2023-07-02T13:45:25.010 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-02T13:45:25.011 DEBUG:teuthology.run:Teuthology command: teuthology --name dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi --owner scheduled_dis@teuthology --description rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} --archive /home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366 --verbose -- /home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366/orig.config.yaml 2023-07-02T13:45:25.043 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-02T13:45:25.132 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366 branch: pacific description: rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} email: idryomov@gmail.com first_in_suite: false job_id: '7324366' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: pacific ceph: conf: global: mon client directed command retry: 5 ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request sha1: 8934282de75bfae9ed7c114e0bb6e46fc588d851 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: extra_packages: - rbd-nbd extra_system_packages: - pv flavor: default sha1: 8934282de75bfae9ed7c114e0bb6e46fc588d851 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-61616-pacific sha1: cbd88633a9c19779ad4b2e367bdc17f83b3a065d owner: scheduled_dis@teuthology priority: 74 repo: https://github.com/ceph/ceph roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - - client.0 seed: 3615 sha1: 8934282de75bfae9ed7c114e0bb6e46fc588d851 sleep_before_teardown: 0 subset: 25/99 suite: rbd suite_branch: wip-61616-pacific suite_path: /home/teuthworker/src/github.com_idryomov_ceph_cbd88633a9c19779ad4b2e367bdc17f83b3a065d/qa suite_relpath: qa suite_repo: https://github.com/idryomov/ceph suite_sha1: cbd88633a9c19779ad4b2e367bdc17f83b3a065d targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKAKQdK43fJ67ZR0UFSSd/x9YyqkWqm7r0EvMSQqiplezjVhkRix4wwgl7eot1veKhyPZmpw4xJNdOF0LHYyJls= smithi022.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAJsgII6kEeeCVaBI90i/iehAOPSQrDC/rPxYXSo44UgV74Crx/QB4JOdm21MI75Pv9b9R4Vt8I2AzdTkMoETQA= smithi163.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlG8ZmXbNG1kKGtt6qrh3TXluF3kt7nELxpD4U3sZvPxNmfk+i05F0KvaqgWwitlcawW5gZzWAqd4jeTIW8MCTKW7jA7Ipp/X6WjdLPEzd3Gl4HBzLACoV13x2/ep16BUlgQgBoDBrf7ye4s5Ud/a2ARRZFB0s5q2Pa+oyjTJjv6SeNkn5rqA5zIwm9+g6bkl7MZtNxHBcm5SPIZpFLJ7P9edWZh5820pOd5LjZEEutwGxUxTUf6TipiRoZYW+oFLTGHrgIFbiUPsZDhhRj9/mJZmhF8JcvkRRTNUQl2Mgl7meT556UkqLDtwbkAaIrOA58XqnvIaTByEaC4Bw1CpuntlENFsJR40BhLj9GgOa7AmDOkurkVwG3yQldNdDQ8EVtX7xRlXDRqHKy6EYh5TMB/zYrNmFiSGFIEKhDzgaRQCAngZ72NVQ3EAlzdqBqQhZlkn3zR71CXe5rzHm4TaZCKIfcFc6UZ+bWSXYs6ob5sTB3YRD788HYsqsuCikcTE= tasks: - install: null - ceph: null - thrashosds: timeout: 1200 - workunit: clients: client.0: - rbd/diff_continuous.sh env: RBD_DEVICE_TYPE: nbd teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b53f236c1077f1d295baafb9e0ca9a4b447019d7 timestamp: 2023-07-02_13:35:41 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-02T13:45:25.132 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_idryomov_ceph_cbd88633a9c19779ad4b2e367bdc17f83b3a065d/qa; will attempt to use it 2023-07-02T13:45:25.133 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_idryomov_ceph_cbd88633a9c19779ad4b2e367bdc17f83b3a065d/qa/tasks 2023-07-02T13:45:25.133 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-02T13:45:25.135 INFO:teuthology.task.internal:Checking packages... 2023-07-02T13:45:25.164 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '8934282de75bfae9ed7c114e0bb6e46fc588d851' 2023-07-02T13:45:25.164 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-02T13:45:25.165 INFO:teuthology.packaging:ref: None 2023-07-02T13:45:25.165 INFO:teuthology.packaging:tag: None 2023-07-02T13:45:25.165 INFO:teuthology.packaging:branch: pacific 2023-07-02T13:45:25.165 INFO:teuthology.packaging:sha1: 8934282de75bfae9ed7c114e0bb6e46fc588d851 2023-07-02T13:45:25.166 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2023-07-02T13:45:25.363 INFO:teuthology.task.internal:Found packages for ceph version 16.2.13-280.g8934282d 2023-07-02T13:45:25.363 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-02T13:45:25.371 INFO:teuthology.task.internal:no buildpackages task found 2023-07-02T13:45:25.371 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-02T13:45:25.421 INFO:teuthology.task.internal:Saving configuration 2023-07-02T13:45:25.434 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-02T13:45:25.447 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-02T13:45:25.469 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-02 13:36:35.227001', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKAKQdK43fJ67ZR0UFSSd/x9YyqkWqm7r0EvMSQqiplezjVhkRix4wwgl7eot1veKhyPZmpw4xJNdOF0LHYyJls='} 2023-07-02T13:45:25.488 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi022.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-02 13:36:35.229451', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAJsgII6kEeeCVaBI90i/iehAOPSQrDC/rPxYXSo44UgV74Crx/QB4JOdm21MI75Pv9b9R4Vt8I2AzdTkMoETQA='} 2023-07-02T13:45:25.508 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-02 13:36:35.228345', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFaaGHzck6+XqhY4JFYUD+C6Ou2tJfw3pVD7A3KKlNX4roBvmKeJOagoQvqdU9RmySDwt8SF+OH1+gl3jIJT9lw='} 2023-07-02T13:45:25.508 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-02T13:45:25.515 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2023-07-02T13:45:25.515 INFO:teuthology.task.internal:roles: ubuntu@smithi022.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5'] 2023-07-02T13:45:25.515 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['client.0'] 2023-07-02T13:45:25.516 INFO:teuthology.run_tasks:Running task console_log... 2023-07-02T13:45:25.639 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f677c08a0d0>, signals=[15]) 2023-07-02T13:45:25.640 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-02T13:45:25.647 INFO:teuthology.task.internal:Opening connections... 2023-07-02T13:45:25.647 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2023-07-02T13:45:25.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T13:45:25.719 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi022.front.sepia.ceph.com 2023-07-02T13:45:25.720 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T13:45:25.787 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi163.front.sepia.ceph.com 2023-07-02T13:45:25.788 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T13:45:25.856 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-02T13:45:25.865 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2023-07-02T13:45:25.883 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2023-07-02T13:45:25.884 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2023-07-02T13:45:25.939 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="Red Hat Enterprise Linux" 2023-07-02T13:45:25.940 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="8.4 (Ootpa)" 2023-07-02T13:45:25.940 INFO:teuthology.orchestra.run.smithi016.stdout:ID="rhel" 2023-07-02T13:45:25.940 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="fedora" 2023-07-02T13:45:25.940 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="8.4" 2023-07-02T13:45:25.940 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el8" 2023-07-02T13:45:25.941 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-07-02T13:45:25.941 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2023-07-02T13:45:25.941 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-07-02T13:45:25.941 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://www.redhat.com/" 2023-07-02T13:45:25.941 INFO:teuthology.orchestra.run.smithi016.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-02T13:45:25.941 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-02T13:45:25.941 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T13:45:25.942 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-02T13:45:25.942 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-07-02T13:45:25.942 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-02T13:45:25.942 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-07-02T13:45:25.942 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2023-07-02T13:45:25.967 DEBUG:teuthology.orchestra.run.smithi022:> uname -m 2023-07-02T13:45:25.985 INFO:teuthology.orchestra.run.smithi022.stdout:x86_64 2023-07-02T13:45:25.985 DEBUG:teuthology.orchestra.run.smithi022:> cat /etc/os-release 2023-07-02T13:45:26.042 INFO:teuthology.orchestra.run.smithi022.stdout:NAME="Red Hat Enterprise Linux" 2023-07-02T13:45:26.042 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION="8.4 (Ootpa)" 2023-07-02T13:45:26.042 INFO:teuthology.orchestra.run.smithi022.stdout:ID="rhel" 2023-07-02T13:45:26.042 INFO:teuthology.orchestra.run.smithi022.stdout:ID_LIKE="fedora" 2023-07-02T13:45:26.043 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION_ID="8.4" 2023-07-02T13:45:26.043 INFO:teuthology.orchestra.run.smithi022.stdout:PLATFORM_ID="platform:el8" 2023-07-02T13:45:26.043 INFO:teuthology.orchestra.run.smithi022.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-07-02T13:45:26.043 INFO:teuthology.orchestra.run.smithi022.stdout:ANSI_COLOR="0;31" 2023-07-02T13:45:26.043 INFO:teuthology.orchestra.run.smithi022.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-07-02T13:45:26.044 INFO:teuthology.orchestra.run.smithi022.stdout:HOME_URL="https://www.redhat.com/" 2023-07-02T13:45:26.044 INFO:teuthology.orchestra.run.smithi022.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-02T13:45:26.044 INFO:teuthology.orchestra.run.smithi022.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-02T13:45:26.045 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T13:45:26.045 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-02T13:45:26.045 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-07-02T13:45:26.045 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-02T13:45:26.046 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-07-02T13:45:26.046 INFO:teuthology.lock.ops:Updating smithi022.front.sepia.ceph.com on lock server 2023-07-02T13:45:26.067 DEBUG:teuthology.orchestra.run.smithi163:> uname -m 2023-07-02T13:45:26.085 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2023-07-02T13:45:26.085 DEBUG:teuthology.orchestra.run.smithi163:> cat /etc/os-release 2023-07-02T13:45:26.139 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="Red Hat Enterprise Linux" 2023-07-02T13:45:26.139 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="8.4 (Ootpa)" 2023-07-02T13:45:26.140 INFO:teuthology.orchestra.run.smithi163.stdout:ID="rhel" 2023-07-02T13:45:26.140 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="fedora" 2023-07-02T13:45:26.140 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="8.4" 2023-07-02T13:45:26.140 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el8" 2023-07-02T13:45:26.140 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-07-02T13:45:26.140 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2023-07-02T13:45:26.140 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-07-02T13:45:26.141 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://www.redhat.com/" 2023-07-02T13:45:26.141 INFO:teuthology.orchestra.run.smithi163.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-02T13:45:26.141 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-02T13:45:26.141 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T13:45:26.141 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-02T13:45:26.141 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-07-02T13:45:26.141 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-02T13:45:26.142 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-07-02T13:45:26.142 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2023-07-02T13:45:26.168 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-02T13:45:26.179 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-02T13:45:26.210 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-02T13:45:26.210 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2023-07-02T13:45:26.213 DEBUG:teuthology.orchestra.run.smithi022:> test '!' -e /home/ubuntu/cephtest 2023-07-02T13:45:26.215 DEBUG:teuthology.orchestra.run.smithi163:> test '!' -e /home/ubuntu/cephtest 2023-07-02T13:45:26.230 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-02T13:45:26.242 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-02T13:45:26.242 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2023-07-02T13:45:26.269 DEBUG:teuthology.orchestra.run.smithi022:> test -z $(ls -A /var/lib/ceph) 2023-07-02T13:45:26.272 DEBUG:teuthology.orchestra.run.smithi163:> test -z $(ls -A /var/lib/ceph) 2023-07-02T13:45:26.284 INFO:teuthology.orchestra.run.smithi016.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-07-02T13:45:26.286 INFO:teuthology.orchestra.run.smithi022.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-07-02T13:45:26.288 INFO:teuthology.orchestra.run.smithi163.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-07-02T13:45:26.289 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-02T13:45:26.370 INFO:teuthology.run_tasks:Running task kernel... 2023-07-02T13:45:26.388 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-07-02T13:45:26.389 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-07-02T13:45:26.389 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-02T13:45:26.390 DEBUG:teuthology.orchestra.run.smithi022:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-02T13:45:26.390 DEBUG:teuthology.orchestra.run.smithi163:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-02T13:45:26.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T13:45:26.406 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2023-07-02T13:45:26.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T13:45:26.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T13:45:26.408 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2023-07-02T13:45:26.409 DEBUG:teuthology.orchestra.run.smithi022:> uname -r 2023-07-02T13:45:26.461 INFO:teuthology.orchestra.run.smithi163.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-07-02T13:45:26.462 INFO:teuthology.task.kernel:Running kernel on smithi163: 4.18.0-305.25.1.el8_4.x86_64 2023-07-02T13:45:26.462 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum install -y kernel 2023-07-02T13:45:26.465 INFO:teuthology.orchestra.run.smithi016.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-07-02T13:45:26.466 INFO:teuthology.orchestra.run.smithi022.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-07-02T13:45:26.466 INFO:teuthology.task.kernel:Running kernel on smithi016: 4.18.0-305.25.1.el8_4.x86_64 2023-07-02T13:45:26.466 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum install -y kernel 2023-07-02T13:45:26.467 INFO:teuthology.task.kernel:Running kernel on smithi022: 4.18.0-305.25.1.el8_4.x86_64 2023-07-02T13:45:26.467 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum install -y kernel 2023-07-02T13:45:27.900 INFO:teuthology.orchestra.run.smithi016.stdout:Updating Subscription Management repositories. 2023-07-02T13:45:27.901 INFO:teuthology.orchestra.run.smithi016.stdout:Unable to read consumer identity 2023-07-02T13:45:27.976 INFO:teuthology.orchestra.run.smithi163.stdout:Updating Subscription Management repositories. 2023-07-02T13:45:27.977 INFO:teuthology.orchestra.run.smithi163.stdout:Unable to read consumer identity 2023-07-02T13:45:27.988 INFO:teuthology.orchestra.run.smithi022.stdout:Updating Subscription Management repositories. 2023-07-02T13:45:27.989 INFO:teuthology.orchestra.run.smithi022.stdout:Unable to read consumer identity 2023-07-02T13:45:31.198 INFO:teuthology.orchestra.run.smithi016.stdout:Copr repo for python3-asyncssh owned by ceph 1.8 kB/s | 3.5 kB 00:01 2023-07-02T13:45:31.633 INFO:teuthology.orchestra.run.smithi022.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2023-07-02T13:45:31.918 INFO:teuthology.orchestra.run.smithi163.stdout:Copr repo for python3-asyncssh owned by ceph 1.3 kB/s | 3.5 kB 00:02 2023-07-02T13:45:33.003 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 8.8 MB/s | 16 MB 00:01 2023-07-02T13:45:33.641 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 16 MB 00:01 2023-07-02T13:45:36.310 INFO:teuthology.orchestra.run.smithi022.stdout:Extra Packages for Enterprise Linux 3.4 MB/s | 16 MB 00:04 2023-07-02T13:45:36.540 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2023-07-02T13:45:37.019 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2023-07-02T13:45:37.250 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-07-02T13:45:37.251 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-07-02T13:45:37.251 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-07-02T13:45:37.403 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2023-07-02T13:45:37.404 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2023-07-02T13:45:37.404 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2023-07-02T13:45:37.526 DEBUG:teuthology.orchestra.run.smithi016:> echo no | sudo yum reinstall kernel || true 2023-07-02T13:45:37.771 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-07-02T13:45:37.772 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-07-02T13:45:37.772 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-07-02T13:45:37.892 INFO:teuthology.orchestra.run.smithi016.stdout:Updating Subscription Management repositories. 2023-07-02T13:45:37.893 INFO:teuthology.orchestra.run.smithi016.stdout:Unable to read consumer identity 2023-07-02T13:45:37.934 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-07-02T13:45:37.935 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2023-07-02T13:45:37.935 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-07-02T13:45:38.068 DEBUG:teuthology.orchestra.run.smithi163:> echo no | sudo yum reinstall kernel || true 2023-07-02T13:45:38.196 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:02 ago on Sun 02 Jul 2023 01:45:36 PM UTC. 2023-07-02T13:45:38.344 INFO:teuthology.orchestra.run.smithi016.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-07-02T13:45:38.344 INFO:teuthology.orchestra.run.smithi016.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-07-02T13:45:38.344 INFO:teuthology.orchestra.run.smithi016.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-07-02T13:45:38.375 INFO:teuthology.orchestra.run.smithi016.stderr:Error: No packages marked for reinstall. 2023-07-02T13:45:38.410 INFO:teuthology.orchestra.run.smithi163.stdout:Updating Subscription Management repositories. 2023-07-02T13:45:38.410 INFO:teuthology.orchestra.run.smithi163.stdout:Unable to read consumer identity 2023-07-02T13:45:38.426 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum reinstall -y kernel || true 2023-07-02T13:45:38.698 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:01 ago on Sun 02 Jul 2023 01:45:37 PM UTC. 2023-07-02T13:45:38.790 INFO:teuthology.orchestra.run.smithi016.stdout:Updating Subscription Management repositories. 2023-07-02T13:45:38.791 INFO:teuthology.orchestra.run.smithi016.stdout:Unable to read consumer identity 2023-07-02T13:45:38.838 INFO:teuthology.orchestra.run.smithi163.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-07-02T13:45:38.838 INFO:teuthology.orchestra.run.smithi163.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-07-02T13:45:38.839 INFO:teuthology.orchestra.run.smithi163.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-07-02T13:45:38.877 INFO:teuthology.orchestra.run.smithi163.stderr:Error: No packages marked for reinstall. 2023-07-02T13:45:38.928 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum reinstall -y kernel || true 2023-07-02T13:45:39.083 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:03 ago on Sun 02 Jul 2023 01:45:36 PM UTC. 2023-07-02T13:45:39.227 INFO:teuthology.orchestra.run.smithi016.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-07-02T13:45:39.227 INFO:teuthology.orchestra.run.smithi016.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-07-02T13:45:39.227 INFO:teuthology.orchestra.run.smithi016.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-07-02T13:45:39.245 INFO:teuthology.orchestra.run.smithi016.stderr:Error: No packages marked for reinstall. 2023-07-02T13:45:39.272 INFO:teuthology.orchestra.run.smithi163.stdout:Updating Subscription Management repositories. 2023-07-02T13:45:39.272 INFO:teuthology.orchestra.run.smithi163.stdout:Unable to read consumer identity 2023-07-02T13:45:39.300 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2023-07-02T13:45:39.429 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:39.430 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:39.430 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:39.430 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-07-02T13:45:39.431 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-07-02T13:45:39.431 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-07-02T13:45:39.431 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi016.front.sepia.ceph.com, path=None, version=distro) 2023-07-02T13:45:39.431 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2023-07-02T13:45:39.552 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:02 ago on Sun 02 Jul 2023 01:45:37 PM UTC. 2023-07-02T13:45:39.561 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:39.562 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:39.562 DEBUG:teuthology.orchestra.run.smithi016:> sudo rpm -qi grub2-tools 2023-07-02T13:45:39.590 INFO:teuthology.orchestra.run.smithi016.stdout:Name : grub2-tools 2023-07-02T13:45:39.590 INFO:teuthology.orchestra.run.smithi016.stdout:Epoch : 1 2023-07-02T13:45:39.590 INFO:teuthology.orchestra.run.smithi016.stdout:Version : 2.02 2023-07-02T13:45:39.590 INFO:teuthology.orchestra.run.smithi016.stdout:Release : 99.el8_4.1 2023-07-02T13:45:39.590 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: x86_64 2023-07-02T13:45:39.590 INFO:teuthology.orchestra.run.smithi016.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-07-02T13:45:39.591 INFO:teuthology.orchestra.run.smithi016.stdout:Group : System Environment/Base 2023-07-02T13:45:39.591 INFO:teuthology.orchestra.run.smithi016.stdout:Size : 9264631 2023-07-02T13:45:39.591 INFO:teuthology.orchestra.run.smithi016.stdout:License : GPLv3+ 2023-07-02T13:45:39.591 INFO:teuthology.orchestra.run.smithi016.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-07-02T13:45:39.591 INFO:teuthology.orchestra.run.smithi016.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-07-02T13:45:39.591 INFO:teuthology.orchestra.run.smithi016.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-07-02T13:45:39.591 INFO:teuthology.orchestra.run.smithi016.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-07-02T13:45:39.591 INFO:teuthology.orchestra.run.smithi016.stdout:Relocations : (not relocatable) 2023-07-02T13:45:39.592 INFO:teuthology.orchestra.run.smithi016.stdout:Packager : Red Hat, Inc. 2023-07-02T13:45:39.592 INFO:teuthology.orchestra.run.smithi016.stdout:Vendor : Red Hat, Inc. 2023-07-02T13:45:39.592 INFO:teuthology.orchestra.run.smithi016.stdout:URL : http://www.gnu.org/software/grub/ 2023-07-02T13:45:39.592 INFO:teuthology.orchestra.run.smithi016.stdout:Summary : Support tools for GRUB. 2023-07-02T13:45:39.592 INFO:teuthology.orchestra.run.smithi016.stdout:Description : 2023-07-02T13:45:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T13:45:39.592 INFO:teuthology.orchestra.run.smithi016.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-07-02T13:45:39.593 INFO:teuthology.orchestra.run.smithi016.stdout:customizable bootloader with modular architecture. It supports a rich 2023-07-02T13:45:39.593 INFO:teuthology.orchestra.run.smithi016.stdout:variety of kernel formats, file systems, computer architectures and 2023-07-02T13:45:39.593 INFO:teuthology.orchestra.run.smithi016.stdout:hardware devices. 2023-07-02T13:45:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T13:45:39.593 INFO:teuthology.orchestra.run.smithi016.stdout:This subpackage provides tools for support of all platforms. 2023-07-02T13:45:39.594 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-07-02T13:45:39.594 INFO:teuthology.task.kernel:Updating grub on smithi016 to boot 4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:39.594 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-07-02T13:45:39.708 INFO:teuthology.orchestra.run.smithi163.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-07-02T13:45:39.708 INFO:teuthology.orchestra.run.smithi163.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-07-02T13:45:39.708 INFO:teuthology.orchestra.run.smithi163.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-07-02T13:45:39.726 INFO:teuthology.orchestra.run.smithi163.stderr:Error: No packages marked for reinstall. 2023-07-02T13:45:39.764 INFO:teuthology.orchestra.run.smithi022.stdout:lab-extras 4.8 MB/s | 24 kB 00:00 2023-07-02T13:45:39.775 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2023-07-02T13:45:39.899 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:39.900 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:39.900 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:39.900 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-07-02T13:45:39.900 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-07-02T13:45:39.901 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2023-07-02T13:45:39.901 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi163.front.sepia.ceph.com, path=None, version=distro) 2023-07-02T13:45:39.901 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2023-07-02T13:45:40.025 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:40.026 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:40.027 DEBUG:teuthology.orchestra.run.smithi163:> sudo rpm -qi grub2-tools 2023-07-02T13:45:40.053 INFO:teuthology.orchestra.run.smithi163.stdout:Name : grub2-tools 2023-07-02T13:45:40.053 INFO:teuthology.orchestra.run.smithi163.stdout:Epoch : 1 2023-07-02T13:45:40.054 INFO:teuthology.orchestra.run.smithi163.stdout:Version : 2.02 2023-07-02T13:45:40.054 INFO:teuthology.orchestra.run.smithi163.stdout:Release : 99.el8_4.1 2023-07-02T13:45:40.054 INFO:teuthology.orchestra.run.smithi163.stdout:Architecture: x86_64 2023-07-02T13:45:40.054 INFO:teuthology.orchestra.run.smithi163.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-07-02T13:45:40.054 INFO:teuthology.orchestra.run.smithi163.stdout:Group : System Environment/Base 2023-07-02T13:45:40.054 INFO:teuthology.orchestra.run.smithi163.stdout:Size : 9264631 2023-07-02T13:45:40.054 INFO:teuthology.orchestra.run.smithi163.stdout:License : GPLv3+ 2023-07-02T13:45:40.055 INFO:teuthology.orchestra.run.smithi163.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-07-02T13:45:40.055 INFO:teuthology.orchestra.run.smithi163.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-07-02T13:45:40.055 INFO:teuthology.orchestra.run.smithi163.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-07-02T13:45:40.055 INFO:teuthology.orchestra.run.smithi163.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-07-02T13:45:40.055 INFO:teuthology.orchestra.run.smithi163.stdout:Relocations : (not relocatable) 2023-07-02T13:45:40.055 INFO:teuthology.orchestra.run.smithi163.stdout:Packager : Red Hat, Inc. 2023-07-02T13:45:40.055 INFO:teuthology.orchestra.run.smithi163.stdout:Vendor : Red Hat, Inc. 2023-07-02T13:45:40.056 INFO:teuthology.orchestra.run.smithi163.stdout:URL : http://www.gnu.org/software/grub/ 2023-07-02T13:45:40.056 INFO:teuthology.orchestra.run.smithi163.stdout:Summary : Support tools for GRUB. 2023-07-02T13:45:40.056 INFO:teuthology.orchestra.run.smithi163.stdout:Description : 2023-07-02T13:45:40.056 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T13:45:40.056 INFO:teuthology.orchestra.run.smithi163.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-07-02T13:45:40.056 INFO:teuthology.orchestra.run.smithi163.stdout:customizable bootloader with modular architecture. It supports a rich 2023-07-02T13:45:40.057 INFO:teuthology.orchestra.run.smithi163.stdout:variety of kernel formats, file systems, computer architectures and 2023-07-02T13:45:40.057 INFO:teuthology.orchestra.run.smithi163.stdout:hardware devices. 2023-07-02T13:45:40.057 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T13:45:40.057 INFO:teuthology.orchestra.run.smithi163.stdout:This subpackage provides tools for support of all platforms. 2023-07-02T13:45:40.058 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-07-02T13:45:40.058 INFO:teuthology.task.kernel:Updating grub on smithi163 to boot 4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:40.058 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-07-02T13:45:40.539 INFO:teuthology.orchestra.run.smithi022.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-07-02T13:45:40.539 INFO:teuthology.orchestra.run.smithi022.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-07-02T13:45:40.539 INFO:teuthology.orchestra.run.smithi022.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-07-02T13:45:40.711 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-07-02T13:45:40.712 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2023-07-02T13:45:40.713 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-07-02T13:45:40.823 DEBUG:teuthology.orchestra.run.smithi022:> echo no | sudo yum reinstall kernel || true 2023-07-02T13:45:40.833 INFO:teuthology.orchestra.run.smithi016.stderr:Generating grub configuration file ... 2023-07-02T13:45:41.226 INFO:teuthology.orchestra.run.smithi022.stdout:Updating Subscription Management repositories. 2023-07-02T13:45:41.227 INFO:teuthology.orchestra.run.smithi022.stdout:Unable to read consumer identity 2023-07-02T13:45:41.357 INFO:teuthology.orchestra.run.smithi163.stderr:Generating grub configuration file ... 2023-07-02T13:45:41.529 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:02 ago on Sun 02 Jul 2023 01:45:39 PM UTC. 2023-07-02T13:45:41.671 INFO:teuthology.orchestra.run.smithi022.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-07-02T13:45:41.671 INFO:teuthology.orchestra.run.smithi022.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-07-02T13:45:41.672 INFO:teuthology.orchestra.run.smithi022.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-07-02T13:45:41.696 INFO:teuthology.orchestra.run.smithi022.stderr:Error: No packages marked for reinstall. 2023-07-02T13:45:41.748 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum reinstall -y kernel || true 2023-07-02T13:45:42.111 INFO:teuthology.orchestra.run.smithi022.stdout:Updating Subscription Management repositories. 2023-07-02T13:45:42.111 INFO:teuthology.orchestra.run.smithi022.stdout:Unable to read consumer identity 2023-07-02T13:45:42.406 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:03 ago on Sun 02 Jul 2023 01:45:39 PM UTC. 2023-07-02T13:45:42.554 INFO:teuthology.orchestra.run.smithi022.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-07-02T13:45:42.554 INFO:teuthology.orchestra.run.smithi022.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-07-02T13:45:42.554 INFO:teuthology.orchestra.run.smithi022.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-07-02T13:45:42.563 INFO:teuthology.orchestra.run.smithi022.stderr:Error: No packages marked for reinstall. 2023-07-02T13:45:42.613 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q kernel | sort -rV | head -n 1 2023-07-02T13:45:42.670 INFO:teuthology.orchestra.run.smithi016.stderr:done 2023-07-02T13:45:42.672 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2023-07-02T13:45:42.686 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.60vs2yj6Xa 2023-07-02T13:45:42.687 DEBUG:teuthology.orchestra.run.smithi016:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.60vs2yj6Xa 2023-07-02T13:45:42.743 INFO:teuthology.orchestra.run.smithi022.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:42.743 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:42.744 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:42.744 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-07-02T13:45:42.744 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-07-02T13:45:42.744 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-07-02T13:45:42.744 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi022.front.sepia.ceph.com, path=None, version=distro) 2023-07-02T13:45:42.745 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q kernel | sort -rV | head -n 1 2023-07-02T13:45:42.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.60vs2yj6Xa 2023-07-02T13:45:42.874 INFO:teuthology.orchestra.run.smithi022.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:42.875 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:42.875 DEBUG:teuthology.orchestra.run.smithi022:> sudo rpm -qi grub2-tools 2023-07-02T13:45:42.902 INFO:teuthology.orchestra.run.smithi022.stdout:Name : grub2-tools 2023-07-02T13:45:42.902 INFO:teuthology.orchestra.run.smithi022.stdout:Epoch : 1 2023-07-02T13:45:42.903 INFO:teuthology.orchestra.run.smithi022.stdout:Version : 2.02 2023-07-02T13:45:42.903 INFO:teuthology.orchestra.run.smithi022.stdout:Release : 99.el8_4.1 2023-07-02T13:45:42.903 INFO:teuthology.orchestra.run.smithi022.stdout:Architecture: x86_64 2023-07-02T13:45:42.903 INFO:teuthology.orchestra.run.smithi022.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-07-02T13:45:42.903 INFO:teuthology.orchestra.run.smithi022.stdout:Group : System Environment/Base 2023-07-02T13:45:42.903 INFO:teuthology.orchestra.run.smithi022.stdout:Size : 9264631 2023-07-02T13:45:42.903 INFO:teuthology.orchestra.run.smithi022.stdout:License : GPLv3+ 2023-07-02T13:45:42.903 INFO:teuthology.orchestra.run.smithi022.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-07-02T13:45:42.904 INFO:teuthology.orchestra.run.smithi022.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-07-02T13:45:42.904 INFO:teuthology.orchestra.run.smithi022.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-07-02T13:45:42.904 INFO:teuthology.orchestra.run.smithi022.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-07-02T13:45:42.904 INFO:teuthology.orchestra.run.smithi022.stdout:Relocations : (not relocatable) 2023-07-02T13:45:42.904 INFO:teuthology.orchestra.run.smithi022.stdout:Packager : Red Hat, Inc. 2023-07-02T13:45:42.904 INFO:teuthology.orchestra.run.smithi022.stdout:Vendor : Red Hat, Inc. 2023-07-02T13:45:42.904 INFO:teuthology.orchestra.run.smithi022.stdout:URL : http://www.gnu.org/software/grub/ 2023-07-02T13:45:42.905 INFO:teuthology.orchestra.run.smithi022.stdout:Summary : Support tools for GRUB. 2023-07-02T13:45:42.905 INFO:teuthology.orchestra.run.smithi022.stdout:Description : 2023-07-02T13:45:42.905 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T13:45:42.905 INFO:teuthology.orchestra.run.smithi022.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-07-02T13:45:42.905 INFO:teuthology.orchestra.run.smithi022.stdout:customizable bootloader with modular architecture. It supports a rich 2023-07-02T13:45:42.905 INFO:teuthology.orchestra.run.smithi022.stdout:variety of kernel formats, file systems, computer architectures and 2023-07-02T13:45:42.905 INFO:teuthology.orchestra.run.smithi022.stdout:hardware devices. 2023-07-02T13:45:42.905 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T13:45:42.906 INFO:teuthology.orchestra.run.smithi022.stdout:This subpackage provides tools for support of all platforms. 2023-07-02T13:45:42.907 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-07-02T13:45:42.907 INFO:teuthology.task.kernel:Updating grub on smithi022 to boot 4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:42.907 DEBUG:teuthology.orchestra.run.smithi022:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-07-02T13:45:42.909 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.60vs2yj6Xa is 6KB 2023-07-02T13:45:42.926 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.60vs2yj6Xa 2023-07-02T13:45:42.939 DEBUG:teuthology.orchestra.run.smithi016:> sudo /bin/ls /boot/loader/entries || true 2023-07-02T13:45:43.009 INFO:teuthology.orchestra.run.smithi016.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-07-02T13:45:43.009 INFO:teuthology.orchestra.run.smithi016.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-07-02T13:45:43.009 INFO:teuthology.orchestra.run.smithi016.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-07-02T13:45:43.009 INFO:teuthology.orchestra.run.smithi016.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-07-02T13:45:43.010 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:43.180 DEBUG:teuthology.orchestra.run.smithi016:> sudo shutdown -r now 2023-07-02T13:45:43.463 INFO:teuthology.orchestra.run.smithi163.stderr:done 2023-07-02T13:45:43.465 DEBUG:teuthology.orchestra.run.smithi163:> mktemp 2023-07-02T13:45:43.478 INFO:teuthology.orchestra.run.smithi163.stdout:/tmp/tmp.WeQWpSOMzq 2023-07-02T13:45:43.479 DEBUG:teuthology.orchestra.run.smithi163:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WeQWpSOMzq 2023-07-02T13:45:43.559 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0666 /tmp/tmp.WeQWpSOMzq 2023-07-02T13:45:43.708 DEBUG:teuthology.orchestra.remote:smithi163:/tmp/tmp.WeQWpSOMzq is 6KB 2023-07-02T13:45:43.754 DEBUG:teuthology.orchestra.run.smithi163:> rm -fr /tmp/tmp.WeQWpSOMzq 2023-07-02T13:45:43.767 DEBUG:teuthology.orchestra.run.smithi163:> sudo /bin/ls /boot/loader/entries || true 2023-07-02T13:45:43.833 INFO:teuthology.orchestra.run.smithi163.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-07-02T13:45:43.834 INFO:teuthology.orchestra.run.smithi163.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-07-02T13:45:43.834 INFO:teuthology.orchestra.run.smithi163.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-07-02T13:45:43.834 INFO:teuthology.orchestra.run.smithi163.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-07-02T13:45:43.835 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:44.010 DEBUG:teuthology.orchestra.run.smithi163:> sudo shutdown -r now 2023-07-02T13:45:44.088 INFO:teuthology.orchestra.run.smithi022.stderr:Generating grub configuration file ... 2023-07-02T13:45:46.150 INFO:teuthology.orchestra.run.smithi022.stderr:done 2023-07-02T13:45:46.152 DEBUG:teuthology.orchestra.run.smithi022:> mktemp 2023-07-02T13:45:46.165 INFO:teuthology.orchestra.run.smithi022.stdout:/tmp/tmp.zhfyulRgCh 2023-07-02T13:45:46.166 DEBUG:teuthology.orchestra.run.smithi022:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zhfyulRgCh 2023-07-02T13:45:46.242 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0666 /tmp/tmp.zhfyulRgCh 2023-07-02T13:45:46.395 DEBUG:teuthology.orchestra.remote:smithi022:/tmp/tmp.zhfyulRgCh is 6KB 2023-07-02T13:45:46.412 DEBUG:teuthology.orchestra.run.smithi022:> rm -fr /tmp/tmp.zhfyulRgCh 2023-07-02T13:45:46.426 DEBUG:teuthology.orchestra.run.smithi022:> sudo /bin/ls /boot/loader/entries || true 2023-07-02T13:45:46.492 INFO:teuthology.orchestra.run.smithi022.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-07-02T13:45:46.492 INFO:teuthology.orchestra.run.smithi022.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-07-02T13:45:46.492 INFO:teuthology.orchestra.run.smithi022.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-07-02T13:45:46.492 INFO:teuthology.orchestra.run.smithi022.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-07-02T13:45:46.493 DEBUG:teuthology.orchestra.run.smithi022:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-07-02T13:45:46.643 DEBUG:teuthology.orchestra.run.smithi022:> sudo shutdown -r now 2023-07-02T13:46:13.208 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-07-02T13:46:13.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-07-02T13:46:13.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T13:46:14.013 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-07-02T13:46:14.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-07-02T13:46:14.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T13:46:16.648 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-07-02T13:46:16.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2023-07-02T13:46:16.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T13:46:31.664 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2023-07-02T13:46:32.436 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2023-07-02T13:46:40.672 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-07-02T13:46:40.673 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T13:46:41.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-07-02T13:46:41.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T13:46:44.500 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2023-07-02T13:46:56.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-07-02T13:46:56.514 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T13:46:59.568 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2023-07-02T13:47:14.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-07-02T13:47:14.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T13:47:16.652 DEBUG:teuthology.orchestra.remote:timed out 2023-07-02T13:47:25.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2023-07-02T13:47:25.661 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T13:47:26.889 DEBUG:teuthology.orchestra.run.smithi022:> true 2023-07-02T13:47:27.718 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi022.front.sepia.ceph.com' 2023-07-02T13:47:27.718 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-425.3.1.el8.x86_64"... 2023-07-02T13:47:27.719 DEBUG:teuthology.orchestra.run.smithi022:> uname -r 2023-07-02T13:47:27.733 INFO:teuthology.orchestra.run.smithi022.stdout:4.18.0-425.3.1.el8.x86_64 2023-07-02T13:47:27.733 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-07-02T13:47:27.733 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-07-02T13:47:27.734 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-07-02T13:47:28.734 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-07-02T13:47:28.735 DEBUG:teuthology.orchestra.run.smithi022:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-02T13:47:28.827 INFO:teuthology.orchestra.run.smithi022.stdout:ttyS1 2023-07-02T13:47:28.881 DEBUG:teuthology.parallel:result is None 2023-07-02T13:47:33.104 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2023-07-02T13:47:40.674 DEBUG:teuthology.orchestra.remote:timed out 2023-07-02T13:47:51.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-07-02T13:47:51.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T13:47:52.019 DEBUG:teuthology.orchestra.run.smithi163:> true 2023-07-02T13:47:52.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-07-02T13:47:52.676 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T13:47:52.699 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-07-02T13:47:52.699 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "4.18.0-425.3.1.el8.x86_64"... 2023-07-02T13:47:52.700 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2023-07-02T13:47:52.714 INFO:teuthology.orchestra.run.smithi163.stdout:4.18.0-425.3.1.el8.x86_64 2023-07-02T13:47:52.714 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-07-02T13:47:52.714 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-07-02T13:47:52.714 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-07-02T13:47:52.938 DEBUG:teuthology.orchestra.run.smithi016:> true 2023-07-02T13:47:53.200 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-07-02T13:47:53.200 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-425.3.1.el8.x86_64"... 2023-07-02T13:47:53.200 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2023-07-02T13:47:53.215 INFO:teuthology.orchestra.run.smithi016.stdout:4.18.0-425.3.1.el8.x86_64 2023-07-02T13:47:53.216 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-07-02T13:47:53.216 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-07-02T13:47:53.216 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-07-02T13:47:53.716 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2023-07-02T13:47:53.716 DEBUG:teuthology.orchestra.run.smithi163:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-02T13:47:53.793 INFO:teuthology.orchestra.run.smithi163.stdout:ttyS1 2023-07-02T13:47:53.805 DEBUG:teuthology.parallel:result is None 2023-07-02T13:47:54.217 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-07-02T13:47:54.218 DEBUG:teuthology.orchestra.run.smithi016:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-02T13:47:54.281 INFO:teuthology.orchestra.run.smithi016.stdout:ttyS1 2023-07-02T13:47:54.293 DEBUG:teuthology.parallel:result is None 2023-07-02T13:47:54.294 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-02T13:47:54.301 INFO:teuthology.task.internal:Creating test directory... 2023-07-02T13:47:54.301 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-02T13:47:54.303 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-02T13:47:54.305 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-02T13:47:54.321 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-02T13:47:54.333 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-02T13:47:54.366 INFO:teuthology.task.internal:Creating archive directory... 2023-07-02T13:47:54.367 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-02T13:47:54.369 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-02T13:47:54.371 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-02T13:47:54.402 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-02T13:47:54.408 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-02T13:47:54.409 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-02T13:47:54.432 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-02T13:47:54.439 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-02T13:47:54.462 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-02T13:47:54.474 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-02T13:47:54.479 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-02T13:47:54.482 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-02T13:47:54.491 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-02T13:47:54.494 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-02T13:47:54.494 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-02T13:47:54.501 INFO:teuthology.task.internal:Configuring sudo... 2023-07-02T13:47:54.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-02T13:47:54.518 DEBUG:teuthology.orchestra.run.smithi022:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-02T13:47:54.534 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-02T13:47:54.560 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-02T13:47:54.569 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-02T13:47:54.570 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-02T13:47:54.586 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-02T13:47:54.602 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-02T13:47:54.617 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-02T13:47:54.684 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-02T13:47:54.722 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-02T13:47:54.806 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-02T13:47:54.844 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-07-02T13:47:54.844 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-02T13:47:54.909 DEBUG:teuthology.orchestra.run.smithi022:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-02T13:47:54.951 DEBUG:teuthology.orchestra.run.smithi022:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-02T13:47:54.997 DEBUG:teuthology.orchestra.run.smithi022:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-02T13:47:55.084 DEBUG:teuthology.orchestra.run.smithi022:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-02T13:47:55.124 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-07-02T13:47:55.124 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-02T13:47:55.190 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-02T13:47:55.236 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-02T13:47:55.280 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-02T13:47:55.365 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-02T13:47:55.401 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-07-02T13:47:55.401 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-02T13:47:55.464 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2023-07-02T13:47:55.466 DEBUG:teuthology.orchestra.run.smithi022:> sudo service rsyslog restart 2023-07-02T13:47:55.468 DEBUG:teuthology.orchestra.run.smithi163:> sudo service rsyslog restart 2023-07-02T13:47:55.516 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-02T13:47:55.524 INFO:teuthology.orchestra.run.smithi022.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-02T13:47:55.554 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-02T13:47:56.037 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-02T13:47:56.045 INFO:teuthology.task.internal:Starting timer... 2023-07-02T13:47:56.045 INFO:teuthology.run_tasks:Running task pcp... 2023-07-02T13:47:56.078 INFO:teuthology.run_tasks:Running task selinux... 2023-07-02T13:47:56.110 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-07-02T13:47:56.111 DEBUG:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2023-07-02T13:47:56.171 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: [ OK ] 2023-07-02T13:47:56.172 DEBUG:teuthology.orchestra.run.smithi022:> sudo service auditd rotate 2023-07-02T13:47:56.235 INFO:teuthology.orchestra.run.smithi022.stdout:Rotating logs: [ OK ] 2023-07-02T13:47:56.236 DEBUG:teuthology.orchestra.run.smithi163:> sudo service auditd rotate 2023-07-02T13:47:56.303 INFO:teuthology.orchestra.run.smithi163.stdout:Rotating logs: [ OK ] 2023-07-02T13:47:56.304 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-02T13:47:56.305 DEBUG:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2023-07-02T13:47:56.343 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2023-07-02T13:47:56.343 DEBUG:teuthology.orchestra.run.smithi022:> /usr/sbin/getenforce 2023-07-02T13:47:56.369 INFO:teuthology.orchestra.run.smithi022.stdout:Permissive 2023-07-02T13:47:56.369 DEBUG:teuthology.orchestra.run.smithi163:> /usr/sbin/getenforce 2023-07-02T13:47:56.397 INFO:teuthology.orchestra.run.smithi163.stdout:Permissive 2023-07-02T13:47:56.397 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi016.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi022.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi163.front.sepia.ceph.com': 'permissive'} 2023-07-02T13:47:56.398 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-02T13:47:56.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T13:47:56.425 DEBUG:teuthology.orchestra.run.smithi022:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-02T13:47:56.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T13:47:56.452 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-02T13:47:56.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T13:47:56.476 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-02T13:47:56.477 DEBUG:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2023-07-02T13:47:56.503 DEBUG:teuthology.orchestra.run.smithi022:> sudo /usr/sbin/setenforce permissive 2023-07-02T13:47:56.533 DEBUG:teuthology.orchestra.run.smithi163:> sudo /usr/sbin/setenforce permissive 2023-07-02T13:47:56.557 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-02T13:47:56.567 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-07-02T13:47:56.572 INFO:teuthology.repo_utils:Fetching main from origin 2023-07-02T13:47:56.705 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-02T13:47:56.723 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-02T13:47:56.725 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com,smithi022.front.sepia.ceph.com,smithi163.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-02T13:56:41.142 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi022.front.sepia.ceph.com'), Remote(name='ubuntu@smithi163.front.sepia.ceph.com')] 2023-07-02T13:56:41.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-07-02T13:56:41.145 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T13:56:41.217 DEBUG:teuthology.orchestra.run.smithi016:> true 2023-07-02T13:56:41.284 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-07-02T13:56:41.284 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2023-07-02T13:56:41.285 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T13:56:41.355 DEBUG:teuthology.orchestra.run.smithi022:> true 2023-07-02T13:56:41.379 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi022.front.sepia.ceph.com' 2023-07-02T13:56:41.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-07-02T13:56:41.381 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-02T13:56:41.453 DEBUG:teuthology.orchestra.run.smithi163:> true 2023-07-02T13:56:41.475 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2023-07-02T13:56:41.476 INFO:teuthology.run_tasks:Running task clock... 2023-07-02T13:56:41.489 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-02T13:56:41.490 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-02T13:56:41.491 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-02T13:56:41.493 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-02T13:56:41.494 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-02T13:56:41.496 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-02T13:56:41.496 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-02T13:56:41.522 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-02T13:56:41.524 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-02T13:56:41.538 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-02T13:56:41.539 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-02T13:56:41.544 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-02T13:56:41.559 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-02T13:56:41.573 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2023-07-02T13:56:41.577 INFO:teuthology.orchestra.run.smithi022.stderr:sudo: ntpd: command not found 2023-07-02T13:56:41.584 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2023-07-02T13:56:41.587 INFO:teuthology.orchestra.run.smithi022.stdout:506 Cannot talk to daemon 2023-07-02T13:56:41.588 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: ntpd: command not found 2023-07-02T13:56:41.598 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-02T13:56:41.599 INFO:teuthology.orchestra.run.smithi163.stdout:506 Cannot talk to daemon 2023-07-02T13:56:41.602 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-02T13:56:41.612 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-02T13:56:41.613 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-02T13:56:41.616 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-02T13:56:41.626 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-02T13:56:41.680 INFO:teuthology.orchestra.run.smithi016.stderr:bash: ntpq: command not found 2023-07-02T13:56:41.683 INFO:teuthology.orchestra.run.smithi016.stdout:210 Number of sources = 4 2023-07-02T13:56:41.683 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-02T13:56:41.683 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2023-07-02T13:56:41.683 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T13:56:41.683 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T13:56:41.683 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T13:56:41.684 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T13:56:41.684 INFO:teuthology.orchestra.run.smithi022.stderr:bash: ntpq: command not found 2023-07-02T13:56:41.686 INFO:teuthology.orchestra.run.smithi022.stdout:210 Number of sources = 4 2023-07-02T13:56:41.686 INFO:teuthology.orchestra.run.smithi022.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-02T13:56:41.686 INFO:teuthology.orchestra.run.smithi022.stdout:=============================================================================== 2023-07-02T13:56:41.687 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T13:56:41.687 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T13:56:41.687 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T13:56:41.687 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T13:56:41.693 INFO:teuthology.orchestra.run.smithi163.stderr:bash: ntpq: command not found 2023-07-02T13:56:41.695 INFO:teuthology.orchestra.run.smithi163.stdout:210 Number of sources = 4 2023-07-02T13:56:41.696 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-02T13:56:41.696 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2023-07-02T13:56:41.697 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T13:56:41.697 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T13:56:41.697 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T13:56:41.697 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T13:56:41.698 INFO:teuthology.run_tasks:Running task install... 2023-07-02T13:56:41.709 DEBUG:teuthology.task.install:project ceph 2023-07-02T13:56:41.710 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'extra_packages': ['rbd-nbd'], 'extra_system_packages': ['pv'], 'flavor': 'default', 'sha1': '8934282de75bfae9ed7c114e0bb6e46fc588d851'}} 2023-07-02T13:56:41.710 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-nbd'], 'extra_system_packages': ['pv'], 'flavor': 'default', 'sha1': '8934282de75bfae9ed7c114e0bb6e46fc588d851'} 2023-07-02T13:56:41.710 INFO:teuthology.task.install:Using flavor: default 2023-07-02T13:56:41.716 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-07-02T13:56:41.716 INFO:teuthology.task.install:extra packages: ['rbd-nbd'] 2023-07-02T13:56:41.718 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-nbd'], 'extra_system_packages': ['pv'], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8934282de75bfae9ed7c114e0bb6e46fc588d851', 'tag': None, 'wait_for_package': False} 2023-07-02T13:56:41.718 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8934282de75bfae9ed7c114e0bb6e46fc588d851 2023-07-02T13:56:41.721 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-nbd'], 'extra_system_packages': ['pv'], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8934282de75bfae9ed7c114e0bb6e46fc588d851', 'tag': None, 'wait_for_package': False} 2023-07-02T13:56:41.721 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8934282de75bfae9ed7c114e0bb6e46fc588d851 2023-07-02T13:56:41.722 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-nbd'], 'extra_system_packages': ['pv'], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8934282de75bfae9ed7c114e0bb6e46fc588d851', 'tag': None, 'wait_for_package': False} 2023-07-02T13:56:41.723 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8934282de75bfae9ed7c114e0bb6e46fc588d851 2023-07-02T13:56:41.876 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/pacific/8934282de75bfae9ed7c114e0bb6e46fc588d851/centos/8/flavors/default/ 2023-07-02T13:56:41.876 INFO:teuthology.task.install.rpm:Package version is 16.2.13-280.g8934282d 2023-07-02T13:56:41.942 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/pacific/8934282de75bfae9ed7c114e0bb6e46fc588d851/centos/8/flavors/default/ 2023-07-02T13:56:41.943 INFO:teuthology.task.install.rpm:Package version is 16.2.13-280.g8934282d 2023-07-02T13:56:41.958 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/pacific/8934282de75bfae9ed7c114e0bb6e46fc588d851/centos/8/flavors/default/ 2023-07-02T13:56:41.958 INFO:teuthology.task.install.rpm:Package version is 16.2.13-280.g8934282d 2023-07-02T13:56:42.060 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/pacific/8934282de75bfae9ed7c114e0bb6e46fc588d851/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/pacific/8934282de75bfae9ed7c114e0bb6e46fc588d851/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/pacific/8934282de75bfae9ed7c114e0bb6e46fc588d851/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-02T13:56:42.061 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-07-02T13:56:42.061 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-02T13:56:42.088 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-nbd, pv, pv, pv, pv on remote rpm x86_64 2023-07-02T13:56:42.089 DEBUG:teuthology.orchestra.run.smithi022:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8934282de75bfae9ed7c114e0bb6e46fc588d851/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-02T13:56:42.116 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/pacific/8934282de75bfae9ed7c114e0bb6e46fc588d851/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/pacific/8934282de75bfae9ed7c114e0bb6e46fc588d851/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/pacific/8934282de75bfae9ed7c114e0bb6e46fc588d851/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-02T13:56:42.117 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-07-02T13:56:42.117 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-02T13:56:42.133 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/pacific/8934282de75bfae9ed7c114e0bb6e46fc588d851/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/pacific/8934282de75bfae9ed7c114e0bb6e46fc588d851/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/pacific/8934282de75bfae9ed7c114e0bb6e46fc588d851/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-02T13:56:42.133 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-07-02T13:56:42.134 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-02T13:56:42.145 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-nbd, pv, pv, pv, pv, pv on remote rpm x86_64 2023-07-02T13:56:42.145 DEBUG:teuthology.orchestra.run.smithi016:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8934282de75bfae9ed7c114e0bb6e46fc588d851/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-02T13:56:42.156 DEBUG:teuthology.orchestra.run.smithi022:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-02T13:56:42.161 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-nbd, pv, pv, pv, pv, pv, pv on remote rpm x86_64 2023-07-02T13:56:42.161 DEBUG:teuthology.orchestra.run.smithi163:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8934282de75bfae9ed7c114e0bb6e46fc588d851/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-02T13:56:42.211 DEBUG:teuthology.orchestra.run.smithi016:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-02T13:56:42.227 DEBUG:teuthology.orchestra.run.smithi163:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-02T13:56:42.238 DEBUG:teuthology.orchestra.run.smithi022:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-02T13:56:42.292 DEBUG:teuthology.orchestra.run.smithi016:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-02T13:56:42.302 DEBUG:teuthology.orchestra.run.smithi163:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-02T13:56:42.304 INFO:teuthology.orchestra.run.smithi022.stdout:check_obsoletes = 1 2023-07-02T13:56:42.305 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum clean all 2023-07-02T13:56:42.357 INFO:teuthology.orchestra.run.smithi016.stdout:check_obsoletes = 1 2023-07-02T13:56:42.359 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum clean all 2023-07-02T13:56:42.367 INFO:teuthology.orchestra.run.smithi163.stdout:check_obsoletes = 1 2023-07-02T13:56:42.368 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum clean all 2023-07-02T13:56:42.715 INFO:teuthology.orchestra.run.smithi022.stdout:Updating Subscription Management repositories. 2023-07-02T13:56:42.759 INFO:teuthology.orchestra.run.smithi163.stdout:Updating Subscription Management repositories. 2023-07-02T13:56:42.771 INFO:teuthology.orchestra.run.smithi016.stdout:Updating Subscription Management repositories. 2023-07-02T13:56:43.514 INFO:teuthology.orchestra.run.smithi016.stdout:44 files removed 2023-07-02T13:56:43.534 INFO:teuthology.orchestra.run.smithi163.stdout:44 files removed 2023-07-02T13:56:43.568 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd rbd-nbd pv pv pv pv pv pv 2023-07-02T13:56:43.586 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd rbd-nbd pv pv pv pv pv pv 2023-07-02T13:56:43.609 INFO:teuthology.orchestra.run.smithi022.stdout:44 files removed 2023-07-02T13:56:43.665 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd rbd-nbd pv pv pv pv pv pv 2023-07-02T13:56:43.935 INFO:teuthology.orchestra.run.smithi163.stdout:Updating Subscription Management repositories. 2023-07-02T13:56:43.937 INFO:teuthology.orchestra.run.smithi016.stdout:Updating Subscription Management repositories. 2023-07-02T13:56:44.037 INFO:teuthology.orchestra.run.smithi022.stdout:Updating Subscription Management repositories. 2023-07-02T13:56:45.599 INFO:teuthology.orchestra.run.smithi163.stdout:ceph packages for x86_64 197 kB/s | 76 kB 00:00 2023-07-02T13:56:45.742 INFO:teuthology.orchestra.run.smithi016.stdout:ceph packages for x86_64 234 kB/s | 76 kB 00:00 2023-07-02T13:56:45.838 INFO:teuthology.orchestra.run.smithi163.stdout:ceph noarch packages 77 kB/s | 15 kB 00:00 2023-07-02T13:56:45.951 INFO:teuthology.orchestra.run.smithi022.stdout:ceph packages for x86_64 259 kB/s | 76 kB 00:00 2023-07-02T13:56:46.044 INFO:teuthology.orchestra.run.smithi016.stdout:ceph noarch packages 59 kB/s | 15 kB 00:00 2023-07-02T13:56:46.182 INFO:teuthology.orchestra.run.smithi163.stdout:ceph source packages 4.6 kB/s | 1.4 kB 00:00 2023-07-02T13:56:46.308 INFO:teuthology.orchestra.run.smithi022.stdout:ceph noarch packages 49 kB/s | 15 kB 00:00 2023-07-02T13:56:46.394 INFO:teuthology.orchestra.run.smithi016.stdout:ceph source packages 4.5 kB/s | 1.4 kB 00:00 2023-07-02T13:56:46.594 INFO:teuthology.orchestra.run.smithi022.stdout:ceph source packages 5.7 kB/s | 1.4 kB 00:00 2023-07-02T13:56:48.072 INFO:teuthology.orchestra.run.smithi163.stdout:Copr repo for python3-asyncssh owned by ceph 1.9 kB/s | 3.5 kB 00:01 2023-07-02T13:56:48.297 INFO:teuthology.orchestra.run.smithi016.stdout:Copr repo for python3-asyncssh owned by ceph 1.9 kB/s | 3.5 kB 00:01 2023-07-02T13:56:48.865 INFO:teuthology.orchestra.run.smithi022.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2023-07-02T13:56:49.316 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:00 2023-07-02T13:56:49.613 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2023-07-02T13:56:50.123 INFO:teuthology.orchestra.run.smithi022.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2023-07-02T13:56:52.746 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 4.1 MB/s | 24 kB 00:00 2023-07-02T13:56:52.939 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 4.1 MB/s | 24 kB 00:00 2023-07-02T13:56:53.390 INFO:teuthology.orchestra.run.smithi016.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 59 MB/s | 33 MB 00:00 2023-07-02T13:56:53.594 INFO:teuthology.orchestra.run.smithi022.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2023-07-02T13:56:53.816 INFO:teuthology.orchestra.run.smithi163.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 42 MB/s | 33 MB 00:00 2023-07-02T13:56:54.464 INFO:teuthology.orchestra.run.smithi022.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 49 MB/s | 38 MB 00:00 2023-07-02T13:56:59.183 INFO:teuthology.orchestra.run.smithi163.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 73 MB/s | 38 MB 00:00 2023-07-02T13:56:59.450 INFO:teuthology.orchestra.run.smithi016.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 MB/s | 38 MB 00:00 2023-07-02T13:57:00.562 INFO:teuthology.orchestra.run.smithi022.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 63 MB/s | 33 MB 00:00 2023-07-02T13:57:07.557 INFO:teuthology.orchestra.run.smithi163.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-02T13:57:07.558 INFO:teuthology.orchestra.run.smithi163.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-02T13:57:08.043 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2023-07-02T13:57:08.053 INFO:teuthology.orchestra.run.smithi163.stdout:====================================================================================================================== 2023-07-02T13:57:08.053 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2023-07-02T13:57:08.054 INFO:teuthology.orchestra.run.smithi163.stdout:====================================================================================================================== 2023-07-02T13:57:08.054 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2023-07-02T13:57:08.054 INFO:teuthology.orchestra.run.smithi163.stdout: ceph x86_64 2:16.2.13-280.g8934282d.el8 ceph 6.4 k 2023-07-02T13:57:08.054 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base x86_64 2:16.2.13-280.g8934282d.el8 ceph 6.1 M 2023-07-02T13:57:08.054 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse x86_64 2:16.2.13-280.g8934282d.el8 ceph 909 k 2023-07-02T13:57:08.055 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache x86_64 2:16.2.13-280.g8934282d.el8 ceph 161 k 2023-07-02T13:57:08.055 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr x86_64 2:16.2.13-280.g8934282d.el8 ceph 1.6 M 2023-07-02T13:57:08.055 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 116 k 2023-07-02T13:57:08.055 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 3.1 M 2023-07-02T13:57:08.055 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 8.6 M 2023-07-02T13:57:08.055 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 27 k 2023-07-02T13:57:08.055 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw x86_64 2:16.2.13-280.g8934282d.el8 ceph 12 M 2023-07-02T13:57:08.056 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test x86_64 2:16.2.13-280.g8934282d.el8 ceph 51 M 2023-07-02T13:57:08.056 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 84 k 2023-07-02T13:57:08.056 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel x86_64 2:16.2.13-280.g8934282d.el8 ceph 25 k 2023-07-02T13:57:08.056 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2 x86_64 2:16.2.13-280.g8934282d.el8 ceph 824 k 2023-07-02T13:57:08.056 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel x86_64 2:16.2.13-280.g8934282d.el8 ceph 131 k 2023-07-02T13:57:08.056 INFO:teuthology.orchestra.run.smithi163.stdout: pv x86_64 1.6.6-7.el8 epel 64 k 2023-07-02T13:57:08.056 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs x86_64 2:16.2.13-280.g8934282d.el8 ceph 215 k 2023-07-02T13:57:08.057 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados x86_64 2:16.2.13-280.g8934282d.el8 ceph 389 k 2023-07-02T13:57:08.057 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd x86_64 2:16.2.13-280.g8934282d.el8 ceph 368 k 2023-07-02T13:57:08.057 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw x86_64 2:16.2.13-280.g8934282d.el8 ceph 114 k 2023-07-02T13:57:08.057 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse x86_64 2:16.2.13-280.g8934282d.el8 ceph 88 k 2023-07-02T13:57:08.057 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror x86_64 2:16.2.13-280.g8934282d.el8 ceph 3.7 M 2023-07-02T13:57:08.058 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd x86_64 2:16.2.13-280.g8934282d.el8 ceph 173 k 2023-07-02T13:57:08.058 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrading: 2023-07-02T13:57:08.058 INFO:teuthology.orchestra.run.smithi163.stdout: librados2 x86_64 2:16.2.13-280.g8934282d.el8 ceph 3.7 M 2023-07-02T13:57:08.058 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1 x86_64 2:16.2.13-280.g8934282d.el8 ceph 4.2 M 2023-07-02T13:57:08.058 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2023-07-02T13:57:08.058 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common x86_64 2:16.2.13-280.g8934282d.el8 ceph 24 M 2023-07-02T13:57:08.058 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 24 k 2023-07-02T13:57:08.059 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds x86_64 2:16.2.13-280.g8934282d.el8 ceph 2.4 M 2023-07-02T13:57:08.059 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 244 k 2023-07-02T13:57:08.059 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon x86_64 2:16.2.13-280.g8934282d.el8 ceph 5.0 M 2023-07-02T13:57:08.059 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd x86_64 2:16.2.13-280.g8934282d.el8 ceph 18 M 2023-07-02T13:57:08.059 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 15 k 2023-07-02T13:57:08.059 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux x86_64 2:16.2.13-280.g8934282d.el8 ceph 26 k 2023-07-02T13:57:08.059 INFO:teuthology.orchestra.run.smithi163.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-02T13:57:08.060 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-02T13:57:08.060 INFO:teuthology.orchestra.run.smithi163.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-07-02T13:57:08.060 INFO:teuthology.orchestra.run.smithi163.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-07-02T13:57:08.060 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite x86_64 2:16.2.13-280.g8934282d.el8 ceph 174 k 2023-07-02T13:57:08.060 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-07-02T13:57:08.060 INFO:teuthology.orchestra.run.smithi163.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-02T13:57:08.060 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-07-02T13:57:08.061 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-02T13:57:08.061 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1 x86_64 2:16.2.13-280.g8934282d.el8 ceph 493 k 2023-07-02T13:57:08.061 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-07-02T13:57:08.061 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2 x86_64 2:16.2.13-280.g8934282d.el8 ceph 3.8 M 2023-07-02T13:57:08.061 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-02T13:57:08.062 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-02T13:57:08.062 INFO:teuthology.orchestra.run.smithi163.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-02T13:57:08.062 INFO:teuthology.orchestra.run.smithi163.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-07-02T13:57:08.062 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-02T13:57:08.063 INFO:teuthology.orchestra.run.smithi163.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-07-02T13:57:08.063 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-02T13:57:08.063 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-02T13:57:08.063 INFO:teuthology.orchestra.run.smithi163.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-02T13:57:08.063 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-02T13:57:08.063 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse x86_64 2:16.2.13-280.g8934282d.el8 ceph 45 k 2023-07-02T13:57:08.063 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common x86_64 2:16.2.13-280.g8934282d.el8 ceph 99 k 2023-07-02T13:57:08.064 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-02T13:57:08.064 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-02T13:57:08.064 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-02T13:57:08.064 INFO:teuthology.orchestra.run.smithi163.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-07-02T13:57:08.064 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-02T13:57:08.065 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-02T13:57:08.065 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-02T13:57:08.065 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-02T13:57:08.065 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-02T13:57:08.065 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-02T13:57:08.065 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-02T13:57:08.065 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-02T13:57:08.066 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-02T13:57:08.066 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-02T13:57:08.066 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-02T13:57:08.066 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-02T13:57:08.066 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-02T13:57:08.066 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-02T13:57:08.066 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-02T13:57:08.067 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-02T13:57:08.067 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-02T13:57:08.067 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-02T13:57:08.067 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-02T13:57:08.067 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-02T13:57:08.067 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-02T13:57:08.067 INFO:teuthology.orchestra.run.smithi163.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-07-02T13:57:08.068 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-02T13:57:08.068 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-02T13:57:08.068 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-02T13:57:08.068 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-02T13:57:08.068 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-07-02T13:57:08.068 INFO:teuthology.orchestra.run.smithi163.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-02T13:57:08.068 INFO:teuthology.orchestra.run.smithi163.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-02T13:57:08.069 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-02T13:57:08.069 INFO:teuthology.orchestra.run.smithi163.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-02T13:57:08.069 INFO:teuthology.orchestra.run.smithi163.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-02T13:57:08.069 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-02T13:57:08.069 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-02T13:57:08.069 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-02T13:57:08.069 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-02T13:57:08.069 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-02T13:57:08.070 INFO:teuthology.orchestra.run.smithi163.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-07-02T13:57:08.070 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-02T13:57:08.070 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T13:57:08.070 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2023-07-02T13:57:08.070 INFO:teuthology.orchestra.run.smithi163.stdout:====================================================================================================================== 2023-07-02T13:57:08.070 INFO:teuthology.orchestra.run.smithi163.stdout:Install 98 Packages 2023-07-02T13:57:08.070 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrade 2 Packages 2023-07-02T13:57:08.071 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T13:57:08.071 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 193 M 2023-07-02T13:57:08.071 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2023-07-02T13:57:08.233 INFO:teuthology.orchestra.run.smithi016.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-02T13:57:08.234 INFO:teuthology.orchestra.run.smithi016.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-02T13:57:08.264 INFO:teuthology.orchestra.run.smithi163.stdout:(1/100): ceph-16.2.13-280.g8934282d.el8.x86_64. 37 kB/s | 6.4 kB 00:00 2023-07-02T13:57:08.753 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2023-07-02T13:57:08.765 INFO:teuthology.orchestra.run.smithi016.stdout:====================================================================================================================== 2023-07-02T13:57:08.765 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2023-07-02T13:57:08.765 INFO:teuthology.orchestra.run.smithi016.stdout:====================================================================================================================== 2023-07-02T13:57:08.765 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2023-07-02T13:57:08.765 INFO:teuthology.orchestra.run.smithi016.stdout: ceph x86_64 2:16.2.13-280.g8934282d.el8 ceph 6.4 k 2023-07-02T13:57:08.765 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base x86_64 2:16.2.13-280.g8934282d.el8 ceph 6.1 M 2023-07-02T13:57:08.766 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse x86_64 2:16.2.13-280.g8934282d.el8 ceph 909 k 2023-07-02T13:57:08.766 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache x86_64 2:16.2.13-280.g8934282d.el8 ceph 161 k 2023-07-02T13:57:08.766 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr x86_64 2:16.2.13-280.g8934282d.el8 ceph 1.6 M 2023-07-02T13:57:08.766 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 116 k 2023-07-02T13:57:08.766 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 3.1 M 2023-07-02T13:57:08.766 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 8.6 M 2023-07-02T13:57:08.766 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 27 k 2023-07-02T13:57:08.767 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw x86_64 2:16.2.13-280.g8934282d.el8 ceph 12 M 2023-07-02T13:57:08.767 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test x86_64 2:16.2.13-280.g8934282d.el8 ceph 51 M 2023-07-02T13:57:08.767 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 84 k 2023-07-02T13:57:08.767 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel x86_64 2:16.2.13-280.g8934282d.el8 ceph 25 k 2023-07-02T13:57:08.767 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2 x86_64 2:16.2.13-280.g8934282d.el8 ceph 824 k 2023-07-02T13:57:08.767 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel x86_64 2:16.2.13-280.g8934282d.el8 ceph 131 k 2023-07-02T13:57:08.767 INFO:teuthology.orchestra.run.smithi016.stdout: pv x86_64 1.6.6-7.el8 epel 64 k 2023-07-02T13:57:08.768 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs x86_64 2:16.2.13-280.g8934282d.el8 ceph 215 k 2023-07-02T13:57:08.768 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados x86_64 2:16.2.13-280.g8934282d.el8 ceph 389 k 2023-07-02T13:57:08.768 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd x86_64 2:16.2.13-280.g8934282d.el8 ceph 368 k 2023-07-02T13:57:08.768 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw x86_64 2:16.2.13-280.g8934282d.el8 ceph 114 k 2023-07-02T13:57:08.768 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse x86_64 2:16.2.13-280.g8934282d.el8 ceph 88 k 2023-07-02T13:57:08.768 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror x86_64 2:16.2.13-280.g8934282d.el8 ceph 3.7 M 2023-07-02T13:57:08.768 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd x86_64 2:16.2.13-280.g8934282d.el8 ceph 173 k 2023-07-02T13:57:08.769 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrading: 2023-07-02T13:57:08.769 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 x86_64 2:16.2.13-280.g8934282d.el8 ceph 3.7 M 2023-07-02T13:57:08.769 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1 x86_64 2:16.2.13-280.g8934282d.el8 ceph 4.2 M 2023-07-02T13:57:08.769 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2023-07-02T13:57:08.769 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common x86_64 2:16.2.13-280.g8934282d.el8 ceph 24 M 2023-07-02T13:57:08.769 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 24 k 2023-07-02T13:57:08.769 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds x86_64 2:16.2.13-280.g8934282d.el8 ceph 2.4 M 2023-07-02T13:57:08.770 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 244 k 2023-07-02T13:57:08.770 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon x86_64 2:16.2.13-280.g8934282d.el8 ceph 5.0 M 2023-07-02T13:57:08.770 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd x86_64 2:16.2.13-280.g8934282d.el8 ceph 18 M 2023-07-02T13:57:08.770 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 15 k 2023-07-02T13:57:08.770 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux x86_64 2:16.2.13-280.g8934282d.el8 ceph 26 k 2023-07-02T13:57:08.770 INFO:teuthology.orchestra.run.smithi016.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-02T13:57:08.770 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-02T13:57:08.771 INFO:teuthology.orchestra.run.smithi016.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-07-02T13:57:08.771 INFO:teuthology.orchestra.run.smithi016.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-07-02T13:57:08.771 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite x86_64 2:16.2.13-280.g8934282d.el8 ceph 174 k 2023-07-02T13:57:08.771 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-07-02T13:57:08.771 INFO:teuthology.orchestra.run.smithi016.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-02T13:57:08.771 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-07-02T13:57:08.771 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-02T13:57:08.771 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 x86_64 2:16.2.13-280.g8934282d.el8 ceph 493 k 2023-07-02T13:57:08.772 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-07-02T13:57:08.772 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2 x86_64 2:16.2.13-280.g8934282d.el8 ceph 3.8 M 2023-07-02T13:57:08.772 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-02T13:57:08.772 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-02T13:57:08.773 INFO:teuthology.orchestra.run.smithi016.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-02T13:57:08.773 INFO:teuthology.orchestra.run.smithi016.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-07-02T13:57:08.773 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-02T13:57:08.773 INFO:teuthology.orchestra.run.smithi016.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-07-02T13:57:08.773 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-02T13:57:08.773 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-02T13:57:08.773 INFO:teuthology.orchestra.run.smithi016.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-02T13:57:08.773 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-02T13:57:08.774 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse x86_64 2:16.2.13-280.g8934282d.el8 ceph 45 k 2023-07-02T13:57:08.774 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common x86_64 2:16.2.13-280.g8934282d.el8 ceph 99 k 2023-07-02T13:57:08.774 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-02T13:57:08.774 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-02T13:57:08.774 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-02T13:57:08.774 INFO:teuthology.orchestra.run.smithi016.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-07-02T13:57:08.774 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-02T13:57:08.775 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-02T13:57:08.775 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-02T13:57:08.775 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-02T13:57:08.775 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-02T13:57:08.775 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-02T13:57:08.775 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-02T13:57:08.775 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-02T13:57:08.776 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-02T13:57:08.776 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-02T13:57:08.776 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-02T13:57:08.776 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-02T13:57:08.776 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-02T13:57:08.776 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-02T13:57:08.776 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-02T13:57:08.777 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-02T13:57:08.777 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-02T13:57:08.777 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-02T13:57:08.777 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-02T13:57:08.777 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-02T13:57:08.777 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-02T13:57:08.777 INFO:teuthology.orchestra.run.smithi016.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-07-02T13:57:08.778 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-02T13:57:08.778 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-02T13:57:08.778 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-02T13:57:08.778 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-02T13:57:08.778 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-07-02T13:57:08.778 INFO:teuthology.orchestra.run.smithi016.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-02T13:57:08.778 INFO:teuthology.orchestra.run.smithi016.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-02T13:57:08.778 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-02T13:57:08.779 INFO:teuthology.orchestra.run.smithi016.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-02T13:57:08.779 INFO:teuthology.orchestra.run.smithi016.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-02T13:57:08.779 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-02T13:57:08.779 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-02T13:57:08.779 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-02T13:57:08.779 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-02T13:57:08.779 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-02T13:57:08.780 INFO:teuthology.orchestra.run.smithi016.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-07-02T13:57:08.780 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-02T13:57:08.780 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T13:57:08.780 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2023-07-02T13:57:08.780 INFO:teuthology.orchestra.run.smithi016.stdout:====================================================================================================================== 2023-07-02T13:57:08.780 INFO:teuthology.orchestra.run.smithi016.stdout:Install 98 Packages 2023-07-02T13:57:08.780 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 2 Packages 2023-07-02T13:57:08.781 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T13:57:08.781 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 193 M 2023-07-02T13:57:08.781 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2023-07-02T13:57:08.865 INFO:teuthology.orchestra.run.smithi022.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-02T13:57:08.866 INFO:teuthology.orchestra.run.smithi022.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-02T13:57:09.078 INFO:teuthology.orchestra.run.smithi016.stdout:(1/100): ceph-16.2.13-280.g8934282d.el8.x86_64. 24 kB/s | 6.4 kB 00:00 2023-07-02T13:57:09.375 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-07-02T13:57:09.385 INFO:teuthology.orchestra.run.smithi022.stdout:====================================================================================================================== 2023-07-02T13:57:09.385 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2023-07-02T13:57:09.386 INFO:teuthology.orchestra.run.smithi022.stdout:====================================================================================================================== 2023-07-02T13:57:09.386 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2023-07-02T13:57:09.386 INFO:teuthology.orchestra.run.smithi022.stdout: ceph x86_64 2:16.2.13-280.g8934282d.el8 ceph 6.4 k 2023-07-02T13:57:09.386 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-base x86_64 2:16.2.13-280.g8934282d.el8 ceph 6.1 M 2023-07-02T13:57:09.386 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-fuse x86_64 2:16.2.13-280.g8934282d.el8 ceph 909 k 2023-07-02T13:57:09.386 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-immutable-object-cache x86_64 2:16.2.13-280.g8934282d.el8 ceph 161 k 2023-07-02T13:57:09.387 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr x86_64 2:16.2.13-280.g8934282d.el8 ceph 1.6 M 2023-07-02T13:57:09.387 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-cephadm noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 116 k 2023-07-02T13:57:09.387 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-dashboard noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 3.1 M 2023-07-02T13:57:09.387 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 8.6 M 2023-07-02T13:57:09.387 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-rook noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 27 k 2023-07-02T13:57:09.387 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-radosgw x86_64 2:16.2.13-280.g8934282d.el8 ceph 12 M 2023-07-02T13:57:09.387 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-test x86_64 2:16.2.13-280.g8934282d.el8 ceph 51 M 2023-07-02T13:57:09.387 INFO:teuthology.orchestra.run.smithi022.stdout: cephadm noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 84 k 2023-07-02T13:57:09.388 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs-devel x86_64 2:16.2.13-280.g8934282d.el8 ceph 25 k 2023-07-02T13:57:09.388 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs2 x86_64 2:16.2.13-280.g8934282d.el8 ceph 824 k 2023-07-02T13:57:09.388 INFO:teuthology.orchestra.run.smithi022.stdout: librados-devel x86_64 2:16.2.13-280.g8934282d.el8 ceph 131 k 2023-07-02T13:57:09.388 INFO:teuthology.orchestra.run.smithi022.stdout: pv x86_64 1.6.6-7.el8 epel 64 k 2023-07-02T13:57:09.388 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cephfs x86_64 2:16.2.13-280.g8934282d.el8 ceph 215 k 2023-07-02T13:57:09.388 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rados x86_64 2:16.2.13-280.g8934282d.el8 ceph 389 k 2023-07-02T13:57:09.388 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rbd x86_64 2:16.2.13-280.g8934282d.el8 ceph 368 k 2023-07-02T13:57:09.389 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rgw x86_64 2:16.2.13-280.g8934282d.el8 ceph 114 k 2023-07-02T13:57:09.389 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-fuse x86_64 2:16.2.13-280.g8934282d.el8 ceph 88 k 2023-07-02T13:57:09.389 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-mirror x86_64 2:16.2.13-280.g8934282d.el8 ceph 3.7 M 2023-07-02T13:57:09.389 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-nbd x86_64 2:16.2.13-280.g8934282d.el8 ceph 173 k 2023-07-02T13:57:09.389 INFO:teuthology.orchestra.run.smithi022.stdout:Upgrading: 2023-07-02T13:57:09.389 INFO:teuthology.orchestra.run.smithi022.stdout: librados2 x86_64 2:16.2.13-280.g8934282d.el8 ceph 3.7 M 2023-07-02T13:57:09.389 INFO:teuthology.orchestra.run.smithi022.stdout: librbd1 x86_64 2:16.2.13-280.g8934282d.el8 ceph 4.2 M 2023-07-02T13:57:09.389 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2023-07-02T13:57:09.390 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-common x86_64 2:16.2.13-280.g8934282d.el8 ceph 24 M 2023-07-02T13:57:09.390 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-grafana-dashboards noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 24 k 2023-07-02T13:57:09.390 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mds x86_64 2:16.2.13-280.g8934282d.el8 ceph 2.4 M 2023-07-02T13:57:09.390 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-modules-core noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 244 k 2023-07-02T13:57:09.390 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mon x86_64 2:16.2.13-280.g8934282d.el8 ceph 5.0 M 2023-07-02T13:57:09.390 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-osd x86_64 2:16.2.13-280.g8934282d.el8 ceph 18 M 2023-07-02T13:57:09.390 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-prometheus-alerts noarch 2:16.2.13-280.g8934282d.el8 ceph-noarch 15 k 2023-07-02T13:57:09.391 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-selinux x86_64 2:16.2.13-280.g8934282d.el8 ceph 26 k 2023-07-02T13:57:09.391 INFO:teuthology.orchestra.run.smithi022.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-02T13:57:09.391 INFO:teuthology.orchestra.run.smithi022.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-02T13:57:09.391 INFO:teuthology.orchestra.run.smithi022.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-07-02T13:57:09.391 INFO:teuthology.orchestra.run.smithi022.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-07-02T13:57:09.391 INFO:teuthology.orchestra.run.smithi022.stdout: libcephsqlite x86_64 2:16.2.13-280.g8934282d.el8 ceph 174 k 2023-07-02T13:57:09.391 INFO:teuthology.orchestra.run.smithi022.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-07-02T13:57:09.391 INFO:teuthology.orchestra.run.smithi022.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-02T13:57:09.392 INFO:teuthology.orchestra.run.smithi022.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-07-02T13:57:09.392 INFO:teuthology.orchestra.run.smithi022.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-02T13:57:09.392 INFO:teuthology.orchestra.run.smithi022.stdout: libradosstriper1 x86_64 2:16.2.13-280.g8934282d.el8 ceph 493 k 2023-07-02T13:57:09.392 INFO:teuthology.orchestra.run.smithi022.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-07-02T13:57:09.392 INFO:teuthology.orchestra.run.smithi022.stdout: librgw2 x86_64 2:16.2.13-280.g8934282d.el8 ceph 3.8 M 2023-07-02T13:57:09.392 INFO:teuthology.orchestra.run.smithi022.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-02T13:57:09.392 INFO:teuthology.orchestra.run.smithi022.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-02T13:57:09.393 INFO:teuthology.orchestra.run.smithi022.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-02T13:57:09.393 INFO:teuthology.orchestra.run.smithi022.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-07-02T13:57:09.393 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-02T13:57:09.393 INFO:teuthology.orchestra.run.smithi022.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-07-02T13:57:09.393 INFO:teuthology.orchestra.run.smithi022.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-02T13:57:09.393 INFO:teuthology.orchestra.run.smithi022.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-02T13:57:09.393 INFO:teuthology.orchestra.run.smithi022.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-02T13:57:09.393 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-02T13:57:09.394 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-argparse x86_64 2:16.2.13-280.g8934282d.el8 ceph 45 k 2023-07-02T13:57:09.394 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-common x86_64 2:16.2.13-280.g8934282d.el8 ceph 99 k 2023-07-02T13:57:09.394 INFO:teuthology.orchestra.run.smithi022.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-02T13:57:09.394 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-02T13:57:09.394 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-02T13:57:09.394 INFO:teuthology.orchestra.run.smithi022.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-07-02T13:57:09.394 INFO:teuthology.orchestra.run.smithi022.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-02T13:57:09.395 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-02T13:57:09.395 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-02T13:57:09.395 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-02T13:57:09.395 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-02T13:57:09.395 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-02T13:57:09.395 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-02T13:57:09.395 INFO:teuthology.orchestra.run.smithi022.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-02T13:57:09.395 INFO:teuthology.orchestra.run.smithi022.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-02T13:57:09.396 INFO:teuthology.orchestra.run.smithi022.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-02T13:57:09.398 INFO:teuthology.orchestra.run.smithi022.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-02T13:57:09.398 INFO:teuthology.orchestra.run.smithi022.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-02T13:57:09.398 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-02T13:57:09.398 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-02T13:57:09.399 INFO:teuthology.orchestra.run.smithi022.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-02T13:57:09.399 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-02T13:57:09.399 INFO:teuthology.orchestra.run.smithi022.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-02T13:57:09.399 INFO:teuthology.orchestra.run.smithi022.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-02T13:57:09.399 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-02T13:57:09.399 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-02T13:57:09.399 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-02T13:57:09.400 INFO:teuthology.orchestra.run.smithi022.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-07-02T13:57:09.400 INFO:teuthology.orchestra.run.smithi022.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-02T13:57:09.400 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-02T13:57:09.400 INFO:teuthology.orchestra.run.smithi022.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-02T13:57:09.400 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-02T13:57:09.400 INFO:teuthology.orchestra.run.smithi022.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-07-02T13:57:09.400 INFO:teuthology.orchestra.run.smithi022.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-02T13:57:09.401 INFO:teuthology.orchestra.run.smithi022.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-02T13:57:09.401 INFO:teuthology.orchestra.run.smithi022.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-02T13:57:09.401 INFO:teuthology.orchestra.run.smithi022.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-02T13:57:09.401 INFO:teuthology.orchestra.run.smithi022.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-02T13:57:09.401 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-02T13:57:09.401 INFO:teuthology.orchestra.run.smithi022.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-02T13:57:09.401 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-02T13:57:09.402 INFO:teuthology.orchestra.run.smithi022.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-02T13:57:09.402 INFO:teuthology.orchestra.run.smithi022.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-02T13:57:09.402 INFO:teuthology.orchestra.run.smithi022.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-07-02T13:57:09.402 INFO:teuthology.orchestra.run.smithi022.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-02T13:57:09.402 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T13:57:09.402 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2023-07-02T13:57:09.402 INFO:teuthology.orchestra.run.smithi022.stdout:====================================================================================================================== 2023-07-02T13:57:09.403 INFO:teuthology.orchestra.run.smithi022.stdout:Install 98 Packages 2023-07-02T13:57:09.403 INFO:teuthology.orchestra.run.smithi022.stdout:Upgrade 2 Packages 2023-07-02T13:57:09.403 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T13:57:09.404 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 193 M 2023-07-02T13:57:09.404 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2023-07-02T13:57:09.523 INFO:teuthology.orchestra.run.smithi163.stdout:(2/100): ceph-common-16.2.13-280.g8934282d.el8. 17 MB/s | 24 MB 00:01 2023-07-02T13:57:09.526 INFO:teuthology.orchestra.run.smithi163.stdout:(3/100): ceph-fuse-16.2.13-280.g8934282d.el8.x8 721 kB/s | 909 kB 00:01 2023-07-02T13:57:09.551 INFO:teuthology.orchestra.run.smithi016.stdout:(2/100): ceph-fuse-16.2.13-280.g8934282d.el8.x8 1.9 MB/s | 909 kB 00:00 2023-07-02T13:57:09.562 INFO:teuthology.orchestra.run.smithi163.stdout:(4/100): ceph-immutable-object-cache-16.2.13-28 4.1 MB/s | 161 kB 00:00 2023-07-02T13:57:09.605 INFO:teuthology.orchestra.run.smithi022.stdout:(1/100): ceph-16.2.13-280.g8934282d.el8.x86_64. 37 kB/s | 6.4 kB 00:00 2023-07-02T13:57:09.651 INFO:teuthology.orchestra.run.smithi016.stdout:(3/100): ceph-immutable-object-cache-16.2.13-28 1.6 MB/s | 161 kB 00:00 2023-07-02T13:57:10.001 INFO:teuthology.orchestra.run.smithi163.stdout:(5/100): ceph-mgr-16.2.13-280.g8934282d.el8.x86 3.7 MB/s | 1.6 MB 00:00 2023-07-02T13:57:10.685 INFO:teuthology.orchestra.run.smithi163.stdout:(6/100): ceph-mon-16.2.13-280.g8934282d.el8.x86 7.3 MB/s | 5.0 MB 00:00 2023-07-02T13:57:10.722 INFO:teuthology.orchestra.run.smithi016.stdout:(4/100): ceph-mds-16.2.13-280.g8934282d.el8.x86 2.3 MB/s | 2.4 MB 00:01 2023-07-02T13:57:10.986 INFO:teuthology.orchestra.run.smithi163.stdout:(7/100): ceph-base-16.2.13-280.g8934282d.el8.x8 2.1 MB/s | 6.1 MB 00:02 2023-07-02T13:57:11.230 INFO:teuthology.orchestra.run.smithi022.stdout:(2/100): ceph-fuse-16.2.13-280.g8934282d.el8.x8 560 kB/s | 909 kB 00:01 2023-07-02T13:57:11.606 INFO:teuthology.orchestra.run.smithi016.stdout:(5/100): ceph-mgr-16.2.13-280.g8934282d.el8.x86 1.8 MB/s | 1.6 MB 00:00 2023-07-02T13:57:11.623 INFO:teuthology.orchestra.run.smithi022.stdout:(3/100): ceph-immutable-object-cache-16.2.13-28 411 kB/s | 161 kB 00:00 2023-07-02T13:57:11.748 INFO:teuthology.orchestra.run.smithi016.stdout:(6/100): ceph-base-16.2.13-280.g8934282d.el8.x8 2.1 MB/s | 6.1 MB 00:02 2023-07-02T13:57:12.699 INFO:teuthology.orchestra.run.smithi163.stdout:(8/100): ceph-mds-16.2.13-280.g8934282d.el8.x86 787 kB/s | 2.4 MB 00:03 2023-07-02T13:57:12.824 INFO:teuthology.orchestra.run.smithi163.stdout:(9/100): ceph-selinux-16.2.13-280.g8934282d.el8 205 kB/s | 26 kB 00:00 2023-07-02T13:57:13.681 INFO:teuthology.orchestra.run.smithi016.stdout:(7/100): ceph-mon-16.2.13-280.g8934282d.el8.x86 2.4 MB/s | 5.0 MB 00:02 2023-07-02T13:57:13.879 INFO:teuthology.orchestra.run.smithi163.stdout:(10/100): ceph-osd-16.2.13-280.g8934282d.el8.x8 5.7 MB/s | 18 MB 00:03 2023-07-02T13:57:13.945 INFO:teuthology.orchestra.run.smithi163.stdout:(11/100): libcephfs-devel-16.2.13-280.g8934282d 391 kB/s | 25 kB 00:00 2023-07-02T13:57:14.101 INFO:teuthology.orchestra.run.smithi163.stdout:(12/100): libcephfs2-16.2.13-280.g8934282d.el8. 5.2 MB/s | 824 kB 00:00 2023-07-02T13:57:14.163 INFO:teuthology.orchestra.run.smithi163.stdout:(13/100): libcephsqlite-16.2.13-280.g8934282d.e 2.7 MB/s | 174 kB 00:00 2023-07-02T13:57:14.228 INFO:teuthology.orchestra.run.smithi163.stdout:(14/100): librados-devel-16.2.13-280.g8934282d. 2.0 MB/s | 131 kB 00:00 2023-07-02T13:57:14.298 INFO:teuthology.orchestra.run.smithi022.stdout:(4/100): ceph-base-16.2.13-280.g8934282d.el8.x8 1.2 MB/s | 6.1 MB 00:04 2023-07-02T13:57:14.353 INFO:teuthology.orchestra.run.smithi163.stdout:(15/100): libradosstriper1-16.2.13-280.g8934282 3.9 MB/s | 493 kB 00:00 2023-07-02T13:57:15.019 INFO:teuthology.orchestra.run.smithi163.stdout:(16/100): librgw2-16.2.13-280.g8934282d.el8.x86 5.7 MB/s | 3.8 MB 00:00 2023-07-02T13:57:15.080 INFO:teuthology.orchestra.run.smithi163.stdout:(17/100): python3-ceph-argparse-16.2.13-280.g89 736 kB/s | 45 kB 00:00 2023-07-02T13:57:15.142 INFO:teuthology.orchestra.run.smithi163.stdout:(18/100): python3-ceph-common-16.2.13-280.g8934 1.5 MB/s | 99 kB 00:00 2023-07-02T13:57:15.222 INFO:teuthology.orchestra.run.smithi163.stdout:(19/100): python3-cephfs-16.2.13-280.g8934282d. 2.7 MB/s | 215 kB 00:00 2023-07-02T13:57:15.300 INFO:teuthology.orchestra.run.smithi163.stdout:(20/100): python3-rados-16.2.13-280.g8934282d.e 4.9 MB/s | 389 kB 00:00 2023-07-02T13:57:15.377 INFO:teuthology.orchestra.run.smithi163.stdout:(21/100): python3-rbd-16.2.13-280.g8934282d.el8 4.6 MB/s | 368 kB 00:00 2023-07-02T13:57:15.446 INFO:teuthology.orchestra.run.smithi163.stdout:(22/100): python3-rgw-16.2.13-280.g8934282d.el8 1.6 MB/s | 114 kB 00:00 2023-07-02T13:57:15.501 INFO:teuthology.orchestra.run.smithi163.stdout:(23/100): rbd-fuse-16.2.13-280.g8934282d.el8.x8 1.6 MB/s | 88 kB 00:00 2023-07-02T13:57:15.765 INFO:teuthology.orchestra.run.smithi022.stdout:(5/100): ceph-mds-16.2.13-280.g8934282d.el8.x86 603 kB/s | 2.4 MB 00:04 2023-07-02T13:57:15.823 INFO:teuthology.orchestra.run.smithi022.stdout:(6/100): ceph-mgr-16.2.13-280.g8934282d.el8.x86 1.1 MB/s | 1.6 MB 00:01 2023-07-02T13:57:16.229 INFO:teuthology.orchestra.run.smithi163.stdout:(24/100): rbd-mirror-16.2.13-280.g8934282d.el8. 5.0 MB/s | 3.7 MB 00:00 2023-07-02T13:57:16.331 INFO:teuthology.orchestra.run.smithi163.stdout:(25/100): rbd-nbd-16.2.13-280.g8934282d.el8.x86 1.6 MB/s | 173 kB 00:00 2023-07-02T13:57:16.460 INFO:teuthology.orchestra.run.smithi163.stdout:(26/100): ceph-grafana-dashboards-16.2.13-280.g 190 kB/s | 24 kB 00:00 2023-07-02T13:57:16.530 INFO:teuthology.orchestra.run.smithi163.stdout:(27/100): ceph-mgr-cephadm-16.2.13-280.g8934282 1.6 MB/s | 116 kB 00:00 2023-07-02T13:57:17.294 INFO:teuthology.orchestra.run.smithi163.stdout:(28/100): ceph-mgr-dashboard-16.2.13-280.g89342 4.1 MB/s | 3.1 MB 00:00 2023-07-02T13:57:17.634 INFO:teuthology.orchestra.run.smithi016.stdout:(8/100): ceph-radosgw-16.2.13-280.g8934282d.el8 2.9 MB/s | 12 MB 00:03 2023-07-02T13:57:17.700 INFO:teuthology.orchestra.run.smithi016.stdout:(9/100): ceph-selinux-16.2.13-280.g8934282d.el8 392 kB/s | 26 kB 00:00 2023-07-02T13:57:18.502 INFO:teuthology.orchestra.run.smithi163.stdout:(29/100): ceph-radosgw-16.2.13-280.g8934282d.el 1.5 MB/s | 12 MB 00:07 2023-07-02T13:57:18.851 INFO:teuthology.orchestra.run.smithi163.stdout:(30/100): ceph-mgr-modules-core-16.2.13-280.g89 700 kB/s | 244 kB 00:00 2023-07-02T13:57:19.001 INFO:teuthology.orchestra.run.smithi163.stdout:(31/100): ceph-mgr-rook-16.2.13-280.g8934282d.e 183 kB/s | 27 kB 00:00 2023-07-02T13:57:19.099 INFO:teuthology.orchestra.run.smithi163.stdout:(32/100): ceph-prometheus-alerts-16.2.13-280.g8 151 kB/s | 15 kB 00:00 2023-07-02T13:57:19.194 INFO:teuthology.orchestra.run.smithi163.stdout:(33/100): ceph-mgr-diskprediction-local-16.2.13 4.5 MB/s | 8.6 MB 00:01 2023-07-02T13:57:19.202 INFO:teuthology.orchestra.run.smithi163.stdout:(34/100): cephadm-16.2.13-280.g8934282d.el8.noa 817 kB/s | 84 kB 00:00 2023-07-02T13:57:19.297 INFO:teuthology.orchestra.run.smithi163.stdout:(35/100): leveldb-1.22-1.el8.x86_64.rpm 1.9 MB/s | 181 kB 00:00 2023-07-02T13:57:19.302 INFO:teuthology.orchestra.run.smithi163.stdout:(36/100): gperftools-libs-2.7-9.el8.x86_64.rpm 2.8 MB/s | 306 kB 00:00 2023-07-02T13:57:19.319 INFO:teuthology.orchestra.run.smithi163.stdout:(37/100): liboath-2.6.2-3.el8.x86_64.rpm 2.6 MB/s | 59 kB 00:00 2023-07-02T13:57:19.321 INFO:teuthology.orchestra.run.smithi163.stdout:(38/100): libunwind-1.3.1-3.el8.x86_64.rpm 4.0 MB/s | 75 kB 00:00 2023-07-02T13:57:19.340 INFO:teuthology.orchestra.run.smithi163.stdout:(39/100): pv-1.6.6-7.el8.x86_64.rpm 3.0 MB/s | 64 kB 00:00 2023-07-02T13:57:19.341 INFO:teuthology.orchestra.run.smithi163.stdout:(40/100): python3-apipkg-1.5-6.el8.noarch.rpm 1.1 MB/s | 21 kB 00:00 2023-07-02T13:57:19.358 INFO:teuthology.orchestra.run.smithi163.stdout:(41/100): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 2.3 MB/s | 44 kB 00:00 2023-07-02T13:57:19.362 INFO:teuthology.orchestra.run.smithi163.stdout:(42/100): python3-beautifulsoup4-4.6.3-2.el8.1. 8.4 MB/s | 185 kB 00:00 2023-07-02T13:57:19.378 INFO:teuthology.orchestra.run.smithi163.stdout:(43/100): python3-cachetools-3.1.1-4.el8.noarch 1.7 MB/s | 33 kB 00:00 2023-07-02T13:57:19.380 INFO:teuthology.orchestra.run.smithi163.stdout:(44/100): python3-certifi-2018.10.15-7.el8.noar 904 kB/s | 16 kB 00:00 2023-07-02T13:57:19.402 INFO:teuthology.orchestra.run.smithi163.stdout:(45/100): python3-cheroot-8.5.2-1.el8.noarch.rp 6.9 MB/s | 173 kB 00:00 2023-07-02T13:57:19.416 INFO:teuthology.orchestra.run.smithi163.stdout:(46/100): python3-cherrypy-18.4.0-1.el8.noarch. 11 MB/s | 384 kB 00:00 2023-07-02T13:57:19.436 INFO:teuthology.orchestra.run.smithi163.stdout:(47/100): python3-google-auth-1.1.1-10.el8.noar 3.8 MB/s | 81 kB 00:00 2023-07-02T13:57:19.441 INFO:teuthology.orchestra.run.smithi163.stdout:(48/100): python3-execnet-1.7.1-1.el8.noarch.rp 9.0 MB/s | 355 kB 00:00 2023-07-02T13:57:19.454 INFO:teuthology.orchestra.run.smithi163.stdout:(49/100): python3-jaraco-6.2-6.el8.noarch.rpm 640 kB/s | 11 kB 00:00 2023-07-02T13:57:19.459 INFO:teuthology.orchestra.run.smithi163.stdout:(50/100): python3-jaraco-functools-2.0-4.el8.no 1.0 MB/s | 18 kB 00:00 2023-07-02T13:57:19.477 INFO:teuthology.orchestra.run.smithi163.stdout:(51/100): python3-logutils-0.3.5-11.el8.noarch. 2.6 MB/s | 49 kB 00:00 2023-07-02T13:57:19.495 INFO:teuthology.orchestra.run.smithi163.stdout:(52/100): python3-more-itertools-7.2.0-3.el8.no 3.2 MB/s | 59 kB 00:00 2023-07-02T13:57:19.500 INFO:teuthology.orchestra.run.smithi163.stdout:(53/100): python3-kubernetes-11.0.0-6.el8.noarc 25 MB/s | 1.2 MB 00:00 2023-07-02T13:57:19.517 INFO:teuthology.orchestra.run.smithi163.stdout:(54/100): python3-pecan-1.3.2-9.el8.noarch.rpm 13 MB/s | 283 kB 00:00 2023-07-02T13:57:19.518 INFO:teuthology.orchestra.run.smithi163.stdout:(55/100): python3-portend-2.6-1.el8.noarch.rpm 893 kB/s | 16 kB 00:00 2023-07-02T13:57:19.538 INFO:teuthology.orchestra.run.smithi163.stdout:(56/100): python3-repoze-lru-0.7-6.el8.noarch.r 1.7 MB/s | 34 kB 00:00 2023-07-02T13:57:19.539 INFO:teuthology.orchestra.run.smithi163.stdout:(57/100): python3-remoto-1.2.1-1.el8.noarch.rpm 2.2 MB/s | 49 kB 00:00 2023-07-02T13:57:19.558 INFO:teuthology.orchestra.run.smithi163.stdout:(58/100): python3-routes-2.4.1-12.el8.noarch.rp 9.8 MB/s | 196 kB 00:00 2023-07-02T13:57:19.561 INFO:teuthology.orchestra.run.smithi163.stdout:(59/100): python3-rsa-4.8-1.el8.noarch.rpm 2.8 MB/s | 61 kB 00:00 2023-07-02T13:57:19.575 INFO:teuthology.orchestra.run.smithi163.stdout:(60/100): python3-simplegeneric-0.8.1-17.el8.no 1.1 MB/s | 19 kB 00:00 2023-07-02T13:57:19.578 INFO:teuthology.orchestra.run.smithi163.stdout:(61/100): python3-singledispatch-3.4.0.3-18.el8 1.3 MB/s | 24 kB 00:00 2023-07-02T13:57:19.593 INFO:teuthology.orchestra.run.smithi163.stdout:(62/100): python3-tempora-1.14.1-5.el8.noarch.r 1.6 MB/s | 29 kB 00:00 2023-07-02T13:57:19.596 INFO:teuthology.orchestra.run.smithi163.stdout:(63/100): python3-trustme-0.6.0-4.el8.noarch.rp 1.5 MB/s | 27 kB 00:00 2023-07-02T13:57:19.614 INFO:teuthology.orchestra.run.smithi163.stdout:(64/100): python3-waitress-1.4.3-1.el8.noarch.r 12 MB/s | 255 kB 00:00 2023-07-02T13:57:19.617 INFO:teuthology.orchestra.run.smithi163.stdout:(65/100): python3-webob-1.8.5-1.el8.1.noarch.rp 12 MB/s | 251 kB 00:00 2023-07-02T13:57:19.633 INFO:teuthology.orchestra.run.smithi163.stdout:(66/100): python3-websocket-client-0.56.0-5.el8 3.1 MB/s | 61 kB 00:00 2023-07-02T13:57:19.636 INFO:teuthology.orchestra.run.smithi163.stdout:(67/100): python3-webtest-2.0.33-1.el8.noarch.r 4.6 MB/s | 85 kB 00:00 2023-07-02T13:57:19.652 INFO:teuthology.orchestra.run.smithi163.stdout:(68/100): python3-zc-lockfile-2.0-2.el8.noarch. 1.3 MB/s | 23 kB 00:00 2023-07-02T13:57:19.943 INFO:teuthology.orchestra.run.smithi163.stdout:(69/100): python3-scipy-1.0.0-20.module+el8.1.0 50 MB/s | 15 MB 00:00 2023-07-02T13:57:19.976 INFO:teuthology.orchestra.run.smithi163.stdout:(70/100): python3-prettytable-0.7.2-14.el8.noar 1.3 MB/s | 44 kB 00:00 2023-07-02T13:57:20.007 INFO:teuthology.orchestra.run.smithi163.stdout:(71/100): python3-jsonpatch-1.21-2.el8.noarch.r 847 kB/s | 27 kB 00:00 2023-07-02T13:57:20.039 INFO:teuthology.orchestra.run.smithi163.stdout:(72/100): xmlstarlet-1.6.1-20.el8.x86_64.rpm 172 kB/s | 70 kB 00:00 2023-07-02T13:57:20.042 INFO:teuthology.orchestra.run.smithi163.stdout:(73/100): jq-1.5-12.el8.x86_64.rpm 4.5 MB/s | 161 kB 00:00 2023-07-02T13:57:20.099 INFO:teuthology.orchestra.run.smithi163.stdout:(74/100): python3-werkzeug-0.12.2-4.el8.noarch. 7.5 MB/s | 457 kB 00:00 2023-07-02T13:57:20.136 INFO:teuthology.orchestra.run.smithi163.stdout:(75/100): python3-babel-2.5.1-5.el8.noarch.rpm 52 MB/s | 4.8 MB 00:00 2023-07-02T13:57:20.136 INFO:teuthology.orchestra.run.smithi163.stdout:(76/100): python3-markupsafe-0.23-19.el8.x86_64 1.0 MB/s | 39 kB 00:00 2023-07-02T13:57:20.200 INFO:teuthology.orchestra.run.smithi163.stdout:(77/100): lttng-ust-2.8.1-11.el8.x86_64.rpm 3.9 MB/s | 259 kB 00:00 2023-07-02T13:57:20.201 INFO:teuthology.orchestra.run.smithi163.stdout:(78/100): python3-numpy-f2py-1.14.3-9.el8.x86_6 3.4 MB/s | 225 kB 00:00 2023-07-02T13:57:20.223 INFO:teuthology.orchestra.run.smithi022.stdout:(7/100): ceph-mon-16.2.13-280.g8934282d.el8.x86 1.1 MB/s | 5.0 MB 00:04 2023-07-02T13:57:20.266 INFO:teuthology.orchestra.run.smithi163.stdout:(79/100): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.9 MB/s | 126 kB 00:00 2023-07-02T13:57:20.306 INFO:teuthology.orchestra.run.smithi163.stdout:(80/100): python3-numpy-1.14.3-9.el8.x86_64.rpm 36 MB/s | 3.7 MB 00:00 2023-07-02T13:57:20.308 INFO:teuthology.orchestra.run.smithi163.stdout:(81/100): oniguruma-6.8.2-2.el8.x86_64.rpm 4.3 MB/s | 187 kB 00:00 2023-07-02T13:57:20.379 INFO:teuthology.orchestra.run.smithi163.stdout:(82/100): socat-1.7.3.3-2.el8.x86_64.rpm 4.2 MB/s | 302 kB 00:00 2023-07-02T13:57:20.384 INFO:teuthology.orchestra.run.smithi163.stdout:(83/100): python3-jinja2-2.10.1-2.el8_0.noarch. 6.7 MB/s | 538 kB 00:00 2023-07-02T13:57:20.423 INFO:teuthology.orchestra.run.smithi163.stdout:(84/100): python3-pyasn1-modules-0.3.7-6.el8.no 2.8 MB/s | 110 kB 00:00 2023-07-02T13:57:20.437 INFO:teuthology.orchestra.run.smithi163.stdout:(85/100): python3-mako-1.0.6-13.el8.noarch.rpm 2.7 MB/s | 157 kB 00:00 2023-07-02T13:57:20.474 INFO:teuthology.orchestra.run.smithi163.stdout:(86/100): python3-jsonpointer-1.10-11.el8.noarc 546 kB/s | 20 kB 00:00 2023-07-02T13:57:20.508 INFO:teuthology.orchestra.run.smithi163.stdout:(87/100): librdkafka-0.11.4-1.el8.x86_64.rpm 4.1 MB/s | 353 kB 00:00 2023-07-02T13:57:20.523 INFO:teuthology.orchestra.run.smithi163.stdout:(88/100): python3-pytz-2017.2-9.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2023-07-02T13:57:20.621 INFO:teuthology.orchestra.run.smithi163.stdout:(89/100): openblas-threads-0.3.12-1.el8.x86_64. 43 MB/s | 4.7 MB 00:00 2023-07-02T13:57:20.639 INFO:teuthology.orchestra.run.smithi163.stdout:(90/100): openblas-0.3.12-1.el8.x86_64.rpm 40 MB/s | 4.6 MB 00:00 2023-07-02T13:57:20.661 INFO:teuthology.orchestra.run.smithi163.stdout:(91/100): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2023-07-02T13:57:20.698 INFO:teuthology.orchestra.run.smithi163.stdout:(92/100): python3-oauthlib-2.1.0-1.el8.noarch.r 2.6 MB/s | 155 kB 00:00 2023-07-02T13:57:20.709 INFO:teuthology.orchestra.run.smithi163.stdout:(93/100): python3-requests-oauthlib-1.0.0-1.el8 889 kB/s | 43 kB 00:00 2023-07-02T13:57:20.734 INFO:teuthology.orchestra.run.smithi163.stdout:(94/100): fuse-2.9.7-12.el8.x86_64.rpm 2.2 MB/s | 83 kB 00:00 2023-07-02T13:57:20.748 INFO:teuthology.orchestra.run.smithi163.stdout:(95/100): libquadmath-8.4.1-1.el8.x86_64.rpm 4.3 MB/s | 169 kB 00:00 2023-07-02T13:57:20.773 INFO:teuthology.orchestra.run.smithi163.stdout:(96/100): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.2 MB/s | 47 kB 00:00 2023-07-02T13:57:20.789 INFO:teuthology.orchestra.run.smithi163.stdout:(97/100): libgfortran-8.4.1-1.el8.x86_64.rpm 15 MB/s | 639 kB 00:00 2023-07-02T13:57:22.529 INFO:teuthology.orchestra.run.smithi163.stdout:(98/100): librbd1-16.2.13-280.g8934282d.el8.x86 2.4 MB/s | 4.2 MB 00:01 2023-07-02T13:57:23.641 INFO:teuthology.orchestra.run.smithi016.stdout:(10/100): ceph-osd-16.2.13-280.g8934282d.el8.x8 1.5 MB/s | 18 MB 00:11 2023-07-02T13:57:23.711 INFO:teuthology.orchestra.run.smithi016.stdout:(11/100): libcephfs-devel-16.2.13-280.g8934282d 365 kB/s | 25 kB 00:00 2023-07-02T13:57:24.223 INFO:teuthology.orchestra.run.smithi163.stdout:(99/100): librados2-16.2.13-280.g8934282d.el8.x 1.1 MB/s | 3.7 MB 00:03 2023-07-02T13:57:24.320 INFO:teuthology.orchestra.run.smithi016.stdout:(12/100): libcephfs2-16.2.13-280.g8934282d.el8. 1.3 MB/s | 824 kB 00:00 2023-07-02T13:57:24.462 INFO:teuthology.orchestra.run.smithi016.stdout:(13/100): libcephsqlite-16.2.13-280.g8934282d.e 1.2 MB/s | 174 kB 00:00 2023-07-02T13:57:24.588 INFO:teuthology.orchestra.run.smithi016.stdout:(14/100): librados-devel-16.2.13-280.g8934282d. 1.0 MB/s | 131 kB 00:00 2023-07-02T13:57:24.925 INFO:teuthology.orchestra.run.smithi016.stdout:(15/100): libradosstriper1-16.2.13-280.g8934282 1.4 MB/s | 493 kB 00:00 2023-07-02T13:57:25.363 INFO:teuthology.orchestra.run.smithi022.stdout:(8/100): ceph-osd-16.2.13-280.g8934282d.el8.x86 1.9 MB/s | 18 MB 00:09 2023-07-02T13:57:25.422 INFO:teuthology.orchestra.run.smithi022.stdout:(9/100): ceph-selinux-16.2.13-280.g8934282d.el8 445 kB/s | 26 kB 00:00 2023-07-02T13:57:27.319 INFO:teuthology.orchestra.run.smithi016.stdout:(16/100): librgw2-16.2.13-280.g8934282d.el8.x86 1.6 MB/s | 3.8 MB 00:02 2023-07-02T13:57:27.383 INFO:teuthology.orchestra.run.smithi016.stdout:(17/100): python3-ceph-argparse-16.2.13-280.g89 705 kB/s | 45 kB 00:00 2023-07-02T13:57:27.452 INFO:teuthology.orchestra.run.smithi016.stdout:(18/100): python3-ceph-common-16.2.13-280.g8934 1.4 MB/s | 99 kB 00:00 2023-07-02T13:57:27.594 INFO:teuthology.orchestra.run.smithi016.stdout:(19/100): python3-cephfs-16.2.13-280.g8934282d. 1.5 MB/s | 215 kB 00:00 2023-07-02T13:57:27.866 INFO:teuthology.orchestra.run.smithi016.stdout:(20/100): python3-rados-16.2.13-280.g8934282d.e 1.4 MB/s | 389 kB 00:00 2023-07-02T13:57:28.140 INFO:teuthology.orchestra.run.smithi016.stdout:(21/100): python3-rbd-16.2.13-280.g8934282d.el8 1.3 MB/s | 368 kB 00:00 2023-07-02T13:57:28.218 INFO:teuthology.orchestra.run.smithi022.stdout:(10/100): ceph-radosgw-16.2.13-280.g8934282d.el 1.4 MB/s | 12 MB 00:07 2023-07-02T13:57:28.267 INFO:teuthology.orchestra.run.smithi016.stdout:(22/100): python3-rgw-16.2.13-280.g8934282d.el8 900 kB/s | 114 kB 00:00 2023-07-02T13:57:28.307 INFO:teuthology.orchestra.run.smithi022.stdout:(11/100): libcephfs-devel-16.2.13-280.g8934282d 287 kB/s | 25 kB 00:00 2023-07-02T13:57:28.334 INFO:teuthology.orchestra.run.smithi016.stdout:(23/100): rbd-fuse-16.2.13-280.g8934282d.el8.x8 1.3 MB/s | 88 kB 00:00 2023-07-02T13:57:28.778 INFO:teuthology.orchestra.run.smithi022.stdout:(12/100): libcephfs2-16.2.13-280.g8934282d.el8. 1.7 MB/s | 824 kB 00:00 2023-07-02T13:57:28.932 INFO:teuthology.orchestra.run.smithi022.stdout:(13/100): libcephsqlite-16.2.13-280.g8934282d.e 1.1 MB/s | 174 kB 00:00 2023-07-02T13:57:29.016 INFO:teuthology.orchestra.run.smithi022.stdout:(14/100): librados-devel-16.2.13-280.g8934282d. 1.5 MB/s | 131 kB 00:00 2023-07-02T13:57:29.027 INFO:teuthology.orchestra.run.smithi016.stdout:(24/100): ceph-common-16.2.13-280.g8934282d.el8 1.2 MB/s | 24 MB 00:20 2023-07-02T13:57:29.298 INFO:teuthology.orchestra.run.smithi022.stdout:(15/100): libradosstriper1-16.2.13-280.g8934282 1.7 MB/s | 493 kB 00:00 2023-07-02T13:57:29.307 INFO:teuthology.orchestra.run.smithi016.stdout:(25/100): rbd-nbd-16.2.13-280.g8934282d.el8.x86 618 kB/s | 173 kB 00:00 2023-07-02T13:57:29.401 INFO:teuthology.orchestra.run.smithi016.stdout:(26/100): ceph-grafana-dashboards-16.2.13-280.g 261 kB/s | 24 kB 00:00 2023-07-02T13:57:29.583 INFO:teuthology.orchestra.run.smithi016.stdout:(27/100): ceph-mgr-cephadm-16.2.13-280.g8934282 640 kB/s | 116 kB 00:00 2023-07-02T13:57:30.565 INFO:teuthology.orchestra.run.smithi016.stdout:(28/100): rbd-mirror-16.2.13-280.g8934282d.el8. 1.6 MB/s | 3.7 MB 00:02 2023-07-02T13:57:31.372 INFO:teuthology.orchestra.run.smithi022.stdout:(16/100): librgw2-16.2.13-280.g8934282d.el8.x86 1.8 MB/s | 3.8 MB 00:02 2023-07-02T13:57:31.474 INFO:teuthology.orchestra.run.smithi022.stdout:(17/100): python3-ceph-argparse-16.2.13-280.g89 441 kB/s | 45 kB 00:00 2023-07-02T13:57:31.584 INFO:teuthology.orchestra.run.smithi022.stdout:(18/100): python3-ceph-common-16.2.13-280.g8934 900 kB/s | 99 kB 00:00 2023-07-02T13:57:31.707 INFO:teuthology.orchestra.run.smithi022.stdout:(19/100): python3-cephfs-16.2.13-280.g8934282d. 1.7 MB/s | 215 kB 00:00 2023-07-02T13:57:31.945 INFO:teuthology.orchestra.run.smithi022.stdout:(20/100): python3-rados-16.2.13-280.g8934282d.e 1.6 MB/s | 389 kB 00:00 2023-07-02T13:57:32.162 INFO:teuthology.orchestra.run.smithi022.stdout:(21/100): python3-rbd-16.2.13-280.g8934282d.el8 1.7 MB/s | 368 kB 00:00 2023-07-02T13:57:32.252 INFO:teuthology.orchestra.run.smithi022.stdout:(22/100): python3-rgw-16.2.13-280.g8934282d.el8 1.2 MB/s | 114 kB 00:00 2023-07-02T13:57:32.344 INFO:teuthology.orchestra.run.smithi022.stdout:(23/100): rbd-fuse-16.2.13-280.g8934282d.el8.x8 970 kB/s | 88 kB 00:00 2023-07-02T13:57:32.845 INFO:teuthology.orchestra.run.smithi022.stdout:(24/100): ceph-common-16.2.13-280.g8934282d.el8 1.0 MB/s | 24 MB 00:23 2023-07-02T13:57:33.148 INFO:teuthology.orchestra.run.smithi022.stdout:(25/100): rbd-nbd-16.2.13-280.g8934282d.el8.x86 573 kB/s | 173 kB 00:00 2023-07-02T13:57:33.241 INFO:teuthology.orchestra.run.smithi022.stdout:(26/100): ceph-grafana-dashboards-16.2.13-280.g 260 kB/s | 24 kB 00:00 2023-07-02T13:57:33.410 INFO:teuthology.orchestra.run.smithi016.stdout:(29/100): ceph-mgr-dashboard-16.2.13-280.g89342 836 kB/s | 3.1 MB 00:03 2023-07-02T13:57:33.430 INFO:teuthology.orchestra.run.smithi022.stdout:(27/100): ceph-mgr-cephadm-16.2.13-280.g8934282 618 kB/s | 116 kB 00:00 2023-07-02T13:57:33.657 INFO:teuthology.orchestra.run.smithi022.stdout:(28/100): rbd-mirror-16.2.13-280.g8934282d.el8. 2.8 MB/s | 3.7 MB 00:01 2023-07-02T13:57:33.725 INFO:teuthology.orchestra.run.smithi016.stdout:(30/100): ceph-mgr-modules-core-16.2.13-280.g89 776 kB/s | 244 kB 00:00 2023-07-02T13:57:33.827 INFO:teuthology.orchestra.run.smithi016.stdout:(31/100): ceph-mgr-rook-16.2.13-280.g8934282d.e 269 kB/s | 27 kB 00:00 2023-07-02T13:57:33.928 INFO:teuthology.orchestra.run.smithi016.stdout:(32/100): ceph-prometheus-alerts-16.2.13-280.g8 148 kB/s | 15 kB 00:00 2023-07-02T13:57:34.028 INFO:teuthology.orchestra.run.smithi016.stdout:(33/100): cephadm-16.2.13-280.g8934282d.el8.noa 840 kB/s | 84 kB 00:00 2023-07-02T13:57:34.272 INFO:teuthology.orchestra.run.smithi016.stdout:(34/100): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2023-07-02T13:57:34.362 INFO:teuthology.orchestra.run.smithi016.stdout:(35/100): leveldb-1.22-1.el8.x86_64.rpm 2.0 MB/s | 181 kB 00:00 2023-07-02T13:57:34.414 INFO:teuthology.orchestra.run.smithi016.stdout:(36/100): liboath-2.6.2-3.el8.x86_64.rpm 1.1 MB/s | 59 kB 00:00 2023-07-02T13:57:34.472 INFO:teuthology.orchestra.run.smithi016.stdout:(37/100): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2023-07-02T13:57:34.548 INFO:teuthology.orchestra.run.smithi016.stdout:(38/100): pv-1.6.6-7.el8.x86_64.rpm 853 kB/s | 64 kB 00:00 2023-07-02T13:57:34.614 INFO:teuthology.orchestra.run.smithi016.stdout:(39/100): python3-apipkg-1.5-6.el8.noarch.rpm 318 kB/s | 21 kB 00:00 2023-07-02T13:57:34.679 INFO:teuthology.orchestra.run.smithi016.stdout:(40/100): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 681 kB/s | 44 kB 00:00 2023-07-02T13:57:34.717 INFO:teuthology.orchestra.run.smithi016.stdout:(41/100): ceph-mgr-diskprediction-local-16.2.13 2.1 MB/s | 8.6 MB 00:04 2023-07-02T13:57:34.766 INFO:teuthology.orchestra.run.smithi016.stdout:(42/100): python3-beautifulsoup4-4.6.3-2.el8.1. 2.1 MB/s | 185 kB 00:00 2023-07-02T13:57:34.813 INFO:teuthology.orchestra.run.smithi016.stdout:(43/100): python3-certifi-2018.10.15-7.el8.noar 345 kB/s | 16 kB 00:00 2023-07-02T13:57:34.814 INFO:teuthology.orchestra.run.smithi016.stdout:(44/100): python3-cachetools-3.1.1-4.el8.noarch 340 kB/s | 33 kB 00:00 2023-07-02T13:57:34.923 INFO:teuthology.orchestra.run.smithi016.stdout:(45/100): python3-cheroot-8.5.2-1.el8.noarch.rp 1.5 MB/s | 173 kB 00:00 2023-07-02T13:57:35.019 INFO:teuthology.orchestra.run.smithi016.stdout:(46/100): python3-cherrypy-18.4.0-1.el8.noarch. 1.8 MB/s | 384 kB 00:00 2023-07-02T13:57:35.091 INFO:teuthology.orchestra.run.smithi016.stdout:(47/100): python3-google-auth-1.1.1-10.el8.noar 1.1 MB/s | 81 kB 00:00 2023-07-02T13:57:35.095 INFO:teuthology.orchestra.run.smithi016.stdout:(48/100): python3-execnet-1.7.1-1.el8.noarch.rp 2.0 MB/s | 355 kB 00:00 2023-07-02T13:57:35.142 INFO:teuthology.orchestra.run.smithi016.stdout:(49/100): python3-jaraco-6.2-6.el8.noarch.rpm 220 kB/s | 11 kB 00:00 2023-07-02T13:57:35.143 INFO:teuthology.orchestra.run.smithi016.stdout:(50/100): python3-jaraco-functools-2.0-4.el8.no 370 kB/s | 18 kB 00:00 2023-07-02T13:57:35.189 INFO:teuthology.orchestra.run.smithi016.stdout:(51/100): python3-logutils-0.3.5-11.el8.noarch. 1.1 MB/s | 49 kB 00:00 2023-07-02T13:57:35.234 INFO:teuthology.orchestra.run.smithi016.stdout:(52/100): python3-more-itertools-7.2.0-3.el8.no 1.3 MB/s | 59 kB 00:00 2023-07-02T13:57:35.321 INFO:teuthology.orchestra.run.smithi016.stdout:(53/100): python3-pecan-1.3.2-9.el8.noarch.rpm 3.2 MB/s | 283 kB 00:00 2023-07-02T13:57:35.365 INFO:teuthology.orchestra.run.smithi016.stdout:(54/100): python3-portend-2.6-1.el8.noarch.rpm 353 kB/s | 16 kB 00:00 2023-07-02T13:57:35.444 INFO:teuthology.orchestra.run.smithi016.stdout:(55/100): python3-remoto-1.2.1-1.el8.noarch.rpm 619 kB/s | 49 kB 00:00 2023-07-02T13:57:35.514 INFO:teuthology.orchestra.run.smithi016.stdout:(56/100): python3-kubernetes-11.0.0-6.el8.noarc 3.1 MB/s | 1.2 MB 00:00 2023-07-02T13:57:35.515 INFO:teuthology.orchestra.run.smithi016.stdout:(57/100): python3-repoze-lru-0.7-6.el8.noarch.r 477 kB/s | 34 kB 00:00 2023-07-02T13:57:35.558 INFO:teuthology.orchestra.run.smithi016.stdout:(58/100): python3-rsa-4.8-1.el8.noarch.rpm 1.4 MB/s | 61 kB 00:00 2023-07-02T13:57:35.590 INFO:teuthology.orchestra.run.smithi016.stdout:(59/100): python3-routes-2.4.1-12.el8.noarch.rp 2.6 MB/s | 196 kB 00:00 2023-07-02T13:57:35.595 INFO:teuthology.orchestra.run.smithi016.stdout:(60/100): python3-simplegeneric-0.8.1-17.el8.no 510 kB/s | 19 kB 00:00 2023-07-02T13:57:35.627 INFO:teuthology.orchestra.run.smithi016.stdout:(61/100): python3-singledispatch-3.4.0.3-18.el8 642 kB/s | 24 kB 00:00 2023-07-02T13:57:35.634 INFO:teuthology.orchestra.run.smithi016.stdout:(62/100): python3-tempora-1.14.1-5.el8.noarch.r 767 kB/s | 29 kB 00:00 2023-07-02T13:57:35.685 INFO:teuthology.orchestra.run.smithi016.stdout:(63/100): python3-trustme-0.6.0-4.el8.noarch.rp 481 kB/s | 27 kB 00:00 2023-07-02T13:57:35.726 INFO:teuthology.orchestra.run.smithi016.stdout:(64/100): python3-waitress-1.4.3-1.el8.noarch.r 2.7 MB/s | 255 kB 00:00 2023-07-02T13:57:35.761 INFO:teuthology.orchestra.run.smithi016.stdout:(65/100): python3-webob-1.8.5-1.el8.1.noarch.rp 3.2 MB/s | 251 kB 00:00 2023-07-02T13:57:35.765 INFO:teuthology.orchestra.run.smithi016.stdout:(66/100): python3-websocket-client-0.56.0-5.el8 1.6 MB/s | 61 kB 00:00 2023-07-02T13:57:35.802 INFO:teuthology.orchestra.run.smithi016.stdout:(67/100): python3-webtest-2.0.33-1.el8.noarch.r 2.1 MB/s | 85 kB 00:00 2023-07-02T13:57:35.806 INFO:teuthology.orchestra.run.smithi016.stdout:(68/100): xmlstarlet-1.6.1-20.el8.x86_64.rpm 20 MB/s | 70 kB 00:00 2023-07-02T13:57:35.808 INFO:teuthology.orchestra.run.smithi016.stdout:(69/100): python3-zc-lockfile-2.0-2.el8.noarch. 527 kB/s | 23 kB 00:00 2023-07-02T13:57:35.859 INFO:teuthology.orchestra.run.smithi016.stdout:(70/100): python3-prettytable-0.7.2-14.el8.noar 860 kB/s | 44 kB 00:00 2023-07-02T13:57:35.880 INFO:teuthology.orchestra.run.smithi016.stdout:(71/100): python3-jsonpatch-1.21-2.el8.noarch.r 1.3 MB/s | 27 kB 00:00 2023-07-02T13:57:35.889 INFO:teuthology.orchestra.run.smithi022.stdout:(29/100): ceph-test-16.2.13-280.g8934282d.el8.x 4.9 MB/s | 51 MB 00:10 2023-07-02T13:57:35.903 INFO:teuthology.orchestra.run.smithi016.stdout:(72/100): jq-1.5-12.el8.x86_64.rpm 6.9 MB/s | 161 kB 00:00 2023-07-02T13:57:35.920 INFO:teuthology.orchestra.run.smithi022.stdout:(30/100): ceph-mgr-diskprediction-local-16.2.13 3.8 MB/s | 8.6 MB 00:02 2023-07-02T13:57:35.930 INFO:teuthology.orchestra.run.smithi016.stdout:(73/100): python3-werkzeug-0.12.2-4.el8.noarch. 17 MB/s | 457 kB 00:00 2023-07-02T13:57:35.986 INFO:teuthology.orchestra.run.smithi022.stdout:(31/100): ceph-mgr-rook-16.2.13-280.g8934282d.e 416 kB/s | 27 kB 00:00 2023-07-02T13:57:35.989 INFO:teuthology.orchestra.run.smithi022.stdout:(32/100): ceph-mgr-modules-core-16.2.13-280.g89 2.4 MB/s | 244 kB 00:00 2023-07-02T13:57:36.025 INFO:teuthology.orchestra.run.smithi016.stdout:(74/100): python3-babel-2.5.1-5.el8.noarch.rpm 51 MB/s | 4.8 MB 00:00 2023-07-02T13:57:36.038 INFO:teuthology.orchestra.run.smithi022.stdout:(33/100): cephadm-16.2.13-280.g8934282d.el8.noa 1.7 MB/s | 84 kB 00:00 2023-07-02T13:57:36.049 INFO:teuthology.orchestra.run.smithi022.stdout:(34/100): ceph-prometheus-alerts-16.2.13-280.g8 238 kB/s | 15 kB 00:00 2023-07-02T13:57:36.139 INFO:teuthology.orchestra.run.smithi016.stdout:(75/100): python3-scipy-1.0.0-20.module+el8.1.0 44 MB/s | 15 MB 00:00 2023-07-02T13:57:36.140 INFO:teuthology.orchestra.run.smithi016.stdout:(76/100): python3-markupsafe-0.23-19.el8.x86_64 336 kB/s | 39 kB 00:00 2023-07-02T13:57:36.170 INFO:teuthology.orchestra.run.smithi016.stdout:(77/100): python3-numpy-f2py-1.14.3-9.el8.x86_6 7.4 MB/s | 225 kB 00:00 2023-07-02T13:57:36.340 INFO:teuthology.orchestra.run.smithi022.stdout:(35/100): ceph-mgr-dashboard-16.2.13-280.g89342 1.1 MB/s | 3.1 MB 00:02 2023-07-02T13:57:36.585 INFO:teuthology.orchestra.run.smithi016.stdout:(78/100): ceph-test-16.2.13-280.g8934282d.el8.x 2.7 MB/s | 51 MB 00:18 2023-07-02T13:57:36.589 INFO:teuthology.orchestra.run.smithi016.stdout:(79/100): python3-pyasn1-0.3.7-6.el8.noarch.rpm 302 kB/s | 126 kB 00:00 2023-07-02T13:57:36.591 INFO:teuthology.orchestra.run.smithi016.stdout:(80/100): lttng-ust-2.8.1-11.el8.x86_64.rpm 573 kB/s | 259 kB 00:00 2023-07-02T13:57:36.611 INFO:teuthology.orchestra.run.smithi016.stdout:(81/100): oniguruma-6.8.2-2.el8.x86_64.rpm 8.3 MB/s | 187 kB 00:00 2023-07-02T13:57:36.618 INFO:teuthology.orchestra.run.smithi016.stdout:(82/100): python3-jinja2-2.10.1-2.el8_0.noarch. 20 MB/s | 538 kB 00:00 2023-07-02T13:57:36.634 INFO:teuthology.orchestra.run.smithi016.stdout:(83/100): socat-1.7.3.3-2.el8.x86_64.rpm 13 MB/s | 302 kB 00:00 2023-07-02T13:57:36.644 INFO:teuthology.orchestra.run.smithi016.stdout:(84/100): python3-mako-1.0.6-13.el8.noarch.rpm 5.8 MB/s | 157 kB 00:00 2023-07-02T13:57:36.655 INFO:teuthology.orchestra.run.smithi016.stdout:(85/100): python3-pyasn1-modules-0.3.7-6.el8.no 5.1 MB/s | 110 kB 00:00 2023-07-02T13:57:36.670 INFO:teuthology.orchestra.run.smithi016.stdout:(86/100): librdkafka-0.11.4-1.el8.x86_64.rpm 13 MB/s | 353 kB 00:00 2023-07-02T13:57:36.673 INFO:teuthology.orchestra.run.smithi016.stdout:(87/100): python3-jsonpointer-1.10-11.el8.noarc 1.1 MB/s | 20 kB 00:00 2023-07-02T13:57:36.690 INFO:teuthology.orchestra.run.smithi016.stdout:(88/100): python3-numpy-1.14.3-9.el8.x86_64.rpm 36 MB/s | 3.7 MB 00:00 2023-07-02T13:57:36.691 INFO:teuthology.orchestra.run.smithi016.stdout:(89/100): python3-pytz-2017.2-9.el8.noarch.rpm 2.6 MB/s | 54 kB 00:00 2023-07-02T13:57:36.722 INFO:teuthology.orchestra.run.smithi016.stdout:(90/100): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2023-07-02T13:57:36.742 INFO:teuthology.orchestra.run.smithi016.stdout:(91/100): python3-oauthlib-2.1.0-1.el8.noarch.r 7.6 MB/s | 155 kB 00:00 2023-07-02T13:57:36.763 INFO:teuthology.orchestra.run.smithi016.stdout:(92/100): openblas-threads-0.3.12-1.el8.x86_64. 53 MB/s | 4.7 MB 00:00 2023-07-02T13:57:36.764 INFO:teuthology.orchestra.run.smithi016.stdout:(93/100): python3-requests-oauthlib-1.0.0-1.el8 1.9 MB/s | 43 kB 00:00 2023-07-02T13:57:36.783 INFO:teuthology.orchestra.run.smithi016.stdout:(94/100): fuse-2.9.7-12.el8.x86_64.rpm 4.0 MB/s | 83 kB 00:00 2023-07-02T13:57:36.785 INFO:teuthology.orchestra.run.smithi016.stdout:(95/100): libquadmath-8.4.1-1.el8.x86_64.rpm 7.9 MB/s | 169 kB 00:00 2023-07-02T13:57:36.801 INFO:teuthology.orchestra.run.smithi016.stdout:(96/100): openblas-0.3.12-1.el8.x86_64.rpm 42 MB/s | 4.6 MB 00:00 2023-07-02T13:57:36.801 INFO:teuthology.orchestra.run.smithi016.stdout:(97/100): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.5 MB/s | 47 kB 00:00 2023-07-02T13:57:36.809 INFO:teuthology.orchestra.run.smithi016.stdout:(98/100): libgfortran-8.4.1-1.el8.x86_64.rpm 26 MB/s | 639 kB 00:00 2023-07-02T13:57:37.477 INFO:teuthology.orchestra.run.smithi016.stdout:(99/100): librados2-16.2.13-280.g8934282d.el8.x 5.5 MB/s | 3.7 MB 00:00 2023-07-02T13:57:37.767 INFO:teuthology.orchestra.run.smithi016.stdout:(100/100): librbd1-16.2.13-280.g8934282d.el8.x8 4.3 MB/s | 4.2 MB 00:00 2023-07-02T13:57:37.770 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2023-07-02T13:57:37.771 INFO:teuthology.orchestra.run.smithi016.stdout:Total 6.7 MB/s | 193 MB 00:28 2023-07-02T13:57:37.802 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2023-07-02T13:57:37.902 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2023-07-02T13:57:37.902 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2023-07-02T13:57:38.928 INFO:teuthology.orchestra.run.smithi163.stdout:(100/100): ceph-test-16.2.13-280.g8934282d.el8. 2.0 MB/s | 51 MB 00:26 2023-07-02T13:57:38.933 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2023-07-02T13:57:38.933 INFO:teuthology.orchestra.run.smithi163.stdout:Total 6.3 MB/s | 193 MB 00:30 2023-07-02T13:57:38.964 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2023-07-02T13:57:39.060 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2023-07-02T13:57:39.060 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2023-07-02T13:57:39.527 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2023-07-02T13:57:39.532 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2023-07-02T13:57:40.635 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2023-07-02T13:57:40.639 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2023-07-02T13:57:43.685 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2023-07-02T13:57:43.877 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/102 2023-07-02T13:57:44.065 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : leveldb-1.22-1.el8.x86_64 2/102 2023-07-02T13:57:44.253 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/102 2023-07-02T13:57:44.417 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 4/102 2023-07-02T13:57:44.510 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2023-07-02T13:57:44.548 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 4/102 2023-07-02T13:57:44.710 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/102 2023-07-02T13:57:44.776 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/102 2023-07-02T13:57:44.900 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : leveldb-1.22-1.el8.x86_64 2/102 2023-07-02T13:57:45.124 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/102 2023-07-02T13:57:45.185 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/102 2023-07-02T13:57:45.316 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 4/102 2023-07-02T13:57:45.367 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librados2-2:16.2.13-280.g8934282d.el8.x86_64 6/102 2023-07-02T13:57:45.431 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 4/102 2023-07-02T13:57:45.632 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/102 2023-07-02T13:57:45.820 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:16.2.13-280.g8934282d.el8.x86_64 6/102 2023-07-02T13:57:46.000 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librbd1-2:16.2.13-280.g8934282d.el8.x86_64 7/102 2023-07-02T13:57:46.025 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/102 2023-07-02T13:57:46.129 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:16.2.13-280.g8934282d.el8.x86_64 7/102 2023-07-02T13:57:46.232 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librados2-2:16.2.13-280.g8934282d.el8.x86_64 6/102 2023-07-02T13:57:46.334 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs2-2:16.2.13-280.g8934282d.el8.x86_64 8/102 2023-07-02T13:57:46.645 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:16.2.13-280.g8934282d.el8.x86_64 6/102 2023-07-02T13:57:46.730 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs2-2:16.2.13-280.g8934282d.el8.x86_64 8/102 2023-07-02T13:57:46.849 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librbd1-2:16.2.13-280.g8934282d.el8.x86_64 7/102 2023-07-02T13:57:46.909 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librgw2-2:16.2.13-280.g8934282d.el8.x86_64 9/102 2023-07-02T13:57:47.003 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:16.2.13-280.g8934282d.el8.x86_64 7/102 2023-07-02T13:57:47.007 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librgw2-2:16.2.13-280.g8934282d.el8.x86_64 9/102 2023-07-02T13:57:47.190 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs2-2:16.2.13-280.g8934282d.el8.x86_64 8/102 2023-07-02T13:57:47.236 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rados-2:16.2.13-280.g8934282d.el8.x86_64 10/102 2023-07-02T13:57:47.480 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/102 2023-07-02T13:57:47.567 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs2-2:16.2.13-280.g8934282d.el8.x86_64 8/102 2023-07-02T13:57:47.641 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/102 2023-07-02T13:57:47.740 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librgw2-2:16.2.13-280.g8934282d.el8.x86_64 9/102 2023-07-02T13:57:47.800 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephsqlite-2:16.2.13-280.g8934282d.el8.x86_64 13/102 2023-07-02T13:57:47.837 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librgw2-2:16.2.13-280.g8934282d.el8.x86_64 9/102 2023-07-02T13:57:47.922 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephsqlite-2:16.2.13-280.g8934282d.el8.x86_64 13/102 2023-07-02T13:57:48.084 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libradosstriper1-2:16.2.13-280.g8934282d.el8.x86 14/102 2023-07-02T13:57:48.097 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rados-2:16.2.13-280.g8934282d.el8.x86_64 10/102 2023-07-02T13:57:48.164 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libradosstriper1-2:16.2.13-280.g8934282d.el8.x86 14/102 2023-07-02T13:57:48.309 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 15/102 2023-07-02T13:57:48.319 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/102 2023-07-02T13:57:48.436 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 15/102 2023-07-02T13:57:48.464 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/102 2023-07-02T13:57:48.608 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 16/102 2023-07-02T13:57:48.649 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephsqlite-2:16.2.13-280.g8934282d.el8.x86_64 13/102 2023-07-02T13:57:48.753 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephsqlite-2:16.2.13-280.g8934282d.el8.x86_64 13/102 2023-07-02T13:57:48.932 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libradosstriper1-2:16.2.13-280.g8934282d.el8.x86 14/102 2023-07-02T13:57:49.019 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libradosstriper1-2:16.2.13-280.g8934282d.el8.x86 14/102 2023-07-02T13:57:49.190 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 16/102 2023-07-02T13:57:49.190 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 15/102 2023-07-02T13:57:49.330 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 15/102 2023-07-02T13:57:49.350 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 17/102 2023-07-02T13:57:49.507 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 16/102 2023-07-02T13:57:49.897 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 17/102 2023-07-02T13:57:50.059 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 16/102 2023-07-02T13:57:50.075 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-0.3.12-1.el8.x86_64 18/102 2023-07-02T13:57:50.240 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 17/102 2023-07-02T13:57:50.777 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 17/102 2023-07-02T13:57:50.814 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 18/102 2023-07-02T13:57:51.006 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-0.3.12-1.el8.x86_64 18/102 2023-07-02T13:57:51.079 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 19/102 2023-07-02T13:57:51.250 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 20/102 2023-07-02T13:57:51.405 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 21/102 2023-07-02T13:57:51.678 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 22/102 2023-07-02T13:57:51.725 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 18/102 2023-07-02T13:57:51.938 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/102 2023-07-02T13:57:52.018 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 19/102 2023-07-02T13:57:52.183 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 24/102 2023-07-02T13:57:52.230 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 20/102 2023-07-02T13:57:52.323 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-common-2:16.2.13-280.g8934282d.el8. 25/102 2023-07-02T13:57:52.410 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 21/102 2023-07-02T13:57:52.448 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-argparse-2:16.2.13-280.g8934282d.el 26/102 2023-07-02T13:57:52.662 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 22/102 2023-07-02T13:57:52.693 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cephfs-2:16.2.13-280.g8934282d.el8.x86_6 27/102 2023-07-02T13:57:52.975 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/102 2023-07-02T13:57:53.196 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 24/102 2023-07-02T13:57:53.354 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-common-2:16.2.13-280.g8934282d.el8. 25/102 2023-07-02T13:57:53.472 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-argparse-2:16.2.13-280.g8934282d.el 26/102 2023-07-02T13:57:53.704 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cephfs-2:16.2.13-280.g8934282d.el8.x86_6 27/102 2023-07-02T13:57:53.721 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 28/102 2023-07-02T13:57:54.052 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 29/102 2023-07-02T13:57:54.328 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 30/102 2023-07-02T13:57:54.675 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 31/102 2023-07-02T13:57:54.765 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 28/102 2023-07-02T13:57:54.939 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 32/102 2023-07-02T13:57:55.112 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 29/102 2023-07-02T13:57:55.174 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 33/102 2023-07-02T13:57:55.389 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 30/102 2023-07-02T13:57:55.636 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 31/102 2023-07-02T13:57:55.921 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 32/102 2023-07-02T13:57:56.147 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 33/102 2023-07-02T13:57:57.336 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 33/102 2023-07-02T13:57:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 34/102 2023-07-02T13:57:57.865 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rsa-4.8-1.el8.noarch 35/102 2023-07-02T13:57:58.078 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 36/102 2023-07-02T13:57:58.204 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 33/102 2023-07-02T13:57:58.253 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rbd-2:16.2.13-280.g8934282d.el8.x86_64 37/102 2023-07-02T13:57:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rgw-2:16.2.13-280.g8934282d.el8.x86_64 38/102 2023-07-02T13:57:58.509 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 34/102 2023-07-02T13:57:58.618 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados-devel-2:16.2.13-280.g8934282d.el8.x86_6 39/102 2023-07-02T13:57:58.736 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rsa-4.8-1.el8.noarch 35/102 2023-07-02T13:57:58.782 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fuse-2.9.7-12.el8.x86_64 40/102 2023-07-02T13:57:58.957 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 36/102 2023-07-02T13:57:58.982 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 41/102 2023-07-02T13:57:59.117 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rbd-2:16.2.13-280.g8934282d.el8.x86_64 37/102 2023-07-02T13:57:59.167 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 42/102 2023-07-02T13:57:59.277 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rgw-2:16.2.13-280.g8934282d.el8.x86_64 38/102 2023-07-02T13:57:59.371 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 43/102 2023-07-02T13:57:59.458 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librados-devel-2:16.2.13-280.g8934282d.el8.x86_6 39/102 2023-07-02T13:57:59.557 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 44/102 2023-07-02T13:57:59.604 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : fuse-2.9.7-12.el8.x86_64 40/102 2023-07-02T13:57:59.650 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 44/102 2023-07-02T13:57:59.779 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 41/102 2023-07-02T13:57:59.826 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : jq-1.5-12.el8.x86_64 45/102 2023-07-02T13:57:59.962 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 46/102 2023-07-02T13:57:59.994 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 42/102 2023-07-02T13:58:00.153 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 47/102 2023-07-02T13:58:00.183 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 43/102 2023-07-02T13:58:00.319 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 48/102 2023-07-02T13:58:00.347 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 44/102 2023-07-02T13:58:00.412 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 44/102 2023-07-02T13:58:00.580 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : jq-1.5-12.el8.x86_64 45/102 2023-07-02T13:58:00.587 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 49/102 2023-07-02T13:58:00.748 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 46/102 2023-07-02T13:58:00.818 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 50/102 2023-07-02T13:58:00.978 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/102 2023-07-02T13:58:01.072 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 47/102 2023-07-02T13:58:01.147 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 52/102 2023-07-02T13:58:01.288 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 53/102 2023-07-02T13:58:01.393 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 48/102 2023-07-02T13:58:01.499 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 54/102 2023-07-02T13:58:01.644 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 49/102 2023-07-02T13:58:01.829 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 50/102 2023-07-02T13:58:01.995 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/102 2023-07-02T13:58:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 55/102 2023-07-02T13:58:02.159 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 52/102 2023-07-02T13:58:02.178 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 56/102 2023-07-02T13:58:02.324 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 53/102 2023-07-02T13:58:02.331 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 57/102 2023-07-02T13:58:02.504 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 54/102 2023-07-02T13:58:02.582 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 58/102 2023-07-02T13:58:02.715 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 55/102 2023-07-02T13:58:02.764 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 59/102 2023-07-02T13:58:02.916 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 56/102 2023-07-02T13:58:02.948 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 60/102 2023-07-02T13:58:03.085 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 57/102 2023-07-02T13:58:03.254 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-portend-2.6-1.el8.noarch 61/102 2023-07-02T13:58:03.306 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 58/102 2023-07-02T13:58:03.465 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 62/102 2023-07-02T13:58:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 59/102 2023-07-02T13:58:03.668 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 63/102 2023-07-02T13:58:03.710 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 60/102 2023-07-02T13:58:03.881 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 64/102 2023-07-02T13:58:03.966 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-portend-2.6-1.el8.noarch 61/102 2023-07-02T13:58:04.203 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 62/102 2023-07-02T13:58:04.386 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 63/102 2023-07-02T13:58:04.602 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 64/102 2023-07-02T13:58:05.057 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 65/102 2023-07-02T13:58:05.341 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 66/102 2023-07-02T13:58:05.569 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 67/102 2023-07-02T13:58:05.725 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 65/102 2023-07-02T13:58:05.876 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 68/102 2023-07-02T13:58:06.044 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 66/102 2023-07-02T13:58:06.134 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 69/102 2023-07-02T13:58:06.264 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 67/102 2023-07-02T13:58:06.330 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 70/102 2023-07-02T13:58:06.553 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 71/102 2023-07-02T13:58:06.578 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 68/102 2023-07-02T13:58:06.798 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 72/102 2023-07-02T13:58:06.808 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 69/102 2023-07-02T13:58:06.966 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 73/102 2023-07-02T13:58:07.027 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 70/102 2023-07-02T13:58:07.168 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 74/102 2023-07-02T13:58:07.260 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 71/102 2023-07-02T13:58:07.348 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 75/102 2023-07-02T13:58:07.534 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 72/102 2023-07-02T13:58:07.722 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 73/102 2023-07-02T13:58:07.900 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 74/102 2023-07-02T13:58:08.112 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 75/102 2023-07-02T13:58:12.396 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:16.2.13-280.g8934282d.el8.x86_64 76/102 2023-07-02T13:58:12.656 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-common-2:16.2.13-280.g8934282d.el8.x86_64 76/102 2023-07-02T13:58:12.883 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:16.2.13-280.g8934282d.el8.x86_64 76/102 2023-07-02T13:58:13.137 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-common-2:16.2.13-280.g8934282d.el8.x86_64 76/102 2023-07-02T13:58:13.405 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:16.2.13-280.g8934282d.el8.x86_64 76/102 2023-07-02T13:58:13.693 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-base-2:16.2.13-280.g8934282d.el8.x86_64 77/102 2023-07-02T13:58:13.782 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-base-2:16.2.13-280.g8934282d.el8.x86_64 77/102 2023-07-02T13:58:13.844 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:16.2.13-280.g8934282d.el8.x86_64 76/102 2023-07-02T13:58:13.905 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-selinux-2:16.2.13-280.g8934282d.el8.x86_64 78/102 2023-07-02T13:58:14.109 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-base-2:16.2.13-280.g8934282d.el8.x86_64 77/102 2023-07-02T13:58:14.194 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-base-2:16.2.13-280.g8934282d.el8.x86_64 77/102 2023-07-02T13:58:14.311 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-selinux-2:16.2.13-280.g8934282d.el8.x86_64 78/102 2023-07-02T13:58:31.920 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-selinux-2:16.2.13-280.g8934282d.el8.x86_64 78/102 2023-07-02T13:58:31.920 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /sys 2023-07-02T13:58:31.920 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /proc 2023-07-02T13:58:31.920 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /dev 2023-07-02T13:58:31.920 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /run 2023-07-02T13:58:31.921 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /mnt 2023-07-02T13:58:31.921 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /var/tmp 2023-07-02T13:58:31.921 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /home 2023-07-02T13:58:31.921 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /tmp 2023-07-02T13:58:31.921 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /dev 2023-07-02T13:58:31.921 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T13:58:32.291 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mds-2:16.2.13-280.g8934282d.el8.x86_64 79/102 2023-07-02T13:58:32.719 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mds-2:16.2.13-280.g8934282d.el8.x86_64 79/102 2023-07-02T13:58:32.900 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mon-2:16.2.13-280.g8934282d.el8.x86_64 80/102 2023-07-02T13:58:32.977 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-selinux-2:16.2.13-280.g8934282d.el8.x86_64 78/102 2023-07-02T13:58:32.978 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /sys 2023-07-02T13:58:32.978 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /proc 2023-07-02T13:58:32.978 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /dev 2023-07-02T13:58:32.978 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /run 2023-07-02T13:58:32.978 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /mnt 2023-07-02T13:58:32.978 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /var/tmp 2023-07-02T13:58:32.979 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /home 2023-07-02T13:58:32.979 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /tmp 2023-07-02T13:58:32.979 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /dev 2023-07-02T13:58:32.979 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T13:58:33.486 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mds-2:16.2.13-280.g8934282d.el8.x86_64 79/102 2023-07-02T13:58:33.942 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mds-2:16.2.13-280.g8934282d.el8.x86_64 79/102 2023-07-02T13:58:34.111 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mon-2:16.2.13-280.g8934282d.el8.x86_64 80/102 2023-07-02T13:58:34.386 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mon-2:16.2.13-280.g8934282d.el8.x86_64 80/102 2023-07-02T13:58:34.599 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-osd-2:16.2.13-280.g8934282d.el8.x86_64 81/102 2023-07-02T13:58:34.620 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-osd-2:16.2.13-280.g8934282d.el8.x86_64 81/102 2023-07-02T13:58:35.691 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mon-2:16.2.13-280.g8934282d.el8.x86_64 80/102 2023-07-02T13:58:35.861 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-osd-2:16.2.13-280.g8934282d.el8.x86_64 81/102 2023-07-02T13:58:35.883 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-osd-2:16.2.13-280.g8934282d.el8.x86_64 81/102 2023-07-02T13:58:37.510 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:16.2.13-280.g8934282d.el8.noarch 82/102 2023-07-02T13:58:37.683 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephadm-2:16.2.13-280.g8934282d.el8.noarch 82/102 2023-07-02T13:58:37.863 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-prometheus-alerts-2:16.2.13-280.g8934282d.e 83/102 2023-07-02T13:58:38.051 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-grafana-dashboards-2:16.2.13-280.g8934282d. 84/102 2023-07-02T13:58:38.249 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-cephadm-2:16.2.13-280.g8934282d.el8.noa 85/102 2023-07-02T13:58:38.848 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:16.2.13-280.g8934282d.el8.noarch 82/102 2023-07-02T13:58:38.967 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephadm-2:16.2.13-280.g8934282d.el8.noarch 82/102 2023-07-02T13:58:39.192 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-prometheus-alerts-2:16.2.13-280.g8934282d.e 83/102 2023-07-02T13:58:39.215 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.13-280.g8934282d.el8.noa 85/102 2023-07-02T13:58:39.356 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-grafana-dashboards-2:16.2.13-280.g8934282d. 84/102 2023-07-02T13:58:39.504 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-dashboard-2:16.2.13-280.g8934282d.el8.n 86/102 2023-07-02T13:58:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-cephadm-2:16.2.13-280.g8934282d.el8.noa 85/102 2023-07-02T13:58:40.549 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.13-280.g8934282d.el8.n 86/102 2023-07-02T13:58:40.569 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.13-280.g8934282d.el8.noa 85/102 2023-07-02T13:58:40.706 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.13-280.g893 87/102 2023-07-02T13:58:40.802 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.13-280.g893 87/102 2023-07-02T13:58:40.839 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-dashboard-2:16.2.13-280.g8934282d.el8.n 86/102 2023-07-02T13:58:41.181 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-modules-core-2:16.2.13-280.g8934282d.el 88/102 2023-07-02T13:58:41.366 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-2:16.2.13-280.g8934282d.el8.x86_64 89/102 2023-07-02T13:58:41.392 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-2:16.2.13-280.g8934282d.el8.x86_64 89/102 2023-07-02T13:58:41.523 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-rook-2:16.2.13-280.g8934282d.el8.noarch 90/102 2023-07-02T13:58:41.534 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.13-280.g8934282d.el8.noarch 90/102 2023-07-02T13:58:41.950 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.13-280.g8934282d.el8.n 86/102 2023-07-02T13:58:42.118 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.13-280.g893 87/102 2023-07-02T13:58:42.218 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.13-280.g893 87/102 2023-07-02T13:58:42.575 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-2:16.2.13-280.g8934282d.el8.x86_64 91/102 2023-07-02T13:58:42.591 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-modules-core-2:16.2.13-280.g8934282d.el 88/102 2023-07-02T13:58:42.745 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-radosgw-2:16.2.13-280.g8934282d.el8.x86_64 92/102 2023-07-02T13:58:42.772 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-2:16.2.13-280.g8934282d.el8.x86_64 89/102 2023-07-02T13:58:42.799 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-2:16.2.13-280.g8934282d.el8.x86_64 89/102 2023-07-02T13:58:42.839 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-radosgw-2:16.2.13-280.g8934282d.el8.x86_64 92/102 2023-07-02T13:58:42.926 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-rook-2:16.2.13-280.g8934282d.el8.noarch 90/102 2023-07-02T13:58:42.938 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.13-280.g8934282d.el8.noarch 90/102 2023-07-02T13:58:43.023 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-immutable-object-cache-2:16.2.13-280.g89342 93/102 2023-07-02T13:58:43.369 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.13-280.g89342 93/102 2023-07-02T13:58:43.523 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-mirror-2:16.2.13-280.g8934282d.el8.x86_64 94/102 2023-07-02T13:58:44.074 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-2:16.2.13-280.g8934282d.el8.x86_64 91/102 2023-07-02T13:58:44.263 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-radosgw-2:16.2.13-280.g8934282d.el8.x86_64 92/102 2023-07-02T13:58:44.340 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-radosgw-2:16.2.13-280.g8934282d.el8.x86_64 92/102 2023-07-02T13:58:44.510 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-immutable-object-cache-2:16.2.13-280.g89342 93/102 2023-07-02T13:58:44.877 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.13-280.g89342 93/102 2023-07-02T13:58:45.026 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-mirror-2:16.2.13-280.g8934282d.el8.x86_64 94/102 2023-07-02T13:58:48.065 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-mirror-2:16.2.13-280.g8934282d.el8.x86_64 94/102 2023-07-02T13:58:48.396 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-test-2:16.2.13-280.g8934282d.el8.x86_64 95/102 2023-07-02T13:58:48.551 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-fuse-2:16.2.13-280.g8934282d.el8.x86_64 96/102 2023-07-02T13:58:48.705 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs-devel-2:16.2.13-280.g8934282d.el8.x86_ 97/102 2023-07-02T13:58:48.887 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-fuse-2:16.2.13-280.g8934282d.el8.x86_64 98/102 2023-07-02T13:58:49.085 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-nbd-2:16.2.13-280.g8934282d.el8.x86_64 99/102 2023-07-02T13:58:49.257 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : pv-1.6.6-7.el8.x86_64 100/102 2023-07-02T13:58:49.257 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 101/102 2023-07-02T13:58:49.466 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 101/102 2023-07-02T13:58:49.466 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 102/102 2023-07-02T13:58:49.845 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-mirror-2:16.2.13-280.g8934282d.el8.x86_64 94/102 2023-07-02T13:58:50.163 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-test-2:16.2.13-280.g8934282d.el8.x86_64 95/102 2023-07-02T13:58:50.285 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 102/102 2023-07-02T13:58:50.285 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-2:16.2.13-280.g8934282d.el8.x86_64 1/102 2023-07-02T13:58:50.286 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-base-2:16.2.13-280.g8934282d.el8.x86_64 2/102 2023-07-02T13:58:50.286 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-common-2:16.2.13-280.g8934282d.el8.x86_64 3/102 2023-07-02T13:58:50.286 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-fuse-2:16.2.13-280.g8934282d.el8.x86_64 4/102 2023-07-02T13:58:50.286 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-immutable-object-cache-2:16.2.13-280.g89342 5/102 2023-07-02T13:58:50.286 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mds-2:16.2.13-280.g8934282d.el8.x86_64 6/102 2023-07-02T13:58:50.286 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-2:16.2.13-280.g8934282d.el8.x86_64 7/102 2023-07-02T13:58:50.287 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mon-2:16.2.13-280.g8934282d.el8.x86_64 8/102 2023-07-02T13:58:50.287 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-osd-2:16.2.13-280.g8934282d.el8.x86_64 9/102 2023-07-02T13:58:50.287 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-radosgw-2:16.2.13-280.g8934282d.el8.x86_64 10/102 2023-07-02T13:58:50.287 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-selinux-2:16.2.13-280.g8934282d.el8.x86_64 11/102 2023-07-02T13:58:50.287 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-test-2:16.2.13-280.g8934282d.el8.x86_64 12/102 2023-07-02T13:58:50.287 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs-devel-2:16.2.13-280.g8934282d.el8.x86_ 13/102 2023-07-02T13:58:50.288 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs2-2:16.2.13-280.g8934282d.el8.x86_64 14/102 2023-07-02T13:58:50.288 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephsqlite-2:16.2.13-280.g8934282d.el8.x86_64 15/102 2023-07-02T13:58:50.288 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados-devel-2:16.2.13-280.g8934282d.el8.x86_6 16/102 2023-07-02T13:58:50.288 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libradosstriper1-2:16.2.13-280.g8934282d.el8.x86 17/102 2023-07-02T13:58:50.288 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librgw2-2:16.2.13-280.g8934282d.el8.x86_64 18/102 2023-07-02T13:58:50.288 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-argparse-2:16.2.13-280.g8934282d.el 19/102 2023-07-02T13:58:50.288 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-common-2:16.2.13-280.g8934282d.el8. 20/102 2023-07-02T13:58:50.289 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cephfs-2:16.2.13-280.g8934282d.el8.x86_6 21/102 2023-07-02T13:58:50.289 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rados-2:16.2.13-280.g8934282d.el8.x86_64 22/102 2023-07-02T13:58:50.289 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rbd-2:16.2.13-280.g8934282d.el8.x86_64 23/102 2023-07-02T13:58:50.289 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rgw-2:16.2.13-280.g8934282d.el8.x86_64 24/102 2023-07-02T13:58:50.289 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-fuse-2:16.2.13-280.g8934282d.el8.x86_64 25/102 2023-07-02T13:58:50.289 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-mirror-2:16.2.13-280.g8934282d.el8.x86_64 26/102 2023-07-02T13:58:50.289 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-nbd-2:16.2.13-280.g8934282d.el8.x86_64 27/102 2023-07-02T13:58:50.289 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-grafana-dashboards-2:16.2.13-280.g8934282d. 28/102 2023-07-02T13:58:50.295 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-cephadm-2:16.2.13-280.g8934282d.el8.noa 29/102 2023-07-02T13:58:50.295 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-dashboard-2:16.2.13-280.g8934282d.el8.n 30/102 2023-07-02T13:58:50.295 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.13-280.g893 31/102 2023-07-02T13:58:50.296 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-modules-core-2:16.2.13-280.g8934282d.el 32/102 2023-07-02T13:58:50.296 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-rook-2:16.2.13-280.g8934282d.el8.noarch 33/102 2023-07-02T13:58:50.296 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-prometheus-alerts-2:16.2.13-280.g8934282d.e 34/102 2023-07-02T13:58:50.296 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephadm-2:16.2.13-280.g8934282d.el8.noarch 35/102 2023-07-02T13:58:50.296 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 36/102 2023-07-02T13:58:50.296 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : leveldb-1.22-1.el8.x86_64 37/102 2023-07-02T13:58:50.296 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 38/102 2023-07-02T13:58:50.297 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 39/102 2023-07-02T13:58:50.297 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : pv-1.6.6-7.el8.x86_64 40/102 2023-07-02T13:58:50.297 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 41/102 2023-07-02T13:58:50.297 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 42/102 2023-07-02T13:58:50.297 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 43/102 2023-07-02T13:58:50.297 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 44/102 2023-07-02T13:58:50.297 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 45/102 2023-07-02T13:58:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 46/102 2023-07-02T13:58:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 47/102 2023-07-02T13:58:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 48/102 2023-07-02T13:58:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/102 2023-07-02T13:58:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/102 2023-07-02T13:58:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/102 2023-07-02T13:58:50.298 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/102 2023-07-02T13:58:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/102 2023-07-02T13:58:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/102 2023-07-02T13:58:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 55/102 2023-07-02T13:58:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-portend-2.6-1.el8.noarch 56/102 2023-07-02T13:58:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 57/102 2023-07-02T13:58:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/102 2023-07-02T13:58:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/102 2023-07-02T13:58:50.299 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/102 2023-07-02T13:58:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/102 2023-07-02T13:58:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/102 2023-07-02T13:58:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/102 2023-07-02T13:58:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/102 2023-07-02T13:58:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/102 2023-07-02T13:58:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/102 2023-07-02T13:58:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/102 2023-07-02T13:58:50.300 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/102 2023-07-02T13:58:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/102 2023-07-02T13:58:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 70/102 2023-07-02T13:58:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 71/102 2023-07-02T13:58:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 72/102 2023-07-02T13:58:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 73/102 2023-07-02T13:58:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : jq-1.5-12.el8.x86_64 74/102 2023-07-02T13:58:50.301 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 75/102 2023-07-02T13:58:50.302 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 76/102 2023-07-02T13:58:50.302 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/102 2023-07-02T13:58:50.302 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 78/102 2023-07-02T13:58:50.302 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 79/102 2023-07-02T13:58:50.302 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 80/102 2023-07-02T13:58:50.302 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 81/102 2023-07-02T13:58:50.302 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 82/102 2023-07-02T13:58:50.302 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 83/102 2023-07-02T13:58:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 84/102 2023-07-02T13:58:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 85/102 2023-07-02T13:58:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 86/102 2023-07-02T13:58:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 87/102 2023-07-02T13:58:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 88/102 2023-07-02T13:58:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 89/102 2023-07-02T13:58:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 90/102 2023-07-02T13:58:50.303 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 91/102 2023-07-02T13:58:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 92/102 2023-07-02T13:58:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 93/102 2023-07-02T13:58:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 94/102 2023-07-02T13:58:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 95/102 2023-07-02T13:58:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 96/102 2023-07-02T13:58:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 97/102 2023-07-02T13:58:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 98/102 2023-07-02T13:58:50.304 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:16.2.13-280.g8934282d.el8.x86_64 99/102 2023-07-02T13:58:50.305 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 100/102 2023-07-02T13:58:50.305 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:16.2.13-280.g8934282d.el8.x86_64 101/102 2023-07-02T13:58:50.305 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-fuse-2:16.2.13-280.g8934282d.el8.x86_64 96/102 2023-07-02T13:58:50.451 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs-devel-2:16.2.13-280.g8934282d.el8.x86_ 97/102 2023-07-02T13:58:50.650 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-fuse-2:16.2.13-280.g8934282d.el8.x86_64 98/102 2023-07-02T13:58:50.823 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-nbd-2:16.2.13-280.g8934282d.el8.x86_64 99/102 2023-07-02T13:58:50.961 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : pv-1.6.6-7.el8.x86_64 100/102 2023-07-02T13:58:50.961 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 101/102 2023-07-02T13:58:51.195 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 101/102 2023-07-02T13:58:51.195 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 102/102 2023-07-02T13:58:52.034 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 102/102 2023-07-02T13:58:52.034 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-2:16.2.13-280.g8934282d.el8.x86_64 1/102 2023-07-02T13:58:52.034 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-base-2:16.2.13-280.g8934282d.el8.x86_64 2/102 2023-07-02T13:58:52.034 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-common-2:16.2.13-280.g8934282d.el8.x86_64 3/102 2023-07-02T13:58:52.035 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-fuse-2:16.2.13-280.g8934282d.el8.x86_64 4/102 2023-07-02T13:58:52.035 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-immutable-object-cache-2:16.2.13-280.g89342 5/102 2023-07-02T13:58:52.035 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mds-2:16.2.13-280.g8934282d.el8.x86_64 6/102 2023-07-02T13:58:52.035 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-2:16.2.13-280.g8934282d.el8.x86_64 7/102 2023-07-02T13:58:52.035 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mon-2:16.2.13-280.g8934282d.el8.x86_64 8/102 2023-07-02T13:58:52.035 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-osd-2:16.2.13-280.g8934282d.el8.x86_64 9/102 2023-07-02T13:58:52.035 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-radosgw-2:16.2.13-280.g8934282d.el8.x86_64 10/102 2023-07-02T13:58:52.036 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-selinux-2:16.2.13-280.g8934282d.el8.x86_64 11/102 2023-07-02T13:58:52.036 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-test-2:16.2.13-280.g8934282d.el8.x86_64 12/102 2023-07-02T13:58:52.036 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs-devel-2:16.2.13-280.g8934282d.el8.x86_ 13/102 2023-07-02T13:58:52.036 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs2-2:16.2.13-280.g8934282d.el8.x86_64 14/102 2023-07-02T13:58:52.036 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephsqlite-2:16.2.13-280.g8934282d.el8.x86_64 15/102 2023-07-02T13:58:52.036 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados-devel-2:16.2.13-280.g8934282d.el8.x86_6 16/102 2023-07-02T13:58:52.036 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libradosstriper1-2:16.2.13-280.g8934282d.el8.x86 17/102 2023-07-02T13:58:52.037 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librgw2-2:16.2.13-280.g8934282d.el8.x86_64 18/102 2023-07-02T13:58:52.037 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-argparse-2:16.2.13-280.g8934282d.el 19/102 2023-07-02T13:58:52.037 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-common-2:16.2.13-280.g8934282d.el8. 20/102 2023-07-02T13:58:52.037 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cephfs-2:16.2.13-280.g8934282d.el8.x86_6 21/102 2023-07-02T13:58:52.037 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rados-2:16.2.13-280.g8934282d.el8.x86_64 22/102 2023-07-02T13:58:52.037 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rbd-2:16.2.13-280.g8934282d.el8.x86_64 23/102 2023-07-02T13:58:52.037 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rgw-2:16.2.13-280.g8934282d.el8.x86_64 24/102 2023-07-02T13:58:52.037 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-fuse-2:16.2.13-280.g8934282d.el8.x86_64 25/102 2023-07-02T13:58:52.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-mirror-2:16.2.13-280.g8934282d.el8.x86_64 26/102 2023-07-02T13:58:52.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-nbd-2:16.2.13-280.g8934282d.el8.x86_64 27/102 2023-07-02T13:58:52.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-grafana-dashboards-2:16.2.13-280.g8934282d. 28/102 2023-07-02T13:58:52.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-cephadm-2:16.2.13-280.g8934282d.el8.noa 29/102 2023-07-02T13:58:52.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-dashboard-2:16.2.13-280.g8934282d.el8.n 30/102 2023-07-02T13:58:52.043 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.13-280.g893 31/102 2023-07-02T13:58:52.043 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-modules-core-2:16.2.13-280.g8934282d.el 32/102 2023-07-02T13:58:52.043 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-rook-2:16.2.13-280.g8934282d.el8.noarch 33/102 2023-07-02T13:58:52.043 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-prometheus-alerts-2:16.2.13-280.g8934282d.e 34/102 2023-07-02T13:58:52.043 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephadm-2:16.2.13-280.g8934282d.el8.noarch 35/102 2023-07-02T13:58:52.044 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 36/102 2023-07-02T13:58:52.044 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : leveldb-1.22-1.el8.x86_64 37/102 2023-07-02T13:58:52.044 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 38/102 2023-07-02T13:58:52.044 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 39/102 2023-07-02T13:58:52.044 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : pv-1.6.6-7.el8.x86_64 40/102 2023-07-02T13:58:52.044 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 41/102 2023-07-02T13:58:52.045 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 42/102 2023-07-02T13:58:52.045 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 43/102 2023-07-02T13:58:52.045 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 44/102 2023-07-02T13:58:52.045 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 45/102 2023-07-02T13:58:52.045 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 46/102 2023-07-02T13:58:52.045 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 47/102 2023-07-02T13:58:52.045 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 48/102 2023-07-02T13:58:52.045 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/102 2023-07-02T13:58:52.046 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/102 2023-07-02T13:58:52.046 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/102 2023-07-02T13:58:52.046 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/102 2023-07-02T13:58:52.046 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/102 2023-07-02T13:58:52.046 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/102 2023-07-02T13:58:52.046 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 55/102 2023-07-02T13:58:52.046 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-portend-2.6-1.el8.noarch 56/102 2023-07-02T13:58:52.047 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 57/102 2023-07-02T13:58:52.047 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/102 2023-07-02T13:58:52.047 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/102 2023-07-02T13:58:52.047 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/102 2023-07-02T13:58:52.047 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/102 2023-07-02T13:58:52.047 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/102 2023-07-02T13:58:52.047 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/102 2023-07-02T13:58:52.047 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/102 2023-07-02T13:58:52.048 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/102 2023-07-02T13:58:52.048 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/102 2023-07-02T13:58:52.048 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/102 2023-07-02T13:58:52.048 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/102 2023-07-02T13:58:52.048 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/102 2023-07-02T13:58:52.048 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 70/102 2023-07-02T13:58:52.048 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 71/102 2023-07-02T13:58:52.048 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 72/102 2023-07-02T13:58:52.049 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 73/102 2023-07-02T13:58:52.049 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : jq-1.5-12.el8.x86_64 74/102 2023-07-02T13:58:52.049 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 75/102 2023-07-02T13:58:52.049 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 76/102 2023-07-02T13:58:52.049 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/102 2023-07-02T13:58:52.049 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 78/102 2023-07-02T13:58:52.049 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 79/102 2023-07-02T13:58:52.049 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 80/102 2023-07-02T13:58:52.050 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 81/102 2023-07-02T13:58:52.050 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 82/102 2023-07-02T13:58:52.050 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 83/102 2023-07-02T13:58:52.050 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 84/102 2023-07-02T13:58:52.050 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 85/102 2023-07-02T13:58:52.050 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 86/102 2023-07-02T13:58:52.050 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 87/102 2023-07-02T13:58:52.051 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 88/102 2023-07-02T13:58:52.051 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 89/102 2023-07-02T13:58:52.051 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 90/102 2023-07-02T13:58:52.051 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 91/102 2023-07-02T13:58:52.051 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 92/102 2023-07-02T13:58:52.051 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 93/102 2023-07-02T13:58:52.051 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 94/102 2023-07-02T13:58:52.051 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 95/102 2023-07-02T13:58:52.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 96/102 2023-07-02T13:58:52.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 97/102 2023-07-02T13:58:52.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 98/102 2023-07-02T13:58:52.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:16.2.13-280.g8934282d.el8.x86_64 99/102 2023-07-02T13:58:52.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 100/102 2023-07-02T13:58:52.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:16.2.13-280.g8934282d.el8.x86_64 101/102 2023-07-02T13:58:52.408 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 102/102 2023-07-02T13:58:52.409 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:53 ago on Sun 02 Jul 2023 01:56:59 PM UTC. 2023-07-02T13:58:54.332 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 102/102 2023-07-02T13:58:54.332 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:55 ago on Sun 02 Jul 2023 01:56:59 PM UTC. 2023-07-02T13:58:54.366 INFO:teuthology.orchestra.run.smithi163.stdout:Installed products updated. 2023-07-02T13:58:54.498 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T13:58:54.498 INFO:teuthology.orchestra.run.smithi163.stdout:Upgraded: 2023-07-02T13:58:54.499 INFO:teuthology.orchestra.run.smithi163.stdout: librados2-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.499 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.499 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2023-07-02T13:58:54.499 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.499 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.499 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.499 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.500 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T13:58:54.500 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.500 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.500 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.500 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T13:58:54.500 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T13:58:54.500 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T13:58:54.501 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T13:58:54.501 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T13:58:54.501 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.501 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.501 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T13:58:54.502 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.502 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.502 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.503 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T13:58:54.503 INFO:teuthology.orchestra.run.smithi163.stdout: fuse-2.9.7-12.el8.x86_64 2023-07-02T13:58:54.503 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-02T13:58:54.504 INFO:teuthology.orchestra.run.smithi163.stdout: jq-1.5-12.el8.x86_64 2023-07-02T13:58:54.504 INFO:teuthology.orchestra.run.smithi163.stdout: leveldb-1.22-1.el8.x86_64 2023-07-02T13:58:54.504 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.505 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.505 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.505 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-07-02T13:58:54.505 INFO:teuthology.orchestra.run.smithi163.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-02T13:58:54.506 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-07-02T13:58:54.506 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-02T13:58:54.506 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.506 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.506 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-07-02T13:58:54.507 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.507 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-02T13:58:54.507 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-02T13:58:54.507 INFO:teuthology.orchestra.run.smithi163.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-02T13:58:54.507 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-0.3.12-1.el8.x86_64 2023-07-02T13:58:54.507 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-07-02T13:58:54.507 INFO:teuthology.orchestra.run.smithi163.stdout: pv-1.6.6-7.el8.x86_64 2023-07-02T13:58:54.508 INFO:teuthology.orchestra.run.smithi163.stdout: python3-apipkg-1.5-6.el8.noarch 2023-07-02T13:58:54.508 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel-2.5.1-5.el8.noarch 2023-07-02T13:58:54.508 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-02T13:58:54.508 INFO:teuthology.orchestra.run.smithi163.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-02T13:58:54.508 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-02T13:58:54.508 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.509 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.509 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.509 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-02T13:58:54.509 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-02T13:58:54.509 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-02T13:58:54.509 INFO:teuthology.orchestra.run.smithi163.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-07-02T13:58:54.509 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-02T13:58:54.509 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-02T13:58:54.510 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-02T13:58:54.510 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-07-02T13:58:54.510 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-02T13:58:54.510 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-02T13:58:54.510 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-02T13:58:54.510 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-02T13:58:54.510 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-02T13:58:54.510 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-02T13:58:54.511 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-02T13:58:54.511 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-02T13:58:54.511 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-07-02T13:58:54.511 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-07-02T13:58:54.511 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-02T13:58:54.511 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-02T13:58:54.511 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend-2.6-1.el8.noarch 2023-07-02T13:58:54.512 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-02T13:58:54.512 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-02T13:58:54.512 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-02T13:58:54.512 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-02T13:58:54.512 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.512 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.512 INFO:teuthology.orchestra.run.smithi163.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-07-02T13:58:54.513 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-02T13:58:54.513 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-02T13:58:54.513 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.513 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-02T13:58:54.513 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-02T13:58:54.513 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-07-02T13:58:54.513 INFO:teuthology.orchestra.run.smithi163.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-02T13:58:54.514 INFO:teuthology.orchestra.run.smithi163.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-02T13:58:54.514 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-02T13:58:54.514 INFO:teuthology.orchestra.run.smithi163.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-02T13:58:54.514 INFO:teuthology.orchestra.run.smithi163.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-02T13:58:54.514 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-02T13:58:54.514 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-02T13:58:54.514 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-02T13:58:54.514 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-02T13:58:54.515 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-02T13:58:54.515 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.515 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.515 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:54.515 INFO:teuthology.orchestra.run.smithi163.stdout: socat-1.7.3.3-2.el8.x86_64 2023-07-02T13:58:54.515 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-02T13:58:54.515 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T13:58:54.515 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2023-07-02T13:58:54.695 DEBUG:teuthology.parallel:result is None 2023-07-02T13:58:56.254 INFO:teuthology.orchestra.run.smithi016.stdout:Installed products updated. 2023-07-02T13:58:56.397 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T13:58:56.397 INFO:teuthology.orchestra.run.smithi016.stdout:Upgraded: 2023-07-02T13:58:56.397 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.397 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.398 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2023-07-02T13:58:56.398 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.398 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.398 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.398 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.398 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T13:58:56.398 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T13:58:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T13:58:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T13:58:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T13:58:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T13:58:56.400 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.400 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.400 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T13:58:56.400 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.400 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.400 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.400 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T13:58:56.401 INFO:teuthology.orchestra.run.smithi016.stdout: fuse-2.9.7-12.el8.x86_64 2023-07-02T13:58:56.401 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-02T13:58:56.401 INFO:teuthology.orchestra.run.smithi016.stdout: jq-1.5-12.el8.x86_64 2023-07-02T13:58:56.401 INFO:teuthology.orchestra.run.smithi016.stdout: leveldb-1.22-1.el8.x86_64 2023-07-02T13:58:56.401 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.401 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.401 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.401 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-07-02T13:58:56.402 INFO:teuthology.orchestra.run.smithi016.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-02T13:58:56.402 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-07-02T13:58:56.402 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-02T13:58:56.402 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.402 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.402 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-07-02T13:58:56.402 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.403 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-02T13:58:56.403 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-02T13:58:56.403 INFO:teuthology.orchestra.run.smithi016.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-02T13:58:56.403 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-0.3.12-1.el8.x86_64 2023-07-02T13:58:56.403 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-07-02T13:58:56.403 INFO:teuthology.orchestra.run.smithi016.stdout: pv-1.6.6-7.el8.x86_64 2023-07-02T13:58:56.403 INFO:teuthology.orchestra.run.smithi016.stdout: python3-apipkg-1.5-6.el8.noarch 2023-07-02T13:58:56.403 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel-2.5.1-5.el8.noarch 2023-07-02T13:58:56.404 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-02T13:58:56.404 INFO:teuthology.orchestra.run.smithi016.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-02T13:58:56.404 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-02T13:58:56.404 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.404 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.404 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.404 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-02T13:58:56.405 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-02T13:58:56.405 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-02T13:58:56.405 INFO:teuthology.orchestra.run.smithi016.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-07-02T13:58:56.405 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-02T13:58:56.405 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-02T13:58:56.405 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-02T13:58:56.405 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-07-02T13:58:56.406 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-02T13:58:56.406 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-02T13:58:56.406 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-02T13:58:56.406 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-02T13:58:56.406 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-02T13:58:56.406 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-02T13:58:56.406 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-02T13:58:56.406 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-02T13:58:56.407 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-07-02T13:58:56.407 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-07-02T13:58:56.407 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-02T13:58:56.407 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-02T13:58:56.407 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend-2.6-1.el8.noarch 2023-07-02T13:58:56.407 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-02T13:58:56.407 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-02T13:58:56.408 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-02T13:58:56.408 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-02T13:58:56.408 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.408 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.408 INFO:teuthology.orchestra.run.smithi016.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-07-02T13:58:56.408 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-02T13:58:56.408 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-02T13:58:56.409 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.409 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-02T13:58:56.409 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-02T13:58:56.409 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-07-02T13:58:56.409 INFO:teuthology.orchestra.run.smithi016.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-02T13:58:56.409 INFO:teuthology.orchestra.run.smithi016.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-02T13:58:56.409 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-02T13:58:56.410 INFO:teuthology.orchestra.run.smithi016.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-02T13:58:56.410 INFO:teuthology.orchestra.run.smithi016.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-02T13:58:56.410 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-02T13:58:56.410 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-02T13:58:56.410 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-02T13:58:56.410 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-02T13:58:56.410 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-02T13:58:56.410 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T13:58:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: socat-1.7.3.3-2.el8.x86_64 2023-07-02T13:58:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-02T13:58:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T13:58:56.411 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2023-07-02T13:58:56.580 DEBUG:teuthology.parallel:result is None 2023-07-02T13:59:45.825 INFO:teuthology.orchestra.run.smithi022.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-07-02T13:59:45.826 INFO:teuthology.orchestra.run.smithi022.stdout:[MIRROR] leveldb-1.22-1.el8.x86_64.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/l/leveldb-1.22-1.el8.x86_64.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-07-02T13:59:45.826 INFO:teuthology.orchestra.run.smithi022.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-07-02T13:59:45.925 INFO:teuthology.orchestra.run.smithi022.stdout:(36/100): liboath-2.6.2-3.el8.x86_64.rpm 464 B/s | 59 kB 02:09 2023-07-02T13:59:45.968 INFO:teuthology.orchestra.run.smithi022.stdout:(37/100): libunwind-1.3.1-3.el8.x86_64.rpm 1.7 MB/s | 75 kB 00:00 2023-07-02T13:59:45.983 INFO:teuthology.orchestra.run.smithi022.stdout:(38/100): leveldb-1.22-1.el8.x86_64.rpm 1.4 kB/s | 181 kB 02:09 2023-07-02T13:59:46.005 INFO:teuthology.orchestra.run.smithi022.stdout:(39/100): python3-apipkg-1.5-6.el8.noarch.rpm 1.0 MB/s | 21 kB 00:00 2023-07-02T13:59:46.008 INFO:teuthology.orchestra.run.smithi022.stdout:(40/100): pv-1.6.6-7.el8.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2023-07-02T13:59:46.025 INFO:teuthology.orchestra.run.smithi022.stdout:(41/100): gperftools-libs-2.7-9.el8.x86_64.rpm 2.4 kB/s | 306 kB 02:09 2023-07-02T13:59:46.026 INFO:teuthology.orchestra.run.smithi022.stdout:(42/100): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 2.0 MB/s | 44 kB 00:00 2023-07-02T13:59:46.046 INFO:teuthology.orchestra.run.smithi022.stdout:(43/100): python3-cachetools-3.1.1-4.el8.noarch 1.5 MB/s | 33 kB 00:00 2023-07-02T13:59:46.047 INFO:teuthology.orchestra.run.smithi022.stdout:(44/100): python3-certifi-2018.10.15-7.el8.noar 795 kB/s | 16 kB 00:00 2023-07-02T13:59:46.090 INFO:teuthology.orchestra.run.smithi022.stdout:(45/100): python3-beautifulsoup4-4.6.3-2.el8.1. 2.2 MB/s | 185 kB 00:00 2023-07-02T13:59:46.109 INFO:teuthology.orchestra.run.smithi022.stdout:(46/100): python3-cheroot-8.5.2-1.el8.noarch.rp 2.7 MB/s | 173 kB 00:00 2023-07-02T13:59:46.151 INFO:teuthology.orchestra.run.smithi022.stdout:(47/100): python3-google-auth-1.1.1-10.el8.noar 1.9 MB/s | 81 kB 00:00 2023-07-02T13:59:46.172 INFO:teuthology.orchestra.run.smithi022.stdout:(48/100): python3-jaraco-6.2-6.el8.noarch.rpm 548 kB/s | 11 kB 00:00 2023-07-02T13:59:46.192 INFO:teuthology.orchestra.run.smithi022.stdout:(49/100): python3-cherrypy-18.4.0-1.el8.noarch. 2.6 MB/s | 384 kB 00:00 2023-07-02T13:59:46.192 INFO:teuthology.orchestra.run.smithi022.stdout:(50/100): python3-jaraco-functools-2.0-4.el8.no 844 kB/s | 18 kB 00:00 2023-07-02T13:59:46.213 INFO:teuthology.orchestra.run.smithi022.stdout:(51/100): python3-execnet-1.7.1-1.el8.noarch.rp 2.8 MB/s | 355 kB 00:00 2023-07-02T13:59:46.214 INFO:teuthology.orchestra.run.smithi022.stdout:(52/100): python3-logutils-0.3.5-11.el8.noarch. 2.2 MB/s | 49 kB 00:00 2023-07-02T13:59:46.236 INFO:teuthology.orchestra.run.smithi022.stdout:(53/100): python3-more-itertools-7.2.0-3.el8.no 2.7 MB/s | 59 kB 00:00 2023-07-02T13:59:46.256 INFO:teuthology.orchestra.run.smithi022.stdout:(54/100): python3-portend-2.6-1.el8.noarch.rpm 764 kB/s | 16 kB 00:00 2023-07-02T13:59:46.278 INFO:teuthology.orchestra.run.smithi022.stdout:(55/100): python3-remoto-1.2.1-1.el8.noarch.rpm 2.2 MB/s | 49 kB 00:00 2023-07-02T13:59:46.297 INFO:teuthology.orchestra.run.smithi022.stdout:(56/100): python3-pecan-1.3.2-9.el8.noarch.rpm 3.3 MB/s | 283 kB 00:00 2023-07-02T13:59:46.299 INFO:teuthology.orchestra.run.smithi022.stdout:(57/100): python3-repoze-lru-0.7-6.el8.noarch.r 1.6 MB/s | 34 kB 00:00 2023-07-02T13:59:46.321 INFO:teuthology.orchestra.run.smithi022.stdout:(58/100): python3-rsa-4.8-1.el8.noarch.rpm 2.8 MB/s | 61 kB 00:00 2023-07-02T13:59:46.342 INFO:teuthology.orchestra.run.smithi022.stdout:(59/100): python3-simplegeneric-0.8.1-17.el8.no 891 kB/s | 19 kB 00:00 2023-07-02T13:59:46.358 INFO:teuthology.orchestra.run.smithi022.stdout:(60/100): python3-routes-2.4.1-12.el8.noarch.rp 3.2 MB/s | 196 kB 00:00 2023-07-02T13:59:46.363 INFO:teuthology.orchestra.run.smithi022.stdout:(61/100): python3-singledispatch-3.4.0.3-18.el8 1.1 MB/s | 24 kB 00:00 2023-07-02T13:59:46.380 INFO:teuthology.orchestra.run.smithi022.stdout:(62/100): python3-tempora-1.14.1-5.el8.noarch.r 1.3 MB/s | 29 kB 00:00 2023-07-02T13:59:46.384 INFO:teuthology.orchestra.run.smithi022.stdout:(63/100): python3-trustme-0.6.0-4.el8.noarch.rp 1.3 MB/s | 27 kB 00:00 2023-07-02T13:59:46.443 INFO:teuthology.orchestra.run.smithi022.stdout:(64/100): python3-waitress-1.4.3-1.el8.noarch.r 4.0 MB/s | 255 kB 00:00 2023-07-02T13:59:46.457 INFO:teuthology.orchestra.run.smithi022.stdout:(65/100): python3-kubernetes-11.0.0-6.el8.noarc 4.4 MB/s | 1.2 MB 00:00 2023-07-02T13:59:46.465 INFO:teuthology.orchestra.run.smithi022.stdout:(66/100): python3-websocket-client-0.56.0-5.el8 2.8 MB/s | 61 kB 00:00 2023-07-02T13:59:46.479 INFO:teuthology.orchestra.run.smithi022.stdout:(67/100): python3-webtest-2.0.33-1.el8.noarch.r 3.9 MB/s | 85 kB 00:00 2023-07-02T13:59:46.484 INFO:teuthology.orchestra.run.smithi022.stdout:(68/100): xmlstarlet-1.6.1-20.el8.x86_64.rpm 15 MB/s | 70 kB 00:00 2023-07-02T13:59:46.485 INFO:teuthology.orchestra.run.smithi022.stdout:(69/100): python3-webob-1.8.5-1.el8.1.noarch.rp 2.4 MB/s | 251 kB 00:00 2023-07-02T13:59:46.487 INFO:teuthology.orchestra.run.smithi022.stdout:(70/100): python3-zc-lockfile-2.0-2.el8.noarch. 1.0 MB/s | 23 kB 00:00 2023-07-02T13:59:46.520 INFO:teuthology.orchestra.run.smithi022.stdout:(71/100): python3-jwt-1.6.1-2.el8.noarch.rpm 1.2 MB/s | 43 kB 00:00 2023-07-02T13:59:46.530 INFO:teuthology.orchestra.run.smithi022.stdout:(72/100): python3-oauthlib-2.1.0-1.el8.noarch.r 3.4 MB/s | 155 kB 00:00 2023-07-02T13:59:46.535 INFO:teuthology.orchestra.run.smithi022.stdout:(73/100): python3-requests-oauthlib-1.0.0-1.el8 890 kB/s | 43 kB 00:00 2023-07-02T13:59:46.538 INFO:teuthology.orchestra.run.smithi022.stdout:(74/100): fuse-2.9.7-12.el8.x86_64.rpm 4.4 MB/s | 83 kB 00:00 2023-07-02T13:59:46.551 INFO:teuthology.orchestra.run.smithi022.stdout:(75/100): librabbitmq-0.9.0-3.el8.x86_64.rpm 3.0 MB/s | 47 kB 00:00 2023-07-02T13:59:46.552 INFO:teuthology.orchestra.run.smithi022.stdout:(76/100): libquadmath-8.4.1-1.el8.x86_64.rpm 7.6 MB/s | 169 kB 00:00 2023-07-02T13:59:46.570 INFO:teuthology.orchestra.run.smithi022.stdout:(77/100): libgfortran-8.4.1-1.el8.x86_64.rpm 20 MB/s | 639 kB 00:00 2023-07-02T13:59:46.571 INFO:teuthology.orchestra.run.smithi022.stdout:(78/100): python3-prettytable-0.7.2-14.el8.noar 2.3 MB/s | 44 kB 00:00 2023-07-02T13:59:46.585 INFO:teuthology.orchestra.run.smithi022.stdout:(79/100): python3-jsonpatch-1.21-2.el8.noarch.r 1.8 MB/s | 27 kB 00:00 2023-07-02T13:59:46.589 INFO:teuthology.orchestra.run.smithi022.stdout:(80/100): jq-1.5-12.el8.x86_64.rpm 9.1 MB/s | 161 kB 00:00 2023-07-02T13:59:46.608 INFO:teuthology.orchestra.run.smithi022.stdout:(81/100): python3-werkzeug-0.12.2-4.el8.noarch. 19 MB/s | 457 kB 00:00 2023-07-02T13:59:46.623 INFO:teuthology.orchestra.run.smithi022.stdout:(82/100): python3-markupsafe-0.23-19.el8.x86_64 2.6 MB/s | 39 kB 00:00 2023-07-02T13:59:46.642 INFO:teuthology.orchestra.run.smithi022.stdout:(83/100): lttng-ust-2.8.1-11.el8.x86_64.rpm 14 MB/s | 259 kB 00:00 2023-07-02T13:59:46.660 INFO:teuthology.orchestra.run.smithi022.stdout:(84/100): python3-numpy-f2py-1.14.3-9.el8.x86_6 13 MB/s | 225 kB 00:00 2023-07-02T13:59:46.674 INFO:teuthology.orchestra.run.smithi022.stdout:(85/100): python3-pyasn1-0.3.7-6.el8.noarch.rpm 8.5 MB/s | 126 kB 00:00 2023-07-02T13:59:46.699 INFO:teuthology.orchestra.run.smithi022.stdout:(86/100): python3-babel-2.5.1-5.el8.noarch.rpm 44 MB/s | 4.8 MB 00:00 2023-07-02T13:59:46.715 INFO:teuthology.orchestra.run.smithi022.stdout:(87/100): oniguruma-6.8.2-2.el8.x86_64.rpm 11 MB/s | 187 kB 00:00 2023-07-02T13:59:46.738 INFO:teuthology.orchestra.run.smithi022.stdout:(88/100): python3-jinja2-2.10.1-2.el8_0.noarch. 23 MB/s | 538 kB 00:00 2023-07-02T13:59:46.756 INFO:teuthology.orchestra.run.smithi022.stdout:(89/100): socat-1.7.3.3-2.el8.x86_64.rpm 17 MB/s | 302 kB 00:00 2023-07-02T13:59:46.771 INFO:teuthology.orchestra.run.smithi022.stdout:(90/100): python3-numpy-1.14.3-9.el8.x86_64.rpm 39 MB/s | 3.7 MB 00:00 2023-07-02T13:59:46.774 INFO:teuthology.orchestra.run.smithi022.stdout:(91/100): python3-mako-1.0.6-13.el8.noarch.rpm 8.3 MB/s | 157 kB 00:00 2023-07-02T13:59:46.785 INFO:teuthology.orchestra.run.smithi022.stdout:(92/100): python3-pyasn1-modules-0.3.7-6.el8.no 7.6 MB/s | 110 kB 00:00 2023-07-02T13:59:46.793 INFO:teuthology.orchestra.run.smithi022.stdout:(93/100): librdkafka-0.11.4-1.el8.x86_64.rpm 19 MB/s | 353 kB 00:00 2023-07-02T13:59:46.799 INFO:teuthology.orchestra.run.smithi022.stdout:(94/100): python3-jsonpointer-1.10-11.el8.noarc 1.5 MB/s | 20 kB 00:00 2023-07-02T13:59:46.807 INFO:teuthology.orchestra.run.smithi022.stdout:(95/100): python3-pytz-2017.2-9.el8.noarch.rpm 3.7 MB/s | 54 kB 00:00 2023-07-02T13:59:46.922 INFO:teuthology.orchestra.run.smithi022.stdout:(96/100): python3-scipy-1.0.0-20.module+el8.1.0 39 MB/s | 15 MB 00:00 2023-07-02T13:59:46.995 INFO:teuthology.orchestra.run.smithi022.stdout:(97/100): openblas-threads-0.3.12-1.el8.x86_64. 24 MB/s | 4.7 MB 00:00 2023-07-02T13:59:47.011 INFO:teuthology.orchestra.run.smithi022.stdout:(98/100): openblas-0.3.12-1.el8.x86_64.rpm 23 MB/s | 4.6 MB 00:00 2023-07-02T13:59:47.467 INFO:teuthology.orchestra.run.smithi022.stdout:(99/100): librados2-16.2.13-280.g8934282d.el8.x 6.9 MB/s | 3.7 MB 00:00 2023-07-02T13:59:47.537 INFO:teuthology.orchestra.run.smithi022.stdout:(100/100): librbd1-16.2.13-280.g8934282d.el8.x8 7.7 MB/s | 4.2 MB 00:00 2023-07-02T13:59:47.540 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2023-07-02T13:59:47.541 INFO:teuthology.orchestra.run.smithi022.stdout:Total 1.2 MB/s | 193 MB 02:38 2023-07-02T13:59:47.572 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2023-07-02T13:59:47.671 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2023-07-02T13:59:47.671 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2023-07-02T13:59:49.188 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2023-07-02T13:59:49.192 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2023-07-02T13:59:53.210 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2023-07-02T13:59:53.403 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/102 2023-07-02T13:59:53.608 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : leveldb-1.22-1.el8.x86_64 2/102 2023-07-02T13:59:53.786 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 3/102 2023-07-02T13:59:53.923 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 3/102 2023-07-02T13:59:54.119 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/102 2023-07-02T13:59:54.561 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/102 2023-07-02T13:59:54.819 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : librados2-2:16.2.13-280.g8934282d.el8.x86_64 5/102 2023-07-02T13:59:55.240 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados2-2:16.2.13-280.g8934282d.el8.x86_64 5/102 2023-07-02T13:59:55.444 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : librbd1-2:16.2.13-280.g8934282d.el8.x86_64 6/102 2023-07-02T13:59:55.558 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librbd1-2:16.2.13-280.g8934282d.el8.x86_64 6/102 2023-07-02T13:59:55.735 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephfs2-2:16.2.13-280.g8934282d.el8.x86_64 7/102 2023-07-02T13:59:55.825 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephfs2-2:16.2.13-280.g8934282d.el8.x86_64 7/102 2023-07-02T13:59:56.033 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rados-2:16.2.13-280.g8934282d.el8.x86_64 8/102 2023-07-02T13:59:56.534 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/102 2023-07-02T13:59:56.702 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librgw2-2:16.2.13-280.g8934282d.el8.x86_64 10/102 2023-07-02T13:59:56.820 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librgw2-2:16.2.13-280.g8934282d.el8.x86_64 10/102 2023-07-02T13:59:57.048 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/102 2023-07-02T13:59:57.235 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/102 2023-07-02T13:59:57.444 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephsqlite-2:16.2.13-280.g8934282d.el8.x86_64 13/102 2023-07-02T13:59:57.548 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephsqlite-2:16.2.13-280.g8934282d.el8.x86_64 13/102 2023-07-02T13:59:57.719 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libradosstriper1-2:16.2.13-280.g8934282d.el8.x86 14/102 2023-07-02T13:59:57.787 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libradosstriper1-2:16.2.13-280.g8934282d.el8.x86 14/102 2023-07-02T13:59:57.957 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 15/102 2023-07-02T13:59:58.187 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 16/102 2023-07-02T13:59:58.439 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 17/102 2023-07-02T13:59:58.577 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 18/102 2023-07-02T13:59:58.716 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 18/102 2023-07-02T13:59:58.910 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 19/102 2023-07-02T13:59:59.478 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 19/102 2023-07-02T13:59:59.635 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 20/102 2023-07-02T14:00:00.184 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 20/102 2023-07-02T14:00:00.368 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : openblas-0.3.12-1.el8.x86_64 21/102 2023-07-02T14:00:01.101 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 21/102 2023-07-02T14:00:01.364 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 22/102 2023-07-02T14:00:01.589 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 23/102 2023-07-02T14:00:01.801 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 24/102 2023-07-02T14:00:01.991 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ceph-common-2:16.2.13-280.g8934282d.el8. 25/102 2023-07-02T14:00:02.142 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ceph-argparse-2:16.2.13-280.g8934282d.el 26/102 2023-07-02T14:00:02.389 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cephfs-2:16.2.13-280.g8934282d.el8.x86_6 27/102 2023-07-02T14:00:02.650 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 28/102 2023-07-02T14:00:02.939 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 29/102 2023-07-02T14:00:03.176 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 30/102 2023-07-02T14:00:05.334 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 30/102 2023-07-02T14:00:05.694 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 31/102 2023-07-02T14:00:06.709 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 32/102 2023-07-02T14:00:07.187 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 33/102 2023-07-02T14:00:07.465 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 34/102 2023-07-02T14:00:07.676 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rsa-4.8-1.el8.noarch 35/102 2023-07-02T14:00:07.897 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 36/102 2023-07-02T14:00:08.096 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rgw-2:16.2.13-280.g8934282d.el8.x86_64 37/102 2023-07-02T14:00:08.256 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rbd-2:16.2.13-280.g8934282d.el8.x86_64 38/102 2023-07-02T14:00:08.475 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librados-devel-2:16.2.13-280.g8934282d.el8.x86_6 39/102 2023-07-02T14:00:08.668 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 40/102 2023-07-02T14:00:08.854 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 41/102 2023-07-02T14:00:09.065 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 42/102 2023-07-02T14:00:09.217 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/102 2023-07-02T14:00:09.285 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/102 2023-07-02T14:00:09.445 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : jq-1.5-12.el8.x86_64 44/102 2023-07-02T14:00:09.629 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 45/102 2023-07-02T14:00:09.771 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : fuse-2.9.7-12.el8.x86_64 46/102 2023-07-02T14:00:09.912 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 47/102 2023-07-02T14:00:10.079 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 48/102 2023-07-02T14:00:10.336 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 49/102 2023-07-02T14:00:10.602 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 50/102 2023-07-02T14:00:10.755 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/102 2023-07-02T14:00:10.948 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 52/102 2023-07-02T14:00:11.097 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 53/102 2023-07-02T14:00:11.283 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 54/102 2023-07-02T14:00:11.630 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 55/102 2023-07-02T14:00:11.946 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 56/102 2023-07-02T14:00:12.149 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 57/102 2023-07-02T14:00:12.383 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 58/102 2023-07-02T14:00:12.581 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 59/102 2023-07-02T14:00:12.807 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 60/102 2023-07-02T14:00:13.087 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-portend-2.6-1.el8.noarch 61/102 2023-07-02T14:00:13.325 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 62/102 2023-07-02T14:00:13.503 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 63/102 2023-07-02T14:00:13.709 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 64/102 2023-07-02T14:00:14.936 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 65/102 2023-07-02T14:00:15.249 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 66/102 2023-07-02T14:00:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 67/102 2023-07-02T14:00:15.802 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 68/102 2023-07-02T14:00:16.042 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 69/102 2023-07-02T14:00:16.248 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 70/102 2023-07-02T14:00:16.495 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 71/102 2023-07-02T14:00:16.766 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 72/102 2023-07-02T14:00:16.959 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 73/102 2023-07-02T14:00:17.144 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 74/102 2023-07-02T14:00:17.374 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 75/102 2023-07-02T14:00:22.484 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-common-2:16.2.13-280.g8934282d.el8.x86_64 76/102 2023-07-02T14:00:22.723 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-common-2:16.2.13-280.g8934282d.el8.x86_64 76/102 2023-07-02T14:00:23.463 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-common-2:16.2.13-280.g8934282d.el8.x86_64 76/102 2023-07-02T14:00:23.699 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-base-2:16.2.13-280.g8934282d.el8.x86_64 77/102 2023-07-02T14:00:23.802 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-base-2:16.2.13-280.g8934282d.el8.x86_64 77/102 2023-07-02T14:00:23.931 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-selinux-2:16.2.13-280.g8934282d.el8.x86_64 78/102 2023-07-02T14:00:42.360 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-selinux-2:16.2.13-280.g8934282d.el8.x86_64 78/102 2023-07-02T14:00:42.361 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /sys 2023-07-02T14:00:42.361 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /proc 2023-07-02T14:00:42.361 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /dev 2023-07-02T14:00:42.361 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /run 2023-07-02T14:00:42.361 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /mnt 2023-07-02T14:00:42.361 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /var/tmp 2023-07-02T14:00:42.362 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /home 2023-07-02T14:00:42.362 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /tmp 2023-07-02T14:00:42.362 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /dev 2023-07-02T14:00:42.362 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:00:42.761 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mds-2:16.2.13-280.g8934282d.el8.x86_64 79/102 2023-07-02T14:00:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mds-2:16.2.13-280.g8934282d.el8.x86_64 79/102 2023-07-02T14:00:43.403 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mon-2:16.2.13-280.g8934282d.el8.x86_64 80/102 2023-07-02T14:00:44.986 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mon-2:16.2.13-280.g8934282d.el8.x86_64 80/102 2023-07-02T14:00:45.186 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-osd-2:16.2.13-280.g8934282d.el8.x86_64 81/102 2023-07-02T14:00:45.208 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-osd-2:16.2.13-280.g8934282d.el8.x86_64 81/102 2023-07-02T14:00:48.215 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: cephadm-2:16.2.13-280.g8934282d.el8.noarch 82/102 2023-07-02T14:00:48.347 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : cephadm-2:16.2.13-280.g8934282d.el8.noarch 82/102 2023-07-02T14:00:48.526 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-prometheus-alerts-2:16.2.13-280.g8934282d.e 83/102 2023-07-02T14:00:48.697 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-grafana-dashboards-2:16.2.13-280.g8934282d. 84/102 2023-07-02T14:00:48.885 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-cephadm-2:16.2.13-280.g8934282d.el8.noa 85/102 2023-07-02T14:00:49.897 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.13-280.g8934282d.el8.noa 85/102 2023-07-02T14:00:50.152 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-dashboard-2:16.2.13-280.g8934282d.el8.n 86/102 2023-07-02T14:00:51.263 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.13-280.g8934282d.el8.n 86/102 2023-07-02T14:00:51.418 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.13-280.g893 87/102 2023-07-02T14:00:51.516 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.13-280.g893 87/102 2023-07-02T14:00:51.931 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-modules-core-2:16.2.13-280.g8934282d.el 88/102 2023-07-02T14:00:52.094 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-2:16.2.13-280.g8934282d.el8.x86_64 89/102 2023-07-02T14:00:52.121 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-2:16.2.13-280.g8934282d.el8.x86_64 89/102 2023-07-02T14:00:52.251 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-rook-2:16.2.13-280.g8934282d.el8.noarch 90/102 2023-07-02T14:00:52.262 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.13-280.g8934282d.el8.noarch 90/102 2023-07-02T14:00:53.398 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-2:16.2.13-280.g8934282d.el8.x86_64 91/102 2023-07-02T14:00:53.610 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-radosgw-2:16.2.13-280.g8934282d.el8.x86_64 92/102 2023-07-02T14:00:53.684 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-radosgw-2:16.2.13-280.g8934282d.el8.x86_64 92/102 2023-07-02T14:00:53.859 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-immutable-object-cache-2:16.2.13-280.g89342 93/102 2023-07-02T14:00:54.227 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.13-280.g89342 93/102 2023-07-02T14:00:54.368 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-mirror-2:16.2.13-280.g8934282d.el8.x86_64 94/102 2023-07-02T14:00:59.191 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-mirror-2:16.2.13-280.g8934282d.el8.x86_64 94/102 2023-07-02T14:00:59.529 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-test-2:16.2.13-280.g8934282d.el8.x86_64 95/102 2023-07-02T14:00:59.671 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-fuse-2:16.2.13-280.g8934282d.el8.x86_64 96/102 2023-07-02T14:00:59.808 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephfs-devel-2:16.2.13-280.g8934282d.el8.x86_ 97/102 2023-07-02T14:00:59.958 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-fuse-2:16.2.13-280.g8934282d.el8.x86_64 98/102 2023-07-02T14:01:00.123 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-nbd-2:16.2.13-280.g8934282d.el8.x86_64 99/102 2023-07-02T14:01:00.286 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : pv-1.6.6-7.el8.x86_64 100/102 2023-07-02T14:01:00.286 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 101/102 2023-07-02T14:01:00.478 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 101/102 2023-07-02T14:01:00.479 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 102/102 2023-07-02T14:01:01.368 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 102/102 2023-07-02T14:01:01.369 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-2:16.2.13-280.g8934282d.el8.x86_64 1/102 2023-07-02T14:01:01.369 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-base-2:16.2.13-280.g8934282d.el8.x86_64 2/102 2023-07-02T14:01:01.369 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-common-2:16.2.13-280.g8934282d.el8.x86_64 3/102 2023-07-02T14:01:01.369 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-fuse-2:16.2.13-280.g8934282d.el8.x86_64 4/102 2023-07-02T14:01:01.369 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-immutable-object-cache-2:16.2.13-280.g89342 5/102 2023-07-02T14:01:01.369 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mds-2:16.2.13-280.g8934282d.el8.x86_64 6/102 2023-07-02T14:01:01.369 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-2:16.2.13-280.g8934282d.el8.x86_64 7/102 2023-07-02T14:01:01.370 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mon-2:16.2.13-280.g8934282d.el8.x86_64 8/102 2023-07-02T14:01:01.370 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-osd-2:16.2.13-280.g8934282d.el8.x86_64 9/102 2023-07-02T14:01:01.370 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-radosgw-2:16.2.13-280.g8934282d.el8.x86_64 10/102 2023-07-02T14:01:01.370 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-selinux-2:16.2.13-280.g8934282d.el8.x86_64 11/102 2023-07-02T14:01:01.370 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-test-2:16.2.13-280.g8934282d.el8.x86_64 12/102 2023-07-02T14:01:01.370 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephfs-devel-2:16.2.13-280.g8934282d.el8.x86_ 13/102 2023-07-02T14:01:01.370 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephfs2-2:16.2.13-280.g8934282d.el8.x86_64 14/102 2023-07-02T14:01:01.370 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephsqlite-2:16.2.13-280.g8934282d.el8.x86_64 15/102 2023-07-02T14:01:01.371 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados-devel-2:16.2.13-280.g8934282d.el8.x86_6 16/102 2023-07-02T14:01:01.371 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libradosstriper1-2:16.2.13-280.g8934282d.el8.x86 17/102 2023-07-02T14:01:01.371 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librgw2-2:16.2.13-280.g8934282d.el8.x86_64 18/102 2023-07-02T14:01:01.371 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ceph-argparse-2:16.2.13-280.g8934282d.el 19/102 2023-07-02T14:01:01.371 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ceph-common-2:16.2.13-280.g8934282d.el8. 20/102 2023-07-02T14:01:01.371 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cephfs-2:16.2.13-280.g8934282d.el8.x86_6 21/102 2023-07-02T14:01:01.371 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rados-2:16.2.13-280.g8934282d.el8.x86_64 22/102 2023-07-02T14:01:01.371 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rbd-2:16.2.13-280.g8934282d.el8.x86_64 23/102 2023-07-02T14:01:01.372 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rgw-2:16.2.13-280.g8934282d.el8.x86_64 24/102 2023-07-02T14:01:01.372 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-fuse-2:16.2.13-280.g8934282d.el8.x86_64 25/102 2023-07-02T14:01:01.372 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-mirror-2:16.2.13-280.g8934282d.el8.x86_64 26/102 2023-07-02T14:01:01.372 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-nbd-2:16.2.13-280.g8934282d.el8.x86_64 27/102 2023-07-02T14:01:01.372 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-grafana-dashboards-2:16.2.13-280.g8934282d. 28/102 2023-07-02T14:01:01.372 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-cephadm-2:16.2.13-280.g8934282d.el8.noa 29/102 2023-07-02T14:01:01.377 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-dashboard-2:16.2.13-280.g8934282d.el8.n 30/102 2023-07-02T14:01:01.377 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.13-280.g893 31/102 2023-07-02T14:01:01.377 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-modules-core-2:16.2.13-280.g8934282d.el 32/102 2023-07-02T14:01:01.377 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-rook-2:16.2.13-280.g8934282d.el8.noarch 33/102 2023-07-02T14:01:01.378 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-prometheus-alerts-2:16.2.13-280.g8934282d.e 34/102 2023-07-02T14:01:01.378 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : cephadm-2:16.2.13-280.g8934282d.el8.noarch 35/102 2023-07-02T14:01:01.378 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 36/102 2023-07-02T14:01:01.378 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : leveldb-1.22-1.el8.x86_64 37/102 2023-07-02T14:01:01.378 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 38/102 2023-07-02T14:01:01.378 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 39/102 2023-07-02T14:01:01.378 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : pv-1.6.6-7.el8.x86_64 40/102 2023-07-02T14:01:01.379 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 41/102 2023-07-02T14:01:01.379 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 42/102 2023-07-02T14:01:01.379 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 43/102 2023-07-02T14:01:01.379 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 44/102 2023-07-02T14:01:01.379 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 45/102 2023-07-02T14:01:01.379 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 46/102 2023-07-02T14:01:01.379 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 47/102 2023-07-02T14:01:01.380 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 48/102 2023-07-02T14:01:01.380 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/102 2023-07-02T14:01:01.380 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/102 2023-07-02T14:01:01.380 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/102 2023-07-02T14:01:01.380 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/102 2023-07-02T14:01:01.380 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/102 2023-07-02T14:01:01.380 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/102 2023-07-02T14:01:01.380 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 55/102 2023-07-02T14:01:01.381 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-portend-2.6-1.el8.noarch 56/102 2023-07-02T14:01:01.381 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 57/102 2023-07-02T14:01:01.381 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/102 2023-07-02T14:01:01.381 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/102 2023-07-02T14:01:01.381 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/102 2023-07-02T14:01:01.381 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/102 2023-07-02T14:01:01.381 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/102 2023-07-02T14:01:01.382 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/102 2023-07-02T14:01:01.382 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/102 2023-07-02T14:01:01.382 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/102 2023-07-02T14:01:01.382 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/102 2023-07-02T14:01:01.382 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/102 2023-07-02T14:01:01.382 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/102 2023-07-02T14:01:01.382 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/102 2023-07-02T14:01:01.383 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 70/102 2023-07-02T14:01:01.383 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 71/102 2023-07-02T14:01:01.383 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 72/102 2023-07-02T14:01:01.383 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 73/102 2023-07-02T14:01:01.383 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 74/102 2023-07-02T14:01:01.383 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 75/102 2023-07-02T14:01:01.383 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 76/102 2023-07-02T14:01:01.383 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 77/102 2023-07-02T14:01:01.384 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 78/102 2023-07-02T14:01:01.384 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 79/102 2023-07-02T14:01:01.384 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 80/102 2023-07-02T14:01:01.384 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : jq-1.5-12.el8.x86_64 81/102 2023-07-02T14:01:01.384 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 82/102 2023-07-02T14:01:01.384 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 83/102 2023-07-02T14:01:01.384 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 84/102 2023-07-02T14:01:01.385 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 85/102 2023-07-02T14:01:01.385 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 86/102 2023-07-02T14:01:01.385 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 87/102 2023-07-02T14:01:01.385 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 88/102 2023-07-02T14:01:01.385 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 89/102 2023-07-02T14:01:01.385 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 90/102 2023-07-02T14:01:01.385 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 91/102 2023-07-02T14:01:01.385 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 92/102 2023-07-02T14:01:01.386 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 93/102 2023-07-02T14:01:01.386 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 94/102 2023-07-02T14:01:01.386 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 95/102 2023-07-02T14:01:01.386 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 96/102 2023-07-02T14:01:01.386 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 97/102 2023-07-02T14:01:01.386 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 98/102 2023-07-02T14:01:01.386 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados2-2:16.2.13-280.g8934282d.el8.x86_64 99/102 2023-07-02T14:01:01.386 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 100/102 2023-07-02T14:01:01.387 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librbd1-2:16.2.13-280.g8934282d.el8.x86_64 101/102 2023-07-02T14:01:03.514 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 102/102 2023-07-02T14:01:03.515 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:04:03 ago on Sun 02 Jul 2023 01:57:00 PM UTC. 2023-07-02T14:01:05.469 INFO:teuthology.orchestra.run.smithi022.stdout:Installed products updated. 2023-07-02T14:01:05.597 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:05.598 INFO:teuthology.orchestra.run.smithi022.stdout:Upgraded: 2023-07-02T14:01:05.598 INFO:teuthology.orchestra.run.smithi022.stdout: librados2-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.598 INFO:teuthology.orchestra.run.smithi022.stdout: librbd1-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.598 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2023-07-02T14:01:05.598 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.598 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-base-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.599 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-common-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.599 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-fuse-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.599 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-grafana-dashboards-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T14:01:05.599 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-immutable-object-cache-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.599 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mds-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.599 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.600 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-cephadm-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T14:01:05.600 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-dashboard-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T14:01:05.600 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-diskprediction-local-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T14:01:05.600 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-modules-core-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T14:01:05.600 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-rook-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T14:01:05.600 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mon-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.600 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-osd-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.601 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-prometheus-alerts-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T14:01:05.601 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-radosgw-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.601 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-selinux-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.601 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-test-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.601 INFO:teuthology.orchestra.run.smithi022.stdout: cephadm-2:16.2.13-280.g8934282d.el8.noarch 2023-07-02T14:01:05.601 INFO:teuthology.orchestra.run.smithi022.stdout: fuse-2.9.7-12.el8.x86_64 2023-07-02T14:01:05.601 INFO:teuthology.orchestra.run.smithi022.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-02T14:01:05.601 INFO:teuthology.orchestra.run.smithi022.stdout: jq-1.5-12.el8.x86_64 2023-07-02T14:01:05.602 INFO:teuthology.orchestra.run.smithi022.stdout: leveldb-1.22-1.el8.x86_64 2023-07-02T14:01:05.602 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs-devel-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.602 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs2-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.602 INFO:teuthology.orchestra.run.smithi022.stdout: libcephsqlite-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.602 INFO:teuthology.orchestra.run.smithi022.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-07-02T14:01:05.602 INFO:teuthology.orchestra.run.smithi022.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-02T14:01:05.602 INFO:teuthology.orchestra.run.smithi022.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-07-02T14:01:05.603 INFO:teuthology.orchestra.run.smithi022.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-02T14:01:05.603 INFO:teuthology.orchestra.run.smithi022.stdout: librados-devel-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.603 INFO:teuthology.orchestra.run.smithi022.stdout: libradosstriper1-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.603 INFO:teuthology.orchestra.run.smithi022.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-07-02T14:01:05.603 INFO:teuthology.orchestra.run.smithi022.stdout: librgw2-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.603 INFO:teuthology.orchestra.run.smithi022.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-02T14:01:05.603 INFO:teuthology.orchestra.run.smithi022.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-02T14:01:05.603 INFO:teuthology.orchestra.run.smithi022.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-02T14:01:05.604 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-0.3.12-1.el8.x86_64 2023-07-02T14:01:05.604 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-07-02T14:01:05.604 INFO:teuthology.orchestra.run.smithi022.stdout: pv-1.6.6-7.el8.x86_64 2023-07-02T14:01:05.604 INFO:teuthology.orchestra.run.smithi022.stdout: python3-apipkg-1.5-6.el8.noarch 2023-07-02T14:01:05.604 INFO:teuthology.orchestra.run.smithi022.stdout: python3-babel-2.5.1-5.el8.noarch 2023-07-02T14:01:05.604 INFO:teuthology.orchestra.run.smithi022.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-02T14:01:05.604 INFO:teuthology.orchestra.run.smithi022.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-02T14:01:05.605 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-02T14:01:05.605 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-argparse-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.605 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-common-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.605 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cephfs-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.605 INFO:teuthology.orchestra.run.smithi022.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-02T14:01:05.605 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-02T14:01:05.605 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-02T14:01:05.605 INFO:teuthology.orchestra.run.smithi022.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-07-02T14:01:05.606 INFO:teuthology.orchestra.run.smithi022.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-02T14:01:05.606 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-02T14:01:05.606 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-02T14:01:05.606 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-07-02T14:01:05.606 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-02T14:01:05.606 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-02T14:01:05.606 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-02T14:01:05.607 INFO:teuthology.orchestra.run.smithi022.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-02T14:01:05.607 INFO:teuthology.orchestra.run.smithi022.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-02T14:01:05.607 INFO:teuthology.orchestra.run.smithi022.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-02T14:01:05.607 INFO:teuthology.orchestra.run.smithi022.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-02T14:01:05.607 INFO:teuthology.orchestra.run.smithi022.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-02T14:01:05.607 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-07-02T14:01:05.607 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-07-02T14:01:05.607 INFO:teuthology.orchestra.run.smithi022.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-02T14:01:05.608 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-02T14:01:05.608 INFO:teuthology.orchestra.run.smithi022.stdout: python3-portend-2.6-1.el8.noarch 2023-07-02T14:01:05.608 INFO:teuthology.orchestra.run.smithi022.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-02T14:01:05.608 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-02T14:01:05.608 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-02T14:01:05.608 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-02T14:01:05.608 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rados-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.609 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rbd-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.609 INFO:teuthology.orchestra.run.smithi022.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-07-02T14:01:05.609 INFO:teuthology.orchestra.run.smithi022.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-02T14:01:05.609 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-02T14:01:05.609 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rgw-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.609 INFO:teuthology.orchestra.run.smithi022.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-02T14:01:05.609 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-02T14:01:05.609 INFO:teuthology.orchestra.run.smithi022.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-07-02T14:01:05.610 INFO:teuthology.orchestra.run.smithi022.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-02T14:01:05.610 INFO:teuthology.orchestra.run.smithi022.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-02T14:01:05.610 INFO:teuthology.orchestra.run.smithi022.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-02T14:01:05.610 INFO:teuthology.orchestra.run.smithi022.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-02T14:01:05.610 INFO:teuthology.orchestra.run.smithi022.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-02T14:01:05.610 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-02T14:01:05.610 INFO:teuthology.orchestra.run.smithi022.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-02T14:01:05.611 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-02T14:01:05.611 INFO:teuthology.orchestra.run.smithi022.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-02T14:01:05.611 INFO:teuthology.orchestra.run.smithi022.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-02T14:01:05.611 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-fuse-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.611 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-mirror-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.611 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-nbd-2:16.2.13-280.g8934282d.el8.x86_64 2023-07-02T14:01:05.611 INFO:teuthology.orchestra.run.smithi022.stdout: socat-1.7.3.3-2.el8.x86_64 2023-07-02T14:01:05.612 INFO:teuthology.orchestra.run.smithi022.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-02T14:01:05.612 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:05.612 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-07-02T14:01:05.784 DEBUG:teuthology.parallel:result is None 2023-07-02T14:01:05.785 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8934282de75bfae9ed7c114e0bb6e46fc588d851 2023-07-02T14:01:05.927 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-02T14:01:05.954 INFO:teuthology.orchestra.run.smithi016.stdout:16.2.13-280.g8934282d.el8 2023-07-02T14:01:05.955 INFO:teuthology.packaging:The installed version of ceph is 16.2.13-280.g8934282d.el8 2023-07-02T14:01:05.955 INFO:teuthology.task.install:The correct ceph version 16.2.13-280.g8934282d is installed. 2023-07-02T14:01:05.956 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8934282de75bfae9ed7c114e0bb6e46fc588d851 2023-07-02T14:01:06.169 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-02T14:01:06.190 INFO:teuthology.orchestra.run.smithi022.stdout:16.2.13-280.g8934282d.el8 2023-07-02T14:01:06.191 INFO:teuthology.packaging:The installed version of ceph is 16.2.13-280.g8934282d.el8 2023-07-02T14:01:06.191 INFO:teuthology.task.install:The correct ceph version 16.2.13-280.g8934282d is installed. 2023-07-02T14:01:06.192 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8934282de75bfae9ed7c114e0bb6e46fc588d851 2023-07-02T14:01:06.401 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-02T14:01:06.427 INFO:teuthology.orchestra.run.smithi163.stdout:16.2.13-280.g8934282d.el8 2023-07-02T14:01:06.427 INFO:teuthology.packaging:The installed version of ceph is 16.2.13-280.g8934282d.el8 2023-07-02T14:01:06.428 INFO:teuthology.task.install:The correct ceph version 16.2.13-280.g8934282d is installed. 2023-07-02T14:01:06.428 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-07-02T14:01:06.429 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-07-02T14:01:06.429 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-02T14:01:06.475 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-07-02T14:01:06.476 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-02T14:01:06.516 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-07-02T14:01:06.517 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-02T14:01:06.560 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-07-02T14:01:06.561 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-07-02T14:01:06.561 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2023-07-02T14:01:06.591 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-02T14:01:06.660 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-07-02T14:01:06.660 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/daemon-helper 2023-07-02T14:01:06.686 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-02T14:01:06.753 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-07-02T14:01:06.754 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/daemon-helper 2023-07-02T14:01:06.785 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-02T14:01:06.853 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-07-02T14:01:06.853 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-07-02T14:01:06.853 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-02T14:01:06.881 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-02T14:01:06.949 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-07-02T14:01:06.949 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-02T14:01:06.974 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-02T14:01:07.041 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-07-02T14:01:07.041 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-02T14:01:07.067 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-02T14:01:07.134 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-07-02T14:01:07.135 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-07-02T14:01:07.136 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/stdin-killer 2023-07-02T14:01:07.164 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-02T14:01:07.231 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-07-02T14:01:07.232 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/stdin-killer 2023-07-02T14:01:07.258 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-02T14:01:07.324 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-07-02T14:01:07.324 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/stdin-killer 2023-07-02T14:01:07.351 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-02T14:01:07.417 INFO:teuthology.run_tasks:Running task ceph... 2023-07-02T14:01:07.489 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-07-02T14:01:07.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /var/log/ceph 2023-07-02T14:01:07.492 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 777 /var/log/ceph 2023-07-02T14:01:07.494 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 777 /var/log/ceph 2023-07-02T14:01:07.519 INFO:tasks.ceph:Disabling ceph logrotate... 2023-07-02T14:01:07.520 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/logrotate.d/ceph 2023-07-02T14:01:07.560 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /etc/logrotate.d/ceph 2023-07-02T14:01:07.563 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/logrotate.d/ceph 2023-07-02T14:01:07.587 INFO:tasks.ceph:Creating extra log directories... 2023-07-02T14:01:07.588 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-07-02T14:01:07.628 DEBUG:teuthology.orchestra.run.smithi022:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-07-02T14:01:07.631 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-07-02T14:01:07.664 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-07-02T14:01:07.665 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-07-02T14:01:07.665 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366', 'branch': 'pacific', 'description': 'rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7324366', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'machine': {'cpus': 1, 'disk': 40, 'ram': 8000}, 'volumes': {'count': 4, 'size': 30}}], 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'pacific'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request'], 'sha1': '8934282de75bfae9ed7c114e0bb6e46fc588d851'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'extra_system_packages': ['pv'], 'flavor': 'default', 'sha1': '8934282de75bfae9ed7c114e0bb6e46fc588d851'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-61616-pacific', 'sha1': 'cbd88633a9c19779ad4b2e367bdc17f83b3a065d'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 74, 'repo': 'https://github.com/ceph/ceph', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5'], ['client.0']], 'seed': 3615, 'sha1': '8934282de75bfae9ed7c114e0bb6e46fc588d851', 'sleep_before_teardown': 0, 'subset': '25/99', 'suite': 'rbd', 'suite_branch': 'wip-61616-pacific', 'suite_path': '/home/teuthworker/src/github.com_idryomov_ceph_cbd88633a9c19779ad4b2e367bdc17f83b3a065d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/idryomov/ceph', 'suite_sha1': 'cbd88633a9c19779ad4b2e367bdc17f83b3a065d', 'targets': {'smithi016.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKAKQdK43fJ67ZR0UFSSd/x9YyqkWqm7r0EvMSQqiplezjVhkRix4wwgl7eot1veKhyPZmpw4xJNdOF0LHYyJls=', 'smithi022.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAJsgII6kEeeCVaBI90i/iehAOPSQrDC/rPxYXSo44UgV74Crx/QB4JOdm21MI75Pv9b9R4Vt8I2AzdTkMoETQA=', 'smithi163.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlG8ZmXbNG1kKGtt6qrh3TXluF3kt7nELxpD4U3sZvPxNmfk+i05F0KvaqgWwitlcawW5gZzWAqd4jeTIW8MCTKW7jA7Ipp/X6WjdLPEzd3Gl4HBzLACoV13x2/ep16BUlgQgBoDBrf7ye4s5Ud/a2ARRZFB0s5q2Pa+oyjTJjv6SeNkn5rqA5zIwm9+g6bkl7MZtNxHBcm5SPIZpFLJ7P9edWZh5820pOd5LjZEEutwGxUxTUf6TipiRoZYW+oFLTGHrgIFbiUPsZDhhRj9/mJZmhF8JcvkRRTNUQl2Mgl7meT556UkqLDtwbkAaIrOA58XqnvIaTByEaC4Bw1CpuntlENFsJR40BhLj9GgOa7AmDOkurkVwG3yQldNdDQ8EVtX7xRlXDRqHKy6EYh5TMB/zYrNmFiSGFIEKhDzgaRQCAngZ72NVQ3EAlzdqBqQhZlkn3zR71CXe5rzHm4TaZCKIfcFc6UZ+bWSXYs6ob5sTB3YRD788HYsqsuCikcTE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rbd/diff_continuous.sh']}, 'env': {'RBD_DEVICE_TYPE': 'nbd'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b53f236c1077f1d295baafb9e0ca9a4b447019d7', 'timestamp': '2023-07-02_13:35:41', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.23459'} 2023-07-02T14:01:07.666 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-07-02T14:01:07.696 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-07-02T14:01:07.698 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-07-02T14:01:07.713 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/run/ceph 2023-07-02T14:01:07.752 DEBUG:teuthology.orchestra.run.smithi022:> sudo install -d -m0777 -- /var/run/ceph 2023-07-02T14:01:07.755 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m0777 -- /var/run/ceph 2023-07-02T14:01:07.780 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-07-02T14:01:07.781 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2023-07-02T14:01:07.834 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-02T14:01:07.834 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2023-07-02T14:01:07.890 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-02T14:01:07.890 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-02T14:01:07.890 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6h/6d Inode: 215361 Links: 1 2023-07-02T14:01:07.891 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-02T14:01:07.891 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2023-07-02T14:01:07.891 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2023-07-02 13:58:51.618610144 +0000 2023-07-02T14:01:07.891 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2023-07-02 13:56:07.898881104 +0000 2023-07-02T14:01:07.891 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2023-07-02 13:56:07.898881104 +0000 2023-07-02T14:01:07.891 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2023-07-02T14:01:07.891 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-02T14:01:07.956 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2023-07-02T14:01:07.956 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2023-07-02T14:01:07.956 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000127106 s, 4.0 MB/s 2023-07-02T14:01:07.957 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-02T14:01:08.014 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2023-07-02T14:01:08.070 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-02T14:01:08.070 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-02T14:01:08.070 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6h/6d Inode: 210611 Links: 1 2023-07-02T14:01:08.070 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-02T14:01:08.070 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2023-07-02T14:01:08.071 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2023-07-02 13:58:51.618610144 +0000 2023-07-02T14:01:08.071 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2023-07-02 13:56:08.415872961 +0000 2023-07-02T14:01:08.071 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2023-07-02 13:56:08.415872961 +0000 2023-07-02T14:01:08.071 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2023-07-02T14:01:08.071 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-02T14:01:08.135 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2023-07-02T14:01:08.135 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2023-07-02T14:01:08.135 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 9.3179e-05 s, 5.5 MB/s 2023-07-02T14:01:08.136 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-02T14:01:08.193 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2023-07-02T14:01:08.249 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-02T14:01:08.249 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-02T14:01:08.249 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6h/6d Inode: 215521 Links: 1 2023-07-02T14:01:08.249 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-02T14:01:08.250 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2023-07-02T14:01:08.250 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2023-07-02 13:58:51.619610128 +0000 2023-07-02T14:01:08.250 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2023-07-02 13:56:08.941864677 +0000 2023-07-02T14:01:08.250 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2023-07-02 13:56:08.941864677 +0000 2023-07-02T14:01:08.250 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2023-07-02T14:01:08.250 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-02T14:01:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2023-07-02T14:01:08.313 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2023-07-02T14:01:08.314 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 9.7311e-05 s, 5.3 MB/s 2023-07-02T14:01:08.315 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-02T14:01:08.372 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2023-07-02T14:01:08.427 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-02T14:01:08.428 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-02T14:01:08.428 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6h/6d Inode: 215701 Links: 1 2023-07-02T14:01:08.428 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-02T14:01:08.428 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2023-07-02T14:01:08.428 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2023-07-02 13:58:51.619610128 +0000 2023-07-02T14:01:08.428 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2023-07-02 13:56:09.450856660 +0000 2023-07-02T14:01:08.428 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2023-07-02 13:56:09.450856660 +0000 2023-07-02T14:01:08.429 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2023-07-02T14:01:08.429 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-02T14:01:08.491 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2023-07-02T14:01:08.491 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2023-07-02T14:01:08.491 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000130328 s, 3.9 MB/s 2023-07-02T14:01:08.492 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-02T14:01:08.548 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-07-02T14:01:08.548 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-07-02T14:01:08.549 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/scratch_devs of=/dev/stdout 2023-07-02T14:01:08.567 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-02T14:01:08.567 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_1 2023-07-02T14:01:08.625 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-02T14:01:08.626 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-02T14:01:08.626 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 6h/6d Inode: 211196 Links: 1 2023-07-02T14:01:08.626 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-02T14:01:08.626 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2023-07-02T14:01:08.626 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2023-07-02 14:01:00.939935473 +0000 2023-07-02T14:01:08.626 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2023-07-02 13:56:07.551606013 +0000 2023-07-02T14:01:08.627 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2023-07-02 13:56:07.551606013 +0000 2023-07-02T14:01:08.627 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: - 2023-07-02T14:01:08.627 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-02T14:01:08.694 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2023-07-02T14:01:08.695 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2023-07-02T14:01:08.695 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 0.000143508 s, 3.6 MB/s 2023-07-02T14:01:08.696 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-02T14:01:08.755 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_2 2023-07-02T14:01:08.813 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-02T14:01:08.813 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-02T14:01:08.814 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 6h/6d Inode: 215296 Links: 1 2023-07-02T14:01:08.814 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-02T14:01:08.814 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2023-07-02T14:01:08.814 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2023-07-02 14:01:00.939935473 +0000 2023-07-02T14:01:08.814 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2023-07-02 13:56:08.067599153 +0000 2023-07-02T14:01:08.814 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2023-07-02 13:56:08.067599153 +0000 2023-07-02T14:01:08.815 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: - 2023-07-02T14:01:08.815 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-02T14:01:08.881 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2023-07-02T14:01:08.881 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2023-07-02T14:01:08.882 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 0.000135945 s, 3.8 MB/s 2023-07-02T14:01:08.882 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-02T14:01:08.941 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_3 2023-07-02T14:01:08.999 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-02T14:01:09.000 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-02T14:01:09.000 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 6h/6d Inode: 211329 Links: 1 2023-07-02T14:01:09.000 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-02T14:01:09.000 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2023-07-02T14:01:09.000 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2023-07-02 14:01:00.939935473 +0000 2023-07-02T14:01:09.001 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2023-07-02 13:56:08.594592147 +0000 2023-07-02T14:01:09.001 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2023-07-02 13:56:08.594592147 +0000 2023-07-02T14:01:09.001 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: - 2023-07-02T14:01:09.001 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-02T14:01:09.068 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2023-07-02T14:01:09.069 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2023-07-02T14:01:09.069 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 0.000132512 s, 3.9 MB/s 2023-07-02T14:01:09.070 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-02T14:01:09.129 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_4 2023-07-02T14:01:09.187 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-02T14:01:09.188 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-02T14:01:09.188 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 6h/6d Inode: 209723 Links: 1 2023-07-02T14:01:09.188 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-02T14:01:09.188 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2023-07-02T14:01:09.188 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2023-07-02 14:01:00.939935473 +0000 2023-07-02T14:01:09.188 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2023-07-02 13:56:09.119585167 +0000 2023-07-02T14:01:09.188 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2023-07-02 13:56:09.119585167 +0000 2023-07-02T14:01:09.189 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: - 2023-07-02T14:01:09.189 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-02T14:01:09.256 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2023-07-02T14:01:09.256 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2023-07-02T14:01:09.256 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 0.000127577 s, 4.0 MB/s 2023-07-02T14:01:09.257 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-02T14:01:09.316 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-07-02T14:01:09.316 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi022.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2023-07-02T14:01:09.316 INFO:tasks.ceph:Generating config... 2023-07-02T14:01:09.318 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-07-02T14:01:09.318 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-07-02T14:01:09.318 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-07-02T14:01:09.318 INFO:tasks.ceph:[mgr] debug ms = 1 2023-07-02T14:01:09.319 INFO:tasks.ceph:[mon] debug mon = 20 2023-07-02T14:01:09.319 INFO:tasks.ceph:[mon] debug ms = 1 2023-07-02T14:01:09.319 INFO:tasks.ceph:[mon] debug paxos = 20 2023-07-02T14:01:09.319 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2023-07-02T14:01:09.319 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2023-07-02T14:01:09.320 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-07-02T14:01:09.320 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-07-02T14:01:09.320 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-07-02T14:01:09.320 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-07-02T14:01:09.320 INFO:tasks.ceph:[osd] debug ms = 1 2023-07-02T14:01:09.321 INFO:tasks.ceph:[osd] debug osd = 20 2023-07-02T14:01:09.321 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-07-02T14:01:09.321 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-07-02T14:01:09.321 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-07-02T14:01:09.321 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-07-02T14:01:09.322 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-07-02T14:01:09.322 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-07-02T14:01:09.322 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-07-02T14:01:09.322 INFO:tasks.ceph:Setting up mon.a... 2023-07-02T14:01:09.323 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-07-02T14:01:09.419 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/ceph.keyring 2023-07-02T14:01:09.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-07-02T14:01:09.464 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-07-02T14:01:09.530 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.16'), ('mon.c', '[v2:172.21.15.16:3301,v1:172.21.15.16:6790]'), ('mon.b', '172.21.15.22')] 2023-07-02T14:01:09.531 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.16,[v2:172.21.15.16:3301,v1:172.21.15.16:6790],172.21.15.22', 'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-07-02T14:01:09.532 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-07-02T14:01:09.532 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-07-02T14:01:09.585 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.16 --addv c '[v2:172.21.15.16:3301,v1:172.21.15.16:6790]' --add b 172.21.15.22 --print /home/ubuntu/cephtest/ceph.monmap 2023-07-02T14:01:09.667 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-07-02T14:01:09.668 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: generated fsid 9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2 2023-07-02T14:01:09.668 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 0 2023-07-02T14:01:09.668 INFO:teuthology.orchestra.run.smithi016.stdout:fsid 9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2 2023-07-02T14:01:09.668 INFO:teuthology.orchestra.run.smithi016.stdout:last_changed 2023-07-02T14:01:09.667648+0000 2023-07-02T14:01:09.669 INFO:teuthology.orchestra.run.smithi016.stdout:created 2023-07-02T14:01:09.667648+0000 2023-07-02T14:01:09.669 INFO:teuthology.orchestra.run.smithi016.stdout:min_mon_release 0 (unknown) 2023-07-02T14:01:09.669 INFO:teuthology.orchestra.run.smithi016.stdout:election_strategy: 1 2023-07-02T14:01:09.669 INFO:teuthology.orchestra.run.smithi016.stdout:0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2023-07-02T14:01:09.669 INFO:teuthology.orchestra.run.smithi016.stdout:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-07-02T14:01:09.669 INFO:teuthology.orchestra.run.smithi016.stdout:2: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.c 2023-07-02T14:01:09.670 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-07-02T14:01:09.671 DEBUG:teuthology.orchestra.run.smithi016:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-07-02T14:01:09.726 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2... 2023-07-02T14:01:09.727 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-07-02T14:01:09.768 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-07-02T14:01:09.772 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-07-02T14:01:09.811 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2023-07-02T14:01:09.811 INFO:teuthology.orchestra.run.smithi016.stdout: chdir = "" 2023-07-02T14:01:09.811 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-07-02T14:01:09.812 INFO:teuthology.orchestra.run.smithi016.stdout: auth supported = cephx 2023-07-02T14:01:09.812 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.812 INFO:teuthology.orchestra.run.smithi016.stdout: filestore xattr use omap = true 2023-07-02T14:01:09.813 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.813 INFO:teuthology.orchestra.run.smithi016.stdout: mon clock drift allowed = 1.000 2023-07-02T14:01:09.813 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.814 INFO:teuthology.orchestra.run.smithi016.stdout: osd crush chooseleaf type = 0 2023-07-02T14:01:09.814 INFO:teuthology.orchestra.run.smithi016.stdout: auth debug = true 2023-07-02T14:01:09.814 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.814 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on old message = true 2023-07-02T14:01:09.815 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on bug = true 2023-07-02T14:01:09.815 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.815 INFO:teuthology.orchestra.run.smithi016.stdout: mon max pg per osd = 10000 # >= luminous 2023-07-02T14:01:09.816 INFO:teuthology.orchestra.run.smithi016.stdout: mon pg warn max object skew = 0 2023-07-02T14:01:09.816 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.816 INFO:teuthology.orchestra.run.smithi016.stdout: # disable pg_autoscaler by default for new pools 2023-07-02T14:01:09.817 INFO:teuthology.orchestra.run.smithi016.stdout: osd_pool_default_pg_autoscale_mode = off 2023-07-02T14:01:09.817 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.817 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default size = 2 2023-07-02T14:01:09.818 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.818 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow primary affinity = true 2023-07-02T14:01:09.818 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow pg remap = true 2023-07-02T14:01:09.818 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on legacy crush tunables = false 2023-07-02T14:01:09.819 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on crush straw calc version zero = false 2023-07-02T14:01:09.819 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on no sortbitwise = false 2023-07-02T14:01:09.819 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on osd down out interval zero = false 2023-07-02T14:01:09.820 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on too few osds = false 2023-07-02T14:01:09.820 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-07-02T14:01:09.820 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_no_redundancy = false 2023-07-02T14:01:09.821 INFO:teuthology.orchestra.run.smithi016.stdout: mon_allow_pool_size_one = true 2023-07-02T14:01:09.821 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.821 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-07-02T14:01:09.821 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.822 INFO:teuthology.orchestra.run.smithi016.stdout: osd default data pool replay window = 5 2023-07-02T14:01:09.822 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.822 INFO:teuthology.orchestra.run.smithi016.stdout: mon allow pool delete = true 2023-07-02T14:01:09.823 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.823 INFO:teuthology.orchestra.run.smithi016.stdout: mon cluster log file level = debug 2023-07-02T14:01:09.823 INFO:teuthology.orchestra.run.smithi016.stdout: debug asserts on shutdown = true 2023-07-02T14:01:09.823 INFO:teuthology.orchestra.run.smithi016.stdout: mon health detail to clog = false 2023-07-02T14:01:09.824 INFO:teuthology.orchestra.run.smithi016.stdout: mon host = "172.21.15.16,[v2:172.21.15.16:3301,v1:172.21.15.16:6790],172.21.15.22" 2023-07-02T14:01:09.824 INFO:teuthology.orchestra.run.smithi016.stdout: mon client directed command retry = 5 2023-07-02T14:01:09.824 INFO:teuthology.orchestra.run.smithi016.stdout: ms inject socket failures = 5000 2023-07-02T14:01:09.825 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = 9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2 2023-07-02T14:01:09.825 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.825 INFO:teuthology.orchestra.run.smithi016.stdout:[osd] 2023-07-02T14:01:09.825 INFO:teuthology.orchestra.run.smithi016.stdout: osd journal size = 100 2023-07-02T14:01:09.826 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.826 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub load threshold = 5.0 2023-07-02T14:01:09.826 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub max interval = 600 2023-07-02T14:01:09.826 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.827 INFO:teuthology.orchestra.run.smithi016.stdout: osd recover clone overlap = true 2023-07-02T14:01:09.827 INFO:teuthology.orchestra.run.smithi016.stdout: osd recovery max chunk = 1048576 2023-07-02T14:01:09.827 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.827 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug shutdown = true 2023-07-02T14:01:09.828 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug op order = true 2023-07-02T14:01:09.828 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify stray on activate = true 2023-07-02T14:01:09.828 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.828 INFO:teuthology.orchestra.run.smithi016.stdout: osd open classes on start = true 2023-07-02T14:01:09.829 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug pg log writeout = true 2023-07-02T14:01:09.829 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.829 INFO:teuthology.orchestra.run.smithi016.stdout: osd deep scrub update digest min age = 30 2023-07-02T14:01:09.829 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.830 INFO:teuthology.orchestra.run.smithi016.stdout: osd map max advance = 10 2023-07-02T14:01:09.830 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.830 INFO:teuthology.orchestra.run.smithi016.stdout: journal zero on create = true 2023-07-02T14:01:09.830 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.831 INFO:teuthology.orchestra.run.smithi016.stdout: filestore ondisk finisher threads = 3 2023-07-02T14:01:09.831 INFO:teuthology.orchestra.run.smithi016.stdout: filestore apply finisher threads = 3 2023-07-02T14:01:09.831 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.831 INFO:teuthology.orchestra.run.smithi016.stdout: bdev debug aio = true 2023-07-02T14:01:09.832 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug misdirected ops = true 2023-07-02T14:01:09.832 INFO:teuthology.orchestra.run.smithi016.stdout: bluefs allocator = hybrid 2023-07-02T14:01:09.832 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore allocator = hybrid 2023-07-02T14:01:09.833 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore block size = 96636764160 2023-07-02T14:01:09.833 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore fsck on mount = True 2023-07-02T14:01:09.833 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluefs = 1/20 2023-07-02T14:01:09.833 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluestore = 1/20 2023-07-02T14:01:09.834 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2023-07-02T14:01:09.834 INFO:teuthology.orchestra.run.smithi016.stdout: debug osd = 20 2023-07-02T14:01:09.834 INFO:teuthology.orchestra.run.smithi016.stdout: debug rocksdb = 4/10 2023-07-02T14:01:09.834 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd backfillfull_ratio = 0.85 2023-07-02T14:01:09.835 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd full ratio = 0.9 2023-07-02T14:01:09.835 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd nearfull ratio = 0.8 2023-07-02T14:01:09.835 INFO:teuthology.orchestra.run.smithi016.stdout: osd failsafe full ratio = 0.95 2023-07-02T14:01:09.836 INFO:teuthology.orchestra.run.smithi016.stdout: osd max markdown count = 1000 2023-07-02T14:01:09.836 INFO:teuthology.orchestra.run.smithi016.stdout: osd objectstore = bluestore 2023-07-02T14:01:09.836 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.836 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr] 2023-07-02T14:01:09.837 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2023-07-02T14:01:09.837 INFO:teuthology.orchestra.run.smithi016.stdout: debug mgr = 20 2023-07-02T14:01:09.837 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2023-07-02T14:01:09.838 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2023-07-02T14:01:09.838 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2023-07-02T14:01:09.838 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2023-07-02T14:01:09.839 INFO:teuthology.orchestra.run.smithi016.stdout: mgr/telemetry/nag = false 2023-07-02T14:01:09.839 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.839 INFO:teuthology.orchestra.run.smithi016.stdout:[mon] 2023-07-02T14:01:09.839 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2023-07-02T14:01:09.840 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2023-07-02T14:01:09.840 INFO:teuthology.orchestra.run.smithi016.stdout: debug paxos = 20 2023-07-02T14:01:09.840 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2023-07-02T14:01:09.841 INFO:teuthology.orchestra.run.smithi016.stdout: mon data avail warn = 5 2023-07-02T14:01:09.841 INFO:teuthology.orchestra.run.smithi016.stdout: mon mgr mkfs grace = 240 2023-07-02T14:01:09.841 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2023-07-02T14:01:09.842 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd reporter subtree level = osd 2023-07-02T14:01:09.842 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd prime pg temp = true 2023-07-02T14:01:09.842 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2023-07-02T14:01:09.842 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.843 INFO:teuthology.orchestra.run.smithi016.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-07-02T14:01:09.843 INFO:teuthology.orchestra.run.smithi016.stdout: auth mon ticket ttl = 660 # 11m 2023-07-02T14:01:09.843 INFO:teuthology.orchestra.run.smithi016.stdout: auth service ticket ttl = 240 # 4m 2023-07-02T14:01:09.844 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.844 INFO:teuthology.orchestra.run.smithi016.stdout: # don't complain about insecure global_id in the test suite 2023-07-02T14:01:09.844 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-07-02T14:01:09.844 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-07-02T14:01:09.845 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:09.845 INFO:teuthology.orchestra.run.smithi016.stdout:[client] 2023-07-02T14:01:09.845 INFO:teuthology.orchestra.run.smithi016.stdout: rgw cache enabled = true 2023-07-02T14:01:09.845 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable ops log = true 2023-07-02T14:01:09.845 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable usage log = true 2023-07-02T14:01:09.845 INFO:teuthology.orchestra.run.smithi016.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-07-02T14:01:09.845 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-07-02T14:01:09.846 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.a] 2023-07-02T14:01:09.846 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.c] 2023-07-02T14:01:09.846 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.b] 2023-07-02T14:01:09.848 INFO:teuthology.orchestra.run.smithi022.stdout:[global] 2023-07-02T14:01:09.848 INFO:teuthology.orchestra.run.smithi022.stdout: chdir = "" 2023-07-02T14:01:09.848 INFO:teuthology.orchestra.run.smithi022.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-07-02T14:01:09.848 INFO:teuthology.orchestra.run.smithi022.stdout: auth supported = cephx 2023-07-02T14:01:09.848 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.848 INFO:teuthology.orchestra.run.smithi022.stdout: filestore xattr use omap = true 2023-07-02T14:01:09.849 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.849 INFO:teuthology.orchestra.run.smithi022.stdout: mon clock drift allowed = 1.000 2023-07-02T14:01:09.849 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.849 INFO:teuthology.orchestra.run.smithi022.stdout: osd crush chooseleaf type = 0 2023-07-02T14:01:09.849 INFO:teuthology.orchestra.run.smithi022.stdout: auth debug = true 2023-07-02T14:01:09.849 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.849 INFO:teuthology.orchestra.run.smithi022.stdout: ms die on old message = true 2023-07-02T14:01:09.850 INFO:teuthology.orchestra.run.smithi022.stdout: ms die on bug = true 2023-07-02T14:01:09.850 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.850 INFO:teuthology.orchestra.run.smithi022.stdout: mon max pg per osd = 10000 # >= luminous 2023-07-02T14:01:09.850 INFO:teuthology.orchestra.run.smithi022.stdout: mon pg warn max object skew = 0 2023-07-02T14:01:09.850 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.850 INFO:teuthology.orchestra.run.smithi022.stdout: # disable pg_autoscaler by default for new pools 2023-07-02T14:01:09.850 INFO:teuthology.orchestra.run.smithi022.stdout: osd_pool_default_pg_autoscale_mode = off 2023-07-02T14:01:09.850 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.851 INFO:teuthology.orchestra.run.smithi022.stdout: osd pool default size = 2 2023-07-02T14:01:09.851 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.851 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd allow primary affinity = true 2023-07-02T14:01:09.851 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd allow pg remap = true 2023-07-02T14:01:09.851 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on legacy crush tunables = false 2023-07-02T14:01:09.851 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on crush straw calc version zero = false 2023-07-02T14:01:09.851 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on no sortbitwise = false 2023-07-02T14:01:09.851 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on osd down out interval zero = false 2023-07-02T14:01:09.852 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on too few osds = false 2023-07-02T14:01:09.852 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-07-02T14:01:09.852 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_pool_no_redundancy = false 2023-07-02T14:01:09.852 INFO:teuthology.orchestra.run.smithi022.stdout: mon_allow_pool_size_one = true 2023-07-02T14:01:09.852 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.852 INFO:teuthology.orchestra.run.smithi022.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-07-02T14:01:09.852 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.853 INFO:teuthology.orchestra.run.smithi022.stdout: osd default data pool replay window = 5 2023-07-02T14:01:09.853 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.853 INFO:teuthology.orchestra.run.smithi022.stdout: mon allow pool delete = true 2023-07-02T14:01:09.853 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.853 INFO:teuthology.orchestra.run.smithi022.stdout: mon cluster log file level = debug 2023-07-02T14:01:09.853 INFO:teuthology.orchestra.run.smithi022.stdout: debug asserts on shutdown = true 2023-07-02T14:01:09.853 INFO:teuthology.orchestra.run.smithi022.stdout: mon health detail to clog = false 2023-07-02T14:01:09.853 INFO:teuthology.orchestra.run.smithi022.stdout: mon host = "172.21.15.16,[v2:172.21.15.16:3301,v1:172.21.15.16:6790],172.21.15.22" 2023-07-02T14:01:09.854 INFO:teuthology.orchestra.run.smithi022.stdout: mon client directed command retry = 5 2023-07-02T14:01:09.854 INFO:teuthology.orchestra.run.smithi022.stdout: ms inject socket failures = 5000 2023-07-02T14:01:09.854 INFO:teuthology.orchestra.run.smithi022.stdout: fsid = 9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2 2023-07-02T14:01:09.854 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.854 INFO:teuthology.orchestra.run.smithi022.stdout:[osd] 2023-07-02T14:01:09.854 INFO:teuthology.orchestra.run.smithi022.stdout: osd journal size = 100 2023-07-02T14:01:09.854 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.854 INFO:teuthology.orchestra.run.smithi022.stdout: osd scrub load threshold = 5.0 2023-07-02T14:01:09.855 INFO:teuthology.orchestra.run.smithi022.stdout: osd scrub max interval = 600 2023-07-02T14:01:09.855 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.855 INFO:teuthology.orchestra.run.smithi022.stdout: osd recover clone overlap = true 2023-07-02T14:01:09.856 INFO:teuthology.orchestra.run.smithi022.stdout: osd recovery max chunk = 1048576 2023-07-02T14:01:09.856 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.856 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug shutdown = true 2023-07-02T14:01:09.856 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug op order = true 2023-07-02T14:01:09.857 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug verify stray on activate = true 2023-07-02T14:01:09.857 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.857 INFO:teuthology.orchestra.run.smithi022.stdout: osd open classes on start = true 2023-07-02T14:01:09.858 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug pg log writeout = true 2023-07-02T14:01:09.858 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.858 INFO:teuthology.orchestra.run.smithi022.stdout: osd deep scrub update digest min age = 30 2023-07-02T14:01:09.858 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.859 INFO:teuthology.orchestra.run.smithi022.stdout: osd map max advance = 10 2023-07-02T14:01:09.859 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.859 INFO:teuthology.orchestra.run.smithi022.stdout: journal zero on create = true 2023-07-02T14:01:09.860 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.860 INFO:teuthology.orchestra.run.smithi022.stdout: filestore ondisk finisher threads = 3 2023-07-02T14:01:09.860 INFO:teuthology.orchestra.run.smithi022.stdout: filestore apply finisher threads = 3 2023-07-02T14:01:09.860 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.861 INFO:teuthology.orchestra.run.smithi022.stdout: bdev debug aio = true 2023-07-02T14:01:09.861 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug misdirected ops = true 2023-07-02T14:01:09.861 INFO:teuthology.orchestra.run.smithi022.stdout: bluefs allocator = hybrid 2023-07-02T14:01:09.862 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore allocator = hybrid 2023-07-02T14:01:09.862 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore block size = 96636764160 2023-07-02T14:01:09.862 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore fsck on mount = True 2023-07-02T14:01:09.862 INFO:teuthology.orchestra.run.smithi022.stdout: debug bluefs = 1/20 2023-07-02T14:01:09.862 INFO:teuthology.orchestra.run.smithi022.stdout: debug bluestore = 1/20 2023-07-02T14:01:09.863 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2023-07-02T14:01:09.863 INFO:teuthology.orchestra.run.smithi022.stdout: debug osd = 20 2023-07-02T14:01:09.863 INFO:teuthology.orchestra.run.smithi022.stdout: debug rocksdb = 4/10 2023-07-02T14:01:09.863 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd backfillfull_ratio = 0.85 2023-07-02T14:01:09.864 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd full ratio = 0.9 2023-07-02T14:01:09.864 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd nearfull ratio = 0.8 2023-07-02T14:01:09.864 INFO:teuthology.orchestra.run.smithi022.stdout: osd failsafe full ratio = 0.95 2023-07-02T14:01:09.864 INFO:teuthology.orchestra.run.smithi022.stdout: osd max markdown count = 1000 2023-07-02T14:01:09.865 INFO:teuthology.orchestra.run.smithi022.stdout: osd objectstore = bluestore 2023-07-02T14:01:09.865 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.865 INFO:teuthology.orchestra.run.smithi022.stdout:[mgr] 2023-07-02T14:01:09.865 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2023-07-02T14:01:09.866 INFO:teuthology.orchestra.run.smithi022.stdout: debug mgr = 20 2023-07-02T14:01:09.866 INFO:teuthology.orchestra.run.smithi022.stdout: debug mon = 20 2023-07-02T14:01:09.866 INFO:teuthology.orchestra.run.smithi022.stdout: debug auth = 20 2023-07-02T14:01:09.866 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min pgs per osd = 4 2023-07-02T14:01:09.866 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min bytes per osd = 10 2023-07-02T14:01:09.867 INFO:teuthology.orchestra.run.smithi022.stdout: mgr/telemetry/nag = false 2023-07-02T14:01:09.867 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.867 INFO:teuthology.orchestra.run.smithi022.stdout:[mon] 2023-07-02T14:01:09.867 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2023-07-02T14:01:09.868 INFO:teuthology.orchestra.run.smithi022.stdout: debug mon = 20 2023-07-02T14:01:09.868 INFO:teuthology.orchestra.run.smithi022.stdout: debug paxos = 20 2023-07-02T14:01:09.868 INFO:teuthology.orchestra.run.smithi022.stdout: debug auth = 20 2023-07-02T14:01:09.868 INFO:teuthology.orchestra.run.smithi022.stdout: mon data avail warn = 5 2023-07-02T14:01:09.869 INFO:teuthology.orchestra.run.smithi022.stdout: mon mgr mkfs grace = 240 2023-07-02T14:01:09.869 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min pgs per osd = 4 2023-07-02T14:01:09.869 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd reporter subtree level = osd 2023-07-02T14:01:09.869 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd prime pg temp = true 2023-07-02T14:01:09.870 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min bytes per osd = 10 2023-07-02T14:01:09.870 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.870 INFO:teuthology.orchestra.run.smithi022.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-07-02T14:01:09.871 INFO:teuthology.orchestra.run.smithi022.stdout: auth mon ticket ttl = 660 # 11m 2023-07-02T14:01:09.871 INFO:teuthology.orchestra.run.smithi022.stdout: auth service ticket ttl = 240 # 4m 2023-07-02T14:01:09.871 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.871 INFO:teuthology.orchestra.run.smithi022.stdout: # don't complain about insecure global_id in the test suite 2023-07-02T14:01:09.872 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-07-02T14:01:09.872 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-07-02T14:01:09.872 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-07-02T14:01:09.873 INFO:teuthology.orchestra.run.smithi022.stdout:[client] 2023-07-02T14:01:09.873 INFO:teuthology.orchestra.run.smithi022.stdout: rgw cache enabled = true 2023-07-02T14:01:09.873 INFO:teuthology.orchestra.run.smithi022.stdout: rgw enable ops log = true 2023-07-02T14:01:09.874 INFO:teuthology.orchestra.run.smithi022.stdout: rgw enable usage log = true 2023-07-02T14:01:09.874 INFO:teuthology.orchestra.run.smithi022.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-07-02T14:01:09.874 INFO:teuthology.orchestra.run.smithi022.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-07-02T14:01:09.874 INFO:teuthology.orchestra.run.smithi022.stdout:[mon.a] 2023-07-02T14:01:09.875 INFO:teuthology.orchestra.run.smithi022.stdout:[mon.c] 2023-07-02T14:01:09.875 INFO:teuthology.orchestra.run.smithi022.stdout:[mon.b] 2023-07-02T14:01:09.875 INFO:teuthology.orchestra.run.smithi163.stdout:[global] 2023-07-02T14:01:09.876 INFO:teuthology.orchestra.run.smithi163.stdout: chdir = "" 2023-07-02T14:01:09.876 INFO:teuthology.orchestra.run.smithi163.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-07-02T14:01:09.876 INFO:teuthology.orchestra.run.smithi163.stdout: auth supported = cephx 2023-07-02T14:01:09.877 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.877 INFO:teuthology.orchestra.run.smithi163.stdout: filestore xattr use omap = true 2023-07-02T14:01:09.877 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.877 INFO:teuthology.orchestra.run.smithi163.stdout: mon clock drift allowed = 1.000 2023-07-02T14:01:09.878 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.878 INFO:teuthology.orchestra.run.smithi163.stdout: osd crush chooseleaf type = 0 2023-07-02T14:01:09.878 INFO:teuthology.orchestra.run.smithi163.stdout: auth debug = true 2023-07-02T14:01:09.879 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.879 INFO:teuthology.orchestra.run.smithi163.stdout: ms die on old message = true 2023-07-02T14:01:09.879 INFO:teuthology.orchestra.run.smithi163.stdout: ms die on bug = true 2023-07-02T14:01:09.880 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.880 INFO:teuthology.orchestra.run.smithi163.stdout: mon max pg per osd = 10000 # >= luminous 2023-07-02T14:01:09.880 INFO:teuthology.orchestra.run.smithi163.stdout: mon pg warn max object skew = 0 2023-07-02T14:01:09.880 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.881 INFO:teuthology.orchestra.run.smithi163.stdout: # disable pg_autoscaler by default for new pools 2023-07-02T14:01:09.881 INFO:teuthology.orchestra.run.smithi163.stdout: osd_pool_default_pg_autoscale_mode = off 2023-07-02T14:01:09.881 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.882 INFO:teuthology.orchestra.run.smithi163.stdout: osd pool default size = 2 2023-07-02T14:01:09.882 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.882 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd allow primary affinity = true 2023-07-02T14:01:09.882 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd allow pg remap = true 2023-07-02T14:01:09.883 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on legacy crush tunables = false 2023-07-02T14:01:09.883 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on crush straw calc version zero = false 2023-07-02T14:01:09.883 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on no sortbitwise = false 2023-07-02T14:01:09.884 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on osd down out interval zero = false 2023-07-02T14:01:09.884 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on too few osds = false 2023-07-02T14:01:09.884 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-07-02T14:01:09.884 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_pool_no_redundancy = false 2023-07-02T14:01:09.885 INFO:teuthology.orchestra.run.smithi163.stdout: mon_allow_pool_size_one = true 2023-07-02T14:01:09.885 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.885 INFO:teuthology.orchestra.run.smithi163.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-07-02T14:01:09.885 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.886 INFO:teuthology.orchestra.run.smithi163.stdout: osd default data pool replay window = 5 2023-07-02T14:01:09.886 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.886 INFO:teuthology.orchestra.run.smithi163.stdout: mon allow pool delete = true 2023-07-02T14:01:09.887 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.887 INFO:teuthology.orchestra.run.smithi163.stdout: mon cluster log file level = debug 2023-07-02T14:01:09.887 INFO:teuthology.orchestra.run.smithi163.stdout: debug asserts on shutdown = true 2023-07-02T14:01:09.887 INFO:teuthology.orchestra.run.smithi163.stdout: mon health detail to clog = false 2023-07-02T14:01:09.888 INFO:teuthology.orchestra.run.smithi163.stdout: mon host = "172.21.15.16,[v2:172.21.15.16:3301,v1:172.21.15.16:6790],172.21.15.22" 2023-07-02T14:01:09.888 INFO:teuthology.orchestra.run.smithi163.stdout: mon client directed command retry = 5 2023-07-02T14:01:09.888 INFO:teuthology.orchestra.run.smithi163.stdout: ms inject socket failures = 5000 2023-07-02T14:01:09.888 INFO:teuthology.orchestra.run.smithi163.stdout: fsid = 9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2 2023-07-02T14:01:09.889 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.889 INFO:teuthology.orchestra.run.smithi163.stdout:[osd] 2023-07-02T14:01:09.889 INFO:teuthology.orchestra.run.smithi163.stdout: osd journal size = 100 2023-07-02T14:01:09.889 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.890 INFO:teuthology.orchestra.run.smithi163.stdout: osd scrub load threshold = 5.0 2023-07-02T14:01:09.890 INFO:teuthology.orchestra.run.smithi163.stdout: osd scrub max interval = 600 2023-07-02T14:01:09.890 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.890 INFO:teuthology.orchestra.run.smithi163.stdout: osd recover clone overlap = true 2023-07-02T14:01:09.891 INFO:teuthology.orchestra.run.smithi163.stdout: osd recovery max chunk = 1048576 2023-07-02T14:01:09.891 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.891 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug shutdown = true 2023-07-02T14:01:09.891 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug op order = true 2023-07-02T14:01:09.891 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug verify stray on activate = true 2023-07-02T14:01:09.892 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.892 INFO:teuthology.orchestra.run.smithi163.stdout: osd open classes on start = true 2023-07-02T14:01:09.892 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug pg log writeout = true 2023-07-02T14:01:09.892 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.893 INFO:teuthology.orchestra.run.smithi163.stdout: osd deep scrub update digest min age = 30 2023-07-02T14:01:09.893 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.893 INFO:teuthology.orchestra.run.smithi163.stdout: osd map max advance = 10 2023-07-02T14:01:09.894 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.894 INFO:teuthology.orchestra.run.smithi163.stdout: journal zero on create = true 2023-07-02T14:01:09.894 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.894 INFO:teuthology.orchestra.run.smithi163.stdout: filestore ondisk finisher threads = 3 2023-07-02T14:01:09.895 INFO:teuthology.orchestra.run.smithi163.stdout: filestore apply finisher threads = 3 2023-07-02T14:01:09.895 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.895 INFO:teuthology.orchestra.run.smithi163.stdout: bdev debug aio = true 2023-07-02T14:01:09.895 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug misdirected ops = true 2023-07-02T14:01:09.896 INFO:teuthology.orchestra.run.smithi163.stdout: bluefs allocator = hybrid 2023-07-02T14:01:09.896 INFO:teuthology.orchestra.run.smithi163.stdout: bluestore allocator = hybrid 2023-07-02T14:01:09.896 INFO:teuthology.orchestra.run.smithi163.stdout: bluestore block size = 96636764160 2023-07-02T14:01:09.896 INFO:teuthology.orchestra.run.smithi163.stdout: bluestore fsck on mount = True 2023-07-02T14:01:09.897 INFO:teuthology.orchestra.run.smithi163.stdout: debug bluefs = 1/20 2023-07-02T14:01:09.897 INFO:teuthology.orchestra.run.smithi163.stdout: debug bluestore = 1/20 2023-07-02T14:01:09.897 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2023-07-02T14:01:09.898 INFO:teuthology.orchestra.run.smithi163.stdout: debug osd = 20 2023-07-02T14:01:09.898 INFO:teuthology.orchestra.run.smithi163.stdout: debug rocksdb = 4/10 2023-07-02T14:01:09.898 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd backfillfull_ratio = 0.85 2023-07-02T14:01:09.898 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd full ratio = 0.9 2023-07-02T14:01:09.899 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd nearfull ratio = 0.8 2023-07-02T14:01:09.899 INFO:teuthology.orchestra.run.smithi163.stdout: osd failsafe full ratio = 0.95 2023-07-02T14:01:09.899 INFO:teuthology.orchestra.run.smithi163.stdout: osd max markdown count = 1000 2023-07-02T14:01:09.900 INFO:teuthology.orchestra.run.smithi163.stdout: osd objectstore = bluestore 2023-07-02T14:01:09.900 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.900 INFO:teuthology.orchestra.run.smithi163.stdout:[mgr] 2023-07-02T14:01:09.901 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2023-07-02T14:01:09.901 INFO:teuthology.orchestra.run.smithi163.stdout: debug mgr = 20 2023-07-02T14:01:09.901 INFO:teuthology.orchestra.run.smithi163.stdout: debug mon = 20 2023-07-02T14:01:09.901 INFO:teuthology.orchestra.run.smithi163.stdout: debug auth = 20 2023-07-02T14:01:09.902 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min pgs per osd = 4 2023-07-02T14:01:09.902 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min bytes per osd = 10 2023-07-02T14:01:09.902 INFO:teuthology.orchestra.run.smithi163.stdout: mgr/telemetry/nag = false 2023-07-02T14:01:09.903 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.903 INFO:teuthology.orchestra.run.smithi163.stdout:[mon] 2023-07-02T14:01:09.903 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2023-07-02T14:01:09.904 INFO:teuthology.orchestra.run.smithi163.stdout: debug mon = 20 2023-07-02T14:01:09.904 INFO:teuthology.orchestra.run.smithi163.stdout: debug paxos = 20 2023-07-02T14:01:09.904 INFO:teuthology.orchestra.run.smithi163.stdout: debug auth = 20 2023-07-02T14:01:09.904 INFO:teuthology.orchestra.run.smithi163.stdout: mon data avail warn = 5 2023-07-02T14:01:09.905 INFO:teuthology.orchestra.run.smithi163.stdout: mon mgr mkfs grace = 240 2023-07-02T14:01:09.905 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min pgs per osd = 4 2023-07-02T14:01:09.905 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd reporter subtree level = osd 2023-07-02T14:01:09.905 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd prime pg temp = true 2023-07-02T14:01:09.906 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min bytes per osd = 10 2023-07-02T14:01:09.906 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.906 INFO:teuthology.orchestra.run.smithi163.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-07-02T14:01:09.906 INFO:teuthology.orchestra.run.smithi163.stdout: auth mon ticket ttl = 660 # 11m 2023-07-02T14:01:09.906 INFO:teuthology.orchestra.run.smithi163.stdout: auth service ticket ttl = 240 # 4m 2023-07-02T14:01:09.906 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.906 INFO:teuthology.orchestra.run.smithi163.stdout: # don't complain about insecure global_id in the test suite 2023-07-02T14:01:09.907 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-07-02T14:01:09.907 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-07-02T14:01:09.907 INFO:teuthology.orchestra.run.smithi163.stdout: 2023-07-02T14:01:09.907 INFO:teuthology.orchestra.run.smithi163.stdout:[client] 2023-07-02T14:01:09.907 INFO:teuthology.orchestra.run.smithi163.stdout: rgw cache enabled = true 2023-07-02T14:01:09.907 INFO:teuthology.orchestra.run.smithi163.stdout: rgw enable ops log = true 2023-07-02T14:01:09.907 INFO:teuthology.orchestra.run.smithi163.stdout: rgw enable usage log = true 2023-07-02T14:01:09.907 INFO:teuthology.orchestra.run.smithi163.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-07-02T14:01:09.908 INFO:teuthology.orchestra.run.smithi163.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-07-02T14:01:09.908 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.a] 2023-07-02T14:01:09.908 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.c] 2023-07-02T14:01:09.908 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.b] 2023-07-02T14:01:09.908 INFO:tasks.ceph:Creating admin key on mon.a... 2023-07-02T14:01:09.909 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-07-02T14:01:09.959 INFO:tasks.ceph:Copying monmap to all nodes... 2023-07-02T14:01:09.960 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-07-02T14:01:09.960 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-07-02T14:01:09.975 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-07-02T14:01:09.976 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-07-02T14:01:10.030 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi016.front.sepia.ceph.com 2023-07-02T14:01:10.031 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-07-02T14:01:10.032 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.keyring 2023-07-02T14:01:10.032 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-07-02T14:01:10.105 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-07-02T14:01:10.106 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-07-02T14:01:10.160 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi022.front.sepia.ceph.com 2023-07-02T14:01:10.160 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-07-02T14:01:10.161 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/ceph/ceph.keyring 2023-07-02T14:01:10.161 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-07-02T14:01:10.201 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-07-02T14:01:10.201 DEBUG:teuthology.orchestra.run.smithi022:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-07-02T14:01:10.257 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi163.front.sepia.ceph.com 2023-07-02T14:01:10.258 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-07-02T14:01:10.258 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.keyring 2023-07-02T14:01:10.259 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-07-02T14:01:10.294 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-07-02T14:01:10.294 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-07-02T14:01:10.348 INFO:tasks.ceph:Setting up mon nodes... 2023-07-02T14:01:10.349 INFO:tasks.ceph:Setting up mgr nodes... 2023-07-02T14:01:10.349 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-07-02T14:01:10.417 INFO:teuthology.orchestra.run.smithi022.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-07-02T14:01:10.420 INFO:tasks.ceph:Setting up mds nodes... 2023-07-02T14:01:10.420 INFO:tasks.ceph_client:Setting up client nodes... 2023-07-02T14:01:10.420 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-07-02T14:01:10.487 INFO:teuthology.orchestra.run.smithi163.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-07-02T14:01:10.500 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-07-02T14:01:10.500 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi022.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2023-07-02T14:01:10.500 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-07-02T14:01:10.525 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-07-02T14:01:10.526 INFO:tasks.ceph:role: osd.0 2023-07-02T14:01:10.526 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com 2023-07-02T14:01:10.526 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-07-02T14:01:10.591 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-07-02T14:01:10.592 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-02T14:01:10.592 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-02T14:01:10.592 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 2023-07-02T14:01:10.592 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-02T14:01:10.592 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2023-07-02T14:01:10.592 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-02T14:01:10.593 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-02T14:01:10.593 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-02T14:01:10.593 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-02T14:01:10.614 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2023-07-02T14:01:10.616 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2023-07-02T14:01:10.616 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-07-02T14:01:10.695 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-07-02T14:01:10.765 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-07-02T14:01:10.830 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-07-02T14:01:10.830 INFO:tasks.ceph:role: osd.1 2023-07-02T14:01:10.831 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com 2023-07-02T14:01:10.831 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-07-02T14:01:10.894 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-07-02T14:01:10.895 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-02T14:01:10.895 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-02T14:01:10.895 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 2023-07-02T14:01:10.895 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-02T14:01:10.895 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2023-07-02T14:01:10.895 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-02T14:01:10.896 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-02T14:01:10.896 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-02T14:01:10.896 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-02T14:01:10.930 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2023-07-02T14:01:10.932 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2023-07-02T14:01:10.933 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-07-02T14:01:11.009 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-07-02T14:01:11.078 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-07-02T14:01:11.143 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-07-02T14:01:11.144 INFO:tasks.ceph:role: osd.2 2023-07-02T14:01:11.144 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com 2023-07-02T14:01:11.144 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-07-02T14:01:11.207 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-07-02T14:01:11.208 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-02T14:01:11.208 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-02T14:01:11.208 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 2023-07-02T14:01:11.208 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-02T14:01:11.208 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2023-07-02T14:01:11.208 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-02T14:01:11.208 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-02T14:01:11.209 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-02T14:01:11.209 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-02T14:01:11.233 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2023-07-02T14:01:11.234 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2023-07-02T14:01:11.235 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-07-02T14:01:11.312 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-07-02T14:01:11.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-07-02T14:01:11.470 INFO:teuthology.orchestra.run.smithi016.stderr:2023-07-02T14:01:11.469+0000 7f011d2a1200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-07-02T14:01:11.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-07-02T14:01:11.469+0000 7f011d2a1200 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-07-02T14:01:11.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-07-02T14:01:11.469+0000 7f011d2a1200 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-07-02T14:01:11.471 INFO:teuthology.orchestra.run.smithi016.stderr:2023-07-02T14:01:11.469+0000 7f011d2a1200 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-07-02T14:01:11.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-07-02T14:01:11.469+0000 7f011d2a1200 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-07-02T14:01:11.472 INFO:teuthology.orchestra.run.smithi016.stderr:2023-07-02T14:01:11.469+0000 7f011d2a1200 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-07-02T14:01:14.318 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-07-02T14:01:14.349 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-07-02T14:01:14.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-07-02T14:01:14.437+0000 7f16b7708200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-07-02T14:01:14.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-07-02T14:01:14.437+0000 7f16b7708200 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-07-02T14:01:14.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-07-02T14:01:14.437+0000 7f16b7708200 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-07-02T14:01:14.438 INFO:teuthology.orchestra.run.smithi016.stderr:2023-07-02T14:01:14.437+0000 7f16b7708200 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-07-02T14:01:14.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-07-02T14:01:14.437+0000 7f16b7708200 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-07-02T14:01:14.439 INFO:teuthology.orchestra.run.smithi016.stderr:2023-07-02T14:01:14.437+0000 7f16b7708200 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-07-02T14:01:17.273 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-07-02T14:01:17.301 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-07-02T14:01:17.385 INFO:teuthology.orchestra.run.smithi016.stderr:2023-07-02T14:01:17.385+0000 7fe668dbb200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-07-02T14:01:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-07-02T14:01:17.385+0000 7fe668dbb200 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-07-02T14:01:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-07-02T14:01:17.385+0000 7fe668dbb200 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-07-02T14:01:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-07-02T14:01:17.385+0000 7fe668dbb200 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-07-02T14:01:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-07-02T14:01:17.385+0000 7fe668dbb200 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-07-02T14:01:17.386 INFO:teuthology.orchestra.run.smithi016.stderr:2023-07-02T14:01:17.385+0000 7fe668dbb200 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-07-02T14:01:20.219 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-07-02T14:01:20.248 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-07-02T14:01:20.276 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-07-02T14:01:20.276 INFO:tasks.ceph:role: osd.3 2023-07-02T14:01:20.276 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi022.front.sepia.ceph.com 2023-07-02T14:01:20.277 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-07-02T14:01:20.341 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-07-02T14:01:20.342 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-02T14:01:20.342 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-02T14:01:20.342 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 2023-07-02T14:01:20.342 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-02T14:01:20.342 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2023-07-02T14:01:20.342 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-02T14:01:20.342 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-02T14:01:20.343 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-02T14:01:20.343 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-02T14:01:20.364 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2023-07-02T14:01:20.366 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2023-07-02T14:01:20.366 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2023-07-02T14:01:20.445 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-07-02T14:01:20.516 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-07-02T14:01:20.581 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-07-02T14:01:20.582 INFO:tasks.ceph:role: osd.4 2023-07-02T14:01:20.582 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi022.front.sepia.ceph.com 2023-07-02T14:01:20.582 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-07-02T14:01:20.646 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-07-02T14:01:20.646 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-02T14:01:20.647 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-02T14:01:20.647 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 2023-07-02T14:01:20.647 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-02T14:01:20.647 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2023-07-02T14:01:20.647 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-02T14:01:20.647 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-02T14:01:20.647 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-02T14:01:20.648 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-02T14:01:20.669 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2023-07-02T14:01:20.670 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2023-07-02T14:01:20.671 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2023-07-02T14:01:20.750 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-07-02T14:01:20.819 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-07-02T14:01:20.884 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-07-02T14:01:20.884 INFO:tasks.ceph:role: osd.5 2023-07-02T14:01:20.885 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi022.front.sepia.ceph.com 2023-07-02T14:01:20.885 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-07-02T14:01:20.948 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-07-02T14:01:20.949 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2023-07-02T14:01:20.949 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-07-02T14:01:20.949 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 2023-07-02T14:01:20.949 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-07-02T14:01:20.949 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2023-07-02T14:01:20.949 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-07-02T14:01:20.949 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-07-02T14:01:20.950 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-07-02T14:01:20.950 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-07-02T14:01:20.971 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2023-07-02T14:01:20.972 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2023-07-02T14:01:20.973 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2023-07-02T14:01:21.050 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-07-02T14:01:21.120 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-07-02T14:01:21.242 INFO:teuthology.orchestra.run.smithi022.stderr:2023-07-02T14:01:21.241+0000 7f3c18c87200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-07-02T14:01:21.243 INFO:teuthology.orchestra.run.smithi022.stderr:2023-07-02T14:01:21.241+0000 7f3c18c87200 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-07-02T14:01:21.243 INFO:teuthology.orchestra.run.smithi022.stderr:2023-07-02T14:01:21.241+0000 7f3c18c87200 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-07-02T14:01:21.243 INFO:teuthology.orchestra.run.smithi022.stderr:2023-07-02T14:01:21.241+0000 7f3c18c87200 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-07-02T14:01:21.243 INFO:teuthology.orchestra.run.smithi022.stderr:2023-07-02T14:01:21.241+0000 7f3c18c87200 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-07-02T14:01:21.243 INFO:teuthology.orchestra.run.smithi022.stderr:2023-07-02T14:01:21.241+0000 7f3c18c87200 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-07-02T14:01:24.084 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-07-02T14:01:24.115 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-07-02T14:01:24.202 INFO:teuthology.orchestra.run.smithi022.stderr:2023-07-02T14:01:24.201+0000 7f4ddd71e200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-07-02T14:01:24.203 INFO:teuthology.orchestra.run.smithi022.stderr:2023-07-02T14:01:24.201+0000 7f4ddd71e200 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-07-02T14:01:24.203 INFO:teuthology.orchestra.run.smithi022.stderr:2023-07-02T14:01:24.201+0000 7f4ddd71e200 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-07-02T14:01:24.203 INFO:teuthology.orchestra.run.smithi022.stderr:2023-07-02T14:01:24.201+0000 7f4ddd71e200 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-07-02T14:01:24.203 INFO:teuthology.orchestra.run.smithi022.stderr:2023-07-02T14:01:24.201+0000 7f4ddd71e200 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-07-02T14:01:24.203 INFO:teuthology.orchestra.run.smithi022.stderr:2023-07-02T14:01:24.201+0000 7f4ddd71e200 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-07-02T14:01:27.040 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-07-02T14:01:27.067 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-07-02T14:01:27.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-07-02T14:01:27.150+0000 7f9d8b802200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-07-02T14:01:27.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-07-02T14:01:27.150+0000 7f9d8b802200 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-07-02T14:01:27.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-07-02T14:01:27.150+0000 7f9d8b802200 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-07-02T14:01:27.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-07-02T14:01:27.150+0000 7f9d8b802200 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-07-02T14:01:27.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-07-02T14:01:27.150+0000 7f9d8b802200 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-07-02T14:01:27.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-07-02T14:01:27.151+0000 7f9d8b802200 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-07-02T14:01:29.998 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-07-02T14:01:30.069 INFO:tasks.ceph:Reading keys from all nodes... 2023-07-02T14:01:30.070 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-07-02T14:01:30.070 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-07-02T14:01:30.098 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-07-02T14:01:30.099 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-07-02T14:01:30.165 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-07-02T14:01:30.166 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-07-02T14:01:30.232 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-07-02T14:01:30.233 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-07-02T14:01:30.263 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-07-02T14:01:30.263 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-07-02T14:01:30.332 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-07-02T14:01:30.332 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-07-02T14:01:30.399 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-07-02T14:01:30.400 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-07-02T14:01:30.466 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-07-02T14:01:30.467 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-07-02T14:01:30.483 INFO:tasks.ceph:Adding keys to all mons... 2023-07-02T14:01:30.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo tee -a /etc/ceph/ceph.keyring 2023-07-02T14:01:30.486 DEBUG:teuthology.orchestra.run.smithi022:> sudo tee -a /etc/ceph/ceph.keyring 2023-07-02T14:01:30.510 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.0] 2023-07-02T14:01:30.511 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAng6FkRvwMHBAA3PgYcnBkbu/xJjnkn+fh5Q== 2023-07-02T14:01:30.511 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.1] 2023-07-02T14:01:30.511 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAqg6FkAFkWGhAAJ4ZSJUeiIFJUCJActQrQmw== 2023-07-02T14:01:30.511 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.2] 2023-07-02T14:01:30.511 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAtg6FkQwH7FhAAHfIkLS6S9pfpDQxwhGdsUQ== 2023-07-02T14:01:30.511 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr.x] 2023-07-02T14:01:30.511 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAmg6FkN1roGBAAj5liPmLJ1TnMYCMt0Th6XA== 2023-07-02T14:01:30.512 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.3] 2023-07-02T14:01:30.512 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAxg6FkdDpxDhAAOhSbJOPRsE+JfV+LDliWyw== 2023-07-02T14:01:30.512 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.4] 2023-07-02T14:01:30.512 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQA0g6FkajwTDBAA8T3m3eOWTSDe7xsR9K5u7w== 2023-07-02T14:01:30.512 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.5] 2023-07-02T14:01:30.512 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQA3g6Fk68wLCRAAi3GXWr3GbKDFByFmSGlSGw== 2023-07-02T14:01:30.512 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2023-07-02T14:01:30.513 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAmg6FkzlsRHRAAW3XQktKAOpzQoV2J3OZENw== 2023-07-02T14:01:30.532 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.0] 2023-07-02T14:01:30.532 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQAng6FkRvwMHBAA3PgYcnBkbu/xJjnkn+fh5Q== 2023-07-02T14:01:30.532 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.1] 2023-07-02T14:01:30.532 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQAqg6FkAFkWGhAAJ4ZSJUeiIFJUCJActQrQmw== 2023-07-02T14:01:30.532 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.2] 2023-07-02T14:01:30.533 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQAtg6FkQwH7FhAAHfIkLS6S9pfpDQxwhGdsUQ== 2023-07-02T14:01:30.533 INFO:teuthology.orchestra.run.smithi022.stdout:[mgr.x] 2023-07-02T14:01:30.533 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQAmg6FkN1roGBAAj5liPmLJ1TnMYCMt0Th6XA== 2023-07-02T14:01:30.533 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.3] 2023-07-02T14:01:30.533 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQAxg6FkdDpxDhAAOhSbJOPRsE+JfV+LDliWyw== 2023-07-02T14:01:30.533 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.4] 2023-07-02T14:01:30.533 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQA0g6FkajwTDBAA8T3m3eOWTSDe7xsR9K5u7w== 2023-07-02T14:01:30.533 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.5] 2023-07-02T14:01:30.534 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQA3g6Fk68wLCRAAi3GXWr3GbKDFByFmSGlSGw== 2023-07-02T14:01:30.534 INFO:teuthology.orchestra.run.smithi022.stdout:[client.0] 2023-07-02T14:01:30.534 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQAmg6FkzlsRHRAAW3XQktKAOpzQoV2J3OZENw== 2023-07-02T14:01:30.535 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-02T14:01:30.556 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-02T14:01:30.674 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-02T14:01:30.676 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-02T14:01:30.850 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-02T14:01:30.853 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-02T14:01:30.945 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-07-02T14:01:30.947 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-07-02T14:01:31.051 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-02T14:01:31.053 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-02T14:01:31.136 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-02T14:01:31.138 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-02T14:01:31.220 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-02T14:01:31.222 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-07-02T14:01:31.383 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-07-02T14:01:31.386 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-07-02T14:01:31.437 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-07-02T14:01:31.438 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-07-02T14:01:31.463 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-07-02T14:01:31.555 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-07-02T14:01:31.579 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-07-02T14:01:31.644 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-07-02T14:01:31.736 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-07-02T14:01:31.761 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-07-02T14:01:31.786 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-07-02T14:01:31.879 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-07-02T14:01:31.904 DEBUG:teuthology.orchestra.run.smithi016:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-07-02T14:01:31.906 DEBUG:teuthology.orchestra.run.smithi022:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-07-02T14:01:31.958 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-07-02T14:01:31.959 INFO:tasks.ceph.mon.a:Restarting daemon 2023-07-02T14:01:31.959 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-07-02T14:01:31.961 INFO:tasks.ceph.mon.a:Started 2023-07-02T14:01:31.962 INFO:tasks.ceph.mon.c:Restarting daemon 2023-07-02T14:01:31.962 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-07-02T14:01:31.963 INFO:tasks.ceph.mon.c:Started 2023-07-02T14:01:31.964 INFO:tasks.ceph.mon.b:Restarting daemon 2023-07-02T14:01:31.964 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-07-02T14:01:32.001 INFO:tasks.ceph.mon.b:Started 2023-07-02T14:01:32.001 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-07-02T14:01:32.001 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-07-02T14:01:32.002 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-07-02T14:01:32.004 INFO:tasks.ceph.mgr.x:Started 2023-07-02T14:01:32.004 INFO:tasks.ceph:Setting crush tunables to default 2023-07-02T14:01:32.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd crush tunables default 2023-07-02T14:01:37.398 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2023-07-02T14:01:37.410 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-07-02T14:01:37.410 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-07-02T14:01:37.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-07-02T14:01:37.436 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-07-02T14:01:37.437 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-07-02T14:01:37.456 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:37.455+0000 7f101451f680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-02T14:01:37.501 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2023-07-02T14:01:37.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-07-02T14:01:37.529 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:37.528+0000 7f101451f680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-02T14:01:37.566 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-07-02T14:01:37.567 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-07-02T14:01:37.592 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-07-02T14:01:37.593 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-07-02T14:01:37.658 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-07-02T14:01:37.658 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-07-02T14:01:37.723 DEBUG:teuthology.orchestra.run.smithi022:> sudo ceph --cluster ceph osd new 43c446fe-c811-4ee3-99e8-d7452edfc1c1 0 2023-07-02T14:01:37.970 INFO:teuthology.orchestra.run.smithi022.stdout:0 2023-07-02T14:01:37.983 DEBUG:teuthology.orchestra.run.smithi022:> sudo ceph --cluster ceph osd new 4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b 1 2023-07-02T14:01:38.038 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:38.037+0000 7f101451f680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-02T14:01:38.196 INFO:teuthology.orchestra.run.smithi022.stdout:1 2023-07-02T14:01:38.208 DEBUG:teuthology.orchestra.run.smithi022:> sudo ceph --cluster ceph osd new 99f8ea0a-5a71-4abe-b087-8e891fd11d1b 2 2023-07-02T14:01:38.420 INFO:teuthology.orchestra.run.smithi022.stdout:2 2023-07-02T14:01:38.433 DEBUG:teuthology.orchestra.run.smithi022:> sudo ceph --cluster ceph osd new d285b4d5-0bef-421a-b84a-943e485b10c2 3 2023-07-02T14:01:38.640 INFO:teuthology.orchestra.run.smithi022.stdout:3 2023-07-02T14:01:38.653 DEBUG:teuthology.orchestra.run.smithi022:> sudo ceph --cluster ceph osd new 7ba0b0db-e407-4c63-8278-aab09f3082c4 4 2023-07-02T14:01:38.834 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:38.833+0000 7f101451f680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-02T14:01:38.905 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:38.904+0000 7f101451f680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-02T14:01:38.907 INFO:teuthology.orchestra.run.smithi022.stdout:4 2023-07-02T14:01:38.920 DEBUG:teuthology.orchestra.run.smithi022:> sudo ceph --cluster ceph osd new 29663bfa-f6fe-4ce7-b45a-a9b7b4424c42 5 2023-07-02T14:01:39.106 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:39.105+0000 7f101451f680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-02T14:01:39.131 INFO:teuthology.orchestra.run.smithi022.stdout:5 2023-07-02T14:01:39.143 INFO:tasks.ceph.osd.0:Restarting daemon 2023-07-02T14:01:39.143 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-07-02T14:01:39.146 INFO:tasks.ceph.osd.0:Started 2023-07-02T14:01:39.146 INFO:tasks.ceph.osd.1:Restarting daemon 2023-07-02T14:01:39.146 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-07-02T14:01:39.148 INFO:tasks.ceph.osd.1:Started 2023-07-02T14:01:39.149 INFO:tasks.ceph.osd.2:Restarting daemon 2023-07-02T14:01:39.149 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-07-02T14:01:39.151 INFO:tasks.ceph.osd.2:Started 2023-07-02T14:01:39.151 INFO:tasks.ceph.osd.3:Restarting daemon 2023-07-02T14:01:39.151 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-07-02T14:01:39.153 INFO:tasks.ceph.osd.3:Started 2023-07-02T14:01:39.153 INFO:tasks.ceph.osd.4:Restarting daemon 2023-07-02T14:01:39.154 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-07-02T14:01:39.155 INFO:tasks.ceph.osd.4:Started 2023-07-02T14:01:39.156 INFO:tasks.ceph.osd.5:Restarting daemon 2023-07-02T14:01:39.156 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-07-02T14:01:39.158 INFO:tasks.ceph.osd.5:Started 2023-07-02T14:01:39.158 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-02T14:01:39.368 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:39.369 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"fsid":"9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2","created":"2023-07-02T14:01:37.298060+0000","modified":"2023-07-02T14:01:39.129121+0000","last_up_change":"0.000000","last_in_change":"2023-07-02T14:01:39.129121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"43c446fe-c811-4ee3-99e8-d7452edfc1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99f8ea0a-5a71-4abe-b087-8e891fd11d1b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d285b4d5-0bef-421a-b84a-943e485b10c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7ba0b0db-e407-4c63-8278-aab09f3082c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"29663bfa-f6fe-4ce7-b45a-a9b7b4424c42","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-02T14:01:39.381 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-07-02T14:01:39.381 INFO:tasks.ceph:Waiting for OSDs to come up 2023-07-02T14:01:39.499 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:01:39.497+0000 7f12259b1200 -1 Falling back to public interface 2023-07-02T14:01:39.574 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:39.573+0000 7f101451f680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-02T14:01:39.682 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:39.681+0000 7f101451f680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-02T14:01:39.750 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:01:39.748+0000 7fee7fd37200 -1 Falling back to public interface 2023-07-02T14:01:39.752 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:01:39.751+0000 7fee8c264200 -1 Falling back to public interface 2023-07-02T14:01:39.753 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:01:39.752+0000 7f16e7f31200 -1 Falling back to public interface 2023-07-02T14:01:39.754 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:01:39.753+0000 7f0839526200 -1 Falling back to public interface 2023-07-02T14:01:39.758 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:01:39.757+0000 7fd93f524200 -1 Falling back to public interface 2023-07-02T14:01:39.842 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:39.841+0000 7f101451f680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-02T14:01:39.923 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:39.922+0000 7f101451f680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-02T14:01:39.985 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-07-02T14:01:39.996 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:39.995+0000 7f101451f680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-02T14:01:40.175 INFO:teuthology.misc.health.smithi016.stdout: 2023-07-02T14:01:40.175 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":8,"fsid":"9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2","created":"2023-07-02T14:01:37.298060+0000","modified":"2023-07-02T14:01:39.129121+0000","last_up_change":"0.000000","last_in_change":"2023-07-02T14:01:39.129121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"43c446fe-c811-4ee3-99e8-d7452edfc1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"99f8ea0a-5a71-4abe-b087-8e891fd11d1b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d285b4d5-0bef-421a-b84a-943e485b10c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7ba0b0db-e407-4c63-8278-aab09f3082c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"29663bfa-f6fe-4ce7-b45a-a9b7b4424c42","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-02T14:01:40.186 DEBUG:teuthology.misc:0 of 6 OSDs are up 2023-07-02T14:01:40.274 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:40.273+0000 7f101451f680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-02T14:01:40.477 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:40.476+0000 7f101451f680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-02T14:01:41.176 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:41.175+0000 7f101451f680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-02T14:01:41.248 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:41.247+0000 7f101451f680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-02T14:01:41.330 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:41.329+0000 7f101451f680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-02T14:01:41.485 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:41.484+0000 7f101451f680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-02T14:01:41.559 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:41.558+0000 7f101451f680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-02T14:01:41.682 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:41.681+0000 7f101451f680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-02T14:01:41.794 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:41.793+0000 7f101451f680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-02T14:01:42.128 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:42.127+0000 7f101451f680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-02T14:01:42.210 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:42.210+0000 7f101451f680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-02T14:01:42.990 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:01:42.988+0000 7f12259b1200 -1 osd.0 0 log_to_monitors {default=true} 2023-07-02T14:01:43.217 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:01:43.216+0000 7f16e7f31200 -1 osd.3 0 log_to_monitors {default=true} 2023-07-02T14:01:43.219 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:01:43.218+0000 7fee7fd37200 -1 osd.1 0 log_to_monitors {default=true} 2023-07-02T14:01:43.234 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:43.233+0000 7f0fca767700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-07-02T14:01:43.234 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-07-02T14:01:43.233+0000 7f0fca767700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-07-02T14:01:43.248 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:01:43.248+0000 7fee8c264200 -1 osd.4 0 log_to_monitors {default=true} 2023-07-02T14:01:43.256 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:01:43.255+0000 7f0839526200 -1 osd.2 0 log_to_monitors {default=true} 2023-07-02T14:01:43.257 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:01:43.257+0000 7fd93f524200 -1 osd.5 0 log_to_monitors {default=true} 2023-07-02T14:01:44.276 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:01:44.275+0000 7f16de8ee700 -1 osd.3 0 waiting for initial osdmap 2023-07-02T14:01:44.277 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:01:44.276+0000 7fee82c21700 -1 osd.4 0 waiting for initial osdmap 2023-07-02T14:01:44.277 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:01:44.276+0000 7fd935ee1700 -1 osd.5 0 waiting for initial osdmap 2023-07-02T14:01:44.278 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:01:44.275+0000 7fee766f4700 -1 osd.1 0 waiting for initial osdmap 2023-07-02T14:01:44.278 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:01:44.275+0000 7f121c36e700 -1 osd.0 0 waiting for initial osdmap 2023-07-02T14:01:44.278 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:01:44.276+0000 7f082fee3700 -1 osd.2 0 waiting for initial osdmap 2023-07-02T14:01:44.281 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:01:44.281+0000 7f16db06a700 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-02T14:01:44.282 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:01:44.281+0000 7fd93265d700 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-02T14:01:44.282 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:01:44.281+0000 7fee7f39d700 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-02T14:01:44.282 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:01:44.280+0000 7f1218aea700 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-02T14:01:44.283 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:01:44.281+0000 7fee72e70700 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-02T14:01:44.283 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:01:44.281+0000 7f082c65f700 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-02T14:01:46.791 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-07-02T14:01:47.096 INFO:teuthology.misc.health.smithi016.stdout: 2023-07-02T14:01:47.096 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":12,"fsid":"9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2","created":"2023-07-02T14:01:37.298060+0000","modified":"2023-07-02T14:01:46.277830+0000","last_up_change":"2023-07-02T14:01:45.271760+0000","last_in_change":"2023-07-02T14:01:39.129121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-07-02T14:01:45.323455+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"43c446fe-c811-4ee3-99e8-d7452edfc1c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":83359},{"type":"v1","addr":"172.21.15.16:6801","nonce":83359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":83359},{"type":"v1","addr":"172.21.15.16:6803","nonce":83359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":83359},{"type":"v1","addr":"172.21.15.16:6807","nonce":83359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":83359},{"type":"v1","addr":"172.21.15.16:6805","nonce":83359}]},"public_addr":"172.21.15.16:6801/83359","cluster_addr":"172.21.15.16:6803/83359","heartbeat_back_addr":"172.21.15.16:6807/83359","heartbeat_front_addr":"172.21.15.16:6805/83359","state":["exists","up"]},{"osd":1,"uuid":"4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":83360},{"type":"v1","addr":"172.21.15.16:6809","nonce":83360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":83360},{"type":"v1","addr":"172.21.15.16:6811","nonce":83360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":83360},{"type":"v1","addr":"172.21.15.16:6815","nonce":83360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":83360},{"type":"v1","addr":"172.21.15.16:6813","nonce":83360}]},"public_addr":"172.21.15.16:6809/83360","cluster_addr":"172.21.15.16:6811/83360","heartbeat_back_addr":"172.21.15.16:6815/83360","heartbeat_front_addr":"172.21.15.16:6813/83360","state":["exists","up"]},{"osd":2,"uuid":"99f8ea0a-5a71-4abe-b087-8e891fd11d1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":83361},{"type":"v1","addr":"172.21.15.16:6817","nonce":83361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":83361},{"type":"v1","addr":"172.21.15.16:6819","nonce":83361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":83361},{"type":"v1","addr":"172.21.15.16:6823","nonce":83361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":83361},{"type":"v1","addr":"172.21.15.16:6821","nonce":83361}]},"public_addr":"172.21.15.16:6817/83361","cluster_addr":"172.21.15.16:6819/83361","heartbeat_back_addr":"172.21.15.16:6823/83361","heartbeat_front_addr":"172.21.15.16:6821/83361","state":["exists","up"]},{"osd":3,"uuid":"d285b4d5-0bef-421a-b84a-943e485b10c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":83350},{"type":"v1","addr":"172.21.15.22:6809","nonce":83350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":83350},{"type":"v1","addr":"172.21.15.22:6811","nonce":83350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":83350},{"type":"v1","addr":"172.21.15.22:6815","nonce":83350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":83350},{"type":"v1","addr":"172.21.15.22:6813","nonce":83350}]},"public_addr":"172.21.15.22:6809/83350","cluster_addr":"172.21.15.22:6811/83350","heartbeat_back_addr":"172.21.15.22:6815/83350","heartbeat_front_addr":"172.21.15.22:6813/83350","state":["exists","up"]},{"osd":4,"uuid":"7ba0b0db-e407-4c63-8278-aab09f3082c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":83351},{"type":"v1","addr":"172.21.15.22:6801","nonce":83351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":83351},{"type":"v1","addr":"172.21.15.22:6803","nonce":83351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":83351},{"type":"v1","addr":"172.21.15.22:6807","nonce":83351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":83351},{"type":"v1","addr":"172.21.15.22:6805","nonce":83351}]},"public_addr":"172.21.15.22:6801/83351","cluster_addr":"172.21.15.22:6803/83351","heartbeat_back_addr":"172.21.15.22:6807/83351","heartbeat_front_addr":"172.21.15.22:6805/83351","state":["exists","up"]},{"osd":5,"uuid":"29663bfa-f6fe-4ce7-b45a-a9b7b4424c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":83352},{"type":"v1","addr":"172.21.15.22:6817","nonce":83352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":83352},{"type":"v1","addr":"172.21.15.22:6819","nonce":83352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":83352},{"type":"v1","addr":"172.21.15.22:6823","nonce":83352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":83352},{"type":"v1","addr":"172.21.15.22:6821","nonce":83352}]},"public_addr":"172.21.15.22:6817/83352","cluster_addr":"172.21.15.22:6819/83352","heartbeat_back_addr":"172.21.15.22:6823/83352","heartbeat_front_addr":"172.21.15.22:6821/83352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.001178+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.184181+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.274854+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.186217+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.220075+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-02T14:01:47.108 DEBUG:teuthology.misc:6 of 6 OSDs are up 2023-07-02T14:01:47.108 INFO:tasks.ceph:Creating RBD pool 2023-07-02T14:01:47.109 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd pool create rbd 8 2023-07-02T14:01:48.294 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'rbd' created 2023-07-02T14:01:48.308 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-07-02T14:01:49.296 INFO:teuthology.orchestra.run.smithi016.stderr:enabled application 'rbd' on pool 'rbd' 2023-07-02T14:01:49.310 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-07-02T14:01:49.310 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-07-02T14:01:49.311 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-07-02T14:01:49.311 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-07-02T14:01:49.311 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-07-02T14:01:49.664 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:49.676 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":82920},{"type":"v1","addr":"172.21.15.22:6825","nonce":82920}]},"active_addr":"172.21.15.22:6825/82920","active_change":"2023-07-02T14:01:42.224927+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":413342397}]},{"addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":1039605688}]}]} 2023-07-02T14:01:49.678 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-07-02T14:01:49.679 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-07-02T14:01:49.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-02T14:01:49.981 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:49.981 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":15,"fsid":"9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2","created":"2023-07-02T14:01:37.298060+0000","modified":"2023-07-02T14:01:49.287069+0000","last_up_change":"2023-07-02T14:01:45.271760+0000","last_in_change":"2023-07-02T14:01:39.129121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-07-02T14:01:45.323455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-02T14:01:47.448897+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"43c446fe-c811-4ee3-99e8-d7452edfc1c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":83359},{"type":"v1","addr":"172.21.15.16:6801","nonce":83359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":83359},{"type":"v1","addr":"172.21.15.16:6803","nonce":83359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":83359},{"type":"v1","addr":"172.21.15.16:6807","nonce":83359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":83359},{"type":"v1","addr":"172.21.15.16:6805","nonce":83359}]},"public_addr":"172.21.15.16:6801/83359","cluster_addr":"172.21.15.16:6803/83359","heartbeat_back_addr":"172.21.15.16:6807/83359","heartbeat_front_addr":"172.21.15.16:6805/83359","state":["exists","up"]},{"osd":1,"uuid":"4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":83360},{"type":"v1","addr":"172.21.15.16:6809","nonce":83360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":83360},{"type":"v1","addr":"172.21.15.16:6811","nonce":83360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":83360},{"type":"v1","addr":"172.21.15.16:6815","nonce":83360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":83360},{"type":"v1","addr":"172.21.15.16:6813","nonce":83360}]},"public_addr":"172.21.15.16:6809/83360","cluster_addr":"172.21.15.16:6811/83360","heartbeat_back_addr":"172.21.15.16:6815/83360","heartbeat_front_addr":"172.21.15.16:6813/83360","state":["exists","up"]},{"osd":2,"uuid":"99f8ea0a-5a71-4abe-b087-8e891fd11d1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":83361},{"type":"v1","addr":"172.21.15.16:6817","nonce":83361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":83361},{"type":"v1","addr":"172.21.15.16:6819","nonce":83361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":83361},{"type":"v1","addr":"172.21.15.16:6823","nonce":83361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":83361},{"type":"v1","addr":"172.21.15.16:6821","nonce":83361}]},"public_addr":"172.21.15.16:6817/83361","cluster_addr":"172.21.15.16:6819/83361","heartbeat_back_addr":"172.21.15.16:6823/83361","heartbeat_front_addr":"172.21.15.16:6821/83361","state":["exists","up"]},{"osd":3,"uuid":"d285b4d5-0bef-421a-b84a-943e485b10c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":83350},{"type":"v1","addr":"172.21.15.22:6809","nonce":83350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":83350},{"type":"v1","addr":"172.21.15.22:6811","nonce":83350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":83350},{"type":"v1","addr":"172.21.15.22:6815","nonce":83350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":83350},{"type":"v1","addr":"172.21.15.22:6813","nonce":83350}]},"public_addr":"172.21.15.22:6809/83350","cluster_addr":"172.21.15.22:6811/83350","heartbeat_back_addr":"172.21.15.22:6815/83350","heartbeat_front_addr":"172.21.15.22:6813/83350","state":["exists","up"]},{"osd":4,"uuid":"7ba0b0db-e407-4c63-8278-aab09f3082c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":83351},{"type":"v1","addr":"172.21.15.22:6801","nonce":83351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":83351},{"type":"v1","addr":"172.21.15.22:6803","nonce":83351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":83351},{"type":"v1","addr":"172.21.15.22:6807","nonce":83351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":83351},{"type":"v1","addr":"172.21.15.22:6805","nonce":83351}]},"public_addr":"172.21.15.22:6801/83351","cluster_addr":"172.21.15.22:6803/83351","heartbeat_back_addr":"172.21.15.22:6807/83351","heartbeat_front_addr":"172.21.15.22:6805/83351","state":["exists","up"]},{"osd":5,"uuid":"29663bfa-f6fe-4ce7-b45a-a9b7b4424c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":83352},{"type":"v1","addr":"172.21.15.22:6817","nonce":83352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":83352},{"type":"v1","addr":"172.21.15.22:6819","nonce":83352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":83352},{"type":"v1","addr":"172.21.15.22:6823","nonce":83352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":83352},{"type":"v1","addr":"172.21.15.22:6821","nonce":83352}]},"public_addr":"172.21.15.22:6817/83352","cluster_addr":"172.21.15.22:6819/83352","heartbeat_back_addr":"172.21.15.22:6823/83352","heartbeat_front_addr":"172.21.15.22:6821/83352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.001178+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.184181+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.274854+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.186217+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.247442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.220075+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-02T14:01:49.994 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-07-02T14:01:49.994 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-02T14:01:50.302 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:50.302 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":16,"fsid":"9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2","created":"2023-07-02T14:01:37.298060+0000","modified":"2023-07-02T14:01:50.288411+0000","last_up_change":"2023-07-02T14:01:45.271760+0000","last_in_change":"2023-07-02T14:01:39.129121+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-07-02T14:01:45.323455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-02T14:01:47.448897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"43c446fe-c811-4ee3-99e8-d7452edfc1c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":83359},{"type":"v1","addr":"172.21.15.16:6801","nonce":83359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":83359},{"type":"v1","addr":"172.21.15.16:6803","nonce":83359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":83359},{"type":"v1","addr":"172.21.15.16:6807","nonce":83359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":83359},{"type":"v1","addr":"172.21.15.16:6805","nonce":83359}]},"public_addr":"172.21.15.16:6801/83359","cluster_addr":"172.21.15.16:6803/83359","heartbeat_back_addr":"172.21.15.16:6807/83359","heartbeat_front_addr":"172.21.15.16:6805/83359","state":["exists","up"]},{"osd":1,"uuid":"4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":83360},{"type":"v1","addr":"172.21.15.16:6809","nonce":83360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":83360},{"type":"v1","addr":"172.21.15.16:6811","nonce":83360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":83360},{"type":"v1","addr":"172.21.15.16:6815","nonce":83360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":83360},{"type":"v1","addr":"172.21.15.16:6813","nonce":83360}]},"public_addr":"172.21.15.16:6809/83360","cluster_addr":"172.21.15.16:6811/83360","heartbeat_back_addr":"172.21.15.16:6815/83360","heartbeat_front_addr":"172.21.15.16:6813/83360","state":["exists","up"]},{"osd":2,"uuid":"99f8ea0a-5a71-4abe-b087-8e891fd11d1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":83361},{"type":"v1","addr":"172.21.15.16:6817","nonce":83361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":83361},{"type":"v1","addr":"172.21.15.16:6819","nonce":83361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":83361},{"type":"v1","addr":"172.21.15.16:6823","nonce":83361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":83361},{"type":"v1","addr":"172.21.15.16:6821","nonce":83361}]},"public_addr":"172.21.15.16:6817/83361","cluster_addr":"172.21.15.16:6819/83361","heartbeat_back_addr":"172.21.15.16:6823/83361","heartbeat_front_addr":"172.21.15.16:6821/83361","state":["exists","up"]},{"osd":3,"uuid":"d285b4d5-0bef-421a-b84a-943e485b10c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":83350},{"type":"v1","addr":"172.21.15.22:6809","nonce":83350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":83350},{"type":"v1","addr":"172.21.15.22:6811","nonce":83350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":83350},{"type":"v1","addr":"172.21.15.22:6815","nonce":83350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":83350},{"type":"v1","addr":"172.21.15.22:6813","nonce":83350}]},"public_addr":"172.21.15.22:6809/83350","cluster_addr":"172.21.15.22:6811/83350","heartbeat_back_addr":"172.21.15.22:6815/83350","heartbeat_front_addr":"172.21.15.22:6813/83350","state":["exists","up"]},{"osd":4,"uuid":"7ba0b0db-e407-4c63-8278-aab09f3082c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":83351},{"type":"v1","addr":"172.21.15.22:6801","nonce":83351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":83351},{"type":"v1","addr":"172.21.15.22:6803","nonce":83351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":83351},{"type":"v1","addr":"172.21.15.22:6807","nonce":83351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":83351},{"type":"v1","addr":"172.21.15.22:6805","nonce":83351}]},"public_addr":"172.21.15.22:6801/83351","cluster_addr":"172.21.15.22:6803/83351","heartbeat_back_addr":"172.21.15.22:6807/83351","heartbeat_front_addr":"172.21.15.22:6805/83351","state":["exists","up"]},{"osd":5,"uuid":"29663bfa-f6fe-4ce7-b45a-a9b7b4424c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":83352},{"type":"v1","addr":"172.21.15.22:6817","nonce":83352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":83352},{"type":"v1","addr":"172.21.15.22:6819","nonce":83352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":83352},{"type":"v1","addr":"172.21.15.22:6823","nonce":83352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":83352},{"type":"v1","addr":"172.21.15.22:6821","nonce":83352}]},"public_addr":"172.21.15.22:6817/83352","cluster_addr":"172.21.15.22:6819/83352","heartbeat_back_addr":"172.21.15.22:6823/83352","heartbeat_front_addr":"172.21.15.22:6821/83352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.001178+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.184181+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.274854+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.186217+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.247442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.220075+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-02T14:01:50.315 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-07-02T14:01:50.464 INFO:teuthology.orchestra.run.smithi016.stdout:47244640259 2023-07-02T14:01:50.465 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-07-02T14:01:50.614 INFO:teuthology.orchestra.run.smithi016.stdout:47244640259 2023-07-02T14:01:50.615 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-07-02T14:01:50.764 INFO:teuthology.orchestra.run.smithi016.stdout:47244640259 2023-07-02T14:01:50.764 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-07-02T14:01:50.915 INFO:teuthology.orchestra.run.smithi016.stdout:47244640259 2023-07-02T14:01:50.915 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-07-02T14:01:51.065 INFO:teuthology.orchestra.run.smithi016.stdout:47244640259 2023-07-02T14:01:51.066 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-07-02T14:01:51.216 INFO:teuthology.orchestra.run.smithi016.stdout:47244640259 2023-07-02T14:01:51.216 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-07-02T14:01:51.520 INFO:teuthology.orchestra.run.smithi016.stdout:47244640257 2023-07-02T14:01:51.534 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.0 2023-07-02T14:01:52.535 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-07-02T14:01:52.846 INFO:teuthology.orchestra.run.smithi016.stdout:47244640257 2023-07-02T14:01:52.859 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.0 2023-07-02T14:01:53.860 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-07-02T14:01:54.174 INFO:teuthology.orchestra.run.smithi016.stdout:47244640259 2023-07-02T14:01:54.188 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.0 2023-07-02T14:01:54.188 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T14:01:54.491 INFO:teuthology.orchestra.run.smithi016.stdout:47244640259 2023-07-02T14:01:54.504 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.1 2023-07-02T14:01:54.504 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-07-02T14:01:54.807 INFO:teuthology.orchestra.run.smithi016.stdout:47244640259 2023-07-02T14:01:54.821 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.2 2023-07-02T14:01:54.821 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-07-02T14:01:55.127 INFO:teuthology.orchestra.run.smithi016.stdout:47244640259 2023-07-02T14:01:55.141 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.3 2023-07-02T14:01:55.141 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-07-02T14:01:55.442 INFO:teuthology.orchestra.run.smithi016.stdout:47244640259 2023-07-02T14:01:55.456 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.4 2023-07-02T14:01:55.456 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-07-02T14:01:55.756 INFO:teuthology.orchestra.run.smithi016.stdout:47244640259 2023-07-02T14:01:55.769 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.5 2023-07-02T14:01:55.769 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-07-02T14:01:55.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:01:56.066 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:56.068 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:01:56.080 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-07-02T14:01:54.232582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":29560,"kb_used_data":840,"kb_used_omap":0,"kb_used_meta":28672,"kb_avail":566201480,"statfs":{"total":579820584960,"available":579790315520,"internally_reserved":0,"allocated":860160,"data_stored":269058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29360128},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.555585+0000","last_change":"2023-07-02T14:01:49.554924+0000","last_active":"2023-07-02T14:01:49.555585+0000","last_peered":"2023-07-02T14:01:49.555585+0000","last_clean":"2023-07-02T14:01:49.555585+0000","last_became_active":"2023-07-02T14:01:49.554578+0000","last_became_peered":"2023-07-02T14:01:49.554578+0000","last_unstale":"2023-07-02T14:01:49.555585+0000","last_undegraded":"2023-07-02T14:01:49.555585+0000","last_fullsized":"2023-07-02T14:01:49.555585+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.297777+0000","last_change":"2023-07-02T14:01:49.295075+0000","last_active":"2023-07-02T14:01:49.297777+0000","last_peered":"2023-07-02T14:01:49.297777+0000","last_clean":"2023-07-02T14:01:49.297777+0000","last_became_active":"2023-07-02T14:01:49.293782+0000","last_became_peered":"2023-07-02T14:01:49.293782+0000","last_unstale":"2023-07-02T14:01:49.297777+0000","last_undegraded":"2023-07-02T14:01:49.297777+0000","last_fullsized":"2023-07-02T14:01:49.297777+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.294236+0000","last_change":"2023-07-02T14:01:49.293139+0000","last_active":"2023-07-02T14:01:49.294236+0000","last_peered":"2023-07-02T14:01:49.294236+0000","last_clean":"2023-07-02T14:01:49.294236+0000","last_became_active":"2023-07-02T14:01:49.292790+0000","last_became_peered":"2023-07-02T14:01:49.292790+0000","last_unstale":"2023-07-02T14:01:49.294236+0000","last_undegraded":"2023-07-02T14:01:49.294236+0000","last_fullsized":"2023-07-02T14:01:49.294236+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.294648+0000","last_change":"2023-07-02T14:01:49.293994+0000","last_active":"2023-07-02T14:01:49.294648+0000","last_peered":"2023-07-02T14:01:49.294648+0000","last_clean":"2023-07-02T14:01:49.294648+0000","last_became_active":"2023-07-02T14:01:49.293608+0000","last_became_peered":"2023-07-02T14:01:49.293608+0000","last_unstale":"2023-07-02T14:01:49.294648+0000","last_undegraded":"2023-07-02T14:01:49.294648+0000","last_fullsized":"2023-07-02T14:01:49.294648+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.291083+0000","last_change":"2023-07-02T14:01:47.288970+0000","last_active":"2023-07-02T14:01:49.291083+0000","last_peered":"2023-07-02T14:01:49.291083+0000","last_clean":"2023-07-02T14:01:49.291083+0000","last_became_active":"2023-07-02T14:01:47.288610+0000","last_became_peered":"2023-07-02T14:01:47.288610+0000","last_unstale":"2023-07-02T14:01:49.291083+0000","last_undegraded":"2023-07-02T14:01:49.291083+0000","last_fullsized":"2023-07-02T14:01:49.291083+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.298964+0000","last_change":"2023-07-02T14:01:49.298280+0000","last_active":"2023-07-02T14:01:49.298964+0000","last_peered":"2023-07-02T14:01:49.298964+0000","last_clean":"2023-07-02T14:01:49.298964+0000","last_became_active":"2023-07-02T14:01:49.297946+0000","last_became_peered":"2023-07-02T14:01:49.297946+0000","last_unstale":"2023-07-02T14:01:49.298964+0000","last_undegraded":"2023-07-02T14:01:49.298964+0000","last_fullsized":"2023-07-02T14:01:49.298964+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.299410+0000","last_change":"2023-07-02T14:01:49.295243+0000","last_active":"2023-07-02T14:01:49.299410+0000","last_peered":"2023-07-02T14:01:49.299410+0000","last_clean":"2023-07-02T14:01:49.299410+0000","last_became_active":"2023-07-02T14:01:49.294964+0000","last_became_peered":"2023-07-02T14:01:49.294964+0000","last_unstale":"2023-07-02T14:01:49.299410+0000","last_undegraded":"2023-07-02T14:01:49.299410+0000","last_fullsized":"2023-07-02T14:01:49.299410+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.298982+0000","last_change":"2023-07-02T14:01:49.298253+0000","last_active":"2023-07-02T14:01:49.298982+0000","last_peered":"2023-07-02T14:01:49.298982+0000","last_clean":"2023-07-02T14:01:49.298982+0000","last_became_active":"2023-07-02T14:01:49.297965+0000","last_became_peered":"2023-07-02T14:01:49.297965+0000","last_unstale":"2023-07-02T14:01:49.298982+0000","last_undegraded":"2023-07-02T14:01:49.298982+0000","last_fullsized":"2023-07-02T14:01:49.298982+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.293782+0000","last_change":"2023-07-02T14:01:49.293092+0000","last_active":"2023-07-02T14:01:49.293782+0000","last_peered":"2023-07-02T14:01:49.293782+0000","last_clean":"2023-07-02T14:01:49.293782+0000","last_became_active":"2023-07-02T14:01:49.292773+0000","last_became_peered":"2023-07-02T14:01:49.292773+0000","last_unstale":"2023-07-02T14:01:49.293782+0000","last_undegraded":"2023-07-02T14:01:49.293782+0000","last_fullsized":"2023-07-02T14:01:49.293782+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4892,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94366948,"statfs":{"total":96636764160,"available":96631754752,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4880,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94366960,"statfs":{"total":96636764160,"available":96631767040,"internally_reserved":0,"allocated":139264,"data_stored":43198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4944,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366896,"statfs":{"total":96636764160,"available":96631701504,"internally_reserved":0,"allocated":139264,"data_stored":43198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4944,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366896,"statfs":{"total":96636764160,"available":96631701504,"internally_reserved":0,"allocated":139264,"data_stored":43198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4944,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366896,"statfs":{"total":96636764160,"available":96631701504,"internally_reserved":0,"allocated":139264,"data_stored":43198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:01:56.081 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:01:56.379 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:56.381 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:01:56.392 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-07-02T14:01:56.232972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":29672,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":28736,"kb_avail":566201368,"statfs":{"total":579820584960,"available":579790200832,"internally_reserved":0,"allocated":909312,"data_stored":288798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29425664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.555585+0000","last_change":"2023-07-02T14:01:49.554924+0000","last_active":"2023-07-02T14:01:49.555585+0000","last_peered":"2023-07-02T14:01:49.555585+0000","last_clean":"2023-07-02T14:01:49.555585+0000","last_became_active":"2023-07-02T14:01:49.554578+0000","last_became_peered":"2023-07-02T14:01:49.554578+0000","last_unstale":"2023-07-02T14:01:49.555585+0000","last_undegraded":"2023-07-02T14:01:49.555585+0000","last_fullsized":"2023-07-02T14:01:49.555585+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.297777+0000","last_change":"2023-07-02T14:01:49.295075+0000","last_active":"2023-07-02T14:01:49.297777+0000","last_peered":"2023-07-02T14:01:49.297777+0000","last_clean":"2023-07-02T14:01:49.297777+0000","last_became_active":"2023-07-02T14:01:49.293782+0000","last_became_peered":"2023-07-02T14:01:49.293782+0000","last_unstale":"2023-07-02T14:01:49.297777+0000","last_undegraded":"2023-07-02T14:01:49.297777+0000","last_fullsized":"2023-07-02T14:01:49.297777+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.294236+0000","last_change":"2023-07-02T14:01:49.293139+0000","last_active":"2023-07-02T14:01:49.294236+0000","last_peered":"2023-07-02T14:01:49.294236+0000","last_clean":"2023-07-02T14:01:49.294236+0000","last_became_active":"2023-07-02T14:01:49.292790+0000","last_became_peered":"2023-07-02T14:01:49.292790+0000","last_unstale":"2023-07-02T14:01:49.294236+0000","last_undegraded":"2023-07-02T14:01:49.294236+0000","last_fullsized":"2023-07-02T14:01:49.294236+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.294648+0000","last_change":"2023-07-02T14:01:49.293994+0000","last_active":"2023-07-02T14:01:49.294648+0000","last_peered":"2023-07-02T14:01:49.294648+0000","last_clean":"2023-07-02T14:01:49.294648+0000","last_became_active":"2023-07-02T14:01:49.293608+0000","last_became_peered":"2023-07-02T14:01:49.293608+0000","last_unstale":"2023-07-02T14:01:49.294648+0000","last_undegraded":"2023-07-02T14:01:49.294648+0000","last_fullsized":"2023-07-02T14:01:49.294648+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.291083+0000","last_change":"2023-07-02T14:01:47.288970+0000","last_active":"2023-07-02T14:01:49.291083+0000","last_peered":"2023-07-02T14:01:49.291083+0000","last_clean":"2023-07-02T14:01:49.291083+0000","last_became_active":"2023-07-02T14:01:47.288610+0000","last_became_peered":"2023-07-02T14:01:47.288610+0000","last_unstale":"2023-07-02T14:01:49.291083+0000","last_undegraded":"2023-07-02T14:01:49.291083+0000","last_fullsized":"2023-07-02T14:01:49.291083+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.298964+0000","last_change":"2023-07-02T14:01:49.298280+0000","last_active":"2023-07-02T14:01:49.298964+0000","last_peered":"2023-07-02T14:01:49.298964+0000","last_clean":"2023-07-02T14:01:49.298964+0000","last_became_active":"2023-07-02T14:01:49.297946+0000","last_became_peered":"2023-07-02T14:01:49.297946+0000","last_unstale":"2023-07-02T14:01:49.298964+0000","last_undegraded":"2023-07-02T14:01:49.298964+0000","last_fullsized":"2023-07-02T14:01:49.298964+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.299410+0000","last_change":"2023-07-02T14:01:49.295243+0000","last_active":"2023-07-02T14:01:49.299410+0000","last_peered":"2023-07-02T14:01:49.299410+0000","last_clean":"2023-07-02T14:01:49.299410+0000","last_became_active":"2023-07-02T14:01:49.294964+0000","last_became_peered":"2023-07-02T14:01:49.294964+0000","last_unstale":"2023-07-02T14:01:49.299410+0000","last_undegraded":"2023-07-02T14:01:49.299410+0000","last_fullsized":"2023-07-02T14:01:49.299410+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.298982+0000","last_change":"2023-07-02T14:01:49.298253+0000","last_active":"2023-07-02T14:01:49.298982+0000","last_peered":"2023-07-02T14:01:49.298982+0000","last_clean":"2023-07-02T14:01:49.298982+0000","last_became_active":"2023-07-02T14:01:49.297965+0000","last_became_peered":"2023-07-02T14:01:49.297965+0000","last_unstale":"2023-07-02T14:01:49.298982+0000","last_undegraded":"2023-07-02T14:01:49.298982+0000","last_fullsized":"2023-07-02T14:01:49.298982+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.293782+0000","last_change":"2023-07-02T14:01:49.293092+0000","last_active":"2023-07-02T14:01:49.293782+0000","last_peered":"2023-07-02T14:01:49.293782+0000","last_clean":"2023-07-02T14:01:49.293782+0000","last_became_active":"2023-07-02T14:01:49.292773+0000","last_became_peered":"2023-07-02T14:01:49.292773+0000","last_unstale":"2023-07-02T14:01:49.293782+0000","last_undegraded":"2023-07-02T14:01:49.293782+0000","last_fullsized":"2023-07-02T14:01:49.293782+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4892,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94366948,"statfs":{"total":96636764160,"available":96631754752,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:01:56.393 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-07-02T14:01:56.393 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-07-02T14:01:56.394 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-07-02T14:01:56.394 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-07-02T14:01:56.735 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:56.748 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-07-02T14:01:56.749 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-07-02T14:01:56.749 INFO:teuthology.run_tasks:Running task thrashosds... 2023-07-02T14:01:56.760 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2023-07-02T14:01:56.760 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'pacific'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request'], 'sha1': '8934282de75bfae9ed7c114e0bb6e46fc588d851'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'extra_system_packages': ['pv'], 'flavor': 'default', 'sha1': '8934282de75bfae9ed7c114e0bb6e46fc588d851'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-61616-pacific', 'sha1': 'cbd88633a9c19779ad4b2e367bdc17f83b3a065d'}} 2023-07-02T14:01:56.760 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2023-07-02T14:01:56.761 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2023-07-02T14:01:57.082 INFO:tasks.thrashosds:Beginning thrashosds... 2023-07-02T14:01:57.083 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-07-02T14:01:57.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:01:57.383 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:57.385 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:01:57.396 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-07-02T14:01:56.232972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":29672,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":28736,"kb_avail":566201368,"statfs":{"total":579820584960,"available":579790200832,"internally_reserved":0,"allocated":909312,"data_stored":288798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29425664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.555585+0000","last_change":"2023-07-02T14:01:49.554924+0000","last_active":"2023-07-02T14:01:49.555585+0000","last_peered":"2023-07-02T14:01:49.555585+0000","last_clean":"2023-07-02T14:01:49.555585+0000","last_became_active":"2023-07-02T14:01:49.554578+0000","last_became_peered":"2023-07-02T14:01:49.554578+0000","last_unstale":"2023-07-02T14:01:49.555585+0000","last_undegraded":"2023-07-02T14:01:49.555585+0000","last_fullsized":"2023-07-02T14:01:49.555585+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.297777+0000","last_change":"2023-07-02T14:01:49.295075+0000","last_active":"2023-07-02T14:01:49.297777+0000","last_peered":"2023-07-02T14:01:49.297777+0000","last_clean":"2023-07-02T14:01:49.297777+0000","last_became_active":"2023-07-02T14:01:49.293782+0000","last_became_peered":"2023-07-02T14:01:49.293782+0000","last_unstale":"2023-07-02T14:01:49.297777+0000","last_undegraded":"2023-07-02T14:01:49.297777+0000","last_fullsized":"2023-07-02T14:01:49.297777+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.294236+0000","last_change":"2023-07-02T14:01:49.293139+0000","last_active":"2023-07-02T14:01:49.294236+0000","last_peered":"2023-07-02T14:01:49.294236+0000","last_clean":"2023-07-02T14:01:49.294236+0000","last_became_active":"2023-07-02T14:01:49.292790+0000","last_became_peered":"2023-07-02T14:01:49.292790+0000","last_unstale":"2023-07-02T14:01:49.294236+0000","last_undegraded":"2023-07-02T14:01:49.294236+0000","last_fullsized":"2023-07-02T14:01:49.294236+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.294648+0000","last_change":"2023-07-02T14:01:49.293994+0000","last_active":"2023-07-02T14:01:49.294648+0000","last_peered":"2023-07-02T14:01:49.294648+0000","last_clean":"2023-07-02T14:01:49.294648+0000","last_became_active":"2023-07-02T14:01:49.293608+0000","last_became_peered":"2023-07-02T14:01:49.293608+0000","last_unstale":"2023-07-02T14:01:49.294648+0000","last_undegraded":"2023-07-02T14:01:49.294648+0000","last_fullsized":"2023-07-02T14:01:49.294648+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.291083+0000","last_change":"2023-07-02T14:01:47.288970+0000","last_active":"2023-07-02T14:01:49.291083+0000","last_peered":"2023-07-02T14:01:49.291083+0000","last_clean":"2023-07-02T14:01:49.291083+0000","last_became_active":"2023-07-02T14:01:47.288610+0000","last_became_peered":"2023-07-02T14:01:47.288610+0000","last_unstale":"2023-07-02T14:01:49.291083+0000","last_undegraded":"2023-07-02T14:01:49.291083+0000","last_fullsized":"2023-07-02T14:01:49.291083+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.298964+0000","last_change":"2023-07-02T14:01:49.298280+0000","last_active":"2023-07-02T14:01:49.298964+0000","last_peered":"2023-07-02T14:01:49.298964+0000","last_clean":"2023-07-02T14:01:49.298964+0000","last_became_active":"2023-07-02T14:01:49.297946+0000","last_became_peered":"2023-07-02T14:01:49.297946+0000","last_unstale":"2023-07-02T14:01:49.298964+0000","last_undegraded":"2023-07-02T14:01:49.298964+0000","last_fullsized":"2023-07-02T14:01:49.298964+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.299410+0000","last_change":"2023-07-02T14:01:49.295243+0000","last_active":"2023-07-02T14:01:49.299410+0000","last_peered":"2023-07-02T14:01:49.299410+0000","last_clean":"2023-07-02T14:01:49.299410+0000","last_became_active":"2023-07-02T14:01:49.294964+0000","last_became_peered":"2023-07-02T14:01:49.294964+0000","last_unstale":"2023-07-02T14:01:49.299410+0000","last_undegraded":"2023-07-02T14:01:49.299410+0000","last_fullsized":"2023-07-02T14:01:49.299410+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.298982+0000","last_change":"2023-07-02T14:01:49.298253+0000","last_active":"2023-07-02T14:01:49.298982+0000","last_peered":"2023-07-02T14:01:49.298982+0000","last_clean":"2023-07-02T14:01:49.298982+0000","last_became_active":"2023-07-02T14:01:49.297965+0000","last_became_peered":"2023-07-02T14:01:49.297965+0000","last_unstale":"2023-07-02T14:01:49.298982+0000","last_undegraded":"2023-07-02T14:01:49.298982+0000","last_fullsized":"2023-07-02T14:01:49.298982+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.293782+0000","last_change":"2023-07-02T14:01:49.293092+0000","last_active":"2023-07-02T14:01:49.293782+0000","last_peered":"2023-07-02T14:01:49.293782+0000","last_clean":"2023-07-02T14:01:49.293782+0000","last_became_active":"2023-07-02T14:01:49.292773+0000","last_became_peered":"2023-07-02T14:01:49.292773+0000","last_unstale":"2023-07-02T14:01:49.293782+0000","last_undegraded":"2023-07-02T14:01:49.293782+0000","last_fullsized":"2023-07-02T14:01:49.293782+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4892,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94366948,"statfs":{"total":96636764160,"available":96631754752,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:01:57.397 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:01:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:57.697 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:01:57.708 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-07-02T14:01:56.232972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":29672,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":28736,"kb_avail":566201368,"statfs":{"total":579820584960,"available":579790200832,"internally_reserved":0,"allocated":909312,"data_stored":288798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29425664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.555585+0000","last_change":"2023-07-02T14:01:49.554924+0000","last_active":"2023-07-02T14:01:49.555585+0000","last_peered":"2023-07-02T14:01:49.555585+0000","last_clean":"2023-07-02T14:01:49.555585+0000","last_became_active":"2023-07-02T14:01:49.554578+0000","last_became_peered":"2023-07-02T14:01:49.554578+0000","last_unstale":"2023-07-02T14:01:49.555585+0000","last_undegraded":"2023-07-02T14:01:49.555585+0000","last_fullsized":"2023-07-02T14:01:49.555585+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.297777+0000","last_change":"2023-07-02T14:01:49.295075+0000","last_active":"2023-07-02T14:01:49.297777+0000","last_peered":"2023-07-02T14:01:49.297777+0000","last_clean":"2023-07-02T14:01:49.297777+0000","last_became_active":"2023-07-02T14:01:49.293782+0000","last_became_peered":"2023-07-02T14:01:49.293782+0000","last_unstale":"2023-07-02T14:01:49.297777+0000","last_undegraded":"2023-07-02T14:01:49.297777+0000","last_fullsized":"2023-07-02T14:01:49.297777+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.294236+0000","last_change":"2023-07-02T14:01:49.293139+0000","last_active":"2023-07-02T14:01:49.294236+0000","last_peered":"2023-07-02T14:01:49.294236+0000","last_clean":"2023-07-02T14:01:49.294236+0000","last_became_active":"2023-07-02T14:01:49.292790+0000","last_became_peered":"2023-07-02T14:01:49.292790+0000","last_unstale":"2023-07-02T14:01:49.294236+0000","last_undegraded":"2023-07-02T14:01:49.294236+0000","last_fullsized":"2023-07-02T14:01:49.294236+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.294648+0000","last_change":"2023-07-02T14:01:49.293994+0000","last_active":"2023-07-02T14:01:49.294648+0000","last_peered":"2023-07-02T14:01:49.294648+0000","last_clean":"2023-07-02T14:01:49.294648+0000","last_became_active":"2023-07-02T14:01:49.293608+0000","last_became_peered":"2023-07-02T14:01:49.293608+0000","last_unstale":"2023-07-02T14:01:49.294648+0000","last_undegraded":"2023-07-02T14:01:49.294648+0000","last_fullsized":"2023-07-02T14:01:49.294648+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.291083+0000","last_change":"2023-07-02T14:01:47.288970+0000","last_active":"2023-07-02T14:01:49.291083+0000","last_peered":"2023-07-02T14:01:49.291083+0000","last_clean":"2023-07-02T14:01:49.291083+0000","last_became_active":"2023-07-02T14:01:47.288610+0000","last_became_peered":"2023-07-02T14:01:47.288610+0000","last_unstale":"2023-07-02T14:01:49.291083+0000","last_undegraded":"2023-07-02T14:01:49.291083+0000","last_fullsized":"2023-07-02T14:01:49.291083+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.298964+0000","last_change":"2023-07-02T14:01:49.298280+0000","last_active":"2023-07-02T14:01:49.298964+0000","last_peered":"2023-07-02T14:01:49.298964+0000","last_clean":"2023-07-02T14:01:49.298964+0000","last_became_active":"2023-07-02T14:01:49.297946+0000","last_became_peered":"2023-07-02T14:01:49.297946+0000","last_unstale":"2023-07-02T14:01:49.298964+0000","last_undegraded":"2023-07-02T14:01:49.298964+0000","last_fullsized":"2023-07-02T14:01:49.298964+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.299410+0000","last_change":"2023-07-02T14:01:49.295243+0000","last_active":"2023-07-02T14:01:49.299410+0000","last_peered":"2023-07-02T14:01:49.299410+0000","last_clean":"2023-07-02T14:01:49.299410+0000","last_became_active":"2023-07-02T14:01:49.294964+0000","last_became_peered":"2023-07-02T14:01:49.294964+0000","last_unstale":"2023-07-02T14:01:49.299410+0000","last_undegraded":"2023-07-02T14:01:49.299410+0000","last_fullsized":"2023-07-02T14:01:49.299410+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.298982+0000","last_change":"2023-07-02T14:01:49.298253+0000","last_active":"2023-07-02T14:01:49.298982+0000","last_peered":"2023-07-02T14:01:49.298982+0000","last_clean":"2023-07-02T14:01:49.298982+0000","last_became_active":"2023-07-02T14:01:49.297965+0000","last_became_peered":"2023-07-02T14:01:49.297965+0000","last_unstale":"2023-07-02T14:01:49.298982+0000","last_undegraded":"2023-07-02T14:01:49.298982+0000","last_fullsized":"2023-07-02T14:01:49.298982+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.293782+0000","last_change":"2023-07-02T14:01:49.293092+0000","last_active":"2023-07-02T14:01:49.293782+0000","last_peered":"2023-07-02T14:01:49.293782+0000","last_clean":"2023-07-02T14:01:49.293782+0000","last_became_active":"2023-07-02T14:01:49.292773+0000","last_became_peered":"2023-07-02T14:01:49.292773+0000","last_unstale":"2023-07-02T14:01:49.293782+0000","last_undegraded":"2023-07-02T14:01:49.293782+0000","last_fullsized":"2023-07-02T14:01:49.293782+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4892,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94366948,"statfs":{"total":96636764160,"available":96631754752,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:01:57.709 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-07-02T14:01:57.709 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2023-07-02T14:01:58.008 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 16 2023-07-02T14:01:58.009 INFO:teuthology.orchestra.run.smithi016.stdout:fsid 9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2 2023-07-02T14:01:58.009 INFO:teuthology.orchestra.run.smithi016.stdout:created 2023-07-02T14:01:37.298060+0000 2023-07-02T14:01:58.009 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2023-07-02T14:01:50.288411+0000 2023-07-02T14:01:58.009 INFO:teuthology.orchestra.run.smithi016.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-07-02T14:01:58.009 INFO:teuthology.orchestra.run.smithi016.stdout:crush_version 4 2023-07-02T14:01:58.010 INFO:teuthology.orchestra.run.smithi016.stdout:full_ratio 0.95 2023-07-02T14:01:58.010 INFO:teuthology.orchestra.run.smithi016.stdout:backfillfull_ratio 0.9 2023-07-02T14:01:58.010 INFO:teuthology.orchestra.run.smithi016.stdout:nearfull_ratio 0.85 2023-07-02T14:01:58.010 INFO:teuthology.orchestra.run.smithi016.stdout:require_min_compat_client luminous 2023-07-02T14:01:58.010 INFO:teuthology.orchestra.run.smithi016.stdout:min_compat_client jewel 2023-07-02T14:01:58.010 INFO:teuthology.orchestra.run.smithi016.stdout:require_osd_release pacific 2023-07-02T14:01:58.010 INFO:teuthology.orchestra.run.smithi016.stdout:stretch_mode_enabled false 2023-07-02T14:01:58.010 INFO:teuthology.orchestra.run.smithi016.stdout:pool 1 'device_health_metrics' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 14 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr_devicehealth 2023-07-02T14:01:58.011 INFO:teuthology.orchestra.run.smithi016.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 16 flags hashpspool stripe_width 0 application rbd 2023-07-02T14:01:58.011 INFO:teuthology.orchestra.run.smithi016.stdout:max_osd 6 2023-07-02T14:01:58.011 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6800/83359,v1:172.21.15.16:6801/83359] [v2:172.21.15.16:6802/83359,v1:172.21.15.16:6803/83359] exists,up 43c446fe-c811-4ee3-99e8-d7452edfc1c1 2023-07-02T14:01:58.011 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 up in weight 1 up_from 11 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] [v2:172.21.15.16:6810/83360,v1:172.21.15.16:6811/83360] exists,up 4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b 2023-07-02T14:01:58.011 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 up in weight 1 up_from 11 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] [v2:172.21.15.16:6818/83361,v1:172.21.15.16:6819/83361] exists,up 99f8ea0a-5a71-4abe-b087-8e891fd11d1b 2023-07-02T14:01:58.011 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 up in weight 1 up_from 11 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.22:6808/83350,v1:172.21.15.22:6809/83350] [v2:172.21.15.22:6810/83350,v1:172.21.15.22:6811/83350] exists,up d285b4d5-0bef-421a-b84a-943e485b10c2 2023-07-02T14:01:58.011 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.22:6800/83351,v1:172.21.15.22:6801/83351] [v2:172.21.15.22:6802/83351,v1:172.21.15.22:6803/83351] exists,up 7ba0b0db-e407-4c63-8278-aab09f3082c4 2023-07-02T14:01:58.012 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5 up in weight 1 up_from 11 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.22:6816/83352,v1:172.21.15.22:6817/83352] [v2:172.21.15.22:6818/83352,v1:172.21.15.22:6819/83352] exists,up 29663bfa-f6fe-4ce7-b45a-a9b7b4424c42 2023-07-02T14:01:58.022 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6800/83359,v1:172.21.15.16:6801/83359] [v2:172.21.15.16:6802/83359,v1:172.21.15.16:6803/83359] exists,up 43c446fe-c811-4ee3-99e8-d7452edfc1c1', 'osd.1 up in weight 1 up_from 11 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] [v2:172.21.15.16:6810/83360,v1:172.21.15.16:6811/83360] exists,up 4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b', 'osd.2 up in weight 1 up_from 11 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] [v2:172.21.15.16:6818/83361,v1:172.21.15.16:6819/83361] exists,up 99f8ea0a-5a71-4abe-b087-8e891fd11d1b', 'osd.3 up in weight 1 up_from 11 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.22:6808/83350,v1:172.21.15.22:6809/83350] [v2:172.21.15.22:6810/83350,v1:172.21.15.22:6811/83350] exists,up d285b4d5-0bef-421a-b84a-943e485b10c2', 'osd.4 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.22:6800/83351,v1:172.21.15.22:6801/83351] [v2:172.21.15.22:6802/83351,v1:172.21.15.22:6803/83351] exists,up 7ba0b0db-e407-4c63-8278-aab09f3082c4', 'osd.5 up in weight 1 up_from 11 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.22:6816/83352,v1:172.21.15.22:6817/83352] [v2:172.21.15.22:6818/83352,v1:172.21.15.22:6819/83352] exists,up 29663bfa-f6fe-4ce7-b45a-a9b7b4424c42'] 2023-07-02T14:01:58.023 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2023-07-02T14:01:58.141 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:01:58.141 INFO:teuthology.orchestra.run.smithi016.stdout: "name": "mon.a", 2023-07-02T14:01:58.141 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster": "ceph", 2023-07-02T14:01:58.142 INFO:teuthology.orchestra.run.smithi016.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2023-07-02T14:01:58.142 INFO:teuthology.orchestra.run.smithi016.stdout: "admin_socket_mode": "", 2023-07-02T14:01:58.142 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2023-07-02T14:01:58.142 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_client_required": "cephx, none", 2023-07-02T14:01:58.142 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_cluster_required": "cephx", 2023-07-02T14:01:58.142 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_debug": "true", 2023-07-02T14:01:58.142 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2023-07-02T14:01:58.143 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_mon_ticket_ttl": "660.000000", 2023-07-02T14:01:58.143 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_service_required": "cephx", 2023-07-02T14:01:58.143 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_service_ticket_ttl": "240.000000", 2023-07-02T14:01:58.143 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_supported": "cephx", 2023-07-02T14:01:58.143 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_aio": "true", 2023-07-02T14:01:58.143 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_aio_max_queue_depth": "1024", 2023-07-02T14:01:58.143 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_aio_poll_ms": "250", 2023-07-02T14:01:58.143 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_aio_reap_max": "16", 2023-07-02T14:01:58.144 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_async_discard": "false", 2023-07-02T14:01:58.144 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_block_size": "4096", 2023-07-02T14:01:58.144 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_debug_aio": "false", 2023-07-02T14:01:58.144 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_debug_aio_log_age": "5.000000", 2023-07-02T14:01:58.144 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2023-07-02T14:01:58.144 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_debug_inflight_ios": "false", 2023-07-02T14:01:58.144 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_enable_discard": "false", 2023-07-02T14:01:58.145 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_flock_retry": "3", 2023-07-02T14:01:58.145 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_flock_retry_interval": "0.100000", 2023-07-02T14:01:58.145 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_inject_crash": "0", 2023-07-02T14:01:58.145 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_inject_crash_flush_delay": "2", 2023-07-02T14:01:58.145 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_ioring": "false", 2023-07-02T14:01:58.145 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_ioring_hipri": "false", 2023-07-02T14:01:58.145 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_ioring_sqthread_poll": "false", 2023-07-02T14:01:58.145 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_nvme_unbind_from_kernel": "false", 2023-07-02T14:01:58.146 INFO:teuthology.orchestra.run.smithi016.stdout: "bdev_type": "", 2023-07-02T14:01:58.146 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_alloc_size": "1048576", 2023-07-02T14:01:58.146 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_allocator": "hybrid", 2023-07-02T14:01:58.146 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_buffered_io": "true", 2023-07-02T14:01:58.146 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_check_for_zeros": "false", 2023-07-02T14:01:58.146 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_compact_log_sync": "false", 2023-07-02T14:01:58.146 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2023-07-02T14:01:58.146 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_log_compact_min_size": "16777216", 2023-07-02T14:01:58.147 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_log_replay_check_allocations": "true", 2023-07-02T14:01:58.147 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_max_log_runway": "4194304", 2023-07-02T14:01:58.147 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_max_prefetch": "1048576", 2023-07-02T14:01:58.147 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_min_flush_size": "524288", 2023-07-02T14:01:58.147 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_min_log_runway": "1048576", 2023-07-02T14:01:58.147 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_replay_recovery": "false", 2023-07-02T14:01:58.147 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_replay_recovery_disable_compact": "false", 2023-07-02T14:01:58.147 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_shared_alloc_size": "65536", 2023-07-02T14:01:58.148 INFO:teuthology.orchestra.run.smithi016.stdout: "bluefs_sync_write": "false", 2023-07-02T14:01:58.148 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2023-07-02T14:01:58.148 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2023-07-02T14:01:58.148 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2023-07-02T14:01:58.148 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_allocator": "hybrid", 2023-07-02T14:01:58.148 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2023-07-02T14:01:58.148 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2023-07-02T14:01:58.149 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2023-07-02T14:01:58.149 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2023-07-02T14:01:58.149 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2023-07-02T14:01:58.149 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_bitmapallocator_span_size": "1024", 2023-07-02T14:01:58.149 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_blobid_prealloc": "10240", 2023-07-02T14:01:58.149 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_create": "true", 2023-07-02T14:01:58.149 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_db_create": "false", 2023-07-02T14:01:58.150 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_db_path": "", 2023-07-02T14:01:58.150 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_db_size": "0", 2023-07-02T14:01:58.150 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_path": "", 2023-07-02T14:01:58.150 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_preallocate_file": "false", 2023-07-02T14:01:58.150 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_size": "107374182400", 2023-07-02T14:01:58.150 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_wal_create": "false", 2023-07-02T14:01:58.150 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_wal_path": "", 2023-07-02T14:01:58.150 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_block_wal_size": "100663296", 2023-07-02T14:01:58.151 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_bluefs": "true", 2023-07-02T14:01:58.151 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2023-07-02T14:01:58.151 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_bluefs_env_mirror": "false", 2023-07-02T14:01:58.151 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_autotune": "true", 2023-07-02T14:01:58.151 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_autotune_interval": "5.000000", 2023-07-02T14:01:58.151 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2023-07-02T14:01:58.151 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_kv_ratio": "0.450000", 2023-07-02T14:01:58.151 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_meta_ratio": "0.450000", 2023-07-02T14:01:58.152 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_size": "0", 2023-07-02T14:01:58.152 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_size_hdd": "1073741824", 2023-07-02T14:01:58.152 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_size_ssd": "3221225472", 2023-07-02T14:01:58.152 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_trim_interval": "0.050000", 2023-07-02T14:01:58.152 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2023-07-02T14:01:58.152 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_cache_type": "2q", 2023-07-02T14:01:58.152 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_clone_cow": "true", 2023-07-02T14:01:58.153 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_algorithm": "snappy", 2023-07-02T14:01:58.153 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_max_blob_size": "0", 2023-07-02T14:01:58.153 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2023-07-02T14:01:58.153 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2023-07-02T14:01:58.153 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_min_blob_size": "0", 2023-07-02T14:01:58.153 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2023-07-02T14:01:58.153 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_min_blob_size_ssd": "8192", 2023-07-02T14:01:58.153 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_mode": "none", 2023-07-02T14:01:58.154 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_compression_required_ratio": "0.875000", 2023-07-02T14:01:58.154 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_csum_type": "crc32c", 2023-07-02T14:01:58.154 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_enforce_settings": "default", 2023-07-02T14:01:58.154 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_freelist": "false", 2023-07-02T14:01:58.154 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_fsck_abort": "false", 2023-07-02T14:01:58.154 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_inject_bug21040": "false", 2023-07-02T14:01:58.154 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2023-07-02T14:01:58.154 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_inject_read_err": "false", 2023-07-02T14:01:58.155 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_legacy_omap": "false", 2023-07-02T14:01:58.155 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_no_reuse_blocks": "false", 2023-07-02T14:01:58.155 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_omit_block_device_write": "false", 2023-07-02T14:01:58.155 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_omit_kv_commit": "false", 2023-07-02T14:01:58.155 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2023-07-02T14:01:58.155 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_prefill": "0.000000", 2023-07-02T14:01:58.155 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_prefragment_max": "1048576", 2023-07-02T14:01:58.155 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_random_read_err": "0.000000", 2023-07-02T14:01:58.156 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2023-07-02T14:01:58.156 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_small_allocations": "0", 2023-07-02T14:01:58.156 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2023-07-02T14:01:58.156 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_default_buffered_read": "true", 2023-07-02T14:01:58.156 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_default_buffered_write": "false", 2023-07-02T14:01:58.156 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_deferred_batch_ops": "0", 2023-07-02T14:01:58.156 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2023-07-02T14:01:58.157 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2023-07-02T14:01:58.157 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2023-07-02T14:01:58.157 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_map_shard_max_size": "1200", 2023-07-02T14:01:58.157 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_map_shard_min_size": "150", 2023-07-02T14:01:58.157 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_map_shard_target_size": "500", 2023-07-02T14:01:58.157 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2023-07-02T14:01:58.157 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_freelist_blocks_per_key": "128", 2023-07-02T14:01:58.157 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2023-07-02T14:01:58.158 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2023-07-02T14:01:58.158 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2023-07-02T14:01:58.158 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_on_mkfs": "true", 2023-07-02T14:01:58.158 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2023-07-02T14:01:58.158 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_on_mount": "false", 2023-07-02T14:01:58.158 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_on_mount_deep": "false", 2023-07-02T14:01:58.158 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_on_umount": "false", 2023-07-02T14:01:58.158 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_on_umount_deep": "false", 2023-07-02T14:01:58.159 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2023-07-02T14:01:58.159 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_quick_fix_threads": "2", 2023-07-02T14:01:58.159 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2023-07-02T14:01:58.159 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2023-07-02T14:01:58.159 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_gc_enable_blob_threshold": "0", 2023-07-02T14:01:58.159 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_gc_enable_total_threshold": "0", 2023-07-02T14:01:58.159 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2023-07-02T14:01:58.159 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_ignore_data_csum": "false", 2023-07-02T14:01:58.160 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2023-07-02T14:01:58.160 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_kvbackend": "rocksdb", 2023-07-02T14:01:58.160 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_log_collection_list_age": "60.000000", 2023-07-02T14:01:58.160 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2023-07-02T14:01:58.160 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_log_op_age": "5.000000", 2023-07-02T14:01:58.160 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_max_alloc_size": "0", 2023-07-02T14:01:58.160 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_max_blob_size": "0", 2023-07-02T14:01:58.160 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_max_blob_size_hdd": "65536", 2023-07-02T14:01:58.161 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_max_blob_size_ssd": "65536", 2023-07-02T14:01:58.161 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_max_defer_interval": "3.000000", 2023-07-02T14:01:58.161 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_max_deferred_txc": "32", 2023-07-02T14:01:58.161 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_min_alloc_size": "0", 2023-07-02T14:01:58.161 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_min_alloc_size_hdd": "4096", 2023-07-02T14:01:58.161 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_min_alloc_size_ssd": "4096", 2023-07-02T14:01:58.161 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_nid_prealloc": "1024", 2023-07-02T14:01:58.161 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_prefer_deferred_size": "0", 2023-07-02T14:01:58.162 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2023-07-02T14:01:58.162 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2023-07-02T14:01:58.162 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_retry_disk_reads": "3", 2023-07-02T14:01:58.162 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_rocksdb_cf": "true", 2023-07-02T14:01:58.162 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2023-07-02T14:01:58.162 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2023-07-02T14:01:58.162 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_rocksdb_options_annex": "", 2023-07-02T14:01:58.163 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_spdk_coremask": "0x1", 2023-07-02T14:01:58.163 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_spdk_io_sleep": "5", 2023-07-02T14:01:58.163 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_spdk_max_io_completion": "0", 2023-07-02T14:01:58.163 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_spdk_mem": "512", 2023-07-02T14:01:58.163 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_sync_submit_transaction": "false", 2023-07-02T14:01:58.163 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_throttle_bytes": "67108864", 2023-07-02T14:01:58.163 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_throttle_cost_per_io": "0", 2023-07-02T14:01:58.163 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2023-07-02T14:01:58.164 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2023-07-02T14:01:58.164 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2023-07-02T14:01:58.164 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_throttle_trace_rate": "0.000000", 2023-07-02T14:01:58.164 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_tracing": "false", 2023-07-02T14:01:58.164 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2023-07-02T14:01:58.164 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_volume_selection_reserved": "0", 2023-07-02T14:01:58.164 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2023-07-02T14:01:58.165 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2023-07-02T14:01:58.165 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_warn_on_legacy_statfs": "true", 2023-07-02T14:01:58.165 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2023-07-02T14:01:58.165 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2023-07-02T14:01:58.165 INFO:teuthology.orchestra.run.smithi016.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2023-07-02T14:01:58.165 INFO:teuthology.orchestra.run.smithi016.stdout: "cephadm_path": "/usr/sbin/cephadm", 2023-07-02T14:01:58.165 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_action_update_interval": "2", 2023-07-02T14:01:58.165 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_directory_scan_interval": "10", 2023-07-02T14:01:58.166 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2023-07-02T14:01:58.166 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2023-07-02T14:01:58.166 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2023-07-02T14:01:58.166 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_mount_timeout": "10", 2023-07-02T14:01:58.166 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2023-07-02T14:01:58.166 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2023-07-02T14:01:58.166 INFO:teuthology.orchestra.run.smithi016.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2023-07-02T14:01:58.166 INFO:teuthology.orchestra.run.smithi016.stdout: "cephsqlite_blocklist_dead_locker": "true", 2023-07-02T14:01:58.167 INFO:teuthology.orchestra.run.smithi016.stdout: "cephsqlite_lock_renewal_interval": "2000", 2023-07-02T14:01:58.167 INFO:teuthology.orchestra.run.smithi016.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2023-07-02T14:01:58.167 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_cluster_require_signatures": "false", 2023-07-02T14:01:58.167 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_cluster_require_version": "2", 2023-07-02T14:01:58.167 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_require_signatures": "false", 2023-07-02T14:01:58.167 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_require_version": "2", 2023-07-02T14:01:58.167 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_service_require_signatures": "false", 2023-07-02T14:01:58.168 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_service_require_version": "2", 2023-07-02T14:01:58.168 INFO:teuthology.orchestra.run.smithi016.stdout: "cephx_sign_messages": "true", 2023-07-02T14:01:58.168 INFO:teuthology.orchestra.run.smithi016.stdout: "chdir": "", 2023-07-02T14:01:58.168 INFO:teuthology.orchestra.run.smithi016.stdout: "client_acl_type": "", 2023-07-02T14:01:58.168 INFO:teuthology.orchestra.run.smithi016.stdout: "client_asio_thread_count": "2", 2023-07-02T14:01:58.168 INFO:teuthology.orchestra.run.smithi016.stdout: "client_cache_mid": "0.750000", 2023-07-02T14:01:58.168 INFO:teuthology.orchestra.run.smithi016.stdout: "client_cache_size": "16384", 2023-07-02T14:01:58.168 INFO:teuthology.orchestra.run.smithi016.stdout: "client_caps_release_delay": "5", 2023-07-02T14:01:58.169 INFO:teuthology.orchestra.run.smithi016.stdout: "client_check_pool_perm": "true", 2023-07-02T14:01:58.169 INFO:teuthology.orchestra.run.smithi016.stdout: "client_collect_and_send_global_metrics": "false", 2023-07-02T14:01:58.169 INFO:teuthology.orchestra.run.smithi016.stdout: "client_debug_force_sync_read": "false", 2023-07-02T14:01:58.169 INFO:teuthology.orchestra.run.smithi016.stdout: "client_debug_getattr_caps": "false", 2023-07-02T14:01:58.169 INFO:teuthology.orchestra.run.smithi016.stdout: "client_debug_inject_tick_delay": "0", 2023-07-02T14:01:58.169 INFO:teuthology.orchestra.run.smithi016.stdout: "client_die_on_failed_dentry_invalidate": "true", 2023-07-02T14:01:58.169 INFO:teuthology.orchestra.run.smithi016.stdout: "client_die_on_failed_remount": "false", 2023-07-02T14:01:58.169 INFO:teuthology.orchestra.run.smithi016.stdout: "client_dirsize_rbytes": "true", 2023-07-02T14:01:58.170 INFO:teuthology.orchestra.run.smithi016.stdout: "client_force_lazyio": "false", 2023-07-02T14:01:58.170 INFO:teuthology.orchestra.run.smithi016.stdout: "client_fs": "", 2023-07-02T14:01:58.170 INFO:teuthology.orchestra.run.smithi016.stdout: "client_inject_fixed_oldest_tid": "false", 2023-07-02T14:01:58.170 INFO:teuthology.orchestra.run.smithi016.stdout: "client_inject_release_failure": "false", 2023-07-02T14:01:58.170 INFO:teuthology.orchestra.run.smithi016.stdout: "client_max_inline_size": "4096", 2023-07-02T14:01:58.170 INFO:teuthology.orchestra.run.smithi016.stdout: "client_max_retries_on_remount_failure": "5", 2023-07-02T14:01:58.170 INFO:teuthology.orchestra.run.smithi016.stdout: "client_mds_namespace": "", 2023-07-02T14:01:58.171 INFO:teuthology.orchestra.run.smithi016.stdout: "client_metadata": "", 2023-07-02T14:01:58.171 INFO:teuthology.orchestra.run.smithi016.stdout: "client_mount_gid": "-1", 2023-07-02T14:01:58.171 INFO:teuthology.orchestra.run.smithi016.stdout: "client_mount_timeout": "300", 2023-07-02T14:01:58.171 INFO:teuthology.orchestra.run.smithi016.stdout: "client_mount_uid": "-1", 2023-07-02T14:01:58.171 INFO:teuthology.orchestra.run.smithi016.stdout: "client_mountpoint": "/", 2023-07-02T14:01:58.171 INFO:teuthology.orchestra.run.smithi016.stdout: "client_notify_timeout": "10", 2023-07-02T14:01:58.171 INFO:teuthology.orchestra.run.smithi016.stdout: "client_oc": "true", 2023-07-02T14:01:58.171 INFO:teuthology.orchestra.run.smithi016.stdout: "client_oc_max_dirty": "104857600", 2023-07-02T14:01:58.172 INFO:teuthology.orchestra.run.smithi016.stdout: "client_oc_max_dirty_age": "5.000000", 2023-07-02T14:01:58.172 INFO:teuthology.orchestra.run.smithi016.stdout: "client_oc_max_objects": "1000", 2023-07-02T14:01:58.172 INFO:teuthology.orchestra.run.smithi016.stdout: "client_oc_size": "209715200", 2023-07-02T14:01:58.172 INFO:teuthology.orchestra.run.smithi016.stdout: "client_oc_target_dirty": "8388608", 2023-07-02T14:01:58.172 INFO:teuthology.orchestra.run.smithi016.stdout: "client_permissions": "true", 2023-07-02T14:01:58.172 INFO:teuthology.orchestra.run.smithi016.stdout: "client_quota": "true", 2023-07-02T14:01:58.172 INFO:teuthology.orchestra.run.smithi016.stdout: "client_quota_df": "true", 2023-07-02T14:01:58.172 INFO:teuthology.orchestra.run.smithi016.stdout: "client_readahead_max_bytes": "0", 2023-07-02T14:01:58.173 INFO:teuthology.orchestra.run.smithi016.stdout: "client_readahead_max_periods": "4", 2023-07-02T14:01:58.173 INFO:teuthology.orchestra.run.smithi016.stdout: "client_readahead_min": "131072", 2023-07-02T14:01:58.173 INFO:teuthology.orchestra.run.smithi016.stdout: "client_reconnect_stale": "false", 2023-07-02T14:01:58.173 INFO:teuthology.orchestra.run.smithi016.stdout: "client_shutdown_timeout": "30", 2023-07-02T14:01:58.173 INFO:teuthology.orchestra.run.smithi016.stdout: "client_snapdir": ".snap", 2023-07-02T14:01:58.173 INFO:teuthology.orchestra.run.smithi016.stdout: "client_tick_interval": "1", 2023-07-02T14:01:58.173 INFO:teuthology.orchestra.run.smithi016.stdout: "client_trace": "", 2023-07-02T14:01:58.174 INFO:teuthology.orchestra.run.smithi016.stdout: "client_try_dentry_invalidate": "false", 2023-07-02T14:01:58.174 INFO:teuthology.orchestra.run.smithi016.stdout: "client_use_faked_inos": "false", 2023-07-02T14:01:58.174 INFO:teuthology.orchestra.run.smithi016.stdout: "client_use_random_mds": "false", 2023-07-02T14:01:58.174 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_graylog": "false", 2023-07-02T14:01:58.174 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_graylog_host": "127.0.0.1", 2023-07-02T14:01:58.174 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_graylog_port": "12201", 2023-07-02T14:01:58.174 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_monitors": "default=true", 2023-07-02T14:01:58.174 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_syslog": "false", 2023-07-02T14:01:58.175 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2023-07-02T14:01:58.175 INFO:teuthology.orchestra.run.smithi016.stdout: "clog_to_syslog_level": "info", 2023-07-02T14:01:58.175 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "-", 2023-07-02T14:01:58.175 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_network": "", 2023-07-02T14:01:58.175 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_network_interface": "", 2023-07-02T14:01:58.175 INFO:teuthology.orchestra.run.smithi016.stdout: "compressor_zlib_isal": "false", 2023-07-02T14:01:58.175 INFO:teuthology.orchestra.run.smithi016.stdout: "compressor_zlib_level": "5", 2023-07-02T14:01:58.175 INFO:teuthology.orchestra.run.smithi016.stdout: "compressor_zlib_winsize": "-15", 2023-07-02T14:01:58.176 INFO:teuthology.orchestra.run.smithi016.stdout: "compressor_zstd_level": "1", 2023-07-02T14:01:58.176 INFO:teuthology.orchestra.run.smithi016.stdout: "container_image": "docker.io/ceph/daemon-base:latest-pacific-devel", 2023-07-02T14:01:58.176 INFO:teuthology.orchestra.run.smithi016.stdout: "crash_dir": "/var/lib/ceph/crash", 2023-07-02T14:01:58.176 INFO:teuthology.orchestra.run.smithi016.stdout: "crimson_osd_obc_lru_size": "10", 2023-07-02T14:01:58.176 INFO:teuthology.orchestra.run.smithi016.stdout: "crimson_osd_scheduler_concurrency": "0", 2023-07-02T14:01:58.176 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_location": "", 2023-07-02T14:01:58.176 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_location_hook": "", 2023-07-02T14:01:58.176 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_location_hook_timeout": "10", 2023-07-02T14:01:58.177 INFO:teuthology.orchestra.run.smithi016.stdout: "daemonize": "false", 2023-07-02T14:01:58.177 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_allow_any_pool_priority": "false", 2023-07-02T14:01:58.177 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_asok": "1/5", 2023-07-02T14:01:58.177 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_asok_assert_abort": "false", 2023-07-02T14:01:58.177 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_asserts_on_shutdown": "true", 2023-07-02T14:01:58.177 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_auth": "20/20", 2023-07-02T14:01:58.177 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_bdev": "1/3", 2023-07-02T14:01:58.177 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_bluefs": "1/5", 2023-07-02T14:01:58.178 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_bluestore": "1/5", 2023-07-02T14:01:58.178 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_buffer": "0/1", 2023-07-02T14:01:58.178 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_cephfs_mirror": "0/5", 2023-07-02T14:01:58.178 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_cephsqlite": "0/5", 2023-07-02T14:01:58.178 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_civetweb": "1/10", 2023-07-02T14:01:58.178 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_client": "0/5", 2023-07-02T14:01:58.179 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_compressor": "1/5", 2023-07-02T14:01:58.179 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_context": "0/1", 2023-07-02T14:01:58.179 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_crush": "1/1", 2023-07-02T14:01:58.179 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_crypto": "1/5", 2023-07-02T14:01:58.179 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_deliberately_leak_memory": "false", 2023-07-02T14:01:58.179 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_disable_randomized_ping": "false", 2023-07-02T14:01:58.179 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_dpdk": "1/5", 2023-07-02T14:01:58.180 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_eventtrace": "1/5", 2023-07-02T14:01:58.180 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_filer": "0/1", 2023-07-02T14:01:58.180 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_filestore": "1/3", 2023-07-02T14:01:58.180 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_finisher": "1/1", 2023-07-02T14:01:58.180 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_fuse": "1/5", 2023-07-02T14:01:58.180 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_heartbeat_testing_span": "0", 2023-07-02T14:01:58.180 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_heartbeatmap": "1/5", 2023-07-02T14:01:58.180 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_immutable_obj_cache": "0/5", 2023-07-02T14:01:58.181 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_javaclient": "1/5", 2023-07-02T14:01:58.181 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_journal": "1/3", 2023-07-02T14:01:58.181 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_journaler": "0/5", 2023-07-02T14:01:58.181 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_kstore": "1/5", 2023-07-02T14:01:58.181 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_leveldb": "4/5", 2023-07-02T14:01:58.181 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_lockdep": "0/1", 2023-07-02T14:01:58.181 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mds": "1/5", 2023-07-02T14:01:58.181 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mds_balancer": "1/5", 2023-07-02T14:01:58.182 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mds_locker": "1/5", 2023-07-02T14:01:58.182 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mds_log": "1/5", 2023-07-02T14:01:58.182 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mds_log_expire": "1/5", 2023-07-02T14:01:58.182 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mds_migrator": "1/5", 2023-07-02T14:01:58.182 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_memdb": "4/5", 2023-07-02T14:01:58.182 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mgr": "2/5", 2023-07-02T14:01:58.182 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mgrc": "1/5", 2023-07-02T14:01:58.182 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_mon": "20/20", 2023-07-02T14:01:58.183 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_monc": "0/10", 2023-07-02T14:01:58.183 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_ms": "1/1", 2023-07-02T14:01:58.183 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_none": "0/5", 2023-07-02T14:01:58.183 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_objclass": "0/5", 2023-07-02T14:01:58.183 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_objectcacher": "0/5", 2023-07-02T14:01:58.183 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_objecter": "0/1", 2023-07-02T14:01:58.183 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_optracker": "0/5", 2023-07-02T14:01:58.183 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_osd": "1/5", 2023-07-02T14:01:58.184 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_paxos": "20/20", 2023-07-02T14:01:58.184 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_perfcounter": "1/5", 2023-07-02T14:01:58.184 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_prioritycache": "1/5", 2023-07-02T14:01:58.184 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rados": "0/5", 2023-07-02T14:01:58.184 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rbd": "0/5", 2023-07-02T14:01:58.184 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rbd_mirror": "0/5", 2023-07-02T14:01:58.184 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rbd_pwl": "0/5", 2023-07-02T14:01:58.185 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rbd_replay": "0/5", 2023-07-02T14:01:58.185 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_refs": "0/0", 2023-07-02T14:01:58.185 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_reserver": "1/1", 2023-07-02T14:01:58.185 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rgw": "1/5", 2023-07-02T14:01:58.185 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rgw_sync": "1/5", 2023-07-02T14:01:58.185 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_rocksdb": "4/5", 2023-07-02T14:01:58.185 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_striper": "0/1", 2023-07-02T14:01:58.185 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_test": "0/5", 2023-07-02T14:01:58.186 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_throttle": "1/1", 2023-07-02T14:01:58.186 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_timer": "0/1", 2023-07-02T14:01:58.186 INFO:teuthology.orchestra.run.smithi016.stdout: "debug_tp": "0/5", 2023-07-02T14:01:58.186 INFO:teuthology.orchestra.run.smithi016.stdout: "device_failure_prediction_mode": "none", 2023-07-02T14:01:58.186 INFO:teuthology.orchestra.run.smithi016.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2023-07-02T14:01:58.186 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2023-07-02T14:01:58.186 INFO:teuthology.orchestra.run.smithi016.stdout: "err_to_graylog": "false", 2023-07-02T14:01:58.186 INFO:teuthology.orchestra.run.smithi016.stdout: "err_to_stderr": "true", 2023-07-02T14:01:58.187 INFO:teuthology.orchestra.run.smithi016.stdout: "err_to_syslog": "false", 2023-07-02T14:01:58.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event_tracing": "false", 2023-07-02T14:01:58.187 INFO:teuthology.orchestra.run.smithi016.stdout: "fake_statfs_for_testing": "0", 2023-07-02T14:01:58.187 INFO:teuthology.orchestra.run.smithi016.stdout: "fatal_signal_handlers": "true", 2023-07-02T14:01:58.187 INFO:teuthology.orchestra.run.smithi016.stdout: "filer_max_purge_ops": "10", 2023-07-02T14:01:58.187 INFO:teuthology.orchestra.run.smithi016.stdout: "filer_max_truncate_ops": "128", 2023-07-02T14:01:58.187 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_apply_finisher_threads": "1", 2023-07-02T14:01:58.187 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_blackhole": "false", 2023-07-02T14:01:58.188 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_btrfs_clone_range": "true", 2023-07-02T14:01:58.188 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_btrfs_snap": "true", 2023-07-02T14:01:58.188 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_caller_concurrency": "10", 2023-07-02T14:01:58.188 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_collect_device_partition_information": "true", 2023-07-02T14:01:58.188 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_commit_timeout": "600.000000", 2023-07-02T14:01:58.188 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_debug_inject_read_err": "false", 2023-07-02T14:01:58.188 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_debug_omap_check": "false", 2023-07-02T14:01:58.188 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_debug_random_read_err": "0.000000", 2023-07-02T14:01:58.189 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_debug_verify_split": "false", 2023-07-02T14:01:58.189 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_dump_file": "", 2023-07-02T14:01:58.189 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2023-07-02T14:01:58.189 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_expected_throughput_ops": "200.000000", 2023-07-02T14:01:58.189 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fadvise": "true", 2023-07-02T14:01:58.189 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fail_eio": "true", 2023-07-02T14:01:58.189 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fd_cache_shards": "16", 2023-07-02T14:01:58.189 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fd_cache_size": "128", 2023-07-02T14:01:58.190 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fiemap": "false", 2023-07-02T14:01:58.190 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fiemap_threshold": "4096", 2023-07-02T14:01:58.190 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_fsync_flushes_journal_data": "false", 2023-07-02T14:01:58.190 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_index_retry_probability": "0.000000", 2023-07-02T14:01:58.190 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_inject_stall": "0", 2023-07-02T14:01:58.190 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_journal_parallel": "false", 2023-07-02T14:01:58.190 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_journal_trailing": "false", 2023-07-02T14:01:58.190 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_journal_writeahead": "false", 2023-07-02T14:01:58.191 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_kill_at": "0", 2023-07-02T14:01:58.191 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_alloc_hint_size": "1048576", 2023-07-02T14:01:58.191 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattr_size": "0", 2023-07-02T14:01:58.191 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2023-07-02T14:01:58.191 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattr_size_other": "512", 2023-07-02T14:01:58.191 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2023-07-02T14:01:58.191 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattrs": "0", 2023-07-02T14:01:58.191 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2023-07-02T14:01:58.192 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattrs_other": "2", 2023-07-02T14:01:58.192 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_inline_xattrs_xfs": "10", 2023-07-02T14:01:58.192 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_sync_interval": "5.000000", 2023-07-02T14:01:58.192 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_xattr_value_size": "0", 2023-07-02T14:01:58.192 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2023-07-02T14:01:58.192 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_xattr_value_size_other": "1024", 2023-07-02T14:01:58.192 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2023-07-02T14:01:58.192 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_merge_threshold": "-10", 2023-07-02T14:01:58.193 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_min_sync_interval": "0.010000", 2023-07-02T14:01:58.193 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_odsync_write": "false", 2023-07-02T14:01:58.193 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_omap_backend": "rocksdb", 2023-07-02T14:01:58.193 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_omap_backend_path": "", 2023-07-02T14:01:58.193 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_omap_header_cache_size": "1024", 2023-07-02T14:01:58.193 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_ondisk_finisher_threads": "1", 2023-07-02T14:01:58.193 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_op_thread_suicide_timeout": "180", 2023-07-02T14:01:58.194 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_op_thread_timeout": "60", 2023-07-02T14:01:58.194 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_op_threads": "2", 2023-07-02T14:01:58.194 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_punch_hole": "false", 2023-07-02T14:01:58.194 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2023-07-02T14:01:58.194 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2023-07-02T14:01:58.194 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2023-07-02T14:01:58.194 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_high_threshhold": "0.900000", 2023-07-02T14:01:58.194 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_low_threshhold": "0.300000", 2023-07-02T14:01:58.195 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_max_bytes": "104857600", 2023-07-02T14:01:58.195 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2023-07-02T14:01:58.195 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2023-07-02T14:01:58.195 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2023-07-02T14:01:58.195 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_queue_max_ops": "50", 2023-07-02T14:01:58.195 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2023-07-02T14:01:58.195 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_seek_data_hole": "false", 2023-07-02T14:01:58.195 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_sloppy_crc": "false", 2023-07-02T14:01:58.196 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_sloppy_crc_block_size": "65536", 2023-07-02T14:01:58.196 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_splice": "false", 2023-07-02T14:01:58.196 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_split_multiple": "2", 2023-07-02T14:01:58.196 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_split_rand_factor": "20", 2023-07-02T14:01:58.196 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_update_to": "1000", 2023-07-02T14:01:58.196 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2023-07-02T14:01:58.196 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2023-07-02T14:01:58.196 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2023-07-02T14:01:58.197 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2023-07-02T14:01:58.197 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2023-07-02T14:01:58.197 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2023-07-02T14:01:58.197 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_enable": "true", 2023-07-02T14:01:58.197 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2023-07-02T14:01:58.197 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2023-07-02T14:01:58.197 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2023-07-02T14:01:58.198 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2023-07-02T14:01:58.198 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2023-07-02T14:01:58.198 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2023-07-02T14:01:58.198 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_xfs_extsize": "false", 2023-07-02T14:01:58.198 INFO:teuthology.orchestra.run.smithi016.stdout: "filestore_zfs_snap": "false", 2023-07-02T14:01:58.198 INFO:teuthology.orchestra.run.smithi016.stdout: "fio_dir": "/tmp/fio", 2023-07-02T14:01:58.198 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2", 2023-07-02T14:01:58.198 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_allow_other": "true", 2023-07-02T14:01:58.199 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_atomic_o_trunc": "true", 2023-07-02T14:01:58.199 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_big_writes": "true", 2023-07-02T14:01:58.199 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_debug": "false", 2023-07-02T14:01:58.199 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_default_permissions": "false", 2023-07-02T14:01:58.199 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_disable_pagecache": "false", 2023-07-02T14:01:58.199 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_max_write": "0", 2023-07-02T14:01:58.199 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_multithreaded": "true", 2023-07-02T14:01:58.199 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_require_active_mds": "true", 2023-07-02T14:01:58.200 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_set_user_groups": "true", 2023-07-02T14:01:58.200 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_splice_move": "true", 2023-07-02T14:01:58.200 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_splice_read": "true", 2023-07-02T14:01:58.200 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_splice_write": "true", 2023-07-02T14:01:58.200 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_syncfs_on_mksnap": "true", 2023-07-02T14:01:58.200 INFO:teuthology.orchestra.run.smithi016.stdout: "fuse_use_invalidate_cb": "true", 2023-07-02T14:01:58.200 INFO:teuthology.orchestra.run.smithi016.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2023-07-02T14:01:58.200 INFO:teuthology.orchestra.run.smithi016.stdout: "gss_target_name": "ceph", 2023-07-02T14:01:58.201 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_file": "", 2023-07-02T14:01:58.201 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_inject_failure": "0", 2023-07-02T14:01:58.201 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_interval": "5", 2023-07-02T14:01:58.201 INFO:teuthology.orchestra.run.smithi016.stdout: "host": "", 2023-07-02T14:01:58.201 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2023-07-02T14:01:58.201 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_max_inflight_ops": "128", 2023-07-02T14:01:58.201 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_max_size": "1073741824", 2023-07-02T14:01:58.201 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2023-07-02T14:01:58.202 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_bps_burst": "0", 2023-07-02T14:01:58.202 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2023-07-02T14:01:58.202 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_bps_limit": "0", 2023-07-02T14:01:58.202 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_iops_burst": "0", 2023-07-02T14:01:58.202 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2023-07-02T14:01:58.202 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_iops_limit": "0", 2023-07-02T14:01:58.202 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2023-07-02T14:01:58.202 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2023-07-02T14:01:58.203 INFO:teuthology.orchestra.run.smithi016.stdout: "immutable_object_cache_watermark": "0.900000", 2023-07-02T14:01:58.203 INFO:teuthology.orchestra.run.smithi016.stdout: "inject_early_sigterm": "false", 2023-07-02T14:01:58.203 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_aio": "true", 2023-07-02T14:01:58.203 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_align_min_size": "65536", 2023-07-02T14:01:58.203 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_block_align": "true", 2023-07-02T14:01:58.203 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_block_size": "4096", 2023-07-02T14:01:58.203 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_dio": "true", 2023-07-02T14:01:58.203 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_discard": "false", 2023-07-02T14:01:58.204 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_force_aio": "false", 2023-07-02T14:01:58.204 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_ignore_corruption": "false", 2023-07-02T14:01:58.204 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_max_write_bytes": "10485760", 2023-07-02T14:01:58.204 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_max_write_entries": "100", 2023-07-02T14:01:58.204 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_replay_from": "0", 2023-07-02T14:01:58.204 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_throttle_high_multiple": "0.000000", 2023-07-02T14:01:58.204 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_throttle_high_threshhold": "0.900000", 2023-07-02T14:01:58.204 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_throttle_low_threshhold": "0.600000", 2023-07-02T14:01:58.205 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_throttle_max_multiple": "0.000000", 2023-07-02T14:01:58.205 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_write_header_frequency": "0", 2023-07-02T14:01:58.205 INFO:teuthology.orchestra.run.smithi016.stdout: "journal_zero_on_create": "false", 2023-07-02T14:01:58.205 INFO:teuthology.orchestra.run.smithi016.stdout: "journaler_prefetch_periods": "10", 2023-07-02T14:01:58.205 INFO:teuthology.orchestra.run.smithi016.stdout: "journaler_prezero_periods": "5", 2023-07-02T14:01:58.205 INFO:teuthology.orchestra.run.smithi016.stdout: "journaler_write_head_interval": "15", 2023-07-02T14:01:58.205 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "", 2023-07-02T14:01:58.206 INFO:teuthology.orchestra.run.smithi016.stdout: "keyfile": "", 2023-07-02T14:01:58.206 INFO:teuthology.orchestra.run.smithi016.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2023-07-02T14:01:58.206 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_backend": "rocksdb", 2023-07-02T14:01:58.206 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_default_stripe_size": "65536", 2023-07-02T14:01:58.206 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_fsck_on_mount": "false", 2023-07-02T14:01:58.206 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_fsck_on_mount_deep": "true", 2023-07-02T14:01:58.206 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_max_bytes": "67108864", 2023-07-02T14:01:58.206 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_max_ops": "512", 2023-07-02T14:01:58.207 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_nid_prealloc": "1024", 2023-07-02T14:01:58.207 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_onode_map_size": "1024", 2023-07-02T14:01:58.207 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2023-07-02T14:01:58.207 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_sync_submit_transaction": "false", 2023-07-02T14:01:58.207 INFO:teuthology.orchestra.run.smithi016.stdout: "kstore_sync_transaction": "false", 2023-07-02T14:01:58.207 INFO:teuthology.orchestra.run.smithi016.stdout: "leveldb_block_size": "65536", 2023-07-02T14:01:58.207 INFO:teuthology.orchestra.run.smithi016.stdout: "leveldb_bloom_size": "0", 2023-07-02T14:01:58.207 INFO:teuthology.orchestra.run.smithi016.stdout: "leveldb_cache_size": "536870912", 2023-07-02T14:01:58.208 INFO:teuthology.orchestra.run.smithi016.stdout: "leveldb_compact_on_mount": "false", 2023-07-02T14:01:58.208 INFO:teuthology.orchestra.run.smithi016.stdout: "leveldb_compression": "false", 2023-07-02T14:01:58.208 INFO:teuthology.orchestra.run.smithi016.stdout: "leveldb_log": "", 2023-07-02T14:01:58.208 INFO:teuthology.orchestra.run.smithi016.stdout: "leveldb_log_to_ceph_log": "true", 2023-07-02T14:01:58.208 INFO:teuthology.orchestra.run.smithi016.stdout: "leveldb_max_open_files": "0", 2023-07-02T14:01:58.208 INFO:teuthology.orchestra.run.smithi016.stdout: "leveldb_paranoid": "false", 2023-07-02T14:01:58.209 INFO:teuthology.orchestra.run.smithi016.stdout: "leveldb_write_buffer_size": "33554432", 2023-07-02T14:01:58.209 INFO:teuthology.orchestra.run.smithi016.stdout: "librados_thread_count": "2", 2023-07-02T14:01:58.209 INFO:teuthology.orchestra.run.smithi016.stdout: "lockdep": "false", 2023-07-02T14:01:58.209 INFO:teuthology.orchestra.run.smithi016.stdout: "lockdep_force_backtrace": "false", 2023-07-02T14:01:58.209 INFO:teuthology.orchestra.run.smithi016.stdout: "log_coarse_timestamps": "true", 2023-07-02T14:01:58.209 INFO:teuthology.orchestra.run.smithi016.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2023-07-02T14:01:58.209 INFO:teuthology.orchestra.run.smithi016.stdout: "log_flush_on_exit": "false", 2023-07-02T14:01:58.209 INFO:teuthology.orchestra.run.smithi016.stdout: "log_graylog_host": "127.0.0.1", 2023-07-02T14:01:58.210 INFO:teuthology.orchestra.run.smithi016.stdout: "log_graylog_port": "12201", 2023-07-02T14:01:58.210 INFO:teuthology.orchestra.run.smithi016.stdout: "log_max_new": "1000", 2023-07-02T14:01:58.210 INFO:teuthology.orchestra.run.smithi016.stdout: "log_max_recent": "10000", 2023-07-02T14:01:58.210 INFO:teuthology.orchestra.run.smithi016.stdout: "log_stderr_prefix": "", 2023-07-02T14:01:58.210 INFO:teuthology.orchestra.run.smithi016.stdout: "log_stop_at_utilization": "0.970000", 2023-07-02T14:01:58.210 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_file": "true", 2023-07-02T14:01:58.210 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_graylog": "false", 2023-07-02T14:01:58.210 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_stderr": "false", 2023-07-02T14:01:58.211 INFO:teuthology.orchestra.run.smithi016.stdout: "log_to_syslog": "false", 2023-07-02T14:01:58.211 INFO:teuthology.orchestra.run.smithi016.stdout: "max_rotating_auth_attempts": "10", 2023-07-02T14:01:58.211 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2023-07-02T14:01:58.211 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_action_on_write_error": "1", 2023-07-02T14:01:58.211 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_alternate_name_max": "8192", 2023-07-02T14:01:58.211 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_asio_thread_count": "2", 2023-07-02T14:01:58.211 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_export_pin": "true", 2023-07-02T14:01:58.211 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_fragment_dirs": "true", 2023-07-02T14:01:58.212 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2023-07-02T14:01:58.212 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_fragment_interval": "5", 2023-07-02T14:01:58.212 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_fragment_size_max": "100000", 2023-07-02T14:01:58.212 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_idle_threshold": "0.000000", 2023-07-02T14:01:58.212 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_interval": "10", 2023-07-02T14:01:58.212 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_max": "-1", 2023-07-02T14:01:58.212 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_max_until": "-1", 2023-07-02T14:01:58.213 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_merge_size": "50", 2023-07-02T14:01:58.213 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_midchunk": "0.300000", 2023-07-02T14:01:58.213 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_min_rebalance": "0.100000", 2023-07-02T14:01:58.213 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_min_start": "0.200000", 2023-07-02T14:01:58.213 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_minchunk": "0.001000", 2023-07-02T14:01:58.213 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_mode": "0", 2023-07-02T14:01:58.213 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_need_max": "1.200000", 2023-07-02T14:01:58.214 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_need_min": "0.800000", 2023-07-02T14:01:58.214 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_replicate_threshold": "8000.000000", 2023-07-02T14:01:58.214 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_sample_interval": "3.000000", 2023-07-02T14:01:58.214 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_split_bits": "3", 2023-07-02T14:01:58.214 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_split_rd": "25000.000000", 2023-07-02T14:01:58.214 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_split_size": "10000", 2023-07-02T14:01:58.214 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_split_wr": "10000.000000", 2023-07-02T14:01:58.214 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_target_decay": "10.000000", 2023-07-02T14:01:58.215 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2023-07-02T14:01:58.215 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_beacon_grace": "15.000000", 2023-07-02T14:01:58.215 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_beacon_interval": "4.000000", 2023-07-02T14:01:58.215 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_beacon_mon_down_grace": "60", 2023-07-02T14:01:58.215 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_memory_limit": "4294967296", 2023-07-02T14:01:58.215 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_mid": "0.700000", 2023-07-02T14:01:58.215 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_release_free_interval": "10", 2023-07-02T14:01:58.215 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_reservation": "0.050000", 2023-07-02T14:01:58.216 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_trim_decay_rate": "1.000000", 2023-07-02T14:01:58.216 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_trim_interval": "1", 2023-07-02T14:01:58.216 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cache_trim_threshold": "262144", 2023-07-02T14:01:58.216 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2023-07-02T14:01:58.216 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2023-07-02T14:01:58.216 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_client_delegate_inos_pct": "50", 2023-07-02T14:01:58.216 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_client_prealloc_inos": "1000", 2023-07-02T14:01:58.216 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2023-07-02T14:01:58.217 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_connect_bootstrapping": "false", 2023-07-02T14:01:58.217 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_damage_table_max_entries": "10000", 2023-07-02T14:01:58.217 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2023-07-02T14:01:58.217 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_debug_auth_pins": "false", 2023-07-02T14:01:58.217 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_debug_frag": "false", 2023-07-02T14:01:58.217 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_debug_scatterstat": "false", 2023-07-02T14:01:58.217 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_debug_subtrees": "false", 2023-07-02T14:01:58.217 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_decay_halflife": "5.000000", 2023-07-02T14:01:58.218 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_default_dir_hash": "2", 2023-07-02T14:01:58.218 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_defer_session_stale": "true", 2023-07-02T14:01:58.218 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_deny_all_reconnect": "false", 2023-07-02T14:01:58.218 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dir_keys_per_op": "16384", 2023-07-02T14:01:58.218 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dir_max_commit_size": "10", 2023-07-02T14:01:58.218 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dir_max_entries": "0", 2023-07-02T14:01:58.218 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dirstat_min_interval": "1.000000", 2023-07-02T14:01:58.219 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dump_cache_after_rejoin": "false", 2023-07-02T14:01:58.219 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dump_cache_on_map": "false", 2023-07-02T14:01:58.219 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dump_cache_threshold_file": "0", 2023-07-02T14:01:58.219 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2023-07-02T14:01:58.219 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_early_reply": "true", 2023-07-02T14:01:58.219 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_enable_op_tracker": "true", 2023-07-02T14:01:58.219 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_enforce_unique_name": "true", 2023-07-02T14:01:58.219 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_export_ephemeral_distributed": "true", 2023-07-02T14:01:58.220 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2023-07-02T14:01:58.220 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_export_ephemeral_random": "true", 2023-07-02T14:01:58.220 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_export_ephemeral_random_max": "0.010000", 2023-07-02T14:01:58.220 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_forward_all_requests_to_auth": "false", 2023-07-02T14:01:58.220 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_freeze_tree_timeout": "30.000000", 2023-07-02T14:01:58.220 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_go_bad_corrupt_dentry": "true", 2023-07-02T14:01:58.220 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2023-07-02T14:01:58.220 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_health_cache_threshold": "1.500000", 2023-07-02T14:01:58.221 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_health_summarize_threshold": "10", 2023-07-02T14:01:58.221 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_heartbeat_grace": "15.000000", 2023-07-02T14:01:58.221 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_heartbeat_reset_grace": "1000", 2023-07-02T14:01:58.221 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_inject_health_dummy": "false", 2023-07-02T14:01:58.221 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2023-07-02T14:01:58.221 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_inject_migrator_session_race": "false", 2023-07-02T14:01:58.221 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2023-07-02T14:01:58.221 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2023-07-02T14:01:58.222 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_join_fs": "", 2023-07-02T14:01:58.222 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_journal_format": "1", 2023-07-02T14:01:58.222 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_create_at": "0", 2023-07-02T14:01:58.222 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_export_at": "0", 2023-07-02T14:01:58.222 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_import_at": "0", 2023-07-02T14:01:58.222 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_journal_at": "0", 2023-07-02T14:01:58.222 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_journal_expire_at": "0", 2023-07-02T14:01:58.223 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_journal_replay_at": "0", 2023-07-02T14:01:58.223 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_link_at": "0", 2023-07-02T14:01:58.223 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_mdstable_at": "0", 2023-07-02T14:01:58.223 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_openc_at": "0", 2023-07-02T14:01:58.223 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_kill_rename_at": "0", 2023-07-02T14:01:58.223 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_events_per_segment": "1024", 2023-07-02T14:01:58.223 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_max_events": "-1", 2023-07-02T14:01:58.223 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_max_segments": "128", 2023-07-02T14:01:58.224 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_pause": "false", 2023-07-02T14:01:58.224 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_segment_size": "0", 2023-07-02T14:01:58.224 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_skip_corrupt_events": "false", 2023-07-02T14:01:58.224 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_log_warn_factor": "2.000000", 2023-07-02T14:01:58.224 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_caps_per_client": "1048576", 2023-07-02T14:01:58.224 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_completed_flushes": "100000", 2023-07-02T14:01:58.224 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_completed_requests": "100000", 2023-07-02T14:01:58.224 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_export_size": "20971520", 2023-07-02T14:01:58.225 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_file_recover": "32", 2023-07-02T14:01:58.225 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_purge_files": "64", 2023-07-02T14:01:58.225 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_purge_ops": "8192", 2023-07-02T14:01:58.225 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2023-07-02T14:01:58.225 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_scrub_ops_in_progress": "5", 2023-07-02T14:01:58.225 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_snaps_per_dir": "100", 2023-07-02T14:01:58.225 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_max_xattr_pairs_size": "65536", 2023-07-02T14:01:58.225 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_metrics_update_interval": "2", 2023-07-02T14:01:58.226 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_min_caps_per_client": "100", 2023-07-02T14:01:58.226 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_min_caps_working_set": "10000", 2023-07-02T14:01:58.226 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_mon_shutdown_timeout": "5.000000", 2023-07-02T14:01:58.226 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_numa_node": "-1", 2023-07-02T14:01:58.226 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_oft_prefetch_dirfrags": "false", 2023-07-02T14:01:58.226 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_op_complaint_time": "30.000000", 2023-07-02T14:01:58.226 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_op_history_duration": "600", 2023-07-02T14:01:58.226 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_op_history_size": "20", 2023-07-02T14:01:58.227 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_op_log_threshold": "5", 2023-07-02T14:01:58.227 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_ping_grace": "15", 2023-07-02T14:01:58.227 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_ping_interval": "5", 2023-07-02T14:01:58.227 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2023-07-02T14:01:58.227 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_recall_global_max_decay_threshold": "131072", 2023-07-02T14:01:58.227 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_recall_max_caps": "30000", 2023-07-02T14:01:58.227 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_recall_max_decay_rate": "1.500000", 2023-07-02T14:01:58.227 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_recall_max_decay_threshold": "131072", 2023-07-02T14:01:58.228 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_recall_warning_decay_rate": "60.000000", 2023-07-02T14:01:58.228 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_recall_warning_threshold": "262144", 2023-07-02T14:01:58.228 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_reconnect_timeout": "45.000000", 2023-07-02T14:01:58.228 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_replay_interval": "1.000000", 2023-07-02T14:01:58.228 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_replay_unsafe_with_closed_session": "false", 2023-07-02T14:01:58.228 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_request_load_average_decay_rate": "60.000000", 2023-07-02T14:01:58.228 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_root_ino_gid": "0", 2023-07-02T14:01:58.228 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_root_ino_uid": "0", 2023-07-02T14:01:58.229 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_scatter_nudge_interval": "5.000000", 2023-07-02T14:01:58.229 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_blocklist_on_evict": "true", 2023-07-02T14:01:58.229 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_blocklist_on_timeout": "true", 2023-07-02T14:01:58.229 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2023-07-02T14:01:58.229 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_cache_liveness_magnitude": "10", 2023-07-02T14:01:58.229 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_cap_acquisition_decay_rate": "10.000000", 2023-07-02T14:01:58.229 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_cap_acquisition_throttle": "500000", 2023-07-02T14:01:58.229 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2023-07-02T14:01:58.230 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_sessionmap_keys_per_op": "1024", 2023-07-02T14:01:58.230 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_shutdown_check": "0", 2023-07-02T14:01:58.230 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_skip_ino": "0", 2023-07-02T14:01:58.230 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_sleep_rank_change": "0.000000", 2023-07-02T14:01:58.230 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_snap_max_uid": "4294967294", 2023-07-02T14:01:58.230 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_snap_min_uid": "0", 2023-07-02T14:01:58.230 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_snap_rstat": "false", 2023-07-02T14:01:58.231 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_standby_replay_damaged": "false", 2023-07-02T14:01:58.231 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_task_status_update_interval": "2.000000", 2023-07-02T14:01:58.231 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_thrash_exports": "0", 2023-07-02T14:01:58.231 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_thrash_fragments": "0", 2023-07-02T14:01:58.231 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_tick_interval": "5.000000", 2023-07-02T14:01:58.231 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_valgrind_exit": "false", 2023-07-02T14:01:58.231 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_verify_backtrace": "1", 2023-07-02T14:01:58.231 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_verify_scatter": "false", 2023-07-02T14:01:58.232 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_wipe_ino_prealloc": "false", 2023-07-02T14:01:58.232 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_wipe_sessions": "false", 2023-07-02T14:01:58.232 INFO:teuthology.orchestra.run.smithi016.stdout: "mempool_debug": "false", 2023-07-02T14:01:58.232 INFO:teuthology.orchestra.run.smithi016.stdout: "memstore_debug_omit_block_device_write": "false", 2023-07-02T14:01:58.232 INFO:teuthology.orchestra.run.smithi016.stdout: "memstore_device_bytes": "1073741824", 2023-07-02T14:01:58.232 INFO:teuthology.orchestra.run.smithi016.stdout: "memstore_page_set": "false", 2023-07-02T14:01:58.232 INFO:teuthology.orchestra.run.smithi016.stdout: "memstore_page_size": "65536", 2023-07-02T14:01:58.232 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_client_bytes": "134217728", 2023-07-02T14:01:58.233 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_client_messages": "512", 2023-07-02T14:01:58.233 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2023-07-02T14:01:58.233 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_connect_retry_interval": "1.000000", 2023-07-02T14:01:58.233 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2023-07-02T14:01:58.233 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2023-07-02T14:01:58.233 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_disabled_modules": "", 2023-07-02T14:01:58.233 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_initial_modules": "restful iostat nfs", 2023-07-02T14:01:58.233 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_max_pg_num_change": "128", 2023-07-02T14:01:58.234 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_mds_bytes": "134217728", 2023-07-02T14:01:58.234 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_mds_messages": "128", 2023-07-02T14:01:58.234 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2023-07-02T14:01:58.234 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_mon_bytes": "134217728", 2023-07-02T14:01:58.234 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_mon_messages": "128", 2023-07-02T14:01:58.234 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_osd_bytes": "536870912", 2023-07-02T14:01:58.234 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_osd_messages": "8192", 2023-07-02T14:01:58.234 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_service_beacon_grace": "60.000000", 2023-07-02T14:01:58.235 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_standby_modules": "true", 2023-07-02T14:01:58.235 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_stats_period": "5", 2023-07-02T14:01:58.235 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_stats_threshold": "5", 2023-07-02T14:01:58.235 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_tick_period": "2", 2023-07-02T14:01:58.235 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_ttl_cache_expire_seconds": "0", 2023-07-02T14:01:58.235 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_accept_timeout_factor": "2.000000", 2023-07-02T14:01:58.235 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_allow_pool_delete": "true", 2023-07-02T14:01:58.235 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_allow_pool_size_one": "true", 2023-07-02T14:01:58.236 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_auth_validate_all_caps": "true", 2023-07-02T14:01:58.236 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2023-07-02T14:01:58.236 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2023-07-02T14:01:58.236 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_bytes": "104857600", 2023-07-02T14:01:58.236 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_directed_command_retry": "5", 2023-07-02T14:01:58.236 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_hunt_interval": "3.000000", 2023-07-02T14:01:58.236 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2023-07-02T14:01:58.237 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2023-07-02T14:01:58.237 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2023-07-02T14:01:58.237 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_hunt_parallel": "3", 2023-07-02T14:01:58.237 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_log_interval": "1.000000", 2023-07-02T14:01:58.237 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_max_log_entries_per_message": "1000", 2023-07-02T14:01:58.237 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_ping_interval": "10.000000", 2023-07-02T14:01:58.237 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_client_ping_timeout": "30.000000", 2023-07-02T14:01:58.237 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_clock_drift_allowed": "1.000000", 2023-07-02T14:01:58.238 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2023-07-02T14:01:58.238 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2023-07-02T14:01:58.238 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_file_level": "debug", 2023-07-02T14:01:58.238 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_file": "true", 2023-07-02T14:01:58.238 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_graylog": "false", 2023-07-02T14:01:58.238 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2023-07-02T14:01:58.238 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_graylog_port": "12201", 2023-07-02T14:01:58.238 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_stderr": "false", 2023-07-02T14:01:58.239 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_syslog": "default=false", 2023-07-02T14:01:58.239 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2023-07-02T14:01:58.239 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cluster_log_to_syslog_level": "info", 2023-07-02T14:01:58.239 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_compact_on_bootstrap": "false", 2023-07-02T14:01:58.239 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_compact_on_start": "false", 2023-07-02T14:01:58.239 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_compact_on_trim": "true", 2023-07-02T14:01:58.239 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_con_tracker_persist_interval": "10", 2023-07-02T14:01:58.239 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_con_tracker_score_halflife": "43200", 2023-07-02T14:01:58.240 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_config_key_max_entry_size": "65536", 2023-07-02T14:01:58.240 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_cpu_threads": "4", 2023-07-02T14:01:58.240 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_crush_min_required_version": "hammer", 2023-07-02T14:01:58.240 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_daemon_bytes": "419430400", 2023-07-02T14:01:58.240 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2023-07-02T14:01:58.240 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_data_avail_crit": "5", 2023-07-02T14:01:58.240 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_data_avail_warn": "5", 2023-07-02T14:01:58.240 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_data_size_warn": "16106127360", 2023-07-02T14:01:58.241 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_block_osdmap_trim": "false", 2023-07-02T14:01:58.241 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_deprecated_as_obsolete": "false", 2023-07-02T14:01:58.241 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_dump_json": "false", 2023-07-02T14:01:58.241 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2023-07-02T14:01:58.241 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_dump_transactions": "false", 2023-07-02T14:01:58.241 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_extra_checks": "false", 2023-07-02T14:01:58.241 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_no_initial_persistent_features": "false", 2023-07-02T14:01:58.242 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2023-07-02T14:01:58.242 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_no_require_octopus": "false", 2023-07-02T14:01:58.242 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_no_require_pacific": "false", 2023-07-02T14:01:58.242 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2023-07-02T14:01:58.242 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_delta_reset_interval": "10.000000", 2023-07-02T14:01:58.242 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_dns_srv_name": "ceph-mon", 2023-07-02T14:01:58.242 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_down_mkfs_grace": "60", 2023-07-02T14:01:58.242 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_election_default_strategy": "1", 2023-07-02T14:01:58.243 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_election_timeout": "5.000000", 2023-07-02T14:01:58.243 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2023-07-02T14:01:58.243 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_elector_ping_divisor": "2", 2023-07-02T14:01:58.243 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_elector_ping_timeout": "2.000000", 2023-07-02T14:01:58.243 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_enable_op_tracker": "true", 2023-07-02T14:01:58.243 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_fake_pool_delete": "false", 2023-07-02T14:01:58.243 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_force_quorum_join": "false", 2023-07-02T14:01:58.243 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_globalid_prealloc": "10000", 2023-07-02T14:01:58.244 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_health_detail_to_clog": "false", 2023-07-02T14:01:58.244 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_health_log_update_period": "5", 2023-07-02T14:01:58.244 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_health_max_detail": "50", 2023-07-02T14:01:58.244 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_health_to_clog": "true", 2023-07-02T14:01:58.244 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_health_to_clog_interval": "600", 2023-07-02T14:01:58.244 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2023-07-02T14:01:58.244 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_host": "172.21.15.16,[v2:172.21.15.16:3301,v1:172.21.15.16:6790],172.21.15.22", 2023-07-02T14:01:58.244 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_host_override": "", 2023-07-02T14:01:58.245 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_initial_members": "", 2023-07-02T14:01:58.245 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2023-07-02T14:01:58.245 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2023-07-02T14:01:58.245 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_inject_transaction_delay_max": "10.000000", 2023-07-02T14:01:58.245 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2023-07-02T14:01:58.245 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_keyvaluedb": "rocksdb", 2023-07-02T14:01:58.245 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_lease": "5.000000", 2023-07-02T14:01:58.245 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2023-07-02T14:01:58.246 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_lease_renew_interval_factor": "0.600000", 2023-07-02T14:01:58.246 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_log_max_summary": "50", 2023-07-02T14:01:58.246 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_log_entries_per_event": "4096", 2023-07-02T14:01:58.246 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_log_epochs": "500", 2023-07-02T14:01:58.246 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_mdsmap_epochs": "500", 2023-07-02T14:01:58.246 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_mgrmap_epochs": "500", 2023-07-02T14:01:58.246 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_osd": "10000", 2023-07-02T14:01:58.246 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_pg_per_osd": "10000", 2023-07-02T14:01:58.247 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_pool_pg_num": "65536", 2023-07-02T14:01:58.247 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_max_snap_prune_per_epoch": "100", 2023-07-02T14:01:58.247 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mds_blocklist_interval": "86400.000000", 2023-07-02T14:01:58.247 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mds_force_trim_to": "0", 2023-07-02T14:01:58.247 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mds_skip_sanity": "false", 2023-07-02T14:01:58.247 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_memory_autotune": "true", 2023-07-02T14:01:58.247 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_memory_target": "2147483648", 2023-07-02T14:01:58.247 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mgr_beacon_grace": "30", 2023-07-02T14:01:58.248 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2023-07-02T14:01:58.248 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mgr_digest_period": "5", 2023-07-02T14:01:58.248 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mgr_inactive_grace": "60", 2023-07-02T14:01:58.248 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mgr_mkfs_grace": "240", 2023-07-02T14:01:58.248 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2023-07-02T14:01:58.248 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_min_osdmap_epochs": "500", 2023-07-02T14:01:58.248 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_op_complaint_time": "30", 2023-07-02T14:01:58.249 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_op_history_duration": "600", 2023-07-02T14:01:58.249 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_op_history_size": "20", 2023-07-02T14:01:58.249 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_op_history_slow_op_size": "20", 2023-07-02T14:01:58.249 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_op_history_slow_op_threshold": "10", 2023-07-02T14:01:58.249 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_op_log_threshold": "5", 2023-07-02T14:01:58.249 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_adjust_down_out_interval": "true", 2023-07-02T14:01:58.249 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2023-07-02T14:01:58.249 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2023-07-02T14:01:58.250 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_auto_mark_in": "false", 2023-07-02T14:01:58.250 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_auto_mark_new_in": "true", 2023-07-02T14:01:58.250 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2023-07-02T14:01:58.250 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2023-07-02T14:01:58.250 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_cache_size": "500", 2023-07-02T14:01:58.250 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_cache_size_min": "134217728", 2023-07-02T14:01:58.250 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_crush_smoke_test": "true", 2023-07-02T14:01:58.250 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_destroyed_out_interval": "600", 2023-07-02T14:01:58.251 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_down_out_interval": "600", 2023-07-02T14:01:58.251 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_down_out_subtree_limit": "rack", 2023-07-02T14:01:58.251 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2023-07-02T14:01:58.251 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_force_trim_to": "0", 2023-07-02T14:01:58.251 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_full_ratio": "0.950000", 2023-07-02T14:01:58.251 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2023-07-02T14:01:58.251 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_laggy_halflife": "3600", 2023-07-02T14:01:58.251 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_laggy_max_interval": "300", 2023-07-02T14:01:58.252 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_laggy_weight": "0.300000", 2023-07-02T14:01:58.252 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2023-07-02T14:01:58.252 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_max_creating_pgs": "1024", 2023-07-02T14:01:58.252 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_max_initial_pgs": "1024", 2023-07-02T14:01:58.252 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_min_down_reporters": "2", 2023-07-02T14:01:58.252 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_min_in_ratio": "0.750000", 2023-07-02T14:01:58.252 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_min_up_ratio": "0.300000", 2023-07-02T14:01:58.252 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_nearfull_ratio": "0.850000", 2023-07-02T14:01:58.253 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_prime_pg_temp": "true", 2023-07-02T14:01:58.253 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2023-07-02T14:01:58.253 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2023-07-02T14:01:58.253 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_report_timeout": "900", 2023-07-02T14:01:58.253 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_reporter_subtree_level": "osd", 2023-07-02T14:01:58.253 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2023-07-02T14:01:58.253 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_warn_num_repaired": "10", 2023-07-02T14:01:58.253 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osd_warn_op_age": "32.000000", 2023-07-02T14:01:58.254 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osdmap_full_prune_enabled": "true", 2023-07-02T14:01:58.254 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osdmap_full_prune_interval": "10", 2023-07-02T14:01:58.254 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osdmap_full_prune_min": "10000", 2023-07-02T14:01:58.254 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_osdmap_full_prune_txsize": "100", 2023-07-02T14:01:58.254 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2023-07-02T14:01:58.254 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pg_stuck_threshold": "60", 2023-07-02T14:01:58.254 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2023-07-02T14:01:58.255 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pg_warn_min_objects": "10000", 2023-07-02T14:01:58.255 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pg_warn_min_per_osd": "0", 2023-07-02T14:01:58.255 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pg_warn_min_pool_objects": "1000", 2023-07-02T14:01:58.255 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pool_quota_crit_threshold": "0", 2023-07-02T14:01:58.255 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_pool_quota_warn_threshold": "0", 2023-07-02T14:01:58.255 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_probe_timeout": "2.000000", 2023-07-02T14:01:58.255 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_reweight_max_change": "0.050000", 2023-07-02T14:01:58.255 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_reweight_max_osds": "4", 2023-07-02T14:01:58.256 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_reweight_min_bytes_per_osd": "10", 2023-07-02T14:01:58.256 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_reweight_min_pgs_per_osd": "4", 2023-07-02T14:01:58.256 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2023-07-02T14:01:58.256 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2023-07-02T14:01:58.256 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2023-07-02T14:01:58.256 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_scrub_interval": "86400", 2023-07-02T14:01:58.256 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_scrub_max_keys": "100", 2023-07-02T14:01:58.256 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_scrub_timeout": "300", 2023-07-02T14:01:58.257 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_session_timeout": "300", 2023-07-02T14:01:58.257 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_smart_report_timeout": "5", 2023-07-02T14:01:58.257 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_stat_smooth_intervals": "6", 2023-07-02T14:01:58.257 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2023-07-02T14:01:58.257 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_stretch_pool_min_size": "2", 2023-07-02T14:01:58.257 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_stretch_pool_size": "4", 2023-07-02T14:01:58.257 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2023-07-02T14:01:58.257 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_subscribe_interval": "86400.000000", 2023-07-02T14:01:58.258 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_sync_debug": "false", 2023-07-02T14:01:58.258 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_sync_max_payload_keys": "2000", 2023-07-02T14:01:58.258 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_sync_max_payload_size": "1048576", 2023-07-02T14:01:58.258 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_sync_provider_kill_at": "0", 2023-07-02T14:01:58.258 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_sync_requester_kill_at": "0", 2023-07-02T14:01:58.258 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_sync_timeout": "60.000000", 2023-07-02T14:01:58.258 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_target_pg_per_osd": "100", 2023-07-02T14:01:58.258 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_tick_interval": "5", 2023-07-02T14:01:58.259 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_timecheck_interval": "300.000000", 2023-07-02T14:01:58.259 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_timecheck_skew_interval": "30.000000", 2023-07-02T14:01:58.259 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_older_version_delay": "604800", 2023-07-02T14:01:58.259 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2023-07-02T14:01:58.259 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2023-07-02T14:01:58.259 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2023-07-02T14:01:58.259 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2023-07-02T14:01:58.259 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2023-07-02T14:01:58.260 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2023-07-02T14:01:58.260 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_misplaced": "false", 2023-07-02T14:01:58.260 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2023-07-02T14:01:58.260 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_older_version": "true", 2023-07-02T14:01:58.260 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2023-07-02T14:01:58.260 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_pool_no_app": "true", 2023-07-02T14:01:58.260 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_pool_no_redundancy": "false", 2023-07-02T14:01:58.261 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2023-07-02T14:01:58.261 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2023-07-02T14:01:58.261 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2023-07-02T14:01:58.261 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_on_too_few_osds": "false", 2023-07-02T14:01:58.261 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2023-07-02T14:01:58.261 INFO:teuthology.orchestra.run.smithi016.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2023-07-02T14:01:58.261 INFO:teuthology.orchestra.run.smithi016.stdout: "monmap": "", 2023-07-02T14:01:58.261 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_max_op_threads": "5", 2023-07-02T14:01:58.262 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_op_threads": "3", 2023-07-02T14:01:58.262 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_buffer_size": "131072", 2023-07-02T14:01:58.262 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_cm": "false", 2023-07-02T14:01:58.262 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_device_name": "", 2023-07-02T14:01:58.262 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_dscp": "96", 2023-07-02T14:01:58.262 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_enable_hugepage": "false", 2023-07-02T14:01:58.262 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_gid_idx": "0", 2023-07-02T14:01:58.262 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_local_gid": "", 2023-07-02T14:01:58.263 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_polling_us": "1000", 2023-07-02T14:01:58.263 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_port_num": "1", 2023-07-02T14:01:58.263 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_receive_buffers": "32768", 2023-07-02T14:01:58.263 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_receive_queue_len": "4096", 2023-07-02T14:01:58.263 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_roce_ver": "1", 2023-07-02T14:01:58.263 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_send_buffers": "1024", 2023-07-02T14:01:58.263 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_sl": "3", 2023-07-02T14:01:58.263 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_support_srq": "true", 2023-07-02T14:01:58.264 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_rdma_type": "ib", 2023-07-02T14:01:58.264 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_async_reap_threshold": "5", 2023-07-02T14:01:58.264 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_before_connect": "false", 2023-07-02T14:01:58.264 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_ipv4": "true", 2023-07-02T14:01:58.264 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_ipv6": "false", 2023-07-02T14:01:58.264 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_msgr1": "true", 2023-07-02T14:01:58.264 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_msgr2": "true", 2023-07-02T14:01:58.265 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_port_max": "7300", 2023-07-02T14:01:58.265 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_port_min": "6800", 2023-07-02T14:01:58.265 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_prefer_ipv4": "false", 2023-07-02T14:01:58.265 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_retry_count": "3", 2023-07-02T14:01:58.265 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_bind_retry_delay": "5", 2023-07-02T14:01:58.265 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_blackhole_client": "false", 2023-07-02T14:01:58.265 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_blackhole_mds": "false", 2023-07-02T14:01:58.265 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_blackhole_mgr": "false", 2023-07-02T14:01:58.266 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_blackhole_mon": "false", 2023-07-02T14:01:58.266 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_blackhole_osd": "false", 2023-07-02T14:01:58.266 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_client_mode": "crc secure", 2023-07-02T14:01:58.266 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_cluster_mode": "crc secure", 2023-07-02T14:01:58.266 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_cluster_type": "", 2023-07-02T14:01:58.266 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_connection_idle_timeout": "900", 2023-07-02T14:01:58.266 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_connection_ready_timeout": "10", 2023-07-02T14:01:58.266 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_crc_data": "true", 2023-07-02T14:01:58.267 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_crc_header": "true", 2023-07-02T14:01:58.267 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_die_on_bad_msg": "false", 2023-07-02T14:01:58.267 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_die_on_bug": "true", 2023-07-02T14:01:58.267 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_die_on_old_message": "true", 2023-07-02T14:01:58.267 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_die_on_skipped_message": "false", 2023-07-02T14:01:58.267 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_die_on_unhandled_msg": "false", 2023-07-02T14:01:58.267 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dispatch_throttle_bytes": "104857600", 2023-07-02T14:01:58.267 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_coremask": "0xF", 2023-07-02T14:01:58.268 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_debug_allow_loopback": "false", 2023-07-02T14:01:58.268 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2023-07-02T14:01:58.268 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_host_ipv4_addr": "", 2023-07-02T14:01:58.268 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_hugepages": "", 2023-07-02T14:01:58.268 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_hw_flow_control": "true", 2023-07-02T14:01:58.268 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2023-07-02T14:01:58.268 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_lro": "true", 2023-07-02T14:01:58.268 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_memory_channel": "4", 2023-07-02T14:01:58.269 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2023-07-02T14:01:58.269 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_pmd": "", 2023-07-02T14:01:58.269 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_port_id": "0", 2023-07-02T14:01:58.269 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2023-07-02T14:01:58.269 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dump_corrupt_message_level": "1", 2023-07-02T14:01:58.269 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_dump_on_send": "false", 2023-07-02T14:01:58.269 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_initial_backoff": "0.200000", 2023-07-02T14:01:58.270 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_inject_delay_max": "1.000000", 2023-07-02T14:01:58.270 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_inject_delay_probability": "0.000000", 2023-07-02T14:01:58.270 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_inject_delay_type": "", 2023-07-02T14:01:58.270 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_inject_internal_delays": "0.000000", 2023-07-02T14:01:58.270 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_inject_socket_failures": "5000", 2023-07-02T14:01:58.270 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_learn_addr_from_peer": "true", 2023-07-02T14:01:58.270 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_max_accept_failures": "4", 2023-07-02T14:01:58.270 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_max_backoff": "15.000000", 2023-07-02T14:01:58.271 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_mon_client_mode": "secure crc", 2023-07-02T14:01:58.271 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_mon_cluster_mode": "secure crc", 2023-07-02T14:01:58.271 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_mon_service_mode": "secure crc", 2023-07-02T14:01:58.271 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2023-07-02T14:01:58.271 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_pq_min_cost": "65536", 2023-07-02T14:01:58.271 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_public_type": "", 2023-07-02T14:01:58.271 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_service_mode": "crc secure", 2023-07-02T14:01:58.271 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_tcp_listen_backlog": "512", 2023-07-02T14:01:58.272 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_tcp_nodelay": "true", 2023-07-02T14:01:58.272 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_tcp_prefetch_max_size": "4096", 2023-07-02T14:01:58.272 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_tcp_rcvbuf": "0", 2023-07-02T14:01:58.272 INFO:teuthology.orchestra.run.smithi016.stdout: "ms_type": "async+posix", 2023-07-02T14:01:58.272 INFO:teuthology.orchestra.run.smithi016.stdout: "no_config_file": "false", 2023-07-02T14:01:58.272 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_completion_locks_per_session": "32", 2023-07-02T14:01:58.272 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_debug_inject_relock_delay": "false", 2023-07-02T14:01:58.272 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_inflight_op_bytes": "104857600", 2023-07-02T14:01:58.273 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_inflight_ops": "1024", 2023-07-02T14:01:58.273 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_inject_no_watch_ping": "false", 2023-07-02T14:01:58.273 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_retry_writes_after_first_reply": "false", 2023-07-02T14:01:58.273 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_tick_interval": "5.000000", 2023-07-02T14:01:58.273 INFO:teuthology.orchestra.run.smithi016.stdout: "objecter_timeout": "10.000000", 2023-07-02T14:01:58.273 INFO:teuthology.orchestra.run.smithi016.stdout: "objectstore_blackhole": "false", 2023-07-02T14:01:58.273 INFO:teuthology.orchestra.run.smithi016.stdout: "openssl_engine_opts": "", 2023-07-02T14:01:58.273 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_delay_time": "5.000000", 2023-07-02T14:01:58.274 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_hist_halflife": "1000", 2023-07-02T14:01:58.274 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_max_low_ops": "2", 2023-07-02T14:01:58.274 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_max_ops": "4", 2023-07-02T14:01:58.274 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_min_evict_effort": "0.100000", 2023-07-02T14:01:58.274 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_quantize_effort": "0.100000", 2023-07-02T14:01:58.274 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_agent_slop": "0.020000", 2023-07-02T14:01:58.274 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_aggregated_slow_ops_logging": "true", 2023-07-02T14:01:58.274 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_allow_recovery_below_min_size": "true", 2023-07-02T14:01:58.275 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_asio_thread_count": "2", 2023-07-02T14:01:58.275 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_async_recovery_min_cost": "100", 2023-07-02T14:01:58.275 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_auto_mark_unfound_lost": "false", 2023-07-02T14:01:58.275 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_backfill_retry_interval": "30.000000", 2023-07-02T14:01:58.275 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_backfill_scan_max": "512", 2023-07-02T14:01:58.275 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_backfill_scan_min": "64", 2023-07-02T14:01:58.275 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_backoff_on_degraded": "false", 2023-07-02T14:01:58.275 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_backoff_on_peering": "false", 2023-07-02T14:01:58.276 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_backoff_on_unfound": "true", 2023-07-02T14:01:58.276 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_beacon_report_interval": "300", 2023-07-02T14:01:58.276 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bench_duration": "30", 2023-07-02T14:01:58.276 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bench_large_size_max_throughput": "104857600", 2023-07-02T14:01:58.276 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bench_max_block_size": "67108864", 2023-07-02T14:01:58.276 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_bench_small_size_max_iops": "100", 2023-07-02T14:01:58.276 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_blkin_trace_all": "false", 2023-07-02T14:01:58.277 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2023-07-02T14:01:58.277 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2023-07-02T14:01:58.277 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_check_for_log_corruption": "false", 2023-07-02T14:01:58.277 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_check_max_object_name_len_on_startup": "true", 2023-07-02T14:01:58.277 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2023-07-02T14:01:58.277 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2023-07-02T14:01:58.277 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2023-07-02T14:01:58.277 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_class_update_on_start": "true", 2023-07-02T14:01:58.278 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_client_message_cap": "256", 2023-07-02T14:01:58.278 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_client_message_size_cap": "524288000", 2023-07-02T14:01:58.278 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_client_op_priority": "63", 2023-07-02T14:01:58.278 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_client_watch_timeout": "30", 2023-07-02T14:01:58.278 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_command_max_records": "256", 2023-07-02T14:01:58.278 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_command_thread_suicide_timeout": "900", 2023-07-02T14:01:58.278 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_command_thread_timeout": "600", 2023-07-02T14:01:58.278 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_compact_on_start": "false", 2023-07-02T14:01:58.279 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_copyfrom_max_chunk": "8388608", 2023-07-02T14:01:58.279 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_crush_chooseleaf_type": "0", 2023-07-02T14:01:58.279 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_crush_initial_weight": "-1.000000", 2023-07-02T14:01:58.279 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_crush_update_on_start": "true", 2023-07-02T14:01:58.279 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_crush_update_weight_set": "true", 2023-07-02T14:01:58.279 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2023-07-02T14:01:58.280 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2023-07-02T14:01:58.280 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2023-07-02T14:01:58.280 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_drop_ping_duration": "0", 2023-07-02T14:01:58.280 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_drop_ping_probability": "0.000000", 2023-07-02T14:01:58.280 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_feed_pullee": "-1", 2023-07-02T14:01:58.280 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_inject_copyfrom_error": "false", 2023-07-02T14:01:58.280 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2023-07-02T14:01:58.280 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2023-07-02T14:01:58.281 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_misdirected_ops": "false", 2023-07-02T14:01:58.281 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_no_acting_change": "false", 2023-07-02T14:01:58.281 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_no_purge_strays": "false", 2023-07-02T14:01:58.281 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_op_order": "false", 2023-07-02T14:01:58.281 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_pg_log_writeout": "false", 2023-07-02T14:01:58.281 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_pretend_recovery_active": "false", 2023-07-02T14:01:58.281 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_random_push_read_error": "0.000000", 2023-07-02T14:01:58.281 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2023-07-02T14:01:58.282 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_shutdown": "false", 2023-07-02T14:01:58.282 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2023-07-02T14:01:58.282 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2023-07-02T14:01:58.282 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_verify_cached_snaps": "false", 2023-07-02T14:01:58.282 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_verify_missing_on_start": "false", 2023-07-02T14:01:58.282 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_verify_snaps": "false", 2023-07-02T14:01:58.282 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_debug_verify_stray_on_activate": "false", 2023-07-02T14:01:58.282 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_interval": "604800.000000", 2023-07-02T14:01:58.283 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_keys": "1024", 2023-07-02T14:01:58.283 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2023-07-02T14:01:58.283 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2023-07-02T14:01:58.283 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2023-07-02T14:01:58.283 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_stride": "524288", 2023-07-02T14:01:58.283 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2023-07-02T14:01:58.283 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_default_data_pool_replay_window": "5", 2023-07-02T14:01:58.283 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_default_notify_timeout": "30", 2023-07-02T14:01:58.284 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_delete_sleep": "0.000000", 2023-07-02T14:01:58.284 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_delete_sleep_hdd": "5.000000", 2023-07-02T14:01:58.284 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_delete_sleep_hybrid": "1.000000", 2023-07-02T14:01:58.284 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_delete_sleep_ssd": "1.000000", 2023-07-02T14:01:58.284 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_discard_disconnected_ops": "true", 2023-07-02T14:01:58.284 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_enable_op_tracker": "true", 2023-07-02T14:01:58.284 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_erasure_code_plugins": "jerasure lrc isa", 2023-07-02T14:01:58.284 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_failsafe_full_ratio": "0.970000", 2023-07-02T14:01:58.285 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_fast_fail_on_connection_refused": "true", 2023-07-02T14:01:58.285 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_fast_info": "true", 2023-07-02T14:01:58.285 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_fast_shutdown": "true", 2023-07-02T14:01:58.285 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_fast_shutdown_notify_mon": "true", 2023-07-02T14:01:58.285 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_find_best_info_ignore_history_les": "false", 2023-07-02T14:01:58.285 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_force_auth_primary_missing_objects": "100", 2023-07-02T14:01:58.285 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2023-07-02T14:01:58.285 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_function_tracing": "false", 2023-07-02T14:01:58.286 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_grace": "20", 2023-07-02T14:01:58.286 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_interval": "6", 2023-07-02T14:01:58.286 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2023-07-02T14:01:58.286 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_min_peers": "10", 2023-07-02T14:01:58.286 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_min_size": "2000", 2023-07-02T14:01:58.286 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_stale": "600", 2023-07-02T14:01:58.286 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2023-07-02T14:01:58.287 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_hit_set_max_size": "100000", 2023-07-02T14:01:58.287 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_hit_set_min_size": "1000", 2023-07-02T14:01:58.287 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_hit_set_namespace": ".ceph-internal", 2023-07-02T14:01:58.287 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_ignore_stale_divergent_priors": "false", 2023-07-02T14:01:58.287 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2023-07-02T14:01:58.287 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_inject_failure_on_pg_removal": "false", 2023-07-02T14:01:58.287 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2023-07-02T14:01:58.287 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_journal_flush_on_shutdown": "true", 2023-07-02T14:01:58.288 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_journal_size": "5120", 2023-07-02T14:01:58.288 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_kill_backfill_at": "0", 2023-07-02T14:01:58.288 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_loop_before_reset_tphandle": "64", 2023-07-02T14:01:58.288 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_map_cache_size": "50", 2023-07-02T14:01:58.288 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_map_dedup": "true", 2023-07-02T14:01:58.288 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_map_message_max": "40", 2023-07-02T14:01:58.288 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_map_message_max_bytes": "10485760", 2023-07-02T14:01:58.289 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_map_share_max_epochs": "40", 2023-07-02T14:01:58.289 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_attr_name_len": "100", 2023-07-02T14:01:58.289 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_attr_size": "0", 2023-07-02T14:01:58.289 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_backfills": "1", 2023-07-02T14:01:58.289 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_markdown_count": "5", 2023-07-02T14:01:58.289 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_markdown_period": "600", 2023-07-02T14:01:58.289 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_object_name_len": "2048", 2023-07-02T14:01:58.289 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_object_namespace_len": "256", 2023-07-02T14:01:58.290 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_object_size": "134217728", 2023-07-02T14:01:58.290 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2023-07-02T14:01:58.290 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_omap_entries_per_request": "1024", 2023-07-02T14:01:58.290 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_pg_blocked_by": "16", 2023-07-02T14:01:58.290 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_pg_log_entries": "10000", 2023-07-02T14:01:58.290 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2023-07-02T14:01:58.290 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_pgls": "1024", 2023-07-02T14:01:58.290 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_push_cost": "8388608", 2023-07-02T14:01:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_push_objects": "10", 2023-07-02T14:01:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_scrubs": "1", 2023-07-02T14:01:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2023-07-02T14:01:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_trimming_pgs": "2", 2023-07-02T14:01:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_write_op_reply_len": "32", 2023-07-02T14:01:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_max_write_size": "90", 2023-07-02T14:01:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_cost_per_byte_usec": "0.000000", 2023-07-02T14:01:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_cost_per_byte_usec_hdd": "5.200000", 2023-07-02T14:01:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_cost_per_byte_usec_ssd": "0.011000", 2023-07-02T14:01:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_cost_per_io_usec": "0.000000", 2023-07-02T14:01:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_cost_per_io_usec_hdd": "25000.000000", 2023-07-02T14:01:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_cost_per_io_usec_ssd": "50.000000", 2023-07-02T14:01:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2023-07-02T14:01:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2023-07-02T14:01:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2023-07-02T14:01:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_profile": "high_client_ops", 2023-07-02T14:01:58.293 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2023-07-02T14:01:58.293 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_background_best_effort_lim": "999999", 2023-07-02T14:01:58.293 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_background_best_effort_res": "1", 2023-07-02T14:01:58.293 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2023-07-02T14:01:58.293 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_background_recovery_lim": "999999", 2023-07-02T14:01:58.293 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_background_recovery_res": "1", 2023-07-02T14:01:58.293 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2023-07-02T14:01:58.294 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_client_lim": "999999", 2023-07-02T14:01:58.294 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_client_res": "1", 2023-07-02T14:01:58.294 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_scheduler_client_wgt": "1", 2023-07-02T14:01:58.294 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mclock_skip_benchmark": "false", 2023-07-02T14:01:58.294 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_base": "805306368", 2023-07-02T14:01:58.294 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_cache_min": "134217728", 2023-07-02T14:01:58.294 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_cache_resize_interval": "1.000000", 2023-07-02T14:01:58.294 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_expected_fragmentation": "0.150000", 2023-07-02T14:01:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_target": "4294967296", 2023-07-02T14:01:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_target_autotune": "false", 2023-07-02T14:01:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2023-07-02T14:01:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_min_pg_log_entries": "250", 2023-07-02T14:01:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_min_recovery_priority": "0", 2023-07-02T14:01:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mon_heartbeat_interval": "30", 2023-07-02T14:01:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2023-07-02T14:01:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mon_report_interval": "5", 2023-07-02T14:01:58.296 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mon_report_max_in_flight": "2", 2023-07-02T14:01:58.296 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_mon_shutdown_timeout": "5.000000", 2023-07-02T14:01:58.296 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_num_cache_shards": "32", 2023-07-02T14:01:58.296 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_num_op_tracker_shard": "32", 2023-07-02T14:01:58.296 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_numa_auto_affinity": "true", 2023-07-02T14:01:58.296 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_numa_node": "-1", 2023-07-02T14:01:58.296 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_numa_prefer_iface": "true", 2023-07-02T14:01:58.296 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_object_clean_region_max_num_intervals": "10", 2023-07-02T14:01:58.297 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_objecter_finishers": "1", 2023-07-02T14:01:58.297 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_objectstore": "bluestore", 2023-07-02T14:01:58.297 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_objectstore_fuse": "false", 2023-07-02T14:01:58.297 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_objectstore_tracing": "false", 2023-07-02T14:01:58.297 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_complaint_time": "30.000000", 2023-07-02T14:01:58.297 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_history_duration": "600", 2023-07-02T14:01:58.297 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_history_size": "20", 2023-07-02T14:01:58.297 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_history_slow_op_size": "20", 2023-07-02T14:01:58.298 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2023-07-02T14:01:58.298 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_log_threshold": "5", 2023-07-02T14:01:58.298 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_num_shards": "0", 2023-07-02T14:01:58.298 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_num_shards_hdd": "5", 2023-07-02T14:01:58.298 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_num_shards_ssd": "8", 2023-07-02T14:01:58.298 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_num_threads_per_shard": "0", 2023-07-02T14:01:58.298 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2023-07-02T14:01:58.298 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2023-07-02T14:01:58.299 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2023-07-02T14:01:58.299 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_pq_min_cost": "65536", 2023-07-02T14:01:58.299 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_queue": "wpq", 2023-07-02T14:01:58.299 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_queue_cut_off": "high", 2023-07-02T14:01:58.299 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_thread_suicide_timeout": "150", 2023-07-02T14:01:58.299 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_op_thread_timeout": "15", 2023-07-02T14:01:58.299 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_open_classes_on_start": "true", 2023-07-02T14:01:58.299 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_os_flags": "0", 2023-07-02T14:01:58.300 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_peering_op_priority": "255", 2023-07-02T14:01:58.300 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_delete_cost": "1048576", 2023-07-02T14:01:58.300 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_delete_priority": "5", 2023-07-02T14:01:58.300 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2023-07-02T14:01:58.300 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2023-07-02T14:01:58.300 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_log_dups_tracked": "3000", 2023-07-02T14:01:58.300 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_log_trim_max": "10000", 2023-07-02T14:01:58.300 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_log_trim_min": "100", 2023-07-02T14:01:58.301 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2023-07-02T14:01:58.301 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_object_context_cache_count": "64", 2023-07-02T14:01:58.301 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pg_stat_report_interval_max": "500", 2023-07-02T14:01:58.301 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2023-07-02T14:01:58.301 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_cache_min_evict_age": "0", 2023-07-02T14:01:58.301 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_cache_min_flush_age": "0", 2023-07-02T14:01:58.301 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2023-07-02T14:01:58.301 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2023-07-02T14:01:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2023-07-02T14:01:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_crush_rule": "-1", 2023-07-02T14:01:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_ec_fast_read": "false", 2023-07-02T14:01:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd", 2023-07-02T14:01:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_flag_bulk": "false", 2023-07-02T14:01:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_flag_hashpspool": "true", 2023-07-02T14:01:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_flag_nodelete": "false", 2023-07-02T14:01:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_flag_nopgchange": "false", 2023-07-02T14:01:58.303 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_flag_nosizechange": "false", 2023-07-02T14:01:58.303 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_flags": "0", 2023-07-02T14:01:58.303 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2023-07-02T14:01:58.303 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_min_size": "0", 2023-07-02T14:01:58.303 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2023-07-02T14:01:58.303 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_pg_num": "32", 2023-07-02T14:01:58.303 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_pgp_num": "0", 2023-07-02T14:01:58.303 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2023-07-02T14:01:58.304 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_size": "2", 2023-07-02T14:01:58.304 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_default_type": "replicated", 2023-07-02T14:01:58.304 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2023-07-02T14:01:58.304 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_pool_use_gmt_hitset": "true", 2023-07-02T14:01:58.304 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_push_per_object_cost": "1000", 2023-07-02T14:01:58.304 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_read_ec_check_for_errors": "false", 2023-07-02T14:01:58.304 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recover_clone_overlap": "true", 2023-07-02T14:01:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recover_clone_overlap_limit": "10", 2023-07-02T14:01:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_cost": "20971520", 2023-07-02T14:01:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_delay_start": "0.000000", 2023-07-02T14:01:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_max_active": "0", 2023-07-02T14:01:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_max_active_hdd": "3", 2023-07-02T14:01:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_max_active_ssd": "10", 2023-07-02T14:01:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_max_chunk": "8388608", 2023-07-02T14:01:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2023-07-02T14:01:58.306 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_max_single_start": "1", 2023-07-02T14:01:58.306 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_op_priority": "3", 2023-07-02T14:01:58.306 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_op_warn_multiple": "16", 2023-07-02T14:01:58.306 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_priority": "5", 2023-07-02T14:01:58.306 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_retry_interval": "30.000000", 2023-07-02T14:01:58.306 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_sleep": "0.000000", 2023-07-02T14:01:58.306 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_sleep_hdd": "0.100000", 2023-07-02T14:01:58.306 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2023-07-02T14:01:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_recovery_sleep_ssd": "0.000000", 2023-07-02T14:01:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_repair_during_recovery": "false", 2023-07-02T14:01:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_requested_scrub_priority": "120", 2023-07-02T14:01:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2023-07-02T14:01:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_rollback_to_cluster_snap": "", 2023-07-02T14:01:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_auto_repair": "false", 2023-07-02T14:01:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_auto_repair_num_errors": "5", 2023-07-02T14:01:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_backoff_ratio": "0.660000", 2023-07-02T14:01:58.308 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_begin_hour": "0", 2023-07-02T14:01:58.308 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_begin_week_day": "0", 2023-07-02T14:01:58.308 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_chunk_max": "25", 2023-07-02T14:01:58.308 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_chunk_min": "5", 2023-07-02T14:01:58.308 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_cost": "52428800", 2023-07-02T14:01:58.308 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_during_recovery": "false", 2023-07-02T14:01:58.308 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_end_hour": "0", 2023-07-02T14:01:58.308 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_end_week_day": "0", 2023-07-02T14:01:58.309 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_extended_sleep": "0.000000", 2023-07-02T14:01:58.309 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2023-07-02T14:01:58.309 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_invalid_stats": "true", 2023-07-02T14:01:58.309 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_load_threshold": "0.500000", 2023-07-02T14:01:58.309 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_max_interval": "604800.000000", 2023-07-02T14:01:58.309 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_max_preemptions": "5", 2023-07-02T14:01:58.310 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_min_interval": "86400.000000", 2023-07-02T14:01:58.310 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_priority": "5", 2023-07-02T14:01:58.310 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_scrub_sleep": "0.000000", 2023-07-02T14:01:58.310 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_shutdown_pgref_assert": "false", 2023-07-02T14:01:58.310 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_skip_data_digest": "false", 2023-07-02T14:01:58.310 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_smart_report_timeout": "5", 2023-07-02T14:01:58.310 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_snap_trim_cost": "1048576", 2023-07-02T14:01:58.310 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_snap_trim_priority": "5", 2023-07-02T14:01:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_snap_trim_sleep": "0.000000", 2023-07-02T14:01:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2023-07-02T14:01:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2023-07-02T14:01:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2023-07-02T14:01:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2023-07-02T14:01:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_target_transaction_size": "30", 2023-07-02T14:01:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_hit_set_count": "4", 2023-07-02T14:01:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2023-07-02T14:01:58.312 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2023-07-02T14:01:58.312 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2023-07-02T14:01:58.312 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2023-07-02T14:01:58.312 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2023-07-02T14:01:58.312 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2023-07-02T14:01:58.312 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_default_cache_mode": "writeback", 2023-07-02T14:01:58.312 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2023-07-02T14:01:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tier_promote_max_objects_sec": "25", 2023-07-02T14:01:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_tracing": "false", 2023-07-02T14:01:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_use_stale_snap": "false", 2023-07-02T14:01:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2023-07-02T14:01:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: "osdc_blkin_trace_all": "false", 2023-07-02T14:01:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_kill_at": "0", 2023-07-02T14:01:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_max_join_drift": "10", 2023-07-02T14:01:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_min": "500", 2023-07-02T14:01:58.314 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_min_wait": "0.050000", 2023-07-02T14:01:58.314 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_propose_interval": "1.000000", 2023-07-02T14:01:58.314 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_service_trim_max": "500", 2023-07-02T14:01:58.314 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_service_trim_max_multiplier": "20", 2023-07-02T14:01:58.314 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_service_trim_min": "250", 2023-07-02T14:01:58.314 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_stash_full_interval": "25", 2023-07-02T14:01:58.314 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_trim_max": "500", 2023-07-02T14:01:58.314 INFO:teuthology.orchestra.run.smithi016.stdout: "paxos_trim_min": "250", 2023-07-02T14:01:58.315 INFO:teuthology.orchestra.run.smithi016.stdout: "perf": "true", 2023-07-02T14:01:58.315 INFO:teuthology.orchestra.run.smithi016.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2023-07-02T14:01:58.315 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin_crypto_accelerator": "crypto_isal", 2023-07-02T14:01:58.315 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin_dir": "/usr/lib64/ceph", 2023-07-02T14:01:58.315 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "-", 2023-07-02T14:01:58.315 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrv": "", 2023-07-02T14:01:58.315 INFO:teuthology.orchestra.run.smithi016.stdout: "public_bind_addr": "-", 2023-07-02T14:01:58.315 INFO:teuthology.orchestra.run.smithi016.stdout: "public_network": "", 2023-07-02T14:01:58.316 INFO:teuthology.orchestra.run.smithi016.stdout: "public_network_interface": "", 2023-07-02T14:01:58.316 INFO:teuthology.orchestra.run.smithi016.stdout: "qat_compressor_enabled": "false", 2023-07-02T14:01:58.316 INFO:teuthology.orchestra.run.smithi016.stdout: "rados_mon_op_timeout": "0", 2023-07-02T14:01:58.316 INFO:teuthology.orchestra.run.smithi016.stdout: "rados_osd_op_timeout": "0", 2023-07-02T14:01:58.316 INFO:teuthology.orchestra.run.smithi016.stdout: "rados_tracing": "false", 2023-07-02T14:01:58.316 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_atime_update_interval": "60", 2023-07-02T14:01:58.316 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2023-07-02T14:01:58.316 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_balance_parent_reads": "false", 2023-07-02T14:01:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_balance_snap_reads": "false", 2023-07-02T14:01:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_blkin_trace_all": "false", 2023-07-02T14:01:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_blocklist_expire_seconds": "0", 2023-07-02T14:01:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_blocklist_on_break_lock": "true", 2023-07-02T14:01:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache": "true", 2023-07-02T14:01:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_block_writes_upfront": "false", 2023-07-02T14:01:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_max_dirty": "25165824", 2023-07-02T14:01:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_max_dirty_age": "1.000000", 2023-07-02T14:01:58.318 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_max_dirty_object": "0", 2023-07-02T14:01:58.318 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_policy": "writearound", 2023-07-02T14:01:58.318 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_size": "33554432", 2023-07-02T14:01:58.318 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_target_dirty": "16777216", 2023-07-02T14:01:58.318 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_cache_writethrough_until_flush": "true", 2023-07-02T14:01:58.318 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_clone_copy_on_read": "false", 2023-07-02T14:01:58.318 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_compression_hint": "none", 2023-07-02T14:01:58.319 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_concurrent_management_ops": "10", 2023-07-02T14:01:58.319 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_config_pool_override_update_timestamp": "0", 2023-07-02T14:01:58.319 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_clone_format": "auto", 2023-07-02T14:01:58.319 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_data_pool": "", 2023-07-02T14:01:58.319 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_features": "61", 2023-07-02T14:01:58.319 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_format": "2", 2023-07-02T14:01:58.319 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_map_options": "", 2023-07-02T14:01:58.319 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_order": "22", 2023-07-02T14:01:58.320 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_pool": "rbd", 2023-07-02T14:01:58.320 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2023-07-02T14:01:58.320 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_stripe_count": "0", 2023-07-02T14:01:58.320 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_default_stripe_unit": "0", 2023-07-02T14:01:58.320 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_disable_zero_copy_writes": "true", 2023-07-02T14:01:58.320 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_discard_granularity_bytes": "65536", 2023-07-02T14:01:58.320 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_discard_on_zeroed_write_same": "true", 2023-07-02T14:01:58.320 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_enable_alloc_hint": "true", 2023-07-02T14:01:58.321 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2023-07-02T14:01:58.321 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_io_scheduler": "simple", 2023-07-02T14:01:58.321 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2023-07-02T14:01:58.321 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_commit_age": "5.000000", 2023-07-02T14:01:58.321 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2023-07-02T14:01:58.321 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_max_payload_bytes": "16384", 2023-07-02T14:01:58.321 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_object_flush_age": "0.000000", 2023-07-02T14:01:58.321 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_object_flush_bytes": "1048576", 2023-07-02T14:01:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_object_flush_interval": "0", 2023-07-02T14:01:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2023-07-02T14:01:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2023-07-02T14:01:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_order": "24", 2023-07-02T14:01:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_pool": "", 2023-07-02T14:01:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_journal_splay_width": "4", 2023-07-02T14:01:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_localize_parent_reads": "false", 2023-07-02T14:01:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_localize_snap_reads": "false", 2023-07-02T14:01:58.323 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2023-07-02T14:01:58.323 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2023-07-02T14:01:58.323 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2023-07-02T14:01:58.323 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2023-07-02T14:01:58.323 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2023-07-02T14:01:58.323 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2023-07-02T14:01:58.323 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_image_policy_type": "simple", 2023-07-02T14:01:58.323 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2023-07-02T14:01:58.324 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_image_state_check_interval": "30", 2023-07-02T14:01:58.324 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2023-07-02T14:01:58.324 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2023-07-02T14:01:58.324 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2023-07-02T14:01:58.324 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2023-07-02T14:01:58.324 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2023-07-02T14:01:58.324 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_autotune": "true", 2023-07-02T14:01:58.324 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_base": "805306368", 2023-07-02T14:01:58.325 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2023-07-02T14:01:58.325 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_cache_min": "134217728", 2023-07-02T14:01:58.325 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2023-07-02T14:01:58.325 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2023-07-02T14:01:58.325 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_memory_target": "4294967296", 2023-07-02T14:01:58.325 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_perf_stats_prio": "5", 2023-07-02T14:01:58.325 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2023-07-02T14:01:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2023-07-02T14:01:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirroring_delete_delay": "0", 2023-07-02T14:01:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2023-07-02T14:01:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirroring_replay_delay": "0", 2023-07-02T14:01:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2023-07-02T14:01:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2023-07-02T14:01:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_move_to_trash_on_remove": "false", 2023-07-02T14:01:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2023-07-02T14:01:58.327 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_mtime_update_interval": "60", 2023-07-02T14:01:58.327 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_non_blocking_aio": "true", 2023-07-02T14:01:58.327 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_op_thread_timeout": "60", 2023-07-02T14:01:58.327 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_op_threads": "1", 2023-07-02T14:01:58.327 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_parent_cache_enabled": "false", 2023-07-02T14:01:58.327 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_persistent_cache_mode": "disabled", 2023-07-02T14:01:58.327 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_persistent_cache_path": "/tmp", 2023-07-02T14:01:58.327 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_persistent_cache_size": "1073741824", 2023-07-02T14:01:58.328 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_plugins": "", 2023-07-02T14:01:58.328 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_bps_burst": "0", 2023-07-02T14:01:58.328 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_bps_burst_seconds": "1", 2023-07-02T14:01:58.328 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_bps_limit": "0", 2023-07-02T14:01:58.328 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_iops_burst": "0", 2023-07-02T14:01:58.328 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_iops_burst_seconds": "1", 2023-07-02T14:01:58.328 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_iops_limit": "0", 2023-07-02T14:01:58.328 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_read_bps_burst": "0", 2023-07-02T14:01:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2023-07-02T14:01:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_read_bps_limit": "0", 2023-07-02T14:01:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_read_iops_burst": "0", 2023-07-02T14:01:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2023-07-02T14:01:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_read_iops_limit": "0", 2023-07-02T14:01:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_schedule_tick_min": "50", 2023-07-02T14:01:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_write_bps_burst": "0", 2023-07-02T14:01:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2023-07-02T14:01:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_write_bps_limit": "0", 2023-07-02T14:01:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_write_iops_burst": "0", 2023-07-02T14:01:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2023-07-02T14:01:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_qos_write_iops_limit": "0", 2023-07-02T14:01:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_quiesce_notification_attempts": "10", 2023-07-02T14:01:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_read_from_replica_policy": "default", 2023-07-02T14:01:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2023-07-02T14:01:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_readahead_max_bytes": "524288", 2023-07-02T14:01:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_readahead_trigger_requests": "10", 2023-07-02T14:01:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_request_timed_out_seconds": "30", 2023-07-02T14:01:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_skip_partial_discard": "true", 2023-07-02T14:01:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2023-07-02T14:01:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_tracing": "false", 2023-07-02T14:01:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_validate_names": "true", 2023-07-02T14:01:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd_validate_pool": "true", 2023-07-02T14:01:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: "restapi_base_url": "", 2023-07-02T14:01:58.332 INFO:teuthology.orchestra.run.smithi016.stdout: "restapi_log_level": "", 2023-07-02T14:01:58.332 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_acl_grants_max_num": "100", 2023-07-02T14:01:58.332 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_admin_entry": "admin", 2023-07-02T14:01:58.332 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_barbican_url": "", 2023-07-02T14:01:58.333 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2023-07-02T14:01:58.333 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_default_quota_max_size": "-1", 2023-07-02T14:01:58.333 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_index_max_aio": "128", 2023-07-02T14:01:58.334 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_quota_cache_size": "10000", 2023-07-02T14:01:58.334 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_bucket_quota_ttl": "600", 2023-07-02T14:01:58.334 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_cache_enabled": "true", 2023-07-02T14:01:58.335 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_cache_expiry_interval": "900", 2023-07-02T14:01:58.335 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_cache_lru_size": "10000", 2023-07-02T14:01:58.335 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_content_length_compat": "false", 2023-07-02T14:01:58.335 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_copy_obj_progress": "true", 2023-07-02T14:01:58.336 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2023-07-02T14:01:58.336 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_cors_rules_max_num": "100", 2023-07-02T14:01:58.336 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_cross_domain_policy": "", 2023-07-02T14:01:58.336 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_default_encryption_key": "", 2023-07-02T14:01:58.337 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_addr": "", 2023-07-02T14:01:58.337 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_ca_path": "", 2023-07-02T14:01:58.337 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_client_cert": "", 2023-07-02T14:01:58.337 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_client_key": "", 2023-07-02T14:01:58.337 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_kms_key_template": "", 2023-07-02T14:01:58.338 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_password": "", 2023-07-02T14:01:58.338 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2023-07-02T14:01:58.338 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_kmip_username": "", 2023-07-02T14:01:58.339 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_require_ssl": "true", 2023-07-02T14:01:58.339 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2023-07-02T14:01:58.339 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2023-07-02T14:01:58.339 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_suppress_logs": "true", 2023-07-02T14:01:58.339 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_addr": "", 2023-07-02T14:01:58.339 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_auth": "token", 2023-07-02T14:01:58.339 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_namespace": "", 2023-07-02T14:01:58.340 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_prefix": "", 2023-07-02T14:01:58.340 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_secret_engine": "transit", 2023-07-02T14:01:58.340 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_ssl_cacert": "", 2023-07-02T14:01:58.340 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2023-07-02T14:01:58.340 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2023-07-02T14:01:58.340 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_token_file": "", 2023-07-02T14:01:58.340 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_crypt_vault_verify_ssl": "true", 2023-07-02T14:01:58.341 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_curl_buffersize": "524288", 2023-07-02T14:01:58.341 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_curl_low_speed_limit": "1024", 2023-07-02T14:01:58.341 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_curl_low_speed_time": "300", 2023-07-02T14:01:58.341 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_curl_wait_timeout_ms": "1000", 2023-07-02T14:01:58.341 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2023-07-02T14:01:58.341 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data_log_changes_size": "1000", 2023-07-02T14:01:58.342 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data_log_num_shards": "128", 2023-07-02T14:01:58.342 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data_log_obj_prefix": "data_log", 2023-07-02T14:01:58.342 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data_log_window": "30", 2023-07-02T14:01:58.342 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_data_notify_interval_msec": "200", 2023-07-02T14:01:58.342 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_default_data_log_backing": "fifo", 2023-07-02T14:01:58.342 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_default_realm_info_oid": "default.realm", 2023-07-02T14:01:58.342 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_default_region_info_oid": "default.region", 2023-07-02T14:01:58.343 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_default_zone_info_oid": "default.zone", 2023-07-02T14:01:58.343 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2023-07-02T14:01:58.343 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_defer_to_bucket_acls": "", 2023-07-02T14:01:58.343 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_delete_multi_obj_max_num": "1000", 2023-07-02T14:01:58.343 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_admin_lim": "0.000000", 2023-07-02T14:01:58.343 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_admin_res": "100.000000", 2023-07-02T14:01:58.343 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2023-07-02T14:01:58.343 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_auth_lim": "0.000000", 2023-07-02T14:01:58.344 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_auth_res": "200.000000", 2023-07-02T14:01:58.344 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2023-07-02T14:01:58.344 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_data_lim": "0.000000", 2023-07-02T14:01:58.344 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_data_res": "500.000000", 2023-07-02T14:01:58.344 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_data_wgt": "500.000000", 2023-07-02T14:01:58.344 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2023-07-02T14:01:58.344 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_metadata_res": "500.000000", 2023-07-02T14:01:58.344 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2023-07-02T14:01:58.345 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dns_name": "", 2023-07-02T14:01:58.345 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dns_s3website_name": "", 2023-07-02T14:01:58.345 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_dynamic_resharding": "true", 2023-07-02T14:01:58.345 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2023-07-02T14:01:58.345 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_gc_threads": "true", 2023-07-02T14:01:58.345 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_lc_threads": "true", 2023-07-02T14:01:58.345 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_ops_log": "false", 2023-07-02T14:01:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_quota_threads": "true", 2023-07-02T14:01:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_static_website": "false", 2023-07-02T14:01:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enable_usage_log": "false", 2023-07-02T14:01:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_enforce_swift_acls": "true", 2023-07-02T14:01:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_exit_timeout_secs": "120", 2023-07-02T14:01:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_expose_bucket": "false", 2023-07-02T14:01:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_extended_http_attrs": "", 2023-07-02T14:01:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_fcgi_socket_backlog": "1024", 2023-07-02T14:01:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2023-07-02T14:01:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_frontends": "beast port=7480", 2023-07-02T14:01:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_max_concurrent_io": "10", 2023-07-02T14:01:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_max_deferred": "50", 2023-07-02T14:01:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2023-07-02T14:01:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_max_objs": "32", 2023-07-02T14:01:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_max_queue_size": "134213632", 2023-07-02T14:01:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_max_trim_chunk": "16", 2023-07-02T14:01:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_obj_min_wait": "7200", 2023-07-02T14:01:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_processor_max_time": "3600", 2023-07-02T14:01:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_gc_processor_period": "3600", 2023-07-02T14:01:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_get_obj_max_req_size": "4194304", 2023-07-02T14:01:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_get_obj_window_size": "16777216", 2023-07-02T14:01:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_healthcheck_disabling_path": "", 2023-07-02T14:01:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_host": "", 2023-07-02T14:01:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ignore_get_invalid_range": "false", 2023-07-02T14:01:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_init_timeout": "300", 2023-07-02T14:01:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2023-07-02T14:01:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_accepted_admin_roles": "", 2023-07-02T14:01:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2023-07-02T14:01:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_domain": "", 2023-07-02T14:01:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_password": "", 2023-07-02T14:01:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_password_path": "", 2023-07-02T14:01:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_project": "", 2023-07-02T14:01:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_tenant": "", 2023-07-02T14:01:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_token": "", 2023-07-02T14:01:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_token_path": "", 2023-07-02T14:01:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_admin_user": "", 2023-07-02T14:01:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_api_version": "2", 2023-07-02T14:01:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_barbican_domain": "", 2023-07-02T14:01:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_barbican_password": "", 2023-07-02T14:01:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_barbican_project": "", 2023-07-02T14:01:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_barbican_tenant": "", 2023-07-02T14:01:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_barbican_user": "", 2023-07-02T14:01:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_implicit_tenants": "false", 2023-07-02T14:01:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_token_cache_size": "10000", 2023-07-02T14:01:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_url": "", 2023-07-02T14:01:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_keystone_verify_ssl": "true", 2023-07-02T14:01:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_debug_interval": "-1", 2023-07-02T14:01:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_lock_max_time": "90", 2023-07-02T14:01:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_max_objs": "32", 2023-07-02T14:01:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_max_rules": "1000", 2023-07-02T14:01:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_max_worker": "3", 2023-07-02T14:01:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_max_wp_worker": "3", 2023-07-02T14:01:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lc_thread_delay": "0", 2023-07-02T14:01:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2023-07-02T14:01:58.353 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ldap_dnattr": "uid", 2023-07-02T14:01:58.353 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2023-07-02T14:01:58.353 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ldap_searchfilter": "", 2023-07-02T14:01:58.354 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2023-07-02T14:01:58.354 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ldap_uri": "ldaps://", 2023-07-02T14:01:58.354 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2023-07-02T14:01:58.354 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_list_bucket_min_readahead": "1000", 2023-07-02T14:01:58.355 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_list_buckets_max_chunk": "1000", 2023-07-02T14:01:58.355 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_log_http_headers": "", 2023-07-02T14:01:58.355 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_log_nonexistent_bucket": "false", 2023-07-02T14:01:58.356 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2023-07-02T14:01:58.356 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_log_object_name_utc": "false", 2023-07-02T14:01:58.356 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2023-07-02T14:01:58.356 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_attr_name_len": "0", 2023-07-02T14:01:58.357 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_attr_size": "0", 2023-07-02T14:01:58.357 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_attrs_num_in_req": "0", 2023-07-02T14:01:58.357 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_chunk_size": "4194304", 2023-07-02T14:01:58.357 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_concurrent_requests": "1024", 2023-07-02T14:01:58.358 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_dynamic_shards": "1999", 2023-07-02T14:01:58.358 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_listing_results": "1000", 2023-07-02T14:01:58.358 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_notify_retries": "3", 2023-07-02T14:01:58.358 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_objs_per_shard": "100000", 2023-07-02T14:01:58.358 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_put_param_size": "1048576", 2023-07-02T14:01:58.359 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_put_size": "5368709120", 2023-07-02T14:01:58.359 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_max_slo_entries": "1000", 2023-07-02T14:01:58.359 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_md_log_max_shards": "64", 2023-07-02T14:01:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_md_notify_interval_msec": "200", 2023-07-02T14:01:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_mime_types_file": "/etc/mime.types", 2023-07-02T14:01:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_mp_lock_max_time": "600", 2023-07-02T14:01:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_multi_obj_del_max_aio": "16", 2023-07-02T14:01:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_multipart_min_part_size": "5242880", 2023-07-02T14:01:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_multipart_part_upload_limit": "10000", 2023-07-02T14:01:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_fhcache_partitions": "3", 2023-07-02T14:01:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_fhcache_size": "2017", 2023-07-02T14:01:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2023-07-02T14:01:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_lru_lanes": "5", 2023-07-02T14:01:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_max_gc": "300", 2023-07-02T14:01:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_namespace_expire_secs": "300", 2023-07-02T14:01:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_run_gc_threads": "false", 2023-07-02T14:01:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_run_lc_threads": "false", 2023-07-02T14:01:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_run_quota_threads": "false", 2023-07-02T14:01:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_run_sync_thread": "false", 2023-07-02T14:01:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_s3_fast_attrs": "false", 2023-07-02T14:01:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_nfs_write_completion_interval_s": "10", 2023-07-02T14:01:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_num_async_rados_threads": "32", 2023-07-02T14:01:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_num_control_oids": "8", 2023-07-02T14:01:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_num_rados_handles": "1", 2023-07-02T14:01:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_numa_node": "-1", 2023-07-02T14:01:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_obj_stripe_size": "4194304", 2023-07-02T14:01:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_obj_tombstone_cache_size": "1000", 2023-07-02T14:01:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_objexp_chunk_size": "100", 2023-07-02T14:01:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_objexp_gc_interval": "600", 2023-07-02T14:01:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_objexp_hints_num_shards": "127", 2023-07-02T14:01:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_olh_pending_timeout_sec": "3600", 2023-07-02T14:01:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_op_thread_suicide_timeout": "0", 2023-07-02T14:01:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_op_thread_timeout": "600", 2023-07-02T14:01:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_op_tracing": "false", 2023-07-02T14:01:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_opa_token": "", 2023-07-02T14:01:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_opa_url": "", 2023-07-02T14:01:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_opa_verify_ssl": "true", 2023-07-02T14:01:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ops_log_data_backlog": "5242880", 2023-07-02T14:01:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ops_log_file_path": "", 2023-07-02T14:01:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ops_log_rados": "true", 2023-07-02T14:01:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_ops_log_socket_path": "", 2023-07-02T14:01:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_override_bucket_index_max_shards": "0", 2023-07-02T14:01:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2023-07-02T14:01:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_period_push_interval": "2.000000", 2023-07-02T14:01:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_period_push_interval_max": "30.000000", 2023-07-02T14:01:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_period_root_pool": ".rgw.root", 2023-07-02T14:01:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_port": "", 2023-07-02T14:01:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_print_continue": "true", 2023-07-02T14:01:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_print_prohibited_content_length": "false", 2023-07-02T14:01:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_put_obj_max_window_size": "67108864", 2023-07-02T14:01:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_put_obj_min_window_size": "16777216", 2023-07-02T14:01:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2023-07-02T14:01:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_rados_pool_recovery_priority": "5", 2023-07-02T14:01:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_rados_tracing": "false", 2023-07-02T14:01:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_realm": "", 2023-07-02T14:01:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_realm_root_pool": ".rgw.root", 2023-07-02T14:01:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_region": "", 2023-07-02T14:01:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_region_root_pool": ".rgw.root", 2023-07-02T14:01:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_relaxed_region_enforcement": "false", 2023-07-02T14:01:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_relaxed_s3_bucket_names": "false", 2023-07-02T14:01:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2023-07-02T14:01:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_request_uri": "", 2023-07-02T14:01:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_reshard_batch_size": "64", 2023-07-02T14:01:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_reshard_bucket_lock_duration": "360", 2023-07-02T14:01:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_reshard_max_aio": "128", 2023-07-02T14:01:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_reshard_num_logs": "16", 2023-07-02T14:01:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_reshard_thread_interval": "600", 2023-07-02T14:01:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_resolve_cname": "false", 2023-07-02T14:01:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_rest_getusage_op_compat": "false", 2023-07-02T14:01:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_run_sync_thread": "true", 2023-07-02T14:01:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_auth_order": "sts, external, local", 2023-07-02T14:01:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_auth_use_keystone": "false", 2023-07-02T14:01:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_auth_use_ldap": "false", 2023-07-02T14:01:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_auth_use_rados": "true", 2023-07-02T14:01:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_auth_use_sts": "false", 2023-07-02T14:01:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_s3_success_create_obj_status": "0", 2023-07-02T14:01:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_safe_max_objects_per_shard": "102400", 2023-07-02T14:01:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_scheduler_type": "throttler", 2023-07-02T14:01:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_script_uri": "", 2023-07-02T14:01:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_service_provider_name": "", 2023-07-02T14:01:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_shard_warning_threshold": "90.000000", 2023-07-02T14:01:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_socket_path": "", 2023-07-02T14:01:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_client_id": "", 2023-07-02T14:01:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_client_secret": "", 2023-07-02T14:01:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_entry": "sts", 2023-07-02T14:01:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_key": "sts", 2023-07-02T14:01:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_max_session_duration": "43200", 2023-07-02T14:01:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_min_session_duration": "900", 2023-07-02T14:01:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sts_token_introspection_url": "", 2023-07-02T14:01:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_account_in_url": "false", 2023-07-02T14:01:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_auth_entry": "auth", 2023-07-02T14:01:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_auth_url": "", 2023-07-02T14:01:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_custom_header": "", 2023-07-02T14:01:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_enforce_content_length": "false", 2023-07-02T14:01:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_need_stats": "true", 2023-07-02T14:01:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_tenant_name": "", 2023-07-02T14:01:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_token_expiration": "86400", 2023-07-02T14:01:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_url": "", 2023-07-02T14:01:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_url_prefix": "swift", 2023-07-02T14:01:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_swift_versioning_enabled": "false", 2023-07-02T14:01:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2023-07-02T14:01:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_lease_period": "120", 2023-07-02T14:01:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2023-07-02T14:01:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_log_trim_interval": "1200", 2023-07-02T14:01:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_log_trim_max_buckets": "16", 2023-07-02T14:01:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2023-07-02T14:01:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2023-07-02T14:01:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_obj_etag_verify": "false", 2023-07-02T14:01:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_trace_history_size": "4096", 2023-07-02T14:01:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_trace_per_node_log_size": "32", 2023-07-02T14:01:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2023-07-02T14:01:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_thread_pool_size": "512", 2023-07-02T14:01:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_comment": "", 2023-07-02T14:01:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_createby": "", 2023-07-02T14:01:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_encoding": "", 2023-07-02T14:01:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_flag": "false", 2023-07-02T14:01:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_origin": "", 2023-07-02T14:01:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_sha_unit": "524288", 2023-07-02T14:01:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_torrent_tracker": "", 2023-07-02T14:01:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_trust_forwarded_https": "false", 2023-07-02T14:01:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_usage_log_flush_threshold": "1024", 2023-07-02T14:01:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_usage_log_tick_interval": "30", 2023-07-02T14:01:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_usage_max_shards": "32", 2023-07-02T14:01:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_usage_max_user_shards": "1", 2023-07-02T14:01:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_use_opa_authz": "false", 2023-07-02T14:01:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_default_quota_max_objects": "-1", 2023-07-02T14:01:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_default_quota_max_size": "-1", 2023-07-02T14:01:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_max_buckets": "1000", 2023-07-02T14:01:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2023-07-02T14:01:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_quota_sync_idle_users": "false", 2023-07-02T14:01:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_quota_sync_interval": "86400", 2023-07-02T14:01:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_quota_sync_wait_time": "86400", 2023-07-02T14:01:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_user_unique_email": "true", 2023-07-02T14:01:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_verify_ssl": "true", 2023-07-02T14:01:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_website_routing_rules_max_num": "50", 2023-07-02T14:01:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_zone": "", 2023-07-02T14:01:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_zone_root_pool": ".rgw.root", 2023-07-02T14:01:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_zonegroup": "", 2023-07-02T14:01:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2023-07-02T14:01:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_block_size": "4096", 2023-07-02T14:01:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_bloom_bits_per_key": "20", 2023-07-02T14:01:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2023-07-02T14:01:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2023-07-02T14:01:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cache_row_ratio": "0.000000", 2023-07-02T14:01:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cache_shard_bits": "4", 2023-07-02T14:01:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cache_size": "536870912", 2023-07-02T14:01:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cache_type": "binned_lru", 2023-07-02T14:01:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cf_compact_on_deletion": "false", 2023-07-02T14:01:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2023-07-02T14:01:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2023-07-02T14:01:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_collect_compaction_stats": "false", 2023-07-02T14:01:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_collect_extended_stats": "false", 2023-07-02T14:01:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_collect_memory_stats": "false", 2023-07-02T14:01:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_delete_range_threshold": "1048576", 2023-07-02T14:01:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_index_type": "binary_search", 2023-07-02T14:01:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_log_to_ceph_log": "true", 2023-07-02T14:01:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_metadata_block_size": "4096", 2023-07-02T14:01:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_partition_filters": "false", 2023-07-02T14:01:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_perf": "false", 2023-07-02T14:01:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2023-07-02T14:01:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "rotating_keys_bootstrap_timeout": "30", 2023-07-02T14:01:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "rotating_keys_renewal_timeout": "10", 2023-07-02T14:01:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "run_dir": "/var/run/ceph", 2023-07-02T14:01:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "setgroup": "", 2023-07-02T14:01:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "setuser": "", 2023-07-02T14:01:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "setuser_match_path": "", 2023-07-02T14:01:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_misplaced_ratio": "0.050000", 2023-07-02T14:01:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "thp": "false", 2023-07-02T14:01:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "threadpool_default_timeout": "60", 2023-07-02T14:01:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "threadpool_empty_queue_max_wait": "2", 2023-07-02T14:01:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "throttler_perf_counter": "true" 2023-07-02T14:01:58.382 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:01:58.384 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2023-07-02T14:01:58.513 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a: {} 2023-07-02T14:01:58.524 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b: {} 2023-07-02T14:01:58.533 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c: {} 2023-07-02T14:01:58.534 INFO:teuthology.orchestra.run.smithi016.stderr:mon.a: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2023-07-02T14:01:58.534 INFO:teuthology.orchestra.run.smithi016.stderr:mon.b: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2023-07-02T14:01:58.534 INFO:teuthology.orchestra.run.smithi016.stderr:mon.c: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2023-07-02T14:01:58.545 DEBUG:teuthology.orchestra.run.smithi022:> type ceph-objectstore-tool 2023-07-02T14:01:58.562 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-07-02T14:01:58.562 DEBUG:teuthology.orchestra.run.smithi016:> type ceph-objectstore-tool 2023-07-02T14:01:58.576 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-07-02T14:01:58.577 INFO:teuthology.run_tasks:Running task workunit... 2023-07-02T14:01:58.590 INFO:tasks.workunit:Pulling workunits from ref cbd88633a9c19779ad4b2e367bdc17f83b3a065d 2023-07-02T14:01:58.590 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-07-02T14:01:58.591 DEBUG:teuthology.orchestra.run.smithi163:> stat -- /home/ubuntu/cephtest/mnt.0 2023-07-02T14:01:58.591 INFO:tasks.thrashosds.thrasher:starting do_thrash 2023-07-02T14:01:58.591 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2023-07-02T14:01:58.592 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:01:58.592 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2023-07-02T14:01:58.592 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-07-02T14:01:58.593 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2023-07-02T14:01:58.593 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2023-07-02T14:01:58.593 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:01:58.594 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2023-07-02T14:01:58.594 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:01:58.594 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2023-07-02T14:01:58.594 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:01:58.604 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:01:58.604+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:01:58.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T14:01:58.610 INFO:teuthology.orchestra.run.smithi163.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-07-02T14:01:58.610 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-07-02T14:01:58.664 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-07-02T14:01:58.665 DEBUG:teuthology.orchestra.run.smithi163:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-07-02T14:01:58.694 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:01:58.694+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:01:58.719 INFO:tasks.workunit:timeout=3h 2023-07-02T14:01:58.719 INFO:tasks.workunit:cleanup=True 2023-07-02T14:01:58.720 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/idryomov/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout cbd88633a9c19779ad4b2e367bdc17f83b3a065d 2023-07-02T14:01:58.759 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:01:58.759 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:01:58.759 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:01:58.759 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:01:58.771 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:01:58.774 INFO:tasks.workunit.client.0.smithi163.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-07-02T14:01:58.775 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:01:58.792 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:01:58.795 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:01:58.794+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:01:58.812 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:01:58.830 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:01:58.849 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:01:58.867 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:01:58.869 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:01:58.869 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:01:58.869 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:01:58.869 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:01:58.869 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:01:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:01:58.896 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:01:58.896+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:01:58.899 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:01:58.899 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:01:58.910 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:01:58.949 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:58.949 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:01:58.950 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 16, 2023-07-02T14:01:58.950 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2", 2023-07-02T14:01:58.950 INFO:teuthology.orchestra.run.smithi016.stdout: "created": "2023-07-02T14:01:37.298060+0000", 2023-07-02T14:01:58.950 INFO:teuthology.orchestra.run.smithi016.stdout: "modified": "2023-07-02T14:01:50.288411+0000", 2023-07-02T14:01:58.950 INFO:teuthology.orchestra.run.smithi016.stdout: "last_up_change": "2023-07-02T14:01:45.271760+0000", 2023-07-02T14:01:58.950 INFO:teuthology.orchestra.run.smithi016.stdout: "last_in_change": "2023-07-02T14:01:39.129121+0000", 2023-07-02T14:01:58.950 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-07-02T14:01:58.951 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_num": 5799936, 2023-07-02T14:01:58.951 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_set": [ 2023-07-02T14:01:58.951 INFO:teuthology.orchestra.run.smithi016.stdout: "pglog_hardlimit", 2023-07-02T14:01:58.951 INFO:teuthology.orchestra.run.smithi016.stdout: "purged_snapdirs", 2023-07-02T14:01:58.951 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_deletes", 2023-07-02T14:01:58.951 INFO:teuthology.orchestra.run.smithi016.stdout: "sortbitwise" 2023-07-02T14:01:58.951 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:01:58.951 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_version": 4, 2023-07-02T14:01:58.952 INFO:teuthology.orchestra.run.smithi016.stdout: "full_ratio": 0.94999998807907104, 2023-07-02T14:01:58.952 INFO:teuthology.orchestra.run.smithi016.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-07-02T14:01:58.952 INFO:teuthology.orchestra.run.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2023-07-02T14:01:58.952 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_snapshot": "", 2023-07-02T14:01:58.952 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_max": 2, 2023-07-02T14:01:58.952 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd": 6, 2023-07-02T14:01:58.952 INFO:teuthology.orchestra.run.smithi016.stdout: "require_min_compat_client": "luminous", 2023-07-02T14:01:58.953 INFO:teuthology.orchestra.run.smithi016.stdout: "min_compat_client": "jewel", 2023-07-02T14:01:58.953 INFO:teuthology.orchestra.run.smithi016.stdout: "require_osd_release": "pacific", 2023-07-02T14:01:58.953 INFO:teuthology.orchestra.run.smithi016.stdout: "pools": [ 2023-07-02T14:01:58.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.953 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 1, 2023-07-02T14:01:58.953 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "device_health_metrics", 2023-07-02T14:01:58.953 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:45.323455+0000", 2023-07-02T14:01:58.953 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2023-07-02T14:01:58.954 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2023-07-02T14:01:58.954 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:01:58.954 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:01:58.954 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:01:58.954 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:01:58.954 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:01:58.954 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:01:58.955 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:01:58.955 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:01:58.955 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:01:58.955 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:01:58.955 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 1, 2023-07-02T14:01:58.955 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 1, 2023-07-02T14:01:58.955 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 1, 2023-07-02T14:01:58.955 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 1, 2023-07-02T14:01:58.956 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 1, 2023-07-02T14:01:58.956 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:01:58.956 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:01:58.956 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:01:58.956 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:01:58.956 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:01:58.956 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:01:58.956 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:01:58.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.957 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "14", 2023-07-02T14:01:58.957 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:01:58.957 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:01:58.957 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:01:58.957 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:01:58.957 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:01:58.958 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2023-07-02T14:01:58.958 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2023-07-02T14:01:58.958 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:01:58.958 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:01:58.958 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:01:58.958 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:01:58.958 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:01:58.958 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:01:58.959 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:01:58.959 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:01:58.959 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:01:58.959 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:01:58.959 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:01:58.959 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:01:58.959 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:01:58.959 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:01:58.960 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:01:58.960 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:01:58.960 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:01:58.960 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:01:58.960 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:01:58.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.960 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:01:58.960 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:01:58.961 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:01:58.961 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:01:58.961 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:01:58.961 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:01:58.961 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:01:58.961 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:01:58.961 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:01:58.962 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:01:58.962 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:01:58.962 INFO:teuthology.orchestra.run.smithi016.stdout: "options": { 2023-07-02T14:01:58.962 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_max": 32, 2023-07-02T14:01:58.962 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_min": 1 2023-07-02T14:01:58.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.962 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:01:58.962 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_devicehealth": {} 2023-07-02T14:01:58.963 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:58.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.963 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 2, 2023-07-02T14:01:58.963 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "rbd", 2023-07-02T14:01:58.963 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:47.448897+0000", 2023-07-02T14:01:58.963 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2023-07-02T14:01:58.964 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2023-07-02T14:01:58.964 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:01:58.964 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:01:58.964 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:01:58.964 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:01:58.964 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:01:58.964 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:01:58.964 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:01:58.965 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:01:58.965 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:01:58.965 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:01:58.965 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2023-07-02T14:01:58.965 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2023-07-02T14:01:58.965 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2023-07-02T14:01:58.965 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2023-07-02T14:01:58.965 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2023-07-02T14:01:58.966 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:01:58.966 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:01:58.966 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:01:58.966 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:01:58.966 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:01:58.966 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:01:58.966 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:01:58.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.967 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "16", 2023-07-02T14:01:58.967 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:01:58.967 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:01:58.967 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:01:58.967 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:01:58.967 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:01:58.967 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2023-07-02T14:01:58.968 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2023-07-02T14:01:58.968 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:01:58.968 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:01:58.968 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:01:58.968 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:01:58.968 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:01:58.968 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:01:58.969 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:01:58.969 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:01:58.969 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:01:58.969 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:01:58.969 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:01:58.969 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:01:58.969 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:01:58.969 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:01:58.970 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:01:58.970 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:01:58.970 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:01:58.970 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:01:58.970 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:01:58.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.970 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:01:58.970 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:01:58.971 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:01:58.971 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:01:58.971 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:01:58.971 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:01:58.971 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:01:58.971 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:01:58.971 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:01:58.971 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:01:58.972 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:01:58.972 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2023-07-02T14:01:58.972 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:01:58.972 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2023-07-02T14:01:58.972 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:58.972 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:58.972 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:01:58.973 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:01:58.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.973 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:01:58.973 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "43c446fe-c811-4ee3-99e8-d7452edfc1c1", 2023-07-02T14:01:58.973 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:01:58.973 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:01:58.973 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:01:58.973 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:01:58.974 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:01:58.974 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:01:58.974 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:01:58.974 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 0, 2023-07-02T14:01:58.974 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:01:58.974 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:01:58.974 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:01:58.974 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:58.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:58.975 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2023-07-02T14:01:58.975 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:01:58.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:58.975 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2023-07-02T14:01:58.976 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:01:58.976 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:58.976 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:58.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.976 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:01:58.976 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:58.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.977 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:58.977 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6802", 2023-07-02T14:01:58.977 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:01:58.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.977 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:58.977 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6803", 2023-07-02T14:01:58.977 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:01:58.978 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:58.978 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:58.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.978 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:01:58.978 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:58.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.978 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:58.978 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6806", 2023-07-02T14:01:58.979 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:01:58.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.979 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:58.979 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6807", 2023-07-02T14:01:58.979 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:01:58.979 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:58.979 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:58.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.980 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:01:58.980 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:58.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.980 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:58.980 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6804", 2023-07-02T14:01:58.980 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:01:58.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.981 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:58.981 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6805", 2023-07-02T14:01:58.981 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:01:58.981 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:58.982 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:58.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.982 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6801/83359", 2023-07-02T14:01:58.982 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6803/83359", 2023-07-02T14:01:58.982 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6807/83359", 2023-07-02T14:01:58.982 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6805/83359", 2023-07-02T14:01:58.982 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:01:58.982 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:01:58.983 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:01:58.983 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:58.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.983 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:01:58.983 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b", 2023-07-02T14:01:58.983 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:01:58.984 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:01:58.984 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:01:58.984 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:01:58.984 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:01:58.984 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:01:58.984 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:01:58.984 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 14, 2023-07-02T14:01:58.984 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:01:58.985 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:01:58.985 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:01:58.985 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:58.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.985 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:58.985 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6808", 2023-07-02T14:01:58.985 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:01:58.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.986 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:58.986 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6809", 2023-07-02T14:01:58.986 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:01:58.986 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:58.986 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:58.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.987 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:01:58.987 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:58.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.987 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:58.987 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6810", 2023-07-02T14:01:58.987 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:01:58.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.988 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:58.988 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6811", 2023-07-02T14:01:58.988 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:01:58.988 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:58.988 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:58.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.988 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:01:58.988 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:58.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.989 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:58.989 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6814", 2023-07-02T14:01:58.989 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:01:58.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.989 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:58.989 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6815", 2023-07-02T14:01:58.990 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:01:58.990 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:58.990 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:58.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.990 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:01:58.990 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:58.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.990 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:58.991 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6812", 2023-07-02T14:01:58.991 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:01:58.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.991 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:58.991 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6813", 2023-07-02T14:01:58.991 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:01:58.992 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:58.992 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:58.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.992 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6809/83360", 2023-07-02T14:01:58.992 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6811/83360", 2023-07-02T14:01:58.992 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6815/83360", 2023-07-02T14:01:58.992 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6813/83360", 2023-07-02T14:01:58.992 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:01:58.993 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:01:58.993 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:01:58.993 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:58.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.993 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:01:58.993 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "99f8ea0a-5a71-4abe-b087-8e891fd11d1b", 2023-07-02T14:01:58.994 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:01:58.994 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:01:58.994 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:01:58.994 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:01:58.994 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:01:58.994 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:01:58.994 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:01:58.994 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 14, 2023-07-02T14:01:58.995 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:01:58.995 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:01:58.995 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:01:58.995 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:58.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:58.995 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6816", 2023-07-02T14:01:58.995 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:01:58.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.996 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:58.996 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6817", 2023-07-02T14:01:58.996 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:01:58.996 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:58.996 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:58.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.997 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:01:58.997 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:58.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.997 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:58.997 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6818", 2023-07-02T14:01:58.997 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:01:58.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.998 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:58.998 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6819", 2023-07-02T14:01:58.998 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:01:58.998 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:58.998 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:58.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.998 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:01:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6822", 2023-07-02T14:01:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:01:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:58.999 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:59.000 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6823", 2023-07-02T14:01:59.000 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:01:59.000 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:59.000 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:59.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.000 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:01:59.000 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:59.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6820", 2023-07-02T14:01:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:01:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6821", 2023-07-02T14:01:59.001 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:01:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6817/83361", 2023-07-02T14:01:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6819/83361", 2023-07-02T14:01:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6823/83361", 2023-07-02T14:01:59.002 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6821/83361", 2023-07-02T14:01:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:01:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:01:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:01:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:01:59.003 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "d285b4d5-0bef-421a-b84a-943e485b10c2", 2023-07-02T14:01:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:01:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:01:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:01:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:01:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:01:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:01:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:01:59.004 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 14, 2023-07-02T14:01:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:01:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:01:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:01:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6808", 2023-07-02T14:01:59.005 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:01:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6809", 2023-07-02T14:01:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:01:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:59.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:01:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6810", 2023-07-02T14:01:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:01:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6811", 2023-07-02T14:01:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:01:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.008 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:01:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6814", 2023-07-02T14:01:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:01:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6815", 2023-07-02T14:01:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:01:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:01:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:59.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6812", 2023-07-02T14:01:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:01:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.011 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6813", 2023-07-02T14:01:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:01:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6809/83350", 2023-07-02T14:01:59.012 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6811/83350", 2023-07-02T14:01:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6815/83350", 2023-07-02T14:01:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6813/83350", 2023-07-02T14:01:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:01:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:01:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:01:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:59.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:01:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "7ba0b0db-e407-4c63-8278-aab09f3082c4", 2023-07-02T14:01:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:01:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:01:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:01:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:01:59.014 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:01:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:01:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:01:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 0, 2023-07-02T14:01:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:01:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:01:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:01:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:59.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6800", 2023-07-02T14:01:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:01:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6801", 2023-07-02T14:01:59.016 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:01:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:01:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.017 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6802", 2023-07-02T14:01:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:01:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6803", 2023-07-02T14:01:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:01:59.018 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:01:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6806", 2023-07-02T14:01:59.019 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:01:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6807", 2023-07-02T14:01:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:01:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:59.020 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:01:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6804", 2023-07-02T14:01:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:01:59.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6805", 2023-07-02T14:01:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:01:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.022 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6801/83351", 2023-07-02T14:01:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6803/83351", 2023-07-02T14:01:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6807/83351", 2023-07-02T14:01:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6805/83351", 2023-07-02T14:01:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:01:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:01:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:01:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:59.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:01:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "29663bfa-f6fe-4ce7-b45a-a9b7b4424c42", 2023-07-02T14:01:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:01:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:01:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:01:59.024 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:01:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:01:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:01:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:01:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 14, 2023-07-02T14:01:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:01:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:01:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:01:59.025 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6816", 2023-07-02T14:01:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:01:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:59.026 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6817", 2023-07-02T14:01:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:01:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:01:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.027 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6818", 2023-07-02T14:01:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:01:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6819", 2023-07-02T14:01:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:01:59.028 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:01:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6822", 2023-07-02T14:01:59.029 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:01:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6823", 2023-07-02T14:01:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:01:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:59.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:01:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:01:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:01:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6820", 2023-07-02T14:01:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:01:59.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:01:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6821", 2023-07-02T14:01:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:01:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6817/83352", 2023-07-02T14:01:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6819/83352", 2023-07-02T14:01:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6823/83352", 2023-07-02T14:01:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6821/83352", 2023-07-02T14:01:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:01:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:01:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:01:59.033 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:01:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:01:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_xinfo": [ 2023-07-02T14:01:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:01:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:01:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:01:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:01:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:01:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:01:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.001178+0000", 2023-07-02T14:01:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:01:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.035 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:01:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:01:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:01:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:01:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:01:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:01:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.184181+0000", 2023-07-02T14:01:59.036 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:01:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:01:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:01:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:01:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:01:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:01:59.037 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:01:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.274854+0000", 2023-07-02T14:01:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:01:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:01:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:01:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:01:59.038 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:01:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:01:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:01:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.186217+0000", 2023-07-02T14:01:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:01:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:01:59.039 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:01:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:01:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:01:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:01:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:01:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.247442+0000", 2023-07-02T14:01:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:01:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:01:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:01:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:01:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:01:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:01:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:01:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:01:59.041 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.220075+0000", 2023-07-02T14:01:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:01:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:01:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap": [], 2023-07-02T14:01:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_items": [], 2023-07-02T14:01:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_temp": [], 2023-07-02T14:01:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_temp": [], 2023-07-02T14:01:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklist": {}, 2023-07-02T14:01:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "range_blocklist": {}, 2023-07-02T14:01:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profiles": { 2023-07-02T14:01:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "default": { 2023-07-02T14:01:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2023-07-02T14:01:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "2", 2023-07-02T14:01:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "1", 2023-07-02T14:01:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "jerasure", 2023-07-02T14:01:59.043 INFO:teuthology.orchestra.run.smithi016.stdout: "ruleset-failure-domain": "osd", 2023-07-02T14:01:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: "technique": "reed_sol_van" 2023-07-02T14:01:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:01:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps_queue": [], 2023-07-02T14:01:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: "new_removed_snaps": [], 2023-07-02T14:01:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: "new_purged_snaps": [], 2023-07-02T14:01:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_node_flags": {}, 2023-07-02T14:01:59.044 INFO:teuthology.orchestra.run.smithi016.stdout: "device_class_flags": {}, 2023-07-02T14:01:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode": { 2023-07-02T14:01:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_enabled": false, 2023-07-02T14:01:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_bucket_count": 0, 2023-07-02T14:01:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: "degraded_stretch_mode": 0, 2023-07-02T14:01:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: "recovering_stretch_mode": 0, 2023-07-02T14:01:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_bucket": 0 2023-07-02T14:01:59.045 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:01:59.045 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:01:59.047 INFO:tasks.thrashosds.thrasher:j is {'epoch': 16, 'fsid': '9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2', 'created': '2023-07-02T14:01:37.298060+0000', 'modified': '2023-07-02T14:01:50.288411+0000', 'last_up_change': '2023-07-02T14:01:45.271760+0000', 'last_in_change': '2023-07-02T14:01:39.129121+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-07-02T14:01:45.323455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-02T14:01:47.448897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '43c446fe-c811-4ee3-99e8-d7452edfc1c1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6800', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6801', 'nonce': 83359}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6802', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6803', 'nonce': 83359}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6806', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6807', 'nonce': 83359}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6804', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6805', 'nonce': 83359}]}, 'public_addr': '172.21.15.16:6801/83359', 'cluster_addr': '172.21.15.16:6803/83359', 'heartbeat_back_addr': '172.21.15.16:6807/83359', 'heartbeat_front_addr': '172.21.15.16:6805/83359', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 14, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6808', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6809', 'nonce': 83360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6810', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6811', 'nonce': 83360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6814', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6815', 'nonce': 83360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6812', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6813', 'nonce': 83360}]}, 'public_addr': '172.21.15.16:6809/83360', 'cluster_addr': '172.21.15.16:6811/83360', 'heartbeat_back_addr': '172.21.15.16:6815/83360', 'heartbeat_front_addr': '172.21.15.16:6813/83360', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '99f8ea0a-5a71-4abe-b087-8e891fd11d1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 14, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6816', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6817', 'nonce': 83361}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6818', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6819', 'nonce': 83361}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6822', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6823', 'nonce': 83361}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6820', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6821', 'nonce': 83361}]}, 'public_addr': '172.21.15.16:6817/83361', 'cluster_addr': '172.21.15.16:6819/83361', 'heartbeat_back_addr': '172.21.15.16:6823/83361', 'heartbeat_front_addr': '172.21.15.16:6821/83361', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd285b4d5-0bef-421a-b84a-943e485b10c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 14, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6808', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6809', 'nonce': 83350}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6810', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6811', 'nonce': 83350}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6814', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6815', 'nonce': 83350}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6812', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6813', 'nonce': 83350}]}, 'public_addr': '172.21.15.22:6809/83350', 'cluster_addr': '172.21.15.22:6811/83350', 'heartbeat_back_addr': '172.21.15.22:6815/83350', 'heartbeat_front_addr': '172.21.15.22:6813/83350', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7ba0b0db-e407-4c63-8278-aab09f3082c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6800', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6801', 'nonce': 83351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6802', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6803', 'nonce': 83351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6806', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6807', 'nonce': 83351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6804', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6805', 'nonce': 83351}]}, 'public_addr': '172.21.15.22:6801/83351', 'cluster_addr': '172.21.15.22:6803/83351', 'heartbeat_back_addr': '172.21.15.22:6807/83351', 'heartbeat_front_addr': '172.21.15.22:6805/83351', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '29663bfa-f6fe-4ce7-b45a-a9b7b4424c42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 14, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6816', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6817', 'nonce': 83352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6818', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6819', 'nonce': 83352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6822', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6823', 'nonce': 83352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6820', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6821', 'nonce': 83352}]}, 'public_addr': '172.21.15.22:6817/83352', 'cluster_addr': '172.21.15.22:6819/83352', 'heartbeat_back_addr': '172.21.15.22:6823/83352', 'heartbeat_front_addr': '172.21.15.22:6821/83352', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.001178+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.184181+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.274854+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.186217+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.247442+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.220075+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-07-02T14:01:59.047 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:01:59.049 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:01:59.046+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:01:59.148 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:01:59.148+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:01:59.171 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:01:59.172 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:01:59.182 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:01:59.248 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:01:59.248+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:01:59.316 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:01:59.316 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:01:59.323 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:01:59.329 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:01:59.349 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:01:59.348+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:01:59.401 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:01:59.403 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:01:59.414 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-07-02T14:01:59.318572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":29672,"kb_used_data":888,"kb_used_omap":0,"kb_used_meta":28736,"kb_avail":566201368,"statfs":{"total":579820584960,"available":579790200832,"internally_reserved":0,"allocated":909312,"data_stored":288798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29425664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.555585+0000","last_change":"2023-07-02T14:01:49.554924+0000","last_active":"2023-07-02T14:01:49.555585+0000","last_peered":"2023-07-02T14:01:49.555585+0000","last_clean":"2023-07-02T14:01:49.555585+0000","last_became_active":"2023-07-02T14:01:49.554578+0000","last_became_peered":"2023-07-02T14:01:49.554578+0000","last_unstale":"2023-07-02T14:01:49.555585+0000","last_undegraded":"2023-07-02T14:01:49.555585+0000","last_fullsized":"2023-07-02T14:01:49.555585+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.297777+0000","last_change":"2023-07-02T14:01:49.295075+0000","last_active":"2023-07-02T14:01:49.297777+0000","last_peered":"2023-07-02T14:01:49.297777+0000","last_clean":"2023-07-02T14:01:49.297777+0000","last_became_active":"2023-07-02T14:01:49.293782+0000","last_became_peered":"2023-07-02T14:01:49.293782+0000","last_unstale":"2023-07-02T14:01:49.297777+0000","last_undegraded":"2023-07-02T14:01:49.297777+0000","last_fullsized":"2023-07-02T14:01:49.297777+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.294236+0000","last_change":"2023-07-02T14:01:49.293139+0000","last_active":"2023-07-02T14:01:49.294236+0000","last_peered":"2023-07-02T14:01:49.294236+0000","last_clean":"2023-07-02T14:01:49.294236+0000","last_became_active":"2023-07-02T14:01:49.292790+0000","last_became_peered":"2023-07-02T14:01:49.292790+0000","last_unstale":"2023-07-02T14:01:49.294236+0000","last_undegraded":"2023-07-02T14:01:49.294236+0000","last_fullsized":"2023-07-02T14:01:49.294236+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.294648+0000","last_change":"2023-07-02T14:01:49.293994+0000","last_active":"2023-07-02T14:01:49.294648+0000","last_peered":"2023-07-02T14:01:49.294648+0000","last_clean":"2023-07-02T14:01:49.294648+0000","last_became_active":"2023-07-02T14:01:49.293608+0000","last_became_peered":"2023-07-02T14:01:49.293608+0000","last_unstale":"2023-07-02T14:01:49.294648+0000","last_undegraded":"2023-07-02T14:01:49.294648+0000","last_fullsized":"2023-07-02T14:01:49.294648+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.291083+0000","last_change":"2023-07-02T14:01:47.288970+0000","last_active":"2023-07-02T14:01:49.291083+0000","last_peered":"2023-07-02T14:01:49.291083+0000","last_clean":"2023-07-02T14:01:49.291083+0000","last_became_active":"2023-07-02T14:01:47.288610+0000","last_became_peered":"2023-07-02T14:01:47.288610+0000","last_unstale":"2023-07-02T14:01:49.291083+0000","last_undegraded":"2023-07-02T14:01:49.291083+0000","last_fullsized":"2023-07-02T14:01:49.291083+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.298964+0000","last_change":"2023-07-02T14:01:49.298280+0000","last_active":"2023-07-02T14:01:49.298964+0000","last_peered":"2023-07-02T14:01:49.298964+0000","last_clean":"2023-07-02T14:01:49.298964+0000","last_became_active":"2023-07-02T14:01:49.297946+0000","last_became_peered":"2023-07-02T14:01:49.297946+0000","last_unstale":"2023-07-02T14:01:49.298964+0000","last_undegraded":"2023-07-02T14:01:49.298964+0000","last_fullsized":"2023-07-02T14:01:49.298964+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.299410+0000","last_change":"2023-07-02T14:01:49.295243+0000","last_active":"2023-07-02T14:01:49.299410+0000","last_peered":"2023-07-02T14:01:49.299410+0000","last_clean":"2023-07-02T14:01:49.299410+0000","last_became_active":"2023-07-02T14:01:49.294964+0000","last_became_peered":"2023-07-02T14:01:49.294964+0000","last_unstale":"2023-07-02T14:01:49.299410+0000","last_undegraded":"2023-07-02T14:01:49.299410+0000","last_fullsized":"2023-07-02T14:01:49.299410+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.298982+0000","last_change":"2023-07-02T14:01:49.298253+0000","last_active":"2023-07-02T14:01:49.298982+0000","last_peered":"2023-07-02T14:01:49.298982+0000","last_clean":"2023-07-02T14:01:49.298982+0000","last_became_active":"2023-07-02T14:01:49.297965+0000","last_became_peered":"2023-07-02T14:01:49.297965+0000","last_unstale":"2023-07-02T14:01:49.298982+0000","last_undegraded":"2023-07-02T14:01:49.298982+0000","last_fullsized":"2023-07-02T14:01:49.298982+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-07-02T14:01:49.293782+0000","last_change":"2023-07-02T14:01:49.293092+0000","last_active":"2023-07-02T14:01:49.293782+0000","last_peered":"2023-07-02T14:01:49.293782+0000","last_clean":"2023-07-02T14:01:49.293782+0000","last_became_active":"2023-07-02T14:01:49.292773+0000","last_became_peered":"2023-07-02T14:01:49.292773+0000","last_unstale":"2023-07-02T14:01:49.293782+0000","last_undegraded":"2023-07-02T14:01:49.293782+0000","last_fullsized":"2023-07-02T14:01:49.293782+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4892,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":94366948,"statfs":{"total":96636764160,"available":96631754752,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4956,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366884,"statfs":{"total":96636764160,"available":96631689216,"internally_reserved":0,"allocated":151552,"data_stored":48133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:01:59.416 INFO:tasks.thrashosds.thrasher:Setting 2.7 to [4, 5, 0, 3] 2023-07-02T14:01:59.416 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.7', '4', '5', '0', '3'] 2023-07-02T14:01:59.416 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.7 4 5 0 3 2023-07-02T14:01:59.450 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:01:59.450+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:01:59.454 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:01:59.455 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:01:59.465 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:01:59.551 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:01:59.550+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:01:59.633 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:01:59.633 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:01:59.644 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:01:59.685 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:01:59.684+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:01:59.753 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:01:59.753+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:01:59.808 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:01:59.809 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:01:59.819 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:01:59.862 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:01:59.861+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:01:59.943 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:01:59.943 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:01:59.954 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:01:59.995 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:01:59.994+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:00.056 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:00.055+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:00.118 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:00.118 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:00.128 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:02:00.179 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:00.178+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:00.258 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:00.257+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:00.268 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:00.269 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:00.279 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:02:00.324 INFO:teuthology.orchestra.run.smithi016.stderr:set 2.7 pg_upmap_items mapping to [4->5,0->3] 2023-07-02T14:02:00.363 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:00.362+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:00.446 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:00.446 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:00.456 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:02:00.460 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:00.459+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:00.561 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:00.560+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:00.619 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:00.620 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:00.629 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:02:00.662 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:00.661+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:00.753 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:00.753 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:00.762 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:00.762+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:00.763 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:02:00.863 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:00.864+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:00.886 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:02:00.927 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:00.927 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:00.937 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:02:00.979 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:00.979+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:01.030 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:01.047 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:01.064 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:01.065 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:01.064+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:01.082 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:01.100 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:01.102 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:01.102 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:01.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:01.102 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [] 2023-07-02T14:02:01.103 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:01.112 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:02:01.118 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:01.119 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:02:01.119 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:02:01.119 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:02:01.119 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:02:01.119 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:02:01.120 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:02:01.166 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:01.165+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:01.268 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:01.268+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:01.277 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:01.277 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:01.277 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:01.278 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:01.288 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:02:01.336 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:02:01.379 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:01.378+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:01.452 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:01.452 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:01.452 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:01.452 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:01.452 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:01.462 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:02:01.489 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:01.488+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:01.590 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:01.589+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:01.612 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:01.612 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:01.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:01.612 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [] 2023-07-02T14:02:01.612 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:01.622 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:02:01.691 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:01.690+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:01.749 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:01.749 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:01.749 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:01.749 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:01.760 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:02:01.792 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:01.792+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:01.893 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:01.892+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:01.925 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:01.925 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:01.925 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:01.926 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:01.926 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:01.935 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:02:01.994 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:01.994+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:02.095 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:02.093+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:02.100 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:02.100 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:02.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:02.100 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:02.100 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:02.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:02:02.195 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:02.195+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:02.275 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:02.275 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:02.275 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:02.275 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:02.285 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:02:02.296 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:02.296+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:02.327 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:02:02.397 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:02.397+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:02.411 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:02.411 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:02.411 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:02.411 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:02.411 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:02.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:02:02.498 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:02.498+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:02.544 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:02.544 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:02.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:02.545 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:02.545 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:02.554 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:02:02.598 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:02.597+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:02.678 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:02.678 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:02.678 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:02.678 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:02.689 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:02:02.699 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:02.700+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:02.800 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:02.799+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:02.853 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:02.853 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:02.854 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:02.854 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:02.854 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:02.863 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:02:02.901 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:02.901+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:03.002 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:03.001+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:03.027 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:03.027 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:03.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:03.027 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:03.028 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:03.037 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:02:03.102 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:03.102+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:03.131 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:02:03.172 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:03.172 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:03.173 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:03.173 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:03.182 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:02:03.224 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:03.224+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:03.268 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:03.285 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:03.302 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:03.303 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:03.302+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:03.320 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:03.338 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:03.349 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:03.349 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:03.356 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:03.357 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:02:03.357 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:02:03.357 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:02:03.357 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:02:03.357 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:02:03.358 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:02:03.359 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:02:03.404 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:03.405+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:03.482 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:03.483 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:03.493 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:02:03.535 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:03.535+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:03.605 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:03.605+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:03.658 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:03.658 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:03.668 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:02:03.709 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:03.709+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:03.806 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:03.807+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:03.830 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:03.831 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:03.841 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:02:03.907 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:03.907+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:04.005 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:04.005 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:04.008 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:04.008+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:04.015 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:02:04.109 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:04.108+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:04.179 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:04.179 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:04.189 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:02:04.210 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:04.210+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:04.310 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:04.311+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:04.312 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:04.313 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:04.323 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:02:04.339 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:02:04.411 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:04.410+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:04.486 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:04.487 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:04.497 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:02:04.512 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:04.513+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:04.614 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:04.613+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:04.621 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:04.621 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:04.632 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:02:04.715 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:04.714+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:04.795 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:04.796 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:04.805 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:02:04.846 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:04.845+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:04.917 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:04.917+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:04.968 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:04.969 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:04.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:02:05.022 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:05.021+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:05.103 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:05.103 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:05.113 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:02:05.154 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:05.153+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:05.219 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:05.218+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:05.278 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:05.278 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:05.288 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:02:05.320 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:05.320+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:05.337 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2023-07-02T14:02:05.337 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:02:05.337 INFO:tasks.thrashosds.thrasher:Reweighting osd 3 to 0.30020895647876544 2023-07-02T14:02:05.337 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 0.30020895647876544 2023-07-02T14:02:05.339 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:02:05.369 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:02:05.436 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:05.435+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:05.457 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:05.458 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:05.470 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:02:05.536 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:05.537 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:05.535+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:05.556 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:05.575 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:05.593 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:05.612 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:05.622 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:05.622+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:05.646 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:05.647 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:02:05.647 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:02:05.648 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:02:05.648 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:02:05.648 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:02:05.648 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:02:05.659 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:05.659 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:05.659 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:05.659 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:05.670 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:02:05.753 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:05.752+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:05.824 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:05.824+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:05.834 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:05.834 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:05.834 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:05.834 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:05.834 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:05.844 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:02:05.925 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:05.925+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:06.008 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:06.008 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:06.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:06.009 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:06.009 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:06.018 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:02:06.062 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:06.062+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:06.126 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:06.126+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:06.156 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:06.156 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:06.156 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:06.156 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:06.166 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:02:06.249 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:06.250+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:06.328 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:06.328+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:06.332 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:06.332 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:06.332 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:06.332 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:06.332 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:06.342 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:02:06.348 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.3 to 0.300209 (4cda) 2023-07-02T14:02:06.429 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:06.428+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:06.508 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:06.508 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:06.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:06.508 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [] 2023-07-02T14:02:06.509 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:06.518 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:02:06.562 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:06.562+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:06.630 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:06.630+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:06.642 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:06.643 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:06.643 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:06.643 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:06.653 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:02:06.736 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:06.736+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:06.818 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:06.818 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:06.819 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:06.819 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:06.819 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:06.828 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:02:06.869 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:06.869+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:06.932 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:06.932+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:06.992 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:06.992 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:06.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:06.994 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [] 2023-07-02T14:02:06.994 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:07.002 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:02:07.033 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:07.033+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:07.127 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:07.127 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:07.127 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:07.127 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:07.133 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:07.134+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:07.137 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:02:07.234 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:07.234+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:07.301 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:07.301 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:07.302 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:07.302 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:07.302 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:07.311 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:02:07.352 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:02:07.353 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:07.353+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:07.436 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:07.435+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:07.475 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:07.476 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:07.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:07.476 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [] 2023-07-02T14:02:07.476 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:07.485 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:02:07.537 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:07.537+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:07.612 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:07.613 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:07.622 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:02:07.637 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:07.637+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:07.663 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:02:07.738 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:07.738+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:07.747 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:07.747 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:07.747 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:07.748 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:07.748 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:07.758 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:02:07.804 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:07.822 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:07.839 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:07.839+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:07.839 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:07.857 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:07.875 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:07.884 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:07.884 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:07.893 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:07.894 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:02:07.895 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:02:07.895 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:02:07.895 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:02:07.895 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:02:07.896 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:02:07.896 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:02:07.940 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:07.940+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:08.019 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:08.019 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:08.029 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:02:08.040 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:08.041+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:08.141 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:08.141+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:08.152 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:08.152 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:08.163 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:02:08.242 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:08.242+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:08.286 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:08.286 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:08.296 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:02:08.343 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:08.344+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:08.355 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:02:08.429 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:08.430 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:08.440 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:02:08.484 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:08.483+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:08.545 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:08.543+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:08.563 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:08.563 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:08.573 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:02:08.646 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:08.646+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:08.736 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:08.737 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:08.746 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:08.746+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:08.747 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:02:08.847 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:08.847+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:08.885 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:08.885 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:08.895 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:02:08.948 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:08.948+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:09.020 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:09.020 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:09.031 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:02:09.049 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:09.049+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:09.150 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:09.150+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:09.194 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:09.194 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:09.204 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:02:09.289 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:09.289+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:09.351 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:09.350+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:09.371 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:09.371 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:09.382 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:02:09.465 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:09.465+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:09.548 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:09.548 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:09.552 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:09.552+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:09.558 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:02:09.653 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:09.654+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:09.722 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:09.723 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:09.733 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:02:09.754 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:09.754+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:09.855 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:09.855+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:09.898 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:09.898 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:09.906 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:02:09.908 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:02:09.991 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:09.991+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:10.057 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:10.056+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:10.057 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:10.072 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:10.072 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:10.075 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:10.082 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:02:10.093 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:10.111 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:10.128 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:10.146 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:10.147 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:02:10.147 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:02:10.147 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:02:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:02:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:02:10.148 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:02:10.166 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:10.166+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:10.246 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:10.246 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:10.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:10.247 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [] 2023-07-02T14:02:10.247 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:10.256 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:02:10.297 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:10.297+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:10.360 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:10.360+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:10.371 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:02:10.383 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:10.383 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:10.383 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:10.383 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:10.394 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:02:10.477 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:10.476+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:10.562 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:10.561+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:10.574 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:10.574 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:10.574 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:10.575 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:10.575 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:10.585 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:02:10.663 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:10.662+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:10.748 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:10.749 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:10.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:10.749 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:10.749 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:10.759 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:02:10.764 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:10.763+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:10.864 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:10.863+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:10.882 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:10.883 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:10.883 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:10.883 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:10.893 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:02:10.965 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:10.964+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:11.056 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:11.057 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:11.057 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:11.057 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:11.057 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:11.065 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:11.064+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:11.066 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:02:11.167 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:11.166+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:11.190 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:11.190 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:11.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:11.190 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:11.190 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:11.200 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:02:11.267 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:11.266+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:11.359 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:02:11.361 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2023-07-02T14:02:11.361 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:02:11.361 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2023-07-02T14:02:11.361 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-07-02T14:02:11.368 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:11.368 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:11.368 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:11.368 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:11.369 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:11.367+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:11.380 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:02:11.468 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:11.467+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:11.547 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:11.548 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:11.548 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:11.548 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:11.548 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:11.558 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:02:11.599 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:11.598+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:11.668 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:02:11.669 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:11.669 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 24, 2023-07-02T14:02:11.669 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2", 2023-07-02T14:02:11.669 INFO:teuthology.orchestra.run.smithi016.stdout: "created": "2023-07-02T14:01:37.298060+0000", 2023-07-02T14:02:11.669 INFO:teuthology.orchestra.run.smithi016.stdout: "modified": "2023-07-02T14:02:11.351389+0000", 2023-07-02T14:02:11.670 INFO:teuthology.orchestra.run.smithi016.stdout: "last_up_change": "2023-07-02T14:01:45.271760+0000", 2023-07-02T14:02:11.670 INFO:teuthology.orchestra.run.smithi016.stdout: "last_in_change": "2023-07-02T14:01:39.129121+0000", 2023-07-02T14:02:11.670 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-07-02T14:02:11.670 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_num": 5801984, 2023-07-02T14:02:11.670 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_set": [ 2023-07-02T14:02:11.670 INFO:teuthology.orchestra.run.smithi016.stdout: "noscrub", 2023-07-02T14:02:11.671 INFO:teuthology.orchestra.run.smithi016.stdout: "pglog_hardlimit", 2023-07-02T14:02:11.671 INFO:teuthology.orchestra.run.smithi016.stdout: "purged_snapdirs", 2023-07-02T14:02:11.671 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_deletes", 2023-07-02T14:02:11.671 INFO:teuthology.orchestra.run.smithi016.stdout: "sortbitwise" 2023-07-02T14:02:11.671 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:11.671 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_version": 4, 2023-07-02T14:02:11.672 INFO:teuthology.orchestra.run.smithi016.stdout: "full_ratio": 0.94999998807907104, 2023-07-02T14:02:11.672 INFO:teuthology.orchestra.run.smithi016.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-07-02T14:02:11.672 INFO:teuthology.orchestra.run.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2023-07-02T14:02:11.672 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_snapshot": "", 2023-07-02T14:02:11.672 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_max": 2, 2023-07-02T14:02:11.672 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd": 6, 2023-07-02T14:02:11.673 INFO:teuthology.orchestra.run.smithi016.stdout: "require_min_compat_client": "luminous", 2023-07-02T14:02:11.673 INFO:teuthology.orchestra.run.smithi016.stdout: "min_compat_client": "luminous", 2023-07-02T14:02:11.673 INFO:teuthology.orchestra.run.smithi016.stdout: "require_osd_release": "pacific", 2023-07-02T14:02:11.673 INFO:teuthology.orchestra.run.smithi016.stdout: "pools": [ 2023-07-02T14:02:11.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.673 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 1, 2023-07-02T14:02:11.673 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "device_health_metrics", 2023-07-02T14:02:11.674 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:45.323455+0000", 2023-07-02T14:02:11.674 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2023-07-02T14:02:11.674 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2023-07-02T14:02:11.674 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:02:11.674 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:02:11.674 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:02:11.674 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:02:11.675 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:02:11.675 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:02:11.675 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:02:11.675 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:02:11.675 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:02:11.675 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:02:11.675 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 1, 2023-07-02T14:02:11.676 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 1, 2023-07-02T14:02:11.676 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 1, 2023-07-02T14:02:11.676 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 1, 2023-07-02T14:02:11.676 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 1, 2023-07-02T14:02:11.676 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:02:11.676 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:02:11.676 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:02:11.677 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:02:11.677 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:02:11.677 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:02:11.677 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:02:11.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.678 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "14", 2023-07-02T14:02:11.678 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:02:11.678 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:02:11.678 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:02:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:02:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:02:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2023-07-02T14:02:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2023-07-02T14:02:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:02:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:02:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:02:11.680 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:02:11.680 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:02:11.680 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:02:11.680 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:02:11.680 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:02:11.680 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:02:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:02:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:02:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:02:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:02:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:02:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:02:11.681 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:02:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:02:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:02:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:02:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:02:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:02:11.682 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:02:11.683 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:02:11.683 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:02:11.683 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:02:11.683 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:02:11.684 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:02:11.684 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:02:11.684 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:02:11.685 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:02:11.685 INFO:teuthology.orchestra.run.smithi016.stdout: "options": { 2023-07-02T14:02:11.685 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_max": 32, 2023-07-02T14:02:11.685 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_min": 1 2023-07-02T14:02:11.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.686 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:02:11.686 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_devicehealth": {} 2023-07-02T14:02:11.687 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.688 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 2, 2023-07-02T14:02:11.688 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "rbd", 2023-07-02T14:02:11.689 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:47.448897+0000", 2023-07-02T14:02:11.689 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2023-07-02T14:02:11.689 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2023-07-02T14:02:11.689 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:02:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:02:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:02:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:02:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:02:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:02:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:02:11.690 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:02:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:02:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:02:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2023-07-02T14:02:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2023-07-02T14:02:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2023-07-02T14:02:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2023-07-02T14:02:11.691 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2023-07-02T14:02:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:02:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:02:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:02:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:02:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:02:11.692 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:02:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:02:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "16", 2023-07-02T14:02:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:02:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:02:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:02:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:02:11.693 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:02:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2023-07-02T14:02:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2023-07-02T14:02:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:02:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:02:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:02:11.694 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:02:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:02:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:02:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:02:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:02:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:02:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:02:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:02:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:02:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:02:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:02:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:02:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:02:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:02:11.696 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:02:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:02:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:02:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:02:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:02:11.697 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:02:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:02:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:02:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:02:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:02:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:02:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:02:11.698 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:02:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2023-07-02T14:02:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:02:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2023-07-02T14:02:11.699 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:02:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:02:11.700 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "43c446fe-c811-4ee3-99e8-d7452edfc1c1", 2023-07-02T14:02:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 0, 2023-07-02T14:02:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2023-07-02T14:02:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:11.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2023-07-02T14:02:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:11.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6802", 2023-07-02T14:02:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.707 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.707 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6803", 2023-07-02T14:02:11.707 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:11.707 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.707 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.708 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:11.708 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.708 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.709 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6806", 2023-07-02T14:02:11.709 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:11.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.710 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.710 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6807", 2023-07-02T14:02:11.710 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:11.710 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.710 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6804", 2023-07-02T14:02:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:11.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6805", 2023-07-02T14:02:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6801/83359", 2023-07-02T14:02:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6803/83359", 2023-07-02T14:02:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6807/83359", 2023-07-02T14:02:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6805/83359", 2023-07-02T14:02:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:11.713 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:02:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b", 2023-07-02T14:02:11.714 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:11.715 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 14, 2023-07-02T14:02:11.716 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:11.716 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:11.716 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:11.716 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.716 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6808", 2023-07-02T14:02:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6809", 2023-07-02T14:02:11.717 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.718 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6810", 2023-07-02T14:02:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6811", 2023-07-02T14:02:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:11.720 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.720 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.720 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6814", 2023-07-02T14:02:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6815", 2023-07-02T14:02:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:11.722 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6812", 2023-07-02T14:02:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.723 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6813", 2023-07-02T14:02:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6809/83360", 2023-07-02T14:02:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6811/83360", 2023-07-02T14:02:11.724 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6815/83360", 2023-07-02T14:02:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6813/83360", 2023-07-02T14:02:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:02:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "99f8ea0a-5a71-4abe-b087-8e891fd11d1b", 2023-07-02T14:02:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:11.726 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 14, 2023-07-02T14:02:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.727 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6816", 2023-07-02T14:02:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6817", 2023-07-02T14:02:11.728 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.729 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6818", 2023-07-02T14:02:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6819", 2023-07-02T14:02:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.730 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.731 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:11.731 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.731 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.731 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6822", 2023-07-02T14:02:11.731 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:11.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6823", 2023-07-02T14:02:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6820", 2023-07-02T14:02:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6821", 2023-07-02T14:02:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6817/83361", 2023-07-02T14:02:11.734 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6819/83361", 2023-07-02T14:02:11.735 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6823/83361", 2023-07-02T14:02:11.735 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6821/83361", 2023-07-02T14:02:11.735 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:11.735 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:11.735 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:11.735 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:02:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "d285b4d5-0bef-421a-b84a-943e485b10c2", 2023-07-02T14:02:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 0.300201416015625, 2023-07-02T14:02:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:11.736 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 18, 2023-07-02T14:02:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6808", 2023-07-02T14:02:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6809", 2023-07-02T14:02:11.738 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.739 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6810", 2023-07-02T14:02:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6811", 2023-07-02T14:02:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.740 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6814", 2023-07-02T14:02:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:11.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6815", 2023-07-02T14:02:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.742 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6812", 2023-07-02T14:02:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6813", 2023-07-02T14:02:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6809/83350", 2023-07-02T14:02:11.744 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6811/83350", 2023-07-02T14:02:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6815/83350", 2023-07-02T14:02:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6813/83350", 2023-07-02T14:02:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:02:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "7ba0b0db-e407-4c63-8278-aab09f3082c4", 2023-07-02T14:02:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:11.746 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 0, 2023-07-02T14:02:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6800", 2023-07-02T14:02:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6801", 2023-07-02T14:02:11.748 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.749 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6802", 2023-07-02T14:02:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6803", 2023-07-02T14:02:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.750 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6806", 2023-07-02T14:02:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:11.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6807", 2023-07-02T14:02:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6804", 2023-07-02T14:02:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6805", 2023-07-02T14:02:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6801/83351", 2023-07-02T14:02:11.754 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6803/83351", 2023-07-02T14:02:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6807/83351", 2023-07-02T14:02:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6805/83351", 2023-07-02T14:02:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:02:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "29663bfa-f6fe-4ce7-b45a-a9b7b4424c42", 2023-07-02T14:02:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:11.756 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 14, 2023-07-02T14:02:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6816", 2023-07-02T14:02:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6817", 2023-07-02T14:02:11.758 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.759 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6818", 2023-07-02T14:02:11.760 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:11.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.760 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.760 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6819", 2023-07-02T14:02:11.760 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:11.760 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6822", 2023-07-02T14:02:11.761 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:11.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.762 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.762 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6823", 2023-07-02T14:02:11.762 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:11.762 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.762 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.763 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:11.763 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:11.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.763 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:11.763 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6820", 2023-07-02T14:02:11.763 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:11.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.764 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:11.764 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6821", 2023-07-02T14:02:11.764 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:11.764 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.764 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.764 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6817/83352", 2023-07-02T14:02:11.764 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6819/83352", 2023-07-02T14:02:11.765 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6823/83352", 2023-07-02T14:02:11.765 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6821/83352", 2023-07-02T14:02:11.765 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:11.765 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:11.765 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:11.765 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.765 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.765 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:11.766 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_xinfo": [ 2023-07-02T14:02:11.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.766 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:02:11.766 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:11.766 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:11.766 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:11.766 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:11.766 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:11.767 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.001178+0000", 2023-07-02T14:02:11.767 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:11.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.767 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:02:11.767 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:11.767 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:11.767 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:11.768 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:11.768 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:11.768 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.184181+0000", 2023-07-02T14:02:11.768 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:11.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.768 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:02:11.769 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:11.769 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:11.769 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:11.769 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:11.769 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:11.769 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.274854+0000", 2023-07-02T14:02:11.769 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:11.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.770 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:02:11.770 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:11.770 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:11.770 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:11.770 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:11.770 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:11.770 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.186217+0000", 2023-07-02T14:02:11.771 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:11.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.771 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:02:11.771 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:11.771 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:11.771 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:11.771 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:11.772 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:11.772 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.247442+0000", 2023-07-02T14:02:11.772 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:11.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.772 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:02:11.772 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:11.772 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:11.773 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:11.773 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:11.773 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:11.773 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.220075+0000", 2023-07-02T14:02:11.773 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:11.773 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.773 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:11.773 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap": [], 2023-07-02T14:02:11.774 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_items": [ 2023-07-02T14:02:11.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.774 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.7", 2023-07-02T14:02:11.774 INFO:teuthology.orchestra.run.smithi016.stdout: "mappings": [ 2023-07-02T14:02:11.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:11.774 INFO:teuthology.orchestra.run.smithi016.stdout: "from": 4, 2023-07-02T14:02:11.774 INFO:teuthology.orchestra.run.smithi016.stdout: "to": 5 2023-07-02T14:02:11.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.775 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:11.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.775 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:11.775 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_temp": [], 2023-07-02T14:02:11.775 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_temp": [], 2023-07-02T14:02:11.775 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklist": {}, 2023-07-02T14:02:11.775 INFO:teuthology.orchestra.run.smithi016.stdout: "range_blocklist": {}, 2023-07-02T14:02:11.776 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profiles": { 2023-07-02T14:02:11.776 INFO:teuthology.orchestra.run.smithi016.stdout: "default": { 2023-07-02T14:02:11.776 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2023-07-02T14:02:11.776 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "2", 2023-07-02T14:02:11.776 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "1", 2023-07-02T14:02:11.776 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "jerasure", 2023-07-02T14:02:11.776 INFO:teuthology.orchestra.run.smithi016.stdout: "ruleset-failure-domain": "osd", 2023-07-02T14:02:11.777 INFO:teuthology.orchestra.run.smithi016.stdout: "technique": "reed_sol_van" 2023-07-02T14:02:11.777 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:11.777 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps_queue": [], 2023-07-02T14:02:11.777 INFO:teuthology.orchestra.run.smithi016.stdout: "new_removed_snaps": [], 2023-07-02T14:02:11.777 INFO:teuthology.orchestra.run.smithi016.stdout: "new_purged_snaps": [], 2023-07-02T14:02:11.777 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_node_flags": {}, 2023-07-02T14:02:11.778 INFO:teuthology.orchestra.run.smithi016.stdout: "device_class_flags": {}, 2023-07-02T14:02:11.778 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode": { 2023-07-02T14:02:11.778 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_enabled": false, 2023-07-02T14:02:11.778 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_bucket_count": 0, 2023-07-02T14:02:11.778 INFO:teuthology.orchestra.run.smithi016.stdout: "degraded_stretch_mode": 0, 2023-07-02T14:02:11.778 INFO:teuthology.orchestra.run.smithi016.stdout: "recovering_stretch_mode": 0, 2023-07-02T14:02:11.778 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_bucket": 0 2023-07-02T14:02:11.779 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:11.779 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:11.780 INFO:tasks.thrashosds.thrasher:j is {'epoch': 24, 'fsid': '9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2', 'created': '2023-07-02T14:01:37.298060+0000', 'modified': '2023-07-02T14:02:11.351389+0000', 'last_up_change': '2023-07-02T14:01:45.271760+0000', 'last_in_change': '2023-07-02T14:01:39.129121+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-07-02T14:01:45.323455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-02T14:01:47.448897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '43c446fe-c811-4ee3-99e8-d7452edfc1c1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6800', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6801', 'nonce': 83359}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6802', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6803', 'nonce': 83359}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6806', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6807', 'nonce': 83359}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6804', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6805', 'nonce': 83359}]}, 'public_addr': '172.21.15.16:6801/83359', 'cluster_addr': '172.21.15.16:6803/83359', 'heartbeat_back_addr': '172.21.15.16:6807/83359', 'heartbeat_front_addr': '172.21.15.16:6805/83359', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 14, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6808', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6809', 'nonce': 83360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6810', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6811', 'nonce': 83360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6814', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6815', 'nonce': 83360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6812', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6813', 'nonce': 83360}]}, 'public_addr': '172.21.15.16:6809/83360', 'cluster_addr': '172.21.15.16:6811/83360', 'heartbeat_back_addr': '172.21.15.16:6815/83360', 'heartbeat_front_addr': '172.21.15.16:6813/83360', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '99f8ea0a-5a71-4abe-b087-8e891fd11d1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 14, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6816', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6817', 'nonce': 83361}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6818', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6819', 'nonce': 83361}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6822', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6823', 'nonce': 83361}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6820', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6821', 'nonce': 83361}]}, 'public_addr': '172.21.15.16:6817/83361', 'cluster_addr': '172.21.15.16:6819/83361', 'heartbeat_back_addr': '172.21.15.16:6823/83361', 'heartbeat_front_addr': '172.21.15.16:6821/83361', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd285b4d5-0bef-421a-b84a-943e485b10c2', 'up': 1, 'in': 1, 'weight': 0.300201416015625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6808', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6809', 'nonce': 83350}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6810', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6811', 'nonce': 83350}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6814', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6815', 'nonce': 83350}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6812', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6813', 'nonce': 83350}]}, 'public_addr': '172.21.15.22:6809/83350', 'cluster_addr': '172.21.15.22:6811/83350', 'heartbeat_back_addr': '172.21.15.22:6815/83350', 'heartbeat_front_addr': '172.21.15.22:6813/83350', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7ba0b0db-e407-4c63-8278-aab09f3082c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6800', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6801', 'nonce': 83351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6802', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6803', 'nonce': 83351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6806', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6807', 'nonce': 83351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6804', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6805', 'nonce': 83351}]}, 'public_addr': '172.21.15.22:6801/83351', 'cluster_addr': '172.21.15.22:6803/83351', 'heartbeat_back_addr': '172.21.15.22:6807/83351', 'heartbeat_front_addr': '172.21.15.22:6805/83351', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '29663bfa-f6fe-4ce7-b45a-a9b7b4424c42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 14, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6816', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6817', 'nonce': 83352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6818', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6819', 'nonce': 83352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6822', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6823', 'nonce': 83352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6820', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6821', 'nonce': 83352}]}, 'public_addr': '172.21.15.22:6817/83352', 'cluster_addr': '172.21.15.22:6819/83352', 'heartbeat_back_addr': '172.21.15.22:6823/83352', 'heartbeat_front_addr': '172.21.15.22:6821/83352', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.001178+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.184181+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.274854+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.186217+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.247442+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.220075+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.7', 'mappings': [{'from': 4, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-07-02T14:02:11.781 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:02:11.782 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:11.782 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:11.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:11.782 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:11.782 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:11.783 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:02:11.783 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:11.779+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:11.881 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:11.880+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:11.921 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:11.921 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:11.921 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:11.921 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:11.931 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:02:11.981 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:11.980+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:12.055 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:12.056 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:12.056 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:12.056 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:12.056 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:12.066 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:02:12.081 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:12.080+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:12.119 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:02:12.121 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:02:12.132 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2023-07-02T14:02:11.355683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":30496,"kb_used_data":1392,"kb_used_omap":0,"kb_used_meta":29056,"kb_avail":566200544,"statfs":{"total":579820584960,"available":579789357056,"internally_reserved":0,"allocated":1425408,"data_stored":483186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29753344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-02T14:02:06.394837+0000","last_change":"2023-07-02T14:02:01.726288+0000","last_active":"2023-07-02T14:02:06.394837+0000","last_peered":"2023-07-02T14:02:06.394837+0000","last_clean":"2023-07-02T14:02:06.394837+0000","last_became_active":"2023-07-02T14:02:01.725984+0000","last_became_peered":"2023-07-02T14:02:01.725984+0000","last_unstale":"2023-07-02T14:02:06.394837+0000","last_undegraded":"2023-07-02T14:02:06.394837+0000","last_fullsized":"2023-07-02T14:02:06.394837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-02T14:02:06.345979+0000","last_change":"2023-07-02T14:01:49.295075+0000","last_active":"2023-07-02T14:02:06.345979+0000","last_peered":"2023-07-02T14:02:06.345979+0000","last_clean":"2023-07-02T14:02:06.345979+0000","last_became_active":"2023-07-02T14:01:49.293782+0000","last_became_peered":"2023-07-02T14:01:49.293782+0000","last_unstale":"2023-07-02T14:02:06.345979+0000","last_undegraded":"2023-07-02T14:02:06.345979+0000","last_fullsized":"2023-07-02T14:02:06.345979+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-02T14:02:06.394821+0000","last_change":"2023-07-02T14:01:49.293139+0000","last_active":"2023-07-02T14:02:06.394821+0000","last_peered":"2023-07-02T14:02:06.394821+0000","last_clean":"2023-07-02T14:02:06.394821+0000","last_became_active":"2023-07-02T14:01:49.292790+0000","last_became_peered":"2023-07-02T14:01:49.292790+0000","last_unstale":"2023-07-02T14:02:06.394821+0000","last_undegraded":"2023-07-02T14:02:06.394821+0000","last_fullsized":"2023-07-02T14:02:06.394821+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-02T14:02:06.394778+0000","last_change":"2023-07-02T14:01:49.293994+0000","last_active":"2023-07-02T14:02:06.394778+0000","last_peered":"2023-07-02T14:02:06.394778+0000","last_clean":"2023-07-02T14:02:06.394778+0000","last_became_active":"2023-07-02T14:01:49.293608+0000","last_became_peered":"2023-07-02T14:01:49.293608+0000","last_unstale":"2023-07-02T14:02:06.394778+0000","last_undegraded":"2023-07-02T14:02:06.394778+0000","last_fullsized":"2023-07-02T14:02:06.394778+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-02T14:02:06.394793+0000","last_change":"2023-07-02T14:01:47.288970+0000","last_active":"2023-07-02T14:02:06.394793+0000","last_peered":"2023-07-02T14:02:06.394793+0000","last_clean":"2023-07-02T14:02:06.394793+0000","last_became_active":"2023-07-02T14:01:47.288610+0000","last_became_peered":"2023-07-02T14:01:47.288610+0000","last_unstale":"2023-07-02T14:02:06.394793+0000","last_undegraded":"2023-07-02T14:02:06.394793+0000","last_fullsized":"2023-07-02T14:02:06.394793+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-02T14:02:06.394627+0000","last_change":"2023-07-02T14:01:49.298280+0000","last_active":"2023-07-02T14:02:06.394627+0000","last_peered":"2023-07-02T14:02:06.394627+0000","last_clean":"2023-07-02T14:02:06.394627+0000","last_became_active":"2023-07-02T14:01:49.297946+0000","last_became_peered":"2023-07-02T14:01:49.297946+0000","last_unstale":"2023-07-02T14:02:06.394627+0000","last_undegraded":"2023-07-02T14:02:06.394627+0000","last_fullsized":"2023-07-02T14:02:06.394627+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-02T14:02:06.394559+0000","last_change":"2023-07-02T14:01:49.295243+0000","last_active":"2023-07-02T14:02:06.394559+0000","last_peered":"2023-07-02T14:02:06.394559+0000","last_clean":"2023-07-02T14:02:06.394559+0000","last_became_active":"2023-07-02T14:01:49.294964+0000","last_became_peered":"2023-07-02T14:01:49.294964+0000","last_unstale":"2023-07-02T14:02:06.394559+0000","last_undegraded":"2023-07-02T14:02:06.394559+0000","last_fullsized":"2023-07-02T14:02:06.394559+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-02T14:02:06.394626+0000","last_change":"2023-07-02T14:01:49.298253+0000","last_active":"2023-07-02T14:02:06.394626+0000","last_peered":"2023-07-02T14:02:06.394626+0000","last_clean":"2023-07-02T14:02:06.394626+0000","last_became_active":"2023-07-02T14:01:49.297965+0000","last_became_peered":"2023-07-02T14:01:49.297965+0000","last_unstale":"2023-07-02T14:02:06.394626+0000","last_undegraded":"2023-07-02T14:02:06.394626+0000","last_fullsized":"2023-07-02T14:02:06.394626+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-07-02T14:02:06.346030+0000","last_change":"2023-07-02T14:01:49.293092+0000","last_active":"2023-07-02T14:02:06.346030+0000","last_peered":"2023-07-02T14:02:06.346030+0000","last_clean":"2023-07-02T14:02:06.346030+0000","last_became_active":"2023-07-02T14:01:49.292773+0000","last_became_peered":"2023-07-02T14:01:49.292773+0000","last_unstale":"2023-07-02T14:02:06.346030+0000","last_undegraded":"2023-07-02T14:02:06.346030+0000","last_fullsized":"2023-07-02T14:02:06.346030+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640263,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5040,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366800,"statfs":{"total":96636764160,"available":96631603200,"internally_reserved":0,"allocated":237568,"data_stored":80531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]}]},{"osd":4,"up_from":11,"seq":47244640263,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5104,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366736,"statfs":{"total":96636764160,"available":96631537664,"internally_reserved":0,"allocated":237568,"data_stored":80531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640263,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5104,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366736,"statfs":{"total":96636764160,"available":96631537664,"internally_reserved":0,"allocated":237568,"data_stored":80531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]}]},{"osd":2,"up_from":11,"seq":47244640263,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5040,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366800,"statfs":{"total":96636764160,"available":96631603200,"internally_reserved":0,"allocated":237568,"data_stored":80531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]}]},{"osd":0,"up_from":11,"seq":47244640263,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5104,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366736,"statfs":{"total":96636764160,"available":96631537664,"internally_reserved":0,"allocated":237568,"data_stored":80531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]}]},{"osd":1,"up_from":11,"seq":47244640263,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5104,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366736,"statfs":{"total":96636764160,"available":96631537664,"internally_reserved":0,"allocated":237568,"data_stored":80531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:02:12.133 INFO:tasks.thrashosds.thrasher:Setting 2.2 to [1, 5] 2023-07-02T14:02:12.133 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.2', '1', '5'] 2023-07-02T14:02:12.134 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.2 1 5 2023-07-02T14:02:12.158 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:02:12.182 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:12.182+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:12.189 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:12.189 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:12.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:12.189 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [] 2023-07-02T14:02:12.189 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:12.199 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:02:12.283 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:12.282+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:12.317 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:12.352 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:12.364 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:12.364 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:12.365 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:12.365 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:12.370 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:12.375 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:02:12.388 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:12.407 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:12.417 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:12.417+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:12.424 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:12.426 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:02:12.426 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:02:12.426 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:02:12.426 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:02:12.426 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:02:12.426 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:02:12.485 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:12.484+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:12.499 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:12.500 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:12.510 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:02:12.585 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:12.585+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:12.673 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:12.673 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:12.683 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:02:12.726 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:12.726+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:12.787 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:12.786+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:12.806 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:12.807 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:12.817 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:02:12.899 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:12.899+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:12.981 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:12.981 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:12.988 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:12.988+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:12.991 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:02:13.089 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:13.088+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:13.156 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:13.156 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:13.166 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:02:13.210 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:13.209+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:13.290 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:13.289+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:13.300 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:13.301 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:13.311 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:02:13.364 INFO:teuthology.orchestra.run.smithi016.stderr:set 2.2 pg_upmap mapping to [1,5] 2023-07-02T14:02:13.371 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:02:13.391 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:13.390+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:13.479 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:13.479 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:13.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:02:13.491 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:13.490+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:13.592 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:13.591+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:13.615 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:13.615 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:13.625 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:02:13.708 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:13.707+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:13.789 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:13.789 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:13.794 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:13.793+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:13.799 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:02:13.894 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:13.893+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:13.963 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:13.963 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:13.973 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:02:14.014 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:14.013+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:14.095 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:14.094+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:14.136 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:14.137 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:14.146 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:02:14.229 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:14.228+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:14.297 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:14.296+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:14.311 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:14.311 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:14.321 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:02:14.364 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:02:14.418 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:14.417+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:14.437 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:02:14.488 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:14.488 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:14.499 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:02:14.500 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:14.498+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:14.579 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:14.597 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:14.600 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:14.599+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:14.615 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:14.626 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:14.626 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:14.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:14.626 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:14.627 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:14.632 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:14.637 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:02:14.651 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:14.669 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:14.670 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:02:14.670 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:02:14.670 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:02:14.671 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:02:14.671 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:02:14.671 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:02:14.722 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:14.721+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:14.773 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:14.773 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:14.774 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:14.774 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:14.784 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:02:14.827 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:14.826+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:14.902 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:14.901+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:14.907 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:14.907 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:14.907 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:14.908 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:14.908 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:14.917 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:02:15.003 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:15.002+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:15.083 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:15.083 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:15.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:15.083 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [] 2023-07-02T14:02:15.084 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:15.093 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:02:15.136 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:15.135+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:15.205 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:15.204+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:15.217 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:15.218 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:15.218 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:15.218 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:15.228 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:02:15.312 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:15.311+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:15.393 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:15.393 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:15.393 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:15.393 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:15.393 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:15.403 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:02:15.406 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:15.406+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:15.507 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:15.506+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:15.567 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:15.567 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:15.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:15.567 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [] 2023-07-02T14:02:15.567 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:15.578 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:02:15.607 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:15.606+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:15.701 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:15.702 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:15.702 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:15.702 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:15.708 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:15.707+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:15.711 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:02:15.809 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:15.808+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:15.835 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:15.836 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:15.836 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:15.836 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:15.836 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:15.845 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:02:15.910 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:15.910+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:15.968 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:15.969 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:15.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:15.969 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [] 2023-07-02T14:02:15.969 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:15.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:02:16.022 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:16.021+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:16.111 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:16.111+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:16.118 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:16.118 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:16.118 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:16.118 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:16.128 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:02:16.212 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:16.212+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:16.293 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:16.294 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:16.294 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:16.294 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:16.294 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:16.303 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:02:16.313 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:16.313+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:16.376 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:02:16.414 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:16.413+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:16.429 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:16.430 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:16.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:16.430 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:16.430 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:16.440 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:02:16.515 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:16.514+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:16.570 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:16.570 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:16.571 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:16.571 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:16.581 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:02:16.622 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:16.621+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:16.681 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:02:16.726 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:16.726+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:16.747 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:16.748 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:16.759 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:02:16.817 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:16.817+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:16.823 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:16.841 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:16.859 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:16.877 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:16.896 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:16.914 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:16.915 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:02:16.916 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:02:16.916 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:02:16.916 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:02:16.916 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:02:16.916 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:02:16.918 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:16.917+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:16.925 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:16.926 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:16.936 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:02:17.019 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:17.018+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:17.101 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:17.101 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:17.111 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:02:17.152 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:17.152+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:17.221 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:17.220+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:17.275 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:17.275 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:17.285 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:02:17.322 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:17.322+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:17.373 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:02:17.412 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:17.412 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:17.422 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:02:17.423 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:17.423+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:17.524 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:17.523+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:17.561 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:17.561 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:17.571 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:02:17.624 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:17.624+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:17.695 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:17.695 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:17.705 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:02:17.750 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:17.749+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:17.826 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:17.825+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:17.828 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:17.829 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:17.839 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:02:17.927 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:17.926+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:18.003 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:18.004 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:18.013 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:02:18.028 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:18.028+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:18.129 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:18.128+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:18.177 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:18.178 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:18.188 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:02:18.230 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:18.229+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:18.331 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:18.330+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:18.354 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:18.354 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:18.364 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:02:18.375 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2023-07-02T14:02:18.375 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-07-02T14:02:18.432 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:18.432+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:18.528 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:18.529 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:18.533 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:18.533+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:18.539 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:02:18.634 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:18.633+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:18.662 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:18.663 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:18.673 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:02:18.757 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:18.756+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:18.836 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:18.835+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:18.837 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:18.837 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:18.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:02:18.928 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:02:18.937 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:18.936+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:18.973 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:18.973 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:18.984 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:02:19.038 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:19.037+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:19.072 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:19.090 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:19.108 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:19.112 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:19.113 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:19.113 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:19.113 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:19.113 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:19.123 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:02:19.126 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:19.139 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:19.138+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:19.144 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:19.163 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:02:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:02:19.164 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:02:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:02:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:02:19.165 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:02:19.240 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:19.240+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:19.248 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:19.249 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:19.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:19.249 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:19.249 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:19.259 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:02:19.341 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:19.340+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:19.381 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.0 to 1 (10000) 2023-07-02T14:02:19.385 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:02:19.397 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-07-02T14:02:19.432 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:19.432 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:19.432 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:19.432 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:19.443 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:02:19.444 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:19.443+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:19.543 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:19.543+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:19.612 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:19.613 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:19.613 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:19.613 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:19.613 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:19.623 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:02:19.644 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:19.643+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:19.744 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:19.744+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:19.789 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:19.789 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:19.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:19.790 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:19.790 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:19.800 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:02:19.883 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:19.883+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:19.946 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:19.946+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:19.964 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:19.964 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:19.965 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:19.965 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:19.974 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:02:20.048 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:20.047+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:20.138 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:20.138 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:20.138 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:20.138 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:20.139 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:20.148 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:02:20.149 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:20.149+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:20.250 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:20.250+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:20.272 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:20.272 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:20.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:20.273 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:20.273 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:20.282 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:02:20.352 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:20.351+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:20.385 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.1 to 1 (10000) 2023-07-02T14:02:20.385 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:02:20.398 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-07-02T14:02:20.422 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:20.422 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:20.422 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:20.422 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:20.433 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:02:20.482 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:20.482+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:20.553 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:20.553+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:20.598 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:20.598 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:20.598 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:20.598 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:20.599 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:20.608 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:02:20.654 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:20.654+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:20.732 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:20.732 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:20.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:20.732 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [] 2023-07-02T14:02:20.732 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:20.742 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:02:20.755 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:20.755+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:20.855 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:20.854+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:20.866 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:20.866 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:20.867 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:20.867 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:20.876 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:02:20.956 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:20.956+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:21.041 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:21.041 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:21.041 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:21.041 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:21.042 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:21.051 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:02:21.095 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:21.094+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:21.158 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:21.157+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:21.176 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:02:21.177 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:21.177 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:21.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:21.177 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [] 2023-07-02T14:02:21.177 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:21.187 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:02:21.259 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:21.259+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:21.320 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:21.337 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:21.352 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:21.352 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:21.352 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:21.353 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:21.355 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:21.360 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:21.360+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:21.362 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:02:21.372 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:21.386 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.2 to 1 (10000) 2023-07-02T14:02:21.393 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:21.397 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-07-02T14:02:21.412 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:21.413 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:02:21.413 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:02:21.414 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:02:21.414 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:02:21.414 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:02:21.414 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:02:21.466 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:21.466+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:21.488 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:21.488 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:21.498 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:02:21.561 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:21.560+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:21.622 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:21.622 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:21.632 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:02:21.673 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:21.672+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:21.756 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:21.756 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:21.762 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:21.762+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:21.767 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:02:21.864 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:21.863+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:21.930 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:21.930 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:21.940 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:02:21.964 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:21.963+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:22.064 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:22.065 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:22.065 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:22.064+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:22.074 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:02:22.166 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:22.165+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:22.238 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:22.238 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:22.248 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:02:22.291 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:22.291+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:22.367 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:22.367+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:22.371 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:22.371 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:22.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:02:22.390 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.3 to 1 (10000) 2023-07-02T14:02:22.400 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:02:22.401 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2023-07-02T14:02:22.484 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:22.484+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:22.517 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:22.517 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:22.529 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:02:22.568 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:22.567+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:22.669 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:22.668+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:22.696 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:22.697 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:22.707 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:02:22.769 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:22.769+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:22.832 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:22.832 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:22.842 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:02:22.870 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:22.869+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:22.971 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:22.971+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:23.005 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:23.006 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:23.015 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:02:23.072 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:23.072+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:23.153 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:23.154 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:23.164 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:02:23.208 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:23.207+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:23.274 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:23.274+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:23.288 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:23.288 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:23.298 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:02:23.381 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:23.380+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:23.397 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:02:23.397 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.4 to 1 (10000) 2023-07-02T14:02:23.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2023-07-02T14:02:23.425 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:02:23.463 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:23.463 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:23.473 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:02:23.515 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:23.514+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:23.577 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:23.577+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:23.595 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:23.612 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:23.629 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:23.639 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:23.639 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:23.647 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:23.650 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:02:23.665 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:23.682 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:23.683 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:02:23.684 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:02:23.684 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:02:23.684 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:02:23.684 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:02:23.684 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:02:23.693 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:23.692+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:23.774 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:23.774 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:23.775 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:23.775 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:23.775 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:23.778 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:23.778+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:23.784 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:02:23.879 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:23.879+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:23.948 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:23.949 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:23.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:23.949 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [] 2023-07-02T14:02:23.949 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:23.958 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:02:24.002 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:24.001+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:24.081 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:24.080+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:24.083 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:24.083 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:24.083 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:24.083 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:24.093 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:02:24.182 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:24.182+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:24.256 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:24.257 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:24.257 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:24.257 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:24.257 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:24.267 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:02:24.282 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:24.282+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:24.383 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:24.383+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:24.390 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:24.391 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:24.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:24.391 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [] 2023-07-02T14:02:24.391 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.5 to 1 (10000) 2023-07-02T14:02:24.402 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:02:24.415 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-07-02T14:02:24.415 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:02:24.484 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:24.483+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:24.536 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:24.536 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:24.536 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:24.537 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:24.548 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:02:24.591 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:24.591+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:24.674 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:24.674 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:24.675 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:24.675 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:24.675 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:24.685 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:02:24.687 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:24.687+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:24.763 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:02:24.765 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:02:24.776 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":48,"stamp":"2023-07-02T14:02:24.395932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":31000,"kb_used_data":1704,"kb_used_omap":0,"kb_used_meta":29248,"kb_avail":566200040,"statfs":{"total":579820584960,"available":579788840960,"internally_reserved":0,"allocated":1744896,"data_stored":604280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29949952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":28,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370920+0000","last_change":"2023-07-02T14:02:01.726288+0000","last_active":"2023-07-02T14:02:17.370920+0000","last_peered":"2023-07-02T14:02:17.370920+0000","last_clean":"2023-07-02T14:02:17.370920+0000","last_became_active":"2023-07-02T14:02:01.725984+0000","last_became_peered":"2023-07-02T14:02:01.725984+0000","last_unstale":"2023-07-02T14:02:17.370920+0000","last_undegraded":"2023-07-02T14:02:17.370920+0000","last_fullsized":"2023-07-02T14:02:17.370920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370429+0000","last_change":"2023-07-02T14:01:49.295075+0000","last_active":"2023-07-02T14:02:17.370429+0000","last_peered":"2023-07-02T14:02:17.370429+0000","last_clean":"2023-07-02T14:02:17.370429+0000","last_became_active":"2023-07-02T14:01:49.293782+0000","last_became_peered":"2023-07-02T14:01:49.293782+0000","last_unstale":"2023-07-02T14:02:17.370429+0000","last_undegraded":"2023-07-02T14:02:17.370429+0000","last_fullsized":"2023-07-02T14:02:17.370429+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370895+0000","last_change":"2023-07-02T14:01:49.293139+0000","last_active":"2023-07-02T14:02:17.370895+0000","last_peered":"2023-07-02T14:02:17.370895+0000","last_clean":"2023-07-02T14:02:17.370895+0000","last_became_active":"2023-07-02T14:01:49.292790+0000","last_became_peered":"2023-07-02T14:01:49.292790+0000","last_unstale":"2023-07-02T14:02:17.370895+0000","last_undegraded":"2023-07-02T14:02:17.370895+0000","last_fullsized":"2023-07-02T14:02:17.370895+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370871+0000","last_change":"2023-07-02T14:01:49.293994+0000","last_active":"2023-07-02T14:02:17.370871+0000","last_peered":"2023-07-02T14:02:17.370871+0000","last_clean":"2023-07-02T14:02:17.370871+0000","last_became_active":"2023-07-02T14:01:49.293608+0000","last_became_peered":"2023-07-02T14:01:49.293608+0000","last_unstale":"2023-07-02T14:02:17.370871+0000","last_undegraded":"2023-07-02T14:02:17.370871+0000","last_fullsized":"2023-07-02T14:02:17.370871+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370875+0000","last_change":"2023-07-02T14:01:47.288970+0000","last_active":"2023-07-02T14:02:17.370875+0000","last_peered":"2023-07-02T14:02:17.370875+0000","last_clean":"2023-07-02T14:02:17.370875+0000","last_became_active":"2023-07-02T14:01:47.288610+0000","last_became_peered":"2023-07-02T14:01:47.288610+0000","last_unstale":"2023-07-02T14:02:17.370875+0000","last_undegraded":"2023-07-02T14:02:17.370875+0000","last_fullsized":"2023-07-02T14:02:17.370875+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-07-02T14:02:15.024606+0000","last_change":"2023-07-02T14:01:49.298280+0000","last_active":"2023-07-02T14:02:15.024606+0000","last_peered":"2023-07-02T14:02:15.024606+0000","last_clean":"2023-07-02T14:02:15.024606+0000","last_became_active":"2023-07-02T14:01:49.297946+0000","last_became_peered":"2023-07-02T14:01:49.297946+0000","last_unstale":"2023-07-02T14:02:15.024606+0000","last_undegraded":"2023-07-02T14:02:15.024606+0000","last_fullsized":"2023-07-02T14:02:15.024606+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.395981+0000","last_change":"2023-07-02T14:01:49.295243+0000","last_active":"2023-07-02T14:02:17.395981+0000","last_peered":"2023-07-02T14:02:17.395981+0000","last_clean":"2023-07-02T14:02:17.395981+0000","last_became_active":"2023-07-02T14:01:49.294964+0000","last_became_peered":"2023-07-02T14:01:49.294964+0000","last_unstale":"2023-07-02T14:02:17.395981+0000","last_undegraded":"2023-07-02T14:02:17.395981+0000","last_fullsized":"2023-07-02T14:02:17.395981+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370424+0000","last_change":"2023-07-02T14:02:15.025638+0000","last_active":"2023-07-02T14:02:17.370424+0000","last_peered":"2023-07-02T14:02:17.370424+0000","last_clean":"2023-07-02T14:02:17.370424+0000","last_became_active":"2023-07-02T14:02:15.025376+0000","last_became_peered":"2023-07-02T14:02:15.025376+0000","last_unstale":"2023-07-02T14:02:17.370424+0000","last_undegraded":"2023-07-02T14:02:17.370424+0000","last_fullsized":"2023-07-02T14:02:17.370424+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370524+0000","last_change":"2023-07-02T14:01:49.293092+0000","last_active":"2023-07-02T14:02:17.370524+0000","last_peered":"2023-07-02T14:02:17.370524+0000","last_clean":"2023-07-02T14:02:17.370524+0000","last_became_active":"2023-07-02T14:01:49.292773+0000","last_became_peered":"2023-07-02T14:01:49.292773+0000","last_unstale":"2023-07-02T14:02:17.370524+0000","last_undegraded":"2023-07-02T14:02:17.370524+0000","last_fullsized":"2023-07-02T14:02:17.370524+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":286720,"data_stored":99159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640265,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5152,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":286720,"data_stored":99159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]}]},{"osd":3,"up_from":11,"seq":47244640265,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":286720,"data_stored":99159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640265,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5164,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366676,"statfs":{"total":96636764160,"available":96631476224,"internally_reserved":0,"allocated":299008,"data_stored":103822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":286720,"data_stored":99159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]}]},{"osd":1,"up_from":11,"seq":47244640265,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5228,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366612,"statfs":{"total":96636764160,"available":96631410688,"internally_reserved":0,"allocated":299008,"data_stored":103822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:02:24.778 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:02:24.788 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:24.787+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:24.852 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:24.852 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:24.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:24.853 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:24.853 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:24.863 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:02:24.890 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:24.889+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:24.990 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:24.989+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:25.029 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:25.029 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:25.029 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:25.029 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:25.039 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:02:25.090 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:25.090+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:25.118 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:02:25.121 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:02:25.132 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":48,"stamp":"2023-07-02T14:02:24.395932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":31000,"kb_used_data":1704,"kb_used_omap":0,"kb_used_meta":29248,"kb_avail":566200040,"statfs":{"total":579820584960,"available":579788840960,"internally_reserved":0,"allocated":1744896,"data_stored":604280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29949952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":28,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370920+0000","last_change":"2023-07-02T14:02:01.726288+0000","last_active":"2023-07-02T14:02:17.370920+0000","last_peered":"2023-07-02T14:02:17.370920+0000","last_clean":"2023-07-02T14:02:17.370920+0000","last_became_active":"2023-07-02T14:02:01.725984+0000","last_became_peered":"2023-07-02T14:02:01.725984+0000","last_unstale":"2023-07-02T14:02:17.370920+0000","last_undegraded":"2023-07-02T14:02:17.370920+0000","last_fullsized":"2023-07-02T14:02:17.370920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370429+0000","last_change":"2023-07-02T14:01:49.295075+0000","last_active":"2023-07-02T14:02:17.370429+0000","last_peered":"2023-07-02T14:02:17.370429+0000","last_clean":"2023-07-02T14:02:17.370429+0000","last_became_active":"2023-07-02T14:01:49.293782+0000","last_became_peered":"2023-07-02T14:01:49.293782+0000","last_unstale":"2023-07-02T14:02:17.370429+0000","last_undegraded":"2023-07-02T14:02:17.370429+0000","last_fullsized":"2023-07-02T14:02:17.370429+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370895+0000","last_change":"2023-07-02T14:01:49.293139+0000","last_active":"2023-07-02T14:02:17.370895+0000","last_peered":"2023-07-02T14:02:17.370895+0000","last_clean":"2023-07-02T14:02:17.370895+0000","last_became_active":"2023-07-02T14:01:49.292790+0000","last_became_peered":"2023-07-02T14:01:49.292790+0000","last_unstale":"2023-07-02T14:02:17.370895+0000","last_undegraded":"2023-07-02T14:02:17.370895+0000","last_fullsized":"2023-07-02T14:02:17.370895+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370871+0000","last_change":"2023-07-02T14:01:49.293994+0000","last_active":"2023-07-02T14:02:17.370871+0000","last_peered":"2023-07-02T14:02:17.370871+0000","last_clean":"2023-07-02T14:02:17.370871+0000","last_became_active":"2023-07-02T14:01:49.293608+0000","last_became_peered":"2023-07-02T14:01:49.293608+0000","last_unstale":"2023-07-02T14:02:17.370871+0000","last_undegraded":"2023-07-02T14:02:17.370871+0000","last_fullsized":"2023-07-02T14:02:17.370871+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370875+0000","last_change":"2023-07-02T14:01:47.288970+0000","last_active":"2023-07-02T14:02:17.370875+0000","last_peered":"2023-07-02T14:02:17.370875+0000","last_clean":"2023-07-02T14:02:17.370875+0000","last_became_active":"2023-07-02T14:01:47.288610+0000","last_became_peered":"2023-07-02T14:01:47.288610+0000","last_unstale":"2023-07-02T14:02:17.370875+0000","last_undegraded":"2023-07-02T14:02:17.370875+0000","last_fullsized":"2023-07-02T14:02:17.370875+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-07-02T14:02:15.024606+0000","last_change":"2023-07-02T14:01:49.298280+0000","last_active":"2023-07-02T14:02:15.024606+0000","last_peered":"2023-07-02T14:02:15.024606+0000","last_clean":"2023-07-02T14:02:15.024606+0000","last_became_active":"2023-07-02T14:01:49.297946+0000","last_became_peered":"2023-07-02T14:01:49.297946+0000","last_unstale":"2023-07-02T14:02:15.024606+0000","last_undegraded":"2023-07-02T14:02:15.024606+0000","last_fullsized":"2023-07-02T14:02:15.024606+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.395981+0000","last_change":"2023-07-02T14:01:49.295243+0000","last_active":"2023-07-02T14:02:17.395981+0000","last_peered":"2023-07-02T14:02:17.395981+0000","last_clean":"2023-07-02T14:02:17.395981+0000","last_became_active":"2023-07-02T14:01:49.294964+0000","last_became_peered":"2023-07-02T14:01:49.294964+0000","last_unstale":"2023-07-02T14:02:17.395981+0000","last_undegraded":"2023-07-02T14:02:17.395981+0000","last_fullsized":"2023-07-02T14:02:17.395981+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370424+0000","last_change":"2023-07-02T14:02:15.025638+0000","last_active":"2023-07-02T14:02:17.370424+0000","last_peered":"2023-07-02T14:02:17.370424+0000","last_clean":"2023-07-02T14:02:17.370424+0000","last_became_active":"2023-07-02T14:02:15.025376+0000","last_became_peered":"2023-07-02T14:02:15.025376+0000","last_unstale":"2023-07-02T14:02:17.370424+0000","last_undegraded":"2023-07-02T14:02:17.370424+0000","last_fullsized":"2023-07-02T14:02:17.370424+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370524+0000","last_change":"2023-07-02T14:01:49.293092+0000","last_active":"2023-07-02T14:02:17.370524+0000","last_peered":"2023-07-02T14:02:17.370524+0000","last_clean":"2023-07-02T14:02:17.370524+0000","last_became_active":"2023-07-02T14:01:49.292773+0000","last_became_peered":"2023-07-02T14:01:49.292773+0000","last_unstale":"2023-07-02T14:02:17.370524+0000","last_undegraded":"2023-07-02T14:02:17.370524+0000","last_fullsized":"2023-07-02T14:02:17.370524+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":286720,"data_stored":99159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640265,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5152,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":286720,"data_stored":99159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]}]},{"osd":3,"up_from":11,"seq":47244640265,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":286720,"data_stored":99159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640265,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5164,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366676,"statfs":{"total":96636764160,"available":96631476224,"internally_reserved":0,"allocated":299008,"data_stored":103822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":286720,"data_stored":99159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]}]},{"osd":1,"up_from":11,"seq":47244640265,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5228,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366612,"statfs":{"total":96636764160,"available":96631410688,"internally_reserved":0,"allocated":299008,"data_stored":103822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:02:25.134 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-07-02T14:02:25.134 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:02:25.134 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2023-07-02T14:02:25.134 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-07-02T14:02:25.163 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:25.164 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:25.164 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:25.164 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:25.164 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:25.174 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:02:25.215 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:25.215+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:25.292 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:25.292+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:25.341 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:25.341 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:25.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:25.342 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:25.342 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:25.352 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:02:25.393 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:25.393+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:25.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:02:25.472 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:02:25.472 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:25.472 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 33, 2023-07-02T14:02:25.472 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2", 2023-07-02T14:02:25.472 INFO:teuthology.orchestra.run.smithi016.stdout: "created": "2023-07-02T14:01:37.298060+0000", 2023-07-02T14:02:25.473 INFO:teuthology.orchestra.run.smithi016.stdout: "modified": "2023-07-02T14:02:24.392728+0000", 2023-07-02T14:02:25.473 INFO:teuthology.orchestra.run.smithi016.stdout: "last_up_change": "2023-07-02T14:01:45.271760+0000", 2023-07-02T14:02:25.473 INFO:teuthology.orchestra.run.smithi016.stdout: "last_in_change": "2023-07-02T14:01:39.129121+0000", 2023-07-02T14:02:25.473 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-07-02T14:02:25.473 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_num": 5801984, 2023-07-02T14:02:25.473 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_set": [ 2023-07-02T14:02:25.473 INFO:teuthology.orchestra.run.smithi016.stdout: "noscrub", 2023-07-02T14:02:25.474 INFO:teuthology.orchestra.run.smithi016.stdout: "pglog_hardlimit", 2023-07-02T14:02:25.474 INFO:teuthology.orchestra.run.smithi016.stdout: "purged_snapdirs", 2023-07-02T14:02:25.474 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_deletes", 2023-07-02T14:02:25.474 INFO:teuthology.orchestra.run.smithi016.stdout: "sortbitwise" 2023-07-02T14:02:25.474 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:25.474 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_version": 4, 2023-07-02T14:02:25.474 INFO:teuthology.orchestra.run.smithi016.stdout: "full_ratio": 0.94999998807907104, 2023-07-02T14:02:25.474 INFO:teuthology.orchestra.run.smithi016.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-07-02T14:02:25.475 INFO:teuthology.orchestra.run.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2023-07-02T14:02:25.475 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_snapshot": "", 2023-07-02T14:02:25.475 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_max": 2, 2023-07-02T14:02:25.475 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd": 6, 2023-07-02T14:02:25.475 INFO:teuthology.orchestra.run.smithi016.stdout: "require_min_compat_client": "luminous", 2023-07-02T14:02:25.475 INFO:teuthology.orchestra.run.smithi016.stdout: "min_compat_client": "luminous", 2023-07-02T14:02:25.475 INFO:teuthology.orchestra.run.smithi016.stdout: "require_osd_release": "pacific", 2023-07-02T14:02:25.476 INFO:teuthology.orchestra.run.smithi016.stdout: "pools": [ 2023-07-02T14:02:25.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.476 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 1, 2023-07-02T14:02:25.476 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "device_health_metrics", 2023-07-02T14:02:25.476 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:45.323455+0000", 2023-07-02T14:02:25.476 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2023-07-02T14:02:25.476 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2023-07-02T14:02:25.477 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:02:25.477 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:02:25.477 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:02:25.477 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:02:25.477 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:02:25.477 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:02:25.477 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:02:25.477 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:02:25.478 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:02:25.478 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:02:25.478 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 1, 2023-07-02T14:02:25.478 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 1, 2023-07-02T14:02:25.478 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 1, 2023-07-02T14:02:25.478 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 1, 2023-07-02T14:02:25.478 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 1, 2023-07-02T14:02:25.478 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:02:25.479 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:02:25.479 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:02:25.479 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:02:25.479 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:02:25.479 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:02:25.479 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:02:25.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.479 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "14", 2023-07-02T14:02:25.480 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:02:25.480 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:02:25.480 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:02:25.480 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:02:25.480 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:02:25.480 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2023-07-02T14:02:25.480 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2023-07-02T14:02:25.481 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:02:25.481 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:02:25.481 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:02:25.481 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:02:25.481 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:02:25.481 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:02:25.481 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:02:25.482 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:02:25.482 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:02:25.482 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:02:25.482 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:02:25.482 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:02:25.482 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:02:25.482 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:02:25.483 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:02:25.483 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:02:25.483 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:02:25.483 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:02:25.483 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:02:25.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.483 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:02:25.483 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:02:25.484 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:02:25.484 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:02:25.484 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:02:25.484 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:02:25.484 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:02:25.484 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:02:25.484 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:02:25.485 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:02:25.485 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:02:25.485 INFO:teuthology.orchestra.run.smithi016.stdout: "options": { 2023-07-02T14:02:25.485 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_max": 32, 2023-07-02T14:02:25.485 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_min": 1 2023-07-02T14:02:25.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.485 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:02:25.485 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_devicehealth": {} 2023-07-02T14:02:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 2, 2023-07-02T14:02:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "rbd", 2023-07-02T14:02:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:47.448897+0000", 2023-07-02T14:02:25.486 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2023-07-02T14:02:25.487 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2023-07-02T14:02:25.487 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:02:25.487 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:02:25.487 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:02:25.487 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:02:25.487 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:02:25.487 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:02:25.487 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:02:25.488 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:02:25.488 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:02:25.488 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:02:25.488 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2023-07-02T14:02:25.488 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2023-07-02T14:02:25.488 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2023-07-02T14:02:25.488 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2023-07-02T14:02:25.489 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2023-07-02T14:02:25.489 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:02:25.489 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:02:25.489 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:02:25.489 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:02:25.489 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:02:25.489 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:02:25.489 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:02:25.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.490 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "16", 2023-07-02T14:02:25.490 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:02:25.490 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:02:25.490 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:02:25.490 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:02:25.490 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:02:25.490 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2023-07-02T14:02:25.491 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2023-07-02T14:02:25.491 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:02:25.491 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:02:25.491 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:02:25.491 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:02:25.491 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:02:25.491 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:02:25.491 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:02:25.492 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:02:25.492 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:02:25.492 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:02:25.492 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:02:25.492 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:02:25.492 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:02:25.492 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:02:25.493 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:02:25.493 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:02:25.493 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:02:25.493 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:02:25.493 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:02:25.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.493 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:02:25.493 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:02:25.494 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:02:25.494 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:02:25.494 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:02:25.494 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:02:25.494 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:02:25.494 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:02:25.494 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:02:25.494 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:02:25.495 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:02:25.495 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2023-07-02T14:02:25.495 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:02:25.495 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2023-07-02T14:02:25.495 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.495 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.495 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:25.496 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:02:25.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.496 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:02:25.496 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "43c446fe-c811-4ee3-99e8-d7452edfc1c1", 2023-07-02T14:02:25.496 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:25.496 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:25.496 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:25.496 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:25.497 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:25.497 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:25.497 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:25.497 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 0, 2023-07-02T14:02:25.497 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:25.497 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:25.497 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:25.497 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.498 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.498 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2023-07-02T14:02:25.498 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:25.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.498 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.498 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2023-07-02T14:02:25.499 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:25.499 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.499 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.499 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:25.499 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.499 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.500 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6802", 2023-07-02T14:02:25.500 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:25.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.500 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.500 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6803", 2023-07-02T14:02:25.500 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:25.501 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.501 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.501 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:25.501 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.501 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.501 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6806", 2023-07-02T14:02:25.502 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:25.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.502 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.502 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6807", 2023-07-02T14:02:25.502 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:25.502 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.502 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.503 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:25.503 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.503 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.503 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6804", 2023-07-02T14:02:25.503 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:25.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.504 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.504 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6805", 2023-07-02T14:02:25.504 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:25.504 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.504 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.505 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6801/83359", 2023-07-02T14:02:25.505 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6803/83359", 2023-07-02T14:02:25.505 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6807/83359", 2023-07-02T14:02:25.505 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6805/83359", 2023-07-02T14:02:25.505 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:25.505 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:25.505 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:25.505 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.506 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:02:25.506 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b", 2023-07-02T14:02:25.506 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:25.506 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:25.506 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:25.506 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:25.507 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:25.507 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:25.507 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:25.507 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 25, 2023-07-02T14:02:25.507 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:25.507 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:25.507 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:25.508 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.508 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.508 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6808", 2023-07-02T14:02:25.508 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:25.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.508 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.509 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6809", 2023-07-02T14:02:25.509 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:25.509 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.509 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.509 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:25.509 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.510 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.510 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6810", 2023-07-02T14:02:25.510 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:25.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.510 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.510 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6811", 2023-07-02T14:02:25.510 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:25.511 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.511 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.511 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:25.511 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.511 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.512 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6814", 2023-07-02T14:02:25.512 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:25.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.512 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.512 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6815", 2023-07-02T14:02:25.512 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:25.512 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.513 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.513 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:25.513 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.513 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.513 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6812", 2023-07-02T14:02:25.513 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:25.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.514 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.514 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6813", 2023-07-02T14:02:25.514 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:25.514 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.514 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.515 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6809/83360", 2023-07-02T14:02:25.515 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6811/83360", 2023-07-02T14:02:25.515 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6815/83360", 2023-07-02T14:02:25.515 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6813/83360", 2023-07-02T14:02:25.515 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:25.515 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:25.515 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:25.516 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.516 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:02:25.516 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "99f8ea0a-5a71-4abe-b087-8e891fd11d1b", 2023-07-02T14:02:25.516 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:25.516 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:25.516 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:25.517 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:25.517 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:25.517 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:25.517 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:25.517 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 14, 2023-07-02T14:02:25.517 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:25.517 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:25.517 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:25.518 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.518 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.518 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6816", 2023-07-02T14:02:25.518 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:25.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.519 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.519 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6817", 2023-07-02T14:02:25.519 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:25.519 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.519 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.519 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:25.519 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.520 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.520 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6818", 2023-07-02T14:02:25.520 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:25.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.520 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.520 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6819", 2023-07-02T14:02:25.521 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:25.521 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.521 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.521 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:25.521 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.522 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.522 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6822", 2023-07-02T14:02:25.522 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:25.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.522 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.522 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6823", 2023-07-02T14:02:25.522 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:25.523 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.523 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.523 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:25.523 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.523 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.524 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6820", 2023-07-02T14:02:25.524 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:25.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.524 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.524 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6821", 2023-07-02T14:02:25.524 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:25.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.525 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.525 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6817/83361", 2023-07-02T14:02:25.525 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6819/83361", 2023-07-02T14:02:25.525 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6823/83361", 2023-07-02T14:02:25.525 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6821/83361", 2023-07-02T14:02:25.525 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:25.525 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:25.526 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:25.526 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.526 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:02:25.526 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "d285b4d5-0bef-421a-b84a-943e485b10c2", 2023-07-02T14:02:25.526 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:25.527 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:25.527 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:25.527 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:25.527 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:25.527 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:25.527 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:25.527 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 18, 2023-07-02T14:02:25.527 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:25.528 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:25.528 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:25.528 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.528 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.528 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6808", 2023-07-02T14:02:25.528 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:25.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.529 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.529 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6809", 2023-07-02T14:02:25.529 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:25.529 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.529 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.530 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:25.530 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.530 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.530 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6810", 2023-07-02T14:02:25.530 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:25.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.531 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.531 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6811", 2023-07-02T14:02:25.531 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:25.531 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.531 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.532 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:25.532 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.532 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.532 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6814", 2023-07-02T14:02:25.532 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:25.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.533 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.533 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6815", 2023-07-02T14:02:25.533 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:25.533 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.533 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.533 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:25.534 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.534 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.534 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6812", 2023-07-02T14:02:25.534 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:25.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.535 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.535 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6813", 2023-07-02T14:02:25.535 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:25.535 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.535 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.535 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6809/83350", 2023-07-02T14:02:25.535 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6811/83350", 2023-07-02T14:02:25.536 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6815/83350", 2023-07-02T14:02:25.536 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6813/83350", 2023-07-02T14:02:25.536 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:25.536 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:25.536 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:25.536 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.537 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:02:25.537 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "7ba0b0db-e407-4c63-8278-aab09f3082c4", 2023-07-02T14:02:25.537 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:25.537 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:25.537 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:25.537 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:25.537 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:25.538 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:25.538 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:25.538 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 0, 2023-07-02T14:02:25.538 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:25.538 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:25.538 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:25.538 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.539 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.539 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6800", 2023-07-02T14:02:25.539 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:25.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.539 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.539 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6801", 2023-07-02T14:02:25.539 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:25.540 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.540 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.540 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:25.540 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.540 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.541 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6802", 2023-07-02T14:02:25.541 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:25.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.541 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.541 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6803", 2023-07-02T14:02:25.541 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:25.541 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.542 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.542 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:25.542 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.542 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.542 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6806", 2023-07-02T14:02:25.542 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:25.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.543 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.543 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6807", 2023-07-02T14:02:25.543 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:25.543 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.543 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.544 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:25.544 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.544 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.544 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6804", 2023-07-02T14:02:25.544 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:25.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.545 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.545 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6805", 2023-07-02T14:02:25.545 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:25.545 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.545 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.545 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6801/83351", 2023-07-02T14:02:25.546 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6803/83351", 2023-07-02T14:02:25.546 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6807/83351", 2023-07-02T14:02:25.546 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6805/83351", 2023-07-02T14:02:25.546 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:25.546 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:25.546 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:25.546 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.547 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:02:25.547 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "29663bfa-f6fe-4ce7-b45a-a9b7b4424c42", 2023-07-02T14:02:25.547 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:25.547 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:25.547 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:25.547 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:25.547 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:25.548 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:25.548 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:25.548 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 14, 2023-07-02T14:02:25.548 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:25.548 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:25.548 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:25.548 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.549 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.549 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6816", 2023-07-02T14:02:25.549 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:25.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.549 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.549 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6817", 2023-07-02T14:02:25.550 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:25.550 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.550 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.550 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:25.550 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.550 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.551 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6818", 2023-07-02T14:02:25.551 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:25.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.551 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.551 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6819", 2023-07-02T14:02:25.551 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:25.552 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.552 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.552 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:25.552 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.552 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.552 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6822", 2023-07-02T14:02:25.553 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:25.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.553 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.553 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6823", 2023-07-02T14:02:25.553 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:25.553 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.553 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.554 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:25.554 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:25.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.554 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:25.554 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6820", 2023-07-02T14:02:25.554 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:25.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.555 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:25.555 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6821", 2023-07-02T14:02:25.555 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:25.555 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.555 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.556 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6817/83352", 2023-07-02T14:02:25.556 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6819/83352", 2023-07-02T14:02:25.556 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6823/83352", 2023-07-02T14:02:25.556 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6821/83352", 2023-07-02T14:02:25.556 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:25.556 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:25.556 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:25.557 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.557 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.557 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:25.557 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_xinfo": [ 2023-07-02T14:02:25.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.557 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:02:25.557 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:25.558 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:25.558 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:25.558 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:25.558 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:25.558 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.001178+0000", 2023-07-02T14:02:25.558 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:25.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.559 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:02:25.559 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:25.559 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:25.559 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:25.559 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:25.559 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:25.559 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.184181+0000", 2023-07-02T14:02:25.560 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:25.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.560 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:02:25.560 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:25.560 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:25.560 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:25.560 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:25.561 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:25.561 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.274854+0000", 2023-07-02T14:02:25.561 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:25.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.561 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:02:25.561 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:25.561 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:25.562 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:25.562 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:25.562 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:25.562 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.186217+0000", 2023-07-02T14:02:25.562 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:25.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.563 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:02:25.563 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:25.563 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:25.563 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:25.563 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:25.563 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:25.563 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.247442+0000", 2023-07-02T14:02:25.563 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:25.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.564 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:02:25.564 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:25.564 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:25.564 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:25.564 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:25.565 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:25.565 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.220075+0000", 2023-07-02T14:02:25.565 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:25.565 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.565 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:25.565 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap": [ 2023-07-02T14:02:25.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.565 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.2", 2023-07-02T14:02:25.566 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:02:25.566 INFO:teuthology.orchestra.run.smithi016.stdout: 1, 2023-07-02T14:02:25.566 INFO:teuthology.orchestra.run.smithi016.stdout: 5 2023-07-02T14:02:25.566 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.566 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.566 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:25.566 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_items": [ 2023-07-02T14:02:25.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.567 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.7", 2023-07-02T14:02:25.567 INFO:teuthology.orchestra.run.smithi016.stdout: "mappings": [ 2023-07-02T14:02:25.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:25.567 INFO:teuthology.orchestra.run.smithi016.stdout: "from": 4, 2023-07-02T14:02:25.567 INFO:teuthology.orchestra.run.smithi016.stdout: "to": 5 2023-07-02T14:02:25.567 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.567 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:25.567 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.568 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:25.568 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_temp": [], 2023-07-02T14:02:25.568 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_temp": [], 2023-07-02T14:02:25.568 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklist": {}, 2023-07-02T14:02:25.568 INFO:teuthology.orchestra.run.smithi016.stdout: "range_blocklist": {}, 2023-07-02T14:02:25.568 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profiles": { 2023-07-02T14:02:25.568 INFO:teuthology.orchestra.run.smithi016.stdout: "default": { 2023-07-02T14:02:25.569 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2023-07-02T14:02:25.569 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "2", 2023-07-02T14:02:25.569 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "1", 2023-07-02T14:02:25.569 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "jerasure", 2023-07-02T14:02:25.569 INFO:teuthology.orchestra.run.smithi016.stdout: "ruleset-failure-domain": "osd", 2023-07-02T14:02:25.569 INFO:teuthology.orchestra.run.smithi016.stdout: "technique": "reed_sol_van" 2023-07-02T14:02:25.569 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:25.570 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps_queue": [], 2023-07-02T14:02:25.570 INFO:teuthology.orchestra.run.smithi016.stdout: "new_removed_snaps": [], 2023-07-02T14:02:25.570 INFO:teuthology.orchestra.run.smithi016.stdout: "new_purged_snaps": [], 2023-07-02T14:02:25.570 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_node_flags": {}, 2023-07-02T14:02:25.570 INFO:teuthology.orchestra.run.smithi016.stdout: "device_class_flags": {}, 2023-07-02T14:02:25.570 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode": { 2023-07-02T14:02:25.570 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_enabled": false, 2023-07-02T14:02:25.570 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_bucket_count": 0, 2023-07-02T14:02:25.571 INFO:teuthology.orchestra.run.smithi016.stdout: "degraded_stretch_mode": 0, 2023-07-02T14:02:25.571 INFO:teuthology.orchestra.run.smithi016.stdout: "recovering_stretch_mode": 0, 2023-07-02T14:02:25.571 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_bucket": 0 2023-07-02T14:02:25.571 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:25.571 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:25.573 INFO:tasks.thrashosds.thrasher:j is {'epoch': 33, 'fsid': '9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2', 'created': '2023-07-02T14:01:37.298060+0000', 'modified': '2023-07-02T14:02:24.392728+0000', 'last_up_change': '2023-07-02T14:01:45.271760+0000', 'last_in_change': '2023-07-02T14:01:39.129121+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-07-02T14:01:45.323455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-02T14:01:47.448897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '43c446fe-c811-4ee3-99e8-d7452edfc1c1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6800', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6801', 'nonce': 83359}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6802', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6803', 'nonce': 83359}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6806', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6807', 'nonce': 83359}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6804', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6805', 'nonce': 83359}]}, 'public_addr': '172.21.15.16:6801/83359', 'cluster_addr': '172.21.15.16:6803/83359', 'heartbeat_back_addr': '172.21.15.16:6807/83359', 'heartbeat_front_addr': '172.21.15.16:6805/83359', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 25, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6808', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6809', 'nonce': 83360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6810', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6811', 'nonce': 83360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6814', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6815', 'nonce': 83360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6812', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6813', 'nonce': 83360}]}, 'public_addr': '172.21.15.16:6809/83360', 'cluster_addr': '172.21.15.16:6811/83360', 'heartbeat_back_addr': '172.21.15.16:6815/83360', 'heartbeat_front_addr': '172.21.15.16:6813/83360', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '99f8ea0a-5a71-4abe-b087-8e891fd11d1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 14, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6816', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6817', 'nonce': 83361}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6818', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6819', 'nonce': 83361}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6822', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6823', 'nonce': 83361}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6820', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6821', 'nonce': 83361}]}, 'public_addr': '172.21.15.16:6817/83361', 'cluster_addr': '172.21.15.16:6819/83361', 'heartbeat_back_addr': '172.21.15.16:6823/83361', 'heartbeat_front_addr': '172.21.15.16:6821/83361', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd285b4d5-0bef-421a-b84a-943e485b10c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6808', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6809', 'nonce': 83350}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6810', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6811', 'nonce': 83350}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6814', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6815', 'nonce': 83350}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6812', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6813', 'nonce': 83350}]}, 'public_addr': '172.21.15.22:6809/83350', 'cluster_addr': '172.21.15.22:6811/83350', 'heartbeat_back_addr': '172.21.15.22:6815/83350', 'heartbeat_front_addr': '172.21.15.22:6813/83350', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7ba0b0db-e407-4c63-8278-aab09f3082c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6800', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6801', 'nonce': 83351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6802', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6803', 'nonce': 83351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6806', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6807', 'nonce': 83351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6804', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6805', 'nonce': 83351}]}, 'public_addr': '172.21.15.22:6801/83351', 'cluster_addr': '172.21.15.22:6803/83351', 'heartbeat_back_addr': '172.21.15.22:6807/83351', 'heartbeat_front_addr': '172.21.15.22:6805/83351', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '29663bfa-f6fe-4ce7-b45a-a9b7b4424c42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 14, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6816', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6817', 'nonce': 83352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6818', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6819', 'nonce': 83352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6822', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6823', 'nonce': 83352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6820', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6821', 'nonce': 83352}]}, 'public_addr': '172.21.15.22:6817/83352', 'cluster_addr': '172.21.15.22:6819/83352', 'heartbeat_back_addr': '172.21.15.22:6823/83352', 'heartbeat_front_addr': '172.21.15.22:6821/83352', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.001178+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.184181+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.274854+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.186217+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.247442+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.220075+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.2', 'osds': [1, 5]}], 'pg_upmap_items': [{'pgid': '2.7', 'mappings': [{'from': 4, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-07-02T14:02:25.574 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:02:25.574 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:25.575 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:25.575 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:25.575 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:25.575 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:02:25.576 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:25.572+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:25.674 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:25.673+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:25.695 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:02:25.743 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:25.743 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:25.754 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:02:25.774 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:25.773+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:25.857 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:25.875 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:25.874+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:25.892 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:25.915 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:25.916 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:25.919 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:25.926 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:02:25.933 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:02:25.935 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:02:25.938 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:25.946 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":48,"stamp":"2023-07-02T14:02:24.395932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":31000,"kb_used_data":1704,"kb_used_omap":0,"kb_used_meta":29248,"kb_avail":566200040,"statfs":{"total":579820584960,"available":579788840960,"internally_reserved":0,"allocated":1744896,"data_stored":604280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29949952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":28,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370920+0000","last_change":"2023-07-02T14:02:01.726288+0000","last_active":"2023-07-02T14:02:17.370920+0000","last_peered":"2023-07-02T14:02:17.370920+0000","last_clean":"2023-07-02T14:02:17.370920+0000","last_became_active":"2023-07-02T14:02:01.725984+0000","last_became_peered":"2023-07-02T14:02:01.725984+0000","last_unstale":"2023-07-02T14:02:17.370920+0000","last_undegraded":"2023-07-02T14:02:17.370920+0000","last_fullsized":"2023-07-02T14:02:17.370920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370429+0000","last_change":"2023-07-02T14:01:49.295075+0000","last_active":"2023-07-02T14:02:17.370429+0000","last_peered":"2023-07-02T14:02:17.370429+0000","last_clean":"2023-07-02T14:02:17.370429+0000","last_became_active":"2023-07-02T14:01:49.293782+0000","last_became_peered":"2023-07-02T14:01:49.293782+0000","last_unstale":"2023-07-02T14:02:17.370429+0000","last_undegraded":"2023-07-02T14:02:17.370429+0000","last_fullsized":"2023-07-02T14:02:17.370429+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370895+0000","last_change":"2023-07-02T14:01:49.293139+0000","last_active":"2023-07-02T14:02:17.370895+0000","last_peered":"2023-07-02T14:02:17.370895+0000","last_clean":"2023-07-02T14:02:17.370895+0000","last_became_active":"2023-07-02T14:01:49.292790+0000","last_became_peered":"2023-07-02T14:01:49.292790+0000","last_unstale":"2023-07-02T14:02:17.370895+0000","last_undegraded":"2023-07-02T14:02:17.370895+0000","last_fullsized":"2023-07-02T14:02:17.370895+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370871+0000","last_change":"2023-07-02T14:01:49.293994+0000","last_active":"2023-07-02T14:02:17.370871+0000","last_peered":"2023-07-02T14:02:17.370871+0000","last_clean":"2023-07-02T14:02:17.370871+0000","last_became_active":"2023-07-02T14:01:49.293608+0000","last_became_peered":"2023-07-02T14:01:49.293608+0000","last_unstale":"2023-07-02T14:02:17.370871+0000","last_undegraded":"2023-07-02T14:02:17.370871+0000","last_fullsized":"2023-07-02T14:02:17.370871+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370875+0000","last_change":"2023-07-02T14:01:47.288970+0000","last_active":"2023-07-02T14:02:17.370875+0000","last_peered":"2023-07-02T14:02:17.370875+0000","last_clean":"2023-07-02T14:02:17.370875+0000","last_became_active":"2023-07-02T14:01:47.288610+0000","last_became_peered":"2023-07-02T14:01:47.288610+0000","last_unstale":"2023-07-02T14:02:17.370875+0000","last_undegraded":"2023-07-02T14:02:17.370875+0000","last_fullsized":"2023-07-02T14:02:17.370875+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-07-02T14:02:15.024606+0000","last_change":"2023-07-02T14:01:49.298280+0000","last_active":"2023-07-02T14:02:15.024606+0000","last_peered":"2023-07-02T14:02:15.024606+0000","last_clean":"2023-07-02T14:02:15.024606+0000","last_became_active":"2023-07-02T14:01:49.297946+0000","last_became_peered":"2023-07-02T14:01:49.297946+0000","last_unstale":"2023-07-02T14:02:15.024606+0000","last_undegraded":"2023-07-02T14:02:15.024606+0000","last_fullsized":"2023-07-02T14:02:15.024606+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.395981+0000","last_change":"2023-07-02T14:01:49.295243+0000","last_active":"2023-07-02T14:02:17.395981+0000","last_peered":"2023-07-02T14:02:17.395981+0000","last_clean":"2023-07-02T14:02:17.395981+0000","last_became_active":"2023-07-02T14:01:49.294964+0000","last_became_peered":"2023-07-02T14:01:49.294964+0000","last_unstale":"2023-07-02T14:02:17.395981+0000","last_undegraded":"2023-07-02T14:02:17.395981+0000","last_fullsized":"2023-07-02T14:02:17.395981+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370424+0000","last_change":"2023-07-02T14:02:15.025638+0000","last_active":"2023-07-02T14:02:17.370424+0000","last_peered":"2023-07-02T14:02:17.370424+0000","last_clean":"2023-07-02T14:02:17.370424+0000","last_became_active":"2023-07-02T14:02:15.025376+0000","last_became_peered":"2023-07-02T14:02:15.025376+0000","last_unstale":"2023-07-02T14:02:17.370424+0000","last_undegraded":"2023-07-02T14:02:17.370424+0000","last_fullsized":"2023-07-02T14:02:17.370424+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-07-02T14:02:17.370524+0000","last_change":"2023-07-02T14:01:49.293092+0000","last_active":"2023-07-02T14:02:17.370524+0000","last_peered":"2023-07-02T14:02:17.370524+0000","last_clean":"2023-07-02T14:02:17.370524+0000","last_became_active":"2023-07-02T14:01:49.292773+0000","last_became_peered":"2023-07-02T14:01:49.292773+0000","last_unstale":"2023-07-02T14:02:17.370524+0000","last_undegraded":"2023-07-02T14:02:17.370524+0000","last_fullsized":"2023-07-02T14:02:17.370524+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":286720,"data_stored":99159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640265,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5152,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":286720,"data_stored":99159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]}]},{"osd":3,"up_from":11,"seq":47244640265,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":286720,"data_stored":99159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640265,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5164,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366676,"statfs":{"total":96636764160,"available":96631476224,"internally_reserved":0,"allocated":299008,"data_stored":103822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5152,"kb_used_data":280,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366688,"statfs":{"total":96636764160,"available":96631488512,"internally_reserved":0,"allocated":286720,"data_stored":99159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]}]},{"osd":1,"up_from":11,"seq":47244640265,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5228,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366612,"statfs":{"total":96636764160,"available":96631410688,"internally_reserved":0,"allocated":299008,"data_stored":103822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:02:25.947 INFO:tasks.thrashosds.thrasher:Setting 2.4 to [5, 0] 2023-07-02T14:02:25.948 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.4', '5', '0'] 2023-07-02T14:02:25.948 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.4 5 0 2023-07-02T14:02:25.957 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:25.975 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:25.974+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:25.975 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:25.976 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:02:25.977 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:02:25.977 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:02:25.977 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:02:25.977 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:02:25.977 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:02:26.062 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:26.062 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:26.073 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:02:26.114 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:26.113+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:26.177 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:26.176+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:26.236 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:26.237 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:26.247 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:02:26.278 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:26.278+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:26.379 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:26.378+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:26.405 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:02:26.405 INFO:teuthology.orchestra.run.smithi016.stderr:set 2.4 pg_upmap mapping to [5,0] 2023-07-02T14:02:26.412 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:26.412 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:26.423 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:02:26.480 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:26.480+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:26.581 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:26.580+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:26.587 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:26.588 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:26.598 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:02:26.682 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:26.682+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:26.721 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:26.722 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:26.731 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:02:26.782 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:26.782+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:26.856 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:26.856 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:26.866 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:02:26.908 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:26.907+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:26.985 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:26.985+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:27.031 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:27.031 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:27.041 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:02:27.086 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:27.085+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:27.165 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:27.166 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:27.176 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:02:27.187 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:27.187+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:27.288 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:27.288+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:27.340 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:27.341 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:27.351 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:02:27.395 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:27.395+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:27.490 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:27.490+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:27.491 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:27.491 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:27.501 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:02:27.591 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:27.591+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:27.666 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:27.666 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:27.676 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:02:27.693 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:27.692+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:27.793 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:27.793+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:27.841 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:27.841 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:27.851 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:02:27.895 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:27.894+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:27.975 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:27.975 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:27.985 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:02:27.989 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:02:28.029 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:28.029+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:28.096 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:28.096+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:28.110 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:28.110 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:28.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:02:28.167 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:28.187 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:28.196 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:28.196+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:28.205 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:28.223 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:28.242 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:28.261 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:28.262 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:02:28.262 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:02:28.262 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:02:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:02:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:02:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:02:28.289 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:28.289 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:28.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:28.290 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:28.290 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:28.297 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:28.297+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:28.299 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:02:28.397 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:28.397+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:28.419 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:02:28.466 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:28.466 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:28.466 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:28.466 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:28.477 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:02:28.521 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:28.521+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:28.598 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:28.598+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:28.637 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:28.638 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:28.638 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:28.638 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:28.638 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:28.648 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:02:28.731 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:28.731+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:28.800 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:28.800+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:28.814 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:28.814 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:28.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:28.815 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:28.815 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:28.824 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:02:28.901 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:28.901+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:28.962 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:28.963 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:28.963 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:28.963 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:28.973 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:02:29.002 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:29.002+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:29.097 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:29.097 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:29.098 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:29.098 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:29.098 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:29.103 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:29.102+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:29.107 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:02:29.204 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:29.203+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:29.271 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:29.272 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:29.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:29.272 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [] 2023-07-02T14:02:29.273 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:29.282 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:02:29.305 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:29.305+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:29.406 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:29.405+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:29.416 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:02:29.446 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:29.447 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:29.447 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:29.447 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:29.457 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:02:29.540 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:29.539+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:29.607 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:29.607+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:29.620 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:29.620 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:29.620 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:29.621 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:29.621 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:29.631 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:02:29.708 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:29.708+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:29.795 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:29.795 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:29.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:29.795 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [] 2023-07-02T14:02:29.796 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:29.805 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:02:29.849 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:29.848+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:29.910 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:29.910+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:29.928 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:29.929 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:29.929 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:29.929 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:29.939 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:02:30.022 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:30.021+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:30.103 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:30.103 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:30.103 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:30.103 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:30.103 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:30.111 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:30.110+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:30.113 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:02:30.212 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:30.212+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:30.274 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:02:30.277 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:30.277 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:30.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:30.277 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [] 2023-07-02T14:02:30.278 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:30.287 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:02:30.312 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:30.312+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:30.413 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:30.413+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:30.428 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:30.446 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:30.451 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:30.451 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:30.461 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:02:30.464 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:30.482 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:30.500 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:30.513 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:30.512+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:30.518 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:30.519 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:02:30.519 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:02:30.519 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:02:30.520 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:02:30.520 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:02:30.520 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:02:30.588 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:30.588 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:30.598 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:02:30.642 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:30.642+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:30.715 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:30.715+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:30.722 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:30.722 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:30.733 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:02:30.816 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:30.815+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:30.897 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:30.898 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:30.907 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:02:30.917 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:30.917+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:31.018 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:31.018+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:31.030 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:31.030 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:31.041 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:02:31.125 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:31.125+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:31.206 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:31.206 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:31.216 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:02:31.257 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:31.257+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:31.320 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:31.319+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:31.380 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:31.380 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:31.390 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:02:31.418 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2023-07-02T14:02:31.418 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:02:31.419 INFO:tasks.thrashosds.thrasher:primary_affinity 2023-07-02T14:02:31.419 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.232982 2023-07-02T14:02:31.419 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 0.2329824245858999 2023-07-02T14:02:31.429 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:02:31.433 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:31.433+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:31.518 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:31.518 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:31.521 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:31.521+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:31.529 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:02:31.622 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:31.622+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:31.696 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:31.697 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:31.707 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:02:31.748 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:31.748+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:31.823 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:31.822+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:31.844 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:31.845 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:31.855 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:02:31.937 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:31.937+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:32.019 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:32.019 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:32.024 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:32.024+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:32.029 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:02:32.125 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:32.125+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:32.193 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:32.217 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:32.218 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:02:32.262 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:32.261+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:32.327 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:32.327+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:32.342 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:32.342 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:32.352 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:02:32.420 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.4 primary-affinity to 0.232982 (8152682) 2023-07-02T14:02:32.420 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:02:32.428 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:32.427+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:32.515 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:32.515 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:32.526 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:02:32.528 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:32.528+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:32.531 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:02:32.628 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:32.628+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:32.650 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:32.651 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:32.661 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:02:32.676 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:32.693 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:32.711 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:32.729 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:32.729 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:32.728+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:32.747 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:32.765 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:32.766 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:02:32.766 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:02:32.767 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:02:32.767 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:02:32.767 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:02:32.767 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:02:32.785 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:32.785 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:32.785 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:32.786 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:32.795 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:02:32.830 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:32.830+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:32.931 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:32.930+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:32.960 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:32.960 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:32.960 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:32.960 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:32.960 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:32.970 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:02:33.032 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:33.032+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:33.132 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:33.132+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:33.134 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:33.134 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:33.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:33.135 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [] 2023-07-02T14:02:33.135 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:33.144 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:02:33.233 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:33.233+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:33.271 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:33.271 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:33.271 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:33.271 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:33.281 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:02:33.334 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:33.334+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:33.405 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:33.405 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:33.405 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:33.405 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:33.406 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:33.415 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:02:33.456 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:33.456+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:33.535 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:33.534+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:33.580 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:33.581 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:33.590 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:02:33.636 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:33.636+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:33.714 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:33.714 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:33.714 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:33.714 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:33.724 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:02:33.765 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:33.765+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:33.838 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:33.838+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:33.888 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:33.888 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:33.889 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:33.889 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:33.889 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:33.898 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:02:33.939 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:33.939+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:34.040 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:34.040+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:34.062 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:34.062 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:34.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:34.063 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:34.063 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:34.072 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:02:34.155 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:34.155+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:34.236 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:34.236 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:34.236 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:34.237 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:34.242 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:34.242+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:34.246 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:02:34.342 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:34.342+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:34.410 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:34.410 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:34.410 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:34.411 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:34.411 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:34.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:02:34.437 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:02:34.443 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:34.442+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:34.544 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:34.543+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:34.591 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:34.591 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:34.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:34.591 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:34.591 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:34.602 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:02:34.645 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:34.644+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:34.740 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:34.740 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:34.740 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:34.740 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:34.745 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:34.745+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:34.750 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:02:34.778 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:02:34.846 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:34.846+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:34.915 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:34.916 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:34.916 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:34.917 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:34.917 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:34.917 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:34.927 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:02:34.933 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:34.950 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:34.967 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:34.969 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:34.969+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:34.985 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:35.003 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:35.004 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:02:35.004 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:02:35.004 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:02:35.004 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:02:35.004 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:02:35.005 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:02:35.050 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:35.051 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:35.056 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:35.056+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:35.060 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:02:35.148 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:35.148+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:35.225 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:35.225 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:35.235 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:02:35.276 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:35.275+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:35.350 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:35.350+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:35.398 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:35.398 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:35.408 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:02:35.425 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:02:35.478 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:35.478+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:35.532 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:35.532 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:35.544 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:02:35.588 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:35.587+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:35.653 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:35.652+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:35.668 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:35.669 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:35.679 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:02:35.754 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:35.754+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:35.842 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:35.842 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:35.852 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:02:35.855 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:35.855+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:35.956 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:35.955+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:35.976 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:35.976 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:35.986 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:02:36.057 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:36.057+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:36.124 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:36.124 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:36.125 INFO:tasks.workunit.client.0.smithi163.stderr:Note: switching to 'cbd88633a9c19779ad4b2e367bdc17f83b3a065d'. 2023-07-02T14:02:36.126 INFO:tasks.workunit.client.0.smithi163.stderr: 2023-07-02T14:02:36.126 INFO:tasks.workunit.client.0.smithi163.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-07-02T14:02:36.126 INFO:tasks.workunit.client.0.smithi163.stderr:changes and commit them, and you can discard any commits you make in this 2023-07-02T14:02:36.126 INFO:tasks.workunit.client.0.smithi163.stderr:state without impacting any branches by switching back to a branch. 2023-07-02T14:02:36.126 INFO:tasks.workunit.client.0.smithi163.stderr: 2023-07-02T14:02:36.126 INFO:tasks.workunit.client.0.smithi163.stderr:If you want to create a new branch to retain commits you create, you may 2023-07-02T14:02:36.127 INFO:tasks.workunit.client.0.smithi163.stderr:do so (now or later) by using -c with the switch command. Example: 2023-07-02T14:02:36.127 INFO:tasks.workunit.client.0.smithi163.stderr: 2023-07-02T14:02:36.127 INFO:tasks.workunit.client.0.smithi163.stderr: git switch -c 2023-07-02T14:02:36.127 INFO:tasks.workunit.client.0.smithi163.stderr: 2023-07-02T14:02:36.127 INFO:tasks.workunit.client.0.smithi163.stderr:Or undo this operation with: 2023-07-02T14:02:36.127 INFO:tasks.workunit.client.0.smithi163.stderr: 2023-07-02T14:02:36.127 INFO:tasks.workunit.client.0.smithi163.stderr: git switch - 2023-07-02T14:02:36.127 INFO:tasks.workunit.client.0.smithi163.stderr: 2023-07-02T14:02:36.128 INFO:tasks.workunit.client.0.smithi163.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-07-02T14:02:36.128 INFO:tasks.workunit.client.0.smithi163.stderr: 2023-07-02T14:02:36.128 INFO:tasks.workunit.client.0.smithi163.stderr:HEAD is now at cbd88633a9c qa/workunits/rbd: make continuous export-diff test actually work 2023-07-02T14:02:36.132 DEBUG:teuthology.orchestra.run.smithi163:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-07-02T14:02:36.134 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:02:36.158 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:36.158+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:36.194 INFO:tasks.workunit.client.0.smithi163.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-07-02T14:02:36.196 INFO:tasks.workunit.client.0.smithi163.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-07-02T14:02:36.197 INFO:tasks.workunit.client.0.smithi163.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-07-02T14:02:36.259 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:36.259+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:36.298 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:36.298 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:36.308 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:02:36.391 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:36.391+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:36.460 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:36.460+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:36.473 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:36.473 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:36.483 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:02:36.561 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:36.560+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:36.648 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:36.648 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:36.658 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:02:36.703 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:36.703+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:36.763 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:36.763+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:36.782 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:36.782 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:36.792 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:02:36.865 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:36.864+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:36.895 INFO:tasks.workunit.client.0.smithi163.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-07-02T14:02:36.956 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:36.956 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:36.965 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:36.965+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:36.966 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:02:36.970 INFO:tasks.workunit.client.0.smithi163.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-07-02T14:02:37.008 INFO:tasks.workunit.client.0.smithi163.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-07-02T14:02:37.010 INFO:tasks.workunit.client.0.smithi163.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-07-02T14:02:37.010 INFO:tasks.workunit.client.0.smithi163.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-07-02T14:02:37.015 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:02:37.047 INFO:tasks.workunit.client.0.smithi163.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-07-02T14:02:37.050 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2023-07-02T14:02:37.050 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-07-02T14:02:37.092 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:37.092+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:37.105 INFO:tasks.workunit:Running workunits matching rbd/diff_continuous.sh on client.0... 2023-07-02T14:02:37.106 INFO:tasks.workunit:Running workunit rbd/diff_continuous.sh... 2023-07-02T14:02:37.106 DEBUG:teuthology.orchestra.run.smithi163:workunit test rbd/diff_continuous.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cbd88633a9c19779ad4b2e367bdc17f83b3a065d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh 2023-07-02T14:02:37.135 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:37.135 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:37.145 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:02:37.164 INFO:tasks.workunit.client.0.smithi163.stderr:+ set -o pipefail 2023-07-02T14:02:37.164 INFO:tasks.workunit.client.0.smithi163.stderr:+ readonly DEVICE_TYPE=nbd 2023-07-02T14:02:37.164 INFO:tasks.workunit.client.0.smithi163.stderr:+ DEVICE_TYPE=nbd 2023-07-02T14:02:37.164 INFO:tasks.workunit.client.0.smithi163.stderr:++ uuidgen 2023-07-02T14:02:37.165 INFO:tasks.workunit.client.0.smithi163.stderr:+ BASE_UUID=b3fbb028-e267-4d25-a750-93ba18a2316a 2023-07-02T14:02:37.165 INFO:tasks.workunit.client.0.smithi163.stderr:+ readonly BASE_UUID 2023-07-02T14:02:37.165 INFO:tasks.workunit.client.0.smithi163.stderr:+ readonly SIZE=2G 2023-07-02T14:02:37.166 INFO:tasks.workunit.client.0.smithi163.stderr:+ SIZE=2G 2023-07-02T14:02:37.166 INFO:tasks.workunit.client.0.smithi163.stderr:+ readonly SRC=b3fbb028-e267-4d25-a750-93ba18a2316a-src 2023-07-02T14:02:37.166 INFO:tasks.workunit.client.0.smithi163.stderr:+ SRC=b3fbb028-e267-4d25-a750-93ba18a2316a-src 2023-07-02T14:02:37.166 INFO:tasks.workunit.client.0.smithi163.stderr:+ readonly DST=b3fbb028-e267-4d25-a750-93ba18a2316a-dst 2023-07-02T14:02:37.166 INFO:tasks.workunit.client.0.smithi163.stderr:+ DST=b3fbb028-e267-4d25-a750-93ba18a2316a-dst 2023-07-02T14:02:37.166 INFO:tasks.workunit.client.0.smithi163.stderr:+ readonly MOUNT=b3fbb028-e267-4d25-a750-93ba18a2316a-mnt 2023-07-02T14:02:37.166 INFO:tasks.workunit.client.0.smithi163.stderr:+ MOUNT=b3fbb028-e267-4d25-a750-93ba18a2316a-mnt 2023-07-02T14:02:37.167 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd create -s 2G --stripe-unit 64K --stripe-count 8 --image-feature exclusive-lock,object-map,fast-diff b3fbb028-e267-4d25-a750-93ba18a2316a-src 2023-07-02T14:02:37.168 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:37.167+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:37.189 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:37.207 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:37.226 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:37.244 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:37.263 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:37.268 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:37.268+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:37.281 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:02:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:02:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:02:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:02:37.283 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:02:37.284 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:02:37.312 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:37.312 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:37.313 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:37.313 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:37.313 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:37.323 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:02:37.369 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:37.368+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:37.437 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2023-07-02T14:02:37.437 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:02:37.437 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2023-07-02T14:02:37.438 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-07-02T14:02:37.438 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:02:37.482 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:37.482+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:37.491 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:37.492 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:37.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:37.492 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [] 2023-07-02T14:02:37.492 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:37.503 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:02:37.570 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:37.570+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:37.642 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:37.642 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:37.642 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:37.642 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:37.652 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:02:37.694 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:37.693+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:37.745 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:02:37.746 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:37.746 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 38, 2023-07-02T14:02:37.746 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2", 2023-07-02T14:02:37.746 INFO:teuthology.orchestra.run.smithi016.stdout: "created": "2023-07-02T14:01:37.298060+0000", 2023-07-02T14:02:37.746 INFO:teuthology.orchestra.run.smithi016.stdout: "modified": "2023-07-02T14:02:35.418238+0000", 2023-07-02T14:02:37.746 INFO:teuthology.orchestra.run.smithi016.stdout: "last_up_change": "2023-07-02T14:01:45.271760+0000", 2023-07-02T14:02:37.747 INFO:teuthology.orchestra.run.smithi016.stdout: "last_in_change": "2023-07-02T14:01:39.129121+0000", 2023-07-02T14:02:37.747 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-07-02T14:02:37.747 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_num": 5801984, 2023-07-02T14:02:37.747 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_set": [ 2023-07-02T14:02:37.747 INFO:teuthology.orchestra.run.smithi016.stdout: "noscrub", 2023-07-02T14:02:37.747 INFO:teuthology.orchestra.run.smithi016.stdout: "pglog_hardlimit", 2023-07-02T14:02:37.747 INFO:teuthology.orchestra.run.smithi016.stdout: "purged_snapdirs", 2023-07-02T14:02:37.748 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_deletes", 2023-07-02T14:02:37.748 INFO:teuthology.orchestra.run.smithi016.stdout: "sortbitwise" 2023-07-02T14:02:37.748 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:37.748 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_version": 4, 2023-07-02T14:02:37.749 INFO:teuthology.orchestra.run.smithi016.stdout: "full_ratio": 0.94999998807907104, 2023-07-02T14:02:37.749 INFO:teuthology.orchestra.run.smithi016.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-07-02T14:02:37.749 INFO:teuthology.orchestra.run.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2023-07-02T14:02:37.749 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_snapshot": "", 2023-07-02T14:02:37.749 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_max": 2, 2023-07-02T14:02:37.749 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd": 6, 2023-07-02T14:02:37.749 INFO:teuthology.orchestra.run.smithi016.stdout: "require_min_compat_client": "luminous", 2023-07-02T14:02:37.750 INFO:teuthology.orchestra.run.smithi016.stdout: "min_compat_client": "luminous", 2023-07-02T14:02:37.750 INFO:teuthology.orchestra.run.smithi016.stdout: "require_osd_release": "pacific", 2023-07-02T14:02:37.750 INFO:teuthology.orchestra.run.smithi016.stdout: "pools": [ 2023-07-02T14:02:37.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.750 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 1, 2023-07-02T14:02:37.751 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "device_health_metrics", 2023-07-02T14:02:37.751 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:45.323455+0000", 2023-07-02T14:02:37.751 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2023-07-02T14:02:37.751 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2023-07-02T14:02:37.751 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:02:37.751 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:02:37.752 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:02:37.752 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:02:37.752 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:02:37.752 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:02:37.752 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:02:37.752 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:02:37.753 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:02:37.753 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:02:37.753 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 1, 2023-07-02T14:02:37.753 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 1, 2023-07-02T14:02:37.753 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 1, 2023-07-02T14:02:37.753 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 1, 2023-07-02T14:02:37.753 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 1, 2023-07-02T14:02:37.754 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:02:37.754 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:02:37.754 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:02:37.754 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:02:37.754 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:02:37.754 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:02:37.754 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:02:37.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.755 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "14", 2023-07-02T14:02:37.755 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:02:37.755 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:02:37.755 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:02:37.755 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:02:37.756 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:02:37.756 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2023-07-02T14:02:37.756 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2023-07-02T14:02:37.756 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:02:37.756 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:02:37.756 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:02:37.756 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:02:37.757 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:02:37.757 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:02:37.757 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:02:37.757 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:02:37.757 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:02:37.757 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:02:37.757 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:02:37.758 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:02:37.758 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:02:37.758 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:02:37.758 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:02:37.758 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:02:37.758 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:02:37.758 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:02:37.759 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:02:37.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.759 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:02:37.759 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:02:37.759 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:02:37.759 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:02:37.759 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:02:37.760 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:02:37.760 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:02:37.760 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:02:37.760 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:02:37.760 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:02:37.761 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:02:37.761 INFO:teuthology.orchestra.run.smithi016.stdout: "options": { 2023-07-02T14:02:37.761 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_max": 32, 2023-07-02T14:02:37.761 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_min": 1 2023-07-02T14:02:37.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.762 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:02:37.762 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_devicehealth": {} 2023-07-02T14:02:37.762 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.762 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 2, 2023-07-02T14:02:37.762 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "rbd", 2023-07-02T14:02:37.763 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:47.448897+0000", 2023-07-02T14:02:37.763 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2023-07-02T14:02:37.763 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2023-07-02T14:02:37.763 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:02:37.763 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:02:37.763 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:02:37.763 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:02:37.764 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:02:37.764 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:02:37.764 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:02:37.764 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:02:37.764 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:02:37.764 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:02:37.764 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2023-07-02T14:02:37.765 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2023-07-02T14:02:37.765 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2023-07-02T14:02:37.765 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2023-07-02T14:02:37.765 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2023-07-02T14:02:37.765 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:02:37.765 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:02:37.765 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:02:37.766 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:02:37.766 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:02:37.766 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:02:37.766 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:02:37.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.766 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "16", 2023-07-02T14:02:37.767 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:02:37.767 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:02:37.767 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:02:37.767 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:02:37.767 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:02:37.767 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2023-07-02T14:02:37.767 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2023-07-02T14:02:37.768 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:02:37.768 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:02:37.768 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:02:37.768 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:02:37.768 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:02:37.768 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:02:37.769 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:02:37.769 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:02:37.769 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:02:37.769 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:02:37.769 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:02:37.769 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:02:37.769 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:02:37.770 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:02:37.770 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:02:37.770 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:02:37.770 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:02:37.770 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:02:37.770 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:02:37.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.771 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:02:37.771 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:02:37.771 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:02:37.771 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:02:37.771 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:02:37.771 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:02:37.771 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:02:37.771 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:02:37.772 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:02:37.772 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:02:37.772 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:02:37.772 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2023-07-02T14:02:37.772 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:02:37.773 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2023-07-02T14:02:37.773 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.773 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.773 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:37.773 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:02:37.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.773 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:02:37.774 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "43c446fe-c811-4ee3-99e8-d7452edfc1c1", 2023-07-02T14:02:37.774 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:37.774 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:37.774 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:37.774 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:37.774 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:37.775 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:37.775 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:37.775 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 0, 2023-07-02T14:02:37.775 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:37.775 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:37.775 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:37.775 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.776 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2023-07-02T14:02:37.776 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:37.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.777 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2023-07-02T14:02:37.777 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:37.777 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.777 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.777 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:37.777 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.778 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.778 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6802", 2023-07-02T14:02:37.778 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:37.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.778 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.778 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6803", 2023-07-02T14:02:37.779 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:37.779 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.779 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.779 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:37.779 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.780 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.780 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6806", 2023-07-02T14:02:37.780 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:37.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.781 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.781 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6807", 2023-07-02T14:02:37.781 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:37.781 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.781 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.781 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:37.782 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.782 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.782 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6804", 2023-07-02T14:02:37.782 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:37.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.783 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.783 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6805", 2023-07-02T14:02:37.783 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:37.783 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.783 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.783 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6801/83359", 2023-07-02T14:02:37.784 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6803/83359", 2023-07-02T14:02:37.784 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6807/83359", 2023-07-02T14:02:37.784 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6805/83359", 2023-07-02T14:02:37.784 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:37.784 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:37.784 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:37.784 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.785 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:02:37.785 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b", 2023-07-02T14:02:37.786 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:37.786 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:37.786 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:37.786 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:37.786 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:37.787 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:37.787 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:37.787 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 25, 2023-07-02T14:02:37.787 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:37.787 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:37.787 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:37.788 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.788 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.788 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6808", 2023-07-02T14:02:37.788 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:37.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.789 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.789 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6809", 2023-07-02T14:02:37.789 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:37.789 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.789 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.790 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:37.790 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.790 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.790 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6810", 2023-07-02T14:02:37.790 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:37.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.791 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.791 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6811", 2023-07-02T14:02:37.791 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:37.791 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.791 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.792 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:37.792 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.792 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.792 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6814", 2023-07-02T14:02:37.792 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:37.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.793 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.793 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6815", 2023-07-02T14:02:37.793 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:37.793 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.793 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.793 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:37.794 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.794 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.794 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6812", 2023-07-02T14:02:37.794 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:37.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.794 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.795 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6813", 2023-07-02T14:02:37.795 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:37.795 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.795 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.795 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6809/83360", 2023-07-02T14:02:37.795 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6811/83360", 2023-07-02T14:02:37.795 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6815/83360", 2023-07-02T14:02:37.796 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6813/83360", 2023-07-02T14:02:37.796 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:37.796 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:37.796 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:37.796 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.796 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:02:37.797 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "99f8ea0a-5a71-4abe-b087-8e891fd11d1b", 2023-07-02T14:02:37.797 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:37.797 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:37.797 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:37.797 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:37.797 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:37.797 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:37.798 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:37.798 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 14, 2023-07-02T14:02:37.798 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:37.798 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:37.798 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:37.798 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.798 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.799 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6816", 2023-07-02T14:02:37.799 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:37.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.799 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.799 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6817", 2023-07-02T14:02:37.799 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:37.799 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.800 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.800 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:37.800 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.800 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.800 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6818", 2023-07-02T14:02:37.800 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:37.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.801 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.801 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6819", 2023-07-02T14:02:37.801 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:37.801 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.801 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.802 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:37.802 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.802 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.802 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6822", 2023-07-02T14:02:37.802 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:37.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.803 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.803 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6823", 2023-07-02T14:02:37.803 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:37.803 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.803 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.803 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:37.804 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.804 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.804 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6820", 2023-07-02T14:02:37.804 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:37.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.805 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.805 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6821", 2023-07-02T14:02:37.805 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:37.805 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.805 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.805 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6817/83361", 2023-07-02T14:02:37.805 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6819/83361", 2023-07-02T14:02:37.806 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6823/83361", 2023-07-02T14:02:37.806 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6821/83361", 2023-07-02T14:02:37.806 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:37.806 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:37.806 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:37.806 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.807 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:02:37.807 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "d285b4d5-0bef-421a-b84a-943e485b10c2", 2023-07-02T14:02:37.807 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:37.807 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:37.807 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:37.807 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:37.807 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:37.807 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:37.808 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:37.808 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 18, 2023-07-02T14:02:37.808 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:37.808 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:37.808 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:37.808 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.808 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.809 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6808", 2023-07-02T14:02:37.809 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:37.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.809 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.809 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6809", 2023-07-02T14:02:37.809 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:37.810 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.810 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.810 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:37.810 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.810 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.810 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6810", 2023-07-02T14:02:37.811 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:37.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.811 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.811 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6811", 2023-07-02T14:02:37.811 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:37.811 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.811 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.812 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:37.812 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.812 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.812 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6814", 2023-07-02T14:02:37.812 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:37.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.813 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.813 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6815", 2023-07-02T14:02:37.813 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:37.813 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.813 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.814 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:37.814 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.814 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.814 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6812", 2023-07-02T14:02:37.814 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:37.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.815 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.815 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6813", 2023-07-02T14:02:37.815 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:37.815 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.815 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.815 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6809/83350", 2023-07-02T14:02:37.815 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6811/83350", 2023-07-02T14:02:37.816 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6815/83350", 2023-07-02T14:02:37.816 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6813/83350", 2023-07-02T14:02:37.816 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:37.816 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:37.816 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:37.816 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.817 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:02:37.817 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "7ba0b0db-e407-4c63-8278-aab09f3082c4", 2023-07-02T14:02:37.817 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:37.817 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:37.817 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:37.817 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 0.23297119140625, 2023-07-02T14:02:37.817 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:37.818 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:37.818 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:37.818 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 0, 2023-07-02T14:02:37.818 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:37.818 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:37.818 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:37.818 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.819 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.819 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6800", 2023-07-02T14:02:37.819 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:37.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.819 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.819 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6801", 2023-07-02T14:02:37.820 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:37.820 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.820 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.820 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:37.820 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.821 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.821 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6802", 2023-07-02T14:02:37.821 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:37.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.821 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.821 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6803", 2023-07-02T14:02:37.821 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:37.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.822 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.822 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:37.822 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.822 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.823 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6806", 2023-07-02T14:02:37.823 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:37.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.823 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.823 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6807", 2023-07-02T14:02:37.823 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:37.823 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.824 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.824 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:37.824 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.824 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.824 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6804", 2023-07-02T14:02:37.824 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:37.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.825 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.825 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6805", 2023-07-02T14:02:37.825 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:37.825 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.825 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.826 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6801/83351", 2023-07-02T14:02:37.826 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6803/83351", 2023-07-02T14:02:37.826 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6807/83351", 2023-07-02T14:02:37.826 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6805/83351", 2023-07-02T14:02:37.826 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:37.826 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:37.826 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:37.826 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:02:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "29663bfa-f6fe-4ce7-b45a-a9b7b4424c42", 2023-07-02T14:02:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:37.828 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:37.828 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:37.828 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:37.828 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:37.828 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 34, 2023-07-02T14:02:37.828 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:37.828 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:37.828 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:37.829 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.829 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.829 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6816", 2023-07-02T14:02:37.829 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:37.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.830 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.830 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6817", 2023-07-02T14:02:37.830 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:37.830 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.830 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.830 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:37.830 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.831 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.831 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6818", 2023-07-02T14:02:37.831 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:37.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.831 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.831 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6819", 2023-07-02T14:02:37.832 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:37.832 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.832 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.832 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:37.832 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.833 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.833 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6822", 2023-07-02T14:02:37.833 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:37.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.833 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.833 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6823", 2023-07-02T14:02:37.833 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:37.834 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.834 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.834 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:37.834 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:37.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.834 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:37.834 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6820", 2023-07-02T14:02:37.835 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:37.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.835 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:37.835 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6821", 2023-07-02T14:02:37.835 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:37.835 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.835 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.836 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6817/83352", 2023-07-02T14:02:37.836 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6819/83352", 2023-07-02T14:02:37.836 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6823/83352", 2023-07-02T14:02:37.836 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6821/83352", 2023-07-02T14:02:37.836 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:37.836 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:37.836 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:37.837 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.837 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.837 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:37.837 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_xinfo": [ 2023-07-02T14:02:37.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.837 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:02:37.837 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:37.837 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:37.838 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:37.838 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:37.838 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:37.838 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.001178+0000", 2023-07-02T14:02:37.838 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:37.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.839 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:02:37.839 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:37.839 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:37.839 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:37.839 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:37.839 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:37.839 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.184181+0000", 2023-07-02T14:02:37.839 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:37.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.840 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:02:37.840 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:37.840 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:37.840 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:37.840 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:37.840 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:37.841 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.274854+0000", 2023-07-02T14:02:37.841 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:37.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.841 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:02:37.841 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:37.841 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:37.841 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:37.842 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:37.842 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:37.842 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.186217+0000", 2023-07-02T14:02:37.842 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:37.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.842 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:02:37.842 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:37.843 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:37.843 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:37.843 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:37.843 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:37.843 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.247442+0000", 2023-07-02T14:02:37.843 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:37.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.844 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:02:37.844 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:37.844 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:37.844 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:37.844 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:37.844 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:37.844 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.220075+0000", 2023-07-02T14:02:37.845 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:37.845 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.845 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:37.845 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap": [ 2023-07-02T14:02:37.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.845 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.2", 2023-07-02T14:02:37.845 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:02:37.845 INFO:teuthology.orchestra.run.smithi016.stdout: 1, 2023-07-02T14:02:37.846 INFO:teuthology.orchestra.run.smithi016.stdout: 5 2023-07-02T14:02:37.846 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.846 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.4", 2023-07-02T14:02:37.846 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:02:37.846 INFO:teuthology.orchestra.run.smithi016.stdout: 5, 2023-07-02T14:02:37.846 INFO:teuthology.orchestra.run.smithi016.stdout: 0 2023-07-02T14:02:37.847 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.847 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.847 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:37.847 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_items": [ 2023-07-02T14:02:37.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.847 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.7", 2023-07-02T14:02:37.847 INFO:teuthology.orchestra.run.smithi016.stdout: "mappings": [ 2023-07-02T14:02:37.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:37.848 INFO:teuthology.orchestra.run.smithi016.stdout: "from": 4, 2023-07-02T14:02:37.848 INFO:teuthology.orchestra.run.smithi016.stdout: "to": 5 2023-07-02T14:02:37.848 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.848 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:37.848 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.848 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:37.848 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_temp": [], 2023-07-02T14:02:37.849 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_temp": [], 2023-07-02T14:02:37.849 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklist": {}, 2023-07-02T14:02:37.849 INFO:teuthology.orchestra.run.smithi016.stdout: "range_blocklist": {}, 2023-07-02T14:02:37.849 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profiles": { 2023-07-02T14:02:37.849 INFO:teuthology.orchestra.run.smithi016.stdout: "default": { 2023-07-02T14:02:37.849 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2023-07-02T14:02:37.849 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "2", 2023-07-02T14:02:37.849 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "1", 2023-07-02T14:02:37.850 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "jerasure", 2023-07-02T14:02:37.850 INFO:teuthology.orchestra.run.smithi016.stdout: "ruleset-failure-domain": "osd", 2023-07-02T14:02:37.850 INFO:teuthology.orchestra.run.smithi016.stdout: "technique": "reed_sol_van" 2023-07-02T14:02:37.850 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:37.850 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps_queue": [], 2023-07-02T14:02:37.850 INFO:teuthology.orchestra.run.smithi016.stdout: "new_removed_snaps": [], 2023-07-02T14:02:37.850 INFO:teuthology.orchestra.run.smithi016.stdout: "new_purged_snaps": [], 2023-07-02T14:02:37.851 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_node_flags": {}, 2023-07-02T14:02:37.851 INFO:teuthology.orchestra.run.smithi016.stdout: "device_class_flags": {}, 2023-07-02T14:02:37.851 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode": { 2023-07-02T14:02:37.851 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_enabled": false, 2023-07-02T14:02:37.851 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_bucket_count": 0, 2023-07-02T14:02:37.851 INFO:teuthology.orchestra.run.smithi016.stdout: "degraded_stretch_mode": 0, 2023-07-02T14:02:37.851 INFO:teuthology.orchestra.run.smithi016.stdout: "recovering_stretch_mode": 0, 2023-07-02T14:02:37.851 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_bucket": 0 2023-07-02T14:02:37.852 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:37.852 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:37.853 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:37.854 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:37.854 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:37.854 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:37.854 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:37.855 INFO:tasks.thrashosds.thrasher:j is {'epoch': 38, 'fsid': '9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2', 'created': '2023-07-02T14:01:37.298060+0000', 'modified': '2023-07-02T14:02:35.418238+0000', 'last_up_change': '2023-07-02T14:01:45.271760+0000', 'last_in_change': '2023-07-02T14:01:39.129121+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-07-02T14:01:45.323455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-02T14:01:47.448897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '43c446fe-c811-4ee3-99e8-d7452edfc1c1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6800', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6801', 'nonce': 83359}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6802', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6803', 'nonce': 83359}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6806', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6807', 'nonce': 83359}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6804', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6805', 'nonce': 83359}]}, 'public_addr': '172.21.15.16:6801/83359', 'cluster_addr': '172.21.15.16:6803/83359', 'heartbeat_back_addr': '172.21.15.16:6807/83359', 'heartbeat_front_addr': '172.21.15.16:6805/83359', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 25, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6808', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6809', 'nonce': 83360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6810', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6811', 'nonce': 83360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6814', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6815', 'nonce': 83360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6812', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6813', 'nonce': 83360}]}, 'public_addr': '172.21.15.16:6809/83360', 'cluster_addr': '172.21.15.16:6811/83360', 'heartbeat_back_addr': '172.21.15.16:6815/83360', 'heartbeat_front_addr': '172.21.15.16:6813/83360', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '99f8ea0a-5a71-4abe-b087-8e891fd11d1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 14, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6816', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6817', 'nonce': 83361}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6818', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6819', 'nonce': 83361}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6822', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6823', 'nonce': 83361}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6820', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6821', 'nonce': 83361}]}, 'public_addr': '172.21.15.16:6817/83361', 'cluster_addr': '172.21.15.16:6819/83361', 'heartbeat_back_addr': '172.21.15.16:6823/83361', 'heartbeat_front_addr': '172.21.15.16:6821/83361', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd285b4d5-0bef-421a-b84a-943e485b10c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6808', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6809', 'nonce': 83350}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6810', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6811', 'nonce': 83350}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6814', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6815', 'nonce': 83350}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6812', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6813', 'nonce': 83350}]}, 'public_addr': '172.21.15.22:6809/83350', 'cluster_addr': '172.21.15.22:6811/83350', 'heartbeat_back_addr': '172.21.15.22:6815/83350', 'heartbeat_front_addr': '172.21.15.22:6813/83350', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7ba0b0db-e407-4c63-8278-aab09f3082c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.23297119140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6800', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6801', 'nonce': 83351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6802', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6803', 'nonce': 83351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6806', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6807', 'nonce': 83351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6804', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6805', 'nonce': 83351}]}, 'public_addr': '172.21.15.22:6801/83351', 'cluster_addr': '172.21.15.22:6803/83351', 'heartbeat_back_addr': '172.21.15.22:6807/83351', 'heartbeat_front_addr': '172.21.15.22:6805/83351', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '29663bfa-f6fe-4ce7-b45a-a9b7b4424c42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6816', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6817', 'nonce': 83352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6818', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6819', 'nonce': 83352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6822', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6823', 'nonce': 83352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6820', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6821', 'nonce': 83352}]}, 'public_addr': '172.21.15.22:6817/83352', 'cluster_addr': '172.21.15.22:6819/83352', 'heartbeat_back_addr': '172.21.15.22:6823/83352', 'heartbeat_front_addr': '172.21.15.22:6821/83352', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.001178+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.184181+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.274854+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.186217+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.247442+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.220075+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.2', 'osds': [1, 5]}, {'pgid': '2.4', 'osds': [5, 0]}], 'pg_upmap_items': [{'pgid': '2.7', 'mappings': [{'from': 4, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-07-02T14:02:37.855 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:02:37.855 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:37.853+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:37.856 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:02:37.954 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:37.953+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:37.979 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:37.980 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:37.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:37.980 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [] 2023-07-02T14:02:37.980 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:37.990 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:02:38.055 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:38.055+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:38.060 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:02:38.155 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:38.155 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:38.155 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:38.155 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:38.156 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:38.155+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:38.165 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:02:38.204 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:02:38.206 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:02:38.218 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2023-07-02T14:02:38.056529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":32144,"kb_used_data":2400,"kb_used_omap":0,"kb_used_meta":29696,"kb_avail":566198896,"statfs":{"total":579820584960,"available":579787669504,"internally_reserved":0,"allocated":2457600,"data_stored":875652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30408704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":37,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-07-02T14:02:29.413318+0000","last_change":"2023-07-02T14:02:01.726288+0000","last_active":"2023-07-02T14:02:29.413318+0000","last_peered":"2023-07-02T14:02:29.413318+0000","last_clean":"2023-07-02T14:02:29.413318+0000","last_became_active":"2023-07-02T14:02:01.725984+0000","last_became_peered":"2023-07-02T14:02:01.725984+0000","last_unstale":"2023-07-02T14:02:29.413318+0000","last_undegraded":"2023-07-02T14:02:29.413318+0000","last_fullsized":"2023-07-02T14:02:29.413318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":30,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-07-02T14:02:29.413433+0000","last_change":"2023-07-02T14:01:49.295075+0000","last_active":"2023-07-02T14:02:29.413433+0000","last_peered":"2023-07-02T14:02:29.413433+0000","last_clean":"2023-07-02T14:02:29.413433+0000","last_became_active":"2023-07-02T14:01:49.293782+0000","last_became_peered":"2023-07-02T14:01:49.293782+0000","last_unstale":"2023-07-02T14:02:29.413433+0000","last_undegraded":"2023-07-02T14:02:29.413433+0000","last_fullsized":"2023-07-02T14:02:29.413433+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":29,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-07-02T14:02:29.413311+0000","last_change":"2023-07-02T14:01:49.293139+0000","last_active":"2023-07-02T14:02:29.413311+0000","last_peered":"2023-07-02T14:02:29.413311+0000","last_clean":"2023-07-02T14:02:29.413311+0000","last_became_active":"2023-07-02T14:01:49.292790+0000","last_became_peered":"2023-07-02T14:01:49.292790+0000","last_unstale":"2023-07-02T14:02:29.413311+0000","last_undegraded":"2023-07-02T14:02:29.413311+0000","last_fullsized":"2023-07-02T14:02:29.413311+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":29,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-07-02T14:02:29.413299+0000","last_change":"2023-07-02T14:01:49.293994+0000","last_active":"2023-07-02T14:02:29.413299+0000","last_peered":"2023-07-02T14:02:29.413299+0000","last_clean":"2023-07-02T14:02:29.413299+0000","last_became_active":"2023-07-02T14:01:49.293608+0000","last_became_peered":"2023-07-02T14:01:49.293608+0000","last_unstale":"2023-07-02T14:02:29.413299+0000","last_undegraded":"2023-07-02T14:02:29.413299+0000","last_fullsized":"2023-07-02T14:02:29.413299+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":31,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-07-02T14:02:29.413298+0000","last_change":"2023-07-02T14:01:47.288970+0000","last_active":"2023-07-02T14:02:29.413298+0000","last_peered":"2023-07-02T14:02:29.413298+0000","last_clean":"2023-07-02T14:02:29.413298+0000","last_became_active":"2023-07-02T14:01:47.288610+0000","last_became_peered":"2023-07-02T14:01:47.288610+0000","last_unstale":"2023-07-02T14:02:29.413298+0000","last_undegraded":"2023-07-02T14:02:29.413298+0000","last_fullsized":"2023-07-02T14:02:29.413298+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":28,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-07-02T14:02:30.328275+0000","last_change":"2023-07-02T14:01:49.298280+0000","last_active":"2023-07-02T14:02:30.328275+0000","last_peered":"2023-07-02T14:02:30.328275+0000","last_clean":"2023-07-02T14:02:30.328275+0000","last_became_active":"2023-07-02T14:01:49.297946+0000","last_became_peered":"2023-07-02T14:01:49.297946+0000","last_unstale":"2023-07-02T14:02:30.328275+0000","last_undegraded":"2023-07-02T14:02:30.328275+0000","last_fullsized":"2023-07-02T14:02:30.328275+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":30,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-07-02T14:02:30.328203+0000","last_change":"2023-07-02T14:01:49.295243+0000","last_active":"2023-07-02T14:02:30.328203+0000","last_peered":"2023-07-02T14:02:30.328203+0000","last_clean":"2023-07-02T14:02:30.328203+0000","last_became_active":"2023-07-02T14:01:49.294964+0000","last_became_peered":"2023-07-02T14:01:49.294964+0000","last_unstale":"2023-07-02T14:02:30.328203+0000","last_undegraded":"2023-07-02T14:02:30.328203+0000","last_fullsized":"2023-07-02T14:02:30.328203+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-07-02T14:02:29.413420+0000","last_change":"2023-07-02T14:02:15.025638+0000","last_active":"2023-07-02T14:02:29.413420+0000","last_peered":"2023-07-02T14:02:29.413420+0000","last_clean":"2023-07-02T14:02:29.413420+0000","last_became_active":"2023-07-02T14:02:15.025376+0000","last_became_peered":"2023-07-02T14:02:15.025376+0000","last_unstale":"2023-07-02T14:02:29.413420+0000","last_undegraded":"2023-07-02T14:02:29.413420+0000","last_fullsized":"2023-07-02T14:02:29.413420+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-07-02T14:02:30.328290+0000","last_change":"2023-07-02T14:02:27.728735+0000","last_active":"2023-07-02T14:02:30.328290+0000","last_peered":"2023-07-02T14:02:30.328290+0000","last_clean":"2023-07-02T14:02:30.328290+0000","last_became_active":"2023-07-02T14:02:27.728488+0000","last_became_peered":"2023-07-02T14:02:27.728488+0000","last_unstale":"2023-07-02T14:02:30.328290+0000","last_undegraded":"2023-07-02T14:02:30.328290+0000","last_fullsized":"2023-07-02T14:02:30.328290+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640268,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5336,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366504,"statfs":{"total":96636764160,"available":96631300096,"internally_reserved":0,"allocated":409600,"data_stored":145942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640268,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5336,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366504,"statfs":{"total":96636764160,"available":96631300096,"internally_reserved":0,"allocated":409600,"data_stored":145942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]}]},{"osd":3,"up_from":11,"seq":47244640268,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5400,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366440,"statfs":{"total":96636764160,"available":96631234560,"internally_reserved":0,"allocated":409600,"data_stored":145942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640268,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5336,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366504,"statfs":{"total":96636764160,"available":96631300096,"internally_reserved":0,"allocated":409600,"data_stored":145942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]}]},{"osd":0,"up_from":11,"seq":47244640268,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5336,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366504,"statfs":{"total":96636764160,"available":96631300096,"internally_reserved":0,"allocated":409600,"data_stored":145942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640268,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5400,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366440,"statfs":{"total":96636764160,"available":96631234560,"internally_reserved":0,"allocated":409600,"data_stored":145942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:02:38.219 INFO:tasks.thrashosds.thrasher:Setting 2.5 to [5, 2, 1, 4] 2023-07-02T14:02:38.219 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.5', '5', '2', '1', '4'] 2023-07-02T14:02:38.219 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.5 5 2 1 4 2023-07-02T14:02:38.256 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:38.255+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:38.288 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:38.288 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:38.289 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:38.289 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:38.289 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:38.298 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:02:38.357 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:38.357+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:38.457 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:38.457+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:38.463 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:38.463 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:38.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:38.464 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [] 2023-07-02T14:02:38.464 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:38.473 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:02:38.558 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:38.558+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:38.638 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:38.639 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:38.639 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:38.639 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:38.649 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:02:38.658 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:38.658+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:38.759 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:38.758+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:38.772 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:38.772 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:38.772 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:38.772 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:38.772 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:38.782 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:02:38.860 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:38.860+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:38.946 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:38.946 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:38.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:38.946 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:02:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4701.0:6 2.2 e39/25 2:5cea7035:::rbd_info:head v 39'1, mlcod=0'0)", 2023-07-02T14:02:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:38.059063+0000", 2023-07-02T14:02:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.887121885, 2023-07-02T14:02:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00095493599999999998, 2023-07-02T14:02:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:38.059063+0000", 2023-07-02T14:02:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:38.059063+0000", 2023-07-02T14:02:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:38.059063+0000", 2023-07-02T14:02:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.614e-06 2023-07-02T14:02:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:38.059068+0000", 2023-07-02T14:02:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7099999999999999e-05 2023-07-02T14:02:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:38.059086+0000", 2023-07-02T14:02:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6169999999999996e-06 2023-07-02T14:02:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:38.059095+0000", 2023-07-02T14:02:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0612999999999998e-05 2023-07-02T14:02:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:38.059146+0000", 2023-07-02T14:02:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0619999999999999e-05 2023-07-02T14:02:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:38.059166+0000", 2023-07-02T14:02:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00082311099999999996 2023-07-02T14:02:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:38.059990+0000", 2023-07-02T14:02:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8504999999999999e-05 2023-07-02T14:02:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:38.060018+0000", 2023-07-02T14:02:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00095493599999999998 2023-07-02T14:02:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:38.955 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:38.955 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:38.955 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:38.956 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:02:39.000 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:39.000+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:39.062 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:39.061+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:39.064 INFO:teuthology.orchestra.run.smithi016.stderr:set 2.5 pg_upmap_items mapping to [5->2,1->4] 2023-07-02T14:02:39.077 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd create -s 2G --object-size 512K b3fbb028-e267-4d25-a750-93ba18a2316a-dst 2023-07-02T14:02:39.087 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:39.087 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:39.087 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:39.088 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:39.098 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:02:39.120 INFO:tasks.workunit.client.0.smithi163.stderr:++ sudo rbd device map -t nbd b3fbb028-e267-4d25-a750-93ba18a2316a-src 2023-07-02T14:02:39.181 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:39.181+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:39.237 INFO:tasks.workunit.client.0.smithi163.stderr:+ dev=/dev/nbd0 2023-07-02T14:02:39.237 INFO:tasks.workunit.client.0.smithi163.stderr:+ sudo mkfs.ext4 /dev/nbd0 2023-07-02T14:02:39.261 INFO:tasks.workunit.client.0.smithi163.stderr:mke2fs 1.45.6 (20-Mar-2020) 2023-07-02T14:02:39.262 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:39.262+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:39.265 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:39.265 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:39.265 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:39.265 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:39.265 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:39.270 INFO:tasks.workunit.client.0.smithi163.stdout:Discarding device blocks: 4096/524288 done 2023-07-02T14:02:39.271 INFO:tasks.workunit.client.0.smithi163.stdout:Creating filesystem with 524288 4k blocks and 131072 inodes 2023-07-02T14:02:39.271 INFO:tasks.workunit.client.0.smithi163.stdout:Filesystem UUID: 287ff972-3e1a-4389-a8d4-1e417f52f6ab 2023-07-02T14:02:39.271 INFO:tasks.workunit.client.0.smithi163.stdout:Superblock backups stored on blocks: 2023-07-02T14:02:39.271 INFO:tasks.workunit.client.0.smithi163.stdout: 32768, 98304, 163840, 229376, 294912 2023-07-02T14:02:39.271 INFO:tasks.workunit.client.0.smithi163.stdout: 2023-07-02T14:02:39.271 INFO:tasks.workunit.client.0.smithi163.stdout:Allocating group tables: 0/16 done 2023-07-02T14:02:39.271 INFO:tasks.workunit.client.0.smithi163.stdout:Writing inode tables: 0/16 done 2023-07-02T14:02:39.276 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:02:39.294 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:02:39.363 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:39.363+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:39.374 INFO:tasks.workunit.client.0.smithi163.stdout:Creating journal (16384 blocks): done 2023-07-02T14:02:39.420 INFO:tasks.workunit.client.0.smithi163.stdout:Writing superblocks and filesystem accounting information: 0/16 done 2023-07-02T14:02:39.420 INFO:tasks.workunit.client.0.smithi163.stdout: 2023-07-02T14:02:39.421 INFO:tasks.workunit.client.0.smithi163.stderr:+ mkdir b3fbb028-e267-4d25-a750-93ba18a2316a-mnt 2023-07-02T14:02:39.423 INFO:tasks.workunit.client.0.smithi163.stderr:+ sudo mount /dev/nbd0 b3fbb028-e267-4d25-a750-93ba18a2316a-mnt 2023-07-02T14:02:39.452 INFO:tasks.workunit.client.0.smithi163.stderr:++ whoami 2023-07-02T14:02:39.452 INFO:tasks.workunit.client.0.smithi163.stderr:+ sudo chown ubuntu b3fbb028-e267-4d25-a750-93ba18a2316a-mnt 2023-07-02T14:02:39.463 INFO:tasks.workunit.client.0.smithi163.stderr:+ wget https://download.ceph.com/qa/linux-5.4.tar.gz 2023-07-02T14:02:39.464 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:39.464+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:39.485 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:02:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.485 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:380 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'10, mlcod=40'1)", 2023-07-02T14:02:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.309796+0000", 2023-07-02T14:02:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.173537845, 2023-07-02T14:02:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034177320000000001, 2023-07-02T14:02:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:39.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309796+0000", 2023-07-02T14:02:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:39.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309796+0000", 2023-07-02T14:02:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309796+0000", 2023-07-02T14:02:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4694999999999997e-05 2023-07-02T14:02:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309851+0000", 2023-07-02T14:02:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6890000000000004e-06 2023-07-02T14:02:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309858+0000", 2023-07-02T14:02:39.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7659999999999998e-06 2023-07-02T14:02:39.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:39.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309863+0000", 2023-07-02T14:02:39.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001070164 2023-07-02T14:02:39.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310933+0000", 2023-07-02T14:02:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1287000000000001e-05 2023-07-02T14:02:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:39.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310954+0000", 2023-07-02T14:02:39.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022261059999999998 2023-07-02T14:02:39.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:39.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313181+0000", 2023-07-02T14:02:39.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3494000000000002e-05 2023-07-02T14:02:39.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:39.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313214+0000", 2023-07-02T14:02:39.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034177320000000001 2023-07-02T14:02:39.493 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.493 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.493 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.494 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:381 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'11, mlcod=40'1)", 2023-07-02T14:02:39.494 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.310792+0000", 2023-07-02T14:02:39.494 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.17254169699999999, 2023-07-02T14:02:39.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038397629999999999, 2023-07-02T14:02:39.494 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:39.494 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:39.495 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:39.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:39.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310792+0000", 2023-07-02T14:02:39.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:39.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:39.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310792+0000", 2023-07-02T14:02:39.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:39.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:39.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310792+0000", 2023-07-02T14:02:39.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9630999999999999e-05 2023-07-02T14:02:39.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:39.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310851+0000", 2023-07-02T14:02:39.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3351e-05 2023-07-02T14:02:39.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310865+0000", 2023-07-02T14:02:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7459999999999995e-06 2023-07-02T14:02:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310874+0000", 2023-07-02T14:02:39.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000478457 2023-07-02T14:02:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311353+0000", 2023-07-02T14:02:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3498000000000001e-05 2023-07-02T14:02:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311376+0000", 2023-07-02T14:02:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003216464 2023-07-02T14:02:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314593+0000", 2023-07-02T14:02:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9368e-05 2023-07-02T14:02:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314632+0000", 2023-07-02T14:02:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038397629999999999 2023-07-02T14:02:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.501 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:382 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'12, mlcod=40'3)", 2023-07-02T14:02:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311736+0000", 2023-07-02T14:02:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.171598005, 2023-07-02T14:02:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029255800000000001, 2023-07-02T14:02:39.502 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311736+0000", 2023-07-02T14:02:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311736+0000", 2023-07-02T14:02:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:39.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311735+0000", 2023-07-02T14:02:39.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9895000000000005e-05 2023-07-02T14:02:39.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:39.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311835+0000", 2023-07-02T14:02:39.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2548000000000001e-05 2023-07-02T14:02:39.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:39.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311848+0000", 2023-07-02T14:02:39.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5810000000000002e-06 2023-07-02T14:02:39.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:39.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311854+0000", 2023-07-02T14:02:39.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4743999999999997e-05 2023-07-02T14:02:39.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:39.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311899+0000", 2023-07-02T14:02:39.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.808e-05 2023-07-02T14:02:39.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311917+0000", 2023-07-02T14:02:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027195069999999999 2023-07-02T14:02:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:39.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314637+0000", 2023-07-02T14:02:39.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5080999999999999e-05 2023-07-02T14:02:39.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:39.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314662+0000", 2023-07-02T14:02:39.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029255800000000001 2023-07-02T14:02:39.509 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.509 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:386 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'16, mlcod=40'10)", 2023-07-02T14:02:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.317468+0000", 2023-07-02T14:02:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.16586583699999999, 2023-07-02T14:02:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027835989999999999, 2023-07-02T14:02:39.510 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317468+0000", 2023-07-02T14:02:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:39.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:39.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317468+0000", 2023-07-02T14:02:39.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:39.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317468+0000", 2023-07-02T14:02:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8161999999999998e-05 2023-07-02T14:02:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317506+0000", 2023-07-02T14:02:39.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5590000000000002e-06 2023-07-02T14:02:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317512+0000", 2023-07-02T14:02:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.186e-06 2023-07-02T14:02:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317518+0000", 2023-07-02T14:02:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4561999999999996e-05 2023-07-02T14:02:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317582+0000", 2023-07-02T14:02:39.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1291e-05 2023-07-02T14:02:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317603+0000", 2023-07-02T14:02:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026133839999999998 2023-07-02T14:02:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320217+0000", 2023-07-02T14:02:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5111999999999999e-05 2023-07-02T14:02:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320252+0000", 2023-07-02T14:02:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027835989999999999 2023-07-02T14:02:39.517 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:387 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'17, mlcod=40'10)", 2023-07-02T14:02:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.317892+0000", 2023-07-02T14:02:39.518 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.16544244299999999, 2023-07-02T14:02:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031330540000000001, 2023-07-02T14:02:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:39.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317892+0000", 2023-07-02T14:02:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317892+0000", 2023-07-02T14:02:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317891+0000", 2023-07-02T14:02:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1758e-05 2023-07-02T14:02:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317963+0000", 2023-07-02T14:02:39.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5237e-05 2023-07-02T14:02:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317978+0000", 2023-07-02T14:02:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9710000000000001e-06 2023-07-02T14:02:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317984+0000", 2023-07-02T14:02:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4039999999999998e-05 2023-07-02T14:02:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318028+0000", 2023-07-02T14:02:39.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8092000000000001e-05 2023-07-02T14:02:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318056+0000", 2023-07-02T14:02:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029412959999999999 2023-07-02T14:02:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320997+0000", 2023-07-02T14:02:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7416999999999999e-05 2023-07-02T14:02:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321025+0000", 2023-07-02T14:02:39.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031330540000000001 2023-07-02T14:02:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:402 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'32, mlcod=40'17)", 2023-07-02T14:02:39.526 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.331582+0000", 2023-07-02T14:02:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.151752576, 2023-07-02T14:02:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002927172, 2023-07-02T14:02:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:39.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331582+0000", 2023-07-02T14:02:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331582+0000", 2023-07-02T14:02:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331581+0000", 2023-07-02T14:02:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2337999999999998e-05 2023-07-02T14:02:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:39.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331643+0000", 2023-07-02T14:02:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0506000000000001e-05 2023-07-02T14:02:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331653+0000", 2023-07-02T14:02:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5650000000000003e-06 2023-07-02T14:02:39.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331658+0000", 2023-07-02T14:02:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3337999999999998e-05 2023-07-02T14:02:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:39.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331691+0000", 2023-07-02T14:02:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4527e-05 2023-07-02T14:02:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331706+0000", 2023-07-02T14:02:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027586830000000001 2023-07-02T14:02:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334464+0000", 2023-07-02T14:02:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4311000000000003e-05 2023-07-02T14:02:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334509+0000", 2023-07-02T14:02:39.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002927172 2023-07-02T14:02:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:403 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'33, mlcod=40'20)", 2023-07-02T14:02:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.331850+0000", 2023-07-02T14:02:39.534 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.15148447100000001, 2023-07-02T14:02:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033780419999999999, 2023-07-02T14:02:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:39.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331850+0000", 2023-07-02T14:02:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331850+0000", 2023-07-02T14:02:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331849+0000", 2023-07-02T14:02:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015039599999999999 2023-07-02T14:02:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:39.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331999+0000", 2023-07-02T14:02:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7082e-05 2023-07-02T14:02:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.332017+0000", 2023-07-02T14:02:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3400000000000003e-06 2023-07-02T14:02:39.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.332023+0000", 2023-07-02T14:02:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8889999999999999e-05 2023-07-02T14:02:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:39.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.332102+0000", 2023-07-02T14:02:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2781000000000003e-05 2023-07-02T14:02:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.332145+0000", 2023-07-02T14:02:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030463410000000002 2023-07-02T14:02:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.335191+0000", 2023-07-02T14:02:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.684e-05 2023-07-02T14:02:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.335228+0000", 2023-07-02T14:02:39.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033780419999999999 2023-07-02T14:02:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:405 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'35, mlcod=40'25)", 2023-07-02T14:02:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.333577+0000", 2023-07-02T14:02:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.14975730300000001, 2023-07-02T14:02:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002930219, 2023-07-02T14:02:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:39.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333577+0000", 2023-07-02T14:02:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333577+0000", 2023-07-02T14:02:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333576+0000", 2023-07-02T14:02:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4240000000000002e-05 2023-07-02T14:02:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333630+0000", 2023-07-02T14:02:39.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2962999999999999e-05 2023-07-02T14:02:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333643+0000", 2023-07-02T14:02:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8390000000000001e-06 2023-07-02T14:02:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333650+0000", 2023-07-02T14:02:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0926000000000005e-05 2023-07-02T14:02:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333721+0000", 2023-07-02T14:02:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2921e-05 2023-07-02T14:02:39.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333744+0000", 2023-07-02T14:02:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027376240000000001 2023-07-02T14:02:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336481+0000", 2023-07-02T14:02:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.565e-05 2023-07-02T14:02:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336507+0000", 2023-07-02T14:02:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002930219 2023-07-02T14:02:39.549 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:406 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'36, mlcod=40'25)", 2023-07-02T14:02:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.334125+0000", 2023-07-02T14:02:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.149209539, 2023-07-02T14:02:39.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003294352, 2023-07-02T14:02:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334125+0000", 2023-07-02T14:02:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334125+0000", 2023-07-02T14:02:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334124+0000", 2023-07-02T14:02:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106912 2023-07-02T14:02:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334231+0000", 2023-07-02T14:02:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6602000000000001e-05 2023-07-02T14:02:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334247+0000", 2023-07-02T14:02:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7350000000000001e-06 2023-07-02T14:02:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334253+0000", 2023-07-02T14:02:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000473144 2023-07-02T14:02:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334726+0000", 2023-07-02T14:02:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4657000000000001e-05 2023-07-02T14:02:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334751+0000", 2023-07-02T14:02:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026279250000000001 2023-07-02T14:02:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337379+0000", 2023-07-02T14:02:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0111999999999998e-05 2023-07-02T14:02:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337419+0000", 2023-07-02T14:02:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003294352 2023-07-02T14:02:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:426 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'42, mlcod=40'38)", 2023-07-02T14:02:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.340864+0000", 2023-07-02T14:02:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.14247021800000001, 2023-07-02T14:02:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065109469999999996, 2023-07-02T14:02:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.340864+0000", 2023-07-02T14:02:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.340864+0000", 2023-07-02T14:02:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.340863+0000", 2023-07-02T14:02:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019354159999999999 2023-07-02T14:02:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342799+0000", 2023-07-02T14:02:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5824e-05 2023-07-02T14:02:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342814+0000", 2023-07-02T14:02:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:02:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342822+0000", 2023-07-02T14:02:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7465999999999994e-05 2023-07-02T14:02:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342889+0000", 2023-07-02T14:02:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5068000000000001e-05 2023-07-02T14:02:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342914+0000", 2023-07-02T14:02:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044200969999999996 2023-07-02T14:02:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347334+0000", 2023-07-02T14:02:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0711999999999999e-05 2023-07-02T14:02:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347375+0000", 2023-07-02T14:02:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065109469999999996 2023-07-02T14:02:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:431 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'4, mlcod=40'2)", 2023-07-02T14:02:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.350267+0000", 2023-07-02T14:02:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.13306675200000001, 2023-07-02T14:02:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077388420000000001, 2023-07-02T14:02:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350267+0000", 2023-07-02T14:02:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350267+0000", 2023-07-02T14:02:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350266+0000", 2023-07-02T14:02:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024191569999999999 2023-07-02T14:02:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352685+0000", 2023-07-02T14:02:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0013e-05 2023-07-02T14:02:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352706+0000", 2023-07-02T14:02:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3250000000000008e-06 2023-07-02T14:02:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352714+0000", 2023-07-02T14:02:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9648000000000003e-05 2023-07-02T14:02:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352763+0000", 2023-07-02T14:02:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4496999999999999e-05 2023-07-02T14:02:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352788+0000", 2023-07-02T14:02:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051724880000000003 2023-07-02T14:02:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357960+0000", 2023-07-02T14:02:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5802e-05 2023-07-02T14:02:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358006+0000", 2023-07-02T14:02:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077388420000000001 2023-07-02T14:02:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:433 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'5, mlcod=40'2)", 2023-07-02T14:02:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.352729+0000", 2023-07-02T14:02:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.13060474799999999, 2023-07-02T14:02:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0053048619999999996, 2023-07-02T14:02:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352729+0000", 2023-07-02T14:02:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352729+0000", 2023-07-02T14:02:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352728+0000", 2023-07-02T14:02:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4173e-05 2023-07-02T14:02:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352763+0000", 2023-07-02T14:02:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7689999999999996e-06 2023-07-02T14:02:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352770+0000", 2023-07-02T14:02:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6689999999999997e-06 2023-07-02T14:02:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352775+0000", 2023-07-02T14:02:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025486380000000002 2023-07-02T14:02:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.355324+0000", 2023-07-02T14:02:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7258999999999999e-05 2023-07-02T14:02:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.355341+0000", 2023-07-02T14:02:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026703569999999999 2023-07-02T14:02:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358011+0000", 2023-07-02T14:02:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3039000000000001e-05 2023-07-02T14:02:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358034+0000", 2023-07-02T14:02:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0053048619999999996 2023-07-02T14:02:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:439 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'6, mlcod=40'4)", 2023-07-02T14:02:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.359868+0000", 2023-07-02T14:02:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.12346644599999999, 2023-07-02T14:02:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031584299999999998, 2023-07-02T14:02:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.359868+0000", 2023-07-02T14:02:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.359868+0000", 2023-07-02T14:02:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:02:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.359866+0000", 2023-07-02T14:02:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00068265100000000005 2023-07-02T14:02:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.360549+0000", 2023-07-02T14:02:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9236999999999999e-05 2023-07-02T14:02:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.360568+0000", 2023-07-02T14:02:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5529999999999998e-06 2023-07-02T14:02:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.360578+0000", 2023-07-02T14:02:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6258000000000003e-05 2023-07-02T14:02:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.360624+0000", 2023-07-02T14:02:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6570000000000001e-05 2023-07-02T14:02:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.360651+0000", 2023-07-02T14:02:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023451359999999998 2023-07-02T14:02:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.362996+0000", 2023-07-02T14:02:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0467000000000002e-05 2023-07-02T14:02:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363026+0000", 2023-07-02T14:02:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031584299999999998 2023-07-02T14:02:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:440 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'7, mlcod=40'4)", 2023-07-02T14:02:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.361026+0000", 2023-07-02T14:02:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.122307976, 2023-07-02T14:02:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029947889999999999, 2023-07-02T14:02:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361026+0000", 2023-07-02T14:02:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361026+0000", 2023-07-02T14:02:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361025+0000", 2023-07-02T14:02:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013836740000000001 2023-07-02T14:02:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.362409+0000", 2023-07-02T14:02:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5155e-05 2023-07-02T14:02:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.362424+0000", 2023-07-02T14:02:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1170000000000004e-06 2023-07-02T14:02:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.362432+0000", 2023-07-02T14:02:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9370999999999999e-05 2023-07-02T14:02:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.362482+0000", 2023-07-02T14:02:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5891999999999999e-05 2023-07-02T14:02:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.362508+0000", 2023-07-02T14:02:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014803920000000001 2023-07-02T14:02:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363988+0000", 2023-07-02T14:02:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2923999999999999e-05 2023-07-02T14:02:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364021+0000", 2023-07-02T14:02:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029947889999999999 2023-07-02T14:02:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:451 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'8, mlcod=40'6)", 2023-07-02T14:02:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.365605+0000", 2023-07-02T14:02:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.11772901199999999, 2023-07-02T14:02:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033622349999999999, 2023-07-02T14:02:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365605+0000", 2023-07-02T14:02:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365605+0000", 2023-07-02T14:02:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:02:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365604+0000", 2023-07-02T14:02:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00056647700000000002 2023-07-02T14:02:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.366170+0000", 2023-07-02T14:02:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2206000000000001e-05 2023-07-02T14:02:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.366192+0000", 2023-07-02T14:02:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1044e-05 2023-07-02T14:02:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.366203+0000", 2023-07-02T14:02:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2663000000000005e-05 2023-07-02T14:02:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.366276+0000", 2023-07-02T14:02:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7101999999999999e-05 2023-07-02T14:02:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.366303+0000", 2023-07-02T14:02:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026248769999999999 2023-07-02T14:02:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368928+0000", 2023-07-02T14:02:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9267000000000002e-05 2023-07-02T14:02:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368967+0000", 2023-07-02T14:02:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033622349999999999 2023-07-02T14:02:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:455 2.5 e40/14 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:head v 40'12, mlcod=40'10)", 2023-07-02T14:02:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.376227+0000", 2023-07-02T14:02:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.10710747399999999, 2023-07-02T14:02:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012681951, 2023-07-02T14:02:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.376227+0000", 2023-07-02T14:02:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.376227+0000", 2023-07-02T14:02:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.376226+0000", 2023-07-02T14:02:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040198860000000003 2023-07-02T14:02:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380245+0000", 2023-07-02T14:02:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7640000000000001e-05 2023-07-02T14:02:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380263+0000", 2023-07-02T14:02:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3299999999999999e-06 2023-07-02T14:02:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380271+0000", 2023-07-02T14:02:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4322e-05 2023-07-02T14:02:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380316+0000", 2023-07-02T14:02:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7682000000000001e-05 2023-07-02T14:02:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380333+0000", 2023-07-02T14:02:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085223439999999994 2023-07-02T14:02:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.388856+0000", 2023-07-02T14:02:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2930999999999999e-05 2023-07-02T14:02:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.388909+0000", 2023-07-02T14:02:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012681951 2023-07-02T14:02:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:456 2.5 e40/14 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:head v 40'13, mlcod=40'10)", 2023-07-02T14:02:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.382266+0000", 2023-07-02T14:02:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.101067847, 2023-07-02T14:02:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075568149999999997, 2023-07-02T14:02:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382266+0000", 2023-07-02T14:02:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382266+0000", 2023-07-02T14:02:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:02:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382264+0000", 2023-07-02T14:02:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012720599999999999 2023-07-02T14:02:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382391+0000", 2023-07-02T14:02:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7598e-05 2023-07-02T14:02:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382409+0000", 2023-07-02T14:02:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8660000000000001e-06 2023-07-02T14:02:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382419+0000", 2023-07-02T14:02:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001837403 2023-07-02T14:02:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384256+0000", 2023-07-02T14:02:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5757000000000001e-05 2023-07-02T14:02:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384282+0000", 2023-07-02T14:02:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055044730000000002 2023-07-02T14:02:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389787+0000", 2023-07-02T14:02:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6622000000000003e-05 2023-07-02T14:02:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389823+0000", 2023-07-02T14:02:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075568149999999997 2023-07-02T14:02:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:457 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'14, mlcod=40'10)", 2023-07-02T14:02:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.382901+0000", 2023-07-02T14:02:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.100432875, 2023-07-02T14:02:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0069428320000000003, 2023-07-02T14:02:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382901+0000", 2023-07-02T14:02:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382901+0000", 2023-07-02T14:02:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:02:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382900+0000", 2023-07-02T14:02:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7738999999999994e-05 2023-07-02T14:02:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382998+0000", 2023-07-02T14:02:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6769000000000002e-05 2023-07-02T14:02:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383014+0000", 2023-07-02T14:02:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0073e-05 2023-07-02T14:02:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383025+0000", 2023-07-02T14:02:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016687410000000001 2023-07-02T14:02:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384693+0000", 2023-07-02T14:02:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4108999999999999e-05 2023-07-02T14:02:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384717+0000", 2023-07-02T14:02:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051086830000000001 2023-07-02T14:02:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389826+0000", 2023-07-02T14:02:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8059000000000001e-05 2023-07-02T14:02:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389844+0000", 2023-07-02T14:02:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0069428320000000003 2023-07-02T14:02:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:479 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'15, mlcod=40'10)", 2023-07-02T14:02:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.383348+0000", 2023-07-02T14:02:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.099985916999999994, 2023-07-02T14:02:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065068890000000001, 2023-07-02T14:02:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383348+0000", 2023-07-02T14:02:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383348+0000", 2023-07-02T14:02:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383347+0000", 2023-07-02T14:02:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000422123 2023-07-02T14:02:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383769+0000", 2023-07-02T14:02:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1688e-05 2023-07-02T14:02:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383791+0000", 2023-07-02T14:02:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0237e-05 2023-07-02T14:02:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383801+0000", 2023-07-02T14:02:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013799940000000001 2023-07-02T14:02:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.385181+0000", 2023-07-02T14:02:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3047000000000001e-05 2023-07-02T14:02:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.385204+0000", 2023-07-02T14:02:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046417339999999998 2023-07-02T14:02:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389846+0000", 2023-07-02T14:02:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1800000000000002e-06 2023-07-02T14:02:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389855+0000", 2023-07-02T14:02:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065068890000000001 2023-07-02T14:02:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:484 2.5 e40/14 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:head v 40'16, mlcod=40'10)", 2023-07-02T14:02:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.384371+0000", 2023-07-02T14:02:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.098962854000000003, 2023-07-02T14:02:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054937609999999998, 2023-07-02T14:02:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384371+0000", 2023-07-02T14:02:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384371+0000", 2023-07-02T14:02:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384370+0000", 2023-07-02T14:02:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00041928199999999999 2023-07-02T14:02:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384790+0000", 2023-07-02T14:02:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1742999999999999e-05 2023-07-02T14:02:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384801+0000", 2023-07-02T14:02:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0380000000000002e-06 2023-07-02T14:02:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384808+0000", 2023-07-02T14:02:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010660999999999999 2023-07-02T14:02:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.385874+0000", 2023-07-02T14:02:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3745000000000003e-05 2023-07-02T14:02:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.385908+0000", 2023-07-02T14:02:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039487560000000003 2023-07-02T14:02:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389857+0000", 2023-07-02T14:02:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1019999999999997e-06 2023-07-02T14:02:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389865+0000", 2023-07-02T14:02:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054937609999999998 2023-07-02T14:02:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:39.646 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:39.647 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:02:39.648 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:39.648 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:39.648 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:39.648 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:39.648 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:39.649 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:39.649 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:02:39.649 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:02:39.650 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:02:39.650 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:02:39.650 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:02:39.650 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:02:39.650 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:39.646+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:39.687 INFO:tasks.workunit.client.0.smithi163.stderr:--2023-07-02 14:02:39-- https://download.ceph.com/qa/linux-5.4.tar.gz 2023-07-02T14:02:39.727 INFO:tasks.workunit.client.0.smithi163.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2023-07-02T14:02:39.747 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:39.747+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:39.771 INFO:tasks.workunit.client.0.smithi163.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:443... connected. 2023-07-02T14:02:39.811 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:39.811 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:39.821 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:02:39.862 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:39.862+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:39.888 INFO:tasks.workunit.client.0.smithi163.stderr:HTTP request sent, awaiting response... 200 OK 2023-07-02T14:02:39.888 INFO:tasks.workunit.client.0.smithi163.stderr:Length: 172616875 (165M) [application/octet-stream] 2023-07-02T14:02:39.889 INFO:tasks.workunit.client.0.smithi163.stderr:Saving to: ‘linux-5.4.tar.gz’ 2023-07-02T14:02:39.889 INFO:tasks.workunit.client.0.smithi163.stderr: 2023-07-02T14:02:39.909 INFO:tasks.workunit.client.0.smithi163.stderr: 0K .......... .......... .......... .......... .......... 0% 2.39M 69s 2023-07-02T14:02:39.930 INFO:tasks.workunit.client.0.smithi163.stderr: 50K .......... .......... .......... .......... .......... 0% 2.40M 69s 2023-07-02T14:02:39.930 INFO:tasks.workunit.client.0.smithi163.stderr: 100K .......... .......... .......... .......... .......... 0% 138M 46s 2023-07-02T14:02:39.948 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:39.948+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:39.951 INFO:tasks.workunit.client.0.smithi163.stderr: 150K .......... .......... .......... .......... .......... 0% 2.43M 52s 2023-07-02T14:02:39.952 INFO:tasks.workunit.client.0.smithi163.stderr: 200K .......... .......... .......... .......... .......... 0% 322M 41s 2023-07-02T14:02:39.952 INFO:tasks.workunit.client.0.smithi163.stderr: 250K .......... .......... .......... .......... .......... 0% 374M 35s 2023-07-02T14:02:39.952 INFO:tasks.workunit.client.0.smithi163.stderr: 300K .......... .......... .......... .......... .......... 0% 79.4M 30s 2023-07-02T14:02:39.972 INFO:tasks.workunit.client.0.smithi163.stderr: 350K .......... .......... .......... .......... .......... 0% 2.24M 35s 2023-07-02T14:02:39.973 INFO:tasks.workunit.client.0.smithi163.stderr: 400K .......... .......... .......... .......... .......... 0% 55.7M 32s 2023-07-02T14:02:39.975 INFO:tasks.workunit.client.0.smithi163.stderr: 450K .......... .......... .......... .......... .......... 0% 22.0M 29s 2023-07-02T14:02:39.976 INFO:tasks.workunit.client.0.smithi163.stderr: 500K .......... .......... .......... .......... .......... 0% 45.5M 27s 2023-07-02T14:02:39.978 INFO:tasks.workunit.client.0.smithi163.stderr: 550K .......... .......... .......... .......... .......... 0% 22.3M 25s 2023-07-02T14:02:39.979 INFO:tasks.workunit.client.0.smithi163.stderr: 600K .......... .......... .......... .......... .......... 0% 39.2M 24s 2023-07-02T14:02:39.982 INFO:tasks.workunit.client.0.smithi163.stderr: 650K .......... .......... .......... .......... .......... 0% 16.8M 23s 2023-07-02T14:02:39.985 INFO:tasks.workunit.client.0.smithi163.stderr: 700K .......... .......... .......... .......... .......... 0% 22.0M 22s 2023-07-02T14:02:39.990 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:39.990 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:40.000 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:02:40.007 INFO:tasks.workunit.client.0.smithi163.stderr: 750K .......... .......... .......... .......... .......... 0% 2.22M 25s 2023-07-02T14:02:40.008 INFO:tasks.workunit.client.0.smithi163.stderr: 800K .......... .......... .......... .......... .......... 0% 67.4M 24s 2023-07-02T14:02:40.009 INFO:tasks.workunit.client.0.smithi163.stderr: 850K .......... .......... .......... .......... .......... 0% 29.0M 23s 2023-07-02T14:02:40.011 INFO:tasks.workunit.client.0.smithi163.stderr: 900K .......... .......... .......... .......... .......... 0% 32.0M 22s 2023-07-02T14:02:40.013 INFO:tasks.workunit.client.0.smithi163.stderr: 950K .......... .......... .......... .......... .......... 0% 16.9M 21s 2023-07-02T14:02:40.014 INFO:tasks.workunit.client.0.smithi163.stderr: 1000K .......... .......... .......... .......... .......... 0% 40.0M 20s 2023-07-02T14:02:40.018 INFO:tasks.workunit.client.0.smithi163.stderr: 1050K .......... .......... .......... .......... .......... 0% 15.7M 20s 2023-07-02T14:02:40.020 INFO:tasks.workunit.client.0.smithi163.stderr: 1100K .......... .......... .......... .......... .......... 0% 21.5M 19s 2023-07-02T14:02:40.024 INFO:tasks.workunit.client.0.smithi163.stderr: 1150K .......... .......... .......... .......... .......... 0% 13.1M 19s 2023-07-02T14:02:40.026 INFO:tasks.workunit.client.0.smithi163.stderr: 1200K .......... .......... .......... .......... .......... 0% 20.8M 18s 2023-07-02T14:02:40.029 INFO:tasks.workunit.client.0.smithi163.stderr: 1250K .......... .......... .......... .......... .......... 0% 16.2M 18s 2023-07-02T14:02:40.030 INFO:tasks.workunit.client.0.smithi163.stderr: 1300K .......... .......... .......... .......... .......... 0% 45.9M 18s 2023-07-02T14:02:40.032 INFO:tasks.workunit.client.0.smithi163.stderr: 1350K .......... .......... .......... .......... .......... 0% 21.5M 17s 2023-07-02T14:02:40.034 INFO:tasks.workunit.client.0.smithi163.stderr: 1400K .......... .......... .......... .......... .......... 0% 40.6M 17s 2023-07-02T14:02:40.037 INFO:tasks.workunit.client.0.smithi163.stderr: 1450K .......... .......... .......... .......... .......... 0% 16.8M 17s 2023-07-02T14:02:40.049 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:40.049+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:40.051 INFO:tasks.workunit.client.0.smithi163.stderr: 1500K .......... .......... .......... .......... .......... 0% 3.42M 18s 2023-07-02T14:02:40.059 INFO:tasks.workunit.client.0.smithi163.stderr: 1550K .......... .......... .......... .......... .......... 0% 6.06M 18s 2023-07-02T14:02:40.062 INFO:tasks.workunit.client.0.smithi163.stderr: 1600K .......... .......... .......... .......... .......... 0% 18.7M 18s 2023-07-02T14:02:40.065 INFO:tasks.workunit.client.0.smithi163.stderr: 1650K .......... .......... .......... .......... .......... 1% 15.6M 17s 2023-07-02T14:02:40.068 INFO:tasks.workunit.client.0.smithi163.stderr: 1700K .......... .......... .......... .......... .......... 1% 13.8M 17s 2023-07-02T14:02:40.070 INFO:tasks.workunit.client.0.smithi163.stderr: 1750K .......... .......... .......... .......... .......... 1% 23.5M 17s 2023-07-02T14:02:40.074 INFO:tasks.workunit.client.0.smithi163.stderr: 1800K .......... .......... .......... .......... .......... 1% 11.2M 17s 2023-07-02T14:02:40.077 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:02:40.078 INFO:tasks.workunit.client.0.smithi163.stderr: 1850K .......... .......... .......... .......... .......... 1% 14.1M 17s 2023-07-02T14:02:40.083 INFO:tasks.workunit.client.0.smithi163.stderr: 1900K .......... .......... .......... .......... .......... 1% 9.93M 17s 2023-07-02T14:02:40.087 INFO:tasks.workunit.client.0.smithi163.stderr: 1950K .......... .......... .......... .......... .......... 1% 13.7M 17s 2023-07-02T14:02:40.088 INFO:tasks.workunit.client.0.smithi163.stderr: 2000K .......... .......... .......... .......... .......... 1% 33.8M 16s 2023-07-02T14:02:40.089 INFO:tasks.workunit.client.0.smithi163.stderr: 2050K .......... .......... .......... .......... .......... 1% 35.9M 16s 2023-07-02T14:02:40.092 INFO:tasks.workunit.client.0.smithi163.stderr: 2100K .......... .......... .......... .......... .......... 1% 17.9M 16s 2023-07-02T14:02:40.093 INFO:tasks.workunit.client.0.smithi163.stderr: 2150K .......... .......... .......... .......... .......... 1% 34.2M 16s 2023-07-02T14:02:40.095 INFO:tasks.workunit.client.0.smithi163.stderr: 2200K .......... .......... .......... .......... .......... 1% 30.6M 15s 2023-07-02T14:02:40.109 INFO:tasks.workunit.client.0.smithi163.stderr: 2250K .......... .......... .......... .......... .......... 1% 3.53M 16s 2023-07-02T14:02:40.114 INFO:tasks.workunit.client.0.smithi163.stderr: 2300K .......... .......... .......... .......... .......... 1% 9.42M 16s 2023-07-02T14:02:40.120 INFO:tasks.workunit.client.0.smithi163.stderr: 2350K .......... .......... .......... .......... .......... 1% 7.91M 16s 2023-07-02T14:02:40.123 INFO:tasks.workunit.client.0.smithi163.stderr: 2400K .......... .......... .......... .......... .......... 1% 18.6M 16s 2023-07-02T14:02:40.126 INFO:tasks.workunit.client.0.smithi163.stderr: 2450K .......... .......... .......... .......... .......... 1% 21.7M 16s 2023-07-02T14:02:40.129 INFO:tasks.workunit.client.0.smithi163.stderr: 2500K .......... .......... .......... .......... .......... 1% 13.6M 16s 2023-07-02T14:02:40.130 INFO:tasks.workunit.client.0.smithi163.stderr: 2550K .......... .......... .......... .......... .......... 1% 28.0M 15s 2023-07-02T14:02:40.134 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:40.134 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:40.135 INFO:tasks.workunit.client.0.smithi163.stderr: 2600K .......... .......... .......... .......... .......... 1% 11.7M 15s 2023-07-02T14:02:40.136 INFO:tasks.workunit.client.0.smithi163.stderr: 2650K .......... .......... .......... .......... .......... 1% 32.6M 15s 2023-07-02T14:02:40.139 INFO:tasks.workunit.client.0.smithi163.stderr: 2700K .......... .......... .......... .......... .......... 1% 14.7M 15s 2023-07-02T14:02:40.142 INFO:tasks.workunit.client.0.smithi163.stderr: 2750K .......... .......... .......... .......... .......... 1% 18.1M 15s 2023-07-02T14:02:40.144 INFO:tasks.workunit.client.0.smithi163.stderr: 2800K .......... .......... .......... .......... .......... 1% 29.9M 15s 2023-07-02T14:02:40.145 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:02:40.147 INFO:tasks.workunit.client.0.smithi163.stderr: 2850K .......... .......... .......... .......... .......... 1% 13.8M 15s 2023-07-02T14:02:40.151 INFO:tasks.workunit.client.0.smithi163.stderr: 2900K .......... .......... .......... .......... .......... 1% 13.4M 15s 2023-07-02T14:02:40.153 INFO:tasks.workunit.client.0.smithi163.stderr: 2950K .......... .......... .......... .......... .......... 1% 21.6M 15s 2023-07-02T14:02:40.156 INFO:tasks.workunit.client.0.smithi163.stderr: 3000K .......... .......... .......... .......... .......... 1% 14.6M 15s 2023-07-02T14:02:40.160 INFO:tasks.workunit.client.0.smithi163.stderr: 3050K .......... .......... .......... .......... .......... 1% 15.4M 15s 2023-07-02T14:02:40.174 INFO:tasks.workunit.client.0.smithi163.stderr: 3100K .......... .......... .......... .......... .......... 1% 3.41M 15s 2023-07-02T14:02:40.179 INFO:tasks.workunit.client.0.smithi163.stderr: 3150K .......... .......... .......... .......... .......... 1% 9.41M 15s 2023-07-02T14:02:40.184 INFO:tasks.workunit.client.0.smithi163.stderr: 3200K .......... .......... .......... .......... .......... 1% 10.7M 15s 2023-07-02T14:02:40.185 INFO:tasks.workunit.client.0.smithi163.stderr: 3250K .......... .......... .......... .......... .......... 1% 26.9M 15s 2023-07-02T14:02:40.188 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:40.188+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:40.188 INFO:tasks.workunit.client.0.smithi163.stderr: 3300K .......... .......... .......... .......... .......... 1% 18.1M 15s 2023-07-02T14:02:40.192 INFO:tasks.workunit.client.0.smithi163.stderr: 3350K .......... .......... .......... .......... .......... 2% 13.4M 15s 2023-07-02T14:02:40.194 INFO:tasks.workunit.client.0.smithi163.stderr: 3400K .......... .......... .......... .......... .......... 2% 19.4M 15s 2023-07-02T14:02:40.197 INFO:tasks.workunit.client.0.smithi163.stderr: 3450K .......... .......... .......... .......... .......... 2% 16.7M 15s 2023-07-02T14:02:40.201 INFO:tasks.workunit.client.0.smithi163.stderr: 3500K .......... .......... .......... .......... .......... 2% 16.1M 15s 2023-07-02T14:02:40.203 INFO:tasks.workunit.client.0.smithi163.stderr: 3550K .......... .......... .......... .......... .......... 2% 20.5M 14s 2023-07-02T14:02:40.204 INFO:tasks.workunit.client.0.smithi163.stderr: 3600K .......... .......... .......... .......... .......... 2% 31.5M 14s 2023-07-02T14:02:40.206 INFO:tasks.workunit.client.0.smithi163.stderr: 3650K .......... .......... .......... .......... .......... 2% 29.2M 14s 2023-07-02T14:02:40.209 INFO:tasks.workunit.client.0.smithi163.stderr: 3700K .......... .......... .......... .......... .......... 2% 13.7M 14s 2023-07-02T14:02:40.215 INFO:tasks.workunit.client.0.smithi163.stderr: 3750K .......... .......... .......... .......... .......... 2% 9.20M 14s 2023-07-02T14:02:40.218 INFO:tasks.workunit.client.0.smithi163.stderr: 3800K .......... .......... .......... .......... .......... 2% 13.2M 14s 2023-07-02T14:02:40.223 INFO:tasks.workunit.client.0.smithi163.stderr: 3850K .......... .......... .......... .......... .......... 2% 10.5M 14s 2023-07-02T14:02:40.227 INFO:tasks.workunit.client.0.smithi163.stderr: 3900K .......... .......... .......... .......... .......... 2% 11.8M 14s 2023-07-02T14:02:40.237 INFO:tasks.workunit.client.0.smithi163.stderr: 3950K .......... .......... .......... .......... .......... 2% 5.15M 14s 2023-07-02T14:02:40.241 INFO:tasks.workunit.client.0.smithi163.stderr: 4000K .......... .......... .......... .......... .......... 2% 11.1M 14s 2023-07-02T14:02:40.245 INFO:tasks.workunit.client.0.smithi163.stderr: 4050K .......... .......... .......... .......... .......... 2% 14.3M 14s 2023-07-02T14:02:40.248 INFO:tasks.workunit.client.0.smithi163.stderr: 4100K .......... .......... .......... .......... .......... 2% 14.6M 14s 2023-07-02T14:02:40.250 INFO:tasks.workunit.client.0.smithi163.stderr: 4150K .......... .......... .......... .......... .......... 2% 27.4M 14s 2023-07-02T14:02:40.250 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:40.249+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:40.252 INFO:tasks.workunit.client.0.smithi163.stderr: 4200K .......... .......... .......... .......... .......... 2% 21.8M 14s 2023-07-02T14:02:40.255 INFO:tasks.workunit.client.0.smithi163.stderr: 4250K .......... .......... .......... .......... .......... 2% 14.6M 14s 2023-07-02T14:02:40.258 INFO:tasks.workunit.client.0.smithi163.stderr: 4300K .......... .......... .......... .......... .......... 2% 15.6M 14s 2023-07-02T14:02:40.262 INFO:tasks.workunit.client.0.smithi163.stderr: 4350K .......... .......... .......... .......... .......... 2% 12.5M 14s 2023-07-02T14:02:40.265 INFO:tasks.workunit.client.0.smithi163.stderr: 4400K .......... .......... .......... .......... .......... 2% 21.1M 14s 2023-07-02T14:02:40.268 INFO:tasks.workunit.client.0.smithi163.stderr: 4450K .......... .......... .......... .......... .......... 2% 16.6M 14s 2023-07-02T14:02:40.270 INFO:tasks.workunit.client.0.smithi163.stderr: 4500K .......... .......... .......... .......... .......... 2% 21.0M 14s 2023-07-02T14:02:40.273 INFO:tasks.workunit.client.0.smithi163.stderr: 4550K .......... .......... .......... .......... .......... 2% 17.5M 14s 2023-07-02T14:02:40.279 INFO:tasks.workunit.client.0.smithi163.stderr: 4600K .......... .......... .......... .......... .......... 2% 8.41M 14s 2023-07-02T14:02:40.279 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:40.279 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:40.284 INFO:tasks.workunit.client.0.smithi163.stderr: 4650K .......... .......... .......... .......... .......... 2% 9.25M 14s 2023-07-02T14:02:40.290 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:02:40.291 INFO:tasks.workunit.client.0.smithi163.stderr: 4700K .......... .......... .......... .......... .......... 2% 7.03M 14s 2023-07-02T14:02:40.297 INFO:tasks.workunit.client.0.smithi163.stderr: 4750K .......... .......... .......... .......... .......... 2% 8.39M 14s 2023-07-02T14:02:40.302 INFO:tasks.workunit.client.0.smithi163.stderr: 4800K .......... .......... .......... .......... .......... 2% 8.31M 14s 2023-07-02T14:02:40.305 INFO:tasks.workunit.client.0.smithi163.stderr: 4850K .......... .......... .......... .......... .......... 2% 23.5M 14s 2023-07-02T14:02:40.308 INFO:tasks.workunit.client.0.smithi163.stderr: 4900K .......... .......... .......... .......... .......... 2% 13.8M 14s 2023-07-02T14:02:40.312 INFO:tasks.workunit.client.0.smithi163.stderr: 4950K .......... .......... .......... .......... .......... 2% 13.8M 14s 2023-07-02T14:02:40.313 INFO:tasks.workunit.client.0.smithi163.stderr: 5000K .......... .......... .......... .......... .......... 2% 30.4M 14s 2023-07-02T14:02:40.316 INFO:tasks.workunit.client.0.smithi163.stderr: 5050K .......... .......... .......... .......... .......... 3% 23.7M 14s 2023-07-02T14:02:40.317 INFO:tasks.workunit.client.0.smithi163.stderr: 5100K .......... .......... .......... .......... .......... 3% 27.0M 14s 2023-07-02T14:02:40.321 INFO:tasks.workunit.client.0.smithi163.stderr: 5150K .......... .......... .......... .......... .......... 3% 12.3M 14s 2023-07-02T14:02:40.323 INFO:tasks.workunit.client.0.smithi163.stderr: 5200K .......... .......... .......... .......... .......... 3% 19.4M 14s 2023-07-02T14:02:40.327 INFO:tasks.workunit.client.0.smithi163.stderr: 5250K .......... .......... .......... .......... .......... 3% 13.4M 14s 2023-07-02T14:02:40.330 INFO:tasks.workunit.client.0.smithi163.stderr: 5300K .......... .......... .......... .......... .......... 3% 16.0M 14s 2023-07-02T14:02:40.334 INFO:tasks.workunit.client.0.smithi163.stderr: 5350K .......... .......... .......... .......... .......... 3% 13.9M 13s 2023-07-02T14:02:40.337 INFO:tasks.workunit.client.0.smithi163.stderr: 5400K .......... .......... .......... .......... .......... 3% 16.6M 13s 2023-07-02T14:02:40.340 INFO:tasks.workunit.client.0.smithi163.stderr: 5450K .......... .......... .......... .......... .......... 3% 17.5M 13s 2023-07-02T14:02:40.342 INFO:tasks.workunit.client.0.smithi163.stderr: 5500K .......... .......... .......... .......... .......... 3% 15.8M 13s 2023-07-02T14:02:40.350 INFO:tasks.workunit.client.0.smithi163.stderr: 5550K .......... .......... .......... .......... .......... 3% 6.38M 13s 2023-07-02T14:02:40.351 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:40.350+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:40.355 INFO:tasks.workunit.client.0.smithi163.stderr: 5600K .......... .......... .......... .......... .......... 3% 10.7M 13s 2023-07-02T14:02:40.357 INFO:tasks.workunit.client.0.smithi163.stderr: 5650K .......... .......... .......... .......... .......... 3% 19.6M 13s 2023-07-02T14:02:40.363 INFO:tasks.workunit.client.0.smithi163.stderr: 5700K .......... .......... .......... .......... .......... 3% 9.99M 13s 2023-07-02T14:02:40.364 INFO:tasks.workunit.client.0.smithi163.stderr: 5750K .......... .......... .......... .......... .......... 3% 22.7M 13s 2023-07-02T14:02:40.368 INFO:tasks.workunit.client.0.smithi163.stderr: 5800K .......... .......... .......... .......... .......... 3% 13.8M 13s 2023-07-02T14:02:40.369 INFO:tasks.workunit.client.0.smithi163.stderr: 5850K .......... .......... .......... .......... .......... 3% 30.9M 13s 2023-07-02T14:02:40.373 INFO:tasks.workunit.client.0.smithi163.stderr: 5900K .......... .......... .......... .......... .......... 3% 15.9M 13s 2023-07-02T14:02:40.376 INFO:tasks.workunit.client.0.smithi163.stderr: 5950K .......... .......... .......... .......... .......... 3% 13.7M 13s 2023-07-02T14:02:40.378 INFO:tasks.workunit.client.0.smithi163.stderr: 6000K .......... .......... .......... .......... .......... 3% 23.8M 13s 2023-07-02T14:02:40.380 INFO:tasks.workunit.client.0.smithi163.stderr: 6050K .......... .......... .......... .......... .......... 3% 23.5M 13s 2023-07-02T14:02:40.382 INFO:tasks.workunit.client.0.smithi163.stderr: 6100K .......... .......... .......... .......... .......... 3% 22.7M 13s 2023-07-02T14:02:40.387 INFO:tasks.workunit.client.0.smithi163.stderr: 6150K .......... .......... .......... .......... .......... 3% 11.4M 13s 2023-07-02T14:02:40.390 INFO:tasks.workunit.client.0.smithi163.stderr: 6200K .......... .......... .......... .......... .......... 3% 14.7M 13s 2023-07-02T14:02:40.391 INFO:tasks.workunit.client.0.smithi163.stderr: 6250K .......... .......... .......... .......... .......... 3% 40.2M 13s 2023-07-02T14:02:40.394 INFO:tasks.workunit.client.0.smithi163.stderr: 6300K .......... .......... .......... .......... .......... 3% 18.1M 13s 2023-07-02T14:02:40.400 INFO:tasks.workunit.client.0.smithi163.stderr: 6350K .......... .......... .......... .......... .......... 3% 8.39M 13s 2023-07-02T14:02:40.403 INFO:tasks.workunit.client.0.smithi163.stderr: 6400K .......... .......... .......... .......... .......... 3% 15.7M 13s 2023-07-02T14:02:40.407 INFO:tasks.workunit.client.0.smithi163.stderr: 6450K .......... .......... .......... .......... .......... 3% 18.2M 13s 2023-07-02T14:02:40.409 INFO:tasks.workunit.client.0.smithi163.stderr: 6500K .......... .......... .......... .......... .......... 3% 15.9M 13s 2023-07-02T14:02:40.415 INFO:tasks.workunit.client.0.smithi163.stderr: 6550K .......... .......... .......... .......... .......... 3% 7.33M 13s 2023-07-02T14:02:40.417 INFO:tasks.workunit.client.0.smithi163.stderr: 6600K .......... .......... .......... .......... .......... 3% 32.7M 13s 2023-07-02T14:02:40.421 INFO:tasks.workunit.client.0.smithi163.stderr: 6650K .......... .......... .......... .......... .......... 3% 11.4M 13s 2023-07-02T14:02:40.424 INFO:tasks.workunit.client.0.smithi163.stderr: 6700K .......... .......... .......... .......... .......... 4% 17.5M 13s 2023-07-02T14:02:40.427 INFO:tasks.workunit.client.0.smithi163.stderr: 6750K .......... .......... .......... .......... .......... 4% 14.1M 13s 2023-07-02T14:02:40.430 INFO:tasks.workunit.client.0.smithi163.stderr: 6800K .......... .......... .......... .......... .......... 4% 21.3M 13s 2023-07-02T14:02:40.431 INFO:tasks.workunit.client.0.smithi163.stderr: 6850K .......... .......... .......... .......... .......... 4% 25.8M 13s 2023-07-02T14:02:40.433 INFO:tasks.workunit.client.0.smithi163.stderr: 6900K .......... .......... .......... .......... .......... 4% 26.9M 13s 2023-07-02T14:02:40.435 INFO:tasks.workunit.client.0.smithi163.stderr: 6950K .......... .......... .......... .......... .......... 4% 21.4M 13s 2023-07-02T14:02:40.438 INFO:tasks.workunit.client.0.smithi163.stderr: 7000K .......... .......... .......... .......... .......... 4% 22.5M 13s 2023-07-02T14:02:40.441 INFO:tasks.workunit.client.0.smithi163.stderr: 7050K .......... .......... .......... .......... .......... 4% 14.4M 13s 2023-07-02T14:02:40.443 INFO:tasks.workunit.client.0.smithi163.stderr: 7100K .......... .......... .......... .......... .......... 4% 28.6M 13s 2023-07-02T14:02:40.446 INFO:tasks.workunit.client.0.smithi163.stderr: 7150K .......... .......... .......... .......... .......... 4% 14.9M 13s 2023-07-02T14:02:40.450 INFO:tasks.workunit.client.0.smithi163.stderr: 7200K .......... .......... .......... .......... .......... 4% 13.5M 13s 2023-07-02T14:02:40.451 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:40.451+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:40.452 INFO:tasks.workunit.client.0.smithi163.stderr: 7250K .......... .......... .......... .......... .......... 4% 33.5M 12s 2023-07-02T14:02:40.454 INFO:tasks.workunit.client.0.smithi163.stderr: 7300K .......... .......... .......... .......... .......... 4% 18.9M 12s 2023-07-02T14:02:40.457 INFO:tasks.workunit.client.0.smithi163.stderr: 7350K .......... .......... .......... .......... .......... 4% 16.3M 12s 2023-07-02T14:02:40.460 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:40.460 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:40.461 INFO:tasks.workunit.client.0.smithi163.stderr: 7400K .......... .......... .......... .......... .......... 4% 16.0M 12s 2023-07-02T14:02:40.463 INFO:tasks.workunit.client.0.smithi163.stderr: 7450K .......... .......... .......... .......... .......... 4% 17.9M 12s 2023-07-02T14:02:40.466 INFO:tasks.workunit.client.0.smithi163.stderr: 7500K .......... .......... .......... .......... .......... 4% 14.3M 12s 2023-07-02T14:02:40.470 INFO:tasks.workunit.client.0.smithi163.stderr: 7550K .......... .......... .......... .......... .......... 4% 11.6M 12s 2023-07-02T14:02:40.473 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:02:40.474 INFO:tasks.workunit.client.0.smithi163.stderr: 7600K .......... .......... .......... .......... .......... 4% 12.5M 12s 2023-07-02T14:02:40.476 INFO:tasks.workunit.client.0.smithi163.stderr: 7650K .......... .......... .......... .......... .......... 4% 24.1M 12s 2023-07-02T14:02:40.480 INFO:tasks.workunit.client.0.smithi163.stderr: 7700K .......... .......... .......... .......... .......... 4% 12.5M 12s 2023-07-02T14:02:40.482 INFO:tasks.workunit.client.0.smithi163.stderr: 7750K .......... .......... .......... .......... .......... 4% 25.6M 12s 2023-07-02T14:02:40.484 INFO:tasks.workunit.client.0.smithi163.stderr: 7800K .......... .......... .......... .......... .......... 4% 25.4M 12s 2023-07-02T14:02:40.486 INFO:tasks.workunit.client.0.smithi163.stderr: 7850K .......... .......... .......... .......... .......... 4% 19.5M 12s 2023-07-02T14:02:40.489 INFO:tasks.workunit.client.0.smithi163.stderr: 7900K .......... .......... .......... .......... .......... 4% 20.8M 12s 2023-07-02T14:02:40.491 INFO:tasks.workunit.client.0.smithi163.stderr: 7950K .......... .......... .......... .......... .......... 4% 20.2M 12s 2023-07-02T14:02:40.493 INFO:tasks.workunit.client.0.smithi163.stderr: 8000K .......... .......... .......... .......... .......... 4% 19.7M 12s 2023-07-02T14:02:40.496 INFO:tasks.workunit.client.0.smithi163.stderr: 8050K .......... .......... .......... .......... .......... 4% 17.9M 12s 2023-07-02T14:02:40.500 INFO:tasks.workunit.client.0.smithi163.stderr: 8100K .......... .......... .......... .......... .......... 4% 16.4M 12s 2023-07-02T14:02:40.501 INFO:tasks.workunit.client.0.smithi163.stderr: 8150K .......... .......... .......... .......... .......... 4% 36.5M 12s 2023-07-02T14:02:40.504 INFO:tasks.workunit.client.0.smithi163.stderr: 8200K .......... .......... .......... .......... .......... 4% 17.7M 12s 2023-07-02T14:02:40.506 INFO:tasks.workunit.client.0.smithi163.stderr: 8250K .......... .......... .......... .......... .......... 4% 20.6M 12s 2023-07-02T14:02:40.509 INFO:tasks.workunit.client.0.smithi163.stderr: 8300K .......... .......... .......... .......... .......... 4% 17.3M 12s 2023-07-02T14:02:40.512 INFO:tasks.workunit.client.0.smithi163.stderr: 8350K .......... .......... .......... .......... .......... 4% 12.9M 12s 2023-07-02T14:02:40.514 INFO:tasks.workunit.client.0.smithi163.stderr: 8400K .......... .......... .......... .......... .......... 5% 32.7M 12s 2023-07-02T14:02:40.516 INFO:tasks.workunit.client.0.smithi163.stderr: 8450K .......... .......... .......... .......... .......... 5% 20.9M 12s 2023-07-02T14:02:40.518 INFO:tasks.workunit.client.0.smithi163.stderr: 8500K .......... .......... .......... .......... .......... 5% 33.7M 12s 2023-07-02T14:02:40.519 INFO:tasks.workunit.client.0.smithi163.stderr: 8550K .......... .......... .......... .......... .......... 5% 34.7M 12s 2023-07-02T14:02:40.521 INFO:tasks.workunit.client.0.smithi163.stderr: 8600K .......... .......... .......... .......... .......... 5% 23.8M 12s 2023-07-02T14:02:40.524 INFO:tasks.workunit.client.0.smithi163.stderr: 8650K .......... .......... .......... .......... .......... 5% 17.5M 12s 2023-07-02T14:02:40.534 INFO:tasks.workunit.client.0.smithi163.stderr: 8700K .......... .......... .......... .......... .......... 5% 4.98M 12s 2023-07-02T14:02:40.537 INFO:tasks.workunit.client.0.smithi163.stderr: 8750K .......... .......... .......... .......... .......... 5% 14.2M 12s 2023-07-02T14:02:40.541 INFO:tasks.workunit.client.0.smithi163.stderr: 8800K .......... .......... .......... .......... .......... 5% 14.7M 12s 2023-07-02T14:02:40.543 INFO:tasks.workunit.client.0.smithi163.stderr: 8850K .......... .......... .......... .......... .......... 5% 23.1M 12s 2023-07-02T14:02:40.545 INFO:tasks.workunit.client.0.smithi163.stderr: 8900K .......... .......... .......... .......... .......... 5% 26.0M 12s 2023-07-02T14:02:40.546 INFO:tasks.workunit.client.0.smithi163.stderr: 8950K .......... .......... .......... .......... .......... 5% 30.3M 12s 2023-07-02T14:02:40.549 INFO:tasks.workunit.client.0.smithi163.stderr: 9000K .......... .......... .......... .......... .......... 5% 19.1M 12s 2023-07-02T14:02:40.551 INFO:tasks.workunit.client.0.smithi163.stderr: 9050K .......... .......... .......... .......... .......... 5% 23.2M 12s 2023-07-02T14:02:40.551 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:40.550+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:40.552 INFO:tasks.workunit.client.0.smithi163.stderr: 9100K .......... .......... .......... .......... .......... 5% 30.4M 12s 2023-07-02T14:02:40.556 INFO:tasks.workunit.client.0.smithi163.stderr: 9150K .......... .......... .......... .......... .......... 5% 12.2M 12s 2023-07-02T14:02:40.559 INFO:tasks.workunit.client.0.smithi163.stderr: 9200K .......... .......... .......... .......... .......... 5% 19.6M 12s 2023-07-02T14:02:40.561 INFO:tasks.workunit.client.0.smithi163.stderr: 9250K .......... .......... .......... .......... .......... 5% 21.2M 12s 2023-07-02T14:02:40.564 INFO:tasks.workunit.client.0.smithi163.stderr: 9300K .......... .......... .......... .......... .......... 5% 20.3M 12s 2023-07-02T14:02:40.567 INFO:tasks.workunit.client.0.smithi163.stderr: 9350K .......... .......... .......... .......... .......... 5% 16.7M 11s 2023-07-02T14:02:40.568 INFO:tasks.workunit.client.0.smithi163.stderr: 9400K .......... .......... .......... .......... .......... 5% 28.0M 11s 2023-07-02T14:02:40.570 INFO:tasks.workunit.client.0.smithi163.stderr: 9450K .......... .......... .......... .......... .......... 5% 26.2M 11s 2023-07-02T14:02:40.573 INFO:tasks.workunit.client.0.smithi163.stderr: 9500K .......... .......... .......... .......... .......... 5% 16.7M 11s 2023-07-02T14:02:40.576 INFO:tasks.workunit.client.0.smithi163.stderr: 9550K .......... .......... .......... .......... .......... 5% 17.0M 11s 2023-07-02T14:02:40.578 INFO:tasks.workunit.client.0.smithi163.stderr: 9600K .......... .......... .......... .......... .......... 5% 27.9M 11s 2023-07-02T14:02:40.580 INFO:tasks.workunit.client.0.smithi163.stderr: 9650K .......... .......... .......... .......... .......... 5% 22.4M 11s 2023-07-02T14:02:40.582 INFO:tasks.workunit.client.0.smithi163.stderr: 9700K .......... .......... .......... .......... .......... 5% 19.5M 11s 2023-07-02T14:02:40.586 INFO:tasks.workunit.client.0.smithi163.stderr: 9750K .......... .......... .......... .......... .......... 5% 14.3M 11s 2023-07-02T14:02:40.594 INFO:tasks.workunit.client.0.smithi163.stderr: 9800K .......... .......... .......... .......... .......... 5% 6.07M 11s 2023-07-02T14:02:40.597 INFO:tasks.workunit.client.0.smithi163.stderr: 9850K .......... .......... .......... .......... .......... 5% 18.2M 11s 2023-07-02T14:02:40.599 INFO:tasks.workunit.client.0.smithi163.stderr: 9900K .......... .......... .......... .......... .......... 5% 17.6M 11s 2023-07-02T14:02:40.602 INFO:tasks.workunit.client.0.smithi163.stderr: 9950K .......... .......... .......... .......... .......... 5% 19.9M 11s 2023-07-02T14:02:40.604 INFO:tasks.workunit.client.0.smithi163.stderr: 10000K .......... .......... .......... .......... .......... 5% 19.6M 11s 2023-07-02T14:02:40.607 INFO:tasks.workunit.client.0.smithi163.stderr: 10050K .......... .......... .......... .......... .......... 5% 18.1M 11s 2023-07-02T14:02:40.609 INFO:tasks.workunit.client.0.smithi163.stderr: 10100K .......... .......... .......... .......... .......... 6% 21.2M 11s 2023-07-02T14:02:40.611 INFO:tasks.workunit.client.0.smithi163.stderr: 10150K .......... .......... .......... .......... .......... 6% 26.1M 11s 2023-07-02T14:02:40.614 INFO:tasks.workunit.client.0.smithi163.stderr: 10200K .......... .......... .......... .......... .......... 6% 16.1M 11s 2023-07-02T14:02:40.616 INFO:tasks.workunit.client.0.smithi163.stderr: 10250K .......... .......... .......... .......... .......... 6% 22.0M 11s 2023-07-02T14:02:40.618 INFO:tasks.workunit.client.0.smithi163.stderr: 10300K .......... .......... .......... .......... .......... 6% 27.7M 11s 2023-07-02T14:02:40.621 INFO:tasks.workunit.client.0.smithi163.stderr: 10350K .......... .......... .......... .......... .......... 6% 16.9M 11s 2023-07-02T14:02:40.625 INFO:tasks.workunit.client.0.smithi163.stderr: 10400K .......... .......... .......... .......... .......... 6% 14.1M 11s 2023-07-02T14:02:40.627 INFO:tasks.workunit.client.0.smithi163.stderr: 10450K .......... .......... .......... .......... .......... 6% 18.0M 11s 2023-07-02T14:02:40.629 INFO:tasks.workunit.client.0.smithi163.stderr: 10500K .......... .......... .......... .......... .......... 6% 29.4M 11s 2023-07-02T14:02:40.633 INFO:tasks.workunit.client.0.smithi163.stderr: 10550K .......... .......... .......... .......... .......... 6% 10.4M 11s 2023-07-02T14:02:40.637 INFO:tasks.workunit.client.0.smithi163.stderr: 10600K .......... .......... .......... .......... .......... 6% 26.5M 11s 2023-07-02T14:02:40.637 INFO:tasks.workunit.client.0.smithi163.stderr: 10650K .......... .......... .......... .......... .......... 6% 91.2M 11s 2023-07-02T14:02:40.637 INFO:tasks.workunit.client.0.smithi163.stderr: 10700K .......... .......... .......... .......... .......... 6% 58.2M 11s 2023-07-02T14:02:40.641 INFO:tasks.workunit.client.0.smithi163.stderr: 10750K .......... .......... .......... .......... .......... 6% 12.8M 11s 2023-07-02T14:02:40.644 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:40.645 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:40.645 INFO:tasks.workunit.client.0.smithi163.stderr: 10800K .......... .......... .......... .......... .......... 6% 10.8M 11s 2023-07-02T14:02:40.648 INFO:tasks.workunit.client.0.smithi163.stderr: 10850K .......... .......... .......... .......... .......... 6% 17.9M 11s 2023-07-02T14:02:40.651 INFO:tasks.workunit.client.0.smithi163.stderr: 10900K .......... .......... .......... .......... .......... 6% 12.8M 11s 2023-07-02T14:02:40.652 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:40.652+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:40.653 INFO:tasks.workunit.client.0.smithi163.stderr: 10950K .......... .......... .......... .......... .......... 6% 22.5M 11s 2023-07-02T14:02:40.655 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:02:40.659 INFO:tasks.workunit.client.0.smithi163.stderr: 11000K .......... .......... .......... .......... .......... 6% 8.83M 11s 2023-07-02T14:02:40.661 INFO:tasks.workunit.client.0.smithi163.stderr: 11050K .......... .......... .......... .......... .......... 6% 26.8M 11s 2023-07-02T14:02:40.664 INFO:tasks.workunit.client.0.smithi163.stderr: 11100K .......... .......... .......... .......... .......... 6% 15.8M 11s 2023-07-02T14:02:40.667 INFO:tasks.workunit.client.0.smithi163.stderr: 11150K .......... .......... .......... .......... .......... 6% 17.8M 11s 2023-07-02T14:02:40.668 INFO:tasks.workunit.client.0.smithi163.stderr: 11200K .......... .......... .......... .......... .......... 6% 39.1M 11s 2023-07-02T14:02:40.671 INFO:tasks.workunit.client.0.smithi163.stderr: 11250K .......... .......... .......... .......... .......... 6% 22.7M 11s 2023-07-02T14:02:40.673 INFO:tasks.workunit.client.0.smithi163.stderr: 11300K .......... .......... .......... .......... .......... 6% 21.0M 11s 2023-07-02T14:02:40.674 INFO:tasks.workunit.client.0.smithi163.stderr: 11350K .......... .......... .......... .......... .......... 6% 26.5M 11s 2023-07-02T14:02:40.677 INFO:tasks.workunit.client.0.smithi163.stderr: 11400K .......... .......... .......... .......... .......... 6% 18.0M 11s 2023-07-02T14:02:40.679 INFO:tasks.workunit.client.0.smithi163.stderr: 11450K .......... .......... .......... .......... .......... 6% 22.5M 11s 2023-07-02T14:02:40.681 INFO:tasks.workunit.client.0.smithi163.stderr: 11500K .......... .......... .......... .......... .......... 6% 26.3M 11s 2023-07-02T14:02:40.684 INFO:tasks.workunit.client.0.smithi163.stderr: 11550K .......... .......... .......... .......... .......... 6% 17.5M 11s 2023-07-02T14:02:40.686 INFO:tasks.workunit.client.0.smithi163.stderr: 11600K .......... .......... .......... .......... .......... 6% 30.8M 11s 2023-07-02T14:02:40.688 INFO:tasks.workunit.client.0.smithi163.stderr: 11650K .......... .......... .......... .......... .......... 6% 18.1M 11s 2023-07-02T14:02:40.690 INFO:tasks.workunit.client.0.smithi163.stderr: 11700K .......... .......... .......... .......... .......... 6% 25.8M 11s 2023-07-02T14:02:40.692 INFO:tasks.workunit.client.0.smithi163.stderr: 11750K .......... .......... .......... .......... .......... 7% 31.5M 11s 2023-07-02T14:02:40.694 INFO:tasks.workunit.client.0.smithi163.stderr: 11800K .......... .......... .......... .......... .......... 7% 17.7M 11s 2023-07-02T14:02:40.696 INFO:tasks.workunit.client.0.smithi163.stderr: 11850K .......... .......... .......... .......... .......... 7% 32.7M 11s 2023-07-02T14:02:40.698 INFO:tasks.workunit.client.0.smithi163.stderr: 11900K .......... .......... .......... .......... .......... 7% 24.2M 11s 2023-07-02T14:02:40.701 INFO:tasks.workunit.client.0.smithi163.stderr: 11950K .......... .......... .......... .......... .......... 7% 14.6M 11s 2023-07-02T14:02:40.704 INFO:tasks.workunit.client.0.smithi163.stderr: 12000K .......... .......... .......... .......... .......... 7% 18.2M 11s 2023-07-02T14:02:40.706 INFO:tasks.workunit.client.0.smithi163.stderr: 12050K .......... .......... .......... .......... .......... 7% 24.9M 11s 2023-07-02T14:02:40.710 INFO:tasks.workunit.client.0.smithi163.stderr: 12100K .......... .......... .......... .......... .......... 7% 13.9M 11s 2023-07-02T14:02:40.714 INFO:tasks.workunit.client.0.smithi163.stderr: 12150K .......... .......... .......... .......... .......... 7% 11.6M 11s 2023-07-02T14:02:40.715 INFO:tasks.workunit.client.0.smithi163.stderr: 12200K .......... .......... .......... .......... .......... 7% 27.5M 11s 2023-07-02T14:02:40.719 INFO:tasks.workunit.client.0.smithi163.stderr: 12250K .......... .......... .......... .......... .......... 7% 14.9M 11s 2023-07-02T14:02:40.722 INFO:tasks.workunit.client.0.smithi163.stderr: 12300K .......... .......... .......... .......... .......... 7% 19.3M 11s 2023-07-02T14:02:40.724 INFO:tasks.workunit.client.0.smithi163.stderr: 12350K .......... .......... .......... .......... .......... 7% 19.8M 11s 2023-07-02T14:02:40.726 INFO:tasks.workunit.client.0.smithi163.stderr: 12400K .......... .......... .......... .......... .......... 7% 23.4M 11s 2023-07-02T14:02:40.728 INFO:tasks.workunit.client.0.smithi163.stderr: 12450K .......... .......... .......... .......... .......... 7% 28.2M 10s 2023-07-02T14:02:40.730 INFO:tasks.workunit.client.0.smithi163.stderr: 12500K .......... .......... .......... .......... .......... 7% 18.5M 10s 2023-07-02T14:02:40.732 INFO:tasks.workunit.client.0.smithi163.stderr: 12550K .......... .......... .......... .......... .......... 7% 30.2M 10s 2023-07-02T14:02:40.734 INFO:tasks.workunit.client.0.smithi163.stderr: 12600K .......... .......... .......... .......... .......... 7% 25.3M 10s 2023-07-02T14:02:40.736 INFO:tasks.workunit.client.0.smithi163.stderr: 12650K .......... .......... .......... .......... .......... 7% 21.2M 10s 2023-07-02T14:02:40.738 INFO:tasks.workunit.client.0.smithi163.stderr: 12700K .......... .......... .......... .......... .......... 7% 29.7M 10s 2023-07-02T14:02:40.741 INFO:tasks.workunit.client.0.smithi163.stderr: 12750K .......... .......... .......... .......... .......... 7% 16.7M 10s 2023-07-02T14:02:40.743 INFO:tasks.workunit.client.0.smithi163.stderr: 12800K .......... .......... .......... .......... .......... 7% 17.0M 10s 2023-07-02T14:02:40.746 INFO:tasks.workunit.client.0.smithi163.stderr: 12850K .......... .......... .......... .......... .......... 7% 19.5M 10s 2023-07-02T14:02:40.748 INFO:tasks.workunit.client.0.smithi163.stderr: 12900K .......... .......... .......... .......... .......... 7% 29.0M 10s 2023-07-02T14:02:40.750 INFO:tasks.workunit.client.0.smithi163.stderr: 12950K .......... .......... .......... .......... .......... 7% 26.7M 10s 2023-07-02T14:02:40.753 INFO:tasks.workunit.client.0.smithi163.stderr: 13000K .......... .......... .......... .......... .......... 7% 16.0M 10s 2023-07-02T14:02:40.753 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:40.752+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:40.756 INFO:tasks.workunit.client.0.smithi163.stderr: 13050K .......... .......... .......... .......... .......... 7% 14.2M 10s 2023-07-02T14:02:40.758 INFO:tasks.workunit.client.0.smithi163.stderr: 13100K .......... .......... .......... .......... .......... 7% 22.0M 10s 2023-07-02T14:02:40.762 INFO:tasks.workunit.client.0.smithi163.stderr: 13150K .......... .......... .......... .......... .......... 7% 13.4M 10s 2023-07-02T14:02:40.766 INFO:tasks.workunit.client.0.smithi163.stderr: 13200K .......... .......... .......... .......... .......... 7% 13.0M 10s 2023-07-02T14:02:40.767 INFO:tasks.workunit.client.0.smithi163.stderr: 13250K .......... .......... .......... .......... .......... 7% 26.4M 10s 2023-07-02T14:02:40.770 INFO:tasks.workunit.client.0.smithi163.stderr: 13300K .......... .......... .......... .......... .......... 7% 17.0M 10s 2023-07-02T14:02:40.773 INFO:tasks.workunit.client.0.smithi163.stderr: 13350K .......... .......... .......... .......... .......... 7% 19.6M 10s 2023-07-02T14:02:40.775 INFO:tasks.workunit.client.0.smithi163.stderr: 13400K .......... .......... .......... .......... .......... 7% 19.1M 10s 2023-07-02T14:02:40.778 INFO:tasks.workunit.client.0.smithi163.stderr: 13450K .......... .......... .......... .......... .......... 8% 21.6M 10s 2023-07-02T14:02:40.780 INFO:tasks.workunit.client.0.smithi163.stderr: 13500K .......... .......... .......... .......... .......... 8% 19.0M 10s 2023-07-02T14:02:40.784 INFO:tasks.workunit.client.0.smithi163.stderr: 13550K .......... .......... .......... .......... .......... 8% 14.4M 10s 2023-07-02T14:02:40.786 INFO:tasks.workunit.client.0.smithi163.stderr: 13600K .......... .......... .......... .......... .......... 8% 23.7M 10s 2023-07-02T14:02:40.788 INFO:tasks.workunit.client.0.smithi163.stderr: 13650K .......... .......... .......... .......... .......... 8% 22.4M 10s 2023-07-02T14:02:40.791 INFO:tasks.workunit.client.0.smithi163.stderr: 13700K .......... .......... .......... .......... .......... 8% 19.9M 10s 2023-07-02T14:02:40.792 INFO:tasks.workunit.client.0.smithi163.stderr: 13750K .......... .......... .......... .......... .......... 8% 29.4M 10s 2023-07-02T14:02:40.795 INFO:tasks.workunit.client.0.smithi163.stderr: 13800K .......... .......... .......... .......... .......... 8% 17.9M 10s 2023-07-02T14:02:40.796 INFO:tasks.workunit.client.0.smithi163.stderr: 13850K .......... .......... .......... .......... .......... 8% 35.2M 10s 2023-07-02T14:02:40.799 INFO:tasks.workunit.client.0.smithi163.stderr: 13900K .......... .......... .......... .......... .......... 8% 18.5M 10s 2023-07-02T14:02:40.802 INFO:tasks.workunit.client.0.smithi163.stderr: 13950K .......... .......... .......... .......... .......... 8% 17.6M 10s 2023-07-02T14:02:40.802 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:40.802 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:40.804 INFO:tasks.workunit.client.0.smithi163.stderr: 14000K .......... .......... .......... .......... .......... 8% 17.5M 10s 2023-07-02T14:02:40.807 INFO:tasks.workunit.client.0.smithi163.stderr: 14050K .......... .......... .......... .......... .......... 8% 22.1M 10s 2023-07-02T14:02:40.810 INFO:tasks.workunit.client.0.smithi163.stderr: 14100K .......... .......... .......... .......... .......... 8% 17.4M 10s 2023-07-02T14:02:40.812 INFO:tasks.workunit.client.0.smithi163.stderr: 14150K .......... .......... .......... .......... .......... 8% 19.1M 10s 2023-07-02T14:02:40.813 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:02:40.814 INFO:tasks.workunit.client.0.smithi163.stderr: 14200K .......... .......... .......... .......... .......... 8% 33.4M 10s 2023-07-02T14:02:40.816 INFO:tasks.workunit.client.0.smithi163.stderr: 14250K .......... .......... .......... .......... .......... 8% 22.6M 10s 2023-07-02T14:02:40.819 INFO:tasks.workunit.client.0.smithi163.stderr: 14300K .......... .......... .......... .......... .......... 8% 13.4M 10s 2023-07-02T14:02:40.821 INFO:tasks.workunit.client.0.smithi163.stderr: 14350K .......... .......... .......... .......... .......... 8% 23.0M 10s 2023-07-02T14:02:40.824 INFO:tasks.workunit.client.0.smithi163.stderr: 14400K .......... .......... .......... .......... .......... 8% 18.4M 10s 2023-07-02T14:02:40.826 INFO:tasks.workunit.client.0.smithi163.stderr: 14450K .......... .......... .......... .......... .......... 8% 20.9M 10s 2023-07-02T14:02:40.828 INFO:tasks.workunit.client.0.smithi163.stderr: 14500K .......... .......... .......... .......... .......... 8% 24.6M 10s 2023-07-02T14:02:40.831 INFO:tasks.workunit.client.0.smithi163.stderr: 14550K .......... .......... .......... .......... .......... 8% 16.9M 10s 2023-07-02T14:02:40.833 INFO:tasks.workunit.client.0.smithi163.stderr: 14600K .......... .......... .......... .......... .......... 8% 25.9M 10s 2023-07-02T14:02:40.835 INFO:tasks.workunit.client.0.smithi163.stderr: 14650K .......... .......... .......... .......... .......... 8% 28.2M 10s 2023-07-02T14:02:40.836 INFO:tasks.workunit.client.0.smithi163.stderr: 14700K .......... .......... .......... .......... .......... 8% 38.1M 10s 2023-07-02T14:02:40.839 INFO:tasks.workunit.client.0.smithi163.stderr: 14750K .......... .......... .......... .......... .......... 8% 17.4M 10s 2023-07-02T14:02:40.842 INFO:tasks.workunit.client.0.smithi163.stderr: 14800K .......... .......... .......... .......... .......... 8% 17.5M 10s 2023-07-02T14:02:40.844 INFO:tasks.workunit.client.0.smithi163.stderr: 14850K .......... .......... .......... .......... .......... 8% 20.9M 10s 2023-07-02T14:02:40.845 INFO:tasks.workunit.client.0.smithi163.stderr: 14900K .......... .......... .......... .......... .......... 8% 37.0M 10s 2023-07-02T14:02:40.848 INFO:tasks.workunit.client.0.smithi163.stderr: 14950K .......... .......... .......... .......... .......... 8% 19.2M 10s 2023-07-02T14:02:40.849 INFO:tasks.workunit.client.0.smithi163.stderr: 15000K .......... .......... .......... .......... .......... 8% 32.2M 10s 2023-07-02T14:02:40.852 INFO:tasks.workunit.client.0.smithi163.stderr: 15050K .......... .......... .......... .......... .......... 8% 22.3M 10s 2023-07-02T14:02:40.854 INFO:tasks.workunit.client.0.smithi163.stderr: 15100K .......... .......... .......... .......... .......... 8% 28.5M 10s 2023-07-02T14:02:40.854 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:40.854+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:40.858 INFO:tasks.workunit.client.0.smithi163.stderr: 15150K .......... .......... .......... .......... .......... 9% 12.5M 10s 2023-07-02T14:02:40.859 INFO:tasks.workunit.client.0.smithi163.stderr: 15200K .......... .......... .......... .......... .......... 9% 26.7M 10s 2023-07-02T14:02:40.861 INFO:tasks.workunit.client.0.smithi163.stderr: 15250K .......... .......... .......... .......... .......... 9% 31.5M 10s 2023-07-02T14:02:40.863 INFO:tasks.workunit.client.0.smithi163.stderr: 15300K .......... .......... .......... .......... .......... 9% 19.4M 10s 2023-07-02T14:02:40.866 INFO:tasks.workunit.client.0.smithi163.stderr: 15350K .......... .......... .......... .......... .......... 9% 20.9M 10s 2023-07-02T14:02:40.868 INFO:tasks.workunit.client.0.smithi163.stderr: 15400K .......... .......... .......... .......... .......... 9% 17.7M 10s 2023-07-02T14:02:40.870 INFO:tasks.workunit.client.0.smithi163.stderr: 15450K .......... .......... .......... .......... .......... 9% 28.6M 10s 2023-07-02T14:02:40.873 INFO:tasks.workunit.client.0.smithi163.stderr: 15500K .......... .......... .......... .......... .......... 9% 18.9M 10s 2023-07-02T14:02:40.876 INFO:tasks.workunit.client.0.smithi163.stderr: 15550K .......... .......... .......... .......... .......... 9% 14.5M 10s 2023-07-02T14:02:40.878 INFO:tasks.workunit.client.0.smithi163.stderr: 15600K .......... .......... .......... .......... .......... 9% 22.3M 10s 2023-07-02T14:02:40.880 INFO:tasks.workunit.client.0.smithi163.stderr: 15650K .......... .......... .......... .......... .......... 9% 20.7M 10s 2023-07-02T14:02:40.883 INFO:tasks.workunit.client.0.smithi163.stderr: 15700K .......... .......... .......... .......... .......... 9% 21.1M 10s 2023-07-02T14:02:40.885 INFO:tasks.workunit.client.0.smithi163.stderr: 15750K .......... .......... .......... .......... .......... 9% 25.7M 10s 2023-07-02T14:02:40.887 INFO:tasks.workunit.client.0.smithi163.stderr: 15800K .......... .......... .......... .......... .......... 9% 26.0M 10s 2023-07-02T14:02:40.889 INFO:tasks.workunit.client.0.smithi163.stderr: 15850K .......... .......... .......... .......... .......... 9% 20.5M 10s 2023-07-02T14:02:40.891 INFO:tasks.workunit.client.0.smithi163.stderr: 15900K .......... .......... .......... .......... .......... 9% 24.5M 10s 2023-07-02T14:02:40.894 INFO:tasks.workunit.client.0.smithi163.stderr: 15950K .......... .......... .......... .......... .......... 9% 19.1M 10s 2023-07-02T14:02:40.896 INFO:tasks.workunit.client.0.smithi163.stderr: 16000K .......... .......... .......... .......... .......... 9% 17.7M 10s 2023-07-02T14:02:40.898 INFO:tasks.workunit.client.0.smithi163.stderr: 16050K .......... .......... .......... .......... .......... 9% 30.3M 10s 2023-07-02T14:02:40.900 INFO:tasks.workunit.client.0.smithi163.stderr: 16100K .......... .......... .......... .......... .......... 9% 18.7M 10s 2023-07-02T14:02:40.902 INFO:tasks.workunit.client.0.smithi163.stderr: 16150K .......... .......... .......... .......... .......... 9% 31.1M 10s 2023-07-02T14:02:40.905 INFO:tasks.workunit.client.0.smithi163.stderr: 16200K .......... .......... .......... .......... .......... 9% 20.3M 10s 2023-07-02T14:02:40.906 INFO:tasks.workunit.client.0.smithi163.stderr: 16250K .......... .......... .......... .......... .......... 9% 24.4M 10s 2023-07-02T14:02:40.908 INFO:tasks.workunit.client.0.smithi163.stderr: 16300K .......... .......... .......... .......... .......... 9% 28.6M 9s 2023-07-02T14:02:40.911 INFO:tasks.workunit.client.0.smithi163.stderr: 16350K .......... .......... .......... .......... .......... 9% 16.0M 9s 2023-07-02T14:02:40.914 INFO:tasks.workunit.client.0.smithi163.stderr: 16400K .......... .......... .......... .......... .......... 9% 17.5M 9s 2023-07-02T14:02:40.916 INFO:tasks.workunit.client.0.smithi163.stderr: 16450K .......... .......... .......... .......... .......... 9% 32.1M 9s 2023-07-02T14:02:40.917 INFO:tasks.workunit.client.0.smithi163.stderr: 16500K .......... .......... .......... .......... .......... 9% 32.5M 9s 2023-07-02T14:02:40.920 INFO:tasks.workunit.client.0.smithi163.stderr: 16550K .......... .......... .......... .......... .......... 9% 18.1M 9s 2023-07-02T14:02:40.922 INFO:tasks.workunit.client.0.smithi163.stderr: 16600K .......... .......... .......... .......... .......... 9% 22.0M 9s 2023-07-02T14:02:40.925 INFO:tasks.workunit.client.0.smithi163.stderr: 16650K .......... .......... .......... .......... .......... 9% 18.0M 9s 2023-07-02T14:02:40.927 INFO:tasks.workunit.client.0.smithi163.stderr: 16700K .......... .......... .......... .......... .......... 9% 19.0M 9s 2023-07-02T14:02:40.930 INFO:tasks.workunit.client.0.smithi163.stderr: 16750K .......... .......... .......... .......... .......... 9% 20.2M 9s 2023-07-02T14:02:40.933 INFO:tasks.workunit.client.0.smithi163.stderr: 16800K .......... .......... .......... .......... .......... 9% 15.8M 9s 2023-07-02T14:02:40.935 INFO:tasks.workunit.client.0.smithi163.stderr: 16850K .......... .......... .......... .......... .......... 10% 24.6M 9s 2023-07-02T14:02:40.937 INFO:tasks.workunit.client.0.smithi163.stderr: 16900K .......... .......... .......... .......... .......... 10% 28.2M 9s 2023-07-02T14:02:40.939 INFO:tasks.workunit.client.0.smithi163.stderr: 16950K .......... .......... .......... .......... .......... 10% 15.5M 9s 2023-07-02T14:02:40.941 INFO:tasks.workunit.client.0.smithi163.stderr: 17000K .......... .......... .......... .......... .......... 10% 30.1M 9s 2023-07-02T14:02:40.944 INFO:tasks.workunit.client.0.smithi163.stderr: 17050K .......... .......... .......... .......... .......... 10% 17.3M 9s 2023-07-02T14:02:40.947 INFO:tasks.workunit.client.0.smithi163.stderr: 17100K .......... .......... .......... .......... .......... 10% 17.3M 9s 2023-07-02T14:02:40.949 INFO:tasks.workunit.client.0.smithi163.stderr: 17150K .......... .......... .......... .......... .......... 10% 21.5M 9s 2023-07-02T14:02:40.951 INFO:tasks.workunit.client.0.smithi163.stderr: 17200K .......... .......... .......... .......... .......... 10% 24.9M 9s 2023-07-02T14:02:40.954 INFO:tasks.workunit.client.0.smithi163.stderr: 17250K .......... .......... .......... .......... .......... 10% 21.0M 9s 2023-07-02T14:02:40.954 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:40.953+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:40.955 INFO:tasks.workunit.client.0.smithi163.stderr: 17300K .......... .......... .......... .......... .......... 10% 30.4M 9s 2023-07-02T14:02:40.957 INFO:tasks.workunit.client.0.smithi163.stderr: 17350K .......... .......... .......... .......... .......... 10% 22.9M 9s 2023-07-02T14:02:40.959 INFO:tasks.workunit.client.0.smithi163.stderr: 17400K .......... .......... .......... .......... .......... 10% 24.0M 9s 2023-07-02T14:02:40.961 INFO:tasks.workunit.client.0.smithi163.stderr: 17450K .......... .......... .......... .......... .......... 10% 21.8M 9s 2023-07-02T14:02:40.963 INFO:tasks.workunit.client.0.smithi163.stderr: 17500K .......... .......... .......... .......... .......... 10% 24.7M 9s 2023-07-02T14:02:40.965 INFO:tasks.workunit.client.0.smithi163.stderr: 17550K .......... .......... .......... .......... .......... 10% 21.7M 9s 2023-07-02T14:02:40.969 INFO:tasks.workunit.client.0.smithi163.stderr: 17600K .......... .......... .......... .......... .......... 10% 20.0M 9s 2023-07-02T14:02:40.971 INFO:tasks.workunit.client.0.smithi163.stderr: 17650K .......... .......... .......... .......... .......... 10% 19.2M 9s 2023-07-02T14:02:40.973 INFO:tasks.workunit.client.0.smithi163.stderr: 17700K .......... .......... .......... .......... .......... 10% 30.1M 9s 2023-07-02T14:02:40.975 INFO:tasks.workunit.client.0.smithi163.stderr: 17750K .......... .......... .......... .......... .......... 10% 16.2M 9s 2023-07-02T14:02:40.976 INFO:tasks.workunit.client.0.smithi163.stderr: 17800K .......... .......... .......... .......... .......... 10% 55.1M 9s 2023-07-02T14:02:40.978 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:40.978 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:40.979 INFO:tasks.workunit.client.0.smithi163.stderr: 17850K .......... .......... .......... .......... .......... 10% 17.8M 9s 2023-07-02T14:02:40.981 INFO:tasks.workunit.client.0.smithi163.stderr: 17900K .......... .......... .......... .......... .......... 10% 31.7M 9s 2023-07-02T14:02:40.983 INFO:tasks.workunit.client.0.smithi163.stderr: 17950K .......... .......... .......... .......... .......... 10% 18.9M 9s 2023-07-02T14:02:40.986 INFO:tasks.workunit.client.0.smithi163.stderr: 18000K .......... .......... .......... .......... .......... 10% 14.7M 9s 2023-07-02T14:02:40.988 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:02:40.989 INFO:tasks.workunit.client.0.smithi163.stderr: 18050K .......... .......... .......... .......... .......... 10% 27.7M 9s 2023-07-02T14:02:40.991 INFO:tasks.workunit.client.0.smithi163.stderr: 18100K .......... .......... .......... .......... .......... 10% 15.4M 9s 2023-07-02T14:02:40.994 INFO:tasks.workunit.client.0.smithi163.stderr: 18150K .......... .......... .......... .......... .......... 10% 23.9M 9s 2023-07-02T14:02:40.996 INFO:tasks.workunit.client.0.smithi163.stderr: 18200K .......... .......... .......... .......... .......... 10% 17.2M 9s 2023-07-02T14:02:40.999 INFO:tasks.workunit.client.0.smithi163.stderr: 18250K .......... .......... .......... .......... .......... 10% 19.4M 9s 2023-07-02T14:02:41.000 INFO:tasks.workunit.client.0.smithi163.stderr: 18300K .......... .......... .......... .......... .......... 10% 40.6M 9s 2023-07-02T14:02:41.003 INFO:tasks.workunit.client.0.smithi163.stderr: 18350K .......... .......... .......... .......... .......... 10% 14.2M 9s 2023-07-02T14:02:41.006 INFO:tasks.workunit.client.0.smithi163.stderr: 18400K .......... .......... .......... .......... .......... 10% 20.7M 9s 2023-07-02T14:02:41.008 INFO:tasks.workunit.client.0.smithi163.stderr: 18450K .......... .......... .......... .......... .......... 10% 38.2M 9s 2023-07-02T14:02:41.010 INFO:tasks.workunit.client.0.smithi163.stderr: 18500K .......... .......... .......... .......... .......... 11% 16.1M 9s 2023-07-02T14:02:41.012 INFO:tasks.workunit.client.0.smithi163.stderr: 18550K .......... .......... .......... .......... .......... 11% 26.7M 9s 2023-07-02T14:02:41.014 INFO:tasks.workunit.client.0.smithi163.stderr: 18600K .......... .......... .......... .......... .......... 11% 23.4M 9s 2023-07-02T14:02:41.016 INFO:tasks.workunit.client.0.smithi163.stderr: 18650K .......... .......... .......... .......... .......... 11% 21.8M 9s 2023-07-02T14:02:41.018 INFO:tasks.workunit.client.0.smithi163.stderr: 18700K .......... .......... .......... .......... .......... 11% 22.4M 9s 2023-07-02T14:02:41.021 INFO:tasks.workunit.client.0.smithi163.stderr: 18750K .......... .......... .......... .......... .......... 11% 16.6M 9s 2023-07-02T14:02:41.023 INFO:tasks.workunit.client.0.smithi163.stderr: 18800K .......... .......... .......... .......... .......... 11% 22.8M 9s 2023-07-02T14:02:41.025 INFO:tasks.workunit.client.0.smithi163.stderr: 18850K .......... .......... .......... .......... .......... 11% 33.7M 9s 2023-07-02T14:02:41.027 INFO:tasks.workunit.client.0.smithi163.stderr: 18900K .......... .......... .......... .......... .......... 11% 20.3M 9s 2023-07-02T14:02:41.029 INFO:tasks.workunit.client.0.smithi163.stderr: 18950K .......... .......... .......... .......... .......... 11% 26.8M 9s 2023-07-02T14:02:41.032 INFO:tasks.workunit.client.0.smithi163.stderr: 19000K .......... .......... .......... .......... .......... 11% 18.7M 9s 2023-07-02T14:02:41.033 INFO:tasks.workunit.client.0.smithi163.stderr: 19050K .......... .......... .......... .......... .......... 11% 50.9M 9s 2023-07-02T14:02:41.036 INFO:tasks.workunit.client.0.smithi163.stderr: 19100K .......... .......... .......... .......... .......... 11% 16.1M 9s 2023-07-02T14:02:41.039 INFO:tasks.workunit.client.0.smithi163.stderr: 19150K .......... .......... .......... .......... .......... 11% 19.2M 9s 2023-07-02T14:02:41.040 INFO:tasks.workunit.client.0.smithi163.stderr: 19200K .......... .......... .......... .......... .......... 11% 24.7M 9s 2023-07-02T14:02:41.042 INFO:tasks.workunit.client.0.smithi163.stderr: 19250K .......... .......... .......... .......... .......... 11% 23.8M 9s 2023-07-02T14:02:41.044 INFO:tasks.workunit.client.0.smithi163.stderr: 19300K .......... .......... .......... .......... .......... 11% 28.3M 9s 2023-07-02T14:02:41.047 INFO:tasks.workunit.client.0.smithi163.stderr: 19350K .......... .......... .......... .......... .......... 11% 16.1M 9s 2023-07-02T14:02:41.049 INFO:tasks.workunit.client.0.smithi163.stderr: 19400K .......... .......... .......... .......... .......... 11% 26.9M 9s 2023-07-02T14:02:41.051 INFO:tasks.workunit.client.0.smithi163.stderr: 19450K .......... .......... .......... .......... .......... 11% 18.3M 9s 2023-07-02T14:02:41.054 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:41.053+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:41.055 INFO:tasks.workunit.client.0.smithi163.stderr: 19500K .......... .......... .......... .......... .......... 11% 14.2M 9s 2023-07-02T14:02:41.058 INFO:tasks.workunit.client.0.smithi163.stderr: 19550K .......... .......... .......... .......... .......... 11% 13.8M 9s 2023-07-02T14:02:41.061 INFO:tasks.workunit.client.0.smithi163.stderr: 19600K .......... .......... .......... .......... .......... 11% 16.9M 9s 2023-07-02T14:02:41.065 INFO:tasks.workunit.client.0.smithi163.stderr: 19650K .......... .......... .......... .......... .......... 11% 13.6M 9s 2023-07-02T14:02:41.066 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:02:41.067 INFO:tasks.workunit.client.0.smithi163.stderr: 19700K .......... .......... .......... .......... .......... 11% 20.3M 9s 2023-07-02T14:02:41.070 INFO:tasks.workunit.client.0.smithi163.stderr: 19750K .......... .......... .......... .......... .......... 11% 16.4M 9s 2023-07-02T14:02:41.073 INFO:tasks.workunit.client.0.smithi163.stderr: 19800K .......... .......... .......... .......... .......... 11% 17.6M 9s 2023-07-02T14:02:41.075 INFO:tasks.workunit.client.0.smithi163.stderr: 19850K .......... .......... .......... .......... .......... 11% 21.1M 9s 2023-07-02T14:02:41.079 INFO:tasks.workunit.client.0.smithi163.stderr: 19900K .......... .......... .......... .......... .......... 11% 15.2M 9s 2023-07-02T14:02:41.081 INFO:tasks.workunit.client.0.smithi163.stderr: 19950K .......... .......... .......... .......... .......... 11% 18.4M 9s 2023-07-02T14:02:41.086 INFO:tasks.workunit.client.0.smithi163.stderr: 20000K .......... .......... .......... .......... .......... 11% 12.2M 9s 2023-07-02T14:02:41.088 INFO:tasks.workunit.client.0.smithi163.stderr: 20050K .......... .......... .......... .......... .......... 11% 25.6M 9s 2023-07-02T14:02:41.112 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:41.112 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:41.122 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:02:41.155 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:41.154+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:41.172 INFO:tasks.workunit.client.0.smithi163.stderr: 20100K .......... .......... .......... .......... .......... 11% 605K 9s 2023-07-02T14:02:41.172 INFO:tasks.workunit.client.0.smithi163.stderr: 20150K .......... .......... .......... .......... .......... 11% 288M 9s 2023-07-02T14:02:41.179 INFO:tasks.workunit.client.0.smithi163.stderr: 20200K .......... .......... .......... .......... .......... 12% 10.7M 9s 2023-07-02T14:02:41.179 INFO:tasks.workunit.client.0.smithi163.stderr: 20250K .......... .......... .......... .......... .......... 12% 308M 9s 2023-07-02T14:02:41.180 INFO:tasks.workunit.client.0.smithi163.stderr: 20300K .......... .......... .......... .......... .......... 12% 362M 9s 2023-07-02T14:02:41.188 INFO:tasks.workunit.client.0.smithi163.stderr: 20350K .......... .......... .......... .......... .......... 12% 8.18M 9s 2023-07-02T14:02:41.189 INFO:tasks.workunit.client.0.smithi163.stderr: 20400K .......... .......... .......... .......... .......... 12% 338M 9s 2023-07-02T14:02:41.189 INFO:tasks.workunit.client.0.smithi163.stderr: 20450K .......... .......... .......... .......... .......... 12% 379M 9s 2023-07-02T14:02:41.189 INFO:tasks.workunit.client.0.smithi163.stderr: 20500K .......... .......... .......... .......... .......... 12% 402M 9s 2023-07-02T14:02:41.189 INFO:tasks.workunit.client.0.smithi163.stderr: 20550K .......... .......... .......... .......... .......... 12% 351M 9s 2023-07-02T14:02:41.190 INFO:tasks.workunit.client.0.smithi163.stderr: 20600K .......... .......... .......... .......... .......... 12% 389M 9s 2023-07-02T14:02:41.243 INFO:tasks.workunit.client.0.smithi163.stderr: 20650K .......... .......... .......... .......... .......... 12% 5.99M 9s 2023-07-02T14:02:41.243 INFO:tasks.workunit.client.0.smithi163.stderr: 20700K .......... .......... .......... .......... .......... 12% 312M 9s 2023-07-02T14:02:41.244 INFO:tasks.workunit.client.0.smithi163.stderr: 20750K .......... .......... .......... .......... .......... 12% 281M 9s 2023-07-02T14:02:41.244 INFO:tasks.workunit.client.0.smithi163.stderr: 20800K .......... .......... .......... .......... .......... 12% 394M 9s 2023-07-02T14:02:41.244 INFO:tasks.workunit.client.0.smithi163.stderr: 20850K .......... .......... .......... .......... .......... 12% 337M 9s 2023-07-02T14:02:41.244 INFO:tasks.workunit.client.0.smithi163.stderr: 20900K .......... .......... .......... .......... .......... 12% 376M 9s 2023-07-02T14:02:41.244 INFO:tasks.workunit.client.0.smithi163.stderr: 20950K .......... .......... .......... .......... .......... 12% 317M 9s 2023-07-02T14:02:41.244 INFO:tasks.workunit.client.0.smithi163.stderr: 21000K .......... .......... .......... .......... .......... 12% 393M 9s 2023-07-02T14:02:41.244 INFO:tasks.workunit.client.0.smithi163.stderr: 21050K .......... .......... .......... .......... .......... 12% 324M 9s 2023-07-02T14:02:41.245 INFO:tasks.workunit.client.0.smithi163.stderr: 21100K .......... .......... .......... .......... .......... 12% 4.56M 9s 2023-07-02T14:02:41.245 INFO:tasks.workunit.client.0.smithi163.stderr: 21150K .......... .......... .......... .......... .......... 12% 198M 9s 2023-07-02T14:02:41.245 INFO:tasks.workunit.client.0.smithi163.stderr: 21200K .......... .......... .......... .......... .......... 12% 369M 9s 2023-07-02T14:02:41.245 INFO:tasks.workunit.client.0.smithi163.stderr: 21250K .......... .......... .......... .......... .......... 12% 362M 9s 2023-07-02T14:02:41.245 INFO:tasks.workunit.client.0.smithi163.stderr: 21300K .......... .......... .......... .......... .......... 12% 359M 9s 2023-07-02T14:02:41.245 INFO:tasks.workunit.client.0.smithi163.stderr: 21350K .......... .......... .......... .......... .......... 12% 326M 9s 2023-07-02T14:02:41.245 INFO:tasks.workunit.client.0.smithi163.stderr: 21400K .......... .......... .......... .......... .......... 12% 346M 9s 2023-07-02T14:02:41.245 INFO:tasks.workunit.client.0.smithi163.stderr: 21450K .......... .......... .......... .......... .......... 12% 351M 9s 2023-07-02T14:02:41.246 INFO:tasks.workunit.client.0.smithi163.stderr: 21500K .......... .......... .......... .......... .......... 12% 333M 9s 2023-07-02T14:02:41.246 INFO:tasks.workunit.client.0.smithi163.stderr: 21550K .......... .......... .......... .......... .......... 12% 300M 9s 2023-07-02T14:02:41.246 INFO:tasks.workunit.client.0.smithi163.stderr: 21600K .......... .......... .......... .......... .......... 12% 343M 9s 2023-07-02T14:02:41.246 INFO:tasks.workunit.client.0.smithi163.stderr: 21650K .......... .......... .......... .......... .......... 12% 334M 9s 2023-07-02T14:02:41.246 INFO:tasks.workunit.client.0.smithi163.stderr: 21700K .......... .......... .......... .......... .......... 12% 367M 9s 2023-07-02T14:02:41.246 INFO:tasks.workunit.client.0.smithi163.stderr: 21750K .......... .......... .......... .......... .......... 12% 307M 9s 2023-07-02T14:02:41.246 INFO:tasks.workunit.client.0.smithi163.stderr: 21800K .......... .......... .......... .......... .......... 12% 357M 9s 2023-07-02T14:02:41.247 INFO:tasks.workunit.client.0.smithi163.stderr: 21850K .......... .......... .......... .......... .......... 12% 351M 9s 2023-07-02T14:02:41.247 INFO:tasks.workunit.client.0.smithi163.stderr: 21900K .......... .......... .......... .......... .......... 13% 369M 9s 2023-07-02T14:02:41.247 INFO:tasks.workunit.client.0.smithi163.stderr: 21950K .......... .......... .......... .......... .......... 13% 291M 9s 2023-07-02T14:02:41.247 INFO:tasks.workunit.client.0.smithi163.stderr: 22000K .......... .......... .......... .......... .......... 13% 383M 9s 2023-07-02T14:02:41.247 INFO:tasks.workunit.client.0.smithi163.stderr: 22050K .......... .......... .......... .......... .......... 13% 393M 9s 2023-07-02T14:02:41.247 INFO:tasks.workunit.client.0.smithi163.stderr: 22100K .......... .......... .......... .......... .......... 13% 396M 9s 2023-07-02T14:02:41.247 INFO:tasks.workunit.client.0.smithi163.stderr: 22150K .......... .......... .......... .......... .......... 13% 307M 9s 2023-07-02T14:02:41.247 INFO:tasks.workunit.client.0.smithi163.stderr: 22200K .......... .......... .......... .......... .......... 13% 368M 9s 2023-07-02T14:02:41.248 INFO:tasks.workunit.client.0.smithi163.stderr: 22250K .......... .......... .......... .......... .......... 13% 383M 9s 2023-07-02T14:02:41.248 INFO:tasks.workunit.client.0.smithi163.stderr: 22300K .......... .......... .......... .......... .......... 13% 339M 9s 2023-07-02T14:02:41.248 INFO:tasks.workunit.client.0.smithi163.stderr: 22350K .......... .......... .......... .......... .......... 13% 337M 9s 2023-07-02T14:02:41.248 INFO:tasks.workunit.client.0.smithi163.stderr: 22400K .......... .......... .......... .......... .......... 13% 403M 9s 2023-07-02T14:02:41.248 INFO:tasks.workunit.client.0.smithi163.stderr: 22450K .......... .......... .......... .......... .......... 13% 32.6M 9s 2023-07-02T14:02:41.248 INFO:tasks.workunit.client.0.smithi163.stderr: 22500K .......... .......... .......... .......... .......... 13% 18.4M 9s 2023-07-02T14:02:41.248 INFO:tasks.workunit.client.0.smithi163.stderr: 22550K .......... .......... .......... .......... .......... 13% 47.1M 9s 2023-07-02T14:02:41.248 INFO:tasks.workunit.client.0.smithi163.stderr: 22600K .......... .......... .......... .......... .......... 13% 18.1M 9s 2023-07-02T14:02:41.249 INFO:tasks.workunit.client.0.smithi163.stderr: 22650K .......... .......... .......... .......... .......... 13% 17.2M 9s 2023-07-02T14:02:41.249 INFO:tasks.workunit.client.0.smithi163.stderr: 22700K .......... .......... .......... .......... .......... 13% 25.7M 9s 2023-07-02T14:02:41.249 INFO:tasks.workunit.client.0.smithi163.stderr: 22750K .......... .......... .......... .......... .......... 13% 14.6M 9s 2023-07-02T14:02:41.249 INFO:tasks.workunit.client.0.smithi163.stderr: 22800K .......... .......... .......... .......... .......... 13% 20.8M 9s 2023-07-02T14:02:41.249 INFO:tasks.workunit.client.0.smithi163.stderr: 22850K .......... .......... .......... .......... .......... 13% 24.6M 9s 2023-07-02T14:02:41.249 INFO:tasks.workunit.client.0.smithi163.stderr: 22900K .......... .......... .......... .......... .......... 13% 17.6M 9s 2023-07-02T14:02:41.249 INFO:tasks.workunit.client.0.smithi163.stderr: 22950K .......... .......... .......... .......... .......... 13% 15.6M 9s 2023-07-02T14:02:41.249 INFO:tasks.workunit.client.0.smithi163.stderr: 23000K .......... .......... .......... .......... .......... 13% 48.9M 8s 2023-07-02T14:02:41.250 INFO:tasks.workunit.client.0.smithi163.stderr: 23050K .......... .......... .......... .......... .......... 13% 14.6M 8s 2023-07-02T14:02:41.250 INFO:tasks.workunit.client.0.smithi163.stderr: 23100K .......... .......... .......... .......... .......... 13% 34.4M 8s 2023-07-02T14:02:41.253 INFO:tasks.workunit.client.0.smithi163.stderr: 23150K .......... .......... .......... .......... .......... 13% 7.23M 8s 2023-07-02T14:02:41.253 INFO:tasks.workunit.client.0.smithi163.stderr: 23200K .......... .......... .......... .......... .......... 13% 20.9M 8s 2023-07-02T14:02:41.253 INFO:tasks.workunit.client.0.smithi163.stderr: 23250K .......... .......... .......... .......... .......... 13% 25.2M 8s 2023-07-02T14:02:41.253 INFO:tasks.workunit.client.0.smithi163.stderr: 23300K .......... .......... .......... .......... .......... 13% 26.2M 8s 2023-07-02T14:02:41.253 INFO:tasks.workunit.client.0.smithi163.stderr: 23350K .......... .......... .......... .......... .......... 13% 21.5M 8s 2023-07-02T14:02:41.255 INFO:tasks.workunit.client.0.smithi163.stderr: 23400K .......... .......... .......... .......... .......... 13% 16.8M 8s 2023-07-02T14:02:41.256 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:41.254+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:41.257 INFO:tasks.workunit.client.0.smithi163.stderr: 23450K .......... .......... .......... .......... .......... 13% 33.7M 8s 2023-07-02T14:02:41.259 INFO:tasks.workunit.client.0.smithi163.stderr: 23500K .......... .......... .......... .......... .......... 13% 23.2M 8s 2023-07-02T14:02:41.262 INFO:tasks.workunit.client.0.smithi163.stderr: 23550K .......... .......... .......... .......... .......... 14% 16.7M 8s 2023-07-02T14:02:41.263 INFO:tasks.workunit.client.0.smithi163.stderr: 23600K .......... .......... .......... .......... .......... 14% 29.6M 8s 2023-07-02T14:02:41.265 INFO:tasks.workunit.client.0.smithi163.stderr: 23650K .......... .......... .......... .......... .......... 14% 25.0M 8s 2023-07-02T14:02:41.268 INFO:tasks.workunit.client.0.smithi163.stderr: 23700K .......... .......... .......... .......... .......... 14% 21.7M 8s 2023-07-02T14:02:41.270 INFO:tasks.workunit.client.0.smithi163.stderr: 23750K .......... .......... .......... .......... .......... 14% 20.9M 8s 2023-07-02T14:02:41.271 INFO:tasks.workunit.client.0.smithi163.stderr: 23800K .......... .......... .......... .......... .......... 14% 30.5M 8s 2023-07-02T14:02:41.274 INFO:tasks.workunit.client.0.smithi163.stderr: 23850K .......... .......... .......... .......... .......... 14% 21.8M 8s 2023-07-02T14:02:41.276 INFO:tasks.workunit.client.0.smithi163.stderr: 23900K .......... .......... .......... .......... .......... 14% 19.1M 8s 2023-07-02T14:02:41.279 INFO:tasks.workunit.client.0.smithi163.stderr: 23950K .......... .......... .......... .......... .......... 14% 15.7M 8s 2023-07-02T14:02:41.281 INFO:tasks.workunit.client.0.smithi163.stderr: 24000K .......... .......... .......... .......... .......... 14% 30.5M 8s 2023-07-02T14:02:41.285 INFO:tasks.workunit.client.0.smithi163.stderr: 24050K .......... .......... .......... .......... .......... 14% 12.9M 8s 2023-07-02T14:02:41.286 INFO:tasks.workunit.client.0.smithi163.stderr: 24100K .......... .......... .......... .......... .......... 14% 37.4M 8s 2023-07-02T14:02:41.287 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:41.287 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:41.289 INFO:tasks.workunit.client.0.smithi163.stderr: 24150K .......... .......... .......... .......... .......... 14% 19.3M 8s 2023-07-02T14:02:41.291 INFO:tasks.workunit.client.0.smithi163.stderr: 24200K .......... .......... .......... .......... .......... 14% 18.1M 8s 2023-07-02T14:02:41.294 INFO:tasks.workunit.client.0.smithi163.stderr: 24250K .......... .......... .......... .......... .......... 14% 18.4M 8s 2023-07-02T14:02:41.295 INFO:tasks.workunit.client.0.smithi163.stderr: 24300K .......... .......... .......... .......... .......... 14% 38.8M 8s 2023-07-02T14:02:41.297 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:02:41.299 INFO:tasks.workunit.client.0.smithi163.stderr: 24350K .......... .......... .......... .......... .......... 14% 11.8M 8s 2023-07-02T14:02:41.302 INFO:tasks.workunit.client.0.smithi163.stderr: 24400K .......... .......... .......... .......... .......... 14% 18.0M 8s 2023-07-02T14:02:41.305 INFO:tasks.workunit.client.0.smithi163.stderr: 24450K .......... .......... .......... .......... .......... 14% 14.8M 8s 2023-07-02T14:02:41.308 INFO:tasks.workunit.client.0.smithi163.stderr: 24500K .......... .......... .......... .......... .......... 14% 21.1M 8s 2023-07-02T14:02:41.309 INFO:tasks.workunit.client.0.smithi163.stderr: 24550K .......... .......... .......... .......... .......... 14% 37.5M 8s 2023-07-02T14:02:41.313 INFO:tasks.workunit.client.0.smithi163.stderr: 24600K .......... .......... .......... .......... .......... 14% 13.0M 8s 2023-07-02T14:02:41.316 INFO:tasks.workunit.client.0.smithi163.stderr: 24650K .......... .......... .......... .......... .......... 14% 16.5M 8s 2023-07-02T14:02:41.318 INFO:tasks.workunit.client.0.smithi163.stderr: 24700K .......... .......... .......... .......... .......... 14% 22.3M 8s 2023-07-02T14:02:41.320 INFO:tasks.workunit.client.0.smithi163.stderr: 24750K .......... .......... .......... .......... .......... 14% 20.5M 8s 2023-07-02T14:02:41.324 INFO:tasks.workunit.client.0.smithi163.stderr: 24800K .......... .......... .......... .......... .......... 14% 14.8M 8s 2023-07-02T14:02:41.325 INFO:tasks.workunit.client.0.smithi163.stderr: 24850K .......... .......... .......... .......... .......... 14% 40.5M 8s 2023-07-02T14:02:41.327 INFO:tasks.workunit.client.0.smithi163.stderr: 24900K .......... .......... .......... .......... .......... 14% 19.3M 8s 2023-07-02T14:02:41.331 INFO:tasks.workunit.client.0.smithi163.stderr: 24950K .......... .......... .......... .......... .......... 14% 16.3M 8s 2023-07-02T14:02:41.331 INFO:tasks.workunit.client.0.smithi163.stderr: 25000K .......... .......... .......... .......... .......... 14% 41.4M 8s 2023-07-02T14:02:41.333 INFO:tasks.workunit.client.0.smithi163.stderr: 25050K .......... .......... .......... .......... .......... 14% 37.1M 8s 2023-07-02T14:02:41.336 INFO:tasks.workunit.client.0.smithi163.stderr: 25100K .......... .......... .......... .......... .......... 14% 18.8M 8s 2023-07-02T14:02:41.338 INFO:tasks.workunit.client.0.smithi163.stderr: 25150K .......... .......... .......... .......... .......... 14% 17.7M 8s 2023-07-02T14:02:41.341 INFO:tasks.workunit.client.0.smithi163.stderr: 25200K .......... .......... .......... .......... .......... 14% 19.1M 8s 2023-07-02T14:02:41.343 INFO:tasks.workunit.client.0.smithi163.stderr: 25250K .......... .......... .......... .......... .......... 15% 23.9M 8s 2023-07-02T14:02:41.345 INFO:tasks.workunit.client.0.smithi163.stderr: 25300K .......... .......... .......... .......... .......... 15% 21.2M 8s 2023-07-02T14:02:41.347 INFO:tasks.workunit.client.0.smithi163.stderr: 25350K .......... .......... .......... .......... .......... 15% 27.6M 8s 2023-07-02T14:02:41.349 INFO:tasks.workunit.client.0.smithi163.stderr: 25400K .......... .......... .......... .......... .......... 15% 20.5M 8s 2023-07-02T14:02:41.351 INFO:tasks.workunit.client.0.smithi163.stderr: 25450K .......... .......... .......... .......... .......... 15% 26.7M 8s 2023-07-02T14:02:41.354 INFO:tasks.workunit.client.0.smithi163.stderr: 25500K .......... .......... .......... .......... .......... 15% 18.2M 8s 2023-07-02T14:02:41.355 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:41.355+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:41.357 INFO:tasks.workunit.client.0.smithi163.stderr: 25550K .......... .......... .......... .......... .......... 15% 14.0M 8s 2023-07-02T14:02:41.359 INFO:tasks.workunit.client.0.smithi163.stderr: 25600K .......... .......... .......... .......... .......... 15% 21.5M 8s 2023-07-02T14:02:41.363 INFO:tasks.workunit.client.0.smithi163.stderr: 25650K .......... .......... .......... .......... .......... 15% 14.5M 8s 2023-07-02T14:02:41.365 INFO:tasks.workunit.client.0.smithi163.stderr: 25700K .......... .......... .......... .......... .......... 15% 25.8M 8s 2023-07-02T14:02:41.367 INFO:tasks.workunit.client.0.smithi163.stderr: 25750K .......... .......... .......... .......... .......... 15% 22.4M 8s 2023-07-02T14:02:41.370 INFO:tasks.workunit.client.0.smithi163.stderr: 25800K .......... .......... .......... .......... .......... 15% 14.2M 8s 2023-07-02T14:02:41.373 INFO:tasks.workunit.client.0.smithi163.stderr: 25850K .......... .......... .......... .......... .......... 15% 19.7M 8s 2023-07-02T14:02:41.375 INFO:tasks.workunit.client.0.smithi163.stderr: 25900K .......... .......... .......... .......... .......... 15% 21.8M 8s 2023-07-02T14:02:41.378 INFO:tasks.workunit.client.0.smithi163.stderr: 25950K .......... .......... .......... .......... .......... 15% 16.0M 8s 2023-07-02T14:02:41.381 INFO:tasks.workunit.client.0.smithi163.stderr: 26000K .......... .......... .......... .......... .......... 15% 17.3M 8s 2023-07-02T14:02:41.382 INFO:tasks.workunit.client.0.smithi163.stderr: 26050K .......... .......... .......... .......... .......... 15% 38.4M 8s 2023-07-02T14:02:41.384 INFO:tasks.workunit.client.0.smithi163.stderr: 26100K .......... .......... .......... .......... .......... 15% 21.7M 8s 2023-07-02T14:02:41.387 INFO:tasks.workunit.client.0.smithi163.stderr: 26150K .......... .......... .......... .......... .......... 15% 18.0M 8s 2023-07-02T14:02:41.390 INFO:tasks.workunit.client.0.smithi163.stderr: 26200K .......... .......... .......... .......... .......... 15% 16.9M 8s 2023-07-02T14:02:41.392 INFO:tasks.workunit.client.0.smithi163.stderr: 26250K .......... .......... .......... .......... .......... 15% 29.4M 8s 2023-07-02T14:02:41.394 INFO:tasks.workunit.client.0.smithi163.stderr: 26300K .......... .......... .......... .......... .......... 15% 22.4M 8s 2023-07-02T14:02:41.397 INFO:tasks.workunit.client.0.smithi163.stderr: 26350K .......... .......... .......... .......... .......... 15% 14.4M 8s 2023-07-02T14:02:41.399 INFO:tasks.workunit.client.0.smithi163.stderr: 26400K .......... .......... .......... .......... .......... 15% 24.1M 8s 2023-07-02T14:02:41.401 INFO:tasks.workunit.client.0.smithi163.stderr: 26450K .......... .......... .......... .......... .......... 15% 24.8M 8s 2023-07-02T14:02:41.403 INFO:tasks.workunit.client.0.smithi163.stderr: 26500K .......... .......... .......... .......... .......... 15% 25.4M 8s 2023-07-02T14:02:41.406 INFO:tasks.workunit.client.0.smithi163.stderr: 26550K .......... .......... .......... .......... .......... 15% 17.4M 8s 2023-07-02T14:02:41.407 INFO:tasks.workunit.client.0.smithi163.stderr: 26600K .......... .......... .......... .......... .......... 15% 38.4M 8s 2023-07-02T14:02:41.410 INFO:tasks.workunit.client.0.smithi163.stderr: 26650K .......... .......... .......... .......... .......... 15% 20.7M 8s 2023-07-02T14:02:41.413 INFO:tasks.workunit.client.0.smithi163.stderr: 26700K .......... .......... .......... .......... .......... 15% 15.0M 8s 2023-07-02T14:02:41.415 INFO:tasks.workunit.client.0.smithi163.stderr: 26750K .......... .......... .......... .......... .......... 15% 20.7M 8s 2023-07-02T14:02:41.417 INFO:tasks.workunit.client.0.smithi163.stderr: 26800K .......... .......... .......... .......... .......... 15% 24.1M 8s 2023-07-02T14:02:41.419 INFO:tasks.workunit.client.0.smithi163.stderr: 26850K .......... .......... .......... .......... .......... 15% 29.5M 8s 2023-07-02T14:02:41.420 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:41.420 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:41.421 INFO:tasks.workunit.client.0.smithi163.stderr: 26900K .......... .......... .......... .......... .......... 15% 19.0M 8s 2023-07-02T14:02:41.424 INFO:tasks.workunit.client.0.smithi163.stderr: 26950K .......... .......... .......... .......... .......... 16% 22.5M 8s 2023-07-02T14:02:41.425 INFO:tasks.workunit.client.0.smithi163.stderr: 27000K .......... .......... .......... .......... .......... 16% 28.0M 8s 2023-07-02T14:02:41.428 INFO:tasks.workunit.client.0.smithi163.stderr: 27050K .......... .......... .......... .......... .......... 16% 17.0M 8s 2023-07-02T14:02:41.430 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:02:41.432 INFO:tasks.workunit.client.0.smithi163.stderr: 27100K .......... .......... .......... .......... .......... 16% 21.0M 8s 2023-07-02T14:02:41.434 INFO:tasks.workunit.client.0.smithi163.stderr: 27150K .......... .......... .......... .......... .......... 16% 16.2M 8s 2023-07-02T14:02:41.436 INFO:tasks.workunit.client.0.smithi163.stderr: 27200K .......... .......... .......... .......... .......... 16% 21.9M 8s 2023-07-02T14:02:41.440 INFO:tasks.workunit.client.0.smithi163.stderr: 27250K .......... .......... .......... .......... .......... 16% 13.4M 8s 2023-07-02T14:02:41.441 INFO:tasks.workunit.client.0.smithi163.stderr: 27300K .......... .......... .......... .......... .......... 16% 31.5M 8s 2023-07-02T14:02:41.443 INFO:tasks.workunit.client.0.smithi163.stderr: 27350K .......... .......... .......... .......... .......... 16% 25.8M 8s 2023-07-02T14:02:41.445 INFO:tasks.workunit.client.0.smithi163.stderr: 27400K .......... .......... .......... .......... .......... 16% 22.6M 8s 2023-07-02T14:02:41.448 INFO:tasks.workunit.client.0.smithi163.stderr: 27450K .......... .......... .......... .......... .......... 16% 17.1M 8s 2023-07-02T14:02:41.451 INFO:tasks.workunit.client.0.smithi163.stderr: 27500K .......... .......... .......... .......... .......... 16% 16.6M 8s 2023-07-02T14:02:41.453 INFO:tasks.workunit.client.0.smithi163.stderr: 27550K .......... .......... .......... .......... .......... 16% 22.1M 8s 2023-07-02T14:02:41.456 INFO:tasks.workunit.client.0.smithi163.stderr: 27600K .......... .......... .......... .......... .......... 16% 15.0M 8s 2023-07-02T14:02:41.457 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:41.455+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:41.458 INFO:tasks.workunit.client.0.smithi163.stderr: 27650K .......... .......... .......... .......... .......... 16% 39.8M 8s 2023-07-02T14:02:41.460 INFO:tasks.workunit.client.0.smithi163.stderr: 27700K .......... .......... .......... .......... .......... 16% 21.4M 8s 2023-07-02T14:02:41.462 INFO:tasks.workunit.client.0.smithi163.stderr: 27750K .......... .......... .......... .......... .......... 16% 22.3M 8s 2023-07-02T14:02:41.464 INFO:tasks.workunit.client.0.smithi163.stderr: 27800K .......... .......... .......... .......... .......... 16% 20.9M 8s 2023-07-02T14:02:41.467 INFO:tasks.workunit.client.0.smithi163.stderr: 27850K .......... .......... .......... .......... .......... 16% 20.8M 8s 2023-07-02T14:02:41.470 INFO:tasks.workunit.client.0.smithi163.stderr: 27900K .......... .......... .......... .......... .......... 16% 18.2M 8s 2023-07-02T14:02:41.472 INFO:tasks.workunit.client.0.smithi163.stderr: 27950K .......... .......... .......... .......... .......... 16% 16.0M 8s 2023-07-02T14:02:41.475 INFO:tasks.workunit.client.0.smithi163.stderr: 28000K .......... .......... .......... .......... .......... 16% 21.7M 8s 2023-07-02T14:02:41.477 INFO:tasks.workunit.client.0.smithi163.stderr: 28050K .......... .......... .......... .......... .......... 16% 25.9M 8s 2023-07-02T14:02:41.480 INFO:tasks.workunit.client.0.smithi163.stderr: 28100K .......... .......... .......... .......... .......... 16% 17.1M 8s 2023-07-02T14:02:41.481 INFO:tasks.workunit.client.0.smithi163.stderr: 28150K .......... .......... .......... .......... .......... 16% 24.5M 8s 2023-07-02T14:02:41.483 INFO:tasks.workunit.client.0.smithi163.stderr: 28200K .......... .......... .......... .......... .......... 16% 25.3M 8s 2023-07-02T14:02:41.485 INFO:tasks.workunit.client.0.smithi163.stderr: 28250K .......... .......... .......... .......... .......... 16% 28.1M 8s 2023-07-02T14:02:41.488 INFO:tasks.workunit.client.0.smithi163.stderr: 28300K .......... .......... .......... .......... .......... 16% 20.0M 8s 2023-07-02T14:02:41.489 INFO:tasks.workunit.client.0.smithi163.stderr: 28350K .......... .......... .......... .......... .......... 16% 25.7M 8s 2023-07-02T14:02:41.492 INFO:tasks.workunit.client.0.smithi163.stderr: 28400K .......... .......... .......... .......... .......... 16% 17.2M 8s 2023-07-02T14:02:41.494 INFO:tasks.workunit.client.0.smithi163.stderr: 28450K .......... .......... .......... .......... .......... 16% 23.2M 8s 2023-07-02T14:02:41.497 INFO:tasks.workunit.client.0.smithi163.stderr: 28500K .......... .......... .......... .......... .......... 16% 19.8M 8s 2023-07-02T14:02:41.499 INFO:tasks.workunit.client.0.smithi163.stderr: 28550K .......... .......... .......... .......... .......... 16% 21.8M 8s 2023-07-02T14:02:41.501 INFO:tasks.workunit.client.0.smithi163.stderr: 28600K .......... .......... .......... .......... .......... 16% 19.8M 8s 2023-07-02T14:02:41.503 INFO:tasks.workunit.client.0.smithi163.stderr: 28650K .......... .......... .......... .......... .......... 17% 27.5M 8s 2023-07-02T14:02:41.505 INFO:tasks.workunit.client.0.smithi163.stderr: 28700K .......... .......... .......... .......... .......... 17% 26.0M 8s 2023-07-02T14:02:41.509 INFO:tasks.workunit.client.0.smithi163.stderr: 28750K .......... .......... .......... .......... .......... 17% 12.6M 8s 2023-07-02T14:02:41.511 INFO:tasks.workunit.client.0.smithi163.stderr: 28800K .......... .......... .......... .......... .......... 17% 28.2M 8s 2023-07-02T14:02:41.513 INFO:tasks.workunit.client.0.smithi163.stderr: 28850K .......... .......... .......... .......... .......... 17% 24.7M 8s 2023-07-02T14:02:41.515 INFO:tasks.workunit.client.0.smithi163.stderr: 28900K .......... .......... .......... .......... .......... 17% 25.8M 8s 2023-07-02T14:02:41.518 INFO:tasks.workunit.client.0.smithi163.stderr: 28950K .......... .......... .......... .......... .......... 17% 16.0M 8s 2023-07-02T14:02:41.520 INFO:tasks.workunit.client.0.smithi163.stderr: 29000K .......... .......... .......... .......... .......... 17% 24.9M 8s 2023-07-02T14:02:41.522 INFO:tasks.workunit.client.0.smithi163.stderr: 29050K .......... .......... .......... .......... .......... 17% 21.4M 8s 2023-07-02T14:02:41.524 INFO:tasks.workunit.client.0.smithi163.stderr: 29100K .......... .......... .......... .......... .......... 17% 26.1M 8s 2023-07-02T14:02:41.528 INFO:tasks.workunit.client.0.smithi163.stderr: 29150K .......... .......... .......... .......... .......... 17% 13.0M 8s 2023-07-02T14:02:41.530 INFO:tasks.workunit.client.0.smithi163.stderr: 29200K .......... .......... .......... .......... .......... 17% 24.4M 8s 2023-07-02T14:02:41.533 INFO:tasks.workunit.client.0.smithi163.stderr: 29250K .......... .......... .......... .......... .......... 17% 16.7M 8s 2023-07-02T14:02:41.535 INFO:tasks.workunit.client.0.smithi163.stderr: 29300K .......... .......... .......... .......... .......... 17% 21.3M 8s 2023-07-02T14:02:41.537 INFO:tasks.workunit.client.0.smithi163.stderr: 29350K .......... .......... .......... .......... .......... 17% 24.3M 8s 2023-07-02T14:02:41.539 INFO:tasks.workunit.client.0.smithi163.stderr: 29400K .......... .......... .......... .......... .......... 17% 22.9M 8s 2023-07-02T14:02:41.540 INFO:tasks.workunit.client.0.smithi163.stderr: 29450K .......... .......... .......... .......... .......... 17% 29.5M 8s 2023-07-02T14:02:41.543 INFO:tasks.workunit.client.0.smithi163.stderr: 29500K .......... .......... .......... .......... .......... 17% 18.3M 8s 2023-07-02T14:02:41.547 INFO:tasks.workunit.client.0.smithi163.stderr: 29550K .......... .......... .......... .......... .......... 17% 16.6M 8s 2023-07-02T14:02:41.548 INFO:tasks.workunit.client.0.smithi163.stderr: 29600K .......... .......... .......... .......... .......... 17% 31.4M 8s 2023-07-02T14:02:41.550 INFO:tasks.workunit.client.0.smithi163.stderr: 29650K .......... .......... .......... .......... .......... 17% 21.8M 8s 2023-07-02T14:02:41.553 INFO:tasks.workunit.client.0.smithi163.stderr: 29700K .......... .......... .......... .......... .......... 17% 18.8M 8s 2023-07-02T14:02:41.555 INFO:tasks.workunit.client.0.smithi163.stderr: 29750K .......... .......... .......... .......... .......... 17% 25.6M 8s 2023-07-02T14:02:41.557 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:41.556+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:41.557 INFO:tasks.workunit.client.0.smithi163.stderr: 29800K .......... .......... .......... .......... .......... 17% 18.0M 8s 2023-07-02T14:02:41.560 INFO:tasks.workunit.client.0.smithi163.stderr: 29850K .......... .......... .......... .......... .......... 17% 20.2M 8s 2023-07-02T14:02:41.561 INFO:tasks.workunit.client.0.smithi163.stderr: 29900K .......... .......... .......... .......... .......... 17% 33.7M 8s 2023-07-02T14:02:41.564 INFO:tasks.workunit.client.0.smithi163.stderr: 29950K .......... .......... .......... .......... .......... 17% 15.6M 8s 2023-07-02T14:02:41.567 INFO:tasks.workunit.client.0.smithi163.stderr: 30000K .......... .......... .......... .......... .......... 17% 22.1M 8s 2023-07-02T14:02:41.568 INFO:tasks.workunit.client.0.smithi163.stderr: 30050K .......... .......... .......... .......... .......... 17% 30.0M 8s 2023-07-02T14:02:41.571 INFO:tasks.workunit.client.0.smithi163.stderr: 30100K .......... .......... .......... .......... .......... 17% 19.2M 8s 2023-07-02T14:02:41.572 INFO:tasks.workunit.client.0.smithi163.stderr: 30150K .......... .......... .......... .......... .......... 17% 31.0M 8s 2023-07-02T14:02:41.575 INFO:tasks.workunit.client.0.smithi163.stderr: 30200K .......... .......... .......... .......... .......... 17% 16.4M 8s 2023-07-02T14:02:41.578 INFO:tasks.workunit.client.0.smithi163.stderr: 30250K .......... .......... .......... .......... .......... 17% 20.5M 8s 2023-07-02T14:02:41.579 INFO:tasks.workunit.client.0.smithi163.stderr: 30300K .......... .......... .......... .......... .......... 18% 26.9M 8s 2023-07-02T14:02:41.583 INFO:tasks.workunit.client.0.smithi163.stderr: 30350K .......... .......... .......... .......... .......... 18% 15.8M 8s 2023-07-02T14:02:41.589 INFO:tasks.workunit.client.0.smithi163.stderr: 30400K .......... .......... .......... .......... .......... 18% 16.7M 8s 2023-07-02T14:02:41.589 INFO:tasks.workunit.client.0.smithi163.stderr: 30450K .......... .......... .......... .......... .......... 18% 36.3M 8s 2023-07-02T14:02:41.589 INFO:tasks.workunit.client.0.smithi163.stderr: 30500K .......... .......... .......... .......... .......... 18% 24.4M 8s 2023-07-02T14:02:41.591 INFO:tasks.workunit.client.0.smithi163.stderr: 30550K .......... .......... .......... .......... .......... 18% 22.8M 8s 2023-07-02T14:02:41.594 INFO:tasks.workunit.client.0.smithi163.stderr: 30600K .......... .......... .......... .......... .......... 18% 16.3M 8s 2023-07-02T14:02:41.595 INFO:tasks.workunit.client.0.smithi163.stderr: 30650K .......... .......... .......... .......... .......... 18% 42.7M 8s 2023-07-02T14:02:41.595 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:41.595 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:41.598 INFO:tasks.workunit.client.0.smithi163.stderr: 30700K .......... .......... .......... .......... .......... 18% 20.6M 8s 2023-07-02T14:02:41.601 INFO:tasks.workunit.client.0.smithi163.stderr: 30750K .......... .......... .......... .......... .......... 18% 17.3M 8s 2023-07-02T14:02:41.603 INFO:tasks.workunit.client.0.smithi163.stderr: 30800K .......... .......... .......... .......... .......... 18% 17.4M 8s 2023-07-02T14:02:41.605 INFO:tasks.workunit.client.0.smithi163.stderr: 30850K .......... .......... .......... .......... .......... 18% 33.3M 8s 2023-07-02T14:02:41.606 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:02:41.607 INFO:tasks.workunit.client.0.smithi163.stderr: 30900K .......... .......... .......... .......... .......... 18% 18.3M 8s 2023-07-02T14:02:41.609 INFO:tasks.workunit.client.0.smithi163.stderr: 30950K .......... .......... .......... .......... .......... 18% 28.7M 8s 2023-07-02T14:02:41.611 INFO:tasks.workunit.client.0.smithi163.stderr: 31000K .......... .......... .......... .......... .......... 18% 18.2M 8s 2023-07-02T14:02:41.614 INFO:tasks.workunit.client.0.smithi163.stderr: 31050K .......... .......... .......... .......... .......... 18% 23.2M 8s 2023-07-02T14:02:41.618 INFO:tasks.workunit.client.0.smithi163.stderr: 31100K .......... .......... .......... .......... .......... 18% 12.1M 8s 2023-07-02T14:02:41.621 INFO:tasks.workunit.client.0.smithi163.stderr: 31150K .......... .......... .......... .......... .......... 18% 13.2M 8s 2023-07-02T14:02:41.624 INFO:tasks.workunit.client.0.smithi163.stderr: 31200K .......... .......... .......... .......... .......... 18% 19.5M 8s 2023-07-02T14:02:41.628 INFO:tasks.workunit.client.0.smithi163.stderr: 31250K .......... .......... .......... .......... .......... 18% 12.9M 8s 2023-07-02T14:02:41.630 INFO:tasks.workunit.client.0.smithi163.stderr: 31300K .......... .......... .......... .......... .......... 18% 27.2M 8s 2023-07-02T14:02:41.635 INFO:tasks.workunit.client.0.smithi163.stderr: 31350K .......... .......... .......... .......... .......... 18% 10.4M 8s 2023-07-02T14:02:41.635 INFO:tasks.workunit.client.0.smithi163.stderr: 31400K .......... .......... .......... .......... .......... 18% 111M 8s 2023-07-02T14:02:41.636 INFO:tasks.workunit.client.0.smithi163.stderr: 31450K .......... .......... .......... .......... .......... 18% 31.8M 8s 2023-07-02T14:02:41.638 INFO:tasks.workunit.client.0.smithi163.stderr: 31500K .......... .......... .......... .......... .......... 18% 24.9M 8s 2023-07-02T14:02:41.642 INFO:tasks.workunit.client.0.smithi163.stderr: 31550K .......... .......... .......... .......... .......... 18% 13.8M 8s 2023-07-02T14:02:41.644 INFO:tasks.workunit.client.0.smithi163.stderr: 31600K .......... .......... .......... .......... .......... 18% 24.9M 8s 2023-07-02T14:02:41.646 INFO:tasks.workunit.client.0.smithi163.stderr: 31650K .......... .......... .......... .......... .......... 18% 23.5M 8s 2023-07-02T14:02:41.647 INFO:tasks.workunit.client.0.smithi163.stderr: 31700K .......... .......... .......... .......... .......... 18% 33.5M 8s 2023-07-02T14:02:41.649 INFO:tasks.workunit.client.0.smithi163.stderr: 31750K .......... .......... .......... .......... .......... 18% 22.5M 8s 2023-07-02T14:02:41.650 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:02:41.653 INFO:tasks.workunit.client.0.smithi163.stderr: 31800K .......... .......... .......... .......... .......... 18% 14.1M 8s 2023-07-02T14:02:41.654 INFO:tasks.workunit.client.0.smithi163.stderr: 31850K .......... .......... .......... .......... .......... 18% 31.0M 8s 2023-07-02T14:02:41.656 INFO:tasks.workunit.client.0.smithi163.stderr: 31900K .......... .......... .......... .......... .......... 18% 25.6M 8s 2023-07-02T14:02:41.660 INFO:tasks.workunit.client.0.smithi163.stderr: 31950K .......... .......... .......... .......... .......... 18% 13.4M 8s 2023-07-02T14:02:41.661 INFO:tasks.workunit.client.0.smithi163.stderr: 32000K .......... .......... .......... .......... .......... 19% 40.8M 8s 2023-07-02T14:02:41.663 INFO:tasks.workunit.client.0.smithi163.stderr: 32050K .......... .......... .......... .......... .......... 19% 20.3M 8s 2023-07-02T14:02:41.666 INFO:tasks.workunit.client.0.smithi163.stderr: 32100K .......... .......... .......... .......... .......... 19% 16.2M 8s 2023-07-02T14:02:41.669 INFO:tasks.workunit.client.0.smithi163.stderr: 32150K .......... .......... .......... .......... .......... 19% 23.1M 8s 2023-07-02T14:02:41.670 INFO:tasks.workunit.client.0.smithi163.stderr: 32200K .......... .......... .......... .......... .......... 19% 39.9M 8s 2023-07-02T14:02:41.672 INFO:tasks.workunit.client.0.smithi163.stderr: 32250K .......... .......... .......... .......... .......... 19% 21.1M 8s 2023-07-02T14:02:41.674 INFO:tasks.workunit.client.0.smithi163.stderr: 32300K .......... .......... .......... .......... .......... 19% 28.0M 8s 2023-07-02T14:02:41.679 INFO:tasks.workunit.client.0.smithi163.stderr: 32350K .......... .......... .......... .......... .......... 19% 10.9M 8s 2023-07-02T14:02:41.680 INFO:tasks.workunit.client.0.smithi163.stderr: 32400K .......... .......... .......... .......... .......... 19% 33.1M 8s 2023-07-02T14:02:41.682 INFO:tasks.workunit.client.0.smithi163.stderr: 32450K .......... .......... .......... .......... .......... 19% 24.2M 8s 2023-07-02T14:02:41.686 INFO:tasks.workunit.client.0.smithi163.stderr: 32500K .......... .......... .......... .......... .......... 19% 14.6M 8s 2023-07-02T14:02:41.688 INFO:tasks.workunit.client.0.smithi163.stderr: 32550K .......... .......... .......... .......... .......... 19% 17.1M 8s 2023-07-02T14:02:41.691 INFO:tasks.workunit.client.0.smithi163.stderr: 32600K .......... .......... .......... .......... .......... 19% 17.0M 8s 2023-07-02T14:02:41.693 INFO:tasks.workunit.client.0.smithi163.stderr: 32650K .......... .......... .......... .......... .......... 19% 19.5M 8s 2023-07-02T14:02:41.694 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:41.693+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:41.696 INFO:tasks.workunit.client.0.smithi163.stderr: 32700K .......... .......... .......... .......... .......... 19% 21.1M 7s 2023-07-02T14:02:41.699 INFO:tasks.workunit.client.0.smithi163.stderr: 32750K .......... .......... .......... .......... .......... 19% 14.1M 7s 2023-07-02T14:02:41.702 INFO:tasks.workunit.client.0.smithi163.stderr: 32800K .......... .......... .......... .......... .......... 19% 16.3M 7s 2023-07-02T14:02:41.705 INFO:tasks.workunit.client.0.smithi163.stderr: 32850K .......... .......... .......... .......... .......... 19% 17.7M 7s 2023-07-02T14:02:41.707 INFO:tasks.workunit.client.0.smithi163.stderr: 32900K .......... .......... .......... .......... .......... 19% 30.8M 7s 2023-07-02T14:02:41.708 INFO:tasks.workunit.client.0.smithi163.stderr: 32950K .......... .......... .......... .......... .......... 19% 24.4M 7s 2023-07-02T14:02:41.711 INFO:tasks.workunit.client.0.smithi163.stderr: 33000K .......... .......... .......... .......... .......... 19% 25.7M 7s 2023-07-02T14:02:41.713 INFO:tasks.workunit.client.0.smithi163.stderr: 33050K .......... .......... .......... .......... .......... 19% 19.6M 7s 2023-07-02T14:02:41.715 INFO:tasks.workunit.client.0.smithi163.stderr: 33100K .......... .......... .......... .......... .......... 19% 20.4M 7s 2023-07-02T14:02:41.718 INFO:tasks.workunit.client.0.smithi163.stderr: 33150K .......... .......... .......... .......... .......... 19% 18.5M 7s 2023-07-02T14:02:41.721 INFO:tasks.workunit.client.0.smithi163.stderr: 33200K .......... .......... .......... .......... .......... 19% 18.9M 7s 2023-07-02T14:02:41.722 INFO:tasks.workunit.client.0.smithi163.stderr: 33250K .......... .......... .......... .......... .......... 19% 29.1M 7s 2023-07-02T14:02:41.724 INFO:tasks.workunit.client.0.smithi163.stderr: 33300K .......... .......... .......... .......... .......... 19% 24.8M 7s 2023-07-02T14:02:41.727 INFO:tasks.workunit.client.0.smithi163.stderr: 33350K .......... .......... .......... .......... .......... 19% 14.7M 7s 2023-07-02T14:02:41.730 INFO:tasks.workunit.client.0.smithi163.stderr: 33400K .......... .......... .......... .......... .......... 19% 21.7M 7s 2023-07-02T14:02:41.731 INFO:tasks.workunit.client.0.smithi163.stderr: 33450K .......... .......... .......... .......... .......... 19% 55.7M 7s 2023-07-02T14:02:41.731 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:41.731 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:41.734 INFO:tasks.workunit.client.0.smithi163.stderr: 33500K .......... .......... .......... .......... .......... 19% 16.6M 7s 2023-07-02T14:02:41.736 INFO:tasks.workunit.client.0.smithi163.stderr: 33550K .......... .......... .......... .......... .......... 19% 20.4M 7s 2023-07-02T14:02:41.739 INFO:tasks.workunit.client.0.smithi163.stderr: 33600K .......... .......... .......... .......... .......... 19% 16.9M 7s 2023-07-02T14:02:41.741 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:02:41.742 INFO:tasks.workunit.client.0.smithi163.stderr: 33650K .......... .......... .......... .......... .......... 19% 21.4M 7s 2023-07-02T14:02:41.744 INFO:tasks.workunit.client.0.smithi163.stderr: 33700K .......... .......... .......... .......... .......... 20% 17.3M 7s 2023-07-02T14:02:41.745 INFO:tasks.workunit.client.0.smithi163.stderr: 33750K .......... .......... .......... .......... .......... 20% 31.4M 7s 2023-07-02T14:02:41.748 INFO:tasks.workunit.client.0.smithi163.stderr: 33800K .......... .......... .......... .......... .......... 20% 22.5M 7s 2023-07-02T14:02:41.750 INFO:tasks.workunit.client.0.smithi163.stderr: 33850K .......... .......... .......... .......... .......... 20% 23.7M 7s 2023-07-02T14:02:41.753 INFO:tasks.workunit.client.0.smithi163.stderr: 33900K .......... .......... .......... .......... .......... 20% 16.9M 7s 2023-07-02T14:02:41.756 INFO:tasks.workunit.client.0.smithi163.stderr: 33950K .......... .......... .......... .......... .......... 20% 16.2M 7s 2023-07-02T14:02:41.758 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:41.757+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:41.758 INFO:tasks.workunit.client.0.smithi163.stderr: 34000K .......... .......... .......... .......... .......... 20% 19.3M 7s 2023-07-02T14:02:41.762 INFO:tasks.workunit.client.0.smithi163.stderr: 34050K .......... .......... .......... .......... .......... 20% 15.4M 7s 2023-07-02T14:02:41.764 INFO:tasks.workunit.client.0.smithi163.stderr: 34100K .......... .......... .......... .......... .......... 20% 16.2M 7s 2023-07-02T14:02:41.766 INFO:tasks.workunit.client.0.smithi163.stderr: 34150K .......... .......... .......... .......... .......... 20% 23.2M 7s 2023-07-02T14:02:41.768 INFO:tasks.workunit.client.0.smithi163.stderr: 34200K .......... .......... .......... .......... .......... 20% 31.0M 7s 2023-07-02T14:02:41.771 INFO:tasks.workunit.client.0.smithi163.stderr: 34250K .......... .......... .......... .......... .......... 20% 17.1M 7s 2023-07-02T14:02:41.773 INFO:tasks.workunit.client.0.smithi163.stderr: 34300K .......... .......... .......... .......... .......... 20% 21.3M 7s 2023-07-02T14:02:41.776 INFO:tasks.workunit.client.0.smithi163.stderr: 34350K .......... .......... .......... .......... .......... 20% 14.8M 7s 2023-07-02T14:02:41.779 INFO:tasks.workunit.client.0.smithi163.stderr: 34400K .......... .......... .......... .......... .......... 20% 19.1M 7s 2023-07-02T14:02:41.782 INFO:tasks.workunit.client.0.smithi163.stderr: 34450K .......... .......... .......... .......... .......... 20% 18.6M 7s 2023-07-02T14:02:41.783 INFO:tasks.workunit.client.0.smithi163.stderr: 34500K .......... .......... .......... .......... .......... 20% 31.3M 7s 2023-07-02T14:02:41.786 INFO:tasks.workunit.client.0.smithi163.stderr: 34550K .......... .......... .......... .......... .......... 20% 20.5M 7s 2023-07-02T14:02:41.787 INFO:tasks.workunit.client.0.smithi163.stderr: 34600K .......... .......... .......... .......... .......... 20% 30.0M 7s 2023-07-02T14:02:41.790 INFO:tasks.workunit.client.0.smithi163.stderr: 34650K .......... .......... .......... .......... .......... 20% 19.4M 7s 2023-07-02T14:02:41.792 INFO:tasks.workunit.client.0.smithi163.stderr: 34700K .......... .......... .......... .......... .......... 20% 22.4M 7s 2023-07-02T14:02:41.795 INFO:tasks.workunit.client.0.smithi163.stderr: 34750K .......... .......... .......... .......... .......... 20% 16.3M 7s 2023-07-02T14:02:41.797 INFO:tasks.workunit.client.0.smithi163.stderr: 34800K .......... .......... .......... .......... .......... 20% 21.2M 7s 2023-07-02T14:02:41.799 INFO:tasks.workunit.client.0.smithi163.stderr: 34850K .......... .......... .......... .......... .......... 20% 29.9M 7s 2023-07-02T14:02:41.801 INFO:tasks.workunit.client.0.smithi163.stderr: 34900K .......... .......... .......... .......... .......... 20% 24.9M 7s 2023-07-02T14:02:41.802 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:41.804 INFO:tasks.workunit.client.0.smithi163.stderr: 34950K .......... .......... .......... .......... .......... 20% 15.5M 7s 2023-07-02T14:02:41.806 INFO:tasks.workunit.client.0.smithi163.stderr: 35000K .......... .......... .......... .......... .......... 20% 19.8M 7s 2023-07-02T14:02:41.808 INFO:tasks.workunit.client.0.smithi163.stderr: 35050K .......... .......... .......... .......... .......... 20% 24.7M 7s 2023-07-02T14:02:41.810 INFO:tasks.workunit.client.0.smithi163.stderr: 35100K .......... .......... .......... .......... .......... 20% 26.6M 7s 2023-07-02T14:02:41.813 INFO:tasks.workunit.client.0.smithi163.stderr: 35150K .......... .......... .......... .......... .......... 20% 15.9M 7s 2023-07-02T14:02:41.815 INFO:tasks.workunit.client.0.smithi163.stderr: 35200K .......... .......... .......... .......... .......... 20% 25.0M 7s 2023-07-02T14:02:41.817 INFO:tasks.workunit.client.0.smithi163.stderr: 35250K .......... .......... .......... .......... .......... 20% 23.2M 7s 2023-07-02T14:02:41.819 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:41.821 INFO:tasks.workunit.client.0.smithi163.stderr: 35300K .......... .......... .......... .......... .......... 20% 14.0M 7s 2023-07-02T14:02:41.823 INFO:tasks.workunit.client.0.smithi163.stderr: 35350K .......... .......... .......... .......... .......... 21% 23.5M 7s 2023-07-02T14:02:41.825 INFO:tasks.workunit.client.0.smithi163.stderr: 35400K .......... .......... .......... .......... .......... 21% 22.1M 7s 2023-07-02T14:02:41.828 INFO:tasks.workunit.client.0.smithi163.stderr: 35450K .......... .......... .......... .......... .......... 21% 17.3M 7s 2023-07-02T14:02:41.830 INFO:tasks.workunit.client.0.smithi163.stderr: 35500K .......... .......... .......... .......... .......... 21% 21.4M 7s 2023-07-02T14:02:41.833 INFO:tasks.workunit.client.0.smithi163.stderr: 35550K .......... .......... .......... .......... .......... 21% 16.4M 7s 2023-07-02T14:02:41.836 INFO:tasks.workunit.client.0.smithi163.stderr: 35600K .......... .......... .......... .......... .......... 21% 15.6M 7s 2023-07-02T14:02:41.837 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:41.838 INFO:tasks.workunit.client.0.smithi163.stderr: 35650K .......... .......... .......... .......... .......... 21% 30.1M 7s 2023-07-02T14:02:41.841 INFO:tasks.workunit.client.0.smithi163.stderr: 35700K .......... .......... .......... .......... .......... 21% 16.1M 7s 2023-07-02T14:02:41.843 INFO:tasks.workunit.client.0.smithi163.stderr: 35750K .......... .......... .......... .......... .......... 21% 20.8M 7s 2023-07-02T14:02:41.846 INFO:tasks.workunit.client.0.smithi163.stderr: 35800K .......... .......... .......... .......... .......... 21% 20.0M 7s 2023-07-02T14:02:41.848 INFO:tasks.workunit.client.0.smithi163.stderr: 35850K .......... .......... .......... .......... .......... 21% 22.1M 7s 2023-07-02T14:02:41.850 INFO:tasks.workunit.client.0.smithi163.stderr: 35900K .......... .......... .......... .......... .......... 21% 20.0M 7s 2023-07-02T14:02:41.854 INFO:tasks.workunit.client.0.smithi163.stderr: 35950K .......... .......... .......... .......... .......... 21% 15.3M 7s 2023-07-02T14:02:41.855 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:41.856 INFO:tasks.workunit.client.0.smithi163.stderr: 36000K .......... .......... .......... .......... .......... 21% 18.7M 7s 2023-07-02T14:02:41.859 INFO:tasks.workunit.client.0.smithi163.stderr: 36050K .......... .......... .......... .......... .......... 21% 22.4M 7s 2023-07-02T14:02:41.859 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:41.858+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:41.860 INFO:tasks.workunit.client.0.smithi163.stderr: 36100K .......... .......... .......... .......... .......... 21% 25.2M 7s 2023-07-02T14:02:41.862 INFO:tasks.workunit.client.0.smithi163.stderr: 36150K .......... .......... .......... .......... .......... 21% 24.2M 7s 2023-07-02T14:02:41.865 INFO:tasks.workunit.client.0.smithi163.stderr: 36200K .......... .......... .......... .......... .......... 21% 22.0M 7s 2023-07-02T14:02:41.865 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:41.866 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:41.867 INFO:tasks.workunit.client.0.smithi163.stderr: 36250K .......... .......... .......... .......... .......... 21% 19.4M 7s 2023-07-02T14:02:41.869 INFO:tasks.workunit.client.0.smithi163.stderr: 36300K .......... .......... .......... .......... .......... 21% 24.0M 7s 2023-07-02T14:02:41.872 INFO:tasks.workunit.client.0.smithi163.stderr: 36350K .......... .......... .......... .......... .......... 21% 18.0M 7s 2023-07-02T14:02:41.873 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:41.875 INFO:tasks.workunit.client.0.smithi163.stderr: 36400K .......... .......... .......... .......... .......... 21% 18.9M 7s 2023-07-02T14:02:41.876 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:02:41.877 INFO:tasks.workunit.client.0.smithi163.stderr: 36450K .......... .......... .......... .......... .......... 21% 21.9M 7s 2023-07-02T14:02:41.879 INFO:tasks.workunit.client.0.smithi163.stderr: 36500K .......... .......... .......... .......... .......... 21% 18.7M 7s 2023-07-02T14:02:41.880 INFO:tasks.workunit.client.0.smithi163.stderr: 36550K .......... .......... .......... .......... .......... 21% 53.3M 7s 2023-07-02T14:02:41.883 INFO:tasks.workunit.client.0.smithi163.stderr: 36600K .......... .......... .......... .......... .......... 21% 16.5M 7s 2023-07-02T14:02:41.886 INFO:tasks.workunit.client.0.smithi163.stderr: 36650K .......... .......... .......... .......... .......... 21% 19.4M 7s 2023-07-02T14:02:41.888 INFO:tasks.workunit.client.0.smithi163.stderr: 36700K .......... .......... .......... .......... .......... 21% 19.0M 7s 2023-07-02T14:02:41.890 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:41.891 INFO:tasks.workunit.client.0.smithi163.stderr: 36750K .......... .......... .......... .......... .......... 21% 18.2M 7s 2023-07-02T14:02:41.892 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:02:41.892 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:02:41.892 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:02:41.892 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:02:41.892 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:02:41.892 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:02:41.893 INFO:tasks.workunit.client.0.smithi163.stderr: 36800K .......... .......... .......... .......... .......... 21% 29.6M 7s 2023-07-02T14:02:41.897 INFO:tasks.workunit.client.0.smithi163.stderr: 36850K .......... .......... .......... .......... .......... 21% 11.2M 7s 2023-07-02T14:02:41.899 INFO:tasks.workunit.client.0.smithi163.stderr: 36900K .......... .......... .......... .......... .......... 21% 33.9M 7s 2023-07-02T14:02:41.901 INFO:tasks.workunit.client.0.smithi163.stderr: 36950K .......... .......... .......... .......... .......... 21% 22.4M 7s 2023-07-02T14:02:41.902 INFO:tasks.workunit.client.0.smithi163.stderr: 37000K .......... .......... .......... .......... .......... 21% 31.1M 7s 2023-07-02T14:02:41.905 INFO:tasks.workunit.client.0.smithi163.stderr: 37050K .......... .......... .......... .......... .......... 22% 19.7M 7s 2023-07-02T14:02:41.907 INFO:tasks.workunit.client.0.smithi163.stderr: 37100K .......... .......... .......... .......... .......... 22% 23.2M 7s 2023-07-02T14:02:41.911 INFO:tasks.workunit.client.0.smithi163.stderr: 37150K .......... .......... .......... .......... .......... 22% 10.7M 7s 2023-07-02T14:02:41.913 INFO:tasks.workunit.client.0.smithi163.stderr: 37200K .......... .......... .......... .......... .......... 22% 26.4M 7s 2023-07-02T14:02:41.917 INFO:tasks.workunit.client.0.smithi163.stderr: 37250K .......... .......... .......... .......... .......... 22% 13.2M 7s 2023-07-02T14:02:41.918 INFO:tasks.workunit.client.0.smithi163.stderr: 37300K .......... .......... .......... .......... .......... 22% 53.8M 7s 2023-07-02T14:02:41.921 INFO:tasks.workunit.client.0.smithi163.stderr: 37350K .......... .......... .......... .......... .......... 22% 17.0M 7s 2023-07-02T14:02:41.924 INFO:tasks.workunit.client.0.smithi163.stderr: 37400K .......... .......... .......... .......... .......... 22% 16.5M 7s 2023-07-02T14:02:41.925 INFO:tasks.workunit.client.0.smithi163.stderr: 37450K .......... .......... .......... .......... .......... 22% 36.9M 7s 2023-07-02T14:02:41.927 INFO:tasks.workunit.client.0.smithi163.stderr: 37500K .......... .......... .......... .......... .......... 22% 21.8M 7s 2023-07-02T14:02:41.931 INFO:tasks.workunit.client.0.smithi163.stderr: 37550K .......... .......... .......... .......... .......... 22% 14.1M 7s 2023-07-02T14:02:41.933 INFO:tasks.workunit.client.0.smithi163.stderr: 37600K .......... .......... .......... .......... .......... 22% 21.4M 7s 2023-07-02T14:02:41.936 INFO:tasks.workunit.client.0.smithi163.stderr: 37650K .......... .......... .......... .......... .......... 22% 16.7M 7s 2023-07-02T14:02:41.938 INFO:tasks.workunit.client.0.smithi163.stderr: 37700K .......... .......... .......... .......... .......... 22% 27.4M 7s 2023-07-02T14:02:41.940 INFO:tasks.workunit.client.0.smithi163.stderr: 37750K .......... .......... .......... .......... .......... 22% 19.3M 7s 2023-07-02T14:02:41.941 INFO:tasks.workunit.client.0.smithi163.stderr: 37800K .......... .......... .......... .......... .......... 22% 37.7M 7s 2023-07-02T14:02:41.944 INFO:tasks.workunit.client.0.smithi163.stderr: 37850K .......... .......... .......... .......... .......... 22% 15.8M 7s 2023-07-02T14:02:41.946 INFO:tasks.workunit.client.0.smithi163.stderr: 37900K .......... .......... .......... .......... .......... 22% 29.4M 7s 2023-07-02T14:02:41.950 INFO:tasks.workunit.client.0.smithi163.stderr: 37950K .......... .......... .......... .......... .......... 22% 13.2M 7s 2023-07-02T14:02:41.951 INFO:tasks.workunit.client.0.smithi163.stderr: 38000K .......... .......... .......... .......... .......... 22% 30.0M 7s 2023-07-02T14:02:41.953 INFO:tasks.workunit.client.0.smithi163.stderr: 38050K .......... .......... .......... .......... .......... 22% 25.1M 7s 2023-07-02T14:02:41.956 INFO:tasks.workunit.client.0.smithi163.stderr: 38100K .......... .......... .......... .......... .......... 22% 15.5M 7s 2023-07-02T14:02:41.959 INFO:tasks.workunit.client.0.smithi163.stderr: 38150K .......... .......... .......... .......... .......... 22% 20.3M 7s 2023-07-02T14:02:41.960 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:41.958+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:41.961 INFO:tasks.workunit.client.0.smithi163.stderr: 38200K .......... .......... .......... .......... .......... 22% 38.4M 7s 2023-07-02T14:02:41.964 INFO:tasks.workunit.client.0.smithi163.stderr: 38250K .......... .......... .......... .......... .......... 22% 14.2M 7s 2023-07-02T14:02:41.965 INFO:tasks.workunit.client.0.smithi163.stderr: 38300K .......... .......... .......... .......... .......... 22% 34.6M 7s 2023-07-02T14:02:41.968 INFO:tasks.workunit.client.0.smithi163.stderr: 38350K .......... .......... .......... .......... .......... 22% 14.9M 7s 2023-07-02T14:02:41.970 INFO:tasks.workunit.client.0.smithi163.stderr: 38400K .......... .......... .......... .......... .......... 22% 25.7M 7s 2023-07-02T14:02:41.973 INFO:tasks.workunit.client.0.smithi163.stderr: 38450K .......... .......... .......... .......... .......... 22% 16.2M 7s 2023-07-02T14:02:41.975 INFO:tasks.workunit.client.0.smithi163.stderr: 38500K .......... .......... .......... .......... .......... 22% 21.2M 7s 2023-07-02T14:02:41.978 INFO:tasks.workunit.client.0.smithi163.stderr: 38550K .......... .......... .......... .......... .......... 22% 20.9M 7s 2023-07-02T14:02:41.981 INFO:tasks.workunit.client.0.smithi163.stderr: 38600K .......... .......... .......... .......... .......... 22% 21.8M 7s 2023-07-02T14:02:41.983 INFO:tasks.workunit.client.0.smithi163.stderr: 38650K .......... .......... .......... .......... .......... 22% 16.4M 7s 2023-07-02T14:02:41.985 INFO:tasks.workunit.client.0.smithi163.stderr: 38700K .......... .......... .......... .......... .......... 22% 28.7M 7s 2023-07-02T14:02:41.988 INFO:tasks.workunit.client.0.smithi163.stderr: 38750K .......... .......... .......... .......... .......... 23% 13.9M 7s 2023-07-02T14:02:41.992 INFO:tasks.workunit.client.0.smithi163.stderr: 38800K .......... .......... .......... .......... .......... 23% 14.8M 7s 2023-07-02T14:02:41.994 INFO:tasks.workunit.client.0.smithi163.stderr: 38850K .......... .......... .......... .......... .......... 23% 19.7M 7s 2023-07-02T14:02:41.996 INFO:tasks.workunit.client.0.smithi163.stderr: 38900K .......... .......... .......... .......... .......... 23% 35.1M 7s 2023-07-02T14:02:41.998 INFO:tasks.workunit.client.0.smithi163.stderr: 38950K .......... .......... .......... .......... .......... 23% 20.9M 7s 2023-07-02T14:02:42.001 INFO:tasks.workunit.client.0.smithi163.stderr: 39000K .......... .......... .......... .......... .......... 23% 17.0M 7s 2023-07-02T14:02:42.002 INFO:tasks.workunit.client.0.smithi163.stderr: 39050K .......... .......... .......... .......... .......... 23% 36.0M 7s 2023-07-02T14:02:42.005 INFO:tasks.workunit.client.0.smithi163.stderr: 39100K .......... .......... .......... .......... .......... 23% 18.7M 7s 2023-07-02T14:02:42.008 INFO:tasks.workunit.client.0.smithi163.stderr: 39150K .......... .......... .......... .......... .......... 23% 15.1M 7s 2023-07-02T14:02:42.011 INFO:tasks.workunit.client.0.smithi163.stderr: 39200K .......... .......... .......... .......... .......... 23% 18.3M 7s 2023-07-02T14:02:42.012 INFO:tasks.workunit.client.0.smithi163.stderr: 39250K .......... .......... .......... .......... .......... 23% 30.3M 7s 2023-07-02T14:02:42.014 INFO:tasks.workunit.client.0.smithi163.stderr: 39300K .......... .......... .......... .......... .......... 23% 30.3M 7s 2023-07-02T14:02:42.016 INFO:tasks.workunit.client.0.smithi163.stderr: 39350K .......... .......... .......... .......... .......... 23% 24.5M 7s 2023-07-02T14:02:42.018 INFO:tasks.workunit.client.0.smithi163.stderr: 39400K .......... .......... .......... .......... .......... 23% 19.3M 7s 2023-07-02T14:02:42.021 INFO:tasks.workunit.client.0.smithi163.stderr: 39450K .......... .......... .......... .......... .......... 23% 16.2M 7s 2023-07-02T14:02:42.024 INFO:tasks.workunit.client.0.smithi163.stderr: 39500K .......... .......... .......... .......... .......... 23% 20.0M 7s 2023-07-02T14:02:42.027 INFO:tasks.workunit.client.0.smithi163.stderr: 39550K .......... .......... .......... .......... .......... 23% 16.5M 7s 2023-07-02T14:02:42.028 INFO:tasks.workunit.client.0.smithi163.stderr: 39600K .......... .......... .......... .......... .......... 23% 35.2M 7s 2023-07-02T14:02:42.031 INFO:tasks.workunit.client.0.smithi163.stderr: 39650K .......... .......... .......... .......... .......... 23% 20.0M 7s 2023-07-02T14:02:42.034 INFO:tasks.workunit.client.0.smithi163.stderr: 39700K .......... .......... .......... .......... .......... 23% 15.6M 7s 2023-07-02T14:02:42.036 INFO:tasks.workunit.client.0.smithi163.stderr: 39750K .......... .......... .......... .......... .......... 23% 19.7M 7s 2023-07-02T14:02:42.037 INFO:tasks.workunit.client.0.smithi163.stderr: 39800K .......... .......... .......... .......... .......... 23% 35.3M 7s 2023-07-02T14:02:42.040 INFO:tasks.workunit.client.0.smithi163.stderr: 39850K .......... .......... .......... .......... .......... 23% 21.1M 7s 2023-07-02T14:02:42.042 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:42.042 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:42.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:42.042 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:02:42.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.042 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:42.042 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:02:42.043 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.7486690989999998, 2023-07-02T14:02:42.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:02:42.043 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:42.043 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.043 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:42.043 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:42.043 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:42.044 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:02:42.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.044 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:42.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:42.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:02:42.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:42.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:42.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:02:42.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:42.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:02:42.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:02:42.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:42.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:02:42.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:02:42.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:42.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:02:42.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:02:42.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:02:42.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:02:42.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:42.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:02:42.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:02:42.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:42.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:02:42.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:02:42.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:42.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:02:42.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:02:42.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:42.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:02:42.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:02:42.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:42.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:02:42.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:02:42.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:42.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:02:42.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:02:42.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:42.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:02:42.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:02:42.052 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.052 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.053 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.053 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:42.053 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:02:42.053 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.7484340760000001, 2023-07-02T14:02:42.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:02:42.053 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:42.054 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.054 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:42.054 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:42.054 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:42.054 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:02:42.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.054 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:42.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:42.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:02:42.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:42.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:42.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:02:42.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:42.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:02:42.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:02:42.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:42.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:02:42.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:02:42.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:42.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:02:42.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:02:42.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:02:42.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:02:42.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:42.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:02:42.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:02:42.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:42.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:02:42.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:02:42.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:42.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:02:42.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:02:42.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:42.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:02:42.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:02:42.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:42.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:02:42.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:02:42.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:42.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:02:42.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:02:42.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:42.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:02:42.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:02:42.063 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.063 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.063 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.064 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:42.064 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:02:42.064 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.7482349419999998, 2023-07-02T14:02:42.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:02:42.064 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:42.064 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.065 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:42.065 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:42.065 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:42.065 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:02:42.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.065 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:42.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:42.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:02:42.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:42.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:42.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:02:42.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:42.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:02:42.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:02:42.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:42.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:02:42.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:02:42.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:42.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:02:42.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:02:42.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:02:42.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:02:42.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:42.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:02:42.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:02:42.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:42.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:02:42.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:02:42.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:42.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:02:42.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:02:42.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:42.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:02:42.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:02:42.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:42.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:02:42.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:02:42.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:42.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:02:42.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:02:42.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:42.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:02:42.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:02:42.074 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.074 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.074 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.074 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:42.074 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:02:42.075 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.7478698179999999, 2023-07-02T14:02:42.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:02:42.075 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:42.075 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.075 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:42.075 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:42.075 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:42.075 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:02:42.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.076 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:42.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:42.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:02:42.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:42.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:42.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:02:42.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:42.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:42.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:02:42.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:02:42.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:42.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:02:42.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:02:42.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:42.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:02:42.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:02:42.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:02:42.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:02:42.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:42.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:02:42.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:02:42.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:42.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:02:42.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:02:42.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:42.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:02:42.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:02:42.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:42.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:02:42.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:02:42.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:42.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:02:42.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:02:42.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:42.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:02:42.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:02:42.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:42.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:02:42.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:02:42.084 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.084 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.084 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.085 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:42.085 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:02:42.085 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.7477887569999999, 2023-07-02T14:02:42.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:02:42.085 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:42.085 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.085 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:42.086 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:42.086 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:42.086 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:02:42.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.086 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:42.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:42.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:02:42.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:42.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:42.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:02:42.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:42.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:42.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:02:42.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:02:42.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:42.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:02:42.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:02:42.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:42.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:02:42.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:02:42.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:02:42.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:02:42.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:42.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:02:42.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:02:42.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:42.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:02:42.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:02:42.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:42.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:02:42.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:02:42.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:42.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:02:42.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:02:42.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:42.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:02:42.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:02:42.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:42.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:02:42.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:02:42.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:42.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:02:42.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:02:42.095 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.095 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.095 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.095 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:42.095 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:02:42.095 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.7476970860000001, 2023-07-02T14:02:42.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:02:42.096 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:42.096 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.096 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:42.096 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:42.096 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:42.096 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:02:42.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.097 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:42.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:42.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:02:42.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:42.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:42.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:02:42.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:42.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:42.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:02:42.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:02:42.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:42.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:02:42.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:02:42.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:42.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:02:42.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:02:42.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:02:42.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:02:42.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:42.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:02:42.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:02:42.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:42.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:02:42.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:02:42.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:42.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:02:42.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:02:42.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:42.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:02:42.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:02:42.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:42.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:02:42.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:02:42.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:42.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:02:42.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:02:42.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:42.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:02:42.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:02:42.105 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.105 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.105 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.106 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:42.106 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:02:42.106 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.747357644, 2023-07-02T14:02:42.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:02:42.106 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:42.106 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.106 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:42.106 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:42.107 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:42.107 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:02:42.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.107 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:42.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:42.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:02:42.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:42.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:42.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:02:42.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:42.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:42.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:02:42.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:02:42.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:42.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:02:42.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:02:42.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:42.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:02:42.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:02:42.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:02:42.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:02:42.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:42.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:02:42.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:02:42.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:42.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:02:42.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:02:42.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:42.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:02:42.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:02:42.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:42.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:02:42.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:02:42.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:42.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:02:42.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:02:42.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:42.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:02:42.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:02:42.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:42.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:02:42.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:02:42.115 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.115 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.116 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.116 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:42.116 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:02:42.116 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.7471715909999999, 2023-07-02T14:02:42.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:02:42.116 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:42.117 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.117 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:42.117 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:42.117 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:42.117 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:02:42.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.117 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:42.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:42.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:02:42.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:42.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:42.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:02:42.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:42.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:42.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:02:42.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:02:42.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:42.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:02:42.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:02:42.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:42.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:02:42.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:02:42.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:02:42.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:02:42.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:42.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:02:42.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:02:42.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:42.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:02:42.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:02:42.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:42.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:02:42.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:02:42.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:42.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:02:42.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:02:42.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:42.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:02:42.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:02:42.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:42.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:02:42.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:02:42.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:42.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:02:42.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:02:42.126 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.126 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.126 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.126 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:42.126 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:02:42.127 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.7470137889999999, 2023-07-02T14:02:42.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:02:42.127 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:42.127 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.127 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:42.127 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:42.127 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:42.128 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:02:42.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.128 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:42.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:42.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:02:42.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:42.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:42.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:02:42.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:42.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:02:42.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:02:42.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:42.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:02:42.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:02:42.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:42.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:02:42.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:02:42.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:02:42.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:02:42.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:42.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:02:42.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:02:42.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:42.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:02:42.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:02:42.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:42.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:02:42.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:02:42.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:42.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:02:42.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:02:42.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:42.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:02:42.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:02:42.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:42.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:02:42.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:02:42.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:42.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:02:42.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:02:42.136 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.136 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.136 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.137 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:42.137 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:02:42.137 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.746822007, 2023-07-02T14:02:42.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:02:42.137 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:42.138 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.138 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:42.138 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:42.138 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:42.138 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:02:42.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.138 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:42.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:42.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:02:42.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:42.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:42.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:02:42.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:42.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:02:42.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:02:42.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:42.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:02:42.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:02:42.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:42.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:02:42.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:02:42.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:02:42.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:02:42.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:42.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:02:42.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:02:42.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:42.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:02:42.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:02:42.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:42.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:02:42.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:02:42.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:42.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:02:42.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:02:42.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:42.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:02:42.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:02:42.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:42.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:02:42.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:02:42.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:42.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:02:42.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:02:42.147 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.147 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.147 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.147 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:42.147 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:02:42.148 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.746646347, 2023-07-02T14:02:42.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:02:42.148 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:42.148 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.148 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:42.148 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:42.148 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:42.149 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:02:42.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.149 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:42.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:42.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:02:42.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:42.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:42.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:02:42.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:42.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:02:42.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:02:42.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:42.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:02:42.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:02:42.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:42.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:02:42.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:02:42.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:02:42.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:02:42.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:42.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:02:42.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:02:42.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:42.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:02:42.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:02:42.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:42.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:02:42.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:02:42.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:42.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:02:42.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:02:42.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:42.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:02:42.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:02:42.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:42.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:02:42.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:02:42.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:42.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:02:42.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:02:42.157 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.157 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.157 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.158 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:42.158 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:02:42.158 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.7465521590000002, 2023-07-02T14:02:42.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:02:42.158 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:42.158 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.159 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:42.159 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:42.159 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:42.159 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:02:42.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.159 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:42.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:42.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:02:42.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:42.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:42.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:02:42.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:42.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:42.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:02:42.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:02:42.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:42.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:02:42.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:02:42.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:42.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:02:42.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:02:42.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:02:42.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:02:42.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:42.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:02:42.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:02:42.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:42.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:02:42.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:02:42.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:42.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:02:42.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:02:42.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:42.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:02:42.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:02:42.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:42.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:02:42.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:02:42.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:42.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:02:42.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:02:42.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:42.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:02:42.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:02:42.168 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.168 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.168 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.168 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:42.168 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:02:42.168 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.7463487729999998, 2023-07-02T14:02:42.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:02:42.169 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:42.169 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.169 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:42.169 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:42.169 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:42.169 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:02:42.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.170 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:42.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:42.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:02:42.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:42.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:42.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:02:42.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:42.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:02:42.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:02:42.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:42.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:02:42.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:02:42.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:42.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:02:42.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:02:42.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:02:42.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:02:42.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:42.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:02:42.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:02:42.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:42.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:02:42.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:02:42.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:42.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:02:42.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:02:42.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:42.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:02:42.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:02:42.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:42.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:02:42.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:02:42.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:42.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:02:42.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:02:42.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:42.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:02:42.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:02:42.178 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.178 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.178 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.179 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:42.179 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:02:42.179 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.746198117, 2023-07-02T14:02:42.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:02:42.179 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:42.179 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.179 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:42.179 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:42.180 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:42.180 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:02:42.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.180 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:42.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:42.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:02:42.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:42.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:42.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:02:42.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:42.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:42.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:02:42.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:02:42.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:42.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:02:42.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:02:42.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:42.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:02:42.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:02:42.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:02:42.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:02:42.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:42.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:02:42.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:02:42.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:42.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:02:42.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:02:42.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:42.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:02:42.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:02:42.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:42.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:02:42.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:02:42.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:42.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:02:42.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:02:42.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:42.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:02:42.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:02:42.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:42.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:02:42.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:02:42.188 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.188 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.189 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.189 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:42.189 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:02:42.189 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.7461254140000002, 2023-07-02T14:02:42.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:02:42.189 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:42.190 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.190 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:42.190 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:42.190 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:42.190 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:02:42.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.190 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:42.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:42.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:02:42.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:42.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:42.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:02:42.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:42.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:42.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:02:42.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:02:42.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:42.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:02:42.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:02:42.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:42.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:02:42.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:02:42.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:02:42.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:02:42.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:42.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:02:42.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:02:42.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:42.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:02:42.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:02:42.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:42.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:02:42.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:02:42.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:42.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:02:42.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:02:42.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:42.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:02:42.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:02:42.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:42.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:02:42.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:02:42.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:42.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:02:42.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:02:42.199 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.199 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.199 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.199 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:42.199 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:02:42.200 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.7460084239999998, 2023-07-02T14:02:42.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:02:42.200 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:42.200 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.200 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:42.200 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:42.200 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:42.200 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:02:42.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.201 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:42.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:42.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:02:42.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:42.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:42.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:02:42.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:42.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:42.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:02:42.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:02:42.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:42.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:02:42.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:02:42.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:42.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:02:42.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:02:42.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:02:42.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:02:42.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:42.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:02:42.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:02:42.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:42.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:02:42.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:02:42.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:42.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:02:42.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:02:42.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:42.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:02:42.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:02:42.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:42.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:02:42.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:02:42.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:42.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:02:42.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:02:42.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:42.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:02:42.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:02:42.209 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.209 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.209 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.210 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:42.210 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:02:42.210 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.7458589729999998, 2023-07-02T14:02:42.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:02:42.210 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:42.210 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.210 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:42.211 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:42.211 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:42.211 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:02:42.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.211 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:42.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:42.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:02:42.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:42.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:42.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:02:42.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:42.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:42.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:02:42.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:02:42.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:42.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:02:42.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:02:42.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:42.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:02:42.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:02:42.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:02:42.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:02:42.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:42.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:02:42.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:02:42.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:42.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:02:42.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:02:42.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:42.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:02:42.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:02:42.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:42.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:02:42.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:02:42.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:42.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:02:42.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:02:42.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:42.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:02:42.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:02:42.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:42.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:02:42.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:02:42.219 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.220 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.220 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.220 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:42.220 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:02:42.220 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.724383, 2023-07-02T14:02:42.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:02:42.221 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:42.221 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.221 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:42.221 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:42.221 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:42.221 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:02:42.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.221 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:42.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:42.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:02:42.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:42.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:42.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:02:42.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:42.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:02:42.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:02:42.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:42.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:02:42.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:02:42.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:42.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:02:42.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:02:42.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:02:42.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:02:42.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:42.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:02:42.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:02:42.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:42.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:02:42.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:02:42.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:02:42.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:02:42.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:02:42.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:42.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:02:42.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:02:42.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:42.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:02:42.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:02:42.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:42.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:02:42.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:02:42.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:42.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:02:42.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:02:42.230 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.230 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.230 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.230 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:42.231 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:02:42.231 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.7242504639999998, 2023-07-02T14:02:42.231 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:02:42.231 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:42.231 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.231 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:42.231 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:42.231 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:42.232 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:02:42.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.232 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:42.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:42.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:02:42.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:42.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:42.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:02:42.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:42.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:02:42.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:02:42.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:42.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:02:42.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:02:42.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:42.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:02:42.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:02:42.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:02:42.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:02:42.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:42.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:02:42.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:02:42.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:42.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:02:42.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:02:42.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:02:42.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:02:42.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:02:42.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:42.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:02:42.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:02:42.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:42.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:02:42.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:02:42.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:42.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:02:42.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:02:42.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.240 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:42.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:02:42.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:02:42.240 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.240 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.240 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.241 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:42.241 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:02:42.241 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.6629138029999999, 2023-07-02T14:02:42.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:02:42.241 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:42.241 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.242 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:42.242 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:42.242 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:42.242 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:02:42.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.242 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:42.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:42.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:02:42.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:42.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:42.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:02:42.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:42.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:02:42.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:02:42.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:42.245 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:02:42.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:02:42.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:42.245 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:02:42.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:02:42.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.246 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.246 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:02:42.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:02:42.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.246 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.246 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:42.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:02:42.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:02:42.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:42.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:02:42.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:02:42.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:42.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:02:42.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:02:42.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:42.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:02:42.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:02:42.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:42.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:02:42.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:02:42.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:42.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:02:42.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:02:42.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:42.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:42.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:42.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:02:42.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:02:42.251 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.251 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.251 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.251 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:42.251 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:42.251 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:42.253 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:42.252+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:42.254 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:02:42.323 INFO:tasks.workunit.client.0.smithi163.stderr: 39900K .......... .......... .......... .......... .......... 23% 18.8M 7s 2023-07-02T14:02:42.323 INFO:tasks.workunit.client.0.smithi163.stderr: 39950K .......... .......... .......... .......... .......... 23% 14.5M 7s 2023-07-02T14:02:42.323 INFO:tasks.workunit.client.0.smithi163.stderr: 40000K .......... .......... .......... .......... .......... 23% 16.6M 7s 2023-07-02T14:02:42.324 INFO:tasks.workunit.client.0.smithi163.stderr: 40050K .......... .......... .......... .......... .......... 23% 19.7M 7s 2023-07-02T14:02:42.324 INFO:tasks.workunit.client.0.smithi163.stderr: 40100K .......... .......... .......... .......... .......... 23% 30.8M 7s 2023-07-02T14:02:42.324 INFO:tasks.workunit.client.0.smithi163.stderr: 40150K .......... .......... .......... .......... .......... 23% 17.1M 7s 2023-07-02T14:02:42.324 INFO:tasks.workunit.client.0.smithi163.stderr: 40200K .......... .......... .......... .......... .......... 23% 21.3M 7s 2023-07-02T14:02:42.324 INFO:tasks.workunit.client.0.smithi163.stderr: 40250K .......... .......... .......... .......... .......... 23% 19.5M 7s 2023-07-02T14:02:42.324 INFO:tasks.workunit.client.0.smithi163.stderr: 40300K .......... .......... .......... .......... .......... 23% 31.7M 7s 2023-07-02T14:02:42.324 INFO:tasks.workunit.client.0.smithi163.stderr: 40350K .......... .......... .......... .......... .......... 23% 17.9M 7s 2023-07-02T14:02:42.325 INFO:tasks.workunit.client.0.smithi163.stderr: 40400K .......... .......... .......... .......... .......... 23% 19.1M 7s 2023-07-02T14:02:42.325 INFO:tasks.workunit.client.0.smithi163.stderr: 40450K .......... .......... .......... .......... .......... 24% 14.9M 7s 2023-07-02T14:02:42.325 INFO:tasks.workunit.client.0.smithi163.stderr: 40500K .......... .......... .......... .......... .......... 24% 23.6M 7s 2023-07-02T14:02:42.325 INFO:tasks.workunit.client.0.smithi163.stderr: 40550K .......... .......... .......... .......... .......... 24% 17.5M 7s 2023-07-02T14:02:42.325 INFO:tasks.workunit.client.0.smithi163.stderr: 40600K .......... .......... .......... .......... .......... 24% 51.3M 7s 2023-07-02T14:02:42.325 INFO:tasks.workunit.client.0.smithi163.stderr: 40650K .......... .......... .......... .......... .......... 24% 15.7M 7s 2023-07-02T14:02:42.325 INFO:tasks.workunit.client.0.smithi163.stderr: 40700K .......... .......... .......... .......... .......... 24% 28.0M 7s 2023-07-02T14:02:42.326 INFO:tasks.workunit.client.0.smithi163.stderr: 40750K .......... .......... .......... .......... .......... 24% 14.5M 7s 2023-07-02T14:02:42.326 INFO:tasks.workunit.client.0.smithi163.stderr: 40800K .......... .......... .......... .......... .......... 24% 24.9M 7s 2023-07-02T14:02:42.326 INFO:tasks.workunit.client.0.smithi163.stderr: 40850K .......... .......... .......... .......... .......... 24% 18.6M 7s 2023-07-02T14:02:42.326 INFO:tasks.workunit.client.0.smithi163.stderr: 40900K .......... .......... .......... .......... .......... 24% 28.6M 7s 2023-07-02T14:02:42.326 INFO:tasks.workunit.client.0.smithi163.stderr: 40950K .......... .......... .......... .......... .......... 24% 17.8M 7s 2023-07-02T14:02:42.326 INFO:tasks.workunit.client.0.smithi163.stderr: 41000K .......... .......... .......... .......... .......... 24% 19.5M 7s 2023-07-02T14:02:42.326 INFO:tasks.workunit.client.0.smithi163.stderr: 41050K .......... .......... .......... .......... .......... 24% 16.0M 7s 2023-07-02T14:02:42.326 INFO:tasks.workunit.client.0.smithi163.stderr: 41100K .......... .......... .......... .......... .......... 24% 46.0M 7s 2023-07-02T14:02:42.327 INFO:tasks.workunit.client.0.smithi163.stderr: 41150K .......... .......... .......... .......... .......... 24% 14.7M 7s 2023-07-02T14:02:42.327 INFO:tasks.workunit.client.0.smithi163.stderr: 41200K .......... .......... .......... .......... .......... 24% 30.3M 7s 2023-07-02T14:02:42.327 INFO:tasks.workunit.client.0.smithi163.stderr: 41250K .......... .......... .......... .......... .......... 24% 23.4M 7s 2023-07-02T14:02:42.327 INFO:tasks.workunit.client.0.smithi163.stderr: 41300K .......... .......... .......... .......... .......... 24% 15.1M 7s 2023-07-02T14:02:42.327 INFO:tasks.workunit.client.0.smithi163.stderr: 41350K .......... .......... .......... .......... .......... 24% 19.6M 7s 2023-07-02T14:02:42.327 INFO:tasks.workunit.client.0.smithi163.stderr: 41400K .......... .......... .......... .......... .......... 24% 21.7M 7s 2023-07-02T14:02:42.327 INFO:tasks.workunit.client.0.smithi163.stderr: 41450K .......... .......... .......... .......... .......... 24% 18.7M 7s 2023-07-02T14:02:42.327 INFO:tasks.workunit.client.0.smithi163.stderr: 41500K .......... .......... .......... .......... .......... 24% 31.8M 7s 2023-07-02T14:02:42.328 INFO:tasks.workunit.client.0.smithi163.stderr: 41550K .......... .......... .......... .......... .......... 24% 16.9M 7s 2023-07-02T14:02:42.328 INFO:tasks.workunit.client.0.smithi163.stderr: 41600K .......... .......... .......... .......... .......... 24% 14.4M 7s 2023-07-02T14:02:42.328 INFO:tasks.workunit.client.0.smithi163.stderr: 41650K .......... .......... .......... .......... .......... 24% 28.2M 7s 2023-07-02T14:02:42.328 INFO:tasks.workunit.client.0.smithi163.stderr: 41700K .......... .......... .......... .......... .......... 24% 25.4M 7s 2023-07-02T14:02:42.328 INFO:tasks.workunit.client.0.smithi163.stderr: 41750K .......... .......... .......... .......... .......... 24% 12.8M 7s 2023-07-02T14:02:42.328 INFO:tasks.workunit.client.0.smithi163.stderr: 41800K .......... .......... .......... .......... .......... 24% 33.0M 7s 2023-07-02T14:02:42.328 INFO:tasks.workunit.client.0.smithi163.stderr: 41850K .......... .......... .......... .......... .......... 24% 20.1M 7s 2023-07-02T14:02:42.328 INFO:tasks.workunit.client.0.smithi163.stderr: 41900K .......... .......... .......... .......... .......... 24% 21.3M 7s 2023-07-02T14:02:42.329 INFO:tasks.workunit.client.0.smithi163.stderr: 41950K .......... .......... .......... .......... .......... 24% 16.9M 7s 2023-07-02T14:02:42.329 INFO:tasks.workunit.client.0.smithi163.stderr: 42000K .......... .......... .......... .......... .......... 24% 18.4M 7s 2023-07-02T14:02:42.329 INFO:tasks.workunit.client.0.smithi163.stderr: 42050K .......... .......... .......... .......... .......... 24% 18.5M 7s 2023-07-02T14:02:42.329 INFO:tasks.workunit.client.0.smithi163.stderr: 42100K .......... .......... .......... .......... .......... 25% 36.5M 7s 2023-07-02T14:02:42.329 INFO:tasks.workunit.client.0.smithi163.stderr: 42150K .......... .......... .......... .......... .......... 25% 15.5M 7s 2023-07-02T14:02:42.329 INFO:tasks.workunit.client.0.smithi163.stderr: 42200K .......... .......... .......... .......... .......... 25% 26.8M 7s 2023-07-02T14:02:42.329 INFO:tasks.workunit.client.0.smithi163.stderr: 42250K .......... .......... .......... .......... .......... 25% 23.0M 7s 2023-07-02T14:02:42.330 INFO:tasks.workunit.client.0.smithi163.stderr: 42300K .......... .......... .......... .......... .......... 25% 15.8M 7s 2023-07-02T14:02:42.330 INFO:tasks.workunit.client.0.smithi163.stderr: 42350K .......... .......... .......... .......... .......... 25% 14.3M 7s 2023-07-02T14:02:42.330 INFO:tasks.workunit.client.0.smithi163.stderr: 42400K .......... .......... .......... .......... .......... 25% 39.1M 7s 2023-07-02T14:02:42.330 INFO:tasks.workunit.client.0.smithi163.stderr: 42450K .......... .......... .......... .......... .......... 25% 22.0M 7s 2023-07-02T14:02:42.330 INFO:tasks.workunit.client.0.smithi163.stderr: 42500K .......... .......... .......... .......... .......... 25% 17.8M 7s 2023-07-02T14:02:42.330 INFO:tasks.workunit.client.0.smithi163.stderr: 42550K .......... .......... .......... .......... .......... 25% 27.3M 7s 2023-07-02T14:02:42.330 INFO:tasks.workunit.client.0.smithi163.stderr: 42600K .......... .......... .......... .......... .......... 25% 32.5M 7s 2023-07-02T14:02:42.330 INFO:tasks.workunit.client.0.smithi163.stderr: 42650K .......... .......... .......... .......... .......... 25% 14.5M 7s 2023-07-02T14:02:42.331 INFO:tasks.workunit.client.0.smithi163.stderr: 42700K .......... .......... .......... .......... .......... 25% 19.7M 7s 2023-07-02T14:02:42.331 INFO:tasks.workunit.client.0.smithi163.stderr: 42750K .......... .......... .......... .......... .......... 25% 16.5M 7s 2023-07-02T14:02:42.331 INFO:tasks.workunit.client.0.smithi163.stderr: 42800K .......... .......... .......... .......... .......... 25% 32.0M 7s 2023-07-02T14:02:42.331 INFO:tasks.workunit.client.0.smithi163.stderr: 42850K .......... .......... .......... .......... .......... 25% 17.2M 7s 2023-07-02T14:02:42.331 INFO:tasks.workunit.client.0.smithi163.stderr: 42900K .......... .......... .......... .......... .......... 25% 26.7M 7s 2023-07-02T14:02:42.331 INFO:tasks.workunit.client.0.smithi163.stderr: 42950K .......... .......... .......... .......... .......... 25% 16.5M 7s 2023-07-02T14:02:42.331 INFO:tasks.workunit.client.0.smithi163.stderr: 43000K .......... .......... .......... .......... .......... 25% 18.5M 7s 2023-07-02T14:02:42.331 INFO:tasks.workunit.client.0.smithi163.stderr: 43050K .......... .......... .......... .......... .......... 25% 20.3M 7s 2023-07-02T14:02:42.332 INFO:tasks.workunit.client.0.smithi163.stderr: 43100K .......... .......... .......... .......... .......... 25% 33.7M 7s 2023-07-02T14:02:42.332 INFO:tasks.workunit.client.0.smithi163.stderr: 43150K .......... .......... .......... .......... .......... 25% 15.3M 7s 2023-07-02T14:02:42.332 INFO:tasks.workunit.client.0.smithi163.stderr: 43200K .......... .......... .......... .......... .......... 25% 24.4M 7s 2023-07-02T14:02:42.332 INFO:tasks.workunit.client.0.smithi163.stderr: 43250K .......... .......... .......... .......... .......... 25% 16.3M 7s 2023-07-02T14:02:42.332 INFO:tasks.workunit.client.0.smithi163.stderr: 43300K .......... .......... .......... .......... .......... 25% 27.0M 7s 2023-07-02T14:02:42.332 INFO:tasks.workunit.client.0.smithi163.stderr: 43350K .......... .......... .......... .......... .......... 25% 11.9M 7s 2023-07-02T14:02:42.332 INFO:tasks.workunit.client.0.smithi163.stderr: 43400K .......... .......... .......... .......... .......... 25% 42.2M 7s 2023-07-02T14:02:42.332 INFO:tasks.workunit.client.0.smithi163.stderr: 43450K .......... .......... .......... .......... .......... 25% 18.6M 7s 2023-07-02T14:02:42.333 INFO:tasks.workunit.client.0.smithi163.stderr: 43500K .......... .......... .......... .......... .......... 25% 17.1M 7s 2023-07-02T14:02:42.333 INFO:tasks.workunit.client.0.smithi163.stderr: 43550K .......... .......... .......... .......... .......... 25% 20.7M 7s 2023-07-02T14:02:42.333 INFO:tasks.workunit.client.0.smithi163.stderr: 43600K .......... .......... .......... .......... .......... 25% 15.8M 7s 2023-07-02T14:02:42.333 INFO:tasks.workunit.client.0.smithi163.stderr: 43650K .......... .......... .......... .......... .......... 25% 35.5M 7s 2023-07-02T14:02:42.333 INFO:tasks.workunit.client.0.smithi163.stderr: 43700K .......... .......... .......... .......... .......... 25% 18.5M 7s 2023-07-02T14:02:42.333 INFO:tasks.workunit.client.0.smithi163.stderr: 43750K .......... .......... .......... .......... .......... 25% 13.9M 7s 2023-07-02T14:02:42.333 INFO:tasks.workunit.client.0.smithi163.stderr: 43800K .......... .......... .......... .......... .......... 26% 31.8M 7s 2023-07-02T14:02:42.333 INFO:tasks.workunit.client.0.smithi163.stderr: 43850K .......... .......... .......... .......... .......... 26% 37.6M 7s 2023-07-02T14:02:42.334 INFO:tasks.workunit.client.0.smithi163.stderr: 43900K .......... .......... .......... .......... .......... 26% 20.2M 7s 2023-07-02T14:02:42.334 INFO:tasks.workunit.client.0.smithi163.stderr: 43950K .......... .......... .......... .......... .......... 26% 13.1M 7s 2023-07-02T14:02:42.334 INFO:tasks.workunit.client.0.smithi163.stderr: 44000K .......... .......... .......... .......... .......... 26% 17.5M 7s 2023-07-02T14:02:42.334 INFO:tasks.workunit.client.0.smithi163.stderr: 44050K .......... .......... .......... .......... .......... 26% 31.8M 7s 2023-07-02T14:02:42.334 INFO:tasks.workunit.client.0.smithi163.stderr: 44100K .......... .......... .......... .......... .......... 26% 16.4M 7s 2023-07-02T14:02:42.334 INFO:tasks.workunit.client.0.smithi163.stderr: 44150K .......... .......... .......... .......... .......... 26% 37.8M 7s 2023-07-02T14:02:42.334 INFO:tasks.workunit.client.0.smithi163.stderr: 44200K .......... .......... .......... .......... .......... 26% 24.5M 7s 2023-07-02T14:02:42.334 INFO:tasks.workunit.client.0.smithi163.stderr: 44250K .......... .......... .......... .......... .......... 26% 14.4M 7s 2023-07-02T14:02:42.335 INFO:tasks.workunit.client.0.smithi163.stderr: 44300K .......... .......... .......... .......... .......... 26% 21.8M 7s 2023-07-02T14:02:42.335 INFO:tasks.workunit.client.0.smithi163.stderr: 44350K .......... .......... .......... .......... .......... 26% 21.5M 7s 2023-07-02T14:02:42.335 INFO:tasks.workunit.client.0.smithi163.stderr: 44400K .......... .......... .......... .......... .......... 26% 20.1M 7s 2023-07-02T14:02:42.335 INFO:tasks.workunit.client.0.smithi163.stderr: 44450K .......... .......... .......... .......... .......... 26% 26.7M 7s 2023-07-02T14:02:42.335 INFO:tasks.workunit.client.0.smithi163.stderr: 44500K .......... .......... .......... .......... .......... 26% 14.8M 7s 2023-07-02T14:02:42.335 INFO:tasks.workunit.client.0.smithi163.stderr: 44550K .......... .......... .......... .......... .......... 26% 43.3M 7s 2023-07-02T14:02:42.335 INFO:tasks.workunit.client.0.smithi163.stderr: 44600K .......... .......... .......... .......... .......... 26% 13.2M 7s 2023-07-02T14:02:42.335 INFO:tasks.workunit.client.0.smithi163.stderr: 44650K .......... .......... .......... .......... .......... 26% 25.9M 7s 2023-07-02T14:02:42.336 INFO:tasks.workunit.client.0.smithi163.stderr: 44700K .......... .......... .......... .......... .......... 26% 19.5M 7s 2023-07-02T14:02:42.336 INFO:tasks.workunit.client.0.smithi163.stderr: 44750K .......... .......... .......... .......... .......... 26% 13.3M 7s 2023-07-02T14:02:42.336 INFO:tasks.workunit.client.0.smithi163.stderr: 44800K .......... .......... .......... .......... .......... 26% 35.7M 7s 2023-07-02T14:02:42.336 INFO:tasks.workunit.client.0.smithi163.stderr: 44850K .......... .......... .......... .......... .......... 26% 18.2M 7s 2023-07-02T14:02:42.336 INFO:tasks.workunit.client.0.smithi163.stderr: 44900K .......... .......... .......... .......... .......... 26% 25.6M 7s 2023-07-02T14:02:42.336 INFO:tasks.workunit.client.0.smithi163.stderr: 44950K .......... .......... .......... .......... .......... 26% 18.9M 7s 2023-07-02T14:02:42.336 INFO:tasks.workunit.client.0.smithi163.stderr: 45000K .......... .......... .......... .......... .......... 26% 17.6M 7s 2023-07-02T14:02:42.336 INFO:tasks.workunit.client.0.smithi163.stderr: 45050K .......... .......... .......... .......... .......... 26% 26.2M 7s 2023-07-02T14:02:42.337 INFO:tasks.workunit.client.0.smithi163.stderr: 45100K .......... .......... .......... .......... .......... 26% 14.8M 7s 2023-07-02T14:02:42.337 INFO:tasks.workunit.client.0.smithi163.stderr: 45150K .......... .......... .......... .......... .......... 26% 18.7M 7s 2023-07-02T14:02:42.337 INFO:tasks.workunit.client.0.smithi163.stderr: 45200K .......... .......... .......... .......... .......... 26% 23.6M 7s 2023-07-02T14:02:42.337 INFO:tasks.workunit.client.0.smithi163.stderr: 45250K .......... .......... .......... .......... .......... 26% 19.1M 7s 2023-07-02T14:02:42.337 INFO:tasks.workunit.client.0.smithi163.stderr: 45300K .......... .......... .......... .......... .......... 26% 32.5M 7s 2023-07-02T14:02:42.337 INFO:tasks.workunit.client.0.smithi163.stderr: 45350K .......... .......... .......... .......... .......... 26% 16.0M 7s 2023-07-02T14:02:42.337 INFO:tasks.workunit.client.0.smithi163.stderr: 45400K .......... .......... .......... .......... .......... 26% 22.4M 7s 2023-07-02T14:02:42.338 INFO:tasks.workunit.client.0.smithi163.stderr: 45450K .......... .......... .......... .......... .......... 26% 21.7M 7s 2023-07-02T14:02:42.338 INFO:tasks.workunit.client.0.smithi163.stderr: 45500K .......... .......... .......... .......... .......... 27% 21.0M 7s 2023-07-02T14:02:42.338 INFO:tasks.workunit.client.0.smithi163.stderr: 45550K .......... .......... .......... .......... .......... 27% 18.6M 7s 2023-07-02T14:02:42.338 INFO:tasks.workunit.client.0.smithi163.stderr: 45600K .......... .......... .......... .......... .......... 27% 19.8M 7s 2023-07-02T14:02:42.338 INFO:tasks.workunit.client.0.smithi163.stderr: 45650K .......... .......... .......... .......... .......... 27% 15.1M 7s 2023-07-02T14:02:42.338 INFO:tasks.workunit.client.0.smithi163.stderr: 45700K .......... .......... .......... .......... .......... 27% 32.4M 7s 2023-07-02T14:02:42.341 INFO:tasks.workunit.client.0.smithi163.stderr: 45750K .......... .......... .......... .......... .......... 27% 20.4M 7s 2023-07-02T14:02:42.341 INFO:tasks.workunit.client.0.smithi163.stderr: 45800K .......... .......... .......... .......... .......... 27% 36.5M 7s 2023-07-02T14:02:42.341 INFO:tasks.workunit.client.0.smithi163.stderr: 45850K .......... .......... .......... .......... .......... 27% 20.0M 7s 2023-07-02T14:02:42.342 INFO:tasks.workunit.client.0.smithi163.stderr: 45900K .......... .......... .......... .......... .......... 27% 16.9M 7s 2023-07-02T14:02:42.342 INFO:tasks.workunit.client.0.smithi163.stderr: 45950K .......... .......... .......... .......... .......... 27% 19.5M 7s 2023-07-02T14:02:42.342 INFO:tasks.workunit.client.0.smithi163.stderr: 46000K .......... .......... .......... .......... .......... 27% 23.7M 7s 2023-07-02T14:02:42.342 INFO:tasks.workunit.client.0.smithi163.stderr: 46050K .......... .......... .......... .......... .......... 27% 15.1M 7s 2023-07-02T14:02:42.343 INFO:tasks.workunit.client.0.smithi163.stderr: 46100K .......... .......... .......... .......... .......... 27% 22.5M 7s 2023-07-02T14:02:42.344 INFO:tasks.workunit.client.0.smithi163.stderr: 46150K .......... .......... .......... .......... .......... 27% 21.1M 7s 2023-07-02T14:02:42.347 INFO:tasks.workunit.client.0.smithi163.stderr: 46200K .......... .......... .......... .......... .......... 27% 16.7M 7s 2023-07-02T14:02:42.349 INFO:tasks.workunit.client.0.smithi163.stderr: 46250K .......... .......... .......... .......... .......... 27% 25.1M 7s 2023-07-02T14:02:42.351 INFO:tasks.workunit.client.0.smithi163.stderr: 46300K .......... .......... .......... .......... .......... 27% 26.3M 6s 2023-07-02T14:02:42.353 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:42.352+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:42.354 INFO:tasks.workunit.client.0.smithi163.stderr: 46350K .......... .......... .......... .......... .......... 27% 14.6M 6s 2023-07-02T14:02:42.357 INFO:tasks.workunit.client.0.smithi163.stderr: 46400K .......... .......... .......... .......... .......... 27% 16.9M 6s 2023-07-02T14:02:42.359 INFO:tasks.workunit.client.0.smithi163.stderr: 46450K .......... .......... .......... .......... .......... 27% 39.5M 6s 2023-07-02T14:02:42.362 INFO:tasks.workunit.client.0.smithi163.stderr: 46500K .......... .......... .......... .......... .......... 27% 17.6M 6s 2023-07-02T14:02:42.363 INFO:tasks.workunit.client.0.smithi163.stderr: 46550K .......... .......... .......... .......... .......... 27% 22.6M 6s 2023-07-02T14:02:42.366 INFO:tasks.workunit.client.0.smithi163.stderr: 46600K .......... .......... .......... .......... .......... 27% 18.9M 6s 2023-07-02T14:02:42.368 INFO:tasks.workunit.client.0.smithi163.stderr: 46650K .......... .......... .......... .......... .......... 27% 34.5M 6s 2023-07-02T14:02:42.371 INFO:tasks.workunit.client.0.smithi163.stderr: 46700K .......... .......... .......... .......... .......... 27% 14.9M 6s 2023-07-02T14:02:42.374 INFO:tasks.workunit.client.0.smithi163.stderr: 46750K .......... .......... .......... .......... .......... 27% 15.9M 6s 2023-07-02T14:02:42.376 INFO:tasks.workunit.client.0.smithi163.stderr: 46800K .......... .......... .......... .......... .......... 27% 20.0M 6s 2023-07-02T14:02:42.378 INFO:tasks.workunit.client.0.smithi163.stderr: 46850K .......... .......... .......... .......... .......... 27% 23.2M 6s 2023-07-02T14:02:42.381 INFO:tasks.workunit.client.0.smithi163.stderr: 46900K .......... .......... .......... .......... .......... 27% 19.5M 6s 2023-07-02T14:02:42.383 INFO:tasks.workunit.client.0.smithi163.stderr: 46950K .......... .......... .......... .......... .......... 27% 25.6M 6s 2023-07-02T14:02:42.384 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:42.384 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:42.384 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:42.384 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:42.386 INFO:tasks.workunit.client.0.smithi163.stderr: 47000K .......... .......... .......... .......... .......... 27% 17.0M 6s 2023-07-02T14:02:42.389 INFO:tasks.workunit.client.0.smithi163.stderr: 47050K .......... .......... .......... .......... .......... 27% 16.9M 6s 2023-07-02T14:02:42.390 INFO:tasks.workunit.client.0.smithi163.stderr: 47100K .......... .......... .......... .......... .......... 27% 23.2M 6s 2023-07-02T14:02:42.393 INFO:tasks.workunit.client.0.smithi163.stderr: 47150K .......... .......... .......... .......... .......... 28% 22.5M 6s 2023-07-02T14:02:42.394 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:02:42.395 INFO:tasks.workunit.client.0.smithi163.stderr: 47200K .......... .......... .......... .......... .......... 28% 31.2M 6s 2023-07-02T14:02:42.398 INFO:tasks.workunit.client.0.smithi163.stderr: 47250K .......... .......... .......... .......... .......... 28% 14.3M 6s 2023-07-02T14:02:42.400 INFO:tasks.workunit.client.0.smithi163.stderr: 47300K .......... .......... .......... .......... .......... 28% 19.3M 6s 2023-07-02T14:02:42.402 INFO:tasks.workunit.client.0.smithi163.stderr: 47350K .......... .......... .......... .......... .......... 28% 25.1M 6s 2023-07-02T14:02:42.405 INFO:tasks.workunit.client.0.smithi163.stderr: 47400K .......... .......... .......... .......... .......... 28% 17.5M 6s 2023-07-02T14:02:42.406 INFO:tasks.workunit.client.0.smithi163.stderr: 47450K .......... .......... .......... .......... .......... 28% 58.5M 6s 2023-07-02T14:02:42.408 INFO:tasks.workunit.client.0.smithi163.stderr: 47500K .......... .......... .......... .......... .......... 28% 20.8M 6s 2023-07-02T14:02:42.412 INFO:tasks.workunit.client.0.smithi163.stderr: 47550K .......... .......... .......... .......... .......... 28% 13.4M 6s 2023-07-02T14:02:42.415 INFO:tasks.workunit.client.0.smithi163.stderr: 47600K .......... .......... .......... .......... .......... 28% 16.9M 6s 2023-07-02T14:02:42.416 INFO:tasks.workunit.client.0.smithi163.stderr: 47650K .......... .......... .......... .......... .......... 28% 29.9M 6s 2023-07-02T14:02:42.419 INFO:tasks.workunit.client.0.smithi163.stderr: 47700K .......... .......... .......... .......... .......... 28% 20.9M 6s 2023-07-02T14:02:42.421 INFO:tasks.workunit.client.0.smithi163.stderr: 47750K .......... .......... .......... .......... .......... 28% 21.5M 6s 2023-07-02T14:02:42.424 INFO:tasks.workunit.client.0.smithi163.stderr: 47800K .......... .......... .......... .......... .......... 28% 16.7M 6s 2023-07-02T14:02:42.426 INFO:tasks.workunit.client.0.smithi163.stderr: 47850K .......... .......... .......... .......... .......... 28% 26.5M 6s 2023-07-02T14:02:42.429 INFO:tasks.workunit.client.0.smithi163.stderr: 47900K .......... .......... .......... .......... .......... 28% 15.2M 6s 2023-07-02T14:02:42.432 INFO:tasks.workunit.client.0.smithi163.stderr: 47950K .......... .......... .......... .......... .......... 28% 17.2M 6s 2023-07-02T14:02:42.434 INFO:tasks.workunit.client.0.smithi163.stderr: 48000K .......... .......... .......... .......... .......... 28% 21.5M 6s 2023-07-02T14:02:42.437 INFO:tasks.workunit.client.0.smithi163.stderr: 48050K .......... .......... .......... .......... .......... 28% 18.8M 6s 2023-07-02T14:02:42.439 INFO:tasks.workunit.client.0.smithi163.stderr: 48100K .......... .......... .......... .......... .......... 28% 36.0M 6s 2023-07-02T14:02:42.441 INFO:tasks.workunit.client.0.smithi163.stderr: 48150K .......... .......... .......... .......... .......... 28% 17.0M 6s 2023-07-02T14:02:42.443 INFO:tasks.workunit.client.0.smithi163.stderr: 48200K .......... .......... .......... .......... .......... 28% 24.3M 6s 2023-07-02T14:02:42.445 INFO:tasks.workunit.client.0.smithi163.stderr: 48250K .......... .......... .......... .......... .......... 28% 31.6M 6s 2023-07-02T14:02:42.448 INFO:tasks.workunit.client.0.smithi163.stderr: 48300K .......... .......... .......... .......... .......... 28% 15.4M 6s 2023-07-02T14:02:42.452 INFO:tasks.workunit.client.0.smithi163.stderr: 48350K .......... .......... .......... .......... .......... 28% 13.3M 6s 2023-07-02T14:02:42.454 INFO:tasks.workunit.client.0.smithi163.stderr: 48400K .......... .......... .......... .......... .......... 28% 23.5M 6s 2023-07-02T14:02:42.454 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:42.453+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:42.457 INFO:tasks.workunit.client.0.smithi163.stderr: 48450K .......... .......... .......... .......... .......... 28% 16.5M 6s 2023-07-02T14:02:42.458 INFO:tasks.workunit.client.0.smithi163.stderr: 48500K .......... .......... .......... .......... .......... 28% 28.5M 6s 2023-07-02T14:02:42.460 INFO:tasks.workunit.client.0.smithi163.stderr: 48550K .......... .......... .......... .......... .......... 28% 27.5M 6s 2023-07-02T14:02:42.462 INFO:tasks.workunit.client.0.smithi163.stderr: 48600K .......... .......... .......... .......... .......... 28% 19.4M 6s 2023-07-02T14:02:42.466 INFO:tasks.workunit.client.0.smithi163.stderr: 48650K .......... .......... .......... .......... .......... 28% 14.9M 6s 2023-07-02T14:02:42.467 INFO:tasks.workunit.client.0.smithi163.stderr: 48700K .......... .......... .......... .......... .......... 28% 37.9M 6s 2023-07-02T14:02:42.470 INFO:tasks.workunit.client.0.smithi163.stderr: 48750K .......... .......... .......... .......... .......... 28% 14.2M 6s 2023-07-02T14:02:42.472 INFO:tasks.workunit.client.0.smithi163.stderr: 48800K .......... .......... .......... .......... .......... 28% 27.5M 6s 2023-07-02T14:02:42.475 INFO:tasks.workunit.client.0.smithi163.stderr: 48850K .......... .......... .......... .......... .......... 29% 18.7M 6s 2023-07-02T14:02:42.477 INFO:tasks.workunit.client.0.smithi163.stderr: 48900K .......... .......... .......... .......... .......... 29% 19.0M 6s 2023-07-02T14:02:42.481 INFO:tasks.workunit.client.0.smithi163.stderr: 48950K .......... .......... .......... .......... .......... 29% 15.2M 6s 2023-07-02T14:02:42.482 INFO:tasks.workunit.client.0.smithi163.stderr: 49000K .......... .......... .......... .......... .......... 29% 73.4M 6s 2023-07-02T14:02:42.483 INFO:tasks.workunit.client.0.smithi163.stderr: 49050K .......... .......... .......... .......... .......... 29% 20.1M 6s 2023-07-02T14:02:42.486 INFO:tasks.workunit.client.0.smithi163.stderr: 49100K .......... .......... .......... .......... .......... 29% 21.0M 6s 2023-07-02T14:02:42.488 INFO:tasks.workunit.client.0.smithi163.stderr: 49150K .......... .......... .......... .......... .......... 29% 22.2M 6s 2023-07-02T14:02:42.493 INFO:tasks.workunit.client.0.smithi163.stderr: 49200K .......... .......... .......... .......... .......... 29% 11.7M 6s 2023-07-02T14:02:42.494 INFO:tasks.workunit.client.0.smithi163.stderr: 49250K .......... .......... .......... .......... .......... 29% 40.6M 6s 2023-07-02T14:02:42.496 INFO:tasks.workunit.client.0.smithi163.stderr: 49300K .......... .......... .......... .......... .......... 29% 22.2M 6s 2023-07-02T14:02:42.499 INFO:tasks.workunit.client.0.smithi163.stderr: 49350K .......... .......... .......... .......... .......... 29% 15.4M 6s 2023-07-02T14:02:42.501 INFO:tasks.workunit.client.0.smithi163.stderr: 49400K .......... .......... .......... .......... .......... 29% 21.6M 6s 2023-07-02T14:02:42.503 INFO:tasks.workunit.client.0.smithi163.stderr: 49450K .......... .......... .......... .......... .......... 29% 28.1M 6s 2023-07-02T14:02:42.505 INFO:tasks.workunit.client.0.smithi163.stderr: 49500K .......... .......... .......... .......... .......... 29% 19.0M 6s 2023-07-02T14:02:42.508 INFO:tasks.workunit.client.0.smithi163.stderr: 49550K .......... .......... .......... .......... .......... 29% 16.4M 6s 2023-07-02T14:02:42.511 INFO:tasks.workunit.client.0.smithi163.stderr: 49600K .......... .......... .......... .......... .......... 29% 18.8M 6s 2023-07-02T14:02:42.513 INFO:tasks.workunit.client.0.smithi163.stderr: 49650K .......... .......... .......... .......... .......... 29% 20.4M 6s 2023-07-02T14:02:42.517 INFO:tasks.workunit.client.0.smithi163.stderr: 49700K .......... .......... .......... .......... .......... 29% 16.0M 6s 2023-07-02T14:02:42.518 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:42.518 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:42.518 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:42.518 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:42.518 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:42.519 INFO:tasks.workunit.client.0.smithi163.stderr: 49750K .......... .......... .......... .......... .......... 29% 37.1M 6s 2023-07-02T14:02:42.520 INFO:tasks.workunit.client.0.smithi163.stderr: 49800K .......... .......... .......... .......... .......... 29% 18.1M 6s 2023-07-02T14:02:42.523 INFO:tasks.workunit.client.0.smithi163.stderr: 49850K .......... .......... .......... .......... .......... 29% 19.7M 6s 2023-07-02T14:02:42.524 INFO:tasks.workunit.client.0.smithi163.stderr: 49900K .......... .......... .......... .......... .......... 29% 39.2M 6s 2023-07-02T14:02:42.528 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:02:42.529 INFO:tasks.workunit.client.0.smithi163.stderr: 49950K .......... .......... .......... .......... .......... 29% 13.2M 6s 2023-07-02T14:02:42.531 INFO:tasks.workunit.client.0.smithi163.stderr: 50000K .......... .......... .......... .......... .......... 29% 18.9M 6s 2023-07-02T14:02:42.533 INFO:tasks.workunit.client.0.smithi163.stderr: 50050K .......... .......... .......... .......... .......... 29% 24.4M 6s 2023-07-02T14:02:42.536 INFO:tasks.workunit.client.0.smithi163.stderr: 50100K .......... .......... .......... .......... .......... 29% 15.4M 6s 2023-07-02T14:02:42.538 INFO:tasks.workunit.client.0.smithi163.stderr: 50150K .......... .......... .......... .......... .......... 29% 27.6M 6s 2023-07-02T14:02:42.540 INFO:tasks.workunit.client.0.smithi163.stderr: 50200K .......... .......... .......... .......... .......... 29% 21.2M 6s 2023-07-02T14:02:42.541 INFO:tasks.workunit.client.0.smithi163.stderr: 50250K .......... .......... .......... .......... .......... 29% 26.1M 6s 2023-07-02T14:02:42.545 INFO:tasks.workunit.client.0.smithi163.stderr: 50300K .......... .......... .......... .......... .......... 29% 15.6M 6s 2023-07-02T14:02:42.547 INFO:tasks.workunit.client.0.smithi163.stderr: 50350K .......... .......... .......... .......... .......... 29% 18.3M 6s 2023-07-02T14:02:42.550 INFO:tasks.workunit.client.0.smithi163.stderr: 50400K .......... .......... .......... .......... .......... 29% 17.0M 6s 2023-07-02T14:02:42.552 INFO:tasks.workunit.client.0.smithi163.stderr: 50450K .......... .......... .......... .......... .......... 29% 29.5M 6s 2023-07-02T14:02:42.553 INFO:tasks.workunit.client.0.smithi163.stderr: 50500K .......... .......... .......... .......... .......... 29% 36.1M 6s 2023-07-02T14:02:42.554 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:42.554+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:42.556 INFO:tasks.workunit.client.0.smithi163.stderr: 50550K .......... .......... .......... .......... .......... 30% 20.4M 6s 2023-07-02T14:02:42.559 INFO:tasks.workunit.client.0.smithi163.stderr: 50600K .......... .......... .......... .......... .......... 30% 16.3M 6s 2023-07-02T14:02:42.561 INFO:tasks.workunit.client.0.smithi163.stderr: 50650K .......... .......... .......... .......... .......... 30% 20.8M 6s 2023-07-02T14:02:42.564 INFO:tasks.workunit.client.0.smithi163.stderr: 50700K .......... .......... .......... .......... .......... 30% 19.0M 6s 2023-07-02T14:02:42.566 INFO:tasks.workunit.client.0.smithi163.stderr: 50750K .......... .......... .......... .......... .......... 30% 21.8M 6s 2023-07-02T14:02:42.568 INFO:tasks.workunit.client.0.smithi163.stderr: 50800K .......... .......... .......... .......... .......... 30% 24.6M 6s 2023-07-02T14:02:42.571 INFO:tasks.workunit.client.0.smithi163.stderr: 50850K .......... .......... .......... .......... .......... 30% 16.4M 6s 2023-07-02T14:02:42.573 INFO:tasks.workunit.client.0.smithi163.stderr: 50900K .......... .......... .......... .......... .......... 30% 19.9M 6s 2023-07-02T14:02:42.576 INFO:tasks.workunit.client.0.smithi163.stderr: 50950K .......... .......... .......... .......... .......... 30% 19.7M 6s 2023-07-02T14:02:42.578 INFO:tasks.workunit.client.0.smithi163.stderr: 51000K .......... .......... .......... .......... .......... 30% 27.9M 6s 2023-07-02T14:02:42.580 INFO:tasks.workunit.client.0.smithi163.stderr: 51050K .......... .......... .......... .......... .......... 30% 19.1M 6s 2023-07-02T14:02:42.583 INFO:tasks.workunit.client.0.smithi163.stderr: 51100K .......... .......... .......... .......... .......... 30% 17.5M 6s 2023-07-02T14:02:42.586 INFO:tasks.workunit.client.0.smithi163.stderr: 51150K .......... .......... .......... .......... .......... 30% 15.7M 6s 2023-07-02T14:02:42.588 INFO:tasks.workunit.client.0.smithi163.stderr: 51200K .......... .......... .......... .......... .......... 30% 26.4M 6s 2023-07-02T14:02:42.591 INFO:tasks.workunit.client.0.smithi163.stderr: 51250K .......... .......... .......... .......... .......... 30% 17.3M 6s 2023-07-02T14:02:42.593 INFO:tasks.workunit.client.0.smithi163.stderr: 51300K .......... .......... .......... .......... .......... 30% 24.8M 6s 2023-07-02T14:02:42.595 INFO:tasks.workunit.client.0.smithi163.stderr: 51350K .......... .......... .......... .......... .......... 30% 19.0M 6s 2023-07-02T14:02:42.597 INFO:tasks.workunit.client.0.smithi163.stderr: 51400K .......... .......... .......... .......... .......... 30% 22.7M 6s 2023-07-02T14:02:42.600 INFO:tasks.workunit.client.0.smithi163.stderr: 51450K .......... .......... .......... .......... .......... 30% 20.4M 6s 2023-07-02T14:02:42.602 INFO:tasks.workunit.client.0.smithi163.stderr: 51500K .......... .......... .......... .......... .......... 30% 20.7M 6s 2023-07-02T14:02:42.604 INFO:tasks.workunit.client.0.smithi163.stderr: 51550K .......... .......... .......... .......... .......... 30% 22.6M 6s 2023-07-02T14:02:42.607 INFO:tasks.workunit.client.0.smithi163.stderr: 51600K .......... .......... .......... .......... .......... 30% 18.7M 6s 2023-07-02T14:02:42.609 INFO:tasks.workunit.client.0.smithi163.stderr: 51650K .......... .......... .......... .......... .......... 30% 18.1M 6s 2023-07-02T14:02:42.612 INFO:tasks.workunit.client.0.smithi163.stderr: 51700K .......... .......... .......... .......... .......... 30% 21.4M 6s 2023-07-02T14:02:42.614 INFO:tasks.workunit.client.0.smithi163.stderr: 51750K .......... .......... .......... .......... .......... 30% 23.8M 6s 2023-07-02T14:02:42.617 INFO:tasks.workunit.client.0.smithi163.stderr: 51800K .......... .......... .......... .......... .......... 30% 18.0M 6s 2023-07-02T14:02:42.619 INFO:tasks.workunit.client.0.smithi163.stderr: 51850K .......... .......... .......... .......... .......... 30% 21.3M 6s 2023-07-02T14:02:42.620 INFO:tasks.workunit.client.0.smithi163.stderr: 51900K .......... .......... .......... .......... .......... 30% 31.8M 6s 2023-07-02T14:02:42.624 INFO:tasks.workunit.client.0.smithi163.stderr: 51950K .......... .......... .......... .......... .......... 30% 14.4M 6s 2023-07-02T14:02:42.627 INFO:tasks.workunit.client.0.smithi163.stderr: 52000K .......... .......... .......... .......... .......... 30% 15.8M 6s 2023-07-02T14:02:42.628 INFO:tasks.workunit.client.0.smithi163.stderr: 52050K .......... .......... .......... .......... .......... 30% 31.0M 6s 2023-07-02T14:02:42.636 INFO:tasks.workunit.client.0.smithi163.stderr: 52100K .......... .......... .......... .......... .......... 30% 8.02M 6s 2023-07-02T14:02:42.636 INFO:tasks.workunit.client.0.smithi163.stderr: 52150K .......... .......... .......... .......... .......... 30% 122M 6s 2023-07-02T14:02:42.636 INFO:tasks.workunit.client.0.smithi163.stderr: 52200K .......... .......... .......... .......... .......... 30% 53.3M 6s 2023-07-02T14:02:42.637 INFO:tasks.workunit.client.0.smithi163.stderr: 52250K .......... .......... .......... .......... .......... 31% 38.9M 6s 2023-07-02T14:02:42.640 INFO:tasks.workunit.client.0.smithi163.stderr: 52300K .......... .......... .......... .......... .......... 31% 21.6M 6s 2023-07-02T14:02:42.643 INFO:tasks.workunit.client.0.smithi163.stderr: 52350K .......... .......... .......... .......... .......... 31% 15.6M 6s 2023-07-02T14:02:42.646 INFO:tasks.workunit.client.0.smithi163.stderr: 52400K .......... .......... .......... .......... .......... 31% 16.0M 6s 2023-07-02T14:02:42.648 INFO:tasks.workunit.client.0.smithi163.stderr: 52450K .......... .......... .......... .......... .......... 31% 22.0M 6s 2023-07-02T14:02:42.650 INFO:tasks.workunit.client.0.smithi163.stderr: 52500K .......... .......... .......... .......... .......... 31% 21.9M 6s 2023-07-02T14:02:42.652 INFO:tasks.workunit.client.0.smithi163.stderr: 52550K .......... .......... .......... .......... .......... 31% 25.4M 6s 2023-07-02T14:02:42.653 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:42.654 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:42.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:42.654 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:02:42.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.654 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:380 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'10, mlcod=40'1)", 2023-07-02T14:02:42.654 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.309796+0000", 2023-07-02T14:02:42.654 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.3426928770000002, 2023-07-02T14:02:42.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034177320000000001, 2023-07-02T14:02:42.655 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:42.655 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.655 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:42.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:42.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309796+0000", 2023-07-02T14:02:42.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:42.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:42.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309796+0000", 2023-07-02T14:02:42.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:42.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:42.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309796+0000", 2023-07-02T14:02:42.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4694999999999997e-05 2023-07-02T14:02:42.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:42.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309851+0000", 2023-07-02T14:02:42.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6890000000000004e-06 2023-07-02T14:02:42.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:42.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309858+0000", 2023-07-02T14:02:42.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7659999999999998e-06 2023-07-02T14:02:42.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309863+0000", 2023-07-02T14:02:42.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001070164 2023-07-02T14:02:42.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:42.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310933+0000", 2023-07-02T14:02:42.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1287000000000001e-05 2023-07-02T14:02:42.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:42.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310954+0000", 2023-07-02T14:02:42.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022261059999999998 2023-07-02T14:02:42.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:42.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313181+0000", 2023-07-02T14:02:42.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3494000000000002e-05 2023-07-02T14:02:42.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:42.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313214+0000", 2023-07-02T14:02:42.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034177320000000001 2023-07-02T14:02:42.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.662 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.662 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:381 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'11, mlcod=40'1)", 2023-07-02T14:02:42.663 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.310792+0000", 2023-07-02T14:02:42.663 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.3416967290000001, 2023-07-02T14:02:42.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038397629999999999, 2023-07-02T14:02:42.663 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:42.663 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.663 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:42.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:42.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310792+0000", 2023-07-02T14:02:42.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:42.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:42.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310792+0000", 2023-07-02T14:02:42.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:42.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310792+0000", 2023-07-02T14:02:42.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9630999999999999e-05 2023-07-02T14:02:42.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:42.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310851+0000", 2023-07-02T14:02:42.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3351e-05 2023-07-02T14:02:42.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:42.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310865+0000", 2023-07-02T14:02:42.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7459999999999995e-06 2023-07-02T14:02:42.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310874+0000", 2023-07-02T14:02:42.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000478457 2023-07-02T14:02:42.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:42.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311353+0000", 2023-07-02T14:02:42.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3498000000000001e-05 2023-07-02T14:02:42.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:42.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311376+0000", 2023-07-02T14:02:42.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003216464 2023-07-02T14:02:42.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:42.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314593+0000", 2023-07-02T14:02:42.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9368e-05 2023-07-02T14:02:42.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:42.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314632+0000", 2023-07-02T14:02:42.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038397629999999999 2023-07-02T14:02:42.670 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.670 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.670 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.671 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:382 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'12, mlcod=40'3)", 2023-07-02T14:02:42.671 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311736+0000", 2023-07-02T14:02:42.671 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.3407530369999998, 2023-07-02T14:02:42.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029255800000000001, 2023-07-02T14:02:42.671 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:42.671 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.671 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:42.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:42.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311736+0000", 2023-07-02T14:02:42.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:42.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:42.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311736+0000", 2023-07-02T14:02:42.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:42.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311735+0000", 2023-07-02T14:02:42.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9895000000000005e-05 2023-07-02T14:02:42.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:42.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311835+0000", 2023-07-02T14:02:42.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2548000000000001e-05 2023-07-02T14:02:42.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:42.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311848+0000", 2023-07-02T14:02:42.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5810000000000002e-06 2023-07-02T14:02:42.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311854+0000", 2023-07-02T14:02:42.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4743999999999997e-05 2023-07-02T14:02:42.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:42.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311899+0000", 2023-07-02T14:02:42.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.808e-05 2023-07-02T14:02:42.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:42.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311917+0000", 2023-07-02T14:02:42.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027195069999999999 2023-07-02T14:02:42.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:42.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314637+0000", 2023-07-02T14:02:42.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5080999999999999e-05 2023-07-02T14:02:42.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:42.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314662+0000", 2023-07-02T14:02:42.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029255800000000001 2023-07-02T14:02:42.678 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.678 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.678 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.679 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:386 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'16, mlcod=40'10)", 2023-07-02T14:02:42.679 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.317468+0000", 2023-07-02T14:02:42.679 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.3350208690000001, 2023-07-02T14:02:42.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027835989999999999, 2023-07-02T14:02:42.679 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:42.679 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.680 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:42.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:42.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317468+0000", 2023-07-02T14:02:42.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:42.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:42.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317468+0000", 2023-07-02T14:02:42.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:42.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:42.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317468+0000", 2023-07-02T14:02:42.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8161999999999998e-05 2023-07-02T14:02:42.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:42.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317506+0000", 2023-07-02T14:02:42.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5590000000000002e-06 2023-07-02T14:02:42.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:42.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317512+0000", 2023-07-02T14:02:42.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.186e-06 2023-07-02T14:02:42.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317518+0000", 2023-07-02T14:02:42.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4561999999999996e-05 2023-07-02T14:02:42.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:42.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317582+0000", 2023-07-02T14:02:42.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1291e-05 2023-07-02T14:02:42.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:42.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317603+0000", 2023-07-02T14:02:42.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026133839999999998 2023-07-02T14:02:42.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:42.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320217+0000", 2023-07-02T14:02:42.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5111999999999999e-05 2023-07-02T14:02:42.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:42.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320252+0000", 2023-07-02T14:02:42.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027835989999999999 2023-07-02T14:02:42.686 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.686 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.686 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.687 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:387 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'17, mlcod=40'10)", 2023-07-02T14:02:42.687 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.317892+0000", 2023-07-02T14:02:42.687 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.3345974749999998, 2023-07-02T14:02:42.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031330540000000001, 2023-07-02T14:02:42.687 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:42.687 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.688 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:42.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:42.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317892+0000", 2023-07-02T14:02:42.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:42.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:42.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317892+0000", 2023-07-02T14:02:42.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:42.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317891+0000", 2023-07-02T14:02:42.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1758e-05 2023-07-02T14:02:42.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:42.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317963+0000", 2023-07-02T14:02:42.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5237e-05 2023-07-02T14:02:42.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:42.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317978+0000", 2023-07-02T14:02:42.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9710000000000001e-06 2023-07-02T14:02:42.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317984+0000", 2023-07-02T14:02:42.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4039999999999998e-05 2023-07-02T14:02:42.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:42.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318028+0000", 2023-07-02T14:02:42.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8092000000000001e-05 2023-07-02T14:02:42.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:42.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318056+0000", 2023-07-02T14:02:42.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029412959999999999 2023-07-02T14:02:42.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:42.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320997+0000", 2023-07-02T14:02:42.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7416999999999999e-05 2023-07-02T14:02:42.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:42.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321025+0000", 2023-07-02T14:02:42.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031330540000000001 2023-07-02T14:02:42.694 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.694 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.694 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.695 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:402 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'32, mlcod=40'17)", 2023-07-02T14:02:42.695 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.331582+0000", 2023-07-02T14:02:42.695 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.3209076080000002, 2023-07-02T14:02:42.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002927172, 2023-07-02T14:02:42.695 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:42.695 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.695 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:42.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:42.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331582+0000", 2023-07-02T14:02:42.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:42.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:42.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331582+0000", 2023-07-02T14:02:42.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:42.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331581+0000", 2023-07-02T14:02:42.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2337999999999998e-05 2023-07-02T14:02:42.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:42.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331643+0000", 2023-07-02T14:02:42.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0506000000000001e-05 2023-07-02T14:02:42.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:42.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331653+0000", 2023-07-02T14:02:42.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5650000000000003e-06 2023-07-02T14:02:42.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331658+0000", 2023-07-02T14:02:42.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3337999999999998e-05 2023-07-02T14:02:42.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:42.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331691+0000", 2023-07-02T14:02:42.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4527e-05 2023-07-02T14:02:42.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:42.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331706+0000", 2023-07-02T14:02:42.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027586830000000001 2023-07-02T14:02:42.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:42.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334464+0000", 2023-07-02T14:02:42.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4311000000000003e-05 2023-07-02T14:02:42.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:42.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334509+0000", 2023-07-02T14:02:42.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002927172 2023-07-02T14:02:42.702 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.702 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.702 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.703 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:403 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'33, mlcod=40'20)", 2023-07-02T14:02:42.703 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.331850+0000", 2023-07-02T14:02:42.703 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.3206395030000002, 2023-07-02T14:02:42.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033780419999999999, 2023-07-02T14:02:42.703 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:42.703 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.703 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:42.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:42.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331850+0000", 2023-07-02T14:02:42.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:42.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:42.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331850+0000", 2023-07-02T14:02:42.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:42.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:42.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331849+0000", 2023-07-02T14:02:42.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015039599999999999 2023-07-02T14:02:42.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:42.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331999+0000", 2023-07-02T14:02:42.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7082e-05 2023-07-02T14:02:42.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:42.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.332017+0000", 2023-07-02T14:02:42.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3400000000000003e-06 2023-07-02T14:02:42.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.332023+0000", 2023-07-02T14:02:42.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8889999999999999e-05 2023-07-02T14:02:42.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:42.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.332102+0000", 2023-07-02T14:02:42.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2781000000000003e-05 2023-07-02T14:02:42.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:42.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.332145+0000", 2023-07-02T14:02:42.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030463410000000002 2023-07-02T14:02:42.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:42.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.335191+0000", 2023-07-02T14:02:42.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.684e-05 2023-07-02T14:02:42.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:42.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.335228+0000", 2023-07-02T14:02:42.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033780419999999999 2023-07-02T14:02:42.710 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.710 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.710 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.710 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:405 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'35, mlcod=40'25)", 2023-07-02T14:02:42.711 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.333577+0000", 2023-07-02T14:02:42.711 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.3189123349999998, 2023-07-02T14:02:42.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002930219, 2023-07-02T14:02:42.711 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:42.711 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.711 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:42.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:42.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333577+0000", 2023-07-02T14:02:42.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:42.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:42.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333577+0000", 2023-07-02T14:02:42.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:42.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333576+0000", 2023-07-02T14:02:42.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4240000000000002e-05 2023-07-02T14:02:42.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:42.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333630+0000", 2023-07-02T14:02:42.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2962999999999999e-05 2023-07-02T14:02:42.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:42.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333643+0000", 2023-07-02T14:02:42.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8390000000000001e-06 2023-07-02T14:02:42.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333650+0000", 2023-07-02T14:02:42.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0926000000000005e-05 2023-07-02T14:02:42.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:42.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333721+0000", 2023-07-02T14:02:42.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2921e-05 2023-07-02T14:02:42.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:42.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333744+0000", 2023-07-02T14:02:42.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027376240000000001 2023-07-02T14:02:42.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:42.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336481+0000", 2023-07-02T14:02:42.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.565e-05 2023-07-02T14:02:42.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:42.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336507+0000", 2023-07-02T14:02:42.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002930219 2023-07-02T14:02:42.718 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.718 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.718 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.718 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:406 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'36, mlcod=40'25)", 2023-07-02T14:02:42.718 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.334125+0000", 2023-07-02T14:02:42.719 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.318364571, 2023-07-02T14:02:42.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003294352, 2023-07-02T14:02:42.719 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:42.719 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.719 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:42.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:42.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334125+0000", 2023-07-02T14:02:42.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:42.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:42.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334125+0000", 2023-07-02T14:02:42.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:42.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334124+0000", 2023-07-02T14:02:42.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106912 2023-07-02T14:02:42.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:42.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334231+0000", 2023-07-02T14:02:42.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6602000000000001e-05 2023-07-02T14:02:42.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:42.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334247+0000", 2023-07-02T14:02:42.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7350000000000001e-06 2023-07-02T14:02:42.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334253+0000", 2023-07-02T14:02:42.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000473144 2023-07-02T14:02:42.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:42.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334726+0000", 2023-07-02T14:02:42.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4657000000000001e-05 2023-07-02T14:02:42.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:42.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334751+0000", 2023-07-02T14:02:42.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026279250000000001 2023-07-02T14:02:42.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:42.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337379+0000", 2023-07-02T14:02:42.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0111999999999998e-05 2023-07-02T14:02:42.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:42.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337419+0000", 2023-07-02T14:02:42.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003294352 2023-07-02T14:02:42.726 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.726 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.726 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.726 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:426 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'42, mlcod=40'38)", 2023-07-02T14:02:42.726 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.340864+0000", 2023-07-02T14:02:42.726 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.3116252500000001, 2023-07-02T14:02:42.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065109469999999996, 2023-07-02T14:02:42.727 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:42.727 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.727 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:42.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:42.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.340864+0000", 2023-07-02T14:02:42.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:42.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:42.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.340864+0000", 2023-07-02T14:02:42.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:42.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.340863+0000", 2023-07-02T14:02:42.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019354159999999999 2023-07-02T14:02:42.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:42.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342799+0000", 2023-07-02T14:02:42.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5824e-05 2023-07-02T14:02:42.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:42.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342814+0000", 2023-07-02T14:02:42.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:02:42.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342822+0000", 2023-07-02T14:02:42.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7465999999999994e-05 2023-07-02T14:02:42.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:42.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342889+0000", 2023-07-02T14:02:42.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5068000000000001e-05 2023-07-02T14:02:42.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:42.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342914+0000", 2023-07-02T14:02:42.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044200969999999996 2023-07-02T14:02:42.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:42.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347334+0000", 2023-07-02T14:02:42.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0711999999999999e-05 2023-07-02T14:02:42.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:42.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347375+0000", 2023-07-02T14:02:42.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065109469999999996 2023-07-02T14:02:42.733 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.734 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.734 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:431 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'4, mlcod=40'2)", 2023-07-02T14:02:42.734 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.350267+0000", 2023-07-02T14:02:42.734 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.3022217839999999, 2023-07-02T14:02:42.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077388420000000001, 2023-07-02T14:02:42.735 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:42.735 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.735 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:42.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:42.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350267+0000", 2023-07-02T14:02:42.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:42.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:42.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350267+0000", 2023-07-02T14:02:42.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:42.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350266+0000", 2023-07-02T14:02:42.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024191569999999999 2023-07-02T14:02:42.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:42.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352685+0000", 2023-07-02T14:02:42.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0013e-05 2023-07-02T14:02:42.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:42.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352706+0000", 2023-07-02T14:02:42.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3250000000000008e-06 2023-07-02T14:02:42.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352714+0000", 2023-07-02T14:02:42.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9648000000000003e-05 2023-07-02T14:02:42.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:42.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352763+0000", 2023-07-02T14:02:42.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4496999999999999e-05 2023-07-02T14:02:42.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:42.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352788+0000", 2023-07-02T14:02:42.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051724880000000003 2023-07-02T14:02:42.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:42.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357960+0000", 2023-07-02T14:02:42.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5802e-05 2023-07-02T14:02:42.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:42.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358006+0000", 2023-07-02T14:02:42.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077388420000000001 2023-07-02T14:02:42.741 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.741 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.742 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.742 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:433 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'5, mlcod=40'2)", 2023-07-02T14:02:42.742 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.352729+0000", 2023-07-02T14:02:42.742 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.29975978, 2023-07-02T14:02:42.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0053048619999999996, 2023-07-02T14:02:42.742 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:42.743 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.743 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:42.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:42.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352729+0000", 2023-07-02T14:02:42.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:42.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:42.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352729+0000", 2023-07-02T14:02:42.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:42.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352728+0000", 2023-07-02T14:02:42.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4173e-05 2023-07-02T14:02:42.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:42.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352763+0000", 2023-07-02T14:02:42.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7689999999999996e-06 2023-07-02T14:02:42.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:42.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352770+0000", 2023-07-02T14:02:42.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6689999999999997e-06 2023-07-02T14:02:42.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352775+0000", 2023-07-02T14:02:42.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025486380000000002 2023-07-02T14:02:42.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:42.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.355324+0000", 2023-07-02T14:02:42.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7258999999999999e-05 2023-07-02T14:02:42.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:42.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.355341+0000", 2023-07-02T14:02:42.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026703569999999999 2023-07-02T14:02:42.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:42.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358011+0000", 2023-07-02T14:02:42.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3039000000000001e-05 2023-07-02T14:02:42.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:42.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358034+0000", 2023-07-02T14:02:42.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0053048619999999996 2023-07-02T14:02:42.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.749 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.750 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:439 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'6, mlcod=40'4)", 2023-07-02T14:02:42.750 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.359868+0000", 2023-07-02T14:02:42.750 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.292621478, 2023-07-02T14:02:42.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031584299999999998, 2023-07-02T14:02:42.750 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:42.750 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.751 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:42.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:42.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.359868+0000", 2023-07-02T14:02:42.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:42.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:42.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.359868+0000", 2023-07-02T14:02:42.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:02:42.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:42.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.359866+0000", 2023-07-02T14:02:42.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00068265100000000005 2023-07-02T14:02:42.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:42.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.360549+0000", 2023-07-02T14:02:42.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9236999999999999e-05 2023-07-02T14:02:42.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:42.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.360568+0000", 2023-07-02T14:02:42.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5529999999999998e-06 2023-07-02T14:02:42.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.360578+0000", 2023-07-02T14:02:42.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6258000000000003e-05 2023-07-02T14:02:42.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:42.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.360624+0000", 2023-07-02T14:02:42.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6570000000000001e-05 2023-07-02T14:02:42.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:42.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.360651+0000", 2023-07-02T14:02:42.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023451359999999998 2023-07-02T14:02:42.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:42.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.362996+0000", 2023-07-02T14:02:42.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0467000000000002e-05 2023-07-02T14:02:42.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:42.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363026+0000", 2023-07-02T14:02:42.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031584299999999998 2023-07-02T14:02:42.757 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.757 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.757 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.758 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:440 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'7, mlcod=40'4)", 2023-07-02T14:02:42.758 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.361026+0000", 2023-07-02T14:02:42.758 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.291463008, 2023-07-02T14:02:42.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029947889999999999, 2023-07-02T14:02:42.758 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:42.758 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.758 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:42.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:42.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361026+0000", 2023-07-02T14:02:42.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:42.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:42.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361026+0000", 2023-07-02T14:02:42.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:42.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361025+0000", 2023-07-02T14:02:42.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013836740000000001 2023-07-02T14:02:42.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:42.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.362409+0000", 2023-07-02T14:02:42.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5155e-05 2023-07-02T14:02:42.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:42.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.362424+0000", 2023-07-02T14:02:42.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1170000000000004e-06 2023-07-02T14:02:42.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.362432+0000", 2023-07-02T14:02:42.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9370999999999999e-05 2023-07-02T14:02:42.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:42.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.362482+0000", 2023-07-02T14:02:42.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5891999999999999e-05 2023-07-02T14:02:42.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:42.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.362508+0000", 2023-07-02T14:02:42.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014803920000000001 2023-07-02T14:02:42.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:42.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363988+0000", 2023-07-02T14:02:42.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2923999999999999e-05 2023-07-02T14:02:42.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:42.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364021+0000", 2023-07-02T14:02:42.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029947889999999999 2023-07-02T14:02:42.765 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.765 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.765 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.765 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:451 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'8, mlcod=40'6)", 2023-07-02T14:02:42.766 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.365605+0000", 2023-07-02T14:02:42.766 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.2868840439999998, 2023-07-02T14:02:42.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033622349999999999, 2023-07-02T14:02:42.766 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:42.766 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.766 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:42.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:42.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365605+0000", 2023-07-02T14:02:42.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:42.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:42.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365605+0000", 2023-07-02T14:02:42.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:02:42.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:42.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365604+0000", 2023-07-02T14:02:42.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00056647700000000002 2023-07-02T14:02:42.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:42.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.366170+0000", 2023-07-02T14:02:42.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2206000000000001e-05 2023-07-02T14:02:42.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:42.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.366192+0000", 2023-07-02T14:02:42.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1044e-05 2023-07-02T14:02:42.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.366203+0000", 2023-07-02T14:02:42.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2663000000000005e-05 2023-07-02T14:02:42.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:42.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.366276+0000", 2023-07-02T14:02:42.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7101999999999999e-05 2023-07-02T14:02:42.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:42.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.366303+0000", 2023-07-02T14:02:42.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026248769999999999 2023-07-02T14:02:42.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:42.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368928+0000", 2023-07-02T14:02:42.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9267000000000002e-05 2023-07-02T14:02:42.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:42.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368967+0000", 2023-07-02T14:02:42.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033622349999999999 2023-07-02T14:02:42.773 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.773 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.773 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.773 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:455 2.5 e40/14 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:head v 40'12, mlcod=40'10)", 2023-07-02T14:02:42.773 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.376227+0000", 2023-07-02T14:02:42.774 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.2762625060000001, 2023-07-02T14:02:42.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012681951, 2023-07-02T14:02:42.774 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:42.774 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.774 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:42.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:42.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.376227+0000", 2023-07-02T14:02:42.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:42.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:42.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.376227+0000", 2023-07-02T14:02:42.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:42.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.376226+0000", 2023-07-02T14:02:42.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040198860000000003 2023-07-02T14:02:42.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:42.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380245+0000", 2023-07-02T14:02:42.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7640000000000001e-05 2023-07-02T14:02:42.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:42.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380263+0000", 2023-07-02T14:02:42.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3299999999999999e-06 2023-07-02T14:02:42.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380271+0000", 2023-07-02T14:02:42.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4322e-05 2023-07-02T14:02:42.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:42.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380316+0000", 2023-07-02T14:02:42.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7682000000000001e-05 2023-07-02T14:02:42.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:42.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380333+0000", 2023-07-02T14:02:42.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085223439999999994 2023-07-02T14:02:42.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:42.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.388856+0000", 2023-07-02T14:02:42.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2930999999999999e-05 2023-07-02T14:02:42.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:42.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.388909+0000", 2023-07-02T14:02:42.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012681951 2023-07-02T14:02:42.781 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.781 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.781 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.781 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:456 2.5 e40/14 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:head v 40'13, mlcod=40'10)", 2023-07-02T14:02:42.781 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.382266+0000", 2023-07-02T14:02:42.781 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.2702228789999999, 2023-07-02T14:02:42.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075568149999999997, 2023-07-02T14:02:42.782 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:42.782 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.782 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:42.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:42.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382266+0000", 2023-07-02T14:02:42.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:42.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:42.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382266+0000", 2023-07-02T14:02:42.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:02:42.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:42.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382264+0000", 2023-07-02T14:02:42.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012720599999999999 2023-07-02T14:02:42.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:42.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382391+0000", 2023-07-02T14:02:42.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7598e-05 2023-07-02T14:02:42.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:42.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382409+0000", 2023-07-02T14:02:42.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8660000000000001e-06 2023-07-02T14:02:42.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382419+0000", 2023-07-02T14:02:42.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001837403 2023-07-02T14:02:42.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:42.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384256+0000", 2023-07-02T14:02:42.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5757000000000001e-05 2023-07-02T14:02:42.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:42.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384282+0000", 2023-07-02T14:02:42.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055044730000000002 2023-07-02T14:02:42.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:42.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389787+0000", 2023-07-02T14:02:42.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6622000000000003e-05 2023-07-02T14:02:42.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:42.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389823+0000", 2023-07-02T14:02:42.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075568149999999997 2023-07-02T14:02:42.788 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.789 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.789 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.789 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:457 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'14, mlcod=40'10)", 2023-07-02T14:02:42.789 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.382901+0000", 2023-07-02T14:02:42.789 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.269587907, 2023-07-02T14:02:42.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0069428320000000003, 2023-07-02T14:02:42.790 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:42.790 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.790 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:42.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:42.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382901+0000", 2023-07-02T14:02:42.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:42.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:42.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382901+0000", 2023-07-02T14:02:42.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:02:42.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:42.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382900+0000", 2023-07-02T14:02:42.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7738999999999994e-05 2023-07-02T14:02:42.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:42.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382998+0000", 2023-07-02T14:02:42.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6769000000000002e-05 2023-07-02T14:02:42.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:42.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383014+0000", 2023-07-02T14:02:42.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0073e-05 2023-07-02T14:02:42.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383025+0000", 2023-07-02T14:02:42.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016687410000000001 2023-07-02T14:02:42.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:42.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384693+0000", 2023-07-02T14:02:42.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4108999999999999e-05 2023-07-02T14:02:42.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:42.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384717+0000", 2023-07-02T14:02:42.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051086830000000001 2023-07-02T14:02:42.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:42.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389826+0000", 2023-07-02T14:02:42.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8059000000000001e-05 2023-07-02T14:02:42.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:42.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389844+0000", 2023-07-02T14:02:42.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0069428320000000003 2023-07-02T14:02:42.796 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.796 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.797 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.797 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:479 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'15, mlcod=40'10)", 2023-07-02T14:02:42.797 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.383348+0000", 2023-07-02T14:02:42.797 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.2691409490000001, 2023-07-02T14:02:42.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065068890000000001, 2023-07-02T14:02:42.797 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:42.798 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.798 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:42.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:42.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383348+0000", 2023-07-02T14:02:42.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:42.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:42.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383348+0000", 2023-07-02T14:02:42.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:42.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383347+0000", 2023-07-02T14:02:42.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000422123 2023-07-02T14:02:42.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:42.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383769+0000", 2023-07-02T14:02:42.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1688e-05 2023-07-02T14:02:42.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:42.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383791+0000", 2023-07-02T14:02:42.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0237e-05 2023-07-02T14:02:42.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383801+0000", 2023-07-02T14:02:42.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013799940000000001 2023-07-02T14:02:42.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:42.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.385181+0000", 2023-07-02T14:02:42.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3047000000000001e-05 2023-07-02T14:02:42.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:42.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.385204+0000", 2023-07-02T14:02:42.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046417339999999998 2023-07-02T14:02:42.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:42.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389846+0000", 2023-07-02T14:02:42.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1800000000000002e-06 2023-07-02T14:02:42.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:42.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389855+0000", 2023-07-02T14:02:42.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065068890000000001 2023-07-02T14:02:42.804 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.804 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.804 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.805 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:484 2.5 e40/14 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:head v 40'16, mlcod=40'10)", 2023-07-02T14:02:42.805 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.384371+0000", 2023-07-02T14:02:42.805 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.2681178860000002, 2023-07-02T14:02:42.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054937609999999998, 2023-07-02T14:02:42.805 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:42.805 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:42.806 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:42.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:42.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384371+0000", 2023-07-02T14:02:42.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:42.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:42.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384371+0000", 2023-07-02T14:02:42.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:42.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:42.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384370+0000", 2023-07-02T14:02:42.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00041928199999999999 2023-07-02T14:02:42.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:42.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384790+0000", 2023-07-02T14:02:42.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1742999999999999e-05 2023-07-02T14:02:42.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:42.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384801+0000", 2023-07-02T14:02:42.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0380000000000002e-06 2023-07-02T14:02:42.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:42.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384808+0000", 2023-07-02T14:02:42.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010660999999999999 2023-07-02T14:02:42.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:42.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.385874+0000", 2023-07-02T14:02:42.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3745000000000003e-05 2023-07-02T14:02:42.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:42.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.385908+0000", 2023-07-02T14:02:42.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039487560000000003 2023-07-02T14:02:42.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:42.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389857+0000", 2023-07-02T14:02:42.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1019999999999997e-06 2023-07-02T14:02:42.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:42.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:42.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:42.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389865+0000", 2023-07-02T14:02:42.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054937609999999998 2023-07-02T14:02:42.812 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.813 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.813 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.813 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:42.813 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:42.813 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:42.815 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:02:42.816 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:42.813+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:42.845 INFO:tasks.workunit.client.0.smithi163.stderr: 52600K .......... .......... .......... .......... .......... 31% 17.7M 6s 2023-07-02T14:02:42.846 INFO:tasks.workunit.client.0.smithi163.stderr: 52650K .......... .......... .......... .......... .......... 31% 29.4M 6s 2023-07-02T14:02:42.846 INFO:tasks.workunit.client.0.smithi163.stderr: 52700K .......... .......... .......... .......... .......... 31% 14.5M 6s 2023-07-02T14:02:42.846 INFO:tasks.workunit.client.0.smithi163.stderr: 52750K .......... .......... .......... .......... .......... 31% 17.2M 6s 2023-07-02T14:02:42.846 INFO:tasks.workunit.client.0.smithi163.stderr: 52800K .......... .......... .......... .......... .......... 31% 31.4M 6s 2023-07-02T14:02:42.846 INFO:tasks.workunit.client.0.smithi163.stderr: 52850K .......... .......... .......... .......... .......... 31% 18.6M 6s 2023-07-02T14:02:42.846 INFO:tasks.workunit.client.0.smithi163.stderr: 52900K .......... .......... .......... .......... .......... 31% 16.9M 6s 2023-07-02T14:02:42.846 INFO:tasks.workunit.client.0.smithi163.stderr: 52950K .......... .......... .......... .......... .......... 31% 30.2M 6s 2023-07-02T14:02:42.847 INFO:tasks.workunit.client.0.smithi163.stderr: 53000K .......... .......... .......... .......... .......... 31% 13.7M 6s 2023-07-02T14:02:42.847 INFO:tasks.workunit.client.0.smithi163.stderr: 53050K .......... .......... .......... .......... .......... 31% 27.6M 6s 2023-07-02T14:02:42.847 INFO:tasks.workunit.client.0.smithi163.stderr: 53100K .......... .......... .......... .......... .......... 31% 26.6M 6s 2023-07-02T14:02:42.847 INFO:tasks.workunit.client.0.smithi163.stderr: 53150K .......... .......... .......... .......... .......... 31% 14.1M 6s 2023-07-02T14:02:42.847 INFO:tasks.workunit.client.0.smithi163.stderr: 53200K .......... .......... .......... .......... .......... 31% 23.9M 6s 2023-07-02T14:02:42.847 INFO:tasks.workunit.client.0.smithi163.stderr: 53250K .......... .......... .......... .......... .......... 31% 17.6M 6s 2023-07-02T14:02:42.847 INFO:tasks.workunit.client.0.smithi163.stderr: 53300K .......... .......... .......... .......... .......... 31% 32.5M 6s 2023-07-02T14:02:42.847 INFO:tasks.workunit.client.0.smithi163.stderr: 53350K .......... .......... .......... .......... .......... 31% 15.8M 6s 2023-07-02T14:02:42.848 INFO:tasks.workunit.client.0.smithi163.stderr: 53400K .......... .......... .......... .......... .......... 31% 17.2M 6s 2023-07-02T14:02:42.848 INFO:tasks.workunit.client.0.smithi163.stderr: 53450K .......... .......... .......... .......... .......... 31% 27.5M 6s 2023-07-02T14:02:42.848 INFO:tasks.workunit.client.0.smithi163.stderr: 53500K .......... .......... .......... .......... .......... 31% 16.9M 6s 2023-07-02T14:02:42.848 INFO:tasks.workunit.client.0.smithi163.stderr: 53550K .......... .......... .......... .......... .......... 31% 17.8M 6s 2023-07-02T14:02:42.848 INFO:tasks.workunit.client.0.smithi163.stderr: 53600K .......... .......... .......... .......... .......... 31% 39.7M 6s 2023-07-02T14:02:42.848 INFO:tasks.workunit.client.0.smithi163.stderr: 53650K .......... .......... .......... .......... .......... 31% 14.8M 6s 2023-07-02T14:02:42.848 INFO:tasks.workunit.client.0.smithi163.stderr: 53700K .......... .......... .......... .......... .......... 31% 23.6M 6s 2023-07-02T14:02:42.848 INFO:tasks.workunit.client.0.smithi163.stderr: 53750K .......... .......... .......... .......... .......... 31% 16.8M 6s 2023-07-02T14:02:42.849 INFO:tasks.workunit.client.0.smithi163.stderr: 53800K .......... .......... .......... .......... .......... 31% 37.1M 6s 2023-07-02T14:02:42.849 INFO:tasks.workunit.client.0.smithi163.stderr: 53850K .......... .......... .......... .......... .......... 31% 24.6M 6s 2023-07-02T14:02:42.849 INFO:tasks.workunit.client.0.smithi163.stderr: 53900K .......... .......... .......... .......... .......... 32% 16.2M 6s 2023-07-02T14:02:42.849 INFO:tasks.workunit.client.0.smithi163.stderr: 53950K .......... .......... .......... .......... .......... 32% 16.8M 6s 2023-07-02T14:02:42.849 INFO:tasks.workunit.client.0.smithi163.stderr: 54000K .......... .......... .......... .......... .......... 32% 19.6M 6s 2023-07-02T14:02:42.849 INFO:tasks.workunit.client.0.smithi163.stderr: 54050K .......... .......... .......... .......... .......... 32% 17.5M 6s 2023-07-02T14:02:42.849 INFO:tasks.workunit.client.0.smithi163.stderr: 54100K .......... .......... .......... .......... .......... 32% 49.4M 6s 2023-07-02T14:02:42.849 INFO:tasks.workunit.client.0.smithi163.stderr: 54150K .......... .......... .......... .......... .......... 32% 16.2M 6s 2023-07-02T14:02:42.850 INFO:tasks.workunit.client.0.smithi163.stderr: 54200K .......... .......... .......... .......... .......... 32% 27.1M 6s 2023-07-02T14:02:42.850 INFO:tasks.workunit.client.0.smithi163.stderr: 54250K .......... .......... .......... .......... .......... 32% 12.8M 6s 2023-07-02T14:02:42.850 INFO:tasks.workunit.client.0.smithi163.stderr: 54300K .......... .......... .......... .......... .......... 32% 40.1M 6s 2023-07-02T14:02:42.850 INFO:tasks.workunit.client.0.smithi163.stderr: 54350K .......... .......... .......... .......... .......... 32% 17.4M 6s 2023-07-02T14:02:42.850 INFO:tasks.workunit.client.0.smithi163.stderr: 54400K .......... .......... .......... .......... .......... 32% 17.7M 6s 2023-07-02T14:02:42.850 INFO:tasks.workunit.client.0.smithi163.stderr: 54450K .......... .......... .......... .......... .......... 32% 22.3M 6s 2023-07-02T14:02:42.850 INFO:tasks.workunit.client.0.smithi163.stderr: 54500K .......... .......... .......... .......... .......... 32% 18.3M 6s 2023-07-02T14:02:42.851 INFO:tasks.workunit.client.0.smithi163.stderr: 54550K .......... .......... .......... .......... .......... 32% 33.5M 6s 2023-07-02T14:02:42.851 INFO:tasks.workunit.client.0.smithi163.stderr: 54600K .......... .......... .......... .......... .......... 32% 17.2M 6s 2023-07-02T14:02:42.851 INFO:tasks.workunit.client.0.smithi163.stderr: 54650K .......... .......... .......... .......... .......... 32% 18.3M 6s 2023-07-02T14:02:42.851 INFO:tasks.workunit.client.0.smithi163.stderr: 54700K .......... .......... .......... .......... .......... 32% 19.7M 6s 2023-07-02T14:02:42.851 INFO:tasks.workunit.client.0.smithi163.stderr: 54750K .......... .......... .......... .......... .......... 32% 16.5M 6s 2023-07-02T14:02:42.851 INFO:tasks.workunit.client.0.smithi163.stderr: 54800K .......... .......... .......... .......... .......... 32% 20.6M 6s 2023-07-02T14:02:42.851 INFO:tasks.workunit.client.0.smithi163.stderr: 54850K .......... .......... .......... .......... .......... 32% 19.2M 6s 2023-07-02T14:02:42.851 INFO:tasks.workunit.client.0.smithi163.stderr: 54900K .......... .......... .......... .......... .......... 32% 23.5M 6s 2023-07-02T14:02:42.852 INFO:tasks.workunit.client.0.smithi163.stderr: 54950K .......... .......... .......... .......... .......... 32% 20.9M 6s 2023-07-02T14:02:42.852 INFO:tasks.workunit.client.0.smithi163.stderr: 55000K .......... .......... .......... .......... .......... 32% 26.8M 6s 2023-07-02T14:02:42.852 INFO:tasks.workunit.client.0.smithi163.stderr: 55050K .......... .......... .......... .......... .......... 32% 18.9M 6s 2023-07-02T14:02:42.852 INFO:tasks.workunit.client.0.smithi163.stderr: 55100K .......... .......... .......... .......... .......... 32% 22.7M 6s 2023-07-02T14:02:42.852 INFO:tasks.workunit.client.0.smithi163.stderr: 55150K .......... .......... .......... .......... .......... 32% 17.2M 6s 2023-07-02T14:02:42.852 INFO:tasks.workunit.client.0.smithi163.stderr: 55200K .......... .......... .......... .......... .......... 32% 16.0M 6s 2023-07-02T14:02:42.852 INFO:tasks.workunit.client.0.smithi163.stderr: 55250K .......... .......... .......... .......... .......... 32% 30.2M 6s 2023-07-02T14:02:42.852 INFO:tasks.workunit.client.0.smithi163.stderr: 55300K .......... .......... .......... .......... .......... 32% 23.7M 6s 2023-07-02T14:02:42.853 INFO:tasks.workunit.client.0.smithi163.stderr: 55350K .......... .......... .......... .......... .......... 32% 16.7M 6s 2023-07-02T14:02:42.853 INFO:tasks.workunit.client.0.smithi163.stderr: 55400K .......... .......... .......... .......... .......... 32% 19.9M 6s 2023-07-02T14:02:42.853 INFO:tasks.workunit.client.0.smithi163.stderr: 55450K .......... .......... .......... .......... .......... 32% 29.7M 6s 2023-07-02T14:02:42.853 INFO:tasks.workunit.client.0.smithi163.stderr: 55500K .......... .......... .......... .......... .......... 32% 17.7M 6s 2023-07-02T14:02:42.853 INFO:tasks.workunit.client.0.smithi163.stderr: 55550K .......... .......... .......... .......... .......... 32% 21.0M 6s 2023-07-02T14:02:42.853 INFO:tasks.workunit.client.0.smithi163.stderr: 55600K .......... .......... .......... .......... .......... 33% 22.4M 6s 2023-07-02T14:02:42.853 INFO:tasks.workunit.client.0.smithi163.stderr: 55650K .......... .......... .......... .......... .......... 33% 18.7M 6s 2023-07-02T14:02:42.853 INFO:tasks.workunit.client.0.smithi163.stderr: 55700K .......... .......... .......... .......... .......... 33% 15.0M 6s 2023-07-02T14:02:42.854 INFO:tasks.workunit.client.0.smithi163.stderr: 55750K .......... .......... .......... .......... .......... 33% 38.3M 6s 2023-07-02T14:02:42.854 INFO:tasks.workunit.client.0.smithi163.stderr: 55800K .......... .......... .......... .......... .......... 33% 14.4M 6s 2023-07-02T14:02:42.854 INFO:tasks.workunit.client.0.smithi163.stderr: 55850K .......... .......... .......... .......... .......... 33% 22.7M 6s 2023-07-02T14:02:42.854 INFO:tasks.workunit.client.0.smithi163.stderr: 55900K .......... .......... .......... .......... .......... 33% 23.3M 6s 2023-07-02T14:02:42.854 INFO:tasks.workunit.client.0.smithi163.stderr: 55950K .......... .......... .......... .......... .......... 33% 12.7M 6s 2023-07-02T14:02:42.854 INFO:tasks.workunit.client.0.smithi163.stderr: 56000K .......... .......... .......... .......... .......... 33% 37.6M 6s 2023-07-02T14:02:42.854 INFO:tasks.workunit.client.0.smithi163.stderr: 56050K .......... .......... .......... .......... .......... 33% 20.0M 6s 2023-07-02T14:02:42.855 INFO:tasks.workunit.client.0.smithi163.stderr: 56100K .......... .......... .......... .......... .......... 33% 18.6M 6s 2023-07-02T14:02:42.855 INFO:tasks.workunit.client.0.smithi163.stderr: 56150K .......... .......... .......... .......... .......... 33% 20.5M 6s 2023-07-02T14:02:42.855 INFO:tasks.workunit.client.0.smithi163.stderr: 56200K .......... .......... .......... .......... .......... 33% 16.6M 6s 2023-07-02T14:02:42.855 INFO:tasks.workunit.client.0.smithi163.stderr: 56250K .......... .......... .......... .......... .......... 33% 22.8M 6s 2023-07-02T14:02:42.855 INFO:tasks.workunit.client.0.smithi163.stderr: 56300K .......... .......... .......... .......... .......... 33% 30.9M 6s 2023-07-02T14:02:42.855 INFO:tasks.workunit.client.0.smithi163.stderr: 56350K .......... .......... .......... .......... .......... 33% 11.6M 6s 2023-07-02T14:02:42.855 INFO:tasks.workunit.client.0.smithi163.stderr: 56400K .......... .......... .......... .......... .......... 33% 25.9M 6s 2023-07-02T14:02:42.856 INFO:tasks.workunit.client.0.smithi163.stderr: 56450K .......... .......... .......... .......... .......... 33% 19.9M 6s 2023-07-02T14:02:42.856 INFO:tasks.workunit.client.0.smithi163.stderr: 56500K .......... .......... .......... .......... .......... 33% 41.4M 6s 2023-07-02T14:02:42.859 INFO:tasks.workunit.client.0.smithi163.stderr: 56550K .......... .......... .......... .......... .......... 33% 14.2M 6s 2023-07-02T14:02:42.859 INFO:tasks.workunit.client.0.smithi163.stderr: 56600K .......... .......... .......... .......... .......... 33% 30.6M 6s 2023-07-02T14:02:42.859 INFO:tasks.workunit.client.0.smithi163.stderr: 56650K .......... .......... .......... .......... .......... 33% 26.2M 6s 2023-07-02T14:02:42.859 INFO:tasks.workunit.client.0.smithi163.stderr: 56700K .......... .......... .......... .......... .......... 33% 16.4M 6s 2023-07-02T14:02:42.859 INFO:tasks.workunit.client.0.smithi163.stderr: 56750K .......... .......... .......... .......... .......... 33% 14.0M 6s 2023-07-02T14:02:42.859 INFO:tasks.workunit.client.0.smithi163.stderr: 56800K .......... .......... .......... .......... .......... 33% 29.4M 6s 2023-07-02T14:02:42.860 INFO:tasks.workunit.client.0.smithi163.stderr: 56850K .......... .......... .......... .......... .......... 33% 25.4M 6s 2023-07-02T14:02:42.863 INFO:tasks.workunit.client.0.smithi163.stderr: 56900K .......... .......... .......... .......... .......... 33% 17.7M 6s 2023-07-02T14:02:42.865 INFO:tasks.workunit.client.0.smithi163.stderr: 56950K .......... .......... .......... .......... .......... 33% 21.9M 6s 2023-07-02T14:02:42.866 INFO:tasks.workunit.client.0.smithi163.stderr: 57000K .......... .......... .......... .......... .......... 33% 30.1M 6s 2023-07-02T14:02:42.870 INFO:tasks.workunit.client.0.smithi163.stderr: 57050K .......... .......... .......... .......... .......... 33% 13.7M 6s 2023-07-02T14:02:42.872 INFO:tasks.workunit.client.0.smithi163.stderr: 57100K .......... .......... .......... .......... .......... 33% 21.5M 6s 2023-07-02T14:02:42.875 INFO:tasks.workunit.client.0.smithi163.stderr: 57150K .......... .......... .......... .......... .......... 33% 17.4M 6s 2023-07-02T14:02:42.876 INFO:tasks.workunit.client.0.smithi163.stderr: 57200K .......... .......... .......... .......... .......... 33% 29.8M 6s 2023-07-02T14:02:42.880 INFO:tasks.workunit.client.0.smithi163.stderr: 57250K .......... .......... .......... .......... .......... 33% 16.4M 6s 2023-07-02T14:02:42.881 INFO:tasks.workunit.client.0.smithi163.stderr: 57300K .......... .......... .......... .......... .......... 34% 25.4M 6s 2023-07-02T14:02:42.884 INFO:tasks.workunit.client.0.smithi163.stderr: 57350K .......... .......... .......... .......... .......... 34% 22.0M 6s 2023-07-02T14:02:42.886 INFO:tasks.workunit.client.0.smithi163.stderr: 57400K .......... .......... .......... .......... .......... 34% 21.0M 6s 2023-07-02T14:02:42.888 INFO:tasks.workunit.client.0.smithi163.stderr: 57450K .......... .......... .......... .......... .......... 34% 21.4M 6s 2023-07-02T14:02:42.890 INFO:tasks.workunit.client.0.smithi163.stderr: 57500K .......... .......... .......... .......... .......... 34% 20.4M 6s 2023-07-02T14:02:42.894 INFO:tasks.workunit.client.0.smithi163.stderr: 57550K .......... .......... .......... .......... .......... 34% 16.2M 6s 2023-07-02T14:02:42.896 INFO:tasks.workunit.client.0.smithi163.stderr: 57600K .......... .......... .......... .......... .......... 34% 18.7M 6s 2023-07-02T14:02:42.899 INFO:tasks.workunit.client.0.smithi163.stderr: 57650K .......... .......... .......... .......... .......... 34% 18.4M 6s 2023-07-02T14:02:42.901 INFO:tasks.workunit.client.0.smithi163.stderr: 57700K .......... .......... .......... .......... .......... 34% 25.1M 6s 2023-07-02T14:02:42.902 INFO:tasks.workunit.client.0.smithi163.stderr: 57750K .......... .......... .......... .......... .......... 34% 30.6M 6s 2023-07-02T14:02:42.906 INFO:tasks.workunit.client.0.smithi163.stderr: 57800K .......... .......... .......... .......... .......... 34% 14.9M 6s 2023-07-02T14:02:42.907 INFO:tasks.workunit.client.0.smithi163.stderr: 57850K .......... .......... .......... .......... .......... 34% 36.4M 6s 2023-07-02T14:02:42.910 INFO:tasks.workunit.client.0.smithi163.stderr: 57900K .......... .......... .......... .......... .......... 34% 18.1M 6s 2023-07-02T14:02:42.914 INFO:tasks.workunit.client.0.smithi163.stderr: 57950K .......... .......... .......... .......... .......... 34% 12.4M 6s 2023-07-02T14:02:42.915 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:42.914+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:42.916 INFO:tasks.workunit.client.0.smithi163.stderr: 58000K .......... .......... .......... .......... .......... 34% 24.5M 6s 2023-07-02T14:02:42.917 INFO:tasks.workunit.client.0.smithi163.stderr: 58050K .......... .......... .......... .......... .......... 34% 29.2M 6s 2023-07-02T14:02:42.920 INFO:tasks.workunit.client.0.smithi163.stderr: 58100K .......... .......... .......... .......... .......... 34% 17.1M 6s 2023-07-02T14:02:42.922 INFO:tasks.workunit.client.0.smithi163.stderr: 58150K .......... .......... .......... .......... .......... 34% 25.0M 6s 2023-07-02T14:02:42.924 INFO:tasks.workunit.client.0.smithi163.stderr: 58200K .......... .......... .......... .......... .......... 34% 21.8M 6s 2023-07-02T14:02:42.926 INFO:tasks.workunit.client.0.smithi163.stderr: 58250K .......... .......... .......... .......... .......... 34% 26.6M 6s 2023-07-02T14:02:42.930 INFO:tasks.workunit.client.0.smithi163.stderr: 58300K .......... .......... .......... .......... .......... 34% 14.4M 6s 2023-07-02T14:02:42.932 INFO:tasks.workunit.client.0.smithi163.stderr: 58350K .......... .......... .......... .......... .......... 34% 24.0M 6s 2023-07-02T14:02:42.934 INFO:tasks.workunit.client.0.smithi163.stderr: 58400K .......... .......... .......... .......... .......... 34% 25.2M 6s 2023-07-02T14:02:42.938 INFO:tasks.workunit.client.0.smithi163.stderr: 58450K .......... .......... .......... .......... .......... 34% 12.3M 6s 2023-07-02T14:02:42.940 INFO:tasks.workunit.client.0.smithi163.stderr: 58500K .......... .......... .......... .......... .......... 34% 24.4M 6s 2023-07-02T14:02:42.941 INFO:tasks.workunit.client.0.smithi163.stderr: 58550K .......... .......... .......... .......... .......... 34% 35.8M 6s 2023-07-02T14:02:42.945 INFO:tasks.workunit.client.0.smithi163.stderr: 58600K .......... .......... .......... .......... .......... 34% 13.1M 6s 2023-07-02T14:02:42.946 INFO:tasks.workunit.client.0.smithi163.stderr: 58650K .......... .......... .......... .......... .......... 34% 68.0M 6s 2023-07-02T14:02:42.948 INFO:tasks.workunit.client.0.smithi163.stderr: 58700K .......... .......... .......... .......... .......... 34% 17.2M 6s 2023-07-02T14:02:42.952 INFO:tasks.workunit.client.0.smithi163.stderr: 58750K .......... .......... .......... .......... .......... 34% 14.1M 6s 2023-07-02T14:02:42.953 INFO:tasks.workunit.client.0.smithi163.stderr: 58800K .......... .......... .......... .......... .......... 34% 33.8M 6s 2023-07-02T14:02:42.956 INFO:tasks.workunit.client.0.smithi163.stderr: 58850K .......... .......... .......... .......... .......... 34% 16.4M 6s 2023-07-02T14:02:42.958 INFO:tasks.workunit.client.0.smithi163.stderr: 58900K .......... .......... .......... .......... .......... 34% 25.3M 6s 2023-07-02T14:02:42.961 INFO:tasks.workunit.client.0.smithi163.stderr: 58950K .......... .......... .......... .......... .......... 35% 20.6M 6s 2023-07-02T14:02:42.963 INFO:tasks.workunit.client.0.smithi163.stderr: 59000K .......... .......... .......... .......... .......... 35% 17.4M 6s 2023-07-02T14:02:42.965 INFO:tasks.workunit.client.0.smithi163.stderr: 59050K .......... .......... .......... .......... .......... 35% 20.6M 6s 2023-07-02T14:02:42.967 INFO:tasks.workunit.client.0.smithi163.stderr: 59100K .......... .......... .......... .......... .......... 35% 25.9M 6s 2023-07-02T14:02:42.971 INFO:tasks.workunit.client.0.smithi163.stderr: 59150K .......... .......... .......... .......... .......... 35% 15.4M 6s 2023-07-02T14:02:42.973 INFO:tasks.workunit.client.0.smithi163.stderr: 59200K .......... .......... .......... .......... .......... 35% 28.4M 6s 2023-07-02T14:02:42.976 INFO:tasks.workunit.client.0.smithi163.stderr: 59250K .......... .......... .......... .......... .......... 35% 15.7M 6s 2023-07-02T14:02:42.977 INFO:tasks.workunit.client.0.smithi163.stderr: 59300K .......... .......... .......... .......... .......... 35% 30.3M 6s 2023-07-02T14:02:42.980 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:42.980 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:42.980 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:42.980 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:42.981 INFO:tasks.workunit.client.0.smithi163.stderr: 59350K .......... .......... .......... .......... .......... 35% 14.1M 6s 2023-07-02T14:02:42.982 INFO:tasks.workunit.client.0.smithi163.stderr: 59400K .......... .......... .......... .......... .......... 35% 26.3M 6s 2023-07-02T14:02:42.984 INFO:tasks.workunit.client.0.smithi163.stderr: 59450K .......... .......... .......... .......... .......... 35% 29.7M 6s 2023-07-02T14:02:42.987 INFO:tasks.workunit.client.0.smithi163.stderr: 59500K .......... .......... .......... .......... .......... 35% 15.8M 6s 2023-07-02T14:02:42.990 INFO:tasks.workunit.client.0.smithi163.stderr: 59550K .......... .......... .......... .......... .......... 35% 18.3M 6s 2023-07-02T14:02:42.991 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:02:42.992 INFO:tasks.workunit.client.0.smithi163.stderr: 59600K .......... .......... .......... .......... .......... 35% 20.6M 6s 2023-07-02T14:02:42.994 INFO:tasks.workunit.client.0.smithi163.stderr: 59650K .......... .......... .......... .......... .......... 35% 24.3M 6s 2023-07-02T14:02:42.997 INFO:tasks.workunit.client.0.smithi163.stderr: 59700K .......... .......... .......... .......... .......... 35% 20.9M 6s 2023-07-02T14:02:42.999 INFO:tasks.workunit.client.0.smithi163.stderr: 59750K .......... .......... .......... .......... .......... 35% 19.7M 6s 2023-07-02T14:02:43.001 INFO:tasks.workunit.client.0.smithi163.stderr: 59800K .......... .......... .......... .......... .......... 35% 24.8M 6s 2023-07-02T14:02:43.005 INFO:tasks.workunit.client.0.smithi163.stderr: 59850K .......... .......... .......... .......... .......... 35% 14.1M 6s 2023-07-02T14:02:43.006 INFO:tasks.workunit.client.0.smithi163.stderr: 59900K .......... .......... .......... .......... .......... 35% 29.7M 6s 2023-07-02T14:02:43.009 INFO:tasks.workunit.client.0.smithi163.stderr: 59950K .......... .......... .......... .......... .......... 35% 20.1M 6s 2023-07-02T14:02:43.011 INFO:tasks.workunit.client.0.smithi163.stderr: 60000K .......... .......... .......... .......... .......... 35% 16.2M 6s 2023-07-02T14:02:43.014 INFO:tasks.workunit.client.0.smithi163.stderr: 60050K .......... .......... .......... .......... .......... 35% 20.4M 6s 2023-07-02T14:02:43.016 INFO:tasks.workunit.client.0.smithi163.stderr: 60100K .......... .......... .......... .......... .......... 35% 26.6M 6s 2023-07-02T14:02:43.017 INFO:tasks.workunit.client.0.smithi163.stderr: 60150K .......... .......... .......... .......... .......... 35% 32.7M 6s 2023-07-02T14:02:43.021 INFO:tasks.workunit.client.0.smithi163.stderr: 60200K .......... .......... .......... .......... .......... 35% 14.8M 6s 2023-07-02T14:02:43.023 INFO:tasks.workunit.client.0.smithi163.stderr: 60250K .......... .......... .......... .......... .......... 35% 18.4M 6s 2023-07-02T14:02:43.026 INFO:tasks.workunit.client.0.smithi163.stderr: 60300K .......... .......... .......... .......... .......... 35% 21.4M 6s 2023-07-02T14:02:43.028 INFO:tasks.workunit.client.0.smithi163.stderr: 60350K .......... .......... .......... .......... .......... 35% 17.4M 6s 2023-07-02T14:02:43.031 INFO:tasks.workunit.client.0.smithi163.stderr: 60400K .......... .......... .......... .......... .......... 35% 18.4M 6s 2023-07-02T14:02:43.032 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:43.031+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:43.033 INFO:tasks.workunit.client.0.smithi163.stderr: 60450K .......... .......... .......... .......... .......... 35% 19.5M 6s 2023-07-02T14:02:43.035 INFO:tasks.workunit.client.0.smithi163.stderr: 60500K .......... .......... .......... .......... .......... 35% 23.2M 6s 2023-07-02T14:02:43.038 INFO:tasks.workunit.client.0.smithi163.stderr: 60550K .......... .......... .......... .......... .......... 35% 22.2M 6s 2023-07-02T14:02:43.040 INFO:tasks.workunit.client.0.smithi163.stderr: 60600K .......... .......... .......... .......... .......... 35% 16.8M 6s 2023-07-02T14:02:43.042 INFO:tasks.workunit.client.0.smithi163.stderr: 60650K .......... .......... .......... .......... .......... 36% 41.6M 6s 2023-07-02T14:02:43.044 INFO:tasks.workunit.client.0.smithi163.stderr: 60700K .......... .......... .......... .......... .......... 36% 22.3M 6s 2023-07-02T14:02:43.047 INFO:tasks.workunit.client.0.smithi163.stderr: 60750K .......... .......... .......... .......... .......... 36% 14.4M 6s 2023-07-02T14:02:43.051 INFO:tasks.workunit.client.0.smithi163.stderr: 60800K .......... .......... .......... .......... .......... 36% 14.3M 6s 2023-07-02T14:02:43.053 INFO:tasks.workunit.client.0.smithi163.stderr: 60850K .......... .......... .......... .......... .......... 36% 27.2M 6s 2023-07-02T14:02:43.056 INFO:tasks.workunit.client.0.smithi163.stderr: 60900K .......... .......... .......... .......... .......... 36% 17.6M 6s 2023-07-02T14:02:43.057 INFO:tasks.workunit.client.0.smithi163.stderr: 60950K .......... .......... .......... .......... .......... 36% 31.6M 6s 2023-07-02T14:02:43.059 INFO:tasks.workunit.client.0.smithi163.stderr: 61000K .......... .......... .......... .......... .......... 36% 24.3M 6s 2023-07-02T14:02:43.062 INFO:tasks.workunit.client.0.smithi163.stderr: 61050K .......... .......... .......... .......... .......... 36% 18.2M 6s 2023-07-02T14:02:43.065 INFO:tasks.workunit.client.0.smithi163.stderr: 61100K .......... .......... .......... .......... .......... 36% 16.4M 6s 2023-07-02T14:02:43.067 INFO:tasks.workunit.client.0.smithi163.stderr: 61150K .......... .......... .......... .......... .......... 36% 22.2M 6s 2023-07-02T14:02:43.069 INFO:tasks.workunit.client.0.smithi163.stderr: 61200K .......... .......... .......... .......... .......... 36% 21.2M 6s 2023-07-02T14:02:43.071 INFO:tasks.workunit.client.0.smithi163.stderr: 61250K .......... .......... .......... .......... .......... 36% 23.3M 6s 2023-07-02T14:02:43.074 INFO:tasks.workunit.client.0.smithi163.stderr: 61300K .......... .......... .......... .......... .......... 36% 18.3M 6s 2023-07-02T14:02:43.076 INFO:tasks.workunit.client.0.smithi163.stderr: 61350K .......... .......... .......... .......... .......... 36% 23.1M 6s 2023-07-02T14:02:43.078 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:02:43.079 INFO:tasks.workunit.client.0.smithi163.stderr: 61400K .......... .......... .......... .......... .......... 36% 19.0M 6s 2023-07-02T14:02:43.081 INFO:tasks.workunit.client.0.smithi163.stderr: 61450K .......... .......... .......... .......... .......... 36% 24.3M 6s 2023-07-02T14:02:43.083 INFO:tasks.workunit.client.0.smithi163.stderr: 61500K .......... .......... .......... .......... .......... 36% 20.9M 6s 2023-07-02T14:02:43.086 INFO:tasks.workunit.client.0.smithi163.stderr: 61550K .......... .......... .......... .......... .......... 36% 16.1M 6s 2023-07-02T14:02:43.088 INFO:tasks.workunit.client.0.smithi163.stderr: 61600K .......... .......... .......... .......... .......... 36% 22.4M 6s 2023-07-02T14:02:43.091 INFO:tasks.workunit.client.0.smithi163.stderr: 61650K .......... .......... .......... .......... .......... 36% 20.4M 6s 2023-07-02T14:02:43.093 INFO:tasks.workunit.client.0.smithi163.stderr: 61700K .......... .......... .......... .......... .......... 36% 20.1M 6s 2023-07-02T14:02:43.096 INFO:tasks.workunit.client.0.smithi163.stderr: 61750K .......... .......... .......... .......... .......... 36% 18.8M 6s 2023-07-02T14:02:43.098 INFO:tasks.workunit.client.0.smithi163.stderr: 61800K .......... .......... .......... .......... .......... 36% 21.7M 6s 2023-07-02T14:02:43.099 INFO:tasks.workunit.client.0.smithi163.stderr: 61850K .......... .......... .......... .......... .......... 36% 29.7M 6s 2023-07-02T14:02:43.101 INFO:tasks.workunit.client.0.smithi163.stderr: 61900K .......... .......... .......... .......... .......... 36% 22.9M 6s 2023-07-02T14:02:43.104 INFO:tasks.workunit.client.0.smithi163.stderr: 61950K .......... .......... .......... .......... .......... 36% 16.8M 6s 2023-07-02T14:02:43.107 INFO:tasks.workunit.client.0.smithi163.stderr: 62000K .......... .......... .......... .......... .......... 36% 18.9M 6s 2023-07-02T14:02:43.110 INFO:tasks.workunit.client.0.smithi163.stderr: 62050K .......... .......... .......... .......... .......... 36% 19.3M 6s 2023-07-02T14:02:43.111 INFO:tasks.workunit.client.0.smithi163.stderr: 62100K .......... .......... .......... .......... .......... 36% 31.1M 6s 2023-07-02T14:02:43.114 INFO:tasks.workunit.client.0.smithi163.stderr: 62150K .......... .......... .......... .......... .......... 36% 18.3M 6s 2023-07-02T14:02:43.116 INFO:tasks.workunit.client.0.smithi163.stderr: 62200K .......... .......... .......... .......... .......... 36% 31.0M 6s 2023-07-02T14:02:43.116 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:43.115+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:43.117 INFO:tasks.workunit.client.0.smithi163.stderr: 62250K .......... .......... .......... .......... .......... 36% 25.8M 6s 2023-07-02T14:02:43.120 INFO:tasks.workunit.client.0.smithi163.stderr: 62300K .......... .......... .......... .......... .......... 36% 17.8M 6s 2023-07-02T14:02:43.123 INFO:tasks.workunit.client.0.smithi163.stderr: 62350K .......... .......... .......... .......... .......... 37% 16.3M 6s 2023-07-02T14:02:43.125 INFO:tasks.workunit.client.0.smithi163.stderr: 62400K .......... .......... .......... .......... .......... 37% 32.9M 6s 2023-07-02T14:02:43.126 INFO:tasks.workunit.client.0.smithi163.stderr: 62450K .......... .......... .......... .......... .......... 37% 30.6M 5s 2023-07-02T14:02:43.128 INFO:tasks.workunit.client.0.smithi163.stderr: 62500K .......... .......... .......... .......... .......... 37% 24.4M 5s 2023-07-02T14:02:43.132 INFO:tasks.workunit.client.0.smithi163.stderr: 62550K .......... .......... .......... .......... .......... 37% 14.1M 5s 2023-07-02T14:02:43.133 INFO:tasks.workunit.client.0.smithi163.stderr: 62600K .......... .......... .......... .......... .......... 37% 42.0M 5s 2023-07-02T14:02:43.136 INFO:tasks.workunit.client.0.smithi163.stderr: 62650K .......... .......... .......... .......... .......... 37% 15.9M 5s 2023-07-02T14:02:43.137 INFO:tasks.workunit.client.0.smithi163.stderr: 62700K .......... .......... .......... .......... .......... 37% 28.3M 5s 2023-07-02T14:02:43.141 INFO:tasks.workunit.client.0.smithi163.stderr: 62750K .......... .......... .......... .......... .......... 37% 14.2M 5s 2023-07-02T14:02:43.143 INFO:tasks.workunit.client.0.smithi163.stderr: 62800K .......... .......... .......... .......... .......... 37% 24.0M 5s 2023-07-02T14:02:43.145 INFO:tasks.workunit.client.0.smithi163.stderr: 62850K .......... .......... .......... .......... .......... 37% 23.8M 5s 2023-07-02T14:02:43.148 INFO:tasks.workunit.client.0.smithi163.stderr: 62900K .......... .......... .......... .......... .......... 37% 18.3M 5s 2023-07-02T14:02:43.149 INFO:tasks.workunit.client.0.smithi163.stderr: 62950K .......... .......... .......... .......... .......... 37% 30.3M 5s 2023-07-02T14:02:43.153 INFO:tasks.workunit.client.0.smithi163.stderr: 63000K .......... .......... .......... .......... .......... 37% 15.4M 5s 2023-07-02T14:02:43.155 INFO:tasks.workunit.client.0.smithi163.stderr: 63050K .......... .......... .......... .......... .......... 37% 18.6M 5s 2023-07-02T14:02:43.156 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:43.157 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:43.157 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:43.157 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:43.157 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:43.158 INFO:tasks.workunit.client.0.smithi163.stderr: 63100K .......... .......... .......... .......... .......... 37% 31.0M 5s 2023-07-02T14:02:43.161 INFO:tasks.workunit.client.0.smithi163.stderr: 63150K .......... .......... .......... .......... .......... 37% 14.8M 5s 2023-07-02T14:02:43.163 INFO:tasks.workunit.client.0.smithi163.stderr: 63200K .......... .......... .......... .......... .......... 37% 19.5M 5s 2023-07-02T14:02:43.165 INFO:tasks.workunit.client.0.smithi163.stderr: 63250K .......... .......... .......... .......... .......... 37% 25.8M 5s 2023-07-02T14:02:43.167 INFO:tasks.workunit.client.0.smithi163.stderr: 63300K .......... .......... .......... .......... .......... 37% 20.4M 5s 2023-07-02T14:02:43.168 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:02:43.169 INFO:tasks.workunit.client.0.smithi163.stderr: 63350K .......... .......... .......... .......... .......... 37% 20.4M 5s 2023-07-02T14:02:43.171 INFO:tasks.workunit.client.0.smithi163.stderr: 63400K .......... .......... .......... .......... .......... 37% 21.7M 5s 2023-07-02T14:02:43.174 INFO:tasks.workunit.client.0.smithi163.stderr: 63450K .......... .......... .......... .......... .......... 37% 23.4M 5s 2023-07-02T14:02:43.176 INFO:tasks.workunit.client.0.smithi163.stderr: 63500K .......... .......... .......... .......... .......... 37% 20.8M 5s 2023-07-02T14:02:43.180 INFO:tasks.workunit.client.0.smithi163.stderr: 63550K .......... .......... .......... .......... .......... 37% 12.5M 5s 2023-07-02T14:02:43.182 INFO:tasks.workunit.client.0.smithi163.stderr: 63600K .......... .......... .......... .......... .......... 37% 29.4M 5s 2023-07-02T14:02:43.184 INFO:tasks.workunit.client.0.smithi163.stderr: 63650K .......... .......... .......... .......... .......... 37% 21.8M 5s 2023-07-02T14:02:43.187 INFO:tasks.workunit.client.0.smithi163.stderr: 63700K .......... .......... .......... .......... .......... 37% 16.5M 5s 2023-07-02T14:02:43.188 INFO:tasks.workunit.client.0.smithi163.stderr: 63750K .......... .......... .......... .......... .......... 37% 29.1M 5s 2023-07-02T14:02:43.191 INFO:tasks.workunit.client.0.smithi163.stderr: 63800K .......... .......... .......... .......... .......... 37% 21.0M 5s 2023-07-02T14:02:43.193 INFO:tasks.workunit.client.0.smithi163.stderr: 63850K .......... .......... .......... .......... .......... 37% 26.7M 5s 2023-07-02T14:02:43.195 INFO:tasks.workunit.client.0.smithi163.stderr: 63900K .......... .......... .......... .......... .......... 37% 17.8M 5s 2023-07-02T14:02:43.198 INFO:tasks.workunit.client.0.smithi163.stderr: 63950K .......... .......... .......... .......... .......... 37% 19.0M 5s 2023-07-02T14:02:43.200 INFO:tasks.workunit.client.0.smithi163.stderr: 64000K .......... .......... .......... .......... .......... 37% 26.4M 5s 2023-07-02T14:02:43.202 INFO:tasks.workunit.client.0.smithi163.stderr: 64050K .......... .......... .......... .......... .......... 38% 28.2M 5s 2023-07-02T14:02:43.205 INFO:tasks.workunit.client.0.smithi163.stderr: 64100K .......... .......... .......... .......... .......... 38% 14.8M 5s 2023-07-02T14:02:43.206 INFO:tasks.workunit.client.0.smithi163.stderr: 64150K .......... .......... .......... .......... .......... 38% 34.2M 5s 2023-07-02T14:02:43.208 INFO:tasks.workunit.client.0.smithi163.stderr: 64200K .......... .......... .......... .......... .......... 38% 27.2M 5s 2023-07-02T14:02:43.210 INFO:tasks.workunit.client.0.smithi163.stderr: 64250K .......... .......... .......... .......... .......... 38% 24.4M 5s 2023-07-02T14:02:43.213 INFO:tasks.workunit.client.0.smithi163.stderr: 64300K .......... .......... .......... .......... .......... 38% 14.9M 5s 2023-07-02T14:02:43.216 INFO:tasks.workunit.client.0.smithi163.stderr: 64350K .......... .......... .......... .......... .......... 38% 21.2M 5s 2023-07-02T14:02:43.217 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:43.216+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:43.218 INFO:tasks.workunit.client.0.smithi163.stderr: 64400K .......... .......... .......... .......... .......... 38% 25.7M 5s 2023-07-02T14:02:43.221 INFO:tasks.workunit.client.0.smithi163.stderr: 64450K .......... .......... .......... .......... .......... 38% 16.2M 5s 2023-07-02T14:02:43.222 INFO:tasks.workunit.client.0.smithi163.stderr: 64500K .......... .......... .......... .......... .......... 38% 40.1M 5s 2023-07-02T14:02:43.223 INFO:tasks.workunit.client.0.smithi163.stderr: 64550K .......... .......... .......... .......... .......... 38% 24.8M 5s 2023-07-02T14:02:43.228 INFO:tasks.workunit.client.0.smithi163.stderr: 64600K .......... .......... .......... .......... .......... 38% 13.4M 5s 2023-07-02T14:02:43.229 INFO:tasks.workunit.client.0.smithi163.stderr: 64650K .......... .......... .......... .......... .......... 38% 29.9M 5s 2023-07-02T14:02:43.231 INFO:tasks.workunit.client.0.smithi163.stderr: 64700K .......... .......... .......... .......... .......... 38% 21.0M 5s 2023-07-02T14:02:43.234 INFO:tasks.workunit.client.0.smithi163.stderr: 64750K .......... .......... .......... .......... .......... 38% 21.0M 5s 2023-07-02T14:02:43.237 INFO:tasks.workunit.client.0.smithi163.stderr: 64800K .......... .......... .......... .......... .......... 38% 16.3M 5s 2023-07-02T14:02:43.239 INFO:tasks.workunit.client.0.smithi163.stderr: 64850K .......... .......... .......... .......... .......... 38% 21.9M 5s 2023-07-02T14:02:43.241 INFO:tasks.workunit.client.0.smithi163.stderr: 64900K .......... .......... .......... .......... .......... 38% 26.6M 5s 2023-07-02T14:02:43.244 INFO:tasks.workunit.client.0.smithi163.stderr: 64950K .......... .......... .......... .......... .......... 38% 15.0M 5s 2023-07-02T14:02:43.246 INFO:tasks.workunit.client.0.smithi163.stderr: 65000K .......... .......... .......... .......... .......... 38% 26.7M 5s 2023-07-02T14:02:43.247 INFO:tasks.workunit.client.0.smithi163.stderr: 65050K .......... .......... .......... .......... .......... 38% 26.4M 5s 2023-07-02T14:02:43.251 INFO:tasks.workunit.client.0.smithi163.stderr: 65100K .......... .......... .......... .......... .......... 38% 16.1M 5s 2023-07-02T14:02:43.254 INFO:tasks.workunit.client.0.smithi163.stderr: 65150K .......... .......... .......... .......... .......... 38% 16.5M 5s 2023-07-02T14:02:43.255 INFO:tasks.workunit.client.0.smithi163.stderr: 65200K .......... .......... .......... .......... .......... 38% 31.6M 5s 2023-07-02T14:02:43.256 INFO:tasks.workunit.client.0.smithi163.stderr: 65250K .......... .......... .......... .......... .......... 38% 37.3M 5s 2023-07-02T14:02:43.260 INFO:tasks.workunit.client.0.smithi163.stderr: 65300K .......... .......... .......... .......... .......... 38% 14.0M 5s 2023-07-02T14:02:43.263 INFO:tasks.workunit.client.0.smithi163.stderr: 65350K .......... .......... .......... .......... .......... 38% 17.3M 5s 2023-07-02T14:02:43.264 INFO:tasks.workunit.client.0.smithi163.stderr: 65400K .......... .......... .......... .......... .......... 38% 33.4M 5s 2023-07-02T14:02:43.267 INFO:tasks.workunit.client.0.smithi163.stderr: 65450K .......... .......... .......... .......... .......... 38% 15.8M 5s 2023-07-02T14:02:43.269 INFO:tasks.workunit.client.0.smithi163.stderr: 65500K .......... .......... .......... .......... .......... 38% 40.7M 5s 2023-07-02T14:02:43.272 INFO:tasks.workunit.client.0.smithi163.stderr: 65550K .......... .......... .......... .......... .......... 38% 13.9M 5s 2023-07-02T14:02:43.274 INFO:tasks.workunit.client.0.smithi163.stderr: 65600K .......... .......... .......... .......... .......... 38% 32.3M 5s 2023-07-02T14:02:43.276 INFO:tasks.workunit.client.0.smithi163.stderr: 65650K .......... .......... .......... .......... .......... 38% 19.9M 5s 2023-07-02T14:02:43.279 INFO:tasks.workunit.client.0.smithi163.stderr: 65700K .......... .......... .......... .......... .......... 39% 17.6M 5s 2023-07-02T14:02:43.280 INFO:tasks.workunit.client.0.smithi163.stderr: 65750K .......... .......... .......... .......... .......... 39% 33.0M 5s 2023-07-02T14:02:43.283 INFO:tasks.workunit.client.0.smithi163.stderr: 65800K .......... .......... .......... .......... .......... 39% 17.7M 5s 2023-07-02T14:02:43.286 INFO:tasks.workunit.client.0.smithi163.stderr: 65850K .......... .......... .......... .......... .......... 39% 14.0M 5s 2023-07-02T14:02:43.288 INFO:tasks.workunit.client.0.smithi163.stderr: 65900K .......... .......... .......... .......... .......... 39% 25.3M 5s 2023-07-02T14:02:43.292 INFO:tasks.workunit.client.0.smithi163.stderr: 65950K .......... .......... .......... .......... .......... 39% 15.4M 5s 2023-07-02T14:02:43.294 INFO:tasks.workunit.client.0.smithi163.stderr: 66000K .......... .......... .......... .......... .......... 39% 18.1M 5s 2023-07-02T14:02:43.296 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:43.296 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:02:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:250 2.2 2:44c63967:::rbd_data.125de1449605.0000000000000101:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.299603+0000", 2023-07-02T14:02:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.9949191719999999, 2023-07-02T14:02:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037580609000000001, 2023-07-02T14:02:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 250 2023-07-02T14:02:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.299603+0000", 2023-07-02T14:02:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.299603+0000", 2023-07-02T14:02:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.299602+0000", 2023-07-02T14:02:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022150699999999999 2023-07-02T14:02:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.299824+0000", 2023-07-02T14:02:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1405e-05 2023-07-02T14:02:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.299835+0000", 2023-07-02T14:02:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1590000000000004e-06 2023-07-02T14:02:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.299842+0000", 2023-07-02T14:02:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017232279999999999 2023-07-02T14:02:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317074+0000", 2023-07-02T14:02:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5123999999999994e-05 2023-07-02T14:02:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317150+0000", 2023-07-02T14:02:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014873800000000001 2023-07-02T14:02:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317298+0000", 2023-07-02T14:02:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035949480000000002 2023-07-02T14:02:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320893+0000", 2023-07-02T14:02:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016194844999999999 2023-07-02T14:02:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337088+0000", 2023-07-02T14:02:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1582999999999997e-05 2023-07-02T14:02:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337140+0000", 2023-07-02T14:02:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3779000000000002e-05 2023-07-02T14:02:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337183+0000", 2023-07-02T14:02:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037580609000000001 2023-07-02T14:02:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:256 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.300278+0000", 2023-07-02T14:02:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.9942437499999999, 2023-07-02T14:02:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037365890999999998, 2023-07-02T14:02:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 256 2023-07-02T14:02:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300278+0000", 2023-07-02T14:02:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300278+0000", 2023-07-02T14:02:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300278+0000", 2023-07-02T14:02:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00066792800000000003 2023-07-02T14:02:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300945+0000", 2023-07-02T14:02:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0150000000000003e-06 2023-07-02T14:02:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300954+0000", 2023-07-02T14:02:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9069999999999998e-06 2023-07-02T14:02:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300958+0000", 2023-07-02T14:02:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018796822000000001 2023-07-02T14:02:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319755+0000", 2023-07-02T14:02:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126038 2023-07-02T14:02:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319881+0000", 2023-07-02T14:02:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015656700000000001 2023-07-02T14:02:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320038+0000", 2023-07-02T14:02:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017229343000000001 2023-07-02T14:02:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337267+0000", 2023-07-02T14:02:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000277378 2023-07-02T14:02:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337545+0000", 2023-07-02T14:02:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4978000000000001e-05 2023-07-02T14:02:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337600+0000", 2023-07-02T14:02:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4635000000000002e-05 2023-07-02T14:02:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337644+0000", 2023-07-02T14:02:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037365890999999998 2023-07-02T14:02:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:279 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.310714+0000", 2023-07-02T14:02:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.9838082859999999, 2023-07-02T14:02:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038119238, 2023-07-02T14:02:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 279 2023-07-02T14:02:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310714+0000", 2023-07-02T14:02:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310714+0000", 2023-07-02T14:02:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310713+0000", 2023-07-02T14:02:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8973e-05 2023-07-02T14:02:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310762+0000", 2023-07-02T14:02:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0530000000000007e-06 2023-07-02T14:02:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310770+0000", 2023-07-02T14:02:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7569999999999993e-06 2023-07-02T14:02:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310779+0000", 2023-07-02T14:02:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033655263999999997 2023-07-02T14:02:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.344434+0000", 2023-07-02T14:02:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4365999999999999e-05 2023-07-02T14:02:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.344508+0000", 2023-07-02T14:02:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016449100000000001 2023-07-02T14:02:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.344673+0000", 2023-07-02T14:02:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019402379999999999 2023-07-02T14:02:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346613+0000", 2023-07-02T14:02:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021203900000000002 2023-07-02T14:02:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348734+0000", 2023-07-02T14:02:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5225000000000001e-05 2023-07-02T14:02:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348789+0000", 2023-07-02T14:02:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4163999999999999e-05 2023-07-02T14:02:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348833+0000", 2023-07-02T14:02:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038119238 2023-07-02T14:02:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:280 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.310785+0000", 2023-07-02T14:02:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.9837374130000001, 2023-07-02T14:02:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038149954999999999, 2023-07-02T14:02:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 280 2023-07-02T14:02:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310785+0000", 2023-07-02T14:02:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310785+0000", 2023-07-02T14:02:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310784+0000", 2023-07-02T14:02:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4364999999999999e-05 2023-07-02T14:02:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310838+0000", 2023-07-02T14:02:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4619999999999996e-06 2023-07-02T14:02:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310847+0000", 2023-07-02T14:02:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5129999999999998e-06 2023-07-02T14:02:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310851+0000", 2023-07-02T14:02:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034215981999999999 2023-07-02T14:02:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.345067+0000", 2023-07-02T14:02:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6117000000000002e-05 2023-07-02T14:02:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.345153+0000", 2023-07-02T14:02:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014067499999999999 2023-07-02T14:02:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.345294+0000", 2023-07-02T14:02:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013560810000000001 2023-07-02T14:02:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346650+0000", 2023-07-02T14:02:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002196239 2023-07-02T14:02:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348846+0000", 2023-07-02T14:02:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7621999999999999e-05 2023-07-02T14:02:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348894+0000", 2023-07-02T14:02:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0518000000000001e-05 2023-07-02T14:02:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348935+0000", 2023-07-02T14:02:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038149954999999999 2023-07-02T14:02:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:281 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.310857+0000", 2023-07-02T14:02:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.9836651509999998, 2023-07-02T14:02:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039347695000000002, 2023-07-02T14:02:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:02:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310857+0000", 2023-07-02T14:02:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310857+0000", 2023-07-02T14:02:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310856+0000", 2023-07-02T14:02:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00081158100000000002 2023-07-02T14:02:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311668+0000", 2023-07-02T14:02:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.257e-05 2023-07-02T14:02:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311680+0000", 2023-07-02T14:02:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1110000000000002e-06 2023-07-02T14:02:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311688+0000", 2023-07-02T14:02:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034985653999999998 2023-07-02T14:02:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346673+0000", 2023-07-02T14:02:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7062999999999999e-05 2023-07-02T14:02:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346730+0000", 2023-07-02T14:02:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000151061 2023-07-02T14:02:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346881+0000", 2023-07-02T14:02:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018356119999999999 2023-07-02T14:02:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348717+0000", 2023-07-02T14:02:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001417385 2023-07-02T14:02:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350134+0000", 2023-07-02T14:02:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7036999999999997e-05 2023-07-02T14:02:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350181+0000", 2023-07-02T14:02:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3187999999999999e-05 2023-07-02T14:02:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350205+0000", 2023-07-02T14:02:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039347695000000002 2023-07-02T14:02:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:282 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311695+0000", 2023-07-02T14:02:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.9828266569999999, 2023-07-02T14:02:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038558453999999999, 2023-07-02T14:02:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 282 2023-07-02T14:02:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311695+0000", 2023-07-02T14:02:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311695+0000", 2023-07-02T14:02:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311695+0000", 2023-07-02T14:02:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6679999999999999e-05 2023-07-02T14:02:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311751+0000", 2023-07-02T14:02:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9199999999999993e-06 2023-07-02T14:02:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311760+0000", 2023-07-02T14:02:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2480000000000004e-06 2023-07-02T14:02:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311765+0000", 2023-07-02T14:02:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035670150999999997 2023-07-02T14:02:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347436+0000", 2023-07-02T14:02:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7861000000000006e-05 2023-07-02T14:02:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347503+0000", 2023-07-02T14:02:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000217443 2023-07-02T14:02:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347721+0000", 2023-07-02T14:02:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023992449999999999 2023-07-02T14:02:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350120+0000", 2023-07-02T14:02:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2219000000000006e-05 2023-07-02T14:02:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350212+0000", 2023-07-02T14:02:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3430000000000001e-05 2023-07-02T14:02:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350236+0000", 2023-07-02T14:02:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8074999999999999e-05 2023-07-02T14:02:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350254+0000", 2023-07-02T14:02:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038558453999999999 2023-07-02T14:02:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.360 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:283 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:43.360 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311771+0000", 2023-07-02T14:02:43.360 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.982750786, 2023-07-02T14:02:43.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039285701999999999, 2023-07-02T14:02:43.360 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 283 2023-07-02T14:02:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311771+0000", 2023-07-02T14:02:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311771+0000", 2023-07-02T14:02:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311771+0000", 2023-07-02T14:02:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6775999999999999e-05 2023-07-02T14:02:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311837+0000", 2023-07-02T14:02:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4290000000000001e-06 2023-07-02T14:02:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311846+0000", 2023-07-02T14:02:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3800000000000002e-06 2023-07-02T14:02:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311851+0000", 2023-07-02T14:02:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036308287000000002 2023-07-02T14:02:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348160+0000", 2023-07-02T14:02:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6475000000000002e-05 2023-07-02T14:02:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348236+0000", 2023-07-02T14:02:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000147966 2023-07-02T14:02:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348384+0000", 2023-07-02T14:02:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019115569999999999 2023-07-02T14:02:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350296+0000", 2023-07-02T14:02:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00063248000000000004 2023-07-02T14:02:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350928+0000", 2023-07-02T14:02:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3355000000000003e-05 2023-07-02T14:02:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351021+0000", 2023-07-02T14:02:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5596000000000003e-05 2023-07-02T14:02:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351057+0000", 2023-07-02T14:02:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039285701999999999 2023-07-02T14:02:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:284 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311859+0000", 2023-07-02T14:02:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.9826634890000001, 2023-07-02T14:02:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039967282999999999, 2023-07-02T14:02:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 284 2023-07-02T14:02:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311859+0000", 2023-07-02T14:02:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311859+0000", 2023-07-02T14:02:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311858+0000", 2023-07-02T14:02:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3902999999999997e-05 2023-07-02T14:02:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311892+0000", 2023-07-02T14:02:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0989999999999999e-06 2023-07-02T14:02:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311900+0000", 2023-07-02T14:02:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9030000000000003e-06 2023-07-02T14:02:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311905+0000", 2023-07-02T14:02:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037060899000000001 2023-07-02T14:02:43.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:43.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348966+0000", 2023-07-02T14:02:43.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3609000000000002e-05 2023-07-02T14:02:43.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:43.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349009+0000", 2023-07-02T14:02:43.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6831999999999998e-05 2023-07-02T14:02:43.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:43.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349096+0000", 2023-07-02T14:02:43.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001975077 2023-07-02T14:02:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351071+0000", 2023-07-02T14:02:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00066166900000000004 2023-07-02T14:02:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351733+0000", 2023-07-02T14:02:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3199999999999999e-05 2023-07-02T14:02:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351786+0000", 2023-07-02T14:02:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9823999999999997e-05 2023-07-02T14:02:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351826+0000", 2023-07-02T14:02:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039967282999999999 2023-07-02T14:02:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:285 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311910+0000", 2023-07-02T14:02:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.9826124570000001, 2023-07-02T14:02:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040060302999999998, 2023-07-02T14:02:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 285 2023-07-02T14:02:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311910+0000", 2023-07-02T14:02:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311910+0000", 2023-07-02T14:02:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311909+0000", 2023-07-02T14:02:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1025999999999997e-05 2023-07-02T14:02:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311940+0000", 2023-07-02T14:02:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4769999999999997e-06 2023-07-02T14:02:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311948+0000", 2023-07-02T14:02:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5059999999999997e-06 2023-07-02T14:02:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311952+0000", 2023-07-02T14:02:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037480332999999998 2023-07-02T14:02:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349432+0000", 2023-07-02T14:02:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2923000000000005e-05 2023-07-02T14:02:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349515+0000", 2023-07-02T14:02:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015231700000000001 2023-07-02T14:02:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349668+0000", 2023-07-02T14:02:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002171849 2023-07-02T14:02:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351839+0000", 2023-07-02T14:02:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2052000000000003e-05 2023-07-02T14:02:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351882+0000", 2023-07-02T14:02:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7683999999999999e-05 2023-07-02T14:02:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351929+0000", 2023-07-02T14:02:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0648000000000001e-05 2023-07-02T14:02:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351970+0000", 2023-07-02T14:02:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040060302999999998 2023-07-02T14:02:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:286 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311957+0000", 2023-07-02T14:02:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.98256459, 2023-07-02T14:02:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042429263000000002, 2023-07-02T14:02:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 286 2023-07-02T14:02:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311957+0000", 2023-07-02T14:02:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311957+0000", 2023-07-02T14:02:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311957+0000", 2023-07-02T14:02:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7862000000000003e-05 2023-07-02T14:02:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312015+0000", 2023-07-02T14:02:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2770000000000005e-06 2023-07-02T14:02:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312023+0000", 2023-07-02T14:02:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0000000000000004e-06 2023-07-02T14:02:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312028+0000", 2023-07-02T14:02:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038291327999999999 2023-07-02T14:02:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350319+0000", 2023-07-02T14:02:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3686000000000001e-05 2023-07-02T14:02:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350373+0000", 2023-07-02T14:02:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012420000000000001 2023-07-02T14:02:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350497+0000", 2023-07-02T14:02:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014893669999999999 2023-07-02T14:02:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351986+0000", 2023-07-02T14:02:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022881580000000002 2023-07-02T14:02:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354275+0000", 2023-07-02T14:02:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7388e-05 2023-07-02T14:02:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354332+0000", 2023-07-02T14:02:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4688999999999998e-05 2023-07-02T14:02:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354387+0000", 2023-07-02T14:02:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042429263000000002 2023-07-02T14:02:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:287 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 589824~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312033+0000", 2023-07-02T14:02:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.9824889720000001, 2023-07-02T14:02:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042515431999999999, 2023-07-02T14:02:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 287 2023-07-02T14:02:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312033+0000", 2023-07-02T14:02:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312033+0000", 2023-07-02T14:02:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312033+0000", 2023-07-02T14:02:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1540999999999999e-05 2023-07-02T14:02:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312094+0000", 2023-07-02T14:02:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.123e-06 2023-07-02T14:02:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312102+0000", 2023-07-02T14:02:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5480000000000003e-06 2023-07-02T14:02:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312107+0000", 2023-07-02T14:02:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039019356999999998 2023-07-02T14:02:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351126+0000", 2023-07-02T14:02:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100107 2023-07-02T14:02:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:43.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351226+0000", 2023-07-02T14:02:43.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177455 2023-07-02T14:02:43.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:43.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351404+0000", 2023-07-02T14:02:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030030819999999998 2023-07-02T14:02:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354407+0000", 2023-07-02T14:02:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5787000000000001e-05 2023-07-02T14:02:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354443+0000", 2023-07-02T14:02:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0627000000000001e-05 2023-07-02T14:02:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354503+0000", 2023-07-02T14:02:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5327000000000002e-05 2023-07-02T14:02:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354548+0000", 2023-07-02T14:02:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042515431999999999 2023-07-02T14:02:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:288 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312112+0000", 2023-07-02T14:02:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.982410019, 2023-07-02T14:02:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051367833000000002, 2023-07-02T14:02:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 288 2023-07-02T14:02:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312112+0000", 2023-07-02T14:02:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312112+0000", 2023-07-02T14:02:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312111+0000", 2023-07-02T14:02:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8513999999999998e-05 2023-07-02T14:02:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312150+0000", 2023-07-02T14:02:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6499999999999996e-06 2023-07-02T14:02:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312157+0000", 2023-07-02T14:02:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7799999999999997e-06 2023-07-02T14:02:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312163+0000", 2023-07-02T14:02:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039867382 2023-07-02T14:02:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352031+0000", 2023-07-02T14:02:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6370000000000002e-05 2023-07-02T14:02:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352107+0000", 2023-07-02T14:02:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000156021 2023-07-02T14:02:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352263+0000", 2023-07-02T14:02:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023521890000000002 2023-07-02T14:02:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354615+0000", 2023-07-02T14:02:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087542309999999995 2023-07-02T14:02:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363369+0000", 2023-07-02T14:02:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3217999999999998e-05 2023-07-02T14:02:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363423+0000", 2023-07-02T14:02:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7228999999999997e-05 2023-07-02T14:02:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363480+0000", 2023-07-02T14:02:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051367833000000002 2023-07-02T14:02:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:289 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:02:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.9823539710000002, 2023-07-02T14:02:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051413158, 2023-07-02T14:02:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 289 2023-07-02T14:02:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:02:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:02:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:02:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1689e-05 2023-07-02T14:02:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312209+0000", 2023-07-02T14:02:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1070000000000005e-06 2023-07-02T14:02:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312217+0000", 2023-07-02T14:02:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.634e-06 2023-07-02T14:02:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312222+0000", 2023-07-02T14:02:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040512761000000001 2023-07-02T14:02:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352735+0000", 2023-07-02T14:02:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000119184 2023-07-02T14:02:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352854+0000", 2023-07-02T14:02:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021481899999999999 2023-07-02T14:02:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353069+0000", 2023-07-02T14:02:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001601707 2023-07-02T14:02:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354670+0000", 2023-07-02T14:02:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088168570000000009 2023-07-02T14:02:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363487+0000", 2023-07-02T14:02:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9635999999999997e-05 2023-07-02T14:02:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363537+0000", 2023-07-02T14:02:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4292999999999998e-05 2023-07-02T14:02:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363581+0000", 2023-07-02T14:02:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051413158 2023-07-02T14:02:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:290 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:02:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.9822947260000001, 2023-07-02T14:02:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051455062000000003, 2023-07-02T14:02:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 290 2023-07-02T14:02:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:02:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:02:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:02:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6469000000000003e-05 2023-07-02T14:02:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312293+0000", 2023-07-02T14:02:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4630000000000007e-06 2023-07-02T14:02:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312303+0000", 2023-07-02T14:02:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8969999999999999e-06 2023-07-02T14:02:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312307+0000", 2023-07-02T14:02:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041261548000000002 2023-07-02T14:02:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353569+0000", 2023-07-02T14:02:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9594999999999999e-05 2023-07-02T14:02:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353659+0000", 2023-07-02T14:02:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018390600000000001 2023-07-02T14:02:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353843+0000", 2023-07-02T14:02:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035173100000000001 2023-07-02T14:02:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357360+0000", 2023-07-02T14:02:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062295670000000001 2023-07-02T14:02:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363589+0000", 2023-07-02T14:02:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9854000000000001e-05 2023-07-02T14:02:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363639+0000", 2023-07-02T14:02:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3133999999999997e-05 2023-07-02T14:02:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363682+0000", 2023-07-02T14:02:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051455062000000003 2023-07-02T14:02:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:291 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:02:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.982208687, 2023-07-02T14:02:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051527860000000002, 2023-07-02T14:02:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 291 2023-07-02T14:02:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:02:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:02:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:02:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2448999999999998e-05 2023-07-02T14:02:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312375+0000", 2023-07-02T14:02:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5259999999999993e-06 2023-07-02T14:02:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312384+0000", 2023-07-02T14:02:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0610000000000003e-06 2023-07-02T14:02:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312389+0000", 2023-07-02T14:02:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042326111 2023-07-02T14:02:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354715+0000", 2023-07-02T14:02:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3998999999999995e-05 2023-07-02T14:02:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354779+0000", 2023-07-02T14:02:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010269500000000001 2023-07-02T14:02:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354882+0000", 2023-07-02T14:02:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002533736 2023-07-02T14:02:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357415+0000", 2023-07-02T14:02:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062765700000000004 2023-07-02T14:02:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363692+0000", 2023-07-02T14:02:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7095000000000001e-05 2023-07-02T14:02:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363749+0000", 2023-07-02T14:02:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2263999999999996e-05 2023-07-02T14:02:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363841+0000", 2023-07-02T14:02:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051527860000000002 2023-07-02T14:02:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:292 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:02:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.982126439, 2023-07-02T14:02:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051556056000000003, 2023-07-02T14:02:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 292 2023-07-02T14:02:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:02:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:02:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312395+0000", 2023-07-02T14:02:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9624e-05 2023-07-02T14:02:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312425+0000", 2023-07-02T14:02:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:02:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312432+0000", 2023-07-02T14:02:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7550000000000002e-06 2023-07-02T14:02:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312437+0000", 2023-07-02T14:02:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.043823036000000003 2023-07-02T14:02:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356260+0000", 2023-07-02T14:02:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6021999999999997e-05 2023-07-02T14:02:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356346+0000", 2023-07-02T14:02:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020076600000000001 2023-07-02T14:02:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356547+0000", 2023-07-02T14:02:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020766410000000002 2023-07-02T14:02:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358623+0000", 2023-07-02T14:02:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005231192 2023-07-02T14:02:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363854+0000", 2023-07-02T14:02:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3423e-05 2023-07-02T14:02:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363908+0000", 2023-07-02T14:02:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3825999999999997e-05 2023-07-02T14:02:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363952+0000", 2023-07-02T14:02:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051556056000000003 2023-07-02T14:02:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:293 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:02:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.9820802689999999, 2023-07-02T14:02:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051614769999999997, 2023-07-02T14:02:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 293 2023-07-02T14:02:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:02:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:02:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312441+0000", 2023-07-02T14:02:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9609999999999999e-05 2023-07-02T14:02:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312471+0000", 2023-07-02T14:02:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7069999999999992e-06 2023-07-02T14:02:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312478+0000", 2023-07-02T14:02:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1190000000000002e-06 2023-07-02T14:02:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312485+0000", 2023-07-02T14:02:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.044956387 2023-07-02T14:02:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357441+0000", 2023-07-02T14:02:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4134999999999994e-05 2023-07-02T14:02:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357515+0000", 2023-07-02T14:02:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000147408 2023-07-02T14:02:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357663+0000", 2023-07-02T14:02:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056472639999999999 2023-07-02T14:02:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363310+0000", 2023-07-02T14:02:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00065402600000000004 2023-07-02T14:02:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363964+0000", 2023-07-02T14:02:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0497000000000001e-05 2023-07-02T14:02:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364014+0000", 2023-07-02T14:02:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2243000000000002e-05 2023-07-02T14:02:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364057+0000", 2023-07-02T14:02:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051614769999999997 2023-07-02T14:02:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:294 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1048576~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:02:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.9820322699999999, 2023-07-02T14:02:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051679527000000003, 2023-07-02T14:02:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 294 2023-07-02T14:02:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:02:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:02:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312489+0000", 2023-07-02T14:02:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8204999999999998e-05 2023-07-02T14:02:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312517+0000", 2023-07-02T14:02:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7759999999999997e-06 2023-07-02T14:02:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312525+0000", 2023-07-02T14:02:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8730000000000001e-06 2023-07-02T14:02:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312532+0000", 2023-07-02T14:02:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.045442206999999998 2023-07-02T14:02:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357974+0000", 2023-07-02T14:02:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4422000000000002e-05 2023-07-02T14:02:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358029+0000", 2023-07-02T14:02:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149519 2023-07-02T14:02:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358178+0000", 2023-07-02T14:02:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051779390000000003 2023-07-02T14:02:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363356+0000", 2023-07-02T14:02:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00070707699999999997 2023-07-02T14:02:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364063+0000", 2023-07-02T14:02:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6835999999999999e-05 2023-07-02T14:02:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364090+0000", 2023-07-02T14:02:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9263000000000003e-05 2023-07-02T14:02:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364169+0000", 2023-07-02T14:02:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051679527000000003 2023-07-02T14:02:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:423 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [write 2555904~1638400] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.330332+0000", 2023-07-02T14:02:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.964190275, 2023-07-02T14:02:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038020074000000001, 2023-07-02T14:02:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 423 2023-07-02T14:02:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.330332+0000", 2023-07-02T14:02:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.330332+0000", 2023-07-02T14:02:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:02:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.330330+0000", 2023-07-02T14:02:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060720390000000004 2023-07-02T14:02:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336402+0000", 2023-07-02T14:02:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8478000000000002e-05 2023-07-02T14:02:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336421+0000", 2023-07-02T14:02:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0624999999999999e-05 2023-07-02T14:02:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336451+0000", 2023-07-02T14:02:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022199402 2023-07-02T14:02:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358651+0000", 2023-07-02T14:02:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1981000000000001e-05 2023-07-02T14:02:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358723+0000", 2023-07-02T14:02:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163117 2023-07-02T14:02:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358886+0000", 2023-07-02T14:02:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005296762 2023-07-02T14:02:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364183+0000", 2023-07-02T14:02:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040861739999999997 2023-07-02T14:02:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368269+0000", 2023-07-02T14:02:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7169000000000003e-05 2023-07-02T14:02:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368306+0000", 2023-07-02T14:02:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5760999999999998e-05 2023-07-02T14:02:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368352+0000", 2023-07-02T14:02:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038020074000000001 2023-07-02T14:02:43.494 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:427 2.2 2:44c63967:::rbd_data.125de1449605.0000000000000101:head [write 2621440~1572864] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.336464+0000", 2023-07-02T14:02:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.958058447, 2023-07-02T14:02:43.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037760331000000001, 2023-07-02T14:02:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 427 2023-07-02T14:02:43.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336464+0000", 2023-07-02T14:02:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336464+0000", 2023-07-02T14:02:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:02:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336462+0000", 2023-07-02T14:02:43.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050934140000000001 2023-07-02T14:02:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.341555+0000", 2023-07-02T14:02:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6209e-05 2023-07-02T14:02:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.341572+0000", 2023-07-02T14:02:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.114e-06 2023-07-02T14:02:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.341579+0000", 2023-07-02T14:02:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019481099000000002 2023-07-02T14:02:43.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361060+0000", 2023-07-02T14:02:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5408000000000005e-05 2023-07-02T14:02:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:43.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361155+0000", 2023-07-02T14:02:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000186515 2023-07-02T14:02:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361342+0000", 2023-07-02T14:02:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003066465 2023-07-02T14:02:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364408+0000", 2023-07-02T14:02:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0096922329999999998 2023-07-02T14:02:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:43.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.374100+0000", 2023-07-02T14:02:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7960999999999995e-05 2023-07-02T14:02:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.374168+0000", 2023-07-02T14:02:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5458000000000003e-05 2023-07-02T14:02:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:43.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.374224+0000", 2023-07-02T14:02:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037760331000000001 2023-07-02T14:02:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:43.505 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:43.506 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:43.508 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:02:43.508 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:43.506+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:43.550 INFO:tasks.workunit.client.0.smithi163.stderr: 66050K .......... .......... .......... .......... .......... 39% 30.3M 5s 2023-07-02T14:02:43.551 INFO:tasks.workunit.client.0.smithi163.stderr: 66100K .......... .......... .......... .......... .......... 39% 22.6M 5s 2023-07-02T14:02:43.551 INFO:tasks.workunit.client.0.smithi163.stderr: 66150K .......... .......... .......... .......... .......... 39% 17.1M 5s 2023-07-02T14:02:43.551 INFO:tasks.workunit.client.0.smithi163.stderr: 66200K .......... .......... .......... .......... .......... 39% 20.2M 5s 2023-07-02T14:02:43.551 INFO:tasks.workunit.client.0.smithi163.stderr: 66250K .......... .......... .......... .......... .......... 39% 23.6M 5s 2023-07-02T14:02:43.551 INFO:tasks.workunit.client.0.smithi163.stderr: 66300K .......... .......... .......... .......... .......... 39% 22.3M 5s 2023-07-02T14:02:43.551 INFO:tasks.workunit.client.0.smithi163.stderr: 66350K .......... .......... .......... .......... .......... 39% 14.6M 5s 2023-07-02T14:02:43.551 INFO:tasks.workunit.client.0.smithi163.stderr: 66400K .......... .......... .......... .......... .......... 39% 17.6M 5s 2023-07-02T14:02:43.552 INFO:tasks.workunit.client.0.smithi163.stderr: 66450K .......... .......... .......... .......... .......... 39% 22.5M 5s 2023-07-02T14:02:43.552 INFO:tasks.workunit.client.0.smithi163.stderr: 66500K .......... .......... .......... .......... .......... 39% 16.7M 5s 2023-07-02T14:02:43.552 INFO:tasks.workunit.client.0.smithi163.stderr: 66550K .......... .......... .......... .......... .......... 39% 39.4M 5s 2023-07-02T14:02:43.552 INFO:tasks.workunit.client.0.smithi163.stderr: 66600K .......... .......... .......... .......... .......... 39% 17.0M 5s 2023-07-02T14:02:43.552 INFO:tasks.workunit.client.0.smithi163.stderr: 66650K .......... .......... .......... .......... .......... 39% 27.3M 5s 2023-07-02T14:02:43.552 INFO:tasks.workunit.client.0.smithi163.stderr: 66700K .......... .......... .......... .......... .......... 39% 16.4M 5s 2023-07-02T14:02:43.552 INFO:tasks.workunit.client.0.smithi163.stderr: 66750K .......... .......... .......... .......... .......... 39% 16.6M 5s 2023-07-02T14:02:43.553 INFO:tasks.workunit.client.0.smithi163.stderr: 66800K .......... .......... .......... .......... .......... 39% 36.5M 5s 2023-07-02T14:02:43.553 INFO:tasks.workunit.client.0.smithi163.stderr: 66850K .......... .......... .......... .......... .......... 39% 16.5M 5s 2023-07-02T14:02:43.553 INFO:tasks.workunit.client.0.smithi163.stderr: 66900K .......... .......... .......... .......... .......... 39% 22.3M 5s 2023-07-02T14:02:43.553 INFO:tasks.workunit.client.0.smithi163.stderr: 66950K .......... .......... .......... .......... .......... 39% 24.8M 5s 2023-07-02T14:02:43.553 INFO:tasks.workunit.client.0.smithi163.stderr: 67000K .......... .......... .......... .......... .......... 39% 36.2M 5s 2023-07-02T14:02:43.553 INFO:tasks.workunit.client.0.smithi163.stderr: 67050K .......... .......... .......... .......... .......... 39% 18.9M 5s 2023-07-02T14:02:43.553 INFO:tasks.workunit.client.0.smithi163.stderr: 67100K .......... .......... .......... .......... .......... 39% 13.5M 5s 2023-07-02T14:02:43.553 INFO:tasks.workunit.client.0.smithi163.stderr: 67150K .......... .......... .......... .......... .......... 39% 21.3M 5s 2023-07-02T14:02:43.554 INFO:tasks.workunit.client.0.smithi163.stderr: 67200K .......... .......... .......... .......... .......... 39% 21.4M 5s 2023-07-02T14:02:43.554 INFO:tasks.workunit.client.0.smithi163.stderr: 67250K .......... .......... .......... .......... .......... 39% 16.4M 5s 2023-07-02T14:02:43.554 INFO:tasks.workunit.client.0.smithi163.stderr: 67300K .......... .......... .......... .......... .......... 39% 29.4M 5s 2023-07-02T14:02:43.554 INFO:tasks.workunit.client.0.smithi163.stderr: 67350K .......... .......... .......... .......... .......... 39% 28.8M 5s 2023-07-02T14:02:43.554 INFO:tasks.workunit.client.0.smithi163.stderr: 67400K .......... .......... .......... .......... .......... 40% 14.0M 5s 2023-07-02T14:02:43.554 INFO:tasks.workunit.client.0.smithi163.stderr: 67450K .......... .......... .......... .......... .......... 40% 33.2M 5s 2023-07-02T14:02:43.554 INFO:tasks.workunit.client.0.smithi163.stderr: 67500K .......... .......... .......... .......... .......... 40% 31.6M 5s 2023-07-02T14:02:43.554 INFO:tasks.workunit.client.0.smithi163.stderr: 67550K .......... .......... .......... .......... .......... 40% 16.9M 5s 2023-07-02T14:02:43.555 INFO:tasks.workunit.client.0.smithi163.stderr: 67600K .......... .......... .......... .......... .......... 40% 13.9M 5s 2023-07-02T14:02:43.555 INFO:tasks.workunit.client.0.smithi163.stderr: 67650K .......... .......... .......... .......... .......... 40% 18.9M 5s 2023-07-02T14:02:43.555 INFO:tasks.workunit.client.0.smithi163.stderr: 67700K .......... .......... .......... .......... .......... 40% 27.9M 5s 2023-07-02T14:02:43.555 INFO:tasks.workunit.client.0.smithi163.stderr: 67750K .......... .......... .......... .......... .......... 40% 17.9M 5s 2023-07-02T14:02:43.555 INFO:tasks.workunit.client.0.smithi163.stderr: 67800K .......... .......... .......... .......... .......... 40% 23.8M 5s 2023-07-02T14:02:43.555 INFO:tasks.workunit.client.0.smithi163.stderr: 67850K .......... .......... .......... .......... .......... 40% 21.3M 5s 2023-07-02T14:02:43.555 INFO:tasks.workunit.client.0.smithi163.stderr: 67900K .......... .......... .......... .......... .......... 40% 28.7M 5s 2023-07-02T14:02:43.555 INFO:tasks.workunit.client.0.smithi163.stderr: 67950K .......... .......... .......... .......... .......... 40% 10.4M 5s 2023-07-02T14:02:43.556 INFO:tasks.workunit.client.0.smithi163.stderr: 68000K .......... .......... .......... .......... .......... 40% 25.7M 5s 2023-07-02T14:02:43.556 INFO:tasks.workunit.client.0.smithi163.stderr: 68050K .......... .......... .......... .......... .......... 40% 35.2M 5s 2023-07-02T14:02:43.556 INFO:tasks.workunit.client.0.smithi163.stderr: 68100K .......... .......... .......... .......... .......... 40% 16.2M 5s 2023-07-02T14:02:43.556 INFO:tasks.workunit.client.0.smithi163.stderr: 68150K .......... .......... .......... .......... .......... 40% 18.4M 5s 2023-07-02T14:02:43.556 INFO:tasks.workunit.client.0.smithi163.stderr: 68200K .......... .......... .......... .......... .......... 40% 31.0M 5s 2023-07-02T14:02:43.556 INFO:tasks.workunit.client.0.smithi163.stderr: 68250K .......... .......... .......... .......... .......... 40% 14.5M 5s 2023-07-02T14:02:43.556 INFO:tasks.workunit.client.0.smithi163.stderr: 68300K .......... .......... .......... .......... .......... 40% 22.8M 5s 2023-07-02T14:02:43.556 INFO:tasks.workunit.client.0.smithi163.stderr: 68350K .......... .......... .......... .......... .......... 40% 19.1M 5s 2023-07-02T14:02:43.557 INFO:tasks.workunit.client.0.smithi163.stderr: 68400K .......... .......... .......... .......... .......... 40% 24.2M 5s 2023-07-02T14:02:43.557 INFO:tasks.workunit.client.0.smithi163.stderr: 68450K .......... .......... .......... .......... .......... 40% 27.5M 5s 2023-07-02T14:02:43.557 INFO:tasks.workunit.client.0.smithi163.stderr: 68500K .......... .......... .......... .......... .......... 40% 16.2M 5s 2023-07-02T14:02:43.557 INFO:tasks.workunit.client.0.smithi163.stderr: 68550K .......... .......... .......... .......... .......... 40% 20.6M 5s 2023-07-02T14:02:43.557 INFO:tasks.workunit.client.0.smithi163.stderr: 68600K .......... .......... .......... .......... .......... 40% 25.2M 5s 2023-07-02T14:02:43.557 INFO:tasks.workunit.client.0.smithi163.stderr: 68650K .......... .......... .......... .......... .......... 40% 16.2M 5s 2023-07-02T14:02:43.557 INFO:tasks.workunit.client.0.smithi163.stderr: 68700K .......... .......... .......... .......... .......... 40% 34.1M 5s 2023-07-02T14:02:43.557 INFO:tasks.workunit.client.0.smithi163.stderr: 68750K .......... .......... .......... .......... .......... 40% 15.7M 5s 2023-07-02T14:02:43.558 INFO:tasks.workunit.client.0.smithi163.stderr: 68800K .......... .......... .......... .......... .......... 40% 19.2M 5s 2023-07-02T14:02:43.558 INFO:tasks.workunit.client.0.smithi163.stderr: 68850K .......... .......... .......... .......... .......... 40% 16.2M 5s 2023-07-02T14:02:43.558 INFO:tasks.workunit.client.0.smithi163.stderr: 68900K .......... .......... .......... .......... .......... 40% 25.3M 5s 2023-07-02T14:02:43.558 INFO:tasks.workunit.client.0.smithi163.stderr: 68950K .......... .......... .......... .......... .......... 40% 35.4M 5s 2023-07-02T14:02:43.558 INFO:tasks.workunit.client.0.smithi163.stderr: 69000K .......... .......... .......... .......... .......... 40% 18.2M 5s 2023-07-02T14:02:43.558 INFO:tasks.workunit.client.0.smithi163.stderr: 69050K .......... .......... .......... .......... .......... 40% 18.0M 5s 2023-07-02T14:02:43.558 INFO:tasks.workunit.client.0.smithi163.stderr: 69100K .......... .......... .......... .......... .......... 41% 30.5M 5s 2023-07-02T14:02:43.559 INFO:tasks.workunit.client.0.smithi163.stderr: 69150K .......... .......... .......... .......... .......... 41% 12.2M 5s 2023-07-02T14:02:43.559 INFO:tasks.workunit.client.0.smithi163.stderr: 69200K .......... .......... .......... .......... .......... 41% 26.4M 5s 2023-07-02T14:02:43.559 INFO:tasks.workunit.client.0.smithi163.stderr: 69250K .......... .......... .......... .......... .......... 41% 33.3M 5s 2023-07-02T14:02:43.559 INFO:tasks.workunit.client.0.smithi163.stderr: 69300K .......... .......... .......... .......... .......... 41% 21.6M 5s 2023-07-02T14:02:43.559 INFO:tasks.workunit.client.0.smithi163.stderr: 69350K .......... .......... .......... .......... .......... 41% 24.2M 5s 2023-07-02T14:02:43.559 INFO:tasks.workunit.client.0.smithi163.stderr: 69400K .......... .......... .......... .......... .......... 41% 14.2M 5s 2023-07-02T14:02:43.559 INFO:tasks.workunit.client.0.smithi163.stderr: 69450K .......... .......... .......... .......... .......... 41% 19.4M 5s 2023-07-02T14:02:43.559 INFO:tasks.workunit.client.0.smithi163.stderr: 69500K .......... .......... .......... .......... .......... 41% 18.7M 5s 2023-07-02T14:02:43.560 INFO:tasks.workunit.client.0.smithi163.stderr: 69550K .......... .......... .......... .......... .......... 41% 15.5M 5s 2023-07-02T14:02:43.560 INFO:tasks.workunit.client.0.smithi163.stderr: 69600K .......... .......... .......... .......... .......... 41% 27.4M 5s 2023-07-02T14:02:43.560 INFO:tasks.workunit.client.0.smithi163.stderr: 69650K .......... .......... .......... .......... .......... 41% 25.6M 5s 2023-07-02T14:02:43.560 INFO:tasks.workunit.client.0.smithi163.stderr: 69700K .......... .......... .......... .......... .......... 41% 14.5M 5s 2023-07-02T14:02:43.560 INFO:tasks.workunit.client.0.smithi163.stderr: 69750K .......... .......... .......... .......... .......... 41% 19.8M 5s 2023-07-02T14:02:43.560 INFO:tasks.workunit.client.0.smithi163.stderr: 69800K .......... .......... .......... .......... .......... 41% 31.7M 5s 2023-07-02T14:02:43.560 INFO:tasks.workunit.client.0.smithi163.stderr: 69850K .......... .......... .......... .......... .......... 41% 24.2M 5s 2023-07-02T14:02:43.560 INFO:tasks.workunit.client.0.smithi163.stderr: 69900K .......... .......... .......... .......... .......... 41% 14.5M 5s 2023-07-02T14:02:43.561 INFO:tasks.workunit.client.0.smithi163.stderr: 69950K .......... .......... .......... .......... .......... 41% 18.2M 5s 2023-07-02T14:02:43.561 INFO:tasks.workunit.client.0.smithi163.stderr: 70000K .......... .......... .......... .......... .......... 41% 19.6M 5s 2023-07-02T14:02:43.561 INFO:tasks.workunit.client.0.smithi163.stderr: 70050K .......... .......... .......... .......... .......... 41% 18.7M 5s 2023-07-02T14:02:43.561 INFO:tasks.workunit.client.0.smithi163.stderr: 70100K .......... .......... .......... .......... .......... 41% 25.4M 5s 2023-07-02T14:02:43.561 INFO:tasks.workunit.client.0.smithi163.stderr: 70150K .......... .......... .......... .......... .......... 41% 30.3M 5s 2023-07-02T14:02:43.561 INFO:tasks.workunit.client.0.smithi163.stderr: 70200K .......... .......... .......... .......... .......... 41% 22.2M 5s 2023-07-02T14:02:43.561 INFO:tasks.workunit.client.0.smithi163.stderr: 70250K .......... .......... .......... .......... .......... 41% 16.0M 5s 2023-07-02T14:02:43.561 INFO:tasks.workunit.client.0.smithi163.stderr: 70300K .......... .......... .......... .......... .......... 41% 37.4M 5s 2023-07-02T14:02:43.562 INFO:tasks.workunit.client.0.smithi163.stderr: 70350K .......... .......... .......... .......... .......... 41% 15.0M 5s 2023-07-02T14:02:43.562 INFO:tasks.workunit.client.0.smithi163.stderr: 70400K .......... .......... .......... .......... .......... 41% 16.0M 5s 2023-07-02T14:02:43.562 INFO:tasks.workunit.client.0.smithi163.stderr: 70450K .......... .......... .......... .......... .......... 41% 24.2M 5s 2023-07-02T14:02:43.562 INFO:tasks.workunit.client.0.smithi163.stderr: 70500K .......... .......... .......... .......... .......... 41% 24.9M 5s 2023-07-02T14:02:43.562 INFO:tasks.workunit.client.0.smithi163.stderr: 70550K .......... .......... .......... .......... .......... 41% 17.9M 5s 2023-07-02T14:02:43.562 INFO:tasks.workunit.client.0.smithi163.stderr: 70600K .......... .......... .......... .......... .......... 41% 44.4M 5s 2023-07-02T14:02:43.562 INFO:tasks.workunit.client.0.smithi163.stderr: 70650K .......... .......... .......... .......... .......... 41% 13.7M 5s 2023-07-02T14:02:43.562 INFO:tasks.workunit.client.0.smithi163.stderr: 70700K .......... .......... .......... .......... .......... 41% 34.3M 5s 2023-07-02T14:02:43.563 INFO:tasks.workunit.client.0.smithi163.stderr: 70750K .......... .......... .......... .......... .......... 42% 13.6M 5s 2023-07-02T14:02:43.563 INFO:tasks.workunit.client.0.smithi163.stderr: 70800K .......... .......... .......... .......... .......... 42% 26.8M 5s 2023-07-02T14:02:43.563 INFO:tasks.workunit.client.0.smithi163.stderr: 70850K .......... .......... .......... .......... .......... 42% 19.7M 5s 2023-07-02T14:02:43.563 INFO:tasks.workunit.client.0.smithi163.stderr: 70900K .......... .......... .......... .......... .......... 42% 18.1M 5s 2023-07-02T14:02:43.563 INFO:tasks.workunit.client.0.smithi163.stderr: 70950K .......... .......... .......... .......... .......... 42% 25.6M 5s 2023-07-02T14:02:43.563 INFO:tasks.workunit.client.0.smithi163.stderr: 71000K .......... .......... .......... .......... .......... 42% 19.8M 5s 2023-07-02T14:02:43.563 INFO:tasks.workunit.client.0.smithi163.stderr: 71050K .......... .......... .......... .......... .......... 42% 18.0M 5s 2023-07-02T14:02:43.563 INFO:tasks.workunit.client.0.smithi163.stderr: 71100K .......... .......... .......... .......... .......... 42% 33.7M 5s 2023-07-02T14:02:43.564 INFO:tasks.workunit.client.0.smithi163.stderr: 71150K .......... .......... .......... .......... .......... 42% 13.3M 5s 2023-07-02T14:02:43.564 INFO:tasks.workunit.client.0.smithi163.stderr: 71200K .......... .......... .......... .......... .......... 42% 24.4M 5s 2023-07-02T14:02:43.564 INFO:tasks.workunit.client.0.smithi163.stderr: 71250K .......... .......... .......... .......... .......... 42% 20.3M 5s 2023-07-02T14:02:43.564 INFO:tasks.workunit.client.0.smithi163.stderr: 71300K .......... .......... .......... .......... .......... 42% 17.0M 5s 2023-07-02T14:02:43.564 INFO:tasks.workunit.client.0.smithi163.stderr: 71350K .......... .......... .......... .......... .......... 42% 28.1M 5s 2023-07-02T14:02:43.567 INFO:tasks.workunit.client.0.smithi163.stderr: 71400K .......... .......... .......... .......... .......... 42% 19.5M 5s 2023-07-02T14:02:43.567 INFO:tasks.workunit.client.0.smithi163.stderr: 71450K .......... .......... .......... .......... .......... 42% 15.8M 5s 2023-07-02T14:02:43.568 INFO:tasks.workunit.client.0.smithi163.stderr: 71500K .......... .......... .......... .......... .......... 42% 27.7M 5s 2023-07-02T14:02:43.568 INFO:tasks.workunit.client.0.smithi163.stderr: 71550K .......... .......... .......... .......... .......... 42% 16.4M 5s 2023-07-02T14:02:43.568 INFO:tasks.workunit.client.0.smithi163.stderr: 71600K .......... .......... .......... .......... .......... 42% 18.2M 5s 2023-07-02T14:02:43.568 INFO:tasks.workunit.client.0.smithi163.stderr: 71650K .......... .......... .......... .......... .......... 42% 22.9M 5s 2023-07-02T14:02:43.568 INFO:tasks.workunit.client.0.smithi163.stderr: 71700K .......... .......... .......... .......... .......... 42% 22.1M 5s 2023-07-02T14:02:43.569 INFO:tasks.workunit.client.0.smithi163.stderr: 71750K .......... .......... .......... .......... .......... 42% 28.2M 5s 2023-07-02T14:02:43.572 INFO:tasks.workunit.client.0.smithi163.stderr: 71800K .......... .......... .......... .......... .......... 42% 15.1M 5s 2023-07-02T14:02:43.574 INFO:tasks.workunit.client.0.smithi163.stderr: 71850K .......... .......... .......... .......... .......... 42% 28.0M 5s 2023-07-02T14:02:43.576 INFO:tasks.workunit.client.0.smithi163.stderr: 71900K .......... .......... .......... .......... .......... 42% 20.6M 5s 2023-07-02T14:02:43.579 INFO:tasks.workunit.client.0.smithi163.stderr: 71950K .......... .......... .......... .......... .......... 42% 16.6M 5s 2023-07-02T14:02:43.582 INFO:tasks.workunit.client.0.smithi163.stderr: 72000K .......... .......... .......... .......... .......... 42% 16.7M 5s 2023-07-02T14:02:43.583 INFO:tasks.workunit.client.0.smithi163.stderr: 72050K .......... .......... .......... .......... .......... 42% 36.7M 5s 2023-07-02T14:02:43.586 INFO:tasks.workunit.client.0.smithi163.stderr: 72100K .......... .......... .......... .......... .......... 42% 22.6M 5s 2023-07-02T14:02:43.588 INFO:tasks.workunit.client.0.smithi163.stderr: 72150K .......... .......... .......... .......... .......... 42% 21.4M 5s 2023-07-02T14:02:43.591 INFO:tasks.workunit.client.0.smithi163.stderr: 72200K .......... .......... .......... .......... .......... 42% 14.1M 5s 2023-07-02T14:02:43.593 INFO:tasks.workunit.client.0.smithi163.stderr: 72250K .......... .......... .......... .......... .......... 42% 32.3M 5s 2023-07-02T14:02:43.595 INFO:tasks.workunit.client.0.smithi163.stderr: 72300K .......... .......... .......... .......... .......... 42% 19.4M 5s 2023-07-02T14:02:43.598 INFO:tasks.workunit.client.0.smithi163.stderr: 72350K .......... .......... .......... .......... .......... 42% 20.5M 5s 2023-07-02T14:02:43.600 INFO:tasks.workunit.client.0.smithi163.stderr: 72400K .......... .......... .......... .......... .......... 42% 18.1M 5s 2023-07-02T14:02:43.605 INFO:tasks.workunit.client.0.smithi163.stderr: 72450K .......... .......... .......... .......... .......... 43% 25.0M 5s 2023-07-02T14:02:43.605 INFO:tasks.workunit.client.0.smithi163.stderr: 72500K .......... .......... .......... .......... .......... 43% 19.6M 5s 2023-07-02T14:02:43.607 INFO:tasks.workunit.client.0.smithi163.stderr: 72550K .......... .......... .......... .......... .......... 43% 25.3M 5s 2023-07-02T14:02:43.608 INFO:tasks.workunit.client.0.smithi163.stderr: 72600K .......... .......... .......... .......... .......... 43% 29.4M 5s 2023-07-02T14:02:43.610 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:43.609+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:43.611 INFO:tasks.workunit.client.0.smithi163.stderr: 72650K .......... .......... .......... .......... .......... 43% 16.8M 5s 2023-07-02T14:02:43.614 INFO:tasks.workunit.client.0.smithi163.stderr: 72700K .......... .......... .......... .......... .......... 43% 16.6M 5s 2023-07-02T14:02:43.617 INFO:tasks.workunit.client.0.smithi163.stderr: 72750K .......... .......... .......... .......... .......... 43% 20.2M 5s 2023-07-02T14:02:43.619 INFO:tasks.workunit.client.0.smithi163.stderr: 72800K .......... .......... .......... .......... .......... 43% 20.1M 5s 2023-07-02T14:02:43.622 INFO:tasks.workunit.client.0.smithi163.stderr: 72850K .......... .......... .......... .......... .......... 43% 15.4M 5s 2023-07-02T14:02:43.624 INFO:tasks.workunit.client.0.smithi163.stderr: 72900K .......... .......... .......... .......... .......... 43% 31.9M 5s 2023-07-02T14:02:43.627 INFO:tasks.workunit.client.0.smithi163.stderr: 72950K .......... .......... .......... .......... .......... 43% 17.1M 5s 2023-07-02T14:02:43.629 INFO:tasks.workunit.client.0.smithi163.stderr: 73000K .......... .......... .......... .......... .......... 43% 22.2M 5s 2023-07-02T14:02:43.635 INFO:tasks.workunit.client.0.smithi163.stderr: 73050K .......... .......... .......... .......... .......... 43% 10.0M 5s 2023-07-02T14:02:43.635 INFO:tasks.workunit.client.0.smithi163.stderr: 73100K .......... .......... .......... .......... .......... 43% 112M 5s 2023-07-02T14:02:43.636 INFO:tasks.workunit.client.0.smithi163.stderr: 73150K .......... .......... .......... .......... .......... 43% 23.8M 5s 2023-07-02T14:02:43.638 INFO:tasks.workunit.client.0.smithi163.stderr: 73200K .......... .......... .......... .......... .......... 43% 25.1M 5s 2023-07-02T14:02:43.643 INFO:tasks.workunit.client.0.smithi163.stderr: 73250K .......... .......... .......... .......... .......... 43% 13.1M 5s 2023-07-02T14:02:43.644 INFO:tasks.workunit.client.0.smithi163.stderr: 73300K .......... .......... .......... .......... .......... 43% 26.5M 5s 2023-07-02T14:02:43.647 INFO:tasks.workunit.client.0.smithi163.stderr: 73350K .......... .......... .......... .......... .......... 43% 16.8M 5s 2023-07-02T14:02:43.648 INFO:tasks.workunit.client.0.smithi163.stderr: 73400K .......... .......... .......... .......... .......... 43% 31.9M 5s 2023-07-02T14:02:43.651 INFO:tasks.workunit.client.0.smithi163.stderr: 73450K .......... .......... .......... .......... .......... 43% 17.7M 5s 2023-07-02T14:02:43.653 INFO:tasks.workunit.client.0.smithi163.stderr: 73500K .......... .......... .......... .......... .......... 43% 31.9M 5s 2023-07-02T14:02:43.657 INFO:tasks.workunit.client.0.smithi163.stderr: 73550K .......... .......... .......... .......... .......... 43% 12.1M 5s 2023-07-02T14:02:43.659 INFO:tasks.workunit.client.0.smithi163.stderr: 73600K .......... .......... .......... .......... .......... 43% 20.1M 5s 2023-07-02T14:02:43.661 INFO:tasks.workunit.client.0.smithi163.stderr: 73650K .......... .......... .......... .......... .......... 43% 32.6M 5s 2023-07-02T14:02:43.664 INFO:tasks.workunit.client.0.smithi163.stderr: 73700K .......... .......... .......... .......... .......... 43% 24.0M 5s 2023-07-02T14:02:43.666 INFO:tasks.workunit.client.0.smithi163.stderr: 73750K .......... .......... .......... .......... .......... 43% 17.3M 5s 2023-07-02T14:02:43.668 INFO:tasks.workunit.client.0.smithi163.stderr: 73800K .......... .......... .......... .......... .......... 43% 19.4M 5s 2023-07-02T14:02:43.669 INFO:tasks.workunit.client.0.smithi163.stderr: 73850K .......... .......... .......... .......... .......... 43% 40.9M 5s 2023-07-02T14:02:43.672 INFO:tasks.workunit.client.0.smithi163.stderr: 73900K .......... .......... .......... .......... .......... 43% 19.1M 5s 2023-07-02T14:02:43.675 INFO:tasks.workunit.client.0.smithi163.stderr: 73950K .......... .......... .......... .......... .......... 43% 13.5M 5s 2023-07-02T14:02:43.678 INFO:tasks.workunit.client.0.smithi163.stderr: 74000K .......... .......... .......... .......... .......... 43% 19.7M 5s 2023-07-02T14:02:43.680 INFO:tasks.workunit.client.0.smithi163.stderr: 74050K .......... .......... .......... .......... .......... 43% 27.3M 5s 2023-07-02T14:02:43.682 INFO:tasks.workunit.client.0.smithi163.stderr: 74100K .......... .......... .......... .......... .......... 43% 18.8M 5s 2023-07-02T14:02:43.683 INFO:tasks.workunit.client.0.smithi163.stderr: 74150K .......... .......... .......... .......... .......... 44% 38.9M 5s 2023-07-02T14:02:43.686 INFO:tasks.workunit.client.0.smithi163.stderr: 74200K .......... .......... .......... .......... .......... 44% 22.4M 5s 2023-07-02T14:02:43.689 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:43.689 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:43.689 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:43.690 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:43.690 INFO:tasks.workunit.client.0.smithi163.stderr: 74250K .......... .......... .......... .......... .......... 44% 14.5M 5s 2023-07-02T14:02:43.691 INFO:tasks.workunit.client.0.smithi163.stderr: 74300K .......... .......... .......... .......... .......... 44% 25.5M 5s 2023-07-02T14:02:43.693 INFO:tasks.workunit.client.0.smithi163.stderr: 74350K .......... .......... .......... .......... .......... 44% 18.7M 5s 2023-07-02T14:02:43.696 INFO:tasks.workunit.client.0.smithi163.stderr: 74400K .......... .......... .......... .......... .......... 44% 18.4M 5s 2023-07-02T14:02:43.698 INFO:tasks.workunit.client.0.smithi163.stderr: 74450K .......... .......... .......... .......... .......... 44% 26.7M 5s 2023-07-02T14:02:43.699 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:02:43.701 INFO:tasks.workunit.client.0.smithi163.stderr: 74500K .......... .......... .......... .......... .......... 44% 14.3M 5s 2023-07-02T14:02:43.704 INFO:tasks.workunit.client.0.smithi163.stderr: 74550K .......... .......... .......... .......... .......... 44% 24.8M 5s 2023-07-02T14:02:43.707 INFO:tasks.workunit.client.0.smithi163.stderr: 74600K .......... .......... .......... .......... .......... 44% 16.8M 5s 2023-07-02T14:02:43.708 INFO:tasks.workunit.client.0.smithi163.stderr: 74650K .......... .......... .......... .......... .......... 44% 24.2M 5s 2023-07-02T14:02:43.710 INFO:tasks.workunit.client.0.smithi163.stderr: 74700K .......... .......... .......... .......... .......... 44% 26.0M 5s 2023-07-02T14:02:43.714 INFO:tasks.workunit.client.0.smithi163.stderr: 74750K .......... .......... .......... .......... .......... 44% 14.5M 5s 2023-07-02T14:02:43.717 INFO:tasks.workunit.client.0.smithi163.stderr: 74800K .......... .......... .......... .......... .......... 44% 17.9M 5s 2023-07-02T14:02:43.718 INFO:tasks.workunit.client.0.smithi163.stderr: 74850K .......... .......... .......... .......... .......... 44% 29.4M 5s 2023-07-02T14:02:43.721 INFO:tasks.workunit.client.0.smithi163.stderr: 74900K .......... .......... .......... .......... .......... 44% 17.4M 5s 2023-07-02T14:02:43.723 INFO:tasks.workunit.client.0.smithi163.stderr: 74950K .......... .......... .......... .......... .......... 44% 28.2M 5s 2023-07-02T14:02:43.725 INFO:tasks.workunit.client.0.smithi163.stderr: 75000K .......... .......... .......... .......... .......... 44% 17.7M 5s 2023-07-02T14:02:43.728 INFO:tasks.workunit.client.0.smithi163.stderr: 75050K .......... .......... .......... .......... .......... 44% 20.2M 5s 2023-07-02T14:02:43.731 INFO:tasks.workunit.client.0.smithi163.stderr: 75100K .......... .......... .......... .......... .......... 44% 15.6M 5s 2023-07-02T14:02:43.733 INFO:tasks.workunit.client.0.smithi163.stderr: 75150K .......... .......... .......... .......... .......... 44% 23.3M 5s 2023-07-02T14:02:43.736 INFO:tasks.workunit.client.0.smithi163.stderr: 75200K .......... .......... .......... .......... .......... 44% 16.8M 5s 2023-07-02T14:02:43.737 INFO:tasks.workunit.client.0.smithi163.stderr: 75250K .......... .......... .......... .......... .......... 44% 28.9M 5s 2023-07-02T14:02:43.739 INFO:tasks.workunit.client.0.smithi163.stderr: 75300K .......... .......... .......... .......... .......... 44% 33.4M 5s 2023-07-02T14:02:43.740 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:43.739+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:43.743 INFO:tasks.workunit.client.0.smithi163.stderr: 75350K .......... .......... .......... .......... .......... 44% 12.0M 5s 2023-07-02T14:02:43.744 INFO:tasks.workunit.client.0.smithi163.stderr: 75400K .......... .......... .......... .......... .......... 44% 29.2M 5s 2023-07-02T14:02:43.747 INFO:tasks.workunit.client.0.smithi163.stderr: 75450K .......... .......... .......... .......... .......... 44% 24.3M 5s 2023-07-02T14:02:43.749 INFO:tasks.workunit.client.0.smithi163.stderr: 75500K .......... .......... .......... .......... .......... 44% 18.6M 5s 2023-07-02T14:02:43.752 INFO:tasks.workunit.client.0.smithi163.stderr: 75550K .......... .......... .......... .......... .......... 44% 18.5M 5s 2023-07-02T14:02:43.755 INFO:tasks.workunit.client.0.smithi163.stderr: 75600K .......... .......... .......... .......... .......... 44% 20.1M 5s 2023-07-02T14:02:43.757 INFO:tasks.workunit.client.0.smithi163.stderr: 75650K .......... .......... .......... .......... .......... 44% 21.8M 5s 2023-07-02T14:02:43.759 INFO:tasks.workunit.client.0.smithi163.stderr: 75700K .......... .......... .......... .......... .......... 44% 23.3M 5s 2023-07-02T14:02:43.761 INFO:tasks.workunit.client.0.smithi163.stderr: 75750K .......... .......... .......... .......... .......... 44% 23.0M 5s 2023-07-02T14:02:43.764 INFO:tasks.workunit.client.0.smithi163.stderr: 75800K .......... .......... .......... .......... .......... 44% 18.5M 5s 2023-07-02T14:02:43.766 INFO:tasks.workunit.client.0.smithi163.stderr: 75850K .......... .......... .......... .......... .......... 45% 22.2M 5s 2023-07-02T14:02:43.767 INFO:tasks.workunit.client.0.smithi163.stderr: 75900K .......... .......... .......... .......... .......... 45% 29.9M 5s 2023-07-02T14:02:43.770 INFO:tasks.workunit.client.0.smithi163.stderr: 75950K .......... .......... .......... .......... .......... 45% 18.4M 5s 2023-07-02T14:02:43.773 INFO:tasks.workunit.client.0.smithi163.stderr: 76000K .......... .......... .......... .......... .......... 45% 16.3M 5s 2023-07-02T14:02:43.776 INFO:tasks.workunit.client.0.smithi163.stderr: 76050K .......... .......... .......... .......... .......... 45% 17.3M 5s 2023-07-02T14:02:43.777 INFO:tasks.workunit.client.0.smithi163.stderr: 76100K .......... .......... .......... .......... .......... 45% 29.4M 5s 2023-07-02T14:02:43.780 INFO:tasks.workunit.client.0.smithi163.stderr: 76150K .......... .......... .......... .......... .......... 45% 19.4M 5s 2023-07-02T14:02:43.782 INFO:tasks.workunit.client.0.smithi163.stderr: 76200K .......... .......... .......... .......... .......... 45% 31.3M 5s 2023-07-02T14:02:43.785 INFO:tasks.workunit.client.0.smithi163.stderr: 76250K .......... .......... .......... .......... .......... 45% 14.5M 5s 2023-07-02T14:02:43.787 INFO:tasks.workunit.client.0.smithi163.stderr: 76300K .......... .......... .......... .......... .......... 45% 26.8M 5s 2023-07-02T14:02:43.790 INFO:tasks.workunit.client.0.smithi163.stderr: 76350K .......... .......... .......... .......... .......... 45% 17.0M 5s 2023-07-02T14:02:43.793 INFO:tasks.workunit.client.0.smithi163.stderr: 76400K .......... .......... .......... .......... .......... 45% 16.8M 5s 2023-07-02T14:02:43.794 INFO:tasks.workunit.client.0.smithi163.stderr: 76450K .......... .......... .......... .......... .......... 45% 25.0M 5s 2023-07-02T14:02:43.796 INFO:tasks.workunit.client.0.smithi163.stderr: 76500K .......... .......... .......... .......... .......... 45% 24.8M 5s 2023-07-02T14:02:43.800 INFO:tasks.workunit.client.0.smithi163.stderr: 76550K .......... .......... .......... .......... .......... 45% 14.9M 5s 2023-07-02T14:02:43.802 INFO:tasks.workunit.client.0.smithi163.stderr: 76600K .......... .......... .......... .......... .......... 45% 23.6M 5s 2023-07-02T14:02:43.803 INFO:tasks.workunit.client.0.smithi163.stderr: 76650K .......... .......... .......... .......... .......... 45% 31.7M 5s 2023-07-02T14:02:43.805 INFO:tasks.workunit.client.0.smithi163.stderr: 76700K .......... .......... .......... .......... .......... 45% 24.1M 5s 2023-07-02T14:02:43.809 INFO:tasks.workunit.client.0.smithi163.stderr: 76750K .......... .......... .......... .......... .......... 45% 14.3M 5s 2023-07-02T14:02:43.809 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:43.807+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:43.811 INFO:tasks.workunit.client.0.smithi163.stderr: 76800K .......... .......... .......... .......... .......... 45% 18.1M 5s 2023-07-02T14:02:43.814 INFO:tasks.workunit.client.0.smithi163.stderr: 76850K .......... .......... .......... .......... .......... 45% 18.4M 5s 2023-07-02T14:02:43.817 INFO:tasks.workunit.client.0.smithi163.stderr: 76900K .......... .......... .......... .......... .......... 45% 16.9M 5s 2023-07-02T14:02:43.818 INFO:tasks.workunit.client.0.smithi163.stderr: 76950K .......... .......... .......... .......... .......... 45% 33.2M 5s 2023-07-02T14:02:43.820 INFO:tasks.workunit.client.0.smithi163.stderr: 77000K .......... .......... .......... .......... .......... 45% 26.2M 5s 2023-07-02T14:02:43.823 INFO:tasks.workunit.client.0.smithi163.stderr: 77050K .......... .......... .......... .......... .......... 45% 17.2M 5s 2023-07-02T14:02:43.825 INFO:tasks.workunit.client.0.smithi163.stderr: 77100K .......... .......... .......... .......... .......... 45% 21.0M 5s 2023-07-02T14:02:43.828 INFO:tasks.workunit.client.0.smithi163.stderr: 77150K .......... .......... .......... .......... .......... 45% 15.4M 5s 2023-07-02T14:02:43.831 INFO:tasks.workunit.client.0.smithi163.stderr: 77200K .......... .......... .......... .......... .......... 45% 25.6M 5s 2023-07-02T14:02:43.833 INFO:tasks.workunit.client.0.smithi163.stderr: 77250K .......... .......... .......... .......... .......... 45% 20.7M 5s 2023-07-02T14:02:43.836 INFO:tasks.workunit.client.0.smithi163.stderr: 77300K .......... .......... .......... .......... .......... 45% 18.7M 5s 2023-07-02T14:02:43.838 INFO:tasks.workunit.client.0.smithi163.stderr: 77350K .......... .......... .......... .......... .......... 45% 22.4M 5s 2023-07-02T14:02:43.841 INFO:tasks.workunit.client.0.smithi163.stderr: 77400K .......... .......... .......... .......... .......... 45% 16.3M 5s 2023-07-02T14:02:43.842 INFO:tasks.workunit.client.0.smithi163.stderr: 77450K .......... .......... .......... .......... .......... 45% 42.8M 5s 2023-07-02T14:02:43.845 INFO:tasks.workunit.client.0.smithi163.stderr: 77500K .......... .......... .......... .......... .......... 46% 16.9M 5s 2023-07-02T14:02:43.847 INFO:tasks.workunit.client.0.smithi163.stderr: 77550K .......... .......... .......... .......... .......... 46% 19.2M 5s 2023-07-02T14:02:43.851 INFO:tasks.workunit.client.0.smithi163.stderr: 77600K .......... .......... .......... .......... .......... 46% 14.9M 5s 2023-07-02T14:02:43.852 INFO:tasks.workunit.client.0.smithi163.stderr: 77650K .......... .......... .......... .......... .......... 46% 28.5M 5s 2023-07-02T14:02:43.854 INFO:tasks.workunit.client.0.smithi163.stderr: 77700K .......... .......... .......... .......... .......... 46% 36.0M 5s 2023-07-02T14:02:43.856 INFO:tasks.workunit.client.0.smithi163.stderr: 77750K .......... .......... .......... .......... .......... 46% 20.1M 5s 2023-07-02T14:02:43.859 INFO:tasks.workunit.client.0.smithi163.stderr: 77800K .......... .......... .......... .......... .......... 46% 14.6M 5s 2023-07-02T14:02:43.861 INFO:tasks.workunit.client.0.smithi163.stderr: 77850K .......... .......... .......... .......... .......... 46% 30.0M 5s 2023-07-02T14:02:43.862 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:43.862 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:43.862 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:43.862 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:43.863 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:43.864 INFO:tasks.workunit.client.0.smithi163.stderr: 77900K .......... .......... .......... .......... .......... 46% 15.6M 5s 2023-07-02T14:02:43.867 INFO:tasks.workunit.client.0.smithi163.stderr: 77950K .......... .......... .......... .......... .......... 46% 21.3M 5s 2023-07-02T14:02:43.869 INFO:tasks.workunit.client.0.smithi163.stderr: 78000K .......... .......... .......... .......... .......... 46% 19.0M 5s 2023-07-02T14:02:43.871 INFO:tasks.workunit.client.0.smithi163.stderr: 78050K .......... .......... .......... .......... .......... 46% 22.0M 5s 2023-07-02T14:02:43.872 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:02:43.873 INFO:tasks.workunit.client.0.smithi163.stderr: 78100K .......... .......... .......... .......... .......... 46% 25.9M 5s 2023-07-02T14:02:43.876 INFO:tasks.workunit.client.0.smithi163.stderr: 78150K .......... .......... .......... .......... .......... 46% 16.9M 5s 2023-07-02T14:02:43.878 INFO:tasks.workunit.client.0.smithi163.stderr: 78200K .......... .......... .......... .......... .......... 46% 20.5M 5s 2023-07-02T14:02:43.881 INFO:tasks.workunit.client.0.smithi163.stderr: 78250K .......... .......... .......... .......... .......... 46% 23.9M 5s 2023-07-02T14:02:43.883 INFO:tasks.workunit.client.0.smithi163.stderr: 78300K .......... .......... .......... .......... .......... 46% 18.9M 5s 2023-07-02T14:02:43.886 INFO:tasks.workunit.client.0.smithi163.stderr: 78350K .......... .......... .......... .......... .......... 46% 14.5M 5s 2023-07-02T14:02:43.888 INFO:tasks.workunit.client.0.smithi163.stderr: 78400K .......... .......... .......... .......... .......... 46% 25.3M 5s 2023-07-02T14:02:43.891 INFO:tasks.workunit.client.0.smithi163.stderr: 78450K .......... .......... .......... .......... .......... 46% 18.1M 5s 2023-07-02T14:02:43.893 INFO:tasks.workunit.client.0.smithi163.stderr: 78500K .......... .......... .......... .......... .......... 46% 19.6M 5s 2023-07-02T14:02:43.895 INFO:tasks.workunit.client.0.smithi163.stderr: 78550K .......... .......... .......... .......... .......... 46% 23.9M 5s 2023-07-02T14:02:43.897 INFO:tasks.workunit.client.0.smithi163.stderr: 78600K .......... .......... .......... .......... .......... 46% 29.0M 5s 2023-07-02T14:02:43.901 INFO:tasks.workunit.client.0.smithi163.stderr: 78650K .......... .......... .......... .......... .......... 46% 13.1M 5s 2023-07-02T14:02:43.903 INFO:tasks.workunit.client.0.smithi163.stderr: 78700K .......... .......... .......... .......... .......... 46% 24.6M 5s 2023-07-02T14:02:43.903 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:02:43.906 INFO:tasks.workunit.client.0.smithi163.stderr: 78750K .......... .......... .......... .......... .......... 46% 19.0M 5s 2023-07-02T14:02:43.908 INFO:tasks.workunit.client.0.smithi163.stderr: 78800K .......... .......... .......... .......... .......... 46% 23.7M 5s 2023-07-02T14:02:43.909 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:43.908+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:43.911 INFO:tasks.workunit.client.0.smithi163.stderr: 78850K .......... .......... .......... .......... .......... 46% 15.3M 5s 2023-07-02T14:02:43.913 INFO:tasks.workunit.client.0.smithi163.stderr: 78900K .......... .......... .......... .......... .......... 46% 25.7M 5s 2023-07-02T14:02:43.914 INFO:tasks.workunit.client.0.smithi163.stderr: 78950K .......... .......... .......... .......... .......... 46% 33.4M 5s 2023-07-02T14:02:43.917 INFO:tasks.workunit.client.0.smithi163.stderr: 79000K .......... .......... .......... .......... .......... 46% 14.4M 5s 2023-07-02T14:02:43.920 INFO:tasks.workunit.client.0.smithi163.stderr: 79050K .......... .......... .......... .......... .......... 46% 21.8M 5s 2023-07-02T14:02:43.922 INFO:tasks.workunit.client.0.smithi163.stderr: 79100K .......... .......... .......... .......... .......... 46% 26.7M 5s 2023-07-02T14:02:43.924 INFO:tasks.workunit.client.0.smithi163.stderr: 79150K .......... .......... .......... .......... .......... 46% 17.7M 5s 2023-07-02T14:02:43.928 INFO:tasks.workunit.client.0.smithi163.stderr: 79200K .......... .......... .......... .......... .......... 47% 15.4M 5s 2023-07-02T14:02:43.929 INFO:tasks.workunit.client.0.smithi163.stderr: 79250K .......... .......... .......... .......... .......... 47% 38.9M 5s 2023-07-02T14:02:43.932 INFO:tasks.workunit.client.0.smithi163.stderr: 79300K .......... .......... .......... .......... .......... 47% 16.6M 5s 2023-07-02T14:02:43.935 INFO:tasks.workunit.client.0.smithi163.stderr: 79350K .......... .......... .......... .......... .......... 47% 19.6M 5s 2023-07-02T14:02:43.937 INFO:tasks.workunit.client.0.smithi163.stderr: 79400K .......... .......... .......... .......... .......... 47% 19.5M 5s 2023-07-02T14:02:43.938 INFO:tasks.workunit.client.0.smithi163.stderr: 79450K .......... .......... .......... .......... .......... 47% 46.9M 5s 2023-07-02T14:02:43.941 INFO:tasks.workunit.client.0.smithi163.stderr: 79500K .......... .......... .......... .......... .......... 47% 19.4M 5s 2023-07-02T14:02:43.943 INFO:tasks.workunit.client.0.smithi163.stderr: 79550K .......... .......... .......... .......... .......... 47% 17.2M 5s 2023-07-02T14:02:43.946 INFO:tasks.workunit.client.0.smithi163.stderr: 79600K .......... .......... .......... .......... .......... 47% 15.2M 5s 2023-07-02T14:02:43.948 INFO:tasks.workunit.client.0.smithi163.stderr: 79650K .......... .......... .......... .......... .......... 47% 29.7M 5s 2023-07-02T14:02:43.950 INFO:tasks.workunit.client.0.smithi163.stderr: 79700K .......... .......... .......... .......... .......... 47% 19.3M 5s 2023-07-02T14:02:43.952 INFO:tasks.workunit.client.0.smithi163.stderr: 79750K .......... .......... .......... .......... .......... 47% 25.4M 5s 2023-07-02T14:02:43.954 INFO:tasks.workunit.client.0.smithi163.stderr: 79800K .......... .......... .......... .......... .......... 47% 21.3M 5s 2023-07-02T14:02:43.958 INFO:tasks.workunit.client.0.smithi163.stderr: 79850K .......... .......... .......... .......... .......... 47% 15.7M 5s 2023-07-02T14:02:43.960 INFO:tasks.workunit.client.0.smithi163.stderr: 79900K .......... .......... .......... .......... .......... 47% 24.3M 5s 2023-07-02T14:02:43.963 INFO:tasks.workunit.client.0.smithi163.stderr: 79950K .......... .......... .......... .......... .......... 47% 16.2M 5s 2023-07-02T14:02:43.964 INFO:tasks.workunit.client.0.smithi163.stderr: 80000K .......... .......... .......... .......... .......... 47% 33.2M 5s 2023-07-02T14:02:43.968 INFO:tasks.workunit.client.0.smithi163.stderr: 80050K .......... .......... .......... .......... .......... 47% 14.7M 5s 2023-07-02T14:02:43.970 INFO:tasks.workunit.client.0.smithi163.stderr: 80100K .......... .......... .......... .......... .......... 47% 23.7M 5s 2023-07-02T14:02:43.972 INFO:tasks.workunit.client.0.smithi163.stderr: 80150K .......... .......... .......... .......... .......... 47% 18.1M 5s 2023-07-02T14:02:43.976 INFO:tasks.workunit.client.0.smithi163.stderr: 80200K .......... .......... .......... .......... .......... 47% 15.7M 4s 2023-07-02T14:02:43.977 INFO:tasks.workunit.client.0.smithi163.stderr: 80250K .......... .......... .......... .......... .......... 47% 38.9M 4s 2023-07-02T14:02:43.979 INFO:tasks.workunit.client.0.smithi163.stderr: 80300K .......... .......... .......... .......... .......... 47% 19.8M 4s 2023-07-02T14:02:43.982 INFO:tasks.workunit.client.0.smithi163.stderr: 80350K .......... .......... .......... .......... .......... 47% 17.4M 4s 2023-07-02T14:02:43.984 INFO:tasks.workunit.client.0.smithi163.stderr: 80400K .......... .......... .......... .......... .......... 47% 20.7M 4s 2023-07-02T14:02:43.987 INFO:tasks.workunit.client.0.smithi163.stderr: 80450K .......... .......... .......... .......... .......... 47% 16.1M 4s 2023-07-02T14:02:43.989 INFO:tasks.workunit.client.0.smithi163.stderr: 80500K .......... .......... .......... .......... .......... 47% 26.5M 4s 2023-07-02T14:02:43.992 INFO:tasks.workunit.client.0.smithi163.stderr: 80550K .......... .......... .......... .......... .......... 47% 17.8M 4s 2023-07-02T14:02:43.995 INFO:tasks.workunit.client.0.smithi163.stderr: 80600K .......... .......... .......... .......... .......... 47% 22.5M 4s 2023-07-02T14:02:43.996 INFO:tasks.workunit.client.0.smithi163.stderr: 80650K .......... .......... .......... .......... .......... 47% 20.9M 4s 2023-07-02T14:02:43.999 INFO:tasks.workunit.client.0.smithi163.stderr: 80700K .......... .......... .......... .......... .......... 47% 16.3M 4s 2023-07-02T14:02:44.002 INFO:tasks.workunit.client.0.smithi163.stderr: 80750K .......... .......... .......... .......... .......... 47% 18.3M 4s 2023-07-02T14:02:44.004 INFO:tasks.workunit.client.0.smithi163.stderr: 80800K .......... .......... .......... .......... .......... 47% 26.3M 4s 2023-07-02T14:02:44.007 INFO:tasks.workunit.client.0.smithi163.stderr: 80850K .......... .......... .......... .......... .......... 47% 19.2M 4s 2023-07-02T14:02:44.009 INFO:tasks.workunit.client.0.smithi163.stderr: 80900K .......... .......... .......... .......... .......... 48% 26.1M 4s 2023-07-02T14:02:44.010 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:44.008+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:44.011 INFO:tasks.workunit.client.0.smithi163.stderr: 80950K .......... .......... .......... .......... .......... 48% 17.4M 4s 2023-07-02T14:02:44.012 INFO:tasks.workunit.client.0.smithi163.stderr: 81000K .......... .......... .......... .......... .......... 48% 37.1M 4s 2023-07-02T14:02:44.015 INFO:tasks.workunit.client.0.smithi163.stderr: 81050K .......... .......... .......... .......... .......... 48% 19.1M 4s 2023-07-02T14:02:44.018 INFO:tasks.workunit.client.0.smithi163.stderr: 81100K .......... .......... .......... .......... .......... 48% 15.4M 4s 2023-07-02T14:02:44.021 INFO:tasks.workunit.client.0.smithi163.stderr: 81150K .......... .......... .......... .......... .......... 48% 18.8M 4s 2023-07-02T14:02:44.023 INFO:tasks.workunit.client.0.smithi163.stderr: 81200K .......... .......... .......... .......... .......... 48% 23.3M 4s 2023-07-02T14:02:44.026 INFO:tasks.workunit.client.0.smithi163.stderr: 81250K .......... .......... .......... .......... .......... 48% 16.1M 4s 2023-07-02T14:02:44.027 INFO:tasks.workunit.client.0.smithi163.stderr: 81300K .......... .......... .......... .......... .......... 48% 40.6M 4s 2023-07-02T14:02:44.029 INFO:tasks.workunit.client.0.smithi163.stderr: 81350K .......... .......... .......... .......... .......... 48% 21.0M 4s 2023-07-02T14:02:44.032 INFO:tasks.workunit.client.0.smithi163.stderr: 81400K .......... .......... .......... .......... .......... 48% 21.4M 4s 2023-07-02T14:02:44.035 INFO:tasks.workunit.client.0.smithi163.stderr: 81450K .......... .......... .......... .......... .......... 48% 17.1M 4s 2023-07-02T14:02:44.036 INFO:tasks.workunit.client.0.smithi163.stderr: 81500K .......... .......... .......... .......... .......... 48% 30.1M 4s 2023-07-02T14:02:44.039 INFO:tasks.workunit.client.0.smithi163.stderr: 81550K .......... .......... .......... .......... .......... 48% 20.9M 4s 2023-07-02T14:02:44.042 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:44.042 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:44.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:44.042 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:02:44.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.042 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:163 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:44.043 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.284743+0000", 2023-07-02T14:02:44.043 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7550888159999998, 2023-07-02T14:02:44.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027177222000000001, 2023-07-02T14:02:44.043 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:44.043 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:44.043 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:44.043 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:44.044 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:44.044 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 163 2023-07-02T14:02:44.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.044 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:44.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:44.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284743+0000", 2023-07-02T14:02:44.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:44.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:44.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284743+0000", 2023-07-02T14:02:44.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:44.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:44.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284742+0000", 2023-07-02T14:02:44.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9390999999999998e-05 2023-07-02T14:02:44.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:44.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284831+0000", 2023-07-02T14:02:44.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6971000000000001e-05 2023-07-02T14:02:44.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:44.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284848+0000", 2023-07-02T14:02:44.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951e-05 2023-07-02T14:02:44.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:44.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284864+0000", 2023-07-02T14:02:44.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015684601999999999 2023-07-02T14:02:44.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:44.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300549+0000", 2023-07-02T14:02:44.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3045000000000003e-05 2023-07-02T14:02:44.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:44.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300622+0000", 2023-07-02T14:02:44.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9452999999999999e-05 2023-07-02T14:02:44.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:44.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300711+0000", 2023-07-02T14:02:44.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059060379999999997 2023-07-02T14:02:44.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:44.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306617+0000", 2023-07-02T14:02:44.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052100280000000002 2023-07-02T14:02:44.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:44.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311827+0000", 2023-07-02T14:02:44.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978999999999998e-05 2023-07-02T14:02:44.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:44.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311879+0000", 2023-07-02T14:02:44.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0349999999999998e-05 2023-07-02T14:02:44.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:44.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311920+0000", 2023-07-02T14:02:44.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027177222000000001 2023-07-02T14:02:44.052 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.053 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.053 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.053 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:165 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:44.053 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.284977+0000", 2023-07-02T14:02:44.053 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7548541929999999, 2023-07-02T14:02:44.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027689147000000001, 2023-07-02T14:02:44.054 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:44.054 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:44.054 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:44.054 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:44.054 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:44.054 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 165 2023-07-02T14:02:44.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.055 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:44.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:44.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284977+0000", 2023-07-02T14:02:44.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:44.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:44.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284977+0000", 2023-07-02T14:02:44.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:44.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:44.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284976+0000", 2023-07-02T14:02:44.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3962999999999996e-05 2023-07-02T14:02:44.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:44.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285060+0000", 2023-07-02T14:02:44.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1260000000000004e-06 2023-07-02T14:02:44.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:44.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285070+0000", 2023-07-02T14:02:44.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0669999999999999e-06 2023-07-02T14:02:44.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:44.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285075+0000", 2023-07-02T14:02:44.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016699978000000001 2023-07-02T14:02:44.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:44.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.301775+0000", 2023-07-02T14:02:44.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6872000000000004e-05 2023-07-02T14:02:44.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:44.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.301862+0000", 2023-07-02T14:02:44.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000155592 2023-07-02T14:02:44.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:44.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.302017+0000", 2023-07-02T14:02:44.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046740250000000001 2023-07-02T14:02:44.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306691+0000", 2023-07-02T14:02:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059023670000000004 2023-07-02T14:02:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:44.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312593+0000", 2023-07-02T14:02:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6545999999999999e-05 2023-07-02T14:02:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312630+0000", 2023-07-02T14:02:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6270999999999999e-05 2023-07-02T14:02:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:44.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312666+0000", 2023-07-02T14:02:44.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027689147000000001 2023-07-02T14:02:44.063 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.063 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.063 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.064 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:168 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1507328~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:44.064 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285081+0000", 2023-07-02T14:02:44.064 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7547501820000004, 2023-07-02T14:02:44.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027656613, 2023-07-02T14:02:44.064 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:44.064 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:44.064 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:44.065 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:44.065 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:44.065 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 168 2023-07-02T14:02:44.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.065 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:44.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:44.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285081+0000", 2023-07-02T14:02:44.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:44.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:44.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285081+0000", 2023-07-02T14:02:44.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:44.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:44.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285080+0000", 2023-07-02T14:02:44.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5372999999999998e-05 2023-07-02T14:02:44.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:44.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285156+0000", 2023-07-02T14:02:44.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0099999999999995e-06 2023-07-02T14:02:44.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:44.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285164+0000", 2023-07-02T14:02:44.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3320000000000002e-06 2023-07-02T14:02:44.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:44.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285168+0000", 2023-07-02T14:02:44.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017721316000000001 2023-07-02T14:02:44.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:44.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.302889+0000", 2023-07-02T14:02:44.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9568999999999998e-05 2023-07-02T14:02:44.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:44.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.302949+0000", 2023-07-02T14:02:44.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121902 2023-07-02T14:02:44.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:44.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303071+0000", 2023-07-02T14:02:44.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0071548599999999999 2023-07-02T14:02:44.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:44.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310226+0000", 2023-07-02T14:02:44.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024455520000000001 2023-07-02T14:02:44.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:44.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312671+0000", 2023-07-02T14:02:44.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6958000000000001e-05 2023-07-02T14:02:44.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:44.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312708+0000", 2023-07-02T14:02:44.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9451e-05 2023-07-02T14:02:44.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:44.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312738+0000", 2023-07-02T14:02:44.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027656613 2023-07-02T14:02:44.074 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.074 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.074 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.074 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:171 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1572864~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:44.075 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285174+0000", 2023-07-02T14:02:44.075 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7546576649999999, 2023-07-02T14:02:44.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027618999000000002, 2023-07-02T14:02:44.075 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:44.075 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:44.075 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:44.075 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:44.075 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:44.076 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 171 2023-07-02T14:02:44.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.076 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:44.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:44.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285174+0000", 2023-07-02T14:02:44.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:44.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:44.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285174+0000", 2023-07-02T14:02:44.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:44.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:44.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285173+0000", 2023-07-02T14:02:44.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1789000000000002e-05 2023-07-02T14:02:44.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:44.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285225+0000", 2023-07-02T14:02:44.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6429999999999995e-06 2023-07-02T14:02:44.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:44.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285232+0000", 2023-07-02T14:02:44.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8380000000000001e-06 2023-07-02T14:02:44.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:44.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285237+0000", 2023-07-02T14:02:44.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018182713 2023-07-02T14:02:44.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:44.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303420+0000", 2023-07-02T14:02:44.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0059000000000001e-05 2023-07-02T14:02:44.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:44.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303490+0000", 2023-07-02T14:02:44.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012003 2023-07-02T14:02:44.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:44.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303610+0000", 2023-07-02T14:02:44.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066642150000000002 2023-07-02T14:02:44.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:44.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310274+0000", 2023-07-02T14:02:44.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024681949999999999 2023-07-02T14:02:44.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:44.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312743+0000", 2023-07-02T14:02:44.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5525999999999999e-05 2023-07-02T14:02:44.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:44.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312768+0000", 2023-07-02T14:02:44.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4613999999999998e-05 2023-07-02T14:02:44.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:44.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312793+0000", 2023-07-02T14:02:44.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027618999000000002 2023-07-02T14:02:44.084 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.084 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.085 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.085 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:174 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:44.085 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285243+0000", 2023-07-02T14:02:44.085 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7545884679999997, 2023-07-02T14:02:44.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027607988, 2023-07-02T14:02:44.086 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:44.086 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:44.086 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:44.086 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:44.086 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:44.086 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 174 2023-07-02T14:02:44.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.086 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:44.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:44.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285243+0000", 2023-07-02T14:02:44.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:44.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:44.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285243+0000", 2023-07-02T14:02:44.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:44.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:44.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285242+0000", 2023-07-02T14:02:44.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6173000000000003e-05 2023-07-02T14:02:44.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:44.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285328+0000", 2023-07-02T14:02:44.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5729999999999997e-06 2023-07-02T14:02:44.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:44.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285338+0000", 2023-07-02T14:02:44.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.081e-06 2023-07-02T14:02:44.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:44.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285347+0000", 2023-07-02T14:02:44.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018567367000000001 2023-07-02T14:02:44.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:44.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303914+0000", 2023-07-02T14:02:44.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9522000000000003e-05 2023-07-02T14:02:44.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:44.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303984+0000", 2023-07-02T14:02:44.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013801800000000001 2023-07-02T14:02:44.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:44.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.304122+0000", 2023-07-02T14:02:44.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061898489999999999 2023-07-02T14:02:44.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:44.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310312+0000", 2023-07-02T14:02:44.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024882519999999998 2023-07-02T14:02:44.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:44.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312800+0000", 2023-07-02T14:02:44.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7100000000000001e-05 2023-07-02T14:02:44.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:44.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312827+0000", 2023-07-02T14:02:44.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3771999999999999e-05 2023-07-02T14:02:44.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:44.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312851+0000", 2023-07-02T14:02:44.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027607988 2023-07-02T14:02:44.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.095 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.096 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:177 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:44.096 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285353+0000", 2023-07-02T14:02:44.096 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7544782850000002, 2023-07-02T14:02:44.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027577522, 2023-07-02T14:02:44.096 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:44.097 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:44.097 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:44.097 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:44.097 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:44.097 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 177 2023-07-02T14:02:44.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.097 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:44.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:44.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285353+0000", 2023-07-02T14:02:44.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:44.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:44.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285353+0000", 2023-07-02T14:02:44.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:44.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:44.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285352+0000", 2023-07-02T14:02:44.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7850999999999996e-05 2023-07-02T14:02:44.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:44.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285440+0000", 2023-07-02T14:02:44.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7960000000000001e-06 2023-07-02T14:02:44.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:44.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285449+0000", 2023-07-02T14:02:44.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0529999999999997e-06 2023-07-02T14:02:44.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:44.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285454+0000", 2023-07-02T14:02:44.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018975296999999999 2023-07-02T14:02:44.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:44.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.304429+0000", 2023-07-02T14:02:44.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0682e-05 2023-07-02T14:02:44.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:44.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.304480+0000", 2023-07-02T14:02:44.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116107 2023-07-02T14:02:44.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:44.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.304596+0000", 2023-07-02T14:02:44.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0057486309999999997 2023-07-02T14:02:44.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:44.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310345+0000", 2023-07-02T14:02:44.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025147289999999998 2023-07-02T14:02:44.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:44.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312860+0000", 2023-07-02T14:02:44.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7252999999999997e-05 2023-07-02T14:02:44.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:44.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312897+0000", 2023-07-02T14:02:44.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3705999999999999e-05 2023-07-02T14:02:44.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:44.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312931+0000", 2023-07-02T14:02:44.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027577522 2023-07-02T14:02:44.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.106 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.106 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:180 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:44.107 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285460+0000", 2023-07-02T14:02:44.107 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.754371409, 2023-07-02T14:02:44.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027547589000000001, 2023-07-02T14:02:44.107 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:44.107 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:44.107 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:44.107 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:44.108 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:44.108 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 180 2023-07-02T14:02:44.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.108 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:44.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:44.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285460+0000", 2023-07-02T14:02:44.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:44.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:44.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285460+0000", 2023-07-02T14:02:44.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:44.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:44.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285459+0000", 2023-07-02T14:02:44.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7868999999999997e-05 2023-07-02T14:02:44.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:44.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285527+0000", 2023-07-02T14:02:44.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6710000000000004e-06 2023-07-02T14:02:44.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:44.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285533+0000", 2023-07-02T14:02:44.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9960000000000002e-06 2023-07-02T14:02:44.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:44.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285536+0000", 2023-07-02T14:02:44.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019509826000000001 2023-07-02T14:02:44.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:44.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.305046+0000", 2023-07-02T14:02:44.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3639000000000004e-05 2023-07-02T14:02:44.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:44.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.305119+0000", 2023-07-02T14:02:44.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014125300000000001 2023-07-02T14:02:44.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:44.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.305260+0000", 2023-07-02T14:02:44.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051273919999999997 2023-07-02T14:02:44.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:44.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310388+0000", 2023-07-02T14:02:44.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025505390000000001 2023-07-02T14:02:44.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:44.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312938+0000", 2023-07-02T14:02:44.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5608999999999997e-05 2023-07-02T14:02:44.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:44.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312974+0000", 2023-07-02T14:02:44.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3562999999999997e-05 2023-07-02T14:02:44.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:44.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313008+0000", 2023-07-02T14:02:44.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027547589000000001 2023-07-02T14:02:44.116 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.116 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.117 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.117 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:183 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:44.117 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:02:44.117 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7542921140000001, 2023-07-02T14:02:44.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003, 2023-07-02T14:02:44.118 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:44.118 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:44.118 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:44.118 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:44.118 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:44.118 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 183 2023-07-02T14:02:44.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.118 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:44.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:44.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:02:44.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:44.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:44.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:02:44.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:44.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:44.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:02:44.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9790999999999997e-05 2023-07-02T14:02:44.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:44.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285598+0000", 2023-07-02T14:02:44.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5809999999999996e-06 2023-07-02T14:02:44.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:44.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285604+0000", 2023-07-02T14:02:44.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7599999999999998e-06 2023-07-02T14:02:44.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:44.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285607+0000", 2023-07-02T14:02:44.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021108188999999999 2023-07-02T14:02:44.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:44.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306715+0000", 2023-07-02T14:02:44.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6548e-05 2023-07-02T14:02:44.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:44.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306792+0000", 2023-07-02T14:02:44.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173869 2023-07-02T14:02:44.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:44.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306965+0000", 2023-07-02T14:02:44.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060498009999999996 2023-07-02T14:02:44.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:44.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313015+0000", 2023-07-02T14:02:44.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049443719999999998 2023-07-02T14:02:44.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:44.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317960+0000", 2023-07-02T14:02:44.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5927999999999998e-05 2023-07-02T14:02:44.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:44.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318006+0000", 2023-07-02T14:02:44.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6004999999999998e-05 2023-07-02T14:02:44.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:44.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318042+0000", 2023-07-02T14:02:44.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003 2023-07-02T14:02:44.127 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.127 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.127 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.128 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:186 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:44.128 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:02:44.128 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7542212880000001, 2023-07-02T14:02:44.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003, 2023-07-02T14:02:44.128 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:44.128 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:44.128 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:44.128 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:44.129 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:44.129 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 186 2023-07-02T14:02:44.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.129 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:44.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:44.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:02:44.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:44.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:44.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:02:44.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:44.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:44.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:02:44.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4279e-05 2023-07-02T14:02:44.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:44.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285664+0000", 2023-07-02T14:02:44.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3789999999999995e-06 2023-07-02T14:02:44.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:44.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285673+0000", 2023-07-02T14:02:44.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499999999999998e-06 2023-07-02T14:02:44.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:44.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285678+0000", 2023-07-02T14:02:44.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023261947000000002 2023-07-02T14:02:44.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:44.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.308940+0000", 2023-07-02T14:02:44.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010323399999999999 2023-07-02T14:02:44.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:44.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309044+0000", 2023-07-02T14:02:44.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018878300000000001 2023-07-02T14:02:44.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:44.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309232+0000", 2023-07-02T14:02:44.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037874150000000001 2023-07-02T14:02:44.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:44.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313020+0000", 2023-07-02T14:02:44.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050976140000000003 2023-07-02T14:02:44.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:44.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318117+0000", 2023-07-02T14:02:44.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3182000000000001e-05 2023-07-02T14:02:44.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:44.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:02:44.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5055999999999998e-05 2023-07-02T14:02:44.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:44.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318196+0000", 2023-07-02T14:02:44.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003 2023-07-02T14:02:44.137 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.137 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.138 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.138 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:189 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:44.138 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:02:44.138 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7541475249999996, 2023-07-02T14:02:44.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997, 2023-07-02T14:02:44.138 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:44.139 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:44.139 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:44.139 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:44.139 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:44.139 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:02:44.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.139 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:44.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:44.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:02:44.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:44.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:44.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:02:44.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:44.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:44.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285683+0000", 2023-07-02T14:02:44.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8451999999999999e-05 2023-07-02T14:02:44.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:44.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285782+0000", 2023-07-02T14:02:44.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3650000000000004e-06 2023-07-02T14:02:44.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:44.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285790+0000", 2023-07-02T14:02:44.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8230000000000002e-06 2023-07-02T14:02:44.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:44.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285795+0000", 2023-07-02T14:02:44.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024619367999999999 2023-07-02T14:02:44.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:44.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310414+0000", 2023-07-02T14:02:44.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014244600000000001 2023-07-02T14:02:44.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:44.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310557+0000", 2023-07-02T14:02:44.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020612699999999999 2023-07-02T14:02:44.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:44.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310763+0000", 2023-07-02T14:02:44.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003427687 2023-07-02T14:02:44.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:44.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314191+0000", 2023-07-02T14:02:44.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040492089999999998 2023-07-02T14:02:44.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:44.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318240+0000", 2023-07-02T14:02:44.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0287999999999999e-05 2023-07-02T14:02:44.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:44.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318270+0000", 2023-07-02T14:02:44.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0868000000000002e-05 2023-07-02T14:02:44.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:44.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318311+0000", 2023-07-02T14:02:44.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997 2023-07-02T14:02:44.148 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.148 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.148 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.149 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:192 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:44.149 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:02:44.149 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7540307520000002, 2023-07-02T14:02:44.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003, 2023-07-02T14:02:44.149 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:44.149 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:44.149 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:44.149 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:44.150 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:44.150 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 192 2023-07-02T14:02:44.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.150 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:44.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:44.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:02:44.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:44.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:44.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:02:44.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:44.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:44.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285800+0000", 2023-07-02T14:02:44.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100417 2023-07-02T14:02:44.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:44.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285900+0000", 2023-07-02T14:02:44.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8244999999999999e-05 2023-07-02T14:02:44.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:44.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285919+0000", 2023-07-02T14:02:44.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2499999999999997e-06 2023-07-02T14:02:44.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:44.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285924+0000", 2023-07-02T14:02:44.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025288155999999999 2023-07-02T14:02:44.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:44.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311212+0000", 2023-07-02T14:02:44.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4073999999999996e-05 2023-07-02T14:02:44.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311286+0000", 2023-07-02T14:02:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016219699999999999 2023-07-02T14:02:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:44.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311448+0000", 2023-07-02T14:02:44.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066020599999999999 2023-07-02T14:02:44.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:44.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318050+0000", 2023-07-02T14:02:44.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000327986 2023-07-02T14:02:44.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:44.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318378+0000", 2023-07-02T14:02:44.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8693999999999998e-05 2023-07-02T14:02:44.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:44.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318427+0000", 2023-07-02T14:02:44.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6656999999999998e-05 2023-07-02T14:02:44.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:44.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318464+0000", 2023-07-02T14:02:44.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003 2023-07-02T14:02:44.158 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.158 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.159 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.159 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:195 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:44.159 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:02:44.159 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7538992479999997, 2023-07-02T14:02:44.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001, 2023-07-02T14:02:44.160 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:44.160 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:44.160 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:44.160 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:44.160 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:44.160 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 195 2023-07-02T14:02:44.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.160 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:44.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:44.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:02:44.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:44.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:44.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:02:44.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:44.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:44.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285931+0000", 2023-07-02T14:02:44.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4530999999999995e-05 2023-07-02T14:02:44.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:44.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286006+0000", 2023-07-02T14:02:44.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4113e-05 2023-07-02T14:02:44.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:44.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286020+0000", 2023-07-02T14:02:44.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7459999999999997e-06 2023-07-02T14:02:44.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:44.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286026+0000", 2023-07-02T14:02:44.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025997769 2023-07-02T14:02:44.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:44.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312023+0000", 2023-07-02T14:02:44.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4403000000000001e-05 2023-07-02T14:02:44.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:44.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312078+0000", 2023-07-02T14:02:44.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020581399999999999 2023-07-02T14:02:44.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:44.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312284+0000", 2023-07-02T14:02:44.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005771043 2023-07-02T14:02:44.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:44.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318055+0000", 2023-07-02T14:02:44.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00046015200000000002 2023-07-02T14:02:44.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:44.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318515+0000", 2023-07-02T14:02:44.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9444000000000003e-05 2023-07-02T14:02:44.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:44.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318564+0000", 2023-07-02T14:02:44.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2141000000000002e-05 2023-07-02T14:02:44.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:44.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318606+0000", 2023-07-02T14:02:44.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001 2023-07-02T14:02:44.169 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.169 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.169 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.170 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:198 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:44.170 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:02:44.170 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7537992889999998, 2023-07-02T14:02:44.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999, 2023-07-02T14:02:44.170 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:44.170 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:44.170 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:44.170 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:44.171 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:44.171 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 198 2023-07-02T14:02:44.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.171 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:44.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:44.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:02:44.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:44.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:44.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:02:44.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:44.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:44.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286031+0000", 2023-07-02T14:02:44.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00049286700000000002 2023-07-02T14:02:44.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:44.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286524+0000", 2023-07-02T14:02:44.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4938e-05 2023-07-02T14:02:44.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:44.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286539+0000", 2023-07-02T14:02:44.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0680000000000002e-06 2023-07-02T14:02:44.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:44.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286545+0000", 2023-07-02T14:02:44.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026497718 2023-07-02T14:02:44.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:44.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313043+0000", 2023-07-02T14:02:44.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0884999999999998e-05 2023-07-02T14:02:44.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:44.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313094+0000", 2023-07-02T14:02:44.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014109999999999999 2023-07-02T14:02:44.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:44.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313235+0000", 2023-07-02T14:02:44.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048258980000000003 2023-07-02T14:02:44.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:44.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318061+0000", 2023-07-02T14:02:44.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014346350000000001 2023-07-02T14:02:44.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:44.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319495+0000", 2023-07-02T14:02:44.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1986e-05 2023-07-02T14:02:44.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:44.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319547+0000", 2023-07-02T14:02:44.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0457e-05 2023-07-02T14:02:44.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:44.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319578+0000", 2023-07-02T14:02:44.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999 2023-07-02T14:02:44.179 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.180 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.180 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.180 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:201 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:44.180 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:02:44.180 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7532788640000003, 2023-07-02T14:02:44.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003, 2023-07-02T14:02:44.181 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:44.181 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:44.181 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:44.181 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:44.181 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:44.181 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 201 2023-07-02T14:02:44.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.182 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:44.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:44.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:02:44.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:44.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:44.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:02:44.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:44.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:44.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:02:44.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019213999999999999 2023-07-02T14:02:44.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:44.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286744+0000", 2023-07-02T14:02:44.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5279999999999992e-06 2023-07-02T14:02:44.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:44.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286753+0000", 2023-07-02T14:02:44.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4200000000000004e-06 2023-07-02T14:02:44.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:44.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286760+0000", 2023-07-02T14:02:44.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026818386999999999 2023-07-02T14:02:44.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:44.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313578+0000", 2023-07-02T14:02:44.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015965 2023-07-02T14:02:44.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:44.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313738+0000", 2023-07-02T14:02:44.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015767299999999999 2023-07-02T14:02:44.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:44.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313895+0000", 2023-07-02T14:02:44.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041714930000000001 2023-07-02T14:02:44.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:44.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318067+0000", 2023-07-02T14:02:44.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001561698 2023-07-02T14:02:44.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:44.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319629+0000", 2023-07-02T14:02:44.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2346000000000003e-05 2023-07-02T14:02:44.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:44.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319681+0000", 2023-07-02T14:02:44.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4691000000000003e-05 2023-07-02T14:02:44.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:44.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319726+0000", 2023-07-02T14:02:44.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003 2023-07-02T14:02:44.190 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.190 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.190 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.191 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:204 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:44.191 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:02:44.191 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7530640259999997, 2023-07-02T14:02:44.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998, 2023-07-02T14:02:44.191 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:44.191 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:44.192 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:44.192 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:44.192 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:44.192 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 204 2023-07-02T14:02:44.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.192 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:44.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:44.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:02:44.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:44.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:44.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:02:44.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:44.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:44.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:02:44.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2497999999999998e-05 2023-07-02T14:02:44.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:44.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286819+0000", 2023-07-02T14:02:44.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0099999999999995e-06 2023-07-02T14:02:44.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:44.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286827+0000", 2023-07-02T14:02:44.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000003e-06 2023-07-02T14:02:44.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:44.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286833+0000", 2023-07-02T14:02:44.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027384716999999999 2023-07-02T14:02:44.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:44.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314218+0000", 2023-07-02T14:02:44.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6204000000000004e-05 2023-07-02T14:02:44.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:44.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314294+0000", 2023-07-02T14:02:44.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015757499999999999 2023-07-02T14:02:44.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:44.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314452+0000", 2023-07-02T14:02:44.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036215560000000002 2023-07-02T14:02:44.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:44.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318073+0000", 2023-07-02T14:02:44.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017023979999999999 2023-07-02T14:02:44.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:44.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319776+0000", 2023-07-02T14:02:44.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9561000000000002e-05 2023-07-02T14:02:44.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:44.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319825+0000", 2023-07-02T14:02:44.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0329e-05 2023-07-02T14:02:44.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:44.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319866+0000", 2023-07-02T14:02:44.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998 2023-07-02T14:02:44.200 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.201 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.201 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.201 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:207 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:44.201 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:02:44.201 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7529893809999999, 2023-07-02T14:02:44.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003, 2023-07-02T14:02:44.202 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:44.202 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:44.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:44.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:44.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:44.202 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 207 2023-07-02T14:02:44.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.203 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:44.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:44.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:02:44.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:44.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:44.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:02:44.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:44.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:44.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286841+0000", 2023-07-02T14:02:44.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00068375100000000002 2023-07-02T14:02:44.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:44.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287525+0000", 2023-07-02T14:02:44.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1601e-05 2023-07-02T14:02:44.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:44.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287537+0000", 2023-07-02T14:02:44.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6409999999999996e-06 2023-07-02T14:02:44.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:44.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287543+0000", 2023-07-02T14:02:44.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027245844000000002 2023-07-02T14:02:44.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:44.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314789+0000", 2023-07-02T14:02:44.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8910000000000003e-05 2023-07-02T14:02:44.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:44.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314858+0000", 2023-07-02T14:02:44.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030992599999999997 2023-07-02T14:02:44.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:44.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315168+0000", 2023-07-02T14:02:44.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029122869999999999 2023-07-02T14:02:44.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:44.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318080+0000", 2023-07-02T14:02:44.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019963020000000001 2023-07-02T14:02:44.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:44.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320077+0000", 2023-07-02T14:02:44.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5116000000000005e-05 2023-07-02T14:02:44.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:44.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320142+0000", 2023-07-02T14:02:44.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3793000000000004e-05 2023-07-02T14:02:44.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:44.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320205+0000", 2023-07-02T14:02:44.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003 2023-07-02T14:02:44.211 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.211 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.211 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.212 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:236 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2424832~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:44.212 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:02:44.212 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7522771979999998, 2023-07-02T14:02:44.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003, 2023-07-02T14:02:44.212 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:44.212 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:44.212 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:44.213 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:44.213 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:44.213 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2023-07-02T14:02:44.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.213 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:44.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:44.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:02:44.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:44.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:44.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:02:44.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:44.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:44.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287553+0000", 2023-07-02T14:02:44.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0667e-05 2023-07-02T14:02:44.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:44.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287614+0000", 2023-07-02T14:02:44.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0454e-05 2023-07-02T14:02:44.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:44.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287624+0000", 2023-07-02T14:02:44.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8564999999999999e-05 2023-07-02T14:02:44.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:44.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287643+0000", 2023-07-02T14:02:44.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027977596 2023-07-02T14:02:44.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:44.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315621+0000", 2023-07-02T14:02:44.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000201728 2023-07-02T14:02:44.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:44.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315822+0000", 2023-07-02T14:02:44.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018436599999999999 2023-07-02T14:02:44.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:44.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316007+0000", 2023-07-02T14:02:44.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042423440000000003 2023-07-02T14:02:44.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:44.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320249+0000", 2023-07-02T14:02:44.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001452633 2023-07-02T14:02:44.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:44.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321702+0000", 2023-07-02T14:02:44.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3616000000000003e-05 2023-07-02T14:02:44.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:44.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321755+0000", 2023-07-02T14:02:44.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840999999999999e-05 2023-07-02T14:02:44.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:44.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321793+0000", 2023-07-02T14:02:44.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003 2023-07-02T14:02:44.221 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.222 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.222 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.222 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:251 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:44.222 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:02:44.222 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7521816609999998, 2023-07-02T14:02:44.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001, 2023-07-02T14:02:44.223 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:44.223 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:44.223 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:44.223 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:44.223 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:44.223 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 251 2023-07-02T14:02:44.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.224 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:44.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:44.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:02:44.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:44.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:44.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:02:44.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:44.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:44.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287649+0000", 2023-07-02T14:02:44.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3232000000000001e-05 2023-07-02T14:02:44.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:44.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287702+0000", 2023-07-02T14:02:44.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0010000000000007e-06 2023-07-02T14:02:44.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:44.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287711+0000", 2023-07-02T14:02:44.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3569999999999997e-06 2023-07-02T14:02:44.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:44.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287717+0000", 2023-07-02T14:02:44.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028686454 2023-07-02T14:02:44.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:44.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316403+0000", 2023-07-02T14:02:44.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7741000000000002e-05 2023-07-02T14:02:44.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:44.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316461+0000", 2023-07-02T14:02:44.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4313000000000004e-05 2023-07-02T14:02:44.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:44.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316545+0000", 2023-07-02T14:02:44.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003734032 2023-07-02T14:02:44.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:44.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320279+0000", 2023-07-02T14:02:44.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015237219999999999 2023-07-02T14:02:44.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:44.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321803+0000", 2023-07-02T14:02:44.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1927e-05 2023-07-02T14:02:44.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:44.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321845+0000", 2023-07-02T14:02:44.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5398999999999998e-05 2023-07-02T14:02:44.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:44.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321880+0000", 2023-07-02T14:02:44.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001 2023-07-02T14:02:44.232 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.232 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.232 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.233 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:255 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2555904~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:44.233 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:02:44.233 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.7517669329999999, 2023-07-02T14:02:44.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002, 2023-07-02T14:02:44.233 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:44.233 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:44.233 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:44.233 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:44.234 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:44.234 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 255 2023-07-02T14:02:44.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.234 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:44.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:44.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:02:44.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:44.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:44.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:02:44.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:44.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:44.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288063+0000", 2023-07-02T14:02:44.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4459999999999997e-05 2023-07-02T14:02:44.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:44.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288118+0000", 2023-07-02T14:02:44.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0998999999999999e-05 2023-07-02T14:02:44.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:44.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288129+0000", 2023-07-02T14:02:44.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3219999999999998e-06 2023-07-02T14:02:44.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:44.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288135+0000", 2023-07-02T14:02:44.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029137386000000001 2023-07-02T14:02:44.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:44.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317273+0000", 2023-07-02T14:02:44.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7819999999999997e-05 2023-07-02T14:02:44.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:44.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317350+0000", 2023-07-02T14:02:44.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000174864 2023-07-02T14:02:44.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:44.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317525+0000", 2023-07-02T14:02:44.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027943519999999999 2023-07-02T14:02:44.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:44.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320320+0000", 2023-07-02T14:02:44.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015681460000000001 2023-07-02T14:02:44.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:44.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321888+0000", 2023-07-02T14:02:44.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9508999999999997e-05 2023-07-02T14:02:44.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:44.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321927+0000", 2023-07-02T14:02:44.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5428e-05 2023-07-02T14:02:44.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:44.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321963+0000", 2023-07-02T14:02:44.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002 2023-07-02T14:02:44.242 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.242 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.243 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.243 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:446 2.1 2:910b2a90:::rbd_data.125de1449605.000000000000010a:head [write 65536~3211264] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:44.243 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:02:44.243 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.6744871809999999, 2023-07-02T14:02:44.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003, 2023-07-02T14:02:44.243 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:44.244 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:44.244 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:44.244 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:44.244 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:44.244 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 446 2023-07-02T14:02:44.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.244 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:44.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:44.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:02:44.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:44.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:44.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:02:44.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:44.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:44.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:02:44.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015907372999999999 2023-07-02T14:02:44.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:44.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381251+0000", 2023-07-02T14:02:44.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5527000000000001e-05 2023-07-02T14:02:44.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:44.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381277+0000", 2023-07-02T14:02:44.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9958000000000001e-05 2023-07-02T14:02:44.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:44.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381297+0000", 2023-07-02T14:02:44.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4941000000000001e-05 2023-07-02T14:02:44.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381352+0000", 2023-07-02T14:02:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011565699999999999 2023-07-02T14:02:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381467+0000", 2023-07-02T14:02:44.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018039799999999999 2023-07-02T14:02:44.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:44.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381648+0000", 2023-07-02T14:02:44.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088183559999999994 2023-07-02T14:02:44.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:44.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.390466+0000", 2023-07-02T14:02:44.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007278278 2023-07-02T14:02:44.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:44.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397744+0000", 2023-07-02T14:02:44.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9289000000000001e-05 2023-07-02T14:02:44.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:44.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397804+0000", 2023-07-02T14:02:44.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0479000000000002e-05 2023-07-02T14:02:44.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:44.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397854+0000", 2023-07-02T14:02:44.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003 2023-07-02T14:02:44.253 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.253 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.253 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.253 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.253 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.253 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:44.254 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2023-07-02T14:02:44.254 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:02:44.254 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2023-07-02T14:02:44.254 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-07-02T14:02:44.256 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:44.255+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:44.257 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:02:44.258 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:44.258 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:44.259 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:02:44.260 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:44.260 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:44.260 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:44.260 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:02:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:02:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:02:44.262 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:02:44.263 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:02:44.263 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:02:44.320 INFO:tasks.workunit.client.0.smithi163.stderr: 81600K .......... .......... .......... .......... .......... 48% 16.8M 4s 2023-07-02T14:02:44.321 INFO:tasks.workunit.client.0.smithi163.stderr: 81650K .......... .......... .......... .......... .......... 48% 16.5M 4s 2023-07-02T14:02:44.321 INFO:tasks.workunit.client.0.smithi163.stderr: 81700K .......... .......... .......... .......... .......... 48% 20.8M 4s 2023-07-02T14:02:44.321 INFO:tasks.workunit.client.0.smithi163.stderr: 81750K .......... .......... .......... .......... .......... 48% 39.7M 4s 2023-07-02T14:02:44.321 INFO:tasks.workunit.client.0.smithi163.stderr: 81800K .......... .......... .......... .......... .......... 48% 18.5M 4s 2023-07-02T14:02:44.321 INFO:tasks.workunit.client.0.smithi163.stderr: 81850K .......... .......... .......... .......... .......... 48% 14.7M 4s 2023-07-02T14:02:44.321 INFO:tasks.workunit.client.0.smithi163.stderr: 81900K .......... .......... .......... .......... .......... 48% 28.3M 4s 2023-07-02T14:02:44.321 INFO:tasks.workunit.client.0.smithi163.stderr: 81950K .......... .......... .......... .......... .......... 48% 18.0M 4s 2023-07-02T14:02:44.322 INFO:tasks.workunit.client.0.smithi163.stderr: 82000K .......... .......... .......... .......... .......... 48% 15.8M 4s 2023-07-02T14:02:44.322 INFO:tasks.workunit.client.0.smithi163.stderr: 82050K .......... .......... .......... .......... .......... 48% 28.5M 4s 2023-07-02T14:02:44.322 INFO:tasks.workunit.client.0.smithi163.stderr: 82100K .......... .......... .......... .......... .......... 48% 20.1M 4s 2023-07-02T14:02:44.322 INFO:tasks.workunit.client.0.smithi163.stderr: 82150K .......... .......... .......... .......... .......... 48% 15.4M 4s 2023-07-02T14:02:44.322 INFO:tasks.workunit.client.0.smithi163.stderr: 82200K .......... .......... .......... .......... .......... 48% 39.0M 4s 2023-07-02T14:02:44.322 INFO:tasks.workunit.client.0.smithi163.stderr: 82250K .......... .......... .......... .......... .......... 48% 20.2M 4s 2023-07-02T14:02:44.322 INFO:tasks.workunit.client.0.smithi163.stderr: 82300K .......... .......... .......... .......... .......... 48% 20.6M 4s 2023-07-02T14:02:44.322 INFO:tasks.workunit.client.0.smithi163.stderr: 82350K .......... .......... .......... .......... .......... 48% 14.7M 4s 2023-07-02T14:02:44.323 INFO:tasks.workunit.client.0.smithi163.stderr: 82400K .......... .......... .......... .......... .......... 48% 19.4M 4s 2023-07-02T14:02:44.323 INFO:tasks.workunit.client.0.smithi163.stderr: 82450K .......... .......... .......... .......... .......... 48% 32.4M 4s 2023-07-02T14:02:44.323 INFO:tasks.workunit.client.0.smithi163.stderr: 82500K .......... .......... .......... .......... .......... 48% 16.6M 4s 2023-07-02T14:02:44.323 INFO:tasks.workunit.client.0.smithi163.stderr: 82550K .......... .......... .......... .......... .......... 49% 20.0M 4s 2023-07-02T14:02:44.323 INFO:tasks.workunit.client.0.smithi163.stderr: 82600K .......... .......... .......... .......... .......... 49% 22.5M 4s 2023-07-02T14:02:44.323 INFO:tasks.workunit.client.0.smithi163.stderr: 82650K .......... .......... .......... .......... .......... 49% 27.9M 4s 2023-07-02T14:02:44.323 INFO:tasks.workunit.client.0.smithi163.stderr: 82700K .......... .......... .......... .......... .......... 49% 17.7M 4s 2023-07-02T14:02:44.323 INFO:tasks.workunit.client.0.smithi163.stderr: 82750K .......... .......... .......... .......... .......... 49% 17.5M 4s 2023-07-02T14:02:44.324 INFO:tasks.workunit.client.0.smithi163.stderr: 82800K .......... .......... .......... .......... .......... 49% 25.7M 4s 2023-07-02T14:02:44.324 INFO:tasks.workunit.client.0.smithi163.stderr: 82850K .......... .......... .......... .......... .......... 49% 17.4M 4s 2023-07-02T14:02:44.324 INFO:tasks.workunit.client.0.smithi163.stderr: 82900K .......... .......... .......... .......... .......... 49% 23.0M 4s 2023-07-02T14:02:44.324 INFO:tasks.workunit.client.0.smithi163.stderr: 82950K .......... .......... .......... .......... .......... 49% 20.1M 4s 2023-07-02T14:02:44.324 INFO:tasks.workunit.client.0.smithi163.stderr: 83000K .......... .......... .......... .......... .......... 49% 21.4M 4s 2023-07-02T14:02:44.324 INFO:tasks.workunit.client.0.smithi163.stderr: 83050K .......... .......... .......... .......... .......... 49% 21.8M 4s 2023-07-02T14:02:44.324 INFO:tasks.workunit.client.0.smithi163.stderr: 83100K .......... .......... .......... .......... .......... 49% 16.6M 4s 2023-07-02T14:02:44.324 INFO:tasks.workunit.client.0.smithi163.stderr: 83150K .......... .......... .......... .......... .......... 49% 19.0M 4s 2023-07-02T14:02:44.325 INFO:tasks.workunit.client.0.smithi163.stderr: 83200K .......... .......... .......... .......... .......... 49% 34.6M 4s 2023-07-02T14:02:44.325 INFO:tasks.workunit.client.0.smithi163.stderr: 83250K .......... .......... .......... .......... .......... 49% 14.0M 4s 2023-07-02T14:02:44.325 INFO:tasks.workunit.client.0.smithi163.stderr: 83300K .......... .......... .......... .......... .......... 49% 29.7M 4s 2023-07-02T14:02:44.325 INFO:tasks.workunit.client.0.smithi163.stderr: 83350K .......... .......... .......... .......... .......... 49% 24.2M 4s 2023-07-02T14:02:44.325 INFO:tasks.workunit.client.0.smithi163.stderr: 83400K .......... .......... .......... .......... .......... 49% 15.1M 4s 2023-07-02T14:02:44.325 INFO:tasks.workunit.client.0.smithi163.stderr: 83450K .......... .......... .......... .......... .......... 49% 29.5M 4s 2023-07-02T14:02:44.325 INFO:tasks.workunit.client.0.smithi163.stderr: 83500K .......... .......... .......... .......... .......... 49% 16.4M 4s 2023-07-02T14:02:44.325 INFO:tasks.workunit.client.0.smithi163.stderr: 83550K .......... .......... .......... .......... .......... 49% 17.3M 4s 2023-07-02T14:02:44.326 INFO:tasks.workunit.client.0.smithi163.stderr: 83600K .......... .......... .......... .......... .......... 49% 23.6M 4s 2023-07-02T14:02:44.326 INFO:tasks.workunit.client.0.smithi163.stderr: 83650K .......... .......... .......... .......... .......... 49% 18.4M 4s 2023-07-02T14:02:44.326 INFO:tasks.workunit.client.0.smithi163.stderr: 83700K .......... .......... .......... .......... .......... 49% 27.6M 4s 2023-07-02T14:02:44.326 INFO:tasks.workunit.client.0.smithi163.stderr: 83750K .......... .......... .......... .......... .......... 49% 16.4M 4s 2023-07-02T14:02:44.326 INFO:tasks.workunit.client.0.smithi163.stderr: 83800K .......... .......... .......... .......... .......... 49% 26.8M 4s 2023-07-02T14:02:44.326 INFO:tasks.workunit.client.0.smithi163.stderr: 83850K .......... .......... .......... .......... .......... 49% 22.5M 4s 2023-07-02T14:02:44.326 INFO:tasks.workunit.client.0.smithi163.stderr: 83900K .......... .......... .......... .......... .......... 49% 14.8M 4s 2023-07-02T14:02:44.327 INFO:tasks.workunit.client.0.smithi163.stderr: 83950K .......... .......... .......... .......... .......... 49% 22.2M 4s 2023-07-02T14:02:44.327 INFO:tasks.workunit.client.0.smithi163.stderr: 84000K .......... .......... .......... .......... .......... 49% 23.5M 4s 2023-07-02T14:02:44.327 INFO:tasks.workunit.client.0.smithi163.stderr: 84050K .......... .......... .......... .......... .......... 49% 18.8M 4s 2023-07-02T14:02:44.327 INFO:tasks.workunit.client.0.smithi163.stderr: 84100K .......... .......... .......... .......... .......... 49% 30.0M 4s 2023-07-02T14:02:44.327 INFO:tasks.workunit.client.0.smithi163.stderr: 84150K .......... .......... .......... .......... .......... 49% 13.8M 4s 2023-07-02T14:02:44.327 INFO:tasks.workunit.client.0.smithi163.stderr: 84200K .......... .......... .......... .......... .......... 49% 27.2M 4s 2023-07-02T14:02:44.327 INFO:tasks.workunit.client.0.smithi163.stderr: 84250K .......... .......... .......... .......... .......... 50% 17.6M 4s 2023-07-02T14:02:44.327 INFO:tasks.workunit.client.0.smithi163.stderr: 84300K .......... .......... .......... .......... .......... 50% 21.6M 4s 2023-07-02T14:02:44.328 INFO:tasks.workunit.client.0.smithi163.stderr: 84350K .......... .......... .......... .......... .......... 50% 17.9M 4s 2023-07-02T14:02:44.328 INFO:tasks.workunit.client.0.smithi163.stderr: 84400K .......... .......... .......... .......... .......... 50% 24.6M 4s 2023-07-02T14:02:44.328 INFO:tasks.workunit.client.0.smithi163.stderr: 84450K .......... .......... .......... .......... .......... 50% 15.9M 4s 2023-07-02T14:02:44.328 INFO:tasks.workunit.client.0.smithi163.stderr: 84500K .......... .......... .......... .......... .......... 50% 23.4M 4s 2023-07-02T14:02:44.328 INFO:tasks.workunit.client.0.smithi163.stderr: 84550K .......... .......... .......... .......... .......... 50% 31.1M 4s 2023-07-02T14:02:44.328 INFO:tasks.workunit.client.0.smithi163.stderr: 84600K .......... .......... .......... .......... .......... 50% 28.3M 4s 2023-07-02T14:02:44.328 INFO:tasks.workunit.client.0.smithi163.stderr: 84650K .......... .......... .......... .......... .......... 50% 16.2M 4s 2023-07-02T14:02:44.328 INFO:tasks.workunit.client.0.smithi163.stderr: 84700K .......... .......... .......... .......... .......... 50% 21.3M 4s 2023-07-02T14:02:44.329 INFO:tasks.workunit.client.0.smithi163.stderr: 84750K .......... .......... .......... .......... .......... 50% 15.0M 4s 2023-07-02T14:02:44.329 INFO:tasks.workunit.client.0.smithi163.stderr: 84800K .......... .......... .......... .......... .......... 50% 22.1M 4s 2023-07-02T14:02:44.329 INFO:tasks.workunit.client.0.smithi163.stderr: 84850K .......... .......... .......... .......... .......... 50% 21.3M 4s 2023-07-02T14:02:44.329 INFO:tasks.workunit.client.0.smithi163.stderr: 84900K .......... .......... .......... .......... .......... 50% 25.2M 4s 2023-07-02T14:02:44.329 INFO:tasks.workunit.client.0.smithi163.stderr: 84950K .......... .......... .......... .......... .......... 50% 15.7M 4s 2023-07-02T14:02:44.329 INFO:tasks.workunit.client.0.smithi163.stderr: 85000K .......... .......... .......... .......... .......... 50% 24.3M 4s 2023-07-02T14:02:44.329 INFO:tasks.workunit.client.0.smithi163.stderr: 85050K .......... .......... .......... .......... .......... 50% 20.0M 4s 2023-07-02T14:02:44.329 INFO:tasks.workunit.client.0.smithi163.stderr: 85100K .......... .......... .......... .......... .......... 50% 24.7M 4s 2023-07-02T14:02:44.330 INFO:tasks.workunit.client.0.smithi163.stderr: 85150K .......... .......... .......... .......... .......... 50% 19.7M 4s 2023-07-02T14:02:44.330 INFO:tasks.workunit.client.0.smithi163.stderr: 85200K .......... .......... .......... .......... .......... 50% 16.6M 4s 2023-07-02T14:02:44.330 INFO:tasks.workunit.client.0.smithi163.stderr: 85250K .......... .......... .......... .......... .......... 50% 24.0M 4s 2023-07-02T14:02:44.330 INFO:tasks.workunit.client.0.smithi163.stderr: 85300K .......... .......... .......... .......... .......... 50% 18.8M 4s 2023-07-02T14:02:44.330 INFO:tasks.workunit.client.0.smithi163.stderr: 85350K .......... .......... .......... .......... .......... 50% 18.0M 4s 2023-07-02T14:02:44.330 INFO:tasks.workunit.client.0.smithi163.stderr: 85400K .......... .......... .......... .......... .......... 50% 23.9M 4s 2023-07-02T14:02:44.330 INFO:tasks.workunit.client.0.smithi163.stderr: 85450K .......... .......... .......... .......... .......... 50% 36.0M 4s 2023-07-02T14:02:44.330 INFO:tasks.workunit.client.0.smithi163.stderr: 85500K .......... .......... .......... .......... .......... 50% 15.2M 4s 2023-07-02T14:02:44.331 INFO:tasks.workunit.client.0.smithi163.stderr: 85550K .......... .......... .......... .......... .......... 50% 14.3M 4s 2023-07-02T14:02:44.331 INFO:tasks.workunit.client.0.smithi163.stderr: 85600K .......... .......... .......... .......... .......... 50% 39.7M 4s 2023-07-02T14:02:44.331 INFO:tasks.workunit.client.0.smithi163.stderr: 85650K .......... .......... .......... .......... .......... 50% 14.8M 4s 2023-07-02T14:02:44.331 INFO:tasks.workunit.client.0.smithi163.stderr: 85700K .......... .......... .......... .......... .......... 50% 19.4M 4s 2023-07-02T14:02:44.331 INFO:tasks.workunit.client.0.smithi163.stderr: 85750K .......... .......... .......... .......... .......... 50% 28.1M 4s 2023-07-02T14:02:44.331 INFO:tasks.workunit.client.0.smithi163.stderr: 85800K .......... .......... .......... .......... .......... 50% 24.7M 4s 2023-07-02T14:02:44.331 INFO:tasks.workunit.client.0.smithi163.stderr: 85850K .......... .......... .......... .......... .......... 50% 23.5M 4s 2023-07-02T14:02:44.331 INFO:tasks.workunit.client.0.smithi163.stderr: 85900K .......... .......... .......... .......... .......... 50% 17.3M 4s 2023-07-02T14:02:44.332 INFO:tasks.workunit.client.0.smithi163.stderr: 85950K .......... .......... .......... .......... .......... 51% 15.5M 4s 2023-07-02T14:02:44.332 INFO:tasks.workunit.client.0.smithi163.stderr: 86000K .......... .......... .......... .......... .......... 51% 24.8M 4s 2023-07-02T14:02:44.332 INFO:tasks.workunit.client.0.smithi163.stderr: 86050K .......... .......... .......... .......... .......... 51% 16.9M 4s 2023-07-02T14:02:44.332 INFO:tasks.workunit.client.0.smithi163.stderr: 86100K .......... .......... .......... .......... .......... 51% 30.2M 4s 2023-07-02T14:02:44.332 INFO:tasks.workunit.client.0.smithi163.stderr: 86150K .......... .......... .......... .......... .......... 51% 16.0M 4s 2023-07-02T14:02:44.332 INFO:tasks.workunit.client.0.smithi163.stderr: 86200K .......... .......... .......... .......... .......... 51% 21.9M 4s 2023-07-02T14:02:44.332 INFO:tasks.workunit.client.0.smithi163.stderr: 86250K .......... .......... .......... .......... .......... 51% 25.8M 4s 2023-07-02T14:02:44.332 INFO:tasks.workunit.client.0.smithi163.stderr: 86300K .......... .......... .......... .......... .......... 51% 24.3M 4s 2023-07-02T14:02:44.333 INFO:tasks.workunit.client.0.smithi163.stderr: 86350K .......... .......... .......... .......... .......... 51% 13.7M 4s 2023-07-02T14:02:44.333 INFO:tasks.workunit.client.0.smithi163.stderr: 86400K .......... .......... .......... .......... .......... 51% 26.9M 4s 2023-07-02T14:02:44.333 INFO:tasks.workunit.client.0.smithi163.stderr: 86450K .......... .......... .......... .......... .......... 51% 21.0M 4s 2023-07-02T14:02:44.333 INFO:tasks.workunit.client.0.smithi163.stderr: 86500K .......... .......... .......... .......... .......... 51% 28.6M 4s 2023-07-02T14:02:44.333 INFO:tasks.workunit.client.0.smithi163.stderr: 86550K .......... .......... .......... .......... .......... 51% 17.1M 4s 2023-07-02T14:02:44.333 INFO:tasks.workunit.client.0.smithi163.stderr: 86600K .......... .......... .......... .......... .......... 51% 18.4M 4s 2023-07-02T14:02:44.333 INFO:tasks.workunit.client.0.smithi163.stderr: 86650K .......... .......... .......... .......... .......... 51% 30.6M 4s 2023-07-02T14:02:44.333 INFO:tasks.workunit.client.0.smithi163.stderr: 86700K .......... .......... .......... .......... .......... 51% 15.9M 4s 2023-07-02T14:02:44.334 INFO:tasks.workunit.client.0.smithi163.stderr: 86750K .......... .......... .......... .......... .......... 51% 14.5M 4s 2023-07-02T14:02:44.334 INFO:tasks.workunit.client.0.smithi163.stderr: 86800K .......... .......... .......... .......... .......... 51% 22.2M 4s 2023-07-02T14:02:44.334 INFO:tasks.workunit.client.0.smithi163.stderr: 86850K .......... .......... .......... .......... .......... 51% 59.3M 4s 2023-07-02T14:02:44.334 INFO:tasks.workunit.client.0.smithi163.stderr: 86900K .......... .......... .......... .......... .......... 51% 13.2M 4s 2023-07-02T14:02:44.334 INFO:tasks.workunit.client.0.smithi163.stderr: 86950K .......... .......... .......... .......... .......... 51% 23.5M 4s 2023-07-02T14:02:44.334 INFO:tasks.workunit.client.0.smithi163.stderr: 87000K .......... .......... .......... .......... .......... 51% 35.0M 4s 2023-07-02T14:02:44.334 INFO:tasks.workunit.client.0.smithi163.stderr: 87050K .......... .......... .......... .......... .......... 51% 14.7M 4s 2023-07-02T14:02:44.335 INFO:tasks.workunit.client.0.smithi163.stderr: 87100K .......... .......... .......... .......... .......... 51% 27.1M 4s 2023-07-02T14:02:44.335 INFO:tasks.workunit.client.0.smithi163.stderr: 87150K .......... .......... .......... .......... .......... 51% 20.7M 4s 2023-07-02T14:02:44.335 INFO:tasks.workunit.client.0.smithi163.stderr: 87200K .......... .......... .......... .......... .......... 51% 16.9M 4s 2023-07-02T14:02:44.335 INFO:tasks.workunit.client.0.smithi163.stderr: 87250K .......... .......... .......... .......... .......... 51% 20.3M 4s 2023-07-02T14:02:44.335 INFO:tasks.workunit.client.0.smithi163.stderr: 87300K .......... .......... .......... .......... .......... 51% 26.7M 4s 2023-07-02T14:02:44.335 INFO:tasks.workunit.client.0.smithi163.stderr: 87350K .......... .......... .......... .......... .......... 51% 22.0M 4s 2023-07-02T14:02:44.335 INFO:tasks.workunit.client.0.smithi163.stderr: 87400K .......... .......... .......... .......... .......... 51% 18.5M 4s 2023-07-02T14:02:44.340 INFO:tasks.workunit.client.0.smithi163.stderr: 87450K .......... .......... .......... .......... .......... 51% 18.7M 4s 2023-07-02T14:02:44.340 INFO:tasks.workunit.client.0.smithi163.stderr: 87500K .......... .......... .......... .......... .......... 51% 20.5M 4s 2023-07-02T14:02:44.341 INFO:tasks.workunit.client.0.smithi163.stderr: 87550K .......... .......... .......... .......... .......... 51% 16.6M 4s 2023-07-02T14:02:44.341 INFO:tasks.workunit.client.0.smithi163.stderr: 87600K .......... .......... .......... .......... .......... 51% 20.1M 4s 2023-07-02T14:02:44.341 INFO:tasks.workunit.client.0.smithi163.stderr: 87650K .......... .......... .......... .......... .......... 52% 33.7M 4s 2023-07-02T14:02:44.341 INFO:tasks.workunit.client.0.smithi163.stderr: 87700K .......... .......... .......... .......... .......... 52% 18.4M 4s 2023-07-02T14:02:44.341 INFO:tasks.workunit.client.0.smithi163.stderr: 87750K .......... .......... .......... .......... .......... 52% 15.0M 4s 2023-07-02T14:02:44.341 INFO:tasks.workunit.client.0.smithi163.stderr: 87800K .......... .......... .......... .......... .......... 52% 24.8M 4s 2023-07-02T14:02:44.341 INFO:tasks.workunit.client.0.smithi163.stderr: 87850K .......... .......... .......... .......... .......... 52% 27.3M 4s 2023-07-02T14:02:44.343 INFO:tasks.workunit.client.0.smithi163.stderr: 87900K .......... .......... .......... .......... .......... 52% 17.8M 4s 2023-07-02T14:02:44.345 INFO:tasks.workunit.client.0.smithi163.stderr: 87950K .......... .......... .......... .......... .......... 52% 22.6M 4s 2023-07-02T14:02:44.348 INFO:tasks.workunit.client.0.smithi163.stderr: 88000K .......... .......... .......... .......... .......... 52% 15.1M 4s 2023-07-02T14:02:44.351 INFO:tasks.workunit.client.0.smithi163.stderr: 88050K .......... .......... .......... .......... .......... 52% 21.7M 4s 2023-07-02T14:02:44.354 INFO:tasks.workunit.client.0.smithi163.stderr: 88100K .......... .......... .......... .......... .......... 52% 17.8M 4s 2023-07-02T14:02:44.355 INFO:tasks.workunit.client.0.smithi163.stderr: 88150K .......... .......... .......... .......... .......... 52% 37.1M 4s 2023-07-02T14:02:44.356 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:44.355+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:44.357 INFO:tasks.workunit.client.0.smithi163.stderr: 88200K .......... .......... .......... .......... .......... 52% 18.4M 4s 2023-07-02T14:02:44.360 INFO:tasks.workunit.client.0.smithi163.stderr: 88250K .......... .......... .......... .......... .......... 52% 22.9M 4s 2023-07-02T14:02:44.362 INFO:tasks.workunit.client.0.smithi163.stderr: 88300K .......... .......... .......... .......... .......... 52% 17.7M 4s 2023-07-02T14:02:44.365 INFO:tasks.workunit.client.0.smithi163.stderr: 88350K .......... .......... .......... .......... .......... 52% 17.3M 4s 2023-07-02T14:02:44.366 INFO:tasks.workunit.client.0.smithi163.stderr: 88400K .......... .......... .......... .......... .......... 52% 40.0M 4s 2023-07-02T14:02:44.370 INFO:tasks.workunit.client.0.smithi163.stderr: 88450K .......... .......... .......... .......... .......... 52% 14.6M 4s 2023-07-02T14:02:44.372 INFO:tasks.workunit.client.0.smithi163.stderr: 88500K .......... .......... .......... .......... .......... 52% 26.4M 4s 2023-07-02T14:02:44.374 INFO:tasks.workunit.client.0.smithi163.stderr: 88550K .......... .......... .......... .......... .......... 52% 17.3M 4s 2023-07-02T14:02:44.378 INFO:tasks.workunit.client.0.smithi163.stderr: 88600K .......... .......... .......... .......... .......... 52% 15.1M 4s 2023-07-02T14:02:44.378 INFO:tasks.workunit.client.0.smithi163.stderr: 88650K .......... .......... .......... .......... .......... 52% 57.4M 4s 2023-07-02T14:02:44.380 INFO:tasks.workunit.client.0.smithi163.stderr: 88700K .......... .......... .......... .......... .......... 52% 21.9M 4s 2023-07-02T14:02:44.384 INFO:tasks.workunit.client.0.smithi163.stderr: 88750K .......... .......... .......... .......... .......... 52% 13.6M 4s 2023-07-02T14:02:44.386 INFO:tasks.workunit.client.0.smithi163.stderr: 88800K .......... .......... .......... .......... .......... 52% 25.4M 4s 2023-07-02T14:02:44.389 INFO:tasks.workunit.client.0.smithi163.stderr: 88850K .......... .......... .......... .......... .......... 52% 18.3M 4s 2023-07-02T14:02:44.391 INFO:tasks.workunit.client.0.smithi163.stderr: 88900K .......... .......... .......... .......... .......... 52% 26.4M 4s 2023-07-02T14:02:44.393 INFO:tasks.workunit.client.0.smithi163.stderr: 88950K .......... .......... .......... .......... .......... 52% 22.2M 4s 2023-07-02T14:02:44.395 INFO:tasks.workunit.client.0.smithi163.stderr: 89000K .......... .......... .......... .......... .......... 52% 19.4M 4s 2023-07-02T14:02:44.398 INFO:tasks.workunit.client.0.smithi163.stderr: 89050K .......... .......... .......... .......... .......... 52% 17.8M 4s 2023-07-02T14:02:44.399 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:44.399 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:44.400 INFO:tasks.workunit.client.0.smithi163.stderr: 89100K .......... .......... .......... .......... .......... 52% 27.6M 4s 2023-07-02T14:02:44.404 INFO:tasks.workunit.client.0.smithi163.stderr: 89150K .......... .......... .......... .......... .......... 52% 11.8M 4s 2023-07-02T14:02:44.405 INFO:tasks.workunit.client.0.smithi163.stderr: 89200K .......... .......... .......... .......... .......... 52% 58.7M 4s 2023-07-02T14:02:44.407 INFO:tasks.workunit.client.0.smithi163.stderr: 89250K .......... .......... .......... .......... .......... 52% 20.3M 4s 2023-07-02T14:02:44.409 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:02:44.410 INFO:tasks.workunit.client.0.smithi163.stderr: 89300K .......... .......... .......... .......... .......... 53% 23.0M 4s 2023-07-02T14:02:44.413 INFO:tasks.workunit.client.0.smithi163.stderr: 89350K .......... .......... .......... .......... .......... 53% 14.3M 4s 2023-07-02T14:02:44.415 INFO:tasks.workunit.client.0.smithi163.stderr: 89400K .......... .......... .......... .......... .......... 53% 25.1M 4s 2023-07-02T14:02:44.417 INFO:tasks.workunit.client.0.smithi163.stderr: 89450K .......... .......... .......... .......... .......... 53% 24.9M 4s 2023-07-02T14:02:44.419 INFO:tasks.workunit.client.0.smithi163.stderr: 89500K .......... .......... .......... .......... .......... 53% 20.6M 4s 2023-07-02T14:02:44.422 INFO:tasks.workunit.client.0.smithi163.stderr: 89550K .......... .......... .......... .......... .......... 53% 15.3M 4s 2023-07-02T14:02:44.425 INFO:tasks.workunit.client.0.smithi163.stderr: 89600K .......... .......... .......... .......... .......... 53% 21.9M 4s 2023-07-02T14:02:44.427 INFO:tasks.workunit.client.0.smithi163.stderr: 89650K .......... .......... .......... .......... .......... 53% 24.0M 4s 2023-07-02T14:02:44.429 INFO:tasks.workunit.client.0.smithi163.stderr: 89700K .......... .......... .......... .......... .......... 53% 20.2M 4s 2023-07-02T14:02:44.432 INFO:tasks.workunit.client.0.smithi163.stderr: 89750K .......... .......... .......... .......... .......... 53% 16.3M 4s 2023-07-02T14:02:44.434 INFO:tasks.workunit.client.0.smithi163.stderr: 89800K .......... .......... .......... .......... .......... 53% 25.8M 4s 2023-07-02T14:02:44.437 INFO:tasks.workunit.client.0.smithi163.stderr: 89850K .......... .......... .......... .......... .......... 53% 15.9M 4s 2023-07-02T14:02:44.438 INFO:tasks.workunit.client.0.smithi163.stderr: 89900K .......... .......... .......... .......... .......... 53% 30.8M 4s 2023-07-02T14:02:44.441 INFO:tasks.workunit.client.0.smithi163.stderr: 89950K .......... .......... .......... .......... .......... 53% 18.2M 4s 2023-07-02T14:02:44.443 INFO:tasks.workunit.client.0.smithi163.stderr: 90000K .......... .......... .......... .......... .......... 53% 24.8M 4s 2023-07-02T14:02:44.446 INFO:tasks.workunit.client.0.smithi163.stderr: 90050K .......... .......... .......... .......... .......... 53% 16.7M 4s 2023-07-02T14:02:44.448 INFO:tasks.workunit.client.0.smithi163.stderr: 90100K .......... .......... .......... .......... .......... 53% 25.7M 4s 2023-07-02T14:02:44.451 INFO:tasks.workunit.client.0.smithi163.stderr: 90150K .......... .......... .......... .......... .......... 53% 19.2M 4s 2023-07-02T14:02:44.452 INFO:tasks.workunit.client.0.smithi163.stderr: 90200K .......... .......... .......... .......... .......... 53% 27.3M 4s 2023-07-02T14:02:44.456 INFO:tasks.workunit.client.0.smithi163.stderr: 90250K .......... .......... .......... .......... .......... 53% 14.8M 4s 2023-07-02T14:02:44.456 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:44.455+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:44.457 INFO:tasks.workunit.client.0.smithi163.stderr: 90300K .......... .......... .......... .......... .......... 53% 25.3M 4s 2023-07-02T14:02:44.460 INFO:tasks.workunit.client.0.smithi163.stderr: 90350K .......... .......... .......... .......... .......... 53% 22.8M 4s 2023-07-02T14:02:44.463 INFO:tasks.workunit.client.0.smithi163.stderr: 90400K .......... .......... .......... .......... .......... 53% 16.0M 4s 2023-07-02T14:02:44.465 INFO:tasks.workunit.client.0.smithi163.stderr: 90450K .......... .......... .......... .......... .......... 53% 17.6M 4s 2023-07-02T14:02:44.467 INFO:tasks.workunit.client.0.smithi163.stderr: 90500K .......... .......... .......... .......... .......... 53% 32.8M 4s 2023-07-02T14:02:44.469 INFO:tasks.workunit.client.0.smithi163.stderr: 90550K .......... .......... .......... .......... .......... 53% 24.0M 4s 2023-07-02T14:02:44.472 INFO:tasks.workunit.client.0.smithi163.stderr: 90600K .......... .......... .......... .......... .......... 53% 14.9M 4s 2023-07-02T14:02:44.474 INFO:tasks.workunit.client.0.smithi163.stderr: 90650K .......... .......... .......... .......... .......... 53% 25.7M 4s 2023-07-02T14:02:44.476 INFO:tasks.workunit.client.0.smithi163.stderr: 90700K .......... .......... .......... .......... .......... 53% 30.6M 4s 2023-07-02T14:02:44.480 INFO:tasks.workunit.client.0.smithi163.stderr: 90750K .......... .......... .......... .......... .......... 53% 12.8M 4s 2023-07-02T14:02:44.482 INFO:tasks.workunit.client.0.smithi163.stderr: 90800K .......... .......... .......... .......... .......... 53% 22.9M 4s 2023-07-02T14:02:44.483 INFO:tasks.workunit.client.0.smithi163.stderr: 90850K .......... .......... .......... .......... .......... 53% 31.1M 4s 2023-07-02T14:02:44.487 INFO:tasks.workunit.client.0.smithi163.stderr: 90900K .......... .......... .......... .......... .......... 53% 13.3M 4s 2023-07-02T14:02:44.489 INFO:tasks.workunit.client.0.smithi163.stderr: 90950K .......... .......... .......... .......... .......... 53% 26.2M 4s 2023-07-02T14:02:44.490 INFO:tasks.workunit.client.0.smithi163.stderr: 91000K .......... .......... .......... .......... .......... 54% 24.8M 4s 2023-07-02T14:02:44.493 INFO:tasks.workunit.client.0.smithi163.stderr: 91050K .......... .......... .......... .......... .......... 54% 21.1M 4s 2023-07-02T14:02:44.496 INFO:tasks.workunit.client.0.smithi163.stderr: 91100K .......... .......... .......... .......... .......... 54% 18.4M 4s 2023-07-02T14:02:44.499 INFO:tasks.workunit.client.0.smithi163.stderr: 91150K .......... .......... .......... .......... .......... 54% 15.9M 4s 2023-07-02T14:02:44.501 INFO:tasks.workunit.client.0.smithi163.stderr: 91200K .......... .......... .......... .......... .......... 54% 20.1M 4s 2023-07-02T14:02:44.504 INFO:tasks.workunit.client.0.smithi163.stderr: 91250K .......... .......... .......... .......... .......... 54% 21.1M 4s 2023-07-02T14:02:44.507 INFO:tasks.workunit.client.0.smithi163.stderr: 91300K .......... .......... .......... .......... .......... 54% 14.6M 4s 2023-07-02T14:02:44.508 INFO:tasks.workunit.client.0.smithi163.stderr: 91350K .......... .......... .......... .......... .......... 54% 28.3M 4s 2023-07-02T14:02:44.511 INFO:tasks.workunit.client.0.smithi163.stderr: 91400K .......... .......... .......... .......... .......... 54% 21.6M 4s 2023-07-02T14:02:44.513 INFO:tasks.workunit.client.0.smithi163.stderr: 91450K .......... .......... .......... .......... .......... 54% 21.1M 4s 2023-07-02T14:02:44.517 INFO:tasks.workunit.client.0.smithi163.stderr: 91500K .......... .......... .......... .......... .......... 54% 13.8M 4s 2023-07-02T14:02:44.519 INFO:tasks.workunit.client.0.smithi163.stderr: 91550K .......... .......... .......... .......... .......... 54% 17.5M 4s 2023-07-02T14:02:44.522 INFO:tasks.workunit.client.0.smithi163.stderr: 91600K .......... .......... .......... .......... .......... 54% 23.7M 4s 2023-07-02T14:02:44.524 INFO:tasks.workunit.client.0.smithi163.stderr: 91650K .......... .......... .......... .......... .......... 54% 22.9M 4s 2023-07-02T14:02:44.527 INFO:tasks.workunit.client.0.smithi163.stderr: 91700K .......... .......... .......... .......... .......... 54% 16.7M 4s 2023-07-02T14:02:44.529 INFO:tasks.workunit.client.0.smithi163.stderr: 91750K .......... .......... .......... .......... .......... 54% 26.0M 4s 2023-07-02T14:02:44.531 INFO:tasks.workunit.client.0.smithi163.stderr: 91800K .......... .......... .......... .......... .......... 54% 18.6M 4s 2023-07-02T14:02:44.532 INFO:tasks.workunit.client.0.smithi163.stderr: 91850K .......... .......... .......... .......... .......... 54% 31.2M 4s 2023-07-02T14:02:44.534 INFO:tasks.workunit.client.0.smithi163.stderr: 91900K .......... .......... .......... .......... .......... 54% 25.8M 4s 2023-07-02T14:02:44.538 INFO:tasks.workunit.client.0.smithi163.stderr: 91950K .......... .......... .......... .......... .......... 54% 16.4M 4s 2023-07-02T14:02:44.540 INFO:tasks.workunit.client.0.smithi163.stderr: 92000K .......... .......... .......... .......... .......... 54% 18.0M 4s 2023-07-02T14:02:44.544 INFO:tasks.workunit.client.0.smithi163.stderr: 92050K .......... .......... .......... .......... .......... 54% 15.7M 4s 2023-07-02T14:02:44.545 INFO:tasks.workunit.client.0.smithi163.stderr: 92100K .......... .......... .......... .......... .......... 54% 26.7M 4s 2023-07-02T14:02:44.548 INFO:tasks.workunit.client.0.smithi163.stderr: 92150K .......... .......... .......... .......... .......... 54% 19.2M 4s 2023-07-02T14:02:44.549 INFO:tasks.workunit.client.0.smithi163.stderr: 92200K .......... .......... .......... .......... .......... 54% 40.0M 4s 2023-07-02T14:02:44.552 INFO:tasks.workunit.client.0.smithi163.stderr: 92250K .......... .......... .......... .......... .......... 54% 14.9M 4s 2023-07-02T14:02:44.555 INFO:tasks.workunit.client.0.smithi163.stderr: 92300K .......... .......... .......... .......... .......... 54% 22.0M 4s 2023-07-02T14:02:44.557 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:44.556+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:44.558 INFO:tasks.workunit.client.0.smithi163.stderr: 92350K .......... .......... .......... .......... .......... 54% 13.2M 4s 2023-07-02T14:02:44.560 INFO:tasks.workunit.client.0.smithi163.stderr: 92400K .......... .......... .......... .......... .......... 54% 29.2M 4s 2023-07-02T14:02:44.562 INFO:tasks.workunit.client.0.smithi163.stderr: 92450K .......... .......... .......... .......... .......... 54% 22.2M 4s 2023-07-02T14:02:44.564 INFO:tasks.workunit.client.0.smithi163.stderr: 92500K .......... .......... .......... .......... .......... 54% 18.9M 4s 2023-07-02T14:02:44.567 INFO:tasks.workunit.client.0.smithi163.stderr: 92550K .......... .......... .......... .......... .......... 54% 25.4M 4s 2023-07-02T14:02:44.568 INFO:tasks.workunit.client.0.smithi163.stderr: 92600K .......... .......... .......... .......... .......... 54% 25.2M 4s 2023-07-02T14:02:44.571 INFO:tasks.workunit.client.0.smithi163.stderr: 92650K .......... .......... .......... .......... .......... 54% 16.2M 4s 2023-07-02T14:02:44.573 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:44.573 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:44.573 INFO:tasks.workunit.client.0.smithi163.stderr: 92700K .......... .......... .......... .......... .......... 55% 27.9M 4s 2023-07-02T14:02:44.577 INFO:tasks.workunit.client.0.smithi163.stderr: 92750K .......... .......... .......... .......... .......... 55% 14.4M 4s 2023-07-02T14:02:44.579 INFO:tasks.workunit.client.0.smithi163.stderr: 92800K .......... .......... .......... .......... .......... 55% 21.1M 4s 2023-07-02T14:02:44.581 INFO:tasks.workunit.client.0.smithi163.stderr: 92850K .......... .......... .......... .......... .......... 55% 24.8M 4s 2023-07-02T14:02:44.583 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:02:44.584 INFO:tasks.workunit.client.0.smithi163.stderr: 92900K .......... .......... .......... .......... .......... 55% 16.9M 4s 2023-07-02T14:02:44.585 INFO:tasks.workunit.client.0.smithi163.stderr: 92950K .......... .......... .......... .......... .......... 55% 32.5M 4s 2023-07-02T14:02:44.588 INFO:tasks.workunit.client.0.smithi163.stderr: 93000K .......... .......... .......... .......... .......... 55% 18.8M 4s 2023-07-02T14:02:44.592 INFO:tasks.workunit.client.0.smithi163.stderr: 93050K .......... .......... .......... .......... .......... 55% 14.0M 4s 2023-07-02T14:02:44.593 INFO:tasks.workunit.client.0.smithi163.stderr: 93100K .......... .......... .......... .......... .......... 55% 34.6M 4s 2023-07-02T14:02:44.596 INFO:tasks.workunit.client.0.smithi163.stderr: 93150K .......... .......... .......... .......... .......... 55% 17.8M 4s 2023-07-02T14:02:44.598 INFO:tasks.workunit.client.0.smithi163.stderr: 93200K .......... .......... .......... .......... .......... 55% 20.4M 4s 2023-07-02T14:02:44.599 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:02:44.599 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:44.599 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 42, 2023-07-02T14:02:44.599 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2", 2023-07-02T14:02:44.600 INFO:teuthology.orchestra.run.smithi016.stdout: "created": "2023-07-02T14:01:37.298060+0000", 2023-07-02T14:02:44.600 INFO:teuthology.orchestra.run.smithi016.stdout: "modified": "2023-07-02T14:02:44.069293+0000", 2023-07-02T14:02:44.600 INFO:teuthology.orchestra.run.smithi016.stdout: "last_up_change": "2023-07-02T14:01:45.271760+0000", 2023-07-02T14:02:44.600 INFO:teuthology.orchestra.run.smithi016.stdout: "last_in_change": "2023-07-02T14:01:39.129121+0000", 2023-07-02T14:02:44.600 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-07-02T14:02:44.600 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_num": 5799936, 2023-07-02T14:02:44.600 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_set": [ 2023-07-02T14:02:44.601 INFO:teuthology.orchestra.run.smithi016.stdout: "pglog_hardlimit", 2023-07-02T14:02:44.601 INFO:teuthology.orchestra.run.smithi016.stdout: "purged_snapdirs", 2023-07-02T14:02:44.601 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_deletes", 2023-07-02T14:02:44.601 INFO:teuthology.orchestra.run.smithi016.stdout: "sortbitwise" 2023-07-02T14:02:44.601 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:44.601 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_version": 4, 2023-07-02T14:02:44.601 INFO:teuthology.orchestra.run.smithi016.stdout: "full_ratio": 0.94999998807907104, 2023-07-02T14:02:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-07-02T14:02:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2023-07-02T14:02:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_snapshot": "", 2023-07-02T14:02:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_max": 2, 2023-07-02T14:02:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd": 6, 2023-07-02T14:02:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "require_min_compat_client": "luminous", 2023-07-02T14:02:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "min_compat_client": "luminous", 2023-07-02T14:02:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "require_osd_release": "pacific", 2023-07-02T14:02:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: "pools": [ 2023-07-02T14:02:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 1, 2023-07-02T14:02:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "device_health_metrics", 2023-07-02T14:02:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:45.323455+0000", 2023-07-02T14:02:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2023-07-02T14:02:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2023-07-02T14:02:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:02:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:02:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:02:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:02:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:02:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:02:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:02:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:02:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:02:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:02:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 1, 2023-07-02T14:02:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 1, 2023-07-02T14:02:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 1, 2023-07-02T14:02:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 1, 2023-07-02T14:02:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 1, 2023-07-02T14:02:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:02:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:02:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:02:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:02:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:02:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:02:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:02:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "14", 2023-07-02T14:02:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:02:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:02:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:02:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:02:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:02:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2023-07-02T14:02:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2023-07-02T14:02:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:02:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:02:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:02:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:02:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:02:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:02:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:02:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:02:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:02:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:02:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:02:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:02:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:02:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:02:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:02:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:02:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:02:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:02:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:02:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:02:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:02:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:02:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:02:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:02:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:02:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:02:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:02:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:02:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:02:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:02:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "options": { 2023-07-02T14:02:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_max": 32, 2023-07-02T14:02:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_min": 1 2023-07-02T14:02:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:02:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_devicehealth": {} 2023-07-02T14:02:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 2, 2023-07-02T14:02:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "rbd", 2023-07-02T14:02:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:47.448897+0000", 2023-07-02T14:02:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 8193, 2023-07-02T14:02:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2023-07-02T14:02:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:02:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:02:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:02:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:02:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:02:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:02:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:02:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:02:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:02:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:02:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2023-07-02T14:02:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2023-07-02T14:02:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2023-07-02T14:02:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2023-07-02T14:02:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2023-07-02T14:02:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:02:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:02:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:02:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:02:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:02:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:02:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:02:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "40", 2023-07-02T14:02:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:02:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:02:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:02:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:02:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:02:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 3, 2023-07-02T14:02:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 40, 2023-07-02T14:02:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:02:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:02:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:02:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:02:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:02:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:02:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:02:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:02:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:02:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:02:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:02:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:02:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:02:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:02:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:02:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:02:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:02:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:02:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:02:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:02:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:02:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:02:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:02:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:02:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:02:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:02:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:02:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:02:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:02:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:02:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2023-07-02T14:02:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:02:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2023-07-02T14:02:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:02:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:02:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "43c446fe-c811-4ee3-99e8-d7452edfc1c1", 2023-07-02T14:02:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 0, 2023-07-02T14:02:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2023-07-02T14:02:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2023-07-02T14:02:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6802", 2023-07-02T14:02:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6803", 2023-07-02T14:02:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6806", 2023-07-02T14:02:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6807", 2023-07-02T14:02:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6804", 2023-07-02T14:02:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6805", 2023-07-02T14:02:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6801/83359", 2023-07-02T14:02:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6803/83359", 2023-07-02T14:02:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6807/83359", 2023-07-02T14:02:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6805/83359", 2023-07-02T14:02:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:02:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b", 2023-07-02T14:02:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 25, 2023-07-02T14:02:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6808", 2023-07-02T14:02:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6809", 2023-07-02T14:02:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6810", 2023-07-02T14:02:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6811", 2023-07-02T14:02:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6814", 2023-07-02T14:02:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6815", 2023-07-02T14:02:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6812", 2023-07-02T14:02:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6813", 2023-07-02T14:02:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6809/83360", 2023-07-02T14:02:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6811/83360", 2023-07-02T14:02:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6815/83360", 2023-07-02T14:02:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6813/83360", 2023-07-02T14:02:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:02:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "99f8ea0a-5a71-4abe-b087-8e891fd11d1b", 2023-07-02T14:02:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 14, 2023-07-02T14:02:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6816", 2023-07-02T14:02:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6817", 2023-07-02T14:02:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6818", 2023-07-02T14:02:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6819", 2023-07-02T14:02:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6822", 2023-07-02T14:02:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6823", 2023-07-02T14:02:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6820", 2023-07-02T14:02:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6821", 2023-07-02T14:02:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6817/83361", 2023-07-02T14:02:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6819/83361", 2023-07-02T14:02:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6823/83361", 2023-07-02T14:02:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6821/83361", 2023-07-02T14:02:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:02:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "d285b4d5-0bef-421a-b84a-943e485b10c2", 2023-07-02T14:02:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 18, 2023-07-02T14:02:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6808", 2023-07-02T14:02:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6809", 2023-07-02T14:02:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6810", 2023-07-02T14:02:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6811", 2023-07-02T14:02:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6814", 2023-07-02T14:02:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6815", 2023-07-02T14:02:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6812", 2023-07-02T14:02:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6813", 2023-07-02T14:02:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6809/83350", 2023-07-02T14:02:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6811/83350", 2023-07-02T14:02:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6815/83350", 2023-07-02T14:02:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6813/83350", 2023-07-02T14:02:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:02:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "7ba0b0db-e407-4c63-8278-aab09f3082c4", 2023-07-02T14:02:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 0.23297119140625, 2023-07-02T14:02:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 0, 2023-07-02T14:02:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6800", 2023-07-02T14:02:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6801", 2023-07-02T14:02:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6802", 2023-07-02T14:02:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6803", 2023-07-02T14:02:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6806", 2023-07-02T14:02:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6807", 2023-07-02T14:02:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6804", 2023-07-02T14:02:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6805", 2023-07-02T14:02:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6801/83351", 2023-07-02T14:02:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6803/83351", 2023-07-02T14:02:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6807/83351", 2023-07-02T14:02:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6805/83351", 2023-07-02T14:02:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:02:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "29663bfa-f6fe-4ce7-b45a-a9b7b4424c42", 2023-07-02T14:02:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 34, 2023-07-02T14:02:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6816", 2023-07-02T14:02:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6817", 2023-07-02T14:02:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6818", 2023-07-02T14:02:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6819", 2023-07-02T14:02:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6822", 2023-07-02T14:02:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6823", 2023-07-02T14:02:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6820", 2023-07-02T14:02:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6821", 2023-07-02T14:02:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6817/83352", 2023-07-02T14:02:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6819/83352", 2023-07-02T14:02:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6823/83352", 2023-07-02T14:02:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6821/83352", 2023-07-02T14:02:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_xinfo": [ 2023-07-02T14:02:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:02:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.001178+0000", 2023-07-02T14:02:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:02:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.184181+0000", 2023-07-02T14:02:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:02:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.274854+0000", 2023-07-02T14:02:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:02:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.186217+0000", 2023-07-02T14:02:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:02:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.247442+0000", 2023-07-02T14:02:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:02:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.220075+0000", 2023-07-02T14:02:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap": [ 2023-07-02T14:02:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.2", 2023-07-02T14:02:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:02:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: 1, 2023-07-02T14:02:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: 5 2023-07-02T14:02:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.4", 2023-07-02T14:02:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:02:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: 5, 2023-07-02T14:02:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: 0 2023-07-02T14:02:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_items": [ 2023-07-02T14:02:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.7", 2023-07-02T14:02:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "mappings": [ 2023-07-02T14:02:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "from": 4, 2023-07-02T14:02:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "to": 5 2023-07-02T14:02:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_temp": [], 2023-07-02T14:02:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_temp": [], 2023-07-02T14:02:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklist": {}, 2023-07-02T14:02:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "range_blocklist": {}, 2023-07-02T14:02:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profiles": { 2023-07-02T14:02:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "default": { 2023-07-02T14:02:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2023-07-02T14:02:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "2", 2023-07-02T14:02:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "1", 2023-07-02T14:02:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "jerasure", 2023-07-02T14:02:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "ruleset-failure-domain": "osd", 2023-07-02T14:02:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "technique": "reed_sol_van" 2023-07-02T14:02:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps_queue": [ 2023-07-02T14:02:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 2, 2023-07-02T14:02:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "snaps": [ 2023-07-02T14:02:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "begin": 2, 2023-07-02T14:02:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "length": 1 2023-07-02T14:02:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "new_removed_snaps": [], 2023-07-02T14:02:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "new_purged_snaps": [], 2023-07-02T14:02:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_node_flags": {}, 2023-07-02T14:02:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "device_class_flags": {}, 2023-07-02T14:02:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode": { 2023-07-02T14:02:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_enabled": false, 2023-07-02T14:02:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_bucket_count": 0, 2023-07-02T14:02:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "degraded_stretch_mode": 0, 2023-07-02T14:02:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "recovering_stretch_mode": 0, 2023-07-02T14:02:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_bucket": 0 2023-07-02T14:02:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:44.699 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:44.701 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:44.700+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:44.702 INFO:tasks.thrashosds.thrasher:j is {'epoch': 42, 'fsid': '9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2', 'created': '2023-07-02T14:01:37.298060+0000', 'modified': '2023-07-02T14:02:44.069293+0000', 'last_up_change': '2023-07-02T14:01:45.271760+0000', 'last_in_change': '2023-07-02T14:01:39.129121+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-07-02T14:01:45.323455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-02T14:01:47.448897+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 40, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '43c446fe-c811-4ee3-99e8-d7452edfc1c1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6800', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6801', 'nonce': 83359}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6802', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6803', 'nonce': 83359}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6806', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6807', 'nonce': 83359}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6804', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6805', 'nonce': 83359}]}, 'public_addr': '172.21.15.16:6801/83359', 'cluster_addr': '172.21.15.16:6803/83359', 'heartbeat_back_addr': '172.21.15.16:6807/83359', 'heartbeat_front_addr': '172.21.15.16:6805/83359', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 25, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6808', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6809', 'nonce': 83360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6810', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6811', 'nonce': 83360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6814', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6815', 'nonce': 83360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6812', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6813', 'nonce': 83360}]}, 'public_addr': '172.21.15.16:6809/83360', 'cluster_addr': '172.21.15.16:6811/83360', 'heartbeat_back_addr': '172.21.15.16:6815/83360', 'heartbeat_front_addr': '172.21.15.16:6813/83360', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '99f8ea0a-5a71-4abe-b087-8e891fd11d1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 14, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6816', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6817', 'nonce': 83361}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6818', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6819', 'nonce': 83361}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6822', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6823', 'nonce': 83361}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6820', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6821', 'nonce': 83361}]}, 'public_addr': '172.21.15.16:6817/83361', 'cluster_addr': '172.21.15.16:6819/83361', 'heartbeat_back_addr': '172.21.15.16:6823/83361', 'heartbeat_front_addr': '172.21.15.16:6821/83361', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd285b4d5-0bef-421a-b84a-943e485b10c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6808', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6809', 'nonce': 83350}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6810', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6811', 'nonce': 83350}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6814', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6815', 'nonce': 83350}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6812', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6813', 'nonce': 83350}]}, 'public_addr': '172.21.15.22:6809/83350', 'cluster_addr': '172.21.15.22:6811/83350', 'heartbeat_back_addr': '172.21.15.22:6815/83350', 'heartbeat_front_addr': '172.21.15.22:6813/83350', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7ba0b0db-e407-4c63-8278-aab09f3082c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.23297119140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6800', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6801', 'nonce': 83351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6802', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6803', 'nonce': 83351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6806', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6807', 'nonce': 83351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6804', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6805', 'nonce': 83351}]}, 'public_addr': '172.21.15.22:6801/83351', 'cluster_addr': '172.21.15.22:6803/83351', 'heartbeat_back_addr': '172.21.15.22:6807/83351', 'heartbeat_front_addr': '172.21.15.22:6805/83351', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '29663bfa-f6fe-4ce7-b45a-a9b7b4424c42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6816', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6817', 'nonce': 83352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6818', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6819', 'nonce': 83352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6822', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6823', 'nonce': 83352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6820', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6821', 'nonce': 83352}]}, 'public_addr': '172.21.15.22:6817/83352', 'cluster_addr': '172.21.15.22:6819/83352', 'heartbeat_back_addr': '172.21.15.22:6823/83352', 'heartbeat_front_addr': '172.21.15.22:6821/83352', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.001178+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.184181+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.274854+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.186217+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.247442+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.220075+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.2', 'osds': [1, 5]}, {'pgid': '2.4', 'osds': [5, 0]}], 'pg_upmap_items': [{'pgid': '2.7', 'mappings': [{'from': 4, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 2, 'snaps': [{'begin': 2, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-07-02T14:02:44.703 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:02:44.707 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:44.707 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:44.720 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:02:44.725 INFO:tasks.workunit.client.0.smithi163.stderr: 93250K .......... .......... .......... .......... .......... 55% 18.8M 4s 2023-07-02T14:02:44.725 INFO:tasks.workunit.client.0.smithi163.stderr: 93300K .......... .......... .......... .......... .......... 55% 23.5M 4s 2023-07-02T14:02:44.726 INFO:tasks.workunit.client.0.smithi163.stderr: 93350K .......... .......... .......... .......... .......... 55% 17.2M 4s 2023-07-02T14:02:44.726 INFO:tasks.workunit.client.0.smithi163.stderr: 93400K .......... .......... .......... .......... .......... 55% 22.7M 4s 2023-07-02T14:02:44.726 INFO:tasks.workunit.client.0.smithi163.stderr: 93450K .......... .......... .......... .......... .......... 55% 31.5M 4s 2023-07-02T14:02:44.726 INFO:tasks.workunit.client.0.smithi163.stderr: 93500K .......... .......... .......... .......... .......... 55% 17.7M 4s 2023-07-02T14:02:44.726 INFO:tasks.workunit.client.0.smithi163.stderr: 93550K .......... .......... .......... .......... .......... 55% 16.7M 4s 2023-07-02T14:02:44.726 INFO:tasks.workunit.client.0.smithi163.stderr: 93600K .......... .......... .......... .......... .......... 55% 15.9M 4s 2023-07-02T14:02:44.726 INFO:tasks.workunit.client.0.smithi163.stderr: 93650K .......... .......... .......... .......... .......... 55% 24.5M 4s 2023-07-02T14:02:44.727 INFO:tasks.workunit.client.0.smithi163.stderr: 93700K .......... .......... .......... .......... .......... 55% 17.8M 4s 2023-07-02T14:02:44.727 INFO:tasks.workunit.client.0.smithi163.stderr: 93750K .......... .......... .......... .......... .......... 55% 29.0M 4s 2023-07-02T14:02:44.727 INFO:tasks.workunit.client.0.smithi163.stderr: 93800K .......... .......... .......... .......... .......... 55% 18.2M 4s 2023-07-02T14:02:44.727 INFO:tasks.workunit.client.0.smithi163.stderr: 93850K .......... .......... .......... .......... .......... 55% 21.9M 4s 2023-07-02T14:02:44.727 INFO:tasks.workunit.client.0.smithi163.stderr: 93900K .......... .......... .......... .......... .......... 55% 11.5M 4s 2023-07-02T14:02:44.727 INFO:tasks.workunit.client.0.smithi163.stderr: 93950K .......... .......... .......... .......... .......... 55% 29.3M 4s 2023-07-02T14:02:44.727 INFO:tasks.workunit.client.0.smithi163.stderr: 94000K .......... .......... .......... .......... .......... 55% 28.3M 4s 2023-07-02T14:02:44.727 INFO:tasks.workunit.client.0.smithi163.stderr: 94050K .......... .......... .......... .......... .......... 55% 23.1M 4s 2023-07-02T14:02:44.728 INFO:tasks.workunit.client.0.smithi163.stderr: 94100K .......... .......... .......... .......... .......... 55% 17.3M 4s 2023-07-02T14:02:44.728 INFO:tasks.workunit.client.0.smithi163.stderr: 94150K .......... .......... .......... .......... .......... 55% 26.4M 4s 2023-07-02T14:02:44.728 INFO:tasks.workunit.client.0.smithi163.stderr: 94200K .......... .......... .......... .......... .......... 55% 24.4M 4s 2023-07-02T14:02:44.728 INFO:tasks.workunit.client.0.smithi163.stderr: 94250K .......... .......... .......... .......... .......... 55% 17.7M 4s 2023-07-02T14:02:44.728 INFO:tasks.workunit.client.0.smithi163.stderr: 94300K .......... .......... .......... .......... .......... 55% 16.6M 4s 2023-07-02T14:02:44.728 INFO:tasks.workunit.client.0.smithi163.stderr: 94350K .......... .......... .......... .......... .......... 56% 21.1M 4s 2023-07-02T14:02:44.728 INFO:tasks.workunit.client.0.smithi163.stderr: 94400K .......... .......... .......... .......... .......... 56% 30.9M 4s 2023-07-02T14:02:44.728 INFO:tasks.workunit.client.0.smithi163.stderr: 94450K .......... .......... .......... .......... .......... 56% 15.6M 4s 2023-07-02T14:02:44.729 INFO:tasks.workunit.client.0.smithi163.stderr: 94500K .......... .......... .......... .......... .......... 56% 25.3M 4s 2023-07-02T14:02:44.729 INFO:tasks.workunit.client.0.smithi163.stderr: 94550K .......... .......... .......... .......... .......... 56% 25.0M 4s 2023-07-02T14:02:44.729 INFO:tasks.workunit.client.0.smithi163.stderr: 94600K .......... .......... .......... .......... .......... 56% 14.1M 4s 2023-07-02T14:02:44.729 INFO:tasks.workunit.client.0.smithi163.stderr: 94650K .......... .......... .......... .......... .......... 56% 29.8M 4s 2023-07-02T14:02:44.729 INFO:tasks.workunit.client.0.smithi163.stderr: 94700K .......... .......... .......... .......... .......... 56% 25.7M 4s 2023-07-02T14:02:44.729 INFO:tasks.workunit.client.0.smithi163.stderr: 94750K .......... .......... .......... .......... .......... 56% 15.6M 4s 2023-07-02T14:02:44.729 INFO:tasks.workunit.client.0.smithi163.stderr: 94800K .......... .......... .......... .......... .......... 56% 20.2M 4s 2023-07-02T14:02:44.730 INFO:tasks.workunit.client.0.smithi163.stderr: 94850K .......... .......... .......... .......... .......... 56% 16.2M 4s 2023-07-02T14:02:44.730 INFO:tasks.workunit.client.0.smithi163.stderr: 94900K .......... .......... .......... .......... .......... 56% 17.3M 4s 2023-07-02T14:02:44.730 INFO:tasks.workunit.client.0.smithi163.stderr: 94950K .......... .......... .......... .......... .......... 56% 18.4M 4s 2023-07-02T14:02:44.730 INFO:tasks.workunit.client.0.smithi163.stderr: 95000K .......... .......... .......... .......... .......... 56% 31.0M 4s 2023-07-02T14:02:44.730 INFO:tasks.workunit.client.0.smithi163.stderr: 95050K .......... .......... .......... .......... .......... 56% 21.8M 4s 2023-07-02T14:02:44.730 INFO:tasks.workunit.client.0.smithi163.stderr: 95100K .......... .......... .......... .......... .......... 56% 34.9M 4s 2023-07-02T14:02:44.730 INFO:tasks.workunit.client.0.smithi163.stderr: 95150K .......... .......... .......... .......... .......... 56% 8.30M 4s 2023-07-02T14:02:44.730 INFO:tasks.workunit.client.0.smithi163.stderr: 95200K .......... .......... .......... .......... .......... 56% 22.4M 4s 2023-07-02T14:02:44.731 INFO:tasks.workunit.client.0.smithi163.stderr: 95250K .......... .......... .......... .......... .......... 56% 33.6M 4s 2023-07-02T14:02:44.731 INFO:tasks.workunit.client.0.smithi163.stderr: 95300K .......... .......... .......... .......... .......... 56% 24.3M 4s 2023-07-02T14:02:44.731 INFO:tasks.workunit.client.0.smithi163.stderr: 95350K .......... .......... .......... .......... .......... 56% 14.8M 4s 2023-07-02T14:02:44.731 INFO:tasks.workunit.client.0.smithi163.stderr: 95400K .......... .......... .......... .......... .......... 56% 25.4M 4s 2023-07-02T14:02:44.731 INFO:tasks.workunit.client.0.smithi163.stderr: 95450K .......... .......... .......... .......... .......... 56% 21.6M 4s 2023-07-02T14:02:44.731 INFO:tasks.workunit.client.0.smithi163.stderr: 95500K .......... .......... .......... .......... .......... 56% 23.8M 4s 2023-07-02T14:02:44.731 INFO:tasks.workunit.client.0.smithi163.stderr: 95550K .......... .......... .......... .......... .......... 56% 18.8M 4s 2023-07-02T14:02:44.731 INFO:tasks.workunit.client.0.smithi163.stderr: 95600K .......... .......... .......... .......... .......... 56% 19.4M 4s 2023-07-02T14:02:44.732 INFO:tasks.workunit.client.0.smithi163.stderr: 95650K .......... .......... .......... .......... .......... 56% 34.7M 4s 2023-07-02T14:02:44.732 INFO:tasks.workunit.client.0.smithi163.stderr: 95700K .......... .......... .......... .......... .......... 56% 13.8M 4s 2023-07-02T14:02:44.732 INFO:tasks.workunit.client.0.smithi163.stderr: 95750K .......... .......... .......... .......... .......... 56% 26.5M 4s 2023-07-02T14:02:44.732 INFO:tasks.workunit.client.0.smithi163.stderr: 95800K .......... .......... .......... .......... .......... 56% 19.4M 4s 2023-07-02T14:02:44.733 INFO:tasks.workunit.client.0.smithi163.stderr: 95850K .......... .......... .......... .......... .......... 56% 18.1M 4s 2023-07-02T14:02:44.734 INFO:tasks.workunit.client.0.smithi163.stderr: 95900K .......... .......... .......... .......... .......... 56% 30.6M 4s 2023-07-02T14:02:44.734 INFO:tasks.workunit.client.0.smithi163.stderr: 95950K .......... .......... .......... .......... .......... 56% 22.2M 4s 2023-07-02T14:02:44.734 INFO:tasks.workunit.client.0.smithi163.stderr: 96000K .......... .......... .......... .......... .......... 56% 13.1M 4s 2023-07-02T14:02:44.736 INFO:tasks.workunit.client.0.smithi163.stderr: 96050K .......... .......... .......... .......... .......... 57% 27.4M 4s 2023-07-02T14:02:44.738 INFO:tasks.workunit.client.0.smithi163.stderr: 96100K .......... .......... .......... .......... .......... 57% 20.6M 4s 2023-07-02T14:02:44.740 INFO:tasks.workunit.client.0.smithi163.stderr: 96150K .......... .......... .......... .......... .......... 57% 29.7M 4s 2023-07-02T14:02:44.743 INFO:tasks.workunit.client.0.smithi163.stderr: 96200K .......... .......... .......... .......... .......... 57% 16.1M 4s 2023-07-02T14:02:44.745 INFO:tasks.workunit.client.0.smithi163.stderr: 96250K .......... .......... .......... .......... .......... 57% 29.1M 4s 2023-07-02T14:02:44.748 INFO:tasks.workunit.client.0.smithi163.stderr: 96300K .......... .......... .......... .......... .......... 57% 17.4M 4s 2023-07-02T14:02:44.750 INFO:tasks.workunit.client.0.smithi163.stderr: 96350K .......... .......... .......... .......... .......... 57% 17.9M 4s 2023-07-02T14:02:44.754 INFO:tasks.workunit.client.0.smithi163.stderr: 96400K .......... .......... .......... .......... .......... 57% 16.1M 4s 2023-07-02T14:02:44.756 INFO:tasks.workunit.client.0.smithi163.stderr: 96450K .......... .......... .......... .......... .......... 57% 21.7M 4s 2023-07-02T14:02:44.757 INFO:tasks.workunit.client.0.smithi163.stderr: 96500K .......... .......... .......... .......... .......... 57% 32.8M 4s 2023-07-02T14:02:44.759 INFO:tasks.workunit.client.0.smithi163.stderr: 96550K .......... .......... .......... .......... .......... 57% 20.7M 4s 2023-07-02T14:02:44.763 INFO:tasks.workunit.client.0.smithi163.stderr: 96600K .......... .......... .......... .......... .......... 57% 13.9M 4s 2023-07-02T14:02:44.764 INFO:tasks.workunit.client.0.smithi163.stderr: 96650K .......... .......... .......... .......... .......... 57% 40.0M 4s 2023-07-02T14:02:44.767 INFO:tasks.workunit.client.0.smithi163.stderr: 96700K .......... .......... .......... .......... .......... 57% 14.9M 4s 2023-07-02T14:02:44.771 INFO:tasks.workunit.client.0.smithi163.stderr: 96750K .......... .......... .......... .......... .......... 57% 13.4M 4s 2023-07-02T14:02:44.772 INFO:tasks.workunit.client.0.smithi163.stderr: 96800K .......... .......... .......... .......... .......... 57% 41.6M 4s 2023-07-02T14:02:44.775 INFO:tasks.workunit.client.0.smithi163.stderr: 96850K .......... .......... .......... .......... .......... 57% 22.3M 4s 2023-07-02T14:02:44.777 INFO:tasks.workunit.client.0.smithi163.stderr: 96900K .......... .......... .......... .......... .......... 57% 19.6M 4s 2023-07-02T14:02:44.779 INFO:tasks.workunit.client.0.smithi163.stderr: 96950K .......... .......... .......... .......... .......... 57% 20.1M 4s 2023-07-02T14:02:44.781 INFO:tasks.workunit.client.0.smithi163.stderr: 97000K .......... .......... .......... .......... .......... 57% 29.0M 4s 2023-07-02T14:02:44.784 INFO:tasks.workunit.client.0.smithi163.stderr: 97050K .......... .......... .......... .......... .......... 57% 18.8M 4s 2023-07-02T14:02:44.786 INFO:tasks.workunit.client.0.smithi163.stderr: 97100K .......... .......... .......... .......... .......... 57% 18.7M 4s 2023-07-02T14:02:44.789 INFO:tasks.workunit.client.0.smithi163.stderr: 97150K .......... .......... .......... .......... .......... 57% 16.4M 4s 2023-07-02T14:02:44.791 INFO:tasks.workunit.client.0.smithi163.stderr: 97200K .......... .......... .......... .......... .......... 57% 30.7M 4s 2023-07-02T14:02:44.795 INFO:tasks.workunit.client.0.smithi163.stderr: 97250K .......... .......... .......... .......... .......... 57% 12.9M 4s 2023-07-02T14:02:44.796 INFO:tasks.workunit.client.0.smithi163.stderr: 97300K .......... .......... .......... .......... .......... 57% 39.8M 4s 2023-07-02T14:02:44.798 INFO:tasks.workunit.client.0.smithi163.stderr: 97350K .......... .......... .......... .......... .......... 57% 21.4M 4s 2023-07-02T14:02:44.801 INFO:tasks.workunit.client.0.smithi163.stderr: 97400K .......... .......... .......... .......... .......... 57% 17.2M 4s 2023-07-02T14:02:44.802 INFO:tasks.workunit.client.0.smithi163.stderr: 97450K .......... .......... .......... .......... .......... 57% 31.1M 4s 2023-07-02T14:02:44.803 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:44.802+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:44.804 INFO:tasks.workunit.client.0.smithi163.stderr: 97500K .......... .......... .......... .......... .......... 57% 24.4M 4s 2023-07-02T14:02:44.807 INFO:tasks.workunit.client.0.smithi163.stderr: 97550K .......... .......... .......... .......... .......... 57% 18.3M 4s 2023-07-02T14:02:44.811 INFO:tasks.workunit.client.0.smithi163.stderr: 97600K .......... .......... .......... .......... .......... 57% 14.6M 4s 2023-07-02T14:02:44.813 INFO:tasks.workunit.client.0.smithi163.stderr: 97650K .......... .......... .......... .......... .......... 57% 19.2M 4s 2023-07-02T14:02:44.815 INFO:tasks.workunit.client.0.smithi163.stderr: 97700K .......... .......... .......... .......... .......... 57% 25.1M 4s 2023-07-02T14:02:44.818 INFO:tasks.workunit.client.0.smithi163.stderr: 97750K .......... .......... .......... .......... .......... 58% 17.1M 4s 2023-07-02T14:02:44.819 INFO:tasks.workunit.client.0.smithi163.stderr: 97800K .......... .......... .......... .......... .......... 58% 39.0M 4s 2023-07-02T14:02:44.822 INFO:tasks.workunit.client.0.smithi163.stderr: 97850K .......... .......... .......... .......... .......... 58% 18.0M 4s 2023-07-02T14:02:44.824 INFO:tasks.workunit.client.0.smithi163.stderr: 97900K .......... .......... .......... .......... .......... 58% 24.0M 4s 2023-07-02T14:02:44.826 INFO:tasks.workunit.client.0.smithi163.stderr: 97950K .......... .......... .......... .......... .......... 58% 20.3M 4s 2023-07-02T14:02:44.829 INFO:tasks.workunit.client.0.smithi163.stderr: 98000K .......... .......... .......... .......... .......... 58% 14.6M 4s 2023-07-02T14:02:44.831 INFO:tasks.workunit.client.0.smithi163.stderr: 98050K .......... .......... .......... .......... .......... 58% 26.5M 4s 2023-07-02T14:02:44.834 INFO:tasks.workunit.client.0.smithi163.stderr: 98100K .......... .......... .......... .......... .......... 58% 21.9M 4s 2023-07-02T14:02:44.837 INFO:tasks.workunit.client.0.smithi163.stderr: 98150K .......... .......... .......... .......... .......... 58% 16.4M 4s 2023-07-02T14:02:44.838 INFO:tasks.workunit.client.0.smithi163.stderr: 98200K .......... .......... .......... .......... .......... 58% 29.4M 4s 2023-07-02T14:02:44.841 INFO:tasks.workunit.client.0.smithi163.stderr: 98250K .......... .......... .......... .......... .......... 58% 16.8M 4s 2023-07-02T14:02:44.843 INFO:tasks.workunit.client.0.smithi163.stderr: 98300K .......... .......... .......... .......... .......... 58% 30.7M 4s 2023-07-02T14:02:44.846 INFO:tasks.workunit.client.0.smithi163.stderr: 98350K .......... .......... .......... .......... .......... 58% 14.2M 4s 2023-07-02T14:02:44.849 INFO:tasks.workunit.client.0.smithi163.stderr: 98400K .......... .......... .......... .......... .......... 58% 18.3M 4s 2023-07-02T14:02:44.851 INFO:tasks.workunit.client.0.smithi163.stderr: 98450K .......... .......... .......... .......... .......... 58% 23.3M 4s 2023-07-02T14:02:44.854 INFO:tasks.workunit.client.0.smithi163.stderr: 98500K .......... .......... .......... .......... .......... 58% 17.0M 4s 2023-07-02T14:02:44.855 INFO:tasks.workunit.client.0.smithi163.stderr: 98550K .......... .......... .......... .......... .......... 58% 30.1M 4s 2023-07-02T14:02:44.858 INFO:tasks.workunit.client.0.smithi163.stderr: 98600K .......... .......... .......... .......... .......... 58% 21.2M 4s 2023-07-02T14:02:44.861 INFO:tasks.workunit.client.0.smithi163.stderr: 98650K .......... .......... .......... .......... .......... 58% 13.6M 4s 2023-07-02T14:02:44.863 INFO:tasks.workunit.client.0.smithi163.stderr: 98700K .......... .......... .......... .......... .......... 58% 24.4M 4s 2023-07-02T14:02:44.866 INFO:tasks.workunit.client.0.smithi163.stderr: 98750K .......... .......... .......... .......... .......... 58% 17.9M 4s 2023-07-02T14:02:44.869 INFO:tasks.workunit.client.0.smithi163.stderr: 98800K .......... .......... .......... .......... .......... 58% 15.5M 4s 2023-07-02T14:02:44.870 INFO:tasks.workunit.client.0.smithi163.stderr: 98850K .......... .......... .......... .......... .......... 58% 41.4M 4s 2023-07-02T14:02:44.873 INFO:tasks.workunit.client.0.smithi163.stderr: 98900K .......... .......... .......... .......... .......... 58% 18.1M 4s 2023-07-02T14:02:44.875 INFO:tasks.workunit.client.0.smithi163.stderr: 98950K .......... .......... .......... .......... .......... 58% 27.4M 4s 2023-07-02T14:02:44.878 INFO:tasks.workunit.client.0.smithi163.stderr: 99000K .......... .......... .......... .......... .......... 58% 17.2M 4s 2023-07-02T14:02:44.879 INFO:tasks.workunit.client.0.smithi163.stderr: 99050K .......... .......... .......... .......... .......... 58% 33.4M 3s 2023-07-02T14:02:44.882 INFO:tasks.workunit.client.0.smithi163.stderr: 99100K .......... .......... .......... .......... .......... 58% 17.7M 3s 2023-07-02T14:02:44.884 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:44.884 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:44.886 INFO:tasks.workunit.client.0.smithi163.stderr: 99150K .......... .......... .......... .......... .......... 58% 15.0M 3s 2023-07-02T14:02:44.888 INFO:tasks.workunit.client.0.smithi163.stderr: 99200K .......... .......... .......... .......... .......... 58% 18.7M 3s 2023-07-02T14:02:44.889 INFO:tasks.workunit.client.0.smithi163.stderr: 99250K .......... .......... .......... .......... .......... 58% 29.7M 3s 2023-07-02T14:02:44.891 INFO:tasks.workunit.client.0.smithi163.stderr: 99300K .......... .......... .......... .......... .......... 58% 32.9M 3s 2023-07-02T14:02:44.894 INFO:tasks.workunit.client.0.smithi163.stderr: 99350K .......... .......... .......... .......... .......... 58% 18.3M 3s 2023-07-02T14:02:44.895 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:02:44.896 INFO:tasks.workunit.client.0.smithi163.stderr: 99400K .......... .......... .......... .......... .......... 58% 18.6M 3s 2023-07-02T14:02:44.899 INFO:tasks.workunit.client.0.smithi163.stderr: 99450K .......... .......... .......... .......... .......... 59% 21.1M 3s 2023-07-02T14:02:44.901 INFO:tasks.workunit.client.0.smithi163.stderr: 99500K .......... .......... .......... .......... .......... 59% 19.0M 3s 2023-07-02T14:02:44.902 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:44.901+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:44.904 INFO:tasks.workunit.client.0.smithi163.stderr: 99550K .......... .......... .......... .......... .......... 59% 15.2M 3s 2023-07-02T14:02:44.906 INFO:tasks.workunit.client.0.smithi163.stderr: 99600K .......... .......... .......... .......... .......... 59% 32.5M 3s 2023-07-02T14:02:44.908 INFO:tasks.workunit.client.0.smithi163.stderr: 99650K .......... .......... .......... .......... .......... 59% 20.9M 3s 2023-07-02T14:02:44.911 INFO:tasks.workunit.client.0.smithi163.stderr: 99700K .......... .......... .......... .......... .......... 59% 15.9M 3s 2023-07-02T14:02:44.913 INFO:tasks.workunit.client.0.smithi163.stderr: 99750K .......... .......... .......... .......... .......... 59% 27.3M 3s 2023-07-02T14:02:44.915 INFO:tasks.workunit.client.0.smithi163.stderr: 99800K .......... .......... .......... .......... .......... 59% 25.4M 3s 2023-07-02T14:02:44.918 INFO:tasks.workunit.client.0.smithi163.stderr: 99850K .......... .......... .......... .......... .......... 59% 21.0M 3s 2023-07-02T14:02:44.921 INFO:tasks.workunit.client.0.smithi163.stderr: 99900K .......... .......... .......... .......... .......... 59% 15.1M 3s 2023-07-02T14:02:44.923 INFO:tasks.workunit.client.0.smithi163.stderr: 99950K .......... .......... .......... .......... .......... 59% 19.7M 3s 2023-07-02T14:02:44.925 INFO:tasks.workunit.client.0.smithi163.stderr:100000K .......... .......... .......... .......... .......... 59% 28.2M 3s 2023-07-02T14:02:44.928 INFO:tasks.workunit.client.0.smithi163.stderr:100050K .......... .......... .......... .......... .......... 59% 15.9M 3s 2023-07-02T14:02:44.930 INFO:tasks.workunit.client.0.smithi163.stderr:100100K .......... .......... .......... .......... .......... 59% 25.6M 3s 2023-07-02T14:02:44.932 INFO:tasks.workunit.client.0.smithi163.stderr:100150K .......... .......... .......... .......... .......... 59% 22.6M 3s 2023-07-02T14:02:44.934 INFO:tasks.workunit.client.0.smithi163.stderr:100200K .......... .......... .......... .......... .......... 59% 23.1M 3s 2023-07-02T14:02:44.937 INFO:tasks.workunit.client.0.smithi163.stderr:100250K .......... .......... .......... .......... .......... 59% 15.1M 3s 2023-07-02T14:02:44.939 INFO:tasks.workunit.client.0.smithi163.stderr:100300K .......... .......... .......... .......... .......... 59% 33.8M 3s 2023-07-02T14:02:44.942 INFO:tasks.workunit.client.0.smithi163.stderr:100350K .......... .......... .......... .......... .......... 59% 16.7M 3s 2023-07-02T14:02:44.944 INFO:tasks.workunit.client.0.smithi163.stderr:100400K .......... .......... .......... .......... .......... 59% 18.5M 3s 2023-07-02T14:02:44.947 INFO:tasks.workunit.client.0.smithi163.stderr:100450K .......... .......... .......... .......... .......... 59% 18.1M 3s 2023-07-02T14:02:44.949 INFO:tasks.workunit.client.0.smithi163.stderr:100500K .......... .......... .......... .......... .......... 59% 27.2M 3s 2023-07-02T14:02:44.952 INFO:tasks.workunit.client.0.smithi163.stderr:100550K .......... .......... .......... .......... .......... 59% 15.5M 3s 2023-07-02T14:02:44.954 INFO:tasks.workunit.client.0.smithi163.stderr:100600K .......... .......... .......... .......... .......... 59% 25.1M 3s 2023-07-02T14:02:44.956 INFO:tasks.workunit.client.0.smithi163.stderr:100650K .......... .......... .......... .......... .......... 59% 21.7M 3s 2023-07-02T14:02:44.957 INFO:tasks.workunit.client.0.smithi163.stderr:100700K .......... .......... .......... .......... .......... 59% 40.4M 3s 2023-07-02T14:02:44.961 INFO:tasks.workunit.client.0.smithi163.stderr:100750K .......... .......... .......... .......... .......... 59% 14.1M 3s 2023-07-02T14:02:44.964 INFO:tasks.workunit.client.0.smithi163.stderr:100800K .......... .......... .......... .......... .......... 59% 16.5M 3s 2023-07-02T14:02:44.966 INFO:tasks.workunit.client.0.smithi163.stderr:100850K .......... .......... .......... .......... .......... 59% 22.6M 3s 2023-07-02T14:02:44.968 INFO:tasks.workunit.client.0.smithi163.stderr:100900K .......... .......... .......... .......... .......... 59% 24.7M 3s 2023-07-02T14:02:44.971 INFO:tasks.workunit.client.0.smithi163.stderr:100950K .......... .......... .......... .......... .......... 59% 17.0M 3s 2023-07-02T14:02:44.972 INFO:tasks.workunit.client.0.smithi163.stderr:101000K .......... .......... .......... .......... .......... 59% 29.4M 3s 2023-07-02T14:02:44.976 INFO:tasks.workunit.client.0.smithi163.stderr:101050K .......... .......... .......... .......... .......... 59% 14.1M 3s 2023-07-02T14:02:44.977 INFO:tasks.workunit.client.0.smithi163.stderr:101100K .......... .......... .......... .......... .......... 60% 49.4M 3s 2023-07-02T14:02:44.980 INFO:tasks.workunit.client.0.smithi163.stderr:101150K .......... .......... .......... .......... .......... 60% 14.3M 3s 2023-07-02T14:02:44.982 INFO:tasks.workunit.client.0.smithi163.stderr:101200K .......... .......... .......... .......... .......... 60% 24.5M 3s 2023-07-02T14:02:44.984 INFO:tasks.workunit.client.0.smithi163.stderr:101250K .......... .......... .......... .......... .......... 60% 23.8M 3s 2023-07-02T14:02:44.987 INFO:tasks.workunit.client.0.smithi163.stderr:101300K .......... .......... .......... .......... .......... 60% 18.6M 3s 2023-07-02T14:02:44.990 INFO:tasks.workunit.client.0.smithi163.stderr:101350K .......... .......... .......... .......... .......... 60% 15.5M 3s 2023-07-02T14:02:44.992 INFO:tasks.workunit.client.0.smithi163.stderr:101400K .......... .......... .......... .......... .......... 60% 31.0M 3s 2023-07-02T14:02:44.995 INFO:tasks.workunit.client.0.smithi163.stderr:101450K .......... .......... .......... .......... .......... 60% 16.2M 3s 2023-07-02T14:02:44.996 INFO:tasks.workunit.client.0.smithi163.stderr:101500K .......... .......... .......... .......... .......... 60% 30.8M 3s 2023-07-02T14:02:44.999 INFO:tasks.workunit.client.0.smithi163.stderr:101550K .......... .......... .......... .......... .......... 60% 16.1M 3s 2023-07-02T14:02:45.001 INFO:tasks.workunit.client.0.smithi163.stderr:101600K .......... .......... .......... .......... .......... 60% 23.6M 3s 2023-07-02T14:02:45.002 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:45.001+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:45.004 INFO:tasks.workunit.client.0.smithi163.stderr:101650K .......... .......... .......... .......... .......... 60% 21.6M 3s 2023-07-02T14:02:45.006 INFO:tasks.workunit.client.0.smithi163.stderr:101700K .......... .......... .......... .......... .......... 60% 18.3M 3s 2023-07-02T14:02:45.008 INFO:tasks.workunit.client.0.smithi163.stderr:101750K .......... .......... .......... .......... .......... 60% 26.4M 3s 2023-07-02T14:02:45.011 INFO:tasks.workunit.client.0.smithi163.stderr:101800K .......... .......... .......... .......... .......... 60% 15.3M 3s 2023-07-02T14:02:45.013 INFO:tasks.workunit.client.0.smithi163.stderr:101850K .......... .......... .......... .......... .......... 60% 29.0M 3s 2023-07-02T14:02:45.015 INFO:tasks.workunit.client.0.smithi163.stderr:101900K .......... .......... .......... .......... .......... 60% 24.1M 3s 2023-07-02T14:02:45.018 INFO:tasks.workunit.client.0.smithi163.stderr:101950K .......... .......... .......... .......... .......... 60% 15.4M 3s 2023-07-02T14:02:45.022 INFO:tasks.workunit.client.0.smithi163.stderr:102000K .......... .......... .......... .......... .......... 60% 14.3M 3s 2023-07-02T14:02:45.023 INFO:tasks.workunit.client.0.smithi163.stderr:102050K .......... .......... .......... .......... .......... 60% 29.5M 3s 2023-07-02T14:02:45.026 INFO:tasks.workunit.client.0.smithi163.stderr:102100K .......... .......... .......... .......... .......... 60% 19.4M 3s 2023-07-02T14:02:45.028 INFO:tasks.workunit.client.0.smithi163.stderr:102150K .......... .......... .......... .......... .......... 60% 20.0M 3s 2023-07-02T14:02:45.030 INFO:tasks.workunit.client.0.smithi163.stderr:102200K .......... .......... .......... .......... .......... 60% 22.5M 3s 2023-07-02T14:02:45.033 INFO:tasks.workunit.client.0.smithi163.stderr:102250K .......... .......... .......... .......... .......... 60% 23.2M 3s 2023-07-02T14:02:45.035 INFO:tasks.workunit.client.0.smithi163.stderr:102300K .......... .......... .......... .......... .......... 60% 17.9M 3s 2023-07-02T14:02:45.038 INFO:tasks.workunit.client.0.smithi163.stderr:102350K .......... .......... .......... .......... .......... 60% 16.2M 3s 2023-07-02T14:02:45.041 INFO:tasks.workunit.client.0.smithi163.stderr:102400K .......... .......... .......... .......... .......... 60% 20.8M 3s 2023-07-02T14:02:45.042 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:02:45.043 INFO:tasks.workunit.client.0.smithi163.stderr:102450K .......... .......... .......... .......... .......... 60% 24.4M 3s 2023-07-02T14:02:45.044 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:02:45.046 INFO:tasks.workunit.client.0.smithi163.stderr:102500K .......... .......... .......... .......... .......... 60% 17.2M 3s 2023-07-02T14:02:45.047 INFO:tasks.workunit.client.0.smithi163.stderr:102550K .......... .......... .......... .......... .......... 60% 25.2M 3s 2023-07-02T14:02:45.050 INFO:tasks.workunit.client.0.smithi163.stderr:102600K .......... .......... .......... .......... .......... 60% 20.1M 3s 2023-07-02T14:02:45.052 INFO:tasks.workunit.client.0.smithi163.stderr:102650K .......... .......... .......... .......... .......... 60% 27.9M 3s 2023-07-02T14:02:45.055 INFO:tasks.workunit.client.0.smithi163.stderr:102700K .......... .......... .......... .......... .......... 60% 16.2M 3s 2023-07-02T14:02:45.055 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2023-07-02T14:02:44.242085+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":87692535,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":256,"num_read_kb":281,"num_write":593,"num_write_kb":81455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":565,"ondisk_log_size":565,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":145368,"kb_used_data":113576,"kb_used_omap":0,"kb_used_meta":31744,"kb_avail":566085672,"statfs":{"total":579820584960,"available":579671728128,"internally_reserved":0,"allocated":116301824,"data_stored":114549680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32505856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.180076"},"pg_stats":[{"pgid":"2.7","version":"40'90","reported_seq":146,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-07-02T14:02:40.295967+0000","last_change":"2023-07-02T14:02:39.068336+0000","last_active":"2023-07-02T14:02:40.295967+0000","last_peered":"2023-07-02T14:02:40.295967+0000","last_clean":"2023-07-02T14:02:40.295967+0000","last_became_active":"2023-07-02T14:02:01.725984+0000","last_became_peered":"2023-07-02T14:02:01.725984+0000","last_unstale":"2023-07-02T14:02:40.295967+0000","last_undegraded":"2023-07-02T14:02:40.295967+0000","last_fullsized":"2023-07-02T14:02:40.295967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":90,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12469282,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":25,"num_write":87,"num_write_kb":12181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"40'71","reported_seq":109,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-07-02T14:02:40.238697+0000","last_change":"2023-07-02T14:02:39.062139+0000","last_active":"2023-07-02T14:02:40.238697+0000","last_peered":"2023-07-02T14:02:40.238697+0000","last_clean":"2023-07-02T14:02:40.238697+0000","last_became_active":"2023-07-02T14:01:49.293782+0000","last_became_peered":"2023-07-02T14:01:49.293782+0000","last_unstale":"2023-07-02T14:02:40.238697+0000","last_undegraded":"2023-07-02T14:02:40.238697+0000","last_fullsized":"2023-07-02T14:02:40.238697+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14565376,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":71,"num_write_kb":14228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"40'16","reported_seq":59,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-07-02T14:02:39.438163+0000","last_change":"2023-07-02T14:02:39.069176+0000","last_active":"2023-07-02T14:02:39.438163+0000","last_peered":"2023-07-02T14:02:39.438163+0000","last_clean":"2023-07-02T14:02:39.438163+0000","last_became_active":"2023-07-02T14:01:49.292790+0000","last_became_peered":"2023-07-02T14:01:49.292790+0000","last_unstale":"2023-07-02T14:02:39.438163+0000","last_undegraded":"2023-07-02T14:02:39.438163+0000","last_fullsized":"2023-07-02T14:02:39.438163+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12582912,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":69,"num_write":16,"num_write_kb":8257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"40'94","reported_seq":136,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-07-02T14:02:40.243297+0000","last_change":"2023-07-02T14:02:39.069112+0000","last_active":"2023-07-02T14:02:40.243297+0000","last_peered":"2023-07-02T14:02:40.243297+0000","last_clean":"2023-07-02T14:02:40.243297+0000","last_became_active":"2023-07-02T14:01:49.293608+0000","last_became_peered":"2023-07-02T14:01:49.293608+0000","last_unstale":"2023-07-02T14:02:40.243297+0000","last_undegraded":"2023-07-02T14:02:40.243297+0000","last_fullsized":"2023-07-02T14:02:40.243297+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":94,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6234290,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":199,"num_read_kb":166,"num_write":125,"num_write_kb":6152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":34,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-07-02T14:02:38.866784+0000","last_change":"2023-07-02T14:01:47.288970+0000","last_active":"2023-07-02T14:02:38.866784+0000","last_peered":"2023-07-02T14:02:38.866784+0000","last_clean":"2023-07-02T14:02:38.866784+0000","last_became_active":"2023-07-02T14:01:47.288610+0000","last_became_peered":"2023-07-02T14:01:47.288610+0000","last_unstale":"2023-07-02T14:02:38.866784+0000","last_undegraded":"2023-07-02T14:02:38.866784+0000","last_fullsized":"2023-07-02T14:02:38.866784+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"40'64","reported_seq":99,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-07-02T14:02:40.243038+0000","last_change":"2023-07-02T14:02:39.064116+0000","last_active":"2023-07-02T14:02:40.243038+0000","last_peered":"2023-07-02T14:02:40.243038+0000","last_clean":"2023-07-02T14:02:40.243038+0000","last_became_active":"2023-07-02T14:01:49.297946+0000","last_became_peered":"2023-07-02T14:01:49.297946+0000","last_unstale":"2023-07-02T14:02:40.243038+0000","last_undegraded":"2023-07-02T14:02:40.243038+0000","last_fullsized":"2023-07-02T14:02:40.243038+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10645520,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":64,"num_write_kb":10213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"40'88","reported_seq":130,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-07-02T14:02:40.242920+0000","last_change":"2023-07-02T14:02:39.061498+0000","last_active":"2023-07-02T14:02:40.242920+0000","last_peered":"2023-07-02T14:02:40.242920+0000","last_clean":"2023-07-02T14:02:40.242920+0000","last_became_active":"2023-07-02T14:01:49.294964+0000","last_became_peered":"2023-07-02T14:01:49.294964+0000","last_unstale":"2023-07-02T14:02:40.242920+0000","last_undegraded":"2023-07-02T14:02:40.242920+0000","last_fullsized":"2023-07-02T14:02:40.242920+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":88,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16580608,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":9,"num_write":88,"num_write_kb":16148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"40'83","reported_seq":116,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-07-02T14:02:39.368307+0000","last_change":"2023-07-02T14:02:39.062067+0000","last_active":"2023-07-02T14:02:39.368307+0000","last_peered":"2023-07-02T14:02:39.368307+0000","last_clean":"2023-07-02T14:02:39.368307+0000","last_became_active":"2023-07-02T14:02:15.025376+0000","last_became_peered":"2023-07-02T14:02:15.025376+0000","last_unstale":"2023-07-02T14:02:39.368307+0000","last_undegraded":"2023-07-02T14:02:39.368307+0000","last_fullsized":"2023-07-02T14:02:39.368307+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":83,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8388627,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":83,"num_write_kb":8194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"40'59","reported_seq":78,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-07-02T14:02:40.242744+0000","last_change":"2023-07-02T14:02:39.064149+0000","last_active":"2023-07-02T14:02:40.242744+0000","last_peered":"2023-07-02T14:02:40.242744+0000","last_clean":"2023-07-02T14:02:40.242744+0000","last_became_active":"2023-07-02T14:02:27.728488+0000","last_became_peered":"2023-07-02T14:02:27.728488+0000","last_unstale":"2023-07-02T14:02:40.242744+0000","last_undegraded":"2023-07-02T14:02:40.242744+0000","last_fullsized":"2023-07-02T14:02:40.242744+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":59,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6225920,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":59,"num_write_kb":6082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":87692535,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":256,"num_read_kb":281,"num_write":593,"num_write_kb":81455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":166932480,"data_stored":166894062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":565,"ondisk_log_size":565,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640269,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39288,"kb_used_data":33776,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":94332552,"statfs":{"total":96636764160,"available":96596533248,"internally_reserved":0,"allocated":34586624,"data_stored":34289808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640269,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":5436,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366404,"statfs":{"total":96636764160,"available":96631197696,"internally_reserved":0,"allocated":446464,"data_stored":160843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640269,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46768,"kb_used_data":41000,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":94325072,"statfs":{"total":96636764160,"available":96588873728,"internally_reserved":0,"allocated":41984000,"data_stored":41687327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]}]},{"osd":2,"up_from":11,"seq":47244640269,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26540,"kb_used_data":21348,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94345300,"statfs":{"total":96636764160,"available":96609587200,"internally_reserved":0,"allocated":21860352,"data_stored":21570651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640269,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":18044,"kb_used_data":12916,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":94353796,"statfs":{"total":96636764160,"available":96618287104,"internally_reserved":0,"allocated":13225984,"data_stored":12940363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]}]},{"osd":1,"up_from":11,"seq":47244640269,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9292,"kb_used_data":4100,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":94362548,"statfs":{"total":96636764160,"available":96627249152,"internally_reserved":0,"allocated":4198400,"data_stored":3900688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14614528,"data_stored":14614528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45662208,"data_stored":45650117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":26918912,"data_stored":26914832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":41795584,"data_stored":41784532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":37941248,"data_stored":37930053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:02:45.057 INFO:tasks.thrashosds.thrasher:Setting 2.0 to [4, 1] 2023-07-02T14:02:45.057 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.0', '4', '1'] 2023-07-02T14:02:45.057 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.0 4 1 2023-07-02T14:02:45.058 INFO:tasks.workunit.client.0.smithi163.stderr:102750K .......... .......... .......... .......... .......... 60% 17.3M 3s 2023-07-02T14:02:45.059 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:45.059 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:45.060 INFO:tasks.workunit.client.0.smithi163.stderr:102800K .......... .......... .......... .......... .......... 61% 24.7M 3s 2023-07-02T14:02:45.061 INFO:tasks.workunit.client.0.smithi163.stderr:102850K .......... .......... .......... .......... .......... 61% 28.1M 3s 2023-07-02T14:02:45.064 INFO:tasks.workunit.client.0.smithi163.stderr:102900K .......... .......... .......... .......... .......... 61% 20.4M 3s 2023-07-02T14:02:45.066 INFO:tasks.workunit.client.0.smithi163.stderr:102950K .......... .......... .......... .......... .......... 61% 16.8M 3s 2023-07-02T14:02:45.068 INFO:tasks.workunit.client.0.smithi163.stderr:103000K .......... .......... .......... .......... .......... 61% 26.9M 3s 2023-07-02T14:02:45.070 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:02:45.071 INFO:tasks.workunit.client.0.smithi163.stderr:103050K .......... .......... .......... .......... .......... 61% 18.1M 3s 2023-07-02T14:02:45.073 INFO:tasks.workunit.client.0.smithi163.stderr:103100K .......... .......... .......... .......... .......... 61% 22.7M 3s 2023-07-02T14:02:45.076 INFO:tasks.workunit.client.0.smithi163.stderr:103150K .......... .......... .......... .......... .......... 61% 18.6M 3s 2023-07-02T14:02:45.078 INFO:tasks.workunit.client.0.smithi163.stderr:103200K .......... .......... .......... .......... .......... 61% 24.5M 3s 2023-07-02T14:02:45.082 INFO:tasks.workunit.client.0.smithi163.stderr:103250K .......... .......... .......... .......... .......... 61% 13.8M 3s 2023-07-02T14:02:45.083 INFO:tasks.workunit.client.0.smithi163.stderr:103300K .......... .......... .......... .......... .......... 61% 30.5M 3s 2023-07-02T14:02:45.086 INFO:tasks.workunit.client.0.smithi163.stderr:103350K .......... .......... .......... .......... .......... 61% 17.8M 3s 2023-07-02T14:02:45.088 INFO:tasks.workunit.client.0.smithi163.stderr:103400K .......... .......... .......... .......... .......... 61% 21.6M 3s 2023-07-02T14:02:45.089 INFO:tasks.workunit.client.0.smithi163.stderr:103450K .......... .......... .......... .......... .......... 61% 31.0M 3s 2023-07-02T14:02:45.092 INFO:tasks.workunit.client.0.smithi163.stderr:103500K .......... .......... .......... .......... .......... 61% 21.6M 3s 2023-07-02T14:02:45.095 INFO:tasks.workunit.client.0.smithi163.stderr:103550K .......... .......... .......... .......... .......... 61% 14.6M 3s 2023-07-02T14:02:45.098 INFO:tasks.workunit.client.0.smithi163.stderr:103600K .......... .......... .......... .......... .......... 61% 18.8M 3s 2023-07-02T14:02:45.100 INFO:tasks.workunit.client.0.smithi163.stderr:103650K .......... .......... .......... .......... .......... 61% 23.2M 3s 2023-07-02T14:02:45.102 INFO:tasks.workunit.client.0.smithi163.stderr:103700K .......... .......... .......... .......... .......... 61% 21.3M 3s 2023-07-02T14:02:45.105 INFO:tasks.workunit.client.0.smithi163.stderr:103750K .......... .......... .......... .......... .......... 61% 18.6M 3s 2023-07-02T14:02:45.106 INFO:tasks.workunit.client.0.smithi163.stderr:103800K .......... .......... .......... .......... .......... 61% 25.7M 3s 2023-07-02T14:02:45.108 INFO:tasks.workunit.client.0.smithi163.stderr:103850K .......... .......... .......... .......... .......... 61% 30.5M 3s 2023-07-02T14:02:45.110 INFO:tasks.workunit.client.0.smithi163.stderr:103900K .......... .......... .......... .......... .......... 61% 21.3M 3s 2023-07-02T14:02:45.113 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:45.112+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:45.114 INFO:tasks.workunit.client.0.smithi163.stderr:103950K .......... .......... .......... .......... .......... 61% 12.6M 3s 2023-07-02T14:02:45.117 INFO:tasks.workunit.client.0.smithi163.stderr:104000K .......... .......... .......... .......... .......... 61% 18.6M 3s 2023-07-02T14:02:45.119 INFO:tasks.workunit.client.0.smithi163.stderr:104050K .......... .......... .......... .......... .......... 61% 22.7M 3s 2023-07-02T14:02:45.122 INFO:tasks.workunit.client.0.smithi163.stderr:104100K .......... .......... .......... .......... .......... 61% 18.3M 3s 2023-07-02T14:02:45.123 INFO:tasks.workunit.client.0.smithi163.stderr:104150K .......... .......... .......... .......... .......... 61% 38.3M 3s 2023-07-02T14:02:45.125 INFO:tasks.workunit.client.0.smithi163.stderr:104200K .......... .......... .......... .......... .......... 61% 19.0M 3s 2023-07-02T14:02:45.129 INFO:tasks.workunit.client.0.smithi163.stderr:104250K .......... .......... .......... .......... .......... 61% 15.6M 3s 2023-07-02T14:02:45.131 INFO:tasks.workunit.client.0.smithi163.stderr:104300K .......... .......... .......... .......... .......... 61% 20.4M 3s 2023-07-02T14:02:45.133 INFO:tasks.workunit.client.0.smithi163.stderr:104350K .......... .......... .......... .......... .......... 61% 19.3M 3s 2023-07-02T14:02:45.135 INFO:tasks.workunit.client.0.smithi163.stderr:104400K .......... .......... .......... .......... .......... 61% 28.2M 3s 2023-07-02T14:02:45.138 INFO:tasks.workunit.client.0.smithi163.stderr:104450K .......... .......... .......... .......... .......... 61% 21.6M 3s 2023-07-02T14:02:45.141 INFO:tasks.workunit.client.0.smithi163.stderr:104500K .......... .......... .......... .......... .......... 62% 14.6M 3s 2023-07-02T14:02:45.143 INFO:tasks.workunit.client.0.smithi163.stderr:104550K .......... .......... .......... .......... .......... 62% 33.6M 3s 2023-07-02T14:02:45.146 INFO:tasks.workunit.client.0.smithi163.stderr:104600K .......... .......... .......... .......... .......... 62% 15.3M 3s 2023-07-02T14:02:45.147 INFO:tasks.workunit.client.0.smithi163.stderr:104650K .......... .......... .......... .......... .......... 62% 26.9M 3s 2023-07-02T14:02:45.149 INFO:tasks.workunit.client.0.smithi163.stderr:104700K .......... .......... .......... .......... .......... 62% 22.5M 3s 2023-07-02T14:02:45.152 INFO:tasks.workunit.client.0.smithi163.stderr:104750K .......... .......... .......... .......... .......... 62% 17.8M 3s 2023-07-02T14:02:45.155 INFO:tasks.workunit.client.0.smithi163.stderr:104800K .......... .......... .......... .......... .......... 62% 20.6M 3s 2023-07-02T14:02:45.157 INFO:tasks.workunit.client.0.smithi163.stderr:104850K .......... .......... .......... .......... .......... 62% 19.3M 3s 2023-07-02T14:02:45.158 INFO:tasks.workunit.client.0.smithi163.stderr:104900K .......... .......... .......... .......... .......... 62% 34.3M 3s 2023-07-02T14:02:45.162 INFO:tasks.workunit.client.0.smithi163.stderr:104950K .......... .......... .......... .......... .......... 62% 17.1M 3s 2023-07-02T14:02:45.164 INFO:tasks.workunit.client.0.smithi163.stderr:105000K .......... .......... .......... .......... .......... 62% 21.6M 3s 2023-07-02T14:02:45.166 INFO:tasks.workunit.client.0.smithi163.stderr:105050K .......... .......... .......... .......... .......... 62% 24.4M 3s 2023-07-02T14:02:45.168 INFO:tasks.workunit.client.0.smithi163.stderr:105100K .......... .......... .......... .......... .......... 62% 18.9M 3s 2023-07-02T14:02:45.172 INFO:tasks.workunit.client.0.smithi163.stderr:105150K .......... .......... .......... .......... .......... 62% 13.4M 3s 2023-07-02T14:02:45.173 INFO:tasks.workunit.client.0.smithi163.stderr:105200K .......... .......... .......... .......... .......... 62% 31.5M 3s 2023-07-02T14:02:45.176 INFO:tasks.workunit.client.0.smithi163.stderr:105250K .......... .......... .......... .......... .......... 62% 20.1M 3s 2023-07-02T14:02:45.178 INFO:tasks.workunit.client.0.smithi163.stderr:105300K .......... .......... .......... .......... .......... 62% 28.7M 3s 2023-07-02T14:02:45.180 INFO:tasks.workunit.client.0.smithi163.stderr:105350K .......... .......... .......... .......... .......... 62% 18.7M 3s 2023-07-02T14:02:45.183 INFO:tasks.workunit.client.0.smithi163.stderr:105400K .......... .......... .......... .......... .......... 62% 18.5M 3s 2023-07-02T14:02:45.186 INFO:tasks.workunit.client.0.smithi163.stderr:105450K .......... .......... .......... .......... .......... 62% 17.6M 3s 2023-07-02T14:02:45.189 INFO:tasks.workunit.client.0.smithi163.stderr:105500K .......... .......... .......... .......... .......... 62% 17.4M 3s 2023-07-02T14:02:45.192 INFO:tasks.workunit.client.0.smithi163.stderr:105550K .......... .......... .......... .......... .......... 62% 15.0M 3s 2023-07-02T14:02:45.193 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:45.193 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:45.194 INFO:tasks.workunit.client.0.smithi163.stderr:105600K .......... .......... .......... .......... .......... 62% 34.7M 3s 2023-07-02T14:02:45.196 INFO:tasks.workunit.client.0.smithi163.stderr:105650K .......... .......... .......... .......... .......... 62% 18.0M 3s 2023-07-02T14:02:45.198 INFO:tasks.workunit.client.0.smithi163.stderr:105700K .......... .......... .......... .......... .......... 62% 25.9M 3s 2023-07-02T14:02:45.200 INFO:tasks.workunit.client.0.smithi163.stderr:105750K .......... .......... .......... .......... .......... 62% 19.4M 3s 2023-07-02T14:02:45.202 INFO:tasks.workunit.client.0.smithi163.stderr:105800K .......... .......... .......... .......... .......... 62% 32.8M 3s 2023-07-02T14:02:45.203 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:02:45.204 INFO:tasks.workunit.client.0.smithi163.stderr:105850K .......... .......... .......... .......... .......... 62% 19.5M 3s 2023-07-02T14:02:45.205 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:45.203+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:45.207 INFO:tasks.workunit.client.0.smithi163.stderr:105900K .......... .......... .......... .......... .......... 62% 17.1M 3s 2023-07-02T14:02:45.210 INFO:tasks.workunit.client.0.smithi163.stderr:105950K .......... .......... .......... .......... .......... 62% 16.9M 3s 2023-07-02T14:02:45.213 INFO:tasks.workunit.client.0.smithi163.stderr:106000K .......... .......... .......... .......... .......... 62% 21.6M 3s 2023-07-02T14:02:45.215 INFO:tasks.workunit.client.0.smithi163.stderr:106050K .......... .......... .......... .......... .......... 62% 17.9M 3s 2023-07-02T14:02:45.217 INFO:tasks.workunit.client.0.smithi163.stderr:106100K .......... .......... .......... .......... .......... 62% 25.2M 3s 2023-07-02T14:02:45.219 INFO:tasks.workunit.client.0.smithi163.stderr:106150K .......... .......... .......... .......... .......... 63% 19.4M 3s 2023-07-02T14:02:45.221 INFO:tasks.workunit.client.0.smithi163.stderr:106200K .......... .......... .......... .......... .......... 63% 26.6M 3s 2023-07-02T14:02:45.224 INFO:tasks.workunit.client.0.smithi163.stderr:106250K .......... .......... .......... .......... .......... 63% 17.5M 3s 2023-07-02T14:02:45.226 INFO:tasks.workunit.client.0.smithi163.stderr:106300K .......... .......... .......... .......... .......... 63% 29.1M 3s 2023-07-02T14:02:45.229 INFO:tasks.workunit.client.0.smithi163.stderr:106350K .......... .......... .......... .......... .......... 63% 17.9M 3s 2023-07-02T14:02:45.232 INFO:tasks.workunit.client.0.smithi163.stderr:106400K .......... .......... .......... .......... .......... 63% 14.7M 3s 2023-07-02T14:02:45.233 INFO:tasks.workunit.client.0.smithi163.stderr:106450K .......... .......... .......... .......... .......... 63% 41.2M 3s 2023-07-02T14:02:45.236 INFO:tasks.workunit.client.0.smithi163.stderr:106500K .......... .......... .......... .......... .......... 63% 18.6M 3s 2023-07-02T14:02:45.238 INFO:tasks.workunit.client.0.smithi163.stderr:106550K .......... .......... .......... .......... .......... 63% 17.8M 3s 2023-07-02T14:02:45.240 INFO:tasks.workunit.client.0.smithi163.stderr:106600K .......... .......... .......... .......... .......... 63% 31.7M 3s 2023-07-02T14:02:45.242 INFO:tasks.workunit.client.0.smithi163.stderr:106650K .......... .......... .......... .......... .......... 63% 28.1M 3s 2023-07-02T14:02:45.245 INFO:tasks.workunit.client.0.smithi163.stderr:106700K .......... .......... .......... .......... .......... 63% 16.1M 3s 2023-07-02T14:02:45.247 INFO:tasks.workunit.client.0.smithi163.stderr:106750K .......... .......... .......... .......... .......... 63% 17.9M 3s 2023-07-02T14:02:45.250 INFO:tasks.workunit.client.0.smithi163.stderr:106800K .......... .......... .......... .......... .......... 63% 21.1M 3s 2023-07-02T14:02:45.252 INFO:tasks.workunit.client.0.smithi163.stderr:106850K .......... .......... .......... .......... .......... 63% 19.5M 3s 2023-07-02T14:02:45.255 INFO:tasks.workunit.client.0.smithi163.stderr:106900K .......... .......... .......... .......... .......... 63% 17.9M 3s 2023-07-02T14:02:45.257 INFO:tasks.workunit.client.0.smithi163.stderr:106950K .......... .......... .......... .......... .......... 63% 24.5M 3s 2023-07-02T14:02:45.258 INFO:tasks.workunit.client.0.smithi163.stderr:107000K .......... .......... .......... .......... .......... 63% 30.7M 3s 2023-07-02T14:02:45.262 INFO:tasks.workunit.client.0.smithi163.stderr:107050K .......... .......... .......... .......... .......... 63% 14.2M 3s 2023-07-02T14:02:45.264 INFO:tasks.workunit.client.0.smithi163.stderr:107100K .......... .......... .......... .......... .......... 63% 28.1M 3s 2023-07-02T14:02:45.266 INFO:tasks.workunit.client.0.smithi163.stderr:107150K .......... .......... .......... .......... .......... 63% 17.8M 3s 2023-07-02T14:02:45.268 INFO:tasks.workunit.client.0.smithi163.stderr:107200K .......... .......... .......... .......... .......... 63% 27.2M 3s 2023-07-02T14:02:45.271 INFO:tasks.workunit.client.0.smithi163.stderr:107250K .......... .......... .......... .......... .......... 63% 17.2M 3s 2023-07-02T14:02:45.274 INFO:tasks.workunit.client.0.smithi163.stderr:107300K .......... .......... .......... .......... .......... 63% 19.3M 3s 2023-07-02T14:02:45.276 INFO:tasks.workunit.client.0.smithi163.stderr:107350K .......... .......... .......... .......... .......... 63% 20.2M 3s 2023-07-02T14:02:45.279 INFO:tasks.workunit.client.0.smithi163.stderr:107400K .......... .......... .......... .......... .......... 63% 17.5M 3s 2023-07-02T14:02:45.280 INFO:tasks.workunit.client.0.smithi163.stderr:107450K .......... .......... .......... .......... .......... 63% 33.2M 3s 2023-07-02T14:02:45.284 INFO:tasks.workunit.client.0.smithi163.stderr:107500K .......... .......... .......... .......... .......... 63% 14.6M 3s 2023-07-02T14:02:45.286 INFO:tasks.workunit.client.0.smithi163.stderr:107550K .......... .......... .......... .......... .......... 63% 21.9M 3s 2023-07-02T14:02:45.289 INFO:tasks.workunit.client.0.smithi163.stderr:107600K .......... .......... .......... .......... .......... 63% 17.0M 3s 2023-07-02T14:02:45.291 INFO:tasks.workunit.client.0.smithi163.stderr:107650K .......... .......... .......... .......... .......... 63% 24.7M 3s 2023-07-02T14:02:45.292 INFO:tasks.workunit.client.0.smithi163.stderr:107700K .......... .......... .......... .......... .......... 63% 34.8M 3s 2023-07-02T14:02:45.294 INFO:tasks.workunit.client.0.smithi163.stderr:107750K .......... .......... .......... .......... .......... 63% 23.9M 3s 2023-07-02T14:02:45.298 INFO:tasks.workunit.client.0.smithi163.stderr:107800K .......... .......... .......... .......... .......... 63% 12.8M 3s 2023-07-02T14:02:45.300 INFO:tasks.workunit.client.0.smithi163.stderr:107850K .......... .......... .......... .......... .......... 64% 23.4M 3s 2023-07-02T14:02:45.303 INFO:tasks.workunit.client.0.smithi163.stderr:107900K .......... .......... .......... .......... .......... 64% 20.3M 3s 2023-07-02T14:02:45.304 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:45.303+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:45.305 INFO:tasks.workunit.client.0.smithi163.stderr:107950K .......... .......... .......... .......... .......... 64% 19.8M 3s 2023-07-02T14:02:45.307 INFO:tasks.workunit.client.0.smithi163.stderr:108000K .......... .......... .......... .......... .......... 64% 22.3M 3s 2023-07-02T14:02:45.309 INFO:tasks.workunit.client.0.smithi163.stderr:108050K .......... .......... .......... .......... .......... 64% 21.2M 3s 2023-07-02T14:02:45.312 INFO:tasks.workunit.client.0.smithi163.stderr:108100K .......... .......... .......... .......... .......... 64% 21.3M 3s 2023-07-02T14:02:45.314 INFO:tasks.workunit.client.0.smithi163.stderr:108150K .......... .......... .......... .......... .......... 64% 18.5M 3s 2023-07-02T14:02:45.316 INFO:tasks.workunit.client.0.smithi163.stderr:108200K .......... .......... .......... .......... .......... 64% 33.5M 3s 2023-07-02T14:02:45.319 INFO:tasks.workunit.client.0.smithi163.stderr:108250K .......... .......... .......... .......... .......... 64% 19.1M 3s 2023-07-02T14:02:45.322 INFO:tasks.workunit.client.0.smithi163.stderr:108300K .......... .......... .......... .......... .......... 64% 14.4M 3s 2023-07-02T14:02:45.324 INFO:tasks.workunit.client.0.smithi163.stderr:108350K .......... .......... .......... .......... .......... 64% 18.1M 3s 2023-07-02T14:02:45.326 INFO:tasks.workunit.client.0.smithi163.stderr:108400K .......... .......... .......... .......... .......... 64% 36.7M 3s 2023-07-02T14:02:45.329 INFO:tasks.workunit.client.0.smithi163.stderr:108450K .......... .......... .......... .......... .......... 64% 16.8M 3s 2023-07-02T14:02:45.330 INFO:tasks.workunit.client.0.smithi163.stderr:108500K .......... .......... .......... .......... .......... 64% 29.8M 3s 2023-07-02T14:02:45.333 INFO:tasks.workunit.client.0.smithi163.stderr:108550K .......... .......... .......... .......... .......... 64% 16.7M 3s 2023-07-02T14:02:45.335 INFO:tasks.workunit.client.0.smithi163.stderr:108600K .......... .......... .......... .......... .......... 64% 39.8M 3s 2023-07-02T14:02:45.338 INFO:tasks.workunit.client.0.smithi163.stderr:108650K .......... .......... .......... .......... .......... 64% 14.5M 3s 2023-07-02T14:02:45.339 INFO:tasks.workunit.client.0.smithi163.stderr:108700K .......... .......... .......... .......... .......... 64% 30.0M 3s 2023-07-02T14:02:45.342 INFO:tasks.workunit.client.0.smithi163.stderr:108750K .......... .......... .......... .......... .......... 64% 17.3M 3s 2023-07-02T14:02:45.344 INFO:tasks.workunit.client.0.smithi163.stderr:108800K .......... .......... .......... .......... .......... 64% 23.8M 3s 2023-07-02T14:02:45.348 INFO:tasks.workunit.client.0.smithi163.stderr:108850K .......... .......... .......... .......... .......... 64% 15.2M 3s 2023-07-02T14:02:45.349 INFO:tasks.workunit.client.0.smithi163.stderr:108900K .......... .......... .......... .......... .......... 64% 37.6M 3s 2023-07-02T14:02:45.352 INFO:tasks.workunit.client.0.smithi163.stderr:108950K .......... .......... .......... .......... .......... 64% 18.4M 3s 2023-07-02T14:02:45.354 INFO:tasks.workunit.client.0.smithi163.stderr:109000K .......... .......... .......... .......... .......... 64% 25.1M 3s 2023-07-02T14:02:45.356 INFO:tasks.workunit.client.0.smithi163.stderr:109050K .......... .......... .......... .......... .......... 64% 17.4M 3s 2023-07-02T14:02:45.359 INFO:tasks.workunit.client.0.smithi163.stderr:109100K .......... .......... .......... .......... .......... 64% 21.8M 3s 2023-07-02T14:02:45.362 INFO:tasks.workunit.client.0.smithi163.stderr:109150K .......... .......... .......... .......... .......... 64% 15.4M 3s 2023-07-02T14:02:45.365 INFO:tasks.workunit.client.0.smithi163.stderr:109200K .......... .......... .......... .......... .......... 64% 17.1M 3s 2023-07-02T14:02:45.367 INFO:tasks.workunit.client.0.smithi163.stderr:109250K .......... .......... .......... .......... .......... 64% 20.7M 3s 2023-07-02T14:02:45.368 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:45.368 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:45.369 INFO:tasks.workunit.client.0.smithi163.stderr:109300K .......... .......... .......... .......... .......... 64% 30.0M 3s 2023-07-02T14:02:45.372 INFO:tasks.workunit.client.0.smithi163.stderr:109350K .......... .......... .......... .......... .......... 64% 16.7M 3s 2023-07-02T14:02:45.374 INFO:tasks.workunit.client.0.smithi163.stderr:109400K .......... .......... .......... .......... .......... 64% 19.5M 3s 2023-07-02T14:02:45.376 INFO:tasks.workunit.client.0.smithi163.stderr:109450K .......... .......... .......... .......... .......... 64% 24.4M 3s 2023-07-02T14:02:45.378 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:02:45.379 INFO:tasks.workunit.client.0.smithi163.stderr:109500K .......... .......... .......... .......... .......... 64% 27.5M 3s 2023-07-02T14:02:45.381 INFO:tasks.workunit.client.0.smithi163.stderr:109550K .......... .......... .......... .......... .......... 65% 17.5M 3s 2023-07-02T14:02:45.384 INFO:tasks.workunit.client.0.smithi163.stderr:109600K .......... .......... .......... .......... .......... 65% 15.4M 3s 2023-07-02T14:02:45.386 INFO:tasks.workunit.client.0.smithi163.stderr:109650K .......... .......... .......... .......... .......... 65% 26.8M 3s 2023-07-02T14:02:45.389 INFO:tasks.workunit.client.0.smithi163.stderr:109700K .......... .......... .......... .......... .......... 65% 14.8M 3s 2023-07-02T14:02:45.391 INFO:tasks.workunit.client.0.smithi163.stderr:109750K .......... .......... .......... .......... .......... 65% 20.7M 3s 2023-07-02T14:02:45.393 INFO:tasks.workunit.client.0.smithi163.stderr:109800K .......... .......... .......... .......... .......... 65% 29.4M 3s 2023-07-02T14:02:45.395 INFO:tasks.workunit.client.0.smithi163.stderr:109850K .......... .......... .......... .......... .......... 65% 28.7M 3s 2023-07-02T14:02:45.398 INFO:tasks.workunit.client.0.smithi163.stderr:109900K .......... .......... .......... .......... .......... 65% 15.5M 3s 2023-07-02T14:02:45.400 INFO:tasks.workunit.client.0.smithi163.stderr:109950K .......... .......... .......... .......... .......... 65% 20.4M 3s 2023-07-02T14:02:45.403 INFO:tasks.workunit.client.0.smithi163.stderr:110000K .......... .......... .......... .......... .......... 65% 19.6M 3s 2023-07-02T14:02:45.405 INFO:tasks.workunit.client.0.smithi163.stderr:110050K .......... .......... .......... .......... .......... 65% 25.7M 3s 2023-07-02T14:02:45.407 INFO:tasks.workunit.client.0.smithi163.stderr:110100K .......... .......... .......... .......... .......... 65% 18.0M 3s 2023-07-02T14:02:45.410 INFO:tasks.workunit.client.0.smithi163.stderr:110150K .......... .......... .......... .......... .......... 65% 19.7M 3s 2023-07-02T14:02:45.413 INFO:tasks.workunit.client.0.smithi163.stderr:110200K .......... .......... .......... .......... .......... 65% 14.6M 3s 2023-07-02T14:02:45.414 INFO:tasks.workunit.client.0.smithi163.stderr:110250K .......... .......... .......... .......... .......... 65% 36.8M 3s 2023-07-02T14:02:45.417 INFO:tasks.workunit.client.0.smithi163.stderr:110300K .......... .......... .......... .......... .......... 65% 20.8M 3s 2023-07-02T14:02:45.419 INFO:tasks.workunit.client.0.smithi163.stderr:110350K .......... .......... .......... .......... .......... 65% 19.4M 3s 2023-07-02T14:02:45.421 INFO:tasks.workunit.client.0.smithi163.stderr:110400K .......... .......... .......... .......... .......... 65% 24.4M 3s 2023-07-02T14:02:45.422 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:45.422+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:45.424 INFO:tasks.workunit.client.0.smithi163.stderr:110450K .......... .......... .......... .......... .......... 65% 16.8M 3s 2023-07-02T14:02:45.426 INFO:tasks.workunit.client.0.smithi163.stderr:110500K .......... .......... .......... .......... .......... 65% 31.7M 3s 2023-07-02T14:02:45.429 INFO:tasks.workunit.client.0.smithi163.stderr:110550K .......... .......... .......... .......... .......... 65% 17.2M 3s 2023-07-02T14:02:45.431 INFO:tasks.workunit.client.0.smithi163.stderr:110600K .......... .......... .......... .......... .......... 65% 20.1M 3s 2023-07-02T14:02:45.433 INFO:tasks.workunit.client.0.smithi163.stderr:110650K .......... .......... .......... .......... .......... 65% 22.6M 3s 2023-07-02T14:02:45.435 INFO:tasks.workunit.client.0.smithi163.stderr:110700K .......... .......... .......... .......... .......... 65% 21.8M 3s 2023-07-02T14:02:45.439 INFO:tasks.workunit.client.0.smithi163.stderr:110750K .......... .......... .......... .......... .......... 65% 14.2M 3s 2023-07-02T14:02:45.441 INFO:tasks.workunit.client.0.smithi163.stderr:110800K .......... .......... .......... .......... .......... 65% 26.5M 3s 2023-07-02T14:02:45.443 INFO:tasks.workunit.client.0.smithi163.stderr:110850K .......... .......... .......... .......... .......... 65% 18.7M 3s 2023-07-02T14:02:45.445 INFO:tasks.workunit.client.0.smithi163.stderr:110900K .......... .......... .......... .......... .......... 65% 36.1M 3s 2023-07-02T14:02:45.447 INFO:tasks.workunit.client.0.smithi163.stderr:110950K .......... .......... .......... .......... .......... 65% 17.4M 3s 2023-07-02T14:02:45.449 INFO:tasks.workunit.client.0.smithi163.stderr:111000K .......... .......... .......... .......... .......... 65% 22.2M 3s 2023-07-02T14:02:45.452 INFO:tasks.workunit.client.0.smithi163.stderr:111050K .......... .......... .......... .......... .......... 65% 19.7M 3s 2023-07-02T14:02:45.455 INFO:tasks.workunit.client.0.smithi163.stderr:111100K .......... .......... .......... .......... .......... 65% 20.0M 3s 2023-07-02T14:02:45.458 INFO:tasks.workunit.client.0.smithi163.stderr:111150K .......... .......... .......... .......... .......... 65% 13.5M 3s 2023-07-02T14:02:45.459 INFO:tasks.workunit.client.0.smithi163.stderr:111200K .......... .......... .......... .......... .......... 65% 36.3M 3s 2023-07-02T14:02:45.463 INFO:tasks.workunit.client.0.smithi163.stderr:111250K .......... .......... .......... .......... .......... 66% 16.2M 3s 2023-07-02T14:02:45.464 INFO:tasks.workunit.client.0.smithi163.stderr:111300K .......... .......... .......... .......... .......... 66% 30.6M 3s 2023-07-02T14:02:45.467 INFO:tasks.workunit.client.0.smithi163.stderr:111350K .......... .......... .......... .......... .......... 66% 15.9M 3s 2023-07-02T14:02:45.468 INFO:tasks.workunit.client.0.smithi163.stderr:111400K .......... .......... .......... .......... .......... 66% 40.8M 3s 2023-07-02T14:02:45.471 INFO:tasks.workunit.client.0.smithi163.stderr:111450K .......... .......... .......... .......... .......... 66% 17.0M 3s 2023-07-02T14:02:45.473 INFO:tasks.workunit.client.0.smithi163.stderr:111500K .......... .......... .......... .......... .......... 66% 22.1M 3s 2023-07-02T14:02:45.476 INFO:tasks.workunit.client.0.smithi163.stderr:111550K .......... .......... .......... .......... .......... 66% 20.6M 3s 2023-07-02T14:02:45.479 INFO:tasks.workunit.client.0.smithi163.stderr:111600K .......... .......... .......... .......... .......... 66% 16.3M 3s 2023-07-02T14:02:45.481 INFO:tasks.workunit.client.0.smithi163.stderr:111650K .......... .......... .......... .......... .......... 66% 20.9M 3s 2023-07-02T14:02:45.483 INFO:tasks.workunit.client.0.smithi163.stderr:111700K .......... .......... .......... .......... .......... 66% 21.4M 3s 2023-07-02T14:02:45.485 INFO:tasks.workunit.client.0.smithi163.stderr:111750K .......... .......... .......... .......... .......... 66% 32.7M 3s 2023-07-02T14:02:45.487 INFO:tasks.workunit.client.0.smithi163.stderr:111800K .......... .......... .......... .......... .......... 66% 22.1M 3s 2023-07-02T14:02:45.490 INFO:tasks.workunit.client.0.smithi163.stderr:111850K .......... .......... .......... .......... .......... 66% 16.3M 3s 2023-07-02T14:02:45.493 INFO:tasks.workunit.client.0.smithi163.stderr:111900K .......... .......... .......... .......... .......... 66% 19.6M 3s 2023-07-02T14:02:45.495 INFO:tasks.workunit.client.0.smithi163.stderr:111950K .......... .......... .......... .......... .......... 66% 18.2M 3s 2023-07-02T14:02:45.498 INFO:tasks.workunit.client.0.smithi163.stderr:112000K .......... .......... .......... .......... .......... 66% 20.6M 3s 2023-07-02T14:02:45.500 INFO:tasks.workunit.client.0.smithi163.stderr:112050K .......... .......... .......... .......... .......... 66% 21.5M 3s 2023-07-02T14:02:45.501 INFO:tasks.workunit.client.0.smithi163.stderr:112100K .......... .......... .......... .......... .......... 66% 30.7M 3s 2023-07-02T14:02:45.503 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:45.503 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:45.506 INFO:tasks.workunit.client.0.smithi163.stderr:112150K .......... .......... .......... .......... .......... 66% 13.4M 3s 2023-07-02T14:02:45.506 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:45.505+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:45.507 INFO:tasks.workunit.client.0.smithi163.stderr:112200K .......... .......... .......... .......... .......... 66% 25.5M 3s 2023-07-02T14:02:45.509 INFO:tasks.workunit.client.0.smithi163.stderr:112250K .......... .......... .......... .......... .......... 66% 26.2M 3s 2023-07-02T14:02:45.511 INFO:tasks.workunit.client.0.smithi163.stderr:112300K .......... .......... .......... .......... .......... 66% 27.7M 3s 2023-07-02T14:02:45.513 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:02:45.514 INFO:tasks.workunit.client.0.smithi163.stderr:112350K .......... .......... .......... .......... .......... 66% 17.4M 3s 2023-07-02T14:02:45.517 INFO:tasks.workunit.client.0.smithi163.stderr:112400K .......... .......... .......... .......... .......... 66% 16.3M 3s 2023-07-02T14:02:45.519 INFO:tasks.workunit.client.0.smithi163.stderr:112450K .......... .......... .......... .......... .......... 66% 19.1M 3s 2023-07-02T14:02:45.521 INFO:tasks.workunit.client.0.smithi163.stderr:112500K .......... .......... .......... .......... .......... 66% 21.3M 3s 2023-07-02T14:02:45.523 INFO:tasks.workunit.client.0.smithi163.stderr:112550K .......... .......... .......... .......... .......... 66% 21.6M 3s 2023-07-02T14:02:45.525 INFO:tasks.workunit.client.0.smithi163.stderr:112600K .......... .......... .......... .......... .......... 66% 24.9M 3s 2023-07-02T14:02:45.527 INFO:tasks.workunit.client.0.smithi163.stderr:112650K .......... .......... .......... .......... .......... 66% 27.2M 3s 2023-07-02T14:02:45.530 INFO:tasks.workunit.client.0.smithi163.stderr:112700K .......... .......... .......... .......... .......... 66% 19.5M 3s 2023-07-02T14:02:45.534 INFO:tasks.workunit.client.0.smithi163.stderr:112750K .......... .......... .......... .......... .......... 66% 12.9M 3s 2023-07-02T14:02:45.535 INFO:tasks.workunit.client.0.smithi163.stderr:112800K .......... .......... .......... .......... .......... 66% 42.3M 3s 2023-07-02T14:02:45.537 INFO:tasks.workunit.client.0.smithi163.stderr:112850K .......... .......... .......... .......... .......... 66% 22.0M 3s 2023-07-02T14:02:45.540 INFO:tasks.workunit.client.0.smithi163.stderr:112900K .......... .......... .......... .......... .......... 67% 14.7M 3s 2023-07-02T14:02:45.542 INFO:tasks.workunit.client.0.smithi163.stderr:112950K .......... .......... .......... .......... .......... 67% 25.0M 3s 2023-07-02T14:02:45.545 INFO:tasks.workunit.client.0.smithi163.stderr:113000K .......... .......... .......... .......... .......... 67% 16.9M 3s 2023-07-02T14:02:45.547 INFO:tasks.workunit.client.0.smithi163.stderr:113050K .......... .......... .......... .......... .......... 67% 32.5M 3s 2023-07-02T14:02:45.550 INFO:tasks.workunit.client.0.smithi163.stderr:113100K .......... .......... .......... .......... .......... 67% 15.4M 3s 2023-07-02T14:02:45.552 INFO:tasks.workunit.client.0.smithi163.stderr:113150K .......... .......... .......... .......... .......... 67% 19.7M 3s 2023-07-02T14:02:45.554 INFO:tasks.workunit.client.0.smithi163.stderr:113200K .......... .......... .......... .......... .......... 67% 23.0M 3s 2023-07-02T14:02:45.557 INFO:tasks.workunit.client.0.smithi163.stderr:113250K .......... .......... .......... .......... .......... 67% 16.2M 3s 2023-07-02T14:02:45.559 INFO:tasks.workunit.client.0.smithi163.stderr:113300K .......... .......... .......... .......... .......... 67% 31.1M 3s 2023-07-02T14:02:45.561 INFO:tasks.workunit.client.0.smithi163.stderr:113350K .......... .......... .......... .......... .......... 67% 22.5M 3s 2023-07-02T14:02:45.565 INFO:tasks.workunit.client.0.smithi163.stderr:113400K .......... .......... .......... .......... .......... 67% 14.1M 3s 2023-07-02T14:02:45.566 INFO:tasks.workunit.client.0.smithi163.stderr:113450K .......... .......... .......... .......... .......... 67% 36.0M 3s 2023-07-02T14:02:45.568 INFO:tasks.workunit.client.0.smithi163.stderr:113500K .......... .......... .......... .......... .......... 67% 22.6M 3s 2023-07-02T14:02:45.572 INFO:tasks.workunit.client.0.smithi163.stderr:113550K .......... .......... .......... .......... .......... 67% 13.3M 3s 2023-07-02T14:02:45.574 INFO:tasks.workunit.client.0.smithi163.stderr:113600K .......... .......... .......... .......... .......... 67% 22.2M 3s 2023-07-02T14:02:45.576 INFO:tasks.workunit.client.0.smithi163.stderr:113650K .......... .......... .......... .......... .......... 67% 25.0M 3s 2023-07-02T14:02:45.578 INFO:tasks.workunit.client.0.smithi163.stderr:113700K .......... .......... .......... .......... .......... 67% 32.1M 3s 2023-07-02T14:02:45.581 INFO:tasks.workunit.client.0.smithi163.stderr:113750K .......... .......... .......... .......... .......... 67% 14.8M 3s 2023-07-02T14:02:45.583 INFO:tasks.workunit.client.0.smithi163.stderr:113800K .......... .......... .......... .......... .......... 67% 23.1M 3s 2023-07-02T14:02:45.586 INFO:tasks.workunit.client.0.smithi163.stderr:113850K .......... .......... .......... .......... .......... 67% 17.4M 3s 2023-07-02T14:02:45.588 INFO:tasks.workunit.client.0.smithi163.stderr:113900K .......... .......... .......... .......... .......... 67% 22.4M 3s 2023-07-02T14:02:45.591 INFO:tasks.workunit.client.0.smithi163.stderr:113950K .......... .......... .......... .......... .......... 67% 16.8M 3s 2023-07-02T14:02:45.592 INFO:tasks.workunit.client.0.smithi163.stderr:114000K .......... .......... .......... .......... .......... 67% 37.3M 3s 2023-07-02T14:02:45.595 INFO:tasks.workunit.client.0.smithi163.stderr:114050K .......... .......... .......... .......... .......... 67% 17.5M 3s 2023-07-02T14:02:45.597 INFO:tasks.workunit.client.0.smithi163.stderr:114100K .......... .......... .......... .......... .......... 67% 28.9M 3s 2023-07-02T14:02:45.599 INFO:tasks.workunit.client.0.smithi163.stderr:114150K .......... .......... .......... .......... .......... 67% 18.8M 3s 2023-07-02T14:02:45.601 INFO:tasks.workunit.client.0.smithi163.stderr:114200K .......... .......... .......... .......... .......... 67% 22.7M 3s 2023-07-02T14:02:45.604 INFO:tasks.workunit.client.0.smithi163.stderr:114250K .......... .......... .......... .......... .......... 67% 18.8M 3s 2023-07-02T14:02:45.606 INFO:tasks.workunit.client.0.smithi163.stderr:114300K .......... .......... .......... .......... .......... 67% 23.3M 3s 2023-07-02T14:02:45.607 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:45.606+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:45.608 INFO:tasks.workunit.client.0.smithi163.stderr:114350K .......... .......... .......... .......... .......... 67% 19.7M 3s 2023-07-02T14:02:45.611 INFO:tasks.workunit.client.0.smithi163.stderr:114400K .......... .......... .......... .......... .......... 67% 18.5M 3s 2023-07-02T14:02:45.614 INFO:tasks.workunit.client.0.smithi163.stderr:114450K .......... .......... .......... .......... .......... 67% 21.3M 3s 2023-07-02T14:02:45.616 INFO:tasks.workunit.client.0.smithi163.stderr:114500K .......... .......... .......... .......... .......... 67% 21.6M 3s 2023-07-02T14:02:45.620 INFO:tasks.workunit.client.0.smithi163.stderr:114550K .......... .......... .......... .......... .......... 67% 24.6M 3s 2023-07-02T14:02:45.621 INFO:tasks.workunit.client.0.smithi163.stderr:114600K .......... .......... .......... .......... .......... 68% 23.4M 3s 2023-07-02T14:02:45.623 INFO:tasks.workunit.client.0.smithi163.stderr:114650K .......... .......... .......... .......... .......... 68% 15.4M 3s 2023-07-02T14:02:45.625 INFO:tasks.workunit.client.0.smithi163.stderr:114700K .......... .......... .......... .......... .......... 68% 22.2M 3s 2023-07-02T14:02:45.628 INFO:tasks.workunit.client.0.smithi163.stderr:114750K .......... .......... .......... .......... .......... 68% 18.2M 3s 2023-07-02T14:02:45.633 INFO:tasks.workunit.client.0.smithi163.stderr:114800K .......... .......... .......... .......... .......... 68% 10.3M 3s 2023-07-02T14:02:45.633 INFO:tasks.workunit.client.0.smithi163.stderr:114850K .......... .......... .......... .......... .......... 68% 98.2M 3s 2023-07-02T14:02:45.635 INFO:tasks.workunit.client.0.smithi163.stderr:114900K .......... .......... .......... .......... .......... 68% 21.8M 3s 2023-07-02T14:02:45.637 INFO:tasks.workunit.client.0.smithi163.stderr:114950K .......... .......... .......... .......... .......... 68% 26.8M 3s 2023-07-02T14:02:45.640 INFO:tasks.workunit.client.0.smithi163.stderr:115000K .......... .......... .......... .......... .......... 68% 16.7M 3s 2023-07-02T14:02:45.642 INFO:tasks.workunit.client.0.smithi163.stderr:115050K .......... .......... .......... .......... .......... 68% 26.7M 3s 2023-07-02T14:02:45.644 INFO:tasks.workunit.client.0.smithi163.stderr:115100K .......... .......... .......... .......... .......... 68% 21.9M 3s 2023-07-02T14:02:45.647 INFO:tasks.workunit.client.0.smithi163.stderr:115150K .......... .......... .......... .......... .......... 68% 17.7M 3s 2023-07-02T14:02:45.650 INFO:tasks.workunit.client.0.smithi163.stderr:115200K .......... .......... .......... .......... .......... 68% 15.1M 3s 2023-07-02T14:02:45.652 INFO:tasks.workunit.client.0.smithi163.stderr:115250K .......... .......... .......... .......... .......... 68% 23.4M 3s 2023-07-02T14:02:45.655 INFO:tasks.workunit.client.0.smithi163.stderr:115300K .......... .......... .......... .......... .......... 68% 20.7M 3s 2023-07-02T14:02:45.656 INFO:tasks.workunit.client.0.smithi163.stderr:115350K .......... .......... .......... .......... .......... 68% 25.2M 3s 2023-07-02T14:02:45.658 INFO:tasks.workunit.client.0.smithi163.stderr:115400K .......... .......... .......... .......... .......... 68% 24.9M 3s 2023-07-02T14:02:45.661 INFO:tasks.workunit.client.0.smithi163.stderr:115450K .......... .......... .......... .......... .......... 68% 17.6M 3s 2023-07-02T14:02:45.663 INFO:tasks.workunit.client.0.smithi163.stderr:115500K .......... .......... .......... .......... .......... 68% 25.9M 3s 2023-07-02T14:02:45.666 INFO:tasks.workunit.client.0.smithi163.stderr:115550K .......... .......... .......... .......... .......... 68% 14.0M 3s 2023-07-02T14:02:45.668 INFO:tasks.workunit.client.0.smithi163.stderr:115600K .......... .......... .......... .......... .......... 68% 39.0M 3s 2023-07-02T14:02:45.671 INFO:tasks.workunit.client.0.smithi163.stderr:115650K .......... .......... .......... .......... .......... 68% 17.1M 3s 2023-07-02T14:02:45.674 INFO:tasks.workunit.client.0.smithi163.stderr:115700K .......... .......... .......... .......... .......... 68% 17.8M 3s 2023-07-02T14:02:45.675 INFO:tasks.workunit.client.0.smithi163.stderr:115750K .......... .......... .......... .......... .......... 68% 22.4M 3s 2023-07-02T14:02:45.676 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:45.676 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:45.678 INFO:tasks.workunit.client.0.smithi163.stderr:115800K .......... .......... .......... .......... .......... 68% 18.3M 3s 2023-07-02T14:02:45.680 INFO:tasks.workunit.client.0.smithi163.stderr:115850K .......... .......... .......... .......... .......... 68% 32.2M 3s 2023-07-02T14:02:45.682 INFO:tasks.workunit.client.0.smithi163.stderr:115900K .......... .......... .......... .......... .......... 68% 21.5M 3s 2023-07-02T14:02:45.685 INFO:tasks.workunit.client.0.smithi163.stderr:115950K .......... .......... .......... .......... .......... 68% 16.0M 3s 2023-07-02T14:02:45.686 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:02:45.687 INFO:tasks.workunit.client.0.smithi163.stderr:116000K .......... .......... .......... .......... .......... 68% 23.7M 3s 2023-07-02T14:02:45.690 INFO:tasks.workunit.client.0.smithi163.stderr:116050K .......... .......... .......... .......... .......... 68% 18.1M 3s 2023-07-02T14:02:45.691 INFO:tasks.workunit.client.0.smithi163.stderr:116100K .......... .......... .......... .......... .......... 68% 36.4M 3s 2023-07-02T14:02:45.694 INFO:tasks.workunit.client.0.smithi163.stderr:116150K .......... .......... .......... .......... .......... 68% 21.5M 3s 2023-07-02T14:02:45.696 INFO:tasks.workunit.client.0.smithi163.stderr:116200K .......... .......... .......... .......... .......... 68% 16.9M 3s 2023-07-02T14:02:45.698 INFO:tasks.workunit.client.0.smithi163.stderr:116250K .......... .......... .......... .......... .......... 68% 26.7M 3s 2023-07-02T14:02:45.700 INFO:tasks.workunit.client.0.smithi163.stderr:116300K .......... .......... .......... .......... .......... 69% 24.4M 3s 2023-07-02T14:02:45.704 INFO:tasks.workunit.client.0.smithi163.stderr:116350K .......... .......... .......... .......... .......... 69% 13.3M 3s 2023-07-02T14:02:45.706 INFO:tasks.workunit.client.0.smithi163.stderr:116400K .......... .......... .......... .......... .......... 69% 23.6M 3s 2023-07-02T14:02:45.708 INFO:tasks.workunit.client.0.smithi163.stderr:116450K .......... .......... .......... .......... .......... 69% 19.3M 3s 2023-07-02T14:02:45.711 INFO:tasks.workunit.client.0.smithi163.stderr:116500K .......... .......... .......... .......... .......... 69% 24.9M 3s 2023-07-02T14:02:45.713 INFO:tasks.workunit.client.0.smithi163.stderr:116550K .......... .......... .......... .......... .......... 69% 16.8M 3s 2023-07-02T14:02:45.715 INFO:tasks.workunit.client.0.smithi163.stderr:116600K .......... .......... .......... .......... .......... 69% 26.1M 3s 2023-07-02T14:02:45.718 INFO:tasks.workunit.client.0.smithi163.stderr:116650K .......... .......... .......... .......... .......... 69% 18.6M 3s 2023-07-02T14:02:45.721 INFO:tasks.workunit.client.0.smithi163.stderr:116700K .......... .......... .......... .......... .......... 69% 18.6M 3s 2023-07-02T14:02:45.724 INFO:tasks.workunit.client.0.smithi163.stderr:116750K .......... .......... .......... .......... .......... 69% 16.2M 3s 2023-07-02T14:02:45.725 INFO:tasks.workunit.client.0.smithi163.stderr:116800K .......... .......... .......... .......... .......... 69% 29.3M 3s 2023-07-02T14:02:45.727 INFO:tasks.workunit.client.0.smithi163.stderr:116850K .......... .......... .......... .......... .......... 69% 30.9M 3s 2023-07-02T14:02:45.728 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:45.727+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:45.729 INFO:tasks.workunit.client.0.smithi163.stderr:116900K .......... .......... .......... .......... .......... 69% 21.0M 3s 2023-07-02T14:02:45.732 INFO:tasks.workunit.client.0.smithi163.stderr:116950K .......... .......... .......... .......... .......... 69% 14.6M 3s 2023-07-02T14:02:45.735 INFO:tasks.workunit.client.0.smithi163.stderr:117000K .......... .......... .......... .......... .......... 69% 23.6M 3s 2023-07-02T14:02:45.738 INFO:tasks.workunit.client.0.smithi163.stderr:117050K .......... .......... .......... .......... .......... 69% 15.9M 3s 2023-07-02T14:02:45.739 INFO:tasks.workunit.client.0.smithi163.stderr:117100K .......... .......... .......... .......... .......... 69% 38.6M 3s 2023-07-02T14:02:45.742 INFO:tasks.workunit.client.0.smithi163.stderr:117150K .......... .......... .......... .......... .......... 69% 14.9M 3s 2023-07-02T14:02:45.744 INFO:tasks.workunit.client.0.smithi163.stderr:117200K .......... .......... .......... .......... .......... 69% 22.2M 3s 2023-07-02T14:02:45.748 INFO:tasks.workunit.client.0.smithi163.stderr:117250K .......... .......... .......... .......... .......... 69% 14.8M 3s 2023-07-02T14:02:45.749 INFO:tasks.workunit.client.0.smithi163.stderr:117300K .......... .......... .......... .......... .......... 69% 36.4M 3s 2023-07-02T14:02:45.751 INFO:tasks.workunit.client.0.smithi163.stderr:117350K .......... .......... .......... .......... .......... 69% 22.8M 3s 2023-07-02T14:02:45.754 INFO:tasks.workunit.client.0.smithi163.stderr:117400K .......... .......... .......... .......... .......... 69% 19.7M 3s 2023-07-02T14:02:45.757 INFO:tasks.workunit.client.0.smithi163.stderr:117450K .......... .......... .......... .......... .......... 69% 17.0M 3s 2023-07-02T14:02:45.759 INFO:tasks.workunit.client.0.smithi163.stderr:117500K .......... .......... .......... .......... .......... 69% 23.7M 3s 2023-07-02T14:02:45.761 INFO:tasks.workunit.client.0.smithi163.stderr:117550K .......... .......... .......... .......... .......... 69% 22.5M 3s 2023-07-02T14:02:45.764 INFO:tasks.workunit.client.0.smithi163.stderr:117600K .......... .......... .......... .......... .......... 69% 14.9M 3s 2023-07-02T14:02:45.765 INFO:tasks.workunit.client.0.smithi163.stderr:117650K .......... .......... .......... .......... .......... 69% 31.4M 3s 2023-07-02T14:02:45.768 INFO:tasks.workunit.client.0.smithi163.stderr:117700K .......... .......... .......... .......... .......... 69% 18.8M 3s 2023-07-02T14:02:45.770 INFO:tasks.workunit.client.0.smithi163.stderr:117750K .......... .......... .......... .......... .......... 69% 24.0M 3s 2023-07-02T14:02:45.773 INFO:tasks.workunit.client.0.smithi163.stderr:117800K .......... .......... .......... .......... .......... 69% 16.8M 3s 2023-07-02T14:02:45.774 INFO:tasks.workunit.client.0.smithi163.stderr:117850K .......... .......... .......... .......... .......... 69% 33.3M 3s 2023-07-02T14:02:45.777 INFO:tasks.workunit.client.0.smithi163.stderr:117900K .......... .......... .......... .......... .......... 69% 18.2M 3s 2023-07-02T14:02:45.780 INFO:tasks.workunit.client.0.smithi163.stderr:117950K .......... .......... .......... .......... .......... 70% 16.4M 3s 2023-07-02T14:02:45.783 INFO:tasks.workunit.client.0.smithi163.stderr:118000K .......... .......... .......... .......... .......... 70% 19.2M 3s 2023-07-02T14:02:45.785 INFO:tasks.workunit.client.0.smithi163.stderr:118050K .......... .......... .......... .......... .......... 70% 24.4M 3s 2023-07-02T14:02:45.787 INFO:tasks.workunit.client.0.smithi163.stderr:118100K .......... .......... .......... .......... .......... 70% 18.1M 3s 2023-07-02T14:02:45.789 INFO:tasks.workunit.client.0.smithi163.stderr:118150K .......... .......... .......... .......... .......... 70% 31.7M 3s 2023-07-02T14:02:45.791 INFO:tasks.workunit.client.0.smithi163.stderr:118200K .......... .......... .......... .......... .......... 70% 20.9M 3s 2023-07-02T14:02:45.794 INFO:tasks.workunit.client.0.smithi163.stderr:118250K .......... .......... .......... .......... .......... 70% 18.0M 3s 2023-07-02T14:02:45.797 INFO:tasks.workunit.client.0.smithi163.stderr:118300K .......... .......... .......... .......... .......... 70% 19.1M 3s 2023-07-02T14:02:45.800 INFO:tasks.workunit.client.0.smithi163.stderr:118350K .......... .......... .......... .......... .......... 70% 15.8M 3s 2023-07-02T14:02:45.801 INFO:tasks.workunit.client.0.smithi163.stderr:118400K .......... .......... .......... .......... .......... 70% 33.2M 3s 2023-07-02T14:02:45.804 INFO:tasks.workunit.client.0.smithi163.stderr:118450K .......... .......... .......... .......... .......... 70% 20.2M 2s 2023-07-02T14:02:45.806 INFO:tasks.workunit.client.0.smithi163.stderr:118500K .......... .......... .......... .......... .......... 70% 17.3M 2s 2023-07-02T14:02:45.807 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:45.807+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:45.808 INFO:tasks.workunit.client.0.smithi163.stderr:118550K .......... .......... .......... .......... .......... 70% 22.8M 2s 2023-07-02T14:02:45.810 INFO:tasks.workunit.client.0.smithi163.stderr:118600K .......... .......... .......... .......... .......... 70% 31.3M 2s 2023-07-02T14:02:45.813 INFO:tasks.workunit.client.0.smithi163.stderr:118650K .......... .......... .......... .......... .......... 70% 19.6M 2s 2023-07-02T14:02:45.813 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:45.813 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:45.815 INFO:tasks.workunit.client.0.smithi163.stderr:118700K .......... .......... .......... .......... .......... 70% 16.7M 2s 2023-07-02T14:02:45.818 INFO:tasks.workunit.client.0.smithi163.stderr:118750K .......... .......... .......... .......... .......... 70% 17.4M 2s 2023-07-02T14:02:45.820 INFO:tasks.workunit.client.0.smithi163.stderr:118800K .......... .......... .......... .......... .......... 70% 25.3M 2s 2023-07-02T14:02:45.823 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:02:45.823 INFO:tasks.workunit.client.0.smithi163.stderr:118850K .......... .......... .......... .......... .......... 70% 16.1M 2s 2023-07-02T14:02:45.825 INFO:tasks.workunit.client.0.smithi163.stderr:118900K .......... .......... .......... .......... .......... 70% 28.7M 2s 2023-07-02T14:02:45.827 INFO:tasks.workunit.client.0.smithi163.stderr:118950K .......... .......... .......... .......... .......... 70% 22.4M 2s 2023-07-02T14:02:45.830 INFO:tasks.workunit.client.0.smithi163.stderr:119000K .......... .......... .......... .......... .......... 70% 16.4M 2s 2023-07-02T14:02:45.832 INFO:tasks.workunit.client.0.smithi163.stderr:119050K .......... .......... .......... .......... .......... 70% 23.5M 2s 2023-07-02T14:02:45.834 INFO:tasks.workunit.client.0.smithi163.stderr:119100K .......... .......... .......... .......... .......... 70% 32.9M 2s 2023-07-02T14:02:45.837 INFO:tasks.workunit.client.0.smithi163.stderr:119150K .......... .......... .......... .......... .......... 70% 18.4M 2s 2023-07-02T14:02:45.840 INFO:tasks.workunit.client.0.smithi163.stderr:119200K .......... .......... .......... .......... .......... 70% 14.7M 2s 2023-07-02T14:02:45.841 INFO:tasks.workunit.client.0.smithi163.stderr:119250K .......... .......... .......... .......... .......... 70% 25.2M 2s 2023-07-02T14:02:45.844 INFO:tasks.workunit.client.0.smithi163.stderr:119300K .......... .......... .......... .......... .......... 70% 17.8M 2s 2023-07-02T14:02:45.847 INFO:tasks.workunit.client.0.smithi163.stderr:119350K .......... .......... .......... .......... .......... 70% 20.6M 2s 2023-07-02T14:02:45.848 INFO:tasks.workunit.client.0.smithi163.stderr:119400K .......... .......... .......... .......... .......... 70% 30.7M 2s 2023-07-02T14:02:45.850 INFO:tasks.workunit.client.0.smithi163.stderr:119450K .......... .......... .......... .......... .......... 70% 21.4M 2s 2023-07-02T14:02:45.853 INFO:tasks.workunit.client.0.smithi163.stderr:119500K .......... .......... .......... .......... .......... 70% 16.5M 2s 2023-07-02T14:02:45.857 INFO:tasks.workunit.client.0.smithi163.stderr:119550K .......... .......... .......... .......... .......... 70% 16.3M 2s 2023-07-02T14:02:45.859 INFO:tasks.workunit.client.0.smithi163.stderr:119600K .......... .......... .......... .......... .......... 70% 21.4M 2s 2023-07-02T14:02:45.860 INFO:tasks.workunit.client.0.smithi163.stderr:119650K .......... .......... .......... .......... .......... 71% 41.6M 2s 2023-07-02T14:02:45.862 INFO:tasks.workunit.client.0.smithi163.stderr:119700K .......... .......... .......... .......... .......... 71% 21.7M 2s 2023-07-02T14:02:45.866 INFO:tasks.workunit.client.0.smithi163.stderr:119750K .......... .......... .......... .......... .......... 71% 15.5M 2s 2023-07-02T14:02:45.867 INFO:tasks.workunit.client.0.smithi163.stderr:119800K .......... .......... .......... .......... .......... 71% 28.4M 2s 2023-07-02T14:02:45.871 INFO:tasks.workunit.client.0.smithi163.stderr:119850K .......... .......... .......... .......... .......... 71% 14.6M 2s 2023-07-02T14:02:45.872 INFO:tasks.workunit.client.0.smithi163.stderr:119900K .......... .......... .......... .......... .......... 71% 33.6M 2s 2023-07-02T14:02:45.874 INFO:tasks.workunit.client.0.smithi163.stderr:119950K .......... .......... .......... .......... .......... 71% 19.6M 2s 2023-07-02T14:02:45.877 INFO:tasks.workunit.client.0.smithi163.stderr:120000K .......... .......... .......... .......... .......... 71% 22.3M 2s 2023-07-02T14:02:45.880 INFO:tasks.workunit.client.0.smithi163.stderr:120050K .......... .......... .......... .......... .......... 71% 16.3M 2s 2023-07-02T14:02:45.882 INFO:tasks.workunit.client.0.smithi163.stderr:120100K .......... .......... .......... .......... .......... 71% 24.6M 2s 2023-07-02T14:02:45.883 INFO:tasks.workunit.client.0.smithi163.stderr:120150K .......... .......... .......... .......... .......... 71% 39.1M 2s 2023-07-02T14:02:45.886 INFO:tasks.workunit.client.0.smithi163.stderr:120200K .......... .......... .......... .......... .......... 71% 15.7M 2s 2023-07-02T14:02:45.889 INFO:tasks.workunit.client.0.smithi163.stderr:120250K .......... .......... .......... .......... .......... 71% 17.5M 2s 2023-07-02T14:02:45.891 INFO:tasks.workunit.client.0.smithi163.stderr:120300K .......... .......... .......... .......... .......... 71% 24.5M 2s 2023-07-02T14:02:45.894 INFO:tasks.workunit.client.0.smithi163.stderr:120350K .......... .......... .......... .......... .......... 71% 16.2M 2s 2023-07-02T14:02:45.896 INFO:tasks.workunit.client.0.smithi163.stderr:120400K .......... .......... .......... .......... .......... 71% 18.4M 2s 2023-07-02T14:02:45.897 INFO:tasks.workunit.client.0.smithi163.stderr:120450K .......... .......... .......... .......... .......... 71% 39.3M 2s 2023-07-02T14:02:45.901 INFO:tasks.workunit.client.0.smithi163.stderr:120500K .......... .......... .......... .......... .......... 71% 14.9M 2s 2023-07-02T14:02:45.903 INFO:tasks.workunit.client.0.smithi163.stderr:120550K .......... .......... .......... .......... .......... 71% 23.9M 2s 2023-07-02T14:02:45.904 INFO:tasks.workunit.client.0.smithi163.stderr:120600K .......... .......... .......... .......... .......... 71% 27.6M 2s 2023-07-02T14:02:45.907 INFO:tasks.workunit.client.0.smithi163.stderr:120650K .......... .......... .......... .......... .......... 71% 20.9M 2s 2023-07-02T14:02:45.908 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:45.907+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:45.910 INFO:tasks.workunit.client.0.smithi163.stderr:120700K .......... .......... .......... .......... .......... 71% 18.9M 2s 2023-07-02T14:02:45.913 INFO:tasks.workunit.client.0.smithi163.stderr:120750K .......... .......... .......... .......... .......... 71% 18.4M 2s 2023-07-02T14:02:45.916 INFO:tasks.workunit.client.0.smithi163.stderr:120800K .......... .......... .......... .......... .......... 71% 13.8M 2s 2023-07-02T14:02:45.918 INFO:tasks.workunit.client.0.smithi163.stderr:120850K .......... .......... .......... .......... .......... 71% 28.3M 2s 2023-07-02T14:02:45.920 INFO:tasks.workunit.client.0.smithi163.stderr:120900K .......... .......... .......... .......... .......... 71% 22.0M 2s 2023-07-02T14:02:45.921 INFO:tasks.workunit.client.0.smithi163.stderr:120950K .......... .......... .......... .......... .......... 71% 27.4M 2s 2023-07-02T14:02:45.923 INFO:tasks.workunit.client.0.smithi163.stderr:121000K .......... .......... .......... .......... .......... 71% 23.4M 2s 2023-07-02T14:02:45.925 INFO:tasks.workunit.client.0.smithi163.stderr:121050K .......... .......... .......... .......... .......... 71% 37.0M 2s 2023-07-02T14:02:45.928 INFO:tasks.workunit.client.0.smithi163.stderr:121100K .......... .......... .......... .......... .......... 71% 15.4M 2s 2023-07-02T14:02:45.932 INFO:tasks.workunit.client.0.smithi163.stderr:121150K .......... .......... .......... .......... .......... 71% 12.3M 2s 2023-07-02T14:02:45.933 INFO:tasks.workunit.client.0.smithi163.stderr:121200K .......... .......... .......... .......... .......... 71% 40.2M 2s 2023-07-02T14:02:45.935 INFO:tasks.workunit.client.0.smithi163.stderr:121250K .......... .......... .......... .......... .......... 71% 28.1M 2s 2023-07-02T14:02:45.938 INFO:tasks.workunit.client.0.smithi163.stderr:121300K .......... .......... .......... .......... .......... 71% 20.9M 2s 2023-07-02T14:02:45.939 INFO:tasks.workunit.client.0.smithi163.stderr:121350K .......... .......... .......... .......... .......... 72% 31.1M 2s 2023-07-02T14:02:45.941 INFO:tasks.workunit.client.0.smithi163.stderr:121400K .......... .......... .......... .......... .......... 72% 25.6M 2s 2023-07-02T14:02:45.943 INFO:tasks.workunit.client.0.smithi163.stderr:121450K .......... .......... .......... .......... .......... 72% 22.2M 2s 2023-07-02T14:02:45.947 INFO:tasks.workunit.client.0.smithi163.stderr:121500K .......... .......... .......... .......... .......... 72% 14.0M 2s 2023-07-02T14:02:45.947 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:45.947 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:45.949 INFO:tasks.workunit.client.0.smithi163.stderr:121550K .......... .......... .......... .......... .......... 72% 17.9M 2s 2023-07-02T14:02:45.951 INFO:tasks.workunit.client.0.smithi163.stderr:121600K .......... .......... .......... .......... .......... 72% 28.5M 2s 2023-07-02T14:02:45.954 INFO:tasks.workunit.client.0.smithi163.stderr:121650K .......... .......... .......... .......... .......... 72% 14.2M 2s 2023-07-02T14:02:45.956 INFO:tasks.workunit.client.0.smithi163.stderr:121700K .......... .......... .......... .......... .......... 72% 39.4M 2s 2023-07-02T14:02:45.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:02:45.958 INFO:tasks.workunit.client.0.smithi163.stderr:121750K .......... .......... .......... .......... .......... 72% 16.8M 2s 2023-07-02T14:02:45.963 INFO:tasks.workunit.client.0.smithi163.stderr:121800K .......... .......... .......... .......... .......... 72% 11.8M 2s 2023-07-02T14:02:45.965 INFO:tasks.workunit.client.0.smithi163.stderr:121850K .......... .......... .......... .......... .......... 72% 22.1M 2s 2023-07-02T14:02:45.966 INFO:tasks.workunit.client.0.smithi163.stderr:121900K .......... .......... .......... .......... .......... 72% 40.9M 2s 2023-07-02T14:02:45.969 INFO:tasks.workunit.client.0.smithi163.stderr:121950K .......... .......... .......... .......... .......... 72% 19.7M 2s 2023-07-02T14:02:45.971 INFO:tasks.workunit.client.0.smithi163.stderr:122000K .......... .......... .......... .......... .......... 72% 19.7M 2s 2023-07-02T14:02:45.974 INFO:tasks.workunit.client.0.smithi163.stderr:122050K .......... .......... .......... .......... .......... 72% 18.2M 2s 2023-07-02T14:02:45.977 INFO:tasks.workunit.client.0.smithi163.stderr:122100K .......... .......... .......... .......... .......... 72% 16.8M 2s 2023-07-02T14:02:45.979 INFO:tasks.workunit.client.0.smithi163.stderr:122150K .......... .......... .......... .......... .......... 72% 19.3M 2s 2023-07-02T14:02:45.981 INFO:tasks.workunit.client.0.smithi163.stderr:122200K .......... .......... .......... .......... .......... 72% 37.6M 2s 2023-07-02T14:02:45.983 INFO:tasks.workunit.client.0.smithi163.stderr:122250K .......... .......... .......... .......... .......... 72% 18.9M 2s 2023-07-02T14:02:45.985 INFO:tasks.workunit.client.0.smithi163.stderr:122300K .......... .......... .......... .......... .......... 72% 27.6M 2s 2023-07-02T14:02:45.989 INFO:tasks.workunit.client.0.smithi163.stderr:122350K .......... .......... .......... .......... .......... 72% 12.6M 2s 2023-07-02T14:02:45.991 INFO:tasks.workunit.client.0.smithi163.stderr:122400K .......... .......... .......... .......... .......... 72% 20.3M 2s 2023-07-02T14:02:45.992 INFO:tasks.workunit.client.0.smithi163.stderr:122450K .......... .......... .......... .......... .......... 72% 30.9M 2s 2023-07-02T14:02:45.995 INFO:tasks.workunit.client.0.smithi163.stderr:122500K .......... .......... .......... .......... .......... 72% 21.3M 2s 2023-07-02T14:02:45.998 INFO:tasks.workunit.client.0.smithi163.stderr:122550K .......... .......... .......... .......... .......... 72% 18.0M 2s 2023-07-02T14:02:46.000 INFO:tasks.workunit.client.0.smithi163.stderr:122600K .......... .......... .......... .......... .......... 72% 24.1M 2s 2023-07-02T14:02:46.003 INFO:tasks.workunit.client.0.smithi163.stderr:122650K .......... .......... .......... .......... .......... 72% 17.9M 2s 2023-07-02T14:02:46.003 INFO:tasks.workunit.client.0.smithi163.stderr:122700K .......... .......... .......... .......... .......... 72% 48.8M 2s 2023-07-02T14:02:46.008 INFO:tasks.workunit.client.0.smithi163.stderr:122750K .......... .......... .......... .......... .......... 72% 11.9M 2s 2023-07-02T14:02:46.009 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:46.008+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:46.009 INFO:tasks.workunit.client.0.smithi163.stderr:122800K .......... .......... .......... .......... .......... 72% 23.9M 2s 2023-07-02T14:02:46.012 INFO:tasks.workunit.client.0.smithi163.stderr:122850K .......... .......... .......... .......... .......... 72% 22.5M 2s 2023-07-02T14:02:46.014 INFO:tasks.workunit.client.0.smithi163.stderr:122900K .......... .......... .......... .......... .......... 72% 26.7M 2s 2023-07-02T14:02:46.015 INFO:tasks.workunit.client.0.smithi163.stderr:122950K .......... .......... .......... .......... .......... 72% 32.7M 2s 2023-07-02T14:02:46.017 INFO:tasks.workunit.client.0.smithi163.stderr:123000K .......... .......... .......... .......... .......... 72% 28.6M 2s 2023-07-02T14:02:46.021 INFO:tasks.workunit.client.0.smithi163.stderr:123050K .......... .......... .......... .......... .......... 73% 14.6M 2s 2023-07-02T14:02:46.022 INFO:tasks.workunit.client.0.smithi163.stderr:123100K .......... .......... .......... .......... .......... 73% 21.1M 2s 2023-07-02T14:02:46.026 INFO:tasks.workunit.client.0.smithi163.stderr:123150K .......... .......... .......... .......... .......... 73% 16.2M 2s 2023-07-02T14:02:46.028 INFO:tasks.workunit.client.0.smithi163.stderr:123200K .......... .......... .......... .......... .......... 73% 23.9M 2s 2023-07-02T14:02:46.029 INFO:tasks.workunit.client.0.smithi163.stderr:123250K .......... .......... .......... .......... .......... 73% 37.8M 2s 2023-07-02T14:02:46.031 INFO:tasks.workunit.client.0.smithi163.stderr:123300K .......... .......... .......... .......... .......... 73% 18.8M 2s 2023-07-02T14:02:46.033 INFO:tasks.workunit.client.0.smithi163.stderr:123350K .......... .......... .......... .......... .......... 73% 28.5M 2s 2023-07-02T14:02:46.036 INFO:tasks.workunit.client.0.smithi163.stderr:123400K .......... .......... .......... .......... .......... 73% 19.9M 2s 2023-07-02T14:02:46.038 INFO:tasks.workunit.client.0.smithi163.stderr:123450K .......... .......... .......... .......... .......... 73% 17.7M 2s 2023-07-02T14:02:46.041 INFO:tasks.workunit.client.0.smithi163.stderr:123500K .......... .......... .......... .......... .......... 73% 19.4M 2s 2023-07-02T14:02:46.043 INFO:tasks.workunit.client.0.smithi163.stderr:123550K .......... .......... .......... .......... .......... 73% 19.2M 2s 2023-07-02T14:02:46.046 INFO:tasks.workunit.client.0.smithi163.stderr:123600K .......... .......... .......... .......... .......... 73% 20.8M 2s 2023-07-02T14:02:46.048 INFO:tasks.workunit.client.0.smithi163.stderr:123650K .......... .......... .......... .......... .......... 73% 26.4M 2s 2023-07-02T14:02:46.052 INFO:tasks.workunit.client.0.smithi163.stderr:123700K .......... .......... .......... .......... .......... 73% 10.3M 2s 2023-07-02T14:02:46.054 INFO:tasks.workunit.client.0.smithi163.stderr:123750K .......... .......... .......... .......... .......... 73% 32.2M 2s 2023-07-02T14:02:46.057 INFO:tasks.workunit.client.0.smithi163.stderr:123800K .......... .......... .......... .......... .......... 73% 17.3M 2s 2023-07-02T14:02:46.059 INFO:tasks.workunit.client.0.smithi163.stderr:123850K .......... .......... .......... .......... .......... 73% 23.9M 2s 2023-07-02T14:02:46.062 INFO:tasks.workunit.client.0.smithi163.stderr:123900K .......... .......... .......... .......... .......... 73% 17.4M 2s 2023-07-02T14:02:46.064 INFO:tasks.workunit.client.0.smithi163.stderr:123950K .......... .......... .......... .......... .......... 73% 19.8M 2s 2023-07-02T14:02:46.065 INFO:tasks.workunit.client.0.smithi163.stderr:124000K .......... .......... .......... .......... .......... 73% 41.7M 2s 2023-07-02T14:02:46.069 INFO:tasks.workunit.client.0.smithi163.stderr:124050K .......... .......... .......... .......... .......... 73% 14.4M 2s 2023-07-02T14:02:46.072 INFO:tasks.workunit.client.0.smithi163.stderr:124100K .......... .......... .......... .......... .......... 73% 18.0M 2s 2023-07-02T14:02:46.073 INFO:tasks.workunit.client.0.smithi163.stderr:124150K .......... .......... .......... .......... .......... 73% 27.8M 2s 2023-07-02T14:02:46.074 INFO:tasks.workunit.client.0.smithi163.stderr:124200K .......... .......... .......... .......... .......... 73% 31.9M 2s 2023-07-02T14:02:46.077 INFO:tasks.workunit.client.0.smithi163.stderr:124250K .......... .......... .......... .......... .......... 73% 17.6M 2s 2023-07-02T14:02:46.079 INFO:tasks.workunit.client.0.smithi163.stderr:124300K .......... .......... .......... .......... .......... 73% 26.7M 2s 2023-07-02T14:02:46.080 INFO:teuthology.orchestra.run.smithi016.stderr:set 2.0 pg_upmap mapping to [4,1] 2023-07-02T14:02:46.084 INFO:tasks.workunit.client.0.smithi163.stderr:124350K .......... .......... .......... .......... .......... 73% 11.7M 2s 2023-07-02T14:02:46.085 INFO:tasks.workunit.client.0.smithi163.stderr:124400K .......... .......... .......... .......... .......... 73% 25.3M 2s 2023-07-02T14:02:46.088 INFO:tasks.workunit.client.0.smithi163.stderr:124450K .......... .......... .......... .......... .......... 73% 23.3M 2s 2023-07-02T14:02:46.089 INFO:tasks.workunit.client.0.smithi163.stderr:124500K .......... .......... .......... .......... .......... 73% 32.3M 2s 2023-07-02T14:02:46.092 INFO:tasks.workunit.client.0.smithi163.stderr:124550K .......... .......... .......... .......... .......... 73% 18.0M 2s 2023-07-02T14:02:46.094 INFO:tasks.workunit.client.0.smithi163.stderr:124600K .......... .......... .......... .......... .......... 73% 19.3M 2s 2023-07-02T14:02:46.096 INFO:tasks.workunit.client.0.smithi163.stderr:124650K .......... .......... .......... .......... .......... 73% 34.1M 2s 2023-07-02T14:02:46.097 INFO:tasks.workunit.client.0.smithi163.stderr:124700K .......... .......... .......... .......... .......... 74% 23.7M 2s 2023-07-02T14:02:46.101 INFO:tasks.workunit.client.0.smithi163.stderr:124750K .......... .......... .......... .......... .......... 74% 14.1M 2s 2023-07-02T14:02:46.103 INFO:tasks.workunit.client.0.smithi163.stderr:124800K .......... .......... .......... .......... .......... 74% 27.5M 2s 2023-07-02T14:02:46.106 INFO:tasks.workunit.client.0.smithi163.stderr:124850K .......... .......... .......... .......... .......... 74% 17.6M 2s 2023-07-02T14:02:46.108 INFO:tasks.workunit.client.0.smithi163.stderr:124900K .......... .......... .......... .......... .......... 74% 26.1M 2s 2023-07-02T14:02:46.109 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:46.108+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:46.110 INFO:tasks.workunit.client.0.smithi163.stderr:124950K .......... .......... .......... .......... .......... 74% 20.3M 2s 2023-07-02T14:02:46.111 INFO:tasks.workunit.client.0.smithi163.stderr:125000K .......... .......... .......... .......... .......... 74% 35.0M 2s 2023-07-02T14:02:46.115 INFO:tasks.workunit.client.0.smithi163.stderr:125050K .......... .......... .......... .......... .......... 74% 13.3M 2s 2023-07-02T14:02:46.117 INFO:tasks.workunit.client.0.smithi163.stderr:125100K .......... .......... .......... .......... .......... 74% 29.8M 2s 2023-07-02T14:02:46.120 INFO:tasks.workunit.client.0.smithi163.stderr:125150K .......... .......... .......... .......... .......... 74% 15.2M 2s 2023-07-02T14:02:46.122 INFO:tasks.workunit.client.0.smithi163.stderr:125200K .......... .......... .......... .......... .......... 74% 26.4M 2s 2023-07-02T14:02:46.123 INFO:tasks.workunit.client.0.smithi163.stderr:125250K .......... .......... .......... .......... .......... 74% 32.2M 2s 2023-07-02T14:02:46.124 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:46.124 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:46.125 INFO:tasks.workunit.client.0.smithi163.stderr:125300K .......... .......... .......... .......... .......... 74% 23.5M 2s 2023-07-02T14:02:46.130 INFO:tasks.workunit.client.0.smithi163.stderr:125350K .......... .......... .......... .......... .......... 74% 10.7M 2s 2023-07-02T14:02:46.131 INFO:tasks.workunit.client.0.smithi163.stderr:125400K .......... .......... .......... .......... .......... 74% 45.5M 2s 2023-07-02T14:02:46.134 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:02:46.135 INFO:tasks.workunit.client.0.smithi163.stderr:125450K .......... .......... .......... .......... .......... 74% 15.6M 2s 2023-07-02T14:02:46.135 INFO:tasks.workunit.client.0.smithi163.stderr:125500K .......... .......... .......... .......... .......... 74% 41.7M 2s 2023-07-02T14:02:46.138 INFO:tasks.workunit.client.0.smithi163.stderr:125550K .......... .......... .......... .......... .......... 74% 15.9M 2s 2023-07-02T14:02:46.140 INFO:tasks.workunit.client.0.smithi163.stderr:125600K .......... .......... .......... .......... .......... 74% 21.0M 2s 2023-07-02T14:02:46.143 INFO:tasks.workunit.client.0.smithi163.stderr:125650K .......... .......... .......... .......... .......... 74% 24.1M 2s 2023-07-02T14:02:46.147 INFO:tasks.workunit.client.0.smithi163.stderr:125700K .......... .......... .......... .......... .......... 74% 11.9M 2s 2023-07-02T14:02:46.149 INFO:tasks.workunit.client.0.smithi163.stderr:125750K .......... .......... .......... .......... .......... 74% 24.1M 2s 2023-07-02T14:02:46.151 INFO:tasks.workunit.client.0.smithi163.stderr:125800K .......... .......... .......... .......... .......... 74% 18.9M 2s 2023-07-02T14:02:46.154 INFO:tasks.workunit.client.0.smithi163.stderr:125850K .......... .......... .......... .......... .......... 74% 18.9M 2s 2023-07-02T14:02:46.156 INFO:tasks.workunit.client.0.smithi163.stderr:125900K .......... .......... .......... .......... .......... 74% 29.4M 2s 2023-07-02T14:02:46.158 INFO:tasks.workunit.client.0.smithi163.stderr:125950K .......... .......... .......... .......... .......... 74% 22.3M 2s 2023-07-02T14:02:46.161 INFO:tasks.workunit.client.0.smithi163.stderr:126000K .......... .......... .......... .......... .......... 74% 16.1M 2s 2023-07-02T14:02:46.163 INFO:tasks.workunit.client.0.smithi163.stderr:126050K .......... .......... .......... .......... .......... 74% 26.8M 2s 2023-07-02T14:02:46.164 INFO:tasks.workunit.client.0.smithi163.stderr:126100K .......... .......... .......... .......... .......... 74% 28.2M 2s 2023-07-02T14:02:46.168 INFO:tasks.workunit.client.0.smithi163.stderr:126150K .......... .......... .......... .......... .......... 74% 15.5M 2s 2023-07-02T14:02:46.170 INFO:tasks.workunit.client.0.smithi163.stderr:126200K .......... .......... .......... .......... .......... 74% 18.7M 2s 2023-07-02T14:02:46.171 INFO:tasks.workunit.client.0.smithi163.stderr:126250K .......... .......... .......... .......... .......... 74% 31.3M 2s 2023-07-02T14:02:46.174 INFO:tasks.workunit.client.0.smithi163.stderr:126300K .......... .......... .......... .......... .......... 74% 23.5M 2s 2023-07-02T14:02:46.178 INFO:tasks.workunit.client.0.smithi163.stderr:126350K .......... .......... .......... .......... .......... 74% 11.4M 2s 2023-07-02T14:02:46.180 INFO:tasks.workunit.client.0.smithi163.stderr:126400K .......... .......... .......... .......... .......... 75% 26.0M 2s 2023-07-02T14:02:46.182 INFO:tasks.workunit.client.0.smithi163.stderr:126450K .......... .......... .......... .......... .......... 75% 28.7M 2s 2023-07-02T14:02:46.183 INFO:tasks.workunit.client.0.smithi163.stderr:126500K .......... .......... .......... .......... .......... 75% 23.8M 2s 2023-07-02T14:02:46.186 INFO:tasks.workunit.client.0.smithi163.stderr:126550K .......... .......... .......... .......... .......... 75% 22.5M 2s 2023-07-02T14:02:46.188 INFO:tasks.workunit.client.0.smithi163.stderr:126600K .......... .......... .......... .......... .......... 75% 21.8M 2s 2023-07-02T14:02:46.191 INFO:tasks.workunit.client.0.smithi163.stderr:126650K .......... .......... .......... .......... .......... 75% 19.9M 2s 2023-07-02T14:02:46.194 INFO:tasks.workunit.client.0.smithi163.stderr:126700K .......... .......... .......... .......... .......... 75% 14.8M 2s 2023-07-02T14:02:46.196 INFO:tasks.workunit.client.0.smithi163.stderr:126750K .......... .......... .......... .......... .......... 75% 23.2M 2s 2023-07-02T14:02:46.197 INFO:tasks.workunit.client.0.smithi163.stderr:126800K .......... .......... .......... .......... .......... 75% 34.0M 2s 2023-07-02T14:02:46.200 INFO:tasks.workunit.client.0.smithi163.stderr:126850K .......... .......... .......... .......... .......... 75% 19.9M 2s 2023-07-02T14:02:46.202 INFO:tasks.workunit.client.0.smithi163.stderr:126900K .......... .......... .......... .......... .......... 75% 20.0M 2s 2023-07-02T14:02:46.204 INFO:tasks.workunit.client.0.smithi163.stderr:126950K .......... .......... .......... .......... .......... 75% 31.3M 2s 2023-07-02T14:02:46.207 INFO:tasks.workunit.client.0.smithi163.stderr:127000K .......... .......... .......... .......... .......... 75% 14.6M 2s 2023-07-02T14:02:46.210 INFO:tasks.workunit.client.0.smithi163.stderr:127050K .......... .......... .......... .......... .......... 75% 21.6M 2s 2023-07-02T14:02:46.211 INFO:tasks.workunit.client.0.smithi163.stderr:127100K .......... .......... .......... .......... .......... 75% 24.1M 2s 2023-07-02T14:02:46.214 INFO:tasks.workunit.client.0.smithi163.stderr:127150K .......... .......... .......... .......... .......... 75% 17.6M 2s 2023-07-02T14:02:46.216 INFO:tasks.workunit.client.0.smithi163.stderr:127200K .......... .......... .......... .......... .......... 75% 28.7M 2s 2023-07-02T14:02:46.217 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:46.216+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:46.219 INFO:tasks.workunit.client.0.smithi163.stderr:127250K .......... .......... .......... .......... .......... 75% 17.3M 2s 2023-07-02T14:02:46.221 INFO:tasks.workunit.client.0.smithi163.stderr:127300K .......... .......... .......... .......... .......... 75% 18.0M 2s 2023-07-02T14:02:46.224 INFO:tasks.workunit.client.0.smithi163.stderr:127350K .......... .......... .......... .......... .......... 75% 20.0M 2s 2023-07-02T14:02:46.226 INFO:tasks.workunit.client.0.smithi163.stderr:127400K .......... .......... .......... .......... .......... 75% 26.0M 2s 2023-07-02T14:02:46.228 INFO:tasks.workunit.client.0.smithi163.stderr:127450K .......... .......... .......... .......... .......... 75% 21.0M 2s 2023-07-02T14:02:46.230 INFO:tasks.workunit.client.0.smithi163.stderr:127500K .......... .......... .......... .......... .......... 75% 29.0M 2s 2023-07-02T14:02:46.233 INFO:tasks.workunit.client.0.smithi163.stderr:127550K .......... .......... .......... .......... .......... 75% 15.4M 2s 2023-07-02T14:02:46.235 INFO:tasks.workunit.client.0.smithi163.stderr:127600K .......... .......... .......... .......... .......... 75% 23.6M 2s 2023-07-02T14:02:46.239 INFO:tasks.workunit.client.0.smithi163.stderr:127650K .......... .......... .......... .......... .......... 75% 11.6M 2s 2023-07-02T14:02:46.241 INFO:tasks.workunit.client.0.smithi163.stderr:127700K .......... .......... .......... .......... .......... 75% 37.5M 2s 2023-07-02T14:02:46.244 INFO:tasks.workunit.client.0.smithi163.stderr:127750K .......... .......... .......... .......... .......... 75% 13.5M 2s 2023-07-02T14:02:46.245 INFO:tasks.workunit.client.0.smithi163.stderr:127800K .......... .......... .......... .......... .......... 75% 44.8M 2s 2023-07-02T14:02:46.247 INFO:tasks.workunit.client.0.smithi163.stderr:127850K .......... .......... .......... .......... .......... 75% 21.4M 2s 2023-07-02T14:02:46.250 INFO:tasks.workunit.client.0.smithi163.stderr:127900K .......... .......... .......... .......... .......... 75% 24.1M 2s 2023-07-02T14:02:46.252 INFO:tasks.workunit.client.0.smithi163.stderr:127950K .......... .......... .......... .......... .......... 75% 22.6M 2s 2023-07-02T14:02:46.254 INFO:tasks.workunit.client.0.smithi163.stderr:128000K .......... .......... .......... .......... .......... 75% 25.0M 2s 2023-07-02T14:02:46.255 INFO:tasks.workunit.client.0.smithi163.stderr:128050K .......... .......... .......... .......... .......... 75% 43.7M 2s 2023-07-02T14:02:46.257 INFO:tasks.workunit.client.0.smithi163.stderr:128100K .......... .......... .......... .......... .......... 76% 20.2M 2s 2023-07-02T14:02:46.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:02:46.263 INFO:tasks.workunit.client.0.smithi163.stderr:128150K .......... .......... .......... .......... .......... 76% 8.90M 2s 2023-07-02T14:02:46.263 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:02:46.264 INFO:tasks.workunit.client.0.smithi163.stderr:128200K .......... .......... .......... .......... .......... 76% 35.7M 2s 2023-07-02T14:02:46.267 INFO:tasks.workunit.client.0.smithi163.stderr:128250K .......... .......... .......... .......... .......... 76% 18.1M 2s 2023-07-02T14:02:46.269 INFO:tasks.workunit.client.0.smithi163.stderr:128300K .......... .......... .......... .......... .......... 76% 26.4M 2s 2023-07-02T14:02:46.272 INFO:tasks.workunit.client.0.smithi163.stderr:128350K .......... .......... .......... .......... .......... 76% 14.0M 2s 2023-07-02T14:02:46.274 INFO:tasks.workunit.client.0.smithi163.stderr:128400K .......... .......... .......... .......... .......... 76% 24.2M 2s 2023-07-02T14:02:46.276 INFO:tasks.workunit.client.0.smithi163.stderr:128450K .......... .......... .......... .......... .......... 76% 25.0M 2s 2023-07-02T14:02:46.279 INFO:tasks.workunit.client.0.smithi163.stderr:128500K .......... .......... .......... .......... .......... 76% 18.1M 2s 2023-07-02T14:02:46.280 INFO:tasks.workunit.client.0.smithi163.stderr:128550K .......... .......... .......... .......... .......... 76% 30.1M 2s 2023-07-02T14:02:46.282 INFO:tasks.workunit.client.0.smithi163.stderr:128600K .......... .......... .......... .......... .......... 76% 23.7M 2s 2023-07-02T14:02:46.287 INFO:tasks.workunit.client.0.smithi163.stderr:128650K .......... .......... .......... .......... .......... 76% 11.6M 2s 2023-07-02T14:02:46.288 INFO:tasks.workunit.client.0.smithi163.stderr:128700K .......... .......... .......... .......... .......... 76% 43.8M 2s 2023-07-02T14:02:46.290 INFO:tasks.workunit.client.0.smithi163.stderr:128750K .......... .......... .......... .......... .......... 76% 21.2M 2s 2023-07-02T14:02:46.293 INFO:tasks.workunit.client.0.smithi163.stderr:128800K .......... .......... .......... .......... .......... 76% 18.3M 2s 2023-07-02T14:02:46.294 INFO:tasks.workunit.client.0.smithi163.stderr:128850K .......... .......... .......... .......... .......... 76% 53.0M 2s 2023-07-02T14:02:46.296 INFO:tasks.workunit.client.0.smithi163.stderr:128900K .......... .......... .......... .......... .......... 76% 16.3M 2s 2023-07-02T14:02:46.300 INFO:tasks.workunit.client.0.smithi163.stderr:128950K .......... .......... .......... .......... .......... 76% 15.8M 2s 2023-07-02T14:02:46.300 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:46.301 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:46.302 INFO:tasks.workunit.client.0.smithi163.stderr:129000K .......... .......... .......... .......... .......... 76% 18.9M 2s 2023-07-02T14:02:46.304 INFO:tasks.workunit.client.0.smithi163.stderr:129050K .......... .......... .......... .......... .......... 76% 31.1M 2s 2023-07-02T14:02:46.306 INFO:tasks.workunit.client.0.smithi163.stderr:129100K .......... .......... .......... .......... .......... 76% 28.0M 2s 2023-07-02T14:02:46.309 INFO:tasks.workunit.client.0.smithi163.stderr:129150K .......... .......... .......... .......... .......... 76% 16.7M 2s 2023-07-02T14:02:46.310 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:46.310+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:46.312 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:02:46.313 INFO:tasks.workunit.client.0.smithi163.stderr:129200K .......... .......... .......... .......... .......... 76% 14.5M 2s 2023-07-02T14:02:46.314 INFO:tasks.workunit.client.0.smithi163.stderr:129250K .......... .......... .......... .......... .......... 76% 23.0M 2s 2023-07-02T14:02:46.316 INFO:tasks.workunit.client.0.smithi163.stderr:129300K .......... .......... .......... .......... .......... 76% 21.8M 2s 2023-07-02T14:02:46.318 INFO:tasks.workunit.client.0.smithi163.stderr:129350K .......... .......... .......... .......... .......... 76% 23.9M 2s 2023-07-02T14:02:46.321 INFO:tasks.workunit.client.0.smithi163.stderr:129400K .......... .......... .......... .......... .......... 76% 16.3M 2s 2023-07-02T14:02:46.323 INFO:tasks.workunit.client.0.smithi163.stderr:129450K .......... .......... .......... .......... .......... 76% 28.8M 2s 2023-07-02T14:02:46.326 INFO:tasks.workunit.client.0.smithi163.stderr:129500K .......... .......... .......... .......... .......... 76% 18.1M 2s 2023-07-02T14:02:46.329 INFO:tasks.workunit.client.0.smithi163.stderr:129550K .......... .......... .......... .......... .......... 76% 18.3M 2s 2023-07-02T14:02:46.330 INFO:tasks.workunit.client.0.smithi163.stderr:129600K .......... .......... .......... .......... .......... 76% 22.7M 2s 2023-07-02T14:02:46.333 INFO:tasks.workunit.client.0.smithi163.stderr:129650K .......... .......... .......... .......... .......... 76% 20.6M 2s 2023-07-02T14:02:46.335 INFO:tasks.workunit.client.0.smithi163.stderr:129700K .......... .......... .......... .......... .......... 76% 23.7M 2s 2023-07-02T14:02:46.337 INFO:tasks.workunit.client.0.smithi163.stderr:129750K .......... .......... .......... .......... .......... 77% 24.9M 2s 2023-07-02T14:02:46.339 INFO:tasks.workunit.client.0.smithi163.stderr:129800K .......... .......... .......... .......... .......... 77% 22.5M 2s 2023-07-02T14:02:46.341 INFO:tasks.workunit.client.0.smithi163.stderr:129850K .......... .......... .......... .......... .......... 77% 31.6M 2s 2023-07-02T14:02:46.344 INFO:tasks.workunit.client.0.smithi163.stderr:129900K .......... .......... .......... .......... .......... 77% 15.6M 2s 2023-07-02T14:02:46.346 INFO:tasks.workunit.client.0.smithi163.stderr:129950K .......... .......... .......... .......... .......... 77% 18.9M 2s 2023-07-02T14:02:46.348 INFO:tasks.workunit.client.0.smithi163.stderr:130000K .......... .......... .......... .......... .......... 77% 37.1M 2s 2023-07-02T14:02:46.352 INFO:tasks.workunit.client.0.smithi163.stderr:130050K .......... .......... .......... .......... .......... 77% 10.9M 2s 2023-07-02T14:02:46.355 INFO:tasks.workunit.client.0.smithi163.stderr:130100K .......... .......... .......... .......... .......... 77% 17.7M 2s 2023-07-02T14:02:46.357 INFO:tasks.workunit.client.0.smithi163.stderr:130150K .......... .......... .......... .......... .......... 77% 25.7M 2s 2023-07-02T14:02:46.359 INFO:tasks.workunit.client.0.smithi163.stderr:130200K .......... .......... .......... .......... .......... 77% 26.2M 2s 2023-07-02T14:02:46.361 INFO:tasks.workunit.client.0.smithi163.stderr:130250K .......... .......... .......... .......... .......... 77% 16.3M 2s 2023-07-02T14:02:46.364 INFO:tasks.workunit.client.0.smithi163.stderr:130300K .......... .......... .......... .......... .......... 77% 19.6M 2s 2023-07-02T14:02:46.366 INFO:tasks.workunit.client.0.smithi163.stderr:130350K .......... .......... .......... .......... .......... 77% 21.5M 2s 2023-07-02T14:02:46.369 INFO:tasks.workunit.client.0.smithi163.stderr:130400K .......... .......... .......... .......... .......... 77% 23.1M 2s 2023-07-02T14:02:46.371 INFO:tasks.workunit.client.0.smithi163.stderr:130450K .......... .......... .......... .......... .......... 77% 21.7M 2s 2023-07-02T14:02:46.372 INFO:tasks.workunit.client.0.smithi163.stderr:130500K .......... .......... .......... .......... .......... 77% 32.1M 2s 2023-07-02T14:02:46.376 INFO:tasks.workunit.client.0.smithi163.stderr:130550K .......... .......... .......... .......... .......... 77% 12.9M 2s 2023-07-02T14:02:46.379 INFO:tasks.workunit.client.0.smithi163.stderr:130600K .......... .......... .......... .......... .......... 77% 18.0M 2s 2023-07-02T14:02:46.380 INFO:tasks.workunit.client.0.smithi163.stderr:130650K .......... .......... .......... .......... .......... 77% 33.0M 2s 2023-07-02T14:02:46.382 INFO:tasks.workunit.client.0.smithi163.stderr:130700K .......... .......... .......... .......... .......... 77% 29.5M 2s 2023-07-02T14:02:46.384 INFO:tasks.workunit.client.0.smithi163.stderr:130750K .......... .......... .......... .......... .......... 77% 23.2M 2s 2023-07-02T14:02:46.387 INFO:tasks.workunit.client.0.smithi163.stderr:130800K .......... .......... .......... .......... .......... 77% 18.3M 2s 2023-07-02T14:02:46.388 INFO:tasks.workunit.client.0.smithi163.stderr:130850K .......... .......... .......... .......... .......... 77% 30.6M 2s 2023-07-02T14:02:46.392 INFO:tasks.workunit.client.0.smithi163.stderr:130900K .......... .......... .......... .......... .......... 77% 13.6M 2s 2023-07-02T14:02:46.395 INFO:tasks.workunit.client.0.smithi163.stderr:130950K .......... .......... .......... .......... .......... 77% 18.9M 2s 2023-07-02T14:02:46.396 INFO:tasks.workunit.client.0.smithi163.stderr:131000K .......... .......... .......... .......... .......... 77% 29.5M 2s 2023-07-02T14:02:46.398 INFO:tasks.workunit.client.0.smithi163.stderr:131050K .......... .......... .......... .......... .......... 77% 29.5M 2s 2023-07-02T14:02:46.400 INFO:tasks.workunit.client.0.smithi163.stderr:131100K .......... .......... .......... .......... .......... 77% 22.1M 2s 2023-07-02T14:02:46.404 INFO:tasks.workunit.client.0.smithi163.stderr:131150K .......... .......... .......... .......... .......... 77% 13.0M 2s 2023-07-02T14:02:46.406 INFO:tasks.workunit.client.0.smithi163.stderr:131200K .......... .......... .......... .......... .......... 77% 21.8M 2s 2023-07-02T14:02:46.409 INFO:tasks.workunit.client.0.smithi163.stderr:131250K .......... .......... .......... .......... .......... 77% 21.7M 2s 2023-07-02T14:02:46.411 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:46.410+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:46.412 INFO:tasks.workunit.client.0.smithi163.stderr:131300K .......... .......... .......... .......... .......... 77% 18.8M 2s 2023-07-02T14:02:46.412 INFO:tasks.workunit.client.0.smithi163.stderr:131350K .......... .......... .......... .......... .......... 77% 34.1M 2s 2023-07-02T14:02:46.416 INFO:tasks.workunit.client.0.smithi163.stderr:131400K .......... .......... .......... .......... .......... 77% 16.8M 2s 2023-07-02T14:02:46.418 INFO:tasks.workunit.client.0.smithi163.stderr:131450K .......... .......... .......... .......... .......... 78% 18.2M 2s 2023-07-02T14:02:46.420 INFO:tasks.workunit.client.0.smithi163.stderr:131500K .......... .......... .......... .......... .......... 78% 32.3M 2s 2023-07-02T14:02:46.423 INFO:tasks.workunit.client.0.smithi163.stderr:131550K .......... .......... .......... .......... .......... 78% 15.6M 2s 2023-07-02T14:02:46.424 INFO:tasks.workunit.client.0.smithi163.stderr:131600K .......... .......... .......... .......... .......... 78% 35.3M 2s 2023-07-02T14:02:46.427 INFO:tasks.workunit.client.0.smithi163.stderr:131650K .......... .......... .......... .......... .......... 78% 19.4M 2s 2023-07-02T14:02:46.429 INFO:tasks.workunit.client.0.smithi163.stderr:131700K .......... .......... .......... .......... .......... 78% 18.9M 2s 2023-07-02T14:02:46.432 INFO:tasks.workunit.client.0.smithi163.stderr:131750K .......... .......... .......... .......... .......... 78% 19.2M 2s 2023-07-02T14:02:46.434 INFO:tasks.workunit.client.0.smithi163.stderr:131800K .......... .......... .......... .......... .......... 78% 18.6M 2s 2023-07-02T14:02:46.436 INFO:tasks.workunit.client.0.smithi163.stderr:131850K .......... .......... .......... .......... .......... 78% 25.4M 2s 2023-07-02T14:02:46.438 INFO:tasks.workunit.client.0.smithi163.stderr:131900K .......... .......... .......... .......... .......... 78% 29.6M 2s 2023-07-02T14:02:46.441 INFO:tasks.workunit.client.0.smithi163.stderr:131950K .......... .......... .......... .......... .......... 78% 16.6M 2s 2023-07-02T14:02:46.442 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:46.445 INFO:tasks.workunit.client.0.smithi163.stderr:132000K .......... .......... .......... .......... .......... 78% 12.4M 2s 2023-07-02T14:02:46.446 INFO:tasks.workunit.client.0.smithi163.stderr:132050K .......... .......... .......... .......... .......... 78% 25.7M 2s 2023-07-02T14:02:46.448 INFO:tasks.workunit.client.0.smithi163.stderr:132100K .......... .......... .......... .......... .......... 78% 24.1M 2s 2023-07-02T14:02:46.451 INFO:tasks.workunit.client.0.smithi163.stderr:132150K .......... .......... .......... .......... .......... 78% 20.7M 2s 2023-07-02T14:02:46.454 INFO:tasks.workunit.client.0.smithi163.stderr:132200K .......... .......... .......... .......... .......... 78% 15.2M 2s 2023-07-02T14:02:46.457 INFO:tasks.workunit.client.0.smithi163.stderr:132250K .......... .......... .......... .......... .......... 78% 20.6M 2s 2023-07-02T14:02:46.459 INFO:tasks.workunit.client.0.smithi163.stderr:132300K .......... .......... .......... .......... .......... 78% 22.5M 2s 2023-07-02T14:02:46.460 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:46.461 INFO:tasks.workunit.client.0.smithi163.stderr:132350K .......... .......... .......... .......... .......... 78% 19.4M 2s 2023-07-02T14:02:46.462 INFO:tasks.workunit.client.0.smithi163.stderr:132400K .......... .......... .......... .......... .......... 78% 34.4M 2s 2023-07-02T14:02:46.465 INFO:tasks.workunit.client.0.smithi163.stderr:132450K .......... .......... .......... .......... .......... 78% 24.4M 2s 2023-07-02T14:02:46.466 INFO:tasks.workunit.client.0.smithi163.stderr:132500K .......... .......... .......... .......... .......... 78% 31.1M 2s 2023-07-02T14:02:46.470 INFO:tasks.workunit.client.0.smithi163.stderr:132550K .......... .......... .......... .......... .......... 78% 11.8M 2s 2023-07-02T14:02:46.473 INFO:tasks.workunit.client.0.smithi163.stderr:132600K .......... .......... .......... .......... .......... 78% 22.8M 2s 2023-07-02T14:02:46.475 INFO:tasks.workunit.client.0.smithi163.stderr:132650K .......... .......... .......... .......... .......... 78% 24.9M 2s 2023-07-02T14:02:46.477 INFO:tasks.workunit.client.0.smithi163.stderr:132700K .......... .......... .......... .......... .......... 78% 22.5M 2s 2023-07-02T14:02:46.479 INFO:tasks.workunit.client.0.smithi163.stderr:132750K .......... .......... .......... .......... .......... 78% 18.6M 2s 2023-07-02T14:02:46.480 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:46.481 INFO:tasks.workunit.client.0.smithi163.stderr:132800K .......... .......... .......... .......... .......... 78% 34.2M 2s 2023-07-02T14:02:46.485 INFO:tasks.workunit.client.0.smithi163.stderr:132850K .......... .......... .......... .......... .......... 78% 10.7M 2s 2023-07-02T14:02:46.487 INFO:tasks.workunit.client.0.smithi163.stderr:132900K .......... .......... .......... .......... .......... 78% 31.3M 2s 2023-07-02T14:02:46.488 INFO:tasks.workunit.client.0.smithi163.stderr:132950K .......... .......... .......... .......... .......... 78% 25.8M 2s 2023-07-02T14:02:46.490 INFO:tasks.workunit.client.0.smithi163.stderr:133000K .......... .......... .......... .......... .......... 78% 28.4M 2s 2023-07-02T14:02:46.493 INFO:tasks.workunit.client.0.smithi163.stderr:133050K .......... .......... .......... .......... .......... 78% 17.4M 2s 2023-07-02T14:02:46.495 INFO:tasks.workunit.client.0.smithi163.stderr:133100K .......... .......... .......... .......... .......... 78% 30.9M 2s 2023-07-02T14:02:46.498 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:46.498 INFO:tasks.workunit.client.0.smithi163.stderr:133150K .......... .......... .......... .......... .......... 79% 14.6M 2s 2023-07-02T14:02:46.499 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:46.499 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:46.499 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:46.499 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:46.499 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:46.501 INFO:tasks.workunit.client.0.smithi163.stderr:133200K .......... .......... .......... .......... .......... 79% 17.2M 2s 2023-07-02T14:02:46.504 INFO:tasks.workunit.client.0.smithi163.stderr:133250K .......... .......... .......... .......... .......... 79% 17.3M 2s 2023-07-02T14:02:46.505 INFO:tasks.workunit.client.0.smithi163.stderr:133300K .......... .......... .......... .......... .......... 79% 40.7M 2s 2023-07-02T14:02:46.508 INFO:tasks.workunit.client.0.smithi163.stderr:133350K .......... .......... .......... .......... .......... 79% 15.9M 2s 2023-07-02T14:02:46.510 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:02:46.510 INFO:tasks.workunit.client.0.smithi163.stderr:133400K .......... .......... .......... .......... .......... 79% 27.6M 2s 2023-07-02T14:02:46.512 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:46.510+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:46.512 INFO:tasks.workunit.client.0.smithi163.stderr:133450K .......... .......... .......... .......... .......... 79% 30.8M 2s 2023-07-02T14:02:46.514 INFO:tasks.workunit.client.0.smithi163.stderr:133500K .......... .......... .......... .......... .......... 79% 19.9M 2s 2023-07-02T14:02:46.517 INFO:tasks.workunit.client.0.smithi163.stderr:133550K .......... .......... .......... .......... .......... 79% 17.7M 2s 2023-07-02T14:02:46.517 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:46.520 INFO:tasks.workunit.client.0.smithi163.stderr:133600K .......... .......... .......... .......... .......... 79% 16.8M 2s 2023-07-02T14:02:46.521 INFO:tasks.workunit.client.0.smithi163.stderr:133650K .......... .......... .......... .......... .......... 79% 30.6M 2s 2023-07-02T14:02:46.524 INFO:tasks.workunit.client.0.smithi163.stderr:133700K .......... .......... .......... .......... .......... 79% 16.6M 2s 2023-07-02T14:02:46.527 INFO:tasks.workunit.client.0.smithi163.stderr:133750K .......... .......... .......... .......... .......... 79% 17.8M 2s 2023-07-02T14:02:46.528 INFO:tasks.workunit.client.0.smithi163.stderr:133800K .......... .......... .......... .......... .......... 79% 27.3M 2s 2023-07-02T14:02:46.530 INFO:tasks.workunit.client.0.smithi163.stderr:133850K .......... .......... .......... .......... .......... 79% 40.0M 2s 2023-07-02T14:02:46.533 INFO:tasks.workunit.client.0.smithi163.stderr:133900K .......... .......... .......... .......... .......... 79% 15.1M 2s 2023-07-02T14:02:46.536 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:46.537 INFO:tasks.workunit.client.0.smithi163.stderr:133950K .......... .......... .......... .......... .......... 79% 13.7M 2s 2023-07-02T14:02:46.537 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:02:46.537 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:02:46.537 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:02:46.538 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:02:46.538 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:02:46.538 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:02:46.539 INFO:tasks.workunit.client.0.smithi163.stderr:134000K .......... .......... .......... .......... .......... 79% 23.3M 2s 2023-07-02T14:02:46.541 INFO:tasks.workunit.client.0.smithi163.stderr:134050K .......... .......... .......... .......... .......... 79% 23.2M 2s 2023-07-02T14:02:46.544 INFO:tasks.workunit.client.0.smithi163.stderr:134100K .......... .......... .......... .......... .......... 79% 16.8M 2s 2023-07-02T14:02:46.545 INFO:tasks.workunit.client.0.smithi163.stderr:134150K .......... .......... .......... .......... .......... 79% 36.7M 2s 2023-07-02T14:02:46.549 INFO:tasks.workunit.client.0.smithi163.stderr:134200K .......... .......... .......... .......... .......... 79% 12.5M 2s 2023-07-02T14:02:46.551 INFO:tasks.workunit.client.0.smithi163.stderr:134250K .......... .......... .......... .......... .......... 79% 29.2M 2s 2023-07-02T14:02:46.553 INFO:tasks.workunit.client.0.smithi163.stderr:134300K .......... .......... .......... .......... .......... 79% 22.8M 2s 2023-07-02T14:02:46.555 INFO:tasks.workunit.client.0.smithi163.stderr:134350K .......... .......... .......... .......... .......... 79% 19.9M 2s 2023-07-02T14:02:46.557 INFO:tasks.workunit.client.0.smithi163.stderr:134400K .......... .......... .......... .......... .......... 79% 34.5M 2s 2023-07-02T14:02:46.559 INFO:tasks.workunit.client.0.smithi163.stderr:134450K .......... .......... .......... .......... .......... 79% 21.4M 2s 2023-07-02T14:02:46.562 INFO:tasks.workunit.client.0.smithi163.stderr:134500K .......... .......... .......... .......... .......... 79% 15.2M 2s 2023-07-02T14:02:46.565 INFO:tasks.workunit.client.0.smithi163.stderr:134550K .......... .......... .......... .......... .......... 79% 15.9M 2s 2023-07-02T14:02:46.568 INFO:tasks.workunit.client.0.smithi163.stderr:134600K .......... .......... .......... .......... .......... 79% 20.8M 2s 2023-07-02T14:02:46.569 INFO:tasks.workunit.client.0.smithi163.stderr:134650K .......... .......... .......... .......... .......... 79% 38.0M 2s 2023-07-02T14:02:46.571 INFO:tasks.workunit.client.0.smithi163.stderr:134700K .......... .......... .......... .......... .......... 79% 23.2M 2s 2023-07-02T14:02:46.574 INFO:tasks.workunit.client.0.smithi163.stderr:134750K .......... .......... .......... .......... .......... 79% 20.7M 2s 2023-07-02T14:02:46.578 INFO:tasks.workunit.client.0.smithi163.stderr:134800K .......... .......... .......... .......... .......... 79% 12.6M 2s 2023-07-02T14:02:46.579 INFO:tasks.workunit.client.0.smithi163.stderr:134850K .......... .......... .......... .......... .......... 80% 25.5M 2s 2023-07-02T14:02:46.581 INFO:tasks.workunit.client.0.smithi163.stderr:134900K .......... .......... .......... .......... .......... 80% 34.5M 2s 2023-07-02T14:02:46.583 INFO:tasks.workunit.client.0.smithi163.stderr:134950K .......... .......... .......... .......... .......... 80% 23.2M 2s 2023-07-02T14:02:46.585 INFO:tasks.workunit.client.0.smithi163.stderr:135000K .......... .......... .......... .......... .......... 80% 19.5M 2s 2023-07-02T14:02:46.587 INFO:tasks.workunit.client.0.smithi163.stderr:135050K .......... .......... .......... .......... .......... 80% 26.5M 2s 2023-07-02T14:02:46.591 INFO:tasks.workunit.client.0.smithi163.stderr:135100K .......... .......... .......... .......... .......... 80% 13.9M 2s 2023-07-02T14:02:46.594 INFO:tasks.workunit.client.0.smithi163.stderr:135150K .......... .......... .......... .......... .......... 80% 15.4M 2s 2023-07-02T14:02:46.595 INFO:tasks.workunit.client.0.smithi163.stderr:135200K .......... .......... .......... .......... .......... 80% 38.0M 2s 2023-07-02T14:02:46.597 INFO:tasks.workunit.client.0.smithi163.stderr:135250K .......... .......... .......... .......... .......... 80% 23.4M 2s 2023-07-02T14:02:46.599 INFO:tasks.workunit.client.0.smithi163.stderr:135300K .......... .......... .......... .......... .......... 80% 27.2M 2s 2023-07-02T14:02:46.602 INFO:tasks.workunit.client.0.smithi163.stderr:135350K .......... .......... .......... .......... .......... 80% 15.4M 2s 2023-07-02T14:02:46.604 INFO:tasks.workunit.client.0.smithi163.stderr:135400K .......... .......... .......... .......... .......... 80% 29.7M 2s 2023-07-02T14:02:46.606 INFO:tasks.workunit.client.0.smithi163.stderr:135450K .......... .......... .......... .......... .......... 80% 21.3M 2s 2023-07-02T14:02:46.609 INFO:tasks.workunit.client.0.smithi163.stderr:135500K .......... .......... .......... .......... .......... 80% 16.7M 2s 2023-07-02T14:02:46.612 INFO:tasks.workunit.client.0.smithi163.stderr:135550K .......... .......... .......... .......... .......... 80% 18.0M 2s 2023-07-02T14:02:46.613 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:46.611+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:46.614 INFO:tasks.workunit.client.0.smithi163.stderr:135600K .......... .......... .......... .......... .......... 80% 25.7M 2s 2023-07-02T14:02:46.617 INFO:tasks.workunit.client.0.smithi163.stderr:135650K .......... .......... .......... .......... .......... 80% 15.5M 2s 2023-07-02T14:02:46.618 INFO:tasks.workunit.client.0.smithi163.stderr:135700K .......... .......... .......... .......... .......... 80% 34.4M 2s 2023-07-02T14:02:46.621 INFO:tasks.workunit.client.0.smithi163.stderr:135750K .......... .......... .......... .......... .......... 80% 16.0M 2s 2023-07-02T14:02:46.622 INFO:tasks.workunit.client.0.smithi163.stderr:135800K .......... .......... .......... .......... .......... 80% 36.7M 2s 2023-07-02T14:02:46.626 INFO:tasks.workunit.client.0.smithi163.stderr:135850K .......... .......... .......... .......... .......... 80% 15.2M 2s 2023-07-02T14:02:46.629 INFO:tasks.workunit.client.0.smithi163.stderr:135900K .......... .......... .......... .......... .......... 80% 15.3M 2s 2023-07-02T14:02:46.635 INFO:tasks.workunit.client.0.smithi163.stderr:135950K .......... .......... .......... .......... .......... 80% 9.67M 2s 2023-07-02T14:02:46.636 INFO:tasks.workunit.client.0.smithi163.stderr:136000K .......... .......... .......... .......... .......... 80% 209M 2s 2023-07-02T14:02:46.637 INFO:tasks.workunit.client.0.smithi163.stderr:136050K .......... .......... .......... .......... .......... 80% 18.6M 2s 2023-07-02T14:02:46.638 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:46.639 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:46.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:46.639 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:02:46.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.639 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:250 2.2 2:44c63967:::rbd_data.125de1449605.0000000000000101:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:46.639 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.299603+0000", 2023-07-02T14:02:46.640 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.3364856080000003, 2023-07-02T14:02:46.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037580609000000001, 2023-07-02T14:02:46.640 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:46.640 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:46.640 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:46.640 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:46.641 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:46.641 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 250 2023-07-02T14:02:46.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.641 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:46.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:46.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.299603+0000", 2023-07-02T14:02:46.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:46.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:46.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.299603+0000", 2023-07-02T14:02:46.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:46.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:46.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.299602+0000", 2023-07-02T14:02:46.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022150699999999999 2023-07-02T14:02:46.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:46.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.299824+0000", 2023-07-02T14:02:46.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1405e-05 2023-07-02T14:02:46.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:46.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.299835+0000", 2023-07-02T14:02:46.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1590000000000004e-06 2023-07-02T14:02:46.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:46.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.299842+0000", 2023-07-02T14:02:46.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017232279999999999 2023-07-02T14:02:46.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:46.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317074+0000", 2023-07-02T14:02:46.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5123999999999994e-05 2023-07-02T14:02:46.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:46.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317150+0000", 2023-07-02T14:02:46.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014873800000000001 2023-07-02T14:02:46.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:46.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317298+0000", 2023-07-02T14:02:46.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035949480000000002 2023-07-02T14:02:46.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:46.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320893+0000", 2023-07-02T14:02:46.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016194844999999999 2023-07-02T14:02:46.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:46.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337088+0000", 2023-07-02T14:02:46.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1582999999999997e-05 2023-07-02T14:02:46.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:46.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337140+0000", 2023-07-02T14:02:46.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3779000000000002e-05 2023-07-02T14:02:46.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:46.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337183+0000", 2023-07-02T14:02:46.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037580609000000001 2023-07-02T14:02:46.650 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.650 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.651 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.651 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:256 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:46.651 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.300278+0000", 2023-07-02T14:02:46.651 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.3358101859999998, 2023-07-02T14:02:46.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037365890999999998, 2023-07-02T14:02:46.652 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:46.652 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:46.652 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:46.652 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:46.652 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:46.652 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 256 2023-07-02T14:02:46.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.653 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:46.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:46.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300278+0000", 2023-07-02T14:02:46.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:46.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:46.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300278+0000", 2023-07-02T14:02:46.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:46.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:46.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300278+0000", 2023-07-02T14:02:46.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00066792800000000003 2023-07-02T14:02:46.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:46.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300945+0000", 2023-07-02T14:02:46.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0150000000000003e-06 2023-07-02T14:02:46.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:46.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300954+0000", 2023-07-02T14:02:46.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9069999999999998e-06 2023-07-02T14:02:46.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:46.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300958+0000", 2023-07-02T14:02:46.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018796822000000001 2023-07-02T14:02:46.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:46.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319755+0000", 2023-07-02T14:02:46.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126038 2023-07-02T14:02:46.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:46.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319881+0000", 2023-07-02T14:02:46.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015656700000000001 2023-07-02T14:02:46.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:46.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320038+0000", 2023-07-02T14:02:46.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017229343000000001 2023-07-02T14:02:46.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:46.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337267+0000", 2023-07-02T14:02:46.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000277378 2023-07-02T14:02:46.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:46.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337545+0000", 2023-07-02T14:02:46.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4978000000000001e-05 2023-07-02T14:02:46.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:46.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337600+0000", 2023-07-02T14:02:46.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4635000000000002e-05 2023-07-02T14:02:46.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:46.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337644+0000", 2023-07-02T14:02:46.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037365890999999998 2023-07-02T14:02:46.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.662 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.663 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:279 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:46.663 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.310714+0000", 2023-07-02T14:02:46.663 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.3253747220000003, 2023-07-02T14:02:46.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038119238, 2023-07-02T14:02:46.663 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:46.663 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:46.664 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:46.664 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:46.664 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:46.664 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 279 2023-07-02T14:02:46.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.664 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:46.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:46.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310714+0000", 2023-07-02T14:02:46.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:46.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:46.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310714+0000", 2023-07-02T14:02:46.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:46.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:46.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310713+0000", 2023-07-02T14:02:46.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8973e-05 2023-07-02T14:02:46.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:46.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310762+0000", 2023-07-02T14:02:46.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0530000000000007e-06 2023-07-02T14:02:46.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:46.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310770+0000", 2023-07-02T14:02:46.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7569999999999993e-06 2023-07-02T14:02:46.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:46.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310779+0000", 2023-07-02T14:02:46.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033655263999999997 2023-07-02T14:02:46.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:46.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.344434+0000", 2023-07-02T14:02:46.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4365999999999999e-05 2023-07-02T14:02:46.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:46.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.344508+0000", 2023-07-02T14:02:46.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016449100000000001 2023-07-02T14:02:46.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:46.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.344673+0000", 2023-07-02T14:02:46.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019402379999999999 2023-07-02T14:02:46.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:46.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346613+0000", 2023-07-02T14:02:46.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021203900000000002 2023-07-02T14:02:46.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:46.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348734+0000", 2023-07-02T14:02:46.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5225000000000001e-05 2023-07-02T14:02:46.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:46.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348789+0000", 2023-07-02T14:02:46.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4163999999999999e-05 2023-07-02T14:02:46.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:46.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348833+0000", 2023-07-02T14:02:46.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038119238 2023-07-02T14:02:46.674 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.674 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.674 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.675 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:280 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:46.675 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.310785+0000", 2023-07-02T14:02:46.675 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.325303849, 2023-07-02T14:02:46.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038149954999999999, 2023-07-02T14:02:46.675 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:46.675 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:46.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:46.676 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:46.676 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:46.676 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 280 2023-07-02T14:02:46.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.676 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:46.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:46.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310785+0000", 2023-07-02T14:02:46.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:46.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:46.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310785+0000", 2023-07-02T14:02:46.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:46.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:46.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310784+0000", 2023-07-02T14:02:46.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4364999999999999e-05 2023-07-02T14:02:46.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:46.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310838+0000", 2023-07-02T14:02:46.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4619999999999996e-06 2023-07-02T14:02:46.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:46.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310847+0000", 2023-07-02T14:02:46.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5129999999999998e-06 2023-07-02T14:02:46.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:46.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310851+0000", 2023-07-02T14:02:46.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034215981999999999 2023-07-02T14:02:46.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:46.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.345067+0000", 2023-07-02T14:02:46.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6117000000000002e-05 2023-07-02T14:02:46.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:46.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.345153+0000", 2023-07-02T14:02:46.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014067499999999999 2023-07-02T14:02:46.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:46.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.345294+0000", 2023-07-02T14:02:46.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013560810000000001 2023-07-02T14:02:46.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:46.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346650+0000", 2023-07-02T14:02:46.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002196239 2023-07-02T14:02:46.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:46.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348846+0000", 2023-07-02T14:02:46.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7621999999999999e-05 2023-07-02T14:02:46.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:46.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348894+0000", 2023-07-02T14:02:46.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0518000000000001e-05 2023-07-02T14:02:46.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:46.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348935+0000", 2023-07-02T14:02:46.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038149954999999999 2023-07-02T14:02:46.686 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.686 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.686 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.686 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:281 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:46.686 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.310857+0000", 2023-07-02T14:02:46.687 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.3252315870000002, 2023-07-02T14:02:46.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039347695000000002, 2023-07-02T14:02:46.687 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:46.687 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:46.687 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:46.687 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:46.687 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:46.688 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:02:46.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.688 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:46.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:46.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310857+0000", 2023-07-02T14:02:46.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:46.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:46.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310857+0000", 2023-07-02T14:02:46.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:46.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:46.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310856+0000", 2023-07-02T14:02:46.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00081158100000000002 2023-07-02T14:02:46.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:46.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311668+0000", 2023-07-02T14:02:46.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.257e-05 2023-07-02T14:02:46.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:46.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311680+0000", 2023-07-02T14:02:46.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1110000000000002e-06 2023-07-02T14:02:46.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:46.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311688+0000", 2023-07-02T14:02:46.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034985653999999998 2023-07-02T14:02:46.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:46.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346673+0000", 2023-07-02T14:02:46.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7062999999999999e-05 2023-07-02T14:02:46.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:46.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346730+0000", 2023-07-02T14:02:46.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000151061 2023-07-02T14:02:46.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:46.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346881+0000", 2023-07-02T14:02:46.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018356119999999999 2023-07-02T14:02:46.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:46.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348717+0000", 2023-07-02T14:02:46.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001417385 2023-07-02T14:02:46.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:46.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350134+0000", 2023-07-02T14:02:46.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7036999999999997e-05 2023-07-02T14:02:46.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:46.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350181+0000", 2023-07-02T14:02:46.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3187999999999999e-05 2023-07-02T14:02:46.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:46.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350205+0000", 2023-07-02T14:02:46.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039347695000000002 2023-07-02T14:02:46.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.698 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.698 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:282 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:46.698 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311695+0000", 2023-07-02T14:02:46.698 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.3243930930000003, 2023-07-02T14:02:46.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038558453999999999, 2023-07-02T14:02:46.699 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:46.699 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:46.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:46.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:46.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:46.700 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 282 2023-07-02T14:02:46.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.700 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:46.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:46.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311695+0000", 2023-07-02T14:02:46.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:46.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:46.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311695+0000", 2023-07-02T14:02:46.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:46.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:46.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311695+0000", 2023-07-02T14:02:46.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6679999999999999e-05 2023-07-02T14:02:46.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:46.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311751+0000", 2023-07-02T14:02:46.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9199999999999993e-06 2023-07-02T14:02:46.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:46.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311760+0000", 2023-07-02T14:02:46.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2480000000000004e-06 2023-07-02T14:02:46.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:46.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311765+0000", 2023-07-02T14:02:46.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035670150999999997 2023-07-02T14:02:46.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:46.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347436+0000", 2023-07-02T14:02:46.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7861000000000006e-05 2023-07-02T14:02:46.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:46.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347503+0000", 2023-07-02T14:02:46.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000217443 2023-07-02T14:02:46.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:46.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347721+0000", 2023-07-02T14:02:46.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023992449999999999 2023-07-02T14:02:46.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:46.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350120+0000", 2023-07-02T14:02:46.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2219000000000006e-05 2023-07-02T14:02:46.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:46.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350212+0000", 2023-07-02T14:02:46.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3430000000000001e-05 2023-07-02T14:02:46.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:46.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350236+0000", 2023-07-02T14:02:46.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8074999999999999e-05 2023-07-02T14:02:46.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:46.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350254+0000", 2023-07-02T14:02:46.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038558453999999999 2023-07-02T14:02:46.709 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.709 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.709 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:283 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311771+0000", 2023-07-02T14:02:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.3243172220000003, 2023-07-02T14:02:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039285701999999999, 2023-07-02T14:02:46.710 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:46.711 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:46.711 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:46.711 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:46.711 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:46.711 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 283 2023-07-02T14:02:46.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.711 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311771+0000", 2023-07-02T14:02:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:46.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311771+0000", 2023-07-02T14:02:46.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:46.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:46.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311771+0000", 2023-07-02T14:02:46.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6775999999999999e-05 2023-07-02T14:02:46.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:46.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311837+0000", 2023-07-02T14:02:46.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4290000000000001e-06 2023-07-02T14:02:46.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311846+0000", 2023-07-02T14:02:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3800000000000002e-06 2023-07-02T14:02:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:46.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311851+0000", 2023-07-02T14:02:46.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036308287000000002 2023-07-02T14:02:46.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:46.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348160+0000", 2023-07-02T14:02:46.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6475000000000002e-05 2023-07-02T14:02:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348236+0000", 2023-07-02T14:02:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000147966 2023-07-02T14:02:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:46.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348384+0000", 2023-07-02T14:02:46.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019115569999999999 2023-07-02T14:02:46.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:46.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350296+0000", 2023-07-02T14:02:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00063248000000000004 2023-07-02T14:02:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350928+0000", 2023-07-02T14:02:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3355000000000003e-05 2023-07-02T14:02:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:46.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351021+0000", 2023-07-02T14:02:46.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5596000000000003e-05 2023-07-02T14:02:46.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:46.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351057+0000", 2023-07-02T14:02:46.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039285701999999999 2023-07-02T14:02:46.721 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.721 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.721 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.721 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:284 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:46.722 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311859+0000", 2023-07-02T14:02:46.722 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.324229925, 2023-07-02T14:02:46.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039967282999999999, 2023-07-02T14:02:46.722 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:46.722 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:46.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:46.723 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:46.723 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:46.723 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 284 2023-07-02T14:02:46.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.723 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:46.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:46.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311859+0000", 2023-07-02T14:02:46.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:46.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:46.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311859+0000", 2023-07-02T14:02:46.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:46.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:46.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311858+0000", 2023-07-02T14:02:46.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3902999999999997e-05 2023-07-02T14:02:46.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:46.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311892+0000", 2023-07-02T14:02:46.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0989999999999999e-06 2023-07-02T14:02:46.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:46.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311900+0000", 2023-07-02T14:02:46.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9030000000000003e-06 2023-07-02T14:02:46.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:46.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311905+0000", 2023-07-02T14:02:46.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037060899000000001 2023-07-02T14:02:46.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:46.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348966+0000", 2023-07-02T14:02:46.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3609000000000002e-05 2023-07-02T14:02:46.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:46.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349009+0000", 2023-07-02T14:02:46.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6831999999999998e-05 2023-07-02T14:02:46.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:46.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349096+0000", 2023-07-02T14:02:46.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001975077 2023-07-02T14:02:46.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:46.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351071+0000", 2023-07-02T14:02:46.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00066166900000000004 2023-07-02T14:02:46.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:46.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351733+0000", 2023-07-02T14:02:46.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3199999999999999e-05 2023-07-02T14:02:46.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:46.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351786+0000", 2023-07-02T14:02:46.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9823999999999997e-05 2023-07-02T14:02:46.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:46.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351826+0000", 2023-07-02T14:02:46.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039967282999999999 2023-07-02T14:02:46.733 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.733 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.733 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.733 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:285 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:46.734 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311910+0000", 2023-07-02T14:02:46.734 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.324178893, 2023-07-02T14:02:46.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040060302999999998, 2023-07-02T14:02:46.734 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:46.734 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:46.734 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:46.734 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:46.735 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:46.735 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 285 2023-07-02T14:02:46.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.735 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:46.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:46.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311910+0000", 2023-07-02T14:02:46.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:46.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:46.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311910+0000", 2023-07-02T14:02:46.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:46.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:46.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311909+0000", 2023-07-02T14:02:46.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1025999999999997e-05 2023-07-02T14:02:46.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:46.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311940+0000", 2023-07-02T14:02:46.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4769999999999997e-06 2023-07-02T14:02:46.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:46.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311948+0000", 2023-07-02T14:02:46.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5059999999999997e-06 2023-07-02T14:02:46.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:46.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311952+0000", 2023-07-02T14:02:46.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037480332999999998 2023-07-02T14:02:46.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:46.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349432+0000", 2023-07-02T14:02:46.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2923000000000005e-05 2023-07-02T14:02:46.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:46.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349515+0000", 2023-07-02T14:02:46.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015231700000000001 2023-07-02T14:02:46.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:46.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349668+0000", 2023-07-02T14:02:46.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002171849 2023-07-02T14:02:46.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:46.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351839+0000", 2023-07-02T14:02:46.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2052000000000003e-05 2023-07-02T14:02:46.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:46.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351882+0000", 2023-07-02T14:02:46.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7683999999999999e-05 2023-07-02T14:02:46.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:46.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351929+0000", 2023-07-02T14:02:46.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0648000000000001e-05 2023-07-02T14:02:46.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:46.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351970+0000", 2023-07-02T14:02:46.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040060302999999998 2023-07-02T14:02:46.746 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.746 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.746 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.747 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:286 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:46.747 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311957+0000", 2023-07-02T14:02:46.747 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.3241310259999999, 2023-07-02T14:02:46.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042429263000000002, 2023-07-02T14:02:46.747 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:46.747 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:46.748 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:46.748 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:46.748 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:46.748 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 286 2023-07-02T14:02:46.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.748 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:46.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:46.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311957+0000", 2023-07-02T14:02:46.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:46.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:46.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311957+0000", 2023-07-02T14:02:46.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:46.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:46.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311957+0000", 2023-07-02T14:02:46.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7862000000000003e-05 2023-07-02T14:02:46.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:46.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312015+0000", 2023-07-02T14:02:46.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2770000000000005e-06 2023-07-02T14:02:46.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:46.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312023+0000", 2023-07-02T14:02:46.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0000000000000004e-06 2023-07-02T14:02:46.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:46.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312028+0000", 2023-07-02T14:02:46.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038291327999999999 2023-07-02T14:02:46.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:46.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350319+0000", 2023-07-02T14:02:46.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3686000000000001e-05 2023-07-02T14:02:46.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:46.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350373+0000", 2023-07-02T14:02:46.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012420000000000001 2023-07-02T14:02:46.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:46.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350497+0000", 2023-07-02T14:02:46.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014893669999999999 2023-07-02T14:02:46.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:46.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351986+0000", 2023-07-02T14:02:46.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022881580000000002 2023-07-02T14:02:46.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:46.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354275+0000", 2023-07-02T14:02:46.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7388e-05 2023-07-02T14:02:46.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:46.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354332+0000", 2023-07-02T14:02:46.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4688999999999998e-05 2023-07-02T14:02:46.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:46.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354387+0000", 2023-07-02T14:02:46.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042429263000000002 2023-07-02T14:02:46.758 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.758 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.758 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.759 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:287 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 589824~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:46.759 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312033+0000", 2023-07-02T14:02:46.759 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.3240554080000004, 2023-07-02T14:02:46.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042515431999999999, 2023-07-02T14:02:46.759 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:46.759 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:46.760 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:46.760 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:46.760 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:46.760 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 287 2023-07-02T14:02:46.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.760 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:46.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:46.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312033+0000", 2023-07-02T14:02:46.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:46.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:46.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312033+0000", 2023-07-02T14:02:46.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:46.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:46.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312033+0000", 2023-07-02T14:02:46.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1540999999999999e-05 2023-07-02T14:02:46.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:46.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312094+0000", 2023-07-02T14:02:46.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.123e-06 2023-07-02T14:02:46.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:46.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312102+0000", 2023-07-02T14:02:46.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5480000000000003e-06 2023-07-02T14:02:46.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:46.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312107+0000", 2023-07-02T14:02:46.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039019356999999998 2023-07-02T14:02:46.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:46.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351126+0000", 2023-07-02T14:02:46.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100107 2023-07-02T14:02:46.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:46.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351226+0000", 2023-07-02T14:02:46.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177455 2023-07-02T14:02:46.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:46.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351404+0000", 2023-07-02T14:02:46.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030030819999999998 2023-07-02T14:02:46.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:46.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354407+0000", 2023-07-02T14:02:46.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5787000000000001e-05 2023-07-02T14:02:46.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:46.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354443+0000", 2023-07-02T14:02:46.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0627000000000001e-05 2023-07-02T14:02:46.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:46.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354503+0000", 2023-07-02T14:02:46.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5327000000000002e-05 2023-07-02T14:02:46.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:46.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354548+0000", 2023-07-02T14:02:46.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042515431999999999 2023-07-02T14:02:46.770 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.770 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.770 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.770 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:288 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:46.771 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312112+0000", 2023-07-02T14:02:46.771 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.3239764550000004, 2023-07-02T14:02:46.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051367833000000002, 2023-07-02T14:02:46.771 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:46.771 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:46.771 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:46.771 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:46.772 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:46.772 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 288 2023-07-02T14:02:46.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.772 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:46.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:46.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312112+0000", 2023-07-02T14:02:46.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:46.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:46.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312112+0000", 2023-07-02T14:02:46.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312111+0000", 2023-07-02T14:02:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8513999999999998e-05 2023-07-02T14:02:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312150+0000", 2023-07-02T14:02:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6499999999999996e-06 2023-07-02T14:02:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:46.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312157+0000", 2023-07-02T14:02:46.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7799999999999997e-06 2023-07-02T14:02:46.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:46.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312163+0000", 2023-07-02T14:02:46.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039867382 2023-07-02T14:02:46.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:46.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352031+0000", 2023-07-02T14:02:46.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6370000000000002e-05 2023-07-02T14:02:46.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:46.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352107+0000", 2023-07-02T14:02:46.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000156021 2023-07-02T14:02:46.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:46.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352263+0000", 2023-07-02T14:02:46.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023521890000000002 2023-07-02T14:02:46.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:46.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354615+0000", 2023-07-02T14:02:46.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087542309999999995 2023-07-02T14:02:46.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:46.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363369+0000", 2023-07-02T14:02:46.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3217999999999998e-05 2023-07-02T14:02:46.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:46.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363423+0000", 2023-07-02T14:02:46.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7228999999999997e-05 2023-07-02T14:02:46.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:46.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363480+0000", 2023-07-02T14:02:46.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051367833000000002 2023-07-02T14:02:46.781 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.782 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.782 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.782 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:289 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:46.782 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:02:46.782 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.3239204070000001, 2023-07-02T14:02:46.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051413158, 2023-07-02T14:02:46.783 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:46.783 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:46.783 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:46.783 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:46.783 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:46.783 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 289 2023-07-02T14:02:46.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.784 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:46.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:46.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:02:46.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:46.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:46.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:02:46.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:46.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:46.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:02:46.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1689e-05 2023-07-02T14:02:46.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:46.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312209+0000", 2023-07-02T14:02:46.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1070000000000005e-06 2023-07-02T14:02:46.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:46.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312217+0000", 2023-07-02T14:02:46.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.634e-06 2023-07-02T14:02:46.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:46.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312222+0000", 2023-07-02T14:02:46.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040512761000000001 2023-07-02T14:02:46.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:46.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352735+0000", 2023-07-02T14:02:46.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000119184 2023-07-02T14:02:46.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:46.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352854+0000", 2023-07-02T14:02:46.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021481899999999999 2023-07-02T14:02:46.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:46.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353069+0000", 2023-07-02T14:02:46.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001601707 2023-07-02T14:02:46.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:46.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354670+0000", 2023-07-02T14:02:46.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088168570000000009 2023-07-02T14:02:46.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363487+0000", 2023-07-02T14:02:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9635999999999997e-05 2023-07-02T14:02:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363537+0000", 2023-07-02T14:02:46.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4292999999999998e-05 2023-07-02T14:02:46.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:46.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363581+0000", 2023-07-02T14:02:46.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051413158 2023-07-02T14:02:46.793 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.793 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.793 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.794 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:290 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:46.794 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:02:46.794 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.323861162, 2023-07-02T14:02:46.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051455062000000003, 2023-07-02T14:02:46.794 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:46.795 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:46.795 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:46.795 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:46.795 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:46.795 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 290 2023-07-02T14:02:46.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.795 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:46.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:46.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:02:46.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:46.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:46.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:02:46.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:46.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:46.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:02:46.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6469000000000003e-05 2023-07-02T14:02:46.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:46.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312293+0000", 2023-07-02T14:02:46.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4630000000000007e-06 2023-07-02T14:02:46.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:46.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312303+0000", 2023-07-02T14:02:46.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8969999999999999e-06 2023-07-02T14:02:46.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:46.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312307+0000", 2023-07-02T14:02:46.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041261548000000002 2023-07-02T14:02:46.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:46.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353569+0000", 2023-07-02T14:02:46.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9594999999999999e-05 2023-07-02T14:02:46.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:46.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353659+0000", 2023-07-02T14:02:46.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018390600000000001 2023-07-02T14:02:46.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:46.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353843+0000", 2023-07-02T14:02:46.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035173100000000001 2023-07-02T14:02:46.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:46.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357360+0000", 2023-07-02T14:02:46.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062295670000000001 2023-07-02T14:02:46.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:46.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363589+0000", 2023-07-02T14:02:46.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9854000000000001e-05 2023-07-02T14:02:46.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:46.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363639+0000", 2023-07-02T14:02:46.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3133999999999997e-05 2023-07-02T14:02:46.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:46.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363682+0000", 2023-07-02T14:02:46.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051455062000000003 2023-07-02T14:02:46.805 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.805 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.805 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.806 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:291 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:46.806 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:02:46.806 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.3237751229999999, 2023-07-02T14:02:46.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051527860000000002, 2023-07-02T14:02:46.806 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:46.806 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:46.807 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:46.807 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:46.807 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:46.807 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 291 2023-07-02T14:02:46.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.808 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:46.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:46.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:02:46.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:46.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:46.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:02:46.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:46.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:46.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:02:46.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2448999999999998e-05 2023-07-02T14:02:46.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:46.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312375+0000", 2023-07-02T14:02:46.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5259999999999993e-06 2023-07-02T14:02:46.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:46.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312384+0000", 2023-07-02T14:02:46.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0610000000000003e-06 2023-07-02T14:02:46.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:46.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312389+0000", 2023-07-02T14:02:46.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042326111 2023-07-02T14:02:46.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:46.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354715+0000", 2023-07-02T14:02:46.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3998999999999995e-05 2023-07-02T14:02:46.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:46.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354779+0000", 2023-07-02T14:02:46.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010269500000000001 2023-07-02T14:02:46.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:46.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354882+0000", 2023-07-02T14:02:46.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002533736 2023-07-02T14:02:46.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:46.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357415+0000", 2023-07-02T14:02:46.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062765700000000004 2023-07-02T14:02:46.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:46.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363692+0000", 2023-07-02T14:02:46.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7095000000000001e-05 2023-07-02T14:02:46.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:46.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363749+0000", 2023-07-02T14:02:46.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2263999999999996e-05 2023-07-02T14:02:46.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:46.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363841+0000", 2023-07-02T14:02:46.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051527860000000002 2023-07-02T14:02:46.816 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.816 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.816 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.817 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:292 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:46.817 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:02:46.817 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.3236928749999999, 2023-07-02T14:02:46.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051556056000000003, 2023-07-02T14:02:46.817 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:46.817 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:46.818 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:46.818 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:46.818 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:46.818 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 292 2023-07-02T14:02:46.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.818 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:46.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:46.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:02:46.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:46.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:46.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:02:46.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:46.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:46.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312395+0000", 2023-07-02T14:02:46.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9624e-05 2023-07-02T14:02:46.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:46.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312425+0000", 2023-07-02T14:02:46.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:02:46.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:46.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312432+0000", 2023-07-02T14:02:46.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7550000000000002e-06 2023-07-02T14:02:46.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:46.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312437+0000", 2023-07-02T14:02:46.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.043823036000000003 2023-07-02T14:02:46.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:46.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356260+0000", 2023-07-02T14:02:46.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6021999999999997e-05 2023-07-02T14:02:46.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:46.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356346+0000", 2023-07-02T14:02:46.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020076600000000001 2023-07-02T14:02:46.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:46.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356547+0000", 2023-07-02T14:02:46.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020766410000000002 2023-07-02T14:02:46.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:46.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358623+0000", 2023-07-02T14:02:46.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005231192 2023-07-02T14:02:46.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:46.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363854+0000", 2023-07-02T14:02:46.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3423e-05 2023-07-02T14:02:46.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:46.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363908+0000", 2023-07-02T14:02:46.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3825999999999997e-05 2023-07-02T14:02:46.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:46.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363952+0000", 2023-07-02T14:02:46.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051556056000000003 2023-07-02T14:02:46.827 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.827 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.827 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.827 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:293 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:46.827 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:02:46.827 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.3236467049999998, 2023-07-02T14:02:46.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051614769999999997, 2023-07-02T14:02:46.828 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:46.828 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:46.828 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:46.828 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:46.828 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:46.828 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 293 2023-07-02T14:02:46.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.829 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:46.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:46.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:02:46.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:46.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:46.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:02:46.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:46.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:46.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312441+0000", 2023-07-02T14:02:46.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9609999999999999e-05 2023-07-02T14:02:46.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:46.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312471+0000", 2023-07-02T14:02:46.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7069999999999992e-06 2023-07-02T14:02:46.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:46.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312478+0000", 2023-07-02T14:02:46.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1190000000000002e-06 2023-07-02T14:02:46.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:46.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312485+0000", 2023-07-02T14:02:46.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.044956387 2023-07-02T14:02:46.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:46.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357441+0000", 2023-07-02T14:02:46.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4134999999999994e-05 2023-07-02T14:02:46.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:46.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357515+0000", 2023-07-02T14:02:46.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000147408 2023-07-02T14:02:46.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:46.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357663+0000", 2023-07-02T14:02:46.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056472639999999999 2023-07-02T14:02:46.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:46.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363310+0000", 2023-07-02T14:02:46.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00065402600000000004 2023-07-02T14:02:46.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:46.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363964+0000", 2023-07-02T14:02:46.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0497000000000001e-05 2023-07-02T14:02:46.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:46.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364014+0000", 2023-07-02T14:02:46.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2243000000000002e-05 2023-07-02T14:02:46.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:46.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364057+0000", 2023-07-02T14:02:46.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051614769999999997 2023-07-02T14:02:46.837 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.837 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.837 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.838 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:294 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1048576~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:46.838 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:02:46.838 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.3235987060000003, 2023-07-02T14:02:46.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051679527000000003, 2023-07-02T14:02:46.838 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:46.838 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:46.838 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:46.839 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:46.839 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:46.839 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 294 2023-07-02T14:02:46.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.839 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:46.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:46.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:02:46.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:46.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:46.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:02:46.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:46.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:46.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312489+0000", 2023-07-02T14:02:46.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8204999999999998e-05 2023-07-02T14:02:46.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:46.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312517+0000", 2023-07-02T14:02:46.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7759999999999997e-06 2023-07-02T14:02:46.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:46.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312525+0000", 2023-07-02T14:02:46.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8730000000000001e-06 2023-07-02T14:02:46.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:46.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312532+0000", 2023-07-02T14:02:46.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.045442206999999998 2023-07-02T14:02:46.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:46.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357974+0000", 2023-07-02T14:02:46.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4422000000000002e-05 2023-07-02T14:02:46.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:46.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358029+0000", 2023-07-02T14:02:46.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149519 2023-07-02T14:02:46.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:46.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358178+0000", 2023-07-02T14:02:46.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051779390000000003 2023-07-02T14:02:46.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:46.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363356+0000", 2023-07-02T14:02:46.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00070707699999999997 2023-07-02T14:02:46.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:46.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364063+0000", 2023-07-02T14:02:46.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6835999999999999e-05 2023-07-02T14:02:46.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:46.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364090+0000", 2023-07-02T14:02:46.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9263000000000003e-05 2023-07-02T14:02:46.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:46.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364169+0000", 2023-07-02T14:02:46.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051679527000000003 2023-07-02T14:02:46.847 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.848 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.848 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.848 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:423 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [write 2555904~1638400] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:46.848 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.330332+0000", 2023-07-02T14:02:46.848 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.3057567109999999, 2023-07-02T14:02:46.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038020074000000001, 2023-07-02T14:02:46.849 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:46.849 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:46.849 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:46.849 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:46.849 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:46.849 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 423 2023-07-02T14:02:46.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.850 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:46.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:46.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.330332+0000", 2023-07-02T14:02:46.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:46.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:46.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.330332+0000", 2023-07-02T14:02:46.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:02:46.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:46.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.330330+0000", 2023-07-02T14:02:46.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060720390000000004 2023-07-02T14:02:46.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:46.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336402+0000", 2023-07-02T14:02:46.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8478000000000002e-05 2023-07-02T14:02:46.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:46.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336421+0000", 2023-07-02T14:02:46.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0624999999999999e-05 2023-07-02T14:02:46.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:46.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336451+0000", 2023-07-02T14:02:46.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022199402 2023-07-02T14:02:46.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:46.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358651+0000", 2023-07-02T14:02:46.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1981000000000001e-05 2023-07-02T14:02:46.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:46.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358723+0000", 2023-07-02T14:02:46.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163117 2023-07-02T14:02:46.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:46.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358886+0000", 2023-07-02T14:02:46.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005296762 2023-07-02T14:02:46.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:46.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364183+0000", 2023-07-02T14:02:46.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040861739999999997 2023-07-02T14:02:46.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:46.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368269+0000", 2023-07-02T14:02:46.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7169000000000003e-05 2023-07-02T14:02:46.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:46.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368306+0000", 2023-07-02T14:02:46.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5760999999999998e-05 2023-07-02T14:02:46.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:46.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368352+0000", 2023-07-02T14:02:46.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038020074000000001 2023-07-02T14:02:46.858 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.858 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.858 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.859 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:427 2.2 2:44c63967:::rbd_data.125de1449605.0000000000000101:head [write 2621440~1572864] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:46.859 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.336464+0000", 2023-07-02T14:02:46.859 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 7.2996248829999999, 2023-07-02T14:02:46.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037760331000000001, 2023-07-02T14:02:46.859 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:46.859 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:46.859 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:46.860 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:46.860 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:46.860 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 427 2023-07-02T14:02:46.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.860 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:46.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:46.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336464+0000", 2023-07-02T14:02:46.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:46.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:46.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336464+0000", 2023-07-02T14:02:46.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:02:46.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:46.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336462+0000", 2023-07-02T14:02:46.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050934140000000001 2023-07-02T14:02:46.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:46.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.341555+0000", 2023-07-02T14:02:46.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6209e-05 2023-07-02T14:02:46.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:46.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.341572+0000", 2023-07-02T14:02:46.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.114e-06 2023-07-02T14:02:46.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:46.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.341579+0000", 2023-07-02T14:02:46.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019481099000000002 2023-07-02T14:02:46.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:46.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361060+0000", 2023-07-02T14:02:46.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5408000000000005e-05 2023-07-02T14:02:46.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:46.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361155+0000", 2023-07-02T14:02:46.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000186515 2023-07-02T14:02:46.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:46.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361342+0000", 2023-07-02T14:02:46.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003066465 2023-07-02T14:02:46.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:46.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364408+0000", 2023-07-02T14:02:46.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0096922329999999998 2023-07-02T14:02:46.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:46.867 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.374100+0000", 2023-07-02T14:02:46.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7960999999999995e-05 2023-07-02T14:02:46.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:46.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.374168+0000", 2023-07-02T14:02:46.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5458000000000003e-05 2023-07-02T14:02:46.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:46.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:46.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:46.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.374224+0000", 2023-07-02T14:02:46.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037760331000000001 2023-07-02T14:02:46.868 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.869 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.869 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.869 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:46.869 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:46.869 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:46.871 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:02:46.875 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:46.870+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:46.911 INFO:tasks.workunit.client.0.smithi163.stderr:136100K .......... .......... .......... .......... .......... 80% 38.9M 2s 2023-07-02T14:02:46.911 INFO:tasks.workunit.client.0.smithi163.stderr:136150K .......... .......... .......... .......... .......... 80% 22.9M 2s 2023-07-02T14:02:46.911 INFO:tasks.workunit.client.0.smithi163.stderr:136200K .......... .......... .......... .......... .......... 80% 14.0M 2s 2023-07-02T14:02:46.911 INFO:tasks.workunit.client.0.smithi163.stderr:136250K .......... .......... .......... .......... .......... 80% 22.8M 2s 2023-07-02T14:02:46.911 INFO:tasks.workunit.client.0.smithi163.stderr:136300K .......... .......... .......... .......... .......... 80% 30.4M 2s 2023-07-02T14:02:46.912 INFO:tasks.workunit.client.0.smithi163.stderr:136350K .......... .......... .......... .......... .......... 80% 19.6M 2s 2023-07-02T14:02:46.912 INFO:tasks.workunit.client.0.smithi163.stderr:136400K .......... .......... .......... .......... .......... 80% 18.7M 2s 2023-07-02T14:02:46.912 INFO:tasks.workunit.client.0.smithi163.stderr:136450K .......... .......... .......... .......... .......... 80% 24.2M 2s 2023-07-02T14:02:46.912 INFO:tasks.workunit.client.0.smithi163.stderr:136500K .......... .......... .......... .......... .......... 81% 14.5M 2s 2023-07-02T14:02:46.912 INFO:tasks.workunit.client.0.smithi163.stderr:136550K .......... .......... .......... .......... .......... 81% 20.2M 2s 2023-07-02T14:02:46.912 INFO:tasks.workunit.client.0.smithi163.stderr:136600K .......... .......... .......... .......... .......... 81% 33.4M 2s 2023-07-02T14:02:46.912 INFO:tasks.workunit.client.0.smithi163.stderr:136650K .......... .......... .......... .......... .......... 81% 33.9M 2s 2023-07-02T14:02:46.912 INFO:tasks.workunit.client.0.smithi163.stderr:136700K .......... .......... .......... .......... .......... 81% 20.4M 2s 2023-07-02T14:02:46.913 INFO:tasks.workunit.client.0.smithi163.stderr:136750K .......... .......... .......... .......... .......... 81% 15.2M 2s 2023-07-02T14:02:46.913 INFO:tasks.workunit.client.0.smithi163.stderr:136800K .......... .......... .......... .......... .......... 81% 13.6M 2s 2023-07-02T14:02:46.913 INFO:tasks.workunit.client.0.smithi163.stderr:136850K .......... .......... .......... .......... .......... 81% 27.2M 2s 2023-07-02T14:02:46.913 INFO:tasks.workunit.client.0.smithi163.stderr:136900K .......... .......... .......... .......... .......... 81% 22.3M 2s 2023-07-02T14:02:46.913 INFO:tasks.workunit.client.0.smithi163.stderr:136950K .......... .......... .......... .......... .......... 81% 26.8M 2s 2023-07-02T14:02:46.913 INFO:tasks.workunit.client.0.smithi163.stderr:137000K .......... .......... .......... .......... .......... 81% 24.2M 2s 2023-07-02T14:02:46.913 INFO:tasks.workunit.client.0.smithi163.stderr:137050K .......... .......... .......... .......... .......... 81% 14.3M 2s 2023-07-02T14:02:46.914 INFO:tasks.workunit.client.0.smithi163.stderr:137100K .......... .......... .......... .......... .......... 81% 24.8M 2s 2023-07-02T14:02:46.914 INFO:tasks.workunit.client.0.smithi163.stderr:137150K .......... .......... .......... .......... .......... 81% 18.0M 2s 2023-07-02T14:02:46.914 INFO:tasks.workunit.client.0.smithi163.stderr:137200K .......... .......... .......... .......... .......... 81% 32.8M 2s 2023-07-02T14:02:46.914 INFO:tasks.workunit.client.0.smithi163.stderr:137250K .......... .......... .......... .......... .......... 81% 21.9M 2s 2023-07-02T14:02:46.914 INFO:tasks.workunit.client.0.smithi163.stderr:137300K .......... .......... .......... .......... .......... 81% 14.8M 2s 2023-07-02T14:02:46.914 INFO:tasks.workunit.client.0.smithi163.stderr:137350K .......... .......... .......... .......... .......... 81% 32.2M 2s 2023-07-02T14:02:46.914 INFO:tasks.workunit.client.0.smithi163.stderr:137400K .......... .......... .......... .......... .......... 81% 18.2M 2s 2023-07-02T14:02:46.914 INFO:tasks.workunit.client.0.smithi163.stderr:137450K .......... .......... .......... .......... .......... 81% 30.9M 2s 2023-07-02T14:02:46.915 INFO:tasks.workunit.client.0.smithi163.stderr:137500K .......... .......... .......... .......... .......... 81% 18.4M 2s 2023-07-02T14:02:46.915 INFO:tasks.workunit.client.0.smithi163.stderr:137550K .......... .......... .......... .......... .......... 81% 15.9M 2s 2023-07-02T14:02:46.915 INFO:tasks.workunit.client.0.smithi163.stderr:137600K .......... .......... .......... .......... .......... 81% 14.0M 2s 2023-07-02T14:02:46.915 INFO:tasks.workunit.client.0.smithi163.stderr:137650K .......... .......... .......... .......... .......... 81% 31.8M 2s 2023-07-02T14:02:46.915 INFO:tasks.workunit.client.0.smithi163.stderr:137700K .......... .......... .......... .......... .......... 81% 26.9M 2s 2023-07-02T14:02:46.915 INFO:tasks.workunit.client.0.smithi163.stderr:137750K .......... .......... .......... .......... .......... 81% 19.7M 2s 2023-07-02T14:02:46.915 INFO:tasks.workunit.client.0.smithi163.stderr:137800K .......... .......... .......... .......... .......... 81% 14.5M 2s 2023-07-02T14:02:46.915 INFO:tasks.workunit.client.0.smithi163.stderr:137850K .......... .......... .......... .......... .......... 81% 26.2M 2s 2023-07-02T14:02:46.916 INFO:tasks.workunit.client.0.smithi163.stderr:137900K .......... .......... .......... .......... .......... 81% 17.5M 2s 2023-07-02T14:02:46.916 INFO:tasks.workunit.client.0.smithi163.stderr:137950K .......... .......... .......... .......... .......... 81% 18.1M 2s 2023-07-02T14:02:46.916 INFO:tasks.workunit.client.0.smithi163.stderr:138000K .......... .......... .......... .......... .......... 81% 35.2M 2s 2023-07-02T14:02:46.916 INFO:tasks.workunit.client.0.smithi163.stderr:138050K .......... .......... .......... .......... .......... 81% 20.3M 2s 2023-07-02T14:02:46.916 INFO:tasks.workunit.client.0.smithi163.stderr:138100K .......... .......... .......... .......... .......... 81% 29.8M 2s 2023-07-02T14:02:46.916 INFO:tasks.workunit.client.0.smithi163.stderr:138150K .......... .......... .......... .......... .......... 81% 10.7M 2s 2023-07-02T14:02:46.916 INFO:tasks.workunit.client.0.smithi163.stderr:138200K .......... .......... .......... .......... .......... 82% 40.8M 2s 2023-07-02T14:02:46.917 INFO:tasks.workunit.client.0.smithi163.stderr:138250K .......... .......... .......... .......... .......... 82% 22.8M 1s 2023-07-02T14:02:46.917 INFO:tasks.workunit.client.0.smithi163.stderr:138300K .......... .......... .......... .......... .......... 82% 21.3M 1s 2023-07-02T14:02:46.917 INFO:tasks.workunit.client.0.smithi163.stderr:138350K .......... .......... .......... .......... .......... 82% 18.0M 1s 2023-07-02T14:02:46.917 INFO:tasks.workunit.client.0.smithi163.stderr:138400K .......... .......... .......... .......... .......... 82% 20.4M 1s 2023-07-02T14:02:46.917 INFO:tasks.workunit.client.0.smithi163.stderr:138450K .......... .......... .......... .......... .......... 82% 14.2M 1s 2023-07-02T14:02:46.917 INFO:tasks.workunit.client.0.smithi163.stderr:138500K .......... .......... .......... .......... .......... 82% 32.3M 1s 2023-07-02T14:02:46.917 INFO:tasks.workunit.client.0.smithi163.stderr:138550K .......... .......... .......... .......... .......... 82% 25.2M 1s 2023-07-02T14:02:46.917 INFO:tasks.workunit.client.0.smithi163.stderr:138600K .......... .......... .......... .......... .......... 82% 32.1M 1s 2023-07-02T14:02:46.918 INFO:tasks.workunit.client.0.smithi163.stderr:138650K .......... .......... .......... .......... .......... 82% 17.3M 1s 2023-07-02T14:02:46.918 INFO:tasks.workunit.client.0.smithi163.stderr:138700K .......... .......... .......... .......... .......... 82% 41.9M 1s 2023-07-02T14:02:46.918 INFO:tasks.workunit.client.0.smithi163.stderr:138750K .......... .......... .......... .......... .......... 82% 13.0M 1s 2023-07-02T14:02:46.918 INFO:tasks.workunit.client.0.smithi163.stderr:138800K .......... .......... .......... .......... .......... 82% 17.5M 1s 2023-07-02T14:02:46.918 INFO:tasks.workunit.client.0.smithi163.stderr:138850K .......... .......... .......... .......... .......... 82% 23.8M 1s 2023-07-02T14:02:46.918 INFO:tasks.workunit.client.0.smithi163.stderr:138900K .......... .......... .......... .......... .......... 82% 24.5M 1s 2023-07-02T14:02:46.918 INFO:tasks.workunit.client.0.smithi163.stderr:138950K .......... .......... .......... .......... .......... 82% 40.1M 1s 2023-07-02T14:02:46.918 INFO:tasks.workunit.client.0.smithi163.stderr:139000K .......... .......... .......... .......... .......... 82% 19.2M 1s 2023-07-02T14:02:46.919 INFO:tasks.workunit.client.0.smithi163.stderr:139050K .......... .......... .......... .......... .......... 82% 14.3M 1s 2023-07-02T14:02:46.919 INFO:tasks.workunit.client.0.smithi163.stderr:139100K .......... .......... .......... .......... .......... 82% 35.4M 1s 2023-07-02T14:02:46.919 INFO:tasks.workunit.client.0.smithi163.stderr:139150K .......... .......... .......... .......... .......... 82% 16.0M 1s 2023-07-02T14:02:46.919 INFO:tasks.workunit.client.0.smithi163.stderr:139200K .......... .......... .......... .......... .......... 82% 18.5M 1s 2023-07-02T14:02:46.919 INFO:tasks.workunit.client.0.smithi163.stderr:139250K .......... .......... .......... .......... .......... 82% 20.4M 1s 2023-07-02T14:02:46.919 INFO:tasks.workunit.client.0.smithi163.stderr:139300K .......... .......... .......... .......... .......... 82% 30.2M 1s 2023-07-02T14:02:46.919 INFO:tasks.workunit.client.0.smithi163.stderr:139350K .......... .......... .......... .......... .......... 82% 15.8M 1s 2023-07-02T14:02:46.920 INFO:tasks.workunit.client.0.smithi163.stderr:139400K .......... .......... .......... .......... .......... 82% 31.0M 1s 2023-07-02T14:02:46.920 INFO:tasks.workunit.client.0.smithi163.stderr:139450K .......... .......... .......... .......... .......... 82% 20.6M 1s 2023-07-02T14:02:46.920 INFO:tasks.workunit.client.0.smithi163.stderr:139500K .......... .......... .......... .......... .......... 82% 17.5M 1s 2023-07-02T14:02:46.920 INFO:tasks.workunit.client.0.smithi163.stderr:139550K .......... .......... .......... .......... .......... 82% 16.9M 1s 2023-07-02T14:02:46.920 INFO:tasks.workunit.client.0.smithi163.stderr:139600K .......... .......... .......... .......... .......... 82% 18.5M 1s 2023-07-02T14:02:46.920 INFO:tasks.workunit.client.0.smithi163.stderr:139650K .......... .......... .......... .......... .......... 82% 16.6M 1s 2023-07-02T14:02:46.920 INFO:tasks.workunit.client.0.smithi163.stderr:139700K .......... .......... .......... .......... .......... 82% 20.7M 1s 2023-07-02T14:02:46.920 INFO:tasks.workunit.client.0.smithi163.stderr:139750K .......... .......... .......... .......... .......... 82% 30.4M 1s 2023-07-02T14:02:46.921 INFO:tasks.workunit.client.0.smithi163.stderr:139800K .......... .......... .......... .......... .......... 82% 16.7M 1s 2023-07-02T14:02:46.921 INFO:tasks.workunit.client.0.smithi163.stderr:139850K .......... .......... .......... .......... .......... 82% 18.4M 1s 2023-07-02T14:02:46.921 INFO:tasks.workunit.client.0.smithi163.stderr:139900K .......... .......... .......... .......... .......... 83% 23.2M 1s 2023-07-02T14:02:46.921 INFO:tasks.workunit.client.0.smithi163.stderr:139950K .......... .......... .......... .......... .......... 83% 15.9M 1s 2023-07-02T14:02:46.921 INFO:tasks.workunit.client.0.smithi163.stderr:140000K .......... .......... .......... .......... .......... 83% 32.7M 1s 2023-07-02T14:02:46.921 INFO:tasks.workunit.client.0.smithi163.stderr:140050K .......... .......... .......... .......... .......... 83% 29.7M 1s 2023-07-02T14:02:46.921 INFO:tasks.workunit.client.0.smithi163.stderr:140100K .......... .......... .......... .......... .......... 83% 15.3M 1s 2023-07-02T14:02:46.921 INFO:tasks.workunit.client.0.smithi163.stderr:140150K .......... .......... .......... .......... .......... 83% 15.8M 1s 2023-07-02T14:02:46.922 INFO:tasks.workunit.client.0.smithi163.stderr:140200K .......... .......... .......... .......... .......... 83% 19.5M 1s 2023-07-02T14:02:46.922 INFO:tasks.workunit.client.0.smithi163.stderr:140250K .......... .......... .......... .......... .......... 83% 39.7M 1s 2023-07-02T14:02:46.922 INFO:tasks.workunit.client.0.smithi163.stderr:140300K .......... .......... .......... .......... .......... 83% 24.2M 1s 2023-07-02T14:02:46.922 INFO:tasks.workunit.client.0.smithi163.stderr:140350K .......... .......... .......... .......... .......... 83% 20.3M 1s 2023-07-02T14:02:46.922 INFO:tasks.workunit.client.0.smithi163.stderr:140400K .......... .......... .......... .......... .......... 83% 16.7M 1s 2023-07-02T14:02:46.922 INFO:tasks.workunit.client.0.smithi163.stderr:140450K .......... .......... .......... .......... .......... 83% 15.1M 1s 2023-07-02T14:02:46.922 INFO:tasks.workunit.client.0.smithi163.stderr:140500K .......... .......... .......... .......... .......... 83% 41.5M 1s 2023-07-02T14:02:46.923 INFO:tasks.workunit.client.0.smithi163.stderr:140550K .......... .......... .......... .......... .......... 83% 25.9M 1s 2023-07-02T14:02:46.923 INFO:tasks.workunit.client.0.smithi163.stderr:140600K .......... .......... .......... .......... .......... 83% 22.4M 1s 2023-07-02T14:02:46.923 INFO:tasks.workunit.client.0.smithi163.stderr:140650K .......... .......... .......... .......... .......... 83% 30.7M 1s 2023-07-02T14:02:46.923 INFO:tasks.workunit.client.0.smithi163.stderr:140700K .......... .......... .......... .......... .......... 83% 12.0M 1s 2023-07-02T14:02:46.923 INFO:tasks.workunit.client.0.smithi163.stderr:140750K .......... .......... .......... .......... .......... 83% 16.6M 1s 2023-07-02T14:02:46.923 INFO:tasks.workunit.client.0.smithi163.stderr:140800K .......... .......... .......... .......... .......... 83% 25.3M 1s 2023-07-02T14:02:46.976 INFO:tasks.workunit.client.0.smithi163.stderr:140850K .......... .......... .......... .......... .......... 83% 607K 1s 2023-07-02T14:02:46.977 INFO:tasks.workunit.client.0.smithi163.stderr:140900K .......... .......... .......... .......... .......... 83% 294M 1s 2023-07-02T14:02:46.977 INFO:tasks.workunit.client.0.smithi163.stderr:140950K .......... .......... .......... .......... .......... 83% 289M 1s 2023-07-02T14:02:46.977 INFO:tasks.workunit.client.0.smithi163.stderr:141000K .......... .......... .......... .......... .......... 83% 365M 1s 2023-07-02T14:02:46.977 INFO:tasks.workunit.client.0.smithi163.stderr:141050K .......... .......... .......... .......... .......... 83% 374M 1s 2023-07-02T14:02:46.977 INFO:tasks.workunit.client.0.smithi163.stderr:141100K .......... .......... .......... .......... .......... 83% 382M 1s 2023-07-02T14:02:46.977 INFO:tasks.workunit.client.0.smithi163.stderr:141150K .......... .......... .......... .......... .......... 83% 298M 1s 2023-07-02T14:02:46.977 INFO:tasks.workunit.client.0.smithi163.stderr:141200K .......... .......... .......... .......... .......... 83% 373M 1s 2023-07-02T14:02:46.978 INFO:tasks.workunit.client.0.smithi163.stderr:141250K .......... .......... .......... .......... .......... 83% 365M 1s 2023-07-02T14:02:46.978 INFO:tasks.workunit.client.0.smithi163.stderr:141300K .......... .......... .......... .......... .......... 83% 363M 1s 2023-07-02T14:02:46.978 INFO:tasks.workunit.client.0.smithi163.stderr:141350K .......... .......... .......... .......... .......... 83% 328M 1s 2023-07-02T14:02:46.978 INFO:tasks.workunit.client.0.smithi163.stderr:141400K .......... .......... .......... .......... .......... 83% 371M 1s 2023-07-02T14:02:46.978 INFO:tasks.workunit.client.0.smithi163.stderr:141450K .......... .......... .......... .......... .......... 83% 363M 1s 2023-07-02T14:02:46.978 INFO:tasks.workunit.client.0.smithi163.stderr:141500K .......... .......... .......... .......... .......... 83% 356M 1s 2023-07-02T14:02:46.978 INFO:tasks.workunit.client.0.smithi163.stderr:141550K .......... .......... .......... .......... .......... 84% 306M 1s 2023-07-02T14:02:46.979 INFO:tasks.workunit.client.0.smithi163.stderr:141600K .......... .......... .......... .......... .......... 84% 351M 1s 2023-07-02T14:02:46.979 INFO:tasks.workunit.client.0.smithi163.stderr:141650K .......... .......... .......... .......... .......... 84% 355M 1s 2023-07-02T14:02:46.979 INFO:tasks.workunit.client.0.smithi163.stderr:141700K .......... .......... .......... .......... .......... 84% 314M 1s 2023-07-02T14:02:46.979 INFO:tasks.workunit.client.0.smithi163.stderr:141750K .......... .......... .......... .......... .......... 84% 315M 1s 2023-07-02T14:02:46.979 INFO:tasks.workunit.client.0.smithi163.stderr:141800K .......... .......... .......... .......... .......... 84% 363M 1s 2023-07-02T14:02:46.979 INFO:tasks.workunit.client.0.smithi163.stderr:141850K .......... .......... .......... .......... .......... 84% 336M 1s 2023-07-02T14:02:46.979 INFO:tasks.workunit.client.0.smithi163.stderr:141900K .......... .......... .......... .......... .......... 84% 352M 1s 2023-07-02T14:02:46.979 INFO:tasks.workunit.client.0.smithi163.stderr:141950K .......... .......... .......... .......... .......... 84% 304M 1s 2023-07-02T14:02:46.980 INFO:tasks.workunit.client.0.smithi163.stderr:142000K .......... .......... .......... .......... .......... 84% 356M 1s 2023-07-02T14:02:46.980 INFO:tasks.workunit.client.0.smithi163.stderr:142050K .......... .......... .......... .......... .......... 84% 358M 1s 2023-07-02T14:02:46.980 INFO:tasks.workunit.client.0.smithi163.stderr:142100K .......... .......... .......... .......... .......... 84% 368M 1s 2023-07-02T14:02:46.980 INFO:tasks.workunit.client.0.smithi163.stderr:142150K .......... .......... .......... .......... .......... 84% 307M 1s 2023-07-02T14:02:46.980 INFO:tasks.workunit.client.0.smithi163.stderr:142200K .......... .......... .......... .......... .......... 84% 299M 1s 2023-07-02T14:02:46.980 INFO:tasks.workunit.client.0.smithi163.stderr:142250K .......... .......... .......... .......... .......... 84% 361M 1s 2023-07-02T14:02:46.980 INFO:tasks.workunit.client.0.smithi163.stderr:142300K .......... .......... .......... .......... .......... 84% 358M 1s 2023-07-02T14:02:46.980 INFO:tasks.workunit.client.0.smithi163.stderr:142350K .......... .......... .......... .......... .......... 84% 274M 1s 2023-07-02T14:02:46.981 INFO:tasks.workunit.client.0.smithi163.stderr:142400K .......... .......... .......... .......... .......... 84% 328M 1s 2023-07-02T14:02:46.981 INFO:tasks.workunit.client.0.smithi163.stderr:142450K .......... .......... .......... .......... .......... 84% 366M 1s 2023-07-02T14:02:46.981 INFO:tasks.workunit.client.0.smithi163.stderr:142500K .......... .......... .......... .......... .......... 84% 361M 1s 2023-07-02T14:02:46.981 INFO:tasks.workunit.client.0.smithi163.stderr:142550K .......... .......... .......... .......... .......... 84% 280M 1s 2023-07-02T14:02:46.981 INFO:tasks.workunit.client.0.smithi163.stderr:142600K .......... .......... .......... .......... .......... 84% 368M 1s 2023-07-02T14:02:46.981 INFO:tasks.workunit.client.0.smithi163.stderr:142650K .......... .......... .......... .......... .......... 84% 378M 1s 2023-07-02T14:02:46.981 INFO:tasks.workunit.client.0.smithi163.stderr:142700K .......... .......... .......... .......... .......... 84% 15.0M 1s 2023-07-02T14:02:46.981 INFO:tasks.workunit.client.0.smithi163.stderr:142750K .......... .......... .......... .......... .......... 84% 14.0M 1s 2023-07-02T14:02:46.982 INFO:tasks.workunit.client.0.smithi163.stderr:142800K .......... .......... .......... .......... .......... 84% 28.8M 1s 2023-07-02T14:02:46.982 INFO:tasks.workunit.client.0.smithi163.stderr:142850K .......... .......... .......... .......... .......... 84% 22.4M 1s 2023-07-02T14:02:46.982 INFO:tasks.workunit.client.0.smithi163.stderr:142900K .......... .......... .......... .......... .......... 84% 13.3M 1s 2023-07-02T14:02:46.982 INFO:tasks.workunit.client.0.smithi163.stderr:142950K .......... .......... .......... .......... .......... 84% 23.9M 1s 2023-07-02T14:02:46.982 INFO:tasks.workunit.client.0.smithi163.stderr:143000K .......... .......... .......... .......... .......... 84% 16.8M 1s 2023-07-02T14:02:46.982 INFO:tasks.workunit.client.0.smithi163.stderr:143050K .......... .......... .......... .......... .......... 84% 20.3M 1s 2023-07-02T14:02:46.982 INFO:tasks.workunit.client.0.smithi163.stderr:143100K .......... .......... .......... .......... .......... 84% 22.2M 1s 2023-07-02T14:02:46.983 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:46.973+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:46.985 INFO:tasks.workunit.client.0.smithi163.stderr:143150K .......... .......... .......... .......... .......... 84% 10.0M 1s 2023-07-02T14:02:46.985 INFO:tasks.workunit.client.0.smithi163.stderr:143200K .......... .......... .......... .......... .......... 84% 17.6M 1s 2023-07-02T14:02:46.985 INFO:tasks.workunit.client.0.smithi163.stderr:143250K .......... .......... .......... .......... .......... 85% 31.3M 1s 2023-07-02T14:02:46.985 INFO:tasks.workunit.client.0.smithi163.stderr:143300K .......... .......... .......... .......... .......... 85% 22.4M 1s 2023-07-02T14:02:46.988 INFO:tasks.workunit.client.0.smithi163.stderr:143350K .......... .......... .......... .......... .......... 85% 12.7M 1s 2023-07-02T14:02:46.991 INFO:tasks.workunit.client.0.smithi163.stderr:143400K .......... .......... .......... .......... .......... 85% 18.0M 1s 2023-07-02T14:02:46.993 INFO:tasks.workunit.client.0.smithi163.stderr:143450K .......... .......... .......... .......... .......... 85% 19.2M 1s 2023-07-02T14:02:46.995 INFO:tasks.workunit.client.0.smithi163.stderr:143500K .......... .......... .......... .......... .......... 85% 36.0M 1s 2023-07-02T14:02:46.999 INFO:tasks.workunit.client.0.smithi163.stderr:143550K .......... .......... .......... .......... .......... 85% 11.5M 1s 2023-07-02T14:02:47.001 INFO:tasks.workunit.client.0.smithi163.stderr:143600K .......... .......... .......... .......... .......... 85% 18.2M 1s 2023-07-02T14:02:47.005 INFO:tasks.workunit.client.0.smithi163.stderr:143650K .......... .......... .......... .......... .......... 85% 17.9M 1s 2023-07-02T14:02:47.006 INFO:tasks.workunit.client.0.smithi163.stderr:143700K .......... .......... .......... .......... .......... 85% 45.7M 1s 2023-07-02T14:02:47.007 INFO:tasks.workunit.client.0.smithi163.stderr:143750K .......... .......... .......... .......... .......... 85% 30.2M 1s 2023-07-02T14:02:47.010 INFO:tasks.workunit.client.0.smithi163.stderr:143800K .......... .......... .......... .......... .......... 85% 14.1M 1s 2023-07-02T14:02:47.012 INFO:tasks.workunit.client.0.smithi163.stderr:143850K .......... .......... .......... .......... .......... 85% 24.9M 1s 2023-07-02T14:02:47.015 INFO:tasks.workunit.client.0.smithi163.stderr:143900K .......... .......... .......... .......... .......... 85% 15.9M 1s 2023-07-02T14:02:47.018 INFO:tasks.workunit.client.0.smithi163.stderr:143950K .......... .......... .......... .......... .......... 85% 24.4M 1s 2023-07-02T14:02:47.020 INFO:tasks.workunit.client.0.smithi163.stderr:144000K .......... .......... .......... .......... .......... 85% 20.4M 1s 2023-07-02T14:02:47.023 INFO:tasks.workunit.client.0.smithi163.stderr:144050K .......... .......... .......... .......... .......... 85% 14.9M 1s 2023-07-02T14:02:47.026 INFO:tasks.workunit.client.0.smithi163.stderr:144100K .......... .......... .......... .......... .......... 85% 18.9M 1s 2023-07-02T14:02:47.028 INFO:tasks.workunit.client.0.smithi163.stderr:144150K .......... .......... .......... .......... .......... 85% 16.6M 1s 2023-07-02T14:02:47.031 INFO:tasks.workunit.client.0.smithi163.stderr:144200K .......... .......... .......... .......... .......... 85% 20.4M 1s 2023-07-02T14:02:47.034 INFO:tasks.workunit.client.0.smithi163.stderr:144250K .......... .......... .......... .......... .......... 85% 19.1M 1s 2023-07-02T14:02:47.037 INFO:tasks.workunit.client.0.smithi163.stderr:144300K .......... .......... .......... .......... .......... 85% 14.4M 1s 2023-07-02T14:02:47.040 INFO:tasks.workunit.client.0.smithi163.stderr:144350K .......... .......... .......... .......... .......... 85% 15.3M 1s 2023-07-02T14:02:47.042 INFO:tasks.workunit.client.0.smithi163.stderr:144400K .......... .......... .......... .......... .......... 85% 20.6M 1s 2023-07-02T14:02:47.045 INFO:tasks.workunit.client.0.smithi163.stderr:144450K .......... .......... .......... .......... .......... 85% 23.9M 1s 2023-07-02T14:02:47.048 INFO:tasks.workunit.client.0.smithi163.stderr:144500K .......... .......... .......... .......... .......... 85% 14.1M 1s 2023-07-02T14:02:47.049 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:47.049 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:47.049 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:47.049 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:47.050 INFO:tasks.workunit.client.0.smithi163.stderr:144550K .......... .......... .......... .......... .......... 85% 27.0M 1s 2023-07-02T14:02:47.053 INFO:tasks.workunit.client.0.smithi163.stderr:144600K .......... .......... .......... .......... .......... 85% 15.4M 1s 2023-07-02T14:02:47.055 INFO:tasks.workunit.client.0.smithi163.stderr:144650K .......... .......... .......... .......... .......... 85% 25.2M 1s 2023-07-02T14:02:47.058 INFO:tasks.workunit.client.0.smithi163.stderr:144700K .......... .......... .......... .......... .......... 85% 15.1M 1s 2023-07-02T14:02:47.059 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:02:47.061 INFO:tasks.workunit.client.0.smithi163.stderr:144750K .......... .......... .......... .......... .......... 85% 17.1M 1s 2023-07-02T14:02:47.062 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:02:47.065 INFO:tasks.workunit.client.0.smithi163.stderr:144800K .......... .......... .......... .......... .......... 85% 13.8M 1s 2023-07-02T14:02:47.067 INFO:tasks.workunit.client.0.smithi163.stderr:144850K .......... .......... .......... .......... .......... 85% 25.9M 1s 2023-07-02T14:02:47.069 INFO:tasks.workunit.client.0.smithi163.stderr:144900K .......... .......... .......... .......... .......... 85% 20.3M 1s 2023-07-02T14:02:47.071 INFO:tasks.workunit.client.0.smithi163.stderr:144950K .......... .......... .......... .......... .......... 86% 28.5M 1s 2023-07-02T14:02:47.073 INFO:tasks.workunit.client.0.smithi163.stderr:145000K .......... .......... .......... .......... .......... 86% 20.9M 1s 2023-07-02T14:02:47.074 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:47.073+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:47.077 INFO:tasks.workunit.client.0.smithi163.stderr:145050K .......... .......... .......... .......... .......... 86% 12.5M 1s 2023-07-02T14:02:47.079 INFO:tasks.workunit.client.0.smithi163.stderr:145100K .......... .......... .......... .......... .......... 86% 23.6M 1s 2023-07-02T14:02:47.081 INFO:tasks.workunit.client.0.smithi163.stderr:145150K .......... .......... .......... .......... .......... 86% 20.2M 1s 2023-07-02T14:02:47.083 INFO:tasks.workunit.client.0.smithi163.stderr:145200K .......... .......... .......... .......... .......... 86% 30.5M 1s 2023-07-02T14:02:47.086 INFO:tasks.workunit.client.0.smithi163.stderr:145250K .......... .......... .......... .......... .......... 86% 15.0M 1s 2023-07-02T14:02:47.089 INFO:tasks.workunit.client.0.smithi163.stderr:145300K .......... .......... .......... .......... .......... 86% 16.9M 1s 2023-07-02T14:02:47.091 INFO:tasks.workunit.client.0.smithi163.stderr:145350K .......... .......... .......... .......... .......... 86% 23.5M 1s 2023-07-02T14:02:47.093 INFO:tasks.workunit.client.0.smithi163.stderr:145400K .......... .......... .......... .......... .......... 86% 23.6M 1s 2023-07-02T14:02:47.095 INFO:tasks.workunit.client.0.smithi163.stderr:145450K .......... .......... .......... .......... .......... 86% 27.4M 1s 2023-07-02T14:02:47.098 INFO:tasks.workunit.client.0.smithi163.stderr:145500K .......... .......... .......... .......... .......... 86% 14.4M 1s 2023-07-02T14:02:47.101 INFO:tasks.workunit.client.0.smithi163.stderr:145550K .......... .......... .......... .......... .......... 86% 17.2M 1s 2023-07-02T14:02:47.105 INFO:tasks.workunit.client.0.smithi163.stderr:145600K .......... .......... .......... .......... .......... 86% 12.4M 1s 2023-07-02T14:02:47.107 INFO:tasks.workunit.client.0.smithi163.stderr:145650K .......... .......... .......... .......... .......... 86% 23.8M 1s 2023-07-02T14:02:47.110 INFO:tasks.workunit.client.0.smithi163.stderr:145700K .......... .......... .......... .......... .......... 86% 19.4M 1s 2023-07-02T14:02:47.113 INFO:tasks.workunit.client.0.smithi163.stderr:145750K .......... .......... .......... .......... .......... 86% 15.5M 1s 2023-07-02T14:02:47.115 INFO:tasks.workunit.client.0.smithi163.stderr:145800K .......... .......... .......... .......... .......... 86% 21.2M 1s 2023-07-02T14:02:47.117 INFO:tasks.workunit.client.0.smithi163.stderr:145850K .......... .......... .......... .......... .......... 86% 26.0M 1s 2023-07-02T14:02:47.120 INFO:tasks.workunit.client.0.smithi163.stderr:145900K .......... .......... .......... .......... .......... 86% 14.6M 1s 2023-07-02T14:02:47.124 INFO:tasks.workunit.client.0.smithi163.stderr:145950K .......... .......... .......... .......... .......... 86% 14.1M 1s 2023-07-02T14:02:47.126 INFO:tasks.workunit.client.0.smithi163.stderr:146000K .......... .......... .......... .......... .......... 86% 20.2M 1s 2023-07-02T14:02:47.129 INFO:tasks.workunit.client.0.smithi163.stderr:146050K .......... .......... .......... .......... .......... 86% 21.6M 1s 2023-07-02T14:02:47.131 INFO:tasks.workunit.client.0.smithi163.stderr:146100K .......... .......... .......... .......... .......... 86% 20.5M 1s 2023-07-02T14:02:47.134 INFO:tasks.workunit.client.0.smithi163.stderr:146150K .......... .......... .......... .......... .......... 86% 16.0M 1s 2023-07-02T14:02:47.136 INFO:tasks.workunit.client.0.smithi163.stderr:146200K .......... .......... .......... .......... .......... 86% 28.8M 1s 2023-07-02T14:02:47.141 INFO:tasks.workunit.client.0.smithi163.stderr:146250K .......... .......... .......... .......... .......... 86% 9.75M 1s 2023-07-02T14:02:47.142 INFO:tasks.workunit.client.0.smithi163.stderr:146300K .......... .......... .......... .......... .......... 86% 35.6M 1s 2023-07-02T14:02:47.144 INFO:tasks.workunit.client.0.smithi163.stderr:146350K .......... .......... .......... .......... .......... 86% 22.1M 1s 2023-07-02T14:02:47.146 INFO:tasks.workunit.client.0.smithi163.stderr:146400K .......... .......... .......... .......... .......... 86% 30.4M 1s 2023-07-02T14:02:47.150 INFO:tasks.workunit.client.0.smithi163.stderr:146450K .......... .......... .......... .......... .......... 86% 13.6M 1s 2023-07-02T14:02:47.152 INFO:tasks.workunit.client.0.smithi163.stderr:146500K .......... .......... .......... .......... .......... 86% 17.7M 1s 2023-07-02T14:02:47.154 INFO:tasks.workunit.client.0.smithi163.stderr:146550K .......... .......... .......... .......... .......... 86% 25.7M 1s 2023-07-02T14:02:47.156 INFO:tasks.workunit.client.0.smithi163.stderr:146600K .......... .......... .......... .......... .......... 86% 27.6M 1s 2023-07-02T14:02:47.158 INFO:tasks.workunit.client.0.smithi163.stderr:146650K .......... .......... .......... .......... .......... 87% 18.1M 1s 2023-07-02T14:02:47.161 INFO:tasks.workunit.client.0.smithi163.stderr:146700K .......... .......... .......... .......... .......... 87% 20.5M 1s 2023-07-02T14:02:47.165 INFO:tasks.workunit.client.0.smithi163.stderr:146750K .......... .......... .......... .......... .......... 87% 12.2M 1s 2023-07-02T14:02:47.168 INFO:tasks.workunit.client.0.smithi163.stderr:146800K .......... .......... .......... .......... .......... 87% 19.8M 1s 2023-07-02T14:02:47.169 INFO:tasks.workunit.client.0.smithi163.stderr:146850K .......... .......... .......... .......... .......... 87% 27.8M 1s 2023-07-02T14:02:47.172 INFO:tasks.workunit.client.0.smithi163.stderr:146900K .......... .......... .......... .......... .......... 87% 18.0M 1s 2023-07-02T14:02:47.175 INFO:tasks.workunit.client.0.smithi163.stderr:146950K .......... .......... .......... .......... .......... 87% 18.3M 1s 2023-07-02T14:02:47.175 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:47.174+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:47.177 INFO:tasks.workunit.client.0.smithi163.stderr:147000K .......... .......... .......... .......... .......... 87% 16.7M 1s 2023-07-02T14:02:47.179 INFO:tasks.workunit.client.0.smithi163.stderr:147050K .......... .......... .......... .......... .......... 87% 28.4M 1s 2023-07-02T14:02:47.182 INFO:tasks.workunit.client.0.smithi163.stderr:147100K .......... .......... .......... .......... .......... 87% 19.6M 1s 2023-07-02T14:02:47.186 INFO:tasks.workunit.client.0.smithi163.stderr:147150K .......... .......... .......... .......... .......... 87% 12.2M 1s 2023-07-02T14:02:47.188 INFO:tasks.workunit.client.0.smithi163.stderr:147200K .......... .......... .......... .......... .......... 87% 18.5M 1s 2023-07-02T14:02:47.191 INFO:tasks.workunit.client.0.smithi163.stderr:147250K .......... .......... .......... .......... .......... 87% 17.7M 1s 2023-07-02T14:02:47.193 INFO:tasks.workunit.client.0.smithi163.stderr:147300K .......... .......... .......... .......... .......... 87% 32.6M 1s 2023-07-02T14:02:47.196 INFO:tasks.workunit.client.0.smithi163.stderr:147350K .......... .......... .......... .......... .......... 87% 15.7M 1s 2023-07-02T14:02:47.198 INFO:tasks.workunit.client.0.smithi163.stderr:147400K .......... .......... .......... .......... .......... 87% 22.0M 1s 2023-07-02T14:02:47.202 INFO:tasks.workunit.client.0.smithi163.stderr:147450K .......... .......... .......... .......... .......... 87% 13.3M 1s 2023-07-02T14:02:47.205 INFO:tasks.workunit.client.0.smithi163.stderr:147500K .......... .......... .......... .......... .......... 87% 15.8M 1s 2023-07-02T14:02:47.207 INFO:tasks.workunit.client.0.smithi163.stderr:147550K .......... .......... .......... .......... .......... 87% 20.5M 1s 2023-07-02T14:02:47.209 INFO:tasks.workunit.client.0.smithi163.stderr:147600K .......... .......... .......... .......... .......... 87% 23.8M 1s 2023-07-02T14:02:47.211 INFO:tasks.workunit.client.0.smithi163.stderr:147650K .......... .......... .......... .......... .......... 87% 22.1M 1s 2023-07-02T14:02:47.215 INFO:tasks.workunit.client.0.smithi163.stderr:147700K .......... .......... .......... .......... .......... 87% 13.9M 1s 2023-07-02T14:02:47.217 INFO:tasks.workunit.client.0.smithi163.stderr:147750K .......... .......... .......... .......... .......... 87% 17.7M 1s 2023-07-02T14:02:47.219 INFO:tasks.workunit.client.0.smithi163.stderr:147800K .......... .......... .......... .......... .......... 87% 33.7M 1s 2023-07-02T14:02:47.221 INFO:tasks.workunit.client.0.smithi163.stderr:147850K .......... .......... .......... .......... .......... 87% 21.5M 1s 2023-07-02T14:02:47.223 INFO:tasks.workunit.client.0.smithi163.stderr:147900K .......... .......... .......... .......... .......... 87% 28.3M 1s 2023-07-02T14:02:47.227 INFO:tasks.workunit.client.0.smithi163.stderr:147950K .......... .......... .......... .......... .......... 87% 11.9M 1s 2023-07-02T14:02:47.230 INFO:tasks.workunit.client.0.smithi163.stderr:148000K .......... .......... .......... .......... .......... 87% 21.3M 1s 2023-07-02T14:02:47.232 INFO:tasks.workunit.client.0.smithi163.stderr:148050K .......... .......... .......... .......... .......... 87% 21.1M 1s 2023-07-02T14:02:47.234 INFO:tasks.workunit.client.0.smithi163.stderr:148100K .......... .......... .......... .......... .......... 87% 20.9M 1s 2023-07-02T14:02:47.235 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:47.236 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:47.236 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:47.236 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:47.236 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:47.237 INFO:tasks.workunit.client.0.smithi163.stderr:148150K .......... .......... .......... .......... .......... 87% 18.0M 1s 2023-07-02T14:02:47.239 INFO:tasks.workunit.client.0.smithi163.stderr:148200K .......... .......... .......... .......... .......... 87% 20.9M 1s 2023-07-02T14:02:47.241 INFO:tasks.workunit.client.0.smithi163.stderr:148250K .......... .......... .......... .......... .......... 87% 29.5M 1s 2023-07-02T14:02:47.244 INFO:tasks.workunit.client.0.smithi163.stderr:148300K .......... .......... .......... .......... .......... 88% 15.9M 1s 2023-07-02T14:02:47.246 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:02:47.247 INFO:tasks.workunit.client.0.smithi163.stderr:148350K .......... .......... .......... .......... .......... 88% 16.8M 1s 2023-07-02T14:02:47.249 INFO:tasks.workunit.client.0.smithi163.stderr:148400K .......... .......... .......... .......... .......... 88% 17.7M 1s 2023-07-02T14:02:47.252 INFO:tasks.workunit.client.0.smithi163.stderr:148450K .......... .......... .......... .......... .......... 88% 19.0M 1s 2023-07-02T14:02:47.255 INFO:tasks.workunit.client.0.smithi163.stderr:148500K .......... .......... .......... .......... .......... 88% 20.1M 1s 2023-07-02T14:02:47.258 INFO:tasks.workunit.client.0.smithi163.stderr:148550K .......... .......... .......... .......... .......... 88% 17.1M 1s 2023-07-02T14:02:47.259 INFO:tasks.workunit.client.0.smithi163.stderr:148600K .......... .......... .......... .......... .......... 88% 23.9M 1s 2023-07-02T14:02:47.262 INFO:tasks.workunit.client.0.smithi163.stderr:148650K .......... .......... .......... .......... .......... 88% 20.8M 1s 2023-07-02T14:02:47.264 INFO:tasks.workunit.client.0.smithi163.stderr:148700K .......... .......... .......... .......... .......... 88% 21.5M 1s 2023-07-02T14:02:47.268 INFO:tasks.workunit.client.0.smithi163.stderr:148750K .......... .......... .......... .......... .......... 88% 13.2M 1s 2023-07-02T14:02:47.271 INFO:tasks.workunit.client.0.smithi163.stderr:148800K .......... .......... .......... .......... .......... 88% 14.5M 1s 2023-07-02T14:02:47.273 INFO:tasks.workunit.client.0.smithi163.stderr:148850K .......... .......... .......... .......... .......... 88% 23.2M 1s 2023-07-02T14:02:47.275 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:47.275+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:47.278 INFO:tasks.workunit.client.0.smithi163.stderr:148900K .......... .......... .......... .......... .......... 88% 11.2M 1s 2023-07-02T14:02:47.279 INFO:tasks.workunit.client.0.smithi163.stderr:148950K .......... .......... .......... .......... .......... 88% 26.0M 1s 2023-07-02T14:02:47.281 INFO:tasks.workunit.client.0.smithi163.stderr:149000K .......... .......... .......... .......... .......... 88% 28.6M 1s 2023-07-02T14:02:47.284 INFO:tasks.workunit.client.0.smithi163.stderr:149050K .......... .......... .......... .......... .......... 88% 18.7M 1s 2023-07-02T14:02:47.286 INFO:tasks.workunit.client.0.smithi163.stderr:149100K .......... .......... .......... .......... .......... 88% 23.1M 1s 2023-07-02T14:02:47.289 INFO:tasks.workunit.client.0.smithi163.stderr:149150K .......... .......... .......... .......... .......... 88% 13.1M 1s 2023-07-02T14:02:47.291 INFO:tasks.workunit.client.0.smithi163.stderr:149200K .......... .......... .......... .......... .......... 88% 28.1M 1s 2023-07-02T14:02:47.295 INFO:tasks.workunit.client.0.smithi163.stderr:149250K .......... .......... .......... .......... .......... 88% 14.2M 1s 2023-07-02T14:02:47.296 INFO:tasks.workunit.client.0.smithi163.stderr:149300K .......... .......... .......... .......... .......... 88% 27.0M 1s 2023-07-02T14:02:47.298 INFO:tasks.workunit.client.0.smithi163.stderr:149350K .......... .......... .......... .......... .......... 88% 28.3M 1s 2023-07-02T14:02:47.301 INFO:tasks.workunit.client.0.smithi163.stderr:149400K .......... .......... .......... .......... .......... 88% 21.9M 1s 2023-07-02T14:02:47.303 INFO:tasks.workunit.client.0.smithi163.stderr:149450K .......... .......... .......... .......... .......... 88% 19.8M 1s 2023-07-02T14:02:47.306 INFO:tasks.workunit.client.0.smithi163.stderr:149500K .......... .......... .......... .......... .......... 88% 17.4M 1s 2023-07-02T14:02:47.309 INFO:tasks.workunit.client.0.smithi163.stderr:149550K .......... .......... .......... .......... .......... 88% 17.8M 1s 2023-07-02T14:02:47.311 INFO:tasks.workunit.client.0.smithi163.stderr:149600K .......... .......... .......... .......... .......... 88% 16.7M 1s 2023-07-02T14:02:47.313 INFO:tasks.workunit.client.0.smithi163.stderr:149650K .......... .......... .......... .......... .......... 88% 30.0M 1s 2023-07-02T14:02:47.315 INFO:tasks.workunit.client.0.smithi163.stderr:149700K .......... .......... .......... .......... .......... 88% 21.2M 1s 2023-07-02T14:02:47.317 INFO:tasks.workunit.client.0.smithi163.stderr:149750K .......... .......... .......... .......... .......... 88% 27.0M 1s 2023-07-02T14:02:47.319 INFO:tasks.workunit.client.0.smithi163.stderr:149800K .......... .......... .......... .......... .......... 88% 20.4M 1s 2023-07-02T14:02:47.321 INFO:tasks.workunit.client.0.smithi163.stderr:149850K .......... .......... .......... .......... .......... 88% 22.8M 1s 2023-07-02T14:02:47.324 INFO:tasks.workunit.client.0.smithi163.stderr:149900K .......... .......... .......... .......... .......... 88% 24.0M 1s 2023-07-02T14:02:47.329 INFO:tasks.workunit.client.0.smithi163.stderr:149950K .......... .......... .......... .......... .......... 88% 9.68M 1s 2023-07-02T14:02:47.333 INFO:tasks.workunit.client.0.smithi163.stderr:150000K .......... .......... .......... .......... .......... 89% 12.0M 1s 2023-07-02T14:02:47.335 INFO:tasks.workunit.client.0.smithi163.stderr:150050K .......... .......... .......... .......... .......... 89% 20.0M 1s 2023-07-02T14:02:47.337 INFO:tasks.workunit.client.0.smithi163.stderr:150100K .......... .......... .......... .......... .......... 89% 27.3M 1s 2023-07-02T14:02:47.339 INFO:tasks.workunit.client.0.smithi163.stderr:150150K .......... .......... .......... .......... .......... 89% 18.7M 1s 2023-07-02T14:02:47.341 INFO:tasks.workunit.client.0.smithi163.stderr:150200K .......... .......... .......... .......... .......... 89% 26.4M 1s 2023-07-02T14:02:47.343 INFO:tasks.workunit.client.0.smithi163.stderr:150250K .......... .......... .......... .......... .......... 89% 39.6M 1s 2023-07-02T14:02:47.347 INFO:tasks.workunit.client.0.smithi163.stderr:150300K .......... .......... .......... .......... .......... 89% 12.3M 1s 2023-07-02T14:02:47.352 INFO:tasks.workunit.client.0.smithi163.stderr:150350K .......... .......... .......... .......... .......... 89% 8.71M 1s 2023-07-02T14:02:47.354 INFO:tasks.workunit.client.0.smithi163.stderr:150400K .......... .......... .......... .......... .......... 89% 24.2M 1s 2023-07-02T14:02:47.357 INFO:tasks.workunit.client.0.smithi163.stderr:150450K .......... .......... .......... .......... .......... 89% 24.4M 1s 2023-07-02T14:02:47.359 INFO:tasks.workunit.client.0.smithi163.stderr:150500K .......... .......... .......... .......... .......... 89% 21.3M 1s 2023-07-02T14:02:47.360 INFO:tasks.workunit.client.0.smithi163.stderr:150550K .......... .......... .......... .......... .......... 89% 26.6M 1s 2023-07-02T14:02:47.363 INFO:tasks.workunit.client.0.smithi163.stderr:150600K .......... .......... .......... .......... .......... 89% 21.0M 1s 2023-07-02T14:02:47.366 INFO:tasks.workunit.client.0.smithi163.stderr:150650K .......... .......... .......... .......... .......... 89% 13.6M 1s 2023-07-02T14:02:47.368 INFO:tasks.workunit.client.0.smithi163.stderr:150700K .......... .......... .......... .......... .......... 89% 24.3M 1s 2023-07-02T14:02:47.372 INFO:tasks.workunit.client.0.smithi163.stderr:150750K .......... .......... .......... .......... .......... 89% 13.3M 1s 2023-07-02T14:02:47.374 INFO:tasks.workunit.client.0.smithi163.stderr:150800K .......... .......... .......... .......... .......... 89% 24.0M 1s 2023-07-02T14:02:47.375 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:47.374+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:47.376 INFO:tasks.workunit.client.0.smithi163.stderr:150850K .......... .......... .......... .......... .......... 89% 23.2M 1s 2023-07-02T14:02:47.379 INFO:tasks.workunit.client.0.smithi163.stderr:150900K .......... .......... .......... .......... .......... 89% 20.9M 1s 2023-07-02T14:02:47.381 INFO:tasks.workunit.client.0.smithi163.stderr:150950K .......... .......... .......... .......... .......... 89% 22.5M 1s 2023-07-02T14:02:47.384 INFO:tasks.workunit.client.0.smithi163.stderr:151000K .......... .......... .......... .......... .......... 89% 14.8M 1s 2023-07-02T14:02:47.386 INFO:tasks.workunit.client.0.smithi163.stderr:151050K .......... .......... .......... .......... .......... 89% 23.3M 1s 2023-07-02T14:02:47.387 INFO:tasks.workunit.client.0.smithi163.stderr:151100K .......... .......... .......... .......... .......... 89% 38.3M 1s 2023-07-02T14:02:47.391 INFO:tasks.workunit.client.0.smithi163.stderr:151150K .......... .......... .......... .......... .......... 89% 14.3M 1s 2023-07-02T14:02:47.394 INFO:tasks.workunit.client.0.smithi163.stderr:151200K .......... .......... .......... .......... .......... 89% 17.7M 1s 2023-07-02T14:02:47.396 INFO:tasks.workunit.client.0.smithi163.stderr:151250K .......... .......... .......... .......... .......... 89% 20.2M 1s 2023-07-02T14:02:47.398 INFO:tasks.workunit.client.0.smithi163.stderr:151300K .......... .......... .......... .......... .......... 89% 24.2M 1s 2023-07-02T14:02:47.401 INFO:tasks.workunit.client.0.smithi163.stderr:151350K .......... .......... .......... .......... .......... 89% 14.8M 1s 2023-07-02T14:02:47.403 INFO:tasks.workunit.client.0.smithi163.stderr:151400K .......... .......... .......... .......... .......... 89% 22.5M 1s 2023-07-02T14:02:47.406 INFO:tasks.workunit.client.0.smithi163.stderr:151450K .......... .......... .......... .......... .......... 89% 20.0M 1s 2023-07-02T14:02:47.409 INFO:tasks.workunit.client.0.smithi163.stderr:151500K .......... .......... .......... .......... .......... 89% 14.5M 1s 2023-07-02T14:02:47.412 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:47.412 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:47.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:47.412 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:02:47.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.413 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:163 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:47.413 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.284743+0000", 2023-07-02T14:02:47.413 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.1256623710000007, 2023-07-02T14:02:47.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027177222000000001, 2023-07-02T14:02:47.413 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:47.413 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:47.413 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:47.414 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:47.414 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:47.414 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 163 2023-07-02T14:02:47.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.414 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:47.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:47.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284743+0000", 2023-07-02T14:02:47.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:47.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:47.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284743+0000", 2023-07-02T14:02:47.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:47.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:47.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284742+0000", 2023-07-02T14:02:47.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9390999999999998e-05 2023-07-02T14:02:47.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:47.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284831+0000", 2023-07-02T14:02:47.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6971000000000001e-05 2023-07-02T14:02:47.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:47.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284848+0000", 2023-07-02T14:02:47.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951e-05 2023-07-02T14:02:47.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:47.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284864+0000", 2023-07-02T14:02:47.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015684601999999999 2023-07-02T14:02:47.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:47.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300549+0000", 2023-07-02T14:02:47.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3045000000000003e-05 2023-07-02T14:02:47.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:47.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300622+0000", 2023-07-02T14:02:47.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9452999999999999e-05 2023-07-02T14:02:47.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:47.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300711+0000", 2023-07-02T14:02:47.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059060379999999997 2023-07-02T14:02:47.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:47.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306617+0000", 2023-07-02T14:02:47.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052100280000000002 2023-07-02T14:02:47.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:47.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311827+0000", 2023-07-02T14:02:47.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978999999999998e-05 2023-07-02T14:02:47.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:47.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311879+0000", 2023-07-02T14:02:47.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0349999999999998e-05 2023-07-02T14:02:47.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:47.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311920+0000", 2023-07-02T14:02:47.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027177222000000001 2023-07-02T14:02:47.431 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.432 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.432 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.433 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:165 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:47.433 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.284977+0000", 2023-07-02T14:02:47.433 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.1254277479999999, 2023-07-02T14:02:47.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027689147000000001, 2023-07-02T14:02:47.434 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:47.434 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:47.434 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:47.434 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:47.435 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:47.435 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 165 2023-07-02T14:02:47.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.435 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:47.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:47.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284977+0000", 2023-07-02T14:02:47.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:47.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:47.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284977+0000", 2023-07-02T14:02:47.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:47.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:47.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284976+0000", 2023-07-02T14:02:47.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3962999999999996e-05 2023-07-02T14:02:47.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:47.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285060+0000", 2023-07-02T14:02:47.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1260000000000004e-06 2023-07-02T14:02:47.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:47.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285070+0000", 2023-07-02T14:02:47.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0669999999999999e-06 2023-07-02T14:02:47.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:47.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285075+0000", 2023-07-02T14:02:47.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016699978000000001 2023-07-02T14:02:47.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:47.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.301775+0000", 2023-07-02T14:02:47.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6872000000000004e-05 2023-07-02T14:02:47.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:47.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.301862+0000", 2023-07-02T14:02:47.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000155592 2023-07-02T14:02:47.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:47.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.302017+0000", 2023-07-02T14:02:47.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046740250000000001 2023-07-02T14:02:47.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:47.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306691+0000", 2023-07-02T14:02:47.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059023670000000004 2023-07-02T14:02:47.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:47.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312593+0000", 2023-07-02T14:02:47.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6545999999999999e-05 2023-07-02T14:02:47.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:47.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312630+0000", 2023-07-02T14:02:47.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6270999999999999e-05 2023-07-02T14:02:47.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:47.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312666+0000", 2023-07-02T14:02:47.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027689147000000001 2023-07-02T14:02:47.452 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.453 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.453 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.453 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:168 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1507328~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:47.453 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285081+0000", 2023-07-02T14:02:47.453 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.1253237370000004, 2023-07-02T14:02:47.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027656613, 2023-07-02T14:02:47.454 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:47.454 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:47.454 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:47.454 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:47.454 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:47.454 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 168 2023-07-02T14:02:47.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.455 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:47.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:47.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285081+0000", 2023-07-02T14:02:47.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:47.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:47.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285081+0000", 2023-07-02T14:02:47.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:47.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:47.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285080+0000", 2023-07-02T14:02:47.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5372999999999998e-05 2023-07-02T14:02:47.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:47.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285156+0000", 2023-07-02T14:02:47.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0099999999999995e-06 2023-07-02T14:02:47.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:47.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285164+0000", 2023-07-02T14:02:47.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3320000000000002e-06 2023-07-02T14:02:47.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:47.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285168+0000", 2023-07-02T14:02:47.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017721316000000001 2023-07-02T14:02:47.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:47.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.302889+0000", 2023-07-02T14:02:47.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9568999999999998e-05 2023-07-02T14:02:47.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:47.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.302949+0000", 2023-07-02T14:02:47.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121902 2023-07-02T14:02:47.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:47.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303071+0000", 2023-07-02T14:02:47.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0071548599999999999 2023-07-02T14:02:47.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:47.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310226+0000", 2023-07-02T14:02:47.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024455520000000001 2023-07-02T14:02:47.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:47.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312671+0000", 2023-07-02T14:02:47.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6958000000000001e-05 2023-07-02T14:02:47.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:47.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312708+0000", 2023-07-02T14:02:47.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9451e-05 2023-07-02T14:02:47.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:47.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312738+0000", 2023-07-02T14:02:47.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027656613 2023-07-02T14:02:47.463 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.463 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.463 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.464 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:171 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1572864~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:47.464 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285174+0000", 2023-07-02T14:02:47.464 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.1252312199999999, 2023-07-02T14:02:47.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027618999000000002, 2023-07-02T14:02:47.464 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:47.465 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:47.465 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:47.465 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:47.465 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:47.465 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 171 2023-07-02T14:02:47.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.465 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:47.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:47.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285174+0000", 2023-07-02T14:02:47.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:47.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:47.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285174+0000", 2023-07-02T14:02:47.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:47.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:47.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285173+0000", 2023-07-02T14:02:47.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1789000000000002e-05 2023-07-02T14:02:47.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:47.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285225+0000", 2023-07-02T14:02:47.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6429999999999995e-06 2023-07-02T14:02:47.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:47.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285232+0000", 2023-07-02T14:02:47.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8380000000000001e-06 2023-07-02T14:02:47.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:47.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285237+0000", 2023-07-02T14:02:47.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018182713 2023-07-02T14:02:47.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:47.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303420+0000", 2023-07-02T14:02:47.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0059000000000001e-05 2023-07-02T14:02:47.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:47.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303490+0000", 2023-07-02T14:02:47.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012003 2023-07-02T14:02:47.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:47.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303610+0000", 2023-07-02T14:02:47.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066642150000000002 2023-07-02T14:02:47.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:47.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310274+0000", 2023-07-02T14:02:47.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024681949999999999 2023-07-02T14:02:47.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:47.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312743+0000", 2023-07-02T14:02:47.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5525999999999999e-05 2023-07-02T14:02:47.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:47.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312768+0000", 2023-07-02T14:02:47.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4613999999999998e-05 2023-07-02T14:02:47.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:47.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312793+0000", 2023-07-02T14:02:47.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027618999000000002 2023-07-02T14:02:47.474 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.474 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.474 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.475 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:174 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:47.475 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285243+0000", 2023-07-02T14:02:47.475 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.1251620229999997, 2023-07-02T14:02:47.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027607988, 2023-07-02T14:02:47.475 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:47.475 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:47.475 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:47.475 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:47.476 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:47.476 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 174 2023-07-02T14:02:47.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.476 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:47.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:47.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285243+0000", 2023-07-02T14:02:47.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:47.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:47.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285243+0000", 2023-07-02T14:02:47.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:47.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:47.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285242+0000", 2023-07-02T14:02:47.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6173000000000003e-05 2023-07-02T14:02:47.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:47.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285328+0000", 2023-07-02T14:02:47.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5729999999999997e-06 2023-07-02T14:02:47.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:47.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285338+0000", 2023-07-02T14:02:47.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.081e-06 2023-07-02T14:02:47.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:47.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285347+0000", 2023-07-02T14:02:47.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018567367000000001 2023-07-02T14:02:47.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:47.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303914+0000", 2023-07-02T14:02:47.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9522000000000003e-05 2023-07-02T14:02:47.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:47.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303984+0000", 2023-07-02T14:02:47.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013801800000000001 2023-07-02T14:02:47.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:47.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.304122+0000", 2023-07-02T14:02:47.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061898489999999999 2023-07-02T14:02:47.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:47.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310312+0000", 2023-07-02T14:02:47.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024882519999999998 2023-07-02T14:02:47.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:47.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312800+0000", 2023-07-02T14:02:47.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7100000000000001e-05 2023-07-02T14:02:47.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:47.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312827+0000", 2023-07-02T14:02:47.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3771999999999999e-05 2023-07-02T14:02:47.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:47.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312851+0000", 2023-07-02T14:02:47.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027607988 2023-07-02T14:02:47.486 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.486 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.486 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.487 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:177 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:47.487 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285353+0000", 2023-07-02T14:02:47.487 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.1250518399999994, 2023-07-02T14:02:47.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027577522, 2023-07-02T14:02:47.488 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:47.488 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:47.489 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:47.489 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:47.489 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:47.489 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 177 2023-07-02T14:02:47.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.490 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:47.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:47.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285353+0000", 2023-07-02T14:02:47.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:47.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:47.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285353+0000", 2023-07-02T14:02:47.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:47.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:47.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285352+0000", 2023-07-02T14:02:47.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7850999999999996e-05 2023-07-02T14:02:47.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:47.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285440+0000", 2023-07-02T14:02:47.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7960000000000001e-06 2023-07-02T14:02:47.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:47.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285449+0000", 2023-07-02T14:02:47.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0529999999999997e-06 2023-07-02T14:02:47.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:47.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285454+0000", 2023-07-02T14:02:47.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018975296999999999 2023-07-02T14:02:47.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:47.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.304429+0000", 2023-07-02T14:02:47.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0682e-05 2023-07-02T14:02:47.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:47.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.304480+0000", 2023-07-02T14:02:47.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116107 2023-07-02T14:02:47.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:47.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.304596+0000", 2023-07-02T14:02:47.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0057486309999999997 2023-07-02T14:02:47.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:47.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310345+0000", 2023-07-02T14:02:47.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025147289999999998 2023-07-02T14:02:47.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:47.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312860+0000", 2023-07-02T14:02:47.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7252999999999997e-05 2023-07-02T14:02:47.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:47.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312897+0000", 2023-07-02T14:02:47.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3705999999999999e-05 2023-07-02T14:02:47.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:47.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312931+0000", 2023-07-02T14:02:47.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027577522 2023-07-02T14:02:47.502 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.502 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.502 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.502 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:180 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:47.502 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285460+0000", 2023-07-02T14:02:47.502 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.1249449640000009, 2023-07-02T14:02:47.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027547589000000001, 2023-07-02T14:02:47.503 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:47.503 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:47.503 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:47.503 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:47.503 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:47.503 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 180 2023-07-02T14:02:47.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.504 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:47.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:47.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285460+0000", 2023-07-02T14:02:47.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:47.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:47.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285460+0000", 2023-07-02T14:02:47.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:47.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:47.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285459+0000", 2023-07-02T14:02:47.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7868999999999997e-05 2023-07-02T14:02:47.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:47.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285527+0000", 2023-07-02T14:02:47.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6710000000000004e-06 2023-07-02T14:02:47.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:47.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285533+0000", 2023-07-02T14:02:47.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9960000000000002e-06 2023-07-02T14:02:47.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:47.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285536+0000", 2023-07-02T14:02:47.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019509826000000001 2023-07-02T14:02:47.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:47.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.305046+0000", 2023-07-02T14:02:47.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3639000000000004e-05 2023-07-02T14:02:47.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:47.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.305119+0000", 2023-07-02T14:02:47.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014125300000000001 2023-07-02T14:02:47.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:47.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.305260+0000", 2023-07-02T14:02:47.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051273919999999997 2023-07-02T14:02:47.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:47.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310388+0000", 2023-07-02T14:02:47.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025505390000000001 2023-07-02T14:02:47.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:47.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312938+0000", 2023-07-02T14:02:47.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5608999999999997e-05 2023-07-02T14:02:47.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:47.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312974+0000", 2023-07-02T14:02:47.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3562999999999997e-05 2023-07-02T14:02:47.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:47.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313008+0000", 2023-07-02T14:02:47.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027547589000000001 2023-07-02T14:02:47.512 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.512 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.513 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.513 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:183 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:47.513 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:02:47.513 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.1248656690000001, 2023-07-02T14:02:47.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003, 2023-07-02T14:02:47.513 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:47.514 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:47.514 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:47.514 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:47.514 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:47.514 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 183 2023-07-02T14:02:47.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.514 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:47.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:47.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:02:47.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:47.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:47.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:02:47.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:47.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:47.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:02:47.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9790999999999997e-05 2023-07-02T14:02:47.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:47.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285598+0000", 2023-07-02T14:02:47.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5809999999999996e-06 2023-07-02T14:02:47.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:47.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285604+0000", 2023-07-02T14:02:47.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7599999999999998e-06 2023-07-02T14:02:47.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:47.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285607+0000", 2023-07-02T14:02:47.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021108188999999999 2023-07-02T14:02:47.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:47.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306715+0000", 2023-07-02T14:02:47.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6548e-05 2023-07-02T14:02:47.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:47.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306792+0000", 2023-07-02T14:02:47.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173869 2023-07-02T14:02:47.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:47.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306965+0000", 2023-07-02T14:02:47.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060498009999999996 2023-07-02T14:02:47.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:47.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313015+0000", 2023-07-02T14:02:47.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049443719999999998 2023-07-02T14:02:47.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:47.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317960+0000", 2023-07-02T14:02:47.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5927999999999998e-05 2023-07-02T14:02:47.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:47.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318006+0000", 2023-07-02T14:02:47.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6004999999999998e-05 2023-07-02T14:02:47.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:47.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318042+0000", 2023-07-02T14:02:47.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003 2023-07-02T14:02:47.523 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.524 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.524 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:186 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:47.524 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:02:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.1247948430000001, 2023-07-02T14:02:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003, 2023-07-02T14:02:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 186 2023-07-02T14:02:47.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.526 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:47.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:47.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:02:47.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:47.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:47.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:02:47.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:47.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:47.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:02:47.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4279e-05 2023-07-02T14:02:47.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:47.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285664+0000", 2023-07-02T14:02:47.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3789999999999995e-06 2023-07-02T14:02:47.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:47.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285673+0000", 2023-07-02T14:02:47.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499999999999998e-06 2023-07-02T14:02:47.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:47.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285678+0000", 2023-07-02T14:02:47.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023261947000000002 2023-07-02T14:02:47.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:47.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.308940+0000", 2023-07-02T14:02:47.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010323399999999999 2023-07-02T14:02:47.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:47.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309044+0000", 2023-07-02T14:02:47.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018878300000000001 2023-07-02T14:02:47.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:47.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309232+0000", 2023-07-02T14:02:47.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037874150000000001 2023-07-02T14:02:47.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:47.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313020+0000", 2023-07-02T14:02:47.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050976140000000003 2023-07-02T14:02:47.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:47.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318117+0000", 2023-07-02T14:02:47.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3182000000000001e-05 2023-07-02T14:02:47.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:47.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:02:47.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5055999999999998e-05 2023-07-02T14:02:47.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:47.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318196+0000", 2023-07-02T14:02:47.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003 2023-07-02T14:02:47.536 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.537 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.537 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.537 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:189 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:47.537 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:02:47.538 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.1247210800000005, 2023-07-02T14:02:47.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997, 2023-07-02T14:02:47.538 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:47.538 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:47.538 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:47.538 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:47.538 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:47.538 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:02:47.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.539 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:47.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:47.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:02:47.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:47.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:47.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:02:47.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:47.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:47.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285683+0000", 2023-07-02T14:02:47.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8451999999999999e-05 2023-07-02T14:02:47.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:47.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285782+0000", 2023-07-02T14:02:47.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3650000000000004e-06 2023-07-02T14:02:47.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:47.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285790+0000", 2023-07-02T14:02:47.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8230000000000002e-06 2023-07-02T14:02:47.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:47.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285795+0000", 2023-07-02T14:02:47.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024619367999999999 2023-07-02T14:02:47.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:47.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310414+0000", 2023-07-02T14:02:47.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014244600000000001 2023-07-02T14:02:47.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:47.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310557+0000", 2023-07-02T14:02:47.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020612699999999999 2023-07-02T14:02:47.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:47.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310763+0000", 2023-07-02T14:02:47.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003427687 2023-07-02T14:02:47.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:47.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314191+0000", 2023-07-02T14:02:47.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040492089999999998 2023-07-02T14:02:47.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:47.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318240+0000", 2023-07-02T14:02:47.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0287999999999999e-05 2023-07-02T14:02:47.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:47.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318270+0000", 2023-07-02T14:02:47.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0868000000000002e-05 2023-07-02T14:02:47.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:47.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318311+0000", 2023-07-02T14:02:47.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997 2023-07-02T14:02:47.548 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.548 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.548 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.548 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:192 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:47.549 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:02:47.549 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.1246043070000002, 2023-07-02T14:02:47.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003, 2023-07-02T14:02:47.549 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:47.549 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:47.549 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:47.549 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:47.549 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:47.550 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 192 2023-07-02T14:02:47.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.550 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:47.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:47.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:02:47.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:47.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:47.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:02:47.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:47.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:47.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285800+0000", 2023-07-02T14:02:47.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100417 2023-07-02T14:02:47.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:47.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285900+0000", 2023-07-02T14:02:47.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8244999999999999e-05 2023-07-02T14:02:47.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:47.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285919+0000", 2023-07-02T14:02:47.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2499999999999997e-06 2023-07-02T14:02:47.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:47.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285924+0000", 2023-07-02T14:02:47.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025288155999999999 2023-07-02T14:02:47.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:47.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311212+0000", 2023-07-02T14:02:47.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4073999999999996e-05 2023-07-02T14:02:47.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:47.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311286+0000", 2023-07-02T14:02:47.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016219699999999999 2023-07-02T14:02:47.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:47.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311448+0000", 2023-07-02T14:02:47.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066020599999999999 2023-07-02T14:02:47.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:47.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318050+0000", 2023-07-02T14:02:47.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000327986 2023-07-02T14:02:47.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:47.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318378+0000", 2023-07-02T14:02:47.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8693999999999998e-05 2023-07-02T14:02:47.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318427+0000", 2023-07-02T14:02:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6656999999999998e-05 2023-07-02T14:02:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318464+0000", 2023-07-02T14:02:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003 2023-07-02T14:02:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.559 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.559 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.559 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:195 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:47.559 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:02:47.559 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.1244728029999997, 2023-07-02T14:02:47.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001, 2023-07-02T14:02:47.560 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:47.560 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:47.560 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:47.560 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:47.560 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:47.560 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 195 2023-07-02T14:02:47.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.561 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:47.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:47.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:02:47.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:47.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:47.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:02:47.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:47.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:47.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285931+0000", 2023-07-02T14:02:47.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4530999999999995e-05 2023-07-02T14:02:47.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:47.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286006+0000", 2023-07-02T14:02:47.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4113e-05 2023-07-02T14:02:47.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:47.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286020+0000", 2023-07-02T14:02:47.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7459999999999997e-06 2023-07-02T14:02:47.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:47.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286026+0000", 2023-07-02T14:02:47.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025997769 2023-07-02T14:02:47.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:47.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312023+0000", 2023-07-02T14:02:47.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4403000000000001e-05 2023-07-02T14:02:47.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:47.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312078+0000", 2023-07-02T14:02:47.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020581399999999999 2023-07-02T14:02:47.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:47.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312284+0000", 2023-07-02T14:02:47.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005771043 2023-07-02T14:02:47.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:47.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318055+0000", 2023-07-02T14:02:47.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00046015200000000002 2023-07-02T14:02:47.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:47.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318515+0000", 2023-07-02T14:02:47.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9444000000000003e-05 2023-07-02T14:02:47.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:47.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318564+0000", 2023-07-02T14:02:47.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2141000000000002e-05 2023-07-02T14:02:47.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:47.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318606+0000", 2023-07-02T14:02:47.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001 2023-07-02T14:02:47.569 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.569 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.570 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.570 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:198 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:47.570 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:02:47.570 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.1243728439999998, 2023-07-02T14:02:47.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999, 2023-07-02T14:02:47.571 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:47.571 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:47.571 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:47.571 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:47.572 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:47.572 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 198 2023-07-02T14:02:47.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.572 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:47.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:47.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:02:47.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:47.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:47.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:02:47.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:47.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:47.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286031+0000", 2023-07-02T14:02:47.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00049286700000000002 2023-07-02T14:02:47.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:47.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286524+0000", 2023-07-02T14:02:47.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4938e-05 2023-07-02T14:02:47.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:47.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286539+0000", 2023-07-02T14:02:47.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0680000000000002e-06 2023-07-02T14:02:47.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:47.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286545+0000", 2023-07-02T14:02:47.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026497718 2023-07-02T14:02:47.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:47.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313043+0000", 2023-07-02T14:02:47.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0884999999999998e-05 2023-07-02T14:02:47.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:47.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313094+0000", 2023-07-02T14:02:47.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014109999999999999 2023-07-02T14:02:47.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:47.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313235+0000", 2023-07-02T14:02:47.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048258980000000003 2023-07-02T14:02:47.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:47.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318061+0000", 2023-07-02T14:02:47.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014346350000000001 2023-07-02T14:02:47.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:47.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319495+0000", 2023-07-02T14:02:47.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1986e-05 2023-07-02T14:02:47.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:47.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319547+0000", 2023-07-02T14:02:47.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0457e-05 2023-07-02T14:02:47.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:47.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319578+0000", 2023-07-02T14:02:47.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999 2023-07-02T14:02:47.581 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.581 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.582 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.582 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:201 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:47.582 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:02:47.582 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.1238524190000003, 2023-07-02T14:02:47.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003, 2023-07-02T14:02:47.583 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:47.583 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:47.583 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:47.583 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:47.583 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:47.583 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 201 2023-07-02T14:02:47.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.583 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:47.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:47.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:02:47.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:47.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:47.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:02:47.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:47.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:47.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:02:47.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019213999999999999 2023-07-02T14:02:47.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:47.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286744+0000", 2023-07-02T14:02:47.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5279999999999992e-06 2023-07-02T14:02:47.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:47.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286753+0000", 2023-07-02T14:02:47.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4200000000000004e-06 2023-07-02T14:02:47.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:47.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286760+0000", 2023-07-02T14:02:47.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026818386999999999 2023-07-02T14:02:47.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:47.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313578+0000", 2023-07-02T14:02:47.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015965 2023-07-02T14:02:47.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:47.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313738+0000", 2023-07-02T14:02:47.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015767299999999999 2023-07-02T14:02:47.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:47.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313895+0000", 2023-07-02T14:02:47.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041714930000000001 2023-07-02T14:02:47.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:47.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318067+0000", 2023-07-02T14:02:47.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001561698 2023-07-02T14:02:47.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:47.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319629+0000", 2023-07-02T14:02:47.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2346000000000003e-05 2023-07-02T14:02:47.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:47.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319681+0000", 2023-07-02T14:02:47.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4691000000000003e-05 2023-07-02T14:02:47.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:47.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319726+0000", 2023-07-02T14:02:47.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003 2023-07-02T14:02:47.593 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.593 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.593 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.594 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:204 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:47.594 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:02:47.594 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.1236375810000006, 2023-07-02T14:02:47.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998, 2023-07-02T14:02:47.594 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:47.594 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:47.594 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:47.595 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:47.595 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:47.595 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 204 2023-07-02T14:02:47.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.595 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:47.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:47.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:02:47.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:47.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:47.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:02:47.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:47.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:47.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:02:47.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2497999999999998e-05 2023-07-02T14:02:47.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:47.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286819+0000", 2023-07-02T14:02:47.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0099999999999995e-06 2023-07-02T14:02:47.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:47.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286827+0000", 2023-07-02T14:02:47.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000003e-06 2023-07-02T14:02:47.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:47.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286833+0000", 2023-07-02T14:02:47.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027384716999999999 2023-07-02T14:02:47.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:47.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314218+0000", 2023-07-02T14:02:47.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6204000000000004e-05 2023-07-02T14:02:47.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:47.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314294+0000", 2023-07-02T14:02:47.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015757499999999999 2023-07-02T14:02:47.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:47.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314452+0000", 2023-07-02T14:02:47.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036215560000000002 2023-07-02T14:02:47.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:47.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318073+0000", 2023-07-02T14:02:47.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017023979999999999 2023-07-02T14:02:47.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:47.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319776+0000", 2023-07-02T14:02:47.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9561000000000002e-05 2023-07-02T14:02:47.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:47.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319825+0000", 2023-07-02T14:02:47.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0329e-05 2023-07-02T14:02:47.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:47.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319866+0000", 2023-07-02T14:02:47.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998 2023-07-02T14:02:47.604 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.605 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.605 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.606 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:207 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:47.606 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:02:47.606 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.1235629360000008, 2023-07-02T14:02:47.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003, 2023-07-02T14:02:47.607 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:47.607 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:47.607 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:47.608 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:47.608 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:47.608 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 207 2023-07-02T14:02:47.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.609 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:47.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:47.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:02:47.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:47.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:47.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:02:47.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:47.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:47.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286841+0000", 2023-07-02T14:02:47.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00068375100000000002 2023-07-02T14:02:47.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:47.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287525+0000", 2023-07-02T14:02:47.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1601e-05 2023-07-02T14:02:47.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:47.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287537+0000", 2023-07-02T14:02:47.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6409999999999996e-06 2023-07-02T14:02:47.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:47.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287543+0000", 2023-07-02T14:02:47.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027245844000000002 2023-07-02T14:02:47.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:47.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314789+0000", 2023-07-02T14:02:47.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8910000000000003e-05 2023-07-02T14:02:47.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:47.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314858+0000", 2023-07-02T14:02:47.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030992599999999997 2023-07-02T14:02:47.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:47.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315168+0000", 2023-07-02T14:02:47.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029122869999999999 2023-07-02T14:02:47.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:47.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318080+0000", 2023-07-02T14:02:47.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019963020000000001 2023-07-02T14:02:47.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:47.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320077+0000", 2023-07-02T14:02:47.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5116000000000005e-05 2023-07-02T14:02:47.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:47.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320142+0000", 2023-07-02T14:02:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3793000000000004e-05 2023-07-02T14:02:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320205+0000", 2023-07-02T14:02:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003 2023-07-02T14:02:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.618 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.618 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:236 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2424832~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:47.618 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:02:47.618 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.1228507529999998, 2023-07-02T14:02:47.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003, 2023-07-02T14:02:47.619 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:47.619 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:47.619 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:47.619 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:47.619 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:47.619 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2023-07-02T14:02:47.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.619 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:02:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:47.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:02:47.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:47.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:47.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287553+0000", 2023-07-02T14:02:47.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0667e-05 2023-07-02T14:02:47.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:47.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287614+0000", 2023-07-02T14:02:47.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0454e-05 2023-07-02T14:02:47.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:47.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287624+0000", 2023-07-02T14:02:47.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8564999999999999e-05 2023-07-02T14:02:47.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:47.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287643+0000", 2023-07-02T14:02:47.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027977596 2023-07-02T14:02:47.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:47.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315621+0000", 2023-07-02T14:02:47.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000201728 2023-07-02T14:02:47.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:47.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315822+0000", 2023-07-02T14:02:47.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018436599999999999 2023-07-02T14:02:47.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:47.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316007+0000", 2023-07-02T14:02:47.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042423440000000003 2023-07-02T14:02:47.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:47.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320249+0000", 2023-07-02T14:02:47.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001452633 2023-07-02T14:02:47.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:47.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321702+0000", 2023-07-02T14:02:47.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3616000000000003e-05 2023-07-02T14:02:47.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:47.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321755+0000", 2023-07-02T14:02:47.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840999999999999e-05 2023-07-02T14:02:47.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:47.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321793+0000", 2023-07-02T14:02:47.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003 2023-07-02T14:02:47.628 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.628 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.629 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.629 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:251 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:47.629 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:02:47.629 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.1227552159999998, 2023-07-02T14:02:47.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001, 2023-07-02T14:02:47.630 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:47.630 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:47.630 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:47.630 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:47.630 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:47.630 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 251 2023-07-02T14:02:47.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.630 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:47.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:47.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:02:47.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:47.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:47.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:02:47.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:47.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:47.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287649+0000", 2023-07-02T14:02:47.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3232000000000001e-05 2023-07-02T14:02:47.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:47.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287702+0000", 2023-07-02T14:02:47.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0010000000000007e-06 2023-07-02T14:02:47.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:47.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287711+0000", 2023-07-02T14:02:47.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3569999999999997e-06 2023-07-02T14:02:47.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:47.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287717+0000", 2023-07-02T14:02:47.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028686454 2023-07-02T14:02:47.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:47.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316403+0000", 2023-07-02T14:02:47.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7741000000000002e-05 2023-07-02T14:02:47.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:47.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316461+0000", 2023-07-02T14:02:47.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4313000000000004e-05 2023-07-02T14:02:47.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:47.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316545+0000", 2023-07-02T14:02:47.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003734032 2023-07-02T14:02:47.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:47.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320279+0000", 2023-07-02T14:02:47.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015237219999999999 2023-07-02T14:02:47.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:47.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321803+0000", 2023-07-02T14:02:47.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1927e-05 2023-07-02T14:02:47.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:47.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321845+0000", 2023-07-02T14:02:47.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5398999999999998e-05 2023-07-02T14:02:47.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:47.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321880+0000", 2023-07-02T14:02:47.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001 2023-07-02T14:02:47.641 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.641 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.641 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.641 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:255 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2555904~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:47.642 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:02:47.642 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.1223404880000007, 2023-07-02T14:02:47.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002, 2023-07-02T14:02:47.642 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:47.642 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:47.642 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:47.642 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:47.642 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:47.643 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 255 2023-07-02T14:02:47.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.643 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:47.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:47.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:02:47.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:47.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:47.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:02:47.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:47.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:47.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288063+0000", 2023-07-02T14:02:47.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4459999999999997e-05 2023-07-02T14:02:47.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:47.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288118+0000", 2023-07-02T14:02:47.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0998999999999999e-05 2023-07-02T14:02:47.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:47.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288129+0000", 2023-07-02T14:02:47.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3219999999999998e-06 2023-07-02T14:02:47.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:47.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288135+0000", 2023-07-02T14:02:47.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029137386000000001 2023-07-02T14:02:47.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:47.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317273+0000", 2023-07-02T14:02:47.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7819999999999997e-05 2023-07-02T14:02:47.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:47.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317350+0000", 2023-07-02T14:02:47.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000174864 2023-07-02T14:02:47.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:47.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317525+0000", 2023-07-02T14:02:47.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027943519999999999 2023-07-02T14:02:47.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:47.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320320+0000", 2023-07-02T14:02:47.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015681460000000001 2023-07-02T14:02:47.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:47.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321888+0000", 2023-07-02T14:02:47.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9508999999999997e-05 2023-07-02T14:02:47.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:47.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321927+0000", 2023-07-02T14:02:47.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5428e-05 2023-07-02T14:02:47.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:47.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321963+0000", 2023-07-02T14:02:47.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002 2023-07-02T14:02:47.651 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.651 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.652 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.652 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:446 2.1 2:910b2a90:::rbd_data.125de1449605.000000000000010a:head [write 65536~3211264] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:47.652 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:02:47.652 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.0450607359999999, 2023-07-02T14:02:47.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003, 2023-07-02T14:02:47.653 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:47.653 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:47.653 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:47.653 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:47.653 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:47.653 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 446 2023-07-02T14:02:47.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.654 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:47.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:47.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:02:47.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:47.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:47.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:02:47.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:47.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:47.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:02:47.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015907372999999999 2023-07-02T14:02:47.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:47.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381251+0000", 2023-07-02T14:02:47.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5527000000000001e-05 2023-07-02T14:02:47.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:47.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381277+0000", 2023-07-02T14:02:47.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9958000000000001e-05 2023-07-02T14:02:47.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:47.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381297+0000", 2023-07-02T14:02:47.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4941000000000001e-05 2023-07-02T14:02:47.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:47.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381352+0000", 2023-07-02T14:02:47.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011565699999999999 2023-07-02T14:02:47.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:47.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381467+0000", 2023-07-02T14:02:47.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018039799999999999 2023-07-02T14:02:47.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:47.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381648+0000", 2023-07-02T14:02:47.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088183559999999994 2023-07-02T14:02:47.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:47.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.390466+0000", 2023-07-02T14:02:47.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007278278 2023-07-02T14:02:47.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:47.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397744+0000", 2023-07-02T14:02:47.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9289000000000001e-05 2023-07-02T14:02:47.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:47.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397804+0000", 2023-07-02T14:02:47.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0479000000000002e-05 2023-07-02T14:02:47.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:47.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:47.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:47.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397854+0000", 2023-07-02T14:02:47.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003 2023-07-02T14:02:47.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.662 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:47.663 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:47.663 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:47.665 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:02:47.665 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:47.663+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:47.707 INFO:tasks.workunit.client.0.smithi163.stderr:151550K .......... .......... .......... .......... .......... 89% 11.9M 1s 2023-07-02T14:02:47.708 INFO:tasks.workunit.client.0.smithi163.stderr:151600K .......... .......... .......... .......... .......... 89% 22.5M 1s 2023-07-02T14:02:47.708 INFO:tasks.workunit.client.0.smithi163.stderr:151650K .......... .......... .......... .......... .......... 89% 30.8M 1s 2023-07-02T14:02:47.708 INFO:tasks.workunit.client.0.smithi163.stderr:151700K .......... .......... .......... .......... .......... 90% 25.1M 1s 2023-07-02T14:02:47.708 INFO:tasks.workunit.client.0.smithi163.stderr:151750K .......... .......... .......... .......... .......... 90% 14.0M 1s 2023-07-02T14:02:47.708 INFO:tasks.workunit.client.0.smithi163.stderr:151800K .......... .......... .......... .......... .......... 90% 17.8M 1s 2023-07-02T14:02:47.708 INFO:tasks.workunit.client.0.smithi163.stderr:151850K .......... .......... .......... .......... .......... 90% 12.4M 1s 2023-07-02T14:02:47.708 INFO:tasks.workunit.client.0.smithi163.stderr:151900K .......... .......... .......... .......... .......... 90% 25.5M 1s 2023-07-02T14:02:47.709 INFO:tasks.workunit.client.0.smithi163.stderr:151950K .......... .......... .......... .......... .......... 90% 15.1M 1s 2023-07-02T14:02:47.709 INFO:tasks.workunit.client.0.smithi163.stderr:152000K .......... .......... .......... .......... .......... 90% 34.7M 1s 2023-07-02T14:02:47.709 INFO:tasks.workunit.client.0.smithi163.stderr:152050K .......... .......... .......... .......... .......... 90% 27.1M 1s 2023-07-02T14:02:47.709 INFO:tasks.workunit.client.0.smithi163.stderr:152100K .......... .......... .......... .......... .......... 90% 13.4M 1s 2023-07-02T14:02:47.709 INFO:tasks.workunit.client.0.smithi163.stderr:152150K .......... .......... .......... .......... .......... 90% 26.3M 1s 2023-07-02T14:02:47.709 INFO:tasks.workunit.client.0.smithi163.stderr:152200K .......... .......... .......... .......... .......... 90% 20.1M 1s 2023-07-02T14:02:47.709 INFO:tasks.workunit.client.0.smithi163.stderr:152250K .......... .......... .......... .......... .......... 90% 16.9M 1s 2023-07-02T14:02:47.709 INFO:tasks.workunit.client.0.smithi163.stderr:152300K .......... .......... .......... .......... .......... 90% 24.9M 1s 2023-07-02T14:02:47.710 INFO:tasks.workunit.client.0.smithi163.stderr:152350K .......... .......... .......... .......... .......... 90% 18.0M 1s 2023-07-02T14:02:47.710 INFO:tasks.workunit.client.0.smithi163.stderr:152400K .......... .......... .......... .......... .......... 90% 17.0M 1s 2023-07-02T14:02:47.710 INFO:tasks.workunit.client.0.smithi163.stderr:152450K .......... .......... .......... .......... .......... 90% 27.0M 1s 2023-07-02T14:02:47.710 INFO:tasks.workunit.client.0.smithi163.stderr:152500K .......... .......... .......... .......... .......... 90% 17.7M 1s 2023-07-02T14:02:47.710 INFO:tasks.workunit.client.0.smithi163.stderr:152550K .......... .......... .......... .......... .......... 90% 36.5M 1s 2023-07-02T14:02:47.710 INFO:tasks.workunit.client.0.smithi163.stderr:152600K .......... .......... .......... .......... .......... 90% 11.6M 1s 2023-07-02T14:02:47.710 INFO:tasks.workunit.client.0.smithi163.stderr:152650K .......... .......... .......... .......... .......... 90% 36.9M 1s 2023-07-02T14:02:47.710 INFO:tasks.workunit.client.0.smithi163.stderr:152700K .......... .......... .......... .......... .......... 90% 17.0M 1s 2023-07-02T14:02:47.711 INFO:tasks.workunit.client.0.smithi163.stderr:152750K .......... .......... .......... .......... .......... 90% 16.1M 1s 2023-07-02T14:02:47.711 INFO:tasks.workunit.client.0.smithi163.stderr:152800K .......... .......... .......... .......... .......... 90% 15.4M 1s 2023-07-02T14:02:47.711 INFO:tasks.workunit.client.0.smithi163.stderr:152850K .......... .......... .......... .......... .......... 90% 24.0M 1s 2023-07-02T14:02:47.711 INFO:tasks.workunit.client.0.smithi163.stderr:152900K .......... .......... .......... .......... .......... 90% 30.5M 1s 2023-07-02T14:02:47.711 INFO:tasks.workunit.client.0.smithi163.stderr:152950K .......... .......... .......... .......... .......... 90% 12.5M 1s 2023-07-02T14:02:47.711 INFO:tasks.workunit.client.0.smithi163.stderr:153000K .......... .......... .......... .......... .......... 90% 22.8M 1s 2023-07-02T14:02:47.711 INFO:tasks.workunit.client.0.smithi163.stderr:153050K .......... .......... .......... .......... .......... 90% 24.4M 1s 2023-07-02T14:02:47.712 INFO:tasks.workunit.client.0.smithi163.stderr:153100K .......... .......... .......... .......... .......... 90% 14.0M 1s 2023-07-02T14:02:47.712 INFO:tasks.workunit.client.0.smithi163.stderr:153150K .......... .......... .......... .......... .......... 90% 18.5M 1s 2023-07-02T14:02:47.712 INFO:tasks.workunit.client.0.smithi163.stderr:153200K .......... .......... .......... .......... .......... 90% 29.7M 1s 2023-07-02T14:02:47.712 INFO:tasks.workunit.client.0.smithi163.stderr:153250K .......... .......... .......... .......... .......... 90% 25.7M 1s 2023-07-02T14:02:47.712 INFO:tasks.workunit.client.0.smithi163.stderr:153300K .......... .......... .......... .......... .......... 90% 18.8M 1s 2023-07-02T14:02:47.712 INFO:tasks.workunit.client.0.smithi163.stderr:153350K .......... .......... .......... .......... .......... 91% 24.3M 1s 2023-07-02T14:02:47.712 INFO:tasks.workunit.client.0.smithi163.stderr:153400K .......... .......... .......... .......... .......... 91% 18.9M 1s 2023-07-02T14:02:47.712 INFO:tasks.workunit.client.0.smithi163.stderr:153450K .......... .......... .......... .......... .......... 91% 10.3M 1s 2023-07-02T14:02:47.713 INFO:tasks.workunit.client.0.smithi163.stderr:153500K .......... .......... .......... .......... .......... 91% 25.9M 1s 2023-07-02T14:02:47.713 INFO:tasks.workunit.client.0.smithi163.stderr:153550K .......... .......... .......... .......... .......... 91% 17.6M 1s 2023-07-02T14:02:47.713 INFO:tasks.workunit.client.0.smithi163.stderr:153600K .......... .......... .......... .......... .......... 91% 34.9M 1s 2023-07-02T14:02:47.713 INFO:tasks.workunit.client.0.smithi163.stderr:153650K .......... .......... .......... .......... .......... 91% 16.1M 1s 2023-07-02T14:02:47.713 INFO:tasks.workunit.client.0.smithi163.stderr:153700K .......... .......... .......... .......... .......... 91% 19.6M 1s 2023-07-02T14:02:47.713 INFO:tasks.workunit.client.0.smithi163.stderr:153750K .......... .......... .......... .......... .......... 91% 27.9M 1s 2023-07-02T14:02:47.713 INFO:tasks.workunit.client.0.smithi163.stderr:153800K .......... .......... .......... .......... .......... 91% 15.5M 1s 2023-07-02T14:02:47.713 INFO:tasks.workunit.client.0.smithi163.stderr:153850K .......... .......... .......... .......... .......... 91% 23.9M 1s 2023-07-02T14:02:47.714 INFO:tasks.workunit.client.0.smithi163.stderr:153900K .......... .......... .......... .......... .......... 91% 24.6M 1s 2023-07-02T14:02:47.714 INFO:tasks.workunit.client.0.smithi163.stderr:153950K .......... .......... .......... .......... .......... 91% 19.8M 1s 2023-07-02T14:02:47.714 INFO:tasks.workunit.client.0.smithi163.stderr:154000K .......... .......... .......... .......... .......... 91% 17.8M 1s 2023-07-02T14:02:47.714 INFO:tasks.workunit.client.0.smithi163.stderr:154050K .......... .......... .......... .......... .......... 91% 10.2M 1s 2023-07-02T14:02:47.714 INFO:tasks.workunit.client.0.smithi163.stderr:154100K .......... .......... .......... .......... .......... 91% 23.9M 1s 2023-07-02T14:02:47.714 INFO:tasks.workunit.client.0.smithi163.stderr:154150K .......... .......... .......... .......... .......... 91% 14.7M 1s 2023-07-02T14:02:47.714 INFO:tasks.workunit.client.0.smithi163.stderr:154200K .......... .......... .......... .......... .......... 91% 18.7M 1s 2023-07-02T14:02:47.714 INFO:tasks.workunit.client.0.smithi163.stderr:154250K .......... .......... .......... .......... .......... 91% 20.3M 1s 2023-07-02T14:02:47.715 INFO:tasks.workunit.client.0.smithi163.stderr:154300K .......... .......... .......... .......... .......... 91% 22.4M 1s 2023-07-02T14:02:47.715 INFO:tasks.workunit.client.0.smithi163.stderr:154350K .......... .......... .......... .......... .......... 91% 17.5M 1s 2023-07-02T14:02:47.715 INFO:tasks.workunit.client.0.smithi163.stderr:154400K .......... .......... .......... .......... .......... 91% 19.2M 1s 2023-07-02T14:02:47.715 INFO:tasks.workunit.client.0.smithi163.stderr:154450K .......... .......... .......... .......... .......... 91% 22.1M 1s 2023-07-02T14:02:47.715 INFO:tasks.workunit.client.0.smithi163.stderr:154500K .......... .......... .......... .......... .......... 91% 20.9M 1s 2023-07-02T14:02:47.715 INFO:tasks.workunit.client.0.smithi163.stderr:154550K .......... .......... .......... .......... .......... 91% 14.5M 1s 2023-07-02T14:02:47.715 INFO:tasks.workunit.client.0.smithi163.stderr:154600K .......... .......... .......... .......... .......... 91% 24.6M 1s 2023-07-02T14:02:47.715 INFO:tasks.workunit.client.0.smithi163.stderr:154650K .......... .......... .......... .......... .......... 91% 18.5M 1s 2023-07-02T14:02:47.716 INFO:tasks.workunit.client.0.smithi163.stderr:154700K .......... .......... .......... .......... .......... 91% 21.6M 1s 2023-07-02T14:02:47.716 INFO:tasks.workunit.client.0.smithi163.stderr:154750K .......... .......... .......... .......... .......... 91% 17.3M 1s 2023-07-02T14:02:47.716 INFO:tasks.workunit.client.0.smithi163.stderr:154800K .......... .......... .......... .......... .......... 91% 32.3M 1s 2023-07-02T14:02:47.716 INFO:tasks.workunit.client.0.smithi163.stderr:154850K .......... .......... .......... .......... .......... 91% 18.4M 1s 2023-07-02T14:02:47.716 INFO:tasks.workunit.client.0.smithi163.stderr:154900K .......... .......... .......... .......... .......... 91% 14.0M 1s 2023-07-02T14:02:47.716 INFO:tasks.workunit.client.0.smithi163.stderr:154950K .......... .......... .......... .......... .......... 91% 17.3M 1s 2023-07-02T14:02:47.716 INFO:tasks.workunit.client.0.smithi163.stderr:155000K .......... .......... .......... .......... .......... 91% 17.0M 1s 2023-07-02T14:02:47.716 INFO:tasks.workunit.client.0.smithi163.stderr:155050K .......... .......... .......... .......... .......... 92% 20.2M 1s 2023-07-02T14:02:47.717 INFO:tasks.workunit.client.0.smithi163.stderr:155100K .......... .......... .......... .......... .......... 92% 35.2M 1s 2023-07-02T14:02:47.717 INFO:tasks.workunit.client.0.smithi163.stderr:155150K .......... .......... .......... .......... .......... 92% 17.9M 1s 2023-07-02T14:02:47.717 INFO:tasks.workunit.client.0.smithi163.stderr:155200K .......... .......... .......... .......... .......... 92% 20.8M 1s 2023-07-02T14:02:47.717 INFO:tasks.workunit.client.0.smithi163.stderr:155250K .......... .......... .......... .......... .......... 92% 14.8M 1s 2023-07-02T14:02:47.717 INFO:tasks.workunit.client.0.smithi163.stderr:155300K .......... .......... .......... .......... .......... 92% 20.7M 1s 2023-07-02T14:02:47.717 INFO:tasks.workunit.client.0.smithi163.stderr:155350K .......... .......... .......... .......... .......... 92% 27.1M 1s 2023-07-02T14:02:47.717 INFO:tasks.workunit.client.0.smithi163.stderr:155400K .......... .......... .......... .......... .......... 92% 16.2M 1s 2023-07-02T14:02:47.718 INFO:tasks.workunit.client.0.smithi163.stderr:155450K .......... .......... .......... .......... .......... 92% 19.2M 1s 2023-07-02T14:02:47.718 INFO:tasks.workunit.client.0.smithi163.stderr:155500K .......... .......... .......... .......... .......... 92% 14.6M 1s 2023-07-02T14:02:47.718 INFO:tasks.workunit.client.0.smithi163.stderr:155550K .......... .......... .......... .......... .......... 92% 15.8M 1s 2023-07-02T14:02:47.718 INFO:tasks.workunit.client.0.smithi163.stderr:155600K .......... .......... .......... .......... .......... 92% 15.7M 1s 2023-07-02T14:02:47.718 INFO:tasks.workunit.client.0.smithi163.stderr:155650K .......... .......... .......... .......... .......... 92% 24.2M 1s 2023-07-02T14:02:47.718 INFO:tasks.workunit.client.0.smithi163.stderr:155700K .......... .......... .......... .......... .......... 92% 21.2M 1s 2023-07-02T14:02:47.718 INFO:tasks.workunit.client.0.smithi163.stderr:155750K .......... .......... .......... .......... .......... 92% 17.4M 1s 2023-07-02T14:02:47.718 INFO:tasks.workunit.client.0.smithi163.stderr:155800K .......... .......... .......... .......... .......... 92% 10.7M 1s 2023-07-02T14:02:47.719 INFO:tasks.workunit.client.0.smithi163.stderr:155850K .......... .......... .......... .......... .......... 92% 141M 1s 2023-07-02T14:02:47.719 INFO:tasks.workunit.client.0.smithi163.stderr:155900K .......... .......... .......... .......... .......... 92% 22.3M 1s 2023-07-02T14:02:47.719 INFO:tasks.workunit.client.0.smithi163.stderr:155950K .......... .......... .......... .......... .......... 92% 15.5M 1s 2023-07-02T14:02:47.719 INFO:tasks.workunit.client.0.smithi163.stderr:156000K .......... .......... .......... .......... .......... 92% 29.6M 1s 2023-07-02T14:02:47.719 INFO:tasks.workunit.client.0.smithi163.stderr:156050K .......... .......... .......... .......... .......... 92% 25.3M 1s 2023-07-02T14:02:47.719 INFO:tasks.workunit.client.0.smithi163.stderr:156100K .......... .......... .......... .......... .......... 92% 13.5M 1s 2023-07-02T14:02:47.719 INFO:tasks.workunit.client.0.smithi163.stderr:156150K .......... .......... .......... .......... .......... 92% 19.8M 1s 2023-07-02T14:02:47.719 INFO:tasks.workunit.client.0.smithi163.stderr:156200K .......... .......... .......... .......... .......... 92% 21.3M 1s 2023-07-02T14:02:47.720 INFO:tasks.workunit.client.0.smithi163.stderr:156250K .......... .......... .......... .......... .......... 92% 18.7M 1s 2023-07-02T14:02:47.720 INFO:tasks.workunit.client.0.smithi163.stderr:156300K .......... .......... .......... .......... .......... 92% 25.3M 1s 2023-07-02T14:02:47.720 INFO:tasks.workunit.client.0.smithi163.stderr:156350K .......... .......... .......... .......... .......... 92% 18.7M 1s 2023-07-02T14:02:47.720 INFO:tasks.workunit.client.0.smithi163.stderr:156400K .......... .......... .......... .......... .......... 92% 20.8M 1s 2023-07-02T14:02:47.720 INFO:tasks.workunit.client.0.smithi163.stderr:156450K .......... .......... .......... .......... .......... 92% 14.1M 1s 2023-07-02T14:02:47.720 INFO:tasks.workunit.client.0.smithi163.stderr:156500K .......... .......... .......... .......... .......... 92% 19.3M 1s 2023-07-02T14:02:47.720 INFO:tasks.workunit.client.0.smithi163.stderr:156550K .......... .......... .......... .......... .......... 92% 18.8M 1s 2023-07-02T14:02:47.720 INFO:tasks.workunit.client.0.smithi163.stderr:156600K .......... .......... .......... .......... .......... 92% 23.5M 1s 2023-07-02T14:02:47.721 INFO:tasks.workunit.client.0.smithi163.stderr:156650K .......... .......... .......... .......... .......... 92% 22.9M 1s 2023-07-02T14:02:47.721 INFO:tasks.workunit.client.0.smithi163.stderr:156700K .......... .......... .......... .......... .......... 92% 18.9M 1s 2023-07-02T14:02:47.721 INFO:tasks.workunit.client.0.smithi163.stderr:156750K .......... .......... .......... .......... .......... 93% 15.3M 1s 2023-07-02T14:02:47.721 INFO:tasks.workunit.client.0.smithi163.stderr:156800K .......... .......... .......... .......... .......... 93% 15.4M 1s 2023-07-02T14:02:47.721 INFO:tasks.workunit.client.0.smithi163.stderr:156850K .......... .......... .......... .......... .......... 93% 19.6M 1s 2023-07-02T14:02:47.721 INFO:tasks.workunit.client.0.smithi163.stderr:156900K .......... .......... .......... .......... .......... 93% 28.7M 1s 2023-07-02T14:02:47.721 INFO:tasks.workunit.client.0.smithi163.stderr:156950K .......... .......... .......... .......... .......... 93% 13.0M 1s 2023-07-02T14:02:47.721 INFO:tasks.workunit.client.0.smithi163.stderr:157000K .......... .......... .......... .......... .......... 93% 19.7M 1s 2023-07-02T14:02:47.722 INFO:tasks.workunit.client.0.smithi163.stderr:157050K .......... .......... .......... .......... .......... 93% 23.4M 1s 2023-07-02T14:02:47.722 INFO:tasks.workunit.client.0.smithi163.stderr:157100K .......... .......... .......... .......... .......... 93% 19.6M 1s 2023-07-02T14:02:47.722 INFO:tasks.workunit.client.0.smithi163.stderr:157150K .......... .......... .......... .......... .......... 93% 14.4M 1s 2023-07-02T14:02:47.722 INFO:tasks.workunit.client.0.smithi163.stderr:157200K .......... .......... .......... .......... .......... 93% 18.6M 1s 2023-07-02T14:02:47.722 INFO:tasks.workunit.client.0.smithi163.stderr:157250K .......... .......... .......... .......... .......... 93% 28.9M 1s 2023-07-02T14:02:47.722 INFO:tasks.workunit.client.0.smithi163.stderr:157300K .......... .......... .......... .......... .......... 93% 23.3M 1s 2023-07-02T14:02:47.725 INFO:tasks.workunit.client.0.smithi163.stderr:157350K .......... .......... .......... .......... .......... 93% 13.6M 1s 2023-07-02T14:02:47.725 INFO:tasks.workunit.client.0.smithi163.stderr:157400K .......... .......... .......... .......... .......... 93% 21.1M 1s 2023-07-02T14:02:47.725 INFO:tasks.workunit.client.0.smithi163.stderr:157450K .......... .......... .......... .......... .......... 93% 35.8M 1s 2023-07-02T14:02:47.725 INFO:tasks.workunit.client.0.smithi163.stderr:157500K .......... .......... .......... .......... .......... 93% 16.4M 1s 2023-07-02T14:02:47.726 INFO:tasks.workunit.client.0.smithi163.stderr:157550K .......... .......... .......... .......... .......... 93% 16.6M 1s 2023-07-02T14:02:47.726 INFO:tasks.workunit.client.0.smithi163.stderr:157600K .......... .......... .......... .......... .......... 93% 18.5M 1s 2023-07-02T14:02:47.726 INFO:tasks.workunit.client.0.smithi163.stderr:157650K .......... .......... .......... .......... .......... 93% 19.2M 1s 2023-07-02T14:02:47.728 INFO:tasks.workunit.client.0.smithi163.stderr:157700K .......... .......... .......... .......... .......... 93% 13.9M 1s 2023-07-02T14:02:47.730 INFO:tasks.workunit.client.0.smithi163.stderr:157750K .......... .......... .......... .......... .......... 93% 23.2M 1s 2023-07-02T14:02:47.731 INFO:tasks.workunit.client.0.smithi163.stderr:157800K .......... .......... .......... .......... .......... 93% 33.1M 1s 2023-07-02T14:02:47.734 INFO:tasks.workunit.client.0.smithi163.stderr:157850K .......... .......... .......... .......... .......... 93% 18.4M 1s 2023-07-02T14:02:47.736 INFO:tasks.workunit.client.0.smithi163.stderr:157900K .......... .......... .......... .......... .......... 93% 29.2M 1s 2023-07-02T14:02:47.740 INFO:tasks.workunit.client.0.smithi163.stderr:157950K .......... .......... .......... .......... .......... 93% 11.3M 1s 2023-07-02T14:02:47.742 INFO:tasks.workunit.client.0.smithi163.stderr:158000K .......... .......... .......... .......... .......... 93% 32.0M 1s 2023-07-02T14:02:47.746 INFO:tasks.workunit.client.0.smithi163.stderr:158050K .......... .......... .......... .......... .......... 93% 12.4M 1s 2023-07-02T14:02:47.747 INFO:tasks.workunit.client.0.smithi163.stderr:158100K .......... .......... .......... .......... .......... 93% 28.9M 1s 2023-07-02T14:02:47.749 INFO:tasks.workunit.client.0.smithi163.stderr:158150K .......... .......... .......... .......... .......... 93% 24.5M 1s 2023-07-02T14:02:47.752 INFO:tasks.workunit.client.0.smithi163.stderr:158200K .......... .......... .......... .......... .......... 93% 21.1M 1s 2023-07-02T14:02:47.753 INFO:tasks.workunit.client.0.smithi163.stderr:158250K .......... .......... .......... .......... .......... 93% 47.4M 1s 2023-07-02T14:02:47.754 INFO:tasks.workunit.client.0.smithi163.stderr:158300K .......... .......... .......... .......... .......... 93% 28.4M 1s 2023-07-02T14:02:47.757 INFO:tasks.workunit.client.0.smithi163.stderr:158350K .......... .......... .......... .......... .......... 93% 19.8M 1s 2023-07-02T14:02:47.759 INFO:tasks.workunit.client.0.smithi163.stderr:158400K .......... .......... .......... .......... .......... 93% 20.0M 1s 2023-07-02T14:02:47.761 INFO:tasks.workunit.client.0.smithi163.stderr:158450K .......... .......... .......... .......... .......... 94% 29.3M 1s 2023-07-02T14:02:47.762 INFO:tasks.workunit.client.0.smithi163.stderr:158500K .......... .......... .......... .......... .......... 94% 33.3M 0s 2023-07-02T14:02:47.764 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:47.763+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:47.765 INFO:tasks.workunit.client.0.smithi163.stderr:158550K .......... .......... .......... .......... .......... 94% 22.1M 0s 2023-07-02T14:02:47.766 INFO:tasks.workunit.client.0.smithi163.stderr:158600K .......... .......... .......... .......... .......... 94% 28.5M 0s 2023-07-02T14:02:47.768 INFO:tasks.workunit.client.0.smithi163.stderr:158650K .......... .......... .......... .......... .......... 94% 29.3M 0s 2023-07-02T14:02:47.770 INFO:tasks.workunit.client.0.smithi163.stderr:158700K .......... .......... .......... .......... .......... 94% 21.6M 0s 2023-07-02T14:02:47.772 INFO:tasks.workunit.client.0.smithi163.stderr:158750K .......... .......... .......... .......... .......... 94% 21.8M 0s 2023-07-02T14:02:47.774 INFO:tasks.workunit.client.0.smithi163.stderr:158800K .......... .......... .......... .......... .......... 94% 31.7M 0s 2023-07-02T14:02:47.775 INFO:tasks.workunit.client.0.smithi163.stderr:158850K .......... .......... .......... .......... .......... 94% 33.2M 0s 2023-07-02T14:02:47.778 INFO:tasks.workunit.client.0.smithi163.stderr:158900K .......... .......... .......... .......... .......... 94% 23.0M 0s 2023-07-02T14:02:47.779 INFO:tasks.workunit.client.0.smithi163.stderr:158950K .......... .......... .......... .......... .......... 94% 35.0M 0s 2023-07-02T14:02:47.780 INFO:tasks.workunit.client.0.smithi163.stderr:159000K .......... .......... .......... .......... .......... 94% 34.8M 0s 2023-07-02T14:02:47.782 INFO:tasks.workunit.client.0.smithi163.stderr:159050K .......... .......... .......... .......... .......... 94% 24.2M 0s 2023-07-02T14:02:47.785 INFO:tasks.workunit.client.0.smithi163.stderr:159100K .......... .......... .......... .......... .......... 94% 22.4M 0s 2023-07-02T14:02:47.787 INFO:tasks.workunit.client.0.smithi163.stderr:159150K .......... .......... .......... .......... .......... 94% 22.1M 0s 2023-07-02T14:02:47.789 INFO:tasks.workunit.client.0.smithi163.stderr:159200K .......... .......... .......... .......... .......... 94% 25.9M 0s 2023-07-02T14:02:47.791 INFO:tasks.workunit.client.0.smithi163.stderr:159250K .......... .......... .......... .......... .......... 94% 21.1M 0s 2023-07-02T14:02:47.792 INFO:tasks.workunit.client.0.smithi163.stderr:159300K .......... .......... .......... .......... .......... 94% 39.0M 0s 2023-07-02T14:02:47.794 INFO:tasks.workunit.client.0.smithi163.stderr:159350K .......... .......... .......... .......... .......... 94% 29.3M 0s 2023-07-02T14:02:47.795 INFO:tasks.workunit.client.0.smithi163.stderr:159400K .......... .......... .......... .......... .......... 94% 30.8M 0s 2023-07-02T14:02:47.799 INFO:tasks.workunit.client.0.smithi163.stderr:159450K .......... .......... .......... .......... .......... 94% 15.0M 0s 2023-07-02T14:02:47.801 INFO:tasks.workunit.client.0.smithi163.stderr:159500K .......... .......... .......... .......... .......... 94% 24.2M 0s 2023-07-02T14:02:47.803 INFO:tasks.workunit.client.0.smithi163.stderr:159550K .......... .......... .......... .......... .......... 94% 22.7M 0s 2023-07-02T14:02:47.805 INFO:tasks.workunit.client.0.smithi163.stderr:159600K .......... .......... .......... .......... .......... 94% 21.4M 0s 2023-07-02T14:02:47.806 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:47.806 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:47.806 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:47.806 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:47.807 INFO:tasks.workunit.client.0.smithi163.stderr:159650K .......... .......... .......... .......... .......... 94% 23.9M 0s 2023-07-02T14:02:47.809 INFO:tasks.workunit.client.0.smithi163.stderr:159700K .......... .......... .......... .......... .......... 94% 27.6M 0s 2023-07-02T14:02:47.811 INFO:tasks.workunit.client.0.smithi163.stderr:159750K .......... .......... .......... .......... .......... 94% 23.5M 0s 2023-07-02T14:02:47.813 INFO:tasks.workunit.client.0.smithi163.stderr:159800K .......... .......... .......... .......... .......... 94% 22.2M 0s 2023-07-02T14:02:47.815 INFO:tasks.workunit.client.0.smithi163.stderr:159850K .......... .......... .......... .......... .......... 94% 22.5M 0s 2023-07-02T14:02:47.816 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:02:47.817 INFO:tasks.workunit.client.0.smithi163.stderr:159900K .......... .......... .......... .......... .......... 94% 29.8M 0s 2023-07-02T14:02:47.820 INFO:tasks.workunit.client.0.smithi163.stderr:159950K .......... .......... .......... .......... .......... 94% 17.0M 0s 2023-07-02T14:02:47.822 INFO:tasks.workunit.client.0.smithi163.stderr:160000K .......... .......... .......... .......... .......... 94% 22.8M 0s 2023-07-02T14:02:47.824 INFO:tasks.workunit.client.0.smithi163.stderr:160050K .......... .......... .......... .......... .......... 94% 27.2M 0s 2023-07-02T14:02:47.826 INFO:tasks.workunit.client.0.smithi163.stderr:160100K .......... .......... .......... .......... .......... 95% 23.9M 0s 2023-07-02T14:02:47.828 INFO:tasks.workunit.client.0.smithi163.stderr:160150K .......... .......... .......... .......... .......... 95% 24.1M 0s 2023-07-02T14:02:47.830 INFO:tasks.workunit.client.0.smithi163.stderr:160200K .......... .......... .......... .......... .......... 95% 21.8M 0s 2023-07-02T14:02:47.831 INFO:tasks.workunit.client.0.smithi163.stderr:160250K .......... .......... .......... .......... .......... 95% 41.1M 0s 2023-07-02T14:02:47.834 INFO:tasks.workunit.client.0.smithi163.stderr:160300K .......... .......... .......... .......... .......... 95% 20.1M 0s 2023-07-02T14:02:47.837 INFO:tasks.workunit.client.0.smithi163.stderr:160350K .......... .......... .......... .......... .......... 95% 17.7M 0s 2023-07-02T14:02:47.839 INFO:tasks.workunit.client.0.smithi163.stderr:160400K .......... .......... .......... .......... .......... 95% 18.4M 0s 2023-07-02T14:02:47.841 INFO:tasks.workunit.client.0.smithi163.stderr:160450K .......... .......... .......... .......... .......... 95% 29.2M 0s 2023-07-02T14:02:47.843 INFO:tasks.workunit.client.0.smithi163.stderr:160500K .......... .......... .......... .......... .......... 95% 25.7M 0s 2023-07-02T14:02:47.845 INFO:tasks.workunit.client.0.smithi163.stderr:160550K .......... .......... .......... .......... .......... 95% 28.9M 0s 2023-07-02T14:02:47.847 INFO:tasks.workunit.client.0.smithi163.stderr:160600K .......... .......... .......... .......... .......... 95% 19.3M 0s 2023-07-02T14:02:47.848 INFO:tasks.workunit.client.0.smithi163.stderr:160650K .......... .......... .......... .......... .......... 95% 41.3M 0s 2023-07-02T14:02:47.852 INFO:tasks.workunit.client.0.smithi163.stderr:160700K .......... .......... .......... .......... .......... 95% 15.4M 0s 2023-07-02T14:02:47.854 INFO:tasks.workunit.client.0.smithi163.stderr:160750K .......... .......... .......... .......... .......... 95% 23.2M 0s 2023-07-02T14:02:47.856 INFO:tasks.workunit.client.0.smithi163.stderr:160800K .......... .......... .......... .......... .......... 95% 21.7M 0s 2023-07-02T14:02:47.859 INFO:tasks.workunit.client.0.smithi163.stderr:160850K .......... .......... .......... .......... .......... 95% 15.3M 0s 2023-07-02T14:02:47.861 INFO:tasks.workunit.client.0.smithi163.stderr:160900K .......... .......... .......... .......... .......... 95% 20.8M 0s 2023-07-02T14:02:47.863 INFO:tasks.workunit.client.0.smithi163.stderr:160950K .......... .......... .......... .......... .......... 95% 34.0M 0s 2023-07-02T14:02:47.864 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:47.864+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:47.865 INFO:tasks.workunit.client.0.smithi163.stderr:161000K .......... .......... .......... .......... .......... 95% 21.6M 0s 2023-07-02T14:02:47.866 INFO:tasks.workunit.client.0.smithi163.stderr:161050K .......... .......... .......... .......... .......... 95% 46.9M 0s 2023-07-02T14:02:47.868 INFO:tasks.workunit.client.0.smithi163.stderr:161100K .......... .......... .......... .......... .......... 95% 20.2M 0s 2023-07-02T14:02:47.871 INFO:tasks.workunit.client.0.smithi163.stderr:161150K .......... .......... .......... .......... .......... 95% 19.1M 0s 2023-07-02T14:02:47.872 INFO:tasks.workunit.client.0.smithi163.stderr:161200K .......... .......... .......... .......... .......... 95% 37.1M 0s 2023-07-02T14:02:47.875 INFO:tasks.workunit.client.0.smithi163.stderr:161250K .......... .......... .......... .......... .......... 95% 19.0M 0s 2023-07-02T14:02:47.876 INFO:tasks.workunit.client.0.smithi163.stderr:161300K .......... .......... .......... .......... .......... 95% 50.3M 0s 2023-07-02T14:02:47.878 INFO:tasks.workunit.client.0.smithi163.stderr:161350K .......... .......... .......... .......... .......... 95% 22.7M 0s 2023-07-02T14:02:47.881 INFO:tasks.workunit.client.0.smithi163.stderr:161400K .......... .......... .......... .......... .......... 95% 19.0M 0s 2023-07-02T14:02:47.882 INFO:tasks.workunit.client.0.smithi163.stderr:161450K .......... .......... .......... .......... .......... 95% 39.6M 0s 2023-07-02T14:02:47.884 INFO:tasks.workunit.client.0.smithi163.stderr:161500K .......... .......... .......... .......... .......... 95% 20.7M 0s 2023-07-02T14:02:47.886 INFO:tasks.workunit.client.0.smithi163.stderr:161550K .......... .......... .......... .......... .......... 95% 23.3M 0s 2023-07-02T14:02:47.888 INFO:tasks.workunit.client.0.smithi163.stderr:161600K .......... .......... .......... .......... .......... 95% 28.3M 0s 2023-07-02T14:02:47.890 INFO:tasks.workunit.client.0.smithi163.stderr:161650K .......... .......... .......... .......... .......... 95% 25.0M 0s 2023-07-02T14:02:47.892 INFO:tasks.workunit.client.0.smithi163.stderr:161700K .......... .......... .......... .......... .......... 95% 21.2M 0s 2023-07-02T14:02:47.895 INFO:tasks.workunit.client.0.smithi163.stderr:161750K .......... .......... .......... .......... .......... 95% 17.6M 0s 2023-07-02T14:02:47.898 INFO:tasks.workunit.client.0.smithi163.stderr:161800K .......... .......... .......... .......... .......... 96% 18.0M 0s 2023-07-02T14:02:47.900 INFO:tasks.workunit.client.0.smithi163.stderr:161850K .......... .......... .......... .......... .......... 96% 19.2M 0s 2023-07-02T14:02:47.902 INFO:tasks.workunit.client.0.smithi163.stderr:161900K .......... .......... .......... .......... .......... 96% 38.1M 0s 2023-07-02T14:02:47.905 INFO:tasks.workunit.client.0.smithi163.stderr:161950K .......... .......... .......... .......... .......... 96% 15.6M 0s 2023-07-02T14:02:47.906 INFO:tasks.workunit.client.0.smithi163.stderr:162000K .......... .......... .......... .......... .......... 96% 30.4M 0s 2023-07-02T14:02:47.909 INFO:tasks.workunit.client.0.smithi163.stderr:162050K .......... .......... .......... .......... .......... 96% 16.9M 0s 2023-07-02T14:02:47.911 INFO:tasks.workunit.client.0.smithi163.stderr:162100K .......... .......... .......... .......... .......... 96% 30.3M 0s 2023-07-02T14:02:47.914 INFO:tasks.workunit.client.0.smithi163.stderr:162150K .......... .......... .......... .......... .......... 96% 17.5M 0s 2023-07-02T14:02:47.915 INFO:tasks.workunit.client.0.smithi163.stderr:162200K .......... .......... .......... .......... .......... 96% 34.5M 0s 2023-07-02T14:02:47.917 INFO:tasks.workunit.client.0.smithi163.stderr:162250K .......... .......... .......... .......... .......... 96% 26.9M 0s 2023-07-02T14:02:47.919 INFO:tasks.workunit.client.0.smithi163.stderr:162300K .......... .......... .......... .......... .......... 96% 19.5M 0s 2023-07-02T14:02:47.922 INFO:tasks.workunit.client.0.smithi163.stderr:162350K .......... .......... .......... .......... .......... 96% 16.2M 0s 2023-07-02T14:02:47.924 INFO:tasks.workunit.client.0.smithi163.stderr:162400K .......... .......... .......... .......... .......... 96% 25.4M 0s 2023-07-02T14:02:47.927 INFO:tasks.workunit.client.0.smithi163.stderr:162450K .......... .......... .......... .......... .......... 96% 19.8M 0s 2023-07-02T14:02:47.928 INFO:tasks.workunit.client.0.smithi163.stderr:162500K .......... .......... .......... .......... .......... 96% 43.9M 0s 2023-07-02T14:02:47.930 INFO:tasks.workunit.client.0.smithi163.stderr:162550K .......... .......... .......... .......... .......... 96% 30.1M 0s 2023-07-02T14:02:47.931 INFO:tasks.workunit.client.0.smithi163.stderr:162600K .......... .......... .......... .......... .......... 96% 26.9M 0s 2023-07-02T14:02:47.934 INFO:tasks.workunit.client.0.smithi163.stderr:162650K .......... .......... .......... .......... .......... 96% 20.0M 0s 2023-07-02T14:02:47.935 INFO:tasks.workunit.client.0.smithi163.stderr:162700K .......... .......... .......... .......... .......... 96% 38.5M 0s 2023-07-02T14:02:47.937 INFO:tasks.workunit.client.0.smithi163.stderr:162750K .......... .......... .......... .......... .......... 96% 21.2M 0s 2023-07-02T14:02:47.940 INFO:tasks.workunit.client.0.smithi163.stderr:162800K .......... .......... .......... .......... .......... 96% 23.9M 0s 2023-07-02T14:02:47.940 INFO:tasks.workunit.client.0.smithi163.stderr:162850K .......... .......... .......... .......... .......... 96% 59.8M 0s 2023-07-02T14:02:47.941 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:47.941 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:47.941 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:47.941 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:47.941 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:47.943 INFO:tasks.workunit.client.0.smithi163.stderr:162900K .......... .......... .......... .......... .......... 96% 20.1M 0s 2023-07-02T14:02:47.944 INFO:tasks.workunit.client.0.smithi163.stderr:162950K .......... .......... .......... .......... .......... 96% 43.8M 0s 2023-07-02T14:02:47.946 INFO:tasks.workunit.client.0.smithi163.stderr:163000K .......... .......... .......... .......... .......... 96% 21.9M 0s 2023-07-02T14:02:47.947 INFO:tasks.workunit.client.0.smithi163.stderr:163050K .......... .......... .......... .......... .......... 96% 45.0M 0s 2023-07-02T14:02:47.949 INFO:tasks.workunit.client.0.smithi163.stderr:163100K .......... .......... .......... .......... .......... 96% 27.0M 0s 2023-07-02T14:02:47.951 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:02:47.952 INFO:tasks.workunit.client.0.smithi163.stderr:163150K .......... .......... .......... .......... .......... 96% 23.8M 0s 2023-07-02T14:02:47.953 INFO:tasks.workunit.client.0.smithi163.stderr:163200K .......... .......... .......... .......... .......... 96% 23.4M 0s 2023-07-02T14:02:47.955 INFO:tasks.workunit.client.0.smithi163.stderr:163250K .......... .......... .......... .......... .......... 96% 31.6M 0s 2023-07-02T14:02:47.957 INFO:tasks.workunit.client.0.smithi163.stderr:163300K .......... .......... .......... .......... .......... 96% 24.0M 0s 2023-07-02T14:02:47.958 INFO:tasks.workunit.client.0.smithi163.stderr:163350K .......... .......... .......... .......... .......... 96% 48.5M 0s 2023-07-02T14:02:47.961 INFO:tasks.workunit.client.0.smithi163.stderr:163400K .......... .......... .......... .......... .......... 96% 16.2M 0s 2023-07-02T14:02:47.962 INFO:tasks.workunit.client.0.smithi163.stderr:163450K .......... .......... .......... .......... .......... 96% 42.3M 0s 2023-07-02T14:02:47.965 INFO:tasks.workunit.client.0.smithi163.stderr:163500K .......... .......... .......... .......... .......... 97% 15.4M 0s 2023-07-02T14:02:47.966 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:47.964+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:47.968 INFO:tasks.workunit.client.0.smithi163.stderr:163550K .......... .......... .......... .......... .......... 97% 14.8M 0s 2023-07-02T14:02:47.969 INFO:tasks.workunit.client.0.smithi163.stderr:163600K .......... .......... .......... .......... .......... 97% 37.9M 0s 2023-07-02T14:02:47.972 INFO:tasks.workunit.client.0.smithi163.stderr:163650K .......... .......... .......... .......... .......... 97% 21.9M 0s 2023-07-02T14:02:47.974 INFO:tasks.workunit.client.0.smithi163.stderr:163700K .......... .......... .......... .......... .......... 97% 18.3M 0s 2023-07-02T14:02:47.976 INFO:tasks.workunit.client.0.smithi163.stderr:163750K .......... .......... .......... .......... .......... 97% 40.0M 0s 2023-07-02T14:02:47.979 INFO:tasks.workunit.client.0.smithi163.stderr:163800K .......... .......... .......... .......... .......... 97% 19.4M 0s 2023-07-02T14:02:47.980 INFO:tasks.workunit.client.0.smithi163.stderr:163850K .......... .......... .......... .......... .......... 97% 46.3M 0s 2023-07-02T14:02:47.982 INFO:tasks.workunit.client.0.smithi163.stderr:163900K .......... .......... .......... .......... .......... 97% 19.6M 0s 2023-07-02T14:02:47.984 INFO:tasks.workunit.client.0.smithi163.stderr:163950K .......... .......... .......... .......... .......... 97% 22.4M 0s 2023-07-02T14:02:47.985 INFO:tasks.workunit.client.0.smithi163.stderr:164000K .......... .......... .......... .......... .......... 97% 43.7M 0s 2023-07-02T14:02:47.988 INFO:tasks.workunit.client.0.smithi163.stderr:164050K .......... .......... .......... .......... .......... 97% 20.4M 0s 2023-07-02T14:02:47.988 INFO:tasks.workunit.client.0.smithi163.stderr:164100K .......... .......... .......... .......... .......... 97% 42.8M 0s 2023-07-02T14:02:47.991 INFO:tasks.workunit.client.0.smithi163.stderr:164150K .......... .......... .......... .......... .......... 97% 22.9M 0s 2023-07-02T14:02:47.993 INFO:tasks.workunit.client.0.smithi163.stderr:164200K .......... .......... .......... .......... .......... 97% 23.3M 0s 2023-07-02T14:02:47.994 INFO:tasks.workunit.client.0.smithi163.stderr:164250K .......... .......... .......... .......... .......... 97% 45.5M 0s 2023-07-02T14:02:47.996 INFO:tasks.workunit.client.0.smithi163.stderr:164300K .......... .......... .......... .......... .......... 97% 38.1M 0s 2023-07-02T14:02:47.999 INFO:tasks.workunit.client.0.smithi163.stderr:164350K .......... .......... .......... .......... .......... 97% 13.5M 0s 2023-07-02T14:02:48.000 INFO:tasks.workunit.client.0.smithi163.stderr:164400K .......... .......... .......... .......... .......... 97% 39.7M 0s 2023-07-02T14:02:48.002 INFO:tasks.workunit.client.0.smithi163.stderr:164450K .......... .......... .......... .......... .......... 97% 21.1M 0s 2023-07-02T14:02:48.004 INFO:tasks.workunit.client.0.smithi163.stderr:164500K .......... .......... .......... .......... .......... 97% 25.0M 0s 2023-07-02T14:02:48.008 INFO:tasks.workunit.client.0.smithi163.stderr:164550K .......... .......... .......... .......... .......... 97% 13.8M 0s 2023-07-02T14:02:48.009 INFO:tasks.workunit.client.0.smithi163.stderr:164600K .......... .......... .......... .......... .......... 97% 38.7M 0s 2023-07-02T14:02:48.012 INFO:tasks.workunit.client.0.smithi163.stderr:164650K .......... .......... .......... .......... .......... 97% 19.1M 0s 2023-07-02T14:02:48.013 INFO:tasks.workunit.client.0.smithi163.stderr:164700K .......... .......... .......... .......... .......... 97% 29.9M 0s 2023-07-02T14:02:48.016 INFO:tasks.workunit.client.0.smithi163.stderr:164750K .......... .......... .......... .......... .......... 97% 22.4M 0s 2023-07-02T14:02:48.019 INFO:tasks.workunit.client.0.smithi163.stderr:164800K .......... .......... .......... .......... .......... 97% 12.7M 0s 2023-07-02T14:02:48.022 INFO:tasks.workunit.client.0.smithi163.stderr:164850K .......... .......... .......... .......... .......... 97% 32.0M 0s 2023-07-02T14:02:48.024 INFO:tasks.workunit.client.0.smithi163.stderr:164900K .......... .......... .......... .......... .......... 97% 16.9M 0s 2023-07-02T14:02:48.028 INFO:tasks.workunit.client.0.smithi163.stderr:164950K .......... .......... .......... .......... .......... 97% 13.0M 0s 2023-07-02T14:02:48.029 INFO:tasks.workunit.client.0.smithi163.stderr:165000K .......... .......... .......... .......... .......... 97% 32.2M 0s 2023-07-02T14:02:48.031 INFO:tasks.workunit.client.0.smithi163.stderr:165050K .......... .......... .......... .......... .......... 97% 22.8M 0s 2023-07-02T14:02:48.032 INFO:tasks.workunit.client.0.smithi163.stderr:165100K .......... .......... .......... .......... .......... 97% 41.0M 0s 2023-07-02T14:02:48.035 INFO:tasks.workunit.client.0.smithi163.stderr:165150K .......... .......... .......... .......... .......... 98% 20.2M 0s 2023-07-02T14:02:48.037 INFO:tasks.workunit.client.0.smithi163.stderr:165200K .......... .......... .......... .......... .......... 98% 23.7M 0s 2023-07-02T14:02:48.038 INFO:tasks.workunit.client.0.smithi163.stderr:165250K .......... .......... .......... .......... .......... 98% 40.8M 0s 2023-07-02T14:02:48.040 INFO:tasks.workunit.client.0.smithi163.stderr:165300K .......... .......... .......... .......... .......... 98% 23.9M 0s 2023-07-02T14:02:48.041 INFO:tasks.workunit.client.0.smithi163.stderr:165350K .......... .......... .......... .......... .......... 98% 35.2M 0s 2023-07-02T14:02:48.044 INFO:tasks.workunit.client.0.smithi163.stderr:165400K .......... .......... .......... .......... .......... 98% 23.3M 0s 2023-07-02T14:02:48.046 INFO:tasks.workunit.client.0.smithi163.stderr:165450K .......... .......... .......... .......... .......... 98% 22.1M 0s 2023-07-02T14:02:48.047 INFO:tasks.workunit.client.0.smithi163.stderr:165500K .......... .......... .......... .......... .......... 98% 43.2M 0s 2023-07-02T14:02:48.050 INFO:tasks.workunit.client.0.smithi163.stderr:165550K .......... .......... .......... .......... .......... 98% 21.5M 0s 2023-07-02T14:02:48.051 INFO:tasks.workunit.client.0.smithi163.stderr:165600K .......... .......... .......... .......... .......... 98% 35.7M 0s 2023-07-02T14:02:48.052 INFO:tasks.workunit.client.0.smithi163.stderr:165650K .......... .......... .......... .......... .......... 98% 30.5M 0s 2023-07-02T14:02:48.055 INFO:tasks.workunit.client.0.smithi163.stderr:165700K .......... .......... .......... .......... .......... 98% 22.2M 0s 2023-07-02T14:02:48.056 INFO:tasks.workunit.client.0.smithi163.stderr:165750K .......... .......... .......... .......... .......... 98% 31.2M 0s 2023-07-02T14:02:48.059 INFO:tasks.workunit.client.0.smithi163.stderr:165800K .......... .......... .......... .......... .......... 98% 14.3M 0s 2023-07-02T14:02:48.063 INFO:tasks.workunit.client.0.smithi163.stderr:165850K .......... .......... .......... .......... .......... 98% 14.9M 0s 2023-07-02T14:02:48.065 INFO:tasks.workunit.client.0.smithi163.stderr:165900K .......... .......... .......... .......... .......... 98% 18.2M 0s 2023-07-02T14:02:48.066 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:48.065+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:48.069 INFO:tasks.workunit.client.0.smithi163.stderr:165950K .......... .......... .......... .......... .......... 98% 15.8M 0s 2023-07-02T14:02:48.071 INFO:tasks.workunit.client.0.smithi163.stderr:166000K .......... .......... .......... .......... .......... 98% 22.0M 0s 2023-07-02T14:02:48.072 INFO:tasks.workunit.client.0.smithi163.stderr:166050K .......... .......... .......... .......... .......... 98% 30.2M 0s 2023-07-02T14:02:48.074 INFO:tasks.workunit.client.0.smithi163.stderr:166100K .......... .......... .......... .......... .......... 98% 24.4M 0s 2023-07-02T14:02:48.075 INFO:tasks.workunit.client.0.smithi163.stderr:166150K .......... .......... .......... .......... .......... 98% 41.0M 0s 2023-07-02T14:02:48.078 INFO:tasks.workunit.client.0.smithi163.stderr:166200K .......... .......... .......... .......... .......... 98% 20.7M 0s 2023-07-02T14:02:48.079 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:48.079 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:48.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:48.080 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:02:48.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.080 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:322 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:48.080 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:02:48.080 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7779223910000006, 2023-07-02T14:02:48.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.03676041, 2023-07-02T14:02:48.081 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:48.081 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:48.081 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:48.081 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:48.081 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:48.081 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 322 2023-07-02T14:02:48.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.081 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:48.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:48.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:02:48.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:48.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:48.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:02:48.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:48.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:48.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298819+0000", 2023-07-02T14:02:48.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8068e-05 2023-07-02T14:02:48.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:48.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298877+0000", 2023-07-02T14:02:48.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3339999999999998e-06 2023-07-02T14:02:48.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:48.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298883+0000", 2023-07-02T14:02:48.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1030000000000004e-06 2023-07-02T14:02:48.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:48.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298887+0000", 2023-07-02T14:02:48.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023640230000000002 2023-07-02T14:02:48.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322528+0000", 2023-07-02T14:02:48.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8424999999999994e-05 2023-07-02T14:02:48.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:48.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322616+0000", 2023-07-02T14:02:48.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116815 2023-07-02T14:02:48.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:48.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322733+0000", 2023-07-02T14:02:48.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048252629999999998 2023-07-02T14:02:48.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:48.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327558+0000", 2023-07-02T14:02:48.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078965240000000003 2023-07-02T14:02:48.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:48.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335455+0000", 2023-07-02T14:02:48.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0511999999999999e-05 2023-07-02T14:02:48.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:48.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335515+0000", 2023-07-02T14:02:48.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4789000000000006e-05 2023-07-02T14:02:48.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:48.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335580+0000", 2023-07-02T14:02:48.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.03676041 2023-07-02T14:02:48.090 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.090 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.090 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.091 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:323 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:48.091 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:02:48.091 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7778510109999992, 2023-07-02T14:02:48.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036875044000000003, 2023-07-02T14:02:48.091 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:48.092 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:48.092 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:48.092 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:48.092 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:48.093 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 323 2023-07-02T14:02:48.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.093 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:48.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:48.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:02:48.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:48.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:48.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:02:48.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:48.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:48.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298890+0000", 2023-07-02T14:02:48.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7277000000000002e-05 2023-07-02T14:02:48.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:48.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298938+0000", 2023-07-02T14:02:48.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0495e-05 2023-07-02T14:02:48.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:48.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298948+0000", 2023-07-02T14:02:48.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6829999999999999e-06 2023-07-02T14:02:48.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:48.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298953+0000", 2023-07-02T14:02:48.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024271435000000001 2023-07-02T14:02:48.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323224+0000", 2023-07-02T14:02:48.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9266999999999998e-05 2023-07-02T14:02:48.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:48.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323304+0000", 2023-07-02T14:02:48.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000157021 2023-07-02T14:02:48.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:48.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323461+0000", 2023-07-02T14:02:48.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041036689999999999 2023-07-02T14:02:48.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:48.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327564+0000", 2023-07-02T14:02:48.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080753490000000008 2023-07-02T14:02:48.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:48.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335640+0000", 2023-07-02T14:02:48.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7398999999999994e-05 2023-07-02T14:02:48.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:48.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335727+0000", 2023-07-02T14:02:48.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8967000000000001e-05 2023-07-02T14:02:48.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:48.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335766+0000", 2023-07-02T14:02:48.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036875044000000003 2023-07-02T14:02:48.104 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.104 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.104 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.104 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:324 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:48.105 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:02:48.105 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7777836330000003, 2023-07-02T14:02:48.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036935673000000002, 2023-07-02T14:02:48.105 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:48.105 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:48.105 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:48.105 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:48.106 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:48.106 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 324 2023-07-02T14:02:48.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.106 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:48.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:48.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:02:48.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:48.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:48.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:02:48.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:48.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:48.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:02:48.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5519999999999997e-05 2023-07-02T14:02:48.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:48.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299013+0000", 2023-07-02T14:02:48.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0701e-05 2023-07-02T14:02:48.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:48.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299024+0000", 2023-07-02T14:02:48.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2379999999999999e-06 2023-07-02T14:02:48.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:48.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299028+0000", 2023-07-02T14:02:48.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024972768999999999 2023-07-02T14:02:48.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324001+0000", 2023-07-02T14:02:48.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1178000000000002e-05 2023-07-02T14:02:48.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:48.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324072+0000", 2023-07-02T14:02:48.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000166119 2023-07-02T14:02:48.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:48.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324238+0000", 2023-07-02T14:02:48.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033314210000000002 2023-07-02T14:02:48.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:48.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327570+0000", 2023-07-02T14:02:48.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0082393369999999994 2023-07-02T14:02:48.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:48.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335809+0000", 2023-07-02T14:02:48.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7071999999999999e-05 2023-07-02T14:02:48.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:48.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335856+0000", 2023-07-02T14:02:48.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7926999999999998e-05 2023-07-02T14:02:48.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:48.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335894+0000", 2023-07-02T14:02:48.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036935673000000002 2023-07-02T14:02:48.115 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.115 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.115 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.116 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:325 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:48.116 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:02:48.116 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7777083059999992, 2023-07-02T14:02:48.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036990838999999998, 2023-07-02T14:02:48.116 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:48.116 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:48.117 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:48.117 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:48.117 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:48.117 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 325 2023-07-02T14:02:48.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.117 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:48.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:48.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:02:48.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:48.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:48.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:02:48.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:48.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:48.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299033+0000", 2023-07-02T14:02:48.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107619 2023-07-02T14:02:48.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:48.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299141+0000", 2023-07-02T14:02:48.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3497e-05 2023-07-02T14:02:48.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:48.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299154+0000", 2023-07-02T14:02:48.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4669e-05 2023-07-02T14:02:48.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:48.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299169+0000", 2023-07-02T14:02:48.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025630159999999999 2023-07-02T14:02:48.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324799+0000", 2023-07-02T14:02:48.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6953000000000007e-05 2023-07-02T14:02:48.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:48.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324876+0000", 2023-07-02T14:02:48.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016346199999999999 2023-07-02T14:02:48.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:48.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325039+0000", 2023-07-02T14:02:48.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003566624 2023-07-02T14:02:48.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:48.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328606+0000", 2023-07-02T14:02:48.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073295540000000003 2023-07-02T14:02:48.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:48.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335936+0000", 2023-07-02T14:02:48.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0044999999999999e-05 2023-07-02T14:02:48.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:48.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335986+0000", 2023-07-02T14:02:48.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8872999999999999e-05 2023-07-02T14:02:48.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:48.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336024+0000", 2023-07-02T14:02:48.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036990838999999998 2023-07-02T14:02:48.126 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.126 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.127 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.127 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:326 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:48.127 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:02:48.127 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7775663779999995, 2023-07-02T14:02:48.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036975239999999999, 2023-07-02T14:02:48.128 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:48.128 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:48.128 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:48.128 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:48.128 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:48.128 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 326 2023-07-02T14:02:48.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.129 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:48.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:48.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:02:48.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:48.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:48.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:02:48.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:48.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:48.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299175+0000", 2023-07-02T14:02:48.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4782000000000003e-05 2023-07-02T14:02:48.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:48.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299250+0000", 2023-07-02T14:02:48.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.277e-05 2023-07-02T14:02:48.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:48.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299262+0000", 2023-07-02T14:02:48.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1460000000000001e-06 2023-07-02T14:02:48.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:48.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299268+0000", 2023-07-02T14:02:48.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026158098000000001 2023-07-02T14:02:48.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325427+0000", 2023-07-02T14:02:48.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4845999999999995e-05 2023-07-02T14:02:48.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:48.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325501+0000", 2023-07-02T14:02:48.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015890699999999999 2023-07-02T14:02:48.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:48.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325660+0000", 2023-07-02T14:02:48.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002952114 2023-07-02T14:02:48.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:48.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328612+0000", 2023-07-02T14:02:48.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074491949999999996 2023-07-02T14:02:48.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:48.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336062+0000", 2023-07-02T14:02:48.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7376999999999997e-05 2023-07-02T14:02:48.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:48.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336109+0000", 2023-07-02T14:02:48.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1814000000000003e-05 2023-07-02T14:02:48.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:48.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336151+0000", 2023-07-02T14:02:48.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036975239999999999 2023-07-02T14:02:48.137 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.137 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.137 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.138 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:327 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 589824~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:48.138 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:02:48.138 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7774671170000005, 2023-07-02T14:02:48.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037234253000000002, 2023-07-02T14:02:48.138 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:48.138 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:48.139 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:48.139 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:48.139 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:48.139 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 327 2023-07-02T14:02:48.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.139 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:48.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:48.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:02:48.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:48.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:48.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:02:48.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:48.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:48.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299274+0000", 2023-07-02T14:02:48.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2213999999999997e-05 2023-07-02T14:02:48.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:48.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299346+0000", 2023-07-02T14:02:48.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1816e-05 2023-07-02T14:02:48.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:48.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299358+0000", 2023-07-02T14:02:48.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2626000000000001e-05 2023-07-02T14:02:48.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:48.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299371+0000", 2023-07-02T14:02:48.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026916853000000001 2023-07-02T14:02:48.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326288+0000", 2023-07-02T14:02:48.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6909000000000004e-05 2023-07-02T14:02:48.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:48.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326375+0000", 2023-07-02T14:02:48.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015430399999999999 2023-07-02T14:02:48.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:48.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326529+0000", 2023-07-02T14:02:48.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096652100000000005 2023-07-02T14:02:48.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:48.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336194+0000", 2023-07-02T14:02:48.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115299 2023-07-02T14:02:48.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:48.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336309+0000", 2023-07-02T14:02:48.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015981200000000001 2023-07-02T14:02:48.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:48.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336469+0000", 2023-07-02T14:02:48.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9867999999999998e-05 2023-07-02T14:02:48.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:48.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336509+0000", 2023-07-02T14:02:48.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037234253000000002 2023-07-02T14:02:48.154 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.154 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.154 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.155 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:328 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:48.156 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:02:48.156 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7773635280000004, 2023-07-02T14:02:48.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037216610999999997, 2023-07-02T14:02:48.156 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:48.157 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:48.157 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:48.157 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:48.157 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:48.157 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 328 2023-07-02T14:02:48.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.158 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:48.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:48.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:02:48.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:48.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:48.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:02:48.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:48.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:48.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:02:48.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1412e-05 2023-07-02T14:02:48.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:48.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299449+0000", 2023-07-02T14:02:48.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0219000000000001e-05 2023-07-02T14:02:48.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:48.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299469+0000", 2023-07-02T14:02:48.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9480000000000003e-06 2023-07-02T14:02:48.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:48.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299479+0000", 2023-07-02T14:02:48.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027444344999999998 2023-07-02T14:02:48.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326924+0000", 2023-07-02T14:02:48.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5165000000000001e-05 2023-07-02T14:02:48.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:48.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327019+0000", 2023-07-02T14:02:48.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019113499999999999 2023-07-02T14:02:48.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:48.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327210+0000", 2023-07-02T14:02:48.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090197449999999992 2023-07-02T14:02:48.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:48.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336230+0000", 2023-07-02T14:02:48.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028764399999999999 2023-07-02T14:02:48.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:48.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336517+0000", 2023-07-02T14:02:48.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1981999999999999e-05 2023-07-02T14:02:48.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:48.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336559+0000", 2023-07-02T14:02:48.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:02:48.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:48.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336595+0000", 2023-07-02T14:02:48.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037216610999999997 2023-07-02T14:02:48.167 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.168 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.168 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.168 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:329 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:48.168 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:02:48.168 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7772536110000008, 2023-07-02T14:02:48.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037191142000000003, 2023-07-02T14:02:48.169 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:48.169 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:48.169 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:48.169 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:48.169 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:48.169 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 329 2023-07-02T14:02:48.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.170 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:48.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:48.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:02:48.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:48.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:48.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:02:48.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:48.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:48.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299487+0000", 2023-07-02T14:02:48.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7871000000000001e-05 2023-07-02T14:02:48.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:48.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299555+0000", 2023-07-02T14:02:48.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0446e-05 2023-07-02T14:02:48.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:48.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299566+0000", 2023-07-02T14:02:48.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7939999999999999e-06 2023-07-02T14:02:48.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:48.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299572+0000", 2023-07-02T14:02:48.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028034019 2023-07-02T14:02:48.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327606+0000", 2023-07-02T14:02:48.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1967999999999999e-05 2023-07-02T14:02:48.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:48.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327678+0000", 2023-07-02T14:02:48.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022216900000000001 2023-07-02T14:02:48.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:48.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327900+0000", 2023-07-02T14:02:48.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084016899999999999 2023-07-02T14:02:48.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:48.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336301+0000", 2023-07-02T14:02:48.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00030122800000000001 2023-07-02T14:02:48.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:48.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336603+0000", 2023-07-02T14:02:48.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0380000000000003e-05 2023-07-02T14:02:48.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:48.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336643+0000", 2023-07-02T14:02:48.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6455999999999998e-05 2023-07-02T14:02:48.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:48.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336679+0000", 2023-07-02T14:02:48.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037191142000000003 2023-07-02T14:02:48.178 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.178 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.178 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.179 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:330 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:48.179 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:02:48.179 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7771628410000009, 2023-07-02T14:02:48.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037849305, 2023-07-02T14:02:48.179 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:48.179 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:48.179 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:48.179 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:48.180 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:48.180 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 330 2023-07-02T14:02:48.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.180 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:48.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:48.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:02:48.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:48.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:48.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:02:48.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:48.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:48.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299578+0000", 2023-07-02T14:02:48.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4708000000000005e-05 2023-07-02T14:02:48.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:48.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299643+0000", 2023-07-02T14:02:48.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7309999999999999e-06 2023-07-02T14:02:48.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:48.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299652+0000", 2023-07-02T14:02:48.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7989999999999999e-06 2023-07-02T14:02:48.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:48.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299658+0000", 2023-07-02T14:02:48.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028981664000000001 2023-07-02T14:02:48.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328639+0000", 2023-07-02T14:02:48.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0939000000000006e-05 2023-07-02T14:02:48.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:48.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328710+0000", 2023-07-02T14:02:48.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000142624 2023-07-02T14:02:48.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:48.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328853+0000", 2023-07-02T14:02:48.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078374640000000006 2023-07-02T14:02:48.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:48.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336690+0000", 2023-07-02T14:02:48.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00068453000000000003 2023-07-02T14:02:48.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:48.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337375+0000", 2023-07-02T14:02:48.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0667999999999998e-05 2023-07-02T14:02:48.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:48.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337406+0000", 2023-07-02T14:02:48.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2858999999999999e-05 2023-07-02T14:02:48.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:48.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337428+0000", 2023-07-02T14:02:48.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037849305 2023-07-02T14:02:48.188 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.188 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.189 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.189 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:331 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:48.189 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:02:48.189 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7770789790000006, 2023-07-02T14:02:48.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037870332, 2023-07-02T14:02:48.190 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:48.190 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:48.190 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:48.190 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:48.190 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:48.190 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 331 2023-07-02T14:02:48.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.190 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:48.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:48.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:02:48.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:48.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:48.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:02:48.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:48.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:48.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299662+0000", 2023-07-02T14:02:48.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2252e-05 2023-07-02T14:02:48.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:48.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299725+0000", 2023-07-02T14:02:48.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8440000000000004e-06 2023-07-02T14:02:48.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:48.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299733+0000", 2023-07-02T14:02:48.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6759999999999998e-06 2023-07-02T14:02:48.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:48.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299738+0000", 2023-07-02T14:02:48.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029459387 2023-07-02T14:02:48.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329197+0000", 2023-07-02T14:02:48.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0087999999999997e-05 2023-07-02T14:02:48.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:48.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329268+0000", 2023-07-02T14:02:48.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014405700000000001 2023-07-02T14:02:48.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:48.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329412+0000", 2023-07-02T14:02:48.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072842100000000002 2023-07-02T14:02:48.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:48.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336696+0000", 2023-07-02T14:02:48.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00076275300000000002 2023-07-02T14:02:48.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:48.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337459+0000", 2023-07-02T14:02:48.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5988000000000002e-05 2023-07-02T14:02:48.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:48.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337495+0000", 2023-07-02T14:02:48.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8686000000000002e-05 2023-07-02T14:02:48.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:48.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337533+0000", 2023-07-02T14:02:48.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037870332 2023-07-02T14:02:48.199 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.199 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.199 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.199 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:332 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:48.200 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:02:48.200 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7769990549999992, 2023-07-02T14:02:48.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037892506999999999, 2023-07-02T14:02:48.200 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:48.200 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:48.200 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:48.200 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:48.200 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:48.201 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 332 2023-07-02T14:02:48.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.201 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:48.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:48.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:02:48.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:48.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:48.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:02:48.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:48.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:48.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299742+0000", 2023-07-02T14:02:48.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1366999999999997e-05 2023-07-02T14:02:48.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:48.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299804+0000", 2023-07-02T14:02:48.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9319999999999996e-06 2023-07-02T14:02:48.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:48.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299812+0000", 2023-07-02T14:02:48.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4359999999999999e-06 2023-07-02T14:02:48.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:48.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299818+0000", 2023-07-02T14:02:48.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029946144000000001 2023-07-02T14:02:48.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329764+0000", 2023-07-02T14:02:48.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000164828 2023-07-02T14:02:48.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:48.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329929+0000", 2023-07-02T14:02:48.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016139400000000001 2023-07-02T14:02:48.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:48.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330090+0000", 2023-07-02T14:02:48.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066134009999999997 2023-07-02T14:02:48.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:48.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336704+0000", 2023-07-02T14:02:48.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00086955400000000003 2023-07-02T14:02:48.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:48.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337573+0000", 2023-07-02T14:02:48.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8447000000000001e-05 2023-07-02T14:02:48.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:48.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337602+0000", 2023-07-02T14:02:48.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.362e-05 2023-07-02T14:02:48.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:48.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337635+0000", 2023-07-02T14:02:48.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037892506999999999 2023-07-02T14:02:48.209 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.209 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.209 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.210 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:333 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:48.210 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:02:48.210 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7769189740000009, 2023-07-02T14:02:48.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037932598999999997, 2023-07-02T14:02:48.210 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:48.210 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:48.211 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:48.211 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:48.211 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:48.211 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 333 2023-07-02T14:02:48.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.211 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:48.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:48.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:02:48.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:48.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:48.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:02:48.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:48.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:48.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299822+0000", 2023-07-02T14:02:48.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0831000000000001e-05 2023-07-02T14:02:48.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:48.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299883+0000", 2023-07-02T14:02:48.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.322e-06 2023-07-02T14:02:48.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:48.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299888+0000", 2023-07-02T14:02:48.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3479999999999997e-06 2023-07-02T14:02:48.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:48.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299893+0000", 2023-07-02T14:02:48.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030831905999999999 2023-07-02T14:02:48.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330725+0000", 2023-07-02T14:02:48.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3560999999999996e-05 2023-07-02T14:02:48.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:48.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330798+0000", 2023-07-02T14:02:48.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015165999999999999 2023-07-02T14:02:48.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:48.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330950+0000", 2023-07-02T14:02:48.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057590319999999999 2023-07-02T14:02:48.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:48.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336709+0000", 2023-07-02T14:02:48.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00096987899999999999 2023-07-02T14:02:48.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:48.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337679+0000", 2023-07-02T14:02:48.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9699000000000001e-05 2023-07-02T14:02:48.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:48.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337719+0000", 2023-07-02T14:02:48.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6961000000000001e-05 2023-07-02T14:02:48.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:48.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337756+0000", 2023-07-02T14:02:48.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037932598999999997 2023-07-02T14:02:48.220 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.220 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.220 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.220 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:372 2.0 2:169af44a:::rbd_data.125de1449605.0000000000000106:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:48.220 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:02:48.220 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7733416490000007, 2023-07-02T14:02:48.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036066963, 2023-07-02T14:02:48.221 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:48.221 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:48.221 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:48.221 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:48.221 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:48.221 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 372 2023-07-02T14:02:48.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.222 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:48.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:48.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:02:48.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:48.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:48.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:02:48.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:48.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:48.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:02:48.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1002000000000003e-05 2023-07-02T14:02:48.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:48.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303461+0000", 2023-07-02T14:02:48.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9306e-05 2023-07-02T14:02:48.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:48.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303490+0000", 2023-07-02T14:02:48.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5026e-05 2023-07-02T14:02:48.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:48.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303505+0000", 2023-07-02T14:02:48.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033623129000000002 2023-07-02T14:02:48.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337128+0000", 2023-07-02T14:02:48.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6968000000000002e-05 2023-07-02T14:02:48.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:48.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337175+0000", 2023-07-02T14:02:48.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000101127 2023-07-02T14:02:48.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:48.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337276+0000", 2023-07-02T14:02:48.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017866920000000001 2023-07-02T14:02:48.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:48.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339063+0000", 2023-07-02T14:02:48.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00031459800000000003 2023-07-02T14:02:48.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:48.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339377+0000", 2023-07-02T14:02:48.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7145999999999999e-05 2023-07-02T14:02:48.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:48.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339425+0000", 2023-07-02T14:02:48.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2632000000000001e-05 2023-07-02T14:02:48.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.230 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:48.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339467+0000", 2023-07-02T14:02:48.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036066963 2023-07-02T14:02:48.230 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.230 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.230 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.231 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:416 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:48.231 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:02:48.231 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7586255949999998, 2023-07-02T14:02:48.231 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039242025, 2023-07-02T14:02:48.231 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:48.231 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:48.231 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:48.231 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:48.232 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:48.232 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:02:48.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.232 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:48.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:48.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:02:48.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:48.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:48.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:02:48.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:48.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:48.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:02:48.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.692e-06 2023-07-02T14:02:48.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:48.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318119+0000", 2023-07-02T14:02:48.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9790000000000005e-06 2023-07-02T14:02:48.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:48.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318129+0000", 2023-07-02T14:02:48.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3640000000000001e-06 2023-07-02T14:02:48.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:48.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318137+0000", 2023-07-02T14:02:48.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022607808 2023-07-02T14:02:48.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340745+0000", 2023-07-02T14:02:48.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0130999999999996e-05 2023-07-02T14:02:48.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340825+0000", 2023-07-02T14:02:48.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039581080000000001 2023-07-02T14:02:48.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344783+0000", 2023-07-02T14:02:48.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000006e-05 2023-07-02T14:02:48.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344854+0000", 2023-07-02T14:02:48.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052105550000000004 2023-07-02T14:02:48.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350065+0000", 2023-07-02T14:02:48.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0321999999999997e-05 2023-07-02T14:02:48.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350115+0000", 2023-07-02T14:02:48.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012378409999999999 2023-07-02T14:02:48.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.240 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351353+0000", 2023-07-02T14:02:48.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5942000000000001e-05 2023-07-02T14:02:48.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:48.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351409+0000", 2023-07-02T14:02:48.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035677700000000002 2023-07-02T14:02:48.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:48.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351766+0000", 2023-07-02T14:02:48.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000926719 2023-07-02T14:02:48.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:48.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352693+0000", 2023-07-02T14:02:48.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045652080000000003 2023-07-02T14:02:48.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:48.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357258+0000", 2023-07-02T14:02:48.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4717e-05 2023-07-02T14:02:48.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:48.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357293+0000", 2023-07-02T14:02:48.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5807000000000002e-05 2023-07-02T14:02:48.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:48.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357358+0000", 2023-07-02T14:02:48.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039242025 2023-07-02T14:02:48.244 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.244 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.244 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.245 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:417 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:48.245 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:02:48.245 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7586001580000001, 2023-07-02T14:02:48.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042407105, 2023-07-02T14:02:48.245 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:48.246 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:48.246 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:48.246 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:48.246 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:48.246 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 417 2023-07-02T14:02:48.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.246 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:48.246 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:48.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:02:48.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:48.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:48.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:02:48.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:48.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:48.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318141+0000", 2023-07-02T14:02:48.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7120000000000002e-06 2023-07-02T14:02:48.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:48.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318145+0000", 2023-07-02T14:02:48.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8869999999999997e-06 2023-07-02T14:02:48.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:48.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318151+0000", 2023-07-02T14:02:48.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9820000000000002e-06 2023-07-02T14:02:48.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:48.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318155+0000", 2023-07-02T14:02:48.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699133 2023-07-02T14:02:48.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340854+0000", 2023-07-02T14:02:48.251 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6849999999999998e-05 2023-07-02T14:02:48.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.251 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.251 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340931+0000", 2023-07-02T14:02:48.251 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039672140000000002 2023-07-02T14:02:48.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.252 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344898+0000", 2023-07-02T14:02:48.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5341999999999998e-05 2023-07-02T14:02:48.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.252 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344973+0000", 2023-07-02T14:02:48.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051819400000000003 2023-07-02T14:02:48.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.253 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350155+0000", 2023-07-02T14:02:48.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3261999999999999e-05 2023-07-02T14:02:48.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350209+0000", 2023-07-02T14:02:48.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00194097 2023-07-02T14:02:48.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352150+0000", 2023-07-02T14:02:48.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6851e-05 2023-07-02T14:02:48.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352226+0000", 2023-07-02T14:02:48.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051831109999999998 2023-07-02T14:02:48.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357409+0000", 2023-07-02T14:02:48.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5325e-05 2023-07-02T14:02:48.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:48.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357495+0000", 2023-07-02T14:02:48.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042107899999999998 2023-07-02T14:02:48.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:48.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357916+0000", 2023-07-02T14:02:48.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0011672449999999999 2023-07-02T14:02:48.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:48.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.359083+0000", 2023-07-02T14:02:48.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013534930000000001 2023-07-02T14:02:48.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:48.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360437+0000", 2023-07-02T14:02:48.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2159000000000002e-05 2023-07-02T14:02:48.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:48.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360469+0000", 2023-07-02T14:02:48.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0092000000000005e-05 2023-07-02T14:02:48.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:48.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360549+0000", 2023-07-02T14:02:48.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042407105 2023-07-02T14:02:48.260 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.260 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.260 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.260 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:418 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:48.260 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:02:48.261 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7585809169999997, 2023-07-02T14:02:48.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.045613663999999998, 2023-07-02T14:02:48.261 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:48.261 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:48.261 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:48.261 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:48.261 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:48.261 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 418 2023-07-02T14:02:48.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.262 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:48.262 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.262 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:48.262 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:02:48.262 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:48.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:48.263 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:02:48.263 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:48.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:48.263 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318160+0000", 2023-07-02T14:02:48.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6490000000000001e-06 2023-07-02T14:02:48.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.264 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:48.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318164+0000", 2023-07-02T14:02:48.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:02:48.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:48.265 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318170+0000", 2023-07-02T14:02:48.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5279999999999999e-06 2023-07-02T14:02:48.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:48.265 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318173+0000", 2023-07-02T14:02:48.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022788856 2023-07-02T14:02:48.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340962+0000", 2023-07-02T14:02:48.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7730000000000003e-05 2023-07-02T14:02:48.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341040+0000", 2023-07-02T14:02:48.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039795899999999999 2023-07-02T14:02:48.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345020+0000", 2023-07-02T14:02:48.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4962000000000006e-05 2023-07-02T14:02:48.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.268 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345085+0000", 2023-07-02T14:02:48.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051955430000000004 2023-07-02T14:02:48.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350280+0000", 2023-07-02T14:02:48.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8913999999999995e-05 2023-07-02T14:02:48.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350379+0000", 2023-07-02T14:02:48.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001887344 2023-07-02T14:02:48.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352266+0000", 2023-07-02T14:02:48.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7865000000000003e-05 2023-07-02T14:02:48.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352314+0000", 2023-07-02T14:02:48.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060621190000000004 2023-07-02T14:02:48.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.271 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358376+0000", 2023-07-02T14:02:48.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2448999999999997e-05 2023-07-02T14:02:48.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358449+0000", 2023-07-02T14:02:48.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0021539929999999999 2023-07-02T14:02:48.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360603+0000", 2023-07-02T14:02:48.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7859000000000002e-05 2023-07-02T14:02:48.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.273 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:48.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360671+0000", 2023-07-02T14:02:48.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026930699999999999 2023-07-02T14:02:48.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:48.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360940+0000", 2023-07-02T14:02:48.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00081179699999999995 2023-07-02T14:02:48.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:48.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361752+0000", 2023-07-02T14:02:48.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0019350490000000001 2023-07-02T14:02:48.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.275 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:48.275 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363687+0000", 2023-07-02T14:02:48.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4462000000000003e-05 2023-07-02T14:02:48.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.275 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:48.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363721+0000", 2023-07-02T14:02:48.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3408000000000001e-05 2023-07-02T14:02:48.276 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:48.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363775+0000", 2023-07-02T14:02:48.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.045613663999999998 2023-07-02T14:02:48.277 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.277 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.277 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.277 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:419 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:48.277 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:02:48.277 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7585645900000006, 2023-07-02T14:02:48.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.047296703000000002, 2023-07-02T14:02:48.278 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:48.278 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:48.278 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:48.278 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:48.278 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:48.278 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 419 2023-07-02T14:02:48.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.279 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:48.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:48.279 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:02:48.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:48.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:48.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:02:48.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:48.280 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:48.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:02:48.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.433e-06 2023-07-02T14:02:48.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:48.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318180+0000", 2023-07-02T14:02:48.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9950000000000002e-06 2023-07-02T14:02:48.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:48.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318186+0000", 2023-07-02T14:02:48.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7290000000000002e-06 2023-07-02T14:02:48.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.282 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:48.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318190+0000", 2023-07-02T14:02:48.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022877989000000001 2023-07-02T14:02:48.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.283 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341068+0000", 2023-07-02T14:02:48.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5803999999999994e-05 2023-07-02T14:02:48.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.283 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341154+0000", 2023-07-02T14:02:48.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039633000000000003 2023-07-02T14:02:48.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345117+0000", 2023-07-02T14:02:48.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7188999999999996e-05 2023-07-02T14:02:48.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345194+0000", 2023-07-02T14:02:48.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052306059999999996 2023-07-02T14:02:48.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350425+0000", 2023-07-02T14:02:48.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8093999999999997e-05 2023-07-02T14:02:48.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350473+0000", 2023-07-02T14:02:48.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001865158 2023-07-02T14:02:48.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352338+0000", 2023-07-02T14:02:48.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5479000000000003e-05 2023-07-02T14:02:48.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352384+0000", 2023-07-02T14:02:48.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061153290000000001 2023-07-02T14:02:48.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358499+0000", 2023-07-02T14:02:48.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5156999999999999e-05 2023-07-02T14:02:48.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358574+0000", 2023-07-02T14:02:48.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027847459999999998 2023-07-02T14:02:48.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361359+0000", 2023-07-02T14:02:48.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2633999999999999e-05 2023-07-02T14:02:48.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361402+0000", 2023-07-02T14:02:48.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024171510000000002 2023-07-02T14:02:48.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363819+0000", 2023-07-02T14:02:48.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6988999999999999e-05 2023-07-02T14:02:48.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:48.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363876+0000", 2023-07-02T14:02:48.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035321300000000001 2023-07-02T14:02:48.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.364229+0000", 2023-07-02T14:02:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00094674799999999997 2023-07-02T14:02:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365176+0000", 2023-07-02T14:02:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019724200000000001 2023-07-02T14:02:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365373+0000", 2023-07-02T14:02:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2116e-05 2023-07-02T14:02:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365425+0000", 2023-07-02T14:02:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8996999999999999e-05 2023-07-02T14:02:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365474+0000", 2023-07-02T14:02:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.047296703000000002 2023-07-02T14:02:48.295 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.295 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.295 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.295 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:472 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:48.295 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:02:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.6991083580000002, 2023-07-02T14:02:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036204721000000002, 2023-07-02T14:02:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 472 2023-07-02T14:02:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:02:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:02:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:02:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377633+0000", 2023-07-02T14:02:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.815e-06 2023-07-02T14:02:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377635+0000", 2023-07-02T14:02:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9459999999999998e-06 2023-07-02T14:02:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377638+0000", 2023-07-02T14:02:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.673e-06 2023-07-02T14:02:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377643+0000", 2023-07-02T14:02:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043004239999999997 2023-07-02T14:02:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.381943+0000", 2023-07-02T14:02:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4321999999999999e-05 2023-07-02T14:02:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.381998+0000", 2023-07-02T14:02:48.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003008377 2023-07-02T14:02:48.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385006+0000", 2023-07-02T14:02:48.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8751999999999998e-05 2023-07-02T14:02:48.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385045+0000", 2023-07-02T14:02:48.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086260880000000005 2023-07-02T14:02:48.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393671+0000", 2023-07-02T14:02:48.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6365999999999997e-05 2023-07-02T14:02:48.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393707+0000", 2023-07-02T14:02:48.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028560289999999999 2023-07-02T14:02:48.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396563+0000", 2023-07-02T14:02:48.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6195e-05 2023-07-02T14:02:48.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396619+0000", 2023-07-02T14:02:48.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023118269999999998 2023-07-02T14:02:48.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.398931+0000", 2023-07-02T14:02:48.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3885999999999999e-05 2023-07-02T14:02:48.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.307 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.398985+0000", 2023-07-02T14:02:48.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043756890000000003 2023-07-02T14:02:48.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.307 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403361+0000", 2023-07-02T14:02:48.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0043000000000001e-05 2023-07-02T14:02:48.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403421+0000", 2023-07-02T14:02:48.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001590972 2023-07-02T14:02:48.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405012+0000", 2023-07-02T14:02:48.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0067000000000005e-05 2023-07-02T14:02:48.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405082+0000", 2023-07-02T14:02:48.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050958610000000001 2023-07-02T14:02:48.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410178+0000", 2023-07-02T14:02:48.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7070000000000004e-05 2023-07-02T14:02:48.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410275+0000", 2023-07-02T14:02:48.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060121100000000004 2023-07-02T14:02:48.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410876+0000", 2023-07-02T14:02:48.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3177999999999999e-05 2023-07-02T14:02:48.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:48.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410919+0000", 2023-07-02T14:02:48.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000271394 2023-07-02T14:02:48.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:48.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411191+0000", 2023-07-02T14:02:48.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001009598 2023-07-02T14:02:48.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:48.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.412200+0000", 2023-07-02T14:02:48.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014789790000000001 2023-07-02T14:02:48.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:48.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413679+0000", 2023-07-02T14:02:48.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2624000000000002e-05 2023-07-02T14:02:48.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:48.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413742+0000", 2023-07-02T14:02:48.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6564999999999994e-05 2023-07-02T14:02:48.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:48.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413838+0000", 2023-07-02T14:02:48.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036204721000000002 2023-07-02T14:02:48.315 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.315 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.316 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.316 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:473 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:48.316 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:02:48.316 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.6990969469999992, 2023-07-02T14:02:48.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037915904, 2023-07-02T14:02:48.316 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:48.317 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:48.317 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:48.317 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:48.317 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:48.317 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 473 2023-07-02T14:02:48.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.317 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:48.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:48.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:02:48.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:48.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:48.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:02:48.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:02:48.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:48.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:02:48.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.119e-06 2023-07-02T14:02:48.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:48.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377647+0000", 2023-07-02T14:02:48.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.875e-06 2023-07-02T14:02:48.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:48.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377650+0000", 2023-07-02T14:02:48.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1949999999999999e-06 2023-07-02T14:02:48.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:48.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377653+0000", 2023-07-02T14:02:48.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004375444 2023-07-02T14:02:48.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382028+0000", 2023-07-02T14:02:48.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1063999999999997e-05 2023-07-02T14:02:48.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.322 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382089+0000", 2023-07-02T14:02:48.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030043869999999999 2023-07-02T14:02:48.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385094+0000", 2023-07-02T14:02:48.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8167999999999997e-05 2023-07-02T14:02:48.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.324 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385142+0000", 2023-07-02T14:02:48.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008603421 2023-07-02T14:02:48.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.324 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393745+0000", 2023-07-02T14:02:48.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3763000000000005e-05 2023-07-02T14:02:48.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393819+0000", 2023-07-02T14:02:48.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028274720000000001 2023-07-02T14:02:48.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396647+0000", 2023-07-02T14:02:48.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7929999999999997e-05 2023-07-02T14:02:48.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396695+0000", 2023-07-02T14:02:48.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023426459999999999 2023-07-02T14:02:48.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399037+0000", 2023-07-02T14:02:48.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5846000000000004e-05 2023-07-02T14:02:48.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399123+0000", 2023-07-02T14:02:48.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043448360000000004 2023-07-02T14:02:48.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403468+0000", 2023-07-02T14:02:48.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6699000000000001e-05 2023-07-02T14:02:48.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403515+0000", 2023-07-02T14:02:48.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001622943 2023-07-02T14:02:48.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405138+0000", 2023-07-02T14:02:48.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6382000000000002e-05 2023-07-02T14:02:48.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405204+0000", 2023-07-02T14:02:48.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051089910000000002 2023-07-02T14:02:48.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410313+0000", 2023-07-02T14:02:48.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8106000000000003e-05 2023-07-02T14:02:48.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410361+0000", 2023-07-02T14:02:48.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012200399999999999 2023-07-02T14:02:48.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411581+0000", 2023-07-02T14:02:48.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5067000000000005e-05 2023-07-02T14:02:48.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411656+0000", 2023-07-02T14:02:48.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022353249999999998 2023-07-02T14:02:48.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413891+0000", 2023-07-02T14:02:48.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9356000000000003e-05 2023-07-02T14:02:48.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:48.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413981+0000", 2023-07-02T14:02:48.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00043818299999999998 2023-07-02T14:02:48.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:48.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.414419+0000", 2023-07-02T14:02:48.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00081559199999999997 2023-07-02T14:02:48.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:48.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415235+0000", 2023-07-02T14:02:48.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026434499999999998 2023-07-02T14:02:48.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:48.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415499+0000", 2023-07-02T14:02:48.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9779999999999999e-05 2023-07-02T14:02:48.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:48.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415529+0000", 2023-07-02T14:02:48.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2159999999999997e-05 2023-07-02T14:02:48.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:48.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415561+0000", 2023-07-02T14:02:48.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037915904 2023-07-02T14:02:48.338 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.338 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.338 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.339 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:475 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:48.339 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:02:48.339 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.699086908, 2023-07-02T14:02:48.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039286962000000002, 2023-07-02T14:02:48.339 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:48.339 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:48.339 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:48.340 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:48.340 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:48.340 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 475 2023-07-02T14:02:48.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.340 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:48.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:48.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:02:48.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:48.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:48.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:02:48.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:48.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:48.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:02:48.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4779999999999997e-06 2023-07-02T14:02:48.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:48.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377664+0000", 2023-07-02T14:02:48.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.268e-06 2023-07-02T14:02:48.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:48.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377671+0000", 2023-07-02T14:02:48.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.861e-06 2023-07-02T14:02:48.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:48.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377674+0000", 2023-07-02T14:02:48.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044395789999999999 2023-07-02T14:02:48.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382114+0000", 2023-07-02T14:02:48.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8096999999999997e-05 2023-07-02T14:02:48.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382162+0000", 2023-07-02T14:02:48.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029949059999999999 2023-07-02T14:02:48.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385157+0000", 2023-07-02T14:02:48.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1158000000000001e-05 2023-07-02T14:02:48.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385198+0000", 2023-07-02T14:02:48.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086477229999999995 2023-07-02T14:02:48.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393846+0000", 2023-07-02T14:02:48.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5990999999999999e-05 2023-07-02T14:02:48.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393912+0000", 2023-07-02T14:02:48.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027979599999999999 2023-07-02T14:02:48.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396710+0000", 2023-07-02T14:02:48.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5911000000000002e-05 2023-07-02T14:02:48.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396746+0000", 2023-07-02T14:02:48.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024039759999999999 2023-07-02T14:02:48.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399150+0000", 2023-07-02T14:02:48.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7834999999999999e-05 2023-07-02T14:02:48.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399188+0000", 2023-07-02T14:02:48.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004364054 2023-07-02T14:02:48.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403552+0000", 2023-07-02T14:02:48.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6274999999999999e-05 2023-07-02T14:02:48.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403608+0000", 2023-07-02T14:02:48.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0016261540000000001 2023-07-02T14:02:48.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405234+0000", 2023-07-02T14:02:48.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6945000000000004e-05 2023-07-02T14:02:48.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405301+0000", 2023-07-02T14:02:48.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050871329999999998 2023-07-02T14:02:48.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410388+0000", 2023-07-02T14:02:48.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0210999999999995e-05 2023-07-02T14:02:48.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410468+0000", 2023-07-02T14:02:48.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001227454 2023-07-02T14:02:48.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411696+0000", 2023-07-02T14:02:48.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8188000000000005e-05 2023-07-02T14:02:48.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411764+0000", 2023-07-02T14:02:48.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031779930000000001 2023-07-02T14:02:48.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.414942+0000", 2023-07-02T14:02:48.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7236000000000004e-05 2023-07-02T14:02:48.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:48.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415019+0000", 2023-07-02T14:02:48.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000573045 2023-07-02T14:02:48.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:48.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415592+0000", 2023-07-02T14:02:48.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7580000000000002e-05 2023-07-02T14:02:48.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:48.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415640+0000", 2023-07-02T14:02:48.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024783200000000002 2023-07-02T14:02:48.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:48.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415888+0000", 2023-07-02T14:02:48.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00080440999999999998 2023-07-02T14:02:48.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:48.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416692+0000", 2023-07-02T14:02:48.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000195787 2023-07-02T14:02:48.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:48.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416888+0000", 2023-07-02T14:02:48.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0787999999999997e-05 2023-07-02T14:02:48.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:48.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416919+0000", 2023-07-02T14:02:48.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.338e-05 2023-07-02T14:02:48.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:48.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:48.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:48.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416942+0000", 2023-07-02T14:02:48.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039286962000000002 2023-07-02T14:02:48.362 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.362 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.362 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.362 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:48.362 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:48.362 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:48.364 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:02:48.365 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:48.363+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:48.386 INFO:tasks.workunit.client.0.smithi163.stderr:166250K .......... .......... .......... .......... .......... 98% 32.5M 0s 2023-07-02T14:02:48.386 INFO:tasks.workunit.client.0.smithi163.stderr:166300K .......... .......... .......... .......... .......... 98% 28.5M 0s 2023-07-02T14:02:48.386 INFO:tasks.workunit.client.0.smithi163.stderr:166350K .......... .......... .......... .......... .......... 98% 23.8M 0s 2023-07-02T14:02:48.387 INFO:tasks.workunit.client.0.smithi163.stderr:166400K .......... .......... .......... .......... .......... 98% 47.7M 0s 2023-07-02T14:02:48.387 INFO:tasks.workunit.client.0.smithi163.stderr:166450K .......... .......... .......... .......... .......... 98% 19.3M 0s 2023-07-02T14:02:48.387 INFO:tasks.workunit.client.0.smithi163.stderr:166500K .......... .......... .......... .......... .......... 98% 24.4M 0s 2023-07-02T14:02:48.387 INFO:tasks.workunit.client.0.smithi163.stderr:166550K .......... .......... .......... .......... .......... 98% 40.3M 0s 2023-07-02T14:02:48.387 INFO:tasks.workunit.client.0.smithi163.stderr:166600K .......... .......... .......... .......... .......... 98% 20.3M 0s 2023-07-02T14:02:48.387 INFO:tasks.workunit.client.0.smithi163.stderr:166650K .......... .......... .......... .......... .......... 98% 57.0M 0s 2023-07-02T14:02:48.387 INFO:tasks.workunit.client.0.smithi163.stderr:166700K .......... .......... .......... .......... .......... 98% 20.2M 0s 2023-07-02T14:02:48.388 INFO:tasks.workunit.client.0.smithi163.stderr:166750K .......... .......... .......... .......... .......... 98% 21.8M 0s 2023-07-02T14:02:48.388 INFO:tasks.workunit.client.0.smithi163.stderr:166800K .......... .......... .......... .......... .......... 98% 36.4M 0s 2023-07-02T14:02:48.388 INFO:tasks.workunit.client.0.smithi163.stderr:166850K .......... .......... .......... .......... .......... 99% 21.1M 0s 2023-07-02T14:02:48.388 INFO:tasks.workunit.client.0.smithi163.stderr:166900K .......... .......... .......... .......... .......... 99% 45.1M 0s 2023-07-02T14:02:48.388 INFO:tasks.workunit.client.0.smithi163.stderr:166950K .......... .......... .......... .......... .......... 99% 25.1M 0s 2023-07-02T14:02:48.388 INFO:tasks.workunit.client.0.smithi163.stderr:167000K .......... .......... .......... .......... .......... 99% 18.2M 0s 2023-07-02T14:02:48.388 INFO:tasks.workunit.client.0.smithi163.stderr:167050K .......... .......... .......... .......... .......... 99% 34.8M 0s 2023-07-02T14:02:48.388 INFO:tasks.workunit.client.0.smithi163.stderr:167100K .......... .......... .......... .......... .......... 99% 43.1M 0s 2023-07-02T14:02:48.389 INFO:tasks.workunit.client.0.smithi163.stderr:167150K .......... .......... .......... .......... .......... 99% 14.3M 0s 2023-07-02T14:02:48.389 INFO:tasks.workunit.client.0.smithi163.stderr:167200K .......... .......... .......... .......... .......... 99% 39.7M 0s 2023-07-02T14:02:48.389 INFO:tasks.workunit.client.0.smithi163.stderr:167250K .......... .......... .......... .......... .......... 99% 23.6M 0s 2023-07-02T14:02:48.389 INFO:tasks.workunit.client.0.smithi163.stderr:167300K .......... .......... .......... .......... .......... 99% 41.9M 0s 2023-07-02T14:02:48.389 INFO:tasks.workunit.client.0.smithi163.stderr:167350K .......... .......... .......... .......... .......... 99% 22.2M 0s 2023-07-02T14:02:48.389 INFO:tasks.workunit.client.0.smithi163.stderr:167400K .......... .......... .......... .......... .......... 99% 44.6M 0s 2023-07-02T14:02:48.389 INFO:tasks.workunit.client.0.smithi163.stderr:167450K .......... .......... .......... .......... .......... 99% 22.2M 0s 2023-07-02T14:02:48.390 INFO:tasks.workunit.client.0.smithi163.stderr:167500K .......... .......... .......... .......... .......... 99% 35.5M 0s 2023-07-02T14:02:48.390 INFO:tasks.workunit.client.0.smithi163.stderr:167550K .......... .......... .......... .......... .......... 99% 17.3M 0s 2023-07-02T14:02:48.390 INFO:tasks.workunit.client.0.smithi163.stderr:167600K .......... .......... .......... .......... .......... 99% 21.5M 0s 2023-07-02T14:02:48.390 INFO:tasks.workunit.client.0.smithi163.stderr:167650K .......... .......... .......... .......... .......... 99% 42.7M 0s 2023-07-02T14:02:48.390 INFO:tasks.workunit.client.0.smithi163.stderr:167700K .......... .......... .......... .......... .......... 99% 20.0M 0s 2023-07-02T14:02:48.390 INFO:tasks.workunit.client.0.smithi163.stderr:167750K .......... .......... .......... .......... .......... 99% 21.2M 0s 2023-07-02T14:02:48.390 INFO:tasks.workunit.client.0.smithi163.stderr:167800K .......... .......... .......... .......... .......... 99% 39.0M 0s 2023-07-02T14:02:48.390 INFO:tasks.workunit.client.0.smithi163.stderr:167850K .......... .......... .......... .......... .......... 99% 21.5M 0s 2023-07-02T14:02:48.391 INFO:tasks.workunit.client.0.smithi163.stderr:167900K .......... .......... .......... .......... .......... 99% 44.6M 0s 2023-07-02T14:02:48.391 INFO:tasks.workunit.client.0.smithi163.stderr:167950K .......... .......... .......... .......... .......... 99% 21.3M 0s 2023-07-02T14:02:48.391 INFO:tasks.workunit.client.0.smithi163.stderr:168000K .......... .......... .......... .......... .......... 99% 24.4M 0s 2023-07-02T14:02:48.391 INFO:tasks.workunit.client.0.smithi163.stderr:168050K .......... .......... .......... .......... .......... 99% 40.4M 0s 2023-07-02T14:02:48.391 INFO:tasks.workunit.client.0.smithi163.stderr:168100K .......... .......... .......... .......... .......... 99% 23.5M 0s 2023-07-02T14:02:48.391 INFO:tasks.workunit.client.0.smithi163.stderr:168150K .......... .......... .......... .......... .......... 99% 34.4M 0s 2023-07-02T14:02:48.391 INFO:tasks.workunit.client.0.smithi163.stderr:168200K .......... .......... .......... .......... .......... 99% 23.7M 0s 2023-07-02T14:02:48.391 INFO:tasks.workunit.client.0.smithi163.stderr:168250K .......... .......... .......... .......... .......... 99% 24.0M 0s 2023-07-02T14:02:48.392 INFO:tasks.workunit.client.0.smithi163.stderr:168300K .......... .......... .......... .......... .......... 99% 40.0M 0s 2023-07-02T14:02:48.392 INFO:tasks.workunit.client.0.smithi163.stderr:168350K .......... .......... .......... .......... .......... 99% 23.3M 0s 2023-07-02T14:02:48.392 INFO:tasks.workunit.client.0.smithi163.stderr:168400K .......... .......... .......... .......... .......... 99% 24.0M 0s 2023-07-02T14:02:48.392 INFO:tasks.workunit.client.0.smithi163.stderr:168450K .......... .......... .......... .......... .......... 99% 38.0M 0s 2023-07-02T14:02:48.392 INFO:tasks.workunit.client.0.smithi163.stderr:168500K .......... .......... .......... .......... .......... 99% 24.5M 0s 2023-07-02T14:02:48.392 INFO:tasks.workunit.client.0.smithi163.stderr:168550K .......... .......... . 100% 279M=8.3s 2023-07-02T14:02:48.392 INFO:tasks.workunit.client.0.smithi163.stderr: 2023-07-02T14:02:48.393 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02 14:02:48 (19.9 MB/s) - ‘linux-5.4.tar.gz’ saved [172616875/172616875] 2023-07-02T14:02:48.393 INFO:tasks.workunit.client.0.smithi163.stderr: 2023-07-02T14:02:48.393 INFO:tasks.workunit.client.0.smithi163.stderr:+ untar_pid=78885 2023-07-02T14:02:48.393 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=1 2023-07-02T14:02:48.393 INFO:tasks.workunit.client.0.smithi163.stderr:+ untar_workload 2023-07-02T14:02:48.393 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap1 2023-07-02T14:02:48.393 INFO:tasks.workunit.client.0.smithi163.stderr:+ local i 2023-07-02T14:02:48.393 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i = 0 )) 2023-07-02T14:02:48.394 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i < 10 )) 2023-07-02T14:02:48.394 INFO:tasks.workunit.client.0.smithi163.stderr:+ pv -L 10M linux-5.4.tar.gz 2023-07-02T14:02:48.465 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:48.464+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:48.531 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:48.532 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:48.532 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:48.532 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:48.541 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:02:48.549 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:02:48.582 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:48.581+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:48.666 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:48.665+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:48.698 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:48.705 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:48.705 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:48.706 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:48.706 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:48.706 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:48.716 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:02:48.716 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:48.732 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:48.750 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:48.767 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:48.766+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:48.768 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:48.785 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:48.786 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:02:48.787 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:02:48.787 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:02:48.787 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:02:48.787 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:02:48.787 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:02:48.868 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:48.867+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:48.880 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:48.881 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:48.891 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:02:48.968 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:48.968+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:49.016 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:49.016 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:49.026 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:02:49.069 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:49.068+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:49.072 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:02:49.152 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:49.152 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:49.162 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:02:49.204 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:49.204+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:49.271 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:49.270+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:49.286 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:49.286 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:49.296 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:02:49.372 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:49.371+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:49.420 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:49.420 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:49.430 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:02:49.474 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:49.473+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:49.554 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:49.554 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:49.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:02:49.605 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:49.604+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:49.674 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:49.674+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:49.728 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:49.728 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:49.738 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:02:49.782 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:49.781+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:49.862 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:49.863 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:49.872 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:02:49.877 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:49.876+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:49.977 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:49.976+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:50.036 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:50.036 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:50.046 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:02:50.069 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:02:50.078 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:50.077+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:50.105 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:02:50.111 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap1 b3fbb028-e267-4d25-a750-93ba18a2316a@snap1.diff 2023-07-02T14:02:50.150 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:02:50.149+0000 7f5616f2f700 0 -- 172.21.15.163:0/1981431750 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x5581754aae10 msgr2=0x5581754ad200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-07-02T14:02:50.151 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:02:50.150+0000 7f5616f2f700 0 -- 172.21.15.163:0/1981431750 >> [v2:172.21.15.22:6816/83352,v1:172.21.15.22:6817/83352] conn(0x5581754b6950 msgr2=0x5581754b8dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:02:50.178 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:50.178+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:50.212 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:50.213 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:50.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:02:50.279 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:50.278+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:50.318 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 18% complete... Exporting image: 31% complete... Exporting image: 43% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 56% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-07-02T14:02:50.324 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:02:50.324 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=2 2023-07-02T14:02:50.324 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap2 2023-07-02T14:02:50.356 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:50.356 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:50.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:02:50.407 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:50.406+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:50.480 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:50.479+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:50.530 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:50.530 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:50.540 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:02:50.581 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:50.581+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:50.681 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:50.681+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:50.704 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:50.704 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:50.714 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:02:50.782 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:50.781+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:50.798 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:02:50.844 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:50.845 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:50.855 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:02:50.883 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:50.882+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:50.936 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:50.953 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:50.970 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:50.979 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:50.979 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:50.984 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:50.983+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:50.989 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:50.990 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:02:51.006 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:51.024 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:51.025 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:02:51.025 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:02:51.025 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:02:51.025 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:02:51.026 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:02:51.026 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:02:51.085 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:51.084+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:51.086 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:02:51.091 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 3 2023-07-02T14:02:51.092 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2023-07-02T14:02:51.093 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:02:51.093 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2023-07-02T14:02:51.093 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-07-02T14:02:51.093 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-07-02T14:02:51.094 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2023-07-02T14:02:51.163 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:51.163 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:51.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:51.163 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:02:51.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.164 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:322 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:51.164 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:02:51.164 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 11.861471656999999, 2023-07-02T14:02:51.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.03676041, 2023-07-02T14:02:51.164 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.164 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:51.164 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:51.164 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:51.165 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:51.165 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 322 2023-07-02T14:02:51.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.165 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:02:51.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:02:51.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298819+0000", 2023-07-02T14:02:51.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8068e-05 2023-07-02T14:02:51.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298877+0000", 2023-07-02T14:02:51.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3339999999999998e-06 2023-07-02T14:02:51.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298883+0000", 2023-07-02T14:02:51.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1030000000000004e-06 2023-07-02T14:02:51.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298887+0000", 2023-07-02T14:02:51.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023640230000000002 2023-07-02T14:02:51.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322528+0000", 2023-07-02T14:02:51.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8424999999999994e-05 2023-07-02T14:02:51.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322616+0000", 2023-07-02T14:02:51.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116815 2023-07-02T14:02:51.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:51.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322733+0000", 2023-07-02T14:02:51.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048252629999999998 2023-07-02T14:02:51.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:51.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327558+0000", 2023-07-02T14:02:51.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078965240000000003 2023-07-02T14:02:51.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:51.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335455+0000", 2023-07-02T14:02:51.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0511999999999999e-05 2023-07-02T14:02:51.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:51.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335515+0000", 2023-07-02T14:02:51.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4789000000000006e-05 2023-07-02T14:02:51.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335580+0000", 2023-07-02T14:02:51.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.03676041 2023-07-02T14:02:51.173 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.173 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.174 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.174 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:323 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:51.174 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:02:51.174 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 11.861400277, 2023-07-02T14:02:51.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036875044000000003, 2023-07-02T14:02:51.174 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.175 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:51.175 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:51.175 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:51.175 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:51.175 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 323 2023-07-02T14:02:51.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.175 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:02:51.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:02:51.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298890+0000", 2023-07-02T14:02:51.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7277000000000002e-05 2023-07-02T14:02:51.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298938+0000", 2023-07-02T14:02:51.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0495e-05 2023-07-02T14:02:51.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298948+0000", 2023-07-02T14:02:51.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6829999999999999e-06 2023-07-02T14:02:51.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298953+0000", 2023-07-02T14:02:51.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024271435000000001 2023-07-02T14:02:51.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323224+0000", 2023-07-02T14:02:51.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9266999999999998e-05 2023-07-02T14:02:51.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323304+0000", 2023-07-02T14:02:51.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000157021 2023-07-02T14:02:51.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:51.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323461+0000", 2023-07-02T14:02:51.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041036689999999999 2023-07-02T14:02:51.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:51.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327564+0000", 2023-07-02T14:02:51.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080753490000000008 2023-07-02T14:02:51.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:51.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335640+0000", 2023-07-02T14:02:51.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7398999999999994e-05 2023-07-02T14:02:51.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:51.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335727+0000", 2023-07-02T14:02:51.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8967000000000001e-05 2023-07-02T14:02:51.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335766+0000", 2023-07-02T14:02:51.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036875044000000003 2023-07-02T14:02:51.184 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.184 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.184 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.184 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:324 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:51.184 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:02:51.185 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 11.861332899000001, 2023-07-02T14:02:51.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036935673000000002, 2023-07-02T14:02:51.185 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.185 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:51.185 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:51.185 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:51.185 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:51.186 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 324 2023-07-02T14:02:51.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.186 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:02:51.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:02:51.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:02:51.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5519999999999997e-05 2023-07-02T14:02:51.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299013+0000", 2023-07-02T14:02:51.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0701e-05 2023-07-02T14:02:51.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299024+0000", 2023-07-02T14:02:51.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2379999999999999e-06 2023-07-02T14:02:51.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299028+0000", 2023-07-02T14:02:51.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024972768999999999 2023-07-02T14:02:51.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324001+0000", 2023-07-02T14:02:51.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1178000000000002e-05 2023-07-02T14:02:51.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324072+0000", 2023-07-02T14:02:51.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000166119 2023-07-02T14:02:51.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:51.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324238+0000", 2023-07-02T14:02:51.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033314210000000002 2023-07-02T14:02:51.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:51.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327570+0000", 2023-07-02T14:02:51.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0082393369999999994 2023-07-02T14:02:51.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:51.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335809+0000", 2023-07-02T14:02:51.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7071999999999999e-05 2023-07-02T14:02:51.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:51.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335856+0000", 2023-07-02T14:02:51.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7926999999999998e-05 2023-07-02T14:02:51.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335894+0000", 2023-07-02T14:02:51.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036935673000000002 2023-07-02T14:02:51.194 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.194 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.194 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.195 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:325 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:51.195 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:02:51.195 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 11.861257572, 2023-07-02T14:02:51.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036990838999999998, 2023-07-02T14:02:51.195 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.195 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:51.195 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:51.196 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:51.196 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:51.196 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 325 2023-07-02T14:02:51.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.196 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:02:51.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:02:51.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299033+0000", 2023-07-02T14:02:51.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107619 2023-07-02T14:02:51.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299141+0000", 2023-07-02T14:02:51.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3497e-05 2023-07-02T14:02:51.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.199 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299154+0000", 2023-07-02T14:02:51.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4669e-05 2023-07-02T14:02:51.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.199 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299169+0000", 2023-07-02T14:02:51.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025630159999999999 2023-07-02T14:02:51.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324799+0000", 2023-07-02T14:02:51.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6953000000000007e-05 2023-07-02T14:02:51.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324876+0000", 2023-07-02T14:02:51.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016346199999999999 2023-07-02T14:02:51.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:51.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325039+0000", 2023-07-02T14:02:51.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003566624 2023-07-02T14:02:51.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:51.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328606+0000", 2023-07-02T14:02:51.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073295540000000003 2023-07-02T14:02:51.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:51.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335936+0000", 2023-07-02T14:02:51.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0044999999999999e-05 2023-07-02T14:02:51.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:51.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335986+0000", 2023-07-02T14:02:51.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8872999999999999e-05 2023-07-02T14:02:51.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336024+0000", 2023-07-02T14:02:51.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036990838999999998 2023-07-02T14:02:51.204 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.204 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.205 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.205 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:326 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:51.205 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:02:51.205 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 11.861115644, 2023-07-02T14:02:51.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036975239999999999, 2023-07-02T14:02:51.205 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.206 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:51.206 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:51.206 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:51.206 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:51.206 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 326 2023-07-02T14:02:51.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.206 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:02:51.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:02:51.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:51.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299175+0000", 2023-07-02T14:02:51.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4782000000000003e-05 2023-07-02T14:02:51.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299250+0000", 2023-07-02T14:02:51.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.277e-05 2023-07-02T14:02:51.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299262+0000", 2023-07-02T14:02:51.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1460000000000001e-06 2023-07-02T14:02:51.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299268+0000", 2023-07-02T14:02:51.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026158098000000001 2023-07-02T14:02:51.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325427+0000", 2023-07-02T14:02:51.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4845999999999995e-05 2023-07-02T14:02:51.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325501+0000", 2023-07-02T14:02:51.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015890699999999999 2023-07-02T14:02:51.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:51.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325660+0000", 2023-07-02T14:02:51.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002952114 2023-07-02T14:02:51.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:51.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328612+0000", 2023-07-02T14:02:51.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074491949999999996 2023-07-02T14:02:51.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:51.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336062+0000", 2023-07-02T14:02:51.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7376999999999997e-05 2023-07-02T14:02:51.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:51.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336109+0000", 2023-07-02T14:02:51.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1814000000000003e-05 2023-07-02T14:02:51.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336151+0000", 2023-07-02T14:02:51.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036975239999999999 2023-07-02T14:02:51.215 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.215 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.215 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.215 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:327 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 589824~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:51.215 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:02:51.215 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 11.861016383000001, 2023-07-02T14:02:51.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037234253000000002, 2023-07-02T14:02:51.216 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.216 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:51.216 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:51.216 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:51.216 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:51.216 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 327 2023-07-02T14:02:51.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.217 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:02:51.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:02:51.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299274+0000", 2023-07-02T14:02:51.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2213999999999997e-05 2023-07-02T14:02:51.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299346+0000", 2023-07-02T14:02:51.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1816e-05 2023-07-02T14:02:51.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299358+0000", 2023-07-02T14:02:51.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2626000000000001e-05 2023-07-02T14:02:51.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299371+0000", 2023-07-02T14:02:51.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026916853000000001 2023-07-02T14:02:51.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.221 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326288+0000", 2023-07-02T14:02:51.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6909000000000004e-05 2023-07-02T14:02:51.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326375+0000", 2023-07-02T14:02:51.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015430399999999999 2023-07-02T14:02:51.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:51.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326529+0000", 2023-07-02T14:02:51.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096652100000000005 2023-07-02T14:02:51.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:51.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336194+0000", 2023-07-02T14:02:51.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115299 2023-07-02T14:02:51.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:51.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336309+0000", 2023-07-02T14:02:51.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015981200000000001 2023-07-02T14:02:51.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:51.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336469+0000", 2023-07-02T14:02:51.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9867999999999998e-05 2023-07-02T14:02:51.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336509+0000", 2023-07-02T14:02:51.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037234253000000002 2023-07-02T14:02:51.225 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.225 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.225 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.226 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:328 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:51.226 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:02:51.226 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 11.860912794000001, 2023-07-02T14:02:51.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037216610999999997, 2023-07-02T14:02:51.226 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.226 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:51.227 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:51.227 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:51.227 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:51.227 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 328 2023-07-02T14:02:51.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.227 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:02:51.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:02:51.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:02:51.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1412e-05 2023-07-02T14:02:51.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299449+0000", 2023-07-02T14:02:51.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0219000000000001e-05 2023-07-02T14:02:51.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.230 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299469+0000", 2023-07-02T14:02:51.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9480000000000003e-06 2023-07-02T14:02:51.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.231 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299479+0000", 2023-07-02T14:02:51.231 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027444344999999998 2023-07-02T14:02:51.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.231 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326924+0000", 2023-07-02T14:02:51.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5165000000000001e-05 2023-07-02T14:02:51.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327019+0000", 2023-07-02T14:02:51.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019113499999999999 2023-07-02T14:02:51.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:51.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327210+0000", 2023-07-02T14:02:51.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090197449999999992 2023-07-02T14:02:51.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:51.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336230+0000", 2023-07-02T14:02:51.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028764399999999999 2023-07-02T14:02:51.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:51.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336517+0000", 2023-07-02T14:02:51.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1981999999999999e-05 2023-07-02T14:02:51.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:51.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336559+0000", 2023-07-02T14:02:51.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:02:51.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336595+0000", 2023-07-02T14:02:51.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037216610999999997 2023-07-02T14:02:51.235 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.236 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.236 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.236 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:329 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:51.236 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:02:51.236 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 11.860802876999999, 2023-07-02T14:02:51.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037191142000000003, 2023-07-02T14:02:51.237 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.237 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:51.237 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:51.237 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:51.237 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:51.237 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 329 2023-07-02T14:02:51.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.238 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:02:51.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:02:51.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:51.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299487+0000", 2023-07-02T14:02:51.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7871000000000001e-05 2023-07-02T14:02:51.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.240 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299555+0000", 2023-07-02T14:02:51.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0446e-05 2023-07-02T14:02:51.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.240 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299566+0000", 2023-07-02T14:02:51.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7939999999999999e-06 2023-07-02T14:02:51.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299572+0000", 2023-07-02T14:02:51.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028034019 2023-07-02T14:02:51.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327606+0000", 2023-07-02T14:02:51.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1967999999999999e-05 2023-07-02T14:02:51.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327678+0000", 2023-07-02T14:02:51.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022216900000000001 2023-07-02T14:02:51.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:51.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327900+0000", 2023-07-02T14:02:51.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084016899999999999 2023-07-02T14:02:51.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:51.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336301+0000", 2023-07-02T14:02:51.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00030122800000000001 2023-07-02T14:02:51.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:51.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336603+0000", 2023-07-02T14:02:51.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0380000000000003e-05 2023-07-02T14:02:51.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:51.245 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336643+0000", 2023-07-02T14:02:51.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6455999999999998e-05 2023-07-02T14:02:51.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336679+0000", 2023-07-02T14:02:51.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037191142000000003 2023-07-02T14:02:51.246 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.246 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.246 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.246 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.246 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:330 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:51.247 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:02:51.247 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 11.860712106999999, 2023-07-02T14:02:51.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037849305, 2023-07-02T14:02:51.247 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.247 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:51.247 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:51.247 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:51.247 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:51.248 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 330 2023-07-02T14:02:51.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.248 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:02:51.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:02:51.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299578+0000", 2023-07-02T14:02:51.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4708000000000005e-05 2023-07-02T14:02:51.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299643+0000", 2023-07-02T14:02:51.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7309999999999999e-06 2023-07-02T14:02:51.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.251 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.251 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299652+0000", 2023-07-02T14:02:51.251 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7989999999999999e-06 2023-07-02T14:02:51.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.251 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.251 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299658+0000", 2023-07-02T14:02:51.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028981664000000001 2023-07-02T14:02:51.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.252 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328639+0000", 2023-07-02T14:02:51.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0939000000000006e-05 2023-07-02T14:02:51.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.252 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.253 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328710+0000", 2023-07-02T14:02:51.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000142624 2023-07-02T14:02:51.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:51.253 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328853+0000", 2023-07-02T14:02:51.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078374640000000006 2023-07-02T14:02:51.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:51.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336690+0000", 2023-07-02T14:02:51.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00068453000000000003 2023-07-02T14:02:51.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:51.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337375+0000", 2023-07-02T14:02:51.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0667999999999998e-05 2023-07-02T14:02:51.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:51.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337406+0000", 2023-07-02T14:02:51.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2858999999999999e-05 2023-07-02T14:02:51.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337428+0000", 2023-07-02T14:02:51.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037849305 2023-07-02T14:02:51.256 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.256 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.256 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.257 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:331 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:51.257 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:02:51.257 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 11.860628244999999, 2023-07-02T14:02:51.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037870332, 2023-07-02T14:02:51.257 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.257 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:51.258 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:51.258 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:51.258 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:51.258 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 331 2023-07-02T14:02:51.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.258 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:02:51.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:02:51.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299662+0000", 2023-07-02T14:02:51.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2252e-05 2023-07-02T14:02:51.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299725+0000", 2023-07-02T14:02:51.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8440000000000004e-06 2023-07-02T14:02:51.261 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.261 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.261 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299733+0000", 2023-07-02T14:02:51.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6759999999999998e-06 2023-07-02T14:02:51.261 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.262 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.262 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299738+0000", 2023-07-02T14:02:51.262 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029459387 2023-07-02T14:02:51.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.262 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.262 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.262 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329197+0000", 2023-07-02T14:02:51.262 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0087999999999997e-05 2023-07-02T14:02:51.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.263 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329268+0000", 2023-07-02T14:02:51.263 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014405700000000001 2023-07-02T14:02:51.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:51.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329412+0000", 2023-07-02T14:02:51.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072842100000000002 2023-07-02T14:02:51.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.264 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:51.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336696+0000", 2023-07-02T14:02:51.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00076275300000000002 2023-07-02T14:02:51.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:51.265 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337459+0000", 2023-07-02T14:02:51.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5988000000000002e-05 2023-07-02T14:02:51.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:51.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337495+0000", 2023-07-02T14:02:51.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8686000000000002e-05 2023-07-02T14:02:51.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337533+0000", 2023-07-02T14:02:51.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037870332 2023-07-02T14:02:51.266 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.267 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.267 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.267 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:332 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:51.267 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:02:51.267 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 11.860548321, 2023-07-02T14:02:51.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037892506999999999, 2023-07-02T14:02:51.268 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.268 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:51.268 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:51.268 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:51.268 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:51.268 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 332 2023-07-02T14:02:51.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.268 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:02:51.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:02:51.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299742+0000", 2023-07-02T14:02:51.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1366999999999997e-05 2023-07-02T14:02:51.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.271 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299804+0000", 2023-07-02T14:02:51.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9319999999999996e-06 2023-07-02T14:02:51.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.271 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299812+0000", 2023-07-02T14:02:51.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4359999999999999e-06 2023-07-02T14:02:51.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299818+0000", 2023-07-02T14:02:51.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029946144000000001 2023-07-02T14:02:51.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.273 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329764+0000", 2023-07-02T14:02:51.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000164828 2023-07-02T14:02:51.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.273 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329929+0000", 2023-07-02T14:02:51.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016139400000000001 2023-07-02T14:02:51.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:51.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330090+0000", 2023-07-02T14:02:51.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066134009999999997 2023-07-02T14:02:51.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:51.275 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336704+0000", 2023-07-02T14:02:51.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00086955400000000003 2023-07-02T14:02:51.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.275 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:51.275 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337573+0000", 2023-07-02T14:02:51.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8447000000000001e-05 2023-07-02T14:02:51.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:51.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337602+0000", 2023-07-02T14:02:51.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.362e-05 2023-07-02T14:02:51.276 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337635+0000", 2023-07-02T14:02:51.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037892506999999999 2023-07-02T14:02:51.277 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.277 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.277 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.277 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:333 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:51.277 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:02:51.278 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 11.860468239999999, 2023-07-02T14:02:51.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037932598999999997, 2023-07-02T14:02:51.278 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.278 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:51.278 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:51.278 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:51.278 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:51.278 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 333 2023-07-02T14:02:51.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.279 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.279 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:02:51.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:02:51.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.280 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299822+0000", 2023-07-02T14:02:51.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0831000000000001e-05 2023-07-02T14:02:51.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299883+0000", 2023-07-02T14:02:51.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.322e-06 2023-07-02T14:02:51.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299888+0000", 2023-07-02T14:02:51.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3479999999999997e-06 2023-07-02T14:02:51.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.282 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299893+0000", 2023-07-02T14:02:51.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030831905999999999 2023-07-02T14:02:51.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.283 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330725+0000", 2023-07-02T14:02:51.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3560999999999996e-05 2023-07-02T14:02:51.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330798+0000", 2023-07-02T14:02:51.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015165999999999999 2023-07-02T14:02:51.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:51.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330950+0000", 2023-07-02T14:02:51.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057590319999999999 2023-07-02T14:02:51.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:51.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336709+0000", 2023-07-02T14:02:51.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00096987899999999999 2023-07-02T14:02:51.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:51.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337679+0000", 2023-07-02T14:02:51.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9699000000000001e-05 2023-07-02T14:02:51.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:51.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337719+0000", 2023-07-02T14:02:51.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6961000000000001e-05 2023-07-02T14:02:51.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337756+0000", 2023-07-02T14:02:51.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037932598999999997 2023-07-02T14:02:51.287 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.287 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.287 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.288 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:372 2.0 2:169af44a:::rbd_data.125de1449605.0000000000000106:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:51.288 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:02:51.288 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 11.856890914999999, 2023-07-02T14:02:51.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036066963, 2023-07-02T14:02:51.288 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.288 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:51.288 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:51.289 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:51.289 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:51.289 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 372 2023-07-02T14:02:51.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.289 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:02:51.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:02:51.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:02:51.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1002000000000003e-05 2023-07-02T14:02:51.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303461+0000", 2023-07-02T14:02:51.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9306e-05 2023-07-02T14:02:51.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303490+0000", 2023-07-02T14:02:51.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5026e-05 2023-07-02T14:02:51.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303505+0000", 2023-07-02T14:02:51.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033623129000000002 2023-07-02T14:02:51.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337128+0000", 2023-07-02T14:02:51.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6968000000000002e-05 2023-07-02T14:02:51.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.294 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337175+0000", 2023-07-02T14:02:51.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000101127 2023-07-02T14:02:51.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:51.294 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337276+0000", 2023-07-02T14:02:51.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017866920000000001 2023-07-02T14:02:51.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.295 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:51.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339063+0000", 2023-07-02T14:02:51.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00031459800000000003 2023-07-02T14:02:51.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:51.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339377+0000", 2023-07-02T14:02:51.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7145999999999999e-05 2023-07-02T14:02:51.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:51.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339425+0000", 2023-07-02T14:02:51.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2632000000000001e-05 2023-07-02T14:02:51.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339467+0000", 2023-07-02T14:02:51.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036066963 2023-07-02T14:02:51.297 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.297 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.298 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.298 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:416 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:51.298 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:02:51.298 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 11.842174861, 2023-07-02T14:02:51.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039242025, 2023-07-02T14:02:51.298 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.299 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:51.299 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:51.299 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:51.299 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:51.299 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:02:51.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.299 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:02:51.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:02:51.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:02:51.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.692e-06 2023-07-02T14:02:51.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318119+0000", 2023-07-02T14:02:51.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9790000000000005e-06 2023-07-02T14:02:51.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318129+0000", 2023-07-02T14:02:51.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3640000000000001e-06 2023-07-02T14:02:51.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318137+0000", 2023-07-02T14:02:51.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022607808 2023-07-02T14:02:51.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340745+0000", 2023-07-02T14:02:51.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0130999999999996e-05 2023-07-02T14:02:51.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340825+0000", 2023-07-02T14:02:51.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039581080000000001 2023-07-02T14:02:51.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344783+0000", 2023-07-02T14:02:51.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000006e-05 2023-07-02T14:02:51.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344854+0000", 2023-07-02T14:02:51.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052105550000000004 2023-07-02T14:02:51.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350065+0000", 2023-07-02T14:02:51.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0321999999999997e-05 2023-07-02T14:02:51.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.307 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350115+0000", 2023-07-02T14:02:51.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012378409999999999 2023-07-02T14:02:51.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.307 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351353+0000", 2023-07-02T14:02:51.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5942000000000001e-05 2023-07-02T14:02:51.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351409+0000", 2023-07-02T14:02:51.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035677700000000002 2023-07-02T14:02:51.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:51.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351766+0000", 2023-07-02T14:02:51.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000926719 2023-07-02T14:02:51.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:51.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352693+0000", 2023-07-02T14:02:51.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045652080000000003 2023-07-02T14:02:51.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:51.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357258+0000", 2023-07-02T14:02:51.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4717e-05 2023-07-02T14:02:51.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:51.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357293+0000", 2023-07-02T14:02:51.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5807000000000002e-05 2023-07-02T14:02:51.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357358+0000", 2023-07-02T14:02:51.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039242025 2023-07-02T14:02:51.311 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.312 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.312 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.312 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:417 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:51.312 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:02:51.312 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 11.842149424, 2023-07-02T14:02:51.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042407105, 2023-07-02T14:02:51.313 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.313 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:51.313 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:51.313 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:51.313 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:51.313 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 417 2023-07-02T14:02:51.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.313 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:02:51.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:02:51.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318141+0000", 2023-07-02T14:02:51.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7120000000000002e-06 2023-07-02T14:02:51.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318145+0000", 2023-07-02T14:02:51.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8869999999999997e-06 2023-07-02T14:02:51.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318151+0000", 2023-07-02T14:02:51.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9820000000000002e-06 2023-07-02T14:02:51.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318155+0000", 2023-07-02T14:02:51.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699133 2023-07-02T14:02:51.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340854+0000", 2023-07-02T14:02:51.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6849999999999998e-05 2023-07-02T14:02:51.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340931+0000", 2023-07-02T14:02:51.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039672140000000002 2023-07-02T14:02:51.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344898+0000", 2023-07-02T14:02:51.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5341999999999998e-05 2023-07-02T14:02:51.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344973+0000", 2023-07-02T14:02:51.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051819400000000003 2023-07-02T14:02:51.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350155+0000", 2023-07-02T14:02:51.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3261999999999999e-05 2023-07-02T14:02:51.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350209+0000", 2023-07-02T14:02:51.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00194097 2023-07-02T14:02:51.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352150+0000", 2023-07-02T14:02:51.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6851e-05 2023-07-02T14:02:51.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.322 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352226+0000", 2023-07-02T14:02:51.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051831109999999998 2023-07-02T14:02:51.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357409+0000", 2023-07-02T14:02:51.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5325e-05 2023-07-02T14:02:51.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357495+0000", 2023-07-02T14:02:51.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042107899999999998 2023-07-02T14:02:51.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.324 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:51.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357916+0000", 2023-07-02T14:02:51.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0011672449999999999 2023-07-02T14:02:51.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:51.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.359083+0000", 2023-07-02T14:02:51.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013534930000000001 2023-07-02T14:02:51.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:51.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360437+0000", 2023-07-02T14:02:51.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2159000000000002e-05 2023-07-02T14:02:51.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:51.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360469+0000", 2023-07-02T14:02:51.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0092000000000005e-05 2023-07-02T14:02:51.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360549+0000", 2023-07-02T14:02:51.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042407105 2023-07-02T14:02:51.327 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.327 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.327 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.328 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:418 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:51.328 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:02:51.328 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 11.842130183, 2023-07-02T14:02:51.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.045613663999999998, 2023-07-02T14:02:51.328 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.328 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:51.328 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:51.329 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:51.329 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:51.329 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 418 2023-07-02T14:02:51.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.329 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:02:51.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:02:51.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318160+0000", 2023-07-02T14:02:51.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6490000000000001e-06 2023-07-02T14:02:51.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318164+0000", 2023-07-02T14:02:51.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:02:51.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318170+0000", 2023-07-02T14:02:51.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5279999999999999e-06 2023-07-02T14:02:51.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318173+0000", 2023-07-02T14:02:51.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022788856 2023-07-02T14:02:51.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340962+0000", 2023-07-02T14:02:51.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7730000000000003e-05 2023-07-02T14:02:51.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341040+0000", 2023-07-02T14:02:51.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039795899999999999 2023-07-02T14:02:51.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345020+0000", 2023-07-02T14:02:51.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4962000000000006e-05 2023-07-02T14:02:51.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345085+0000", 2023-07-02T14:02:51.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051955430000000004 2023-07-02T14:02:51.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350280+0000", 2023-07-02T14:02:51.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8913999999999995e-05 2023-07-02T14:02:51.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350379+0000", 2023-07-02T14:02:51.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001887344 2023-07-02T14:02:51.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352266+0000", 2023-07-02T14:02:51.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7865000000000003e-05 2023-07-02T14:02:51.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352314+0000", 2023-07-02T14:02:51.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060621190000000004 2023-07-02T14:02:51.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358376+0000", 2023-07-02T14:02:51.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2448999999999997e-05 2023-07-02T14:02:51.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358449+0000", 2023-07-02T14:02:51.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0021539929999999999 2023-07-02T14:02:51.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360603+0000", 2023-07-02T14:02:51.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7859000000000002e-05 2023-07-02T14:02:51.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360671+0000", 2023-07-02T14:02:51.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026930699999999999 2023-07-02T14:02:51.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:51.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360940+0000", 2023-07-02T14:02:51.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00081179699999999995 2023-07-02T14:02:51.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:51.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361752+0000", 2023-07-02T14:02:51.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0019350490000000001 2023-07-02T14:02:51.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:51.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363687+0000", 2023-07-02T14:02:51.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4462000000000003e-05 2023-07-02T14:02:51.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:51.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363721+0000", 2023-07-02T14:02:51.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3408000000000001e-05 2023-07-02T14:02:51.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363775+0000", 2023-07-02T14:02:51.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.045613663999999998 2023-07-02T14:02:51.345 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.345 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.345 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.345 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:419 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:51.346 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:02:51.346 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 11.842113855999999, 2023-07-02T14:02:51.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.047296703000000002, 2023-07-02T14:02:51.346 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.346 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:51.346 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:51.346 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:51.346 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:51.347 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 419 2023-07-02T14:02:51.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.347 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:02:51.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:02:51.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:02:51.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.433e-06 2023-07-02T14:02:51.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318180+0000", 2023-07-02T14:02:51.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9950000000000002e-06 2023-07-02T14:02:51.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318186+0000", 2023-07-02T14:02:51.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7290000000000002e-06 2023-07-02T14:02:51.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318190+0000", 2023-07-02T14:02:51.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022877989000000001 2023-07-02T14:02:51.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341068+0000", 2023-07-02T14:02:51.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5803999999999994e-05 2023-07-02T14:02:51.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341154+0000", 2023-07-02T14:02:51.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039633000000000003 2023-07-02T14:02:51.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345117+0000", 2023-07-02T14:02:51.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7188999999999996e-05 2023-07-02T14:02:51.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345194+0000", 2023-07-02T14:02:51.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052306059999999996 2023-07-02T14:02:51.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350425+0000", 2023-07-02T14:02:51.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8093999999999997e-05 2023-07-02T14:02:51.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350473+0000", 2023-07-02T14:02:51.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001865158 2023-07-02T14:02:51.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352338+0000", 2023-07-02T14:02:51.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5479000000000003e-05 2023-07-02T14:02:51.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352384+0000", 2023-07-02T14:02:51.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061153290000000001 2023-07-02T14:02:51.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358499+0000", 2023-07-02T14:02:51.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5156999999999999e-05 2023-07-02T14:02:51.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358574+0000", 2023-07-02T14:02:51.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027847459999999998 2023-07-02T14:02:51.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361359+0000", 2023-07-02T14:02:51.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2633999999999999e-05 2023-07-02T14:02:51.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361402+0000", 2023-07-02T14:02:51.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024171510000000002 2023-07-02T14:02:51.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363819+0000", 2023-07-02T14:02:51.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6988999999999999e-05 2023-07-02T14:02:51.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363876+0000", 2023-07-02T14:02:51.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035321300000000001 2023-07-02T14:02:51.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:51.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.364229+0000", 2023-07-02T14:02:51.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00094674799999999997 2023-07-02T14:02:51.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:51.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365176+0000", 2023-07-02T14:02:51.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019724200000000001 2023-07-02T14:02:51.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:51.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365373+0000", 2023-07-02T14:02:51.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2116e-05 2023-07-02T14:02:51.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:51.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365425+0000", 2023-07-02T14:02:51.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8996999999999999e-05 2023-07-02T14:02:51.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365474+0000", 2023-07-02T14:02:51.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.047296703000000002 2023-07-02T14:02:51.368 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.368 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.368 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.368 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:472 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:51.369 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:02:51.369 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 11.782657624, 2023-07-02T14:02:51.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036204721000000002, 2023-07-02T14:02:51.369 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.369 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:51.369 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:51.369 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:51.369 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:51.370 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 472 2023-07-02T14:02:51.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.370 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:02:51.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:02:51.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:02:51.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377633+0000", 2023-07-02T14:02:51.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.815e-06 2023-07-02T14:02:51.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377635+0000", 2023-07-02T14:02:51.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9459999999999998e-06 2023-07-02T14:02:51.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377638+0000", 2023-07-02T14:02:51.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.673e-06 2023-07-02T14:02:51.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377643+0000", 2023-07-02T14:02:51.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043004239999999997 2023-07-02T14:02:51.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.381943+0000", 2023-07-02T14:02:51.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4321999999999999e-05 2023-07-02T14:02:51.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.381998+0000", 2023-07-02T14:02:51.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003008377 2023-07-02T14:02:51.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385006+0000", 2023-07-02T14:02:51.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8751999999999998e-05 2023-07-02T14:02:51.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385045+0000", 2023-07-02T14:02:51.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086260880000000005 2023-07-02T14:02:51.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393671+0000", 2023-07-02T14:02:51.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6365999999999997e-05 2023-07-02T14:02:51.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393707+0000", 2023-07-02T14:02:51.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028560289999999999 2023-07-02T14:02:51.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396563+0000", 2023-07-02T14:02:51.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6195e-05 2023-07-02T14:02:51.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396619+0000", 2023-07-02T14:02:51.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023118269999999998 2023-07-02T14:02:51.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.398931+0000", 2023-07-02T14:02:51.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3885999999999999e-05 2023-07-02T14:02:51.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.398985+0000", 2023-07-02T14:02:51.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043756890000000003 2023-07-02T14:02:51.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403361+0000", 2023-07-02T14:02:51.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0043000000000001e-05 2023-07-02T14:02:51.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403421+0000", 2023-07-02T14:02:51.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001590972 2023-07-02T14:02:51.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405012+0000", 2023-07-02T14:02:51.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0067000000000005e-05 2023-07-02T14:02:51.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405082+0000", 2023-07-02T14:02:51.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050958610000000001 2023-07-02T14:02:51.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410178+0000", 2023-07-02T14:02:51.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7070000000000004e-05 2023-07-02T14:02:51.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410275+0000", 2023-07-02T14:02:51.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060121100000000004 2023-07-02T14:02:51.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410876+0000", 2023-07-02T14:02:51.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3177999999999999e-05 2023-07-02T14:02:51.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410919+0000", 2023-07-02T14:02:51.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000271394 2023-07-02T14:02:51.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:51.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411191+0000", 2023-07-02T14:02:51.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001009598 2023-07-02T14:02:51.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:51.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.412200+0000", 2023-07-02T14:02:51.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014789790000000001 2023-07-02T14:02:51.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:51.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413679+0000", 2023-07-02T14:02:51.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2624000000000002e-05 2023-07-02T14:02:51.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:51.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413742+0000", 2023-07-02T14:02:51.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6564999999999994e-05 2023-07-02T14:02:51.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413838+0000", 2023-07-02T14:02:51.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036204721000000002 2023-07-02T14:02:51.403 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.403 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.403 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.404 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:473 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:51.405 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:02:51.405 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 11.782646213, 2023-07-02T14:02:51.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037915904, 2023-07-02T14:02:51.405 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.406 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:51.406 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:51.406 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:51.406 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:51.407 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 473 2023-07-02T14:02:51.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.407 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:02:51.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:02:51.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:02:51.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:02:51.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.119e-06 2023-07-02T14:02:51.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377647+0000", 2023-07-02T14:02:51.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.875e-06 2023-07-02T14:02:51.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377650+0000", 2023-07-02T14:02:51.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1949999999999999e-06 2023-07-02T14:02:51.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377653+0000", 2023-07-02T14:02:51.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004375444 2023-07-02T14:02:51.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382028+0000", 2023-07-02T14:02:51.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1063999999999997e-05 2023-07-02T14:02:51.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382089+0000", 2023-07-02T14:02:51.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030043869999999999 2023-07-02T14:02:51.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385094+0000", 2023-07-02T14:02:51.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8167999999999997e-05 2023-07-02T14:02:51.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385142+0000", 2023-07-02T14:02:51.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008603421 2023-07-02T14:02:51.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393745+0000", 2023-07-02T14:02:51.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3763000000000005e-05 2023-07-02T14:02:51.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393819+0000", 2023-07-02T14:02:51.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028274720000000001 2023-07-02T14:02:51.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396647+0000", 2023-07-02T14:02:51.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7929999999999997e-05 2023-07-02T14:02:51.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396695+0000", 2023-07-02T14:02:51.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023426459999999999 2023-07-02T14:02:51.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399037+0000", 2023-07-02T14:02:51.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5846000000000004e-05 2023-07-02T14:02:51.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399123+0000", 2023-07-02T14:02:51.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043448360000000004 2023-07-02T14:02:51.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403468+0000", 2023-07-02T14:02:51.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6699000000000001e-05 2023-07-02T14:02:51.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403515+0000", 2023-07-02T14:02:51.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001622943 2023-07-02T14:02:51.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405138+0000", 2023-07-02T14:02:51.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6382000000000002e-05 2023-07-02T14:02:51.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405204+0000", 2023-07-02T14:02:51.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051089910000000002 2023-07-02T14:02:51.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410313+0000", 2023-07-02T14:02:51.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8106000000000003e-05 2023-07-02T14:02:51.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410361+0000", 2023-07-02T14:02:51.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012200399999999999 2023-07-02T14:02:51.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411581+0000", 2023-07-02T14:02:51.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5067000000000005e-05 2023-07-02T14:02:51.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411656+0000", 2023-07-02T14:02:51.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022353249999999998 2023-07-02T14:02:51.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413891+0000", 2023-07-02T14:02:51.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9356000000000003e-05 2023-07-02T14:02:51.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413981+0000", 2023-07-02T14:02:51.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00043818299999999998 2023-07-02T14:02:51.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:51.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.414419+0000", 2023-07-02T14:02:51.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00081559199999999997 2023-07-02T14:02:51.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:51.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415235+0000", 2023-07-02T14:02:51.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026434499999999998 2023-07-02T14:02:51.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:51.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415499+0000", 2023-07-02T14:02:51.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9779999999999999e-05 2023-07-02T14:02:51.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:51.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415529+0000", 2023-07-02T14:02:51.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2159999999999997e-05 2023-07-02T14:02:51.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415561+0000", 2023-07-02T14:02:51.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037915904 2023-07-02T14:02:51.438 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.438 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.438 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.438 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:475 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:51.439 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:02:51.439 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 11.782636174, 2023-07-02T14:02:51.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039286962000000002, 2023-07-02T14:02:51.439 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.439 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:51.439 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:51.439 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:51.439 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:51.440 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 475 2023-07-02T14:02:51.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.440 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:02:51.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:02:51.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:02:51.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4779999999999997e-06 2023-07-02T14:02:51.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377664+0000", 2023-07-02T14:02:51.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.268e-06 2023-07-02T14:02:51.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377671+0000", 2023-07-02T14:02:51.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.861e-06 2023-07-02T14:02:51.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377674+0000", 2023-07-02T14:02:51.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044395789999999999 2023-07-02T14:02:51.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382114+0000", 2023-07-02T14:02:51.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8096999999999997e-05 2023-07-02T14:02:51.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382162+0000", 2023-07-02T14:02:51.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029949059999999999 2023-07-02T14:02:51.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385157+0000", 2023-07-02T14:02:51.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1158000000000001e-05 2023-07-02T14:02:51.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385198+0000", 2023-07-02T14:02:51.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086477229999999995 2023-07-02T14:02:51.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393846+0000", 2023-07-02T14:02:51.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5990999999999999e-05 2023-07-02T14:02:51.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393912+0000", 2023-07-02T14:02:51.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027979599999999999 2023-07-02T14:02:51.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396710+0000", 2023-07-02T14:02:51.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5911000000000002e-05 2023-07-02T14:02:51.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396746+0000", 2023-07-02T14:02:51.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024039759999999999 2023-07-02T14:02:51.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399150+0000", 2023-07-02T14:02:51.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7834999999999999e-05 2023-07-02T14:02:51.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399188+0000", 2023-07-02T14:02:51.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004364054 2023-07-02T14:02:51.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403552+0000", 2023-07-02T14:02:51.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6274999999999999e-05 2023-07-02T14:02:51.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403608+0000", 2023-07-02T14:02:51.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0016261540000000001 2023-07-02T14:02:51.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405234+0000", 2023-07-02T14:02:51.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6945000000000004e-05 2023-07-02T14:02:51.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405301+0000", 2023-07-02T14:02:51.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050871329999999998 2023-07-02T14:02:51.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410388+0000", 2023-07-02T14:02:51.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0210999999999995e-05 2023-07-02T14:02:51.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410468+0000", 2023-07-02T14:02:51.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001227454 2023-07-02T14:02:51.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411696+0000", 2023-07-02T14:02:51.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8188000000000005e-05 2023-07-02T14:02:51.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411764+0000", 2023-07-02T14:02:51.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031779930000000001 2023-07-02T14:02:51.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.414942+0000", 2023-07-02T14:02:51.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7236000000000004e-05 2023-07-02T14:02:51.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:02:51.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415019+0000", 2023-07-02T14:02:51.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000573045 2023-07-02T14:02:51.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415592+0000", 2023-07-02T14:02:51.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7580000000000002e-05 2023-07-02T14:02:51.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415640+0000", 2023-07-02T14:02:51.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024783200000000002 2023-07-02T14:02:51.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:51.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415888+0000", 2023-07-02T14:02:51.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00080440999999999998 2023-07-02T14:02:51.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:51.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416692+0000", 2023-07-02T14:02:51.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000195787 2023-07-02T14:02:51.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:51.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416888+0000", 2023-07-02T14:02:51.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0787999999999997e-05 2023-07-02T14:02:51.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:51.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416919+0000", 2023-07-02T14:02:51.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.338e-05 2023-07-02T14:02:51.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416942+0000", 2023-07-02T14:02:51.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039286962000000002 2023-07-02T14:02:51.461 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.461 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.461 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.461 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.461 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.461 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:51.463 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:02:51.464 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:51.464 INFO:teuthology.orchestra.run.smithi022.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-07-02T14:02:51.464 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:51.464 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:51.462+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:51.563 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:51.563+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:51.629 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:51.629 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:51.629 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:51.629 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:51.639 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:02:51.680 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:51.679+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:51.765 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:51.764+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:51.802 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:51.803 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:51.803 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:51.803 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:51.803 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:51.813 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:02:51.865 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:51.865+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:51.937 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:51.938 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:51.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:51.938 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:02:51.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.938 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:1e298de8:::rbd_id.b3fbb028-e267-4d25-a750-93ba18a2316a-src:head, version: 40'1, data_included: [0~16], data_size: 16, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:1e298de8:::rbd_id.b3fbb028-e267-4d25-a750-93ba18a2316a-src:head@40'1, size: 16, copy_subset: [0~16], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:16, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-07-02T14:02:51.938 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.133561+0000", 2023-07-02T14:02:51.939 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.8031057229999998, 2023-07-02T14:02:51.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000366984, 2023-07-02T14:02:51.939 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.939 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:02:51.939 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.133561+0000", 2023-07-02T14:02:51.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.133561+0000", 2023-07-02T14:02:51.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:51.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.133560+0000", 2023-07-02T14:02:51.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2972e-05 2023-07-02T14:02:51.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.133573+0000", 2023-07-02T14:02:51.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6759e-05 2023-07-02T14:02:51.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.133590+0000", 2023-07-02T14:02:51.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4440000000000008e-06 2023-07-02T14:02:51.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.133598+0000", 2023-07-02T14:02:51.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9666000000000003e-05 2023-07-02T14:02:51.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.133638+0000", 2023-07-02T14:02:51.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1415e-05 2023-07-02T14:02:51.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.133650+0000", 2023-07-02T14:02:51.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00027846499999999998 2023-07-02T14:02:51.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.133928+0000", 2023-07-02T14:02:51.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000366984 2023-07-02T14:02:51.946 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.946 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.946 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.947 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:169af44a:::rbd_data.125de1449605.0000000000000106:head, version: 40'57, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:169af44a:::rbd_data.125de1449605.0000000000000106:head@40'57, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-07-02T14:02:51.947 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.136138+0000", 2023-07-02T14:02:51.947 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.8005291200000002, 2023-07-02T14:02:51.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033651969999999999, 2023-07-02T14:02:51.947 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.948 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:02:51.948 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.136138+0000", 2023-07-02T14:02:51.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.136138+0000", 2023-07-02T14:02:51.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.136137+0000", 2023-07-02T14:02:51.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001911903 2023-07-02T14:02:51.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.138049+0000", 2023-07-02T14:02:51.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6398000000000001e-05 2023-07-02T14:02:51.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.138065+0000", 2023-07-02T14:02:51.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8220000000000007e-06 2023-07-02T14:02:51.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.138073+0000", 2023-07-02T14:02:51.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8141999999999998e-05 2023-07-02T14:02:51.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.138151+0000", 2023-07-02T14:02:51.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5259999999999999e-06 2023-07-02T14:02:51.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.138161+0000", 2023-07-02T14:02:51.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001342085 2023-07-02T14:02:51.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.954 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.139503+0000", 2023-07-02T14:02:51.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033651969999999999 2023-07-02T14:02:51.954 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.955 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.955 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.955 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:169af44a:::rbd_data.125de1449605.0000000000000106:head, version: 40'57, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:169af44a:::rbd_data.125de1449605.0000000000000106:head@40'57, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:02:51.955 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.141582+0000", 2023-07-02T14:02:51.955 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.795085271, 2023-07-02T14:02:51.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029211950000000001, 2023-07-02T14:02:51.956 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.956 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:02:51.956 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.141582+0000", 2023-07-02T14:02:51.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.141582+0000", 2023-07-02T14:02:51.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.141581+0000", 2023-07-02T14:02:51.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001864533 2023-07-02T14:02:51.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.143445+0000", 2023-07-02T14:02:51.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5650000000000001e-05 2023-07-02T14:02:51.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.143461+0000", 2023-07-02T14:02:51.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.109e-06 2023-07-02T14:02:51.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.143468+0000", 2023-07-02T14:02:51.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0136999999999996e-05 2023-07-02T14:02:51.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.143538+0000", 2023-07-02T14:02:51.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4500000000000004e-06 2023-07-02T14:02:51.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.143547+0000", 2023-07-02T14:02:51.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00095602800000000004 2023-07-02T14:02:51.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.144503+0000", 2023-07-02T14:02:51.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029211950000000001 2023-07-02T14:02:51.962 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.962 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.963 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.963 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:169af44a:::rbd_data.125de1449605.0000000000000106:head, version: 40'57, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:169af44a:::rbd_data.125de1449605.0000000000000106:head@40'57, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:02:51.963 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.146696+0000", 2023-07-02T14:02:51.963 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.7899709399999999, 2023-07-02T14:02:51.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032948410000000002, 2023-07-02T14:02:51.963 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.964 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:02:51.964 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.146696+0000", 2023-07-02T14:02:51.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.146696+0000", 2023-07-02T14:02:51.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:51.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.146695+0000", 2023-07-02T14:02:51.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022307080000000001 2023-07-02T14:02:51.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.148926+0000", 2023-07-02T14:02:51.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.626e-05 2023-07-02T14:02:51.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.148942+0000", 2023-07-02T14:02:51.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4460000000000004e-06 2023-07-02T14:02:51.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.148949+0000", 2023-07-02T14:02:51.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2187999999999994e-05 2023-07-02T14:02:51.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.149022+0000", 2023-07-02T14:02:51.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3311000000000001e-05 2023-07-02T14:02:51.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.149045+0000", 2023-07-02T14:02:51.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00094571599999999998 2023-07-02T14:02:51.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.149991+0000", 2023-07-02T14:02:51.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032948410000000002 2023-07-02T14:02:51.970 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.970 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.970 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.971 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:169af44a:::rbd_data.125de1449605.0000000000000106:head, version: 40'57, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:169af44a:::rbd_data.125de1449605.0000000000000106:head@40'57, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:02:51.971 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.151904+0000", 2023-07-02T14:02:51.971 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.784763018, 2023-07-02T14:02:51.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031662180000000002, 2023-07-02T14:02:51.971 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.971 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:02:51.971 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.151904+0000", 2023-07-02T14:02:51.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.151904+0000", 2023-07-02T14:02:51.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.151903+0000", 2023-07-02T14:02:51.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001946234 2023-07-02T14:02:51.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.153849+0000", 2023-07-02T14:02:51.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3519999999999999e-05 2023-07-02T14:02:51.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.153863+0000", 2023-07-02T14:02:51.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6320000000000003e-06 2023-07-02T14:02:51.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.153873+0000", 2023-07-02T14:02:51.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0098000000000005e-05 2023-07-02T14:02:51.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.153943+0000", 2023-07-02T14:02:51.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1810000000000007e-06 2023-07-02T14:02:51.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.153952+0000", 2023-07-02T14:02:51.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0011181240000000001 2023-07-02T14:02:51.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.155070+0000", 2023-07-02T14:02:51.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031662180000000002 2023-07-02T14:02:51.978 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.978 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.978 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.978 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:1971cf55:::rbd_data.125de1449605.00000000000000e0:head, version: 40'79, data_included: [0~8192], data_size: 8192, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:1971cf55:::rbd_data.125de1449605.00000000000000e0:head@40'79, size: 8192, copy_subset: [0~8192], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:8192, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-07-02T14:02:51.978 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.156685+0000", 2023-07-02T14:02:51.978 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.779981308, 2023-07-02T14:02:51.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00038141899999999997, 2023-07-02T14:02:51.979 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.979 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:02:51.979 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.156685+0000", 2023-07-02T14:02:51.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.156685+0000", 2023-07-02T14:02:51.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.156685+0000", 2023-07-02T14:02:51.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3300999999999999e-05 2023-07-02T14:02:51.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.156708+0000", 2023-07-02T14:02:51.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3427e-05 2023-07-02T14:02:51.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.156722+0000", 2023-07-02T14:02:51.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3539999999999996e-06 2023-07-02T14:02:51.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.156728+0000", 2023-07-02T14:02:51.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9743999999999998e-05 2023-07-02T14:02:51.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.156768+0000", 2023-07-02T14:02:51.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3409999999999996e-06 2023-07-02T14:02:51.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.156772+0000", 2023-07-02T14:02:51.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000294841 2023-07-02T14:02:51.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.157067+0000", 2023-07-02T14:02:51.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00038141899999999997 2023-07-02T14:02:51.985 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.985 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.985 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.986 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:15e08356:::rbd_data.125de1449605.0000000000000002:head, version: 40'112, data_included: [0~131072,135168~917504], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:15e08356:::rbd_data.125de1449605.0000000000000002:head@40'112, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1052672, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-07-02T14:02:51.986 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.159064+0000", 2023-07-02T14:02:51.986 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.777602275, 2023-07-02T14:02:51.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036074649999999998, 2023-07-02T14:02:51.986 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.986 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:02:51.986 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.159064+0000", 2023-07-02T14:02:51.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.159064+0000", 2023-07-02T14:02:51.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:51.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.159064+0000", 2023-07-02T14:02:51.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024697149999999999 2023-07-02T14:02:51.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.161533+0000", 2023-07-02T14:02:51.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5977999999999998e-05 2023-07-02T14:02:51.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.161549+0000", 2023-07-02T14:02:51.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5580000000000003e-06 2023-07-02T14:02:51.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.161557+0000", 2023-07-02T14:02:51.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4352999999999998e-05 2023-07-02T14:02:51.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.161631+0000", 2023-07-02T14:02:51.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8870000000000004e-06 2023-07-02T14:02:51.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:51.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.161641+0000", 2023-07-02T14:02:51.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0010307560000000001 2023-07-02T14:02:51.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:51.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.162672+0000", 2023-07-02T14:02:51.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036074649999999998 2023-07-02T14:02:51.993 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.993 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:51.993 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:51.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.993 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:15e08356:::rbd_data.125de1449605.0000000000000002:head, version: 40'112, data_included: [1052672~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:15e08356:::rbd_data.125de1449605.0000000000000002:head@40'112, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2101248, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1052672, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:02:51.993 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.164941+0000", 2023-07-02T14:02:51.994 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.7717258440000001, 2023-07-02T14:02:51.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002871644, 2023-07-02T14:02:51.994 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:51.994 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:02:51.994 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:51.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:51.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.164941+0000", 2023-07-02T14:02:51.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:51.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:51.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.164941+0000", 2023-07-02T14:02:51.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:51.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:51.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.164940+0000", 2023-07-02T14:02:51.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001801223 2023-07-02T14:02:51.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:51.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.166741+0000", 2023-07-02T14:02:51.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5262999999999999e-05 2023-07-02T14:02:51.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:51.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.166757+0000", 2023-07-02T14:02:51.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2327000000000001e-05 2023-07-02T14:02:51.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:51.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.166769+0000", 2023-07-02T14:02:51.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9894000000000005e-05 2023-07-02T14:02:51.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:51.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.166839+0000", 2023-07-02T14:02:51.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4280000000000002e-06 2023-07-02T14:02:51.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:51.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:51.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.166848+0000", 2023-07-02T14:02:52.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00096417599999999997 2023-07-02T14:02:52.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.167813+0000", 2023-07-02T14:02:52.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002871644 2023-07-02T14:02:52.001 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.001 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.001 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.001 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:15e08356:::rbd_data.125de1449605.0000000000000002:head, version: 40'112, data_included: [2101248~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:15e08356:::rbd_data.125de1449605.0000000000000002:head@40'112, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3149824, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2101248, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:02:52.001 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.169810+0000", 2023-07-02T14:02:52.001 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.7668568950000001, 2023-07-02T14:02:52.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025331210000000002, 2023-07-02T14:02:52.002 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.002 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:02:52.002 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.169810+0000", 2023-07-02T14:02:52.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.169810+0000", 2023-07-02T14:02:52.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:52.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.169809+0000", 2023-07-02T14:02:52.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0016680919999999999 2023-07-02T14:02:52.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.171477+0000", 2023-07-02T14:02:52.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5795999999999999e-05 2023-07-02T14:02:52.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.171493+0000", 2023-07-02T14:02:52.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7649999999999993e-06 2023-07-02T14:02:52.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.171501+0000", 2023-07-02T14:02:52.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4740000000000002e-05 2023-07-02T14:02:52.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.171546+0000", 2023-07-02T14:02:52.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6750000000000001e-06 2023-07-02T14:02:52.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.171550+0000", 2023-07-02T14:02:52.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00079280099999999999 2023-07-02T14:02:52.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.172343+0000", 2023-07-02T14:02:52.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025331210000000002 2023-07-02T14:02:52.008 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.008 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.008 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.009 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:15e08356:::rbd_data.125de1449605.0000000000000002:head, version: 40'112, data_included: [3149824~1044480], data_size: 1044480, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:15e08356:::rbd_data.125de1449605.0000000000000002:head@40'112, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3149824, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:02:52.009 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.174306+0000", 2023-07-02T14:02:52.009 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.762360557, 2023-07-02T14:02:52.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0021748200000000001, 2023-07-02T14:02:52.009 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.010 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:02:52.010 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.174306+0000", 2023-07-02T14:02:52.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.174306+0000", 2023-07-02T14:02:52.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:52.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.174305+0000", 2023-07-02T14:02:52.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00092398899999999995 2023-07-02T14:02:52.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.175229+0000", 2023-07-02T14:02:52.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5741999999999999e-05 2023-07-02T14:02:52.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.175245+0000", 2023-07-02T14:02:52.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2989999999999999e-06 2023-07-02T14:02:52.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.175252+0000", 2023-07-02T14:02:52.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9908999999999998e-05 2023-07-02T14:02:52.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.175322+0000", 2023-07-02T14:02:52.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4709999999999996e-06 2023-07-02T14:02:52.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.175332+0000", 2023-07-02T14:02:52.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0011491889999999999 2023-07-02T14:02:52.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.176481+0000", 2023-07-02T14:02:52.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0021748200000000001 2023-07-02T14:02:52.016 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.016 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.016 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.016 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:1c72c23d:::rbd_object_map.125de1449605:head, version: 40'119, data_included: [0~162], data_size: 162, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2:1c72c23d:::rbd_object_map.125de1449605:head@40'119, size: 162, copy_subset: [0~162], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:162, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-07-02T14:02:52.016 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.177916+0000", 2023-07-02T14:02:52.017 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.7587507790000001, 2023-07-02T14:02:52.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00034433199999999998, 2023-07-02T14:02:52.017 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.017 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:02:52.017 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.177916+0000", 2023-07-02T14:02:52.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.177916+0000", 2023-07-02T14:02:52.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:52.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.177915+0000", 2023-07-02T14:02:52.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0332999999999999e-05 2023-07-02T14:02:52.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.177926+0000", 2023-07-02T14:02:52.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2816000000000001e-05 2023-07-02T14:02:52.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.177939+0000", 2023-07-02T14:02:52.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4629999999999998e-06 2023-07-02T14:02:52.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.177945+0000", 2023-07-02T14:02:52.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7746000000000001e-05 2023-07-02T14:02:52.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.177983+0000", 2023-07-02T14:02:52.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2640000000000002e-06 2023-07-02T14:02:52.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.177987+0000", 2023-07-02T14:02:52.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00027325400000000001 2023-07-02T14:02:52.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.178260+0000", 2023-07-02T14:02:52.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00034433199999999998 2023-07-02T14:02:52.023 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.023 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.023 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.024 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:1752944e:::rbd_data.125de1449605.000000000000000d:head, version: 40'120, data_included: [0~131072], data_size: 131072, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:1752944e:::rbd_data.125de1449605.000000000000000d:head@40'120, size: 131072, copy_subset: [0~131072], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131072, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-07-02T14:02:52.024 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.179464+0000", 2023-07-02T14:02:52.024 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.7572028249999998, 2023-07-02T14:02:52.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00049279599999999999, 2023-07-02T14:02:52.024 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.025 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:02:52.025 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.179464+0000", 2023-07-02T14:02:52.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.179464+0000", 2023-07-02T14:02:52.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:52.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.179463+0000", 2023-07-02T14:02:52.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000114058 2023-07-02T14:02:52.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.179577+0000", 2023-07-02T14:02:52.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5682999999999999e-05 2023-07-02T14:02:52.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.179593+0000", 2023-07-02T14:02:52.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2030000000000003e-06 2023-07-02T14:02:52.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.179600+0000", 2023-07-02T14:02:52.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7911999999999999e-05 2023-07-02T14:02:52.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.179638+0000", 2023-07-02T14:02:52.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2679999999999996e-06 2023-07-02T14:02:52.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.179642+0000", 2023-07-02T14:02:52.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00031440100000000001 2023-07-02T14:02:52.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.179957+0000", 2023-07-02T14:02:52.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00049279599999999999 2023-07-02T14:02:52.031 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.031 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.031 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.032 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:905 2.0 e48/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 48'136, mlcod=47'134)", 2023-07-02T14:02:52.032 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:51.077909+0000", 2023-07-02T14:02:52.032 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.85875738000000001, 2023-07-02T14:02:52.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00069600100000000002, 2023-07-02T14:02:52.032 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.032 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.032 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:51.077909+0000", 2023-07-02T14:02:52.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:51.077909+0000", 2023-07-02T14:02:52.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:02:52.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:51.077909+0000", 2023-07-02T14:02:52.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4449999999999998e-06 2023-07-02T14:02:52.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:51.077914+0000", 2023-07-02T14:02:52.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3623e-05 2023-07-02T14:02:52.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:51.077927+0000", 2023-07-02T14:02:52.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.682e-06 2023-07-02T14:02:52.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:51.077933+0000", 2023-07-02T14:02:52.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9118000000000001e-05 2023-07-02T14:02:52.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:51.077972+0000", 2023-07-02T14:02:52.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3832000000000001e-05 2023-07-02T14:02:52.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:51.077986+0000", 2023-07-02T14:02:52.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00059268799999999996 2023-07-02T14:02:52.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:51.078579+0000", 2023-07-02T14:02:52.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6829e-05 2023-07-02T14:02:52.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:51.078605+0000", 2023-07-02T14:02:52.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00069600100000000002 2023-07-02T14:02:52.039 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.039 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.039 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.039 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.039 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.040 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:52.041 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:02:52.042 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:52.040+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:52.087 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:02:52.142 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:52.141+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:52.205 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:52.206 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:52.206 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:52.206 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:52.216 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:02:52.257 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:52.256+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:52.343 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:52.342+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:52.380 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:52.380 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:52.380 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:02:52.380 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:02:52.381 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:52.390 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:02:52.445 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:52.444+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:52.516 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:52.516 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:02:52.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:02:52.517 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:02:52.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.517 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:52.517 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:02:52.517 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.223267916999999, 2023-07-02T14:02:52.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:02:52.517 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.517 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.518 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:52.518 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:52.518 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:52.518 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:02:52.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.518 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:02:52.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:02:52.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:52.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:02:52.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:02:52.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:02:52.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:02:52.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:02:52.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:02:52.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:02:52.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:02:52.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:02:52.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:02:52.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:02:52.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:02:52.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:52.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:02:52.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:02:52.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:52.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:02:52.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:02:52.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:52.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:02:52.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:02:52.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:02:52.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:02:52.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:02:52.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:02:52.528 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.528 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.528 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.528 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:52.528 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:02:52.529 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.223032893999999, 2023-07-02T14:02:52.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:02:52.529 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.530 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.530 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:52.530 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:52.530 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:52.530 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:02:52.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.530 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:02:52.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:02:52.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:52.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:02:52.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:02:52.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:02:52.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:02:52.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:02:52.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:02:52.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:02:52.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:02:52.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:02:52.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:02:52.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:02:52.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:02:52.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:52.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:02:52.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:02:52.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:52.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:02:52.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:02:52.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:52.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:02:52.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:02:52.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:02:52.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:02:52.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:02:52.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:02:52.541 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.541 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.542 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.542 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:52.542 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:02:52.542 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.22283376, 2023-07-02T14:02:52.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:02:52.542 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.543 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.543 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:52.543 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:52.543 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:52.543 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:02:52.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.543 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:02:52.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:02:52.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:52.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:02:52.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:02:52.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:02:52.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:02:52.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:02:52.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:02:52.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:02:52.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:02:52.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:02:52.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:02:52.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:02:52.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:02:52.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:52.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:02:52.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:02:52.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:52.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:02:52.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:02:52.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:52.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:02:52.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:02:52.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:02:52.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:02:52.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:02:52.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:02:52.552 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.552 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.552 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.552 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:52.553 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:02:52.553 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.222468636, 2023-07-02T14:02:52.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:02:52.553 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.553 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.553 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:52.554 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:52.554 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:52.554 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:02:52.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.554 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:02:52.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:02:52.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:52.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:02:52.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:02:52.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:02:52.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:02:52.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:02:52.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:02:52.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:02:52.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:02:52.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:02:52.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:02:52.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:02:52.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:02:52.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:52.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:02:52.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:02:52.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:52.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:02:52.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:02:52.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:52.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:02:52.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:02:52.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:02:52.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:02:52.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:02:52.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:02:52.563 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.563 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.563 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.563 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:52.563 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:02:52.563 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.222387575000001, 2023-07-02T14:02:52.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:02:52.564 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.564 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.564 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:52.564 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:52.564 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:52.564 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:02:52.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.565 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:02:52.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:02:52.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:52.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:02:52.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:02:52.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:02:52.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:02:52.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:02:52.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:02:52.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:02:52.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:02:52.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:02:52.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:02:52.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:02:52.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:02:52.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:52.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:02:52.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:02:52.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:52.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:02:52.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:02:52.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:52.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:02:52.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:02:52.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:02:52.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:02:52.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:02:52.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:02:52.573 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.573 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.573 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.574 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:52.574 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:02:52.574 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.222295903999999, 2023-07-02T14:02:52.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:02:52.574 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.574 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.574 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:52.575 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:52.575 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:52.575 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:02:52.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.575 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:02:52.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:02:52.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:52.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:02:52.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:02:52.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:02:52.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:02:52.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:02:52.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:02:52.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:02:52.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:02:52.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:02:52.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:02:52.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:02:52.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:02:52.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:52.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:02:52.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:02:52.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:52.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:02:52.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:02:52.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:52.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:02:52.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:02:52.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:02:52.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:02:52.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:02:52.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:02:52.583 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.583 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.584 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.584 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:52.584 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:02:52.584 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.221956462, 2023-07-02T14:02:52.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:02:52.585 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.585 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.585 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:52.585 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:52.585 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:52.585 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:02:52.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.585 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:02:52.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:02:52.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:52.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:02:52.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:02:52.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:02:52.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:02:52.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:02:52.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:02:52.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:02:52.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:02:52.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:02:52.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:02:52.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:02:52.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:02:52.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:52.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:02:52.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:02:52.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:52.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:02:52.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:02:52.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:52.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:02:52.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:02:52.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:02:52.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:02:52.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:02:52.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:02:52.594 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.594 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.594 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.594 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:52.595 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:02:52.595 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.221770408999999, 2023-07-02T14:02:52.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:02:52.595 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.595 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.595 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:52.595 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:52.595 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:52.596 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:02:52.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.596 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:02:52.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:02:52.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:52.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:02:52.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:02:52.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:02:52.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:02:52.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:02:52.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:02:52.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:02:52.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:02:52.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:02:52.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:02:52.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:02:52.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:02:52.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:52.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:02:52.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:02:52.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:52.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:02:52.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:02:52.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:52.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:02:52.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:02:52.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:02:52.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:02:52.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.604 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.604 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:02:52.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:02:52.604 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.604 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.604 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.605 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:52.605 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:02:52.605 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.221612607000001, 2023-07-02T14:02:52.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:02:52.605 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.605 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.605 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:52.606 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:52.606 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:52.606 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:02:52.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.606 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:02:52.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:02:52.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:52.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:02:52.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:02:52.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:02:52.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:02:52.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:02:52.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:02:52.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:02:52.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:02:52.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:02:52.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:02:52.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:02:52.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:02:52.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:52.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:02:52.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:02:52.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:52.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:02:52.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:02:52.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:52.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:02:52.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:02:52.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:02:52.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:02:52.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:02:52.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:02:52.615 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.615 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.615 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.615 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:52.615 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:02:52.615 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.221420824999999, 2023-07-02T14:02:52.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:02:52.616 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.616 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.616 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:52.616 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:52.616 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:52.616 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:02:52.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.617 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:02:52.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:02:52.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:52.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:02:52.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:02:52.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:02:52.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:02:52.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:02:52.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:02:52.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:02:52.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:02:52.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:02:52.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:02:52.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:02:52.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:02:52.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:52.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:02:52.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:02:52.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:52.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:02:52.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:02:52.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:52.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:02:52.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:02:52.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:02:52.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:02:52.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.625 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:02:52.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:02:52.625 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.625 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.625 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.626 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:52.626 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:02:52.626 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.221245164999999, 2023-07-02T14:02:52.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:02:52.626 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.626 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.626 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:52.626 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:52.627 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:52.627 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:02:52.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.627 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:02:52.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:02:52.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:52.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:02:52.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:02:52.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:02:52.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:02:52.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:02:52.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:02:52.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:02:52.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:02:52.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:02:52.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:02:52.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:02:52.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:02:52.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:52.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:02:52.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:02:52.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:52.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:02:52.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:02:52.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:52.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:02:52.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:02:52.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:02:52.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:02:52.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:02:52.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:02:52.635 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.635 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.635 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.636 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:52.636 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:02:52.636 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.221150977000001, 2023-07-02T14:02:52.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:02:52.636 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.637 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.637 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:52.637 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:52.637 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:52.637 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:02:52.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.637 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:02:52.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:02:52.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:52.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:02:52.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:02:52.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:02:52.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:02:52.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:02:52.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:02:52.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:02:52.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:02:52.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:02:52.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:02:52.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:02:52.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:02:52.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:52.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:02:52.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:02:52.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:52.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:02:52.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:02:52.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:52.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:02:52.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:02:52.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:02:52.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:02:52.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:02:52.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:02:52.646 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.646 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.646 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.646 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:52.646 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:02:52.647 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.220947591, 2023-07-02T14:02:52.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:02:52.647 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.647 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.647 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:52.647 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:52.647 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:52.647 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:02:52.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.648 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:02:52.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:02:52.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:52.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:02:52.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:02:52.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:02:52.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:02:52.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:02:52.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:02:52.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:02:52.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:02:52.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:02:52.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:02:52.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:02:52.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:02:52.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:52.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:02:52.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:02:52.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:52.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:02:52.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:02:52.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:52.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:02:52.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:02:52.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:02:52.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:02:52.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:02:52.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:02:52.656 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.656 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.656 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.657 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:52.657 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:02:52.657 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.220796934999999, 2023-07-02T14:02:52.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:02:52.657 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.657 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.657 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:52.657 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:52.658 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:52.658 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:02:52.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.658 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:02:52.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:02:52.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:52.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:02:52.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:02:52.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:02:52.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:02:52.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:02:52.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:02:52.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:02:52.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:02:52.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:02:52.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:02:52.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:02:52.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:02:52.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:52.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:02:52.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:02:52.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:52.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:02:52.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:02:52.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:52.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:02:52.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:02:52.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:02:52.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:02:52.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:02:52.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:02:52.666 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.667 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.667 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.667 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:52.667 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:02:52.667 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.220724232, 2023-07-02T14:02:52.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:02:52.668 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.668 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.668 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:52.668 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:52.668 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:52.668 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:02:52.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.668 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:02:52.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:02:52.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:52.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:02:52.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:02:52.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:02:52.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:02:52.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:02:52.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:02:52.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:02:52.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:02:52.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:02:52.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:02:52.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:02:52.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:02:52.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:52.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:02:52.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:02:52.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:52.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:02:52.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:02:52.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:52.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:02:52.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:02:52.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:02:52.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:02:52.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:02:52.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:02:52.677 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.677 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.677 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.677 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:52.678 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:02:52.678 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.220607242, 2023-07-02T14:02:52.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:02:52.678 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.678 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.678 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:52.678 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:52.678 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:52.679 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:02:52.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.679 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:02:52.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:02:52.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:52.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:02:52.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:02:52.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:02:52.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:02:52.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:02:52.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:02:52.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:02:52.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:02:52.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:02:52.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:02:52.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:02:52.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:02:52.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:52.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:02:52.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:02:52.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:52.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:02:52.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:02:52.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:52.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:02:52.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:02:52.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:02:52.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:02:52.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:02:52.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:02:52.687 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.687 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.687 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.688 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:52.688 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:02:52.688 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.220457790999999, 2023-07-02T14:02:52.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:02:52.688 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.688 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.688 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:52.689 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:52.689 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:52.689 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:02:52.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.689 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:02:52.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:02:52.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:52.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:02:52.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:02:52.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:02:52.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:02:52.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:02:52.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:02:52.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:02:52.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:02:52.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:02:52.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:02:52.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:02:52.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:02:52.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:52.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:02:52.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:02:52.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:52.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:02:52.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:02:52.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:52.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:02:52.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:02:52.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:02:52.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:02:52.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:02:52.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:02:52.697 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.698 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.698 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.698 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:52.698 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:02:52.698 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.198981818, 2023-07-02T14:02:52.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:02:52.699 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.699 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.699 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:52.699 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:52.699 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:52.699 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:02:52.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.700 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:02:52.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:02:52.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:52.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:02:52.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:02:52.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:02:52.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:02:52.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:02:52.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:02:52.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:02:52.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:02:52.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:02:52.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:02:52.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:02:52.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:02:52.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:02:52.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:02:52.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:02:52.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:52.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:02:52.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:02:52.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:52.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:02:52.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:02:52.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:02:52.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:02:52.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:02:52.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:02:52.708 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.708 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.708 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.708 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:52.709 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:02:52.709 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.198849281999999, 2023-07-02T14:02:52.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:02:52.709 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.709 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.709 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:52.709 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:52.709 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:52.710 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:02:52.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.710 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:02:52.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:02:52.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:52.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:02:52.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:02:52.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:02:52.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:02:52.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:02:52.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:02:52.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:02:52.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:02:52.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:02:52.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:02:52.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:02:52.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:02:52.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:02:52.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:02:52.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:02:52.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:52.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:02:52.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:02:52.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:52.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:02:52.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:02:52.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:02:52.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:02:52.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:02:52.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:02:52.718 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.718 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.718 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.719 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:52.719 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:02:52.719 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.137512621000001, 2023-07-02T14:02:52.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:02:52.719 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:02:52.719 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:52.720 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:02:52.720 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:02:52.720 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:52.720 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:02:52.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.720 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:02:52.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:02:52.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:02:52.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:02:52.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:02:52.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:02:52.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:02:52.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:02:52.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:02:52.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:02:52.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:02:52.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:02:52.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:02:52.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:02:52.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:02:52.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:02:52.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:02:52.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:02:52.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:02:52.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:02:52.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:02:52.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:02:52.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:02:52.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:02:52.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:02:52.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:02:52.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:02:52.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:02:52.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:02:52.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:02:52.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:02:52.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:52.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:02:52.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:02:52.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:02:52.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:02:52.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:02:52.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:02:52.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:02:52.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:02:52.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:02:52.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:02:52.728 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.729 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.729 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.729 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:02:52.729 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:02:52.729 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:52.730 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:52.729+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:52.731 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:02:52.831 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:52.830+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:52.856 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:52.856 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:52.857 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:52.857 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:52.866 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:02:52.949 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:52.948+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:53.031 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:53.031 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:53.031 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:53.031 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:53.032 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:53.033 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:53.032+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:53.036 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:02:53.041 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:02:53.080 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:02:53.134 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:53.133+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:53.181 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:53.198 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:53.207 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:53.207 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:53.216 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:53.217 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:02:53.234 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:53.235 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:53.233+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:53.253 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:53.335 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:53.334+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:53.360 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:53.360 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:53.370 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:02:53.436 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:53.436+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:53.472 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:02:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:02:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:02:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:02:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:02:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:02:53.535 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:53.536 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:53.537 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:53.536+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:53.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:02:53.637 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:53.637+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:53.669 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:53.669 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:53.680 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:02:53.738 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:53.737+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:53.807 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:53.807 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:53.817 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:02:53.858 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:53.857+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:53.939 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:53.939+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:53.981 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:53.981 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:53.991 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:02:54.040 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:54.039+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:54.092 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap1 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap2 b3fbb028-e267-4d25-a750-93ba18a2316a@snap2.diff 2023-07-02T14:02:54.116 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:54.116 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:54.127 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:02:54.168 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:54.167+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:54.180 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 6% complete... Exporting image: 100% complete...done. 2023-07-02T14:02:54.183 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:02:54.183 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=3 2023-07-02T14:02:54.183 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap3 2023-07-02T14:02:54.241 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:54.241+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:54.263 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:54.263 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:54.274 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:02:54.342 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:54.341+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:54.438 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:54.438 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:54.442 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:54.441+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:54.448 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:02:54.543 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:54.542+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:54.612 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:54.613 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:54.624 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:02:54.665 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:54.664+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:54.745 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:54.744+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:54.788 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:54.788 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:54.798 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:02:54.846 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:54.845+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:54.947 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:54.945+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:54.961 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:54.961 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:54.972 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:02:55.047 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:55.046+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:55.093 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:02:55.097 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:55.098 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:55.099 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:02:55.103 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 3 2023-07-02T14:02:55.109 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:02:55.148 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:55.147+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:55.249 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:55.248+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:55.273 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:55.274 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:55.283 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:02:55.350 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:55.349+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:55.408 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:55.408 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:55.418 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:02:55.462 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:55.461+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:55.485 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:02:55.541 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:55.541 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:55.551 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:02:55.552 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:55.550+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:55.624 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:55.642 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:55.652 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:55.652+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:55.660 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:55.678 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:55.678 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:55.678 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:55.678 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:55.679 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:55.689 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:02:55.699 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:55.716 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:55.718 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:02:55.718 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:02:55.718 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:02:55.718 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:02:55.718 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:02:55.718 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:02:55.770 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:55.769+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:55.812 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:55.812 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:55.813 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:55.813 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:55.813 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:55.822 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:02:55.863 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:55.862+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:55.955 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:55.955+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:55.987 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:55.988 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:55.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:55.988 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:02:55.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:55.988 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:380 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'10, mlcod=40'1)", 2023-07-02T14:02:55.988 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.309796+0000", 2023-07-02T14:02:55.988 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.676574308999999, 2023-07-02T14:02:55.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034177320000000001, 2023-07-02T14:02:55.989 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:55.989 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:55.989 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:55.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:55.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:55.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309796+0000", 2023-07-02T14:02:55.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:55.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:55.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:55.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:55.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309796+0000", 2023-07-02T14:02:55.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:55.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:55.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:55.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:55.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309796+0000", 2023-07-02T14:02:55.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4694999999999997e-05 2023-07-02T14:02:55.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:55.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:55.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:55.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309851+0000", 2023-07-02T14:02:55.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6890000000000004e-06 2023-07-02T14:02:55.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:55.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:55.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:55.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309858+0000", 2023-07-02T14:02:55.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7659999999999998e-06 2023-07-02T14:02:55.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:55.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:55.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:55.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309863+0000", 2023-07-02T14:02:55.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001070164 2023-07-02T14:02:55.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:55.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:55.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:55.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310933+0000", 2023-07-02T14:02:55.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1287000000000001e-05 2023-07-02T14:02:55.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:55.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:55.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:55.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310954+0000", 2023-07-02T14:02:55.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022261059999999998 2023-07-02T14:02:55.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:55.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:55.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:55.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313181+0000", 2023-07-02T14:02:55.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3494000000000002e-05 2023-07-02T14:02:55.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:55.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:55.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:55.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313214+0000", 2023-07-02T14:02:55.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034177320000000001 2023-07-02T14:02:55.995 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:55.996 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:55.996 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:55.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:55.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:55.996 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:381 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'11, mlcod=40'1)", 2023-07-02T14:02:55.996 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.310792+0000", 2023-07-02T14:02:55.996 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.675578161000001, 2023-07-02T14:02:55.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038397629999999999, 2023-07-02T14:02:55.997 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:55.997 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:55.997 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:55.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:55.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:55.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310792+0000", 2023-07-02T14:02:55.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:55.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:55.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:55.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:55.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310792+0000", 2023-07-02T14:02:55.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:55.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:55.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:55.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:55.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310792+0000", 2023-07-02T14:02:55.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9630999999999999e-05 2023-07-02T14:02:55.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:55.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:55.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:55.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310851+0000", 2023-07-02T14:02:55.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3351e-05 2023-07-02T14:02:55.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:55.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310865+0000", 2023-07-02T14:02:56.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7459999999999995e-06 2023-07-02T14:02:56.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310874+0000", 2023-07-02T14:02:56.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000478457 2023-07-02T14:02:56.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311353+0000", 2023-07-02T14:02:56.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3498000000000001e-05 2023-07-02T14:02:56.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311376+0000", 2023-07-02T14:02:56.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003216464 2023-07-02T14:02:56.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314593+0000", 2023-07-02T14:02:56.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9368e-05 2023-07-02T14:02:56.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314632+0000", 2023-07-02T14:02:56.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038397629999999999 2023-07-02T14:02:56.003 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.003 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.003 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.004 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:382 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'12, mlcod=40'3)", 2023-07-02T14:02:56.004 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311736+0000", 2023-07-02T14:02:56.004 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.674634469000001, 2023-07-02T14:02:56.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029255800000000001, 2023-07-02T14:02:56.004 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.004 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.005 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311736+0000", 2023-07-02T14:02:56.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311736+0000", 2023-07-02T14:02:56.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:56.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311735+0000", 2023-07-02T14:02:56.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9895000000000005e-05 2023-07-02T14:02:56.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311835+0000", 2023-07-02T14:02:56.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2548000000000001e-05 2023-07-02T14:02:56.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311848+0000", 2023-07-02T14:02:56.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5810000000000002e-06 2023-07-02T14:02:56.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311854+0000", 2023-07-02T14:02:56.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4743999999999997e-05 2023-07-02T14:02:56.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311899+0000", 2023-07-02T14:02:56.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.808e-05 2023-07-02T14:02:56.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311917+0000", 2023-07-02T14:02:56.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027195069999999999 2023-07-02T14:02:56.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314637+0000", 2023-07-02T14:02:56.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5080999999999999e-05 2023-07-02T14:02:56.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314662+0000", 2023-07-02T14:02:56.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029255800000000001 2023-07-02T14:02:56.011 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.011 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.011 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.012 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:386 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'16, mlcod=40'10)", 2023-07-02T14:02:56.012 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.317468+0000", 2023-07-02T14:02:56.012 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.668902300999999, 2023-07-02T14:02:56.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027835989999999999, 2023-07-02T14:02:56.012 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.012 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.012 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317468+0000", 2023-07-02T14:02:56.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317468+0000", 2023-07-02T14:02:56.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:56.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317468+0000", 2023-07-02T14:02:56.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8161999999999998e-05 2023-07-02T14:02:56.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317506+0000", 2023-07-02T14:02:56.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5590000000000002e-06 2023-07-02T14:02:56.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317512+0000", 2023-07-02T14:02:56.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.186e-06 2023-07-02T14:02:56.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317518+0000", 2023-07-02T14:02:56.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4561999999999996e-05 2023-07-02T14:02:56.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317582+0000", 2023-07-02T14:02:56.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1291e-05 2023-07-02T14:02:56.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317603+0000", 2023-07-02T14:02:56.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026133839999999998 2023-07-02T14:02:56.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320217+0000", 2023-07-02T14:02:56.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5111999999999999e-05 2023-07-02T14:02:56.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320252+0000", 2023-07-02T14:02:56.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027835989999999999 2023-07-02T14:02:56.019 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.019 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.019 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.019 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:387 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'17, mlcod=40'10)", 2023-07-02T14:02:56.019 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.317892+0000", 2023-07-02T14:02:56.020 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.668478907000001, 2023-07-02T14:02:56.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031330540000000001, 2023-07-02T14:02:56.020 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.020 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.020 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317892+0000", 2023-07-02T14:02:56.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317892+0000", 2023-07-02T14:02:56.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:56.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317891+0000", 2023-07-02T14:02:56.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1758e-05 2023-07-02T14:02:56.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317963+0000", 2023-07-02T14:02:56.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5237e-05 2023-07-02T14:02:56.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317978+0000", 2023-07-02T14:02:56.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9710000000000001e-06 2023-07-02T14:02:56.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317984+0000", 2023-07-02T14:02:56.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4039999999999998e-05 2023-07-02T14:02:56.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318028+0000", 2023-07-02T14:02:56.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8092000000000001e-05 2023-07-02T14:02:56.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318056+0000", 2023-07-02T14:02:56.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029412959999999999 2023-07-02T14:02:56.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320997+0000", 2023-07-02T14:02:56.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7416999999999999e-05 2023-07-02T14:02:56.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321025+0000", 2023-07-02T14:02:56.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031330540000000001 2023-07-02T14:02:56.026 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.027 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.027 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.027 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:402 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'32, mlcod=40'17)", 2023-07-02T14:02:56.027 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.331582+0000", 2023-07-02T14:02:56.027 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.654789040000001, 2023-07-02T14:02:56.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002927172, 2023-07-02T14:02:56.028 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.028 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.028 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331582+0000", 2023-07-02T14:02:56.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331582+0000", 2023-07-02T14:02:56.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:56.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331581+0000", 2023-07-02T14:02:56.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2337999999999998e-05 2023-07-02T14:02:56.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331643+0000", 2023-07-02T14:02:56.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0506000000000001e-05 2023-07-02T14:02:56.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331653+0000", 2023-07-02T14:02:56.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5650000000000003e-06 2023-07-02T14:02:56.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331658+0000", 2023-07-02T14:02:56.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3337999999999998e-05 2023-07-02T14:02:56.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331691+0000", 2023-07-02T14:02:56.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4527e-05 2023-07-02T14:02:56.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331706+0000", 2023-07-02T14:02:56.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027586830000000001 2023-07-02T14:02:56.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334464+0000", 2023-07-02T14:02:56.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4311000000000003e-05 2023-07-02T14:02:56.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334509+0000", 2023-07-02T14:02:56.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002927172 2023-07-02T14:02:56.034 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.034 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.034 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.035 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:403 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'33, mlcod=40'20)", 2023-07-02T14:02:56.035 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.331850+0000", 2023-07-02T14:02:56.035 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.654520935000001, 2023-07-02T14:02:56.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033780419999999999, 2023-07-02T14:02:56.035 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.035 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.036 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331850+0000", 2023-07-02T14:02:56.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331850+0000", 2023-07-02T14:02:56.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:56.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331849+0000", 2023-07-02T14:02:56.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015039599999999999 2023-07-02T14:02:56.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.331999+0000", 2023-07-02T14:02:56.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7082e-05 2023-07-02T14:02:56.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.332017+0000", 2023-07-02T14:02:56.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3400000000000003e-06 2023-07-02T14:02:56.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.332023+0000", 2023-07-02T14:02:56.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8889999999999999e-05 2023-07-02T14:02:56.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.332102+0000", 2023-07-02T14:02:56.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2781000000000003e-05 2023-07-02T14:02:56.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.332145+0000", 2023-07-02T14:02:56.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030463410000000002 2023-07-02T14:02:56.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.335191+0000", 2023-07-02T14:02:56.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.684e-05 2023-07-02T14:02:56.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.335228+0000", 2023-07-02T14:02:56.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033780419999999999 2023-07-02T14:02:56.042 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.042 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.042 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.043 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:405 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'35, mlcod=40'25)", 2023-07-02T14:02:56.043 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.333577+0000", 2023-07-02T14:02:56.043 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.652793766999999, 2023-07-02T14:02:56.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002930219, 2023-07-02T14:02:56.043 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.043 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.043 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333577+0000", 2023-07-02T14:02:56.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333577+0000", 2023-07-02T14:02:56.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:56.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333576+0000", 2023-07-02T14:02:56.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4240000000000002e-05 2023-07-02T14:02:56.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333630+0000", 2023-07-02T14:02:56.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2962999999999999e-05 2023-07-02T14:02:56.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333643+0000", 2023-07-02T14:02:56.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8390000000000001e-06 2023-07-02T14:02:56.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333650+0000", 2023-07-02T14:02:56.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0926000000000005e-05 2023-07-02T14:02:56.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333721+0000", 2023-07-02T14:02:56.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2921e-05 2023-07-02T14:02:56.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.333744+0000", 2023-07-02T14:02:56.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027376240000000001 2023-07-02T14:02:56.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336481+0000", 2023-07-02T14:02:56.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.565e-05 2023-07-02T14:02:56.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336507+0000", 2023-07-02T14:02:56.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002930219 2023-07-02T14:02:56.050 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.050 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.050 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.051 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:406 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'36, mlcod=40'25)", 2023-07-02T14:02:56.051 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.334125+0000", 2023-07-02T14:02:56.051 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.652246002999998, 2023-07-02T14:02:56.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003294352, 2023-07-02T14:02:56.051 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.051 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.051 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334125+0000", 2023-07-02T14:02:56.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334125+0000", 2023-07-02T14:02:56.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:56.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334124+0000", 2023-07-02T14:02:56.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106912 2023-07-02T14:02:56.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334231+0000", 2023-07-02T14:02:56.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6602000000000001e-05 2023-07-02T14:02:56.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334247+0000", 2023-07-02T14:02:56.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7350000000000001e-06 2023-07-02T14:02:56.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334253+0000", 2023-07-02T14:02:56.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000473144 2023-07-02T14:02:56.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334726+0000", 2023-07-02T14:02:56.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4657000000000001e-05 2023-07-02T14:02:56.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.334751+0000", 2023-07-02T14:02:56.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026279250000000001 2023-07-02T14:02:56.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337379+0000", 2023-07-02T14:02:56.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0111999999999998e-05 2023-07-02T14:02:56.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337419+0000", 2023-07-02T14:02:56.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003294352 2023-07-02T14:02:56.058 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.058 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.058 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.058 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:426 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'42, mlcod=40'38)", 2023-07-02T14:02:56.058 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.340864+0000", 2023-07-02T14:02:56.059 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.645506682000001, 2023-07-02T14:02:56.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065109469999999996, 2023-07-02T14:02:56.059 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.059 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.059 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.340864+0000", 2023-07-02T14:02:56.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.340864+0000", 2023-07-02T14:02:56.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:56.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.340863+0000", 2023-07-02T14:02:56.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019354159999999999 2023-07-02T14:02:56.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342799+0000", 2023-07-02T14:02:56.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5824e-05 2023-07-02T14:02:56.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342814+0000", 2023-07-02T14:02:56.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:02:56.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342822+0000", 2023-07-02T14:02:56.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7465999999999994e-05 2023-07-02T14:02:56.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342889+0000", 2023-07-02T14:02:56.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5068000000000001e-05 2023-07-02T14:02:56.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342914+0000", 2023-07-02T14:02:56.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044200969999999996 2023-07-02T14:02:56.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347334+0000", 2023-07-02T14:02:56.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0711999999999999e-05 2023-07-02T14:02:56.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347375+0000", 2023-07-02T14:02:56.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065109469999999996 2023-07-02T14:02:56.065 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.066 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.066 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.066 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:431 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'4, mlcod=40'2)", 2023-07-02T14:02:56.066 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.350267+0000", 2023-07-02T14:02:56.066 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.636103215999999, 2023-07-02T14:02:56.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077388420000000001, 2023-07-02T14:02:56.067 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.067 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.067 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350267+0000", 2023-07-02T14:02:56.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350267+0000", 2023-07-02T14:02:56.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:56.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350266+0000", 2023-07-02T14:02:56.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024191569999999999 2023-07-02T14:02:56.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352685+0000", 2023-07-02T14:02:56.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0013e-05 2023-07-02T14:02:56.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352706+0000", 2023-07-02T14:02:56.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3250000000000008e-06 2023-07-02T14:02:56.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352714+0000", 2023-07-02T14:02:56.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9648000000000003e-05 2023-07-02T14:02:56.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352763+0000", 2023-07-02T14:02:56.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4496999999999999e-05 2023-07-02T14:02:56.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352788+0000", 2023-07-02T14:02:56.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051724880000000003 2023-07-02T14:02:56.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357960+0000", 2023-07-02T14:02:56.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5802e-05 2023-07-02T14:02:56.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358006+0000", 2023-07-02T14:02:56.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077388420000000001 2023-07-02T14:02:56.073 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.073 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.073 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.074 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:433 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'5, mlcod=40'2)", 2023-07-02T14:02:56.074 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.352729+0000", 2023-07-02T14:02:56.074 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.633641212000001, 2023-07-02T14:02:56.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0053048619999999996, 2023-07-02T14:02:56.074 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.074 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.075 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352729+0000", 2023-07-02T14:02:56.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352729+0000", 2023-07-02T14:02:56.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:56.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352728+0000", 2023-07-02T14:02:56.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4173e-05 2023-07-02T14:02:56.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352763+0000", 2023-07-02T14:02:56.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7689999999999996e-06 2023-07-02T14:02:56.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352770+0000", 2023-07-02T14:02:56.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6689999999999997e-06 2023-07-02T14:02:56.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352775+0000", 2023-07-02T14:02:56.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025486380000000002 2023-07-02T14:02:56.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.355324+0000", 2023-07-02T14:02:56.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7258999999999999e-05 2023-07-02T14:02:56.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.355341+0000", 2023-07-02T14:02:56.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026703569999999999 2023-07-02T14:02:56.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358011+0000", 2023-07-02T14:02:56.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3039000000000001e-05 2023-07-02T14:02:56.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358034+0000", 2023-07-02T14:02:56.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0053048619999999996 2023-07-02T14:02:56.081 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.081 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.081 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.082 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:439 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'6, mlcod=40'4)", 2023-07-02T14:02:56.082 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.359868+0000", 2023-07-02T14:02:56.082 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.626502909999999, 2023-07-02T14:02:56.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031584299999999998, 2023-07-02T14:02:56.082 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.082 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.082 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.359868+0000", 2023-07-02T14:02:56.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.359868+0000", 2023-07-02T14:02:56.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:02:56.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.359866+0000", 2023-07-02T14:02:56.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00068265100000000005 2023-07-02T14:02:56.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.360549+0000", 2023-07-02T14:02:56.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9236999999999999e-05 2023-07-02T14:02:56.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.360568+0000", 2023-07-02T14:02:56.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5529999999999998e-06 2023-07-02T14:02:56.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.360578+0000", 2023-07-02T14:02:56.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6258000000000003e-05 2023-07-02T14:02:56.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.360624+0000", 2023-07-02T14:02:56.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6570000000000001e-05 2023-07-02T14:02:56.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.360651+0000", 2023-07-02T14:02:56.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023451359999999998 2023-07-02T14:02:56.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.362996+0000", 2023-07-02T14:02:56.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0467000000000002e-05 2023-07-02T14:02:56.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363026+0000", 2023-07-02T14:02:56.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031584299999999998 2023-07-02T14:02:56.089 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.089 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.089 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.090 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:440 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'7, mlcod=40'4)", 2023-07-02T14:02:56.090 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.361026+0000", 2023-07-02T14:02:56.090 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.625344439999999, 2023-07-02T14:02:56.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029947889999999999, 2023-07-02T14:02:56.090 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.090 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.090 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361026+0000", 2023-07-02T14:02:56.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361026+0000", 2023-07-02T14:02:56.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:56.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361025+0000", 2023-07-02T14:02:56.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013836740000000001 2023-07-02T14:02:56.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.362409+0000", 2023-07-02T14:02:56.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5155e-05 2023-07-02T14:02:56.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.362424+0000", 2023-07-02T14:02:56.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1170000000000004e-06 2023-07-02T14:02:56.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.362432+0000", 2023-07-02T14:02:56.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9370999999999999e-05 2023-07-02T14:02:56.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.362482+0000", 2023-07-02T14:02:56.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5891999999999999e-05 2023-07-02T14:02:56.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.362508+0000", 2023-07-02T14:02:56.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014803920000000001 2023-07-02T14:02:56.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363988+0000", 2023-07-02T14:02:56.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2923999999999999e-05 2023-07-02T14:02:56.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364021+0000", 2023-07-02T14:02:56.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029947889999999999 2023-07-02T14:02:56.097 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.097 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.097 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.097 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:451 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'8, mlcod=40'6)", 2023-07-02T14:02:56.097 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.365605+0000", 2023-07-02T14:02:56.098 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.620765475999999, 2023-07-02T14:02:56.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033622349999999999, 2023-07-02T14:02:56.098 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.098 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.098 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365605+0000", 2023-07-02T14:02:56.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365605+0000", 2023-07-02T14:02:56.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:02:56.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365604+0000", 2023-07-02T14:02:56.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00056647700000000002 2023-07-02T14:02:56.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.366170+0000", 2023-07-02T14:02:56.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2206000000000001e-05 2023-07-02T14:02:56.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.366192+0000", 2023-07-02T14:02:56.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1044e-05 2023-07-02T14:02:56.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.366203+0000", 2023-07-02T14:02:56.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2663000000000005e-05 2023-07-02T14:02:56.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.366276+0000", 2023-07-02T14:02:56.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7101999999999999e-05 2023-07-02T14:02:56.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.366303+0000", 2023-07-02T14:02:56.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026248769999999999 2023-07-02T14:02:56.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368928+0000", 2023-07-02T14:02:56.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9267000000000002e-05 2023-07-02T14:02:56.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368967+0000", 2023-07-02T14:02:56.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033622349999999999 2023-07-02T14:02:56.104 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.105 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.105 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.105 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:455 2.5 e40/14 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:head v 40'12, mlcod=40'10)", 2023-07-02T14:02:56.105 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.376227+0000", 2023-07-02T14:02:56.105 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.610143938, 2023-07-02T14:02:56.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012681951, 2023-07-02T14:02:56.106 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.106 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.106 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.376227+0000", 2023-07-02T14:02:56.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.376227+0000", 2023-07-02T14:02:56.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:56.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.376226+0000", 2023-07-02T14:02:56.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040198860000000003 2023-07-02T14:02:56.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380245+0000", 2023-07-02T14:02:56.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7640000000000001e-05 2023-07-02T14:02:56.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380263+0000", 2023-07-02T14:02:56.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3299999999999999e-06 2023-07-02T14:02:56.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380271+0000", 2023-07-02T14:02:56.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4322e-05 2023-07-02T14:02:56.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380316+0000", 2023-07-02T14:02:56.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7682000000000001e-05 2023-07-02T14:02:56.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380333+0000", 2023-07-02T14:02:56.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085223439999999994 2023-07-02T14:02:56.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.388856+0000", 2023-07-02T14:02:56.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2930999999999999e-05 2023-07-02T14:02:56.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.388909+0000", 2023-07-02T14:02:56.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012681951 2023-07-02T14:02:56.112 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.112 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.112 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.113 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:456 2.5 e40/14 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:head v 40'13, mlcod=40'10)", 2023-07-02T14:02:56.113 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.382266+0000", 2023-07-02T14:02:56.113 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.604104311, 2023-07-02T14:02:56.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075568149999999997, 2023-07-02T14:02:56.113 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.114 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.114 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382266+0000", 2023-07-02T14:02:56.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382266+0000", 2023-07-02T14:02:56.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:02:56.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382264+0000", 2023-07-02T14:02:56.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012720599999999999 2023-07-02T14:02:56.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382391+0000", 2023-07-02T14:02:56.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7598e-05 2023-07-02T14:02:56.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382409+0000", 2023-07-02T14:02:56.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8660000000000001e-06 2023-07-02T14:02:56.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382419+0000", 2023-07-02T14:02:56.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001837403 2023-07-02T14:02:56.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384256+0000", 2023-07-02T14:02:56.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5757000000000001e-05 2023-07-02T14:02:56.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384282+0000", 2023-07-02T14:02:56.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055044730000000002 2023-07-02T14:02:56.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389787+0000", 2023-07-02T14:02:56.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6622000000000003e-05 2023-07-02T14:02:56.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389823+0000", 2023-07-02T14:02:56.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075568149999999997 2023-07-02T14:02:56.120 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.120 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.120 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.121 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:457 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'14, mlcod=40'10)", 2023-07-02T14:02:56.121 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.382901+0000", 2023-07-02T14:02:56.121 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.603469339, 2023-07-02T14:02:56.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0069428320000000003, 2023-07-02T14:02:56.121 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.121 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.121 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382901+0000", 2023-07-02T14:02:56.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382901+0000", 2023-07-02T14:02:56.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:02:56.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382900+0000", 2023-07-02T14:02:56.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7738999999999994e-05 2023-07-02T14:02:56.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382998+0000", 2023-07-02T14:02:56.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6769000000000002e-05 2023-07-02T14:02:56.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383014+0000", 2023-07-02T14:02:56.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0073e-05 2023-07-02T14:02:56.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383025+0000", 2023-07-02T14:02:56.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016687410000000001 2023-07-02T14:02:56.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384693+0000", 2023-07-02T14:02:56.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4108999999999999e-05 2023-07-02T14:02:56.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384717+0000", 2023-07-02T14:02:56.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051086830000000001 2023-07-02T14:02:56.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389826+0000", 2023-07-02T14:02:56.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8059000000000001e-05 2023-07-02T14:02:56.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389844+0000", 2023-07-02T14:02:56.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0069428320000000003 2023-07-02T14:02:56.128 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.128 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.128 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.128 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:479 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'15, mlcod=40'10)", 2023-07-02T14:02:56.129 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.383348+0000", 2023-07-02T14:02:56.129 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.603022380999999, 2023-07-02T14:02:56.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065068890000000001, 2023-07-02T14:02:56.129 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.129 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.129 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383348+0000", 2023-07-02T14:02:56.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383348+0000", 2023-07-02T14:02:56.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:56.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383347+0000", 2023-07-02T14:02:56.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000422123 2023-07-02T14:02:56.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383769+0000", 2023-07-02T14:02:56.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1688e-05 2023-07-02T14:02:56.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383791+0000", 2023-07-02T14:02:56.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0237e-05 2023-07-02T14:02:56.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383801+0000", 2023-07-02T14:02:56.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013799940000000001 2023-07-02T14:02:56.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.385181+0000", 2023-07-02T14:02:56.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3047000000000001e-05 2023-07-02T14:02:56.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.385204+0000", 2023-07-02T14:02:56.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046417339999999998 2023-07-02T14:02:56.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389846+0000", 2023-07-02T14:02:56.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1800000000000002e-06 2023-07-02T14:02:56.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389855+0000", 2023-07-02T14:02:56.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065068890000000001 2023-07-02T14:02:56.136 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.136 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.136 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.136 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:484 2.5 e40/14 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:head v 40'16, mlcod=40'10)", 2023-07-02T14:02:56.136 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.384371+0000", 2023-07-02T14:02:56.137 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 16.601999318000001, 2023-07-02T14:02:56.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054937609999999998, 2023-07-02T14:02:56.137 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.137 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.137 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384371+0000", 2023-07-02T14:02:56.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384371+0000", 2023-07-02T14:02:56.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:56.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384370+0000", 2023-07-02T14:02:56.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00041928199999999999 2023-07-02T14:02:56.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384790+0000", 2023-07-02T14:02:56.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1742999999999999e-05 2023-07-02T14:02:56.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384801+0000", 2023-07-02T14:02:56.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0380000000000002e-06 2023-07-02T14:02:56.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384808+0000", 2023-07-02T14:02:56.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010660999999999999 2023-07-02T14:02:56.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.385874+0000", 2023-07-02T14:02:56.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3745000000000003e-05 2023-07-02T14:02:56.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.385908+0000", 2023-07-02T14:02:56.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039487560000000003 2023-07-02T14:02:56.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389857+0000", 2023-07-02T14:02:56.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1019999999999997e-06 2023-07-02T14:02:56.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389865+0000", 2023-07-02T14:02:56.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054937609999999998 2023-07-02T14:02:56.143 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.144 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.144 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.144 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.144 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.144 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:56.145 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:02:56.146 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:02:56.146 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:56.144+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:56.246 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:56.245+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:56.310 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:56.310 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:56.310 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:56.310 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:56.320 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:02:56.346 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:56.346+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:56.447 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:56.446+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:56.465 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2023-07-02T14:02:56.465 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:02:56.466 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2023-07-02T14:02:56.466 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-07-02T14:02:56.485 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:56.485 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:56.485 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:56.485 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:56.485 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:56.496 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:02:56.548 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:56.548+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:56.649 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:56.648+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:56.665 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:56.665 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:56.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:56.665 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:02:56.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.666 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:250 2.2 2:44c63967:::rbd_data.125de1449605.0000000000000101:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:56.666 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.299603+0000", 2023-07-02T14:02:56.666 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.363578932999999, 2023-07-02T14:02:56.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037580609000000001, 2023-07-02T14:02:56.666 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.666 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.667 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:56.667 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:56.667 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:56.667 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 250 2023-07-02T14:02:56.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.667 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.299603+0000", 2023-07-02T14:02:56.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.299603+0000", 2023-07-02T14:02:56.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:56.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.299602+0000", 2023-07-02T14:02:56.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022150699999999999 2023-07-02T14:02:56.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.299824+0000", 2023-07-02T14:02:56.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1405e-05 2023-07-02T14:02:56.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.299835+0000", 2023-07-02T14:02:56.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1590000000000004e-06 2023-07-02T14:02:56.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.299842+0000", 2023-07-02T14:02:56.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017232279999999999 2023-07-02T14:02:56.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317074+0000", 2023-07-02T14:02:56.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5123999999999994e-05 2023-07-02T14:02:56.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317150+0000", 2023-07-02T14:02:56.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014873800000000001 2023-07-02T14:02:56.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:56.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317298+0000", 2023-07-02T14:02:56.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035949480000000002 2023-07-02T14:02:56.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:56.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320893+0000", 2023-07-02T14:02:56.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016194844999999999 2023-07-02T14:02:56.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:56.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337088+0000", 2023-07-02T14:02:56.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1582999999999997e-05 2023-07-02T14:02:56.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337140+0000", 2023-07-02T14:02:56.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3779000000000002e-05 2023-07-02T14:02:56.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337183+0000", 2023-07-02T14:02:56.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037580609000000001 2023-07-02T14:02:56.676 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.676 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.676 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.677 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:256 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:56.677 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.300278+0000", 2023-07-02T14:02:56.677 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.362903510999999, 2023-07-02T14:02:56.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037365890999999998, 2023-07-02T14:02:56.677 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.677 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.677 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:56.678 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:56.678 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:56.678 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 256 2023-07-02T14:02:56.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.678 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300278+0000", 2023-07-02T14:02:56.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300278+0000", 2023-07-02T14:02:56.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:56.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300278+0000", 2023-07-02T14:02:56.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00066792800000000003 2023-07-02T14:02:56.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300945+0000", 2023-07-02T14:02:56.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0150000000000003e-06 2023-07-02T14:02:56.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300954+0000", 2023-07-02T14:02:56.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9069999999999998e-06 2023-07-02T14:02:56.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300958+0000", 2023-07-02T14:02:56.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018796822000000001 2023-07-02T14:02:56.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319755+0000", 2023-07-02T14:02:56.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126038 2023-07-02T14:02:56.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319881+0000", 2023-07-02T14:02:56.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015656700000000001 2023-07-02T14:02:56.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:56.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320038+0000", 2023-07-02T14:02:56.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017229343000000001 2023-07-02T14:02:56.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:56.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337267+0000", 2023-07-02T14:02:56.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000277378 2023-07-02T14:02:56.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:56.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337545+0000", 2023-07-02T14:02:56.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4978000000000001e-05 2023-07-02T14:02:56.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337600+0000", 2023-07-02T14:02:56.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4635000000000002e-05 2023-07-02T14:02:56.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.337644+0000", 2023-07-02T14:02:56.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037365890999999998 2023-07-02T14:02:56.687 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.687 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.687 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.688 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:279 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:56.688 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.310714+0000", 2023-07-02T14:02:56.688 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.352468046999999, 2023-07-02T14:02:56.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038119238, 2023-07-02T14:02:56.688 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.688 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.688 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:56.689 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:56.689 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:56.689 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 279 2023-07-02T14:02:56.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.689 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310714+0000", 2023-07-02T14:02:56.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310714+0000", 2023-07-02T14:02:56.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:56.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310713+0000", 2023-07-02T14:02:56.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8973e-05 2023-07-02T14:02:56.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310762+0000", 2023-07-02T14:02:56.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0530000000000007e-06 2023-07-02T14:02:56.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310770+0000", 2023-07-02T14:02:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7569999999999993e-06 2023-07-02T14:02:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310779+0000", 2023-07-02T14:02:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033655263999999997 2023-07-02T14:02:56.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.344434+0000", 2023-07-02T14:02:56.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4365999999999999e-05 2023-07-02T14:02:56.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.344508+0000", 2023-07-02T14:02:56.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016449100000000001 2023-07-02T14:02:56.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:56.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.344673+0000", 2023-07-02T14:02:56.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019402379999999999 2023-07-02T14:02:56.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:56.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346613+0000", 2023-07-02T14:02:56.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021203900000000002 2023-07-02T14:02:56.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:56.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348734+0000", 2023-07-02T14:02:56.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5225000000000001e-05 2023-07-02T14:02:56.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348789+0000", 2023-07-02T14:02:56.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4163999999999999e-05 2023-07-02T14:02:56.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348833+0000", 2023-07-02T14:02:56.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038119238 2023-07-02T14:02:56.699 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.699 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.699 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.699 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:280 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:56.699 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.310785+0000", 2023-07-02T14:02:56.700 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.352397174, 2023-07-02T14:02:56.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038149954999999999, 2023-07-02T14:02:56.700 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.700 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.700 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:56.701 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:56.701 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:56.701 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 280 2023-07-02T14:02:56.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.701 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310785+0000", 2023-07-02T14:02:56.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310785+0000", 2023-07-02T14:02:56.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:56.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310784+0000", 2023-07-02T14:02:56.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4364999999999999e-05 2023-07-02T14:02:56.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310838+0000", 2023-07-02T14:02:56.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4619999999999996e-06 2023-07-02T14:02:56.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310847+0000", 2023-07-02T14:02:56.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5129999999999998e-06 2023-07-02T14:02:56.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310851+0000", 2023-07-02T14:02:56.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034215981999999999 2023-07-02T14:02:56.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.345067+0000", 2023-07-02T14:02:56.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6117000000000002e-05 2023-07-02T14:02:56.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.345153+0000", 2023-07-02T14:02:56.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014067499999999999 2023-07-02T14:02:56.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:56.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.345294+0000", 2023-07-02T14:02:56.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013560810000000001 2023-07-02T14:02:56.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:56.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346650+0000", 2023-07-02T14:02:56.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002196239 2023-07-02T14:02:56.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:56.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348846+0000", 2023-07-02T14:02:56.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7621999999999999e-05 2023-07-02T14:02:56.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348894+0000", 2023-07-02T14:02:56.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0518000000000001e-05 2023-07-02T14:02:56.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348935+0000", 2023-07-02T14:02:56.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038149954999999999 2023-07-02T14:02:56.710 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.710 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.710 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.711 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:281 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:56.711 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.310857+0000", 2023-07-02T14:02:56.711 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.352324912, 2023-07-02T14:02:56.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039347695000000002, 2023-07-02T14:02:56.711 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.712 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.712 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:56.712 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:56.712 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:56.712 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:02:56.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.712 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310857+0000", 2023-07-02T14:02:56.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310857+0000", 2023-07-02T14:02:56.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:56.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310856+0000", 2023-07-02T14:02:56.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00081158100000000002 2023-07-02T14:02:56.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311668+0000", 2023-07-02T14:02:56.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.257e-05 2023-07-02T14:02:56.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311680+0000", 2023-07-02T14:02:56.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1110000000000002e-06 2023-07-02T14:02:56.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311688+0000", 2023-07-02T14:02:56.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034985653999999998 2023-07-02T14:02:56.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346673+0000", 2023-07-02T14:02:56.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7062999999999999e-05 2023-07-02T14:02:56.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346730+0000", 2023-07-02T14:02:56.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000151061 2023-07-02T14:02:56.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:56.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346881+0000", 2023-07-02T14:02:56.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018356119999999999 2023-07-02T14:02:56.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:56.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348717+0000", 2023-07-02T14:02:56.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001417385 2023-07-02T14:02:56.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:56.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350134+0000", 2023-07-02T14:02:56.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7036999999999997e-05 2023-07-02T14:02:56.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350181+0000", 2023-07-02T14:02:56.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3187999999999999e-05 2023-07-02T14:02:56.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350205+0000", 2023-07-02T14:02:56.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039347695000000002 2023-07-02T14:02:56.721 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.722 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.722 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.722 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:282 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:56.722 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311695+0000", 2023-07-02T14:02:56.722 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.351486418, 2023-07-02T14:02:56.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038558453999999999, 2023-07-02T14:02:56.723 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.723 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.723 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:56.723 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:56.723 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:56.723 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 282 2023-07-02T14:02:56.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.724 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311695+0000", 2023-07-02T14:02:56.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311695+0000", 2023-07-02T14:02:56.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:56.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311695+0000", 2023-07-02T14:02:56.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6679999999999999e-05 2023-07-02T14:02:56.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311751+0000", 2023-07-02T14:02:56.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9199999999999993e-06 2023-07-02T14:02:56.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311760+0000", 2023-07-02T14:02:56.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2480000000000004e-06 2023-07-02T14:02:56.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311765+0000", 2023-07-02T14:02:56.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035670150999999997 2023-07-02T14:02:56.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347436+0000", 2023-07-02T14:02:56.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7861000000000006e-05 2023-07-02T14:02:56.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347503+0000", 2023-07-02T14:02:56.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000217443 2023-07-02T14:02:56.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:56.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347721+0000", 2023-07-02T14:02:56.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023992449999999999 2023-07-02T14:02:56.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:56.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350120+0000", 2023-07-02T14:02:56.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2219000000000006e-05 2023-07-02T14:02:56.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:56.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350212+0000", 2023-07-02T14:02:56.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3430000000000001e-05 2023-07-02T14:02:56.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350236+0000", 2023-07-02T14:02:56.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8074999999999999e-05 2023-07-02T14:02:56.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350254+0000", 2023-07-02T14:02:56.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038558453999999999 2023-07-02T14:02:56.733 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.733 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.733 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.733 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:283 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:56.733 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311771+0000", 2023-07-02T14:02:56.733 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.351410547, 2023-07-02T14:02:56.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039285701999999999, 2023-07-02T14:02:56.734 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.734 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.734 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:56.734 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:56.734 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:56.734 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 283 2023-07-02T14:02:56.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.735 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311771+0000", 2023-07-02T14:02:56.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311771+0000", 2023-07-02T14:02:56.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:56.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311771+0000", 2023-07-02T14:02:56.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6775999999999999e-05 2023-07-02T14:02:56.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311837+0000", 2023-07-02T14:02:56.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4290000000000001e-06 2023-07-02T14:02:56.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311846+0000", 2023-07-02T14:02:56.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3800000000000002e-06 2023-07-02T14:02:56.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311851+0000", 2023-07-02T14:02:56.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036308287000000002 2023-07-02T14:02:56.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348160+0000", 2023-07-02T14:02:56.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6475000000000002e-05 2023-07-02T14:02:56.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348236+0000", 2023-07-02T14:02:56.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000147966 2023-07-02T14:02:56.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:56.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348384+0000", 2023-07-02T14:02:56.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019115569999999999 2023-07-02T14:02:56.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:56.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350296+0000", 2023-07-02T14:02:56.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00063248000000000004 2023-07-02T14:02:56.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:56.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350928+0000", 2023-07-02T14:02:56.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3355000000000003e-05 2023-07-02T14:02:56.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351021+0000", 2023-07-02T14:02:56.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5596000000000003e-05 2023-07-02T14:02:56.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351057+0000", 2023-07-02T14:02:56.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039285701999999999 2023-07-02T14:02:56.743 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.743 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.744 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:284 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:56.744 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311859+0000", 2023-07-02T14:02:56.744 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.35132325, 2023-07-02T14:02:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039967282999999999, 2023-07-02T14:02:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 284 2023-07-02T14:02:56.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.746 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311859+0000", 2023-07-02T14:02:56.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311859+0000", 2023-07-02T14:02:56.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:56.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311858+0000", 2023-07-02T14:02:56.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3902999999999997e-05 2023-07-02T14:02:56.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311892+0000", 2023-07-02T14:02:56.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0989999999999999e-06 2023-07-02T14:02:56.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311900+0000", 2023-07-02T14:02:56.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9030000000000003e-06 2023-07-02T14:02:56.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311905+0000", 2023-07-02T14:02:56.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037060899000000001 2023-07-02T14:02:56.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348966+0000", 2023-07-02T14:02:56.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3609000000000002e-05 2023-07-02T14:02:56.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349009+0000", 2023-07-02T14:02:56.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6831999999999998e-05 2023-07-02T14:02:56.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:56.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349096+0000", 2023-07-02T14:02:56.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001975077 2023-07-02T14:02:56.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:56.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351071+0000", 2023-07-02T14:02:56.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00066166900000000004 2023-07-02T14:02:56.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:56.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351733+0000", 2023-07-02T14:02:56.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3199999999999999e-05 2023-07-02T14:02:56.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351786+0000", 2023-07-02T14:02:56.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9823999999999997e-05 2023-07-02T14:02:56.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351826+0000", 2023-07-02T14:02:56.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039967282999999999 2023-07-02T14:02:56.754 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.755 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.755 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.755 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:285 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:56.755 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311910+0000", 2023-07-02T14:02:56.755 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.351272217999998, 2023-07-02T14:02:56.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040060302999999998, 2023-07-02T14:02:56.756 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.756 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.756 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:56.756 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:56.756 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:56.757 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 285 2023-07-02T14:02:56.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.757 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311910+0000", 2023-07-02T14:02:56.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311910+0000", 2023-07-02T14:02:56.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:56.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311909+0000", 2023-07-02T14:02:56.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1025999999999997e-05 2023-07-02T14:02:56.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311940+0000", 2023-07-02T14:02:56.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4769999999999997e-06 2023-07-02T14:02:56.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311948+0000", 2023-07-02T14:02:56.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5059999999999997e-06 2023-07-02T14:02:56.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311952+0000", 2023-07-02T14:02:56.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037480332999999998 2023-07-02T14:02:56.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349432+0000", 2023-07-02T14:02:56.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2923000000000005e-05 2023-07-02T14:02:56.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349515+0000", 2023-07-02T14:02:56.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015231700000000001 2023-07-02T14:02:56.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:56.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349668+0000", 2023-07-02T14:02:56.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002171849 2023-07-02T14:02:56.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:56.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351839+0000", 2023-07-02T14:02:56.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2052000000000003e-05 2023-07-02T14:02:56.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:56.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351882+0000", 2023-07-02T14:02:56.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7683999999999999e-05 2023-07-02T14:02:56.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351929+0000", 2023-07-02T14:02:56.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0648000000000001e-05 2023-07-02T14:02:56.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351970+0000", 2023-07-02T14:02:56.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040060302999999998 2023-07-02T14:02:56.766 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.766 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.767 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.767 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:286 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:56.767 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311957+0000", 2023-07-02T14:02:56.767 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.351224350999999, 2023-07-02T14:02:56.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042429263000000002, 2023-07-02T14:02:56.767 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.768 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.768 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:56.768 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:56.768 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:56.768 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 286 2023-07-02T14:02:56.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.768 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311957+0000", 2023-07-02T14:02:56.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311957+0000", 2023-07-02T14:02:56.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:56.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311957+0000", 2023-07-02T14:02:56.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7862000000000003e-05 2023-07-02T14:02:56.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312015+0000", 2023-07-02T14:02:56.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2770000000000005e-06 2023-07-02T14:02:56.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312023+0000", 2023-07-02T14:02:56.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0000000000000004e-06 2023-07-02T14:02:56.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312028+0000", 2023-07-02T14:02:56.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038291327999999999 2023-07-02T14:02:56.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350319+0000", 2023-07-02T14:02:56.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3686000000000001e-05 2023-07-02T14:02:56.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350373+0000", 2023-07-02T14:02:56.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012420000000000001 2023-07-02T14:02:56.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:56.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350497+0000", 2023-07-02T14:02:56.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014893669999999999 2023-07-02T14:02:56.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:56.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351986+0000", 2023-07-02T14:02:56.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022881580000000002 2023-07-02T14:02:56.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:56.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354275+0000", 2023-07-02T14:02:56.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7388e-05 2023-07-02T14:02:56.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354332+0000", 2023-07-02T14:02:56.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4688999999999998e-05 2023-07-02T14:02:56.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354387+0000", 2023-07-02T14:02:56.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042429263000000002 2023-07-02T14:02:56.779 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.779 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.779 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.779 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:287 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 589824~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:56.780 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312033+0000", 2023-07-02T14:02:56.780 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.351148732999999, 2023-07-02T14:02:56.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042515431999999999, 2023-07-02T14:02:56.780 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.780 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.780 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:56.780 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:56.781 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:56.781 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 287 2023-07-02T14:02:56.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.781 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312033+0000", 2023-07-02T14:02:56.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312033+0000", 2023-07-02T14:02:56.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:56.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312033+0000", 2023-07-02T14:02:56.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1540999999999999e-05 2023-07-02T14:02:56.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312094+0000", 2023-07-02T14:02:56.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.123e-06 2023-07-02T14:02:56.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312102+0000", 2023-07-02T14:02:56.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5480000000000003e-06 2023-07-02T14:02:56.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312107+0000", 2023-07-02T14:02:56.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039019356999999998 2023-07-02T14:02:56.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351126+0000", 2023-07-02T14:02:56.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100107 2023-07-02T14:02:56.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351226+0000", 2023-07-02T14:02:56.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177455 2023-07-02T14:02:56.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:56.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351404+0000", 2023-07-02T14:02:56.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030030819999999998 2023-07-02T14:02:56.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:56.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354407+0000", 2023-07-02T14:02:56.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5787000000000001e-05 2023-07-02T14:02:56.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:56.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354443+0000", 2023-07-02T14:02:56.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0627000000000001e-05 2023-07-02T14:02:56.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354503+0000", 2023-07-02T14:02:56.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5327000000000002e-05 2023-07-02T14:02:56.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354548+0000", 2023-07-02T14:02:56.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042515431999999999 2023-07-02T14:02:56.790 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.790 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.790 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.790 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:288 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:56.791 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312112+0000", 2023-07-02T14:02:56.791 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.35106978, 2023-07-02T14:02:56.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051367833000000002, 2023-07-02T14:02:56.791 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.791 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.791 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:56.791 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:56.791 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:56.792 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 288 2023-07-02T14:02:56.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.792 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312112+0000", 2023-07-02T14:02:56.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312112+0000", 2023-07-02T14:02:56.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:56.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312111+0000", 2023-07-02T14:02:56.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8513999999999998e-05 2023-07-02T14:02:56.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312150+0000", 2023-07-02T14:02:56.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6499999999999996e-06 2023-07-02T14:02:56.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312157+0000", 2023-07-02T14:02:56.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7799999999999997e-06 2023-07-02T14:02:56.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312163+0000", 2023-07-02T14:02:56.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039867382 2023-07-02T14:02:56.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352031+0000", 2023-07-02T14:02:56.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6370000000000002e-05 2023-07-02T14:02:56.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352107+0000", 2023-07-02T14:02:56.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000156021 2023-07-02T14:02:56.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:56.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352263+0000", 2023-07-02T14:02:56.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023521890000000002 2023-07-02T14:02:56.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:56.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354615+0000", 2023-07-02T14:02:56.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087542309999999995 2023-07-02T14:02:56.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:56.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363369+0000", 2023-07-02T14:02:56.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3217999999999998e-05 2023-07-02T14:02:56.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363423+0000", 2023-07-02T14:02:56.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7228999999999997e-05 2023-07-02T14:02:56.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363480+0000", 2023-07-02T14:02:56.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051367833000000002 2023-07-02T14:02:56.800 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.801 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.801 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.801 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:289 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:56.801 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:02:56.801 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.351013731999998, 2023-07-02T14:02:56.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051413158, 2023-07-02T14:02:56.802 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.802 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.802 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:56.802 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:56.802 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:56.802 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 289 2023-07-02T14:02:56.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.803 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:02:56.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:02:56.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:56.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:02:56.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1689e-05 2023-07-02T14:02:56.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312209+0000", 2023-07-02T14:02:56.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1070000000000005e-06 2023-07-02T14:02:56.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312217+0000", 2023-07-02T14:02:56.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.634e-06 2023-07-02T14:02:56.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312222+0000", 2023-07-02T14:02:56.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040512761000000001 2023-07-02T14:02:56.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352735+0000", 2023-07-02T14:02:56.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000119184 2023-07-02T14:02:56.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352854+0000", 2023-07-02T14:02:56.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021481899999999999 2023-07-02T14:02:56.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:56.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353069+0000", 2023-07-02T14:02:56.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001601707 2023-07-02T14:02:56.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:56.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354670+0000", 2023-07-02T14:02:56.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088168570000000009 2023-07-02T14:02:56.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:56.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363487+0000", 2023-07-02T14:02:56.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9635999999999997e-05 2023-07-02T14:02:56.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363537+0000", 2023-07-02T14:02:56.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4292999999999998e-05 2023-07-02T14:02:56.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363581+0000", 2023-07-02T14:02:56.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051413158 2023-07-02T14:02:56.811 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.811 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.812 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.812 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:290 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:56.812 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:02:56.812 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.350954486999999, 2023-07-02T14:02:56.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051455062000000003, 2023-07-02T14:02:56.813 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.813 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.813 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:56.813 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:56.813 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:56.813 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 290 2023-07-02T14:02:56.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.813 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:02:56.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:02:56.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:56.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:02:56.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6469000000000003e-05 2023-07-02T14:02:56.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312293+0000", 2023-07-02T14:02:56.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4630000000000007e-06 2023-07-02T14:02:56.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312303+0000", 2023-07-02T14:02:56.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8969999999999999e-06 2023-07-02T14:02:56.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312307+0000", 2023-07-02T14:02:56.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041261548000000002 2023-07-02T14:02:56.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353569+0000", 2023-07-02T14:02:56.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9594999999999999e-05 2023-07-02T14:02:56.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353659+0000", 2023-07-02T14:02:56.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018390600000000001 2023-07-02T14:02:56.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:56.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353843+0000", 2023-07-02T14:02:56.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035173100000000001 2023-07-02T14:02:56.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:56.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357360+0000", 2023-07-02T14:02:56.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062295670000000001 2023-07-02T14:02:56.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:56.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363589+0000", 2023-07-02T14:02:56.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9854000000000001e-05 2023-07-02T14:02:56.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363639+0000", 2023-07-02T14:02:56.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3133999999999997e-05 2023-07-02T14:02:56.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363682+0000", 2023-07-02T14:02:56.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051455062000000003 2023-07-02T14:02:56.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.822 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.823 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:291 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:56.823 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:02:56.823 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.350868448, 2023-07-02T14:02:56.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051527860000000002, 2023-07-02T14:02:56.823 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.824 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.824 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:56.824 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:56.824 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:56.824 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 291 2023-07-02T14:02:56.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.824 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:02:56.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:02:56.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:56.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:02:56.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2448999999999998e-05 2023-07-02T14:02:56.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312375+0000", 2023-07-02T14:02:56.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5259999999999993e-06 2023-07-02T14:02:56.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312384+0000", 2023-07-02T14:02:56.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0610000000000003e-06 2023-07-02T14:02:56.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312389+0000", 2023-07-02T14:02:56.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042326111 2023-07-02T14:02:56.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354715+0000", 2023-07-02T14:02:56.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3998999999999995e-05 2023-07-02T14:02:56.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354779+0000", 2023-07-02T14:02:56.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010269500000000001 2023-07-02T14:02:56.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:56.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354882+0000", 2023-07-02T14:02:56.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002533736 2023-07-02T14:02:56.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:56.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357415+0000", 2023-07-02T14:02:56.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062765700000000004 2023-07-02T14:02:56.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:56.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363692+0000", 2023-07-02T14:02:56.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7095000000000001e-05 2023-07-02T14:02:56.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363749+0000", 2023-07-02T14:02:56.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2263999999999996e-05 2023-07-02T14:02:56.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363841+0000", 2023-07-02T14:02:56.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051527860000000002 2023-07-02T14:02:56.833 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.833 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.833 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.834 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:292 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:56.834 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:02:56.834 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.350786200000002, 2023-07-02T14:02:56.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051556056000000003, 2023-07-02T14:02:56.834 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.834 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.834 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:56.834 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:56.835 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:56.835 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 292 2023-07-02T14:02:56.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.835 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:02:56.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:02:56.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:56.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312395+0000", 2023-07-02T14:02:56.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9624e-05 2023-07-02T14:02:56.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312425+0000", 2023-07-02T14:02:56.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:02:56.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312432+0000", 2023-07-02T14:02:56.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7550000000000002e-06 2023-07-02T14:02:56.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312437+0000", 2023-07-02T14:02:56.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.043823036000000003 2023-07-02T14:02:56.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356260+0000", 2023-07-02T14:02:56.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6021999999999997e-05 2023-07-02T14:02:56.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356346+0000", 2023-07-02T14:02:56.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020076600000000001 2023-07-02T14:02:56.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:56.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356547+0000", 2023-07-02T14:02:56.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020766410000000002 2023-07-02T14:02:56.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:56.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358623+0000", 2023-07-02T14:02:56.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005231192 2023-07-02T14:02:56.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:56.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363854+0000", 2023-07-02T14:02:56.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3423e-05 2023-07-02T14:02:56.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363908+0000", 2023-07-02T14:02:56.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3825999999999997e-05 2023-07-02T14:02:56.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363952+0000", 2023-07-02T14:02:56.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051556056000000003 2023-07-02T14:02:56.843 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.844 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.844 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.844 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:293 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:56.844 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:02:56.845 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.350740030000001, 2023-07-02T14:02:56.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051614769999999997, 2023-07-02T14:02:56.845 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.845 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.845 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:56.845 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:56.845 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:56.845 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 293 2023-07-02T14:02:56.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.846 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:02:56.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:02:56.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:56.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312441+0000", 2023-07-02T14:02:56.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9609999999999999e-05 2023-07-02T14:02:56.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312471+0000", 2023-07-02T14:02:56.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7069999999999992e-06 2023-07-02T14:02:56.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312478+0000", 2023-07-02T14:02:56.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1190000000000002e-06 2023-07-02T14:02:56.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312485+0000", 2023-07-02T14:02:56.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.044956387 2023-07-02T14:02:56.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357441+0000", 2023-07-02T14:02:56.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4134999999999994e-05 2023-07-02T14:02:56.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357515+0000", 2023-07-02T14:02:56.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000147408 2023-07-02T14:02:56.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:56.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357663+0000", 2023-07-02T14:02:56.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056472639999999999 2023-07-02T14:02:56.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:56.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363310+0000", 2023-07-02T14:02:56.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00065402600000000004 2023-07-02T14:02:56.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:56.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363964+0000", 2023-07-02T14:02:56.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0497000000000001e-05 2023-07-02T14:02:56.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364014+0000", 2023-07-02T14:02:56.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2243000000000002e-05 2023-07-02T14:02:56.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364057+0000", 2023-07-02T14:02:56.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051614769999999997 2023-07-02T14:02:56.854 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.854 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.854 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.855 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:294 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1048576~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:56.855 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:02:56.855 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.350692031000001, 2023-07-02T14:02:56.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051679527000000003, 2023-07-02T14:02:56.855 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.855 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.856 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:56.856 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:56.856 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:56.856 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 294 2023-07-02T14:02:56.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.856 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:02:56.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:02:56.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:56.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312489+0000", 2023-07-02T14:02:56.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8204999999999998e-05 2023-07-02T14:02:56.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312517+0000", 2023-07-02T14:02:56.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7759999999999997e-06 2023-07-02T14:02:56.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312525+0000", 2023-07-02T14:02:56.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8730000000000001e-06 2023-07-02T14:02:56.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312532+0000", 2023-07-02T14:02:56.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.045442206999999998 2023-07-02T14:02:56.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357974+0000", 2023-07-02T14:02:56.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4422000000000002e-05 2023-07-02T14:02:56.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358029+0000", 2023-07-02T14:02:56.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149519 2023-07-02T14:02:56.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:56.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358178+0000", 2023-07-02T14:02:56.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051779390000000003 2023-07-02T14:02:56.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:56.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363356+0000", 2023-07-02T14:02:56.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00070707699999999997 2023-07-02T14:02:56.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:56.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364063+0000", 2023-07-02T14:02:56.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6835999999999999e-05 2023-07-02T14:02:56.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364090+0000", 2023-07-02T14:02:56.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9263000000000003e-05 2023-07-02T14:02:56.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364169+0000", 2023-07-02T14:02:56.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051679527000000003 2023-07-02T14:02:56.865 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.865 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.865 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.865 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:423 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [write 2555904~1638400] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:56.866 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.330332+0000", 2023-07-02T14:02:56.866 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.332850036, 2023-07-02T14:02:56.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038020074000000001, 2023-07-02T14:02:56.866 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.866 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.866 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:56.867 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:56.867 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:56.867 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 423 2023-07-02T14:02:56.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.867 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.867 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.330332+0000", 2023-07-02T14:02:56.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.330332+0000", 2023-07-02T14:02:56.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:02:56.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.330330+0000", 2023-07-02T14:02:56.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060720390000000004 2023-07-02T14:02:56.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336402+0000", 2023-07-02T14:02:56.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8478000000000002e-05 2023-07-02T14:02:56.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336421+0000", 2023-07-02T14:02:56.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0624999999999999e-05 2023-07-02T14:02:56.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336451+0000", 2023-07-02T14:02:56.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022199402 2023-07-02T14:02:56.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358651+0000", 2023-07-02T14:02:56.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1981000000000001e-05 2023-07-02T14:02:56.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358723+0000", 2023-07-02T14:02:56.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163117 2023-07-02T14:02:56.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:56.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358886+0000", 2023-07-02T14:02:56.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005296762 2023-07-02T14:02:56.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:56.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364183+0000", 2023-07-02T14:02:56.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040861739999999997 2023-07-02T14:02:56.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.874 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:56.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368269+0000", 2023-07-02T14:02:56.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7169000000000003e-05 2023-07-02T14:02:56.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368306+0000", 2023-07-02T14:02:56.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5760999999999998e-05 2023-07-02T14:02:56.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368352+0000", 2023-07-02T14:02:56.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038020074000000001 2023-07-02T14:02:56.876 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.876 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.876 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.877 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:427 2.2 2:44c63967:::rbd_data.125de1449605.0000000000000101:head [write 2621440~1572864] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:56.877 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.336464+0000", 2023-07-02T14:02:56.877 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.326718207999999, 2023-07-02T14:02:56.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037760331000000001, 2023-07-02T14:02:56.877 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:56.877 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:56.878 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:56.878 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:56.878 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:56.878 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 427 2023-07-02T14:02:56.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.878 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:56.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:56.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336464+0000", 2023-07-02T14:02:56.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:56.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:56.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336464+0000", 2023-07-02T14:02:56.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:02:56.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:56.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336462+0000", 2023-07-02T14:02:56.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050934140000000001 2023-07-02T14:02:56.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:56.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.341555+0000", 2023-07-02T14:02:56.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6209e-05 2023-07-02T14:02:56.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:56.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.341572+0000", 2023-07-02T14:02:56.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.114e-06 2023-07-02T14:02:56.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:56.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.341579+0000", 2023-07-02T14:02:56.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019481099000000002 2023-07-02T14:02:56.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:56.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361060+0000", 2023-07-02T14:02:56.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5408000000000005e-05 2023-07-02T14:02:56.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:56.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361155+0000", 2023-07-02T14:02:56.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000186515 2023-07-02T14:02:56.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:02:56.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361342+0000", 2023-07-02T14:02:56.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003066465 2023-07-02T14:02:56.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:56.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364408+0000", 2023-07-02T14:02:56.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0096922329999999998 2023-07-02T14:02:56.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.885 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:56.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.374100+0000", 2023-07-02T14:02:56.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7960999999999995e-05 2023-07-02T14:02:56.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:56.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.374168+0000", 2023-07-02T14:02:56.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5458000000000003e-05 2023-07-02T14:02:56.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:56.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.374224+0000", 2023-07-02T14:02:56.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037760331000000001 2023-07-02T14:02:56.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.887 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.888 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.888 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:56.889 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:02:56.890 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:02:56.891 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:56.891 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 51, 2023-07-02T14:02:56.891 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2", 2023-07-02T14:02:56.891 INFO:teuthology.orchestra.run.smithi016.stdout: "created": "2023-07-02T14:01:37.298060+0000", 2023-07-02T14:02:56.891 INFO:teuthology.orchestra.run.smithi016.stdout: "modified": "2023-07-02T14:02:56.080301+0000", 2023-07-02T14:02:56.891 INFO:teuthology.orchestra.run.smithi016.stdout: "last_up_change": "2023-07-02T14:01:45.271760+0000", 2023-07-02T14:02:56.891 INFO:teuthology.orchestra.run.smithi016.stdout: "last_in_change": "2023-07-02T14:01:39.129121+0000", 2023-07-02T14:02:56.892 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-07-02T14:02:56.892 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_num": 5799936, 2023-07-02T14:02:56.892 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_set": [ 2023-07-02T14:02:56.892 INFO:teuthology.orchestra.run.smithi016.stdout: "pglog_hardlimit", 2023-07-02T14:02:56.892 INFO:teuthology.orchestra.run.smithi016.stdout: "purged_snapdirs", 2023-07-02T14:02:56.892 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_deletes", 2023-07-02T14:02:56.892 INFO:teuthology.orchestra.run.smithi016.stdout: "sortbitwise" 2023-07-02T14:02:56.893 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:56.893 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_version": 4, 2023-07-02T14:02:56.893 INFO:teuthology.orchestra.run.smithi016.stdout: "full_ratio": 0.94999998807907104, 2023-07-02T14:02:56.893 INFO:teuthology.orchestra.run.smithi016.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-07-02T14:02:56.893 INFO:teuthology.orchestra.run.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2023-07-02T14:02:56.893 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_snapshot": "", 2023-07-02T14:02:56.893 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_max": 2, 2023-07-02T14:02:56.894 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd": 6, 2023-07-02T14:02:56.894 INFO:teuthology.orchestra.run.smithi016.stdout: "require_min_compat_client": "luminous", 2023-07-02T14:02:56.894 INFO:teuthology.orchestra.run.smithi016.stdout: "min_compat_client": "luminous", 2023-07-02T14:02:56.894 INFO:teuthology.orchestra.run.smithi016.stdout: "require_osd_release": "pacific", 2023-07-02T14:02:56.894 INFO:teuthology.orchestra.run.smithi016.stdout: "pools": [ 2023-07-02T14:02:56.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.894 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 1, 2023-07-02T14:02:56.894 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "device_health_metrics", 2023-07-02T14:02:56.895 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:45.323455+0000", 2023-07-02T14:02:56.895 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2023-07-02T14:02:56.895 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2023-07-02T14:02:56.895 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:02:56.895 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:02:56.895 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:02:56.895 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:02:56.895 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:02:56.896 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:02:56.896 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:02:56.896 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:02:56.896 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:02:56.896 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:02:56.896 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 1, 2023-07-02T14:02:56.897 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 1, 2023-07-02T14:02:56.897 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 1, 2023-07-02T14:02:56.897 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 1, 2023-07-02T14:02:56.897 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 1, 2023-07-02T14:02:56.898 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:02:56.898 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:02:56.898 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:02:56.898 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:02:56.898 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:02:56.899 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:02:56.899 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:02:56.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.899 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "14", 2023-07-02T14:02:56.899 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:02:56.899 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:02:56.899 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:02:56.900 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:02:56.900 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:02:56.900 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2023-07-02T14:02:56.900 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2023-07-02T14:02:56.900 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:02:56.900 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:02:56.900 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:02:56.900 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:02:56.901 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:02:56.901 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:02:56.901 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:02:56.901 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:02:56.901 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:02:56.901 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:02:56.901 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:02:56.902 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:02:56.902 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:02:56.902 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:02:56.902 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:02:56.902 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:02:56.902 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:02:56.902 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:02:56.902 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:02:56.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.903 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:02:56.903 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:02:56.903 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:02:56.903 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:02:56.903 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:02:56.904 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:02:56.904 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:02:56.904 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:02:56.904 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:02:56.904 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:02:56.904 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:02:56.905 INFO:teuthology.orchestra.run.smithi016.stdout: "options": { 2023-07-02T14:02:56.905 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_max": 32, 2023-07-02T14:02:56.905 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_min": 1 2023-07-02T14:02:56.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.905 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:02:56.905 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_devicehealth": {} 2023-07-02T14:02:56.905 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.906 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 2, 2023-07-02T14:02:56.906 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "rbd", 2023-07-02T14:02:56.906 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:47.448897+0000", 2023-07-02T14:02:56.906 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 8193, 2023-07-02T14:02:56.906 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2023-07-02T14:02:56.906 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:02:56.906 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:02:56.907 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:02:56.907 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:02:56.907 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:02:56.907 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:02:56.907 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:02:56.907 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:02:56.907 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:02:56.907 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:02:56.908 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2023-07-02T14:02:56.908 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2023-07-02T14:02:56.908 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2023-07-02T14:02:56.908 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2023-07-02T14:02:56.908 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2023-07-02T14:02:56.909 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:02:56.909 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:02:56.909 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:02:56.909 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:02:56.909 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:02:56.909 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:02:56.910 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:02:56.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.910 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "50", 2023-07-02T14:02:56.910 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:02:56.910 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:02:56.910 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:02:56.910 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:02:56.911 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:02:56.911 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 6, 2023-07-02T14:02:56.911 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 50, 2023-07-02T14:02:56.911 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:02:56.911 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:02:56.911 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:02:56.911 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:02:56.912 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:02:56.912 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:02:56.912 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:02:56.912 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:02:56.912 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:02:56.912 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:02:56.912 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:02:56.912 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:02:56.913 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:02:56.913 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:02:56.913 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:02:56.913 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:02:56.913 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:02:56.913 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:02:56.913 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:02:56.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.914 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:02:56.914 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:02:56.914 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:02:56.914 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:02:56.914 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:02:56.914 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:02:56.914 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:02:56.915 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:02:56.915 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:02:56.915 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:02:56.915 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:02:56.915 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2023-07-02T14:02:56.915 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:02:56.915 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2023-07-02T14:02:56.915 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.916 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.916 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:56.916 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:02:56.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.916 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:02:56.916 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "43c446fe-c811-4ee3-99e8-d7452edfc1c1", 2023-07-02T14:02:56.916 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:56.917 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:56.917 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:56.917 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:56.917 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:56.917 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:56.917 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:56.917 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 0, 2023-07-02T14:02:56.918 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:56.918 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:56.918 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:56.918 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.918 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.919 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2023-07-02T14:02:56.919 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:56.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.920 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.920 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2023-07-02T14:02:56.920 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:56.920 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.920 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.921 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:56.921 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.921 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.921 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6802", 2023-07-02T14:02:56.921 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:56.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.922 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.922 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6803", 2023-07-02T14:02:56.922 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:56.922 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.922 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.923 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:56.923 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.923 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.923 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6806", 2023-07-02T14:02:56.924 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:56.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.924 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.924 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6807", 2023-07-02T14:02:56.925 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:56.925 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.925 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.925 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:56.925 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.925 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.926 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6804", 2023-07-02T14:02:56.926 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:56.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.926 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.926 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6805", 2023-07-02T14:02:56.926 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:02:56.927 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.927 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.927 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6801/83359", 2023-07-02T14:02:56.927 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6803/83359", 2023-07-02T14:02:56.927 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6807/83359", 2023-07-02T14:02:56.927 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6805/83359", 2023-07-02T14:02:56.927 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:56.928 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:56.928 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:56.928 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.928 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:02:56.928 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b", 2023-07-02T14:02:56.929 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:56.929 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:56.929 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:56.929 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:56.929 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:56.930 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:56.930 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:56.930 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 46, 2023-07-02T14:02:56.930 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:56.930 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:56.931 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:56.931 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.931 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.931 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6808", 2023-07-02T14:02:56.931 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:56.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.932 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.932 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6809", 2023-07-02T14:02:56.932 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:56.932 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.932 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.932 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:56.933 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.933 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.933 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6810", 2023-07-02T14:02:56.933 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:56.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.933 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.934 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6811", 2023-07-02T14:02:56.934 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:56.934 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.934 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.934 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:56.934 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.935 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.935 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6814", 2023-07-02T14:02:56.935 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:56.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.935 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.935 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6815", 2023-07-02T14:02:56.936 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:56.936 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.936 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.936 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:56.936 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.937 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.937 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6812", 2023-07-02T14:02:56.937 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:56.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.937 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.937 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6813", 2023-07-02T14:02:56.937 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:02:56.938 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.938 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.938 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6809/83360", 2023-07-02T14:02:56.938 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6811/83360", 2023-07-02T14:02:56.938 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6815/83360", 2023-07-02T14:02:56.938 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6813/83360", 2023-07-02T14:02:56.938 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:56.939 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:56.939 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:56.939 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.939 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:02:56.939 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "99f8ea0a-5a71-4abe-b087-8e891fd11d1b", 2023-07-02T14:02:56.940 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:56.940 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:56.940 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:56.940 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:56.940 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:56.940 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:56.941 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:56.941 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 14, 2023-07-02T14:02:56.941 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:56.941 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:56.941 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:56.942 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.942 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.942 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6816", 2023-07-02T14:02:56.942 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:56.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.942 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.943 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6817", 2023-07-02T14:02:56.943 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:56.943 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.943 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.943 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:56.943 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.944 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.944 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6818", 2023-07-02T14:02:56.944 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:56.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.944 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.944 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6819", 2023-07-02T14:02:56.945 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:56.945 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.945 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.945 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:56.945 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.946 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.946 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6822", 2023-07-02T14:02:56.946 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:56.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.946 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.946 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6823", 2023-07-02T14:02:56.947 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:56.947 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.947 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.947 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:56.947 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.948 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.948 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6820", 2023-07-02T14:02:56.948 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:56.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.948 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.948 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6821", 2023-07-02T14:02:56.949 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:02:56.949 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.949 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.949 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6817/83361", 2023-07-02T14:02:56.949 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6819/83361", 2023-07-02T14:02:56.949 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6823/83361", 2023-07-02T14:02:56.950 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6821/83361", 2023-07-02T14:02:56.950 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:56.950 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:56.950 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:56.950 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.951 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:02:56.951 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "d285b4d5-0bef-421a-b84a-943e485b10c2", 2023-07-02T14:02:56.951 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:56.951 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:56.952 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:56.952 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:56.952 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:56.952 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:56.952 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:56.952 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 18, 2023-07-02T14:02:56.953 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:56.953 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:56.953 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:56.953 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.953 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.953 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6808", 2023-07-02T14:02:56.953 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:56.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.954 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.954 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6809", 2023-07-02T14:02:56.955 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:56.955 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.955 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.955 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:56.955 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.956 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.956 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6810", 2023-07-02T14:02:56.956 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:56.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.956 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.956 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6811", 2023-07-02T14:02:56.957 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:56.957 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.957 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.957 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:56.957 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.957 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.958 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6814", 2023-07-02T14:02:56.958 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:56.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.958 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.958 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6815", 2023-07-02T14:02:56.958 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:56.958 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.959 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.959 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:56.959 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.959 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.959 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6812", 2023-07-02T14:02:56.960 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:56.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.960 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.960 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6813", 2023-07-02T14:02:56.960 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:02:56.960 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.960 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.961 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6809/83350", 2023-07-02T14:02:56.961 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6811/83350", 2023-07-02T14:02:56.961 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6815/83350", 2023-07-02T14:02:56.961 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6813/83350", 2023-07-02T14:02:56.961 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:56.961 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:56.961 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:56.962 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.962 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:02:56.962 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "7ba0b0db-e407-4c63-8278-aab09f3082c4", 2023-07-02T14:02:56.962 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:56.962 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:56.963 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:56.963 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 0.23297119140625, 2023-07-02T14:02:56.963 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:56.963 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:56.963 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:56.963 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 0, 2023-07-02T14:02:56.963 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:56.963 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:56.964 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:56.964 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.964 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.964 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6800", 2023-07-02T14:02:56.964 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:56.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.965 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.965 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6801", 2023-07-02T14:02:56.965 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:56.965 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.965 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.965 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:56.966 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.966 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.966 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6802", 2023-07-02T14:02:56.966 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:56.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.966 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.967 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6803", 2023-07-02T14:02:56.967 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:56.967 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.967 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.967 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:56.967 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.968 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.968 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6806", 2023-07-02T14:02:56.968 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:56.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.968 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.968 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6807", 2023-07-02T14:02:56.969 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:56.969 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.969 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.969 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:56.969 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.969 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.970 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6804", 2023-07-02T14:02:56.970 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:56.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.970 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.970 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6805", 2023-07-02T14:02:56.970 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:02:56.971 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.971 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.971 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6801/83351", 2023-07-02T14:02:56.971 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6803/83351", 2023-07-02T14:02:56.971 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6807/83351", 2023-07-02T14:02:56.971 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6805/83351", 2023-07-02T14:02:56.971 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:56.972 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:56.972 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:56.972 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.972 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:02:56.972 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "29663bfa-f6fe-4ce7-b45a-a9b7b4424c42", 2023-07-02T14:02:56.972 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:02:56.973 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:02:56.973 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:02:56.973 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:02:56.973 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:02:56.973 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:02:56.973 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:02:56.973 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 34, 2023-07-02T14:02:56.973 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:02:56.974 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:02:56.974 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:02:56.974 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.974 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.974 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6816", 2023-07-02T14:02:56.974 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:56.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.975 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6817", 2023-07-02T14:02:56.975 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:56.975 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.975 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.976 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:02:56.976 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.976 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.976 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6818", 2023-07-02T14:02:56.976 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:56.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.977 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.977 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6819", 2023-07-02T14:02:56.977 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:56.977 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.977 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.977 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:02:56.977 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.978 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.978 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6822", 2023-07-02T14:02:56.978 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:56.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.978 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.978 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6823", 2023-07-02T14:02:56.979 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:56.979 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.979 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.979 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:02:56.979 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:02:56.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.979 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:02:56.980 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6820", 2023-07-02T14:02:56.980 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:56.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.980 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:02:56.980 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6821", 2023-07-02T14:02:56.980 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:02:56.981 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.981 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.981 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6817/83352", 2023-07-02T14:02:56.981 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6819/83352", 2023-07-02T14:02:56.981 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6823/83352", 2023-07-02T14:02:56.981 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6821/83352", 2023-07-02T14:02:56.981 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:02:56.982 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:02:56.982 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:02:56.982 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.982 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.982 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:56.982 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_xinfo": [ 2023-07-02T14:02:56.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.983 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:02:56.983 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:56.983 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:56.983 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:56.983 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:56.983 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:56.983 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.001178+0000", 2023-07-02T14:02:56.983 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:56.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.984 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:02:56.984 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:56.984 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:56.984 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:56.984 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:56.984 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:56.985 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.184181+0000", 2023-07-02T14:02:56.985 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:56.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.985 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:02:56.985 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:56.985 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:56.986 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:56.986 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:56.986 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:56.986 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.274854+0000", 2023-07-02T14:02:56.986 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:56.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.986 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:02:56.987 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:56.987 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:56.987 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:56.987 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:56.987 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:56.987 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.186217+0000", 2023-07-02T14:02:56.987 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:56.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.988 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:02:56.988 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:56.988 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:56.988 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:56.988 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:56.988 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:56.989 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.247442+0000", 2023-07-02T14:02:56.989 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:56.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.989 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:02:56.989 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:02:56.989 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:02:56.989 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:02:56.990 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:02:56.990 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:02:56.990 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.220075+0000", 2023-07-02T14:02:56.990 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:02:56.990 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.990 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:56.990 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap": [ 2023-07-02T14:02:56.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.991 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.0", 2023-07-02T14:02:56.991 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:02:56.991 INFO:teuthology.orchestra.run.smithi016.stdout: 4, 2023-07-02T14:02:56.991 INFO:teuthology.orchestra.run.smithi016.stdout: 1 2023-07-02T14:02:56.991 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.991 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.2", 2023-07-02T14:02:56.992 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:02:56.992 INFO:teuthology.orchestra.run.smithi016.stdout: 1, 2023-07-02T14:02:56.992 INFO:teuthology.orchestra.run.smithi016.stdout: 5 2023-07-02T14:02:56.992 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.992 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.4", 2023-07-02T14:02:56.992 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:02:56.993 INFO:teuthology.orchestra.run.smithi016.stdout: 5, 2023-07-02T14:02:56.993 INFO:teuthology.orchestra.run.smithi016.stdout: 0 2023-07-02T14:02:56.993 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.993 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.993 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:56.993 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_items": [ 2023-07-02T14:02:56.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.994 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.7", 2023-07-02T14:02:56.994 INFO:teuthology.orchestra.run.smithi016.stdout: "mappings": [ 2023-07-02T14:02:56.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:56.994 INFO:teuthology.orchestra.run.smithi016.stdout: "from": 4, 2023-07-02T14:02:56.994 INFO:teuthology.orchestra.run.smithi016.stdout: "to": 5 2023-07-02T14:02:56.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.994 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:56.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.995 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:02:56.995 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_temp": [], 2023-07-02T14:02:56.995 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_temp": [], 2023-07-02T14:02:56.995 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklist": {}, 2023-07-02T14:02:56.995 INFO:teuthology.orchestra.run.smithi016.stdout: "range_blocklist": {}, 2023-07-02T14:02:56.995 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profiles": { 2023-07-02T14:02:56.995 INFO:teuthology.orchestra.run.smithi016.stdout: "default": { 2023-07-02T14:02:56.995 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2023-07-02T14:02:56.996 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "2", 2023-07-02T14:02:56.996 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "1", 2023-07-02T14:02:56.996 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "jerasure", 2023-07-02T14:02:56.996 INFO:teuthology.orchestra.run.smithi016.stdout: "ruleset-failure-domain": "osd", 2023-07-02T14:02:56.996 INFO:teuthology.orchestra.run.smithi016.stdout: "technique": "reed_sol_van" 2023-07-02T14:02:56.996 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:56.997 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps_queue": [], 2023-07-02T14:02:56.997 INFO:teuthology.orchestra.run.smithi016.stdout: "new_removed_snaps": [], 2023-07-02T14:02:56.997 INFO:teuthology.orchestra.run.smithi016.stdout: "new_purged_snaps": [], 2023-07-02T14:02:56.997 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_node_flags": {}, 2023-07-02T14:02:56.997 INFO:teuthology.orchestra.run.smithi016.stdout: "device_class_flags": {}, 2023-07-02T14:02:56.997 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode": { 2023-07-02T14:02:56.997 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_enabled": false, 2023-07-02T14:02:56.997 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_bucket_count": 0, 2023-07-02T14:02:56.998 INFO:teuthology.orchestra.run.smithi016.stdout: "degraded_stretch_mode": 0, 2023-07-02T14:02:56.998 INFO:teuthology.orchestra.run.smithi016.stdout: "recovering_stretch_mode": 0, 2023-07-02T14:02:56.998 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_bucket": 0 2023-07-02T14:02:56.998 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:56.998 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:56.999 INFO:tasks.thrashosds.thrasher:j is {'epoch': 51, 'fsid': '9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2', 'created': '2023-07-02T14:01:37.298060+0000', 'modified': '2023-07-02T14:02:56.080301+0000', 'last_up_change': '2023-07-02T14:01:45.271760+0000', 'last_in_change': '2023-07-02T14:01:39.129121+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-07-02T14:01:45.323455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-02T14:01:47.448897+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 50, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '43c446fe-c811-4ee3-99e8-d7452edfc1c1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6800', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6801', 'nonce': 83359}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6802', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6803', 'nonce': 83359}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6806', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6807', 'nonce': 83359}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6804', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6805', 'nonce': 83359}]}, 'public_addr': '172.21.15.16:6801/83359', 'cluster_addr': '172.21.15.16:6803/83359', 'heartbeat_back_addr': '172.21.15.16:6807/83359', 'heartbeat_front_addr': '172.21.15.16:6805/83359', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 46, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6808', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6809', 'nonce': 83360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6810', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6811', 'nonce': 83360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6814', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6815', 'nonce': 83360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6812', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6813', 'nonce': 83360}]}, 'public_addr': '172.21.15.16:6809/83360', 'cluster_addr': '172.21.15.16:6811/83360', 'heartbeat_back_addr': '172.21.15.16:6815/83360', 'heartbeat_front_addr': '172.21.15.16:6813/83360', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '99f8ea0a-5a71-4abe-b087-8e891fd11d1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 14, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6816', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6817', 'nonce': 83361}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6818', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6819', 'nonce': 83361}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6822', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6823', 'nonce': 83361}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6820', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6821', 'nonce': 83361}]}, 'public_addr': '172.21.15.16:6817/83361', 'cluster_addr': '172.21.15.16:6819/83361', 'heartbeat_back_addr': '172.21.15.16:6823/83361', 'heartbeat_front_addr': '172.21.15.16:6821/83361', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd285b4d5-0bef-421a-b84a-943e485b10c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6808', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6809', 'nonce': 83350}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6810', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6811', 'nonce': 83350}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6814', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6815', 'nonce': 83350}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6812', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6813', 'nonce': 83350}]}, 'public_addr': '172.21.15.22:6809/83350', 'cluster_addr': '172.21.15.22:6811/83350', 'heartbeat_back_addr': '172.21.15.22:6815/83350', 'heartbeat_front_addr': '172.21.15.22:6813/83350', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7ba0b0db-e407-4c63-8278-aab09f3082c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.23297119140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6800', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6801', 'nonce': 83351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6802', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6803', 'nonce': 83351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6806', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6807', 'nonce': 83351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6804', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6805', 'nonce': 83351}]}, 'public_addr': '172.21.15.22:6801/83351', 'cluster_addr': '172.21.15.22:6803/83351', 'heartbeat_back_addr': '172.21.15.22:6807/83351', 'heartbeat_front_addr': '172.21.15.22:6805/83351', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '29663bfa-f6fe-4ce7-b45a-a9b7b4424c42', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6816', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6817', 'nonce': 83352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6818', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6819', 'nonce': 83352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6822', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6823', 'nonce': 83352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6820', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6821', 'nonce': 83352}]}, 'public_addr': '172.21.15.22:6817/83352', 'cluster_addr': '172.21.15.22:6819/83352', 'heartbeat_back_addr': '172.21.15.22:6823/83352', 'heartbeat_front_addr': '172.21.15.22:6821/83352', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.001178+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.184181+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.274854+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.186217+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.247442+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.220075+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.0', 'osds': [4, 1]}, {'pgid': '2.2', 'osds': [1, 5]}, {'pgid': '2.4', 'osds': [5, 0]}], 'pg_upmap_items': [{'pgid': '2.7', 'mappings': [{'from': 4, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-07-02T14:02:57.000 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:02:57.000 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:56.888+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:57.001 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:57.001+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:57.103 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:57.102+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:57.131 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:57.132 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:57.132 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:02:57.132 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:57.142 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:02:57.203 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:57.202+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:57.267 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:57.267 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:02:57.267 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:02:57.267 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:02:57.267 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:57.277 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:02:57.305 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:57.304+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:57.308 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:02:57.310 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:02:57.321 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2023-07-02T14:02:56.244015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134895325,"num_objects":65,"num_object_clones":9,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":893,"num_read_kb":138426,"num_write":842,"num_write_kb":119563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8523954,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":813,"ondisk_log_size":813,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":276824,"kb_used_data":242152,"kb_used_omap":0,"kb_used_meta":34624,"kb_avail":565954216,"statfs":{"total":579820584960,"available":579537117184,"internally_reserved":0,"allocated":247963648,"data_stored":282438278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35454976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28996,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":341,"num_read_kb":20180,"num_write":12,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.172305"},"pg_stats":[{"pgid":"2.7","version":"50'145","reported_seq":372,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-07-02T14:02:55.100335+0000","last_change":"2023-07-02T14:02:39.068336+0000","last_active":"2023-07-02T14:02:55.100335+0000","last_peered":"2023-07-02T14:02:55.100335+0000","last_clean":"2023-07-02T14:02:55.100335+0000","last_became_active":"2023-07-02T14:02:01.725984+0000","last_became_peered":"2023-07-02T14:02:01.725984+0000","last_unstale":"2023-07-02T14:02:55.100335+0000","last_undegraded":"2023-07-02T14:02:55.100335+0000","last_fullsized":"2023-07-02T14:02:55.100335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":145,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20595908,"num_objects":12,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":388,"num_read_kb":19384,"num_write":135,"num_write_kb":19100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"49'113","reported_seq":178,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-07-02T14:02:54.138493+0000","last_change":"2023-07-02T14:02:39.062139+0000","last_active":"2023-07-02T14:02:54.138493+0000","last_peered":"2023-07-02T14:02:54.138493+0000","last_clean":"2023-07-02T14:02:54.138493+0000","last_became_active":"2023-07-02T14:01:49.293782+0000","last_became_peered":"2023-07-02T14:01:49.293782+0000","last_unstale":"2023-07-02T14:02:54.138493+0000","last_undegraded":"2023-07-02T14:02:54.138493+0000","last_fullsized":"2023-07-02T14:02:54.138493+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":113,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20680704,"num_objects":10,"num_object_clones":2,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":19,"num_read_kb":24988,"num_write":111,"num_write_kb":19204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"47'33","reported_seq":159,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-07-02T14:02:54.146809+0000","last_change":"2023-07-02T14:02:39.069176+0000","last_active":"2023-07-02T14:02:54.146809+0000","last_peered":"2023-07-02T14:02:54.146809+0000","last_clean":"2023-07-02T14:02:54.146809+0000","last_became_active":"2023-07-02T14:01:49.292790+0000","last_became_peered":"2023-07-02T14:01:49.292790+0000","last_unstale":"2023-07-02T14:02:54.146809+0000","last_undegraded":"2023-07-02T14:02:54.146809+0000","last_fullsized":"2023-07-02T14:02:54.146809+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23330816,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":15749,"num_write":30,"num_write_kb":15681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"50'138","reported_seq":240,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-07-02T14:02:55.086026+0000","last_change":"2023-07-02T14:02:50.069577+0000","last_active":"2023-07-02T14:02:55.086026+0000","last_peered":"2023-07-02T14:02:55.086026+0000","last_clean":"2023-07-02T14:02:55.086026+0000","last_became_active":"2023-07-02T14:02:50.068789+0000","last_became_peered":"2023-07-02T14:02:50.068789+0000","last_unstale":"2023-07-02T14:02:55.086026+0000","last_undegraded":"2023-07-02T14:02:55.086026+0000","last_fullsized":"2023-07-02T14:02:55.086026+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":138,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12132692,"num_objects":9,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":334,"num_read_kb":15173,"num_write":184,"num_write_kb":10924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8523954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":44,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-07-02T14:02:53.077719+0000","last_change":"2023-07-02T14:01:47.288970+0000","last_active":"2023-07-02T14:02:53.077719+0000","last_peered":"2023-07-02T14:02:53.077719+0000","last_clean":"2023-07-02T14:02:53.077719+0000","last_became_active":"2023-07-02T14:01:47.288610+0000","last_became_peered":"2023-07-02T14:01:47.288610+0000","last_unstale":"2023-07-02T14:02:53.077719+0000","last_undegraded":"2023-07-02T14:02:53.077719+0000","last_fullsized":"2023-07-02T14:02:53.077719+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"47'91","reported_seq":151,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-07-02T14:02:54.142448+0000","last_change":"2023-07-02T14:02:39.064116+0000","last_active":"2023-07-02T14:02:54.142448+0000","last_peered":"2023-07-02T14:02:54.142448+0000","last_clean":"2023-07-02T14:02:54.142448+0000","last_became_active":"2023-07-02T14:01:49.297946+0000","last_became_peered":"2023-07-02T14:01:49.297946+0000","last_unstale":"2023-07-02T14:02:54.142448+0000","last_undegraded":"2023-07-02T14:02:54.142448+0000","last_fullsized":"2023-07-02T14:02:54.142448+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":91,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16478224,"num_objects":8,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":17,"num_read_kb":19060,"num_write":90,"num_write_kb":15017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"50'125","reported_seq":189,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-07-02T14:02:55.088188+0000","last_change":"2023-07-02T14:02:39.061498+0000","last_active":"2023-07-02T14:02:55.088188+0000","last_peered":"2023-07-02T14:02:55.088188+0000","last_clean":"2023-07-02T14:02:55.088188+0000","last_became_active":"2023-07-02T14:01:49.294964+0000","last_became_peered":"2023-07-02T14:01:49.294964+0000","last_unstale":"2023-07-02T14:02:55.088188+0000","last_undegraded":"2023-07-02T14:02:55.088188+0000","last_fullsized":"2023-07-02T14:02:55.088188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":125,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20971682,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":16,"num_read_kb":20441,"num_write":125,"num_write_kb":20437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"40'85","reported_seq":147,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-07-02T14:02:55.094425+0000","last_change":"2023-07-02T14:02:39.062067+0000","last_active":"2023-07-02T14:02:55.094425+0000","last_peered":"2023-07-02T14:02:55.094425+0000","last_clean":"2023-07-02T14:02:55.094425+0000","last_became_active":"2023-07-02T14:02:15.025376+0000","last_became_peered":"2023-07-02T14:02:15.025376+0000","last_unstale":"2023-07-02T14:02:55.094425+0000","last_undegraded":"2023-07-02T14:02:55.094425+0000","last_fullsized":"2023-07-02T14:02:55.094425+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8519699,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":8463,"num_write":85,"num_write_kb":8322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"47'83","reported_seq":122,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-07-02T14:02:54.141768+0000","last_change":"2023-07-02T14:02:39.064149+0000","last_active":"2023-07-02T14:02:54.141768+0000","last_peered":"2023-07-02T14:02:54.141768+0000","last_clean":"2023-07-02T14:02:54.141768+0000","last_became_active":"2023-07-02T14:02:27.728488+0000","last_became_peered":"2023-07-02T14:02:27.728488+0000","last_unstale":"2023-07-02T14:02:54.141768+0000","last_undegraded":"2023-07-02T14:02:54.141768+0000","last_fullsized":"2023-07-02T14:02:54.141768+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":83,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12185600,"num_objects":6,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":15168,"num_write":82,"num_write_kb":10878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":134895325,"num_objects":65,"num_object_clones":9,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":893,"num_read_kb":138426,"num_write":842,"num_write_kb":119563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8523954,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":244604928,"data_stored":281202106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":813,"ondisk_log_size":813,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640272,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59764,"kb_used_data":53868,"kb_used_omap":0,"kb_used_meta":5888,"kb_avail":94312076,"statfs":{"total":96636764160,"available":96575565824,"internally_reserved":0,"allocated":55160832,"data_stored":60028705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6029312},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.41199999999999998}]},{"osd":1,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.45000000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.374}]},{"osd":3,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.34000000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.29099999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640272,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":16644,"kb_used_data":11388,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94355196,"statfs":{"total":96636764160,"available":96619720704,"internally_reserved":0,"allocated":11661312,"data_stored":13124494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.44}]},{"osd":1,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.36499999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.33700000000000002}]},{"osd":3,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.314},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.42199999999999999}]},{"osd":5,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.35399999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640272,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60780,"kb_used_data":54500,"kb_used_omap":0,"kb_used_meta":6272,"kb_avail":94311060,"statfs":{"total":96636764160,"available":96574525440,"internally_reserved":0,"allocated":55808000,"data_stored":68469547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6422528},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.47199999999999998}]},{"osd":1,"last update":"Sun Jul 2 14:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.46000000000000002}]},{"osd":2,"last update":"Sun Jul 2 14:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.51800000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.317}]},{"osd":5,"last update":"Sun Jul 2 14:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.33300000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640272,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41500,"kb_used_data":35988,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":94330340,"statfs":{"total":96636764160,"available":96594268160,"internally_reserved":0,"allocated":36851712,"data_stored":38198903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.34100000000000003}]},{"osd":1,"last update":"Sun Jul 2 14:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.307},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.35399999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.51800000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.57299999999999995}]},{"osd":5,"last update":"Sun Jul 2 14:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.63},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.59799999999999998}]}]},{"osd":0,"up_from":11,"seq":47244640272,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32420,"kb_used_data":27100,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94339420,"statfs":{"total":96636764160,"available":96603566080,"internally_reserved":0,"allocated":27750400,"data_stored":34475623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 2 14:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.432,"5min":0.432,"15min":0.432},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.32600000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.314}]},{"osd":3,"last update":"Sun Jul 2 14:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.55800000000000005}]},{"osd":4,"last update":"Sun Jul 2 14:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.53500000000000003}]},{"osd":5,"last update":"Sun Jul 2 14:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.502},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.57699999999999996}]}]},{"osd":1,"up_from":11,"seq":47244640272,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65716,"kb_used_data":59308,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":94306124,"statfs":{"total":96636764160,"available":96569470976,"internally_reserved":0,"allocated":60731392,"data_stored":68141006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.496,"5min":0.496,"15min":0.496},"last":0.30399999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.32100000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.42399999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.48699999999999999}]},{"osd":5,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.432}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":27193344,"data_stored":34271232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":60178432,"data_stored":67936777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36298752,"data_stored":37994674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":55250944,"data_stored":68265156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11091968,"data_stored":12915028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":54591488,"data_stored":59819239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:02:57.323 INFO:tasks.thrashosds.thrasher:Setting 2.7 to [2, 1] 2023-07-02T14:02:57.323 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.7', '2', '1'] 2023-07-02T14:02:57.324 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.7 2 1 2023-07-02T14:02:57.405 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:57.404+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:57.434 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:02:57.434 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:02:57.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:02:57.435 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:02:57.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.435 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:163 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:57.435 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.284743+0000", 2023-07-02T14:02:57.435 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.147674491, 2023-07-02T14:02:57.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027177222000000001, 2023-07-02T14:02:57.436 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:57.436 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:57.436 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:57.436 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:57.436 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:57.436 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 163 2023-07-02T14:02:57.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.437 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:57.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:57.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284743+0000", 2023-07-02T14:02:57.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:57.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:57.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284743+0000", 2023-07-02T14:02:57.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:57.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:57.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284742+0000", 2023-07-02T14:02:57.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9390999999999998e-05 2023-07-02T14:02:57.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:57.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284831+0000", 2023-07-02T14:02:57.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6971000000000001e-05 2023-07-02T14:02:57.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:57.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284848+0000", 2023-07-02T14:02:57.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951e-05 2023-07-02T14:02:57.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:57.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284864+0000", 2023-07-02T14:02:57.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015684601999999999 2023-07-02T14:02:57.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:57.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300549+0000", 2023-07-02T14:02:57.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3045000000000003e-05 2023-07-02T14:02:57.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:57.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300622+0000", 2023-07-02T14:02:57.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9452999999999999e-05 2023-07-02T14:02:57.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:57.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300711+0000", 2023-07-02T14:02:57.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059060379999999997 2023-07-02T14:02:57.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:57.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306617+0000", 2023-07-02T14:02:57.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052100280000000002 2023-07-02T14:02:57.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:57.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311827+0000", 2023-07-02T14:02:57.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978999999999998e-05 2023-07-02T14:02:57.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:57.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311879+0000", 2023-07-02T14:02:57.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0349999999999998e-05 2023-07-02T14:02:57.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:57.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311920+0000", 2023-07-02T14:02:57.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027177222000000001 2023-07-02T14:02:57.445 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.445 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.445 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.446 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:165 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:57.446 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.284977+0000", 2023-07-02T14:02:57.446 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.147439867999999, 2023-07-02T14:02:57.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027689147000000001, 2023-07-02T14:02:57.446 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:57.446 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:57.446 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:57.447 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:57.447 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:57.447 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 165 2023-07-02T14:02:57.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.447 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:57.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:57.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284977+0000", 2023-07-02T14:02:57.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:57.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:57.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284977+0000", 2023-07-02T14:02:57.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:57.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:57.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284976+0000", 2023-07-02T14:02:57.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3962999999999996e-05 2023-07-02T14:02:57.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:57.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285060+0000", 2023-07-02T14:02:57.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1260000000000004e-06 2023-07-02T14:02:57.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:57.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285070+0000", 2023-07-02T14:02:57.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0669999999999999e-06 2023-07-02T14:02:57.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:57.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285075+0000", 2023-07-02T14:02:57.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016699978000000001 2023-07-02T14:02:57.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:57.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.301775+0000", 2023-07-02T14:02:57.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6872000000000004e-05 2023-07-02T14:02:57.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:57.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.301862+0000", 2023-07-02T14:02:57.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000155592 2023-07-02T14:02:57.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:57.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.302017+0000", 2023-07-02T14:02:57.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046740250000000001 2023-07-02T14:02:57.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:57.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306691+0000", 2023-07-02T14:02:57.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059023670000000004 2023-07-02T14:02:57.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:57.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312593+0000", 2023-07-02T14:02:57.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6545999999999999e-05 2023-07-02T14:02:57.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:57.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312630+0000", 2023-07-02T14:02:57.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6270999999999999e-05 2023-07-02T14:02:57.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:57.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312666+0000", 2023-07-02T14:02:57.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027689147000000001 2023-07-02T14:02:57.457 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.457 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.457 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.457 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:168 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1507328~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:57.457 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285081+0000", 2023-07-02T14:02:57.458 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.147335857000002, 2023-07-02T14:02:57.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027656613, 2023-07-02T14:02:57.458 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:57.458 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:57.458 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:57.458 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:57.458 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:57.458 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 168 2023-07-02T14:02:57.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.459 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:57.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:57.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285081+0000", 2023-07-02T14:02:57.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:57.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:57.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285081+0000", 2023-07-02T14:02:57.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:57.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:57.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285080+0000", 2023-07-02T14:02:57.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5372999999999998e-05 2023-07-02T14:02:57.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:57.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285156+0000", 2023-07-02T14:02:57.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0099999999999995e-06 2023-07-02T14:02:57.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:57.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285164+0000", 2023-07-02T14:02:57.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3320000000000002e-06 2023-07-02T14:02:57.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:57.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285168+0000", 2023-07-02T14:02:57.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017721316000000001 2023-07-02T14:02:57.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:57.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.302889+0000", 2023-07-02T14:02:57.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9568999999999998e-05 2023-07-02T14:02:57.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:57.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.302949+0000", 2023-07-02T14:02:57.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121902 2023-07-02T14:02:57.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:57.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303071+0000", 2023-07-02T14:02:57.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0071548599999999999 2023-07-02T14:02:57.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:57.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310226+0000", 2023-07-02T14:02:57.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024455520000000001 2023-07-02T14:02:57.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:57.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312671+0000", 2023-07-02T14:02:57.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6958000000000001e-05 2023-07-02T14:02:57.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:57.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312708+0000", 2023-07-02T14:02:57.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9451e-05 2023-07-02T14:02:57.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:57.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312738+0000", 2023-07-02T14:02:57.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027656613 2023-07-02T14:02:57.467 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.467 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.468 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.468 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:171 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1572864~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:57.468 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285174+0000", 2023-07-02T14:02:57.468 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.147243339999999, 2023-07-02T14:02:57.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027618999000000002, 2023-07-02T14:02:57.469 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:57.469 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:57.469 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:57.469 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:57.469 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:57.469 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 171 2023-07-02T14:02:57.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.469 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:57.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:57.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285174+0000", 2023-07-02T14:02:57.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:57.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:57.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285174+0000", 2023-07-02T14:02:57.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:57.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:57.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285173+0000", 2023-07-02T14:02:57.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1789000000000002e-05 2023-07-02T14:02:57.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:57.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285225+0000", 2023-07-02T14:02:57.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6429999999999995e-06 2023-07-02T14:02:57.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:57.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285232+0000", 2023-07-02T14:02:57.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8380000000000001e-06 2023-07-02T14:02:57.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:57.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285237+0000", 2023-07-02T14:02:57.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018182713 2023-07-02T14:02:57.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:57.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303420+0000", 2023-07-02T14:02:57.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0059000000000001e-05 2023-07-02T14:02:57.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:57.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303490+0000", 2023-07-02T14:02:57.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012003 2023-07-02T14:02:57.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:57.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303610+0000", 2023-07-02T14:02:57.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066642150000000002 2023-07-02T14:02:57.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:57.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310274+0000", 2023-07-02T14:02:57.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024681949999999999 2023-07-02T14:02:57.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:57.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312743+0000", 2023-07-02T14:02:57.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5525999999999999e-05 2023-07-02T14:02:57.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:57.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312768+0000", 2023-07-02T14:02:57.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4613999999999998e-05 2023-07-02T14:02:57.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:57.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312793+0000", 2023-07-02T14:02:57.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027618999000000002 2023-07-02T14:02:57.478 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.478 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.478 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.479 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:174 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:57.479 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285243+0000", 2023-07-02T14:02:57.479 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.147174143000001, 2023-07-02T14:02:57.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027607988, 2023-07-02T14:02:57.479 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:57.479 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:57.480 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:57.480 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:57.480 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:57.480 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 174 2023-07-02T14:02:57.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.480 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:57.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:57.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285243+0000", 2023-07-02T14:02:57.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:57.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:57.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285243+0000", 2023-07-02T14:02:57.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:57.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:57.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285242+0000", 2023-07-02T14:02:57.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6173000000000003e-05 2023-07-02T14:02:57.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:57.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285328+0000", 2023-07-02T14:02:57.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5729999999999997e-06 2023-07-02T14:02:57.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:57.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285338+0000", 2023-07-02T14:02:57.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.081e-06 2023-07-02T14:02:57.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:57.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285347+0000", 2023-07-02T14:02:57.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018567367000000001 2023-07-02T14:02:57.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:57.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303914+0000", 2023-07-02T14:02:57.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9522000000000003e-05 2023-07-02T14:02:57.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:57.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303984+0000", 2023-07-02T14:02:57.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013801800000000001 2023-07-02T14:02:57.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:57.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.304122+0000", 2023-07-02T14:02:57.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061898489999999999 2023-07-02T14:02:57.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:57.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310312+0000", 2023-07-02T14:02:57.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024882519999999998 2023-07-02T14:02:57.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:57.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312800+0000", 2023-07-02T14:02:57.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7100000000000001e-05 2023-07-02T14:02:57.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:57.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312827+0000", 2023-07-02T14:02:57.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3771999999999999e-05 2023-07-02T14:02:57.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:57.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312851+0000", 2023-07-02T14:02:57.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027607988 2023-07-02T14:02:57.489 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.489 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.489 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.489 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:177 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:57.489 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285353+0000", 2023-07-02T14:02:57.490 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.147063960000001, 2023-07-02T14:02:57.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027577522, 2023-07-02T14:02:57.490 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:57.490 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:57.490 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:57.490 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:57.490 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:57.490 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 177 2023-07-02T14:02:57.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.491 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:57.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:57.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285353+0000", 2023-07-02T14:02:57.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:57.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:57.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285353+0000", 2023-07-02T14:02:57.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:57.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:57.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285352+0000", 2023-07-02T14:02:57.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7850999999999996e-05 2023-07-02T14:02:57.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:57.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285440+0000", 2023-07-02T14:02:57.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7960000000000001e-06 2023-07-02T14:02:57.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:57.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285449+0000", 2023-07-02T14:02:57.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0529999999999997e-06 2023-07-02T14:02:57.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:57.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285454+0000", 2023-07-02T14:02:57.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018975296999999999 2023-07-02T14:02:57.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:57.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.304429+0000", 2023-07-02T14:02:57.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0682e-05 2023-07-02T14:02:57.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:57.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.304480+0000", 2023-07-02T14:02:57.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116107 2023-07-02T14:02:57.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:57.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.304596+0000", 2023-07-02T14:02:57.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0057486309999999997 2023-07-02T14:02:57.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:57.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310345+0000", 2023-07-02T14:02:57.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025147289999999998 2023-07-02T14:02:57.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:57.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312860+0000", 2023-07-02T14:02:57.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7252999999999997e-05 2023-07-02T14:02:57.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:57.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312897+0000", 2023-07-02T14:02:57.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3705999999999999e-05 2023-07-02T14:02:57.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:57.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312931+0000", 2023-07-02T14:02:57.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027577522 2023-07-02T14:02:57.499 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.499 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.499 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.500 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:180 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:57.500 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285460+0000", 2023-07-02T14:02:57.500 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.146957084, 2023-07-02T14:02:57.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027547589000000001, 2023-07-02T14:02:57.500 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:57.501 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:57.501 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:57.501 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:57.501 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:57.501 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 180 2023-07-02T14:02:57.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.501 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:57.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:57.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285460+0000", 2023-07-02T14:02:57.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:57.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:57.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285460+0000", 2023-07-02T14:02:57.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:57.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:57.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285459+0000", 2023-07-02T14:02:57.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7868999999999997e-05 2023-07-02T14:02:57.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:57.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285527+0000", 2023-07-02T14:02:57.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6710000000000004e-06 2023-07-02T14:02:57.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:57.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285533+0000", 2023-07-02T14:02:57.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9960000000000002e-06 2023-07-02T14:02:57.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:57.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285536+0000", 2023-07-02T14:02:57.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019509826000000001 2023-07-02T14:02:57.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:57.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.305046+0000", 2023-07-02T14:02:57.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3639000000000004e-05 2023-07-02T14:02:57.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:57.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.305119+0000", 2023-07-02T14:02:57.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014125300000000001 2023-07-02T14:02:57.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:57.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.305260+0000", 2023-07-02T14:02:57.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051273919999999997 2023-07-02T14:02:57.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:57.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310388+0000", 2023-07-02T14:02:57.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025505390000000001 2023-07-02T14:02:57.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:57.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312938+0000", 2023-07-02T14:02:57.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5608999999999997e-05 2023-07-02T14:02:57.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:57.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312974+0000", 2023-07-02T14:02:57.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3562999999999997e-05 2023-07-02T14:02:57.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:57.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313008+0000", 2023-07-02T14:02:57.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027547589000000001 2023-07-02T14:02:57.510 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.510 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.510 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.510 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:183 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:57.510 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:02:57.511 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.146877789000001, 2023-07-02T14:02:57.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003, 2023-07-02T14:02:57.511 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:57.511 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:57.511 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:57.511 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:57.511 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:57.511 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 183 2023-07-02T14:02:57.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.512 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:57.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:57.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:02:57.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:57.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:57.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:02:57.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:57.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:57.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:02:57.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9790999999999997e-05 2023-07-02T14:02:57.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:57.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285598+0000", 2023-07-02T14:02:57.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5809999999999996e-06 2023-07-02T14:02:57.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:57.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285604+0000", 2023-07-02T14:02:57.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7599999999999998e-06 2023-07-02T14:02:57.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:57.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285607+0000", 2023-07-02T14:02:57.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021108188999999999 2023-07-02T14:02:57.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:57.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306715+0000", 2023-07-02T14:02:57.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6548e-05 2023-07-02T14:02:57.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:57.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306792+0000", 2023-07-02T14:02:57.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173869 2023-07-02T14:02:57.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:57.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306965+0000", 2023-07-02T14:02:57.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060498009999999996 2023-07-02T14:02:57.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:57.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313015+0000", 2023-07-02T14:02:57.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049443719999999998 2023-07-02T14:02:57.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:57.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317960+0000", 2023-07-02T14:02:57.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5927999999999998e-05 2023-07-02T14:02:57.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:57.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318006+0000", 2023-07-02T14:02:57.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6004999999999998e-05 2023-07-02T14:02:57.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:57.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318042+0000", 2023-07-02T14:02:57.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003 2023-07-02T14:02:57.520 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.520 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.520 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.521 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:186 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:57.521 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:02:57.521 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.146806963, 2023-07-02T14:02:57.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003, 2023-07-02T14:02:57.521 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:57.522 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:57.522 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:57.522 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:57.522 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:57.522 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 186 2023-07-02T14:02:57.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.522 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:57.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:57.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:02:57.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:57.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:57.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:02:57.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:57.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:57.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:02:57.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4279e-05 2023-07-02T14:02:57.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:57.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285664+0000", 2023-07-02T14:02:57.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3789999999999995e-06 2023-07-02T14:02:57.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:57.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285673+0000", 2023-07-02T14:02:57.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499999999999998e-06 2023-07-02T14:02:57.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:57.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285678+0000", 2023-07-02T14:02:57.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023261947000000002 2023-07-02T14:02:57.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:57.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.308940+0000", 2023-07-02T14:02:57.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010323399999999999 2023-07-02T14:02:57.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:57.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309044+0000", 2023-07-02T14:02:57.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018878300000000001 2023-07-02T14:02:57.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:57.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309232+0000", 2023-07-02T14:02:57.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037874150000000001 2023-07-02T14:02:57.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:57.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313020+0000", 2023-07-02T14:02:57.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050976140000000003 2023-07-02T14:02:57.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:57.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318117+0000", 2023-07-02T14:02:57.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3182000000000001e-05 2023-07-02T14:02:57.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:57.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:02:57.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5055999999999998e-05 2023-07-02T14:02:57.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:57.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318196+0000", 2023-07-02T14:02:57.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003 2023-07-02T14:02:57.531 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.531 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.532 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.532 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:189 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:57.532 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:02:57.532 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.1467332, 2023-07-02T14:02:57.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997, 2023-07-02T14:02:57.532 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:57.533 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:57.533 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:57.533 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:57.533 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:57.533 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:02:57.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.533 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:57.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:57.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:02:57.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:57.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:57.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:02:57.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:57.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:57.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285683+0000", 2023-07-02T14:02:57.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8451999999999999e-05 2023-07-02T14:02:57.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:57.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285782+0000", 2023-07-02T14:02:57.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3650000000000004e-06 2023-07-02T14:02:57.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:57.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285790+0000", 2023-07-02T14:02:57.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8230000000000002e-06 2023-07-02T14:02:57.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:57.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285795+0000", 2023-07-02T14:02:57.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024619367999999999 2023-07-02T14:02:57.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:57.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310414+0000", 2023-07-02T14:02:57.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014244600000000001 2023-07-02T14:02:57.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:57.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310557+0000", 2023-07-02T14:02:57.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020612699999999999 2023-07-02T14:02:57.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:57.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310763+0000", 2023-07-02T14:02:57.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003427687 2023-07-02T14:02:57.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:57.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314191+0000", 2023-07-02T14:02:57.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040492089999999998 2023-07-02T14:02:57.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:57.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318240+0000", 2023-07-02T14:02:57.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0287999999999999e-05 2023-07-02T14:02:57.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:57.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318270+0000", 2023-07-02T14:02:57.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0868000000000002e-05 2023-07-02T14:02:57.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:57.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318311+0000", 2023-07-02T14:02:57.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997 2023-07-02T14:02:57.542 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.542 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.542 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.542 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:192 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:57.543 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:02:57.543 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.146616427000001, 2023-07-02T14:02:57.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003, 2023-07-02T14:02:57.543 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:57.543 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:57.543 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:57.543 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:57.544 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:57.544 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 192 2023-07-02T14:02:57.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.544 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:57.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:57.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:02:57.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:57.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:57.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:02:57.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:57.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:57.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285800+0000", 2023-07-02T14:02:57.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100417 2023-07-02T14:02:57.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:57.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285900+0000", 2023-07-02T14:02:57.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8244999999999999e-05 2023-07-02T14:02:57.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:57.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285919+0000", 2023-07-02T14:02:57.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2499999999999997e-06 2023-07-02T14:02:57.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:57.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285924+0000", 2023-07-02T14:02:57.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025288155999999999 2023-07-02T14:02:57.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:57.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311212+0000", 2023-07-02T14:02:57.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4073999999999996e-05 2023-07-02T14:02:57.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:57.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311286+0000", 2023-07-02T14:02:57.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016219699999999999 2023-07-02T14:02:57.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:57.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311448+0000", 2023-07-02T14:02:57.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066020599999999999 2023-07-02T14:02:57.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:57.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318050+0000", 2023-07-02T14:02:57.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000327986 2023-07-02T14:02:57.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:57.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318378+0000", 2023-07-02T14:02:57.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8693999999999998e-05 2023-07-02T14:02:57.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:57.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318427+0000", 2023-07-02T14:02:57.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6656999999999998e-05 2023-07-02T14:02:57.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:57.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318464+0000", 2023-07-02T14:02:57.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003 2023-07-02T14:02:57.552 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.552 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.553 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.553 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:195 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:57.553 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:02:57.553 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.146484922999999, 2023-07-02T14:02:57.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001, 2023-07-02T14:02:57.553 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:57.554 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:57.554 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:57.554 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:57.554 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:57.554 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 195 2023-07-02T14:02:57.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.554 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:57.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:57.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:02:57.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:57.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:57.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:02:57.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:57.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:57.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285931+0000", 2023-07-02T14:02:57.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4530999999999995e-05 2023-07-02T14:02:57.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:57.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286006+0000", 2023-07-02T14:02:57.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4113e-05 2023-07-02T14:02:57.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:57.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286020+0000", 2023-07-02T14:02:57.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7459999999999997e-06 2023-07-02T14:02:57.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:57.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286026+0000", 2023-07-02T14:02:57.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025997769 2023-07-02T14:02:57.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:57.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312023+0000", 2023-07-02T14:02:57.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4403000000000001e-05 2023-07-02T14:02:57.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:57.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312078+0000", 2023-07-02T14:02:57.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020581399999999999 2023-07-02T14:02:57.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:57.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312284+0000", 2023-07-02T14:02:57.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005771043 2023-07-02T14:02:57.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:57.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318055+0000", 2023-07-02T14:02:57.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00046015200000000002 2023-07-02T14:02:57.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:57.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318515+0000", 2023-07-02T14:02:57.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9444000000000003e-05 2023-07-02T14:02:57.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:57.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318564+0000", 2023-07-02T14:02:57.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2141000000000002e-05 2023-07-02T14:02:57.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:57.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318606+0000", 2023-07-02T14:02:57.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001 2023-07-02T14:02:57.563 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.563 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.563 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.563 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:198 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:02:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.146384963999999, 2023-07-02T14:02:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999, 2023-07-02T14:02:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 198 2023-07-02T14:02:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:02:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:02:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286031+0000", 2023-07-02T14:02:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00049286700000000002 2023-07-02T14:02:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286524+0000", 2023-07-02T14:02:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4938e-05 2023-07-02T14:02:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286539+0000", 2023-07-02T14:02:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0680000000000002e-06 2023-07-02T14:02:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286545+0000", 2023-07-02T14:02:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026497718 2023-07-02T14:02:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313043+0000", 2023-07-02T14:02:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0884999999999998e-05 2023-07-02T14:02:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313094+0000", 2023-07-02T14:02:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014109999999999999 2023-07-02T14:02:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313235+0000", 2023-07-02T14:02:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048258980000000003 2023-07-02T14:02:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318061+0000", 2023-07-02T14:02:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014346350000000001 2023-07-02T14:02:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319495+0000", 2023-07-02T14:02:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1986e-05 2023-07-02T14:02:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319547+0000", 2023-07-02T14:02:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0457e-05 2023-07-02T14:02:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319578+0000", 2023-07-02T14:02:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999 2023-07-02T14:02:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:201 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:02:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.145864539000002, 2023-07-02T14:02:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003, 2023-07-02T14:02:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 201 2023-07-02T14:02:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:02:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:02:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:02:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019213999999999999 2023-07-02T14:02:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286744+0000", 2023-07-02T14:02:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5279999999999992e-06 2023-07-02T14:02:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286753+0000", 2023-07-02T14:02:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4200000000000004e-06 2023-07-02T14:02:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286760+0000", 2023-07-02T14:02:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026818386999999999 2023-07-02T14:02:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313578+0000", 2023-07-02T14:02:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015965 2023-07-02T14:02:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313738+0000", 2023-07-02T14:02:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015767299999999999 2023-07-02T14:02:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313895+0000", 2023-07-02T14:02:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041714930000000001 2023-07-02T14:02:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318067+0000", 2023-07-02T14:02:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001561698 2023-07-02T14:02:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319629+0000", 2023-07-02T14:02:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2346000000000003e-05 2023-07-02T14:02:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319681+0000", 2023-07-02T14:02:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4691000000000003e-05 2023-07-02T14:02:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319726+0000", 2023-07-02T14:02:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003 2023-07-02T14:02:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:204 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:02:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.145649701, 2023-07-02T14:02:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998, 2023-07-02T14:02:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 204 2023-07-02T14:02:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:02:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:02:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:02:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2497999999999998e-05 2023-07-02T14:02:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286819+0000", 2023-07-02T14:02:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0099999999999995e-06 2023-07-02T14:02:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286827+0000", 2023-07-02T14:02:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000003e-06 2023-07-02T14:02:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286833+0000", 2023-07-02T14:02:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027384716999999999 2023-07-02T14:02:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314218+0000", 2023-07-02T14:02:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6204000000000004e-05 2023-07-02T14:02:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314294+0000", 2023-07-02T14:02:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015757499999999999 2023-07-02T14:02:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314452+0000", 2023-07-02T14:02:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036215560000000002 2023-07-02T14:02:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318073+0000", 2023-07-02T14:02:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017023979999999999 2023-07-02T14:02:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319776+0000", 2023-07-02T14:02:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9561000000000002e-05 2023-07-02T14:02:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319825+0000", 2023-07-02T14:02:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0329e-05 2023-07-02T14:02:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319866+0000", 2023-07-02T14:02:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998 2023-07-02T14:02:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:207 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:02:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.145575055999998, 2023-07-02T14:02:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003, 2023-07-02T14:02:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 207 2023-07-02T14:02:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:02:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:02:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286841+0000", 2023-07-02T14:02:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00068375100000000002 2023-07-02T14:02:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287525+0000", 2023-07-02T14:02:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1601e-05 2023-07-02T14:02:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287537+0000", 2023-07-02T14:02:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6409999999999996e-06 2023-07-02T14:02:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:57.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287543+0000", 2023-07-02T14:02:57.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027245844000000002 2023-07-02T14:02:57.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:57.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314789+0000", 2023-07-02T14:02:57.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8910000000000003e-05 2023-07-02T14:02:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314858+0000", 2023-07-02T14:02:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030992599999999997 2023-07-02T14:02:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315168+0000", 2023-07-02T14:02:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029122869999999999 2023-07-02T14:02:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318080+0000", 2023-07-02T14:02:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019963020000000001 2023-07-02T14:02:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320077+0000", 2023-07-02T14:02:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5116000000000005e-05 2023-07-02T14:02:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320142+0000", 2023-07-02T14:02:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3793000000000004e-05 2023-07-02T14:02:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320205+0000", 2023-07-02T14:02:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003 2023-07-02T14:02:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:236 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2424832~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:02:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.144862873000001, 2023-07-02T14:02:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003, 2023-07-02T14:02:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2023-07-02T14:02:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:02:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:02:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287553+0000", 2023-07-02T14:02:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0667e-05 2023-07-02T14:02:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:57.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287614+0000", 2023-07-02T14:02:57.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0454e-05 2023-07-02T14:02:57.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:57.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287624+0000", 2023-07-02T14:02:57.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8564999999999999e-05 2023-07-02T14:02:57.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:57.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287643+0000", 2023-07-02T14:02:57.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027977596 2023-07-02T14:02:57.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315621+0000", 2023-07-02T14:02:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000201728 2023-07-02T14:02:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315822+0000", 2023-07-02T14:02:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018436599999999999 2023-07-02T14:02:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316007+0000", 2023-07-02T14:02:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042423440000000003 2023-07-02T14:02:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320249+0000", 2023-07-02T14:02:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001452633 2023-07-02T14:02:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321702+0000", 2023-07-02T14:02:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3616000000000003e-05 2023-07-02T14:02:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321755+0000", 2023-07-02T14:02:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840999999999999e-05 2023-07-02T14:02:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321793+0000", 2023-07-02T14:02:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003 2023-07-02T14:02:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:251 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:02:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.144767336000001, 2023-07-02T14:02:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001, 2023-07-02T14:02:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 251 2023-07-02T14:02:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:02:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:02:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287649+0000", 2023-07-02T14:02:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3232000000000001e-05 2023-07-02T14:02:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287702+0000", 2023-07-02T14:02:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0010000000000007e-06 2023-07-02T14:02:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287711+0000", 2023-07-02T14:02:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3569999999999997e-06 2023-07-02T14:02:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287717+0000", 2023-07-02T14:02:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028686454 2023-07-02T14:02:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316403+0000", 2023-07-02T14:02:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7741000000000002e-05 2023-07-02T14:02:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316461+0000", 2023-07-02T14:02:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4313000000000004e-05 2023-07-02T14:02:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316545+0000", 2023-07-02T14:02:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003734032 2023-07-02T14:02:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320279+0000", 2023-07-02T14:02:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015237219999999999 2023-07-02T14:02:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321803+0000", 2023-07-02T14:02:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1927e-05 2023-07-02T14:02:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321845+0000", 2023-07-02T14:02:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5398999999999998e-05 2023-07-02T14:02:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321880+0000", 2023-07-02T14:02:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001 2023-07-02T14:02:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:255 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2555904~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:02:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.144352607999998, 2023-07-02T14:02:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002, 2023-07-02T14:02:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 255 2023-07-02T14:02:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:02:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:02:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:02:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288063+0000", 2023-07-02T14:02:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4459999999999997e-05 2023-07-02T14:02:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288118+0000", 2023-07-02T14:02:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0998999999999999e-05 2023-07-02T14:02:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288129+0000", 2023-07-02T14:02:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3219999999999998e-06 2023-07-02T14:02:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288135+0000", 2023-07-02T14:02:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029137386000000001 2023-07-02T14:02:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317273+0000", 2023-07-02T14:02:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7819999999999997e-05 2023-07-02T14:02:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317350+0000", 2023-07-02T14:02:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000174864 2023-07-02T14:02:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317525+0000", 2023-07-02T14:02:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027943519999999999 2023-07-02T14:02:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320320+0000", 2023-07-02T14:02:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015681460000000001 2023-07-02T14:02:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321888+0000", 2023-07-02T14:02:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9508999999999997e-05 2023-07-02T14:02:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321927+0000", 2023-07-02T14:02:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5428e-05 2023-07-02T14:02:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:57.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321963+0000", 2023-07-02T14:02:57.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002 2023-07-02T14:02:57.636 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.636 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.636 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.636 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:446 2.1 2:910b2a90:::rbd_data.125de1449605.000000000000010a:head [write 65536~3211264] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:02:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:02:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.067072855999999, 2023-07-02T14:02:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003, 2023-07-02T14:02:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:02:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:02:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:02:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:02:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:02:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 446 2023-07-02T14:02:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:02:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:02:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:02:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:02:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:02:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:02:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:02:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:02:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:02:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015907372999999999 2023-07-02T14:02:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:02:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381251+0000", 2023-07-02T14:02:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5527000000000001e-05 2023-07-02T14:02:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:02:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381277+0000", 2023-07-02T14:02:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9958000000000001e-05 2023-07-02T14:02:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:02:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381297+0000", 2023-07-02T14:02:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4941000000000001e-05 2023-07-02T14:02:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:02:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381352+0000", 2023-07-02T14:02:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011565699999999999 2023-07-02T14:02:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:02:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381467+0000", 2023-07-02T14:02:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018039799999999999 2023-07-02T14:02:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:02:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381648+0000", 2023-07-02T14:02:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088183559999999994 2023-07-02T14:02:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:02:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.390466+0000", 2023-07-02T14:02:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007278278 2023-07-02T14:02:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:02:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397744+0000", 2023-07-02T14:02:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9289000000000001e-05 2023-07-02T14:02:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:02:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397804+0000", 2023-07-02T14:02:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0479000000000002e-05 2023-07-02T14:02:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:02:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:02:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:02:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397854+0000", 2023-07-02T14:02:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003 2023-07-02T14:02:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.647 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:02:57.647 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:02:57.647 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:02:57.648 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:02:57.649 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:57.648+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:57.730 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:02:57.749 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:57.748+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:57.786 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:02:57.844 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:02:57.844 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:02:57.844 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:02:57.845 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:02:57.869 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:02:57.886 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:02:57.887 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:57.886+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:57.904 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:02:57.922 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:02:57.940 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:02:57.950 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:57.949+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:57.957 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:02:57.958 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:02:57.959 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:02:57.959 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:02:57.959 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:02:57.959 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:02:57.959 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:02:58.013 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:58.013 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:58.023 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:02:58.064 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:58.063+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:58.096 INFO:teuthology.orchestra.run.smithi016.stderr:set 2.7 pg_upmap mapping to [2,1] 2023-07-02T14:02:58.105 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap2 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap3 b3fbb028-e267-4d25-a750-93ba18a2316a@snap3.diff 2023-07-02T14:02:58.148 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:02:58.189 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:58.190 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:58.192 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:58.191+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:58.200 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:02:58.253 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:58.252+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:58.324 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:58.324 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:58.335 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:02:58.378 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:58.377+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:58.455 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:58.454+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:58.458 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:58.458 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:58.468 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:02:58.556 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:58.555+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:58.633 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:58.633 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:58.643 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:02:58.684 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:58.683+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:58.758 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:58.757+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:58.807 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:58.807 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:58.817 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:02:58.858 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:58.857+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:58.959 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:58.958+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:58.980 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:58.980 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:58.990 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:02:59.060 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:02:59.059+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:02:59.102 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:02:59.115 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:59.115 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:59.125 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:02:59.199 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:59.198+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:59.261 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:59.261+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:59.285 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:59.285 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:59.296 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:02:59.361 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:59.361+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:59.459 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:59.460 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:59.462 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:02:59.461+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:02:59.470 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:02:59.563 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:02:59.562+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:02:59.635 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:59.635 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:59.645 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:02:59.663 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:02:59.663+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:02:59.764 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:59.764+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:59.809 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:59.810 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:59.820 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:02:59.865 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:02:59.864+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:02:59.942 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:02:59.943 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:02:59.953 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:02:59.966 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:02:59.965+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:02:59.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:03:00.066 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:00.065+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:00.133 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:00.133 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:00.139 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 51% complete...2023-07-02T14:03:00.139+0000 7fec2bfff700 0 -- 172.21.15.163:0/1691066546 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x560b0ef9c9b0 msgr2=0x560b0ef9eda0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:03:00.143 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 100% complete...done. 2023-07-02T14:03:00.146 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:03:00.147 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=4 2023-07-02T14:03:00.147 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap4 2023-07-02T14:03:00.149 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:03:00.150 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:00.167 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:00.166+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:00.173 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:00.195 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:00.216 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:00.252 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:00.268 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:00.267+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:00.286 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:00.287 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:03:00.288 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:03:00.288 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:03:00.288 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:03:00.288 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:03:00.288 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:03:00.318 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:00.319 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:00.319 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:00.319 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:00.329 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:03:00.370 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:00.369+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:00.469 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:00.468+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:00.493 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:00.493 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:00.493 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:00.494 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:00.494 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:00.503 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:03:00.570 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:00.570+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:00.630 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:00.630 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:03:00.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:03:00.631 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:00.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.631 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:163 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:00.631 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.284743+0000", 2023-07-02T14:03:00.631 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.343135895, 2023-07-02T14:03:00.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027177222000000001, 2023-07-02T14:03:00.631 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:00.632 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:00.632 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:00.632 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:00.632 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:00.632 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 163 2023-07-02T14:03:00.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.632 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:00.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:00.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284743+0000", 2023-07-02T14:03:00.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:00.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:00.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284743+0000", 2023-07-02T14:03:00.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:00.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:00.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284742+0000", 2023-07-02T14:03:00.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9390999999999998e-05 2023-07-02T14:03:00.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:00.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284831+0000", 2023-07-02T14:03:00.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6971000000000001e-05 2023-07-02T14:03:00.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:00.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284848+0000", 2023-07-02T14:03:00.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5951e-05 2023-07-02T14:03:00.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:00.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284864+0000", 2023-07-02T14:03:00.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015684601999999999 2023-07-02T14:03:00.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:00.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300549+0000", 2023-07-02T14:03:00.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3045000000000003e-05 2023-07-02T14:03:00.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:00.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300622+0000", 2023-07-02T14:03:00.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9452999999999999e-05 2023-07-02T14:03:00.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:00.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.300711+0000", 2023-07-02T14:03:00.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059060379999999997 2023-07-02T14:03:00.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:00.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306617+0000", 2023-07-02T14:03:00.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052100280000000002 2023-07-02T14:03:00.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:00.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311827+0000", 2023-07-02T14:03:00.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978999999999998e-05 2023-07-02T14:03:00.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:00.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311879+0000", 2023-07-02T14:03:00.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0349999999999998e-05 2023-07-02T14:03:00.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:00.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311920+0000", 2023-07-02T14:03:00.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027177222000000001 2023-07-02T14:03:00.641 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.641 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.641 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.641 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:165 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:00.642 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.284977+0000", 2023-07-02T14:03:00.642 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.342901271999999, 2023-07-02T14:03:00.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027689147000000001, 2023-07-02T14:03:00.642 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:00.642 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:00.642 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:00.642 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:00.642 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:00.643 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 165 2023-07-02T14:03:00.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.643 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:00.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:00.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284977+0000", 2023-07-02T14:03:00.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:00.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:00.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284977+0000", 2023-07-02T14:03:00.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:00.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:00.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284976+0000", 2023-07-02T14:03:00.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3962999999999996e-05 2023-07-02T14:03:00.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:00.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285060+0000", 2023-07-02T14:03:00.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1260000000000004e-06 2023-07-02T14:03:00.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:00.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285070+0000", 2023-07-02T14:03:00.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0669999999999999e-06 2023-07-02T14:03:00.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:00.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285075+0000", 2023-07-02T14:03:00.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016699978000000001 2023-07-02T14:03:00.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:00.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.301775+0000", 2023-07-02T14:03:00.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6872000000000004e-05 2023-07-02T14:03:00.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:00.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.301862+0000", 2023-07-02T14:03:00.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000155592 2023-07-02T14:03:00.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:00.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.302017+0000", 2023-07-02T14:03:00.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046740250000000001 2023-07-02T14:03:00.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:00.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306691+0000", 2023-07-02T14:03:00.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059023670000000004 2023-07-02T14:03:00.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:00.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312593+0000", 2023-07-02T14:03:00.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6545999999999999e-05 2023-07-02T14:03:00.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:00.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312630+0000", 2023-07-02T14:03:00.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6270999999999999e-05 2023-07-02T14:03:00.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:00.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312666+0000", 2023-07-02T14:03:00.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027689147000000001 2023-07-02T14:03:00.651 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.651 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.651 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.652 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:168 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1507328~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:00.652 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285081+0000", 2023-07-02T14:03:00.652 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.342797261000001, 2023-07-02T14:03:00.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027656613, 2023-07-02T14:03:00.652 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:00.652 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:00.653 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:00.653 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:00.653 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:00.653 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 168 2023-07-02T14:03:00.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.653 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:00.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:00.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285081+0000", 2023-07-02T14:03:00.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:00.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:00.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285081+0000", 2023-07-02T14:03:00.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:00.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:00.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285080+0000", 2023-07-02T14:03:00.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5372999999999998e-05 2023-07-02T14:03:00.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:00.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285156+0000", 2023-07-02T14:03:00.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0099999999999995e-06 2023-07-02T14:03:00.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:00.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285164+0000", 2023-07-02T14:03:00.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3320000000000002e-06 2023-07-02T14:03:00.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:00.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285168+0000", 2023-07-02T14:03:00.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017721316000000001 2023-07-02T14:03:00.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.302889+0000", 2023-07-02T14:03:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9568999999999998e-05 2023-07-02T14:03:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.302949+0000", 2023-07-02T14:03:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121902 2023-07-02T14:03:00.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303071+0000", 2023-07-02T14:03:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0071548599999999999 2023-07-02T14:03:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:00.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310226+0000", 2023-07-02T14:03:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024455520000000001 2023-07-02T14:03:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312671+0000", 2023-07-02T14:03:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6958000000000001e-05 2023-07-02T14:03:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312708+0000", 2023-07-02T14:03:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9451e-05 2023-07-02T14:03:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312738+0000", 2023-07-02T14:03:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027656613 2023-07-02T14:03:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:171 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1572864~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285174+0000", 2023-07-02T14:03:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.342704743999999, 2023-07-02T14:03:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027618999000000002, 2023-07-02T14:03:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 171 2023-07-02T14:03:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285174+0000", 2023-07-02T14:03:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285174+0000", 2023-07-02T14:03:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285173+0000", 2023-07-02T14:03:00.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1789000000000002e-05 2023-07-02T14:03:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285225+0000", 2023-07-02T14:03:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6429999999999995e-06 2023-07-02T14:03:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285232+0000", 2023-07-02T14:03:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8380000000000001e-06 2023-07-02T14:03:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285237+0000", 2023-07-02T14:03:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018182713 2023-07-02T14:03:00.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:00.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303420+0000", 2023-07-02T14:03:00.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0059000000000001e-05 2023-07-02T14:03:00.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:00.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303490+0000", 2023-07-02T14:03:00.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012003 2023-07-02T14:03:00.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:00.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303610+0000", 2023-07-02T14:03:00.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066642150000000002 2023-07-02T14:03:00.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310274+0000", 2023-07-02T14:03:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024681949999999999 2023-07-02T14:03:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:00.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312743+0000", 2023-07-02T14:03:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5525999999999999e-05 2023-07-02T14:03:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312768+0000", 2023-07-02T14:03:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4613999999999998e-05 2023-07-02T14:03:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:00.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312793+0000", 2023-07-02T14:03:00.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027618999000000002 2023-07-02T14:03:00.672 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.672 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.672 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:174 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285243+0000", 2023-07-02T14:03:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.342635547, 2023-07-02T14:03:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027607988, 2023-07-02T14:03:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:00.673 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:00.674 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:00.674 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 174 2023-07-02T14:03:00.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.674 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:00.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:00.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285243+0000", 2023-07-02T14:03:00.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285243+0000", 2023-07-02T14:03:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:00.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285242+0000", 2023-07-02T14:03:00.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6173000000000003e-05 2023-07-02T14:03:00.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:00.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285328+0000", 2023-07-02T14:03:00.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5729999999999997e-06 2023-07-02T14:03:00.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:00.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285338+0000", 2023-07-02T14:03:00.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.081e-06 2023-07-02T14:03:00.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:00.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285347+0000", 2023-07-02T14:03:00.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018567367000000001 2023-07-02T14:03:00.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:00.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303914+0000", 2023-07-02T14:03:00.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9522000000000003e-05 2023-07-02T14:03:00.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:00.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303984+0000", 2023-07-02T14:03:00.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013801800000000001 2023-07-02T14:03:00.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:00.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.304122+0000", 2023-07-02T14:03:00.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061898489999999999 2023-07-02T14:03:00.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:00.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310312+0000", 2023-07-02T14:03:00.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024882519999999998 2023-07-02T14:03:00.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:00.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312800+0000", 2023-07-02T14:03:00.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7100000000000001e-05 2023-07-02T14:03:00.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:00.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312827+0000", 2023-07-02T14:03:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3771999999999999e-05 2023-07-02T14:03:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312851+0000", 2023-07-02T14:03:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027607988 2023-07-02T14:03:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.682 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:177 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285353+0000", 2023-07-02T14:03:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.342525364, 2023-07-02T14:03:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027577522, 2023-07-02T14:03:00.683 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:00.684 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:00.684 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:00.684 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:00.684 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:00.684 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 177 2023-07-02T14:03:00.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.684 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:00.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:00.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285353+0000", 2023-07-02T14:03:00.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:00.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:00.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285353+0000", 2023-07-02T14:03:00.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:00.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:00.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285352+0000", 2023-07-02T14:03:00.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7850999999999996e-05 2023-07-02T14:03:00.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:00.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285440+0000", 2023-07-02T14:03:00.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7960000000000001e-06 2023-07-02T14:03:00.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:00.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285449+0000", 2023-07-02T14:03:00.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0529999999999997e-06 2023-07-02T14:03:00.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:00.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285454+0000", 2023-07-02T14:03:00.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018975296999999999 2023-07-02T14:03:00.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:00.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.304429+0000", 2023-07-02T14:03:00.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0682e-05 2023-07-02T14:03:00.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:00.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.304480+0000", 2023-07-02T14:03:00.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116107 2023-07-02T14:03:00.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.304596+0000", 2023-07-02T14:03:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0057486309999999997 2023-07-02T14:03:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:00.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310345+0000", 2023-07-02T14:03:00.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025147289999999998 2023-07-02T14:03:00.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:00.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312860+0000", 2023-07-02T14:03:00.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7252999999999997e-05 2023-07-02T14:03:00.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:00.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312897+0000", 2023-07-02T14:03:00.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3705999999999999e-05 2023-07-02T14:03:00.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:00.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312931+0000", 2023-07-02T14:03:00.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027577522 2023-07-02T14:03:00.693 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.693 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.693 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.693 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:180 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:00.693 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285460+0000", 2023-07-02T14:03:00.693 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.342418488, 2023-07-02T14:03:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027547589000000001, 2023-07-02T14:03:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 180 2023-07-02T14:03:00.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.695 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:00.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:00.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285460+0000", 2023-07-02T14:03:00.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:00.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285460+0000", 2023-07-02T14:03:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285459+0000", 2023-07-02T14:03:00.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7868999999999997e-05 2023-07-02T14:03:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285527+0000", 2023-07-02T14:03:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6710000000000004e-06 2023-07-02T14:03:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285533+0000", 2023-07-02T14:03:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9960000000000002e-06 2023-07-02T14:03:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285536+0000", 2023-07-02T14:03:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019509826000000001 2023-07-02T14:03:00.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.305046+0000", 2023-07-02T14:03:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3639000000000004e-05 2023-07-02T14:03:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:00.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.305119+0000", 2023-07-02T14:03:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014125300000000001 2023-07-02T14:03:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.305260+0000", 2023-07-02T14:03:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051273919999999997 2023-07-02T14:03:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310388+0000", 2023-07-02T14:03:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025505390000000001 2023-07-02T14:03:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312938+0000", 2023-07-02T14:03:00.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5608999999999997e-05 2023-07-02T14:03:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312974+0000", 2023-07-02T14:03:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3562999999999997e-05 2023-07-02T14:03:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313008+0000", 2023-07-02T14:03:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027547589000000001 2023-07-02T14:03:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:183 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.342339193000001, 2023-07-02T14:03:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003, 2023-07-02T14:03:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:00.704 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 183 2023-07-02T14:03:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:00.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9790999999999997e-05 2023-07-02T14:03:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285598+0000", 2023-07-02T14:03:00.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5809999999999996e-06 2023-07-02T14:03:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285604+0000", 2023-07-02T14:03:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7599999999999998e-06 2023-07-02T14:03:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285607+0000", 2023-07-02T14:03:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021108188999999999 2023-07-02T14:03:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306715+0000", 2023-07-02T14:03:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6548e-05 2023-07-02T14:03:00.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306792+0000", 2023-07-02T14:03:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173869 2023-07-02T14:03:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:00.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306965+0000", 2023-07-02T14:03:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060498009999999996 2023-07-02T14:03:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313015+0000", 2023-07-02T14:03:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049443719999999998 2023-07-02T14:03:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317960+0000", 2023-07-02T14:03:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5927999999999998e-05 2023-07-02T14:03:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:00.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318006+0000", 2023-07-02T14:03:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6004999999999998e-05 2023-07-02T14:03:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318042+0000", 2023-07-02T14:03:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003 2023-07-02T14:03:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.713 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:186 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.342268366999999, 2023-07-02T14:03:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003, 2023-07-02T14:03:00.714 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 186 2023-07-02T14:03:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:00.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:00.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4279e-05 2023-07-02T14:03:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:00.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285664+0000", 2023-07-02T14:03:00.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3789999999999995e-06 2023-07-02T14:03:00.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:00.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285673+0000", 2023-07-02T14:03:00.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499999999999998e-06 2023-07-02T14:03:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285678+0000", 2023-07-02T14:03:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023261947000000002 2023-07-02T14:03:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.308940+0000", 2023-07-02T14:03:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010323399999999999 2023-07-02T14:03:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309044+0000", 2023-07-02T14:03:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018878300000000001 2023-07-02T14:03:00.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:00.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309232+0000", 2023-07-02T14:03:00.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037874150000000001 2023-07-02T14:03:00.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:00.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313020+0000", 2023-07-02T14:03:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050976140000000003 2023-07-02T14:03:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318117+0000", 2023-07-02T14:03:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3182000000000001e-05 2023-07-02T14:03:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5055999999999998e-05 2023-07-02T14:03:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318196+0000", 2023-07-02T14:03:00.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003 2023-07-02T14:03:00.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.724 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.724 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:189 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:00.724 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:03:00.724 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.342194603999999, 2023-07-02T14:03:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997, 2023-07-02T14:03:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:03:00.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.726 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:00.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:00.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:03:00.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:00.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:00.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:03:00.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:00.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:00.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285683+0000", 2023-07-02T14:03:00.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8451999999999999e-05 2023-07-02T14:03:00.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:00.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285782+0000", 2023-07-02T14:03:00.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3650000000000004e-06 2023-07-02T14:03:00.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:00.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285790+0000", 2023-07-02T14:03:00.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8230000000000002e-06 2023-07-02T14:03:00.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:00.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285795+0000", 2023-07-02T14:03:00.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024619367999999999 2023-07-02T14:03:00.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:00.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310414+0000", 2023-07-02T14:03:00.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014244600000000001 2023-07-02T14:03:00.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:00.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310557+0000", 2023-07-02T14:03:00.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020612699999999999 2023-07-02T14:03:00.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:00.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310763+0000", 2023-07-02T14:03:00.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003427687 2023-07-02T14:03:00.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:00.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314191+0000", 2023-07-02T14:03:00.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040492089999999998 2023-07-02T14:03:00.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:00.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318240+0000", 2023-07-02T14:03:00.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0287999999999999e-05 2023-07-02T14:03:00.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:00.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318270+0000", 2023-07-02T14:03:00.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0868000000000002e-05 2023-07-02T14:03:00.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:00.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318311+0000", 2023-07-02T14:03:00.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997 2023-07-02T14:03:00.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.734 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.735 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:192 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:00.735 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:03:00.735 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.342077831000001, 2023-07-02T14:03:00.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003, 2023-07-02T14:03:00.735 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:00.735 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:00.735 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:00.735 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:00.736 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:00.736 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 192 2023-07-02T14:03:00.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.736 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:00.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:00.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:03:00.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:00.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:00.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:03:00.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:00.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:00.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285800+0000", 2023-07-02T14:03:00.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100417 2023-07-02T14:03:00.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:00.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285900+0000", 2023-07-02T14:03:00.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8244999999999999e-05 2023-07-02T14:03:00.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:00.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285919+0000", 2023-07-02T14:03:00.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2499999999999997e-06 2023-07-02T14:03:00.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:00.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285924+0000", 2023-07-02T14:03:00.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025288155999999999 2023-07-02T14:03:00.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:00.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311212+0000", 2023-07-02T14:03:00.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4073999999999996e-05 2023-07-02T14:03:00.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:00.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311286+0000", 2023-07-02T14:03:00.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016219699999999999 2023-07-02T14:03:00.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:00.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311448+0000", 2023-07-02T14:03:00.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066020599999999999 2023-07-02T14:03:00.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:00.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318050+0000", 2023-07-02T14:03:00.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000327986 2023-07-02T14:03:00.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:00.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318378+0000", 2023-07-02T14:03:00.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8693999999999998e-05 2023-07-02T14:03:00.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:00.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318427+0000", 2023-07-02T14:03:00.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6656999999999998e-05 2023-07-02T14:03:00.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:00.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318464+0000", 2023-07-02T14:03:00.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003 2023-07-02T14:03:00.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.744 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.745 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.745 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:195 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:00.745 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:03:00.745 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.341946326999999, 2023-07-02T14:03:00.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001, 2023-07-02T14:03:00.745 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:00.746 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:00.746 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:00.746 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:00.746 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:00.746 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 195 2023-07-02T14:03:00.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.746 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:00.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:00.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:03:00.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:00.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:00.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:03:00.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:00.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:00.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285931+0000", 2023-07-02T14:03:00.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4530999999999995e-05 2023-07-02T14:03:00.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:00.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286006+0000", 2023-07-02T14:03:00.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4113e-05 2023-07-02T14:03:00.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:00.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286020+0000", 2023-07-02T14:03:00.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7459999999999997e-06 2023-07-02T14:03:00.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:00.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286026+0000", 2023-07-02T14:03:00.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025997769 2023-07-02T14:03:00.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:00.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312023+0000", 2023-07-02T14:03:00.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4403000000000001e-05 2023-07-02T14:03:00.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:00.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312078+0000", 2023-07-02T14:03:00.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020581399999999999 2023-07-02T14:03:00.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:00.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312284+0000", 2023-07-02T14:03:00.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005771043 2023-07-02T14:03:00.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:00.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318055+0000", 2023-07-02T14:03:00.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00046015200000000002 2023-07-02T14:03:00.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:00.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318515+0000", 2023-07-02T14:03:00.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9444000000000003e-05 2023-07-02T14:03:00.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:00.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318564+0000", 2023-07-02T14:03:00.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2141000000000002e-05 2023-07-02T14:03:00.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:00.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318606+0000", 2023-07-02T14:03:00.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001 2023-07-02T14:03:00.755 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.755 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.755 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.755 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:198 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:00.755 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:03:00.756 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.341846367999999, 2023-07-02T14:03:00.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999, 2023-07-02T14:03:00.756 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:00.756 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:00.756 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:00.756 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:00.756 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:00.756 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 198 2023-07-02T14:03:00.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.757 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:00.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:00.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:03:00.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:00.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:00.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:03:00.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:00.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:00.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286031+0000", 2023-07-02T14:03:00.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00049286700000000002 2023-07-02T14:03:00.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:00.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286524+0000", 2023-07-02T14:03:00.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4938e-05 2023-07-02T14:03:00.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:00.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286539+0000", 2023-07-02T14:03:00.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0680000000000002e-06 2023-07-02T14:03:00.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:00.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286545+0000", 2023-07-02T14:03:00.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026497718 2023-07-02T14:03:00.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:00.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313043+0000", 2023-07-02T14:03:00.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0884999999999998e-05 2023-07-02T14:03:00.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:00.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313094+0000", 2023-07-02T14:03:00.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014109999999999999 2023-07-02T14:03:00.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:00.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313235+0000", 2023-07-02T14:03:00.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048258980000000003 2023-07-02T14:03:00.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:00.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318061+0000", 2023-07-02T14:03:00.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014346350000000001 2023-07-02T14:03:00.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:00.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319495+0000", 2023-07-02T14:03:00.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1986e-05 2023-07-02T14:03:00.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:00.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319547+0000", 2023-07-02T14:03:00.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0457e-05 2023-07-02T14:03:00.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:00.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319578+0000", 2023-07-02T14:03:00.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999 2023-07-02T14:03:00.765 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.765 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.765 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.766 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:201 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:00.766 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:00.766 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.341325943000001, 2023-07-02T14:03:00.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003, 2023-07-02T14:03:00.766 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:00.766 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:00.766 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:00.766 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:00.767 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:00.767 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 201 2023-07-02T14:03:00.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.767 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:00.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:00.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:00.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:00.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:00.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:00.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:00.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:00.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:00.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019213999999999999 2023-07-02T14:03:00.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:00.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286744+0000", 2023-07-02T14:03:00.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5279999999999992e-06 2023-07-02T14:03:00.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:00.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286753+0000", 2023-07-02T14:03:00.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4200000000000004e-06 2023-07-02T14:03:00.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:00.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286760+0000", 2023-07-02T14:03:00.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026818386999999999 2023-07-02T14:03:00.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:00.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313578+0000", 2023-07-02T14:03:00.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015965 2023-07-02T14:03:00.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:00.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313738+0000", 2023-07-02T14:03:00.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015767299999999999 2023-07-02T14:03:00.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:00.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313895+0000", 2023-07-02T14:03:00.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041714930000000001 2023-07-02T14:03:00.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:00.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318067+0000", 2023-07-02T14:03:00.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001561698 2023-07-02T14:03:00.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:00.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319629+0000", 2023-07-02T14:03:00.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2346000000000003e-05 2023-07-02T14:03:00.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:00.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319681+0000", 2023-07-02T14:03:00.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4691000000000003e-05 2023-07-02T14:03:00.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:00.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319726+0000", 2023-07-02T14:03:00.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003 2023-07-02T14:03:00.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.775 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.776 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:204 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:00.776 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:00.776 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.341111105, 2023-07-02T14:03:00.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998, 2023-07-02T14:03:00.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:00.776 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:00.777 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:00.777 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:00.777 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:00.777 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 204 2023-07-02T14:03:00.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.777 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:00.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:00.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:00.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:00.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:00.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:00.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:00.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:00.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:00.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2497999999999998e-05 2023-07-02T14:03:00.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:00.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286819+0000", 2023-07-02T14:03:00.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0099999999999995e-06 2023-07-02T14:03:00.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:00.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286827+0000", 2023-07-02T14:03:00.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000003e-06 2023-07-02T14:03:00.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:00.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286833+0000", 2023-07-02T14:03:00.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027384716999999999 2023-07-02T14:03:00.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:00.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314218+0000", 2023-07-02T14:03:00.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6204000000000004e-05 2023-07-02T14:03:00.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:00.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314294+0000", 2023-07-02T14:03:00.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015757499999999999 2023-07-02T14:03:00.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:00.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314452+0000", 2023-07-02T14:03:00.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036215560000000002 2023-07-02T14:03:00.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:00.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318073+0000", 2023-07-02T14:03:00.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017023979999999999 2023-07-02T14:03:00.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:00.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319776+0000", 2023-07-02T14:03:00.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9561000000000002e-05 2023-07-02T14:03:00.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:00.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319825+0000", 2023-07-02T14:03:00.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0329e-05 2023-07-02T14:03:00.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:00.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319866+0000", 2023-07-02T14:03:00.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998 2023-07-02T14:03:00.785 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.786 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.786 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:207 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:00.786 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:03:00.786 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.341036460000002, 2023-07-02T14:03:00.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003, 2023-07-02T14:03:00.787 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:00.787 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:00.787 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:00.787 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:00.787 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:00.787 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 207 2023-07-02T14:03:00.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.788 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:00.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:00.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:03:00.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:00.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:00.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:03:00.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:00.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:00.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286841+0000", 2023-07-02T14:03:00.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00068375100000000002 2023-07-02T14:03:00.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:00.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287525+0000", 2023-07-02T14:03:00.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1601e-05 2023-07-02T14:03:00.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:00.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287537+0000", 2023-07-02T14:03:00.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6409999999999996e-06 2023-07-02T14:03:00.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:00.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287543+0000", 2023-07-02T14:03:00.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027245844000000002 2023-07-02T14:03:00.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:00.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314789+0000", 2023-07-02T14:03:00.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8910000000000003e-05 2023-07-02T14:03:00.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:00.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314858+0000", 2023-07-02T14:03:00.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030992599999999997 2023-07-02T14:03:00.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:00.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315168+0000", 2023-07-02T14:03:00.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029122869999999999 2023-07-02T14:03:00.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:00.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318080+0000", 2023-07-02T14:03:00.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019963020000000001 2023-07-02T14:03:00.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:00.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320077+0000", 2023-07-02T14:03:00.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5116000000000005e-05 2023-07-02T14:03:00.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:00.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320142+0000", 2023-07-02T14:03:00.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3793000000000004e-05 2023-07-02T14:03:00.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:00.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320205+0000", 2023-07-02T14:03:00.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003 2023-07-02T14:03:00.796 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.796 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.796 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.797 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:236 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2424832~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:00.797 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:03:00.797 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.340324277000001, 2023-07-02T14:03:00.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003, 2023-07-02T14:03:00.797 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:00.797 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:00.797 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:00.797 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:00.798 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:00.798 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2023-07-02T14:03:00.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.798 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:00.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:00.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:03:00.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:00.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:00.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:03:00.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:00.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:00.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287553+0000", 2023-07-02T14:03:00.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0667e-05 2023-07-02T14:03:00.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:00.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287614+0000", 2023-07-02T14:03:00.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0454e-05 2023-07-02T14:03:00.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:00.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287624+0000", 2023-07-02T14:03:00.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8564999999999999e-05 2023-07-02T14:03:00.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:00.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287643+0000", 2023-07-02T14:03:00.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027977596 2023-07-02T14:03:00.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:00.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315621+0000", 2023-07-02T14:03:00.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000201728 2023-07-02T14:03:00.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:00.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315822+0000", 2023-07-02T14:03:00.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018436599999999999 2023-07-02T14:03:00.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:00.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316007+0000", 2023-07-02T14:03:00.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042423440000000003 2023-07-02T14:03:00.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:00.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320249+0000", 2023-07-02T14:03:00.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001452633 2023-07-02T14:03:00.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:00.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321702+0000", 2023-07-02T14:03:00.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3616000000000003e-05 2023-07-02T14:03:00.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:00.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321755+0000", 2023-07-02T14:03:00.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840999999999999e-05 2023-07-02T14:03:00.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:00.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321793+0000", 2023-07-02T14:03:00.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003 2023-07-02T14:03:00.806 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.806 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.806 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.807 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:251 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:00.807 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:03:00.807 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.340228740000001, 2023-07-02T14:03:00.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001, 2023-07-02T14:03:00.807 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:00.807 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:00.808 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:00.808 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:00.808 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:00.808 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 251 2023-07-02T14:03:00.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.808 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:00.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:00.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:03:00.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:00.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:00.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:03:00.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:00.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:00.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287649+0000", 2023-07-02T14:03:00.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3232000000000001e-05 2023-07-02T14:03:00.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:00.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287702+0000", 2023-07-02T14:03:00.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0010000000000007e-06 2023-07-02T14:03:00.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:00.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287711+0000", 2023-07-02T14:03:00.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3569999999999997e-06 2023-07-02T14:03:00.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:00.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287717+0000", 2023-07-02T14:03:00.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028686454 2023-07-02T14:03:00.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:00.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316403+0000", 2023-07-02T14:03:00.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7741000000000002e-05 2023-07-02T14:03:00.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:00.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316461+0000", 2023-07-02T14:03:00.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4313000000000004e-05 2023-07-02T14:03:00.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:00.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316545+0000", 2023-07-02T14:03:00.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003734032 2023-07-02T14:03:00.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:00.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320279+0000", 2023-07-02T14:03:00.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015237219999999999 2023-07-02T14:03:00.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:00.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321803+0000", 2023-07-02T14:03:00.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1927e-05 2023-07-02T14:03:00.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:00.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321845+0000", 2023-07-02T14:03:00.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5398999999999998e-05 2023-07-02T14:03:00.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:00.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321880+0000", 2023-07-02T14:03:00.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001 2023-07-02T14:03:00.817 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.817 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.817 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.817 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:255 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2555904~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:00.817 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:03:00.817 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.339814012000001, 2023-07-02T14:03:00.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002, 2023-07-02T14:03:00.818 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:00.818 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:00.818 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:00.818 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:00.818 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:00.818 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 255 2023-07-02T14:03:00.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.819 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:00.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:00.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:03:00.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:00.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:00.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:03:00.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:00.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:00.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288063+0000", 2023-07-02T14:03:00.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4459999999999997e-05 2023-07-02T14:03:00.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:00.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288118+0000", 2023-07-02T14:03:00.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0998999999999999e-05 2023-07-02T14:03:00.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:00.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288129+0000", 2023-07-02T14:03:00.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3219999999999998e-06 2023-07-02T14:03:00.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:00.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288135+0000", 2023-07-02T14:03:00.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029137386000000001 2023-07-02T14:03:00.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:00.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317273+0000", 2023-07-02T14:03:00.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7819999999999997e-05 2023-07-02T14:03:00.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:00.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317350+0000", 2023-07-02T14:03:00.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000174864 2023-07-02T14:03:00.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:00.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317525+0000", 2023-07-02T14:03:00.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027943519999999999 2023-07-02T14:03:00.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:00.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320320+0000", 2023-07-02T14:03:00.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015681460000000001 2023-07-02T14:03:00.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:00.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321888+0000", 2023-07-02T14:03:00.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9508999999999997e-05 2023-07-02T14:03:00.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:00.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321927+0000", 2023-07-02T14:03:00.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5428e-05 2023-07-02T14:03:00.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:00.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321963+0000", 2023-07-02T14:03:00.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002 2023-07-02T14:03:00.827 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.827 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.827 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.828 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:446 2.1 2:910b2a90:::rbd_data.125de1449605.000000000000010a:head [write 65536~3211264] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:00.828 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:00.828 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.262534259999999, 2023-07-02T14:03:00.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003, 2023-07-02T14:03:00.828 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:00.828 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:00.828 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:00.829 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:00.829 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:00.829 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 446 2023-07-02T14:03:00.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.829 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:00.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:00.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:00.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:00.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:00.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:00.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:00.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:00.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:00.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015907372999999999 2023-07-02T14:03:00.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:00.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381251+0000", 2023-07-02T14:03:00.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5527000000000001e-05 2023-07-02T14:03:00.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:00.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381277+0000", 2023-07-02T14:03:00.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9958000000000001e-05 2023-07-02T14:03:00.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:00.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381297+0000", 2023-07-02T14:03:00.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4941000000000001e-05 2023-07-02T14:03:00.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:00.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381352+0000", 2023-07-02T14:03:00.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011565699999999999 2023-07-02T14:03:00.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:00.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381467+0000", 2023-07-02T14:03:00.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018039799999999999 2023-07-02T14:03:00.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:00.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381648+0000", 2023-07-02T14:03:00.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088183559999999994 2023-07-02T14:03:00.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:00.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.390466+0000", 2023-07-02T14:03:00.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007278278 2023-07-02T14:03:00.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:00.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397744+0000", 2023-07-02T14:03:00.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9289000000000001e-05 2023-07-02T14:03:00.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:00.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397804+0000", 2023-07-02T14:03:00.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0479000000000002e-05 2023-07-02T14:03:00.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:00.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:00.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:00.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397854+0000", 2023-07-02T14:03:00.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003 2023-07-02T14:03:00.837 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.838 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.838 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.838 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:00.838 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:00.838 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:00.840 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:03:00.840 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:00.839+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:00.940 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:00.940+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:00.975 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:00.975 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:00.975 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:03:00.975 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:00.985 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:03:01.041 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:01.041+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:01.111 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:01.112 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:01.112 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:01.112 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:01.112 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:01.116 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:03:01.122 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:03:01.166 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:01.165+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:01.243 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:01.242+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:01.249 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:01.249 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:01.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:01.250 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:01.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.250 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:322 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:01.250 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:01.250 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 21.948090111999999, 2023-07-02T14:03:01.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.03676041, 2023-07-02T14:03:01.250 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.251 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:01.251 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:01.251 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:01.251 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:01.251 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 322 2023-07-02T14:03:01.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.251 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:01.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.252 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:01.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:01.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.253 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298819+0000", 2023-07-02T14:03:01.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8068e-05 2023-07-02T14:03:01.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298877+0000", 2023-07-02T14:03:01.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3339999999999998e-06 2023-07-02T14:03:01.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298883+0000", 2023-07-02T14:03:01.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1030000000000004e-06 2023-07-02T14:03:01.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298887+0000", 2023-07-02T14:03:01.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023640230000000002 2023-07-02T14:03:01.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322528+0000", 2023-07-02T14:03:01.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8424999999999994e-05 2023-07-02T14:03:01.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322616+0000", 2023-07-02T14:03:01.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116815 2023-07-02T14:03:01.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:01.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322733+0000", 2023-07-02T14:03:01.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048252629999999998 2023-07-02T14:03:01.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:01.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327558+0000", 2023-07-02T14:03:01.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078965240000000003 2023-07-02T14:03:01.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:01.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335455+0000", 2023-07-02T14:03:01.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0511999999999999e-05 2023-07-02T14:03:01.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:01.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335515+0000", 2023-07-02T14:03:01.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4789000000000006e-05 2023-07-02T14:03:01.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335580+0000", 2023-07-02T14:03:01.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.03676041 2023-07-02T14:03:01.260 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.260 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.260 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.261 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.261 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:323 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:01.261 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:01.261 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 21.948018732000001, 2023-07-02T14:03:01.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036875044000000003, 2023-07-02T14:03:01.261 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.261 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:01.262 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:01.262 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:01.262 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:01.262 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 323 2023-07-02T14:03:01.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.262 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.263 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:01.263 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.263 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:01.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:01.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.264 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298890+0000", 2023-07-02T14:03:01.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7277000000000002e-05 2023-07-02T14:03:01.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.265 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298938+0000", 2023-07-02T14:03:01.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0495e-05 2023-07-02T14:03:01.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.265 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298948+0000", 2023-07-02T14:03:01.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6829999999999999e-06 2023-07-02T14:03:01.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298953+0000", 2023-07-02T14:03:01.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024271435000000001 2023-07-02T14:03:01.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323224+0000", 2023-07-02T14:03:01.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9266999999999998e-05 2023-07-02T14:03:01.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323304+0000", 2023-07-02T14:03:01.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000157021 2023-07-02T14:03:01.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:01.268 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323461+0000", 2023-07-02T14:03:01.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041036689999999999 2023-07-02T14:03:01.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:01.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327564+0000", 2023-07-02T14:03:01.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080753490000000008 2023-07-02T14:03:01.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:01.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335640+0000", 2023-07-02T14:03:01.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7398999999999994e-05 2023-07-02T14:03:01.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:01.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335727+0000", 2023-07-02T14:03:01.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8967000000000001e-05 2023-07-02T14:03:01.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335766+0000", 2023-07-02T14:03:01.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036875044000000003 2023-07-02T14:03:01.271 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.271 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.271 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.271 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:324 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:01.272 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:01.272 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 21.947951354000001, 2023-07-02T14:03:01.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036935673000000002, 2023-07-02T14:03:01.272 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.272 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:01.272 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:01.272 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:01.272 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:01.273 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 324 2023-07-02T14:03:01.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.273 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.273 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:01.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:01.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:01.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.275 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:01.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5519999999999997e-05 2023-07-02T14:03:01.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.275 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.275 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299013+0000", 2023-07-02T14:03:01.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0701e-05 2023-07-02T14:03:01.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299024+0000", 2023-07-02T14:03:01.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2379999999999999e-06 2023-07-02T14:03:01.276 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.277 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299028+0000", 2023-07-02T14:03:01.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024972768999999999 2023-07-02T14:03:01.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.277 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.277 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324001+0000", 2023-07-02T14:03:01.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1178000000000002e-05 2023-07-02T14:03:01.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324072+0000", 2023-07-02T14:03:01.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000166119 2023-07-02T14:03:01.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:01.279 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324238+0000", 2023-07-02T14:03:01.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033314210000000002 2023-07-02T14:03:01.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:01.279 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327570+0000", 2023-07-02T14:03:01.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0082393369999999994 2023-07-02T14:03:01.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:01.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335809+0000", 2023-07-02T14:03:01.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7071999999999999e-05 2023-07-02T14:03:01.280 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:01.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335856+0000", 2023-07-02T14:03:01.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7926999999999998e-05 2023-07-02T14:03:01.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335894+0000", 2023-07-02T14:03:01.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036935673000000002 2023-07-02T14:03:01.281 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.282 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.282 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.282 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.282 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:325 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:01.282 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:01.282 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 21.947876027, 2023-07-02T14:03:01.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036990838999999998, 2023-07-02T14:03:01.283 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.283 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:01.283 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:01.283 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:01.283 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:01.283 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 325 2023-07-02T14:03:01.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.284 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:01.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:01.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:01.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299033+0000", 2023-07-02T14:03:01.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107619 2023-07-02T14:03:01.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299141+0000", 2023-07-02T14:03:01.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3497e-05 2023-07-02T14:03:01.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299154+0000", 2023-07-02T14:03:01.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4669e-05 2023-07-02T14:03:01.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299169+0000", 2023-07-02T14:03:01.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025630159999999999 2023-07-02T14:03:01.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324799+0000", 2023-07-02T14:03:01.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6953000000000007e-05 2023-07-02T14:03:01.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324876+0000", 2023-07-02T14:03:01.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016346199999999999 2023-07-02T14:03:01.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:01.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325039+0000", 2023-07-02T14:03:01.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003566624 2023-07-02T14:03:01.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:01.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328606+0000", 2023-07-02T14:03:01.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073295540000000003 2023-07-02T14:03:01.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:01.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335936+0000", 2023-07-02T14:03:01.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0044999999999999e-05 2023-07-02T14:03:01.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:01.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335986+0000", 2023-07-02T14:03:01.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8872999999999999e-05 2023-07-02T14:03:01.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336024+0000", 2023-07-02T14:03:01.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036990838999999998 2023-07-02T14:03:01.292 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.292 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.292 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.293 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:326 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:01.293 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:01.293 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 21.947734099000002, 2023-07-02T14:03:01.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036975239999999999, 2023-07-02T14:03:01.293 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.293 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:01.293 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:01.294 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:01.294 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:01.294 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 326 2023-07-02T14:03:01.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.294 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.294 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:01.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.295 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:01.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:01.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299175+0000", 2023-07-02T14:03:01.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4782000000000003e-05 2023-07-02T14:03:01.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299250+0000", 2023-07-02T14:03:01.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.277e-05 2023-07-02T14:03:01.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299262+0000", 2023-07-02T14:03:01.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1460000000000001e-06 2023-07-02T14:03:01.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299268+0000", 2023-07-02T14:03:01.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026158098000000001 2023-07-02T14:03:01.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325427+0000", 2023-07-02T14:03:01.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4845999999999995e-05 2023-07-02T14:03:01.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325501+0000", 2023-07-02T14:03:01.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015890699999999999 2023-07-02T14:03:01.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:01.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325660+0000", 2023-07-02T14:03:01.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002952114 2023-07-02T14:03:01.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:01.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328612+0000", 2023-07-02T14:03:01.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074491949999999996 2023-07-02T14:03:01.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:01.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336062+0000", 2023-07-02T14:03:01.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7376999999999997e-05 2023-07-02T14:03:01.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:01.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336109+0000", 2023-07-02T14:03:01.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1814000000000003e-05 2023-07-02T14:03:01.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336151+0000", 2023-07-02T14:03:01.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036975239999999999 2023-07-02T14:03:01.302 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.302 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.303 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.303 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:327 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 589824~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:01.303 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:01.303 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 21.947634837999999, 2023-07-02T14:03:01.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037234253000000002, 2023-07-02T14:03:01.303 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.304 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:01.304 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:01.304 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:01.304 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:01.304 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 327 2023-07-02T14:03:01.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.304 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:01.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:01.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:01.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299274+0000", 2023-07-02T14:03:01.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2213999999999997e-05 2023-07-02T14:03:01.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.307 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299346+0000", 2023-07-02T14:03:01.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1816e-05 2023-07-02T14:03:01.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299358+0000", 2023-07-02T14:03:01.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2626000000000001e-05 2023-07-02T14:03:01.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299371+0000", 2023-07-02T14:03:01.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026916853000000001 2023-07-02T14:03:01.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326288+0000", 2023-07-02T14:03:01.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6909000000000004e-05 2023-07-02T14:03:01.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326375+0000", 2023-07-02T14:03:01.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015430399999999999 2023-07-02T14:03:01.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:01.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326529+0000", 2023-07-02T14:03:01.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096652100000000005 2023-07-02T14:03:01.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:01.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336194+0000", 2023-07-02T14:03:01.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115299 2023-07-02T14:03:01.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:01.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336309+0000", 2023-07-02T14:03:01.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015981200000000001 2023-07-02T14:03:01.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:01.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336469+0000", 2023-07-02T14:03:01.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9867999999999998e-05 2023-07-02T14:03:01.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336509+0000", 2023-07-02T14:03:01.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037234253000000002 2023-07-02T14:03:01.313 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.314 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.314 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.314 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:328 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:01.314 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:01.314 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 21.947531249000001, 2023-07-02T14:03:01.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037216610999999997, 2023-07-02T14:03:01.315 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.315 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:01.315 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:01.315 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:01.315 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:01.315 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 328 2023-07-02T14:03:01.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.316 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:01.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:01.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:01.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:01.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1412e-05 2023-07-02T14:03:01.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299449+0000", 2023-07-02T14:03:01.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0219000000000001e-05 2023-07-02T14:03:01.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299469+0000", 2023-07-02T14:03:01.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9480000000000003e-06 2023-07-02T14:03:01.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299479+0000", 2023-07-02T14:03:01.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027444344999999998 2023-07-02T14:03:01.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326924+0000", 2023-07-02T14:03:01.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5165000000000001e-05 2023-07-02T14:03:01.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327019+0000", 2023-07-02T14:03:01.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019113499999999999 2023-07-02T14:03:01.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:01.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327210+0000", 2023-07-02T14:03:01.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090197449999999992 2023-07-02T14:03:01.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.322 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:01.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336230+0000", 2023-07-02T14:03:01.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028764399999999999 2023-07-02T14:03:01.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:01.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336517+0000", 2023-07-02T14:03:01.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1981999999999999e-05 2023-07-02T14:03:01.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:01.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336559+0000", 2023-07-02T14:03:01.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:03:01.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.324 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336595+0000", 2023-07-02T14:03:01.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037216610999999997 2023-07-02T14:03:01.325 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.325 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.325 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.325 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:329 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:01.325 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:01.325 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 21.947421332000001, 2023-07-02T14:03:01.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037191142000000003, 2023-07-02T14:03:01.326 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.326 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:01.326 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:01.326 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:01.326 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:01.326 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 329 2023-07-02T14:03:01.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.327 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:01.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:01.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:01.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299487+0000", 2023-07-02T14:03:01.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7871000000000001e-05 2023-07-02T14:03:01.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299555+0000", 2023-07-02T14:03:01.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0446e-05 2023-07-02T14:03:01.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299566+0000", 2023-07-02T14:03:01.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7939999999999999e-06 2023-07-02T14:03:01.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299572+0000", 2023-07-02T14:03:01.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028034019 2023-07-02T14:03:01.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327606+0000", 2023-07-02T14:03:01.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1967999999999999e-05 2023-07-02T14:03:01.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327678+0000", 2023-07-02T14:03:01.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022216900000000001 2023-07-02T14:03:01.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:01.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327900+0000", 2023-07-02T14:03:01.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084016899999999999 2023-07-02T14:03:01.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:01.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336301+0000", 2023-07-02T14:03:01.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00030122800000000001 2023-07-02T14:03:01.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:01.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336603+0000", 2023-07-02T14:03:01.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0380000000000003e-05 2023-07-02T14:03:01.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:01.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336643+0000", 2023-07-02T14:03:01.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6455999999999998e-05 2023-07-02T14:03:01.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336679+0000", 2023-07-02T14:03:01.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037191142000000003 2023-07-02T14:03:01.336 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.336 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.336 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.337 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:330 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:01.337 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:01.337 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 21.947330562000001, 2023-07-02T14:03:01.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037849305, 2023-07-02T14:03:01.337 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.337 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:01.337 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:01.337 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:01.338 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:01.338 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 330 2023-07-02T14:03:01.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.338 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:01.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:01.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:01.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299578+0000", 2023-07-02T14:03:01.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4708000000000005e-05 2023-07-02T14:03:01.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299643+0000", 2023-07-02T14:03:01.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7309999999999999e-06 2023-07-02T14:03:01.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299652+0000", 2023-07-02T14:03:01.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7989999999999999e-06 2023-07-02T14:03:01.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299658+0000", 2023-07-02T14:03:01.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028981664000000001 2023-07-02T14:03:01.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328639+0000", 2023-07-02T14:03:01.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0939000000000006e-05 2023-07-02T14:03:01.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328710+0000", 2023-07-02T14:03:01.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000142624 2023-07-02T14:03:01.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:01.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328853+0000", 2023-07-02T14:03:01.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078374640000000006 2023-07-02T14:03:01.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:01.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336690+0000", 2023-07-02T14:03:01.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00068453000000000003 2023-07-02T14:03:01.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:01.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337375+0000", 2023-07-02T14:03:01.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0667999999999998e-05 2023-07-02T14:03:01.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:01.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337406+0000", 2023-07-02T14:03:01.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2858999999999999e-05 2023-07-02T14:03:01.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337428+0000", 2023-07-02T14:03:01.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037849305 2023-07-02T14:03:01.346 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.346 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.347 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.347 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:331 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:01.347 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:01.347 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 21.947246700000001, 2023-07-02T14:03:01.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037870332, 2023-07-02T14:03:01.347 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.348 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:01.348 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:01.348 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:01.348 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:01.348 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 331 2023-07-02T14:03:01.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.348 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:01.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:01.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:01.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299662+0000", 2023-07-02T14:03:01.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2252e-05 2023-07-02T14:03:01.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299725+0000", 2023-07-02T14:03:01.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8440000000000004e-06 2023-07-02T14:03:01.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299733+0000", 2023-07-02T14:03:01.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6759999999999998e-06 2023-07-02T14:03:01.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299738+0000", 2023-07-02T14:03:01.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029459387 2023-07-02T14:03:01.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329197+0000", 2023-07-02T14:03:01.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0087999999999997e-05 2023-07-02T14:03:01.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329268+0000", 2023-07-02T14:03:01.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014405700000000001 2023-07-02T14:03:01.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:01.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329412+0000", 2023-07-02T14:03:01.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072842100000000002 2023-07-02T14:03:01.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:01.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336696+0000", 2023-07-02T14:03:01.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00076275300000000002 2023-07-02T14:03:01.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:01.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337459+0000", 2023-07-02T14:03:01.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5988000000000002e-05 2023-07-02T14:03:01.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:01.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337495+0000", 2023-07-02T14:03:01.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8686000000000002e-05 2023-07-02T14:03:01.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337533+0000", 2023-07-02T14:03:01.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037870332 2023-07-02T14:03:01.357 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.357 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.357 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.358 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:332 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:01.358 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:01.358 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 21.947166776, 2023-07-02T14:03:01.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037892506999999999, 2023-07-02T14:03:01.358 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.358 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:01.358 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:01.358 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:01.359 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:01.359 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 332 2023-07-02T14:03:01.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.359 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:01.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:01.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:01.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299742+0000", 2023-07-02T14:03:01.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1366999999999997e-05 2023-07-02T14:03:01.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299804+0000", 2023-07-02T14:03:01.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9319999999999996e-06 2023-07-02T14:03:01.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299812+0000", 2023-07-02T14:03:01.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4359999999999999e-06 2023-07-02T14:03:01.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299818+0000", 2023-07-02T14:03:01.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029946144000000001 2023-07-02T14:03:01.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329764+0000", 2023-07-02T14:03:01.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000164828 2023-07-02T14:03:01.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329929+0000", 2023-07-02T14:03:01.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016139400000000001 2023-07-02T14:03:01.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:01.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330090+0000", 2023-07-02T14:03:01.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066134009999999997 2023-07-02T14:03:01.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:01.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336704+0000", 2023-07-02T14:03:01.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00086955400000000003 2023-07-02T14:03:01.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:01.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337573+0000", 2023-07-02T14:03:01.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8447000000000001e-05 2023-07-02T14:03:01.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:01.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337602+0000", 2023-07-02T14:03:01.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.362e-05 2023-07-02T14:03:01.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337635+0000", 2023-07-02T14:03:01.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037892506999999999 2023-07-02T14:03:01.368 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.368 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.368 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.369 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:333 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:01.369 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:01.369 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 21.947086694999999, 2023-07-02T14:03:01.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037932598999999997, 2023-07-02T14:03:01.369 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.369 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:01.369 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:01.370 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:01.370 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:01.370 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 333 2023-07-02T14:03:01.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.370 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:01.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:01.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:01.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299822+0000", 2023-07-02T14:03:01.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0831000000000001e-05 2023-07-02T14:03:01.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299883+0000", 2023-07-02T14:03:01.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.322e-06 2023-07-02T14:03:01.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299888+0000", 2023-07-02T14:03:01.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3479999999999997e-06 2023-07-02T14:03:01.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299893+0000", 2023-07-02T14:03:01.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030831905999999999 2023-07-02T14:03:01.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330725+0000", 2023-07-02T14:03:01.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3560999999999996e-05 2023-07-02T14:03:01.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330798+0000", 2023-07-02T14:03:01.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015165999999999999 2023-07-02T14:03:01.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:01.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330950+0000", 2023-07-02T14:03:01.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057590319999999999 2023-07-02T14:03:01.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:01.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336709+0000", 2023-07-02T14:03:01.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00096987899999999999 2023-07-02T14:03:01.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:01.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337679+0000", 2023-07-02T14:03:01.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9699000000000001e-05 2023-07-02T14:03:01.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:01.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337719+0000", 2023-07-02T14:03:01.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6961000000000001e-05 2023-07-02T14:03:01.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337756+0000", 2023-07-02T14:03:01.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037932598999999997 2023-07-02T14:03:01.378 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.379 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.379 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.379 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:372 2.0 2:169af44a:::rbd_data.125de1449605.0000000000000106:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:01.379 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:03:01.379 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 21.943509370000001, 2023-07-02T14:03:01.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036066963, 2023-07-02T14:03:01.380 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.380 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:01.380 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:01.380 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:01.380 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:01.380 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 372 2023-07-02T14:03:01.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.380 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:03:01.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:03:01.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:01.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:03:01.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1002000000000003e-05 2023-07-02T14:03:01.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303461+0000", 2023-07-02T14:03:01.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9306e-05 2023-07-02T14:03:01.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303490+0000", 2023-07-02T14:03:01.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5026e-05 2023-07-02T14:03:01.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303505+0000", 2023-07-02T14:03:01.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033623129000000002 2023-07-02T14:03:01.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337128+0000", 2023-07-02T14:03:01.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6968000000000002e-05 2023-07-02T14:03:01.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337175+0000", 2023-07-02T14:03:01.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000101127 2023-07-02T14:03:01.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:01.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337276+0000", 2023-07-02T14:03:01.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017866920000000001 2023-07-02T14:03:01.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:01.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339063+0000", 2023-07-02T14:03:01.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00031459800000000003 2023-07-02T14:03:01.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:01.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339377+0000", 2023-07-02T14:03:01.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7145999999999999e-05 2023-07-02T14:03:01.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:01.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339425+0000", 2023-07-02T14:03:01.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2632000000000001e-05 2023-07-02T14:03:01.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339467+0000", 2023-07-02T14:03:01.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036066963 2023-07-02T14:03:01.389 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.389 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.389 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.389 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:416 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:01.390 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:01.390 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 21.928793316, 2023-07-02T14:03:01.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039242025, 2023-07-02T14:03:01.390 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.390 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:01.390 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:01.390 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:01.390 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:01.391 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:03:01.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.391 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:01.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:01.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:01.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:01.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.692e-06 2023-07-02T14:03:01.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318119+0000", 2023-07-02T14:03:01.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9790000000000005e-06 2023-07-02T14:03:01.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318129+0000", 2023-07-02T14:03:01.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3640000000000001e-06 2023-07-02T14:03:01.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318137+0000", 2023-07-02T14:03:01.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022607808 2023-07-02T14:03:01.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340745+0000", 2023-07-02T14:03:01.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0130999999999996e-05 2023-07-02T14:03:01.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340825+0000", 2023-07-02T14:03:01.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039581080000000001 2023-07-02T14:03:01.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344783+0000", 2023-07-02T14:03:01.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000006e-05 2023-07-02T14:03:01.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344854+0000", 2023-07-02T14:03:01.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052105550000000004 2023-07-02T14:03:01.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350065+0000", 2023-07-02T14:03:01.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0321999999999997e-05 2023-07-02T14:03:01.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350115+0000", 2023-07-02T14:03:01.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012378409999999999 2023-07-02T14:03:01.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351353+0000", 2023-07-02T14:03:01.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5942000000000001e-05 2023-07-02T14:03:01.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351409+0000", 2023-07-02T14:03:01.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035677700000000002 2023-07-02T14:03:01.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:01.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351766+0000", 2023-07-02T14:03:01.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000926719 2023-07-02T14:03:01.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:01.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352693+0000", 2023-07-02T14:03:01.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045652080000000003 2023-07-02T14:03:01.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:01.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357258+0000", 2023-07-02T14:03:01.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4717e-05 2023-07-02T14:03:01.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:01.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357293+0000", 2023-07-02T14:03:01.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5807000000000002e-05 2023-07-02T14:03:01.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357358+0000", 2023-07-02T14:03:01.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039242025 2023-07-02T14:03:01.403 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.403 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.403 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.404 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:417 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:01.404 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:01.404 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 21.928767878999999, 2023-07-02T14:03:01.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042407105, 2023-07-02T14:03:01.404 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.405 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:01.405 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:01.405 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:01.405 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:01.405 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 417 2023-07-02T14:03:01.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.405 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:01.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:01.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:01.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318141+0000", 2023-07-02T14:03:01.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7120000000000002e-06 2023-07-02T14:03:01.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318145+0000", 2023-07-02T14:03:01.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8869999999999997e-06 2023-07-02T14:03:01.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318151+0000", 2023-07-02T14:03:01.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9820000000000002e-06 2023-07-02T14:03:01.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318155+0000", 2023-07-02T14:03:01.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699133 2023-07-02T14:03:01.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340854+0000", 2023-07-02T14:03:01.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6849999999999998e-05 2023-07-02T14:03:01.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340931+0000", 2023-07-02T14:03:01.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039672140000000002 2023-07-02T14:03:01.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344898+0000", 2023-07-02T14:03:01.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5341999999999998e-05 2023-07-02T14:03:01.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344973+0000", 2023-07-02T14:03:01.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051819400000000003 2023-07-02T14:03:01.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350155+0000", 2023-07-02T14:03:01.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3261999999999999e-05 2023-07-02T14:03:01.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350209+0000", 2023-07-02T14:03:01.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00194097 2023-07-02T14:03:01.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352150+0000", 2023-07-02T14:03:01.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6851e-05 2023-07-02T14:03:01.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352226+0000", 2023-07-02T14:03:01.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051831109999999998 2023-07-02T14:03:01.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357409+0000", 2023-07-02T14:03:01.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5325e-05 2023-07-02T14:03:01.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357495+0000", 2023-07-02T14:03:01.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042107899999999998 2023-07-02T14:03:01.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:01.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357916+0000", 2023-07-02T14:03:01.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0011672449999999999 2023-07-02T14:03:01.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:01.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.359083+0000", 2023-07-02T14:03:01.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013534930000000001 2023-07-02T14:03:01.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:01.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360437+0000", 2023-07-02T14:03:01.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2159000000000002e-05 2023-07-02T14:03:01.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:01.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360469+0000", 2023-07-02T14:03:01.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0092000000000005e-05 2023-07-02T14:03:01.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360549+0000", 2023-07-02T14:03:01.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042407105 2023-07-02T14:03:01.420 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.420 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.420 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.420 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:418 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:01.420 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:01.421 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 21.928748637999998, 2023-07-02T14:03:01.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.045613663999999998, 2023-07-02T14:03:01.421 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.421 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:01.421 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:01.421 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:01.421 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:01.421 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 418 2023-07-02T14:03:01.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.422 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:01.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:01.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:01.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318160+0000", 2023-07-02T14:03:01.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6490000000000001e-06 2023-07-02T14:03:01.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318164+0000", 2023-07-02T14:03:01.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:03:01.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318170+0000", 2023-07-02T14:03:01.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5279999999999999e-06 2023-07-02T14:03:01.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318173+0000", 2023-07-02T14:03:01.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022788856 2023-07-02T14:03:01.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340962+0000", 2023-07-02T14:03:01.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7730000000000003e-05 2023-07-02T14:03:01.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341040+0000", 2023-07-02T14:03:01.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039795899999999999 2023-07-02T14:03:01.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345020+0000", 2023-07-02T14:03:01.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4962000000000006e-05 2023-07-02T14:03:01.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345085+0000", 2023-07-02T14:03:01.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051955430000000004 2023-07-02T14:03:01.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350280+0000", 2023-07-02T14:03:01.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8913999999999995e-05 2023-07-02T14:03:01.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350379+0000", 2023-07-02T14:03:01.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001887344 2023-07-02T14:03:01.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352266+0000", 2023-07-02T14:03:01.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7865000000000003e-05 2023-07-02T14:03:01.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352314+0000", 2023-07-02T14:03:01.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060621190000000004 2023-07-02T14:03:01.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358376+0000", 2023-07-02T14:03:01.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2448999999999997e-05 2023-07-02T14:03:01.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358449+0000", 2023-07-02T14:03:01.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0021539929999999999 2023-07-02T14:03:01.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360603+0000", 2023-07-02T14:03:01.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7859000000000002e-05 2023-07-02T14:03:01.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360671+0000", 2023-07-02T14:03:01.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026930699999999999 2023-07-02T14:03:01.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:01.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360940+0000", 2023-07-02T14:03:01.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00081179699999999995 2023-07-02T14:03:01.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:01.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361752+0000", 2023-07-02T14:03:01.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0019350490000000001 2023-07-02T14:03:01.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:01.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363687+0000", 2023-07-02T14:03:01.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4462000000000003e-05 2023-07-02T14:03:01.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:01.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363721+0000", 2023-07-02T14:03:01.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3408000000000001e-05 2023-07-02T14:03:01.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363775+0000", 2023-07-02T14:03:01.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.045613663999999998 2023-07-02T14:03:01.437 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.437 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.437 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.437 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:419 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:01.438 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:01.438 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 21.928732311000001, 2023-07-02T14:03:01.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.047296703000000002, 2023-07-02T14:03:01.438 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.438 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:01.438 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:01.439 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:01.439 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:01.439 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 419 2023-07-02T14:03:01.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.439 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:01.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:01.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:01.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:01.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.433e-06 2023-07-02T14:03:01.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318180+0000", 2023-07-02T14:03:01.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9950000000000002e-06 2023-07-02T14:03:01.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318186+0000", 2023-07-02T14:03:01.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7290000000000002e-06 2023-07-02T14:03:01.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318190+0000", 2023-07-02T14:03:01.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022877989000000001 2023-07-02T14:03:01.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341068+0000", 2023-07-02T14:03:01.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5803999999999994e-05 2023-07-02T14:03:01.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341154+0000", 2023-07-02T14:03:01.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039633000000000003 2023-07-02T14:03:01.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345117+0000", 2023-07-02T14:03:01.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7188999999999996e-05 2023-07-02T14:03:01.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345194+0000", 2023-07-02T14:03:01.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052306059999999996 2023-07-02T14:03:01.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350425+0000", 2023-07-02T14:03:01.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8093999999999997e-05 2023-07-02T14:03:01.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350473+0000", 2023-07-02T14:03:01.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001865158 2023-07-02T14:03:01.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352338+0000", 2023-07-02T14:03:01.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5479000000000003e-05 2023-07-02T14:03:01.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352384+0000", 2023-07-02T14:03:01.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061153290000000001 2023-07-02T14:03:01.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358499+0000", 2023-07-02T14:03:01.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5156999999999999e-05 2023-07-02T14:03:01.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358574+0000", 2023-07-02T14:03:01.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027847459999999998 2023-07-02T14:03:01.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361359+0000", 2023-07-02T14:03:01.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2633999999999999e-05 2023-07-02T14:03:01.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361402+0000", 2023-07-02T14:03:01.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024171510000000002 2023-07-02T14:03:01.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363819+0000", 2023-07-02T14:03:01.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6988999999999999e-05 2023-07-02T14:03:01.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363876+0000", 2023-07-02T14:03:01.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035321300000000001 2023-07-02T14:03:01.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:01.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.364229+0000", 2023-07-02T14:03:01.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00094674799999999997 2023-07-02T14:03:01.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:01.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365176+0000", 2023-07-02T14:03:01.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019724200000000001 2023-07-02T14:03:01.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:01.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365373+0000", 2023-07-02T14:03:01.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2116e-05 2023-07-02T14:03:01.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:01.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365425+0000", 2023-07-02T14:03:01.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8996999999999999e-05 2023-07-02T14:03:01.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365474+0000", 2023-07-02T14:03:01.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.047296703000000002 2023-07-02T14:03:01.455 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.455 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.456 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.456 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:472 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:01.456 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:01.456 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 21.869276078999999, 2023-07-02T14:03:01.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036204721000000002, 2023-07-02T14:03:01.457 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.457 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:01.457 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:01.457 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:01.457 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:01.457 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 472 2023-07-02T14:03:01.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.457 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:01.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:01.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:03:01.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377633+0000", 2023-07-02T14:03:01.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.815e-06 2023-07-02T14:03:01.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377635+0000", 2023-07-02T14:03:01.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9459999999999998e-06 2023-07-02T14:03:01.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377638+0000", 2023-07-02T14:03:01.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.673e-06 2023-07-02T14:03:01.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377643+0000", 2023-07-02T14:03:01.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043004239999999997 2023-07-02T14:03:01.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.381943+0000", 2023-07-02T14:03:01.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4321999999999999e-05 2023-07-02T14:03:01.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.381998+0000", 2023-07-02T14:03:01.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003008377 2023-07-02T14:03:01.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385006+0000", 2023-07-02T14:03:01.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8751999999999998e-05 2023-07-02T14:03:01.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385045+0000", 2023-07-02T14:03:01.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086260880000000005 2023-07-02T14:03:01.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393671+0000", 2023-07-02T14:03:01.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6365999999999997e-05 2023-07-02T14:03:01.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393707+0000", 2023-07-02T14:03:01.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028560289999999999 2023-07-02T14:03:01.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396563+0000", 2023-07-02T14:03:01.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6195e-05 2023-07-02T14:03:01.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396619+0000", 2023-07-02T14:03:01.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023118269999999998 2023-07-02T14:03:01.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.398931+0000", 2023-07-02T14:03:01.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3885999999999999e-05 2023-07-02T14:03:01.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.398985+0000", 2023-07-02T14:03:01.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043756890000000003 2023-07-02T14:03:01.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403361+0000", 2023-07-02T14:03:01.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0043000000000001e-05 2023-07-02T14:03:01.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403421+0000", 2023-07-02T14:03:01.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001590972 2023-07-02T14:03:01.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405012+0000", 2023-07-02T14:03:01.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0067000000000005e-05 2023-07-02T14:03:01.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405082+0000", 2023-07-02T14:03:01.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050958610000000001 2023-07-02T14:03:01.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410178+0000", 2023-07-02T14:03:01.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7070000000000004e-05 2023-07-02T14:03:01.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410275+0000", 2023-07-02T14:03:01.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060121100000000004 2023-07-02T14:03:01.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410876+0000", 2023-07-02T14:03:01.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3177999999999999e-05 2023-07-02T14:03:01.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410919+0000", 2023-07-02T14:03:01.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000271394 2023-07-02T14:03:01.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:01.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411191+0000", 2023-07-02T14:03:01.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001009598 2023-07-02T14:03:01.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:01.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.412200+0000", 2023-07-02T14:03:01.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014789790000000001 2023-07-02T14:03:01.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:01.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413679+0000", 2023-07-02T14:03:01.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2624000000000002e-05 2023-07-02T14:03:01.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:01.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413742+0000", 2023-07-02T14:03:01.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6564999999999994e-05 2023-07-02T14:03:01.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413838+0000", 2023-07-02T14:03:01.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036204721000000002 2023-07-02T14:03:01.476 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.476 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.476 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.477 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:473 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:01.477 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:01.477 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 21.869264668, 2023-07-02T14:03:01.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037915904, 2023-07-02T14:03:01.477 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.477 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:01.477 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:01.478 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:01.478 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:01.478 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 473 2023-07-02T14:03:01.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.478 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:01.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:01.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:03:01.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:01.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.119e-06 2023-07-02T14:03:01.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377647+0000", 2023-07-02T14:03:01.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.875e-06 2023-07-02T14:03:01.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377650+0000", 2023-07-02T14:03:01.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1949999999999999e-06 2023-07-02T14:03:01.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377653+0000", 2023-07-02T14:03:01.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004375444 2023-07-02T14:03:01.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382028+0000", 2023-07-02T14:03:01.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1063999999999997e-05 2023-07-02T14:03:01.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382089+0000", 2023-07-02T14:03:01.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030043869999999999 2023-07-02T14:03:01.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385094+0000", 2023-07-02T14:03:01.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8167999999999997e-05 2023-07-02T14:03:01.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385142+0000", 2023-07-02T14:03:01.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008603421 2023-07-02T14:03:01.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393745+0000", 2023-07-02T14:03:01.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3763000000000005e-05 2023-07-02T14:03:01.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393819+0000", 2023-07-02T14:03:01.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028274720000000001 2023-07-02T14:03:01.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396647+0000", 2023-07-02T14:03:01.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7929999999999997e-05 2023-07-02T14:03:01.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396695+0000", 2023-07-02T14:03:01.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023426459999999999 2023-07-02T14:03:01.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399037+0000", 2023-07-02T14:03:01.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5846000000000004e-05 2023-07-02T14:03:01.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399123+0000", 2023-07-02T14:03:01.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043448360000000004 2023-07-02T14:03:01.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403468+0000", 2023-07-02T14:03:01.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6699000000000001e-05 2023-07-02T14:03:01.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403515+0000", 2023-07-02T14:03:01.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001622943 2023-07-02T14:03:01.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405138+0000", 2023-07-02T14:03:01.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6382000000000002e-05 2023-07-02T14:03:01.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405204+0000", 2023-07-02T14:03:01.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051089910000000002 2023-07-02T14:03:01.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410313+0000", 2023-07-02T14:03:01.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8106000000000003e-05 2023-07-02T14:03:01.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410361+0000", 2023-07-02T14:03:01.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012200399999999999 2023-07-02T14:03:01.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411581+0000", 2023-07-02T14:03:01.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5067000000000005e-05 2023-07-02T14:03:01.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411656+0000", 2023-07-02T14:03:01.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022353249999999998 2023-07-02T14:03:01.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413891+0000", 2023-07-02T14:03:01.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9356000000000003e-05 2023-07-02T14:03:01.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413981+0000", 2023-07-02T14:03:01.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00043818299999999998 2023-07-02T14:03:01.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:01.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.414419+0000", 2023-07-02T14:03:01.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00081559199999999997 2023-07-02T14:03:01.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:01.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415235+0000", 2023-07-02T14:03:01.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026434499999999998 2023-07-02T14:03:01.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:01.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415499+0000", 2023-07-02T14:03:01.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9779999999999999e-05 2023-07-02T14:03:01.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:01.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415529+0000", 2023-07-02T14:03:01.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2159999999999997e-05 2023-07-02T14:03:01.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415561+0000", 2023-07-02T14:03:01.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037915904 2023-07-02T14:03:01.498 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.498 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.498 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.499 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:475 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:01.499 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:01.499 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 21.869254629, 2023-07-02T14:03:01.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039286962000000002, 2023-07-02T14:03:01.499 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.499 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:01.500 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:01.500 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:01.500 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:01.500 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 475 2023-07-02T14:03:01.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.500 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:01.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:01.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:01.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:01.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4779999999999997e-06 2023-07-02T14:03:01.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377664+0000", 2023-07-02T14:03:01.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.268e-06 2023-07-02T14:03:01.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377671+0000", 2023-07-02T14:03:01.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.861e-06 2023-07-02T14:03:01.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377674+0000", 2023-07-02T14:03:01.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044395789999999999 2023-07-02T14:03:01.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382114+0000", 2023-07-02T14:03:01.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8096999999999997e-05 2023-07-02T14:03:01.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382162+0000", 2023-07-02T14:03:01.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029949059999999999 2023-07-02T14:03:01.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385157+0000", 2023-07-02T14:03:01.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1158000000000001e-05 2023-07-02T14:03:01.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385198+0000", 2023-07-02T14:03:01.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086477229999999995 2023-07-02T14:03:01.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393846+0000", 2023-07-02T14:03:01.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5990999999999999e-05 2023-07-02T14:03:01.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393912+0000", 2023-07-02T14:03:01.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027979599999999999 2023-07-02T14:03:01.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396710+0000", 2023-07-02T14:03:01.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5911000000000002e-05 2023-07-02T14:03:01.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396746+0000", 2023-07-02T14:03:01.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024039759999999999 2023-07-02T14:03:01.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399150+0000", 2023-07-02T14:03:01.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7834999999999999e-05 2023-07-02T14:03:01.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399188+0000", 2023-07-02T14:03:01.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004364054 2023-07-02T14:03:01.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403552+0000", 2023-07-02T14:03:01.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6274999999999999e-05 2023-07-02T14:03:01.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403608+0000", 2023-07-02T14:03:01.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0016261540000000001 2023-07-02T14:03:01.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405234+0000", 2023-07-02T14:03:01.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6945000000000004e-05 2023-07-02T14:03:01.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405301+0000", 2023-07-02T14:03:01.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050871329999999998 2023-07-02T14:03:01.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410388+0000", 2023-07-02T14:03:01.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0210999999999995e-05 2023-07-02T14:03:01.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410468+0000", 2023-07-02T14:03:01.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001227454 2023-07-02T14:03:01.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411696+0000", 2023-07-02T14:03:01.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8188000000000005e-05 2023-07-02T14:03:01.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411764+0000", 2023-07-02T14:03:01.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031779930000000001 2023-07-02T14:03:01.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.414942+0000", 2023-07-02T14:03:01.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7236000000000004e-05 2023-07-02T14:03:01.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:01.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415019+0000", 2023-07-02T14:03:01.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000573045 2023-07-02T14:03:01.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415592+0000", 2023-07-02T14:03:01.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7580000000000002e-05 2023-07-02T14:03:01.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415640+0000", 2023-07-02T14:03:01.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024783200000000002 2023-07-02T14:03:01.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:01.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415888+0000", 2023-07-02T14:03:01.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00080440999999999998 2023-07-02T14:03:01.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:01.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416692+0000", 2023-07-02T14:03:01.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000195787 2023-07-02T14:03:01.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:01.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416888+0000", 2023-07-02T14:03:01.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0787999999999997e-05 2023-07-02T14:03:01.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:01.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416919+0000", 2023-07-02T14:03:01.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.338e-05 2023-07-02T14:03:01.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416942+0000", 2023-07-02T14:03:01.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039286962000000002 2023-07-02T14:03:01.521 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.521 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.521 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.522 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.522 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.522 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:01.523 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:03:01.524 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:01.523+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:01.624 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:01.623+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:01.688 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:01.688 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:01.688 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:03:01.688 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:01.698 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:03:01.725 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:01.724+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:01.821 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:01.821 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:01.821 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:01.821 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:01.822 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:01.826 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:01.825+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:01.831 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:03:01.927 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:01.926+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:01.955 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:01.956 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:01.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:01.956 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:01.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.956 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:1e298de8:::rbd_id.b3fbb028-e267-4d25-a750-93ba18a2316a-src:head, version: 40'1, data_included: [0~16], data_size: 16, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:1e298de8:::rbd_id.b3fbb028-e267-4d25-a750-93ba18a2316a-src:head@40'1, size: 16, copy_subset: [0~16], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:16, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-07-02T14:03:01.956 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.133561+0000", 2023-07-02T14:03:01.957 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.821065748000001, 2023-07-02T14:03:01.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000366984, 2023-07-02T14:03:01.957 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.957 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:01.957 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.133561+0000", 2023-07-02T14:03:01.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.133561+0000", 2023-07-02T14:03:01.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:01.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.133560+0000", 2023-07-02T14:03:01.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2972e-05 2023-07-02T14:03:01.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.133573+0000", 2023-07-02T14:03:01.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6759e-05 2023-07-02T14:03:01.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.133590+0000", 2023-07-02T14:03:01.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4440000000000008e-06 2023-07-02T14:03:01.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.133598+0000", 2023-07-02T14:03:01.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9666000000000003e-05 2023-07-02T14:03:01.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.133638+0000", 2023-07-02T14:03:01.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1415e-05 2023-07-02T14:03:01.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.133650+0000", 2023-07-02T14:03:01.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00027846499999999998 2023-07-02T14:03:01.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.133928+0000", 2023-07-02T14:03:01.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000366984 2023-07-02T14:03:01.964 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.964 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.964 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.964 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:169af44a:::rbd_data.125de1449605.0000000000000106:head, version: 40'57, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:169af44a:::rbd_data.125de1449605.0000000000000106:head@40'57, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-07-02T14:03:01.964 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.136138+0000", 2023-07-02T14:03:01.964 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.818489144999999, 2023-07-02T14:03:01.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033651969999999999, 2023-07-02T14:03:01.965 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.965 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:01.965 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.136138+0000", 2023-07-02T14:03:01.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.136138+0000", 2023-07-02T14:03:01.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:01.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.136137+0000", 2023-07-02T14:03:01.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001911903 2023-07-02T14:03:01.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.138049+0000", 2023-07-02T14:03:01.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6398000000000001e-05 2023-07-02T14:03:01.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.138065+0000", 2023-07-02T14:03:01.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8220000000000007e-06 2023-07-02T14:03:01.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.138073+0000", 2023-07-02T14:03:01.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8141999999999998e-05 2023-07-02T14:03:01.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.138151+0000", 2023-07-02T14:03:01.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5259999999999999e-06 2023-07-02T14:03:01.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.138161+0000", 2023-07-02T14:03:01.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001342085 2023-07-02T14:03:01.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.139503+0000", 2023-07-02T14:03:01.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033651969999999999 2023-07-02T14:03:01.971 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.971 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.971 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.971 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:169af44a:::rbd_data.125de1449605.0000000000000106:head, version: 40'57, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:169af44a:::rbd_data.125de1449605.0000000000000106:head@40'57, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:03:01.972 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.141582+0000", 2023-07-02T14:03:01.972 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.813045296, 2023-07-02T14:03:01.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029211950000000001, 2023-07-02T14:03:01.972 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.972 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:01.972 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.141582+0000", 2023-07-02T14:03:01.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.141582+0000", 2023-07-02T14:03:01.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:01.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.141581+0000", 2023-07-02T14:03:01.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001864533 2023-07-02T14:03:01.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.143445+0000", 2023-07-02T14:03:01.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5650000000000001e-05 2023-07-02T14:03:01.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.143461+0000", 2023-07-02T14:03:01.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.109e-06 2023-07-02T14:03:01.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.143468+0000", 2023-07-02T14:03:01.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0136999999999996e-05 2023-07-02T14:03:01.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.143538+0000", 2023-07-02T14:03:01.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4500000000000004e-06 2023-07-02T14:03:01.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.143547+0000", 2023-07-02T14:03:01.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00095602800000000004 2023-07-02T14:03:01.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.144503+0000", 2023-07-02T14:03:01.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029211950000000001 2023-07-02T14:03:01.979 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.979 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.979 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.980 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:169af44a:::rbd_data.125de1449605.0000000000000106:head, version: 40'57, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:169af44a:::rbd_data.125de1449605.0000000000000106:head@40'57, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:03:01.980 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.146696+0000", 2023-07-02T14:03:01.980 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.807930965000001, 2023-07-02T14:03:01.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032948410000000002, 2023-07-02T14:03:01.980 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.980 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:01.980 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.146696+0000", 2023-07-02T14:03:01.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.146696+0000", 2023-07-02T14:03:01.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:01.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.146695+0000", 2023-07-02T14:03:01.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022307080000000001 2023-07-02T14:03:01.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.148926+0000", 2023-07-02T14:03:01.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.626e-05 2023-07-02T14:03:01.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.148942+0000", 2023-07-02T14:03:01.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4460000000000004e-06 2023-07-02T14:03:01.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.148949+0000", 2023-07-02T14:03:01.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2187999999999994e-05 2023-07-02T14:03:01.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.149022+0000", 2023-07-02T14:03:01.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3311000000000001e-05 2023-07-02T14:03:01.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.149045+0000", 2023-07-02T14:03:01.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00094571599999999998 2023-07-02T14:03:01.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.149991+0000", 2023-07-02T14:03:01.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032948410000000002 2023-07-02T14:03:01.987 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.987 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.987 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.988 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:169af44a:::rbd_data.125de1449605.0000000000000106:head, version: 40'57, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:169af44a:::rbd_data.125de1449605.0000000000000106:head@40'57, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:03:01.988 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.151904+0000", 2023-07-02T14:03:01.988 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.802723043, 2023-07-02T14:03:01.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031662180000000002, 2023-07-02T14:03:01.988 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.989 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:01.989 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.151904+0000", 2023-07-02T14:03:01.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.151904+0000", 2023-07-02T14:03:01.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:01.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.151903+0000", 2023-07-02T14:03:01.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001946234 2023-07-02T14:03:01.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.153849+0000", 2023-07-02T14:03:01.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3519999999999999e-05 2023-07-02T14:03:01.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.153863+0000", 2023-07-02T14:03:01.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6320000000000003e-06 2023-07-02T14:03:01.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:01.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.153873+0000", 2023-07-02T14:03:01.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0098000000000005e-05 2023-07-02T14:03:01.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:01.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.153943+0000", 2023-07-02T14:03:01.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1810000000000007e-06 2023-07-02T14:03:01.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:01.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.153952+0000", 2023-07-02T14:03:01.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0011181240000000001 2023-07-02T14:03:01.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:01.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.155070+0000", 2023-07-02T14:03:01.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031662180000000002 2023-07-02T14:03:01.995 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.995 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:01.995 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:01.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.995 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:1971cf55:::rbd_data.125de1449605.00000000000000e0:head, version: 40'79, data_included: [0~8192], data_size: 8192, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:1971cf55:::rbd_data.125de1449605.00000000000000e0:head@40'79, size: 8192, copy_subset: [0~8192], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:8192, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-07-02T14:03:01.995 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.156685+0000", 2023-07-02T14:03:01.996 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.797941333000001, 2023-07-02T14:03:01.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00038141899999999997, 2023-07-02T14:03:01.996 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:01.996 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:01.996 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:01.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:01.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.156685+0000", 2023-07-02T14:03:01.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:01.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:01.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.156685+0000", 2023-07-02T14:03:01.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:01.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:01.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.156685+0000", 2023-07-02T14:03:01.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3300999999999999e-05 2023-07-02T14:03:01.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:01.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.156708+0000", 2023-07-02T14:03:01.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3427e-05 2023-07-02T14:03:01.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:01.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:01.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:01.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.156722+0000", 2023-07-02T14:03:02.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3539999999999996e-06 2023-07-02T14:03:02.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:02.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.156728+0000", 2023-07-02T14:03:02.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9743999999999998e-05 2023-07-02T14:03:02.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:02.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.156768+0000", 2023-07-02T14:03:02.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3409999999999996e-06 2023-07-02T14:03:02.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:02.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.156772+0000", 2023-07-02T14:03:02.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000294841 2023-07-02T14:03:02.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:02.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.157067+0000", 2023-07-02T14:03:02.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00038141899999999997 2023-07-02T14:03:02.002 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:02.002 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:02.003 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:02.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.003 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:15e08356:::rbd_data.125de1449605.0000000000000002:head, version: 40'112, data_included: [0~131072,135168~917504], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:15e08356:::rbd_data.125de1449605.0000000000000002:head@40'112, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1052672, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-07-02T14:03:02.003 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.159064+0000", 2023-07-02T14:03:02.003 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.7955623, 2023-07-02T14:03:02.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036074649999999998, 2023-07-02T14:03:02.003 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:02.004 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:02.004 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:02.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:02.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.159064+0000", 2023-07-02T14:03:02.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:02.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:02.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.159064+0000", 2023-07-02T14:03:02.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:02.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:02.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.159064+0000", 2023-07-02T14:03:02.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024697149999999999 2023-07-02T14:03:02.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:02.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.161533+0000", 2023-07-02T14:03:02.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5977999999999998e-05 2023-07-02T14:03:02.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:02.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.161549+0000", 2023-07-02T14:03:02.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5580000000000003e-06 2023-07-02T14:03:02.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:02.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.161557+0000", 2023-07-02T14:03:02.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4352999999999998e-05 2023-07-02T14:03:02.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:02.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.161631+0000", 2023-07-02T14:03:02.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8870000000000004e-06 2023-07-02T14:03:02.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:02.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.161641+0000", 2023-07-02T14:03:02.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0010307560000000001 2023-07-02T14:03:02.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:02.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.162672+0000", 2023-07-02T14:03:02.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036074649999999998 2023-07-02T14:03:02.010 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:02.010 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:02.010 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:02.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.011 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:15e08356:::rbd_data.125de1449605.0000000000000002:head, version: 40'112, data_included: [1052672~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:15e08356:::rbd_data.125de1449605.0000000000000002:head@40'112, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2101248, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1052672, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:03:02.011 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.164941+0000", 2023-07-02T14:03:02.011 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.789685868999999, 2023-07-02T14:03:02.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002871644, 2023-07-02T14:03:02.011 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:02.011 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:02.011 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:02.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:02.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.164941+0000", 2023-07-02T14:03:02.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:02.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:02.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.164941+0000", 2023-07-02T14:03:02.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:02.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:02.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.164940+0000", 2023-07-02T14:03:02.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001801223 2023-07-02T14:03:02.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:02.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.166741+0000", 2023-07-02T14:03:02.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5262999999999999e-05 2023-07-02T14:03:02.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:02.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.166757+0000", 2023-07-02T14:03:02.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2327000000000001e-05 2023-07-02T14:03:02.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:02.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.166769+0000", 2023-07-02T14:03:02.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9894000000000005e-05 2023-07-02T14:03:02.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:02.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.166839+0000", 2023-07-02T14:03:02.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4280000000000002e-06 2023-07-02T14:03:02.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:02.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.166848+0000", 2023-07-02T14:03:02.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00096417599999999997 2023-07-02T14:03:02.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:02.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.167813+0000", 2023-07-02T14:03:02.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002871644 2023-07-02T14:03:02.017 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:02.018 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:02.018 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:02.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.018 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:15e08356:::rbd_data.125de1449605.0000000000000002:head, version: 40'112, data_included: [2101248~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:15e08356:::rbd_data.125de1449605.0000000000000002:head@40'112, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3149824, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2101248, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:03:02.018 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.169810+0000", 2023-07-02T14:03:02.018 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.784816920000001, 2023-07-02T14:03:02.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025331210000000002, 2023-07-02T14:03:02.019 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:02.019 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:02.019 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:02.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:02.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.169810+0000", 2023-07-02T14:03:02.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:02.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:02.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.169810+0000", 2023-07-02T14:03:02.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:02.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:02.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.169809+0000", 2023-07-02T14:03:02.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0016680919999999999 2023-07-02T14:03:02.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:02.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.171477+0000", 2023-07-02T14:03:02.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5795999999999999e-05 2023-07-02T14:03:02.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:02.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.171493+0000", 2023-07-02T14:03:02.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7649999999999993e-06 2023-07-02T14:03:02.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:02.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.171501+0000", 2023-07-02T14:03:02.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4740000000000002e-05 2023-07-02T14:03:02.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:02.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.171546+0000", 2023-07-02T14:03:02.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6750000000000001e-06 2023-07-02T14:03:02.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:02.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.171550+0000", 2023-07-02T14:03:02.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00079280099999999999 2023-07-02T14:03:02.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:02.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.172343+0000", 2023-07-02T14:03:02.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025331210000000002 2023-07-02T14:03:02.025 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:02.025 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:02.025 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:02.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.026 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:15e08356:::rbd_data.125de1449605.0000000000000002:head, version: 40'112, data_included: [3149824~1044480], data_size: 1044480, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:15e08356:::rbd_data.125de1449605.0000000000000002:head@40'112, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3149824, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:03:02.026 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.174306+0000", 2023-07-02T14:03:02.026 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.780320582, 2023-07-02T14:03:02.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0021748200000000001, 2023-07-02T14:03:02.026 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:02.026 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:02.027 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:02.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:02.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.174306+0000", 2023-07-02T14:03:02.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:02.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:02.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.174306+0000", 2023-07-02T14:03:02.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:02.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:02.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.174305+0000", 2023-07-02T14:03:02.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00092398899999999995 2023-07-02T14:03:02.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:02.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.175229+0000", 2023-07-02T14:03:02.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5741999999999999e-05 2023-07-02T14:03:02.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:02.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.175245+0000", 2023-07-02T14:03:02.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2989999999999999e-06 2023-07-02T14:03:02.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:02.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.175252+0000", 2023-07-02T14:03:02.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9908999999999998e-05 2023-07-02T14:03:02.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:02.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.175322+0000", 2023-07-02T14:03:02.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4709999999999996e-06 2023-07-02T14:03:02.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:02.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.175332+0000", 2023-07-02T14:03:02.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0011491889999999999 2023-07-02T14:03:02.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:02.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.176481+0000", 2023-07-02T14:03:02.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0021748200000000001 2023-07-02T14:03:02.033 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:02.033 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:02.034 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:02.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.034 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:1c72c23d:::rbd_object_map.125de1449605:head, version: 40'119, data_included: [0~162], data_size: 162, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(2:1c72c23d:::rbd_object_map.125de1449605:head@40'119, size: 162, copy_subset: [0~162], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:162, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-07-02T14:03:02.034 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.177916+0000", 2023-07-02T14:03:02.034 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.776710804, 2023-07-02T14:03:02.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00034433199999999998, 2023-07-02T14:03:02.034 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:02.035 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:02.035 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:02.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:02.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.177916+0000", 2023-07-02T14:03:02.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:02.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:02.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.177916+0000", 2023-07-02T14:03:02.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:02.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:02.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.177915+0000", 2023-07-02T14:03:02.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0332999999999999e-05 2023-07-02T14:03:02.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:02.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.177926+0000", 2023-07-02T14:03:02.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2816000000000001e-05 2023-07-02T14:03:02.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:02.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.177939+0000", 2023-07-02T14:03:02.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4629999999999998e-06 2023-07-02T14:03:02.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:02.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.177945+0000", 2023-07-02T14:03:02.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7746000000000001e-05 2023-07-02T14:03:02.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:02.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.177983+0000", 2023-07-02T14:03:02.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2640000000000002e-06 2023-07-02T14:03:02.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:02.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.177987+0000", 2023-07-02T14:03:02.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00027325400000000001 2023-07-02T14:03:02.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:02.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.178260+0000", 2023-07-02T14:03:02.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00034433199999999998 2023-07-02T14:03:02.040 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:02.041 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:02.041 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:02.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.041 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 45/44 [PushOp(2:1752944e:::rbd_data.125de1449605.000000000000000d:head, version: 40'120, data_included: [0~131072], data_size: 131072, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:1752944e:::rbd_data.125de1449605.000000000000000d:head@40'120, size: 131072, copy_subset: [0~131072], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:131072, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-07-02T14:03:02.041 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:48.179464+0000", 2023-07-02T14:03:02.041 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.775162849999999, 2023-07-02T14:03:02.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00049279599999999999, 2023-07-02T14:03:02.042 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:02.042 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:02.042 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:02.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:02.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.179464+0000", 2023-07-02T14:03:02.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:02.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:02.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.179464+0000", 2023-07-02T14:03:02.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:02.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:02.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.179463+0000", 2023-07-02T14:03:02.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000114058 2023-07-02T14:03:02.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:02.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.179577+0000", 2023-07-02T14:03:02.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5682999999999999e-05 2023-07-02T14:03:02.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:02.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.179593+0000", 2023-07-02T14:03:02.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2030000000000003e-06 2023-07-02T14:03:02.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:02.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.179600+0000", 2023-07-02T14:03:02.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7911999999999999e-05 2023-07-02T14:03:02.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:02.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.179638+0000", 2023-07-02T14:03:02.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2679999999999996e-06 2023-07-02T14:03:02.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:02.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.179642+0000", 2023-07-02T14:03:02.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00031440100000000001 2023-07-02T14:03:02.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:02.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:48.179957+0000", 2023-07-02T14:03:02.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00049279599999999999 2023-07-02T14:03:02.048 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:02.048 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:02.048 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:02.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.048 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:905 2.0 e48/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 48'136, mlcod=47'134)", 2023-07-02T14:03:02.049 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:51.077909+0000", 2023-07-02T14:03:02.049 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 10.876717405000001, 2023-07-02T14:03:02.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00069600100000000002, 2023-07-02T14:03:02.049 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:02.049 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:02.049 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:02.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:02.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:51.077909+0000", 2023-07-02T14:03:02.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:02.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:02.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:51.077909+0000", 2023-07-02T14:03:02.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:03:02.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:02.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:51.077909+0000", 2023-07-02T14:03:02.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4449999999999998e-06 2023-07-02T14:03:02.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:02.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:51.077914+0000", 2023-07-02T14:03:02.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3623e-05 2023-07-02T14:03:02.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:02.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:51.077927+0000", 2023-07-02T14:03:02.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.682e-06 2023-07-02T14:03:02.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:02.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:51.077933+0000", 2023-07-02T14:03:02.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9118000000000001e-05 2023-07-02T14:03:02.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:02.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:51.077972+0000", 2023-07-02T14:03:02.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3832000000000001e-05 2023-07-02T14:03:02.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:02.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:51.077986+0000", 2023-07-02T14:03:02.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00059268799999999996 2023-07-02T14:03:02.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:02.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:51.078579+0000", 2023-07-02T14:03:02.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6829e-05 2023-07-02T14:03:02.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:02.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:51.078605+0000", 2023-07-02T14:03:02.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00069600100000000002 2023-07-02T14:03:02.056 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:02.056 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:02.056 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:02.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.056 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:924 2.0 e49/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 49'137, mlcod=47'135)", 2023-07-02T14:03:02.056 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:53.171281+0000", 2023-07-02T14:03:02.056 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7833456420000005, 2023-07-02T14:03:02.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001000246, 2023-07-02T14:03:02.057 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:02.057 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:02.057 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:02.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:02.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:53.171281+0000", 2023-07-02T14:03:02.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:02.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:02.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:53.171281+0000", 2023-07-02T14:03:02.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:03:02.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:02.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:53.171281+0000", 2023-07-02T14:03:02.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8080000000000001e-06 2023-07-02T14:03:02.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:02.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:53.171287+0000", 2023-07-02T14:03:02.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6954000000000001e-05 2023-07-02T14:03:02.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:02.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:53.171304+0000", 2023-07-02T14:03:02.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7519999999999996e-06 2023-07-02T14:03:02.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:02.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:53.171311+0000", 2023-07-02T14:03:02.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4465999999999997e-05 2023-07-02T14:03:02.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:02.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:53.171356+0000", 2023-07-02T14:03:02.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7768e-05 2023-07-02T14:03:02.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:02.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:53.171374+0000", 2023-07-02T14:03:02.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087457099999999999 2023-07-02T14:03:02.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:02.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:53.172248+0000", 2023-07-02T14:03:02.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3141e-05 2023-07-02T14:03:02.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:02.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:02.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:02.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:53.172281+0000", 2023-07-02T14:03:02.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001000246 2023-07-02T14:03:02.063 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:02.063 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:02.064 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:02.064 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:02.064 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:02.064 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:02.065 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:03:02.066 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:02.064+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:02.113 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:03:02.168 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:02.167+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:02.233 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:02.233 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:02.234 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:03:02.234 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:02.244 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:03:02.266 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:02.265+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:02.304 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:03:02.367 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:02.366+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:02.368 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:02.368 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:02.368 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:02.368 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:02.368 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:02.378 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:03:02.452 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:02.468 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:02.467+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:02.469 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:02.487 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:02.504 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:02.522 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:02.541 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:03:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:03:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:03:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:03:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:03:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:03:02.544 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:02.544 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:02.554 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:03:02.595 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:02.594+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:02.670 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:02.670+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:02.678 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:02.678 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:02.688 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:03:02.771 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:02.770+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:02.812 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:02.813 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:02.822 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:03:02.872 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:02.872+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:02.972 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:02.972+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:02.985 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:02.985 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:02.996 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:03:03.074 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:03.073+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:03.108 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2023-07-02T14:03:03.108 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:03:03.109 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2023-07-02T14:03:03.109 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-07-02T14:03:03.109 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-07-02T14:03:03.109 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2023-07-02T14:03:03.120 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:03.120 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:03.131 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:03:03.175 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:03.174+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:03.238 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:03.238 INFO:teuthology.orchestra.run.smithi022.stdout: "success": "" 2023-07-02T14:03:03.238 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:03.258 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:03.258 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:03.268 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:03:03.311 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:03.310+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:03.376 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:03.375+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:03.392 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:03.392 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:03.402 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:03:03.485 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:03.484+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:03.566 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:03.567 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:03.577 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:03:03.618 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:03.617+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:03.678 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:03.678+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:03.741 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:03.741 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:03.751 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:03:03.795 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:03.794+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:03.875 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:03.875 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:03.881 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:03.880+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:03.885 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:03:03.981 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:03.981+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:04.012 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:04.012 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:04.022 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:03:04.082 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:04.081+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:04.128 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:03:04.144 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:04.145 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:04.155 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:03:04.182 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:04.181+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:04.279 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:04.279 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:04.283 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:04.282+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:04.289 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:03:04.384 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:04.383+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:04.413 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:04.413 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:04.423 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:03:04.484 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:04.483+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:04.555 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:03:04.586 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:04.586 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:04.596 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:03:04.598 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:04.597+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:04.629 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete...+ tar -C b3fbb028-e267-4d25-a750-93ba18a2316a-mnt -xzf b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4.tar.gz 2023-07-02T14:03:04.686 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:04.685+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:04.693 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:04.710 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:04.719 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:04.719 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:04.727 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:04.729 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:03:04.745 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:04.763 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:04.781 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:03:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:03:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:03:04.782 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:03:04.783 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:03:04.783 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:03:04.787 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:04.786+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:04.887 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:04.886+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:04.893 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:04.894 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:04.894 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:03:04.894 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:04.903 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:03:04.987 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:04.986+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:05.026 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:05.026 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:05.027 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:05.027 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:05.027 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:05.036 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:03:05.088 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:05.087+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:05.162 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:05.162 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:05.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:05.162 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:05.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.163 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:05.163 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:05.163 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.869383976999998, 2023-07-02T14:03:05.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:03:05.163 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:05.163 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.164 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:05.164 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:05.164 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:05.164 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:03:05.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.164 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:05.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:05.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:05.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:05.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:05.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:05.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:05.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:05.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:03:05.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:03:05.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:05.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:03:05.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:03:05.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:05.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:03:05.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:03:05.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:03:05.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:03:05.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:05.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:03:05.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:03:05.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:05.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:03:05.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:03:05.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:05.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:03:05.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:03:05.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:05.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:03:05.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:03:05.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:05.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:03:05.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:03:05.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:05.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:03:05.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:03:05.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:05.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:03:05.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:03:05.173 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.173 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.173 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.174 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:05.174 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:05.174 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.869148954, 2023-07-02T14:03:05.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:03:05.174 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:05.174 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.174 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:05.175 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:05.175 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:05.175 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:03:05.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.175 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:05.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:05.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:05.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:05.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:05.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:05.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:05.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:05.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:03:05.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:03:05.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:05.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:03:05.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:03:05.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:05.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:03:05.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:03:05.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:03:05.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:03:05.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:05.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:03:05.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:03:05.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:05.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:03:05.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:03:05.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:05.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:03:05.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:03:05.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:05.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:03:05.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:03:05.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:05.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:03:05.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:03:05.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:05.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:03:05.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:03:05.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:05.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:03:05.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:03:05.187 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.188 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.188 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.189 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:05.189 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:05.189 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.868949820000001, 2023-07-02T14:03:05.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:03:05.189 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:05.189 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.189 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:05.190 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:05.190 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:05.190 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:03:05.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.190 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:05.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:05.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:05.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:05.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:05.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:05.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:05.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:05.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:03:05.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:03:05.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:05.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:03:05.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:03:05.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:05.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:03:05.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:03:05.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:03:05.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:03:05.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:05.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:03:05.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:03:05.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:05.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:03:05.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:03:05.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:05.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:03:05.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:03:05.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:05.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:03:05.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:03:05.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:05.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:03:05.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:03:05.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:05.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:03:05.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:03:05.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:05.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:03:05.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:03:05.199 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.199 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.199 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.199 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:05.200 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:05.200 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.868584695999999, 2023-07-02T14:03:05.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:03:05.200 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:05.200 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.200 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:05.201 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:05.201 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:05.201 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:03:05.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.201 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:05.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:05.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:05.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:05.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:05.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:05.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:05.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:05.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:03:05.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:03:05.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:05.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:03:05.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:03:05.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:05.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:03:05.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:03:05.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:03:05.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:03:05.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:05.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:03:05.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:03:05.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:05.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:03:05.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:03:05.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:05.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:03:05.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:03:05.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:05.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:03:05.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:03:05.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:05.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:03:05.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:03:05.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:05.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:03:05.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:03:05.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:05.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:03:05.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:03:05.210 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.210 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.210 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.210 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:05.210 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:05.210 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.868503635, 2023-07-02T14:03:05.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:03:05.211 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:05.211 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.211 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:05.211 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:05.211 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:05.212 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:03:05.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.212 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:05.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:05.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:05.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:05.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:05.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:05.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:05.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:05.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:03:05.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:03:05.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:05.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:03:05.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:03:05.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:05.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:03:05.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:03:05.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:03:05.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:03:05.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:05.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:03:05.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:03:05.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:05.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:03:05.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:03:05.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:05.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:03:05.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:03:05.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:05.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:03:05.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:03:05.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:05.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:03:05.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:03:05.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:05.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:03:05.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:03:05.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:05.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:03:05.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:03:05.220 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.220 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.221 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.221 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:05.221 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:05.221 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.868411964, 2023-07-02T14:03:05.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:03:05.222 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:05.222 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.222 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:05.222 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:05.222 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:05.222 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:03:05.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.223 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:05.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:05.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:05.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:05.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:05.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:05.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:05.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:05.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:03:05.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:03:05.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:05.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:03:05.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:03:05.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:05.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:03:05.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:03:05.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:03:05.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:03:05.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:05.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:03:05.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:03:05.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:05.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:03:05.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:03:05.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:05.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:03:05.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:03:05.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:05.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:03:05.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:03:05.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:05.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:03:05.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:03:05.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.230 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:05.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:03:05.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:03:05.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:05.231 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:03:05.231 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:03:05.231 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.231 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.231 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.232 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:05.232 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:05.232 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.868072521999999, 2023-07-02T14:03:05.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:03:05.232 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:05.232 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.233 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:05.233 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:05.233 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:05.233 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:03:05.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.233 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:05.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:05.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:05.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:05.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:05.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:05.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:05.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:05.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:03:05.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:03:05.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:05.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:03:05.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:03:05.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:05.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:03:05.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:03:05.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:03:05.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:03:05.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:05.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:03:05.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:03:05.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:05.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:03:05.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:03:05.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:05.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:03:05.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:03:05.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:05.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:03:05.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:03:05.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.240 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:05.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:03:05.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:03:05.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:05.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:03:05.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:03:05.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:05.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:03:05.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:03:05.242 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.242 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.242 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.242 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:05.243 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:05.243 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.867886468999998, 2023-07-02T14:03:05.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:03:05.243 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:05.243 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.243 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:05.243 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:05.244 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:05.244 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:03:05.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.244 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:05.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:05.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:05.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:05.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:05.245 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:05.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:05.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:05.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:05.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:03:05.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.246 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.246 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:05.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:03:05.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:03:05.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:05.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:03:05.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:03:05.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:03:05.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:03:05.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:05.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:03:05.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:03:05.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:05.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:03:05.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:03:05.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:05.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:03:05.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:03:05.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:05.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:03:05.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:03:05.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.251 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:05.251 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:03:05.251 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:03:05.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.251 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:05.251 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:03:05.251 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:03:05.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.252 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:05.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:03:05.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:03:05.252 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.253 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.253 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.253 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:05.253 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:05.253 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.867728667000002, 2023-07-02T14:03:05.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:03:05.254 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:05.254 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.254 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:05.254 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:05.254 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:05.254 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:03:05.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.255 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:05.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:05.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:05.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:05.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:05.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:05.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:05.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:05.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:03:05.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:03:05.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:05.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:03:05.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:03:05.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:05.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:03:05.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:03:05.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:03:05.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:03:05.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:05.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:03:05.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:03:05.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:05.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:03:05.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:03:05.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:05.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:03:05.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:03:05.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.261 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:05.261 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:03:05.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:03:05.261 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.261 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:05.261 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:03:05.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:03:05.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.262 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.262 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:05.262 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:03:05.262 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:03:05.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.262 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:05.263 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:03:05.263 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:03:05.263 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.263 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.263 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.264 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:05.264 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:05.264 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.867536885, 2023-07-02T14:03:05.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:03:05.264 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:05.265 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.265 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:05.265 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:05.265 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:05.265 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:03:05.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.265 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:05.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:05.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:05.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:05.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:05.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:05.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:05.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:05.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:03:05.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:03:05.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:05.268 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:03:05.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:03:05.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:05.268 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:03:05.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:03:05.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:03:05.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:03:05.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:05.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:03:05.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:03:05.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:05.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:03:05.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:03:05.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.271 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:05.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:03:05.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:03:05.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.271 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:05.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:03:05.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:03:05.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:05.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:03:05.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:03:05.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.273 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:05.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:03:05.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:03:05.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.273 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:05.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:03:05.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:03:05.274 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.274 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.274 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.275 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:05.275 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:05.275 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.867361225, 2023-07-02T14:03:05.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:03:05.275 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:05.275 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.275 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:05.275 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:05.276 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:05.276 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:03:05.276 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.276 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:05.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:05.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:05.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:05.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.277 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:05.277 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:05.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:05.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:05.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:05.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:03:05.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:05.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:03:05.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:03:05.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:05.279 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:03:05.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:03:05.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:03:05.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:03:05.280 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:05.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:03:05.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:03:05.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:05.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:03:05.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:03:05.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:05.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:03:05.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:03:05.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.282 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:05.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:03:05.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:03:05.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:05.283 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:03:05.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:03:05.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:05.283 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:03:05.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:03:05.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:05.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:03:05.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:03:05.285 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.285 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.285 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.285 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:05.285 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:05.286 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.867267037000001, 2023-07-02T14:03:05.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:03:05.286 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:05.286 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.286 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:05.286 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:05.286 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:05.286 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:03:05.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.287 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:05.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:05.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:05.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:05.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:05.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:05.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:05.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:05.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:03:05.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:03:05.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:05.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:03:05.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:03:05.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:05.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:03:05.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:03:05.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:03:05.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:03:05.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:05.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:03:05.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:03:05.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:05.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:03:05.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:03:05.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:05.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:03:05.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:03:05.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:05.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:03:05.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:03:05.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:05.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:03:05.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:03:05.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:05.294 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:03:05.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:03:05.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.295 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:05.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:03:05.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:03:05.295 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.295 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.296 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.296 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:05.296 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:05.296 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.867063650999999, 2023-07-02T14:03:05.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:03:05.297 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:05.297 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.297 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:05.297 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:05.297 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:05.297 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:03:05.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.297 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:05.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:05.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:05.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:05.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:05.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:05.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:05.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:05.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:03:05.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:03:05.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:05.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:03:05.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:03:05.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:05.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:03:05.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:03:05.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:03:05.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:03:05.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:05.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:03:05.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:03:05.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:05.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:03:05.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:03:05.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:05.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:03:05.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:03:05.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:05.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:03:05.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:03:05.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:05.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:03:05.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:03:05.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:05.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:03:05.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:03:05.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:05.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:03:05.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:03:05.306 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.306 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.306 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.307 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:05.307 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:05.307 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.866912995, 2023-07-02T14:03:05.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:03:05.307 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:05.307 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.307 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:05.308 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:05.308 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:05.308 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:03:05.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.308 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:05.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:05.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:05.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:05.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:05.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:05.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:05.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:05.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:03:05.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:03:05.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:05.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:03:05.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:03:05.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:05.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:03:05.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:03:05.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:03:05.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:03:05.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:05.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:03:05.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:03:05.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:05.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:03:05.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:03:05.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:05.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:03:05.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:03:05.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:05.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:03:05.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:03:05.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:05.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:03:05.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:03:05.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:05.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:03:05.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:03:05.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:05.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:03:05.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:03:05.317 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.317 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.317 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.317 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:05.317 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:05.318 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.866840291999999, 2023-07-02T14:03:05.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:03:05.318 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:05.318 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.318 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:05.318 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:05.318 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:05.318 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:03:05.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.319 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:05.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:05.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:05.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:05.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:05.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:05.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:05.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:05.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:05.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:03:05.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:05.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:03:05.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:03:05.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.322 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:05.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:03:05.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:03:05.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.322 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:03:05.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:03:05.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:05.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:03:05.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:03:05.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:05.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:03:05.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:03:05.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.324 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:05.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:03:05.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:03:05.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:05.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:03:05.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:03:05.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:05.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:03:05.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:03:05.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:05.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:03:05.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:03:05.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:05.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:03:05.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:03:05.327 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.327 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.328 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.328 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:05.328 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:05.328 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.866723302, 2023-07-02T14:03:05.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:03:05.329 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:05.329 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.329 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:05.329 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:05.329 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:05.329 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:03:05.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.329 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:05.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:05.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:05.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:05.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:05.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:05.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:05.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:05.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:05.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:03:05.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:05.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:03:05.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:03:05.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:05.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:03:05.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:03:05.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:03:05.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:03:05.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:05.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:03:05.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:03:05.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:05.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:03:05.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:03:05.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:05.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:03:05.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:03:05.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:05.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:03:05.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:03:05.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:05.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:03:05.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:03:05.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:05.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:03:05.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:03:05.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:05.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:03:05.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:03:05.338 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.338 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.338 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.339 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:05.339 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:05.339 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.866573850999998, 2023-07-02T14:03:05.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:03:05.339 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:05.339 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.339 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:05.340 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:05.340 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:05.340 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:03:05.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.340 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:05.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:05.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:05.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:05.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:05.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:05.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:05.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:05.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:03:05.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:03:05.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:05.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:03:05.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:03:05.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:05.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:03:05.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:03:05.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:03:05.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:03:05.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:05.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:03:05.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:03:05.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:05.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:03:05.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:03:05.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:05.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:03:05.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:03:05.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:05.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:03:05.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:03:05.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:05.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:03:05.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:03:05.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:05.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:03:05.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:03:05.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:05.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:03:05.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:03:05.349 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.349 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.349 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.349 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:05.350 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:05.350 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.845097878000001, 2023-07-02T14:03:05.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:03:05.350 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:05.350 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.350 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:05.350 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:05.350 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:05.351 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:03:05.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.351 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:05.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:05.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:05.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:05.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:05.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:05.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:05.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:05.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:03:05.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:03:05.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:05.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:03:05.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:03:05.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:05.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:03:05.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:03:05.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:03:05.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:03:05.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:05.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:03:05.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:03:05.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:05.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:03:05.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:03:05.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:05.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:03:05.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:03:05.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:05.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:03:05.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:03:05.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:05.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:03:05.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:03:05.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:05.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:03:05.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:03:05.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:05.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:03:05.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:03:05.360 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.360 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.360 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.360 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:05.360 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:05.361 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.844965341999998, 2023-07-02T14:03:05.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:03:05.361 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:05.361 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.361 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:05.361 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:05.361 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:05.361 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:03:05.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.362 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:05.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:05.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:05.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:05.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:05.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:05.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:05.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:05.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:03:05.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:03:05.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:05.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:03:05.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:03:05.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:05.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:03:05.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:03:05.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:03:05.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:03:05.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:05.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:03:05.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:03:05.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:05.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:03:05.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:03:05.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:05.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:03:05.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:03:05.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:05.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:03:05.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:03:05.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:05.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:03:05.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:03:05.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:05.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:03:05.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:03:05.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:05.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:03:05.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:03:05.370 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.370 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.371 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.371 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:05.371 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:05.371 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.783628681, 2023-07-02T14:03:05.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:03:05.371 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:05.372 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.372 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:05.372 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:05.372 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:05.372 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:03:05.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.372 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:05.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:05.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:05.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:05.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:05.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:05.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:05.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:05.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:03:05.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:03:05.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:05.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:03:05.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:03:05.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:05.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:03:05.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:03:05.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:03:05.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:03:05.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:05.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:03:05.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:03:05.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:05.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:03:05.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:03:05.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:05.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:03:05.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:03:05.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:05.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:03:05.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:03:05.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:05.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:03:05.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:03:05.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:05.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:03:05.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:03:05.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:05.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:05.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:05.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:03:05.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:03:05.381 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.381 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.381 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.381 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:05.381 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:05.382 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:05.383 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 100% complete...done. 2023-07-02T14:03:05.383 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 4 2023-07-02T14:03:05.384 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:03:05.384 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:05.383+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:05.384 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:03:05.483 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:05.482+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:05.574 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:05.574 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:05.574 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:05.574 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:05.584 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:05.582+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:05.593 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:03:05.685 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:05.684+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:05.786 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:05.784+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:05.787 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:05.787 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:05.787 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:05.788 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:05.788 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:05.797 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:03:05.886 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:05.885+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:05.962 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:05.963 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:03:05.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:03:05.963 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:05.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.963 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:426 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'42, mlcod=40'38)", 2023-07-02T14:03:05.963 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.340864+0000", 2023-07-02T14:03:05.963 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.620037496999998, 2023-07-02T14:03:05.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065109469999999996, 2023-07-02T14:03:05.964 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:05.964 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.964 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:05.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:05.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.340864+0000", 2023-07-02T14:03:05.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:05.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:05.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.340864+0000", 2023-07-02T14:03:05.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:05.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:05.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.340863+0000", 2023-07-02T14:03:05.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019354159999999999 2023-07-02T14:03:05.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:05.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342799+0000", 2023-07-02T14:03:05.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5824e-05 2023-07-02T14:03:05.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:05.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342814+0000", 2023-07-02T14:03:05.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:03:05.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342822+0000", 2023-07-02T14:03:05.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7465999999999994e-05 2023-07-02T14:03:05.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:05.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342889+0000", 2023-07-02T14:03:05.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5068000000000001e-05 2023-07-02T14:03:05.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:05.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342914+0000", 2023-07-02T14:03:05.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044200969999999996 2023-07-02T14:03:05.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:05.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347334+0000", 2023-07-02T14:03:05.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0711999999999999e-05 2023-07-02T14:03:05.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:05.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347375+0000", 2023-07-02T14:03:05.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065109469999999996 2023-07-02T14:03:05.970 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:05.971 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:05.971 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:05.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.971 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:431 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'4, mlcod=40'2)", 2023-07-02T14:03:05.971 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.350267+0000", 2023-07-02T14:03:05.971 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.610634031, 2023-07-02T14:03:05.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077388420000000001, 2023-07-02T14:03:05.972 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:05.972 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.972 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:05.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:05.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350267+0000", 2023-07-02T14:03:05.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:05.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:05.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350267+0000", 2023-07-02T14:03:05.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:05.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:05.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350266+0000", 2023-07-02T14:03:05.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024191569999999999 2023-07-02T14:03:05.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:05.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352685+0000", 2023-07-02T14:03:05.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0013e-05 2023-07-02T14:03:05.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:05.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352706+0000", 2023-07-02T14:03:05.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3250000000000008e-06 2023-07-02T14:03:05.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352714+0000", 2023-07-02T14:03:05.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9648000000000003e-05 2023-07-02T14:03:05.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:05.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352763+0000", 2023-07-02T14:03:05.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4496999999999999e-05 2023-07-02T14:03:05.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:05.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352788+0000", 2023-07-02T14:03:05.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051724880000000003 2023-07-02T14:03:05.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:05.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357960+0000", 2023-07-02T14:03:05.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5802e-05 2023-07-02T14:03:05.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:05.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358006+0000", 2023-07-02T14:03:05.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077388420000000001 2023-07-02T14:03:05.978 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:05.978 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:05.978 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:05.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.979 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:455 2.5 e40/14 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:head v 40'12, mlcod=40'10)", 2023-07-02T14:03:05.979 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.376227+0000", 2023-07-02T14:03:05.979 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.584674753000002, 2023-07-02T14:03:05.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012681951, 2023-07-02T14:03:05.979 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:05.980 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.980 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:05.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:05.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.376227+0000", 2023-07-02T14:03:05.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:05.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:05.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.376227+0000", 2023-07-02T14:03:05.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:05.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:05.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.376226+0000", 2023-07-02T14:03:05.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040198860000000003 2023-07-02T14:03:05.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:05.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380245+0000", 2023-07-02T14:03:05.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7640000000000001e-05 2023-07-02T14:03:05.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:05.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380263+0000", 2023-07-02T14:03:05.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3299999999999999e-06 2023-07-02T14:03:05.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380271+0000", 2023-07-02T14:03:05.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4322e-05 2023-07-02T14:03:05.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:05.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380316+0000", 2023-07-02T14:03:05.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7682000000000001e-05 2023-07-02T14:03:05.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:05.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380333+0000", 2023-07-02T14:03:05.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085223439999999994 2023-07-02T14:03:05.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:05.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.388856+0000", 2023-07-02T14:03:05.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2930999999999999e-05 2023-07-02T14:03:05.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:05.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.388909+0000", 2023-07-02T14:03:05.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012681951 2023-07-02T14:03:05.986 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:05.986 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:05.986 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:05.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.987 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:456 2.5 e40/14 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:head v 40'13, mlcod=40'10)", 2023-07-02T14:03:05.987 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.382266+0000", 2023-07-02T14:03:05.987 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.578635126000002, 2023-07-02T14:03:05.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075568149999999997, 2023-07-02T14:03:05.987 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:05.987 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.987 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:05.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:05.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382266+0000", 2023-07-02T14:03:05.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:05.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:05.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382266+0000", 2023-07-02T14:03:05.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:05.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:05.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382264+0000", 2023-07-02T14:03:05.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012720599999999999 2023-07-02T14:03:05.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:05.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382391+0000", 2023-07-02T14:03:05.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7598e-05 2023-07-02T14:03:05.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:05.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382409+0000", 2023-07-02T14:03:05.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8660000000000001e-06 2023-07-02T14:03:05.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382419+0000", 2023-07-02T14:03:05.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001837403 2023-07-02T14:03:05.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:05.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384256+0000", 2023-07-02T14:03:05.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5757000000000001e-05 2023-07-02T14:03:05.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:05.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384282+0000", 2023-07-02T14:03:05.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055044730000000002 2023-07-02T14:03:05.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:05.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389787+0000", 2023-07-02T14:03:05.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6622000000000003e-05 2023-07-02T14:03:05.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:05.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389823+0000", 2023-07-02T14:03:05.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075568149999999997 2023-07-02T14:03:05.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:05.994 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:05.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:05.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.994 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:457 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'14, mlcod=40'10)", 2023-07-02T14:03:05.995 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.382901+0000", 2023-07-02T14:03:05.995 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.578000154000001, 2023-07-02T14:03:05.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0069428320000000003, 2023-07-02T14:03:05.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:05.995 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:05.995 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:05.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:05.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382901+0000", 2023-07-02T14:03:05.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:05.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:05.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382901+0000", 2023-07-02T14:03:05.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:05.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:05.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382900+0000", 2023-07-02T14:03:05.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7738999999999994e-05 2023-07-02T14:03:05.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:05.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382998+0000", 2023-07-02T14:03:05.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6769000000000002e-05 2023-07-02T14:03:05.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:05.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383014+0000", 2023-07-02T14:03:05.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0073e-05 2023-07-02T14:03:05.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:05.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383025+0000", 2023-07-02T14:03:05.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016687410000000001 2023-07-02T14:03:05.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:05.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:05.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:05.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384693+0000", 2023-07-02T14:03:05.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4108999999999999e-05 2023-07-02T14:03:06.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384717+0000", 2023-07-02T14:03:06.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051086830000000001 2023-07-02T14:03:06.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389826+0000", 2023-07-02T14:03:06.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8059000000000001e-05 2023-07-02T14:03:06.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389844+0000", 2023-07-02T14:03:06.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0069428320000000003 2023-07-02T14:03:06.002 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.002 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.002 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.002 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:479 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'15, mlcod=40'10)", 2023-07-02T14:03:06.002 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.383348+0000", 2023-07-02T14:03:06.002 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.577553196, 2023-07-02T14:03:06.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065068890000000001, 2023-07-02T14:03:06.003 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.003 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.003 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383348+0000", 2023-07-02T14:03:06.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383348+0000", 2023-07-02T14:03:06.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:06.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383347+0000", 2023-07-02T14:03:06.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000422123 2023-07-02T14:03:06.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383769+0000", 2023-07-02T14:03:06.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1688e-05 2023-07-02T14:03:06.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383791+0000", 2023-07-02T14:03:06.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0237e-05 2023-07-02T14:03:06.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383801+0000", 2023-07-02T14:03:06.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013799940000000001 2023-07-02T14:03:06.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.385181+0000", 2023-07-02T14:03:06.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3047000000000001e-05 2023-07-02T14:03:06.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.385204+0000", 2023-07-02T14:03:06.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046417339999999998 2023-07-02T14:03:06.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389846+0000", 2023-07-02T14:03:06.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1800000000000002e-06 2023-07-02T14:03:06.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389855+0000", 2023-07-02T14:03:06.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065068890000000001 2023-07-02T14:03:06.009 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.009 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.010 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.010 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1031 2.4 e57/34 2:2f8e3c69:::rbd_data.125de1449605.0000000000000028:head v 57'88, mlcod=57'86)", 2023-07-02T14:03:06.010 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.551534+0000", 2023-07-02T14:03:06.010 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.40936743799999997, 2023-07-02T14:03:06.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085593449999999995, 2023-07-02T14:03:06.010 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.011 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.011 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.551534+0000", 2023-07-02T14:03:06.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.551534+0000", 2023-07-02T14:03:06.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:06.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.551533+0000", 2023-07-02T14:03:06.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003576339 2023-07-02T14:03:06.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.555109+0000", 2023-07-02T14:03:06.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9423e-05 2023-07-02T14:03:06.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.555129+0000", 2023-07-02T14:03:06.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2002e-05 2023-07-02T14:03:06.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.555141+0000", 2023-07-02T14:03:06.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0914000000000001e-05 2023-07-02T14:03:06.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.555192+0000", 2023-07-02T14:03:06.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8541e-05 2023-07-02T14:03:06.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.555220+0000", 2023-07-02T14:03:06.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048274249999999998 2023-07-02T14:03:06.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.560048+0000", 2023-07-02T14:03:06.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5726999999999998e-05 2023-07-02T14:03:06.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.560093+0000", 2023-07-02T14:03:06.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085593449999999995 2023-07-02T14:03:06.017 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.017 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.017 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.018 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1097 2.5 e57/14 2:a5de3e1e:::rbd_data.125de1449605.0000000000000022:head v 57'53, mlcod=57'47)", 2023-07-02T14:03:06.018 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.594133+0000", 2023-07-02T14:03:06.018 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.36676882999999999, 2023-07-02T14:03:06.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056946139999999998, 2023-07-02T14:03:06.018 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.018 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.019 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.594133+0000", 2023-07-02T14:03:06.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.594133+0000", 2023-07-02T14:03:06.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:06.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.594132+0000", 2023-07-02T14:03:06.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022406090000000002 2023-07-02T14:03:06.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.596373+0000", 2023-07-02T14:03:06.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0052e-05 2023-07-02T14:03:06.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.596383+0000", 2023-07-02T14:03:06.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7830000000000004e-06 2023-07-02T14:03:06.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.596388+0000", 2023-07-02T14:03:06.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045496300000000001 2023-07-02T14:03:06.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.596843+0000", 2023-07-02T14:03:06.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0905999999999999e-05 2023-07-02T14:03:06.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.596864+0000", 2023-07-02T14:03:06.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029289530000000002 2023-07-02T14:03:06.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.599793+0000", 2023-07-02T14:03:06.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4078999999999997e-05 2023-07-02T14:03:06.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.599827+0000", 2023-07-02T14:03:06.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056946139999999998 2023-07-02T14:03:06.025 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.025 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.025 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.026 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1103 2.4 e57/34 2:3e650bab:::rbd_data.125de1449605.0000000000000031:head v 57'101, mlcod=57'97)", 2023-07-02T14:03:06.026 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.599608+0000", 2023-07-02T14:03:06.026 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.361293437, 2023-07-02T14:03:06.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0058272369999999999, 2023-07-02T14:03:06.026 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.026 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.026 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.599608+0000", 2023-07-02T14:03:06.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.599608+0000", 2023-07-02T14:03:06.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:06.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.599607+0000", 2023-07-02T14:03:06.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014553140000000001 2023-07-02T14:03:06.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.601062+0000", 2023-07-02T14:03:06.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1060000000000006e-06 2023-07-02T14:03:06.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.601071+0000", 2023-07-02T14:03:06.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6079999999999998e-06 2023-07-02T14:03:06.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.601076+0000", 2023-07-02T14:03:06.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3705000000000003e-05 2023-07-02T14:03:06.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.601120+0000", 2023-07-02T14:03:06.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9527000000000001e-05 2023-07-02T14:03:06.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.601139+0000", 2023-07-02T14:03:06.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042608289999999998 2023-07-02T14:03:06.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.605400+0000", 2023-07-02T14:03:06.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5345000000000001e-05 2023-07-02T14:03:06.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.605435+0000", 2023-07-02T14:03:06.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0058272369999999999 2023-07-02T14:03:06.033 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.033 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.033 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.033 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1134 2.4 e57/34 2:2211c235:::rbd_data.125de1449605.0000000000000034:head v 57'108, mlcod=57'105)", 2023-07-02T14:03:06.033 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.614673+0000", 2023-07-02T14:03:06.034 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.34622855000000002, 2023-07-02T14:03:06.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068419489999999999, 2023-07-02T14:03:06.034 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.034 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.034 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.614673+0000", 2023-07-02T14:03:06.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.614673+0000", 2023-07-02T14:03:06.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:06.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.614672+0000", 2023-07-02T14:03:06.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014647250000000001 2023-07-02T14:03:06.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.616137+0000", 2023-07-02T14:03:06.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3174e-05 2023-07-02T14:03:06.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.616150+0000", 2023-07-02T14:03:06.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6930000000000007e-06 2023-07-02T14:03:06.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.616158+0000", 2023-07-02T14:03:06.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4897000000000001e-05 2023-07-02T14:03:06.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.616202+0000", 2023-07-02T14:03:06.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1438e-05 2023-07-02T14:03:06.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.616224+0000", 2023-07-02T14:03:06.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052322990000000002 2023-07-02T14:03:06.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.621456+0000", 2023-07-02T14:03:06.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8683e-05 2023-07-02T14:03:06.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.621515+0000", 2023-07-02T14:03:06.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068419489999999999 2023-07-02T14:03:06.040 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.040 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.041 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.041 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1174 2.5 e57/14 2:a79a9f1f:::rbd_data.125de1449605.0000000000000039:head v 57'59, mlcod=57'57)", 2023-07-02T14:03:06.041 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.641967+0000", 2023-07-02T14:03:06.041 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.31893459200000002, 2023-07-02T14:03:06.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056519020000000003, 2023-07-02T14:03:06.042 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.042 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.042 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.641967+0000", 2023-07-02T14:03:06.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.641967+0000", 2023-07-02T14:03:06.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:06.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.641966+0000", 2023-07-02T14:03:06.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001901694 2023-07-02T14:03:06.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.643868+0000", 2023-07-02T14:03:06.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6974999999999999e-05 2023-07-02T14:03:06.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.643885+0000", 2023-07-02T14:03:06.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1940000000000003e-06 2023-07-02T14:03:06.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.643891+0000", 2023-07-02T14:03:06.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8118000000000002e-05 2023-07-02T14:03:06.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.643939+0000", 2023-07-02T14:03:06.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6815e-05 2023-07-02T14:03:06.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.643966+0000", 2023-07-02T14:03:06.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003614117 2023-07-02T14:03:06.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.647580+0000", 2023-07-02T14:03:06.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8729000000000001e-05 2023-07-02T14:03:06.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.647619+0000", 2023-07-02T14:03:06.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056519020000000003 2023-07-02T14:03:06.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.048 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.049 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1187 2.5 e57/14 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:head v 57'65, mlcod=57'63)", 2023-07-02T14:03:06.049 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.654963+0000", 2023-07-02T14:03:06.049 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.305938035, 2023-07-02T14:03:06.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012171026, 2023-07-02T14:03:06.049 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.049 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.050 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.654963+0000", 2023-07-02T14:03:06.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.654963+0000", 2023-07-02T14:03:06.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:06.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.654963+0000", 2023-07-02T14:03:06.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064185329999999997 2023-07-02T14:03:06.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661381+0000", 2023-07-02T14:03:06.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3459e-05 2023-07-02T14:03:06.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661405+0000", 2023-07-02T14:03:06.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0638e-05 2023-07-02T14:03:06.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661416+0000", 2023-07-02T14:03:06.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3569000000000001e-05 2023-07-02T14:03:06.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661479+0000", 2023-07-02T14:03:06.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4239000000000001e-05 2023-07-02T14:03:06.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661523+0000", 2023-07-02T14:03:06.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055722319999999999 2023-07-02T14:03:06.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.667096+0000", 2023-07-02T14:03:06.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8906999999999999e-05 2023-07-02T14:03:06.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.667134+0000", 2023-07-02T14:03:06.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012171026 2023-07-02T14:03:06.056 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.056 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.056 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.057 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1188 2.5 e57/14 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:head v 57'66, mlcod=57'63)", 2023-07-02T14:03:06.057 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.661435+0000", 2023-07-02T14:03:06.057 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.29946650600000002, 2023-07-02T14:03:06.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061507050000000002, 2023-07-02T14:03:06.057 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.057 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.057 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661435+0000", 2023-07-02T14:03:06.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661435+0000", 2023-07-02T14:03:06.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:06.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661434+0000", 2023-07-02T14:03:06.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8148e-05 2023-07-02T14:03:06.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661482+0000", 2023-07-02T14:03:06.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0789e-05 2023-07-02T14:03:06.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661492+0000", 2023-07-02T14:03:06.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8880000000000002e-06 2023-07-02T14:03:06.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661498+0000", 2023-07-02T14:03:06.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034011409999999999 2023-07-02T14:03:06.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.664900+0000", 2023-07-02T14:03:06.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4694000000000001e-05 2023-07-02T14:03:06.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.664924+0000", 2023-07-02T14:03:06.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026326829999999998 2023-07-02T14:03:06.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.667557+0000", 2023-07-02T14:03:06.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8770000000000001e-05 2023-07-02T14:03:06.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.667586+0000", 2023-07-02T14:03:06.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061507050000000002 2023-07-02T14:03:06.064 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.064 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.064 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.064 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1189 2.5 e57/14 2:adf18789:::rbd_data.125de1449605.000000000000003c:head v 57'67, mlcod=57'63)", 2023-07-02T14:03:06.064 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.661508+0000", 2023-07-02T14:03:06.065 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.299393568, 2023-07-02T14:03:06.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015893727, 2023-07-02T14:03:06.065 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.065 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.065 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661508+0000", 2023-07-02T14:03:06.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661508+0000", 2023-07-02T14:03:06.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:06.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661507+0000", 2023-07-02T14:03:06.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074929860000000001 2023-07-02T14:03:06.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669000+0000", 2023-07-02T14:03:06.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5695999999999999e-05 2023-07-02T14:03:06.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669026+0000", 2023-07-02T14:03:06.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6399999999999997e-06 2023-07-02T14:03:06.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669034+0000", 2023-07-02T14:03:06.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1940000000000001e-05 2023-07-02T14:03:06.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669086+0000", 2023-07-02T14:03:06.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5514999999999999e-05 2023-07-02T14:03:06.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669111+0000", 2023-07-02T14:03:06.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082441170000000005 2023-07-02T14:03:06.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.677355+0000", 2023-07-02T14:03:06.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6285000000000002e-05 2023-07-02T14:03:06.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.677402+0000", 2023-07-02T14:03:06.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015893727 2023-07-02T14:03:06.071 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.071 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.072 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.072 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1190 2.5 e57/14 2:adf18789:::rbd_data.125de1449605.000000000000003c:head v 57'68, mlcod=57'63)", 2023-07-02T14:03:06.072 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.669047+0000", 2023-07-02T14:03:06.072 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.29185424300000001, 2023-07-02T14:03:06.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.008388431, 2023-07-02T14:03:06.073 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.073 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.073 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669047+0000", 2023-07-02T14:03:06.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669047+0000", 2023-07-02T14:03:06.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:06.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669046+0000", 2023-07-02T14:03:06.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8430000000000001e-05 2023-07-02T14:03:06.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669074+0000", 2023-07-02T14:03:06.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9349999999999997e-06 2023-07-02T14:03:06.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669081+0000", 2023-07-02T14:03:06.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0469999999999998e-06 2023-07-02T14:03:06.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669084+0000", 2023-07-02T14:03:06.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034289910000000002 2023-07-02T14:03:06.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.672513+0000", 2023-07-02T14:03:06.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6016e-05 2023-07-02T14:03:06.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.672529+0000", 2023-07-02T14:03:06.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048754319999999999 2023-07-02T14:03:06.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.677405+0000", 2023-07-02T14:03:06.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0834000000000003e-05 2023-07-02T14:03:06.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.677436+0000", 2023-07-02T14:03:06.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.008388431 2023-07-02T14:03:06.079 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.079 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.079 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.080 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1196 2.5 e57/14 2:a79a9f1f:::rbd_data.125de1449605.0000000000000039:head v 57'69, mlcod=57'63)", 2023-07-02T14:03:06.080 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.669091+0000", 2023-07-02T14:03:06.080 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.29181080799999998, 2023-07-02T14:03:06.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0083840430000000007, 2023-07-02T14:03:06.080 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.080 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.081 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669091+0000", 2023-07-02T14:03:06.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669091+0000", 2023-07-02T14:03:06.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:06.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669090+0000", 2023-07-02T14:03:06.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037209230000000001 2023-07-02T14:03:06.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.672811+0000", 2023-07-02T14:03:06.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5011000000000001e-05 2023-07-02T14:03:06.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.672826+0000", 2023-07-02T14:03:06.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3039999999999999e-06 2023-07-02T14:03:06.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.672834+0000", 2023-07-02T14:03:06.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9644999999999997e-05 2023-07-02T14:03:06.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.672873+0000", 2023-07-02T14:03:06.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8957000000000002e-05 2023-07-02T14:03:06.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.672892+0000", 2023-07-02T14:03:06.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045453769999999998 2023-07-02T14:03:06.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.677438+0000", 2023-07-02T14:03:06.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147000000000002e-05 2023-07-02T14:03:06.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.677475+0000", 2023-07-02T14:03:06.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0083840430000000007 2023-07-02T14:03:06.087 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.087 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.087 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.087 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1217 2.5 e57/14 2:adf18789:::rbd_data.125de1449605.000000000000003c:head v 57'72, mlcod=57'70)", 2023-07-02T14:03:06.088 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.679558+0000", 2023-07-02T14:03:06.088 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.281343748, 2023-07-02T14:03:06.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065831600000000002, 2023-07-02T14:03:06.088 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.088 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.088 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.679558+0000", 2023-07-02T14:03:06.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.679558+0000", 2023-07-02T14:03:06.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:06.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.679557+0000", 2023-07-02T14:03:06.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030605350000000001 2023-07-02T14:03:06.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.682617+0000", 2023-07-02T14:03:06.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8680000000000002e-05 2023-07-02T14:03:06.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.682656+0000", 2023-07-02T14:03:06.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4589999999999998e-06 2023-07-02T14:03:06.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.682664+0000", 2023-07-02T14:03:06.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2956e-05 2023-07-02T14:03:06.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.682707+0000", 2023-07-02T14:03:06.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4885000000000002e-05 2023-07-02T14:03:06.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.682742+0000", 2023-07-02T14:03:06.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033564200000000001 2023-07-02T14:03:06.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.686098+0000", 2023-07-02T14:03:06.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2419999999999997e-05 2023-07-02T14:03:06.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.686141+0000", 2023-07-02T14:03:06.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065831600000000002 2023-07-02T14:03:06.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.095 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.095 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1218 2.5 e57/14 2:a79a9f1f:::rbd_data.125de1449605.0000000000000039:head v 57'73, mlcod=57'70)", 2023-07-02T14:03:06.095 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.682676+0000", 2023-07-02T14:03:06.095 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.27822519600000001, 2023-07-02T14:03:06.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064358790000000003, 2023-07-02T14:03:06.096 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.096 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.096 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.682676+0000", 2023-07-02T14:03:06.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.682676+0000", 2023-07-02T14:03:06.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:06.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.682675+0000", 2023-07-02T14:03:06.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016076420000000001 2023-07-02T14:03:06.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.684283+0000", 2023-07-02T14:03:06.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9286000000000001e-05 2023-07-02T14:03:06.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.684302+0000", 2023-07-02T14:03:06.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0291e-05 2023-07-02T14:03:06.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.684313+0000", 2023-07-02T14:03:06.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050923199999999998 2023-07-02T14:03:06.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.684822+0000", 2023-07-02T14:03:06.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8663e-05 2023-07-02T14:03:06.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.684850+0000", 2023-07-02T14:03:06.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004224139 2023-07-02T14:03:06.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.689075+0000", 2023-07-02T14:03:06.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7571000000000003e-05 2023-07-02T14:03:06.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.689112+0000", 2023-07-02T14:03:06.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064358790000000003 2023-07-02T14:03:06.102 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.102 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.103 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.103 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1230 2.5 e57/14 2:be65b5f1:::rbd_data.125de1449605.0000000000000043:head v 57'75, mlcod=57'73)", 2023-07-02T14:03:06.103 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.705587+0000", 2023-07-02T14:03:06.103 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.25531428299999998, 2023-07-02T14:03:06.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059983939999999998, 2023-07-02T14:03:06.104 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.104 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.104 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.705587+0000", 2023-07-02T14:03:06.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.705587+0000", 2023-07-02T14:03:06.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:06.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.705586+0000", 2023-07-02T14:03:06.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029122179999999998 2023-07-02T14:03:06.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.708498+0000", 2023-07-02T14:03:06.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2082999999999999e-05 2023-07-02T14:03:06.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.708510+0000", 2023-07-02T14:03:06.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7269999999999998e-06 2023-07-02T14:03:06.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.708515+0000", 2023-07-02T14:03:06.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8591999999999998e-05 2023-07-02T14:03:06.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.708564+0000", 2023-07-02T14:03:06.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8493e-05 2023-07-02T14:03:06.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.708592+0000", 2023-07-02T14:03:06.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029272510000000001 2023-07-02T14:03:06.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.711519+0000", 2023-07-02T14:03:06.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6088999999999997e-05 2023-07-02T14:03:06.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.711586+0000", 2023-07-02T14:03:06.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059983939999999998 2023-07-02T14:03:06.110 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.110 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.110 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.111 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1234 2.4 e57/34 2:3e72164b:::rbd_data.125de1449605.0000000000000044:head v 57'125, mlcod=57'123)", 2023-07-02T14:03:06.111 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.711100+0000", 2023-07-02T14:03:06.111 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.249801563, 2023-07-02T14:03:06.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061259950000000004, 2023-07-02T14:03:06.111 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.111 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.111 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.711100+0000", 2023-07-02T14:03:06.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.711100+0000", 2023-07-02T14:03:06.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:06.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.711098+0000", 2023-07-02T14:03:06.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026477919999999999 2023-07-02T14:03:06.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.713746+0000", 2023-07-02T14:03:06.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0956999999999999e-05 2023-07-02T14:03:06.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.713767+0000", 2023-07-02T14:03:06.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8929999999999994e-06 2023-07-02T14:03:06.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.713776+0000", 2023-07-02T14:03:06.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1447999999999999e-05 2023-07-02T14:03:06.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.713827+0000", 2023-07-02T14:03:06.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7498999999999999e-05 2023-07-02T14:03:06.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.713855+0000", 2023-07-02T14:03:06.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033264359999999999 2023-07-02T14:03:06.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.717181+0000", 2023-07-02T14:03:06.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4620000000000003e-05 2023-07-02T14:03:06.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.717226+0000", 2023-07-02T14:03:06.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061259950000000004 2023-07-02T14:03:06.118 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.118 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.118 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.118 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.118 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.118 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:06.120 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:03:06.120 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:06.119+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:06.221 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:06.221+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:06.284 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:06.285 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:06.285 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:06.285 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:06.295 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:03:06.338 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:06.337+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:06.423 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:06.422+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:06.433 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:06.434 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:06.434 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:06.434 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:06.434 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:06.444 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:03:06.527 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:06.526+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:06.610 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:06.610 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:03:06.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:03:06.611 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:06.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.611 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:279 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:06.611 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.310714+0000", 2023-07-02T14:03:06.611 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.297312205000001, 2023-07-02T14:03:06.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038119238, 2023-07-02T14:03:06.611 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.611 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.612 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:06.612 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:06.612 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:06.612 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 279 2023-07-02T14:03:06.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.612 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310714+0000", 2023-07-02T14:03:06.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310714+0000", 2023-07-02T14:03:06.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:06.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310713+0000", 2023-07-02T14:03:06.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8973e-05 2023-07-02T14:03:06.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310762+0000", 2023-07-02T14:03:06.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0530000000000007e-06 2023-07-02T14:03:06.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310770+0000", 2023-07-02T14:03:06.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7569999999999993e-06 2023-07-02T14:03:06.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310779+0000", 2023-07-02T14:03:06.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033655263999999997 2023-07-02T14:03:06.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.344434+0000", 2023-07-02T14:03:06.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4365999999999999e-05 2023-07-02T14:03:06.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.344508+0000", 2023-07-02T14:03:06.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016449100000000001 2023-07-02T14:03:06.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:06.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.344673+0000", 2023-07-02T14:03:06.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019402379999999999 2023-07-02T14:03:06.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:06.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346613+0000", 2023-07-02T14:03:06.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021203900000000002 2023-07-02T14:03:06.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:06.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348734+0000", 2023-07-02T14:03:06.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5225000000000001e-05 2023-07-02T14:03:06.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348789+0000", 2023-07-02T14:03:06.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4163999999999999e-05 2023-07-02T14:03:06.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348833+0000", 2023-07-02T14:03:06.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038119238 2023-07-02T14:03:06.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.621 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.621 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:280 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:06.622 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.310785+0000", 2023-07-02T14:03:06.622 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.297241331999999, 2023-07-02T14:03:06.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038149954999999999, 2023-07-02T14:03:06.622 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.622 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.622 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:06.622 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:06.622 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:06.623 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 280 2023-07-02T14:03:06.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.623 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310785+0000", 2023-07-02T14:03:06.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310785+0000", 2023-07-02T14:03:06.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:06.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310784+0000", 2023-07-02T14:03:06.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4364999999999999e-05 2023-07-02T14:03:06.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310838+0000", 2023-07-02T14:03:06.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4619999999999996e-06 2023-07-02T14:03:06.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310847+0000", 2023-07-02T14:03:06.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5129999999999998e-06 2023-07-02T14:03:06.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310851+0000", 2023-07-02T14:03:06.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034215981999999999 2023-07-02T14:03:06.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.345067+0000", 2023-07-02T14:03:06.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6117000000000002e-05 2023-07-02T14:03:06.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.345153+0000", 2023-07-02T14:03:06.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014067499999999999 2023-07-02T14:03:06.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:06.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.345294+0000", 2023-07-02T14:03:06.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013560810000000001 2023-07-02T14:03:06.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:06.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346650+0000", 2023-07-02T14:03:06.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002196239 2023-07-02T14:03:06.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:06.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348846+0000", 2023-07-02T14:03:06.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7621999999999999e-05 2023-07-02T14:03:06.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348894+0000", 2023-07-02T14:03:06.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0518000000000001e-05 2023-07-02T14:03:06.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348935+0000", 2023-07-02T14:03:06.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038149954999999999 2023-07-02T14:03:06.631 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.631 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.631 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.632 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:281 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:06.632 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.310857+0000", 2023-07-02T14:03:06.632 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.297169069999999, 2023-07-02T14:03:06.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039347695000000002, 2023-07-02T14:03:06.632 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.632 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.633 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:06.633 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:06.633 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:06.633 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:03:06.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.633 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310857+0000", 2023-07-02T14:03:06.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310857+0000", 2023-07-02T14:03:06.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:06.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310856+0000", 2023-07-02T14:03:06.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00081158100000000002 2023-07-02T14:03:06.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311668+0000", 2023-07-02T14:03:06.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.257e-05 2023-07-02T14:03:06.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311680+0000", 2023-07-02T14:03:06.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1110000000000002e-06 2023-07-02T14:03:06.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311688+0000", 2023-07-02T14:03:06.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034985653999999998 2023-07-02T14:03:06.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346673+0000", 2023-07-02T14:03:06.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7062999999999999e-05 2023-07-02T14:03:06.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346730+0000", 2023-07-02T14:03:06.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000151061 2023-07-02T14:03:06.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:06.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346881+0000", 2023-07-02T14:03:06.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018356119999999999 2023-07-02T14:03:06.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:06.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348717+0000", 2023-07-02T14:03:06.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001417385 2023-07-02T14:03:06.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:06.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350134+0000", 2023-07-02T14:03:06.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7036999999999997e-05 2023-07-02T14:03:06.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350181+0000", 2023-07-02T14:03:06.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3187999999999999e-05 2023-07-02T14:03:06.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350205+0000", 2023-07-02T14:03:06.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039347695000000002 2023-07-02T14:03:06.642 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.642 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.642 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.642 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:282 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:06.642 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311695+0000", 2023-07-02T14:03:06.642 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.296330575999999, 2023-07-02T14:03:06.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038558453999999999, 2023-07-02T14:03:06.643 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.643 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.643 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:06.643 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:06.643 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:06.643 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 282 2023-07-02T14:03:06.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.644 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311695+0000", 2023-07-02T14:03:06.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311695+0000", 2023-07-02T14:03:06.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:06.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311695+0000", 2023-07-02T14:03:06.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6679999999999999e-05 2023-07-02T14:03:06.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311751+0000", 2023-07-02T14:03:06.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9199999999999993e-06 2023-07-02T14:03:06.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311760+0000", 2023-07-02T14:03:06.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2480000000000004e-06 2023-07-02T14:03:06.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311765+0000", 2023-07-02T14:03:06.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035670150999999997 2023-07-02T14:03:06.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347436+0000", 2023-07-02T14:03:06.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7861000000000006e-05 2023-07-02T14:03:06.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347503+0000", 2023-07-02T14:03:06.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000217443 2023-07-02T14:03:06.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:06.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347721+0000", 2023-07-02T14:03:06.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023992449999999999 2023-07-02T14:03:06.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:06.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350120+0000", 2023-07-02T14:03:06.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2219000000000006e-05 2023-07-02T14:03:06.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:06.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350212+0000", 2023-07-02T14:03:06.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3430000000000001e-05 2023-07-02T14:03:06.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350236+0000", 2023-07-02T14:03:06.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8074999999999999e-05 2023-07-02T14:03:06.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350254+0000", 2023-07-02T14:03:06.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038558453999999999 2023-07-02T14:03:06.652 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.652 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.652 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.653 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:283 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:06.653 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311771+0000", 2023-07-02T14:03:06.653 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.296254704999999, 2023-07-02T14:03:06.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039285701999999999, 2023-07-02T14:03:06.653 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.653 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.653 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:06.653 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:06.654 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:06.654 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 283 2023-07-02T14:03:06.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.654 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311771+0000", 2023-07-02T14:03:06.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311771+0000", 2023-07-02T14:03:06.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:06.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311771+0000", 2023-07-02T14:03:06.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6775999999999999e-05 2023-07-02T14:03:06.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311837+0000", 2023-07-02T14:03:06.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4290000000000001e-06 2023-07-02T14:03:06.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311846+0000", 2023-07-02T14:03:06.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3800000000000002e-06 2023-07-02T14:03:06.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311851+0000", 2023-07-02T14:03:06.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036308287000000002 2023-07-02T14:03:06.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348160+0000", 2023-07-02T14:03:06.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6475000000000002e-05 2023-07-02T14:03:06.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348236+0000", 2023-07-02T14:03:06.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000147966 2023-07-02T14:03:06.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:06.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348384+0000", 2023-07-02T14:03:06.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019115569999999999 2023-07-02T14:03:06.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:06.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350296+0000", 2023-07-02T14:03:06.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00063248000000000004 2023-07-02T14:03:06.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:06.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350928+0000", 2023-07-02T14:03:06.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3355000000000003e-05 2023-07-02T14:03:06.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351021+0000", 2023-07-02T14:03:06.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5596000000000003e-05 2023-07-02T14:03:06.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351057+0000", 2023-07-02T14:03:06.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039285701999999999 2023-07-02T14:03:06.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.662 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.663 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.663 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:284 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:06.663 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311859+0000", 2023-07-02T14:03:06.663 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.296167407999999, 2023-07-02T14:03:06.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039967282999999999, 2023-07-02T14:03:06.663 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.664 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.664 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:06.664 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:06.664 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:06.664 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 284 2023-07-02T14:03:06.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.664 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311859+0000", 2023-07-02T14:03:06.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311859+0000", 2023-07-02T14:03:06.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:06.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311858+0000", 2023-07-02T14:03:06.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3902999999999997e-05 2023-07-02T14:03:06.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311892+0000", 2023-07-02T14:03:06.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0989999999999999e-06 2023-07-02T14:03:06.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311900+0000", 2023-07-02T14:03:06.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9030000000000003e-06 2023-07-02T14:03:06.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311905+0000", 2023-07-02T14:03:06.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037060899000000001 2023-07-02T14:03:06.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348966+0000", 2023-07-02T14:03:06.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3609000000000002e-05 2023-07-02T14:03:06.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349009+0000", 2023-07-02T14:03:06.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6831999999999998e-05 2023-07-02T14:03:06.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:06.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349096+0000", 2023-07-02T14:03:06.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001975077 2023-07-02T14:03:06.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:06.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351071+0000", 2023-07-02T14:03:06.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00066166900000000004 2023-07-02T14:03:06.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:06.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351733+0000", 2023-07-02T14:03:06.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3199999999999999e-05 2023-07-02T14:03:06.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351786+0000", 2023-07-02T14:03:06.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9823999999999997e-05 2023-07-02T14:03:06.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351826+0000", 2023-07-02T14:03:06.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039967282999999999 2023-07-02T14:03:06.673 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.673 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.673 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.673 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:285 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:06.674 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311910+0000", 2023-07-02T14:03:06.674 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.296116376000001, 2023-07-02T14:03:06.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040060302999999998, 2023-07-02T14:03:06.674 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.674 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.674 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:06.674 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:06.674 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:06.675 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 285 2023-07-02T14:03:06.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.675 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311910+0000", 2023-07-02T14:03:06.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311910+0000", 2023-07-02T14:03:06.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:06.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311909+0000", 2023-07-02T14:03:06.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1025999999999997e-05 2023-07-02T14:03:06.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311940+0000", 2023-07-02T14:03:06.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4769999999999997e-06 2023-07-02T14:03:06.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311948+0000", 2023-07-02T14:03:06.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5059999999999997e-06 2023-07-02T14:03:06.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311952+0000", 2023-07-02T14:03:06.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037480332999999998 2023-07-02T14:03:06.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349432+0000", 2023-07-02T14:03:06.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2923000000000005e-05 2023-07-02T14:03:06.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349515+0000", 2023-07-02T14:03:06.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015231700000000001 2023-07-02T14:03:06.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:06.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349668+0000", 2023-07-02T14:03:06.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002171849 2023-07-02T14:03:06.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:06.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351839+0000", 2023-07-02T14:03:06.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2052000000000003e-05 2023-07-02T14:03:06.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:06.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351882+0000", 2023-07-02T14:03:06.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7683999999999999e-05 2023-07-02T14:03:06.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351929+0000", 2023-07-02T14:03:06.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0648000000000001e-05 2023-07-02T14:03:06.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351970+0000", 2023-07-02T14:03:06.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040060302999999998 2023-07-02T14:03:06.683 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.683 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.683 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.684 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:286 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:06.684 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311957+0000", 2023-07-02T14:03:06.684 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.296068509000001, 2023-07-02T14:03:06.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042429263000000002, 2023-07-02T14:03:06.684 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.684 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.684 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:06.685 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:06.685 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:06.685 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 286 2023-07-02T14:03:06.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.685 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311957+0000", 2023-07-02T14:03:06.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311957+0000", 2023-07-02T14:03:06.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:06.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311957+0000", 2023-07-02T14:03:06.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7862000000000003e-05 2023-07-02T14:03:06.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312015+0000", 2023-07-02T14:03:06.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2770000000000005e-06 2023-07-02T14:03:06.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312023+0000", 2023-07-02T14:03:06.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0000000000000004e-06 2023-07-02T14:03:06.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312028+0000", 2023-07-02T14:03:06.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038291327999999999 2023-07-02T14:03:06.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350319+0000", 2023-07-02T14:03:06.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3686000000000001e-05 2023-07-02T14:03:06.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350373+0000", 2023-07-02T14:03:06.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012420000000000001 2023-07-02T14:03:06.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:06.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350497+0000", 2023-07-02T14:03:06.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014893669999999999 2023-07-02T14:03:06.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:06.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351986+0000", 2023-07-02T14:03:06.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022881580000000002 2023-07-02T14:03:06.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:06.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354275+0000", 2023-07-02T14:03:06.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7388e-05 2023-07-02T14:03:06.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354332+0000", 2023-07-02T14:03:06.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4688999999999998e-05 2023-07-02T14:03:06.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354387+0000", 2023-07-02T14:03:06.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042429263000000002 2023-07-02T14:03:06.693 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.694 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.694 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.694 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:287 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 589824~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:06.694 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312033+0000", 2023-07-02T14:03:06.694 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.295992891000001, 2023-07-02T14:03:06.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042515431999999999, 2023-07-02T14:03:06.695 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.695 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.695 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:06.695 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:06.695 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:06.695 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 287 2023-07-02T14:03:06.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.696 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312033+0000", 2023-07-02T14:03:06.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312033+0000", 2023-07-02T14:03:06.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:06.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312033+0000", 2023-07-02T14:03:06.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1540999999999999e-05 2023-07-02T14:03:06.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312094+0000", 2023-07-02T14:03:06.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.123e-06 2023-07-02T14:03:06.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312102+0000", 2023-07-02T14:03:06.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5480000000000003e-06 2023-07-02T14:03:06.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312107+0000", 2023-07-02T14:03:06.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039019356999999998 2023-07-02T14:03:06.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351126+0000", 2023-07-02T14:03:06.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100107 2023-07-02T14:03:06.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351226+0000", 2023-07-02T14:03:06.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177455 2023-07-02T14:03:06.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:06.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351404+0000", 2023-07-02T14:03:06.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030030819999999998 2023-07-02T14:03:06.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:06.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354407+0000", 2023-07-02T14:03:06.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5787000000000001e-05 2023-07-02T14:03:06.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:06.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354443+0000", 2023-07-02T14:03:06.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0627000000000001e-05 2023-07-02T14:03:06.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354503+0000", 2023-07-02T14:03:06.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5327000000000002e-05 2023-07-02T14:03:06.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354548+0000", 2023-07-02T14:03:06.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042515431999999999 2023-07-02T14:03:06.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.704 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.705 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:288 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:06.705 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312112+0000", 2023-07-02T14:03:06.705 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.295913938000002, 2023-07-02T14:03:06.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051367833000000002, 2023-07-02T14:03:06.705 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.705 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.705 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:06.705 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:06.706 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:06.706 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 288 2023-07-02T14:03:06.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.706 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312112+0000", 2023-07-02T14:03:06.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312112+0000", 2023-07-02T14:03:06.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:06.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312111+0000", 2023-07-02T14:03:06.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8513999999999998e-05 2023-07-02T14:03:06.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312150+0000", 2023-07-02T14:03:06.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6499999999999996e-06 2023-07-02T14:03:06.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312157+0000", 2023-07-02T14:03:06.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7799999999999997e-06 2023-07-02T14:03:06.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312163+0000", 2023-07-02T14:03:06.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039867382 2023-07-02T14:03:06.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352031+0000", 2023-07-02T14:03:06.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6370000000000002e-05 2023-07-02T14:03:06.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352107+0000", 2023-07-02T14:03:06.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000156021 2023-07-02T14:03:06.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:06.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352263+0000", 2023-07-02T14:03:06.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023521890000000002 2023-07-02T14:03:06.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:06.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354615+0000", 2023-07-02T14:03:06.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087542309999999995 2023-07-02T14:03:06.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:06.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363369+0000", 2023-07-02T14:03:06.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3217999999999998e-05 2023-07-02T14:03:06.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363423+0000", 2023-07-02T14:03:06.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7228999999999997e-05 2023-07-02T14:03:06.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363480+0000", 2023-07-02T14:03:06.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051367833000000002 2023-07-02T14:03:06.714 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.714 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.714 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.715 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:289 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:06.715 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:06.715 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.295857890000001, 2023-07-02T14:03:06.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051413158, 2023-07-02T14:03:06.715 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.715 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.716 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:06.716 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:06.716 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:06.716 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 289 2023-07-02T14:03:06.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.716 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:06.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:06.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:06.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:06.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1689e-05 2023-07-02T14:03:06.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312209+0000", 2023-07-02T14:03:06.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1070000000000005e-06 2023-07-02T14:03:06.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312217+0000", 2023-07-02T14:03:06.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.634e-06 2023-07-02T14:03:06.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312222+0000", 2023-07-02T14:03:06.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040512761000000001 2023-07-02T14:03:06.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352735+0000", 2023-07-02T14:03:06.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000119184 2023-07-02T14:03:06.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352854+0000", 2023-07-02T14:03:06.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021481899999999999 2023-07-02T14:03:06.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:06.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353069+0000", 2023-07-02T14:03:06.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001601707 2023-07-02T14:03:06.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:06.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354670+0000", 2023-07-02T14:03:06.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088168570000000009 2023-07-02T14:03:06.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:06.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363487+0000", 2023-07-02T14:03:06.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9635999999999997e-05 2023-07-02T14:03:06.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363537+0000", 2023-07-02T14:03:06.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4292999999999998e-05 2023-07-02T14:03:06.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363581+0000", 2023-07-02T14:03:06.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051413158 2023-07-02T14:03:06.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.725 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.725 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.725 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:290 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:06.725 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:06.725 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.295798645000001, 2023-07-02T14:03:06.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051455062000000003, 2023-07-02T14:03:06.726 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.726 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.726 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:06.726 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:06.726 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:06.726 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 290 2023-07-02T14:03:06.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.727 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:06.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:06.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:06.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:06.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6469000000000003e-05 2023-07-02T14:03:06.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312293+0000", 2023-07-02T14:03:06.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4630000000000007e-06 2023-07-02T14:03:06.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312303+0000", 2023-07-02T14:03:06.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8969999999999999e-06 2023-07-02T14:03:06.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312307+0000", 2023-07-02T14:03:06.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041261548000000002 2023-07-02T14:03:06.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353569+0000", 2023-07-02T14:03:06.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9594999999999999e-05 2023-07-02T14:03:06.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353659+0000", 2023-07-02T14:03:06.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018390600000000001 2023-07-02T14:03:06.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:06.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353843+0000", 2023-07-02T14:03:06.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035173100000000001 2023-07-02T14:03:06.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:06.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357360+0000", 2023-07-02T14:03:06.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062295670000000001 2023-07-02T14:03:06.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:06.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363589+0000", 2023-07-02T14:03:06.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9854000000000001e-05 2023-07-02T14:03:06.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363639+0000", 2023-07-02T14:03:06.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3133999999999997e-05 2023-07-02T14:03:06.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363682+0000", 2023-07-02T14:03:06.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051455062000000003 2023-07-02T14:03:06.735 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.735 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.735 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.735 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:291 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:06.736 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:06.736 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.295712605999999, 2023-07-02T14:03:06.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051527860000000002, 2023-07-02T14:03:06.736 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.736 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.736 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:06.736 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:06.736 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:06.737 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 291 2023-07-02T14:03:06.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.737 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:06.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:06.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:06.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:06.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2448999999999998e-05 2023-07-02T14:03:06.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312375+0000", 2023-07-02T14:03:06.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5259999999999993e-06 2023-07-02T14:03:06.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312384+0000", 2023-07-02T14:03:06.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0610000000000003e-06 2023-07-02T14:03:06.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312389+0000", 2023-07-02T14:03:06.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042326111 2023-07-02T14:03:06.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354715+0000", 2023-07-02T14:03:06.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3998999999999995e-05 2023-07-02T14:03:06.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354779+0000", 2023-07-02T14:03:06.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010269500000000001 2023-07-02T14:03:06.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:06.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354882+0000", 2023-07-02T14:03:06.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002533736 2023-07-02T14:03:06.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:06.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357415+0000", 2023-07-02T14:03:06.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062765700000000004 2023-07-02T14:03:06.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:06.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363692+0000", 2023-07-02T14:03:06.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7095000000000001e-05 2023-07-02T14:03:06.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363749+0000", 2023-07-02T14:03:06.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2263999999999996e-05 2023-07-02T14:03:06.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363841+0000", 2023-07-02T14:03:06.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051527860000000002 2023-07-02T14:03:06.745 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.745 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.745 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.746 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:292 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:06.746 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:03:06.746 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.295630358, 2023-07-02T14:03:06.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051556056000000003, 2023-07-02T14:03:06.746 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.746 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.747 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:06.747 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:06.747 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:06.747 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 292 2023-07-02T14:03:06.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.747 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:03:06.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:03:06.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:06.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312395+0000", 2023-07-02T14:03:06.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9624e-05 2023-07-02T14:03:06.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312425+0000", 2023-07-02T14:03:06.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:03:06.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312432+0000", 2023-07-02T14:03:06.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7550000000000002e-06 2023-07-02T14:03:06.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312437+0000", 2023-07-02T14:03:06.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.043823036000000003 2023-07-02T14:03:06.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356260+0000", 2023-07-02T14:03:06.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6021999999999997e-05 2023-07-02T14:03:06.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356346+0000", 2023-07-02T14:03:06.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020076600000000001 2023-07-02T14:03:06.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:06.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356547+0000", 2023-07-02T14:03:06.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020766410000000002 2023-07-02T14:03:06.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:06.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358623+0000", 2023-07-02T14:03:06.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005231192 2023-07-02T14:03:06.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:06.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363854+0000", 2023-07-02T14:03:06.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3423e-05 2023-07-02T14:03:06.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363908+0000", 2023-07-02T14:03:06.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3825999999999997e-05 2023-07-02T14:03:06.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363952+0000", 2023-07-02T14:03:06.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051556056000000003 2023-07-02T14:03:06.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.756 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.756 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:293 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:06.756 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:03:06.756 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.295584187999999, 2023-07-02T14:03:06.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051614769999999997, 2023-07-02T14:03:06.757 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.757 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.757 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:06.757 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:06.757 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:06.757 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 293 2023-07-02T14:03:06.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.758 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:03:06.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:03:06.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:06.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312441+0000", 2023-07-02T14:03:06.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9609999999999999e-05 2023-07-02T14:03:06.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312471+0000", 2023-07-02T14:03:06.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7069999999999992e-06 2023-07-02T14:03:06.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312478+0000", 2023-07-02T14:03:06.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1190000000000002e-06 2023-07-02T14:03:06.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312485+0000", 2023-07-02T14:03:06.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.044956387 2023-07-02T14:03:06.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357441+0000", 2023-07-02T14:03:06.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4134999999999994e-05 2023-07-02T14:03:06.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357515+0000", 2023-07-02T14:03:06.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000147408 2023-07-02T14:03:06.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:06.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357663+0000", 2023-07-02T14:03:06.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056472639999999999 2023-07-02T14:03:06.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:06.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363310+0000", 2023-07-02T14:03:06.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00065402600000000004 2023-07-02T14:03:06.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:06.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363964+0000", 2023-07-02T14:03:06.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0497000000000001e-05 2023-07-02T14:03:06.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364014+0000", 2023-07-02T14:03:06.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2243000000000002e-05 2023-07-02T14:03:06.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364057+0000", 2023-07-02T14:03:06.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051614769999999997 2023-07-02T14:03:06.766 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.766 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.766 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.766 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:294 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1048576~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:06.767 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:03:06.767 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.295536189, 2023-07-02T14:03:06.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051679527000000003, 2023-07-02T14:03:06.767 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.767 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.767 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:06.767 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:06.767 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:06.768 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 294 2023-07-02T14:03:06.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.768 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:03:06.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:03:06.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:06.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312489+0000", 2023-07-02T14:03:06.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8204999999999998e-05 2023-07-02T14:03:06.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312517+0000", 2023-07-02T14:03:06.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7759999999999997e-06 2023-07-02T14:03:06.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312525+0000", 2023-07-02T14:03:06.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8730000000000001e-06 2023-07-02T14:03:06.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312532+0000", 2023-07-02T14:03:06.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.045442206999999998 2023-07-02T14:03:06.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357974+0000", 2023-07-02T14:03:06.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4422000000000002e-05 2023-07-02T14:03:06.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358029+0000", 2023-07-02T14:03:06.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149519 2023-07-02T14:03:06.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:06.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358178+0000", 2023-07-02T14:03:06.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051779390000000003 2023-07-02T14:03:06.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:06.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363356+0000", 2023-07-02T14:03:06.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00070707699999999997 2023-07-02T14:03:06.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:06.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364063+0000", 2023-07-02T14:03:06.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6835999999999999e-05 2023-07-02T14:03:06.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364090+0000", 2023-07-02T14:03:06.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9263000000000003e-05 2023-07-02T14:03:06.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364169+0000", 2023-07-02T14:03:06.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051679527000000003 2023-07-02T14:03:06.776 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.776 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.776 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.777 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:423 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [write 2555904~1638400] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:06.777 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.330332+0000", 2023-07-02T14:03:06.777 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.277694193999999, 2023-07-02T14:03:06.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038020074000000001, 2023-07-02T14:03:06.777 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.777 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.777 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:06.778 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:06.778 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:06.778 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 423 2023-07-02T14:03:06.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.778 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.330332+0000", 2023-07-02T14:03:06.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.330332+0000", 2023-07-02T14:03:06.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:06.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.330330+0000", 2023-07-02T14:03:06.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060720390000000004 2023-07-02T14:03:06.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336402+0000", 2023-07-02T14:03:06.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8478000000000002e-05 2023-07-02T14:03:06.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336421+0000", 2023-07-02T14:03:06.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0624999999999999e-05 2023-07-02T14:03:06.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336451+0000", 2023-07-02T14:03:06.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022199402 2023-07-02T14:03:06.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358651+0000", 2023-07-02T14:03:06.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1981000000000001e-05 2023-07-02T14:03:06.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358723+0000", 2023-07-02T14:03:06.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163117 2023-07-02T14:03:06.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:06.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358886+0000", 2023-07-02T14:03:06.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005296762 2023-07-02T14:03:06.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:06.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364183+0000", 2023-07-02T14:03:06.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040861739999999997 2023-07-02T14:03:06.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:06.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368269+0000", 2023-07-02T14:03:06.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7169000000000003e-05 2023-07-02T14:03:06.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368306+0000", 2023-07-02T14:03:06.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5760999999999998e-05 2023-07-02T14:03:06.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368352+0000", 2023-07-02T14:03:06.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038020074000000001 2023-07-02T14:03:06.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:427 2.2 2:44c63967:::rbd_data.125de1449605.0000000000000101:head [write 2621440~1572864] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.336464+0000", 2023-07-02T14:03:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.271562366000001, 2023-07-02T14:03:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037760331000000001, 2023-07-02T14:03:06.788 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.788 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.788 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:06.788 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:06.788 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:06.788 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 427 2023-07-02T14:03:06.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.788 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336464+0000", 2023-07-02T14:03:06.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336464+0000", 2023-07-02T14:03:06.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:06.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336462+0000", 2023-07-02T14:03:06.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050934140000000001 2023-07-02T14:03:06.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.341555+0000", 2023-07-02T14:03:06.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6209e-05 2023-07-02T14:03:06.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.341572+0000", 2023-07-02T14:03:06.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.114e-06 2023-07-02T14:03:06.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.341579+0000", 2023-07-02T14:03:06.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019481099000000002 2023-07-02T14:03:06.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361060+0000", 2023-07-02T14:03:06.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5408000000000005e-05 2023-07-02T14:03:06.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361155+0000", 2023-07-02T14:03:06.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000186515 2023-07-02T14:03:06.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:06.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361342+0000", 2023-07-02T14:03:06.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003066465 2023-07-02T14:03:06.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:06.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364408+0000", 2023-07-02T14:03:06.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0096922329999999998 2023-07-02T14:03:06.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:06.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.374100+0000", 2023-07-02T14:03:06.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7960999999999995e-05 2023-07-02T14:03:06.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.374168+0000", 2023-07-02T14:03:06.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5458000000000003e-05 2023-07-02T14:03:06.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.374224+0000", 2023-07-02T14:03:06.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037760331000000001 2023-07-02T14:03:06.797 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.797 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.797 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.797 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:1066 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e57)", 2023-07-02T14:03:06.797 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.577749+0000", 2023-07-02T14:03:06.798 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.030277082, 2023-07-02T14:03:06.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038354245000000002, 2023-07-02T14:03:06.798 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.798 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.798 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:06.798 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:06.798 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:06.798 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1066 2023-07-02T14:03:06.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.799 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577749+0000", 2023-07-02T14:03:06.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577749+0000", 2023-07-02T14:03:06.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:06.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577748+0000", 2023-07-02T14:03:06.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2019999999999998e-06 2023-07-02T14:03:06.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577756+0000", 2023-07-02T14:03:06.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.528e-06 2023-07-02T14:03:06.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577762+0000", 2023-07-02T14:03:06.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4035999999999997e-05 2023-07-02T14:03:06.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577796+0000", 2023-07-02T14:03:06.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068442729999999997 2023-07-02T14:03:06.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.584640+0000", 2023-07-02T14:03:06.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2489000000000003e-05 2023-07-02T14:03:06.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:06.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.584723+0000", 2023-07-02T14:03:06.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0053785780000000002 2023-07-02T14:03:06.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.590101+0000", 2023-07-02T14:03:06.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0708999999999998e-05 2023-07-02T14:03:06.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:06.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.590162+0000", 2023-07-02T14:03:06.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047004220000000001 2023-07-02T14:03:06.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.594863+0000", 2023-07-02T14:03:06.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0859999999999997e-05 2023-07-02T14:03:06.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:06.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.594923+0000", 2023-07-02T14:03:06.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038761630000000002 2023-07-02T14:03:06.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.598800+0000", 2023-07-02T14:03:06.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106016 2023-07-02T14:03:06.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:06.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.598906+0000", 2023-07-02T14:03:06.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061340400000000003 2023-07-02T14:03:06.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.605040+0000", 2023-07-02T14:03:06.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2472000000000002e-05 2023-07-02T14:03:06.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:06.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.605092+0000", 2023-07-02T14:03:06.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002500927 2023-07-02T14:03:06.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.607593+0000", 2023-07-02T14:03:06.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2676000000000006e-05 2023-07-02T14:03:06.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:06.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.607676+0000", 2023-07-02T14:03:06.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041623739999999999 2023-07-02T14:03:06.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.611838+0000", 2023-07-02T14:03:06.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1777000000000002e-05 2023-07-02T14:03:06.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.611900+0000", 2023-07-02T14:03:06.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00035127200000000002 2023-07-02T14:03:06.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:06.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.612251+0000", 2023-07-02T14:03:06.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022406100000000001 2023-07-02T14:03:06.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:06.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.614492+0000", 2023-07-02T14:03:06.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001502139 2023-07-02T14:03:06.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:06.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.615994+0000", 2023-07-02T14:03:06.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5522000000000001e-05 2023-07-02T14:03:06.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.616049+0000", 2023-07-02T14:03:06.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3705000000000002e-05 2023-07-02T14:03:06.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.616103+0000", 2023-07-02T14:03:06.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038354245000000002 2023-07-02T14:03:06.815 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.815 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.815 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.815 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:1067 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e57)", 2023-07-02T14:03:06.815 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.577810+0000", 2023-07-02T14:03:06.816 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.0302154889999999, 2023-07-02T14:03:06.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040743256999999998, 2023-07-02T14:03:06.816 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:06.816 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:06.816 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:06.816 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:06.816 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:06.816 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1067 2023-07-02T14:03:06.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.817 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:06.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:06.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577810+0000", 2023-07-02T14:03:06.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:06.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:06.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577810+0000", 2023-07-02T14:03:06.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:06.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:06.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577810+0000", 2023-07-02T14:03:06.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7410000000000003e-06 2023-07-02T14:03:06.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:06.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577816+0000", 2023-07-02T14:03:06.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4979999999999995e-06 2023-07-02T14:03:06.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:06.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577826+0000", 2023-07-02T14:03:06.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5599999999999999e-06 2023-07-02T14:03:06.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:06.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577832+0000", 2023-07-02T14:03:06.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0069249979999999999 2023-07-02T14:03:06.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.584757+0000", 2023-07-02T14:03:06.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8571999999999993e-05 2023-07-02T14:03:06.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:06.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.584826+0000", 2023-07-02T14:03:06.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0053597239999999997 2023-07-02T14:03:06.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.590186+0000", 2023-07-02T14:03:06.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3909999999999999e-05 2023-07-02T14:03:06.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:06.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.590220+0000", 2023-07-02T14:03:06.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047859419999999996 2023-07-02T14:03:06.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.595006+0000", 2023-07-02T14:03:06.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5887000000000001e-05 2023-07-02T14:03:06.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:06.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.595071+0000", 2023-07-02T14:03:06.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038991120000000002 2023-07-02T14:03:06.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.598971+0000", 2023-07-02T14:03:06.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0792999999999999e-05 2023-07-02T14:03:06.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:06.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.599031+0000", 2023-07-02T14:03:06.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060912229999999998 2023-07-02T14:03:06.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.605123+0000", 2023-07-02T14:03:06.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3111e-05 2023-07-02T14:03:06.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:06.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.605176+0000", 2023-07-02T14:03:06.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002557372 2023-07-02T14:03:06.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.607733+0000", 2023-07-02T14:03:06.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013124799999999999 2023-07-02T14:03:06.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:06.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.607864+0000", 2023-07-02T14:03:06.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048035049999999996 2023-07-02T14:03:06.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.612668+0000", 2023-07-02T14:03:06.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0899e-05 2023-07-02T14:03:06.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:06.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.612739+0000", 2023-07-02T14:03:06.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033911969999999999 2023-07-02T14:03:06.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:06.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.616130+0000", 2023-07-02T14:03:06.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9387e-05 2023-07-02T14:03:06.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:06.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.616179+0000", 2023-07-02T14:03:06.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042055800000000002 2023-07-02T14:03:06.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:06.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.616600+0000", 2023-07-02T14:03:06.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001567188 2023-07-02T14:03:06.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:06.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.618167+0000", 2023-07-02T14:03:06.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030074700000000003 2023-07-02T14:03:06.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:06.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.618468+0000", 2023-07-02T14:03:06.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9024999999999999e-05 2023-07-02T14:03:06.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:06.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.618517+0000", 2023-07-02T14:03:06.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6959000000000003e-05 2023-07-02T14:03:06.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:06.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:06.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:06.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.618554+0000", 2023-07-02T14:03:06.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040743256999999998 2023-07-02T14:03:06.835 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.835 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.835 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.835 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:06.835 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:06.835 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:06.836 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:03:06.837 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:06.836+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:06.838 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:03:06.937 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:06.936+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:07.006 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:07.006 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:07.007 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:07.007 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:07.017 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:03:07.018 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:07.036 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:07.038 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:07.037+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:07.054 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:07.073 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:07.092 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:07.110 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:07.111 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:03:07.112 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:03:07.112 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:03:07.112 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:03:07.112 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:03:07.112 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:03:07.140 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:07.139+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:07.143 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:07.143 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:07.153 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:03:07.240 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:07.239+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:07.318 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:07.319 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:07.329 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:03:07.370 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:07.369+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:07.385 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:03:07.441 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:07.440+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:07.466 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:07.466 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:07.476 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:03:07.561 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:07.560+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:07.641 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:07.642 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:07.642 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:07.641+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:07.651 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:03:07.742 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:07.741+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:07.815 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:07.815 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:07.825 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:03:07.869 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:07.868+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:07.944 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:07.943+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:07.949 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:07.949 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:07.959 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:03:08.045 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:08.045+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:08.124 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:08.124 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:08.134 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:03:08.146 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:08.145+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:08.246 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:08.245+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:08.249 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2023-07-02T14:03:08.249 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:03:08.249 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5] 2023-07-02T14:03:08.249 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2023-07-02T14:03:08.258 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:08.258 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:08.268 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:03:08.320 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:03:08.347 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:08.347+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:08.428 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:08.428 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:08.445 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:03:08.486 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:08.485+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:08.550 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:08.548+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:08.650 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:08.650 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:08.651 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:08.649+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:08.662 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:03:08.751 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:08.750+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:08.772 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-07-02T14:03:08.770+0000 7ff490b05700 -1 mon.a@0(leader).osd e58 definitely_dead 0 2023-07-02T14:03:08.811 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:08.811 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:08.825 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:03:08.871 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:08.869+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:08.954 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:08.953+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:09.052 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:09.052 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:09.055 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:09.053+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:09.073 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:03:09.123 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:03:09.158 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:09.157+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:09.187 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap3 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap4 b3fbb028-e267-4d25-a750-93ba18a2316a@snap4.diff 2023-07-02T14:03:09.256 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:09.255+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:09.329 INFO:teuthology.orchestra.run.smithi016.stderr:marked out osd.5. 2023-07-02T14:03:09.340 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:09.340 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:09.356 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:09.356+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:09.359 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:03:09.426 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:09.447 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:09.457 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:09.456+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:09.481 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:09.514 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:09.532 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:09.550 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:09.552 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:03:09.552 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:03:09.552 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:03:09.552 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:03:09.552 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:03:09.552 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:03:09.557 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:09.557+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:09.561 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:09.561 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:03:09.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:03:09.561 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:09.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.562 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:426 2.4 e40/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 40'42, mlcod=40'38)", 2023-07-02T14:03:09.562 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.340864+0000", 2023-07-02T14:03:09.562 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.218116059, 2023-07-02T14:03:09.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065109469999999996, 2023-07-02T14:03:09.562 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.562 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:09.562 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.340864+0000", 2023-07-02T14:03:09.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.340864+0000", 2023-07-02T14:03:09.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:09.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.340863+0000", 2023-07-02T14:03:09.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019354159999999999 2023-07-02T14:03:09.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342799+0000", 2023-07-02T14:03:09.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5824e-05 2023-07-02T14:03:09.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342814+0000", 2023-07-02T14:03:09.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:03:09.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342822+0000", 2023-07-02T14:03:09.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7465999999999994e-05 2023-07-02T14:03:09.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342889+0000", 2023-07-02T14:03:09.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5068000000000001e-05 2023-07-02T14:03:09.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:09.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.342914+0000", 2023-07-02T14:03:09.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044200969999999996 2023-07-02T14:03:09.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:09.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347334+0000", 2023-07-02T14:03:09.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0711999999999999e-05 2023-07-02T14:03:09.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:09.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347375+0000", 2023-07-02T14:03:09.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065109469999999996 2023-07-02T14:03:09.569 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.569 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.569 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.570 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:431 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'4, mlcod=40'2)", 2023-07-02T14:03:09.570 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.350267+0000", 2023-07-02T14:03:09.570 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.208712593000001, 2023-07-02T14:03:09.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077388420000000001, 2023-07-02T14:03:09.570 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.570 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:09.570 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350267+0000", 2023-07-02T14:03:09.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350267+0000", 2023-07-02T14:03:09.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:09.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350266+0000", 2023-07-02T14:03:09.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024191569999999999 2023-07-02T14:03:09.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352685+0000", 2023-07-02T14:03:09.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0013e-05 2023-07-02T14:03:09.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352706+0000", 2023-07-02T14:03:09.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3250000000000008e-06 2023-07-02T14:03:09.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352714+0000", 2023-07-02T14:03:09.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9648000000000003e-05 2023-07-02T14:03:09.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352763+0000", 2023-07-02T14:03:09.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4496999999999999e-05 2023-07-02T14:03:09.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:09.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352788+0000", 2023-07-02T14:03:09.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051724880000000003 2023-07-02T14:03:09.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:09.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357960+0000", 2023-07-02T14:03:09.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5802e-05 2023-07-02T14:03:09.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:09.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358006+0000", 2023-07-02T14:03:09.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077388420000000001 2023-07-02T14:03:09.577 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.577 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.577 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.577 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:455 2.5 e40/14 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:head v 40'12, mlcod=40'10)", 2023-07-02T14:03:09.577 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.376227+0000", 2023-07-02T14:03:09.578 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.182753314999999, 2023-07-02T14:03:09.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012681951, 2023-07-02T14:03:09.578 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.578 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:09.578 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.376227+0000", 2023-07-02T14:03:09.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.376227+0000", 2023-07-02T14:03:09.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:09.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.376226+0000", 2023-07-02T14:03:09.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040198860000000003 2023-07-02T14:03:09.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380245+0000", 2023-07-02T14:03:09.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7640000000000001e-05 2023-07-02T14:03:09.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380263+0000", 2023-07-02T14:03:09.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3299999999999999e-06 2023-07-02T14:03:09.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380271+0000", 2023-07-02T14:03:09.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4322e-05 2023-07-02T14:03:09.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380316+0000", 2023-07-02T14:03:09.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7682000000000001e-05 2023-07-02T14:03:09.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:09.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.380333+0000", 2023-07-02T14:03:09.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085223439999999994 2023-07-02T14:03:09.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:09.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.388856+0000", 2023-07-02T14:03:09.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2930999999999999e-05 2023-07-02T14:03:09.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:09.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.388909+0000", 2023-07-02T14:03:09.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012681951 2023-07-02T14:03:09.584 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.585 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.585 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.585 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:456 2.5 e40/14 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:head v 40'13, mlcod=40'10)", 2023-07-02T14:03:09.585 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.382266+0000", 2023-07-02T14:03:09.585 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.176713688, 2023-07-02T14:03:09.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075568149999999997, 2023-07-02T14:03:09.586 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.586 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:09.586 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382266+0000", 2023-07-02T14:03:09.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382266+0000", 2023-07-02T14:03:09.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:09.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382264+0000", 2023-07-02T14:03:09.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012720599999999999 2023-07-02T14:03:09.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382391+0000", 2023-07-02T14:03:09.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7598e-05 2023-07-02T14:03:09.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382409+0000", 2023-07-02T14:03:09.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8660000000000001e-06 2023-07-02T14:03:09.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382419+0000", 2023-07-02T14:03:09.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001837403 2023-07-02T14:03:09.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384256+0000", 2023-07-02T14:03:09.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5757000000000001e-05 2023-07-02T14:03:09.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:09.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384282+0000", 2023-07-02T14:03:09.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055044730000000002 2023-07-02T14:03:09.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:09.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389787+0000", 2023-07-02T14:03:09.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6622000000000003e-05 2023-07-02T14:03:09.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:09.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389823+0000", 2023-07-02T14:03:09.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075568149999999997 2023-07-02T14:03:09.592 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.592 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.593 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.593 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:457 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'14, mlcod=40'10)", 2023-07-02T14:03:09.593 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.382901+0000", 2023-07-02T14:03:09.593 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.176078715999999, 2023-07-02T14:03:09.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0069428320000000003, 2023-07-02T14:03:09.593 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.594 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:09.594 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382901+0000", 2023-07-02T14:03:09.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382901+0000", 2023-07-02T14:03:09.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:09.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382900+0000", 2023-07-02T14:03:09.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7738999999999994e-05 2023-07-02T14:03:09.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.382998+0000", 2023-07-02T14:03:09.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6769000000000002e-05 2023-07-02T14:03:09.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383014+0000", 2023-07-02T14:03:09.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0073e-05 2023-07-02T14:03:09.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383025+0000", 2023-07-02T14:03:09.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016687410000000001 2023-07-02T14:03:09.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384693+0000", 2023-07-02T14:03:09.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4108999999999999e-05 2023-07-02T14:03:09.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:09.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.384717+0000", 2023-07-02T14:03:09.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051086830000000001 2023-07-02T14:03:09.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:09.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389826+0000", 2023-07-02T14:03:09.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8059000000000001e-05 2023-07-02T14:03:09.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:09.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389844+0000", 2023-07-02T14:03:09.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0069428320000000003 2023-07-02T14:03:09.600 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.600 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.600 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.601 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:479 2.5 e40/14 2:bbaed619:::rbd_data.125de1449605.0000000000000109:head v 40'15, mlcod=40'10)", 2023-07-02T14:03:09.601 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.383348+0000", 2023-07-02T14:03:09.601 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.175631758000002, 2023-07-02T14:03:09.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065068890000000001, 2023-07-02T14:03:09.601 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.601 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:09.601 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383348+0000", 2023-07-02T14:03:09.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383348+0000", 2023-07-02T14:03:09.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:09.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383347+0000", 2023-07-02T14:03:09.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000422123 2023-07-02T14:03:09.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383769+0000", 2023-07-02T14:03:09.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1688e-05 2023-07-02T14:03:09.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383791+0000", 2023-07-02T14:03:09.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0237e-05 2023-07-02T14:03:09.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.383801+0000", 2023-07-02T14:03:09.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013799940000000001 2023-07-02T14:03:09.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.385181+0000", 2023-07-02T14:03:09.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3047000000000001e-05 2023-07-02T14:03:09.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:09.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.385204+0000", 2023-07-02T14:03:09.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046417339999999998 2023-07-02T14:03:09.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:09.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389846+0000", 2023-07-02T14:03:09.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1800000000000002e-06 2023-07-02T14:03:09.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:09.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.389855+0000", 2023-07-02T14:03:09.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065068890000000001 2023-07-02T14:03:09.608 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.608 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.608 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.608 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1031 2.4 e57/34 2:2f8e3c69:::rbd_data.125de1449605.0000000000000028:head v 57'88, mlcod=57'86)", 2023-07-02T14:03:09.609 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.551534+0000", 2023-07-02T14:03:09.609 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.0074459999999998, 2023-07-02T14:03:09.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085593449999999995, 2023-07-02T14:03:09.609 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.609 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:09.609 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.551534+0000", 2023-07-02T14:03:09.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.551534+0000", 2023-07-02T14:03:09.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.551533+0000", 2023-07-02T14:03:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003576339 2023-07-02T14:03:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.555109+0000", 2023-07-02T14:03:09.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9423e-05 2023-07-02T14:03:09.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.555129+0000", 2023-07-02T14:03:09.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2002e-05 2023-07-02T14:03:09.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.555141+0000", 2023-07-02T14:03:09.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0914000000000001e-05 2023-07-02T14:03:09.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.555192+0000", 2023-07-02T14:03:09.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8541e-05 2023-07-02T14:03:09.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:09.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.555220+0000", 2023-07-02T14:03:09.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048274249999999998 2023-07-02T14:03:09.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:09.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.560048+0000", 2023-07-02T14:03:09.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5726999999999998e-05 2023-07-02T14:03:09.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:09.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.560093+0000", 2023-07-02T14:03:09.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085593449999999995 2023-07-02T14:03:09.616 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.616 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.616 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.616 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1097 2.5 e57/14 2:a5de3e1e:::rbd_data.125de1449605.0000000000000022:head v 57'53, mlcod=57'47)", 2023-07-02T14:03:09.616 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.594133+0000", 2023-07-02T14:03:09.617 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.9648473919999998, 2023-07-02T14:03:09.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056946139999999998, 2023-07-02T14:03:09.617 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.617 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:09.617 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.594133+0000", 2023-07-02T14:03:09.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.594133+0000", 2023-07-02T14:03:09.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:09.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.594132+0000", 2023-07-02T14:03:09.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022406090000000002 2023-07-02T14:03:09.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.596373+0000", 2023-07-02T14:03:09.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0052e-05 2023-07-02T14:03:09.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.596383+0000", 2023-07-02T14:03:09.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7830000000000004e-06 2023-07-02T14:03:09.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.596388+0000", 2023-07-02T14:03:09.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045496300000000001 2023-07-02T14:03:09.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.596843+0000", 2023-07-02T14:03:09.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0905999999999999e-05 2023-07-02T14:03:09.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:09.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.596864+0000", 2023-07-02T14:03:09.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029289530000000002 2023-07-02T14:03:09.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:09.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.599793+0000", 2023-07-02T14:03:09.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4078999999999997e-05 2023-07-02T14:03:09.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:09.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.599827+0000", 2023-07-02T14:03:09.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056946139999999998 2023-07-02T14:03:09.623 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.624 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.624 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.624 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1103 2.4 e57/34 2:3e650bab:::rbd_data.125de1449605.0000000000000031:head v 57'101, mlcod=57'97)", 2023-07-02T14:03:09.624 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.599608+0000", 2023-07-02T14:03:09.624 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.959371999, 2023-07-02T14:03:09.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0058272369999999999, 2023-07-02T14:03:09.625 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.625 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:09.625 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.599608+0000", 2023-07-02T14:03:09.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.599608+0000", 2023-07-02T14:03:09.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:09.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.599607+0000", 2023-07-02T14:03:09.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014553140000000001 2023-07-02T14:03:09.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.601062+0000", 2023-07-02T14:03:09.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1060000000000006e-06 2023-07-02T14:03:09.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.601071+0000", 2023-07-02T14:03:09.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6079999999999998e-06 2023-07-02T14:03:09.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.601076+0000", 2023-07-02T14:03:09.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3705000000000003e-05 2023-07-02T14:03:09.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.601120+0000", 2023-07-02T14:03:09.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9527000000000001e-05 2023-07-02T14:03:09.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:09.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.601139+0000", 2023-07-02T14:03:09.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042608289999999998 2023-07-02T14:03:09.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:09.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.605400+0000", 2023-07-02T14:03:09.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5345000000000001e-05 2023-07-02T14:03:09.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:09.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.605435+0000", 2023-07-02T14:03:09.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0058272369999999999 2023-07-02T14:03:09.631 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.631 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.631 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.632 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1134 2.4 e57/34 2:2211c235:::rbd_data.125de1449605.0000000000000034:head v 57'108, mlcod=57'105)", 2023-07-02T14:03:09.632 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.614673+0000", 2023-07-02T14:03:09.632 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.9443071120000002, 2023-07-02T14:03:09.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068419489999999999, 2023-07-02T14:03:09.632 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.632 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:09.633 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.614673+0000", 2023-07-02T14:03:09.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.614673+0000", 2023-07-02T14:03:09.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:09.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.614672+0000", 2023-07-02T14:03:09.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014647250000000001 2023-07-02T14:03:09.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.616137+0000", 2023-07-02T14:03:09.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3174e-05 2023-07-02T14:03:09.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.616150+0000", 2023-07-02T14:03:09.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6930000000000007e-06 2023-07-02T14:03:09.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.616158+0000", 2023-07-02T14:03:09.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4897000000000001e-05 2023-07-02T14:03:09.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.616202+0000", 2023-07-02T14:03:09.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1438e-05 2023-07-02T14:03:09.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:09.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.616224+0000", 2023-07-02T14:03:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052322990000000002 2023-07-02T14:03:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.621456+0000", 2023-07-02T14:03:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8683e-05 2023-07-02T14:03:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:09.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.621515+0000", 2023-07-02T14:03:09.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068419489999999999 2023-07-02T14:03:09.639 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.639 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.639 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.640 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1187 2.5 e57/14 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:head v 57'65, mlcod=57'63)", 2023-07-02T14:03:09.640 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.654963+0000", 2023-07-02T14:03:09.640 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.904016597, 2023-07-02T14:03:09.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012171026, 2023-07-02T14:03:09.640 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.640 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:09.640 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.654963+0000", 2023-07-02T14:03:09.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.654963+0000", 2023-07-02T14:03:09.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:09.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.654963+0000", 2023-07-02T14:03:09.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064185329999999997 2023-07-02T14:03:09.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661381+0000", 2023-07-02T14:03:09.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3459e-05 2023-07-02T14:03:09.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661405+0000", 2023-07-02T14:03:09.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0638e-05 2023-07-02T14:03:09.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661416+0000", 2023-07-02T14:03:09.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3569000000000001e-05 2023-07-02T14:03:09.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661479+0000", 2023-07-02T14:03:09.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4239000000000001e-05 2023-07-02T14:03:09.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:09.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661523+0000", 2023-07-02T14:03:09.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055722319999999999 2023-07-02T14:03:09.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:09.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.667096+0000", 2023-07-02T14:03:09.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8906999999999999e-05 2023-07-02T14:03:09.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:09.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.667134+0000", 2023-07-02T14:03:09.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012171026 2023-07-02T14:03:09.647 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.647 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.647 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.647 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1188 2.5 e57/14 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:head v 57'66, mlcod=57'63)", 2023-07-02T14:03:09.648 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.661435+0000", 2023-07-02T14:03:09.648 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.8975450679999999, 2023-07-02T14:03:09.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061507050000000002, 2023-07-02T14:03:09.648 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.648 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:09.648 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661435+0000", 2023-07-02T14:03:09.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661435+0000", 2023-07-02T14:03:09.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:09.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661434+0000", 2023-07-02T14:03:09.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8148e-05 2023-07-02T14:03:09.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661482+0000", 2023-07-02T14:03:09.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0789e-05 2023-07-02T14:03:09.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661492+0000", 2023-07-02T14:03:09.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8880000000000002e-06 2023-07-02T14:03:09.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661498+0000", 2023-07-02T14:03:09.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034011409999999999 2023-07-02T14:03:09.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.664900+0000", 2023-07-02T14:03:09.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4694000000000001e-05 2023-07-02T14:03:09.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:09.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.664924+0000", 2023-07-02T14:03:09.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026326829999999998 2023-07-02T14:03:09.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:09.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.667557+0000", 2023-07-02T14:03:09.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8770000000000001e-05 2023-07-02T14:03:09.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:09.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.667586+0000", 2023-07-02T14:03:09.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061507050000000002 2023-07-02T14:03:09.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.655 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.655 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1189 2.5 e57/14 2:adf18789:::rbd_data.125de1449605.000000000000003c:head v 57'67, mlcod=57'63)", 2023-07-02T14:03:09.655 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.661508+0000", 2023-07-02T14:03:09.655 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.8974721300000001, 2023-07-02T14:03:09.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015893727, 2023-07-02T14:03:09.656 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.656 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:09.656 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661508+0000", 2023-07-02T14:03:09.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661508+0000", 2023-07-02T14:03:09.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:09.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.661507+0000", 2023-07-02T14:03:09.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074929860000000001 2023-07-02T14:03:09.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669000+0000", 2023-07-02T14:03:09.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5695999999999999e-05 2023-07-02T14:03:09.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669026+0000", 2023-07-02T14:03:09.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6399999999999997e-06 2023-07-02T14:03:09.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669034+0000", 2023-07-02T14:03:09.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1940000000000001e-05 2023-07-02T14:03:09.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669086+0000", 2023-07-02T14:03:09.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5514999999999999e-05 2023-07-02T14:03:09.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:09.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669111+0000", 2023-07-02T14:03:09.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082441170000000005 2023-07-02T14:03:09.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:09.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.677355+0000", 2023-07-02T14:03:09.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6285000000000002e-05 2023-07-02T14:03:09.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:09.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.677402+0000", 2023-07-02T14:03:09.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015893727 2023-07-02T14:03:09.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.662 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.663 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.663 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1190 2.5 e57/14 2:adf18789:::rbd_data.125de1449605.000000000000003c:head v 57'68, mlcod=57'63)", 2023-07-02T14:03:09.663 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.669047+0000", 2023-07-02T14:03:09.663 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.8899328049999999, 2023-07-02T14:03:09.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.008388431, 2023-07-02T14:03:09.663 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.664 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:09.664 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669047+0000", 2023-07-02T14:03:09.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669047+0000", 2023-07-02T14:03:09.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:09.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669046+0000", 2023-07-02T14:03:09.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8430000000000001e-05 2023-07-02T14:03:09.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669074+0000", 2023-07-02T14:03:09.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9349999999999997e-06 2023-07-02T14:03:09.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669081+0000", 2023-07-02T14:03:09.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0469999999999998e-06 2023-07-02T14:03:09.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669084+0000", 2023-07-02T14:03:09.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034289910000000002 2023-07-02T14:03:09.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.672513+0000", 2023-07-02T14:03:09.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6016e-05 2023-07-02T14:03:09.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:09.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.672529+0000", 2023-07-02T14:03:09.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048754319999999999 2023-07-02T14:03:09.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:09.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.677405+0000", 2023-07-02T14:03:09.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0834000000000003e-05 2023-07-02T14:03:09.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:09.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.677436+0000", 2023-07-02T14:03:09.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.008388431 2023-07-02T14:03:09.670 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.670 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.670 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.671 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1196 2.5 e57/14 2:a79a9f1f:::rbd_data.125de1449605.0000000000000039:head v 57'69, mlcod=57'63)", 2023-07-02T14:03:09.671 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.669091+0000", 2023-07-02T14:03:09.671 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.8898893700000001, 2023-07-02T14:03:09.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0083840430000000007, 2023-07-02T14:03:09.671 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.671 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:09.671 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669091+0000", 2023-07-02T14:03:09.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669091+0000", 2023-07-02T14:03:09.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:09.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.669090+0000", 2023-07-02T14:03:09.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037209230000000001 2023-07-02T14:03:09.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.672811+0000", 2023-07-02T14:03:09.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5011000000000001e-05 2023-07-02T14:03:09.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.672826+0000", 2023-07-02T14:03:09.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3039999999999999e-06 2023-07-02T14:03:09.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.672834+0000", 2023-07-02T14:03:09.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9644999999999997e-05 2023-07-02T14:03:09.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.672873+0000", 2023-07-02T14:03:09.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8957000000000002e-05 2023-07-02T14:03:09.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:09.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.672892+0000", 2023-07-02T14:03:09.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045453769999999998 2023-07-02T14:03:09.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:09.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.677438+0000", 2023-07-02T14:03:09.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147000000000002e-05 2023-07-02T14:03:09.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:09.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.677475+0000", 2023-07-02T14:03:09.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0083840430000000007 2023-07-02T14:03:09.678 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.678 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.678 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.679 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1217 2.5 e57/14 2:adf18789:::rbd_data.125de1449605.000000000000003c:head v 57'72, mlcod=57'70)", 2023-07-02T14:03:09.679 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.679558+0000", 2023-07-02T14:03:09.679 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.8794223099999998, 2023-07-02T14:03:09.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065831600000000002, 2023-07-02T14:03:09.679 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.679 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:09.679 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.679558+0000", 2023-07-02T14:03:09.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.679558+0000", 2023-07-02T14:03:09.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:09.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.679557+0000", 2023-07-02T14:03:09.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030605350000000001 2023-07-02T14:03:09.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.682617+0000", 2023-07-02T14:03:09.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8680000000000002e-05 2023-07-02T14:03:09.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.682656+0000", 2023-07-02T14:03:09.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4589999999999998e-06 2023-07-02T14:03:09.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.682664+0000", 2023-07-02T14:03:09.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2956e-05 2023-07-02T14:03:09.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.682707+0000", 2023-07-02T14:03:09.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4885000000000002e-05 2023-07-02T14:03:09.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:09.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.682742+0000", 2023-07-02T14:03:09.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033564200000000001 2023-07-02T14:03:09.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:09.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.686098+0000", 2023-07-02T14:03:09.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2419999999999997e-05 2023-07-02T14:03:09.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:09.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.686141+0000", 2023-07-02T14:03:09.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065831600000000002 2023-07-02T14:03:09.686 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.686 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.686 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.686 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1218 2.5 e57/14 2:a79a9f1f:::rbd_data.125de1449605.0000000000000039:head v 57'73, mlcod=57'70)", 2023-07-02T14:03:09.686 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.682676+0000", 2023-07-02T14:03:09.687 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.8763037580000002, 2023-07-02T14:03:09.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064358790000000003, 2023-07-02T14:03:09.687 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.687 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:09.687 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.682676+0000", 2023-07-02T14:03:09.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.682676+0000", 2023-07-02T14:03:09.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:09.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.682675+0000", 2023-07-02T14:03:09.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016076420000000001 2023-07-02T14:03:09.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.684283+0000", 2023-07-02T14:03:09.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9286000000000001e-05 2023-07-02T14:03:09.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.684302+0000", 2023-07-02T14:03:09.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0291e-05 2023-07-02T14:03:09.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.684313+0000", 2023-07-02T14:03:09.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050923199999999998 2023-07-02T14:03:09.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.684822+0000", 2023-07-02T14:03:09.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8663e-05 2023-07-02T14:03:09.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:09.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.684850+0000", 2023-07-02T14:03:09.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004224139 2023-07-02T14:03:09.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:09.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.689075+0000", 2023-07-02T14:03:09.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7571000000000003e-05 2023-07-02T14:03:09.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:09.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.689112+0000", 2023-07-02T14:03:09.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064358790000000003 2023-07-02T14:03:09.693 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.694 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.694 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.694 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1230 2.5 e57/14 2:be65b5f1:::rbd_data.125de1449605.0000000000000043:head v 57'75, mlcod=57'73)", 2023-07-02T14:03:09.694 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.705587+0000", 2023-07-02T14:03:09.694 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.8533928450000001, 2023-07-02T14:03:09.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059983939999999998, 2023-07-02T14:03:09.695 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.695 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:09.695 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.705587+0000", 2023-07-02T14:03:09.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.705587+0000", 2023-07-02T14:03:09.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:09.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.705586+0000", 2023-07-02T14:03:09.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029122179999999998 2023-07-02T14:03:09.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.708498+0000", 2023-07-02T14:03:09.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2082999999999999e-05 2023-07-02T14:03:09.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.708510+0000", 2023-07-02T14:03:09.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7269999999999998e-06 2023-07-02T14:03:09.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.708515+0000", 2023-07-02T14:03:09.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8591999999999998e-05 2023-07-02T14:03:09.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.708564+0000", 2023-07-02T14:03:09.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8493e-05 2023-07-02T14:03:09.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:09.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.708592+0000", 2023-07-02T14:03:09.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029272510000000001 2023-07-02T14:03:09.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:09.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.711519+0000", 2023-07-02T14:03:09.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6088999999999997e-05 2023-07-02T14:03:09.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:09.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.711586+0000", 2023-07-02T14:03:09.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059983939999999998 2023-07-02T14:03:09.701 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.701 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.701 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.702 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1234 2.4 e57/34 2:3e72164b:::rbd_data.125de1449605.0000000000000044:head v 57'125, mlcod=57'123)", 2023-07-02T14:03:09.702 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.711100+0000", 2023-07-02T14:03:09.702 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.8478801250000001, 2023-07-02T14:03:09.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061259950000000004, 2023-07-02T14:03:09.702 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.702 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:09.703 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.711100+0000", 2023-07-02T14:03:09.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.711100+0000", 2023-07-02T14:03:09.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:09.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.711098+0000", 2023-07-02T14:03:09.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026477919999999999 2023-07-02T14:03:09.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.713746+0000", 2023-07-02T14:03:09.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0956999999999999e-05 2023-07-02T14:03:09.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.713767+0000", 2023-07-02T14:03:09.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8929999999999994e-06 2023-07-02T14:03:09.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.713776+0000", 2023-07-02T14:03:09.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1447999999999999e-05 2023-07-02T14:03:09.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.713827+0000", 2023-07-02T14:03:09.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7498999999999999e-05 2023-07-02T14:03:09.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:09.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.713855+0000", 2023-07-02T14:03:09.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033264359999999999 2023-07-02T14:03:09.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:09.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.717181+0000", 2023-07-02T14:03:09.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4620000000000003e-05 2023-07-02T14:03:09.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:09.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.717226+0000", 2023-07-02T14:03:09.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061259950000000004 2023-07-02T14:03:09.709 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.709 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.709 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.710 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:1272 2.4 e57/34 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head v 57'128, mlcod=57'124)", 2023-07-02T14:03:09.710 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:06.415871+0000", 2023-07-02T14:03:09.710 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.1431092309999999, 2023-07-02T14:03:09.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0094971880000000002, 2023-07-02T14:03:09.710 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.710 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:09.710 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:06.415871+0000", 2023-07-02T14:03:09.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:06.415871+0000", 2023-07-02T14:03:09.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:09.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:06.415869+0000", 2023-07-02T14:03:09.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047800580000000002 2023-07-02T14:03:09.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:06.420649+0000", 2023-07-02T14:03:09.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1348e-05 2023-07-02T14:03:09.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:06.420671+0000", 2023-07-02T14:03:09.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0730000000000001e-05 2023-07-02T14:03:09.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:06.420681+0000", 2023-07-02T14:03:09.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000119998 2023-07-02T14:03:09.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:06.420801+0000", 2023-07-02T14:03:09.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8399000000000003e-05 2023-07-02T14:03:09.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:09.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:06.420840+0000", 2023-07-02T14:03:09.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044850649999999999 2023-07-02T14:03:09.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:09.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:06.425325+0000", 2023-07-02T14:03:09.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3269999999999997e-05 2023-07-02T14:03:09.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:09.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:06.425368+0000", 2023-07-02T14:03:09.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0094971880000000002 2023-07-02T14:03:09.717 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.717 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.717 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.717 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.717 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.717 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:09.719 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:03:09.720 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:09.718+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:09.819 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:09.819+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:09.883 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:09.884 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:09.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.884 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:03:09.884 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:09.884 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.43156053500000002, 2023-07-02T14:03:09.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.43158670599999999, 2023-07-02T14:03:09.885 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:09.885 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "delayed", 2023-07-02T14:03:09.885 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:09.885 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:09.885 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:09.885 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:03:09.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.886 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:09.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:09.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:09.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:09.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:09.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:09.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:09.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:09.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:03:09.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:03:09.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:09.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:03:09.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:03:09.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:09.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:03:09.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:03:09.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:09.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:03:09.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:03:09.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:09.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:03:09.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:03:09.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:09.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:09.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:09.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:03:09.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012924599999999999 2023-07-02T14:03:09.891 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.891 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:09.892 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.892 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:09.892 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:03:09.892 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 1 2023-07-02T14:03:09.892 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:09.894 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:03:09.937 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:09.936+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:10.017 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:10.018 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:10.018 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:10.018 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:10.018 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:10.021 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:10.021+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:10.028 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:03:10.123 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:10.121+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:10.195 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:10.196 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:03:10.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:03:10.196 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:10.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.196 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:279 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.197 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.310714+0000", 2023-07-02T14:03:10.197 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.882851674000001, 2023-07-02T14:03:10.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038119238, 2023-07-02T14:03:10.197 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.197 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.197 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.197 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.197 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.198 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 279 2023-07-02T14:03:10.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.198 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310714+0000", 2023-07-02T14:03:10.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310714+0000", 2023-07-02T14:03:10.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:10.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310713+0000", 2023-07-02T14:03:10.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8973e-05 2023-07-02T14:03:10.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310762+0000", 2023-07-02T14:03:10.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0530000000000007e-06 2023-07-02T14:03:10.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310770+0000", 2023-07-02T14:03:10.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7569999999999993e-06 2023-07-02T14:03:10.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310779+0000", 2023-07-02T14:03:10.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033655263999999997 2023-07-02T14:03:10.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.344434+0000", 2023-07-02T14:03:10.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4365999999999999e-05 2023-07-02T14:03:10.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.344508+0000", 2023-07-02T14:03:10.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016449100000000001 2023-07-02T14:03:10.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:10.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.344673+0000", 2023-07-02T14:03:10.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019402379999999999 2023-07-02T14:03:10.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346613+0000", 2023-07-02T14:03:10.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021203900000000002 2023-07-02T14:03:10.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348734+0000", 2023-07-02T14:03:10.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5225000000000001e-05 2023-07-02T14:03:10.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348789+0000", 2023-07-02T14:03:10.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4163999999999999e-05 2023-07-02T14:03:10.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348833+0000", 2023-07-02T14:03:10.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038119238 2023-07-02T14:03:10.207 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.207 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.207 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.207 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:280 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.207 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.310785+0000", 2023-07-02T14:03:10.208 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.882780800999999, 2023-07-02T14:03:10.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038149954999999999, 2023-07-02T14:03:10.208 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.208 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.208 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.208 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.208 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.209 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 280 2023-07-02T14:03:10.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.209 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310785+0000", 2023-07-02T14:03:10.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310785+0000", 2023-07-02T14:03:10.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:10.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310784+0000", 2023-07-02T14:03:10.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4364999999999999e-05 2023-07-02T14:03:10.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310838+0000", 2023-07-02T14:03:10.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4619999999999996e-06 2023-07-02T14:03:10.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310847+0000", 2023-07-02T14:03:10.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5129999999999998e-06 2023-07-02T14:03:10.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310851+0000", 2023-07-02T14:03:10.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034215981999999999 2023-07-02T14:03:10.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.345067+0000", 2023-07-02T14:03:10.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6117000000000002e-05 2023-07-02T14:03:10.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.345153+0000", 2023-07-02T14:03:10.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014067499999999999 2023-07-02T14:03:10.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:10.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.345294+0000", 2023-07-02T14:03:10.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013560810000000001 2023-07-02T14:03:10.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346650+0000", 2023-07-02T14:03:10.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002196239 2023-07-02T14:03:10.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348846+0000", 2023-07-02T14:03:10.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7621999999999999e-05 2023-07-02T14:03:10.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348894+0000", 2023-07-02T14:03:10.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0518000000000001e-05 2023-07-02T14:03:10.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348935+0000", 2023-07-02T14:03:10.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038149954999999999 2023-07-02T14:03:10.218 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.218 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.218 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.219 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:281 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.219 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.310857+0000", 2023-07-02T14:03:10.219 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.882708538999999, 2023-07-02T14:03:10.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039347695000000002, 2023-07-02T14:03:10.219 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.219 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.219 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.219 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.220 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.220 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:03:10.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.220 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310857+0000", 2023-07-02T14:03:10.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310857+0000", 2023-07-02T14:03:10.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:10.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310856+0000", 2023-07-02T14:03:10.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00081158100000000002 2023-07-02T14:03:10.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311668+0000", 2023-07-02T14:03:10.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.257e-05 2023-07-02T14:03:10.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311680+0000", 2023-07-02T14:03:10.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1110000000000002e-06 2023-07-02T14:03:10.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311688+0000", 2023-07-02T14:03:10.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034985653999999998 2023-07-02T14:03:10.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346673+0000", 2023-07-02T14:03:10.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7062999999999999e-05 2023-07-02T14:03:10.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346730+0000", 2023-07-02T14:03:10.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000151061 2023-07-02T14:03:10.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:10.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.346881+0000", 2023-07-02T14:03:10.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018356119999999999 2023-07-02T14:03:10.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348717+0000", 2023-07-02T14:03:10.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001417385 2023-07-02T14:03:10.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350134+0000", 2023-07-02T14:03:10.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7036999999999997e-05 2023-07-02T14:03:10.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350181+0000", 2023-07-02T14:03:10.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3187999999999999e-05 2023-07-02T14:03:10.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350205+0000", 2023-07-02T14:03:10.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039347695000000002 2023-07-02T14:03:10.229 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.229 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.229 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.229 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:282 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.230 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311695+0000", 2023-07-02T14:03:10.230 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.881870044999999, 2023-07-02T14:03:10.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038558453999999999, 2023-07-02T14:03:10.230 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.230 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.230 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.230 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.230 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.231 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 282 2023-07-02T14:03:10.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.231 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311695+0000", 2023-07-02T14:03:10.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311695+0000", 2023-07-02T14:03:10.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:10.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311695+0000", 2023-07-02T14:03:10.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6679999999999999e-05 2023-07-02T14:03:10.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311751+0000", 2023-07-02T14:03:10.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9199999999999993e-06 2023-07-02T14:03:10.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311760+0000", 2023-07-02T14:03:10.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2480000000000004e-06 2023-07-02T14:03:10.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311765+0000", 2023-07-02T14:03:10.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035670150999999997 2023-07-02T14:03:10.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347436+0000", 2023-07-02T14:03:10.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7861000000000006e-05 2023-07-02T14:03:10.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347503+0000", 2023-07-02T14:03:10.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000217443 2023-07-02T14:03:10.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:10.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.347721+0000", 2023-07-02T14:03:10.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023992449999999999 2023-07-02T14:03:10.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350120+0000", 2023-07-02T14:03:10.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2219000000000006e-05 2023-07-02T14:03:10.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350212+0000", 2023-07-02T14:03:10.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3430000000000001e-05 2023-07-02T14:03:10.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350236+0000", 2023-07-02T14:03:10.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8074999999999999e-05 2023-07-02T14:03:10.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350254+0000", 2023-07-02T14:03:10.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038558453999999999 2023-07-02T14:03:10.240 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.240 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.240 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.240 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:283 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.240 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311771+0000", 2023-07-02T14:03:10.240 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.881794173999999, 2023-07-02T14:03:10.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039285701999999999, 2023-07-02T14:03:10.241 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.241 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.241 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.241 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.241 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.241 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 283 2023-07-02T14:03:10.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.242 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311771+0000", 2023-07-02T14:03:10.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311771+0000", 2023-07-02T14:03:10.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:10.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311771+0000", 2023-07-02T14:03:10.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6775999999999999e-05 2023-07-02T14:03:10.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311837+0000", 2023-07-02T14:03:10.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4290000000000001e-06 2023-07-02T14:03:10.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311846+0000", 2023-07-02T14:03:10.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3800000000000002e-06 2023-07-02T14:03:10.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311851+0000", 2023-07-02T14:03:10.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036308287000000002 2023-07-02T14:03:10.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348160+0000", 2023-07-02T14:03:10.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6475000000000002e-05 2023-07-02T14:03:10.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348236+0000", 2023-07-02T14:03:10.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000147966 2023-07-02T14:03:10.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:10.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348384+0000", 2023-07-02T14:03:10.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019115569999999999 2023-07-02T14:03:10.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350296+0000", 2023-07-02T14:03:10.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00063248000000000004 2023-07-02T14:03:10.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350928+0000", 2023-07-02T14:03:10.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3355000000000003e-05 2023-07-02T14:03:10.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351021+0000", 2023-07-02T14:03:10.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5596000000000003e-05 2023-07-02T14:03:10.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351057+0000", 2023-07-02T14:03:10.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039285701999999999 2023-07-02T14:03:10.250 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.251 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.251 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.251 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:284 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.251 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311859+0000", 2023-07-02T14:03:10.251 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.881706876999999, 2023-07-02T14:03:10.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039967282999999999, 2023-07-02T14:03:10.252 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.252 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.252 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.252 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.252 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.252 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 284 2023-07-02T14:03:10.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.252 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311859+0000", 2023-07-02T14:03:10.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311859+0000", 2023-07-02T14:03:10.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:10.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311858+0000", 2023-07-02T14:03:10.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3902999999999997e-05 2023-07-02T14:03:10.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311892+0000", 2023-07-02T14:03:10.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0989999999999999e-06 2023-07-02T14:03:10.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311900+0000", 2023-07-02T14:03:10.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9030000000000003e-06 2023-07-02T14:03:10.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311905+0000", 2023-07-02T14:03:10.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037060899000000001 2023-07-02T14:03:10.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.348966+0000", 2023-07-02T14:03:10.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3609000000000002e-05 2023-07-02T14:03:10.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349009+0000", 2023-07-02T14:03:10.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6831999999999998e-05 2023-07-02T14:03:10.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:10.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349096+0000", 2023-07-02T14:03:10.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001975077 2023-07-02T14:03:10.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351071+0000", 2023-07-02T14:03:10.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00066166900000000004 2023-07-02T14:03:10.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351733+0000", 2023-07-02T14:03:10.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3199999999999999e-05 2023-07-02T14:03:10.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351786+0000", 2023-07-02T14:03:10.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9823999999999997e-05 2023-07-02T14:03:10.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351826+0000", 2023-07-02T14:03:10.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039967282999999999 2023-07-02T14:03:10.262 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.262 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.262 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.263 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:285 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.263 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311910+0000", 2023-07-02T14:03:10.263 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.881655845000001, 2023-07-02T14:03:10.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040060302999999998, 2023-07-02T14:03:10.263 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.263 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.263 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.263 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.264 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.264 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 285 2023-07-02T14:03:10.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.264 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311910+0000", 2023-07-02T14:03:10.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311910+0000", 2023-07-02T14:03:10.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:10.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311909+0000", 2023-07-02T14:03:10.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1025999999999997e-05 2023-07-02T14:03:10.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311940+0000", 2023-07-02T14:03:10.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4769999999999997e-06 2023-07-02T14:03:10.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311948+0000", 2023-07-02T14:03:10.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5059999999999997e-06 2023-07-02T14:03:10.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311952+0000", 2023-07-02T14:03:10.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037480332999999998 2023-07-02T14:03:10.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349432+0000", 2023-07-02T14:03:10.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2923000000000005e-05 2023-07-02T14:03:10.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349515+0000", 2023-07-02T14:03:10.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015231700000000001 2023-07-02T14:03:10.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:10.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.349668+0000", 2023-07-02T14:03:10.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002171849 2023-07-02T14:03:10.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351839+0000", 2023-07-02T14:03:10.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2052000000000003e-05 2023-07-02T14:03:10.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351882+0000", 2023-07-02T14:03:10.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7683999999999999e-05 2023-07-02T14:03:10.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351929+0000", 2023-07-02T14:03:10.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0648000000000001e-05 2023-07-02T14:03:10.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351970+0000", 2023-07-02T14:03:10.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040060302999999998 2023-07-02T14:03:10.273 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.273 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.273 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.273 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:286 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.274 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.311957+0000", 2023-07-02T14:03:10.274 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.881607978000002, 2023-07-02T14:03:10.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042429263000000002, 2023-07-02T14:03:10.274 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.274 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.274 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.274 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.274 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.275 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 286 2023-07-02T14:03:10.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.275 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311957+0000", 2023-07-02T14:03:10.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311957+0000", 2023-07-02T14:03:10.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:10.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311957+0000", 2023-07-02T14:03:10.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7862000000000003e-05 2023-07-02T14:03:10.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312015+0000", 2023-07-02T14:03:10.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2770000000000005e-06 2023-07-02T14:03:10.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312023+0000", 2023-07-02T14:03:10.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0000000000000004e-06 2023-07-02T14:03:10.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312028+0000", 2023-07-02T14:03:10.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038291327999999999 2023-07-02T14:03:10.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350319+0000", 2023-07-02T14:03:10.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3686000000000001e-05 2023-07-02T14:03:10.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350373+0000", 2023-07-02T14:03:10.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012420000000000001 2023-07-02T14:03:10.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:10.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.350497+0000", 2023-07-02T14:03:10.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014893669999999999 2023-07-02T14:03:10.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351986+0000", 2023-07-02T14:03:10.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022881580000000002 2023-07-02T14:03:10.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354275+0000", 2023-07-02T14:03:10.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7388e-05 2023-07-02T14:03:10.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354332+0000", 2023-07-02T14:03:10.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4688999999999998e-05 2023-07-02T14:03:10.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354387+0000", 2023-07-02T14:03:10.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042429263000000002 2023-07-02T14:03:10.284 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.284 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.284 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.284 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:287 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 589824~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.284 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312033+0000", 2023-07-02T14:03:10.285 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.881532360000001, 2023-07-02T14:03:10.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042515431999999999, 2023-07-02T14:03:10.285 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.285 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.285 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.285 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.285 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.286 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 287 2023-07-02T14:03:10.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.286 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312033+0000", 2023-07-02T14:03:10.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312033+0000", 2023-07-02T14:03:10.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:10.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312033+0000", 2023-07-02T14:03:10.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1540999999999999e-05 2023-07-02T14:03:10.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312094+0000", 2023-07-02T14:03:10.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.123e-06 2023-07-02T14:03:10.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312102+0000", 2023-07-02T14:03:10.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5480000000000003e-06 2023-07-02T14:03:10.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312107+0000", 2023-07-02T14:03:10.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039019356999999998 2023-07-02T14:03:10.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351126+0000", 2023-07-02T14:03:10.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100107 2023-07-02T14:03:10.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351226+0000", 2023-07-02T14:03:10.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000177455 2023-07-02T14:03:10.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:10.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.351404+0000", 2023-07-02T14:03:10.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030030819999999998 2023-07-02T14:03:10.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354407+0000", 2023-07-02T14:03:10.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5787000000000001e-05 2023-07-02T14:03:10.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354443+0000", 2023-07-02T14:03:10.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0627000000000001e-05 2023-07-02T14:03:10.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354503+0000", 2023-07-02T14:03:10.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5327000000000002e-05 2023-07-02T14:03:10.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354548+0000", 2023-07-02T14:03:10.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042515431999999999 2023-07-02T14:03:10.294 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.294 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.295 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.295 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:288 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.295 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312112+0000", 2023-07-02T14:03:10.295 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.881453406999999, 2023-07-02T14:03:10.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051367833000000002, 2023-07-02T14:03:10.295 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.296 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.296 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.296 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.296 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.296 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 288 2023-07-02T14:03:10.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.296 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312112+0000", 2023-07-02T14:03:10.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312112+0000", 2023-07-02T14:03:10.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:10.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312111+0000", 2023-07-02T14:03:10.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8513999999999998e-05 2023-07-02T14:03:10.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312150+0000", 2023-07-02T14:03:10.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6499999999999996e-06 2023-07-02T14:03:10.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312157+0000", 2023-07-02T14:03:10.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7799999999999997e-06 2023-07-02T14:03:10.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312163+0000", 2023-07-02T14:03:10.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039867382 2023-07-02T14:03:10.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352031+0000", 2023-07-02T14:03:10.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6370000000000002e-05 2023-07-02T14:03:10.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352107+0000", 2023-07-02T14:03:10.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000156021 2023-07-02T14:03:10.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:10.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352263+0000", 2023-07-02T14:03:10.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023521890000000002 2023-07-02T14:03:10.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354615+0000", 2023-07-02T14:03:10.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087542309999999995 2023-07-02T14:03:10.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363369+0000", 2023-07-02T14:03:10.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3217999999999998e-05 2023-07-02T14:03:10.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363423+0000", 2023-07-02T14:03:10.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7228999999999997e-05 2023-07-02T14:03:10.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363480+0000", 2023-07-02T14:03:10.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051367833000000002 2023-07-02T14:03:10.305 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.305 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.305 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.305 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:289 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.306 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:10.306 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.881397359000001, 2023-07-02T14:03:10.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051413158, 2023-07-02T14:03:10.306 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.306 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.306 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.306 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.307 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.307 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 289 2023-07-02T14:03:10.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.307 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:10.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:10.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:10.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:10.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1689e-05 2023-07-02T14:03:10.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312209+0000", 2023-07-02T14:03:10.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1070000000000005e-06 2023-07-02T14:03:10.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312217+0000", 2023-07-02T14:03:10.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.634e-06 2023-07-02T14:03:10.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312222+0000", 2023-07-02T14:03:10.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040512761000000001 2023-07-02T14:03:10.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352735+0000", 2023-07-02T14:03:10.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000119184 2023-07-02T14:03:10.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352854+0000", 2023-07-02T14:03:10.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021481899999999999 2023-07-02T14:03:10.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:10.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353069+0000", 2023-07-02T14:03:10.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001601707 2023-07-02T14:03:10.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354670+0000", 2023-07-02T14:03:10.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088168570000000009 2023-07-02T14:03:10.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363487+0000", 2023-07-02T14:03:10.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9635999999999997e-05 2023-07-02T14:03:10.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363537+0000", 2023-07-02T14:03:10.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4292999999999998e-05 2023-07-02T14:03:10.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363581+0000", 2023-07-02T14:03:10.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051413158 2023-07-02T14:03:10.315 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.316 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.316 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.316 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:290 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.316 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:10.316 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.881338113999998, 2023-07-02T14:03:10.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051455062000000003, 2023-07-02T14:03:10.317 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.317 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.317 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.317 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.317 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.317 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 290 2023-07-02T14:03:10.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.318 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:10.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:10.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:10.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:10.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6469000000000003e-05 2023-07-02T14:03:10.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312293+0000", 2023-07-02T14:03:10.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4630000000000007e-06 2023-07-02T14:03:10.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312303+0000", 2023-07-02T14:03:10.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8969999999999999e-06 2023-07-02T14:03:10.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312307+0000", 2023-07-02T14:03:10.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041261548000000002 2023-07-02T14:03:10.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353569+0000", 2023-07-02T14:03:10.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9594999999999999e-05 2023-07-02T14:03:10.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353659+0000", 2023-07-02T14:03:10.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018390600000000001 2023-07-02T14:03:10.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:10.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353843+0000", 2023-07-02T14:03:10.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035173100000000001 2023-07-02T14:03:10.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357360+0000", 2023-07-02T14:03:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062295670000000001 2023-07-02T14:03:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363589+0000", 2023-07-02T14:03:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9854000000000001e-05 2023-07-02T14:03:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363639+0000", 2023-07-02T14:03:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3133999999999997e-05 2023-07-02T14:03:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363682+0000", 2023-07-02T14:03:10.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051455062000000003 2023-07-02T14:03:10.326 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.326 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.326 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.327 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:291 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.327 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:10.327 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.881252074999999, 2023-07-02T14:03:10.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051527860000000002, 2023-07-02T14:03:10.328 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.328 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.328 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.329 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.329 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.329 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 291 2023-07-02T14:03:10.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.330 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:10.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:10.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:10.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:10.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2448999999999998e-05 2023-07-02T14:03:10.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312375+0000", 2023-07-02T14:03:10.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5259999999999993e-06 2023-07-02T14:03:10.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312384+0000", 2023-07-02T14:03:10.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0610000000000003e-06 2023-07-02T14:03:10.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312389+0000", 2023-07-02T14:03:10.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042326111 2023-07-02T14:03:10.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354715+0000", 2023-07-02T14:03:10.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3998999999999995e-05 2023-07-02T14:03:10.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354779+0000", 2023-07-02T14:03:10.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010269500000000001 2023-07-02T14:03:10.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:10.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354882+0000", 2023-07-02T14:03:10.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002533736 2023-07-02T14:03:10.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357415+0000", 2023-07-02T14:03:10.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062765700000000004 2023-07-02T14:03:10.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363692+0000", 2023-07-02T14:03:10.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7095000000000001e-05 2023-07-02T14:03:10.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363749+0000", 2023-07-02T14:03:10.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2263999999999996e-05 2023-07-02T14:03:10.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363841+0000", 2023-07-02T14:03:10.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051527860000000002 2023-07-02T14:03:10.348 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.348 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.348 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.349 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:292 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.349 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:03:10.350 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.881169827000001, 2023-07-02T14:03:10.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051556056000000003, 2023-07-02T14:03:10.350 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.351 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.351 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.351 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.351 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.352 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 292 2023-07-02T14:03:10.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.352 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:03:10.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:03:10.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:10.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312395+0000", 2023-07-02T14:03:10.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9624e-05 2023-07-02T14:03:10.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312425+0000", 2023-07-02T14:03:10.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:03:10.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312432+0000", 2023-07-02T14:03:10.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7550000000000002e-06 2023-07-02T14:03:10.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312437+0000", 2023-07-02T14:03:10.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.043823036000000003 2023-07-02T14:03:10.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356260+0000", 2023-07-02T14:03:10.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6021999999999997e-05 2023-07-02T14:03:10.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356346+0000", 2023-07-02T14:03:10.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020076600000000001 2023-07-02T14:03:10.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:10.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356547+0000", 2023-07-02T14:03:10.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020766410000000002 2023-07-02T14:03:10.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358623+0000", 2023-07-02T14:03:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005231192 2023-07-02T14:03:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363854+0000", 2023-07-02T14:03:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3423e-05 2023-07-02T14:03:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363908+0000", 2023-07-02T14:03:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3825999999999997e-05 2023-07-02T14:03:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363952+0000", 2023-07-02T14:03:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051556056000000003 2023-07-02T14:03:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.365 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.365 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:293 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.365 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:03:10.365 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.881123657, 2023-07-02T14:03:10.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051614769999999997, 2023-07-02T14:03:10.366 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.366 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.366 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.366 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.366 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.366 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 293 2023-07-02T14:03:10.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.367 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:03:10.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:03:10.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:10.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312441+0000", 2023-07-02T14:03:10.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9609999999999999e-05 2023-07-02T14:03:10.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312471+0000", 2023-07-02T14:03:10.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7069999999999992e-06 2023-07-02T14:03:10.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312478+0000", 2023-07-02T14:03:10.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1190000000000002e-06 2023-07-02T14:03:10.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312485+0000", 2023-07-02T14:03:10.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.044956387 2023-07-02T14:03:10.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357441+0000", 2023-07-02T14:03:10.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4134999999999994e-05 2023-07-02T14:03:10.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357515+0000", 2023-07-02T14:03:10.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000147408 2023-07-02T14:03:10.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:10.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357663+0000", 2023-07-02T14:03:10.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056472639999999999 2023-07-02T14:03:10.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363310+0000", 2023-07-02T14:03:10.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00065402600000000004 2023-07-02T14:03:10.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363964+0000", 2023-07-02T14:03:10.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0497000000000001e-05 2023-07-02T14:03:10.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364014+0000", 2023-07-02T14:03:10.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2243000000000002e-05 2023-07-02T14:03:10.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364057+0000", 2023-07-02T14:03:10.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051614769999999997 2023-07-02T14:03:10.379 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.379 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.379 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.380 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:294 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1048576~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.380 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:03:10.380 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.881075658, 2023-07-02T14:03:10.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051679527000000003, 2023-07-02T14:03:10.380 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.381 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.381 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.381 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.381 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.381 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 294 2023-07-02T14:03:10.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.381 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:03:10.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:03:10.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:10.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312489+0000", 2023-07-02T14:03:10.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8204999999999998e-05 2023-07-02T14:03:10.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312517+0000", 2023-07-02T14:03:10.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7759999999999997e-06 2023-07-02T14:03:10.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312525+0000", 2023-07-02T14:03:10.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8730000000000001e-06 2023-07-02T14:03:10.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312532+0000", 2023-07-02T14:03:10.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.045442206999999998 2023-07-02T14:03:10.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357974+0000", 2023-07-02T14:03:10.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4422000000000002e-05 2023-07-02T14:03:10.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358029+0000", 2023-07-02T14:03:10.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149519 2023-07-02T14:03:10.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:10.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358178+0000", 2023-07-02T14:03:10.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051779390000000003 2023-07-02T14:03:10.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363356+0000", 2023-07-02T14:03:10.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00070707699999999997 2023-07-02T14:03:10.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364063+0000", 2023-07-02T14:03:10.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6835999999999999e-05 2023-07-02T14:03:10.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364090+0000", 2023-07-02T14:03:10.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9263000000000003e-05 2023-07-02T14:03:10.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364169+0000", 2023-07-02T14:03:10.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051679527000000003 2023-07-02T14:03:10.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.390 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.391 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:423 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [write 2555904~1638400] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.391 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.330332+0000", 2023-07-02T14:03:10.391 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.863233662999999, 2023-07-02T14:03:10.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038020074000000001, 2023-07-02T14:03:10.391 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.391 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.391 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.391 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.392 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.392 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 423 2023-07-02T14:03:10.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.392 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.330332+0000", 2023-07-02T14:03:10.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.330332+0000", 2023-07-02T14:03:10.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:10.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.330330+0000", 2023-07-02T14:03:10.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060720390000000004 2023-07-02T14:03:10.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336402+0000", 2023-07-02T14:03:10.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8478000000000002e-05 2023-07-02T14:03:10.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336421+0000", 2023-07-02T14:03:10.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0624999999999999e-05 2023-07-02T14:03:10.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336451+0000", 2023-07-02T14:03:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022199402 2023-07-02T14:03:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358651+0000", 2023-07-02T14:03:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1981000000000001e-05 2023-07-02T14:03:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358723+0000", 2023-07-02T14:03:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000163117 2023-07-02T14:03:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:10.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358886+0000", 2023-07-02T14:03:10.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005296762 2023-07-02T14:03:10.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364183+0000", 2023-07-02T14:03:10.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040861739999999997 2023-07-02T14:03:10.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368269+0000", 2023-07-02T14:03:10.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7169000000000003e-05 2023-07-02T14:03:10.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368306+0000", 2023-07-02T14:03:10.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5760999999999998e-05 2023-07-02T14:03:10.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.368352+0000", 2023-07-02T14:03:10.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038020074000000001 2023-07-02T14:03:10.401 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.401 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.401 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.401 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:427 2.2 2:44c63967:::rbd_data.125de1449605.0000000000000101:head [write 2621440~1572864] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.401 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.336464+0000", 2023-07-02T14:03:10.401 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 30.857101835000002, 2023-07-02T14:03:10.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037760331000000001, 2023-07-02T14:03:10.402 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.402 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.402 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.402 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.402 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.402 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 427 2023-07-02T14:03:10.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.403 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336464+0000", 2023-07-02T14:03:10.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336464+0000", 2023-07-02T14:03:10.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:10.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.336462+0000", 2023-07-02T14:03:10.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050934140000000001 2023-07-02T14:03:10.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.341555+0000", 2023-07-02T14:03:10.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6209e-05 2023-07-02T14:03:10.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.341572+0000", 2023-07-02T14:03:10.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.114e-06 2023-07-02T14:03:10.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.341579+0000", 2023-07-02T14:03:10.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019481099000000002 2023-07-02T14:03:10.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361060+0000", 2023-07-02T14:03:10.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5408000000000005e-05 2023-07-02T14:03:10.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361155+0000", 2023-07-02T14:03:10.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000186515 2023-07-02T14:03:10.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:10.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.361342+0000", 2023-07-02T14:03:10.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003066465 2023-07-02T14:03:10.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364408+0000", 2023-07-02T14:03:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0096922329999999998 2023-07-02T14:03:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.374100+0000", 2023-07-02T14:03:10.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7960999999999995e-05 2023-07-02T14:03:10.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.374168+0000", 2023-07-02T14:03:10.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5458000000000003e-05 2023-07-02T14:03:10.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.374224+0000", 2023-07-02T14:03:10.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037760331000000001 2023-07-02T14:03:10.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.411 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.412 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:1066 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e57)", 2023-07-02T14:03:10.412 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.577749+0000", 2023-07-02T14:03:10.412 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.615816551, 2023-07-02T14:03:10.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038354245000000002, 2023-07-02T14:03:10.412 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.412 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.412 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.413 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.413 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.413 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1066 2023-07-02T14:03:10.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.413 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577749+0000", 2023-07-02T14:03:10.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577749+0000", 2023-07-02T14:03:10.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:10.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577748+0000", 2023-07-02T14:03:10.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2019999999999998e-06 2023-07-02T14:03:10.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577756+0000", 2023-07-02T14:03:10.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.528e-06 2023-07-02T14:03:10.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577762+0000", 2023-07-02T14:03:10.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4035999999999997e-05 2023-07-02T14:03:10.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577796+0000", 2023-07-02T14:03:10.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068442729999999997 2023-07-02T14:03:10.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.584640+0000", 2023-07-02T14:03:10.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2489000000000003e-05 2023-07-02T14:03:10.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:10.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.584723+0000", 2023-07-02T14:03:10.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0053785780000000002 2023-07-02T14:03:10.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.590101+0000", 2023-07-02T14:03:10.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0708999999999998e-05 2023-07-02T14:03:10.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:10.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.590162+0000", 2023-07-02T14:03:10.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047004220000000001 2023-07-02T14:03:10.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.594863+0000", 2023-07-02T14:03:10.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0859999999999997e-05 2023-07-02T14:03:10.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:10.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.594923+0000", 2023-07-02T14:03:10.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038761630000000002 2023-07-02T14:03:10.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.598800+0000", 2023-07-02T14:03:10.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106016 2023-07-02T14:03:10.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:10.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.598906+0000", 2023-07-02T14:03:10.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061340400000000003 2023-07-02T14:03:10.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.605040+0000", 2023-07-02T14:03:10.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2472000000000002e-05 2023-07-02T14:03:10.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:10.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.605092+0000", 2023-07-02T14:03:10.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002500927 2023-07-02T14:03:10.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.607593+0000", 2023-07-02T14:03:10.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2676000000000006e-05 2023-07-02T14:03:10.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:10.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.607676+0000", 2023-07-02T14:03:10.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041623739999999999 2023-07-02T14:03:10.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.611838+0000", 2023-07-02T14:03:10.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1777000000000002e-05 2023-07-02T14:03:10.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.611900+0000", 2023-07-02T14:03:10.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00035127200000000002 2023-07-02T14:03:10.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:10.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.612251+0000", 2023-07-02T14:03:10.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022406100000000001 2023-07-02T14:03:10.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.614492+0000", 2023-07-02T14:03:10.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001502139 2023-07-02T14:03:10.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.615994+0000", 2023-07-02T14:03:10.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5522000000000001e-05 2023-07-02T14:03:10.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.616049+0000", 2023-07-02T14:03:10.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3705000000000002e-05 2023-07-02T14:03:10.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.616103+0000", 2023-07-02T14:03:10.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038354245000000002 2023-07-02T14:03:10.430 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.431 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.431 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.432 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:1067 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e57)", 2023-07-02T14:03:10.432 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:05.577810+0000", 2023-07-02T14:03:10.432 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.6157549580000001, 2023-07-02T14:03:10.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040743256999999998, 2023-07-02T14:03:10.432 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.432 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.433 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.433 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.433 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.433 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1067 2023-07-02T14:03:10.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.433 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577810+0000", 2023-07-02T14:03:10.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577810+0000", 2023-07-02T14:03:10.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:10.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577810+0000", 2023-07-02T14:03:10.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7410000000000003e-06 2023-07-02T14:03:10.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577816+0000", 2023-07-02T14:03:10.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4979999999999995e-06 2023-07-02T14:03:10.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577826+0000", 2023-07-02T14:03:10.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5599999999999999e-06 2023-07-02T14:03:10.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.577832+0000", 2023-07-02T14:03:10.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0069249979999999999 2023-07-02T14:03:10.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.584757+0000", 2023-07-02T14:03:10.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8571999999999993e-05 2023-07-02T14:03:10.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:10.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.584826+0000", 2023-07-02T14:03:10.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0053597239999999997 2023-07-02T14:03:10.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.590186+0000", 2023-07-02T14:03:10.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3909999999999999e-05 2023-07-02T14:03:10.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:10.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.590220+0000", 2023-07-02T14:03:10.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047859419999999996 2023-07-02T14:03:10.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.595006+0000", 2023-07-02T14:03:10.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5887000000000001e-05 2023-07-02T14:03:10.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:10.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.595071+0000", 2023-07-02T14:03:10.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038991120000000002 2023-07-02T14:03:10.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.598971+0000", 2023-07-02T14:03:10.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0792999999999999e-05 2023-07-02T14:03:10.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:10.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.599031+0000", 2023-07-02T14:03:10.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060912229999999998 2023-07-02T14:03:10.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.605123+0000", 2023-07-02T14:03:10.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3111e-05 2023-07-02T14:03:10.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:10.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.605176+0000", 2023-07-02T14:03:10.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002557372 2023-07-02T14:03:10.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.607733+0000", 2023-07-02T14:03:10.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013124799999999999 2023-07-02T14:03:10.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:10.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.607864+0000", 2023-07-02T14:03:10.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048035049999999996 2023-07-02T14:03:10.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.612668+0000", 2023-07-02T14:03:10.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0899e-05 2023-07-02T14:03:10.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:10.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.612739+0000", 2023-07-02T14:03:10.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033911969999999999 2023-07-02T14:03:10.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.616130+0000", 2023-07-02T14:03:10.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9387e-05 2023-07-02T14:03:10.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.616179+0000", 2023-07-02T14:03:10.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042055800000000002 2023-07-02T14:03:10.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:10.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.616600+0000", 2023-07-02T14:03:10.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001567188 2023-07-02T14:03:10.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.618167+0000", 2023-07-02T14:03:10.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030074700000000003 2023-07-02T14:03:10.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.618468+0000", 2023-07-02T14:03:10.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9024999999999999e-05 2023-07-02T14:03:10.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.618517+0000", 2023-07-02T14:03:10.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6959000000000003e-05 2023-07-02T14:03:10.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:05.618554+0000", 2023-07-02T14:03:10.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040743256999999998 2023-07-02T14:03:10.451 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.451 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.451 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.451 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.451 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.451 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:10.452 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:03:10.454 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:03:10.454 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:10.451+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:10.554 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:10.553+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:10.620 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:10.621 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:10.621 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:10.621 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:10.631 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:03:10.675 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:10.674+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:10.755 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:10.754+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:10.756 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:10.756 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:10.757 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:10.757 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:10.757 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:10.767 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:03:10.856 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:10.856+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:10.933 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:03:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:03:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:165 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.284977+0000", 2023-07-02T14:03:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.646269233000002, 2023-07-02T14:03:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027689147000000001, 2023-07-02T14:03:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 165 2023-07-02T14:03:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284977+0000", 2023-07-02T14:03:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284977+0000", 2023-07-02T14:03:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.284976+0000", 2023-07-02T14:03:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3962999999999996e-05 2023-07-02T14:03:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285060+0000", 2023-07-02T14:03:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1260000000000004e-06 2023-07-02T14:03:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285070+0000", 2023-07-02T14:03:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0669999999999999e-06 2023-07-02T14:03:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285075+0000", 2023-07-02T14:03:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016699978000000001 2023-07-02T14:03:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.301775+0000", 2023-07-02T14:03:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6872000000000004e-05 2023-07-02T14:03:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.301862+0000", 2023-07-02T14:03:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000155592 2023-07-02T14:03:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.302017+0000", 2023-07-02T14:03:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046740250000000001 2023-07-02T14:03:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306691+0000", 2023-07-02T14:03:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059023670000000004 2023-07-02T14:03:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312593+0000", 2023-07-02T14:03:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6545999999999999e-05 2023-07-02T14:03:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312630+0000", 2023-07-02T14:03:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6270999999999999e-05 2023-07-02T14:03:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312666+0000", 2023-07-02T14:03:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027689147000000001 2023-07-02T14:03:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:168 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1507328~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285081+0000", 2023-07-02T14:03:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.646165222, 2023-07-02T14:03:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027656613, 2023-07-02T14:03:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 168 2023-07-02T14:03:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285081+0000", 2023-07-02T14:03:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285081+0000", 2023-07-02T14:03:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285080+0000", 2023-07-02T14:03:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5372999999999998e-05 2023-07-02T14:03:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285156+0000", 2023-07-02T14:03:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0099999999999995e-06 2023-07-02T14:03:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285164+0000", 2023-07-02T14:03:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3320000000000002e-06 2023-07-02T14:03:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285168+0000", 2023-07-02T14:03:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017721316000000001 2023-07-02T14:03:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.302889+0000", 2023-07-02T14:03:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9568999999999998e-05 2023-07-02T14:03:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.302949+0000", 2023-07-02T14:03:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121902 2023-07-02T14:03:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303071+0000", 2023-07-02T14:03:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0071548599999999999 2023-07-02T14:03:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310226+0000", 2023-07-02T14:03:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024455520000000001 2023-07-02T14:03:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312671+0000", 2023-07-02T14:03:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6958000000000001e-05 2023-07-02T14:03:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312708+0000", 2023-07-02T14:03:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9451e-05 2023-07-02T14:03:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312738+0000", 2023-07-02T14:03:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027656613 2023-07-02T14:03:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:171 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1572864~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285174+0000", 2023-07-02T14:03:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.646072705000002, 2023-07-02T14:03:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027618999000000002, 2023-07-02T14:03:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 171 2023-07-02T14:03:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285174+0000", 2023-07-02T14:03:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285174+0000", 2023-07-02T14:03:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285173+0000", 2023-07-02T14:03:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1789000000000002e-05 2023-07-02T14:03:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285225+0000", 2023-07-02T14:03:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6429999999999995e-06 2023-07-02T14:03:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285232+0000", 2023-07-02T14:03:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8380000000000001e-06 2023-07-02T14:03:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285237+0000", 2023-07-02T14:03:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018182713 2023-07-02T14:03:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303420+0000", 2023-07-02T14:03:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0059000000000001e-05 2023-07-02T14:03:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303490+0000", 2023-07-02T14:03:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012003 2023-07-02T14:03:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303610+0000", 2023-07-02T14:03:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066642150000000002 2023-07-02T14:03:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310274+0000", 2023-07-02T14:03:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024681949999999999 2023-07-02T14:03:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312743+0000", 2023-07-02T14:03:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5525999999999999e-05 2023-07-02T14:03:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312768+0000", 2023-07-02T14:03:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4613999999999998e-05 2023-07-02T14:03:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312793+0000", 2023-07-02T14:03:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027618999000000002 2023-07-02T14:03:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:174 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285243+0000", 2023-07-02T14:03:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.646003508, 2023-07-02T14:03:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027607988, 2023-07-02T14:03:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 174 2023-07-02T14:03:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285243+0000", 2023-07-02T14:03:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285243+0000", 2023-07-02T14:03:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285242+0000", 2023-07-02T14:03:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6173000000000003e-05 2023-07-02T14:03:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285328+0000", 2023-07-02T14:03:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5729999999999997e-06 2023-07-02T14:03:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285338+0000", 2023-07-02T14:03:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.081e-06 2023-07-02T14:03:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285347+0000", 2023-07-02T14:03:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018567367000000001 2023-07-02T14:03:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303914+0000", 2023-07-02T14:03:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9522000000000003e-05 2023-07-02T14:03:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.303984+0000", 2023-07-02T14:03:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013801800000000001 2023-07-02T14:03:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.304122+0000", 2023-07-02T14:03:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061898489999999999 2023-07-02T14:03:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310312+0000", 2023-07-02T14:03:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024882519999999998 2023-07-02T14:03:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312800+0000", 2023-07-02T14:03:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7100000000000001e-05 2023-07-02T14:03:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312827+0000", 2023-07-02T14:03:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3771999999999999e-05 2023-07-02T14:03:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312851+0000", 2023-07-02T14:03:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027607988 2023-07-02T14:03:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:177 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285353+0000", 2023-07-02T14:03:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.645893324999999, 2023-07-02T14:03:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027577522, 2023-07-02T14:03:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 177 2023-07-02T14:03:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285353+0000", 2023-07-02T14:03:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285353+0000", 2023-07-02T14:03:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285352+0000", 2023-07-02T14:03:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7850999999999996e-05 2023-07-02T14:03:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285440+0000", 2023-07-02T14:03:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7960000000000001e-06 2023-07-02T14:03:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285449+0000", 2023-07-02T14:03:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0529999999999997e-06 2023-07-02T14:03:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285454+0000", 2023-07-02T14:03:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018975296999999999 2023-07-02T14:03:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.304429+0000", 2023-07-02T14:03:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0682e-05 2023-07-02T14:03:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.304480+0000", 2023-07-02T14:03:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116107 2023-07-02T14:03:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.304596+0000", 2023-07-02T14:03:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0057486309999999997 2023-07-02T14:03:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310345+0000", 2023-07-02T14:03:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025147289999999998 2023-07-02T14:03:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312860+0000", 2023-07-02T14:03:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7252999999999997e-05 2023-07-02T14:03:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312897+0000", 2023-07-02T14:03:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3705999999999999e-05 2023-07-02T14:03:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312931+0000", 2023-07-02T14:03:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027577522 2023-07-02T14:03:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:183 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.645707154, 2023-07-02T14:03:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003, 2023-07-02T14:03:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 183 2023-07-02T14:03:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9790999999999997e-05 2023-07-02T14:03:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285598+0000", 2023-07-02T14:03:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5809999999999996e-06 2023-07-02T14:03:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285604+0000", 2023-07-02T14:03:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7599999999999998e-06 2023-07-02T14:03:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285607+0000", 2023-07-02T14:03:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021108188999999999 2023-07-02T14:03:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306715+0000", 2023-07-02T14:03:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6548e-05 2023-07-02T14:03:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306792+0000", 2023-07-02T14:03:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173869 2023-07-02T14:03:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306965+0000", 2023-07-02T14:03:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060498009999999996 2023-07-02T14:03:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313015+0000", 2023-07-02T14:03:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049443719999999998 2023-07-02T14:03:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317960+0000", 2023-07-02T14:03:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5927999999999998e-05 2023-07-02T14:03:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318006+0000", 2023-07-02T14:03:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6004999999999998e-05 2023-07-02T14:03:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318042+0000", 2023-07-02T14:03:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003 2023-07-02T14:03:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:186 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.645636327999998, 2023-07-02T14:03:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003, 2023-07-02T14:03:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 186 2023-07-02T14:03:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4279e-05 2023-07-02T14:03:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285664+0000", 2023-07-02T14:03:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3789999999999995e-06 2023-07-02T14:03:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285673+0000", 2023-07-02T14:03:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499999999999998e-06 2023-07-02T14:03:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285678+0000", 2023-07-02T14:03:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023261947000000002 2023-07-02T14:03:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.308940+0000", 2023-07-02T14:03:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010323399999999999 2023-07-02T14:03:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309044+0000", 2023-07-02T14:03:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018878300000000001 2023-07-02T14:03:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309232+0000", 2023-07-02T14:03:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037874150000000001 2023-07-02T14:03:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313020+0000", 2023-07-02T14:03:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050976140000000003 2023-07-02T14:03:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318117+0000", 2023-07-02T14:03:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3182000000000001e-05 2023-07-02T14:03:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5055999999999998e-05 2023-07-02T14:03:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318196+0000", 2023-07-02T14:03:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003 2023-07-02T14:03:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:189 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:03:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.645562564999999, 2023-07-02T14:03:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997, 2023-07-02T14:03:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:03:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:03:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:03:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285683+0000", 2023-07-02T14:03:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8451999999999999e-05 2023-07-02T14:03:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285782+0000", 2023-07-02T14:03:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3650000000000004e-06 2023-07-02T14:03:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285790+0000", 2023-07-02T14:03:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8230000000000002e-06 2023-07-02T14:03:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285795+0000", 2023-07-02T14:03:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024619367999999999 2023-07-02T14:03:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310414+0000", 2023-07-02T14:03:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014244600000000001 2023-07-02T14:03:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310557+0000", 2023-07-02T14:03:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020612699999999999 2023-07-02T14:03:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310763+0000", 2023-07-02T14:03:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003427687 2023-07-02T14:03:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314191+0000", 2023-07-02T14:03:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040492089999999998 2023-07-02T14:03:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318240+0000", 2023-07-02T14:03:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0287999999999999e-05 2023-07-02T14:03:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318270+0000", 2023-07-02T14:03:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0868000000000002e-05 2023-07-02T14:03:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318311+0000", 2023-07-02T14:03:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997 2023-07-02T14:03:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:192 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:03:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.645445792, 2023-07-02T14:03:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003, 2023-07-02T14:03:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 192 2023-07-02T14:03:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:03:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:03:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285800+0000", 2023-07-02T14:03:11.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100417 2023-07-02T14:03:11.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:11.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285900+0000", 2023-07-02T14:03:11.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8244999999999999e-05 2023-07-02T14:03:11.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:11.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285919+0000", 2023-07-02T14:03:11.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2499999999999997e-06 2023-07-02T14:03:11.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285924+0000", 2023-07-02T14:03:11.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025288155999999999 2023-07-02T14:03:11.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:11.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311212+0000", 2023-07-02T14:03:11.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4073999999999996e-05 2023-07-02T14:03:11.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:11.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311286+0000", 2023-07-02T14:03:11.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016219699999999999 2023-07-02T14:03:11.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311448+0000", 2023-07-02T14:03:11.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066020599999999999 2023-07-02T14:03:11.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:11.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318050+0000", 2023-07-02T14:03:11.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000327986 2023-07-02T14:03:11.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318378+0000", 2023-07-02T14:03:11.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8693999999999998e-05 2023-07-02T14:03:11.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:11.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318427+0000", 2023-07-02T14:03:11.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6656999999999998e-05 2023-07-02T14:03:11.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:11.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318464+0000", 2023-07-02T14:03:11.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003 2023-07-02T14:03:11.031 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.031 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:11.031 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.032 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:195 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.032 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:03:11.032 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.645314288000002, 2023-07-02T14:03:11.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001, 2023-07-02T14:03:11.032 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:11.032 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.033 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:11.033 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:11.033 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.033 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 195 2023-07-02T14:03:11.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.033 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:11.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:11.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:03:11.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:11.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:11.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:03:11.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:11.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:11.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285931+0000", 2023-07-02T14:03:11.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4530999999999995e-05 2023-07-02T14:03:11.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:11.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286006+0000", 2023-07-02T14:03:11.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4113e-05 2023-07-02T14:03:11.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:11.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286020+0000", 2023-07-02T14:03:11.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7459999999999997e-06 2023-07-02T14:03:11.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286026+0000", 2023-07-02T14:03:11.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025997769 2023-07-02T14:03:11.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:11.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312023+0000", 2023-07-02T14:03:11.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4403000000000001e-05 2023-07-02T14:03:11.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:11.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312078+0000", 2023-07-02T14:03:11.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020581399999999999 2023-07-02T14:03:11.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312284+0000", 2023-07-02T14:03:11.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005771043 2023-07-02T14:03:11.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:11.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318055+0000", 2023-07-02T14:03:11.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00046015200000000002 2023-07-02T14:03:11.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318515+0000", 2023-07-02T14:03:11.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9444000000000003e-05 2023-07-02T14:03:11.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:11.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318564+0000", 2023-07-02T14:03:11.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2141000000000002e-05 2023-07-02T14:03:11.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:11.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318606+0000", 2023-07-02T14:03:11.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001 2023-07-02T14:03:11.042 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.042 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:11.042 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.043 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:198 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.043 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:03:11.043 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.645214329000002, 2023-07-02T14:03:11.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999, 2023-07-02T14:03:11.043 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:11.043 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.044 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:11.044 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:11.044 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.044 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 198 2023-07-02T14:03:11.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.044 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:11.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:11.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:03:11.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:11.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:11.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:03:11.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:11.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:11.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286031+0000", 2023-07-02T14:03:11.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00049286700000000002 2023-07-02T14:03:11.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:11.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286524+0000", 2023-07-02T14:03:11.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4938e-05 2023-07-02T14:03:11.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:11.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286539+0000", 2023-07-02T14:03:11.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0680000000000002e-06 2023-07-02T14:03:11.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286545+0000", 2023-07-02T14:03:11.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026497718 2023-07-02T14:03:11.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:11.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313043+0000", 2023-07-02T14:03:11.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0884999999999998e-05 2023-07-02T14:03:11.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:11.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313094+0000", 2023-07-02T14:03:11.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014109999999999999 2023-07-02T14:03:11.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313235+0000", 2023-07-02T14:03:11.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048258980000000003 2023-07-02T14:03:11.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:11.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318061+0000", 2023-07-02T14:03:11.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014346350000000001 2023-07-02T14:03:11.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319495+0000", 2023-07-02T14:03:11.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1986e-05 2023-07-02T14:03:11.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:11.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319547+0000", 2023-07-02T14:03:11.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0457e-05 2023-07-02T14:03:11.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:11.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319578+0000", 2023-07-02T14:03:11.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999 2023-07-02T14:03:11.053 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.053 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:11.053 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.054 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:201 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.054 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:11.054 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.644693904, 2023-07-02T14:03:11.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003, 2023-07-02T14:03:11.054 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:11.054 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.054 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:11.054 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:11.055 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.055 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 201 2023-07-02T14:03:11.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.055 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:11.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:11.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:11.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:11.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:11.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:11.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:11.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:11.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:11.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019213999999999999 2023-07-02T14:03:11.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:11.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286744+0000", 2023-07-02T14:03:11.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5279999999999992e-06 2023-07-02T14:03:11.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:11.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286753+0000", 2023-07-02T14:03:11.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4200000000000004e-06 2023-07-02T14:03:11.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286760+0000", 2023-07-02T14:03:11.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026818386999999999 2023-07-02T14:03:11.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:11.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313578+0000", 2023-07-02T14:03:11.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015965 2023-07-02T14:03:11.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:11.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313738+0000", 2023-07-02T14:03:11.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015767299999999999 2023-07-02T14:03:11.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313895+0000", 2023-07-02T14:03:11.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041714930000000001 2023-07-02T14:03:11.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:11.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318067+0000", 2023-07-02T14:03:11.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001561698 2023-07-02T14:03:11.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319629+0000", 2023-07-02T14:03:11.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2346000000000003e-05 2023-07-02T14:03:11.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:11.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319681+0000", 2023-07-02T14:03:11.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4691000000000003e-05 2023-07-02T14:03:11.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:11.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319726+0000", 2023-07-02T14:03:11.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003 2023-07-02T14:03:11.063 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.064 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:11.064 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.064 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:204 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.064 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:11.064 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.644479065999999, 2023-07-02T14:03:11.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998, 2023-07-02T14:03:11.065 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:11.065 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.065 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:11.065 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:11.065 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.065 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 204 2023-07-02T14:03:11.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.065 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:11.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:11.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:11.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:11.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:11.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:11.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:11.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:11.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:11.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2497999999999998e-05 2023-07-02T14:03:11.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:11.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286819+0000", 2023-07-02T14:03:11.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0099999999999995e-06 2023-07-02T14:03:11.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:11.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286827+0000", 2023-07-02T14:03:11.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000003e-06 2023-07-02T14:03:11.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286833+0000", 2023-07-02T14:03:11.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027384716999999999 2023-07-02T14:03:11.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:11.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314218+0000", 2023-07-02T14:03:11.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6204000000000004e-05 2023-07-02T14:03:11.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:11.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314294+0000", 2023-07-02T14:03:11.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015757499999999999 2023-07-02T14:03:11.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314452+0000", 2023-07-02T14:03:11.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036215560000000002 2023-07-02T14:03:11.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:11.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318073+0000", 2023-07-02T14:03:11.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017023979999999999 2023-07-02T14:03:11.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319776+0000", 2023-07-02T14:03:11.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9561000000000002e-05 2023-07-02T14:03:11.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:11.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319825+0000", 2023-07-02T14:03:11.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0329e-05 2023-07-02T14:03:11.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:11.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319866+0000", 2023-07-02T14:03:11.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998 2023-07-02T14:03:11.074 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.074 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:11.074 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.075 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:207 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.075 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:03:11.075 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.644404421000001, 2023-07-02T14:03:11.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003, 2023-07-02T14:03:11.075 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:11.075 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.075 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:11.076 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:11.076 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.076 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 207 2023-07-02T14:03:11.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.076 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:11.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:11.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:03:11.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:11.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:11.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:03:11.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:11.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:11.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286841+0000", 2023-07-02T14:03:11.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00068375100000000002 2023-07-02T14:03:11.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:11.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287525+0000", 2023-07-02T14:03:11.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1601e-05 2023-07-02T14:03:11.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:11.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287537+0000", 2023-07-02T14:03:11.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6409999999999996e-06 2023-07-02T14:03:11.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287543+0000", 2023-07-02T14:03:11.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027245844000000002 2023-07-02T14:03:11.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:11.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314789+0000", 2023-07-02T14:03:11.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8910000000000003e-05 2023-07-02T14:03:11.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:11.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314858+0000", 2023-07-02T14:03:11.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030992599999999997 2023-07-02T14:03:11.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315168+0000", 2023-07-02T14:03:11.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029122869999999999 2023-07-02T14:03:11.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:11.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318080+0000", 2023-07-02T14:03:11.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019963020000000001 2023-07-02T14:03:11.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320077+0000", 2023-07-02T14:03:11.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5116000000000005e-05 2023-07-02T14:03:11.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:11.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320142+0000", 2023-07-02T14:03:11.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3793000000000004e-05 2023-07-02T14:03:11.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:11.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320205+0000", 2023-07-02T14:03:11.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003 2023-07-02T14:03:11.084 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.085 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:11.085 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.085 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:236 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2424832~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.085 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:03:11.085 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.643692238, 2023-07-02T14:03:11.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003, 2023-07-02T14:03:11.086 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:11.086 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.086 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:11.086 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:11.086 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.086 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2023-07-02T14:03:11.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.087 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:11.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:11.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:03:11.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:11.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:11.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:03:11.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:11.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:11.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287553+0000", 2023-07-02T14:03:11.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0667e-05 2023-07-02T14:03:11.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:11.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287614+0000", 2023-07-02T14:03:11.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0454e-05 2023-07-02T14:03:11.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:11.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287624+0000", 2023-07-02T14:03:11.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8564999999999999e-05 2023-07-02T14:03:11.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287643+0000", 2023-07-02T14:03:11.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027977596 2023-07-02T14:03:11.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:11.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315621+0000", 2023-07-02T14:03:11.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000201728 2023-07-02T14:03:11.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:11.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315822+0000", 2023-07-02T14:03:11.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018436599999999999 2023-07-02T14:03:11.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316007+0000", 2023-07-02T14:03:11.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042423440000000003 2023-07-02T14:03:11.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320249+0000", 2023-07-02T14:03:11.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001452633 2023-07-02T14:03:11.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:11.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321702+0000", 2023-07-02T14:03:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3616000000000003e-05 2023-07-02T14:03:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321755+0000", 2023-07-02T14:03:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840999999999999e-05 2023-07-02T14:03:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321793+0000", 2023-07-02T14:03:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003 2023-07-02T14:03:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.096 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:251 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.096 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:03:11.096 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.643596701, 2023-07-02T14:03:11.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001, 2023-07-02T14:03:11.096 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:11.096 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.097 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:11.097 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:11.097 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.097 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 251 2023-07-02T14:03:11.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.097 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:11.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:11.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:03:11.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:11.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:11.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:03:11.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:11.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:11.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287649+0000", 2023-07-02T14:03:11.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3232000000000001e-05 2023-07-02T14:03:11.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:11.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287702+0000", 2023-07-02T14:03:11.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0010000000000007e-06 2023-07-02T14:03:11.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:11.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287711+0000", 2023-07-02T14:03:11.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3569999999999997e-06 2023-07-02T14:03:11.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287717+0000", 2023-07-02T14:03:11.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028686454 2023-07-02T14:03:11.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:11.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316403+0000", 2023-07-02T14:03:11.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7741000000000002e-05 2023-07-02T14:03:11.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:11.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316461+0000", 2023-07-02T14:03:11.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4313000000000004e-05 2023-07-02T14:03:11.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316545+0000", 2023-07-02T14:03:11.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003734032 2023-07-02T14:03:11.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320279+0000", 2023-07-02T14:03:11.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015237219999999999 2023-07-02T14:03:11.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:11.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321803+0000", 2023-07-02T14:03:11.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1927e-05 2023-07-02T14:03:11.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:11.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321845+0000", 2023-07-02T14:03:11.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5398999999999998e-05 2023-07-02T14:03:11.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:11.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321880+0000", 2023-07-02T14:03:11.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001 2023-07-02T14:03:11.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.106 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:11.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.107 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:255 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2555904~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.107 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:03:11.107 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.643181973000001, 2023-07-02T14:03:11.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002, 2023-07-02T14:03:11.107 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:11.107 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.107 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:11.108 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:11.108 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.108 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 255 2023-07-02T14:03:11.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.108 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:11.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:11.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:03:11.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:11.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:11.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:03:11.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:11.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:11.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288063+0000", 2023-07-02T14:03:11.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4459999999999997e-05 2023-07-02T14:03:11.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:11.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288118+0000", 2023-07-02T14:03:11.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0998999999999999e-05 2023-07-02T14:03:11.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:11.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288129+0000", 2023-07-02T14:03:11.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3219999999999998e-06 2023-07-02T14:03:11.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288135+0000", 2023-07-02T14:03:11.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029137386000000001 2023-07-02T14:03:11.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317273+0000", 2023-07-02T14:03:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7819999999999997e-05 2023-07-02T14:03:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317350+0000", 2023-07-02T14:03:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000174864 2023-07-02T14:03:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317525+0000", 2023-07-02T14:03:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027943519999999999 2023-07-02T14:03:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320320+0000", 2023-07-02T14:03:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015681460000000001 2023-07-02T14:03:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321888+0000", 2023-07-02T14:03:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9508999999999997e-05 2023-07-02T14:03:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321927+0000", 2023-07-02T14:03:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5428e-05 2023-07-02T14:03:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321963+0000", 2023-07-02T14:03:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002 2023-07-02T14:03:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:446 2.1 2:910b2a90:::rbd_data.125de1449605.000000000000010a:head [write 65536~3211264] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:11.118 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.565902221000002, 2023-07-02T14:03:11.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003, 2023-07-02T14:03:11.118 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:11.118 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.118 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:11.118 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 446 2023-07-02T14:03:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015907372999999999 2023-07-02T14:03:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381251+0000", 2023-07-02T14:03:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5527000000000001e-05 2023-07-02T14:03:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381277+0000", 2023-07-02T14:03:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9958000000000001e-05 2023-07-02T14:03:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381297+0000", 2023-07-02T14:03:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4941000000000001e-05 2023-07-02T14:03:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381352+0000", 2023-07-02T14:03:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011565699999999999 2023-07-02T14:03:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381467+0000", 2023-07-02T14:03:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018039799999999999 2023-07-02T14:03:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381648+0000", 2023-07-02T14:03:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088183559999999994 2023-07-02T14:03:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.390466+0000", 2023-07-02T14:03:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007278278 2023-07-02T14:03:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397744+0000", 2023-07-02T14:03:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9289000000000001e-05 2023-07-02T14:03:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397804+0000", 2023-07-02T14:03:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0479000000000002e-05 2023-07-02T14:03:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397854+0000", 2023-07-02T14:03:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003 2023-07-02T14:03:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:03:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:03:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.8235714269999992, 2023-07-02T14:03:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:03:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:03:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:03:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:03:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:03:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:03:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:03:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:03:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:03:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:03:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:03:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:03:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:03:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:03:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:03:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:03:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:03:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:03:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:03:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:03:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:03:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:03:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:03:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:03:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:03:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.8235106890000008, 2023-07-02T14:03:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:03:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:03:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:03:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:03:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:03:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:03:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:03:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:03:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:03:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:03:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:03:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:03:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:03:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:03:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:03:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:03:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:03:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:03:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:03:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:03:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:03:11.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:03:11.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:11.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:03:11.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:03:11.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:03:11.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:03:11.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:11.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:03:11.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:03:11.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:11.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:11.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:11.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:03:11.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:03:11.150 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.150 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:11.150 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.150 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:11.150 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:11.150 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:11.152 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:03:11.152 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:11.151+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:11.253 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:11.252+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:11.289 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:11.289 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:11.289 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:03:11.290 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:11.300 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:03:11.332 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:03:11.356 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:11.355+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:11.451 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 51% complete...2023-07-02T14:03:11.450+0000 7ff902563700 0 -- 172.21.15.163:0/1039679439 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x55f9bb382d90 msgr2=0x55f9bb3851c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:03:11.455 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:11.454+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:11.467 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 100% complete...done. 2023-07-02T14:03:11.470 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:03:11.471 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=5 2023-07-02T14:03:11.471 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap5 2023-07-02T14:03:11.484 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:11.484 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:11.485 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:11.485 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:11.485 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:11.496 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:03:11.556 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:11.555+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:11.656 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:11.655+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:11.669 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:11.669 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:11.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:11.670 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:11.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.670 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:322 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.670 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:11.670 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.368095003999997, 2023-07-02T14:03:11.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.03676041, 2023-07-02T14:03:11.671 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:11.671 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.671 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:11.671 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:11.671 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.671 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 322 2023-07-02T14:03:11.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.672 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:11.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:11.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:11.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:11.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:11.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:11.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:11.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:11.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298819+0000", 2023-07-02T14:03:11.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8068e-05 2023-07-02T14:03:11.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:11.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298877+0000", 2023-07-02T14:03:11.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3339999999999998e-06 2023-07-02T14:03:11.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:11.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298883+0000", 2023-07-02T14:03:11.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1030000000000004e-06 2023-07-02T14:03:11.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298887+0000", 2023-07-02T14:03:11.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023640230000000002 2023-07-02T14:03:11.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322528+0000", 2023-07-02T14:03:11.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8424999999999994e-05 2023-07-02T14:03:11.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:11.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322616+0000", 2023-07-02T14:03:11.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116815 2023-07-02T14:03:11.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:11.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322733+0000", 2023-07-02T14:03:11.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048252629999999998 2023-07-02T14:03:11.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:11.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327558+0000", 2023-07-02T14:03:11.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078965240000000003 2023-07-02T14:03:11.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335455+0000", 2023-07-02T14:03:11.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0511999999999999e-05 2023-07-02T14:03:11.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:11.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335515+0000", 2023-07-02T14:03:11.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4789000000000006e-05 2023-07-02T14:03:11.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:11.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335580+0000", 2023-07-02T14:03:11.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.03676041 2023-07-02T14:03:11.680 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.680 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.680 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.681 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:323 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.681 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:11.681 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.368023624000003, 2023-07-02T14:03:11.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036875044000000003, 2023-07-02T14:03:11.681 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:11.681 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.681 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:11.682 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:11.682 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.682 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 323 2023-07-02T14:03:11.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.682 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:11.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:11.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:11.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:11.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:11.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:11.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:11.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:11.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298890+0000", 2023-07-02T14:03:11.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7277000000000002e-05 2023-07-02T14:03:11.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:11.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298938+0000", 2023-07-02T14:03:11.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0495e-05 2023-07-02T14:03:11.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:11.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298948+0000", 2023-07-02T14:03:11.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6829999999999999e-06 2023-07-02T14:03:11.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298953+0000", 2023-07-02T14:03:11.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024271435000000001 2023-07-02T14:03:11.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323224+0000", 2023-07-02T14:03:11.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9266999999999998e-05 2023-07-02T14:03:11.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:11.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323304+0000", 2023-07-02T14:03:11.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000157021 2023-07-02T14:03:11.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:11.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323461+0000", 2023-07-02T14:03:11.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041036689999999999 2023-07-02T14:03:11.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:11.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327564+0000", 2023-07-02T14:03:11.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080753490000000008 2023-07-02T14:03:11.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335640+0000", 2023-07-02T14:03:11.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7398999999999994e-05 2023-07-02T14:03:11.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:11.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335727+0000", 2023-07-02T14:03:11.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8967000000000001e-05 2023-07-02T14:03:11.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:11.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335766+0000", 2023-07-02T14:03:11.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036875044000000003 2023-07-02T14:03:11.691 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.691 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.691 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.692 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:324 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.692 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:11.692 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.367956245999999, 2023-07-02T14:03:11.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036935673000000002, 2023-07-02T14:03:11.692 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:11.693 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.693 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:11.693 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:11.693 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.693 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 324 2023-07-02T14:03:11.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.693 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:11.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:11.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:11.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:11.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:11.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:11.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:11.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:11.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:11.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5519999999999997e-05 2023-07-02T14:03:11.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:11.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299013+0000", 2023-07-02T14:03:11.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0701e-05 2023-07-02T14:03:11.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:11.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299024+0000", 2023-07-02T14:03:11.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2379999999999999e-06 2023-07-02T14:03:11.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299028+0000", 2023-07-02T14:03:11.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024972768999999999 2023-07-02T14:03:11.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324001+0000", 2023-07-02T14:03:11.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1178000000000002e-05 2023-07-02T14:03:11.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:11.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324072+0000", 2023-07-02T14:03:11.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000166119 2023-07-02T14:03:11.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:11.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324238+0000", 2023-07-02T14:03:11.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033314210000000002 2023-07-02T14:03:11.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:11.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327570+0000", 2023-07-02T14:03:11.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0082393369999999994 2023-07-02T14:03:11.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335809+0000", 2023-07-02T14:03:11.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7071999999999999e-05 2023-07-02T14:03:11.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:11.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335856+0000", 2023-07-02T14:03:11.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7926999999999998e-05 2023-07-02T14:03:11.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:11.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335894+0000", 2023-07-02T14:03:11.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036935673000000002 2023-07-02T14:03:11.702 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.702 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.702 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.702 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:325 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.702 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:11.703 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.367880919000001, 2023-07-02T14:03:11.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036990838999999998, 2023-07-02T14:03:11.703 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:11.703 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.703 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:11.703 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:11.703 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.703 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 325 2023-07-02T14:03:11.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.704 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:11.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:11.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:11.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:11.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:11.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:11.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:11.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:11.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299033+0000", 2023-07-02T14:03:11.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107619 2023-07-02T14:03:11.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:11.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299141+0000", 2023-07-02T14:03:11.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3497e-05 2023-07-02T14:03:11.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:11.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299154+0000", 2023-07-02T14:03:11.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4669e-05 2023-07-02T14:03:11.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299169+0000", 2023-07-02T14:03:11.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025630159999999999 2023-07-02T14:03:11.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324799+0000", 2023-07-02T14:03:11.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6953000000000007e-05 2023-07-02T14:03:11.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:11.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324876+0000", 2023-07-02T14:03:11.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016346199999999999 2023-07-02T14:03:11.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:11.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325039+0000", 2023-07-02T14:03:11.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003566624 2023-07-02T14:03:11.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:11.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328606+0000", 2023-07-02T14:03:11.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073295540000000003 2023-07-02T14:03:11.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335936+0000", 2023-07-02T14:03:11.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0044999999999999e-05 2023-07-02T14:03:11.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:11.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335986+0000", 2023-07-02T14:03:11.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8872999999999999e-05 2023-07-02T14:03:11.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:11.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336024+0000", 2023-07-02T14:03:11.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036990838999999998 2023-07-02T14:03:11.712 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.712 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.712 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.713 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:326 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.713 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:11.713 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.367738991000003, 2023-07-02T14:03:11.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036975239999999999, 2023-07-02T14:03:11.713 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:11.713 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.713 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:11.714 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:11.714 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.714 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 326 2023-07-02T14:03:11.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.714 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:11.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:11.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:11.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:11.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:11.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:11.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:11.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:11.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299175+0000", 2023-07-02T14:03:11.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4782000000000003e-05 2023-07-02T14:03:11.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:11.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299250+0000", 2023-07-02T14:03:11.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.277e-05 2023-07-02T14:03:11.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:11.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299262+0000", 2023-07-02T14:03:11.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1460000000000001e-06 2023-07-02T14:03:11.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299268+0000", 2023-07-02T14:03:11.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026158098000000001 2023-07-02T14:03:11.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325427+0000", 2023-07-02T14:03:11.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4845999999999995e-05 2023-07-02T14:03:11.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:11.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325501+0000", 2023-07-02T14:03:11.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015890699999999999 2023-07-02T14:03:11.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:11.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325660+0000", 2023-07-02T14:03:11.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002952114 2023-07-02T14:03:11.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:11.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328612+0000", 2023-07-02T14:03:11.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074491949999999996 2023-07-02T14:03:11.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336062+0000", 2023-07-02T14:03:11.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7376999999999997e-05 2023-07-02T14:03:11.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:11.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336109+0000", 2023-07-02T14:03:11.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1814000000000003e-05 2023-07-02T14:03:11.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:11.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336151+0000", 2023-07-02T14:03:11.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036975239999999999 2023-07-02T14:03:11.722 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.723 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.723 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.723 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:327 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 589824~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.723 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:11.723 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.36763973, 2023-07-02T14:03:11.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037234253000000002, 2023-07-02T14:03:11.724 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:11.724 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.724 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:11.724 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:11.724 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.724 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 327 2023-07-02T14:03:11.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.725 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:11.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:11.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:11.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:11.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:11.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:11.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:11.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:11.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299274+0000", 2023-07-02T14:03:11.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2213999999999997e-05 2023-07-02T14:03:11.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:11.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299346+0000", 2023-07-02T14:03:11.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1816e-05 2023-07-02T14:03:11.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:11.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299358+0000", 2023-07-02T14:03:11.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2626000000000001e-05 2023-07-02T14:03:11.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299371+0000", 2023-07-02T14:03:11.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026916853000000001 2023-07-02T14:03:11.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326288+0000", 2023-07-02T14:03:11.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6909000000000004e-05 2023-07-02T14:03:11.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:11.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326375+0000", 2023-07-02T14:03:11.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015430399999999999 2023-07-02T14:03:11.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:11.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326529+0000", 2023-07-02T14:03:11.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096652100000000005 2023-07-02T14:03:11.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336194+0000", 2023-07-02T14:03:11.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115299 2023-07-02T14:03:11.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:11.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336309+0000", 2023-07-02T14:03:11.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015981200000000001 2023-07-02T14:03:11.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:11.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336469+0000", 2023-07-02T14:03:11.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9867999999999998e-05 2023-07-02T14:03:11.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336509+0000", 2023-07-02T14:03:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037234253000000002 2023-07-02T14:03:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:328 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.367536141000002, 2023-07-02T14:03:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037216610999999997, 2023-07-02T14:03:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 328 2023-07-02T14:03:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:11.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:11.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:11.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:11.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:11.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1412e-05 2023-07-02T14:03:11.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:11.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299449+0000", 2023-07-02T14:03:11.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0219000000000001e-05 2023-07-02T14:03:11.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:11.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299469+0000", 2023-07-02T14:03:11.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9480000000000003e-06 2023-07-02T14:03:11.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299479+0000", 2023-07-02T14:03:11.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027444344999999998 2023-07-02T14:03:11.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326924+0000", 2023-07-02T14:03:11.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5165000000000001e-05 2023-07-02T14:03:11.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:11.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327019+0000", 2023-07-02T14:03:11.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019113499999999999 2023-07-02T14:03:11.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:11.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327210+0000", 2023-07-02T14:03:11.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090197449999999992 2023-07-02T14:03:11.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336230+0000", 2023-07-02T14:03:11.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028764399999999999 2023-07-02T14:03:11.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:11.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336517+0000", 2023-07-02T14:03:11.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1981999999999999e-05 2023-07-02T14:03:11.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:11.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336559+0000", 2023-07-02T14:03:11.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:03:11.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:11.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336595+0000", 2023-07-02T14:03:11.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037216610999999997 2023-07-02T14:03:11.743 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.743 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.743 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.744 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:329 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.744 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:11.744 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.367426223999999, 2023-07-02T14:03:11.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037191142000000003, 2023-07-02T14:03:11.744 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:11.744 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.745 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:11.745 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:11.745 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.745 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 329 2023-07-02T14:03:11.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.745 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:11.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:11.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:11.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:11.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:11.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:11.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:11.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:11.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299487+0000", 2023-07-02T14:03:11.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7871000000000001e-05 2023-07-02T14:03:11.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:11.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299555+0000", 2023-07-02T14:03:11.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0446e-05 2023-07-02T14:03:11.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:11.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299566+0000", 2023-07-02T14:03:11.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7939999999999999e-06 2023-07-02T14:03:11.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299572+0000", 2023-07-02T14:03:11.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028034019 2023-07-02T14:03:11.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327606+0000", 2023-07-02T14:03:11.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1967999999999999e-05 2023-07-02T14:03:11.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:11.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327678+0000", 2023-07-02T14:03:11.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022216900000000001 2023-07-02T14:03:11.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:11.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327900+0000", 2023-07-02T14:03:11.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084016899999999999 2023-07-02T14:03:11.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336301+0000", 2023-07-02T14:03:11.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00030122800000000001 2023-07-02T14:03:11.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:11.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336603+0000", 2023-07-02T14:03:11.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0380000000000003e-05 2023-07-02T14:03:11.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:11.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336643+0000", 2023-07-02T14:03:11.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6455999999999998e-05 2023-07-02T14:03:11.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:11.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336679+0000", 2023-07-02T14:03:11.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037191142000000003 2023-07-02T14:03:11.754 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.754 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.754 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.754 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:330 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.754 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:11.754 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.367335453999999, 2023-07-02T14:03:11.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037849305, 2023-07-02T14:03:11.755 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:11.755 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.755 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:11.755 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:11.755 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.755 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 330 2023-07-02T14:03:11.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.756 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:11.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:11.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:11.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:11.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:11.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:11.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:11.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:11.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299578+0000", 2023-07-02T14:03:11.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4708000000000005e-05 2023-07-02T14:03:11.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:11.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299643+0000", 2023-07-02T14:03:11.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7309999999999999e-06 2023-07-02T14:03:11.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:11.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299652+0000", 2023-07-02T14:03:11.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7989999999999999e-06 2023-07-02T14:03:11.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299658+0000", 2023-07-02T14:03:11.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028981664000000001 2023-07-02T14:03:11.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328639+0000", 2023-07-02T14:03:11.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0939000000000006e-05 2023-07-02T14:03:11.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:11.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328710+0000", 2023-07-02T14:03:11.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000142624 2023-07-02T14:03:11.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:11.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328853+0000", 2023-07-02T14:03:11.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078374640000000006 2023-07-02T14:03:11.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:11.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336690+0000", 2023-07-02T14:03:11.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00068453000000000003 2023-07-02T14:03:11.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337375+0000", 2023-07-02T14:03:11.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0667999999999998e-05 2023-07-02T14:03:11.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:11.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337406+0000", 2023-07-02T14:03:11.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2858999999999999e-05 2023-07-02T14:03:11.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:11.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337428+0000", 2023-07-02T14:03:11.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037849305 2023-07-02T14:03:11.764 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.764 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.764 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.765 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:331 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.765 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:11.765 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.367251592000002, 2023-07-02T14:03:11.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037870332, 2023-07-02T14:03:11.765 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:11.765 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.765 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:11.765 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:11.766 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.766 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 331 2023-07-02T14:03:11.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.766 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:11.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:11.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:11.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:11.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:11.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:11.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:11.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:11.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299662+0000", 2023-07-02T14:03:11.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2252e-05 2023-07-02T14:03:11.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:11.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299725+0000", 2023-07-02T14:03:11.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8440000000000004e-06 2023-07-02T14:03:11.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:11.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299733+0000", 2023-07-02T14:03:11.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6759999999999998e-06 2023-07-02T14:03:11.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299738+0000", 2023-07-02T14:03:11.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029459387 2023-07-02T14:03:11.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329197+0000", 2023-07-02T14:03:11.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0087999999999997e-05 2023-07-02T14:03:11.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:11.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329268+0000", 2023-07-02T14:03:11.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014405700000000001 2023-07-02T14:03:11.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:11.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329412+0000", 2023-07-02T14:03:11.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072842100000000002 2023-07-02T14:03:11.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:11.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336696+0000", 2023-07-02T14:03:11.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00076275300000000002 2023-07-02T14:03:11.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337459+0000", 2023-07-02T14:03:11.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5988000000000002e-05 2023-07-02T14:03:11.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:11.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337495+0000", 2023-07-02T14:03:11.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8686000000000002e-05 2023-07-02T14:03:11.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:11.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337533+0000", 2023-07-02T14:03:11.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037870332 2023-07-02T14:03:11.774 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.774 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.775 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.775 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:332 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.775 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:11.775 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.367171667999997, 2023-07-02T14:03:11.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037892506999999999, 2023-07-02T14:03:11.775 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:11.776 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.776 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:11.776 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:11.776 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.776 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 332 2023-07-02T14:03:11.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.776 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:11.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:11.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:11.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:11.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:11.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:11.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:11.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:11.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299742+0000", 2023-07-02T14:03:11.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1366999999999997e-05 2023-07-02T14:03:11.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:11.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299804+0000", 2023-07-02T14:03:11.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9319999999999996e-06 2023-07-02T14:03:11.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:11.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299812+0000", 2023-07-02T14:03:11.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4359999999999999e-06 2023-07-02T14:03:11.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299818+0000", 2023-07-02T14:03:11.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029946144000000001 2023-07-02T14:03:11.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329764+0000", 2023-07-02T14:03:11.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000164828 2023-07-02T14:03:11.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:11.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329929+0000", 2023-07-02T14:03:11.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016139400000000001 2023-07-02T14:03:11.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:11.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330090+0000", 2023-07-02T14:03:11.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066134009999999997 2023-07-02T14:03:11.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:11.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336704+0000", 2023-07-02T14:03:11.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00086955400000000003 2023-07-02T14:03:11.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337573+0000", 2023-07-02T14:03:11.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8447000000000001e-05 2023-07-02T14:03:11.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:11.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337602+0000", 2023-07-02T14:03:11.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.362e-05 2023-07-02T14:03:11.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:11.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337635+0000", 2023-07-02T14:03:11.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037892506999999999 2023-07-02T14:03:11.785 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.785 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.785 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.785 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:333 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.785 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:11.786 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.367091586999997, 2023-07-02T14:03:11.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037932598999999997, 2023-07-02T14:03:11.786 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:11.786 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.786 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:11.786 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:11.786 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.786 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 333 2023-07-02T14:03:11.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.787 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:11.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:11.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:11.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:11.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:11.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:11.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:11.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:11.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299822+0000", 2023-07-02T14:03:11.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0831000000000001e-05 2023-07-02T14:03:11.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:11.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299883+0000", 2023-07-02T14:03:11.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.322e-06 2023-07-02T14:03:11.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:11.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299888+0000", 2023-07-02T14:03:11.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3479999999999997e-06 2023-07-02T14:03:11.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299893+0000", 2023-07-02T14:03:11.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030831905999999999 2023-07-02T14:03:11.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330725+0000", 2023-07-02T14:03:11.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3560999999999996e-05 2023-07-02T14:03:11.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:11.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330798+0000", 2023-07-02T14:03:11.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015165999999999999 2023-07-02T14:03:11.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:11.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330950+0000", 2023-07-02T14:03:11.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057590319999999999 2023-07-02T14:03:11.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:11.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336709+0000", 2023-07-02T14:03:11.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00096987899999999999 2023-07-02T14:03:11.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337679+0000", 2023-07-02T14:03:11.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9699000000000001e-05 2023-07-02T14:03:11.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:11.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337719+0000", 2023-07-02T14:03:11.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6961000000000001e-05 2023-07-02T14:03:11.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:11.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337756+0000", 2023-07-02T14:03:11.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037932598999999997 2023-07-02T14:03:11.795 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.795 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.795 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.796 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:372 2.0 2:169af44a:::rbd_data.125de1449605.0000000000000106:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.796 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:03:11.796 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.363514262000002, 2023-07-02T14:03:11.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036066963, 2023-07-02T14:03:11.796 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:11.796 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.796 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:11.797 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:11.797 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.797 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 372 2023-07-02T14:03:11.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.797 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:11.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:11.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:03:11.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:11.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:11.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:03:11.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:11.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:11.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:03:11.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1002000000000003e-05 2023-07-02T14:03:11.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:11.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303461+0000", 2023-07-02T14:03:11.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9306e-05 2023-07-02T14:03:11.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:11.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303490+0000", 2023-07-02T14:03:11.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5026e-05 2023-07-02T14:03:11.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303505+0000", 2023-07-02T14:03:11.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033623129000000002 2023-07-02T14:03:11.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337128+0000", 2023-07-02T14:03:11.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6968000000000002e-05 2023-07-02T14:03:11.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:11.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337175+0000", 2023-07-02T14:03:11.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000101127 2023-07-02T14:03:11.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337276+0000", 2023-07-02T14:03:11.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017866920000000001 2023-07-02T14:03:11.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:11.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339063+0000", 2023-07-02T14:03:11.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00031459800000000003 2023-07-02T14:03:11.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339377+0000", 2023-07-02T14:03:11.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7145999999999999e-05 2023-07-02T14:03:11.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:11.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339425+0000", 2023-07-02T14:03:11.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2632000000000001e-05 2023-07-02T14:03:11.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:11.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339467+0000", 2023-07-02T14:03:11.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036066963 2023-07-02T14:03:11.805 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.806 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.806 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.806 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:416 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.806 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:11.806 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.348798207999998, 2023-07-02T14:03:11.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039242025, 2023-07-02T14:03:11.807 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:11.807 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.807 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:11.807 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:11.807 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.807 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:03:11.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.808 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:11.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:11.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:11.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:11.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:11.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:11.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:11.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:11.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:11.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.692e-06 2023-07-02T14:03:11.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:11.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318119+0000", 2023-07-02T14:03:11.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9790000000000005e-06 2023-07-02T14:03:11.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:11.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318129+0000", 2023-07-02T14:03:11.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3640000000000001e-06 2023-07-02T14:03:11.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318137+0000", 2023-07-02T14:03:11.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022607808 2023-07-02T14:03:11.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340745+0000", 2023-07-02T14:03:11.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0130999999999996e-05 2023-07-02T14:03:11.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340825+0000", 2023-07-02T14:03:11.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039581080000000001 2023-07-02T14:03:11.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344783+0000", 2023-07-02T14:03:11.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000006e-05 2023-07-02T14:03:11.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344854+0000", 2023-07-02T14:03:11.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052105550000000004 2023-07-02T14:03:11.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350065+0000", 2023-07-02T14:03:11.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0321999999999997e-05 2023-07-02T14:03:11.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350115+0000", 2023-07-02T14:03:11.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012378409999999999 2023-07-02T14:03:11.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351353+0000", 2023-07-02T14:03:11.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5942000000000001e-05 2023-07-02T14:03:11.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:11.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351409+0000", 2023-07-02T14:03:11.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035677700000000002 2023-07-02T14:03:11.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351766+0000", 2023-07-02T14:03:11.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000926719 2023-07-02T14:03:11.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:11.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352693+0000", 2023-07-02T14:03:11.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045652080000000003 2023-07-02T14:03:11.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357258+0000", 2023-07-02T14:03:11.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4717e-05 2023-07-02T14:03:11.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:11.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357293+0000", 2023-07-02T14:03:11.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5807000000000002e-05 2023-07-02T14:03:11.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:11.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357358+0000", 2023-07-02T14:03:11.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039242025 2023-07-02T14:03:11.820 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.820 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.820 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.820 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:417 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.820 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:11.821 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.348772771, 2023-07-02T14:03:11.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042407105, 2023-07-02T14:03:11.821 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:11.821 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.821 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:11.821 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:11.821 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.821 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 417 2023-07-02T14:03:11.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.822 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:11.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:11.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:11.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:11.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:11.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:11.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:11.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:11.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318141+0000", 2023-07-02T14:03:11.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7120000000000002e-06 2023-07-02T14:03:11.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:11.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318145+0000", 2023-07-02T14:03:11.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8869999999999997e-06 2023-07-02T14:03:11.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:11.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318151+0000", 2023-07-02T14:03:11.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9820000000000002e-06 2023-07-02T14:03:11.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318155+0000", 2023-07-02T14:03:11.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699133 2023-07-02T14:03:11.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340854+0000", 2023-07-02T14:03:11.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6849999999999998e-05 2023-07-02T14:03:11.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340931+0000", 2023-07-02T14:03:11.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039672140000000002 2023-07-02T14:03:11.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344898+0000", 2023-07-02T14:03:11.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5341999999999998e-05 2023-07-02T14:03:11.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344973+0000", 2023-07-02T14:03:11.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051819400000000003 2023-07-02T14:03:11.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350155+0000", 2023-07-02T14:03:11.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3261999999999999e-05 2023-07-02T14:03:11.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350209+0000", 2023-07-02T14:03:11.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00194097 2023-07-02T14:03:11.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352150+0000", 2023-07-02T14:03:11.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6851e-05 2023-07-02T14:03:11.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352226+0000", 2023-07-02T14:03:11.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051831109999999998 2023-07-02T14:03:11.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357409+0000", 2023-07-02T14:03:11.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5325e-05 2023-07-02T14:03:11.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:11.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357495+0000", 2023-07-02T14:03:11.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042107899999999998 2023-07-02T14:03:11.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357916+0000", 2023-07-02T14:03:11.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0011672449999999999 2023-07-02T14:03:11.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:11.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.359083+0000", 2023-07-02T14:03:11.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013534930000000001 2023-07-02T14:03:11.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360437+0000", 2023-07-02T14:03:11.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2159000000000002e-05 2023-07-02T14:03:11.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:11.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360469+0000", 2023-07-02T14:03:11.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0092000000000005e-05 2023-07-02T14:03:11.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:11.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360549+0000", 2023-07-02T14:03:11.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042407105 2023-07-02T14:03:11.838 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.838 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.838 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.839 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:418 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.839 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:11.839 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.348753530000003, 2023-07-02T14:03:11.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.045613663999999998, 2023-07-02T14:03:11.839 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:11.839 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.839 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:11.839 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:11.840 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.840 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 418 2023-07-02T14:03:11.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.840 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:11.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:11.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:11.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:11.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:11.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:11.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:11.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:11.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318160+0000", 2023-07-02T14:03:11.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6490000000000001e-06 2023-07-02T14:03:11.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:11.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318164+0000", 2023-07-02T14:03:11.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:03:11.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:11.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318170+0000", 2023-07-02T14:03:11.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5279999999999999e-06 2023-07-02T14:03:11.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318173+0000", 2023-07-02T14:03:11.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022788856 2023-07-02T14:03:11.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340962+0000", 2023-07-02T14:03:11.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7730000000000003e-05 2023-07-02T14:03:11.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341040+0000", 2023-07-02T14:03:11.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039795899999999999 2023-07-02T14:03:11.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345020+0000", 2023-07-02T14:03:11.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4962000000000006e-05 2023-07-02T14:03:11.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345085+0000", 2023-07-02T14:03:11.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051955430000000004 2023-07-02T14:03:11.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350280+0000", 2023-07-02T14:03:11.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8913999999999995e-05 2023-07-02T14:03:11.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350379+0000", 2023-07-02T14:03:11.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001887344 2023-07-02T14:03:11.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352266+0000", 2023-07-02T14:03:11.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7865000000000003e-05 2023-07-02T14:03:11.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352314+0000", 2023-07-02T14:03:11.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060621190000000004 2023-07-02T14:03:11.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358376+0000", 2023-07-02T14:03:11.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2448999999999997e-05 2023-07-02T14:03:11.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358449+0000", 2023-07-02T14:03:11.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0021539929999999999 2023-07-02T14:03:11.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360603+0000", 2023-07-02T14:03:11.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7859000000000002e-05 2023-07-02T14:03:11.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:11.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360671+0000", 2023-07-02T14:03:11.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026930699999999999 2023-07-02T14:03:11.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360940+0000", 2023-07-02T14:03:11.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00081179699999999995 2023-07-02T14:03:11.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:11.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361752+0000", 2023-07-02T14:03:11.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0019350490000000001 2023-07-02T14:03:11.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363687+0000", 2023-07-02T14:03:11.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4462000000000003e-05 2023-07-02T14:03:11.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:11.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363721+0000", 2023-07-02T14:03:11.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3408000000000001e-05 2023-07-02T14:03:11.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:11.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363775+0000", 2023-07-02T14:03:11.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.045613663999999998 2023-07-02T14:03:11.855 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.855 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.855 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.855 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:419 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.855 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:11.856 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.348737202999999, 2023-07-02T14:03:11.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.047296703000000002, 2023-07-02T14:03:11.856 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:11.856 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.856 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:11.856 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:11.856 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.856 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 419 2023-07-02T14:03:11.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.857 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:11.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:11.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:11.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:11.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:11.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:11.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:11.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:11.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:11.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.433e-06 2023-07-02T14:03:11.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:11.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318180+0000", 2023-07-02T14:03:11.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9950000000000002e-06 2023-07-02T14:03:11.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:11.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318186+0000", 2023-07-02T14:03:11.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7290000000000002e-06 2023-07-02T14:03:11.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318190+0000", 2023-07-02T14:03:11.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022877989000000001 2023-07-02T14:03:11.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341068+0000", 2023-07-02T14:03:11.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5803999999999994e-05 2023-07-02T14:03:11.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341154+0000", 2023-07-02T14:03:11.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039633000000000003 2023-07-02T14:03:11.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345117+0000", 2023-07-02T14:03:11.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7188999999999996e-05 2023-07-02T14:03:11.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345194+0000", 2023-07-02T14:03:11.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052306059999999996 2023-07-02T14:03:11.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350425+0000", 2023-07-02T14:03:11.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8093999999999997e-05 2023-07-02T14:03:11.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350473+0000", 2023-07-02T14:03:11.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001865158 2023-07-02T14:03:11.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352338+0000", 2023-07-02T14:03:11.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5479000000000003e-05 2023-07-02T14:03:11.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352384+0000", 2023-07-02T14:03:11.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061153290000000001 2023-07-02T14:03:11.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358499+0000", 2023-07-02T14:03:11.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5156999999999999e-05 2023-07-02T14:03:11.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358574+0000", 2023-07-02T14:03:11.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027847459999999998 2023-07-02T14:03:11.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361359+0000", 2023-07-02T14:03:11.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2633999999999999e-05 2023-07-02T14:03:11.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361402+0000", 2023-07-02T14:03:11.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024171510000000002 2023-07-02T14:03:11.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363819+0000", 2023-07-02T14:03:11.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6988999999999999e-05 2023-07-02T14:03:11.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:11.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363876+0000", 2023-07-02T14:03:11.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035321300000000001 2023-07-02T14:03:11.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.364229+0000", 2023-07-02T14:03:11.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00094674799999999997 2023-07-02T14:03:11.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:11.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365176+0000", 2023-07-02T14:03:11.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019724200000000001 2023-07-02T14:03:11.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365373+0000", 2023-07-02T14:03:11.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2116e-05 2023-07-02T14:03:11.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:11.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365425+0000", 2023-07-02T14:03:11.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8996999999999999e-05 2023-07-02T14:03:11.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:11.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365474+0000", 2023-07-02T14:03:11.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.047296703000000002 2023-07-02T14:03:11.873 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.873 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.873 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.873 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:472 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.873 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:11.874 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.289280970999997, 2023-07-02T14:03:11.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036204721000000002, 2023-07-02T14:03:11.874 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:11.874 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.874 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:11.874 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:11.874 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.875 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 472 2023-07-02T14:03:11.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.875 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:11.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:11.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:11.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:11.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:11.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:11.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:03:11.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:11.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377633+0000", 2023-07-02T14:03:11.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.815e-06 2023-07-02T14:03:11.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:11.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377635+0000", 2023-07-02T14:03:11.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9459999999999998e-06 2023-07-02T14:03:11.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:11.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377638+0000", 2023-07-02T14:03:11.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.673e-06 2023-07-02T14:03:11.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377643+0000", 2023-07-02T14:03:11.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043004239999999997 2023-07-02T14:03:11.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.381943+0000", 2023-07-02T14:03:11.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4321999999999999e-05 2023-07-02T14:03:11.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.381998+0000", 2023-07-02T14:03:11.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003008377 2023-07-02T14:03:11.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385006+0000", 2023-07-02T14:03:11.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8751999999999998e-05 2023-07-02T14:03:11.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385045+0000", 2023-07-02T14:03:11.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086260880000000005 2023-07-02T14:03:11.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393671+0000", 2023-07-02T14:03:11.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6365999999999997e-05 2023-07-02T14:03:11.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393707+0000", 2023-07-02T14:03:11.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028560289999999999 2023-07-02T14:03:11.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396563+0000", 2023-07-02T14:03:11.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6195e-05 2023-07-02T14:03:11.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396619+0000", 2023-07-02T14:03:11.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023118269999999998 2023-07-02T14:03:11.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.398931+0000", 2023-07-02T14:03:11.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3885999999999999e-05 2023-07-02T14:03:11.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.398985+0000", 2023-07-02T14:03:11.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043756890000000003 2023-07-02T14:03:11.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403361+0000", 2023-07-02T14:03:11.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0043000000000001e-05 2023-07-02T14:03:11.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403421+0000", 2023-07-02T14:03:11.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001590972 2023-07-02T14:03:11.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405012+0000", 2023-07-02T14:03:11.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0067000000000005e-05 2023-07-02T14:03:11.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405082+0000", 2023-07-02T14:03:11.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050958610000000001 2023-07-02T14:03:11.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410178+0000", 2023-07-02T14:03:11.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7070000000000004e-05 2023-07-02T14:03:11.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410275+0000", 2023-07-02T14:03:11.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060121100000000004 2023-07-02T14:03:11.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410876+0000", 2023-07-02T14:03:11.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3177999999999999e-05 2023-07-02T14:03:11.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:11.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410919+0000", 2023-07-02T14:03:11.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000271394 2023-07-02T14:03:11.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411191+0000", 2023-07-02T14:03:11.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001009598 2023-07-02T14:03:11.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.412200+0000", 2023-07-02T14:03:11.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014789790000000001 2023-07-02T14:03:11.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:11.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413679+0000", 2023-07-02T14:03:11.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2624000000000002e-05 2023-07-02T14:03:11.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:11.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413742+0000", 2023-07-02T14:03:11.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6564999999999994e-05 2023-07-02T14:03:11.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.893 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:11.893 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413838+0000", 2023-07-02T14:03:11.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036204721000000002 2023-07-02T14:03:11.893 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.893 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.894 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.894 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:473 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.894 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:11.894 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.289269560000001, 2023-07-02T14:03:11.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037915904, 2023-07-02T14:03:11.894 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:11.895 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.895 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:11.895 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:11.895 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.895 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 473 2023-07-02T14:03:11.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.895 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:11.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:11.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:11.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:11.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:11.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:11.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:03:11.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:11.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:11.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.119e-06 2023-07-02T14:03:11.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:11.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377647+0000", 2023-07-02T14:03:11.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.875e-06 2023-07-02T14:03:11.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:11.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377650+0000", 2023-07-02T14:03:11.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1949999999999999e-06 2023-07-02T14:03:11.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377653+0000", 2023-07-02T14:03:11.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004375444 2023-07-02T14:03:11.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382028+0000", 2023-07-02T14:03:11.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1063999999999997e-05 2023-07-02T14:03:11.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382089+0000", 2023-07-02T14:03:11.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030043869999999999 2023-07-02T14:03:11.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385094+0000", 2023-07-02T14:03:11.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8167999999999997e-05 2023-07-02T14:03:11.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385142+0000", 2023-07-02T14:03:11.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008603421 2023-07-02T14:03:11.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393745+0000", 2023-07-02T14:03:11.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3763000000000005e-05 2023-07-02T14:03:11.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393819+0000", 2023-07-02T14:03:11.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028274720000000001 2023-07-02T14:03:11.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396647+0000", 2023-07-02T14:03:11.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7929999999999997e-05 2023-07-02T14:03:11.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396695+0000", 2023-07-02T14:03:11.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023426459999999999 2023-07-02T14:03:11.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399037+0000", 2023-07-02T14:03:11.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5846000000000004e-05 2023-07-02T14:03:11.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399123+0000", 2023-07-02T14:03:11.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043448360000000004 2023-07-02T14:03:11.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403468+0000", 2023-07-02T14:03:11.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6699000000000001e-05 2023-07-02T14:03:11.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403515+0000", 2023-07-02T14:03:11.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001622943 2023-07-02T14:03:11.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405138+0000", 2023-07-02T14:03:11.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6382000000000002e-05 2023-07-02T14:03:11.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405204+0000", 2023-07-02T14:03:11.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051089910000000002 2023-07-02T14:03:11.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410313+0000", 2023-07-02T14:03:11.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8106000000000003e-05 2023-07-02T14:03:11.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410361+0000", 2023-07-02T14:03:11.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012200399999999999 2023-07-02T14:03:11.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411581+0000", 2023-07-02T14:03:11.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5067000000000005e-05 2023-07-02T14:03:11.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411656+0000", 2023-07-02T14:03:11.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022353249999999998 2023-07-02T14:03:11.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413891+0000", 2023-07-02T14:03:11.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9356000000000003e-05 2023-07-02T14:03:11.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:11.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413981+0000", 2023-07-02T14:03:11.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00043818299999999998 2023-07-02T14:03:11.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.414419+0000", 2023-07-02T14:03:11.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00081559199999999997 2023-07-02T14:03:11.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:11.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415235+0000", 2023-07-02T14:03:11.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026434499999999998 2023-07-02T14:03:11.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415499+0000", 2023-07-02T14:03:11.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9779999999999999e-05 2023-07-02T14:03:11.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:11.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415529+0000", 2023-07-02T14:03:11.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2159999999999997e-05 2023-07-02T14:03:11.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:11.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415561+0000", 2023-07-02T14:03:11.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037915904 2023-07-02T14:03:11.915 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.915 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.915 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.916 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:475 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:11.916 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:11.916 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.289259520999998, 2023-07-02T14:03:11.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039286962000000002, 2023-07-02T14:03:11.916 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:11.916 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:11.917 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:11.917 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:11.917 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:11.917 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 475 2023-07-02T14:03:11.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.917 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:11.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:11.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:11.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:11.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:11.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:11.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:11.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:11.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:11.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4779999999999997e-06 2023-07-02T14:03:11.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:11.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377664+0000", 2023-07-02T14:03:11.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.268e-06 2023-07-02T14:03:11.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:11.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377671+0000", 2023-07-02T14:03:11.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.861e-06 2023-07-02T14:03:11.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:11.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377674+0000", 2023-07-02T14:03:11.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044395789999999999 2023-07-02T14:03:11.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382114+0000", 2023-07-02T14:03:11.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8096999999999997e-05 2023-07-02T14:03:11.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.922 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382162+0000", 2023-07-02T14:03:11.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029949059999999999 2023-07-02T14:03:11.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385157+0000", 2023-07-02T14:03:11.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1158000000000001e-05 2023-07-02T14:03:11.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385198+0000", 2023-07-02T14:03:11.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086477229999999995 2023-07-02T14:03:11.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393846+0000", 2023-07-02T14:03:11.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5990999999999999e-05 2023-07-02T14:03:11.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393912+0000", 2023-07-02T14:03:11.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027979599999999999 2023-07-02T14:03:11.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396710+0000", 2023-07-02T14:03:11.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5911000000000002e-05 2023-07-02T14:03:11.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396746+0000", 2023-07-02T14:03:11.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024039759999999999 2023-07-02T14:03:11.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399150+0000", 2023-07-02T14:03:11.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7834999999999999e-05 2023-07-02T14:03:11.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399188+0000", 2023-07-02T14:03:11.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004364054 2023-07-02T14:03:11.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403552+0000", 2023-07-02T14:03:11.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6274999999999999e-05 2023-07-02T14:03:11.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403608+0000", 2023-07-02T14:03:11.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0016261540000000001 2023-07-02T14:03:11.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405234+0000", 2023-07-02T14:03:11.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6945000000000004e-05 2023-07-02T14:03:11.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405301+0000", 2023-07-02T14:03:11.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050871329999999998 2023-07-02T14:03:11.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410388+0000", 2023-07-02T14:03:11.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0210999999999995e-05 2023-07-02T14:03:11.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410468+0000", 2023-07-02T14:03:11.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001227454 2023-07-02T14:03:11.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411696+0000", 2023-07-02T14:03:11.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8188000000000005e-05 2023-07-02T14:03:11.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411764+0000", 2023-07-02T14:03:11.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031779930000000001 2023-07-02T14:03:11.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.414942+0000", 2023-07-02T14:03:11.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7236000000000004e-05 2023-07-02T14:03:11.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:11.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415019+0000", 2023-07-02T14:03:11.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000573045 2023-07-02T14:03:11.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:11.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415592+0000", 2023-07-02T14:03:11.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7580000000000002e-05 2023-07-02T14:03:11.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:11.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415640+0000", 2023-07-02T14:03:11.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024783200000000002 2023-07-02T14:03:11.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:11.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415888+0000", 2023-07-02T14:03:11.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00080440999999999998 2023-07-02T14:03:11.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:11.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416692+0000", 2023-07-02T14:03:11.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000195787 2023-07-02T14:03:11.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:11.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416888+0000", 2023-07-02T14:03:11.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0787999999999997e-05 2023-07-02T14:03:11.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:11.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416919+0000", 2023-07-02T14:03:11.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.338e-05 2023-07-02T14:03:11.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:11.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:11.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:11.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416942+0000", 2023-07-02T14:03:11.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039286962000000002 2023-07-02T14:03:11.938 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.938 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.939 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.939 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:11.939 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:11.939 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:11.939 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:03:11.941 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:11.939+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:11.941 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:03:12.041 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:12.040+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:12.095 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:12.095 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:12.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:12.096 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.7 62/60 [PushOp(2:f30916f5:::rbd_data.125de1449605.0000000000000075:head, version: 58'249, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:f30916f5:::rbd_data.125de1449605.0000000000000075:head@58'249, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-07-02T14:03:12.096 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:12.091624+0000", 2023-07-02T14:03:12.096 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0028731719999999998, 2023-07-02T14:03:12.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002929891, 2023-07-02T14:03:12.097 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:12.097 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:12.097 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:12.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:12.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:12.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.091624+0000", 2023-07-02T14:03:12.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:12.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:12.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:12.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:12.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.091624+0000", 2023-07-02T14:03:12.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:12.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:12.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:12.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:12.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.091624+0000", 2023-07-02T14:03:12.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024253930000000001 2023-07-02T14:03:12.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:12.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:12.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:12.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094049+0000", 2023-07-02T14:03:12.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1549999999999999e-05 2023-07-02T14:03:12.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:12.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:12.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:12.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094071+0000", 2023-07-02T14:03:12.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5319999999999995e-06 2023-07-02T14:03:12.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:12.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:12.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:12.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094080+0000", 2023-07-02T14:03:12.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1773999999999993e-05 2023-07-02T14:03:12.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:12.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:12.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:12.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094162+0000", 2023-07-02T14:03:12.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5380000000000008e-06 2023-07-02T14:03:12.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:12.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:12.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:12.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094172+0000", 2023-07-02T14:03:12.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025471700000000001 2023-07-02T14:03:12.107 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:12.108 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:12.108 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:12.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:12.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:12.109 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:3046 2.0 e62/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 62'696, mlcod=62'694)", 2023-07-02T14:03:12.109 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:12.094097+0000", 2023-07-02T14:03:12.110 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.00040014599999999999, 2023-07-02T14:03:12.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00053592099999999997, 2023-07-02T14:03:12.110 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:12.110 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:12.111 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:12.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:12.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:12.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094097+0000", 2023-07-02T14:03:12.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:12.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:12.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:12.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:12.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094097+0000", 2023-07-02T14:03:12.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:12.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:12.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:12.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:12.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094097+0000", 2023-07-02T14:03:12.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6799999999999998e-06 2023-07-02T14:03:12.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:12.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:12.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:12.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094102+0000", 2023-07-02T14:03:12.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2429999999999994e-06 2023-07-02T14:03:12.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094112+0000", 2023-07-02T14:03:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3230000000000002e-06 2023-07-02T14:03:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094115+0000", 2023-07-02T14:03:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0043999999999997e-05 2023-07-02T14:03:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094165+0000", 2023-07-02T14:03:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1022e-05 2023-07-02T14:03:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094186+0000", 2023-07-02T14:03:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8572999999999994e-05 2023-07-02T14:03:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:12.120 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:12.120 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:12.120 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:12.120 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2023-07-02T14:03:12.120 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 2 2023-07-02T14:03:12.120 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:12.121 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:03:12.129 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:12.158 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:12.165 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:12.164+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:12.183 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:12.211 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:12.240 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:12.243 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:12.242+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:12.261 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:12.262 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:03:12.262 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:03:12.263 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:03:12.263 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:03:12.263 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:03:12.263 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:03:12.291 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:12.291 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:12.306 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:03:12.343 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:12.343+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:12.444 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:12.443+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:12.470 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:12.470 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:12.480 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:03:12.545 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:12.545+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:12.644 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:12.645 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:12.645 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:12.645+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:12.654 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:03:12.747 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:12.746+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:12.818 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:12.818 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:12.828 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:03:12.848 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:12.847+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:12.949 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:12.948+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:12.991 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:12.992 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:13.001 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:03:13.050 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:13.050+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:13.145 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:13.145 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:13.151 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:13.150+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:13.155 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:03:13.252 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:13.251+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:13.319 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:13.319 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:13.330 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:03:13.352 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:03:13.353 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:13.353+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:13.454 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:13.454+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:13.497 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:13.497 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:13.508 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:03:13.555 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:13.555+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:13.632 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:13.632 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:13.642 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:03:13.686 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:13.685+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:13.756 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:13.755+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:13.765 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:13.766 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:13.775 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:03:13.860 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:13.859+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:13.941 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:13.941 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:13.951 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:03:13.958 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:13.958+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:14.058 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:14.058+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:14.073 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:03:14.156 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:14.157 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:14.159 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:14.158+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:14.174 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:03:14.260 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:14.259+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:14.277 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:03:14.341 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:14.342 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:14.352 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:03:14.352 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2023-07-02T14:03:14.353 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:03:14.353 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2023-07-02T14:03:14.353 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-07-02T14:03:14.353 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-07-02T14:03:14.354 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2023-07-02T14:03:14.361 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:14.361+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:14.419 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:14.437 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:14.456 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:14.461 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:14.460+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:14.475 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:14.485 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:14.485 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:03:14.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:03:14.485 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:14.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.486 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:183 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:14.486 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:14.486 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.197183111999998, 2023-07-02T14:03:14.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003, 2023-07-02T14:03:14.486 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:14.486 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:14.487 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:14.487 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:14.487 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:14.487 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 183 2023-07-02T14:03:14.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.487 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:14.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:14.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:14.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:14.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:14.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:14.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:14.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:14.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:14.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9790999999999997e-05 2023-07-02T14:03:14.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:14.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285598+0000", 2023-07-02T14:03:14.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5809999999999996e-06 2023-07-02T14:03:14.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:14.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285604+0000", 2023-07-02T14:03:14.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7599999999999998e-06 2023-07-02T14:03:14.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:14.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285607+0000", 2023-07-02T14:03:14.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021108188999999999 2023-07-02T14:03:14.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306715+0000", 2023-07-02T14:03:14.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6548e-05 2023-07-02T14:03:14.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:14.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306792+0000", 2023-07-02T14:03:14.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173869 2023-07-02T14:03:14.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:14.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306965+0000", 2023-07-02T14:03:14.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060498009999999996 2023-07-02T14:03:14.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:14.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313015+0000", 2023-07-02T14:03:14.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049443719999999998 2023-07-02T14:03:14.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:14.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317960+0000", 2023-07-02T14:03:14.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5927999999999998e-05 2023-07-02T14:03:14.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:14.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318006+0000", 2023-07-02T14:03:14.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6004999999999998e-05 2023-07-02T14:03:14.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:14.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318042+0000", 2023-07-02T14:03:14.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003 2023-07-02T14:03:14.497 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.497 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.497 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.497 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:186 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:14.497 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:14.498 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.197112285999999, 2023-07-02T14:03:14.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003, 2023-07-02T14:03:14.498 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:14.498 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:14.498 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:14.498 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:14.498 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:14.499 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 186 2023-07-02T14:03:14.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.499 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:14.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:14.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:14.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:14.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:14.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:14.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:14.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:14.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:14.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4279e-05 2023-07-02T14:03:14.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:14.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285664+0000", 2023-07-02T14:03:14.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3789999999999995e-06 2023-07-02T14:03:14.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:14.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285673+0000", 2023-07-02T14:03:14.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499999999999998e-06 2023-07-02T14:03:14.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:14.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285678+0000", 2023-07-02T14:03:14.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023261947000000002 2023-07-02T14:03:14.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.308940+0000", 2023-07-02T14:03:14.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010323399999999999 2023-07-02T14:03:14.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:14.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309044+0000", 2023-07-02T14:03:14.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018878300000000001 2023-07-02T14:03:14.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:14.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309232+0000", 2023-07-02T14:03:14.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037874150000000001 2023-07-02T14:03:14.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:14.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313020+0000", 2023-07-02T14:03:14.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050976140000000003 2023-07-02T14:03:14.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:14.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318117+0000", 2023-07-02T14:03:14.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3182000000000001e-05 2023-07-02T14:03:14.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:14.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:14.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5055999999999998e-05 2023-07-02T14:03:14.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:14.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318196+0000", 2023-07-02T14:03:14.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003 2023-07-02T14:03:14.508 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.508 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.508 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.509 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:189 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:14.509 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:03:14.509 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.197038523000003, 2023-07-02T14:03:14.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997, 2023-07-02T14:03:14.509 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:14.509 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:14.509 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:14.510 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:14.510 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:14.510 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:03:14.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.510 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:14.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:14.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:03:14.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:14.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:14.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:03:14.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:14.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:14.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285683+0000", 2023-07-02T14:03:14.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8451999999999999e-05 2023-07-02T14:03:14.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:14.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285782+0000", 2023-07-02T14:03:14.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3650000000000004e-06 2023-07-02T14:03:14.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:14.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285790+0000", 2023-07-02T14:03:14.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8230000000000002e-06 2023-07-02T14:03:14.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:14.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285795+0000", 2023-07-02T14:03:14.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024619367999999999 2023-07-02T14:03:14.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310414+0000", 2023-07-02T14:03:14.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014244600000000001 2023-07-02T14:03:14.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:14.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310557+0000", 2023-07-02T14:03:14.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020612699999999999 2023-07-02T14:03:14.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:14.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310763+0000", 2023-07-02T14:03:14.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003427687 2023-07-02T14:03:14.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:14.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314191+0000", 2023-07-02T14:03:14.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040492089999999998 2023-07-02T14:03:14.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:14.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318240+0000", 2023-07-02T14:03:14.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0287999999999999e-05 2023-07-02T14:03:14.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:14.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318270+0000", 2023-07-02T14:03:14.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0868000000000002e-05 2023-07-02T14:03:14.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:14.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318311+0000", 2023-07-02T14:03:14.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997 2023-07-02T14:03:14.519 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.519 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.520 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.520 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:192 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:14.520 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:03:14.520 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.196921750000001, 2023-07-02T14:03:14.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003, 2023-07-02T14:03:14.521 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:14.521 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:14.521 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:14.521 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:14.521 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:14.521 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 192 2023-07-02T14:03:14.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.522 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:14.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:14.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:03:14.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:14.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:14.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:03:14.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:14.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:14.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285800+0000", 2023-07-02T14:03:14.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100417 2023-07-02T14:03:14.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:14.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285900+0000", 2023-07-02T14:03:14.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8244999999999999e-05 2023-07-02T14:03:14.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:14.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285919+0000", 2023-07-02T14:03:14.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2499999999999997e-06 2023-07-02T14:03:14.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:14.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285924+0000", 2023-07-02T14:03:14.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025288155999999999 2023-07-02T14:03:14.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311212+0000", 2023-07-02T14:03:14.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4073999999999996e-05 2023-07-02T14:03:14.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:14.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311286+0000", 2023-07-02T14:03:14.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016219699999999999 2023-07-02T14:03:14.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:14.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311448+0000", 2023-07-02T14:03:14.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066020599999999999 2023-07-02T14:03:14.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:14.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318050+0000", 2023-07-02T14:03:14.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000327986 2023-07-02T14:03:14.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:14.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318378+0000", 2023-07-02T14:03:14.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8693999999999998e-05 2023-07-02T14:03:14.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:14.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318427+0000", 2023-07-02T14:03:14.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6656999999999998e-05 2023-07-02T14:03:14.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:14.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318464+0000", 2023-07-02T14:03:14.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003 2023-07-02T14:03:14.531 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.531 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.531 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.531 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:195 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:14.531 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:03:14.532 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.196790245999999, 2023-07-02T14:03:14.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001, 2023-07-02T14:03:14.532 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:14.532 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:14.532 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:14.532 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:14.532 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:14.533 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 195 2023-07-02T14:03:14.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.533 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:14.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:14.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:03:14.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:14.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:14.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:03:14.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:14.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:14.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285931+0000", 2023-07-02T14:03:14.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4530999999999995e-05 2023-07-02T14:03:14.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:14.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286006+0000", 2023-07-02T14:03:14.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4113e-05 2023-07-02T14:03:14.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:14.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286020+0000", 2023-07-02T14:03:14.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7459999999999997e-06 2023-07-02T14:03:14.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286026+0000", 2023-07-02T14:03:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025997769 2023-07-02T14:03:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312023+0000", 2023-07-02T14:03:14.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4403000000000001e-05 2023-07-02T14:03:14.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:14.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312078+0000", 2023-07-02T14:03:14.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020581399999999999 2023-07-02T14:03:14.539 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:14.539 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-07-02T14:03:14.539 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:14.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:14.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312284+0000", 2023-07-02T14:03:14.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005771043 2023-07-02T14:03:14.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:14.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318055+0000", 2023-07-02T14:03:14.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00046015200000000002 2023-07-02T14:03:14.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:14.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318515+0000", 2023-07-02T14:03:14.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9444000000000003e-05 2023-07-02T14:03:14.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:14.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318564+0000", 2023-07-02T14:03:14.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2141000000000002e-05 2023-07-02T14:03:14.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:14.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318606+0000", 2023-07-02T14:03:14.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001 2023-07-02T14:03:14.543 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.544 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.544 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.544 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:198 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:14.544 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:03:14.544 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.196690287000003, 2023-07-02T14:03:14.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999, 2023-07-02T14:03:14.545 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:14.545 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:14.545 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:14.545 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:14.545 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:14.545 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 198 2023-07-02T14:03:14.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.546 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:14.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:14.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:03:14.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:14.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:14.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:03:14.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:14.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:14.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286031+0000", 2023-07-02T14:03:14.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00049286700000000002 2023-07-02T14:03:14.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:14.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286524+0000", 2023-07-02T14:03:14.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4938e-05 2023-07-02T14:03:14.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:14.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286539+0000", 2023-07-02T14:03:14.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0680000000000002e-06 2023-07-02T14:03:14.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:14.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286545+0000", 2023-07-02T14:03:14.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026497718 2023-07-02T14:03:14.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313043+0000", 2023-07-02T14:03:14.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0884999999999998e-05 2023-07-02T14:03:14.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:14.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313094+0000", 2023-07-02T14:03:14.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014109999999999999 2023-07-02T14:03:14.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:14.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313235+0000", 2023-07-02T14:03:14.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048258980000000003 2023-07-02T14:03:14.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:14.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318061+0000", 2023-07-02T14:03:14.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014346350000000001 2023-07-02T14:03:14.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:14.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319495+0000", 2023-07-02T14:03:14.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1986e-05 2023-07-02T14:03:14.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:14.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319547+0000", 2023-07-02T14:03:14.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0457e-05 2023-07-02T14:03:14.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:14.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319578+0000", 2023-07-02T14:03:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999 2023-07-02T14:03:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.555 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:201 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:14.556 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:14.556 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.196169861999998, 2023-07-02T14:03:14.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003, 2023-07-02T14:03:14.556 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:14.556 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:14.556 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:14.556 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:14.557 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:14.557 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 201 2023-07-02T14:03:14.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.557 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:14.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:14.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:14.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:14.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:14.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:14.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:14.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:14.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:14.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019213999999999999 2023-07-02T14:03:14.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:14.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286744+0000", 2023-07-02T14:03:14.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5279999999999992e-06 2023-07-02T14:03:14.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:14.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286753+0000", 2023-07-02T14:03:14.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4200000000000004e-06 2023-07-02T14:03:14.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:14.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286760+0000", 2023-07-02T14:03:14.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026818386999999999 2023-07-02T14:03:14.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313578+0000", 2023-07-02T14:03:14.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015965 2023-07-02T14:03:14.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:14.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313738+0000", 2023-07-02T14:03:14.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015767299999999999 2023-07-02T14:03:14.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:14.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313895+0000", 2023-07-02T14:03:14.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041714930000000001 2023-07-02T14:03:14.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:14.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318067+0000", 2023-07-02T14:03:14.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001561698 2023-07-02T14:03:14.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:14.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319629+0000", 2023-07-02T14:03:14.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2346000000000003e-05 2023-07-02T14:03:14.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:14.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319681+0000", 2023-07-02T14:03:14.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4691000000000003e-05 2023-07-02T14:03:14.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:14.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319726+0000", 2023-07-02T14:03:14.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003 2023-07-02T14:03:14.566 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.566 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.566 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.567 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:204 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:14.567 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:14.567 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.195955024, 2023-07-02T14:03:14.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998, 2023-07-02T14:03:14.567 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:14.567 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:14.568 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:14.568 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:14.568 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:14.568 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 204 2023-07-02T14:03:14.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.568 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:14.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:14.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:14.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:14.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:14.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:14.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:14.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:14.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:14.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2497999999999998e-05 2023-07-02T14:03:14.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:14.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286819+0000", 2023-07-02T14:03:14.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0099999999999995e-06 2023-07-02T14:03:14.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:14.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286827+0000", 2023-07-02T14:03:14.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000003e-06 2023-07-02T14:03:14.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:14.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286833+0000", 2023-07-02T14:03:14.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027384716999999999 2023-07-02T14:03:14.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314218+0000", 2023-07-02T14:03:14.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6204000000000004e-05 2023-07-02T14:03:14.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:14.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314294+0000", 2023-07-02T14:03:14.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015757499999999999 2023-07-02T14:03:14.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:14.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314452+0000", 2023-07-02T14:03:14.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036215560000000002 2023-07-02T14:03:14.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:14.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318073+0000", 2023-07-02T14:03:14.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017023979999999999 2023-07-02T14:03:14.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:14.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319776+0000", 2023-07-02T14:03:14.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9561000000000002e-05 2023-07-02T14:03:14.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:14.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319825+0000", 2023-07-02T14:03:14.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0329e-05 2023-07-02T14:03:14.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:14.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319866+0000", 2023-07-02T14:03:14.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998 2023-07-02T14:03:14.577 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.577 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.577 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.578 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:207 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:14.578 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:03:14.578 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.195880379000002, 2023-07-02T14:03:14.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003, 2023-07-02T14:03:14.578 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:14.579 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:14.579 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:14.579 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:14.579 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:14.579 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 207 2023-07-02T14:03:14.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.579 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:14.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:14.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:03:14.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:14.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:14.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:03:14.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:14.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:14.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286841+0000", 2023-07-02T14:03:14.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00068375100000000002 2023-07-02T14:03:14.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:14.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287525+0000", 2023-07-02T14:03:14.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1601e-05 2023-07-02T14:03:14.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:14.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287537+0000", 2023-07-02T14:03:14.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6409999999999996e-06 2023-07-02T14:03:14.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:14.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287543+0000", 2023-07-02T14:03:14.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027245844000000002 2023-07-02T14:03:14.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314789+0000", 2023-07-02T14:03:14.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8910000000000003e-05 2023-07-02T14:03:14.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:14.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314858+0000", 2023-07-02T14:03:14.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030992599999999997 2023-07-02T14:03:14.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:14.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315168+0000", 2023-07-02T14:03:14.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029122869999999999 2023-07-02T14:03:14.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:14.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318080+0000", 2023-07-02T14:03:14.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019963020000000001 2023-07-02T14:03:14.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:14.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320077+0000", 2023-07-02T14:03:14.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5116000000000005e-05 2023-07-02T14:03:14.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:14.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320142+0000", 2023-07-02T14:03:14.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3793000000000004e-05 2023-07-02T14:03:14.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:14.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320205+0000", 2023-07-02T14:03:14.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003 2023-07-02T14:03:14.588 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.589 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.589 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.589 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:236 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2424832~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:14.589 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:03:14.589 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.195168195999997, 2023-07-02T14:03:14.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003, 2023-07-02T14:03:14.590 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:14.590 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:14.590 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:14.590 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:14.590 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:14.590 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2023-07-02T14:03:14.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.591 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:14.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:14.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:03:14.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:14.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:14.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:03:14.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:14.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:14.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287553+0000", 2023-07-02T14:03:14.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0667e-05 2023-07-02T14:03:14.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:14.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287614+0000", 2023-07-02T14:03:14.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0454e-05 2023-07-02T14:03:14.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:14.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287624+0000", 2023-07-02T14:03:14.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8564999999999999e-05 2023-07-02T14:03:14.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:14.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287643+0000", 2023-07-02T14:03:14.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027977596 2023-07-02T14:03:14.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315621+0000", 2023-07-02T14:03:14.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000201728 2023-07-02T14:03:14.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:14.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315822+0000", 2023-07-02T14:03:14.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018436599999999999 2023-07-02T14:03:14.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:14.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316007+0000", 2023-07-02T14:03:14.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042423440000000003 2023-07-02T14:03:14.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:14.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320249+0000", 2023-07-02T14:03:14.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001452633 2023-07-02T14:03:14.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:14.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321702+0000", 2023-07-02T14:03:14.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3616000000000003e-05 2023-07-02T14:03:14.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:14.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321755+0000", 2023-07-02T14:03:14.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840999999999999e-05 2023-07-02T14:03:14.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:14.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321793+0000", 2023-07-02T14:03:14.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003 2023-07-02T14:03:14.600 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.600 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.600 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.600 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:251 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:14.601 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:03:14.601 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.195072658999997, 2023-07-02T14:03:14.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001, 2023-07-02T14:03:14.601 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:14.601 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:14.601 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:14.601 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:14.602 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:14.602 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 251 2023-07-02T14:03:14.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.602 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:14.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:14.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:03:14.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:14.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:14.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:03:14.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:14.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:14.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287649+0000", 2023-07-02T14:03:14.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3232000000000001e-05 2023-07-02T14:03:14.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:14.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287702+0000", 2023-07-02T14:03:14.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0010000000000007e-06 2023-07-02T14:03:14.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:14.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287711+0000", 2023-07-02T14:03:14.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3569999999999997e-06 2023-07-02T14:03:14.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:14.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287717+0000", 2023-07-02T14:03:14.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028686454 2023-07-02T14:03:14.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316403+0000", 2023-07-02T14:03:14.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7741000000000002e-05 2023-07-02T14:03:14.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:14.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316461+0000", 2023-07-02T14:03:14.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4313000000000004e-05 2023-07-02T14:03:14.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:14.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316545+0000", 2023-07-02T14:03:14.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003734032 2023-07-02T14:03:14.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:14.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320279+0000", 2023-07-02T14:03:14.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015237219999999999 2023-07-02T14:03:14.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:14.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321803+0000", 2023-07-02T14:03:14.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1927e-05 2023-07-02T14:03:14.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:14.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321845+0000", 2023-07-02T14:03:14.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5398999999999998e-05 2023-07-02T14:03:14.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:14.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321880+0000", 2023-07-02T14:03:14.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001 2023-07-02T14:03:14.611 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.611 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.611 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.612 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:255 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2555904~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:14.612 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:03:14.612 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.194657931000002, 2023-07-02T14:03:14.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002, 2023-07-02T14:03:14.612 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:14.612 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:14.613 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:14.613 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:14.613 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:14.613 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 255 2023-07-02T14:03:14.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.613 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:14.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:14.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:03:14.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:14.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:14.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:03:14.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:14.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:14.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288063+0000", 2023-07-02T14:03:14.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4459999999999997e-05 2023-07-02T14:03:14.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:14.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288118+0000", 2023-07-02T14:03:14.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0998999999999999e-05 2023-07-02T14:03:14.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:14.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288129+0000", 2023-07-02T14:03:14.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3219999999999998e-06 2023-07-02T14:03:14.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:14.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288135+0000", 2023-07-02T14:03:14.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029137386000000001 2023-07-02T14:03:14.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317273+0000", 2023-07-02T14:03:14.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7819999999999997e-05 2023-07-02T14:03:14.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:14.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317350+0000", 2023-07-02T14:03:14.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000174864 2023-07-02T14:03:14.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:14.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317525+0000", 2023-07-02T14:03:14.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027943519999999999 2023-07-02T14:03:14.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:14.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320320+0000", 2023-07-02T14:03:14.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015681460000000001 2023-07-02T14:03:14.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:14.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321888+0000", 2023-07-02T14:03:14.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9508999999999997e-05 2023-07-02T14:03:14.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:14.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321927+0000", 2023-07-02T14:03:14.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5428e-05 2023-07-02T14:03:14.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:14.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321963+0000", 2023-07-02T14:03:14.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002 2023-07-02T14:03:14.622 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.622 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.623 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.623 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:446 2.1 2:910b2a90:::rbd_data.125de1449605.000000000000010a:head [write 65536~3211264] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:14.623 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:14.623 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.117378178999999, 2023-07-02T14:03:14.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003, 2023-07-02T14:03:14.624 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:14.624 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:14.624 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:14.624 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:14.624 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:14.624 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 446 2023-07-02T14:03:14.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.625 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:14.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:14.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:14.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:14.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:14.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:14.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:14.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:14.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:14.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015907372999999999 2023-07-02T14:03:14.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:14.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381251+0000", 2023-07-02T14:03:14.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5527000000000001e-05 2023-07-02T14:03:14.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:14.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381277+0000", 2023-07-02T14:03:14.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9958000000000001e-05 2023-07-02T14:03:14.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:14.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381297+0000", 2023-07-02T14:03:14.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4941000000000001e-05 2023-07-02T14:03:14.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381352+0000", 2023-07-02T14:03:14.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011565699999999999 2023-07-02T14:03:14.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:14.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381467+0000", 2023-07-02T14:03:14.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018039799999999999 2023-07-02T14:03:14.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:14.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381648+0000", 2023-07-02T14:03:14.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088183559999999994 2023-07-02T14:03:14.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:14.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.390466+0000", 2023-07-02T14:03:14.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007278278 2023-07-02T14:03:14.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:14.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397744+0000", 2023-07-02T14:03:14.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9289000000000001e-05 2023-07-02T14:03:14.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:14.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397804+0000", 2023-07-02T14:03:14.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0479000000000002e-05 2023-07-02T14:03:14.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:14.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397854+0000", 2023-07-02T14:03:14.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003 2023-07-02T14:03:14.634 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.634 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.634 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.634 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:03:14.634 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:03:14.634 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.375047385, 2023-07-02T14:03:14.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:03:14.635 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:14.635 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:03:14.635 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:14.635 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:14.635 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:14.635 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:03:14.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.636 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:14.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:14.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:03:14.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:14.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:14.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:03:14.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:14.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:14.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:03:14.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:03:14.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:14.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:03:14.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:03:14.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:14.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:03:14.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:03:14.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:14.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:03:14.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:03:14.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:03:14.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:03:14.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:14.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:03:14.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:03:14.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:03:14.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:03:14.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:14.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:03:14.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:03:14.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:14.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:03:14.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:03:14.643 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.643 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.644 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.644 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:03:14.644 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:03:14.644 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.374986647, 2023-07-02T14:03:14.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:03:14.645 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:14.645 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:14.645 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:14.645 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:14.645 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:14.645 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:03:14.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.646 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:14.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:14.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:03:14.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:14.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:14.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:03:14.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:14.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:14.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:03:14.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:03:14.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:14.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:03:14.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:03:14.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:14.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:03:14.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:03:14.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:14.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:03:14.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:03:14.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:03:14.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:03:14.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:14.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:03:14.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:03:14.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:03:14.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:03:14.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:14.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:03:14.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:03:14.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:14.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:03:14.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:03:14.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:14.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:03:14.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:03:14.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:14.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:03:14.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:03:14.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:14.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:03:14.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:03:14.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:14.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:03:14.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:03:14.656 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.656 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.656 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.657 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:14.657 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:03:14.657 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.1445323169999999, 2023-07-02T14:03:14.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:03:14.657 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:14.657 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:14.658 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:14.658 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:14.658 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:14.658 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:03:14.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.658 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:14.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:14.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:03:14.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:14.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:14.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:03:14.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:14.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:14.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:03:14.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:03:14.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:14.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:03:14.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:03:14.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:14.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:03:14.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:03:14.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:14.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:03:14.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:03:14.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:03:14.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:03:14.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:14.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:03:14.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:03:14.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:14.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:03:14.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:03:14.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:14.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:03:14.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:03:14.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:14.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:03:14.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:03:14.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:14.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:03:14.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:03:14.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:14.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:03:14.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:03:14.667 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.667 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.668 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.668 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2941 2.3 2:d3762e63:::rbd_data.125de1449605.00000000000000f8:head [write 65536~2031616] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:14.668 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:03:14.668 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.015429363, 2023-07-02T14:03:14.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002, 2023-07-02T14:03:14.669 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:14.669 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:14.669 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:14.669 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:14.669 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:14.669 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2941 2023-07-02T14:03:14.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.670 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:14.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:14.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:03:14.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:14.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:14.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:03:14.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:14.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:14.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467291+0000", 2023-07-02T14:03:14.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043022859999999998 2023-07-02T14:03:14.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:14.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471593+0000", 2023-07-02T14:03:14.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0999999999999999e-05 2023-07-02T14:03:14.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:14.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471614+0000", 2023-07-02T14:03:14.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0097e-05 2023-07-02T14:03:14.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:14.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471625+0000", 2023-07-02T14:03:14.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069808499999999998 2023-07-02T14:03:14.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472323+0000", 2023-07-02T14:03:14.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135635 2023-07-02T14:03:14.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:14.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472458+0000", 2023-07-02T14:03:14.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223365 2023-07-02T14:03:14.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:14.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472682+0000", 2023-07-02T14:03:14.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013761944999999999 2023-07-02T14:03:14.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:14.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.486444+0000", 2023-07-02T14:03:14.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016859747000000001 2023-07-02T14:03:14.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:14.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503303+0000", 2023-07-02T14:03:14.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6635000000000001e-05 2023-07-02T14:03:14.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:14.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503380+0000", 2023-07-02T14:03:14.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000161803 2023-07-02T14:03:14.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:14.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503542+0000", 2023-07-02T14:03:14.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002 2023-07-02T14:03:14.679 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.679 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.679 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.679 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2950 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 1507328~786432] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:14.679 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:03:14.680 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.007256055, 2023-07-02T14:03:14.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999, 2023-07-02T14:03:14.680 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:14.680 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:14.680 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:14.680 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:14.680 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:14.681 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2950 2023-07-02T14:03:14.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.681 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:14.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:14.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:03:14.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:14.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:14.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:03:14.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:14.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:14.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:03:14.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001147854 2023-07-02T14:03:14.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:14.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476614+0000", 2023-07-02T14:03:14.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.802e-05 2023-07-02T14:03:14.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:14.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476632+0000", 2023-07-02T14:03:14.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1860000000000003e-06 2023-07-02T14:03:14.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476639+0000", 2023-07-02T14:03:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061211240000000004 2023-07-02T14:03:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482760+0000", 2023-07-02T14:03:14.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8833999999999996e-05 2023-07-02T14:03:14.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:14.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482859+0000", 2023-07-02T14:03:14.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019270600000000001 2023-07-02T14:03:14.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:14.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.483051+0000", 2023-07-02T14:03:14.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060385630000000003 2023-07-02T14:03:14.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:14.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489090+0000", 2023-07-02T14:03:14.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024506918999999999 2023-07-02T14:03:14.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:14.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513597+0000", 2023-07-02T14:03:14.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9672999999999995e-05 2023-07-02T14:03:14.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:14.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513667+0000", 2023-07-02T14:03:14.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024329500000000001 2023-07-02T14:03:14.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:14.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513910+0000", 2023-07-02T14:03:14.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999 2023-07-02T14:03:14.690 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.690 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.690 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.691 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2951 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2686976~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:14.691 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:03:14.691 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.0060752769999999, 2023-07-02T14:03:14.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999, 2023-07-02T14:03:14.691 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:14.691 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:14.691 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:14.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:14.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:14.692 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2951 2023-07-02T14:03:14.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.692 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:14.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:14.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:03:14.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:14.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:14.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:03:14.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:14.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:14.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476646+0000", 2023-07-02T14:03:14.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7639000000000003e-05 2023-07-02T14:03:14.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:14.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476684+0000", 2023-07-02T14:03:14.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3508000000000002e-05 2023-07-02T14:03:14.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:14.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476747+0000", 2023-07-02T14:03:14.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8499999999999994e-06 2023-07-02T14:03:14.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:14.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476755+0000", 2023-07-02T14:03:14.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078346160000000008 2023-07-02T14:03:14.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484590+0000", 2023-07-02T14:03:14.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7206000000000004e-05 2023-07-02T14:03:14.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:14.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484687+0000", 2023-07-02T14:03:14.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016243099999999999 2023-07-02T14:03:14.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:14.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484849+0000", 2023-07-02T14:03:14.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042482860000000004 2023-07-02T14:03:14.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:14.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489097+0000", 2023-07-02T14:03:14.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.02489504 2023-07-02T14:03:14.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:14.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513992+0000", 2023-07-02T14:03:14.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9972000000000001e-05 2023-07-02T14:03:14.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:14.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514052+0000", 2023-07-02T14:03:14.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149264 2023-07-02T14:03:14.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:14.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514202+0000", 2023-07-02T14:03:14.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999 2023-07-02T14:03:14.701 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.701 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.701 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.702 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2952 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2883584~12288] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:14.702 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:03:14.702 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.0059614419999998, 2023-07-02T14:03:14.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651, 2023-07-02T14:03:14.702 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:14.703 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:14.703 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:14.703 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:14.703 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:14.703 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2952 2023-07-02T14:03:14.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.703 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:14.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:14.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:03:14.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:14.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:14.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:03:14.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:14.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:14.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476760+0000", 2023-07-02T14:03:14.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3664e-05 2023-07-02T14:03:14.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:14.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476774+0000", 2023-07-02T14:03:14.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.166e-06 2023-07-02T14:03:14.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:14.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476783+0000", 2023-07-02T14:03:14.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8720000000000001e-06 2023-07-02T14:03:14.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:14.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476788+0000", 2023-07-02T14:03:14.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087180529999999999 2023-07-02T14:03:14.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:14.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485506+0000", 2023-07-02T14:03:14.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5680999999999994e-05 2023-07-02T14:03:14.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:14.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485602+0000", 2023-07-02T14:03:14.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000222365 2023-07-02T14:03:14.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:14.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485824+0000", 2023-07-02T14:03:14.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003277379 2023-07-02T14:03:14.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:14.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489101+0000", 2023-07-02T14:03:14.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025151739999999999 2023-07-02T14:03:14.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:14.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514253+0000", 2023-07-02T14:03:14.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4749000000000002e-05 2023-07-02T14:03:14.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:14.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514288+0000", 2023-07-02T14:03:14.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130634 2023-07-02T14:03:14.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:14.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:14.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:14.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514419+0000", 2023-07-02T14:03:14.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651 2023-07-02T14:03:14.712 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.713 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.713 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.713 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:14.713 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:14.713 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:14.714 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:14.715 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:03:14.715 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:03:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:03:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:03:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:03:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:03:14.718 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:03:14.756 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:14.755+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:14.791 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete...+ sync b3fbb028-e267-4d25-a750-93ba18a2316a-mnt 2023-07-02T14:03:14.812 INFO:tasks.workunit.client.0.smithi163.stderr:+ rm -rf b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4 b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4.tar.gz 2023-07-02T14:03:14.818 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:14.817+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:14.852 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:14.852 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:14.852 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:03:14.853 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:14.862 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:03:14.919 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:14.919+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:15.020 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:15.019+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:15.025 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:15.025 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:15.026 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:15.026 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:15.026 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:15.035 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:03:15.121 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:15.121+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:15.203 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:15.204 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:15.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:15.204 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:15.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.204 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:322 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:15.204 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:15.205 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.902131429000001, 2023-07-02T14:03:15.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.03676041, 2023-07-02T14:03:15.205 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.205 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.205 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:15.205 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:15.205 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:15.206 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 322 2023-07-02T14:03:15.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.206 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:15.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:15.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:15.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298819+0000", 2023-07-02T14:03:15.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8068e-05 2023-07-02T14:03:15.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298877+0000", 2023-07-02T14:03:15.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3339999999999998e-06 2023-07-02T14:03:15.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298883+0000", 2023-07-02T14:03:15.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1030000000000004e-06 2023-07-02T14:03:15.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:15.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298887+0000", 2023-07-02T14:03:15.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023640230000000002 2023-07-02T14:03:15.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322528+0000", 2023-07-02T14:03:15.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8424999999999994e-05 2023-07-02T14:03:15.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:15.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322616+0000", 2023-07-02T14:03:15.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116815 2023-07-02T14:03:15.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:15.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322733+0000", 2023-07-02T14:03:15.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048252629999999998 2023-07-02T14:03:15.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:15.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327558+0000", 2023-07-02T14:03:15.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078965240000000003 2023-07-02T14:03:15.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:15.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335455+0000", 2023-07-02T14:03:15.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0511999999999999e-05 2023-07-02T14:03:15.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:15.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335515+0000", 2023-07-02T14:03:15.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4789000000000006e-05 2023-07-02T14:03:15.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:15.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335580+0000", 2023-07-02T14:03:15.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.03676041 2023-07-02T14:03:15.214 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.214 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.214 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.215 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:323 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:15.215 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:15.215 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.902060048999999, 2023-07-02T14:03:15.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036875044000000003, 2023-07-02T14:03:15.215 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.216 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.216 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:15.216 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:15.216 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:15.216 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 323 2023-07-02T14:03:15.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.216 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:15.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:15.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:15.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298890+0000", 2023-07-02T14:03:15.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7277000000000002e-05 2023-07-02T14:03:15.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298938+0000", 2023-07-02T14:03:15.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0495e-05 2023-07-02T14:03:15.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298948+0000", 2023-07-02T14:03:15.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6829999999999999e-06 2023-07-02T14:03:15.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:15.221 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298953+0000", 2023-07-02T14:03:15.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024271435000000001 2023-07-02T14:03:15.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323224+0000", 2023-07-02T14:03:15.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9266999999999998e-05 2023-07-02T14:03:15.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:15.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323304+0000", 2023-07-02T14:03:15.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000157021 2023-07-02T14:03:15.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:15.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323461+0000", 2023-07-02T14:03:15.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041036689999999999 2023-07-02T14:03:15.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:15.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327564+0000", 2023-07-02T14:03:15.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080753490000000008 2023-07-02T14:03:15.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:15.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335640+0000", 2023-07-02T14:03:15.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7398999999999994e-05 2023-07-02T14:03:15.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:15.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335727+0000", 2023-07-02T14:03:15.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8967000000000001e-05 2023-07-02T14:03:15.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:15.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335766+0000", 2023-07-02T14:03:15.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036875044000000003 2023-07-02T14:03:15.230 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.230 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.230 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.231 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:324 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:15.231 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:15.232 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.901992671000002, 2023-07-02T14:03:15.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036935673000000002, 2023-07-02T14:03:15.232 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.233 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.233 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:15.233 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:15.234 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:15.234 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 324 2023-07-02T14:03:15.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.234 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:15.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:15.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:15.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:15.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5519999999999997e-05 2023-07-02T14:03:15.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299013+0000", 2023-07-02T14:03:15.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0701e-05 2023-07-02T14:03:15.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299024+0000", 2023-07-02T14:03:15.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2379999999999999e-06 2023-07-02T14:03:15.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:15.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299028+0000", 2023-07-02T14:03:15.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024972768999999999 2023-07-02T14:03:15.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324001+0000", 2023-07-02T14:03:15.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1178000000000002e-05 2023-07-02T14:03:15.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:15.245 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324072+0000", 2023-07-02T14:03:15.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000166119 2023-07-02T14:03:15.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.246 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.246 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:15.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324238+0000", 2023-07-02T14:03:15.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033314210000000002 2023-07-02T14:03:15.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:15.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327570+0000", 2023-07-02T14:03:15.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0082393369999999994 2023-07-02T14:03:15.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:15.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335809+0000", 2023-07-02T14:03:15.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7071999999999999e-05 2023-07-02T14:03:15.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:15.251 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335856+0000", 2023-07-02T14:03:15.251 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7926999999999998e-05 2023-07-02T14:03:15.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:15.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335894+0000", 2023-07-02T14:03:15.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036935673000000002 2023-07-02T14:03:15.252 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.253 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.253 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.254 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:325 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:15.254 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:15.254 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.901917343999997, 2023-07-02T14:03:15.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036990838999999998, 2023-07-02T14:03:15.254 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.255 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.255 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:15.255 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:15.256 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:15.256 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 325 2023-07-02T14:03:15.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.256 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:15.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:15.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:15.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299033+0000", 2023-07-02T14:03:15.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107619 2023-07-02T14:03:15.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.261 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.261 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299141+0000", 2023-07-02T14:03:15.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3497e-05 2023-07-02T14:03:15.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.262 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.262 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.262 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299154+0000", 2023-07-02T14:03:15.262 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4669e-05 2023-07-02T14:03:15.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.262 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:15.263 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299169+0000", 2023-07-02T14:03:15.263 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025630159999999999 2023-07-02T14:03:15.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.263 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324799+0000", 2023-07-02T14:03:15.263 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6953000000000007e-05 2023-07-02T14:03:15.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.264 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:15.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324876+0000", 2023-07-02T14:03:15.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016346199999999999 2023-07-02T14:03:15.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:15.265 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325039+0000", 2023-07-02T14:03:15.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003566624 2023-07-02T14:03:15.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:15.265 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328606+0000", 2023-07-02T14:03:15.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073295540000000003 2023-07-02T14:03:15.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:15.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335936+0000", 2023-07-02T14:03:15.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0044999999999999e-05 2023-07-02T14:03:15.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:15.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335986+0000", 2023-07-02T14:03:15.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8872999999999999e-05 2023-07-02T14:03:15.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:15.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336024+0000", 2023-07-02T14:03:15.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036990838999999998 2023-07-02T14:03:15.267 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.268 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.268 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.268 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:326 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:15.268 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:15.268 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.901775416, 2023-07-02T14:03:15.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036975239999999999, 2023-07-02T14:03:15.269 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.269 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.269 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:15.269 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:15.269 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:15.269 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 326 2023-07-02T14:03:15.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.270 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:15.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:15.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:15.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.271 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299175+0000", 2023-07-02T14:03:15.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4782000000000003e-05 2023-07-02T14:03:15.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299250+0000", 2023-07-02T14:03:15.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.277e-05 2023-07-02T14:03:15.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299262+0000", 2023-07-02T14:03:15.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1460000000000001e-06 2023-07-02T14:03:15.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.273 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:15.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299268+0000", 2023-07-02T14:03:15.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026158098000000001 2023-07-02T14:03:15.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325427+0000", 2023-07-02T14:03:15.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4845999999999995e-05 2023-07-02T14:03:15.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:15.275 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325501+0000", 2023-07-02T14:03:15.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015890699999999999 2023-07-02T14:03:15.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.275 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:15.275 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325660+0000", 2023-07-02T14:03:15.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002952114 2023-07-02T14:03:15.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:15.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328612+0000", 2023-07-02T14:03:15.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074491949999999996 2023-07-02T14:03:15.276 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:15.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336062+0000", 2023-07-02T14:03:15.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7376999999999997e-05 2023-07-02T14:03:15.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.277 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:15.277 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336109+0000", 2023-07-02T14:03:15.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1814000000000003e-05 2023-07-02T14:03:15.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:15.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336151+0000", 2023-07-02T14:03:15.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036975239999999999 2023-07-02T14:03:15.278 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.278 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.278 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.279 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:327 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 589824~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:15.279 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:15.279 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.901676154999997, 2023-07-02T14:03:15.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037234253000000002, 2023-07-02T14:03:15.279 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.279 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.279 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:15.280 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:15.280 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:15.280 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 327 2023-07-02T14:03:15.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.281 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:15.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.283 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:15.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:15.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299274+0000", 2023-07-02T14:03:15.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2213999999999997e-05 2023-07-02T14:03:15.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299346+0000", 2023-07-02T14:03:15.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1816e-05 2023-07-02T14:03:15.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299358+0000", 2023-07-02T14:03:15.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2626000000000001e-05 2023-07-02T14:03:15.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:15.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299371+0000", 2023-07-02T14:03:15.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026916853000000001 2023-07-02T14:03:15.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326288+0000", 2023-07-02T14:03:15.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6909000000000004e-05 2023-07-02T14:03:15.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:15.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326375+0000", 2023-07-02T14:03:15.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015430399999999999 2023-07-02T14:03:15.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:15.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326529+0000", 2023-07-02T14:03:15.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096652100000000005 2023-07-02T14:03:15.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:15.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336194+0000", 2023-07-02T14:03:15.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115299 2023-07-02T14:03:15.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:15.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336309+0000", 2023-07-02T14:03:15.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015981200000000001 2023-07-02T14:03:15.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:15.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336469+0000", 2023-07-02T14:03:15.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9867999999999998e-05 2023-07-02T14:03:15.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:15.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336509+0000", 2023-07-02T14:03:15.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037234253000000002 2023-07-02T14:03:15.299 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.300 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.300 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.301 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:328 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:15.301 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:15.301 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.901572565999999, 2023-07-02T14:03:15.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037216610999999997, 2023-07-02T14:03:15.302 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.302 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.302 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:15.302 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:15.302 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:15.303 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 328 2023-07-02T14:03:15.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.303 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:15.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:15.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:15.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:15.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1412e-05 2023-07-02T14:03:15.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299449+0000", 2023-07-02T14:03:15.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0219000000000001e-05 2023-07-02T14:03:15.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299469+0000", 2023-07-02T14:03:15.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9480000000000003e-06 2023-07-02T14:03:15.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:15.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299479+0000", 2023-07-02T14:03:15.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027444344999999998 2023-07-02T14:03:15.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.307 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326924+0000", 2023-07-02T14:03:15.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5165000000000001e-05 2023-07-02T14:03:15.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:15.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327019+0000", 2023-07-02T14:03:15.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019113499999999999 2023-07-02T14:03:15.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:15.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327210+0000", 2023-07-02T14:03:15.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090197449999999992 2023-07-02T14:03:15.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:15.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336230+0000", 2023-07-02T14:03:15.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028764399999999999 2023-07-02T14:03:15.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:15.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336517+0000", 2023-07-02T14:03:15.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1981999999999999e-05 2023-07-02T14:03:15.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:15.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336559+0000", 2023-07-02T14:03:15.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:03:15.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:15.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336595+0000", 2023-07-02T14:03:15.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037216610999999997 2023-07-02T14:03:15.311 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.311 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.311 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.312 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:329 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:15.312 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:15.312 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.901462649000003, 2023-07-02T14:03:15.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037191142000000003, 2023-07-02T14:03:15.312 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.313 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.313 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:15.313 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:15.313 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:15.313 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 329 2023-07-02T14:03:15.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.313 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:15.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:15.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:15.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299487+0000", 2023-07-02T14:03:15.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7871000000000001e-05 2023-07-02T14:03:15.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299555+0000", 2023-07-02T14:03:15.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0446e-05 2023-07-02T14:03:15.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299566+0000", 2023-07-02T14:03:15.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7939999999999999e-06 2023-07-02T14:03:15.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:15.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299572+0000", 2023-07-02T14:03:15.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028034019 2023-07-02T14:03:15.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327606+0000", 2023-07-02T14:03:15.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1967999999999999e-05 2023-07-02T14:03:15.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:15.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327678+0000", 2023-07-02T14:03:15.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022216900000000001 2023-07-02T14:03:15.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:15.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327900+0000", 2023-07-02T14:03:15.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084016899999999999 2023-07-02T14:03:15.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:15.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336301+0000", 2023-07-02T14:03:15.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00030122800000000001 2023-07-02T14:03:15.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:15.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336603+0000", 2023-07-02T14:03:15.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0380000000000003e-05 2023-07-02T14:03:15.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:15.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336643+0000", 2023-07-02T14:03:15.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6455999999999998e-05 2023-07-02T14:03:15.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:15.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336679+0000", 2023-07-02T14:03:15.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037191142000000003 2023-07-02T14:03:15.322 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.322 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.322 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.322 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:330 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:15.322 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:15.322 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.901371879000003, 2023-07-02T14:03:15.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037849305, 2023-07-02T14:03:15.323 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.323 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.323 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:15.323 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:15.323 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:15.323 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 330 2023-07-02T14:03:15.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.324 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.324 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:15.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:15.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:15.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299578+0000", 2023-07-02T14:03:15.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4708000000000005e-05 2023-07-02T14:03:15.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299643+0000", 2023-07-02T14:03:15.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7309999999999999e-06 2023-07-02T14:03:15.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299652+0000", 2023-07-02T14:03:15.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7989999999999999e-06 2023-07-02T14:03:15.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:15.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299658+0000", 2023-07-02T14:03:15.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028981664000000001 2023-07-02T14:03:15.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328639+0000", 2023-07-02T14:03:15.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0939000000000006e-05 2023-07-02T14:03:15.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:15.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328710+0000", 2023-07-02T14:03:15.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000142624 2023-07-02T14:03:15.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:15.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328853+0000", 2023-07-02T14:03:15.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078374640000000006 2023-07-02T14:03:15.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:15.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336690+0000", 2023-07-02T14:03:15.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00068453000000000003 2023-07-02T14:03:15.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:15.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337375+0000", 2023-07-02T14:03:15.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0667999999999998e-05 2023-07-02T14:03:15.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:15.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337406+0000", 2023-07-02T14:03:15.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2858999999999999e-05 2023-07-02T14:03:15.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:15.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337428+0000", 2023-07-02T14:03:15.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037849305 2023-07-02T14:03:15.332 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.332 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.332 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.333 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:331 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:15.333 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:15.333 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.901288016999999, 2023-07-02T14:03:15.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037870332, 2023-07-02T14:03:15.333 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.333 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.333 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:15.334 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:15.334 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:15.334 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 331 2023-07-02T14:03:15.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.334 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:15.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:15.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:15.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299662+0000", 2023-07-02T14:03:15.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2252e-05 2023-07-02T14:03:15.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299725+0000", 2023-07-02T14:03:15.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8440000000000004e-06 2023-07-02T14:03:15.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299733+0000", 2023-07-02T14:03:15.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6759999999999998e-06 2023-07-02T14:03:15.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:15.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299738+0000", 2023-07-02T14:03:15.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029459387 2023-07-02T14:03:15.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329197+0000", 2023-07-02T14:03:15.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0087999999999997e-05 2023-07-02T14:03:15.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:15.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329268+0000", 2023-07-02T14:03:15.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014405700000000001 2023-07-02T14:03:15.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:15.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329412+0000", 2023-07-02T14:03:15.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072842100000000002 2023-07-02T14:03:15.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:15.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336696+0000", 2023-07-02T14:03:15.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00076275300000000002 2023-07-02T14:03:15.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:15.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337459+0000", 2023-07-02T14:03:15.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5988000000000002e-05 2023-07-02T14:03:15.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:15.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337495+0000", 2023-07-02T14:03:15.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8686000000000002e-05 2023-07-02T14:03:15.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:15.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337533+0000", 2023-07-02T14:03:15.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037870332 2023-07-02T14:03:15.342 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.342 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.343 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.343 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:332 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:15.343 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:15.343 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.901208093000001, 2023-07-02T14:03:15.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037892506999999999, 2023-07-02T14:03:15.344 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.344 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.344 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:15.344 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:15.344 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:15.344 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 332 2023-07-02T14:03:15.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.344 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:15.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:15.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:15.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299742+0000", 2023-07-02T14:03:15.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1366999999999997e-05 2023-07-02T14:03:15.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299804+0000", 2023-07-02T14:03:15.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9319999999999996e-06 2023-07-02T14:03:15.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299812+0000", 2023-07-02T14:03:15.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4359999999999999e-06 2023-07-02T14:03:15.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:15.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299818+0000", 2023-07-02T14:03:15.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029946144000000001 2023-07-02T14:03:15.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329764+0000", 2023-07-02T14:03:15.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000164828 2023-07-02T14:03:15.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:15.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329929+0000", 2023-07-02T14:03:15.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016139400000000001 2023-07-02T14:03:15.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:15.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330090+0000", 2023-07-02T14:03:15.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066134009999999997 2023-07-02T14:03:15.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:15.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336704+0000", 2023-07-02T14:03:15.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00086955400000000003 2023-07-02T14:03:15.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:15.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337573+0000", 2023-07-02T14:03:15.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8447000000000001e-05 2023-07-02T14:03:15.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:15.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337602+0000", 2023-07-02T14:03:15.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.362e-05 2023-07-02T14:03:15.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:15.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337635+0000", 2023-07-02T14:03:15.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037892506999999999 2023-07-02T14:03:15.353 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.353 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.353 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.353 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:333 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:15.353 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:15.354 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.901128012000001, 2023-07-02T14:03:15.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037932598999999997, 2023-07-02T14:03:15.354 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.354 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.354 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:15.354 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:15.354 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:15.355 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 333 2023-07-02T14:03:15.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.355 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:15.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:15.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:15.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299822+0000", 2023-07-02T14:03:15.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0831000000000001e-05 2023-07-02T14:03:15.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299883+0000", 2023-07-02T14:03:15.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.322e-06 2023-07-02T14:03:15.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299888+0000", 2023-07-02T14:03:15.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3479999999999997e-06 2023-07-02T14:03:15.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:15.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299893+0000", 2023-07-02T14:03:15.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030831905999999999 2023-07-02T14:03:15.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330725+0000", 2023-07-02T14:03:15.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3560999999999996e-05 2023-07-02T14:03:15.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:15.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330798+0000", 2023-07-02T14:03:15.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015165999999999999 2023-07-02T14:03:15.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:15.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330950+0000", 2023-07-02T14:03:15.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057590319999999999 2023-07-02T14:03:15.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:15.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336709+0000", 2023-07-02T14:03:15.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00096987899999999999 2023-07-02T14:03:15.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:15.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337679+0000", 2023-07-02T14:03:15.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9699000000000001e-05 2023-07-02T14:03:15.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:15.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337719+0000", 2023-07-02T14:03:15.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6961000000000001e-05 2023-07-02T14:03:15.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:15.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337756+0000", 2023-07-02T14:03:15.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037932598999999997 2023-07-02T14:03:15.363 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.363 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.363 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.364 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:372 2.0 2:169af44a:::rbd_data.125de1449605.0000000000000106:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:15.364 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:03:15.364 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.897550686999999, 2023-07-02T14:03:15.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036066963, 2023-07-02T14:03:15.364 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.364 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.364 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:15.365 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:15.365 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:15.365 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 372 2023-07-02T14:03:15.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.365 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:03:15.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:03:15.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:15.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:03:15.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1002000000000003e-05 2023-07-02T14:03:15.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303461+0000", 2023-07-02T14:03:15.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9306e-05 2023-07-02T14:03:15.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303490+0000", 2023-07-02T14:03:15.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5026e-05 2023-07-02T14:03:15.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:15.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303505+0000", 2023-07-02T14:03:15.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033623129000000002 2023-07-02T14:03:15.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337128+0000", 2023-07-02T14:03:15.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6968000000000002e-05 2023-07-02T14:03:15.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:15.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337175+0000", 2023-07-02T14:03:15.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000101127 2023-07-02T14:03:15.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:15.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337276+0000", 2023-07-02T14:03:15.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017866920000000001 2023-07-02T14:03:15.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:15.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339063+0000", 2023-07-02T14:03:15.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00031459800000000003 2023-07-02T14:03:15.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:15.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339377+0000", 2023-07-02T14:03:15.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7145999999999999e-05 2023-07-02T14:03:15.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:15.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339425+0000", 2023-07-02T14:03:15.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2632000000000001e-05 2023-07-02T14:03:15.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:15.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339467+0000", 2023-07-02T14:03:15.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036066963 2023-07-02T14:03:15.373 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.373 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.374 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.374 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:416 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:15.374 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:15.374 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.882834633000002, 2023-07-02T14:03:15.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039242025, 2023-07-02T14:03:15.375 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.375 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.375 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:15.375 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:15.375 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:15.375 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:03:15.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.375 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:15.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:15.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:15.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:15.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.692e-06 2023-07-02T14:03:15.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318119+0000", 2023-07-02T14:03:15.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9790000000000005e-06 2023-07-02T14:03:15.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318129+0000", 2023-07-02T14:03:15.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3640000000000001e-06 2023-07-02T14:03:15.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:15.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318137+0000", 2023-07-02T14:03:15.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022607808 2023-07-02T14:03:15.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340745+0000", 2023-07-02T14:03:15.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0130999999999996e-05 2023-07-02T14:03:15.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340825+0000", 2023-07-02T14:03:15.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039581080000000001 2023-07-02T14:03:15.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344783+0000", 2023-07-02T14:03:15.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000006e-05 2023-07-02T14:03:15.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344854+0000", 2023-07-02T14:03:15.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052105550000000004 2023-07-02T14:03:15.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350065+0000", 2023-07-02T14:03:15.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0321999999999997e-05 2023-07-02T14:03:15.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350115+0000", 2023-07-02T14:03:15.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012378409999999999 2023-07-02T14:03:15.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351353+0000", 2023-07-02T14:03:15.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5942000000000001e-05 2023-07-02T14:03:15.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:15.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351409+0000", 2023-07-02T14:03:15.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035677700000000002 2023-07-02T14:03:15.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:15.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351766+0000", 2023-07-02T14:03:15.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000926719 2023-07-02T14:03:15.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:15.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352693+0000", 2023-07-02T14:03:15.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045652080000000003 2023-07-02T14:03:15.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:15.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357258+0000", 2023-07-02T14:03:15.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4717e-05 2023-07-02T14:03:15.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:15.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357293+0000", 2023-07-02T14:03:15.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5807000000000002e-05 2023-07-02T14:03:15.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:15.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357358+0000", 2023-07-02T14:03:15.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039242025 2023-07-02T14:03:15.387 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.388 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.388 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.388 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:417 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:15.388 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:15.388 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.882809195999997, 2023-07-02T14:03:15.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042407105, 2023-07-02T14:03:15.389 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.389 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.389 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:15.389 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:15.389 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:15.389 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 417 2023-07-02T14:03:15.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.390 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:15.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:15.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:15.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318141+0000", 2023-07-02T14:03:15.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7120000000000002e-06 2023-07-02T14:03:15.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318145+0000", 2023-07-02T14:03:15.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8869999999999997e-06 2023-07-02T14:03:15.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318151+0000", 2023-07-02T14:03:15.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9820000000000002e-06 2023-07-02T14:03:15.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:15.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318155+0000", 2023-07-02T14:03:15.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699133 2023-07-02T14:03:15.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340854+0000", 2023-07-02T14:03:15.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6849999999999998e-05 2023-07-02T14:03:15.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340931+0000", 2023-07-02T14:03:15.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039672140000000002 2023-07-02T14:03:15.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344898+0000", 2023-07-02T14:03:15.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5341999999999998e-05 2023-07-02T14:03:15.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344973+0000", 2023-07-02T14:03:15.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051819400000000003 2023-07-02T14:03:15.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350155+0000", 2023-07-02T14:03:15.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3261999999999999e-05 2023-07-02T14:03:15.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350209+0000", 2023-07-02T14:03:15.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00194097 2023-07-02T14:03:15.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352150+0000", 2023-07-02T14:03:15.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6851e-05 2023-07-02T14:03:15.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352226+0000", 2023-07-02T14:03:15.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051831109999999998 2023-07-02T14:03:15.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357409+0000", 2023-07-02T14:03:15.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5325e-05 2023-07-02T14:03:15.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:15.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357495+0000", 2023-07-02T14:03:15.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042107899999999998 2023-07-02T14:03:15.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:15.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357916+0000", 2023-07-02T14:03:15.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0011672449999999999 2023-07-02T14:03:15.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:15.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.359083+0000", 2023-07-02T14:03:15.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013534930000000001 2023-07-02T14:03:15.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:15.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360437+0000", 2023-07-02T14:03:15.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2159000000000002e-05 2023-07-02T14:03:15.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:15.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360469+0000", 2023-07-02T14:03:15.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0092000000000005e-05 2023-07-02T14:03:15.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:15.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360549+0000", 2023-07-02T14:03:15.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042407105 2023-07-02T14:03:15.403 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.403 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.403 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.403 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:418 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:15.404 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:15.404 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.882789955, 2023-07-02T14:03:15.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.045613663999999998, 2023-07-02T14:03:15.404 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.404 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.404 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:15.404 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:15.404 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:15.405 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 418 2023-07-02T14:03:15.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.405 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.405 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:15.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:15.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:15.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318160+0000", 2023-07-02T14:03:15.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6490000000000001e-06 2023-07-02T14:03:15.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318164+0000", 2023-07-02T14:03:15.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:03:15.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318170+0000", 2023-07-02T14:03:15.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5279999999999999e-06 2023-07-02T14:03:15.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:15.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318173+0000", 2023-07-02T14:03:15.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022788856 2023-07-02T14:03:15.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340962+0000", 2023-07-02T14:03:15.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7730000000000003e-05 2023-07-02T14:03:15.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341040+0000", 2023-07-02T14:03:15.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039795899999999999 2023-07-02T14:03:15.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345020+0000", 2023-07-02T14:03:15.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4962000000000006e-05 2023-07-02T14:03:15.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345085+0000", 2023-07-02T14:03:15.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051955430000000004 2023-07-02T14:03:15.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350280+0000", 2023-07-02T14:03:15.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8913999999999995e-05 2023-07-02T14:03:15.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350379+0000", 2023-07-02T14:03:15.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001887344 2023-07-02T14:03:15.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352266+0000", 2023-07-02T14:03:15.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7865000000000003e-05 2023-07-02T14:03:15.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352314+0000", 2023-07-02T14:03:15.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060621190000000004 2023-07-02T14:03:15.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358376+0000", 2023-07-02T14:03:15.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2448999999999997e-05 2023-07-02T14:03:15.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358449+0000", 2023-07-02T14:03:15.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0021539929999999999 2023-07-02T14:03:15.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360603+0000", 2023-07-02T14:03:15.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7859000000000002e-05 2023-07-02T14:03:15.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:15.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360671+0000", 2023-07-02T14:03:15.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026930699999999999 2023-07-02T14:03:15.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:15.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360940+0000", 2023-07-02T14:03:15.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00081179699999999995 2023-07-02T14:03:15.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:15.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361752+0000", 2023-07-02T14:03:15.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0019350490000000001 2023-07-02T14:03:15.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:15.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363687+0000", 2023-07-02T14:03:15.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4462000000000003e-05 2023-07-02T14:03:15.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:15.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363721+0000", 2023-07-02T14:03:15.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3408000000000001e-05 2023-07-02T14:03:15.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:15.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363775+0000", 2023-07-02T14:03:15.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.045613663999999998 2023-07-02T14:03:15.419 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.420 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.420 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.420 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:419 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:15.420 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:15.420 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.882773628000002, 2023-07-02T14:03:15.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.047296703000000002, 2023-07-02T14:03:15.421 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.421 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.421 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:15.421 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:15.421 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:15.421 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 419 2023-07-02T14:03:15.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.422 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:15.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:15.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:15.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:15.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.433e-06 2023-07-02T14:03:15.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318180+0000", 2023-07-02T14:03:15.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9950000000000002e-06 2023-07-02T14:03:15.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318186+0000", 2023-07-02T14:03:15.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7290000000000002e-06 2023-07-02T14:03:15.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:15.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318190+0000", 2023-07-02T14:03:15.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022877989000000001 2023-07-02T14:03:15.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341068+0000", 2023-07-02T14:03:15.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5803999999999994e-05 2023-07-02T14:03:15.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341154+0000", 2023-07-02T14:03:15.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039633000000000003 2023-07-02T14:03:15.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345117+0000", 2023-07-02T14:03:15.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7188999999999996e-05 2023-07-02T14:03:15.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345194+0000", 2023-07-02T14:03:15.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052306059999999996 2023-07-02T14:03:15.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350425+0000", 2023-07-02T14:03:15.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8093999999999997e-05 2023-07-02T14:03:15.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350473+0000", 2023-07-02T14:03:15.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001865158 2023-07-02T14:03:15.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352338+0000", 2023-07-02T14:03:15.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5479000000000003e-05 2023-07-02T14:03:15.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352384+0000", 2023-07-02T14:03:15.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061153290000000001 2023-07-02T14:03:15.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358499+0000", 2023-07-02T14:03:15.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5156999999999999e-05 2023-07-02T14:03:15.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358574+0000", 2023-07-02T14:03:15.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027847459999999998 2023-07-02T14:03:15.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361359+0000", 2023-07-02T14:03:15.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2633999999999999e-05 2023-07-02T14:03:15.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361402+0000", 2023-07-02T14:03:15.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024171510000000002 2023-07-02T14:03:15.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363819+0000", 2023-07-02T14:03:15.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6988999999999999e-05 2023-07-02T14:03:15.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:15.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363876+0000", 2023-07-02T14:03:15.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035321300000000001 2023-07-02T14:03:15.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:15.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.364229+0000", 2023-07-02T14:03:15.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00094674799999999997 2023-07-02T14:03:15.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:15.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365176+0000", 2023-07-02T14:03:15.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019724200000000001 2023-07-02T14:03:15.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:15.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365373+0000", 2023-07-02T14:03:15.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2116e-05 2023-07-02T14:03:15.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:15.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365425+0000", 2023-07-02T14:03:15.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8996999999999999e-05 2023-07-02T14:03:15.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:15.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365474+0000", 2023-07-02T14:03:15.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.047296703000000002 2023-07-02T14:03:15.437 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.438 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.438 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.438 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:472 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:15.438 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:15.438 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.823317396, 2023-07-02T14:03:15.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036204721000000002, 2023-07-02T14:03:15.439 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.439 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.439 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:15.439 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:15.439 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:15.439 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 472 2023-07-02T14:03:15.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.439 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:15.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:15.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:03:15.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377633+0000", 2023-07-02T14:03:15.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.815e-06 2023-07-02T14:03:15.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377635+0000", 2023-07-02T14:03:15.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9459999999999998e-06 2023-07-02T14:03:15.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377638+0000", 2023-07-02T14:03:15.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.673e-06 2023-07-02T14:03:15.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:15.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377643+0000", 2023-07-02T14:03:15.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043004239999999997 2023-07-02T14:03:15.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.381943+0000", 2023-07-02T14:03:15.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4321999999999999e-05 2023-07-02T14:03:15.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.381998+0000", 2023-07-02T14:03:15.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003008377 2023-07-02T14:03:15.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385006+0000", 2023-07-02T14:03:15.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8751999999999998e-05 2023-07-02T14:03:15.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385045+0000", 2023-07-02T14:03:15.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086260880000000005 2023-07-02T14:03:15.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393671+0000", 2023-07-02T14:03:15.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6365999999999997e-05 2023-07-02T14:03:15.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393707+0000", 2023-07-02T14:03:15.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028560289999999999 2023-07-02T14:03:15.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396563+0000", 2023-07-02T14:03:15.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6195e-05 2023-07-02T14:03:15.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396619+0000", 2023-07-02T14:03:15.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023118269999999998 2023-07-02T14:03:15.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.398931+0000", 2023-07-02T14:03:15.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3885999999999999e-05 2023-07-02T14:03:15.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.398985+0000", 2023-07-02T14:03:15.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043756890000000003 2023-07-02T14:03:15.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403361+0000", 2023-07-02T14:03:15.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0043000000000001e-05 2023-07-02T14:03:15.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403421+0000", 2023-07-02T14:03:15.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001590972 2023-07-02T14:03:15.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405012+0000", 2023-07-02T14:03:15.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0067000000000005e-05 2023-07-02T14:03:15.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405082+0000", 2023-07-02T14:03:15.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050958610000000001 2023-07-02T14:03:15.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410178+0000", 2023-07-02T14:03:15.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7070000000000004e-05 2023-07-02T14:03:15.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410275+0000", 2023-07-02T14:03:15.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060121100000000004 2023-07-02T14:03:15.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410876+0000", 2023-07-02T14:03:15.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3177999999999999e-05 2023-07-02T14:03:15.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:15.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410919+0000", 2023-07-02T14:03:15.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000271394 2023-07-02T14:03:15.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:15.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411191+0000", 2023-07-02T14:03:15.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001009598 2023-07-02T14:03:15.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:15.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.412200+0000", 2023-07-02T14:03:15.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014789790000000001 2023-07-02T14:03:15.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:15.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413679+0000", 2023-07-02T14:03:15.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2624000000000002e-05 2023-07-02T14:03:15.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:15.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413742+0000", 2023-07-02T14:03:15.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6564999999999994e-05 2023-07-02T14:03:15.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:15.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413838+0000", 2023-07-02T14:03:15.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036204721000000002 2023-07-02T14:03:15.458 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.458 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.458 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.459 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:473 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:15.459 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:15.459 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.823305984999998, 2023-07-02T14:03:15.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037915904, 2023-07-02T14:03:15.459 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.459 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.459 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:15.459 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:15.460 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:15.460 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 473 2023-07-02T14:03:15.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.460 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:15.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:15.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:03:15.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:15.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.119e-06 2023-07-02T14:03:15.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377647+0000", 2023-07-02T14:03:15.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.875e-06 2023-07-02T14:03:15.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377650+0000", 2023-07-02T14:03:15.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1949999999999999e-06 2023-07-02T14:03:15.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:15.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377653+0000", 2023-07-02T14:03:15.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004375444 2023-07-02T14:03:15.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382028+0000", 2023-07-02T14:03:15.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1063999999999997e-05 2023-07-02T14:03:15.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382089+0000", 2023-07-02T14:03:15.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030043869999999999 2023-07-02T14:03:15.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385094+0000", 2023-07-02T14:03:15.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8167999999999997e-05 2023-07-02T14:03:15.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385142+0000", 2023-07-02T14:03:15.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008603421 2023-07-02T14:03:15.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393745+0000", 2023-07-02T14:03:15.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3763000000000005e-05 2023-07-02T14:03:15.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393819+0000", 2023-07-02T14:03:15.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028274720000000001 2023-07-02T14:03:15.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396647+0000", 2023-07-02T14:03:15.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7929999999999997e-05 2023-07-02T14:03:15.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396695+0000", 2023-07-02T14:03:15.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023426459999999999 2023-07-02T14:03:15.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399037+0000", 2023-07-02T14:03:15.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5846000000000004e-05 2023-07-02T14:03:15.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399123+0000", 2023-07-02T14:03:15.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043448360000000004 2023-07-02T14:03:15.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403468+0000", 2023-07-02T14:03:15.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6699000000000001e-05 2023-07-02T14:03:15.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403515+0000", 2023-07-02T14:03:15.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001622943 2023-07-02T14:03:15.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405138+0000", 2023-07-02T14:03:15.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6382000000000002e-05 2023-07-02T14:03:15.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405204+0000", 2023-07-02T14:03:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051089910000000002 2023-07-02T14:03:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410313+0000", 2023-07-02T14:03:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8106000000000003e-05 2023-07-02T14:03:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410361+0000", 2023-07-02T14:03:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012200399999999999 2023-07-02T14:03:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411581+0000", 2023-07-02T14:03:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5067000000000005e-05 2023-07-02T14:03:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411656+0000", 2023-07-02T14:03:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022353249999999998 2023-07-02T14:03:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413891+0000", 2023-07-02T14:03:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9356000000000003e-05 2023-07-02T14:03:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413981+0000", 2023-07-02T14:03:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00043818299999999998 2023-07-02T14:03:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.414419+0000", 2023-07-02T14:03:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00081559199999999997 2023-07-02T14:03:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415235+0000", 2023-07-02T14:03:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026434499999999998 2023-07-02T14:03:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415499+0000", 2023-07-02T14:03:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9779999999999999e-05 2023-07-02T14:03:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415529+0000", 2023-07-02T14:03:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2159999999999997e-05 2023-07-02T14:03:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415561+0000", 2023-07-02T14:03:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037915904 2023-07-02T14:03:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:475 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.823295946000002, 2023-07-02T14:03:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039286962000000002, 2023-07-02T14:03:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 475 2023-07-02T14:03:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4779999999999997e-06 2023-07-02T14:03:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377664+0000", 2023-07-02T14:03:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.268e-06 2023-07-02T14:03:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377671+0000", 2023-07-02T14:03:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.861e-06 2023-07-02T14:03:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377674+0000", 2023-07-02T14:03:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044395789999999999 2023-07-02T14:03:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382114+0000", 2023-07-02T14:03:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8096999999999997e-05 2023-07-02T14:03:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382162+0000", 2023-07-02T14:03:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029949059999999999 2023-07-02T14:03:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385157+0000", 2023-07-02T14:03:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1158000000000001e-05 2023-07-02T14:03:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385198+0000", 2023-07-02T14:03:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086477229999999995 2023-07-02T14:03:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393846+0000", 2023-07-02T14:03:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5990999999999999e-05 2023-07-02T14:03:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393912+0000", 2023-07-02T14:03:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027979599999999999 2023-07-02T14:03:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396710+0000", 2023-07-02T14:03:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5911000000000002e-05 2023-07-02T14:03:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396746+0000", 2023-07-02T14:03:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024039759999999999 2023-07-02T14:03:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399150+0000", 2023-07-02T14:03:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7834999999999999e-05 2023-07-02T14:03:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399188+0000", 2023-07-02T14:03:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004364054 2023-07-02T14:03:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403552+0000", 2023-07-02T14:03:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6274999999999999e-05 2023-07-02T14:03:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403608+0000", 2023-07-02T14:03:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0016261540000000001 2023-07-02T14:03:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405234+0000", 2023-07-02T14:03:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6945000000000004e-05 2023-07-02T14:03:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405301+0000", 2023-07-02T14:03:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050871329999999998 2023-07-02T14:03:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410388+0000", 2023-07-02T14:03:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0210999999999995e-05 2023-07-02T14:03:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410468+0000", 2023-07-02T14:03:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001227454 2023-07-02T14:03:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411696+0000", 2023-07-02T14:03:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8188000000000005e-05 2023-07-02T14:03:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411764+0000", 2023-07-02T14:03:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031779930000000001 2023-07-02T14:03:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.414942+0000", 2023-07-02T14:03:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7236000000000004e-05 2023-07-02T14:03:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415019+0000", 2023-07-02T14:03:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000573045 2023-07-02T14:03:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415592+0000", 2023-07-02T14:03:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7580000000000002e-05 2023-07-02T14:03:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415640+0000", 2023-07-02T14:03:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024783200000000002 2023-07-02T14:03:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415888+0000", 2023-07-02T14:03:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00080440999999999998 2023-07-02T14:03:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416692+0000", 2023-07-02T14:03:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000195787 2023-07-02T14:03:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416888+0000", 2023-07-02T14:03:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0787999999999997e-05 2023-07-02T14:03:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416919+0000", 2023-07-02T14:03:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.338e-05 2023-07-02T14:03:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416942+0000", 2023-07-02T14:03:15.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039286962000000002 2023-07-02T14:03:15.503 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.503 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.503 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.503 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:15.503 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:15.503 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:15.505 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:03:15.505 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:15.504+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:15.605 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:15.605+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:15.678 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:15.679 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:15.679 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:03:15.679 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:15.689 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:03:15.730 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:15.730+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:15.750 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i++ )) 2023-07-02T14:03:15.750 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i < 10 )) 2023-07-02T14:03:15.751 INFO:tasks.workunit.client.0.smithi163.stderr:+ pv -L 10M linux-5.4.tar.gz 2023-07-02T14:03:15.807 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:15.806+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:15.852 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:15.853 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:15.853 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:15.853 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:15.853 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:15.864 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:03:15.908 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:15.909+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:15.990 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:15.991 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:15.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:15.991 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:15.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.992 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:1053 2.0 e57/46 2:1c3726b3:::rbd_data.125de1449605.000000000000002f:head v 57'164, mlcod=57'161)", 2023-07-02T14:03:15.992 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:05.578013+0000", 2023-07-02T14:03:15.992 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 10.410644528000001, 2023-07-02T14:03:15.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099458040000000008, 2023-07-02T14:03:15.993 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:15.993 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:15.993 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:15.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:15.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.578013+0000", 2023-07-02T14:03:15.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:15.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:15.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.578013+0000", 2023-07-02T14:03:15.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:15.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:15.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.578012+0000", 2023-07-02T14:03:15.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046313150000000004 2023-07-02T14:03:15.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:15.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582643+0000", 2023-07-02T14:03:15.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0795999999999999e-05 2023-07-02T14:03:15.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:15.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:15.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:15.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582664+0000", 2023-07-02T14:03:16.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2939999999999994e-06 2023-07-02T14:03:16.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582673+0000", 2023-07-02T14:03:16.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3214999999999998e-05 2023-07-02T14:03:16.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582726+0000", 2023-07-02T14:03:16.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8241999999999997e-05 2023-07-02T14:03:16.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582765+0000", 2023-07-02T14:03:16.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051597769999999999 2023-07-02T14:03:16.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.587924+0000", 2023-07-02T14:03:16.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4795999999999997e-05 2023-07-02T14:03:16.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.587959+0000", 2023-07-02T14:03:16.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099458040000000008 2023-07-02T14:03:16.006 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.007 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.007 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.008 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:1054 2.0 e57/46 2:1c3726b3:::rbd_data.125de1449605.000000000000002f:head v 57'165, mlcod=57'161)", 2023-07-02T14:03:16.008 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:05.582711+0000", 2023-07-02T14:03:16.008 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 10.405946809, 2023-07-02T14:03:16.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053281630000000003, 2023-07-02T14:03:16.009 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.009 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.009 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582711+0000", 2023-07-02T14:03:16.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582711+0000", 2023-07-02T14:03:16.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:16.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582710+0000", 2023-07-02T14:03:16.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0908000000000002e-05 2023-07-02T14:03:16.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582751+0000", 2023-07-02T14:03:16.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1470000000000007e-06 2023-07-02T14:03:16.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582760+0000", 2023-07-02T14:03:16.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1669999999999998e-06 2023-07-02T14:03:16.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582766+0000", 2023-07-02T14:03:16.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002795905 2023-07-02T14:03:16.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.585561+0000", 2023-07-02T14:03:16.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6288e-05 2023-07-02T14:03:16.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.585588+0000", 2023-07-02T14:03:16.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024248550000000001 2023-07-02T14:03:16.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.588013+0000", 2023-07-02T14:03:16.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6682e-05 2023-07-02T14:03:16.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.588039+0000", 2023-07-02T14:03:16.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053281630000000003 2023-07-02T14:03:16.022 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.022 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.022 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.023 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:1094 2.0 e57/46 2:1c3726b3:::rbd_data.125de1449605.000000000000002f:head v 57'170, mlcod=57'167)", 2023-07-02T14:03:16.023 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:05.595461+0000", 2023-07-02T14:03:16.024 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 10.393197242999999, 2023-07-02T14:03:16.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070844539999999996, 2023-07-02T14:03:16.024 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.024 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.025 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.595461+0000", 2023-07-02T14:03:16.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.595461+0000", 2023-07-02T14:03:16.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:16.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.595460+0000", 2023-07-02T14:03:16.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033907640000000001 2023-07-02T14:03:16.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.598850+0000", 2023-07-02T14:03:16.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8046e-05 2023-07-02T14:03:16.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.598868+0000", 2023-07-02T14:03:16.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1449e-05 2023-07-02T14:03:16.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.598880+0000", 2023-07-02T14:03:16.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0612000000000003e-05 2023-07-02T14:03:16.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.598931+0000", 2023-07-02T14:03:16.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.421e-05 2023-07-02T14:03:16.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.598955+0000", 2023-07-02T14:03:16.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0035568819999999999 2023-07-02T14:03:16.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.602512+0000", 2023-07-02T14:03:16.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3525999999999997e-05 2023-07-02T14:03:16.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.602545+0000", 2023-07-02T14:03:16.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070844539999999996 2023-07-02T14:03:16.037 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.038 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.038 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.039 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:1105 2.0 e57/46 2:1beb29b4:::rbd_data.125de1449605.0000000000000025:head v 57'173, mlcod=57'168)", 2023-07-02T14:03:16.039 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:05.599412+0000", 2023-07-02T14:03:16.039 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 10.389246139000001, 2023-07-02T14:03:16.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043199980000000002, 2023-07-02T14:03:16.040 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.040 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.040 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.599412+0000", 2023-07-02T14:03:16.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.599412+0000", 2023-07-02T14:03:16.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:16.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.599411+0000", 2023-07-02T14:03:16.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013543450000000001 2023-07-02T14:03:16.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.600765+0000", 2023-07-02T14:03:16.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4677e-05 2023-07-02T14:03:16.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.600780+0000", 2023-07-02T14:03:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7959999999999998e-06 2023-07-02T14:03:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.600787+0000", 2023-07-02T14:03:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00073057299999999999 2023-07-02T14:03:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.601517+0000", 2023-07-02T14:03:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4372e-05 2023-07-02T14:03:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.601542+0000", 2023-07-02T14:03:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0021526969999999999 2023-07-02T14:03:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.603695+0000", 2023-07-02T14:03:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7258000000000001e-05 2023-07-02T14:03:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.603732+0000", 2023-07-02T14:03:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043199980000000002 2023-07-02T14:03:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:1120 2.0 e57/46 2:1fa9788a:::rbd_data.125de1449605.0000000000000030:head v 57'177, mlcod=57'174)", 2023-07-02T14:03:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:05.608346+0000", 2023-07-02T14:03:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 10.380311733999999, 2023-07-02T14:03:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067962459999999997, 2023-07-02T14:03:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.608346+0000", 2023-07-02T14:03:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.608346+0000", 2023-07-02T14:03:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.608345+0000", 2023-07-02T14:03:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002673539 2023-07-02T14:03:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611019+0000", 2023-07-02T14:03:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8331000000000001e-05 2023-07-02T14:03:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611037+0000", 2023-07-02T14:03:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8879999999999996e-06 2023-07-02T14:03:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611042+0000", 2023-07-02T14:03:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7515999999999997e-05 2023-07-02T14:03:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611090+0000", 2023-07-02T14:03:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5330999999999999e-05 2023-07-02T14:03:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611115+0000", 2023-07-02T14:03:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039866559999999999 2023-07-02T14:03:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.615102+0000", 2023-07-02T14:03:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0865999999999998e-05 2023-07-02T14:03:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.615142+0000", 2023-07-02T14:03:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067962459999999997 2023-07-02T14:03:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:1121 2.0 e57/46 2:1fa9788a:::rbd_data.125de1449605.0000000000000030:head v 57'178, mlcod=57'174)", 2023-07-02T14:03:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:05.611066+0000", 2023-07-02T14:03:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 10.377592107, 2023-07-02T14:03:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041697710000000001, 2023-07-02T14:03:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611066+0000", 2023-07-02T14:03:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611066+0000", 2023-07-02T14:03:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611065+0000", 2023-07-02T14:03:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00090598600000000001 2023-07-02T14:03:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611971+0000", 2023-07-02T14:03:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2669e-05 2023-07-02T14:03:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611984+0000", 2023-07-02T14:03:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5730000000000001e-06 2023-07-02T14:03:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611989+0000", 2023-07-02T14:03:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060156399999999996 2023-07-02T14:03:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.612590+0000", 2023-07-02T14:03:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6229999999999999e-05 2023-07-02T14:03:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.612606+0000", 2023-07-02T14:03:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026060950000000001 2023-07-02T14:03:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.615212+0000", 2023-07-02T14:03:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3113999999999999e-05 2023-07-02T14:03:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.615236+0000", 2023-07-02T14:03:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041697710000000001 2023-07-02T14:03:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:1198 2.0 e57/46 2:107a9547:::rbd_data.125de1449605.000000000000003f:head v 57'195, mlcod=57'193)", 2023-07-02T14:03:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:05.668454+0000", 2023-07-02T14:03:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 10.320203438, 2023-07-02T14:03:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0087668139999999995, 2023-07-02T14:03:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.668454+0000", 2023-07-02T14:03:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.668454+0000", 2023-07-02T14:03:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.668453+0000", 2023-07-02T14:03:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045315149999999998 2023-07-02T14:03:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.672985+0000", 2023-07-02T14:03:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1236000000000001e-05 2023-07-02T14:03:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.673006+0000", 2023-07-02T14:03:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.776e-06 2023-07-02T14:03:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.673013+0000", 2023-07-02T14:03:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1847e-05 2023-07-02T14:03:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.673065+0000", 2023-07-02T14:03:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8389999999999998e-05 2023-07-02T14:03:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.673083+0000", 2023-07-02T14:03:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041092359999999996 2023-07-02T14:03:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.677192+0000", 2023-07-02T14:03:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9028999999999999e-05 2023-07-02T14:03:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.677221+0000", 2023-07-02T14:03:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0087668139999999995 2023-07-02T14:03:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:1199 2.0 e57/46 2:107a9547:::rbd_data.125de1449605.000000000000003f:head v 57'196, mlcod=57'193)", 2023-07-02T14:03:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:05.673219+0000", 2023-07-02T14:03:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 10.315438921, 2023-07-02T14:03:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043126400000000004, 2023-07-02T14:03:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.673219+0000", 2023-07-02T14:03:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.673219+0000", 2023-07-02T14:03:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.673218+0000", 2023-07-02T14:03:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8931999999999996e-05 2023-07-02T14:03:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.673297+0000", 2023-07-02T14:03:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2189999999999998e-06 2023-07-02T14:03:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.673304+0000", 2023-07-02T14:03:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4929999999999998e-06 2023-07-02T14:03:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.673308+0000", 2023-07-02T14:03:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023961500000000001 2023-07-02T14:03:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.675704+0000", 2023-07-02T14:03:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6027000000000001e-05 2023-07-02T14:03:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.675720+0000", 2023-07-02T14:03:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785841 2023-07-02T14:03:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.677506+0000", 2023-07-02T14:03:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5726000000000001e-05 2023-07-02T14:03:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.677532+0000", 2023-07-02T14:03:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043126400000000004 2023-07-02T14:03:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:1226 2.0 e57/46 2:047f2192:::rbd_data.125de1449605.0000000000000041:head v 57'206, mlcod=57'203)", 2023-07-02T14:03:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:05.703321+0000", 2023-07-02T14:03:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 10.285336791000001, 2023-07-02T14:03:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006518228, 2023-07-02T14:03:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.703321+0000", 2023-07-02T14:03:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.703321+0000", 2023-07-02T14:03:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.703321+0000", 2023-07-02T14:03:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031170590000000001 2023-07-02T14:03:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.706438+0000", 2023-07-02T14:03:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1039999999999998e-05 2023-07-02T14:03:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.706459+0000", 2023-07-02T14:03:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3070000000000008e-06 2023-07-02T14:03:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.706468+0000", 2023-07-02T14:03:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0488000000000002e-05 2023-07-02T14:03:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.706519+0000", 2023-07-02T14:03:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7299000000000002e-05 2023-07-02T14:03:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.706536+0000", 2023-07-02T14:03:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032408950000000001 2023-07-02T14:03:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.709777+0000", 2023-07-02T14:03:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2612000000000003e-05 2023-07-02T14:03:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.709839+0000", 2023-07-02T14:03:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006518228 2023-07-02T14:03:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:1270 2.0 e57/46 2:169af44a:::rbd_data.125de1449605.0000000000000106:head v 57'216, mlcod=57'212)", 2023-07-02T14:03:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:06.420574+0000", 2023-07-02T14:03:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.5680834650000008, 2023-07-02T14:03:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055088890000000003, 2023-07-02T14:03:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:06.420574+0000", 2023-07-02T14:03:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:06.420574+0000", 2023-07-02T14:03:16.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:16.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:06.420573+0000", 2023-07-02T14:03:16.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001758765 2023-07-02T14:03:16.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:06.422332+0000", 2023-07-02T14:03:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5381000000000001e-05 2023-07-02T14:03:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:06.422348+0000", 2023-07-02T14:03:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1210000000000001e-06 2023-07-02T14:03:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:06.422355+0000", 2023-07-02T14:03:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0161999999999998e-05 2023-07-02T14:03:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:06.422405+0000", 2023-07-02T14:03:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5510000000000001e-05 2023-07-02T14:03:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:06.422430+0000", 2023-07-02T14:03:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036230210000000001 2023-07-02T14:03:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:06.426053+0000", 2023-07-02T14:03:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0014000000000001e-05 2023-07-02T14:03:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:06.426083+0000", 2023-07-02T14:03:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055088890000000003 2023-07-02T14:03:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:2840 2.0 e61/46 2:1c81f085:::rbd_data.125de1449605.00000000000000f0:head v 61'639, mlcod=61'637)", 2023-07-02T14:03:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:11.368484+0000", 2023-07-02T14:03:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.6201735499999996, 2023-07-02T14:03:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038949549999999999, 2023-07-02T14:03:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.368484+0000", 2023-07-02T14:03:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.368484+0000", 2023-07-02T14:03:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.368483+0000", 2023-07-02T14:03:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00099835700000000006 2023-07-02T14:03:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.369481+0000", 2023-07-02T14:03:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7671000000000001e-05 2023-07-02T14:03:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.369499+0000", 2023-07-02T14:03:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7360000000000004e-06 2023-07-02T14:03:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.369505+0000", 2023-07-02T14:03:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3460000000000001e-05 2023-07-02T14:03:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.369549+0000", 2023-07-02T14:03:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9530000000000001e-05 2023-07-02T14:03:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.369568+0000", 2023-07-02T14:03:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027731679999999999 2023-07-02T14:03:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.372342+0000", 2023-07-02T14:03:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7713000000000003e-05 2023-07-02T14:03:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.372379+0000", 2023-07-02T14:03:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038949549999999999 2023-07-02T14:03:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:2863 2.0 e61/46 2:1c81f085:::rbd_data.125de1449605.00000000000000f0:head v 61'641, mlcod=61'639)", 2023-07-02T14:03:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:11.383026+0000", 2023-07-02T14:03:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.6056318559999996, 2023-07-02T14:03:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037956190000000001, 2023-07-02T14:03:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.383026+0000", 2023-07-02T14:03:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.383026+0000", 2023-07-02T14:03:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.383025+0000", 2023-07-02T14:03:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012165629999999999 2023-07-02T14:03:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.384241+0000", 2023-07-02T14:03:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9899999999999999e-05 2023-07-02T14:03:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.384261+0000", 2023-07-02T14:03:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6300000000000004e-06 2023-07-02T14:03:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.384270+0000", 2023-07-02T14:03:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6109999999999997e-05 2023-07-02T14:03:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.384316+0000", 2023-07-02T14:03:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4791999999999999e-05 2023-07-02T14:03:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.384341+0000", 2023-07-02T14:03:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002455357 2023-07-02T14:03:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.386796+0000", 2023-07-02T14:03:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5592000000000002e-05 2023-07-02T14:03:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.386822+0000", 2023-07-02T14:03:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037956190000000001 2023-07-02T14:03:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:2896 2.0 e61/46 2:08f47745:::rbd_data.125de1449605.00000000000000eb:head v 61'652, mlcod=61'648)", 2023-07-02T14:03:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:11.422798+0000", 2023-07-02T14:03:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.5658596210000004, 2023-07-02T14:03:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077822680000000002, 2023-07-02T14:03:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.422798+0000", 2023-07-02T14:03:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.422798+0000", 2023-07-02T14:03:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.422797+0000", 2023-07-02T14:03:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030387439999999999 2023-07-02T14:03:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.425836+0000", 2023-07-02T14:03:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1531000000000001e-05 2023-07-02T14:03:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.425857+0000", 2023-07-02T14:03:16.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3219999999999999e-06 2023-07-02T14:03:16.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.425867+0000", 2023-07-02T14:03:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0667000000000001e-05 2023-07-02T14:03:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.425917+0000", 2023-07-02T14:03:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8943999999999999e-05 2023-07-02T14:03:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.425946+0000", 2023-07-02T14:03:16.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046048809999999999 2023-07-02T14:03:16.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.430551+0000", 2023-07-02T14:03:16.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9351000000000001e-05 2023-07-02T14:03:16.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.430581+0000", 2023-07-02T14:03:16.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077822680000000002 2023-07-02T14:03:16.119 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.119 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.119 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.120 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:2958 2.0 e61/46 2:187db15e:::rbd_data.125de1449605.00000000000000fc:head v 61'667, mlcod=61'665)", 2023-07-02T14:03:16.120 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:11.493615+0000", 2023-07-02T14:03:16.120 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.4950432219999996, 2023-07-02T14:03:16.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057044310000000003, 2023-07-02T14:03:16.120 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.120 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.121 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.493615+0000", 2023-07-02T14:03:16.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.493615+0000", 2023-07-02T14:03:16.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999943 2023-07-02T14:03:16.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.493609+0000", 2023-07-02T14:03:16.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020804550000000002 2023-07-02T14:03:16.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.495689+0000", 2023-07-02T14:03:16.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4980000000000001e-05 2023-07-02T14:03:16.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.495714+0000", 2023-07-02T14:03:16.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1089999999999998e-06 2023-07-02T14:03:16.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.495722+0000", 2023-07-02T14:03:16.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5745999999999999e-05 2023-07-02T14:03:16.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.495768+0000", 2023-07-02T14:03:16.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9123999999999999e-05 2023-07-02T14:03:16.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.495787+0000", 2023-07-02T14:03:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0035009989999999999 2023-07-02T14:03:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.499288+0000", 2023-07-02T14:03:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0925e-05 2023-07-02T14:03:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.499319+0000", 2023-07-02T14:03:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057044310000000003 2023-07-02T14:03:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.128 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:3017 2.0 e61/46 2:1f6e35fd:::rbd_data.125de1449605.0000000000000111:head v 61'683, mlcod=61'680)", 2023-07-02T14:03:16.128 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:11.548521+0000", 2023-07-02T14:03:16.128 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.4401372849999996, 2023-07-02T14:03:16.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006595465, 2023-07-02T14:03:16.128 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.128 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.128 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.548521+0000", 2023-07-02T14:03:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.548521+0000", 2023-07-02T14:03:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.548520+0000", 2023-07-02T14:03:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027544179999999998 2023-07-02T14:03:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.551274+0000", 2023-07-02T14:03:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.451e-05 2023-07-02T14:03:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.551289+0000", 2023-07-02T14:03:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4489999999999996e-06 2023-07-02T14:03:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.551295+0000", 2023-07-02T14:03:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2367000000000003e-05 2023-07-02T14:03:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.551338+0000", 2023-07-02T14:03:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6847999999999999e-05 2023-07-02T14:03:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.551355+0000", 2023-07-02T14:03:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037245009999999999 2023-07-02T14:03:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.555079+0000", 2023-07-02T14:03:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6934999999999998e-05 2023-07-02T14:03:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.555116+0000", 2023-07-02T14:03:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006595465 2023-07-02T14:03:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:3028 2.0 e61/46 2:014344e8:::rbd_data.125de1449605.0000000000000114:head v 61'690, mlcod=61'685)", 2023-07-02T14:03:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:11.562915+0000", 2023-07-02T14:03:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.4257430339999999, 2023-07-02T14:03:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075509189999999997, 2023-07-02T14:03:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.562915+0000", 2023-07-02T14:03:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.562915+0000", 2023-07-02T14:03:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.562914+0000", 2023-07-02T14:03:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032299809999999998 2023-07-02T14:03:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.566144+0000", 2023-07-02T14:03:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1277999999999999e-05 2023-07-02T14:03:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.566165+0000", 2023-07-02T14:03:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6469999999999993e-06 2023-07-02T14:03:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.566175+0000", 2023-07-02T14:03:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0241000000000003e-05 2023-07-02T14:03:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.566215+0000", 2023-07-02T14:03:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2554000000000001e-05 2023-07-02T14:03:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.566238+0000", 2023-07-02T14:03:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041783059999999997 2023-07-02T14:03:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.570416+0000", 2023-07-02T14:03:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9579000000000001e-05 2023-07-02T14:03:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.570466+0000", 2023-07-02T14:03:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075509189999999997 2023-07-02T14:03:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.7 62/60 [PushOp(2:f30916f5:::rbd_data.125de1449605.0000000000000075:head, version: 58'249, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:f30916f5:::rbd_data.125de1449605.0000000000000075:head@58'249, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-07-02T14:03:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:12.091624+0000", 2023-07-02T14:03:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.8970335469999999, 2023-07-02T14:03:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040768849999999997, 2023-07-02T14:03:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.091624+0000", 2023-07-02T14:03:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.091624+0000", 2023-07-02T14:03:16.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.091624+0000", 2023-07-02T14:03:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024253930000000001 2023-07-02T14:03:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094049+0000", 2023-07-02T14:03:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1549999999999999e-05 2023-07-02T14:03:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094071+0000", 2023-07-02T14:03:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5319999999999995e-06 2023-07-02T14:03:16.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094080+0000", 2023-07-02T14:03:16.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1773999999999993e-05 2023-07-02T14:03:16.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094162+0000", 2023-07-02T14:03:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5380000000000008e-06 2023-07-02T14:03:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094172+0000", 2023-07-02T14:03:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0015297150000000001 2023-07-02T14:03:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.095701+0000", 2023-07-02T14:03:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040768849999999997 2023-07-02T14:03:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:3073 2.0 e62/46 2:1f6e35fd:::rbd_data.125de1449605.0000000000000111:head v 62'700, mlcod=62'697)", 2023-07-02T14:03:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:12.104450+0000", 2023-07-02T14:03:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.884207784, 2023-07-02T14:03:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049432629999999998, 2023-07-02T14:03:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.104450+0000", 2023-07-02T14:03:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.104450+0000", 2023-07-02T14:03:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.104449+0000", 2023-07-02T14:03:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0015084779999999999 2023-07-02T14:03:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.105958+0000", 2023-07-02T14:03:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0789e-05 2023-07-02T14:03:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.105969+0000", 2023-07-02T14:03:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3319999999999999e-06 2023-07-02T14:03:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.105974+0000", 2023-07-02T14:03:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1780999999999999e-05 2023-07-02T14:03:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.106016+0000", 2023-07-02T14:03:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8377e-05 2023-07-02T14:03:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.106034+0000", 2023-07-02T14:03:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033141310000000001 2023-07-02T14:03:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.109348+0000", 2023-07-02T14:03:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5173000000000003e-05 2023-07-02T14:03:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.109393+0000", 2023-07-02T14:03:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049432629999999998 2023-07-02T14:03:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:3081 2.0 e62/46 2:014344e8:::rbd_data.125de1449605.0000000000000114:head v 62'705, mlcod=62'702)", 2023-07-02T14:03:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:12.118968+0000", 2023-07-02T14:03:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.8696902959999999, 2023-07-02T14:03:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042841040000000004, 2023-07-02T14:03:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.118968+0000", 2023-07-02T14:03:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.118968+0000", 2023-07-02T14:03:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.118967+0000", 2023-07-02T14:03:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013399429999999999 2023-07-02T14:03:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.120307+0000", 2023-07-02T14:03:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1103e-05 2023-07-02T14:03:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.120318+0000", 2023-07-02T14:03:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3369999999999999e-06 2023-07-02T14:03:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.120323+0000", 2023-07-02T14:03:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1192999999999998e-05 2023-07-02T14:03:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.120354+0000", 2023-07-02T14:03:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.109e-05 2023-07-02T14:03:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.120376+0000", 2023-07-02T14:03:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028409590000000001 2023-07-02T14:03:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.123216+0000", 2023-07-02T14:03:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5249000000000001e-05 2023-07-02T14:03:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.123252+0000", 2023-07-02T14:03:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042841040000000004 2023-07-02T14:03:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.7 62/60 [PushOp(2:eed8de36:::rbd_data.125de1449605.0000000000000073:head, version: 58'251, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:eed8de36:::rbd_data.125de1449605.0000000000000073:head@58'251, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:03:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:12.132531+0000", 2023-07-02T14:03:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.8561269149999999, 2023-07-02T14:03:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037686489999999998, 2023-07-02T14:03:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.132531+0000", 2023-07-02T14:03:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.132531+0000", 2023-07-02T14:03:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.132530+0000", 2023-07-02T14:03:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00096131099999999996 2023-07-02T14:03:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.133491+0000", 2023-07-02T14:03:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0018e-05 2023-07-02T14:03:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.133511+0000", 2023-07-02T14:03:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4400000000000005e-06 2023-07-02T14:03:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.133520+0000", 2023-07-02T14:03:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3308999999999999e-05 2023-07-02T14:03:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.133593+0000", 2023-07-02T14:03:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1190000000000003e-06 2023-07-02T14:03:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.133602+0000", 2023-07-02T14:03:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026974099999999999 2023-07-02T14:03:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.136300+0000", 2023-07-02T14:03:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037686489999999998 2023-07-02T14:03:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.174 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.174 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.174 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.174 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:16.175 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:03:16.176 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:03:16.176 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:16.175+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:16.276 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:16.275+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:16.342 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:16.342 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:16.342 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:03:16.342 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:16.352 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:03:16.377 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:16.376+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:16.476 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:16.476 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:16.476 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:16.477 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:16.477 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:16.478 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:16.476+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:16.486 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:03:16.498 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 100% complete...done. 2023-07-02T14:03:16.503 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 1 2023-07-02T14:03:16.579 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:16.578+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:16.612 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:16.612 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:16.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:16.612 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:16.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.612 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:16.613 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:16.613 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.319331536, 2023-07-02T14:03:16.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:03:16.613 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.613 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.613 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:16.613 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:16.614 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:16.614 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:03:16.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.614 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:16.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.615 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:16.615 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:16.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.616 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:03:16.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:03:16.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.616 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.616 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.616 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:03:16.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:03:16.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:03:16.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:03:16.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:03:16.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:03:16.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:03:16.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:03:16.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:03:16.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:03:16.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:16.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:03:16.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:03:16.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:16.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:03:16.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:03:16.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:16.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:03:16.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:03:16.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:03:16.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:03:16.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:03:16.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:03:16.622 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.622 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.622 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.623 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:16.623 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:16.623 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.319096512999998, 2023-07-02T14:03:16.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:03:16.623 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.624 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.624 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:16.624 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:16.624 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:16.624 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:03:16.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.624 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.625 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:16.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.625 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:16.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:16.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.626 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:03:16.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:03:16.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:03:16.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:03:16.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:03:16.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:03:16.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:03:16.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:03:16.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:03:16.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:03:16.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:03:16.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:03:16.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:16.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:03:16.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:03:16.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:16.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:03:16.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:03:16.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:16.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:03:16.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:03:16.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:03:16.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:03:16.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:03:16.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:03:16.633 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.633 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.633 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.633 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:16.633 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:16.633 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.318897378999999, 2023-07-02T14:03:16.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:03:16.634 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.634 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.634 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:16.634 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:16.634 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:16.634 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:03:16.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.635 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:16.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:16.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:16.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:03:16.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:03:16.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:03:16.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:03:16.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:03:16.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:03:16.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:03:16.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:03:16.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:03:16.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:03:16.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:03:16.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:03:16.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:16.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:03:16.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:03:16.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:16.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:03:16.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:03:16.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:16.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:03:16.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:03:16.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:03:16.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:03:16.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:03:16.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:03:16.643 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.643 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.643 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.643 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:16.644 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:16.644 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.318532255000001, 2023-07-02T14:03:16.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:03:16.644 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.644 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.644 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:16.644 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:16.644 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:16.645 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:03:16.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.645 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:16.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:16.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:16.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:03:16.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:03:16.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:03:16.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:03:16.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:03:16.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:03:16.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:03:16.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:03:16.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:03:16.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:03:16.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:03:16.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:03:16.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:16.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:03:16.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:03:16.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:16.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:03:16.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:03:16.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:16.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:03:16.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:03:16.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:03:16.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:03:16.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:03:16.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:03:16.653 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.653 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.653 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.654 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:16.654 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:16.654 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.318451193999998, 2023-07-02T14:03:16.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:03:16.654 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.654 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.654 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:16.655 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:16.655 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:16.655 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:03:16.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.655 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:16.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:16.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:16.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:03:16.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:03:16.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:03:16.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:03:16.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:03:16.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:03:16.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:03:16.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:03:16.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:03:16.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:03:16.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:03:16.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:03:16.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:16.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:03:16.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:03:16.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:16.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:03:16.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:03:16.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:16.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:03:16.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:03:16.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:03:16.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:03:16.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:03:16.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:03:16.663 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.663 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.664 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.664 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:16.664 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:16.664 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.318359522999998, 2023-07-02T14:03:16.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:03:16.664 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.665 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.665 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:16.665 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:16.665 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:16.665 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:03:16.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.665 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:16.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:16.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:16.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:03:16.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:03:16.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:03:16.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:03:16.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:03:16.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:03:16.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:03:16.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:03:16.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:03:16.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:03:16.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:03:16.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:03:16.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:16.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:03:16.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:03:16.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:16.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:03:16.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:03:16.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:16.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:03:16.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:03:16.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:03:16.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:03:16.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:03:16.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:03:16.674 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.674 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.674 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.674 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:16.674 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:16.674 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.318020081, 2023-07-02T14:03:16.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:03:16.675 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.675 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.675 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:16.675 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:16.675 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:16.675 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:03:16.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.676 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:16.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:16.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:16.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:03:16.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:03:16.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:03:16.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:03:16.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:03:16.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:03:16.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:03:16.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:03:16.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:03:16.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:03:16.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:03:16.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:03:16.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:16.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:03:16.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:03:16.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:16.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:03:16.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:03:16.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:16.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:03:16.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:03:16.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:03:16.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:03:16.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:03:16.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:03:16.684 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.684 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.684 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.684 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:16.685 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:16.685 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.317834028, 2023-07-02T14:03:16.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:03:16.685 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.685 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.685 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:16.685 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:16.685 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:16.686 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:03:16.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.686 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:16.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:16.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:16.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:16.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:03:16.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:03:16.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:03:16.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:03:16.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:03:16.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:03:16.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:03:16.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:03:16.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:03:16.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:03:16.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:03:16.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:16.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:03:16.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:03:16.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:16.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:03:16.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:03:16.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:16.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:03:16.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:03:16.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:03:16.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:03:16.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:03:16.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:03:16.694 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.694 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.694 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.695 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:16.695 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:16.695 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.317676226000003, 2023-07-02T14:03:16.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:03:16.695 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.695 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.695 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:16.696 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:16.696 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:16.696 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:03:16.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.696 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:16.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:16.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:16.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:03:16.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:03:16.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:03:16.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:03:16.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:03:16.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:03:16.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:03:16.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:03:16.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:03:16.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:03:16.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:03:16.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:03:16.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:16.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:03:16.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:03:16.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:16.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:03:16.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:03:16.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:16.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:03:16.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:03:16.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:03:16.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:03:16.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:03:16.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:03:16.704 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.704 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.705 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.705 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:16.705 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:16.705 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.317484444000002, 2023-07-02T14:03:16.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:03:16.705 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.706 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.706 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:16.706 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:16.706 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:16.706 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:03:16.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.706 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:16.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:16.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:16.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:03:16.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:03:16.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:03:16.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:03:16.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:03:16.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:03:16.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:03:16.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:03:16.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:03:16.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:03:16.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:03:16.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:03:16.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:16.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:03:16.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:03:16.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:16.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:03:16.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:03:16.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:16.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:03:16.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:03:16.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:03:16.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:03:16.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:03:16.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:03:16.715 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.715 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.715 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.715 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:16.715 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:16.715 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.317308783999998, 2023-07-02T14:03:16.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:03:16.716 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.716 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.716 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:16.716 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:16.716 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:16.716 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:03:16.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.717 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:16.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:16.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:16.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:16.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:03:16.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:03:16.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:03:16.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:03:16.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:03:16.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:03:16.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:03:16.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:03:16.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:03:16.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:03:16.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:03:16.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:16.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:03:16.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:03:16.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:16.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:03:16.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:03:16.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:16.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:03:16.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:03:16.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:03:16.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:03:16.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:03:16.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:03:16.725 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.725 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.725 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.725 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:16.726 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:16.726 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.317214595999999, 2023-07-02T14:03:16.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:03:16.726 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.726 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.726 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:16.726 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:16.726 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:16.727 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:03:16.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.727 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:16.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:16.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:16.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:03:16.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:03:16.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:03:16.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:03:16.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:03:16.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:03:16.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:03:16.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:03:16.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:03:16.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:03:16.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:03:16.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:03:16.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:16.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:03:16.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:03:16.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:16.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:03:16.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:03:16.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:16.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:03:16.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:03:16.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:03:16.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:03:16.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:03:16.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:03:16.735 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.735 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.735 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.736 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:16.736 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:16.736 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.317011209999997, 2023-07-02T14:03:16.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:03:16.736 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.736 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.736 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:16.737 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:16.737 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:16.737 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:03:16.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.737 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:16.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:16.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:16.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:03:16.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:03:16.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:03:16.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:03:16.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:03:16.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:03:16.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:03:16.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:03:16.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:03:16.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:03:16.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:03:16.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:03:16.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:16.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:03:16.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:03:16.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:16.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:03:16.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:03:16.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:16.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:03:16.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:03:16.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:03:16.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:03:16.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:03:16.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:03:16.745 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.745 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.745 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.746 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:16.746 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:16.746 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.316860554000002, 2023-07-02T14:03:16.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:03:16.746 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.747 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.747 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:16.747 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:16.747 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:16.747 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:03:16.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.747 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:16.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:16.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:16.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:03:16.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:03:16.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:03:16.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:03:16.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:03:16.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:03:16.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:03:16.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:03:16.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:03:16.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:03:16.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:03:16.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:03:16.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:16.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:03:16.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:03:16.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:16.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:03:16.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:03:16.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:16.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:03:16.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:03:16.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:03:16.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:03:16.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:03:16.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:03:16.756 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.756 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.756 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.756 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:16.756 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:16.756 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.316787851000001, 2023-07-02T14:03:16.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:03:16.757 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.757 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.757 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:16.757 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:16.757 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:16.757 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:03:16.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.758 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:16.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:16.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:16.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:16.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:03:16.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:03:16.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:03:16.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:03:16.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:03:16.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:03:16.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:03:16.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:03:16.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:03:16.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:03:16.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:03:16.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:16.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:03:16.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:03:16.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:16.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:03:16.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:03:16.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:16.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:03:16.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:03:16.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:03:16.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:03:16.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:03:16.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:03:16.766 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.766 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.766 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.766 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:16.767 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:16.767 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.316670860999999, 2023-07-02T14:03:16.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:03:16.767 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.767 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.767 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:16.767 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:16.767 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:16.768 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:03:16.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.768 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:16.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:16.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:16.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:16.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:03:16.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:03:16.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:03:16.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:03:16.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:03:16.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:03:16.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:03:16.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:03:16.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:03:16.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:03:16.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:03:16.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:16.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:03:16.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:03:16.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:16.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:03:16.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:03:16.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:16.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:03:16.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:03:16.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:03:16.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:03:16.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:03:16.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:03:16.776 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.776 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.776 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.777 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:16.777 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:16.777 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.31652141, 2023-07-02T14:03:16.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:03:16.777 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.777 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.777 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:16.777 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:16.778 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:16.778 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:03:16.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.778 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:16.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:16.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:16.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:03:16.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:03:16.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:03:16.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:03:16.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:03:16.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:03:16.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:03:16.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:03:16.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:03:16.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:03:16.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:03:16.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:03:16.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:16.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:03:16.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:03:16.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:16.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:03:16.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:03:16.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:16.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:03:16.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:03:16.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:03:16.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:03:16.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:03:16.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:03:16.786 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.786 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.786 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.787 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:16.787 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:16.787 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.295045436999999, 2023-07-02T14:03:16.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:03:16.787 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.787 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.788 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:16.788 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:16.788 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:16.788 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:03:16.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.788 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:16.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:16.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:16.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:03:16.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:03:16.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:03:16.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:03:16.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:03:16.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:03:16.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:03:16.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:03:16.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:03:16.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:03:16.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:03:16.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:03:16.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:16.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:03:16.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:03:16.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:16.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:03:16.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:03:16.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:16.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:03:16.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:03:16.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:03:16.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:03:16.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:03:16.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:03:16.796 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.797 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.797 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.797 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:16.797 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:16.797 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.294912900999996, 2023-07-02T14:03:16.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:03:16.798 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.798 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.798 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:16.798 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:16.798 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:16.798 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:03:16.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.798 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:16.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:16.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:16.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:03:16.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:03:16.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:03:16.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:03:16.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:03:16.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:03:16.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:03:16.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:03:16.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:03:16.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:03:16.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:03:16.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:03:16.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:16.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:03:16.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:03:16.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:16.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:03:16.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:03:16.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:16.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:03:16.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:03:16.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:03:16.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:03:16.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:03:16.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:03:16.807 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.807 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.807 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.807 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:16.808 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:16.808 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.233576239999998, 2023-07-02T14:03:16.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:03:16.808 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:16.808 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:16.808 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:16.808 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:16.808 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:16.809 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:03:16.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.809 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:16.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:16.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:16.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:16.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:16.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:16.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:16.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:16.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:03:16.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:03:16.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:16.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:03:16.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:03:16.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:16.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:03:16.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:03:16.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:16.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:03:16.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:03:16.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:16.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:03:16.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:03:16.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:16.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:03:16.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:03:16.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:16.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:03:16.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:03:16.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:16.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:03:16.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:03:16.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:16.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:03:16.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:03:16.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:16.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:03:16.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:03:16.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:16.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:16.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:16.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:03:16.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:03:16.817 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.817 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.817 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.817 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:16.818 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:16.818 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:16.818 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:03:16.819 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:16.818+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:16.820 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:03:16.919 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:16.918+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:16.948 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:16.948 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:16.948 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:16.949 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:16.959 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:03:16.961 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:16.979 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:16.996 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:17.014 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:17.020 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:17.019+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:17.033 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:17.050 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:17.052 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:03:17.052 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:03:17.052 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:03:17.052 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:03:17.052 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:03:17.053 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:03:17.087 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:17.087 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:17.097 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:03:17.138 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:17.137+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:17.223 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:17.223+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:17.260 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:17.261 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:17.271 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:03:17.323 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:17.322+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:17.336 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:03:17.399 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:17.400 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:17.409 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:03:17.453 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:17.452+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:17.504 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap4 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap5 b3fbb028-e267-4d25-a750-93ba18a2316a@snap5.diff 2023-07-02T14:03:17.525 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:17.525+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:17.533 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:17.533 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:17.544 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:03:17.567 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:03:17.567+0000 7f1e9172b700 0 -- 172.21.15.163:0/1637143032 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x5589383b3480 msgr2=0x5589383b5870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:03:17.572 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:03:17.572+0000 7f1e91f2c700 0 -- 172.21.15.163:0/1637143032 >> [v2:172.21.15.22:6808/83350,v1:172.21.15.22:6809/83350] conn(0x7f1e70008780 msgr2=0x7f1e7000abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:03:17.626 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:17.626+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:17.726 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:17.725+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:17.741 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:17.741 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:17.754 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:03:17.839 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:17.838+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:17.923 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:17.924 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:17.928 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:17.928+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:17.935 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:03:18.028 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:18.029+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:18.118 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:18.118 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:18.129 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:18.130+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:18.130 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:03:18.231 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:18.231+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:18.262 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:18.326 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:18.327 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:03:18.331 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:18.330+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:18.404 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete...2023-07-02T14:03:18.402+0000 7f1e91f2c700 0 -- 172.21.15.163:0/1637143032 >> [v2:172.21.15.22:6808/83350,v1:172.21.15.22:6809/83350] conn(0x5589383b3480 msgr2=0x7f1e7403b730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:03:18.411 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:03:18.410+0000 7f1e9172b700 0 -- 172.21.15.163:0/1637143032 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7f1e7000e9b0 msgr2=0x7f1e70010de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:03:18.414 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:03:18.413+0000 7f1e91f2c700 0 -- 172.21.15.163:0/1637143032 >> [v2:172.21.15.22:6808/83350,v1:172.21.15.22:6809/83350] conn(0x7f1e70008780 msgr2=0x7f1e7403bc20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:03:18.431 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:18.432+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:18.466 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:18.466 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:18.477 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:03:18.493 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 37% complete...2023-07-02T14:03:18.492+0000 7f1e91f2c700 0 -- 172.21.15.163:0/1637143032 >> [v2:172.21.15.22:6808/83350,v1:172.21.15.22:6809/83350] conn(0x7f1e7000e9b0 msgr2=0x7f1e7403c110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:03:18.533 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:18.532+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:18.634 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:18.634+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:18.640 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:18.640 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:18.650 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:03:18.664 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 100% complete...done. 2023-07-02T14:03:18.670 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:03:18.671 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=6 2023-07-02T14:03:18.671 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap6 2023-07-02T14:03:18.735 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:18.733+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:18.775 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:18.776 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:18.785 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:03:18.836 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:18.835+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:18.909 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:18.909 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:18.919 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:03:18.936 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:18.936+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:19.037 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:19.037+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:19.042 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:19.042 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:19.052 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:03:19.065 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:03:19.138 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:19.139+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:19.175 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:19.175 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:19.185 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:03:19.210 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:19.228 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:19.245 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:19.263 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:19.267 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:19.268+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:19.281 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:19.299 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:19.300 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:03:19.300 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:03:19.300 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:03:19.301 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:03:19.301 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:03:19.301 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:03:19.341 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:19.340+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:19.347 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:03:19.351 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:19.351 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:19.351 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:03:19.351 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:19.359 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:03:19.362 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:03:19.363 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 4 2023-07-02T14:03:19.447 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:19.447+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:19.526 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:19.527 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:19.527 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:19.527 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:19.527 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:19.537 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:03:19.543 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:19.542+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:19.644 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:19.643+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:19.702 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:19.703 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:19.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:19.703 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:19.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.703 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:19.703 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:19.703 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.410042605999998, 2023-07-02T14:03:19.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:03:19.704 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:19.704 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:19.704 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:19.704 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:19.704 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:19.704 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:03:19.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.705 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:19.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:19.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:19.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:19.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:19.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:19.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:19.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:19.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:03:19.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:03:19.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:19.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:03:19.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:03:19.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:19.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:03:19.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:03:19.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:19.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:03:19.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:03:19.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:19.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:03:19.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:03:19.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:19.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:03:19.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:03:19.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:19.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:03:19.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:03:19.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:19.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:03:19.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:03:19.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:19.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:03:19.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:03:19.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:19.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:03:19.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:03:19.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:19.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:03:19.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:03:19.713 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.713 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.713 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.714 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:19.714 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:19.714 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.409807583000003, 2023-07-02T14:03:19.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:03:19.714 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:19.714 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:19.715 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:19.715 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:19.715 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:19.715 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:03:19.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.715 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:19.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:19.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:19.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:19.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:19.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:19.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:19.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:19.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:03:19.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:03:19.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:19.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:03:19.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:03:19.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:19.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:03:19.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:03:19.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:19.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:03:19.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:03:19.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:19.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:03:19.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:03:19.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:19.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:03:19.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:03:19.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:19.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:03:19.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:03:19.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:19.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:03:19.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:03:19.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:19.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:03:19.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:03:19.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:19.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:03:19.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:03:19.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:19.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:03:19.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:03:19.723 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.724 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.724 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.724 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:19.724 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:19.724 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.409608448999997, 2023-07-02T14:03:19.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:03:19.725 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:19.725 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:19.725 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:19.725 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:19.725 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:19.725 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:03:19.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.726 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:19.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:19.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:19.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:19.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:19.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:19.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:19.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:19.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:03:19.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:03:19.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:19.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:03:19.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:03:19.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:19.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:03:19.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:03:19.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:19.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:03:19.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:03:19.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:19.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:03:19.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:03:19.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:19.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:03:19.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:03:19.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:19.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:03:19.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:03:19.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:19.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:03:19.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:03:19.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:19.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:03:19.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:03:19.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:19.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:03:19.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:03:19.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:19.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:03:19.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:03:19.734 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.734 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.734 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.734 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:19.735 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:19.735 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.409243324999998, 2023-07-02T14:03:19.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:03:19.735 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:19.735 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:19.735 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:19.735 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:19.736 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:19.736 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:03:19.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.736 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:19.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:19.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:19.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:19.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:19.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:19.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:19.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:19.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:03:19.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:03:19.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:19.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:03:19.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:03:19.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:19.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:03:19.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:03:19.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:19.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:03:19.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:03:19.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:19.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:03:19.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:03:19.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:19.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:03:19.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:03:19.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:19.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:03:19.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:03:19.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:19.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:03:19.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:03:19.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:19.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:03:19.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:03:19.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:19.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:03:19.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:03:19.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:19.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:03:19.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:03:19.744 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.744 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.745 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.745 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:19.745 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:19.745 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.409162264000003, 2023-07-02T14:03:19.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:03:19.745 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:19.746 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:19.746 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:19.746 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:19.746 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:19.746 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:03:19.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.746 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:19.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:19.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:19.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:19.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:19.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:19.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:19.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:19.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:03:19.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:03:19.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:19.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:03:19.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:03:19.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:19.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:03:19.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:03:19.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:19.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:03:19.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:03:19.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:19.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:03:19.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:03:19.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:19.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:03:19.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:03:19.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:19.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:03:19.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:03:19.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:19.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:03:19.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:03:19.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:19.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:03:19.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:03:19.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:19.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:03:19.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:03:19.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:19.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:03:19.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:03:19.755 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.755 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.755 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.755 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:19.755 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:19.756 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.409070593000003, 2023-07-02T14:03:19.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:03:19.756 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:19.756 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:19.756 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:19.756 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:19.756 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:19.757 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:03:19.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.757 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:19.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:19.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:19.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:19.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:19.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:19.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:19.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:19.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:03:19.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:03:19.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:19.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:03:19.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:03:19.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:19.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:03:19.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:03:19.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:19.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:03:19.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:03:19.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:19.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:03:19.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:03:19.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:19.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:03:19.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:03:19.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:19.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:03:19.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:03:19.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:19.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:03:19.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:03:19.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:19.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:03:19.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:03:19.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:19.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:03:19.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:03:19.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:19.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:03:19.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:03:19.765 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.765 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.765 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.766 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:19.766 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:19.766 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.408731150999998, 2023-07-02T14:03:19.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:03:19.766 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:19.766 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:19.766 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:19.767 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:19.767 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:19.767 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:03:19.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.767 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:19.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:19.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:19.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:19.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:19.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:19.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:19.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:19.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:03:19.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:03:19.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:19.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:03:19.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:03:19.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:19.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:03:19.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:03:19.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:19.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:03:19.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:03:19.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:19.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:03:19.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:03:19.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:19.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:03:19.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:03:19.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:19.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:03:19.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:03:19.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:19.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:03:19.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:03:19.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:19.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:03:19.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:03:19.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:19.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:03:19.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:03:19.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:19.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:03:19.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:03:19.775 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.775 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.776 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.776 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:19.776 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:19.776 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.408545097999998, 2023-07-02T14:03:19.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:03:19.776 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:19.777 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:19.777 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:19.777 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:19.777 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:19.777 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:03:19.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.777 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:19.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:19.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:19.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:19.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:19.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:19.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:19.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:19.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:19.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:03:19.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:19.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:03:19.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:03:19.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:19.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:03:19.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:03:19.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:19.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:03:19.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:03:19.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:19.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:03:19.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:03:19.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:19.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:03:19.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:03:19.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:19.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:03:19.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:03:19.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:19.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:03:19.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:03:19.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:19.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:03:19.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:03:19.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:19.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:03:19.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:03:19.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:19.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:03:19.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:03:19.786 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.786 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.786 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.786 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:19.786 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:19.786 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.408387296000001, 2023-07-02T14:03:19.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:03:19.787 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:19.787 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:19.787 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:19.787 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:19.787 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:19.787 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:03:19.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.788 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:19.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:19.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:19.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:19.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:19.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:19.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:19.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:19.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:03:19.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:03:19.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:19.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:03:19.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:03:19.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:19.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:03:19.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:03:19.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:19.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:03:19.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:03:19.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:19.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:03:19.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:03:19.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:19.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:03:19.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:03:19.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:19.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:03:19.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:03:19.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:19.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:03:19.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:03:19.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:19.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:03:19.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:03:19.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:19.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:03:19.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:03:19.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:19.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:03:19.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:03:19.796 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.796 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.796 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.797 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:19.797 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:19.797 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.408195513999999, 2023-07-02T14:03:19.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:03:19.797 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:19.797 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:19.797 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:19.797 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:19.798 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:19.798 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:03:19.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.798 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:19.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:19.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:19.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:19.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:19.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:19.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:19.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:19.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:03:19.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:03:19.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:19.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:03:19.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:03:19.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:19.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:03:19.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:03:19.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:19.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:03:19.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:03:19.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:19.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:03:19.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:03:19.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:19.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:03:19.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:03:19.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:19.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:03:19.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:03:19.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:19.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:03:19.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:03:19.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:19.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:03:19.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:03:19.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:19.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:03:19.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:03:19.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:19.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:03:19.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:03:19.806 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.806 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.806 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.807 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:19.807 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:19.807 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.408019854000003, 2023-07-02T14:03:19.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:03:19.807 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:19.807 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:19.808 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:19.808 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:19.808 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:19.808 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:03:19.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.808 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:19.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:19.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:19.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:19.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:19.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:19.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:19.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:19.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:19.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:03:19.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:19.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:03:19.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:03:19.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:19.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:03:19.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:03:19.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:19.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:03:19.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:03:19.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:19.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:03:19.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:03:19.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:19.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:03:19.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:03:19.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:19.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:03:19.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:03:19.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:19.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:03:19.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:03:19.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:19.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:03:19.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:03:19.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:19.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:03:19.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:03:19.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:19.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:03:19.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:03:19.817 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.817 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.817 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.817 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:19.817 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:19.818 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.407925665999997, 2023-07-02T14:03:19.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:03:19.818 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:19.818 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:19.818 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:19.818 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:19.818 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:19.818 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:03:19.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.819 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:19.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:19.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:19.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:19.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:19.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:19.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:19.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:19.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:03:19.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:03:19.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:19.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:03:19.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:03:19.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:19.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:03:19.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:03:19.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:19.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:03:19.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:03:19.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:19.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:03:19.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:03:19.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:19.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:03:19.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:03:19.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:19.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:03:19.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:03:19.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:19.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:03:19.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:03:19.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:19.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:03:19.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:03:19.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:19.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:03:19.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:03:19.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:19.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:03:19.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:03:19.827 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.827 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.827 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.828 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:19.828 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:19.828 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.407722280000002, 2023-07-02T14:03:19.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:03:19.828 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:19.828 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:19.828 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:19.829 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:19.829 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:19.829 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:03:19.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.829 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:19.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:19.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:19.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:19.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:19.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:19.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:19.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:19.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:03:19.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:03:19.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:19.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:03:19.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:03:19.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:19.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:03:19.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:03:19.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:19.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:03:19.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:03:19.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:19.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:03:19.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:03:19.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:19.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:03:19.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:03:19.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:19.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:03:19.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:03:19.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:19.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:03:19.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:03:19.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:19.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:03:19.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:03:19.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:19.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:03:19.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:03:19.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:19.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:03:19.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:03:19.838 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.838 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.838 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.838 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:19.838 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:19.838 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.407571623999999, 2023-07-02T14:03:19.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:03:19.839 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:19.839 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:19.839 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:19.839 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:19.839 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:19.839 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:03:19.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.840 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:19.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:19.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:19.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:19.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:19.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:19.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:19.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:19.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:03:19.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:03:19.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:19.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:03:19.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:03:19.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:19.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:03:19.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:03:19.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:19.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:03:19.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:03:19.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:19.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:03:19.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:03:19.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:19.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:03:19.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:03:19.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:19.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:03:19.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:03:19.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:19.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:03:19.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:03:19.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:19.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:03:19.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:03:19.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:19.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:03:19.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:03:19.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:19.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:03:19.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:03:19.848 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.848 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.848 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.849 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:19.849 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:19.849 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.407498920999998, 2023-07-02T14:03:19.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:03:19.849 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:19.849 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:19.849 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:19.849 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:19.850 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:19.850 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:03:19.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.850 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:19.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:19.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:19.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:19.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:19.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:19.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:19.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:19.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:19.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:03:19.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:19.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:03:19.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:03:19.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:19.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:03:19.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:03:19.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:19.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:03:19.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:03:19.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:19.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:03:19.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:03:19.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:19.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:03:19.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:03:19.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:19.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:03:19.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:03:19.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:19.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:03:19.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:03:19.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:19.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:03:19.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:03:19.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:19.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:03:19.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:03:19.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:19.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:03:19.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:03:19.858 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.858 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.858 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.859 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:19.859 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:19.859 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.407381931000003, 2023-07-02T14:03:19.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:03:19.859 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:19.859 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:19.860 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:19.860 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:19.860 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:19.860 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:03:19.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.860 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:19.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:19.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:19.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:19.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:19.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:19.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:19.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:19.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:19.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:03:19.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:19.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:03:19.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:03:19.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:19.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:03:19.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:03:19.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:19.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:03:19.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:03:19.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:19.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:03:19.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:03:19.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:19.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:03:19.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:03:19.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:19.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:03:19.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:03:19.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:19.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:03:19.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:03:19.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:19.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:03:19.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:03:19.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:19.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:03:19.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:03:19.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:19.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:03:19.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:03:19.868 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.869 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.869 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.869 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:19.869 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:19.869 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.407232479999998, 2023-07-02T14:03:19.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:03:19.870 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:19.870 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:19.870 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:19.870 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:19.870 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:19.870 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:03:19.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.870 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:19.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:19.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:19.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:19.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:19.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:19.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:19.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:19.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:03:19.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:03:19.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:19.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:03:19.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:03:19.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:19.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:03:19.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:03:19.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:19.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:03:19.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:03:19.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:19.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:03:19.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:03:19.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:19.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:03:19.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:03:19.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:19.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:03:19.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:03:19.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:19.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:03:19.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:03:19.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:19.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:03:19.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:03:19.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:19.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:03:19.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:03:19.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:19.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:03:19.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:03:19.879 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.879 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.879 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.880 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:19.880 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:19.880 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.385756507000004, 2023-07-02T14:03:19.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:03:19.880 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:19.880 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:19.880 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:19.881 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:19.881 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:19.881 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:03:19.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.881 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:19.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:19.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:19.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:19.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:19.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:19.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:19.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:19.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:03:19.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:03:19.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:19.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:03:19.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:03:19.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:19.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:03:19.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:03:19.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:19.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:03:19.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:03:19.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:19.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:03:19.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:03:19.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:19.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:03:19.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:03:19.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:19.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:03:19.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:03:19.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:19.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:03:19.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:03:19.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:19.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:03:19.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:03:19.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:19.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:03:19.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:03:19.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:19.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:03:19.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:03:19.889 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.889 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.890 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.890 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:19.890 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:19.890 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.385623971000001, 2023-07-02T14:03:19.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:03:19.890 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:19.891 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:19.891 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:19.891 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:19.892 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:19.892 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:03:19.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.892 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:19.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.893 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:19.893 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:19.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:19.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:19.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:19.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:19.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:19.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:03:19.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:03:19.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:19.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:03:19.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:03:19.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:19.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:03:19.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:03:19.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:19.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:03:19.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:03:19.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:19.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:03:19.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:03:19.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:19.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:03:19.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:03:19.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:19.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:03:19.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:03:19.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:19.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:03:19.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:03:19.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:19.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:03:19.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:03:19.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:19.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:03:19.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:03:19.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:19.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:03:19.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:03:19.901 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.901 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.901 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.902 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:19.902 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:19.902 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.324287310000003, 2023-07-02T14:03:19.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:03:19.902 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:19.902 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:19.903 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:19.903 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:19.903 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:19.903 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:03:19.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.903 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:19.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:19.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:19.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:19.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:19.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:19.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:19.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:19.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:03:19.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:03:19.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:19.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:03:19.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:03:19.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:19.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:03:19.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:03:19.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:19.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:03:19.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:03:19.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:19.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:03:19.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:03:19.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:19.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:03:19.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:03:19.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:19.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:03:19.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:03:19.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:19.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:03:19.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:03:19.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:19.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:03:19.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:03:19.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:19.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:03:19.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:03:19.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:19.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:19.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:19.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:03:19.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:03:19.912 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.912 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.912 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.912 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:19.912 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:19.912 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:19.912 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2023-07-02T14:03:19.913 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:03:19.913 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2023-07-02T14:03:19.913 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-07-02T14:03:19.915 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:19.915+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:19.915 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:03:20.015 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:20.015+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:20.042 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:20.043 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:20.043 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:20.043 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:20.053 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:03:20.116 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:20.116+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:20.178 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:20.178 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:20.178 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:20.178 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:20.178 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:20.188 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:03:20.223 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:03:20.224 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:20.224 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 66, 2023-07-02T14:03:20.224 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2", 2023-07-02T14:03:20.224 INFO:teuthology.orchestra.run.smithi016.stdout: "created": "2023-07-02T14:01:37.298060+0000", 2023-07-02T14:03:20.224 INFO:teuthology.orchestra.run.smithi016.stdout: "modified": "2023-07-02T14:03:19.338597+0000", 2023-07-02T14:03:20.224 INFO:teuthology.orchestra.run.smithi016.stdout: "last_up_change": "2023-07-02T14:01:45.271760+0000", 2023-07-02T14:03:20.225 INFO:teuthology.orchestra.run.smithi016.stdout: "last_in_change": "2023-07-02T14:03:09.309538+0000", 2023-07-02T14:03:20.225 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-07-02T14:03:20.225 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_num": 5804032, 2023-07-02T14:03:20.225 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_set": [ 2023-07-02T14:03:20.225 INFO:teuthology.orchestra.run.smithi016.stdout: "nodeep-scrub", 2023-07-02T14:03:20.225 INFO:teuthology.orchestra.run.smithi016.stdout: "pglog_hardlimit", 2023-07-02T14:03:20.225 INFO:teuthology.orchestra.run.smithi016.stdout: "purged_snapdirs", 2023-07-02T14:03:20.226 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_deletes", 2023-07-02T14:03:20.226 INFO:teuthology.orchestra.run.smithi016.stdout: "sortbitwise" 2023-07-02T14:03:20.226 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:03:20.226 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_version": 4, 2023-07-02T14:03:20.226 INFO:teuthology.orchestra.run.smithi016.stdout: "full_ratio": 0.94999998807907104, 2023-07-02T14:03:20.226 INFO:teuthology.orchestra.run.smithi016.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-07-02T14:03:20.226 INFO:teuthology.orchestra.run.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2023-07-02T14:03:20.227 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_snapshot": "", 2023-07-02T14:03:20.227 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_max": 2, 2023-07-02T14:03:20.227 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd": 6, 2023-07-02T14:03:20.227 INFO:teuthology.orchestra.run.smithi016.stdout: "require_min_compat_client": "luminous", 2023-07-02T14:03:20.227 INFO:teuthology.orchestra.run.smithi016.stdout: "min_compat_client": "luminous", 2023-07-02T14:03:20.227 INFO:teuthology.orchestra.run.smithi016.stdout: "require_osd_release": "pacific", 2023-07-02T14:03:20.227 INFO:teuthology.orchestra.run.smithi016.stdout: "pools": [ 2023-07-02T14:03:20.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.228 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 1, 2023-07-02T14:03:20.228 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "device_health_metrics", 2023-07-02T14:03:20.228 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:45.323455+0000", 2023-07-02T14:03:20.228 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2023-07-02T14:03:20.228 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2023-07-02T14:03:20.228 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:03:20.228 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:03:20.229 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:03:20.229 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:03:20.229 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:03:20.229 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:03:20.229 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:03:20.229 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:03:20.229 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:03:20.229 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:03:20.230 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 1, 2023-07-02T14:03:20.230 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 1, 2023-07-02T14:03:20.230 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 1, 2023-07-02T14:03:20.230 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 1, 2023-07-02T14:03:20.230 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 1, 2023-07-02T14:03:20.230 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:03:20.230 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:03:20.230 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:03:20.231 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:03:20.231 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:03:20.231 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:03:20.231 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:03:20.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.231 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "14", 2023-07-02T14:03:20.231 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:03:20.232 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:03:20.232 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:03:20.232 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:03:20.232 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:03:20.232 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2023-07-02T14:03:20.232 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2023-07-02T14:03:20.232 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:03:20.232 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:03:20.233 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:03:20.233 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:03:20.233 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:03:20.233 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:03:20.233 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:03:20.234 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:03:20.234 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:03:20.234 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:03:20.234 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:03:20.234 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:03:20.234 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:03:20.234 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:03:20.234 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:03:20.235 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:03:20.235 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:03:20.235 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:03:20.235 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:03:20.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.235 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:03:20.235 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:03:20.235 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:03:20.236 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:03:20.236 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:03:20.236 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:03:20.236 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:03:20.236 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:03:20.236 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:03:20.236 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:03:20.237 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:03:20.237 INFO:teuthology.orchestra.run.smithi016.stdout: "options": { 2023-07-02T14:03:20.237 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_max": 32, 2023-07-02T14:03:20.237 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_min": 1 2023-07-02T14:03:20.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.237 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:03:20.237 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_devicehealth": {} 2023-07-02T14:03:20.238 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.238 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 2, 2023-07-02T14:03:20.238 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "rbd", 2023-07-02T14:03:20.238 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:47.448897+0000", 2023-07-02T14:03:20.238 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 8193, 2023-07-02T14:03:20.238 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2023-07-02T14:03:20.239 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:03:20.239 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:03:20.239 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:03:20.239 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:03:20.239 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:03:20.239 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:03:20.239 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:03:20.239 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:03:20.240 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:03:20.240 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:03:20.240 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2023-07-02T14:03:20.240 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2023-07-02T14:03:20.240 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2023-07-02T14:03:20.240 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2023-07-02T14:03:20.240 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2023-07-02T14:03:20.241 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:03:20.241 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:03:20.241 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:03:20.241 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:03:20.241 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:03:20.241 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:03:20.241 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:03:20.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.242 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "66", 2023-07-02T14:03:20.242 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:03:20.242 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:03:20.242 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:03:20.242 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:03:20.242 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:03:20.242 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 9, 2023-07-02T14:03:20.242 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 66, 2023-07-02T14:03:20.243 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:03:20.243 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:03:20.243 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:03:20.243 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:03:20.243 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:03:20.243 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:03:20.243 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:03:20.243 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:03:20.244 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:03:20.244 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:03:20.244 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:03:20.244 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:03:20.244 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:03:20.244 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:03:20.244 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:03:20.245 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:03:20.245 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:03:20.245 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:03:20.245 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:03:20.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.245 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:03:20.245 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:03:20.245 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:03:20.246 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:03:20.246 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:03:20.246 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:03:20.246 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:03:20.246 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:03:20.246 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:03:20.246 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:03:20.247 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:03:20.247 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2023-07-02T14:03:20.247 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:03:20.247 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2023-07-02T14:03:20.247 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.247 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.247 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:03:20.247 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:03:20.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.248 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:03:20.248 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "43c446fe-c811-4ee3-99e8-d7452edfc1c1", 2023-07-02T14:03:20.248 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:03:20.248 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:03:20.248 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:03:20.248 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:03:20.249 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:03:20.249 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:03:20.249 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:03:20.249 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 62, 2023-07-02T14:03:20.249 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:03:20.249 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:03:20.249 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:03:20.249 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.250 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.250 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2023-07-02T14:03:20.250 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:20.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.250 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.250 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2023-07-02T14:03:20.251 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:20.251 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.251 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.251 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:03:20.251 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.251 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.252 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6802", 2023-07-02T14:03:20.252 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:20.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.252 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.252 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6803", 2023-07-02T14:03:20.252 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:20.253 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.253 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.253 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:03:20.253 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.253 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.253 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6806", 2023-07-02T14:03:20.254 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:20.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.254 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.254 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6807", 2023-07-02T14:03:20.254 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:20.254 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.254 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.255 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:03:20.255 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.255 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.255 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6804", 2023-07-02T14:03:20.255 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:20.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.256 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.256 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6805", 2023-07-02T14:03:20.256 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:20.256 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.256 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.257 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6801/83359", 2023-07-02T14:03:20.257 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6803/83359", 2023-07-02T14:03:20.257 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6807/83359", 2023-07-02T14:03:20.257 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6805/83359", 2023-07-02T14:03:20.257 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:03:20.257 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:03:20.257 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:03:20.258 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.258 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:03:20.258 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b", 2023-07-02T14:03:20.258 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:03:20.258 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:03:20.258 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:03:20.259 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:03:20.259 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:03:20.259 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:03:20.259 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:03:20.259 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 63, 2023-07-02T14:03:20.259 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:03:20.259 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:03:20.259 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:03:20.260 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.260 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.260 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6808", 2023-07-02T14:03:20.260 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:20.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.261 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.261 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6809", 2023-07-02T14:03:20.261 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:20.261 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.261 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.261 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:03:20.261 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.262 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.262 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6810", 2023-07-02T14:03:20.262 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:20.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.262 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.263 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6811", 2023-07-02T14:03:20.263 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:20.263 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.263 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.263 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:03:20.263 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.264 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.264 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6814", 2023-07-02T14:03:20.264 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:20.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.264 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.264 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6815", 2023-07-02T14:03:20.264 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:20.265 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.265 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.265 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:03:20.265 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.265 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.265 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6812", 2023-07-02T14:03:20.266 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:20.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.266 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.266 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6813", 2023-07-02T14:03:20.266 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:20.266 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.267 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.267 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6809/83360", 2023-07-02T14:03:20.267 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6811/83360", 2023-07-02T14:03:20.267 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6815/83360", 2023-07-02T14:03:20.267 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6813/83360", 2023-07-02T14:03:20.267 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:03:20.267 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:03:20.268 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:03:20.268 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.268 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:03:20.268 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "99f8ea0a-5a71-4abe-b087-8e891fd11d1b", 2023-07-02T14:03:20.268 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:03:20.268 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:03:20.269 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:03:20.269 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:03:20.269 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:03:20.269 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:03:20.269 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:03:20.269 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 62, 2023-07-02T14:03:20.269 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:03:20.270 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:03:20.270 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:03:20.270 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.270 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.270 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6816", 2023-07-02T14:03:20.270 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:20.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.271 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.271 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6817", 2023-07-02T14:03:20.271 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:20.271 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.271 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.271 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:03:20.272 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.272 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.272 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6818", 2023-07-02T14:03:20.272 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:20.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.272 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.273 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6819", 2023-07-02T14:03:20.273 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:20.273 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.273 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.273 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:03:20.273 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.274 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.274 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6822", 2023-07-02T14:03:20.274 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:20.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.274 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.274 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6823", 2023-07-02T14:03:20.275 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:20.275 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.275 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.275 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:03:20.275 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.275 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.276 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6820", 2023-07-02T14:03:20.276 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:20.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.276 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.276 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6821", 2023-07-02T14:03:20.276 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:20.276 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.277 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.277 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6817/83361", 2023-07-02T14:03:20.277 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6819/83361", 2023-07-02T14:03:20.277 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6823/83361", 2023-07-02T14:03:20.277 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6821/83361", 2023-07-02T14:03:20.277 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:03:20.277 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:03:20.278 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:03:20.278 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.278 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:03:20.278 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "d285b4d5-0bef-421a-b84a-943e485b10c2", 2023-07-02T14:03:20.278 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:03:20.279 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:03:20.279 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:03:20.279 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:03:20.279 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:03:20.279 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:03:20.279 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:03:20.279 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 63, 2023-07-02T14:03:20.279 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:03:20.280 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:03:20.280 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:03:20.280 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.280 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.280 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6808", 2023-07-02T14:03:20.280 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:20.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.281 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.281 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6809", 2023-07-02T14:03:20.281 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:20.281 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.281 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.281 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:03:20.282 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.282 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.282 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6810", 2023-07-02T14:03:20.282 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:20.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.282 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.283 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6811", 2023-07-02T14:03:20.283 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:20.283 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.283 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.283 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:03:20.283 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.284 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.284 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6814", 2023-07-02T14:03:20.284 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:20.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.284 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.284 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6815", 2023-07-02T14:03:20.285 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:20.285 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.285 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.285 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:03:20.285 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.285 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.286 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6812", 2023-07-02T14:03:20.286 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:20.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.286 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.286 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6813", 2023-07-02T14:03:20.286 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:20.286 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.287 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.287 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6809/83350", 2023-07-02T14:03:20.287 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6811/83350", 2023-07-02T14:03:20.287 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6815/83350", 2023-07-02T14:03:20.287 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6813/83350", 2023-07-02T14:03:20.287 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:03:20.287 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:03:20.288 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:03:20.288 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.288 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:03:20.288 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "7ba0b0db-e407-4c63-8278-aab09f3082c4", 2023-07-02T14:03:20.288 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:03:20.289 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:03:20.289 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:03:20.289 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 0.23297119140625, 2023-07-02T14:03:20.289 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:03:20.289 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:03:20.289 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:03:20.289 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 0, 2023-07-02T14:03:20.289 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:03:20.290 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:03:20.290 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:03:20.290 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.290 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.290 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6800", 2023-07-02T14:03:20.290 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:20.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.291 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.291 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6801", 2023-07-02T14:03:20.291 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:20.291 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.291 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.292 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:03:20.292 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.292 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.292 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6802", 2023-07-02T14:03:20.292 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:20.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.293 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.293 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6803", 2023-07-02T14:03:20.293 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:20.293 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.293 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.293 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:03:20.294 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.294 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.294 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6806", 2023-07-02T14:03:20.294 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:20.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.294 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.295 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6807", 2023-07-02T14:03:20.295 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:20.295 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.295 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.295 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:03:20.295 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.296 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.296 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6804", 2023-07-02T14:03:20.296 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:20.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.296 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.296 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6805", 2023-07-02T14:03:20.297 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:20.297 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.297 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.297 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6801/83351", 2023-07-02T14:03:20.297 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6803/83351", 2023-07-02T14:03:20.297 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6807/83351", 2023-07-02T14:03:20.297 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6805/83351", 2023-07-02T14:03:20.298 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:03:20.298 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:03:20.298 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:03:20.298 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.298 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:03:20.298 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "29663bfa-f6fe-4ce7-b45a-a9b7b4424c42", 2023-07-02T14:03:20.299 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:03:20.299 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 0, 2023-07-02T14:03:20.299 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 0, 2023-07-02T14:03:20.299 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:03:20.299 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:03:20.299 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:03:20.299 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:03:20.300 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 34, 2023-07-02T14:03:20.300 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:03:20.300 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:03:20.300 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:03:20.300 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.300 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.300 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6816", 2023-07-02T14:03:20.301 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:20.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.301 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6817", 2023-07-02T14:03:20.301 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:20.301 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.301 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.302 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:03:20.302 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.302 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.302 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6818", 2023-07-02T14:03:20.302 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:20.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.303 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.303 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6819", 2023-07-02T14:03:20.303 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:20.303 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.303 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.304 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:03:20.304 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.304 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.304 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6822", 2023-07-02T14:03:20.304 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:20.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.305 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.305 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6823", 2023-07-02T14:03:20.305 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:20.305 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.305 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.305 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:03:20.306 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:20.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.306 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:20.306 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6820", 2023-07-02T14:03:20.306 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:20.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.306 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:20.307 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6821", 2023-07-02T14:03:20.307 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:20.307 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.307 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.307 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6817/83352", 2023-07-02T14:03:20.307 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6819/83352", 2023-07-02T14:03:20.307 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6823/83352", 2023-07-02T14:03:20.308 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6821/83352", 2023-07-02T14:03:20.308 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:03:20.308 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:03:20.308 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:03:20.308 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.308 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.308 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:03:20.309 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_xinfo": [ 2023-07-02T14:03:20.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.309 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:03:20.309 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:03:20.309 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:03:20.309 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:03:20.309 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:03:20.309 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:03:20.310 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.001178+0000", 2023-07-02T14:03:20.310 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:03:20.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.310 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:03:20.310 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:03:20.310 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:03:20.310 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:03:20.311 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:03:20.311 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:03:20.311 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.184181+0000", 2023-07-02T14:03:20.311 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:03:20.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.311 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:03:20.312 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:03:20.312 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:03:20.312 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:03:20.312 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:03:20.312 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:03:20.312 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.274854+0000", 2023-07-02T14:03:20.312 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:03:20.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.313 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:03:20.313 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:03:20.313 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:03:20.313 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:03:20.313 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:03:20.313 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:03:20.313 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.186217+0000", 2023-07-02T14:03:20.314 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:03:20.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.314 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:03:20.314 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:03:20.314 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:03:20.314 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:03:20.314 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:03:20.315 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:03:20.315 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.247442+0000", 2023-07-02T14:03:20.315 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:03:20.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.315 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:03:20.315 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:03:20.315 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:03:20.316 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:03:20.316 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:03:20.316 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 65536, 2023-07-02T14:03:20.316 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.220075+0000", 2023-07-02T14:03:20.316 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:03:20.316 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.316 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:03:20.317 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap": [ 2023-07-02T14:03:20.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.317 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.0", 2023-07-02T14:03:20.317 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:03:20.317 INFO:teuthology.orchestra.run.smithi016.stdout: 4, 2023-07-02T14:03:20.317 INFO:teuthology.orchestra.run.smithi016.stdout: 1 2023-07-02T14:03:20.317 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.318 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.2", 2023-07-02T14:03:20.318 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:03:20.318 INFO:teuthology.orchestra.run.smithi016.stdout: 1, 2023-07-02T14:03:20.318 INFO:teuthology.orchestra.run.smithi016.stdout: 5 2023-07-02T14:03:20.318 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.319 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.4", 2023-07-02T14:03:20.319 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:03:20.319 INFO:teuthology.orchestra.run.smithi016.stdout: 5, 2023-07-02T14:03:20.319 INFO:teuthology.orchestra.run.smithi016.stdout: 0 2023-07-02T14:03:20.319 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.319 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.319 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:03:20.319 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_items": [], 2023-07-02T14:03:20.320 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_temp": [], 2023-07-02T14:03:20.320 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_temp": [], 2023-07-02T14:03:20.320 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklist": {}, 2023-07-02T14:03:20.320 INFO:teuthology.orchestra.run.smithi016.stdout: "range_blocklist": {}, 2023-07-02T14:03:20.320 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profiles": { 2023-07-02T14:03:20.320 INFO:teuthology.orchestra.run.smithi016.stdout: "default": { 2023-07-02T14:03:20.320 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2023-07-02T14:03:20.320 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "2", 2023-07-02T14:03:20.321 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "1", 2023-07-02T14:03:20.321 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "jerasure", 2023-07-02T14:03:20.321 INFO:teuthology.orchestra.run.smithi016.stdout: "ruleset-failure-domain": "osd", 2023-07-02T14:03:20.321 INFO:teuthology.orchestra.run.smithi016.stdout: "technique": "reed_sol_van" 2023-07-02T14:03:20.321 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.321 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps_queue": [], 2023-07-02T14:03:20.321 INFO:teuthology.orchestra.run.smithi016.stdout: "new_removed_snaps": [], 2023-07-02T14:03:20.322 INFO:teuthology.orchestra.run.smithi016.stdout: "new_purged_snaps": [], 2023-07-02T14:03:20.322 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_node_flags": {}, 2023-07-02T14:03:20.322 INFO:teuthology.orchestra.run.smithi016.stdout: "device_class_flags": {}, 2023-07-02T14:03:20.322 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode": { 2023-07-02T14:03:20.322 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_enabled": false, 2023-07-02T14:03:20.322 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_bucket_count": 0, 2023-07-02T14:03:20.322 INFO:teuthology.orchestra.run.smithi016.stdout: "degraded_stretch_mode": 0, 2023-07-02T14:03:20.323 INFO:teuthology.orchestra.run.smithi016.stdout: "recovering_stretch_mode": 0, 2023-07-02T14:03:20.323 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_bucket": 0 2023-07-02T14:03:20.323 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.323 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:20.323 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:20.222+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:20.325 INFO:tasks.thrashosds.thrasher:j is {'epoch': 66, 'fsid': '9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2', 'created': '2023-07-02T14:01:37.298060+0000', 'modified': '2023-07-02T14:03:19.338597+0000', 'last_up_change': '2023-07-02T14:01:45.271760+0000', 'last_in_change': '2023-07-02T14:03:09.309538+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-07-02T14:01:45.323455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-02T14:01:47.448897+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 9, 'snap_epoch': 66, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '43c446fe-c811-4ee3-99e8-d7452edfc1c1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6800', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6801', 'nonce': 83359}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6802', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6803', 'nonce': 83359}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6806', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6807', 'nonce': 83359}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6804', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6805', 'nonce': 83359}]}, 'public_addr': '172.21.15.16:6801/83359', 'cluster_addr': '172.21.15.16:6803/83359', 'heartbeat_back_addr': '172.21.15.16:6807/83359', 'heartbeat_front_addr': '172.21.15.16:6805/83359', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 63, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6808', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6809', 'nonce': 83360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6810', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6811', 'nonce': 83360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6814', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6815', 'nonce': 83360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6812', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6813', 'nonce': 83360}]}, 'public_addr': '172.21.15.16:6809/83360', 'cluster_addr': '172.21.15.16:6811/83360', 'heartbeat_back_addr': '172.21.15.16:6815/83360', 'heartbeat_front_addr': '172.21.15.16:6813/83360', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '99f8ea0a-5a71-4abe-b087-8e891fd11d1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6816', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6817', 'nonce': 83361}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6818', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6819', 'nonce': 83361}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6822', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6823', 'nonce': 83361}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6820', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6821', 'nonce': 83361}]}, 'public_addr': '172.21.15.16:6817/83361', 'cluster_addr': '172.21.15.16:6819/83361', 'heartbeat_back_addr': '172.21.15.16:6823/83361', 'heartbeat_front_addr': '172.21.15.16:6821/83361', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd285b4d5-0bef-421a-b84a-943e485b10c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 63, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6808', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6809', 'nonce': 83350}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6810', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6811', 'nonce': 83350}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6814', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6815', 'nonce': 83350}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6812', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6813', 'nonce': 83350}]}, 'public_addr': '172.21.15.22:6809/83350', 'cluster_addr': '172.21.15.22:6811/83350', 'heartbeat_back_addr': '172.21.15.22:6815/83350', 'heartbeat_front_addr': '172.21.15.22:6813/83350', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7ba0b0db-e407-4c63-8278-aab09f3082c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.23297119140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6800', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6801', 'nonce': 83351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6802', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6803', 'nonce': 83351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6806', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6807', 'nonce': 83351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6804', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6805', 'nonce': 83351}]}, 'public_addr': '172.21.15.22:6801/83351', 'cluster_addr': '172.21.15.22:6803/83351', 'heartbeat_back_addr': '172.21.15.22:6807/83351', 'heartbeat_front_addr': '172.21.15.22:6805/83351', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '29663bfa-f6fe-4ce7-b45a-a9b7b4424c42', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6816', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6817', 'nonce': 83352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6818', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6819', 'nonce': 83352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6822', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6823', 'nonce': 83352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6820', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6821', 'nonce': 83352}]}, 'public_addr': '172.21.15.22:6817/83352', 'cluster_addr': '172.21.15.22:6819/83352', 'heartbeat_back_addr': '172.21.15.22:6823/83352', 'heartbeat_front_addr': '172.21.15.22:6821/83352', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.001178+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.184181+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.274854+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.186217+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.247442+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.220075+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.0', 'osds': [4, 1]}, {'pgid': '2.2', 'osds': [1, 5]}, {'pgid': '2.4', 'osds': [5, 0]}], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-07-02T14:03:20.325 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2023-07-02T14:03:20.326 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:20.327 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:03:20.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:03:20.327 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:20.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.327 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2620 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 196608~786432] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:20.327 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:03:20.327 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.9896699350000002, 2023-07-02T14:03:20.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966, 2023-07-02T14:03:20.328 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:20.328 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:20.328 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:20.328 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:20.328 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:20.328 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2620 2023-07-02T14:03:20.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.329 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:20.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:20.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:03:20.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:20.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:20.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:03:20.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:20.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:20.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324571+0000", 2023-07-02T14:03:20.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016686649999999999 2023-07-02T14:03:20.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:20.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326240+0000", 2023-07-02T14:03:20.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1844e-05 2023-07-02T14:03:20.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:20.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326252+0000", 2023-07-02T14:03:20.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7760000000000003e-06 2023-07-02T14:03:20.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:20.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326258+0000", 2023-07-02T14:03:20.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0634000000000003e-05 2023-07-02T14:03:20.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326318+0000", 2023-07-02T14:03:20.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9629999999999993e-06 2023-07-02T14:03:20.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:20.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326328+0000", 2023-07-02T14:03:20.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.010877045 2023-07-02T14:03:20.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337205+0000", 2023-07-02T14:03:20.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022401300000000001 2023-07-02T14:03:20.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:20.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337429+0000", 2023-07-02T14:03:20.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000139822 2023-07-02T14:03:20.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:20.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337569+0000", 2023-07-02T14:03:20.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032942119999999999 2023-07-02T14:03:20.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:20.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340863+0000", 2023-07-02T14:03:20.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021302560000000001 2023-07-02T14:03:20.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:20.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342994+0000", 2023-07-02T14:03:20.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5634000000000002e-05 2023-07-02T14:03:20.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:20.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343059+0000", 2023-07-02T14:03:20.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143721 2023-07-02T14:03:20.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:20.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343203+0000", 2023-07-02T14:03:20.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966 2023-07-02T14:03:20.338 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.338 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.338 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.339 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2622 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 983040~36864] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:20.339 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:03:20.339 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.9879774240000003, 2023-07-02T14:03:20.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664, 2023-07-02T14:03:20.339 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:20.340 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:20.340 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:20.340 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:20.340 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:20.340 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2622 2023-07-02T14:03:20.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.340 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:20.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:20.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:03:20.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:20.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:20.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:03:20.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:20.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:20.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326265+0000", 2023-07-02T14:03:20.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1282000000000001e-05 2023-07-02T14:03:20.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:20.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326286+0000", 2023-07-02T14:03:20.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9480000000000002e-06 2023-07-02T14:03:20.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:20.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326293+0000", 2023-07-02T14:03:20.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3610000000000003e-06 2023-07-02T14:03:20.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:20.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326297+0000", 2023-07-02T14:03:20.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7707999999999994e-05 2023-07-02T14:03:20.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326385+0000", 2023-07-02T14:03:20.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8070000000000006e-06 2023-07-02T14:03:20.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:20.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326394+0000", 2023-07-02T14:03:20.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.014535878 2023-07-02T14:03:20.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340930+0000", 2023-07-02T14:03:20.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000154551 2023-07-02T14:03:20.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:20.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341084+0000", 2023-07-02T14:03:20.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00032375900000000002 2023-07-02T14:03:20.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:20.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341408+0000", 2023-07-02T14:03:20.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012631066 2023-07-02T14:03:20.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:20.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354039+0000", 2023-07-02T14:03:20.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5741000000000001e-05 2023-07-02T14:03:20.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:20.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354055+0000", 2023-07-02T14:03:20.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8794000000000006e-05 2023-07-02T14:03:20.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:20.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354124+0000", 2023-07-02T14:03:20.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014667700000000001 2023-07-02T14:03:20.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:20.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354270+0000", 2023-07-02T14:03:20.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664 2023-07-02T14:03:20.350 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.350 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.350 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.350 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2727 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~12288] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:20.351 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:03:20.351 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.9879416489999997, 2023-07-02T14:03:20.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999, 2023-07-02T14:03:20.351 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:20.351 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:20.351 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:20.351 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:20.352 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:20.352 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2727 2023-07-02T14:03:20.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.352 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:20.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:20.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:03:20.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:20.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:20.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:03:20.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:20.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:20.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:03:20.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3834e-05 2023-07-02T14:03:20.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:20.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326315+0000", 2023-07-02T14:03:20.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3029999999999996e-06 2023-07-02T14:03:20.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:20.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326321+0000", 2023-07-02T14:03:20.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7590000000000002e-06 2023-07-02T14:03:20.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:20.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326324+0000", 2023-07-02T14:03:20.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112872 2023-07-02T14:03:20.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326437+0000", 2023-07-02T14:03:20.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2290000000000002e-06 2023-07-02T14:03:20.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:20.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326442+0000", 2023-07-02T14:03:20.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0155896879999999 2023-07-02T14:03:20.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342031+0000", 2023-07-02T14:03:20.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016496100000000001 2023-07-02T14:03:20.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:20.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342196+0000", 2023-07-02T14:03:20.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021998699999999999 2023-07-02T14:03:20.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:20.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342416+0000", 2023-07-02T14:03:20.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011869744999999999 2023-07-02T14:03:20.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:20.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354286+0000", 2023-07-02T14:03:20.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0361e-05 2023-07-02T14:03:20.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:20.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354347+0000", 2023-07-02T14:03:20.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3312000000000005e-05 2023-07-02T14:03:20.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:20.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354430+0000", 2023-07-02T14:03:20.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126444 2023-07-02T14:03:20.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:20.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354556+0000", 2023-07-02T14:03:20.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999 2023-07-02T14:03:20.361 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.362 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.362 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.362 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2730 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [set-alloc-hint object_size 4194304 write_size 4194304,write 454656~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:20.362 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:03:20.362 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.9879166399999999, 2023-07-02T14:03:20.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999, 2023-07-02T14:03:20.363 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:20.363 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:20.363 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:20.363 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:20.363 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:20.363 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2730 2023-07-02T14:03:20.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.363 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:20.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:20.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:03:20.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:20.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:20.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:03:20.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:20.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:20.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:03:20.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6049999999999997e-06 2023-07-02T14:03:20.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:20.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326332+0000", 2023-07-02T14:03:20.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3000000000000002e-06 2023-07-02T14:03:20.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:20.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326335+0000", 2023-07-02T14:03:20.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.719e-05 2023-07-02T14:03:20.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:20.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326352+0000", 2023-07-02T14:03:20.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011583599999999999 2023-07-02T14:03:20.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326468+0000", 2023-07-02T14:03:20.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.831e-06 2023-07-02T14:03:20.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:20.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326473+0000", 2023-07-02T14:03:20.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.016769915 2023-07-02T14:03:20.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343243+0000", 2023-07-02T14:03:20.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197352 2023-07-02T14:03:20.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:20.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343440+0000", 2023-07-02T14:03:20.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197581 2023-07-02T14:03:20.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:20.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343638+0000", 2023-07-02T14:03:20.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010980139 2023-07-02T14:03:20.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:20.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354618+0000", 2023-07-02T14:03:20.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056779530000000003 2023-07-02T14:03:20.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:20.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360296+0000", 2023-07-02T14:03:20.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101109 2023-07-02T14:03:20.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:20.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360397+0000", 2023-07-02T14:03:20.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000160273 2023-07-02T14:03:20.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:20.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360557+0000", 2023-07-02T14:03:20.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999 2023-07-02T14:03:20.374 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.374 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.374 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.374 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2752 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 1019904~3174400] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:20.374 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:03:20.374 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.9878704159999998, 2023-07-02T14:03:20.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001, 2023-07-02T14:03:20.375 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:20.375 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:20.375 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:20.375 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:20.375 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:20.375 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2752 2023-07-02T14:03:20.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.376 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:20.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:20.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:03:20.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:20.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:20.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:03:20.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:20.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:20.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326372+0000", 2023-07-02T14:03:20.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052050380000000004 2023-07-02T14:03:20.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:20.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331577+0000", 2023-07-02T14:03:20.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6832000000000001e-05 2023-07-02T14:03:20.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:20.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331594+0000", 2023-07-02T14:03:20.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8220000000000007e-06 2023-07-02T14:03:20.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:20.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331602+0000", 2023-07-02T14:03:20.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9895999999999999e-05 2023-07-02T14:03:20.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331642+0000", 2023-07-02T14:03:20.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:03:20.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:20.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331649+0000", 2023-07-02T14:03:20.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0139141709999999 2023-07-02T14:03:20.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345563+0000", 2023-07-02T14:03:20.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113817 2023-07-02T14:03:20.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:20.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345677+0000", 2023-07-02T14:03:20.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018803400000000001 2023-07-02T14:03:20.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:20.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345865+0000", 2023-07-02T14:03:20.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087950359999999991 2023-07-02T14:03:20.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:20.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354660+0000", 2023-07-02T14:03:20.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059116669999999998 2023-07-02T14:03:20.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:20.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360572+0000", 2023-07-02T14:03:20.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0992000000000002e-05 2023-07-02T14:03:20.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:20.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360633+0000", 2023-07-02T14:03:20.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013601600000000001 2023-07-02T14:03:20.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:20.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360769+0000", 2023-07-02T14:03:20.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001 2023-07-02T14:03:20.385 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.385 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.386 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.386 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2759 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 0~454656] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:20.386 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:03:20.386 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.982630619, 2023-07-02T14:03:20.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999, 2023-07-02T14:03:20.386 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:20.387 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:20.387 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:20.387 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:20.387 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:20.387 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2759 2023-07-02T14:03:20.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.387 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:20.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:20.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:03:20.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:20.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:20.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:03:20.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:20.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:20.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331611+0000", 2023-07-02T14:03:20.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053795999999999998 2023-07-02T14:03:20.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:20.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332149+0000", 2023-07-02T14:03:20.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4970000000000007e-06 2023-07-02T14:03:20.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:20.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332158+0000", 2023-07-02T14:03:20.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:03:20.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:20.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332162+0000", 2023-07-02T14:03:20.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3526000000000001e-05 2023-07-02T14:03:20.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332185+0000", 2023-07-02T14:03:20.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:03:20.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:20.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332189+0000", 2023-07-02T14:03:20.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0186295910000001 2023-07-02T14:03:20.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350819+0000", 2023-07-02T14:03:20.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000111123 2023-07-02T14:03:20.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:20.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350930+0000", 2023-07-02T14:03:20.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019575300000000001 2023-07-02T14:03:20.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:20.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.351126+0000", 2023-07-02T14:03:20.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011503298 2023-07-02T14:03:20.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:20.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362629+0000", 2023-07-02T14:03:20.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085443989999999994 2023-07-02T14:03:20.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:20.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371173+0000", 2023-07-02T14:03:20.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3547999999999995e-05 2023-07-02T14:03:20.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:20.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371237+0000", 2023-07-02T14:03:20.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790300000000001 2023-07-02T14:03:20.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:20.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371375+0000", 2023-07-02T14:03:20.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999 2023-07-02T14:03:20.397 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.397 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.397 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.397 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2760 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 458752~131072] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:20.398 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:03:20.398 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.9820744339999994, 2023-07-02T14:03:20.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001, 2023-07-02T14:03:20.398 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:20.398 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:20.398 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:20.398 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:20.399 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:20.399 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2760 2023-07-02T14:03:20.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.399 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:20.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:20.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:03:20.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:20.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:20.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:03:20.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:20.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:20.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332168+0000", 2023-07-02T14:03:20.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013281100000000001 2023-07-02T14:03:20.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:20.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332301+0000", 2023-07-02T14:03:20.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7389999999999999e-06 2023-07-02T14:03:20.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:20.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332308+0000", 2023-07-02T14:03:20.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0639999999999998e-06 2023-07-02T14:03:20.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:20.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332312+0000", 2023-07-02T14:03:20.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1735000000000001e-05 2023-07-02T14:03:20.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332333+0000", 2023-07-02T14:03:20.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8342000000000002e-05 2023-07-02T14:03:20.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:20.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332352+0000", 2023-07-02T14:03:20.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02234782 2023-07-02T14:03:20.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354699+0000", 2023-07-02T14:03:20.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110371 2023-07-02T14:03:20.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:20.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354810+0000", 2023-07-02T14:03:20.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022174900000000001 2023-07-02T14:03:20.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:20.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355032+0000", 2023-07-02T14:03:20.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076053329999999997 2023-07-02T14:03:20.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:20.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362637+0000", 2023-07-02T14:03:20.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087913609999999993 2023-07-02T14:03:20.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:20.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371428+0000", 2023-07-02T14:03:20.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8532999999999998e-05 2023-07-02T14:03:20.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:20.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371477+0000", 2023-07-02T14:03:20.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010990800000000001 2023-07-02T14:03:20.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:20.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371587+0000", 2023-07-02T14:03:20.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001 2023-07-02T14:03:20.409 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.409 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.409 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.409 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2766 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 589824~45056] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:20.409 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:03:20.409 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.9819258089999998, 2023-07-02T14:03:20.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188, 2023-07-02T14:03:20.410 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:20.410 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:20.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:20.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:20.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:20.410 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2766 2023-07-02T14:03:20.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.411 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:20.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:20.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:03:20.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:20.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:20.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:03:20.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:20.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:20.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:03:20.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2247999999999998e-05 2023-07-02T14:03:20.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:20.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332369+0000", 2023-07-02T14:03:20.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4189999999999996e-06 2023-07-02T14:03:20.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:20.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332376+0000", 2023-07-02T14:03:20.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1049999999999999e-06 2023-07-02T14:03:20.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:20.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332379+0000", 2023-07-02T14:03:20.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6894000000000003e-05 2023-07-02T14:03:20.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332416+0000", 2023-07-02T14:03:20.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3970000000000002e-06 2023-07-02T14:03:20.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:20.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332422+0000", 2023-07-02T14:03:20.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0231382280000001 2023-07-02T14:03:20.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355560+0000", 2023-07-02T14:03:20.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9079000000000005e-05 2023-07-02T14:03:20.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:20.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355659+0000", 2023-07-02T14:03:20.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018005000000000001 2023-07-02T14:03:20.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:20.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355839+0000", 2023-07-02T14:03:20.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068044439999999998 2023-07-02T14:03:20.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:20.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362644+0000", 2023-07-02T14:03:20.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090155589999999994 2023-07-02T14:03:20.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:20.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371659+0000", 2023-07-02T14:03:20.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1297999999999994e-05 2023-07-02T14:03:20.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:20.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371730+0000", 2023-07-02T14:03:20.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013100699999999999 2023-07-02T14:03:20.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:20.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371861+0000", 2023-07-02T14:03:20.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188 2023-07-02T14:03:20.420 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.420 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.420 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.421 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2784 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 0~393216] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:20.421 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:03:20.421 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.9818573570000009, 2023-07-02T14:03:20.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999, 2023-07-02T14:03:20.421 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:20.421 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:20.422 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:20.422 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:20.422 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:20.422 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2784 2023-07-02T14:03:20.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.422 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:20.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:20.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:03:20.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:20.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:20.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:03:20.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:20.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:20.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332385+0000", 2023-07-02T14:03:20.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040252400000000002 2023-07-02T14:03:20.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:20.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332787+0000", 2023-07-02T14:03:20.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0208e-05 2023-07-02T14:03:20.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:20.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332798+0000", 2023-07-02T14:03:20.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2023-07-02T14:03:20.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:20.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332801+0000", 2023-07-02T14:03:20.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4838000000000001e-05 2023-07-02T14:03:20.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332836+0000", 2023-07-02T14:03:20.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5319999999999998e-06 2023-07-02T14:03:20.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:20.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332839+0000", 2023-07-02T14:03:20.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0234290109999999 2023-07-02T14:03:20.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356268+0000", 2023-07-02T14:03:20.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011165700000000001 2023-07-02T14:03:20.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:20.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356380+0000", 2023-07-02T14:03:20.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022504799999999999 2023-07-02T14:03:20.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:20.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356605+0000", 2023-07-02T14:03:20.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015276201 2023-07-02T14:03:20.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:20.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371881+0000", 2023-07-02T14:03:20.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5496999999999999e-05 2023-07-02T14:03:20.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:20.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371957+0000", 2023-07-02T14:03:20.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4570000000000002e-05 2023-07-02T14:03:20.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:20.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372001+0000", 2023-07-02T14:03:20.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3880000000000003e-05 2023-07-02T14:03:20.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:20.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372085+0000", 2023-07-02T14:03:20.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999 2023-07-02T14:03:20.432 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.432 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.432 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.432 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2785 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 405504~446464] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:20.433 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:03:20.433 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.9814359059999997, 2023-07-02T14:03:20.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999, 2023-07-02T14:03:20.433 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:20.433 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:20.433 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:20.433 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:20.433 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:20.434 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2785 2023-07-02T14:03:20.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.434 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:20.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:20.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:03:20.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:20.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:20.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:03:20.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:20.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:20.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:03:20.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059527500000000004 2023-07-02T14:03:20.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:20.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333402+0000", 2023-07-02T14:03:20.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7059999999999993e-06 2023-07-02T14:03:20.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:20.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333411+0000", 2023-07-02T14:03:20.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0450000000000001e-06 2023-07-02T14:03:20.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:20.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333414+0000", 2023-07-02T14:03:20.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5790999999999997e-05 2023-07-02T14:03:20.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:03:20.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3320000000000002e-06 2023-07-02T14:03:20.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:20.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333454+0000", 2023-07-02T14:03:20.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.025310269 2023-07-02T14:03:20.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358764+0000", 2023-07-02T14:03:20.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012414899999999999 2023-07-02T14:03:20.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:20.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358888+0000", 2023-07-02T14:03:20.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000217028 2023-07-02T14:03:20.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:20.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.359105+0000", 2023-07-02T14:03:20.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012788212 2023-07-02T14:03:20.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:20.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371893+0000", 2023-07-02T14:03:20.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012201569999999999 2023-07-02T14:03:20.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:20.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373114+0000", 2023-07-02T14:03:20.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4512999999999995e-05 2023-07-02T14:03:20.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:20.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373198+0000", 2023-07-02T14:03:20.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132373 2023-07-02T14:03:20.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:20.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373330+0000", 2023-07-02T14:03:20.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999 2023-07-02T14:03:20.443 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.443 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.444 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.444 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2786 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1310720~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:20.444 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:03:20.444 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.9808232130000007, 2023-07-02T14:03:20.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001, 2023-07-02T14:03:20.444 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:20.445 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:20.445 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:20.445 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:20.445 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:20.445 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2786 2023-07-02T14:03:20.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.445 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:20.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:20.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:03:20.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:20.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:20.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:03:20.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:20.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:20.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333419+0000", 2023-07-02T14:03:20.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2009999999999999e-06 2023-07-02T14:03:20.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:20.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333427+0000", 2023-07-02T14:03:20.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0210000000000001e-06 2023-07-02T14:03:20.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:20.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333431+0000", 2023-07-02T14:03:20.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3769999999999999e-06 2023-07-02T14:03:20.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:20.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333434+0000", 2023-07-02T14:03:20.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1161e-05 2023-07-02T14:03:20.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333485+0000", 2023-07-02T14:03:20.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1549000000000001e-05 2023-07-02T14:03:20.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:20.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333496+0000", 2023-07-02T14:03:20.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0273116339999999 2023-07-02T14:03:20.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360808+0000", 2023-07-02T14:03:20.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000115128 2023-07-02T14:03:20.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:20.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360923+0000", 2023-07-02T14:03:20.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018367700000000001 2023-07-02T14:03:20.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:20.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361107+0000", 2023-07-02T14:03:20.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010796923999999999 2023-07-02T14:03:20.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:20.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371904+0000", 2023-07-02T14:03:20.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014983520000000001 2023-07-02T14:03:20.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:20.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373402+0000", 2023-07-02T14:03:20.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0520999999999999e-05 2023-07-02T14:03:20.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:20.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373463+0000", 2023-07-02T14:03:20.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000129095 2023-07-02T14:03:20.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:20.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373592+0000", 2023-07-02T14:03:20.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001 2023-07-02T14:03:20.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.455 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.455 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2787 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1433600~8192] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:20.456 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:03:20.456 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.9808064129999998, 2023-07-02T14:03:20.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999, 2023-07-02T14:03:20.456 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:20.456 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:20.456 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:20.456 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:20.457 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:20.457 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2787 2023-07-02T14:03:20.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.457 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:20.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:20.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:03:20.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:20.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:20.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:03:20.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:20.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:20.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333436+0000", 2023-07-02T14:03:20.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2731999999999999e-05 2023-07-02T14:03:20.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:20.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:03:20.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7440000000000001e-06 2023-07-02T14:03:20.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:20.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333456+0000", 2023-07-02T14:03:20.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8650000000000003e-06 2023-07-02T14:03:20.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:20.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333460+0000", 2023-07-02T14:03:20.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0077e-05 2023-07-02T14:03:20.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333530+0000", 2023-07-02T14:03:20.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4180000000000001e-06 2023-07-02T14:03:20.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:20.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333533+0000", 2023-07-02T14:03:20.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0284002720000001 2023-07-02T14:03:20.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361933+0000", 2023-07-02T14:03:20.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6781999999999996e-05 2023-07-02T14:03:20.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:20.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362030+0000", 2023-07-02T14:03:20.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018305400000000001 2023-07-02T14:03:20.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:20.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362213+0000", 2023-07-02T14:03:20.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097015109999999995 2023-07-02T14:03:20.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:20.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371915+0000", 2023-07-02T14:03:20.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001742295 2023-07-02T14:03:20.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:20.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373657+0000", 2023-07-02T14:03:20.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5980000000000003e-05 2023-07-02T14:03:20.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:20.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373713+0000", 2023-07-02T14:03:20.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011783800000000001 2023-07-02T14:03:20.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:20.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373831+0000", 2023-07-02T14:03:20.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999 2023-07-02T14:03:20.466 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.467 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.467 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.467 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2803 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 851968~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:20.467 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:03:20.467 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.9807645610000009, 2023-07-02T14:03:20.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514, 2023-07-02T14:03:20.468 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:20.468 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:20.468 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:20.468 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:20.468 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:20.468 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2803 2023-07-02T14:03:20.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.469 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:20.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:20.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:03:20.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:20.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:20.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:03:20.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:20.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:20.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:03:20.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4943000000000001e-05 2023-07-02T14:03:20.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:20.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333493+0000", 2023-07-02T14:03:20.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2629999999999997e-06 2023-07-02T14:03:20.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:20.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333500+0000", 2023-07-02T14:03:20.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4159999999999997e-06 2023-07-02T14:03:20.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:20.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333505+0000", 2023-07-02T14:03:20.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4518999999999998e-05 2023-07-02T14:03:20.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333559+0000", 2023-07-02T14:03:20.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5900000000000008e-06 2023-07-02T14:03:20.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:20.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333568+0000", 2023-07-02T14:03:20.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0291061779999999 2023-07-02T14:03:20.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362674+0000", 2023-07-02T14:03:20.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110412 2023-07-02T14:03:20.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:20.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362784+0000", 2023-07-02T14:03:20.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015384599999999999 2023-07-02T14:03:20.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:20.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362938+0000", 2023-07-02T14:03:20.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091682310000000006 2023-07-02T14:03:20.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:20.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372106+0000", 2023-07-02T14:03:20.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017703230000000001 2023-07-02T14:03:20.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:20.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373877+0000", 2023-07-02T14:03:20.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6387000000000002e-05 2023-07-02T14:03:20.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:20.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373923+0000", 2023-07-02T14:03:20.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014099199999999999 2023-07-02T14:03:20.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:20.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374064+0000", 2023-07-02T14:03:20.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514 2023-07-02T14:03:20.478 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.478 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.478 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.479 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2805 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 634880~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:20.479 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:03:20.479 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.9807313860000004, 2023-07-02T14:03:20.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814, 2023-07-02T14:03:20.479 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:20.479 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:20.479 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:20.480 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:20.480 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:20.480 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2805 2023-07-02T14:03:20.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.480 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:20.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:20.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:03:20.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:20.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:20.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:03:20.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:20.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:20.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333511+0000", 2023-07-02T14:03:20.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207092 2023-07-02T14:03:20.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:20.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333718+0000", 2023-07-02T14:03:20.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5110000000000003e-06 2023-07-02T14:03:20.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:20.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333727+0000", 2023-07-02T14:03:20.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3270000000000007e-06 2023-07-02T14:03:20.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:20.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333736+0000", 2023-07-02T14:03:20.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0319999999999999e-05 2023-07-02T14:03:20.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333756+0000", 2023-07-02T14:03:20.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1695000000000001e-05 2023-07-02T14:03:20.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:20.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:03:20.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0295750560000001 2023-07-02T14:03:20.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363343+0000", 2023-07-02T14:03:20.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3387999999999994e-05 2023-07-02T14:03:20.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:20.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363437+0000", 2023-07-02T14:03:20.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018373 2023-07-02T14:03:20.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:20.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363620+0000", 2023-07-02T14:03:20.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0084948769999999996 2023-07-02T14:03:20.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:20.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372115+0000", 2023-07-02T14:03:20.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020082170000000001 2023-07-02T14:03:20.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:20.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374123+0000", 2023-07-02T14:03:20.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8875999999999997e-05 2023-07-02T14:03:20.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:20.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374182+0000", 2023-07-02T14:03:20.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0001272 2023-07-02T14:03:20.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:20.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374309+0000", 2023-07-02T14:03:20.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814 2023-07-02T14:03:20.490 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.490 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.490 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.490 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2806 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 1359872~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:20.490 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:03:20.490 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.9805010549999995, 2023-07-02T14:03:20.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234, 2023-07-02T14:03:20.491 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:20.491 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:20.491 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:20.491 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:20.491 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:20.491 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2806 2023-07-02T14:03:20.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.492 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:20.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:20.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:03:20.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:20.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:20.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:03:20.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:20.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:20.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333741+0000", 2023-07-02T14:03:20.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3978999999999999e-05 2023-07-02T14:03:20.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:20.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333755+0000", 2023-07-02T14:03:20.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:03:20.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:20.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333761+0000", 2023-07-02T14:03:20.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7010000000000001e-06 2023-07-02T14:03:20.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:20.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333764+0000", 2023-07-02T14:03:20.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5086000000000003e-05 2023-07-02T14:03:20.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333799+0000", 2023-07-02T14:03:20.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6389999999999992e-06 2023-07-02T14:03:20.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:20.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333806+0000", 2023-07-02T14:03:20.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.030496675 2023-07-02T14:03:20.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364303+0000", 2023-07-02T14:03:20.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011382700000000001 2023-07-02T14:03:20.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:20.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364417+0000", 2023-07-02T14:03:20.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192058 2023-07-02T14:03:20.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:20.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364609+0000", 2023-07-02T14:03:20.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075132130000000004 2023-07-02T14:03:20.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:20.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372122+0000", 2023-07-02T14:03:20.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022731750000000001 2023-07-02T14:03:20.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:20.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374395+0000", 2023-07-02T14:03:20.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5277000000000005e-05 2023-07-02T14:03:20.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:20.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374460+0000", 2023-07-02T14:03:20.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017074599999999999 2023-07-02T14:03:20.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:20.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374631+0000", 2023-07-02T14:03:20.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234 2023-07-02T14:03:20.501 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.501 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.501 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.502 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 868352~49152] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:20.502 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:03:20.502 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.9804743410000007, 2023-07-02T14:03:20.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001, 2023-07-02T14:03:20.502 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:20.502 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:20.503 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:20.503 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:20.503 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:20.503 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:03:20.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.503 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:20.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:20.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:03:20.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:20.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:20.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:03:20.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:20.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:20.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:03:20.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6355000000000001e-05 2023-07-02T14:03:20.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:20.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333794+0000", 2023-07-02T14:03:20.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8329999999999996e-06 2023-07-02T14:03:20.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:20.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333801+0000", 2023-07-02T14:03:20.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5419999999999996e-06 2023-07-02T14:03:20.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:20.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333807+0000", 2023-07-02T14:03:20.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2023-07-02T14:03:20.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333826+0000", 2023-07-02T14:03:20.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.303e-06 2023-07-02T14:03:20.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:20.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333830+0000", 2023-07-02T14:03:20.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.031137559 2023-07-02T14:03:20.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364967+0000", 2023-07-02T14:03:20.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1390000000000004e-05 2023-07-02T14:03:20.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:20.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365059+0000", 2023-07-02T14:03:20.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017597399999999999 2023-07-02T14:03:20.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:20.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365234+0000", 2023-07-02T14:03:20.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068943950000000002 2023-07-02T14:03:20.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:20.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372129+0000", 2023-07-02T14:03:20.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026189910000000002 2023-07-02T14:03:20.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:20.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374748+0000", 2023-07-02T14:03:20.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6613999999999996e-05 2023-07-02T14:03:20.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:20.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374814+0000", 2023-07-02T14:03:20.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502300000000001 2023-07-02T14:03:20.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:20.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374950+0000", 2023-07-02T14:03:20.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001 2023-07-02T14:03:20.513 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.513 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.513 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.513 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2815 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 917504~49152] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:20.513 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:03:20.514 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.976260108, 2023-07-02T14:03:20.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998, 2023-07-02T14:03:20.514 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:20.514 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:20.514 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:20.514 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:20.514 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:20.514 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2815 2023-07-02T14:03:20.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.515 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:20.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:20.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:03:20.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:20.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:20.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:03:20.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:03:20.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:20.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337980+0000", 2023-07-02T14:03:20.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017126700000000001 2023-07-02T14:03:20.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:20.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338152+0000", 2023-07-02T14:03:20.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2938999999999999e-05 2023-07-02T14:03:20.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:20.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338164+0000", 2023-07-02T14:03:20.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4448999999999999e-05 2023-07-02T14:03:20.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:20.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:03:20.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0281812 2023-07-02T14:03:20.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366370+0000", 2023-07-02T14:03:20.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1840000000000002e-05 2023-07-02T14:03:20.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:20.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366462+0000", 2023-07-02T14:03:20.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000186112 2023-07-02T14:03:20.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:20.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366648+0000", 2023-07-02T14:03:20.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054905709999999996 2023-07-02T14:03:20.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:20.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372139+0000", 2023-07-02T14:03:20.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044954790000000001 2023-07-02T14:03:20.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:20.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376634+0000", 2023-07-02T14:03:20.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4621000000000004e-05 2023-07-02T14:03:20.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:20.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376699+0000", 2023-07-02T14:03:20.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012862299999999999 2023-07-02T14:03:20.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:20.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376827+0000", 2023-07-02T14:03:20.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998 2023-07-02T14:03:20.523 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.523 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.523 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.524 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2816 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 917504~65536] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:20.524 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:03:20.524 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.9760457290000009, 2023-07-02T14:03:20.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001, 2023-07-02T14:03:20.524 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:20.524 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:20.524 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:20.525 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:20.525 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:20.525 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2816 2023-07-02T14:03:20.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.525 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:20.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:20.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:03:20.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:20.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:20.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:03:20.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:20.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:20.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338196+0000", 2023-07-02T14:03:20.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6286999999999999e-05 2023-07-02T14:03:20.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:20.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338243+0000", 2023-07-02T14:03:20.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7030000000000001e-06 2023-07-02T14:03:20.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:20.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338252+0000", 2023-07-02T14:03:20.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:03:20.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:20.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338258+0000", 2023-07-02T14:03:20.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032085734999999997 2023-07-02T14:03:20.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370344+0000", 2023-07-02T14:03:20.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108621 2023-07-02T14:03:20.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:20.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370452+0000", 2023-07-02T14:03:20.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014936499999999999 2023-07-02T14:03:20.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:20.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370602+0000", 2023-07-02T14:03:20.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001778012 2023-07-02T14:03:20.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:20.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372380+0000", 2023-07-02T14:03:20.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044876380000000004 2023-07-02T14:03:20.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:20.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376867+0000", 2023-07-02T14:03:20.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9870999999999999e-05 2023-07-02T14:03:20.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:20.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376897+0000", 2023-07-02T14:03:20.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011401 2023-07-02T14:03:20.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:20.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.377011+0000", 2023-07-02T14:03:20.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001 2023-07-02T14:03:20.533 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.534 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.534 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.534 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2858 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1441792~2359296] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:20.534 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.378969+0000", 2023-07-02T14:03:20.534 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.9352735320000001, 2023-07-02T14:03:20.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029176497999999999, 2023-07-02T14:03:20.535 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:20.535 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:20.535 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:20.535 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:20.535 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:20.535 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2858 2023-07-02T14:03:20.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.535 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:20.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:20.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.378969+0000", 2023-07-02T14:03:20.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:20.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:20.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.378969+0000", 2023-07-02T14:03:20.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:20.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:20.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.378969+0000", 2023-07-02T14:03:20.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011672914 2023-07-02T14:03:20.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:20.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.390642+0000", 2023-07-02T14:03:20.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2140999999999999e-05 2023-07-02T14:03:20.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:20.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.390654+0000", 2023-07-02T14:03:20.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5440000000000004e-06 2023-07-02T14:03:20.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:20.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.390660+0000", 2023-07-02T14:03:20.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011227500000000001 2023-07-02T14:03:20.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.390773+0000", 2023-07-02T14:03:20.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100979 2023-07-02T14:03:20.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:20.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.390873+0000", 2023-07-02T14:03:20.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021888100000000001 2023-07-02T14:03:20.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:20.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.391092+0000", 2023-07-02T14:03:20.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015316722 2023-07-02T14:03:20.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:20.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.406409+0000", 2023-07-02T14:03:20.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015790299999999999 2023-07-02T14:03:20.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:20.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.407988+0000", 2023-07-02T14:03:20.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7111000000000003e-05 2023-07-02T14:03:20.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:20.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.408045+0000", 2023-07-02T14:03:20.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010076799999999999 2023-07-02T14:03:20.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:20.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.408146+0000", 2023-07-02T14:03:20.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029176497999999999 2023-07-02T14:03:20.544 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.544 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.544 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.544 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2944 2.4 2:25711f4d:::rbd_data.125de1449605.00000000000000fa:head [write 65536~2097152] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:20.545 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.470424+0000", 2023-07-02T14:03:20.545 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.8438191160000006, 2023-07-02T14:03:20.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029419590999999998, 2023-07-02T14:03:20.545 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:20.545 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:20.545 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:20.545 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:20.545 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:20.546 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2944 2023-07-02T14:03:20.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.546 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:20.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:20.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.470424+0000", 2023-07-02T14:03:20.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:20.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:20.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.470424+0000", 2023-07-02T14:03:20.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:03:20.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:20.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.470421+0000", 2023-07-02T14:03:20.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0086348350000000004 2023-07-02T14:03:20.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:20.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479056+0000", 2023-07-02T14:03:20.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1942000000000001e-05 2023-07-02T14:03:20.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:20.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479078+0000", 2023-07-02T14:03:20.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8130000000000002e-06 2023-07-02T14:03:20.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:20.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479087+0000", 2023-07-02T14:03:20.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4845999999999995e-05 2023-07-02T14:03:20.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:20.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479152+0000", 2023-07-02T14:03:20.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013724700000000001 2023-07-02T14:03:20.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:20.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479289+0000", 2023-07-02T14:03:20.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000234665 2023-07-02T14:03:20.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:20.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479524+0000", 2023-07-02T14:03:20.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081727950000000001 2023-07-02T14:03:20.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:20.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.487697+0000", 2023-07-02T14:03:20.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011956079999999999 2023-07-02T14:03:20.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:20.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.499653+0000", 2023-07-02T14:03:20.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8978999999999999e-05 2023-07-02T14:03:20.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:20.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.499712+0000", 2023-07-02T14:03:20.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013181299999999999 2023-07-02T14:03:20.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.499844+0000", 2023-07-02T14:03:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029419590999999998 2023-07-02T14:03:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:20.555 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:20.555 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:20.555 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:03:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:03:20.557 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:20.556+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:20.657 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:20.657+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:20.738 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:20.738 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:20.748 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:03:20.791 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:20.790+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:20.859 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:20.858+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:20.872 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:20.873 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:20.873 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:20.873 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:20.873 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:20.882 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:03:20.961 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:20.961+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:21.050 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:21.050 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:03:21.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:03:21.050 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:21.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.050 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:289 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:21.051 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:21.051 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.734860884, 2023-07-02T14:03:21.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051413158, 2023-07-02T14:03:21.051 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:21.051 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:21.051 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:21.051 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:21.052 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:21.052 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 289 2023-07-02T14:03:21.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.052 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:21.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:21.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:21.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:21.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:21.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:21.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:21.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:21.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:21.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1689e-05 2023-07-02T14:03:21.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:21.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312209+0000", 2023-07-02T14:03:21.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1070000000000005e-06 2023-07-02T14:03:21.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:21.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312217+0000", 2023-07-02T14:03:21.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.634e-06 2023-07-02T14:03:21.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:21.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312222+0000", 2023-07-02T14:03:21.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040512761000000001 2023-07-02T14:03:21.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352735+0000", 2023-07-02T14:03:21.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000119184 2023-07-02T14:03:21.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:21.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352854+0000", 2023-07-02T14:03:21.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021481899999999999 2023-07-02T14:03:21.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:21.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353069+0000", 2023-07-02T14:03:21.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001601707 2023-07-02T14:03:21.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:21.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354670+0000", 2023-07-02T14:03:21.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088168570000000009 2023-07-02T14:03:21.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:21.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363487+0000", 2023-07-02T14:03:21.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9635999999999997e-05 2023-07-02T14:03:21.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:21.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363537+0000", 2023-07-02T14:03:21.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4292999999999998e-05 2023-07-02T14:03:21.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:21.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363581+0000", 2023-07-02T14:03:21.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051413158 2023-07-02T14:03:21.060 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.060 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.061 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.061 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:290 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:21.061 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:21.061 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.734801638999997, 2023-07-02T14:03:21.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051455062000000003, 2023-07-02T14:03:21.062 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:21.062 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:21.062 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:21.062 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:21.062 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:21.062 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 290 2023-07-02T14:03:21.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.062 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:21.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:21.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:21.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:21.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:21.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:21.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:21.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:21.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:21.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6469000000000003e-05 2023-07-02T14:03:21.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:21.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312293+0000", 2023-07-02T14:03:21.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4630000000000007e-06 2023-07-02T14:03:21.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:21.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312303+0000", 2023-07-02T14:03:21.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8969999999999999e-06 2023-07-02T14:03:21.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:21.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312307+0000", 2023-07-02T14:03:21.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041261548000000002 2023-07-02T14:03:21.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353569+0000", 2023-07-02T14:03:21.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9594999999999999e-05 2023-07-02T14:03:21.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:21.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353659+0000", 2023-07-02T14:03:21.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018390600000000001 2023-07-02T14:03:21.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:21.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353843+0000", 2023-07-02T14:03:21.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035173100000000001 2023-07-02T14:03:21.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:21.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357360+0000", 2023-07-02T14:03:21.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062295670000000001 2023-07-02T14:03:21.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:21.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363589+0000", 2023-07-02T14:03:21.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9854000000000001e-05 2023-07-02T14:03:21.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:21.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363639+0000", 2023-07-02T14:03:21.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3133999999999997e-05 2023-07-02T14:03:21.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:21.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363682+0000", 2023-07-02T14:03:21.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051455062000000003 2023-07-02T14:03:21.071 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.071 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.071 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.071 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:291 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:21.071 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:21.072 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.734715600000001, 2023-07-02T14:03:21.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051527860000000002, 2023-07-02T14:03:21.072 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:21.072 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:21.072 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:21.072 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:21.072 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:21.072 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 291 2023-07-02T14:03:21.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.073 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:21.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:21.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:21.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:21.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:21.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:21.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:21.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:21.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:21.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2448999999999998e-05 2023-07-02T14:03:21.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:21.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312375+0000", 2023-07-02T14:03:21.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5259999999999993e-06 2023-07-02T14:03:21.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:21.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312384+0000", 2023-07-02T14:03:21.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0610000000000003e-06 2023-07-02T14:03:21.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:21.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312389+0000", 2023-07-02T14:03:21.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042326111 2023-07-02T14:03:21.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354715+0000", 2023-07-02T14:03:21.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3998999999999995e-05 2023-07-02T14:03:21.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:21.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354779+0000", 2023-07-02T14:03:21.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010269500000000001 2023-07-02T14:03:21.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:21.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354882+0000", 2023-07-02T14:03:21.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002533736 2023-07-02T14:03:21.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:21.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357415+0000", 2023-07-02T14:03:21.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062765700000000004 2023-07-02T14:03:21.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:21.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363692+0000", 2023-07-02T14:03:21.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7095000000000001e-05 2023-07-02T14:03:21.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:21.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363749+0000", 2023-07-02T14:03:21.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2263999999999996e-05 2023-07-02T14:03:21.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:21.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363841+0000", 2023-07-02T14:03:21.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051527860000000002 2023-07-02T14:03:21.081 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.081 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.081 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.082 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:292 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:21.082 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:03:21.082 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.734633352000003, 2023-07-02T14:03:21.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051556056000000003, 2023-07-02T14:03:21.082 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:21.082 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:21.082 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:21.083 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:21.083 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:21.083 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 292 2023-07-02T14:03:21.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.083 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:21.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:21.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:03:21.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:21.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:21.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:03:21.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:21.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:21.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312395+0000", 2023-07-02T14:03:21.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9624e-05 2023-07-02T14:03:21.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:21.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312425+0000", 2023-07-02T14:03:21.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:03:21.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:21.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312432+0000", 2023-07-02T14:03:21.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7550000000000002e-06 2023-07-02T14:03:21.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:21.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312437+0000", 2023-07-02T14:03:21.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.043823036000000003 2023-07-02T14:03:21.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356260+0000", 2023-07-02T14:03:21.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6021999999999997e-05 2023-07-02T14:03:21.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:21.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356346+0000", 2023-07-02T14:03:21.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020076600000000001 2023-07-02T14:03:21.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:21.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356547+0000", 2023-07-02T14:03:21.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020766410000000002 2023-07-02T14:03:21.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:21.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358623+0000", 2023-07-02T14:03:21.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005231192 2023-07-02T14:03:21.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:21.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363854+0000", 2023-07-02T14:03:21.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3423e-05 2023-07-02T14:03:21.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:21.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363908+0000", 2023-07-02T14:03:21.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3825999999999997e-05 2023-07-02T14:03:21.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:21.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363952+0000", 2023-07-02T14:03:21.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051556056000000003 2023-07-02T14:03:21.091 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.091 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.092 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.092 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:293 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:21.092 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:03:21.092 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.734587181999999, 2023-07-02T14:03:21.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051614769999999997, 2023-07-02T14:03:21.093 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:21.093 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:21.093 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:21.093 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:21.093 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:21.093 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 293 2023-07-02T14:03:21.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.093 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:21.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:21.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:03:21.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:21.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:21.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:03:21.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:21.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:21.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312441+0000", 2023-07-02T14:03:21.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9609999999999999e-05 2023-07-02T14:03:21.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:21.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312471+0000", 2023-07-02T14:03:21.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7069999999999992e-06 2023-07-02T14:03:21.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:21.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312478+0000", 2023-07-02T14:03:21.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1190000000000002e-06 2023-07-02T14:03:21.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:21.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312485+0000", 2023-07-02T14:03:21.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.044956387 2023-07-02T14:03:21.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357441+0000", 2023-07-02T14:03:21.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4134999999999994e-05 2023-07-02T14:03:21.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:21.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357515+0000", 2023-07-02T14:03:21.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000147408 2023-07-02T14:03:21.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:21.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357663+0000", 2023-07-02T14:03:21.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056472639999999999 2023-07-02T14:03:21.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:21.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363310+0000", 2023-07-02T14:03:21.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00065402600000000004 2023-07-02T14:03:21.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:21.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363964+0000", 2023-07-02T14:03:21.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0497000000000001e-05 2023-07-02T14:03:21.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:21.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364014+0000", 2023-07-02T14:03:21.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2243000000000002e-05 2023-07-02T14:03:21.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:21.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364057+0000", 2023-07-02T14:03:21.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051614769999999997 2023-07-02T14:03:21.102 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.102 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.102 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.102 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:294 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1048576~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:21.102 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:03:21.103 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.734539183000003, 2023-07-02T14:03:21.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051679527000000003, 2023-07-02T14:03:21.103 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:21.103 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:21.103 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:21.103 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:21.103 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:21.104 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 294 2023-07-02T14:03:21.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.104 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:21.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:21.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:03:21.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:21.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:21.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:03:21.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:21.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:21.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312489+0000", 2023-07-02T14:03:21.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8204999999999998e-05 2023-07-02T14:03:21.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:21.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312517+0000", 2023-07-02T14:03:21.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7759999999999997e-06 2023-07-02T14:03:21.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:21.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312525+0000", 2023-07-02T14:03:21.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8730000000000001e-06 2023-07-02T14:03:21.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:21.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312532+0000", 2023-07-02T14:03:21.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.045442206999999998 2023-07-02T14:03:21.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357974+0000", 2023-07-02T14:03:21.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4422000000000002e-05 2023-07-02T14:03:21.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:21.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358029+0000", 2023-07-02T14:03:21.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149519 2023-07-02T14:03:21.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:21.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358178+0000", 2023-07-02T14:03:21.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051779390000000003 2023-07-02T14:03:21.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:21.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363356+0000", 2023-07-02T14:03:21.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00070707699999999997 2023-07-02T14:03:21.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:21.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364063+0000", 2023-07-02T14:03:21.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6835999999999999e-05 2023-07-02T14:03:21.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:21.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364090+0000", 2023-07-02T14:03:21.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9263000000000003e-05 2023-07-02T14:03:21.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:21.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364169+0000", 2023-07-02T14:03:21.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051679527000000003 2023-07-02T14:03:21.112 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.113 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.113 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.113 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:03:21.113 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:21.113 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.594928981000001, 2023-07-02T14:03:21.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:03:21.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:21.114 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:21.114 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:21.114 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:21.114 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:21.114 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:03:21.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.114 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:21.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:21.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:21.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:21.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:21.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:21.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:21.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:21.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:03:21.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:03:21.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:21.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:03:21.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:03:21.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:21.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:03:21.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:03:21.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:21.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:03:21.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:03:21.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:03:21.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:03:21.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:21.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:03:21.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:03:21.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:03:21.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:03:21.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:21.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:03:21.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:03:21.121 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.121 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.121 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.121 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:03:21.121 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:03:21.122 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.724235973000001, 2023-07-02T14:03:21.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:03:21.122 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:21.122 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:03:21.122 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:21.122 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:03:21.122 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:03:21.123 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:03:21.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.123 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:21.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:21.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:03:21.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:21.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:21.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:03:21.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:21.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:21.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:03:21.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:03:21.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:21.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:03:21.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:03:21.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:21.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:03:21.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:03:21.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:21.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:03:21.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:03:21.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:03:21.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:03:21.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:21.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:03:21.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:03:21.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:03:21.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:03:21.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:21.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:03:21.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:03:21.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:21.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:03:21.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:03:21.130 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.130 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.130 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.130 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:21.131 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:03:21.131 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.724111252, 2023-07-02T14:03:21.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:03:21.131 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:21.131 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:21.131 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:21.131 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:21.131 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:21.132 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:03:21.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.132 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:21.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:21.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:03:21.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:21.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:21.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:03:21.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:21.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:21.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:03:21.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:03:21.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:21.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:03:21.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:03:21.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:21.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:03:21.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:03:21.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:21.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:03:21.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:03:21.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:03:21.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:03:21.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:21.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:03:21.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:03:21.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:03:21.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:03:21.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:21.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:03:21.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:03:21.138 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.138 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.138 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.139 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:21.139 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:03:21.139 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.717261368000001, 2023-07-02T14:03:21.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:03:21.139 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:21.139 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:21.140 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:21.140 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:21.140 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:21.140 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:03:21.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.140 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:21.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:21.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:03:21.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:21.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:21.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:03:21.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:03:21.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:21.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:03:21.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:03:21.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:21.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:03:21.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:03:21.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:21.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:03:21.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:03:21.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:21.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:03:21.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:03:21.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:03:21.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:03:21.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:21.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:03:21.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:03:21.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:03:21.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:03:21.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:21.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:03:21.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:03:21.147 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.147 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.147 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.147 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:21.147 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:03:21.147 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.717198442999999, 2023-07-02T14:03:21.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:03:21.148 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:21.148 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:21.148 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:21.148 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:21.148 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:21.148 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:03:21.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.149 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:21.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:21.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:03:21.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:21.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:21.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:03:21.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:21.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:21.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:03:21.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:03:21.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:21.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:03:21.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:03:21.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:21.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:03:21.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:03:21.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:21.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:03:21.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:03:21.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:03:21.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:03:21.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:21.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:03:21.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:03:21.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:03:21.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:03:21.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:21.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:03:21.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:03:21.155 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.155 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.155 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.156 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:21.156 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:03:21.156 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.716898022000001, 2023-07-02T14:03:21.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:03:21.156 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:21.156 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:21.156 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:21.156 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:21.157 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:21.157 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:03:21.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.157 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:21.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:21.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:03:21.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:21.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:21.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:03:21.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:21.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:21.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:03:21.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:03:21.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:21.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:03:21.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:03:21.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:21.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:03:21.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:03:21.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:21.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:03:21.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:03:21.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:03:21.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:03:21.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:21.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:03:21.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:03:21.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:03:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:03:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:03:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:03:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:03:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:03:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:03:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:21.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:03:21.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:03:21.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:21.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:03:21.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:03:21.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:21.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:03:21.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:03:21.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:21.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:03:21.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:03:21.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:21.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:03:21.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:03:21.167 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.167 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.167 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.168 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:03:21.168 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:21.168 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.716373522, 2023-07-02T14:03:21.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:03:21.168 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:21.168 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:03:21.169 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:21.169 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:21.169 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:21.169 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:03:21.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.169 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:21.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:21.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:21.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:21.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:21.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:21.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:21.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:21.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:21.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:03:21.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:21.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:03:21.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:03:21.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:21.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:03:21.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:03:21.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:21.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:03:21.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:03:21.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:03:21.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:03:21.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:21.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:03:21.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:03:21.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:03:21.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:03:21.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:21.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:03:21.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:03:21.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:21.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:03:21.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:03:21.176 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.176 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.176 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.177 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2871 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:21.177 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.382460+0000", 2023-07-02T14:03:21.177 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.6645686939999997, 2023-07-02T14:03:21.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.062523707999999997, 2023-07-02T14:03:21.177 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:21.177 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:21.178 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:21.178 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:21.178 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:21.178 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2871 2023-07-02T14:03:21.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.178 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:21.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:21.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382460+0000", 2023-07-02T14:03:21.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:21.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:21.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382460+0000", 2023-07-02T14:03:21.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:21.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:21.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382460+0000", 2023-07-02T14:03:21.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3189999999999999e-06 2023-07-02T14:03:21.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:21.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382463+0000", 2023-07-02T14:03:21.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7400000000000004e-06 2023-07-02T14:03:21.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:21.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382471+0000", 2023-07-02T14:03:21.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1640000000000003e-06 2023-07-02T14:03:21.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:21.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382475+0000", 2023-07-02T14:03:21.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067762170000000002 2023-07-02T14:03:21.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.389251+0000", 2023-07-02T14:03:21.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3442000000000005e-05 2023-07-02T14:03:21.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.389335+0000", 2023-07-02T14:03:21.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028393860000000002 2023-07-02T14:03:21.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.392174+0000", 2023-07-02T14:03:21.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1433000000000002e-05 2023-07-02T14:03:21.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.392206+0000", 2023-07-02T14:03:21.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051865440000000004 2023-07-02T14:03:21.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.397392+0000", 2023-07-02T14:03:21.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.321e-05 2023-07-02T14:03:21.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.397455+0000", 2023-07-02T14:03:21.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012230776 2023-07-02T14:03:21.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.409686+0000", 2023-07-02T14:03:21.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8430999999999996e-05 2023-07-02T14:03:21.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.409754+0000", 2023-07-02T14:03:21.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052440610000000004 2023-07-02T14:03:21.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.414999+0000", 2023-07-02T14:03:21.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0063999999999999e-05 2023-07-02T14:03:21.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.415069+0000", 2023-07-02T14:03:21.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035077749999999999 2023-07-02T14:03:21.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.418576+0000", 2023-07-02T14:03:21.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0323999999999999e-05 2023-07-02T14:03:21.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.418657+0000", 2023-07-02T14:03:21.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018539512000000001 2023-07-02T14:03:21.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.437196+0000", 2023-07-02T14:03:21.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0704000000000004e-05 2023-07-02T14:03:21.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:21.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.437287+0000", 2023-07-02T14:03:21.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042943000000000001 2023-07-02T14:03:21.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:21.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.437716+0000", 2023-07-02T14:03:21.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016934700000000001 2023-07-02T14:03:21.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:21.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.439410+0000", 2023-07-02T14:03:21.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0053641770000000004 2023-07-02T14:03:21.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:21.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.444774+0000", 2023-07-02T14:03:21.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0989000000000003e-05 2023-07-02T14:03:21.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:21.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.444825+0000", 2023-07-02T14:03:21.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000158982 2023-07-02T14:03:21.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:21.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.444984+0000", 2023-07-02T14:03:21.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.062523707999999997 2023-07-02T14:03:21.194 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.194 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.194 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.195 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2872 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:21.195 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.382481+0000", 2023-07-02T14:03:21.195 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.6645480930000005, 2023-07-02T14:03:21.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.065455442000000003, 2023-07-02T14:03:21.195 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:21.195 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:21.196 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:21.196 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:21.196 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:21.196 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2872 2023-07-02T14:03:21.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.196 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:21.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:21.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382481+0000", 2023-07-02T14:03:21.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:21.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:21.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382481+0000", 2023-07-02T14:03:21.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:21.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:21.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382480+0000", 2023-07-02T14:03:21.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2509999999999999e-06 2023-07-02T14:03:21.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:21.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382484+0000", 2023-07-02T14:03:21.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:03:21.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:21.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382490+0000", 2023-07-02T14:03:21.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3000000000000003e-06 2023-07-02T14:03:21.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:21.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382494+0000", 2023-07-02T14:03:21.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0069223369999999998 2023-07-02T14:03:21.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.389416+0000", 2023-07-02T14:03:21.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5201000000000001e-05 2023-07-02T14:03:21.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.389481+0000", 2023-07-02T14:03:21.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027921510000000001 2023-07-02T14:03:21.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.392274+0000", 2023-07-02T14:03:21.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6381000000000002e-05 2023-07-02T14:03:21.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.392320+0000", 2023-07-02T14:03:21.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051686450000000004 2023-07-02T14:03:21.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.397489+0000", 2023-07-02T14:03:21.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0806e-05 2023-07-02T14:03:21.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.397549+0000", 2023-07-02T14:03:21.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012259144 2023-07-02T14:03:21.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.409809+0000", 2023-07-02T14:03:21.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7465000000000005e-05 2023-07-02T14:03:21.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.409876+0000", 2023-07-02T14:03:21.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052799830000000002 2023-07-02T14:03:21.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.415156+0000", 2023-07-02T14:03:21.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2080000000000001e-05 2023-07-02T14:03:21.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.415228+0000", 2023-07-02T14:03:21.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034778159999999999 2023-07-02T14:03:21.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.418706+0000", 2023-07-02T14:03:21.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7431000000000001e-05 2023-07-02T14:03:21.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.418743+0000", 2023-07-02T14:03:21.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020264971999999999 2023-07-02T14:03:21.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.439008+0000", 2023-07-02T14:03:21.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2211000000000003e-05 2023-07-02T14:03:21.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.439090+0000", 2023-07-02T14:03:21.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060620709999999996 2023-07-02T14:03:21.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.445153+0000", 2023-07-02T14:03:21.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5259000000000006e-05 2023-07-02T14:03:21.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:21.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.445228+0000", 2023-07-02T14:03:21.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030277499999999997 2023-07-02T14:03:21.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:21.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.445531+0000", 2023-07-02T14:03:21.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013477540000000001 2023-07-02T14:03:21.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:21.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.446878+0000", 2023-07-02T14:03:21.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00088152200000000003 2023-07-02T14:03:21.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:21.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.447760+0000", 2023-07-02T14:03:21.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7722999999999998e-05 2023-07-02T14:03:21.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:21.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.447798+0000", 2023-07-02T14:03:21.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013871599999999999 2023-07-02T14:03:21.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:21.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.447936+0000", 2023-07-02T14:03:21.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.065455442000000003 2023-07-02T14:03:21.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.213 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.214 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.214 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3000 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:21.214 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.518762+0000", 2023-07-02T14:03:21.214 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.5282671870000009, 2023-07-02T14:03:21.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.052957575, 2023-07-02T14:03:21.214 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:21.215 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:21.215 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:21.215 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:21.215 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:21.215 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3000 2023-07-02T14:03:21.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.215 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:21.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:21.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518762+0000", 2023-07-02T14:03:21.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:21.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:21.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518762+0000", 2023-07-02T14:03:21.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:21.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:21.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518761+0000", 2023-07-02T14:03:21.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3069999999999996e-06 2023-07-02T14:03:21.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:21.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518765+0000", 2023-07-02T14:03:21.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6170000000000001e-06 2023-07-02T14:03:21.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:21.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518773+0000", 2023-07-02T14:03:21.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.172e-06 2023-07-02T14:03:21.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:21.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518777+0000", 2023-07-02T14:03:21.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055680249999999999 2023-07-02T14:03:21.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.524345+0000", 2023-07-02T14:03:21.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0938000000000003e-05 2023-07-02T14:03:21.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.524426+0000", 2023-07-02T14:03:21.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091411249999999999 2023-07-02T14:03:21.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.533567+0000", 2023-07-02T14:03:21.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5704000000000002e-05 2023-07-02T14:03:21.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.533603+0000", 2023-07-02T14:03:21.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046842289999999998 2023-07-02T14:03:21.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.538287+0000", 2023-07-02T14:03:21.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5780999999999999e-05 2023-07-02T14:03:21.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.538353+0000", 2023-07-02T14:03:21.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034785279999999998 2023-07-02T14:03:21.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.541832+0000", 2023-07-02T14:03:21.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0024999999999997e-05 2023-07-02T14:03:21.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.541872+0000", 2023-07-02T14:03:21.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024189239999999998 2023-07-02T14:03:21.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.544291+0000", 2023-07-02T14:03:21.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9709e-05 2023-07-02T14:03:21.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.544320+0000", 2023-07-02T14:03:21.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037311990000000001 2023-07-02T14:03:21.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.548051+0000", 2023-07-02T14:03:21.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3347000000000002e-05 2023-07-02T14:03:21.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.548115+0000", 2023-07-02T14:03:21.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078550089999999996 2023-07-02T14:03:21.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.555970+0000", 2023-07-02T14:03:21.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8730000000000003e-05 2023-07-02T14:03:21.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.556019+0000", 2023-07-02T14:03:21.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052420460000000002 2023-07-02T14:03:21.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.561261+0000", 2023-07-02T14:03:21.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4456999999999995e-05 2023-07-02T14:03:21.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:21.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.561345+0000", 2023-07-02T14:03:21.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00047298300000000001 2023-07-02T14:03:21.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:21.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.561818+0000", 2023-07-02T14:03:21.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0098041989999999996 2023-07-02T14:03:21.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:21.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.571622+0000", 2023-07-02T14:03:21.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6109000000000003e-05 2023-07-02T14:03:21.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:21.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.571658+0000", 2023-07-02T14:03:21.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9556e-05 2023-07-02T14:03:21.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:21.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.571688+0000", 2023-07-02T14:03:21.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1454000000000001e-05 2023-07-02T14:03:21.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:21.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.571719+0000", 2023-07-02T14:03:21.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.052957575 2023-07-02T14:03:21.232 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.233 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.233 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.233 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3056 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2023-07-02T14:03:21.233 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:12.080431+0000", 2023-07-02T14:03:21.233 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.9665983439999994, 2023-07-02T14:03:21.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051668310000000002, 2023-07-02T14:03:21.234 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:21.234 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:21.234 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:21.234 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:21.234 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:21.234 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3056 2023-07-02T14:03:21.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.234 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:21.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:21.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080431+0000", 2023-07-02T14:03:21.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:21.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:21.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080431+0000", 2023-07-02T14:03:21.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:21.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:21.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080430+0000", 2023-07-02T14:03:21.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:03:21.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:21.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080434+0000", 2023-07-02T14:03:21.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0520000000000004e-06 2023-07-02T14:03:21.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:21.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080441+0000", 2023-07-02T14:03:21.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3860000000000004e-06 2023-07-02T14:03:21.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:21.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080448+0000", 2023-07-02T14:03:21.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015622699999999999 2023-07-02T14:03:21.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080604+0000", 2023-07-02T14:03:21.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9212000000000003e-05 2023-07-02T14:03:21.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080643+0000", 2023-07-02T14:03:21.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064524869999999998 2023-07-02T14:03:21.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.087096+0000", 2023-07-02T14:03:21.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0238000000000003e-05 2023-07-02T14:03:21.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.087136+0000", 2023-07-02T14:03:21.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028586150000000001 2023-07-02T14:03:21.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.089994+0000", 2023-07-02T14:03:21.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1024999999999999e-05 2023-07-02T14:03:21.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090055+0000", 2023-07-02T14:03:21.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050689359999999996 2023-07-02T14:03:21.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095124+0000", 2023-07-02T14:03:21.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9534999999999998e-05 2023-07-02T14:03:21.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095184+0000", 2023-07-02T14:03:21.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043677159999999998 2023-07-02T14:03:21.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.099552+0000", 2023-07-02T14:03:21.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4551000000000001e-05 2023-07-02T14:03:21.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.099596+0000", 2023-07-02T14:03:21.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038310229999999998 2023-07-02T14:03:21.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103427+0000", 2023-07-02T14:03:21.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0380999999999997e-05 2023-07-02T14:03:21.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103488+0000", 2023-07-02T14:03:21.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056513550000000003 2023-07-02T14:03:21.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109139+0000", 2023-07-02T14:03:21.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1032999999999996e-05 2023-07-02T14:03:21.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109210+0000", 2023-07-02T14:03:21.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025622259999999999 2023-07-02T14:03:21.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.111772+0000", 2023-07-02T14:03:21.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5722000000000005e-05 2023-07-02T14:03:21.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.111838+0000", 2023-07-02T14:03:21.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047256870000000001 2023-07-02T14:03:21.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116564+0000", 2023-07-02T14:03:21.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8287000000000005e-05 2023-07-02T14:03:21.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116632+0000", 2023-07-02T14:03:21.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059677539999999996 2023-07-02T14:03:21.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122600+0000", 2023-07-02T14:03:21.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7799999999999995e-05 2023-07-02T14:03:21.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122667+0000", 2023-07-02T14:03:21.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033116230000000001 2023-07-02T14:03:21.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.125979+0000", 2023-07-02T14:03:21.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1271000000000002e-05 2023-07-02T14:03:21.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126060+0000", 2023-07-02T14:03:21.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019737790000000002 2023-07-02T14:03:21.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.128034+0000", 2023-07-02T14:03:21.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00054109199999999996 2023-07-02T14:03:21.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:21.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.128575+0000", 2023-07-02T14:03:21.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050589700000000005 2023-07-02T14:03:21.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:21.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.129081+0000", 2023-07-02T14:03:21.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018622420000000001 2023-07-02T14:03:21.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:21.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.130943+0000", 2023-07-02T14:03:21.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010268079999999999 2023-07-02T14:03:21.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:21.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.131970+0000", 2023-07-02T14:03:21.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3554000000000003e-05 2023-07-02T14:03:21.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:21.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.132024+0000", 2023-07-02T14:03:21.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5173999999999995e-05 2023-07-02T14:03:21.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:21.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.132099+0000", 2023-07-02T14:03:21.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051668310000000002 2023-07-02T14:03:21.257 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.257 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.257 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.257 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3057 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2023-07-02T14:03:21.257 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:12.080462+0000", 2023-07-02T14:03:21.258 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.966566791, 2023-07-02T14:03:21.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.055924880000000003, 2023-07-02T14:03:21.258 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:21.258 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:21.258 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:21.258 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:21.258 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:21.258 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3057 2023-07-02T14:03:21.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.259 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:21.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:21.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080462+0000", 2023-07-02T14:03:21.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:21.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:21.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080462+0000", 2023-07-02T14:03:21.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:21.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:21.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080462+0000", 2023-07-02T14:03:21.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6289999999999999e-06 2023-07-02T14:03:21.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:21.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080465+0000", 2023-07-02T14:03:21.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3088000000000001e-05 2023-07-02T14:03:21.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:21.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080478+0000", 2023-07-02T14:03:21.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.101e-06 2023-07-02T14:03:21.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:21.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080483+0000", 2023-07-02T14:03:21.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018486899999999999 2023-07-02T14:03:21.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080668+0000", 2023-07-02T14:03:21.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7156999999999997e-05 2023-07-02T14:03:21.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080706+0000", 2023-07-02T14:03:21.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093726839999999992 2023-07-02T14:03:21.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090078+0000", 2023-07-02T14:03:21.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3180999999999998e-05 2023-07-02T14:03:21.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090131+0000", 2023-07-02T14:03:21.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050952510000000003 2023-07-02T14:03:21.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095227+0000", 2023-07-02T14:03:21.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8848999999999998e-05 2023-07-02T14:03:21.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095285+0000", 2023-07-02T14:03:21.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043427029999999998 2023-07-02T14:03:21.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.099628+0000", 2023-07-02T14:03:21.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1265000000000004e-05 2023-07-02T14:03:21.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.099689+0000", 2023-07-02T14:03:21.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038415099999999998 2023-07-02T14:03:21.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103531+0000", 2023-07-02T14:03:21.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0980000000000001e-05 2023-07-02T14:03:21.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103602+0000", 2023-07-02T14:03:21.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056603110000000003 2023-07-02T14:03:21.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109262+0000", 2023-07-02T14:03:21.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3421000000000001e-05 2023-07-02T14:03:21.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109326+0000", 2023-07-02T14:03:21.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025528510000000001 2023-07-02T14:03:21.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.111879+0000", 2023-07-02T14:03:21.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3395999999999999e-05 2023-07-02T14:03:21.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.111962+0000", 2023-07-02T14:03:21.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047126889999999999 2023-07-02T14:03:21.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116675+0000", 2023-07-02T14:03:21.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7849000000000001e-05 2023-07-02T14:03:21.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116732+0000", 2023-07-02T14:03:21.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059724649999999997 2023-07-02T14:03:21.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122705+0000", 2023-07-02T14:03:21.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3373999999999993e-05 2023-07-02T14:03:21.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122768+0000", 2023-07-02T14:03:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033541339999999999 2023-07-02T14:03:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126122+0000", 2023-07-02T14:03:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2749000000000003e-05 2023-07-02T14:03:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126215+0000", 2023-07-02T14:03:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036440750000000001 2023-07-02T14:03:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.129859+0000", 2023-07-02T14:03:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000103273 2023-07-02T14:03:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.129963+0000", 2023-07-02T14:03:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021629449999999999 2023-07-02T14:03:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.132125+0000", 2023-07-02T14:03:21.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8305e-05 2023-07-02T14:03:21.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:21.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.132164+0000", 2023-07-02T14:03:21.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00046304499999999998 2023-07-02T14:03:21.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:21.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.132627+0000", 2023-07-02T14:03:21.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022532979999999999 2023-07-02T14:03:21.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:21.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.134880+0000", 2023-07-02T14:03:21.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013186859999999999 2023-07-02T14:03:21.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:21.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.136199+0000", 2023-07-02T14:03:21.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1561000000000002e-05 2023-07-02T14:03:21.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:21.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.136260+0000", 2023-07-02T14:03:21.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126685 2023-07-02T14:03:21.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:21.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.136387+0000", 2023-07-02T14:03:21.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.055924880000000003 2023-07-02T14:03:21.281 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.281 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.281 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.281 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3058 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2023-07-02T14:03:21.282 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:12.080489+0000", 2023-07-02T14:03:21.282 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.9665402620000005, 2023-07-02T14:03:21.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.059911803, 2023-07-02T14:03:21.282 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:21.282 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:21.282 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:21.282 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:21.282 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:21.283 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3058 2023-07-02T14:03:21.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.283 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:21.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:21.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080489+0000", 2023-07-02T14:03:21.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:21.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:21.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080489+0000", 2023-07-02T14:03:21.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:21.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:21.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080488+0000", 2023-07-02T14:03:21.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6950000000000002e-06 2023-07-02T14:03:21.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:21.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080492+0000", 2023-07-02T14:03:21.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7950000000000004e-06 2023-07-02T14:03:21.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:21.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080498+0000", 2023-07-02T14:03:21.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1189999999999999e-06 2023-07-02T14:03:21.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:21.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080502+0000", 2023-07-02T14:03:21.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000227817 2023-07-02T14:03:21.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080730+0000", 2023-07-02T14:03:21.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0629e-05 2023-07-02T14:03:21.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080770+0000", 2023-07-02T14:03:21.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093982159999999992 2023-07-02T14:03:21.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090168+0000", 2023-07-02T14:03:21.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5840000000000007e-05 2023-07-02T14:03:21.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090234+0000", 2023-07-02T14:03:21.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050953109999999999 2023-07-02T14:03:21.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095330+0000", 2023-07-02T14:03:21.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104215 2023-07-02T14:03:21.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095434+0000", 2023-07-02T14:03:21.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043199060000000001 2023-07-02T14:03:21.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.099754+0000", 2023-07-02T14:03:21.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000336504 2023-07-02T14:03:21.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.100090+0000", 2023-07-02T14:03:21.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035759479999999998 2023-07-02T14:03:21.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103666+0000", 2023-07-02T14:03:21.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3940000000000001e-05 2023-07-02T14:03:21.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103730+0000", 2023-07-02T14:03:21.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005666333 2023-07-02T14:03:21.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109396+0000", 2023-07-02T14:03:21.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2536999999999998e-05 2023-07-02T14:03:21.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109489+0000", 2023-07-02T14:03:21.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025086790000000002 2023-07-02T14:03:21.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.111998+0000", 2023-07-02T14:03:21.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7811000000000005e-05 2023-07-02T14:03:21.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.112065+0000", 2023-07-02T14:03:21.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046994519999999998 2023-07-02T14:03:21.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116765+0000", 2023-07-02T14:03:21.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1928000000000002e-05 2023-07-02T14:03:21.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116817+0000", 2023-07-02T14:03:21.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005988345 2023-07-02T14:03:21.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122805+0000", 2023-07-02T14:03:21.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7315999999999998e-05 2023-07-02T14:03:21.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122862+0000", 2023-07-02T14:03:21.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034274489999999999 2023-07-02T14:03:21.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126290+0000", 2023-07-02T14:03:21.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8703999999999998e-05 2023-07-02T14:03:21.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126359+0000", 2023-07-02T14:03:21.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003699889 2023-07-02T14:03:21.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.130058+0000", 2023-07-02T14:03:21.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110308 2023-07-02T14:03:21.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.130169+0000", 2023-07-02T14:03:21.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003130295 2023-07-02T14:03:21.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.133299+0000", 2023-07-02T14:03:21.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6588999999999995e-05 2023-07-02T14:03:21.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.133376+0000", 2023-07-02T14:03:21.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030628389999999999 2023-07-02T14:03:21.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.136439+0000", 2023-07-02T14:03:21.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010477 2023-07-02T14:03:21.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:21.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.136543+0000", 2023-07-02T14:03:21.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00047791199999999999 2023-07-02T14:03:21.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:21.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.137021+0000", 2023-07-02T14:03:21.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00150596 2023-07-02T14:03:21.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:21.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.138527+0000", 2023-07-02T14:03:21.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017228510000000001 2023-07-02T14:03:21.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:21.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140250+0000", 2023-07-02T14:03:21.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8615000000000006e-05 2023-07-02T14:03:21.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:21.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140319+0000", 2023-07-02T14:03:21.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1852000000000002e-05 2023-07-02T14:03:21.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:21.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140400+0000", 2023-07-02T14:03:21.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.059911803 2023-07-02T14:03:21.306 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.306 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.307 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.307 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3059 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2023-07-02T14:03:21.307 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:12.080509+0000", 2023-07-02T14:03:21.307 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.9665197259999996, 2023-07-02T14:03:21.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.062340277999999999, 2023-07-02T14:03:21.307 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:21.308 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:21.308 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:21.308 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:21.308 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:21.308 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3059 2023-07-02T14:03:21.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.308 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:21.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:21.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080509+0000", 2023-07-02T14:03:21.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:21.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:21.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080509+0000", 2023-07-02T14:03:21.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:21.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:21.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080509+0000", 2023-07-02T14:03:21.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5099999999999999e-06 2023-07-02T14:03:21.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:21.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080512+0000", 2023-07-02T14:03:21.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8880000000000002e-06 2023-07-02T14:03:21.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:21.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080518+0000", 2023-07-02T14:03:21.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1520000000000002e-06 2023-07-02T14:03:21.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:21.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080522+0000", 2023-07-02T14:03:21.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026573699999999999 2023-07-02T14:03:21.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080788+0000", 2023-07-02T14:03:21.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7478999999999998e-05 2023-07-02T14:03:21.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080826+0000", 2023-07-02T14:03:21.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0094689049999999997 2023-07-02T14:03:21.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090294+0000", 2023-07-02T14:03:21.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5257000000000001e-05 2023-07-02T14:03:21.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090370+0000", 2023-07-02T14:03:21.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005117229 2023-07-02T14:03:21.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095487+0000", 2023-07-02T14:03:21.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7698e-05 2023-07-02T14:03:21.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095575+0000", 2023-07-02T14:03:21.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045423210000000002 2023-07-02T14:03:21.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.100117+0000", 2023-07-02T14:03:21.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1317999999999999e-05 2023-07-02T14:03:21.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.100168+0000", 2023-07-02T14:03:21.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035847650000000002 2023-07-02T14:03:21.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103753+0000", 2023-07-02T14:03:21.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3754999999999998e-05 2023-07-02T14:03:21.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103787+0000", 2023-07-02T14:03:21.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0057476209999999996 2023-07-02T14:03:21.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109534+0000", 2023-07-02T14:03:21.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9867000000000001e-05 2023-07-02T14:03:21.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109594+0000", 2023-07-02T14:03:21.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024983840000000002 2023-07-02T14:03:21.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.112093+0000", 2023-07-02T14:03:21.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0919e-05 2023-07-02T14:03:21.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.112124+0000", 2023-07-02T14:03:21.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047877880000000003 2023-07-02T14:03:21.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116911+0000", 2023-07-02T14:03:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1793999999999996e-05 2023-07-02T14:03:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.117003+0000", 2023-07-02T14:03:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059009860000000004 2023-07-02T14:03:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122904+0000", 2023-07-02T14:03:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8408000000000001e-05 2023-07-02T14:03:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122953+0000", 2023-07-02T14:03:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034502040000000001 2023-07-02T14:03:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126403+0000", 2023-07-02T14:03:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4770000000000005e-05 2023-07-02T14:03:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126468+0000", 2023-07-02T14:03:21.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037365950000000001 2023-07-02T14:03:21.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.130204+0000", 2023-07-02T14:03:21.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7163000000000002e-05 2023-07-02T14:03:21.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.130251+0000", 2023-07-02T14:03:21.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031607509999999998 2023-07-02T14:03:21.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.133412+0000", 2023-07-02T14:03:21.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1032000000000005e-05 2023-07-02T14:03:21.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.133503+0000", 2023-07-02T14:03:21.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041497349999999999 2023-07-02T14:03:21.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.137653+0000", 2023-07-02T14:03:21.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7430999999999997e-05 2023-07-02T14:03:21.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:21.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.137740+0000", 2023-07-02T14:03:21.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026871349999999998 2023-07-02T14:03:21.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:21.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140427+0000", 2023-07-02T14:03:21.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7179999999999998e-05 2023-07-02T14:03:21.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:21.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140485+0000", 2023-07-02T14:03:21.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00039492900000000003 2023-07-02T14:03:21.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:21.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140879+0000", 2023-07-02T14:03:21.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018682359999999999 2023-07-02T14:03:21.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:21.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.142748+0000", 2023-07-02T14:03:21.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4937e-05 2023-07-02T14:03:21.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:21.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.142793+0000", 2023-07-02T14:03:21.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4743999999999998e-05 2023-07-02T14:03:21.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:21.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.142827+0000", 2023-07-02T14:03:21.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2118999999999999e-05 2023-07-02T14:03:21.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.142849+0000", 2023-07-02T14:03:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.062340277999999999 2023-07-02T14:03:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:21.334 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:21.334 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:21.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:03:21.336 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:03:21.336 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:21.334+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:21.436 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:21.436+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:21.504 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:21.505 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:21.515 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:03:21.517 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:21.535 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:21.536 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:21.536+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:21.553 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:21.572 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:21.590 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:21.609 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:03:21.610 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:03:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:03:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:03:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:03:21.611 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:03:21.638 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:21.638+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:21.641 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:21.642 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:21.652 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:03:21.738 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:21.739+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:21.816 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:21.817 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:21.826 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:03:21.839 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:21.839+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:21.939 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:21.939+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:21.964 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:21.964 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:21.974 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:03:22.041 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:22.041+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:22.098 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:22.098 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:22.108 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:03:22.152 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:22.152+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:22.232 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:22.233 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:22.242 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:03:22.243 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:22.242+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:22.343 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:22.343+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:22.407 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:22.407 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:22.418 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:03:22.461 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:22.461+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:22.541 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:22.542 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:22.544 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:22.544+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:22.551 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:03:22.557 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:03:22.645 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:22.645+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:22.716 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:22.717 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:22.726 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:03:22.770 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:22.770+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:22.847 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:22.847+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:22.850 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:22.850 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:22.860 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:03:22.948 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:22.948+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:23.023 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:23.023 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:23.033 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:03:23.049 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:23.049+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:23.150 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:23.150+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:23.156 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:23.156 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:23.166 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:03:23.250 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:23.250+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:23.294 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:23.295 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:23.305 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:03:23.351 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:23.351+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:23.356 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:03:23.364 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap5 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap6 b3fbb028-e267-4d25-a750-93ba18a2316a@snap6.diff 2023-07-02T14:03:23.418 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete... Exporting image: 2% complete...2023-07-02T14:03:23.417+0000 7fcb66bba700 0 -- 172.21.15.163:0/4080229048 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x55cd9df78a90 msgr2=0x55cd9df7ae80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:03:23.420 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:03:23.419+0000 7fcb65bb8700 0 -- 172.21.15.163:0/4080229048 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7fcb4000ee00 msgr2=0x7fcb40011250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-07-02T14:03:23.452 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:23.452+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:23.496 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:23.496 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:23.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:03:23.553 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:23.554+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:23.622 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:03:23.642 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:23.642 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:23.653 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:03:23.680 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 3% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 100% complete...done. 2023-07-02T14:03:23.683 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:03:23.683 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=7 2023-07-02T14:03:23.684 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap7 2023-07-02T14:03:23.694 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:23.694+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:23.755 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:23.756+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:23.762 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:23.780 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:23.798 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:23.816 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:23.820 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:23.821 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:23.821 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:23.821 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:23.831 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:03:23.834 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:23.853 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:23.854 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:03:23.854 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:03:23.854 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:03:23.855 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:03:23.855 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:03:23.855 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:03:23.856 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:23.856+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:23.955 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:23.955 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:23.955 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:23.955 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:23.956 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:23.957 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:23.957+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:23.965 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:03:24.057 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:24.057+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:24.091 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:24.091 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:03:24.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:03:24.091 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:24.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.092 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:289 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:24.092 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:24.092 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.776463640000003, 2023-07-02T14:03:24.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051413158, 2023-07-02T14:03:24.092 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.092 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.093 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.093 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.093 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.093 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 289 2023-07-02T14:03:24.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.093 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:24.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1689e-05 2023-07-02T14:03:24.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312209+0000", 2023-07-02T14:03:24.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1070000000000005e-06 2023-07-02T14:03:24.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312217+0000", 2023-07-02T14:03:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.634e-06 2023-07-02T14:03:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312222+0000", 2023-07-02T14:03:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040512761000000001 2023-07-02T14:03:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352735+0000", 2023-07-02T14:03:24.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000119184 2023-07-02T14:03:24.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352854+0000", 2023-07-02T14:03:24.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021481899999999999 2023-07-02T14:03:24.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:24.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353069+0000", 2023-07-02T14:03:24.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001601707 2023-07-02T14:03:24.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354670+0000", 2023-07-02T14:03:24.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088168570000000009 2023-07-02T14:03:24.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363487+0000", 2023-07-02T14:03:24.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9635999999999997e-05 2023-07-02T14:03:24.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363537+0000", 2023-07-02T14:03:24.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4292999999999998e-05 2023-07-02T14:03:24.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363581+0000", 2023-07-02T14:03:24.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051413158 2023-07-02T14:03:24.102 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.102 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.102 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.103 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:290 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:24.103 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:24.103 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.776404395, 2023-07-02T14:03:24.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051455062000000003, 2023-07-02T14:03:24.103 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.103 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.103 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.104 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.104 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.104 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 290 2023-07-02T14:03:24.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.104 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:24.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:24.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:24.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:24.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6469000000000003e-05 2023-07-02T14:03:24.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312293+0000", 2023-07-02T14:03:24.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4630000000000007e-06 2023-07-02T14:03:24.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312303+0000", 2023-07-02T14:03:24.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8969999999999999e-06 2023-07-02T14:03:24.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312307+0000", 2023-07-02T14:03:24.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041261548000000002 2023-07-02T14:03:24.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353569+0000", 2023-07-02T14:03:24.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9594999999999999e-05 2023-07-02T14:03:24.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353659+0000", 2023-07-02T14:03:24.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018390600000000001 2023-07-02T14:03:24.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:24.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353843+0000", 2023-07-02T14:03:24.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035173100000000001 2023-07-02T14:03:24.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357360+0000", 2023-07-02T14:03:24.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062295670000000001 2023-07-02T14:03:24.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363589+0000", 2023-07-02T14:03:24.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9854000000000001e-05 2023-07-02T14:03:24.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363639+0000", 2023-07-02T14:03:24.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3133999999999997e-05 2023-07-02T14:03:24.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363682+0000", 2023-07-02T14:03:24.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051455062000000003 2023-07-02T14:03:24.113 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.113 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.113 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.113 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:291 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:24.113 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:24.113 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.776318355999997, 2023-07-02T14:03:24.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051527860000000002, 2023-07-02T14:03:24.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.114 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.114 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.114 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.114 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.114 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 291 2023-07-02T14:03:24.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.115 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:24.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:24.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:24.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:24.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2448999999999998e-05 2023-07-02T14:03:24.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312375+0000", 2023-07-02T14:03:24.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5259999999999993e-06 2023-07-02T14:03:24.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312384+0000", 2023-07-02T14:03:24.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0610000000000003e-06 2023-07-02T14:03:24.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312389+0000", 2023-07-02T14:03:24.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042326111 2023-07-02T14:03:24.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354715+0000", 2023-07-02T14:03:24.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3998999999999995e-05 2023-07-02T14:03:24.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354779+0000", 2023-07-02T14:03:24.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010269500000000001 2023-07-02T14:03:24.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:24.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354882+0000", 2023-07-02T14:03:24.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002533736 2023-07-02T14:03:24.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357415+0000", 2023-07-02T14:03:24.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062765700000000004 2023-07-02T14:03:24.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363692+0000", 2023-07-02T14:03:24.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7095000000000001e-05 2023-07-02T14:03:24.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363749+0000", 2023-07-02T14:03:24.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2263999999999996e-05 2023-07-02T14:03:24.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363841+0000", 2023-07-02T14:03:24.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051527860000000002 2023-07-02T14:03:24.123 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.123 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.123 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.124 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:292 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:24.124 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:03:24.124 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.776236107999999, 2023-07-02T14:03:24.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051556056000000003, 2023-07-02T14:03:24.124 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.124 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.124 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.124 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.125 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.125 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 292 2023-07-02T14:03:24.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.125 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:03:24.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:03:24.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:24.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312395+0000", 2023-07-02T14:03:24.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9624e-05 2023-07-02T14:03:24.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312425+0000", 2023-07-02T14:03:24.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:03:24.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312432+0000", 2023-07-02T14:03:24.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7550000000000002e-06 2023-07-02T14:03:24.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312437+0000", 2023-07-02T14:03:24.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.043823036000000003 2023-07-02T14:03:24.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356260+0000", 2023-07-02T14:03:24.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6021999999999997e-05 2023-07-02T14:03:24.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356346+0000", 2023-07-02T14:03:24.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020076600000000001 2023-07-02T14:03:24.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:24.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356547+0000", 2023-07-02T14:03:24.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020766410000000002 2023-07-02T14:03:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358623+0000", 2023-07-02T14:03:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005231192 2023-07-02T14:03:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363854+0000", 2023-07-02T14:03:24.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3423e-05 2023-07-02T14:03:24.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363908+0000", 2023-07-02T14:03:24.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3825999999999997e-05 2023-07-02T14:03:24.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363952+0000", 2023-07-02T14:03:24.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051556056000000003 2023-07-02T14:03:24.133 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.134 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.134 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.134 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:293 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:24.134 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:03:24.134 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.776189938000002, 2023-07-02T14:03:24.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051614769999999997, 2023-07-02T14:03:24.135 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.135 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.135 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.135 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.135 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.135 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 293 2023-07-02T14:03:24.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.136 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:03:24.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:03:24.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:24.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312441+0000", 2023-07-02T14:03:24.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9609999999999999e-05 2023-07-02T14:03:24.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312471+0000", 2023-07-02T14:03:24.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7069999999999992e-06 2023-07-02T14:03:24.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312478+0000", 2023-07-02T14:03:24.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1190000000000002e-06 2023-07-02T14:03:24.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312485+0000", 2023-07-02T14:03:24.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.044956387 2023-07-02T14:03:24.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357441+0000", 2023-07-02T14:03:24.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4134999999999994e-05 2023-07-02T14:03:24.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357515+0000", 2023-07-02T14:03:24.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000147408 2023-07-02T14:03:24.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:24.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357663+0000", 2023-07-02T14:03:24.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056472639999999999 2023-07-02T14:03:24.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363310+0000", 2023-07-02T14:03:24.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00065402600000000004 2023-07-02T14:03:24.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363964+0000", 2023-07-02T14:03:24.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0497000000000001e-05 2023-07-02T14:03:24.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364014+0000", 2023-07-02T14:03:24.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2243000000000002e-05 2023-07-02T14:03:24.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364057+0000", 2023-07-02T14:03:24.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051614769999999997 2023-07-02T14:03:24.145 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.145 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.145 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.145 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:294 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1048576~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:24.145 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:03:24.146 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.776141938999999, 2023-07-02T14:03:24.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051679527000000003, 2023-07-02T14:03:24.146 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.146 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.146 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.146 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.146 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.146 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 294 2023-07-02T14:03:24.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.147 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:03:24.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:03:24.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:24.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312489+0000", 2023-07-02T14:03:24.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8204999999999998e-05 2023-07-02T14:03:24.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312517+0000", 2023-07-02T14:03:24.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7759999999999997e-06 2023-07-02T14:03:24.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312525+0000", 2023-07-02T14:03:24.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8730000000000001e-06 2023-07-02T14:03:24.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312532+0000", 2023-07-02T14:03:24.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.045442206999999998 2023-07-02T14:03:24.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357974+0000", 2023-07-02T14:03:24.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4422000000000002e-05 2023-07-02T14:03:24.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358029+0000", 2023-07-02T14:03:24.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149519 2023-07-02T14:03:24.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:24.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358178+0000", 2023-07-02T14:03:24.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051779390000000003 2023-07-02T14:03:24.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363356+0000", 2023-07-02T14:03:24.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00070707699999999997 2023-07-02T14:03:24.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364063+0000", 2023-07-02T14:03:24.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6835999999999999e-05 2023-07-02T14:03:24.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364090+0000", 2023-07-02T14:03:24.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9263000000000003e-05 2023-07-02T14:03:24.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364169+0000", 2023-07-02T14:03:24.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051679527000000003 2023-07-02T14:03:24.156 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.156 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.156 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.156 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:03:24.156 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:24.156 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.636531737, 2023-07-02T14:03:24.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:03:24.157 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.157 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:24.157 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.157 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.157 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.157 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:03:24.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.158 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:24.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:24.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:24.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:03:24.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:03:24.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:03:24.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:03:24.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:03:24.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:03:24.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:03:24.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:03:24.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:03:24.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:03:24.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:24.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:03:24.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:03:24.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:03:24.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:03:24.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:03:24.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:03:24.164 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.165 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.165 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.165 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:03:24.165 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:03:24.165 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.765838729, 2023-07-02T14:03:24.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:03:24.166 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.166 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:03:24.166 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.166 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:03:24.166 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:03:24.166 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:03:24.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.167 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:03:24.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:03:24.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:24.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:03:24.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:03:24.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:03:24.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:03:24.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:03:24.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:03:24.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:03:24.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:03:24.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:03:24.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:03:24.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:24.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:03:24.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:03:24.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:03:24.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:03:24.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:03:24.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:03:24.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:03:24.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:03:24.174 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.174 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.174 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.174 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:24.174 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:03:24.175 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.765714008, 2023-07-02T14:03:24.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:03:24.175 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.175 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:24.175 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.175 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.175 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.176 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:03:24.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.176 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:03:24.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:03:24.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:24.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:03:24.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:03:24.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:03:24.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:03:24.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:03:24.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:03:24.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:03:24.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:03:24.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:03:24.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:03:24.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:24.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:03:24.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:03:24.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:03:24.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:03:24.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:03:24.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:03:24.182 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.182 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.182 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.183 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:24.183 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:03:24.183 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.758864124, 2023-07-02T14:03:24.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:03:24.183 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.184 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:24.184 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.184 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.184 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.184 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:03:24.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.185 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:03:24.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:03:24.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:03:24.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:03:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:03:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:03:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:03:24.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:03:24.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:03:24.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:03:24.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:03:24.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:03:24.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:03:24.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:03:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:03:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:03:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:03:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:03:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:03:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:03:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.758801199000001, 2023-07-02T14:03:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:03:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:03:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:03:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:03:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:03:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:03:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:03:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:03:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:03:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:03:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:03:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:03:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:03:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:03:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:03:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:03:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:03:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:03:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:03:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:03:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:03:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.758500778, 2023-07-02T14:03:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:03:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:03:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:03:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:03:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:03:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:03:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:03:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:03:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:03:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:03:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:03:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:03:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:03:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:03:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:03:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:03:24.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:03:24.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:03:24.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:03:24.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:03:24.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:03:24.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:03:24.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:03:24.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:03:24.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:03:24.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:03:24.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:03:24.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:03:24.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:03:24.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:03:24.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:03:24.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:03:24.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:03:24.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.213 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.213 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:03:24.214 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:24.214 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.757976277999999, 2023-07-02T14:03:24.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:03:24.214 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.214 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:03:24.214 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.214 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.214 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.215 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:03:24.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.215 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:24.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:24.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:24.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:24.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:03:24.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:03:24.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:03:24.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:03:24.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:03:24.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:03:24.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:03:24.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:03:24.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:03:24.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:24.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:03:24.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:03:24.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:03:24.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:03:24.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:03:24.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:03:24.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:03:24.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:03:24.222 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.222 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.222 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.223 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2871 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:24.223 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.382460+0000", 2023-07-02T14:03:24.223 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.706171449999999, 2023-07-02T14:03:24.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.062523707999999997, 2023-07-02T14:03:24.223 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.223 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.224 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.224 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.224 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.224 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2871 2023-07-02T14:03:24.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.224 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382460+0000", 2023-07-02T14:03:24.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382460+0000", 2023-07-02T14:03:24.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:24.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382460+0000", 2023-07-02T14:03:24.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3189999999999999e-06 2023-07-02T14:03:24.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382463+0000", 2023-07-02T14:03:24.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7400000000000004e-06 2023-07-02T14:03:24.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382471+0000", 2023-07-02T14:03:24.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1640000000000003e-06 2023-07-02T14:03:24.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382475+0000", 2023-07-02T14:03:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067762170000000002 2023-07-02T14:03:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.389251+0000", 2023-07-02T14:03:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3442000000000005e-05 2023-07-02T14:03:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.389335+0000", 2023-07-02T14:03:24.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028393860000000002 2023-07-02T14:03:24.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.392174+0000", 2023-07-02T14:03:24.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1433000000000002e-05 2023-07-02T14:03:24.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.392206+0000", 2023-07-02T14:03:24.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051865440000000004 2023-07-02T14:03:24.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.397392+0000", 2023-07-02T14:03:24.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.321e-05 2023-07-02T14:03:24.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.397455+0000", 2023-07-02T14:03:24.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012230776 2023-07-02T14:03:24.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.409686+0000", 2023-07-02T14:03:24.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8430999999999996e-05 2023-07-02T14:03:24.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.409754+0000", 2023-07-02T14:03:24.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052440610000000004 2023-07-02T14:03:24.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.414999+0000", 2023-07-02T14:03:24.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0063999999999999e-05 2023-07-02T14:03:24.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.415069+0000", 2023-07-02T14:03:24.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035077749999999999 2023-07-02T14:03:24.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.418576+0000", 2023-07-02T14:03:24.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0323999999999999e-05 2023-07-02T14:03:24.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.418657+0000", 2023-07-02T14:03:24.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018539512000000001 2023-07-02T14:03:24.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.437196+0000", 2023-07-02T14:03:24.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0704000000000004e-05 2023-07-02T14:03:24.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.437287+0000", 2023-07-02T14:03:24.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042943000000000001 2023-07-02T14:03:24.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:24.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.437716+0000", 2023-07-02T14:03:24.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016934700000000001 2023-07-02T14:03:24.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.439410+0000", 2023-07-02T14:03:24.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0053641770000000004 2023-07-02T14:03:24.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.444774+0000", 2023-07-02T14:03:24.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0989000000000003e-05 2023-07-02T14:03:24.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.444825+0000", 2023-07-02T14:03:24.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000158982 2023-07-02T14:03:24.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.444984+0000", 2023-07-02T14:03:24.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.062523707999999997 2023-07-02T14:03:24.241 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.241 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.241 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.242 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2872 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:24.242 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.382481+0000", 2023-07-02T14:03:24.242 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.706150849, 2023-07-02T14:03:24.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.065455442000000003, 2023-07-02T14:03:24.242 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.242 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.243 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.243 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.243 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.243 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2872 2023-07-02T14:03:24.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.243 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382481+0000", 2023-07-02T14:03:24.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382481+0000", 2023-07-02T14:03:24.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:24.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382480+0000", 2023-07-02T14:03:24.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2509999999999999e-06 2023-07-02T14:03:24.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382484+0000", 2023-07-02T14:03:24.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:03:24.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382490+0000", 2023-07-02T14:03:24.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3000000000000003e-06 2023-07-02T14:03:24.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382494+0000", 2023-07-02T14:03:24.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0069223369999999998 2023-07-02T14:03:24.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.389416+0000", 2023-07-02T14:03:24.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5201000000000001e-05 2023-07-02T14:03:24.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.389481+0000", 2023-07-02T14:03:24.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027921510000000001 2023-07-02T14:03:24.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.392274+0000", 2023-07-02T14:03:24.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6381000000000002e-05 2023-07-02T14:03:24.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.392320+0000", 2023-07-02T14:03:24.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051686450000000004 2023-07-02T14:03:24.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.397489+0000", 2023-07-02T14:03:24.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0806e-05 2023-07-02T14:03:24.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.397549+0000", 2023-07-02T14:03:24.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012259144 2023-07-02T14:03:24.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.409809+0000", 2023-07-02T14:03:24.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7465000000000005e-05 2023-07-02T14:03:24.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.409876+0000", 2023-07-02T14:03:24.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052799830000000002 2023-07-02T14:03:24.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.415156+0000", 2023-07-02T14:03:24.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2080000000000001e-05 2023-07-02T14:03:24.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.415228+0000", 2023-07-02T14:03:24.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034778159999999999 2023-07-02T14:03:24.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.418706+0000", 2023-07-02T14:03:24.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7431000000000001e-05 2023-07-02T14:03:24.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.418743+0000", 2023-07-02T14:03:24.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020264971999999999 2023-07-02T14:03:24.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.439008+0000", 2023-07-02T14:03:24.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2211000000000003e-05 2023-07-02T14:03:24.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.439090+0000", 2023-07-02T14:03:24.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060620709999999996 2023-07-02T14:03:24.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.445153+0000", 2023-07-02T14:03:24.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5259000000000006e-05 2023-07-02T14:03:24.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.445228+0000", 2023-07-02T14:03:24.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030277499999999997 2023-07-02T14:03:24.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:24.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.445531+0000", 2023-07-02T14:03:24.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013477540000000001 2023-07-02T14:03:24.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.446878+0000", 2023-07-02T14:03:24.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00088152200000000003 2023-07-02T14:03:24.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.447760+0000", 2023-07-02T14:03:24.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7722999999999998e-05 2023-07-02T14:03:24.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.447798+0000", 2023-07-02T14:03:24.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013871599999999999 2023-07-02T14:03:24.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.447936+0000", 2023-07-02T14:03:24.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.065455442000000003 2023-07-02T14:03:24.261 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.261 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.261 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.262 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3000 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:24.262 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.518762+0000", 2023-07-02T14:03:24.262 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.569869943, 2023-07-02T14:03:24.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.052957575, 2023-07-02T14:03:24.262 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.263 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.263 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.263 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.263 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.263 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3000 2023-07-02T14:03:24.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.263 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518762+0000", 2023-07-02T14:03:24.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518762+0000", 2023-07-02T14:03:24.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:24.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518761+0000", 2023-07-02T14:03:24.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3069999999999996e-06 2023-07-02T14:03:24.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518765+0000", 2023-07-02T14:03:24.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6170000000000001e-06 2023-07-02T14:03:24.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518773+0000", 2023-07-02T14:03:24.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.172e-06 2023-07-02T14:03:24.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518777+0000", 2023-07-02T14:03:24.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055680249999999999 2023-07-02T14:03:24.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.524345+0000", 2023-07-02T14:03:24.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0938000000000003e-05 2023-07-02T14:03:24.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.524426+0000", 2023-07-02T14:03:24.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091411249999999999 2023-07-02T14:03:24.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.533567+0000", 2023-07-02T14:03:24.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5704000000000002e-05 2023-07-02T14:03:24.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.533603+0000", 2023-07-02T14:03:24.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046842289999999998 2023-07-02T14:03:24.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.538287+0000", 2023-07-02T14:03:24.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5780999999999999e-05 2023-07-02T14:03:24.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.538353+0000", 2023-07-02T14:03:24.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034785279999999998 2023-07-02T14:03:24.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.541832+0000", 2023-07-02T14:03:24.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0024999999999997e-05 2023-07-02T14:03:24.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.541872+0000", 2023-07-02T14:03:24.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024189239999999998 2023-07-02T14:03:24.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.544291+0000", 2023-07-02T14:03:24.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9709e-05 2023-07-02T14:03:24.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.544320+0000", 2023-07-02T14:03:24.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037311990000000001 2023-07-02T14:03:24.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.548051+0000", 2023-07-02T14:03:24.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3347000000000002e-05 2023-07-02T14:03:24.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.548115+0000", 2023-07-02T14:03:24.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078550089999999996 2023-07-02T14:03:24.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.555970+0000", 2023-07-02T14:03:24.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8730000000000003e-05 2023-07-02T14:03:24.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.556019+0000", 2023-07-02T14:03:24.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052420460000000002 2023-07-02T14:03:24.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.561261+0000", 2023-07-02T14:03:24.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4456999999999995e-05 2023-07-02T14:03:24.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.561345+0000", 2023-07-02T14:03:24.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00047298300000000001 2023-07-02T14:03:24.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:24.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.561818+0000", 2023-07-02T14:03:24.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0098041989999999996 2023-07-02T14:03:24.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.571622+0000", 2023-07-02T14:03:24.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6109000000000003e-05 2023-07-02T14:03:24.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.571658+0000", 2023-07-02T14:03:24.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9556e-05 2023-07-02T14:03:24.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.571688+0000", 2023-07-02T14:03:24.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1454000000000001e-05 2023-07-02T14:03:24.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.571719+0000", 2023-07-02T14:03:24.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.052957575 2023-07-02T14:03:24.282 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.282 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.282 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.282 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3056 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2023-07-02T14:03:24.282 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:12.080431+0000", 2023-07-02T14:03:24.282 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.008201100000001, 2023-07-02T14:03:24.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051668310000000002, 2023-07-02T14:03:24.283 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.283 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.283 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.283 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.283 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.283 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3056 2023-07-02T14:03:24.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.284 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080431+0000", 2023-07-02T14:03:24.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080431+0000", 2023-07-02T14:03:24.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:24.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080430+0000", 2023-07-02T14:03:24.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:03:24.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080434+0000", 2023-07-02T14:03:24.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0520000000000004e-06 2023-07-02T14:03:24.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080441+0000", 2023-07-02T14:03:24.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3860000000000004e-06 2023-07-02T14:03:24.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080448+0000", 2023-07-02T14:03:24.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015622699999999999 2023-07-02T14:03:24.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080604+0000", 2023-07-02T14:03:24.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9212000000000003e-05 2023-07-02T14:03:24.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080643+0000", 2023-07-02T14:03:24.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064524869999999998 2023-07-02T14:03:24.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.087096+0000", 2023-07-02T14:03:24.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0238000000000003e-05 2023-07-02T14:03:24.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.087136+0000", 2023-07-02T14:03:24.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028586150000000001 2023-07-02T14:03:24.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.089994+0000", 2023-07-02T14:03:24.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1024999999999999e-05 2023-07-02T14:03:24.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090055+0000", 2023-07-02T14:03:24.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050689359999999996 2023-07-02T14:03:24.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095124+0000", 2023-07-02T14:03:24.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9534999999999998e-05 2023-07-02T14:03:24.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095184+0000", 2023-07-02T14:03:24.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043677159999999998 2023-07-02T14:03:24.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.099552+0000", 2023-07-02T14:03:24.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4551000000000001e-05 2023-07-02T14:03:24.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.099596+0000", 2023-07-02T14:03:24.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038310229999999998 2023-07-02T14:03:24.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103427+0000", 2023-07-02T14:03:24.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0380999999999997e-05 2023-07-02T14:03:24.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103488+0000", 2023-07-02T14:03:24.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056513550000000003 2023-07-02T14:03:24.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109139+0000", 2023-07-02T14:03:24.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1032999999999996e-05 2023-07-02T14:03:24.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109210+0000", 2023-07-02T14:03:24.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025622259999999999 2023-07-02T14:03:24.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.111772+0000", 2023-07-02T14:03:24.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5722000000000005e-05 2023-07-02T14:03:24.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.111838+0000", 2023-07-02T14:03:24.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047256870000000001 2023-07-02T14:03:24.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116564+0000", 2023-07-02T14:03:24.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8287000000000005e-05 2023-07-02T14:03:24.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116632+0000", 2023-07-02T14:03:24.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059677539999999996 2023-07-02T14:03:24.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122600+0000", 2023-07-02T14:03:24.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7799999999999995e-05 2023-07-02T14:03:24.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122667+0000", 2023-07-02T14:03:24.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033116230000000001 2023-07-02T14:03:24.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.125979+0000", 2023-07-02T14:03:24.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1271000000000002e-05 2023-07-02T14:03:24.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126060+0000", 2023-07-02T14:03:24.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019737790000000002 2023-07-02T14:03:24.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.128034+0000", 2023-07-02T14:03:24.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00054109199999999996 2023-07-02T14:03:24.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.128575+0000", 2023-07-02T14:03:24.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050589700000000005 2023-07-02T14:03:24.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:24.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.129081+0000", 2023-07-02T14:03:24.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018622420000000001 2023-07-02T14:03:24.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.130943+0000", 2023-07-02T14:03:24.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010268079999999999 2023-07-02T14:03:24.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.131970+0000", 2023-07-02T14:03:24.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3554000000000003e-05 2023-07-02T14:03:24.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.132024+0000", 2023-07-02T14:03:24.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5173999999999995e-05 2023-07-02T14:03:24.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.132099+0000", 2023-07-02T14:03:24.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051668310000000002 2023-07-02T14:03:24.307 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.307 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.307 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.307 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3057 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2023-07-02T14:03:24.307 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:12.080462+0000", 2023-07-02T14:03:24.307 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.008169547, 2023-07-02T14:03:24.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.055924880000000003, 2023-07-02T14:03:24.308 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.308 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.308 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.308 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.308 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.308 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3057 2023-07-02T14:03:24.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.309 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080462+0000", 2023-07-02T14:03:24.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080462+0000", 2023-07-02T14:03:24.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:24.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080462+0000", 2023-07-02T14:03:24.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6289999999999999e-06 2023-07-02T14:03:24.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080465+0000", 2023-07-02T14:03:24.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3088000000000001e-05 2023-07-02T14:03:24.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080478+0000", 2023-07-02T14:03:24.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.101e-06 2023-07-02T14:03:24.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080483+0000", 2023-07-02T14:03:24.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018486899999999999 2023-07-02T14:03:24.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080668+0000", 2023-07-02T14:03:24.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7156999999999997e-05 2023-07-02T14:03:24.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080706+0000", 2023-07-02T14:03:24.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093726839999999992 2023-07-02T14:03:24.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090078+0000", 2023-07-02T14:03:24.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3180999999999998e-05 2023-07-02T14:03:24.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090131+0000", 2023-07-02T14:03:24.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050952510000000003 2023-07-02T14:03:24.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095227+0000", 2023-07-02T14:03:24.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8848999999999998e-05 2023-07-02T14:03:24.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095285+0000", 2023-07-02T14:03:24.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043427029999999998 2023-07-02T14:03:24.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.099628+0000", 2023-07-02T14:03:24.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1265000000000004e-05 2023-07-02T14:03:24.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.099689+0000", 2023-07-02T14:03:24.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038415099999999998 2023-07-02T14:03:24.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103531+0000", 2023-07-02T14:03:24.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0980000000000001e-05 2023-07-02T14:03:24.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103602+0000", 2023-07-02T14:03:24.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056603110000000003 2023-07-02T14:03:24.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109262+0000", 2023-07-02T14:03:24.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3421000000000001e-05 2023-07-02T14:03:24.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109326+0000", 2023-07-02T14:03:24.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025528510000000001 2023-07-02T14:03:24.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.111879+0000", 2023-07-02T14:03:24.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3395999999999999e-05 2023-07-02T14:03:24.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.111962+0000", 2023-07-02T14:03:24.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047126889999999999 2023-07-02T14:03:24.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116675+0000", 2023-07-02T14:03:24.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7849000000000001e-05 2023-07-02T14:03:24.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116732+0000", 2023-07-02T14:03:24.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059724649999999997 2023-07-02T14:03:24.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122705+0000", 2023-07-02T14:03:24.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3373999999999993e-05 2023-07-02T14:03:24.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122768+0000", 2023-07-02T14:03:24.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033541339999999999 2023-07-02T14:03:24.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126122+0000", 2023-07-02T14:03:24.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2749000000000003e-05 2023-07-02T14:03:24.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126215+0000", 2023-07-02T14:03:24.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036440750000000001 2023-07-02T14:03:24.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.129859+0000", 2023-07-02T14:03:24.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000103273 2023-07-02T14:03:24.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.129963+0000", 2023-07-02T14:03:24.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021629449999999999 2023-07-02T14:03:24.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.132125+0000", 2023-07-02T14:03:24.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8305e-05 2023-07-02T14:03:24.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.132164+0000", 2023-07-02T14:03:24.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00046304499999999998 2023-07-02T14:03:24.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:24.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.132627+0000", 2023-07-02T14:03:24.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022532979999999999 2023-07-02T14:03:24.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.134880+0000", 2023-07-02T14:03:24.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013186859999999999 2023-07-02T14:03:24.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.136199+0000", 2023-07-02T14:03:24.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1561000000000002e-05 2023-07-02T14:03:24.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.136260+0000", 2023-07-02T14:03:24.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126685 2023-07-02T14:03:24.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.136387+0000", 2023-07-02T14:03:24.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.055924880000000003 2023-07-02T14:03:24.331 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.332 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.332 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.332 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3058 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2023-07-02T14:03:24.332 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:12.080489+0000", 2023-07-02T14:03:24.333 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.008143018, 2023-07-02T14:03:24.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.059911803, 2023-07-02T14:03:24.333 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.333 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.333 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.333 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.333 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.333 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3058 2023-07-02T14:03:24.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.334 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080489+0000", 2023-07-02T14:03:24.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080489+0000", 2023-07-02T14:03:24.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:24.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080488+0000", 2023-07-02T14:03:24.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6950000000000002e-06 2023-07-02T14:03:24.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080492+0000", 2023-07-02T14:03:24.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7950000000000004e-06 2023-07-02T14:03:24.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080498+0000", 2023-07-02T14:03:24.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1189999999999999e-06 2023-07-02T14:03:24.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080502+0000", 2023-07-02T14:03:24.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000227817 2023-07-02T14:03:24.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080730+0000", 2023-07-02T14:03:24.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0629e-05 2023-07-02T14:03:24.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080770+0000", 2023-07-02T14:03:24.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093982159999999992 2023-07-02T14:03:24.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090168+0000", 2023-07-02T14:03:24.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5840000000000007e-05 2023-07-02T14:03:24.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090234+0000", 2023-07-02T14:03:24.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050953109999999999 2023-07-02T14:03:24.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095330+0000", 2023-07-02T14:03:24.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104215 2023-07-02T14:03:24.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095434+0000", 2023-07-02T14:03:24.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043199060000000001 2023-07-02T14:03:24.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.099754+0000", 2023-07-02T14:03:24.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000336504 2023-07-02T14:03:24.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.100090+0000", 2023-07-02T14:03:24.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035759479999999998 2023-07-02T14:03:24.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103666+0000", 2023-07-02T14:03:24.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3940000000000001e-05 2023-07-02T14:03:24.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103730+0000", 2023-07-02T14:03:24.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005666333 2023-07-02T14:03:24.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109396+0000", 2023-07-02T14:03:24.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2536999999999998e-05 2023-07-02T14:03:24.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109489+0000", 2023-07-02T14:03:24.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025086790000000002 2023-07-02T14:03:24.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.111998+0000", 2023-07-02T14:03:24.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7811000000000005e-05 2023-07-02T14:03:24.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.112065+0000", 2023-07-02T14:03:24.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046994519999999998 2023-07-02T14:03:24.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116765+0000", 2023-07-02T14:03:24.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1928000000000002e-05 2023-07-02T14:03:24.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116817+0000", 2023-07-02T14:03:24.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005988345 2023-07-02T14:03:24.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122805+0000", 2023-07-02T14:03:24.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7315999999999998e-05 2023-07-02T14:03:24.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122862+0000", 2023-07-02T14:03:24.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034274489999999999 2023-07-02T14:03:24.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126290+0000", 2023-07-02T14:03:24.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8703999999999998e-05 2023-07-02T14:03:24.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126359+0000", 2023-07-02T14:03:24.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003699889 2023-07-02T14:03:24.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.130058+0000", 2023-07-02T14:03:24.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110308 2023-07-02T14:03:24.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.130169+0000", 2023-07-02T14:03:24.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003130295 2023-07-02T14:03:24.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.133299+0000", 2023-07-02T14:03:24.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6588999999999995e-05 2023-07-02T14:03:24.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.133376+0000", 2023-07-02T14:03:24.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030628389999999999 2023-07-02T14:03:24.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.136439+0000", 2023-07-02T14:03:24.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010477 2023-07-02T14:03:24.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.136543+0000", 2023-07-02T14:03:24.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00047791199999999999 2023-07-02T14:03:24.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:24.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.137021+0000", 2023-07-02T14:03:24.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00150596 2023-07-02T14:03:24.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.138527+0000", 2023-07-02T14:03:24.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017228510000000001 2023-07-02T14:03:24.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140250+0000", 2023-07-02T14:03:24.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8615000000000006e-05 2023-07-02T14:03:24.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140319+0000", 2023-07-02T14:03:24.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1852000000000002e-05 2023-07-02T14:03:24.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140400+0000", 2023-07-02T14:03:24.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.059911803 2023-07-02T14:03:24.358 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.358 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.359 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.359 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3059 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2023-07-02T14:03:24.359 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:12.080509+0000", 2023-07-02T14:03:24.359 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.008122481999999, 2023-07-02T14:03:24.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.062340277999999999, 2023-07-02T14:03:24.359 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.360 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.360 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.360 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.360 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.360 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3059 2023-07-02T14:03:24.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.360 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080509+0000", 2023-07-02T14:03:24.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080509+0000", 2023-07-02T14:03:24.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:24.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080509+0000", 2023-07-02T14:03:24.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5099999999999999e-06 2023-07-02T14:03:24.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080512+0000", 2023-07-02T14:03:24.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8880000000000002e-06 2023-07-02T14:03:24.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080518+0000", 2023-07-02T14:03:24.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1520000000000002e-06 2023-07-02T14:03:24.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080522+0000", 2023-07-02T14:03:24.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026573699999999999 2023-07-02T14:03:24.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080788+0000", 2023-07-02T14:03:24.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7478999999999998e-05 2023-07-02T14:03:24.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080826+0000", 2023-07-02T14:03:24.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0094689049999999997 2023-07-02T14:03:24.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090294+0000", 2023-07-02T14:03:24.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5257000000000001e-05 2023-07-02T14:03:24.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090370+0000", 2023-07-02T14:03:24.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005117229 2023-07-02T14:03:24.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095487+0000", 2023-07-02T14:03:24.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7698e-05 2023-07-02T14:03:24.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095575+0000", 2023-07-02T14:03:24.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045423210000000002 2023-07-02T14:03:24.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.100117+0000", 2023-07-02T14:03:24.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1317999999999999e-05 2023-07-02T14:03:24.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.100168+0000", 2023-07-02T14:03:24.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035847650000000002 2023-07-02T14:03:24.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103753+0000", 2023-07-02T14:03:24.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3754999999999998e-05 2023-07-02T14:03:24.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103787+0000", 2023-07-02T14:03:24.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0057476209999999996 2023-07-02T14:03:24.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109534+0000", 2023-07-02T14:03:24.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9867000000000001e-05 2023-07-02T14:03:24.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109594+0000", 2023-07-02T14:03:24.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024983840000000002 2023-07-02T14:03:24.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.112093+0000", 2023-07-02T14:03:24.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0919e-05 2023-07-02T14:03:24.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.112124+0000", 2023-07-02T14:03:24.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047877880000000003 2023-07-02T14:03:24.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116911+0000", 2023-07-02T14:03:24.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1793999999999996e-05 2023-07-02T14:03:24.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.117003+0000", 2023-07-02T14:03:24.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059009860000000004 2023-07-02T14:03:24.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122904+0000", 2023-07-02T14:03:24.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8408000000000001e-05 2023-07-02T14:03:24.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122953+0000", 2023-07-02T14:03:24.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034502040000000001 2023-07-02T14:03:24.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126403+0000", 2023-07-02T14:03:24.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4770000000000005e-05 2023-07-02T14:03:24.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126468+0000", 2023-07-02T14:03:24.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037365950000000001 2023-07-02T14:03:24.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.130204+0000", 2023-07-02T14:03:24.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7163000000000002e-05 2023-07-02T14:03:24.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.130251+0000", 2023-07-02T14:03:24.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031607509999999998 2023-07-02T14:03:24.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.133412+0000", 2023-07-02T14:03:24.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1032000000000005e-05 2023-07-02T14:03:24.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.133503+0000", 2023-07-02T14:03:24.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041497349999999999 2023-07-02T14:03:24.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.137653+0000", 2023-07-02T14:03:24.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7430999999999997e-05 2023-07-02T14:03:24.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:24.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.137740+0000", 2023-07-02T14:03:24.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026871349999999998 2023-07-02T14:03:24.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140427+0000", 2023-07-02T14:03:24.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7179999999999998e-05 2023-07-02T14:03:24.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140485+0000", 2023-07-02T14:03:24.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00039492900000000003 2023-07-02T14:03:24.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:24.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140879+0000", 2023-07-02T14:03:24.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018682359999999999 2023-07-02T14:03:24.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.142748+0000", 2023-07-02T14:03:24.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4937e-05 2023-07-02T14:03:24.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.142793+0000", 2023-07-02T14:03:24.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4743999999999998e-05 2023-07-02T14:03:24.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.142827+0000", 2023-07-02T14:03:24.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2118999999999999e-05 2023-07-02T14:03:24.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.142849+0000", 2023-07-02T14:03:24.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.062340277999999999 2023-07-02T14:03:24.387 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.387 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.387 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.387 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.387 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.387 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:24.388 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:03:24.388 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 1 2023-07-02T14:03:24.389 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:03:24.390 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:24.389+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:24.490 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:24.490+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:24.555 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:24.555 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:24.555 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:24.555 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:24.565 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:03:24.590 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:24.591+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:24.689 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:24.689 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:24.689 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:24.690 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:24.690 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:24.691 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:24.691+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:24.699 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:03:24.792 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:24.792+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:24.825 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:24.825 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:03:24.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:03:24.825 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:24.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.825 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:183 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:24.826 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:24.826 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.537527715000003, 2023-07-02T14:03:24.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003, 2023-07-02T14:03:24.826 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.826 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.826 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.826 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.827 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.827 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 183 2023-07-02T14:03:24.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.827 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:24.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:24.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:24.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:24.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9790999999999997e-05 2023-07-02T14:03:24.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285598+0000", 2023-07-02T14:03:24.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5809999999999996e-06 2023-07-02T14:03:24.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285604+0000", 2023-07-02T14:03:24.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7599999999999998e-06 2023-07-02T14:03:24.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285607+0000", 2023-07-02T14:03:24.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021108188999999999 2023-07-02T14:03:24.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306715+0000", 2023-07-02T14:03:24.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6548e-05 2023-07-02T14:03:24.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306792+0000", 2023-07-02T14:03:24.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173869 2023-07-02T14:03:24.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:24.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306965+0000", 2023-07-02T14:03:24.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060498009999999996 2023-07-02T14:03:24.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313015+0000", 2023-07-02T14:03:24.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049443719999999998 2023-07-02T14:03:24.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317960+0000", 2023-07-02T14:03:24.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5927999999999998e-05 2023-07-02T14:03:24.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318006+0000", 2023-07-02T14:03:24.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6004999999999998e-05 2023-07-02T14:03:24.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318042+0000", 2023-07-02T14:03:24.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003 2023-07-02T14:03:24.835 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.835 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.835 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.836 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:186 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:24.836 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:24.836 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.537456888999998, 2023-07-02T14:03:24.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003, 2023-07-02T14:03:24.836 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.837 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.837 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.837 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.837 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.837 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 186 2023-07-02T14:03:24.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.837 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:24.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:24.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:24.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:24.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4279e-05 2023-07-02T14:03:24.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285664+0000", 2023-07-02T14:03:24.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3789999999999995e-06 2023-07-02T14:03:24.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285673+0000", 2023-07-02T14:03:24.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499999999999998e-06 2023-07-02T14:03:24.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285678+0000", 2023-07-02T14:03:24.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023261947000000002 2023-07-02T14:03:24.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.308940+0000", 2023-07-02T14:03:24.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010323399999999999 2023-07-02T14:03:24.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309044+0000", 2023-07-02T14:03:24.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018878300000000001 2023-07-02T14:03:24.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:24.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309232+0000", 2023-07-02T14:03:24.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037874150000000001 2023-07-02T14:03:24.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313020+0000", 2023-07-02T14:03:24.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050976140000000003 2023-07-02T14:03:24.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318117+0000", 2023-07-02T14:03:24.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3182000000000001e-05 2023-07-02T14:03:24.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:24.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5055999999999998e-05 2023-07-02T14:03:24.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318196+0000", 2023-07-02T14:03:24.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003 2023-07-02T14:03:24.846 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.846 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.846 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.846 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:189 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:24.846 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:03:24.846 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.537383126000002, 2023-07-02T14:03:24.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997, 2023-07-02T14:03:24.847 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.847 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.847 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.847 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.847 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.847 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:03:24.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.848 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:03:24.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:03:24.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:24.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285683+0000", 2023-07-02T14:03:24.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8451999999999999e-05 2023-07-02T14:03:24.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285782+0000", 2023-07-02T14:03:24.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3650000000000004e-06 2023-07-02T14:03:24.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285790+0000", 2023-07-02T14:03:24.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8230000000000002e-06 2023-07-02T14:03:24.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285795+0000", 2023-07-02T14:03:24.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024619367999999999 2023-07-02T14:03:24.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310414+0000", 2023-07-02T14:03:24.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014244600000000001 2023-07-02T14:03:24.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310557+0000", 2023-07-02T14:03:24.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020612699999999999 2023-07-02T14:03:24.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:24.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310763+0000", 2023-07-02T14:03:24.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003427687 2023-07-02T14:03:24.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314191+0000", 2023-07-02T14:03:24.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040492089999999998 2023-07-02T14:03:24.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318240+0000", 2023-07-02T14:03:24.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0287999999999999e-05 2023-07-02T14:03:24.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318270+0000", 2023-07-02T14:03:24.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0868000000000002e-05 2023-07-02T14:03:24.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318311+0000", 2023-07-02T14:03:24.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997 2023-07-02T14:03:24.856 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.856 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.856 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.857 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:192 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:24.857 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:03:24.857 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.537266353, 2023-07-02T14:03:24.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003, 2023-07-02T14:03:24.857 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.857 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.857 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.857 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 192 2023-07-02T14:03:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:03:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:03:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285800+0000", 2023-07-02T14:03:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100417 2023-07-02T14:03:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285900+0000", 2023-07-02T14:03:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8244999999999999e-05 2023-07-02T14:03:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285919+0000", 2023-07-02T14:03:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2499999999999997e-06 2023-07-02T14:03:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285924+0000", 2023-07-02T14:03:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025288155999999999 2023-07-02T14:03:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311212+0000", 2023-07-02T14:03:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4073999999999996e-05 2023-07-02T14:03:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311286+0000", 2023-07-02T14:03:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016219699999999999 2023-07-02T14:03:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311448+0000", 2023-07-02T14:03:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066020599999999999 2023-07-02T14:03:24.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318050+0000", 2023-07-02T14:03:24.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000327986 2023-07-02T14:03:24.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318378+0000", 2023-07-02T14:03:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8693999999999998e-05 2023-07-02T14:03:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318427+0000", 2023-07-02T14:03:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6656999999999998e-05 2023-07-02T14:03:24.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318464+0000", 2023-07-02T14:03:24.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003 2023-07-02T14:03:24.866 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.866 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.866 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.867 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:195 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:24.867 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:03:24.867 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.537134848999997, 2023-07-02T14:03:24.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001, 2023-07-02T14:03:24.867 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.867 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 195 2023-07-02T14:03:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:03:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:03:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285931+0000", 2023-07-02T14:03:24.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4530999999999995e-05 2023-07-02T14:03:24.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286006+0000", 2023-07-02T14:03:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4113e-05 2023-07-02T14:03:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286020+0000", 2023-07-02T14:03:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7459999999999997e-06 2023-07-02T14:03:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286026+0000", 2023-07-02T14:03:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025997769 2023-07-02T14:03:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312023+0000", 2023-07-02T14:03:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4403000000000001e-05 2023-07-02T14:03:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312078+0000", 2023-07-02T14:03:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020581399999999999 2023-07-02T14:03:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312284+0000", 2023-07-02T14:03:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005771043 2023-07-02T14:03:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318055+0000", 2023-07-02T14:03:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00046015200000000002 2023-07-02T14:03:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318515+0000", 2023-07-02T14:03:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9444000000000003e-05 2023-07-02T14:03:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318564+0000", 2023-07-02T14:03:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2141000000000002e-05 2023-07-02T14:03:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318606+0000", 2023-07-02T14:03:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001 2023-07-02T14:03:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:198 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:03:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.537034890000001, 2023-07-02T14:03:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999, 2023-07-02T14:03:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 198 2023-07-02T14:03:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:03:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:03:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286031+0000", 2023-07-02T14:03:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00049286700000000002 2023-07-02T14:03:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286524+0000", 2023-07-02T14:03:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4938e-05 2023-07-02T14:03:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286539+0000", 2023-07-02T14:03:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0680000000000002e-06 2023-07-02T14:03:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286545+0000", 2023-07-02T14:03:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026497718 2023-07-02T14:03:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313043+0000", 2023-07-02T14:03:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0884999999999998e-05 2023-07-02T14:03:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313094+0000", 2023-07-02T14:03:24.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014109999999999999 2023-07-02T14:03:24.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:24.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313235+0000", 2023-07-02T14:03:24.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048258980000000003 2023-07-02T14:03:24.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318061+0000", 2023-07-02T14:03:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014346350000000001 2023-07-02T14:03:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319495+0000", 2023-07-02T14:03:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1986e-05 2023-07-02T14:03:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319547+0000", 2023-07-02T14:03:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0457e-05 2023-07-02T14:03:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319578+0000", 2023-07-02T14:03:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999 2023-07-02T14:03:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:201 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.536514465000003, 2023-07-02T14:03:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003, 2023-07-02T14:03:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 201 2023-07-02T14:03:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:24.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:24.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019213999999999999 2023-07-02T14:03:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286744+0000", 2023-07-02T14:03:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5279999999999992e-06 2023-07-02T14:03:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286753+0000", 2023-07-02T14:03:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4200000000000004e-06 2023-07-02T14:03:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286760+0000", 2023-07-02T14:03:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026818386999999999 2023-07-02T14:03:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313578+0000", 2023-07-02T14:03:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015965 2023-07-02T14:03:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313738+0000", 2023-07-02T14:03:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015767299999999999 2023-07-02T14:03:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313895+0000", 2023-07-02T14:03:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041714930000000001 2023-07-02T14:03:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318067+0000", 2023-07-02T14:03:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001561698 2023-07-02T14:03:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319629+0000", 2023-07-02T14:03:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2346000000000003e-05 2023-07-02T14:03:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319681+0000", 2023-07-02T14:03:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4691000000000003e-05 2023-07-02T14:03:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319726+0000", 2023-07-02T14:03:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003 2023-07-02T14:03:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:204 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.536299626999998, 2023-07-02T14:03:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998, 2023-07-02T14:03:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 204 2023-07-02T14:03:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2497999999999998e-05 2023-07-02T14:03:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286819+0000", 2023-07-02T14:03:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0099999999999995e-06 2023-07-02T14:03:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286827+0000", 2023-07-02T14:03:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000003e-06 2023-07-02T14:03:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286833+0000", 2023-07-02T14:03:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027384716999999999 2023-07-02T14:03:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314218+0000", 2023-07-02T14:03:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6204000000000004e-05 2023-07-02T14:03:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314294+0000", 2023-07-02T14:03:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015757499999999999 2023-07-02T14:03:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314452+0000", 2023-07-02T14:03:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036215560000000002 2023-07-02T14:03:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318073+0000", 2023-07-02T14:03:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017023979999999999 2023-07-02T14:03:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319776+0000", 2023-07-02T14:03:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9561000000000002e-05 2023-07-02T14:03:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319825+0000", 2023-07-02T14:03:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0329e-05 2023-07-02T14:03:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319866+0000", 2023-07-02T14:03:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998 2023-07-02T14:03:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:207 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:03:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.536224982, 2023-07-02T14:03:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003, 2023-07-02T14:03:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 207 2023-07-02T14:03:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:03:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:03:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286841+0000", 2023-07-02T14:03:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00068375100000000002 2023-07-02T14:03:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287525+0000", 2023-07-02T14:03:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1601e-05 2023-07-02T14:03:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287537+0000", 2023-07-02T14:03:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6409999999999996e-06 2023-07-02T14:03:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287543+0000", 2023-07-02T14:03:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027245844000000002 2023-07-02T14:03:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314789+0000", 2023-07-02T14:03:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8910000000000003e-05 2023-07-02T14:03:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314858+0000", 2023-07-02T14:03:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030992599999999997 2023-07-02T14:03:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315168+0000", 2023-07-02T14:03:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029122869999999999 2023-07-02T14:03:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318080+0000", 2023-07-02T14:03:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019963020000000001 2023-07-02T14:03:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320077+0000", 2023-07-02T14:03:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5116000000000005e-05 2023-07-02T14:03:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320142+0000", 2023-07-02T14:03:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3793000000000004e-05 2023-07-02T14:03:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320205+0000", 2023-07-02T14:03:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003 2023-07-02T14:03:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:236 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2424832~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:03:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.535512799000003, 2023-07-02T14:03:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003, 2023-07-02T14:03:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2023-07-02T14:03:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:03:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:03:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287553+0000", 2023-07-02T14:03:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0667e-05 2023-07-02T14:03:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287614+0000", 2023-07-02T14:03:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0454e-05 2023-07-02T14:03:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287624+0000", 2023-07-02T14:03:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8564999999999999e-05 2023-07-02T14:03:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287643+0000", 2023-07-02T14:03:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027977596 2023-07-02T14:03:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315621+0000", 2023-07-02T14:03:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000201728 2023-07-02T14:03:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315822+0000", 2023-07-02T14:03:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018436599999999999 2023-07-02T14:03:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316007+0000", 2023-07-02T14:03:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042423440000000003 2023-07-02T14:03:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320249+0000", 2023-07-02T14:03:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001452633 2023-07-02T14:03:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321702+0000", 2023-07-02T14:03:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3616000000000003e-05 2023-07-02T14:03:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321755+0000", 2023-07-02T14:03:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840999999999999e-05 2023-07-02T14:03:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321793+0000", 2023-07-02T14:03:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003 2023-07-02T14:03:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.929 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:251 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:24.929 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:03:24.929 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.535417262000003, 2023-07-02T14:03:24.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001, 2023-07-02T14:03:24.929 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.929 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.930 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.930 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.930 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.930 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 251 2023-07-02T14:03:24.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.930 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:03:24.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:03:24.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:24.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287649+0000", 2023-07-02T14:03:24.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3232000000000001e-05 2023-07-02T14:03:24.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287702+0000", 2023-07-02T14:03:24.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0010000000000007e-06 2023-07-02T14:03:24.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287711+0000", 2023-07-02T14:03:24.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3569999999999997e-06 2023-07-02T14:03:24.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287717+0000", 2023-07-02T14:03:24.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028686454 2023-07-02T14:03:24.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316403+0000", 2023-07-02T14:03:24.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7741000000000002e-05 2023-07-02T14:03:24.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316461+0000", 2023-07-02T14:03:24.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4313000000000004e-05 2023-07-02T14:03:24.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:24.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316545+0000", 2023-07-02T14:03:24.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003734032 2023-07-02T14:03:24.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320279+0000", 2023-07-02T14:03:24.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015237219999999999 2023-07-02T14:03:24.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321803+0000", 2023-07-02T14:03:24.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1927e-05 2023-07-02T14:03:24.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321845+0000", 2023-07-02T14:03:24.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5398999999999998e-05 2023-07-02T14:03:24.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321880+0000", 2023-07-02T14:03:24.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001 2023-07-02T14:03:24.938 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.939 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.939 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.939 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:255 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2555904~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:24.939 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:03:24.939 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.535002534, 2023-07-02T14:03:24.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002, 2023-07-02T14:03:24.940 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.940 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.940 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.940 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.940 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.940 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 255 2023-07-02T14:03:24.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.940 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:03:24.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:03:24.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:24.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288063+0000", 2023-07-02T14:03:24.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4459999999999997e-05 2023-07-02T14:03:24.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288118+0000", 2023-07-02T14:03:24.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0998999999999999e-05 2023-07-02T14:03:24.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288129+0000", 2023-07-02T14:03:24.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3219999999999998e-06 2023-07-02T14:03:24.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288135+0000", 2023-07-02T14:03:24.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029137386000000001 2023-07-02T14:03:24.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317273+0000", 2023-07-02T14:03:24.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7819999999999997e-05 2023-07-02T14:03:24.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317350+0000", 2023-07-02T14:03:24.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000174864 2023-07-02T14:03:24.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:24.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317525+0000", 2023-07-02T14:03:24.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027943519999999999 2023-07-02T14:03:24.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320320+0000", 2023-07-02T14:03:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015681460000000001 2023-07-02T14:03:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321888+0000", 2023-07-02T14:03:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9508999999999997e-05 2023-07-02T14:03:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321927+0000", 2023-07-02T14:03:24.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5428e-05 2023-07-02T14:03:24.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321963+0000", 2023-07-02T14:03:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002 2023-07-02T14:03:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:446 2.1 2:910b2a90:::rbd_data.125de1449605.000000000000010a:head [write 65536~3211264] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.457722781999998, 2023-07-02T14:03:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003, 2023-07-02T14:03:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 446 2023-07-02T14:03:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:24.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015907372999999999 2023-07-02T14:03:24.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381251+0000", 2023-07-02T14:03:24.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5527000000000001e-05 2023-07-02T14:03:24.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381277+0000", 2023-07-02T14:03:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9958000000000001e-05 2023-07-02T14:03:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381297+0000", 2023-07-02T14:03:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4941000000000001e-05 2023-07-02T14:03:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381352+0000", 2023-07-02T14:03:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011565699999999999 2023-07-02T14:03:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381467+0000", 2023-07-02T14:03:24.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018039799999999999 2023-07-02T14:03:24.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:24.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381648+0000", 2023-07-02T14:03:24.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088183559999999994 2023-07-02T14:03:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.390466+0000", 2023-07-02T14:03:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007278278 2023-07-02T14:03:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397744+0000", 2023-07-02T14:03:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9289000000000001e-05 2023-07-02T14:03:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397804+0000", 2023-07-02T14:03:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0479000000000002e-05 2023-07-02T14:03:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397854+0000", 2023-07-02T14:03:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003 2023-07-02T14:03:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.960 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:03:24.960 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:03:24.960 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.715391988, 2023-07-02T14:03:24.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:03:24.960 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.960 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:03:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:03:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:03:24.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:03:24.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:24.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:03:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:03:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:03:24.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:03:24.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:03:24.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:03:24.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:03:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:03:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:03:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:03:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:03:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:03:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:03:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:03:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:03:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:03:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:03:24.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:03:24.968 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.968 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.968 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.969 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:03:24.969 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:03:24.969 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.715331249999998, 2023-07-02T14:03:24.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:03:24.969 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.969 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.970 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.970 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.970 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.970 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:03:24.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.970 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:03:24.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:03:24.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:24.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:03:24.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:03:24.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:03:24.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:03:24.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:03:24.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:03:24.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:03:24.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:03:24.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:03:24.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:03:24.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:24.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:03:24.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:03:24.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:03:24.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:03:24.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:03:24.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:03:24.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:24.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:03:24.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:03:24.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:03:24.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:03:24.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:03:24.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:03:24.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:03:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:03:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:03:24.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:03:24.980 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.980 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.980 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.980 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:24.980 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:03:24.981 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.48487692, 2023-07-02T14:03:24.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:03:24.981 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.981 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.981 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.981 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.981 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.981 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:03:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:03:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:03:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:03:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:03:24.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:03:24.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:03:24.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:03:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:03:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:03:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:03:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:03:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:03:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:03:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:03:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:03:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:03:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:03:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:03:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:03:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:03:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:03:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:03:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:03:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:03:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2941 2.3 2:d3762e63:::rbd_data.125de1449605.00000000000000f8:head [write 65536~2031616] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:03:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.355773965999999, 2023-07-02T14:03:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002, 2023-07-02T14:03:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:24.992 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:24.992 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2941 2023-07-02T14:03:24.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.992 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:24.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:24.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:03:24.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:24.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:24.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:03:24.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:24.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:24.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467291+0000", 2023-07-02T14:03:24.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043022859999999998 2023-07-02T14:03:24.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:24.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471593+0000", 2023-07-02T14:03:24.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0999999999999999e-05 2023-07-02T14:03:24.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:24.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471614+0000", 2023-07-02T14:03:24.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0097e-05 2023-07-02T14:03:24.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:24.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471625+0000", 2023-07-02T14:03:24.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069808499999999998 2023-07-02T14:03:24.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:24.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472323+0000", 2023-07-02T14:03:24.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135635 2023-07-02T14:03:24.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472458+0000", 2023-07-02T14:03:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223365 2023-07-02T14:03:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472682+0000", 2023-07-02T14:03:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013761944999999999 2023-07-02T14:03:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.486444+0000", 2023-07-02T14:03:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016859747000000001 2023-07-02T14:03:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503303+0000", 2023-07-02T14:03:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6635000000000001e-05 2023-07-02T14:03:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503380+0000", 2023-07-02T14:03:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000161803 2023-07-02T14:03:25.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:25.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503542+0000", 2023-07-02T14:03:25.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002 2023-07-02T14:03:25.000 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.000 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.000 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.001 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2950 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 1507328~786432] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:25.001 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:03:25.001 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.347600657999999, 2023-07-02T14:03:25.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999, 2023-07-02T14:03:25.001 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:25.001 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.002 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:25.002 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:25.002 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.002 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2950 2023-07-02T14:03:25.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.002 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:25.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:25.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:03:25.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:25.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:25.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:03:25.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:25.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:25.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:03:25.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001147854 2023-07-02T14:03:25.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:25.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476614+0000", 2023-07-02T14:03:25.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.802e-05 2023-07-02T14:03:25.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:25.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476632+0000", 2023-07-02T14:03:25.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1860000000000003e-06 2023-07-02T14:03:25.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476639+0000", 2023-07-02T14:03:25.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061211240000000004 2023-07-02T14:03:25.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:25.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482760+0000", 2023-07-02T14:03:25.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8833999999999996e-05 2023-07-02T14:03:25.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:25.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482859+0000", 2023-07-02T14:03:25.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019270600000000001 2023-07-02T14:03:25.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:25.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.483051+0000", 2023-07-02T14:03:25.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060385630000000003 2023-07-02T14:03:25.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:25.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489090+0000", 2023-07-02T14:03:25.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024506918999999999 2023-07-02T14:03:25.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513597+0000", 2023-07-02T14:03:25.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9672999999999995e-05 2023-07-02T14:03:25.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:25.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513667+0000", 2023-07-02T14:03:25.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024329500000000001 2023-07-02T14:03:25.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:25.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513910+0000", 2023-07-02T14:03:25.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999 2023-07-02T14:03:25.011 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.011 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.011 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.011 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2951 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2686976~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:25.011 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:03:25.011 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.346419879999999, 2023-07-02T14:03:25.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999, 2023-07-02T14:03:25.012 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:25.012 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.012 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:25.012 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:25.012 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.012 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2951 2023-07-02T14:03:25.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.013 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:25.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:25.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:03:25.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:25.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:25.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:03:25.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:25.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:25.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476646+0000", 2023-07-02T14:03:25.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7639000000000003e-05 2023-07-02T14:03:25.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:25.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476684+0000", 2023-07-02T14:03:25.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3508000000000002e-05 2023-07-02T14:03:25.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:25.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476747+0000", 2023-07-02T14:03:25.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8499999999999994e-06 2023-07-02T14:03:25.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476755+0000", 2023-07-02T14:03:25.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078346160000000008 2023-07-02T14:03:25.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:25.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484590+0000", 2023-07-02T14:03:25.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7206000000000004e-05 2023-07-02T14:03:25.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:25.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484687+0000", 2023-07-02T14:03:25.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016243099999999999 2023-07-02T14:03:25.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:25.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484849+0000", 2023-07-02T14:03:25.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042482860000000004 2023-07-02T14:03:25.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:25.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489097+0000", 2023-07-02T14:03:25.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.02489504 2023-07-02T14:03:25.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513992+0000", 2023-07-02T14:03:25.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9972000000000001e-05 2023-07-02T14:03:25.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:25.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514052+0000", 2023-07-02T14:03:25.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149264 2023-07-02T14:03:25.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:25.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514202+0000", 2023-07-02T14:03:25.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999 2023-07-02T14:03:25.021 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.021 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.021 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.022 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2952 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2883584~12288] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:25.022 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:03:25.022 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.346306045, 2023-07-02T14:03:25.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651, 2023-07-02T14:03:25.022 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:25.022 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.022 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:25.022 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:25.023 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.023 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2952 2023-07-02T14:03:25.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.023 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:25.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:25.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:03:25.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:25.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:25.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:03:25.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:25.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:25.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476760+0000", 2023-07-02T14:03:25.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3664e-05 2023-07-02T14:03:25.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:25.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476774+0000", 2023-07-02T14:03:25.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.166e-06 2023-07-02T14:03:25.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:25.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476783+0000", 2023-07-02T14:03:25.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8720000000000001e-06 2023-07-02T14:03:25.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476788+0000", 2023-07-02T14:03:25.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087180529999999999 2023-07-02T14:03:25.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:25.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485506+0000", 2023-07-02T14:03:25.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5680999999999994e-05 2023-07-02T14:03:25.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:25.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485602+0000", 2023-07-02T14:03:25.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000222365 2023-07-02T14:03:25.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:25.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485824+0000", 2023-07-02T14:03:25.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003277379 2023-07-02T14:03:25.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:25.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489101+0000", 2023-07-02T14:03:25.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025151739999999999 2023-07-02T14:03:25.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514253+0000", 2023-07-02T14:03:25.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4749000000000002e-05 2023-07-02T14:03:25.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:25.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514288+0000", 2023-07-02T14:03:25.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130634 2023-07-02T14:03:25.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:25.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514419+0000", 2023-07-02T14:03:25.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651 2023-07-02T14:03:25.031 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.031 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.031 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.032 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.032 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.032 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:25.033 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:03:25.034 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:25.032+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:25.134 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:25.133+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:25.170 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:25.171 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:25.171 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:03:25.171 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:25.181 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:03:25.235 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:25.235+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:25.326 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2023-07-02T14:03:25.326 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:03:25.326 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2023-07-02T14:03:25.326 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-07-02T14:03:25.346 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:25.347 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:25.347 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:25.347 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:25.347 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:25.356 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:03:25.368 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:03:25.370 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:25.370+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:25.378 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap6 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap7 b3fbb028-e267-4d25-a750-93ba18a2316a@snap7.diff 2023-07-02T14:03:25.437 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:25.437+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:25.441 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:03:25.440+0000 7fda9684f700 0 -- 172.21.15.163:0/2483456693 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x564605804d10 msgr2=0x564605807100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:03:25.445 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:03:25.444+0000 7fda9604e700 0 -- 172.21.15.163:0/2483456693 >> [v2:172.21.15.22:6808/83350,v1:172.21.15.22:6809/83350] conn(0x7fda7c008780 msgr2=0x7fda7c00abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:03:25.489 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:25.489 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:25.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:25.489 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:25.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.490 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:322 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:25.490 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:25.490 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.188024061999997, 2023-07-02T14:03:25.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.03676041, 2023-07-02T14:03:25.490 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:25.490 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.490 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:25.491 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:25.491 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.491 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 322 2023-07-02T14:03:25.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.491 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:25.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:25.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:25.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:25.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:25.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:25.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:25.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:25.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298819+0000", 2023-07-02T14:03:25.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8068e-05 2023-07-02T14:03:25.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:25.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298877+0000", 2023-07-02T14:03:25.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3339999999999998e-06 2023-07-02T14:03:25.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:25.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298883+0000", 2023-07-02T14:03:25.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1030000000000004e-06 2023-07-02T14:03:25.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298887+0000", 2023-07-02T14:03:25.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023640230000000002 2023-07-02T14:03:25.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322528+0000", 2023-07-02T14:03:25.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8424999999999994e-05 2023-07-02T14:03:25.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:25.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322616+0000", 2023-07-02T14:03:25.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116815 2023-07-02T14:03:25.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:25.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322733+0000", 2023-07-02T14:03:25.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048252629999999998 2023-07-02T14:03:25.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:25.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327558+0000", 2023-07-02T14:03:25.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078965240000000003 2023-07-02T14:03:25.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335455+0000", 2023-07-02T14:03:25.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0511999999999999e-05 2023-07-02T14:03:25.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:25.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335515+0000", 2023-07-02T14:03:25.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4789000000000006e-05 2023-07-02T14:03:25.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:25.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335580+0000", 2023-07-02T14:03:25.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.03676041 2023-07-02T14:03:25.500 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.500 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.500 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.500 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:323 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:25.500 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:25.501 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.187952682000002, 2023-07-02T14:03:25.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036875044000000003, 2023-07-02T14:03:25.501 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:25.501 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.501 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:25.501 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:25.501 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.501 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 323 2023-07-02T14:03:25.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.502 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:25.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:25.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:25.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:25.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:25.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:25.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:25.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:25.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298890+0000", 2023-07-02T14:03:25.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7277000000000002e-05 2023-07-02T14:03:25.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:25.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298938+0000", 2023-07-02T14:03:25.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0495e-05 2023-07-02T14:03:25.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:25.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298948+0000", 2023-07-02T14:03:25.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6829999999999999e-06 2023-07-02T14:03:25.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298953+0000", 2023-07-02T14:03:25.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024271435000000001 2023-07-02T14:03:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323224+0000", 2023-07-02T14:03:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9266999999999998e-05 2023-07-02T14:03:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:25.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323304+0000", 2023-07-02T14:03:25.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000157021 2023-07-02T14:03:25.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:25.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323461+0000", 2023-07-02T14:03:25.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041036689999999999 2023-07-02T14:03:25.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:25.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327564+0000", 2023-07-02T14:03:25.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080753490000000008 2023-07-02T14:03:25.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335640+0000", 2023-07-02T14:03:25.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7398999999999994e-05 2023-07-02T14:03:25.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:25.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335727+0000", 2023-07-02T14:03:25.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8967000000000001e-05 2023-07-02T14:03:25.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:25.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335766+0000", 2023-07-02T14:03:25.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036875044000000003 2023-07-02T14:03:25.510 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.510 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.510 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.511 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:324 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:25.511 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:25.511 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.187885303999998, 2023-07-02T14:03:25.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036935673000000002, 2023-07-02T14:03:25.511 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:25.511 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.512 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:25.512 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:25.512 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.512 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 324 2023-07-02T14:03:25.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.512 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:25.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:25.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:25.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:25.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:25.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:25.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:25.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:25.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:25.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5519999999999997e-05 2023-07-02T14:03:25.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:25.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299013+0000", 2023-07-02T14:03:25.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0701e-05 2023-07-02T14:03:25.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:25.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299024+0000", 2023-07-02T14:03:25.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2379999999999999e-06 2023-07-02T14:03:25.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299028+0000", 2023-07-02T14:03:25.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024972768999999999 2023-07-02T14:03:25.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324001+0000", 2023-07-02T14:03:25.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1178000000000002e-05 2023-07-02T14:03:25.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:25.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324072+0000", 2023-07-02T14:03:25.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000166119 2023-07-02T14:03:25.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:25.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324238+0000", 2023-07-02T14:03:25.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033314210000000002 2023-07-02T14:03:25.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:25.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327570+0000", 2023-07-02T14:03:25.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0082393369999999994 2023-07-02T14:03:25.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335809+0000", 2023-07-02T14:03:25.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7071999999999999e-05 2023-07-02T14:03:25.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:25.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335856+0000", 2023-07-02T14:03:25.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7926999999999998e-05 2023-07-02T14:03:25.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:25.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335894+0000", 2023-07-02T14:03:25.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036935673000000002 2023-07-02T14:03:25.521 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.521 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.521 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.521 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:325 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:25.521 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:25.521 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.187809977000001, 2023-07-02T14:03:25.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036990838999999998, 2023-07-02T14:03:25.522 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:25.522 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.522 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:25.522 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:25.522 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.522 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 325 2023-07-02T14:03:25.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.523 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:25.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:25.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:25.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:25.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:25.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:25.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:25.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:25.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299033+0000", 2023-07-02T14:03:25.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107619 2023-07-02T14:03:25.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:25.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299141+0000", 2023-07-02T14:03:25.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3497e-05 2023-07-02T14:03:25.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:25.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299154+0000", 2023-07-02T14:03:25.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4669e-05 2023-07-02T14:03:25.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299169+0000", 2023-07-02T14:03:25.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025630159999999999 2023-07-02T14:03:25.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324799+0000", 2023-07-02T14:03:25.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6953000000000007e-05 2023-07-02T14:03:25.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:25.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324876+0000", 2023-07-02T14:03:25.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016346199999999999 2023-07-02T14:03:25.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:25.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325039+0000", 2023-07-02T14:03:25.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003566624 2023-07-02T14:03:25.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:25.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328606+0000", 2023-07-02T14:03:25.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073295540000000003 2023-07-02T14:03:25.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335936+0000", 2023-07-02T14:03:25.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0044999999999999e-05 2023-07-02T14:03:25.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:25.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335986+0000", 2023-07-02T14:03:25.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8872999999999999e-05 2023-07-02T14:03:25.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:25.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336024+0000", 2023-07-02T14:03:25.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036990838999999998 2023-07-02T14:03:25.531 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.531 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.531 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.532 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:326 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:25.532 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:25.532 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.187668049000003, 2023-07-02T14:03:25.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036975239999999999, 2023-07-02T14:03:25.532 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:25.532 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.532 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:25.533 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:25.533 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.533 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 326 2023-07-02T14:03:25.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.533 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:25.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:25.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:25.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:25.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:25.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:25.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:25.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:25.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299175+0000", 2023-07-02T14:03:25.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4782000000000003e-05 2023-07-02T14:03:25.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:25.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299250+0000", 2023-07-02T14:03:25.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.277e-05 2023-07-02T14:03:25.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:25.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299262+0000", 2023-07-02T14:03:25.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1460000000000001e-06 2023-07-02T14:03:25.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299268+0000", 2023-07-02T14:03:25.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026158098000000001 2023-07-02T14:03:25.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325427+0000", 2023-07-02T14:03:25.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4845999999999995e-05 2023-07-02T14:03:25.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:25.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325501+0000", 2023-07-02T14:03:25.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015890699999999999 2023-07-02T14:03:25.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:25.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325660+0000", 2023-07-02T14:03:25.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002952114 2023-07-02T14:03:25.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:25.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328612+0000", 2023-07-02T14:03:25.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074491949999999996 2023-07-02T14:03:25.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336062+0000", 2023-07-02T14:03:25.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7376999999999997e-05 2023-07-02T14:03:25.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:25.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336109+0000", 2023-07-02T14:03:25.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1814000000000003e-05 2023-07-02T14:03:25.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:25.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336151+0000", 2023-07-02T14:03:25.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036975239999999999 2023-07-02T14:03:25.541 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.542 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.542 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.542 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:327 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 589824~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:25.542 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:25.542 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.187568788, 2023-07-02T14:03:25.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037234253000000002, 2023-07-02T14:03:25.543 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:25.543 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.543 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:25.543 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:25.543 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.543 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 327 2023-07-02T14:03:25.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.543 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:25.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:25.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:25.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:25.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:25.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:25.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:25.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:25.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299274+0000", 2023-07-02T14:03:25.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2213999999999997e-05 2023-07-02T14:03:25.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:25.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299346+0000", 2023-07-02T14:03:25.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1816e-05 2023-07-02T14:03:25.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:25.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299358+0000", 2023-07-02T14:03:25.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2626000000000001e-05 2023-07-02T14:03:25.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299371+0000", 2023-07-02T14:03:25.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026916853000000001 2023-07-02T14:03:25.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326288+0000", 2023-07-02T14:03:25.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6909000000000004e-05 2023-07-02T14:03:25.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:25.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326375+0000", 2023-07-02T14:03:25.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015430399999999999 2023-07-02T14:03:25.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:25.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326529+0000", 2023-07-02T14:03:25.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096652100000000005 2023-07-02T14:03:25.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336194+0000", 2023-07-02T14:03:25.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115299 2023-07-02T14:03:25.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:25.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336309+0000", 2023-07-02T14:03:25.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015981200000000001 2023-07-02T14:03:25.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:25.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336469+0000", 2023-07-02T14:03:25.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9867999999999998e-05 2023-07-02T14:03:25.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:25.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336509+0000", 2023-07-02T14:03:25.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037234253000000002 2023-07-02T14:03:25.552 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.552 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.552 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.552 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:328 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:25.553 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:25.553 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.187465199000002, 2023-07-02T14:03:25.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037216610999999997, 2023-07-02T14:03:25.553 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:25.553 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.553 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:25.553 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:25.553 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.554 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 328 2023-07-02T14:03:25.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.554 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:25.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:25.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:25.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:25.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:25.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:25.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:25.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:25.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:25.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1412e-05 2023-07-02T14:03:25.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:25.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299449+0000", 2023-07-02T14:03:25.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0219000000000001e-05 2023-07-02T14:03:25.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:25.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299469+0000", 2023-07-02T14:03:25.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9480000000000003e-06 2023-07-02T14:03:25.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299479+0000", 2023-07-02T14:03:25.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027444344999999998 2023-07-02T14:03:25.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326924+0000", 2023-07-02T14:03:25.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5165000000000001e-05 2023-07-02T14:03:25.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:25.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327019+0000", 2023-07-02T14:03:25.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019113499999999999 2023-07-02T14:03:25.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:25.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327210+0000", 2023-07-02T14:03:25.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090197449999999992 2023-07-02T14:03:25.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336230+0000", 2023-07-02T14:03:25.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028764399999999999 2023-07-02T14:03:25.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:25.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336517+0000", 2023-07-02T14:03:25.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1981999999999999e-05 2023-07-02T14:03:25.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:25.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336559+0000", 2023-07-02T14:03:25.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:03:25.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:25.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336595+0000", 2023-07-02T14:03:25.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037216610999999997 2023-07-02T14:03:25.562 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.562 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.562 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.563 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:329 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:25.563 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:25.563 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.187355281999999, 2023-07-02T14:03:25.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037191142000000003, 2023-07-02T14:03:25.563 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:25.564 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.564 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:25.564 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:25.564 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.564 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 329 2023-07-02T14:03:25.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.564 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:25.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:25.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:25.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:25.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:25.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:25.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:25.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:25.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299487+0000", 2023-07-02T14:03:25.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7871000000000001e-05 2023-07-02T14:03:25.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:25.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299555+0000", 2023-07-02T14:03:25.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0446e-05 2023-07-02T14:03:25.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:25.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299566+0000", 2023-07-02T14:03:25.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7939999999999999e-06 2023-07-02T14:03:25.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299572+0000", 2023-07-02T14:03:25.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028034019 2023-07-02T14:03:25.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327606+0000", 2023-07-02T14:03:25.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1967999999999999e-05 2023-07-02T14:03:25.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:25.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327678+0000", 2023-07-02T14:03:25.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022216900000000001 2023-07-02T14:03:25.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:25.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327900+0000", 2023-07-02T14:03:25.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084016899999999999 2023-07-02T14:03:25.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336301+0000", 2023-07-02T14:03:25.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00030122800000000001 2023-07-02T14:03:25.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:25.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336603+0000", 2023-07-02T14:03:25.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0380000000000003e-05 2023-07-02T14:03:25.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:25.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336643+0000", 2023-07-02T14:03:25.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6455999999999998e-05 2023-07-02T14:03:25.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:25.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336679+0000", 2023-07-02T14:03:25.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037191142000000003 2023-07-02T14:03:25.573 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.573 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.573 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.573 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:330 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:25.573 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:25.574 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.187264511999999, 2023-07-02T14:03:25.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037849305, 2023-07-02T14:03:25.574 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:25.574 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.574 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:25.574 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:25.574 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.574 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 330 2023-07-02T14:03:25.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.575 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:25.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:25.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:25.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:25.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:25.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:25.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:25.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:25.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299578+0000", 2023-07-02T14:03:25.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4708000000000005e-05 2023-07-02T14:03:25.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:25.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299643+0000", 2023-07-02T14:03:25.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7309999999999999e-06 2023-07-02T14:03:25.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:25.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299652+0000", 2023-07-02T14:03:25.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7989999999999999e-06 2023-07-02T14:03:25.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299658+0000", 2023-07-02T14:03:25.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028981664000000001 2023-07-02T14:03:25.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328639+0000", 2023-07-02T14:03:25.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0939000000000006e-05 2023-07-02T14:03:25.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:25.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328710+0000", 2023-07-02T14:03:25.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000142624 2023-07-02T14:03:25.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:25.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328853+0000", 2023-07-02T14:03:25.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078374640000000006 2023-07-02T14:03:25.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:25.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336690+0000", 2023-07-02T14:03:25.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00068453000000000003 2023-07-02T14:03:25.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337375+0000", 2023-07-02T14:03:25.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0667999999999998e-05 2023-07-02T14:03:25.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:25.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337406+0000", 2023-07-02T14:03:25.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2858999999999999e-05 2023-07-02T14:03:25.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:25.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337428+0000", 2023-07-02T14:03:25.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037849305 2023-07-02T14:03:25.583 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.583 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.583 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.584 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:331 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:25.584 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:25.584 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.187180650000002, 2023-07-02T14:03:25.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037870332, 2023-07-02T14:03:25.584 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:25.584 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.584 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:25.585 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:25.585 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.585 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 331 2023-07-02T14:03:25.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.585 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:25.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:25.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:25.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:25.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:25.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:25.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:25.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:25.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299662+0000", 2023-07-02T14:03:25.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2252e-05 2023-07-02T14:03:25.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:25.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299725+0000", 2023-07-02T14:03:25.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8440000000000004e-06 2023-07-02T14:03:25.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:25.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299733+0000", 2023-07-02T14:03:25.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6759999999999998e-06 2023-07-02T14:03:25.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299738+0000", 2023-07-02T14:03:25.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029459387 2023-07-02T14:03:25.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329197+0000", 2023-07-02T14:03:25.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0087999999999997e-05 2023-07-02T14:03:25.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:25.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329268+0000", 2023-07-02T14:03:25.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014405700000000001 2023-07-02T14:03:25.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:25.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329412+0000", 2023-07-02T14:03:25.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072842100000000002 2023-07-02T14:03:25.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:25.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336696+0000", 2023-07-02T14:03:25.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00076275300000000002 2023-07-02T14:03:25.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337459+0000", 2023-07-02T14:03:25.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5988000000000002e-05 2023-07-02T14:03:25.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:25.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337495+0000", 2023-07-02T14:03:25.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8686000000000002e-05 2023-07-02T14:03:25.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:25.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337533+0000", 2023-07-02T14:03:25.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037870332 2023-07-02T14:03:25.593 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.594 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.594 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.594 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:332 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:25.594 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:25.594 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.187100725999997, 2023-07-02T14:03:25.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037892506999999999, 2023-07-02T14:03:25.595 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:25.595 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.595 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:25.595 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:25.595 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.595 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 332 2023-07-02T14:03:25.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.595 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:25.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:25.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:25.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:25.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:25.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:25.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:25.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:25.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299742+0000", 2023-07-02T14:03:25.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1366999999999997e-05 2023-07-02T14:03:25.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:25.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299804+0000", 2023-07-02T14:03:25.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9319999999999996e-06 2023-07-02T14:03:25.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:25.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299812+0000", 2023-07-02T14:03:25.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4359999999999999e-06 2023-07-02T14:03:25.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299818+0000", 2023-07-02T14:03:25.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029946144000000001 2023-07-02T14:03:25.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329764+0000", 2023-07-02T14:03:25.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000164828 2023-07-02T14:03:25.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:25.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329929+0000", 2023-07-02T14:03:25.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016139400000000001 2023-07-02T14:03:25.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:25.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330090+0000", 2023-07-02T14:03:25.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066134009999999997 2023-07-02T14:03:25.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:25.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336704+0000", 2023-07-02T14:03:25.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00086955400000000003 2023-07-02T14:03:25.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337573+0000", 2023-07-02T14:03:25.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8447000000000001e-05 2023-07-02T14:03:25.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:25.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337602+0000", 2023-07-02T14:03:25.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.362e-05 2023-07-02T14:03:25.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:25.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337635+0000", 2023-07-02T14:03:25.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037892506999999999 2023-07-02T14:03:25.604 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.604 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.604 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.604 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:333 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:25.605 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:25.605 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.187020644999997, 2023-07-02T14:03:25.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037932598999999997, 2023-07-02T14:03:25.605 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:25.605 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.605 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:25.605 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:25.605 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.606 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 333 2023-07-02T14:03:25.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.606 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:25.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:25.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:25.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:25.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:25.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:25.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:25.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:25.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299822+0000", 2023-07-02T14:03:25.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0831000000000001e-05 2023-07-02T14:03:25.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:25.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299883+0000", 2023-07-02T14:03:25.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.322e-06 2023-07-02T14:03:25.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:25.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299888+0000", 2023-07-02T14:03:25.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3479999999999997e-06 2023-07-02T14:03:25.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299893+0000", 2023-07-02T14:03:25.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030831905999999999 2023-07-02T14:03:25.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330725+0000", 2023-07-02T14:03:25.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3560999999999996e-05 2023-07-02T14:03:25.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:25.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330798+0000", 2023-07-02T14:03:25.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015165999999999999 2023-07-02T14:03:25.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:25.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330950+0000", 2023-07-02T14:03:25.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057590319999999999 2023-07-02T14:03:25.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:25.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336709+0000", 2023-07-02T14:03:25.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00096987899999999999 2023-07-02T14:03:25.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337679+0000", 2023-07-02T14:03:25.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9699000000000001e-05 2023-07-02T14:03:25.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:25.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337719+0000", 2023-07-02T14:03:25.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6961000000000001e-05 2023-07-02T14:03:25.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:25.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337756+0000", 2023-07-02T14:03:25.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037932598999999997 2023-07-02T14:03:25.614 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.614 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.614 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.615 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:372 2.0 2:169af44a:::rbd_data.125de1449605.0000000000000106:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:25.615 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:03:25.615 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.183443320000002, 2023-07-02T14:03:25.615 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036066963, 2023-07-02T14:03:25.615 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:25.615 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.616 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:25.616 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:25.616 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.616 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 372 2023-07-02T14:03:25.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.616 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:25.616 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.616 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:25.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:03:25.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:25.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:25.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:03:25.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:25.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:25.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:03:25.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1002000000000003e-05 2023-07-02T14:03:25.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:25.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303461+0000", 2023-07-02T14:03:25.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9306e-05 2023-07-02T14:03:25.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:25.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303490+0000", 2023-07-02T14:03:25.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5026e-05 2023-07-02T14:03:25.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303505+0000", 2023-07-02T14:03:25.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033623129000000002 2023-07-02T14:03:25.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337128+0000", 2023-07-02T14:03:25.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6968000000000002e-05 2023-07-02T14:03:25.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:25.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337175+0000", 2023-07-02T14:03:25.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000101127 2023-07-02T14:03:25.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:25.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337276+0000", 2023-07-02T14:03:25.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017866920000000001 2023-07-02T14:03:25.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:25.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339063+0000", 2023-07-02T14:03:25.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00031459800000000003 2023-07-02T14:03:25.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339377+0000", 2023-07-02T14:03:25.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7145999999999999e-05 2023-07-02T14:03:25.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:25.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339425+0000", 2023-07-02T14:03:25.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2632000000000001e-05 2023-07-02T14:03:25.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:25.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339467+0000", 2023-07-02T14:03:25.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036066963 2023-07-02T14:03:25.625 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.625 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.625 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.625 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:416 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:25.625 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:25.625 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.168727265999998, 2023-07-02T14:03:25.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039242025, 2023-07-02T14:03:25.626 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:25.626 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.626 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:25.626 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:25.626 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.626 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:03:25.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.627 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:25.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:25.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:25.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:25.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:25.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:25.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:25.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:25.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:25.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.692e-06 2023-07-02T14:03:25.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:25.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318119+0000", 2023-07-02T14:03:25.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9790000000000005e-06 2023-07-02T14:03:25.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:25.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318129+0000", 2023-07-02T14:03:25.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3640000000000001e-06 2023-07-02T14:03:25.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318137+0000", 2023-07-02T14:03:25.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022607808 2023-07-02T14:03:25.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340745+0000", 2023-07-02T14:03:25.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0130999999999996e-05 2023-07-02T14:03:25.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340825+0000", 2023-07-02T14:03:25.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039581080000000001 2023-07-02T14:03:25.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344783+0000", 2023-07-02T14:03:25.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000006e-05 2023-07-02T14:03:25.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344854+0000", 2023-07-02T14:03:25.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052105550000000004 2023-07-02T14:03:25.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350065+0000", 2023-07-02T14:03:25.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0321999999999997e-05 2023-07-02T14:03:25.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350115+0000", 2023-07-02T14:03:25.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012378409999999999 2023-07-02T14:03:25.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351353+0000", 2023-07-02T14:03:25.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5942000000000001e-05 2023-07-02T14:03:25.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:25.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351409+0000", 2023-07-02T14:03:25.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035677700000000002 2023-07-02T14:03:25.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:25.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351766+0000", 2023-07-02T14:03:25.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000926719 2023-07-02T14:03:25.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:25.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352693+0000", 2023-07-02T14:03:25.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045652080000000003 2023-07-02T14:03:25.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357258+0000", 2023-07-02T14:03:25.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4717e-05 2023-07-02T14:03:25.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:25.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357293+0000", 2023-07-02T14:03:25.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5807000000000002e-05 2023-07-02T14:03:25.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:25.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357358+0000", 2023-07-02T14:03:25.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039242025 2023-07-02T14:03:25.639 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.639 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.639 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.639 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:417 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:25.640 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:25.640 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.168701829, 2023-07-02T14:03:25.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042407105, 2023-07-02T14:03:25.640 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:25.640 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.640 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:25.640 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:25.640 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.641 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 417 2023-07-02T14:03:25.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.641 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:25.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:25.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:25.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:25.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:25.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:25.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:25.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:25.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318141+0000", 2023-07-02T14:03:25.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7120000000000002e-06 2023-07-02T14:03:25.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:25.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318145+0000", 2023-07-02T14:03:25.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8869999999999997e-06 2023-07-02T14:03:25.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:25.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318151+0000", 2023-07-02T14:03:25.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9820000000000002e-06 2023-07-02T14:03:25.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318155+0000", 2023-07-02T14:03:25.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699133 2023-07-02T14:03:25.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340854+0000", 2023-07-02T14:03:25.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6849999999999998e-05 2023-07-02T14:03:25.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340931+0000", 2023-07-02T14:03:25.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039672140000000002 2023-07-02T14:03:25.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344898+0000", 2023-07-02T14:03:25.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5341999999999998e-05 2023-07-02T14:03:25.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344973+0000", 2023-07-02T14:03:25.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051819400000000003 2023-07-02T14:03:25.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350155+0000", 2023-07-02T14:03:25.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3261999999999999e-05 2023-07-02T14:03:25.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350209+0000", 2023-07-02T14:03:25.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00194097 2023-07-02T14:03:25.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352150+0000", 2023-07-02T14:03:25.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6851e-05 2023-07-02T14:03:25.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352226+0000", 2023-07-02T14:03:25.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051831109999999998 2023-07-02T14:03:25.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357409+0000", 2023-07-02T14:03:25.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5325e-05 2023-07-02T14:03:25.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:25.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357495+0000", 2023-07-02T14:03:25.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042107899999999998 2023-07-02T14:03:25.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:25.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357916+0000", 2023-07-02T14:03:25.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0011672449999999999 2023-07-02T14:03:25.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:25.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.359083+0000", 2023-07-02T14:03:25.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013534930000000001 2023-07-02T14:03:25.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360437+0000", 2023-07-02T14:03:25.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2159000000000002e-05 2023-07-02T14:03:25.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:25.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360469+0000", 2023-07-02T14:03:25.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0092000000000005e-05 2023-07-02T14:03:25.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:25.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360549+0000", 2023-07-02T14:03:25.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042407105 2023-07-02T14:03:25.654 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.654 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.654 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.655 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:418 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:25.655 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:25.655 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.168682588000003, 2023-07-02T14:03:25.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.045613663999999998, 2023-07-02T14:03:25.655 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:25.656 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.656 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:25.656 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:25.656 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.656 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 418 2023-07-02T14:03:25.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.656 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:25.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:25.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:25.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:25.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:25.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:25.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:25.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:25.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318160+0000", 2023-07-02T14:03:25.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6490000000000001e-06 2023-07-02T14:03:25.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:25.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318164+0000", 2023-07-02T14:03:25.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:03:25.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:25.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318170+0000", 2023-07-02T14:03:25.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5279999999999999e-06 2023-07-02T14:03:25.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318173+0000", 2023-07-02T14:03:25.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022788856 2023-07-02T14:03:25.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340962+0000", 2023-07-02T14:03:25.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7730000000000003e-05 2023-07-02T14:03:25.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341040+0000", 2023-07-02T14:03:25.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039795899999999999 2023-07-02T14:03:25.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345020+0000", 2023-07-02T14:03:25.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4962000000000006e-05 2023-07-02T14:03:25.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345085+0000", 2023-07-02T14:03:25.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051955430000000004 2023-07-02T14:03:25.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350280+0000", 2023-07-02T14:03:25.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8913999999999995e-05 2023-07-02T14:03:25.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350379+0000", 2023-07-02T14:03:25.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001887344 2023-07-02T14:03:25.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352266+0000", 2023-07-02T14:03:25.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7865000000000003e-05 2023-07-02T14:03:25.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352314+0000", 2023-07-02T14:03:25.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060621190000000004 2023-07-02T14:03:25.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358376+0000", 2023-07-02T14:03:25.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2448999999999997e-05 2023-07-02T14:03:25.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358449+0000", 2023-07-02T14:03:25.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0021539929999999999 2023-07-02T14:03:25.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360603+0000", 2023-07-02T14:03:25.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7859000000000002e-05 2023-07-02T14:03:25.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:25.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360671+0000", 2023-07-02T14:03:25.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026930699999999999 2023-07-02T14:03:25.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:25.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360940+0000", 2023-07-02T14:03:25.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00081179699999999995 2023-07-02T14:03:25.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:25.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361752+0000", 2023-07-02T14:03:25.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0019350490000000001 2023-07-02T14:03:25.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363687+0000", 2023-07-02T14:03:25.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4462000000000003e-05 2023-07-02T14:03:25.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:25.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363721+0000", 2023-07-02T14:03:25.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3408000000000001e-05 2023-07-02T14:03:25.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:25.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363775+0000", 2023-07-02T14:03:25.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.045613663999999998 2023-07-02T14:03:25.671 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.671 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.671 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.672 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:419 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:25.672 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:25.672 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.168666260999998, 2023-07-02T14:03:25.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.047296703000000002, 2023-07-02T14:03:25.672 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:25.672 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.672 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:25.673 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:25.673 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.673 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 419 2023-07-02T14:03:25.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.673 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:25.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:25.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:25.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:25.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:25.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:25.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:25.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:25.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:25.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.433e-06 2023-07-02T14:03:25.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:25.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318180+0000", 2023-07-02T14:03:25.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9950000000000002e-06 2023-07-02T14:03:25.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:25.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318186+0000", 2023-07-02T14:03:25.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7290000000000002e-06 2023-07-02T14:03:25.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318190+0000", 2023-07-02T14:03:25.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022877989000000001 2023-07-02T14:03:25.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341068+0000", 2023-07-02T14:03:25.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5803999999999994e-05 2023-07-02T14:03:25.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341154+0000", 2023-07-02T14:03:25.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039633000000000003 2023-07-02T14:03:25.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345117+0000", 2023-07-02T14:03:25.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7188999999999996e-05 2023-07-02T14:03:25.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345194+0000", 2023-07-02T14:03:25.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052306059999999996 2023-07-02T14:03:25.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350425+0000", 2023-07-02T14:03:25.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8093999999999997e-05 2023-07-02T14:03:25.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350473+0000", 2023-07-02T14:03:25.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001865158 2023-07-02T14:03:25.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352338+0000", 2023-07-02T14:03:25.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5479000000000003e-05 2023-07-02T14:03:25.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352384+0000", 2023-07-02T14:03:25.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061153290000000001 2023-07-02T14:03:25.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358499+0000", 2023-07-02T14:03:25.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5156999999999999e-05 2023-07-02T14:03:25.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358574+0000", 2023-07-02T14:03:25.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027847459999999998 2023-07-02T14:03:25.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361359+0000", 2023-07-02T14:03:25.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2633999999999999e-05 2023-07-02T14:03:25.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361402+0000", 2023-07-02T14:03:25.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024171510000000002 2023-07-02T14:03:25.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363819+0000", 2023-07-02T14:03:25.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6988999999999999e-05 2023-07-02T14:03:25.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:25.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363876+0000", 2023-07-02T14:03:25.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035321300000000001 2023-07-02T14:03:25.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:25.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.364229+0000", 2023-07-02T14:03:25.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00094674799999999997 2023-07-02T14:03:25.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:25.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365176+0000", 2023-07-02T14:03:25.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019724200000000001 2023-07-02T14:03:25.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365373+0000", 2023-07-02T14:03:25.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2116e-05 2023-07-02T14:03:25.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:25.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365425+0000", 2023-07-02T14:03:25.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8996999999999999e-05 2023-07-02T14:03:25.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:25.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365474+0000", 2023-07-02T14:03:25.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.047296703000000002 2023-07-02T14:03:25.689 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.689 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.689 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.690 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:472 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:25.690 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:25.690 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.109210029000003, 2023-07-02T14:03:25.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036204721000000002, 2023-07-02T14:03:25.690 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:25.690 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.690 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:25.691 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:25.691 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.691 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 472 2023-07-02T14:03:25.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.691 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:25.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:25.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:25.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:25.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:25.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:25.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:03:25.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:25.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377633+0000", 2023-07-02T14:03:25.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.815e-06 2023-07-02T14:03:25.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:25.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377635+0000", 2023-07-02T14:03:25.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9459999999999998e-06 2023-07-02T14:03:25.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:25.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377638+0000", 2023-07-02T14:03:25.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.673e-06 2023-07-02T14:03:25.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377643+0000", 2023-07-02T14:03:25.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043004239999999997 2023-07-02T14:03:25.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.381943+0000", 2023-07-02T14:03:25.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4321999999999999e-05 2023-07-02T14:03:25.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.381998+0000", 2023-07-02T14:03:25.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003008377 2023-07-02T14:03:25.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385006+0000", 2023-07-02T14:03:25.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8751999999999998e-05 2023-07-02T14:03:25.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385045+0000", 2023-07-02T14:03:25.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086260880000000005 2023-07-02T14:03:25.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393671+0000", 2023-07-02T14:03:25.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6365999999999997e-05 2023-07-02T14:03:25.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393707+0000", 2023-07-02T14:03:25.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028560289999999999 2023-07-02T14:03:25.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396563+0000", 2023-07-02T14:03:25.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6195e-05 2023-07-02T14:03:25.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396619+0000", 2023-07-02T14:03:25.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023118269999999998 2023-07-02T14:03:25.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.398931+0000", 2023-07-02T14:03:25.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3885999999999999e-05 2023-07-02T14:03:25.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.398985+0000", 2023-07-02T14:03:25.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043756890000000003 2023-07-02T14:03:25.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403361+0000", 2023-07-02T14:03:25.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0043000000000001e-05 2023-07-02T14:03:25.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403421+0000", 2023-07-02T14:03:25.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001590972 2023-07-02T14:03:25.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405012+0000", 2023-07-02T14:03:25.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0067000000000005e-05 2023-07-02T14:03:25.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405082+0000", 2023-07-02T14:03:25.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050958610000000001 2023-07-02T14:03:25.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410178+0000", 2023-07-02T14:03:25.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7070000000000004e-05 2023-07-02T14:03:25.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410275+0000", 2023-07-02T14:03:25.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060121100000000004 2023-07-02T14:03:25.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410876+0000", 2023-07-02T14:03:25.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3177999999999999e-05 2023-07-02T14:03:25.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:25.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410919+0000", 2023-07-02T14:03:25.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000271394 2023-07-02T14:03:25.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:25.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411191+0000", 2023-07-02T14:03:25.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001009598 2023-07-02T14:03:25.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.412200+0000", 2023-07-02T14:03:25.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014789790000000001 2023-07-02T14:03:25.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:25.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413679+0000", 2023-07-02T14:03:25.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2624000000000002e-05 2023-07-02T14:03:25.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:25.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413742+0000", 2023-07-02T14:03:25.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6564999999999994e-05 2023-07-02T14:03:25.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:25.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413838+0000", 2023-07-02T14:03:25.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036204721000000002 2023-07-02T14:03:25.710 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.710 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.710 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.710 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:473 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:25.711 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:25.711 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.109198618000001, 2023-07-02T14:03:25.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037915904, 2023-07-02T14:03:25.711 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:25.711 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.711 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:25.711 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:25.711 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.712 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 473 2023-07-02T14:03:25.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.712 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:25.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:25.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:25.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:25.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:25.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:25.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:03:25.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:25.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:25.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.119e-06 2023-07-02T14:03:25.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:25.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377647+0000", 2023-07-02T14:03:25.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.875e-06 2023-07-02T14:03:25.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:25.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377650+0000", 2023-07-02T14:03:25.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1949999999999999e-06 2023-07-02T14:03:25.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377653+0000", 2023-07-02T14:03:25.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004375444 2023-07-02T14:03:25.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382028+0000", 2023-07-02T14:03:25.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1063999999999997e-05 2023-07-02T14:03:25.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382089+0000", 2023-07-02T14:03:25.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030043869999999999 2023-07-02T14:03:25.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385094+0000", 2023-07-02T14:03:25.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8167999999999997e-05 2023-07-02T14:03:25.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385142+0000", 2023-07-02T14:03:25.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008603421 2023-07-02T14:03:25.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393745+0000", 2023-07-02T14:03:25.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3763000000000005e-05 2023-07-02T14:03:25.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393819+0000", 2023-07-02T14:03:25.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028274720000000001 2023-07-02T14:03:25.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396647+0000", 2023-07-02T14:03:25.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7929999999999997e-05 2023-07-02T14:03:25.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396695+0000", 2023-07-02T14:03:25.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023426459999999999 2023-07-02T14:03:25.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399037+0000", 2023-07-02T14:03:25.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5846000000000004e-05 2023-07-02T14:03:25.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399123+0000", 2023-07-02T14:03:25.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043448360000000004 2023-07-02T14:03:25.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403468+0000", 2023-07-02T14:03:25.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6699000000000001e-05 2023-07-02T14:03:25.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403515+0000", 2023-07-02T14:03:25.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001622943 2023-07-02T14:03:25.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405138+0000", 2023-07-02T14:03:25.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6382000000000002e-05 2023-07-02T14:03:25.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405204+0000", 2023-07-02T14:03:25.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051089910000000002 2023-07-02T14:03:25.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410313+0000", 2023-07-02T14:03:25.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8106000000000003e-05 2023-07-02T14:03:25.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410361+0000", 2023-07-02T14:03:25.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012200399999999999 2023-07-02T14:03:25.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411581+0000", 2023-07-02T14:03:25.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5067000000000005e-05 2023-07-02T14:03:25.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411656+0000", 2023-07-02T14:03:25.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022353249999999998 2023-07-02T14:03:25.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413891+0000", 2023-07-02T14:03:25.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9356000000000003e-05 2023-07-02T14:03:25.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:25.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413981+0000", 2023-07-02T14:03:25.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00043818299999999998 2023-07-02T14:03:25.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:25.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.414419+0000", 2023-07-02T14:03:25.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00081559199999999997 2023-07-02T14:03:25.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:25.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415235+0000", 2023-07-02T14:03:25.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026434499999999998 2023-07-02T14:03:25.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415499+0000", 2023-07-02T14:03:25.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9779999999999999e-05 2023-07-02T14:03:25.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:25.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415529+0000", 2023-07-02T14:03:25.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2159999999999997e-05 2023-07-02T14:03:25.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:25.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415561+0000", 2023-07-02T14:03:25.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037915904 2023-07-02T14:03:25.732 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.732 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.732 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.732 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:475 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:25.732 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:25.732 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.109188578999998, 2023-07-02T14:03:25.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039286962000000002, 2023-07-02T14:03:25.733 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:25.733 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:25.733 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:25.733 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:25.733 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:25.733 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 475 2023-07-02T14:03:25.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.734 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:25.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:25.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:25.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:25.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:25.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:25.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:25.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:25.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:25.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4779999999999997e-06 2023-07-02T14:03:25.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:25.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377664+0000", 2023-07-02T14:03:25.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.268e-06 2023-07-02T14:03:25.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:25.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377671+0000", 2023-07-02T14:03:25.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.861e-06 2023-07-02T14:03:25.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:25.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377674+0000", 2023-07-02T14:03:25.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044395789999999999 2023-07-02T14:03:25.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382114+0000", 2023-07-02T14:03:25.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8096999999999997e-05 2023-07-02T14:03:25.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382162+0000", 2023-07-02T14:03:25.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029949059999999999 2023-07-02T14:03:25.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385157+0000", 2023-07-02T14:03:25.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1158000000000001e-05 2023-07-02T14:03:25.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385198+0000", 2023-07-02T14:03:25.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086477229999999995 2023-07-02T14:03:25.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393846+0000", 2023-07-02T14:03:25.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5990999999999999e-05 2023-07-02T14:03:25.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393912+0000", 2023-07-02T14:03:25.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027979599999999999 2023-07-02T14:03:25.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396710+0000", 2023-07-02T14:03:25.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5911000000000002e-05 2023-07-02T14:03:25.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396746+0000", 2023-07-02T14:03:25.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024039759999999999 2023-07-02T14:03:25.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399150+0000", 2023-07-02T14:03:25.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7834999999999999e-05 2023-07-02T14:03:25.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399188+0000", 2023-07-02T14:03:25.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004364054 2023-07-02T14:03:25.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403552+0000", 2023-07-02T14:03:25.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6274999999999999e-05 2023-07-02T14:03:25.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403608+0000", 2023-07-02T14:03:25.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0016261540000000001 2023-07-02T14:03:25.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405234+0000", 2023-07-02T14:03:25.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6945000000000004e-05 2023-07-02T14:03:25.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405301+0000", 2023-07-02T14:03:25.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050871329999999998 2023-07-02T14:03:25.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410388+0000", 2023-07-02T14:03:25.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0210999999999995e-05 2023-07-02T14:03:25.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410468+0000", 2023-07-02T14:03:25.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001227454 2023-07-02T14:03:25.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411696+0000", 2023-07-02T14:03:25.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8188000000000005e-05 2023-07-02T14:03:25.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411764+0000", 2023-07-02T14:03:25.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031779930000000001 2023-07-02T14:03:25.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.414942+0000", 2023-07-02T14:03:25.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7236000000000004e-05 2023-07-02T14:03:25.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:25.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415019+0000", 2023-07-02T14:03:25.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000573045 2023-07-02T14:03:25.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:25.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415592+0000", 2023-07-02T14:03:25.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7580000000000002e-05 2023-07-02T14:03:25.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:25.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415640+0000", 2023-07-02T14:03:25.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024783200000000002 2023-07-02T14:03:25.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:25.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415888+0000", 2023-07-02T14:03:25.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00080440999999999998 2023-07-02T14:03:25.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:25.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416692+0000", 2023-07-02T14:03:25.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000195787 2023-07-02T14:03:25.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:25.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416888+0000", 2023-07-02T14:03:25.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0787999999999997e-05 2023-07-02T14:03:25.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:25.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416919+0000", 2023-07-02T14:03:25.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.338e-05 2023-07-02T14:03:25.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:25.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:25.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:25.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416942+0000", 2023-07-02T14:03:25.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039286962000000002 2023-07-02T14:03:25.755 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.755 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.755 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.755 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:25.755 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:25.755 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:25.757 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 100% complete...done. 2023-07-02T14:03:25.758 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:03:25.758 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=8 2023-07-02T14:03:25.758 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap8 2023-07-02T14:03:25.758 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:03:25.759 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:25.756+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:25.759 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:03:25.759 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:25.759 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 69, 2023-07-02T14:03:25.759 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2", 2023-07-02T14:03:25.760 INFO:teuthology.orchestra.run.smithi016.stdout: "created": "2023-07-02T14:01:37.298060+0000", 2023-07-02T14:03:25.760 INFO:teuthology.orchestra.run.smithi016.stdout: "modified": "2023-07-02T14:03:24.352048+0000", 2023-07-02T14:03:25.760 INFO:teuthology.orchestra.run.smithi016.stdout: "last_up_change": "2023-07-02T14:01:45.271760+0000", 2023-07-02T14:03:25.760 INFO:teuthology.orchestra.run.smithi016.stdout: "last_in_change": "2023-07-02T14:03:09.309538+0000", 2023-07-02T14:03:25.760 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-07-02T14:03:25.760 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_num": 5801984, 2023-07-02T14:03:25.760 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_set": [ 2023-07-02T14:03:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: "noscrub", 2023-07-02T14:03:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: "pglog_hardlimit", 2023-07-02T14:03:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: "purged_snapdirs", 2023-07-02T14:03:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_deletes", 2023-07-02T14:03:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: "sortbitwise" 2023-07-02T14:03:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:03:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_version": 4, 2023-07-02T14:03:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: "full_ratio": 0.94999998807907104, 2023-07-02T14:03:25.762 INFO:teuthology.orchestra.run.smithi016.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-07-02T14:03:25.762 INFO:teuthology.orchestra.run.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2023-07-02T14:03:25.762 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_snapshot": "", 2023-07-02T14:03:25.762 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_max": 2, 2023-07-02T14:03:25.762 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd": 6, 2023-07-02T14:03:25.762 INFO:teuthology.orchestra.run.smithi016.stdout: "require_min_compat_client": "luminous", 2023-07-02T14:03:25.762 INFO:teuthology.orchestra.run.smithi016.stdout: "min_compat_client": "luminous", 2023-07-02T14:03:25.762 INFO:teuthology.orchestra.run.smithi016.stdout: "require_osd_release": "pacific", 2023-07-02T14:03:25.763 INFO:teuthology.orchestra.run.smithi016.stdout: "pools": [ 2023-07-02T14:03:25.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.763 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 1, 2023-07-02T14:03:25.763 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "device_health_metrics", 2023-07-02T14:03:25.763 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:45.323455+0000", 2023-07-02T14:03:25.763 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2023-07-02T14:03:25.763 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2023-07-02T14:03:25.763 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:03:25.764 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:03:25.764 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:03:25.764 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:03:25.764 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:03:25.764 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:03:25.764 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:03:25.764 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:03:25.764 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:03:25.765 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:03:25.765 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 1, 2023-07-02T14:03:25.765 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 1, 2023-07-02T14:03:25.765 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 1, 2023-07-02T14:03:25.765 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 1, 2023-07-02T14:03:25.765 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 1, 2023-07-02T14:03:25.765 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:03:25.766 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:03:25.766 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:03:25.766 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:03:25.766 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:03:25.766 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:03:25.766 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:03:25.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.766 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "14", 2023-07-02T14:03:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:03:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:03:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:03:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:03:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:03:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2023-07-02T14:03:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2023-07-02T14:03:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:03:25.768 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:03:25.768 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:03:25.768 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:03:25.768 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:03:25.768 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:03:25.768 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:03:25.768 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:03:25.768 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:03:25.769 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:03:25.769 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:03:25.769 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:03:25.769 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:03:25.769 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:03:25.769 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:03:25.769 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:03:25.769 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:03:25.770 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:03:25.770 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:03:25.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.770 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:03:25.770 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:03:25.770 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:03:25.770 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:03:25.771 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:03:25.771 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:03:25.771 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:03:25.771 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:03:25.771 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:03:25.771 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:03:25.771 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:03:25.771 INFO:teuthology.orchestra.run.smithi016.stdout: "options": { 2023-07-02T14:03:25.772 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_max": 32, 2023-07-02T14:03:25.772 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_min": 1 2023-07-02T14:03:25.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.772 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:03:25.772 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_devicehealth": {} 2023-07-02T14:03:25.772 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.773 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 2, 2023-07-02T14:03:25.773 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "rbd", 2023-07-02T14:03:25.773 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:47.448897+0000", 2023-07-02T14:03:25.773 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 8193, 2023-07-02T14:03:25.773 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2023-07-02T14:03:25.773 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:03:25.773 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:03:25.773 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:03:25.774 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:03:25.774 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:03:25.774 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:03:25.774 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:03:25.774 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:03:25.774 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:03:25.774 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:03:25.774 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2023-07-02T14:03:25.775 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2023-07-02T14:03:25.775 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2023-07-02T14:03:25.775 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2023-07-02T14:03:25.775 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2023-07-02T14:03:25.775 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:03:25.775 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:03:25.775 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:03:25.775 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:03:25.776 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:03:25.776 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:03:25.776 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:03:25.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.776 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "69", 2023-07-02T14:03:25.776 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:03:25.776 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:03:25.777 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:03:25.777 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:03:25.777 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:03:25.777 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 10, 2023-07-02T14:03:25.777 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 69, 2023-07-02T14:03:25.777 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:03:25.777 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:03:25.777 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:03:25.778 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:03:25.778 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:03:25.778 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:03:25.778 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:03:25.778 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:03:25.778 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:03:25.778 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:03:25.778 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:03:25.779 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:03:25.779 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:03:25.779 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:03:25.779 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:03:25.779 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:03:25.779 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:03:25.779 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:03:25.779 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:03:25.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.780 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:03:25.780 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:03:25.780 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:03:25.780 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:03:25.780 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:03:25.780 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:03:25.780 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:03:25.781 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:03:25.781 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:03:25.781 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:03:25.781 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:03:25.781 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2023-07-02T14:03:25.781 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:03:25.781 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2023-07-02T14:03:25.781 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.782 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.782 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:03:25.782 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:03:25.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.782 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:03:25.782 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "43c446fe-c811-4ee3-99e8-d7452edfc1c1", 2023-07-02T14:03:25.782 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:03:25.783 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:03:25.783 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:03:25.783 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:03:25.783 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:03:25.783 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:03:25.783 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:03:25.783 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 62, 2023-07-02T14:03:25.783 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:03:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:03:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:03:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2023-07-02T14:03:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2023-07-02T14:03:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:03:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6802", 2023-07-02T14:03:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.787 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6803", 2023-07-02T14:03:25.787 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:25.787 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.787 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.787 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:03:25.787 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.788 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.788 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6806", 2023-07-02T14:03:25.788 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:25.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.788 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.788 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6807", 2023-07-02T14:03:25.789 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:25.789 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.789 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.789 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:03:25.789 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.789 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.790 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6804", 2023-07-02T14:03:25.790 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:25.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.790 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.790 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6805", 2023-07-02T14:03:25.790 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:25.790 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.791 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.791 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6801/83359", 2023-07-02T14:03:25.791 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6803/83359", 2023-07-02T14:03:25.791 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6807/83359", 2023-07-02T14:03:25.791 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6805/83359", 2023-07-02T14:03:25.791 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:03:25.791 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:03:25.792 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:03:25.792 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.792 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:03:25.792 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b", 2023-07-02T14:03:25.792 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:03:25.792 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:03:25.793 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:03:25.793 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:03:25.793 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:03:25.793 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:03:25.793 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:03:25.793 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 63, 2023-07-02T14:03:25.793 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:03:25.793 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:03:25.794 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:03:25.794 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.794 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.794 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6808", 2023-07-02T14:03:25.794 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:25.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.795 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.795 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6809", 2023-07-02T14:03:25.795 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:25.795 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.795 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.795 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:03:25.796 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.796 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.796 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6810", 2023-07-02T14:03:25.796 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:25.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.796 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.797 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6811", 2023-07-02T14:03:25.797 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:25.797 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.797 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.797 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:03:25.797 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.798 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.798 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6814", 2023-07-02T14:03:25.798 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:25.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.798 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.798 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6815", 2023-07-02T14:03:25.798 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:25.799 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.799 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.799 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:03:25.799 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.799 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.799 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6812", 2023-07-02T14:03:25.800 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:25.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.800 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.800 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6813", 2023-07-02T14:03:25.800 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:25.800 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.801 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.801 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6809/83360", 2023-07-02T14:03:25.801 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6811/83360", 2023-07-02T14:03:25.801 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6815/83360", 2023-07-02T14:03:25.801 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6813/83360", 2023-07-02T14:03:25.801 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:03:25.801 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:03:25.802 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:03:25.802 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.802 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:03:25.802 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "99f8ea0a-5a71-4abe-b087-8e891fd11d1b", 2023-07-02T14:03:25.802 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:03:25.802 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:03:25.803 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:03:25.803 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:03:25.803 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:03:25.803 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:03:25.803 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:03:25.803 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 62, 2023-07-02T14:03:25.803 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:03:25.803 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:03:25.804 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:03:25.804 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.804 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.804 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6816", 2023-07-02T14:03:25.804 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:25.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.805 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.805 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6817", 2023-07-02T14:03:25.805 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:25.805 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.805 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.805 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:03:25.805 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.806 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.806 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6818", 2023-07-02T14:03:25.806 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:25.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.806 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.806 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6819", 2023-07-02T14:03:25.807 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:25.807 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.807 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.807 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:03:25.807 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.807 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.808 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6822", 2023-07-02T14:03:25.808 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:25.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.808 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.808 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6823", 2023-07-02T14:03:25.808 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:25.808 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.809 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.809 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:03:25.809 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.809 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.809 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6820", 2023-07-02T14:03:25.809 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:25.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.810 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.810 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6821", 2023-07-02T14:03:25.810 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:25.810 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.810 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.811 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6817/83361", 2023-07-02T14:03:25.811 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6819/83361", 2023-07-02T14:03:25.811 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6823/83361", 2023-07-02T14:03:25.811 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6821/83361", 2023-07-02T14:03:25.811 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:03:25.811 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:03:25.811 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:03:25.812 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.812 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:03:25.812 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "d285b4d5-0bef-421a-b84a-943e485b10c2", 2023-07-02T14:03:25.812 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:03:25.812 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:03:25.812 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:03:25.813 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:03:25.813 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:03:25.813 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:03:25.813 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:03:25.813 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 63, 2023-07-02T14:03:25.813 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:03:25.813 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:03:25.813 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:03:25.814 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.814 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.814 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6808", 2023-07-02T14:03:25.814 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:25.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.814 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.815 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6809", 2023-07-02T14:03:25.815 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:25.815 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.815 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.815 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:03:25.815 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.816 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.816 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6810", 2023-07-02T14:03:25.816 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:25.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.816 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.816 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6811", 2023-07-02T14:03:25.816 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:25.817 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.817 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.817 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:03:25.817 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.817 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.817 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6814", 2023-07-02T14:03:25.818 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:25.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.818 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.818 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6815", 2023-07-02T14:03:25.818 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:25.818 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.818 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.819 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:03:25.819 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.819 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.819 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6812", 2023-07-02T14:03:25.819 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:25.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.820 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.820 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6813", 2023-07-02T14:03:25.820 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:25.820 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.820 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.821 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6809/83350", 2023-07-02T14:03:25.821 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6811/83350", 2023-07-02T14:03:25.821 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6815/83350", 2023-07-02T14:03:25.821 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6813/83350", 2023-07-02T14:03:25.821 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:03:25.821 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:03:25.821 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:03:25.821 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.822 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:03:25.822 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "7ba0b0db-e407-4c63-8278-aab09f3082c4", 2023-07-02T14:03:25.822 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:03:25.822 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:03:25.822 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:03:25.822 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 0.23297119140625, 2023-07-02T14:03:25.823 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:03:25.823 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:03:25.823 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:03:25.823 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 0, 2023-07-02T14:03:25.823 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:03:25.823 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:03:25.823 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:03:25.823 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.824 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.824 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6800", 2023-07-02T14:03:25.824 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:25.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.824 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.824 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6801", 2023-07-02T14:03:25.825 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:25.825 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.825 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.825 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:03:25.825 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.826 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.826 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6802", 2023-07-02T14:03:25.826 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:25.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.826 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.826 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6803", 2023-07-02T14:03:25.826 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:25.827 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.827 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.827 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:03:25.827 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.827 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.827 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6806", 2023-07-02T14:03:25.828 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:25.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.828 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.828 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6807", 2023-07-02T14:03:25.828 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:25.828 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.828 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.829 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:03:25.829 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.829 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.829 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6804", 2023-07-02T14:03:25.829 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:25.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.830 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.830 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6805", 2023-07-02T14:03:25.830 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:25.830 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.830 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.830 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6801/83351", 2023-07-02T14:03:25.831 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6803/83351", 2023-07-02T14:03:25.831 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6807/83351", 2023-07-02T14:03:25.831 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6805/83351", 2023-07-02T14:03:25.831 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:03:25.831 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:03:25.831 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:03:25.831 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.832 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:03:25.832 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "29663bfa-f6fe-4ce7-b45a-a9b7b4424c42", 2023-07-02T14:03:25.832 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:03:25.832 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 0, 2023-07-02T14:03:25.832 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 0, 2023-07-02T14:03:25.832 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:03:25.832 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:03:25.833 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:03:25.833 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:03:25.833 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 34, 2023-07-02T14:03:25.833 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:03:25.833 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:03:25.833 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:03:25.833 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.834 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.834 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6816", 2023-07-02T14:03:25.834 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:25.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.834 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.834 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6817", 2023-07-02T14:03:25.834 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:25.835 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.835 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.835 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:03:25.835 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.835 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.836 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6818", 2023-07-02T14:03:25.836 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:25.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.836 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.836 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6819", 2023-07-02T14:03:25.836 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:25.836 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.837 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.837 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:03:25.837 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.837 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.837 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6822", 2023-07-02T14:03:25.837 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:25.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.838 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.838 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6823", 2023-07-02T14:03:25.838 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:25.838 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.838 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.839 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:03:25.839 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:25.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.839 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:25.839 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6820", 2023-07-02T14:03:25.839 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:25.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.840 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:25.840 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6821", 2023-07-02T14:03:25.840 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:25.840 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.840 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.840 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6817/83352", 2023-07-02T14:03:25.840 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6819/83352", 2023-07-02T14:03:25.841 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6823/83352", 2023-07-02T14:03:25.841 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6821/83352", 2023-07-02T14:03:25.841 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:03:25.841 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:03:25.841 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:03:25.841 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.841 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.841 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:03:25.842 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_xinfo": [ 2023-07-02T14:03:25.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.842 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:03:25.842 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:03:25.842 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:03:25.842 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:03:25.842 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:03:25.843 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:03:25.843 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.001178+0000", 2023-07-02T14:03:25.843 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:03:25.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.843 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:03:25.843 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:03:25.843 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:03:25.844 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:03:25.844 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:03:25.844 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:03:25.844 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.184181+0000", 2023-07-02T14:03:25.844 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:03:25.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.844 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:03:25.845 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:03:25.845 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:03:25.845 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:03:25.845 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:03:25.845 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:03:25.845 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.274854+0000", 2023-07-02T14:03:25.845 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:03:25.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.846 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:03:25.846 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:03:25.846 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:03:25.846 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:03:25.846 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:03:25.846 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:03:25.846 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.186217+0000", 2023-07-02T14:03:25.847 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:03:25.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.847 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:03:25.847 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:03:25.847 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:03:25.847 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:03:25.847 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:03:25.848 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:03:25.848 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.247442+0000", 2023-07-02T14:03:25.848 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:03:25.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.848 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:03:25.848 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:03:25.848 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:03:25.849 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:03:25.849 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:03:25.849 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 65536, 2023-07-02T14:03:25.849 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.220075+0000", 2023-07-02T14:03:25.849 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:03:25.849 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.849 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:03:25.850 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap": [ 2023-07-02T14:03:25.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.850 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.0", 2023-07-02T14:03:25.850 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:03:25.850 INFO:teuthology.orchestra.run.smithi016.stdout: 4, 2023-07-02T14:03:25.850 INFO:teuthology.orchestra.run.smithi016.stdout: 1 2023-07-02T14:03:25.850 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.851 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.2", 2023-07-02T14:03:25.851 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:03:25.851 INFO:teuthology.orchestra.run.smithi016.stdout: 1, 2023-07-02T14:03:25.851 INFO:teuthology.orchestra.run.smithi016.stdout: 5 2023-07-02T14:03:25.851 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:25.852 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.4", 2023-07-02T14:03:25.852 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:03:25.852 INFO:teuthology.orchestra.run.smithi016.stdout: 5, 2023-07-02T14:03:25.852 INFO:teuthology.orchestra.run.smithi016.stdout: 0 2023-07-02T14:03:25.852 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:25.852 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.852 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:03:25.852 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_items": [], 2023-07-02T14:03:25.853 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_temp": [], 2023-07-02T14:03:25.853 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_temp": [], 2023-07-02T14:03:25.853 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklist": {}, 2023-07-02T14:03:25.853 INFO:teuthology.orchestra.run.smithi016.stdout: "range_blocklist": {}, 2023-07-02T14:03:25.853 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profiles": { 2023-07-02T14:03:25.853 INFO:teuthology.orchestra.run.smithi016.stdout: "default": { 2023-07-02T14:03:25.853 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2023-07-02T14:03:25.853 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "2", 2023-07-02T14:03:25.854 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "1", 2023-07-02T14:03:25.854 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "jerasure", 2023-07-02T14:03:25.854 INFO:teuthology.orchestra.run.smithi016.stdout: "ruleset-failure-domain": "osd", 2023-07-02T14:03:25.854 INFO:teuthology.orchestra.run.smithi016.stdout: "technique": "reed_sol_van" 2023-07-02T14:03:25.854 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:25.854 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps_queue": [], 2023-07-02T14:03:25.854 INFO:teuthology.orchestra.run.smithi016.stdout: "new_removed_snaps": [], 2023-07-02T14:03:25.855 INFO:teuthology.orchestra.run.smithi016.stdout: "new_purged_snaps": [], 2023-07-02T14:03:25.855 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_node_flags": {}, 2023-07-02T14:03:25.855 INFO:teuthology.orchestra.run.smithi016.stdout: "device_class_flags": {}, 2023-07-02T14:03:25.855 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode": { 2023-07-02T14:03:25.855 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_enabled": false, 2023-07-02T14:03:25.855 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_bucket_count": 0, 2023-07-02T14:03:25.855 INFO:teuthology.orchestra.run.smithi016.stdout: "degraded_stretch_mode": 0, 2023-07-02T14:03:25.855 INFO:teuthology.orchestra.run.smithi016.stdout: "recovering_stretch_mode": 0, 2023-07-02T14:03:25.856 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_bucket": 0 2023-07-02T14:03:25.856 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:25.856 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:25.857 INFO:tasks.thrashosds.thrasher:j is {'epoch': 69, 'fsid': '9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2', 'created': '2023-07-02T14:01:37.298060+0000', 'modified': '2023-07-02T14:03:24.352048+0000', 'last_up_change': '2023-07-02T14:01:45.271760+0000', 'last_in_change': '2023-07-02T14:03:09.309538+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-07-02T14:01:45.323455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-02T14:01:47.448897+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 10, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '43c446fe-c811-4ee3-99e8-d7452edfc1c1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6800', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6801', 'nonce': 83359}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6802', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6803', 'nonce': 83359}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6806', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6807', 'nonce': 83359}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6804', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6805', 'nonce': 83359}]}, 'public_addr': '172.21.15.16:6801/83359', 'cluster_addr': '172.21.15.16:6803/83359', 'heartbeat_back_addr': '172.21.15.16:6807/83359', 'heartbeat_front_addr': '172.21.15.16:6805/83359', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 63, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6808', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6809', 'nonce': 83360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6810', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6811', 'nonce': 83360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6814', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6815', 'nonce': 83360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6812', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6813', 'nonce': 83360}]}, 'public_addr': '172.21.15.16:6809/83360', 'cluster_addr': '172.21.15.16:6811/83360', 'heartbeat_back_addr': '172.21.15.16:6815/83360', 'heartbeat_front_addr': '172.21.15.16:6813/83360', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '99f8ea0a-5a71-4abe-b087-8e891fd11d1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6816', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6817', 'nonce': 83361}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6818', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6819', 'nonce': 83361}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6822', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6823', 'nonce': 83361}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6820', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6821', 'nonce': 83361}]}, 'public_addr': '172.21.15.16:6817/83361', 'cluster_addr': '172.21.15.16:6819/83361', 'heartbeat_back_addr': '172.21.15.16:6823/83361', 'heartbeat_front_addr': '172.21.15.16:6821/83361', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd285b4d5-0bef-421a-b84a-943e485b10c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 63, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6808', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6809', 'nonce': 83350}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6810', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6811', 'nonce': 83350}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6814', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6815', 'nonce': 83350}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6812', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6813', 'nonce': 83350}]}, 'public_addr': '172.21.15.22:6809/83350', 'cluster_addr': '172.21.15.22:6811/83350', 'heartbeat_back_addr': '172.21.15.22:6815/83350', 'heartbeat_front_addr': '172.21.15.22:6813/83350', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7ba0b0db-e407-4c63-8278-aab09f3082c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.23297119140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6800', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6801', 'nonce': 83351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6802', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6803', 'nonce': 83351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6806', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6807', 'nonce': 83351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6804', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6805', 'nonce': 83351}]}, 'public_addr': '172.21.15.22:6801/83351', 'cluster_addr': '172.21.15.22:6803/83351', 'heartbeat_back_addr': '172.21.15.22:6807/83351', 'heartbeat_front_addr': '172.21.15.22:6805/83351', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '29663bfa-f6fe-4ce7-b45a-a9b7b4424c42', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6816', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6817', 'nonce': 83352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6818', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6819', 'nonce': 83352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6822', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6823', 'nonce': 83352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6820', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6821', 'nonce': 83352}]}, 'public_addr': '172.21.15.22:6817/83352', 'cluster_addr': '172.21.15.22:6819/83352', 'heartbeat_back_addr': '172.21.15.22:6823/83352', 'heartbeat_front_addr': '172.21.15.22:6821/83352', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.001178+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.184181+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.274854+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.186217+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.247442+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.220075+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.0', 'osds': [4, 1]}, {'pgid': '2.2', 'osds': [1, 5]}, {'pgid': '2.4', 'osds': [5, 0]}], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-07-02T14:03:25.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:03:25.858 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:25.858+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:25.866 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:03:25.959 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:25.959+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:25.982 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:25.982 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:25.983 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:03:25.983 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:25.993 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:03:26.042 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:26.059 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:26.059+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:26.061 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:26.079 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:26.096 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:26.115 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:26.118 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:26.119 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:26.129 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:03:26.133 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:26.134 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:03:26.135 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:03:26.135 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:03:26.135 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:03:26.135 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:03:26.135 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:03:26.170 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:03:26.170 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:26.169+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:26.171 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:03:26.183 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2023-07-02T14:03:24.354814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1270225091,"num_objects":355,"num_object_clones":44,"num_object_copies":710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":355,"num_whiteouts":0,"num_read":3841,"num_read_kb":1273996,"num_write":3576,"num_write_kb":1224081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":484858461,"num_keys_recovered":32,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3302,"ondisk_log_size":3302,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":2520708,"kb_used_data":2450460,"kb_used_omap":0,"kb_used_meta":70208,"kb_avail":469338492,"statfs":{"total":483183820800,"available":480602615808,"internally_reserved":0,"allocated":2509271040,"data_stored":2629204175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":71892992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17412258,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":530,"num_read_kb":1125200,"num_write":48,"num_write_kb":16294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.012423"},"pg_stats":[{"pgid":"2.7","version":"66'457","reported_seq":1019,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-07-02T14:03:19.359647+0000","last_change":"2023-07-02T14:03:14.075264+0000","last_active":"2023-07-02T14:03:19.359647+0000","last_peered":"2023-07-02T14:03:19.359647+0000","last_clean":"2023-07-02T14:03:19.359647+0000","last_became_active":"2023-07-02T14:03:14.073933+0000","last_became_peered":"2023-07-02T14:03:14.073933+0000","last_unstale":"2023-07-02T14:03:19.359647+0000","last_undegraded":"2023-07-02T14:03:19.359647+0000","last_fullsized":"2023-07-02T14:03:19.359647+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":457,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":168625510,"num_objects":51,"num_object_clones":7,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":1136,"num_read_kb":171087,"num_write":426,"num_write_kb":163163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":161921416,"num_keys_recovered":28,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"66'383","reported_seq":623,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-07-02T14:03:19.343787+0000","last_change":"2023-07-02T14:02:39.062139+0000","last_active":"2023-07-02T14:03:19.343787+0000","last_peered":"2023-07-02T14:03:19.343787+0000","last_clean":"2023-07-02T14:03:19.343787+0000","last_became_active":"2023-07-02T14:01:49.293782+0000","last_became_peered":"2023-07-02T14:01:49.293782+0000","last_unstale":"2023-07-02T14:03:19.343787+0000","last_undegraded":"2023-07-02T14:03:19.343787+0000","last_fullsized":"2023-07-02T14:03:19.343787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":383,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":162312354,"num_objects":48,"num_object_clones":9,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":185,"num_read_kb":167784,"num_write":374,"num_write_kb":156885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"64'362","reported_seq":543,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-07-02T14:03:18.502961+0000","last_change":"2023-07-02T14:02:39.069176+0000","last_active":"2023-07-02T14:03:18.502961+0000","last_peered":"2023-07-02T14:03:18.502961+0000","last_clean":"2023-07-02T14:03:18.502961+0000","last_became_active":"2023-07-02T14:01:49.292790+0000","last_became_peered":"2023-07-02T14:01:49.292790+0000","last_unstale":"2023-07-02T14:03:18.502961+0000","last_undegraded":"2023-07-02T14:03:18.502961+0000","last_fullsized":"2023-07-02T14:03:18.502961+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":362,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165896192,"num_objects":47,"num_object_clones":8,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":127,"num_read_kb":194053,"num_write":354,"num_write_kb":154265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"66'760","reported_seq":931,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-07-02T14:03:19.345978+0000","last_change":"2023-07-02T14:02:50.069577+0000","last_active":"2023-07-02T14:03:19.345978+0000","last_peered":"2023-07-02T14:03:19.345978+0000","last_clean":"2023-07-02T14:03:19.345978+0000","last_became_active":"2023-07-02T14:02:50.068789+0000","last_became_peered":"2023-07-02T14:02:50.068789+0000","last_unstale":"2023-07-02T14:03:19.345978+0000","last_undegraded":"2023-07-02T14:03:19.345978+0000","last_fullsized":"2023-07-02T14:03:19.345978+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":760,"ondisk_log_size":760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":173015542,"num_objects":50,"num_object_clones":5,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":2179,"num_read_kb":175518,"num_write":1097,"num_write_kb":167738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8523954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-07-02T14:03:17.333764+0000","last_change":"2023-07-02T14:01:47.288970+0000","last_active":"2023-07-02T14:03:17.333764+0000","last_peered":"2023-07-02T14:03:17.333764+0000","last_clean":"2023-07-02T14:03:17.333764+0000","last_became_active":"2023-07-02T14:01:47.288610+0000","last_became_peered":"2023-07-02T14:01:47.288610+0000","last_unstale":"2023-07-02T14:03:17.333764+0000","last_undegraded":"2023-07-02T14:03:17.333764+0000","last_fullsized":"2023-07-02T14:03:17.333764+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"64'376","reported_seq":577,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-07-02T14:03:18.465223+0000","last_change":"2023-07-02T14:03:13.072471+0000","last_active":"2023-07-02T14:03:18.465223+0000","last_peered":"2023-07-02T14:03:18.465223+0000","last_clean":"2023-07-02T14:03:18.465223+0000","last_became_active":"2023-07-02T14:03:13.071863+0000","last_became_peered":"2023-07-02T14:03:13.071863+0000","last_unstale":"2023-07-02T14:03:18.465223+0000","last_undegraded":"2023-07-02T14:03:18.465223+0000","last_fullsized":"2023-07-02T14:03:18.465223+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":376,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":181436432,"num_objects":49,"num_object_clones":5,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":56,"num_read_kb":178164,"num_write":371,"num_write_kb":175725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":129863696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"64'404","reported_seq":517,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-07-02T14:03:18.508555+0000","last_change":"2023-07-02T14:02:39.061498+0000","last_active":"2023-07-02T14:03:18.508555+0000","last_peered":"2023-07-02T14:03:18.508555+0000","last_clean":"2023-07-02T14:03:18.508555+0000","last_became_active":"2023-07-02T14:01:49.294964+0000","last_became_peered":"2023-07-02T14:01:49.294964+0000","last_unstale":"2023-07-02T14:03:18.508555+0000","last_undegraded":"2023-07-02T14:03:18.508555+0000","last_fullsized":"2023-07-02T14:03:18.508555+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":404,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":173809826,"num_objects":43,"num_object_clones":3,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":55,"num_read_kb":151136,"num_write":401,"num_write_kb":169177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"64'248","reported_seq":430,"reported_epoch":66,"state":"active+clean","last_fresh":"2023-07-02T14:03:19.352801+0000","last_change":"2023-07-02T14:03:13.073775+0000","last_active":"2023-07-02T14:03:19.352801+0000","last_peered":"2023-07-02T14:03:19.352801+0000","last_clean":"2023-07-02T14:03:19.352801+0000","last_became_active":"2023-07-02T14:03:13.073206+0000","last_became_peered":"2023-07-02T14:03:13.073206+0000","last_unstale":"2023-07-02T14:03:19.352801+0000","last_undegraded":"2023-07-02T14:03:19.352801+0000","last_fullsized":"2023-07-02T14:03:19.352801+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":248,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":115408915,"num_objects":30,"num_object_clones":2,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":61,"num_read_kb":106782,"num_write":246,"num_write_kb":112322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":96600083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"64'312","reported_seq":432,"reported_epoch":65,"state":"active+clean","last_fresh":"2023-07-02T14:03:18.497056+0000","last_change":"2023-07-02T14:03:14.074306+0000","last_active":"2023-07-02T14:03:18.497056+0000","last_peered":"2023-07-02T14:03:18.497056+0000","last_clean":"2023-07-02T14:03:18.497056+0000","last_became_active":"2023-07-02T14:03:14.073735+0000","last_became_peered":"2023-07-02T14:03:14.073735+0000","last_unstale":"2023-07-02T14:03:18.497056+0000","last_undegraded":"2023-07-02T14:03:18.497056+0000","last_fullsized":"2023-07-02T14:03:18.497056+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":312,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":129720320,"num_objects":37,"num_object_clones":5,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":42,"num_read_kb":129472,"num_write":307,"num_write_kb":124806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":87949312,"num_keys_recovered":4,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":1270225091,"num_objects":355,"num_object_clones":44,"num_object_copies":710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":355,"num_whiteouts":0,"num_read":3841,"num_read_kb":1273996,"num_write":3576,"num_write_kb":1224081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":484858461,"num_keys_recovered":32,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2505441280,"data_stored":2627768710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3302,"ondisk_log_size":3302,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640277,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640277,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":345048,"kb_used_data":330960,"kb_used_omap":0,"kb_used_meta":14080,"kb_avail":94026792,"statfs":{"total":96636764160,"available":96283435008,"internally_reserved":0,"allocated":338903040,"data_stored":355657937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14417920},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.36399999999999999}]},{"osd":1,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.34200000000000003}]},{"osd":2,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.38},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.39600000000000002}]},{"osd":3,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.29399999999999998}]},{"osd":5,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.309}]}]},{"osd":3,"up_from":11,"seq":47244640277,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":488220,"kb_used_data":475028,"kb_used_omap":0,"kb_used_meta":13184,"kb_avail":93883620,"statfs":{"total":96636764160,"available":96136826880,"internally_reserved":0,"allocated":486428672,"data_stored":525027197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":13500416},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.35799999999999998}]},{"osd":1,"last update":"Sun Jul 2 14:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.374}]},{"osd":2,"last update":"Sun Jul 2 14:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.42199999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.432},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.41699999999999998}]},{"osd":5,"last update":"Sun Jul 2 14:02:48 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.47499999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640277,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":357940,"kb_used_data":345644,"kb_used_omap":0,"kb_used_meta":12288,"kb_avail":94013900,"statfs":{"total":96636764160,"available":96270233600,"internally_reserved":0,"allocated":353939456,"data_stored":358261287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":12582912},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.32000000000000001}]},{"osd":1,"last update":"Sun Jul 2 14:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.254}]},{"osd":3,"last update":"Sun Jul 2 14:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.39300000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.51200000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.433},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.52500000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640277,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403860,"kb_used_data":392140,"kb_used_omap":0,"kb_used_meta":11712,"kb_avail":93967980,"statfs":{"total":96636764160,"available":96223211520,"internally_reserved":0,"allocated":401551360,"data_stored":422994312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":11993088},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 2 14:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.432,"5min":0.432,"15min":0.432},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.30399999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.46300000000000002}]},{"osd":3,"last update":"Sun Jul 2 14:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.434}]},{"osd":4,"last update":"Sun Jul 2 14:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.48599999999999999}]},{"osd":5,"last update":"Sun Jul 2 14:02:45 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.51200000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640277,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":925640,"kb_used_data":906688,"kb_used_omap":0,"kb_used_meta":18944,"kb_avail":93446200,"statfs":{"total":96636764160,"available":95688908800,"internally_reserved":0,"allocated":928448512,"data_stored":967263442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19398656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.496,"5min":0.496,"15min":0.496},"last":0.42399999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.439},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.48799999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.624},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.54200000000000004}]},{"osd":4,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.72599999999999998}]},{"osd":5,"last update":"Sun Jul 2 14:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.71799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":400785408,"data_stored":422707219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":927682560,"data_stored":966976349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":353173504,"data_stored":357974194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":485662720,"data_stored":524740104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":338137088,"data_stored":355370844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:03:26.184 INFO:tasks.thrashosds.thrasher:Setting 1.0 to [0, 3] 2023-07-02T14:03:26.184 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.0', '0', '3'] 2023-07-02T14:03:26.185 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 1.0 0 3 2023-07-02T14:03:26.254 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:26.255 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:26.261 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:26.260+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:26.265 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:03:26.362 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:26.362+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:26.364 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:03:26.379 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:03:26.384 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 2 2023-07-02T14:03:26.434 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:26.434 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:26.444 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:03:26.463 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:26.463+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:26.563 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:26.563+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:26.567 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:26.568 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:26.578 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:03:26.664 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:26.664+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:26.702 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:26.702 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:26.712 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:03:26.765 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:26.764+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:26.837 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:26.837 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:26.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:03:26.890 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:26.890+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:26.966 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:26.965+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:26.970 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:26.971 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:26.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:03:27.067 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:27.066+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:27.144 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:27.145 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:27.154 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:03:27.198 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:27.198+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:27.269 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:27.269+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:27.278 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:27.278 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:27.288 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:03:27.365 INFO:teuthology.orchestra.run.smithi016.stderr:set 1.0 pg_upmap mapping to [0,3] 2023-07-02T14:03:27.370 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:27.370+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:27.455 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:27.456 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:27.466 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:03:27.509 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:27.509+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:27.573 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:27.571+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:27.590 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:27.590 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:27.600 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:03:27.684 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:27.684+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:27.768 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:27.768 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:27.773 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:27.772+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:27.778 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:03:27.874 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:27.874+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:27.942 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:27.942 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:27.953 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:03:27.994 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:27.994+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:28.076 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:28.076+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:28.117 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:28.117 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:28.128 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:03:28.146 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:03:28.212 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:28.212+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:28.264 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:28.264 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:28.274 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:03:28.307 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:28.318 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:28.317+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:28.324 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:28.341 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:28.359 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:28.378 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:28.378+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:28.379 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:28.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:03:28.385 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap7 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap8 b3fbb028-e267-4d25-a750-93ba18a2316a@snap8.diff 2023-07-02T14:03:28.397 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:28.399 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:03:28.399 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:03:28.399 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:03:28.399 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:03:28.399 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:03:28.399 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:03:28.400 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:28.400 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:28.400 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:28.400 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:28.401 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:28.411 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:03:28.428 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 100% complete...done. 2023-07-02T14:03:28.431 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:03:28.431 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=9 2023-07-02T14:03:28.431 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap9 2023-07-02T14:03:28.494 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:28.493+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:28.579 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:28.579 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:28.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:28.580 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:28.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.580 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:322 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:28.580 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:28.580 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.277696548000002, 2023-07-02T14:03:28.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.03676041, 2023-07-02T14:03:28.580 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:28.581 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:28.581 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:28.581 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:28.581 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:28.581 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 322 2023-07-02T14:03:28.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.581 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:28.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:28.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:28.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:28.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:28.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:28.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:28.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:28.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298819+0000", 2023-07-02T14:03:28.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8068e-05 2023-07-02T14:03:28.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:28.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298877+0000", 2023-07-02T14:03:28.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3339999999999998e-06 2023-07-02T14:03:28.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:28.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298883+0000", 2023-07-02T14:03:28.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1030000000000004e-06 2023-07-02T14:03:28.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:28.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298887+0000", 2023-07-02T14:03:28.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023640230000000002 2023-07-02T14:03:28.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322528+0000", 2023-07-02T14:03:28.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8424999999999994e-05 2023-07-02T14:03:28.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:28.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322616+0000", 2023-07-02T14:03:28.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116815 2023-07-02T14:03:28.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:28.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322733+0000", 2023-07-02T14:03:28.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048252629999999998 2023-07-02T14:03:28.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:28.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327558+0000", 2023-07-02T14:03:28.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078965240000000003 2023-07-02T14:03:28.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:28.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335455+0000", 2023-07-02T14:03:28.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0511999999999999e-05 2023-07-02T14:03:28.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:28.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335515+0000", 2023-07-02T14:03:28.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4789000000000006e-05 2023-07-02T14:03:28.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:28.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335580+0000", 2023-07-02T14:03:28.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.03676041 2023-07-02T14:03:28.590 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.590 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.590 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.591 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:323 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:28.591 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:28.591 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.277625168, 2023-07-02T14:03:28.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036875044000000003, 2023-07-02T14:03:28.591 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:28.591 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:28.591 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:28.591 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:28.592 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:28.592 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 323 2023-07-02T14:03:28.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.592 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:28.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:28.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:28.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:28.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:28.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:28.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:28.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:28.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298890+0000", 2023-07-02T14:03:28.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7277000000000002e-05 2023-07-02T14:03:28.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:28.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298938+0000", 2023-07-02T14:03:28.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0495e-05 2023-07-02T14:03:28.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:28.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298948+0000", 2023-07-02T14:03:28.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6829999999999999e-06 2023-07-02T14:03:28.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:28.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298953+0000", 2023-07-02T14:03:28.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024271435000000001 2023-07-02T14:03:28.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323224+0000", 2023-07-02T14:03:28.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9266999999999998e-05 2023-07-02T14:03:28.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:28.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323304+0000", 2023-07-02T14:03:28.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000157021 2023-07-02T14:03:28.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:28.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323461+0000", 2023-07-02T14:03:28.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041036689999999999 2023-07-02T14:03:28.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:28.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327564+0000", 2023-07-02T14:03:28.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080753490000000008 2023-07-02T14:03:28.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:28.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335640+0000", 2023-07-02T14:03:28.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7398999999999994e-05 2023-07-02T14:03:28.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:28.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335727+0000", 2023-07-02T14:03:28.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8967000000000001e-05 2023-07-02T14:03:28.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:28.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335766+0000", 2023-07-02T14:03:28.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036875044000000003 2023-07-02T14:03:28.600 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.601 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.601 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.601 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:324 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:28.601 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:28.601 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.277557790000003, 2023-07-02T14:03:28.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036935673000000002, 2023-07-02T14:03:28.602 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:28.602 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:28.602 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:28.602 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:28.602 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:28.602 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 324 2023-07-02T14:03:28.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.603 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:28.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:28.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:28.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:28.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:28.604 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:28.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:28.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.604 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:28.604 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:28.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5519999999999997e-05 2023-07-02T14:03:28.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:28.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299013+0000", 2023-07-02T14:03:28.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0701e-05 2023-07-02T14:03:28.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:28.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299024+0000", 2023-07-02T14:03:28.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2379999999999999e-06 2023-07-02T14:03:28.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:28.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299028+0000", 2023-07-02T14:03:28.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024972768999999999 2023-07-02T14:03:28.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324001+0000", 2023-07-02T14:03:28.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1178000000000002e-05 2023-07-02T14:03:28.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:28.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324072+0000", 2023-07-02T14:03:28.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000166119 2023-07-02T14:03:28.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:28.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324238+0000", 2023-07-02T14:03:28.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033314210000000002 2023-07-02T14:03:28.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:28.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327570+0000", 2023-07-02T14:03:28.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0082393369999999994 2023-07-02T14:03:28.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:28.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335809+0000", 2023-07-02T14:03:28.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7071999999999999e-05 2023-07-02T14:03:28.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:28.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335856+0000", 2023-07-02T14:03:28.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7926999999999998e-05 2023-07-02T14:03:28.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:28.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335894+0000", 2023-07-02T14:03:28.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036935673000000002 2023-07-02T14:03:28.611 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.611 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.611 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.612 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:325 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:28.612 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:28.612 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.277482462999998, 2023-07-02T14:03:28.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036990838999999998, 2023-07-02T14:03:28.612 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:28.612 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:28.612 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:28.613 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:28.613 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:28.613 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 325 2023-07-02T14:03:28.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.613 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:28.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:28.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:28.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:28.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:28.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:28.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:28.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:28.615 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299033+0000", 2023-07-02T14:03:28.615 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107619 2023-07-02T14:03:28.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:28.615 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299141+0000", 2023-07-02T14:03:28.615 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3497e-05 2023-07-02T14:03:28.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.616 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.616 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:28.616 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299154+0000", 2023-07-02T14:03:28.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4669e-05 2023-07-02T14:03:28.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.616 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:28.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299169+0000", 2023-07-02T14:03:28.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025630159999999999 2023-07-02T14:03:28.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324799+0000", 2023-07-02T14:03:28.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6953000000000007e-05 2023-07-02T14:03:28.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:28.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324876+0000", 2023-07-02T14:03:28.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016346199999999999 2023-07-02T14:03:28.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:28.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325039+0000", 2023-07-02T14:03:28.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003566624 2023-07-02T14:03:28.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:28.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328606+0000", 2023-07-02T14:03:28.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073295540000000003 2023-07-02T14:03:28.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:28.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335936+0000", 2023-07-02T14:03:28.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0044999999999999e-05 2023-07-02T14:03:28.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:28.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335986+0000", 2023-07-02T14:03:28.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8872999999999999e-05 2023-07-02T14:03:28.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:28.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336024+0000", 2023-07-02T14:03:28.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036990838999999998 2023-07-02T14:03:28.621 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.622 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.622 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.622 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:326 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:28.622 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:28.622 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.277340535, 2023-07-02T14:03:28.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036975239999999999, 2023-07-02T14:03:28.623 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:28.623 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:28.623 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:28.623 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:28.623 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:28.623 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 326 2023-07-02T14:03:28.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.624 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:28.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:28.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:28.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:28.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:28.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:28.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:28.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.625 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:28.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299175+0000", 2023-07-02T14:03:28.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4782000000000003e-05 2023-07-02T14:03:28.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:28.626 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299250+0000", 2023-07-02T14:03:28.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.277e-05 2023-07-02T14:03:28.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:28.626 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299262+0000", 2023-07-02T14:03:28.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1460000000000001e-06 2023-07-02T14:03:28.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:28.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299268+0000", 2023-07-02T14:03:28.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026158098000000001 2023-07-02T14:03:28.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325427+0000", 2023-07-02T14:03:28.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4845999999999995e-05 2023-07-02T14:03:28.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:28.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325501+0000", 2023-07-02T14:03:28.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015890699999999999 2023-07-02T14:03:28.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:28.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325660+0000", 2023-07-02T14:03:28.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002952114 2023-07-02T14:03:28.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:28.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328612+0000", 2023-07-02T14:03:28.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074491949999999996 2023-07-02T14:03:28.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:28.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336062+0000", 2023-07-02T14:03:28.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7376999999999997e-05 2023-07-02T14:03:28.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:28.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336109+0000", 2023-07-02T14:03:28.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1814000000000003e-05 2023-07-02T14:03:28.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:28.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336151+0000", 2023-07-02T14:03:28.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036975239999999999 2023-07-02T14:03:28.632 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.632 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.632 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.633 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:327 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 589824~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:28.633 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:28.633 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.277241273999998, 2023-07-02T14:03:28.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037234253000000002, 2023-07-02T14:03:28.633 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:28.633 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:28.633 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:28.633 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:28.634 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:28.634 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 327 2023-07-02T14:03:28.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.634 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:28.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:28.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:28.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:28.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:28.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:28.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:28.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:28.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299274+0000", 2023-07-02T14:03:28.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2213999999999997e-05 2023-07-02T14:03:28.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:28.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299346+0000", 2023-07-02T14:03:28.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1816e-05 2023-07-02T14:03:28.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:28.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299358+0000", 2023-07-02T14:03:28.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2626000000000001e-05 2023-07-02T14:03:28.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:28.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299371+0000", 2023-07-02T14:03:28.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026916853000000001 2023-07-02T14:03:28.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326288+0000", 2023-07-02T14:03:28.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6909000000000004e-05 2023-07-02T14:03:28.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:28.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326375+0000", 2023-07-02T14:03:28.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015430399999999999 2023-07-02T14:03:28.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:28.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326529+0000", 2023-07-02T14:03:28.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096652100000000005 2023-07-02T14:03:28.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:28.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336194+0000", 2023-07-02T14:03:28.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115299 2023-07-02T14:03:28.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:28.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336309+0000", 2023-07-02T14:03:28.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015981200000000001 2023-07-02T14:03:28.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:28.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336469+0000", 2023-07-02T14:03:28.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9867999999999998e-05 2023-07-02T14:03:28.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:28.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336509+0000", 2023-07-02T14:03:28.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037234253000000002 2023-07-02T14:03:28.642 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.642 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.642 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.643 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:328 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:28.643 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:28.643 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.277137685, 2023-07-02T14:03:28.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037216610999999997, 2023-07-02T14:03:28.643 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:28.644 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:28.644 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:28.644 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:28.644 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:28.644 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 328 2023-07-02T14:03:28.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.644 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:28.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:28.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:28.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:28.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:28.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:28.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:28.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:28.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:28.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1412e-05 2023-07-02T14:03:28.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:28.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299449+0000", 2023-07-02T14:03:28.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0219000000000001e-05 2023-07-02T14:03:28.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:28.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299469+0000", 2023-07-02T14:03:28.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9480000000000003e-06 2023-07-02T14:03:28.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:28.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299479+0000", 2023-07-02T14:03:28.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027444344999999998 2023-07-02T14:03:28.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326924+0000", 2023-07-02T14:03:28.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5165000000000001e-05 2023-07-02T14:03:28.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:28.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327019+0000", 2023-07-02T14:03:28.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019113499999999999 2023-07-02T14:03:28.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:28.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327210+0000", 2023-07-02T14:03:28.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090197449999999992 2023-07-02T14:03:28.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:28.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336230+0000", 2023-07-02T14:03:28.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028764399999999999 2023-07-02T14:03:28.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:28.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336517+0000", 2023-07-02T14:03:28.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1981999999999999e-05 2023-07-02T14:03:28.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:28.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336559+0000", 2023-07-02T14:03:28.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:03:28.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:28.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336595+0000", 2023-07-02T14:03:28.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037216610999999997 2023-07-02T14:03:28.653 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.653 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.653 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.654 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:329 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:28.654 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:28.654 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.277027768000004, 2023-07-02T14:03:28.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037191142000000003, 2023-07-02T14:03:28.654 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:28.654 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:28.654 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:28.654 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:28.655 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:28.655 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 329 2023-07-02T14:03:28.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.655 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:28.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:28.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:28.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:28.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:28.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:28.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:28.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:28.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299487+0000", 2023-07-02T14:03:28.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7871000000000001e-05 2023-07-02T14:03:28.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:28.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299555+0000", 2023-07-02T14:03:28.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0446e-05 2023-07-02T14:03:28.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:28.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299566+0000", 2023-07-02T14:03:28.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7939999999999999e-06 2023-07-02T14:03:28.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:28.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299572+0000", 2023-07-02T14:03:28.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028034019 2023-07-02T14:03:28.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327606+0000", 2023-07-02T14:03:28.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1967999999999999e-05 2023-07-02T14:03:28.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:28.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327678+0000", 2023-07-02T14:03:28.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022216900000000001 2023-07-02T14:03:28.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:28.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327900+0000", 2023-07-02T14:03:28.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084016899999999999 2023-07-02T14:03:28.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:28.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336301+0000", 2023-07-02T14:03:28.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00030122800000000001 2023-07-02T14:03:28.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:28.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336603+0000", 2023-07-02T14:03:28.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0380000000000003e-05 2023-07-02T14:03:28.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:28.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336643+0000", 2023-07-02T14:03:28.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6455999999999998e-05 2023-07-02T14:03:28.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:28.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336679+0000", 2023-07-02T14:03:28.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037191142000000003 2023-07-02T14:03:28.663 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.663 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.664 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.664 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:330 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:28.664 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:28.664 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.276936997999996, 2023-07-02T14:03:28.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037849305, 2023-07-02T14:03:28.664 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:28.665 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:28.665 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:28.665 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:28.665 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:28.665 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 330 2023-07-02T14:03:28.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.665 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:28.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:28.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:28.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:28.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:28.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:28.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:28.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:28.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299578+0000", 2023-07-02T14:03:28.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4708000000000005e-05 2023-07-02T14:03:28.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:28.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299643+0000", 2023-07-02T14:03:28.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7309999999999999e-06 2023-07-02T14:03:28.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:28.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299652+0000", 2023-07-02T14:03:28.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7989999999999999e-06 2023-07-02T14:03:28.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:28.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299658+0000", 2023-07-02T14:03:28.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028981664000000001 2023-07-02T14:03:28.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328639+0000", 2023-07-02T14:03:28.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0939000000000006e-05 2023-07-02T14:03:28.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:28.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328710+0000", 2023-07-02T14:03:28.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000142624 2023-07-02T14:03:28.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:28.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328853+0000", 2023-07-02T14:03:28.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078374640000000006 2023-07-02T14:03:28.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:28.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336690+0000", 2023-07-02T14:03:28.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00068453000000000003 2023-07-02T14:03:28.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:28.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337375+0000", 2023-07-02T14:03:28.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0667999999999998e-05 2023-07-02T14:03:28.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:28.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337406+0000", 2023-07-02T14:03:28.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2858999999999999e-05 2023-07-02T14:03:28.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:28.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337428+0000", 2023-07-02T14:03:28.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037849305 2023-07-02T14:03:28.674 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.674 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.674 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.674 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:331 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:28.674 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:28.675 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.276853136, 2023-07-02T14:03:28.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037870332, 2023-07-02T14:03:28.675 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:28.675 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:28.675 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:28.675 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:28.675 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:28.675 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 331 2023-07-02T14:03:28.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.676 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:28.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:28.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:28.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:28.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:28.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:28.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:28.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:28.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299662+0000", 2023-07-02T14:03:28.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2252e-05 2023-07-02T14:03:28.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:28.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299725+0000", 2023-07-02T14:03:28.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8440000000000004e-06 2023-07-02T14:03:28.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:28.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299733+0000", 2023-07-02T14:03:28.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6759999999999998e-06 2023-07-02T14:03:28.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:28.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299738+0000", 2023-07-02T14:03:28.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029459387 2023-07-02T14:03:28.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329197+0000", 2023-07-02T14:03:28.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0087999999999997e-05 2023-07-02T14:03:28.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:28.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329268+0000", 2023-07-02T14:03:28.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014405700000000001 2023-07-02T14:03:28.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:28.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329412+0000", 2023-07-02T14:03:28.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072842100000000002 2023-07-02T14:03:28.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:28.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336696+0000", 2023-07-02T14:03:28.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00076275300000000002 2023-07-02T14:03:28.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:28.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337459+0000", 2023-07-02T14:03:28.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5988000000000002e-05 2023-07-02T14:03:28.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:28.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337495+0000", 2023-07-02T14:03:28.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8686000000000002e-05 2023-07-02T14:03:28.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:28.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337533+0000", 2023-07-02T14:03:28.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037870332 2023-07-02T14:03:28.684 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.684 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.684 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.685 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:332 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:28.685 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:28.685 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.276773212000002, 2023-07-02T14:03:28.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037892506999999999, 2023-07-02T14:03:28.685 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:28.685 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:28.686 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:28.686 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:28.686 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:28.686 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 332 2023-07-02T14:03:28.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.686 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:28.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:28.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:28.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:28.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:28.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:28.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:28.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:28.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299742+0000", 2023-07-02T14:03:28.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1366999999999997e-05 2023-07-02T14:03:28.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:28.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299804+0000", 2023-07-02T14:03:28.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9319999999999996e-06 2023-07-02T14:03:28.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:28.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299812+0000", 2023-07-02T14:03:28.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4359999999999999e-06 2023-07-02T14:03:28.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:28.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299818+0000", 2023-07-02T14:03:28.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029946144000000001 2023-07-02T14:03:28.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329764+0000", 2023-07-02T14:03:28.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000164828 2023-07-02T14:03:28.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:28.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329929+0000", 2023-07-02T14:03:28.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016139400000000001 2023-07-02T14:03:28.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:28.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330090+0000", 2023-07-02T14:03:28.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066134009999999997 2023-07-02T14:03:28.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:28.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336704+0000", 2023-07-02T14:03:28.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00086955400000000003 2023-07-02T14:03:28.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:28.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337573+0000", 2023-07-02T14:03:28.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8447000000000001e-05 2023-07-02T14:03:28.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:28.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337602+0000", 2023-07-02T14:03:28.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.362e-05 2023-07-02T14:03:28.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:28.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337635+0000", 2023-07-02T14:03:28.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037892506999999999 2023-07-02T14:03:28.694 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.695 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.695 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.695 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:333 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:28.695 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:28.695 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.276693131000002, 2023-07-02T14:03:28.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037932598999999997, 2023-07-02T14:03:28.696 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:28.696 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:28.696 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:28.696 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:28.696 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:28.696 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 333 2023-07-02T14:03:28.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.697 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:28.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:28.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:28.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:28.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:28.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:28.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:28.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:28.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299822+0000", 2023-07-02T14:03:28.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0831000000000001e-05 2023-07-02T14:03:28.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:28.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299883+0000", 2023-07-02T14:03:28.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.322e-06 2023-07-02T14:03:28.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:28.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299888+0000", 2023-07-02T14:03:28.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3479999999999997e-06 2023-07-02T14:03:28.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:28.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299893+0000", 2023-07-02T14:03:28.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030831905999999999 2023-07-02T14:03:28.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330725+0000", 2023-07-02T14:03:28.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3560999999999996e-05 2023-07-02T14:03:28.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:28.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330798+0000", 2023-07-02T14:03:28.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015165999999999999 2023-07-02T14:03:28.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:28.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330950+0000", 2023-07-02T14:03:28.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057590319999999999 2023-07-02T14:03:28.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:28.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336709+0000", 2023-07-02T14:03:28.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00096987899999999999 2023-07-02T14:03:28.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:28.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337679+0000", 2023-07-02T14:03:28.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9699000000000001e-05 2023-07-02T14:03:28.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:28.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337719+0000", 2023-07-02T14:03:28.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6961000000000001e-05 2023-07-02T14:03:28.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:28.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337756+0000", 2023-07-02T14:03:28.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037932598999999997 2023-07-02T14:03:28.705 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.705 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.705 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.706 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:372 2.0 2:169af44a:::rbd_data.125de1449605.0000000000000106:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:28.706 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:03:28.706 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.273115806, 2023-07-02T14:03:28.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036066963, 2023-07-02T14:03:28.706 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:28.706 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:28.706 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:28.706 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:28.707 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:28.707 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 372 2023-07-02T14:03:28.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.707 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:28.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:28.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:03:28.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:28.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:28.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:03:28.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:28.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:28.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303400+0000", 2023-07-02T14:03:28.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1002000000000003e-05 2023-07-02T14:03:28.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:28.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303461+0000", 2023-07-02T14:03:28.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9306e-05 2023-07-02T14:03:28.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:28.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303490+0000", 2023-07-02T14:03:28.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5026e-05 2023-07-02T14:03:28.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:28.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.303505+0000", 2023-07-02T14:03:28.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033623129000000002 2023-07-02T14:03:28.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337128+0000", 2023-07-02T14:03:28.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6968000000000002e-05 2023-07-02T14:03:28.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:28.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337175+0000", 2023-07-02T14:03:28.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000101127 2023-07-02T14:03:28.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:28.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337276+0000", 2023-07-02T14:03:28.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017866920000000001 2023-07-02T14:03:28.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:28.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339063+0000", 2023-07-02T14:03:28.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00031459800000000003 2023-07-02T14:03:28.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:28.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339377+0000", 2023-07-02T14:03:28.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7145999999999999e-05 2023-07-02T14:03:28.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:28.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339425+0000", 2023-07-02T14:03:28.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2632000000000001e-05 2023-07-02T14:03:28.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:28.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339467+0000", 2023-07-02T14:03:28.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036066963 2023-07-02T14:03:28.715 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.716 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.716 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.716 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:416 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:28.716 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:28.716 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.258399752000003, 2023-07-02T14:03:28.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039242025, 2023-07-02T14:03:28.717 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:28.717 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:28.717 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:28.717 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:28.717 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:28.717 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:03:28.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.718 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:28.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:28.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:28.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:28.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:28.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:28.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:28.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:28.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:28.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.692e-06 2023-07-02T14:03:28.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:28.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318119+0000", 2023-07-02T14:03:28.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9790000000000005e-06 2023-07-02T14:03:28.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:28.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318129+0000", 2023-07-02T14:03:28.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3640000000000001e-06 2023-07-02T14:03:28.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:28.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318137+0000", 2023-07-02T14:03:28.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022607808 2023-07-02T14:03:28.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340745+0000", 2023-07-02T14:03:28.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0130999999999996e-05 2023-07-02T14:03:28.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340825+0000", 2023-07-02T14:03:28.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039581080000000001 2023-07-02T14:03:28.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344783+0000", 2023-07-02T14:03:28.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000006e-05 2023-07-02T14:03:28.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344854+0000", 2023-07-02T14:03:28.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052105550000000004 2023-07-02T14:03:28.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350065+0000", 2023-07-02T14:03:28.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0321999999999997e-05 2023-07-02T14:03:28.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350115+0000", 2023-07-02T14:03:28.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012378409999999999 2023-07-02T14:03:28.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351353+0000", 2023-07-02T14:03:28.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5942000000000001e-05 2023-07-02T14:03:28.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:28.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351409+0000", 2023-07-02T14:03:28.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035677700000000002 2023-07-02T14:03:28.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:28.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351766+0000", 2023-07-02T14:03:28.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000926719 2023-07-02T14:03:28.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:28.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352693+0000", 2023-07-02T14:03:28.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045652080000000003 2023-07-02T14:03:28.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:28.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357258+0000", 2023-07-02T14:03:28.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4717e-05 2023-07-02T14:03:28.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:28.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357293+0000", 2023-07-02T14:03:28.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5807000000000002e-05 2023-07-02T14:03:28.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:28.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357358+0000", 2023-07-02T14:03:28.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039242025 2023-07-02T14:03:28.730 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.730 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.730 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.730 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:417 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:28.730 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:28.731 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.258374314999998, 2023-07-02T14:03:28.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042407105, 2023-07-02T14:03:28.731 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:28.731 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:28.731 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:28.731 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:28.731 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:28.731 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 417 2023-07-02T14:03:28.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.732 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:28.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:28.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:28.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:28.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:28.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:28.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:28.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:28.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318141+0000", 2023-07-02T14:03:28.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7120000000000002e-06 2023-07-02T14:03:28.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:28.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318145+0000", 2023-07-02T14:03:28.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8869999999999997e-06 2023-07-02T14:03:28.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:28.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318151+0000", 2023-07-02T14:03:28.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9820000000000002e-06 2023-07-02T14:03:28.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:28.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318155+0000", 2023-07-02T14:03:28.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699133 2023-07-02T14:03:28.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340854+0000", 2023-07-02T14:03:28.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6849999999999998e-05 2023-07-02T14:03:28.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340931+0000", 2023-07-02T14:03:28.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039672140000000002 2023-07-02T14:03:28.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344898+0000", 2023-07-02T14:03:28.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5341999999999998e-05 2023-07-02T14:03:28.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344973+0000", 2023-07-02T14:03:28.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051819400000000003 2023-07-02T14:03:28.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350155+0000", 2023-07-02T14:03:28.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3261999999999999e-05 2023-07-02T14:03:28.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350209+0000", 2023-07-02T14:03:28.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00194097 2023-07-02T14:03:28.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352150+0000", 2023-07-02T14:03:28.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6851e-05 2023-07-02T14:03:28.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352226+0000", 2023-07-02T14:03:28.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051831109999999998 2023-07-02T14:03:28.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357409+0000", 2023-07-02T14:03:28.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5325e-05 2023-07-02T14:03:28.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:28.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357495+0000", 2023-07-02T14:03:28.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042107899999999998 2023-07-02T14:03:28.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:28.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357916+0000", 2023-07-02T14:03:28.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0011672449999999999 2023-07-02T14:03:28.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:28.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.359083+0000", 2023-07-02T14:03:28.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013534930000000001 2023-07-02T14:03:28.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:28.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360437+0000", 2023-07-02T14:03:28.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2159000000000002e-05 2023-07-02T14:03:28.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:28.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360469+0000", 2023-07-02T14:03:28.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0092000000000005e-05 2023-07-02T14:03:28.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:28.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360549+0000", 2023-07-02T14:03:28.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042407105 2023-07-02T14:03:28.745 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.745 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.745 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.746 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:418 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:28.746 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:28.746 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.258355074000001, 2023-07-02T14:03:28.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.045613663999999998, 2023-07-02T14:03:28.746 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:28.746 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:28.747 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:28.747 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:28.747 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:28.747 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 418 2023-07-02T14:03:28.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.747 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:28.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:28.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:28.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:28.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:28.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:28.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:28.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:28.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318160+0000", 2023-07-02T14:03:28.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6490000000000001e-06 2023-07-02T14:03:28.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:28.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318164+0000", 2023-07-02T14:03:28.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:03:28.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:28.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318170+0000", 2023-07-02T14:03:28.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5279999999999999e-06 2023-07-02T14:03:28.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:28.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318173+0000", 2023-07-02T14:03:28.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022788856 2023-07-02T14:03:28.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340962+0000", 2023-07-02T14:03:28.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7730000000000003e-05 2023-07-02T14:03:28.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341040+0000", 2023-07-02T14:03:28.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039795899999999999 2023-07-02T14:03:28.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345020+0000", 2023-07-02T14:03:28.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4962000000000006e-05 2023-07-02T14:03:28.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345085+0000", 2023-07-02T14:03:28.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051955430000000004 2023-07-02T14:03:28.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350280+0000", 2023-07-02T14:03:28.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8913999999999995e-05 2023-07-02T14:03:28.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350379+0000", 2023-07-02T14:03:28.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001887344 2023-07-02T14:03:28.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352266+0000", 2023-07-02T14:03:28.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7865000000000003e-05 2023-07-02T14:03:28.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352314+0000", 2023-07-02T14:03:28.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060621190000000004 2023-07-02T14:03:28.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358376+0000", 2023-07-02T14:03:28.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2448999999999997e-05 2023-07-02T14:03:28.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358449+0000", 2023-07-02T14:03:28.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0021539929999999999 2023-07-02T14:03:28.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360603+0000", 2023-07-02T14:03:28.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7859000000000002e-05 2023-07-02T14:03:28.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:28.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360671+0000", 2023-07-02T14:03:28.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026930699999999999 2023-07-02T14:03:28.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:28.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360940+0000", 2023-07-02T14:03:28.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00081179699999999995 2023-07-02T14:03:28.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:28.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361752+0000", 2023-07-02T14:03:28.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0019350490000000001 2023-07-02T14:03:28.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:28.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363687+0000", 2023-07-02T14:03:28.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4462000000000003e-05 2023-07-02T14:03:28.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:28.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363721+0000", 2023-07-02T14:03:28.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3408000000000001e-05 2023-07-02T14:03:28.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:28.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363775+0000", 2023-07-02T14:03:28.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.045613663999999998 2023-07-02T14:03:28.762 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.762 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.762 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.763 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:419 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:28.763 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:28.763 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.258338747000003, 2023-07-02T14:03:28.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.047296703000000002, 2023-07-02T14:03:28.763 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:28.763 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:28.763 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:28.764 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:28.764 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:28.764 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 419 2023-07-02T14:03:28.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.764 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:28.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:28.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:28.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:28.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:28.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:28.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:28.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:28.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:28.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.433e-06 2023-07-02T14:03:28.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:28.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318180+0000", 2023-07-02T14:03:28.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9950000000000002e-06 2023-07-02T14:03:28.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:28.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318186+0000", 2023-07-02T14:03:28.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7290000000000002e-06 2023-07-02T14:03:28.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:28.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318190+0000", 2023-07-02T14:03:28.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022877989000000001 2023-07-02T14:03:28.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341068+0000", 2023-07-02T14:03:28.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5803999999999994e-05 2023-07-02T14:03:28.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341154+0000", 2023-07-02T14:03:28.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039633000000000003 2023-07-02T14:03:28.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345117+0000", 2023-07-02T14:03:28.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7188999999999996e-05 2023-07-02T14:03:28.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345194+0000", 2023-07-02T14:03:28.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052306059999999996 2023-07-02T14:03:28.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350425+0000", 2023-07-02T14:03:28.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8093999999999997e-05 2023-07-02T14:03:28.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350473+0000", 2023-07-02T14:03:28.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001865158 2023-07-02T14:03:28.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352338+0000", 2023-07-02T14:03:28.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5479000000000003e-05 2023-07-02T14:03:28.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352384+0000", 2023-07-02T14:03:28.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061153290000000001 2023-07-02T14:03:28.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358499+0000", 2023-07-02T14:03:28.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5156999999999999e-05 2023-07-02T14:03:28.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358574+0000", 2023-07-02T14:03:28.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027847459999999998 2023-07-02T14:03:28.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361359+0000", 2023-07-02T14:03:28.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2633999999999999e-05 2023-07-02T14:03:28.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361402+0000", 2023-07-02T14:03:28.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024171510000000002 2023-07-02T14:03:28.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363819+0000", 2023-07-02T14:03:28.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6988999999999999e-05 2023-07-02T14:03:28.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:28.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363876+0000", 2023-07-02T14:03:28.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035321300000000001 2023-07-02T14:03:28.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:28.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.364229+0000", 2023-07-02T14:03:28.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00094674799999999997 2023-07-02T14:03:28.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:28.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365176+0000", 2023-07-02T14:03:28.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019724200000000001 2023-07-02T14:03:28.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:28.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365373+0000", 2023-07-02T14:03:28.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2116e-05 2023-07-02T14:03:28.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:28.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365425+0000", 2023-07-02T14:03:28.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8996999999999999e-05 2023-07-02T14:03:28.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:28.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365474+0000", 2023-07-02T14:03:28.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.047296703000000002 2023-07-02T14:03:28.780 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.780 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.780 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.781 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:472 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:28.781 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:28.781 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.198882515000001, 2023-07-02T14:03:28.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036204721000000002, 2023-07-02T14:03:28.781 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:28.781 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:28.781 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:28.782 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:28.782 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:28.782 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 472 2023-07-02T14:03:28.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.782 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:28.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:28.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:28.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:28.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:28.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:28.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:03:28.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:28.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377633+0000", 2023-07-02T14:03:28.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.815e-06 2023-07-02T14:03:28.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:28.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377635+0000", 2023-07-02T14:03:28.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9459999999999998e-06 2023-07-02T14:03:28.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:28.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377638+0000", 2023-07-02T14:03:28.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.673e-06 2023-07-02T14:03:28.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:28.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377643+0000", 2023-07-02T14:03:28.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043004239999999997 2023-07-02T14:03:28.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.381943+0000", 2023-07-02T14:03:28.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4321999999999999e-05 2023-07-02T14:03:28.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.381998+0000", 2023-07-02T14:03:28.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003008377 2023-07-02T14:03:28.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385006+0000", 2023-07-02T14:03:28.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8751999999999998e-05 2023-07-02T14:03:28.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385045+0000", 2023-07-02T14:03:28.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086260880000000005 2023-07-02T14:03:28.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393671+0000", 2023-07-02T14:03:28.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6365999999999997e-05 2023-07-02T14:03:28.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393707+0000", 2023-07-02T14:03:28.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028560289999999999 2023-07-02T14:03:28.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396563+0000", 2023-07-02T14:03:28.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6195e-05 2023-07-02T14:03:28.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396619+0000", 2023-07-02T14:03:28.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023118269999999998 2023-07-02T14:03:28.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.398931+0000", 2023-07-02T14:03:28.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3885999999999999e-05 2023-07-02T14:03:28.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.398985+0000", 2023-07-02T14:03:28.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043756890000000003 2023-07-02T14:03:28.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403361+0000", 2023-07-02T14:03:28.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0043000000000001e-05 2023-07-02T14:03:28.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403421+0000", 2023-07-02T14:03:28.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001590972 2023-07-02T14:03:28.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405012+0000", 2023-07-02T14:03:28.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0067000000000005e-05 2023-07-02T14:03:28.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405082+0000", 2023-07-02T14:03:28.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050958610000000001 2023-07-02T14:03:28.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410178+0000", 2023-07-02T14:03:28.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7070000000000004e-05 2023-07-02T14:03:28.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410275+0000", 2023-07-02T14:03:28.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060121100000000004 2023-07-02T14:03:28.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410876+0000", 2023-07-02T14:03:28.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3177999999999999e-05 2023-07-02T14:03:28.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:28.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410919+0000", 2023-07-02T14:03:28.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000271394 2023-07-02T14:03:28.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:28.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411191+0000", 2023-07-02T14:03:28.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001009598 2023-07-02T14:03:28.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:28.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.412200+0000", 2023-07-02T14:03:28.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014789790000000001 2023-07-02T14:03:28.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:28.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413679+0000", 2023-07-02T14:03:28.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2624000000000002e-05 2023-07-02T14:03:28.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:28.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413742+0000", 2023-07-02T14:03:28.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6564999999999994e-05 2023-07-02T14:03:28.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:28.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413838+0000", 2023-07-02T14:03:28.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036204721000000002 2023-07-02T14:03:28.801 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.801 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.801 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.801 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:473 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:28.801 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:28.802 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.198871103999998, 2023-07-02T14:03:28.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037915904, 2023-07-02T14:03:28.802 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:28.802 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:28.802 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:28.802 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:28.802 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:28.802 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 473 2023-07-02T14:03:28.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.803 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:28.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:28.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:28.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:28.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:28.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:28.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:03:28.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:28.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:28.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.119e-06 2023-07-02T14:03:28.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:28.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377647+0000", 2023-07-02T14:03:28.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.875e-06 2023-07-02T14:03:28.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:28.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377650+0000", 2023-07-02T14:03:28.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1949999999999999e-06 2023-07-02T14:03:28.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:28.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377653+0000", 2023-07-02T14:03:28.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004375444 2023-07-02T14:03:28.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382028+0000", 2023-07-02T14:03:28.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1063999999999997e-05 2023-07-02T14:03:28.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382089+0000", 2023-07-02T14:03:28.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030043869999999999 2023-07-02T14:03:28.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385094+0000", 2023-07-02T14:03:28.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8167999999999997e-05 2023-07-02T14:03:28.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385142+0000", 2023-07-02T14:03:28.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008603421 2023-07-02T14:03:28.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393745+0000", 2023-07-02T14:03:28.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3763000000000005e-05 2023-07-02T14:03:28.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393819+0000", 2023-07-02T14:03:28.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028274720000000001 2023-07-02T14:03:28.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396647+0000", 2023-07-02T14:03:28.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7929999999999997e-05 2023-07-02T14:03:28.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396695+0000", 2023-07-02T14:03:28.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023426459999999999 2023-07-02T14:03:28.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399037+0000", 2023-07-02T14:03:28.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5846000000000004e-05 2023-07-02T14:03:28.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399123+0000", 2023-07-02T14:03:28.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043448360000000004 2023-07-02T14:03:28.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403468+0000", 2023-07-02T14:03:28.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6699000000000001e-05 2023-07-02T14:03:28.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403515+0000", 2023-07-02T14:03:28.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001622943 2023-07-02T14:03:28.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405138+0000", 2023-07-02T14:03:28.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6382000000000002e-05 2023-07-02T14:03:28.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405204+0000", 2023-07-02T14:03:28.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051089910000000002 2023-07-02T14:03:28.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410313+0000", 2023-07-02T14:03:28.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8106000000000003e-05 2023-07-02T14:03:28.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410361+0000", 2023-07-02T14:03:28.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012200399999999999 2023-07-02T14:03:28.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411581+0000", 2023-07-02T14:03:28.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5067000000000005e-05 2023-07-02T14:03:28.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411656+0000", 2023-07-02T14:03:28.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022353249999999998 2023-07-02T14:03:28.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413891+0000", 2023-07-02T14:03:28.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9356000000000003e-05 2023-07-02T14:03:28.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:28.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413981+0000", 2023-07-02T14:03:28.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00043818299999999998 2023-07-02T14:03:28.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:28.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.414419+0000", 2023-07-02T14:03:28.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00081559199999999997 2023-07-02T14:03:28.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:28.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415235+0000", 2023-07-02T14:03:28.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026434499999999998 2023-07-02T14:03:28.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:28.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415499+0000", 2023-07-02T14:03:28.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9779999999999999e-05 2023-07-02T14:03:28.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:28.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415529+0000", 2023-07-02T14:03:28.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2159999999999997e-05 2023-07-02T14:03:28.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:28.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415561+0000", 2023-07-02T14:03:28.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037915904 2023-07-02T14:03:28.823 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.823 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.823 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.824 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:475 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:28.824 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:28.824 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.198861065000003, 2023-07-02T14:03:28.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039286962000000002, 2023-07-02T14:03:28.824 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:28.824 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:28.824 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:28.824 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:28.825 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:28.825 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 475 2023-07-02T14:03:28.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.825 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:28.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:28.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:28.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:28.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:28.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:28.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:28.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:28.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:28.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4779999999999997e-06 2023-07-02T14:03:28.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:28.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377664+0000", 2023-07-02T14:03:28.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.268e-06 2023-07-02T14:03:28.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:28.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377671+0000", 2023-07-02T14:03:28.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.861e-06 2023-07-02T14:03:28.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:28.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377674+0000", 2023-07-02T14:03:28.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044395789999999999 2023-07-02T14:03:28.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382114+0000", 2023-07-02T14:03:28.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8096999999999997e-05 2023-07-02T14:03:28.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382162+0000", 2023-07-02T14:03:28.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029949059999999999 2023-07-02T14:03:28.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385157+0000", 2023-07-02T14:03:28.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1158000000000001e-05 2023-07-02T14:03:28.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385198+0000", 2023-07-02T14:03:28.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086477229999999995 2023-07-02T14:03:28.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393846+0000", 2023-07-02T14:03:28.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5990999999999999e-05 2023-07-02T14:03:28.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393912+0000", 2023-07-02T14:03:28.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027979599999999999 2023-07-02T14:03:28.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396710+0000", 2023-07-02T14:03:28.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5911000000000002e-05 2023-07-02T14:03:28.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396746+0000", 2023-07-02T14:03:28.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024039759999999999 2023-07-02T14:03:28.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399150+0000", 2023-07-02T14:03:28.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7834999999999999e-05 2023-07-02T14:03:28.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399188+0000", 2023-07-02T14:03:28.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004364054 2023-07-02T14:03:28.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403552+0000", 2023-07-02T14:03:28.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6274999999999999e-05 2023-07-02T14:03:28.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403608+0000", 2023-07-02T14:03:28.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0016261540000000001 2023-07-02T14:03:28.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405234+0000", 2023-07-02T14:03:28.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6945000000000004e-05 2023-07-02T14:03:28.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405301+0000", 2023-07-02T14:03:28.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050871329999999998 2023-07-02T14:03:28.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410388+0000", 2023-07-02T14:03:28.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0210999999999995e-05 2023-07-02T14:03:28.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410468+0000", 2023-07-02T14:03:28.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001227454 2023-07-02T14:03:28.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411696+0000", 2023-07-02T14:03:28.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8188000000000005e-05 2023-07-02T14:03:28.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411764+0000", 2023-07-02T14:03:28.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031779930000000001 2023-07-02T14:03:28.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.414942+0000", 2023-07-02T14:03:28.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7236000000000004e-05 2023-07-02T14:03:28.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:28.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415019+0000", 2023-07-02T14:03:28.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000573045 2023-07-02T14:03:28.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:28.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415592+0000", 2023-07-02T14:03:28.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7580000000000002e-05 2023-07-02T14:03:28.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:28.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415640+0000", 2023-07-02T14:03:28.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024783200000000002 2023-07-02T14:03:28.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:28.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415888+0000", 2023-07-02T14:03:28.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00080440999999999998 2023-07-02T14:03:28.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:28.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416692+0000", 2023-07-02T14:03:28.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000195787 2023-07-02T14:03:28.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:28.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416888+0000", 2023-07-02T14:03:28.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0787999999999997e-05 2023-07-02T14:03:28.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:28.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416919+0000", 2023-07-02T14:03:28.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.338e-05 2023-07-02T14:03:28.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:28.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:28.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:28.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416942+0000", 2023-07-02T14:03:28.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039286962000000002 2023-07-02T14:03:28.846 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.846 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.846 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.846 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:28.847 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:28.847 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:28.848 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:03:28.849 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:28.848+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:28.949 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:28.948+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:29.015 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:29.015 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:29.015 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:03:29.015 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:29.025 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:03:29.050 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:29.050+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:29.148 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:29.148 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:29.148 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:29.149 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:29.149 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:29.151 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:29.150+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:29.158 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:03:29.252 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:29.252+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:29.323 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:29.323 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:29.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:29.324 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:29.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.324 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:1053 2.0 e57/46 2:1c3726b3:::rbd_data.125de1449605.000000000000002f:head v 57'164, mlcod=57'161)", 2023-07-02T14:03:29.324 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:05.578013+0000", 2023-07-02T14:03:29.324 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 23.744074609999998, 2023-07-02T14:03:29.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099458040000000008, 2023-07-02T14:03:29.325 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.325 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:29.325 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.578013+0000", 2023-07-02T14:03:29.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.578013+0000", 2023-07-02T14:03:29.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:29.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:29.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.578012+0000", 2023-07-02T14:03:29.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046313150000000004 2023-07-02T14:03:29.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:29.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582643+0000", 2023-07-02T14:03:29.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0795999999999999e-05 2023-07-02T14:03:29.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:29.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582664+0000", 2023-07-02T14:03:29.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2939999999999994e-06 2023-07-02T14:03:29.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:29.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582673+0000", 2023-07-02T14:03:29.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3214999999999998e-05 2023-07-02T14:03:29.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:29.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582726+0000", 2023-07-02T14:03:29.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8241999999999997e-05 2023-07-02T14:03:29.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:29.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582765+0000", 2023-07-02T14:03:29.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051597769999999999 2023-07-02T14:03:29.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:29.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.587924+0000", 2023-07-02T14:03:29.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4795999999999997e-05 2023-07-02T14:03:29.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:29.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.587959+0000", 2023-07-02T14:03:29.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099458040000000008 2023-07-02T14:03:29.331 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.331 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.332 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.332 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:1054 2.0 e57/46 2:1c3726b3:::rbd_data.125de1449605.000000000000002f:head v 57'165, mlcod=57'161)", 2023-07-02T14:03:29.332 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:05.582711+0000", 2023-07-02T14:03:29.332 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 23.739376890999999, 2023-07-02T14:03:29.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053281630000000003, 2023-07-02T14:03:29.333 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.333 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:29.333 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582711+0000", 2023-07-02T14:03:29.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582711+0000", 2023-07-02T14:03:29.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:29.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:29.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582710+0000", 2023-07-02T14:03:29.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0908000000000002e-05 2023-07-02T14:03:29.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:29.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582751+0000", 2023-07-02T14:03:29.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1470000000000007e-06 2023-07-02T14:03:29.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:29.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582760+0000", 2023-07-02T14:03:29.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1669999999999998e-06 2023-07-02T14:03:29.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:29.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.582766+0000", 2023-07-02T14:03:29.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002795905 2023-07-02T14:03:29.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:29.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.585561+0000", 2023-07-02T14:03:29.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6288e-05 2023-07-02T14:03:29.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:29.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.585588+0000", 2023-07-02T14:03:29.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024248550000000001 2023-07-02T14:03:29.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:29.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.588013+0000", 2023-07-02T14:03:29.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6682e-05 2023-07-02T14:03:29.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:29.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.588039+0000", 2023-07-02T14:03:29.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053281630000000003 2023-07-02T14:03:29.339 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.339 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.339 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.340 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:1094 2.0 e57/46 2:1c3726b3:::rbd_data.125de1449605.000000000000002f:head v 57'170, mlcod=57'167)", 2023-07-02T14:03:29.340 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:05.595461+0000", 2023-07-02T14:03:29.340 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 23.726627324999999, 2023-07-02T14:03:29.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070844539999999996, 2023-07-02T14:03:29.340 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.340 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:29.341 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.595461+0000", 2023-07-02T14:03:29.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.595461+0000", 2023-07-02T14:03:29.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:29.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:29.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.595460+0000", 2023-07-02T14:03:29.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033907640000000001 2023-07-02T14:03:29.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:29.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.598850+0000", 2023-07-02T14:03:29.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8046e-05 2023-07-02T14:03:29.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:29.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.598868+0000", 2023-07-02T14:03:29.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1449e-05 2023-07-02T14:03:29.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:29.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.598880+0000", 2023-07-02T14:03:29.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0612000000000003e-05 2023-07-02T14:03:29.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:29.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.598931+0000", 2023-07-02T14:03:29.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.421e-05 2023-07-02T14:03:29.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:29.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.598955+0000", 2023-07-02T14:03:29.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0035568819999999999 2023-07-02T14:03:29.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:29.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.602512+0000", 2023-07-02T14:03:29.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3525999999999997e-05 2023-07-02T14:03:29.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:29.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.602545+0000", 2023-07-02T14:03:29.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070844539999999996 2023-07-02T14:03:29.347 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.347 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.347 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.348 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:1105 2.0 e57/46 2:1beb29b4:::rbd_data.125de1449605.0000000000000025:head v 57'173, mlcod=57'168)", 2023-07-02T14:03:29.348 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:05.599412+0000", 2023-07-02T14:03:29.348 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 23.722676221, 2023-07-02T14:03:29.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043199980000000002, 2023-07-02T14:03:29.348 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.348 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:29.348 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.599412+0000", 2023-07-02T14:03:29.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.599412+0000", 2023-07-02T14:03:29.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:29.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:29.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.599411+0000", 2023-07-02T14:03:29.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013543450000000001 2023-07-02T14:03:29.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:29.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.600765+0000", 2023-07-02T14:03:29.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4677e-05 2023-07-02T14:03:29.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:29.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.600780+0000", 2023-07-02T14:03:29.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7959999999999998e-06 2023-07-02T14:03:29.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:29.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.600787+0000", 2023-07-02T14:03:29.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00073057299999999999 2023-07-02T14:03:29.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:29.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.601517+0000", 2023-07-02T14:03:29.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4372e-05 2023-07-02T14:03:29.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:29.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.601542+0000", 2023-07-02T14:03:29.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0021526969999999999 2023-07-02T14:03:29.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:29.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.603695+0000", 2023-07-02T14:03:29.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7258000000000001e-05 2023-07-02T14:03:29.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:29.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.603732+0000", 2023-07-02T14:03:29.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043199980000000002 2023-07-02T14:03:29.355 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.355 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.355 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.355 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:1120 2.0 e57/46 2:1fa9788a:::rbd_data.125de1449605.0000000000000030:head v 57'177, mlcod=57'174)", 2023-07-02T14:03:29.356 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:05.608346+0000", 2023-07-02T14:03:29.356 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 23.713741815999999, 2023-07-02T14:03:29.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067962459999999997, 2023-07-02T14:03:29.356 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.356 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:29.356 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.608346+0000", 2023-07-02T14:03:29.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.608346+0000", 2023-07-02T14:03:29.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:29.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:29.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.608345+0000", 2023-07-02T14:03:29.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002673539 2023-07-02T14:03:29.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:29.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611019+0000", 2023-07-02T14:03:29.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8331000000000001e-05 2023-07-02T14:03:29.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:29.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611037+0000", 2023-07-02T14:03:29.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8879999999999996e-06 2023-07-02T14:03:29.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:29.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611042+0000", 2023-07-02T14:03:29.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7515999999999997e-05 2023-07-02T14:03:29.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:29.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611090+0000", 2023-07-02T14:03:29.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5330999999999999e-05 2023-07-02T14:03:29.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:29.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611115+0000", 2023-07-02T14:03:29.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039866559999999999 2023-07-02T14:03:29.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:29.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.615102+0000", 2023-07-02T14:03:29.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0865999999999998e-05 2023-07-02T14:03:29.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:29.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.615142+0000", 2023-07-02T14:03:29.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067962459999999997 2023-07-02T14:03:29.363 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.363 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.363 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.363 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:1121 2.0 e57/46 2:1fa9788a:::rbd_data.125de1449605.0000000000000030:head v 57'178, mlcod=57'174)", 2023-07-02T14:03:29.363 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:05.611066+0000", 2023-07-02T14:03:29.363 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 23.711022189000001, 2023-07-02T14:03:29.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041697710000000001, 2023-07-02T14:03:29.364 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.364 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:29.364 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611066+0000", 2023-07-02T14:03:29.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611066+0000", 2023-07-02T14:03:29.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:29.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:29.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611065+0000", 2023-07-02T14:03:29.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00090598600000000001 2023-07-02T14:03:29.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:29.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611971+0000", 2023-07-02T14:03:29.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2669e-05 2023-07-02T14:03:29.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:29.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611984+0000", 2023-07-02T14:03:29.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5730000000000001e-06 2023-07-02T14:03:29.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:29.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.611989+0000", 2023-07-02T14:03:29.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060156399999999996 2023-07-02T14:03:29.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:29.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.612590+0000", 2023-07-02T14:03:29.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6229999999999999e-05 2023-07-02T14:03:29.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:29.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.612606+0000", 2023-07-02T14:03:29.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026060950000000001 2023-07-02T14:03:29.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:29.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.615212+0000", 2023-07-02T14:03:29.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3113999999999999e-05 2023-07-02T14:03:29.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:29.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.615236+0000", 2023-07-02T14:03:29.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041697710000000001 2023-07-02T14:03:29.370 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.370 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.371 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.371 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:1198 2.0 e57/46 2:107a9547:::rbd_data.125de1449605.000000000000003f:head v 57'195, mlcod=57'193)", 2023-07-02T14:03:29.371 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:05.668454+0000", 2023-07-02T14:03:29.371 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 23.65363352, 2023-07-02T14:03:29.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0087668139999999995, 2023-07-02T14:03:29.371 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.372 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:29.372 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.668454+0000", 2023-07-02T14:03:29.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.668454+0000", 2023-07-02T14:03:29.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:29.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:29.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.668453+0000", 2023-07-02T14:03:29.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045315149999999998 2023-07-02T14:03:29.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:29.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.672985+0000", 2023-07-02T14:03:29.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1236000000000001e-05 2023-07-02T14:03:29.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:29.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.673006+0000", 2023-07-02T14:03:29.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.776e-06 2023-07-02T14:03:29.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:29.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.673013+0000", 2023-07-02T14:03:29.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1847e-05 2023-07-02T14:03:29.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:29.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.673065+0000", 2023-07-02T14:03:29.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8389999999999998e-05 2023-07-02T14:03:29.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:29.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.673083+0000", 2023-07-02T14:03:29.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041092359999999996 2023-07-02T14:03:29.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:29.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.677192+0000", 2023-07-02T14:03:29.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9028999999999999e-05 2023-07-02T14:03:29.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:29.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.677221+0000", 2023-07-02T14:03:29.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0087668139999999995 2023-07-02T14:03:29.378 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.378 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.378 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.379 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:1199 2.0 e57/46 2:107a9547:::rbd_data.125de1449605.000000000000003f:head v 57'196, mlcod=57'193)", 2023-07-02T14:03:29.379 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:05.673219+0000", 2023-07-02T14:03:29.379 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 23.648869003000001, 2023-07-02T14:03:29.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043126400000000004, 2023-07-02T14:03:29.379 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.379 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:29.380 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.673219+0000", 2023-07-02T14:03:29.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.673219+0000", 2023-07-02T14:03:29.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:29.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:29.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.673218+0000", 2023-07-02T14:03:29.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8931999999999996e-05 2023-07-02T14:03:29.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:29.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.673297+0000", 2023-07-02T14:03:29.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2189999999999998e-06 2023-07-02T14:03:29.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:29.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.673304+0000", 2023-07-02T14:03:29.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4929999999999998e-06 2023-07-02T14:03:29.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:29.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.673308+0000", 2023-07-02T14:03:29.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023961500000000001 2023-07-02T14:03:29.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:29.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.675704+0000", 2023-07-02T14:03:29.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6027000000000001e-05 2023-07-02T14:03:29.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:29.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.675720+0000", 2023-07-02T14:03:29.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785841 2023-07-02T14:03:29.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:29.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.677506+0000", 2023-07-02T14:03:29.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5726000000000001e-05 2023-07-02T14:03:29.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:29.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.677532+0000", 2023-07-02T14:03:29.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043126400000000004 2023-07-02T14:03:29.386 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.386 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.386 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.387 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:1226 2.0 e57/46 2:047f2192:::rbd_data.125de1449605.0000000000000041:head v 57'206, mlcod=57'203)", 2023-07-02T14:03:29.387 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:05.703321+0000", 2023-07-02T14:03:29.387 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 23.618766872999998, 2023-07-02T14:03:29.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006518228, 2023-07-02T14:03:29.387 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.387 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:29.387 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.703321+0000", 2023-07-02T14:03:29.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.703321+0000", 2023-07-02T14:03:29.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:29.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:29.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.703321+0000", 2023-07-02T14:03:29.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031170590000000001 2023-07-02T14:03:29.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:29.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.706438+0000", 2023-07-02T14:03:29.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1039999999999998e-05 2023-07-02T14:03:29.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:29.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.706459+0000", 2023-07-02T14:03:29.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3070000000000008e-06 2023-07-02T14:03:29.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:29.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.706468+0000", 2023-07-02T14:03:29.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0488000000000002e-05 2023-07-02T14:03:29.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:29.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.706519+0000", 2023-07-02T14:03:29.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7299000000000002e-05 2023-07-02T14:03:29.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:29.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.706536+0000", 2023-07-02T14:03:29.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032408950000000001 2023-07-02T14:03:29.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:29.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.709777+0000", 2023-07-02T14:03:29.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2612000000000003e-05 2023-07-02T14:03:29.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:29.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:05.709839+0000", 2023-07-02T14:03:29.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006518228 2023-07-02T14:03:29.394 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.394 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.394 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.394 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:1270 2.0 e57/46 2:169af44a:::rbd_data.125de1449605.0000000000000106:head v 57'216, mlcod=57'212)", 2023-07-02T14:03:29.394 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:06.420574+0000", 2023-07-02T14:03:29.395 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.901513547, 2023-07-02T14:03:29.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055088890000000003, 2023-07-02T14:03:29.395 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.395 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:29.395 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:06.420574+0000", 2023-07-02T14:03:29.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:06.420574+0000", 2023-07-02T14:03:29.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:29.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:29.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:06.420573+0000", 2023-07-02T14:03:29.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001758765 2023-07-02T14:03:29.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:29.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:06.422332+0000", 2023-07-02T14:03:29.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5381000000000001e-05 2023-07-02T14:03:29.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:29.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:06.422348+0000", 2023-07-02T14:03:29.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1210000000000001e-06 2023-07-02T14:03:29.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:29.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:06.422355+0000", 2023-07-02T14:03:29.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0161999999999998e-05 2023-07-02T14:03:29.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:29.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:06.422405+0000", 2023-07-02T14:03:29.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5510000000000001e-05 2023-07-02T14:03:29.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:29.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:06.422430+0000", 2023-07-02T14:03:29.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036230210000000001 2023-07-02T14:03:29.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:29.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:06.426053+0000", 2023-07-02T14:03:29.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0014000000000001e-05 2023-07-02T14:03:29.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:29.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:06.426083+0000", 2023-07-02T14:03:29.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055088890000000003 2023-07-02T14:03:29.402 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.402 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.402 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.402 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:2840 2.0 e61/46 2:1c81f085:::rbd_data.125de1449605.00000000000000f0:head v 61'639, mlcod=61'637)", 2023-07-02T14:03:29.402 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:11.368484+0000", 2023-07-02T14:03:29.402 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.953603632, 2023-07-02T14:03:29.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038949549999999999, 2023-07-02T14:03:29.403 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.403 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:29.403 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.368484+0000", 2023-07-02T14:03:29.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.404 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.368484+0000", 2023-07-02T14:03:29.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:29.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.404 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:29.405 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.368483+0000", 2023-07-02T14:03:29.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00099835700000000006 2023-07-02T14:03:29.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:29.405 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.369481+0000", 2023-07-02T14:03:29.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7671000000000001e-05 2023-07-02T14:03:29.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:29.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.369499+0000", 2023-07-02T14:03:29.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7360000000000004e-06 2023-07-02T14:03:29.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:29.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.369505+0000", 2023-07-02T14:03:29.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3460000000000001e-05 2023-07-02T14:03:29.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:29.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.369549+0000", 2023-07-02T14:03:29.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9530000000000001e-05 2023-07-02T14:03:29.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:29.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.369568+0000", 2023-07-02T14:03:29.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027731679999999999 2023-07-02T14:03:29.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:29.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.372342+0000", 2023-07-02T14:03:29.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7713000000000003e-05 2023-07-02T14:03:29.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:29.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.372379+0000", 2023-07-02T14:03:29.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038949549999999999 2023-07-02T14:03:29.409 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.409 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.410 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.410 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:2863 2.0 e61/46 2:1c81f085:::rbd_data.125de1449605.00000000000000f0:head v 61'641, mlcod=61'639)", 2023-07-02T14:03:29.410 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:11.383026+0000", 2023-07-02T14:03:29.410 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.939061937999998, 2023-07-02T14:03:29.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037956190000000001, 2023-07-02T14:03:29.410 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.411 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:29.411 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.383026+0000", 2023-07-02T14:03:29.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.383026+0000", 2023-07-02T14:03:29.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:29.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:29.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.383025+0000", 2023-07-02T14:03:29.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012165629999999999 2023-07-02T14:03:29.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:29.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.384241+0000", 2023-07-02T14:03:29.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9899999999999999e-05 2023-07-02T14:03:29.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:29.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.384261+0000", 2023-07-02T14:03:29.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6300000000000004e-06 2023-07-02T14:03:29.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:29.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.384270+0000", 2023-07-02T14:03:29.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6109999999999997e-05 2023-07-02T14:03:29.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:29.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.384316+0000", 2023-07-02T14:03:29.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4791999999999999e-05 2023-07-02T14:03:29.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:29.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.384341+0000", 2023-07-02T14:03:29.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002455357 2023-07-02T14:03:29.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:29.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.386796+0000", 2023-07-02T14:03:29.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5592000000000002e-05 2023-07-02T14:03:29.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:29.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.386822+0000", 2023-07-02T14:03:29.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037956190000000001 2023-07-02T14:03:29.417 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.417 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.417 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.418 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:2896 2.0 e61/46 2:08f47745:::rbd_data.125de1449605.00000000000000eb:head v 61'652, mlcod=61'648)", 2023-07-02T14:03:29.418 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:11.422798+0000", 2023-07-02T14:03:29.418 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.899289703000001, 2023-07-02T14:03:29.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077822680000000002, 2023-07-02T14:03:29.418 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.418 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:29.418 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.422798+0000", 2023-07-02T14:03:29.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.422798+0000", 2023-07-02T14:03:29.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:29.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:29.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.422797+0000", 2023-07-02T14:03:29.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030387439999999999 2023-07-02T14:03:29.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:29.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.425836+0000", 2023-07-02T14:03:29.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1531000000000001e-05 2023-07-02T14:03:29.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:29.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.425857+0000", 2023-07-02T14:03:29.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3219999999999999e-06 2023-07-02T14:03:29.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:29.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.425867+0000", 2023-07-02T14:03:29.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0667000000000001e-05 2023-07-02T14:03:29.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:29.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.425917+0000", 2023-07-02T14:03:29.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8943999999999999e-05 2023-07-02T14:03:29.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:29.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.425946+0000", 2023-07-02T14:03:29.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046048809999999999 2023-07-02T14:03:29.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:29.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.430551+0000", 2023-07-02T14:03:29.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9351000000000001e-05 2023-07-02T14:03:29.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:29.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.430581+0000", 2023-07-02T14:03:29.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077822680000000002 2023-07-02T14:03:29.425 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.425 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.425 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.426 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:2958 2.0 e61/46 2:187db15e:::rbd_data.125de1449605.00000000000000fc:head v 61'667, mlcod=61'665)", 2023-07-02T14:03:29.426 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:11.493615+0000", 2023-07-02T14:03:29.426 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.828473303999999, 2023-07-02T14:03:29.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057044310000000003, 2023-07-02T14:03:29.426 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.426 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:29.426 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.493615+0000", 2023-07-02T14:03:29.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.493615+0000", 2023-07-02T14:03:29.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999943 2023-07-02T14:03:29.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:29.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.493609+0000", 2023-07-02T14:03:29.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020804550000000002 2023-07-02T14:03:29.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:29.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.495689+0000", 2023-07-02T14:03:29.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4980000000000001e-05 2023-07-02T14:03:29.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:29.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.495714+0000", 2023-07-02T14:03:29.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1089999999999998e-06 2023-07-02T14:03:29.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:29.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.495722+0000", 2023-07-02T14:03:29.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5745999999999999e-05 2023-07-02T14:03:29.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:29.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.495768+0000", 2023-07-02T14:03:29.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9123999999999999e-05 2023-07-02T14:03:29.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:29.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.495787+0000", 2023-07-02T14:03:29.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0035009989999999999 2023-07-02T14:03:29.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:29.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.499288+0000", 2023-07-02T14:03:29.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0925e-05 2023-07-02T14:03:29.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:29.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.499319+0000", 2023-07-02T14:03:29.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057044310000000003 2023-07-02T14:03:29.433 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.433 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.433 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.433 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:3017 2.0 e61/46 2:1f6e35fd:::rbd_data.125de1449605.0000000000000111:head v 61'683, mlcod=61'680)", 2023-07-02T14:03:29.433 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:11.548521+0000", 2023-07-02T14:03:29.434 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.773567366999998, 2023-07-02T14:03:29.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006595465, 2023-07-02T14:03:29.434 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.434 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:29.434 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.548521+0000", 2023-07-02T14:03:29.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.548521+0000", 2023-07-02T14:03:29.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:29.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:29.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.548520+0000", 2023-07-02T14:03:29.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027544179999999998 2023-07-02T14:03:29.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:29.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.551274+0000", 2023-07-02T14:03:29.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.451e-05 2023-07-02T14:03:29.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:29.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.551289+0000", 2023-07-02T14:03:29.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4489999999999996e-06 2023-07-02T14:03:29.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:29.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.551295+0000", 2023-07-02T14:03:29.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2367000000000003e-05 2023-07-02T14:03:29.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:29.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.551338+0000", 2023-07-02T14:03:29.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6847999999999999e-05 2023-07-02T14:03:29.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:29.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.551355+0000", 2023-07-02T14:03:29.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037245009999999999 2023-07-02T14:03:29.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:29.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.555079+0000", 2023-07-02T14:03:29.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6934999999999998e-05 2023-07-02T14:03:29.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:29.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.555116+0000", 2023-07-02T14:03:29.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006595465 2023-07-02T14:03:29.440 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.441 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.441 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.441 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:3028 2.0 e61/46 2:014344e8:::rbd_data.125de1449605.0000000000000114:head v 61'690, mlcod=61'685)", 2023-07-02T14:03:29.441 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:11.562915+0000", 2023-07-02T14:03:29.441 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.759173115999999, 2023-07-02T14:03:29.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075509189999999997, 2023-07-02T14:03:29.442 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.442 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:29.442 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.562915+0000", 2023-07-02T14:03:29.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.562915+0000", 2023-07-02T14:03:29.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:29.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:29.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.562914+0000", 2023-07-02T14:03:29.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032299809999999998 2023-07-02T14:03:29.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:29.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.566144+0000", 2023-07-02T14:03:29.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1277999999999999e-05 2023-07-02T14:03:29.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:29.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.566165+0000", 2023-07-02T14:03:29.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6469999999999993e-06 2023-07-02T14:03:29.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:29.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.566175+0000", 2023-07-02T14:03:29.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0241000000000003e-05 2023-07-02T14:03:29.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:29.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.566215+0000", 2023-07-02T14:03:29.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2554000000000001e-05 2023-07-02T14:03:29.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:29.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.566238+0000", 2023-07-02T14:03:29.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041783059999999997 2023-07-02T14:03:29.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:29.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.570416+0000", 2023-07-02T14:03:29.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9579000000000001e-05 2023-07-02T14:03:29.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:29.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:11.570466+0000", 2023-07-02T14:03:29.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075509189999999997 2023-07-02T14:03:29.448 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.448 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.449 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.449 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.7 62/60 [PushOp(2:f30916f5:::rbd_data.125de1449605.0000000000000075:head, version: 58'249, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:f30916f5:::rbd_data.125de1449605.0000000000000075:head@58'249, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-07-02T14:03:29.449 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:12.091624+0000", 2023-07-02T14:03:29.449 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.230463628999999, 2023-07-02T14:03:29.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040768849999999997, 2023-07-02T14:03:29.449 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.450 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:29.450 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.091624+0000", 2023-07-02T14:03:29.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.091624+0000", 2023-07-02T14:03:29.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:29.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:29.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.091624+0000", 2023-07-02T14:03:29.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024253930000000001 2023-07-02T14:03:29.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:29.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094049+0000", 2023-07-02T14:03:29.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1549999999999999e-05 2023-07-02T14:03:29.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:29.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094071+0000", 2023-07-02T14:03:29.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5319999999999995e-06 2023-07-02T14:03:29.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:29.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094080+0000", 2023-07-02T14:03:29.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1773999999999993e-05 2023-07-02T14:03:29.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:29.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094162+0000", 2023-07-02T14:03:29.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5380000000000008e-06 2023-07-02T14:03:29.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:29.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.094172+0000", 2023-07-02T14:03:29.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0015297150000000001 2023-07-02T14:03:29.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:29.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.095701+0000", 2023-07-02T14:03:29.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040768849999999997 2023-07-02T14:03:29.455 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.456 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.456 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.456 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:3073 2.0 e62/46 2:1f6e35fd:::rbd_data.125de1449605.0000000000000111:head v 62'700, mlcod=62'697)", 2023-07-02T14:03:29.456 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:12.104450+0000", 2023-07-02T14:03:29.456 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.217637866, 2023-07-02T14:03:29.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049432629999999998, 2023-07-02T14:03:29.457 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.457 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:29.457 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.104450+0000", 2023-07-02T14:03:29.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.104450+0000", 2023-07-02T14:03:29.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:29.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:29.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.104449+0000", 2023-07-02T14:03:29.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0015084779999999999 2023-07-02T14:03:29.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:29.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.105958+0000", 2023-07-02T14:03:29.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0789e-05 2023-07-02T14:03:29.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:29.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.105969+0000", 2023-07-02T14:03:29.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3319999999999999e-06 2023-07-02T14:03:29.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:29.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.105974+0000", 2023-07-02T14:03:29.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1780999999999999e-05 2023-07-02T14:03:29.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:29.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.106016+0000", 2023-07-02T14:03:29.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8377e-05 2023-07-02T14:03:29.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:29.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.106034+0000", 2023-07-02T14:03:29.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033141310000000001 2023-07-02T14:03:29.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:29.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.109348+0000", 2023-07-02T14:03:29.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5173000000000003e-05 2023-07-02T14:03:29.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:29.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.109393+0000", 2023-07-02T14:03:29.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049432629999999998 2023-07-02T14:03:29.463 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.463 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.464 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.464 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:3081 2.0 e62/46 2:014344e8:::rbd_data.125de1449605.0000000000000114:head v 62'705, mlcod=62'702)", 2023-07-02T14:03:29.464 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:12.118968+0000", 2023-07-02T14:03:29.464 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.203120378000001, 2023-07-02T14:03:29.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042841040000000004, 2023-07-02T14:03:29.464 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.465 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:29.465 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.118968+0000", 2023-07-02T14:03:29.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.118968+0000", 2023-07-02T14:03:29.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:29.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:29.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.118967+0000", 2023-07-02T14:03:29.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013399429999999999 2023-07-02T14:03:29.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:29.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.120307+0000", 2023-07-02T14:03:29.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1103e-05 2023-07-02T14:03:29.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:29.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.120318+0000", 2023-07-02T14:03:29.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3369999999999999e-06 2023-07-02T14:03:29.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:29.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.120323+0000", 2023-07-02T14:03:29.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1192999999999998e-05 2023-07-02T14:03:29.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:29.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.120354+0000", 2023-07-02T14:03:29.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.109e-05 2023-07-02T14:03:29.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:29.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.120376+0000", 2023-07-02T14:03:29.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028409590000000001 2023-07-02T14:03:29.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:29.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.123216+0000", 2023-07-02T14:03:29.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5249000000000001e-05 2023-07-02T14:03:29.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:29.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.123252+0000", 2023-07-02T14:03:29.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042841040000000004 2023-07-02T14:03:29.471 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.471 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.471 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.472 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.7 62/60 [PushOp(2:eed8de36:::rbd_data.125de1449605.0000000000000073:head, version: 58'251, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:eed8de36:::rbd_data.125de1449605.0000000000000073:head@58'251, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:03:29.472 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:12.132531+0000", 2023-07-02T14:03:29.472 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.189556997, 2023-07-02T14:03:29.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037686489999999998, 2023-07-02T14:03:29.472 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.472 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:29.473 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.132531+0000", 2023-07-02T14:03:29.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.132531+0000", 2023-07-02T14:03:29.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:29.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:29.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.132530+0000", 2023-07-02T14:03:29.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00096131099999999996 2023-07-02T14:03:29.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:29.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.133491+0000", 2023-07-02T14:03:29.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0018e-05 2023-07-02T14:03:29.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:29.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.133511+0000", 2023-07-02T14:03:29.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4400000000000005e-06 2023-07-02T14:03:29.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:29.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.133520+0000", 2023-07-02T14:03:29.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3308999999999999e-05 2023-07-02T14:03:29.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:29.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.133593+0000", 2023-07-02T14:03:29.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1190000000000003e-06 2023-07-02T14:03:29.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:29.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.133602+0000", 2023-07-02T14:03:29.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026974099999999999 2023-07-02T14:03:29.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:29.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:12.136300+0000", 2023-07-02T14:03:29.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037686489999999998 2023-07-02T14:03:29.478 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.478 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.479 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.479 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:29.479 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:29.479 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:29.480 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:03:29.480 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 4 2023-07-02T14:03:29.481 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:03:29.482 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:03:29.482 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:29.479+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:29.581 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:29.580+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:29.645 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:29.645 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:29.645 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:03:29.646 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:29.656 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:03:29.697 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:29.696+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:29.782 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:29.781+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:29.819 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:29.819 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:29.820 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:29.820 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:29.820 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:29.830 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:03:29.914 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:29.913+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:29.984 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:29.983+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:29.996 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:29.996 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:29.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:29.997 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:29.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.997 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:29.997 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:29.997 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.703748654000002, 2023-07-02T14:03:29.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:03:29.997 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:29.998 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:29.998 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:29.998 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:29.998 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:29.998 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:03:29.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.998 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:29.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:29.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:29.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:29.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:29.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:29.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:29.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:30.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:30.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:30.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:03:30.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:03:30.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:30.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:03:30.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:03:30.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:30.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:03:30.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:03:30.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:30.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:03:30.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:03:30.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:30.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:03:30.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:03:30.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:30.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:03:30.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:03:30.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:30.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:03:30.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:03:30.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:30.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:03:30.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:03:30.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:30.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:03:30.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:03:30.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:30.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:03:30.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:03:30.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:30.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:03:30.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:03:30.007 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.007 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.007 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.008 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:30.008 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:30.008 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.703513631, 2023-07-02T14:03:30.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:03:30.008 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:30.008 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:30.008 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:30.008 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:30.009 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:30.009 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:03:30.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.009 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:30.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:30.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:30.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:30.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:30.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:30.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:30.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:30.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:03:30.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:03:30.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:30.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:03:30.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:03:30.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:30.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:03:30.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:03:30.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:30.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:03:30.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:03:30.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:30.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:03:30.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:03:30.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:30.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:03:30.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:03:30.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:30.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:03:30.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:03:30.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:30.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:03:30.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:03:30.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:30.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:03:30.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:03:30.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:30.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:03:30.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:03:30.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:30.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:03:30.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:03:30.017 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.017 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.018 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.018 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:30.018 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:30.018 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.703314497000001, 2023-07-02T14:03:30.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:03:30.018 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:30.019 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:30.019 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:30.019 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:30.019 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:30.019 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:03:30.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.019 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:30.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:30.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:30.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:30.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:30.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:30.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:30.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:30.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:03:30.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:03:30.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:30.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:03:30.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:03:30.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:30.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:03:30.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:03:30.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:30.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:03:30.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:03:30.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:30.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:03:30.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:03:30.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:30.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:03:30.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:03:30.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:30.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:03:30.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:03:30.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:30.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:03:30.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:03:30.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:30.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:03:30.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:03:30.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:30.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:03:30.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:03:30.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:30.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:03:30.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:03:30.028 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.028 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.028 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.028 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:30.029 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:30.029 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.702949373000003, 2023-07-02T14:03:30.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:03:30.029 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:30.029 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:30.029 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:30.029 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:30.029 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:30.030 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:03:30.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.030 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:30.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:30.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:30.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:30.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:30.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:30.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:30.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:30.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:03:30.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:03:30.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:30.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:03:30.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:03:30.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:30.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:03:30.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:03:30.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:30.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:03:30.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:03:30.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:30.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:03:30.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:03:30.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:30.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:03:30.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:03:30.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:30.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:03:30.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:03:30.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:30.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:03:30.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:03:30.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:30.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:03:30.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:03:30.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:30.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:03:30.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:03:30.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:30.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:03:30.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:03:30.039 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.039 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.039 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.040 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:30.040 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:30.040 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.702868312, 2023-07-02T14:03:30.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:03:30.040 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:30.041 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:30.041 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:30.041 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:30.041 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:30.041 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:03:30.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.041 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:30.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:30.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:30.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:30.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:30.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:30.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:30.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:30.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:03:30.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:03:30.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:30.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:03:30.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:03:30.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:30.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:03:30.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:03:30.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:30.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:03:30.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:03:30.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:30.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:03:30.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:03:30.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:30.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:03:30.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:03:30.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:30.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:03:30.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:03:30.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:30.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:03:30.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:03:30.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:30.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:03:30.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:03:30.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:30.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:03:30.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:03:30.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:30.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:03:30.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:03:30.050 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.050 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.050 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.050 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:30.050 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:30.051 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.702776641, 2023-07-02T14:03:30.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:03:30.051 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:30.051 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:30.051 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:30.051 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:30.051 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:30.052 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:03:30.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.052 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:30.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:30.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:30.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:30.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:30.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:30.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:30.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:30.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:03:30.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:03:30.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:30.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:03:30.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:03:30.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:30.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:03:30.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:03:30.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:30.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:03:30.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:03:30.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:30.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:03:30.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:03:30.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:30.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:03:30.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:03:30.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:30.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:03:30.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:03:30.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:30.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:03:30.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:03:30.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:30.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:03:30.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:03:30.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:30.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:03:30.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:03:30.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:30.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:03:30.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:03:30.060 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.060 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.060 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.061 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:30.061 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:30.061 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.702437199000002, 2023-07-02T14:03:30.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:03:30.061 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:30.061 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:30.062 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:30.062 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:30.062 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:30.062 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:03:30.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.062 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:30.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:30.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:30.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:30.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:30.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:30.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:30.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:30.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:03:30.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:03:30.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:30.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:03:30.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:03:30.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:30.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:03:30.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:03:30.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:30.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:03:30.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:03:30.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:30.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:03:30.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:03:30.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:30.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:03:30.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:03:30.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:30.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:03:30.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:03:30.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:30.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:03:30.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:03:30.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:30.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:03:30.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:03:30.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:30.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:03:30.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:03:30.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:30.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:03:30.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:03:30.071 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.071 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.071 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.071 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:30.071 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:30.071 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.702251146000002, 2023-07-02T14:03:30.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:03:30.072 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:30.072 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:30.072 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:30.072 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:30.072 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:30.072 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:03:30.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.073 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:30.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:30.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:30.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:30.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:30.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:30.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:30.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:30.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:30.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:03:30.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:30.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:03:30.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:03:30.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:30.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:03:30.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:03:30.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:30.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:03:30.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:03:30.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:30.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:03:30.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:03:30.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:30.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:03:30.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:03:30.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:30.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:03:30.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:03:30.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:30.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:03:30.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:03:30.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:30.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:03:30.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:03:30.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:30.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:03:30.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:03:30.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:30.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:03:30.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:03:30.081 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.081 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.081 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.082 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:30.082 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:30.082 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.702093343999998, 2023-07-02T14:03:30.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:03:30.082 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:30.082 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:30.082 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:30.082 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:30.083 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:30.083 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:03:30.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.083 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:30.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:30.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:30.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:30.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:30.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:30.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:30.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:30.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:03:30.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:03:30.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:30.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:03:30.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:03:30.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:30.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:03:30.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:03:30.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:30.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:03:30.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:03:30.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:30.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:03:30.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:03:30.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:30.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:03:30.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:03:30.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:30.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:03:30.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:03:30.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:30.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:03:30.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:03:30.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:30.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:03:30.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:03:30.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:30.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:03:30.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:03:30.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:30.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:03:30.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:03:30.091 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.091 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.092 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.092 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:30.092 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:30.092 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.701901562000003, 2023-07-02T14:03:30.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:03:30.092 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:30.093 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:30.093 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:30.093 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:30.093 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:30.093 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:03:30.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.093 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:30.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:30.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:30.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:30.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:30.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:30.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:30.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:30.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:03:30.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:03:30.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:30.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:03:30.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:03:30.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:30.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:03:30.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:03:30.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:30.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:03:30.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:03:30.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:30.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:03:30.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:03:30.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:30.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:03:30.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:03:30.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:30.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:03:30.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:03:30.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:30.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:03:30.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:03:30.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:30.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:03:30.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:03:30.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:30.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:03:30.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:03:30.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:30.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:03:30.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:03:30.102 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.102 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.102 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.102 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:30.103 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:30.103 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.701725902, 2023-07-02T14:03:30.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:03:30.103 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:30.103 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:30.103 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:30.103 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:30.103 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:30.104 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:03:30.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.104 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:30.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:30.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:30.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:30.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:30.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:30.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:30.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:30.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:30.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:03:30.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:30.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:03:30.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:03:30.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:30.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:03:30.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:03:30.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:30.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:03:30.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:03:30.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:30.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:03:30.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:03:30.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:30.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:03:30.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:03:30.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:30.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:03:30.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:03:30.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:30.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:03:30.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:03:30.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:30.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:03:30.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:03:30.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:30.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:03:30.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:03:30.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:30.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:03:30.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:03:30.112 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.112 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.112 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.113 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:30.113 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:30.113 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.701631714000001, 2023-07-02T14:03:30.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:03:30.113 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:30.113 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:30.113 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:30.114 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:30.114 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:30.114 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:03:30.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.114 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:30.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:30.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:30.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:30.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:30.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:30.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:30.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:30.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:03:30.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:03:30.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:30.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:03:30.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:03:30.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:30.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:03:30.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:03:30.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:30.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:03:30.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:03:30.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:30.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:03:30.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:03:30.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:30.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:03:30.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:03:30.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:30.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:03:30.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:03:30.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:30.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:03:30.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:03:30.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:30.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:03:30.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:03:30.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:30.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:03:30.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:03:30.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:30.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:03:30.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:03:30.122 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.123 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.123 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.123 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:30.123 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:30.123 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.701428327999999, 2023-07-02T14:03:30.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:03:30.124 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:30.124 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:30.124 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:30.124 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:30.124 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:30.124 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:03:30.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.125 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:30.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:30.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:30.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:30.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:30.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:30.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:30.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:30.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:03:30.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:03:30.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:30.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:03:30.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:03:30.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:30.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:03:30.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:03:30.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:30.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:03:30.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:03:30.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:30.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:03:30.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:03:30.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:30.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:03:30.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:03:30.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:30.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:03:30.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:03:30.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:30.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:03:30.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:03:30.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:30.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:03:30.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:03:30.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:30.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:03:30.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:03:30.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:30.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:03:30.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:03:30.133 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.133 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.133 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.133 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:30.134 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:30.134 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.701277672000003, 2023-07-02T14:03:30.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:03:30.134 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:30.134 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:30.134 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:30.134 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:30.135 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:30.135 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:03:30.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.135 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:30.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:30.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:30.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:30.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:30.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:30.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:30.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:30.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:03:30.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:03:30.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:30.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:03:30.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:03:30.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:30.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:03:30.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:03:30.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:30.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:03:30.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:03:30.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:30.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:03:30.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:03:30.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:30.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:03:30.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:03:30.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:30.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:03:30.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:03:30.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:30.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:03:30.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:03:30.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:30.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:03:30.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:03:30.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:30.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:03:30.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:03:30.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:30.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:03:30.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:03:30.143 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.143 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.143 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.144 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:30.144 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:30.144 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.701204969000003, 2023-07-02T14:03:30.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:03:30.144 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:30.144 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:30.145 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:30.145 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:30.145 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:30.145 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:03:30.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.145 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:30.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:30.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:30.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:30.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:30.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:30.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:30.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:30.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:30.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:03:30.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:30.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:03:30.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:03:30.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:30.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:03:30.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:03:30.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:30.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:03:30.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:03:30.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:30.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:03:30.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:03:30.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:30.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:03:30.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:03:30.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:30.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:03:30.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:03:30.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:30.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:03:30.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:03:30.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:30.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:03:30.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:03:30.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:30.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:03:30.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:03:30.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:30.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:03:30.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:03:30.154 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.154 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.154 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.154 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:30.154 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:30.154 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.701087979, 2023-07-02T14:03:30.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:03:30.155 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:30.155 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:30.155 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:30.155 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:30.155 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:30.155 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:03:30.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.156 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:30.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:30.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:30.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:30.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:30.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:30.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:30.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:30.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:30.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:03:30.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:30.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:03:30.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:03:30.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:30.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:03:30.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:03:30.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:30.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:03:30.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:03:30.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:30.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:03:30.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:03:30.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:30.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:03:30.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:03:30.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:30.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:03:30.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:03:30.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:30.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:03:30.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:03:30.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:30.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:03:30.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:03:30.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:30.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:03:30.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:03:30.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:30.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:03:30.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:03:30.164 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.164 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.164 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.165 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:30.165 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:30.165 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.700938528000002, 2023-07-02T14:03:30.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:03:30.165 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:30.165 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:30.165 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:30.165 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:30.166 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:30.166 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:03:30.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.166 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:30.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:30.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:30.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:30.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:30.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:30.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:30.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:30.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:03:30.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:03:30.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:30.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:03:30.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:03:30.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:30.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:03:30.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:03:30.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:30.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:03:30.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:03:30.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:30.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:03:30.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:03:30.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:30.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:03:30.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:03:30.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:30.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:03:30.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:03:30.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:30.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:03:30.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:03:30.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:30.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:03:30.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:03:30.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:30.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:03:30.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:03:30.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:30.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:03:30.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:03:30.174 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.174 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.175 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.175 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:30.175 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:30.175 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.679462555000001, 2023-07-02T14:03:30.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:03:30.175 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:30.176 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:30.176 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:30.176 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:30.176 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:30.176 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:03:30.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.176 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:30.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:30.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:30.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:30.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:30.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:30.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:30.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:30.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:03:30.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:03:30.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:30.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:03:30.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:03:30.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:30.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:03:30.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:03:30.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:30.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:03:30.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:03:30.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:30.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:03:30.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:03:30.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:30.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:03:30.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:03:30.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:30.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:03:30.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:03:30.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:30.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:03:30.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:03:30.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:30.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:03:30.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:03:30.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:30.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:03:30.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:03:30.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:30.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:03:30.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:03:30.185 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.185 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.185 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.185 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:30.185 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:30.186 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.679330018999998, 2023-07-02T14:03:30.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:03:30.186 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:30.186 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:30.186 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:30.186 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:30.186 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:30.186 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:03:30.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.187 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:30.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:30.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:30.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:30.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:30.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:30.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:30.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:30.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:03:30.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:03:30.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:30.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:03:30.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:03:30.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:30.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:03:30.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:03:30.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:30.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:03:30.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:03:30.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:30.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:03:30.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:03:30.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:30.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:03:30.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:03:30.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:30.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:03:30.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:03:30.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:30.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:03:30.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:03:30.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:30.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:03:30.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:03:30.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:30.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:03:30.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:03:30.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:30.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:03:30.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:03:30.195 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.195 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.195 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.196 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:30.196 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:30.196 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.617993358, 2023-07-02T14:03:30.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:03:30.196 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:30.196 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:30.196 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:30.197 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:30.197 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:30.197 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:03:30.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.197 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:30.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:30.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:30.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:30.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:30.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:30.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:30.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.199 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:30.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:03:30.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:03:30.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.199 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:30.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:03:30.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:03:30.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:30.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:03:30.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:03:30.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:30.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:03:30.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:03:30.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:30.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:03:30.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:03:30.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:30.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:03:30.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:03:30.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:30.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:03:30.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:03:30.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:30.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:03:30.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:03:30.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:30.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:03:30.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:03:30.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:30.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:03:30.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:03:30.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:30.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:30.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:30.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:03:30.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:03:30.205 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.205 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.206 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.206 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:30.206 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:30.206 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:30.207 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:30.206+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:30.208 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:03:30.308 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:30.307+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:30.346 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:30.346 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:30.347 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:30.347 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:30.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:03:30.409 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:30.409+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:30.410 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:03:30.483 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:30.484 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:30.484 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:30.484 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:30.484 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:30.494 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:03:30.538 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:30.538+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:30.551 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:30.569 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:30.587 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:30.606 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:30.610 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:30.610+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:30.634 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:30.635 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:30.641 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:30.648 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:03:30.661 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:03:30.662 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:03:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:03:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:03:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:03:30.663 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:03:30.715 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:30.715+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:30.786 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:30.786 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:30.796 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:03:30.812 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:30.811+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:30.913 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:30.913+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:30.961 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:30.961 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:30.971 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:03:31.014 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:31.013+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:31.114 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:31.114+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:31.135 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:31.135 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:31.146 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:03:31.215 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:31.215+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:31.313 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:31.313 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:31.316 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:31.315+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:31.323 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:03:31.417 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:31.416+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:31.447 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:31.448 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:31.458 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:03:31.483 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:03:31.543 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:31.543+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:31.585 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:31.585 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:31.596 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:03:31.637 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:31.637+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:31.720 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:31.720+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:31.734 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:31.735 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:31.745 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:03:31.821 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:31.821+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:31.870 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:31.870 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:31.881 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:03:31.921 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:31.921+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:32.004 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:32.005 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:32.014 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:03:32.058 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:32.057+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:32.123 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:32.123+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:32.138 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:32.139 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:32.149 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:03:32.182 INFO:tasks.workunit.client.0.smithi163.stderr:+ tar -C b3fbb028-e267-4d25-a750-93ba18a2316a-mnt -xzf b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4.tar.gz 2023-07-02T14:03:32.232 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:32.231+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:32.313 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:32.313 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:32.323 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:03:32.364 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:32.363+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:32.377 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2023-07-02T14:03:32.377 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:03:32.378 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 4] 2023-07-02T14:03:32.378 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 2023-07-02T14:03:32.382 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:03:32.435 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:32.435+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:32.487 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:32.487 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:32.497 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:03:32.527 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:32.527+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:32.621 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:32.621 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:32.628 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:32.627+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:32.631 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:03:32.675 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:03:32.688 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-07-02T14:03:32.688+0000 7ff490b05700 -1 mon.a@0(leader).osd e74 definitely_dead 0 2023-07-02T14:03:32.729 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:32.728+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:32.753 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:32.754 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:32.764 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:03:32.812 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:32.830 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:32.846 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:32.847 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:32.846+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:32.864 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:32.883 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:32.901 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:32.903 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:03:32.903 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:03:32.903 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:03:32.903 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:03:32.903 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:03:32.903 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:03:32.931 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:32.932 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:32.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:32.932 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:32.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.932 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:32.932 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:32.932 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 53.639001825000001, 2023-07-02T14:03:32.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:03:32.933 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:32.933 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:32.933 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:32.933 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:32.933 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:32.933 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:03:32.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.934 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:32.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:32.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:32.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:32.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:32.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:32.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:32.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:32.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:03:32.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:03:32.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:32.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:03:32.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:03:32.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:32.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:03:32.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:03:32.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:32.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:03:32.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:03:32.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:32.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:03:32.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:03:32.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:32.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:03:32.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:03:32.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:32.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:03:32.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:03:32.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:32.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:03:32.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:03:32.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:32.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:03:32.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:03:32.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:32.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:03:32.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:03:32.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:32.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:03:32.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:03:32.942 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:32.942 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:32.942 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:32.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.943 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:32.943 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:32.943 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 53.638766801999999, 2023-07-02T14:03:32.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:03:32.943 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:32.943 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:32.943 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:32.943 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:32.944 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:32.944 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:03:32.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.944 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:32.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:32.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:32.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:32.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:32.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:32.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:32.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:32.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:03:32.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:03:32.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:32.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:03:32.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:03:32.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:32.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:03:32.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:03:32.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:32.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:03:32.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:03:32.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:32.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:03:32.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:03:32.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:32.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:03:32.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:03:32.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:32.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:03:32.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:03:32.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:32.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:03:32.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:03:32.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:32.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:03:32.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:03:32.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:32.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:03:32.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:03:32.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:32.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:03:32.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:03:32.952 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:32.952 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:32.952 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:32.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.953 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:32.953 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:32.953 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 53.638567668, 2023-07-02T14:03:32.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:03:32.953 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:32.954 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:32.954 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:32.954 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:32.954 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:32.954 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:03:32.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.954 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:32.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:32.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:32.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:32.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:32.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:32.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:32.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:32.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:03:32.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:03:32.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:32.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:03:32.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:03:32.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:32.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:03:32.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:03:32.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:32.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:03:32.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:03:32.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:32.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:03:32.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:03:32.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:32.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:03:32.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:03:32.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:32.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:03:32.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:03:32.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:32.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:03:32.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:03:32.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:32.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:03:32.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:03:32.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:32.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:03:32.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:03:32.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:32.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:03:32.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:03:32.963 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:32.963 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:32.963 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:32.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.963 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:32.963 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:32.963 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 53.638202544000002, 2023-07-02T14:03:32.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:03:32.964 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:32.964 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:32.964 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:32.964 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:32.964 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:32.964 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:03:32.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.965 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:32.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:32.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:32.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:32.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:32.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:32.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:32.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:32.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:03:32.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:03:32.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:32.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:03:32.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:03:32.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:32.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:03:32.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:03:32.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:32.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:03:32.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:03:32.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:32.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:03:32.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:03:32.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:32.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:03:32.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:03:32.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:32.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:03:32.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:03:32.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:32.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:03:32.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:03:32.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:32.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:03:32.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:03:32.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:32.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:03:32.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:03:32.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:32.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:03:32.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:03:32.973 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:32.973 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:32.973 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:32.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.974 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:32.974 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:32.974 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 53.638121482999999, 2023-07-02T14:03:32.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:03:32.974 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:32.974 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:32.974 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:32.974 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:32.975 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:32.975 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:03:32.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.975 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:32.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:32.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:32.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:32.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:32.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:32.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:32.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:32.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:03:32.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:03:32.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:32.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:03:32.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:03:32.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:32.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:03:32.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:03:32.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:32.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:03:32.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:03:32.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:32.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:03:32.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:03:32.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:32.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:03:32.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:03:32.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:32.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:03:32.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:03:32.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:32.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:03:32.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:03:32.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:32.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:03:32.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:03:32.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:32.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:03:32.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:03:32.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:32.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:03:32.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:03:32.983 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:32.983 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:32.983 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:32.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.984 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:32.984 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:32.984 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 53.638029811999999, 2023-07-02T14:03:32.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:03:32.984 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:32.985 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:32.985 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:32.985 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:32.985 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:32.985 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:03:32.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.985 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:32.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:32.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:32.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:32.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:32.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:32.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:32.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:32.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:03:32.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:03:32.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:32.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:03:32.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:03:32.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:32.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:03:32.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:03:32.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:32.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:03:32.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:03:32.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:32.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:03:32.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:03:32.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:32.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:03:32.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:03:32.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:32.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:03:32.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:03:32.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:32.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:03:32.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:03:32.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:32.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:03:32.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:03:32.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:32.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:03:32.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:03:32.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:32.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:03:32.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:03:32.994 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:32.994 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:32.994 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:32.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.994 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:32.994 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:32.994 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 53.637690370000001, 2023-07-02T14:03:32.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:03:32.995 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:32.995 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:32.995 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:32.995 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:32.995 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:32.995 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:03:32.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.996 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:32.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:32.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:32.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:32.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:32.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:32.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:32.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:32.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:03:32.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:03:32.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:32.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:03:32.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:03:32.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:32.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:03:32.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:03:32.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:32.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:32.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:32.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:03:32.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:03:32.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:33.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:03:33.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:03:33.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:33.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:03:33.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:03:33.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:33.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:03:33.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:03:33.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:33.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:03:33.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:03:33.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:03:33.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:03:33.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:33.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:03:33.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:03:33.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:33.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:03:33.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:03:33.004 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.004 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:33.004 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.005 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:33.005 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:33.005 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 53.637504317000001, 2023-07-02T14:03:33.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:03:33.005 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:33.005 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.005 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:33.006 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:33.006 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.006 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:03:33.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.006 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:33.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:33.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:33.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:33.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:33.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:33.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:33.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:33.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:33.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:03:33.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:33.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:03:33.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:03:33.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:33.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:03:33.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:03:33.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:03:33.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:03:33.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:33.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:03:33.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:03:33.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:33.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:03:33.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:03:33.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:33.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:03:33.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:03:33.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:33.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:03:33.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:03:33.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:03:33.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:03:33.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:33.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:03:33.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:03:33.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:33.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:03:33.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:03:33.014 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.014 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:33.015 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.015 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:33.015 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:33.015 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 53.637346514999997, 2023-07-02T14:03:33.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:03:33.015 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:33.016 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.016 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:33.016 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:33.016 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.016 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:03:33.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.016 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:33.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:33.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:33.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:33.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:33.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:33.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:33.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:33.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:03:33.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:03:33.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:33.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:03:33.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:03:33.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:33.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:03:33.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:03:33.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:03:33.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:03:33.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:33.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:03:33.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:03:33.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:33.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:03:33.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:03:33.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:33.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:03:33.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:03:33.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:33.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:03:33.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:03:33.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:03:33.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:03:33.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:33.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:03:33.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:03:33.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:33.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:03:33.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:03:33.025 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.025 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:33.025 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.025 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:33.025 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:33.025 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 53.637154733000003, 2023-07-02T14:03:33.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:03:33.026 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:33.026 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.026 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:33.026 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:33.026 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.026 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:03:33.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.027 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:33.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:33.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:33.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:33.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:33.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:33.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:33.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:33.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:03:33.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:03:33.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:33.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:03:33.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:03:33.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:33.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:03:33.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:03:33.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:03:33.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:03:33.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:33.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:03:33.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:03:33.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:33.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:03:33.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:03:33.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:33.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:03:33.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:03:33.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:03:33.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:03:33.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:33.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:03:33.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:03:33.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:33.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:03:33.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:03:33.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:33.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:03:33.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:03:33.035 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.035 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:33.035 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.036 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:33.036 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:33.036 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 53.636979072999999, 2023-07-02T14:03:33.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:03:33.036 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:33.036 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.036 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:33.036 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:33.037 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.037 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:03:33.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.037 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:33.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:33.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:33.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:33.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:33.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:33.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:33.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:33.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:33.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:03:33.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:33.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:03:33.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:03:33.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:33.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:03:33.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:03:33.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:03:33.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:03:33.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:33.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:03:33.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:03:33.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:33.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:03:33.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:03:33.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:33.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:03:33.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:03:33.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:03:33.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:03:33.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:33.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:03:33.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:03:33.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:33.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:03:33.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:03:33.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:33.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:03:33.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:03:33.045 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.045 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:33.045 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.046 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:33.046 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:33.046 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 53.636884885000001, 2023-07-02T14:03:33.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:03:33.046 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:33.046 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.047 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:33.047 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:33.047 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.047 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:03:33.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.047 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:33.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:33.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:33.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:33.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:33.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:33.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:33.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:33.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:03:33.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:03:33.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:33.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:03:33.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:03:33.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:33.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:03:33.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:03:33.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:03:33.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:03:33.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:33.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:03:33.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:03:33.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:33.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:03:33.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:03:33.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:33.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:03:33.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:03:33.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:03:33.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:03:33.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:33.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:03:33.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:03:33.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:33.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:03:33.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:03:33.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:33.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:03:33.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:03:33.055 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.056 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:33.056 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.056 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:33.056 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:33.056 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 53.636681498999998, 2023-07-02T14:03:33.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:03:33.057 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:33.057 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.057 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:33.057 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:33.057 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.057 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:03:33.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.058 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:33.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:33.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:33.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:33.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:33.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:33.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:33.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:33.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:03:33.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:03:33.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:33.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:03:33.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:03:33.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:33.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:03:33.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:03:33.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:03:33.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:03:33.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:33.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:03:33.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:03:33.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:33.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:03:33.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:03:33.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:33.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:03:33.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:03:33.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:03:33.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:03:33.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:33.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:03:33.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:03:33.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:33.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:03:33.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:03:33.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:33.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:03:33.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:03:33.066 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.066 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:33.066 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.066 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:33.067 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:33.067 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 53.636530843000003, 2023-07-02T14:03:33.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:03:33.067 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:33.067 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.067 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:33.067 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:33.067 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.068 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:03:33.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.068 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:33.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:33.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:33.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:33.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:33.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:33.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:33.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:33.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:03:33.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:03:33.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:33.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:03:33.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:03:33.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:33.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:03:33.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:03:33.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:03:33.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:03:33.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:33.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:03:33.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:03:33.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:33.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:03:33.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:03:33.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:33.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:03:33.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:03:33.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:03:33.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:03:33.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:33.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:03:33.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:03:33.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:33.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:03:33.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:03:33.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:33.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:03:33.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:03:33.076 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.076 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:33.076 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.077 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:33.077 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:33.077 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 53.636458140000002, 2023-07-02T14:03:33.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:03:33.077 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:33.077 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.078 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:33.078 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:33.078 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.078 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:03:33.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.078 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:33.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:33.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:33.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:33.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:33.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:33.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:33.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:33.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:33.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:03:33.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:33.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:03:33.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:03:33.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:33.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:03:33.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:03:33.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:03:33.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:03:33.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:33.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:03:33.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:03:33.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:33.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:03:33.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:03:33.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:33.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:03:33.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:03:33.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:03:33.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:03:33.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:33.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:03:33.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:03:33.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:33.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:03:33.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:03:33.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:33.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:03:33.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:03:33.086 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.087 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:33.087 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.087 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:33.087 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:33.087 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 53.63634115, 2023-07-02T14:03:33.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:03:33.088 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:33.088 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.088 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:33.088 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:33.088 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.088 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:03:33.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.088 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:33.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:33.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:33.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:33.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:33.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:33.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:33.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:33.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:33.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:03:33.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:33.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:03:33.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:03:33.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:33.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:03:33.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:03:33.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:03:33.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:03:33.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:33.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:03:33.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:03:33.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:33.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:03:33.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:03:33.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:33.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:03:33.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:03:33.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:03:33.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:03:33.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:33.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:03:33.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:03:33.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:33.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:03:33.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:03:33.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:33.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:03:33.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:03:33.097 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.097 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:33.097 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.097 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:33.097 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:33.098 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 53.636191699000001, 2023-07-02T14:03:33.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:03:33.098 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:33.098 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.098 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:33.098 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:33.098 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.098 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:03:33.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.099 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:33.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:33.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:33.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:33.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:33.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:33.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:33.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:33.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:03:33.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:03:33.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:33.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:03:33.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:03:33.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:33.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:03:33.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:03:33.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:03:33.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:03:33.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:33.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:03:33.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:03:33.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:33.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:03:33.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:03:33.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:33.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:03:33.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:03:33.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:03:33.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:03:33.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:33.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:03:33.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:03:33.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:33.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:03:33.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:03:33.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:33.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:03:33.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:03:33.107 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.107 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:33.107 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.108 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:33.108 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:33.108 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 53.614715726, 2023-07-02T14:03:33.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:03:33.108 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:33.108 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.108 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:33.109 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:33.109 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.109 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:03:33.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.109 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:33.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:33.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:33.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:33.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:33.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:33.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:33.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:33.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:03:33.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:03:33.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:33.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:03:33.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:03:33.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:33.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:03:33.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:03:33.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:03:33.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:03:33.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:33.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:03:33.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:03:33.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:33.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:03:33.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:03:33.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:33.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:03:33.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:03:33.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:33.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:03:33.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:03:33.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:03:33.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:03:33.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:33.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:03:33.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:03:33.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:33.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:03:33.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:03:33.117 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.117 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:33.118 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.118 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:33.118 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:33.118 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 53.614583189999998, 2023-07-02T14:03:33.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:03:33.118 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:33.119 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.119 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:33.119 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:33.119 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.119 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:03:33.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.119 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:33.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:33.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:33.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:33.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:33.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:33.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:33.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:33.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:03:33.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:03:33.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:33.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:03:33.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:03:33.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:33.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:03:33.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:03:33.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:03:33.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:03:33.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:33.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:03:33.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:03:33.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:33.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:03:33.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:03:33.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:33.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:03:33.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:03:33.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:33.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:03:33.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:03:33.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:03:33.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:03:33.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:33.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:03:33.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:03:33.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:33.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:03:33.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:03:33.128 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.128 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:33.128 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.128 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:33.128 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:33.129 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 53.553246528999999, 2023-07-02T14:03:33.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:03:33.129 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:33.129 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.129 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:33.129 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:33.129 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.129 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:03:33.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.130 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:33.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:33.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:33.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:33.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:33.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:33.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:33.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:33.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:03:33.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:03:33.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:33.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:03:33.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:03:33.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:33.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:03:33.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:03:33.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:03:33.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:03:33.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:33.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:03:33.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:03:33.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:33.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:03:33.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:03:33.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:33.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:03:33.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:03:33.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:33.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:03:33.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:03:33.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:03:33.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:03:33.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:33.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:03:33.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:03:33.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:33.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:33.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:33.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:03:33.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:03:33.138 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.138 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:33.138 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.138 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:33.138 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:33.139 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:33.140 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:32.954+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:33.140 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:33.139+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:33.141 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:03:33.241 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:33.240+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:33.265 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:33.265 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:33.266 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:33.266 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:33.275 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:03:33.341 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:33.341+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:33.385 INFO:teuthology.orchestra.run.smithi016.stderr:marked out osd.2. 2023-07-02T14:03:33.399 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap8 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap9 b3fbb028-e267-4d25-a750-93ba18a2316a@snap9.diff 2023-07-02T14:03:33.403 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:33.403 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:33.403 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:33.403 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:33.403 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:33.413 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:03:33.442 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:33.441+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:33.454 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 51% complete...2023-07-02T14:03:33.453+0000 7fe435e24700 0 -- 172.21.15.163:0/2809024549 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x55b3534c4fd0 msgr2=0x55b3534c7400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:03:33.457 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 100% complete...done. 2023-07-02T14:03:33.460 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:03:33.461 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=10 2023-07-02T14:03:33.461 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap10 2023-07-02T14:03:33.543 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:33.543+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:33.579 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:03:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:03:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2620 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 196608~786432] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:03:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.252895170999999, 2023-07-02T14:03:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966, 2023-07-02T14:03:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2620 2023-07-02T14:03:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:03:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:03:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324571+0000", 2023-07-02T14:03:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016686649999999999 2023-07-02T14:03:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326240+0000", 2023-07-02T14:03:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1844e-05 2023-07-02T14:03:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326252+0000", 2023-07-02T14:03:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7760000000000003e-06 2023-07-02T14:03:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326258+0000", 2023-07-02T14:03:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0634000000000003e-05 2023-07-02T14:03:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326318+0000", 2023-07-02T14:03:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9629999999999993e-06 2023-07-02T14:03:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326328+0000", 2023-07-02T14:03:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.010877045 2023-07-02T14:03:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337205+0000", 2023-07-02T14:03:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022401300000000001 2023-07-02T14:03:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337429+0000", 2023-07-02T14:03:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000139822 2023-07-02T14:03:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337569+0000", 2023-07-02T14:03:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032942119999999999 2023-07-02T14:03:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340863+0000", 2023-07-02T14:03:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021302560000000001 2023-07-02T14:03:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342994+0000", 2023-07-02T14:03:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5634000000000002e-05 2023-07-02T14:03:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343059+0000", 2023-07-02T14:03:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143721 2023-07-02T14:03:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343203+0000", 2023-07-02T14:03:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966 2023-07-02T14:03:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2622 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 983040~36864] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:03:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.251202660000001, 2023-07-02T14:03:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664, 2023-07-02T14:03:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2622 2023-07-02T14:03:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:03:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:03:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326265+0000", 2023-07-02T14:03:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1282000000000001e-05 2023-07-02T14:03:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326286+0000", 2023-07-02T14:03:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9480000000000002e-06 2023-07-02T14:03:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326293+0000", 2023-07-02T14:03:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3610000000000003e-06 2023-07-02T14:03:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326297+0000", 2023-07-02T14:03:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7707999999999994e-05 2023-07-02T14:03:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326385+0000", 2023-07-02T14:03:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8070000000000006e-06 2023-07-02T14:03:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326394+0000", 2023-07-02T14:03:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.014535878 2023-07-02T14:03:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340930+0000", 2023-07-02T14:03:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000154551 2023-07-02T14:03:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341084+0000", 2023-07-02T14:03:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00032375900000000002 2023-07-02T14:03:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341408+0000", 2023-07-02T14:03:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012631066 2023-07-02T14:03:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354039+0000", 2023-07-02T14:03:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5741000000000001e-05 2023-07-02T14:03:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354055+0000", 2023-07-02T14:03:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8794000000000006e-05 2023-07-02T14:03:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354124+0000", 2023-07-02T14:03:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014667700000000001 2023-07-02T14:03:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354270+0000", 2023-07-02T14:03:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664 2023-07-02T14:03:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2727 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~12288] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:03:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.251166885, 2023-07-02T14:03:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999, 2023-07-02T14:03:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2727 2023-07-02T14:03:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:03:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:03:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:03:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3834e-05 2023-07-02T14:03:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326315+0000", 2023-07-02T14:03:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3029999999999996e-06 2023-07-02T14:03:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326321+0000", 2023-07-02T14:03:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7590000000000002e-06 2023-07-02T14:03:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326324+0000", 2023-07-02T14:03:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112872 2023-07-02T14:03:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326437+0000", 2023-07-02T14:03:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2290000000000002e-06 2023-07-02T14:03:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326442+0000", 2023-07-02T14:03:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0155896879999999 2023-07-02T14:03:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342031+0000", 2023-07-02T14:03:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016496100000000001 2023-07-02T14:03:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342196+0000", 2023-07-02T14:03:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021998699999999999 2023-07-02T14:03:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342416+0000", 2023-07-02T14:03:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011869744999999999 2023-07-02T14:03:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354286+0000", 2023-07-02T14:03:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0361e-05 2023-07-02T14:03:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354347+0000", 2023-07-02T14:03:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3312000000000005e-05 2023-07-02T14:03:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354430+0000", 2023-07-02T14:03:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126444 2023-07-02T14:03:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354556+0000", 2023-07-02T14:03:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999 2023-07-02T14:03:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2730 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [set-alloc-hint object_size 4194304 write_size 4194304,write 454656~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:03:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.251141875999998, 2023-07-02T14:03:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999, 2023-07-02T14:03:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2730 2023-07-02T14:03:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:03:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:03:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:03:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6049999999999997e-06 2023-07-02T14:03:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326332+0000", 2023-07-02T14:03:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3000000000000002e-06 2023-07-02T14:03:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326335+0000", 2023-07-02T14:03:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.719e-05 2023-07-02T14:03:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326352+0000", 2023-07-02T14:03:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011583599999999999 2023-07-02T14:03:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326468+0000", 2023-07-02T14:03:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.831e-06 2023-07-02T14:03:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326473+0000", 2023-07-02T14:03:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.016769915 2023-07-02T14:03:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343243+0000", 2023-07-02T14:03:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197352 2023-07-02T14:03:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343440+0000", 2023-07-02T14:03:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197581 2023-07-02T14:03:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343638+0000", 2023-07-02T14:03:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010980139 2023-07-02T14:03:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354618+0000", 2023-07-02T14:03:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056779530000000003 2023-07-02T14:03:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360296+0000", 2023-07-02T14:03:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101109 2023-07-02T14:03:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360397+0000", 2023-07-02T14:03:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000160273 2023-07-02T14:03:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360557+0000", 2023-07-02T14:03:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999 2023-07-02T14:03:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2752 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 1019904~3174400] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:03:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.251095652, 2023-07-02T14:03:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001, 2023-07-02T14:03:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2752 2023-07-02T14:03:33.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.629 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:33.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:33.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:03:33.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:33.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:33.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:03:33.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:33.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:33.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326372+0000", 2023-07-02T14:03:33.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052050380000000004 2023-07-02T14:03:33.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:33.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331577+0000", 2023-07-02T14:03:33.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6832000000000001e-05 2023-07-02T14:03:33.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:33.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331594+0000", 2023-07-02T14:03:33.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8220000000000007e-06 2023-07-02T14:03:33.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331602+0000", 2023-07-02T14:03:33.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9895999999999999e-05 2023-07-02T14:03:33.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331642+0000", 2023-07-02T14:03:33.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:03:33.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331649+0000", 2023-07-02T14:03:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0139141709999999 2023-07-02T14:03:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345563+0000", 2023-07-02T14:03:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113817 2023-07-02T14:03:33.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:33.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345677+0000", 2023-07-02T14:03:33.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018803400000000001 2023-07-02T14:03:33.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:33.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345865+0000", 2023-07-02T14:03:33.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087950359999999991 2023-07-02T14:03:33.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354660+0000", 2023-07-02T14:03:33.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059116669999999998 2023-07-02T14:03:33.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360572+0000", 2023-07-02T14:03:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0992000000000002e-05 2023-07-02T14:03:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360633+0000", 2023-07-02T14:03:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013601600000000001 2023-07-02T14:03:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360769+0000", 2023-07-02T14:03:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001 2023-07-02T14:03:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2759 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 0~454656] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:03:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.245855854999999, 2023-07-02T14:03:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999, 2023-07-02T14:03:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2759 2023-07-02T14:03:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:03:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:03:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331611+0000", 2023-07-02T14:03:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053795999999999998 2023-07-02T14:03:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332149+0000", 2023-07-02T14:03:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4970000000000007e-06 2023-07-02T14:03:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332158+0000", 2023-07-02T14:03:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:03:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332162+0000", 2023-07-02T14:03:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3526000000000001e-05 2023-07-02T14:03:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332185+0000", 2023-07-02T14:03:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:03:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332189+0000", 2023-07-02T14:03:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0186295910000001 2023-07-02T14:03:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350819+0000", 2023-07-02T14:03:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000111123 2023-07-02T14:03:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350930+0000", 2023-07-02T14:03:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019575300000000001 2023-07-02T14:03:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.351126+0000", 2023-07-02T14:03:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011503298 2023-07-02T14:03:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362629+0000", 2023-07-02T14:03:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085443989999999994 2023-07-02T14:03:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371173+0000", 2023-07-02T14:03:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3547999999999995e-05 2023-07-02T14:03:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371237+0000", 2023-07-02T14:03:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790300000000001 2023-07-02T14:03:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371375+0000", 2023-07-02T14:03:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999 2023-07-02T14:03:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2760 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 458752~131072] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:03:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.245299670000001, 2023-07-02T14:03:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001, 2023-07-02T14:03:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2760 2023-07-02T14:03:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:03:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:03:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332168+0000", 2023-07-02T14:03:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013281100000000001 2023-07-02T14:03:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332301+0000", 2023-07-02T14:03:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7389999999999999e-06 2023-07-02T14:03:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332308+0000", 2023-07-02T14:03:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0639999999999998e-06 2023-07-02T14:03:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332312+0000", 2023-07-02T14:03:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1735000000000001e-05 2023-07-02T14:03:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332333+0000", 2023-07-02T14:03:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8342000000000002e-05 2023-07-02T14:03:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332352+0000", 2023-07-02T14:03:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02234782 2023-07-02T14:03:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354699+0000", 2023-07-02T14:03:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110371 2023-07-02T14:03:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354810+0000", 2023-07-02T14:03:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022174900000000001 2023-07-02T14:03:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355032+0000", 2023-07-02T14:03:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076053329999999997 2023-07-02T14:03:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:33.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362637+0000", 2023-07-02T14:03:33.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087913609999999993 2023-07-02T14:03:33.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371428+0000", 2023-07-02T14:03:33.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8532999999999998e-05 2023-07-02T14:03:33.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:33.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371477+0000", 2023-07-02T14:03:33.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010990800000000001 2023-07-02T14:03:33.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:33.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371587+0000", 2023-07-02T14:03:33.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001 2023-07-02T14:03:33.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.662 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.662 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2766 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 589824~45056] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:33.663 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:03:33.663 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.245151045, 2023-07-02T14:03:33.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188, 2023-07-02T14:03:33.663 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:33.663 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.663 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:33.663 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:33.663 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.664 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2766 2023-07-02T14:03:33.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.664 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:33.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:33.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:03:33.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:33.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:33.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:03:33.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:33.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:33.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:03:33.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2247999999999998e-05 2023-07-02T14:03:33.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:33.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332369+0000", 2023-07-02T14:03:33.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4189999999999996e-06 2023-07-02T14:03:33.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:33.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332376+0000", 2023-07-02T14:03:33.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1049999999999999e-06 2023-07-02T14:03:33.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332379+0000", 2023-07-02T14:03:33.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6894000000000003e-05 2023-07-02T14:03:33.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332416+0000", 2023-07-02T14:03:33.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3970000000000002e-06 2023-07-02T14:03:33.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:33.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332422+0000", 2023-07-02T14:03:33.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0231382280000001 2023-07-02T14:03:33.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355560+0000", 2023-07-02T14:03:33.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9079000000000005e-05 2023-07-02T14:03:33.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:33.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355659+0000", 2023-07-02T14:03:33.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018005000000000001 2023-07-02T14:03:33.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:33.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355839+0000", 2023-07-02T14:03:33.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068044439999999998 2023-07-02T14:03:33.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:33.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362644+0000", 2023-07-02T14:03:33.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090155589999999994 2023-07-02T14:03:33.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371659+0000", 2023-07-02T14:03:33.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1297999999999994e-05 2023-07-02T14:03:33.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:33.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371730+0000", 2023-07-02T14:03:33.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013100699999999999 2023-07-02T14:03:33.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:33.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371861+0000", 2023-07-02T14:03:33.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188 2023-07-02T14:03:33.673 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.674 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.674 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.674 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2784 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 0~393216] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:33.674 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:03:33.674 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.245082592999999, 2023-07-02T14:03:33.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999, 2023-07-02T14:03:33.675 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:33.675 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:33.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:33.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.675 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2784 2023-07-02T14:03:33.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.675 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:33.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:33.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:03:33.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:33.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:33.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:03:33.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:33.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:33.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332385+0000", 2023-07-02T14:03:33.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040252400000000002 2023-07-02T14:03:33.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:33.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332787+0000", 2023-07-02T14:03:33.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0208e-05 2023-07-02T14:03:33.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:33.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332798+0000", 2023-07-02T14:03:33.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2023-07-02T14:03:33.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332801+0000", 2023-07-02T14:03:33.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4838000000000001e-05 2023-07-02T14:03:33.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332836+0000", 2023-07-02T14:03:33.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5319999999999998e-06 2023-07-02T14:03:33.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:33.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332839+0000", 2023-07-02T14:03:33.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0234290109999999 2023-07-02T14:03:33.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356268+0000", 2023-07-02T14:03:33.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011165700000000001 2023-07-02T14:03:33.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:33.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356380+0000", 2023-07-02T14:03:33.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022504799999999999 2023-07-02T14:03:33.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:33.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356605+0000", 2023-07-02T14:03:33.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015276201 2023-07-02T14:03:33.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:33.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371881+0000", 2023-07-02T14:03:33.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5496999999999999e-05 2023-07-02T14:03:33.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371957+0000", 2023-07-02T14:03:33.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4570000000000002e-05 2023-07-02T14:03:33.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:33.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372001+0000", 2023-07-02T14:03:33.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3880000000000003e-05 2023-07-02T14:03:33.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:33.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372085+0000", 2023-07-02T14:03:33.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999 2023-07-02T14:03:33.685 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.685 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.685 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.686 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2785 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 405504~446464] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:33.686 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:03:33.686 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.244661142000002, 2023-07-02T14:03:33.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999, 2023-07-02T14:03:33.686 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:33.686 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.686 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:33.687 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:33.687 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.687 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2785 2023-07-02T14:03:33.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.687 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:33.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:33.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:03:33.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:33.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:33.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:03:33.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:33.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:33.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:03:33.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059527500000000004 2023-07-02T14:03:33.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:33.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333402+0000", 2023-07-02T14:03:33.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7059999999999993e-06 2023-07-02T14:03:33.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:33.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333411+0000", 2023-07-02T14:03:33.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0450000000000001e-06 2023-07-02T14:03:33.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333414+0000", 2023-07-02T14:03:33.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5790999999999997e-05 2023-07-02T14:03:33.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:03:33.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3320000000000002e-06 2023-07-02T14:03:33.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:33.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333454+0000", 2023-07-02T14:03:33.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.025310269 2023-07-02T14:03:33.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358764+0000", 2023-07-02T14:03:33.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012414899999999999 2023-07-02T14:03:33.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:33.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358888+0000", 2023-07-02T14:03:33.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000217028 2023-07-02T14:03:33.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:33.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.359105+0000", 2023-07-02T14:03:33.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012788212 2023-07-02T14:03:33.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:33.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371893+0000", 2023-07-02T14:03:33.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012201569999999999 2023-07-02T14:03:33.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373114+0000", 2023-07-02T14:03:33.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4512999999999995e-05 2023-07-02T14:03:33.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:33.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373198+0000", 2023-07-02T14:03:33.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132373 2023-07-02T14:03:33.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:33.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373330+0000", 2023-07-02T14:03:33.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999 2023-07-02T14:03:33.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.697 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.697 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2786 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1310720~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:33.697 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:03:33.698 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.244048449000001, 2023-07-02T14:03:33.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001, 2023-07-02T14:03:33.698 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:33.698 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:33.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:33.698 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.699 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2786 2023-07-02T14:03:33.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.699 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:33.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:33.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:03:33.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:33.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:33.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:03:33.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:33.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:33.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333419+0000", 2023-07-02T14:03:33.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2009999999999999e-06 2023-07-02T14:03:33.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:33.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333427+0000", 2023-07-02T14:03:33.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0210000000000001e-06 2023-07-02T14:03:33.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:33.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333431+0000", 2023-07-02T14:03:33.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3769999999999999e-06 2023-07-02T14:03:33.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333434+0000", 2023-07-02T14:03:33.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1161e-05 2023-07-02T14:03:33.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333485+0000", 2023-07-02T14:03:33.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1549000000000001e-05 2023-07-02T14:03:33.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:33.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333496+0000", 2023-07-02T14:03:33.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0273116339999999 2023-07-02T14:03:33.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360808+0000", 2023-07-02T14:03:33.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000115128 2023-07-02T14:03:33.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:33.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360923+0000", 2023-07-02T14:03:33.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018367700000000001 2023-07-02T14:03:33.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:33.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361107+0000", 2023-07-02T14:03:33.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010796923999999999 2023-07-02T14:03:33.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:33.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371904+0000", 2023-07-02T14:03:33.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014983520000000001 2023-07-02T14:03:33.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373402+0000", 2023-07-02T14:03:33.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0520999999999999e-05 2023-07-02T14:03:33.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:33.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373463+0000", 2023-07-02T14:03:33.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000129095 2023-07-02T14:03:33.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:33.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373592+0000", 2023-07-02T14:03:33.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001 2023-07-02T14:03:33.708 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.709 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.709 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.709 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2787 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1433600~8192] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:33.709 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:03:33.709 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.244031649, 2023-07-02T14:03:33.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999, 2023-07-02T14:03:33.710 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:33.710 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.710 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:33.710 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:33.710 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.710 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2787 2023-07-02T14:03:33.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.710 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:33.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:33.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:03:33.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:33.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:33.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:03:33.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:33.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:33.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333436+0000", 2023-07-02T14:03:33.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2731999999999999e-05 2023-07-02T14:03:33.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:33.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:03:33.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7440000000000001e-06 2023-07-02T14:03:33.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:33.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333456+0000", 2023-07-02T14:03:33.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8650000000000003e-06 2023-07-02T14:03:33.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333460+0000", 2023-07-02T14:03:33.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0077e-05 2023-07-02T14:03:33.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333530+0000", 2023-07-02T14:03:33.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4180000000000001e-06 2023-07-02T14:03:33.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:33.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333533+0000", 2023-07-02T14:03:33.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0284002720000001 2023-07-02T14:03:33.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361933+0000", 2023-07-02T14:03:33.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6781999999999996e-05 2023-07-02T14:03:33.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:33.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362030+0000", 2023-07-02T14:03:33.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018305400000000001 2023-07-02T14:03:33.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:33.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362213+0000", 2023-07-02T14:03:33.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097015109999999995 2023-07-02T14:03:33.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:33.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371915+0000", 2023-07-02T14:03:33.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001742295 2023-07-02T14:03:33.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373657+0000", 2023-07-02T14:03:33.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5980000000000003e-05 2023-07-02T14:03:33.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:33.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373713+0000", 2023-07-02T14:03:33.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011783800000000001 2023-07-02T14:03:33.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:33.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373831+0000", 2023-07-02T14:03:33.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999 2023-07-02T14:03:33.720 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.720 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.720 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.721 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2803 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 851968~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:33.721 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:03:33.721 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.243989797000001, 2023-07-02T14:03:33.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514, 2023-07-02T14:03:33.721 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:33.721 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:33.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:33.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.722 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2803 2023-07-02T14:03:33.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.722 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:33.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:33.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:03:33.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:33.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:33.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:03:33.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:33.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:33.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:03:33.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4943000000000001e-05 2023-07-02T14:03:33.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:33.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333493+0000", 2023-07-02T14:03:33.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2629999999999997e-06 2023-07-02T14:03:33.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:33.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333500+0000", 2023-07-02T14:03:33.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4159999999999997e-06 2023-07-02T14:03:33.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333505+0000", 2023-07-02T14:03:33.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4518999999999998e-05 2023-07-02T14:03:33.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333559+0000", 2023-07-02T14:03:33.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5900000000000008e-06 2023-07-02T14:03:33.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:33.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333568+0000", 2023-07-02T14:03:33.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0291061779999999 2023-07-02T14:03:33.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362674+0000", 2023-07-02T14:03:33.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110412 2023-07-02T14:03:33.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:33.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362784+0000", 2023-07-02T14:03:33.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015384599999999999 2023-07-02T14:03:33.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:33.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362938+0000", 2023-07-02T14:03:33.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091682310000000006 2023-07-02T14:03:33.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:33.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372106+0000", 2023-07-02T14:03:33.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017703230000000001 2023-07-02T14:03:33.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373877+0000", 2023-07-02T14:03:33.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6387000000000002e-05 2023-07-02T14:03:33.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:33.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373923+0000", 2023-07-02T14:03:33.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014099199999999999 2023-07-02T14:03:33.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:33.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374064+0000", 2023-07-02T14:03:33.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514 2023-07-02T14:03:33.732 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.732 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.732 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.733 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2805 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 634880~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:33.733 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:03:33.733 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.243956621999999, 2023-07-02T14:03:33.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814, 2023-07-02T14:03:33.733 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:33.733 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.733 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:33.734 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:33.734 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.734 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2805 2023-07-02T14:03:33.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.734 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:33.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:33.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:03:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:03:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333511+0000", 2023-07-02T14:03:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207092 2023-07-02T14:03:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333718+0000", 2023-07-02T14:03:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5110000000000003e-06 2023-07-02T14:03:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333727+0000", 2023-07-02T14:03:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3270000000000007e-06 2023-07-02T14:03:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333736+0000", 2023-07-02T14:03:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0319999999999999e-05 2023-07-02T14:03:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333756+0000", 2023-07-02T14:03:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1695000000000001e-05 2023-07-02T14:03:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:03:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0295750560000001 2023-07-02T14:03:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363343+0000", 2023-07-02T14:03:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3387999999999994e-05 2023-07-02T14:03:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363437+0000", 2023-07-02T14:03:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018373 2023-07-02T14:03:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363620+0000", 2023-07-02T14:03:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0084948769999999996 2023-07-02T14:03:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372115+0000", 2023-07-02T14:03:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020082170000000001 2023-07-02T14:03:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374123+0000", 2023-07-02T14:03:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8875999999999997e-05 2023-07-02T14:03:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374182+0000", 2023-07-02T14:03:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0001272 2023-07-02T14:03:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374309+0000", 2023-07-02T14:03:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814 2023-07-02T14:03:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2806 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 1359872~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:03:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.243726291000002, 2023-07-02T14:03:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234, 2023-07-02T14:03:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2806 2023-07-02T14:03:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:03:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:03:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333741+0000", 2023-07-02T14:03:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3978999999999999e-05 2023-07-02T14:03:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333755+0000", 2023-07-02T14:03:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:03:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333761+0000", 2023-07-02T14:03:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7010000000000001e-06 2023-07-02T14:03:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333764+0000", 2023-07-02T14:03:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5086000000000003e-05 2023-07-02T14:03:33.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333799+0000", 2023-07-02T14:03:33.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6389999999999992e-06 2023-07-02T14:03:33.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:33.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333806+0000", 2023-07-02T14:03:33.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.030496675 2023-07-02T14:03:33.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364303+0000", 2023-07-02T14:03:33.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011382700000000001 2023-07-02T14:03:33.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:33.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364417+0000", 2023-07-02T14:03:33.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192058 2023-07-02T14:03:33.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:33.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364609+0000", 2023-07-02T14:03:33.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075132130000000004 2023-07-02T14:03:33.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:33.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372122+0000", 2023-07-02T14:03:33.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022731750000000001 2023-07-02T14:03:33.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374395+0000", 2023-07-02T14:03:33.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5277000000000005e-05 2023-07-02T14:03:33.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:33.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374460+0000", 2023-07-02T14:03:33.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017074599999999999 2023-07-02T14:03:33.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:33.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374631+0000", 2023-07-02T14:03:33.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234 2023-07-02T14:03:33.755 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.755 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.756 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 868352~49152] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:33.756 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:03:33.756 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.243699577000001, 2023-07-02T14:03:33.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001, 2023-07-02T14:03:33.756 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:33.757 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.757 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:33.757 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:33.757 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.757 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:03:33.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.757 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:33.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:33.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:03:33.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:33.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:33.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:03:33.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:33.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:33.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:03:33.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6355000000000001e-05 2023-07-02T14:03:33.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:33.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333794+0000", 2023-07-02T14:03:33.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8329999999999996e-06 2023-07-02T14:03:33.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:33.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333801+0000", 2023-07-02T14:03:33.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5419999999999996e-06 2023-07-02T14:03:33.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333807+0000", 2023-07-02T14:03:33.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2023-07-02T14:03:33.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333826+0000", 2023-07-02T14:03:33.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.303e-06 2023-07-02T14:03:33.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:33.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333830+0000", 2023-07-02T14:03:33.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.031137559 2023-07-02T14:03:33.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364967+0000", 2023-07-02T14:03:33.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1390000000000004e-05 2023-07-02T14:03:33.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:33.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365059+0000", 2023-07-02T14:03:33.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017597399999999999 2023-07-02T14:03:33.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:33.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365234+0000", 2023-07-02T14:03:33.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068943950000000002 2023-07-02T14:03:33.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:33.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372129+0000", 2023-07-02T14:03:33.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026189910000000002 2023-07-02T14:03:33.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374748+0000", 2023-07-02T14:03:33.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6613999999999996e-05 2023-07-02T14:03:33.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:33.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374814+0000", 2023-07-02T14:03:33.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502300000000001 2023-07-02T14:03:33.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:33.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374950+0000", 2023-07-02T14:03:33.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001 2023-07-02T14:03:33.767 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.767 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.767 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.768 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2815 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 917504~49152] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:33.768 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:03:33.768 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.239485343999998, 2023-07-02T14:03:33.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998, 2023-07-02T14:03:33.768 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:33.768 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.768 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:33.768 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:33.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.769 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2815 2023-07-02T14:03:33.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.769 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:33.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:33.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:03:33.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:33.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:33.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:03:33.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:03:33.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:33.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337980+0000", 2023-07-02T14:03:33.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017126700000000001 2023-07-02T14:03:33.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:33.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338152+0000", 2023-07-02T14:03:33.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2938999999999999e-05 2023-07-02T14:03:33.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:33.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338164+0000", 2023-07-02T14:03:33.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4448999999999999e-05 2023-07-02T14:03:33.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:03:33.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0281812 2023-07-02T14:03:33.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366370+0000", 2023-07-02T14:03:33.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1840000000000002e-05 2023-07-02T14:03:33.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:33.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366462+0000", 2023-07-02T14:03:33.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000186112 2023-07-02T14:03:33.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:33.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366648+0000", 2023-07-02T14:03:33.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054905709999999996 2023-07-02T14:03:33.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:33.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372139+0000", 2023-07-02T14:03:33.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044954790000000001 2023-07-02T14:03:33.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376634+0000", 2023-07-02T14:03:33.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4621000000000004e-05 2023-07-02T14:03:33.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:33.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376699+0000", 2023-07-02T14:03:33.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012862299999999999 2023-07-02T14:03:33.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:33.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376827+0000", 2023-07-02T14:03:33.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998 2023-07-02T14:03:33.777 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.777 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.778 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.778 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2816 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 917504~65536] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:33.778 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:03:33.778 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.239270964999999, 2023-07-02T14:03:33.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001, 2023-07-02T14:03:33.778 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:33.779 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.779 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:33.779 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:33.779 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.779 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2816 2023-07-02T14:03:33.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.779 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:33.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:33.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:03:33.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:33.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:33.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:03:33.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:33.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:33.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338196+0000", 2023-07-02T14:03:33.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6286999999999999e-05 2023-07-02T14:03:33.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:33.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338243+0000", 2023-07-02T14:03:33.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7030000000000001e-06 2023-07-02T14:03:33.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:33.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338252+0000", 2023-07-02T14:03:33.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:03:33.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338258+0000", 2023-07-02T14:03:33.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032085734999999997 2023-07-02T14:03:33.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370344+0000", 2023-07-02T14:03:33.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108621 2023-07-02T14:03:33.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:33.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370452+0000", 2023-07-02T14:03:33.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014936499999999999 2023-07-02T14:03:33.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:33.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370602+0000", 2023-07-02T14:03:33.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001778012 2023-07-02T14:03:33.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:33.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372380+0000", 2023-07-02T14:03:33.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044876380000000004 2023-07-02T14:03:33.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376867+0000", 2023-07-02T14:03:33.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9870999999999999e-05 2023-07-02T14:03:33.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:33.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376897+0000", 2023-07-02T14:03:33.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011401 2023-07-02T14:03:33.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:33.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.377011+0000", 2023-07-02T14:03:33.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001 2023-07-02T14:03:33.788 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.788 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.788 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.788 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2858 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1441792~2359296] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:33.788 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.378969+0000", 2023-07-02T14:03:33.789 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.198498768, 2023-07-02T14:03:33.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029176497999999999, 2023-07-02T14:03:33.789 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:33.789 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.789 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:33.789 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:33.789 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.789 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2858 2023-07-02T14:03:33.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.790 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:33.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:33.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.378969+0000", 2023-07-02T14:03:33.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:33.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:33.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.378969+0000", 2023-07-02T14:03:33.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:33.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:33.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.378969+0000", 2023-07-02T14:03:33.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011672914 2023-07-02T14:03:33.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:33.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.390642+0000", 2023-07-02T14:03:33.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2140999999999999e-05 2023-07-02T14:03:33.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:33.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.390654+0000", 2023-07-02T14:03:33.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5440000000000004e-06 2023-07-02T14:03:33.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.390660+0000", 2023-07-02T14:03:33.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011227500000000001 2023-07-02T14:03:33.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.390773+0000", 2023-07-02T14:03:33.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100979 2023-07-02T14:03:33.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:33.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.390873+0000", 2023-07-02T14:03:33.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021888100000000001 2023-07-02T14:03:33.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:33.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.391092+0000", 2023-07-02T14:03:33.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015316722 2023-07-02T14:03:33.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:33.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.406409+0000", 2023-07-02T14:03:33.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015790299999999999 2023-07-02T14:03:33.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.407988+0000", 2023-07-02T14:03:33.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7111000000000003e-05 2023-07-02T14:03:33.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:33.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.408045+0000", 2023-07-02T14:03:33.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010076799999999999 2023-07-02T14:03:33.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:33.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.408146+0000", 2023-07-02T14:03:33.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029176497999999999 2023-07-02T14:03:33.798 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.798 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.798 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.799 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2944 2.4 2:25711f4d:::rbd_data.125de1449605.00000000000000fa:head [write 65536~2097152] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:33.799 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.470424+0000", 2023-07-02T14:03:33.799 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.107044351999999, 2023-07-02T14:03:33.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029419590999999998, 2023-07-02T14:03:33.799 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:33.799 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:33.799 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:33.800 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:33.800 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:33.800 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2944 2023-07-02T14:03:33.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.800 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:33.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:33.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.470424+0000", 2023-07-02T14:03:33.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:33.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:33.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.470424+0000", 2023-07-02T14:03:33.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:03:33.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:33.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.470421+0000", 2023-07-02T14:03:33.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0086348350000000004 2023-07-02T14:03:33.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:33.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479056+0000", 2023-07-02T14:03:33.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1942000000000001e-05 2023-07-02T14:03:33.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:33.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479078+0000", 2023-07-02T14:03:33.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8130000000000002e-06 2023-07-02T14:03:33.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:33.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479087+0000", 2023-07-02T14:03:33.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4845999999999995e-05 2023-07-02T14:03:33.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:33.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479152+0000", 2023-07-02T14:03:33.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013724700000000001 2023-07-02T14:03:33.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:33.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479289+0000", 2023-07-02T14:03:33.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000234665 2023-07-02T14:03:33.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:33.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479524+0000", 2023-07-02T14:03:33.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081727950000000001 2023-07-02T14:03:33.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:33.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.487697+0000", 2023-07-02T14:03:33.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011956079999999999 2023-07-02T14:03:33.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:33.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.499653+0000", 2023-07-02T14:03:33.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8978999999999999e-05 2023-07-02T14:03:33.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:33.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.499712+0000", 2023-07-02T14:03:33.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013181299999999999 2023-07-02T14:03:33.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:33.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:33.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:33.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.499844+0000", 2023-07-02T14:03:33.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029419590999999998 2023-07-02T14:03:33.808 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.809 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.809 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.809 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:33.809 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:33.809 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:33.810 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:03:33.811 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:33.809+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:33.912 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:33.911+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:33.976 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:33.976 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:33.976 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:33.977 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:33.987 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:03:34.012 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:34.011+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:34.110 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:34.111 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:34.111 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:34.111 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:34.111 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:34.113 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:34.113+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:34.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:03:34.214 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:34.214+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:34.289 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:34.290 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:03:34.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:03:34.290 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:34.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.291 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:289 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:34.291 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:34.292 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 54.973758314999998, 2023-07-02T14:03:34.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051413158, 2023-07-02T14:03:34.292 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:34.292 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:34.293 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:34.293 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:34.293 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:34.293 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 289 2023-07-02T14:03:34.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.294 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:34.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:34.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:34.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:34.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:34.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:34.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:34.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:34.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:34.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1689e-05 2023-07-02T14:03:34.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:34.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312209+0000", 2023-07-02T14:03:34.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1070000000000005e-06 2023-07-02T14:03:34.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:34.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312217+0000", 2023-07-02T14:03:34.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.634e-06 2023-07-02T14:03:34.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:34.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312222+0000", 2023-07-02T14:03:34.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040512761000000001 2023-07-02T14:03:34.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352735+0000", 2023-07-02T14:03:34.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000119184 2023-07-02T14:03:34.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352854+0000", 2023-07-02T14:03:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021481899999999999 2023-07-02T14:03:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:34.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353069+0000", 2023-07-02T14:03:34.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001601707 2023-07-02T14:03:34.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:34.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354670+0000", 2023-07-02T14:03:34.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088168570000000009 2023-07-02T14:03:34.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:34.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363487+0000", 2023-07-02T14:03:34.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9635999999999997e-05 2023-07-02T14:03:34.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:34.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363537+0000", 2023-07-02T14:03:34.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4292999999999998e-05 2023-07-02T14:03:34.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:34.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363581+0000", 2023-07-02T14:03:34.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051413158 2023-07-02T14:03:34.312 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.312 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.312 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.313 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:290 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:34.313 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:34.314 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 54.973699070000002, 2023-07-02T14:03:34.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051455062000000003, 2023-07-02T14:03:34.314 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:34.315 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:34.315 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:34.315 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:34.315 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 290 2023-07-02T14:03:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6469000000000003e-05 2023-07-02T14:03:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312293+0000", 2023-07-02T14:03:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4630000000000007e-06 2023-07-02T14:03:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312303+0000", 2023-07-02T14:03:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8969999999999999e-06 2023-07-02T14:03:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:34.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312307+0000", 2023-07-02T14:03:34.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041261548000000002 2023-07-02T14:03:34.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353569+0000", 2023-07-02T14:03:34.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9594999999999999e-05 2023-07-02T14:03:34.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:34.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353659+0000", 2023-07-02T14:03:34.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018390600000000001 2023-07-02T14:03:34.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:34.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353843+0000", 2023-07-02T14:03:34.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035173100000000001 2023-07-02T14:03:34.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357360+0000", 2023-07-02T14:03:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062295670000000001 2023-07-02T14:03:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:34.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363589+0000", 2023-07-02T14:03:34.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9854000000000001e-05 2023-07-02T14:03:34.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:34.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363639+0000", 2023-07-02T14:03:34.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3133999999999997e-05 2023-07-02T14:03:34.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363682+0000", 2023-07-02T14:03:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051455062000000003 2023-07-02T14:03:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:291 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 54.973613030999999, 2023-07-02T14:03:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051527860000000002, 2023-07-02T14:03:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 291 2023-07-02T14:03:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2448999999999998e-05 2023-07-02T14:03:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312375+0000", 2023-07-02T14:03:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5259999999999993e-06 2023-07-02T14:03:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312384+0000", 2023-07-02T14:03:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0610000000000003e-06 2023-07-02T14:03:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312389+0000", 2023-07-02T14:03:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042326111 2023-07-02T14:03:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354715+0000", 2023-07-02T14:03:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3998999999999995e-05 2023-07-02T14:03:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354779+0000", 2023-07-02T14:03:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010269500000000001 2023-07-02T14:03:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354882+0000", 2023-07-02T14:03:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002533736 2023-07-02T14:03:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357415+0000", 2023-07-02T14:03:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062765700000000004 2023-07-02T14:03:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363692+0000", 2023-07-02T14:03:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7095000000000001e-05 2023-07-02T14:03:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363749+0000", 2023-07-02T14:03:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2263999999999996e-05 2023-07-02T14:03:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363841+0000", 2023-07-02T14:03:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051527860000000002 2023-07-02T14:03:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.340 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:292 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:34.340 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:03:34.340 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 54.973530783000001, 2023-07-02T14:03:34.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051556056000000003, 2023-07-02T14:03:34.340 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:34.340 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:34.340 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:34.341 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:34.341 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:34.341 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 292 2023-07-02T14:03:34.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.341 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:34.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:34.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:03:34.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:34.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:34.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:03:34.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:34.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:34.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312395+0000", 2023-07-02T14:03:34.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9624e-05 2023-07-02T14:03:34.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:34.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312425+0000", 2023-07-02T14:03:34.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:03:34.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:34.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312432+0000", 2023-07-02T14:03:34.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7550000000000002e-06 2023-07-02T14:03:34.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:34.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312437+0000", 2023-07-02T14:03:34.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.043823036000000003 2023-07-02T14:03:34.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356260+0000", 2023-07-02T14:03:34.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6021999999999997e-05 2023-07-02T14:03:34.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:34.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356346+0000", 2023-07-02T14:03:34.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020076600000000001 2023-07-02T14:03:34.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:34.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356547+0000", 2023-07-02T14:03:34.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020766410000000002 2023-07-02T14:03:34.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:34.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358623+0000", 2023-07-02T14:03:34.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005231192 2023-07-02T14:03:34.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:34.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363854+0000", 2023-07-02T14:03:34.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3423e-05 2023-07-02T14:03:34.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363908+0000", 2023-07-02T14:03:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3825999999999997e-05 2023-07-02T14:03:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363952+0000", 2023-07-02T14:03:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051556056000000003 2023-07-02T14:03:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:293 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:03:34.351 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 54.973484612999997, 2023-07-02T14:03:34.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051614769999999997, 2023-07-02T14:03:34.351 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:34.351 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:34.351 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:34.351 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:34.351 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:34.351 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 293 2023-07-02T14:03:34.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.352 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:34.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:34.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:03:34.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:34.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:34.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:03:34.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:34.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:34.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312441+0000", 2023-07-02T14:03:34.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9609999999999999e-05 2023-07-02T14:03:34.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:34.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312471+0000", 2023-07-02T14:03:34.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7069999999999992e-06 2023-07-02T14:03:34.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:34.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312478+0000", 2023-07-02T14:03:34.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1190000000000002e-06 2023-07-02T14:03:34.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:34.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312485+0000", 2023-07-02T14:03:34.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.044956387 2023-07-02T14:03:34.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357441+0000", 2023-07-02T14:03:34.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4134999999999994e-05 2023-07-02T14:03:34.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:34.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357515+0000", 2023-07-02T14:03:34.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000147408 2023-07-02T14:03:34.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:34.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357663+0000", 2023-07-02T14:03:34.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056472639999999999 2023-07-02T14:03:34.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:34.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363310+0000", 2023-07-02T14:03:34.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00065402600000000004 2023-07-02T14:03:34.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:34.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363964+0000", 2023-07-02T14:03:34.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0497000000000001e-05 2023-07-02T14:03:34.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:34.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364014+0000", 2023-07-02T14:03:34.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2243000000000002e-05 2023-07-02T14:03:34.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:34.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364057+0000", 2023-07-02T14:03:34.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051614769999999997 2023-07-02T14:03:34.363 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.363 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.363 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.364 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:294 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1048576~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:34.365 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:03:34.365 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 54.973436614000001, 2023-07-02T14:03:34.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051679527000000003, 2023-07-02T14:03:34.365 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:34.366 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:34.366 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:34.366 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:34.367 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:34.367 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 294 2023-07-02T14:03:34.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.367 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:34.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:34.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:03:34.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:34.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:34.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:03:34.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:34.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:34.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312489+0000", 2023-07-02T14:03:34.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8204999999999998e-05 2023-07-02T14:03:34.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:34.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312517+0000", 2023-07-02T14:03:34.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7759999999999997e-06 2023-07-02T14:03:34.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:34.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312525+0000", 2023-07-02T14:03:34.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8730000000000001e-06 2023-07-02T14:03:34.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:34.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312532+0000", 2023-07-02T14:03:34.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.045442206999999998 2023-07-02T14:03:34.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357974+0000", 2023-07-02T14:03:34.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4422000000000002e-05 2023-07-02T14:03:34.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:34.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358029+0000", 2023-07-02T14:03:34.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149519 2023-07-02T14:03:34.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:34.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358178+0000", 2023-07-02T14:03:34.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051779390000000003 2023-07-02T14:03:34.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:34.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363356+0000", 2023-07-02T14:03:34.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00070707699999999997 2023-07-02T14:03:34.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:34.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364063+0000", 2023-07-02T14:03:34.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6835999999999999e-05 2023-07-02T14:03:34.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:34.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364090+0000", 2023-07-02T14:03:34.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9263000000000003e-05 2023-07-02T14:03:34.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:34.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364169+0000", 2023-07-02T14:03:34.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051679527000000003 2023-07-02T14:03:34.377 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.378 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.378 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.378 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:03:34.378 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:34.378 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.833826412000001, 2023-07-02T14:03:34.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:03:34.379 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:34.379 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:34.379 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:34.379 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:34.379 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:34.379 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:03:34.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.379 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:34.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:34.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:34.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:34.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:34.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:34.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:34.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:34.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:03:34.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:03:34.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:34.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:03:34.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:03:34.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:34.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:03:34.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:03:34.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:34.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:03:34.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:03:34.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:03:34.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:03:34.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:34.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:03:34.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:03:34.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:03:34.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:03:34.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:34.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:03:34.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:03:34.386 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.386 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.387 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.387 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:03:34.387 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:03:34.387 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.963133404000001, 2023-07-02T14:03:34.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:03:34.387 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:34.388 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:03:34.388 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:34.388 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:03:34.388 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:03:34.388 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:03:34.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.388 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:34.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:34.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:03:34.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:34.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:34.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:03:34.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:34.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:34.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:03:34.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:03:34.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:34.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:03:34.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:03:34.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:34.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:03:34.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:03:34.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:34.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:03:34.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:03:34.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:03:34.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:03:34.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:34.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:03:34.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:03:34.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:03:34.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:03:34.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:34.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:03:34.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:03:34.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:34.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:03:34.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:03:34.396 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.396 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.396 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.397 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:34.397 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:03:34.398 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.963008683000002, 2023-07-02T14:03:34.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:03:34.398 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:34.398 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:34.399 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:34.399 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:34.399 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:34.399 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:03:34.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.400 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:34.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:34.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:03:34.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:34.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:34.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:03:34.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:34.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:34.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:03:34.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:03:34.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:34.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:03:34.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:03:34.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:34.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:03:34.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:03:34.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:34.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:03:34.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:03:34.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:03:34.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:03:34.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:34.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:03:34.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:03:34.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:03:34.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:03:34.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:34.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:03:34.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:03:34.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.411 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.412 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:34.412 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:03:34.412 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.956158799000001, 2023-07-02T14:03:34.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:03:34.412 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:34.412 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:34.412 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:34.413 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:34.413 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:34.413 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:03:34.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.413 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:34.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:34.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:03:34.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:34.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:34.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:03:34.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:03:34.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:34.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:03:34.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:03:34.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:34.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:03:34.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:03:34.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:34.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:03:34.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:03:34.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:34.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:03:34.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:03:34.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:03:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:03:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:03:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:03:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:03:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:03:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:03:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:03:34.420 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.420 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.420 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.420 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:34.420 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:03:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.956095873999999, 2023-07-02T14:03:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:03:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:03:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:03:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:34.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:03:34.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:34.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:34.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:03:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:03:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:03:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:03:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:34.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:03:34.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:03:34.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:34.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:03:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:03:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:03:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:03:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:03:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:03:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:03:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:03:34.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:34.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:03:34.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:03:34.428 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.428 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.429 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.429 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:34.429 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:03:34.429 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.955795453, 2023-07-02T14:03:34.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:03:34.429 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:34.430 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:34.430 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:34.430 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:34.430 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:34.430 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:03:34.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.430 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:34.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:03:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:03:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:34.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:34.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:03:34.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:03:34.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:03:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:03:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:03:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:03:34.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:34.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:03:34.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:03:34.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:03:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:03:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:03:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:03:34.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:03:34.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:03:34.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:03:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:03:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:03:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:03:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:03:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:03:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:03:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:34.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:03:34.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:03:34.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:34.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:03:34.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:03:34.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:34.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:03:34.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:03:34.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:34.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:03:34.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:03:34.441 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.441 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.441 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.441 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:03:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.955270952999999, 2023-07-02T14:03:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:03:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:03:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:34.443 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:03:34.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.443 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:34.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:34.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:34.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:34.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:34.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:03:34.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:34.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:03:34.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:03:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:03:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:03:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:03:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:03:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:03:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:03:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:34.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:03:34.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:03:34.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:03:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:03:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:03:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:03:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:34.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:03:34.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:03:34.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.450 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.451 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2871 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:34.451 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.382460+0000", 2023-07-02T14:03:34.451 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.903466125000001, 2023-07-02T14:03:34.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.062523707999999997, 2023-07-02T14:03:34.451 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:34.451 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:34.452 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:34.452 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:34.452 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:34.452 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2871 2023-07-02T14:03:34.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.452 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:34.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:34.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382460+0000", 2023-07-02T14:03:34.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:34.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:34.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382460+0000", 2023-07-02T14:03:34.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:34.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:34.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382460+0000", 2023-07-02T14:03:34.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3189999999999999e-06 2023-07-02T14:03:34.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:34.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382463+0000", 2023-07-02T14:03:34.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7400000000000004e-06 2023-07-02T14:03:34.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:34.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382471+0000", 2023-07-02T14:03:34.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1640000000000003e-06 2023-07-02T14:03:34.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:34.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382475+0000", 2023-07-02T14:03:34.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067762170000000002 2023-07-02T14:03:34.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.389251+0000", 2023-07-02T14:03:34.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3442000000000005e-05 2023-07-02T14:03:34.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.389335+0000", 2023-07-02T14:03:34.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028393860000000002 2023-07-02T14:03:34.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.392174+0000", 2023-07-02T14:03:34.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1433000000000002e-05 2023-07-02T14:03:34.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.392206+0000", 2023-07-02T14:03:34.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051865440000000004 2023-07-02T14:03:34.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.397392+0000", 2023-07-02T14:03:34.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.321e-05 2023-07-02T14:03:34.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.397455+0000", 2023-07-02T14:03:34.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012230776 2023-07-02T14:03:34.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.409686+0000", 2023-07-02T14:03:34.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8430999999999996e-05 2023-07-02T14:03:34.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.409754+0000", 2023-07-02T14:03:34.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052440610000000004 2023-07-02T14:03:34.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.414999+0000", 2023-07-02T14:03:34.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0063999999999999e-05 2023-07-02T14:03:34.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.415069+0000", 2023-07-02T14:03:34.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035077749999999999 2023-07-02T14:03:34.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.418576+0000", 2023-07-02T14:03:34.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0323999999999999e-05 2023-07-02T14:03:34.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.418657+0000", 2023-07-02T14:03:34.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018539512000000001 2023-07-02T14:03:34.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.437196+0000", 2023-07-02T14:03:34.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0704000000000004e-05 2023-07-02T14:03:34.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:34.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.437287+0000", 2023-07-02T14:03:34.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042943000000000001 2023-07-02T14:03:34.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:34.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.437716+0000", 2023-07-02T14:03:34.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016934700000000001 2023-07-02T14:03:34.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:34.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.439410+0000", 2023-07-02T14:03:34.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0053641770000000004 2023-07-02T14:03:34.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:34.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.444774+0000", 2023-07-02T14:03:34.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0989000000000003e-05 2023-07-02T14:03:34.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:34.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.444825+0000", 2023-07-02T14:03:34.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000158982 2023-07-02T14:03:34.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:34.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.444984+0000", 2023-07-02T14:03:34.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.062523707999999997 2023-07-02T14:03:34.468 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.469 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.469 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.469 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2872 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:34.469 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.382481+0000", 2023-07-02T14:03:34.469 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.903445523999999, 2023-07-02T14:03:34.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.065455442000000003, 2023-07-02T14:03:34.470 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:34.470 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:34.470 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:34.470 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:34.470 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:34.470 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2872 2023-07-02T14:03:34.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.471 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:34.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:34.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382481+0000", 2023-07-02T14:03:34.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:34.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:34.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382481+0000", 2023-07-02T14:03:34.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:34.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:34.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382480+0000", 2023-07-02T14:03:34.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2509999999999999e-06 2023-07-02T14:03:34.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:34.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382484+0000", 2023-07-02T14:03:34.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:03:34.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:34.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382490+0000", 2023-07-02T14:03:34.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3000000000000003e-06 2023-07-02T14:03:34.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:34.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382494+0000", 2023-07-02T14:03:34.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0069223369999999998 2023-07-02T14:03:34.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.389416+0000", 2023-07-02T14:03:34.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5201000000000001e-05 2023-07-02T14:03:34.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.389481+0000", 2023-07-02T14:03:34.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027921510000000001 2023-07-02T14:03:34.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.392274+0000", 2023-07-02T14:03:34.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6381000000000002e-05 2023-07-02T14:03:34.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.392320+0000", 2023-07-02T14:03:34.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051686450000000004 2023-07-02T14:03:34.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.397489+0000", 2023-07-02T14:03:34.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0806e-05 2023-07-02T14:03:34.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.397549+0000", 2023-07-02T14:03:34.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012259144 2023-07-02T14:03:34.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.409809+0000", 2023-07-02T14:03:34.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7465000000000005e-05 2023-07-02T14:03:34.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.409876+0000", 2023-07-02T14:03:34.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052799830000000002 2023-07-02T14:03:34.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.415156+0000", 2023-07-02T14:03:34.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2080000000000001e-05 2023-07-02T14:03:34.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.415228+0000", 2023-07-02T14:03:34.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034778159999999999 2023-07-02T14:03:34.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.418706+0000", 2023-07-02T14:03:34.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7431000000000001e-05 2023-07-02T14:03:34.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.418743+0000", 2023-07-02T14:03:34.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020264971999999999 2023-07-02T14:03:34.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.439008+0000", 2023-07-02T14:03:34.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2211000000000003e-05 2023-07-02T14:03:34.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.439090+0000", 2023-07-02T14:03:34.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060620709999999996 2023-07-02T14:03:34.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.445153+0000", 2023-07-02T14:03:34.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5259000000000006e-05 2023-07-02T14:03:34.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:34.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.445228+0000", 2023-07-02T14:03:34.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030277499999999997 2023-07-02T14:03:34.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:34.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.445531+0000", 2023-07-02T14:03:34.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013477540000000001 2023-07-02T14:03:34.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:34.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.446878+0000", 2023-07-02T14:03:34.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00088152200000000003 2023-07-02T14:03:34.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:34.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.447760+0000", 2023-07-02T14:03:34.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7722999999999998e-05 2023-07-02T14:03:34.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:34.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.447798+0000", 2023-07-02T14:03:34.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013871599999999999 2023-07-02T14:03:34.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:34.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.447936+0000", 2023-07-02T14:03:34.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.065455442000000003 2023-07-02T14:03:34.488 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.489 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.489 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.489 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3000 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:34.489 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.518762+0000", 2023-07-02T14:03:34.489 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.767164617999999, 2023-07-02T14:03:34.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.052957575, 2023-07-02T14:03:34.490 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:34.490 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:34.490 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:34.490 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:34.490 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:34.490 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3000 2023-07-02T14:03:34.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.490 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:34.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:34.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518762+0000", 2023-07-02T14:03:34.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:34.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:34.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518762+0000", 2023-07-02T14:03:34.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:34.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:34.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518761+0000", 2023-07-02T14:03:34.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3069999999999996e-06 2023-07-02T14:03:34.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:34.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518765+0000", 2023-07-02T14:03:34.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6170000000000001e-06 2023-07-02T14:03:34.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:34.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518773+0000", 2023-07-02T14:03:34.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.172e-06 2023-07-02T14:03:34.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:34.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518777+0000", 2023-07-02T14:03:34.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055680249999999999 2023-07-02T14:03:34.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.524345+0000", 2023-07-02T14:03:34.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0938000000000003e-05 2023-07-02T14:03:34.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.524426+0000", 2023-07-02T14:03:34.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091411249999999999 2023-07-02T14:03:34.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.533567+0000", 2023-07-02T14:03:34.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5704000000000002e-05 2023-07-02T14:03:34.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.533603+0000", 2023-07-02T14:03:34.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046842289999999998 2023-07-02T14:03:34.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.538287+0000", 2023-07-02T14:03:34.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5780999999999999e-05 2023-07-02T14:03:34.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.538353+0000", 2023-07-02T14:03:34.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034785279999999998 2023-07-02T14:03:34.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.541832+0000", 2023-07-02T14:03:34.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0024999999999997e-05 2023-07-02T14:03:34.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.541872+0000", 2023-07-02T14:03:34.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024189239999999998 2023-07-02T14:03:34.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.544291+0000", 2023-07-02T14:03:34.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9709e-05 2023-07-02T14:03:34.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.544320+0000", 2023-07-02T14:03:34.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037311990000000001 2023-07-02T14:03:34.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.548051+0000", 2023-07-02T14:03:34.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3347000000000002e-05 2023-07-02T14:03:34.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.548115+0000", 2023-07-02T14:03:34.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078550089999999996 2023-07-02T14:03:34.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.555970+0000", 2023-07-02T14:03:34.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8730000000000003e-05 2023-07-02T14:03:34.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.556019+0000", 2023-07-02T14:03:34.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052420460000000002 2023-07-02T14:03:34.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.561261+0000", 2023-07-02T14:03:34.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4456999999999995e-05 2023-07-02T14:03:34.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:34.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.561345+0000", 2023-07-02T14:03:34.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00047298300000000001 2023-07-02T14:03:34.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:34.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.561818+0000", 2023-07-02T14:03:34.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0098041989999999996 2023-07-02T14:03:34.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:34.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.571622+0000", 2023-07-02T14:03:34.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6109000000000003e-05 2023-07-02T14:03:34.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:34.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.571658+0000", 2023-07-02T14:03:34.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9556e-05 2023-07-02T14:03:34.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.571688+0000", 2023-07-02T14:03:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1454000000000001e-05 2023-07-02T14:03:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.571719+0000", 2023-07-02T14:03:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.052957575 2023-07-02T14:03:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3056 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2023-07-02T14:03:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:12.080431+0000", 2023-07-02T14:03:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.205495774999999, 2023-07-02T14:03:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051668310000000002, 2023-07-02T14:03:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3056 2023-07-02T14:03:34.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.511 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:34.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:34.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080431+0000", 2023-07-02T14:03:34.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:34.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:34.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080431+0000", 2023-07-02T14:03:34.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:34.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:34.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080430+0000", 2023-07-02T14:03:34.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:03:34.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:34.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080434+0000", 2023-07-02T14:03:34.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0520000000000004e-06 2023-07-02T14:03:34.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:34.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080441+0000", 2023-07-02T14:03:34.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3860000000000004e-06 2023-07-02T14:03:34.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:34.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080448+0000", 2023-07-02T14:03:34.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015622699999999999 2023-07-02T14:03:34.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080604+0000", 2023-07-02T14:03:34.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9212000000000003e-05 2023-07-02T14:03:34.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080643+0000", 2023-07-02T14:03:34.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064524869999999998 2023-07-02T14:03:34.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.087096+0000", 2023-07-02T14:03:34.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0238000000000003e-05 2023-07-02T14:03:34.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.087136+0000", 2023-07-02T14:03:34.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028586150000000001 2023-07-02T14:03:34.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.089994+0000", 2023-07-02T14:03:34.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1024999999999999e-05 2023-07-02T14:03:34.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090055+0000", 2023-07-02T14:03:34.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050689359999999996 2023-07-02T14:03:34.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095124+0000", 2023-07-02T14:03:34.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9534999999999998e-05 2023-07-02T14:03:34.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095184+0000", 2023-07-02T14:03:34.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043677159999999998 2023-07-02T14:03:34.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.099552+0000", 2023-07-02T14:03:34.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4551000000000001e-05 2023-07-02T14:03:34.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.099596+0000", 2023-07-02T14:03:34.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038310229999999998 2023-07-02T14:03:34.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103427+0000", 2023-07-02T14:03:34.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0380999999999997e-05 2023-07-02T14:03:34.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103488+0000", 2023-07-02T14:03:34.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056513550000000003 2023-07-02T14:03:34.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109139+0000", 2023-07-02T14:03:34.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1032999999999996e-05 2023-07-02T14:03:34.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109210+0000", 2023-07-02T14:03:34.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025622259999999999 2023-07-02T14:03:34.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.111772+0000", 2023-07-02T14:03:34.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5722000000000005e-05 2023-07-02T14:03:34.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.111838+0000", 2023-07-02T14:03:34.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047256870000000001 2023-07-02T14:03:34.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116564+0000", 2023-07-02T14:03:34.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8287000000000005e-05 2023-07-02T14:03:34.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116632+0000", 2023-07-02T14:03:34.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059677539999999996 2023-07-02T14:03:34.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122600+0000", 2023-07-02T14:03:34.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7799999999999995e-05 2023-07-02T14:03:34.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122667+0000", 2023-07-02T14:03:34.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033116230000000001 2023-07-02T14:03:34.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.125979+0000", 2023-07-02T14:03:34.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1271000000000002e-05 2023-07-02T14:03:34.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126060+0000", 2023-07-02T14:03:34.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019737790000000002 2023-07-02T14:03:34.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.128034+0000", 2023-07-02T14:03:34.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00054109199999999996 2023-07-02T14:03:34.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:34.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.128575+0000", 2023-07-02T14:03:34.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050589700000000005 2023-07-02T14:03:34.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:34.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.129081+0000", 2023-07-02T14:03:34.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018622420000000001 2023-07-02T14:03:34.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:34.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.130943+0000", 2023-07-02T14:03:34.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010268079999999999 2023-07-02T14:03:34.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:34.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.131970+0000", 2023-07-02T14:03:34.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3554000000000003e-05 2023-07-02T14:03:34.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:34.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.132024+0000", 2023-07-02T14:03:34.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5173999999999995e-05 2023-07-02T14:03:34.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:34.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.132099+0000", 2023-07-02T14:03:34.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051668310000000002 2023-07-02T14:03:34.534 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.534 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.534 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.534 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3057 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2023-07-02T14:03:34.534 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:12.080462+0000", 2023-07-02T14:03:34.534 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.205464222, 2023-07-02T14:03:34.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.055924880000000003, 2023-07-02T14:03:34.535 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:34.535 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:34.535 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:34.535 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:34.535 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:34.535 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3057 2023-07-02T14:03:34.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.536 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:34.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:34.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080462+0000", 2023-07-02T14:03:34.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:34.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:34.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080462+0000", 2023-07-02T14:03:34.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:34.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:34.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080462+0000", 2023-07-02T14:03:34.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6289999999999999e-06 2023-07-02T14:03:34.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:34.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080465+0000", 2023-07-02T14:03:34.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3088000000000001e-05 2023-07-02T14:03:34.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:34.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080478+0000", 2023-07-02T14:03:34.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.101e-06 2023-07-02T14:03:34.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:34.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080483+0000", 2023-07-02T14:03:34.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018486899999999999 2023-07-02T14:03:34.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080668+0000", 2023-07-02T14:03:34.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7156999999999997e-05 2023-07-02T14:03:34.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080706+0000", 2023-07-02T14:03:34.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093726839999999992 2023-07-02T14:03:34.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090078+0000", 2023-07-02T14:03:34.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3180999999999998e-05 2023-07-02T14:03:34.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090131+0000", 2023-07-02T14:03:34.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050952510000000003 2023-07-02T14:03:34.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095227+0000", 2023-07-02T14:03:34.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8848999999999998e-05 2023-07-02T14:03:34.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095285+0000", 2023-07-02T14:03:34.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043427029999999998 2023-07-02T14:03:34.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.099628+0000", 2023-07-02T14:03:34.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1265000000000004e-05 2023-07-02T14:03:34.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.099689+0000", 2023-07-02T14:03:34.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038415099999999998 2023-07-02T14:03:34.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103531+0000", 2023-07-02T14:03:34.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0980000000000001e-05 2023-07-02T14:03:34.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103602+0000", 2023-07-02T14:03:34.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056603110000000003 2023-07-02T14:03:34.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109262+0000", 2023-07-02T14:03:34.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3421000000000001e-05 2023-07-02T14:03:34.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109326+0000", 2023-07-02T14:03:34.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025528510000000001 2023-07-02T14:03:34.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.111879+0000", 2023-07-02T14:03:34.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3395999999999999e-05 2023-07-02T14:03:34.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.111962+0000", 2023-07-02T14:03:34.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047126889999999999 2023-07-02T14:03:34.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116675+0000", 2023-07-02T14:03:34.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7849000000000001e-05 2023-07-02T14:03:34.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116732+0000", 2023-07-02T14:03:34.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059724649999999997 2023-07-02T14:03:34.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122705+0000", 2023-07-02T14:03:34.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3373999999999993e-05 2023-07-02T14:03:34.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122768+0000", 2023-07-02T14:03:34.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033541339999999999 2023-07-02T14:03:34.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126122+0000", 2023-07-02T14:03:34.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2749000000000003e-05 2023-07-02T14:03:34.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126215+0000", 2023-07-02T14:03:34.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036440750000000001 2023-07-02T14:03:34.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.129859+0000", 2023-07-02T14:03:34.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000103273 2023-07-02T14:03:34.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.129963+0000", 2023-07-02T14:03:34.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021629449999999999 2023-07-02T14:03:34.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.132125+0000", 2023-07-02T14:03:34.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8305e-05 2023-07-02T14:03:34.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:34.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.132164+0000", 2023-07-02T14:03:34.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00046304499999999998 2023-07-02T14:03:34.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:34.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.132627+0000", 2023-07-02T14:03:34.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022532979999999999 2023-07-02T14:03:34.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:34.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.134880+0000", 2023-07-02T14:03:34.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013186859999999999 2023-07-02T14:03:34.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:34.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.136199+0000", 2023-07-02T14:03:34.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1561000000000002e-05 2023-07-02T14:03:34.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:34.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.136260+0000", 2023-07-02T14:03:34.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126685 2023-07-02T14:03:34.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:34.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.136387+0000", 2023-07-02T14:03:34.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.055924880000000003 2023-07-02T14:03:34.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.562 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.563 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.563 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3058 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2023-07-02T14:03:34.564 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:12.080489+0000", 2023-07-02T14:03:34.564 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.205437693, 2023-07-02T14:03:34.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.059911803, 2023-07-02T14:03:34.565 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:34.565 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:34.565 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:34.565 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:34.566 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:34.566 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3058 2023-07-02T14:03:34.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.566 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:34.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:34.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080489+0000", 2023-07-02T14:03:34.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:34.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:34.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080489+0000", 2023-07-02T14:03:34.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:34.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:34.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080488+0000", 2023-07-02T14:03:34.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6950000000000002e-06 2023-07-02T14:03:34.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:34.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080492+0000", 2023-07-02T14:03:34.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7950000000000004e-06 2023-07-02T14:03:34.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:34.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080498+0000", 2023-07-02T14:03:34.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1189999999999999e-06 2023-07-02T14:03:34.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:34.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080502+0000", 2023-07-02T14:03:34.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000227817 2023-07-02T14:03:34.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080730+0000", 2023-07-02T14:03:34.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0629e-05 2023-07-02T14:03:34.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080770+0000", 2023-07-02T14:03:34.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093982159999999992 2023-07-02T14:03:34.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090168+0000", 2023-07-02T14:03:34.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5840000000000007e-05 2023-07-02T14:03:34.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090234+0000", 2023-07-02T14:03:34.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050953109999999999 2023-07-02T14:03:34.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095330+0000", 2023-07-02T14:03:34.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104215 2023-07-02T14:03:34.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095434+0000", 2023-07-02T14:03:34.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043199060000000001 2023-07-02T14:03:34.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.099754+0000", 2023-07-02T14:03:34.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000336504 2023-07-02T14:03:34.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.100090+0000", 2023-07-02T14:03:34.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035759479999999998 2023-07-02T14:03:34.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103666+0000", 2023-07-02T14:03:34.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3940000000000001e-05 2023-07-02T14:03:34.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103730+0000", 2023-07-02T14:03:34.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005666333 2023-07-02T14:03:34.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109396+0000", 2023-07-02T14:03:34.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2536999999999998e-05 2023-07-02T14:03:34.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109489+0000", 2023-07-02T14:03:34.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025086790000000002 2023-07-02T14:03:34.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.111998+0000", 2023-07-02T14:03:34.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7811000000000005e-05 2023-07-02T14:03:34.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.112065+0000", 2023-07-02T14:03:34.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046994519999999998 2023-07-02T14:03:34.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116765+0000", 2023-07-02T14:03:34.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1928000000000002e-05 2023-07-02T14:03:34.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116817+0000", 2023-07-02T14:03:34.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005988345 2023-07-02T14:03:34.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122805+0000", 2023-07-02T14:03:34.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7315999999999998e-05 2023-07-02T14:03:34.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122862+0000", 2023-07-02T14:03:34.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034274489999999999 2023-07-02T14:03:34.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126290+0000", 2023-07-02T14:03:34.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8703999999999998e-05 2023-07-02T14:03:34.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126359+0000", 2023-07-02T14:03:34.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003699889 2023-07-02T14:03:34.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.130058+0000", 2023-07-02T14:03:34.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110308 2023-07-02T14:03:34.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.130169+0000", 2023-07-02T14:03:34.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003130295 2023-07-02T14:03:34.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.133299+0000", 2023-07-02T14:03:34.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6588999999999995e-05 2023-07-02T14:03:34.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.133376+0000", 2023-07-02T14:03:34.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030628389999999999 2023-07-02T14:03:34.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.136439+0000", 2023-07-02T14:03:34.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010477 2023-07-02T14:03:34.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:34.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.136543+0000", 2023-07-02T14:03:34.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00047791199999999999 2023-07-02T14:03:34.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:34.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.137021+0000", 2023-07-02T14:03:34.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00150596 2023-07-02T14:03:34.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:34.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.138527+0000", 2023-07-02T14:03:34.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017228510000000001 2023-07-02T14:03:34.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:34.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140250+0000", 2023-07-02T14:03:34.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8615000000000006e-05 2023-07-02T14:03:34.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:34.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140319+0000", 2023-07-02T14:03:34.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1852000000000002e-05 2023-07-02T14:03:34.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:34.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140400+0000", 2023-07-02T14:03:34.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.059911803 2023-07-02T14:03:34.601 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.601 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.601 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.601 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3059 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2023-07-02T14:03:34.601 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:12.080509+0000", 2023-07-02T14:03:34.602 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.205417156999999, 2023-07-02T14:03:34.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.062340277999999999, 2023-07-02T14:03:34.602 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:34.602 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:34.602 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:34.602 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:34.602 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:34.602 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3059 2023-07-02T14:03:34.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.603 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:34.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:34.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080509+0000", 2023-07-02T14:03:34.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:34.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:34.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080509+0000", 2023-07-02T14:03:34.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:34.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:34.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080509+0000", 2023-07-02T14:03:34.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5099999999999999e-06 2023-07-02T14:03:34.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:34.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080512+0000", 2023-07-02T14:03:34.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8880000000000002e-06 2023-07-02T14:03:34.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:34.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080518+0000", 2023-07-02T14:03:34.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1520000000000002e-06 2023-07-02T14:03:34.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:34.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080522+0000", 2023-07-02T14:03:34.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026573699999999999 2023-07-02T14:03:34.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080788+0000", 2023-07-02T14:03:34.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7478999999999998e-05 2023-07-02T14:03:34.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080826+0000", 2023-07-02T14:03:34.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0094689049999999997 2023-07-02T14:03:34.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090294+0000", 2023-07-02T14:03:34.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5257000000000001e-05 2023-07-02T14:03:34.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090370+0000", 2023-07-02T14:03:34.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005117229 2023-07-02T14:03:34.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095487+0000", 2023-07-02T14:03:34.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7698e-05 2023-07-02T14:03:34.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095575+0000", 2023-07-02T14:03:34.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045423210000000002 2023-07-02T14:03:34.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.100117+0000", 2023-07-02T14:03:34.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1317999999999999e-05 2023-07-02T14:03:34.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.100168+0000", 2023-07-02T14:03:34.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035847650000000002 2023-07-02T14:03:34.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103753+0000", 2023-07-02T14:03:34.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3754999999999998e-05 2023-07-02T14:03:34.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103787+0000", 2023-07-02T14:03:34.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0057476209999999996 2023-07-02T14:03:34.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109534+0000", 2023-07-02T14:03:34.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9867000000000001e-05 2023-07-02T14:03:34.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109594+0000", 2023-07-02T14:03:34.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024983840000000002 2023-07-02T14:03:34.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.112093+0000", 2023-07-02T14:03:34.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0919e-05 2023-07-02T14:03:34.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.112124+0000", 2023-07-02T14:03:34.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047877880000000003 2023-07-02T14:03:34.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116911+0000", 2023-07-02T14:03:34.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1793999999999996e-05 2023-07-02T14:03:34.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.117003+0000", 2023-07-02T14:03:34.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059009860000000004 2023-07-02T14:03:34.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122904+0000", 2023-07-02T14:03:34.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8408000000000001e-05 2023-07-02T14:03:34.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122953+0000", 2023-07-02T14:03:34.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034502040000000001 2023-07-02T14:03:34.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126403+0000", 2023-07-02T14:03:34.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4770000000000005e-05 2023-07-02T14:03:34.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126468+0000", 2023-07-02T14:03:34.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037365950000000001 2023-07-02T14:03:34.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.130204+0000", 2023-07-02T14:03:34.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7163000000000002e-05 2023-07-02T14:03:34.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.130251+0000", 2023-07-02T14:03:34.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031607509999999998 2023-07-02T14:03:34.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.133412+0000", 2023-07-02T14:03:34.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1032000000000005e-05 2023-07-02T14:03:34.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.133503+0000", 2023-07-02T14:03:34.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041497349999999999 2023-07-02T14:03:34.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.137653+0000", 2023-07-02T14:03:34.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7430999999999997e-05 2023-07-02T14:03:34.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:34.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.137740+0000", 2023-07-02T14:03:34.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026871349999999998 2023-07-02T14:03:34.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:34.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140427+0000", 2023-07-02T14:03:34.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7179999999999998e-05 2023-07-02T14:03:34.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:34.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140485+0000", 2023-07-02T14:03:34.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00039492900000000003 2023-07-02T14:03:34.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:34.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140879+0000", 2023-07-02T14:03:34.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018682359999999999 2023-07-02T14:03:34.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:34.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.142748+0000", 2023-07-02T14:03:34.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4937e-05 2023-07-02T14:03:34.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:34.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.142793+0000", 2023-07-02T14:03:34.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4743999999999998e-05 2023-07-02T14:03:34.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:34.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.142827+0000", 2023-07-02T14:03:34.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2118999999999999e-05 2023-07-02T14:03:34.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:34.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:34.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:34.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.142849+0000", 2023-07-02T14:03:34.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.062340277999999999 2023-07-02T14:03:34.628 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.628 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.628 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.628 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:34.628 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:34.629 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:34.629 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:03:34.630 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:34.629+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:34.631 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:03:34.730 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:34.729+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:34.803 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:34.803 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:34.803 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:34.804 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:34.815 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:03:34.858 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:34.858+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:34.914 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:03:34.932 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:34.931+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:34.940 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:34.940 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:34.940 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:34.940 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:34.940 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:34.951 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:03:35.032 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:35.031+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:35.054 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:35.072 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:35.090 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:35.108 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:35.117 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:35.117 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:35.127 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:03:35.128 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:35.133 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:35.133+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:35.145 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:35.147 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:03:35.147 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:03:35.147 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:03:35.147 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:03:35.147 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:03:35.147 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:03:35.234 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:35.234+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:35.265 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:35.265 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:35.276 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:03:35.334 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:35.333+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:35.392 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:03:35.404 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:35.404 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:35.417 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:03:35.435 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:03:35.441 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 1 2023-07-02T14:03:35.449 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:35.449+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:35.536 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:35.536+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:35.595 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:35.595 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:35.606 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:03:35.637 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:35.637+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:35.737 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:35.737+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:35.738 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:35.738 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:35.750 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:03:35.839 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:35.838+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:35.922 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:35.923 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:35.935 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:03:35.976 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:35.975+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:36.040 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:36.040+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:36.112 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:36.112 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:36.124 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:03:36.168 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:36.167+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:36.241 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:36.241+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:36.251 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:36.251 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:36.262 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:03:36.342 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:36.342+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:36.428 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:36.429 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:36.438 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:03:36.442 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap9 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap10 b3fbb028-e267-4d25-a750-93ba18a2316a@snap10.diff 2023-07-02T14:03:36.443 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:36.442+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:36.543 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:36.542+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:36.563 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:36.563 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:36.573 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:03:36.643 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:36.643+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:36.711 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:36.711 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:36.721 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:03:36.744 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:36.743+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:36.844 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:36.844 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:36.845 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:36.844+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:36.854 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:03:36.945 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:36.944+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:36.977 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:36.977 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:36.987 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:03:37.046 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:37.046+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:37.147 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:37.147 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:37.147 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:37.147+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:37.158 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:03:37.166 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:03:37.252 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:37.252+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:37.348 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:37.348+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:37.383 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:37.409 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:03:37.417 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:37.417 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:37.418 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:37.434 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:03:37.449 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:37.448+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:37.549 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:37.549+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:37.651 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:37.650+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:37.657 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:37.684 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:37.684 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:03:37.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:03:37.685 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:37.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.685 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:289 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:37.685 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:37.685 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 58.367077129000002, 2023-07-02T14:03:37.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051413158, 2023-07-02T14:03:37.686 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:37.686 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:37.686 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:37.686 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:37.686 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:37.686 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 289 2023-07-02T14:03:37.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.686 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:37.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:37.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:37.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:37.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:37.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:37.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:37.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:37.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312168+0000", 2023-07-02T14:03:37.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1689e-05 2023-07-02T14:03:37.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:37.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312209+0000", 2023-07-02T14:03:37.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1070000000000005e-06 2023-07-02T14:03:37.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:37.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312217+0000", 2023-07-02T14:03:37.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.634e-06 2023-07-02T14:03:37.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:37.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312222+0000", 2023-07-02T14:03:37.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040512761000000001 2023-07-02T14:03:37.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352735+0000", 2023-07-02T14:03:37.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000119184 2023-07-02T14:03:37.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:37.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.352854+0000", 2023-07-02T14:03:37.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021481899999999999 2023-07-02T14:03:37.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:37.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353069+0000", 2023-07-02T14:03:37.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001601707 2023-07-02T14:03:37.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:37.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354670+0000", 2023-07-02T14:03:37.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088168570000000009 2023-07-02T14:03:37.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:37.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363487+0000", 2023-07-02T14:03:37.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9635999999999997e-05 2023-07-02T14:03:37.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:37.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363537+0000", 2023-07-02T14:03:37.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4292999999999998e-05 2023-07-02T14:03:37.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:37.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363581+0000", 2023-07-02T14:03:37.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051413158 2023-07-02T14:03:37.695 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.695 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:37.696 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.696 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:290 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:37.696 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:37.696 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 58.367017883999999, 2023-07-02T14:03:37.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051455062000000003, 2023-07-02T14:03:37.697 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:37.697 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:37.697 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:37.697 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:37.697 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:37.698 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 290 2023-07-02T14:03:37.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.698 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:37.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:37.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:37.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:37.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:37.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:37.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:37.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:37.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312227+0000", 2023-07-02T14:03:37.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6469000000000003e-05 2023-07-02T14:03:37.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:37.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312293+0000", 2023-07-02T14:03:37.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4630000000000007e-06 2023-07-02T14:03:37.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:37.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312303+0000", 2023-07-02T14:03:37.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8969999999999999e-06 2023-07-02T14:03:37.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:37.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312307+0000", 2023-07-02T14:03:37.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041261548000000002 2023-07-02T14:03:37.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353569+0000", 2023-07-02T14:03:37.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9594999999999999e-05 2023-07-02T14:03:37.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:37.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353659+0000", 2023-07-02T14:03:37.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018390600000000001 2023-07-02T14:03:37.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:37.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.353843+0000", 2023-07-02T14:03:37.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035173100000000001 2023-07-02T14:03:37.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:37.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357360+0000", 2023-07-02T14:03:37.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062295670000000001 2023-07-02T14:03:37.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:37.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363589+0000", 2023-07-02T14:03:37.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9854000000000001e-05 2023-07-02T14:03:37.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:37.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363639+0000", 2023-07-02T14:03:37.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3133999999999997e-05 2023-07-02T14:03:37.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:37.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363682+0000", 2023-07-02T14:03:37.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051455062000000003 2023-07-02T14:03:37.707 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.707 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:37.707 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.708 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:291 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:37.708 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:37.708 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 58.366931845000003, 2023-07-02T14:03:37.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051527860000000002, 2023-07-02T14:03:37.708 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:37.709 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:37.709 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:37.709 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:37.709 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:37.709 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 291 2023-07-02T14:03:37.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.709 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:37.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:37.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:37.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:37.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:37.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:37.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:37.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:37.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312313+0000", 2023-07-02T14:03:37.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2448999999999998e-05 2023-07-02T14:03:37.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:37.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312375+0000", 2023-07-02T14:03:37.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5259999999999993e-06 2023-07-02T14:03:37.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:37.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312384+0000", 2023-07-02T14:03:37.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0610000000000003e-06 2023-07-02T14:03:37.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:37.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312389+0000", 2023-07-02T14:03:37.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042326111 2023-07-02T14:03:37.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354715+0000", 2023-07-02T14:03:37.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3998999999999995e-05 2023-07-02T14:03:37.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:37.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354779+0000", 2023-07-02T14:03:37.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010269500000000001 2023-07-02T14:03:37.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:37.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.354882+0000", 2023-07-02T14:03:37.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002533736 2023-07-02T14:03:37.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:37.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357415+0000", 2023-07-02T14:03:37.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062765700000000004 2023-07-02T14:03:37.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:37.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363692+0000", 2023-07-02T14:03:37.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7095000000000001e-05 2023-07-02T14:03:37.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:37.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363749+0000", 2023-07-02T14:03:37.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2263999999999996e-05 2023-07-02T14:03:37.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:37.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363841+0000", 2023-07-02T14:03:37.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051527860000000002 2023-07-02T14:03:37.718 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.718 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:37.718 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.719 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:292 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:37.719 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:03:37.719 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 58.366849596999998, 2023-07-02T14:03:37.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051556056000000003, 2023-07-02T14:03:37.719 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:37.719 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:37.719 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:37.720 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:37.720 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:37.720 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 292 2023-07-02T14:03:37.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.720 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:37.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:37.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:03:37.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:37.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:37.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312396+0000", 2023-07-02T14:03:37.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:37.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:37.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312395+0000", 2023-07-02T14:03:37.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9624e-05 2023-07-02T14:03:37.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:37.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312425+0000", 2023-07-02T14:03:37.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:03:37.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:37.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312432+0000", 2023-07-02T14:03:37.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7550000000000002e-06 2023-07-02T14:03:37.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:37.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312437+0000", 2023-07-02T14:03:37.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.043823036000000003 2023-07-02T14:03:37.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356260+0000", 2023-07-02T14:03:37.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6021999999999997e-05 2023-07-02T14:03:37.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:37.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356346+0000", 2023-07-02T14:03:37.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020076600000000001 2023-07-02T14:03:37.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:37.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.356547+0000", 2023-07-02T14:03:37.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020766410000000002 2023-07-02T14:03:37.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:37.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358623+0000", 2023-07-02T14:03:37.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005231192 2023-07-02T14:03:37.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:37.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363854+0000", 2023-07-02T14:03:37.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3423e-05 2023-07-02T14:03:37.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:37.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363908+0000", 2023-07-02T14:03:37.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3825999999999997e-05 2023-07-02T14:03:37.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:37.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363952+0000", 2023-07-02T14:03:37.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051556056000000003 2023-07-02T14:03:37.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.734 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:37.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.734 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:293 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:37.734 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:03:37.735 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 58.366803427000001, 2023-07-02T14:03:37.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051614769999999997, 2023-07-02T14:03:37.735 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:37.735 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:37.735 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:37.735 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:37.736 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:37.736 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 293 2023-07-02T14:03:37.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.736 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:37.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:37.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:03:37.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:37.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:37.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312442+0000", 2023-07-02T14:03:37.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:37.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:37.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312441+0000", 2023-07-02T14:03:37.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9609999999999999e-05 2023-07-02T14:03:37.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:37.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312471+0000", 2023-07-02T14:03:37.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7069999999999992e-06 2023-07-02T14:03:37.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:37.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312478+0000", 2023-07-02T14:03:37.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1190000000000002e-06 2023-07-02T14:03:37.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:37.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312485+0000", 2023-07-02T14:03:37.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.044956387 2023-07-02T14:03:37.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357441+0000", 2023-07-02T14:03:37.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4134999999999994e-05 2023-07-02T14:03:37.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:37.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357515+0000", 2023-07-02T14:03:37.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000147408 2023-07-02T14:03:37.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:37.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357663+0000", 2023-07-02T14:03:37.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056472639999999999 2023-07-02T14:03:37.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:37.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363310+0000", 2023-07-02T14:03:37.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00065402600000000004 2023-07-02T14:03:37.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:37.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363964+0000", 2023-07-02T14:03:37.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0497000000000001e-05 2023-07-02T14:03:37.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:37.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364014+0000", 2023-07-02T14:03:37.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2243000000000002e-05 2023-07-02T14:03:37.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:37.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364057+0000", 2023-07-02T14:03:37.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051614769999999997 2023-07-02T14:03:37.754 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.755 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:37.755 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.756 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:294 2.2 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1048576~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:37.756 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:03:37.756 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 58.366755427999998, 2023-07-02T14:03:37.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051679527000000003, 2023-07-02T14:03:37.757 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:37.757 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:37.758 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:37.758 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:37.758 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:37.758 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 294 2023-07-02T14:03:37.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.759 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:37.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:37.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:03:37.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:37.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:37.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312490+0000", 2023-07-02T14:03:37.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:37.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:37.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312489+0000", 2023-07-02T14:03:37.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8204999999999998e-05 2023-07-02T14:03:37.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:37.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312517+0000", 2023-07-02T14:03:37.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7759999999999997e-06 2023-07-02T14:03:37.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:37.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312525+0000", 2023-07-02T14:03:37.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8730000000000001e-06 2023-07-02T14:03:37.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:37.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312532+0000", 2023-07-02T14:03:37.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.045442206999999998 2023-07-02T14:03:37.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.357974+0000", 2023-07-02T14:03:37.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4422000000000002e-05 2023-07-02T14:03:37.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:37.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358029+0000", 2023-07-02T14:03:37.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149519 2023-07-02T14:03:37.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:37.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.358178+0000", 2023-07-02T14:03:37.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051779390000000003 2023-07-02T14:03:37.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:37.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.363356+0000", 2023-07-02T14:03:37.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00070707699999999997 2023-07-02T14:03:37.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:37.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364063+0000", 2023-07-02T14:03:37.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6835999999999999e-05 2023-07-02T14:03:37.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:37.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364090+0000", 2023-07-02T14:03:37.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9263000000000003e-05 2023-07-02T14:03:37.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:37.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.364169+0000", 2023-07-02T14:03:37.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051679527000000003 2023-07-02T14:03:37.776 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.777 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:37.777 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.778 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:03:37.778 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:37.778 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 28.227145226000001, 2023-07-02T14:03:37.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:03:37.779 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:37.779 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:37.779 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:37.779 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:37.780 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:37.780 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:03:37.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.780 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:37.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:37.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:37.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:37.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:37.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:37.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:37.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:37.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:03:37.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:03:37.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:37.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:03:37.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:03:37.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:37.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:03:37.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:03:37.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:37.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:03:37.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:03:37.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:03:37.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:03:37.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:37.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:03:37.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:03:37.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:03:37.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:03:37.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:37.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:03:37.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:03:37.788 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.788 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:37.788 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.788 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:03:37.788 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:03:37.789 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.356452218000001, 2023-07-02T14:03:37.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:03:37.789 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:37.789 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:03:37.789 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:37.789 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:03:37.789 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:03:37.790 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:03:37.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.790 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:37.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:37.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:03:37.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:37.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:37.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:03:37.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:37.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:37.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:03:37.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:03:37.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:37.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:03:37.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:03:37.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:37.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:03:37.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:03:37.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:37.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:03:37.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:03:37.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:03:37.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:03:37.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:37.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:03:37.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:03:37.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:03:37.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:03:37.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:37.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:03:37.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:03:37.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:37.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:03:37.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:03:37.797 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.797 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:37.797 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.797 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:37.798 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:03:37.798 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.356327496999999, 2023-07-02T14:03:37.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:03:37.798 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:37.798 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:37.798 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:37.798 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:37.798 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:37.799 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:03:37.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.799 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:37.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:37.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:03:37.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:37.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:37.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:03:37.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:37.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:37.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:03:37.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:03:37.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:37.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:03:37.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:03:37.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:37.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:03:37.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:03:37.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:37.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:03:37.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:03:37.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:03:37.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:03:37.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:37.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:03:37.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:03:37.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:03:37.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:03:37.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:37.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:03:37.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:03:37.805 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.805 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:37.805 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.806 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:37.806 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:03:37.806 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.349477613000001, 2023-07-02T14:03:37.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:03:37.806 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:37.806 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:37.807 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:37.807 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:37.807 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:37.807 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:03:37.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.807 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:37.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:37.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:03:37.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:37.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:37.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:03:37.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:03:37.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:37.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:03:37.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:03:37.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:37.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:03:37.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:03:37.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:37.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:03:37.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:03:37.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:37.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:03:37.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:03:37.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:03:37.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:03:37.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:37.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:03:37.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:03:37.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:03:37.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:03:37.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:37.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:03:37.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:03:37.814 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.814 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:37.814 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.814 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:37.814 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:03:37.815 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.349414688, 2023-07-02T14:03:37.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:03:37.815 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:37.815 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:37.815 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:37.815 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:37.815 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:37.815 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:03:37.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.816 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:37.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:37.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:03:37.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:37.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:37.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:03:37.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:37.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:37.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:03:37.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:03:37.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:37.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:03:37.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:03:37.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:37.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:03:37.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:03:37.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:37.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:03:37.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:03:37.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:03:37.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:03:37.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:37.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:03:37.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:03:37.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:03:37.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:03:37.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:37.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:03:37.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:03:37.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.822 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:37.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.823 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:37.823 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:03:37.823 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.349114267000001, 2023-07-02T14:03:37.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:03:37.823 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:37.823 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:37.824 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:37.824 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:37.824 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:37.824 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:03:37.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.824 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:37.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:37.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:03:37.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:37.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:37.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:03:37.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:37.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:37.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:03:37.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:03:37.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:37.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:03:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:03:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:03:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:03:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:37.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:03:37.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:03:37.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:03:37.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:03:37.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:37.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:03:37.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:03:37.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:03:37.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:03:37.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:37.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:03:37.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:03:37.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:03:37.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:03:37.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:03:37.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:37.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:03:37.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:03:37.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:37.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:03:37.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:03:37.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:37.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:03:37.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:03:37.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:37.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:03:37.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:03:37.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:37.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:03:37.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:03:37.834 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.834 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:37.835 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.835 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:03:37.835 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:37.835 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.348589767, 2023-07-02T14:03:37.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:03:37.835 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:37.836 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:03:37.836 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:37.836 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:37.836 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:37.836 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:03:37.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.836 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:37.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:37.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:37.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:37.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:37.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:37.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:37.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:37.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:37.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:03:37.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:37.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:03:37.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:03:37.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:37.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:03:37.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:03:37.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:37.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:03:37.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:03:37.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:03:37.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:03:37.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:37.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:03:37.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:03:37.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:03:37.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:03:37.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:37.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:03:37.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:03:37.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:37.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:03:37.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:03:37.843 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.844 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:37.844 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.844 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2871 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:37.844 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.382460+0000", 2023-07-02T14:03:37.844 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.296784938999998, 2023-07-02T14:03:37.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.062523707999999997, 2023-07-02T14:03:37.845 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:37.845 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:37.845 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:37.845 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:37.845 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:37.845 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2871 2023-07-02T14:03:37.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.846 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:37.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:37.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382460+0000", 2023-07-02T14:03:37.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:37.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:37.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382460+0000", 2023-07-02T14:03:37.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:37.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:37.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382460+0000", 2023-07-02T14:03:37.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3189999999999999e-06 2023-07-02T14:03:37.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:37.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382463+0000", 2023-07-02T14:03:37.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7400000000000004e-06 2023-07-02T14:03:37.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:37.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382471+0000", 2023-07-02T14:03:37.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1640000000000003e-06 2023-07-02T14:03:37.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:37.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382475+0000", 2023-07-02T14:03:37.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067762170000000002 2023-07-02T14:03:37.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.389251+0000", 2023-07-02T14:03:37.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3442000000000005e-05 2023-07-02T14:03:37.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.389335+0000", 2023-07-02T14:03:37.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028393860000000002 2023-07-02T14:03:37.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.392174+0000", 2023-07-02T14:03:37.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1433000000000002e-05 2023-07-02T14:03:37.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.392206+0000", 2023-07-02T14:03:37.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051865440000000004 2023-07-02T14:03:37.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.397392+0000", 2023-07-02T14:03:37.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.321e-05 2023-07-02T14:03:37.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.397455+0000", 2023-07-02T14:03:37.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012230776 2023-07-02T14:03:37.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.409686+0000", 2023-07-02T14:03:37.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8430999999999996e-05 2023-07-02T14:03:37.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.409754+0000", 2023-07-02T14:03:37.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052440610000000004 2023-07-02T14:03:37.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.414999+0000", 2023-07-02T14:03:37.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0063999999999999e-05 2023-07-02T14:03:37.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.415069+0000", 2023-07-02T14:03:37.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035077749999999999 2023-07-02T14:03:37.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.418576+0000", 2023-07-02T14:03:37.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0323999999999999e-05 2023-07-02T14:03:37.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.418657+0000", 2023-07-02T14:03:37.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018539512000000001 2023-07-02T14:03:37.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.437196+0000", 2023-07-02T14:03:37.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0704000000000004e-05 2023-07-02T14:03:37.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:37.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.437287+0000", 2023-07-02T14:03:37.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00042943000000000001 2023-07-02T14:03:37.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:37.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.437716+0000", 2023-07-02T14:03:37.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016934700000000001 2023-07-02T14:03:37.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:37.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.439410+0000", 2023-07-02T14:03:37.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0053641770000000004 2023-07-02T14:03:37.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:37.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.444774+0000", 2023-07-02T14:03:37.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0989000000000003e-05 2023-07-02T14:03:37.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:37.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.444825+0000", 2023-07-02T14:03:37.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000158982 2023-07-02T14:03:37.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:37.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.444984+0000", 2023-07-02T14:03:37.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.062523707999999997 2023-07-02T14:03:37.862 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.862 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:37.862 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.862 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2872 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:37.862 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.382481+0000", 2023-07-02T14:03:37.863 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.296764337999999, 2023-07-02T14:03:37.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.065455442000000003, 2023-07-02T14:03:37.863 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:37.863 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:37.863 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:37.863 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:37.863 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:37.863 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2872 2023-07-02T14:03:37.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.864 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:37.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:37.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382481+0000", 2023-07-02T14:03:37.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:37.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:37.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382481+0000", 2023-07-02T14:03:37.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:37.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:37.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382480+0000", 2023-07-02T14:03:37.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2509999999999999e-06 2023-07-02T14:03:37.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:37.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382484+0000", 2023-07-02T14:03:37.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:03:37.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:37.867 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382490+0000", 2023-07-02T14:03:37.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3000000000000003e-06 2023-07-02T14:03:37.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:37.867 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.382494+0000", 2023-07-02T14:03:37.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0069223369999999998 2023-07-02T14:03:37.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.389416+0000", 2023-07-02T14:03:37.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5201000000000001e-05 2023-07-02T14:03:37.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.389481+0000", 2023-07-02T14:03:37.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027921510000000001 2023-07-02T14:03:37.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.392274+0000", 2023-07-02T14:03:37.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6381000000000002e-05 2023-07-02T14:03:37.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.392320+0000", 2023-07-02T14:03:37.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051686450000000004 2023-07-02T14:03:37.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.397489+0000", 2023-07-02T14:03:37.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0806e-05 2023-07-02T14:03:37.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.397549+0000", 2023-07-02T14:03:37.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012259144 2023-07-02T14:03:37.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.409809+0000", 2023-07-02T14:03:37.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7465000000000005e-05 2023-07-02T14:03:37.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.409876+0000", 2023-07-02T14:03:37.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052799830000000002 2023-07-02T14:03:37.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.415156+0000", 2023-07-02T14:03:37.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2080000000000001e-05 2023-07-02T14:03:37.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.415228+0000", 2023-07-02T14:03:37.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034778159999999999 2023-07-02T14:03:37.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.874 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.418706+0000", 2023-07-02T14:03:37.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7431000000000001e-05 2023-07-02T14:03:37.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.418743+0000", 2023-07-02T14:03:37.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020264971999999999 2023-07-02T14:03:37.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.439008+0000", 2023-07-02T14:03:37.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2211000000000003e-05 2023-07-02T14:03:37.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.439090+0000", 2023-07-02T14:03:37.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060620709999999996 2023-07-02T14:03:37.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.445153+0000", 2023-07-02T14:03:37.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5259000000000006e-05 2023-07-02T14:03:37.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:37.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.445228+0000", 2023-07-02T14:03:37.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030277499999999997 2023-07-02T14:03:37.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:37.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.445531+0000", 2023-07-02T14:03:37.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013477540000000001 2023-07-02T14:03:37.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:37.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.446878+0000", 2023-07-02T14:03:37.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00088152200000000003 2023-07-02T14:03:37.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:37.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.447760+0000", 2023-07-02T14:03:37.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7722999999999998e-05 2023-07-02T14:03:37.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:37.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.447798+0000", 2023-07-02T14:03:37.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013871599999999999 2023-07-02T14:03:37.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:37.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.447936+0000", 2023-07-02T14:03:37.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.065455442000000003 2023-07-02T14:03:37.881 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.881 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:37.881 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.882 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3000 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:37.882 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.518762+0000", 2023-07-02T14:03:37.882 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.160483431999999, 2023-07-02T14:03:37.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.052957575, 2023-07-02T14:03:37.882 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:37.882 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:37.882 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:37.882 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:37.883 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:37.883 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3000 2023-07-02T14:03:37.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.883 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:37.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:37.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518762+0000", 2023-07-02T14:03:37.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:37.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:37.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518762+0000", 2023-07-02T14:03:37.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:37.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:37.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518761+0000", 2023-07-02T14:03:37.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3069999999999996e-06 2023-07-02T14:03:37.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.885 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:37.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518765+0000", 2023-07-02T14:03:37.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6170000000000001e-06 2023-07-02T14:03:37.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:37.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518773+0000", 2023-07-02T14:03:37.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.172e-06 2023-07-02T14:03:37.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:37.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.518777+0000", 2023-07-02T14:03:37.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055680249999999999 2023-07-02T14:03:37.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.524345+0000", 2023-07-02T14:03:37.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0938000000000003e-05 2023-07-02T14:03:37.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.524426+0000", 2023-07-02T14:03:37.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091411249999999999 2023-07-02T14:03:37.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.533567+0000", 2023-07-02T14:03:37.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5704000000000002e-05 2023-07-02T14:03:37.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.533603+0000", 2023-07-02T14:03:37.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046842289999999998 2023-07-02T14:03:37.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.538287+0000", 2023-07-02T14:03:37.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5780999999999999e-05 2023-07-02T14:03:37.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.538353+0000", 2023-07-02T14:03:37.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034785279999999998 2023-07-02T14:03:37.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.541832+0000", 2023-07-02T14:03:37.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0024999999999997e-05 2023-07-02T14:03:37.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.541872+0000", 2023-07-02T14:03:37.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024189239999999998 2023-07-02T14:03:37.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.544291+0000", 2023-07-02T14:03:37.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9709e-05 2023-07-02T14:03:37.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.544320+0000", 2023-07-02T14:03:37.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037311990000000001 2023-07-02T14:03:37.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.548051+0000", 2023-07-02T14:03:37.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3347000000000002e-05 2023-07-02T14:03:37.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.548115+0000", 2023-07-02T14:03:37.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078550089999999996 2023-07-02T14:03:37.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.555970+0000", 2023-07-02T14:03:37.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8730000000000003e-05 2023-07-02T14:03:37.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.556019+0000", 2023-07-02T14:03:37.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052420460000000002 2023-07-02T14:03:37.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.561261+0000", 2023-07-02T14:03:37.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4456999999999995e-05 2023-07-02T14:03:37.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:37.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.561345+0000", 2023-07-02T14:03:37.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00047298300000000001 2023-07-02T14:03:37.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:37.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.561818+0000", 2023-07-02T14:03:37.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0098041989999999996 2023-07-02T14:03:37.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:37.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.571622+0000", 2023-07-02T14:03:37.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6109000000000003e-05 2023-07-02T14:03:37.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:37.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.571658+0000", 2023-07-02T14:03:37.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9556e-05 2023-07-02T14:03:37.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:37.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.571688+0000", 2023-07-02T14:03:37.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1454000000000001e-05 2023-07-02T14:03:37.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:37.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.571719+0000", 2023-07-02T14:03:37.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.052957575 2023-07-02T14:03:37.900 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.900 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:37.901 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.901 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3056 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2023-07-02T14:03:37.901 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:12.080431+0000", 2023-07-02T14:03:37.901 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 25.598814589, 2023-07-02T14:03:37.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051668310000000002, 2023-07-02T14:03:37.901 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:37.902 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:37.902 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:37.902 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:37.902 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:37.902 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3056 2023-07-02T14:03:37.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.902 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:37.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:37.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080431+0000", 2023-07-02T14:03:37.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:37.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:37.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080431+0000", 2023-07-02T14:03:37.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:37.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:37.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080430+0000", 2023-07-02T14:03:37.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:03:37.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:37.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080434+0000", 2023-07-02T14:03:37.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0520000000000004e-06 2023-07-02T14:03:37.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:37.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080441+0000", 2023-07-02T14:03:37.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3860000000000004e-06 2023-07-02T14:03:37.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:37.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080448+0000", 2023-07-02T14:03:37.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015622699999999999 2023-07-02T14:03:37.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080604+0000", 2023-07-02T14:03:37.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9212000000000003e-05 2023-07-02T14:03:37.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080643+0000", 2023-07-02T14:03:37.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064524869999999998 2023-07-02T14:03:37.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.087096+0000", 2023-07-02T14:03:37.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0238000000000003e-05 2023-07-02T14:03:37.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.087136+0000", 2023-07-02T14:03:37.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028586150000000001 2023-07-02T14:03:37.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.089994+0000", 2023-07-02T14:03:37.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1024999999999999e-05 2023-07-02T14:03:37.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090055+0000", 2023-07-02T14:03:37.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050689359999999996 2023-07-02T14:03:37.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095124+0000", 2023-07-02T14:03:37.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9534999999999998e-05 2023-07-02T14:03:37.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095184+0000", 2023-07-02T14:03:37.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043677159999999998 2023-07-02T14:03:37.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.099552+0000", 2023-07-02T14:03:37.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4551000000000001e-05 2023-07-02T14:03:37.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.099596+0000", 2023-07-02T14:03:37.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038310229999999998 2023-07-02T14:03:37.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103427+0000", 2023-07-02T14:03:37.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0380999999999997e-05 2023-07-02T14:03:37.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103488+0000", 2023-07-02T14:03:37.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056513550000000003 2023-07-02T14:03:37.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109139+0000", 2023-07-02T14:03:37.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1032999999999996e-05 2023-07-02T14:03:37.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109210+0000", 2023-07-02T14:03:37.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025622259999999999 2023-07-02T14:03:37.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.111772+0000", 2023-07-02T14:03:37.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5722000000000005e-05 2023-07-02T14:03:37.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.111838+0000", 2023-07-02T14:03:37.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047256870000000001 2023-07-02T14:03:37.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116564+0000", 2023-07-02T14:03:37.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8287000000000005e-05 2023-07-02T14:03:37.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116632+0000", 2023-07-02T14:03:37.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059677539999999996 2023-07-02T14:03:37.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122600+0000", 2023-07-02T14:03:37.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7799999999999995e-05 2023-07-02T14:03:37.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122667+0000", 2023-07-02T14:03:37.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033116230000000001 2023-07-02T14:03:37.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.125979+0000", 2023-07-02T14:03:37.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1271000000000002e-05 2023-07-02T14:03:37.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126060+0000", 2023-07-02T14:03:37.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019737790000000002 2023-07-02T14:03:37.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.128034+0000", 2023-07-02T14:03:37.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00054109199999999996 2023-07-02T14:03:37.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:37.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.128575+0000", 2023-07-02T14:03:37.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050589700000000005 2023-07-02T14:03:37.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:37.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.129081+0000", 2023-07-02T14:03:37.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018622420000000001 2023-07-02T14:03:37.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:37.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.130943+0000", 2023-07-02T14:03:37.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010268079999999999 2023-07-02T14:03:37.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:37.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.131970+0000", 2023-07-02T14:03:37.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3554000000000003e-05 2023-07-02T14:03:37.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:37.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.132024+0000", 2023-07-02T14:03:37.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5173999999999995e-05 2023-07-02T14:03:37.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:37.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.132099+0000", 2023-07-02T14:03:37.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051668310000000002 2023-07-02T14:03:37.925 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.925 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:37.925 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.925 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3057 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2023-07-02T14:03:37.926 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:12.080462+0000", 2023-07-02T14:03:37.926 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 25.598783036, 2023-07-02T14:03:37.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.055924880000000003, 2023-07-02T14:03:37.926 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:37.926 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:37.926 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:37.926 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:37.926 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:37.927 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3057 2023-07-02T14:03:37.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.927 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:37.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:37.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080462+0000", 2023-07-02T14:03:37.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:37.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:37.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080462+0000", 2023-07-02T14:03:37.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:37.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:37.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080462+0000", 2023-07-02T14:03:37.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6289999999999999e-06 2023-07-02T14:03:37.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:37.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080465+0000", 2023-07-02T14:03:37.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3088000000000001e-05 2023-07-02T14:03:37.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:37.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080478+0000", 2023-07-02T14:03:37.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.101e-06 2023-07-02T14:03:37.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:37.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080483+0000", 2023-07-02T14:03:37.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018486899999999999 2023-07-02T14:03:37.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080668+0000", 2023-07-02T14:03:37.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7156999999999997e-05 2023-07-02T14:03:37.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080706+0000", 2023-07-02T14:03:37.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093726839999999992 2023-07-02T14:03:37.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090078+0000", 2023-07-02T14:03:37.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3180999999999998e-05 2023-07-02T14:03:37.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090131+0000", 2023-07-02T14:03:37.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050952510000000003 2023-07-02T14:03:37.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095227+0000", 2023-07-02T14:03:37.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8848999999999998e-05 2023-07-02T14:03:37.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095285+0000", 2023-07-02T14:03:37.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043427029999999998 2023-07-02T14:03:37.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.099628+0000", 2023-07-02T14:03:37.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1265000000000004e-05 2023-07-02T14:03:37.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.099689+0000", 2023-07-02T14:03:37.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038415099999999998 2023-07-02T14:03:37.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103531+0000", 2023-07-02T14:03:37.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0980000000000001e-05 2023-07-02T14:03:37.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103602+0000", 2023-07-02T14:03:37.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056603110000000003 2023-07-02T14:03:37.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109262+0000", 2023-07-02T14:03:37.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3421000000000001e-05 2023-07-02T14:03:37.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109326+0000", 2023-07-02T14:03:37.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025528510000000001 2023-07-02T14:03:37.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.111879+0000", 2023-07-02T14:03:37.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3395999999999999e-05 2023-07-02T14:03:37.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.111962+0000", 2023-07-02T14:03:37.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047126889999999999 2023-07-02T14:03:37.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116675+0000", 2023-07-02T14:03:37.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7849000000000001e-05 2023-07-02T14:03:37.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116732+0000", 2023-07-02T14:03:37.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059724649999999997 2023-07-02T14:03:37.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122705+0000", 2023-07-02T14:03:37.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3373999999999993e-05 2023-07-02T14:03:37.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122768+0000", 2023-07-02T14:03:37.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033541339999999999 2023-07-02T14:03:37.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126122+0000", 2023-07-02T14:03:37.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2749000000000003e-05 2023-07-02T14:03:37.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126215+0000", 2023-07-02T14:03:37.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036440750000000001 2023-07-02T14:03:37.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.129859+0000", 2023-07-02T14:03:37.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000103273 2023-07-02T14:03:37.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.129963+0000", 2023-07-02T14:03:37.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021629449999999999 2023-07-02T14:03:37.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.132125+0000", 2023-07-02T14:03:37.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8305e-05 2023-07-02T14:03:37.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:37.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.132164+0000", 2023-07-02T14:03:37.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00046304499999999998 2023-07-02T14:03:37.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:37.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.132627+0000", 2023-07-02T14:03:37.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022532979999999999 2023-07-02T14:03:37.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:37.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.134880+0000", 2023-07-02T14:03:37.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013186859999999999 2023-07-02T14:03:37.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:37.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.136199+0000", 2023-07-02T14:03:37.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1561000000000002e-05 2023-07-02T14:03:37.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:37.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.136260+0000", 2023-07-02T14:03:37.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126685 2023-07-02T14:03:37.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:37.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.136387+0000", 2023-07-02T14:03:37.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.055924880000000003 2023-07-02T14:03:37.950 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.950 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:37.950 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.951 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3058 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2023-07-02T14:03:37.951 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:12.080489+0000", 2023-07-02T14:03:37.951 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 25.598756507000001, 2023-07-02T14:03:37.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.059911803, 2023-07-02T14:03:37.951 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:37.951 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:37.952 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:37.952 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:37.952 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:37.952 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3058 2023-07-02T14:03:37.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.952 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:37.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:37.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080489+0000", 2023-07-02T14:03:37.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:37.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:37.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080489+0000", 2023-07-02T14:03:37.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:37.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:37.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080488+0000", 2023-07-02T14:03:37.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6950000000000002e-06 2023-07-02T14:03:37.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:37.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080492+0000", 2023-07-02T14:03:37.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7950000000000004e-06 2023-07-02T14:03:37.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:37.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080498+0000", 2023-07-02T14:03:37.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1189999999999999e-06 2023-07-02T14:03:37.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:37.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080502+0000", 2023-07-02T14:03:37.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000227817 2023-07-02T14:03:37.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080730+0000", 2023-07-02T14:03:37.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0629e-05 2023-07-02T14:03:37.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080770+0000", 2023-07-02T14:03:37.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093982159999999992 2023-07-02T14:03:37.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090168+0000", 2023-07-02T14:03:37.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5840000000000007e-05 2023-07-02T14:03:37.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090234+0000", 2023-07-02T14:03:37.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050953109999999999 2023-07-02T14:03:37.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095330+0000", 2023-07-02T14:03:37.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104215 2023-07-02T14:03:37.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095434+0000", 2023-07-02T14:03:37.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043199060000000001 2023-07-02T14:03:37.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.099754+0000", 2023-07-02T14:03:37.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000336504 2023-07-02T14:03:37.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.100090+0000", 2023-07-02T14:03:37.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035759479999999998 2023-07-02T14:03:37.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103666+0000", 2023-07-02T14:03:37.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3940000000000001e-05 2023-07-02T14:03:37.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103730+0000", 2023-07-02T14:03:37.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005666333 2023-07-02T14:03:37.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109396+0000", 2023-07-02T14:03:37.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2536999999999998e-05 2023-07-02T14:03:37.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109489+0000", 2023-07-02T14:03:37.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025086790000000002 2023-07-02T14:03:37.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.111998+0000", 2023-07-02T14:03:37.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7811000000000005e-05 2023-07-02T14:03:37.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.112065+0000", 2023-07-02T14:03:37.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046994519999999998 2023-07-02T14:03:37.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116765+0000", 2023-07-02T14:03:37.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1928000000000002e-05 2023-07-02T14:03:37.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116817+0000", 2023-07-02T14:03:37.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005988345 2023-07-02T14:03:37.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122805+0000", 2023-07-02T14:03:37.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7315999999999998e-05 2023-07-02T14:03:37.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122862+0000", 2023-07-02T14:03:37.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034274489999999999 2023-07-02T14:03:37.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126290+0000", 2023-07-02T14:03:37.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8703999999999998e-05 2023-07-02T14:03:37.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126359+0000", 2023-07-02T14:03:37.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003699889 2023-07-02T14:03:37.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.130058+0000", 2023-07-02T14:03:37.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110308 2023-07-02T14:03:37.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.130169+0000", 2023-07-02T14:03:37.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003130295 2023-07-02T14:03:37.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.133299+0000", 2023-07-02T14:03:37.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6588999999999995e-05 2023-07-02T14:03:37.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.133376+0000", 2023-07-02T14:03:37.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030628389999999999 2023-07-02T14:03:37.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.136439+0000", 2023-07-02T14:03:37.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010477 2023-07-02T14:03:37.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:37.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.136543+0000", 2023-07-02T14:03:37.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00047791199999999999 2023-07-02T14:03:37.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:37.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.137021+0000", 2023-07-02T14:03:37.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00150596 2023-07-02T14:03:37.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:37.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.138527+0000", 2023-07-02T14:03:37.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017228510000000001 2023-07-02T14:03:37.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:37.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140250+0000", 2023-07-02T14:03:37.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8615000000000006e-05 2023-07-02T14:03:37.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:37.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140319+0000", 2023-07-02T14:03:37.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1852000000000002e-05 2023-07-02T14:03:37.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:37.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140400+0000", 2023-07-02T14:03:37.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.059911803 2023-07-02T14:03:37.976 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3059 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2023-07-02T14:03:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:12.080509+0000", 2023-07-02T14:03:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 25.598735971, 2023-07-02T14:03:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.062340277999999999, 2023-07-02T14:03:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3059 2023-07-02T14:03:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.979 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:37.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:37.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080509+0000", 2023-07-02T14:03:37.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:37.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:37.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080509+0000", 2023-07-02T14:03:37.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:37.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:37.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080509+0000", 2023-07-02T14:03:37.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5099999999999999e-06 2023-07-02T14:03:37.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:37.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080512+0000", 2023-07-02T14:03:37.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8880000000000002e-06 2023-07-02T14:03:37.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:37.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080518+0000", 2023-07-02T14:03:37.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1520000000000002e-06 2023-07-02T14:03:37.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:37.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080522+0000", 2023-07-02T14:03:37.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026573699999999999 2023-07-02T14:03:37.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080788+0000", 2023-07-02T14:03:37.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7478999999999998e-05 2023-07-02T14:03:37.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.080826+0000", 2023-07-02T14:03:37.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0094689049999999997 2023-07-02T14:03:37.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090294+0000", 2023-07-02T14:03:37.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5257000000000001e-05 2023-07-02T14:03:37.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.090370+0000", 2023-07-02T14:03:37.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005117229 2023-07-02T14:03:37.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095487+0000", 2023-07-02T14:03:37.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7698e-05 2023-07-02T14:03:37.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.095575+0000", 2023-07-02T14:03:37.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045423210000000002 2023-07-02T14:03:37.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.100117+0000", 2023-07-02T14:03:37.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1317999999999999e-05 2023-07-02T14:03:37.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.100168+0000", 2023-07-02T14:03:37.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035847650000000002 2023-07-02T14:03:37.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103753+0000", 2023-07-02T14:03:37.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3754999999999998e-05 2023-07-02T14:03:37.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.103787+0000", 2023-07-02T14:03:37.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0057476209999999996 2023-07-02T14:03:37.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109534+0000", 2023-07-02T14:03:37.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9867000000000001e-05 2023-07-02T14:03:37.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.109594+0000", 2023-07-02T14:03:37.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024983840000000002 2023-07-02T14:03:37.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.112093+0000", 2023-07-02T14:03:37.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0919e-05 2023-07-02T14:03:37.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.112124+0000", 2023-07-02T14:03:37.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047877880000000003 2023-07-02T14:03:37.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.116911+0000", 2023-07-02T14:03:37.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1793999999999996e-05 2023-07-02T14:03:37.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.117003+0000", 2023-07-02T14:03:37.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059009860000000004 2023-07-02T14:03:37.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122904+0000", 2023-07-02T14:03:37.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8408000000000001e-05 2023-07-02T14:03:37.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.122953+0000", 2023-07-02T14:03:37.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034502040000000001 2023-07-02T14:03:37.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126403+0000", 2023-07-02T14:03:37.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4770000000000005e-05 2023-07-02T14:03:37.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.126468+0000", 2023-07-02T14:03:37.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037365950000000001 2023-07-02T14:03:37.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.130204+0000", 2023-07-02T14:03:37.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7163000000000002e-05 2023-07-02T14:03:37.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.130251+0000", 2023-07-02T14:03:37.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031607509999999998 2023-07-02T14:03:37.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.133412+0000", 2023-07-02T14:03:37.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1032000000000005e-05 2023-07-02T14:03:37.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.133503+0000", 2023-07-02T14:03:37.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041497349999999999 2023-07-02T14:03:37.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:37.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.137653+0000", 2023-07-02T14:03:37.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7430999999999997e-05 2023-07-02T14:03:37.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:37.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.137740+0000", 2023-07-02T14:03:37.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026871349999999998 2023-07-02T14:03:37.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:37.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:37.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140427+0000", 2023-07-02T14:03:38.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7179999999999998e-05 2023-07-02T14:03:38.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140485+0000", 2023-07-02T14:03:38.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00039492900000000003 2023-07-02T14:03:38.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:03:38.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.140879+0000", 2023-07-02T14:03:38.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018682359999999999 2023-07-02T14:03:38.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:38.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.142748+0000", 2023-07-02T14:03:38.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4937e-05 2023-07-02T14:03:38.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:38.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.142793+0000", 2023-07-02T14:03:38.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4743999999999998e-05 2023-07-02T14:03:38.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:38.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.142827+0000", 2023-07-02T14:03:38.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2118999999999999e-05 2023-07-02T14:03:38.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:12.142849+0000", 2023-07-02T14:03:38.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.062340277999999999 2023-07-02T14:03:38.004 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.004 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.004 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.004 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.004 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.004 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:38.005 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:38.006 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:03:38.007 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:38.007 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:38.008 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:03:38.008 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:03:38.008 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:03:38.008 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:03:38.008 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:03:38.009 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:03:38.009 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:38.005+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:38.072 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:03:38.081 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete...2023-07-02T14:03:38.080+0000 7fe5f29f6700 0 -- 172.21.15.163:0/320804245 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7fe5d4036c50 msgr2=0x7fe5d4039040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:03:38.132 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:38.132+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:38.147 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:03:38.146+0000 7fe5f29f6700 0 -- 172.21.15.163:0/320804245 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x55bcb337a0d0 msgr2=0x7fe5d403ba10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:03:38.164 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 51% complete... Exporting image: 100% complete...done. 2023-07-02T14:03:38.168 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:03:38.169 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=11 2023-07-02T14:03:38.169 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap11 2023-07-02T14:03:38.208 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:38.207+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:38.254 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:38.255 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:38.255 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:38.255 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:38.269 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:03:38.310 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:38.310+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:38.397 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4] out_osds: [5, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2023-07-02T14:03:38.398 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:03:38.398 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2023-07-02T14:03:38.399 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-07-02T14:03:38.434 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:38.434 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:38.434 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:38.435 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:38.435 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:38.435 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:38.434+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:38.445 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:03:38.530 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:38.530+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:38.610 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:38.609+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:38.615 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:38.616 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:03:38.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:03:38.616 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:38.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.616 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:183 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:38.616 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:38.617 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.328448162000001, 2023-07-02T14:03:38.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003, 2023-07-02T14:03:38.617 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:38.617 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:38.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:38.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:38.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:38.618 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 183 2023-07-02T14:03:38.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.618 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:38.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:38.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:38.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:38.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:38.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:38.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:38.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:38.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:38.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9790999999999997e-05 2023-07-02T14:03:38.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:38.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285598+0000", 2023-07-02T14:03:38.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5809999999999996e-06 2023-07-02T14:03:38.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:38.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285604+0000", 2023-07-02T14:03:38.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7599999999999998e-06 2023-07-02T14:03:38.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:38.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285607+0000", 2023-07-02T14:03:38.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021108188999999999 2023-07-02T14:03:38.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306715+0000", 2023-07-02T14:03:38.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6548e-05 2023-07-02T14:03:38.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306792+0000", 2023-07-02T14:03:38.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173869 2023-07-02T14:03:38.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:38.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306965+0000", 2023-07-02T14:03:38.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060498009999999996 2023-07-02T14:03:38.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:38.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313015+0000", 2023-07-02T14:03:38.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049443719999999998 2023-07-02T14:03:38.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:38.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317960+0000", 2023-07-02T14:03:38.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5927999999999998e-05 2023-07-02T14:03:38.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:38.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318006+0000", 2023-07-02T14:03:38.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6004999999999998e-05 2023-07-02T14:03:38.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318042+0000", 2023-07-02T14:03:38.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003 2023-07-02T14:03:38.626 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.626 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.626 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.627 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:186 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:38.627 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:38.627 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.328377336000003, 2023-07-02T14:03:38.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003, 2023-07-02T14:03:38.627 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:38.627 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:38.628 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:38.628 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:38.628 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:38.628 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 186 2023-07-02T14:03:38.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.628 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:38.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:38.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:38.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:38.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:38.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:38.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:38.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:38.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:38.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4279e-05 2023-07-02T14:03:38.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:38.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285664+0000", 2023-07-02T14:03:38.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3789999999999995e-06 2023-07-02T14:03:38.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:38.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285673+0000", 2023-07-02T14:03:38.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499999999999998e-06 2023-07-02T14:03:38.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:38.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285678+0000", 2023-07-02T14:03:38.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023261947000000002 2023-07-02T14:03:38.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.308940+0000", 2023-07-02T14:03:38.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010323399999999999 2023-07-02T14:03:38.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309044+0000", 2023-07-02T14:03:38.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018878300000000001 2023-07-02T14:03:38.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:38.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309232+0000", 2023-07-02T14:03:38.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037874150000000001 2023-07-02T14:03:38.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:38.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313020+0000", 2023-07-02T14:03:38.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050976140000000003 2023-07-02T14:03:38.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:38.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318117+0000", 2023-07-02T14:03:38.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3182000000000001e-05 2023-07-02T14:03:38.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:38.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:38.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5055999999999998e-05 2023-07-02T14:03:38.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318196+0000", 2023-07-02T14:03:38.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003 2023-07-02T14:03:38.637 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.637 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.637 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.638 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:189 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:38.638 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:03:38.638 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.328303572999999, 2023-07-02T14:03:38.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997, 2023-07-02T14:03:38.638 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:38.638 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:38.638 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:38.638 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:38.639 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:38.639 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:03:38.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.639 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:38.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:38.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:03:38.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:38.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:38.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:03:38.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:38.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:38.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285683+0000", 2023-07-02T14:03:38.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8451999999999999e-05 2023-07-02T14:03:38.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:38.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285782+0000", 2023-07-02T14:03:38.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3650000000000004e-06 2023-07-02T14:03:38.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:38.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285790+0000", 2023-07-02T14:03:38.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8230000000000002e-06 2023-07-02T14:03:38.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:38.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285795+0000", 2023-07-02T14:03:38.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024619367999999999 2023-07-02T14:03:38.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310414+0000", 2023-07-02T14:03:38.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014244600000000001 2023-07-02T14:03:38.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310557+0000", 2023-07-02T14:03:38.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020612699999999999 2023-07-02T14:03:38.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:38.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310763+0000", 2023-07-02T14:03:38.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003427687 2023-07-02T14:03:38.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:38.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314191+0000", 2023-07-02T14:03:38.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040492089999999998 2023-07-02T14:03:38.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:38.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318240+0000", 2023-07-02T14:03:38.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0287999999999999e-05 2023-07-02T14:03:38.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:38.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318270+0000", 2023-07-02T14:03:38.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0868000000000002e-05 2023-07-02T14:03:38.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318311+0000", 2023-07-02T14:03:38.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997 2023-07-02T14:03:38.647 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.648 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.648 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.648 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:192 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:38.648 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:03:38.648 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.328186799999997, 2023-07-02T14:03:38.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003, 2023-07-02T14:03:38.649 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:38.649 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:38.649 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:38.649 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:38.649 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:38.649 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 192 2023-07-02T14:03:38.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.650 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:38.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:38.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:03:38.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:38.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:38.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:03:38.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:38.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:38.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285800+0000", 2023-07-02T14:03:38.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100417 2023-07-02T14:03:38.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:38.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285900+0000", 2023-07-02T14:03:38.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8244999999999999e-05 2023-07-02T14:03:38.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:38.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285919+0000", 2023-07-02T14:03:38.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2499999999999997e-06 2023-07-02T14:03:38.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:38.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285924+0000", 2023-07-02T14:03:38.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025288155999999999 2023-07-02T14:03:38.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311212+0000", 2023-07-02T14:03:38.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4073999999999996e-05 2023-07-02T14:03:38.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311286+0000", 2023-07-02T14:03:38.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016219699999999999 2023-07-02T14:03:38.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:38.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311448+0000", 2023-07-02T14:03:38.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066020599999999999 2023-07-02T14:03:38.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:38.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318050+0000", 2023-07-02T14:03:38.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000327986 2023-07-02T14:03:38.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:38.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318378+0000", 2023-07-02T14:03:38.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8693999999999998e-05 2023-07-02T14:03:38.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:38.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318427+0000", 2023-07-02T14:03:38.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6656999999999998e-05 2023-07-02T14:03:38.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318464+0000", 2023-07-02T14:03:38.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003 2023-07-02T14:03:38.658 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.658 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.658 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.659 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:195 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:38.659 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:03:38.659 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.328055296000002, 2023-07-02T14:03:38.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001, 2023-07-02T14:03:38.659 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:38.659 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:38.659 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:38.659 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:38.660 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:38.660 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 195 2023-07-02T14:03:38.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.660 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:38.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:38.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:03:38.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:38.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:38.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:03:38.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:38.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:38.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285931+0000", 2023-07-02T14:03:38.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4530999999999995e-05 2023-07-02T14:03:38.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:38.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286006+0000", 2023-07-02T14:03:38.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4113e-05 2023-07-02T14:03:38.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:38.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286020+0000", 2023-07-02T14:03:38.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7459999999999997e-06 2023-07-02T14:03:38.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:38.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286026+0000", 2023-07-02T14:03:38.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025997769 2023-07-02T14:03:38.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312023+0000", 2023-07-02T14:03:38.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4403000000000001e-05 2023-07-02T14:03:38.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312078+0000", 2023-07-02T14:03:38.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020581399999999999 2023-07-02T14:03:38.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:38.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312284+0000", 2023-07-02T14:03:38.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005771043 2023-07-02T14:03:38.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:38.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318055+0000", 2023-07-02T14:03:38.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00046015200000000002 2023-07-02T14:03:38.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:38.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318515+0000", 2023-07-02T14:03:38.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9444000000000003e-05 2023-07-02T14:03:38.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:38.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318564+0000", 2023-07-02T14:03:38.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2141000000000002e-05 2023-07-02T14:03:38.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318606+0000", 2023-07-02T14:03:38.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001 2023-07-02T14:03:38.669 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.670 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.670 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.670 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:198 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:38.670 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:03:38.670 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.327955336999999, 2023-07-02T14:03:38.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999, 2023-07-02T14:03:38.671 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:38.671 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:38.671 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:38.671 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:38.671 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:38.671 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 198 2023-07-02T14:03:38.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.671 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:38.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:38.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:03:38.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:38.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:38.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:03:38.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:38.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:38.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286031+0000", 2023-07-02T14:03:38.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00049286700000000002 2023-07-02T14:03:38.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:38.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286524+0000", 2023-07-02T14:03:38.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4938e-05 2023-07-02T14:03:38.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:38.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286539+0000", 2023-07-02T14:03:38.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0680000000000002e-06 2023-07-02T14:03:38.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:38.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286545+0000", 2023-07-02T14:03:38.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026497718 2023-07-02T14:03:38.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313043+0000", 2023-07-02T14:03:38.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0884999999999998e-05 2023-07-02T14:03:38.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313094+0000", 2023-07-02T14:03:38.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014109999999999999 2023-07-02T14:03:38.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:38.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313235+0000", 2023-07-02T14:03:38.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048258980000000003 2023-07-02T14:03:38.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:38.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318061+0000", 2023-07-02T14:03:38.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014346350000000001 2023-07-02T14:03:38.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:38.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319495+0000", 2023-07-02T14:03:38.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1986e-05 2023-07-02T14:03:38.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:38.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319547+0000", 2023-07-02T14:03:38.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0457e-05 2023-07-02T14:03:38.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319578+0000", 2023-07-02T14:03:38.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999 2023-07-02T14:03:38.680 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.680 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.680 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.681 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:201 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:38.681 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:38.681 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.327434912000001, 2023-07-02T14:03:38.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003, 2023-07-02T14:03:38.681 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:38.681 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:38.681 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:38.682 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:38.682 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:38.682 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 201 2023-07-02T14:03:38.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.682 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:38.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:38.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:38.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:38.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:38.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:38.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:38.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:38.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:38.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019213999999999999 2023-07-02T14:03:38.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:38.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286744+0000", 2023-07-02T14:03:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5279999999999992e-06 2023-07-02T14:03:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286753+0000", 2023-07-02T14:03:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4200000000000004e-06 2023-07-02T14:03:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:38.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286760+0000", 2023-07-02T14:03:38.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026818386999999999 2023-07-02T14:03:38.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313578+0000", 2023-07-02T14:03:38.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015965 2023-07-02T14:03:38.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313738+0000", 2023-07-02T14:03:38.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015767299999999999 2023-07-02T14:03:38.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:38.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313895+0000", 2023-07-02T14:03:38.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041714930000000001 2023-07-02T14:03:38.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:38.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318067+0000", 2023-07-02T14:03:38.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001561698 2023-07-02T14:03:38.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:38.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319629+0000", 2023-07-02T14:03:38.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2346000000000003e-05 2023-07-02T14:03:38.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:38.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319681+0000", 2023-07-02T14:03:38.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4691000000000003e-05 2023-07-02T14:03:38.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319726+0000", 2023-07-02T14:03:38.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003 2023-07-02T14:03:38.691 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.691 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.691 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.691 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:204 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:38.692 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:38.692 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.327220074000003, 2023-07-02T14:03:38.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998, 2023-07-02T14:03:38.692 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:38.692 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:38.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:38.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:38.693 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:38.693 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 204 2023-07-02T14:03:38.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.693 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:38.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:38.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:38.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:38.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:38.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:38.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:38.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:38.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:38.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2497999999999998e-05 2023-07-02T14:03:38.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:38.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286819+0000", 2023-07-02T14:03:38.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0099999999999995e-06 2023-07-02T14:03:38.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:38.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286827+0000", 2023-07-02T14:03:38.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000003e-06 2023-07-02T14:03:38.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:38.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286833+0000", 2023-07-02T14:03:38.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027384716999999999 2023-07-02T14:03:38.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314218+0000", 2023-07-02T14:03:38.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6204000000000004e-05 2023-07-02T14:03:38.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314294+0000", 2023-07-02T14:03:38.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015757499999999999 2023-07-02T14:03:38.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:38.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314452+0000", 2023-07-02T14:03:38.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036215560000000002 2023-07-02T14:03:38.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:38.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318073+0000", 2023-07-02T14:03:38.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017023979999999999 2023-07-02T14:03:38.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:38.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319776+0000", 2023-07-02T14:03:38.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9561000000000002e-05 2023-07-02T14:03:38.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:38.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319825+0000", 2023-07-02T14:03:38.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0329e-05 2023-07-02T14:03:38.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319866+0000", 2023-07-02T14:03:38.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998 2023-07-02T14:03:38.701 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.701 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.702 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.702 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:207 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:38.702 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:03:38.702 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.327145428999998, 2023-07-02T14:03:38.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003, 2023-07-02T14:03:38.702 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:38.703 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:38.703 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:38.703 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:38.703 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:38.703 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 207 2023-07-02T14:03:38.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.703 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:38.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:38.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:03:38.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:38.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:38.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:03:38.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:38.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:38.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286841+0000", 2023-07-02T14:03:38.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00068375100000000002 2023-07-02T14:03:38.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:38.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287525+0000", 2023-07-02T14:03:38.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1601e-05 2023-07-02T14:03:38.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:38.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287537+0000", 2023-07-02T14:03:38.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6409999999999996e-06 2023-07-02T14:03:38.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:38.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287543+0000", 2023-07-02T14:03:38.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027245844000000002 2023-07-02T14:03:38.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314789+0000", 2023-07-02T14:03:38.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8910000000000003e-05 2023-07-02T14:03:38.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314858+0000", 2023-07-02T14:03:38.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030992599999999997 2023-07-02T14:03:38.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:38.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315168+0000", 2023-07-02T14:03:38.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029122869999999999 2023-07-02T14:03:38.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:38.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318080+0000", 2023-07-02T14:03:38.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019963020000000001 2023-07-02T14:03:38.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:38.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320077+0000", 2023-07-02T14:03:38.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5116000000000005e-05 2023-07-02T14:03:38.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:38.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320142+0000", 2023-07-02T14:03:38.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3793000000000004e-05 2023-07-02T14:03:38.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320205+0000", 2023-07-02T14:03:38.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003 2023-07-02T14:03:38.712 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.712 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.712 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.712 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:236 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2424832~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:38.713 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:03:38.713 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.326433246000001, 2023-07-02T14:03:38.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003, 2023-07-02T14:03:38.713 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:38.713 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:38.713 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:38.714 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:38.714 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:38.714 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2023-07-02T14:03:38.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.714 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:38.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:38.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:03:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:03:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287553+0000", 2023-07-02T14:03:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0667e-05 2023-07-02T14:03:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287614+0000", 2023-07-02T14:03:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0454e-05 2023-07-02T14:03:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287624+0000", 2023-07-02T14:03:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8564999999999999e-05 2023-07-02T14:03:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287643+0000", 2023-07-02T14:03:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027977596 2023-07-02T14:03:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315621+0000", 2023-07-02T14:03:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000201728 2023-07-02T14:03:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315822+0000", 2023-07-02T14:03:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018436599999999999 2023-07-02T14:03:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316007+0000", 2023-07-02T14:03:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042423440000000003 2023-07-02T14:03:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320249+0000", 2023-07-02T14:03:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001452633 2023-07-02T14:03:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321702+0000", 2023-07-02T14:03:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3616000000000003e-05 2023-07-02T14:03:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321755+0000", 2023-07-02T14:03:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840999999999999e-05 2023-07-02T14:03:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321793+0000", 2023-07-02T14:03:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003 2023-07-02T14:03:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:251 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:03:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.326337709000001, 2023-07-02T14:03:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001, 2023-07-02T14:03:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 251 2023-07-02T14:03:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:03:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:03:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287649+0000", 2023-07-02T14:03:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3232000000000001e-05 2023-07-02T14:03:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287702+0000", 2023-07-02T14:03:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0010000000000007e-06 2023-07-02T14:03:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287711+0000", 2023-07-02T14:03:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3569999999999997e-06 2023-07-02T14:03:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287717+0000", 2023-07-02T14:03:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028686454 2023-07-02T14:03:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316403+0000", 2023-07-02T14:03:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7741000000000002e-05 2023-07-02T14:03:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316461+0000", 2023-07-02T14:03:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4313000000000004e-05 2023-07-02T14:03:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316545+0000", 2023-07-02T14:03:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003734032 2023-07-02T14:03:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320279+0000", 2023-07-02T14:03:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015237219999999999 2023-07-02T14:03:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321803+0000", 2023-07-02T14:03:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1927e-05 2023-07-02T14:03:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321845+0000", 2023-07-02T14:03:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5398999999999998e-05 2023-07-02T14:03:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321880+0000", 2023-07-02T14:03:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001 2023-07-02T14:03:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:255 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2555904~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:03:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.325922980999998, 2023-07-02T14:03:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002, 2023-07-02T14:03:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 255 2023-07-02T14:03:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:03:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:03:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288063+0000", 2023-07-02T14:03:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4459999999999997e-05 2023-07-02T14:03:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288118+0000", 2023-07-02T14:03:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0998999999999999e-05 2023-07-02T14:03:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288129+0000", 2023-07-02T14:03:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3219999999999998e-06 2023-07-02T14:03:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288135+0000", 2023-07-02T14:03:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029137386000000001 2023-07-02T14:03:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317273+0000", 2023-07-02T14:03:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7819999999999997e-05 2023-07-02T14:03:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317350+0000", 2023-07-02T14:03:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000174864 2023-07-02T14:03:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317525+0000", 2023-07-02T14:03:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027943519999999999 2023-07-02T14:03:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320320+0000", 2023-07-02T14:03:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015681460000000001 2023-07-02T14:03:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321888+0000", 2023-07-02T14:03:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9508999999999997e-05 2023-07-02T14:03:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321927+0000", 2023-07-02T14:03:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5428e-05 2023-07-02T14:03:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321963+0000", 2023-07-02T14:03:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002 2023-07-02T14:03:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:446 2.1 2:910b2a90:::rbd_data.125de1449605.000000000000010a:head [write 65536~3211264] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.248643229000002, 2023-07-02T14:03:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003, 2023-07-02T14:03:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 446 2023-07-02T14:03:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015907372999999999 2023-07-02T14:03:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381251+0000", 2023-07-02T14:03:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5527000000000001e-05 2023-07-02T14:03:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381277+0000", 2023-07-02T14:03:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9958000000000001e-05 2023-07-02T14:03:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381297+0000", 2023-07-02T14:03:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4941000000000001e-05 2023-07-02T14:03:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381352+0000", 2023-07-02T14:03:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011565699999999999 2023-07-02T14:03:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381467+0000", 2023-07-02T14:03:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018039799999999999 2023-07-02T14:03:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381648+0000", 2023-07-02T14:03:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088183559999999994 2023-07-02T14:03:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.390466+0000", 2023-07-02T14:03:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007278278 2023-07-02T14:03:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397744+0000", 2023-07-02T14:03:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9289000000000001e-05 2023-07-02T14:03:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397804+0000", 2023-07-02T14:03:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0479000000000002e-05 2023-07-02T14:03:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397854+0000", 2023-07-02T14:03:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003 2023-07-02T14:03:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:03:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:03:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.506312434999998, 2023-07-02T14:03:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:03:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:03:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:03:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:03:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:38.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:03:38.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:38.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:38.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:03:38.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:03:38.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:38.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:03:38.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:03:38.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:38.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:03:38.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:03:38.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:38.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:03:38.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:03:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:03:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:03:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:03:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:03:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:03:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:03:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:03:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:03:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:03:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:03:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:03:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:03:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.506251697000003, 2023-07-02T14:03:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:03:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:03:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:03:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:03:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:03:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:03:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:03:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:03:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:03:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:03:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:03:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:03:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:03:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:03:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:03:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:03:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:03:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:03:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:03:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:03:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:03:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:03:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:03:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:03:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:03:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:03:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:03:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:03:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:03:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:03:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:03:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.275797366999999, 2023-07-02T14:03:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:03:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:38.781 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:03:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:03:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:38.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:03:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:38.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:03:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:03:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:03:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:03:38.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:03:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:03:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:38.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:03:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:03:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:03:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:03:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:03:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:03:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:03:38.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:03:38.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:38.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:03:38.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:03:38.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:03:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:03:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:03:38.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:03:38.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:03:38.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:03:38.790 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.790 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.790 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2941 2.3 2:d3762e63:::rbd_data.125de1449605.00000000000000f8:head [write 65536~2031616] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:03:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.146694412999999, 2023-07-02T14:03:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002, 2023-07-02T14:03:38.791 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2941 2023-07-02T14:03:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:38.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:03:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:03:38.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467291+0000", 2023-07-02T14:03:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043022859999999998 2023-07-02T14:03:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471593+0000", 2023-07-02T14:03:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0999999999999999e-05 2023-07-02T14:03:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471614+0000", 2023-07-02T14:03:38.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0097e-05 2023-07-02T14:03:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471625+0000", 2023-07-02T14:03:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069808499999999998 2023-07-02T14:03:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472323+0000", 2023-07-02T14:03:38.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135635 2023-07-02T14:03:38.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472458+0000", 2023-07-02T14:03:38.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223365 2023-07-02T14:03:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472682+0000", 2023-07-02T14:03:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013761944999999999 2023-07-02T14:03:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.486444+0000", 2023-07-02T14:03:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016859747000000001 2023-07-02T14:03:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503303+0000", 2023-07-02T14:03:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6635000000000001e-05 2023-07-02T14:03:38.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503380+0000", 2023-07-02T14:03:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000161803 2023-07-02T14:03:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503542+0000", 2023-07-02T14:03:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002 2023-07-02T14:03:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2950 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 1507328~786432] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:03:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.138521104999999, 2023-07-02T14:03:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999, 2023-07-02T14:03:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:38.802 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2950 2023-07-02T14:03:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:03:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:03:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:03:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001147854 2023-07-02T14:03:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476614+0000", 2023-07-02T14:03:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.802e-05 2023-07-02T14:03:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476632+0000", 2023-07-02T14:03:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1860000000000003e-06 2023-07-02T14:03:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476639+0000", 2023-07-02T14:03:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061211240000000004 2023-07-02T14:03:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482760+0000", 2023-07-02T14:03:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8833999999999996e-05 2023-07-02T14:03:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482859+0000", 2023-07-02T14:03:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019270600000000001 2023-07-02T14:03:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.483051+0000", 2023-07-02T14:03:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060385630000000003 2023-07-02T14:03:38.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489090+0000", 2023-07-02T14:03:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024506918999999999 2023-07-02T14:03:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513597+0000", 2023-07-02T14:03:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9672999999999995e-05 2023-07-02T14:03:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513667+0000", 2023-07-02T14:03:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024329500000000001 2023-07-02T14:03:38.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513910+0000", 2023-07-02T14:03:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999 2023-07-02T14:03:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2951 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2686976~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:03:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.137340327, 2023-07-02T14:03:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999, 2023-07-02T14:03:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:38.812 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2951 2023-07-02T14:03:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:03:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:03:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:38.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476646+0000", 2023-07-02T14:03:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7639000000000003e-05 2023-07-02T14:03:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:38.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476684+0000", 2023-07-02T14:03:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3508000000000002e-05 2023-07-02T14:03:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476747+0000", 2023-07-02T14:03:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8499999999999994e-06 2023-07-02T14:03:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476755+0000", 2023-07-02T14:03:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078346160000000008 2023-07-02T14:03:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484590+0000", 2023-07-02T14:03:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7206000000000004e-05 2023-07-02T14:03:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484687+0000", 2023-07-02T14:03:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016243099999999999 2023-07-02T14:03:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484849+0000", 2023-07-02T14:03:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042482860000000004 2023-07-02T14:03:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489097+0000", 2023-07-02T14:03:38.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.02489504 2023-07-02T14:03:38.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:38.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513992+0000", 2023-07-02T14:03:38.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9972000000000001e-05 2023-07-02T14:03:38.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:38.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514052+0000", 2023-07-02T14:03:38.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149264 2023-07-02T14:03:38.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514202+0000", 2023-07-02T14:03:38.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999 2023-07-02T14:03:38.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.822 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.822 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2952 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2883584~12288] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:38.822 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:03:38.823 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.137226492, 2023-07-02T14:03:38.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651, 2023-07-02T14:03:38.823 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:38.823 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:38.823 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:38.823 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:38.823 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:38.823 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2952 2023-07-02T14:03:38.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.824 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:38.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:38.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:03:38.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:38.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:38.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:03:38.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:38.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:38.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476760+0000", 2023-07-02T14:03:38.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3664e-05 2023-07-02T14:03:38.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:38.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476774+0000", 2023-07-02T14:03:38.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.166e-06 2023-07-02T14:03:38.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:38.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476783+0000", 2023-07-02T14:03:38.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8720000000000001e-06 2023-07-02T14:03:38.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:38.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476788+0000", 2023-07-02T14:03:38.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087180529999999999 2023-07-02T14:03:38.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:38.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485506+0000", 2023-07-02T14:03:38.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5680999999999994e-05 2023-07-02T14:03:38.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:38.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485602+0000", 2023-07-02T14:03:38.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000222365 2023-07-02T14:03:38.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:38.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485824+0000", 2023-07-02T14:03:38.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003277379 2023-07-02T14:03:38.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:38.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489101+0000", 2023-07-02T14:03:38.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025151739999999999 2023-07-02T14:03:38.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:38.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514253+0000", 2023-07-02T14:03:38.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4749000000000002e-05 2023-07-02T14:03:38.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:38.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514288+0000", 2023-07-02T14:03:38.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130634 2023-07-02T14:03:38.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:38.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514419+0000", 2023-07-02T14:03:38.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651 2023-07-02T14:03:38.832 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.832 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.832 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.833 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.833 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.833 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:38.834 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:03:38.835 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:03:38.835 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:38.835 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 80, 2023-07-02T14:03:38.836 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2", 2023-07-02T14:03:38.836 INFO:teuthology.orchestra.run.smithi016.stdout: "created": "2023-07-02T14:01:37.298060+0000", 2023-07-02T14:03:38.836 INFO:teuthology.orchestra.run.smithi016.stdout: "modified": "2023-07-02T14:03:38.065168+0000", 2023-07-02T14:03:38.836 INFO:teuthology.orchestra.run.smithi016.stdout: "last_up_change": "2023-07-02T14:01:45.271760+0000", 2023-07-02T14:03:38.836 INFO:teuthology.orchestra.run.smithi016.stdout: "last_in_change": "2023-07-02T14:03:33.379208+0000", 2023-07-02T14:03:38.836 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-07-02T14:03:38.837 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_num": 5806080, 2023-07-02T14:03:38.837 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_set": [ 2023-07-02T14:03:38.837 INFO:teuthology.orchestra.run.smithi016.stdout: "nodeep-scrub", 2023-07-02T14:03:38.837 INFO:teuthology.orchestra.run.smithi016.stdout: "noscrub", 2023-07-02T14:03:38.837 INFO:teuthology.orchestra.run.smithi016.stdout: "pglog_hardlimit", 2023-07-02T14:03:38.837 INFO:teuthology.orchestra.run.smithi016.stdout: "purged_snapdirs", 2023-07-02T14:03:38.837 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_deletes", 2023-07-02T14:03:38.837 INFO:teuthology.orchestra.run.smithi016.stdout: "sortbitwise" 2023-07-02T14:03:38.838 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:03:38.838 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_version": 4, 2023-07-02T14:03:38.838 INFO:teuthology.orchestra.run.smithi016.stdout: "full_ratio": 0.94999998807907104, 2023-07-02T14:03:38.838 INFO:teuthology.orchestra.run.smithi016.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-07-02T14:03:38.838 INFO:teuthology.orchestra.run.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2023-07-02T14:03:38.838 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_snapshot": "", 2023-07-02T14:03:38.838 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_max": 2, 2023-07-02T14:03:38.838 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd": 6, 2023-07-02T14:03:38.839 INFO:teuthology.orchestra.run.smithi016.stdout: "require_min_compat_client": "luminous", 2023-07-02T14:03:38.839 INFO:teuthology.orchestra.run.smithi016.stdout: "min_compat_client": "luminous", 2023-07-02T14:03:38.839 INFO:teuthology.orchestra.run.smithi016.stdout: "require_osd_release": "pacific", 2023-07-02T14:03:38.839 INFO:teuthology.orchestra.run.smithi016.stdout: "pools": [ 2023-07-02T14:03:38.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.839 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 1, 2023-07-02T14:03:38.839 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "device_health_metrics", 2023-07-02T14:03:38.840 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:45.323455+0000", 2023-07-02T14:03:38.840 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2023-07-02T14:03:38.840 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2023-07-02T14:03:38.840 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:03:38.840 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:03:38.840 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:03:38.840 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:03:38.840 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:03:38.841 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:03:38.841 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:03:38.841 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:03:38.841 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:03:38.841 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:03:38.841 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 1, 2023-07-02T14:03:38.841 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 1, 2023-07-02T14:03:38.841 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 1, 2023-07-02T14:03:38.842 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 1, 2023-07-02T14:03:38.842 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 1, 2023-07-02T14:03:38.842 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:03:38.842 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:03:38.842 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:03:38.843 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:03:38.843 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:03:38.843 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:03:38.843 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:03:38.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.843 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "14", 2023-07-02T14:03:38.843 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:03:38.844 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:03:38.844 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:03:38.844 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:03:38.844 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:03:38.844 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2023-07-02T14:03:38.844 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2023-07-02T14:03:38.844 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:03:38.845 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:03:38.845 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:03:38.845 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:03:38.845 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:03:38.845 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:03:38.845 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:03:38.845 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:03:38.845 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:03:38.846 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:03:38.846 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:03:38.846 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:03:38.846 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:03:38.846 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:03:38.846 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:03:38.846 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:03:38.846 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:03:38.847 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:03:38.847 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:03:38.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.847 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:03:38.847 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:03:38.847 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:03:38.847 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:03:38.848 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:03:38.848 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:03:38.848 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:03:38.848 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:03:38.848 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:03:38.848 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:03:38.848 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:03:38.848 INFO:teuthology.orchestra.run.smithi016.stdout: "options": { 2023-07-02T14:03:38.849 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_max": 32, 2023-07-02T14:03:38.849 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_min": 1 2023-07-02T14:03:38.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.849 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:03:38.849 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_devicehealth": {} 2023-07-02T14:03:38.849 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.850 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 2, 2023-07-02T14:03:38.850 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "rbd", 2023-07-02T14:03:38.850 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:47.448897+0000", 2023-07-02T14:03:38.850 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 8193, 2023-07-02T14:03:38.850 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2023-07-02T14:03:38.850 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:03:38.850 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:03:38.851 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:03:38.851 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:03:38.851 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:03:38.851 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:03:38.851 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:03:38.851 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:03:38.851 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:03:38.851 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:03:38.852 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2023-07-02T14:03:38.852 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2023-07-02T14:03:38.852 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2023-07-02T14:03:38.852 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2023-07-02T14:03:38.852 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2023-07-02T14:03:38.852 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:03:38.853 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:03:38.853 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:03:38.853 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:03:38.853 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:03:38.854 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:03:38.854 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:03:38.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.854 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "76", 2023-07-02T14:03:38.855 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:03:38.855 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:03:38.855 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:03:38.856 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:03:38.856 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:03:38.856 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 13, 2023-07-02T14:03:38.856 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 76, 2023-07-02T14:03:38.857 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:03:38.857 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:03:38.857 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:03:38.858 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:03:38.858 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:03:38.858 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:03:38.858 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:03:38.858 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:03:38.859 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:03:38.859 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:03:38.859 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:03:38.859 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:03:38.859 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:03:38.859 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:03:38.859 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:03:38.859 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:03:38.860 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:03:38.860 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:03:38.860 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:03:38.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.860 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:03:38.860 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:03:38.861 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:03:38.861 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:03:38.861 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:03:38.861 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:03:38.862 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:03:38.862 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:03:38.862 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:03:38.862 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:03:38.862 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:03:38.862 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2023-07-02T14:03:38.863 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:03:38.863 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2023-07-02T14:03:38.863 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.863 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.863 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:03:38.863 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:03:38.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.863 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:03:38.864 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "43c446fe-c811-4ee3-99e8-d7452edfc1c1", 2023-07-02T14:03:38.864 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:03:38.864 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:03:38.864 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:03:38.864 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:03:38.864 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:03:38.864 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:03:38.864 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:03:38.865 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 71, 2023-07-02T14:03:38.865 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:03:38.865 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:03:38.865 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:03:38.866 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.866 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.866 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2023-07-02T14:03:38.866 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:38.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.867 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.867 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2023-07-02T14:03:38.867 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:38.867 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.867 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.867 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:03:38.867 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.868 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.868 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6802", 2023-07-02T14:03:38.868 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:38.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.868 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.869 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6803", 2023-07-02T14:03:38.869 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:38.869 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.869 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.869 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:03:38.869 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.870 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.870 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6806", 2023-07-02T14:03:38.870 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:38.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.871 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.871 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6807", 2023-07-02T14:03:38.872 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:38.872 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.872 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.872 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:03:38.872 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.873 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.873 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6804", 2023-07-02T14:03:38.873 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:38.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.873 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.874 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6805", 2023-07-02T14:03:38.874 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83359 2023-07-02T14:03:38.874 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.874 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.874 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6801/83359", 2023-07-02T14:03:38.874 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6803/83359", 2023-07-02T14:03:38.874 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6807/83359", 2023-07-02T14:03:38.875 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6805/83359", 2023-07-02T14:03:38.875 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:03:38.875 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:03:38.875 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:03:38.875 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.876 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:03:38.876 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b", 2023-07-02T14:03:38.876 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:03:38.876 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:03:38.876 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:03:38.876 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:03:38.876 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:03:38.876 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:03:38.877 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:03:38.877 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 79, 2023-07-02T14:03:38.877 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:03:38.877 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:03:38.877 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:03:38.877 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.878 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.878 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6808", 2023-07-02T14:03:38.878 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:38.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.878 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.878 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6809", 2023-07-02T14:03:38.879 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:38.879 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.879 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.879 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:03:38.879 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.880 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.880 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6810", 2023-07-02T14:03:38.880 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:38.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.880 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.880 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6811", 2023-07-02T14:03:38.881 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:38.881 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.881 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.881 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:03:38.881 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.882 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.882 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6814", 2023-07-02T14:03:38.882 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:38.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.882 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.882 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6815", 2023-07-02T14:03:38.882 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:38.883 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.883 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.883 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:03:38.883 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.883 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.883 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6812", 2023-07-02T14:03:38.884 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:38.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.884 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.884 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6813", 2023-07-02T14:03:38.884 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:03:38.884 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.885 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.885 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6809/83360", 2023-07-02T14:03:38.885 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6811/83360", 2023-07-02T14:03:38.885 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6815/83360", 2023-07-02T14:03:38.885 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6813/83360", 2023-07-02T14:03:38.885 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:03:38.885 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:03:38.886 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:03:38.886 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.886 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:03:38.887 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "99f8ea0a-5a71-4abe-b087-8e891fd11d1b", 2023-07-02T14:03:38.887 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:03:38.887 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 0, 2023-07-02T14:03:38.888 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 0, 2023-07-02T14:03:38.888 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:03:38.888 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:03:38.889 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:03:38.889 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:03:38.889 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 62, 2023-07-02T14:03:38.889 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:03:38.890 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:03:38.890 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:03:38.890 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.890 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.890 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6816", 2023-07-02T14:03:38.891 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:38.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.891 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.891 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6817", 2023-07-02T14:03:38.891 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:38.891 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.891 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.892 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:03:38.892 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.892 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.892 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6818", 2023-07-02T14:03:38.892 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:38.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.893 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.893 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6819", 2023-07-02T14:03:38.893 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:38.893 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.893 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.894 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:03:38.894 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.894 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.894 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6822", 2023-07-02T14:03:38.894 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:38.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.895 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.895 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6823", 2023-07-02T14:03:38.895 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:38.895 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.895 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.895 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:03:38.895 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.896 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.896 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6820", 2023-07-02T14:03:38.896 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:38.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.896 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.896 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6821", 2023-07-02T14:03:38.897 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:03:38.897 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.897 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.897 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6817/83361", 2023-07-02T14:03:38.897 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6819/83361", 2023-07-02T14:03:38.897 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6823/83361", 2023-07-02T14:03:38.897 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6821/83361", 2023-07-02T14:03:38.898 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:03:38.898 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:03:38.898 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:03:38.898 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.898 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:03:38.898 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "d285b4d5-0bef-421a-b84a-943e485b10c2", 2023-07-02T14:03:38.899 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:03:38.899 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:03:38.899 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:03:38.899 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:03:38.899 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:03:38.899 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:03:38.899 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:03:38.899 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 63, 2023-07-02T14:03:38.900 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:03:38.900 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:03:38.900 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:03:38.900 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.900 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.900 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6808", 2023-07-02T14:03:38.901 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:38.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.901 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.901 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6809", 2023-07-02T14:03:38.901 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:38.901 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.901 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.902 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:03:38.902 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.902 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.902 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6810", 2023-07-02T14:03:38.902 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:38.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.903 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.903 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6811", 2023-07-02T14:03:38.903 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:38.903 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.903 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.903 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:03:38.904 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.904 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.904 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6814", 2023-07-02T14:03:38.904 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:38.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.904 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.905 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6815", 2023-07-02T14:03:38.905 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:38.905 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.905 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.905 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:03:38.905 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.906 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.906 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6812", 2023-07-02T14:03:38.906 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:38.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.906 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.906 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6813", 2023-07-02T14:03:38.906 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83350 2023-07-02T14:03:38.907 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.907 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.907 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6809/83350", 2023-07-02T14:03:38.907 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6811/83350", 2023-07-02T14:03:38.907 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6815/83350", 2023-07-02T14:03:38.907 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6813/83350", 2023-07-02T14:03:38.908 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:03:38.908 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:03:38.908 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:03:38.908 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.908 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:03:38.908 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "7ba0b0db-e407-4c63-8278-aab09f3082c4", 2023-07-02T14:03:38.909 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:03:38.909 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:03:38.909 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:03:38.909 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 0.23297119140625, 2023-07-02T14:03:38.909 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:03:38.909 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:03:38.909 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:03:38.909 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 0, 2023-07-02T14:03:38.910 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:03:38.910 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:03:38.910 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:03:38.910 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.910 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.910 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6800", 2023-07-02T14:03:38.911 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:38.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.911 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.911 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6801", 2023-07-02T14:03:38.911 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:38.911 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.911 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.912 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:03:38.912 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.912 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.912 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6802", 2023-07-02T14:03:38.912 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:38.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.913 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.913 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6803", 2023-07-02T14:03:38.913 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:38.913 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.913 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.913 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:03:38.914 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.914 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.914 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6806", 2023-07-02T14:03:38.914 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:38.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.914 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.915 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6807", 2023-07-02T14:03:38.915 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:38.915 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.915 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.915 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:03:38.915 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.916 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.916 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6804", 2023-07-02T14:03:38.916 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:38.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.916 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.916 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6805", 2023-07-02T14:03:38.916 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:03:38.917 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.917 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.917 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6801/83351", 2023-07-02T14:03:38.917 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6803/83351", 2023-07-02T14:03:38.917 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6807/83351", 2023-07-02T14:03:38.917 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6805/83351", 2023-07-02T14:03:38.917 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:03:38.918 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:03:38.918 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:03:38.918 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.918 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:03:38.918 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "29663bfa-f6fe-4ce7-b45a-a9b7b4424c42", 2023-07-02T14:03:38.919 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:03:38.919 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 0, 2023-07-02T14:03:38.919 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 0, 2023-07-02T14:03:38.919 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:03:38.919 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:03:38.919 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:03:38.919 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:03:38.919 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 34, 2023-07-02T14:03:38.920 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:03:38.920 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:03:38.920 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:03:38.920 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.920 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.920 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6816", 2023-07-02T14:03:38.920 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:38.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.921 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.921 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6817", 2023-07-02T14:03:38.921 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:38.921 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.921 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.922 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:03:38.922 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.922 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.922 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6818", 2023-07-02T14:03:38.922 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:38.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.923 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.923 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6819", 2023-07-02T14:03:38.923 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:38.923 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.923 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.923 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:03:38.923 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.924 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.924 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6822", 2023-07-02T14:03:38.924 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:38.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.925 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.925 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6823", 2023-07-02T14:03:38.925 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:38.925 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.925 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.926 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:03:38.926 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:03:38.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.926 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:03:38.926 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6820", 2023-07-02T14:03:38.926 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:38.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.927 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:03:38.927 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6821", 2023-07-02T14:03:38.927 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:03:38.927 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.927 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.927 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6817/83352", 2023-07-02T14:03:38.927 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6819/83352", 2023-07-02T14:03:38.928 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6823/83352", 2023-07-02T14:03:38.928 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6821/83352", 2023-07-02T14:03:38.928 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:03:38.928 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:03:38.928 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:03:38.928 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.928 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.928 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:03:38.929 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_xinfo": [ 2023-07-02T14:03:38.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.929 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:03:38.929 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:03:38.929 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:03:38.929 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:03:38.929 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:03:38.929 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:03:38.930 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.001178+0000", 2023-07-02T14:03:38.930 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:03:38.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.930 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:03:38.930 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:03:38.930 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:03:38.930 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:03:38.931 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:03:38.931 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:03:38.931 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.184181+0000", 2023-07-02T14:03:38.931 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:03:38.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.931 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:03:38.932 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:03:38.932 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:03:38.932 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:03:38.932 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:03:38.932 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 65536, 2023-07-02T14:03:38.932 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.274854+0000", 2023-07-02T14:03:38.932 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:03:38.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.933 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:03:38.933 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:03:38.933 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:03:38.933 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:03:38.933 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:03:38.934 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:03:38.934 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.186217+0000", 2023-07-02T14:03:38.934 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:03:38.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.934 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:03:38.934 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:03:38.934 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:03:38.935 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:03:38.935 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:03:38.935 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:03:38.935 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.247442+0000", 2023-07-02T14:03:38.935 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:03:38.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.935 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:03:38.936 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:03:38.936 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:03:38.936 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:03:38.936 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:03:38.936 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 65536, 2023-07-02T14:03:38.936 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.220075+0000", 2023-07-02T14:03:38.936 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:03:38.936 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.937 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:03:38.937 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap": [ 2023-07-02T14:03:38.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.937 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "1.0", 2023-07-02T14:03:38.937 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:03:38.937 INFO:teuthology.orchestra.run.smithi016.stdout: 0, 2023-07-02T14:03:38.937 INFO:teuthology.orchestra.run.smithi016.stdout: 3 2023-07-02T14:03:38.937 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.938 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.0", 2023-07-02T14:03:38.938 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:03:38.938 INFO:teuthology.orchestra.run.smithi016.stdout: 4, 2023-07-02T14:03:38.938 INFO:teuthology.orchestra.run.smithi016.stdout: 1 2023-07-02T14:03:38.938 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.939 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.2", 2023-07-02T14:03:38.939 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:03:38.939 INFO:teuthology.orchestra.run.smithi016.stdout: 1, 2023-07-02T14:03:38.939 INFO:teuthology.orchestra.run.smithi016.stdout: 5 2023-07-02T14:03:38.939 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:38.940 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.4", 2023-07-02T14:03:38.940 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:03:38.940 INFO:teuthology.orchestra.run.smithi016.stdout: 5, 2023-07-02T14:03:38.940 INFO:teuthology.orchestra.run.smithi016.stdout: 0 2023-07-02T14:03:38.940 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:38.940 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.940 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:03:38.941 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_items": [], 2023-07-02T14:03:38.941 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_temp": [], 2023-07-02T14:03:38.941 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_temp": [], 2023-07-02T14:03:38.941 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklist": {}, 2023-07-02T14:03:38.941 INFO:teuthology.orchestra.run.smithi016.stdout: "range_blocklist": {}, 2023-07-02T14:03:38.941 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profiles": { 2023-07-02T14:03:38.941 INFO:teuthology.orchestra.run.smithi016.stdout: "default": { 2023-07-02T14:03:38.941 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2023-07-02T14:03:38.942 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "2", 2023-07-02T14:03:38.942 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "1", 2023-07-02T14:03:38.942 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "jerasure", 2023-07-02T14:03:38.942 INFO:teuthology.orchestra.run.smithi016.stdout: "ruleset-failure-domain": "osd", 2023-07-02T14:03:38.942 INFO:teuthology.orchestra.run.smithi016.stdout: "technique": "reed_sol_van" 2023-07-02T14:03:38.942 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:38.942 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps_queue": [], 2023-07-02T14:03:38.943 INFO:teuthology.orchestra.run.smithi016.stdout: "new_removed_snaps": [], 2023-07-02T14:03:38.943 INFO:teuthology.orchestra.run.smithi016.stdout: "new_purged_snaps": [], 2023-07-02T14:03:38.943 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_node_flags": {}, 2023-07-02T14:03:38.943 INFO:teuthology.orchestra.run.smithi016.stdout: "device_class_flags": {}, 2023-07-02T14:03:38.943 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode": { 2023-07-02T14:03:38.943 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_enabled": false, 2023-07-02T14:03:38.943 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_bucket_count": 0, 2023-07-02T14:03:38.943 INFO:teuthology.orchestra.run.smithi016.stdout: "degraded_stretch_mode": 0, 2023-07-02T14:03:38.944 INFO:teuthology.orchestra.run.smithi016.stdout: "recovering_stretch_mode": 0, 2023-07-02T14:03:38.944 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_bucket": 0 2023-07-02T14:03:38.944 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:38.944 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:38.945 INFO:tasks.thrashosds.thrasher:j is {'epoch': 80, 'fsid': '9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2', 'created': '2023-07-02T14:01:37.298060+0000', 'modified': '2023-07-02T14:03:38.065168+0000', 'last_up_change': '2023-07-02T14:01:45.271760+0000', 'last_in_change': '2023-07-02T14:03:33.379208+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-07-02T14:01:45.323455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-02T14:01:47.448897+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 13, 'snap_epoch': 76, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '43c446fe-c811-4ee3-99e8-d7452edfc1c1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 71, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6800', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6801', 'nonce': 83359}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6802', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6803', 'nonce': 83359}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6806', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6807', 'nonce': 83359}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6804', 'nonce': 83359}, {'type': 'v1', 'addr': '172.21.15.16:6805', 'nonce': 83359}]}, 'public_addr': '172.21.15.16:6801/83359', 'cluster_addr': '172.21.15.16:6803/83359', 'heartbeat_back_addr': '172.21.15.16:6807/83359', 'heartbeat_front_addr': '172.21.15.16:6805/83359', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 79, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6808', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6809', 'nonce': 83360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6810', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6811', 'nonce': 83360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6814', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6815', 'nonce': 83360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6812', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6813', 'nonce': 83360}]}, 'public_addr': '172.21.15.16:6809/83360', 'cluster_addr': '172.21.15.16:6811/83360', 'heartbeat_back_addr': '172.21.15.16:6815/83360', 'heartbeat_front_addr': '172.21.15.16:6813/83360', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '99f8ea0a-5a71-4abe-b087-8e891fd11d1b', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6816', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6817', 'nonce': 83361}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6818', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6819', 'nonce': 83361}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6822', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6823', 'nonce': 83361}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6820', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6821', 'nonce': 83361}]}, 'public_addr': '172.21.15.16:6817/83361', 'cluster_addr': '172.21.15.16:6819/83361', 'heartbeat_back_addr': '172.21.15.16:6823/83361', 'heartbeat_front_addr': '172.21.15.16:6821/83361', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd285b4d5-0bef-421a-b84a-943e485b10c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 63, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6808', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6809', 'nonce': 83350}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6810', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6811', 'nonce': 83350}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6814', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6815', 'nonce': 83350}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6812', 'nonce': 83350}, {'type': 'v1', 'addr': '172.21.15.22:6813', 'nonce': 83350}]}, 'public_addr': '172.21.15.22:6809/83350', 'cluster_addr': '172.21.15.22:6811/83350', 'heartbeat_back_addr': '172.21.15.22:6815/83350', 'heartbeat_front_addr': '172.21.15.22:6813/83350', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7ba0b0db-e407-4c63-8278-aab09f3082c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.23297119140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6800', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6801', 'nonce': 83351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6802', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6803', 'nonce': 83351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6806', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6807', 'nonce': 83351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6804', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6805', 'nonce': 83351}]}, 'public_addr': '172.21.15.22:6801/83351', 'cluster_addr': '172.21.15.22:6803/83351', 'heartbeat_back_addr': '172.21.15.22:6807/83351', 'heartbeat_front_addr': '172.21.15.22:6805/83351', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '29663bfa-f6fe-4ce7-b45a-a9b7b4424c42', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6816', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6817', 'nonce': 83352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6818', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6819', 'nonce': 83352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6822', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6823', 'nonce': 83352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6820', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6821', 'nonce': 83352}]}, 'public_addr': '172.21.15.22:6817/83352', 'cluster_addr': '172.21.15.22:6819/83352', 'heartbeat_back_addr': '172.21.15.22:6823/83352', 'heartbeat_front_addr': '172.21.15.22:6821/83352', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.001178+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.184181+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.274854+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.186217+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.247442+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.220075+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [0, 3]}, {'pgid': '2.0', 'osds': [4, 1]}, {'pgid': '2.2', 'osds': [1, 5]}, {'pgid': '2.4', 'osds': [5, 0]}], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-07-02T14:03:38.945 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2023-07-02T14:03:38.946 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:38.833+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:38.946 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:38.946+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:39.047 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:39.047+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:39.079 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.5116.0:24 2.7 2:f1a24948:::rbd_header.125de1449605:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected e81)", 2023-07-02T14:03:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.079361+0000", 2023-07-02T14:03:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.00021504400000000001, 2023-07-02T14:03:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000240566, 2023-07-02T14:03:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "reached pg", 2023-07-02T14:03:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.5116", 2023-07-02T14:03:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/4292539177", 2023-07-02T14:03:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 24 2023-07-02T14:03:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.079361+0000", 2023-07-02T14:03:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.079361+0000", 2023-07-02T14:03:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.079360+0000", 2023-07-02T14:03:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0500000000000003e-06 2023-07-02T14:03:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.079367+0000", 2023-07-02T14:03:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0608000000000001e-05 2023-07-02T14:03:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.079377+0000", 2023-07-02T14:03:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1890000000000001e-06 2023-07-02T14:03:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.079385+0000", 2023-07-02T14:03:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014775000000000001 2023-07-02T14:03:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.079532+0000", 2023-07-02T14:03:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9723999999999996e-05 2023-07-02T14:03:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.079632+0000", 2023-07-02T14:03:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00027108200000000002 2023-07-02T14:03:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2023-07-02T14:03:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 1 2023-07-02T14:03:39.087 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:39.092 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:03:39.093 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:03:39.098 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 4 2023-07-02T14:03:39.148 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:39.148+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:39.249 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:39.248+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:39.263 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:39.264 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:39.264 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:39.264 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:39.264 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:39.277 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:03:39.350 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:39.350+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:39.451 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:39.450+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:39.459 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:39.459 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:39.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:39.459 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:39.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.460 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:322 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:39.460 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:39.460 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.156365100000002, 2023-07-02T14:03:39.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.03676041, 2023-07-02T14:03:39.460 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.460 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:39.460 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.461 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:39.461 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:39.461 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 322 2023-07-02T14:03:39.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.461 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:39.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:39.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:39.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298819+0000", 2023-07-02T14:03:39.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8068e-05 2023-07-02T14:03:39.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298877+0000", 2023-07-02T14:03:39.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3339999999999998e-06 2023-07-02T14:03:39.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298883+0000", 2023-07-02T14:03:39.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1030000000000004e-06 2023-07-02T14:03:39.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298887+0000", 2023-07-02T14:03:39.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023640230000000002 2023-07-02T14:03:39.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322528+0000", 2023-07-02T14:03:39.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8424999999999994e-05 2023-07-02T14:03:39.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322616+0000", 2023-07-02T14:03:39.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116815 2023-07-02T14:03:39.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:39.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322733+0000", 2023-07-02T14:03:39.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048252629999999998 2023-07-02T14:03:39.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:39.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327558+0000", 2023-07-02T14:03:39.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078965240000000003 2023-07-02T14:03:39.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:39.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335455+0000", 2023-07-02T14:03:39.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0511999999999999e-05 2023-07-02T14:03:39.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:39.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335515+0000", 2023-07-02T14:03:39.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4789000000000006e-05 2023-07-02T14:03:39.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:39.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335580+0000", 2023-07-02T14:03:39.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.03676041 2023-07-02T14:03:39.470 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.470 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.470 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.471 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:323 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:39.471 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:39.471 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.156293720000001, 2023-07-02T14:03:39.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036875044000000003, 2023-07-02T14:03:39.471 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.471 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:39.471 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.471 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:39.472 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:39.472 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 323 2023-07-02T14:03:39.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.472 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:39.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:39.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:39.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298890+0000", 2023-07-02T14:03:39.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7277000000000002e-05 2023-07-02T14:03:39.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298938+0000", 2023-07-02T14:03:39.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0495e-05 2023-07-02T14:03:39.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298948+0000", 2023-07-02T14:03:39.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6829999999999999e-06 2023-07-02T14:03:39.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298953+0000", 2023-07-02T14:03:39.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024271435000000001 2023-07-02T14:03:39.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323224+0000", 2023-07-02T14:03:39.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9266999999999998e-05 2023-07-02T14:03:39.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323304+0000", 2023-07-02T14:03:39.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000157021 2023-07-02T14:03:39.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:39.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323461+0000", 2023-07-02T14:03:39.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041036689999999999 2023-07-02T14:03:39.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:39.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327564+0000", 2023-07-02T14:03:39.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080753490000000008 2023-07-02T14:03:39.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:39.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335640+0000", 2023-07-02T14:03:39.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7398999999999994e-05 2023-07-02T14:03:39.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:39.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335727+0000", 2023-07-02T14:03:39.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8967000000000001e-05 2023-07-02T14:03:39.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:39.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335766+0000", 2023-07-02T14:03:39.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036875044000000003 2023-07-02T14:03:39.480 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.481 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.481 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.481 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:324 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:39.481 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:39.481 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.156226341999997, 2023-07-02T14:03:39.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036935673000000002, 2023-07-02T14:03:39.482 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.482 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:39.482 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.482 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:39.482 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:39.482 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 324 2023-07-02T14:03:39.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.482 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:39.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:39.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:39.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:39.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5519999999999997e-05 2023-07-02T14:03:39.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299013+0000", 2023-07-02T14:03:39.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0701e-05 2023-07-02T14:03:39.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299024+0000", 2023-07-02T14:03:39.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2379999999999999e-06 2023-07-02T14:03:39.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299028+0000", 2023-07-02T14:03:39.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024972768999999999 2023-07-02T14:03:39.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324001+0000", 2023-07-02T14:03:39.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1178000000000002e-05 2023-07-02T14:03:39.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324072+0000", 2023-07-02T14:03:39.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000166119 2023-07-02T14:03:39.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:39.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324238+0000", 2023-07-02T14:03:39.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033314210000000002 2023-07-02T14:03:39.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:39.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327570+0000", 2023-07-02T14:03:39.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0082393369999999994 2023-07-02T14:03:39.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:39.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335809+0000", 2023-07-02T14:03:39.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7071999999999999e-05 2023-07-02T14:03:39.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:39.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335856+0000", 2023-07-02T14:03:39.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7926999999999998e-05 2023-07-02T14:03:39.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:39.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335894+0000", 2023-07-02T14:03:39.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036935673000000002 2023-07-02T14:03:39.491 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.491 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.491 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.492 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:325 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:39.492 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:39.492 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.156151014999999, 2023-07-02T14:03:39.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036990838999999998, 2023-07-02T14:03:39.493 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.493 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:39.493 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.494 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:39.494 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:39.494 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 325 2023-07-02T14:03:39.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.494 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:39.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:39.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:39.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299033+0000", 2023-07-02T14:03:39.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107619 2023-07-02T14:03:39.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299141+0000", 2023-07-02T14:03:39.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3497e-05 2023-07-02T14:03:39.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299154+0000", 2023-07-02T14:03:39.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4669e-05 2023-07-02T14:03:39.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299169+0000", 2023-07-02T14:03:39.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025630159999999999 2023-07-02T14:03:39.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324799+0000", 2023-07-02T14:03:39.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6953000000000007e-05 2023-07-02T14:03:39.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324876+0000", 2023-07-02T14:03:39.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016346199999999999 2023-07-02T14:03:39.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:39.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325039+0000", 2023-07-02T14:03:39.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003566624 2023-07-02T14:03:39.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:39.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328606+0000", 2023-07-02T14:03:39.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073295540000000003 2023-07-02T14:03:39.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:39.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335936+0000", 2023-07-02T14:03:39.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0044999999999999e-05 2023-07-02T14:03:39.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:39.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335986+0000", 2023-07-02T14:03:39.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8872999999999999e-05 2023-07-02T14:03:39.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:39.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336024+0000", 2023-07-02T14:03:39.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036990838999999998 2023-07-02T14:03:39.506 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.507 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.507 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.507 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:326 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:39.507 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:39.507 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.156009087000001, 2023-07-02T14:03:39.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036975239999999999, 2023-07-02T14:03:39.508 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.508 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:39.508 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.508 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:39.508 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:39.508 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 326 2023-07-02T14:03:39.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.509 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:39.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:39.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:39.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299175+0000", 2023-07-02T14:03:39.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4782000000000003e-05 2023-07-02T14:03:39.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299250+0000", 2023-07-02T14:03:39.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.277e-05 2023-07-02T14:03:39.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299262+0000", 2023-07-02T14:03:39.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1460000000000001e-06 2023-07-02T14:03:39.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299268+0000", 2023-07-02T14:03:39.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026158098000000001 2023-07-02T14:03:39.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325427+0000", 2023-07-02T14:03:39.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4845999999999995e-05 2023-07-02T14:03:39.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325501+0000", 2023-07-02T14:03:39.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015890699999999999 2023-07-02T14:03:39.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:39.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325660+0000", 2023-07-02T14:03:39.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002952114 2023-07-02T14:03:39.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:39.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328612+0000", 2023-07-02T14:03:39.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074491949999999996 2023-07-02T14:03:39.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:39.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336062+0000", 2023-07-02T14:03:39.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7376999999999997e-05 2023-07-02T14:03:39.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:39.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336109+0000", 2023-07-02T14:03:39.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1814000000000003e-05 2023-07-02T14:03:39.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:39.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336151+0000", 2023-07-02T14:03:39.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036975239999999999 2023-07-02T14:03:39.517 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.517 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.517 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.518 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:327 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 589824~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:39.518 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:39.518 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.155909825999998, 2023-07-02T14:03:39.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037234253000000002, 2023-07-02T14:03:39.518 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.518 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:39.518 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.518 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:39.519 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:39.519 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 327 2023-07-02T14:03:39.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.519 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:39.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:39.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:39.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299274+0000", 2023-07-02T14:03:39.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2213999999999997e-05 2023-07-02T14:03:39.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299346+0000", 2023-07-02T14:03:39.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1816e-05 2023-07-02T14:03:39.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299358+0000", 2023-07-02T14:03:39.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2626000000000001e-05 2023-07-02T14:03:39.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299371+0000", 2023-07-02T14:03:39.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026916853000000001 2023-07-02T14:03:39.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326288+0000", 2023-07-02T14:03:39.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6909000000000004e-05 2023-07-02T14:03:39.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326375+0000", 2023-07-02T14:03:39.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015430399999999999 2023-07-02T14:03:39.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:39.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326529+0000", 2023-07-02T14:03:39.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096652100000000005 2023-07-02T14:03:39.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:39.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336194+0000", 2023-07-02T14:03:39.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115299 2023-07-02T14:03:39.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:39.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336309+0000", 2023-07-02T14:03:39.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015981200000000001 2023-07-02T14:03:39.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:39.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336469+0000", 2023-07-02T14:03:39.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9867999999999998e-05 2023-07-02T14:03:39.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:39.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336509+0000", 2023-07-02T14:03:39.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037234253000000002 2023-07-02T14:03:39.528 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.528 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.529 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.529 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:328 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:39.530 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:39.530 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.155806237, 2023-07-02T14:03:39.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037216610999999997, 2023-07-02T14:03:39.531 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.531 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:39.531 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.532 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:39.532 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:39.532 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 328 2023-07-02T14:03:39.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.533 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:39.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:39.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:39.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:39.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1412e-05 2023-07-02T14:03:39.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299449+0000", 2023-07-02T14:03:39.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0219000000000001e-05 2023-07-02T14:03:39.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299469+0000", 2023-07-02T14:03:39.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9480000000000003e-06 2023-07-02T14:03:39.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299479+0000", 2023-07-02T14:03:39.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027444344999999998 2023-07-02T14:03:39.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326924+0000", 2023-07-02T14:03:39.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5165000000000001e-05 2023-07-02T14:03:39.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327019+0000", 2023-07-02T14:03:39.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019113499999999999 2023-07-02T14:03:39.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:39.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327210+0000", 2023-07-02T14:03:39.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090197449999999992 2023-07-02T14:03:39.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:39.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336230+0000", 2023-07-02T14:03:39.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028764399999999999 2023-07-02T14:03:39.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:39.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336517+0000", 2023-07-02T14:03:39.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1981999999999999e-05 2023-07-02T14:03:39.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:39.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336559+0000", 2023-07-02T14:03:39.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:03:39.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:39.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336595+0000", 2023-07-02T14:03:39.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037216610999999997 2023-07-02T14:03:39.544 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.544 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.544 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.544 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:329 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:39.544 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:39.545 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.155696319999997, 2023-07-02T14:03:39.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037191142000000003, 2023-07-02T14:03:39.545 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.545 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:39.545 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.545 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:39.545 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:39.546 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 329 2023-07-02T14:03:39.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.546 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:39.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:39.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:39.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299487+0000", 2023-07-02T14:03:39.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7871000000000001e-05 2023-07-02T14:03:39.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299555+0000", 2023-07-02T14:03:39.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0446e-05 2023-07-02T14:03:39.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299566+0000", 2023-07-02T14:03:39.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7939999999999999e-06 2023-07-02T14:03:39.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299572+0000", 2023-07-02T14:03:39.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028034019 2023-07-02T14:03:39.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327606+0000", 2023-07-02T14:03:39.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1967999999999999e-05 2023-07-02T14:03:39.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327678+0000", 2023-07-02T14:03:39.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022216900000000001 2023-07-02T14:03:39.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:39.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327900+0000", 2023-07-02T14:03:39.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084016899999999999 2023-07-02T14:03:39.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:39.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336301+0000", 2023-07-02T14:03:39.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00030122800000000001 2023-07-02T14:03:39.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:39.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336603+0000", 2023-07-02T14:03:39.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0380000000000003e-05 2023-07-02T14:03:39.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:39.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336643+0000", 2023-07-02T14:03:39.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6455999999999998e-05 2023-07-02T14:03:39.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:39.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336679+0000", 2023-07-02T14:03:39.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037191142000000003 2023-07-02T14:03:39.554 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.554 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.554 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.555 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:330 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:39.555 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:39.555 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.155605549999997, 2023-07-02T14:03:39.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037849305, 2023-07-02T14:03:39.555 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.555 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:39.556 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.556 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:39.556 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:39.556 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 330 2023-07-02T14:03:39.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.556 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:39.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:39.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:39.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299578+0000", 2023-07-02T14:03:39.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4708000000000005e-05 2023-07-02T14:03:39.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299643+0000", 2023-07-02T14:03:39.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7309999999999999e-06 2023-07-02T14:03:39.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299652+0000", 2023-07-02T14:03:39.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7989999999999999e-06 2023-07-02T14:03:39.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299658+0000", 2023-07-02T14:03:39.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028981664000000001 2023-07-02T14:03:39.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328639+0000", 2023-07-02T14:03:39.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0939000000000006e-05 2023-07-02T14:03:39.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328710+0000", 2023-07-02T14:03:39.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000142624 2023-07-02T14:03:39.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:39.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328853+0000", 2023-07-02T14:03:39.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078374640000000006 2023-07-02T14:03:39.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:39.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336690+0000", 2023-07-02T14:03:39.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00068453000000000003 2023-07-02T14:03:39.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:39.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337375+0000", 2023-07-02T14:03:39.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0667999999999998e-05 2023-07-02T14:03:39.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:39.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337406+0000", 2023-07-02T14:03:39.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2858999999999999e-05 2023-07-02T14:03:39.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:39.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337428+0000", 2023-07-02T14:03:39.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037849305 2023-07-02T14:03:39.565 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.565 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.565 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.566 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:331 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:39.566 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:39.566 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.155521688, 2023-07-02T14:03:39.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037870332, 2023-07-02T14:03:39.566 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.566 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:39.566 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.567 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:39.567 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:39.567 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 331 2023-07-02T14:03:39.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.567 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:39.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:39.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:39.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299662+0000", 2023-07-02T14:03:39.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2252e-05 2023-07-02T14:03:39.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299725+0000", 2023-07-02T14:03:39.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8440000000000004e-06 2023-07-02T14:03:39.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299733+0000", 2023-07-02T14:03:39.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6759999999999998e-06 2023-07-02T14:03:39.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299738+0000", 2023-07-02T14:03:39.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029459387 2023-07-02T14:03:39.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329197+0000", 2023-07-02T14:03:39.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0087999999999997e-05 2023-07-02T14:03:39.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329268+0000", 2023-07-02T14:03:39.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014405700000000001 2023-07-02T14:03:39.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:39.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329412+0000", 2023-07-02T14:03:39.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072842100000000002 2023-07-02T14:03:39.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:39.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336696+0000", 2023-07-02T14:03:39.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00076275300000000002 2023-07-02T14:03:39.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:39.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337459+0000", 2023-07-02T14:03:39.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5988000000000002e-05 2023-07-02T14:03:39.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:39.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337495+0000", 2023-07-02T14:03:39.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8686000000000002e-05 2023-07-02T14:03:39.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:39.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337533+0000", 2023-07-02T14:03:39.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037870332 2023-07-02T14:03:39.580 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.580 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.580 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.580 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:332 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:39.581 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:39.581 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.155441764000003, 2023-07-02T14:03:39.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037892506999999999, 2023-07-02T14:03:39.581 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.581 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:39.582 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.582 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:39.582 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:39.582 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 332 2023-07-02T14:03:39.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.583 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:39.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:39.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:39.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299742+0000", 2023-07-02T14:03:39.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1366999999999997e-05 2023-07-02T14:03:39.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299804+0000", 2023-07-02T14:03:39.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9319999999999996e-06 2023-07-02T14:03:39.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299812+0000", 2023-07-02T14:03:39.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4359999999999999e-06 2023-07-02T14:03:39.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299818+0000", 2023-07-02T14:03:39.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029946144000000001 2023-07-02T14:03:39.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329764+0000", 2023-07-02T14:03:39.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000164828 2023-07-02T14:03:39.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329929+0000", 2023-07-02T14:03:39.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016139400000000001 2023-07-02T14:03:39.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:39.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330090+0000", 2023-07-02T14:03:39.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066134009999999997 2023-07-02T14:03:39.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:39.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336704+0000", 2023-07-02T14:03:39.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00086955400000000003 2023-07-02T14:03:39.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:39.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337573+0000", 2023-07-02T14:03:39.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8447000000000001e-05 2023-07-02T14:03:39.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:39.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337602+0000", 2023-07-02T14:03:39.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.362e-05 2023-07-02T14:03:39.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:39.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337635+0000", 2023-07-02T14:03:39.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037892506999999999 2023-07-02T14:03:39.591 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.591 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.592 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.592 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:333 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:39.592 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:39.592 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.155361683000002, 2023-07-02T14:03:39.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037932598999999997, 2023-07-02T14:03:39.592 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.593 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:39.593 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.593 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:39.593 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:39.593 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 333 2023-07-02T14:03:39.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.593 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:39.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:39.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:39.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299822+0000", 2023-07-02T14:03:39.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0831000000000001e-05 2023-07-02T14:03:39.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299883+0000", 2023-07-02T14:03:39.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.322e-06 2023-07-02T14:03:39.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299888+0000", 2023-07-02T14:03:39.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3479999999999997e-06 2023-07-02T14:03:39.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299893+0000", 2023-07-02T14:03:39.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030831905999999999 2023-07-02T14:03:39.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330725+0000", 2023-07-02T14:03:39.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3560999999999996e-05 2023-07-02T14:03:39.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330798+0000", 2023-07-02T14:03:39.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015165999999999999 2023-07-02T14:03:39.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:39.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330950+0000", 2023-07-02T14:03:39.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057590319999999999 2023-07-02T14:03:39.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:39.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336709+0000", 2023-07-02T14:03:39.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00096987899999999999 2023-07-02T14:03:39.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:39.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337679+0000", 2023-07-02T14:03:39.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9699000000000001e-05 2023-07-02T14:03:39.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:39.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337719+0000", 2023-07-02T14:03:39.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6961000000000001e-05 2023-07-02T14:03:39.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:39.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337756+0000", 2023-07-02T14:03:39.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037932598999999997 2023-07-02T14:03:39.602 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.602 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.602 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.602 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:416 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:39.602 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:39.603 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.137068304000003, 2023-07-02T14:03:39.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039242025, 2023-07-02T14:03:39.603 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.603 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:39.603 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.603 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:39.603 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:39.603 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:03:39.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.604 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.604 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.604 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:39.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:39.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:39.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:39.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.692e-06 2023-07-02T14:03:39.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318119+0000", 2023-07-02T14:03:39.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9790000000000005e-06 2023-07-02T14:03:39.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318129+0000", 2023-07-02T14:03:39.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3640000000000001e-06 2023-07-02T14:03:39.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318137+0000", 2023-07-02T14:03:39.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022607808 2023-07-02T14:03:39.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340745+0000", 2023-07-02T14:03:39.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0130999999999996e-05 2023-07-02T14:03:39.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340825+0000", 2023-07-02T14:03:39.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039581080000000001 2023-07-02T14:03:39.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344783+0000", 2023-07-02T14:03:39.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000006e-05 2023-07-02T14:03:39.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344854+0000", 2023-07-02T14:03:39.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052105550000000004 2023-07-02T14:03:39.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350065+0000", 2023-07-02T14:03:39.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0321999999999997e-05 2023-07-02T14:03:39.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350115+0000", 2023-07-02T14:03:39.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012378409999999999 2023-07-02T14:03:39.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351353+0000", 2023-07-02T14:03:39.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5942000000000001e-05 2023-07-02T14:03:39.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351409+0000", 2023-07-02T14:03:39.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035677700000000002 2023-07-02T14:03:39.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:39.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351766+0000", 2023-07-02T14:03:39.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000926719 2023-07-02T14:03:39.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:39.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352693+0000", 2023-07-02T14:03:39.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045652080000000003 2023-07-02T14:03:39.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:39.615 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357258+0000", 2023-07-02T14:03:39.615 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4717e-05 2023-07-02T14:03:39.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.616 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:39.616 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357293+0000", 2023-07-02T14:03:39.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5807000000000002e-05 2023-07-02T14:03:39.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.616 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:39.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357358+0000", 2023-07-02T14:03:39.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039242025 2023-07-02T14:03:39.617 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.617 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.617 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.618 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:417 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:39.618 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:39.618 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.137042866999998, 2023-07-02T14:03:39.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042407105, 2023-07-02T14:03:39.618 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.618 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:39.618 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.618 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:39.619 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:39.619 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 417 2023-07-02T14:03:39.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.619 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:39.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:39.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:39.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318141+0000", 2023-07-02T14:03:39.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7120000000000002e-06 2023-07-02T14:03:39.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318145+0000", 2023-07-02T14:03:39.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8869999999999997e-06 2023-07-02T14:03:39.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318151+0000", 2023-07-02T14:03:39.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9820000000000002e-06 2023-07-02T14:03:39.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318155+0000", 2023-07-02T14:03:39.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699133 2023-07-02T14:03:39.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340854+0000", 2023-07-02T14:03:39.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6849999999999998e-05 2023-07-02T14:03:39.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340931+0000", 2023-07-02T14:03:39.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039672140000000002 2023-07-02T14:03:39.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344898+0000", 2023-07-02T14:03:39.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5341999999999998e-05 2023-07-02T14:03:39.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.625 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344973+0000", 2023-07-02T14:03:39.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051819400000000003 2023-07-02T14:03:39.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.626 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350155+0000", 2023-07-02T14:03:39.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3261999999999999e-05 2023-07-02T14:03:39.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.626 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350209+0000", 2023-07-02T14:03:39.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00194097 2023-07-02T14:03:39.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352150+0000", 2023-07-02T14:03:39.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6851e-05 2023-07-02T14:03:39.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352226+0000", 2023-07-02T14:03:39.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051831109999999998 2023-07-02T14:03:39.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357409+0000", 2023-07-02T14:03:39.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5325e-05 2023-07-02T14:03:39.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357495+0000", 2023-07-02T14:03:39.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042107899999999998 2023-07-02T14:03:39.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:39.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357916+0000", 2023-07-02T14:03:39.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0011672449999999999 2023-07-02T14:03:39.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:39.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.359083+0000", 2023-07-02T14:03:39.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013534930000000001 2023-07-02T14:03:39.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:39.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360437+0000", 2023-07-02T14:03:39.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2159000000000002e-05 2023-07-02T14:03:39.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:39.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360469+0000", 2023-07-02T14:03:39.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0092000000000005e-05 2023-07-02T14:03:39.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:39.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360549+0000", 2023-07-02T14:03:39.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042407105 2023-07-02T14:03:39.633 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.633 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.633 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.633 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:418 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:39.633 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:39.633 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.137023626000001, 2023-07-02T14:03:39.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.045613663999999998, 2023-07-02T14:03:39.634 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.634 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:39.634 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.634 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:39.634 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:39.634 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 418 2023-07-02T14:03:39.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.635 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:39.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:39.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:39.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318160+0000", 2023-07-02T14:03:39.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6490000000000001e-06 2023-07-02T14:03:39.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318164+0000", 2023-07-02T14:03:39.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:03:39.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318170+0000", 2023-07-02T14:03:39.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5279999999999999e-06 2023-07-02T14:03:39.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318173+0000", 2023-07-02T14:03:39.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022788856 2023-07-02T14:03:39.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340962+0000", 2023-07-02T14:03:39.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7730000000000003e-05 2023-07-02T14:03:39.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341040+0000", 2023-07-02T14:03:39.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039795899999999999 2023-07-02T14:03:39.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345020+0000", 2023-07-02T14:03:39.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4962000000000006e-05 2023-07-02T14:03:39.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345085+0000", 2023-07-02T14:03:39.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051955430000000004 2023-07-02T14:03:39.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350280+0000", 2023-07-02T14:03:39.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8913999999999995e-05 2023-07-02T14:03:39.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350379+0000", 2023-07-02T14:03:39.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001887344 2023-07-02T14:03:39.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352266+0000", 2023-07-02T14:03:39.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7865000000000003e-05 2023-07-02T14:03:39.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352314+0000", 2023-07-02T14:03:39.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060621190000000004 2023-07-02T14:03:39.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358376+0000", 2023-07-02T14:03:39.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2448999999999997e-05 2023-07-02T14:03:39.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358449+0000", 2023-07-02T14:03:39.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0021539929999999999 2023-07-02T14:03:39.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360603+0000", 2023-07-02T14:03:39.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7859000000000002e-05 2023-07-02T14:03:39.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360671+0000", 2023-07-02T14:03:39.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026930699999999999 2023-07-02T14:03:39.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:39.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360940+0000", 2023-07-02T14:03:39.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00081179699999999995 2023-07-02T14:03:39.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:39.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361752+0000", 2023-07-02T14:03:39.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0019350490000000001 2023-07-02T14:03:39.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:39.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363687+0000", 2023-07-02T14:03:39.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4462000000000003e-05 2023-07-02T14:03:39.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:39.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363721+0000", 2023-07-02T14:03:39.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3408000000000001e-05 2023-07-02T14:03:39.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:39.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363775+0000", 2023-07-02T14:03:39.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.045613663999999998 2023-07-02T14:03:39.657 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.657 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.657 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.658 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:419 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:39.659 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:39.659 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.137007298999997, 2023-07-02T14:03:39.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.047296703000000002, 2023-07-02T14:03:39.659 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.660 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:39.660 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.660 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:39.660 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:39.661 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 419 2023-07-02T14:03:39.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.661 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:39.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:39.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:39.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:39.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.433e-06 2023-07-02T14:03:39.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318180+0000", 2023-07-02T14:03:39.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9950000000000002e-06 2023-07-02T14:03:39.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318186+0000", 2023-07-02T14:03:39.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7290000000000002e-06 2023-07-02T14:03:39.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318190+0000", 2023-07-02T14:03:39.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022877989000000001 2023-07-02T14:03:39.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341068+0000", 2023-07-02T14:03:39.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5803999999999994e-05 2023-07-02T14:03:39.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341154+0000", 2023-07-02T14:03:39.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039633000000000003 2023-07-02T14:03:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345117+0000", 2023-07-02T14:03:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7188999999999996e-05 2023-07-02T14:03:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345194+0000", 2023-07-02T14:03:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052306059999999996 2023-07-02T14:03:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350425+0000", 2023-07-02T14:03:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8093999999999997e-05 2023-07-02T14:03:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350473+0000", 2023-07-02T14:03:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001865158 2023-07-02T14:03:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352338+0000", 2023-07-02T14:03:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5479000000000003e-05 2023-07-02T14:03:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352384+0000", 2023-07-02T14:03:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061153290000000001 2023-07-02T14:03:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358499+0000", 2023-07-02T14:03:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5156999999999999e-05 2023-07-02T14:03:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358574+0000", 2023-07-02T14:03:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027847459999999998 2023-07-02T14:03:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361359+0000", 2023-07-02T14:03:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2633999999999999e-05 2023-07-02T14:03:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361402+0000", 2023-07-02T14:03:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024171510000000002 2023-07-02T14:03:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363819+0000", 2023-07-02T14:03:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6988999999999999e-05 2023-07-02T14:03:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363876+0000", 2023-07-02T14:03:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035321300000000001 2023-07-02T14:03:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.364229+0000", 2023-07-02T14:03:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00094674799999999997 2023-07-02T14:03:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365176+0000", 2023-07-02T14:03:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019724200000000001 2023-07-02T14:03:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365373+0000", 2023-07-02T14:03:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2116e-05 2023-07-02T14:03:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365425+0000", 2023-07-02T14:03:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8996999999999999e-05 2023-07-02T14:03:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365474+0000", 2023-07-02T14:03:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.047296703000000002 2023-07-02T14:03:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:472 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.077551067000002, 2023-07-02T14:03:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036204721000000002, 2023-07-02T14:03:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 472 2023-07-02T14:03:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:03:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377633+0000", 2023-07-02T14:03:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.815e-06 2023-07-02T14:03:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377635+0000", 2023-07-02T14:03:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9459999999999998e-06 2023-07-02T14:03:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377638+0000", 2023-07-02T14:03:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.673e-06 2023-07-02T14:03:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377643+0000", 2023-07-02T14:03:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043004239999999997 2023-07-02T14:03:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.381943+0000", 2023-07-02T14:03:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4321999999999999e-05 2023-07-02T14:03:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.381998+0000", 2023-07-02T14:03:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003008377 2023-07-02T14:03:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385006+0000", 2023-07-02T14:03:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8751999999999998e-05 2023-07-02T14:03:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385045+0000", 2023-07-02T14:03:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086260880000000005 2023-07-02T14:03:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393671+0000", 2023-07-02T14:03:39.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6365999999999997e-05 2023-07-02T14:03:39.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393707+0000", 2023-07-02T14:03:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028560289999999999 2023-07-02T14:03:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396563+0000", 2023-07-02T14:03:39.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6195e-05 2023-07-02T14:03:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396619+0000", 2023-07-02T14:03:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023118269999999998 2023-07-02T14:03:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.398931+0000", 2023-07-02T14:03:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3885999999999999e-05 2023-07-02T14:03:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.398985+0000", 2023-07-02T14:03:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043756890000000003 2023-07-02T14:03:39.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403361+0000", 2023-07-02T14:03:39.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0043000000000001e-05 2023-07-02T14:03:39.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403421+0000", 2023-07-02T14:03:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001590972 2023-07-02T14:03:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405012+0000", 2023-07-02T14:03:39.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0067000000000005e-05 2023-07-02T14:03:39.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405082+0000", 2023-07-02T14:03:39.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050958610000000001 2023-07-02T14:03:39.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410178+0000", 2023-07-02T14:03:39.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7070000000000004e-05 2023-07-02T14:03:39.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410275+0000", 2023-07-02T14:03:39.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060121100000000004 2023-07-02T14:03:39.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410876+0000", 2023-07-02T14:03:39.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3177999999999999e-05 2023-07-02T14:03:39.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410919+0000", 2023-07-02T14:03:39.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000271394 2023-07-02T14:03:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411191+0000", 2023-07-02T14:03:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001009598 2023-07-02T14:03:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.412200+0000", 2023-07-02T14:03:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014789790000000001 2023-07-02T14:03:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413679+0000", 2023-07-02T14:03:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2624000000000002e-05 2023-07-02T14:03:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413742+0000", 2023-07-02T14:03:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6564999999999994e-05 2023-07-02T14:03:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413838+0000", 2023-07-02T14:03:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036204721000000002 2023-07-02T14:03:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:473 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.077539655999999, 2023-07-02T14:03:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037915904, 2023-07-02T14:03:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 473 2023-07-02T14:03:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:03:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.119e-06 2023-07-02T14:03:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377647+0000", 2023-07-02T14:03:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.875e-06 2023-07-02T14:03:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377650+0000", 2023-07-02T14:03:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1949999999999999e-06 2023-07-02T14:03:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377653+0000", 2023-07-02T14:03:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004375444 2023-07-02T14:03:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382028+0000", 2023-07-02T14:03:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1063999999999997e-05 2023-07-02T14:03:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382089+0000", 2023-07-02T14:03:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030043869999999999 2023-07-02T14:03:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385094+0000", 2023-07-02T14:03:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8167999999999997e-05 2023-07-02T14:03:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385142+0000", 2023-07-02T14:03:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008603421 2023-07-02T14:03:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393745+0000", 2023-07-02T14:03:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3763000000000005e-05 2023-07-02T14:03:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393819+0000", 2023-07-02T14:03:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028274720000000001 2023-07-02T14:03:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396647+0000", 2023-07-02T14:03:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7929999999999997e-05 2023-07-02T14:03:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396695+0000", 2023-07-02T14:03:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023426459999999999 2023-07-02T14:03:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399037+0000", 2023-07-02T14:03:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5846000000000004e-05 2023-07-02T14:03:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399123+0000", 2023-07-02T14:03:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043448360000000004 2023-07-02T14:03:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403468+0000", 2023-07-02T14:03:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6699000000000001e-05 2023-07-02T14:03:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403515+0000", 2023-07-02T14:03:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001622943 2023-07-02T14:03:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405138+0000", 2023-07-02T14:03:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6382000000000002e-05 2023-07-02T14:03:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405204+0000", 2023-07-02T14:03:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051089910000000002 2023-07-02T14:03:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410313+0000", 2023-07-02T14:03:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8106000000000003e-05 2023-07-02T14:03:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410361+0000", 2023-07-02T14:03:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012200399999999999 2023-07-02T14:03:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411581+0000", 2023-07-02T14:03:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5067000000000005e-05 2023-07-02T14:03:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411656+0000", 2023-07-02T14:03:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022353249999999998 2023-07-02T14:03:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413891+0000", 2023-07-02T14:03:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9356000000000003e-05 2023-07-02T14:03:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413981+0000", 2023-07-02T14:03:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00043818299999999998 2023-07-02T14:03:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.414419+0000", 2023-07-02T14:03:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00081559199999999997 2023-07-02T14:03:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415235+0000", 2023-07-02T14:03:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026434499999999998 2023-07-02T14:03:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415499+0000", 2023-07-02T14:03:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9779999999999999e-05 2023-07-02T14:03:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415529+0000", 2023-07-02T14:03:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2159999999999997e-05 2023-07-02T14:03:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415561+0000", 2023-07-02T14:03:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037915904 2023-07-02T14:03:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:475 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.077529617000003, 2023-07-02T14:03:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039286962000000002, 2023-07-02T14:03:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 475 2023-07-02T14:03:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4779999999999997e-06 2023-07-02T14:03:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377664+0000", 2023-07-02T14:03:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.268e-06 2023-07-02T14:03:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377671+0000", 2023-07-02T14:03:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.861e-06 2023-07-02T14:03:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377674+0000", 2023-07-02T14:03:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044395789999999999 2023-07-02T14:03:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382114+0000", 2023-07-02T14:03:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8096999999999997e-05 2023-07-02T14:03:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382162+0000", 2023-07-02T14:03:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029949059999999999 2023-07-02T14:03:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385157+0000", 2023-07-02T14:03:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1158000000000001e-05 2023-07-02T14:03:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385198+0000", 2023-07-02T14:03:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086477229999999995 2023-07-02T14:03:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393846+0000", 2023-07-02T14:03:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5990999999999999e-05 2023-07-02T14:03:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393912+0000", 2023-07-02T14:03:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027979599999999999 2023-07-02T14:03:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396710+0000", 2023-07-02T14:03:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5911000000000002e-05 2023-07-02T14:03:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396746+0000", 2023-07-02T14:03:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024039759999999999 2023-07-02T14:03:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399150+0000", 2023-07-02T14:03:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7834999999999999e-05 2023-07-02T14:03:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399188+0000", 2023-07-02T14:03:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004364054 2023-07-02T14:03:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403552+0000", 2023-07-02T14:03:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6274999999999999e-05 2023-07-02T14:03:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403608+0000", 2023-07-02T14:03:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0016261540000000001 2023-07-02T14:03:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405234+0000", 2023-07-02T14:03:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6945000000000004e-05 2023-07-02T14:03:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405301+0000", 2023-07-02T14:03:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050871329999999998 2023-07-02T14:03:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410388+0000", 2023-07-02T14:03:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0210999999999995e-05 2023-07-02T14:03:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410468+0000", 2023-07-02T14:03:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001227454 2023-07-02T14:03:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411696+0000", 2023-07-02T14:03:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8188000000000005e-05 2023-07-02T14:03:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411764+0000", 2023-07-02T14:03:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031779930000000001 2023-07-02T14:03:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.414942+0000", 2023-07-02T14:03:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7236000000000004e-05 2023-07-02T14:03:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415019+0000", 2023-07-02T14:03:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000573045 2023-07-02T14:03:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415592+0000", 2023-07-02T14:03:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7580000000000002e-05 2023-07-02T14:03:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415640+0000", 2023-07-02T14:03:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024783200000000002 2023-07-02T14:03:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415888+0000", 2023-07-02T14:03:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00080440999999999998 2023-07-02T14:03:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416692+0000", 2023-07-02T14:03:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000195787 2023-07-02T14:03:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416888+0000", 2023-07-02T14:03:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0787999999999997e-05 2023-07-02T14:03:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416919+0000", 2023-07-02T14:03:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.338e-05 2023-07-02T14:03:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416942+0000", 2023-07-02T14:03:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039286962000000002 2023-07-02T14:03:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:5269 2.5 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:head [write 446464~1847296] snapc e=[e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e81)", 2023-07-02T14:03:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.147365+0000", 2023-07-02T14:03:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.30781948199999998, 2023-07-02T14:03:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036564266999999998, 2023-07-02T14:03:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 5269 2023-07-02T14:03:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.147365+0000", 2023-07-02T14:03:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.147365+0000", 2023-07-02T14:03:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.147365+0000", 2023-07-02T14:03:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001616302 2023-07-02T14:03:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.148981+0000", 2023-07-02T14:03:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1630000000000002e-06 2023-07-02T14:03:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.148990+0000", 2023-07-02T14:03:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4989999999999999e-06 2023-07-02T14:03:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.148997+0000", 2023-07-02T14:03:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012255096 2023-07-02T14:03:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.161252+0000", 2023-07-02T14:03:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3946e-05 2023-07-02T14:03:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.161326+0000", 2023-07-02T14:03:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022859399999999999 2023-07-02T14:03:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.161554+0000", 2023-07-02T14:03:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062552529999999997 2023-07-02T14:03:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.167810+0000", 2023-07-02T14:03:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016036499999999999 2023-07-02T14:03:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.183846+0000", 2023-07-02T14:03:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7889000000000003e-05 2023-07-02T14:03:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.183884+0000", 2023-07-02T14:03:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5410000000000001e-05 2023-07-02T14:03:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.183929+0000", 2023-07-02T14:03:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036564266999999998 2023-07-02T14:03:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.769 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:39.771 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:03:39.771 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:39.770+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:39.871 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:39.871+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:39.971 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:39.971+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:39.974 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6171 2.0 e81/46)", 2023-07-02T14:03:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.971963+0000", 2023-07-02T14:03:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.00054330100000000005, 2023-07-02T14:03:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00055760300000000005, 2023-07-02T14:03:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "queued for pg", 2023-07-02T14:03:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.971963+0000", 2023-07-02T14:03:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.971963+0000", 2023-07-02T14:03:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.971962+0000", 2023-07-02T14:03:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00030258500000000002 2023-07-02T14:03:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972265+0000", 2023-07-02T14:03:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1640000000000001e-06 2023-07-02T14:03:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972273+0000", 2023-07-02T14:03:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4780000000000002e-06 2023-07-02T14:03:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972278+0000", 2023-07-02T14:03:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00031459599999999999 2023-07-02T14:03:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:03:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:03:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.027712541, 2023-07-02T14:03:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027763122000000001, 2023-07-02T14:03:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:03:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:03:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:03:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:03:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:03:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:03:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:03:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:03:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:03:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:03:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:03:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:03:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:03:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052164810000000002 2023-07-02T14:03:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:03:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:03:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.023055183, 2023-07-02T14:03:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023143174999999998, 2023-07-02T14:03:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:03:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:03:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:03:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:03:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:03:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:03:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:03:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:03:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:03:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:03:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:03:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:03:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:03:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058276309999999998 2023-07-02T14:03:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:03:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:03:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.022115320000000001, 2023-07-02T14:03:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022248931999999999, 2023-07-02T14:03:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:03:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:03:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:03:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:03:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:03:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:03:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:03:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:03:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:03:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:03:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:03:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:03:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:03:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006213952 2023-07-02T14:03:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:03:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:03:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.020717441999999999, 2023-07-02T14:03:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020899750000000002, 2023-07-02T14:03:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:03:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:03:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:03:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:03:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:03:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:03:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:03:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:03:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:03:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:03:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:03:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:03:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:03:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061581830000000002 2023-07-02T14:03:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6148 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1344, mlcod=81'1335)", 2023-07-02T14:03:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.956703+0000", 2023-07-02T14:03:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.015803149999999998, 2023-07-02T14:03:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016024096000000002, 2023-07-02T14:03:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956703+0000", 2023-07-02T14:03:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956703+0000", 2023-07-02T14:03:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956702+0000", 2023-07-02T14:03:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012505 2023-07-02T14:03:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956827+0000", 2023-07-02T14:03:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7309999999999999e-05 2023-07-02T14:03:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956845+0000", 2023-07-02T14:03:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7570000000000001e-06 2023-07-02T14:03:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956850+0000", 2023-07-02T14:03:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071775149999999998 2023-07-02T14:03:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964028+0000", 2023-07-02T14:03:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4773000000000002e-05 2023-07-02T14:03:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964053+0000", 2023-07-02T14:03:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073495749999999997 2023-07-02T14:03:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6150 2.0 e81/46 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 81'1345, mlcod=81'1338)", 2023-07-02T14:03:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:03:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.012694495, 2023-07-02T14:03:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012951562999999999, 2023-07-02T14:03:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:03:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:03:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959810+0000", 2023-07-02T14:03:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048438650000000001 2023-07-02T14:03:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964653+0000", 2023-07-02T14:03:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2136e-05 2023-07-02T14:03:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964666+0000", 2023-07-02T14:03:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2169999999999999e-06 2023-07-02T14:03:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964672+0000", 2023-07-02T14:03:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00047561800000000003 2023-07-02T14:03:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965147+0000", 2023-07-02T14:03:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5891e-05 2023-07-02T14:03:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965173+0000", 2023-07-02T14:03:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053615820000000002 2023-07-02T14:03:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6151 2.0 e81/46 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 81'1346, mlcod=81'1338)", 2023-07-02T14:03:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.964682+0000", 2023-07-02T14:03:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0078246940000000001, 2023-07-02T14:03:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081194560000000006, 2023-07-02T14:03:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964682+0000", 2023-07-02T14:03:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964682+0000", 2023-07-02T14:03:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964681+0000", 2023-07-02T14:03:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5908000000000001e-05 2023-07-02T14:03:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964697+0000", 2023-07-02T14:03:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1920000000000005e-06 2023-07-02T14:03:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964705+0000", 2023-07-02T14:03:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.493e-06 2023-07-02T14:03:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964709+0000", 2023-07-02T14:03:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075784019999999997 2023-07-02T14:03:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972288+0000", 2023-07-02T14:03:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3762e-05 2023-07-02T14:03:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972302+0000", 2023-07-02T14:03:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076200479999999999 2023-07-02T14:03:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6153 2.0 e81/46)", 2023-07-02T14:03:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.964714+0000", 2023-07-02T14:03:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0077918830000000003, 2023-07-02T14:03:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081381590000000007, 2023-07-02T14:03:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "queued for pg", 2023-07-02T14:03:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964714+0000", 2023-07-02T14:03:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964714+0000", 2023-07-02T14:03:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964714+0000", 2023-07-02T14:03:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2535999999999999e-05 2023-07-02T14:03:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964737+0000", 2023-07-02T14:03:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.892e-06 2023-07-02T14:03:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964744+0000", 2023-07-02T14:03:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5040000000000004e-06 2023-07-02T14:03:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964749+0000", 2023-07-02T14:03:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4579999999999998e-05 2023-07-02T14:03:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6154 2.0 e81/46)", 2023-07-02T14:03:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.965245+0000", 2023-07-02T14:03:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0072610909999999999, 2023-07-02T14:03:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076664319999999999, 2023-07-02T14:03:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "queued for pg", 2023-07-02T14:03:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965245+0000", 2023-07-02T14:03:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965245+0000", 2023-07-02T14:03:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965245+0000", 2023-07-02T14:03:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7710000000000001e-06 2023-07-02T14:03:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965254+0000", 2023-07-02T14:03:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5920000000000001e-06 2023-07-02T14:03:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965263+0000", 2023-07-02T14:03:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3670000000000002e-06 2023-07-02T14:03:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965269+0000", 2023-07-02T14:03:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4079000000000001e-05 2023-07-02T14:03:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6158 2.0 e81/46)", 2023-07-02T14:03:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.965909+0000", 2023-07-02T14:03:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.006596869, 2023-07-02T14:03:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070543539999999997, 2023-07-02T14:03:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "queued for pg", 2023-07-02T14:03:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965909+0000", 2023-07-02T14:03:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965909+0000", 2023-07-02T14:03:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965909+0000", 2023-07-02T14:03:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1002999999999999e-05 2023-07-02T14:03:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965960+0000", 2023-07-02T14:03:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5530000000000004e-06 2023-07-02T14:03:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965967+0000", 2023-07-02T14:03:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.138e-06 2023-07-02T14:03:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965971+0000", 2023-07-02T14:03:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1935999999999998e-05 2023-07-02T14:03:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6161 2.0 e81/46)", 2023-07-02T14:03:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.966591+0000", 2023-07-02T14:03:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0059151869999999997, 2023-07-02T14:03:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006510531, 2023-07-02T14:03:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "queued for pg", 2023-07-02T14:03:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.966591+0000", 2023-07-02T14:03:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.966591+0000", 2023-07-02T14:03:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.966590+0000", 2023-07-02T14:03:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4656e-05 2023-07-02T14:03:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.966605+0000", 2023-07-02T14:03:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4800999999999999e-05 2023-07-02T14:03:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.966620+0000", 2023-07-02T14:03:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9399999999999999e-06 2023-07-02T14:03:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.966626+0000", 2023-07-02T14:03:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4656999999999997e-05 2023-07-02T14:03:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6162 2.0 e81/46)", 2023-07-02T14:03:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.967387+0000", 2023-07-02T14:03:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0051195499999999996, 2023-07-02T14:03:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057810270000000002, 2023-07-02T14:03:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "queued for pg", 2023-07-02T14:03:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.967387+0000", 2023-07-02T14:03:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.967387+0000", 2023-07-02T14:03:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.967386+0000", 2023-07-02T14:03:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000183673 2023-07-02T14:03:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.967570+0000", 2023-07-02T14:03:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0186999999999999e-05 2023-07-02T14:03:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.967580+0000", 2023-07-02T14:03:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0530000000000003e-06 2023-07-02T14:03:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.967586+0000", 2023-07-02T14:03:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000199192 2023-07-02T14:03:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6163 2.0 e81/46)", 2023-07-02T14:03:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.968964+0000", 2023-07-02T14:03:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0035426199999999998, 2023-07-02T14:03:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042431270000000002, 2023-07-02T14:03:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "queued for pg", 2023-07-02T14:03:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.968964+0000", 2023-07-02T14:03:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.968964+0000", 2023-07-02T14:03:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.968963+0000", 2023-07-02T14:03:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3839999999999996e-05 2023-07-02T14:03:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.969057+0000", 2023-07-02T14:03:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.944e-06 2023-07-02T14:03:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.969067+0000", 2023-07-02T14:03:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8220000000000003e-06 2023-07-02T14:03:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.969072+0000", 2023-07-02T14:03:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010880200000000001 2023-07-02T14:03:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6182 2.7 e81/63 2:ec5a4750:::rbd_data.125de1449605.000000000000009f:head v 81'845, mlcod=81'842)", 2023-07-02T14:03:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.969109+0000", 2023-07-02T14:03:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0033975640000000001, 2023-07-02T14:03:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041528570000000002, 2023-07-02T14:03:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:03:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.969109+0000", 2023-07-02T14:03:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.969109+0000", 2023-07-02T14:03:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.969108+0000", 2023-07-02T14:03:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016349499999999999 2023-07-02T14:03:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.969271+0000", 2023-07-02T14:03:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5930000000000006e-06 2023-07-02T14:03:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.969280+0000", 2023-07-02T14:03:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5510000000000001e-06 2023-07-02T14:03:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.969284+0000", 2023-07-02T14:03:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3998999999999998e-05 2023-07-02T14:03:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.969318+0000", 2023-07-02T14:03:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7422999999999999e-05 2023-07-02T14:03:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.969336+0000", 2023-07-02T14:03:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022718 2023-07-02T14:03:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6164 2.0 e81/46)", 2023-07-02T14:03:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.969879+0000", 2023-07-02T14:03:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0026274800000000002, 2023-07-02T14:03:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0034409810000000001, 2023-07-02T14:03:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "queued for pg", 2023-07-02T14:03:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.969879+0000", 2023-07-02T14:03:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.969879+0000", 2023-07-02T14:03:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.969878+0000", 2023-07-02T14:03:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0136999999999995e-05 2023-07-02T14:03:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.969958+0000", 2023-07-02T14:03:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6030999999999999e-05 2023-07-02T14:03:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.969974+0000", 2023-07-02T14:03:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1349999999999997e-06 2023-07-02T14:03:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.969980+0000", 2023-07-02T14:03:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000101489 2023-07-02T14:03:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2023-07-02T14:03:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 16 2023-07-02T14:03:40.083 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:40.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:03:40.085 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:03:40.086 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:40.086+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:40.091 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:03:40.186 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:40.186+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:40.265 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:40.265 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:40.266 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:40.266 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:40.266 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:40.281 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:03:40.325 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:40.324+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:40.348 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:40.380 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:40.388 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:40.387+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:40.414 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:40.428 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:40.428 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:40.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:40.429 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:40.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.429 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:5545 2.0 e81/46 2:159aabad:::rbd_data.125de1449605.000000000000005d:head v 81'1159, mlcod=81'1155)", 2023-07-02T14:03:40.429 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.389719+0000", 2023-07-02T14:03:40.429 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 1.037560042, 2023-07-02T14:03:40.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014352986, 2023-07-02T14:03:40.430 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.430 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:40.430 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.389719+0000", 2023-07-02T14:03:40.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.389719+0000", 2023-07-02T14:03:40.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:40.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.389718+0000", 2023-07-02T14:03:40.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037957160000000002 2023-07-02T14:03:40.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.393513+0000", 2023-07-02T14:03:40.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7331000000000001e-05 2023-07-02T14:03:40.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.393531+0000", 2023-07-02T14:03:40.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0229999999999998e-06 2023-07-02T14:03:40.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.393540+0000", 2023-07-02T14:03:40.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0124000000000002e-05 2023-07-02T14:03:40.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.393600+0000", 2023-07-02T14:03:40.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2815999999999999e-05 2023-07-02T14:03:40.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.393633+0000", 2023-07-02T14:03:40.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010399037 2023-07-02T14:03:40.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:40.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.404032+0000", 2023-07-02T14:03:40.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9761000000000001e-05 2023-07-02T14:03:40.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:40.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.404072+0000", 2023-07-02T14:03:40.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014352986 2023-07-02T14:03:40.436 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.437 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.437 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.437 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:03:40.437 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:03:40.437 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.52966441099999995, 2023-07-02T14:03:40.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:03:40.438 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.438 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:40.438 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:03:40.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:03:40.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:40.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:03:40.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:03:40.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:03:40.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:03:40.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:03:40.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:03:40.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:03:40.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:03:40.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:03:40.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:03:40.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:03:40.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:03:40.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:40.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:03:40.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:03:40.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:40.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:03:40.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:03:40.444 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.444 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.445 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.445 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:03:40.445 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:03:40.445 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.52557869300000004, 2023-07-02T14:03:40.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:03:40.445 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.446 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:40.446 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:03:40.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:03:40.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:40.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:03:40.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:03:40.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:03:40.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:03:40.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:03:40.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:03:40.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:03:40.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:03:40.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:03:40.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:03:40.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:03:40.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:03:40.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:40.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:03:40.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:03:40.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:40.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:03:40.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:03:40.452 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.452 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.452 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.453 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:03:40.453 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:03:40.453 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.52171391199999995, 2023-07-02T14:03:40.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:03:40.453 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.453 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:40.454 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:03:40.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:03:40.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:40.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:03:40.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:03:40.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:03:40.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:03:40.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:03:40.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:03:40.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:03:40.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:03:40.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:03:40.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:03:40.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:03:40.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:03:40.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:40.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:03:40.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:03:40.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:40.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:03:40.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:03:40.460 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.460 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.460 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.461 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:03:40.461 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:03:40.461 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.51818965100000003, 2023-07-02T14:03:40.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:03:40.461 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.461 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:40.461 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:03:40.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:03:40.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:40.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:03:40.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:03:40.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:03:40.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:03:40.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:03:40.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:03:40.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:03:40.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:03:40.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:03:40.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:03:40.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:03:40.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:03:40.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:40.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:03:40.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:03:40.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:40.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:03:40.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:03:40.468 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.468 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.468 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.469 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6124 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1328, mlcod=81'1319)", 2023-07-02T14:03:40.469 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:03:40.469 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.51467426800000005, 2023-07-02T14:03:40.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019096974999999999, 2023-07-02T14:03:40.469 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.469 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:40.469 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:03:40.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:03:40.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:40.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912602+0000", 2023-07-02T14:03:40.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6741999999999999e-05 2023-07-02T14:03:40.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912629+0000", 2023-07-02T14:03:40.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5163e-05 2023-07-02T14:03:40.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912644+0000", 2023-07-02T14:03:40.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4800000000000001e-06 2023-07-02T14:03:40.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912653+0000", 2023-07-02T14:03:40.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071988909999999998 2023-07-02T14:03:40.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.919851+0000", 2023-07-02T14:03:40.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2949999999999999e-05 2023-07-02T14:03:40.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.919874+0000", 2023-07-02T14:03:40.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011800804 2023-07-02T14:03:40.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:40.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931675+0000", 2023-07-02T14:03:40.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6033999999999998e-05 2023-07-02T14:03:40.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:40.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931701+0000", 2023-07-02T14:03:40.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019096974999999999 2023-07-02T14:03:40.476 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.476 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.476 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.477 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6125 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1330, mlcod=81'1319)", 2023-07-02T14:03:40.477 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.915930+0000", 2023-07-02T14:03:40.477 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.51134858299999997, 2023-07-02T14:03:40.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015810346999999999, 2023-07-02T14:03:40.477 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.477 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:40.477 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.915930+0000", 2023-07-02T14:03:40.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.915930+0000", 2023-07-02T14:03:40.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:40.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.915929+0000", 2023-07-02T14:03:40.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8398e-05 2023-07-02T14:03:40.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.915978+0000", 2023-07-02T14:03:40.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2022e-05 2023-07-02T14:03:40.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.916000+0000", 2023-07-02T14:03:40.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2780000000000001e-06 2023-07-02T14:03:40.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.916005+0000", 2023-07-02T14:03:40.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078621239999999998 2023-07-02T14:03:40.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923867+0000", 2023-07-02T14:03:40.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1858999999999999e-05 2023-07-02T14:03:40.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923889+0000", 2023-07-02T14:03:40.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078253369999999999 2023-07-02T14:03:40.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:40.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931714+0000", 2023-07-02T14:03:40.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5936999999999999e-05 2023-07-02T14:03:40.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:40.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931740+0000", 2023-07-02T14:03:40.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015810346999999999 2023-07-02T14:03:40.484 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.484 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.484 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.484 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6137 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1335, mlcod=81'1333)", 2023-07-02T14:03:40.484 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.934629+0000", 2023-07-02T14:03:40.485 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.49264944199999999, 2023-07-02T14:03:40.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014672614, 2023-07-02T14:03:40.485 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.485 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:40.485 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.934629+0000", 2023-07-02T14:03:40.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.934629+0000", 2023-07-02T14:03:40.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:40.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.934628+0000", 2023-07-02T14:03:40.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030856529999999998 2023-07-02T14:03:40.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.937714+0000", 2023-07-02T14:03:40.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6008999999999999e-05 2023-07-02T14:03:40.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.937730+0000", 2023-07-02T14:03:40.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9440000000000005e-06 2023-07-02T14:03:40.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.937738+0000", 2023-07-02T14:03:40.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4077000000000003e-05 2023-07-02T14:03:40.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.937802+0000", 2023-07-02T14:03:40.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5251e-05 2023-07-02T14:03:40.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.937817+0000", 2023-07-02T14:03:40.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011457106 2023-07-02T14:03:40.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:40.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949275+0000", 2023-07-02T14:03:40.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.7217999999999999e-05 2023-07-02T14:03:40.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:40.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949302+0000", 2023-07-02T14:03:40.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014672614 2023-07-02T14:03:40.491 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.492 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.492 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.492 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6139 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1337, mlcod=81'1333)", 2023-07-02T14:03:40.492 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:03:40.492 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.48798440799999998, 2023-07-02T14:03:40.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018571681, 2023-07-02T14:03:40.493 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.493 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:40.493 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:03:40.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:03:40.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:40.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939293+0000", 2023-07-02T14:03:40.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003764613 2023-07-02T14:03:40.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943058+0000", 2023-07-02T14:03:40.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5797000000000001e-05 2023-07-02T14:03:40.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943074+0000", 2023-07-02T14:03:40.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5089999999999996e-06 2023-07-02T14:03:40.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943081+0000", 2023-07-02T14:03:40.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014727100000000001 2023-07-02T14:03:40.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943228+0000", 2023-07-02T14:03:40.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2246000000000003e-05 2023-07-02T14:03:40.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943261+0000", 2023-07-02T14:03:40.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014576317 2023-07-02T14:03:40.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:40.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957837+0000", 2023-07-02T14:03:40.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8962000000000002e-05 2023-07-02T14:03:40.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:40.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957866+0000", 2023-07-02T14:03:40.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018571681 2023-07-02T14:03:40.499 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.499 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.500 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.500 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6140 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1338, mlcod=81'1333)", 2023-07-02T14:03:40.500 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.943097+0000", 2023-07-02T14:03:40.500 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.48418160399999999, 2023-07-02T14:03:40.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014787599, 2023-07-02T14:03:40.500 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.501 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:40.501 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943097+0000", 2023-07-02T14:03:40.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943097+0000", 2023-07-02T14:03:40.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:40.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943096+0000", 2023-07-02T14:03:40.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4756999999999999e-05 2023-07-02T14:03:40.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943111+0000", 2023-07-02T14:03:40.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9079999999999995e-06 2023-07-02T14:03:40.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943119+0000", 2023-07-02T14:03:40.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4150000000000001e-06 2023-07-02T14:03:40.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943123+0000", 2023-07-02T14:03:40.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005177999 2023-07-02T14:03:40.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.948301+0000", 2023-07-02T14:03:40.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5087000000000003e-05 2023-07-02T14:03:40.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.948356+0000", 2023-07-02T14:03:40.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0095146889999999998 2023-07-02T14:03:40.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:40.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957871+0000", 2023-07-02T14:03:40.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.347e-05 2023-07-02T14:03:40.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:40.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957885+0000", 2023-07-02T14:03:40.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014787599 2023-07-02T14:03:40.507 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.507 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.507 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.508 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6141 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1339, mlcod=81'1333)", 2023-07-02T14:03:40.508 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.943978+0000", 2023-07-02T14:03:40.508 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.48330055100000002, 2023-07-02T14:03:40.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013921674, 2023-07-02T14:03:40.508 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.508 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:40.509 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943978+0000", 2023-07-02T14:03:40.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943978+0000", 2023-07-02T14:03:40.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:40.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943977+0000", 2023-07-02T14:03:40.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0682999999999999e-05 2023-07-02T14:03:40.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943988+0000", 2023-07-02T14:03:40.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0648999999999999e-05 2023-07-02T14:03:40.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943999+0000", 2023-07-02T14:03:40.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3510000000000001e-06 2023-07-02T14:03:40.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944004+0000", 2023-07-02T14:03:40.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053320290000000003 2023-07-02T14:03:40.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949336+0000", 2023-07-02T14:03:40.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5068e-05 2023-07-02T14:03:40.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949351+0000", 2023-07-02T14:03:40.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0085357869999999995 2023-07-02T14:03:40.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:40.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957887+0000", 2023-07-02T14:03:40.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2781e-05 2023-07-02T14:03:40.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:40.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957900+0000", 2023-07-02T14:03:40.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013921674 2023-07-02T14:03:40.515 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.515 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.515 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.516 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:03:40.516 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:03:40.516 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.48248485200000002, 2023-07-02T14:03:40.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:03:40.516 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.516 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:40.516 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:03:40.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:03:40.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:40.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:03:40.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:03:40.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:03:40.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:03:40.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:03:40.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:03:40.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:03:40.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:03:40.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:03:40.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:03:40.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:03:40.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:03:40.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:40.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:03:40.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:03:40.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:40.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:03:40.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:03:40.523 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.523 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.523 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.523 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:03:40.523 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:03:40.524 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.47782749400000002, 2023-07-02T14:03:40.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:03:40.524 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.524 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:40.524 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:03:40.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:03:40.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:40.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:03:40.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:03:40.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:03:40.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:03:40.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:03:40.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:03:40.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:03:40.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:03:40.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:03:40.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:03:40.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:03:40.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:03:40.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:40.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:03:40.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:03:40.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:40.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:03:40.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:03:40.530 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.531 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.531 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.531 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:03:40.531 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:03:40.531 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.47688763099999998, 2023-07-02T14:03:40.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:03:40.532 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.532 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:40.532 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:03:40.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:03:40.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:40.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:03:40.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:03:40.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:03:40.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:03:40.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:03:40.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:03:40.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:03:40.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:03:40.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:03:40.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:03:40.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:03:40.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:03:40.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:40.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:03:40.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:03:40.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:40.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:03:40.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:03:40.538 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.538 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.538 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.539 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:03:40.539 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:03:40.539 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.47548975300000001, 2023-07-02T14:03:40.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:03:40.539 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.539 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:40.540 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:03:40.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:03:40.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:40.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:03:40.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:03:40.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:03:40.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:03:40.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:03:40.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:03:40.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:03:40.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:03:40.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:03:40.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:03:40.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:03:40.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:03:40.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:40.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:03:40.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:03:40.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:40.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:03:40.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:03:40.546 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.546 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.546 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.547 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6148 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1344, mlcod=81'1335)", 2023-07-02T14:03:40.547 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.956703+0000", 2023-07-02T14:03:40.547 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.47057546099999997, 2023-07-02T14:03:40.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017064030000000001, 2023-07-02T14:03:40.547 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.547 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:40.547 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956703+0000", 2023-07-02T14:03:40.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956703+0000", 2023-07-02T14:03:40.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:40.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956702+0000", 2023-07-02T14:03:40.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012505 2023-07-02T14:03:40.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956827+0000", 2023-07-02T14:03:40.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7309999999999999e-05 2023-07-02T14:03:40.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956845+0000", 2023-07-02T14:03:40.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7570000000000001e-06 2023-07-02T14:03:40.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956850+0000", 2023-07-02T14:03:40.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071775149999999998 2023-07-02T14:03:40.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964028+0000", 2023-07-02T14:03:40.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4773000000000002e-05 2023-07-02T14:03:40.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964053+0000", 2023-07-02T14:03:40.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096921820000000006 2023-07-02T14:03:40.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:40.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973745+0000", 2023-07-02T14:03:40.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2272999999999998e-05 2023-07-02T14:03:40.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:40.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973767+0000", 2023-07-02T14:03:40.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017064030000000001 2023-07-02T14:03:40.554 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.554 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.554 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.554 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6150 2.0 e81/46 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 81'1345, mlcod=81'1338)", 2023-07-02T14:03:40.554 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:03:40.555 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.46746680600000001, 2023-07-02T14:03:40.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019172812000000001, 2023-07-02T14:03:40.555 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.555 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:40.555 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:03:40.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:03:40.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:40.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959810+0000", 2023-07-02T14:03:40.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048438650000000001 2023-07-02T14:03:40.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964653+0000", 2023-07-02T14:03:40.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2136e-05 2023-07-02T14:03:40.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964666+0000", 2023-07-02T14:03:40.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2169999999999999e-06 2023-07-02T14:03:40.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964672+0000", 2023-07-02T14:03:40.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00047561800000000003 2023-07-02T14:03:40.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965147+0000", 2023-07-02T14:03:40.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5891e-05 2023-07-02T14:03:40.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965173+0000", 2023-07-02T14:03:40.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013770973000000001 2023-07-02T14:03:40.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:40.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978944+0000", 2023-07-02T14:03:40.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0256999999999998e-05 2023-07-02T14:03:40.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:40.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978985+0000", 2023-07-02T14:03:40.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019172812000000001 2023-07-02T14:03:40.562 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.562 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.562 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.563 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6151 2.0 e81/46 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 81'1346, mlcod=81'1338)", 2023-07-02T14:03:40.563 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.964682+0000", 2023-07-02T14:03:40.563 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.46259700500000001, 2023-07-02T14:03:40.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014325582, 2023-07-02T14:03:40.563 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.563 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:40.563 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964682+0000", 2023-07-02T14:03:40.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964682+0000", 2023-07-02T14:03:40.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:40.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964681+0000", 2023-07-02T14:03:40.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5908000000000001e-05 2023-07-02T14:03:40.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964697+0000", 2023-07-02T14:03:40.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1920000000000005e-06 2023-07-02T14:03:40.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964705+0000", 2023-07-02T14:03:40.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.493e-06 2023-07-02T14:03:40.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964709+0000", 2023-07-02T14:03:40.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075784019999999997 2023-07-02T14:03:40.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972288+0000", 2023-07-02T14:03:40.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3762e-05 2023-07-02T14:03:40.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972302+0000", 2023-07-02T14:03:40.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066885219999999997 2023-07-02T14:03:40.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:40.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978990+0000", 2023-07-02T14:03:40.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7011999999999999e-05 2023-07-02T14:03:40.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:40.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.979007+0000", 2023-07-02T14:03:40.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014325582 2023-07-02T14:03:40.570 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.570 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.570 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.570 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6153 2.0 e81/46)", 2023-07-02T14:03:40.571 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.964714+0000", 2023-07-02T14:03:40.571 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.46256419399999998, 2023-07-02T14:03:40.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014315782000000001, 2023-07-02T14:03:40.571 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.571 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:40.571 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964714+0000", 2023-07-02T14:03:40.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964714+0000", 2023-07-02T14:03:40.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:40.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964714+0000", 2023-07-02T14:03:40.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2535999999999999e-05 2023-07-02T14:03:40.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964737+0000", 2023-07-02T14:03:40.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.892e-06 2023-07-02T14:03:40.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964744+0000", 2023-07-02T14:03:40.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5040000000000004e-06 2023-07-02T14:03:40.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964749+0000", 2023-07-02T14:03:40.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0083469990000000008 2023-07-02T14:03:40.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973096+0000", 2023-07-02T14:03:40.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8771e-05 2023-07-02T14:03:40.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973115+0000", 2023-07-02T14:03:40.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058955850000000001 2023-07-02T14:03:40.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:40.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.979010+0000", 2023-07-02T14:03:40.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9847000000000002e-05 2023-07-02T14:03:40.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:40.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.979030+0000", 2023-07-02T14:03:40.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014315782000000001 2023-07-02T14:03:40.578 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.578 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.578 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.578 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6320 2.7 e81/63 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head v 81'852, mlcod=81'849)", 2023-07-02T14:03:40.578 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:40.070929+0000", 2023-07-02T14:03:40.578 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.35634948399999999, 2023-07-02T14:03:40.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014144276000000001, 2023-07-02T14:03:40.579 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:40.579 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:40.579 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:40.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:40.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.070929+0000", 2023-07-02T14:03:40.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:40.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:40.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.070929+0000", 2023-07-02T14:03:40.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:40.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:40.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.070928+0000", 2023-07-02T14:03:40.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0010305239999999999 2023-07-02T14:03:40.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:40.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.071959+0000", 2023-07-02T14:03:40.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8085999999999999e-05 2023-07-02T14:03:40.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:40.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.071977+0000", 2023-07-02T14:03:40.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3080000000000008e-06 2023-07-02T14:03:40.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:40.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.071985+0000", 2023-07-02T14:03:40.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7088999999999994e-05 2023-07-02T14:03:40.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:40.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.072052+0000", 2023-07-02T14:03:40.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1041000000000003e-05 2023-07-02T14:03:40.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:40.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.072083+0000", 2023-07-02T14:03:40.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012949866000000001 2023-07-02T14:03:40.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:40.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.085033+0000", 2023-07-02T14:03:40.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0262000000000002e-05 2023-07-02T14:03:40.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:40.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:40.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:40.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.085073+0000", 2023-07-02T14:03:40.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014144276000000001 2023-07-02T14:03:40.585 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.585 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.586 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.586 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:40.586 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:40.586 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:40.587 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:40.587 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:40.588 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:40.588 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:03:40.589 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:40.586+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:40.590 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:03:40.590 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:03:40.590 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:03:40.590 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:03:40.590 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:03:40.590 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:03:40.688 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:40.687+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:40.753 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:40.753 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:40.763 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:03:40.788 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:40.788+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:40.889 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:40.889+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:40.927 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:40.928 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:40.938 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:03:40.991 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:40.990+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:41.061 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:41.061 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:41.071 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:03:41.091 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:41.090+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:41.191 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:41.191+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:41.220 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:41.220 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:41.230 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:03:41.292 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:41.291+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:41.346 INFO:tasks.workunit.client.0.smithi163.stderr:+ sync b3fbb028-e267-4d25-a750-93ba18a2316a-mnt 2023-07-02T14:03:41.360 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:41.360 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:41.375 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:03:41.398 INFO:tasks.workunit.client.0.smithi163.stderr:+ rm -rf b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4 b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4.tar.gz 2023-07-02T14:03:41.413 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:03:41.414 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:41.414+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:41.494 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:41.493+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:41.505 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:41.505 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:41.516 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:03:41.595 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:41.594+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:41.640 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:41.641 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:41.650 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:03:41.695 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:41.695+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:41.774 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:41.775 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:41.784 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:03:41.825 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:41.825+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:41.897 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:41.897+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:41.949 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:41.950 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:41.960 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:03:42.001 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:42.001+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:42.099 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:42.099+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:42.125 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:42.126 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:42.136 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:03:42.222 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:42.221+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:42.301 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:42.301+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:42.313 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:42.313 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:42.323 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:03:42.325 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i++ )) 2023-07-02T14:03:42.325 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i < 10 )) 2023-07-02T14:03:42.325 INFO:tasks.workunit.client.0.smithi163.stderr:+ pv -L 10M linux-5.4.tar.gz 2023-07-02T14:03:42.402 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:42.401+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:42.488 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:42.488 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:42.498 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:03:42.542 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:42.541+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:42.591 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:03:42.603 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:42.602+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:42.636 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:42.637 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:42.647 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:03:42.729 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:42.731 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:42.730+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:42.747 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:42.764 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:42.782 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:42.800 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:42.804 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:42.804+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:42.812 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:42.812 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:42.812 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:42.813 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:42.813 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:42.818 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:42.819 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:03:42.820 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:03:42.820 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:03:42.820 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:03:42.820 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:03:42.820 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:03:42.823 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:03:42.905 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:42.905+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:42.989 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:42.989 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:42.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:42.990 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:42.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:42.990 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:322 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:42.990 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:42.990 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.687758813999999, 2023-07-02T14:03:42.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.03676041, 2023-07-02T14:03:42.990 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:42.991 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:42.991 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:42.991 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:42.991 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:42.992 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 322 2023-07-02T14:03:42.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:42.992 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:42.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:42.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:42.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:42.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:42.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:42.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:42.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:42.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:42.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:42.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:42.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:42.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:42.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298819+0000", 2023-07-02T14:03:42.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8068e-05 2023-07-02T14:03:42.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:42.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:42.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:42.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298877+0000", 2023-07-02T14:03:42.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3339999999999998e-06 2023-07-02T14:03:42.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:42.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:42.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:42.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298883+0000", 2023-07-02T14:03:42.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1030000000000004e-06 2023-07-02T14:03:42.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:42.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:42.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:42.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298887+0000", 2023-07-02T14:03:42.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023640230000000002 2023-07-02T14:03:42.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:42.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:42.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:42.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322528+0000", 2023-07-02T14:03:42.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8424999999999994e-05 2023-07-02T14:03:42.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:42.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:42.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:42.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322616+0000", 2023-07-02T14:03:42.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116815 2023-07-02T14:03:42.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:42.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:42.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:42.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322733+0000", 2023-07-02T14:03:42.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048252629999999998 2023-07-02T14:03:42.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:42.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:42.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:42.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327558+0000", 2023-07-02T14:03:42.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078965240000000003 2023-07-02T14:03:42.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:42.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:42.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:42.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335455+0000", 2023-07-02T14:03:42.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0511999999999999e-05 2023-07-02T14:03:42.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:42.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335515+0000", 2023-07-02T14:03:43.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4789000000000006e-05 2023-07-02T14:03:43.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335580+0000", 2023-07-02T14:03:43.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.03676041 2023-07-02T14:03:43.001 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.001 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.001 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.001 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:323 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:43.001 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:43.002 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.687687433999997, 2023-07-02T14:03:43.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036875044000000003, 2023-07-02T14:03:43.002 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.002 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.002 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:43.002 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:43.002 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:43.002 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 323 2023-07-02T14:03:43.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.003 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:43.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:43.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298890+0000", 2023-07-02T14:03:43.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7277000000000002e-05 2023-07-02T14:03:43.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298938+0000", 2023-07-02T14:03:43.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0495e-05 2023-07-02T14:03:43.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298948+0000", 2023-07-02T14:03:43.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6829999999999999e-06 2023-07-02T14:03:43.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298953+0000", 2023-07-02T14:03:43.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024271435000000001 2023-07-02T14:03:43.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323224+0000", 2023-07-02T14:03:43.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9266999999999998e-05 2023-07-02T14:03:43.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323304+0000", 2023-07-02T14:03:43.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000157021 2023-07-02T14:03:43.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:43.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323461+0000", 2023-07-02T14:03:43.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041036689999999999 2023-07-02T14:03:43.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:43.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327564+0000", 2023-07-02T14:03:43.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080753490000000008 2023-07-02T14:03:43.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:43.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335640+0000", 2023-07-02T14:03:43.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7398999999999994e-05 2023-07-02T14:03:43.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335727+0000", 2023-07-02T14:03:43.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8967000000000001e-05 2023-07-02T14:03:43.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335766+0000", 2023-07-02T14:03:43.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036875044000000003 2023-07-02T14:03:43.011 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.011 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.011 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.012 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:324 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:43.012 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:43.012 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.687620056, 2023-07-02T14:03:43.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036935673000000002, 2023-07-02T14:03:43.012 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.012 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.013 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:43.013 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:43.013 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:43.013 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 324 2023-07-02T14:03:43.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.013 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:43.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:43.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5519999999999997e-05 2023-07-02T14:03:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299013+0000", 2023-07-02T14:03:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0701e-05 2023-07-02T14:03:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299024+0000", 2023-07-02T14:03:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2379999999999999e-06 2023-07-02T14:03:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299028+0000", 2023-07-02T14:03:43.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024972768999999999 2023-07-02T14:03:43.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324001+0000", 2023-07-02T14:03:43.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1178000000000002e-05 2023-07-02T14:03:43.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324072+0000", 2023-07-02T14:03:43.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000166119 2023-07-02T14:03:43.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:43.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324238+0000", 2023-07-02T14:03:43.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033314210000000002 2023-07-02T14:03:43.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:43.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327570+0000", 2023-07-02T14:03:43.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0082393369999999994 2023-07-02T14:03:43.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:43.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335809+0000", 2023-07-02T14:03:43.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7071999999999999e-05 2023-07-02T14:03:43.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335856+0000", 2023-07-02T14:03:43.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7926999999999998e-05 2023-07-02T14:03:43.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335894+0000", 2023-07-02T14:03:43.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036935673000000002 2023-07-02T14:03:43.022 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.022 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.022 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.022 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:325 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:43.022 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:43.022 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.687544729000003, 2023-07-02T14:03:43.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036990838999999998, 2023-07-02T14:03:43.023 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.023 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.023 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:43.023 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:43.023 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:43.023 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 325 2023-07-02T14:03:43.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.024 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:43.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:43.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299033+0000", 2023-07-02T14:03:43.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107619 2023-07-02T14:03:43.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299141+0000", 2023-07-02T14:03:43.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3497e-05 2023-07-02T14:03:43.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299154+0000", 2023-07-02T14:03:43.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4669e-05 2023-07-02T14:03:43.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299169+0000", 2023-07-02T14:03:43.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025630159999999999 2023-07-02T14:03:43.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324799+0000", 2023-07-02T14:03:43.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6953000000000007e-05 2023-07-02T14:03:43.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324876+0000", 2023-07-02T14:03:43.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016346199999999999 2023-07-02T14:03:43.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:43.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325039+0000", 2023-07-02T14:03:43.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003566624 2023-07-02T14:03:43.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:43.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328606+0000", 2023-07-02T14:03:43.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073295540000000003 2023-07-02T14:03:43.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:43.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335936+0000", 2023-07-02T14:03:43.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0044999999999999e-05 2023-07-02T14:03:43.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335986+0000", 2023-07-02T14:03:43.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8872999999999999e-05 2023-07-02T14:03:43.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336024+0000", 2023-07-02T14:03:43.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036990838999999998 2023-07-02T14:03:43.032 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.032 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.032 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.032 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:326 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:43.033 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:43.033 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.687402800999998, 2023-07-02T14:03:43.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036975239999999999, 2023-07-02T14:03:43.033 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.033 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.033 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:43.033 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:43.034 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:43.034 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 326 2023-07-02T14:03:43.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.034 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:43.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:43.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:43.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299175+0000", 2023-07-02T14:03:43.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4782000000000003e-05 2023-07-02T14:03:43.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299250+0000", 2023-07-02T14:03:43.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.277e-05 2023-07-02T14:03:43.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299262+0000", 2023-07-02T14:03:43.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1460000000000001e-06 2023-07-02T14:03:43.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299268+0000", 2023-07-02T14:03:43.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026158098000000001 2023-07-02T14:03:43.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325427+0000", 2023-07-02T14:03:43.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4845999999999995e-05 2023-07-02T14:03:43.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325501+0000", 2023-07-02T14:03:43.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015890699999999999 2023-07-02T14:03:43.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:43.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325660+0000", 2023-07-02T14:03:43.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002952114 2023-07-02T14:03:43.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:43.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328612+0000", 2023-07-02T14:03:43.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074491949999999996 2023-07-02T14:03:43.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:43.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336062+0000", 2023-07-02T14:03:43.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7376999999999997e-05 2023-07-02T14:03:43.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336109+0000", 2023-07-02T14:03:43.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1814000000000003e-05 2023-07-02T14:03:43.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336151+0000", 2023-07-02T14:03:43.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036975239999999999 2023-07-02T14:03:43.042 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.042 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.042 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.043 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:327 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 589824~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:43.043 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:43.043 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.687303540000002, 2023-07-02T14:03:43.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037234253000000002, 2023-07-02T14:03:43.043 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.043 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.044 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:43.044 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:43.044 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:43.044 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 327 2023-07-02T14:03:43.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.044 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:43.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:43.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299274+0000", 2023-07-02T14:03:43.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2213999999999997e-05 2023-07-02T14:03:43.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299346+0000", 2023-07-02T14:03:43.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1816e-05 2023-07-02T14:03:43.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299358+0000", 2023-07-02T14:03:43.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2626000000000001e-05 2023-07-02T14:03:43.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299371+0000", 2023-07-02T14:03:43.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026916853000000001 2023-07-02T14:03:43.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326288+0000", 2023-07-02T14:03:43.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6909000000000004e-05 2023-07-02T14:03:43.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326375+0000", 2023-07-02T14:03:43.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015430399999999999 2023-07-02T14:03:43.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:43.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326529+0000", 2023-07-02T14:03:43.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096652100000000005 2023-07-02T14:03:43.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:43.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336194+0000", 2023-07-02T14:03:43.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115299 2023-07-02T14:03:43.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:43.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336309+0000", 2023-07-02T14:03:43.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015981200000000001 2023-07-02T14:03:43.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336469+0000", 2023-07-02T14:03:43.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9867999999999998e-05 2023-07-02T14:03:43.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336509+0000", 2023-07-02T14:03:43.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037234253000000002 2023-07-02T14:03:43.053 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.053 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.053 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.053 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:328 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:43.053 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:43.053 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.687199950999997, 2023-07-02T14:03:43.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037216610999999997, 2023-07-02T14:03:43.054 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.054 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.054 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:43.054 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:43.054 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:43.054 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 328 2023-07-02T14:03:43.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.055 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:43.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:43.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:43.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1412e-05 2023-07-02T14:03:43.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299449+0000", 2023-07-02T14:03:43.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0219000000000001e-05 2023-07-02T14:03:43.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299469+0000", 2023-07-02T14:03:43.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9480000000000003e-06 2023-07-02T14:03:43.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299479+0000", 2023-07-02T14:03:43.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027444344999999998 2023-07-02T14:03:43.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326924+0000", 2023-07-02T14:03:43.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5165000000000001e-05 2023-07-02T14:03:43.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327019+0000", 2023-07-02T14:03:43.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019113499999999999 2023-07-02T14:03:43.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:43.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327210+0000", 2023-07-02T14:03:43.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090197449999999992 2023-07-02T14:03:43.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:43.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336230+0000", 2023-07-02T14:03:43.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028764399999999999 2023-07-02T14:03:43.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:43.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336517+0000", 2023-07-02T14:03:43.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1981999999999999e-05 2023-07-02T14:03:43.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336559+0000", 2023-07-02T14:03:43.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:03:43.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336595+0000", 2023-07-02T14:03:43.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037216610999999997 2023-07-02T14:03:43.064 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.064 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.064 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.064 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:329 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:43.064 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:43.065 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.687090034000001, 2023-07-02T14:03:43.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037191142000000003, 2023-07-02T14:03:43.065 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.065 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.065 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:43.065 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:43.065 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:43.066 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 329 2023-07-02T14:03:43.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.066 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:43.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:43.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:43.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299487+0000", 2023-07-02T14:03:43.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7871000000000001e-05 2023-07-02T14:03:43.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299555+0000", 2023-07-02T14:03:43.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0446e-05 2023-07-02T14:03:43.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299566+0000", 2023-07-02T14:03:43.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7939999999999999e-06 2023-07-02T14:03:43.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299572+0000", 2023-07-02T14:03:43.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028034019 2023-07-02T14:03:43.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327606+0000", 2023-07-02T14:03:43.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1967999999999999e-05 2023-07-02T14:03:43.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327678+0000", 2023-07-02T14:03:43.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022216900000000001 2023-07-02T14:03:43.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:43.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327900+0000", 2023-07-02T14:03:43.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084016899999999999 2023-07-02T14:03:43.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:43.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336301+0000", 2023-07-02T14:03:43.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00030122800000000001 2023-07-02T14:03:43.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:43.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336603+0000", 2023-07-02T14:03:43.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0380000000000003e-05 2023-07-02T14:03:43.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336643+0000", 2023-07-02T14:03:43.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6455999999999998e-05 2023-07-02T14:03:43.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336679+0000", 2023-07-02T14:03:43.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037191142000000003 2023-07-02T14:03:43.074 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.074 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.074 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.075 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:330 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:43.075 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:43.075 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.686999264000001, 2023-07-02T14:03:43.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037849305, 2023-07-02T14:03:43.075 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.075 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.075 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:43.076 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:43.076 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:43.076 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 330 2023-07-02T14:03:43.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.076 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:43.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:43.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299578+0000", 2023-07-02T14:03:43.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4708000000000005e-05 2023-07-02T14:03:43.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299643+0000", 2023-07-02T14:03:43.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7309999999999999e-06 2023-07-02T14:03:43.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299652+0000", 2023-07-02T14:03:43.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7989999999999999e-06 2023-07-02T14:03:43.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299658+0000", 2023-07-02T14:03:43.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028981664000000001 2023-07-02T14:03:43.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328639+0000", 2023-07-02T14:03:43.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0939000000000006e-05 2023-07-02T14:03:43.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328710+0000", 2023-07-02T14:03:43.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000142624 2023-07-02T14:03:43.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:43.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328853+0000", 2023-07-02T14:03:43.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078374640000000006 2023-07-02T14:03:43.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:43.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336690+0000", 2023-07-02T14:03:43.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00068453000000000003 2023-07-02T14:03:43.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:43.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337375+0000", 2023-07-02T14:03:43.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0667999999999998e-05 2023-07-02T14:03:43.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337406+0000", 2023-07-02T14:03:43.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2858999999999999e-05 2023-07-02T14:03:43.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337428+0000", 2023-07-02T14:03:43.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037849305 2023-07-02T14:03:43.085 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.085 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.085 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.086 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:331 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:43.086 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:43.086 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.686915401999997, 2023-07-02T14:03:43.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037870332, 2023-07-02T14:03:43.086 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.086 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.086 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:43.087 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:43.087 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:43.087 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 331 2023-07-02T14:03:43.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.087 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:43.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:43.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299662+0000", 2023-07-02T14:03:43.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2252e-05 2023-07-02T14:03:43.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299725+0000", 2023-07-02T14:03:43.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8440000000000004e-06 2023-07-02T14:03:43.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299733+0000", 2023-07-02T14:03:43.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6759999999999998e-06 2023-07-02T14:03:43.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299738+0000", 2023-07-02T14:03:43.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029459387 2023-07-02T14:03:43.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329197+0000", 2023-07-02T14:03:43.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0087999999999997e-05 2023-07-02T14:03:43.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329268+0000", 2023-07-02T14:03:43.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014405700000000001 2023-07-02T14:03:43.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:43.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329412+0000", 2023-07-02T14:03:43.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072842100000000002 2023-07-02T14:03:43.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:43.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336696+0000", 2023-07-02T14:03:43.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00076275300000000002 2023-07-02T14:03:43.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:43.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337459+0000", 2023-07-02T14:03:43.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5988000000000002e-05 2023-07-02T14:03:43.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337495+0000", 2023-07-02T14:03:43.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8686000000000002e-05 2023-07-02T14:03:43.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337533+0000", 2023-07-02T14:03:43.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037870332 2023-07-02T14:03:43.095 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.095 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.096 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.096 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:332 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:43.096 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:43.096 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.686835477999999, 2023-07-02T14:03:43.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037892506999999999, 2023-07-02T14:03:43.097 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.097 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.097 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:43.097 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:43.097 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:43.097 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 332 2023-07-02T14:03:43.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.097 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:43.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:43.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299742+0000", 2023-07-02T14:03:43.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1366999999999997e-05 2023-07-02T14:03:43.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299804+0000", 2023-07-02T14:03:43.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9319999999999996e-06 2023-07-02T14:03:43.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299812+0000", 2023-07-02T14:03:43.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4359999999999999e-06 2023-07-02T14:03:43.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299818+0000", 2023-07-02T14:03:43.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029946144000000001 2023-07-02T14:03:43.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329764+0000", 2023-07-02T14:03:43.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000164828 2023-07-02T14:03:43.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329929+0000", 2023-07-02T14:03:43.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016139400000000001 2023-07-02T14:03:43.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:43.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330090+0000", 2023-07-02T14:03:43.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066134009999999997 2023-07-02T14:03:43.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:43.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336704+0000", 2023-07-02T14:03:43.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00086955400000000003 2023-07-02T14:03:43.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:43.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337573+0000", 2023-07-02T14:03:43.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8447000000000001e-05 2023-07-02T14:03:43.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337602+0000", 2023-07-02T14:03:43.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.362e-05 2023-07-02T14:03:43.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337635+0000", 2023-07-02T14:03:43.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037892506999999999 2023-07-02T14:03:43.106 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.106 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.106 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.107 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:333 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:43.107 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:43.107 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.686755396999999, 2023-07-02T14:03:43.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037932598999999997, 2023-07-02T14:03:43.107 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.107 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.107 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:43.107 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:43.108 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:43.108 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 333 2023-07-02T14:03:43.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.108 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:43.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:43.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299822+0000", 2023-07-02T14:03:43.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0831000000000001e-05 2023-07-02T14:03:43.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299883+0000", 2023-07-02T14:03:43.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.322e-06 2023-07-02T14:03:43.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299888+0000", 2023-07-02T14:03:43.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3479999999999997e-06 2023-07-02T14:03:43.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299893+0000", 2023-07-02T14:03:43.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030831905999999999 2023-07-02T14:03:43.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330725+0000", 2023-07-02T14:03:43.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3560999999999996e-05 2023-07-02T14:03:43.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330798+0000", 2023-07-02T14:03:43.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015165999999999999 2023-07-02T14:03:43.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:43.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330950+0000", 2023-07-02T14:03:43.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057590319999999999 2023-07-02T14:03:43.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:43.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336709+0000", 2023-07-02T14:03:43.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00096987899999999999 2023-07-02T14:03:43.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:43.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337679+0000", 2023-07-02T14:03:43.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9699000000000001e-05 2023-07-02T14:03:43.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337719+0000", 2023-07-02T14:03:43.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6961000000000001e-05 2023-07-02T14:03:43.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337756+0000", 2023-07-02T14:03:43.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037932598999999997 2023-07-02T14:03:43.116 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.116 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.116 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.117 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:416 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:43.117 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:43.117 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.668462018, 2023-07-02T14:03:43.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039242025, 2023-07-02T14:03:43.117 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.118 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.118 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:43.118 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:43.118 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:43.118 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:03:43.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.118 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:43.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:43.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:43.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.692e-06 2023-07-02T14:03:43.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318119+0000", 2023-07-02T14:03:43.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9790000000000005e-06 2023-07-02T14:03:43.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318129+0000", 2023-07-02T14:03:43.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3640000000000001e-06 2023-07-02T14:03:43.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318137+0000", 2023-07-02T14:03:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022607808 2023-07-02T14:03:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340745+0000", 2023-07-02T14:03:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0130999999999996e-05 2023-07-02T14:03:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340825+0000", 2023-07-02T14:03:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039581080000000001 2023-07-02T14:03:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344783+0000", 2023-07-02T14:03:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000006e-05 2023-07-02T14:03:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344854+0000", 2023-07-02T14:03:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052105550000000004 2023-07-02T14:03:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350065+0000", 2023-07-02T14:03:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0321999999999997e-05 2023-07-02T14:03:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350115+0000", 2023-07-02T14:03:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012378409999999999 2023-07-02T14:03:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351353+0000", 2023-07-02T14:03:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5942000000000001e-05 2023-07-02T14:03:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351409+0000", 2023-07-02T14:03:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035677700000000002 2023-07-02T14:03:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351766+0000", 2023-07-02T14:03:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000926719 2023-07-02T14:03:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352693+0000", 2023-07-02T14:03:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045652080000000003 2023-07-02T14:03:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357258+0000", 2023-07-02T14:03:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4717e-05 2023-07-02T14:03:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357293+0000", 2023-07-02T14:03:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5807000000000002e-05 2023-07-02T14:03:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357358+0000", 2023-07-02T14:03:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039242025 2023-07-02T14:03:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:417 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.668436581000002, 2023-07-02T14:03:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042407105, 2023-07-02T14:03:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 417 2023-07-02T14:03:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318141+0000", 2023-07-02T14:03:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7120000000000002e-06 2023-07-02T14:03:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318145+0000", 2023-07-02T14:03:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8869999999999997e-06 2023-07-02T14:03:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318151+0000", 2023-07-02T14:03:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9820000000000002e-06 2023-07-02T14:03:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318155+0000", 2023-07-02T14:03:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699133 2023-07-02T14:03:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340854+0000", 2023-07-02T14:03:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6849999999999998e-05 2023-07-02T14:03:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340931+0000", 2023-07-02T14:03:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039672140000000002 2023-07-02T14:03:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344898+0000", 2023-07-02T14:03:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5341999999999998e-05 2023-07-02T14:03:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344973+0000", 2023-07-02T14:03:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051819400000000003 2023-07-02T14:03:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350155+0000", 2023-07-02T14:03:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3261999999999999e-05 2023-07-02T14:03:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350209+0000", 2023-07-02T14:03:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00194097 2023-07-02T14:03:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352150+0000", 2023-07-02T14:03:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6851e-05 2023-07-02T14:03:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352226+0000", 2023-07-02T14:03:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051831109999999998 2023-07-02T14:03:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357409+0000", 2023-07-02T14:03:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5325e-05 2023-07-02T14:03:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357495+0000", 2023-07-02T14:03:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042107899999999998 2023-07-02T14:03:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357916+0000", 2023-07-02T14:03:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0011672449999999999 2023-07-02T14:03:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.359083+0000", 2023-07-02T14:03:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013534930000000001 2023-07-02T14:03:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360437+0000", 2023-07-02T14:03:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2159000000000002e-05 2023-07-02T14:03:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360469+0000", 2023-07-02T14:03:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0092000000000005e-05 2023-07-02T14:03:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360549+0000", 2023-07-02T14:03:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042407105 2023-07-02T14:03:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:418 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.668417339999998, 2023-07-02T14:03:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.045613663999999998, 2023-07-02T14:03:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 418 2023-07-02T14:03:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318160+0000", 2023-07-02T14:03:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6490000000000001e-06 2023-07-02T14:03:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318164+0000", 2023-07-02T14:03:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:03:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318170+0000", 2023-07-02T14:03:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5279999999999999e-06 2023-07-02T14:03:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318173+0000", 2023-07-02T14:03:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022788856 2023-07-02T14:03:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340962+0000", 2023-07-02T14:03:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7730000000000003e-05 2023-07-02T14:03:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341040+0000", 2023-07-02T14:03:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039795899999999999 2023-07-02T14:03:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345020+0000", 2023-07-02T14:03:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4962000000000006e-05 2023-07-02T14:03:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345085+0000", 2023-07-02T14:03:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051955430000000004 2023-07-02T14:03:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350280+0000", 2023-07-02T14:03:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8913999999999995e-05 2023-07-02T14:03:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350379+0000", 2023-07-02T14:03:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001887344 2023-07-02T14:03:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352266+0000", 2023-07-02T14:03:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7865000000000003e-05 2023-07-02T14:03:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352314+0000", 2023-07-02T14:03:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060621190000000004 2023-07-02T14:03:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358376+0000", 2023-07-02T14:03:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2448999999999997e-05 2023-07-02T14:03:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358449+0000", 2023-07-02T14:03:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0021539929999999999 2023-07-02T14:03:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360603+0000", 2023-07-02T14:03:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7859000000000002e-05 2023-07-02T14:03:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360671+0000", 2023-07-02T14:03:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026930699999999999 2023-07-02T14:03:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360940+0000", 2023-07-02T14:03:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00081179699999999995 2023-07-02T14:03:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361752+0000", 2023-07-02T14:03:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0019350490000000001 2023-07-02T14:03:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363687+0000", 2023-07-02T14:03:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4462000000000003e-05 2023-07-02T14:03:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363721+0000", 2023-07-02T14:03:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3408000000000001e-05 2023-07-02T14:03:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363775+0000", 2023-07-02T14:03:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.045613663999999998 2023-07-02T14:03:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:419 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.668401013, 2023-07-02T14:03:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.047296703000000002, 2023-07-02T14:03:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 419 2023-07-02T14:03:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:43.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.433e-06 2023-07-02T14:03:43.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318180+0000", 2023-07-02T14:03:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9950000000000002e-06 2023-07-02T14:03:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318186+0000", 2023-07-02T14:03:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7290000000000002e-06 2023-07-02T14:03:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318190+0000", 2023-07-02T14:03:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022877989000000001 2023-07-02T14:03:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341068+0000", 2023-07-02T14:03:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5803999999999994e-05 2023-07-02T14:03:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341154+0000", 2023-07-02T14:03:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039633000000000003 2023-07-02T14:03:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345117+0000", 2023-07-02T14:03:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7188999999999996e-05 2023-07-02T14:03:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345194+0000", 2023-07-02T14:03:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052306059999999996 2023-07-02T14:03:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350425+0000", 2023-07-02T14:03:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8093999999999997e-05 2023-07-02T14:03:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350473+0000", 2023-07-02T14:03:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001865158 2023-07-02T14:03:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352338+0000", 2023-07-02T14:03:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5479000000000003e-05 2023-07-02T14:03:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352384+0000", 2023-07-02T14:03:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061153290000000001 2023-07-02T14:03:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358499+0000", 2023-07-02T14:03:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5156999999999999e-05 2023-07-02T14:03:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358574+0000", 2023-07-02T14:03:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027847459999999998 2023-07-02T14:03:43.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361359+0000", 2023-07-02T14:03:43.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2633999999999999e-05 2023-07-02T14:03:43.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361402+0000", 2023-07-02T14:03:43.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024171510000000002 2023-07-02T14:03:43.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363819+0000", 2023-07-02T14:03:43.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6988999999999999e-05 2023-07-02T14:03:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363876+0000", 2023-07-02T14:03:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035321300000000001 2023-07-02T14:03:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.364229+0000", 2023-07-02T14:03:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00094674799999999997 2023-07-02T14:03:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365176+0000", 2023-07-02T14:03:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019724200000000001 2023-07-02T14:03:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365373+0000", 2023-07-02T14:03:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2116e-05 2023-07-02T14:03:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365425+0000", 2023-07-02T14:03:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8996999999999999e-05 2023-07-02T14:03:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365474+0000", 2023-07-02T14:03:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.047296703000000002 2023-07-02T14:03:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:472 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.608944780999998, 2023-07-02T14:03:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036204721000000002, 2023-07-02T14:03:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 472 2023-07-02T14:03:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:03:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377633+0000", 2023-07-02T14:03:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.815e-06 2023-07-02T14:03:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377635+0000", 2023-07-02T14:03:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9459999999999998e-06 2023-07-02T14:03:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377638+0000", 2023-07-02T14:03:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.673e-06 2023-07-02T14:03:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377643+0000", 2023-07-02T14:03:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043004239999999997 2023-07-02T14:03:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.381943+0000", 2023-07-02T14:03:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4321999999999999e-05 2023-07-02T14:03:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.381998+0000", 2023-07-02T14:03:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003008377 2023-07-02T14:03:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385006+0000", 2023-07-02T14:03:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8751999999999998e-05 2023-07-02T14:03:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385045+0000", 2023-07-02T14:03:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086260880000000005 2023-07-02T14:03:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393671+0000", 2023-07-02T14:03:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6365999999999997e-05 2023-07-02T14:03:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393707+0000", 2023-07-02T14:03:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028560289999999999 2023-07-02T14:03:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396563+0000", 2023-07-02T14:03:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6195e-05 2023-07-02T14:03:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396619+0000", 2023-07-02T14:03:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023118269999999998 2023-07-02T14:03:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.398931+0000", 2023-07-02T14:03:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3885999999999999e-05 2023-07-02T14:03:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.398985+0000", 2023-07-02T14:03:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043756890000000003 2023-07-02T14:03:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403361+0000", 2023-07-02T14:03:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0043000000000001e-05 2023-07-02T14:03:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403421+0000", 2023-07-02T14:03:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001590972 2023-07-02T14:03:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405012+0000", 2023-07-02T14:03:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0067000000000005e-05 2023-07-02T14:03:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405082+0000", 2023-07-02T14:03:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050958610000000001 2023-07-02T14:03:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410178+0000", 2023-07-02T14:03:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7070000000000004e-05 2023-07-02T14:03:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410275+0000", 2023-07-02T14:03:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060121100000000004 2023-07-02T14:03:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410876+0000", 2023-07-02T14:03:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3177999999999999e-05 2023-07-02T14:03:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410919+0000", 2023-07-02T14:03:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000271394 2023-07-02T14:03:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411191+0000", 2023-07-02T14:03:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001009598 2023-07-02T14:03:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.412200+0000", 2023-07-02T14:03:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014789790000000001 2023-07-02T14:03:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413679+0000", 2023-07-02T14:03:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2624000000000002e-05 2023-07-02T14:03:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413742+0000", 2023-07-02T14:03:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6564999999999994e-05 2023-07-02T14:03:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413838+0000", 2023-07-02T14:03:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036204721000000002 2023-07-02T14:03:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:473 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.608933370000003, 2023-07-02T14:03:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037915904, 2023-07-02T14:03:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 473 2023-07-02T14:03:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:03:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.119e-06 2023-07-02T14:03:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377647+0000", 2023-07-02T14:03:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.875e-06 2023-07-02T14:03:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377650+0000", 2023-07-02T14:03:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1949999999999999e-06 2023-07-02T14:03:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377653+0000", 2023-07-02T14:03:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004375444 2023-07-02T14:03:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382028+0000", 2023-07-02T14:03:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1063999999999997e-05 2023-07-02T14:03:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382089+0000", 2023-07-02T14:03:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030043869999999999 2023-07-02T14:03:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385094+0000", 2023-07-02T14:03:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8167999999999997e-05 2023-07-02T14:03:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385142+0000", 2023-07-02T14:03:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008603421 2023-07-02T14:03:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393745+0000", 2023-07-02T14:03:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3763000000000005e-05 2023-07-02T14:03:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393819+0000", 2023-07-02T14:03:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028274720000000001 2023-07-02T14:03:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396647+0000", 2023-07-02T14:03:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7929999999999997e-05 2023-07-02T14:03:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396695+0000", 2023-07-02T14:03:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023426459999999999 2023-07-02T14:03:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399037+0000", 2023-07-02T14:03:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5846000000000004e-05 2023-07-02T14:03:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399123+0000", 2023-07-02T14:03:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043448360000000004 2023-07-02T14:03:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403468+0000", 2023-07-02T14:03:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6699000000000001e-05 2023-07-02T14:03:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403515+0000", 2023-07-02T14:03:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001622943 2023-07-02T14:03:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405138+0000", 2023-07-02T14:03:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6382000000000002e-05 2023-07-02T14:03:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405204+0000", 2023-07-02T14:03:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051089910000000002 2023-07-02T14:03:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410313+0000", 2023-07-02T14:03:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8106000000000003e-05 2023-07-02T14:03:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410361+0000", 2023-07-02T14:03:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012200399999999999 2023-07-02T14:03:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411581+0000", 2023-07-02T14:03:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5067000000000005e-05 2023-07-02T14:03:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411656+0000", 2023-07-02T14:03:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022353249999999998 2023-07-02T14:03:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413891+0000", 2023-07-02T14:03:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9356000000000003e-05 2023-07-02T14:03:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413981+0000", 2023-07-02T14:03:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00043818299999999998 2023-07-02T14:03:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.414419+0000", 2023-07-02T14:03:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00081559199999999997 2023-07-02T14:03:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415235+0000", 2023-07-02T14:03:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026434499999999998 2023-07-02T14:03:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415499+0000", 2023-07-02T14:03:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9779999999999999e-05 2023-07-02T14:03:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415529+0000", 2023-07-02T14:03:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2159999999999997e-05 2023-07-02T14:03:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415561+0000", 2023-07-02T14:03:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037915904 2023-07-02T14:03:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:475 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.608923331, 2023-07-02T14:03:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039286962000000002, 2023-07-02T14:03:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 475 2023-07-02T14:03:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4779999999999997e-06 2023-07-02T14:03:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377664+0000", 2023-07-02T14:03:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.268e-06 2023-07-02T14:03:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377671+0000", 2023-07-02T14:03:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.861e-06 2023-07-02T14:03:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377674+0000", 2023-07-02T14:03:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044395789999999999 2023-07-02T14:03:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382114+0000", 2023-07-02T14:03:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8096999999999997e-05 2023-07-02T14:03:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382162+0000", 2023-07-02T14:03:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029949059999999999 2023-07-02T14:03:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385157+0000", 2023-07-02T14:03:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1158000000000001e-05 2023-07-02T14:03:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385198+0000", 2023-07-02T14:03:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086477229999999995 2023-07-02T14:03:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393846+0000", 2023-07-02T14:03:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5990999999999999e-05 2023-07-02T14:03:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393912+0000", 2023-07-02T14:03:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027979599999999999 2023-07-02T14:03:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396710+0000", 2023-07-02T14:03:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5911000000000002e-05 2023-07-02T14:03:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396746+0000", 2023-07-02T14:03:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024039759999999999 2023-07-02T14:03:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399150+0000", 2023-07-02T14:03:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7834999999999999e-05 2023-07-02T14:03:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399188+0000", 2023-07-02T14:03:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004364054 2023-07-02T14:03:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403552+0000", 2023-07-02T14:03:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6274999999999999e-05 2023-07-02T14:03:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403608+0000", 2023-07-02T14:03:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0016261540000000001 2023-07-02T14:03:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405234+0000", 2023-07-02T14:03:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6945000000000004e-05 2023-07-02T14:03:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405301+0000", 2023-07-02T14:03:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050871329999999998 2023-07-02T14:03:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410388+0000", 2023-07-02T14:03:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0210999999999995e-05 2023-07-02T14:03:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410468+0000", 2023-07-02T14:03:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001227454 2023-07-02T14:03:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411696+0000", 2023-07-02T14:03:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8188000000000005e-05 2023-07-02T14:03:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411764+0000", 2023-07-02T14:03:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031779930000000001 2023-07-02T14:03:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.414942+0000", 2023-07-02T14:03:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7236000000000004e-05 2023-07-02T14:03:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415019+0000", 2023-07-02T14:03:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000573045 2023-07-02T14:03:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415592+0000", 2023-07-02T14:03:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7580000000000002e-05 2023-07-02T14:03:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415640+0000", 2023-07-02T14:03:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024783200000000002 2023-07-02T14:03:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415888+0000", 2023-07-02T14:03:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00080440999999999998 2023-07-02T14:03:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416692+0000", 2023-07-02T14:03:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000195787 2023-07-02T14:03:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416888+0000", 2023-07-02T14:03:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0787999999999997e-05 2023-07-02T14:03:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416919+0000", 2023-07-02T14:03:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.338e-05 2023-07-02T14:03:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416942+0000", 2023-07-02T14:03:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039286962000000002 2023-07-02T14:03:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:5269 2.5 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:head [write 446464~1847296] snapc e=[e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e81)", 2023-07-02T14:03:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.147365+0000", 2023-07-02T14:03:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.8392131960000002, 2023-07-02T14:03:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036564266999999998, 2023-07-02T14:03:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 5269 2023-07-02T14:03:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.147365+0000", 2023-07-02T14:03:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.147365+0000", 2023-07-02T14:03:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.147365+0000", 2023-07-02T14:03:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001616302 2023-07-02T14:03:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.148981+0000", 2023-07-02T14:03:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1630000000000002e-06 2023-07-02T14:03:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.148990+0000", 2023-07-02T14:03:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4989999999999999e-06 2023-07-02T14:03:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.148997+0000", 2023-07-02T14:03:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012255096 2023-07-02T14:03:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.161252+0000", 2023-07-02T14:03:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3946e-05 2023-07-02T14:03:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.161326+0000", 2023-07-02T14:03:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022859399999999999 2023-07-02T14:03:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.161554+0000", 2023-07-02T14:03:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062552529999999997 2023-07-02T14:03:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.167810+0000", 2023-07-02T14:03:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016036499999999999 2023-07-02T14:03:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.183846+0000", 2023-07-02T14:03:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7889000000000003e-05 2023-07-02T14:03:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.183884+0000", 2023-07-02T14:03:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5410000000000001e-05 2023-07-02T14:03:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.183929+0000", 2023-07-02T14:03:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036564266999999998 2023-07-02T14:03:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.258 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:43.259 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap10 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap11 b3fbb028-e267-4d25-a750-93ba18a2316a@snap11.diff 2023-07-02T14:03:43.259 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:03:43.164+0000 7f3f5a346700 0 -- 172.21.15.163:0/3318643050 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x56082e276df0 msgr2=0x56082e2791e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:03:43.259 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:03:43.172+0000 7f3f59b45700 0 -- 172.21.15.163:0/3318643050 >> [v2:172.21.15.22:6808/83350,v1:172.21.15.22:6809/83350] conn(0x7f3f40008780 msgr2=0x7f3f4000abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:03:43.260 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:03:43.260 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:43.258+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:43.360 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:43.360+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:43.426 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:03:43.430 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:43.431 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:43.431 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:03:43.431 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:43.441 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:03:43.483 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:43.483+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:43.561 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:43.560+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:43.565 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:43.565 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:43.566 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:43.566 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:43.566 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:43.576 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:03:43.585 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 50% complete... Exporting image: 100% complete...done. 2023-07-02T14:03:43.591 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:03:43.592 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=12 2023-07-02T14:03:43.592 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap12 2023-07-02T14:03:43.662 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:43.661+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:43.742 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:43.742 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:43.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:43.742 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:43.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.743 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:5545 2.0 e81/46 2:159aabad:::rbd_data.125de1449605.000000000000005d:head v 81'1159, mlcod=81'1155)", 2023-07-02T14:03:43.743 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.389719+0000", 2023-07-02T14:03:43.743 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.3508788389999999, 2023-07-02T14:03:43.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014352986, 2023-07-02T14:03:43.743 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.743 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.743 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.389719+0000", 2023-07-02T14:03:43.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.389719+0000", 2023-07-02T14:03:43.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:43.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.389718+0000", 2023-07-02T14:03:43.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037957160000000002 2023-07-02T14:03:43.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.393513+0000", 2023-07-02T14:03:43.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7331000000000001e-05 2023-07-02T14:03:43.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.393531+0000", 2023-07-02T14:03:43.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0229999999999998e-06 2023-07-02T14:03:43.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.393540+0000", 2023-07-02T14:03:43.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0124000000000002e-05 2023-07-02T14:03:43.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.393600+0000", 2023-07-02T14:03:43.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2815999999999999e-05 2023-07-02T14:03:43.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.393633+0000", 2023-07-02T14:03:43.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010399037 2023-07-02T14:03:43.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.404032+0000", 2023-07-02T14:03:43.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9761000000000001e-05 2023-07-02T14:03:43.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.404072+0000", 2023-07-02T14:03:43.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014352986 2023-07-02T14:03:43.750 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.750 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.750 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.751 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:03:43.751 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:03:43.751 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.8429832080000002, 2023-07-02T14:03:43.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:03:43.751 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.751 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.752 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:03:43.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:03:43.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:43.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:03:43.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:03:43.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:03:43.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:03:43.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:03:43.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:03:43.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:03:43.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:03:43.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:03:43.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:03:43.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:03:43.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:03:43.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:03:43.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:03:43.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:03:43.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:03:43.759 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.759 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.759 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.760 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:03:43.760 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:03:43.760 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.8388974899999999, 2023-07-02T14:03:43.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:03:43.760 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.761 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.761 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:03:43.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:03:43.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:43.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:03:43.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:03:43.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:03:43.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:03:43.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:03:43.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:03:43.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:03:43.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:03:43.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:03:43.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:03:43.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:03:43.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:03:43.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:03:43.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:03:43.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:03:43.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:03:43.767 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.767 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.767 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.768 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:03:43.768 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:03:43.768 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.835032709, 2023-07-02T14:03:43.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:03:43.768 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.769 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.769 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:03:43.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:03:43.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:43.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:03:43.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:03:43.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:03:43.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:03:43.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:03:43.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:03:43.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:03:43.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:03:43.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:03:43.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:03:43.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:03:43.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:03:43.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:03:43.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:03:43.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:03:43.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:03:43.775 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.775 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.775 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.776 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:03:43.776 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:03:43.776 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.8315084480000001, 2023-07-02T14:03:43.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:03:43.776 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.776 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.777 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:03:43.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:03:43.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:43.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:03:43.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:03:43.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:03:43.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:03:43.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:03:43.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:03:43.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:03:43.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:03:43.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:03:43.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:03:43.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:03:43.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:03:43.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:03:43.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:03:43.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:03:43.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:03:43.783 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.783 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.783 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.784 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6124 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1328, mlcod=81'1319)", 2023-07-02T14:03:43.784 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:03:43.784 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.8279930649999998, 2023-07-02T14:03:43.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019096974999999999, 2023-07-02T14:03:43.784 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.784 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.784 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:03:43.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:03:43.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:43.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912602+0000", 2023-07-02T14:03:43.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6741999999999999e-05 2023-07-02T14:03:43.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912629+0000", 2023-07-02T14:03:43.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5163e-05 2023-07-02T14:03:43.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912644+0000", 2023-07-02T14:03:43.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4800000000000001e-06 2023-07-02T14:03:43.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912653+0000", 2023-07-02T14:03:43.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071988909999999998 2023-07-02T14:03:43.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.919851+0000", 2023-07-02T14:03:43.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2949999999999999e-05 2023-07-02T14:03:43.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.919874+0000", 2023-07-02T14:03:43.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011800804 2023-07-02T14:03:43.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931675+0000", 2023-07-02T14:03:43.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6033999999999998e-05 2023-07-02T14:03:43.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931701+0000", 2023-07-02T14:03:43.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019096974999999999 2023-07-02T14:03:43.791 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.791 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.791 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.791 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6125 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1330, mlcod=81'1319)", 2023-07-02T14:03:43.791 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.915930+0000", 2023-07-02T14:03:43.792 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.8246673800000002, 2023-07-02T14:03:43.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015810346999999999, 2023-07-02T14:03:43.792 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.792 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.792 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.915930+0000", 2023-07-02T14:03:43.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.915930+0000", 2023-07-02T14:03:43.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.915929+0000", 2023-07-02T14:03:43.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8398e-05 2023-07-02T14:03:43.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.915978+0000", 2023-07-02T14:03:43.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2022e-05 2023-07-02T14:03:43.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.916000+0000", 2023-07-02T14:03:43.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2780000000000001e-06 2023-07-02T14:03:43.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.916005+0000", 2023-07-02T14:03:43.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078621239999999998 2023-07-02T14:03:43.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923867+0000", 2023-07-02T14:03:43.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1858999999999999e-05 2023-07-02T14:03:43.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923889+0000", 2023-07-02T14:03:43.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078253369999999999 2023-07-02T14:03:43.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931714+0000", 2023-07-02T14:03:43.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5936999999999999e-05 2023-07-02T14:03:43.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931740+0000", 2023-07-02T14:03:43.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015810346999999999 2023-07-02T14:03:43.798 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.799 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.799 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.799 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6137 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1335, mlcod=81'1333)", 2023-07-02T14:03:43.799 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.934629+0000", 2023-07-02T14:03:43.799 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.8059682389999998, 2023-07-02T14:03:43.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014672614, 2023-07-02T14:03:43.800 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.800 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.800 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.934629+0000", 2023-07-02T14:03:43.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.934629+0000", 2023-07-02T14:03:43.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.934628+0000", 2023-07-02T14:03:43.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030856529999999998 2023-07-02T14:03:43.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.937714+0000", 2023-07-02T14:03:43.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6008999999999999e-05 2023-07-02T14:03:43.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.937730+0000", 2023-07-02T14:03:43.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9440000000000005e-06 2023-07-02T14:03:43.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.937738+0000", 2023-07-02T14:03:43.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4077000000000003e-05 2023-07-02T14:03:43.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.937802+0000", 2023-07-02T14:03:43.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5251e-05 2023-07-02T14:03:43.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.937817+0000", 2023-07-02T14:03:43.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011457106 2023-07-02T14:03:43.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949275+0000", 2023-07-02T14:03:43.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.7217999999999999e-05 2023-07-02T14:03:43.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949302+0000", 2023-07-02T14:03:43.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014672614 2023-07-02T14:03:43.806 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.806 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.806 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.807 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6139 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1337, mlcod=81'1333)", 2023-07-02T14:03:43.807 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:03:43.807 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.801303205, 2023-07-02T14:03:43.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018571681, 2023-07-02T14:03:43.807 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.808 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.808 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:03:43.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:03:43.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:43.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939293+0000", 2023-07-02T14:03:43.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003764613 2023-07-02T14:03:43.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943058+0000", 2023-07-02T14:03:43.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5797000000000001e-05 2023-07-02T14:03:43.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943074+0000", 2023-07-02T14:03:43.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5089999999999996e-06 2023-07-02T14:03:43.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943081+0000", 2023-07-02T14:03:43.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014727100000000001 2023-07-02T14:03:43.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943228+0000", 2023-07-02T14:03:43.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2246000000000003e-05 2023-07-02T14:03:43.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943261+0000", 2023-07-02T14:03:43.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014576317 2023-07-02T14:03:43.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957837+0000", 2023-07-02T14:03:43.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8962000000000002e-05 2023-07-02T14:03:43.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957866+0000", 2023-07-02T14:03:43.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018571681 2023-07-02T14:03:43.814 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.814 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.814 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.815 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6140 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1338, mlcod=81'1333)", 2023-07-02T14:03:43.815 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.943097+0000", 2023-07-02T14:03:43.815 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.7975004010000002, 2023-07-02T14:03:43.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014787599, 2023-07-02T14:03:43.815 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.815 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.815 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943097+0000", 2023-07-02T14:03:43.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943097+0000", 2023-07-02T14:03:43.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:43.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943096+0000", 2023-07-02T14:03:43.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4756999999999999e-05 2023-07-02T14:03:43.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943111+0000", 2023-07-02T14:03:43.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9079999999999995e-06 2023-07-02T14:03:43.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943119+0000", 2023-07-02T14:03:43.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4150000000000001e-06 2023-07-02T14:03:43.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943123+0000", 2023-07-02T14:03:43.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005177999 2023-07-02T14:03:43.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.948301+0000", 2023-07-02T14:03:43.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5087000000000003e-05 2023-07-02T14:03:43.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.948356+0000", 2023-07-02T14:03:43.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0095146889999999998 2023-07-02T14:03:43.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957871+0000", 2023-07-02T14:03:43.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.347e-05 2023-07-02T14:03:43.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957885+0000", 2023-07-02T14:03:43.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014787599 2023-07-02T14:03:43.822 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.822 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.822 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.822 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6141 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1339, mlcod=81'1333)", 2023-07-02T14:03:43.823 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.943978+0000", 2023-07-02T14:03:43.823 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.7966193480000001, 2023-07-02T14:03:43.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013921674, 2023-07-02T14:03:43.823 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.823 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.823 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943978+0000", 2023-07-02T14:03:43.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943978+0000", 2023-07-02T14:03:43.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943977+0000", 2023-07-02T14:03:43.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0682999999999999e-05 2023-07-02T14:03:43.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943988+0000", 2023-07-02T14:03:43.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0648999999999999e-05 2023-07-02T14:03:43.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943999+0000", 2023-07-02T14:03:43.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3510000000000001e-06 2023-07-02T14:03:43.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944004+0000", 2023-07-02T14:03:43.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053320290000000003 2023-07-02T14:03:43.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949336+0000", 2023-07-02T14:03:43.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5068e-05 2023-07-02T14:03:43.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949351+0000", 2023-07-02T14:03:43.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0085357869999999995 2023-07-02T14:03:43.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957887+0000", 2023-07-02T14:03:43.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2781e-05 2023-07-02T14:03:43.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957900+0000", 2023-07-02T14:03:43.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013921674 2023-07-02T14:03:43.830 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.830 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.830 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.830 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:03:43.830 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:03:43.830 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.7958036489999998, 2023-07-02T14:03:43.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:03:43.831 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.831 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.831 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:03:43.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:03:43.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:03:43.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:03:43.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:03:43.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:03:43.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:03:43.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:03:43.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:03:43.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:03:43.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:03:43.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:03:43.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:03:43.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:03:43.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:03:43.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:03:43.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:03:43.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:03:43.837 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.837 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.838 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.838 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:03:43.838 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:03:43.838 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.791146291, 2023-07-02T14:03:43.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:03:43.838 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.839 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.839 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:03:43.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:03:43.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:43.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:03:43.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:03:43.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:03:43.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:03:43.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:03:43.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:03:43.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:03:43.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:03:43.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:03:43.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:03:43.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:03:43.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:03:43.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:03:43.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:03:43.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:03:43.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:03:43.845 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.845 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.845 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.846 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:03:43.846 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:03:43.846 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.7902064279999998, 2023-07-02T14:03:43.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:03:43.846 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.846 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.847 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:03:43.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:03:43.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:43.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:03:43.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:03:43.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:03:43.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:03:43.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:03:43.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:03:43.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:03:43.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:03:43.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:03:43.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:03:43.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:03:43.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:03:43.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:03:43.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:03:43.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:03:43.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:03:43.853 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.853 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.853 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.854 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:03:43.854 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:03:43.854 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.7888085500000002, 2023-07-02T14:03:43.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:03:43.854 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.854 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.854 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:03:43.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:03:43.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:43.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:03:43.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:03:43.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:03:43.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:03:43.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:03:43.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:03:43.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:03:43.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:03:43.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:03:43.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:03:43.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:03:43.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:03:43.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:03:43.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:03:43.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:03:43.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:03:43.861 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.861 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.861 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.861 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6148 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1344, mlcod=81'1335)", 2023-07-02T14:03:43.862 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.956703+0000", 2023-07-02T14:03:43.862 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.7838942580000001, 2023-07-02T14:03:43.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017064030000000001, 2023-07-02T14:03:43.862 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.862 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.862 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956703+0000", 2023-07-02T14:03:43.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956703+0000", 2023-07-02T14:03:43.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:43.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956702+0000", 2023-07-02T14:03:43.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012505 2023-07-02T14:03:43.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956827+0000", 2023-07-02T14:03:43.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7309999999999999e-05 2023-07-02T14:03:43.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956845+0000", 2023-07-02T14:03:43.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7570000000000001e-06 2023-07-02T14:03:43.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956850+0000", 2023-07-02T14:03:43.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071775149999999998 2023-07-02T14:03:43.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964028+0000", 2023-07-02T14:03:43.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4773000000000002e-05 2023-07-02T14:03:43.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964053+0000", 2023-07-02T14:03:43.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096921820000000006 2023-07-02T14:03:43.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973745+0000", 2023-07-02T14:03:43.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2272999999999998e-05 2023-07-02T14:03:43.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973767+0000", 2023-07-02T14:03:43.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017064030000000001 2023-07-02T14:03:43.869 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.869 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.869 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.869 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6150 2.0 e81/46 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 81'1345, mlcod=81'1338)", 2023-07-02T14:03:43.869 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:03:43.869 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.780785603, 2023-07-02T14:03:43.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019172812000000001, 2023-07-02T14:03:43.870 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.870 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.870 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:03:43.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:03:43.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:43.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959810+0000", 2023-07-02T14:03:43.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048438650000000001 2023-07-02T14:03:43.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964653+0000", 2023-07-02T14:03:43.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2136e-05 2023-07-02T14:03:43.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964666+0000", 2023-07-02T14:03:43.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2169999999999999e-06 2023-07-02T14:03:43.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964672+0000", 2023-07-02T14:03:43.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00047561800000000003 2023-07-02T14:03:43.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965147+0000", 2023-07-02T14:03:43.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5891e-05 2023-07-02T14:03:43.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965173+0000", 2023-07-02T14:03:43.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013770973000000001 2023-07-02T14:03:43.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978944+0000", 2023-07-02T14:03:43.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0256999999999998e-05 2023-07-02T14:03:43.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978985+0000", 2023-07-02T14:03:43.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019172812000000001 2023-07-02T14:03:43.876 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.877 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.877 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.877 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6151 2.0 e81/46 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 81'1346, mlcod=81'1338)", 2023-07-02T14:03:43.877 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.964682+0000", 2023-07-02T14:03:43.877 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.7759158020000001, 2023-07-02T14:03:43.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014325582, 2023-07-02T14:03:43.878 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.878 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.878 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964682+0000", 2023-07-02T14:03:43.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964682+0000", 2023-07-02T14:03:43.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964681+0000", 2023-07-02T14:03:43.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5908000000000001e-05 2023-07-02T14:03:43.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964697+0000", 2023-07-02T14:03:43.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1920000000000005e-06 2023-07-02T14:03:43.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964705+0000", 2023-07-02T14:03:43.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.493e-06 2023-07-02T14:03:43.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964709+0000", 2023-07-02T14:03:43.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075784019999999997 2023-07-02T14:03:43.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972288+0000", 2023-07-02T14:03:43.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3762e-05 2023-07-02T14:03:43.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972302+0000", 2023-07-02T14:03:43.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066885219999999997 2023-07-02T14:03:43.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978990+0000", 2023-07-02T14:03:43.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7011999999999999e-05 2023-07-02T14:03:43.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.979007+0000", 2023-07-02T14:03:43.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014325582 2023-07-02T14:03:43.884 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.884 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.884 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.885 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6153 2.0 e81/46)", 2023-07-02T14:03:43.885 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.964714+0000", 2023-07-02T14:03:43.885 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.775882991, 2023-07-02T14:03:43.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014315782000000001, 2023-07-02T14:03:43.885 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.885 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.886 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964714+0000", 2023-07-02T14:03:43.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964714+0000", 2023-07-02T14:03:43.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:43.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964714+0000", 2023-07-02T14:03:43.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2535999999999999e-05 2023-07-02T14:03:43.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964737+0000", 2023-07-02T14:03:43.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.892e-06 2023-07-02T14:03:43.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964744+0000", 2023-07-02T14:03:43.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5040000000000004e-06 2023-07-02T14:03:43.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964749+0000", 2023-07-02T14:03:43.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0083469990000000008 2023-07-02T14:03:43.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973096+0000", 2023-07-02T14:03:43.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8771e-05 2023-07-02T14:03:43.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973115+0000", 2023-07-02T14:03:43.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058955850000000001 2023-07-02T14:03:43.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.979010+0000", 2023-07-02T14:03:43.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9847000000000002e-05 2023-07-02T14:03:43.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.979030+0000", 2023-07-02T14:03:43.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014315782000000001 2023-07-02T14:03:43.892 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.892 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.892 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.893 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6320 2.7 e81/63 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head v 81'852, mlcod=81'849)", 2023-07-02T14:03:43.893 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:40.070929+0000", 2023-07-02T14:03:43.893 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.6696682809999999, 2023-07-02T14:03:43.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014144276000000001, 2023-07-02T14:03:43.893 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:43.893 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:43.893 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:43.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:43.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.070929+0000", 2023-07-02T14:03:43.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:43.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:43.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.070929+0000", 2023-07-02T14:03:43.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:43.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:43.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.070928+0000", 2023-07-02T14:03:43.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0010305239999999999 2023-07-02T14:03:43.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:43.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.071959+0000", 2023-07-02T14:03:43.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8085999999999999e-05 2023-07-02T14:03:43.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:43.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.071977+0000", 2023-07-02T14:03:43.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3080000000000008e-06 2023-07-02T14:03:43.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:43.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.071985+0000", 2023-07-02T14:03:43.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7088999999999994e-05 2023-07-02T14:03:43.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:43.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.072052+0000", 2023-07-02T14:03:43.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1041000000000003e-05 2023-07-02T14:03:43.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:43.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.072083+0000", 2023-07-02T14:03:43.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012949866000000001 2023-07-02T14:03:43.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:43.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.085033+0000", 2023-07-02T14:03:43.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0262000000000002e-05 2023-07-02T14:03:43.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:43.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:43.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:43.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.085073+0000", 2023-07-02T14:03:43.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014144276000000001 2023-07-02T14:03:43.900 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.900 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.900 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.900 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:43.900 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:43.900 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:43.902 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:03:43.903 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:43.902+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:43.946 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4] out_osds: [5, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2023-07-02T14:03:43.946 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:03:43.947 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2023-07-02T14:03:43.947 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-07-02T14:03:43.947 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-07-02T14:03:43.947 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2023-07-02T14:03:44.028 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:44.027+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:44.069 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:44.070 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:44.070 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:03:44.070 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:44.080 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:03:44.104 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:44.104+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:44.110 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:44.110 INFO:teuthology.orchestra.run.smithi022.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-07-02T14:03:44.110 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:44.204 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:44.204+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:44.234 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:44.234 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:44.234 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:44.234 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:44.235 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:44.244 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:03:44.305 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:44.304+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:44.370 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:44.370 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:44.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:44.370 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:44.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.371 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:44.371 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:44.371 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 65.077372189000002, 2023-07-02T14:03:44.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:03:44.371 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:44.371 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:44.371 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:44.372 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:44.372 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:44.372 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:03:44.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.372 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:44.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:44.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:44.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:44.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:44.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:44.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:44.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:44.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:03:44.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:03:44.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:44.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:03:44.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:03:44.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:44.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:03:44.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:03:44.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:44.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:03:44.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:03:44.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:44.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:03:44.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:03:44.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:44.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:03:44.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:03:44.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:44.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:03:44.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:03:44.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:44.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:03:44.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:03:44.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:44.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:03:44.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:03:44.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:44.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:03:44.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:03:44.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:44.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:03:44.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:03:44.380 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.381 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.381 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.381 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:44.381 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:44.381 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 65.077137166, 2023-07-02T14:03:44.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:03:44.382 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:44.382 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:44.382 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:44.382 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:44.382 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:44.382 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:03:44.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.383 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:44.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:44.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:44.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:44.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:44.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:44.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:44.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:44.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:03:44.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:03:44.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:44.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:03:44.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:03:44.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:44.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:03:44.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:03:44.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:44.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:03:44.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:03:44.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:44.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:03:44.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:03:44.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:44.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:03:44.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:03:44.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:44.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:03:44.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:03:44.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:44.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:03:44.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:03:44.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:44.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:03:44.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:03:44.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:44.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:03:44.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:03:44.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:44.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:03:44.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:03:44.391 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.391 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.391 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.391 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:44.392 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:44.392 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 65.076938032000001, 2023-07-02T14:03:44.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:03:44.392 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:44.392 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:44.392 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:44.392 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:44.392 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:44.393 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:03:44.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.393 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:44.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:44.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:44.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:44.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:44.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:44.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:44.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:44.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:03:44.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:03:44.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:44.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:03:44.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:03:44.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:44.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:03:44.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:03:44.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:44.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:03:44.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:03:44.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:44.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:03:44.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:03:44.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:44.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:03:44.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:03:44.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:44.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:03:44.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:03:44.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:44.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:03:44.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:03:44.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:44.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:03:44.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:03:44.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:44.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:03:44.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:03:44.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:44.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:03:44.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:03:44.401 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.401 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.401 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.402 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:44.402 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:44.402 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 65.076572908000003, 2023-07-02T14:03:44.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:03:44.402 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:44.402 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:44.403 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:44.403 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:44.403 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:44.403 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:03:44.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.403 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:44.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:44.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:44.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:44.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.404 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:44.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:44.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:44.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:44.405 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:03:44.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:03:44.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:44.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:03:44.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:03:44.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:44.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:03:44.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:03:44.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:44.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:03:44.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:03:44.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:44.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:03:44.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:03:44.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:44.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:03:44.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:03:44.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:44.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:03:44.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:03:44.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:44.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:03:44.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:03:44.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:44.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:03:44.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:03:44.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:44.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:03:44.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:03:44.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:44.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:03:44.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:03:44.411 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.412 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.412 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.412 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:44.412 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:44.412 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 65.076491847, 2023-07-02T14:03:44.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:03:44.413 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:44.413 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:44.413 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:44.413 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:44.413 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:44.413 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:03:44.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.414 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:44.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:44.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:44.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:44.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:44.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:44.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:44.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:44.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:03:44.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:03:44.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:44.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:03:44.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:03:44.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:44.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:03:44.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:03:44.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:44.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:03:44.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:03:44.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:03:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:03:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:03:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:03:44.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:44.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:03:44.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:03:44.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:44.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:03:44.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:03:44.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:44.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:03:44.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:03:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:03:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:03:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:44.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:03:44.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:03:44.422 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.422 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.422 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.422 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:44.423 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:44.423 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 65.076400176000007, 2023-07-02T14:03:44.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:03:44.423 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:44.423 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:44.423 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:44.423 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:44.423 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:03:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:44.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:44.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:44.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:44.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:03:44.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:03:44.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:44.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:03:44.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:03:44.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:03:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:03:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:03:44.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:03:44.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:44.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:03:44.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:03:44.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:44.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:03:44.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:03:44.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:44.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:03:44.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:03:44.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:44.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:03:44.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:03:44.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:03:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:03:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:03:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:03:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:03:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:03:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.433 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:44.433 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:44.433 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 65.076060733999995, 2023-07-02T14:03:44.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:03:44.433 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:44.433 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:44.434 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:44.434 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:44.434 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:44.434 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:03:44.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.434 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:44.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:44.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:44.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:44.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:44.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:44.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:44.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:44.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:03:44.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:03:44.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:44.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:03:44.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:03:44.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:44.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:03:44.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:03:44.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:44.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:03:44.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:03:44.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:44.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:03:44.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:03:44.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:44.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:03:44.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:03:44.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:44.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:03:44.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:03:44.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:44.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:03:44.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:03:44.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:44.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:03:44.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:03:44.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:44.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:03:44.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:03:44.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:44.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:03:44.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:03:44.442 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.443 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.443 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.443 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:44.443 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:44.443 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 65.075874681000002, 2023-07-02T14:03:44.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:03:44.444 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:44.444 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:44.444 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:44.444 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:44.444 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:44.444 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:03:44.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.444 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:44.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:44.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:44.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:44.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:44.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:44.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:44.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:44.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:44.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:03:44.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:44.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:03:44.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:03:44.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:44.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:03:44.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:03:44.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:44.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:03:44.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:03:44.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:44.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:03:44.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:03:44.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:44.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:03:44.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:03:44.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:44.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:03:44.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:03:44.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:44.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:03:44.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:03:44.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:44.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:03:44.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:03:44.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:44.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:03:44.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:03:44.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:44.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:03:44.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:03:44.453 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.453 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.453 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.453 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:44.453 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:44.454 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 65.075716878999998, 2023-07-02T14:03:44.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:03:44.454 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:44.454 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:44.454 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:44.454 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:44.454 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:44.454 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:03:44.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.455 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:44.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:44.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:44.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:44.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:44.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:44.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:44.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:44.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:03:44.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:03:44.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:44.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:03:44.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:03:44.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:44.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:03:44.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:03:44.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:44.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:03:44.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:03:44.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:44.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:03:44.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:03:44.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:44.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:03:44.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:03:44.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:44.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:03:44.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:03:44.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:44.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:03:44.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:03:44.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:44.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:03:44.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:03:44.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:44.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:03:44.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:03:44.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:44.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:03:44.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:03:44.463 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.463 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.463 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.464 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:44.464 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:44.464 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 65.075525096999996, 2023-07-02T14:03:44.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:03:44.464 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:44.464 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:44.464 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:44.465 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:44.465 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:44.465 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:03:44.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.465 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:44.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:44.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:44.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:44.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:44.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:44.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:44.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:44.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:03:44.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:03:44.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:44.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:03:44.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:03:44.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:44.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:03:44.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:03:44.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:44.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:03:44.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:03:44.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:44.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:03:44.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:03:44.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:44.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:03:44.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:03:44.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:44.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:03:44.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:03:44.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:44.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:03:44.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:03:44.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:44.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:03:44.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:03:44.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:44.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:03:44.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:03:44.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:44.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:03:44.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:03:44.473 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.473 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.474 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.474 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:44.474 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:44.474 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 65.075349437, 2023-07-02T14:03:44.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:03:44.474 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:44.475 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:44.475 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:44.475 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:44.475 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:44.475 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:03:44.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.475 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:44.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:44.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:44.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:44.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:44.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:44.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:44.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:44.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:44.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:03:44.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:44.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:03:44.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:03:44.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:44.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:03:44.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:03:44.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:44.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:03:44.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:03:44.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:44.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:03:44.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:03:44.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:44.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:03:44.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:03:44.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:44.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:03:44.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:03:44.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:44.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:03:44.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:03:44.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:44.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:03:44.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:03:44.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:44.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:03:44.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:03:44.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:44.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:03:44.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:03:44.484 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.484 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.484 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.484 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:44.484 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:44.485 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 65.075255248999994, 2023-07-02T14:03:44.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:03:44.485 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:44.485 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:44.485 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:44.485 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:44.485 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:44.485 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:03:44.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.486 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:44.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:44.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:44.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:44.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:44.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:44.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:44.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:44.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:03:44.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:03:44.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:44.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:03:44.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:03:44.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:44.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:03:44.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:03:44.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:44.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:03:44.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:03:44.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:44.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:03:44.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:03:44.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:44.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:03:44.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:03:44.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:44.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:03:44.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:03:44.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:44.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:03:44.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:03:44.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:44.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:03:44.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:03:44.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:44.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:03:44.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:03:44.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:44.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:03:44.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:03:44.494 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.494 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.494 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.495 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:44.495 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:44.495 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 65.075051862999999, 2023-07-02T14:03:44.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:03:44.495 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:44.495 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:44.495 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:44.495 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:44.496 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:44.496 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:03:44.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.496 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:44.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:44.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:44.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:44.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:44.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:44.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:44.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:44.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:03:44.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:03:44.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:44.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:03:44.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:03:44.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:44.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:03:44.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:03:44.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:44.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:03:44.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:03:44.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:44.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:03:44.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:03:44.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:44.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:03:44.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:03:44.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:44.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:03:44.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:03:44.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:44.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:03:44.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:03:44.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:44.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:03:44.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:03:44.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:44.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:03:44.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:03:44.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:44.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:03:44.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:03:44.504 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.504 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.505 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.505 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:44.505 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:44.505 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 65.074901206999996, 2023-07-02T14:03:44.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:03:44.505 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:44.506 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:44.506 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:44.506 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:44.506 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:44.506 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:03:44.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.506 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:44.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:44.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:44.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:44.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:44.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:44.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:44.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:44.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:03:44.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:03:44.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:44.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:03:44.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:03:44.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:44.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:03:44.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:03:44.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:44.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:03:44.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:03:44.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:44.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:03:44.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:03:44.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:44.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:03:44.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:03:44.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:44.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:03:44.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:03:44.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:44.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:03:44.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:03:44.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:44.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:03:44.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:03:44.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:44.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:03:44.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:03:44.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:44.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:03:44.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:03:44.515 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.515 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.515 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.515 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:44.515 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:44.516 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 65.074828503999996, 2023-07-02T14:03:44.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:03:44.516 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:44.516 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:44.516 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:44.516 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:44.516 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:44.517 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:03:44.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.517 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:44.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:44.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:44.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:44.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:44.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:44.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:44.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:44.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:44.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:03:44.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:44.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:03:44.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:03:44.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:44.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:03:44.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:03:44.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:44.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:03:44.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:03:44.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:44.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:03:44.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:03:44.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:44.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:03:44.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:03:44.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:44.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:03:44.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:03:44.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:44.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:03:44.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:03:44.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:44.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:03:44.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:03:44.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:44.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:03:44.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:03:44.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:44.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:03:44.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:03:44.525 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.525 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.525 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.526 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:44.526 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:44.526 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 65.074711514000001, 2023-07-02T14:03:44.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:03:44.526 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:44.526 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:44.526 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:44.527 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:44.527 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:44.527 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:03:44.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.527 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:44.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:44.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:44.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:44.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:44.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:44.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:44.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:44.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:44.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:03:44.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:44.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:03:44.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:03:44.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:44.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:03:44.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:03:44.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:44.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:03:44.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:03:44.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:44.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:03:44.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:03:44.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:44.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:03:44.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:03:44.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:44.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:03:44.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:03:44.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:44.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:03:44.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:03:44.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:44.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:03:44.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:03:44.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:44.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:03:44.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:03:44.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:44.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:03:44.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:03:44.535 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.535 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.536 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.536 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:44.536 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:44.536 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 65.074562063000002, 2023-07-02T14:03:44.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:03:44.536 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:44.537 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:44.537 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:44.537 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:44.537 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:44.537 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:03:44.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.537 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:44.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:44.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:44.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:44.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:44.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:44.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:44.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:44.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:03:44.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:03:44.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:44.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:03:44.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:03:44.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:44.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:03:44.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:03:44.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:44.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:03:44.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:03:44.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:44.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:03:44.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:03:44.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:44.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:03:44.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:03:44.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:44.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:03:44.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:03:44.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:44.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:03:44.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:03:44.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:44.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:03:44.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:03:44.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:44.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:03:44.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:03:44.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:44.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:03:44.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:03:44.546 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.546 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.546 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.546 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:44.546 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:44.547 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 65.053086089999994, 2023-07-02T14:03:44.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:03:44.547 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:44.547 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:44.547 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:44.547 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:44.547 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:44.547 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:03:44.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.548 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:44.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:44.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:44.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:44.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:44.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:44.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:44.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:44.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:03:44.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:03:44.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:44.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:03:44.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:03:44.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:44.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:03:44.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:03:44.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:44.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:03:44.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:03:44.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:44.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:03:44.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:03:44.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:44.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:03:44.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:03:44.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:44.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:03:44.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:03:44.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:44.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:03:44.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:03:44.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:44.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:03:44.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:03:44.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:44.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:03:44.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:03:44.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:44.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:03:44.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:03:44.556 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.556 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.556 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.557 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:44.557 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:44.557 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 65.052953553999998, 2023-07-02T14:03:44.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:03:44.557 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:44.557 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:44.557 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:44.557 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:44.558 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:44.558 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:03:44.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.558 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:44.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:44.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:44.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:44.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:44.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:44.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:44.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:44.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:03:44.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:03:44.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:44.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:03:44.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:03:44.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:44.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:03:44.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:03:44.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:44.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:03:44.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:03:44.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:44.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:03:44.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:03:44.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:44.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:03:44.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:03:44.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:44.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:03:44.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:03:44.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:44.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:03:44.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:03:44.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:44.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:03:44.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:03:44.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:44.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:03:44.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:03:44.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:44.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:03:44.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:03:44.566 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.566 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.566 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.567 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:44.567 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:44.567 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 64.991616893, 2023-07-02T14:03:44.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:03:44.567 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:44.568 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:44.568 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:44.568 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:44.568 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:44.568 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:03:44.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.568 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:44.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:44.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:44.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:44.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:44.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:44.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:44.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:44.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:03:44.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:03:44.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:44.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:03:44.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:03:44.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:44.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:03:44.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:03:44.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:44.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:03:44.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:03:44.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:44.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:03:44.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:03:44.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:44.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:03:44.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:03:44.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:44.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:03:44.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:03:44.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:44.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:03:44.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:03:44.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:44.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:03:44.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:03:44.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:44.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:03:44.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:03:44.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:44.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:44.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:44.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:03:44.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:03:44.577 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.577 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.577 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.577 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:44.577 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:44.577 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:44.578 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:03:44.579 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 1 2023-07-02T14:03:44.579 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:03:44.580 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:44.578+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:44.580 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:03:44.679 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:44.679+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:44.755 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:44.756 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:44.756 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:44.756 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:44.766 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:03:44.780 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:44.779+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:44.830 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:03:44.881 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:44.880+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:44.932 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:44.933 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:44.933 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:44.933 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:44.933 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:44.943 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:03:44.982 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:44.981+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:44.990 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:45.009 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:45.027 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:45.045 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:45.063 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:45.071 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:45.071 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:45.082 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:03:45.083 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:45.083 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:03:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:03:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:03:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:03:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:03:45.084 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:03:45.085 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:45.083+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:45.184 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:45.183+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:45.207 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:45.207 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:45.219 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:03:45.285 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:45.284+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:45.383 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:45.384 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:45.386 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:45.384+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:45.393 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:03:45.450 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap11 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap12 b3fbb028-e267-4d25-a750-93ba18a2316a@snap12.diff 2023-07-02T14:03:45.487 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:45.486+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:45.508 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete...2023-07-02T14:03:45.507+0000 7f19cf0fd700 0 -- 172.21.15.163:0/801005910 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f19b4036bb0 msgr2=0x7f19b4038fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:03:45.514 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:03:45.513+0000 7f19cf0fd700 0 -- 172.21.15.163:0/801005910 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x559c1ee98e50 msgr2=0x7f19b403b750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:03:45.579 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:45.579 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:45.588 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:45.587+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:45.591 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:03:45.689 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:45.688+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:45.729 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:45.729 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:45.740 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:03:45.790 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:45.789+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:45.883 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:45.883 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:45.890 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:45.889+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:45.899 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:03:45.991 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:45.990+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:46.092 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:46.091+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:46.096 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:46.097 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:46.114 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:03:46.193 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:46.192+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:46.239 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:46.239 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:46.249 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:03:46.294 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:46.293+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:46.378 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:46.379 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:46.389 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:03:46.432 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:46.432+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:46.481 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 7% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 100% complete...done. 2023-07-02T14:03:46.496 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:46.495+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:46.501 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:03:46.501 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=13 2023-07-02T14:03:46.501 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap13 2023-07-02T14:03:46.513 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:46.513 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:46.523 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:03:46.580 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:03:46.606 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:46.605+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:46.687 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:46.687 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:46.697 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:03:46.698 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:46.696+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:46.798 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:46.797+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:46.861 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:46.862 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:46.872 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:03:46.899 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:46.898+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:46.996 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:46.996 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:46.999 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:46.999+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:47.006 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:03:47.096 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:03:47.100 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:47.100+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:47.170 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:47.170 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:47.180 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:03:47.201 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:47.200+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:47.234 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:47.251 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:47.268 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:47.286 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:47.302 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:47.301+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:47.304 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:47.322 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:47.323 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:03:47.323 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:03:47.324 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:03:47.324 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:03:47.324 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:03:47.324 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:03:47.345 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:47.345 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:47.345 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:47.345 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:47.345 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:47.355 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:03:47.403 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:47.402+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:47.424 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:03:47.444 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:03:47.450 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 4 2023-07-02T14:03:47.489 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:47.490 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:47.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:47.490 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:47.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.490 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:47.490 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:47.490 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.196940389000005, 2023-07-02T14:03:47.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:03:47.491 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:47.491 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:47.491 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:47.491 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:47.491 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:47.491 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:03:47.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.492 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:47.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:47.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:47.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:47.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:47.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:47.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:47.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:47.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:03:47.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:03:47.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:47.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:03:47.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:03:47.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:47.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:03:47.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:03:47.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:47.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:03:47.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:03:47.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:47.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:03:47.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:03:47.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:47.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:03:47.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:03:47.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:47.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:03:47.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:03:47.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:47.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:03:47.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:03:47.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:47.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:03:47.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:03:47.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:47.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:03:47.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:03:47.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:47.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:03:47.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:03:47.500 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.500 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.500 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.501 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:47.501 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:47.501 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.196705366000003, 2023-07-02T14:03:47.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:03:47.501 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:47.501 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:47.501 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:47.502 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:47.502 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:47.502 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:03:47.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.502 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:47.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:47.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:47.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:47.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:47.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:47.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:47.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:47.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:03:47.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:03:47.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:47.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:03:47.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:03:47.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:47.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:03:47.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:03:47.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:47.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:03:47.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:03:47.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:47.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:03:47.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:03:47.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:47.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:03:47.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:03:47.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:47.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:03:47.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:03:47.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:47.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:03:47.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:03:47.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:47.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:03:47.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:03:47.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:47.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:03:47.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:03:47.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:47.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:03:47.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:03:47.511 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.511 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.511 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.511 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:47.511 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:47.511 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.196506232000004, 2023-07-02T14:03:47.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:03:47.512 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:47.512 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:47.512 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:47.512 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:47.512 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:47.512 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:03:47.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.513 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:47.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:47.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:47.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:47.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:47.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:47.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:47.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:47.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:03:47.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:03:47.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:47.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:03:47.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:03:47.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:47.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:03:47.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:03:47.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:47.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:03:47.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:03:47.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:47.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:03:47.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:03:47.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:47.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:03:47.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:03:47.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:47.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:03:47.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:03:47.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:47.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:03:47.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:03:47.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:47.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:03:47.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:03:47.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:47.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:03:47.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:03:47.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:47.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:03:47.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:03:47.521 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.521 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.521 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.522 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:47.522 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:47.522 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.196141108000006, 2023-07-02T14:03:47.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:03:47.522 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:47.522 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:47.523 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:47.523 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:47.523 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:47.523 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:03:47.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.523 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:47.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:47.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:47.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:47.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:47.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:47.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:47.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:47.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:03:47.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:03:47.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:47.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:03:47.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:03:47.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:47.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:03:47.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:03:47.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:47.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:03:47.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:03:47.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:47.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:03:47.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:03:47.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:47.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:03:47.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:03:47.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:47.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:03:47.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:03:47.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:47.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:03:47.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:03:47.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:47.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:03:47.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:03:47.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:47.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:03:47.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:03:47.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:47.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:03:47.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:03:47.531 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.532 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.532 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.532 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:47.532 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:47.532 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.196060047000003, 2023-07-02T14:03:47.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:03:47.533 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:47.533 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:47.533 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:47.533 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:47.533 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:47.533 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:03:47.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.534 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:47.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:47.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:47.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:47.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:47.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:47.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:47.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:47.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:03:47.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:03:47.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:47.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:03:47.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:03:47.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:47.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:03:47.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:03:47.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:47.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:03:47.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:03:47.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:47.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:03:47.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:03:47.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:47.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:03:47.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:03:47.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:47.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:03:47.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:03:47.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:47.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:03:47.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:03:47.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:47.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:03:47.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:03:47.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:47.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:03:47.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:03:47.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:47.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:03:47.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:03:47.542 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.542 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.542 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.542 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:47.543 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:47.543 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.195968375999996, 2023-07-02T14:03:47.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:03:47.543 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:47.543 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:47.543 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:47.543 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:47.544 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:47.544 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:03:47.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.544 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:47.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:47.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:47.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:47.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:47.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:47.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:47.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:47.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:03:47.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:03:47.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:47.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:03:47.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:03:47.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:47.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:03:47.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:03:47.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:47.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:03:47.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:03:47.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:47.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:03:47.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:03:47.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:47.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:03:47.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:03:47.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:47.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:03:47.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:03:47.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:47.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:03:47.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:03:47.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:47.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:03:47.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:03:47.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:47.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:03:47.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:03:47.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:47.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:03:47.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:03:47.552 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.552 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.553 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.553 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:47.553 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:47.553 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.195628933999998, 2023-07-02T14:03:47.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:03:47.553 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:47.554 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:47.554 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:47.554 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:47.554 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:47.554 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:03:47.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.554 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:47.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:47.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:47.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:47.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:47.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:47.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:47.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:47.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:03:47.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:03:47.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:47.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:03:47.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:03:47.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:47.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:03:47.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:03:47.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:47.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:03:47.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:03:47.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:47.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:03:47.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:03:47.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:47.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:03:47.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:03:47.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:47.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:03:47.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:03:47.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:47.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:03:47.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:03:47.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:47.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:03:47.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:03:47.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:47.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:03:47.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:03:47.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:47.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:03:47.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:03:47.563 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.563 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.563 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.563 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:47.563 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:47.564 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.195442881000005, 2023-07-02T14:03:47.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:03:47.564 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:47.564 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:47.564 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:47.564 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:47.564 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:47.564 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:03:47.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.565 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:47.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:47.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:47.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:47.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:47.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:47.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:47.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:47.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:47.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:03:47.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:47.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:03:47.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:03:47.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:47.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:03:47.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:03:47.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:47.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:03:47.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:03:47.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:47.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:03:47.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:03:47.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:47.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:03:47.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:03:47.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:47.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:03:47.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:03:47.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:47.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:03:47.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:03:47.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:47.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:03:47.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:03:47.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:47.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:03:47.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:03:47.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:47.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:03:47.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:03:47.573 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.573 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.573 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.574 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:47.574 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:47.574 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.195285079000001, 2023-07-02T14:03:47.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:03:47.574 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:47.574 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:47.575 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:47.575 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:47.575 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:47.575 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:03:47.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.575 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:47.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:47.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:47.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:47.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:47.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:47.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:47.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:47.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:03:47.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:03:47.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:47.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:03:47.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:03:47.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:47.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:03:47.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:03:47.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:47.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:03:47.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:03:47.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:47.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:03:47.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:03:47.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:47.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:03:47.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:03:47.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:47.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:03:47.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:03:47.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:47.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:03:47.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:03:47.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:47.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:03:47.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:03:47.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:47.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:03:47.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:03:47.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:47.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:03:47.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:03:47.584 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.584 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.584 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.584 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:47.584 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:47.584 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.195093297, 2023-07-02T14:03:47.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:03:47.585 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:47.585 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:47.585 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:47.585 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:47.585 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:47.585 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:03:47.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.586 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:47.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:47.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:47.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:47.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:47.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:47.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:47.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:47.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:03:47.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:03:47.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:47.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:03:47.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:03:47.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:47.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:03:47.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:03:47.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:47.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:03:47.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:03:47.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:47.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:03:47.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:03:47.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:47.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:03:47.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:03:47.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:47.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:03:47.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:03:47.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:47.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:03:47.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:03:47.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:47.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:03:47.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:03:47.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:47.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:03:47.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:03:47.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:47.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:03:47.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:03:47.594 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.594 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.594 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.595 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:47.595 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:47.595 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.194917637000003, 2023-07-02T14:03:47.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:03:47.595 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:47.595 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:47.595 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:47.595 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:47.596 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:47.596 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:03:47.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.596 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:47.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:47.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:47.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:47.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:47.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:47.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:47.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:47.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:47.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:03:47.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:47.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:03:47.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:03:47.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:47.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:03:47.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:03:47.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:47.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:03:47.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:03:47.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:47.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:03:47.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:03:47.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:47.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:03:47.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:03:47.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:47.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:03:47.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:03:47.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:47.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:03:47.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:03:47.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:47.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:03:47.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:03:47.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:47.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:03:47.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:03:47.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.604 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:47.604 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:03:47.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:03:47.604 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.604 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.604 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.605 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:47.605 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:47.605 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.194823448999998, 2023-07-02T14:03:47.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:03:47.605 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:47.605 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:47.606 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:47.606 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:47.606 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:47.606 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:03:47.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.606 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:47.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:47.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:47.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:47.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:47.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:47.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:47.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:47.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:03:47.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:03:47.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:47.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:03:47.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:03:47.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:47.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:03:47.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:03:47.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:47.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:03:47.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:03:47.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:47.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:03:47.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:03:47.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:47.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:03:47.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:03:47.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:47.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:03:47.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:03:47.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:47.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:03:47.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:03:47.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:47.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:03:47.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:03:47.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:47.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:03:47.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:03:47.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:47.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:03:47.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:03:47.615 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.615 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.615 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.615 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:47.615 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:47.615 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.194620063000002, 2023-07-02T14:03:47.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:03:47.616 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:47.616 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:47.616 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:47.616 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:47.616 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:47.616 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:03:47.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.617 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:47.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:47.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:47.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:47.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:47.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:47.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:47.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:47.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:03:47.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:03:47.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:47.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:03:47.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:03:47.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:47.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:03:47.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:03:47.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:47.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:03:47.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:03:47.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:47.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:03:47.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:03:47.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:47.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:03:47.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:03:47.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:47.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:03:47.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:03:47.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:47.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:03:47.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:03:47.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:47.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:03:47.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:03:47.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:47.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:03:47.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:03:47.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:47.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:03:47.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:03:47.625 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.625 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.625 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.625 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:47.626 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:47.626 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.194469407, 2023-07-02T14:03:47.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:03:47.626 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:47.626 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:47.626 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:47.626 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:47.626 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:47.627 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:03:47.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.627 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:47.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:47.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:47.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:47.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:47.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:47.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:47.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:47.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:03:47.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:03:47.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:47.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:03:47.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:03:47.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:47.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:03:47.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:03:47.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:47.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:03:47.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:03:47.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:47.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:03:47.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:03:47.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:47.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:03:47.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:03:47.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:47.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:03:47.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:03:47.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:47.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:03:47.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:03:47.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:47.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:03:47.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:03:47.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:47.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:03:47.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:03:47.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:47.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:03:47.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:03:47.635 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.635 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.636 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.636 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:47.636 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:47.636 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.194396703999999, 2023-07-02T14:03:47.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:03:47.636 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:47.637 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:47.637 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:47.637 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:47.637 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:47.637 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:03:47.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.637 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:47.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:47.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:47.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:47.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:47.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:47.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:47.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:47.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:47.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:03:47.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:47.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:03:47.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:03:47.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:47.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:03:47.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:03:47.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:47.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:03:47.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:03:47.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:47.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:03:47.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:03:47.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:47.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:03:47.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:03:47.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:47.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:03:47.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:03:47.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:47.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:03:47.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:03:47.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:47.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:03:47.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:03:47.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:47.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:03:47.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:03:47.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:47.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:03:47.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:03:47.646 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.646 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.646 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.646 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:47.646 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:47.646 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.194279714000004, 2023-07-02T14:03:47.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:03:47.647 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:47.647 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:47.647 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:47.647 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:47.647 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:47.647 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:03:47.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.648 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:47.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:47.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:47.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:47.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:47.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:47.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:47.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:47.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:47.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:03:47.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:47.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:03:47.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:03:47.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:47.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:03:47.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:03:47.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:47.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:03:47.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:03:47.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:47.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:03:47.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:03:47.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:47.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:03:47.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:03:47.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:47.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:03:47.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:03:47.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:47.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:03:47.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:03:47.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:47.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:03:47.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:03:47.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:47.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:03:47.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:03:47.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:47.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:03:47.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:03:47.656 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.656 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.656 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.656 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:47.657 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:47.657 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.194130263000005, 2023-07-02T14:03:47.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:03:47.657 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:47.657 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:47.657 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:47.657 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:47.657 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:47.658 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:03:47.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.658 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:47.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:47.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:47.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:47.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:47.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:47.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:47.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:47.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:03:47.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:03:47.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:47.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:03:47.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:03:47.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:47.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:03:47.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:03:47.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:47.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:03:47.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:03:47.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:47.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:03:47.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:03:47.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:47.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:03:47.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:03:47.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:47.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:03:47.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:03:47.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:47.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:03:47.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:03:47.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:47.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:03:47.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:03:47.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:47.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:03:47.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:03:47.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:47.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:03:47.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:03:47.666 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.666 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.666 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.667 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:47.667 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:47.667 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.172654289999997, 2023-07-02T14:03:47.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:03:47.667 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:47.667 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:47.668 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:47.668 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:47.668 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:47.668 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:03:47.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.668 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:47.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:47.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:47.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:47.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:47.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:47.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:47.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:47.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:03:47.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:03:47.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:47.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:03:47.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:03:47.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:47.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:03:47.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:03:47.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:47.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:03:47.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:03:47.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:47.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:03:47.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:03:47.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:47.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:03:47.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:03:47.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:47.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:03:47.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:03:47.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:47.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:03:47.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:03:47.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:47.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:03:47.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:03:47.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:47.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:03:47.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:03:47.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:47.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:03:47.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:03:47.677 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.677 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.677 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.677 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:47.677 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:47.677 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.172521754000002, 2023-07-02T14:03:47.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:03:47.678 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:47.678 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:47.678 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:47.678 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:47.678 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:47.678 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:03:47.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.679 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:47.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:47.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:47.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:47.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:47.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:47.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:47.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:47.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:03:47.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:03:47.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:47.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:03:47.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:03:47.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:47.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:03:47.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:03:47.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:47.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:03:47.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:03:47.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:47.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:03:47.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:03:47.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:47.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:03:47.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:03:47.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:47.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:03:47.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:03:47.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:47.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:03:47.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:03:47.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:47.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:03:47.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:03:47.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:47.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:03:47.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:03:47.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:47.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:03:47.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:03:47.687 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.687 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.687 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.687 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:47.688 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:47.688 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.111185093000003, 2023-07-02T14:03:47.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:03:47.688 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:47.688 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:47.688 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:47.688 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:47.688 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:47.689 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:03:47.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.689 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:47.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:47.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:47.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:47.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:47.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:47.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:47.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:47.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:03:47.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:03:47.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:47.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:03:47.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:03:47.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:47.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:03:47.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:03:47.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:47.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:03:47.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:03:47.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:47.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:03:47.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:03:47.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:47.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:03:47.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:03:47.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:47.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:03:47.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:03:47.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:47.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:03:47.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:03:47.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:47.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:03:47.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:03:47.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:47.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:03:47.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:03:47.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:47.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:47.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:47.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:03:47.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:03:47.697 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.697 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.698 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.698 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:47.698 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:47.698 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:47.699 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:47.698+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:47.700 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:03:47.800 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:47.799+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:47.825 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:47.826 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:47.826 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:47.826 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:47.835 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:03:47.901 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:47.900+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:47.999 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:48.000 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:48.000 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:48.000 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:48.000 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:48.001 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:48.000+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:48.010 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:03:48.102 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:48.101+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:48.177 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:48.177 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:03:48.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:03:48.177 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2620 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 196608~786432] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:03:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.850486209000003, 2023-07-02T14:03:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966, 2023-07-02T14:03:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2620 2023-07-02T14:03:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:03:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:03:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324571+0000", 2023-07-02T14:03:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016686649999999999 2023-07-02T14:03:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326240+0000", 2023-07-02T14:03:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1844e-05 2023-07-02T14:03:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326252+0000", 2023-07-02T14:03:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7760000000000003e-06 2023-07-02T14:03:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326258+0000", 2023-07-02T14:03:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0634000000000003e-05 2023-07-02T14:03:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326318+0000", 2023-07-02T14:03:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9629999999999993e-06 2023-07-02T14:03:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326328+0000", 2023-07-02T14:03:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.010877045 2023-07-02T14:03:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337205+0000", 2023-07-02T14:03:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022401300000000001 2023-07-02T14:03:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337429+0000", 2023-07-02T14:03:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000139822 2023-07-02T14:03:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337569+0000", 2023-07-02T14:03:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032942119999999999 2023-07-02T14:03:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340863+0000", 2023-07-02T14:03:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021302560000000001 2023-07-02T14:03:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342994+0000", 2023-07-02T14:03:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5634000000000002e-05 2023-07-02T14:03:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343059+0000", 2023-07-02T14:03:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143721 2023-07-02T14:03:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343203+0000", 2023-07-02T14:03:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966 2023-07-02T14:03:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2622 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 983040~36864] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:03:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.848793698000001, 2023-07-02T14:03:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664, 2023-07-02T14:03:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2622 2023-07-02T14:03:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:03:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:03:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326265+0000", 2023-07-02T14:03:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1282000000000001e-05 2023-07-02T14:03:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326286+0000", 2023-07-02T14:03:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9480000000000002e-06 2023-07-02T14:03:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326293+0000", 2023-07-02T14:03:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3610000000000003e-06 2023-07-02T14:03:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326297+0000", 2023-07-02T14:03:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7707999999999994e-05 2023-07-02T14:03:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326385+0000", 2023-07-02T14:03:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8070000000000006e-06 2023-07-02T14:03:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326394+0000", 2023-07-02T14:03:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.014535878 2023-07-02T14:03:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340930+0000", 2023-07-02T14:03:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000154551 2023-07-02T14:03:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341084+0000", 2023-07-02T14:03:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00032375900000000002 2023-07-02T14:03:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341408+0000", 2023-07-02T14:03:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012631066 2023-07-02T14:03:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354039+0000", 2023-07-02T14:03:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5741000000000001e-05 2023-07-02T14:03:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354055+0000", 2023-07-02T14:03:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8794000000000006e-05 2023-07-02T14:03:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354124+0000", 2023-07-02T14:03:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014667700000000001 2023-07-02T14:03:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354270+0000", 2023-07-02T14:03:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664 2023-07-02T14:03:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2727 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~12288] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:03:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.848757923000001, 2023-07-02T14:03:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999, 2023-07-02T14:03:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2727 2023-07-02T14:03:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:03:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:03:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:03:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3834e-05 2023-07-02T14:03:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326315+0000", 2023-07-02T14:03:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3029999999999996e-06 2023-07-02T14:03:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326321+0000", 2023-07-02T14:03:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7590000000000002e-06 2023-07-02T14:03:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326324+0000", 2023-07-02T14:03:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112872 2023-07-02T14:03:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326437+0000", 2023-07-02T14:03:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2290000000000002e-06 2023-07-02T14:03:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326442+0000", 2023-07-02T14:03:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0155896879999999 2023-07-02T14:03:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342031+0000", 2023-07-02T14:03:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016496100000000001 2023-07-02T14:03:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342196+0000", 2023-07-02T14:03:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021998699999999999 2023-07-02T14:03:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342416+0000", 2023-07-02T14:03:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011869744999999999 2023-07-02T14:03:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354286+0000", 2023-07-02T14:03:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0361e-05 2023-07-02T14:03:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354347+0000", 2023-07-02T14:03:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3312000000000005e-05 2023-07-02T14:03:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354430+0000", 2023-07-02T14:03:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126444 2023-07-02T14:03:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354556+0000", 2023-07-02T14:03:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999 2023-07-02T14:03:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2730 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [set-alloc-hint object_size 4194304 write_size 4194304,write 454656~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:03:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.848732914000003, 2023-07-02T14:03:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999, 2023-07-02T14:03:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2730 2023-07-02T14:03:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:03:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:03:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:03:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6049999999999997e-06 2023-07-02T14:03:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326332+0000", 2023-07-02T14:03:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3000000000000002e-06 2023-07-02T14:03:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326335+0000", 2023-07-02T14:03:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.719e-05 2023-07-02T14:03:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326352+0000", 2023-07-02T14:03:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011583599999999999 2023-07-02T14:03:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326468+0000", 2023-07-02T14:03:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.831e-06 2023-07-02T14:03:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326473+0000", 2023-07-02T14:03:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.016769915 2023-07-02T14:03:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343243+0000", 2023-07-02T14:03:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197352 2023-07-02T14:03:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343440+0000", 2023-07-02T14:03:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197581 2023-07-02T14:03:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343638+0000", 2023-07-02T14:03:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010980139 2023-07-02T14:03:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354618+0000", 2023-07-02T14:03:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056779530000000003 2023-07-02T14:03:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360296+0000", 2023-07-02T14:03:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101109 2023-07-02T14:03:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360397+0000", 2023-07-02T14:03:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000160273 2023-07-02T14:03:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360557+0000", 2023-07-02T14:03:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999 2023-07-02T14:03:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2752 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 1019904~3174400] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:03:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.848686690000001, 2023-07-02T14:03:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001, 2023-07-02T14:03:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2752 2023-07-02T14:03:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:03:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:03:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326372+0000", 2023-07-02T14:03:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052050380000000004 2023-07-02T14:03:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331577+0000", 2023-07-02T14:03:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6832000000000001e-05 2023-07-02T14:03:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331594+0000", 2023-07-02T14:03:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8220000000000007e-06 2023-07-02T14:03:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331602+0000", 2023-07-02T14:03:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9895999999999999e-05 2023-07-02T14:03:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331642+0000", 2023-07-02T14:03:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:03:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331649+0000", 2023-07-02T14:03:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0139141709999999 2023-07-02T14:03:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345563+0000", 2023-07-02T14:03:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113817 2023-07-02T14:03:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345677+0000", 2023-07-02T14:03:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018803400000000001 2023-07-02T14:03:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345865+0000", 2023-07-02T14:03:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087950359999999991 2023-07-02T14:03:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354660+0000", 2023-07-02T14:03:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059116669999999998 2023-07-02T14:03:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360572+0000", 2023-07-02T14:03:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0992000000000002e-05 2023-07-02T14:03:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360633+0000", 2023-07-02T14:03:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013601600000000001 2023-07-02T14:03:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360769+0000", 2023-07-02T14:03:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001 2023-07-02T14:03:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.237 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2759 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 0~454656] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:48.237 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:03:48.237 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.843446892999999, 2023-07-02T14:03:48.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999, 2023-07-02T14:03:48.237 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.237 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.237 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.237 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.238 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.238 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2759 2023-07-02T14:03:48.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.238 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:03:48.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:03:48.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:48.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331611+0000", 2023-07-02T14:03:48.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053795999999999998 2023-07-02T14:03:48.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332149+0000", 2023-07-02T14:03:48.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4970000000000007e-06 2023-07-02T14:03:48.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332158+0000", 2023-07-02T14:03:48.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:03:48.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332162+0000", 2023-07-02T14:03:48.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3526000000000001e-05 2023-07-02T14:03:48.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332185+0000", 2023-07-02T14:03:48.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:03:48.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332189+0000", 2023-07-02T14:03:48.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0186295910000001 2023-07-02T14:03:48.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350819+0000", 2023-07-02T14:03:48.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000111123 2023-07-02T14:03:48.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350930+0000", 2023-07-02T14:03:48.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019575300000000001 2023-07-02T14:03:48.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:48.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.351126+0000", 2023-07-02T14:03:48.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011503298 2023-07-02T14:03:48.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362629+0000", 2023-07-02T14:03:48.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085443989999999994 2023-07-02T14:03:48.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371173+0000", 2023-07-02T14:03:48.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3547999999999995e-05 2023-07-02T14:03:48.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371237+0000", 2023-07-02T14:03:48.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790300000000001 2023-07-02T14:03:48.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371375+0000", 2023-07-02T14:03:48.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999 2023-07-02T14:03:48.248 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.248 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.248 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.248 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2760 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 458752~131072] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:48.248 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:03:48.248 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.842890707999999, 2023-07-02T14:03:48.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001, 2023-07-02T14:03:48.249 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.249 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.249 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.249 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.249 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.249 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2760 2023-07-02T14:03:48.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.250 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:03:48.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:03:48.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:48.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332168+0000", 2023-07-02T14:03:48.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013281100000000001 2023-07-02T14:03:48.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332301+0000", 2023-07-02T14:03:48.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7389999999999999e-06 2023-07-02T14:03:48.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332308+0000", 2023-07-02T14:03:48.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0639999999999998e-06 2023-07-02T14:03:48.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332312+0000", 2023-07-02T14:03:48.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1735000000000001e-05 2023-07-02T14:03:48.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332333+0000", 2023-07-02T14:03:48.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8342000000000002e-05 2023-07-02T14:03:48.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332352+0000", 2023-07-02T14:03:48.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02234782 2023-07-02T14:03:48.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354699+0000", 2023-07-02T14:03:48.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110371 2023-07-02T14:03:48.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354810+0000", 2023-07-02T14:03:48.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022174900000000001 2023-07-02T14:03:48.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:48.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355032+0000", 2023-07-02T14:03:48.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076053329999999997 2023-07-02T14:03:48.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362637+0000", 2023-07-02T14:03:48.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087913609999999993 2023-07-02T14:03:48.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371428+0000", 2023-07-02T14:03:48.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8532999999999998e-05 2023-07-02T14:03:48.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371477+0000", 2023-07-02T14:03:48.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010990800000000001 2023-07-02T14:03:48.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371587+0000", 2023-07-02T14:03:48.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001 2023-07-02T14:03:48.259 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.259 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.259 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.260 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2766 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 589824~45056] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:48.260 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:03:48.260 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.842742082999997, 2023-07-02T14:03:48.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188, 2023-07-02T14:03:48.260 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.261 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.261 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.261 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.261 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.261 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2766 2023-07-02T14:03:48.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.261 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:03:48.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:03:48.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:48.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:03:48.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2247999999999998e-05 2023-07-02T14:03:48.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332369+0000", 2023-07-02T14:03:48.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4189999999999996e-06 2023-07-02T14:03:48.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332376+0000", 2023-07-02T14:03:48.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1049999999999999e-06 2023-07-02T14:03:48.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332379+0000", 2023-07-02T14:03:48.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6894000000000003e-05 2023-07-02T14:03:48.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332416+0000", 2023-07-02T14:03:48.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3970000000000002e-06 2023-07-02T14:03:48.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332422+0000", 2023-07-02T14:03:48.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0231382280000001 2023-07-02T14:03:48.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355560+0000", 2023-07-02T14:03:48.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9079000000000005e-05 2023-07-02T14:03:48.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355659+0000", 2023-07-02T14:03:48.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018005000000000001 2023-07-02T14:03:48.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:48.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355839+0000", 2023-07-02T14:03:48.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068044439999999998 2023-07-02T14:03:48.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362644+0000", 2023-07-02T14:03:48.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090155589999999994 2023-07-02T14:03:48.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371659+0000", 2023-07-02T14:03:48.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1297999999999994e-05 2023-07-02T14:03:48.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371730+0000", 2023-07-02T14:03:48.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013100699999999999 2023-07-02T14:03:48.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371861+0000", 2023-07-02T14:03:48.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188 2023-07-02T14:03:48.271 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.271 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.271 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.272 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2784 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 0~393216] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:48.272 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:03:48.272 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.842673630999997, 2023-07-02T14:03:48.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999, 2023-07-02T14:03:48.272 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.272 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.272 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.272 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.273 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.273 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2784 2023-07-02T14:03:48.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.273 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:03:48.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:03:48.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:48.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332385+0000", 2023-07-02T14:03:48.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040252400000000002 2023-07-02T14:03:48.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332787+0000", 2023-07-02T14:03:48.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0208e-05 2023-07-02T14:03:48.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332798+0000", 2023-07-02T14:03:48.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2023-07-02T14:03:48.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332801+0000", 2023-07-02T14:03:48.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4838000000000001e-05 2023-07-02T14:03:48.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332836+0000", 2023-07-02T14:03:48.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5319999999999998e-06 2023-07-02T14:03:48.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332839+0000", 2023-07-02T14:03:48.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0234290109999999 2023-07-02T14:03:48.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356268+0000", 2023-07-02T14:03:48.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011165700000000001 2023-07-02T14:03:48.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356380+0000", 2023-07-02T14:03:48.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022504799999999999 2023-07-02T14:03:48.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:48.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356605+0000", 2023-07-02T14:03:48.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015276201 2023-07-02T14:03:48.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371881+0000", 2023-07-02T14:03:48.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5496999999999999e-05 2023-07-02T14:03:48.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371957+0000", 2023-07-02T14:03:48.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4570000000000002e-05 2023-07-02T14:03:48.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372001+0000", 2023-07-02T14:03:48.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3880000000000003e-05 2023-07-02T14:03:48.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372085+0000", 2023-07-02T14:03:48.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999 2023-07-02T14:03:48.283 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.283 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.283 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.283 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2785 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 405504~446464] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:48.283 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:03:48.284 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.842252180000003, 2023-07-02T14:03:48.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999, 2023-07-02T14:03:48.284 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.284 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.284 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.284 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.284 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.285 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2785 2023-07-02T14:03:48.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.285 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:03:48.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:03:48.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:48.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:03:48.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059527500000000004 2023-07-02T14:03:48.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333402+0000", 2023-07-02T14:03:48.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7059999999999993e-06 2023-07-02T14:03:48.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333411+0000", 2023-07-02T14:03:48.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0450000000000001e-06 2023-07-02T14:03:48.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333414+0000", 2023-07-02T14:03:48.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5790999999999997e-05 2023-07-02T14:03:48.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:03:48.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3320000000000002e-06 2023-07-02T14:03:48.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333454+0000", 2023-07-02T14:03:48.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.025310269 2023-07-02T14:03:48.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358764+0000", 2023-07-02T14:03:48.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012414899999999999 2023-07-02T14:03:48.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358888+0000", 2023-07-02T14:03:48.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000217028 2023-07-02T14:03:48.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:48.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.359105+0000", 2023-07-02T14:03:48.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012788212 2023-07-02T14:03:48.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371893+0000", 2023-07-02T14:03:48.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012201569999999999 2023-07-02T14:03:48.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373114+0000", 2023-07-02T14:03:48.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4512999999999995e-05 2023-07-02T14:03:48.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373198+0000", 2023-07-02T14:03:48.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132373 2023-07-02T14:03:48.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373330+0000", 2023-07-02T14:03:48.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999 2023-07-02T14:03:48.294 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.295 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.295 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.295 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2786 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1310720~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:48.295 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:03:48.295 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.841639487000002, 2023-07-02T14:03:48.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001, 2023-07-02T14:03:48.296 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.296 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.296 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.296 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.296 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.296 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2786 2023-07-02T14:03:48.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.297 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:03:48.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:03:48.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:48.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333419+0000", 2023-07-02T14:03:48.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2009999999999999e-06 2023-07-02T14:03:48.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333427+0000", 2023-07-02T14:03:48.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0210000000000001e-06 2023-07-02T14:03:48.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333431+0000", 2023-07-02T14:03:48.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3769999999999999e-06 2023-07-02T14:03:48.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333434+0000", 2023-07-02T14:03:48.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1161e-05 2023-07-02T14:03:48.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333485+0000", 2023-07-02T14:03:48.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1549000000000001e-05 2023-07-02T14:03:48.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333496+0000", 2023-07-02T14:03:48.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0273116339999999 2023-07-02T14:03:48.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360808+0000", 2023-07-02T14:03:48.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000115128 2023-07-02T14:03:48.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360923+0000", 2023-07-02T14:03:48.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018367700000000001 2023-07-02T14:03:48.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:48.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361107+0000", 2023-07-02T14:03:48.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010796923999999999 2023-07-02T14:03:48.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371904+0000", 2023-07-02T14:03:48.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014983520000000001 2023-07-02T14:03:48.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373402+0000", 2023-07-02T14:03:48.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0520999999999999e-05 2023-07-02T14:03:48.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373463+0000", 2023-07-02T14:03:48.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000129095 2023-07-02T14:03:48.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373592+0000", 2023-07-02T14:03:48.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001 2023-07-02T14:03:48.306 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.306 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.306 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.307 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2787 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1433600~8192] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:48.307 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:03:48.307 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.841622686999997, 2023-07-02T14:03:48.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999, 2023-07-02T14:03:48.307 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.307 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.308 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.308 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.308 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.308 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2787 2023-07-02T14:03:48.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.308 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:03:48.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:03:48.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:48.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333436+0000", 2023-07-02T14:03:48.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2731999999999999e-05 2023-07-02T14:03:48.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:03:48.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7440000000000001e-06 2023-07-02T14:03:48.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333456+0000", 2023-07-02T14:03:48.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8650000000000003e-06 2023-07-02T14:03:48.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333460+0000", 2023-07-02T14:03:48.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0077e-05 2023-07-02T14:03:48.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333530+0000", 2023-07-02T14:03:48.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4180000000000001e-06 2023-07-02T14:03:48.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333533+0000", 2023-07-02T14:03:48.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0284002720000001 2023-07-02T14:03:48.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361933+0000", 2023-07-02T14:03:48.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6781999999999996e-05 2023-07-02T14:03:48.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362030+0000", 2023-07-02T14:03:48.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018305400000000001 2023-07-02T14:03:48.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:48.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362213+0000", 2023-07-02T14:03:48.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097015109999999995 2023-07-02T14:03:48.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371915+0000", 2023-07-02T14:03:48.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001742295 2023-07-02T14:03:48.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373657+0000", 2023-07-02T14:03:48.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5980000000000003e-05 2023-07-02T14:03:48.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373713+0000", 2023-07-02T14:03:48.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011783800000000001 2023-07-02T14:03:48.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373831+0000", 2023-07-02T14:03:48.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999 2023-07-02T14:03:48.318 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.318 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.318 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.318 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2803 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 851968~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:48.319 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:03:48.319 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.841580835000002, 2023-07-02T14:03:48.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514, 2023-07-02T14:03:48.319 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.319 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.319 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.319 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.320 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.320 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2803 2023-07-02T14:03:48.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.320 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:03:48.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:03:48.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:48.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:03:48.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4943000000000001e-05 2023-07-02T14:03:48.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333493+0000", 2023-07-02T14:03:48.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2629999999999997e-06 2023-07-02T14:03:48.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333500+0000", 2023-07-02T14:03:48.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4159999999999997e-06 2023-07-02T14:03:48.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333505+0000", 2023-07-02T14:03:48.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4518999999999998e-05 2023-07-02T14:03:48.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333559+0000", 2023-07-02T14:03:48.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5900000000000008e-06 2023-07-02T14:03:48.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333568+0000", 2023-07-02T14:03:48.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0291061779999999 2023-07-02T14:03:48.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362674+0000", 2023-07-02T14:03:48.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110412 2023-07-02T14:03:48.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362784+0000", 2023-07-02T14:03:48.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015384599999999999 2023-07-02T14:03:48.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:48.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362938+0000", 2023-07-02T14:03:48.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091682310000000006 2023-07-02T14:03:48.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372106+0000", 2023-07-02T14:03:48.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017703230000000001 2023-07-02T14:03:48.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373877+0000", 2023-07-02T14:03:48.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6387000000000002e-05 2023-07-02T14:03:48.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373923+0000", 2023-07-02T14:03:48.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014099199999999999 2023-07-02T14:03:48.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374064+0000", 2023-07-02T14:03:48.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514 2023-07-02T14:03:48.330 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.330 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.330 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.330 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2805 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 634880~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:48.330 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:03:48.331 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.841547660000003, 2023-07-02T14:03:48.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814, 2023-07-02T14:03:48.331 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.331 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.331 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.331 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.331 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.331 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2805 2023-07-02T14:03:48.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.332 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:03:48.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:03:48.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:48.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333511+0000", 2023-07-02T14:03:48.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207092 2023-07-02T14:03:48.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333718+0000", 2023-07-02T14:03:48.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5110000000000003e-06 2023-07-02T14:03:48.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333727+0000", 2023-07-02T14:03:48.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3270000000000007e-06 2023-07-02T14:03:48.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333736+0000", 2023-07-02T14:03:48.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0319999999999999e-05 2023-07-02T14:03:48.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333756+0000", 2023-07-02T14:03:48.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1695000000000001e-05 2023-07-02T14:03:48.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:03:48.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0295750560000001 2023-07-02T14:03:48.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363343+0000", 2023-07-02T14:03:48.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3387999999999994e-05 2023-07-02T14:03:48.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363437+0000", 2023-07-02T14:03:48.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018373 2023-07-02T14:03:48.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:48.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363620+0000", 2023-07-02T14:03:48.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0084948769999999996 2023-07-02T14:03:48.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372115+0000", 2023-07-02T14:03:48.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020082170000000001 2023-07-02T14:03:48.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374123+0000", 2023-07-02T14:03:48.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8875999999999997e-05 2023-07-02T14:03:48.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374182+0000", 2023-07-02T14:03:48.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0001272 2023-07-02T14:03:48.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374309+0000", 2023-07-02T14:03:48.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814 2023-07-02T14:03:48.341 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.341 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.342 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.342 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2806 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 1359872~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:48.342 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:03:48.342 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.841317328999999, 2023-07-02T14:03:48.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234, 2023-07-02T14:03:48.342 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.343 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.343 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.343 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.343 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.343 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2806 2023-07-02T14:03:48.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.343 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:03:48.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:03:48.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:48.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333741+0000", 2023-07-02T14:03:48.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3978999999999999e-05 2023-07-02T14:03:48.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333755+0000", 2023-07-02T14:03:48.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:03:48.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333761+0000", 2023-07-02T14:03:48.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7010000000000001e-06 2023-07-02T14:03:48.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333764+0000", 2023-07-02T14:03:48.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5086000000000003e-05 2023-07-02T14:03:48.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333799+0000", 2023-07-02T14:03:48.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6389999999999992e-06 2023-07-02T14:03:48.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333806+0000", 2023-07-02T14:03:48.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.030496675 2023-07-02T14:03:48.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364303+0000", 2023-07-02T14:03:48.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011382700000000001 2023-07-02T14:03:48.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364417+0000", 2023-07-02T14:03:48.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192058 2023-07-02T14:03:48.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:48.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364609+0000", 2023-07-02T14:03:48.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075132130000000004 2023-07-02T14:03:48.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372122+0000", 2023-07-02T14:03:48.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022731750000000001 2023-07-02T14:03:48.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374395+0000", 2023-07-02T14:03:48.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5277000000000005e-05 2023-07-02T14:03:48.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374460+0000", 2023-07-02T14:03:48.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017074599999999999 2023-07-02T14:03:48.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374631+0000", 2023-07-02T14:03:48.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234 2023-07-02T14:03:48.353 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.353 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.353 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.354 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 868352~49152] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:48.354 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:03:48.354 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.841290614999998, 2023-07-02T14:03:48.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001, 2023-07-02T14:03:48.354 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.355 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.355 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.355 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.355 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.355 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:03:48.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.355 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:03:48.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:03:48.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:48.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:03:48.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6355000000000001e-05 2023-07-02T14:03:48.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333794+0000", 2023-07-02T14:03:48.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8329999999999996e-06 2023-07-02T14:03:48.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333801+0000", 2023-07-02T14:03:48.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5419999999999996e-06 2023-07-02T14:03:48.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333807+0000", 2023-07-02T14:03:48.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2023-07-02T14:03:48.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333826+0000", 2023-07-02T14:03:48.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.303e-06 2023-07-02T14:03:48.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333830+0000", 2023-07-02T14:03:48.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.031137559 2023-07-02T14:03:48.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364967+0000", 2023-07-02T14:03:48.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1390000000000004e-05 2023-07-02T14:03:48.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365059+0000", 2023-07-02T14:03:48.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017597399999999999 2023-07-02T14:03:48.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:48.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365234+0000", 2023-07-02T14:03:48.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068943950000000002 2023-07-02T14:03:48.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372129+0000", 2023-07-02T14:03:48.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026189910000000002 2023-07-02T14:03:48.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374748+0000", 2023-07-02T14:03:48.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6613999999999996e-05 2023-07-02T14:03:48.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374814+0000", 2023-07-02T14:03:48.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502300000000001 2023-07-02T14:03:48.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374950+0000", 2023-07-02T14:03:48.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001 2023-07-02T14:03:48.365 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.365 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.365 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.366 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2815 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 917504~49152] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:48.366 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:03:48.366 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.837076381999999, 2023-07-02T14:03:48.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998, 2023-07-02T14:03:48.366 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.366 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.366 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.367 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.367 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.367 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2815 2023-07-02T14:03:48.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.367 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:03:48.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:03:48.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:03:48.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337980+0000", 2023-07-02T14:03:48.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017126700000000001 2023-07-02T14:03:48.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338152+0000", 2023-07-02T14:03:48.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2938999999999999e-05 2023-07-02T14:03:48.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338164+0000", 2023-07-02T14:03:48.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4448999999999999e-05 2023-07-02T14:03:48.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:03:48.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0281812 2023-07-02T14:03:48.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366370+0000", 2023-07-02T14:03:48.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1840000000000002e-05 2023-07-02T14:03:48.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366462+0000", 2023-07-02T14:03:48.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000186112 2023-07-02T14:03:48.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:48.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366648+0000", 2023-07-02T14:03:48.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054905709999999996 2023-07-02T14:03:48.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372139+0000", 2023-07-02T14:03:48.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044954790000000001 2023-07-02T14:03:48.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376634+0000", 2023-07-02T14:03:48.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4621000000000004e-05 2023-07-02T14:03:48.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376699+0000", 2023-07-02T14:03:48.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012862299999999999 2023-07-02T14:03:48.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376827+0000", 2023-07-02T14:03:48.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998 2023-07-02T14:03:48.376 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.376 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.376 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.376 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2816 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 917504~65536] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:48.376 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:03:48.377 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.836862003, 2023-07-02T14:03:48.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001, 2023-07-02T14:03:48.377 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.377 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.377 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.377 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.377 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.377 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2816 2023-07-02T14:03:48.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.378 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:03:48.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:03:48.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:48.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338196+0000", 2023-07-02T14:03:48.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6286999999999999e-05 2023-07-02T14:03:48.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338243+0000", 2023-07-02T14:03:48.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7030000000000001e-06 2023-07-02T14:03:48.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338252+0000", 2023-07-02T14:03:48.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:03:48.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338258+0000", 2023-07-02T14:03:48.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032085734999999997 2023-07-02T14:03:48.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370344+0000", 2023-07-02T14:03:48.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108621 2023-07-02T14:03:48.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370452+0000", 2023-07-02T14:03:48.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014936499999999999 2023-07-02T14:03:48.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:48.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370602+0000", 2023-07-02T14:03:48.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001778012 2023-07-02T14:03:48.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372380+0000", 2023-07-02T14:03:48.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044876380000000004 2023-07-02T14:03:48.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376867+0000", 2023-07-02T14:03:48.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9870999999999999e-05 2023-07-02T14:03:48.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376897+0000", 2023-07-02T14:03:48.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011401 2023-07-02T14:03:48.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.377011+0000", 2023-07-02T14:03:48.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001 2023-07-02T14:03:48.386 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.386 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.387 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.387 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2944 2.4 2:25711f4d:::rbd_data.125de1449605.00000000000000fa:head [write 65536~2097152] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:48.387 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.470424+0000", 2023-07-02T14:03:48.387 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.70463539, 2023-07-02T14:03:48.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029419590999999998, 2023-07-02T14:03:48.387 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.388 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.388 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.388 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.388 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.388 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2944 2023-07-02T14:03:48.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.388 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.470424+0000", 2023-07-02T14:03:48.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.470424+0000", 2023-07-02T14:03:48.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:03:48.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.470421+0000", 2023-07-02T14:03:48.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0086348350000000004 2023-07-02T14:03:48.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479056+0000", 2023-07-02T14:03:48.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1942000000000001e-05 2023-07-02T14:03:48.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479078+0000", 2023-07-02T14:03:48.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8130000000000002e-06 2023-07-02T14:03:48.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479087+0000", 2023-07-02T14:03:48.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4845999999999995e-05 2023-07-02T14:03:48.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479152+0000", 2023-07-02T14:03:48.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013724700000000001 2023-07-02T14:03:48.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479289+0000", 2023-07-02T14:03:48.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000234665 2023-07-02T14:03:48.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:48.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479524+0000", 2023-07-02T14:03:48.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081727950000000001 2023-07-02T14:03:48.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.487697+0000", 2023-07-02T14:03:48.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011956079999999999 2023-07-02T14:03:48.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.499653+0000", 2023-07-02T14:03:48.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8978999999999999e-05 2023-07-02T14:03:48.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.499712+0000", 2023-07-02T14:03:48.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013181299999999999 2023-07-02T14:03:48.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.499844+0000", 2023-07-02T14:03:48.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029419590999999998 2023-07-02T14:03:48.397 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.397 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.397 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.397 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:3878 2.1 e80/79 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head v 80'436, mlcod=80'429)", 2023-07-02T14:03:48.397 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:03:48.398 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.067608505999999, 2023-07-02T14:03:48.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037799472000000001, 2023-07-02T14:03:48.398 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.398 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.398 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:03:48.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:03:48.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:48.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107450+0000", 2023-07-02T14:03:48.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023914520000000001 2023-07-02T14:03:48.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109841+0000", 2023-07-02T14:03:48.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0823e-05 2023-07-02T14:03:48.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109862+0000", 2023-07-02T14:03:48.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2023-07-02T14:03:48.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109874+0000", 2023-07-02T14:03:48.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075009600000000001 2023-07-02T14:03:48.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.110624+0000", 2023-07-02T14:03:48.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5176000000000001e-05 2023-07-02T14:03:48.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.110649+0000", 2023-07-02T14:03:48.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034557310000000001 2023-07-02T14:03:48.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145206+0000", 2023-07-02T14:03:48.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4110999999999998e-05 2023-07-02T14:03:48.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145250+0000", 2023-07-02T14:03:48.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037799472000000001 2023-07-02T14:03:48.404 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.405 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.405 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.405 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.405 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.405 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:48.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:03:48.407 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:48.405+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:48.508 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:48.507+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:48.573 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:48.573 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:48.573 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:48.573 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:48.583 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:03:48.626 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:48.625+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:48.707 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:48.707 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:48.707 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:48.707 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:48.708 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:48.710 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:48.709+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:48.717 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:03:48.810 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:48.809+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:48.883 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:48.883 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:03:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:03:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:03:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 39.429240325999999, 2023-07-02T14:03:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:03:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:03:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:03:48.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:03:48.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:03:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:03:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:03:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:03:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:03:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:03:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:03:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:03:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:03:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:03:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:03:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:03:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:03:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:03:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:03:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:03:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.558547318000002, 2023-07-02T14:03:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:03:48.896 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.896 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:03:48.896 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.896 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:03:48.896 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:03:48.896 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:03:48.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.897 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:03:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:03:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:03:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:03:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:03:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:03:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:03:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:03:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:03:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:03:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:03:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:03:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:03:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:03:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:03:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:03:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:03:48.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:03:48.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:03:48.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:03:48.904 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:03:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.558422597000003, 2023-07-02T14:03:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:03:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:03:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:03:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:03:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:03:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:03:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:03:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:03:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:03:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:03:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:03:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:03:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:03:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:03:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:03:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:03:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:03:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:03:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:03:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:03:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:03:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.551572712999999, 2023-07-02T14:03:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:03:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:03:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:03:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:03:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:03:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:03:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:03:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:03:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:03:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:03:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:03:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:03:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:03:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:03:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:03:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:03:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:03:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:03:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:03:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:03:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:03:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:03:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.551509787999997, 2023-07-02T14:03:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:03:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:03:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:03:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:03:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:03:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:03:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:03:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:03:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:03:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:03:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:03:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:03:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:03:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:03:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:03:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:03:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:03:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:03:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:03:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:03:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:03:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.551209366999998, 2023-07-02T14:03:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:03:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:03:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:03:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:03:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:03:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:03:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:03:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:03:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:03:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:03:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:03:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:03:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:03:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:03:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:03:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:03:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:03:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:03:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:03:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:03:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:03:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:03:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:03:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:03:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:03:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:03:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:03:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:03:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:03:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:03:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:03:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:03:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:03:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:03:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.550684867000001, 2023-07-02T14:03:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:03:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:03:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:03:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:03:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:03:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:03:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:03:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:03:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:03:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:03:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:03:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:03:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:03:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:03:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:03:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:03:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:03:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:03:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:03:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:03:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:03:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:03:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.426665742000001, 2023-07-02T14:03:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:03:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:03:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:03:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:03:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:03:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:03:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:03:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:03:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:03:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:03:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:03:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:03:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:03:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:03:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:03:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:03:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:03:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:03:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:03:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:03:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.961 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.961 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.961 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:48.961 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:03:48.961 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.101183038, 2023-07-02T14:03:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:03:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:03:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:03:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:03:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:03:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:03:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:03:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:03:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:03:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:03:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:03:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:03:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:03:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:03:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:03:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:03:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:03:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:03:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:03:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:03:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:03:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:03:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:03:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:03:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:03:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:03:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:03:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:03:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:03:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:03:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3887 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:03:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.097319849, 2023-07-02T14:03:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998, 2023-07-02T14:03:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3887 2023-07-02T14:03:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:03:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:03:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:48.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784019+0000", 2023-07-02T14:03:48.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4073999999999998e-05 2023-07-02T14:03:48.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:48.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:48.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784063+0000", 2023-07-02T14:03:48.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6630000000000004e-06 2023-07-02T14:03:48.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:48.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:49.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784070+0000", 2023-07-02T14:03:49.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.286e-06 2023-07-02T14:03:49.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:49.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784073+0000", 2023-07-02T14:03:49.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4941999999999998e-05 2023-07-02T14:03:49.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784118+0000", 2023-07-02T14:03:49.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8120000000000008e-06 2023-07-02T14:03:49.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:49.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784126+0000", 2023-07-02T14:03:49.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32999709399999999 2023-07-02T14:03:49.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114123+0000", 2023-07-02T14:03:49.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000157257 2023-07-02T14:03:49.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:49.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114280+0000", 2023-07-02T14:03:49.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000341152 2023-07-02T14:03:49.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:49.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114622+0000", 2023-07-02T14:03:49.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019539766 2023-07-02T14:03:49.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:49.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.134161+0000", 2023-07-02T14:03:49.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015820796000000002 2023-07-02T14:03:49.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:49.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149982+0000", 2023-07-02T14:03:49.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.05e-05 2023-07-02T14:03:49.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:49.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150043+0000", 2023-07-02T14:03:49.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6453999999999999e-05 2023-07-02T14:03:49.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:49.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150089+0000", 2023-07-02T14:03:49.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998 2023-07-02T14:03:49.032 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.032 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:49.032 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.033 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3939 2.1 2:9ef34a83:::rbd_data.125de1449605.00000000000000b8:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:49.033 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:03:49.033 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.097262341, 2023-07-02T14:03:49.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001, 2023-07-02T14:03:49.033 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:49.033 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:49.033 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:49.033 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:49.034 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:49.034 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3939 2023-07-02T14:03:49.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.034 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:49.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:49.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:03:49.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:49.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:49.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:03:49.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:49.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:49.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:03:49.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046852830000000002 2023-07-02T14:03:49.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:49.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788763+0000", 2023-07-02T14:03:49.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3185e-05 2023-07-02T14:03:49.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:49.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788776+0000", 2023-07-02T14:03:49.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8470000000000003e-06 2023-07-02T14:03:49.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:49.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788781+0000", 2023-07-02T14:03:49.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7104000000000002e-05 2023-07-02T14:03:49.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788828+0000", 2023-07-02T14:03:49.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3516000000000001e-05 2023-07-02T14:03:49.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:49.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788841+0000", 2023-07-02T14:03:49.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33064513600000001 2023-07-02T14:03:49.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119487+0000", 2023-07-02T14:03:49.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9876999999999997e-05 2023-07-02T14:03:49.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:49.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119537+0000", 2023-07-02T14:03:49.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014160999999999999 2023-07-02T14:03:49.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:49.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119678+0000", 2023-07-02T14:03:49.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023054392 2023-07-02T14:03:49.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:49.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.142733+0000", 2023-07-02T14:03:49.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074065470000000003 2023-07-02T14:03:49.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:49.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150139+0000", 2023-07-02T14:03:49.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5025999999999997e-05 2023-07-02T14:03:49.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:49.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150204+0000", 2023-07-02T14:03:49.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147999999999997e-05 2023-07-02T14:03:49.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:49.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150241+0000", 2023-07-02T14:03:49.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001 2023-07-02T14:03:49.044 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.044 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:49.044 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.044 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3940 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:49.044 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:03:49.044 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.092551308999999, 2023-07-02T14:03:49.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001, 2023-07-02T14:03:49.045 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:49.045 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:49.045 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:49.045 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:49.045 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:49.045 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3940 2023-07-02T14:03:49.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.046 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:49.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:49.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:03:49.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:49.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:49.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:03:49.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:49.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:49.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788788+0000", 2023-07-02T14:03:49.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043166560000000003 2023-07-02T14:03:49.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:49.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793105+0000", 2023-07-02T14:03:49.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5970000000000001e-05 2023-07-02T14:03:49.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:49.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793121+0000", 2023-07-02T14:03:49.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0939999999999998e-06 2023-07-02T14:03:49.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:49.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793126+0000", 2023-07-02T14:03:49.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2438000000000002e-05 2023-07-02T14:03:49.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793178+0000", 2023-07-02T14:03:49.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2019e-05 2023-07-02T14:03:49.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:49.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793190+0000", 2023-07-02T14:03:49.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33208757799999999 2023-07-02T14:03:49.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125278+0000", 2023-07-02T14:03:49.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5255999999999997e-05 2023-07-02T14:03:49.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:49.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125373+0000", 2023-07-02T14:03:49.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015674999999999999 2023-07-02T14:03:49.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:49.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125530+0000", 2023-07-02T14:03:49.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018767308 2023-07-02T14:03:49.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:49.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144297+0000", 2023-07-02T14:03:49.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060291959999999997 2023-07-02T14:03:49.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:49.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150326+0000", 2023-07-02T14:03:49.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3886999999999993e-05 2023-07-02T14:03:49.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:49.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150390+0000", 2023-07-02T14:03:49.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010255 2023-07-02T14:03:49.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:49.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150493+0000", 2023-07-02T14:03:49.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001 2023-07-02T14:03:49.055 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.055 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:49.055 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.056 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3947 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:49.056 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:03:49.056 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.088205781999999, 2023-07-02T14:03:49.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999, 2023-07-02T14:03:49.056 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:49.056 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:49.056 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:49.057 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:49.057 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:49.057 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3947 2023-07-02T14:03:49.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.057 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:49.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:49.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:03:49.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:49.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:49.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:03:49.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:49.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:49.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793133+0000", 2023-07-02T14:03:49.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2519999999999999e-05 2023-07-02T14:03:49.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:49.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793176+0000", 2023-07-02T14:03:49.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:03:49.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:49.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793180+0000", 2023-07-02T14:03:49.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1709999999999998e-06 2023-07-02T14:03:49.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:49.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793184+0000", 2023-07-02T14:03:49.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1721999999999999e-05 2023-07-02T14:03:49.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793215+0000", 2023-07-02T14:03:49.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1280000000000008e-06 2023-07-02T14:03:49.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:49.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793224+0000", 2023-07-02T14:03:49.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34218549599999998 2023-07-02T14:03:49.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135409+0000", 2023-07-02T14:03:49.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117442 2023-07-02T14:03:49.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:49.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135526+0000", 2023-07-02T14:03:49.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000206257 2023-07-02T14:03:49.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:49.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135733+0000", 2023-07-02T14:03:49.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085700029999999996 2023-07-02T14:03:49.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:49.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144303+0000", 2023-07-02T14:03:49.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006278652 2023-07-02T14:03:49.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:49.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150581+0000", 2023-07-02T14:03:49.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4765999999999996e-05 2023-07-02T14:03:49.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:49.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150646+0000", 2023-07-02T14:03:49.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4777999999999995e-05 2023-07-02T14:03:49.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:49.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150711+0000", 2023-07-02T14:03:49.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999 2023-07-02T14:03:49.067 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.067 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:49.067 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.067 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3980 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 65536~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:49.067 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:03:49.067 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.0881519, 2023-07-02T14:03:49.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999, 2023-07-02T14:03:49.068 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:49.068 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:49.068 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:49.068 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:49.068 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:49.068 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3980 2023-07-02T14:03:49.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.069 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:49.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:49.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:03:49.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:49.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:49.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:03:49.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:49.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:49.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:03:49.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020244260000000002 2023-07-02T14:03:49.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:49.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795212+0000", 2023-07-02T14:03:49.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0005e-05 2023-07-02T14:03:49.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:49.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795222+0000", 2023-07-02T14:03:49.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9860000000000002e-06 2023-07-02T14:03:49.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:49.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795227+0000", 2023-07-02T14:03:49.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6369999999999999e-05 2023-07-02T14:03:49.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795264+0000", 2023-07-02T14:03:49.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1052e-05 2023-07-02T14:03:49.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:49.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795275+0000", 2023-07-02T14:03:49.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.343850459 2023-07-02T14:03:49.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139125+0000", 2023-07-02T14:03:49.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6760000000000004e-05 2023-07-02T14:03:49.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:49.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139202+0000", 2023-07-02T14:03:49.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015092799999999999 2023-07-02T14:03:49.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:49.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139353+0000", 2023-07-02T14:03:49.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011425940000000001 2023-07-02T14:03:49.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:49.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150779+0000", 2023-07-02T14:03:49.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033223060000000001 2023-07-02T14:03:49.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:49.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154101+0000", 2023-07-02T14:03:49.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9268999999999998e-05 2023-07-02T14:03:49.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:49.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154161+0000", 2023-07-02T14:03:49.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6499999999999999e-05 2023-07-02T14:03:49.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:49.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154207+0000", 2023-07-02T14:03:49.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999 2023-07-02T14:03:49.078 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.078 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:49.078 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.079 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3982 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1835008~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:49.079 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:03:49.079 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.086106613, 2023-07-02T14:03:49.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002, 2023-07-02T14:03:49.079 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:49.079 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:49.079 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:49.080 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:49.080 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:49.080 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3982 2023-07-02T14:03:49.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.080 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:49.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:49.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:03:49.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:49.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:49.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:03:49.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:49.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:49.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795233+0000", 2023-07-02T14:03:49.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8680999999999998e-05 2023-07-02T14:03:49.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:49.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795272+0000", 2023-07-02T14:03:49.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5629999999999999e-06 2023-07-02T14:03:49.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:49.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795277+0000", 2023-07-02T14:03:49.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8700000000000001e-06 2023-07-02T14:03:49.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:49.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795280+0000", 2023-07-02T14:03:49.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1964000000000001e-05 2023-07-02T14:03:49.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795312+0000", 2023-07-02T14:03:49.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.314e-06 2023-07-02T14:03:49.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:49.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795319+0000", 2023-07-02T14:03:49.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34828442599999998 2023-07-02T14:03:49.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143603+0000", 2023-07-02T14:03:49.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2438000000000001e-05 2023-07-02T14:03:49.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:49.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143665+0000", 2023-07-02T14:03:49.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3079999999999997e-05 2023-07-02T14:03:49.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:49.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143759+0000", 2023-07-02T14:03:49.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070633079999999999 2023-07-02T14:03:49.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:49.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150822+0000", 2023-07-02T14:03:49.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034430210000000001 2023-07-02T14:03:49.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:49.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154265+0000", 2023-07-02T14:03:49.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4249999999999998e-05 2023-07-02T14:03:49.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:49.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154309+0000", 2023-07-02T14:03:49.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2020000000000001e-05 2023-07-02T14:03:49.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:49.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154351+0000", 2023-07-02T14:03:49.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002 2023-07-02T14:03:49.090 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.090 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:49.090 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.090 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4054 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1900544~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:49.090 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:03:49.090 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.086054942000001, 2023-07-02T14:03:49.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002, 2023-07-02T14:03:49.091 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:49.091 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:49.091 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:49.091 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:49.091 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:49.091 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4054 2023-07-02T14:03:49.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.092 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:49.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:49.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:03:49.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:49.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:49.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:03:49.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:49.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:49.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:03:49.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018874860000000001 2023-07-02T14:03:49.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:49.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797172+0000", 2023-07-02T14:03:49.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3000000000000007e-06 2023-07-02T14:03:49.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:49.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797182+0000", 2023-07-02T14:03:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6260000000000003e-06 2023-07-02T14:03:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797186+0000", 2023-07-02T14:03:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9013e-05 2023-07-02T14:03:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797235+0000", 2023-07-02T14:03:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5990000000000008e-06 2023-07-02T14:03:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:49.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797245+0000", 2023-07-02T14:03:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34785621999999999 2023-07-02T14:03:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145101+0000", 2023-07-02T14:03:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7762000000000006e-05 2023-07-02T14:03:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145169+0000", 2023-07-02T14:03:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013469400000000001 2023-07-02T14:03:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145304+0000", 2023-07-02T14:03:49.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089538039999999992 2023-07-02T14:03:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154257+0000", 2023-07-02T14:03:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104218 2023-07-02T14:03:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154362+0000", 2023-07-02T14:03:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4168000000000001e-05 2023-07-02T14:03:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154406+0000", 2023-07-02T14:03:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3098000000000001e-05 2023-07-02T14:03:49.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154429+0000", 2023-07-02T14:03:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002 2023-07-02T14:03:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4056 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3670016~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:03:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.084148429000001, 2023-07-02T14:03:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997, 2023-07-02T14:03:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:49.102 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4056 2023-07-02T14:03:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:49.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:03:49.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:49.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:49.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:03:49.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:49.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:49.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797191+0000", 2023-07-02T14:03:49.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0438999999999997e-05 2023-07-02T14:03:49.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:49.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797232+0000", 2023-07-02T14:03:49.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6929999999999998e-06 2023-07-02T14:03:49.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:49.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797237+0000", 2023-07-02T14:03:49.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9739999999999998e-06 2023-07-02T14:03:49.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:49.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797240+0000", 2023-07-02T14:03:49.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0243000000000002e-05 2023-07-02T14:03:49.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797270+0000", 2023-07-02T14:03:49.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9889999999999997e-06 2023-07-02T14:03:49.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:49.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797276+0000", 2023-07-02T14:03:49.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35358319799999999 2023-07-02T14:03:49.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150859+0000", 2023-07-02T14:03:49.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1357e-05 2023-07-02T14:03:49.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:49.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150940+0000", 2023-07-02T14:03:49.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014903 2023-07-02T14:03:49.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:49.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151089+0000", 2023-07-02T14:03:49.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033512519999999999 2023-07-02T14:03:49.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:49.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154441+0000", 2023-07-02T14:03:49.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001050945 2023-07-02T14:03:49.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:49.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155492+0000", 2023-07-02T14:03:49.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2523e-05 2023-07-02T14:03:49.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:49.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155524+0000", 2023-07-02T14:03:49.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.057e-05 2023-07-02T14:03:49.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:49.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155555+0000", 2023-07-02T14:03:49.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997 2023-07-02T14:03:49.113 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.113 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:49.113 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.113 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4088 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3735552~458752] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:49.113 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:03:49.113 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.084096234, 2023-07-02T14:03:49.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998, 2023-07-02T14:03:49.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:49.114 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:49.114 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:49.114 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:49.114 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:49.114 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4088 2023-07-02T14:03:49.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.115 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:49.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:49.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:03:49.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:49.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:49.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:03:49.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:49.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:49.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:03:49.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00038503499999999999 2023-07-02T14:03:49.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:49.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797629+0000", 2023-07-02T14:03:49.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8739999999999995e-06 2023-07-02T14:03:49.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:49.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797637+0000", 2023-07-02T14:03:49.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7160000000000001e-06 2023-07-02T14:03:49.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:49.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797640+0000", 2023-07-02T14:03:49.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4342999999999998e-05 2023-07-02T14:03:49.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797695+0000", 2023-07-02T14:03:49.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0939999999999996e-06 2023-07-02T14:03:49.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:49.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797701+0000", 2023-07-02T14:03:49.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35408540799999999 2023-07-02T14:03:49.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151786+0000", 2023-07-02T14:03:49.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1953000000000006e-05 2023-07-02T14:03:49.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:49.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151868+0000", 2023-07-02T14:03:49.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015162800000000001 2023-07-02T14:03:49.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:49.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.152020+0000", 2023-07-02T14:03:49.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054561870000000004 2023-07-02T14:03:49.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:49.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.157476+0000", 2023-07-02T14:03:49.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059153199999999997 2023-07-02T14:03:49.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:49.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158068+0000", 2023-07-02T14:03:49.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2799e-05 2023-07-02T14:03:49.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:49.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158090+0000", 2023-07-02T14:03:49.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7235999999999999e-05 2023-07-02T14:03:49.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:49.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158108+0000", 2023-07-02T14:03:49.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998 2023-07-02T14:03:49.124 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.124 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:49.124 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.125 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4110 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:49.125 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:03:49.125 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.083694957000001, 2023-07-02T14:03:49.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864, 2023-07-02T14:03:49.125 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:49.125 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:49.125 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:49.126 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:49.126 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:49.126 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4110 2023-07-02T14:03:49.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.126 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:49.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:49.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:03:49.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:49.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:49.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:03:49.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:49.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:49.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:03:49.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9331e-05 2023-07-02T14:03:49.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:49.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797684+0000", 2023-07-02T14:03:49.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.147e-06 2023-07-02T14:03:49.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:49.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797689+0000", 2023-07-02T14:03:49.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5020000000000001e-06 2023-07-02T14:03:49.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:49.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797693+0000", 2023-07-02T14:03:49.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5254999999999999e-05 2023-07-02T14:03:49.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797748+0000", 2023-07-02T14:03:49.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5520000000000005e-06 2023-07-02T14:03:49.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:49.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797757+0000", 2023-07-02T14:03:49.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35674245700000001 2023-07-02T14:03:49.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154499+0000", 2023-07-02T14:03:49.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0090000000000001e-06 2023-07-02T14:03:49.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:49.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154506+0000", 2023-07-02T14:03:49.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864 2023-07-02T14:03:49.132 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.133 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:49.133 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.133 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:49.133 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:03:49.133 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 11.083642871, 2023-07-02T14:03:49.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998, 2023-07-02T14:03:49.134 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:49.134 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:49.134 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:49.134 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:49.134 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:49.134 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:03:49.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.134 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:49.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:49.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:03:49.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:49.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:49.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:03:49.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:49.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:49.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:03:49.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001714819 2023-07-02T14:03:49.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:49.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799412+0000", 2023-07-02T14:03:49.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7779999999999996e-06 2023-07-02T14:03:49.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:49.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799421+0000", 2023-07-02T14:03:49.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3510000000000002e-06 2023-07-02T14:03:49.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:49.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799424+0000", 2023-07-02T14:03:49.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0275999999999999e-05 2023-07-02T14:03:49.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799464+0000", 2023-07-02T14:03:49.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8389999999999996e-06 2023-07-02T14:03:49.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:49.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799473+0000", 2023-07-02T14:03:49.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35507638800000002 2023-07-02T14:03:49.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:49.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154550+0000", 2023-07-02T14:03:49.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6309999999999998e-06 2023-07-02T14:03:49.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:49.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:49.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:49.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154559+0000", 2023-07-02T14:03:49.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998 2023-07-02T14:03:49.141 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.141 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:49.141 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.141 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:49.141 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:49.141 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:49.142 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4] out_osds: [5, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2023-07-02T14:03:49.142 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:03:49.142 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2023-07-02T14:03:49.142 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-07-02T14:03:49.143 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-07-02T14:03:49.143 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2023-07-02T14:03:49.144 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:03:49.145 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:49.142+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:49.243 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:49.242+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:49.312 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-07-02T14:03:49.313 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:49.313 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:49.313 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:49.314 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:49.314 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:49.325 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:03:49.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:03:49.344 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:49.343+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:49.438 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:03:49.482 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:49.481+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:49.492 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:49.493 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:49.493 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:49.493 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:49.493 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:49.503 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:03:49.506 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:49.526 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:49.545 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:49.545 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:49.544+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:49.570 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:49.605 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:49.636 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:49.637 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:03:49.638 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:03:49.638 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:03:49.638 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:03:49.638 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:03:49.638 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:03:49.646 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:49.646+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:49.670 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:49.670 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:49.681 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:03:49.747 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:49.746+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:49.817 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:49.817 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:49.827 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:03:49.848 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:49.847+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:49.949 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:49.948+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:49.992 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:49.992 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:50.002 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:03:50.050 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:50.049+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:50.125 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:50.126 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:50.136 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:03:50.150 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:50.149+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:50.252 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:50.251+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:50.259 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:50.259 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:50.270 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:03:50.353 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:50.352+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:50.434 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:50.434 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:50.444 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:03:50.485 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:50.484+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:50.549 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:03:50.555 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:50.554+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:50.608 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:50.608 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:50.618 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:03:50.656 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:50.655+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:50.741 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:50.742 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:50.751 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:03:50.795 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:50.794+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:50.857 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:50.856+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:50.874 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:50.875 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:50.885 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:03:50.958 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:50.957+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:51.049 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:51.049 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:51.058 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:51.057+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:51.059 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:03:51.160 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:51.159+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:51.198 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:51.198 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:51.209 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:03:51.292 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:51.291+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:51.362 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:51.361+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:51.374 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:51.374 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:51.384 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:03:51.452 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap12 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap13 b3fbb028-e267-4d25-a750-93ba18a2316a@snap13.diff 2023-07-02T14:03:51.468 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:51.467+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:51.499 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 100% complete...done. 2023-07-02T14:03:51.502 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:03:51.502 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=14 2023-07-02T14:03:51.502 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap14 2023-07-02T14:03:51.549 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:51.549 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:51.560 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:03:51.563 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:51.562+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:51.649 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:03:51.693 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:51.692+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:51.727 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:51.727 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:51.738 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:03:51.765 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:51.765+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:51.788 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:51.806 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:51.823 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:51.841 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:51.859 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:51.866 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:51.865+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:51.877 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:51.878 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:03:51.878 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:03:51.879 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:03:51.879 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:03:51.879 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:03:51.879 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:03:51.904 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:51.904 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:51.904 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:51.904 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:51.905 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:51.914 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:03:51.997 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:51.996+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:52.068 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:52.067+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:52.081 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:52.081 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:03:52.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:03:52.082 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:52.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.082 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:03:52.082 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:52.082 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 42.627177873000001, 2023-07-02T14:03:52.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:03:52.082 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.083 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:52.083 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.083 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.083 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.083 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:03:52.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.083 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:52.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:03:52.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:52.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:03:52.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:03:52.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:03:52.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:03:52.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:03:52.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:03:52.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:03:52.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:03:52.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:03:52.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:03:52.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:03:52.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:03:52.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:03:52.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:03:52.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:03:52.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:03:52.090 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.090 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.090 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.090 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:03:52.091 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:03:52.091 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.756484864999997, 2023-07-02T14:03:52.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:03:52.091 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.091 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:03:52.091 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.091 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:03:52.092 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:03:52.092 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:03:52.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.092 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:03:52.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:03:52.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:52.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:03:52.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:03:52.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:03:52.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:03:52.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:03:52.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:03:52.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:03:52.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:03:52.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:03:52.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:03:52.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:03:52.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:03:52.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:03:52.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:03:52.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:03:52.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:03:52.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:03:52.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:03:52.099 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.099 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.100 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.100 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:52.100 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:03:52.100 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.756360143999999, 2023-07-02T14:03:52.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:03:52.101 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.101 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:52.101 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.101 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.101 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.101 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:03:52.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.101 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:03:52.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:03:52.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:52.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:03:52.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:03:52.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:03:52.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:03:52.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:03:52.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:03:52.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:03:52.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:03:52.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:03:52.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:03:52.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:03:52.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:03:52.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:03:52.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:03:52.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:03:52.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:03:52.108 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.108 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.108 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.108 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:52.109 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:03:52.109 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.749510260000001, 2023-07-02T14:03:52.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:03:52.109 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.109 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:52.109 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.109 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.110 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.110 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:03:52.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.110 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:03:52.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:03:52.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:03:52.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:03:52.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:03:52.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:03:52.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:03:52.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:03:52.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:03:52.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:03:52.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:03:52.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:03:52.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:03:52.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:03:52.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:03:52.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:03:52.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:03:52.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:03:52.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:03:52.116 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.117 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.117 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.117 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:52.117 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:03:52.117 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.749447334999999, 2023-07-02T14:03:52.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:03:52.118 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.118 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:52.118 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.118 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.118 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.118 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:03:52.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.118 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:03:52.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:03:52.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:52.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:03:52.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:03:52.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:03:52.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:03:52.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:03:52.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:03:52.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:03:52.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:03:52.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:03:52.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:03:52.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:03:52.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:03:52.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:03:52.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:03:52.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:03:52.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:03:52.125 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.125 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.125 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.125 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:52.125 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:03:52.126 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.749146914000001, 2023-07-02T14:03:52.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:03:52.126 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.126 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.126 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.126 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.126 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.127 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:03:52.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.127 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:03:52.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:03:52.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:52.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:03:52.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:03:52.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:03:52.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:03:52.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:03:52.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:03:52.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:03:52.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:03:52.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:03:52.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:03:52.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:03:52.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:03:52.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:03:52.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:03:52.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:03:52.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:03:52.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:03:52.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:03:52.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:03:52.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:03:52.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:03:52.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:03:52.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:03:52.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:03:52.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:03:52.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:03:52.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:03:52.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:03:52.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:03:52.143 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.143 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.143 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.144 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:03:52.144 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:52.144 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.748622414000003, 2023-07-02T14:03:52.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:03:52.148 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.148 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:03:52.148 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.148 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.148 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.149 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:03:52.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.149 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:52.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:52.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:52.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:03:52.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:03:52.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:03:52.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:03:52.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:03:52.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:03:52.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:03:52.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:03:52.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:03:52.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:03:52.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:03:52.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:03:52.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:03:52.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:03:52.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:03:52.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:03:52.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:03:52.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:03:52.156 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.156 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.156 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.157 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:03:52.157 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:03:52.157 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.624603289, 2023-07-02T14:03:52.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:03:52.157 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.157 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:52.157 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.158 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.158 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.158 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:03:52.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.158 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:03:52.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:03:52.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:52.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:03:52.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:03:52.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:03:52.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:03:52.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:03:52.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:03:52.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:03:52.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:03:52.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:03:52.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:03:52.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:03:52.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:03:52.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:03:52.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:03:52.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:03:52.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:03:52.164 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.165 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.165 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.165 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:52.165 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:03:52.165 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.299120585000001, 2023-07-02T14:03:52.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:03:52.166 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.166 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.166 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.166 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.166 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.166 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:03:52.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.166 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:03:52.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:03:52.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:52.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:03:52.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:03:52.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:03:52.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:03:52.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:03:52.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:03:52.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:03:52.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:03:52.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:03:52.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:03:52.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:03:52.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:03:52.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:03:52.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:03:52.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:03:52.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:03:52.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:52.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:03:52.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:03:52.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:03:52.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:03:52.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:03:52.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:03:52.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:03:52.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:03:52.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:03:52.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:03:52.176 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.176 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.176 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.177 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3887 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:52.177 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:03:52.177 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.295257396, 2023-07-02T14:03:52.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998, 2023-07-02T14:03:52.177 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.177 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.177 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.178 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.178 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.178 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3887 2023-07-02T14:03:52.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.178 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:03:52.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:03:52.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:52.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784019+0000", 2023-07-02T14:03:52.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4073999999999998e-05 2023-07-02T14:03:52.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784063+0000", 2023-07-02T14:03:52.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6630000000000004e-06 2023-07-02T14:03:52.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784070+0000", 2023-07-02T14:03:52.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.286e-06 2023-07-02T14:03:52.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784073+0000", 2023-07-02T14:03:52.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4941999999999998e-05 2023-07-02T14:03:52.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784118+0000", 2023-07-02T14:03:52.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8120000000000008e-06 2023-07-02T14:03:52.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784126+0000", 2023-07-02T14:03:52.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32999709399999999 2023-07-02T14:03:52.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114123+0000", 2023-07-02T14:03:52.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000157257 2023-07-02T14:03:52.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114280+0000", 2023-07-02T14:03:52.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000341152 2023-07-02T14:03:52.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:52.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114622+0000", 2023-07-02T14:03:52.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019539766 2023-07-02T14:03:52.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.134161+0000", 2023-07-02T14:03:52.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015820796000000002 2023-07-02T14:03:52.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149982+0000", 2023-07-02T14:03:52.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.05e-05 2023-07-02T14:03:52.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150043+0000", 2023-07-02T14:03:52.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6453999999999999e-05 2023-07-02T14:03:52.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150089+0000", 2023-07-02T14:03:52.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998 2023-07-02T14:03:52.188 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.188 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.188 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.188 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3939 2.1 2:9ef34a83:::rbd_data.125de1449605.00000000000000b8:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:52.188 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:03:52.189 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.295199888000001, 2023-07-02T14:03:52.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001, 2023-07-02T14:03:52.189 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.189 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.189 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.189 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.189 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.189 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3939 2023-07-02T14:03:52.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.190 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:03:52.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:03:52.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:52.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:03:52.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046852830000000002 2023-07-02T14:03:52.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788763+0000", 2023-07-02T14:03:52.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3185e-05 2023-07-02T14:03:52.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788776+0000", 2023-07-02T14:03:52.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8470000000000003e-06 2023-07-02T14:03:52.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788781+0000", 2023-07-02T14:03:52.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7104000000000002e-05 2023-07-02T14:03:52.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788828+0000", 2023-07-02T14:03:52.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3516000000000001e-05 2023-07-02T14:03:52.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788841+0000", 2023-07-02T14:03:52.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33064513600000001 2023-07-02T14:03:52.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119487+0000", 2023-07-02T14:03:52.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9876999999999997e-05 2023-07-02T14:03:52.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119537+0000", 2023-07-02T14:03:52.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014160999999999999 2023-07-02T14:03:52.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:52.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119678+0000", 2023-07-02T14:03:52.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023054392 2023-07-02T14:03:52.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.142733+0000", 2023-07-02T14:03:52.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074065470000000003 2023-07-02T14:03:52.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150139+0000", 2023-07-02T14:03:52.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5025999999999997e-05 2023-07-02T14:03:52.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150204+0000", 2023-07-02T14:03:52.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147999999999997e-05 2023-07-02T14:03:52.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150241+0000", 2023-07-02T14:03:52.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001 2023-07-02T14:03:52.199 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.199 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.200 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.200 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3940 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:52.200 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:03:52.200 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.290488856, 2023-07-02T14:03:52.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001, 2023-07-02T14:03:52.200 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.201 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.201 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.201 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.201 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.201 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3940 2023-07-02T14:03:52.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.201 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:03:52.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:03:52.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:52.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788788+0000", 2023-07-02T14:03:52.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043166560000000003 2023-07-02T14:03:52.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793105+0000", 2023-07-02T14:03:52.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5970000000000001e-05 2023-07-02T14:03:52.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793121+0000", 2023-07-02T14:03:52.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0939999999999998e-06 2023-07-02T14:03:52.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793126+0000", 2023-07-02T14:03:52.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2438000000000002e-05 2023-07-02T14:03:52.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793178+0000", 2023-07-02T14:03:52.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2019e-05 2023-07-02T14:03:52.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793190+0000", 2023-07-02T14:03:52.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33208757799999999 2023-07-02T14:03:52.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125278+0000", 2023-07-02T14:03:52.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5255999999999997e-05 2023-07-02T14:03:52.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125373+0000", 2023-07-02T14:03:52.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015674999999999999 2023-07-02T14:03:52.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:52.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125530+0000", 2023-07-02T14:03:52.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018767308 2023-07-02T14:03:52.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144297+0000", 2023-07-02T14:03:52.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060291959999999997 2023-07-02T14:03:52.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150326+0000", 2023-07-02T14:03:52.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3886999999999993e-05 2023-07-02T14:03:52.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150390+0000", 2023-07-02T14:03:52.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010255 2023-07-02T14:03:52.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150493+0000", 2023-07-02T14:03:52.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001 2023-07-02T14:03:52.211 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.211 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.211 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.211 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3947 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:52.212 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:03:52.212 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.286143329, 2023-07-02T14:03:52.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999, 2023-07-02T14:03:52.212 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.212 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.212 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.212 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.213 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.213 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3947 2023-07-02T14:03:52.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.213 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:03:52.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:03:52.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:52.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793133+0000", 2023-07-02T14:03:52.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2519999999999999e-05 2023-07-02T14:03:52.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793176+0000", 2023-07-02T14:03:52.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:03:52.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793180+0000", 2023-07-02T14:03:52.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1709999999999998e-06 2023-07-02T14:03:52.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793184+0000", 2023-07-02T14:03:52.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1721999999999999e-05 2023-07-02T14:03:52.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793215+0000", 2023-07-02T14:03:52.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1280000000000008e-06 2023-07-02T14:03:52.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793224+0000", 2023-07-02T14:03:52.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34218549599999998 2023-07-02T14:03:52.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135409+0000", 2023-07-02T14:03:52.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117442 2023-07-02T14:03:52.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135526+0000", 2023-07-02T14:03:52.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000206257 2023-07-02T14:03:52.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:52.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135733+0000", 2023-07-02T14:03:52.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085700029999999996 2023-07-02T14:03:52.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144303+0000", 2023-07-02T14:03:52.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006278652 2023-07-02T14:03:52.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150581+0000", 2023-07-02T14:03:52.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4765999999999996e-05 2023-07-02T14:03:52.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150646+0000", 2023-07-02T14:03:52.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4777999999999995e-05 2023-07-02T14:03:52.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150711+0000", 2023-07-02T14:03:52.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999 2023-07-02T14:03:52.222 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.223 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.223 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.223 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3980 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 65536~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:52.223 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:03:52.223 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.286089447, 2023-07-02T14:03:52.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999, 2023-07-02T14:03:52.224 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.224 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.224 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.224 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.224 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.224 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3980 2023-07-02T14:03:52.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.224 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:03:52.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:03:52.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:52.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:03:52.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020244260000000002 2023-07-02T14:03:52.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795212+0000", 2023-07-02T14:03:52.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0005e-05 2023-07-02T14:03:52.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795222+0000", 2023-07-02T14:03:52.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9860000000000002e-06 2023-07-02T14:03:52.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795227+0000", 2023-07-02T14:03:52.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6369999999999999e-05 2023-07-02T14:03:52.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795264+0000", 2023-07-02T14:03:52.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1052e-05 2023-07-02T14:03:52.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795275+0000", 2023-07-02T14:03:52.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.343850459 2023-07-02T14:03:52.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139125+0000", 2023-07-02T14:03:52.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6760000000000004e-05 2023-07-02T14:03:52.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139202+0000", 2023-07-02T14:03:52.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015092799999999999 2023-07-02T14:03:52.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:52.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139353+0000", 2023-07-02T14:03:52.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011425940000000001 2023-07-02T14:03:52.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150779+0000", 2023-07-02T14:03:52.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033223060000000001 2023-07-02T14:03:52.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154101+0000", 2023-07-02T14:03:52.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9268999999999998e-05 2023-07-02T14:03:52.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154161+0000", 2023-07-02T14:03:52.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6499999999999999e-05 2023-07-02T14:03:52.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154207+0000", 2023-07-02T14:03:52.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999 2023-07-02T14:03:52.234 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.234 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.234 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.235 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3982 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1835008~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:52.235 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:03:52.235 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.284044160000001, 2023-07-02T14:03:52.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002, 2023-07-02T14:03:52.235 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.235 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.235 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.235 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.236 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.236 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3982 2023-07-02T14:03:52.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.236 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:03:52.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:03:52.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:52.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795233+0000", 2023-07-02T14:03:52.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8680999999999998e-05 2023-07-02T14:03:52.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795272+0000", 2023-07-02T14:03:52.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5629999999999999e-06 2023-07-02T14:03:52.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795277+0000", 2023-07-02T14:03:52.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8700000000000001e-06 2023-07-02T14:03:52.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795280+0000", 2023-07-02T14:03:52.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1964000000000001e-05 2023-07-02T14:03:52.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795312+0000", 2023-07-02T14:03:52.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.314e-06 2023-07-02T14:03:52.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795319+0000", 2023-07-02T14:03:52.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34828442599999998 2023-07-02T14:03:52.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143603+0000", 2023-07-02T14:03:52.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2438000000000001e-05 2023-07-02T14:03:52.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143665+0000", 2023-07-02T14:03:52.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3079999999999997e-05 2023-07-02T14:03:52.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:52.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143759+0000", 2023-07-02T14:03:52.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070633079999999999 2023-07-02T14:03:52.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150822+0000", 2023-07-02T14:03:52.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034430210000000001 2023-07-02T14:03:52.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154265+0000", 2023-07-02T14:03:52.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4249999999999998e-05 2023-07-02T14:03:52.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154309+0000", 2023-07-02T14:03:52.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2020000000000001e-05 2023-07-02T14:03:52.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154351+0000", 2023-07-02T14:03:52.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002 2023-07-02T14:03:52.245 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.246 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.246 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.246 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4054 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1900544~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:52.246 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:03:52.246 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.283992488999999, 2023-07-02T14:03:52.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002, 2023-07-02T14:03:52.247 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.247 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.247 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.247 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.247 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.247 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4054 2023-07-02T14:03:52.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.247 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:03:52.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:03:52.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:52.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:03:52.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018874860000000001 2023-07-02T14:03:52.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797172+0000", 2023-07-02T14:03:52.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3000000000000007e-06 2023-07-02T14:03:52.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797182+0000", 2023-07-02T14:03:52.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6260000000000003e-06 2023-07-02T14:03:52.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797186+0000", 2023-07-02T14:03:52.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9013e-05 2023-07-02T14:03:52.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797235+0000", 2023-07-02T14:03:52.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5990000000000008e-06 2023-07-02T14:03:52.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797245+0000", 2023-07-02T14:03:52.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34785621999999999 2023-07-02T14:03:52.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145101+0000", 2023-07-02T14:03:52.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7762000000000006e-05 2023-07-02T14:03:52.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145169+0000", 2023-07-02T14:03:52.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013469400000000001 2023-07-02T14:03:52.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:52.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145304+0000", 2023-07-02T14:03:52.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089538039999999992 2023-07-02T14:03:52.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154257+0000", 2023-07-02T14:03:52.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104218 2023-07-02T14:03:52.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154362+0000", 2023-07-02T14:03:52.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4168000000000001e-05 2023-07-02T14:03:52.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154406+0000", 2023-07-02T14:03:52.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3098000000000001e-05 2023-07-02T14:03:52.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154429+0000", 2023-07-02T14:03:52.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002 2023-07-02T14:03:52.257 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.257 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.257 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.258 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4056 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3670016~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:52.258 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:03:52.258 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.282085975999999, 2023-07-02T14:03:52.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997, 2023-07-02T14:03:52.258 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.258 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.258 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.258 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.259 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.259 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4056 2023-07-02T14:03:52.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.259 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:03:52.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:03:52.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:52.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797191+0000", 2023-07-02T14:03:52.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0438999999999997e-05 2023-07-02T14:03:52.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797232+0000", 2023-07-02T14:03:52.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6929999999999998e-06 2023-07-02T14:03:52.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797237+0000", 2023-07-02T14:03:52.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9739999999999998e-06 2023-07-02T14:03:52.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797240+0000", 2023-07-02T14:03:52.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0243000000000002e-05 2023-07-02T14:03:52.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797270+0000", 2023-07-02T14:03:52.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9889999999999997e-06 2023-07-02T14:03:52.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797276+0000", 2023-07-02T14:03:52.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35358319799999999 2023-07-02T14:03:52.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150859+0000", 2023-07-02T14:03:52.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1357e-05 2023-07-02T14:03:52.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150940+0000", 2023-07-02T14:03:52.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014903 2023-07-02T14:03:52.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:52.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151089+0000", 2023-07-02T14:03:52.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033512519999999999 2023-07-02T14:03:52.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154441+0000", 2023-07-02T14:03:52.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001050945 2023-07-02T14:03:52.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155492+0000", 2023-07-02T14:03:52.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2523e-05 2023-07-02T14:03:52.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155524+0000", 2023-07-02T14:03:52.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.057e-05 2023-07-02T14:03:52.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155555+0000", 2023-07-02T14:03:52.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997 2023-07-02T14:03:52.269 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.269 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.269 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.269 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4088 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3735552~458752] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:52.269 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:03:52.269 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.282033781000001, 2023-07-02T14:03:52.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998, 2023-07-02T14:03:52.270 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.270 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.270 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4088 2023-07-02T14:03:52.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.271 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:03:52.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:03:52.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:52.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:03:52.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00038503499999999999 2023-07-02T14:03:52.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797629+0000", 2023-07-02T14:03:52.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8739999999999995e-06 2023-07-02T14:03:52.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797637+0000", 2023-07-02T14:03:52.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7160000000000001e-06 2023-07-02T14:03:52.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797640+0000", 2023-07-02T14:03:52.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4342999999999998e-05 2023-07-02T14:03:52.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797695+0000", 2023-07-02T14:03:52.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0939999999999996e-06 2023-07-02T14:03:52.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797701+0000", 2023-07-02T14:03:52.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35408540799999999 2023-07-02T14:03:52.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151786+0000", 2023-07-02T14:03:52.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1953000000000006e-05 2023-07-02T14:03:52.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151868+0000", 2023-07-02T14:03:52.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015162800000000001 2023-07-02T14:03:52.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:52.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.152020+0000", 2023-07-02T14:03:52.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054561870000000004 2023-07-02T14:03:52.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.157476+0000", 2023-07-02T14:03:52.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059153199999999997 2023-07-02T14:03:52.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158068+0000", 2023-07-02T14:03:52.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2799e-05 2023-07-02T14:03:52.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158090+0000", 2023-07-02T14:03:52.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7235999999999999e-05 2023-07-02T14:03:52.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158108+0000", 2023-07-02T14:03:52.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998 2023-07-02T14:03:52.280 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.280 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.280 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.281 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4110 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:52.281 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:03:52.281 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.281632503999999, 2023-07-02T14:03:52.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864, 2023-07-02T14:03:52.281 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.281 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:52.281 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.282 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.282 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.282 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4110 2023-07-02T14:03:52.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.282 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:03:52.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:03:52.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:52.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:03:52.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9331e-05 2023-07-02T14:03:52.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797684+0000", 2023-07-02T14:03:52.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.147e-06 2023-07-02T14:03:52.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797689+0000", 2023-07-02T14:03:52.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5020000000000001e-06 2023-07-02T14:03:52.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797693+0000", 2023-07-02T14:03:52.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5254999999999999e-05 2023-07-02T14:03:52.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797748+0000", 2023-07-02T14:03:52.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5520000000000005e-06 2023-07-02T14:03:52.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797757+0000", 2023-07-02T14:03:52.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35674245700000001 2023-07-02T14:03:52.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154499+0000", 2023-07-02T14:03:52.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0090000000000001e-06 2023-07-02T14:03:52.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154506+0000", 2023-07-02T14:03:52.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864 2023-07-02T14:03:52.288 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.289 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.289 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.289 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:03:52.289 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:03:52.289 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.281580418000001, 2023-07-02T14:03:52.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998, 2023-07-02T14:03:52.290 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.290 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:03:52.290 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.290 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.290 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.290 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:03:52.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.290 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:03:52.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:03:52.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:52.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:03:52.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001714819 2023-07-02T14:03:52.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799412+0000", 2023-07-02T14:03:52.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7779999999999996e-06 2023-07-02T14:03:52.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799421+0000", 2023-07-02T14:03:52.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3510000000000002e-06 2023-07-02T14:03:52.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799424+0000", 2023-07-02T14:03:52.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0275999999999999e-05 2023-07-02T14:03:52.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799464+0000", 2023-07-02T14:03:52.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8389999999999996e-06 2023-07-02T14:03:52.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799473+0000", 2023-07-02T14:03:52.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35507638800000002 2023-07-02T14:03:52.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154550+0000", 2023-07-02T14:03:52.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6309999999999998e-06 2023-07-02T14:03:52.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154559+0000", 2023-07-02T14:03:52.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998 2023-07-02T14:03:52.297 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.297 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.297 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.297 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.297 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.297 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:52.299 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:52.298+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:52.299 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:03:52.399 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:52.398+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:52.465 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:52.465 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:52.465 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:52.465 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:52.475 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:03:52.500 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:52.500+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:52.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:03:52.571 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:03:52.577 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 2 2023-07-02T14:03:52.601 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:52.601+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:52.644 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:52.644 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:52.644 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:52.644 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:52.645 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:52.655 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:03:52.703 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:52.702+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:52.796 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:52.797 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:03:52.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:03:52.797 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:52.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.797 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:183 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:52.797 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:52.798 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.509196497999994, 2023-07-02T14:03:52.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003, 2023-07-02T14:03:52.798 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.798 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.798 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.798 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.798 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.799 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 183 2023-07-02T14:03:52.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.799 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:52.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:52.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:52.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:03:52.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9790999999999997e-05 2023-07-02T14:03:52.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285598+0000", 2023-07-02T14:03:52.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5809999999999996e-06 2023-07-02T14:03:52.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285604+0000", 2023-07-02T14:03:52.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7599999999999998e-06 2023-07-02T14:03:52.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285607+0000", 2023-07-02T14:03:52.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021108188999999999 2023-07-02T14:03:52.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306715+0000", 2023-07-02T14:03:52.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6548e-05 2023-07-02T14:03:52.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306792+0000", 2023-07-02T14:03:52.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173869 2023-07-02T14:03:52.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:52.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306965+0000", 2023-07-02T14:03:52.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060498009999999996 2023-07-02T14:03:52.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313015+0000", 2023-07-02T14:03:52.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049443719999999998 2023-07-02T14:03:52.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317960+0000", 2023-07-02T14:03:52.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5927999999999998e-05 2023-07-02T14:03:52.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318006+0000", 2023-07-02T14:03:52.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6004999999999998e-05 2023-07-02T14:03:52.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318042+0000", 2023-07-02T14:03:52.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003 2023-07-02T14:03:52.809 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.809 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.809 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.809 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:186 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:52.810 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:52.810 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.509125671999996, 2023-07-02T14:03:52.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003, 2023-07-02T14:03:52.810 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.810 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.810 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.810 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.810 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.811 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 186 2023-07-02T14:03:52.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.811 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:52.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:52.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:52.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:03:52.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4279e-05 2023-07-02T14:03:52.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285664+0000", 2023-07-02T14:03:52.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3789999999999995e-06 2023-07-02T14:03:52.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285673+0000", 2023-07-02T14:03:52.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499999999999998e-06 2023-07-02T14:03:52.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285678+0000", 2023-07-02T14:03:52.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023261947000000002 2023-07-02T14:03:52.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.308940+0000", 2023-07-02T14:03:52.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010323399999999999 2023-07-02T14:03:52.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309044+0000", 2023-07-02T14:03:52.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018878300000000001 2023-07-02T14:03:52.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:52.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309232+0000", 2023-07-02T14:03:52.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037874150000000001 2023-07-02T14:03:52.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313020+0000", 2023-07-02T14:03:52.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050976140000000003 2023-07-02T14:03:52.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318117+0000", 2023-07-02T14:03:52.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3182000000000001e-05 2023-07-02T14:03:52.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:52.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5055999999999998e-05 2023-07-02T14:03:52.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318196+0000", 2023-07-02T14:03:52.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003 2023-07-02T14:03:52.819 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.819 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.819 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.820 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:189 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:52.820 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:03:52.820 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.509051908999993, 2023-07-02T14:03:52.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997, 2023-07-02T14:03:52.820 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.821 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.821 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.821 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.821 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.821 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:03:52.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.821 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:03:52.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:03:52.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:52.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285683+0000", 2023-07-02T14:03:52.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8451999999999999e-05 2023-07-02T14:03:52.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285782+0000", 2023-07-02T14:03:52.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3650000000000004e-06 2023-07-02T14:03:52.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285790+0000", 2023-07-02T14:03:52.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8230000000000002e-06 2023-07-02T14:03:52.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285795+0000", 2023-07-02T14:03:52.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024619367999999999 2023-07-02T14:03:52.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310414+0000", 2023-07-02T14:03:52.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014244600000000001 2023-07-02T14:03:52.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310557+0000", 2023-07-02T14:03:52.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020612699999999999 2023-07-02T14:03:52.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:52.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310763+0000", 2023-07-02T14:03:52.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003427687 2023-07-02T14:03:52.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314191+0000", 2023-07-02T14:03:52.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040492089999999998 2023-07-02T14:03:52.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318240+0000", 2023-07-02T14:03:52.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0287999999999999e-05 2023-07-02T14:03:52.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318270+0000", 2023-07-02T14:03:52.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0868000000000002e-05 2023-07-02T14:03:52.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318311+0000", 2023-07-02T14:03:52.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997 2023-07-02T14:03:52.830 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.830 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.830 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.830 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:192 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:52.830 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:03:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.508935136000005, 2023-07-02T14:03:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003, 2023-07-02T14:03:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 192 2023-07-02T14:03:52.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.832 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:03:52.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:03:52.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:52.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285800+0000", 2023-07-02T14:03:52.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100417 2023-07-02T14:03:52.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285900+0000", 2023-07-02T14:03:52.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8244999999999999e-05 2023-07-02T14:03:52.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285919+0000", 2023-07-02T14:03:52.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2499999999999997e-06 2023-07-02T14:03:52.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285924+0000", 2023-07-02T14:03:52.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025288155999999999 2023-07-02T14:03:52.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311212+0000", 2023-07-02T14:03:52.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4073999999999996e-05 2023-07-02T14:03:52.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311286+0000", 2023-07-02T14:03:52.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016219699999999999 2023-07-02T14:03:52.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:52.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311448+0000", 2023-07-02T14:03:52.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066020599999999999 2023-07-02T14:03:52.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318050+0000", 2023-07-02T14:03:52.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000327986 2023-07-02T14:03:52.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318378+0000", 2023-07-02T14:03:52.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8693999999999998e-05 2023-07-02T14:03:52.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318427+0000", 2023-07-02T14:03:52.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6656999999999998e-05 2023-07-02T14:03:52.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318464+0000", 2023-07-02T14:03:52.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003 2023-07-02T14:03:52.840 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.840 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.840 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.841 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:195 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:52.841 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:03:52.841 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.508803631999996, 2023-07-02T14:03:52.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001, 2023-07-02T14:03:52.841 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.841 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.841 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.841 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.842 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.842 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 195 2023-07-02T14:03:52.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.842 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:03:52.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:03:52.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:52.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285931+0000", 2023-07-02T14:03:52.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4530999999999995e-05 2023-07-02T14:03:52.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286006+0000", 2023-07-02T14:03:52.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4113e-05 2023-07-02T14:03:52.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286020+0000", 2023-07-02T14:03:52.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7459999999999997e-06 2023-07-02T14:03:52.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286026+0000", 2023-07-02T14:03:52.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025997769 2023-07-02T14:03:52.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312023+0000", 2023-07-02T14:03:52.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4403000000000001e-05 2023-07-02T14:03:52.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312078+0000", 2023-07-02T14:03:52.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020581399999999999 2023-07-02T14:03:52.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:52.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312284+0000", 2023-07-02T14:03:52.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005771043 2023-07-02T14:03:52.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318055+0000", 2023-07-02T14:03:52.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00046015200000000002 2023-07-02T14:03:52.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318515+0000", 2023-07-02T14:03:52.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9444000000000003e-05 2023-07-02T14:03:52.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318564+0000", 2023-07-02T14:03:52.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2141000000000002e-05 2023-07-02T14:03:52.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318606+0000", 2023-07-02T14:03:52.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001 2023-07-02T14:03:52.850 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.850 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.850 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.851 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:198 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:52.851 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:03:52.851 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.508703672999999, 2023-07-02T14:03:52.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999, 2023-07-02T14:03:52.851 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.852 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.852 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.852 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.852 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.852 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 198 2023-07-02T14:03:52.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.852 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:03:52.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:03:52.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:52.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286031+0000", 2023-07-02T14:03:52.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00049286700000000002 2023-07-02T14:03:52.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286524+0000", 2023-07-02T14:03:52.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4938e-05 2023-07-02T14:03:52.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286539+0000", 2023-07-02T14:03:52.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0680000000000002e-06 2023-07-02T14:03:52.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286545+0000", 2023-07-02T14:03:52.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026497718 2023-07-02T14:03:52.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313043+0000", 2023-07-02T14:03:52.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0884999999999998e-05 2023-07-02T14:03:52.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313094+0000", 2023-07-02T14:03:52.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014109999999999999 2023-07-02T14:03:52.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:52.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313235+0000", 2023-07-02T14:03:52.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048258980000000003 2023-07-02T14:03:52.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318061+0000", 2023-07-02T14:03:52.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014346350000000001 2023-07-02T14:03:52.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319495+0000", 2023-07-02T14:03:52.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1986e-05 2023-07-02T14:03:52.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319547+0000", 2023-07-02T14:03:52.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0457e-05 2023-07-02T14:03:52.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319578+0000", 2023-07-02T14:03:52.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999 2023-07-02T14:03:52.860 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.861 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.861 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.861 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:201 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:52.861 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:52.861 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.508183247999995, 2023-07-02T14:03:52.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003, 2023-07-02T14:03:52.862 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.862 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.862 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.862 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.862 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.862 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 201 2023-07-02T14:03:52.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.862 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:52.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:52.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:52.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:03:52.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019213999999999999 2023-07-02T14:03:52.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286744+0000", 2023-07-02T14:03:52.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5279999999999992e-06 2023-07-02T14:03:52.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286753+0000", 2023-07-02T14:03:52.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4200000000000004e-06 2023-07-02T14:03:52.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286760+0000", 2023-07-02T14:03:52.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026818386999999999 2023-07-02T14:03:52.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.867 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313578+0000", 2023-07-02T14:03:52.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015965 2023-07-02T14:03:52.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.867 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313738+0000", 2023-07-02T14:03:52.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015767299999999999 2023-07-02T14:03:52.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:52.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313895+0000", 2023-07-02T14:03:52.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041714930000000001 2023-07-02T14:03:52.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318067+0000", 2023-07-02T14:03:52.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001561698 2023-07-02T14:03:52.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319629+0000", 2023-07-02T14:03:52.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2346000000000003e-05 2023-07-02T14:03:52.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319681+0000", 2023-07-02T14:03:52.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4691000000000003e-05 2023-07-02T14:03:52.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319726+0000", 2023-07-02T14:03:52.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003 2023-07-02T14:03:52.871 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.871 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.871 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.871 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:204 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:52.871 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:52.872 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.507968410000004, 2023-07-02T14:03:52.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998, 2023-07-02T14:03:52.872 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.872 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.872 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.872 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.872 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 204 2023-07-02T14:03:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:03:52.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2497999999999998e-05 2023-07-02T14:03:52.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286819+0000", 2023-07-02T14:03:52.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0099999999999995e-06 2023-07-02T14:03:52.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286827+0000", 2023-07-02T14:03:52.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000003e-06 2023-07-02T14:03:52.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286833+0000", 2023-07-02T14:03:52.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027384716999999999 2023-07-02T14:03:52.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314218+0000", 2023-07-02T14:03:52.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6204000000000004e-05 2023-07-02T14:03:52.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314294+0000", 2023-07-02T14:03:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015757499999999999 2023-07-02T14:03:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314452+0000", 2023-07-02T14:03:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036215560000000002 2023-07-02T14:03:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318073+0000", 2023-07-02T14:03:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017023979999999999 2023-07-02T14:03:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319776+0000", 2023-07-02T14:03:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9561000000000002e-05 2023-07-02T14:03:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319825+0000", 2023-07-02T14:03:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0329e-05 2023-07-02T14:03:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319866+0000", 2023-07-02T14:03:52.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998 2023-07-02T14:03:52.881 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.881 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.881 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.882 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:207 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:52.882 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:03:52.882 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.507893765000006, 2023-07-02T14:03:52.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003, 2023-07-02T14:03:52.882 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.882 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.882 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.882 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.883 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.883 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 207 2023-07-02T14:03:52.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.883 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:03:52.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:03:52.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:52.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.885 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286841+0000", 2023-07-02T14:03:52.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00068375100000000002 2023-07-02T14:03:52.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.885 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287525+0000", 2023-07-02T14:03:52.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1601e-05 2023-07-02T14:03:52.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287537+0000", 2023-07-02T14:03:52.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6409999999999996e-06 2023-07-02T14:03:52.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287543+0000", 2023-07-02T14:03:52.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027245844000000002 2023-07-02T14:03:52.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314789+0000", 2023-07-02T14:03:52.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8910000000000003e-05 2023-07-02T14:03:52.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314858+0000", 2023-07-02T14:03:52.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030992599999999997 2023-07-02T14:03:52.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:52.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315168+0000", 2023-07-02T14:03:52.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029122869999999999 2023-07-02T14:03:52.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318080+0000", 2023-07-02T14:03:52.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019963020000000001 2023-07-02T14:03:52.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320077+0000", 2023-07-02T14:03:52.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5116000000000005e-05 2023-07-02T14:03:52.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320142+0000", 2023-07-02T14:03:52.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3793000000000004e-05 2023-07-02T14:03:52.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320205+0000", 2023-07-02T14:03:52.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003 2023-07-02T14:03:52.891 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.891 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.891 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.892 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:236 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2424832~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:52.892 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:03:52.892 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.507181582000001, 2023-07-02T14:03:52.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003, 2023-07-02T14:03:52.892 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.892 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.893 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.893 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.893 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.893 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2023-07-02T14:03:52.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.893 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:03:52.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:03:52.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:52.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287553+0000", 2023-07-02T14:03:52.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0667e-05 2023-07-02T14:03:52.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287614+0000", 2023-07-02T14:03:52.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0454e-05 2023-07-02T14:03:52.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287624+0000", 2023-07-02T14:03:52.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8564999999999999e-05 2023-07-02T14:03:52.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287643+0000", 2023-07-02T14:03:52.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027977596 2023-07-02T14:03:52.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315621+0000", 2023-07-02T14:03:52.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000201728 2023-07-02T14:03:52.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315822+0000", 2023-07-02T14:03:52.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018436599999999999 2023-07-02T14:03:52.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:52.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316007+0000", 2023-07-02T14:03:52.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042423440000000003 2023-07-02T14:03:52.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320249+0000", 2023-07-02T14:03:52.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001452633 2023-07-02T14:03:52.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321702+0000", 2023-07-02T14:03:52.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3616000000000003e-05 2023-07-02T14:03:52.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321755+0000", 2023-07-02T14:03:52.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840999999999999e-05 2023-07-02T14:03:52.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321793+0000", 2023-07-02T14:03:52.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003 2023-07-02T14:03:52.901 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.902 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.902 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.902 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:251 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:52.902 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:03:52.902 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.507086044999994, 2023-07-02T14:03:52.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001, 2023-07-02T14:03:52.903 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.903 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.903 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.903 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.903 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.903 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 251 2023-07-02T14:03:52.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.903 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:03:52.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:03:52.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:52.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287649+0000", 2023-07-02T14:03:52.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3232000000000001e-05 2023-07-02T14:03:52.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287702+0000", 2023-07-02T14:03:52.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0010000000000007e-06 2023-07-02T14:03:52.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287711+0000", 2023-07-02T14:03:52.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3569999999999997e-06 2023-07-02T14:03:52.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287717+0000", 2023-07-02T14:03:52.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028686454 2023-07-02T14:03:52.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316403+0000", 2023-07-02T14:03:52.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7741000000000002e-05 2023-07-02T14:03:52.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316461+0000", 2023-07-02T14:03:52.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4313000000000004e-05 2023-07-02T14:03:52.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:52.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316545+0000", 2023-07-02T14:03:52.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003734032 2023-07-02T14:03:52.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320279+0000", 2023-07-02T14:03:52.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015237219999999999 2023-07-02T14:03:52.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321803+0000", 2023-07-02T14:03:52.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1927e-05 2023-07-02T14:03:52.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321845+0000", 2023-07-02T14:03:52.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5398999999999998e-05 2023-07-02T14:03:52.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321880+0000", 2023-07-02T14:03:52.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001 2023-07-02T14:03:52.912 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.912 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.912 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.912 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:255 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2555904~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:52.912 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:03:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.506671316999999, 2023-07-02T14:03:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002, 2023-07-02T14:03:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 255 2023-07-02T14:03:52.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.914 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:03:52.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:03:52.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:52.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288063+0000", 2023-07-02T14:03:52.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4459999999999997e-05 2023-07-02T14:03:52.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288118+0000", 2023-07-02T14:03:52.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0998999999999999e-05 2023-07-02T14:03:52.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288129+0000", 2023-07-02T14:03:52.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3219999999999998e-06 2023-07-02T14:03:52.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288135+0000", 2023-07-02T14:03:52.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029137386000000001 2023-07-02T14:03:52.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317273+0000", 2023-07-02T14:03:52.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7819999999999997e-05 2023-07-02T14:03:52.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317350+0000", 2023-07-02T14:03:52.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000174864 2023-07-02T14:03:52.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:52.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317525+0000", 2023-07-02T14:03:52.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027943519999999999 2023-07-02T14:03:52.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320320+0000", 2023-07-02T14:03:52.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015681460000000001 2023-07-02T14:03:52.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321888+0000", 2023-07-02T14:03:52.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9508999999999997e-05 2023-07-02T14:03:52.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321927+0000", 2023-07-02T14:03:52.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5428e-05 2023-07-02T14:03:52.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321963+0000", 2023-07-02T14:03:52.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002 2023-07-02T14:03:52.922 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.922 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.922 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.923 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:446 2.1 2:910b2a90:::rbd_data.125de1449605.000000000000010a:head [write 65536~3211264] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:52.923 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:52.923 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.429391565000003, 2023-07-02T14:03:52.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003, 2023-07-02T14:03:52.923 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.923 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.923 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.923 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 446 2023-07-02T14:03:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:52.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:52.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:52.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:03:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015907372999999999 2023-07-02T14:03:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381251+0000", 2023-07-02T14:03:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5527000000000001e-05 2023-07-02T14:03:52.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381277+0000", 2023-07-02T14:03:52.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9958000000000001e-05 2023-07-02T14:03:52.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381297+0000", 2023-07-02T14:03:52.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4941000000000001e-05 2023-07-02T14:03:52.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381352+0000", 2023-07-02T14:03:52.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011565699999999999 2023-07-02T14:03:52.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381467+0000", 2023-07-02T14:03:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018039799999999999 2023-07-02T14:03:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:52.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381648+0000", 2023-07-02T14:03:52.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088183559999999994 2023-07-02T14:03:52.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.390466+0000", 2023-07-02T14:03:52.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007278278 2023-07-02T14:03:52.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397744+0000", 2023-07-02T14:03:52.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9289000000000001e-05 2023-07-02T14:03:52.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397804+0000", 2023-07-02T14:03:52.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0479000000000002e-05 2023-07-02T14:03:52.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397854+0000", 2023-07-02T14:03:52.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003 2023-07-02T14:03:52.932 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.932 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.932 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:03:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:03:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 51.687060770999999, 2023-07-02T14:03:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:03:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.933 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:03:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:03:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:03:52.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:03:52.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:52.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:03:52.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:03:52.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:03:52.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:03:52.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:03:52.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:03:52.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:03:52.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:03:52.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:03:52.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:03:52.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:03:52.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:03:52.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:03:52.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:03:52.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:03:52.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:03:52.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:03:52.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:03:52.941 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.941 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.941 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.942 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:03:52.942 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:03:52.942 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 51.687000032999997, 2023-07-02T14:03:52.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:03:52.942 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.942 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.943 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.943 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.943 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.943 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:03:52.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.943 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:03:52.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:03:52.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:52.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:03:52.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:03:52.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:03:52.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:03:52.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:03:52.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:03:52.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:03:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:03:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:03:52.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:03:52.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:52.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:03:52.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:03:52.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:03:52.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:03:52.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:03:52.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:03:52.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:52.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:03:52.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:03:52.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:03:52.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:03:52.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:03:52.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:03:52.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:03:52.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:03:52.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:03:52.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:03:52.953 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.953 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.953 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.953 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:52.953 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:03:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.456545703000003, 2023-07-02T14:03:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:03:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.954 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:03:52.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.955 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:03:52.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:03:52.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:52.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:03:52.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:03:52.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:03:52.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:03:52.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:03:52.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:03:52.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:03:52.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:03:52.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:03:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:03:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:03:52.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:03:52.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:52.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:03:52.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:03:52.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:03:52.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:03:52.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:03:52.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:03:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:03:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:03:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:03:52.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:03:52.963 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.963 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.963 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.963 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2941 2.3 2:d3762e63:::rbd_data.125de1449605.00000000000000f8:head [write 65536~2031616] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:52.964 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:03:52.964 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.327442748999999, 2023-07-02T14:03:52.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002, 2023-07-02T14:03:52.964 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.964 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.964 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.964 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.965 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.965 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2941 2023-07-02T14:03:52.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.965 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:03:52.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:03:52.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:52.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467291+0000", 2023-07-02T14:03:52.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043022859999999998 2023-07-02T14:03:52.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471593+0000", 2023-07-02T14:03:52.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0999999999999999e-05 2023-07-02T14:03:52.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471614+0000", 2023-07-02T14:03:52.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0097e-05 2023-07-02T14:03:52.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471625+0000", 2023-07-02T14:03:52.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069808499999999998 2023-07-02T14:03:52.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472323+0000", 2023-07-02T14:03:52.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135635 2023-07-02T14:03:52.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472458+0000", 2023-07-02T14:03:52.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223365 2023-07-02T14:03:52.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:52.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472682+0000", 2023-07-02T14:03:52.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013761944999999999 2023-07-02T14:03:52.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.486444+0000", 2023-07-02T14:03:52.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016859747000000001 2023-07-02T14:03:52.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503303+0000", 2023-07-02T14:03:52.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6635000000000001e-05 2023-07-02T14:03:52.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503380+0000", 2023-07-02T14:03:52.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000161803 2023-07-02T14:03:52.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503542+0000", 2023-07-02T14:03:52.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002 2023-07-02T14:03:52.973 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.973 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.973 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2950 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 1507328~786432] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:03:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.319269441000003, 2023-07-02T14:03:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999, 2023-07-02T14:03:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.974 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.975 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.975 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.975 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.975 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2950 2023-07-02T14:03:52.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.975 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:03:52.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:03:52.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:52.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:03:52.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001147854 2023-07-02T14:03:52.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476614+0000", 2023-07-02T14:03:52.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.802e-05 2023-07-02T14:03:52.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476632+0000", 2023-07-02T14:03:52.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1860000000000003e-06 2023-07-02T14:03:52.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476639+0000", 2023-07-02T14:03:52.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061211240000000004 2023-07-02T14:03:52.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482760+0000", 2023-07-02T14:03:52.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8833999999999996e-05 2023-07-02T14:03:52.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482859+0000", 2023-07-02T14:03:52.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019270600000000001 2023-07-02T14:03:52.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:52.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.483051+0000", 2023-07-02T14:03:52.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060385630000000003 2023-07-02T14:03:52.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489090+0000", 2023-07-02T14:03:52.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024506918999999999 2023-07-02T14:03:52.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513597+0000", 2023-07-02T14:03:52.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9672999999999995e-05 2023-07-02T14:03:52.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513667+0000", 2023-07-02T14:03:52.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024329500000000001 2023-07-02T14:03:52.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513910+0000", 2023-07-02T14:03:52.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999 2023-07-02T14:03:52.983 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.983 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.984 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.984 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2951 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2686976~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:52.984 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:03:52.984 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.318088662999997, 2023-07-02T14:03:52.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999, 2023-07-02T14:03:52.985 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.985 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.985 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.985 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.985 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.985 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2951 2023-07-02T14:03:52.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.985 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:03:52.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:03:52.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:52.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476646+0000", 2023-07-02T14:03:52.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7639000000000003e-05 2023-07-02T14:03:52.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476684+0000", 2023-07-02T14:03:52.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3508000000000002e-05 2023-07-02T14:03:52.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476747+0000", 2023-07-02T14:03:52.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8499999999999994e-06 2023-07-02T14:03:52.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:52.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476755+0000", 2023-07-02T14:03:52.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078346160000000008 2023-07-02T14:03:52.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:52.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484590+0000", 2023-07-02T14:03:52.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7206000000000004e-05 2023-07-02T14:03:52.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:52.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484687+0000", 2023-07-02T14:03:52.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016243099999999999 2023-07-02T14:03:52.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:52.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484849+0000", 2023-07-02T14:03:52.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042482860000000004 2023-07-02T14:03:52.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:52.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489097+0000", 2023-07-02T14:03:52.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.02489504 2023-07-02T14:03:52.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:52.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513992+0000", 2023-07-02T14:03:52.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9972000000000001e-05 2023-07-02T14:03:52.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:52.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514052+0000", 2023-07-02T14:03:52.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149264 2023-07-02T14:03:52.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:52.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514202+0000", 2023-07-02T14:03:52.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999 2023-07-02T14:03:52.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.994 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:52.995 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:52.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.995 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2952 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2883584~12288] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:52.995 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:03:52.995 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.317974827999997, 2023-07-02T14:03:52.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651, 2023-07-02T14:03:52.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:52.996 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:52.996 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:52.996 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:52.996 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:52.996 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2952 2023-07-02T14:03:52.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.996 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:52.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:52.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:03:52.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:52.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:52.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:03:52.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:52.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:52.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476760+0000", 2023-07-02T14:03:52.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3664e-05 2023-07-02T14:03:52.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:52.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476774+0000", 2023-07-02T14:03:52.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.166e-06 2023-07-02T14:03:52.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:52.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:52.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:52.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476783+0000", 2023-07-02T14:03:52.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8720000000000001e-06 2023-07-02T14:03:53.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:53.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:53.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476788+0000", 2023-07-02T14:03:53.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087180529999999999 2023-07-02T14:03:53.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:53.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:53.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:53.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485506+0000", 2023-07-02T14:03:53.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5680999999999994e-05 2023-07-02T14:03:53.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:53.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:53.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:53.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485602+0000", 2023-07-02T14:03:53.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000222365 2023-07-02T14:03:53.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:53.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:53.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:53.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485824+0000", 2023-07-02T14:03:53.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003277379 2023-07-02T14:03:53.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:53.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:53.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:53.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489101+0000", 2023-07-02T14:03:53.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025151739999999999 2023-07-02T14:03:53.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:53.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:53.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514253+0000", 2023-07-02T14:03:53.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4749000000000002e-05 2023-07-02T14:03:53.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:53.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:53.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:53.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514288+0000", 2023-07-02T14:03:53.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130634 2023-07-02T14:03:53.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:53.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:53.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:53.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514419+0000", 2023-07-02T14:03:53.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651 2023-07-02T14:03:53.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:53.005 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:53.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:53.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:53.005 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:53.005 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:53.007 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:03:53.007 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:53.006+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:53.108 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:53.107+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:53.145 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:53.145 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:53.146 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:03:53.146 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:53.155 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:03:53.209 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:53.208+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:53.309 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:53.308+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:53.320 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:53.321 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:53.321 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:53.321 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:53.321 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:53.331 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:03:53.410 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:53.408+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:53.456 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:322 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 74.155203971999995, 2023-07-02T14:03:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.03676041, 2023-07-02T14:03:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 322 2023-07-02T14:03:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298820+0000", 2023-07-02T14:03:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298819+0000", 2023-07-02T14:03:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8068e-05 2023-07-02T14:03:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298877+0000", 2023-07-02T14:03:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3339999999999998e-06 2023-07-02T14:03:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298883+0000", 2023-07-02T14:03:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1030000000000004e-06 2023-07-02T14:03:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298887+0000", 2023-07-02T14:03:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023640230000000002 2023-07-02T14:03:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322528+0000", 2023-07-02T14:03:53.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8424999999999994e-05 2023-07-02T14:03:53.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322616+0000", 2023-07-02T14:03:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116815 2023-07-02T14:03:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322733+0000", 2023-07-02T14:03:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048252629999999998 2023-07-02T14:03:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327558+0000", 2023-07-02T14:03:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078965240000000003 2023-07-02T14:03:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335455+0000", 2023-07-02T14:03:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0511999999999999e-05 2023-07-02T14:03:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335515+0000", 2023-07-02T14:03:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4789000000000006e-05 2023-07-02T14:03:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335580+0000", 2023-07-02T14:03:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.03676041 2023-07-02T14:03:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:323 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 74.155132592000001, 2023-07-02T14:03:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036875044000000003, 2023-07-02T14:03:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:53.469 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:53.469 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:53.469 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 323 2023-07-02T14:03:53.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298891+0000", 2023-07-02T14:03:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298890+0000", 2023-07-02T14:03:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7277000000000002e-05 2023-07-02T14:03:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298938+0000", 2023-07-02T14:03:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0495e-05 2023-07-02T14:03:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298948+0000", 2023-07-02T14:03:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6829999999999999e-06 2023-07-02T14:03:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298953+0000", 2023-07-02T14:03:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024271435000000001 2023-07-02T14:03:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323224+0000", 2023-07-02T14:03:53.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9266999999999998e-05 2023-07-02T14:03:53.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:53.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323304+0000", 2023-07-02T14:03:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000157021 2023-07-02T14:03:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323461+0000", 2023-07-02T14:03:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041036689999999999 2023-07-02T14:03:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327564+0000", 2023-07-02T14:03:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080753490000000008 2023-07-02T14:03:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335640+0000", 2023-07-02T14:03:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7398999999999994e-05 2023-07-02T14:03:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335727+0000", 2023-07-02T14:03:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8967000000000001e-05 2023-07-02T14:03:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335766+0000", 2023-07-02T14:03:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036875044000000003 2023-07-02T14:03:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:324 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 74.155065214000004, 2023-07-02T14:03:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036935673000000002, 2023-07-02T14:03:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 324 2023-07-02T14:03:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.298958+0000", 2023-07-02T14:03:53.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5519999999999997e-05 2023-07-02T14:03:53.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:53.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299013+0000", 2023-07-02T14:03:53.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0701e-05 2023-07-02T14:03:53.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:53.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299024+0000", 2023-07-02T14:03:53.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2379999999999999e-06 2023-07-02T14:03:53.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299028+0000", 2023-07-02T14:03:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024972768999999999 2023-07-02T14:03:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324001+0000", 2023-07-02T14:03:53.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1178000000000002e-05 2023-07-02T14:03:53.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:53.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324072+0000", 2023-07-02T14:03:53.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000166119 2023-07-02T14:03:53.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:53.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324238+0000", 2023-07-02T14:03:53.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033314210000000002 2023-07-02T14:03:53.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327570+0000", 2023-07-02T14:03:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0082393369999999994 2023-07-02T14:03:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335809+0000", 2023-07-02T14:03:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7071999999999999e-05 2023-07-02T14:03:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335856+0000", 2023-07-02T14:03:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7926999999999998e-05 2023-07-02T14:03:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335894+0000", 2023-07-02T14:03:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036935673000000002 2023-07-02T14:03:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.492 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:325 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 458752~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:53.492 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:53.492 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 74.154989886999999, 2023-07-02T14:03:53.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036990838999999998, 2023-07-02T14:03:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 325 2023-07-02T14:03:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299034+0000", 2023-07-02T14:03:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299033+0000", 2023-07-02T14:03:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107619 2023-07-02T14:03:53.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:53.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299141+0000", 2023-07-02T14:03:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3497e-05 2023-07-02T14:03:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299154+0000", 2023-07-02T14:03:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4669e-05 2023-07-02T14:03:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299169+0000", 2023-07-02T14:03:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025630159999999999 2023-07-02T14:03:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324799+0000", 2023-07-02T14:03:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6953000000000007e-05 2023-07-02T14:03:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324876+0000", 2023-07-02T14:03:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016346199999999999 2023-07-02T14:03:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:53.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325039+0000", 2023-07-02T14:03:53.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003566624 2023-07-02T14:03:53.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328606+0000", 2023-07-02T14:03:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073295540000000003 2023-07-02T14:03:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335936+0000", 2023-07-02T14:03:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0044999999999999e-05 2023-07-02T14:03:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335986+0000", 2023-07-02T14:03:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8872999999999999e-05 2023-07-02T14:03:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336024+0000", 2023-07-02T14:03:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036990838999999998 2023-07-02T14:03:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.504 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:326 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:53.504 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:53.504 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 74.154847958999994, 2023-07-02T14:03:53.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036975239999999999, 2023-07-02T14:03:53.504 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:53.504 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:53.505 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:53.505 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:53.505 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:53.505 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 326 2023-07-02T14:03:53.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.505 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:53.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:53.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:53.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:53.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:53.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299176+0000", 2023-07-02T14:03:53.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:53.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:53.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299175+0000", 2023-07-02T14:03:53.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4782000000000003e-05 2023-07-02T14:03:53.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:53.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299250+0000", 2023-07-02T14:03:53.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.277e-05 2023-07-02T14:03:53.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:53.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299262+0000", 2023-07-02T14:03:53.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1460000000000001e-06 2023-07-02T14:03:53.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299268+0000", 2023-07-02T14:03:53.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026158098000000001 2023-07-02T14:03:53.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325427+0000", 2023-07-02T14:03:53.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4845999999999995e-05 2023-07-02T14:03:53.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:53.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325501+0000", 2023-07-02T14:03:53.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015890699999999999 2023-07-02T14:03:53.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:53.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325660+0000", 2023-07-02T14:03:53.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002952114 2023-07-02T14:03:53.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:53.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328612+0000", 2023-07-02T14:03:53.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074491949999999996 2023-07-02T14:03:53.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336062+0000", 2023-07-02T14:03:53.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7376999999999997e-05 2023-07-02T14:03:53.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:53.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336109+0000", 2023-07-02T14:03:53.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1814000000000003e-05 2023-07-02T14:03:53.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:53.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336151+0000", 2023-07-02T14:03:53.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036975239999999999 2023-07-02T14:03:53.515 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.515 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.515 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.516 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:327 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 589824~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:53.516 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:53.516 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 74.154748698000006, 2023-07-02T14:03:53.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037234253000000002, 2023-07-02T14:03:53.517 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:53.517 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:53.517 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:53.517 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:53.517 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:53.518 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 327 2023-07-02T14:03:53.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.518 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:53.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:53.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:53.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:53.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:53.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299275+0000", 2023-07-02T14:03:53.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:53.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:53.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299274+0000", 2023-07-02T14:03:53.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2213999999999997e-05 2023-07-02T14:03:53.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:53.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299346+0000", 2023-07-02T14:03:53.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1816e-05 2023-07-02T14:03:53.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:53.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299358+0000", 2023-07-02T14:03:53.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2626000000000001e-05 2023-07-02T14:03:53.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299371+0000", 2023-07-02T14:03:53.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026916853000000001 2023-07-02T14:03:53.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326288+0000", 2023-07-02T14:03:53.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6909000000000004e-05 2023-07-02T14:03:53.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:53.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326375+0000", 2023-07-02T14:03:53.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015430399999999999 2023-07-02T14:03:53.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:53.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326529+0000", 2023-07-02T14:03:53.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096652100000000005 2023-07-02T14:03:53.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336194+0000", 2023-07-02T14:03:53.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115299 2023-07-02T14:03:53.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:53.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336309+0000", 2023-07-02T14:03:53.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015981200000000001 2023-07-02T14:03:53.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:53.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336469+0000", 2023-07-02T14:03:53.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9867999999999998e-05 2023-07-02T14:03:53.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:53.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336509+0000", 2023-07-02T14:03:53.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037234253000000002 2023-07-02T14:03:53.527 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.527 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.527 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.528 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:328 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:53.528 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:53.529 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 74.154645109000001, 2023-07-02T14:03:53.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037216610999999997, 2023-07-02T14:03:53.529 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:53.530 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:53.530 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:53.530 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:53.530 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:53.531 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 328 2023-07-02T14:03:53.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.531 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:53.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:53.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:53.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:53.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:53.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:53.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:53.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:53.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299378+0000", 2023-07-02T14:03:53.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1412e-05 2023-07-02T14:03:53.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:53.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299449+0000", 2023-07-02T14:03:53.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0219000000000001e-05 2023-07-02T14:03:53.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:53.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299469+0000", 2023-07-02T14:03:53.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9480000000000003e-06 2023-07-02T14:03:53.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299479+0000", 2023-07-02T14:03:53.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027444344999999998 2023-07-02T14:03:53.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326924+0000", 2023-07-02T14:03:53.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5165000000000001e-05 2023-07-02T14:03:53.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:53.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327019+0000", 2023-07-02T14:03:53.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019113499999999999 2023-07-02T14:03:53.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:53.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327210+0000", 2023-07-02T14:03:53.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090197449999999992 2023-07-02T14:03:53.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336230+0000", 2023-07-02T14:03:53.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028764399999999999 2023-07-02T14:03:53.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:53.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336517+0000", 2023-07-02T14:03:53.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1981999999999999e-05 2023-07-02T14:03:53.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:53.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336559+0000", 2023-07-02T14:03:53.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:03:53.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:53.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336595+0000", 2023-07-02T14:03:53.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037216610999999997 2023-07-02T14:03:53.544 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.544 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.544 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.545 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:329 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 720896~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:53.545 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:53.545 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 74.154535191999997, 2023-07-02T14:03:53.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037191142000000003, 2023-07-02T14:03:53.545 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:53.546 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:53.546 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:53.546 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:53.546 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:53.546 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 329 2023-07-02T14:03:53.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.546 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:53.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:53.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:53.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:53.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:53.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299488+0000", 2023-07-02T14:03:53.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:53.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:53.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299487+0000", 2023-07-02T14:03:53.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7871000000000001e-05 2023-07-02T14:03:53.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:53.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299555+0000", 2023-07-02T14:03:53.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0446e-05 2023-07-02T14:03:53.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:53.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299566+0000", 2023-07-02T14:03:53.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7939999999999999e-06 2023-07-02T14:03:53.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299572+0000", 2023-07-02T14:03:53.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028034019 2023-07-02T14:03:53.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327606+0000", 2023-07-02T14:03:53.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1967999999999999e-05 2023-07-02T14:03:53.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:53.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327678+0000", 2023-07-02T14:03:53.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022216900000000001 2023-07-02T14:03:53.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:53.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327900+0000", 2023-07-02T14:03:53.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084016899999999999 2023-07-02T14:03:53.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336301+0000", 2023-07-02T14:03:53.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00030122800000000001 2023-07-02T14:03:53.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:53.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336603+0000", 2023-07-02T14:03:53.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0380000000000003e-05 2023-07-02T14:03:53.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:53.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336643+0000", 2023-07-02T14:03:53.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6455999999999998e-05 2023-07-02T14:03:53.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:53.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336679+0000", 2023-07-02T14:03:53.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037191142000000003 2023-07-02T14:03:53.556 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.556 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.556 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.557 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:330 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:53.557 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:53.557 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 74.154444421999997, 2023-07-02T14:03:53.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037849305, 2023-07-02T14:03:53.557 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:53.557 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:53.558 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:53.558 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:53.558 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:53.558 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 330 2023-07-02T14:03:53.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.558 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:53.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:53.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:53.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:53.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:53.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299579+0000", 2023-07-02T14:03:53.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:53.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:53.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299578+0000", 2023-07-02T14:03:53.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4708000000000005e-05 2023-07-02T14:03:53.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:53.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299643+0000", 2023-07-02T14:03:53.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7309999999999999e-06 2023-07-02T14:03:53.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:53.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299652+0000", 2023-07-02T14:03:53.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7989999999999999e-06 2023-07-02T14:03:53.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299658+0000", 2023-07-02T14:03:53.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028981664000000001 2023-07-02T14:03:53.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328639+0000", 2023-07-02T14:03:53.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0939000000000006e-05 2023-07-02T14:03:53.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:53.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328710+0000", 2023-07-02T14:03:53.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000142624 2023-07-02T14:03:53.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:53.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.328853+0000", 2023-07-02T14:03:53.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078374640000000006 2023-07-02T14:03:53.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:53.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336690+0000", 2023-07-02T14:03:53.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00068453000000000003 2023-07-02T14:03:53.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337375+0000", 2023-07-02T14:03:53.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0667999999999998e-05 2023-07-02T14:03:53.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:53.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337406+0000", 2023-07-02T14:03:53.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2858999999999999e-05 2023-07-02T14:03:53.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:53.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337428+0000", 2023-07-02T14:03:53.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037849305 2023-07-02T14:03:53.568 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.568 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.568 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.569 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:331 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:53.569 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:53.569 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 74.154360560000001, 2023-07-02T14:03:53.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037870332, 2023-07-02T14:03:53.569 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:53.569 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:53.570 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:53.570 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:53.570 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:53.570 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 331 2023-07-02T14:03:53.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.570 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:53.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:53.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:53.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:53.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:53.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299663+0000", 2023-07-02T14:03:53.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:53.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:53.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299662+0000", 2023-07-02T14:03:53.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2252e-05 2023-07-02T14:03:53.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:53.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299725+0000", 2023-07-02T14:03:53.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8440000000000004e-06 2023-07-02T14:03:53.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:53.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299733+0000", 2023-07-02T14:03:53.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6759999999999998e-06 2023-07-02T14:03:53.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299738+0000", 2023-07-02T14:03:53.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029459387 2023-07-02T14:03:53.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329197+0000", 2023-07-02T14:03:53.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0087999999999997e-05 2023-07-02T14:03:53.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:53.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329268+0000", 2023-07-02T14:03:53.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014405700000000001 2023-07-02T14:03:53.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:53.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329412+0000", 2023-07-02T14:03:53.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072842100000000002 2023-07-02T14:03:53.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:53.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336696+0000", 2023-07-02T14:03:53.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00076275300000000002 2023-07-02T14:03:53.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337459+0000", 2023-07-02T14:03:53.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5988000000000002e-05 2023-07-02T14:03:53.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:53.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337495+0000", 2023-07-02T14:03:53.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8686000000000002e-05 2023-07-02T14:03:53.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:53.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337533+0000", 2023-07-02T14:03:53.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037870332 2023-07-02T14:03:53.582 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.582 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.582 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.583 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:332 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:53.583 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:53.584 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 74.154280635999996, 2023-07-02T14:03:53.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037892506999999999, 2023-07-02T14:03:53.584 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:53.584 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:53.584 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:53.585 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:53.585 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:53.585 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 332 2023-07-02T14:03:53.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.585 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:53.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:53.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:53.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:53.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:53.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299743+0000", 2023-07-02T14:03:53.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:53.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:53.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299742+0000", 2023-07-02T14:03:53.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1366999999999997e-05 2023-07-02T14:03:53.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:53.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299804+0000", 2023-07-02T14:03:53.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9319999999999996e-06 2023-07-02T14:03:53.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:53.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299812+0000", 2023-07-02T14:03:53.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4359999999999999e-06 2023-07-02T14:03:53.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299818+0000", 2023-07-02T14:03:53.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029946144000000001 2023-07-02T14:03:53.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329764+0000", 2023-07-02T14:03:53.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000164828 2023-07-02T14:03:53.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:53.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329929+0000", 2023-07-02T14:03:53.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016139400000000001 2023-07-02T14:03:53.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:53.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330090+0000", 2023-07-02T14:03:53.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066134009999999997 2023-07-02T14:03:53.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:53.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336704+0000", 2023-07-02T14:03:53.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00086955400000000003 2023-07-02T14:03:53.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337573+0000", 2023-07-02T14:03:53.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8447000000000001e-05 2023-07-02T14:03:53.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:53.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337602+0000", 2023-07-02T14:03:53.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.362e-05 2023-07-02T14:03:53.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:53.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337635+0000", 2023-07-02T14:03:53.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037892506999999999 2023-07-02T14:03:53.595 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.595 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.595 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.595 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:333 2.7 2:eb94024b:::rbd_data.125de1449605.0000000000000105:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:53.596 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:53.596 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 74.154200555000003, 2023-07-02T14:03:53.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037932598999999997, 2023-07-02T14:03:53.596 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:53.596 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:53.596 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:53.596 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:53.597 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:53.597 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 333 2023-07-02T14:03:53.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.597 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:53.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:53.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:53.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:53.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:53.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299823+0000", 2023-07-02T14:03:53.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:53.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:53.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299822+0000", 2023-07-02T14:03:53.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0831000000000001e-05 2023-07-02T14:03:53.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:53.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299883+0000", 2023-07-02T14:03:53.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.322e-06 2023-07-02T14:03:53.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:53.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299888+0000", 2023-07-02T14:03:53.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3479999999999997e-06 2023-07-02T14:03:53.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.299893+0000", 2023-07-02T14:03:53.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030831905999999999 2023-07-02T14:03:53.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330725+0000", 2023-07-02T14:03:53.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3560999999999996e-05 2023-07-02T14:03:53.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:53.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330798+0000", 2023-07-02T14:03:53.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015165999999999999 2023-07-02T14:03:53.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 5", 2023-07-02T14:03:53.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330950+0000", 2023-07-02T14:03:53.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057590319999999999 2023-07-02T14:03:53.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.604 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:53.604 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.336709+0000", 2023-07-02T14:03:53.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00096987899999999999 2023-07-02T14:03:53.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337679+0000", 2023-07-02T14:03:53.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9699000000000001e-05 2023-07-02T14:03:53.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:53.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337719+0000", 2023-07-02T14:03:53.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6961000000000001e-05 2023-07-02T14:03:53.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:53.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.337756+0000", 2023-07-02T14:03:53.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037932598999999997 2023-07-02T14:03:53.607 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.607 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.607 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.607 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:416 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:53.607 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:53.607 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 74.135907176000003, 2023-07-02T14:03:53.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039242025, 2023-07-02T14:03:53.608 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:53.608 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:53.608 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:53.608 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:53.608 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:53.609 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:03:53.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.609 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:53.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:53.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:53.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:53.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:53.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:53.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:53.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:53.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318116+0000", 2023-07-02T14:03:53.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.692e-06 2023-07-02T14:03:53.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:53.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318119+0000", 2023-07-02T14:03:53.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9790000000000005e-06 2023-07-02T14:03:53.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:53.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318129+0000", 2023-07-02T14:03:53.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3640000000000001e-06 2023-07-02T14:03:53.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318137+0000", 2023-07-02T14:03:53.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022607808 2023-07-02T14:03:53.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340745+0000", 2023-07-02T14:03:53.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0130999999999996e-05 2023-07-02T14:03:53.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340825+0000", 2023-07-02T14:03:53.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039581080000000001 2023-07-02T14:03:53.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.615 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344783+0000", 2023-07-02T14:03:53.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000006e-05 2023-07-02T14:03:53.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.616 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.616 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.616 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344854+0000", 2023-07-02T14:03:53.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052105550000000004 2023-07-02T14:03:53.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350065+0000", 2023-07-02T14:03:53.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0321999999999997e-05 2023-07-02T14:03:53.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350115+0000", 2023-07-02T14:03:53.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012378409999999999 2023-07-02T14:03:53.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351353+0000", 2023-07-02T14:03:53.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5942000000000001e-05 2023-07-02T14:03:53.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:53.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351409+0000", 2023-07-02T14:03:53.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035677700000000002 2023-07-02T14:03:53.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:53.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.351766+0000", 2023-07-02T14:03:53.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000926719 2023-07-02T14:03:53.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:53.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352693+0000", 2023-07-02T14:03:53.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045652080000000003 2023-07-02T14:03:53.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357258+0000", 2023-07-02T14:03:53.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4717e-05 2023-07-02T14:03:53.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:53.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357293+0000", 2023-07-02T14:03:53.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5807000000000002e-05 2023-07-02T14:03:53.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:53.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357358+0000", 2023-07-02T14:03:53.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039242025 2023-07-02T14:03:53.623 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.623 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.623 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.624 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:417 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:53.624 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:53.624 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 74.135881738999998, 2023-07-02T14:03:53.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042407105, 2023-07-02T14:03:53.624 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:53.624 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:53.624 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:53.625 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:53.625 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:53.625 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 417 2023-07-02T14:03:53.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.625 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:53.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:53.626 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:53.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:53.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:53.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318142+0000", 2023-07-02T14:03:53.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:53.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:53.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318141+0000", 2023-07-02T14:03:53.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7120000000000002e-06 2023-07-02T14:03:53.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:53.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318145+0000", 2023-07-02T14:03:53.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8869999999999997e-06 2023-07-02T14:03:53.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:53.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318151+0000", 2023-07-02T14:03:53.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9820000000000002e-06 2023-07-02T14:03:53.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318155+0000", 2023-07-02T14:03:53.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699133 2023-07-02T14:03:53.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340854+0000", 2023-07-02T14:03:53.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6849999999999998e-05 2023-07-02T14:03:53.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340931+0000", 2023-07-02T14:03:53.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039672140000000002 2023-07-02T14:03:53.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344898+0000", 2023-07-02T14:03:53.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5341999999999998e-05 2023-07-02T14:03:53.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.344973+0000", 2023-07-02T14:03:53.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051819400000000003 2023-07-02T14:03:53.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350155+0000", 2023-07-02T14:03:53.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3261999999999999e-05 2023-07-02T14:03:53.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350209+0000", 2023-07-02T14:03:53.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00194097 2023-07-02T14:03:53.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352150+0000", 2023-07-02T14:03:53.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6851e-05 2023-07-02T14:03:53.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352226+0000", 2023-07-02T14:03:53.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051831109999999998 2023-07-02T14:03:53.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357409+0000", 2023-07-02T14:03:53.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5325e-05 2023-07-02T14:03:53.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:53.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357495+0000", 2023-07-02T14:03:53.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042107899999999998 2023-07-02T14:03:53.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:53.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.357916+0000", 2023-07-02T14:03:53.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0011672449999999999 2023-07-02T14:03:53.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:53.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.359083+0000", 2023-07-02T14:03:53.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013534930000000001 2023-07-02T14:03:53.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360437+0000", 2023-07-02T14:03:53.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2159000000000002e-05 2023-07-02T14:03:53.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:53.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360469+0000", 2023-07-02T14:03:53.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0092000000000005e-05 2023-07-02T14:03:53.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:53.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360549+0000", 2023-07-02T14:03:53.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042407105 2023-07-02T14:03:53.644 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.644 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.644 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.644 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:418 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:53.644 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:53.645 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 74.135862497999995, 2023-07-02T14:03:53.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.045613663999999998, 2023-07-02T14:03:53.645 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:53.645 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:53.645 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:53.645 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:53.645 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:53.646 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 418 2023-07-02T14:03:53.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.646 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:53.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:53.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:53.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:53.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:53.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:03:53.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:53.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:53.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318160+0000", 2023-07-02T14:03:53.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6490000000000001e-06 2023-07-02T14:03:53.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:53.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318164+0000", 2023-07-02T14:03:53.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:03:53.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:53.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318170+0000", 2023-07-02T14:03:53.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5279999999999999e-06 2023-07-02T14:03:53.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318173+0000", 2023-07-02T14:03:53.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022788856 2023-07-02T14:03:53.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340962+0000", 2023-07-02T14:03:53.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7730000000000003e-05 2023-07-02T14:03:53.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341040+0000", 2023-07-02T14:03:53.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039795899999999999 2023-07-02T14:03:53.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345020+0000", 2023-07-02T14:03:53.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4962000000000006e-05 2023-07-02T14:03:53.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345085+0000", 2023-07-02T14:03:53.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051955430000000004 2023-07-02T14:03:53.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350280+0000", 2023-07-02T14:03:53.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8913999999999995e-05 2023-07-02T14:03:53.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350379+0000", 2023-07-02T14:03:53.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001887344 2023-07-02T14:03:53.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352266+0000", 2023-07-02T14:03:53.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7865000000000003e-05 2023-07-02T14:03:53.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352314+0000", 2023-07-02T14:03:53.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060621190000000004 2023-07-02T14:03:53.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358376+0000", 2023-07-02T14:03:53.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2448999999999997e-05 2023-07-02T14:03:53.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358449+0000", 2023-07-02T14:03:53.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0021539929999999999 2023-07-02T14:03:53.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360603+0000", 2023-07-02T14:03:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7859000000000002e-05 2023-07-02T14:03:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360671+0000", 2023-07-02T14:03:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026930699999999999 2023-07-02T14:03:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.360940+0000", 2023-07-02T14:03:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00081179699999999995 2023-07-02T14:03:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:53.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361752+0000", 2023-07-02T14:03:53.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0019350490000000001 2023-07-02T14:03:53.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363687+0000", 2023-07-02T14:03:53.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4462000000000003e-05 2023-07-02T14:03:53.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:53.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363721+0000", 2023-07-02T14:03:53.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3408000000000001e-05 2023-07-02T14:03:53.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:53.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363775+0000", 2023-07-02T14:03:53.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.045613663999999998 2023-07-02T14:03:53.663 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.663 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.663 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.664 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:419 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:53.664 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:53.664 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 74.135846170999997, 2023-07-02T14:03:53.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.047296703000000002, 2023-07-02T14:03:53.664 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:53.665 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:53.665 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:53.665 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:53.665 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:53.665 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 419 2023-07-02T14:03:53.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.665 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:53.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:53.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:53.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:53.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:53.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:53.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:53.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:53.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318177+0000", 2023-07-02T14:03:53.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.433e-06 2023-07-02T14:03:53.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:53.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318180+0000", 2023-07-02T14:03:53.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9950000000000002e-06 2023-07-02T14:03:53.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:53.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318186+0000", 2023-07-02T14:03:53.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7290000000000002e-06 2023-07-02T14:03:53.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318190+0000", 2023-07-02T14:03:53.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022877989000000001 2023-07-02T14:03:53.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341068+0000", 2023-07-02T14:03:53.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5803999999999994e-05 2023-07-02T14:03:53.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.341154+0000", 2023-07-02T14:03:53.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039633000000000003 2023-07-02T14:03:53.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345117+0000", 2023-07-02T14:03:53.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7188999999999996e-05 2023-07-02T14:03:53.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.345194+0000", 2023-07-02T14:03:53.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052306059999999996 2023-07-02T14:03:53.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350425+0000", 2023-07-02T14:03:53.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8093999999999997e-05 2023-07-02T14:03:53.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.350473+0000", 2023-07-02T14:03:53.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001865158 2023-07-02T14:03:53.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352338+0000", 2023-07-02T14:03:53.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5479000000000003e-05 2023-07-02T14:03:53.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.352384+0000", 2023-07-02T14:03:53.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061153290000000001 2023-07-02T14:03:53.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358499+0000", 2023-07-02T14:03:53.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5156999999999999e-05 2023-07-02T14:03:53.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.358574+0000", 2023-07-02T14:03:53.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027847459999999998 2023-07-02T14:03:53.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361359+0000", 2023-07-02T14:03:53.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2633999999999999e-05 2023-07-02T14:03:53.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.361402+0000", 2023-07-02T14:03:53.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024171510000000002 2023-07-02T14:03:53.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363819+0000", 2023-07-02T14:03:53.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6988999999999999e-05 2023-07-02T14:03:53.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:53.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.363876+0000", 2023-07-02T14:03:53.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035321300000000001 2023-07-02T14:03:53.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:53.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.364229+0000", 2023-07-02T14:03:53.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00094674799999999997 2023-07-02T14:03:53.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:53.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365176+0000", 2023-07-02T14:03:53.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019724200000000001 2023-07-02T14:03:53.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365373+0000", 2023-07-02T14:03:53.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2116e-05 2023-07-02T14:03:53.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:53.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365425+0000", 2023-07-02T14:03:53.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8996999999999999e-05 2023-07-02T14:03:53.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:53.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.365474+0000", 2023-07-02T14:03:53.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.047296703000000002 2023-07-02T14:03:53.684 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.684 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.684 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.685 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:472 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:53.685 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:53.685 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 74.076389938999995, 2023-07-02T14:03:53.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036204721000000002, 2023-07-02T14:03:53.685 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:53.685 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:53.685 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:53.686 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:53.686 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:53.686 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 472 2023-07-02T14:03:53.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.686 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:53.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:53.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:53.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:53.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:53.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377634+0000", 2023-07-02T14:03:53.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:03:53.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:53.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377633+0000", 2023-07-02T14:03:53.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.815e-06 2023-07-02T14:03:53.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:53.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377635+0000", 2023-07-02T14:03:53.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9459999999999998e-06 2023-07-02T14:03:53.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:53.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377638+0000", 2023-07-02T14:03:53.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.673e-06 2023-07-02T14:03:53.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377643+0000", 2023-07-02T14:03:53.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043004239999999997 2023-07-02T14:03:53.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.381943+0000", 2023-07-02T14:03:53.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4321999999999999e-05 2023-07-02T14:03:53.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.381998+0000", 2023-07-02T14:03:53.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003008377 2023-07-02T14:03:53.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385006+0000", 2023-07-02T14:03:53.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8751999999999998e-05 2023-07-02T14:03:53.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385045+0000", 2023-07-02T14:03:53.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086260880000000005 2023-07-02T14:03:53.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393671+0000", 2023-07-02T14:03:53.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6365999999999997e-05 2023-07-02T14:03:53.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393707+0000", 2023-07-02T14:03:53.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028560289999999999 2023-07-02T14:03:53.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396563+0000", 2023-07-02T14:03:53.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6195e-05 2023-07-02T14:03:53.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396619+0000", 2023-07-02T14:03:53.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023118269999999998 2023-07-02T14:03:53.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.398931+0000", 2023-07-02T14:03:53.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3885999999999999e-05 2023-07-02T14:03:53.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.398985+0000", 2023-07-02T14:03:53.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043756890000000003 2023-07-02T14:03:53.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403361+0000", 2023-07-02T14:03:53.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0043000000000001e-05 2023-07-02T14:03:53.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403421+0000", 2023-07-02T14:03:53.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001590972 2023-07-02T14:03:53.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405012+0000", 2023-07-02T14:03:53.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0067000000000005e-05 2023-07-02T14:03:53.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405082+0000", 2023-07-02T14:03:53.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050958610000000001 2023-07-02T14:03:53.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410178+0000", 2023-07-02T14:03:53.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7070000000000004e-05 2023-07-02T14:03:53.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410275+0000", 2023-07-02T14:03:53.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060121100000000004 2023-07-02T14:03:53.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410876+0000", 2023-07-02T14:03:53.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3177999999999999e-05 2023-07-02T14:03:53.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:53.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410919+0000", 2023-07-02T14:03:53.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000271394 2023-07-02T14:03:53.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:53.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411191+0000", 2023-07-02T14:03:53.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001009598 2023-07-02T14:03:53.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.412200+0000", 2023-07-02T14:03:53.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014789790000000001 2023-07-02T14:03:53.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:53.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413679+0000", 2023-07-02T14:03:53.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2624000000000002e-05 2023-07-02T14:03:53.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:53.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413742+0000", 2023-07-02T14:03:53.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6564999999999994e-05 2023-07-02T14:03:53.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:53.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413838+0000", 2023-07-02T14:03:53.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036204721000000002 2023-07-02T14:03:53.707 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.707 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.707 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.708 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:473 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:53.708 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:53.708 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 74.076378528000006, 2023-07-02T14:03:53.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037915904, 2023-07-02T14:03:53.708 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:53.709 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:53.709 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:53.709 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:53.709 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:53.709 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 473 2023-07-02T14:03:53.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.709 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:53.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:53.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:53.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:53.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:53.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:53.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:03:53.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:53.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377645+0000", 2023-07-02T14:03:53.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.119e-06 2023-07-02T14:03:53.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:53.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377647+0000", 2023-07-02T14:03:53.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.875e-06 2023-07-02T14:03:53.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:53.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377650+0000", 2023-07-02T14:03:53.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1949999999999999e-06 2023-07-02T14:03:53.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377653+0000", 2023-07-02T14:03:53.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004375444 2023-07-02T14:03:53.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382028+0000", 2023-07-02T14:03:53.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1063999999999997e-05 2023-07-02T14:03:53.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382089+0000", 2023-07-02T14:03:53.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030043869999999999 2023-07-02T14:03:53.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385094+0000", 2023-07-02T14:03:53.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8167999999999997e-05 2023-07-02T14:03:53.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385142+0000", 2023-07-02T14:03:53.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008603421 2023-07-02T14:03:53.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393745+0000", 2023-07-02T14:03:53.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3763000000000005e-05 2023-07-02T14:03:53.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393819+0000", 2023-07-02T14:03:53.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028274720000000001 2023-07-02T14:03:53.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396647+0000", 2023-07-02T14:03:53.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7929999999999997e-05 2023-07-02T14:03:53.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396695+0000", 2023-07-02T14:03:53.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023426459999999999 2023-07-02T14:03:53.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399037+0000", 2023-07-02T14:03:53.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5846000000000004e-05 2023-07-02T14:03:53.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399123+0000", 2023-07-02T14:03:53.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043448360000000004 2023-07-02T14:03:53.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403468+0000", 2023-07-02T14:03:53.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6699000000000001e-05 2023-07-02T14:03:53.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403515+0000", 2023-07-02T14:03:53.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001622943 2023-07-02T14:03:53.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405138+0000", 2023-07-02T14:03:53.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6382000000000002e-05 2023-07-02T14:03:53.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405204+0000", 2023-07-02T14:03:53.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051089910000000002 2023-07-02T14:03:53.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410313+0000", 2023-07-02T14:03:53.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8106000000000003e-05 2023-07-02T14:03:53.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410361+0000", 2023-07-02T14:03:53.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012200399999999999 2023-07-02T14:03:53.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411581+0000", 2023-07-02T14:03:53.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5067000000000005e-05 2023-07-02T14:03:53.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411656+0000", 2023-07-02T14:03:53.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022353249999999998 2023-07-02T14:03:53.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413891+0000", 2023-07-02T14:03:53.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9356000000000003e-05 2023-07-02T14:03:53.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:53.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.413981+0000", 2023-07-02T14:03:53.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00043818299999999998 2023-07-02T14:03:53.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:53.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.414419+0000", 2023-07-02T14:03:53.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00081559199999999997 2023-07-02T14:03:53.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:53.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415235+0000", 2023-07-02T14:03:53.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026434499999999998 2023-07-02T14:03:53.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415499+0000", 2023-07-02T14:03:53.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9779999999999999e-05 2023-07-02T14:03:53.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:53.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415529+0000", 2023-07-02T14:03:53.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2159999999999997e-05 2023-07-02T14:03:53.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:53.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415561+0000", 2023-07-02T14:03:53.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037915904 2023-07-02T14:03:53.732 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.732 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.732 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.733 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:475 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:53.733 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:53.733 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 74.076368489000004, 2023-07-02T14:03:53.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039286962000000002, 2023-07-02T14:03:53.733 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:53.733 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:53.734 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:53.734 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:53.734 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:53.734 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 475 2023-07-02T14:03:53.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.734 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:53.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:53.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:53.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:53.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:53.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:53.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:53.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:53.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377655+0000", 2023-07-02T14:03:53.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4779999999999997e-06 2023-07-02T14:03:53.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:53.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377664+0000", 2023-07-02T14:03:53.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.268e-06 2023-07-02T14:03:53.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:53.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377671+0000", 2023-07-02T14:03:53.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.861e-06 2023-07-02T14:03:53.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.377674+0000", 2023-07-02T14:03:53.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044395789999999999 2023-07-02T14:03:53.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382114+0000", 2023-07-02T14:03:53.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8096999999999997e-05 2023-07-02T14:03:53.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.382162+0000", 2023-07-02T14:03:53.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029949059999999999 2023-07-02T14:03:53.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385157+0000", 2023-07-02T14:03:53.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1158000000000001e-05 2023-07-02T14:03:53.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.385198+0000", 2023-07-02T14:03:53.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086477229999999995 2023-07-02T14:03:53.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393846+0000", 2023-07-02T14:03:53.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5990999999999999e-05 2023-07-02T14:03:53.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.393912+0000", 2023-07-02T14:03:53.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027979599999999999 2023-07-02T14:03:53.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396710+0000", 2023-07-02T14:03:53.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5911000000000002e-05 2023-07-02T14:03:53.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.396746+0000", 2023-07-02T14:03:53.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024039759999999999 2023-07-02T14:03:53.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399150+0000", 2023-07-02T14:03:53.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7834999999999999e-05 2023-07-02T14:03:53.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.399188+0000", 2023-07-02T14:03:53.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004364054 2023-07-02T14:03:53.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403552+0000", 2023-07-02T14:03:53.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6274999999999999e-05 2023-07-02T14:03:53.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.403608+0000", 2023-07-02T14:03:53.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0016261540000000001 2023-07-02T14:03:53.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405234+0000", 2023-07-02T14:03:53.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6945000000000004e-05 2023-07-02T14:03:53.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.405301+0000", 2023-07-02T14:03:53.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050871329999999998 2023-07-02T14:03:53.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410388+0000", 2023-07-02T14:03:53.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0210999999999995e-05 2023-07-02T14:03:53.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.410468+0000", 2023-07-02T14:03:53.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001227454 2023-07-02T14:03:53.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411696+0000", 2023-07-02T14:03:53.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8188000000000005e-05 2023-07-02T14:03:53.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.411764+0000", 2023-07-02T14:03:53.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031779930000000001 2023-07-02T14:03:53.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.414942+0000", 2023-07-02T14:03:53.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7236000000000004e-05 2023-07-02T14:03:53.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:03:53.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415019+0000", 2023-07-02T14:03:53.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000573045 2023-07-02T14:03:53.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415592+0000", 2023-07-02T14:03:53.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7580000000000002e-05 2023-07-02T14:03:53.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:53.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415640+0000", 2023-07-02T14:03:53.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024783200000000002 2023-07-02T14:03:53.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:53.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.415888+0000", 2023-07-02T14:03:53.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00080440999999999998 2023-07-02T14:03:53.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:53.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416692+0000", 2023-07-02T14:03:53.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000195787 2023-07-02T14:03:53.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416888+0000", 2023-07-02T14:03:53.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0787999999999997e-05 2023-07-02T14:03:53.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:53.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416919+0000", 2023-07-02T14:03:53.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.338e-05 2023-07-02T14:03:53.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:53.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.416942+0000", 2023-07-02T14:03:53.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039286962000000002 2023-07-02T14:03:53.758 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.758 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.758 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.759 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:5269 2.5 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:head [write 446464~1847296] snapc e=[e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e81)", 2023-07-02T14:03:53.759 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.147365+0000", 2023-07-02T14:03:53.759 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.306658354, 2023-07-02T14:03:53.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036564266999999998, 2023-07-02T14:03:53.759 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:53.759 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:53.759 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:53.760 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:53.760 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:53.760 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 5269 2023-07-02T14:03:53.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.760 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:53.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:53.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.147365+0000", 2023-07-02T14:03:53.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:53.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:53.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.147365+0000", 2023-07-02T14:03:53.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:53.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:53.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.147365+0000", 2023-07-02T14:03:53.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001616302 2023-07-02T14:03:53.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:53.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.148981+0000", 2023-07-02T14:03:53.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1630000000000002e-06 2023-07-02T14:03:53.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:53.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.148990+0000", 2023-07-02T14:03:53.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4989999999999999e-06 2023-07-02T14:03:53.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:53.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.148997+0000", 2023-07-02T14:03:53.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012255096 2023-07-02T14:03:53.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:53.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.161252+0000", 2023-07-02T14:03:53.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3946e-05 2023-07-02T14:03:53.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:53.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.161326+0000", 2023-07-02T14:03:53.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022859399999999999 2023-07-02T14:03:53.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:53.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.161554+0000", 2023-07-02T14:03:53.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062552529999999997 2023-07-02T14:03:53.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:53.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.167810+0000", 2023-07-02T14:03:53.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016036499999999999 2023-07-02T14:03:53.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:53.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.183846+0000", 2023-07-02T14:03:53.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7889000000000003e-05 2023-07-02T14:03:53.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:53.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.183884+0000", 2023-07-02T14:03:53.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5410000000000001e-05 2023-07-02T14:03:53.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:53.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:53.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:53.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.183929+0000", 2023-07-02T14:03:53.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036564266999999998 2023-07-02T14:03:53.770 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.770 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.770 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.770 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:53.771 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:53.771 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:53.772 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:03:53.773 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:03:53.773 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:53.771+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:53.873 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:53.872+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:53.891 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:03:53.938 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:53.938 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:53.938 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:03:53.939 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:53.949 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:03:53.974 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:53.973+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:54.037 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:54.055 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:54.072 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:54.072 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:54.073 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:54.073 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:54.073 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:54.073 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:54.074 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:54.073+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:54.082 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:03:54.089 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:03:54.107 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:54.125 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:54.126 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:03:54.126 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:03:54.126 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:03:54.126 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:03:54.127 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:03:54.127 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:03:54.174 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:03:54.174+0000 7f16df8f0700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83343) UID: 0 2023-07-02T14:03:54.247 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:54.247 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:54.257 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:03:54.299 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:54.298+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:54.325 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4] out_osds: [5, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2023-07-02T14:03:54.325 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:03:54.325 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 2, 3, 4, 5] 2023-07-02T14:03:54.326 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-07-02T14:03:54.326 INFO:teuthology.orchestra.run:waiting for 300 2023-07-02T14:03:54.376 INFO:tasks.ceph.osd.3:Stopped 2023-07-02T14:03:54.377 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.3 2023-07-02T14:03:54.377 INFO:tasks.thrashosds.thrasher:remote for osd 3 is ubuntu@smithi022.front.sepia.ceph.com 2023-07-02T14:03:54.378 DEBUG:teuthology.orchestra.run.smithi022:> type ceph-objectstore-tool 2023-07-02T14:03:54.378 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:54.375+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:54.423 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:54.423 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:54.432 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-07-02T14:03:54.433 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2023-07-02T14:03:54.434 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:03:54.477 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:54.476+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:54.578 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:54.577+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:54.578 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap13 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap14 b3fbb028-e267-4d25-a750-93ba18a2316a@snap14.diff 2023-07-02T14:03:54.603 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:54.604 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:54.614 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:03:54.699 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:54.698+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:54.778 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:54.778 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:54.779 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:54.778+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:54.789 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:03:54.880 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:54.879+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:54.918 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:54.918 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:54.929 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:03:54.981 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:54.979+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:55.052 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:55.053 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:55.062 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:03:55.103 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:55.102+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:55.183 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:55.181+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:55.226 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:55.226 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:55.236 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:03:55.284 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:55.283+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:55.384 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:55.383+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:55.400 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:55.401 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:55.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:03:55.485 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:55.484+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:55.582 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:55.582 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:55.585 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:55.585+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:55.595 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:03:55.626 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 100% complete...done. 2023-07-02T14:03:55.629 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:03:55.629 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=15 2023-07-02T14:03:55.629 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap15 2023-07-02T14:03:55.687 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:55.686+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:55.721 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:55.721 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:55.731 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:03:55.774 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:03:55.787 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:55.787+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:55.857 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:55.858 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:55.869 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:03:55.912 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:55.911+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:55.990 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:55.988+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:55.994 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:55.995 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:56.005 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:03:56.090 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:56.089+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:56.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:03:56.170 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:56.170 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:56.180 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:03:56.187 INFO:teuthology.orchestra.run.smithi022.stdout:2.6 2023-07-02T14:03:56.187 INFO:teuthology.orchestra.run.smithi022.stdout:2.5 2023-07-02T14:03:56.188 INFO:teuthology.orchestra.run.smithi022.stdout:2.3 2023-07-02T14:03:56.188 INFO:teuthology.orchestra.run.smithi022.stdout:2.7 2023-07-02T14:03:56.188 INFO:teuthology.orchestra.run.smithi022.stdout:1.0 2023-07-02T14:03:56.221 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:56.220+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:56.276 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:56.291 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:56.290+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:56.293 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:56.305 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:56.306 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:56.310 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:56.316 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:03:56.328 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:56.346 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:56.347 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:03:56.347 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:03:56.347 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:03:56.347 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:03:56.347 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:03:56.347 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:03:56.349 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:03:56.350 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:03:56.398 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:56.398+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:56.479 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:56.479 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:56.479 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:56.480 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:56.480 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:56.489 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:03:56.494 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:56.493+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:56.563 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:03:56.581 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:03:56.586 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 2 2023-07-02T14:03:56.616 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:56.615+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:56.659 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:56.659 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:56.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:56.660 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:56.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.660 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:5545 2.0 e81/46 2:159aabad:::rbd_data.125de1449605.000000000000005d:head v 81'1159, mlcod=81'1155)", 2023-07-02T14:03:56.660 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.389719+0000", 2023-07-02T14:03:56.660 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.268118466000001, 2023-07-02T14:03:56.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014352986, 2023-07-02T14:03:56.660 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:56.661 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:56.661 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:56.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:56.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.389719+0000", 2023-07-02T14:03:56.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:56.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:56.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.389719+0000", 2023-07-02T14:03:56.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:56.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:56.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.389718+0000", 2023-07-02T14:03:56.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037957160000000002 2023-07-02T14:03:56.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:56.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.393513+0000", 2023-07-02T14:03:56.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7331000000000001e-05 2023-07-02T14:03:56.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:56.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.393531+0000", 2023-07-02T14:03:56.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0229999999999998e-06 2023-07-02T14:03:56.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:56.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.393540+0000", 2023-07-02T14:03:56.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0124000000000002e-05 2023-07-02T14:03:56.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:56.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.393600+0000", 2023-07-02T14:03:56.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2815999999999999e-05 2023-07-02T14:03:56.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:56.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.393633+0000", 2023-07-02T14:03:56.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010399037 2023-07-02T14:03:56.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:56.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.404032+0000", 2023-07-02T14:03:56.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9761000000000001e-05 2023-07-02T14:03:56.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:56.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.404072+0000", 2023-07-02T14:03:56.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014352986 2023-07-02T14:03:56.667 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.667 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.667 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.668 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:03:56.668 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:03:56.668 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 16.760222835, 2023-07-02T14:03:56.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:03:56.668 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:56.668 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:56.669 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:56.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:56.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:03:56.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:56.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:56.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:03:56.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:56.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:56.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:03:56.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:03:56.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:56.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:03:56.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:03:56.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:56.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:03:56.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:03:56.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:56.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:03:56.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:03:56.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:56.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:03:56.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:03:56.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:56.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:03:56.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:03:56.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:56.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:03:56.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:03:56.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:56.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:03:56.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:03:56.675 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.675 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.675 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.675 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:03:56.676 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:03:56.676 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 16.756137117000002, 2023-07-02T14:03:56.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:03:56.676 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:56.676 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:56.676 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:56.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:56.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:03:56.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:56.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:56.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:03:56.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:56.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:56.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:03:56.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:03:56.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:56.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:03:56.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:03:56.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:56.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:03:56.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:03:56.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:56.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:03:56.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:03:56.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:56.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:03:56.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:03:56.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:56.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:03:56.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:03:56.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:56.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:03:56.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:03:56.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:56.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:03:56.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:03:56.683 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.683 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.683 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.683 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:03:56.683 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:03:56.684 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 16.752272336000001, 2023-07-02T14:03:56.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:03:56.684 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:56.684 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:56.684 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:56.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:56.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:03:56.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:56.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:56.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:03:56.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:56.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:56.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:03:56.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:03:56.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:56.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:03:56.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:03:56.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:56.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:03:56.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:03:56.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:56.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:03:56.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:03:56.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:56.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:03:56.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:03:56.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:56.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:03:56.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:03:56.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:56.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:03:56.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:03:56.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:56.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:03:56.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:03:56.690 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.691 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.691 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.691 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:03:56.691 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:03:56.691 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 16.748748075000002, 2023-07-02T14:03:56.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:03:56.692 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:56.692 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:56.692 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:56.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:56.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:03:56.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:56.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:56.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:03:56.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:56.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:56.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:03:56.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:03:56.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:56.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:03:56.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:03:56.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:56.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:03:56.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:03:56.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:56.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:03:56.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:03:56.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:56.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:03:56.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:03:56.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:56.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:03:56.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:03:56.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:56.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:03:56.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:03:56.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:56.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:03:56.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:03:56.698 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.698 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.698 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.699 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6124 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1328, mlcod=81'1319)", 2023-07-02T14:03:56.699 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:03:56.699 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 16.745232691999998, 2023-07-02T14:03:56.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019096974999999999, 2023-07-02T14:03:56.699 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:56.699 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:56.700 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:56.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:56.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:03:56.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:56.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:56.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:03:56.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:56.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:56.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912602+0000", 2023-07-02T14:03:56.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6741999999999999e-05 2023-07-02T14:03:56.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:56.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912629+0000", 2023-07-02T14:03:56.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5163e-05 2023-07-02T14:03:56.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:56.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912644+0000", 2023-07-02T14:03:56.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4800000000000001e-06 2023-07-02T14:03:56.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:56.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912653+0000", 2023-07-02T14:03:56.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071988909999999998 2023-07-02T14:03:56.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:56.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.919851+0000", 2023-07-02T14:03:56.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2949999999999999e-05 2023-07-02T14:03:56.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:56.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.919874+0000", 2023-07-02T14:03:56.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011800804 2023-07-02T14:03:56.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:56.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931675+0000", 2023-07-02T14:03:56.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6033999999999998e-05 2023-07-02T14:03:56.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:56.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931701+0000", 2023-07-02T14:03:56.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019096974999999999 2023-07-02T14:03:56.706 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.706 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.706 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.706 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6125 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1330, mlcod=81'1319)", 2023-07-02T14:03:56.707 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.915930+0000", 2023-07-02T14:03:56.707 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 16.741907006999998, 2023-07-02T14:03:56.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015810346999999999, 2023-07-02T14:03:56.707 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:56.707 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:56.707 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:56.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:56.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.915930+0000", 2023-07-02T14:03:56.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:56.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:56.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.915930+0000", 2023-07-02T14:03:56.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:56.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:56.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.915929+0000", 2023-07-02T14:03:56.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8398e-05 2023-07-02T14:03:56.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:56.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.915978+0000", 2023-07-02T14:03:56.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2022e-05 2023-07-02T14:03:56.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:56.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.916000+0000", 2023-07-02T14:03:56.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2780000000000001e-06 2023-07-02T14:03:56.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:56.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.916005+0000", 2023-07-02T14:03:56.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078621239999999998 2023-07-02T14:03:56.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:56.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923867+0000", 2023-07-02T14:03:56.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1858999999999999e-05 2023-07-02T14:03:56.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:56.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923889+0000", 2023-07-02T14:03:56.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078253369999999999 2023-07-02T14:03:56.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:56.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931714+0000", 2023-07-02T14:03:56.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5936999999999999e-05 2023-07-02T14:03:56.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:56.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931740+0000", 2023-07-02T14:03:56.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015810346999999999 2023-07-02T14:03:56.714 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.714 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.714 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.714 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6137 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1335, mlcod=81'1333)", 2023-07-02T14:03:56.714 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.934629+0000", 2023-07-02T14:03:56.715 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 16.723207865999999, 2023-07-02T14:03:56.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014672614, 2023-07-02T14:03:56.715 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:56.715 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:56.715 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:56.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:56.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.934629+0000", 2023-07-02T14:03:56.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:56.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:56.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.934629+0000", 2023-07-02T14:03:56.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:56.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:56.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.934628+0000", 2023-07-02T14:03:56.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030856529999999998 2023-07-02T14:03:56.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:56.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.937714+0000", 2023-07-02T14:03:56.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6008999999999999e-05 2023-07-02T14:03:56.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:56.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.937730+0000", 2023-07-02T14:03:56.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9440000000000005e-06 2023-07-02T14:03:56.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:56.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.937738+0000", 2023-07-02T14:03:56.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4077000000000003e-05 2023-07-02T14:03:56.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:56.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.937802+0000", 2023-07-02T14:03:56.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5251e-05 2023-07-02T14:03:56.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:56.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.937817+0000", 2023-07-02T14:03:56.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011457106 2023-07-02T14:03:56.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:56.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949275+0000", 2023-07-02T14:03:56.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.7217999999999999e-05 2023-07-02T14:03:56.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:56.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949302+0000", 2023-07-02T14:03:56.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014672614 2023-07-02T14:03:56.721 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.722 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.722 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.722 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6139 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1337, mlcod=81'1333)", 2023-07-02T14:03:56.722 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:03:56.722 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 16.718542832000001, 2023-07-02T14:03:56.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018571681, 2023-07-02T14:03:56.723 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:56.723 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:56.723 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:56.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:56.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:03:56.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:56.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:56.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:03:56.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:56.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:56.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939293+0000", 2023-07-02T14:03:56.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003764613 2023-07-02T14:03:56.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:56.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943058+0000", 2023-07-02T14:03:56.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5797000000000001e-05 2023-07-02T14:03:56.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:56.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943074+0000", 2023-07-02T14:03:56.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5089999999999996e-06 2023-07-02T14:03:56.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:56.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943081+0000", 2023-07-02T14:03:56.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014727100000000001 2023-07-02T14:03:56.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:56.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943228+0000", 2023-07-02T14:03:56.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2246000000000003e-05 2023-07-02T14:03:56.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:56.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943261+0000", 2023-07-02T14:03:56.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014576317 2023-07-02T14:03:56.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:56.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957837+0000", 2023-07-02T14:03:56.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8962000000000002e-05 2023-07-02T14:03:56.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:56.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957866+0000", 2023-07-02T14:03:56.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018571681 2023-07-02T14:03:56.729 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.729 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.729 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.730 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6140 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1338, mlcod=81'1333)", 2023-07-02T14:03:56.730 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.943097+0000", 2023-07-02T14:03:56.730 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 16.714740028000001, 2023-07-02T14:03:56.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014787599, 2023-07-02T14:03:56.730 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:56.730 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:56.731 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:56.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:56.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943097+0000", 2023-07-02T14:03:56.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:56.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:56.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943097+0000", 2023-07-02T14:03:56.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:56.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:56.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943096+0000", 2023-07-02T14:03:56.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4756999999999999e-05 2023-07-02T14:03:56.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943111+0000", 2023-07-02T14:03:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9079999999999995e-06 2023-07-02T14:03:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943119+0000", 2023-07-02T14:03:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4150000000000001e-06 2023-07-02T14:03:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943123+0000", 2023-07-02T14:03:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005177999 2023-07-02T14:03:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.948301+0000", 2023-07-02T14:03:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5087000000000003e-05 2023-07-02T14:03:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.948356+0000", 2023-07-02T14:03:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0095146889999999998 2023-07-02T14:03:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957871+0000", 2023-07-02T14:03:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.347e-05 2023-07-02T14:03:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957885+0000", 2023-07-02T14:03:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014787599 2023-07-02T14:03:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6141 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1339, mlcod=81'1333)", 2023-07-02T14:03:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.943978+0000", 2023-07-02T14:03:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 16.713858975000001, 2023-07-02T14:03:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013921674, 2023-07-02T14:03:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943978+0000", 2023-07-02T14:03:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943978+0000", 2023-07-02T14:03:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943977+0000", 2023-07-02T14:03:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0682999999999999e-05 2023-07-02T14:03:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943988+0000", 2023-07-02T14:03:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0648999999999999e-05 2023-07-02T14:03:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943999+0000", 2023-07-02T14:03:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3510000000000001e-06 2023-07-02T14:03:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944004+0000", 2023-07-02T14:03:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053320290000000003 2023-07-02T14:03:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949336+0000", 2023-07-02T14:03:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5068e-05 2023-07-02T14:03:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949351+0000", 2023-07-02T14:03:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0085357869999999995 2023-07-02T14:03:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957887+0000", 2023-07-02T14:03:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2781e-05 2023-07-02T14:03:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957900+0000", 2023-07-02T14:03:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013921674 2023-07-02T14:03:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:03:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:03:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 16.713043276000001, 2023-07-02T14:03:56.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:03:56.746 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:56.746 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:56.746 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:56.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:56.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:03:56.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:03:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:03:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:03:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:03:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:03:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:03:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:03:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:03:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:03:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:03:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:03:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:03:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:03:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:03:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:03:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:03:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:03:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.753 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.753 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:03:56.753 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:03:56.753 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 16.708385918000001, 2023-07-02T14:03:56.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:03:56.753 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:56.754 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:56.754 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:56.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:56.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:03:56.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:56.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:56.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:03:56.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:56.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:56.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:03:56.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:03:56.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:56.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:03:56.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:03:56.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:56.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:03:56.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:03:56.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:56.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:03:56.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:03:56.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:56.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:03:56.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:03:56.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:56.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:03:56.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:03:56.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:56.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:03:56.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:03:56.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:56.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:03:56.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:03:56.760 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.760 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.760 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:03:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:03:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 16.707446054999998, 2023-07-02T14:03:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:03:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:56.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:56.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:03:56.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:56.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:56.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:03:56.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:56.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:56.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:03:56.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:03:56.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:03:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:03:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:03:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:03:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:03:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:03:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:03:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:03:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:03:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:03:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:03:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:03:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:03:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:03:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:03:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:03:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 16.706048177, 2023-07-02T14:03:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:03:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:03:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:03:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:03:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:03:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:03:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:03:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:03:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:03:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:03:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:03:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:03:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:03:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:03:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:03:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:03:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:03:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:03:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:03:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6148 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1344, mlcod=81'1335)", 2023-07-02T14:03:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.956703+0000", 2023-07-02T14:03:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 16.701133885000001, 2023-07-02T14:03:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017064030000000001, 2023-07-02T14:03:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956703+0000", 2023-07-02T14:03:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956703+0000", 2023-07-02T14:03:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956702+0000", 2023-07-02T14:03:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012505 2023-07-02T14:03:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956827+0000", 2023-07-02T14:03:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7309999999999999e-05 2023-07-02T14:03:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956845+0000", 2023-07-02T14:03:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7570000000000001e-06 2023-07-02T14:03:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956850+0000", 2023-07-02T14:03:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071775149999999998 2023-07-02T14:03:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964028+0000", 2023-07-02T14:03:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4773000000000002e-05 2023-07-02T14:03:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964053+0000", 2023-07-02T14:03:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096921820000000006 2023-07-02T14:03:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973745+0000", 2023-07-02T14:03:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2272999999999998e-05 2023-07-02T14:03:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973767+0000", 2023-07-02T14:03:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017064030000000001 2023-07-02T14:03:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6150 2.0 e81/46 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 81'1345, mlcod=81'1338)", 2023-07-02T14:03:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:03:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 16.698025229999999, 2023-07-02T14:03:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019172812000000001, 2023-07-02T14:03:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:03:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:03:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:03:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959810+0000", 2023-07-02T14:03:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048438650000000001 2023-07-02T14:03:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964653+0000", 2023-07-02T14:03:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2136e-05 2023-07-02T14:03:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964666+0000", 2023-07-02T14:03:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2169999999999999e-06 2023-07-02T14:03:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964672+0000", 2023-07-02T14:03:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00047561800000000003 2023-07-02T14:03:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965147+0000", 2023-07-02T14:03:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5891e-05 2023-07-02T14:03:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965173+0000", 2023-07-02T14:03:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013770973000000001 2023-07-02T14:03:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978944+0000", 2023-07-02T14:03:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0256999999999998e-05 2023-07-02T14:03:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978985+0000", 2023-07-02T14:03:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019172812000000001 2023-07-02T14:03:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6151 2.0 e81/46 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 81'1346, mlcod=81'1338)", 2023-07-02T14:03:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.964682+0000", 2023-07-02T14:03:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 16.693155429000001, 2023-07-02T14:03:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014325582, 2023-07-02T14:03:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964682+0000", 2023-07-02T14:03:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964682+0000", 2023-07-02T14:03:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964681+0000", 2023-07-02T14:03:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5908000000000001e-05 2023-07-02T14:03:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964697+0000", 2023-07-02T14:03:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1920000000000005e-06 2023-07-02T14:03:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964705+0000", 2023-07-02T14:03:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.493e-06 2023-07-02T14:03:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964709+0000", 2023-07-02T14:03:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075784019999999997 2023-07-02T14:03:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972288+0000", 2023-07-02T14:03:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3762e-05 2023-07-02T14:03:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972302+0000", 2023-07-02T14:03:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066885219999999997 2023-07-02T14:03:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978990+0000", 2023-07-02T14:03:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7011999999999999e-05 2023-07-02T14:03:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.979007+0000", 2023-07-02T14:03:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014325582 2023-07-02T14:03:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6153 2.0 e81/46)", 2023-07-02T14:03:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.964714+0000", 2023-07-02T14:03:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 16.693122618, 2023-07-02T14:03:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014315782000000001, 2023-07-02T14:03:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964714+0000", 2023-07-02T14:03:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964714+0000", 2023-07-02T14:03:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964714+0000", 2023-07-02T14:03:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2535999999999999e-05 2023-07-02T14:03:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964737+0000", 2023-07-02T14:03:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.892e-06 2023-07-02T14:03:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964744+0000", 2023-07-02T14:03:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5040000000000004e-06 2023-07-02T14:03:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964749+0000", 2023-07-02T14:03:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0083469990000000008 2023-07-02T14:03:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973096+0000", 2023-07-02T14:03:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8771e-05 2023-07-02T14:03:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973115+0000", 2023-07-02T14:03:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058955850000000001 2023-07-02T14:03:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.979010+0000", 2023-07-02T14:03:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9847000000000002e-05 2023-07-02T14:03:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.979030+0000", 2023-07-02T14:03:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014315782000000001 2023-07-02T14:03:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6320 2.7 e81/63 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head v 81'852, mlcod=81'849)", 2023-07-02T14:03:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:40.070929+0000", 2023-07-02T14:03:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 16.586907908000001, 2023-07-02T14:03:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014144276000000001, 2023-07-02T14:03:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.070929+0000", 2023-07-02T14:03:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.070929+0000", 2023-07-02T14:03:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.070928+0000", 2023-07-02T14:03:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0010305239999999999 2023-07-02T14:03:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.071959+0000", 2023-07-02T14:03:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8085999999999999e-05 2023-07-02T14:03:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.071977+0000", 2023-07-02T14:03:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3080000000000008e-06 2023-07-02T14:03:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.071985+0000", 2023-07-02T14:03:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7088999999999994e-05 2023-07-02T14:03:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.072052+0000", 2023-07-02T14:03:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1041000000000003e-05 2023-07-02T14:03:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.072083+0000", 2023-07-02T14:03:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012949866000000001 2023-07-02T14:03:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.085033+0000", 2023-07-02T14:03:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0262000000000002e-05 2023-07-02T14:03:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.085073+0000", 2023-07-02T14:03:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014144276000000001 2023-07-02T14:03:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:56.815 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:56.816 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:56.815+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:56.816 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:03:56.817 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op export --pgid 1.0 --file /var/log/ceph/exp.1.0.3 2023-07-02T14:03:56.916 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:56.916+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:56.986 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:56.986 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:56.986 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:03:56.987 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:56.997 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:03:57.038 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:57.037+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:57.118 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:57.117+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:57.160 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:57.161 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:03:57.161 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:03:57.161 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:03:57.161 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:57.173 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:03:57.219 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:57.218+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:57.312 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:03:57.312 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:03:57.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:03:57.313 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:03:57.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.313 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:57.313 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:57.313 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 78.019418231000003, 2023-07-02T14:03:57.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:03:57.313 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:57.314 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.314 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:57.314 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:57.314 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.314 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:03:57.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.314 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:57.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:57.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:57.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:57.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:57.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:03:57.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:57.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:57.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:03:57.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:03:57.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:57.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:03:57.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:03:57.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:57.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:03:57.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:03:57.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:03:57.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:03:57.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:57.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:03:57.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:03:57.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:57.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:03:57.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:03:57.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:57.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:03:57.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:03:57.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:03:57.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:03:57.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:57.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:03:57.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:03:57.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.322 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:57.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:03:57.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:03:57.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.322 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:57.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:03:57.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:03:57.323 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.323 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.323 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.323 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:57.323 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:57.324 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 78.019183208000001, 2023-07-02T14:03:57.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:03:57.324 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:57.324 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.324 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:57.324 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:57.324 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.325 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:03:57.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.325 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:57.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:57.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:57.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:57.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:57.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:03:57.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:57.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:57.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:03:57.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:03:57.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:57.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:03:57.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:03:57.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:57.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:03:57.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:03:57.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:03:57.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:03:57.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:57.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:03:57.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:03:57.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:57.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:03:57.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:03:57.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:57.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:03:57.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:03:57.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:03:57.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:03:57.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:57.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:03:57.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:03:57.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:57.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:03:57.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:03:57.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:57.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:03:57.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:03:57.333 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.333 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.333 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.334 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:57.334 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:57.334 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 78.018984074000002, 2023-07-02T14:03:57.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:03:57.334 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:57.334 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.334 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:57.335 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:57.335 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.335 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:03:57.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.335 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:57.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:57.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:57.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:57.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:57.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:03:57.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:57.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:57.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:03:57.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:03:57.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:57.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:03:57.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:03:57.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:57.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:03:57.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:03:57.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:03:57.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:03:57.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:57.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:03:57.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:03:57.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:57.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:03:57.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:03:57.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:57.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:03:57.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:03:57.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:03:57.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:03:57.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:57.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:03:57.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:03:57.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:57.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:03:57.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:03:57.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:57.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:03:57.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:03:57.343 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.343 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.344 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.344 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:57.344 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:57.344 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 78.018618950000004, 2023-07-02T14:03:57.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:03:57.345 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:57.345 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.345 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:57.345 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:57.345 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.345 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:03:57.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.345 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:57.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:57.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:57.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:57.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:57.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:03:57.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:57.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:57.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:03:57.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:03:57.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:57.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:03:57.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:03:57.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:57.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:03:57.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:03:57.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:03:57.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:03:57.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:57.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:03:57.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:03:57.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:57.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:03:57.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:03:57.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:57.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:03:57.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:03:57.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:03:57.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:03:57.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:57.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:03:57.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:03:57.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:57.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:03:57.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:03:57.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:57.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:03:57.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:03:57.354 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.354 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.354 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.354 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:57.354 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:57.355 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 78.018537889000001, 2023-07-02T14:03:57.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:03:57.355 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:57.355 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.355 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:57.355 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:57.355 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.355 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:03:57.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.356 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:57.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:57.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:57.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:57.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:57.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:03:57.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:57.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:57.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:03:57.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:03:57.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:57.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:03:57.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:03:57.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:57.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:03:57.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:03:57.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:03:57.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:03:57.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:57.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:03:57.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:03:57.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:57.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:03:57.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:03:57.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:57.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:03:57.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:03:57.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:03:57.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:03:57.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:57.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:03:57.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:03:57.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:57.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:03:57.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:03:57.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:57.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:03:57.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:03:57.364 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.364 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.364 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.365 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:57.365 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:57.365 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 78.018446217999994, 2023-07-02T14:03:57.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:03:57.365 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:57.365 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.365 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:57.366 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:57.366 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.366 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:03:57.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.366 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:57.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:57.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:57.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:57.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:57.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:03:57.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:57.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:57.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:03:57.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:03:57.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:57.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:03:57.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:03:57.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:57.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:03:57.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:03:57.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:03:57.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:03:57.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:57.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:03:57.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:03:57.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:57.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:03:57.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:03:57.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:57.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:03:57.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:03:57.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:57.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:03:57.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:03:57.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:03:57.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:03:57.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:57.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:03:57.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:03:57.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:57.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:03:57.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:03:57.374 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.374 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.375 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.375 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:57.375 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:57.375 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 78.018106775999996, 2023-07-02T14:03:57.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:03:57.375 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:57.376 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.376 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:57.376 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:57.376 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.376 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:03:57.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.376 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:57.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:57.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:57.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:57.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:57.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:03:57.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:57.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:57.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:03:57.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:03:57.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:57.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:03:57.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:03:57.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:57.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:03:57.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:03:57.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:03:57.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:03:57.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:57.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:03:57.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:03:57.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:57.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:03:57.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:03:57.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:57.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:03:57.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:03:57.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:57.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:03:57.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:03:57.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:03:57.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:03:57.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:57.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:03:57.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:03:57.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:57.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:03:57.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:03:57.385 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.385 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.385 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.385 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:57.385 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:57.385 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 78.017920723000003, 2023-07-02T14:03:57.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:03:57.386 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:57.386 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.386 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:57.386 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:57.386 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.386 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:03:57.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.387 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:57.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:57.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:57.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:57.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:57.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:57.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:57.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:57.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:03:57.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:03:57.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:57.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:03:57.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:03:57.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:57.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:03:57.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:03:57.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:03:57.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:03:57.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:57.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:03:57.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:03:57.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:57.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:03:57.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:03:57.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:57.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:03:57.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:03:57.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:57.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:03:57.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:03:57.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:03:57.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:03:57.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:57.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:03:57.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:03:57.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:57.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:03:57.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:03:57.395 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.395 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.395 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.395 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:57.396 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:57.396 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 78.017762920999999, 2023-07-02T14:03:57.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:03:57.396 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:57.396 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.396 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:57.396 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:57.396 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.397 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:03:57.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.397 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:57.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:57.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:57.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:57.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:57.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:03:57.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:57.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:57.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:03:57.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:03:57.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:57.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:03:57.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:03:57.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:57.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:03:57.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:03:57.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:03:57.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:03:57.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:57.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:03:57.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:03:57.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:57.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:03:57.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:03:57.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:57.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:03:57.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:03:57.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:57.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:03:57.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:03:57.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:03:57.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:03:57.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.404 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:57.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:03:57.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:03:57.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:57.405 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:03:57.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:03:57.405 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.405 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.405 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.406 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:57.406 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:57.406 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 78.017571138999998, 2023-07-02T14:03:57.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:03:57.406 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:57.406 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.406 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:57.407 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:57.407 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.407 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:03:57.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.407 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:57.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:57.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:57.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:57.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:57.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:03:57.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:57.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:57.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:03:57.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:03:57.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:57.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:03:57.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:03:57.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:57.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:03:57.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:03:57.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:03:57.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:03:57.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:57.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:03:57.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:03:57.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:57.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:03:57.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:03:57.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:57.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:03:57.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:03:57.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:03:57.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:03:57.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:57.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:03:57.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:03:57.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:57.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:03:57.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:03:57.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:57.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:03:57.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:03:57.415 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.415 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.416 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.416 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:57.416 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:57.416 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 78.017395479000001, 2023-07-02T14:03:57.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:03:57.416 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:57.417 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.417 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:57.417 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:57.417 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.417 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:03:57.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.417 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:57.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:57.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:57.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:57.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:57.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:57.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:57.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:57.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:03:57.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:03:57.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:57.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:03:57.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:03:57.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:57.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:03:57.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:03:57.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:03:57.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:03:57.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:57.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:03:57.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:03:57.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:57.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:03:57.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:03:57.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:57.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:03:57.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:03:57.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:03:57.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:03:57.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:57.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:03:57.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:03:57.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:57.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:03:57.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:03:57.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:57.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:03:57.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:03:57.426 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.426 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.426 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.426 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:57.426 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:57.426 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 78.017301290999995, 2023-07-02T14:03:57.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:03:57.427 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:57.427 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.427 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:57.427 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:57.427 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.427 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:03:57.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.428 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:57.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:57.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:57.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:57.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:57.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:03:57.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:57.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:57.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:03:57.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:03:57.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:57.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:03:57.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:03:57.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:57.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:03:57.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:03:57.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:03:57.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:03:57.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:57.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:03:57.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:03:57.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:57.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:03:57.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:03:57.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:57.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:03:57.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:03:57.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:03:57.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:03:57.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:57.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:03:57.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:03:57.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:57.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:03:57.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:03:57.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:57.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:03:57.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:03:57.436 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.436 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.436 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.436 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:57.437 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:57.437 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 78.017097905, 2023-07-02T14:03:57.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:03:57.437 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:57.437 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.437 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:57.437 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:57.437 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.438 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:03:57.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.438 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:57.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:57.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:57.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:57.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:57.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:03:57.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:57.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:57.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:03:57.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:03:57.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:57.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:03:57.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:03:57.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:57.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:03:57.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:03:57.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:03:57.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:03:57.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:57.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:03:57.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:03:57.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:57.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:03:57.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:03:57.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:57.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:03:57.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:03:57.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:03:57.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:03:57.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:57.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:03:57.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:03:57.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:57.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:03:57.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:03:57.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:57.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:03:57.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:03:57.446 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.446 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.446 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.447 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:57.447 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:57.447 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 78.016947248999998, 2023-07-02T14:03:57.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:03:57.447 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:57.447 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.447 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:57.448 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:57.448 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.448 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:03:57.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.448 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:57.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:57.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:57.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:57.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:57.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:03:57.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:57.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:57.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:03:57.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:03:57.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:57.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:03:57.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:03:57.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:57.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:03:57.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:03:57.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:03:57.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:03:57.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:57.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:03:57.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:03:57.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:57.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:03:57.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:03:57.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:57.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:03:57.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:03:57.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:03:57.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:03:57.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:57.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:03:57.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:03:57.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:57.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:03:57.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:03:57.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:57.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:03:57.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:03:57.456 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.456 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.457 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.457 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:57.457 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:57.457 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 78.016874545999997, 2023-07-02T14:03:57.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:03:57.457 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:57.458 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.458 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:57.458 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:57.458 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.458 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:03:57.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.458 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:57.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:57.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:57.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:57.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:57.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:57.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:57.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:57.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:03:57.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:03:57.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:57.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:03:57.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:03:57.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:57.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:03:57.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:03:57.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:03:57.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:03:57.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:57.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:03:57.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:03:57.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:57.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:03:57.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:03:57.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:57.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:03:57.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:03:57.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:03:57.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:03:57.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:57.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:03:57.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:03:57.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:57.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:03:57.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:03:57.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:57.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:03:57.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:03:57.467 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.467 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.467 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.467 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:57.467 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:57.467 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 78.016757556000002, 2023-07-02T14:03:57.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:03:57.468 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:57.468 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.468 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:57.468 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:57.468 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.468 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:03:57.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.469 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:57.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:57.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:57.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:57.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:57.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:57.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:57.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:57.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:03:57.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:03:57.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:57.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:03:57.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:03:57.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:57.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:03:57.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:03:57.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:03:57.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:03:57.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:57.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:03:57.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:03:57.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:57.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:03:57.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:03:57.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:57.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:03:57.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:03:57.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:03:57.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:03:57.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:57.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:03:57.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:03:57.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:57.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:03:57.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:03:57.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:57.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:03:57.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:03:57.477 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.477 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.477 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.477 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:57.478 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:57.478 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 78.016608105000003, 2023-07-02T14:03:57.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:03:57.478 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:57.478 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.478 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:57.478 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:57.478 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.479 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:03:57.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.479 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:57.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:57.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:57.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:57.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:57.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:03:57.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:57.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:57.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:03:57.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:03:57.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:57.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:03:57.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:03:57.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:57.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:03:57.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:03:57.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:03:57.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:03:57.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:57.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:03:57.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:03:57.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:57.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:03:57.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:03:57.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:57.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:03:57.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:03:57.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:03:57.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:03:57.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:57.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:03:57.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:03:57.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:57.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:03:57.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:03:57.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:57.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:03:57.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:03:57.487 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.487 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.487 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.488 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:57.488 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:57.488 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 77.995132131999995, 2023-07-02T14:03:57.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:03:57.488 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:57.488 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.488 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:57.489 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:57.489 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.489 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:03:57.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.489 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:57.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:57.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:57.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:57.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:57.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:03:57.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:57.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:57.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:03:57.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:03:57.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:57.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:03:57.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:03:57.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:57.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:03:57.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:03:57.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:03:57.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:03:57.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:57.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:03:57.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:03:57.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:57.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:03:57.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:03:57.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:57.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:03:57.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:03:57.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:57.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:03:57.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:03:57.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:03:57.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:03:57.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:57.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:03:57.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:03:57.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:57.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:03:57.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:03:57.497 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.497 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.498 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.498 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:57.498 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:57.498 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 77.994999596, 2023-07-02T14:03:57.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:03:57.498 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:57.499 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.499 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:57.499 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:57.499 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.499 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:03:57.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.499 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:57.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:57.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:57.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:57.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:57.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:03:57.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:57.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:57.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:03:57.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:03:57.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:57.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:03:57.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:03:57.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:57.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:03:57.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:03:57.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:03:57.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:03:57.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:57.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:03:57.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:03:57.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:57.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:03:57.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:03:57.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:03:57.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:03:57.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:03:57.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:57.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:03:57.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:03:57.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:03:57.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:03:57.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:57.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:03:57.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:03:57.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:57.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:03:57.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:03:57.508 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.508 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.508 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.508 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:03:57.508 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:57.508 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 77.933662935000001, 2023-07-02T14:03:57.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:03:57.509 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:03:57.509 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.509 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:03:57.509 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:03:57.509 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.509 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:03:57.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.510 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:03:57.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:03:57.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:57.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:03:57.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:03:57.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:03:57.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:03:57.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:03:57.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:03:57.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:03:57.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:03:57.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:03:57.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:03:57.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:03:57.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:03:57.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:03:57.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:03:57.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:03:57.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:03:57.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:03:57.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:03:57.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:03:57.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:03:57.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:03:57.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:03:57.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:03:57.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:03:57.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:03:57.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:03:57.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:03:57.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:03:57.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:03:57.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:03:57.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:03:57.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:03:57.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:03:57.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:03:57.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:03:57.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:03:57.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:03:57.518 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.518 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.518 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.518 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:03:57.518 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:03:57.518 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:03:57.520 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:03:57.520 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:57.518+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:57.620 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:57.619+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:57.658 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:57.658 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:57.659 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:57.659 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:57.669 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:03:57.722 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:57.720+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:57.792 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:57.793 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:57.793 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:03:57.793 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:03:57.793 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:57.803 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:03:57.844 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:57.843+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:57.923 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:57.922+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:57.969 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:57.970 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:03:57.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:03:57.970 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:03:57.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.970 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2620 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 196608~786432] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:57.970 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:03:57.970 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.642777670000001, 2023-07-02T14:03:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966, 2023-07-02T14:03:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2620 2023-07-02T14:03:57.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:03:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:57.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:03:57.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:57.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:57.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324571+0000", 2023-07-02T14:03:57.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016686649999999999 2023-07-02T14:03:57.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:57.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326240+0000", 2023-07-02T14:03:57.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1844e-05 2023-07-02T14:03:57.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326252+0000", 2023-07-02T14:03:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7760000000000003e-06 2023-07-02T14:03:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326258+0000", 2023-07-02T14:03:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0634000000000003e-05 2023-07-02T14:03:57.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:57.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326318+0000", 2023-07-02T14:03:57.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9629999999999993e-06 2023-07-02T14:03:57.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:57.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326328+0000", 2023-07-02T14:03:57.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.010877045 2023-07-02T14:03:57.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:57.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337205+0000", 2023-07-02T14:03:57.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022401300000000001 2023-07-02T14:03:57.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337429+0000", 2023-07-02T14:03:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000139822 2023-07-02T14:03:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337569+0000", 2023-07-02T14:03:57.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032942119999999999 2023-07-02T14:03:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340863+0000", 2023-07-02T14:03:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021302560000000001 2023-07-02T14:03:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342994+0000", 2023-07-02T14:03:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5634000000000002e-05 2023-07-02T14:03:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343059+0000", 2023-07-02T14:03:57.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143721 2023-07-02T14:03:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343203+0000", 2023-07-02T14:03:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966 2023-07-02T14:03:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:57.981 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2622 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 983040~36864] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:03:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.641085158999999, 2023-07-02T14:03:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664, 2023-07-02T14:03:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:57.982 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2622 2023-07-02T14:03:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:03:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:03:57.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326265+0000", 2023-07-02T14:03:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1282000000000001e-05 2023-07-02T14:03:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:57.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326286+0000", 2023-07-02T14:03:57.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9480000000000002e-06 2023-07-02T14:03:57.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:57.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326293+0000", 2023-07-02T14:03:57.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3610000000000003e-06 2023-07-02T14:03:57.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326297+0000", 2023-07-02T14:03:57.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7707999999999994e-05 2023-07-02T14:03:57.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:57.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326385+0000", 2023-07-02T14:03:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8070000000000006e-06 2023-07-02T14:03:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326394+0000", 2023-07-02T14:03:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.014535878 2023-07-02T14:03:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340930+0000", 2023-07-02T14:03:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000154551 2023-07-02T14:03:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341084+0000", 2023-07-02T14:03:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00032375900000000002 2023-07-02T14:03:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341408+0000", 2023-07-02T14:03:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012631066 2023-07-02T14:03:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354039+0000", 2023-07-02T14:03:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5741000000000001e-05 2023-07-02T14:03:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354055+0000", 2023-07-02T14:03:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8794000000000006e-05 2023-07-02T14:03:57.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354124+0000", 2023-07-02T14:03:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014667700000000001 2023-07-02T14:03:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354270+0000", 2023-07-02T14:03:57.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664 2023-07-02T14:03:57.993 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:57.993 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:57.993 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:57.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.993 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2727 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~12288] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:03:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.641049383999999, 2023-07-02T14:03:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999, 2023-07-02T14:03:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2727 2023-07-02T14:03:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:03:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:57.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:03:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:57.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:03:57.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3834e-05 2023-07-02T14:03:57.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:57.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326315+0000", 2023-07-02T14:03:57.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3029999999999996e-06 2023-07-02T14:03:57.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326321+0000", 2023-07-02T14:03:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7590000000000002e-06 2023-07-02T14:03:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326324+0000", 2023-07-02T14:03:57.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112872 2023-07-02T14:03:57.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:57.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326437+0000", 2023-07-02T14:03:57.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2290000000000002e-06 2023-07-02T14:03:57.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:57.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:57.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326442+0000", 2023-07-02T14:03:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0155896879999999 2023-07-02T14:03:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342031+0000", 2023-07-02T14:03:58.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016496100000000001 2023-07-02T14:03:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342196+0000", 2023-07-02T14:03:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021998699999999999 2023-07-02T14:03:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342416+0000", 2023-07-02T14:03:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011869744999999999 2023-07-02T14:03:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354286+0000", 2023-07-02T14:03:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0361e-05 2023-07-02T14:03:58.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354347+0000", 2023-07-02T14:03:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3312000000000005e-05 2023-07-02T14:03:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:58.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354430+0000", 2023-07-02T14:03:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126444 2023-07-02T14:03:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354556+0000", 2023-07-02T14:03:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999 2023-07-02T14:03:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.004 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2730 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [set-alloc-hint object_size 4194304 write_size 4194304,write 454656~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:03:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.641024375000001, 2023-07-02T14:03:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999, 2023-07-02T14:03:58.005 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:58.006 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:58.006 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:58.006 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:58.006 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:58.006 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2730 2023-07-02T14:03:58.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.006 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:58.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:58.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:03:58.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:58.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:58.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:03:58.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:58.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:58.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:03:58.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6049999999999997e-06 2023-07-02T14:03:58.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326332+0000", 2023-07-02T14:03:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3000000000000002e-06 2023-07-02T14:03:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326335+0000", 2023-07-02T14:03:58.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.719e-05 2023-07-02T14:03:58.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:58.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326352+0000", 2023-07-02T14:03:58.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011583599999999999 2023-07-02T14:03:58.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326468+0000", 2023-07-02T14:03:58.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.831e-06 2023-07-02T14:03:58.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:58.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326473+0000", 2023-07-02T14:03:58.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.016769915 2023-07-02T14:03:58.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343243+0000", 2023-07-02T14:03:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197352 2023-07-02T14:03:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:58.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343440+0000", 2023-07-02T14:03:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197581 2023-07-02T14:03:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343638+0000", 2023-07-02T14:03:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010980139 2023-07-02T14:03:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354618+0000", 2023-07-02T14:03:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056779530000000003 2023-07-02T14:03:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360296+0000", 2023-07-02T14:03:58.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101109 2023-07-02T14:03:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360397+0000", 2023-07-02T14:03:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000160273 2023-07-02T14:03:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:58.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360557+0000", 2023-07-02T14:03:58.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999 2023-07-02T14:03:58.016 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.016 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:58.016 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.016 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2752 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 1019904~3174400] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:03:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.640978150999999, 2023-07-02T14:03:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001, 2023-07-02T14:03:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:58.017 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:58.018 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2752 2023-07-02T14:03:58.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.018 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:58.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:58.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:03:58.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:58.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:58.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:03:58.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:58.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:58.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326372+0000", 2023-07-02T14:03:58.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052050380000000004 2023-07-02T14:03:58.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:58.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331577+0000", 2023-07-02T14:03:58.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6832000000000001e-05 2023-07-02T14:03:58.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:58.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331594+0000", 2023-07-02T14:03:58.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8220000000000007e-06 2023-07-02T14:03:58.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:58.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331602+0000", 2023-07-02T14:03:58.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9895999999999999e-05 2023-07-02T14:03:58.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331642+0000", 2023-07-02T14:03:58.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:03:58.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:58.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331649+0000", 2023-07-02T14:03:58.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0139141709999999 2023-07-02T14:03:58.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345563+0000", 2023-07-02T14:03:58.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113817 2023-07-02T14:03:58.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:58.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345677+0000", 2023-07-02T14:03:58.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018803400000000001 2023-07-02T14:03:58.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:58.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345865+0000", 2023-07-02T14:03:58.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087950359999999991 2023-07-02T14:03:58.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:58.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354660+0000", 2023-07-02T14:03:58.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059116669999999998 2023-07-02T14:03:58.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:58.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360572+0000", 2023-07-02T14:03:58.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0992000000000002e-05 2023-07-02T14:03:58.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:58.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360633+0000", 2023-07-02T14:03:58.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013601600000000001 2023-07-02T14:03:58.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:58.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360769+0000", 2023-07-02T14:03:58.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001 2023-07-02T14:03:58.027 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.028 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:58.028 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.028 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2759 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 0~454656] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:58.028 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:03:58.028 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.635738353999997, 2023-07-02T14:03:58.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999, 2023-07-02T14:03:58.029 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:58.029 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:58.029 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:58.029 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:58.029 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:58.029 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2759 2023-07-02T14:03:58.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.029 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:58.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:58.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:03:58.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:58.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:58.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:03:58.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:03:58.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:58.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331611+0000", 2023-07-02T14:03:58.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053795999999999998 2023-07-02T14:03:58.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:58.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332149+0000", 2023-07-02T14:03:58.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4970000000000007e-06 2023-07-02T14:03:58.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:58.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332158+0000", 2023-07-02T14:03:58.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:03:58.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:58.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332162+0000", 2023-07-02T14:03:58.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3526000000000001e-05 2023-07-02T14:03:58.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332185+0000", 2023-07-02T14:03:58.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:03:58.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:58.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332189+0000", 2023-07-02T14:03:58.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0186295910000001 2023-07-02T14:03:58.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350819+0000", 2023-07-02T14:03:58.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000111123 2023-07-02T14:03:58.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:58.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350930+0000", 2023-07-02T14:03:58.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019575300000000001 2023-07-02T14:03:58.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:58.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.351126+0000", 2023-07-02T14:03:58.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011503298 2023-07-02T14:03:58.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:58.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362629+0000", 2023-07-02T14:03:58.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085443989999999994 2023-07-02T14:03:58.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:58.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371173+0000", 2023-07-02T14:03:58.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3547999999999995e-05 2023-07-02T14:03:58.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:58.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371237+0000", 2023-07-02T14:03:58.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790300000000001 2023-07-02T14:03:58.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:58.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371375+0000", 2023-07-02T14:03:58.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999 2023-07-02T14:03:58.039 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.039 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:58.039 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.040 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2760 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 458752~131072] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:58.040 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:03:58.040 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.635182168999997, 2023-07-02T14:03:58.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001, 2023-07-02T14:03:58.040 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:58.040 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:58.040 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:58.040 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:58.041 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:58.041 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2760 2023-07-02T14:03:58.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.041 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:58.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:58.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:03:58.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:58.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:58.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:03:58.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:58.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:58.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332168+0000", 2023-07-02T14:03:58.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013281100000000001 2023-07-02T14:03:58.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:58.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332301+0000", 2023-07-02T14:03:58.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7389999999999999e-06 2023-07-02T14:03:58.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:58.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332308+0000", 2023-07-02T14:03:58.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0639999999999998e-06 2023-07-02T14:03:58.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:58.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332312+0000", 2023-07-02T14:03:58.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1735000000000001e-05 2023-07-02T14:03:58.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332333+0000", 2023-07-02T14:03:58.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8342000000000002e-05 2023-07-02T14:03:58.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:58.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332352+0000", 2023-07-02T14:03:58.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02234782 2023-07-02T14:03:58.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354699+0000", 2023-07-02T14:03:58.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110371 2023-07-02T14:03:58.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:58.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354810+0000", 2023-07-02T14:03:58.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022174900000000001 2023-07-02T14:03:58.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:58.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355032+0000", 2023-07-02T14:03:58.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076053329999999997 2023-07-02T14:03:58.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:58.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362637+0000", 2023-07-02T14:03:58.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087913609999999993 2023-07-02T14:03:58.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:58.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371428+0000", 2023-07-02T14:03:58.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8532999999999998e-05 2023-07-02T14:03:58.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:58.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371477+0000", 2023-07-02T14:03:58.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010990800000000001 2023-07-02T14:03:58.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:58.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371587+0000", 2023-07-02T14:03:58.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001 2023-07-02T14:03:58.050 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.051 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:58.051 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.051 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2766 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 589824~45056] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:58.051 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:03:58.051 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.635033544000002, 2023-07-02T14:03:58.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188, 2023-07-02T14:03:58.052 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:58.052 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:58.052 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:58.052 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:58.052 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:58.052 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2766 2023-07-02T14:03:58.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.053 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:58.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:58.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:03:58.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:58.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:58.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:03:58.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:58.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:58.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:03:58.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2247999999999998e-05 2023-07-02T14:03:58.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:58.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332369+0000", 2023-07-02T14:03:58.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4189999999999996e-06 2023-07-02T14:03:58.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:58.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332376+0000", 2023-07-02T14:03:58.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1049999999999999e-06 2023-07-02T14:03:58.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:58.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332379+0000", 2023-07-02T14:03:58.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6894000000000003e-05 2023-07-02T14:03:58.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332416+0000", 2023-07-02T14:03:58.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3970000000000002e-06 2023-07-02T14:03:58.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:58.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332422+0000", 2023-07-02T14:03:58.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0231382280000001 2023-07-02T14:03:58.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355560+0000", 2023-07-02T14:03:58.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9079000000000005e-05 2023-07-02T14:03:58.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:58.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355659+0000", 2023-07-02T14:03:58.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018005000000000001 2023-07-02T14:03:58.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:58.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355839+0000", 2023-07-02T14:03:58.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068044439999999998 2023-07-02T14:03:58.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:58.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362644+0000", 2023-07-02T14:03:58.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090155589999999994 2023-07-02T14:03:58.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:58.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371659+0000", 2023-07-02T14:03:58.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1297999999999994e-05 2023-07-02T14:03:58.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:58.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371730+0000", 2023-07-02T14:03:58.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013100699999999999 2023-07-02T14:03:58.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:58.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371861+0000", 2023-07-02T14:03:58.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188 2023-07-02T14:03:58.062 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.062 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:58.062 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.063 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2784 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 0~393216] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:58.063 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:03:58.063 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.634965092000002, 2023-07-02T14:03:58.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999, 2023-07-02T14:03:58.063 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:58.063 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:58.063 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:58.064 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:58.064 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:58.064 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2784 2023-07-02T14:03:58.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.064 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:58.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:58.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:03:58.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:58.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:58.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:03:58.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:58.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:58.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332385+0000", 2023-07-02T14:03:58.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040252400000000002 2023-07-02T14:03:58.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:58.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332787+0000", 2023-07-02T14:03:58.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0208e-05 2023-07-02T14:03:58.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:58.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332798+0000", 2023-07-02T14:03:58.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2023-07-02T14:03:58.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:58.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332801+0000", 2023-07-02T14:03:58.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4838000000000001e-05 2023-07-02T14:03:58.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332836+0000", 2023-07-02T14:03:58.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5319999999999998e-06 2023-07-02T14:03:58.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:58.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332839+0000", 2023-07-02T14:03:58.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0234290109999999 2023-07-02T14:03:58.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356268+0000", 2023-07-02T14:03:58.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011165700000000001 2023-07-02T14:03:58.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:58.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356380+0000", 2023-07-02T14:03:58.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022504799999999999 2023-07-02T14:03:58.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:58.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356605+0000", 2023-07-02T14:03:58.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015276201 2023-07-02T14:03:58.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:58.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371881+0000", 2023-07-02T14:03:58.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5496999999999999e-05 2023-07-02T14:03:58.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:58.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371957+0000", 2023-07-02T14:03:58.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4570000000000002e-05 2023-07-02T14:03:58.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:58.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372001+0000", 2023-07-02T14:03:58.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3880000000000003e-05 2023-07-02T14:03:58.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:58.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372085+0000", 2023-07-02T14:03:58.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999 2023-07-02T14:03:58.074 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.074 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:58.074 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.074 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2785 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 405504~446464] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:58.074 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:03:58.074 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.634543641, 2023-07-02T14:03:58.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999, 2023-07-02T14:03:58.075 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:58.075 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:58.075 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:58.075 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:58.075 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:58.075 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2785 2023-07-02T14:03:58.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.076 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:58.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:58.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:03:58.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:58.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:58.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:03:58.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:58.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:58.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:03:58.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059527500000000004 2023-07-02T14:03:58.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:58.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333402+0000", 2023-07-02T14:03:58.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7059999999999993e-06 2023-07-02T14:03:58.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:58.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333411+0000", 2023-07-02T14:03:58.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0450000000000001e-06 2023-07-02T14:03:58.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:58.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333414+0000", 2023-07-02T14:03:58.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5790999999999997e-05 2023-07-02T14:03:58.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:03:58.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3320000000000002e-06 2023-07-02T14:03:58.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:58.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333454+0000", 2023-07-02T14:03:58.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.025310269 2023-07-02T14:03:58.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358764+0000", 2023-07-02T14:03:58.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012414899999999999 2023-07-02T14:03:58.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:58.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358888+0000", 2023-07-02T14:03:58.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000217028 2023-07-02T14:03:58.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:58.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.359105+0000", 2023-07-02T14:03:58.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012788212 2023-07-02T14:03:58.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:58.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371893+0000", 2023-07-02T14:03:58.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012201569999999999 2023-07-02T14:03:58.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:58.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373114+0000", 2023-07-02T14:03:58.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4512999999999995e-05 2023-07-02T14:03:58.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:58.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373198+0000", 2023-07-02T14:03:58.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132373 2023-07-02T14:03:58.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:58.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373330+0000", 2023-07-02T14:03:58.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999 2023-07-02T14:03:58.085 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.085 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:58.085 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.086 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2786 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1310720~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:58.086 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:03:58.086 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.633930948, 2023-07-02T14:03:58.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001, 2023-07-02T14:03:58.086 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:58.086 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:58.086 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:58.087 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:58.087 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:58.087 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2786 2023-07-02T14:03:58.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.087 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:58.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:58.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:03:58.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:58.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:58.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:03:58.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:58.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:58.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333419+0000", 2023-07-02T14:03:58.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2009999999999999e-06 2023-07-02T14:03:58.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:58.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333427+0000", 2023-07-02T14:03:58.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0210000000000001e-06 2023-07-02T14:03:58.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:58.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333431+0000", 2023-07-02T14:03:58.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3769999999999999e-06 2023-07-02T14:03:58.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:58.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333434+0000", 2023-07-02T14:03:58.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1161e-05 2023-07-02T14:03:58.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333485+0000", 2023-07-02T14:03:58.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1549000000000001e-05 2023-07-02T14:03:58.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:58.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333496+0000", 2023-07-02T14:03:58.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0273116339999999 2023-07-02T14:03:58.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360808+0000", 2023-07-02T14:03:58.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000115128 2023-07-02T14:03:58.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:58.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360923+0000", 2023-07-02T14:03:58.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018367700000000001 2023-07-02T14:03:58.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:58.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361107+0000", 2023-07-02T14:03:58.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010796923999999999 2023-07-02T14:03:58.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:58.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371904+0000", 2023-07-02T14:03:58.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014983520000000001 2023-07-02T14:03:58.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:58.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373402+0000", 2023-07-02T14:03:58.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0520999999999999e-05 2023-07-02T14:03:58.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:58.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373463+0000", 2023-07-02T14:03:58.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000129095 2023-07-02T14:03:58.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:58.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373592+0000", 2023-07-02T14:03:58.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001 2023-07-02T14:03:58.097 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.097 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:58.097 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.097 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2787 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1433600~8192] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:58.097 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:03:58.097 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.633914148000002, 2023-07-02T14:03:58.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999, 2023-07-02T14:03:58.098 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:58.098 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:58.098 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:58.098 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:58.098 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:58.098 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2787 2023-07-02T14:03:58.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.099 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:58.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:58.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:03:58.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:58.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:58.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:03:58.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:58.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:58.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333436+0000", 2023-07-02T14:03:58.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2731999999999999e-05 2023-07-02T14:03:58.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:58.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:03:58.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7440000000000001e-06 2023-07-02T14:03:58.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:58.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333456+0000", 2023-07-02T14:03:58.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8650000000000003e-06 2023-07-02T14:03:58.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:58.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333460+0000", 2023-07-02T14:03:58.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0077e-05 2023-07-02T14:03:58.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333530+0000", 2023-07-02T14:03:58.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4180000000000001e-06 2023-07-02T14:03:58.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:58.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333533+0000", 2023-07-02T14:03:58.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0284002720000001 2023-07-02T14:03:58.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361933+0000", 2023-07-02T14:03:58.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6781999999999996e-05 2023-07-02T14:03:58.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:58.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362030+0000", 2023-07-02T14:03:58.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018305400000000001 2023-07-02T14:03:58.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:58.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362213+0000", 2023-07-02T14:03:58.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097015109999999995 2023-07-02T14:03:58.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:58.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371915+0000", 2023-07-02T14:03:58.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001742295 2023-07-02T14:03:58.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:58.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373657+0000", 2023-07-02T14:03:58.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5980000000000003e-05 2023-07-02T14:03:58.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:58.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373713+0000", 2023-07-02T14:03:58.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011783800000000001 2023-07-02T14:03:58.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:58.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373831+0000", 2023-07-02T14:03:58.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999 2023-07-02T14:03:58.108 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.108 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:58.108 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.109 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2803 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 851968~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:58.109 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:03:58.109 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.633872296, 2023-07-02T14:03:58.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514, 2023-07-02T14:03:58.109 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:58.109 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:58.109 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:58.109 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:58.110 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:58.110 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2803 2023-07-02T14:03:58.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.110 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:58.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:58.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:03:58.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:58.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:58.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:03:58.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:58.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:58.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:03:58.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4943000000000001e-05 2023-07-02T14:03:58.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:58.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333493+0000", 2023-07-02T14:03:58.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2629999999999997e-06 2023-07-02T14:03:58.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:58.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333500+0000", 2023-07-02T14:03:58.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4159999999999997e-06 2023-07-02T14:03:58.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:58.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333505+0000", 2023-07-02T14:03:58.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4518999999999998e-05 2023-07-02T14:03:58.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333559+0000", 2023-07-02T14:03:58.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5900000000000008e-06 2023-07-02T14:03:58.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:58.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333568+0000", 2023-07-02T14:03:58.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0291061779999999 2023-07-02T14:03:58.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362674+0000", 2023-07-02T14:03:58.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110412 2023-07-02T14:03:58.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:58.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362784+0000", 2023-07-02T14:03:58.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015384599999999999 2023-07-02T14:03:58.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:58.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362938+0000", 2023-07-02T14:03:58.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091682310000000006 2023-07-02T14:03:58.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:58.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372106+0000", 2023-07-02T14:03:58.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017703230000000001 2023-07-02T14:03:58.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:58.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373877+0000", 2023-07-02T14:03:58.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6387000000000002e-05 2023-07-02T14:03:58.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:58.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373923+0000", 2023-07-02T14:03:58.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014099199999999999 2023-07-02T14:03:58.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:58.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374064+0000", 2023-07-02T14:03:58.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514 2023-07-02T14:03:58.119 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.120 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:58.120 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.120 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2805 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 634880~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:58.120 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:03:58.120 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.633839121000001, 2023-07-02T14:03:58.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814, 2023-07-02T14:03:58.121 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:58.121 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:58.121 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:58.121 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:58.121 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:58.121 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2805 2023-07-02T14:03:58.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.122 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:58.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:58.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:03:58.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:58.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:58.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:03:58.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:58.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:58.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333511+0000", 2023-07-02T14:03:58.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207092 2023-07-02T14:03:58.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:58.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333718+0000", 2023-07-02T14:03:58.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5110000000000003e-06 2023-07-02T14:03:58.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:58.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333727+0000", 2023-07-02T14:03:58.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3270000000000007e-06 2023-07-02T14:03:58.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:58.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333736+0000", 2023-07-02T14:03:58.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0319999999999999e-05 2023-07-02T14:03:58.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333756+0000", 2023-07-02T14:03:58.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1695000000000001e-05 2023-07-02T14:03:58.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:58.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:03:58.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0295750560000001 2023-07-02T14:03:58.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363343+0000", 2023-07-02T14:03:58.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3387999999999994e-05 2023-07-02T14:03:58.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:58.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363437+0000", 2023-07-02T14:03:58.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018373 2023-07-02T14:03:58.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:58.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363620+0000", 2023-07-02T14:03:58.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0084948769999999996 2023-07-02T14:03:58.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:58.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372115+0000", 2023-07-02T14:03:58.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020082170000000001 2023-07-02T14:03:58.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:58.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374123+0000", 2023-07-02T14:03:58.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8875999999999997e-05 2023-07-02T14:03:58.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:58.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374182+0000", 2023-07-02T14:03:58.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0001272 2023-07-02T14:03:58.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:58.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374309+0000", 2023-07-02T14:03:58.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814 2023-07-02T14:03:58.131 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.131 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:58.131 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.132 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2806 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 1359872~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:58.132 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:03:58.132 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.633608789999997, 2023-07-02T14:03:58.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234, 2023-07-02T14:03:58.132 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:58.132 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:58.132 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:58.133 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:58.133 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:58.133 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2806 2023-07-02T14:03:58.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.133 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:58.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:58.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:03:58.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:58.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:58.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:03:58.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:58.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:58.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333741+0000", 2023-07-02T14:03:58.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3978999999999999e-05 2023-07-02T14:03:58.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:58.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333755+0000", 2023-07-02T14:03:58.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:03:58.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:58.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333761+0000", 2023-07-02T14:03:58.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7010000000000001e-06 2023-07-02T14:03:58.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:58.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333764+0000", 2023-07-02T14:03:58.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5086000000000003e-05 2023-07-02T14:03:58.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333799+0000", 2023-07-02T14:03:58.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6389999999999992e-06 2023-07-02T14:03:58.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:58.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333806+0000", 2023-07-02T14:03:58.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.030496675 2023-07-02T14:03:58.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364303+0000", 2023-07-02T14:03:58.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011382700000000001 2023-07-02T14:03:58.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:58.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364417+0000", 2023-07-02T14:03:58.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192058 2023-07-02T14:03:58.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:58.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364609+0000", 2023-07-02T14:03:58.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075132130000000004 2023-07-02T14:03:58.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:58.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372122+0000", 2023-07-02T14:03:58.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022731750000000001 2023-07-02T14:03:58.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:58.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374395+0000", 2023-07-02T14:03:58.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5277000000000005e-05 2023-07-02T14:03:58.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:58.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374460+0000", 2023-07-02T14:03:58.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017074599999999999 2023-07-02T14:03:58.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:58.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374631+0000", 2023-07-02T14:03:58.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234 2023-07-02T14:03:58.143 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.143 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:58.143 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.143 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 868352~49152] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:03:58.143 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:03:58.143 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.633582076000003, 2023-07-02T14:03:58.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001, 2023-07-02T14:03:58.144 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:58.144 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:58.144 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:58.144 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:58.144 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:58.144 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:03:58.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.145 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:58.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:58.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:03:58.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:58.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:58.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:03:58.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:03:58.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:58.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:03:58.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6355000000000001e-05 2023-07-02T14:03:58.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:58.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333794+0000", 2023-07-02T14:03:58.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8329999999999996e-06 2023-07-02T14:03:58.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:58.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333801+0000", 2023-07-02T14:03:58.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5419999999999996e-06 2023-07-02T14:03:58.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:58.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333807+0000", 2023-07-02T14:03:58.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2023-07-02T14:03:58.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333826+0000", 2023-07-02T14:03:58.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.303e-06 2023-07-02T14:03:58.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:03:58.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333830+0000", 2023-07-02T14:03:58.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.031137559 2023-07-02T14:03:58.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364967+0000", 2023-07-02T14:03:58.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1390000000000004e-05 2023-07-02T14:03:58.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:58.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365059+0000", 2023-07-02T14:03:58.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017597399999999999 2023-07-02T14:03:58.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:58.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365234+0000", 2023-07-02T14:03:58.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068943950000000002 2023-07-02T14:03:58.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:58.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372129+0000", 2023-07-02T14:03:58.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026189910000000002 2023-07-02T14:03:58.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:58.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374748+0000", 2023-07-02T14:03:58.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6613999999999996e-05 2023-07-02T14:03:58.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:58.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374814+0000", 2023-07-02T14:03:58.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502300000000001 2023-07-02T14:03:58.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:58.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374950+0000", 2023-07-02T14:03:58.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001 2023-07-02T14:03:58.154 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.154 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:58.154 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.155 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2815 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 917504~49152] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:58.155 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:03:58.155 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.629367842999997, 2023-07-02T14:03:58.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998, 2023-07-02T14:03:58.155 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:58.155 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:58.155 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:58.156 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:58.156 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:58.156 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2815 2023-07-02T14:03:58.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.156 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:58.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:58.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:03:58.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:58.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:58.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:03:58.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:03:58.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:58.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337980+0000", 2023-07-02T14:03:58.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017126700000000001 2023-07-02T14:03:58.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:58.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338152+0000", 2023-07-02T14:03:58.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2938999999999999e-05 2023-07-02T14:03:58.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:58.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338164+0000", 2023-07-02T14:03:58.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4448999999999999e-05 2023-07-02T14:03:58.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:58.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:03:58.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0281812 2023-07-02T14:03:58.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366370+0000", 2023-07-02T14:03:58.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1840000000000002e-05 2023-07-02T14:03:58.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:58.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366462+0000", 2023-07-02T14:03:58.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000186112 2023-07-02T14:03:58.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:58.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366648+0000", 2023-07-02T14:03:58.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054905709999999996 2023-07-02T14:03:58.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:58.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372139+0000", 2023-07-02T14:03:58.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044954790000000001 2023-07-02T14:03:58.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:58.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376634+0000", 2023-07-02T14:03:58.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4621000000000004e-05 2023-07-02T14:03:58.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:58.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376699+0000", 2023-07-02T14:03:58.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012862299999999999 2023-07-02T14:03:58.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:58.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376827+0000", 2023-07-02T14:03:58.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998 2023-07-02T14:03:58.164 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.164 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:58.165 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.165 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2816 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 917504~65536] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:58.165 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:03:58.165 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.629153463999998, 2023-07-02T14:03:58.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001, 2023-07-02T14:03:58.165 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:58.166 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:58.166 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:58.166 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:58.166 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:58.166 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2816 2023-07-02T14:03:58.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.166 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:58.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:58.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:03:58.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:58.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:58.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:03:58.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:58.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:58.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338196+0000", 2023-07-02T14:03:58.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6286999999999999e-05 2023-07-02T14:03:58.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:58.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338243+0000", 2023-07-02T14:03:58.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7030000000000001e-06 2023-07-02T14:03:58.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:58.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338252+0000", 2023-07-02T14:03:58.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:03:58.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:58.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338258+0000", 2023-07-02T14:03:58.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032085734999999997 2023-07-02T14:03:58.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370344+0000", 2023-07-02T14:03:58.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108621 2023-07-02T14:03:58.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:58.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370452+0000", 2023-07-02T14:03:58.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014936499999999999 2023-07-02T14:03:58.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:58.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370602+0000", 2023-07-02T14:03:58.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001778012 2023-07-02T14:03:58.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:58.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372380+0000", 2023-07-02T14:03:58.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044876380000000004 2023-07-02T14:03:58.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:58.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376867+0000", 2023-07-02T14:03:58.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9870999999999999e-05 2023-07-02T14:03:58.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:58.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376897+0000", 2023-07-02T14:03:58.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011401 2023-07-02T14:03:58.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:58.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.377011+0000", 2023-07-02T14:03:58.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001 2023-07-02T14:03:58.175 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.175 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:58.175 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.175 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2944 2.4 2:25711f4d:::rbd_data.125de1449605.00000000000000fa:head [write 65536~2097152] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:03:58.175 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.470424+0000", 2023-07-02T14:03:58.175 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.496926850999998, 2023-07-02T14:03:58.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029419590999999998, 2023-07-02T14:03:58.176 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:58.176 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:58.176 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:03:58.176 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:03:58.176 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:03:58.176 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2944 2023-07-02T14:03:58.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.177 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:58.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:58.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.470424+0000", 2023-07-02T14:03:58.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:58.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:58.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.470424+0000", 2023-07-02T14:03:58.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:03:58.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:58.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.470421+0000", 2023-07-02T14:03:58.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0086348350000000004 2023-07-02T14:03:58.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:58.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479056+0000", 2023-07-02T14:03:58.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1942000000000001e-05 2023-07-02T14:03:58.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:58.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479078+0000", 2023-07-02T14:03:58.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8130000000000002e-06 2023-07-02T14:03:58.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:58.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479087+0000", 2023-07-02T14:03:58.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4845999999999995e-05 2023-07-02T14:03:58.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479152+0000", 2023-07-02T14:03:58.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013724700000000001 2023-07-02T14:03:58.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:58.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479289+0000", 2023-07-02T14:03:58.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000234665 2023-07-02T14:03:58.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:03:58.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.479524+0000", 2023-07-02T14:03:58.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0081727950000000001 2023-07-02T14:03:58.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:03:58.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.487697+0000", 2023-07-02T14:03:58.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011956079999999999 2023-07-02T14:03:58.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:03:58.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.499653+0000", 2023-07-02T14:03:58.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8978999999999999e-05 2023-07-02T14:03:58.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:58.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.499712+0000", 2023-07-02T14:03:58.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013181299999999999 2023-07-02T14:03:58.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:58.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.499844+0000", 2023-07-02T14:03:58.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029419590999999998 2023-07-02T14:03:58.185 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.185 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:58.185 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.185 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:3878 2.1 e80/79 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head v 80'436, mlcod=80'429)", 2023-07-02T14:03:58.186 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:03:58.186 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.859899967, 2023-07-02T14:03:58.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037799472000000001, 2023-07-02T14:03:58.186 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:03:58.186 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:03:58.186 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:03:58.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:03:58.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:03:58.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:03:58.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:03:58.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:03:58.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:03:58.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:03:58.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107450+0000", 2023-07-02T14:03:58.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023914520000000001 2023-07-02T14:03:58.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:03:58.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109841+0000", 2023-07-02T14:03:58.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0823e-05 2023-07-02T14:03:58.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:03:58.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109862+0000", 2023-07-02T14:03:58.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2023-07-02T14:03:58.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:03:58.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109874+0000", 2023-07-02T14:03:58.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075009600000000001 2023-07-02T14:03:58.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:03:58.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.110624+0000", 2023-07-02T14:03:58.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5176000000000001e-05 2023-07-02T14:03:58.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:03:58.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.110649+0000", 2023-07-02T14:03:58.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034557310000000001 2023-07-02T14:03:58.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:03:58.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145206+0000", 2023-07-02T14:03:58.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4110999999999998e-05 2023-07-02T14:03:58.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:03:58.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:03:58.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:03:58.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145250+0000", 2023-07-02T14:03:58.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037799472000000001 2023-07-02T14:03:58.193 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.193 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:58.193 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.193 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:03:58.193 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:03:58.193 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:58.197 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:03:58.198 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:58.196+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:58.297 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:58.296+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:58.351 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:03:58.363 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:03:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:03:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:03:58.364 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:03:58.374 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:03:58.415 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:58.414+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:58.494 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:03:58.499 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:58.499+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:58.510 INFO:teuthology.orchestra.run.smithi022.stderr:Exporting 1.0 info 1.0( empty local-lis/les=71/72 n=0 ec=12/12 lis/c=71/71 les/c/f=72/72/0 sis=71) 2023-07-02T14:03:58.511 INFO:teuthology.orchestra.run.smithi022.stderr:Export successful 2023-07-02T14:03:58.513 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:03:58.530 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:03:58.544 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:58.544 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:58.549 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:03:58.554 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:03:58.568 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:03:58.569 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:03:58.569 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:03:58.569 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:03:58.569 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:03:58.569 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:03:58.570 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:03:58.572 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:03:58.573 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:03:58.575 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:03:58.587 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap14 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap15 b3fbb028-e267-4d25-a750-93ba18a2316a@snap15.diff 2023-07-02T14:03:58.601 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:58.600+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:58.655 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 51% complete...2023-07-02T14:03:58.655+0000 7fbb546d0700 0 -- 172.21.15.163:0/2270872552 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7fbb34036bb0 msgr2=0x7fbb34038fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-07-02T14:03:58.660 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:03:58.660+0000 7fbb546d0700 0 -- 172.21.15.163:0/2270872552 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x562993c1ef90 msgr2=0x7fbb3403b670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:03:58.665 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 100% complete...done. 2023-07-02T14:03:58.668 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:03:58.668 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=16 2023-07-02T14:03:58.668 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap16 2023-07-02T14:03:58.689 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:58.690 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:58.701 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:03:58.702 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:58.701+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:58.794 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete...+ tar -C b3fbb028-e267-4d25-a750-93ba18a2316a-mnt -xzf b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4.tar.gz 2023-07-02T14:03:58.803 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:03:58.802+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:03:58.829 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:58.829 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:58.839 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:03:58.904 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:58.903+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:59.002 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:59.003 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:59.005 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:59.004+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:59.012 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:03:59.029 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --force --op remove --pgid 1.0 2023-07-02T14:03:59.105 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:59.104+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:59.177 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:59.178 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:59.188 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:03:59.231 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:59.230+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:59.307 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:59.305+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:59.314 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:59.314 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:59.325 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:03:59.409 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:03:59.408+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:03:59.490 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:59.490 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:59.503 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:03:59.508 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:03:59.508+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:03:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:03:59.589 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 100% complete...done. 2023-07-02T14:03:59.594 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 4 2023-07-02T14:03:59.624 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:59.623+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:59.675 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:59.675 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:59.685 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:03:59.729 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:59.728+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:59.810 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:59.810 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:59.811 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:03:59.809+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:03:59.821 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:03:59.911 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:03:59.911+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:03:59.988 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:03:59.989 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:03:59.999 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:00.012 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:00.011+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:00.112 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:00.112+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:00.122 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:00.122 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:00.132 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:00.214 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:00.213+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:00.270 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:00.271 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:00.281 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:04:00.325 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:00.324+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:00.405 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:00.405 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:00.416 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:04:00.416 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:00.414+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:00.516 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:00.515+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:00.574 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:04:00.596 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:00.597 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:00.607 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:04:00.648 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:00.647+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:00.716 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:00.718 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:00.717+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:00.724 INFO:teuthology.orchestra.run.smithi022.stdout: marking collection for removal 2023-07-02T14:04:00.724 INFO:teuthology.orchestra.run.smithi022.stdout:setting '_remove' omap key 2023-07-02T14:04:00.729 INFO:teuthology.orchestra.run.smithi022.stdout:finish_remove_pgs 1.0_head removing 1.0 2023-07-02T14:04:00.730 INFO:teuthology.orchestra.run.smithi022.stdout:Remove successful 2023-07-02T14:04:00.734 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:00.751 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:00.770 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:00.774 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:00.775 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:00.775 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:00.775 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:00.785 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:04:00.788 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:00.789 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:04:00.789 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:04:00.789 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:04:00.789 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:00.789 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:04:00.789 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:04:00.791 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:00.791 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:00.832 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:00.831+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:00.911 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:00.911 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:00.912 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:00.912 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:00.912 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:00.920 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:00.920+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:00.922 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:04:01.022 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:01.021+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:01.088 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:01.089 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:01.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:01.089 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:01.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.089 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:04:01.089 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:01.089 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 51.634321227999997, 2023-07-02T14:04:01.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:04:01.090 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.090 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:01.090 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.090 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.090 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.090 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:04:01.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.091 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:01.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:01.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:01.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:04:01.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:04:01.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:04:01.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:04:01.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:04:01.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:04:01.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:04:01.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:04:01.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:04:01.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:04:01.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:04:01.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:04:01.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:04:01.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:04:01.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:04:01.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:04:01.097 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.097 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.097 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.098 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:04:01.098 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:01.098 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.763628220000001, 2023-07-02T14:04:01.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:04:01.098 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.098 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:01.099 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.099 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:04:01.099 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:04:01.099 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:04:01.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.099 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:01.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:01.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:01.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:04:01.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:04:01.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:04:01.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:04:01.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:04:01.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:04:01.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:04:01.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:04:01.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:04:01.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:04:01.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:04:01.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:04:01.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:04:01.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:04:01.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:04:01.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:04:01.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:04:01.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:04:01.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.107 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.107 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.107 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:01.107 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:01.107 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.763503499000002, 2023-07-02T14:04:01.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:04:01.108 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.108 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:01.108 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.108 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.108 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.108 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:04:01.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.109 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:01.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:01.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:01.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:04:01.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:04:01.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:04:01.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:04:01.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:04:01.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:04:01.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:04:01.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:04:01.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:04:01.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:04:01.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:04:01.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:04:01.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:04:01.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:04:01.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:04:01.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:04:01.115 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.115 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.115 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.116 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:01.116 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:01.116 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.756653614999998, 2023-07-02T14:04:01.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:04:01.116 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.116 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:01.117 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.117 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.117 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.117 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:04:01.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.117 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:01.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:01.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:04:01.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:04:01.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:04:01.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:04:01.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:04:01.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:04:01.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:04:01.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:04:01.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:04:01.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:04:01.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:04:01.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:04:01.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:04:01.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:04:01.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:04:01.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:04:01.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:04:01.124 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.124 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.124 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.124 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:01.124 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:01.125 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.756590690000003, 2023-07-02T14:04:01.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:04:01.125 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.125 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:01.125 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.125 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.125 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.125 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:04:01.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.126 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:01.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:01.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:01.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:04:01.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:04:01.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:04:01.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:04:01.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:04:01.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:04:01.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:04:01.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:04:01.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:04:01.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:04:01.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:04:01.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:04:01.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:04:01.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:04:01.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:04:01.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:04:01.132 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.132 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.133 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.133 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:01.133 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:01.133 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.756290268999997, 2023-07-02T14:04:01.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:04:01.133 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.134 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.134 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.134 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.134 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.134 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:04:01.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.134 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:01.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:01.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:01.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:04:01.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:04:01.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:04:01.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:04:01.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:04:01.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:04:01.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:04:01.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:04:01.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:04:01.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:04:01.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:04:01.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:04:01.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:04:01.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:04:01.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:04:01.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:04:01.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:04:01.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:04:01.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:04:01.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:04:01.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:04:01.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:04:01.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:04:01.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:04:01.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:04:01.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:04:01.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:04:01.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:04:01.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:04:01.145 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.145 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.145 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.145 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:04:01.146 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:01.146 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.755765769, 2023-07-02T14:04:01.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:04:01.146 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.146 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:01.146 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.146 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.146 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.147 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:04:01.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.147 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:01.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:01.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:01.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:01.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:04:01.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:04:01.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:04:01.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:04:01.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:04:01.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:04:01.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:04:01.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:04:01.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:04:01.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:04:01.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:04:01.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:04:01.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:04:01.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:04:01.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:04:01.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:04:01.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:04:01.154 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.154 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.154 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.155 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:04:01.155 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:01.155 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.631746644, 2023-07-02T14:04:01.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:04:01.155 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.155 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:01.155 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.156 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.156 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.156 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:04:01.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.156 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:01.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:01.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:01.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:04:01.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:04:01.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:04:01.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:04:01.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:04:01.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:04:01.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:04:01.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:04:01.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:04:01.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:04:01.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:04:01.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:04:01.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:04:01.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:04:01.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:04:01.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:04:01.162 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.163 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.163 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.163 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:01.163 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:01.163 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.306263940000001, 2023-07-02T14:04:01.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:04:01.164 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.164 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.164 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.164 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.164 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.164 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:04:01.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.165 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:01.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:01.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:01.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:04:01.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:04:01.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:04:01.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:04:01.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:04:01.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:04:01.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:04:01.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:04:01.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:01.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:04:01.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:04:01.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:04:01.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:04:01.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:04:01.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:04:01.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:04:01.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:01.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:04:01.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:04:01.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:04:01.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:04:01.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:04:01.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:04:01.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:04:01.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:04:01.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:04:01.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:04:01.174 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.174 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.175 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.175 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3887 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:01.175 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:01.175 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.302400751, 2023-07-02T14:04:01.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998, 2023-07-02T14:04:01.176 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.176 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.176 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.176 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.176 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.176 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3887 2023-07-02T14:04:01.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.176 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:01.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:01.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:01.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784019+0000", 2023-07-02T14:04:01.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4073999999999998e-05 2023-07-02T14:04:01.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784063+0000", 2023-07-02T14:04:01.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6630000000000004e-06 2023-07-02T14:04:01.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784070+0000", 2023-07-02T14:04:01.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.286e-06 2023-07-02T14:04:01.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784073+0000", 2023-07-02T14:04:01.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4941999999999998e-05 2023-07-02T14:04:01.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784118+0000", 2023-07-02T14:04:01.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8120000000000008e-06 2023-07-02T14:04:01.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784126+0000", 2023-07-02T14:04:01.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32999709399999999 2023-07-02T14:04:01.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114123+0000", 2023-07-02T14:04:01.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000157257 2023-07-02T14:04:01.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114280+0000", 2023-07-02T14:04:01.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000341152 2023-07-02T14:04:01.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:01.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114622+0000", 2023-07-02T14:04:01.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019539766 2023-07-02T14:04:01.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.134161+0000", 2023-07-02T14:04:01.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015820796000000002 2023-07-02T14:04:01.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149982+0000", 2023-07-02T14:04:01.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.05e-05 2023-07-02T14:04:01.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150043+0000", 2023-07-02T14:04:01.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6453999999999999e-05 2023-07-02T14:04:01.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150089+0000", 2023-07-02T14:04:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998 2023-07-02T14:04:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3939 2.1 2:9ef34a83:::rbd_data.125de1449605.00000000000000b8:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.302343242999999, 2023-07-02T14:04:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001, 2023-07-02T14:04:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.187 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3939 2023-07-02T14:04:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046852830000000002 2023-07-02T14:04:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788763+0000", 2023-07-02T14:04:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3185e-05 2023-07-02T14:04:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788776+0000", 2023-07-02T14:04:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8470000000000003e-06 2023-07-02T14:04:01.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788781+0000", 2023-07-02T14:04:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7104000000000002e-05 2023-07-02T14:04:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788828+0000", 2023-07-02T14:04:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3516000000000001e-05 2023-07-02T14:04:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788841+0000", 2023-07-02T14:04:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33064513600000001 2023-07-02T14:04:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119487+0000", 2023-07-02T14:04:01.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9876999999999997e-05 2023-07-02T14:04:01.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119537+0000", 2023-07-02T14:04:01.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014160999999999999 2023-07-02T14:04:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119678+0000", 2023-07-02T14:04:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023054392 2023-07-02T14:04:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.142733+0000", 2023-07-02T14:04:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074065470000000003 2023-07-02T14:04:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150139+0000", 2023-07-02T14:04:01.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5025999999999997e-05 2023-07-02T14:04:01.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150204+0000", 2023-07-02T14:04:01.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147999999999997e-05 2023-07-02T14:04:01.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150241+0000", 2023-07-02T14:04:01.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001 2023-07-02T14:04:01.198 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.198 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.198 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.199 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3940 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:01.199 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:01.199 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.297632211, 2023-07-02T14:04:01.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001, 2023-07-02T14:04:01.199 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.199 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.199 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.199 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.200 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.200 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3940 2023-07-02T14:04:01.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.200 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:01.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:01.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:01.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788788+0000", 2023-07-02T14:04:01.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043166560000000003 2023-07-02T14:04:01.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793105+0000", 2023-07-02T14:04:01.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5970000000000001e-05 2023-07-02T14:04:01.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793121+0000", 2023-07-02T14:04:01.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0939999999999998e-06 2023-07-02T14:04:01.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793126+0000", 2023-07-02T14:04:01.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2438000000000002e-05 2023-07-02T14:04:01.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793178+0000", 2023-07-02T14:04:01.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2019e-05 2023-07-02T14:04:01.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793190+0000", 2023-07-02T14:04:01.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33208757799999999 2023-07-02T14:04:01.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125278+0000", 2023-07-02T14:04:01.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5255999999999997e-05 2023-07-02T14:04:01.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125373+0000", 2023-07-02T14:04:01.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015674999999999999 2023-07-02T14:04:01.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:01.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125530+0000", 2023-07-02T14:04:01.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018767308 2023-07-02T14:04:01.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144297+0000", 2023-07-02T14:04:01.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060291959999999997 2023-07-02T14:04:01.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150326+0000", 2023-07-02T14:04:01.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3886999999999993e-05 2023-07-02T14:04:01.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150390+0000", 2023-07-02T14:04:01.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010255 2023-07-02T14:04:01.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150493+0000", 2023-07-02T14:04:01.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001 2023-07-02T14:04:01.210 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.210 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.210 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.210 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3947 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:01.210 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:01.211 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.293286684000002, 2023-07-02T14:04:01.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999, 2023-07-02T14:04:01.211 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.211 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.211 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.211 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.211 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.211 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3947 2023-07-02T14:04:01.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.212 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:01.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:01.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:01.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793133+0000", 2023-07-02T14:04:01.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2519999999999999e-05 2023-07-02T14:04:01.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793176+0000", 2023-07-02T14:04:01.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:04:01.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793180+0000", 2023-07-02T14:04:01.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1709999999999998e-06 2023-07-02T14:04:01.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793184+0000", 2023-07-02T14:04:01.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1721999999999999e-05 2023-07-02T14:04:01.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793215+0000", 2023-07-02T14:04:01.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1280000000000008e-06 2023-07-02T14:04:01.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793224+0000", 2023-07-02T14:04:01.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34218549599999998 2023-07-02T14:04:01.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135409+0000", 2023-07-02T14:04:01.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117442 2023-07-02T14:04:01.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135526+0000", 2023-07-02T14:04:01.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000206257 2023-07-02T14:04:01.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:01.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135733+0000", 2023-07-02T14:04:01.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085700029999999996 2023-07-02T14:04:01.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144303+0000", 2023-07-02T14:04:01.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006278652 2023-07-02T14:04:01.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150581+0000", 2023-07-02T14:04:01.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4765999999999996e-05 2023-07-02T14:04:01.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150646+0000", 2023-07-02T14:04:01.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4777999999999995e-05 2023-07-02T14:04:01.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150711+0000", 2023-07-02T14:04:01.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999 2023-07-02T14:04:01.222 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.222 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.222 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.222 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3980 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 65536~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:01.222 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:01.222 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.293232801999999, 2023-07-02T14:04:01.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999, 2023-07-02T14:04:01.223 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.223 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.223 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.223 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.223 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.223 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3980 2023-07-02T14:04:01.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.224 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:01.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:01.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:01.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:01.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020244260000000002 2023-07-02T14:04:01.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795212+0000", 2023-07-02T14:04:01.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0005e-05 2023-07-02T14:04:01.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795222+0000", 2023-07-02T14:04:01.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9860000000000002e-06 2023-07-02T14:04:01.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795227+0000", 2023-07-02T14:04:01.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6369999999999999e-05 2023-07-02T14:04:01.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795264+0000", 2023-07-02T14:04:01.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1052e-05 2023-07-02T14:04:01.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795275+0000", 2023-07-02T14:04:01.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.343850459 2023-07-02T14:04:01.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139125+0000", 2023-07-02T14:04:01.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6760000000000004e-05 2023-07-02T14:04:01.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139202+0000", 2023-07-02T14:04:01.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015092799999999999 2023-07-02T14:04:01.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:01.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139353+0000", 2023-07-02T14:04:01.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011425940000000001 2023-07-02T14:04:01.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150779+0000", 2023-07-02T14:04:01.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033223060000000001 2023-07-02T14:04:01.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154101+0000", 2023-07-02T14:04:01.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9268999999999998e-05 2023-07-02T14:04:01.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154161+0000", 2023-07-02T14:04:01.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6499999999999999e-05 2023-07-02T14:04:01.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154207+0000", 2023-07-02T14:04:01.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999 2023-07-02T14:04:01.233 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.233 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.234 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.234 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3982 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1835008~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:01.234 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:01.234 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.291187515000001, 2023-07-02T14:04:01.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002, 2023-07-02T14:04:01.234 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.235 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.235 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.235 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.235 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.235 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3982 2023-07-02T14:04:01.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.235 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:01.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:01.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:01.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795233+0000", 2023-07-02T14:04:01.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8680999999999998e-05 2023-07-02T14:04:01.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795272+0000", 2023-07-02T14:04:01.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5629999999999999e-06 2023-07-02T14:04:01.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795277+0000", 2023-07-02T14:04:01.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8700000000000001e-06 2023-07-02T14:04:01.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795280+0000", 2023-07-02T14:04:01.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1964000000000001e-05 2023-07-02T14:04:01.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795312+0000", 2023-07-02T14:04:01.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.314e-06 2023-07-02T14:04:01.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795319+0000", 2023-07-02T14:04:01.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34828442599999998 2023-07-02T14:04:01.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143603+0000", 2023-07-02T14:04:01.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2438000000000001e-05 2023-07-02T14:04:01.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143665+0000", 2023-07-02T14:04:01.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3079999999999997e-05 2023-07-02T14:04:01.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:01.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143759+0000", 2023-07-02T14:04:01.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070633079999999999 2023-07-02T14:04:01.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150822+0000", 2023-07-02T14:04:01.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034430210000000001 2023-07-02T14:04:01.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154265+0000", 2023-07-02T14:04:01.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4249999999999998e-05 2023-07-02T14:04:01.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154309+0000", 2023-07-02T14:04:01.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2020000000000001e-05 2023-07-02T14:04:01.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154351+0000", 2023-07-02T14:04:01.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002 2023-07-02T14:04:01.245 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.245 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.245 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.246 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4054 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1900544~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:01.246 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:01.246 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.291135843999999, 2023-07-02T14:04:01.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002, 2023-07-02T14:04:01.246 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.246 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.246 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.247 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.247 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.247 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4054 2023-07-02T14:04:01.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.247 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:01.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:01.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:01.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:01.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018874860000000001 2023-07-02T14:04:01.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797172+0000", 2023-07-02T14:04:01.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3000000000000007e-06 2023-07-02T14:04:01.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797182+0000", 2023-07-02T14:04:01.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6260000000000003e-06 2023-07-02T14:04:01.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797186+0000", 2023-07-02T14:04:01.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9013e-05 2023-07-02T14:04:01.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797235+0000", 2023-07-02T14:04:01.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5990000000000008e-06 2023-07-02T14:04:01.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797245+0000", 2023-07-02T14:04:01.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34785621999999999 2023-07-02T14:04:01.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145101+0000", 2023-07-02T14:04:01.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7762000000000006e-05 2023-07-02T14:04:01.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145169+0000", 2023-07-02T14:04:01.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013469400000000001 2023-07-02T14:04:01.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:01.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145304+0000", 2023-07-02T14:04:01.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089538039999999992 2023-07-02T14:04:01.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154257+0000", 2023-07-02T14:04:01.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104218 2023-07-02T14:04:01.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154362+0000", 2023-07-02T14:04:01.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4168000000000001e-05 2023-07-02T14:04:01.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154406+0000", 2023-07-02T14:04:01.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3098000000000001e-05 2023-07-02T14:04:01.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154429+0000", 2023-07-02T14:04:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002 2023-07-02T14:04:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4056 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3670016~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.289229331000001, 2023-07-02T14:04:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997, 2023-07-02T14:04:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.258 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.259 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.259 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.259 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4056 2023-07-02T14:04:01.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.259 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:01.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:01.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:01.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797191+0000", 2023-07-02T14:04:01.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0438999999999997e-05 2023-07-02T14:04:01.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797232+0000", 2023-07-02T14:04:01.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6929999999999998e-06 2023-07-02T14:04:01.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797237+0000", 2023-07-02T14:04:01.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9739999999999998e-06 2023-07-02T14:04:01.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797240+0000", 2023-07-02T14:04:01.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0243000000000002e-05 2023-07-02T14:04:01.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797270+0000", 2023-07-02T14:04:01.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9889999999999997e-06 2023-07-02T14:04:01.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797276+0000", 2023-07-02T14:04:01.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35358319799999999 2023-07-02T14:04:01.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150859+0000", 2023-07-02T14:04:01.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1357e-05 2023-07-02T14:04:01.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150940+0000", 2023-07-02T14:04:01.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014903 2023-07-02T14:04:01.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:01.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151089+0000", 2023-07-02T14:04:01.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033512519999999999 2023-07-02T14:04:01.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154441+0000", 2023-07-02T14:04:01.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001050945 2023-07-02T14:04:01.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155492+0000", 2023-07-02T14:04:01.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2523e-05 2023-07-02T14:04:01.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155524+0000", 2023-07-02T14:04:01.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.057e-05 2023-07-02T14:04:01.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155555+0000", 2023-07-02T14:04:01.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997 2023-07-02T14:04:01.269 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.269 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.269 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.270 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4088 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3735552~458752] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:01.270 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:01.270 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.289177135999999, 2023-07-02T14:04:01.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998, 2023-07-02T14:04:01.270 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.270 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.271 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.271 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.271 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4088 2023-07-02T14:04:01.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.271 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:01.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:01.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:01.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:01.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00038503499999999999 2023-07-02T14:04:01.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797629+0000", 2023-07-02T14:04:01.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8739999999999995e-06 2023-07-02T14:04:01.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797637+0000", 2023-07-02T14:04:01.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7160000000000001e-06 2023-07-02T14:04:01.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797640+0000", 2023-07-02T14:04:01.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4342999999999998e-05 2023-07-02T14:04:01.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797695+0000", 2023-07-02T14:04:01.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0939999999999996e-06 2023-07-02T14:04:01.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797701+0000", 2023-07-02T14:04:01.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35408540799999999 2023-07-02T14:04:01.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151786+0000", 2023-07-02T14:04:01.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1953000000000006e-05 2023-07-02T14:04:01.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151868+0000", 2023-07-02T14:04:01.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015162800000000001 2023-07-02T14:04:01.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:01.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.152020+0000", 2023-07-02T14:04:01.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054561870000000004 2023-07-02T14:04:01.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.157476+0000", 2023-07-02T14:04:01.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059153199999999997 2023-07-02T14:04:01.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158068+0000", 2023-07-02T14:04:01.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2799e-05 2023-07-02T14:04:01.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158090+0000", 2023-07-02T14:04:01.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7235999999999999e-05 2023-07-02T14:04:01.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158108+0000", 2023-07-02T14:04:01.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998 2023-07-02T14:04:01.281 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.281 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.281 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.281 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4110 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.288775859000001, 2023-07-02T14:04:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864, 2023-07-02T14:04:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.282 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.283 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4110 2023-07-02T14:04:01.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.283 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:01.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:01.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:01.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:01.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9331e-05 2023-07-02T14:04:01.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797684+0000", 2023-07-02T14:04:01.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.147e-06 2023-07-02T14:04:01.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797689+0000", 2023-07-02T14:04:01.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5020000000000001e-06 2023-07-02T14:04:01.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797693+0000", 2023-07-02T14:04:01.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5254999999999999e-05 2023-07-02T14:04:01.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797748+0000", 2023-07-02T14:04:01.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5520000000000005e-06 2023-07-02T14:04:01.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797757+0000", 2023-07-02T14:04:01.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35674245700000001 2023-07-02T14:04:01.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154499+0000", 2023-07-02T14:04:01.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0090000000000001e-06 2023-07-02T14:04:01.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154506+0000", 2023-07-02T14:04:01.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864 2023-07-02T14:04:01.290 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.290 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.290 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.290 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:01.291 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:01.291 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.288723773000001, 2023-07-02T14:04:01.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998, 2023-07-02T14:04:01.291 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.291 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:01.291 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.291 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.292 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.292 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:04:01.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.292 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:01.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:01.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:01.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:01.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001714819 2023-07-02T14:04:01.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799412+0000", 2023-07-02T14:04:01.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7779999999999996e-06 2023-07-02T14:04:01.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799421+0000", 2023-07-02T14:04:01.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3510000000000002e-06 2023-07-02T14:04:01.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799424+0000", 2023-07-02T14:04:01.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0275999999999999e-05 2023-07-02T14:04:01.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799464+0000", 2023-07-02T14:04:01.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8389999999999996e-06 2023-07-02T14:04:01.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799473+0000", 2023-07-02T14:04:01.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35507638800000002 2023-07-02T14:04:01.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154550+0000", 2023-07-02T14:04:01.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6309999999999998e-06 2023-07-02T14:04:01.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154559+0000", 2023-07-02T14:04:01.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998 2023-07-02T14:04:01.298 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.299 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.299 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.299 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.299 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.299 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:01.301 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.1.0.3 2023-07-02T14:04:01.301 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:01.300+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:01.302 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:04:01.401 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:01.400+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:01.484 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:01.484 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:01.485 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:01.485 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:01.502 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:01.501+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:01.504 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:04:01.584 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:04:01.627 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:01.626+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:01.657 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:01.657 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:01.657 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:01.657 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:01.657 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:01.668 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:04:01.710 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:01.709+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:01.804 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:01.803+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:01.835 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:01.835 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:01.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:01.836 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:01.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.836 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:183 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:01.836 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:04:01.836 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.547874931999999, 2023-07-02T14:04:01.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003, 2023-07-02T14:04:01.836 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.837 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.837 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.837 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.837 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.837 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 183 2023-07-02T14:04:01.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.837 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:04:01.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:04:01.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:01.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:04:01.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9790999999999997e-05 2023-07-02T14:04:01.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285598+0000", 2023-07-02T14:04:01.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5809999999999996e-06 2023-07-02T14:04:01.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285604+0000", 2023-07-02T14:04:01.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7599999999999998e-06 2023-07-02T14:04:01.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285607+0000", 2023-07-02T14:04:01.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021108188999999999 2023-07-02T14:04:01.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306715+0000", 2023-07-02T14:04:01.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6548e-05 2023-07-02T14:04:01.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306792+0000", 2023-07-02T14:04:01.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173869 2023-07-02T14:04:01.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:01.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306965+0000", 2023-07-02T14:04:01.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060498009999999996 2023-07-02T14:04:01.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313015+0000", 2023-07-02T14:04:01.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049443719999999998 2023-07-02T14:04:01.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317960+0000", 2023-07-02T14:04:01.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5927999999999998e-05 2023-07-02T14:04:01.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318006+0000", 2023-07-02T14:04:01.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6004999999999998e-05 2023-07-02T14:04:01.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318042+0000", 2023-07-02T14:04:01.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003 2023-07-02T14:04:01.846 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.846 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.846 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.847 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:186 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:01.847 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:04:01.847 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.547804106000001, 2023-07-02T14:04:01.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003, 2023-07-02T14:04:01.847 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.847 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.847 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.848 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.848 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.848 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 186 2023-07-02T14:04:01.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.848 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:04:01.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:04:01.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:01.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:04:01.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4279e-05 2023-07-02T14:04:01.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285664+0000", 2023-07-02T14:04:01.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3789999999999995e-06 2023-07-02T14:04:01.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285673+0000", 2023-07-02T14:04:01.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499999999999998e-06 2023-07-02T14:04:01.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285678+0000", 2023-07-02T14:04:01.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023261947000000002 2023-07-02T14:04:01.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.308940+0000", 2023-07-02T14:04:01.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010323399999999999 2023-07-02T14:04:01.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309044+0000", 2023-07-02T14:04:01.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018878300000000001 2023-07-02T14:04:01.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:01.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309232+0000", 2023-07-02T14:04:01.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037874150000000001 2023-07-02T14:04:01.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313020+0000", 2023-07-02T14:04:01.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050976140000000003 2023-07-02T14:04:01.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318117+0000", 2023-07-02T14:04:01.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3182000000000001e-05 2023-07-02T14:04:01.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:04:01.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5055999999999998e-05 2023-07-02T14:04:01.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318196+0000", 2023-07-02T14:04:01.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003 2023-07-02T14:04:01.857 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.857 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.857 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.857 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:189 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:01.857 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:04:01.857 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.547730342999998, 2023-07-02T14:04:01.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997, 2023-07-02T14:04:01.858 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.858 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.858 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.858 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.858 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.858 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:04:01.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.859 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:04:01.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:04:01.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:01.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285683+0000", 2023-07-02T14:04:01.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8451999999999999e-05 2023-07-02T14:04:01.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285782+0000", 2023-07-02T14:04:01.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3650000000000004e-06 2023-07-02T14:04:01.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285790+0000", 2023-07-02T14:04:01.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8230000000000002e-06 2023-07-02T14:04:01.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285795+0000", 2023-07-02T14:04:01.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024619367999999999 2023-07-02T14:04:01.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310414+0000", 2023-07-02T14:04:01.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014244600000000001 2023-07-02T14:04:01.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310557+0000", 2023-07-02T14:04:01.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020612699999999999 2023-07-02T14:04:01.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:01.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310763+0000", 2023-07-02T14:04:01.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003427687 2023-07-02T14:04:01.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314191+0000", 2023-07-02T14:04:01.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040492089999999998 2023-07-02T14:04:01.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318240+0000", 2023-07-02T14:04:01.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0287999999999999e-05 2023-07-02T14:04:01.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318270+0000", 2023-07-02T14:04:01.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0868000000000002e-05 2023-07-02T14:04:01.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.867 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318311+0000", 2023-07-02T14:04:01.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997 2023-07-02T14:04:01.867 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.867 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.868 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.868 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:192 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:01.868 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:04:01.868 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.547613569999996, 2023-07-02T14:04:01.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003, 2023-07-02T14:04:01.868 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.869 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.869 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.869 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.869 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.869 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 192 2023-07-02T14:04:01.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.869 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:04:01.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:04:01.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:01.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285800+0000", 2023-07-02T14:04:01.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100417 2023-07-02T14:04:01.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285900+0000", 2023-07-02T14:04:01.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8244999999999999e-05 2023-07-02T14:04:01.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285919+0000", 2023-07-02T14:04:01.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2499999999999997e-06 2023-07-02T14:04:01.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285924+0000", 2023-07-02T14:04:01.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025288155999999999 2023-07-02T14:04:01.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311212+0000", 2023-07-02T14:04:01.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4073999999999996e-05 2023-07-02T14:04:01.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.874 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311286+0000", 2023-07-02T14:04:01.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016219699999999999 2023-07-02T14:04:01.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:01.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311448+0000", 2023-07-02T14:04:01.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066020599999999999 2023-07-02T14:04:01.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318050+0000", 2023-07-02T14:04:01.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000327986 2023-07-02T14:04:01.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318378+0000", 2023-07-02T14:04:01.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8693999999999998e-05 2023-07-02T14:04:01.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318427+0000", 2023-07-02T14:04:01.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6656999999999998e-05 2023-07-02T14:04:01.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318464+0000", 2023-07-02T14:04:01.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003 2023-07-02T14:04:01.878 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.878 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.878 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.878 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:195 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:01.879 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:04:01.879 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.547482066000001, 2023-07-02T14:04:01.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001, 2023-07-02T14:04:01.879 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.879 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.879 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.879 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.879 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.880 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 195 2023-07-02T14:04:01.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.880 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:04:01.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:04:01.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:01.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285931+0000", 2023-07-02T14:04:01.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4530999999999995e-05 2023-07-02T14:04:01.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286006+0000", 2023-07-02T14:04:01.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4113e-05 2023-07-02T14:04:01.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286020+0000", 2023-07-02T14:04:01.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7459999999999997e-06 2023-07-02T14:04:01.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286026+0000", 2023-07-02T14:04:01.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025997769 2023-07-02T14:04:01.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312023+0000", 2023-07-02T14:04:01.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4403000000000001e-05 2023-07-02T14:04:01.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.885 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312078+0000", 2023-07-02T14:04:01.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020581399999999999 2023-07-02T14:04:01.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:01.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312284+0000", 2023-07-02T14:04:01.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005771043 2023-07-02T14:04:01.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318055+0000", 2023-07-02T14:04:01.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00046015200000000002 2023-07-02T14:04:01.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318515+0000", 2023-07-02T14:04:01.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9444000000000003e-05 2023-07-02T14:04:01.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318564+0000", 2023-07-02T14:04:01.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2141000000000002e-05 2023-07-02T14:04:01.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318606+0000", 2023-07-02T14:04:01.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001 2023-07-02T14:04:01.889 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.889 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.889 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.889 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:198 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:01.889 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:01.890 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.547382107000004, 2023-07-02T14:04:01.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999, 2023-07-02T14:04:01.890 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.890 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.890 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.890 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.890 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.891 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 198 2023-07-02T14:04:01.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.891 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:01.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:01.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:01.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286031+0000", 2023-07-02T14:04:01.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00049286700000000002 2023-07-02T14:04:01.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286524+0000", 2023-07-02T14:04:01.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4938e-05 2023-07-02T14:04:01.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286539+0000", 2023-07-02T14:04:01.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0680000000000002e-06 2023-07-02T14:04:01.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286545+0000", 2023-07-02T14:04:01.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026497718 2023-07-02T14:04:01.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313043+0000", 2023-07-02T14:04:01.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0884999999999998e-05 2023-07-02T14:04:01.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313094+0000", 2023-07-02T14:04:01.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014109999999999999 2023-07-02T14:04:01.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:01.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313235+0000", 2023-07-02T14:04:01.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048258980000000003 2023-07-02T14:04:01.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318061+0000", 2023-07-02T14:04:01.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014346350000000001 2023-07-02T14:04:01.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319495+0000", 2023-07-02T14:04:01.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1986e-05 2023-07-02T14:04:01.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319547+0000", 2023-07-02T14:04:01.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0457e-05 2023-07-02T14:04:01.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319578+0000", 2023-07-02T14:04:01.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999 2023-07-02T14:04:01.899 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.899 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.899 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.900 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:201 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:01.900 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:04:01.900 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.546861681999999, 2023-07-02T14:04:01.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003, 2023-07-02T14:04:01.900 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.900 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.901 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.901 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.901 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.901 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 201 2023-07-02T14:04:01.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.901 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:04:01.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:04:01.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:01.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:04:01.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019213999999999999 2023-07-02T14:04:01.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286744+0000", 2023-07-02T14:04:01.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5279999999999992e-06 2023-07-02T14:04:01.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286753+0000", 2023-07-02T14:04:01.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4200000000000004e-06 2023-07-02T14:04:01.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286760+0000", 2023-07-02T14:04:01.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026818386999999999 2023-07-02T14:04:01.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313578+0000", 2023-07-02T14:04:01.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015965 2023-07-02T14:04:01.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313738+0000", 2023-07-02T14:04:01.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015767299999999999 2023-07-02T14:04:01.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:01.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313895+0000", 2023-07-02T14:04:01.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041714930000000001 2023-07-02T14:04:01.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318067+0000", 2023-07-02T14:04:01.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001561698 2023-07-02T14:04:01.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319629+0000", 2023-07-02T14:04:01.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2346000000000003e-05 2023-07-02T14:04:01.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319681+0000", 2023-07-02T14:04:01.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4691000000000003e-05 2023-07-02T14:04:01.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319726+0000", 2023-07-02T14:04:01.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003 2023-07-02T14:04:01.910 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.910 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.910 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.910 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:204 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:01.910 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:04:01.911 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.546646843999994, 2023-07-02T14:04:01.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998, 2023-07-02T14:04:01.911 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.911 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.911 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.911 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.911 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.911 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 204 2023-07-02T14:04:01.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.912 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:04:01.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:04:01.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:01.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:04:01.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2497999999999998e-05 2023-07-02T14:04:01.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286819+0000", 2023-07-02T14:04:01.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0099999999999995e-06 2023-07-02T14:04:01.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286827+0000", 2023-07-02T14:04:01.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000003e-06 2023-07-02T14:04:01.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286833+0000", 2023-07-02T14:04:01.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027384716999999999 2023-07-02T14:04:01.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314218+0000", 2023-07-02T14:04:01.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6204000000000004e-05 2023-07-02T14:04:01.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314294+0000", 2023-07-02T14:04:01.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015757499999999999 2023-07-02T14:04:01.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:01.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314452+0000", 2023-07-02T14:04:01.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036215560000000002 2023-07-02T14:04:01.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318073+0000", 2023-07-02T14:04:01.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017023979999999999 2023-07-02T14:04:01.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319776+0000", 2023-07-02T14:04:01.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9561000000000002e-05 2023-07-02T14:04:01.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319825+0000", 2023-07-02T14:04:01.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0329e-05 2023-07-02T14:04:01.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319866+0000", 2023-07-02T14:04:01.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998 2023-07-02T14:04:01.920 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.920 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.921 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.921 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:207 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:01.921 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:01.921 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.546572198999996, 2023-07-02T14:04:01.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003, 2023-07-02T14:04:01.922 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.922 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.922 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.922 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.922 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.922 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 207 2023-07-02T14:04:01.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.922 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:01.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:01.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:01.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286841+0000", 2023-07-02T14:04:01.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00068375100000000002 2023-07-02T14:04:01.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287525+0000", 2023-07-02T14:04:01.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1601e-05 2023-07-02T14:04:01.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287537+0000", 2023-07-02T14:04:01.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6409999999999996e-06 2023-07-02T14:04:01.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287543+0000", 2023-07-02T14:04:01.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027245844000000002 2023-07-02T14:04:01.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314789+0000", 2023-07-02T14:04:01.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8910000000000003e-05 2023-07-02T14:04:01.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314858+0000", 2023-07-02T14:04:01.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030992599999999997 2023-07-02T14:04:01.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:01.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315168+0000", 2023-07-02T14:04:01.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029122869999999999 2023-07-02T14:04:01.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318080+0000", 2023-07-02T14:04:01.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019963020000000001 2023-07-02T14:04:01.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320077+0000", 2023-07-02T14:04:01.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5116000000000005e-05 2023-07-02T14:04:01.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320142+0000", 2023-07-02T14:04:01.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3793000000000004e-05 2023-07-02T14:04:01.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320205+0000", 2023-07-02T14:04:01.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003 2023-07-02T14:04:01.931 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.931 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.931 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.932 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:236 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2424832~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:01.932 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:01.932 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.545860016000006, 2023-07-02T14:04:01.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003, 2023-07-02T14:04:01.932 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.932 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.932 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.932 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.933 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.933 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2023-07-02T14:04:01.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.933 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:01.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:01.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:01.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287553+0000", 2023-07-02T14:04:01.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0667e-05 2023-07-02T14:04:01.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287614+0000", 2023-07-02T14:04:01.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0454e-05 2023-07-02T14:04:01.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287624+0000", 2023-07-02T14:04:01.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8564999999999999e-05 2023-07-02T14:04:01.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287643+0000", 2023-07-02T14:04:01.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027977596 2023-07-02T14:04:01.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315621+0000", 2023-07-02T14:04:01.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000201728 2023-07-02T14:04:01.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315822+0000", 2023-07-02T14:04:01.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018436599999999999 2023-07-02T14:04:01.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:01.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316007+0000", 2023-07-02T14:04:01.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042423440000000003 2023-07-02T14:04:01.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320249+0000", 2023-07-02T14:04:01.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001452633 2023-07-02T14:04:01.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321702+0000", 2023-07-02T14:04:01.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3616000000000003e-05 2023-07-02T14:04:01.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321755+0000", 2023-07-02T14:04:01.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840999999999999e-05 2023-07-02T14:04:01.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321793+0000", 2023-07-02T14:04:01.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003 2023-07-02T14:04:01.942 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.942 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.942 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.942 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:251 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:01.942 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:01.942 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.545764478999999, 2023-07-02T14:04:01.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001, 2023-07-02T14:04:01.943 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.943 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.943 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.943 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.943 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.943 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 251 2023-07-02T14:04:01.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287649+0000", 2023-07-02T14:04:01.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3232000000000001e-05 2023-07-02T14:04:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287702+0000", 2023-07-02T14:04:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0010000000000007e-06 2023-07-02T14:04:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287711+0000", 2023-07-02T14:04:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3569999999999997e-06 2023-07-02T14:04:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287717+0000", 2023-07-02T14:04:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028686454 2023-07-02T14:04:01.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316403+0000", 2023-07-02T14:04:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7741000000000002e-05 2023-07-02T14:04:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316461+0000", 2023-07-02T14:04:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4313000000000004e-05 2023-07-02T14:04:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316545+0000", 2023-07-02T14:04:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003734032 2023-07-02T14:04:01.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320279+0000", 2023-07-02T14:04:01.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015237219999999999 2023-07-02T14:04:01.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321803+0000", 2023-07-02T14:04:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1927e-05 2023-07-02T14:04:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321845+0000", 2023-07-02T14:04:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5398999999999998e-05 2023-07-02T14:04:01.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321880+0000", 2023-07-02T14:04:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001 2023-07-02T14:04:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:255 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2555904~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.545349751000003, 2023-07-02T14:04:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002, 2023-07-02T14:04:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.953 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 255 2023-07-02T14:04:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:01.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:01.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:01.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288063+0000", 2023-07-02T14:04:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4459999999999997e-05 2023-07-02T14:04:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288118+0000", 2023-07-02T14:04:01.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0998999999999999e-05 2023-07-02T14:04:01.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288129+0000", 2023-07-02T14:04:01.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3219999999999998e-06 2023-07-02T14:04:01.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288135+0000", 2023-07-02T14:04:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029137386000000001 2023-07-02T14:04:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317273+0000", 2023-07-02T14:04:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7819999999999997e-05 2023-07-02T14:04:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317350+0000", 2023-07-02T14:04:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000174864 2023-07-02T14:04:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:01.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317525+0000", 2023-07-02T14:04:01.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027943519999999999 2023-07-02T14:04:01.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320320+0000", 2023-07-02T14:04:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015681460000000001 2023-07-02T14:04:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321888+0000", 2023-07-02T14:04:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9508999999999997e-05 2023-07-02T14:04:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321927+0000", 2023-07-02T14:04:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5428e-05 2023-07-02T14:04:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321963+0000", 2023-07-02T14:04:01.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002 2023-07-02T14:04:01.963 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.963 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.963 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.963 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:446 2.1 2:910b2a90:::rbd_data.125de1449605.000000000000010a:head [write 65536~3211264] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:01.963 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:04:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.468069998999994, 2023-07-02T14:04:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003, 2023-07-02T14:04:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.964 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 446 2023-07-02T14:04:01.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.965 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:04:01.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:04:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:04:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015907372999999999 2023-07-02T14:04:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381251+0000", 2023-07-02T14:04:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5527000000000001e-05 2023-07-02T14:04:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381277+0000", 2023-07-02T14:04:01.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9958000000000001e-05 2023-07-02T14:04:01.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381297+0000", 2023-07-02T14:04:01.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4941000000000001e-05 2023-07-02T14:04:01.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381352+0000", 2023-07-02T14:04:01.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011565699999999999 2023-07-02T14:04:01.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381467+0000", 2023-07-02T14:04:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018039799999999999 2023-07-02T14:04:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:01.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381648+0000", 2023-07-02T14:04:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088183559999999994 2023-07-02T14:04:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.390466+0000", 2023-07-02T14:04:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007278278 2023-07-02T14:04:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397744+0000", 2023-07-02T14:04:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9289000000000001e-05 2023-07-02T14:04:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397804+0000", 2023-07-02T14:04:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0479000000000002e-05 2023-07-02T14:04:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397854+0000", 2023-07-02T14:04:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003 2023-07-02T14:04:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.973 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:04:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 60.725739204999996, 2023-07-02T14:04:01.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:04:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:04:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.975 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:04:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:04:01.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:04:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:04:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:04:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:04:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:04:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:04:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:04:01.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:04:01.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:04:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:04:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:04:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:04:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:04:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:04:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:04:01.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:04:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:04:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:01.983 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 60.725678467000002, 2023-07-02T14:04:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:04:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.984 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:04:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:01.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:01.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:04:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:04:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:04:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:04:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:04:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:04:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:04:01.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:04:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:04:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:04:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:04:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:04:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:04:01.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:04:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:04:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:04:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:04:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:04:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:04:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:04:01.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:04:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:04:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:04:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:04:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:04:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:04:01.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.495224137000001, 2023-07-02T14:04:01.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:04:01.996 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:01.996 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:01.996 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:01.996 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:01.996 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:01.996 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:04:01.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:04:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:04:01.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:04:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:04:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:01.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:02.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:04:02.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:04:02.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:04:02.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:04:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:04:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:04:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:04:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:04:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:04:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:04:02.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:04:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:04:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:04:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:04:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:04:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:04:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:04:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:04:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:02.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2941 2.3 2:d3762e63:::rbd_data.125de1449605.00000000000000f8:head [write 65536~2031616] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.366121182999997, 2023-07-02T14:04:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002, 2023-07-02T14:04:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:02.006 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2941 2023-07-02T14:04:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:02.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467291+0000", 2023-07-02T14:04:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043022859999999998 2023-07-02T14:04:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471593+0000", 2023-07-02T14:04:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0999999999999999e-05 2023-07-02T14:04:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471614+0000", 2023-07-02T14:04:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0097e-05 2023-07-02T14:04:02.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471625+0000", 2023-07-02T14:04:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069808499999999998 2023-07-02T14:04:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:02.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472323+0000", 2023-07-02T14:04:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135635 2023-07-02T14:04:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472458+0000", 2023-07-02T14:04:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223365 2023-07-02T14:04:02.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472682+0000", 2023-07-02T14:04:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013761944999999999 2023-07-02T14:04:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:02.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.486444+0000", 2023-07-02T14:04:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016859747000000001 2023-07-02T14:04:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503303+0000", 2023-07-02T14:04:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6635000000000001e-05 2023-07-02T14:04:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503380+0000", 2023-07-02T14:04:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000161803 2023-07-02T14:04:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:02.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503542+0000", 2023-07-02T14:04:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002 2023-07-02T14:04:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.016 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2950 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 1507328~786432] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.357947875000001, 2023-07-02T14:04:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999, 2023-07-02T14:04:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:02.017 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2950 2023-07-02T14:04:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:02.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001147854 2023-07-02T14:04:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476614+0000", 2023-07-02T14:04:02.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.802e-05 2023-07-02T14:04:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476632+0000", 2023-07-02T14:04:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1860000000000003e-06 2023-07-02T14:04:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476639+0000", 2023-07-02T14:04:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061211240000000004 2023-07-02T14:04:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482760+0000", 2023-07-02T14:04:02.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8833999999999996e-05 2023-07-02T14:04:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482859+0000", 2023-07-02T14:04:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019270600000000001 2023-07-02T14:04:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.483051+0000", 2023-07-02T14:04:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060385630000000003 2023-07-02T14:04:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489090+0000", 2023-07-02T14:04:02.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024506918999999999 2023-07-02T14:04:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513597+0000", 2023-07-02T14:04:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9672999999999995e-05 2023-07-02T14:04:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513667+0000", 2023-07-02T14:04:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024329500000000001 2023-07-02T14:04:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513910+0000", 2023-07-02T14:04:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999 2023-07-02T14:04:02.026 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2951 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2686976~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.356767097000002, 2023-07-02T14:04:02.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999, 2023-07-02T14:04:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2951 2023-07-02T14:04:02.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476646+0000", 2023-07-02T14:04:02.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7639000000000003e-05 2023-07-02T14:04:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476684+0000", 2023-07-02T14:04:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3508000000000002e-05 2023-07-02T14:04:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476747+0000", 2023-07-02T14:04:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8499999999999994e-06 2023-07-02T14:04:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476755+0000", 2023-07-02T14:04:02.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078346160000000008 2023-07-02T14:04:02.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:02.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484590+0000", 2023-07-02T14:04:02.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7206000000000004e-05 2023-07-02T14:04:02.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:02.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484687+0000", 2023-07-02T14:04:02.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016243099999999999 2023-07-02T14:04:02.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:02.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484849+0000", 2023-07-02T14:04:02.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042482860000000004 2023-07-02T14:04:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489097+0000", 2023-07-02T14:04:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.02489504 2023-07-02T14:04:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513992+0000", 2023-07-02T14:04:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9972000000000001e-05 2023-07-02T14:04:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514052+0000", 2023-07-02T14:04:02.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149264 2023-07-02T14:04:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514202+0000", 2023-07-02T14:04:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999 2023-07-02T14:04:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:02.037 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2952 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2883584~12288] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.356653262000002, 2023-07-02T14:04:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651, 2023-07-02T14:04:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:02.038 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.039 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:02.039 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:02.039 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:02.039 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2952 2023-07-02T14:04:02.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.039 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:02.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:02.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476760+0000", 2023-07-02T14:04:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3664e-05 2023-07-02T14:04:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476774+0000", 2023-07-02T14:04:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.166e-06 2023-07-02T14:04:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476783+0000", 2023-07-02T14:04:02.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8720000000000001e-06 2023-07-02T14:04:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476788+0000", 2023-07-02T14:04:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087180529999999999 2023-07-02T14:04:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485506+0000", 2023-07-02T14:04:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5680999999999994e-05 2023-07-02T14:04:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485602+0000", 2023-07-02T14:04:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000222365 2023-07-02T14:04:02.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485824+0000", 2023-07-02T14:04:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003277379 2023-07-02T14:04:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:02.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489101+0000", 2023-07-02T14:04:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025151739999999999 2023-07-02T14:04:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514253+0000", 2023-07-02T14:04:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4749000000000002e-05 2023-07-02T14:04:02.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514288+0000", 2023-07-02T14:04:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130634 2023-07-02T14:04:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:02.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514419+0000", 2023-07-02T14:04:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651 2023-07-02T14:04:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:02.048 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:02.048 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:02.050 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:04:02.050 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:02.049+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:02.150 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:02.150+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:02.178 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:02.178 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:02.178 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:04:02.178 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:02.189 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:04:02.251 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:02.250+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:02.317 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:02.318 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:02.318 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:04:02.318 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:04:02.318 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:02.328 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:04:02.352 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:02.351+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:02.453 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:02.452+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:02.472 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:02.472 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:04:02.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:04:02.472 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:02.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.473 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:5545 2.0 e81/46 2:159aabad:::rbd_data.125de1449605.000000000000005d:head v 81'1159, mlcod=81'1155)", 2023-07-02T14:04:02.473 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.389719+0000", 2023-07-02T14:04:02.473 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 23.080967007999998, 2023-07-02T14:04:02.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014352986, 2023-07-02T14:04:02.473 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:02.473 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.474 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:02.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:02.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.389719+0000", 2023-07-02T14:04:02.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:02.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:02.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.389719+0000", 2023-07-02T14:04:02.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:02.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:02.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.389718+0000", 2023-07-02T14:04:02.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037957160000000002 2023-07-02T14:04:02.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:02.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.393513+0000", 2023-07-02T14:04:02.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7331000000000001e-05 2023-07-02T14:04:02.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:02.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.393531+0000", 2023-07-02T14:04:02.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0229999999999998e-06 2023-07-02T14:04:02.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.393540+0000", 2023-07-02T14:04:02.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0124000000000002e-05 2023-07-02T14:04:02.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:02.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.393600+0000", 2023-07-02T14:04:02.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2815999999999999e-05 2023-07-02T14:04:02.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:02.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.393633+0000", 2023-07-02T14:04:02.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010399037 2023-07-02T14:04:02.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:02.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.404032+0000", 2023-07-02T14:04:02.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9761000000000001e-05 2023-07-02T14:04:02.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:02.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.404072+0000", 2023-07-02T14:04:02.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014352986 2023-07-02T14:04:02.480 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.480 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.481 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.481 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:04:02.481 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:02.481 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.573071377000002, 2023-07-02T14:04:02.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:04:02.482 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:02.482 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.482 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:02.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:02.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:02.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:02.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:02.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:02.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:02.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:02.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:04:02.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:04:02.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:02.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:04:02.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:04:02.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:02.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:04:02.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:04:02.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:04:02.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:04:02.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:02.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:04:02.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:04:02.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:02.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:04:02.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:04:02.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:02.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:04:02.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:04:02.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:02.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:04:02.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:04:02.488 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.488 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.489 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.489 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:04:02.489 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:02.489 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.568985658999999, 2023-07-02T14:04:02.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:04:02.490 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:02.490 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.490 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:02.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:02.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:02.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:02.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:02.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:02.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:02.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:02.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:04:02.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:04:02.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:02.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:04:02.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:04:02.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:02.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:04:02.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:04:02.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:04:02.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:04:02.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:02.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:04:02.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:04:02.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:02.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:04:02.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:04:02.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:02.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:04:02.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:04:02.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:02.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:04:02.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:04:02.496 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.496 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.497 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.497 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:04:02.497 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:02.497 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.565120877999998, 2023-07-02T14:04:02.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:04:02.497 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:02.498 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.498 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:02.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:02.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:02.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:02.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:02.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:02.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:02.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:02.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:04:02.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:04:02.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:02.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:04:02.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:04:02.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:02.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:04:02.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:04:02.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:04:02.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:04:02.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:02.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:04:02.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:04:02.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:02.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:04:02.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:04:02.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:02.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:04:02.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:04:02.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:02.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:04:02.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:04:02.504 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.504 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.505 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.505 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:04:02.505 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:02.505 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.561596616999999, 2023-07-02T14:04:02.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:04:02.505 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:02.506 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.506 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:02.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:02.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:02.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:02.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:02.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:02.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:02.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:02.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:04:02.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:04:02.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:02.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:04:02.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:04:02.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:02.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:04:02.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:04:02.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:04:02.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:04:02.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:02.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:04:02.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:04:02.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:02.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:04:02.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:04:02.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:02.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:04:02.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:04:02.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:02.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:04:02.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:04:02.512 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.512 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.513 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.513 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6124 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1328, mlcod=81'1319)", 2023-07-02T14:04:02.513 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:02.513 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.558081233999999, 2023-07-02T14:04:02.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019096974999999999, 2023-07-02T14:04:02.514 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:02.514 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.514 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:02.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:02.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:02.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:02.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:02.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:02.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:02.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:02.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912602+0000", 2023-07-02T14:04:02.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6741999999999999e-05 2023-07-02T14:04:02.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:02.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912629+0000", 2023-07-02T14:04:02.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5163e-05 2023-07-02T14:04:02.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:02.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912644+0000", 2023-07-02T14:04:02.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4800000000000001e-06 2023-07-02T14:04:02.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912653+0000", 2023-07-02T14:04:02.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071988909999999998 2023-07-02T14:04:02.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:02.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.919851+0000", 2023-07-02T14:04:02.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2949999999999999e-05 2023-07-02T14:04:02.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:02.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.919874+0000", 2023-07-02T14:04:02.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011800804 2023-07-02T14:04:02.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:02.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931675+0000", 2023-07-02T14:04:02.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6033999999999998e-05 2023-07-02T14:04:02.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:02.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931701+0000", 2023-07-02T14:04:02.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019096974999999999 2023-07-02T14:04:02.520 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.521 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.521 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.521 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6125 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1330, mlcod=81'1319)", 2023-07-02T14:04:02.521 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.915930+0000", 2023-07-02T14:04:02.521 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.554755548999999, 2023-07-02T14:04:02.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015810346999999999, 2023-07-02T14:04:02.522 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:02.522 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.522 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:02.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:02.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.915930+0000", 2023-07-02T14:04:02.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:02.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:02.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.915930+0000", 2023-07-02T14:04:02.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:02.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:02.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.915929+0000", 2023-07-02T14:04:02.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8398e-05 2023-07-02T14:04:02.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:02.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.915978+0000", 2023-07-02T14:04:02.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2022e-05 2023-07-02T14:04:02.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:02.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.916000+0000", 2023-07-02T14:04:02.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2780000000000001e-06 2023-07-02T14:04:02.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.916005+0000", 2023-07-02T14:04:02.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078621239999999998 2023-07-02T14:04:02.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:02.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923867+0000", 2023-07-02T14:04:02.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1858999999999999e-05 2023-07-02T14:04:02.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:02.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923889+0000", 2023-07-02T14:04:02.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078253369999999999 2023-07-02T14:04:02.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:02.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931714+0000", 2023-07-02T14:04:02.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5936999999999999e-05 2023-07-02T14:04:02.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:02.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931740+0000", 2023-07-02T14:04:02.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015810346999999999 2023-07-02T14:04:02.528 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.529 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.529 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.529 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6137 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1335, mlcod=81'1333)", 2023-07-02T14:04:02.529 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.934629+0000", 2023-07-02T14:04:02.529 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.536056408, 2023-07-02T14:04:02.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014672614, 2023-07-02T14:04:02.530 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:02.530 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.530 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:02.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:02.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.934629+0000", 2023-07-02T14:04:02.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:02.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:02.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.934629+0000", 2023-07-02T14:04:02.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:02.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:02.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.934628+0000", 2023-07-02T14:04:02.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030856529999999998 2023-07-02T14:04:02.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:02.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.937714+0000", 2023-07-02T14:04:02.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6008999999999999e-05 2023-07-02T14:04:02.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:02.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.937730+0000", 2023-07-02T14:04:02.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9440000000000005e-06 2023-07-02T14:04:02.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.937738+0000", 2023-07-02T14:04:02.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4077000000000003e-05 2023-07-02T14:04:02.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:02.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.937802+0000", 2023-07-02T14:04:02.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5251e-05 2023-07-02T14:04:02.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:02.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.937817+0000", 2023-07-02T14:04:02.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011457106 2023-07-02T14:04:02.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:02.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949275+0000", 2023-07-02T14:04:02.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.7217999999999999e-05 2023-07-02T14:04:02.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:02.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949302+0000", 2023-07-02T14:04:02.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014672614 2023-07-02T14:04:02.536 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.537 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.537 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.537 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6139 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1337, mlcod=81'1333)", 2023-07-02T14:04:02.537 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:02.537 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.531391373999998, 2023-07-02T14:04:02.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018571681, 2023-07-02T14:04:02.538 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:02.538 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.538 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:02.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:02.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:02.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:02.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:02.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:02.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:02.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:02.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939293+0000", 2023-07-02T14:04:02.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003764613 2023-07-02T14:04:02.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:02.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943058+0000", 2023-07-02T14:04:02.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5797000000000001e-05 2023-07-02T14:04:02.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:02.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943074+0000", 2023-07-02T14:04:02.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5089999999999996e-06 2023-07-02T14:04:02.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943081+0000", 2023-07-02T14:04:02.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014727100000000001 2023-07-02T14:04:02.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:02.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943228+0000", 2023-07-02T14:04:02.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2246000000000003e-05 2023-07-02T14:04:02.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:02.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943261+0000", 2023-07-02T14:04:02.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014576317 2023-07-02T14:04:02.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:02.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957837+0000", 2023-07-02T14:04:02.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8962000000000002e-05 2023-07-02T14:04:02.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:02.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957866+0000", 2023-07-02T14:04:02.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018571681 2023-07-02T14:04:02.545 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.545 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.545 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.545 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6140 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1338, mlcod=81'1333)", 2023-07-02T14:04:02.545 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.943097+0000", 2023-07-02T14:04:02.546 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.527588569999999, 2023-07-02T14:04:02.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014787599, 2023-07-02T14:04:02.546 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:02.546 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.546 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:02.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:02.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943097+0000", 2023-07-02T14:04:02.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:02.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:02.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943097+0000", 2023-07-02T14:04:02.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:02.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:02.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943096+0000", 2023-07-02T14:04:02.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4756999999999999e-05 2023-07-02T14:04:02.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:02.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943111+0000", 2023-07-02T14:04:02.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9079999999999995e-06 2023-07-02T14:04:02.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:02.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943119+0000", 2023-07-02T14:04:02.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4150000000000001e-06 2023-07-02T14:04:02.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943123+0000", 2023-07-02T14:04:02.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005177999 2023-07-02T14:04:02.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:02.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.948301+0000", 2023-07-02T14:04:02.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5087000000000003e-05 2023-07-02T14:04:02.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:02.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.948356+0000", 2023-07-02T14:04:02.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0095146889999999998 2023-07-02T14:04:02.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:02.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957871+0000", 2023-07-02T14:04:02.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.347e-05 2023-07-02T14:04:02.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:02.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957885+0000", 2023-07-02T14:04:02.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014787599 2023-07-02T14:04:02.553 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.553 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.553 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.553 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:04:02.553 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:02.553 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.525891818000002, 2023-07-02T14:04:02.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:04:02.554 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:02.554 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.554 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:02.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:02.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:02.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:02.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:02.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:02.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:02.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:02.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:04:02.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:04:02.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:02.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:04:02.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:04:02.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:02.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:04:02.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:04:02.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:04:02.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:04:02.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:02.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:04:02.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:04:02.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:02.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:04:02.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:04:02.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:02.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:04:02.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:04:02.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:02.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:04:02.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:04:02.560 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.561 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.561 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.561 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:04:02.561 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:02.561 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.521234459999999, 2023-07-02T14:04:02.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:04:02.562 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:02.562 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.562 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:02.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:02.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:02.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:02.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:02.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:02.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:02.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:02.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:04:02.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:04:02.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:02.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:04:02.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:04:02.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:02.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:04:02.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:04:02.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:04:02.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:04:02.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:02.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:04:02.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:04:02.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:02.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:04:02.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:04:02.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:02.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:04:02.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:04:02.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:02.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:04:02.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:04:02.569 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.569 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.569 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.570 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:04:02.570 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:02.570 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.520294596999999, 2023-07-02T14:04:02.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:04:02.570 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:02.570 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.570 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:02.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:02.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:02.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:02.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:02.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:02.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:02.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:02.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:04:02.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:04:02.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:02.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:04:02.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:04:02.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:02.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:04:02.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:04:02.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:04:02.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:04:02.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:02.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:04:02.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:04:02.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:02.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:04:02.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:04:02.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:02.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:04:02.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:04:02.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:02.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:04:02.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:04:02.577 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.577 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.577 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.578 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:04:02.578 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:02.578 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.518896719000001, 2023-07-02T14:04:02.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:04:02.578 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:02.578 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.578 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:02.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:02.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:02.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:02.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:02.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:02.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:02.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:02.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:04:02.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:04:02.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:02.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:04:02.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:04:02.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:02.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:04:02.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:04:02.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:04:02.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:04:02.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:02.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:04:02.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:04:02.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:02.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:04:02.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:04:02.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:02.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:04:02.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:04:02.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:02.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:04:02.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:04:02.585 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.585 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.585 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.586 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6148 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1344, mlcod=81'1335)", 2023-07-02T14:04:02.586 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.956703+0000", 2023-07-02T14:04:02.586 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.513982426999998, 2023-07-02T14:04:02.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017064030000000001, 2023-07-02T14:04:02.586 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:02.586 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.586 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:02.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:02.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956703+0000", 2023-07-02T14:04:02.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:02.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:02.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956703+0000", 2023-07-02T14:04:02.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:02.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:02.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956702+0000", 2023-07-02T14:04:02.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012505 2023-07-02T14:04:02.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:02.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956827+0000", 2023-07-02T14:04:02.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7309999999999999e-05 2023-07-02T14:04:02.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:02.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956845+0000", 2023-07-02T14:04:02.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7570000000000001e-06 2023-07-02T14:04:02.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956850+0000", 2023-07-02T14:04:02.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071775149999999998 2023-07-02T14:04:02.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:02.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964028+0000", 2023-07-02T14:04:02.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4773000000000002e-05 2023-07-02T14:04:02.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:02.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964053+0000", 2023-07-02T14:04:02.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096921820000000006 2023-07-02T14:04:02.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:02.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973745+0000", 2023-07-02T14:04:02.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2272999999999998e-05 2023-07-02T14:04:02.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:02.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973767+0000", 2023-07-02T14:04:02.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017064030000000001 2023-07-02T14:04:02.593 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.593 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.593 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.594 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6150 2.0 e81/46 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 81'1345, mlcod=81'1338)", 2023-07-02T14:04:02.594 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:02.594 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.510873772, 2023-07-02T14:04:02.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019172812000000001, 2023-07-02T14:04:02.594 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:02.594 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.594 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:02.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:02.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:02.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:02.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:02.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:02.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:02.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:02.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959810+0000", 2023-07-02T14:04:02.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048438650000000001 2023-07-02T14:04:02.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:02.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964653+0000", 2023-07-02T14:04:02.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2136e-05 2023-07-02T14:04:02.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:02.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964666+0000", 2023-07-02T14:04:02.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2169999999999999e-06 2023-07-02T14:04:02.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964672+0000", 2023-07-02T14:04:02.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00047561800000000003 2023-07-02T14:04:02.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:02.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965147+0000", 2023-07-02T14:04:02.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5891e-05 2023-07-02T14:04:02.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:02.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965173+0000", 2023-07-02T14:04:02.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013770973000000001 2023-07-02T14:04:02.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:02.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978944+0000", 2023-07-02T14:04:02.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0256999999999998e-05 2023-07-02T14:04:02.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:02.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978985+0000", 2023-07-02T14:04:02.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019172812000000001 2023-07-02T14:04:02.601 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.601 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.601 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.602 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6151 2.0 e81/46 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 81'1346, mlcod=81'1338)", 2023-07-02T14:04:02.602 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.964682+0000", 2023-07-02T14:04:02.602 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.506003970999998, 2023-07-02T14:04:02.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014325582, 2023-07-02T14:04:02.602 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:02.602 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.603 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:02.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:02.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964682+0000", 2023-07-02T14:04:02.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:02.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:02.604 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964682+0000", 2023-07-02T14:04:02.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:02.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.604 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:02.604 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964681+0000", 2023-07-02T14:04:02.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5908000000000001e-05 2023-07-02T14:04:02.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:02.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964697+0000", 2023-07-02T14:04:02.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1920000000000005e-06 2023-07-02T14:04:02.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964705+0000", 2023-07-02T14:04:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.493e-06 2023-07-02T14:04:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964709+0000", 2023-07-02T14:04:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075784019999999997 2023-07-02T14:04:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:02.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972288+0000", 2023-07-02T14:04:02.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3762e-05 2023-07-02T14:04:02.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:02.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972302+0000", 2023-07-02T14:04:02.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066885219999999997 2023-07-02T14:04:02.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:02.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978990+0000", 2023-07-02T14:04:02.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7011999999999999e-05 2023-07-02T14:04:02.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:02.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.979007+0000", 2023-07-02T14:04:02.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014325582 2023-07-02T14:04:02.609 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.609 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.609 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.610 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6153 2.0 e81/46)", 2023-07-02T14:04:02.610 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.964714+0000", 2023-07-02T14:04:02.610 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.505971160000001, 2023-07-02T14:04:02.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014315782000000001, 2023-07-02T14:04:02.610 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:02.610 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.611 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:02.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:02.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964714+0000", 2023-07-02T14:04:02.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:02.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:02.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964714+0000", 2023-07-02T14:04:02.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:02.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:02.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964714+0000", 2023-07-02T14:04:02.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2535999999999999e-05 2023-07-02T14:04:02.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:02.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964737+0000", 2023-07-02T14:04:02.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.892e-06 2023-07-02T14:04:02.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:02.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964744+0000", 2023-07-02T14:04:02.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5040000000000004e-06 2023-07-02T14:04:02.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964749+0000", 2023-07-02T14:04:02.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0083469990000000008 2023-07-02T14:04:02.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:02.615 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973096+0000", 2023-07-02T14:04:02.615 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8771e-05 2023-07-02T14:04:02.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:02.616 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973115+0000", 2023-07-02T14:04:02.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058955850000000001 2023-07-02T14:04:02.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.616 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.616 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:02.616 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.979010+0000", 2023-07-02T14:04:02.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9847000000000002e-05 2023-07-02T14:04:02.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:02.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.979030+0000", 2023-07-02T14:04:02.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014315782000000001 2023-07-02T14:04:02.617 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.617 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.617 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.618 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6320 2.7 e81/63 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head v 81'852, mlcod=81'849)", 2023-07-02T14:04:02.618 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:40.070929+0000", 2023-07-02T14:04:02.618 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.399756450000002, 2023-07-02T14:04:02.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014144276000000001, 2023-07-02T14:04:02.618 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:02.618 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.619 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:02.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:02.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.070929+0000", 2023-07-02T14:04:02.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:02.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:02.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.070929+0000", 2023-07-02T14:04:02.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:02.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:02.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.070928+0000", 2023-07-02T14:04:02.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0010305239999999999 2023-07-02T14:04:02.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:02.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.071959+0000", 2023-07-02T14:04:02.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8085999999999999e-05 2023-07-02T14:04:02.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:02.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.071977+0000", 2023-07-02T14:04:02.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3080000000000008e-06 2023-07-02T14:04:02.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.071985+0000", 2023-07-02T14:04:02.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7088999999999994e-05 2023-07-02T14:04:02.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:02.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.072052+0000", 2023-07-02T14:04:02.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1041000000000003e-05 2023-07-02T14:04:02.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:02.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.072083+0000", 2023-07-02T14:04:02.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012949866000000001 2023-07-02T14:04:02.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:02.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.085033+0000", 2023-07-02T14:04:02.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0262000000000002e-05 2023-07-02T14:04:02.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.625 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:02.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:40.085073+0000", 2023-07-02T14:04:02.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014144276000000001 2023-07-02T14:04:02.625 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.625 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.625 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.626 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:04:02.626 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:02.626 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 1.108706223, 2023-07-02T14:04:02.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:04:02.626 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:02.627 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:02.627 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:02.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:02.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:02.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:02.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:02.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:02.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:02.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:02.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:04:02.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:04:02.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:02.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:04:02.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:04:02.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:04:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:04:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:04:02.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:04:02.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:02.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:04:02.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:04:02.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:02.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:04:02.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:04:02.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:02.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:04:02.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:04:02.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:02.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:02.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:02.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:04:02.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:04:02.633 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.633 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.634 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.634 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:02.634 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:02.634 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:02.636 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:04:02.636 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:04:02.637 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:02.634+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:02.736 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:02.734+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:02.793 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:04:02.838 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:02.836+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:02.842 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:02.842 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:02.842 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:04:02.842 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:02.855 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:04:02.938 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:02.937+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:03.005 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:03.006 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:03.006 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:04:03.006 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:04:03.006 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:03.018 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:03.026 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:03.032 INFO:teuthology.orchestra.run.smithi022.stdout:get_pg_num_history pg_num_history pg_num_history(e87 pg_nums {1={12=1},2={14=8}} deleted_pools ) 2023-07-02T14:04:03.032 INFO:teuthology.orchestra.run.smithi022.stdout:Importing pgid 1.0 2023-07-02T14:04:03.032 INFO:teuthology.orchestra.run.smithi022.stdout:write_pg epoch 72 info 1.0( empty local-lis/les=71/72 n=0 ec=12/12 lis/c=71/71 les/c/f=72/72/0 sis=71) 2023-07-02T14:04:03.037 INFO:teuthology.orchestra.run.smithi022.stdout:Import successful 2023-07-02T14:04:03.038 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:03.037+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:03.058 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:03.092 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:03.125 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:03.139 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:03.138+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:03.154 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:03.154 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:04:03.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:04:03.154 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:03.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.154 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:03.155 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:03.155 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.860526913000001, 2023-07-02T14:04:03.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:04:03.155 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:03.155 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:03.155 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:03.155 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:03.156 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:03.156 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:04:03.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.156 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:03.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:03.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:03.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:03.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:03.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:03.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:03.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:03.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:04:03.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:04:03.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:03.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:04:03.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:04:03.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:03.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:04:03.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:04:03.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:03.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:04:03.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:04:03.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:03.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:04:03.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:04:03.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:03.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:04:03.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:04:03.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:03.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:04:03.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:04:03.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:03.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:04:03.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:04:03.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:03.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:04:03.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:04:03.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:03.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:04:03.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:04:03.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:03.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:04:03.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:04:03.165 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.165 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.165 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.165 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:03.165 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:03.165 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.860291889999999, 2023-07-02T14:04:03.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:04:03.166 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:03.166 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:03.166 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:03.166 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:03.166 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:03.166 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:04:03.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.167 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:03.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:03.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:03.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:03.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:03.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:03.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:03.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:03.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:04:03.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:04:03.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:03.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:04:03.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:04:03.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:03.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:04:03.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:04:03.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:03.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:04:03.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:04:03.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:03.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:04:03.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:04:03.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:03.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:04:03.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:04:03.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:03.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:04:03.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:04:03.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:03.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:04:03.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:04:03.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:03.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:04:03.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:04:03.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:03.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:04:03.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:04:03.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:03.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:04:03.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:04:03.175 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.175 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.176 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.176 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:03.176 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:03.176 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.860092756, 2023-07-02T14:04:03.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:04:03.176 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:03.177 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:03.177 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:03.177 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:03.177 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:03.177 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:04:03.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.177 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:03.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:03.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:03.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:03.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:03.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:03.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:03.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:03.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:04:03.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:04:03.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:03.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:04:03.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:04:03.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:03.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:04:03.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:04:03.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:03.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:04:03.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:04:03.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:03.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:04:03.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:04:03.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:03.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:04:03.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:04:03.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:03.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:04:03.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:04:03.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:03.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:04:03.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:04:03.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:03.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:04:03.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:04:03.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:03.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:04:03.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:04:03.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:03.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:04:03.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:04:03.186 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.186 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.186 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.187 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:03.187 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:03.187 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.859727632000002, 2023-07-02T14:04:03.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:04:03.187 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:03.187 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:03.187 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:03.187 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:03.188 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:03.188 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:04:03.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.188 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:03.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:03.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:03.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:03.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:03.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:03.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:03.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:03.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:04:03.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:04:03.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:03.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:04:03.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:04:03.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:03.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:04:03.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:04:03.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:03.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:04:03.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:04:03.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:03.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:04:03.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:04:03.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:03.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:04:03.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:04:03.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:03.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:04:03.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:04:03.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:03.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:04:03.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:04:03.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:03.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:04:03.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:04:03.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:03.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:04:03.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:04:03.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:03.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:04:03.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:04:03.197 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.197 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.197 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.197 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:03.198 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:03.198 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.859646570999999, 2023-07-02T14:04:03.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:04:03.198 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:03.198 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:03.198 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:03.198 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:03.198 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:03.199 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:04:03.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.199 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:03.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.199 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:03.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:03.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:03.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:03.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:03.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:03.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:03.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:04:03.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:04:03.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:03.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:04:03.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:04:03.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:03.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:04:03.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:04:03.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:03.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:04:03.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:04:03.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:03.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:04:03.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:04:03.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:03.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:04:03.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:04:03.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:03.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:04:03.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:04:03.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:03.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:04:03.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:04:03.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:03.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:04:03.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:04:03.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:03.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:04:03.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:04:03.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:03.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:04:03.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:04:03.208 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.208 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.208 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.208 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:03.209 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:03.209 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.859554900000006, 2023-07-02T14:04:03.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:04:03.209 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:03.209 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:03.209 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:03.209 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:03.209 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:03.210 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:04:03.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.210 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:03.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:03.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:03.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:03.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:03.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:03.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:03.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:03.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:04:03.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:04:03.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:03.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:04:03.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:04:03.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:03.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:04:03.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:04:03.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:03.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:04:03.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:04:03.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:03.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:04:03.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:04:03.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:03.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:04:03.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:04:03.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:03.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:04:03.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:04:03.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:03.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:04:03.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:04:03.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:03.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:04:03.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:04:03.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:03.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:04:03.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:04:03.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:03.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:04:03.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:04:03.218 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.219 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.219 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.219 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:03.219 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:03.219 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.859215457999994, 2023-07-02T14:04:03.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:04:03.220 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:03.220 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:03.220 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:03.220 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:03.220 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:03.220 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:04:03.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.221 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:03.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:03.221 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:03.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:03.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:03.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:03.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:03.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:03.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:04:03.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:04:03.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:03.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:04:03.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:04:03.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:03.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:04:03.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:04:03.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:03.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:04:03.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:04:03.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:03.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:04:03.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:04:03.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:03.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:04:03.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:04:03.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:03.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:04:03.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:04:03.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:03.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:04:03.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:04:03.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:03.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:04:03.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:04:03.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:03.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:04:03.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:04:03.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:03.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:04:03.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:04:03.229 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.229 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.229 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.230 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:03.230 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:03.230 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.859029405000001, 2023-07-02T14:04:03.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:04:03.230 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:03.230 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:03.231 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:03.231 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:03.231 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:03.231 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:04:03.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.231 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:03.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:03.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:03.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:03.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:03.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:03.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:03.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:03.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:03.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:04:03.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:03.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:04:03.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:04:03.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:03.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:04:03.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:04:03.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:03.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:04:03.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:04:03.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:03.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:04:03.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:04:03.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:03.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:04:03.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:04:03.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:03.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:04:03.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:04:03.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:03.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:04:03.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:04:03.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:03.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:04:03.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:04:03.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:03.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:04:03.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:04:03.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.240 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:03.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:04:03.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:04:03.240 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.240 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.240 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.241 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:03.241 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:03.241 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.858871602999997, 2023-07-02T14:04:03.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:04:03.241 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:03.241 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:03.241 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:03.241 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:03.242 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:03.242 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:04:03.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.242 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:03.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:03.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:03.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:03.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:03.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:03.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:03.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:03.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:04:03.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:04:03.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:03.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:04:03.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:04:03.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:03.245 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:04:03.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:04:03.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:03.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:04:03.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:04:03.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.246 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.246 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:03.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:04:03.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:04:03.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:03.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:04:03.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:04:03.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:03.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:04:03.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:04:03.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:03.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:04:03.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:04:03.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:03.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:04:03.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:04:03.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:03.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:04:03.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:04:03.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:03.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:04:03.251 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:04:03.251 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.251 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.251 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.251 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:03.251 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:03.252 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.858679820999996, 2023-07-02T14:04:03.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:04:03.252 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:03.252 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:03.252 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:03.252 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:03.252 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:03.252 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:04:03.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.253 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:03.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:03.253 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:03.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:03.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:03.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:03.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:03.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:03.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:04:03.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:04:03.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:03.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:04:03.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:04:03.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:03.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:04:03.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:04:03.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:03.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:04:03.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:04:03.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:03.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:04:03.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:04:03.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:03.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:04:03.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:04:03.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:03.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:04:03.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:04:03.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:03.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:04:03.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:04:03.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:03.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:04:03.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:04:03.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:03.261 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:04:03.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:04:03.261 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.261 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:03.261 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:04:03.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:04:03.261 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.262 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.262 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.262 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.262 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:03.262 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:03.262 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.858504160999999, 2023-07-02T14:04:03.262 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:04:03.263 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:03.263 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:03.263 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:03.263 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:03.263 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:03.263 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:04:03.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.263 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:03.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.264 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:03.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:03.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:03.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.264 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:03.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:03.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:03.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:03.265 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:03.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:04:03.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:03.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:04:03.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:04:03.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:03.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:04:03.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:04:03.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:03.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:04:03.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:04:03.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:03.268 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:04:03.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:04:03.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:03.268 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:04:03.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:04:03.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:03.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:04:03.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:04:03.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:03.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:04:03.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:04:03.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.271 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:03.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:04:03.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:04:03.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.271 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:03.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:04:03.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:04:03.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:03.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:04:03.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:04:03.272 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.273 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.273 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.273 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:03.273 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:03.273 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.858409972999993, 2023-07-02T14:04:03.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:04:03.274 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:03.274 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:03.274 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:03.274 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:03.274 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:03.274 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:04:03.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.275 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:03.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.275 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:03.275 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:03.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:03.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.275 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:03.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:03.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:03.276 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:03.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:04:03.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:04:03.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.277 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:03.277 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:04:03.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:04:03.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.277 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:03.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:04:03.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:04:03.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:03.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:04:03.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:04:03.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:03.279 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:04:03.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:04:03.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:03.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:04:03.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:04:03.280 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:03.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:04:03.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:04:03.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:03.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:04:03.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:04:03.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.282 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:03.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:04:03.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:04:03.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.282 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:03.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:04:03.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:04:03.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:03.283 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:04:03.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:04:03.283 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.283 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.284 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.284 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:03.284 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:03.284 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.858206586999998, 2023-07-02T14:04:03.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:04:03.285 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:03.285 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:03.285 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:03.285 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:03.285 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:03.285 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:04:03.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.285 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:03.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:03.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:03.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:03.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:03.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:03.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:03.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:03.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:04:03.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:04:03.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:03.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:04:03.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:04:03.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:03.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:04:03.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:04:03.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:03.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:04:03.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:04:03.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:03.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:04:03.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:04:03.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:03.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:04:03.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:04:03.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:03.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:04:03.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:04:03.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:03.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:04:03.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:04:03.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:03.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:04:03.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:04:03.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:03.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:04:03.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:04:03.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:03.294 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:04:03.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:04:03.294 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.294 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.294 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.295 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:03.295 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:03.295 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.858055930999996, 2023-07-02T14:04:03.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:04:03.295 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:03.295 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:03.296 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:03.296 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:03.296 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:03.296 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:04:03.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.296 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:03.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:03.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:03.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:03.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:03.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:03.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:03.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:03.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:04:03.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:04:03.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:03.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:04:03.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:04:03.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:03.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:04:03.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:04:03.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:03.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:04:03.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:04:03.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:03.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:04:03.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:04:03.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:03.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:04:03.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:04:03.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:03.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:04:03.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:04:03.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:03.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:04:03.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:04:03.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:03.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:04:03.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:04:03.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:03.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:04:03.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:04:03.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:03.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:04:03.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:04:03.305 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.305 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.305 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.306 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:03.306 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:03.306 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.857983227999995, 2023-07-02T14:04:03.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:04:03.306 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:03.306 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:03.307 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:03.307 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:03.307 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:03.307 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:04:03.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.307 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:03.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:03.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:03.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:03.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:03.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:03.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:03.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:03.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:03.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:04:03.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:03.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:04:03.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:04:03.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:03.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:04:03.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:04:03.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:03.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:04:03.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:04:03.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:03.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:04:03.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:04:03.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:03.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:04:03.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:04:03.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:03.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:04:03.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:04:03.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:03.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:04:03.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:04:03.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:03.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:04:03.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:04:03.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:03.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:04:03.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:04:03.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:03.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:04:03.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:04:03.316 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.316 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.316 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.317 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:03.317 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:03.317 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.857866238, 2023-07-02T14:04:03.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:04:03.317 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:03.318 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:03.318 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:03.318 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:03.318 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:03.318 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:04:03.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.318 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:03.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:03.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:03.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:03.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:03.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:03.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:03.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:03.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:03.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:04:03.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:03.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:04:03.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:04:03.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:03.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:04:03.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:04:03.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.322 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:03.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:04:03.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:04:03.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:03.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:04:03.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:04:03.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:03.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:04:03.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:04:03.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.324 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:03.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:04:03.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:04:03.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:03.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:04:03.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:04:03.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:03.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:04:03.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:04:03.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:03.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:04:03.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:04:03.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:03.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:04:03.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:04:03.327 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.327 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.327 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.328 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:03.328 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:03.328 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.857716787000001, 2023-07-02T14:04:03.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:04:03.328 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:03.329 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:03.329 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:03.329 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:03.329 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:03.329 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:04:03.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.329 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:03.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:03.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:03.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:03.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:03.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:03.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:03.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:03.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:04:03.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:04:03.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:03.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:04:03.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:04:03.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:03.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:04:03.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:04:03.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:03.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:04:03.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:04:03.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:03.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:04:03.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:04:03.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:03.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:04:03.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:04:03.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:03.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:04:03.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:04:03.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:03.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:04:03.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:04:03.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:03.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:04:03.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:04:03.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:03.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:04:03.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:04:03.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:03.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:04:03.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:04:03.338 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.338 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.338 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.339 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:03.339 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:03.339 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.836240814000007, 2023-07-02T14:04:03.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:04:03.339 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:03.339 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:03.340 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:03.340 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:03.340 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:03.340 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:04:03.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.340 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:03.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:03.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:03.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:03.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:03.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:03.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:03.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:03.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:04:03.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:04:03.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:03.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:04:03.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:04:03.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:03.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:04:03.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:04:03.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:03.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:04:03.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:04:03.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:03.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:04:03.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:04:03.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:03.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:04:03.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:04:03.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:03.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:04:03.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:04:03.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:03.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:04:03.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:04:03.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:03.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:04:03.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:04:03.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:03.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:04:03.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:04:03.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:03.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:04:03.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:04:03.349 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.349 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.349 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.349 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:03.350 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:03.350 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.836108277999998, 2023-07-02T14:04:03.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:04:03.350 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:03.350 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:03.350 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:03.350 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:03.350 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:03.351 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:04:03.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.351 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:03.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:03.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:03.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:03.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:03.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:03.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:03.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:03.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:04:03.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:04:03.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:03.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:04:03.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:04:03.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:03.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:04:03.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:04:03.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:03.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:04:03.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:04:03.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:03.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:04:03.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:04:03.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:03.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:04:03.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:04:03.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:03.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:04:03.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:04:03.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:03.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:04:03.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:04:03.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:03.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:04:03.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:04:03.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:03.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:04:03.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:04:03.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:03.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:04:03.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:04:03.360 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.360 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.360 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.360 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:03.360 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:03.360 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.774771616999999, 2023-07-02T14:04:03.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:04:03.361 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:03.361 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:03.361 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:03.361 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:03.361 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:03.361 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:04:03.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.362 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:03.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:03.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:03.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:03.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:03.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:03.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:03.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:03.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:04:03.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:04:03.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:03.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:04:03.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:04:03.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:03.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:04:03.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:04:03.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:03.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:04:03.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:04:03.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:03.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:04:03.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:04:03.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:03.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:04:03.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:04:03.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:03.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:04:03.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:04:03.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:03.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:04:03.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:04:03.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:03.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:04:03.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:04:03.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:03.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:04:03.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:04:03.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:03.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:03.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:03.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:04:03.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:04:03.370 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.370 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.371 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.371 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:03.371 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:03.371 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:03.372 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:03.373 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:03.374 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:04:03.374 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:04:03.374 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:04:03.375 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:03.375 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:04:03.375 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:04:03.375 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:03.376 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:03.371+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:03.475 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:03.472+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:03.552 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f /var/log/ceph/exp.1.0.3 2023-07-02T14:04:03.573 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:03.572+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:03.578 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-02T14:04:03.595 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap15 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap16 b3fbb028-e267-4d25-a750-93ba18a2316a@snap16.diff 2023-07-02T14:04:03.617 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:03.617 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:03.638 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:04:03.680 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:03.679+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:03.689 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete...2023-07-02T14:04:03.688+0000 7f1940140700 0 -- 172.21.15.163:0/925391742 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x55a94205a440 msgr2=0x55a94205c830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:03.713 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:03.711+0000 7f1940140700 0 -- 172.21.15.163:0/925391742 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f1924036bb0 msgr2=0x7f1924038fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:03.744 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 51% complete... Exporting image: 100% complete...done. 2023-07-02T14:04:03.748 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:04:03.749 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=17 2023-07-02T14:04:03.749 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap17 2023-07-02T14:04:03.775 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:03.774+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:03.875 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:03.874+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:03.900 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:03.901 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:03.911 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:04:03.994 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:03.993+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:04.023 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:04:04.023 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":92,"fsid":"9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2","created":"2023-07-02T14:01:37.298060+0000","modified":"2023-07-02T14:04:02.572914+0000","last_up_change":"2023-07-02T14:03:54.552411+0000","last_in_change":"2023-07-02T14:03:33.379208+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-07-02T14:01:45.323455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-02T14:01:47.448897+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":91,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"43c446fe-c811-4ee3-99e8-d7452edfc1c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":83359},{"type":"v1","addr":"172.21.15.16:6801","nonce":83359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":83359},{"type":"v1","addr":"172.21.15.16:6803","nonce":83359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":83359},{"type":"v1","addr":"172.21.15.16:6807","nonce":83359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":83359},{"type":"v1","addr":"172.21.15.16:6805","nonce":83359}]},"public_addr":"172.21.15.16:6801/83359","cluster_addr":"172.21.15.16:6803/83359","heartbeat_back_addr":"172.21.15.16:6807/83359","heartbeat_front_addr":"172.21.15.16:6805/83359","state":["exists","up"]},{"osd":1,"uuid":"4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":83360},{"type":"v1","addr":"172.21.15.16:6809","nonce":83360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":83360},{"type":"v1","addr":"172.21.15.16:6811","nonce":83360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":83360},{"type":"v1","addr":"172.21.15.16:6815","nonce":83360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":83360},{"type":"v1","addr":"172.21.15.16:6813","nonce":83360}]},"public_addr":"172.21.15.16:6809/83360","cluster_addr":"172.21.15.16:6811/83360","heartbeat_back_addr":"172.21.15.16:6815/83360","heartbeat_front_addr":"172.21.15.16:6813/83360","state":["exists","up"]},{"osd":2,"uuid":"99f8ea0a-5a71-4abe-b087-8e891fd11d1b","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":83361},{"type":"v1","addr":"172.21.15.16:6817","nonce":83361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":83361},{"type":"v1","addr":"172.21.15.16:6819","nonce":83361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":83361},{"type":"v1","addr":"172.21.15.16:6823","nonce":83361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":83361},{"type":"v1","addr":"172.21.15.16:6821","nonce":83361}]},"public_addr":"172.21.15.16:6817/83361","cluster_addr":"172.21.15.16:6819/83361","heartbeat_back_addr":"172.21.15.16:6823/83361","heartbeat_front_addr":"172.21.15.16:6821/83361","state":["exists","up"]},{"osd":3,"uuid":"d285b4d5-0bef-421a-b84a-943e485b10c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":63,"down_at":88,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":83350},{"type":"v1","addr":"172.21.15.22:6809","nonce":83350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":83350},{"type":"v1","addr":"172.21.15.22:6811","nonce":83350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":83350},{"type":"v1","addr":"172.21.15.22:6815","nonce":83350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":83350},{"type":"v1","addr":"172.21.15.22:6813","nonce":83350}]},"public_addr":"172.21.15.22:6809/83350","cluster_addr":"172.21.15.22:6811/83350","heartbeat_back_addr":"172.21.15.22:6815/83350","heartbeat_front_addr":"172.21.15.22:6813/83350","state":["exists"]},{"osd":4,"uuid":"7ba0b0db-e407-4c63-8278-aab09f3082c4","up":1,"in":1,"weight":1,"primary_affinity":0.23297119140625,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":83351},{"type":"v1","addr":"172.21.15.22:6801","nonce":83351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":83351},{"type":"v1","addr":"172.21.15.22:6803","nonce":83351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":83351},{"type":"v1","addr":"172.21.15.22:6807","nonce":83351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":83351},{"type":"v1","addr":"172.21.15.22:6805","nonce":83351}]},"public_addr":"172.21.15.22:6801/83351","cluster_addr":"172.21.15.22:6803/83351","heartbeat_back_addr":"172.21.15.22:6807/83351","heartbeat_front_addr":"172.21.15.22:6805/83351","state":["exists","up"]},{"osd":5,"uuid":"29663bfa-f6fe-4ce7-b45a-a9b7b4424c42","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":83352},{"type":"v1","addr":"172.21.15.22:6817","nonce":83352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":83352},{"type":"v1","addr":"172.21.15.22:6819","nonce":83352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":83352},{"type":"v1","addr":"172.21.15.22:6823","nonce":83352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":83352},{"type":"v1","addr":"172.21.15.22:6821","nonce":83352}]},"public_addr":"172.21.15.22:6817/83352","cluster_addr":"172.21.15.22:6819/83352","heartbeat_back_addr":"172.21.15.22:6823/83352","heartbeat_front_addr":"172.21.15.22:6821/83352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.001178+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.184181+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":65536,"last_purged_snaps_scrub":"2023-07-02T14:01:44.274854+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.186217+0000","dead_epoch":88},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.247442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":65536,"last_purged_snaps_scrub":"2023-07-02T14:01:44.220075+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[0,3]},{"pgid":"2.0","osds":[4,1]},{"pgid":"2.2","osds":[1,5]},{"pgid":"2.4","osds":[5,0]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-02T14:04:04.037 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-07-02T14:01:45.323455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-02T14:01:47.448897+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 19, 'snap_epoch': 91, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-07-02T14:04:04.037 DEBUG:teuthology.orchestra.run.smithi022:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op apply-layout-settings --pool device_health_metrics 2023-07-02T14:04:04.075 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:04.075 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:04.076 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:04.075+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:04.085 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:04:04.177 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:04.176+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:04.251 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:04.251 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:04.261 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:04:04.278 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:04.278+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:04.379 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:04.378+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:04.385 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:04.385 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:04.395 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:04:04.480 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:04.479+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:04.559 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:04.560 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:04.570 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:04:04.606 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:04:04.611 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:04.610+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:04.612 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 3 2023-07-02T14:04:04.638 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:04:04.682 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:04.681+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:04.783 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:04.782+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:04.802 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:04.802 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:04.814 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:04:04.885 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:04.884+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:04.985 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:04.984+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:05.035 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:05.035 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:05.056 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:04:05.097 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:05.096+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:05.186 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:05.185+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:05.221 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:05.221 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:05.231 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:04:05.287 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:05.287+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:05.355 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:05.356 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:05.366 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:04:05.377 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:04:05.420 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:05.419+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:05.489 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:05.489+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:05.490 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:05.490 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:05.501 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:04:05.514 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:05.532 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:05.549 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:05.567 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:05.585 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:05.585 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:04:05.585 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:04:05.585 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:04:05.585 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:05.586 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:04:05.586 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:04:05.588 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:05.589 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:05.591 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:04:05.592 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:05.591+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:05.683 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:05.683 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:04:05.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:04:05.684 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:05.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.684 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:04:05.684 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:05.684 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.783557059, 2023-07-02T14:04:05.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:04:05.685 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:05.685 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:05.685 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:05.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:05.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:05.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:05.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:05.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:05.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:05.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:05.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:04:05.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:04:05.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:05.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:04:05.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:04:05.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:05.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:04:05.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:04:05.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:05.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:04:05.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:04:05.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:05.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:04:05.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:04:05.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:05.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:04:05.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:04:05.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:05.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:04:05.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:04:05.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:05.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:04:05.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:04:05.691 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.692 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.692 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.692 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:04:05.692 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:05.692 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.779471341000001, 2023-07-02T14:04:05.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:04:05.693 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:05.693 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:05.693 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:05.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:05.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:05.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:05.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:05.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:05.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:05.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:05.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:04:05.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:04:05.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:05.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:04:05.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:04:05.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:05.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:04:05.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:04:05.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:05.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:04:05.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:04:05.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:05.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:04:05.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:04:05.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:05.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:04:05.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:04:05.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:05.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:04:05.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:04:05.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:05.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:04:05.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:04:05.699 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.699 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.699 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.700 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:04:05.700 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:05.700 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.77560656, 2023-07-02T14:04:05.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:04:05.700 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:05.701 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:05.701 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:05.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:05.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:05.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:05.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:05.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:05.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:05.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:05.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:04:05.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:04:05.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:05.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:04:05.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:04:05.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:05.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:04:05.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:04:05.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:05.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:04:05.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:04:05.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:05.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:04:05.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:04:05.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:05.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:04:05.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:04:05.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:05.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:04:05.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:04:05.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:05.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:04:05.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:04:05.707 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.707 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.707 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.708 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:04:05.708 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:05.708 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.772082299000001, 2023-07-02T14:04:05.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:04:05.708 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:05.708 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:05.708 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:05.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:05.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:05.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:05.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:05.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:05.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:05.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:05.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:04:05.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:04:05.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:05.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:04:05.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:04:05.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:05.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:04:05.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:04:05.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:05.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:04:05.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:04:05.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:05.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:04:05.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:04:05.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:05.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:04:05.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:04:05.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:05.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:04:05.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:04:05.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:05.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:04:05.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:04:05.715 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.715 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.715 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.716 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6124 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1328, mlcod=81'1319)", 2023-07-02T14:04:05.716 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:05.716 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.768566916000001, 2023-07-02T14:04:05.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019096974999999999, 2023-07-02T14:04:05.716 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:05.716 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:05.716 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:05.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:05.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:05.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:05.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:05.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:05.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:05.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:05.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912602+0000", 2023-07-02T14:04:05.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6741999999999999e-05 2023-07-02T14:04:05.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:05.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912629+0000", 2023-07-02T14:04:05.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5163e-05 2023-07-02T14:04:05.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:05.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912644+0000", 2023-07-02T14:04:05.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4800000000000001e-06 2023-07-02T14:04:05.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:05.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912653+0000", 2023-07-02T14:04:05.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071988909999999998 2023-07-02T14:04:05.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:05.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.919851+0000", 2023-07-02T14:04:05.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2949999999999999e-05 2023-07-02T14:04:05.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:05.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.919874+0000", 2023-07-02T14:04:05.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011800804 2023-07-02T14:04:05.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:05.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931675+0000", 2023-07-02T14:04:05.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6033999999999998e-05 2023-07-02T14:04:05.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:05.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931701+0000", 2023-07-02T14:04:05.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019096974999999999 2023-07-02T14:04:05.723 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.723 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.723 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.723 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6139 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1337, mlcod=81'1333)", 2023-07-02T14:04:05.723 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:05.724 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.741877056, 2023-07-02T14:04:05.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018571681, 2023-07-02T14:04:05.724 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:05.724 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:05.724 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:05.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:05.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:05.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:05.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:05.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:05.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:05.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:05.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939293+0000", 2023-07-02T14:04:05.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003764613 2023-07-02T14:04:05.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:05.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943058+0000", 2023-07-02T14:04:05.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5797000000000001e-05 2023-07-02T14:04:05.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:05.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943074+0000", 2023-07-02T14:04:05.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5089999999999996e-06 2023-07-02T14:04:05.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:05.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943081+0000", 2023-07-02T14:04:05.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014727100000000001 2023-07-02T14:04:05.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:05.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943228+0000", 2023-07-02T14:04:05.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2246000000000003e-05 2023-07-02T14:04:05.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:05.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943261+0000", 2023-07-02T14:04:05.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014576317 2023-07-02T14:04:05.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:05.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957837+0000", 2023-07-02T14:04:05.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8962000000000002e-05 2023-07-02T14:04:05.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:05.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957866+0000", 2023-07-02T14:04:05.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018571681 2023-07-02T14:04:05.730 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.731 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.731 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.731 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:04:05.731 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:05.731 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.7363775, 2023-07-02T14:04:05.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:04:05.732 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:05.732 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:05.732 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:05.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:05.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:05.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:05.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:05.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:05.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:05.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:05.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:04:05.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:04:05.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:05.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:04:05.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:04:05.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:05.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:04:05.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:04:05.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:05.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:04:05.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:04:05.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:05.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:04:05.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:04:05.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:05.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:04:05.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:04:05.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:05.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:04:05.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:04:05.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:05.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:04:05.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:04:05.738 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.738 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.739 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.739 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:04:05.739 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:05.739 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.731720142, 2023-07-02T14:04:05.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:04:05.739 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:05.740 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:05.740 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:05.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:05.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:05.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:05.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:05.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:05.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:05.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:05.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:04:05.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:04:05.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:05.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:04:05.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:04:05.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:05.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:04:05.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:04:05.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:05.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:04:05.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:04:05.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:05.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:04:05.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:04:05.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:05.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:04:05.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:04:05.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:05.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:04:05.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:04:05.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:05.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:04:05.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:04:05.746 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.746 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.746 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.747 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:04:05.747 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:05.747 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.730780279000001, 2023-07-02T14:04:05.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:04:05.747 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:05.747 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:05.747 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:05.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:05.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:05.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:05.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:05.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:05.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:05.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:05.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:04:05.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:04:05.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:05.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:04:05.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:04:05.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:05.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:04:05.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:04:05.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:05.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:04:05.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:04:05.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:05.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:04:05.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:04:05.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:05.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:04:05.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:04:05.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:05.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:04:05.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:04:05.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:05.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:04:05.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:04:05.754 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.754 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.754 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.754 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:04:05.754 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:05.755 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.729382400999999, 2023-07-02T14:04:05.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:04:05.755 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:05.755 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:05.755 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:05.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:05.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:05.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:05.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:05.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:05.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:05.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:05.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:04:05.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:04:05.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:05.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:04:05.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:04:05.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:05.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:04:05.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:04:05.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:05.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:04:05.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:04:05.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:05.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:04:05.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:04:05.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:05.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:04:05.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:04:05.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:05.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:04:05.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:04:05.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:05.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:04:05.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:04:05.762 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.762 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.762 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.762 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6150 2.0 e81/46 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 81'1345, mlcod=81'1338)", 2023-07-02T14:04:05.762 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:05.762 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 25.721359454000002, 2023-07-02T14:04:05.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019172812000000001, 2023-07-02T14:04:05.763 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:05.763 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:05.763 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:05.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:05.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:05.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:05.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:05.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:05.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:05.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:05.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959810+0000", 2023-07-02T14:04:05.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048438650000000001 2023-07-02T14:04:05.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:05.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964653+0000", 2023-07-02T14:04:05.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2136e-05 2023-07-02T14:04:05.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:05.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964666+0000", 2023-07-02T14:04:05.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2169999999999999e-06 2023-07-02T14:04:05.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:05.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964672+0000", 2023-07-02T14:04:05.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00047561800000000003 2023-07-02T14:04:05.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:05.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965147+0000", 2023-07-02T14:04:05.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5891e-05 2023-07-02T14:04:05.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:05.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965173+0000", 2023-07-02T14:04:05.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013770973000000001 2023-07-02T14:04:05.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:05.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978944+0000", 2023-07-02T14:04:05.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0256999999999998e-05 2023-07-02T14:04:05.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:05.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978985+0000", 2023-07-02T14:04:05.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019172812000000001 2023-07-02T14:04:05.769 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.769 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.770 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.770 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:04:05.770 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:05.770 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.3191919050000003, 2023-07-02T14:04:05.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:04:05.770 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:05.771 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:05.771 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:05.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:05.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:05.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:05.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:05.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:05.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:05.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:05.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:04:05.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:04:05.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:05.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:04:05.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:04:05.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:05.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:04:05.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:04:05.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:05.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:04:05.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:04:05.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:05.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:04:05.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:04:05.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:05.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:04:05.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:04:05.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:05.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:04:05.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:04:05.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:05.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:04:05.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:04:05.777 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.777 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.777 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.778 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:05.778 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:05.778 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.952842221, 2023-07-02T14:04:05.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:04:05.778 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:05.778 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:05.778 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:05.779 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:05.779 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:05.779 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:04:05.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.779 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:05.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:05.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:05.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:05.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:05.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:05.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:05.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:05.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:04:05.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:04:05.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:05.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:04:05.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:04:05.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:05.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:04:05.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:04:05.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:05.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:04:05.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:04:05.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:05.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:04:05.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:04:05.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:05.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:04:05.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:04:05.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:05.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:04:05.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:04:05.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:05.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:04:05.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:04:05.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:05.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:04:05.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:04:05.786 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.786 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.786 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.787 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7706 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 0~53248] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:05.787 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:05.787 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.9489154740000001, 2023-07-02T14:04:05.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019507096000000002, 2023-07-02T14:04:05.787 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:05.787 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:05.787 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:05.788 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:05.788 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:05.788 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7706 2023-07-02T14:04:05.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.788 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:05.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:05.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:05.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:05.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:05.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:05.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:05.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:05.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732254+0000", 2023-07-02T14:04:05.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4764999999999997e-05 2023-07-02T14:04:05.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:05.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732289+0000", 2023-07-02T14:04:05.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0166999999999999e-05 2023-07-02T14:04:05.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:05.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732299+0000", 2023-07-02T14:04:05.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5310000000000001e-06 2023-07-02T14:04:05.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:05.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732305+0000", 2023-07-02T14:04:05.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060619599999999999 2023-07-02T14:04:05.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:05.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738367+0000", 2023-07-02T14:04:05.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011864 2023-07-02T14:04:05.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:05.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738486+0000", 2023-07-02T14:04:05.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013098291 2023-07-02T14:04:05.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:05.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751584+0000", 2023-07-02T14:04:05.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7800000000000002e-05 2023-07-02T14:04:05.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:05.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751642+0000", 2023-07-02T14:04:05.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000121198 2023-07-02T14:04:05.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:05.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751763+0000", 2023-07-02T14:04:05.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019507096000000002 2023-07-02T14:04:05.795 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.795 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.795 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.796 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7707 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 3145728~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:05.796 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:05.796 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.9488587540000002, 2023-07-02T14:04:05.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999, 2023-07-02T14:04:05.796 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:05.796 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:05.796 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:05.797 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:05.797 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:05.797 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7707 2023-07-02T14:04:05.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.797 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:05.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:05.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:05.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:05.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:05.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:05.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:05.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:05.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:05.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5017e-05 2023-07-02T14:04:05.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:05.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732347+0000", 2023-07-02T14:04:05.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5821e-05 2023-07-02T14:04:05.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:05.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732363+0000", 2023-07-02T14:04:05.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1270000000000002e-06 2023-07-02T14:04:05.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:05.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732368+0000", 2023-07-02T14:04:05.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008444666 2023-07-02T14:04:05.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:05.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740812+0000", 2023-07-02T14:04:05.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011266599999999999 2023-07-02T14:04:05.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:05.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740925+0000", 2023-07-02T14:04:05.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010848808999999999 2023-07-02T14:04:05.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:05.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751774+0000", 2023-07-02T14:04:05.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8421999999999997e-05 2023-07-02T14:04:05.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:05.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751832+0000", 2023-07-02T14:04:05.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107453 2023-07-02T14:04:05.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:05.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751940+0000", 2023-07-02T14:04:05.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999 2023-07-02T14:04:05.804 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.804 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.804 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.805 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:05.805 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:05.805 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.9487948770000001, 2023-07-02T14:04:05.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:04:05.805 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:05.805 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:05.805 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:05.806 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:05.806 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:05.806 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:04:05.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.806 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:05.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:05.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:05.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:05.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:05.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:05.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:05.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:05.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:05.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:04:05.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:05.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:04:05.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:04:05.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:05.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:04:05.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:04:05.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:05.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:04:05.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:04:05.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:05.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:04:05.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:04:05.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:05.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:04:05.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:04:05.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:05.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:04:05.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:04:05.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:05.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:04:05.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:04:05.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:05.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:04:05.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:04:05.813 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.813 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.813 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.814 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:05.814 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:05.814 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.9442310520000001, 2023-07-02T14:04:05.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:04:05.814 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:05.814 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:05.814 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:05.815 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:05.815 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:05.815 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:04:05.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.815 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:05.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:05.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:05.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:05.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:05.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:05.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:05.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:05.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:04:05.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:04:05.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:05.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:04:05.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:04:05.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:05.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:04:05.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:04:05.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:05.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:04:05.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:04:05.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:05.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:04:05.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:04:05.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:05.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:04:05.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:04:05.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:05.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:04:05.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:04:05.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:05.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:04:05.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:04:05.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:05.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:04:05.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:04:05.822 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.822 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.822 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.823 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:05.823 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:05.823 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.9441811859999998, 2023-07-02T14:04:05.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:04:05.823 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:05.823 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:05.823 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:05.824 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:05.824 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:05.824 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:04:05.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.824 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:05.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:05.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:05.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:05.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:05.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:05.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:05.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:05.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:04:05.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:04:05.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:05.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:04:05.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:04:05.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:05.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:04:05.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:04:05.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:05.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:04:05.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:04:05.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:05.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:04:05.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:04:05.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:05.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:04:05.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:04:05.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:05.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:04:05.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:04:05.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:05.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:04:05.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:04:05.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:05.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:04:05.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:04:05.831 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.831 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.831 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.832 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:05.832 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:05.832 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.9431421929999999, 2023-07-02T14:04:05.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:04:05.832 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:05.832 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:05.832 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:05.833 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:05.833 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:05.833 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:04:05.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.833 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:05.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:05.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:05.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:05.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:05.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:05.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:05.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:05.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:04:05.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:04:05.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:05.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:04:05.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:04:05.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:05.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:04:05.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:04:05.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:05.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:04:05.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:04:05.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:05.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:04:05.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:04:05.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:05.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:04:05.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:04:05.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:05.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:04:05.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:04:05.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:05.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:04:05.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:04:05.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:05.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:04:05.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:04:05.840 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.840 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.840 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.841 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7993 2.7 2:fe72db53:::rbd_data.125de1449605.0000000000000072:head [write 65536~3276800] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:05.841 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:05.841 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.7561841450000002, 2023-07-02T14:04:05.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919, 2023-07-02T14:04:05.841 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:05.841 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:05.841 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:05.842 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:05.842 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:05.842 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7993 2023-07-02T14:04:05.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.842 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:05.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:05.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:05.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:05.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:05.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:05.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:05.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:05.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924986+0000", 2023-07-02T14:04:05.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049542070000000004 2023-07-02T14:04:05.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:05.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929940+0000", 2023-07-02T14:04:05.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9740000000000001e-05 2023-07-02T14:04:05.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:05.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929960+0000", 2023-07-02T14:04:05.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.311e-05 2023-07-02T14:04:05.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:05.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929973+0000", 2023-07-02T14:04:05.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9629000000000002e-05 2023-07-02T14:04:05.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:05.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930023+0000", 2023-07-02T14:04:05.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012751799999999999 2023-07-02T14:04:05.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:05.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930150+0000", 2023-07-02T14:04:05.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016449042000000001 2023-07-02T14:04:05.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:05.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946599+0000", 2023-07-02T14:04:05.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3241e-05 2023-07-02T14:04:05.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:05.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946663+0000", 2023-07-02T14:04:05.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000102296 2023-07-02T14:04:05.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:05.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:05.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:05.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946765+0000", 2023-07-02T14:04:05.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919 2023-07-02T14:04:05.849 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.849 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.849 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.849 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:05.850 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:05.850 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:05.851 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:05.850+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:05.851 INFO:tasks.workunit.client.0.smithi163.stderr:+ sync b3fbb028-e267-4d25-a750-93ba18a2316a-mnt 2023-07-02T14:04:05.851 INFO:tasks.workunit.client.0.smithi163.stderr:+ rm -rf b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4 b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4.tar.gz 2023-07-02T14:04:05.852 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:04:05.852 INFO:teuthology.orchestra.run.smithi022.stderr:Nothing to do for non-filestore backend 2023-07-02T14:04:05.951 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:05.950+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:05.976 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:05.977 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:05.977 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:04:05.977 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:05.986 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:04:06.070 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:06.069+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:06.150 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:06.151 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:06.151 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:04:06.151 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:04:06.151 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:06.152 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:06.151+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:06.161 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:06.230 DEBUG:teuthology.orchestra.run.smithi022:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op apply-layout-settings --pool rbd 2023-07-02T14:04:06.271 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:06.270+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:06.331 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:06.331 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:04:06.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:04:06.331 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:06.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.331 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:06.332 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:06.332 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.038132235999996, 2023-07-02T14:04:06.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:04:06.332 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:06.332 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:06.332 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:06.332 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:06.333 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:06.333 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:04:06.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.333 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:06.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:06.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:06.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:06.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:06.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:06.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:06.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:06.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:04:06.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:04:06.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:06.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:04:06.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:04:06.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:06.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:04:06.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:04:06.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:06.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:04:06.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:04:06.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:06.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:04:06.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:04:06.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:06.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:04:06.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:04:06.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:06.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:04:06.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:04:06.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:06.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:04:06.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:04:06.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:06.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:04:06.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:04:06.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:06.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:04:06.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:04:06.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:06.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:04:06.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:04:06.341 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.341 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.342 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.342 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:06.342 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:06.342 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.037897212999994, 2023-07-02T14:04:06.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:04:06.343 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:06.343 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:06.343 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:06.343 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:06.343 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:06.343 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:04:06.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.343 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:06.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:06.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:06.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:06.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:06.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:06.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:06.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:06.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:04:06.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:04:06.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:06.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:04:06.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:04:06.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:06.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:04:06.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:04:06.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:06.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:04:06.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:04:06.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:06.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:04:06.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:04:06.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:06.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:04:06.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:04:06.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:06.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:04:06.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:04:06.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:06.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:04:06.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:04:06.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:06.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:04:06.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:04:06.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:06.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:04:06.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:04:06.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:06.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:04:06.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:04:06.352 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.352 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.352 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.352 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:06.352 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:06.353 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.037698078999995, 2023-07-02T14:04:06.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:04:06.353 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:06.353 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:06.353 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:06.353 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:06.353 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:06.353 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:04:06.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.354 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:06.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:06.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:06.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:06.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:06.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:06.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:06.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:06.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:04:06.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:04:06.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:06.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:04:06.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:04:06.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:06.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:04:06.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:04:06.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:06.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:04:06.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:04:06.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:06.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:04:06.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:04:06.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:06.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:04:06.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:04:06.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:06.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:04:06.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:04:06.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:06.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:04:06.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:04:06.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:06.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:04:06.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:04:06.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:06.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:04:06.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:04:06.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:06.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:04:06.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:04:06.362 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.362 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.362 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.363 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:06.363 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:06.363 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.037332954999997, 2023-07-02T14:04:06.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:04:06.363 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:06.363 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:06.364 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:06.364 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:06.364 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:06.364 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:04:06.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.364 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:06.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:06.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:06.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:06.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:06.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:06.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:06.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:06.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:04:06.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:04:06.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:06.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:04:06.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:04:06.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:06.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:04:06.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:04:06.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:06.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:04:06.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:04:06.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:06.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:04:06.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:04:06.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:06.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:04:06.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:04:06.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:06.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:04:06.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:04:06.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:06.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:04:06.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:04:06.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:06.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:04:06.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:04:06.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:06.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:04:06.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:04:06.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:06.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:04:06.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:04:06.372 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.373 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.373 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.373 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:06.373 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:06.373 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.037251893999994, 2023-07-02T14:04:06.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:04:06.374 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:06.374 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:06.374 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:06.374 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:06.374 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:06.374 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:04:06.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.374 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:06.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:06.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:06.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:06.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:06.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:06.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:06.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:06.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:04:06.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:04:06.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:06.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:04:06.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:04:06.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:06.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:04:06.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:04:06.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:06.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:04:06.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:04:06.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:06.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:04:06.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:04:06.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:06.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:04:06.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:04:06.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:06.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:04:06.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:04:06.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:06.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:04:06.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:04:06.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:06.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:04:06.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:04:06.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:06.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:04:06.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:04:06.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:06.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:04:06.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:04:06.383 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.383 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.383 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.383 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:06.384 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:06.384 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.037160223000001, 2023-07-02T14:04:06.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:04:06.384 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:06.384 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:06.384 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:06.384 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:06.384 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:06.385 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:04:06.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.385 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:06.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:06.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:06.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:06.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:06.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:06.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:06.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:06.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:04:06.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:04:06.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:06.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:04:06.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:04:06.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:06.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:04:06.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:04:06.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:06.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:04:06.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:04:06.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:06.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:04:06.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:04:06.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:06.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:04:06.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:04:06.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:06.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:04:06.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:04:06.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:06.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:04:06.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:04:06.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:06.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:04:06.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:04:06.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:06.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:04:06.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:04:06.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:06.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:04:06.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:04:06.393 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.393 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.393 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.394 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:06.394 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:06.394 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.036820781000003, 2023-07-02T14:04:06.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:04:06.394 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:06.394 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:06.395 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:06.395 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:06.395 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:06.395 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:04:06.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.395 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:06.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:06.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:06.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:06.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:06.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:06.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:06.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:06.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:04:06.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:04:06.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:06.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:04:06.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:04:06.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:06.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:04:06.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:04:06.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:06.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:04:06.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:04:06.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:06.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:04:06.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:04:06.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:06.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:04:06.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:04:06.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:06.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:04:06.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:04:06.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:06.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:04:06.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:04:06.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:06.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:04:06.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:04:06.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:06.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:04:06.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:04:06.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:06.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:04:06.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:04:06.403 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.404 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.404 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.404 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:06.404 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:06.404 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.036634727999996, 2023-07-02T14:04:06.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:04:06.405 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:06.405 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:06.405 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:06.405 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:06.405 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:06.405 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:04:06.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.406 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:06.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:06.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:06.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:06.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:06.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:06.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:06.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:06.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:06.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:04:06.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:06.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:04:06.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:04:06.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:06.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:04:06.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:04:06.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:06.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:04:06.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:04:06.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:06.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:04:06.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:04:06.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:06.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:04:06.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:04:06.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:06.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:04:06.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:04:06.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:06.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:04:06.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:04:06.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:06.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:04:06.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:04:06.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:06.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:04:06.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:04:06.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:06.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:04:06.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:04:06.414 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.414 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.414 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.414 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:06.415 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:06.415 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.036476926000006, 2023-07-02T14:04:06.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:04:06.415 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:06.415 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:06.415 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:06.415 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:06.415 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:06.416 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:04:06.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.416 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:06.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:06.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:06.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:06.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:06.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:06.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:06.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:06.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:04:06.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:04:06.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:06.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:04:06.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:04:06.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:06.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:04:06.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:04:06.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:06.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:04:06.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:04:06.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:06.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:04:06.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:04:06.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:06.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:04:06.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:04:06.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:06.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:04:06.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:04:06.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:06.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:04:06.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:04:06.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:06.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:04:06.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:04:06.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:06.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:04:06.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:04:06.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:06.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:04:06.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:04:06.424 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.424 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.424 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.425 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:06.425 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:06.425 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.036285144000004, 2023-07-02T14:04:06.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:04:06.425 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:06.425 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:06.425 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:06.426 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:06.426 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:06.426 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:04:06.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.426 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:06.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:06.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:06.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:06.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:06.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:06.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:06.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:06.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:04:06.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:04:06.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:06.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:04:06.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:04:06.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:06.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:04:06.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:04:06.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:06.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:04:06.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:04:06.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:06.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:04:06.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:04:06.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:06.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:04:06.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:04:06.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:06.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:04:06.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:04:06.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:06.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:04:06.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:04:06.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:06.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:04:06.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:04:06.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:06.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:04:06.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:04:06.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:06.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:04:06.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:04:06.434 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.434 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.435 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.435 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:06.435 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:06.435 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.036109483999994, 2023-07-02T14:04:06.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:04:06.435 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:06.436 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:06.436 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:06.436 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:06.436 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:06.436 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:04:06.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.436 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:06.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:06.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:06.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:06.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:06.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:06.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:06.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:06.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:06.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:04:06.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:06.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:04:06.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:04:06.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:06.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:04:06.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:04:06.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:06.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:04:06.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:04:06.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:06.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:04:06.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:04:06.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:06.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:04:06.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:04:06.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:06.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:04:06.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:04:06.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:06.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:04:06.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:04:06.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:06.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:04:06.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:04:06.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:06.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:04:06.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:04:06.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:06.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:04:06.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:04:06.445 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.445 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.445 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.445 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:06.445 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:06.446 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.036015296000002, 2023-07-02T14:04:06.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:04:06.446 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:06.446 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:06.446 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:06.446 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:06.446 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:06.446 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:04:06.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.447 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:06.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:06.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:06.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:06.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:06.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:06.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:06.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:06.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:04:06.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:04:06.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:06.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:04:06.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:04:06.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:06.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:04:06.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:04:06.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:06.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:04:06.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:04:06.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:06.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:04:06.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:04:06.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:06.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:04:06.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:04:06.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:06.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:04:06.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:04:06.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:06.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:04:06.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:04:06.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:06.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:04:06.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:04:06.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:06.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:04:06.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:04:06.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:06.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:04:06.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:04:06.455 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.455 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.455 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.456 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:06.456 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:06.456 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.035811910000007, 2023-07-02T14:04:06.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:04:06.456 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:06.456 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:06.456 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:06.456 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:06.457 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:06.457 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:04:06.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.457 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:06.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:06.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:06.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:06.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:06.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:06.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:06.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:06.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:04:06.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:04:06.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:06.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:04:06.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:04:06.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:06.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:04:06.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:04:06.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:06.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:04:06.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:04:06.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:06.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:04:06.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:04:06.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:06.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:04:06.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:04:06.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:06.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:04:06.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:04:06.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:06.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:04:06.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:04:06.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:06.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:04:06.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:04:06.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:06.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:04:06.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:04:06.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:06.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:04:06.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:04:06.465 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.465 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.466 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.466 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:06.466 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:06.466 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.035661254000004, 2023-07-02T14:04:06.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:04:06.466 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:06.467 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:06.467 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:06.467 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:06.467 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:06.467 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:04:06.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.467 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:06.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:06.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:06.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:06.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:06.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:06.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:06.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:06.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:04:06.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:04:06.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:06.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:04:06.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:04:06.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:06.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:04:06.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:04:06.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:06.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:04:06.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:04:06.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:06.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:04:06.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:04:06.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:06.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:04:06.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:04:06.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:06.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:04:06.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:04:06.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:06.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:04:06.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:04:06.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:06.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:04:06.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:04:06.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:06.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:04:06.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:04:06.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:06.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:04:06.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:04:06.476 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.476 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.476 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.476 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:06.476 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:06.476 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.035588551000004, 2023-07-02T14:04:06.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:04:06.477 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:06.477 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:06.477 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:06.477 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:06.477 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:06.477 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:04:06.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.478 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:06.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:06.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:06.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:06.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:06.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:06.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:06.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:06.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:06.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:04:06.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:06.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:04:06.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:04:06.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:06.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:04:06.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:04:06.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:06.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:04:06.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:04:06.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:06.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:04:06.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:04:06.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:06.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:04:06.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:04:06.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:06.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:04:06.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:04:06.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:06.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:04:06.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:04:06.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:06.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:04:06.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:04:06.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:06.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:04:06.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:04:06.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:06.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:04:06.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:04:06.486 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.486 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.486 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.486 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:06.487 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:06.487 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.035471560999994, 2023-07-02T14:04:06.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:04:06.487 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:06.487 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:06.487 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:06.487 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:06.487 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:06.488 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:04:06.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.488 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:06.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:06.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:06.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:06.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:06.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:06.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:06.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:06.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:06.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:04:06.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:06.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:04:06.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:04:06.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:06.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:04:06.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:04:06.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:06.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:04:06.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:04:06.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:06.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:04:06.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:04:06.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:06.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:04:06.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:04:06.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:06.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:04:06.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:04:06.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:06.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:04:06.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:04:06.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:06.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:04:06.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:04:06.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:06.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:04:06.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:04:06.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:06.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:04:06.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:04:06.496 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.496 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.496 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.497 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:06.497 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:06.497 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.035322109999996, 2023-07-02T14:04:06.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:04:06.497 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:06.497 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:06.497 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:06.498 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:06.498 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:06.498 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:04:06.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.498 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:06.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:06.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:06.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:06.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:06.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:06.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:06.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:06.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:04:06.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:04:06.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:06.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:04:06.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:04:06.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:06.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:04:06.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:04:06.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:06.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:04:06.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:04:06.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:06.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:04:06.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:04:06.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:06.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:04:06.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:04:06.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:06.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:04:06.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:04:06.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:06.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:04:06.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:04:06.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:06.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:04:06.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:04:06.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:06.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:04:06.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:04:06.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:06.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:04:06.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:04:06.506 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.506 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.507 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.507 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:06.507 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:06.507 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.013846137000002, 2023-07-02T14:04:06.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:04:06.508 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:06.508 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:06.508 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:06.508 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:06.508 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:06.508 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:04:06.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.508 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:06.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:06.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:06.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:06.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:06.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:06.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:06.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:06.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:04:06.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:04:06.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:06.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:04:06.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:04:06.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:06.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:04:06.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:04:06.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:06.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:04:06.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:04:06.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:06.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:04:06.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:04:06.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:06.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:04:06.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:04:06.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:06.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:04:06.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:04:06.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:06.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:04:06.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:04:06.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:06.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:04:06.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:04:06.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:06.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:04:06.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:04:06.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:06.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:04:06.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:04:06.517 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.517 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.517 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.517 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:06.517 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:06.517 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.013713601000006, 2023-07-02T14:04:06.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:04:06.518 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:06.518 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:06.518 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:06.518 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:06.518 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:06.518 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:04:06.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.519 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:06.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:06.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:06.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:06.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:06.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:06.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:06.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:06.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:04:06.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:04:06.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:06.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:04:06.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:04:06.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:06.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:04:06.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:04:06.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:06.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:04:06.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:04:06.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:06.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:04:06.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:04:06.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:06.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:04:06.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:04:06.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:06.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:04:06.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:04:06.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:06.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:04:06.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:04:06.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:06.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:04:06.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:04:06.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:06.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:04:06.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:04:06.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:06.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:04:06.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:04:06.527 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.527 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.527 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.528 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:06.528 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:06.528 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 86.952376939999994, 2023-07-02T14:04:06.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:04:06.528 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:06.528 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:06.528 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:06.528 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:06.529 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:06.529 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:04:06.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.529 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:06.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:06.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:06.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:06.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:06.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:06.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:06.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:06.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:04:06.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:04:06.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:06.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:04:06.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:04:06.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:06.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:04:06.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:04:06.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:06.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:04:06.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:04:06.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:06.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:04:06.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:04:06.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:06.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:04:06.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:04:06.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:06.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:04:06.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:04:06.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:06.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:04:06.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:04:06.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:06.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:04:06.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:04:06.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:06.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:04:06.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:04:06.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:06.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:06.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:06.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:04:06.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:04:06.537 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.537 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.537 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.538 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:06.538 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:06.538 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:06.539 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:06.539+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:06.539 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:06.639 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:06.639+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:06.703 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:06.703 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:06.703 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:06.703 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:06.713 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:04:06.756 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:06.755+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:06.765 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i++ )) 2023-07-02T14:04:06.766 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i < 10 )) 2023-07-02T14:04:06.766 INFO:tasks.workunit.client.0.smithi163.stderr:+ pv -L 10M linux-5.4.tar.gz 2023-07-02T14:04:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:06.837 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:06.837 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:06.837 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:06.841 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:06.840+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:06.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:04:06.942 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:06.940+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:07.013 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:07.013 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:07.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2620 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 196608~786432] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.686443322000002, 2023-07-02T14:04:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966, 2023-07-02T14:04:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:07.014 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2620 2023-07-02T14:04:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:07.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324571+0000", 2023-07-02T14:04:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016686649999999999 2023-07-02T14:04:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326240+0000", 2023-07-02T14:04:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1844e-05 2023-07-02T14:04:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:07.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326252+0000", 2023-07-02T14:04:07.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7760000000000003e-06 2023-07-02T14:04:07.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:07.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326258+0000", 2023-07-02T14:04:07.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0634000000000003e-05 2023-07-02T14:04:07.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326318+0000", 2023-07-02T14:04:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9629999999999993e-06 2023-07-02T14:04:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326328+0000", 2023-07-02T14:04:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.010877045 2023-07-02T14:04:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337205+0000", 2023-07-02T14:04:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022401300000000001 2023-07-02T14:04:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337429+0000", 2023-07-02T14:04:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000139822 2023-07-02T14:04:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337569+0000", 2023-07-02T14:04:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032942119999999999 2023-07-02T14:04:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340863+0000", 2023-07-02T14:04:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021302560000000001 2023-07-02T14:04:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342994+0000", 2023-07-02T14:04:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5634000000000002e-05 2023-07-02T14:04:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343059+0000", 2023-07-02T14:04:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143721 2023-07-02T14:04:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343203+0000", 2023-07-02T14:04:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966 2023-07-02T14:04:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2622 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 983040~36864] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.684750811000001, 2023-07-02T14:04:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664, 2023-07-02T14:04:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2622 2023-07-02T14:04:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326265+0000", 2023-07-02T14:04:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1282000000000001e-05 2023-07-02T14:04:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326286+0000", 2023-07-02T14:04:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9480000000000002e-06 2023-07-02T14:04:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326293+0000", 2023-07-02T14:04:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3610000000000003e-06 2023-07-02T14:04:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326297+0000", 2023-07-02T14:04:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7707999999999994e-05 2023-07-02T14:04:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326385+0000", 2023-07-02T14:04:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8070000000000006e-06 2023-07-02T14:04:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326394+0000", 2023-07-02T14:04:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.014535878 2023-07-02T14:04:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340930+0000", 2023-07-02T14:04:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000154551 2023-07-02T14:04:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341084+0000", 2023-07-02T14:04:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00032375900000000002 2023-07-02T14:04:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341408+0000", 2023-07-02T14:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012631066 2023-07-02T14:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354039+0000", 2023-07-02T14:04:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5741000000000001e-05 2023-07-02T14:04:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354055+0000", 2023-07-02T14:04:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8794000000000006e-05 2023-07-02T14:04:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354124+0000", 2023-07-02T14:04:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014667700000000001 2023-07-02T14:04:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354270+0000", 2023-07-02T14:04:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664 2023-07-02T14:04:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2727 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~12288] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.684715036, 2023-07-02T14:04:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999, 2023-07-02T14:04:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2727 2023-07-02T14:04:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3834e-05 2023-07-02T14:04:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326315+0000", 2023-07-02T14:04:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3029999999999996e-06 2023-07-02T14:04:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326321+0000", 2023-07-02T14:04:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7590000000000002e-06 2023-07-02T14:04:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326324+0000", 2023-07-02T14:04:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112872 2023-07-02T14:04:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326437+0000", 2023-07-02T14:04:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2290000000000002e-06 2023-07-02T14:04:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326442+0000", 2023-07-02T14:04:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0155896879999999 2023-07-02T14:04:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342031+0000", 2023-07-02T14:04:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016496100000000001 2023-07-02T14:04:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342196+0000", 2023-07-02T14:04:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021998699999999999 2023-07-02T14:04:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342416+0000", 2023-07-02T14:04:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011869744999999999 2023-07-02T14:04:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354286+0000", 2023-07-02T14:04:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0361e-05 2023-07-02T14:04:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354347+0000", 2023-07-02T14:04:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3312000000000005e-05 2023-07-02T14:04:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354430+0000", 2023-07-02T14:04:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126444 2023-07-02T14:04:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354556+0000", 2023-07-02T14:04:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999 2023-07-02T14:04:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2730 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [set-alloc-hint object_size 4194304 write_size 4194304,write 454656~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.684690027000002, 2023-07-02T14:04:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999, 2023-07-02T14:04:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2730 2023-07-02T14:04:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6049999999999997e-06 2023-07-02T14:04:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326332+0000", 2023-07-02T14:04:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3000000000000002e-06 2023-07-02T14:04:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326335+0000", 2023-07-02T14:04:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.719e-05 2023-07-02T14:04:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326352+0000", 2023-07-02T14:04:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011583599999999999 2023-07-02T14:04:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326468+0000", 2023-07-02T14:04:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.831e-06 2023-07-02T14:04:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326473+0000", 2023-07-02T14:04:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.016769915 2023-07-02T14:04:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343243+0000", 2023-07-02T14:04:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197352 2023-07-02T14:04:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343440+0000", 2023-07-02T14:04:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197581 2023-07-02T14:04:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343638+0000", 2023-07-02T14:04:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010980139 2023-07-02T14:04:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354618+0000", 2023-07-02T14:04:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056779530000000003 2023-07-02T14:04:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360296+0000", 2023-07-02T14:04:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101109 2023-07-02T14:04:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360397+0000", 2023-07-02T14:04:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000160273 2023-07-02T14:04:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360557+0000", 2023-07-02T14:04:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999 2023-07-02T14:04:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2752 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 1019904~3174400] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.684643803, 2023-07-02T14:04:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001, 2023-07-02T14:04:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2752 2023-07-02T14:04:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:07.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:07.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326372+0000", 2023-07-02T14:04:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052050380000000004 2023-07-02T14:04:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331577+0000", 2023-07-02T14:04:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6832000000000001e-05 2023-07-02T14:04:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331594+0000", 2023-07-02T14:04:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8220000000000007e-06 2023-07-02T14:04:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331602+0000", 2023-07-02T14:04:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9895999999999999e-05 2023-07-02T14:04:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331642+0000", 2023-07-02T14:04:07.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:04:07.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:07.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331649+0000", 2023-07-02T14:04:07.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0139141709999999 2023-07-02T14:04:07.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345563+0000", 2023-07-02T14:04:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113817 2023-07-02T14:04:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345677+0000", 2023-07-02T14:04:07.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018803400000000001 2023-07-02T14:04:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345865+0000", 2023-07-02T14:04:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087950359999999991 2023-07-02T14:04:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354660+0000", 2023-07-02T14:04:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059116669999999998 2023-07-02T14:04:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360572+0000", 2023-07-02T14:04:07.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0992000000000002e-05 2023-07-02T14:04:07.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:07.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360633+0000", 2023-07-02T14:04:07.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013601600000000001 2023-07-02T14:04:07.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:07.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360769+0000", 2023-07-02T14:04:07.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001 2023-07-02T14:04:07.074 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.074 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.074 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.075 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2759 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 0~454656] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:07.075 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:07.075 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.679404005999999, 2023-07-02T14:04:07.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999, 2023-07-02T14:04:07.075 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:07.075 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:07.076 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:07.076 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:07.076 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:07.076 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2759 2023-07-02T14:04:07.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.076 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:07.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:07.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:07.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:07.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:07.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:07.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:07.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:07.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331611+0000", 2023-07-02T14:04:07.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053795999999999998 2023-07-02T14:04:07.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:07.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332149+0000", 2023-07-02T14:04:07.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4970000000000007e-06 2023-07-02T14:04:07.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:07.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332158+0000", 2023-07-02T14:04:07.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:04:07.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:07.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332162+0000", 2023-07-02T14:04:07.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3526000000000001e-05 2023-07-02T14:04:07.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332185+0000", 2023-07-02T14:04:07.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:04:07.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:07.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332189+0000", 2023-07-02T14:04:07.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0186295910000001 2023-07-02T14:04:07.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350819+0000", 2023-07-02T14:04:07.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000111123 2023-07-02T14:04:07.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:07.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350930+0000", 2023-07-02T14:04:07.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019575300000000001 2023-07-02T14:04:07.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:07.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.351126+0000", 2023-07-02T14:04:07.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011503298 2023-07-02T14:04:07.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:07.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362629+0000", 2023-07-02T14:04:07.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085443989999999994 2023-07-02T14:04:07.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:07.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371173+0000", 2023-07-02T14:04:07.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3547999999999995e-05 2023-07-02T14:04:07.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:07.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371237+0000", 2023-07-02T14:04:07.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790300000000001 2023-07-02T14:04:07.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:07.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371375+0000", 2023-07-02T14:04:07.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999 2023-07-02T14:04:07.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.086 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.087 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2760 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 458752~131072] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:07.087 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:07.087 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.678847820999998, 2023-07-02T14:04:07.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001, 2023-07-02T14:04:07.087 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:07.087 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:07.087 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:07.088 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:07.088 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:07.088 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2760 2023-07-02T14:04:07.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.088 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:07.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:07.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:07.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:07.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:07.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:07.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:07.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:07.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332168+0000", 2023-07-02T14:04:07.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013281100000000001 2023-07-02T14:04:07.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:07.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332301+0000", 2023-07-02T14:04:07.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7389999999999999e-06 2023-07-02T14:04:07.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:07.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332308+0000", 2023-07-02T14:04:07.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0639999999999998e-06 2023-07-02T14:04:07.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:07.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332312+0000", 2023-07-02T14:04:07.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1735000000000001e-05 2023-07-02T14:04:07.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332333+0000", 2023-07-02T14:04:07.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8342000000000002e-05 2023-07-02T14:04:07.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:07.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332352+0000", 2023-07-02T14:04:07.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02234782 2023-07-02T14:04:07.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354699+0000", 2023-07-02T14:04:07.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110371 2023-07-02T14:04:07.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:07.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354810+0000", 2023-07-02T14:04:07.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022174900000000001 2023-07-02T14:04:07.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:07.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355032+0000", 2023-07-02T14:04:07.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076053329999999997 2023-07-02T14:04:07.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:07.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362637+0000", 2023-07-02T14:04:07.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087913609999999993 2023-07-02T14:04:07.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:07.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371428+0000", 2023-07-02T14:04:07.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8532999999999998e-05 2023-07-02T14:04:07.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:07.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371477+0000", 2023-07-02T14:04:07.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010990800000000001 2023-07-02T14:04:07.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:07.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371587+0000", 2023-07-02T14:04:07.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001 2023-07-02T14:04:07.098 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.098 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.098 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.099 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2766 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 589824~45056] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:07.099 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:07.099 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.678699195999997, 2023-07-02T14:04:07.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188, 2023-07-02T14:04:07.099 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:07.099 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:07.099 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:07.100 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:07.100 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:07.100 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2766 2023-07-02T14:04:07.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.100 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:07.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:07.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:07.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:07.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:07.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:07.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:07.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:07.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:07.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2247999999999998e-05 2023-07-02T14:04:07.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:07.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332369+0000", 2023-07-02T14:04:07.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4189999999999996e-06 2023-07-02T14:04:07.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:07.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332376+0000", 2023-07-02T14:04:07.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1049999999999999e-06 2023-07-02T14:04:07.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:07.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332379+0000", 2023-07-02T14:04:07.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6894000000000003e-05 2023-07-02T14:04:07.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332416+0000", 2023-07-02T14:04:07.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3970000000000002e-06 2023-07-02T14:04:07.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:07.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332422+0000", 2023-07-02T14:04:07.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0231382280000001 2023-07-02T14:04:07.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355560+0000", 2023-07-02T14:04:07.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9079000000000005e-05 2023-07-02T14:04:07.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:07.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355659+0000", 2023-07-02T14:04:07.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018005000000000001 2023-07-02T14:04:07.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:07.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355839+0000", 2023-07-02T14:04:07.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068044439999999998 2023-07-02T14:04:07.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:07.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362644+0000", 2023-07-02T14:04:07.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090155589999999994 2023-07-02T14:04:07.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:07.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371659+0000", 2023-07-02T14:04:07.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1297999999999994e-05 2023-07-02T14:04:07.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:07.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371730+0000", 2023-07-02T14:04:07.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013100699999999999 2023-07-02T14:04:07.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:07.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371861+0000", 2023-07-02T14:04:07.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188 2023-07-02T14:04:07.110 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.110 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.110 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.110 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2784 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 0~393216] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:07.111 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:07.111 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.678630744000003, 2023-07-02T14:04:07.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999, 2023-07-02T14:04:07.111 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:07.111 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:07.111 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:07.111 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:07.111 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:07.112 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2784 2023-07-02T14:04:07.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.112 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:07.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:07.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:07.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:07.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:07.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:07.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:07.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:07.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332385+0000", 2023-07-02T14:04:07.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040252400000000002 2023-07-02T14:04:07.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:07.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332787+0000", 2023-07-02T14:04:07.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0208e-05 2023-07-02T14:04:07.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:07.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332798+0000", 2023-07-02T14:04:07.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2023-07-02T14:04:07.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:07.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332801+0000", 2023-07-02T14:04:07.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4838000000000001e-05 2023-07-02T14:04:07.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332836+0000", 2023-07-02T14:04:07.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5319999999999998e-06 2023-07-02T14:04:07.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:07.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332839+0000", 2023-07-02T14:04:07.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0234290109999999 2023-07-02T14:04:07.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356268+0000", 2023-07-02T14:04:07.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011165700000000001 2023-07-02T14:04:07.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:07.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356380+0000", 2023-07-02T14:04:07.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022504799999999999 2023-07-02T14:04:07.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:07.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356605+0000", 2023-07-02T14:04:07.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015276201 2023-07-02T14:04:07.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:07.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371881+0000", 2023-07-02T14:04:07.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5496999999999999e-05 2023-07-02T14:04:07.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:07.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371957+0000", 2023-07-02T14:04:07.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4570000000000002e-05 2023-07-02T14:04:07.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:07.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372001+0000", 2023-07-02T14:04:07.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3880000000000003e-05 2023-07-02T14:04:07.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:07.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372085+0000", 2023-07-02T14:04:07.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999 2023-07-02T14:04:07.122 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.122 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.122 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.122 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2785 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 405504~446464] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:07.122 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:07.123 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.678209293000002, 2023-07-02T14:04:07.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999, 2023-07-02T14:04:07.123 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:07.123 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:07.123 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:07.123 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:07.123 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:07.123 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2785 2023-07-02T14:04:07.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.124 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:07.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:07.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:07.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:07.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:07.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:07.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:07.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:07.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:07.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059527500000000004 2023-07-02T14:04:07.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:07.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333402+0000", 2023-07-02T14:04:07.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7059999999999993e-06 2023-07-02T14:04:07.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:07.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333411+0000", 2023-07-02T14:04:07.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0450000000000001e-06 2023-07-02T14:04:07.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:07.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333414+0000", 2023-07-02T14:04:07.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5790999999999997e-05 2023-07-02T14:04:07.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:04:07.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3320000000000002e-06 2023-07-02T14:04:07.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:07.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333454+0000", 2023-07-02T14:04:07.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.025310269 2023-07-02T14:04:07.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358764+0000", 2023-07-02T14:04:07.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012414899999999999 2023-07-02T14:04:07.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:07.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358888+0000", 2023-07-02T14:04:07.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000217028 2023-07-02T14:04:07.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:07.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.359105+0000", 2023-07-02T14:04:07.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012788212 2023-07-02T14:04:07.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:07.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371893+0000", 2023-07-02T14:04:07.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012201569999999999 2023-07-02T14:04:07.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:07.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373114+0000", 2023-07-02T14:04:07.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4512999999999995e-05 2023-07-02T14:04:07.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:07.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373198+0000", 2023-07-02T14:04:07.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132373 2023-07-02T14:04:07.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:07.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373330+0000", 2023-07-02T14:04:07.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999 2023-07-02T14:04:07.134 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.134 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.134 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.134 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2786 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1310720~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:07.135 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:07.135 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.677596600000001, 2023-07-02T14:04:07.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001, 2023-07-02T14:04:07.135 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:07.135 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:07.135 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:07.135 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:07.135 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:07.136 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2786 2023-07-02T14:04:07.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.136 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:07.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:07.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:07.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:07.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:07.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:07.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:07.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:07.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333419+0000", 2023-07-02T14:04:07.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2009999999999999e-06 2023-07-02T14:04:07.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:07.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333427+0000", 2023-07-02T14:04:07.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0210000000000001e-06 2023-07-02T14:04:07.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:07.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333431+0000", 2023-07-02T14:04:07.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3769999999999999e-06 2023-07-02T14:04:07.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:07.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333434+0000", 2023-07-02T14:04:07.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1161e-05 2023-07-02T14:04:07.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333485+0000", 2023-07-02T14:04:07.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1549000000000001e-05 2023-07-02T14:04:07.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:07.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333496+0000", 2023-07-02T14:04:07.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0273116339999999 2023-07-02T14:04:07.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360808+0000", 2023-07-02T14:04:07.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000115128 2023-07-02T14:04:07.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:07.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360923+0000", 2023-07-02T14:04:07.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018367700000000001 2023-07-02T14:04:07.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:07.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361107+0000", 2023-07-02T14:04:07.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010796923999999999 2023-07-02T14:04:07.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:07.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371904+0000", 2023-07-02T14:04:07.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014983520000000001 2023-07-02T14:04:07.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:07.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373402+0000", 2023-07-02T14:04:07.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0520999999999999e-05 2023-07-02T14:04:07.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:07.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373463+0000", 2023-07-02T14:04:07.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000129095 2023-07-02T14:04:07.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:07.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373592+0000", 2023-07-02T14:04:07.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001 2023-07-02T14:04:07.146 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.146 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.146 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.146 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2787 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1433600~8192] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:07.147 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:07.147 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.677579799999997, 2023-07-02T14:04:07.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999, 2023-07-02T14:04:07.147 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:07.147 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:07.147 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:07.148 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:07.148 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:07.148 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2787 2023-07-02T14:04:07.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.148 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:07.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:07.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:07.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333436+0000", 2023-07-02T14:04:07.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2731999999999999e-05 2023-07-02T14:04:07.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:07.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:04:07.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7440000000000001e-06 2023-07-02T14:04:07.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:07.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333456+0000", 2023-07-02T14:04:07.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8650000000000003e-06 2023-07-02T14:04:07.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:07.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333460+0000", 2023-07-02T14:04:07.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0077e-05 2023-07-02T14:04:07.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333530+0000", 2023-07-02T14:04:07.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4180000000000001e-06 2023-07-02T14:04:07.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:07.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333533+0000", 2023-07-02T14:04:07.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0284002720000001 2023-07-02T14:04:07.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361933+0000", 2023-07-02T14:04:07.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6781999999999996e-05 2023-07-02T14:04:07.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:07.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362030+0000", 2023-07-02T14:04:07.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018305400000000001 2023-07-02T14:04:07.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:07.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362213+0000", 2023-07-02T14:04:07.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097015109999999995 2023-07-02T14:04:07.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:07.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371915+0000", 2023-07-02T14:04:07.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001742295 2023-07-02T14:04:07.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:07.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373657+0000", 2023-07-02T14:04:07.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5980000000000003e-05 2023-07-02T14:04:07.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:07.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373713+0000", 2023-07-02T14:04:07.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011783800000000001 2023-07-02T14:04:07.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:07.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373831+0000", 2023-07-02T14:04:07.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999 2023-07-02T14:04:07.158 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.158 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.159 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.159 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2803 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 851968~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:07.159 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:07.159 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.677537948000001, 2023-07-02T14:04:07.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514, 2023-07-02T14:04:07.160 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:07.160 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:07.160 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:07.160 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:07.160 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:07.160 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2803 2023-07-02T14:04:07.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.160 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:07.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:07.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:07.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:07.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:07.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:07.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:07.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:07.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:07.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4943000000000001e-05 2023-07-02T14:04:07.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:07.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333493+0000", 2023-07-02T14:04:07.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2629999999999997e-06 2023-07-02T14:04:07.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:07.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333500+0000", 2023-07-02T14:04:07.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4159999999999997e-06 2023-07-02T14:04:07.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:07.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333505+0000", 2023-07-02T14:04:07.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4518999999999998e-05 2023-07-02T14:04:07.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333559+0000", 2023-07-02T14:04:07.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5900000000000008e-06 2023-07-02T14:04:07.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:07.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333568+0000", 2023-07-02T14:04:07.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0291061779999999 2023-07-02T14:04:07.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362674+0000", 2023-07-02T14:04:07.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110412 2023-07-02T14:04:07.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:07.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362784+0000", 2023-07-02T14:04:07.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015384599999999999 2023-07-02T14:04:07.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:07.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362938+0000", 2023-07-02T14:04:07.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091682310000000006 2023-07-02T14:04:07.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:07.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372106+0000", 2023-07-02T14:04:07.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017703230000000001 2023-07-02T14:04:07.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:07.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373877+0000", 2023-07-02T14:04:07.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6387000000000002e-05 2023-07-02T14:04:07.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:07.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373923+0000", 2023-07-02T14:04:07.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014099199999999999 2023-07-02T14:04:07.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:07.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374064+0000", 2023-07-02T14:04:07.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514 2023-07-02T14:04:07.171 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.171 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.171 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.171 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2805 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 634880~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:07.171 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:07.171 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.677504773000003, 2023-07-02T14:04:07.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814, 2023-07-02T14:04:07.172 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:07.172 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:07.172 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:07.172 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:07.172 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:07.172 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2805 2023-07-02T14:04:07.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.173 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:07.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:07.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:07.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:07.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:07.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:07.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:07.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:07.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333511+0000", 2023-07-02T14:04:07.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207092 2023-07-02T14:04:07.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:07.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333718+0000", 2023-07-02T14:04:07.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5110000000000003e-06 2023-07-02T14:04:07.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:07.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333727+0000", 2023-07-02T14:04:07.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3270000000000007e-06 2023-07-02T14:04:07.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:07.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333736+0000", 2023-07-02T14:04:07.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0319999999999999e-05 2023-07-02T14:04:07.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333756+0000", 2023-07-02T14:04:07.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1695000000000001e-05 2023-07-02T14:04:07.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:07.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:04:07.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0295750560000001 2023-07-02T14:04:07.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363343+0000", 2023-07-02T14:04:07.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3387999999999994e-05 2023-07-02T14:04:07.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:07.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363437+0000", 2023-07-02T14:04:07.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018373 2023-07-02T14:04:07.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:07.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363620+0000", 2023-07-02T14:04:07.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0084948769999999996 2023-07-02T14:04:07.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:07.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372115+0000", 2023-07-02T14:04:07.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020082170000000001 2023-07-02T14:04:07.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:07.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374123+0000", 2023-07-02T14:04:07.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8875999999999997e-05 2023-07-02T14:04:07.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:07.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374182+0000", 2023-07-02T14:04:07.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0001272 2023-07-02T14:04:07.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:07.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374309+0000", 2023-07-02T14:04:07.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814 2023-07-02T14:04:07.183 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.183 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.183 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.183 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2806 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 1359872~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:07.183 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:07.184 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.677274441999998, 2023-07-02T14:04:07.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234, 2023-07-02T14:04:07.184 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:07.184 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:07.184 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:07.184 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:07.184 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:07.184 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2806 2023-07-02T14:04:07.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.185 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:07.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:07.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:07.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:07.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:07.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:07.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:07.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:07.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333741+0000", 2023-07-02T14:04:07.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3978999999999999e-05 2023-07-02T14:04:07.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:07.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333755+0000", 2023-07-02T14:04:07.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:04:07.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:07.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333761+0000", 2023-07-02T14:04:07.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7010000000000001e-06 2023-07-02T14:04:07.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:07.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333764+0000", 2023-07-02T14:04:07.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5086000000000003e-05 2023-07-02T14:04:07.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333799+0000", 2023-07-02T14:04:07.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6389999999999992e-06 2023-07-02T14:04:07.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:07.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333806+0000", 2023-07-02T14:04:07.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.030496675 2023-07-02T14:04:07.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364303+0000", 2023-07-02T14:04:07.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011382700000000001 2023-07-02T14:04:07.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:07.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364417+0000", 2023-07-02T14:04:07.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192058 2023-07-02T14:04:07.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:07.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364609+0000", 2023-07-02T14:04:07.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075132130000000004 2023-07-02T14:04:07.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:07.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372122+0000", 2023-07-02T14:04:07.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022731750000000001 2023-07-02T14:04:07.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:07.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374395+0000", 2023-07-02T14:04:07.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5277000000000005e-05 2023-07-02T14:04:07.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:07.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374460+0000", 2023-07-02T14:04:07.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017074599999999999 2023-07-02T14:04:07.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:07.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374631+0000", 2023-07-02T14:04:07.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234 2023-07-02T14:04:07.195 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.195 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.195 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.195 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 868352~49152] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:07.195 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:07.196 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.677247727999998, 2023-07-02T14:04:07.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001, 2023-07-02T14:04:07.196 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:07.196 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:07.196 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:07.196 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:07.196 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:07.196 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:04:07.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.197 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:07.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:07.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:07.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:07.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:07.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:07.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:07.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:07.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:04:07.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6355000000000001e-05 2023-07-02T14:04:07.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:07.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333794+0000", 2023-07-02T14:04:07.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8329999999999996e-06 2023-07-02T14:04:07.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:07.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333801+0000", 2023-07-02T14:04:07.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5419999999999996e-06 2023-07-02T14:04:07.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:07.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333807+0000", 2023-07-02T14:04:07.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2023-07-02T14:04:07.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333826+0000", 2023-07-02T14:04:07.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.303e-06 2023-07-02T14:04:07.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:07.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333830+0000", 2023-07-02T14:04:07.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.031137559 2023-07-02T14:04:07.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364967+0000", 2023-07-02T14:04:07.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1390000000000004e-05 2023-07-02T14:04:07.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:07.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365059+0000", 2023-07-02T14:04:07.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017597399999999999 2023-07-02T14:04:07.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:07.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365234+0000", 2023-07-02T14:04:07.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068943950000000002 2023-07-02T14:04:07.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:07.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372129+0000", 2023-07-02T14:04:07.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026189910000000002 2023-07-02T14:04:07.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:07.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374748+0000", 2023-07-02T14:04:07.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6613999999999996e-05 2023-07-02T14:04:07.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:07.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374814+0000", 2023-07-02T14:04:07.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502300000000001 2023-07-02T14:04:07.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:07.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374950+0000", 2023-07-02T14:04:07.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001 2023-07-02T14:04:07.206 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.207 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.207 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.207 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2815 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 917504~49152] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:07.207 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:07.207 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.673033494999999, 2023-07-02T14:04:07.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998, 2023-07-02T14:04:07.208 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:07.208 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:07.208 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:07.208 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:07.208 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:07.208 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2815 2023-07-02T14:04:07.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.209 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:07.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:07.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:07.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:07.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:07.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:07.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:04:07.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:07.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337980+0000", 2023-07-02T14:04:07.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017126700000000001 2023-07-02T14:04:07.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:07.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338152+0000", 2023-07-02T14:04:07.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2938999999999999e-05 2023-07-02T14:04:07.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:07.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338164+0000", 2023-07-02T14:04:07.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4448999999999999e-05 2023-07-02T14:04:07.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:07.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:04:07.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0281812 2023-07-02T14:04:07.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366370+0000", 2023-07-02T14:04:07.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1840000000000002e-05 2023-07-02T14:04:07.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:07.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366462+0000", 2023-07-02T14:04:07.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000186112 2023-07-02T14:04:07.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:07.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366648+0000", 2023-07-02T14:04:07.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054905709999999996 2023-07-02T14:04:07.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:07.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372139+0000", 2023-07-02T14:04:07.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044954790000000001 2023-07-02T14:04:07.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376634+0000", 2023-07-02T14:04:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4621000000000004e-05 2023-07-02T14:04:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376699+0000", 2023-07-02T14:04:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012862299999999999 2023-07-02T14:04:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376827+0000", 2023-07-02T14:04:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998 2023-07-02T14:04:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.218 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2816 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 917504~65536] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:07.218 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:07.218 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.672819115999999, 2023-07-02T14:04:07.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001, 2023-07-02T14:04:07.218 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:07.218 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:07.218 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:07.219 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:07.219 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:07.219 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2816 2023-07-02T14:04:07.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.219 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:07.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:07.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:07.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:07.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:07.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:07.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:07.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:07.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338196+0000", 2023-07-02T14:04:07.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6286999999999999e-05 2023-07-02T14:04:07.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:07.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338243+0000", 2023-07-02T14:04:07.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7030000000000001e-06 2023-07-02T14:04:07.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:07.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338252+0000", 2023-07-02T14:04:07.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:04:07.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:07.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338258+0000", 2023-07-02T14:04:07.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032085734999999997 2023-07-02T14:04:07.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370344+0000", 2023-07-02T14:04:07.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108621 2023-07-02T14:04:07.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:07.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370452+0000", 2023-07-02T14:04:07.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014936499999999999 2023-07-02T14:04:07.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:07.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370602+0000", 2023-07-02T14:04:07.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001778012 2023-07-02T14:04:07.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:07.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372380+0000", 2023-07-02T14:04:07.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044876380000000004 2023-07-02T14:04:07.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:07.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376867+0000", 2023-07-02T14:04:07.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9870999999999999e-05 2023-07-02T14:04:07.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:07.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376897+0000", 2023-07-02T14:04:07.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011401 2023-07-02T14:04:07.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:07.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.377011+0000", 2023-07-02T14:04:07.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001 2023-07-02T14:04:07.228 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.228 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.228 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.228 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:3878 2.1 e80/79 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head v 80'436, mlcod=80'429)", 2023-07-02T14:04:07.229 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:07.229 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 28.903565618999998, 2023-07-02T14:04:07.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037799472000000001, 2023-07-02T14:04:07.229 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:07.229 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:07.229 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:07.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:07.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:07.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:07.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:07.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:07.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:07.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:07.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107450+0000", 2023-07-02T14:04:07.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023914520000000001 2023-07-02T14:04:07.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:07.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109841+0000", 2023-07-02T14:04:07.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0823e-05 2023-07-02T14:04:07.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:07.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109862+0000", 2023-07-02T14:04:07.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2023-07-02T14:04:07.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:07.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109874+0000", 2023-07-02T14:04:07.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075009600000000001 2023-07-02T14:04:07.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.110624+0000", 2023-07-02T14:04:07.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5176000000000001e-05 2023-07-02T14:04:07.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:07.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.110649+0000", 2023-07-02T14:04:07.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034557310000000001 2023-07-02T14:04:07.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:07.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145206+0000", 2023-07-02T14:04:07.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4110999999999998e-05 2023-07-02T14:04:07.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:07.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145250+0000", 2023-07-02T14:04:07.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037799472000000001 2023-07-02T14:04:07.236 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.236 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.236 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.236 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:7177 2.5 2:a6e4b3cf:::rbd_data.125de1449605.00000000000000c4:head [write 0~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e91)", 2023-07-02T14:04:07.236 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:01.433694+0000", 2023-07-02T14:04:07.237 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.5773225000000002, 2023-07-02T14:04:07.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033301720999999999, 2023-07-02T14:04:07.237 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:07.237 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:07.237 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:07.237 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:07.237 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:07.237 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7177 2023-07-02T14:04:07.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.238 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:07.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:07.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433694+0000", 2023-07-02T14:04:07.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:07.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:07.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433694+0000", 2023-07-02T14:04:07.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:07.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:07.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433693+0000", 2023-07-02T14:04:07.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6749999999999997e-05 2023-07-02T14:04:07.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:07.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433750+0000", 2023-07-02T14:04:07.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0451e-05 2023-07-02T14:04:07.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:07.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433760+0000", 2023-07-02T14:04:07.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8200000000000002e-06 2023-07-02T14:04:07.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:07.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433766+0000", 2023-07-02T14:04:07.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012906031 2023-07-02T14:04:07.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:07.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.446672+0000", 2023-07-02T14:04:07.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3190999999999998e-05 2023-07-02T14:04:07.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:07.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.446745+0000", 2023-07-02T14:04:07.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020003231999999999 2023-07-02T14:04:07.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:07.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.466749+0000", 2023-07-02T14:04:07.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010595 2023-07-02T14:04:07.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:07.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.466855+0000", 2023-07-02T14:04:07.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014102799999999999 2023-07-02T14:04:07.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:07.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:07.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:07.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.466996+0000", 2023-07-02T14:04:07.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033301720999999999 2023-07-02T14:04:07.245 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.245 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.245 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.245 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:07.246 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:07.246 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:07.247 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:04:07.247 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:07.246+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:07.348 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:07.347+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:07.413 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:07.413 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:07.414 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:07.414 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:07.423 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:04:07.464 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:07.463+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:07.549 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:07.548+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:07.587 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:07.588 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:07.589 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:04:07.597 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:04:07.604 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:04:07.614 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap16 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap17 b3fbb028-e267-4d25-a750-93ba18a2316a@snap17.diff 2023-07-02T14:04:07.650 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:07.650+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:07.679 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:07.678+0000 7fcd8c999700 0 -- 172.21.15.163:0/2156580 >> [v2:172.21.15.16:6800/83359,v1:172.21.15.16:6801/83359] conn(0x7fcd6c00fe50 msgr2=0x7fcd6c010250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:07.683 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:07.682+0000 7fcd8d19a700 0 -- 172.21.15.163:0/2156580 >> [v2:172.21.15.22:6800/83351,v1:172.21.15.22:6801/83351] conn(0x7fcd68008780 msgr2=0x7fcd6800abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:07.752 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:07.751+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:07.768 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:07.788 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:07.807 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:07.808 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:07.808 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:07.822 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:04:07.828 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:07.850 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:07.850 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:04:07.850 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:04:07.851 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:04:07.851 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:07.851 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:04:07.851 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:04:07.852 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:07.851+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:07.853 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:07.854 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:07.953 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:07.952+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:07.954 INFO:teuthology.orchestra.run.smithi022.stderr:Nothing to do for non-filestore backend 2023-07-02T14:04:07.960 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:07.960 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:07.974 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:04:08.057 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:08.055+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:08.074 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:04:08.106 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:08.106 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:08.117 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:04:08.157 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:08.157+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:08.259 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:08.258+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:08.305 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:08.306 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:08.319 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:04:08.360 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:08.358+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:08.445 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:08.446 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:08.456 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:04:08.461 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:08.460+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:08.561 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:08.560+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:08.580 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:08.580 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:08.591 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:04:08.675 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:08.674+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:08.730 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete...2023-07-02T14:04:08.729+0000 7fcd8c999700 0 -- 172.21.15.163:0/2156580 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x5576cbed3ed0 msgr2=0x5576cbed62c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:08.740 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:08.739+0000 7fcd8d19a700 0 -- 172.21.15.163:0/2156580 >> [v2:172.21.15.22:6800/83351,v1:172.21.15.22:6801/83351] conn(0x7fcd6c00fe50 msgr2=0x7fcd70039620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:08.744 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:08.743+0000 7fcd8c999700 0 -- 172.21.15.163:0/2156580 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7fcd68008780 msgr2=0x7fcd7003e0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:08.759 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:08.759 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:08.763 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:08.761+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:08.770 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:04:08.859 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 34% complete... Exporting image: 35% complete...2023-07-02T14:04:08.858+0000 7fcd8c999700 0 -- 172.21.15.163:0/2156580 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7fcd6c00fe50 msgr2=0x7fcd700395c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:08.863 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:08.862+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:08.935 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:08.935 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:08.946 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:04:08.988 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:08.988+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:09.031 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 100% complete...done. 2023-07-02T14:04:09.051 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:04:09.051 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=18 2023-07-02T14:04:09.051 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap18 2023-07-02T14:04:09.064 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:09.063+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:09.070 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:09.070 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:09.081 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:09.165 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:09.164+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:09.245 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:09.246 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:09.256 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:09.300 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:09.299+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:09.367 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:09.366+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:09.382 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:09.382 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:09.392 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:04:09.476 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:09.475+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:09.556 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:09.556 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:09.567 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:04:09.608 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:09.607+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:09.669 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:09.668+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:09.731 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:09.731 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:09.741 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:04:09.770 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:09.770+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:09.855 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:04:09.866 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:09.866 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:09.870 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:09.870+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:09.877 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:04:09.971 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:09.970+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:09.995 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:10.013 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:10.031 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:10.045 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:10.045 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:10.045 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:10.045 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:10.045 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:10.050 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:10.056 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:04:10.069 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:10.070 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:04:10.070 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:04:10.070 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:04:10.070 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:10.071 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:04:10.071 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:04:10.072 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:10.071+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:10.073 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:10.074 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:10.087 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:04:10.103 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:04:10.109 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 3 2023-07-02T14:04:10.172 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:10.171+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:10.200 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:10.200 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:10.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:10.200 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:10.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.201 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:04:10.201 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:10.201 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 60.745822936000003, 2023-07-02T14:04:10.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:04:10.201 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.201 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:10.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.202 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:04:10.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.202 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:10.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:10.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:10.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:04:10.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:04:10.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:04:10.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:04:10.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:04:10.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:04:10.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:04:10.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:04:10.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:04:10.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:04:10.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:10.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:04:10.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:04:10.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:04:10.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:04:10.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:04:10.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:04:10.209 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.210 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.210 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.210 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:04:10.210 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:10.210 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.875129928, 2023-07-02T14:04:10.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:04:10.211 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.211 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:10.211 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.211 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:04:10.211 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:04:10.211 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:04:10.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.211 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:10.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:10.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:10.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:04:10.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:04:10.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:04:10.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:04:10.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:04:10.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:04:10.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:04:10.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:04:10.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:04:10.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:04:10.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:10.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:04:10.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:04:10.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:04:10.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:04:10.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:10.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:04:10.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:04:10.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:04:10.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:04:10.219 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.219 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.219 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.220 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:10.220 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:10.220 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.875005207000001, 2023-07-02T14:04:10.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:04:10.220 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.220 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:10.220 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.221 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.221 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.221 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:04:10.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.221 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:10.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:10.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:10.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:04:10.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:04:10.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:04:10.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:04:10.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:04:10.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:04:10.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:04:10.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:04:10.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:04:10.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:04:10.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:10.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:04:10.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:04:10.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:04:10.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:04:10.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:04:10.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:04:10.229 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.229 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.229 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.229 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:10.229 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:10.230 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.868155323000003, 2023-07-02T14:04:10.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:04:10.230 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.230 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:10.230 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.230 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.230 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.230 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:04:10.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.231 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:10.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:10.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:04:10.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:04:10.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:04:10.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:04:10.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:04:10.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:04:10.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:04:10.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:04:10.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:04:10.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:04:10.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:04:10.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:10.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:04:10.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:04:10.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:04:10.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:04:10.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:04:10.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:04:10.238 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.238 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.238 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.239 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:10.239 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:10.239 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.868092398000002, 2023-07-02T14:04:10.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:04:10.239 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.239 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:10.240 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.240 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.240 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.240 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:04:10.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.240 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:10.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:10.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:10.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:04:10.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:04:10.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:04:10.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:04:10.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:04:10.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:04:10.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:04:10.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:04:10.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:04:10.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:04:10.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:10.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:04:10.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:04:10.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:04:10.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:04:10.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:04:10.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:04:10.247 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.247 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.247 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.247 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:10.247 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:10.247 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.867791977000003, 2023-07-02T14:04:10.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:04:10.248 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.248 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:10.248 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.248 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.248 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.248 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:04:10.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.249 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:10.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:10.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:10.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:04:10.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:04:10.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:04:10.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:04:10.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:04:10.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:04:10.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:04:10.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:04:10.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:04:10.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:04:10.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:10.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:04:10.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:04:10.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:04:10.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:04:10.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:10.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:04:10.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:04:10.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:04:10.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:04:10.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:04:10.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:10.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:04:10.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:04:10.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:10.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:04:10.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:04:10.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:10.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:04:10.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:04:10.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:10.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:04:10.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:04:10.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:04:10.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:04:10.259 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.259 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.259 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.260 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:04:10.260 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:10.260 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.867267476999999, 2023-07-02T14:04:10.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:04:10.260 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.260 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:10.261 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.261 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.261 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.261 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:04:10.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.261 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:10.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:10.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:10.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:10.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:04:10.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:04:10.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:04:10.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:04:10.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:04:10.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:04:10.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:04:10.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:04:10.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:04:10.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:10.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:04:10.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:04:10.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:04:10.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:04:10.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:10.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:04:10.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:04:10.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:04:10.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:04:10.269 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.270 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.270 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.270 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:04:10.270 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:10.270 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.743248352000002, 2023-07-02T14:04:10.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:04:10.271 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.271 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:10.271 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.271 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.271 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.271 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:04:10.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.272 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:10.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:10.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:10.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:04:10.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:04:10.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:04:10.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:04:10.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:04:10.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:04:10.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:04:10.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:04:10.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:04:10.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:04:10.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:10.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:04:10.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:04:10.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:04:10.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:04:10.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:04:10.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:04:10.278 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.278 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.279 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.279 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:10.279 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:10.279 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.417765648, 2023-07-02T14:04:10.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:04:10.279 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.280 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:10.280 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.280 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.280 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.280 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:04:10.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.280 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:10.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:10.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:10.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:04:10.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:04:10.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:04:10.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:04:10.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:04:10.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:04:10.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:04:10.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:04:10.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:10.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:04:10.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:10.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:04:10.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:04:10.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:04:10.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:04:10.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:10.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:04:10.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:04:10.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:10.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:04:10.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:04:10.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:10.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:04:10.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:04:10.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:10.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:04:10.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:04:10.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:10.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:04:10.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:04:10.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:04:10.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:04:10.290 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.290 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.290 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.291 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3887 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:10.291 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:10.291 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.413902458999999, 2023-07-02T14:04:10.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998, 2023-07-02T14:04:10.291 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.291 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:10.292 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.292 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.292 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.292 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3887 2023-07-02T14:04:10.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.292 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:10.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:10.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:10.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784019+0000", 2023-07-02T14:04:10.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4073999999999998e-05 2023-07-02T14:04:10.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784063+0000", 2023-07-02T14:04:10.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6630000000000004e-06 2023-07-02T14:04:10.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784070+0000", 2023-07-02T14:04:10.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.286e-06 2023-07-02T14:04:10.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784073+0000", 2023-07-02T14:04:10.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4941999999999998e-05 2023-07-02T14:04:10.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784118+0000", 2023-07-02T14:04:10.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8120000000000008e-06 2023-07-02T14:04:10.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:10.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784126+0000", 2023-07-02T14:04:10.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32999709399999999 2023-07-02T14:04:10.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114123+0000", 2023-07-02T14:04:10.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000157257 2023-07-02T14:04:10.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:10.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114280+0000", 2023-07-02T14:04:10.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000341152 2023-07-02T14:04:10.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:10.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114622+0000", 2023-07-02T14:04:10.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019539766 2023-07-02T14:04:10.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:10.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.134161+0000", 2023-07-02T14:04:10.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015820796000000002 2023-07-02T14:04:10.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:10.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149982+0000", 2023-07-02T14:04:10.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.05e-05 2023-07-02T14:04:10.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:10.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150043+0000", 2023-07-02T14:04:10.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6453999999999999e-05 2023-07-02T14:04:10.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150089+0000", 2023-07-02T14:04:10.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998 2023-07-02T14:04:10.305 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.305 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.305 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.306 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3939 2.1 2:9ef34a83:::rbd_data.125de1449605.00000000000000b8:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:10.306 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:10.306 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.413844951000002, 2023-07-02T14:04:10.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001, 2023-07-02T14:04:10.306 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.307 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:10.307 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.307 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.307 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.307 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3939 2023-07-02T14:04:10.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.307 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:10.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:10.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:10.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:10.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046852830000000002 2023-07-02T14:04:10.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788763+0000", 2023-07-02T14:04:10.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3185e-05 2023-07-02T14:04:10.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788776+0000", 2023-07-02T14:04:10.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8470000000000003e-06 2023-07-02T14:04:10.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788781+0000", 2023-07-02T14:04:10.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7104000000000002e-05 2023-07-02T14:04:10.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788828+0000", 2023-07-02T14:04:10.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3516000000000001e-05 2023-07-02T14:04:10.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:10.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788841+0000", 2023-07-02T14:04:10.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33064513600000001 2023-07-02T14:04:10.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119487+0000", 2023-07-02T14:04:10.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9876999999999997e-05 2023-07-02T14:04:10.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:10.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119537+0000", 2023-07-02T14:04:10.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014160999999999999 2023-07-02T14:04:10.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:10.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119678+0000", 2023-07-02T14:04:10.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023054392 2023-07-02T14:04:10.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:10.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.142733+0000", 2023-07-02T14:04:10.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074065470000000003 2023-07-02T14:04:10.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:10.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150139+0000", 2023-07-02T14:04:10.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5025999999999997e-05 2023-07-02T14:04:10.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:10.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150204+0000", 2023-07-02T14:04:10.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147999999999997e-05 2023-07-02T14:04:10.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150241+0000", 2023-07-02T14:04:10.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001 2023-07-02T14:04:10.318 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.318 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.318 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.319 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3940 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:10.319 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:10.319 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.409133918999999, 2023-07-02T14:04:10.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001, 2023-07-02T14:04:10.319 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.319 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:10.320 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.320 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.320 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.320 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3940 2023-07-02T14:04:10.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.320 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:10.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:10.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:10.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788788+0000", 2023-07-02T14:04:10.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043166560000000003 2023-07-02T14:04:10.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793105+0000", 2023-07-02T14:04:10.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5970000000000001e-05 2023-07-02T14:04:10.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793121+0000", 2023-07-02T14:04:10.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0939999999999998e-06 2023-07-02T14:04:10.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793126+0000", 2023-07-02T14:04:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2438000000000002e-05 2023-07-02T14:04:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793178+0000", 2023-07-02T14:04:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2019e-05 2023-07-02T14:04:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793190+0000", 2023-07-02T14:04:10.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33208757799999999 2023-07-02T14:04:10.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125278+0000", 2023-07-02T14:04:10.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5255999999999997e-05 2023-07-02T14:04:10.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:10.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125373+0000", 2023-07-02T14:04:10.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015674999999999999 2023-07-02T14:04:10.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:10.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125530+0000", 2023-07-02T14:04:10.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018767308 2023-07-02T14:04:10.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:10.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144297+0000", 2023-07-02T14:04:10.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060291959999999997 2023-07-02T14:04:10.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:10.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150326+0000", 2023-07-02T14:04:10.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3886999999999993e-05 2023-07-02T14:04:10.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:10.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150390+0000", 2023-07-02T14:04:10.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010255 2023-07-02T14:04:10.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150493+0000", 2023-07-02T14:04:10.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001 2023-07-02T14:04:10.333 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.333 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.333 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.333 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3947 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:10.333 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:10.334 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.404788392, 2023-07-02T14:04:10.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999, 2023-07-02T14:04:10.334 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.334 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:10.334 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.334 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.334 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.334 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3947 2023-07-02T14:04:10.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.335 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:10.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:10.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:10.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793133+0000", 2023-07-02T14:04:10.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2519999999999999e-05 2023-07-02T14:04:10.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793176+0000", 2023-07-02T14:04:10.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:04:10.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793180+0000", 2023-07-02T14:04:10.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1709999999999998e-06 2023-07-02T14:04:10.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793184+0000", 2023-07-02T14:04:10.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1721999999999999e-05 2023-07-02T14:04:10.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793215+0000", 2023-07-02T14:04:10.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1280000000000008e-06 2023-07-02T14:04:10.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:10.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793224+0000", 2023-07-02T14:04:10.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34218549599999998 2023-07-02T14:04:10.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135409+0000", 2023-07-02T14:04:10.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117442 2023-07-02T14:04:10.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:10.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135526+0000", 2023-07-02T14:04:10.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000206257 2023-07-02T14:04:10.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:10.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135733+0000", 2023-07-02T14:04:10.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085700029999999996 2023-07-02T14:04:10.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:10.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144303+0000", 2023-07-02T14:04:10.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006278652 2023-07-02T14:04:10.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:10.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150581+0000", 2023-07-02T14:04:10.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4765999999999996e-05 2023-07-02T14:04:10.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:10.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150646+0000", 2023-07-02T14:04:10.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4777999999999995e-05 2023-07-02T14:04:10.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150711+0000", 2023-07-02T14:04:10.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999 2023-07-02T14:04:10.344 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.345 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.345 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.345 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3980 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 65536~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:10.345 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:10.345 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.404734509999997, 2023-07-02T14:04:10.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999, 2023-07-02T14:04:10.346 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.346 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:10.346 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.346 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.346 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.346 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3980 2023-07-02T14:04:10.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.346 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:10.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:10.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:10.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:10.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020244260000000002 2023-07-02T14:04:10.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795212+0000", 2023-07-02T14:04:10.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0005e-05 2023-07-02T14:04:10.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795222+0000", 2023-07-02T14:04:10.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9860000000000002e-06 2023-07-02T14:04:10.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795227+0000", 2023-07-02T14:04:10.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6369999999999999e-05 2023-07-02T14:04:10.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795264+0000", 2023-07-02T14:04:10.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1052e-05 2023-07-02T14:04:10.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:10.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795275+0000", 2023-07-02T14:04:10.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.343850459 2023-07-02T14:04:10.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139125+0000", 2023-07-02T14:04:10.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6760000000000004e-05 2023-07-02T14:04:10.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:10.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139202+0000", 2023-07-02T14:04:10.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015092799999999999 2023-07-02T14:04:10.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:10.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139353+0000", 2023-07-02T14:04:10.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011425940000000001 2023-07-02T14:04:10.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:10.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150779+0000", 2023-07-02T14:04:10.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033223060000000001 2023-07-02T14:04:10.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:10.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154101+0000", 2023-07-02T14:04:10.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9268999999999998e-05 2023-07-02T14:04:10.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:10.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154161+0000", 2023-07-02T14:04:10.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6499999999999999e-05 2023-07-02T14:04:10.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154207+0000", 2023-07-02T14:04:10.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999 2023-07-02T14:04:10.356 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.356 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.356 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.357 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3982 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1835008~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:10.357 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:10.357 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.402689223000003, 2023-07-02T14:04:10.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002, 2023-07-02T14:04:10.357 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.357 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:10.358 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.358 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.358 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.358 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3982 2023-07-02T14:04:10.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.358 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:10.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:10.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:10.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795233+0000", 2023-07-02T14:04:10.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8680999999999998e-05 2023-07-02T14:04:10.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795272+0000", 2023-07-02T14:04:10.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5629999999999999e-06 2023-07-02T14:04:10.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795277+0000", 2023-07-02T14:04:10.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8700000000000001e-06 2023-07-02T14:04:10.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795280+0000", 2023-07-02T14:04:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1964000000000001e-05 2023-07-02T14:04:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795312+0000", 2023-07-02T14:04:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.314e-06 2023-07-02T14:04:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795319+0000", 2023-07-02T14:04:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34828442599999998 2023-07-02T14:04:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143603+0000", 2023-07-02T14:04:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2438000000000001e-05 2023-07-02T14:04:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143665+0000", 2023-07-02T14:04:10.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3079999999999997e-05 2023-07-02T14:04:10.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:10.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143759+0000", 2023-07-02T14:04:10.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070633079999999999 2023-07-02T14:04:10.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:10.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150822+0000", 2023-07-02T14:04:10.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034430210000000001 2023-07-02T14:04:10.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:10.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154265+0000", 2023-07-02T14:04:10.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4249999999999998e-05 2023-07-02T14:04:10.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:10.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154309+0000", 2023-07-02T14:04:10.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2020000000000001e-05 2023-07-02T14:04:10.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154351+0000", 2023-07-02T14:04:10.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002 2023-07-02T14:04:10.368 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.368 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.368 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.368 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4054 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1900544~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:10.369 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:10.369 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.402637552000002, 2023-07-02T14:04:10.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002, 2023-07-02T14:04:10.369 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.369 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:10.369 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.369 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.369 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.370 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4054 2023-07-02T14:04:10.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.370 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:10.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:10.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:10.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:10.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018874860000000001 2023-07-02T14:04:10.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797172+0000", 2023-07-02T14:04:10.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3000000000000007e-06 2023-07-02T14:04:10.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797182+0000", 2023-07-02T14:04:10.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6260000000000003e-06 2023-07-02T14:04:10.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797186+0000", 2023-07-02T14:04:10.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9013e-05 2023-07-02T14:04:10.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797235+0000", 2023-07-02T14:04:10.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5990000000000008e-06 2023-07-02T14:04:10.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:10.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797245+0000", 2023-07-02T14:04:10.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34785621999999999 2023-07-02T14:04:10.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145101+0000", 2023-07-02T14:04:10.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7762000000000006e-05 2023-07-02T14:04:10.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:10.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145169+0000", 2023-07-02T14:04:10.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013469400000000001 2023-07-02T14:04:10.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:10.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145304+0000", 2023-07-02T14:04:10.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089538039999999992 2023-07-02T14:04:10.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:10.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154257+0000", 2023-07-02T14:04:10.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104218 2023-07-02T14:04:10.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:10.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154362+0000", 2023-07-02T14:04:10.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4168000000000001e-05 2023-07-02T14:04:10.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:10.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154406+0000", 2023-07-02T14:04:10.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3098000000000001e-05 2023-07-02T14:04:10.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154429+0000", 2023-07-02T14:04:10.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002 2023-07-02T14:04:10.379 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.380 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.380 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.380 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4056 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3670016~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:10.380 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:10.380 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.400731039, 2023-07-02T14:04:10.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997, 2023-07-02T14:04:10.381 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.381 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:10.381 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.381 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.381 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.381 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4056 2023-07-02T14:04:10.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.381 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:10.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:10.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:10.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797191+0000", 2023-07-02T14:04:10.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0438999999999997e-05 2023-07-02T14:04:10.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797232+0000", 2023-07-02T14:04:10.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6929999999999998e-06 2023-07-02T14:04:10.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797237+0000", 2023-07-02T14:04:10.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9739999999999998e-06 2023-07-02T14:04:10.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797240+0000", 2023-07-02T14:04:10.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0243000000000002e-05 2023-07-02T14:04:10.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797270+0000", 2023-07-02T14:04:10.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9889999999999997e-06 2023-07-02T14:04:10.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:10.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797276+0000", 2023-07-02T14:04:10.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35358319799999999 2023-07-02T14:04:10.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150859+0000", 2023-07-02T14:04:10.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1357e-05 2023-07-02T14:04:10.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:10.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150940+0000", 2023-07-02T14:04:10.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014903 2023-07-02T14:04:10.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:10.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151089+0000", 2023-07-02T14:04:10.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033512519999999999 2023-07-02T14:04:10.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:10.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154441+0000", 2023-07-02T14:04:10.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001050945 2023-07-02T14:04:10.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:10.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155492+0000", 2023-07-02T14:04:10.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2523e-05 2023-07-02T14:04:10.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:10.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155524+0000", 2023-07-02T14:04:10.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.057e-05 2023-07-02T14:04:10.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155555+0000", 2023-07-02T14:04:10.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997 2023-07-02T14:04:10.391 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.391 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.391 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.392 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4088 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3735552~458752] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:10.392 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:10.392 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.400678843999998, 2023-07-02T14:04:10.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998, 2023-07-02T14:04:10.392 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.392 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:10.393 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.393 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.393 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.393 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4088 2023-07-02T14:04:10.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.393 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:10.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:10.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:10.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:10.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00038503499999999999 2023-07-02T14:04:10.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797629+0000", 2023-07-02T14:04:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8739999999999995e-06 2023-07-02T14:04:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797637+0000", 2023-07-02T14:04:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7160000000000001e-06 2023-07-02T14:04:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797640+0000", 2023-07-02T14:04:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4342999999999998e-05 2023-07-02T14:04:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797695+0000", 2023-07-02T14:04:10.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0939999999999996e-06 2023-07-02T14:04:10.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:10.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797701+0000", 2023-07-02T14:04:10.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35408540799999999 2023-07-02T14:04:10.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151786+0000", 2023-07-02T14:04:10.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1953000000000006e-05 2023-07-02T14:04:10.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:10.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151868+0000", 2023-07-02T14:04:10.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015162800000000001 2023-07-02T14:04:10.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:10.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.152020+0000", 2023-07-02T14:04:10.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054561870000000004 2023-07-02T14:04:10.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:10.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.157476+0000", 2023-07-02T14:04:10.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059153199999999997 2023-07-02T14:04:10.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:10.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158068+0000", 2023-07-02T14:04:10.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2799e-05 2023-07-02T14:04:10.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:10.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158090+0000", 2023-07-02T14:04:10.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7235999999999999e-05 2023-07-02T14:04:10.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158108+0000", 2023-07-02T14:04:10.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998 2023-07-02T14:04:10.403 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.403 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.403 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.403 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4110 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:10.403 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:10.404 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.400277567000003, 2023-07-02T14:04:10.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864, 2023-07-02T14:04:10.404 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.404 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:10.404 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.404 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.404 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.405 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4110 2023-07-02T14:04:10.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.405 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:10.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:10.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:10.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:10.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9331e-05 2023-07-02T14:04:10.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797684+0000", 2023-07-02T14:04:10.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.147e-06 2023-07-02T14:04:10.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797689+0000", 2023-07-02T14:04:10.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5020000000000001e-06 2023-07-02T14:04:10.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797693+0000", 2023-07-02T14:04:10.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5254999999999999e-05 2023-07-02T14:04:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797748+0000", 2023-07-02T14:04:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5520000000000005e-06 2023-07-02T14:04:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:10.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797757+0000", 2023-07-02T14:04:10.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35674245700000001 2023-07-02T14:04:10.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154499+0000", 2023-07-02T14:04:10.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0090000000000001e-06 2023-07-02T14:04:10.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154506+0000", 2023-07-02T14:04:10.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864 2023-07-02T14:04:10.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.411 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.412 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:10.412 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:10.412 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.400225481, 2023-07-02T14:04:10.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998, 2023-07-02T14:04:10.412 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.412 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:10.413 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.413 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.413 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.413 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:04:10.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.413 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:10.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:10.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:10.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:10.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001714819 2023-07-02T14:04:10.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799412+0000", 2023-07-02T14:04:10.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7779999999999996e-06 2023-07-02T14:04:10.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799421+0000", 2023-07-02T14:04:10.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3510000000000002e-06 2023-07-02T14:04:10.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799424+0000", 2023-07-02T14:04:10.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0275999999999999e-05 2023-07-02T14:04:10.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799464+0000", 2023-07-02T14:04:10.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8389999999999996e-06 2023-07-02T14:04:10.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:10.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799473+0000", 2023-07-02T14:04:10.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35507638800000002 2023-07-02T14:04:10.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154550+0000", 2023-07-02T14:04:10.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6309999999999998e-06 2023-07-02T14:04:10.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154559+0000", 2023-07-02T14:04:10.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998 2023-07-02T14:04:10.420 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.420 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.420 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.420 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.420 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.420 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:10.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:04:10.423 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:10.421+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:10.522 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:10.521+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:10.587 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:10.587 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:10.588 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:10.588 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:10.597 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:04:10.623 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:10.622+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:10.724 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:10.723+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:10.760 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:10.761 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:10.761 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:10.761 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:10.761 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:10.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:04:10.825 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:10.825+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:10.925 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:10.925+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:10.936 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:183 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:04:10.937 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.649293258, 2023-07-02T14:04:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003, 2023-07-02T14:04:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.938 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 183 2023-07-02T14:04:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:04:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:04:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:04:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9790999999999997e-05 2023-07-02T14:04:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285598+0000", 2023-07-02T14:04:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5809999999999996e-06 2023-07-02T14:04:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285604+0000", 2023-07-02T14:04:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7599999999999998e-06 2023-07-02T14:04:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285607+0000", 2023-07-02T14:04:10.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021108188999999999 2023-07-02T14:04:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306715+0000", 2023-07-02T14:04:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6548e-05 2023-07-02T14:04:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306792+0000", 2023-07-02T14:04:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173869 2023-07-02T14:04:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306965+0000", 2023-07-02T14:04:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060498009999999996 2023-07-02T14:04:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313015+0000", 2023-07-02T14:04:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049443719999999998 2023-07-02T14:04:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317960+0000", 2023-07-02T14:04:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5927999999999998e-05 2023-07-02T14:04:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318006+0000", 2023-07-02T14:04:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6004999999999998e-05 2023-07-02T14:04:10.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318042+0000", 2023-07-02T14:04:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003 2023-07-02T14:04:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:186 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:04:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.649222432000002, 2023-07-02T14:04:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003, 2023-07-02T14:04:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.948 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 186 2023-07-02T14:04:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:04:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:04:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:10.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:04:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4279e-05 2023-07-02T14:04:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285664+0000", 2023-07-02T14:04:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3789999999999995e-06 2023-07-02T14:04:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285673+0000", 2023-07-02T14:04:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499999999999998e-06 2023-07-02T14:04:10.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285678+0000", 2023-07-02T14:04:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023261947000000002 2023-07-02T14:04:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.308940+0000", 2023-07-02T14:04:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010323399999999999 2023-07-02T14:04:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309044+0000", 2023-07-02T14:04:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018878300000000001 2023-07-02T14:04:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309232+0000", 2023-07-02T14:04:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037874150000000001 2023-07-02T14:04:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:10.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313020+0000", 2023-07-02T14:04:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050976140000000003 2023-07-02T14:04:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318117+0000", 2023-07-02T14:04:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3182000000000001e-05 2023-07-02T14:04:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:04:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5055999999999998e-05 2023-07-02T14:04:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318196+0000", 2023-07-02T14:04:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003 2023-07-02T14:04:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:189 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:10.958 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:04:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.649148668999999, 2023-07-02T14:04:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997, 2023-07-02T14:04:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.959 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:04:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:04:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:04:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285683+0000", 2023-07-02T14:04:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8451999999999999e-05 2023-07-02T14:04:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285782+0000", 2023-07-02T14:04:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3650000000000004e-06 2023-07-02T14:04:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285790+0000", 2023-07-02T14:04:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8230000000000002e-06 2023-07-02T14:04:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285795+0000", 2023-07-02T14:04:10.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024619367999999999 2023-07-02T14:04:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310414+0000", 2023-07-02T14:04:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014244600000000001 2023-07-02T14:04:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310557+0000", 2023-07-02T14:04:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020612699999999999 2023-07-02T14:04:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310763+0000", 2023-07-02T14:04:10.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003427687 2023-07-02T14:04:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314191+0000", 2023-07-02T14:04:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040492089999999998 2023-07-02T14:04:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318240+0000", 2023-07-02T14:04:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0287999999999999e-05 2023-07-02T14:04:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318270+0000", 2023-07-02T14:04:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0868000000000002e-05 2023-07-02T14:04:10.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318311+0000", 2023-07-02T14:04:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997 2023-07-02T14:04:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:192 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:04:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.649031895999997, 2023-07-02T14:04:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003, 2023-07-02T14:04:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:10.969 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 192 2023-07-02T14:04:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:04:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:04:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:10.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285800+0000", 2023-07-02T14:04:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100417 2023-07-02T14:04:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285900+0000", 2023-07-02T14:04:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8244999999999999e-05 2023-07-02T14:04:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285919+0000", 2023-07-02T14:04:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2499999999999997e-06 2023-07-02T14:04:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285924+0000", 2023-07-02T14:04:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025288155999999999 2023-07-02T14:04:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311212+0000", 2023-07-02T14:04:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4073999999999996e-05 2023-07-02T14:04:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311286+0000", 2023-07-02T14:04:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016219699999999999 2023-07-02T14:04:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311448+0000", 2023-07-02T14:04:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066020599999999999 2023-07-02T14:04:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318050+0000", 2023-07-02T14:04:10.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000327986 2023-07-02T14:04:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318378+0000", 2023-07-02T14:04:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8693999999999998e-05 2023-07-02T14:04:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318427+0000", 2023-07-02T14:04:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6656999999999998e-05 2023-07-02T14:04:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318464+0000", 2023-07-02T14:04:10.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003 2023-07-02T14:04:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:195 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:04:10.979 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.648900392000002, 2023-07-02T14:04:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001, 2023-07-02T14:04:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 195 2023-07-02T14:04:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:04:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:04:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285931+0000", 2023-07-02T14:04:10.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4530999999999995e-05 2023-07-02T14:04:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286006+0000", 2023-07-02T14:04:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4113e-05 2023-07-02T14:04:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286020+0000", 2023-07-02T14:04:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7459999999999997e-06 2023-07-02T14:04:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286026+0000", 2023-07-02T14:04:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025997769 2023-07-02T14:04:10.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312023+0000", 2023-07-02T14:04:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4403000000000001e-05 2023-07-02T14:04:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:10.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312078+0000", 2023-07-02T14:04:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020581399999999999 2023-07-02T14:04:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312284+0000", 2023-07-02T14:04:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005771043 2023-07-02T14:04:10.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318055+0000", 2023-07-02T14:04:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00046015200000000002 2023-07-02T14:04:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:10.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318515+0000", 2023-07-02T14:04:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9444000000000003e-05 2023-07-02T14:04:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318564+0000", 2023-07-02T14:04:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2141000000000002e-05 2023-07-02T14:04:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318606+0000", 2023-07-02T14:04:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001 2023-07-02T14:04:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:198 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.648800433000005, 2023-07-02T14:04:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999, 2023-07-02T14:04:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:10.990 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 198 2023-07-02T14:04:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:10.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286031+0000", 2023-07-02T14:04:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00049286700000000002 2023-07-02T14:04:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286524+0000", 2023-07-02T14:04:10.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4938e-05 2023-07-02T14:04:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286539+0000", 2023-07-02T14:04:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0680000000000002e-06 2023-07-02T14:04:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286545+0000", 2023-07-02T14:04:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026497718 2023-07-02T14:04:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313043+0000", 2023-07-02T14:04:10.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0884999999999998e-05 2023-07-02T14:04:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313094+0000", 2023-07-02T14:04:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014109999999999999 2023-07-02T14:04:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313235+0000", 2023-07-02T14:04:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048258980000000003 2023-07-02T14:04:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318061+0000", 2023-07-02T14:04:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014346350000000001 2023-07-02T14:04:10.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319495+0000", 2023-07-02T14:04:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1986e-05 2023-07-02T14:04:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319547+0000", 2023-07-02T14:04:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0457e-05 2023-07-02T14:04:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319578+0000", 2023-07-02T14:04:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999 2023-07-02T14:04:10.999 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:201 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:04:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.648280008, 2023-07-02T14:04:11.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003, 2023-07-02T14:04:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 201 2023-07-02T14:04:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.001 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:04:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:11.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:04:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:04:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019213999999999999 2023-07-02T14:04:11.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286744+0000", 2023-07-02T14:04:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5279999999999992e-06 2023-07-02T14:04:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:11.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286753+0000", 2023-07-02T14:04:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4200000000000004e-06 2023-07-02T14:04:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286760+0000", 2023-07-02T14:04:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026818386999999999 2023-07-02T14:04:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313578+0000", 2023-07-02T14:04:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015965 2023-07-02T14:04:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313738+0000", 2023-07-02T14:04:11.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015767299999999999 2023-07-02T14:04:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313895+0000", 2023-07-02T14:04:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041714930000000001 2023-07-02T14:04:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318067+0000", 2023-07-02T14:04:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001561698 2023-07-02T14:04:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319629+0000", 2023-07-02T14:04:11.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2346000000000003e-05 2023-07-02T14:04:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319681+0000", 2023-07-02T14:04:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4691000000000003e-05 2023-07-02T14:04:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319726+0000", 2023-07-02T14:04:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003 2023-07-02T14:04:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.010 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:204 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:04:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.648065169999995, 2023-07-02T14:04:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998, 2023-07-02T14:04:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:11.011 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 204 2023-07-02T14:04:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:04:11.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:04:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:04:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2497999999999998e-05 2023-07-02T14:04:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286819+0000", 2023-07-02T14:04:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0099999999999995e-06 2023-07-02T14:04:11.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286827+0000", 2023-07-02T14:04:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000003e-06 2023-07-02T14:04:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286833+0000", 2023-07-02T14:04:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027384716999999999 2023-07-02T14:04:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314218+0000", 2023-07-02T14:04:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6204000000000004e-05 2023-07-02T14:04:11.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314294+0000", 2023-07-02T14:04:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015757499999999999 2023-07-02T14:04:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:11.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314452+0000", 2023-07-02T14:04:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036215560000000002 2023-07-02T14:04:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318073+0000", 2023-07-02T14:04:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017023979999999999 2023-07-02T14:04:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319776+0000", 2023-07-02T14:04:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9561000000000002e-05 2023-07-02T14:04:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319825+0000", 2023-07-02T14:04:11.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0329e-05 2023-07-02T14:04:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319866+0000", 2023-07-02T14:04:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998 2023-07-02T14:04:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:11.020 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:207 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.647990524999997, 2023-07-02T14:04:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003, 2023-07-02T14:04:11.021 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 207 2023-07-02T14:04:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:11.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286841+0000", 2023-07-02T14:04:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00068375100000000002 2023-07-02T14:04:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:11.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287525+0000", 2023-07-02T14:04:11.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1601e-05 2023-07-02T14:04:11.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:11.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287537+0000", 2023-07-02T14:04:11.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6409999999999996e-06 2023-07-02T14:04:11.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287543+0000", 2023-07-02T14:04:11.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027245844000000002 2023-07-02T14:04:11.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:11.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314789+0000", 2023-07-02T14:04:11.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8910000000000003e-05 2023-07-02T14:04:11.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:11.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314858+0000", 2023-07-02T14:04:11.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030992599999999997 2023-07-02T14:04:11.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:11.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315168+0000", 2023-07-02T14:04:11.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029122869999999999 2023-07-02T14:04:11.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:11.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318080+0000", 2023-07-02T14:04:11.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019963020000000001 2023-07-02T14:04:11.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:11.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320077+0000", 2023-07-02T14:04:11.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5116000000000005e-05 2023-07-02T14:04:11.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:11.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320142+0000", 2023-07-02T14:04:11.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3793000000000004e-05 2023-07-02T14:04:11.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:11.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320205+0000", 2023-07-02T14:04:11.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003 2023-07-02T14:04:11.031 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.031 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:11.031 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.031 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:236 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2424832~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:11.031 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:11.032 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.647278342000007, 2023-07-02T14:04:11.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003, 2023-07-02T14:04:11.032 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:11.032 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.032 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:11.032 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:11.032 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.032 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2023-07-02T14:04:11.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.033 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:11.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:11.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:11.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:11.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:11.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:11.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:11.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:11.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287553+0000", 2023-07-02T14:04:11.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0667e-05 2023-07-02T14:04:11.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:11.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287614+0000", 2023-07-02T14:04:11.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0454e-05 2023-07-02T14:04:11.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:11.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287624+0000", 2023-07-02T14:04:11.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8564999999999999e-05 2023-07-02T14:04:11.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287643+0000", 2023-07-02T14:04:11.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027977596 2023-07-02T14:04:11.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:11.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315621+0000", 2023-07-02T14:04:11.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000201728 2023-07-02T14:04:11.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:11.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315822+0000", 2023-07-02T14:04:11.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018436599999999999 2023-07-02T14:04:11.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:11.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316007+0000", 2023-07-02T14:04:11.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042423440000000003 2023-07-02T14:04:11.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:11.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320249+0000", 2023-07-02T14:04:11.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001452633 2023-07-02T14:04:11.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:11.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321702+0000", 2023-07-02T14:04:11.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3616000000000003e-05 2023-07-02T14:04:11.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:11.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321755+0000", 2023-07-02T14:04:11.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840999999999999e-05 2023-07-02T14:04:11.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:11.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321793+0000", 2023-07-02T14:04:11.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003 2023-07-02T14:04:11.041 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.041 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:11.041 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.042 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:251 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:11.042 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:11.042 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.647182805, 2023-07-02T14:04:11.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001, 2023-07-02T14:04:11.042 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:11.042 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.043 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:11.043 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:11.043 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.043 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 251 2023-07-02T14:04:11.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.043 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:11.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:11.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:11.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:11.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:11.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:11.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:11.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:11.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287649+0000", 2023-07-02T14:04:11.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3232000000000001e-05 2023-07-02T14:04:11.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:11.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287702+0000", 2023-07-02T14:04:11.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0010000000000007e-06 2023-07-02T14:04:11.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:11.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287711+0000", 2023-07-02T14:04:11.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3569999999999997e-06 2023-07-02T14:04:11.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287717+0000", 2023-07-02T14:04:11.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028686454 2023-07-02T14:04:11.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:11.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316403+0000", 2023-07-02T14:04:11.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7741000000000002e-05 2023-07-02T14:04:11.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:11.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316461+0000", 2023-07-02T14:04:11.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4313000000000004e-05 2023-07-02T14:04:11.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:11.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316545+0000", 2023-07-02T14:04:11.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003734032 2023-07-02T14:04:11.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:11.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320279+0000", 2023-07-02T14:04:11.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015237219999999999 2023-07-02T14:04:11.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:11.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321803+0000", 2023-07-02T14:04:11.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1927e-05 2023-07-02T14:04:11.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:11.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321845+0000", 2023-07-02T14:04:11.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5398999999999998e-05 2023-07-02T14:04:11.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:11.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321880+0000", 2023-07-02T14:04:11.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001 2023-07-02T14:04:11.052 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.052 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:11.052 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.052 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:255 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2555904~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:11.052 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:11.052 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.646768077000004, 2023-07-02T14:04:11.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002, 2023-07-02T14:04:11.053 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:11.053 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.053 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:11.053 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:11.053 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.053 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 255 2023-07-02T14:04:11.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.054 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:11.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:11.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:11.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:11.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:11.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:11.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:11.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:11.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288063+0000", 2023-07-02T14:04:11.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4459999999999997e-05 2023-07-02T14:04:11.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:11.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288118+0000", 2023-07-02T14:04:11.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0998999999999999e-05 2023-07-02T14:04:11.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:11.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288129+0000", 2023-07-02T14:04:11.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3219999999999998e-06 2023-07-02T14:04:11.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288135+0000", 2023-07-02T14:04:11.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029137386000000001 2023-07-02T14:04:11.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:11.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317273+0000", 2023-07-02T14:04:11.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7819999999999997e-05 2023-07-02T14:04:11.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:11.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317350+0000", 2023-07-02T14:04:11.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000174864 2023-07-02T14:04:11.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:11.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317525+0000", 2023-07-02T14:04:11.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027943519999999999 2023-07-02T14:04:11.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:11.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320320+0000", 2023-07-02T14:04:11.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015681460000000001 2023-07-02T14:04:11.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:11.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321888+0000", 2023-07-02T14:04:11.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9508999999999997e-05 2023-07-02T14:04:11.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:11.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321927+0000", 2023-07-02T14:04:11.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5428e-05 2023-07-02T14:04:11.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:11.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321963+0000", 2023-07-02T14:04:11.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002 2023-07-02T14:04:11.062 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.062 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:11.062 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.063 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:446 2.1 2:910b2a90:::rbd_data.125de1449605.000000000000010a:head [write 65536~3211264] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:11.063 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:04:11.063 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.569488324999995, 2023-07-02T14:04:11.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003, 2023-07-02T14:04:11.063 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:11.063 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.063 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:11.063 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:11.064 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.064 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 446 2023-07-02T14:04:11.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.064 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:11.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:11.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:04:11.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:11.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:11.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:04:11.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:11.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:11.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:04:11.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015907372999999999 2023-07-02T14:04:11.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:11.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381251+0000", 2023-07-02T14:04:11.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5527000000000001e-05 2023-07-02T14:04:11.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:11.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381277+0000", 2023-07-02T14:04:11.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9958000000000001e-05 2023-07-02T14:04:11.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381297+0000", 2023-07-02T14:04:11.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4941000000000001e-05 2023-07-02T14:04:11.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:11.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381352+0000", 2023-07-02T14:04:11.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011565699999999999 2023-07-02T14:04:11.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:11.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381467+0000", 2023-07-02T14:04:11.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018039799999999999 2023-07-02T14:04:11.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:11.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381648+0000", 2023-07-02T14:04:11.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088183559999999994 2023-07-02T14:04:11.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:11.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.390466+0000", 2023-07-02T14:04:11.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007278278 2023-07-02T14:04:11.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:11.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397744+0000", 2023-07-02T14:04:11.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9289000000000001e-05 2023-07-02T14:04:11.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:11.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397804+0000", 2023-07-02T14:04:11.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0479000000000002e-05 2023-07-02T14:04:11.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:11.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397854+0000", 2023-07-02T14:04:11.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003 2023-07-02T14:04:11.072 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.072 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:11.073 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.073 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:04:11.073 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:11.073 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 69.827157530999997, 2023-07-02T14:04:11.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:04:11.073 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:11.074 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:11.074 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:11.074 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:11.074 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.074 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:04:11.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.074 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:11.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:11.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:11.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:11.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:11.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:11.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:11.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:11.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:04:11.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:04:11.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:11.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:04:11.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:04:11.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:11.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:04:11.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:04:11.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:04:11.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:04:11.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:11.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:04:11.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:04:11.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:11.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:04:11.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:04:11.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:11.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:04:11.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:04:11.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:11.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:04:11.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:04:11.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:11.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:04:11.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:04:11.082 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.082 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:11.082 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.082 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:04:11.082 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:11.082 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 69.827096792999996, 2023-07-02T14:04:11.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:04:11.083 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:11.083 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.083 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:11.083 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:11.083 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.083 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:04:11.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.084 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:11.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:11.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:11.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:11.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:11.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:11.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:11.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:11.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:04:11.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:04:11.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:11.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:04:11.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:04:11.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:11.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:04:11.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:04:11.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:04:11.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:04:11.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:11.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:04:11.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:04:11.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:11.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:04:11.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:04:11.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:11.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:04:11.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:04:11.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:11.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:04:11.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:04:11.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:11.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:04:11.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:04:11.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:11.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:04:11.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:04:11.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:11.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:04:11.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:04:11.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:11.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:04:11.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:04:11.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:11.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:04:11.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:04:11.093 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.093 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:11.093 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.596642463000002, 2023-07-02T14:04:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:04:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:11.094 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:04:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:11.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:11.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:11.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:11.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:11.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:11.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:11.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:11.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:04:11.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:04:11.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:11.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:04:11.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:04:11.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:11.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:04:11.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:04:11.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:04:11.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:04:11.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:11.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:04:11.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:04:11.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:11.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:04:11.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:04:11.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:11.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:04:11.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:04:11.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:11.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:04:11.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:04:11.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:11.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:04:11.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:04:11.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:11.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:04:11.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:04:11.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:11.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:04:11.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:04:11.103 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.104 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:11.104 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.104 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2941 2.3 2:d3762e63:::rbd_data.125de1449605.00000000000000f8:head [write 65536~2031616] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:11.104 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:11.104 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.467539508999998, 2023-07-02T14:04:11.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002, 2023-07-02T14:04:11.105 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:11.105 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.105 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:11.105 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:11.105 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.105 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2941 2023-07-02T14:04:11.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.106 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:11.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:11.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:11.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:11.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:11.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:11.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:11.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:11.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467291+0000", 2023-07-02T14:04:11.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043022859999999998 2023-07-02T14:04:11.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:11.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471593+0000", 2023-07-02T14:04:11.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0999999999999999e-05 2023-07-02T14:04:11.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:11.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471614+0000", 2023-07-02T14:04:11.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0097e-05 2023-07-02T14:04:11.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471625+0000", 2023-07-02T14:04:11.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069808499999999998 2023-07-02T14:04:11.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:11.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472323+0000", 2023-07-02T14:04:11.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135635 2023-07-02T14:04:11.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:11.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472458+0000", 2023-07-02T14:04:11.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223365 2023-07-02T14:04:11.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:11.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472682+0000", 2023-07-02T14:04:11.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013761944999999999 2023-07-02T14:04:11.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:11.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.486444+0000", 2023-07-02T14:04:11.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016859747000000001 2023-07-02T14:04:11.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:11.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503303+0000", 2023-07-02T14:04:11.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6635000000000001e-05 2023-07-02T14:04:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503380+0000", 2023-07-02T14:04:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000161803 2023-07-02T14:04:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503542+0000", 2023-07-02T14:04:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002 2023-07-02T14:04:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2950 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 1507328~786432] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.459366201000002, 2023-07-02T14:04:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999, 2023-07-02T14:04:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2950 2023-07-02T14:04:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:11.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:11.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001147854 2023-07-02T14:04:11.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:11.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476614+0000", 2023-07-02T14:04:11.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.802e-05 2023-07-02T14:04:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476632+0000", 2023-07-02T14:04:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1860000000000003e-06 2023-07-02T14:04:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476639+0000", 2023-07-02T14:04:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061211240000000004 2023-07-02T14:04:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482760+0000", 2023-07-02T14:04:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8833999999999996e-05 2023-07-02T14:04:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482859+0000", 2023-07-02T14:04:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019270600000000001 2023-07-02T14:04:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.483051+0000", 2023-07-02T14:04:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060385630000000003 2023-07-02T14:04:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489090+0000", 2023-07-02T14:04:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024506918999999999 2023-07-02T14:04:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513597+0000", 2023-07-02T14:04:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9672999999999995e-05 2023-07-02T14:04:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513667+0000", 2023-07-02T14:04:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024329500000000001 2023-07-02T14:04:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513910+0000", 2023-07-02T14:04:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999 2023-07-02T14:04:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2951 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2686976~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.458185423000003, 2023-07-02T14:04:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999, 2023-07-02T14:04:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2951 2023-07-02T14:04:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476646+0000", 2023-07-02T14:04:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7639000000000003e-05 2023-07-02T14:04:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476684+0000", 2023-07-02T14:04:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3508000000000002e-05 2023-07-02T14:04:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476747+0000", 2023-07-02T14:04:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8499999999999994e-06 2023-07-02T14:04:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476755+0000", 2023-07-02T14:04:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078346160000000008 2023-07-02T14:04:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484590+0000", 2023-07-02T14:04:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7206000000000004e-05 2023-07-02T14:04:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484687+0000", 2023-07-02T14:04:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016243099999999999 2023-07-02T14:04:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484849+0000", 2023-07-02T14:04:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042482860000000004 2023-07-02T14:04:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489097+0000", 2023-07-02T14:04:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.02489504 2023-07-02T14:04:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513992+0000", 2023-07-02T14:04:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9972000000000001e-05 2023-07-02T14:04:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514052+0000", 2023-07-02T14:04:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149264 2023-07-02T14:04:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514202+0000", 2023-07-02T14:04:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999 2023-07-02T14:04:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2952 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2883584~12288] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.458071588000003, 2023-07-02T14:04:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651, 2023-07-02T14:04:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2952 2023-07-02T14:04:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476760+0000", 2023-07-02T14:04:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3664e-05 2023-07-02T14:04:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476774+0000", 2023-07-02T14:04:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.166e-06 2023-07-02T14:04:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476783+0000", 2023-07-02T14:04:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8720000000000001e-06 2023-07-02T14:04:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476788+0000", 2023-07-02T14:04:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087180529999999999 2023-07-02T14:04:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485506+0000", 2023-07-02T14:04:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5680999999999994e-05 2023-07-02T14:04:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485602+0000", 2023-07-02T14:04:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000222365 2023-07-02T14:04:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485824+0000", 2023-07-02T14:04:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003277379 2023-07-02T14:04:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489101+0000", 2023-07-02T14:04:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025151739999999999 2023-07-02T14:04:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514253+0000", 2023-07-02T14:04:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4749000000000002e-05 2023-07-02T14:04:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514288+0000", 2023-07-02T14:04:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130634 2023-07-02T14:04:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514419+0000", 2023-07-02T14:04:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651 2023-07-02T14:04:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:11.146 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:11.147 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:04:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:04:11.148 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:11.146+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:11.247 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:11.247+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:11.286 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:11.286 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:11.286 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:04:11.286 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:11.296 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:04:11.380 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:11.379+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:11.450 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:11.449+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:11.461 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:11.461 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:11.461 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:04:11.462 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:04:11.462 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:11.472 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:04:11.555 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:11.554+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:11.637 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:11.638 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:04:11.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:04:11.638 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:11.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.638 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:04:11.638 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:11.638 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.738459777999999, 2023-07-02T14:04:11.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:04:11.639 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:11.639 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.639 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:11.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:11.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:11.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:11.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:11.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:11.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:11.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:11.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:04:11.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:04:11.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:11.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:04:11.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:04:11.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:11.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:04:11.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:04:11.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:04:11.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:04:11.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:11.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:04:11.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:04:11.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:11.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:04:11.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:04:11.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:11.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:04:11.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:04:11.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:11.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:04:11.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:04:11.646 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.646 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.646 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.646 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:04:11.646 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:11.646 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.73437406, 2023-07-02T14:04:11.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:04:11.647 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:11.647 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.647 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:11.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:11.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:11.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:11.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:11.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:11.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:11.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:11.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:04:11.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:04:11.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:11.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:04:11.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:04:11.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:11.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:04:11.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:04:11.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:04:11.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:04:11.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:11.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:04:11.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:04:11.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:11.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:04:11.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:04:11.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:11.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:04:11.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:04:11.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:11.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:04:11.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:04:11.653 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.654 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.654 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.654 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:04:11.654 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:11.654 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.730509279, 2023-07-02T14:04:11.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:04:11.655 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:11.655 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.655 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:11.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:11.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:11.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:11.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:11.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:11.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:11.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:11.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:04:11.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:04:11.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:11.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:04:11.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:04:11.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:11.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:04:11.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:04:11.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:04:11.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:04:11.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:11.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:04:11.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:04:11.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:11.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:04:11.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:04:11.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:11.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:04:11.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:04:11.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:11.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:04:11.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:04:11.661 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.662 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.662 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.662 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:04:11.662 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:11.662 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.726985018000001, 2023-07-02T14:04:11.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:04:11.663 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:11.663 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.663 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:11.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:11.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:11.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:11.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:11.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:11.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:11.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:11.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:04:11.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:04:11.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:11.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:04:11.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:04:11.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:11.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:04:11.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:04:11.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:04:11.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:04:11.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:11.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:04:11.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:04:11.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:11.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:04:11.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:04:11.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:11.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:04:11.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:04:11.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:11.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:04:11.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:04:11.669 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.670 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.670 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.670 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6124 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1328, mlcod=81'1319)", 2023-07-02T14:04:11.670 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:11.670 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.723469635000001, 2023-07-02T14:04:11.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019096974999999999, 2023-07-02T14:04:11.671 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:11.671 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.671 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:11.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:11.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:11.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:11.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:11.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:11.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:11.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:11.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912602+0000", 2023-07-02T14:04:11.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6741999999999999e-05 2023-07-02T14:04:11.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:11.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912629+0000", 2023-07-02T14:04:11.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5163e-05 2023-07-02T14:04:11.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:11.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912644+0000", 2023-07-02T14:04:11.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4800000000000001e-06 2023-07-02T14:04:11.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912653+0000", 2023-07-02T14:04:11.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071988909999999998 2023-07-02T14:04:11.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:11.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.919851+0000", 2023-07-02T14:04:11.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2949999999999999e-05 2023-07-02T14:04:11.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:11.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.919874+0000", 2023-07-02T14:04:11.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011800804 2023-07-02T14:04:11.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:11.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931675+0000", 2023-07-02T14:04:11.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6033999999999998e-05 2023-07-02T14:04:11.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:11.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931701+0000", 2023-07-02T14:04:11.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019096974999999999 2023-07-02T14:04:11.677 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.677 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.678 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.678 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6139 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1337, mlcod=81'1333)", 2023-07-02T14:04:11.678 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:11.678 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.696779775, 2023-07-02T14:04:11.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018571681, 2023-07-02T14:04:11.678 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:11.679 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.679 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:11.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:11.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:11.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:11.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:11.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:11.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:11.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:11.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939293+0000", 2023-07-02T14:04:11.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003764613 2023-07-02T14:04:11.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:11.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943058+0000", 2023-07-02T14:04:11.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5797000000000001e-05 2023-07-02T14:04:11.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:11.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943074+0000", 2023-07-02T14:04:11.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5089999999999996e-06 2023-07-02T14:04:11.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943081+0000", 2023-07-02T14:04:11.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014727100000000001 2023-07-02T14:04:11.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:11.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943228+0000", 2023-07-02T14:04:11.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2246000000000003e-05 2023-07-02T14:04:11.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:11.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943261+0000", 2023-07-02T14:04:11.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014576317 2023-07-02T14:04:11.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:11.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957837+0000", 2023-07-02T14:04:11.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8962000000000002e-05 2023-07-02T14:04:11.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:11.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957866+0000", 2023-07-02T14:04:11.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018571681 2023-07-02T14:04:11.685 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.685 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.685 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.686 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:04:11.686 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:11.686 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.691280218999999, 2023-07-02T14:04:11.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:04:11.686 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:11.687 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.687 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:11.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:11.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:11.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:11.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:11.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:11.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:11.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:11.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:04:11.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:04:11.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:11.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:04:11.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:04:11.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:11.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:04:11.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:04:11.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:04:11.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:04:11.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:11.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:04:11.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:04:11.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:11.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:04:11.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:04:11.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:11.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:04:11.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:04:11.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:11.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:04:11.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:04:11.693 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.693 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.693 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.694 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:04:11.694 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:11.694 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.686622861, 2023-07-02T14:04:11.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:04:11.694 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:11.694 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.695 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:11.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:11.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:11.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:11.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:11.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:11.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:11.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:11.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:04:11.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:04:11.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:11.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:04:11.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:04:11.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:11.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:04:11.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:04:11.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:04:11.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:04:11.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:11.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:04:11.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:04:11.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:11.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:04:11.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:04:11.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:11.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:04:11.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:04:11.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:11.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:04:11.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:04:11.701 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.701 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.701 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.702 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:04:11.702 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:11.702 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.685682998000001, 2023-07-02T14:04:11.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:04:11.702 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:11.702 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.703 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:11.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:11.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:11.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:11.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:11.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:11.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:11.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:11.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:04:11.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:04:11.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:11.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:04:11.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:04:11.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:11.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:04:11.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:04:11.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:04:11.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:04:11.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:11.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:04:11.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:04:11.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:11.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:04:11.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:04:11.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:11.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:04:11.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:04:11.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:11.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:04:11.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:04:11.709 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.709 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.709 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.710 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:04:11.710 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:11.710 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.684285119999998, 2023-07-02T14:04:11.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:04:11.710 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:11.710 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.710 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:11.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:11.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:11.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:11.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:11.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:11.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:11.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:11.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:04:11.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:04:11.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:11.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:04:11.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:04:11.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:11.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:04:11.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:04:11.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:04:11.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:04:11.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:11.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:04:11.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:04:11.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:11.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:04:11.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:04:11.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:11.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:04:11.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:04:11.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:11.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:04:11.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:04:11.717 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.717 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.717 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.718 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6150 2.0 e81/46 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 81'1345, mlcod=81'1338)", 2023-07-02T14:04:11.718 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:11.718 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.676262173000001, 2023-07-02T14:04:11.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019172812000000001, 2023-07-02T14:04:11.718 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:11.718 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.718 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:11.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:11.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:11.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:11.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:11.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:11.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:11.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:11.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959810+0000", 2023-07-02T14:04:11.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048438650000000001 2023-07-02T14:04:11.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:11.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964653+0000", 2023-07-02T14:04:11.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2136e-05 2023-07-02T14:04:11.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:11.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964666+0000", 2023-07-02T14:04:11.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2169999999999999e-06 2023-07-02T14:04:11.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964672+0000", 2023-07-02T14:04:11.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00047561800000000003 2023-07-02T14:04:11.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:11.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965147+0000", 2023-07-02T14:04:11.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5891e-05 2023-07-02T14:04:11.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:11.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965173+0000", 2023-07-02T14:04:11.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013770973000000001 2023-07-02T14:04:11.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:11.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978944+0000", 2023-07-02T14:04:11.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0256999999999998e-05 2023-07-02T14:04:11.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:11.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978985+0000", 2023-07-02T14:04:11.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019172812000000001 2023-07-02T14:04:11.725 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.725 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.725 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.725 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:04:11.726 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:11.726 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 10.274094624, 2023-07-02T14:04:11.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:04:11.726 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:11.726 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.726 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:11.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:11.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:11.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:11.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:11.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:11.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:11.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:11.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:04:11.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:04:11.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:11.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:04:11.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:04:11.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:11.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:04:11.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:04:11.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:04:11.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:04:11.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:11.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:04:11.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:04:11.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:11.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:04:11.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:04:11.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:11.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:04:11.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:04:11.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:11.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:04:11.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:04:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.9077449400000006, 2023-07-02T14:04:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:04:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.734 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:04:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:11.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:11.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:11.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:11.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:04:11.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:04:11.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:11.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:04:11.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:04:11.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:11.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:04:11.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:04:11.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:04:11.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:04:11.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:11.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:04:11.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:04:11.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:11.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:04:11.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:04:11.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:11.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:04:11.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:04:11.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:11.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:04:11.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:04:11.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:11.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:04:11.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:04:11.742 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.742 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.742 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.742 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7706 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 0~53248] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:11.743 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:11.743 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.9038181929999993, 2023-07-02T14:04:11.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019507096000000002, 2023-07-02T14:04:11.743 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:11.743 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.743 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:11.743 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:11.743 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.744 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7706 2023-07-02T14:04:11.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.744 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:11.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:11.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:11.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:11.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:11.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:11.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:11.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:11.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732254+0000", 2023-07-02T14:04:11.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4764999999999997e-05 2023-07-02T14:04:11.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:11.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732289+0000", 2023-07-02T14:04:11.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0166999999999999e-05 2023-07-02T14:04:11.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:11.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732299+0000", 2023-07-02T14:04:11.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5310000000000001e-06 2023-07-02T14:04:11.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732305+0000", 2023-07-02T14:04:11.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060619599999999999 2023-07-02T14:04:11.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:11.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738367+0000", 2023-07-02T14:04:11.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011864 2023-07-02T14:04:11.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:11.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738486+0000", 2023-07-02T14:04:11.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013098291 2023-07-02T14:04:11.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:11.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751584+0000", 2023-07-02T14:04:11.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7800000000000002e-05 2023-07-02T14:04:11.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:11.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751642+0000", 2023-07-02T14:04:11.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000121198 2023-07-02T14:04:11.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:11.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751763+0000", 2023-07-02T14:04:11.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019507096000000002 2023-07-02T14:04:11.751 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.751 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.751 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.752 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7707 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 3145728~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:11.752 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:11.752 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.9037614729999994, 2023-07-02T14:04:11.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999, 2023-07-02T14:04:11.752 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:11.752 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.752 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:11.752 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:11.753 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.753 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7707 2023-07-02T14:04:11.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.753 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:11.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:11.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:11.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:11.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:11.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:11.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:11.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:11.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:11.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5017e-05 2023-07-02T14:04:11.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:11.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732347+0000", 2023-07-02T14:04:11.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5821e-05 2023-07-02T14:04:11.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:11.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732363+0000", 2023-07-02T14:04:11.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1270000000000002e-06 2023-07-02T14:04:11.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732368+0000", 2023-07-02T14:04:11.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008444666 2023-07-02T14:04:11.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:11.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740812+0000", 2023-07-02T14:04:11.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011266599999999999 2023-07-02T14:04:11.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:11.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740925+0000", 2023-07-02T14:04:11.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010848808999999999 2023-07-02T14:04:11.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:11.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751774+0000", 2023-07-02T14:04:11.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8421999999999997e-05 2023-07-02T14:04:11.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:11.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751832+0000", 2023-07-02T14:04:11.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107453 2023-07-02T14:04:11.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:11.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751940+0000", 2023-07-02T14:04:11.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999 2023-07-02T14:04:11.760 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.760 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.760 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.761 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:11.761 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:11.761 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.9036975960000007, 2023-07-02T14:04:11.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:04:11.761 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:11.761 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.761 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:11.762 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:11.762 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.762 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:04:11.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.762 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:11.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:11.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:11.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:11.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:11.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:11.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:11.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:11.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:11.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:04:11.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:11.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:04:11.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:04:11.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:11.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:04:11.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:04:11.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:04:11.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:04:11.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:11.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:04:11.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:04:11.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:11.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:04:11.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:04:11.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:11.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:04:11.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:04:11.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:11.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:04:11.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:04:11.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:11.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:04:11.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:04:11.769 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.769 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.769 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.770 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:11.770 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:11.770 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.8991337710000007, 2023-07-02T14:04:11.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:04:11.770 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:11.771 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.771 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:11.771 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:11.771 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.771 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:04:11.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.771 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:11.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:11.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:11.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:11.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:11.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:11.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:11.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:11.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:04:11.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:04:11.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:11.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:04:11.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:04:11.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:11.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:04:11.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:04:11.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:04:11.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:04:11.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:11.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:04:11.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:04:11.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:11.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:04:11.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:04:11.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:11.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:04:11.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:04:11.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:11.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:04:11.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:04:11.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:11.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:04:11.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:04:11.778 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.778 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.779 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.779 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:11.779 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:11.779 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.8990839049999995, 2023-07-02T14:04:11.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:04:11.779 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:11.780 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.780 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:11.780 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:11.780 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.780 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:04:11.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.780 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:11.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:11.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:11.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:11.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:11.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:11.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:11.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:11.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:04:11.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:04:11.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:11.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:04:11.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:04:11.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:11.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:04:11.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:04:11.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:04:11.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:04:11.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:11.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:04:11.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:04:11.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:11.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:04:11.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:04:11.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:11.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:04:11.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:04:11.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:11.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:04:11.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:04:11.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:11.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:04:11.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:04:11.788 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.788 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.788 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.788 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:11.788 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:11.788 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.8980449119999996, 2023-07-02T14:04:11.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:04:11.789 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:11.789 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.789 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:11.789 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:11.789 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.789 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:04:11.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.790 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:11.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:11.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:11.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:11.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:11.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:11.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:11.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:11.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:04:11.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:04:11.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:11.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:04:11.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:04:11.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:11.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:04:11.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:04:11.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:04:11.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:04:11.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:11.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:04:11.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:04:11.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:11.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:04:11.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:04:11.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:11.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:04:11.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:04:11.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:11.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:04:11.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:04:11.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:11.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:04:11.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:04:11.797 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.797 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.797 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.797 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7993 2.7 2:fe72db53:::rbd_data.125de1449605.0000000000000072:head [write 65536~3276800] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:11.797 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:11.798 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7110868640000003, 2023-07-02T14:04:11.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919, 2023-07-02T14:04:11.798 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:11.798 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:11.798 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:11.798 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:11.798 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:11.798 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7993 2023-07-02T14:04:11.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.799 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:11.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:11.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:11.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:11.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:11.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:11.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:11.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:11.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924986+0000", 2023-07-02T14:04:11.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049542070000000004 2023-07-02T14:04:11.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:11.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929940+0000", 2023-07-02T14:04:11.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9740000000000001e-05 2023-07-02T14:04:11.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:11.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929960+0000", 2023-07-02T14:04:11.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.311e-05 2023-07-02T14:04:11.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:11.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929973+0000", 2023-07-02T14:04:11.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9629000000000002e-05 2023-07-02T14:04:11.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:11.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930023+0000", 2023-07-02T14:04:11.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012751799999999999 2023-07-02T14:04:11.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:11.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930150+0000", 2023-07-02T14:04:11.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016449042000000001 2023-07-02T14:04:11.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:11.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946599+0000", 2023-07-02T14:04:11.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3241e-05 2023-07-02T14:04:11.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:11.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946663+0000", 2023-07-02T14:04:11.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000102296 2023-07-02T14:04:11.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:11.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:11.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:11.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946765+0000", 2023-07-02T14:04:11.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919 2023-07-02T14:04:11.806 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.806 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.806 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.806 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:11.806 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:11.806 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:11.808 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:04:11.808 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:11.807+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:11.908 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:11.907+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:11.972 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:11.972 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:11.973 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:04:11.973 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:11.983 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:04:12.026 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:12.025+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:12.075 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:04:12.106 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:12.107 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:12.107 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:04:12.107 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:04:12.107 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:12.117 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:12.119 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:12.118+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:12.209 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:12.209+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:12.228 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:12.246 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:12.263 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:12.282 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:12.283 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:12.283 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:04:12.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:04:12.283 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:12.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.284 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:12.284 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:12.284 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 92.990692288000005, 2023-07-02T14:04:12.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:04:12.284 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:12.284 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:12.285 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:12.285 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:12.285 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:12.285 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:04:12.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.285 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:12.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:12.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:12.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:12.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:12.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:12.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:12.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:12.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:04:12.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:04:12.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:12.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:04:12.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:04:12.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:12.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:04:12.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:04:12.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:12.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:04:12.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:04:12.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:12.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:04:12.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:04:12.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:12.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:04:12.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:04:12.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:12.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:04:12.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:04:12.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:12.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:04:12.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:04:12.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:12.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:04:12.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:04:12.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:12.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:04:12.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:04:12.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:12.294 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:04:12.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:04:12.294 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.294 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.294 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.295 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:12.295 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:12.295 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 92.990457265000003, 2023-07-02T14:04:12.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:04:12.295 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:12.295 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:12.295 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:12.296 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:12.296 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:12.296 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:04:12.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.296 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:12.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:12.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:12.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:12.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:12.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:12.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:12.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:12.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:04:12.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:04:12.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:12.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:04:12.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:04:12.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:12.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:04:12.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:04:12.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:12.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:04:12.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:04:12.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:12.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:04:12.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:04:12.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:12.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:04:12.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:04:12.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:12.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:04:12.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:04:12.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:12.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:04:12.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:04:12.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:12.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:04:12.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:04:12.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:12.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:04:12.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:04:12.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:12.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:04:12.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:04:12.305 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.305 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.305 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.305 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:12.305 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:12.306 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 92.990258131000004, 2023-07-02T14:04:12.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:04:12.306 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:12.306 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:12.306 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:12.306 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:12.306 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:12.306 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:04:12.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.307 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:12.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:12.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:12.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:12.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:12.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:12.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:12.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:12.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:04:12.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:04:12.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:12.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:04:12.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:04:12.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:12.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:04:12.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:04:12.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:12.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:04:12.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:04:12.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:12.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:04:12.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:04:12.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:12.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:04:12.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:04:12.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:12.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:04:12.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:04:12.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:12.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:04:12.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:04:12.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:12.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:04:12.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:04:12.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:12.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:04:12.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:04:12.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:12.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:04:12.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:04:12.317 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.317 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.317 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.318 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:12.318 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:12.318 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 92.989893007000006, 2023-07-02T14:04:12.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:04:12.318 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:12.318 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:12.319 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:12.319 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:12.319 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:12.319 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:04:12.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.319 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:12.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:12.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:12.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:12.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:12.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:12.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:12.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:12.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:04:12.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:04:12.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:12.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:04:12.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:04:12.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.322 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:12.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:04:12.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:04:12.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:12.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:04:12.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:04:12.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.324 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:12.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:04:12.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:04:12.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:12.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:04:12.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:04:12.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:12.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:04:12.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:04:12.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:12.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:04:12.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:04:12.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:12.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:04:12.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:04:12.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:12.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:04:12.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:04:12.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:12.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:04:12.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:04:12.328 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.328 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.329 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.329 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:12.329 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:12.329 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 92.989811946000003, 2023-07-02T14:04:12.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:04:12.329 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:12.330 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:12.330 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:12.330 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:12.330 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:12.330 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:04:12.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.330 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:12.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:12.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:12.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:12.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:12.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:12.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:12.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:12.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:04:12.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:04:12.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:12.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:04:12.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:04:12.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:12.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:04:12.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:04:12.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:12.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:04:12.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:04:12.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:12.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:04:12.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:04:12.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:12.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:04:12.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:04:12.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:12.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:04:12.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:04:12.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:12.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:04:12.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:04:12.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:12.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:04:12.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:04:12.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:12.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:04:12.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:04:12.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:12.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:04:12.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:04:12.339 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.339 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.339 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.340 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:12.340 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:12.340 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 92.989720274999996, 2023-07-02T14:04:12.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:04:12.340 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:12.340 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:12.340 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:12.341 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:12.341 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:12.341 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:04:12.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.341 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:12.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:12.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:12.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:12.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:12.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:12.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:12.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:12.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:04:12.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:04:12.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:12.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:04:12.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:04:12.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:12.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:04:12.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:04:12.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:12.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:04:12.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:04:12.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:12.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:04:12.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:04:12.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:12.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:04:12.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:04:12.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:12.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:04:12.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:04:12.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:12.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:04:12.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:04:12.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:12.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:04:12.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:04:12.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:12.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:04:12.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:04:12.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:12.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:04:12.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:04:12.349 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.350 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.350 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.350 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:12.350 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:12.350 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 92.989380832999998, 2023-07-02T14:04:12.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:04:12.351 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:12.351 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:12.351 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:12.351 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:12.351 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:12.351 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:04:12.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.352 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:12.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:12.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:12.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:12.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:12.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:12.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:12.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:12.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:04:12.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:04:12.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:12.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:04:12.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:04:12.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:12.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:04:12.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:04:12.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:12.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:04:12.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:04:12.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:12.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:04:12.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:04:12.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:12.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:04:12.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:04:12.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:12.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:04:12.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:04:12.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:12.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:04:12.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:04:12.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:12.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:04:12.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:04:12.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:12.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:04:12.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:04:12.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:12.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:04:12.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:04:12.360 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.360 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.360 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.361 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:12.361 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:12.361 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 92.989194780000005, 2023-07-02T14:04:12.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:04:12.361 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:12.361 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:12.361 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:12.362 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:12.362 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:12.362 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:04:12.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.362 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:12.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:12.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:12.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:12.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:12.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:12.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:12.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:12.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:12.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:04:12.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:12.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:04:12.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:04:12.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:12.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:04:12.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:04:12.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:12.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:04:12.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:04:12.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:12.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:04:12.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:04:12.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:12.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:04:12.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:04:12.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:12.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:04:12.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:04:12.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:12.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:04:12.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:04:12.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:12.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:04:12.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:04:12.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:12.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:04:12.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:04:12.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:12.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:04:12.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:04:12.370 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.371 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.371 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.371 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:12.371 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:12.371 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 92.989036978000001, 2023-07-02T14:04:12.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:04:12.372 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:12.372 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:12.372 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:12.372 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:12.372 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:12.372 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:04:12.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.372 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:12.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:12.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:12.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:12.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:12.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:12.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:12.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:12.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:04:12.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:04:12.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:12.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:04:12.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:04:12.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:12.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:04:12.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:04:12.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:12.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:04:12.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:04:12.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:12.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:04:12.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:04:12.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:12.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:04:12.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:04:12.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:12.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:04:12.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:04:12.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:12.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:04:12.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:04:12.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:12.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:04:12.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:04:12.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:12.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:04:12.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:04:12.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:12.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:04:12.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:04:12.381 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.381 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.381 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.382 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:12.382 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:12.382 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 92.988845196, 2023-07-02T14:04:12.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:04:12.382 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:12.382 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:12.382 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:12.382 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:12.383 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:12.383 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:04:12.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.383 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:12.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:12.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:12.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:12.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:12.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:12.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:12.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:12.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:04:12.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:04:12.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:12.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:04:12.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:04:12.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:12.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:04:12.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:04:12.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:12.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:04:12.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:04:12.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:12.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:04:12.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:04:12.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:12.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:04:12.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:04:12.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:12.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:04:12.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:04:12.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:12.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:04:12.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:04:12.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:12.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:04:12.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:04:12.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:12.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:04:12.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:04:12.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:12.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:04:12.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:04:12.392 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.392 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.392 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.393 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:12.393 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:12.393 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 92.988669536000003, 2023-07-02T14:04:12.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:04:12.393 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:12.393 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:12.393 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:12.394 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:12.394 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:12.394 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:04:12.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.394 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:12.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:12.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:12.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:12.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:12.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:12.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:12.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:12.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:12.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:04:12.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:12.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:04:12.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:04:12.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:12.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:04:12.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:04:12.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:12.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:04:12.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:04:12.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:12.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:04:12.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:04:12.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:12.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:04:12.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:04:12.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:12.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:04:12.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:04:12.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:12.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:04:12.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:04:12.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:12.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:04:12.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:04:12.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:12.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:04:12.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:04:12.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:12.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:04:12.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:04:12.403 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.403 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.403 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.404 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:12.404 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:12.404 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 92.988575347999998, 2023-07-02T14:04:12.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:04:12.404 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:12.404 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:12.404 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:12.404 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:12.405 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:12.405 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:04:12.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.405 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:12.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:12.405 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:12.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:12.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:12.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:12.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:12.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:12.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:04:12.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:04:12.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:12.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:04:12.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:04:12.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:12.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:04:12.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:04:12.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:12.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:04:12.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:04:12.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:12.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:04:12.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:04:12.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:12.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:04:12.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:04:12.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:12.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:04:12.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:04:12.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:12.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:04:12.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:04:12.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:12.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:04:12.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:04:12.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:12.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:04:12.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:04:12.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:12.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:04:12.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:04:12.414 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.414 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.414 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.414 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:12.415 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:12.415 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 92.988371962000002, 2023-07-02T14:04:12.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:04:12.415 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:12.415 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:12.415 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:12.415 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:12.416 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:12.416 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:04:12.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.416 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:12.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:12.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:12.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:12.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:12.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:12.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:12.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:12.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:04:12.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:04:12.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:12.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:04:12.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:04:12.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:12.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:04:12.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:04:12.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:12.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:04:12.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:04:12.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:12.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:04:12.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:04:12.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:12.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:04:12.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:04:12.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:12.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:04:12.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:04:12.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:12.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:04:12.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:04:12.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:12.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:04:12.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:04:12.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:12.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:04:12.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:04:12.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:12.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:04:12.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:04:12.424 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.424 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.425 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.425 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:12.425 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:12.425 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 92.988221306, 2023-07-02T14:04:12.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:04:12.426 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:12.426 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:12.426 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:12.426 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:12.426 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:12.426 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:04:12.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.427 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:12.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:12.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:12.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:12.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:12.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:12.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:12.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:12.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:04:12.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:04:12.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:12.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:04:12.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:04:12.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:12.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:04:12.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:04:12.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:12.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:04:12.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:04:12.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:12.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:04:12.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:04:12.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:12.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:04:12.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:04:12.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:12.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:04:12.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:04:12.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:12.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:04:12.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:04:12.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:12.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:04:12.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:04:12.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:12.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:04:12.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:04:12.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:12.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:04:12.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:04:12.435 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.435 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.435 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.436 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:12.436 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:12.436 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 92.988148602999999, 2023-07-02T14:04:12.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:04:12.436 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:12.436 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:12.437 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:12.437 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:12.437 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:12.437 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:04:12.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.437 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:12.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:12.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:12.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:12.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:12.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:12.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:12.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:12.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:12.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:04:12.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:12.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:04:12.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:04:12.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:12.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:04:12.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:04:12.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:12.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:04:12.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:04:12.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:12.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:04:12.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:04:12.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:12.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:04:12.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:04:12.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:12.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:04:12.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:04:12.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:12.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:04:12.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:04:12.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:12.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:04:12.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:04:12.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:12.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:04:12.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:04:12.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:12.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:04:12.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:04:12.446 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.446 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.446 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.446 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:12.446 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:12.447 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 92.988031613000004, 2023-07-02T14:04:12.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:04:12.447 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:12.447 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:12.447 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:12.447 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:12.447 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:12.447 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:04:12.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.448 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:12.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:12.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:12.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:12.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:12.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:12.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:12.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:12.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:12.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:04:12.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:12.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:04:12.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:04:12.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:12.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:04:12.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:04:12.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:12.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:04:12.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:04:12.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:12.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:04:12.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:04:12.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:12.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:04:12.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:04:12.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:12.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:04:12.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:04:12.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:12.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:04:12.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:04:12.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:12.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:04:12.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:04:12.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:12.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:04:12.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:04:12.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:12.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:04:12.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:04:12.456 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.456 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.456 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.457 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:12.457 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:12.457 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 92.987882162000005, 2023-07-02T14:04:12.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:04:12.457 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:12.457 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:12.458 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:12.458 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:12.458 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:12.458 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:04:12.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.458 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:12.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:12.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:12.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:12.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:12.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:12.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:12.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:12.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:04:12.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:04:12.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:12.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:04:12.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:04:12.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:12.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:04:12.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:04:12.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:12.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:04:12.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:04:12.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:12.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:04:12.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:04:12.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:12.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:04:12.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:04:12.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:12.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:04:12.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:04:12.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:12.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:04:12.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:04:12.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:12.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:04:12.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:04:12.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:12.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:04:12.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:04:12.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:12.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:04:12.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:04:12.467 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.467 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.467 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.467 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:12.468 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:12.468 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 92.966406188999997, 2023-07-02T14:04:12.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:04:12.468 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:12.468 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:12.468 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:12.468 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:12.468 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:12.469 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:04:12.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.469 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:12.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:12.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:12.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:12.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:12.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:12.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:12.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:12.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:04:12.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:04:12.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:12.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:04:12.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:04:12.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:12.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:04:12.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:04:12.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:12.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:04:12.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:04:12.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:12.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:04:12.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:04:12.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:12.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:04:12.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:04:12.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:12.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:04:12.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:04:12.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:12.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:04:12.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:04:12.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:12.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:04:12.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:04:12.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:12.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:04:12.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:04:12.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:12.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:04:12.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:04:12.477 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.477 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.478 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.478 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:12.478 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:12.478 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 92.966273653000002, 2023-07-02T14:04:12.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:04:12.478 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:12.479 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:12.479 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:12.479 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:12.479 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:12.479 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:04:12.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.479 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:12.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:12.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:12.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:12.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:12.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:12.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:12.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:12.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:04:12.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:04:12.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:12.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:04:12.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:04:12.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:12.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:04:12.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:04:12.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:12.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:04:12.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:04:12.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:12.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:04:12.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:04:12.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:12.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:04:12.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:04:12.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:12.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:04:12.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:04:12.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:12.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:04:12.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:04:12.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:12.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:04:12.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:04:12.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:12.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:04:12.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:04:12.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:12.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:04:12.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:04:12.488 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.488 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.488 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.488 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:12.489 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:12.489 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 92.904936992000003, 2023-07-02T14:04:12.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:04:12.489 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:12.489 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:12.489 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:12.489 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:12.489 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:12.490 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:04:12.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.490 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:12.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:12.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:12.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:12.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:12.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:12.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:12.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:12.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:04:12.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:04:12.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:12.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:04:12.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:04:12.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:12.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:04:12.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:04:12.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:12.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:04:12.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:04:12.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:12.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:04:12.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:04:12.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:12.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:04:12.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:04:12.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:12.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:04:12.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:04:12.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:12.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:04:12.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:04:12.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:12.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:04:12.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:04:12.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:12.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:04:12.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:04:12.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:12.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:12.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:12.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:04:12.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:04:12.498 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.498 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.498 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.499 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:12.499 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:12.499 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:12.500 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:12.501 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:12.501 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:12.499+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:12.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:04:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:04:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:04:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:04:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:04:12.503 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:12.600 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:12.599+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:12.667 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:12.667 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:12.678 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:04:12.719 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:12.718+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:12.802 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:12.802+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:12.841 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:12.842 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:12.852 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:04:12.903 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:12.902+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:12.975 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:12.975 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:12.986 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:04:13.029 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:13.028+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:13.103 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:13.102+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:13.110 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap17 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap18 b3fbb028-e267-4d25-a750-93ba18a2316a@snap18.diff 2023-07-02T14:04:13.111 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:13.111 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:13.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:04:13.149 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:04:13.162 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete...2023-07-02T14:04:13.161+0000 7fd909a7e700 0 -- 172.21.15.163:0/1058316775 >> [v2:172.21.15.22:6800/83351,v1:172.21.15.22:6801/83351] conn(0x7fd8f0008780 msgr2=0x7fd8f000abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:13.163 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:13.163+0000 7fd909a7e700 0 -- 172.21.15.163:0/1058316775 >> [v2:172.21.15.22:6800/83351,v1:172.21.15.22:6801/83351] conn(0x7fd8ec036bb0 msgr2=0x7fd8ec038fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:13.207 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:13.205+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:13.305 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:13.304+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:13.338 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:13.338 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:13.359 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:04:13.443 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:13.442+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:13.484 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4] out_osds: [5, 2] dead_osds: [3] live_osds: [0, 1, 2, 4, 5] 2023-07-02T14:04:13.484 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:04:13.484 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2023-07-02T14:04:13.485 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-07-02T14:04:13.485 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-07-02T14:04:13.485 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2023-07-02T14:04:13.529 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:13.528+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:13.533 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:13.533 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:13.543 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:04:13.607 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:13.606+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:13.620 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:13.620 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-07-02T14:04:13.620 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:13.656 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 34% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 100% complete...done. 2023-07-02T14:04:13.662 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:04:13.663 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=19 2023-07-02T14:04:13.663 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap19 2023-07-02T14:04:13.708 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:13.707+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:13.714 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:13.714 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:13.724 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:04:13.809 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:13.808+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:13.847 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:13.847 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:13.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:04:13.910 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:13.910+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:14.010 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:14.009+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:14.020 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:14.021 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:14.031 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:04:14.090 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:04:14.111 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:14.110+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:14.112 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:04:14.118 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 1 2023-07-02T14:04:14.172 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:14.172 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:14.182 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:04:14.212 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:14.212+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:14.305 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:14.306 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:14.313 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:14.312+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:14.316 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:04:14.414 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:14.413+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:14.440 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:14.440 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:14.450 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:04:14.504 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:04:14.547 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:14.546+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:14.614 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:14.614 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:14.616 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:14.615+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:14.624 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:04:14.644 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:14.662 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:14.679 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:14.696 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:14.714 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:04:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:04:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:04:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:14.716 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:04:14.716 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:04:14.717 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:14.716+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:14.717 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:14.718 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:14.747 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:14.748 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:14.748 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:04:14.748 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:04:14.748 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:14.758 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:14.818 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:14.817+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:14.918 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:14.917+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:14.924 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:14.924 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:04:14.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:04:14.924 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:14.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.924 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:14.925 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:14.925 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 95.631204209000003, 2023-07-02T14:04:14.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:04:14.925 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:14.925 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:14.925 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:14.925 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:14.925 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:14.926 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:04:14.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.926 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:14.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:14.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:14.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:14.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:14.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:14.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:14.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:14.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:04:14.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:04:14.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:14.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:04:14.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:04:14.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:14.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:04:14.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:04:14.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:14.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:04:14.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:04:14.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:14.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:04:14.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:04:14.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:14.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:04:14.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:04:14.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:14.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:04:14.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:04:14.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:14.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:04:14.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:04:14.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:14.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:04:14.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:04:14.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:14.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:04:14.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:04:14.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:14.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:04:14.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:04:14.934 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:14.934 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:14.935 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:14.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.935 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:14.935 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:14.935 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 95.630969186000002, 2023-07-02T14:04:14.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:04:14.935 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:14.936 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:14.936 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:14.936 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:14.936 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:14.936 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:04:14.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.936 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:14.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:14.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:14.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:14.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:14.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:14.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:14.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:14.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:04:14.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:04:14.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:14.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:04:14.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:04:14.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:14.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:04:14.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:04:14.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:14.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:04:14.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:04:14.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:14.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:04:14.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:04:14.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:14.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:04:14.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:04:14.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:14.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:04:14.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:04:14.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:14.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:04:14.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:04:14.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:14.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:04:14.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:04:14.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:14.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:04:14.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:04:14.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:14.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:04:14.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:04:14.945 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:14.945 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:14.945 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:14.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.945 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:14.945 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:14.946 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 95.630770052000003, 2023-07-02T14:04:14.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:04:14.946 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:14.946 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:14.946 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:14.946 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:14.946 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:14.946 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:04:14.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.947 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:14.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:14.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:14.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:14.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:14.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:14.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:14.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:14.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:04:14.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:04:14.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:14.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:04:14.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:04:14.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:14.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:04:14.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:04:14.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:14.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:04:14.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:04:14.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:14.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:04:14.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:04:14.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:14.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:04:14.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:04:14.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:14.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:04:14.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:04:14.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:14.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:04:14.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:04:14.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:14.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:04:14.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:04:14.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.954 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:14.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:04:14.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:04:14.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:14.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:04:14.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:04:14.955 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:14.955 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:14.955 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:14.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.956 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:14.956 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:14.956 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 95.630404928000004, 2023-07-02T14:04:14.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:04:14.956 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:14.956 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:14.956 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:14.957 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:14.957 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:14.957 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:04:14.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.957 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:14.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:14.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:14.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:14.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:14.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:14.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:14.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:14.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:04:14.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:04:14.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:14.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:04:14.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:04:14.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:14.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:04:14.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:04:14.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:14.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:04:14.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:04:14.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:14.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:04:14.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:04:14.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:14.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:04:14.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:04:14.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:14.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:04:14.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:04:14.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:14.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:04:14.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:04:14.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:14.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:04:14.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:04:14.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:14.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:04:14.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:04:14.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:14.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:04:14.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:04:14.965 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:14.966 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:14.966 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:14.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.966 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:14.966 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:14.966 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 95.630323867000001, 2023-07-02T14:04:14.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:04:14.967 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:14.967 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:14.967 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:14.967 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:14.967 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:14.967 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:04:14.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.967 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:14.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:14.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:14.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:14.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:14.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:14.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:14.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:14.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:04:14.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:04:14.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:14.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:04:14.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:04:14.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:14.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:04:14.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:04:14.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:14.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:04:14.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:04:14.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:14.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:04:14.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:04:14.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:14.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:04:14.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:04:14.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:14.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:04:14.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:04:14.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:14.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:04:14.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:04:14.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:14.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:04:14.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:04:14.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:14.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:04:14.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:04:14.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:14.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:04:14.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:04:14.976 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:14.976 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:14.976 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:14.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.976 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:14.977 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:14.977 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 95.630232195999994, 2023-07-02T14:04:14.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:04:14.977 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:14.977 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:14.977 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:14.977 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:14.978 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:14.978 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:04:14.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.978 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:14.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:14.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:14.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:14.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:14.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:14.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:14.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:14.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:04:14.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:04:14.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:14.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:04:14.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:04:14.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:14.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:04:14.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:04:14.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:14.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:04:14.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:04:14.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:14.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:04:14.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:04:14.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:14.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:04:14.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:04:14.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:14.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:04:14.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:04:14.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:14.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:04:14.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:04:14.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:14.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:04:14.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:04:14.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:14.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:04:14.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:04:14.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:14.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:04:14.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:04:14.986 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:14.986 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:14.986 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:14.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.987 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:14.987 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:14.987 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 95.629892753999997, 2023-07-02T14:04:14.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:04:14.987 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:14.987 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:14.988 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:14.988 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:14.988 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:14.988 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:04:14.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.988 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:14.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:14.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:14.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:14.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:14.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:14.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:14.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:14.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:04:14.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:04:14.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:14.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:04:14.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:04:14.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:14.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:04:14.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:04:14.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:14.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:04:14.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:04:14.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:14.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:04:14.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:04:14.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:14.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:04:14.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:04:14.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:14.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:04:14.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:04:14.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:14.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:04:14.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:04:14.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:14.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:04:14.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:04:14.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:14.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:04:14.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:04:14.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:14.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:04:14.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:04:14.996 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:14.997 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:14.997 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:14.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.997 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:14.997 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:14.997 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 95.629706701000003, 2023-07-02T14:04:14.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:04:14.998 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:14.998 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:14.998 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:14.998 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:14.998 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:14.998 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:04:14.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.998 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:14.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:14.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:14.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:14.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:14.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:14.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:15.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:15.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:15.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:15.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:15.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:04:15.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:15.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:04:15.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:04:15.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:15.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:04:15.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:04:15.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:04:15.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:04:15.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:15.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:04:15.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:04:15.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:15.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:04:15.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:04:15.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:15.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:04:15.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:04:15.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:15.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:04:15.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:04:15.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:04:15.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:04:15.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:15.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:04:15.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:04:15.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:15.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:04:15.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:04:15.007 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.007 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:15.007 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.007 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:15.008 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:15.008 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 95.629548899, 2023-07-02T14:04:15.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:04:15.008 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:15.008 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.008 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:15.008 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:15.008 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.009 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:04:15.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.009 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:15.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:15.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:15.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:15.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:15.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:15.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:15.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:15.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:04:15.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:04:15.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:15.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:04:15.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:04:15.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:15.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:04:15.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:04:15.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:04:15.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:04:15.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:15.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:04:15.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:04:15.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:15.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:04:15.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:04:15.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:15.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:04:15.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:04:15.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:15.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:04:15.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:04:15.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:04:15.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:04:15.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:15.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:04:15.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:04:15.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:15.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:04:15.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:04:15.017 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.017 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:15.017 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.018 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:15.018 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:15.018 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 95.629357116999998, 2023-07-02T14:04:15.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:04:15.018 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:15.018 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.018 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:15.019 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:15.019 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.019 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:04:15.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.019 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:15.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:15.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:15.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:15.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:15.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:15.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:15.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:15.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:04:15.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:04:15.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:15.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:04:15.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:04:15.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:15.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:04:15.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:04:15.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:04:15.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:04:15.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:15.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:04:15.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:04:15.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:15.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:04:15.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:04:15.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:15.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:04:15.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:04:15.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:04:15.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:04:15.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:15.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:04:15.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:04:15.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:15.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:04:15.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:04:15.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:15.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:04:15.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:04:15.027 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.027 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:15.028 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.028 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:15.028 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:15.028 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 95.629181457000001, 2023-07-02T14:04:15.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:04:15.028 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:15.029 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.029 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:15.029 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:15.029 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.029 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:04:15.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.029 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:15.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:15.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:15.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:15.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:15.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:15.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:15.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:15.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:15.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:04:15.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:15.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:04:15.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:04:15.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:15.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:04:15.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:04:15.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:04:15.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:04:15.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:15.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:04:15.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:04:15.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:15.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:04:15.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:04:15.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:15.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:04:15.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:04:15.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:04:15.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:04:15.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:15.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:04:15.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:04:15.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:15.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:04:15.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:04:15.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:15.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:04:15.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:04:15.038 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.038 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:15.038 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.038 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:15.038 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:15.038 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 95.629087268999996, 2023-07-02T14:04:15.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:04:15.039 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:15.039 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.039 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:15.039 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:15.039 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.039 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:04:15.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.040 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:15.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:15.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:15.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:15.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:15.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:15.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:15.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:15.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:04:15.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:04:15.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:15.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:04:15.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:04:15.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:15.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:04:15.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:04:15.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:04:15.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:04:15.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:15.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:04:15.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:04:15.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:15.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:04:15.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:04:15.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:15.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:04:15.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:04:15.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:04:15.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:04:15.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:15.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:04:15.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:04:15.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:15.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:04:15.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:04:15.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:15.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:04:15.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:04:15.048 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.048 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:15.048 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.049 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:15.049 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:15.049 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 95.628883883, 2023-07-02T14:04:15.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:04:15.049 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:15.049 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.049 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:15.049 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:15.050 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.050 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:04:15.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.050 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:15.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:15.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:15.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:15.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:15.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:15.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:15.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:15.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:04:15.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:04:15.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:15.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:04:15.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:04:15.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:15.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:04:15.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:04:15.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:04:15.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:04:15.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:15.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:04:15.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:04:15.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:15.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:04:15.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:04:15.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:15.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:04:15.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:04:15.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:04:15.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:04:15.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:15.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:04:15.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:04:15.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:15.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:04:15.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:04:15.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:15.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:04:15.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:04:15.058 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.059 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:15.059 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.059 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:15.059 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:15.059 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 95.628733226999998, 2023-07-02T14:04:15.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:04:15.060 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:15.060 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.060 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:15.060 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:15.060 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.060 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:04:15.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.061 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:15.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:15.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:15.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:15.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:15.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:15.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:15.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:15.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:04:15.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:04:15.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:15.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:04:15.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:04:15.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:15.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:04:15.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:04:15.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:04:15.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:04:15.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:15.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:04:15.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:04:15.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:15.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:04:15.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:04:15.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:15.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:04:15.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:04:15.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:04:15.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:04:15.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:15.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:04:15.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:04:15.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:15.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:04:15.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:04:15.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:15.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:04:15.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:04:15.069 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.069 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:15.069 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.070 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:15.070 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:15.070 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 95.628660523999997, 2023-07-02T14:04:15.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:04:15.070 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:15.070 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.070 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:15.071 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:15.071 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.071 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:04:15.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.071 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:15.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:15.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:15.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:15.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:15.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:15.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:15.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:15.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:15.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:04:15.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:15.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:04:15.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:04:15.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:15.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:04:15.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:04:15.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:04:15.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:04:15.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:15.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:04:15.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:04:15.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:15.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:04:15.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:04:15.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:15.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:04:15.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:04:15.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:04:15.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:04:15.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:15.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:04:15.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:04:15.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:15.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:04:15.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:04:15.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:15.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:04:15.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:04:15.080 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.080 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:15.080 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.080 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:15.080 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:15.080 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 95.628543534000002, 2023-07-02T14:04:15.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:04:15.081 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:15.081 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.081 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:15.081 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:15.081 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.081 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:04:15.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.082 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:15.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:15.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:15.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:15.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:15.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:15.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:15.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:15.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:15.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:04:15.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:15.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:04:15.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:04:15.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:15.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:04:15.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:04:15.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:04:15.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:04:15.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:15.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:04:15.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:04:15.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:15.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:04:15.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:04:15.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:15.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:04:15.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:04:15.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:04:15.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:04:15.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:15.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:04:15.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:04:15.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:15.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:04:15.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:04:15.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:15.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:04:15.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:04:15.090 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.090 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:15.090 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.091 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:15.091 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:15.091 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 95.628394083000003, 2023-07-02T14:04:15.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:04:15.091 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:15.091 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.091 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:15.092 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:15.092 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.092 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:04:15.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.092 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:15.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:15.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:15.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:15.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:15.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:15.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:15.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:15.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:04:15.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:04:15.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:15.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:04:15.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:04:15.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:15.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:04:15.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:04:15.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:04:15.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:04:15.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:15.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:04:15.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:04:15.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:15.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:04:15.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:04:15.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:15.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:04:15.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:04:15.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:04:15.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:04:15.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:15.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:04:15.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:04:15.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:15.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:04:15.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:04:15.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:15.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:04:15.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:04:15.100 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.101 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:15.101 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.101 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:15.101 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:15.101 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 95.606918109999995, 2023-07-02T14:04:15.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:04:15.102 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:15.102 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.102 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:15.102 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:15.102 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.102 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:04:15.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.103 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:15.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:15.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:15.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:15.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:15.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:15.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:15.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:15.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:04:15.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:04:15.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:15.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:04:15.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:04:15.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:15.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:04:15.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:04:15.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:04:15.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:04:15.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:15.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:04:15.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:04:15.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:15.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:04:15.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:04:15.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:15.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:04:15.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:04:15.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:15.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:04:15.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:04:15.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:04:15.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:04:15.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:15.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:04:15.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:04:15.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:15.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:04:15.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:04:15.111 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.111 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:15.111 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.111 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:15.112 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:15.112 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 95.606785574, 2023-07-02T14:04:15.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:04:15.112 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:15.112 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.112 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:15.112 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:15.112 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.113 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:04:15.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.113 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:15.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:15.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:15.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:15.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:15.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:15.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:15.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:15.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:04:15.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:04:15.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:15.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:04:15.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:04:15.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:15.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:04:15.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:04:15.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:04:15.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:04:15.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:15.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:04:15.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:04:15.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:15.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:04:15.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:04:15.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:15.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:04:15.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:04:15.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:15.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:04:15.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:04:15.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:04:15.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:04:15.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:15.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:04:15.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:04:15.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:15.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:04:15.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:04:15.121 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.121 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:15.121 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.122 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:15.122 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:15.122 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 95.545448913000001, 2023-07-02T14:04:15.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:04:15.122 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:15.122 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.123 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:15.123 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:15.123 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.123 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:04:15.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.123 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:15.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:15.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:15.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:15.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:15.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:15.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:15.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:15.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:04:15.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:04:15.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:15.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:04:15.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:04:15.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:15.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:04:15.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:04:15.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:04:15.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:04:15.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:15.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:04:15.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:04:15.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:15.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:04:15.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:04:15.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:15.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:04:15.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:04:15.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:15.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:04:15.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:04:15.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:04:15.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:04:15.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:15.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:04:15.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:04:15.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:15.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:15.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:15.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:04:15.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:04:15.132 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.132 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:15.132 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.132 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:15.132 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:15.132 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:15.133 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap18 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap19 b3fbb028-e267-4d25-a750-93ba18a2316a@snap19.diff 2023-07-02T14:04:15.134 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:15.133+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:15.135 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:15.221 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:15.219+0000 7f497eeda700 0 -- 172.21.15.163:0/3941721740 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x561967091030 msgr2=0x561967093420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:15.234 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:15.233+0000 7f497eeda700 0 -- 172.21.15.163:0/3941721740 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f4960036bb0 msgr2=0x7f4960038fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:15.234 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:15.233+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:15.262 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 100% complete...done. 2023-07-02T14:04:15.265 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:04:15.266 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=20 2023-07-02T14:04:15.266 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap20 2023-07-02T14:04:15.270 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:15.270 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:15.271 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:15.271 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:15.281 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:04:15.364 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:15.363+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:15.436 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:15.435+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:15.445 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:15.445 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:15.445 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:15.445 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:15.445 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:15.455 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:04:15.537 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:15.537+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:15.621 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2620 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 196608~786432] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.295033528999994, 2023-07-02T14:04:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966, 2023-07-02T14:04:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2620 2023-07-02T14:04:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324571+0000", 2023-07-02T14:04:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016686649999999999 2023-07-02T14:04:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326240+0000", 2023-07-02T14:04:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1844e-05 2023-07-02T14:04:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326252+0000", 2023-07-02T14:04:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7760000000000003e-06 2023-07-02T14:04:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326258+0000", 2023-07-02T14:04:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0634000000000003e-05 2023-07-02T14:04:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326318+0000", 2023-07-02T14:04:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9629999999999993e-06 2023-07-02T14:04:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326328+0000", 2023-07-02T14:04:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.010877045 2023-07-02T14:04:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337205+0000", 2023-07-02T14:04:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022401300000000001 2023-07-02T14:04:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337429+0000", 2023-07-02T14:04:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000139822 2023-07-02T14:04:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337569+0000", 2023-07-02T14:04:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032942119999999999 2023-07-02T14:04:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340863+0000", 2023-07-02T14:04:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021302560000000001 2023-07-02T14:04:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342994+0000", 2023-07-02T14:04:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5634000000000002e-05 2023-07-02T14:04:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343059+0000", 2023-07-02T14:04:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143721 2023-07-02T14:04:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343203+0000", 2023-07-02T14:04:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966 2023-07-02T14:04:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2622 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 983040~36864] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.293341018000007, 2023-07-02T14:04:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664, 2023-07-02T14:04:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2622 2023-07-02T14:04:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:15.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:15.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:15.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:15.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326265+0000", 2023-07-02T14:04:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1282000000000001e-05 2023-07-02T14:04:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326286+0000", 2023-07-02T14:04:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9480000000000002e-06 2023-07-02T14:04:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326293+0000", 2023-07-02T14:04:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3610000000000003e-06 2023-07-02T14:04:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326297+0000", 2023-07-02T14:04:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7707999999999994e-05 2023-07-02T14:04:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326385+0000", 2023-07-02T14:04:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8070000000000006e-06 2023-07-02T14:04:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326394+0000", 2023-07-02T14:04:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.014535878 2023-07-02T14:04:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340930+0000", 2023-07-02T14:04:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000154551 2023-07-02T14:04:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341084+0000", 2023-07-02T14:04:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00032375900000000002 2023-07-02T14:04:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341408+0000", 2023-07-02T14:04:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012631066 2023-07-02T14:04:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354039+0000", 2023-07-02T14:04:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5741000000000001e-05 2023-07-02T14:04:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354055+0000", 2023-07-02T14:04:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8794000000000006e-05 2023-07-02T14:04:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354124+0000", 2023-07-02T14:04:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014667700000000001 2023-07-02T14:04:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354270+0000", 2023-07-02T14:04:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664 2023-07-02T14:04:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2727 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~12288] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.293305243000006, 2023-07-02T14:04:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999, 2023-07-02T14:04:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2727 2023-07-02T14:04:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3834e-05 2023-07-02T14:04:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326315+0000", 2023-07-02T14:04:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3029999999999996e-06 2023-07-02T14:04:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326321+0000", 2023-07-02T14:04:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7590000000000002e-06 2023-07-02T14:04:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326324+0000", 2023-07-02T14:04:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112872 2023-07-02T14:04:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326437+0000", 2023-07-02T14:04:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2290000000000002e-06 2023-07-02T14:04:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326442+0000", 2023-07-02T14:04:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0155896879999999 2023-07-02T14:04:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342031+0000", 2023-07-02T14:04:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016496100000000001 2023-07-02T14:04:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342196+0000", 2023-07-02T14:04:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021998699999999999 2023-07-02T14:04:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342416+0000", 2023-07-02T14:04:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011869744999999999 2023-07-02T14:04:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354286+0000", 2023-07-02T14:04:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0361e-05 2023-07-02T14:04:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354347+0000", 2023-07-02T14:04:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3312000000000005e-05 2023-07-02T14:04:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354430+0000", 2023-07-02T14:04:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126444 2023-07-02T14:04:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354556+0000", 2023-07-02T14:04:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999 2023-07-02T14:04:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2730 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [set-alloc-hint object_size 4194304 write_size 4194304,write 454656~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.293280233999994, 2023-07-02T14:04:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999, 2023-07-02T14:04:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2730 2023-07-02T14:04:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6049999999999997e-06 2023-07-02T14:04:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326332+0000", 2023-07-02T14:04:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3000000000000002e-06 2023-07-02T14:04:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326335+0000", 2023-07-02T14:04:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.719e-05 2023-07-02T14:04:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326352+0000", 2023-07-02T14:04:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011583599999999999 2023-07-02T14:04:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326468+0000", 2023-07-02T14:04:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.831e-06 2023-07-02T14:04:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326473+0000", 2023-07-02T14:04:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.016769915 2023-07-02T14:04:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343243+0000", 2023-07-02T14:04:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197352 2023-07-02T14:04:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343440+0000", 2023-07-02T14:04:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197581 2023-07-02T14:04:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343638+0000", 2023-07-02T14:04:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010980139 2023-07-02T14:04:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354618+0000", 2023-07-02T14:04:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056779530000000003 2023-07-02T14:04:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360296+0000", 2023-07-02T14:04:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101109 2023-07-02T14:04:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360397+0000", 2023-07-02T14:04:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000160273 2023-07-02T14:04:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360557+0000", 2023-07-02T14:04:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999 2023-07-02T14:04:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2752 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 1019904~3174400] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:15.670 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.293234010000006, 2023-07-02T14:04:15.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001, 2023-07-02T14:04:15.670 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:15.670 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.670 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:15.670 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:15.670 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.670 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2752 2023-07-02T14:04:15.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.671 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:15.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:15.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:15.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:15.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:15.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:15.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:15.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:15.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326372+0000", 2023-07-02T14:04:15.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052050380000000004 2023-07-02T14:04:15.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:15.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331577+0000", 2023-07-02T14:04:15.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6832000000000001e-05 2023-07-02T14:04:15.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:15.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331594+0000", 2023-07-02T14:04:15.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8220000000000007e-06 2023-07-02T14:04:15.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331602+0000", 2023-07-02T14:04:15.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9895999999999999e-05 2023-07-02T14:04:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331642+0000", 2023-07-02T14:04:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:04:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331649+0000", 2023-07-02T14:04:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0139141709999999 2023-07-02T14:04:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345563+0000", 2023-07-02T14:04:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113817 2023-07-02T14:04:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:15.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345677+0000", 2023-07-02T14:04:15.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018803400000000001 2023-07-02T14:04:15.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:15.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345865+0000", 2023-07-02T14:04:15.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087950359999999991 2023-07-02T14:04:15.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354660+0000", 2023-07-02T14:04:15.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059116669999999998 2023-07-02T14:04:15.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:15.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360572+0000", 2023-07-02T14:04:15.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0992000000000002e-05 2023-07-02T14:04:15.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:15.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360633+0000", 2023-07-02T14:04:15.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013601600000000001 2023-07-02T14:04:15.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:15.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360769+0000", 2023-07-02T14:04:15.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001 2023-07-02T14:04:15.680 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.680 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.681 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.681 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2759 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 0~454656] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:15.681 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:15.681 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.287994213000005, 2023-07-02T14:04:15.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999, 2023-07-02T14:04:15.681 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:15.682 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.682 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:15.682 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:15.682 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.682 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2759 2023-07-02T14:04:15.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.682 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:15.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:15.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:15.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:15.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:15.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:15.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:15.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:15.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331611+0000", 2023-07-02T14:04:15.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053795999999999998 2023-07-02T14:04:15.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:15.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332149+0000", 2023-07-02T14:04:15.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4970000000000007e-06 2023-07-02T14:04:15.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:15.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332158+0000", 2023-07-02T14:04:15.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:04:15.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332162+0000", 2023-07-02T14:04:15.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3526000000000001e-05 2023-07-02T14:04:15.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332185+0000", 2023-07-02T14:04:15.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:04:15.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:15.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332189+0000", 2023-07-02T14:04:15.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0186295910000001 2023-07-02T14:04:15.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350819+0000", 2023-07-02T14:04:15.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000111123 2023-07-02T14:04:15.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:15.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350930+0000", 2023-07-02T14:04:15.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019575300000000001 2023-07-02T14:04:15.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:15.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.351126+0000", 2023-07-02T14:04:15.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011503298 2023-07-02T14:04:15.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:15.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362629+0000", 2023-07-02T14:04:15.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085443989999999994 2023-07-02T14:04:15.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371173+0000", 2023-07-02T14:04:15.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3547999999999995e-05 2023-07-02T14:04:15.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:15.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371237+0000", 2023-07-02T14:04:15.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790300000000001 2023-07-02T14:04:15.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:15.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371375+0000", 2023-07-02T14:04:15.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999 2023-07-02T14:04:15.692 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.692 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.692 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.692 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2760 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 458752~131072] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:15.693 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:15.693 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.287438027999997, 2023-07-02T14:04:15.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001, 2023-07-02T14:04:15.693 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:15.693 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.693 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:15.693 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:15.693 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.694 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2760 2023-07-02T14:04:15.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.694 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:15.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:15.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:15.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:15.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:15.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:15.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:15.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:15.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332168+0000", 2023-07-02T14:04:15.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013281100000000001 2023-07-02T14:04:15.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:15.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332301+0000", 2023-07-02T14:04:15.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7389999999999999e-06 2023-07-02T14:04:15.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:15.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332308+0000", 2023-07-02T14:04:15.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0639999999999998e-06 2023-07-02T14:04:15.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332312+0000", 2023-07-02T14:04:15.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1735000000000001e-05 2023-07-02T14:04:15.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332333+0000", 2023-07-02T14:04:15.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8342000000000002e-05 2023-07-02T14:04:15.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:15.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332352+0000", 2023-07-02T14:04:15.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02234782 2023-07-02T14:04:15.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354699+0000", 2023-07-02T14:04:15.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110371 2023-07-02T14:04:15.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:15.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354810+0000", 2023-07-02T14:04:15.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022174900000000001 2023-07-02T14:04:15.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:15.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355032+0000", 2023-07-02T14:04:15.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076053329999999997 2023-07-02T14:04:15.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:15.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362637+0000", 2023-07-02T14:04:15.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087913609999999993 2023-07-02T14:04:15.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371428+0000", 2023-07-02T14:04:15.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8532999999999998e-05 2023-07-02T14:04:15.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:15.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371477+0000", 2023-07-02T14:04:15.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010990800000000001 2023-07-02T14:04:15.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:15.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371587+0000", 2023-07-02T14:04:15.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001 2023-07-02T14:04:15.703 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.703 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.704 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2766 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 589824~45056] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:15.704 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:15.704 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.287289403000003, 2023-07-02T14:04:15.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188, 2023-07-02T14:04:15.705 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:15.705 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.705 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:15.705 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:15.705 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.705 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2766 2023-07-02T14:04:15.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.705 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:15.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:15.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:15.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:15.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:15.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:15.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:15.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:15.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:15.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2247999999999998e-05 2023-07-02T14:04:15.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:15.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332369+0000", 2023-07-02T14:04:15.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4189999999999996e-06 2023-07-02T14:04:15.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:15.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332376+0000", 2023-07-02T14:04:15.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1049999999999999e-06 2023-07-02T14:04:15.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332379+0000", 2023-07-02T14:04:15.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6894000000000003e-05 2023-07-02T14:04:15.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332416+0000", 2023-07-02T14:04:15.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3970000000000002e-06 2023-07-02T14:04:15.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:15.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332422+0000", 2023-07-02T14:04:15.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0231382280000001 2023-07-02T14:04:15.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355560+0000", 2023-07-02T14:04:15.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9079000000000005e-05 2023-07-02T14:04:15.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:15.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355659+0000", 2023-07-02T14:04:15.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018005000000000001 2023-07-02T14:04:15.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:15.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355839+0000", 2023-07-02T14:04:15.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068044439999999998 2023-07-02T14:04:15.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:15.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362644+0000", 2023-07-02T14:04:15.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090155589999999994 2023-07-02T14:04:15.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371659+0000", 2023-07-02T14:04:15.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1297999999999994e-05 2023-07-02T14:04:15.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:15.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371730+0000", 2023-07-02T14:04:15.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013100699999999999 2023-07-02T14:04:15.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:15.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371861+0000", 2023-07-02T14:04:15.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188 2023-07-02T14:04:15.715 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.715 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.715 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.716 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2784 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 0~393216] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:15.716 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:15.716 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.287220950999995, 2023-07-02T14:04:15.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999, 2023-07-02T14:04:15.716 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:15.716 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.716 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:15.716 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:15.717 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.717 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2784 2023-07-02T14:04:15.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.717 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:15.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:15.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:15.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:15.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:15.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:15.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:15.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:15.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332385+0000", 2023-07-02T14:04:15.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040252400000000002 2023-07-02T14:04:15.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:15.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332787+0000", 2023-07-02T14:04:15.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0208e-05 2023-07-02T14:04:15.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:15.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332798+0000", 2023-07-02T14:04:15.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2023-07-02T14:04:15.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332801+0000", 2023-07-02T14:04:15.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4838000000000001e-05 2023-07-02T14:04:15.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332836+0000", 2023-07-02T14:04:15.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5319999999999998e-06 2023-07-02T14:04:15.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:15.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332839+0000", 2023-07-02T14:04:15.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0234290109999999 2023-07-02T14:04:15.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356268+0000", 2023-07-02T14:04:15.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011165700000000001 2023-07-02T14:04:15.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:15.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356380+0000", 2023-07-02T14:04:15.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022504799999999999 2023-07-02T14:04:15.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:15.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356605+0000", 2023-07-02T14:04:15.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015276201 2023-07-02T14:04:15.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:15.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371881+0000", 2023-07-02T14:04:15.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5496999999999999e-05 2023-07-02T14:04:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371957+0000", 2023-07-02T14:04:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4570000000000002e-05 2023-07-02T14:04:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:15.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372001+0000", 2023-07-02T14:04:15.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3880000000000003e-05 2023-07-02T14:04:15.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:15.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372085+0000", 2023-07-02T14:04:15.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999 2023-07-02T14:04:15.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.727 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.727 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2785 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 405504~446464] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:15.727 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:15.727 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.286799500000001, 2023-07-02T14:04:15.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999, 2023-07-02T14:04:15.728 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:15.728 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.728 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:15.728 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:15.728 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.728 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2785 2023-07-02T14:04:15.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.729 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:15.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:15.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:15.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:15.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:15.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:15.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:15.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:15.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:15.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059527500000000004 2023-07-02T14:04:15.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:15.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333402+0000", 2023-07-02T14:04:15.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7059999999999993e-06 2023-07-02T14:04:15.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:15.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333411+0000", 2023-07-02T14:04:15.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0450000000000001e-06 2023-07-02T14:04:15.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333414+0000", 2023-07-02T14:04:15.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5790999999999997e-05 2023-07-02T14:04:15.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:04:15.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3320000000000002e-06 2023-07-02T14:04:15.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:15.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333454+0000", 2023-07-02T14:04:15.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.025310269 2023-07-02T14:04:15.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358764+0000", 2023-07-02T14:04:15.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012414899999999999 2023-07-02T14:04:15.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:15.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358888+0000", 2023-07-02T14:04:15.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000217028 2023-07-02T14:04:15.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:15.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.359105+0000", 2023-07-02T14:04:15.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012788212 2023-07-02T14:04:15.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:15.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371893+0000", 2023-07-02T14:04:15.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012201569999999999 2023-07-02T14:04:15.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373114+0000", 2023-07-02T14:04:15.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4512999999999995e-05 2023-07-02T14:04:15.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:15.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373198+0000", 2023-07-02T14:04:15.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132373 2023-07-02T14:04:15.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:15.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373330+0000", 2023-07-02T14:04:15.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999 2023-07-02T14:04:15.738 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.738 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.738 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.739 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2786 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1310720~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:15.739 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:15.739 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.286186807000007, 2023-07-02T14:04:15.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001, 2023-07-02T14:04:15.739 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:15.739 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.739 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:15.740 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:15.740 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.740 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2786 2023-07-02T14:04:15.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.740 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:15.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:15.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:15.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:15.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:15.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:15.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:15.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:15.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333419+0000", 2023-07-02T14:04:15.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2009999999999999e-06 2023-07-02T14:04:15.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:15.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333427+0000", 2023-07-02T14:04:15.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0210000000000001e-06 2023-07-02T14:04:15.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:15.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333431+0000", 2023-07-02T14:04:15.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3769999999999999e-06 2023-07-02T14:04:15.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333434+0000", 2023-07-02T14:04:15.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1161e-05 2023-07-02T14:04:15.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333485+0000", 2023-07-02T14:04:15.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1549000000000001e-05 2023-07-02T14:04:15.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:15.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333496+0000", 2023-07-02T14:04:15.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0273116339999999 2023-07-02T14:04:15.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360808+0000", 2023-07-02T14:04:15.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000115128 2023-07-02T14:04:15.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:15.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360923+0000", 2023-07-02T14:04:15.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018367700000000001 2023-07-02T14:04:15.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:15.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361107+0000", 2023-07-02T14:04:15.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010796923999999999 2023-07-02T14:04:15.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:15.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371904+0000", 2023-07-02T14:04:15.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014983520000000001 2023-07-02T14:04:15.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373402+0000", 2023-07-02T14:04:15.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0520999999999999e-05 2023-07-02T14:04:15.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:15.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373463+0000", 2023-07-02T14:04:15.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000129095 2023-07-02T14:04:15.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:15.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373592+0000", 2023-07-02T14:04:15.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001 2023-07-02T14:04:15.750 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.750 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.750 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.750 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2787 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1433600~8192] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:15.750 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:15.751 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.286170006999996, 2023-07-02T14:04:15.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999, 2023-07-02T14:04:15.751 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:15.751 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.751 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:15.751 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:15.751 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.751 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2787 2023-07-02T14:04:15.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.752 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:15.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:15.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:15.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:15.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:15.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:15.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:15.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:15.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333436+0000", 2023-07-02T14:04:15.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2731999999999999e-05 2023-07-02T14:04:15.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:15.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:04:15.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7440000000000001e-06 2023-07-02T14:04:15.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:15.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333456+0000", 2023-07-02T14:04:15.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8650000000000003e-06 2023-07-02T14:04:15.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333460+0000", 2023-07-02T14:04:15.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0077e-05 2023-07-02T14:04:15.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333530+0000", 2023-07-02T14:04:15.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4180000000000001e-06 2023-07-02T14:04:15.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:15.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333533+0000", 2023-07-02T14:04:15.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0284002720000001 2023-07-02T14:04:15.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361933+0000", 2023-07-02T14:04:15.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6781999999999996e-05 2023-07-02T14:04:15.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:15.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362030+0000", 2023-07-02T14:04:15.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018305400000000001 2023-07-02T14:04:15.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:15.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362213+0000", 2023-07-02T14:04:15.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097015109999999995 2023-07-02T14:04:15.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:15.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371915+0000", 2023-07-02T14:04:15.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001742295 2023-07-02T14:04:15.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373657+0000", 2023-07-02T14:04:15.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5980000000000003e-05 2023-07-02T14:04:15.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:15.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373713+0000", 2023-07-02T14:04:15.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011783800000000001 2023-07-02T14:04:15.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:15.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373831+0000", 2023-07-02T14:04:15.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999 2023-07-02T14:04:15.761 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.761 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.761 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.762 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2803 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 851968~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:15.762 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:15.762 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.286128155, 2023-07-02T14:04:15.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514, 2023-07-02T14:04:15.762 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:15.762 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.763 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:15.763 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:15.763 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.763 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2803 2023-07-02T14:04:15.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.763 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:15.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:15.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:15.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:15.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:15.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:15.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:15.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:15.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:15.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4943000000000001e-05 2023-07-02T14:04:15.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:15.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333493+0000", 2023-07-02T14:04:15.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2629999999999997e-06 2023-07-02T14:04:15.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:15.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333500+0000", 2023-07-02T14:04:15.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4159999999999997e-06 2023-07-02T14:04:15.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333505+0000", 2023-07-02T14:04:15.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4518999999999998e-05 2023-07-02T14:04:15.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333559+0000", 2023-07-02T14:04:15.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5900000000000008e-06 2023-07-02T14:04:15.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:15.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333568+0000", 2023-07-02T14:04:15.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0291061779999999 2023-07-02T14:04:15.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362674+0000", 2023-07-02T14:04:15.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110412 2023-07-02T14:04:15.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:15.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362784+0000", 2023-07-02T14:04:15.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015384599999999999 2023-07-02T14:04:15.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:15.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362938+0000", 2023-07-02T14:04:15.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091682310000000006 2023-07-02T14:04:15.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:15.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372106+0000", 2023-07-02T14:04:15.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017703230000000001 2023-07-02T14:04:15.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373877+0000", 2023-07-02T14:04:15.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6387000000000002e-05 2023-07-02T14:04:15.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:15.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373923+0000", 2023-07-02T14:04:15.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014099199999999999 2023-07-02T14:04:15.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:15.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374064+0000", 2023-07-02T14:04:15.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514 2023-07-02T14:04:15.773 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.773 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.773 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.773 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2805 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 634880~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:15.774 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:15.774 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.286094980000001, 2023-07-02T14:04:15.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814, 2023-07-02T14:04:15.774 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:15.774 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.774 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:15.774 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:15.774 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.775 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2805 2023-07-02T14:04:15.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.775 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:15.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:15.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:15.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:15.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:15.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:15.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:15.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:15.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333511+0000", 2023-07-02T14:04:15.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207092 2023-07-02T14:04:15.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:15.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333718+0000", 2023-07-02T14:04:15.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5110000000000003e-06 2023-07-02T14:04:15.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:15.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333727+0000", 2023-07-02T14:04:15.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3270000000000007e-06 2023-07-02T14:04:15.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333736+0000", 2023-07-02T14:04:15.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0319999999999999e-05 2023-07-02T14:04:15.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333756+0000", 2023-07-02T14:04:15.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1695000000000001e-05 2023-07-02T14:04:15.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:15.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:04:15.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0295750560000001 2023-07-02T14:04:15.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363343+0000", 2023-07-02T14:04:15.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3387999999999994e-05 2023-07-02T14:04:15.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:15.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363437+0000", 2023-07-02T14:04:15.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018373 2023-07-02T14:04:15.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:15.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363620+0000", 2023-07-02T14:04:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0084948769999999996 2023-07-02T14:04:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372115+0000", 2023-07-02T14:04:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020082170000000001 2023-07-02T14:04:15.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374123+0000", 2023-07-02T14:04:15.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8875999999999997e-05 2023-07-02T14:04:15.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:15.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374182+0000", 2023-07-02T14:04:15.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0001272 2023-07-02T14:04:15.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:15.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374309+0000", 2023-07-02T14:04:15.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814 2023-07-02T14:04:15.784 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.784 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.785 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.785 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2806 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 1359872~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:15.785 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:15.785 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.285864649000004, 2023-07-02T14:04:15.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234, 2023-07-02T14:04:15.785 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:15.786 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.786 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:15.786 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:15.786 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.786 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2806 2023-07-02T14:04:15.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.786 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:15.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:15.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:15.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:15.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:15.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:15.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:15.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:15.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333741+0000", 2023-07-02T14:04:15.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3978999999999999e-05 2023-07-02T14:04:15.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:15.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333755+0000", 2023-07-02T14:04:15.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:04:15.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:15.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333761+0000", 2023-07-02T14:04:15.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7010000000000001e-06 2023-07-02T14:04:15.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333764+0000", 2023-07-02T14:04:15.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5086000000000003e-05 2023-07-02T14:04:15.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333799+0000", 2023-07-02T14:04:15.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6389999999999992e-06 2023-07-02T14:04:15.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:15.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333806+0000", 2023-07-02T14:04:15.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.030496675 2023-07-02T14:04:15.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364303+0000", 2023-07-02T14:04:15.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011382700000000001 2023-07-02T14:04:15.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:15.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364417+0000", 2023-07-02T14:04:15.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192058 2023-07-02T14:04:15.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:15.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364609+0000", 2023-07-02T14:04:15.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075132130000000004 2023-07-02T14:04:15.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:15.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372122+0000", 2023-07-02T14:04:15.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022731750000000001 2023-07-02T14:04:15.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374395+0000", 2023-07-02T14:04:15.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5277000000000005e-05 2023-07-02T14:04:15.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:15.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374460+0000", 2023-07-02T14:04:15.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017074599999999999 2023-07-02T14:04:15.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:15.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374631+0000", 2023-07-02T14:04:15.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234 2023-07-02T14:04:15.796 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.796 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.796 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.796 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 868352~49152] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:15.797 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:15.797 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.285837935000004, 2023-07-02T14:04:15.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001, 2023-07-02T14:04:15.797 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:15.797 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.797 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:15.797 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:15.797 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.798 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:04:15.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.798 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:15.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:15.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:15.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:15.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:15.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:15.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:15.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:15.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:04:15.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6355000000000001e-05 2023-07-02T14:04:15.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:15.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333794+0000", 2023-07-02T14:04:15.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8329999999999996e-06 2023-07-02T14:04:15.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:15.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333801+0000", 2023-07-02T14:04:15.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5419999999999996e-06 2023-07-02T14:04:15.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333807+0000", 2023-07-02T14:04:15.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2023-07-02T14:04:15.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333826+0000", 2023-07-02T14:04:15.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.303e-06 2023-07-02T14:04:15.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:15.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333830+0000", 2023-07-02T14:04:15.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.031137559 2023-07-02T14:04:15.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364967+0000", 2023-07-02T14:04:15.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1390000000000004e-05 2023-07-02T14:04:15.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:15.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365059+0000", 2023-07-02T14:04:15.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017597399999999999 2023-07-02T14:04:15.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:15.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365234+0000", 2023-07-02T14:04:15.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068943950000000002 2023-07-02T14:04:15.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:15.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372129+0000", 2023-07-02T14:04:15.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026189910000000002 2023-07-02T14:04:15.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374748+0000", 2023-07-02T14:04:15.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6613999999999996e-05 2023-07-02T14:04:15.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:15.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374814+0000", 2023-07-02T14:04:15.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502300000000001 2023-07-02T14:04:15.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:15.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374950+0000", 2023-07-02T14:04:15.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001 2023-07-02T14:04:15.807 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.808 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.808 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.808 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2815 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 917504~49152] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:15.808 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:15.808 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.281623702000005, 2023-07-02T14:04:15.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998, 2023-07-02T14:04:15.809 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:15.809 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.809 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:15.809 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:15.809 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.809 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2815 2023-07-02T14:04:15.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.809 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:15.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:15.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:15.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:15.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:15.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:15.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:04:15.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:15.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337980+0000", 2023-07-02T14:04:15.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017126700000000001 2023-07-02T14:04:15.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:15.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338152+0000", 2023-07-02T14:04:15.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2938999999999999e-05 2023-07-02T14:04:15.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:15.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338164+0000", 2023-07-02T14:04:15.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4448999999999999e-05 2023-07-02T14:04:15.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:04:15.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0281812 2023-07-02T14:04:15.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366370+0000", 2023-07-02T14:04:15.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1840000000000002e-05 2023-07-02T14:04:15.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:15.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366462+0000", 2023-07-02T14:04:15.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000186112 2023-07-02T14:04:15.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:15.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366648+0000", 2023-07-02T14:04:15.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054905709999999996 2023-07-02T14:04:15.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:15.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372139+0000", 2023-07-02T14:04:15.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044954790000000001 2023-07-02T14:04:15.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376634+0000", 2023-07-02T14:04:15.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4621000000000004e-05 2023-07-02T14:04:15.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376699+0000", 2023-07-02T14:04:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012862299999999999 2023-07-02T14:04:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:15.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376827+0000", 2023-07-02T14:04:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998 2023-07-02T14:04:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2816 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 917504~65536] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:15.818 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:15.819 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.281409323000005, 2023-07-02T14:04:15.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001, 2023-07-02T14:04:15.819 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:15.819 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.819 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:15.819 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:15.819 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.819 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2816 2023-07-02T14:04:15.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.820 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:15.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:15.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:15.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:15.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:15.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:15.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:15.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:15.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338196+0000", 2023-07-02T14:04:15.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6286999999999999e-05 2023-07-02T14:04:15.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:15.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338243+0000", 2023-07-02T14:04:15.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7030000000000001e-06 2023-07-02T14:04:15.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:15.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338252+0000", 2023-07-02T14:04:15.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:04:15.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338258+0000", 2023-07-02T14:04:15.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032085734999999997 2023-07-02T14:04:15.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370344+0000", 2023-07-02T14:04:15.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108621 2023-07-02T14:04:15.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:15.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370452+0000", 2023-07-02T14:04:15.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014936499999999999 2023-07-02T14:04:15.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:15.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370602+0000", 2023-07-02T14:04:15.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001778012 2023-07-02T14:04:15.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:15.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372380+0000", 2023-07-02T14:04:15.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044876380000000004 2023-07-02T14:04:15.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:15.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376867+0000", 2023-07-02T14:04:15.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9870999999999999e-05 2023-07-02T14:04:15.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:15.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376897+0000", 2023-07-02T14:04:15.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011401 2023-07-02T14:04:15.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:15.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.377011+0000", 2023-07-02T14:04:15.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001 2023-07-02T14:04:15.828 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.828 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.828 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.829 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:3878 2.1 e80/79 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head v 80'436, mlcod=80'429)", 2023-07-02T14:04:15.829 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:15.829 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.512155825999997, 2023-07-02T14:04:15.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037799472000000001, 2023-07-02T14:04:15.829 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:15.829 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.829 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:15.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:15.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:15.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:15.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:15.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:15.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:15.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:15.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107450+0000", 2023-07-02T14:04:15.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023914520000000001 2023-07-02T14:04:15.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:15.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109841+0000", 2023-07-02T14:04:15.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0823e-05 2023-07-02T14:04:15.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:15.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109862+0000", 2023-07-02T14:04:15.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2023-07-02T14:04:15.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109874+0000", 2023-07-02T14:04:15.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075009600000000001 2023-07-02T14:04:15.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.110624+0000", 2023-07-02T14:04:15.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5176000000000001e-05 2023-07-02T14:04:15.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:15.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.110649+0000", 2023-07-02T14:04:15.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034557310000000001 2023-07-02T14:04:15.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:15.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145206+0000", 2023-07-02T14:04:15.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4110999999999998e-05 2023-07-02T14:04:15.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:15.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145250+0000", 2023-07-02T14:04:15.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037799472000000001 2023-07-02T14:04:15.836 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.836 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.836 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.836 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:7177 2.5 2:a6e4b3cf:::rbd_data.125de1449605.00000000000000c4:head [write 0~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e91)", 2023-07-02T14:04:15.837 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:01.433694+0000", 2023-07-02T14:04:15.837 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.185912707, 2023-07-02T14:04:15.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033301720999999999, 2023-07-02T14:04:15.837 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:15.837 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:15.837 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:15.837 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:15.837 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:15.838 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7177 2023-07-02T14:04:15.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.838 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:15.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:15.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433694+0000", 2023-07-02T14:04:15.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:15.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:15.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433694+0000", 2023-07-02T14:04:15.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:15.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:15.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433693+0000", 2023-07-02T14:04:15.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6749999999999997e-05 2023-07-02T14:04:15.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:15.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433750+0000", 2023-07-02T14:04:15.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0451e-05 2023-07-02T14:04:15.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433760+0000", 2023-07-02T14:04:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8200000000000002e-06 2023-07-02T14:04:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:15.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433766+0000", 2023-07-02T14:04:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012906031 2023-07-02T14:04:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.446672+0000", 2023-07-02T14:04:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3190999999999998e-05 2023-07-02T14:04:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.446745+0000", 2023-07-02T14:04:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020003231999999999 2023-07-02T14:04:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.466749+0000", 2023-07-02T14:04:15.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010595 2023-07-02T14:04:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.466855+0000", 2023-07-02T14:04:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014102799999999999 2023-07-02T14:04:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:15.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.466996+0000", 2023-07-02T14:04:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033301720999999999 2023-07-02T14:04:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:15.845 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:15.846 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:15.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:04:15.847 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:15.846+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:15.947 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:15.946+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:16.011 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:16.012 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:16.012 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:16.012 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:16.022 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:04:16.065 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:16.064+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:16.102 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:04:16.115 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:04:16.120 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 3 2023-07-02T14:04:16.149 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:16.148+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:16.150 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:16.151 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:16.151 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:16.151 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:16.151 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:16.161 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:04:16.249 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:16.248+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:16.290 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:16.291 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:16.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:16.291 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:16.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.291 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:04:16.291 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:16.291 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 66.836484866999996, 2023-07-02T14:04:16.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:04:16.292 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:16.292 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:16.292 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:16.292 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:16.292 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:16.292 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:04:16.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.293 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:16.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:16.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:16.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:16.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:16.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:16.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:16.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:16.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:04:16.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:04:16.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:16.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:04:16.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:04:16.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:16.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:04:16.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:04:16.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:16.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:04:16.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:04:16.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:04:16.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:04:16.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:16.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:04:16.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:04:16.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:04:16.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:04:16.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:16.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:04:16.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:04:16.299 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.299 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.299 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.300 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:04:16.300 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:16.300 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.965791859000007, 2023-07-02T14:04:16.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:04:16.300 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:16.300 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:16.300 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:16.301 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:04:16.301 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:04:16.301 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:04:16.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.301 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:16.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:16.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:16.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:16.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:16.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:16.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:16.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:16.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:04:16.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:04:16.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:16.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:04:16.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:04:16.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:16.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:04:16.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:04:16.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:16.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:04:16.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:04:16.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:04:16.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:04:16.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:16.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:04:16.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:04:16.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:04:16.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:04:16.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:16.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:04:16.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:04:16.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:16.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:04:16.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:04:16.308 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.308 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.308 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.309 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:16.309 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:16.309 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.965667138000001, 2023-07-02T14:04:16.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:04:16.309 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:16.309 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:16.309 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:16.310 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:16.310 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:16.310 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:04:16.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.310 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:16.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:16.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:16.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:16.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:16.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:16.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:16.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:16.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:04:16.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:04:16.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:16.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:04:16.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:04:16.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:16.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:04:16.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:04:16.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:16.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:04:16.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:04:16.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:04:16.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:04:16.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:16.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:04:16.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:04:16.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:04:16.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:04:16.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:16.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:04:16.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:04:16.317 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.317 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.317 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.317 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:16.317 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:16.317 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.958817253999996, 2023-07-02T14:04:16.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:04:16.318 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:16.318 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:16.318 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:16.318 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:16.318 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:16.318 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:04:16.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.319 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:16.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:16.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:16.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:16.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:16.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:16.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:04:16.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:16.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:04:16.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:04:16.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:16.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:04:16.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:04:16.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:16.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:04:16.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:04:16.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:16.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:04:16.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:04:16.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:04:16.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:04:16.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:16.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:04:16.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:04:16.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:04:16.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:04:16.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:16.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:04:16.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:04:16.325 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.325 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.325 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.326 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:16.326 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:16.326 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.958754329000001, 2023-07-02T14:04:16.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:04:16.326 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:16.326 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:16.326 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:16.326 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:16.327 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:16.327 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:04:16.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.327 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:16.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:16.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:16.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:16.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:16.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:16.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:16.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:16.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:04:16.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:04:16.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:16.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:04:16.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:04:16.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:16.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:04:16.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:04:16.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:16.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:04:16.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:04:16.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:04:16.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:04:16.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:16.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:04:16.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:04:16.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:04:16.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:04:16.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:16.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:04:16.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:04:16.333 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.333 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.334 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.334 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:16.334 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:16.334 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.958453907999996, 2023-07-02T14:04:16.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:04:16.335 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:16.335 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:16.335 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:16.335 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:16.335 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:16.335 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:04:16.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.335 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:16.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:16.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:16.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:16.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:16.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:16.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:16.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:16.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:04:16.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:04:16.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:16.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:04:16.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:04:16.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:16.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:04:16.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:04:16.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:16.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:04:16.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:04:16.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:04:16.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:04:16.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:16.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:04:16.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:04:16.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:04:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:04:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:04:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:04:16.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:04:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:04:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:04:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:04:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:04:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:04:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:04:16.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:04:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:04:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:16.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:04:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:04:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:04:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:04:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.346 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:04:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.957929407999998, 2023-07-02T14:04:16.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:04:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:04:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.348 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:16.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:04:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:04:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:04:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:04:16.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:04:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:04:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:04:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:04:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:04:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:04:16.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:04:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:04:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:04:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:04:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:04:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:04:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:04:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:04:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.833910283000002, 2023-07-02T14:04:16.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:04:16.357 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:16.357 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:16.357 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:16.357 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:16.357 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:16.357 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:04:16.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.357 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:16.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:16.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:16.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:16.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:16.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:16.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:16.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:16.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:04:16.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:04:16.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:16.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:04:16.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:04:16.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:16.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:04:16.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:04:16.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:16.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:04:16.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:04:16.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:04:16.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:04:16.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:16.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:04:16.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:04:16.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:04:16.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:04:16.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:16.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:04:16.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:04:16.364 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.364 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.365 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.365 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:16.365 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:16.365 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.508427578999999, 2023-07-02T14:04:16.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:04:16.365 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:16.366 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:16.366 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:16.366 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:16.366 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:16.366 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:04:16.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.366 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:16.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:16.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:16.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:16.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:16.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:16.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:16.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:16.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:04:16.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:04:16.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:16.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:04:16.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:04:16.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:16.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:04:16.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:04:16.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:16.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:04:16.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:04:16.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:16.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:04:16.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:16.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:04:16.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:04:16.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:04:16.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:04:16.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:16.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:04:16.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:04:16.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:16.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:04:16.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:04:16.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:16.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:04:16.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:04:16.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:16.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:04:16.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:04:16.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:16.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:04:16.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:04:16.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:16.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:04:16.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:04:16.376 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.376 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.376 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.377 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3887 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:16.377 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:16.377 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.504564389999999, 2023-07-02T14:04:16.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998, 2023-07-02T14:04:16.377 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:16.377 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:16.377 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:16.377 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:16.378 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:16.378 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3887 2023-07-02T14:04:16.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.378 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:16.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:16.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:16.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:16.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:16.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:16.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:16.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:16.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784019+0000", 2023-07-02T14:04:16.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4073999999999998e-05 2023-07-02T14:04:16.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:16.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784063+0000", 2023-07-02T14:04:16.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6630000000000004e-06 2023-07-02T14:04:16.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:16.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784070+0000", 2023-07-02T14:04:16.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.286e-06 2023-07-02T14:04:16.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:16.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784073+0000", 2023-07-02T14:04:16.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4941999999999998e-05 2023-07-02T14:04:16.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784118+0000", 2023-07-02T14:04:16.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8120000000000008e-06 2023-07-02T14:04:16.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:16.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784126+0000", 2023-07-02T14:04:16.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32999709399999999 2023-07-02T14:04:16.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114123+0000", 2023-07-02T14:04:16.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000157257 2023-07-02T14:04:16.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:16.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114280+0000", 2023-07-02T14:04:16.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000341152 2023-07-02T14:04:16.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:16.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114622+0000", 2023-07-02T14:04:16.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019539766 2023-07-02T14:04:16.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:16.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.134161+0000", 2023-07-02T14:04:16.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015820796000000002 2023-07-02T14:04:16.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:16.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149982+0000", 2023-07-02T14:04:16.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.05e-05 2023-07-02T14:04:16.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:16.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150043+0000", 2023-07-02T14:04:16.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6453999999999999e-05 2023-07-02T14:04:16.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:16.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150089+0000", 2023-07-02T14:04:16.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998 2023-07-02T14:04:16.388 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.388 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.388 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.388 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3939 2.1 2:9ef34a83:::rbd_data.125de1449605.00000000000000b8:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:16.388 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:16.388 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.504506882000001, 2023-07-02T14:04:16.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001, 2023-07-02T14:04:16.389 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:16.389 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:16.389 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:16.389 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:16.389 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:16.389 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3939 2023-07-02T14:04:16.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.390 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:16.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:16.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:16.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:16.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:16.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:16.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:16.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:16.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:16.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046852830000000002 2023-07-02T14:04:16.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:16.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788763+0000", 2023-07-02T14:04:16.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3185e-05 2023-07-02T14:04:16.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:16.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788776+0000", 2023-07-02T14:04:16.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8470000000000003e-06 2023-07-02T14:04:16.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:16.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788781+0000", 2023-07-02T14:04:16.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7104000000000002e-05 2023-07-02T14:04:16.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788828+0000", 2023-07-02T14:04:16.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3516000000000001e-05 2023-07-02T14:04:16.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:16.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788841+0000", 2023-07-02T14:04:16.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33064513600000001 2023-07-02T14:04:16.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119487+0000", 2023-07-02T14:04:16.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9876999999999997e-05 2023-07-02T14:04:16.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:16.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119537+0000", 2023-07-02T14:04:16.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014160999999999999 2023-07-02T14:04:16.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:16.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119678+0000", 2023-07-02T14:04:16.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023054392 2023-07-02T14:04:16.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:16.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.142733+0000", 2023-07-02T14:04:16.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074065470000000003 2023-07-02T14:04:16.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:16.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150139+0000", 2023-07-02T14:04:16.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5025999999999997e-05 2023-07-02T14:04:16.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:16.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150204+0000", 2023-07-02T14:04:16.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147999999999997e-05 2023-07-02T14:04:16.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:16.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150241+0000", 2023-07-02T14:04:16.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001 2023-07-02T14:04:16.399 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.399 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.399 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.400 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3940 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:16.400 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:16.400 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.499795849999998, 2023-07-02T14:04:16.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001, 2023-07-02T14:04:16.400 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:16.400 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:16.401 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:16.401 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:16.401 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:16.401 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3940 2023-07-02T14:04:16.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.401 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:16.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:16.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:16.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:16.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:16.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:16.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:16.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:16.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788788+0000", 2023-07-02T14:04:16.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043166560000000003 2023-07-02T14:04:16.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:16.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793105+0000", 2023-07-02T14:04:16.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5970000000000001e-05 2023-07-02T14:04:16.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:16.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793121+0000", 2023-07-02T14:04:16.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0939999999999998e-06 2023-07-02T14:04:16.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:16.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793126+0000", 2023-07-02T14:04:16.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2438000000000002e-05 2023-07-02T14:04:16.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793178+0000", 2023-07-02T14:04:16.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2019e-05 2023-07-02T14:04:16.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:16.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793190+0000", 2023-07-02T14:04:16.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33208757799999999 2023-07-02T14:04:16.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125278+0000", 2023-07-02T14:04:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5255999999999997e-05 2023-07-02T14:04:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125373+0000", 2023-07-02T14:04:16.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015674999999999999 2023-07-02T14:04:16.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:16.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125530+0000", 2023-07-02T14:04:16.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018767308 2023-07-02T14:04:16.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:16.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144297+0000", 2023-07-02T14:04:16.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060291959999999997 2023-07-02T14:04:16.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:16.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150326+0000", 2023-07-02T14:04:16.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3886999999999993e-05 2023-07-02T14:04:16.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:16.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150390+0000", 2023-07-02T14:04:16.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010255 2023-07-02T14:04:16.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:16.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150493+0000", 2023-07-02T14:04:16.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001 2023-07-02T14:04:16.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.411 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.411 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3947 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:16.411 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:16.412 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.495450323, 2023-07-02T14:04:16.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999, 2023-07-02T14:04:16.412 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:16.412 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:16.412 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:16.412 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:16.412 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:16.413 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3947 2023-07-02T14:04:16.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.413 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:16.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:16.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:16.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:16.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:16.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:16.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:16.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:16.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793133+0000", 2023-07-02T14:04:16.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2519999999999999e-05 2023-07-02T14:04:16.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:16.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793176+0000", 2023-07-02T14:04:16.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:04:16.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:16.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793180+0000", 2023-07-02T14:04:16.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1709999999999998e-06 2023-07-02T14:04:16.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:16.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793184+0000", 2023-07-02T14:04:16.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1721999999999999e-05 2023-07-02T14:04:16.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793215+0000", 2023-07-02T14:04:16.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1280000000000008e-06 2023-07-02T14:04:16.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:16.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793224+0000", 2023-07-02T14:04:16.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34218549599999998 2023-07-02T14:04:16.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135409+0000", 2023-07-02T14:04:16.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117442 2023-07-02T14:04:16.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:16.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135526+0000", 2023-07-02T14:04:16.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000206257 2023-07-02T14:04:16.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:16.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135733+0000", 2023-07-02T14:04:16.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085700029999999996 2023-07-02T14:04:16.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:16.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144303+0000", 2023-07-02T14:04:16.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006278652 2023-07-02T14:04:16.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:16.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150581+0000", 2023-07-02T14:04:16.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4765999999999996e-05 2023-07-02T14:04:16.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:16.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150646+0000", 2023-07-02T14:04:16.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4777999999999995e-05 2023-07-02T14:04:16.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:16.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150711+0000", 2023-07-02T14:04:16.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999 2023-07-02T14:04:16.422 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.423 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.423 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.423 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3980 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 65536~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:16.423 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:16.423 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.495396440999997, 2023-07-02T14:04:16.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999, 2023-07-02T14:04:16.424 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:16.424 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:16.424 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:16.424 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:16.424 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:16.424 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3980 2023-07-02T14:04:16.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.425 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:16.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:16.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:16.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:16.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:16.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:16.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:16.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:16.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:16.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020244260000000002 2023-07-02T14:04:16.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:16.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795212+0000", 2023-07-02T14:04:16.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0005e-05 2023-07-02T14:04:16.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:16.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795222+0000", 2023-07-02T14:04:16.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9860000000000002e-06 2023-07-02T14:04:16.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:16.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795227+0000", 2023-07-02T14:04:16.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6369999999999999e-05 2023-07-02T14:04:16.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795264+0000", 2023-07-02T14:04:16.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1052e-05 2023-07-02T14:04:16.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:16.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795275+0000", 2023-07-02T14:04:16.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.343850459 2023-07-02T14:04:16.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139125+0000", 2023-07-02T14:04:16.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6760000000000004e-05 2023-07-02T14:04:16.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:16.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139202+0000", 2023-07-02T14:04:16.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015092799999999999 2023-07-02T14:04:16.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:16.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139353+0000", 2023-07-02T14:04:16.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011425940000000001 2023-07-02T14:04:16.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:16.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150779+0000", 2023-07-02T14:04:16.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033223060000000001 2023-07-02T14:04:16.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:16.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154101+0000", 2023-07-02T14:04:16.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9268999999999998e-05 2023-07-02T14:04:16.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:16.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154161+0000", 2023-07-02T14:04:16.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6499999999999999e-05 2023-07-02T14:04:16.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:16.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154207+0000", 2023-07-02T14:04:16.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999 2023-07-02T14:04:16.434 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.434 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.434 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.435 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3982 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1835008~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:16.435 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:16.435 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.493351154000003, 2023-07-02T14:04:16.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002, 2023-07-02T14:04:16.435 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:16.435 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:16.436 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:16.436 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:16.436 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:16.436 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3982 2023-07-02T14:04:16.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.436 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:16.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:16.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:16.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:16.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:16.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:16.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:16.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:16.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795233+0000", 2023-07-02T14:04:16.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8680999999999998e-05 2023-07-02T14:04:16.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:16.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795272+0000", 2023-07-02T14:04:16.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5629999999999999e-06 2023-07-02T14:04:16.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:16.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795277+0000", 2023-07-02T14:04:16.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8700000000000001e-06 2023-07-02T14:04:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795280+0000", 2023-07-02T14:04:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1964000000000001e-05 2023-07-02T14:04:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795312+0000", 2023-07-02T14:04:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.314e-06 2023-07-02T14:04:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795319+0000", 2023-07-02T14:04:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34828442599999998 2023-07-02T14:04:16.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143603+0000", 2023-07-02T14:04:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2438000000000001e-05 2023-07-02T14:04:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143665+0000", 2023-07-02T14:04:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3079999999999997e-05 2023-07-02T14:04:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143759+0000", 2023-07-02T14:04:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070633079999999999 2023-07-02T14:04:16.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150822+0000", 2023-07-02T14:04:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034430210000000001 2023-07-02T14:04:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:16.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154265+0000", 2023-07-02T14:04:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4249999999999998e-05 2023-07-02T14:04:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154309+0000", 2023-07-02T14:04:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2020000000000001e-05 2023-07-02T14:04:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154351+0000", 2023-07-02T14:04:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002 2023-07-02T14:04:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4054 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1900544~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.493299483000001, 2023-07-02T14:04:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002, 2023-07-02T14:04:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:16.447 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:16.448 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:16.448 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:16.448 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4054 2023-07-02T14:04:16.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.448 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:16.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:16.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:16.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:16.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:16.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:16.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:16.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:16.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:16.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018874860000000001 2023-07-02T14:04:16.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:16.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797172+0000", 2023-07-02T14:04:16.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3000000000000007e-06 2023-07-02T14:04:16.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:16.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797182+0000", 2023-07-02T14:04:16.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6260000000000003e-06 2023-07-02T14:04:16.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:16.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797186+0000", 2023-07-02T14:04:16.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9013e-05 2023-07-02T14:04:16.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797235+0000", 2023-07-02T14:04:16.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5990000000000008e-06 2023-07-02T14:04:16.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:16.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797245+0000", 2023-07-02T14:04:16.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34785621999999999 2023-07-02T14:04:16.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145101+0000", 2023-07-02T14:04:16.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7762000000000006e-05 2023-07-02T14:04:16.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:16.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145169+0000", 2023-07-02T14:04:16.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013469400000000001 2023-07-02T14:04:16.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:16.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145304+0000", 2023-07-02T14:04:16.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089538039999999992 2023-07-02T14:04:16.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:16.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154257+0000", 2023-07-02T14:04:16.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104218 2023-07-02T14:04:16.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:16.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154362+0000", 2023-07-02T14:04:16.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4168000000000001e-05 2023-07-02T14:04:16.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:16.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154406+0000", 2023-07-02T14:04:16.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3098000000000001e-05 2023-07-02T14:04:16.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:16.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154429+0000", 2023-07-02T14:04:16.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002 2023-07-02T14:04:16.458 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.458 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.458 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.458 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4056 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3670016~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:16.458 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:16.459 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.49139297, 2023-07-02T14:04:16.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997, 2023-07-02T14:04:16.459 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:16.459 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:16.459 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:16.459 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:16.459 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:16.459 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4056 2023-07-02T14:04:16.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.460 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:16.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:16.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:16.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:16.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:16.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:16.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:16.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:16.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797191+0000", 2023-07-02T14:04:16.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0438999999999997e-05 2023-07-02T14:04:16.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:16.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797232+0000", 2023-07-02T14:04:16.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6929999999999998e-06 2023-07-02T14:04:16.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:16.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797237+0000", 2023-07-02T14:04:16.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9739999999999998e-06 2023-07-02T14:04:16.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:16.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797240+0000", 2023-07-02T14:04:16.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0243000000000002e-05 2023-07-02T14:04:16.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797270+0000", 2023-07-02T14:04:16.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9889999999999997e-06 2023-07-02T14:04:16.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:16.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797276+0000", 2023-07-02T14:04:16.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35358319799999999 2023-07-02T14:04:16.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150859+0000", 2023-07-02T14:04:16.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1357e-05 2023-07-02T14:04:16.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:16.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150940+0000", 2023-07-02T14:04:16.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014903 2023-07-02T14:04:16.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:16.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151089+0000", 2023-07-02T14:04:16.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033512519999999999 2023-07-02T14:04:16.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:16.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154441+0000", 2023-07-02T14:04:16.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001050945 2023-07-02T14:04:16.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:16.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155492+0000", 2023-07-02T14:04:16.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2523e-05 2023-07-02T14:04:16.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:16.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155524+0000", 2023-07-02T14:04:16.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.057e-05 2023-07-02T14:04:16.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:16.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155555+0000", 2023-07-02T14:04:16.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997 2023-07-02T14:04:16.469 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.470 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.470 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.470 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4088 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3735552~458752] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:16.470 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:16.470 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.491340774999998, 2023-07-02T14:04:16.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998, 2023-07-02T14:04:16.471 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:16.471 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:16.471 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:16.471 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:16.471 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:16.471 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4088 2023-07-02T14:04:16.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.471 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:16.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:16.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:16.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:16.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:16.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:16.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:16.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:16.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:16.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00038503499999999999 2023-07-02T14:04:16.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:16.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797629+0000", 2023-07-02T14:04:16.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8739999999999995e-06 2023-07-02T14:04:16.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:16.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797637+0000", 2023-07-02T14:04:16.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7160000000000001e-06 2023-07-02T14:04:16.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:16.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797640+0000", 2023-07-02T14:04:16.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4342999999999998e-05 2023-07-02T14:04:16.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797695+0000", 2023-07-02T14:04:16.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0939999999999996e-06 2023-07-02T14:04:16.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:16.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797701+0000", 2023-07-02T14:04:16.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35408540799999999 2023-07-02T14:04:16.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151786+0000", 2023-07-02T14:04:16.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1953000000000006e-05 2023-07-02T14:04:16.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:16.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151868+0000", 2023-07-02T14:04:16.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015162800000000001 2023-07-02T14:04:16.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:16.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.152020+0000", 2023-07-02T14:04:16.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054561870000000004 2023-07-02T14:04:16.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:16.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.157476+0000", 2023-07-02T14:04:16.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059153199999999997 2023-07-02T14:04:16.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:16.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158068+0000", 2023-07-02T14:04:16.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2799e-05 2023-07-02T14:04:16.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:16.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158090+0000", 2023-07-02T14:04:16.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7235999999999999e-05 2023-07-02T14:04:16.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:16.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158108+0000", 2023-07-02T14:04:16.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998 2023-07-02T14:04:16.481 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.481 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.481 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.482 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4110 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:16.482 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:16.482 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.490939498000003, 2023-07-02T14:04:16.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864, 2023-07-02T14:04:16.482 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:16.482 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:16.482 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:16.483 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:16.483 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:16.483 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4110 2023-07-02T14:04:16.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.483 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:16.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:16.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:16.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:16.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:16.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:16.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:16.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:16.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:16.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9331e-05 2023-07-02T14:04:16.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:16.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797684+0000", 2023-07-02T14:04:16.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.147e-06 2023-07-02T14:04:16.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:16.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797689+0000", 2023-07-02T14:04:16.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5020000000000001e-06 2023-07-02T14:04:16.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:16.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797693+0000", 2023-07-02T14:04:16.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5254999999999999e-05 2023-07-02T14:04:16.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797748+0000", 2023-07-02T14:04:16.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5520000000000005e-06 2023-07-02T14:04:16.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:16.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797757+0000", 2023-07-02T14:04:16.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35674245700000001 2023-07-02T14:04:16.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154499+0000", 2023-07-02T14:04:16.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0090000000000001e-06 2023-07-02T14:04:16.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:16.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154506+0000", 2023-07-02T14:04:16.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864 2023-07-02T14:04:16.490 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.490 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.490 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.490 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:16.490 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:16.491 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.490887411999999, 2023-07-02T14:04:16.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998, 2023-07-02T14:04:16.491 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:16.491 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:16.491 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:16.491 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:16.491 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:16.492 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:04:16.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.492 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:16.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:16.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:16.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:16.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:16.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:16.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:16.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:16.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:16.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001714819 2023-07-02T14:04:16.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:16.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799412+0000", 2023-07-02T14:04:16.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7779999999999996e-06 2023-07-02T14:04:16.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:16.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799421+0000", 2023-07-02T14:04:16.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3510000000000002e-06 2023-07-02T14:04:16.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:16.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799424+0000", 2023-07-02T14:04:16.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0275999999999999e-05 2023-07-02T14:04:16.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799464+0000", 2023-07-02T14:04:16.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8389999999999996e-06 2023-07-02T14:04:16.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:16.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799473+0000", 2023-07-02T14:04:16.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35507638800000002 2023-07-02T14:04:16.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:16.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154550+0000", 2023-07-02T14:04:16.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6309999999999998e-06 2023-07-02T14:04:16.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:16.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:16.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:16.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154559+0000", 2023-07-02T14:04:16.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998 2023-07-02T14:04:16.498 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.498 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.498 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.499 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:16.499 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:16.499 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:16.500 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:04:16.501 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:16.500+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:16.601 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:16.600+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:16.665 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:16.665 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:16.675 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:04:16.716 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:16.715+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:16.720 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:04:16.802 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:16.802+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:16.840 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:16.841 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:16.841 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:16.841 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:16.841 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:16.852 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:04:16.900 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:16.903 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:16.902+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:16.918 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:16.936 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:16.955 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:16.973 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:16.974 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:04:16.974 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:04:16.974 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:04:16.974 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:16.974 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:04:16.974 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:04:16.976 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:16.977 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:16.980 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:16.980 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:16.990 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:04:17.031 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:17.030+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:17.096 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:04:17.104 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:17.103+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:17.127 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:17.127 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:17.138 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:04:17.205 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:17.204+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:17.262 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:17.262 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:17.272 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:04:17.313 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:17.312+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:17.407 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:17.406+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:17.435 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:17.436 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:17.446 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:04:17.508 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:17.507+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:17.570 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:17.570 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:17.580 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:04:17.622 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:17.621+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:17.709 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:17.708+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:17.744 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:17.745 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:17.755 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:17.810 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:17.809+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:17.910 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:17.910+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:17.918 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:17.919 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:17.929 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:18.011 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:18.010+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:18.065 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:18.065 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:18.076 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:04:18.111 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:18.110+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:18.212 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:18.212+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:18.240 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:18.241 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:18.251 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:04:18.313 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:18.313+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:18.374 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:18.375 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:18.385 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:04:18.428 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:18.427+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:18.508 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:18.509 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:18.515 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:18.515+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:18.519 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:04:18.615 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:18.614+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:18.631 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4] out_osds: [5, 2] dead_osds: [3] live_osds: [0, 1, 2, 4, 5] 2023-07-02T14:04:18.631 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:04:18.631 INFO:tasks.thrashosds.thrasher:Adding osd 2 2023-07-02T14:04:18.632 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 2 2023-07-02T14:04:18.684 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:18.684 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:18.695 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:04:18.738 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:18.737+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:18.817 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:18.817+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:18.820 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:18.820 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:18.831 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:04:18.918 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:18.917+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:18.949 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-07-02T14:04:18.947+0000 7ff490b05700 -1 mon.a@0(leader).osd e100 definitely_dead 0 2023-07-02T14:04:18.978 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:04:18.996 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:18.996 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:19.007 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:04:19.018 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:19.018+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:19.108 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:04:19.109 INFO:teuthology.orchestra.run.smithi016.stderr:marked in osd.2. 2023-07-02T14:04:19.121 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap19 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap20 b3fbb028-e267-4d25-a750-93ba18a2316a@snap20.diff 2023-07-02T14:04:19.122 INFO:tasks.thrashosds.thrasher:Added osd 2 2023-07-02T14:04:19.122 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:19.121+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:19.124 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:19.143 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:19.162 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:19.173 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 100% complete...done. 2023-07-02T14:04:19.176 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:04:19.176 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=21 2023-07-02T14:04:19.176 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap21 2023-07-02T14:04:19.194 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:19.194 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:19.194 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:19.195 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:19.195 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:19.195 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:19.207 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:04:19.216 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:04:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:04:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:04:19.217 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:19.217 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:04:19.217 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:04:19.219 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:19.220 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:19.221 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:19.220+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:19.322 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:19.321+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:19.344 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:19.344 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:19.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:19.344 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:19.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.345 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:183 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:19.345 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:04:19.345 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 100.05653666000001, 2023-07-02T14:04:19.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003, 2023-07-02T14:04:19.345 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:19.345 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:19.345 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:19.346 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:19.346 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:19.346 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 183 2023-07-02T14:04:19.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.346 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:19.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:19.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:04:19.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:19.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:19.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:04:19.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:19.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:19.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285539+0000", 2023-07-02T14:04:19.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9790999999999997e-05 2023-07-02T14:04:19.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:19.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285598+0000", 2023-07-02T14:04:19.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5809999999999996e-06 2023-07-02T14:04:19.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:19.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285604+0000", 2023-07-02T14:04:19.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7599999999999998e-06 2023-07-02T14:04:19.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:19.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285607+0000", 2023-07-02T14:04:19.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021108188999999999 2023-07-02T14:04:19.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306715+0000", 2023-07-02T14:04:19.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6548e-05 2023-07-02T14:04:19.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:19.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306792+0000", 2023-07-02T14:04:19.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000173869 2023-07-02T14:04:19.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:19.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.306965+0000", 2023-07-02T14:04:19.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060498009999999996 2023-07-02T14:04:19.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:19.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313015+0000", 2023-07-02T14:04:19.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049443719999999998 2023-07-02T14:04:19.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:19.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317960+0000", 2023-07-02T14:04:19.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5927999999999998e-05 2023-07-02T14:04:19.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:19.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318006+0000", 2023-07-02T14:04:19.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6004999999999998e-05 2023-07-02T14:04:19.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:19.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318042+0000", 2023-07-02T14:04:19.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032502304000000003 2023-07-02T14:04:19.355 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.355 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.355 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.355 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:186 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:19.355 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:04:19.355 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 100.05646583399999, 2023-07-02T14:04:19.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003, 2023-07-02T14:04:19.356 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:19.356 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:19.356 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:19.356 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:19.356 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:19.356 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 186 2023-07-02T14:04:19.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.357 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:19.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:19.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:04:19.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:19.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:19.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:04:19.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:19.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:19.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:04:19.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4279e-05 2023-07-02T14:04:19.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:19.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285664+0000", 2023-07-02T14:04:19.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3789999999999995e-06 2023-07-02T14:04:19.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:19.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285673+0000", 2023-07-02T14:04:19.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499999999999998e-06 2023-07-02T14:04:19.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:19.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285678+0000", 2023-07-02T14:04:19.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023261947000000002 2023-07-02T14:04:19.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.308940+0000", 2023-07-02T14:04:19.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010323399999999999 2023-07-02T14:04:19.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:19.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309044+0000", 2023-07-02T14:04:19.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018878300000000001 2023-07-02T14:04:19.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:19.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309232+0000", 2023-07-02T14:04:19.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037874150000000001 2023-07-02T14:04:19.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:19.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313020+0000", 2023-07-02T14:04:19.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050976140000000003 2023-07-02T14:04:19.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:19.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318117+0000", 2023-07-02T14:04:19.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3182000000000001e-05 2023-07-02T14:04:19.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:19.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:04:19.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5055999999999998e-05 2023-07-02T14:04:19.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:19.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318196+0000", 2023-07-02T14:04:19.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003 2023-07-02T14:04:19.365 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.365 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.365 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.366 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:189 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:19.366 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:04:19.366 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 100.056392071, 2023-07-02T14:04:19.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997, 2023-07-02T14:04:19.366 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:19.366 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:19.366 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:19.366 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:19.367 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:19.367 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:04:19.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.367 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:19.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:19.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:04:19.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:19.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:19.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:04:19.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:19.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:19.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285683+0000", 2023-07-02T14:04:19.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8451999999999999e-05 2023-07-02T14:04:19.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:19.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285782+0000", 2023-07-02T14:04:19.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3650000000000004e-06 2023-07-02T14:04:19.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:19.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285790+0000", 2023-07-02T14:04:19.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8230000000000002e-06 2023-07-02T14:04:19.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:19.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285795+0000", 2023-07-02T14:04:19.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024619367999999999 2023-07-02T14:04:19.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310414+0000", 2023-07-02T14:04:19.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014244600000000001 2023-07-02T14:04:19.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:19.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310557+0000", 2023-07-02T14:04:19.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020612699999999999 2023-07-02T14:04:19.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:19.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310763+0000", 2023-07-02T14:04:19.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003427687 2023-07-02T14:04:19.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:19.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314191+0000", 2023-07-02T14:04:19.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040492089999999998 2023-07-02T14:04:19.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:19.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318240+0000", 2023-07-02T14:04:19.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0287999999999999e-05 2023-07-02T14:04:19.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:19.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318270+0000", 2023-07-02T14:04:19.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0868000000000002e-05 2023-07-02T14:04:19.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:19.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318311+0000", 2023-07-02T14:04:19.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997 2023-07-02T14:04:19.375 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.375 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.376 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.376 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:192 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:19.376 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:04:19.376 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 100.056275298, 2023-07-02T14:04:19.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003, 2023-07-02T14:04:19.376 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:19.377 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:19.377 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:19.377 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:19.377 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:19.377 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 192 2023-07-02T14:04:19.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.377 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:19.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:19.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:04:19.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:19.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:19.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:04:19.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:19.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:19.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285800+0000", 2023-07-02T14:04:19.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100417 2023-07-02T14:04:19.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:19.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285900+0000", 2023-07-02T14:04:19.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8244999999999999e-05 2023-07-02T14:04:19.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:19.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285919+0000", 2023-07-02T14:04:19.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2499999999999997e-06 2023-07-02T14:04:19.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:19.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285924+0000", 2023-07-02T14:04:19.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025288155999999999 2023-07-02T14:04:19.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311212+0000", 2023-07-02T14:04:19.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4073999999999996e-05 2023-07-02T14:04:19.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:19.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311286+0000", 2023-07-02T14:04:19.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016219699999999999 2023-07-02T14:04:19.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:19.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311448+0000", 2023-07-02T14:04:19.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066020599999999999 2023-07-02T14:04:19.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:19.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318050+0000", 2023-07-02T14:04:19.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000327986 2023-07-02T14:04:19.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:19.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318378+0000", 2023-07-02T14:04:19.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8693999999999998e-05 2023-07-02T14:04:19.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:19.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318427+0000", 2023-07-02T14:04:19.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6656999999999998e-05 2023-07-02T14:04:19.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:19.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318464+0000", 2023-07-02T14:04:19.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003 2023-07-02T14:04:19.386 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.386 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.386 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.386 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:195 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:19.386 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:04:19.387 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 100.05614379399999, 2023-07-02T14:04:19.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001, 2023-07-02T14:04:19.387 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:19.387 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:19.387 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:19.387 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:19.387 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:19.387 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 195 2023-07-02T14:04:19.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.388 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:19.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:19.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:04:19.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:19.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:19.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:04:19.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:19.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:19.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285931+0000", 2023-07-02T14:04:19.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4530999999999995e-05 2023-07-02T14:04:19.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:19.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286006+0000", 2023-07-02T14:04:19.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4113e-05 2023-07-02T14:04:19.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:19.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286020+0000", 2023-07-02T14:04:19.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7459999999999997e-06 2023-07-02T14:04:19.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:19.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286026+0000", 2023-07-02T14:04:19.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025997769 2023-07-02T14:04:19.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312023+0000", 2023-07-02T14:04:19.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4403000000000001e-05 2023-07-02T14:04:19.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:19.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312078+0000", 2023-07-02T14:04:19.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020581399999999999 2023-07-02T14:04:19.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:19.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312284+0000", 2023-07-02T14:04:19.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005771043 2023-07-02T14:04:19.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:19.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318055+0000", 2023-07-02T14:04:19.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00046015200000000002 2023-07-02T14:04:19.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:19.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318515+0000", 2023-07-02T14:04:19.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9444000000000003e-05 2023-07-02T14:04:19.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:19.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318564+0000", 2023-07-02T14:04:19.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2141000000000002e-05 2023-07-02T14:04:19.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:19.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318606+0000", 2023-07-02T14:04:19.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001 2023-07-02T14:04:19.396 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.396 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.396 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.397 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:198 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:19.397 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:19.397 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 100.056043835, 2023-07-02T14:04:19.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999, 2023-07-02T14:04:19.397 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:19.397 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:19.397 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:19.398 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:19.398 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:19.398 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 198 2023-07-02T14:04:19.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.398 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:19.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:19.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:19.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:19.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:19.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:19.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:19.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:19.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286031+0000", 2023-07-02T14:04:19.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00049286700000000002 2023-07-02T14:04:19.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:19.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286524+0000", 2023-07-02T14:04:19.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4938e-05 2023-07-02T14:04:19.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:19.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286539+0000", 2023-07-02T14:04:19.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0680000000000002e-06 2023-07-02T14:04:19.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:19.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286545+0000", 2023-07-02T14:04:19.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026497718 2023-07-02T14:04:19.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313043+0000", 2023-07-02T14:04:19.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0884999999999998e-05 2023-07-02T14:04:19.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:19.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313094+0000", 2023-07-02T14:04:19.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014109999999999999 2023-07-02T14:04:19.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:19.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313235+0000", 2023-07-02T14:04:19.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048258980000000003 2023-07-02T14:04:19.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:19.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318061+0000", 2023-07-02T14:04:19.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014346350000000001 2023-07-02T14:04:19.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:19.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319495+0000", 2023-07-02T14:04:19.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1986e-05 2023-07-02T14:04:19.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:19.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319547+0000", 2023-07-02T14:04:19.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0457e-05 2023-07-02T14:04:19.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:19.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319578+0000", 2023-07-02T14:04:19.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999 2023-07-02T14:04:19.406 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.407 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.407 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.407 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:201 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:19.407 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:04:19.407 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 100.05552341000001, 2023-07-02T14:04:19.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003, 2023-07-02T14:04:19.408 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:19.408 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:19.408 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:19.408 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:19.408 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:19.408 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 201 2023-07-02T14:04:19.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.408 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:19.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:19.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:04:19.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:19.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:19.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:04:19.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:19.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:19.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:04:19.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019213999999999999 2023-07-02T14:04:19.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:19.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286744+0000", 2023-07-02T14:04:19.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5279999999999992e-06 2023-07-02T14:04:19.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:19.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286753+0000", 2023-07-02T14:04:19.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4200000000000004e-06 2023-07-02T14:04:19.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:19.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286760+0000", 2023-07-02T14:04:19.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026818386999999999 2023-07-02T14:04:19.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313578+0000", 2023-07-02T14:04:19.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015965 2023-07-02T14:04:19.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:19.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313738+0000", 2023-07-02T14:04:19.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015767299999999999 2023-07-02T14:04:19.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:19.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313895+0000", 2023-07-02T14:04:19.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041714930000000001 2023-07-02T14:04:19.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:19.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318067+0000", 2023-07-02T14:04:19.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001561698 2023-07-02T14:04:19.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:19.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319629+0000", 2023-07-02T14:04:19.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2346000000000003e-05 2023-07-02T14:04:19.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:19.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319681+0000", 2023-07-02T14:04:19.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4691000000000003e-05 2023-07-02T14:04:19.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:19.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319726+0000", 2023-07-02T14:04:19.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003 2023-07-02T14:04:19.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.417 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.417 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:204 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:19.418 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:04:19.418 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 100.055308572, 2023-07-02T14:04:19.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998, 2023-07-02T14:04:19.418 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:19.418 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:19.418 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:19.418 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:19.418 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:19.419 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 204 2023-07-02T14:04:19.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.419 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:19.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:19.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:04:19.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:19.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:19.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:04:19.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:19.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:19.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:04:19.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2497999999999998e-05 2023-07-02T14:04:19.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:19.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286819+0000", 2023-07-02T14:04:19.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0099999999999995e-06 2023-07-02T14:04:19.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:19.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286827+0000", 2023-07-02T14:04:19.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000003e-06 2023-07-02T14:04:19.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:19.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286833+0000", 2023-07-02T14:04:19.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027384716999999999 2023-07-02T14:04:19.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314218+0000", 2023-07-02T14:04:19.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6204000000000004e-05 2023-07-02T14:04:19.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:19.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314294+0000", 2023-07-02T14:04:19.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015757499999999999 2023-07-02T14:04:19.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:19.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314452+0000", 2023-07-02T14:04:19.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036215560000000002 2023-07-02T14:04:19.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:19.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318073+0000", 2023-07-02T14:04:19.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017023979999999999 2023-07-02T14:04:19.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:19.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319776+0000", 2023-07-02T14:04:19.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9561000000000002e-05 2023-07-02T14:04:19.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:19.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319825+0000", 2023-07-02T14:04:19.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0329e-05 2023-07-02T14:04:19.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:19.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319866+0000", 2023-07-02T14:04:19.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998 2023-07-02T14:04:19.427 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.427 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.427 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.428 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:207 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:19.428 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:19.428 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 100.055233927, 2023-07-02T14:04:19.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003, 2023-07-02T14:04:19.428 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:19.428 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:19.428 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:19.429 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:19.429 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:19.429 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 207 2023-07-02T14:04:19.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.429 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:19.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:19.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:19.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:19.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:19.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:19.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:19.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:19.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286841+0000", 2023-07-02T14:04:19.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00068375100000000002 2023-07-02T14:04:19.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:19.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287525+0000", 2023-07-02T14:04:19.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1601e-05 2023-07-02T14:04:19.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:19.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287537+0000", 2023-07-02T14:04:19.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6409999999999996e-06 2023-07-02T14:04:19.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:19.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287543+0000", 2023-07-02T14:04:19.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027245844000000002 2023-07-02T14:04:19.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314789+0000", 2023-07-02T14:04:19.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8910000000000003e-05 2023-07-02T14:04:19.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:19.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314858+0000", 2023-07-02T14:04:19.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030992599999999997 2023-07-02T14:04:19.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:19.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315168+0000", 2023-07-02T14:04:19.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029122869999999999 2023-07-02T14:04:19.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:19.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318080+0000", 2023-07-02T14:04:19.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019963020000000001 2023-07-02T14:04:19.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:19.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320077+0000", 2023-07-02T14:04:19.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5116000000000005e-05 2023-07-02T14:04:19.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:19.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320142+0000", 2023-07-02T14:04:19.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3793000000000004e-05 2023-07-02T14:04:19.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:19.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320205+0000", 2023-07-02T14:04:19.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003 2023-07-02T14:04:19.437 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.437 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.438 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.438 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:236 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2424832~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:19.438 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:19.438 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 100.054521744, 2023-07-02T14:04:19.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003, 2023-07-02T14:04:19.438 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:19.439 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:19.439 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:19.439 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:19.439 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:19.439 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2023-07-02T14:04:19.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.439 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:19.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:19.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:19.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:19.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:19.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:19.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:19.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:19.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287553+0000", 2023-07-02T14:04:19.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0667e-05 2023-07-02T14:04:19.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:19.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287614+0000", 2023-07-02T14:04:19.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0454e-05 2023-07-02T14:04:19.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:19.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287624+0000", 2023-07-02T14:04:19.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8564999999999999e-05 2023-07-02T14:04:19.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:19.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287643+0000", 2023-07-02T14:04:19.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027977596 2023-07-02T14:04:19.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315621+0000", 2023-07-02T14:04:19.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000201728 2023-07-02T14:04:19.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:19.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315822+0000", 2023-07-02T14:04:19.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018436599999999999 2023-07-02T14:04:19.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:19.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316007+0000", 2023-07-02T14:04:19.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042423440000000003 2023-07-02T14:04:19.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:19.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320249+0000", 2023-07-02T14:04:19.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001452633 2023-07-02T14:04:19.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:19.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321702+0000", 2023-07-02T14:04:19.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3616000000000003e-05 2023-07-02T14:04:19.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:19.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321755+0000", 2023-07-02T14:04:19.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840999999999999e-05 2023-07-02T14:04:19.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:19.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321793+0000", 2023-07-02T14:04:19.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003 2023-07-02T14:04:19.448 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.448 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.448 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.448 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:251 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:19.448 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:19.449 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 100.05442620700001, 2023-07-02T14:04:19.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001, 2023-07-02T14:04:19.449 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:19.449 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:19.449 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:19.449 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:19.449 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:19.449 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 251 2023-07-02T14:04:19.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.450 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:19.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:19.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:19.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:19.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:19.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:19.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:19.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:19.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287649+0000", 2023-07-02T14:04:19.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3232000000000001e-05 2023-07-02T14:04:19.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:19.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287702+0000", 2023-07-02T14:04:19.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0010000000000007e-06 2023-07-02T14:04:19.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:19.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287711+0000", 2023-07-02T14:04:19.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3569999999999997e-06 2023-07-02T14:04:19.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:19.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287717+0000", 2023-07-02T14:04:19.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028686454 2023-07-02T14:04:19.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316403+0000", 2023-07-02T14:04:19.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7741000000000002e-05 2023-07-02T14:04:19.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:19.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316461+0000", 2023-07-02T14:04:19.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4313000000000004e-05 2023-07-02T14:04:19.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:19.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316545+0000", 2023-07-02T14:04:19.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003734032 2023-07-02T14:04:19.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:19.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320279+0000", 2023-07-02T14:04:19.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015237219999999999 2023-07-02T14:04:19.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:19.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321803+0000", 2023-07-02T14:04:19.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1927e-05 2023-07-02T14:04:19.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:19.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321845+0000", 2023-07-02T14:04:19.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5398999999999998e-05 2023-07-02T14:04:19.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:19.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321880+0000", 2023-07-02T14:04:19.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001 2023-07-02T14:04:19.458 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.458 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.458 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.459 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:255 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2555904~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:19.459 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:19.459 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 100.054011479, 2023-07-02T14:04:19.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002, 2023-07-02T14:04:19.459 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:19.459 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:19.459 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:19.459 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:19.460 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:19.460 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 255 2023-07-02T14:04:19.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.460 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:19.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:19.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:19.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:19.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:19.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:19.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:19.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:19.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288063+0000", 2023-07-02T14:04:19.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4459999999999997e-05 2023-07-02T14:04:19.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:19.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288118+0000", 2023-07-02T14:04:19.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0998999999999999e-05 2023-07-02T14:04:19.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:19.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288129+0000", 2023-07-02T14:04:19.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3219999999999998e-06 2023-07-02T14:04:19.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:19.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288135+0000", 2023-07-02T14:04:19.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029137386000000001 2023-07-02T14:04:19.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317273+0000", 2023-07-02T14:04:19.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7819999999999997e-05 2023-07-02T14:04:19.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:19.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317350+0000", 2023-07-02T14:04:19.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000174864 2023-07-02T14:04:19.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:19.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317525+0000", 2023-07-02T14:04:19.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027943519999999999 2023-07-02T14:04:19.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:19.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320320+0000", 2023-07-02T14:04:19.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015681460000000001 2023-07-02T14:04:19.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:19.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321888+0000", 2023-07-02T14:04:19.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9508999999999997e-05 2023-07-02T14:04:19.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:19.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321927+0000", 2023-07-02T14:04:19.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5428e-05 2023-07-02T14:04:19.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:19.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321963+0000", 2023-07-02T14:04:19.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002 2023-07-02T14:04:19.468 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.468 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.469 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.469 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:446 2.1 2:910b2a90:::rbd_data.125de1449605.000000000000010a:head [write 65536~3211264] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:19.469 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:04:19.469 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 99.976731727000001, 2023-07-02T14:04:19.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003, 2023-07-02T14:04:19.469 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:19.470 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:19.470 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:19.470 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:19.470 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:19.470 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 446 2023-07-02T14:04:19.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.470 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:19.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:19.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:04:19.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:19.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:19.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:04:19.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:19.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:19.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.365344+0000", 2023-07-02T14:04:19.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015907372999999999 2023-07-02T14:04:19.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:19.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381251+0000", 2023-07-02T14:04:19.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5527000000000001e-05 2023-07-02T14:04:19.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:19.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381277+0000", 2023-07-02T14:04:19.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9958000000000001e-05 2023-07-02T14:04:19.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:19.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381297+0000", 2023-07-02T14:04:19.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4941000000000001e-05 2023-07-02T14:04:19.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381352+0000", 2023-07-02T14:04:19.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011565699999999999 2023-07-02T14:04:19.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:19.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381467+0000", 2023-07-02T14:04:19.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018039799999999999 2023-07-02T14:04:19.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:19.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.381648+0000", 2023-07-02T14:04:19.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088183559999999994 2023-07-02T14:04:19.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:19.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.390466+0000", 2023-07-02T14:04:19.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007278278 2023-07-02T14:04:19.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:19.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397744+0000", 2023-07-02T14:04:19.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9289000000000001e-05 2023-07-02T14:04:19.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:19.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397804+0000", 2023-07-02T14:04:19.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0479000000000002e-05 2023-07-02T14:04:19.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:19.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.397854+0000", 2023-07-02T14:04:19.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032509871000000003 2023-07-02T14:04:19.479 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.479 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.479 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.479 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:04:19.479 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:19.479 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.234400933000003, 2023-07-02T14:04:19.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:04:19.480 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:19.480 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:19.480 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:19.480 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:19.480 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:19.480 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:04:19.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.481 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:19.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:19.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:19.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:19.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:19.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:19.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:19.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:19.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:04:19.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:04:19.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:19.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:04:19.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:04:19.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:19.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:04:19.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:04:19.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:19.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:04:19.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:04:19.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:04:19.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:04:19.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:19.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:04:19.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:04:19.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:04:19.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:04:19.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:19.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:04:19.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:04:19.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:19.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:04:19.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:04:19.488 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.488 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.488 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.488 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:04:19.488 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:19.488 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.234340195000001, 2023-07-02T14:04:19.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:04:19.489 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:19.489 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:19.489 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:19.489 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:19.489 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:19.489 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:04:19.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.490 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:19.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:19.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:19.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:19.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:19.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:19.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:19.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:19.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:04:19.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:04:19.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:19.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:04:19.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:04:19.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:19.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:04:19.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:04:19.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:19.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:04:19.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:04:19.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:04:19.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:04:19.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:19.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:04:19.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:04:19.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:04:19.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:04:19.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:19.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:04:19.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:04:19.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:19.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:04:19.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:04:19.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:19.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:04:19.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:04:19.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:19.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:04:19.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:04:19.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:19.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:04:19.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:04:19.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:19.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:04:19.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:04:19.499 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.499 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.499 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.500 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:19.500 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:19.500 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 68.003885865000001, 2023-07-02T14:04:19.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:04:19.500 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:19.500 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:19.501 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:19.501 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:19.501 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:19.501 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:04:19.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.501 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:19.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:19.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:19.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:19.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:19.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:19.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:19.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:19.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:04:19.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:04:19.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:19.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:04:19.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:04:19.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:19.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:04:19.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:04:19.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:19.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:04:19.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:04:19.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:04:19.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:04:19.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:19.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:04:19.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:04:19.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:19.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:04:19.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:04:19.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:19.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:04:19.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:04:19.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:19.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:04:19.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:04:19.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:19.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:04:19.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:04:19.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:19.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:04:19.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:04:19.509 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.510 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.510 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.510 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2941 2.3 2:d3762e63:::rbd_data.125de1449605.00000000000000f8:head [write 65536~2031616] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:19.510 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:19.510 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 67.874782910999997, 2023-07-02T14:04:19.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002, 2023-07-02T14:04:19.511 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:19.511 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:19.511 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:19.511 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:19.511 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:19.511 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2941 2023-07-02T14:04:19.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.512 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:19.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:19.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:19.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:19.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:19.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:19.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:19.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:19.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467291+0000", 2023-07-02T14:04:19.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043022859999999998 2023-07-02T14:04:19.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:19.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471593+0000", 2023-07-02T14:04:19.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0999999999999999e-05 2023-07-02T14:04:19.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:19.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471614+0000", 2023-07-02T14:04:19.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0097e-05 2023-07-02T14:04:19.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:19.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471625+0000", 2023-07-02T14:04:19.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069808499999999998 2023-07-02T14:04:19.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472323+0000", 2023-07-02T14:04:19.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135635 2023-07-02T14:04:19.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:19.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472458+0000", 2023-07-02T14:04:19.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223365 2023-07-02T14:04:19.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:19.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472682+0000", 2023-07-02T14:04:19.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013761944999999999 2023-07-02T14:04:19.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:19.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.486444+0000", 2023-07-02T14:04:19.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016859747000000001 2023-07-02T14:04:19.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:19.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503303+0000", 2023-07-02T14:04:19.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6635000000000001e-05 2023-07-02T14:04:19.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:19.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503380+0000", 2023-07-02T14:04:19.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000161803 2023-07-02T14:04:19.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:19.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503542+0000", 2023-07-02T14:04:19.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002 2023-07-02T14:04:19.520 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.520 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.520 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.520 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2950 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 1507328~786432] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:19.521 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:19.521 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 67.866609603000001, 2023-07-02T14:04:19.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999, 2023-07-02T14:04:19.521 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:19.521 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:19.521 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:19.521 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:19.521 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:19.522 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2950 2023-07-02T14:04:19.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.522 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:19.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:19.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:19.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:19.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:19.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:19.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:19.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:19.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:19.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001147854 2023-07-02T14:04:19.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:19.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476614+0000", 2023-07-02T14:04:19.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.802e-05 2023-07-02T14:04:19.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:19.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476632+0000", 2023-07-02T14:04:19.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1860000000000003e-06 2023-07-02T14:04:19.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:19.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476639+0000", 2023-07-02T14:04:19.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061211240000000004 2023-07-02T14:04:19.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482760+0000", 2023-07-02T14:04:19.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8833999999999996e-05 2023-07-02T14:04:19.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:19.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482859+0000", 2023-07-02T14:04:19.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019270600000000001 2023-07-02T14:04:19.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:19.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.483051+0000", 2023-07-02T14:04:19.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060385630000000003 2023-07-02T14:04:19.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:19.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489090+0000", 2023-07-02T14:04:19.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024506918999999999 2023-07-02T14:04:19.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:19.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513597+0000", 2023-07-02T14:04:19.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9672999999999995e-05 2023-07-02T14:04:19.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:19.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513667+0000", 2023-07-02T14:04:19.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024329500000000001 2023-07-02T14:04:19.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:19.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513910+0000", 2023-07-02T14:04:19.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999 2023-07-02T14:04:19.530 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.530 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.530 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.531 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2951 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2686976~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:19.531 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:19.531 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 67.865428824999995, 2023-07-02T14:04:19.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999, 2023-07-02T14:04:19.531 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:19.531 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:19.532 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:19.532 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:19.532 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:19.532 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2951 2023-07-02T14:04:19.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.532 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:19.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:19.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:19.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:19.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:19.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:19.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:19.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:19.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476646+0000", 2023-07-02T14:04:19.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7639000000000003e-05 2023-07-02T14:04:19.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:19.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476684+0000", 2023-07-02T14:04:19.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3508000000000002e-05 2023-07-02T14:04:19.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:19.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476747+0000", 2023-07-02T14:04:19.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8499999999999994e-06 2023-07-02T14:04:19.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:19.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476755+0000", 2023-07-02T14:04:19.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078346160000000008 2023-07-02T14:04:19.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484590+0000", 2023-07-02T14:04:19.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7206000000000004e-05 2023-07-02T14:04:19.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:19.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484687+0000", 2023-07-02T14:04:19.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016243099999999999 2023-07-02T14:04:19.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:19.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484849+0000", 2023-07-02T14:04:19.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042482860000000004 2023-07-02T14:04:19.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:19.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489097+0000", 2023-07-02T14:04:19.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.02489504 2023-07-02T14:04:19.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:19.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513992+0000", 2023-07-02T14:04:19.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9972000000000001e-05 2023-07-02T14:04:19.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:19.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514052+0000", 2023-07-02T14:04:19.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149264 2023-07-02T14:04:19.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:19.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514202+0000", 2023-07-02T14:04:19.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999 2023-07-02T14:04:19.541 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.541 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.541 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.541 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2952 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2883584~12288] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:19.541 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:19.541 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 67.865314990000002, 2023-07-02T14:04:19.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651, 2023-07-02T14:04:19.542 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:19.542 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:19.542 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:19.542 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:19.542 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:19.542 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2952 2023-07-02T14:04:19.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.543 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:19.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:19.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:19.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:19.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:19.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:19.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:19.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:19.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476760+0000", 2023-07-02T14:04:19.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3664e-05 2023-07-02T14:04:19.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:19.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476774+0000", 2023-07-02T14:04:19.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.166e-06 2023-07-02T14:04:19.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:19.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476783+0000", 2023-07-02T14:04:19.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8720000000000001e-06 2023-07-02T14:04:19.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:19.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476788+0000", 2023-07-02T14:04:19.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087180529999999999 2023-07-02T14:04:19.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:19.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485506+0000", 2023-07-02T14:04:19.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5680999999999994e-05 2023-07-02T14:04:19.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:19.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485602+0000", 2023-07-02T14:04:19.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000222365 2023-07-02T14:04:19.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:19.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485824+0000", 2023-07-02T14:04:19.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003277379 2023-07-02T14:04:19.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:19.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489101+0000", 2023-07-02T14:04:19.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025151739999999999 2023-07-02T14:04:19.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:19.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514253+0000", 2023-07-02T14:04:19.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4749000000000002e-05 2023-07-02T14:04:19.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:19.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514288+0000", 2023-07-02T14:04:19.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130634 2023-07-02T14:04:19.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:19.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:19.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:19.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514419+0000", 2023-07-02T14:04:19.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651 2023-07-02T14:04:19.551 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.551 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.551 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.551 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:19.551 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:19.552 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:19.553 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:04:19.553 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:19.552+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:19.653 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:19.653+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:19.688 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:19.688 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:19.688 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:04:19.689 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:19.698 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:04:19.754 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:19.753+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:19.822 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:19.822 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:19.822 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:04:19.822 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:04:19.822 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:19.832 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:04:19.873 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:19.872+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:19.956 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:19.956+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:19.998 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:19.998 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:04:19.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:04:19.998 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:19.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:19.999 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:04:19.999 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:19.999 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.098699779, 2023-07-02T14:04:19.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:04:19.999 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:19.999 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:19.999 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:20.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:20.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:04:20.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:04:20.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:04:20.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:04:20.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:04:20.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:04:20.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:04:20.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:04:20.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:04:20.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:04:20.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:04:20.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:04:20.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:04:20.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:04:20.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:04:20.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:04:20.006 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.006 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.006 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.007 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:04:20.007 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:20.007 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.094614061000001, 2023-07-02T14:04:20.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:04:20.007 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.007 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.007 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:20.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:20.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:04:20.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:04:20.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:04:20.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:04:20.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:04:20.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:04:20.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:04:20.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:04:20.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:04:20.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:04:20.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:04:20.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:04:20.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:04:20.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:04:20.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:04:20.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:04:20.014 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.014 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.014 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.014 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:04:20.015 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:20.015 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.090749279999997, 2023-07-02T14:04:20.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:04:20.015 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.015 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.015 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:20.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:20.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:20.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:04:20.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:04:20.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:04:20.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:04:20.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:04:20.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:04:20.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:04:20.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:04:20.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:04:20.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:04:20.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:04:20.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:04:20.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:04:20.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:04:20.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:04:20.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:04:20.022 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.022 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.022 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.022 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:04:20.022 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:20.022 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.087225019000002, 2023-07-02T14:04:20.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:04:20.023 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.023 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.023 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:20.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:20.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:04:20.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:04:20.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:04:20.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:04:20.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:04:20.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:04:20.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:04:20.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:04:20.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:04:20.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:04:20.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:04:20.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:04:20.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:04:20.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:04:20.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:04:20.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:04:20.029 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.029 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.030 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.030 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6124 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1328, mlcod=81'1319)", 2023-07-02T14:04:20.030 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:20.030 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.083709636000002, 2023-07-02T14:04:20.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019096974999999999, 2023-07-02T14:04:20.030 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912602+0000", 2023-07-02T14:04:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6741999999999999e-05 2023-07-02T14:04:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912629+0000", 2023-07-02T14:04:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5163e-05 2023-07-02T14:04:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912644+0000", 2023-07-02T14:04:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4800000000000001e-06 2023-07-02T14:04:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912653+0000", 2023-07-02T14:04:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071988909999999998 2023-07-02T14:04:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.919851+0000", 2023-07-02T14:04:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2949999999999999e-05 2023-07-02T14:04:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.919874+0000", 2023-07-02T14:04:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011800804 2023-07-02T14:04:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931675+0000", 2023-07-02T14:04:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6033999999999998e-05 2023-07-02T14:04:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931701+0000", 2023-07-02T14:04:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019096974999999999 2023-07-02T14:04:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6139 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1337, mlcod=81'1333)", 2023-07-02T14:04:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.057019775999997, 2023-07-02T14:04:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018571681, 2023-07-02T14:04:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939293+0000", 2023-07-02T14:04:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003764613 2023-07-02T14:04:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943058+0000", 2023-07-02T14:04:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5797000000000001e-05 2023-07-02T14:04:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943074+0000", 2023-07-02T14:04:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5089999999999996e-06 2023-07-02T14:04:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943081+0000", 2023-07-02T14:04:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014727100000000001 2023-07-02T14:04:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943228+0000", 2023-07-02T14:04:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2246000000000003e-05 2023-07-02T14:04:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943261+0000", 2023-07-02T14:04:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014576317 2023-07-02T14:04:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957837+0000", 2023-07-02T14:04:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8962000000000002e-05 2023-07-02T14:04:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957866+0000", 2023-07-02T14:04:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018571681 2023-07-02T14:04:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:04:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.05152022, 2023-07-02T14:04:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:04:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:04:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:04:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:04:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:04:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:04:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:04:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:04:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:04:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:04:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:04:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:04:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:04:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:04:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:04:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:04:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:04:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:04:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.046862861999998, 2023-07-02T14:04:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:04:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:04:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:04:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:04:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:04:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:04:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:04:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:04:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:04:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:04:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:04:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:04:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:04:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:04:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:04:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:04:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:04:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:04:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.045922998999998, 2023-07-02T14:04:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:04:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:04:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:04:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:04:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:04:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:04:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:04:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:04:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:04:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:04:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:04:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:04:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:04:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:04:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:04:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:04:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:04:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:04:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.044525121, 2023-07-02T14:04:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:04:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:04:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:04:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:04:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:04:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:04:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:04:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:04:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:04:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:04:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:04:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:04:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:04:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:04:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:04:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:04:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:04:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6150 2.0 e81/46 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 81'1345, mlcod=81'1338)", 2023-07-02T14:04:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.036502173999999, 2023-07-02T14:04:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019172812000000001, 2023-07-02T14:04:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959810+0000", 2023-07-02T14:04:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048438650000000001 2023-07-02T14:04:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964653+0000", 2023-07-02T14:04:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2136e-05 2023-07-02T14:04:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964666+0000", 2023-07-02T14:04:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2169999999999999e-06 2023-07-02T14:04:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964672+0000", 2023-07-02T14:04:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00047561800000000003 2023-07-02T14:04:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965147+0000", 2023-07-02T14:04:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5891e-05 2023-07-02T14:04:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965173+0000", 2023-07-02T14:04:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013770973000000001 2023-07-02T14:04:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978944+0000", 2023-07-02T14:04:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0256999999999998e-05 2023-07-02T14:04:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978985+0000", 2023-07-02T14:04:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019172812000000001 2023-07-02T14:04:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:04:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 18.634334625000001, 2023-07-02T14:04:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:04:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:04:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:04:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:04:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:04:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:04:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:04:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:04:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:04:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:04:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:04:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:04:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:04:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:04:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:04:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:04:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:04:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.267984941000002, 2023-07-02T14:04:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:04:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:04:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:04:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:04:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:04:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:04:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:04:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:04:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:04:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:04:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:04:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:04:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:04:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:04:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:04:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:04:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:04:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:04:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:04:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:04:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7706 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 0~53248] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.264058194, 2023-07-02T14:04:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019507096000000002, 2023-07-02T14:04:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7706 2023-07-02T14:04:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732254+0000", 2023-07-02T14:04:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4764999999999997e-05 2023-07-02T14:04:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732289+0000", 2023-07-02T14:04:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0166999999999999e-05 2023-07-02T14:04:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732299+0000", 2023-07-02T14:04:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5310000000000001e-06 2023-07-02T14:04:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732305+0000", 2023-07-02T14:04:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060619599999999999 2023-07-02T14:04:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738367+0000", 2023-07-02T14:04:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011864 2023-07-02T14:04:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738486+0000", 2023-07-02T14:04:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013098291 2023-07-02T14:04:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751584+0000", 2023-07-02T14:04:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7800000000000002e-05 2023-07-02T14:04:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751642+0000", 2023-07-02T14:04:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000121198 2023-07-02T14:04:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751763+0000", 2023-07-02T14:04:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019507096000000002 2023-07-02T14:04:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7707 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 3145728~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.264001474000001, 2023-07-02T14:04:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999, 2023-07-02T14:04:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7707 2023-07-02T14:04:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5017e-05 2023-07-02T14:04:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732347+0000", 2023-07-02T14:04:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5821e-05 2023-07-02T14:04:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732363+0000", 2023-07-02T14:04:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1270000000000002e-06 2023-07-02T14:04:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732368+0000", 2023-07-02T14:04:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008444666 2023-07-02T14:04:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740812+0000", 2023-07-02T14:04:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011266599999999999 2023-07-02T14:04:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740925+0000", 2023-07-02T14:04:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010848808999999999 2023-07-02T14:04:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751774+0000", 2023-07-02T14:04:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8421999999999997e-05 2023-07-02T14:04:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751832+0000", 2023-07-02T14:04:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107453 2023-07-02T14:04:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751940+0000", 2023-07-02T14:04:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999 2023-07-02T14:04:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.263937597000002, 2023-07-02T14:04:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:04:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:04:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:04:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:04:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:04:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:04:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:04:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:04:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:04:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:04:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:04:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:04:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:04:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:04:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:04:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:04:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:04:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:04:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:04:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.259373772, 2023-07-02T14:04:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:04:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:04:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:04:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:04:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:04:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:04:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:04:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:04:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:04:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:04:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:04:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:04:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:04:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:04:20.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:04:20.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:04:20.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:04:20.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:04:20.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:04:20.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:04:20.137 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.137 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.137 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.137 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:20.137 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:20.137 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.259323905999999, 2023-07-02T14:04:20.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:04:20.138 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.138 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.138 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.138 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.138 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.138 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:04:20.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.139 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:20.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:20.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:20.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:04:20.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:04:20.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:04:20.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:04:20.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:04:20.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:04:20.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:04:20.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:04:20.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:04:20.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:04:20.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:04:20.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:04:20.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:04:20.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:04:20.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:04:20.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:04:20.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:04:20.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:04:20.146 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.146 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.146 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.146 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:20.146 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:20.146 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.258284913000001, 2023-07-02T14:04:20.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:04:20.147 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.147 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.147 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.147 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.147 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.147 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:04:20.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.148 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:20.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:20.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:04:20.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:04:20.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:04:20.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:04:20.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:04:20.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:04:20.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:04:20.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:04:20.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:04:20.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:04:20.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:04:20.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:04:20.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:04:20.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:04:20.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:04:20.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:04:20.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:04:20.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:04:20.155 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.155 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.155 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.155 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7993 2.7 2:fe72db53:::rbd_data.125de1449605.0000000000000072:head [write 65536~3276800] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:20.155 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:20.155 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.071326865, 2023-07-02T14:04:20.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919, 2023-07-02T14:04:20.156 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.156 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.156 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.156 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.156 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.156 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7993 2023-07-02T14:04:20.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.157 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:20.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:20.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924986+0000", 2023-07-02T14:04:20.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049542070000000004 2023-07-02T14:04:20.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929940+0000", 2023-07-02T14:04:20.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9740000000000001e-05 2023-07-02T14:04:20.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929960+0000", 2023-07-02T14:04:20.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.311e-05 2023-07-02T14:04:20.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929973+0000", 2023-07-02T14:04:20.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9629000000000002e-05 2023-07-02T14:04:20.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930023+0000", 2023-07-02T14:04:20.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012751799999999999 2023-07-02T14:04:20.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930150+0000", 2023-07-02T14:04:20.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016449042000000001 2023-07-02T14:04:20.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946599+0000", 2023-07-02T14:04:20.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3241e-05 2023-07-02T14:04:20.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946663+0000", 2023-07-02T14:04:20.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000102296 2023-07-02T14:04:20.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946765+0000", 2023-07-02T14:04:20.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919 2023-07-02T14:04:20.163 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.164 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.164 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.164 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.164 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.164 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:20.165 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:04:20.166 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 3 2023-07-02T14:04:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:04:20.167 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:04:20.167 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:20.165+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:20.266 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:20.266+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:20.332 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:20.333 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:20.333 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:04:20.333 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:20.343 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:04:20.386 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:20.385+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:20.466 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:20.466 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:20.466 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:04:20.466 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:04:20.467 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:20.467 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:20.466+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:20.476 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:20.569 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:20.568+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:20.642 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:20.642 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:04:20.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:04:20.642 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:20.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.643 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:20.643 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:20.643 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 101.34919753600001, 2023-07-02T14:04:20.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:04:20.643 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.643 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.644 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.644 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.644 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.644 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:04:20.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.644 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:20.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:20.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:04:20.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:04:20.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:04:20.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:04:20.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:04:20.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:04:20.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:04:20.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:04:20.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:04:20.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:04:20.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:04:20.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:04:20.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:20.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:04:20.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:04:20.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:20.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:04:20.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:04:20.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:04:20.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:04:20.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:04:20.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:04:20.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:04:20.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:04:20.653 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.654 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.654 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.654 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:20.654 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:20.654 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 101.348962513, 2023-07-02T14:04:20.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:04:20.655 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.655 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.655 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.655 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.655 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.655 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:04:20.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.656 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:20.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:20.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:04:20.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:04:20.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:04:20.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:04:20.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:04:20.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:04:20.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:04:20.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:04:20.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:04:20.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:04:20.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:04:20.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:04:20.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:20.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:04:20.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:04:20.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:20.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:04:20.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:04:20.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:04:20.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:04:20.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:04:20.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:04:20.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:04:20.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:04:20.665 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.665 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.665 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.665 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:20.666 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:20.666 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 101.348763379, 2023-07-02T14:04:20.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:04:20.666 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.666 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.666 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.666 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.666 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.667 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:04:20.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.667 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:20.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:20.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:04:20.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:04:20.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:04:20.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:04:20.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:04:20.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:04:20.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:04:20.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:04:20.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:04:20.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:04:20.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:04:20.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:04:20.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:20.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:04:20.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:04:20.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:20.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:04:20.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:04:20.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:04:20.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:04:20.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:04:20.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:04:20.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:04:20.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:04:20.676 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.676 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.676 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.677 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:20.677 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:20.677 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 101.34839825500001, 2023-07-02T14:04:20.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:04:20.677 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.677 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.677 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.678 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.678 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.678 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:04:20.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.678 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:20.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:20.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:20.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:04:20.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:04:20.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:04:20.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:04:20.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:04:20.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:04:20.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:04:20.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:04:20.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:04:20.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:04:20.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:04:20.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:04:20.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:20.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:04:20.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:04:20.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:20.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:04:20.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:04:20.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:04:20.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:04:20.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:04:20.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:04:20.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:04:20.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:04:20.687 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.687 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.687 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.688 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:20.688 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:20.688 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 101.348317194, 2023-07-02T14:04:20.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:04:20.688 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.689 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.689 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.689 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.689 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.689 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:04:20.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.689 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:20.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:20.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:20.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:04:20.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:04:20.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:04:20.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:04:20.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:04:20.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:04:20.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:04:20.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:04:20.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:04:20.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:04:20.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:04:20.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:04:20.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:20.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:04:20.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:04:20.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:20.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:04:20.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:04:20.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:04:20.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:04:20.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:04:20.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:04:20.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:04:20.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:04:20.698 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.699 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.699 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.699 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:20.699 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:20.699 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 101.348225523, 2023-07-02T14:04:20.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:04:20.700 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.700 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.700 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.700 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.700 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.700 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:04:20.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.701 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:20.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:20.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:20.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:04:20.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:04:20.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:04:20.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:04:20.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:04:20.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:04:20.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:04:20.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:04:20.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:04:20.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:04:20.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:04:20.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:04:20.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:20.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:04:20.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:04:20.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:04:20.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:04:20.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:20.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:04:20.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:04:20.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:04:20.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:04:20.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:04:20.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:04:20.710 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.710 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.710 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.710 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:20.710 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:20.711 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 101.347886081, 2023-07-02T14:04:20.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:04:20.711 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.711 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.711 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.711 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.711 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.712 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:04:20.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.712 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:20.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:20.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:20.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:04:20.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:04:20.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:04:20.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:04:20.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:04:20.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:04:20.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:04:20.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:04:20.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:04:20.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:04:20.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:04:20.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:04:20.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:20.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:04:20.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:04:20.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:04:20.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:04:20.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:20.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:04:20.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:04:20.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:04:20.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:04:20.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:04:20.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:04:20.721 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.721 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.721 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.722 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:20.722 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:20.722 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 101.34770002800001, 2023-07-02T14:04:20.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:04:20.722 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.722 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.722 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.722 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.723 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.723 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:04:20.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.723 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:20.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:20.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:20.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:20.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:04:20.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:04:20.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:04:20.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:04:20.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:04:20.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:04:20.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:04:20.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:04:20.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:04:20.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:04:20.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:04:20.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:20.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:04:20.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:04:20.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:04:20.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:04:20.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:20.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:04:20.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:04:20.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:04:20.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:04:20.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:04:20.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:04:20.732 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.732 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.732 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.733 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:20.733 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:20.733 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 101.347542226, 2023-07-02T14:04:20.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:04:20.733 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.733 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.733 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.734 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.734 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.734 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:04:20.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.734 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:20.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:20.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:04:20.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:04:20.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:04:20.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:04:20.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:04:20.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:04:20.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:04:20.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:04:20.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:04:20.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:04:20.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:04:20.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:04:20.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:20.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:04:20.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:04:20.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:04:20.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:04:20.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:20.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:04:20.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:04:20.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:04:20.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:04:20.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:04:20.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:04:20.743 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.743 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.743 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.744 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:20.744 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:20.744 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 101.347350444, 2023-07-02T14:04:20.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:04:20.744 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.744 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.745 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.745 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.745 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.745 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:04:20.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.745 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:20.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:20.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:04:20.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:04:20.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:04:20.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:04:20.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:04:20.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:04:20.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:04:20.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:04:20.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:04:20.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:04:20.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:04:20.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:04:20.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:20.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:04:20.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:04:20.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:20.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:04:20.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:04:20.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:04:20.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:04:20.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:04:20.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:04:20.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:04:20.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:04:20.754 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.754 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.755 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.755 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:20.755 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:20.755 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 101.347174784, 2023-07-02T14:04:20.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:04:20.756 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.756 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.756 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.756 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.756 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.756 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:04:20.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.757 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:20.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:20.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:20.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:04:20.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:04:20.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:04:20.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:04:20.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:04:20.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:04:20.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:04:20.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:04:20.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:04:20.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:04:20.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:04:20.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:20.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:04:20.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:04:20.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:20.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:04:20.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:04:20.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:04:20.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:04:20.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:04:20.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:04:20.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:04:20.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:04:20.766 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.766 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.766 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.766 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:20.766 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:20.767 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 101.347080596, 2023-07-02T14:04:20.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:04:20.767 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.767 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.767 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.767 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.767 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.767 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:04:20.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.768 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:20.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:20.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:20.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:04:20.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:04:20.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:04:20.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:04:20.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:04:20.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:04:20.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:04:20.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:04:20.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:04:20.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:04:20.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:04:20.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:04:20.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:20.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:04:20.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:04:20.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:20.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:04:20.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:04:20.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:04:20.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:04:20.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:04:20.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:04:20.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:04:20.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:04:20.777 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.777 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.777 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.777 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:20.778 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:20.778 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 101.34687721, 2023-07-02T14:04:20.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:04:20.778 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.778 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.778 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.778 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.779 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.779 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:04:20.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.779 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:20.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:20.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:04:20.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:04:20.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:04:20.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:04:20.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:04:20.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:04:20.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:04:20.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:04:20.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:04:20.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:04:20.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:04:20.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:04:20.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:20.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:04:20.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:04:20.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:20.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:04:20.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:04:20.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:04:20.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:04:20.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:04:20.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:04:20.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:04:20.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:04:20.788 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.788 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.788 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.789 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:20.789 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:20.789 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 101.346726554, 2023-07-02T14:04:20.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:04:20.789 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.789 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.789 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.790 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.790 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.790 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:04:20.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.790 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:20.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:20.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:20.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:04:20.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:04:20.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:04:20.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:04:20.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:04:20.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:04:20.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:04:20.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:04:20.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:04:20.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:04:20.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:04:20.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:04:20.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:20.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:04:20.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:04:20.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:20.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:04:20.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:04:20.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:04:20.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:04:20.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:04:20.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:04:20.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:04:20.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:04:20.799 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.799 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.799 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.800 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:20.800 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:20.800 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 101.346653851, 2023-07-02T14:04:20.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:04:20.800 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.800 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.801 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.801 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.801 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.801 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:04:20.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.801 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:20.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:20.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:20.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:20.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:04:20.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:04:20.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:04:20.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:04:20.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:04:20.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:04:20.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:04:20.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:04:20.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:04:20.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:04:20.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:04:20.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:20.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:04:20.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:04:20.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:20.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:04:20.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:04:20.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:04:20.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:04:20.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:04:20.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:04:20.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:04:20.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:04:20.810 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.810 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.810 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.811 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:20.811 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:20.811 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 101.346536861, 2023-07-02T14:04:20.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:04:20.811 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.812 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.812 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.812 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.812 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.812 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:04:20.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.812 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:20.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:20.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:20.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:20.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:04:20.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:04:20.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:04:20.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:04:20.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:04:20.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:04:20.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:04:20.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:04:20.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:04:20.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:04:20.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:04:20.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:20.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:04:20.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:04:20.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:20.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:04:20.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:04:20.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:04:20.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:04:20.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:04:20.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:04:20.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:04:20.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:04:20.821 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.821 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.822 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.822 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:20.822 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:20.822 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 101.34638741000001, 2023-07-02T14:04:20.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:04:20.823 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.823 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.823 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.823 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.823 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.823 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:04:20.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.824 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:20.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:20.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:20.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:04:20.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:04:20.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:04:20.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:04:20.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:04:20.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:04:20.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:04:20.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:04:20.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:04:20.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:04:20.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:04:20.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:04:20.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:20.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:04:20.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:04:20.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:20.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:04:20.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:04:20.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:04:20.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:04:20.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:04:20.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:04:20.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:04:20.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:04:20.833 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.833 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.833 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.833 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:20.833 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:20.834 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 101.324911437, 2023-07-02T14:04:20.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:04:20.834 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.834 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.834 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.834 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.834 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.834 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:04:20.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.835 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:20.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:20.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:04:20.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:04:20.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:04:20.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:04:20.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:04:20.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:04:20.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:04:20.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:04:20.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:04:20.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:04:20.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:04:20.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:04:20.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:20.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:04:20.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:04:20.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:04:20.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:04:20.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:20.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:04:20.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:04:20.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:04:20.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:04:20.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:04:20.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:04:20.844 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.844 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.844 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.844 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:20.845 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:20.845 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 101.324778901, 2023-07-02T14:04:20.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:04:20.845 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.845 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.845 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.845 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.846 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.846 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:04:20.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.846 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:20.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:20.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:04:20.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:04:20.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:04:20.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:04:20.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:04:20.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:04:20.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:04:20.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:04:20.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:04:20.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:04:20.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:04:20.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:04:20.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:20.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:04:20.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:04:20.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:04:20.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:04:20.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:20.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:04:20.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:04:20.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:04:20.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:04:20.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:04:20.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:04:20.855 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.855 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.855 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.856 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:20.856 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:20.856 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 101.26344224, 2023-07-02T14:04:20.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:04:20.856 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:20.856 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:20.857 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:20.857 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:20.857 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:20.857 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:04:20.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.857 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:20.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:20.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:20.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:20.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:20.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:20.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:20.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:20.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:04:20.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:04:20.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:20.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:04:20.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:04:20.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:20.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:04:20.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:04:20.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:20.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:04:20.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:04:20.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:20.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:04:20.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:04:20.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:20.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:04:20.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:04:20.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:20.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:04:20.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:04:20.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:20.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:04:20.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:04:20.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:04:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:04:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:04:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:04:20.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:20.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:20.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:20.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:04:20.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:04:20.866 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.866 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.867 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.867 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:20.867 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:20.867 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:20.868 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:20.869 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:20.868+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:20.969 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:20.968+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:21.006 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:21.007 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:21.007 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:21.007 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:21.017 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:04:21.070 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:21.070+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:21.170 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:21.169+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:21.194 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:21.194 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:21.194 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:21.195 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:21.195 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:21.207 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:04:21.220 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:04:21.272 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:21.271+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:21.372 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:21.372+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:21.395 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:21.395 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:21.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:21.395 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:21.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.396 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2620 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 196608~786432] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:21.396 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:21.396 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.068209276000005, 2023-07-02T14:04:21.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966, 2023-07-02T14:04:21.396 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:21.397 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:21.397 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:21.397 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:21.397 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:21.397 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2620 2023-07-02T14:04:21.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.397 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:21.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:21.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:21.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:21.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:21.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:21.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:21.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:21.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324571+0000", 2023-07-02T14:04:21.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016686649999999999 2023-07-02T14:04:21.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:21.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326240+0000", 2023-07-02T14:04:21.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1844e-05 2023-07-02T14:04:21.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:21.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326252+0000", 2023-07-02T14:04:21.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7760000000000003e-06 2023-07-02T14:04:21.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:21.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326258+0000", 2023-07-02T14:04:21.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0634000000000003e-05 2023-07-02T14:04:21.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326318+0000", 2023-07-02T14:04:21.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9629999999999993e-06 2023-07-02T14:04:21.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:21.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326328+0000", 2023-07-02T14:04:21.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.010877045 2023-07-02T14:04:21.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337205+0000", 2023-07-02T14:04:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022401300000000001 2023-07-02T14:04:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337429+0000", 2023-07-02T14:04:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000139822 2023-07-02T14:04:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:21.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337569+0000", 2023-07-02T14:04:21.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032942119999999999 2023-07-02T14:04:21.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:21.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340863+0000", 2023-07-02T14:04:21.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021302560000000001 2023-07-02T14:04:21.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:21.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342994+0000", 2023-07-02T14:04:21.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5634000000000002e-05 2023-07-02T14:04:21.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:21.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343059+0000", 2023-07-02T14:04:21.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143721 2023-07-02T14:04:21.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:21.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343203+0000", 2023-07-02T14:04:21.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966 2023-07-02T14:04:21.408 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.408 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.408 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.409 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2622 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 983040~36864] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:21.409 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:21.409 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.066516765000003, 2023-07-02T14:04:21.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664, 2023-07-02T14:04:21.409 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:21.409 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:21.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:21.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:21.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:21.410 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2622 2023-07-02T14:04:21.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.410 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:21.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:21.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:21.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:21.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:21.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:21.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:21.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:21.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326265+0000", 2023-07-02T14:04:21.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1282000000000001e-05 2023-07-02T14:04:21.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326286+0000", 2023-07-02T14:04:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9480000000000002e-06 2023-07-02T14:04:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326293+0000", 2023-07-02T14:04:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3610000000000003e-06 2023-07-02T14:04:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326297+0000", 2023-07-02T14:04:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7707999999999994e-05 2023-07-02T14:04:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326385+0000", 2023-07-02T14:04:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8070000000000006e-06 2023-07-02T14:04:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326394+0000", 2023-07-02T14:04:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.014535878 2023-07-02T14:04:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340930+0000", 2023-07-02T14:04:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000154551 2023-07-02T14:04:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341084+0000", 2023-07-02T14:04:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00032375900000000002 2023-07-02T14:04:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341408+0000", 2023-07-02T14:04:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012631066 2023-07-02T14:04:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354039+0000", 2023-07-02T14:04:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5741000000000001e-05 2023-07-02T14:04:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354055+0000", 2023-07-02T14:04:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8794000000000006e-05 2023-07-02T14:04:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354124+0000", 2023-07-02T14:04:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014667700000000001 2023-07-02T14:04:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354270+0000", 2023-07-02T14:04:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664 2023-07-02T14:04:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2727 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~12288] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.066480990000002, 2023-07-02T14:04:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999, 2023-07-02T14:04:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2727 2023-07-02T14:04:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3834e-05 2023-07-02T14:04:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326315+0000", 2023-07-02T14:04:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3029999999999996e-06 2023-07-02T14:04:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326321+0000", 2023-07-02T14:04:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7590000000000002e-06 2023-07-02T14:04:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326324+0000", 2023-07-02T14:04:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112872 2023-07-02T14:04:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326437+0000", 2023-07-02T14:04:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2290000000000002e-06 2023-07-02T14:04:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326442+0000", 2023-07-02T14:04:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0155896879999999 2023-07-02T14:04:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342031+0000", 2023-07-02T14:04:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016496100000000001 2023-07-02T14:04:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342196+0000", 2023-07-02T14:04:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021998699999999999 2023-07-02T14:04:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342416+0000", 2023-07-02T14:04:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011869744999999999 2023-07-02T14:04:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354286+0000", 2023-07-02T14:04:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0361e-05 2023-07-02T14:04:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354347+0000", 2023-07-02T14:04:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3312000000000005e-05 2023-07-02T14:04:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354430+0000", 2023-07-02T14:04:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126444 2023-07-02T14:04:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354556+0000", 2023-07-02T14:04:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999 2023-07-02T14:04:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2730 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [set-alloc-hint object_size 4194304 write_size 4194304,write 454656~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.066455981000004, 2023-07-02T14:04:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999, 2023-07-02T14:04:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2730 2023-07-02T14:04:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6049999999999997e-06 2023-07-02T14:04:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326332+0000", 2023-07-02T14:04:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3000000000000002e-06 2023-07-02T14:04:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326335+0000", 2023-07-02T14:04:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.719e-05 2023-07-02T14:04:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326352+0000", 2023-07-02T14:04:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011583599999999999 2023-07-02T14:04:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326468+0000", 2023-07-02T14:04:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.831e-06 2023-07-02T14:04:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326473+0000", 2023-07-02T14:04:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.016769915 2023-07-02T14:04:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343243+0000", 2023-07-02T14:04:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197352 2023-07-02T14:04:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343440+0000", 2023-07-02T14:04:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197581 2023-07-02T14:04:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343638+0000", 2023-07-02T14:04:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010980139 2023-07-02T14:04:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354618+0000", 2023-07-02T14:04:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056779530000000003 2023-07-02T14:04:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360296+0000", 2023-07-02T14:04:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101109 2023-07-02T14:04:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360397+0000", 2023-07-02T14:04:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000160273 2023-07-02T14:04:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360557+0000", 2023-07-02T14:04:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999 2023-07-02T14:04:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2752 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 1019904~3174400] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.066409757000002, 2023-07-02T14:04:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001, 2023-07-02T14:04:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2752 2023-07-02T14:04:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326372+0000", 2023-07-02T14:04:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052050380000000004 2023-07-02T14:04:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331577+0000", 2023-07-02T14:04:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6832000000000001e-05 2023-07-02T14:04:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331594+0000", 2023-07-02T14:04:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8220000000000007e-06 2023-07-02T14:04:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331602+0000", 2023-07-02T14:04:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9895999999999999e-05 2023-07-02T14:04:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331642+0000", 2023-07-02T14:04:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:04:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331649+0000", 2023-07-02T14:04:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0139141709999999 2023-07-02T14:04:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345563+0000", 2023-07-02T14:04:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113817 2023-07-02T14:04:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345677+0000", 2023-07-02T14:04:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018803400000000001 2023-07-02T14:04:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345865+0000", 2023-07-02T14:04:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087950359999999991 2023-07-02T14:04:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354660+0000", 2023-07-02T14:04:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059116669999999998 2023-07-02T14:04:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360572+0000", 2023-07-02T14:04:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0992000000000002e-05 2023-07-02T14:04:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360633+0000", 2023-07-02T14:04:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013601600000000001 2023-07-02T14:04:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360769+0000", 2023-07-02T14:04:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001 2023-07-02T14:04:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2759 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 0~454656] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.061169960000001, 2023-07-02T14:04:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999, 2023-07-02T14:04:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2759 2023-07-02T14:04:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331611+0000", 2023-07-02T14:04:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053795999999999998 2023-07-02T14:04:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332149+0000", 2023-07-02T14:04:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4970000000000007e-06 2023-07-02T14:04:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332158+0000", 2023-07-02T14:04:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:04:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332162+0000", 2023-07-02T14:04:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3526000000000001e-05 2023-07-02T14:04:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332185+0000", 2023-07-02T14:04:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:04:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332189+0000", 2023-07-02T14:04:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0186295910000001 2023-07-02T14:04:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350819+0000", 2023-07-02T14:04:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000111123 2023-07-02T14:04:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350930+0000", 2023-07-02T14:04:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019575300000000001 2023-07-02T14:04:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.351126+0000", 2023-07-02T14:04:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011503298 2023-07-02T14:04:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362629+0000", 2023-07-02T14:04:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085443989999999994 2023-07-02T14:04:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371173+0000", 2023-07-02T14:04:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3547999999999995e-05 2023-07-02T14:04:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371237+0000", 2023-07-02T14:04:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790300000000001 2023-07-02T14:04:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371375+0000", 2023-07-02T14:04:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999 2023-07-02T14:04:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2760 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 458752~131072] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.060613774999993, 2023-07-02T14:04:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001, 2023-07-02T14:04:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2760 2023-07-02T14:04:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332168+0000", 2023-07-02T14:04:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013281100000000001 2023-07-02T14:04:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332301+0000", 2023-07-02T14:04:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7389999999999999e-06 2023-07-02T14:04:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332308+0000", 2023-07-02T14:04:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0639999999999998e-06 2023-07-02T14:04:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332312+0000", 2023-07-02T14:04:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1735000000000001e-05 2023-07-02T14:04:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332333+0000", 2023-07-02T14:04:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8342000000000002e-05 2023-07-02T14:04:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332352+0000", 2023-07-02T14:04:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02234782 2023-07-02T14:04:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354699+0000", 2023-07-02T14:04:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110371 2023-07-02T14:04:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354810+0000", 2023-07-02T14:04:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022174900000000001 2023-07-02T14:04:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355032+0000", 2023-07-02T14:04:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076053329999999997 2023-07-02T14:04:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362637+0000", 2023-07-02T14:04:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087913609999999993 2023-07-02T14:04:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371428+0000", 2023-07-02T14:04:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8532999999999998e-05 2023-07-02T14:04:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371477+0000", 2023-07-02T14:04:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010990800000000001 2023-07-02T14:04:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371587+0000", 2023-07-02T14:04:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001 2023-07-02T14:04:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2766 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 589824~45056] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.060465149999999, 2023-07-02T14:04:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188, 2023-07-02T14:04:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2766 2023-07-02T14:04:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2247999999999998e-05 2023-07-02T14:04:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332369+0000", 2023-07-02T14:04:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4189999999999996e-06 2023-07-02T14:04:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332376+0000", 2023-07-02T14:04:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1049999999999999e-06 2023-07-02T14:04:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332379+0000", 2023-07-02T14:04:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6894000000000003e-05 2023-07-02T14:04:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332416+0000", 2023-07-02T14:04:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3970000000000002e-06 2023-07-02T14:04:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332422+0000", 2023-07-02T14:04:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0231382280000001 2023-07-02T14:04:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355560+0000", 2023-07-02T14:04:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9079000000000005e-05 2023-07-02T14:04:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355659+0000", 2023-07-02T14:04:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018005000000000001 2023-07-02T14:04:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355839+0000", 2023-07-02T14:04:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068044439999999998 2023-07-02T14:04:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362644+0000", 2023-07-02T14:04:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090155589999999994 2023-07-02T14:04:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371659+0000", 2023-07-02T14:04:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1297999999999994e-05 2023-07-02T14:04:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371730+0000", 2023-07-02T14:04:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013100699999999999 2023-07-02T14:04:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371861+0000", 2023-07-02T14:04:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188 2023-07-02T14:04:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2784 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 0~393216] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.060396698000005, 2023-07-02T14:04:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999, 2023-07-02T14:04:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2784 2023-07-02T14:04:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332385+0000", 2023-07-02T14:04:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040252400000000002 2023-07-02T14:04:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332787+0000", 2023-07-02T14:04:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0208e-05 2023-07-02T14:04:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332798+0000", 2023-07-02T14:04:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2023-07-02T14:04:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332801+0000", 2023-07-02T14:04:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4838000000000001e-05 2023-07-02T14:04:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332836+0000", 2023-07-02T14:04:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5319999999999998e-06 2023-07-02T14:04:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332839+0000", 2023-07-02T14:04:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0234290109999999 2023-07-02T14:04:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356268+0000", 2023-07-02T14:04:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011165700000000001 2023-07-02T14:04:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356380+0000", 2023-07-02T14:04:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022504799999999999 2023-07-02T14:04:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356605+0000", 2023-07-02T14:04:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015276201 2023-07-02T14:04:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371881+0000", 2023-07-02T14:04:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5496999999999999e-05 2023-07-02T14:04:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371957+0000", 2023-07-02T14:04:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4570000000000002e-05 2023-07-02T14:04:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372001+0000", 2023-07-02T14:04:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3880000000000003e-05 2023-07-02T14:04:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372085+0000", 2023-07-02T14:04:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999 2023-07-02T14:04:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2785 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 405504~446464] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.059975246999997, 2023-07-02T14:04:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999, 2023-07-02T14:04:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2785 2023-07-02T14:04:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059527500000000004 2023-07-02T14:04:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333402+0000", 2023-07-02T14:04:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7059999999999993e-06 2023-07-02T14:04:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333411+0000", 2023-07-02T14:04:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0450000000000001e-06 2023-07-02T14:04:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333414+0000", 2023-07-02T14:04:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5790999999999997e-05 2023-07-02T14:04:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:04:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3320000000000002e-06 2023-07-02T14:04:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333454+0000", 2023-07-02T14:04:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.025310269 2023-07-02T14:04:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358764+0000", 2023-07-02T14:04:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012414899999999999 2023-07-02T14:04:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358888+0000", 2023-07-02T14:04:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000217028 2023-07-02T14:04:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.359105+0000", 2023-07-02T14:04:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012788212 2023-07-02T14:04:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371893+0000", 2023-07-02T14:04:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012201569999999999 2023-07-02T14:04:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373114+0000", 2023-07-02T14:04:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4512999999999995e-05 2023-07-02T14:04:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373198+0000", 2023-07-02T14:04:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132373 2023-07-02T14:04:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373330+0000", 2023-07-02T14:04:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999 2023-07-02T14:04:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2786 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1310720~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.059362554000003, 2023-07-02T14:04:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001, 2023-07-02T14:04:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2786 2023-07-02T14:04:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333419+0000", 2023-07-02T14:04:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2009999999999999e-06 2023-07-02T14:04:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333427+0000", 2023-07-02T14:04:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0210000000000001e-06 2023-07-02T14:04:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333431+0000", 2023-07-02T14:04:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3769999999999999e-06 2023-07-02T14:04:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333434+0000", 2023-07-02T14:04:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1161e-05 2023-07-02T14:04:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333485+0000", 2023-07-02T14:04:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1549000000000001e-05 2023-07-02T14:04:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333496+0000", 2023-07-02T14:04:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0273116339999999 2023-07-02T14:04:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360808+0000", 2023-07-02T14:04:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000115128 2023-07-02T14:04:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360923+0000", 2023-07-02T14:04:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018367700000000001 2023-07-02T14:04:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361107+0000", 2023-07-02T14:04:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010796923999999999 2023-07-02T14:04:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371904+0000", 2023-07-02T14:04:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014983520000000001 2023-07-02T14:04:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373402+0000", 2023-07-02T14:04:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0520999999999999e-05 2023-07-02T14:04:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:21.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373463+0000", 2023-07-02T14:04:21.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000129095 2023-07-02T14:04:21.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:21.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373592+0000", 2023-07-02T14:04:21.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001 2023-07-02T14:04:21.535 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.535 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.535 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.535 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2787 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1433600~8192] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:21.536 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:21.536 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.059345754000006, 2023-07-02T14:04:21.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999, 2023-07-02T14:04:21.536 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:21.536 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:21.536 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:21.536 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:21.537 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:21.537 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2787 2023-07-02T14:04:21.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.537 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:21.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:21.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:21.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:21.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:21.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:21.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:21.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:21.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333436+0000", 2023-07-02T14:04:21.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2731999999999999e-05 2023-07-02T14:04:21.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:21.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:04:21.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7440000000000001e-06 2023-07-02T14:04:21.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:21.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333456+0000", 2023-07-02T14:04:21.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8650000000000003e-06 2023-07-02T14:04:21.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:21.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333460+0000", 2023-07-02T14:04:21.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0077e-05 2023-07-02T14:04:21.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333530+0000", 2023-07-02T14:04:21.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4180000000000001e-06 2023-07-02T14:04:21.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:21.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333533+0000", 2023-07-02T14:04:21.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0284002720000001 2023-07-02T14:04:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361933+0000", 2023-07-02T14:04:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6781999999999996e-05 2023-07-02T14:04:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:21.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362030+0000", 2023-07-02T14:04:21.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018305400000000001 2023-07-02T14:04:21.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:21.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362213+0000", 2023-07-02T14:04:21.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097015109999999995 2023-07-02T14:04:21.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:21.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371915+0000", 2023-07-02T14:04:21.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001742295 2023-07-02T14:04:21.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:21.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373657+0000", 2023-07-02T14:04:21.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5980000000000003e-05 2023-07-02T14:04:21.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:21.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373713+0000", 2023-07-02T14:04:21.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011783800000000001 2023-07-02T14:04:21.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:21.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373831+0000", 2023-07-02T14:04:21.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999 2023-07-02T14:04:21.547 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.548 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.548 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.548 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2803 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 851968~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:21.548 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:21.548 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.059303901999996, 2023-07-02T14:04:21.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514, 2023-07-02T14:04:21.549 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:21.549 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:21.549 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:21.549 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:21.549 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:21.549 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2803 2023-07-02T14:04:21.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.550 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:21.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:21.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:21.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:21.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:21.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:21.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:21.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:21.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:21.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4943000000000001e-05 2023-07-02T14:04:21.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:21.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333493+0000", 2023-07-02T14:04:21.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2629999999999997e-06 2023-07-02T14:04:21.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:21.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333500+0000", 2023-07-02T14:04:21.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4159999999999997e-06 2023-07-02T14:04:21.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:21.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333505+0000", 2023-07-02T14:04:21.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4518999999999998e-05 2023-07-02T14:04:21.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333559+0000", 2023-07-02T14:04:21.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5900000000000008e-06 2023-07-02T14:04:21.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:21.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333568+0000", 2023-07-02T14:04:21.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0291061779999999 2023-07-02T14:04:21.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362674+0000", 2023-07-02T14:04:21.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110412 2023-07-02T14:04:21.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:21.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362784+0000", 2023-07-02T14:04:21.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015384599999999999 2023-07-02T14:04:21.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:21.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362938+0000", 2023-07-02T14:04:21.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091682310000000006 2023-07-02T14:04:21.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:21.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372106+0000", 2023-07-02T14:04:21.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017703230000000001 2023-07-02T14:04:21.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:21.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373877+0000", 2023-07-02T14:04:21.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6387000000000002e-05 2023-07-02T14:04:21.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:21.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373923+0000", 2023-07-02T14:04:21.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014099199999999999 2023-07-02T14:04:21.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:21.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374064+0000", 2023-07-02T14:04:21.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514 2023-07-02T14:04:21.560 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.560 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.560 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.561 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2805 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 634880~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:21.561 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:21.561 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.059270726999998, 2023-07-02T14:04:21.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814, 2023-07-02T14:04:21.561 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:21.561 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:21.562 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:21.562 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:21.562 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:21.562 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2805 2023-07-02T14:04:21.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.562 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:21.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:21.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:21.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:21.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:21.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:21.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:21.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:21.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333511+0000", 2023-07-02T14:04:21.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207092 2023-07-02T14:04:21.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:21.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333718+0000", 2023-07-02T14:04:21.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5110000000000003e-06 2023-07-02T14:04:21.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:21.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333727+0000", 2023-07-02T14:04:21.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3270000000000007e-06 2023-07-02T14:04:21.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:21.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333736+0000", 2023-07-02T14:04:21.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0319999999999999e-05 2023-07-02T14:04:21.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333756+0000", 2023-07-02T14:04:21.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1695000000000001e-05 2023-07-02T14:04:21.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:21.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:04:21.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0295750560000001 2023-07-02T14:04:21.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363343+0000", 2023-07-02T14:04:21.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3387999999999994e-05 2023-07-02T14:04:21.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:21.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363437+0000", 2023-07-02T14:04:21.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018373 2023-07-02T14:04:21.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:21.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363620+0000", 2023-07-02T14:04:21.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0084948769999999996 2023-07-02T14:04:21.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:21.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372115+0000", 2023-07-02T14:04:21.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020082170000000001 2023-07-02T14:04:21.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:21.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374123+0000", 2023-07-02T14:04:21.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8875999999999997e-05 2023-07-02T14:04:21.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:21.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374182+0000", 2023-07-02T14:04:21.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0001272 2023-07-02T14:04:21.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:21.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374309+0000", 2023-07-02T14:04:21.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814 2023-07-02T14:04:21.573 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.573 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.573 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.573 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2806 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 1359872~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:21.574 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:21.574 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.059040396, 2023-07-02T14:04:21.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234, 2023-07-02T14:04:21.574 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:21.574 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:21.574 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:21.574 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:21.574 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:21.575 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2806 2023-07-02T14:04:21.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.575 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:21.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:21.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:21.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:21.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:21.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:21.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:21.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:21.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333741+0000", 2023-07-02T14:04:21.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3978999999999999e-05 2023-07-02T14:04:21.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:21.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333755+0000", 2023-07-02T14:04:21.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:04:21.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:21.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333761+0000", 2023-07-02T14:04:21.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7010000000000001e-06 2023-07-02T14:04:21.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:21.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333764+0000", 2023-07-02T14:04:21.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5086000000000003e-05 2023-07-02T14:04:21.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333799+0000", 2023-07-02T14:04:21.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6389999999999992e-06 2023-07-02T14:04:21.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:21.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333806+0000", 2023-07-02T14:04:21.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.030496675 2023-07-02T14:04:21.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364303+0000", 2023-07-02T14:04:21.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011382700000000001 2023-07-02T14:04:21.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:21.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364417+0000", 2023-07-02T14:04:21.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192058 2023-07-02T14:04:21.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:21.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364609+0000", 2023-07-02T14:04:21.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075132130000000004 2023-07-02T14:04:21.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:21.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372122+0000", 2023-07-02T14:04:21.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022731750000000001 2023-07-02T14:04:21.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:21.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374395+0000", 2023-07-02T14:04:21.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5277000000000005e-05 2023-07-02T14:04:21.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:21.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374460+0000", 2023-07-02T14:04:21.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017074599999999999 2023-07-02T14:04:21.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:21.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374631+0000", 2023-07-02T14:04:21.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234 2023-07-02T14:04:21.585 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.586 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.586 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.586 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 868352~49152] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:21.586 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:21.586 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.059013682, 2023-07-02T14:04:21.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001, 2023-07-02T14:04:21.587 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:21.587 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:21.587 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:21.587 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:21.587 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:21.587 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:04:21.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.588 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:21.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:21.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:21.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:21.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:21.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:21.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:21.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:21.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:04:21.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6355000000000001e-05 2023-07-02T14:04:21.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:21.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333794+0000", 2023-07-02T14:04:21.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8329999999999996e-06 2023-07-02T14:04:21.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:21.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333801+0000", 2023-07-02T14:04:21.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5419999999999996e-06 2023-07-02T14:04:21.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:21.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333807+0000", 2023-07-02T14:04:21.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2023-07-02T14:04:21.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333826+0000", 2023-07-02T14:04:21.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.303e-06 2023-07-02T14:04:21.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:21.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333830+0000", 2023-07-02T14:04:21.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.031137559 2023-07-02T14:04:21.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364967+0000", 2023-07-02T14:04:21.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1390000000000004e-05 2023-07-02T14:04:21.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:21.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365059+0000", 2023-07-02T14:04:21.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017597399999999999 2023-07-02T14:04:21.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:21.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365234+0000", 2023-07-02T14:04:21.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068943950000000002 2023-07-02T14:04:21.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:21.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372129+0000", 2023-07-02T14:04:21.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026189910000000002 2023-07-02T14:04:21.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:21.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374748+0000", 2023-07-02T14:04:21.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6613999999999996e-05 2023-07-02T14:04:21.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:21.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374814+0000", 2023-07-02T14:04:21.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502300000000001 2023-07-02T14:04:21.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:21.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374950+0000", 2023-07-02T14:04:21.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001 2023-07-02T14:04:21.598 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.598 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.598 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.599 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2815 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 917504~49152] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:21.599 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:21.599 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.054799449000001, 2023-07-02T14:04:21.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998, 2023-07-02T14:04:21.599 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:21.599 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:21.600 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:21.600 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:21.600 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:21.600 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2815 2023-07-02T14:04:21.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.600 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:21.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:21.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:21.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:21.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:21.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:21.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:04:21.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:21.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337980+0000", 2023-07-02T14:04:21.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017126700000000001 2023-07-02T14:04:21.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:21.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338152+0000", 2023-07-02T14:04:21.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2938999999999999e-05 2023-07-02T14:04:21.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:21.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338164+0000", 2023-07-02T14:04:21.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4448999999999999e-05 2023-07-02T14:04:21.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:21.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:04:21.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0281812 2023-07-02T14:04:21.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366370+0000", 2023-07-02T14:04:21.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1840000000000002e-05 2023-07-02T14:04:21.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:21.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366462+0000", 2023-07-02T14:04:21.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000186112 2023-07-02T14:04:21.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:21.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366648+0000", 2023-07-02T14:04:21.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054905709999999996 2023-07-02T14:04:21.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:21.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372139+0000", 2023-07-02T14:04:21.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044954790000000001 2023-07-02T14:04:21.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:21.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376634+0000", 2023-07-02T14:04:21.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4621000000000004e-05 2023-07-02T14:04:21.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:21.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376699+0000", 2023-07-02T14:04:21.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012862299999999999 2023-07-02T14:04:21.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:21.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376827+0000", 2023-07-02T14:04:21.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998 2023-07-02T14:04:21.609 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.610 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.610 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.610 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2816 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 917504~65536] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:21.610 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:21.610 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.054585070000002, 2023-07-02T14:04:21.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001, 2023-07-02T14:04:21.611 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:21.611 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:21.611 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:21.611 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:21.611 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:21.611 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2816 2023-07-02T14:04:21.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.612 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:21.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:21.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:21.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:21.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:21.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:21.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:21.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:21.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338196+0000", 2023-07-02T14:04:21.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6286999999999999e-05 2023-07-02T14:04:21.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:21.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338243+0000", 2023-07-02T14:04:21.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7030000000000001e-06 2023-07-02T14:04:21.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:21.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338252+0000", 2023-07-02T14:04:21.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:04:21.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:21.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338258+0000", 2023-07-02T14:04:21.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032085734999999997 2023-07-02T14:04:21.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370344+0000", 2023-07-02T14:04:21.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108621 2023-07-02T14:04:21.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:21.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370452+0000", 2023-07-02T14:04:21.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014936499999999999 2023-07-02T14:04:21.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:21.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370602+0000", 2023-07-02T14:04:21.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001778012 2023-07-02T14:04:21.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:21.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372380+0000", 2023-07-02T14:04:21.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044876380000000004 2023-07-02T14:04:21.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:21.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376867+0000", 2023-07-02T14:04:21.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9870999999999999e-05 2023-07-02T14:04:21.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:21.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376897+0000", 2023-07-02T14:04:21.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011401 2023-07-02T14:04:21.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:21.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.377011+0000", 2023-07-02T14:04:21.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001 2023-07-02T14:04:21.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.621 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.621 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:3878 2.1 e80/79 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head v 80'436, mlcod=80'429)", 2023-07-02T14:04:21.621 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:21.622 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 43.285331573000001, 2023-07-02T14:04:21.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037799472000000001, 2023-07-02T14:04:21.622 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:21.622 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:21.622 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:21.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:21.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:21.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:21.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:21.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:21.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:21.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:21.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107450+0000", 2023-07-02T14:04:21.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023914520000000001 2023-07-02T14:04:21.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:21.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109841+0000", 2023-07-02T14:04:21.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0823e-05 2023-07-02T14:04:21.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:21.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109862+0000", 2023-07-02T14:04:21.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2023-07-02T14:04:21.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:21.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109874+0000", 2023-07-02T14:04:21.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075009600000000001 2023-07-02T14:04:21.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.110624+0000", 2023-07-02T14:04:21.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5176000000000001e-05 2023-07-02T14:04:21.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:21.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.110649+0000", 2023-07-02T14:04:21.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034557310000000001 2023-07-02T14:04:21.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:21.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145206+0000", 2023-07-02T14:04:21.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4110999999999998e-05 2023-07-02T14:04:21.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:21.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145250+0000", 2023-07-02T14:04:21.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037799472000000001 2023-07-02T14:04:21.629 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.629 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.629 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.630 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:7177 2.5 2:a6e4b3cf:::rbd_data.125de1449605.00000000000000c4:head [write 0~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e91)", 2023-07-02T14:04:21.630 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:01.433694+0000", 2023-07-02T14:04:21.630 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.959088454, 2023-07-02T14:04:21.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033301720999999999, 2023-07-02T14:04:21.630 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:21.631 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:21.631 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:21.631 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:21.631 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:21.631 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7177 2023-07-02T14:04:21.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.631 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:21.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:21.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433694+0000", 2023-07-02T14:04:21.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:21.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:21.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433694+0000", 2023-07-02T14:04:21.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:21.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:21.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433693+0000", 2023-07-02T14:04:21.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6749999999999997e-05 2023-07-02T14:04:21.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:21.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433750+0000", 2023-07-02T14:04:21.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0451e-05 2023-07-02T14:04:21.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:21.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433760+0000", 2023-07-02T14:04:21.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8200000000000002e-06 2023-07-02T14:04:21.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:21.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433766+0000", 2023-07-02T14:04:21.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012906031 2023-07-02T14:04:21.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:21.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.446672+0000", 2023-07-02T14:04:21.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3190999999999998e-05 2023-07-02T14:04:21.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:21.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.446745+0000", 2023-07-02T14:04:21.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020003231999999999 2023-07-02T14:04:21.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:21.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.466749+0000", 2023-07-02T14:04:21.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010595 2023-07-02T14:04:21.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:21.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.466855+0000", 2023-07-02T14:04:21.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014102799999999999 2023-07-02T14:04:21.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:21.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:21.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:21.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.466996+0000", 2023-07-02T14:04:21.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033301720999999999 2023-07-02T14:04:21.639 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.639 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.639 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.639 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:21.640 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:21.640 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:21.641 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:04:21.642 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:21.642 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:21.642 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:21.642 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:21.643 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:21.643 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:21.644 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:04:21.644 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:04:21.644 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:04:21.644 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:21.644 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:04:21.644 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:04:21.645 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:21.645 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:21.640+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:21.741 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:21.740+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:21.828 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:21.828 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:21.842 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:04:21.843 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:21.841+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:21.943 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:21.943+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:22.017 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:22.018 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:22.029 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:04:22.073 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:22.072+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:22.144 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:22.143+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:22.168 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:04:22.169 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:22.169 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:22.183 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:04:22.266 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:22.265+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:22.345 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:22.344+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:22.374 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:22.375 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:22.389 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:04:22.472 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:22.471+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:22.546 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:22.546+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:22.575 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:22.575 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:22.587 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:04:22.647 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:22.646+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:22.748 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:22.747+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:22.772 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:22.772 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:22.784 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:04:22.848 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:22.847+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:22.924 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:22.925 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:22.935 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:04:22.948 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:22.947+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:23.049 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:23.048+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:23.058 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:23.058 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:23.069 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:04:23.118 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:04:23.137 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap20 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap21 b3fbb028-e267-4d25-a750-93ba18a2316a@snap21.diff 2023-07-02T14:04:23.150 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:23.149+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:23.196 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:23.196 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:23.200 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete...2023-07-02T14:04:23.200+0000 7f3900511700 0 -- 172.21.15.163:0/811370542 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x55868656b110 msgr2=0x55868656d500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-07-02T14:04:23.205 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:23.204+0000 7f3900511700 0 -- 172.21.15.163:0/811370542 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f38e4036c00 msgr2=0x7f38e4039030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:23.206 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:04:23.217 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 51% complete... Exporting image: 100% complete...done. 2023-07-02T14:04:23.221 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:04:23.221 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=22 2023-07-02T14:04:23.221 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap22 2023-07-02T14:04:23.230 INFO:tasks.workunit.client.0.smithi163.stderr:+ tar -C b3fbb028-e267-4d25-a750-93ba18a2316a-mnt -xzf b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4.tar.gz 2023-07-02T14:04:23.250 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:23.249+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:23.351 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:23.350+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:23.373 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:23.373 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:23.383 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:04:23.466 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:23.465+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:23.546 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:23.546 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:23.552 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:23.551+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:23.556 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:23.646 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:04:23.689 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:23.688+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:23.720 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:23.720 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:23.730 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:23.774 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:23.773+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:23.825 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:23.849 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:23.854 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:23.853+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:23.871 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:23.892 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:23.892 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:23.892 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:23.893 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:23.895 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:23.906 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:04:23.918 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:23.918 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:04:23.918 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:04:23.919 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:04:23.919 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:23.919 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:04:23.919 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:04:23.921 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:23.922 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:23.963 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:23.962+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:24.048 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:24.048 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:24.048 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:24.048 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:24.049 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:24.056 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:24.056+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:24.058 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:04:24.123 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 2] out_osds: [5] dead_osds: [3] live_osds: [0, 1, 2, 4, 5] 2023-07-02T14:04:24.123 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:04:24.124 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2023-07-02T14:04:24.124 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-07-02T14:04:24.124 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-07-02T14:04:24.124 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2023-07-02T14:04:24.134 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:04:24.140 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 1 2023-07-02T14:04:24.167 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:24.167+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:24.228 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:24.228 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:24.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:24.228 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:24.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2620 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 196608~786432] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.901168869000003, 2023-07-02T14:04:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966, 2023-07-02T14:04:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.229 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.230 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.230 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.230 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2620 2023-07-02T14:04:24.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.230 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:24.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:24.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:24.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324571+0000", 2023-07-02T14:04:24.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016686649999999999 2023-07-02T14:04:24.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326240+0000", 2023-07-02T14:04:24.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1844e-05 2023-07-02T14:04:24.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326252+0000", 2023-07-02T14:04:24.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7760000000000003e-06 2023-07-02T14:04:24.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326258+0000", 2023-07-02T14:04:24.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0634000000000003e-05 2023-07-02T14:04:24.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326318+0000", 2023-07-02T14:04:24.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9629999999999993e-06 2023-07-02T14:04:24.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326328+0000", 2023-07-02T14:04:24.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.010877045 2023-07-02T14:04:24.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337205+0000", 2023-07-02T14:04:24.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022401300000000001 2023-07-02T14:04:24.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337429+0000", 2023-07-02T14:04:24.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000139822 2023-07-02T14:04:24.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:24.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337569+0000", 2023-07-02T14:04:24.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032942119999999999 2023-07-02T14:04:24.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:24.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340863+0000", 2023-07-02T14:04:24.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021302560000000001 2023-07-02T14:04:24.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:24.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342994+0000", 2023-07-02T14:04:24.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5634000000000002e-05 2023-07-02T14:04:24.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:24.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343059+0000", 2023-07-02T14:04:24.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143721 2023-07-02T14:04:24.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343203+0000", 2023-07-02T14:04:24.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966 2023-07-02T14:04:24.240 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.241 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.241 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.241 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2622 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 983040~36864] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:24.241 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:24.241 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.899476358000001, 2023-07-02T14:04:24.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664, 2023-07-02T14:04:24.242 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.242 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.242 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.242 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.242 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.242 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2622 2023-07-02T14:04:24.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.242 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:24.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:24.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:24.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326265+0000", 2023-07-02T14:04:24.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1282000000000001e-05 2023-07-02T14:04:24.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326286+0000", 2023-07-02T14:04:24.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9480000000000002e-06 2023-07-02T14:04:24.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326293+0000", 2023-07-02T14:04:24.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3610000000000003e-06 2023-07-02T14:04:24.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326297+0000", 2023-07-02T14:04:24.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7707999999999994e-05 2023-07-02T14:04:24.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326385+0000", 2023-07-02T14:04:24.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8070000000000006e-06 2023-07-02T14:04:24.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326394+0000", 2023-07-02T14:04:24.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.014535878 2023-07-02T14:04:24.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340930+0000", 2023-07-02T14:04:24.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000154551 2023-07-02T14:04:24.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341084+0000", 2023-07-02T14:04:24.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00032375900000000002 2023-07-02T14:04:24.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:24.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341408+0000", 2023-07-02T14:04:24.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012631066 2023-07-02T14:04:24.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:24.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354039+0000", 2023-07-02T14:04:24.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5741000000000001e-05 2023-07-02T14:04:24.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:24.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354055+0000", 2023-07-02T14:04:24.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8794000000000006e-05 2023-07-02T14:04:24.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:24.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354124+0000", 2023-07-02T14:04:24.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014667700000000001 2023-07-02T14:04:24.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354270+0000", 2023-07-02T14:04:24.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664 2023-07-02T14:04:24.252 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.252 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.252 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.253 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2727 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~12288] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:24.253 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:24.253 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.899440583000001, 2023-07-02T14:04:24.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999, 2023-07-02T14:04:24.253 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.254 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.254 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.254 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.254 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.254 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2727 2023-07-02T14:04:24.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.254 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:24.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:24.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:24.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:24.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3834e-05 2023-07-02T14:04:24.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326315+0000", 2023-07-02T14:04:24.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3029999999999996e-06 2023-07-02T14:04:24.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326321+0000", 2023-07-02T14:04:24.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7590000000000002e-06 2023-07-02T14:04:24.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326324+0000", 2023-07-02T14:04:24.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112872 2023-07-02T14:04:24.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326437+0000", 2023-07-02T14:04:24.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2290000000000002e-06 2023-07-02T14:04:24.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326442+0000", 2023-07-02T14:04:24.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0155896879999999 2023-07-02T14:04:24.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342031+0000", 2023-07-02T14:04:24.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016496100000000001 2023-07-02T14:04:24.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342196+0000", 2023-07-02T14:04:24.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021998699999999999 2023-07-02T14:04:24.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:24.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342416+0000", 2023-07-02T14:04:24.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011869744999999999 2023-07-02T14:04:24.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:24.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354286+0000", 2023-07-02T14:04:24.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0361e-05 2023-07-02T14:04:24.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:24.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354347+0000", 2023-07-02T14:04:24.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3312000000000005e-05 2023-07-02T14:04:24.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:24.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354430+0000", 2023-07-02T14:04:24.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126444 2023-07-02T14:04:24.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354556+0000", 2023-07-02T14:04:24.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999 2023-07-02T14:04:24.265 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.265 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.265 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.265 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2730 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [set-alloc-hint object_size 4194304 write_size 4194304,write 454656~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:24.265 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:24.266 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.899415574000002, 2023-07-02T14:04:24.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999, 2023-07-02T14:04:24.266 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.266 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.266 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.266 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.266 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.266 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2730 2023-07-02T14:04:24.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.267 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:24.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:24.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:24.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:24.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6049999999999997e-06 2023-07-02T14:04:24.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326332+0000", 2023-07-02T14:04:24.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3000000000000002e-06 2023-07-02T14:04:24.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326335+0000", 2023-07-02T14:04:24.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.719e-05 2023-07-02T14:04:24.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326352+0000", 2023-07-02T14:04:24.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011583599999999999 2023-07-02T14:04:24.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326468+0000", 2023-07-02T14:04:24.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.831e-06 2023-07-02T14:04:24.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326473+0000", 2023-07-02T14:04:24.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.016769915 2023-07-02T14:04:24.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343243+0000", 2023-07-02T14:04:24.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197352 2023-07-02T14:04:24.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343440+0000", 2023-07-02T14:04:24.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197581 2023-07-02T14:04:24.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:24.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343638+0000", 2023-07-02T14:04:24.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010980139 2023-07-02T14:04:24.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:24.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354618+0000", 2023-07-02T14:04:24.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056779530000000003 2023-07-02T14:04:24.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:24.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360296+0000", 2023-07-02T14:04:24.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101109 2023-07-02T14:04:24.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:24.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360397+0000", 2023-07-02T14:04:24.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000160273 2023-07-02T14:04:24.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360557+0000", 2023-07-02T14:04:24.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999 2023-07-02T14:04:24.277 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.277 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.277 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.277 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2752 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 1019904~3174400] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:24.278 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:24.278 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.899369350000001, 2023-07-02T14:04:24.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001, 2023-07-02T14:04:24.278 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.278 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.278 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.278 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.278 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.279 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2752 2023-07-02T14:04:24.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.279 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:24.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:24.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:24.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326372+0000", 2023-07-02T14:04:24.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052050380000000004 2023-07-02T14:04:24.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331577+0000", 2023-07-02T14:04:24.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6832000000000001e-05 2023-07-02T14:04:24.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331594+0000", 2023-07-02T14:04:24.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8220000000000007e-06 2023-07-02T14:04:24.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331602+0000", 2023-07-02T14:04:24.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9895999999999999e-05 2023-07-02T14:04:24.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331642+0000", 2023-07-02T14:04:24.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:04:24.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331649+0000", 2023-07-02T14:04:24.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0139141709999999 2023-07-02T14:04:24.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345563+0000", 2023-07-02T14:04:24.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113817 2023-07-02T14:04:24.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345677+0000", 2023-07-02T14:04:24.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018803400000000001 2023-07-02T14:04:24.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:24.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345865+0000", 2023-07-02T14:04:24.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087950359999999991 2023-07-02T14:04:24.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:24.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354660+0000", 2023-07-02T14:04:24.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059116669999999998 2023-07-02T14:04:24.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:24.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360572+0000", 2023-07-02T14:04:24.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0992000000000002e-05 2023-07-02T14:04:24.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:24.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360633+0000", 2023-07-02T14:04:24.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013601600000000001 2023-07-02T14:04:24.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360769+0000", 2023-07-02T14:04:24.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001 2023-07-02T14:04:24.289 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.290 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.290 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.290 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2759 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 0~454656] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:24.290 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:24.290 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.894129552999999, 2023-07-02T14:04:24.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999, 2023-07-02T14:04:24.291 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.291 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.291 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.291 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.291 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.291 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2759 2023-07-02T14:04:24.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.291 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:24.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:24.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:24.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331611+0000", 2023-07-02T14:04:24.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053795999999999998 2023-07-02T14:04:24.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332149+0000", 2023-07-02T14:04:24.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4970000000000007e-06 2023-07-02T14:04:24.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332158+0000", 2023-07-02T14:04:24.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:04:24.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332162+0000", 2023-07-02T14:04:24.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3526000000000001e-05 2023-07-02T14:04:24.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332185+0000", 2023-07-02T14:04:24.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:04:24.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332189+0000", 2023-07-02T14:04:24.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0186295910000001 2023-07-02T14:04:24.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350819+0000", 2023-07-02T14:04:24.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000111123 2023-07-02T14:04:24.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350930+0000", 2023-07-02T14:04:24.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019575300000000001 2023-07-02T14:04:24.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:24.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.351126+0000", 2023-07-02T14:04:24.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011503298 2023-07-02T14:04:24.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:24.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362629+0000", 2023-07-02T14:04:24.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085443989999999994 2023-07-02T14:04:24.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:24.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371173+0000", 2023-07-02T14:04:24.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3547999999999995e-05 2023-07-02T14:04:24.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:24.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371237+0000", 2023-07-02T14:04:24.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790300000000001 2023-07-02T14:04:24.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371375+0000", 2023-07-02T14:04:24.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999 2023-07-02T14:04:24.301 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.301 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.302 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.302 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2760 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 458752~131072] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:24.302 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:24.302 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.893573368000006, 2023-07-02T14:04:24.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001, 2023-07-02T14:04:24.302 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.303 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.303 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.303 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.303 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.303 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2760 2023-07-02T14:04:24.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.303 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:24.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:24.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:24.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332168+0000", 2023-07-02T14:04:24.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013281100000000001 2023-07-02T14:04:24.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332301+0000", 2023-07-02T14:04:24.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7389999999999999e-06 2023-07-02T14:04:24.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332308+0000", 2023-07-02T14:04:24.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0639999999999998e-06 2023-07-02T14:04:24.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332312+0000", 2023-07-02T14:04:24.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1735000000000001e-05 2023-07-02T14:04:24.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332333+0000", 2023-07-02T14:04:24.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8342000000000002e-05 2023-07-02T14:04:24.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332352+0000", 2023-07-02T14:04:24.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02234782 2023-07-02T14:04:24.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354699+0000", 2023-07-02T14:04:24.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110371 2023-07-02T14:04:24.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354810+0000", 2023-07-02T14:04:24.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022174900000000001 2023-07-02T14:04:24.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:24.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355032+0000", 2023-07-02T14:04:24.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076053329999999997 2023-07-02T14:04:24.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:24.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362637+0000", 2023-07-02T14:04:24.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087913609999999993 2023-07-02T14:04:24.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:24.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371428+0000", 2023-07-02T14:04:24.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8532999999999998e-05 2023-07-02T14:04:24.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:24.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371477+0000", 2023-07-02T14:04:24.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010990800000000001 2023-07-02T14:04:24.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371587+0000", 2023-07-02T14:04:24.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001 2023-07-02T14:04:24.315 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.315 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.315 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.316 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2766 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 589824~45056] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:24.316 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:24.316 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.893424742999997, 2023-07-02T14:04:24.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188, 2023-07-02T14:04:24.316 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.316 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.316 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.317 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.317 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.317 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2766 2023-07-02T14:04:24.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.317 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:24.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:24.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:24.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:24.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2247999999999998e-05 2023-07-02T14:04:24.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332369+0000", 2023-07-02T14:04:24.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4189999999999996e-06 2023-07-02T14:04:24.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332376+0000", 2023-07-02T14:04:24.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1049999999999999e-06 2023-07-02T14:04:24.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332379+0000", 2023-07-02T14:04:24.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6894000000000003e-05 2023-07-02T14:04:24.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332416+0000", 2023-07-02T14:04:24.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3970000000000002e-06 2023-07-02T14:04:24.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332422+0000", 2023-07-02T14:04:24.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0231382280000001 2023-07-02T14:04:24.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355560+0000", 2023-07-02T14:04:24.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9079000000000005e-05 2023-07-02T14:04:24.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355659+0000", 2023-07-02T14:04:24.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018005000000000001 2023-07-02T14:04:24.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:24.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355839+0000", 2023-07-02T14:04:24.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068044439999999998 2023-07-02T14:04:24.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:24.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362644+0000", 2023-07-02T14:04:24.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090155589999999994 2023-07-02T14:04:24.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:24.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371659+0000", 2023-07-02T14:04:24.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1297999999999994e-05 2023-07-02T14:04:24.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:24.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371730+0000", 2023-07-02T14:04:24.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013100699999999999 2023-07-02T14:04:24.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371861+0000", 2023-07-02T14:04:24.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188 2023-07-02T14:04:24.329 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.329 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.329 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.329 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2784 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 0~393216] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:24.329 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:24.330 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.893356291000003, 2023-07-02T14:04:24.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999, 2023-07-02T14:04:24.330 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.330 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.330 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.330 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.330 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.330 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2784 2023-07-02T14:04:24.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.331 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:24.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:24.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:24.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332385+0000", 2023-07-02T14:04:24.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040252400000000002 2023-07-02T14:04:24.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332787+0000", 2023-07-02T14:04:24.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0208e-05 2023-07-02T14:04:24.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332798+0000", 2023-07-02T14:04:24.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2023-07-02T14:04:24.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332801+0000", 2023-07-02T14:04:24.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4838000000000001e-05 2023-07-02T14:04:24.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332836+0000", 2023-07-02T14:04:24.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5319999999999998e-06 2023-07-02T14:04:24.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332839+0000", 2023-07-02T14:04:24.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0234290109999999 2023-07-02T14:04:24.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356268+0000", 2023-07-02T14:04:24.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011165700000000001 2023-07-02T14:04:24.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356380+0000", 2023-07-02T14:04:24.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022504799999999999 2023-07-02T14:04:24.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:24.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356605+0000", 2023-07-02T14:04:24.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015276201 2023-07-02T14:04:24.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:24.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371881+0000", 2023-07-02T14:04:24.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5496999999999999e-05 2023-07-02T14:04:24.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:24.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371957+0000", 2023-07-02T14:04:24.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4570000000000002e-05 2023-07-02T14:04:24.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:24.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372001+0000", 2023-07-02T14:04:24.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3880000000000003e-05 2023-07-02T14:04:24.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372085+0000", 2023-07-02T14:04:24.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999 2023-07-02T14:04:24.341 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.341 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.341 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.341 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2785 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 405504~446464] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:24.341 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:24.342 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.892934839999995, 2023-07-02T14:04:24.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999, 2023-07-02T14:04:24.342 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.342 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.342 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.342 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.342 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.342 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2785 2023-07-02T14:04:24.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.343 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:24.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:24.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:24.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:24.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059527500000000004 2023-07-02T14:04:24.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333402+0000", 2023-07-02T14:04:24.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7059999999999993e-06 2023-07-02T14:04:24.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333411+0000", 2023-07-02T14:04:24.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0450000000000001e-06 2023-07-02T14:04:24.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333414+0000", 2023-07-02T14:04:24.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5790999999999997e-05 2023-07-02T14:04:24.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:04:24.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3320000000000002e-06 2023-07-02T14:04:24.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333454+0000", 2023-07-02T14:04:24.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.025310269 2023-07-02T14:04:24.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358764+0000", 2023-07-02T14:04:24.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012414899999999999 2023-07-02T14:04:24.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358888+0000", 2023-07-02T14:04:24.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000217028 2023-07-02T14:04:24.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:24.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.359105+0000", 2023-07-02T14:04:24.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012788212 2023-07-02T14:04:24.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:24.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371893+0000", 2023-07-02T14:04:24.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012201569999999999 2023-07-02T14:04:24.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:24.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373114+0000", 2023-07-02T14:04:24.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4512999999999995e-05 2023-07-02T14:04:24.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:24.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373198+0000", 2023-07-02T14:04:24.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132373 2023-07-02T14:04:24.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373330+0000", 2023-07-02T14:04:24.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999 2023-07-02T14:04:24.353 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.353 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.353 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.353 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2786 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1310720~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:24.353 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:24.353 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.892322147000002, 2023-07-02T14:04:24.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001, 2023-07-02T14:04:24.354 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.354 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.354 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.354 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.354 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.354 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2786 2023-07-02T14:04:24.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.355 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:24.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:24.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:24.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333419+0000", 2023-07-02T14:04:24.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2009999999999999e-06 2023-07-02T14:04:24.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333427+0000", 2023-07-02T14:04:24.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0210000000000001e-06 2023-07-02T14:04:24.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333431+0000", 2023-07-02T14:04:24.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3769999999999999e-06 2023-07-02T14:04:24.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333434+0000", 2023-07-02T14:04:24.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1161e-05 2023-07-02T14:04:24.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333485+0000", 2023-07-02T14:04:24.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1549000000000001e-05 2023-07-02T14:04:24.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333496+0000", 2023-07-02T14:04:24.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0273116339999999 2023-07-02T14:04:24.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360808+0000", 2023-07-02T14:04:24.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000115128 2023-07-02T14:04:24.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360923+0000", 2023-07-02T14:04:24.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018367700000000001 2023-07-02T14:04:24.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:24.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361107+0000", 2023-07-02T14:04:24.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010796923999999999 2023-07-02T14:04:24.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:24.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371904+0000", 2023-07-02T14:04:24.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014983520000000001 2023-07-02T14:04:24.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:24.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373402+0000", 2023-07-02T14:04:24.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0520999999999999e-05 2023-07-02T14:04:24.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:24.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373463+0000", 2023-07-02T14:04:24.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000129095 2023-07-02T14:04:24.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373592+0000", 2023-07-02T14:04:24.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001 2023-07-02T14:04:24.364 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.365 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.365 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.365 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2787 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1433600~8192] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:24.365 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:24.365 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.892305347000004, 2023-07-02T14:04:24.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999, 2023-07-02T14:04:24.366 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.366 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.366 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.366 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.366 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.366 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2787 2023-07-02T14:04:24.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.366 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:24.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:24.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:24.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333436+0000", 2023-07-02T14:04:24.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2731999999999999e-05 2023-07-02T14:04:24.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:04:24.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7440000000000001e-06 2023-07-02T14:04:24.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333456+0000", 2023-07-02T14:04:24.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8650000000000003e-06 2023-07-02T14:04:24.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333460+0000", 2023-07-02T14:04:24.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0077e-05 2023-07-02T14:04:24.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333530+0000", 2023-07-02T14:04:24.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4180000000000001e-06 2023-07-02T14:04:24.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333533+0000", 2023-07-02T14:04:24.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0284002720000001 2023-07-02T14:04:24.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361933+0000", 2023-07-02T14:04:24.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6781999999999996e-05 2023-07-02T14:04:24.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362030+0000", 2023-07-02T14:04:24.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018305400000000001 2023-07-02T14:04:24.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:24.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362213+0000", 2023-07-02T14:04:24.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097015109999999995 2023-07-02T14:04:24.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:24.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371915+0000", 2023-07-02T14:04:24.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001742295 2023-07-02T14:04:24.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:24.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373657+0000", 2023-07-02T14:04:24.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5980000000000003e-05 2023-07-02T14:04:24.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:24.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373713+0000", 2023-07-02T14:04:24.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011783800000000001 2023-07-02T14:04:24.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373831+0000", 2023-07-02T14:04:24.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999 2023-07-02T14:04:24.376 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.376 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.377 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.377 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2803 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 851968~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:24.377 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:24.377 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.892263494999995, 2023-07-02T14:04:24.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514, 2023-07-02T14:04:24.377 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.378 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.378 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.378 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.378 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.378 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2803 2023-07-02T14:04:24.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.378 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:24.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:24.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:24.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:24.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4943000000000001e-05 2023-07-02T14:04:24.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333493+0000", 2023-07-02T14:04:24.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2629999999999997e-06 2023-07-02T14:04:24.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333500+0000", 2023-07-02T14:04:24.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4159999999999997e-06 2023-07-02T14:04:24.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333505+0000", 2023-07-02T14:04:24.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4518999999999998e-05 2023-07-02T14:04:24.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333559+0000", 2023-07-02T14:04:24.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5900000000000008e-06 2023-07-02T14:04:24.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333568+0000", 2023-07-02T14:04:24.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0291061779999999 2023-07-02T14:04:24.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362674+0000", 2023-07-02T14:04:24.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110412 2023-07-02T14:04:24.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362784+0000", 2023-07-02T14:04:24.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015384599999999999 2023-07-02T14:04:24.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:24.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362938+0000", 2023-07-02T14:04:24.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091682310000000006 2023-07-02T14:04:24.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:24.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372106+0000", 2023-07-02T14:04:24.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017703230000000001 2023-07-02T14:04:24.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:24.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373877+0000", 2023-07-02T14:04:24.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6387000000000002e-05 2023-07-02T14:04:24.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:24.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373923+0000", 2023-07-02T14:04:24.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014099199999999999 2023-07-02T14:04:24.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374064+0000", 2023-07-02T14:04:24.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514 2023-07-02T14:04:24.388 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.388 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.388 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.389 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2805 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 634880~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:24.389 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:24.389 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.892230319999996, 2023-07-02T14:04:24.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814, 2023-07-02T14:04:24.389 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.389 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.389 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.390 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.390 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.390 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2805 2023-07-02T14:04:24.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.390 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:24.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:24.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:24.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333511+0000", 2023-07-02T14:04:24.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207092 2023-07-02T14:04:24.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333718+0000", 2023-07-02T14:04:24.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5110000000000003e-06 2023-07-02T14:04:24.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333727+0000", 2023-07-02T14:04:24.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3270000000000007e-06 2023-07-02T14:04:24.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333736+0000", 2023-07-02T14:04:24.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0319999999999999e-05 2023-07-02T14:04:24.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333756+0000", 2023-07-02T14:04:24.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1695000000000001e-05 2023-07-02T14:04:24.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:04:24.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0295750560000001 2023-07-02T14:04:24.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363343+0000", 2023-07-02T14:04:24.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3387999999999994e-05 2023-07-02T14:04:24.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363437+0000", 2023-07-02T14:04:24.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018373 2023-07-02T14:04:24.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:24.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363620+0000", 2023-07-02T14:04:24.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0084948769999999996 2023-07-02T14:04:24.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:24.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372115+0000", 2023-07-02T14:04:24.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020082170000000001 2023-07-02T14:04:24.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:24.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374123+0000", 2023-07-02T14:04:24.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8875999999999997e-05 2023-07-02T14:04:24.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:24.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374182+0000", 2023-07-02T14:04:24.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0001272 2023-07-02T14:04:24.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374309+0000", 2023-07-02T14:04:24.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814 2023-07-02T14:04:24.400 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.400 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.400 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.400 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2806 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 1359872~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:24.400 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:24.401 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.891999988999999, 2023-07-02T14:04:24.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234, 2023-07-02T14:04:24.401 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.401 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.401 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.401 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.401 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.402 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2806 2023-07-02T14:04:24.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.402 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:24.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:24.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:24.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333741+0000", 2023-07-02T14:04:24.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3978999999999999e-05 2023-07-02T14:04:24.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333755+0000", 2023-07-02T14:04:24.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:04:24.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333761+0000", 2023-07-02T14:04:24.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7010000000000001e-06 2023-07-02T14:04:24.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333764+0000", 2023-07-02T14:04:24.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5086000000000003e-05 2023-07-02T14:04:24.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333799+0000", 2023-07-02T14:04:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6389999999999992e-06 2023-07-02T14:04:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333806+0000", 2023-07-02T14:04:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.030496675 2023-07-02T14:04:24.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364303+0000", 2023-07-02T14:04:24.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011382700000000001 2023-07-02T14:04:24.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364417+0000", 2023-07-02T14:04:24.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192058 2023-07-02T14:04:24.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:24.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364609+0000", 2023-07-02T14:04:24.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075132130000000004 2023-07-02T14:04:24.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:24.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372122+0000", 2023-07-02T14:04:24.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022731750000000001 2023-07-02T14:04:24.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:24.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374395+0000", 2023-07-02T14:04:24.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5277000000000005e-05 2023-07-02T14:04:24.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:24.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374460+0000", 2023-07-02T14:04:24.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017074599999999999 2023-07-02T14:04:24.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374631+0000", 2023-07-02T14:04:24.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234 2023-07-02T14:04:24.415 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.415 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.415 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.416 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 868352~49152] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:24.416 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:24.416 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.891973274999998, 2023-07-02T14:04:24.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001, 2023-07-02T14:04:24.416 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.416 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.416 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.417 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.417 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.417 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:04:24.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.417 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:24.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:24.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:24.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:04:24.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6355000000000001e-05 2023-07-02T14:04:24.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333794+0000", 2023-07-02T14:04:24.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8329999999999996e-06 2023-07-02T14:04:24.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333801+0000", 2023-07-02T14:04:24.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5419999999999996e-06 2023-07-02T14:04:24.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333807+0000", 2023-07-02T14:04:24.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2023-07-02T14:04:24.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333826+0000", 2023-07-02T14:04:24.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.303e-06 2023-07-02T14:04:24.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333830+0000", 2023-07-02T14:04:24.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.031137559 2023-07-02T14:04:24.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364967+0000", 2023-07-02T14:04:24.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1390000000000004e-05 2023-07-02T14:04:24.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365059+0000", 2023-07-02T14:04:24.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017597399999999999 2023-07-02T14:04:24.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:24.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365234+0000", 2023-07-02T14:04:24.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068943950000000002 2023-07-02T14:04:24.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:24.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372129+0000", 2023-07-02T14:04:24.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026189910000000002 2023-07-02T14:04:24.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:24.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374748+0000", 2023-07-02T14:04:24.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6613999999999996e-05 2023-07-02T14:04:24.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:24.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374814+0000", 2023-07-02T14:04:24.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502300000000001 2023-07-02T14:04:24.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374950+0000", 2023-07-02T14:04:24.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001 2023-07-02T14:04:24.427 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.427 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.427 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.427 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2815 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 917504~49152] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:24.427 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:24.428 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 72.887759041999999, 2023-07-02T14:04:24.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998, 2023-07-02T14:04:24.428 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.428 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.428 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.428 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.428 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.428 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2815 2023-07-02T14:04:24.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.429 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:24.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:24.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:04:24.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337980+0000", 2023-07-02T14:04:24.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017126700000000001 2023-07-02T14:04:24.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338152+0000", 2023-07-02T14:04:24.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2938999999999999e-05 2023-07-02T14:04:24.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338164+0000", 2023-07-02T14:04:24.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4448999999999999e-05 2023-07-02T14:04:24.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:04:24.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0281812 2023-07-02T14:04:24.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366370+0000", 2023-07-02T14:04:24.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1840000000000002e-05 2023-07-02T14:04:24.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366462+0000", 2023-07-02T14:04:24.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000186112 2023-07-02T14:04:24.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:24.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366648+0000", 2023-07-02T14:04:24.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054905709999999996 2023-07-02T14:04:24.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:24.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372139+0000", 2023-07-02T14:04:24.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044954790000000001 2023-07-02T14:04:24.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:24.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376634+0000", 2023-07-02T14:04:24.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4621000000000004e-05 2023-07-02T14:04:24.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:24.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376699+0000", 2023-07-02T14:04:24.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012862299999999999 2023-07-02T14:04:24.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376827+0000", 2023-07-02T14:04:24.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998 2023-07-02T14:04:24.437 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.437 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.437 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.438 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2816 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 917504~65536] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:24.438 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:24.438 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 72.887544663, 2023-07-02T14:04:24.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001, 2023-07-02T14:04:24.438 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.438 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.438 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.439 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.439 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.439 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2816 2023-07-02T14:04:24.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.439 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:24.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:24.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:24.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338196+0000", 2023-07-02T14:04:24.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6286999999999999e-05 2023-07-02T14:04:24.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338243+0000", 2023-07-02T14:04:24.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7030000000000001e-06 2023-07-02T14:04:24.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338252+0000", 2023-07-02T14:04:24.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:04:24.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338258+0000", 2023-07-02T14:04:24.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032085734999999997 2023-07-02T14:04:24.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370344+0000", 2023-07-02T14:04:24.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108621 2023-07-02T14:04:24.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370452+0000", 2023-07-02T14:04:24.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014936499999999999 2023-07-02T14:04:24.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:24.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370602+0000", 2023-07-02T14:04:24.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001778012 2023-07-02T14:04:24.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:24.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372380+0000", 2023-07-02T14:04:24.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044876380000000004 2023-07-02T14:04:24.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:24.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376867+0000", 2023-07-02T14:04:24.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9870999999999999e-05 2023-07-02T14:04:24.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:24.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376897+0000", 2023-07-02T14:04:24.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011401 2023-07-02T14:04:24.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.377011+0000", 2023-07-02T14:04:24.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001 2023-07-02T14:04:24.447 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.448 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.448 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.448 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:3878 2.1 e80/79 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head v 80'436, mlcod=80'429)", 2023-07-02T14:04:24.448 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:24.448 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 46.118291165999999, 2023-07-02T14:04:24.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037799472000000001, 2023-07-02T14:04:24.449 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.449 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.449 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:24.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:24.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:24.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107450+0000", 2023-07-02T14:04:24.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023914520000000001 2023-07-02T14:04:24.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109841+0000", 2023-07-02T14:04:24.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0823e-05 2023-07-02T14:04:24.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109862+0000", 2023-07-02T14:04:24.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2023-07-02T14:04:24.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109874+0000", 2023-07-02T14:04:24.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075009600000000001 2023-07-02T14:04:24.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.110624+0000", 2023-07-02T14:04:24.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5176000000000001e-05 2023-07-02T14:04:24.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.110649+0000", 2023-07-02T14:04:24.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034557310000000001 2023-07-02T14:04:24.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:24.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145206+0000", 2023-07-02T14:04:24.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4110999999999998e-05 2023-07-02T14:04:24.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145250+0000", 2023-07-02T14:04:24.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037799472000000001 2023-07-02T14:04:24.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.455 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.456 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.456 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:7177 2.5 2:a6e4b3cf:::rbd_data.125de1449605.00000000000000c4:head [write 0~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e91)", 2023-07-02T14:04:24.456 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:01.433694+0000", 2023-07-02T14:04:24.456 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.792048047000002, 2023-07-02T14:04:24.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033301720999999999, 2023-07-02T14:04:24.456 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.457 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.457 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.457 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.457 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.457 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 7177 2023-07-02T14:04:24.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.457 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433694+0000", 2023-07-02T14:04:24.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433694+0000", 2023-07-02T14:04:24.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:24.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433693+0000", 2023-07-02T14:04:24.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6749999999999997e-05 2023-07-02T14:04:24.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433750+0000", 2023-07-02T14:04:24.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0451e-05 2023-07-02T14:04:24.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433760+0000", 2023-07-02T14:04:24.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8200000000000002e-06 2023-07-02T14:04:24.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.433766+0000", 2023-07-02T14:04:24.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012906031 2023-07-02T14:04:24.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.446672+0000", 2023-07-02T14:04:24.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3190999999999998e-05 2023-07-02T14:04:24.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.446745+0000", 2023-07-02T14:04:24.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020003231999999999 2023-07-02T14:04:24.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:24.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.466749+0000", 2023-07-02T14:04:24.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010595 2023-07-02T14:04:24.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:24.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.466855+0000", 2023-07-02T14:04:24.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014102799999999999 2023-07-02T14:04:24.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:01.466996+0000", 2023-07-02T14:04:24.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033301720999999999 2023-07-02T14:04:24.464 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.465 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.465 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.465 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.465 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.465 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:24.467 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:24.467 INFO:teuthology.orchestra.run.smithi022.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-07-02T14:04:24.467 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:24.467 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:04:24.468 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:24.466+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:24.567 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:24.567+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:24.633 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:24.634 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:24.634 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:24.634 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:24.644 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:04:24.687 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:24.687+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:24.767 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:24.767 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:24.768 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:24.768 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:24.768 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:24.769 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:24.769+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:24.777 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:04:24.870 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:24.870+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:24.944 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:24.944 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:24.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:24.945 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:24.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.945 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:04:24.945 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:24.945 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 75.490198719999995, 2023-07-02T14:04:24.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:04:24.946 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.946 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:24.946 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.946 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.946 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.946 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:04:24.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:24.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:24.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:04:24.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:04:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:04:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:04:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:04:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:04:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:04:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:04:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:04:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:04:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:04:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:04:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:04:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:04:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:04:24.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:04:24.953 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.953 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.953 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:04:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.619505712000006, 2023-07-02T14:04:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:04:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:04:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:04:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:04:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:24.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:24.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:04:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:04:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:04:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:04:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:04:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:04:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:04:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:04:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:04:24.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:04:24.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:04:24.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:04:24.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:04:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:04:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:04:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:04:24.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:04:24.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:04:24.962 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.962 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.619380991, 2023-07-02T14:04:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:04:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.964 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:24.964 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.964 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.964 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.964 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:04:24.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.964 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:04:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:04:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:04:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:04:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:04:24.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:04:24.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:04:24.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:04:24.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:04:24.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:04:24.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:04:24.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:04:24.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:04:24.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:04:24.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:04:24.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:04:24.975 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.975 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.976 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.976 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:24.976 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:24.977 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.612531106999995, 2023-07-02T14:04:24.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:04:24.977 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.978 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:24.978 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.978 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.978 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.978 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:04:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:24.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:04:24.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:04:24.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:04:24.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:04:24.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:04:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:04:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:04:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:04:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:04:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:04:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:04:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:04:24.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:04:24.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:04:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:04:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:04:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:04:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.612468182000001, 2023-07-02T14:04:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:04:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:04:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:04:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:04:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:04:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:04:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:04:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:04:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:04:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:04:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:04:24.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:04:24.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:24.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:04:24.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:04:24.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:24.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:04:24.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:04:24.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:24.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:04:24.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:04:24.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.994 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:24.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:24.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.995 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:24.995 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:24.995 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.612167760999995, 2023-07-02T14:04:24.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:04:24.996 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:24.996 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:24.996 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:24.996 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:24.996 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:24.996 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:04:24.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:04:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:04:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:04:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:04:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:04:25.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:04:25.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:04:25.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:04:25.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:04:25.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:04:25.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:25.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:04:25.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:04:25.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:04:25.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:04:25.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:04:25.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:04:25.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:04:25.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:04:25.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:04:25.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:04:25.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:04:25.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:04:25.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:04:25.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:04:25.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:04:25.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:04:25.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:04:25.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:04:25.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:04:25.007 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.007 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.007 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.008 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:04:25.008 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:25.008 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.611643260999998, 2023-07-02T14:04:25.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:04:25.008 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.008 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:25.008 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.008 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.009 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.009 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:04:25.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.009 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:25.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:25.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:25.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:25.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:04:25.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:04:25.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:04:25.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:04:25.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:04:25.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:04:25.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:04:25.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:04:25.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:04:25.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:25.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:04:25.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:04:25.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:04:25.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:04:25.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:04:25.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:04:25.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:04:25.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:04:25.016 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.016 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.017 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.017 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:04:25.017 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:25.017 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.487624136000001, 2023-07-02T14:04:25.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:04:25.017 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.018 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:25.018 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.018 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.018 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.018 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:04:25.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.018 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:25.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:25.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:25.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:04:25.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:04:25.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:04:25.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:04:25.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:04:25.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:04:25.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:04:25.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:04:25.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:04:25.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:04:25.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:25.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:04:25.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:04:25.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:04:25.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:04:25.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:04:25.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:04:25.025 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.025 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.025 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.025 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:25.026 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:25.026 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.162141431999999, 2023-07-02T14:04:25.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:04:25.026 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.026 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.026 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.026 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.027 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.027 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:04:25.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.027 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:25.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:25.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:25.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:04:25.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:04:25.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:04:25.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:04:25.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:04:25.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:04:25.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:04:25.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:04:25.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:25.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:04:25.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:25.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:04:25.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:04:25.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:04:25.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:04:25.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:04:25.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:04:25.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:25.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:04:25.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:04:25.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:04:25.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:04:25.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:04:25.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:04:25.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:04:25.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:04:25.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:04:25.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:04:25.037 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.037 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.037 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.037 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3887 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:25.037 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:25.038 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.158278242999998, 2023-07-02T14:04:25.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998, 2023-07-02T14:04:25.038 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.038 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.038 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.038 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.038 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.038 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3887 2023-07-02T14:04:25.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.039 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:25.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:25.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:25.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784019+0000", 2023-07-02T14:04:25.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4073999999999998e-05 2023-07-02T14:04:25.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784063+0000", 2023-07-02T14:04:25.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6630000000000004e-06 2023-07-02T14:04:25.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784070+0000", 2023-07-02T14:04:25.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.286e-06 2023-07-02T14:04:25.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784073+0000", 2023-07-02T14:04:25.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4941999999999998e-05 2023-07-02T14:04:25.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784118+0000", 2023-07-02T14:04:25.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8120000000000008e-06 2023-07-02T14:04:25.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:25.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784126+0000", 2023-07-02T14:04:25.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32999709399999999 2023-07-02T14:04:25.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114123+0000", 2023-07-02T14:04:25.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000157257 2023-07-02T14:04:25.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114280+0000", 2023-07-02T14:04:25.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000341152 2023-07-02T14:04:25.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:25.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114622+0000", 2023-07-02T14:04:25.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019539766 2023-07-02T14:04:25.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.134161+0000", 2023-07-02T14:04:25.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015820796000000002 2023-07-02T14:04:25.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149982+0000", 2023-07-02T14:04:25.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.05e-05 2023-07-02T14:04:25.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150043+0000", 2023-07-02T14:04:25.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6453999999999999e-05 2023-07-02T14:04:25.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150089+0000", 2023-07-02T14:04:25.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998 2023-07-02T14:04:25.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.049 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.049 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.049 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3939 2.1 2:9ef34a83:::rbd_data.125de1449605.00000000000000b8:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:25.049 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:25.049 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.158220735, 2023-07-02T14:04:25.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001, 2023-07-02T14:04:25.050 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.050 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.050 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.050 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.050 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.050 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3939 2023-07-02T14:04:25.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.050 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:25.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:25.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:25.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:25.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046852830000000002 2023-07-02T14:04:25.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788763+0000", 2023-07-02T14:04:25.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3185e-05 2023-07-02T14:04:25.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788776+0000", 2023-07-02T14:04:25.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8470000000000003e-06 2023-07-02T14:04:25.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788781+0000", 2023-07-02T14:04:25.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7104000000000002e-05 2023-07-02T14:04:25.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788828+0000", 2023-07-02T14:04:25.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3516000000000001e-05 2023-07-02T14:04:25.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:25.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788841+0000", 2023-07-02T14:04:25.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33064513600000001 2023-07-02T14:04:25.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119487+0000", 2023-07-02T14:04:25.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9876999999999997e-05 2023-07-02T14:04:25.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119537+0000", 2023-07-02T14:04:25.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014160999999999999 2023-07-02T14:04:25.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:25.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119678+0000", 2023-07-02T14:04:25.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023054392 2023-07-02T14:04:25.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.142733+0000", 2023-07-02T14:04:25.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074065470000000003 2023-07-02T14:04:25.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150139+0000", 2023-07-02T14:04:25.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5025999999999997e-05 2023-07-02T14:04:25.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150204+0000", 2023-07-02T14:04:25.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147999999999997e-05 2023-07-02T14:04:25.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150241+0000", 2023-07-02T14:04:25.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001 2023-07-02T14:04:25.060 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.060 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.060 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.061 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3940 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:25.061 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:25.061 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.153509702999997, 2023-07-02T14:04:25.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001, 2023-07-02T14:04:25.061 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.061 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.062 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.062 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.062 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.062 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3940 2023-07-02T14:04:25.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.062 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:25.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:25.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:25.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788788+0000", 2023-07-02T14:04:25.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043166560000000003 2023-07-02T14:04:25.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793105+0000", 2023-07-02T14:04:25.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5970000000000001e-05 2023-07-02T14:04:25.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793121+0000", 2023-07-02T14:04:25.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0939999999999998e-06 2023-07-02T14:04:25.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793126+0000", 2023-07-02T14:04:25.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2438000000000002e-05 2023-07-02T14:04:25.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793178+0000", 2023-07-02T14:04:25.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2019e-05 2023-07-02T14:04:25.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:25.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793190+0000", 2023-07-02T14:04:25.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33208757799999999 2023-07-02T14:04:25.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125278+0000", 2023-07-02T14:04:25.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5255999999999997e-05 2023-07-02T14:04:25.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125373+0000", 2023-07-02T14:04:25.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015674999999999999 2023-07-02T14:04:25.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:25.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125530+0000", 2023-07-02T14:04:25.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018767308 2023-07-02T14:04:25.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144297+0000", 2023-07-02T14:04:25.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060291959999999997 2023-07-02T14:04:25.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150326+0000", 2023-07-02T14:04:25.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3886999999999993e-05 2023-07-02T14:04:25.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150390+0000", 2023-07-02T14:04:25.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010255 2023-07-02T14:04:25.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150493+0000", 2023-07-02T14:04:25.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001 2023-07-02T14:04:25.072 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.072 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.072 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.073 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3947 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:25.073 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:25.073 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.149164175999999, 2023-07-02T14:04:25.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999, 2023-07-02T14:04:25.073 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.073 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.073 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.073 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.074 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.074 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3947 2023-07-02T14:04:25.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.074 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:25.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:25.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:25.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793133+0000", 2023-07-02T14:04:25.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2519999999999999e-05 2023-07-02T14:04:25.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793176+0000", 2023-07-02T14:04:25.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:04:25.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793180+0000", 2023-07-02T14:04:25.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1709999999999998e-06 2023-07-02T14:04:25.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793184+0000", 2023-07-02T14:04:25.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1721999999999999e-05 2023-07-02T14:04:25.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793215+0000", 2023-07-02T14:04:25.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1280000000000008e-06 2023-07-02T14:04:25.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:25.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793224+0000", 2023-07-02T14:04:25.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34218549599999998 2023-07-02T14:04:25.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135409+0000", 2023-07-02T14:04:25.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117442 2023-07-02T14:04:25.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135526+0000", 2023-07-02T14:04:25.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000206257 2023-07-02T14:04:25.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:25.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135733+0000", 2023-07-02T14:04:25.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085700029999999996 2023-07-02T14:04:25.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144303+0000", 2023-07-02T14:04:25.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006278652 2023-07-02T14:04:25.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150581+0000", 2023-07-02T14:04:25.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4765999999999996e-05 2023-07-02T14:04:25.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150646+0000", 2023-07-02T14:04:25.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4777999999999995e-05 2023-07-02T14:04:25.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150711+0000", 2023-07-02T14:04:25.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999 2023-07-02T14:04:25.084 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.084 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.084 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.084 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3980 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 65536~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:25.085 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:25.085 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.149110294000003, 2023-07-02T14:04:25.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999, 2023-07-02T14:04:25.085 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.085 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.085 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.085 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.085 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.086 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3980 2023-07-02T14:04:25.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.086 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:25.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:25.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:25.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:25.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020244260000000002 2023-07-02T14:04:25.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795212+0000", 2023-07-02T14:04:25.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0005e-05 2023-07-02T14:04:25.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795222+0000", 2023-07-02T14:04:25.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9860000000000002e-06 2023-07-02T14:04:25.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795227+0000", 2023-07-02T14:04:25.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6369999999999999e-05 2023-07-02T14:04:25.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795264+0000", 2023-07-02T14:04:25.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1052e-05 2023-07-02T14:04:25.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:25.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795275+0000", 2023-07-02T14:04:25.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.343850459 2023-07-02T14:04:25.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139125+0000", 2023-07-02T14:04:25.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6760000000000004e-05 2023-07-02T14:04:25.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139202+0000", 2023-07-02T14:04:25.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015092799999999999 2023-07-02T14:04:25.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:25.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139353+0000", 2023-07-02T14:04:25.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011425940000000001 2023-07-02T14:04:25.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150779+0000", 2023-07-02T14:04:25.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033223060000000001 2023-07-02T14:04:25.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154101+0000", 2023-07-02T14:04:25.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9268999999999998e-05 2023-07-02T14:04:25.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154161+0000", 2023-07-02T14:04:25.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6499999999999999e-05 2023-07-02T14:04:25.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154207+0000", 2023-07-02T14:04:25.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999 2023-07-02T14:04:25.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.095 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.096 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.096 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3982 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1835008~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:25.096 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:25.096 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.147065007000002, 2023-07-02T14:04:25.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002, 2023-07-02T14:04:25.097 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.097 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.097 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.097 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.097 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.097 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3982 2023-07-02T14:04:25.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.097 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:25.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:25.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:25.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795233+0000", 2023-07-02T14:04:25.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8680999999999998e-05 2023-07-02T14:04:25.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795272+0000", 2023-07-02T14:04:25.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5629999999999999e-06 2023-07-02T14:04:25.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795277+0000", 2023-07-02T14:04:25.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8700000000000001e-06 2023-07-02T14:04:25.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795280+0000", 2023-07-02T14:04:25.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1964000000000001e-05 2023-07-02T14:04:25.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795312+0000", 2023-07-02T14:04:25.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.314e-06 2023-07-02T14:04:25.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:25.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795319+0000", 2023-07-02T14:04:25.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34828442599999998 2023-07-02T14:04:25.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143603+0000", 2023-07-02T14:04:25.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2438000000000001e-05 2023-07-02T14:04:25.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143665+0000", 2023-07-02T14:04:25.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3079999999999997e-05 2023-07-02T14:04:25.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:25.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143759+0000", 2023-07-02T14:04:25.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070633079999999999 2023-07-02T14:04:25.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150822+0000", 2023-07-02T14:04:25.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034430210000000001 2023-07-02T14:04:25.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154265+0000", 2023-07-02T14:04:25.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4249999999999998e-05 2023-07-02T14:04:25.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154309+0000", 2023-07-02T14:04:25.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2020000000000001e-05 2023-07-02T14:04:25.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154351+0000", 2023-07-02T14:04:25.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002 2023-07-02T14:04:25.107 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.107 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.107 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.108 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4054 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1900544~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:25.108 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:25.108 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.147013336000001, 2023-07-02T14:04:25.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002, 2023-07-02T14:04:25.108 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.108 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.109 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.109 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.109 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.109 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4054 2023-07-02T14:04:25.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.109 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:25.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:25.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:25.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:25.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018874860000000001 2023-07-02T14:04:25.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797172+0000", 2023-07-02T14:04:25.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3000000000000007e-06 2023-07-02T14:04:25.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797182+0000", 2023-07-02T14:04:25.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6260000000000003e-06 2023-07-02T14:04:25.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797186+0000", 2023-07-02T14:04:25.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9013e-05 2023-07-02T14:04:25.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797235+0000", 2023-07-02T14:04:25.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5990000000000008e-06 2023-07-02T14:04:25.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:25.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797245+0000", 2023-07-02T14:04:25.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34785621999999999 2023-07-02T14:04:25.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145101+0000", 2023-07-02T14:04:25.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7762000000000006e-05 2023-07-02T14:04:25.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145169+0000", 2023-07-02T14:04:25.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013469400000000001 2023-07-02T14:04:25.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:25.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145304+0000", 2023-07-02T14:04:25.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089538039999999992 2023-07-02T14:04:25.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154257+0000", 2023-07-02T14:04:25.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104218 2023-07-02T14:04:25.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154362+0000", 2023-07-02T14:04:25.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4168000000000001e-05 2023-07-02T14:04:25.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154406+0000", 2023-07-02T14:04:25.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3098000000000001e-05 2023-07-02T14:04:25.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154429+0000", 2023-07-02T14:04:25.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002 2023-07-02T14:04:25.119 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.119 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.119 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.120 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4056 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3670016~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:25.120 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:25.120 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.145106822999999, 2023-07-02T14:04:25.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997, 2023-07-02T14:04:25.120 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.120 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.121 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.121 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.121 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.121 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4056 2023-07-02T14:04:25.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.121 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:25.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:25.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:25.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797191+0000", 2023-07-02T14:04:25.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0438999999999997e-05 2023-07-02T14:04:25.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797232+0000", 2023-07-02T14:04:25.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6929999999999998e-06 2023-07-02T14:04:25.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797237+0000", 2023-07-02T14:04:25.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9739999999999998e-06 2023-07-02T14:04:25.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797240+0000", 2023-07-02T14:04:25.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0243000000000002e-05 2023-07-02T14:04:25.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797270+0000", 2023-07-02T14:04:25.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9889999999999997e-06 2023-07-02T14:04:25.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:25.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797276+0000", 2023-07-02T14:04:25.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35358319799999999 2023-07-02T14:04:25.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150859+0000", 2023-07-02T14:04:25.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1357e-05 2023-07-02T14:04:25.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150940+0000", 2023-07-02T14:04:25.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014903 2023-07-02T14:04:25.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:25.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151089+0000", 2023-07-02T14:04:25.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033512519999999999 2023-07-02T14:04:25.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154441+0000", 2023-07-02T14:04:25.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001050945 2023-07-02T14:04:25.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155492+0000", 2023-07-02T14:04:25.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2523e-05 2023-07-02T14:04:25.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155524+0000", 2023-07-02T14:04:25.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.057e-05 2023-07-02T14:04:25.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155555+0000", 2023-07-02T14:04:25.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997 2023-07-02T14:04:25.131 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.131 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.131 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.131 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4088 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3735552~458752] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:25.131 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:25.132 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.145054627999997, 2023-07-02T14:04:25.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998, 2023-07-02T14:04:25.132 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.132 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.132 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.132 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.132 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.132 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4088 2023-07-02T14:04:25.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.133 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:25.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:25.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:25.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:25.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00038503499999999999 2023-07-02T14:04:25.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797629+0000", 2023-07-02T14:04:25.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8739999999999995e-06 2023-07-02T14:04:25.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797637+0000", 2023-07-02T14:04:25.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7160000000000001e-06 2023-07-02T14:04:25.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797640+0000", 2023-07-02T14:04:25.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4342999999999998e-05 2023-07-02T14:04:25.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797695+0000", 2023-07-02T14:04:25.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0939999999999996e-06 2023-07-02T14:04:25.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:25.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797701+0000", 2023-07-02T14:04:25.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35408540799999999 2023-07-02T14:04:25.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151786+0000", 2023-07-02T14:04:25.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1953000000000006e-05 2023-07-02T14:04:25.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151868+0000", 2023-07-02T14:04:25.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015162800000000001 2023-07-02T14:04:25.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:25.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.152020+0000", 2023-07-02T14:04:25.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054561870000000004 2023-07-02T14:04:25.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.157476+0000", 2023-07-02T14:04:25.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059153199999999997 2023-07-02T14:04:25.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158068+0000", 2023-07-02T14:04:25.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2799e-05 2023-07-02T14:04:25.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158090+0000", 2023-07-02T14:04:25.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7235999999999999e-05 2023-07-02T14:04:25.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158108+0000", 2023-07-02T14:04:25.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998 2023-07-02T14:04:25.142 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.142 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.143 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.143 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4110 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:25.143 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:25.143 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.144653351000002, 2023-07-02T14:04:25.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864, 2023-07-02T14:04:25.143 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.144 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:25.144 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.144 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.144 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.144 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4110 2023-07-02T14:04:25.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.144 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:25.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:25.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:25.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:25.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9331e-05 2023-07-02T14:04:25.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797684+0000", 2023-07-02T14:04:25.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.147e-06 2023-07-02T14:04:25.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797689+0000", 2023-07-02T14:04:25.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5020000000000001e-06 2023-07-02T14:04:25.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797693+0000", 2023-07-02T14:04:25.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5254999999999999e-05 2023-07-02T14:04:25.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797748+0000", 2023-07-02T14:04:25.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5520000000000005e-06 2023-07-02T14:04:25.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:25.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797757+0000", 2023-07-02T14:04:25.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35674245700000001 2023-07-02T14:04:25.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154499+0000", 2023-07-02T14:04:25.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0090000000000001e-06 2023-07-02T14:04:25.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154506+0000", 2023-07-02T14:04:25.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864 2023-07-02T14:04:25.151 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.151 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.151 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.151 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:25.152 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:25.152 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.144601264999999, 2023-07-02T14:04:25.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998, 2023-07-02T14:04:25.152 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.152 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:25.152 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.152 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.153 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.153 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:04:25.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.153 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:25.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:25.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:25.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:25.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001714819 2023-07-02T14:04:25.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799412+0000", 2023-07-02T14:04:25.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7779999999999996e-06 2023-07-02T14:04:25.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799421+0000", 2023-07-02T14:04:25.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3510000000000002e-06 2023-07-02T14:04:25.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799424+0000", 2023-07-02T14:04:25.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0275999999999999e-05 2023-07-02T14:04:25.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799464+0000", 2023-07-02T14:04:25.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8389999999999996e-06 2023-07-02T14:04:25.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:25.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799473+0000", 2023-07-02T14:04:25.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35507638800000002 2023-07-02T14:04:25.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154550+0000", 2023-07-02T14:04:25.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6309999999999998e-06 2023-07-02T14:04:25.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154559+0000", 2023-07-02T14:04:25.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998 2023-07-02T14:04:25.159 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.160 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.160 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.160 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.160 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.160 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:25.161 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:04:25.162 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap21 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap22 b3fbb028-e267-4d25-a750-93ba18a2316a@snap22.diff 2023-07-02T14:04:25.162 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:25.161+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:25.163 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:04:25.199 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 51% complete...2023-07-02T14:04:25.198+0000 7f3751480700 0 -- 172.21.15.163:0/779255073 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x55d8a55fbd40 msgr2=0x55d8a55fe130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:25.201 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 100% complete...done. 2023-07-02T14:04:25.205 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:04:25.205 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=23 2023-07-02T14:04:25.205 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap23 2023-07-02T14:04:25.262 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:25.262+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:25.341 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:25.342 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:25.342 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:25.342 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:25.352 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:04:25.393 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:25.393+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:25.465 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:25.464+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:25.516 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:25.517 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:25.517 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:25.517 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:25.517 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:25.527 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:04:25.568 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:25.568+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:25.666 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:25.666+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:25.693 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:25.693 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:25.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:25.694 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:25.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.694 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:186 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:25.694 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:04:25.694 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 106.405605549, 2023-07-02T14:04:25.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003, 2023-07-02T14:04:25.694 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.695 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.695 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.695 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.695 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.695 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 186 2023-07-02T14:04:25.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.695 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:04:25.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:04:25.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:25.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285610+0000", 2023-07-02T14:04:25.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4279e-05 2023-07-02T14:04:25.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285664+0000", 2023-07-02T14:04:25.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3789999999999995e-06 2023-07-02T14:04:25.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285673+0000", 2023-07-02T14:04:25.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499999999999998e-06 2023-07-02T14:04:25.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285678+0000", 2023-07-02T14:04:25.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023261947000000002 2023-07-02T14:04:25.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.308940+0000", 2023-07-02T14:04:25.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010323399999999999 2023-07-02T14:04:25.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309044+0000", 2023-07-02T14:04:25.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018878300000000001 2023-07-02T14:04:25.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:25.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.309232+0000", 2023-07-02T14:04:25.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037874150000000001 2023-07-02T14:04:25.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313020+0000", 2023-07-02T14:04:25.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050976140000000003 2023-07-02T14:04:25.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318117+0000", 2023-07-02T14:04:25.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3182000000000001e-05 2023-07-02T14:04:25.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318161+0000", 2023-07-02T14:04:25.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5055999999999998e-05 2023-07-02T14:04:25.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318196+0000", 2023-07-02T14:04:25.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032585659000000003 2023-07-02T14:04:25.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.704 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.705 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:189 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:25.705 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:04:25.705 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 106.405531786, 2023-07-02T14:04:25.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997, 2023-07-02T14:04:25.705 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.705 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.705 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.705 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.706 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.706 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:04:25.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.706 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:04:25.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:04:25.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:25.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285683+0000", 2023-07-02T14:04:25.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8451999999999999e-05 2023-07-02T14:04:25.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285782+0000", 2023-07-02T14:04:25.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3650000000000004e-06 2023-07-02T14:04:25.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285790+0000", 2023-07-02T14:04:25.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8230000000000002e-06 2023-07-02T14:04:25.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285795+0000", 2023-07-02T14:04:25.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024619367999999999 2023-07-02T14:04:25.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310414+0000", 2023-07-02T14:04:25.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014244600000000001 2023-07-02T14:04:25.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310557+0000", 2023-07-02T14:04:25.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020612699999999999 2023-07-02T14:04:25.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:25.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310763+0000", 2023-07-02T14:04:25.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003427687 2023-07-02T14:04:25.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314191+0000", 2023-07-02T14:04:25.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040492089999999998 2023-07-02T14:04:25.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318240+0000", 2023-07-02T14:04:25.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0287999999999999e-05 2023-07-02T14:04:25.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318270+0000", 2023-07-02T14:04:25.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0868000000000002e-05 2023-07-02T14:04:25.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318311+0000", 2023-07-02T14:04:25.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997 2023-07-02T14:04:25.714 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.715 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.715 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.715 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:192 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:25.715 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:04:25.715 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 106.405415013, 2023-07-02T14:04:25.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003, 2023-07-02T14:04:25.716 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.716 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.716 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.717 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.717 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.717 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 192 2023-07-02T14:04:25.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.718 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:04:25.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:04:25.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:25.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285800+0000", 2023-07-02T14:04:25.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100417 2023-07-02T14:04:25.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285900+0000", 2023-07-02T14:04:25.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8244999999999999e-05 2023-07-02T14:04:25.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285919+0000", 2023-07-02T14:04:25.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2499999999999997e-06 2023-07-02T14:04:25.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285924+0000", 2023-07-02T14:04:25.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025288155999999999 2023-07-02T14:04:25.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311212+0000", 2023-07-02T14:04:25.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4073999999999996e-05 2023-07-02T14:04:25.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311286+0000", 2023-07-02T14:04:25.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016219699999999999 2023-07-02T14:04:25.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:25.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311448+0000", 2023-07-02T14:04:25.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066020599999999999 2023-07-02T14:04:25.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318050+0000", 2023-07-02T14:04:25.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000327986 2023-07-02T14:04:25.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318378+0000", 2023-07-02T14:04:25.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8693999999999998e-05 2023-07-02T14:04:25.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318427+0000", 2023-07-02T14:04:25.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6656999999999998e-05 2023-07-02T14:04:25.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318464+0000", 2023-07-02T14:04:25.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003 2023-07-02T14:04:25.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.734 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.735 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.735 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:195 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:25.736 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:04:25.736 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 106.405283509, 2023-07-02T14:04:25.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001, 2023-07-02T14:04:25.736 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.737 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.737 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.737 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.737 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.738 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 195 2023-07-02T14:04:25.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.738 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:04:25.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:04:25.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:25.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285931+0000", 2023-07-02T14:04:25.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4530999999999995e-05 2023-07-02T14:04:25.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286006+0000", 2023-07-02T14:04:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4113e-05 2023-07-02T14:04:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286020+0000", 2023-07-02T14:04:25.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7459999999999997e-06 2023-07-02T14:04:25.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286026+0000", 2023-07-02T14:04:25.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025997769 2023-07-02T14:04:25.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312023+0000", 2023-07-02T14:04:25.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4403000000000001e-05 2023-07-02T14:04:25.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312078+0000", 2023-07-02T14:04:25.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020581399999999999 2023-07-02T14:04:25.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:25.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312284+0000", 2023-07-02T14:04:25.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005771043 2023-07-02T14:04:25.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318055+0000", 2023-07-02T14:04:25.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00046015200000000002 2023-07-02T14:04:25.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318515+0000", 2023-07-02T14:04:25.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9444000000000003e-05 2023-07-02T14:04:25.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318564+0000", 2023-07-02T14:04:25.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2141000000000002e-05 2023-07-02T14:04:25.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318606+0000", 2023-07-02T14:04:25.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001 2023-07-02T14:04:25.755 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.755 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.755 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.756 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:198 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:25.756 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:25.757 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 106.40518355, 2023-07-02T14:04:25.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999, 2023-07-02T14:04:25.757 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.757 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.758 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.758 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.758 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.758 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 198 2023-07-02T14:04:25.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.759 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:25.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286031+0000", 2023-07-02T14:04:25.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00049286700000000002 2023-07-02T14:04:25.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286524+0000", 2023-07-02T14:04:25.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4938e-05 2023-07-02T14:04:25.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286539+0000", 2023-07-02T14:04:25.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0680000000000002e-06 2023-07-02T14:04:25.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286545+0000", 2023-07-02T14:04:25.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026497718 2023-07-02T14:04:25.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313043+0000", 2023-07-02T14:04:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0884999999999998e-05 2023-07-02T14:04:25.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313094+0000", 2023-07-02T14:04:25.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014109999999999999 2023-07-02T14:04:25.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:25.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313235+0000", 2023-07-02T14:04:25.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048258980000000003 2023-07-02T14:04:25.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318061+0000", 2023-07-02T14:04:25.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014346350000000001 2023-07-02T14:04:25.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319495+0000", 2023-07-02T14:04:25.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1986e-05 2023-07-02T14:04:25.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319547+0000", 2023-07-02T14:04:25.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0457e-05 2023-07-02T14:04:25.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319578+0000", 2023-07-02T14:04:25.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999 2023-07-02T14:04:25.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.776 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.776 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.777 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:201 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:25.777 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:04:25.777 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 106.404663125, 2023-07-02T14:04:25.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003, 2023-07-02T14:04:25.778 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.778 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.778 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.778 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.779 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.779 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 201 2023-07-02T14:04:25.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.779 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:04:25.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:04:25.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:25.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:04:25.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019213999999999999 2023-07-02T14:04:25.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286744+0000", 2023-07-02T14:04:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5279999999999992e-06 2023-07-02T14:04:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286753+0000", 2023-07-02T14:04:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4200000000000004e-06 2023-07-02T14:04:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286760+0000", 2023-07-02T14:04:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026818386999999999 2023-07-02T14:04:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313578+0000", 2023-07-02T14:04:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015965 2023-07-02T14:04:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313738+0000", 2023-07-02T14:04:25.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015767299999999999 2023-07-02T14:04:25.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:25.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313895+0000", 2023-07-02T14:04:25.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041714930000000001 2023-07-02T14:04:25.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318067+0000", 2023-07-02T14:04:25.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001561698 2023-07-02T14:04:25.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319629+0000", 2023-07-02T14:04:25.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2346000000000003e-05 2023-07-02T14:04:25.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319681+0000", 2023-07-02T14:04:25.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4691000000000003e-05 2023-07-02T14:04:25.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319726+0000", 2023-07-02T14:04:25.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003 2023-07-02T14:04:25.790 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.790 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.790 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.791 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:204 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:25.791 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:04:25.791 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 106.40444828699999, 2023-07-02T14:04:25.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998, 2023-07-02T14:04:25.791 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.791 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.791 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.792 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.792 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.792 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 204 2023-07-02T14:04:25.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.792 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:04:25.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:04:25.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:25.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:04:25.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2497999999999998e-05 2023-07-02T14:04:25.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286819+0000", 2023-07-02T14:04:25.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0099999999999995e-06 2023-07-02T14:04:25.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286827+0000", 2023-07-02T14:04:25.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000003e-06 2023-07-02T14:04:25.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286833+0000", 2023-07-02T14:04:25.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027384716999999999 2023-07-02T14:04:25.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314218+0000", 2023-07-02T14:04:25.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6204000000000004e-05 2023-07-02T14:04:25.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314294+0000", 2023-07-02T14:04:25.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015757499999999999 2023-07-02T14:04:25.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:25.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314452+0000", 2023-07-02T14:04:25.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036215560000000002 2023-07-02T14:04:25.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318073+0000", 2023-07-02T14:04:25.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017023979999999999 2023-07-02T14:04:25.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319776+0000", 2023-07-02T14:04:25.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9561000000000002e-05 2023-07-02T14:04:25.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319825+0000", 2023-07-02T14:04:25.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0329e-05 2023-07-02T14:04:25.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319866+0000", 2023-07-02T14:04:25.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998 2023-07-02T14:04:25.800 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.800 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.801 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.801 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:207 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:25.801 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:25.801 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 106.404373642, 2023-07-02T14:04:25.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003, 2023-07-02T14:04:25.801 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.802 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.802 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.802 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.802 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.802 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 207 2023-07-02T14:04:25.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.802 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:25.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:25.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:25.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286841+0000", 2023-07-02T14:04:25.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00068375100000000002 2023-07-02T14:04:25.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287525+0000", 2023-07-02T14:04:25.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1601e-05 2023-07-02T14:04:25.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287537+0000", 2023-07-02T14:04:25.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6409999999999996e-06 2023-07-02T14:04:25.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287543+0000", 2023-07-02T14:04:25.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027245844000000002 2023-07-02T14:04:25.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314789+0000", 2023-07-02T14:04:25.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8910000000000003e-05 2023-07-02T14:04:25.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314858+0000", 2023-07-02T14:04:25.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030992599999999997 2023-07-02T14:04:25.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:25.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315168+0000", 2023-07-02T14:04:25.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029122869999999999 2023-07-02T14:04:25.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318080+0000", 2023-07-02T14:04:25.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019963020000000001 2023-07-02T14:04:25.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320077+0000", 2023-07-02T14:04:25.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5116000000000005e-05 2023-07-02T14:04:25.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320142+0000", 2023-07-02T14:04:25.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3793000000000004e-05 2023-07-02T14:04:25.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320205+0000", 2023-07-02T14:04:25.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003 2023-07-02T14:04:25.811 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.811 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.811 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.811 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:236 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2424832~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:25.811 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:25.812 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 106.40366145900001, 2023-07-02T14:04:25.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003, 2023-07-02T14:04:25.812 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.812 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.812 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.812 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.812 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.812 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2023-07-02T14:04:25.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.813 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:25.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:25.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:25.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287553+0000", 2023-07-02T14:04:25.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0667e-05 2023-07-02T14:04:25.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287614+0000", 2023-07-02T14:04:25.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0454e-05 2023-07-02T14:04:25.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287624+0000", 2023-07-02T14:04:25.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8564999999999999e-05 2023-07-02T14:04:25.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287643+0000", 2023-07-02T14:04:25.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027977596 2023-07-02T14:04:25.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315621+0000", 2023-07-02T14:04:25.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000201728 2023-07-02T14:04:25.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315822+0000", 2023-07-02T14:04:25.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018436599999999999 2023-07-02T14:04:25.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:25.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316007+0000", 2023-07-02T14:04:25.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042423440000000003 2023-07-02T14:04:25.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320249+0000", 2023-07-02T14:04:25.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001452633 2023-07-02T14:04:25.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321702+0000", 2023-07-02T14:04:25.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3616000000000003e-05 2023-07-02T14:04:25.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321755+0000", 2023-07-02T14:04:25.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840999999999999e-05 2023-07-02T14:04:25.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321793+0000", 2023-07-02T14:04:25.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003 2023-07-02T14:04:25.821 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.821 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.821 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.822 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:251 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:25.822 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:25.822 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 106.403565922, 2023-07-02T14:04:25.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001, 2023-07-02T14:04:25.822 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.822 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.822 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.822 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.823 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.823 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 251 2023-07-02T14:04:25.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.823 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:25.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:25.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:25.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287649+0000", 2023-07-02T14:04:25.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3232000000000001e-05 2023-07-02T14:04:25.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287702+0000", 2023-07-02T14:04:25.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0010000000000007e-06 2023-07-02T14:04:25.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287711+0000", 2023-07-02T14:04:25.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3569999999999997e-06 2023-07-02T14:04:25.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287717+0000", 2023-07-02T14:04:25.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028686454 2023-07-02T14:04:25.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316403+0000", 2023-07-02T14:04:25.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7741000000000002e-05 2023-07-02T14:04:25.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316461+0000", 2023-07-02T14:04:25.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4313000000000004e-05 2023-07-02T14:04:25.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:25.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316545+0000", 2023-07-02T14:04:25.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003734032 2023-07-02T14:04:25.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320279+0000", 2023-07-02T14:04:25.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015237219999999999 2023-07-02T14:04:25.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321803+0000", 2023-07-02T14:04:25.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1927e-05 2023-07-02T14:04:25.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321845+0000", 2023-07-02T14:04:25.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5398999999999998e-05 2023-07-02T14:04:25.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321880+0000", 2023-07-02T14:04:25.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001 2023-07-02T14:04:25.831 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.831 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.831 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.832 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:255 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2555904~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:25.832 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:25.832 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 106.403151194, 2023-07-02T14:04:25.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002, 2023-07-02T14:04:25.832 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.832 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.833 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.833 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.833 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.833 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 255 2023-07-02T14:04:25.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.833 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:25.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:25.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:25.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288063+0000", 2023-07-02T14:04:25.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4459999999999997e-05 2023-07-02T14:04:25.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288118+0000", 2023-07-02T14:04:25.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0998999999999999e-05 2023-07-02T14:04:25.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288129+0000", 2023-07-02T14:04:25.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3219999999999998e-06 2023-07-02T14:04:25.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288135+0000", 2023-07-02T14:04:25.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029137386000000001 2023-07-02T14:04:25.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317273+0000", 2023-07-02T14:04:25.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7819999999999997e-05 2023-07-02T14:04:25.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317350+0000", 2023-07-02T14:04:25.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000174864 2023-07-02T14:04:25.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:25.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317525+0000", 2023-07-02T14:04:25.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027943519999999999 2023-07-02T14:04:25.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320320+0000", 2023-07-02T14:04:25.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015681460000000001 2023-07-02T14:04:25.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321888+0000", 2023-07-02T14:04:25.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9508999999999997e-05 2023-07-02T14:04:25.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321927+0000", 2023-07-02T14:04:25.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5428e-05 2023-07-02T14:04:25.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321963+0000", 2023-07-02T14:04:25.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002 2023-07-02T14:04:25.841 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.842 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.842 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.842 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:04:25.842 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:25.842 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 84.583540647999996, 2023-07-02T14:04:25.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:04:25.843 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.843 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:25.843 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.843 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.843 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.843 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:04:25.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.844 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:25.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:25.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:25.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:04:25.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:04:25.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:04:25.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:04:25.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:04:25.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:04:25.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:04:25.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:04:25.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:04:25.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:04:25.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:25.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:04:25.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:04:25.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:04:25.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:04:25.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:04:25.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:04:25.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:04:25.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:04:25.851 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.851 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.851 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.851 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:04:25.851 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:25.851 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 84.583479909999994, 2023-07-02T14:04:25.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:04:25.852 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.852 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.852 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.852 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.852 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.852 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:04:25.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.853 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:25.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:25.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:25.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:04:25.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:04:25.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:04:25.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:04:25.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:04:25.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:04:25.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:04:25.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:04:25.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:04:25.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:04:25.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:25.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:04:25.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:04:25.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:04:25.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:04:25.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:04:25.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:04:25.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:25.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:04:25.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:04:25.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:04:25.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:04:25.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:04:25.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:04:25.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:04:25.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:04:25.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:04:25.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:04:25.862 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.862 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.862 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.863 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:25.863 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:25.863 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.353025579999994, 2023-07-02T14:04:25.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:04:25.863 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.863 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.863 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.864 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.864 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.864 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:04:25.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.864 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:25.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:25.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:25.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:04:25.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:04:25.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:04:25.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:04:25.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.867 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:04:25.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:04:25.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:04:25.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:04:25.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:04:25.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:04:25.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:04:25.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:04:25.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:25.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:04:25.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:04:25.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:04:25.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:04:25.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:04:25.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:04:25.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:04:25.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:04:25.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:04:25.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:04:25.872 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.872 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.873 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.873 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2941 2.3 2:d3762e63:::rbd_data.125de1449605.00000000000000f8:head [write 65536~2031616] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:25.873 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:25.873 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.223922626000004, 2023-07-02T14:04:25.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002, 2023-07-02T14:04:25.873 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.874 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.874 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.874 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.874 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.874 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2941 2023-07-02T14:04:25.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.874 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:25.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:25.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:25.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467291+0000", 2023-07-02T14:04:25.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043022859999999998 2023-07-02T14:04:25.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471593+0000", 2023-07-02T14:04:25.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0999999999999999e-05 2023-07-02T14:04:25.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471614+0000", 2023-07-02T14:04:25.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0097e-05 2023-07-02T14:04:25.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471625+0000", 2023-07-02T14:04:25.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069808499999999998 2023-07-02T14:04:25.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472323+0000", 2023-07-02T14:04:25.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135635 2023-07-02T14:04:25.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472458+0000", 2023-07-02T14:04:25.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223365 2023-07-02T14:04:25.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:25.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472682+0000", 2023-07-02T14:04:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013761944999999999 2023-07-02T14:04:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.486444+0000", 2023-07-02T14:04:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016859747000000001 2023-07-02T14:04:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503303+0000", 2023-07-02T14:04:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6635000000000001e-05 2023-07-02T14:04:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503380+0000", 2023-07-02T14:04:25.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000161803 2023-07-02T14:04:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503542+0000", 2023-07-02T14:04:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002 2023-07-02T14:04:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2950 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 1507328~786432] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.215749317999993, 2023-07-02T14:04:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999, 2023-07-02T14:04:25.884 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2950 2023-07-02T14:04:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:25.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001147854 2023-07-02T14:04:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476614+0000", 2023-07-02T14:04:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.802e-05 2023-07-02T14:04:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476632+0000", 2023-07-02T14:04:25.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1860000000000003e-06 2023-07-02T14:04:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476639+0000", 2023-07-02T14:04:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061211240000000004 2023-07-02T14:04:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482760+0000", 2023-07-02T14:04:25.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8833999999999996e-05 2023-07-02T14:04:25.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482859+0000", 2023-07-02T14:04:25.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019270600000000001 2023-07-02T14:04:25.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:25.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.483051+0000", 2023-07-02T14:04:25.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060385630000000003 2023-07-02T14:04:25.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489090+0000", 2023-07-02T14:04:25.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024506918999999999 2023-07-02T14:04:25.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513597+0000", 2023-07-02T14:04:25.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9672999999999995e-05 2023-07-02T14:04:25.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513667+0000", 2023-07-02T14:04:25.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024329500000000001 2023-07-02T14:04:25.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513910+0000", 2023-07-02T14:04:25.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999 2023-07-02T14:04:25.894 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.894 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.894 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.894 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2951 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2686976~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:25.894 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:25.894 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.214568540000002, 2023-07-02T14:04:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999, 2023-07-02T14:04:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2951 2023-07-02T14:04:25.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.896 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:25.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476646+0000", 2023-07-02T14:04:25.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7639000000000003e-05 2023-07-02T14:04:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476684+0000", 2023-07-02T14:04:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3508000000000002e-05 2023-07-02T14:04:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476747+0000", 2023-07-02T14:04:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8499999999999994e-06 2023-07-02T14:04:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476755+0000", 2023-07-02T14:04:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078346160000000008 2023-07-02T14:04:25.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484590+0000", 2023-07-02T14:04:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7206000000000004e-05 2023-07-02T14:04:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484687+0000", 2023-07-02T14:04:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016243099999999999 2023-07-02T14:04:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484849+0000", 2023-07-02T14:04:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042482860000000004 2023-07-02T14:04:25.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489097+0000", 2023-07-02T14:04:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.02489504 2023-07-02T14:04:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513992+0000", 2023-07-02T14:04:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9972000000000001e-05 2023-07-02T14:04:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514052+0000", 2023-07-02T14:04:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149264 2023-07-02T14:04:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514202+0000", 2023-07-02T14:04:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999 2023-07-02T14:04:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2952 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2883584~12288] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.214454704999994, 2023-07-02T14:04:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651, 2023-07-02T14:04:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.905 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2952 2023-07-02T14:04:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:25.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476760+0000", 2023-07-02T14:04:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3664e-05 2023-07-02T14:04:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476774+0000", 2023-07-02T14:04:25.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.166e-06 2023-07-02T14:04:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476783+0000", 2023-07-02T14:04:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8720000000000001e-06 2023-07-02T14:04:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476788+0000", 2023-07-02T14:04:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087180529999999999 2023-07-02T14:04:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485506+0000", 2023-07-02T14:04:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5680999999999994e-05 2023-07-02T14:04:25.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485602+0000", 2023-07-02T14:04:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000222365 2023-07-02T14:04:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:25.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485824+0000", 2023-07-02T14:04:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003277379 2023-07-02T14:04:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489101+0000", 2023-07-02T14:04:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025151739999999999 2023-07-02T14:04:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514253+0000", 2023-07-02T14:04:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4749000000000002e-05 2023-07-02T14:04:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514288+0000", 2023-07-02T14:04:25.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130634 2023-07-02T14:04:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514419+0000", 2023-07-02T14:04:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651 2023-07-02T14:04:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.914 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9620 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 589824~65536] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:04:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:04:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.66901098199999998, 2023-07-02T14:04:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001, 2023-07-02T14:04:25.915 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9620 2023-07-02T14:04:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:04:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:04:25.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022203+0000", 2023-07-02T14:04:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0346000000000001e-05 2023-07-02T14:04:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022263+0000", 2023-07-02T14:04:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4657999999999999e-05 2023-07-02T14:04:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022278+0000", 2023-07-02T14:04:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:04:25.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022285+0000", 2023-07-02T14:04:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0096000000000003e-05 2023-07-02T14:04:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022335+0000", 2023-07-02T14:04:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2334000000000001e-05 2023-07-02T14:04:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022347+0000", 2023-07-02T14:04:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095653701999999993 2023-07-02T14:04:25.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118001+0000", 2023-07-02T14:04:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000185797 2023-07-02T14:04:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118187+0000", 2023-07-02T14:04:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026397900000000001 2023-07-02T14:04:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118451+0000", 2023-07-02T14:04:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051832800000000002 2023-07-02T14:04:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.123634+0000", 2023-07-02T14:04:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001227332 2023-07-02T14:04:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124861+0000", 2023-07-02T14:04:25.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9493999999999998e-05 2023-07-02T14:04:25.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124901+0000", 2023-07-02T14:04:25.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0462000000000001e-05 2023-07-02T14:04:25.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124931+0000", 2023-07-02T14:04:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001 2023-07-02T14:04:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9621 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 655360~1966080] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:04:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:04:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.66882359999999996, 2023-07-02T14:04:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999, 2023-07-02T14:04:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:25.927 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9621 2023-07-02T14:04:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:04:25.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:04:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022391+0000", 2023-07-02T14:04:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032053939999999999 2023-07-02T14:04:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025597+0000", 2023-07-02T14:04:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4307999999999999e-05 2023-07-02T14:04:25.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025611+0000", 2023-07-02T14:04:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.032e-06 2023-07-02T14:04:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:25.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025617+0000", 2023-07-02T14:04:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8977999999999998e-05 2023-07-02T14:04:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025656+0000", 2023-07-02T14:04:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:04:25.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:25.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025663+0000", 2023-07-02T14:04:25.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095220739999999998 2023-07-02T14:04:25.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:25.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120884+0000", 2023-07-02T14:04:25.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2525000000000001e-05 2023-07-02T14:04:25.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:25.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120956+0000", 2023-07-02T14:04:25.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016232400000000001 2023-07-02T14:04:25.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.121119+0000", 2023-07-02T14:04:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044007380000000004 2023-07-02T14:04:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.125519+0000", 2023-07-02T14:04:25.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000797209 2023-07-02T14:04:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126317+0000", 2023-07-02T14:04:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1228000000000004e-05 2023-07-02T14:04:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126378+0000", 2023-07-02T14:04:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978000000000003e-05 2023-07-02T14:04:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126430+0000", 2023-07-02T14:04:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999 2023-07-02T14:04:25.937 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.938 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.938 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.938 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:25.938 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:25.938 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:25.939 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:04:25.940 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:25.938+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:25.941 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:04:26.040 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:26.040+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:26.075 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:26.076 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:26.076 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:04:26.076 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:26.086 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:04:26.121 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:26.125 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:04:26.146 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:26.147 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:26.147+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:26.153 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:04:26.159 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 3 2023-07-02T14:04:26.179 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:26.208 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:26.224 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:26.224 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:26.228 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:26.228 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:04:26.229 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:04:26.229 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:04:26.229 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:26.229 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:04:26.229 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:04:26.231 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:26.232 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:26.234 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:04:26.278 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:26.278+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:26.343 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:26.342+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:26.358 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:26.358 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:26.368 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:04:26.451 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:26.450+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:26.532 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:26.532 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:26.543 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:04:26.544 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:26.544+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:26.645 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:26.645+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:26.707 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:26.707 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:26.717 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:26.761 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:26.760+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:26.846 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:26.847 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:26.847 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:26.846+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:26.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:26.947 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:26.947+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:27.048 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:27.047+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:27.065 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:27.066 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:27.086 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:04:27.149 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:27.148+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:27.250 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:27.250+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:27.328 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:27.329 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:27.350 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:04:27.351 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:27.349+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:27.451 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:27.451+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:27.553 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:27.552+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:27.598 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:27.598 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:27.614 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:04:27.653 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:27.652+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:27.753 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:27.753+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:27.855 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:27.854+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:27.881 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:27.881 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:27.900 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:04:27.986 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:27.985+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:28.056 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:28.056+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:28.140 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:04:28.145 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:28.145 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:28.156 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:28.156+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:28.167 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:04:28.233 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:04:28.298 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:28.296+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:28.358 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:28.357+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:28.425 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:28.426 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:28.439 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:04:28.459 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:28.458+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:28.479 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:28.514 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:28.548 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:28.570 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:28.570+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:28.584 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:28.616 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:28.617 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:04:28.617 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:04:28.617 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:04:28.617 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:28.617 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:04:28.618 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:04:28.622 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:28.623 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:28.664 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:28.664+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:28.677 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:28.678 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:28.678 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:28.678 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:28.697 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:04:28.782 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:28.781+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:28.862 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:28.862+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:28.949 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:28.949 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:28.949 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:28.949 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:28.949 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:28.962 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:28.962+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:28.965 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:04:29.064 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:29.063+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:29.132 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:29.133 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:29.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:29.133 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:29.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.133 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:189 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:29.133 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:04:29.134 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 109.845015499, 2023-07-02T14:04:29.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997, 2023-07-02T14:04:29.134 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:29.134 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.134 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:29.134 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:29.134 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.134 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:04:29.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.135 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:29.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:29.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:04:29.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:29.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:29.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285684+0000", 2023-07-02T14:04:29.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:29.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:29.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285683+0000", 2023-07-02T14:04:29.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8451999999999999e-05 2023-07-02T14:04:29.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:29.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285782+0000", 2023-07-02T14:04:29.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3650000000000004e-06 2023-07-02T14:04:29.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:29.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285790+0000", 2023-07-02T14:04:29.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8230000000000002e-06 2023-07-02T14:04:29.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285795+0000", 2023-07-02T14:04:29.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024619367999999999 2023-07-02T14:04:29.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310414+0000", 2023-07-02T14:04:29.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014244600000000001 2023-07-02T14:04:29.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:29.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310557+0000", 2023-07-02T14:04:29.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020612699999999999 2023-07-02T14:04:29.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:29.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.310763+0000", 2023-07-02T14:04:29.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003427687 2023-07-02T14:04:29.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:29.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314191+0000", 2023-07-02T14:04:29.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040492089999999998 2023-07-02T14:04:29.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:29.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318240+0000", 2023-07-02T14:04:29.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0287999999999999e-05 2023-07-02T14:04:29.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:29.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318270+0000", 2023-07-02T14:04:29.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0868000000000002e-05 2023-07-02T14:04:29.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:29.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318311+0000", 2023-07-02T14:04:29.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032627063999999997 2023-07-02T14:04:29.143 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.143 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.143 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.144 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:192 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:29.144 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:04:29.144 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 109.844898726, 2023-07-02T14:04:29.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003, 2023-07-02T14:04:29.144 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:29.144 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.145 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:29.145 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:29.145 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.145 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 192 2023-07-02T14:04:29.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.145 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:29.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:29.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:04:29.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:29.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:29.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285801+0000", 2023-07-02T14:04:29.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:29.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:29.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285800+0000", 2023-07-02T14:04:29.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100417 2023-07-02T14:04:29.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:29.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285900+0000", 2023-07-02T14:04:29.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8244999999999999e-05 2023-07-02T14:04:29.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:29.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285919+0000", 2023-07-02T14:04:29.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2499999999999997e-06 2023-07-02T14:04:29.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285924+0000", 2023-07-02T14:04:29.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025288155999999999 2023-07-02T14:04:29.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311212+0000", 2023-07-02T14:04:29.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4073999999999996e-05 2023-07-02T14:04:29.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:29.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311286+0000", 2023-07-02T14:04:29.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016219699999999999 2023-07-02T14:04:29.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:29.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.311448+0000", 2023-07-02T14:04:29.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066020599999999999 2023-07-02T14:04:29.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:29.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318050+0000", 2023-07-02T14:04:29.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000327986 2023-07-02T14:04:29.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:29.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318378+0000", 2023-07-02T14:04:29.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8693999999999998e-05 2023-07-02T14:04:29.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:29.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318427+0000", 2023-07-02T14:04:29.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6656999999999998e-05 2023-07-02T14:04:29.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:29.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318464+0000", 2023-07-02T14:04:29.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032663082000000003 2023-07-02T14:04:29.154 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.154 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.154 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.154 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:195 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:29.154 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:04:29.155 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 109.844767222, 2023-07-02T14:04:29.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001, 2023-07-02T14:04:29.155 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:29.155 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.155 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:29.155 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:29.155 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.155 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 195 2023-07-02T14:04:29.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.156 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:29.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:29.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:04:29.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:29.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:29.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285932+0000", 2023-07-02T14:04:29.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:29.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.285931+0000", 2023-07-02T14:04:29.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4530999999999995e-05 2023-07-02T14:04:29.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:29.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286006+0000", 2023-07-02T14:04:29.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4113e-05 2023-07-02T14:04:29.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:29.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286020+0000", 2023-07-02T14:04:29.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7459999999999997e-06 2023-07-02T14:04:29.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286026+0000", 2023-07-02T14:04:29.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025997769 2023-07-02T14:04:29.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312023+0000", 2023-07-02T14:04:29.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4403000000000001e-05 2023-07-02T14:04:29.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:29.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312078+0000", 2023-07-02T14:04:29.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020581399999999999 2023-07-02T14:04:29.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:29.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.312284+0000", 2023-07-02T14:04:29.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005771043 2023-07-02T14:04:29.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:29.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318055+0000", 2023-07-02T14:04:29.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00046015200000000002 2023-07-02T14:04:29.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:29.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318515+0000", 2023-07-02T14:04:29.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9444000000000003e-05 2023-07-02T14:04:29.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:29.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318564+0000", 2023-07-02T14:04:29.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2141000000000002e-05 2023-07-02T14:04:29.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:29.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318606+0000", 2023-07-02T14:04:29.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032674384000000001 2023-07-02T14:04:29.164 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.164 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.164 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.165 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:198 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:29.165 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:29.165 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 109.84466726300001, 2023-07-02T14:04:29.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999, 2023-07-02T14:04:29.165 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:29.165 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.166 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:29.166 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:29.166 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.166 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 198 2023-07-02T14:04:29.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.166 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:29.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:29.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:29.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:29.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:29.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:29.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:29.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286031+0000", 2023-07-02T14:04:29.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00049286700000000002 2023-07-02T14:04:29.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:29.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286524+0000", 2023-07-02T14:04:29.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4938e-05 2023-07-02T14:04:29.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:29.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286539+0000", 2023-07-02T14:04:29.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0680000000000002e-06 2023-07-02T14:04:29.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286545+0000", 2023-07-02T14:04:29.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026497718 2023-07-02T14:04:29.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313043+0000", 2023-07-02T14:04:29.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0884999999999998e-05 2023-07-02T14:04:29.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:29.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313094+0000", 2023-07-02T14:04:29.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014109999999999999 2023-07-02T14:04:29.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:29.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313235+0000", 2023-07-02T14:04:29.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048258980000000003 2023-07-02T14:04:29.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:29.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318061+0000", 2023-07-02T14:04:29.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014346350000000001 2023-07-02T14:04:29.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:29.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319495+0000", 2023-07-02T14:04:29.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1986e-05 2023-07-02T14:04:29.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:29.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319547+0000", 2023-07-02T14:04:29.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0457e-05 2023-07-02T14:04:29.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:29.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319578+0000", 2023-07-02T14:04:29.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999 2023-07-02T14:04:29.175 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.175 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.175 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.175 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:201 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:29.175 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:04:29.175 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 109.844146838, 2023-07-02T14:04:29.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003, 2023-07-02T14:04:29.176 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:29.176 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.176 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:29.176 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:29.176 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.176 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 201 2023-07-02T14:04:29.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.177 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:29.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:29.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:04:29.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:29.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:29.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:04:29.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:29.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286552+0000", 2023-07-02T14:04:29.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019213999999999999 2023-07-02T14:04:29.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:29.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286744+0000", 2023-07-02T14:04:29.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5279999999999992e-06 2023-07-02T14:04:29.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:29.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286753+0000", 2023-07-02T14:04:29.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4200000000000004e-06 2023-07-02T14:04:29.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286760+0000", 2023-07-02T14:04:29.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026818386999999999 2023-07-02T14:04:29.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313578+0000", 2023-07-02T14:04:29.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015965 2023-07-02T14:04:29.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:29.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313738+0000", 2023-07-02T14:04:29.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015767299999999999 2023-07-02T14:04:29.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:29.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313895+0000", 2023-07-02T14:04:29.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041714930000000001 2023-07-02T14:04:29.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:29.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318067+0000", 2023-07-02T14:04:29.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001561698 2023-07-02T14:04:29.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:29.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319629+0000", 2023-07-02T14:04:29.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2346000000000003e-05 2023-07-02T14:04:29.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:29.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319681+0000", 2023-07-02T14:04:29.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4691000000000003e-05 2023-07-02T14:04:29.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:29.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319726+0000", 2023-07-02T14:04:29.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033173195000000003 2023-07-02T14:04:29.185 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.185 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.185 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.186 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:204 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:29.186 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:04:29.186 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 109.843932, 2023-07-02T14:04:29.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998, 2023-07-02T14:04:29.186 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:29.186 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.186 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:29.187 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:29.187 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.187 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 204 2023-07-02T14:04:29.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.187 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:29.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:29.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:04:29.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:29.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:29.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:04:29.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:29.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286767+0000", 2023-07-02T14:04:29.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2497999999999998e-05 2023-07-02T14:04:29.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:29.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286819+0000", 2023-07-02T14:04:29.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.0099999999999995e-06 2023-07-02T14:04:29.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:29.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286827+0000", 2023-07-02T14:04:29.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1480000000000003e-06 2023-07-02T14:04:29.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286833+0000", 2023-07-02T14:04:29.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027384716999999999 2023-07-02T14:04:29.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314218+0000", 2023-07-02T14:04:29.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6204000000000004e-05 2023-07-02T14:04:29.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:29.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314294+0000", 2023-07-02T14:04:29.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015757499999999999 2023-07-02T14:04:29.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:29.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314452+0000", 2023-07-02T14:04:29.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036215560000000002 2023-07-02T14:04:29.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:29.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318073+0000", 2023-07-02T14:04:29.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017023979999999999 2023-07-02T14:04:29.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:29.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319776+0000", 2023-07-02T14:04:29.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9561000000000002e-05 2023-07-02T14:04:29.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:29.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319825+0000", 2023-07-02T14:04:29.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0329e-05 2023-07-02T14:04:29.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:29.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319866+0000", 2023-07-02T14:04:29.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033098245999999998 2023-07-02T14:04:29.195 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.195 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.196 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.196 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:207 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:29.196 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:29.196 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 109.843857355, 2023-07-02T14:04:29.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003, 2023-07-02T14:04:29.197 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:29.197 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.197 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:29.197 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:29.197 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.197 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 207 2023-07-02T14:04:29.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.197 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:29.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:29.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:29.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:29.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:29.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:29.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:29.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286841+0000", 2023-07-02T14:04:29.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00068375100000000002 2023-07-02T14:04:29.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:29.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287525+0000", 2023-07-02T14:04:29.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1601e-05 2023-07-02T14:04:29.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:29.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287537+0000", 2023-07-02T14:04:29.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6409999999999996e-06 2023-07-02T14:04:29.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287543+0000", 2023-07-02T14:04:29.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027245844000000002 2023-07-02T14:04:29.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314789+0000", 2023-07-02T14:04:29.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8910000000000003e-05 2023-07-02T14:04:29.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:29.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314858+0000", 2023-07-02T14:04:29.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030992599999999997 2023-07-02T14:04:29.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:29.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315168+0000", 2023-07-02T14:04:29.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029122869999999999 2023-07-02T14:04:29.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:29.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318080+0000", 2023-07-02T14:04:29.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019963020000000001 2023-07-02T14:04:29.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:29.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320077+0000", 2023-07-02T14:04:29.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5116000000000005e-05 2023-07-02T14:04:29.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:29.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320142+0000", 2023-07-02T14:04:29.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3793000000000004e-05 2023-07-02T14:04:29.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:29.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320205+0000", 2023-07-02T14:04:29.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003 2023-07-02T14:04:29.206 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.206 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.206 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.206 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:236 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2424832~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:29.207 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:29.207 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 109.84314517200001, 2023-07-02T14:04:29.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003, 2023-07-02T14:04:29.207 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:29.207 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.207 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:29.207 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:29.207 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.208 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2023-07-02T14:04:29.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.208 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:29.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:29.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:29.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:29.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:29.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:29.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:29.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287553+0000", 2023-07-02T14:04:29.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0667e-05 2023-07-02T14:04:29.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:29.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287614+0000", 2023-07-02T14:04:29.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0454e-05 2023-07-02T14:04:29.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:29.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287624+0000", 2023-07-02T14:04:29.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8564999999999999e-05 2023-07-02T14:04:29.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287643+0000", 2023-07-02T14:04:29.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027977596 2023-07-02T14:04:29.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315621+0000", 2023-07-02T14:04:29.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000201728 2023-07-02T14:04:29.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:29.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315822+0000", 2023-07-02T14:04:29.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018436599999999999 2023-07-02T14:04:29.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:29.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316007+0000", 2023-07-02T14:04:29.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042423440000000003 2023-07-02T14:04:29.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:29.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320249+0000", 2023-07-02T14:04:29.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001452633 2023-07-02T14:04:29.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:29.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321702+0000", 2023-07-02T14:04:29.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3616000000000003e-05 2023-07-02T14:04:29.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:29.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321755+0000", 2023-07-02T14:04:29.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840999999999999e-05 2023-07-02T14:04:29.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:29.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321793+0000", 2023-07-02T14:04:29.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003 2023-07-02T14:04:29.216 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.216 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.217 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.217 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:251 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:29.217 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:29.217 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 109.843049635, 2023-07-02T14:04:29.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001, 2023-07-02T14:04:29.217 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:29.218 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.218 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:29.218 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:29.218 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.218 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 251 2023-07-02T14:04:29.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.218 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:29.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:29.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:29.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:29.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:29.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:29.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:29.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287649+0000", 2023-07-02T14:04:29.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3232000000000001e-05 2023-07-02T14:04:29.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:29.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287702+0000", 2023-07-02T14:04:29.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0010000000000007e-06 2023-07-02T14:04:29.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:29.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287711+0000", 2023-07-02T14:04:29.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3569999999999997e-06 2023-07-02T14:04:29.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287717+0000", 2023-07-02T14:04:29.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028686454 2023-07-02T14:04:29.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316403+0000", 2023-07-02T14:04:29.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7741000000000002e-05 2023-07-02T14:04:29.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:29.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316461+0000", 2023-07-02T14:04:29.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4313000000000004e-05 2023-07-02T14:04:29.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:29.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316545+0000", 2023-07-02T14:04:29.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003734032 2023-07-02T14:04:29.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:29.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320279+0000", 2023-07-02T14:04:29.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015237219999999999 2023-07-02T14:04:29.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:29.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321803+0000", 2023-07-02T14:04:29.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1927e-05 2023-07-02T14:04:29.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:29.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321845+0000", 2023-07-02T14:04:29.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5398999999999998e-05 2023-07-02T14:04:29.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:29.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321880+0000", 2023-07-02T14:04:29.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001 2023-07-02T14:04:29.227 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.227 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.227 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.227 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:255 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2555904~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:29.227 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:29.228 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 109.842634907, 2023-07-02T14:04:29.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002, 2023-07-02T14:04:29.228 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:29.228 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.228 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:29.228 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:29.228 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.228 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 255 2023-07-02T14:04:29.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.229 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:29.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:29.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:29.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:29.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:29.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:29.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:29.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288063+0000", 2023-07-02T14:04:29.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4459999999999997e-05 2023-07-02T14:04:29.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:29.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288118+0000", 2023-07-02T14:04:29.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0998999999999999e-05 2023-07-02T14:04:29.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:29.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288129+0000", 2023-07-02T14:04:29.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3219999999999998e-06 2023-07-02T14:04:29.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288135+0000", 2023-07-02T14:04:29.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029137386000000001 2023-07-02T14:04:29.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317273+0000", 2023-07-02T14:04:29.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7819999999999997e-05 2023-07-02T14:04:29.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:29.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317350+0000", 2023-07-02T14:04:29.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000174864 2023-07-02T14:04:29.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:29.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317525+0000", 2023-07-02T14:04:29.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027943519999999999 2023-07-02T14:04:29.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:29.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320320+0000", 2023-07-02T14:04:29.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015681460000000001 2023-07-02T14:04:29.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:29.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321888+0000", 2023-07-02T14:04:29.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9508999999999997e-05 2023-07-02T14:04:29.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:29.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321927+0000", 2023-07-02T14:04:29.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5428e-05 2023-07-02T14:04:29.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:29.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321963+0000", 2023-07-02T14:04:29.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002 2023-07-02T14:04:29.237 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.237 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.237 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.238 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:04:29.238 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:29.238 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.023024360999997, 2023-07-02T14:04:29.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:04:29.238 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:29.238 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:29.239 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:29.239 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:29.239 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.239 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:04:29.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.239 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:29.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:29.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:29.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:29.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:29.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:29.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:29.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:29.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:04:29.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:04:29.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:29.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:04:29.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:04:29.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:29.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:04:29.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:04:29.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:04:29.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:04:29.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:04:29.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:04:29.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:29.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:04:29.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:04:29.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:04:29.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:04:29.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:29.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:04:29.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:04:29.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:29.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:04:29.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:04:29.246 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.246 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.246 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.247 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:04:29.247 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:29.247 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.022963622999995, 2023-07-02T14:04:29.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:04:29.247 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:29.247 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.248 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:29.248 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:29.248 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.248 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:04:29.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.248 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:29.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:29.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:29.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:29.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:29.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:29.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:29.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:04:29.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:04:29.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:29.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:04:29.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:04:29.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:29.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:04:29.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:04:29.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:04:29.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:04:29.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:04:29.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:04:29.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:29.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:04:29.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:04:29.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:04:29.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:04:29.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:29.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:04:29.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:04:29.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:29.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:04:29.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:04:29.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:29.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:04:29.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:04:29.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:29.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:04:29.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:04:29.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:29.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:04:29.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:04:29.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:29.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:04:29.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:04:29.258 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.258 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.258 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.259 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:29.259 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:29.259 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 77.792509292999995, 2023-07-02T14:04:29.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:04:29.259 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:29.259 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.259 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:29.259 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:29.260 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.260 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:04:29.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.260 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:29.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:29.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:29.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:29.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:29.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:29.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:29.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:04:29.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:04:29.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:29.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:04:29.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:04:29.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:29.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:04:29.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:04:29.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:04:29.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:04:29.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:04:29.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:04:29.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:29.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:04:29.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:04:29.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:29.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:04:29.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:04:29.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:29.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:04:29.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:04:29.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:29.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:04:29.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:04:29.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:29.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:04:29.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:04:29.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:29.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:04:29.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:04:29.268 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.268 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.269 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.269 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2941 2.3 2:d3762e63:::rbd_data.125de1449605.00000000000000f8:head [write 65536~2031616] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:29.269 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:29.269 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 77.663406339000005, 2023-07-02T14:04:29.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002, 2023-07-02T14:04:29.269 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:29.270 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:29.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:29.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.270 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2941 2023-07-02T14:04:29.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.270 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:29.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:29.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:29.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:29.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:29.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:29.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:29.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:29.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467291+0000", 2023-07-02T14:04:29.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043022859999999998 2023-07-02T14:04:29.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:29.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471593+0000", 2023-07-02T14:04:29.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0999999999999999e-05 2023-07-02T14:04:29.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:29.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471614+0000", 2023-07-02T14:04:29.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0097e-05 2023-07-02T14:04:29.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471625+0000", 2023-07-02T14:04:29.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069808499999999998 2023-07-02T14:04:29.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472323+0000", 2023-07-02T14:04:29.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135635 2023-07-02T14:04:29.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:29.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472458+0000", 2023-07-02T14:04:29.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223365 2023-07-02T14:04:29.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:29.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472682+0000", 2023-07-02T14:04:29.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013761944999999999 2023-07-02T14:04:29.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:29.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.486444+0000", 2023-07-02T14:04:29.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016859747000000001 2023-07-02T14:04:29.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:29.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503303+0000", 2023-07-02T14:04:29.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6635000000000001e-05 2023-07-02T14:04:29.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:29.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503380+0000", 2023-07-02T14:04:29.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000161803 2023-07-02T14:04:29.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:29.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503542+0000", 2023-07-02T14:04:29.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002 2023-07-02T14:04:29.279 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.279 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.279 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.279 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2950 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 1507328~786432] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:29.279 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:29.280 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 77.655233030999995, 2023-07-02T14:04:29.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999, 2023-07-02T14:04:29.280 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:29.280 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.280 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:29.280 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:29.280 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.281 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2950 2023-07-02T14:04:29.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.281 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:29.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:29.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:29.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:29.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:29.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:29.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:29.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:29.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:29.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001147854 2023-07-02T14:04:29.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:29.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476614+0000", 2023-07-02T14:04:29.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.802e-05 2023-07-02T14:04:29.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:29.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476632+0000", 2023-07-02T14:04:29.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1860000000000003e-06 2023-07-02T14:04:29.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476639+0000", 2023-07-02T14:04:29.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061211240000000004 2023-07-02T14:04:29.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482760+0000", 2023-07-02T14:04:29.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8833999999999996e-05 2023-07-02T14:04:29.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:29.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482859+0000", 2023-07-02T14:04:29.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019270600000000001 2023-07-02T14:04:29.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:29.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.483051+0000", 2023-07-02T14:04:29.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060385630000000003 2023-07-02T14:04:29.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:29.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489090+0000", 2023-07-02T14:04:29.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024506918999999999 2023-07-02T14:04:29.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:29.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513597+0000", 2023-07-02T14:04:29.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9672999999999995e-05 2023-07-02T14:04:29.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:29.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513667+0000", 2023-07-02T14:04:29.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024329500000000001 2023-07-02T14:04:29.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:29.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513910+0000", 2023-07-02T14:04:29.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999 2023-07-02T14:04:29.289 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.289 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.289 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.290 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2951 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2686976~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:29.290 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:29.290 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 77.654052253000003, 2023-07-02T14:04:29.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999, 2023-07-02T14:04:29.290 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:29.290 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.291 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:29.291 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:29.291 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.291 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2951 2023-07-02T14:04:29.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.291 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:29.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:29.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:29.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:29.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:29.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:29.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:29.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476646+0000", 2023-07-02T14:04:29.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7639000000000003e-05 2023-07-02T14:04:29.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:29.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476684+0000", 2023-07-02T14:04:29.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3508000000000002e-05 2023-07-02T14:04:29.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:29.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476747+0000", 2023-07-02T14:04:29.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8499999999999994e-06 2023-07-02T14:04:29.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476755+0000", 2023-07-02T14:04:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078346160000000008 2023-07-02T14:04:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484590+0000", 2023-07-02T14:04:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7206000000000004e-05 2023-07-02T14:04:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484687+0000", 2023-07-02T14:04:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016243099999999999 2023-07-02T14:04:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484849+0000", 2023-07-02T14:04:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042482860000000004 2023-07-02T14:04:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489097+0000", 2023-07-02T14:04:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.02489504 2023-07-02T14:04:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513992+0000", 2023-07-02T14:04:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9972000000000001e-05 2023-07-02T14:04:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514052+0000", 2023-07-02T14:04:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149264 2023-07-02T14:04:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514202+0000", 2023-07-02T14:04:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999 2023-07-02T14:04:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2952 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2883584~12288] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 77.653938417999996, 2023-07-02T14:04:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651, 2023-07-02T14:04:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2952 2023-07-02T14:04:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476760+0000", 2023-07-02T14:04:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3664e-05 2023-07-02T14:04:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476774+0000", 2023-07-02T14:04:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.166e-06 2023-07-02T14:04:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476783+0000", 2023-07-02T14:04:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8720000000000001e-06 2023-07-02T14:04:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476788+0000", 2023-07-02T14:04:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087180529999999999 2023-07-02T14:04:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485506+0000", 2023-07-02T14:04:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5680999999999994e-05 2023-07-02T14:04:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485602+0000", 2023-07-02T14:04:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000222365 2023-07-02T14:04:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485824+0000", 2023-07-02T14:04:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003277379 2023-07-02T14:04:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489101+0000", 2023-07-02T14:04:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025151739999999999 2023-07-02T14:04:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514253+0000", 2023-07-02T14:04:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4749000000000002e-05 2023-07-02T14:04:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514288+0000", 2023-07-02T14:04:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130634 2023-07-02T14:04:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514419+0000", 2023-07-02T14:04:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651 2023-07-02T14:04:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9620 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 589824~65536] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:04:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:04:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.1084946950000001, 2023-07-02T14:04:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001, 2023-07-02T14:04:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9620 2023-07-02T14:04:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:04:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:04:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022203+0000", 2023-07-02T14:04:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0346000000000001e-05 2023-07-02T14:04:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022263+0000", 2023-07-02T14:04:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4657999999999999e-05 2023-07-02T14:04:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022278+0000", 2023-07-02T14:04:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:04:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022285+0000", 2023-07-02T14:04:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0096000000000003e-05 2023-07-02T14:04:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022335+0000", 2023-07-02T14:04:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2334000000000001e-05 2023-07-02T14:04:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022347+0000", 2023-07-02T14:04:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095653701999999993 2023-07-02T14:04:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118001+0000", 2023-07-02T14:04:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000185797 2023-07-02T14:04:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118187+0000", 2023-07-02T14:04:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026397900000000001 2023-07-02T14:04:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118451+0000", 2023-07-02T14:04:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051832800000000002 2023-07-02T14:04:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.123634+0000", 2023-07-02T14:04:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001227332 2023-07-02T14:04:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124861+0000", 2023-07-02T14:04:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9493999999999998e-05 2023-07-02T14:04:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124901+0000", 2023-07-02T14:04:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0462000000000001e-05 2023-07-02T14:04:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124931+0000", 2023-07-02T14:04:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001 2023-07-02T14:04:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9621 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 655360~1966080] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:04:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:04:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.1083073130000001, 2023-07-02T14:04:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999, 2023-07-02T14:04:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9621 2023-07-02T14:04:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:04:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:04:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022391+0000", 2023-07-02T14:04:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032053939999999999 2023-07-02T14:04:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025597+0000", 2023-07-02T14:04:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4307999999999999e-05 2023-07-02T14:04:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025611+0000", 2023-07-02T14:04:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.032e-06 2023-07-02T14:04:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025617+0000", 2023-07-02T14:04:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8977999999999998e-05 2023-07-02T14:04:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025656+0000", 2023-07-02T14:04:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:04:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025663+0000", 2023-07-02T14:04:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095220739999999998 2023-07-02T14:04:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120884+0000", 2023-07-02T14:04:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2525000000000001e-05 2023-07-02T14:04:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120956+0000", 2023-07-02T14:04:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016232400000000001 2023-07-02T14:04:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.121119+0000", 2023-07-02T14:04:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044007380000000004 2023-07-02T14:04:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.125519+0000", 2023-07-02T14:04:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000797209 2023-07-02T14:04:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126317+0000", 2023-07-02T14:04:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1228000000000004e-05 2023-07-02T14:04:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126378+0000", 2023-07-02T14:04:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978000000000003e-05 2023-07-02T14:04:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126430+0000", 2023-07-02T14:04:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999 2023-07-02T14:04:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:11434 2.1 2:8269d003:::rbd_data.125de1449605.00000000000000ce:head [write 65536~4128768] snapc 1a=[1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e107)", 2023-07-02T14:04:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:04:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.21286904400000001, 2023-07-02T14:04:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036409966000000002, 2023-07-02T14:04:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11434 2023-07-02T14:04:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:29.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:29.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:04:29.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:29.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:29.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:04:29.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:29.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:29.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917828+0000", 2023-07-02T14:04:29.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052044650000000001 2023-07-02T14:04:29.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:29.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923033+0000", 2023-07-02T14:04:29.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0097000000000001e-05 2023-07-02T14:04:29.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:29.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923053+0000", 2023-07-02T14:04:29.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.49e-06 2023-07-02T14:04:29.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923061+0000", 2023-07-02T14:04:29.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0090999999999998e-05 2023-07-02T14:04:29.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:29.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923121+0000", 2023-07-02T14:04:29.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000114095 2023-07-02T14:04:29.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:29.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923236+0000", 2023-07-02T14:04:29.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020094199999999999 2023-07-02T14:04:29.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:29.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923436+0000", 2023-07-02T14:04:29.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088693520000000005 2023-07-02T14:04:29.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:29.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.932306+0000", 2023-07-02T14:04:29.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021874112000000001 2023-07-02T14:04:29.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:29.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954180+0000", 2023-07-02T14:04:29.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4433e-05 2023-07-02T14:04:29.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:29.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954214+0000", 2023-07-02T14:04:29.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5884000000000002e-05 2023-07-02T14:04:29.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:29.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:29.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:29.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954240+0000", 2023-07-02T14:04:29.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036409966000000002 2023-07-02T14:04:29.344 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.344 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.344 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.344 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:29.344 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:29.344 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:29.346 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:04:29.347 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap22 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap23 b3fbb028-e267-4d25-a750-93ba18a2316a@snap23.diff 2023-07-02T14:04:29.347 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:29.234+0000 7f170ca97700 0 -- 172.21.15.163:0/2856555239 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x55dd4baa8ed0 msgr2=0x55dd4baab2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:29.347 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:29.248+0000 7f170ca97700 0 -- 172.21.15.163:0/2856555239 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f16f0036bb0 msgr2=0x7f16f0038fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:29.348 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 100% complete...done. 2023-07-02T14:04:29.348 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:04:29.348 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=24 2023-07-02T14:04:29.348 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap24 2023-07-02T14:04:29.348 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:04:29.349 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:29.346+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:29.446 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:29.446+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:29.469 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 2] out_osds: [5] dead_osds: [3] live_osds: [0, 1, 2, 4, 5] 2023-07-02T14:04:29.470 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:04:29.470 INFO:tasks.thrashosds.thrasher:primary_affinity 2023-07-02T14:04:29.470 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.952225 2023-07-02T14:04:29.470 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 0.9522252423221346 2023-07-02T14:04:29.473 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:29.473 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:29.473 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:04:29.474 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:29.483 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:04:29.547 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:29.547+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:29.607 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:29.607 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:29.607 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:04:29.607 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:04:29.607 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:29.617 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:04:29.648 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:29.648+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:29.742 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:29.742 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:04:29.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:04:29.743 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:29.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.743 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:04:29.743 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:29.743 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.843443344000001, 2023-07-02T14:04:29.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:04:29.744 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:29.744 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.744 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:29.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:29.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:29.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:29.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:29.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:29.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:29.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:04:29.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:04:29.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:29.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:04:29.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:04:29.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:29.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:04:29.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:04:29.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:04:29.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:04:29.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:29.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:04:29.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:04:29.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:29.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:04:29.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:04:29.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:29.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:04:29.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:04:29.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:29.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:04:29.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:04:29.751 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.751 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.752 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.752 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:04:29.752 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:29.752 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.839357626000002, 2023-07-02T14:04:29.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:04:29.752 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:29.753 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.753 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:29.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:29.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:29.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:29.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:29.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:29.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:29.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:04:29.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:04:29.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:29.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:04:29.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:04:29.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:29.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:04:29.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:04:29.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:04:29.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:04:29.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:29.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:04:29.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:04:29.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:29.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:04:29.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:04:29.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:29.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:04:29.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:04:29.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:29.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:04:29.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:04:29.759 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.759 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.760 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.760 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:04:29.760 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:29.760 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.835492844999997, 2023-07-02T14:04:29.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:04:29.761 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:29.761 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.761 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:29.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:29.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:29.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:29.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:29.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:29.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:29.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:29.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:04:29.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:04:29.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:29.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:04:29.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:04:29.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:29.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:04:29.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:04:29.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:04:29.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:04:29.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:29.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:04:29.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:04:29.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:29.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:04:29.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:04:29.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:29.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:04:29.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:04:29.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:29.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:04:29.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:04:29.767 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.768 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.768 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.768 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:04:29.768 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:29.768 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.831968584000002, 2023-07-02T14:04:29.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:04:29.769 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:29.769 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.769 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:29.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:29.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:29.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:29.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:29.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:29.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:29.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:04:29.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:04:29.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:29.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:04:29.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:04:29.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:29.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:04:29.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:04:29.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:04:29.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:04:29.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:29.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:04:29.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:04:29.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:29.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:04:29.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:04:29.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:29.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:04:29.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:04:29.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:29.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:04:29.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:04:29.775 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.775 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.776 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.776 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6124 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1328, mlcod=81'1319)", 2023-07-02T14:04:29.776 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:29.776 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.828453201000002, 2023-07-02T14:04:29.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019096974999999999, 2023-07-02T14:04:29.776 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:29.777 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.777 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:29.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:29.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:29.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:29.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:29.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:29.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:29.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:29.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912602+0000", 2023-07-02T14:04:29.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6741999999999999e-05 2023-07-02T14:04:29.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:29.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912629+0000", 2023-07-02T14:04:29.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5163e-05 2023-07-02T14:04:29.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:29.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912644+0000", 2023-07-02T14:04:29.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4800000000000001e-06 2023-07-02T14:04:29.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912653+0000", 2023-07-02T14:04:29.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071988909999999998 2023-07-02T14:04:29.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:29.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.919851+0000", 2023-07-02T14:04:29.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2949999999999999e-05 2023-07-02T14:04:29.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:29.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.919874+0000", 2023-07-02T14:04:29.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011800804 2023-07-02T14:04:29.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:29.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931675+0000", 2023-07-02T14:04:29.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6033999999999998e-05 2023-07-02T14:04:29.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:29.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931701+0000", 2023-07-02T14:04:29.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019096974999999999 2023-07-02T14:04:29.784 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.784 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.784 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.784 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6139 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1337, mlcod=81'1333)", 2023-07-02T14:04:29.784 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:29.784 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.801763340999997, 2023-07-02T14:04:29.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018571681, 2023-07-02T14:04:29.785 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:29.785 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.785 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:29.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:29.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:29.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:29.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:29.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:29.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:29.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939293+0000", 2023-07-02T14:04:29.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003764613 2023-07-02T14:04:29.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:29.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943058+0000", 2023-07-02T14:04:29.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5797000000000001e-05 2023-07-02T14:04:29.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:29.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943074+0000", 2023-07-02T14:04:29.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5089999999999996e-06 2023-07-02T14:04:29.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943081+0000", 2023-07-02T14:04:29.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014727100000000001 2023-07-02T14:04:29.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:29.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943228+0000", 2023-07-02T14:04:29.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2246000000000003e-05 2023-07-02T14:04:29.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:29.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943261+0000", 2023-07-02T14:04:29.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014576317 2023-07-02T14:04:29.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:29.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957837+0000", 2023-07-02T14:04:29.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8962000000000002e-05 2023-07-02T14:04:29.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:29.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957866+0000", 2023-07-02T14:04:29.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018571681 2023-07-02T14:04:29.792 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.792 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.792 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.792 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:04:29.793 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:29.793 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.796263785000001, 2023-07-02T14:04:29.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:04:29.793 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:29.793 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.793 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:29.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:29.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:29.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:29.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:29.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:29.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:29.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:29.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:04:29.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:04:29.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:29.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:04:29.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:04:29.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:29.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:04:29.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:04:29.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:04:29.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:04:29.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:29.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:04:29.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:04:29.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:29.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:04:29.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:04:29.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:29.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:04:29.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:04:29.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:29.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:04:29.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:04:29.800 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.800 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.800 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.801 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:04:29.801 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:29.801 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.791606426999998, 2023-07-02T14:04:29.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:04:29.801 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:29.801 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.801 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:29.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:29.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:29.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:29.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:29.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:29.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:29.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:04:29.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:04:29.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:29.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:04:29.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:04:29.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:29.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:04:29.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:04:29.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:04:29.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:04:29.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:29.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:04:29.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:04:29.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:29.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:04:29.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:04:29.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:29.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:04:29.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:04:29.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:29.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:04:29.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:04:29.810 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.810 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.811 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.811 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:04:29.812 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:29.812 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.790666563999999, 2023-07-02T14:04:29.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:04:29.812 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:29.813 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.813 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:29.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:29.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:29.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:29.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:29.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:29.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:29.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:04:29.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:04:29.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:29.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:04:29.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:04:29.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:29.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:04:29.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:04:29.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:04:29.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:04:29.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:29.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:04:29.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:04:29.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:29.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:04:29.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:04:29.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:29.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:04:29.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:04:29.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:29.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:04:29.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:04:29.823 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.823 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.823 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.823 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:04:29.823 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:29.824 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.789268686, 2023-07-02T14:04:29.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:04:29.824 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:29.824 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.824 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:29.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:29.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:29.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:29.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:29.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:29.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:29.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:29.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:04:29.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:04:29.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:29.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:04:29.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:04:29.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:29.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:04:29.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:04:29.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:04:29.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:04:29.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:29.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:04:29.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:04:29.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:29.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:04:29.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:04:29.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:29.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:04:29.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:04:29.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:29.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:04:29.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:04:29.831 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.831 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.831 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.831 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6150 2.0 e81/46 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 81'1345, mlcod=81'1338)", 2023-07-02T14:04:29.831 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:29.831 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 49.781245738999999, 2023-07-02T14:04:29.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019172812000000001, 2023-07-02T14:04:29.832 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:29.832 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.832 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:29.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:29.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:29.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:29.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:29.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:29.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:29.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:29.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959810+0000", 2023-07-02T14:04:29.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048438650000000001 2023-07-02T14:04:29.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:29.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964653+0000", 2023-07-02T14:04:29.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2136e-05 2023-07-02T14:04:29.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:29.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964666+0000", 2023-07-02T14:04:29.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2169999999999999e-06 2023-07-02T14:04:29.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964672+0000", 2023-07-02T14:04:29.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00047561800000000003 2023-07-02T14:04:29.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:29.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965147+0000", 2023-07-02T14:04:29.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5891e-05 2023-07-02T14:04:29.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:29.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965173+0000", 2023-07-02T14:04:29.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013770973000000001 2023-07-02T14:04:29.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:29.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978944+0000", 2023-07-02T14:04:29.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0256999999999998e-05 2023-07-02T14:04:29.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:29.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978985+0000", 2023-07-02T14:04:29.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019172812000000001 2023-07-02T14:04:29.838 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.839 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.839 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.839 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:04:29.839 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:29.839 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 28.379078190000001, 2023-07-02T14:04:29.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:04:29.840 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:29.840 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.840 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:29.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:29.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:29.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:29.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:29.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:29.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:29.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:04:29.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:04:29.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:29.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:04:29.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:04:29.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:29.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:04:29.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:04:29.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:04:29.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:04:29.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:29.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:04:29.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:04:29.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:29.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:04:29.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:04:29.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:29.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:04:29.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:04:29.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:29.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:04:29.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:04:29.846 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.846 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.847 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.847 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:29.847 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:29.847 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 27.012728505999998, 2023-07-02T14:04:29.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:04:29.848 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:29.848 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.848 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:29.848 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:29.848 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.848 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:04:29.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.848 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:29.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:29.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:29.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:29.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:29.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:29.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:29.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:04:29.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:04:29.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:29.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:04:29.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:04:29.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:29.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:04:29.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:04:29.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:04:29.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:04:29.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:29.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:04:29.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:04:29.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:29.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:04:29.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:04:29.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:29.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:04:29.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:04:29.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:29.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:04:29.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:04:29.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:29.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:04:29.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:04:29.856 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.856 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.856 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.856 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7706 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 0~53248] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:29.856 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:29.856 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 27.008801759000001, 2023-07-02T14:04:29.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019507096000000002, 2023-07-02T14:04:29.857 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:29.857 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.857 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:29.857 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:29.857 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.857 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7706 2023-07-02T14:04:29.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.858 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:29.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:29.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:29.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:29.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:29.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:29.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:29.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:29.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732254+0000", 2023-07-02T14:04:29.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4764999999999997e-05 2023-07-02T14:04:29.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:29.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732289+0000", 2023-07-02T14:04:29.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0166999999999999e-05 2023-07-02T14:04:29.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:29.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732299+0000", 2023-07-02T14:04:29.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5310000000000001e-06 2023-07-02T14:04:29.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732305+0000", 2023-07-02T14:04:29.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060619599999999999 2023-07-02T14:04:29.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:29.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738367+0000", 2023-07-02T14:04:29.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011864 2023-07-02T14:04:29.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:29.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738486+0000", 2023-07-02T14:04:29.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013098291 2023-07-02T14:04:29.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:29.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751584+0000", 2023-07-02T14:04:29.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7800000000000002e-05 2023-07-02T14:04:29.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:29.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751642+0000", 2023-07-02T14:04:29.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000121198 2023-07-02T14:04:29.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:29.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751763+0000", 2023-07-02T14:04:29.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019507096000000002 2023-07-02T14:04:29.865 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.865 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.865 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.865 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7707 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 3145728~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:29.865 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:29.865 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 27.008745039000001, 2023-07-02T14:04:29.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999, 2023-07-02T14:04:29.866 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:29.866 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.866 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:29.866 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:29.866 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.866 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7707 2023-07-02T14:04:29.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.867 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:29.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:29.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:29.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:29.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:29.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:29.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:29.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:29.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5017e-05 2023-07-02T14:04:29.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:29.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732347+0000", 2023-07-02T14:04:29.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5821e-05 2023-07-02T14:04:29.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:29.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732363+0000", 2023-07-02T14:04:29.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1270000000000002e-06 2023-07-02T14:04:29.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732368+0000", 2023-07-02T14:04:29.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008444666 2023-07-02T14:04:29.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:29.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740812+0000", 2023-07-02T14:04:29.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011266599999999999 2023-07-02T14:04:29.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:29.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740925+0000", 2023-07-02T14:04:29.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010848808999999999 2023-07-02T14:04:29.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:29.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751774+0000", 2023-07-02T14:04:29.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8421999999999997e-05 2023-07-02T14:04:29.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:29.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751832+0000", 2023-07-02T14:04:29.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107453 2023-07-02T14:04:29.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:29.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751940+0000", 2023-07-02T14:04:29.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999 2023-07-02T14:04:29.874 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.874 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.874 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.874 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:29.874 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:29.874 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 27.008681161999998, 2023-07-02T14:04:29.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:04:29.875 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:29.875 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.875 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:29.875 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:29.875 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.875 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:04:29.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.876 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:29.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:29.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:29.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:29.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:29.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:29.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:29.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:29.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:29.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:04:29.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:29.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:04:29.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:04:29.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:29.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:04:29.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:04:29.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:04:29.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:04:29.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:29.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:04:29.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:04:29.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:29.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:04:29.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:04:29.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:29.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:04:29.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:04:29.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:29.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:04:29.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:04:29.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:29.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:04:29.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:04:29.883 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.883 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.883 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.883 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:29.883 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:29.883 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 27.004117337, 2023-07-02T14:04:29.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:04:29.884 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:29.884 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.884 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:29.884 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:29.884 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.884 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:04:29.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.885 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:29.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:29.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:29.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:29.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:29.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:29.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:29.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:29.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:04:29.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:04:29.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:29.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:04:29.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:04:29.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:29.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:04:29.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:04:29.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:04:29.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:04:29.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:29.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:04:29.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:04:29.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:29.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:04:29.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:04:29.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:29.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:04:29.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:04:29.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:29.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:04:29.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:04:29.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:29.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:04:29.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:04:29.892 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.892 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.892 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.892 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:29.892 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:29.893 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 27.004067470999999, 2023-07-02T14:04:29.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:04:29.893 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:29.893 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.893 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:29.893 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:29.893 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.894 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:04:29.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.894 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:29.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:29.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:29.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:29.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:29.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:29.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:29.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:29.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:04:29.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:04:29.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:29.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:04:29.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:04:29.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:29.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:04:29.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:04:29.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:04:29.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:04:29.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:29.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:04:29.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:04:29.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:29.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:04:29.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:04:29.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:29.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:04:29.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:04:29.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:29.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:04:29.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:04:29.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:29.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:04:29.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:04:29.901 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.901 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.901 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.901 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:29.902 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:29.902 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 27.003028478000001, 2023-07-02T14:04:29.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:04:29.902 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:29.902 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.902 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:29.902 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:29.902 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.903 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:04:29.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.903 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:29.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:29.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:29.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:29.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:29.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:29.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:29.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:04:29.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:04:29.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:29.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:04:29.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:04:29.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:29.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:04:29.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:04:29.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:04:29.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:04:29.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:29.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:04:29.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:04:29.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:29.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:04:29.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:04:29.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:29.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:04:29.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:04:29.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:29.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:04:29.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:04:29.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:29.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:04:29.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:04:29.910 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.910 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.910 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.910 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7993 2.7 2:fe72db53:::rbd_data.125de1449605.0000000000000072:head [write 65536~3276800] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:29.911 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:29.911 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 26.81607043, 2023-07-02T14:04:29.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919, 2023-07-02T14:04:29.911 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:29.911 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:29.911 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:29.911 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:29.911 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:29.912 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7993 2023-07-02T14:04:29.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.912 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:29.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:29.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:29.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:29.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:29.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:29.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:29.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:29.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924986+0000", 2023-07-02T14:04:29.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049542070000000004 2023-07-02T14:04:29.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:29.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929940+0000", 2023-07-02T14:04:29.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9740000000000001e-05 2023-07-02T14:04:29.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:29.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929960+0000", 2023-07-02T14:04:29.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.311e-05 2023-07-02T14:04:29.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:29.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929973+0000", 2023-07-02T14:04:29.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9629000000000002e-05 2023-07-02T14:04:29.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:29.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930023+0000", 2023-07-02T14:04:29.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012751799999999999 2023-07-02T14:04:29.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:29.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930150+0000", 2023-07-02T14:04:29.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016449042000000001 2023-07-02T14:04:29.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:29.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946599+0000", 2023-07-02T14:04:29.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3241e-05 2023-07-02T14:04:29.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:29.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946663+0000", 2023-07-02T14:04:29.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000102296 2023-07-02T14:04:29.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:29.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:29.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:29.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946765+0000", 2023-07-02T14:04:29.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919 2023-07-02T14:04:29.919 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.919 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.919 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.919 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:29.920 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:29.920 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:29.921 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:04:29.922 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:29.921+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:30.022 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:30.022+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:30.085 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:30.086 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:30.086 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:04:30.086 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:30.096 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:04:30.139 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:30.139+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:30.219 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:30.219 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:30.219 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:04:30.219 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:04:30.220 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:30.223 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:30.223+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:30.229 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:30.323 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:30.323+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:30.329 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.2 primary-affinity to 0.952225 (8624052) 2023-07-02T14:04:30.355 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:04:30.365 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 2 2023-07-02T14:04:30.401 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:30.401 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:04:30.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:04:30.402 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:30.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.402 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:30.402 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:30.402 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 111.109048152, 2023-07-02T14:04:30.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:04:30.402 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:30.403 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:30.403 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:30.403 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:30.403 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:30.403 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:04:30.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.403 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:30.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.404 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:30.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:30.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:30.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.404 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:30.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:30.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:30.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:30.405 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:04:30.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:04:30.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:30.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:04:30.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:04:30.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:30.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:04:30.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:04:30.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:30.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:04:30.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:04:30.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:30.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:04:30.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:04:30.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:30.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:04:30.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:04:30.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:30.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:04:30.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:04:30.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:30.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:04:30.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:04:30.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:30.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:04:30.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:04:30.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:30.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:04:30.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:04:30.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:30.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:04:30.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:04:30.412 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.412 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.412 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.412 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:30.413 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:30.413 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 111.108813129, 2023-07-02T14:04:30.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:04:30.413 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:30.413 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:30.413 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:30.413 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:30.413 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:30.414 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:04:30.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.414 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:30.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:30.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:30.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:30.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:30.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:30.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:30.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:30.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:04:30.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:04:30.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:30.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:04:30.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:04:30.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:30.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:04:30.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:04:30.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:30.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:04:30.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:04:30.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:30.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:04:30.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:04:30.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:30.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:04:30.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:04:30.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:30.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:04:30.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:04:30.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:30.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:04:30.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:04:30.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:30.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:04:30.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:04:30.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:30.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:04:30.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:04:30.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:30.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:04:30.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:04:30.422 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.422 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.422 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.423 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:30.423 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:30.423 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 111.108613995, 2023-07-02T14:04:30.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:04:30.423 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:30.423 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:30.423 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:30.424 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:30.424 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:30.424 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:04:30.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.424 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:30.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:30.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:30.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:30.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:30.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:30.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:30.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:30.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:04:30.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:04:30.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:30.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:04:30.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:04:30.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:30.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:04:30.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:04:30.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:30.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:04:30.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:04:30.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:30.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:04:30.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:04:30.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:30.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:04:30.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:04:30.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:30.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:04:30.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:04:30.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:30.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:04:30.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:04:30.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:30.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:04:30.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:04:30.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:30.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:04:30.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:04:30.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:30.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:04:30.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:04:30.432 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.433 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.433 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.433 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:30.433 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:30.433 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 111.108248871, 2023-07-02T14:04:30.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:04:30.434 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:30.434 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:30.434 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:30.434 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:30.434 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:30.434 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:04:30.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.434 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:30.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:30.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:30.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:30.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:30.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:30.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:30.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:30.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:04:30.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:04:30.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:30.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:04:30.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:04:30.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:30.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:04:30.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:04:30.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:30.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:04:30.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:04:30.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:30.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:04:30.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:04:30.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:30.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:04:30.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:04:30.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:30.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:04:30.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:04:30.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:30.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:04:30.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:04:30.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:30.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:04:30.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:04:30.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:30.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:04:30.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:04:30.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:30.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:04:30.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:04:30.443 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.443 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.443 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.443 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:30.444 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:30.444 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 111.10816781, 2023-07-02T14:04:30.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:04:30.444 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:30.444 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:30.444 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:30.444 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:30.444 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:30.445 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:04:30.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.445 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:30.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:30.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:30.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:30.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:30.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:30.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:30.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:30.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:04:30.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:04:30.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:30.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:04:30.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:04:30.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:30.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:04:30.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:04:30.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:30.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:04:30.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:04:30.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:30.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:04:30.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:04:30.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:30.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:04:30.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:04:30.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:30.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:04:30.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:04:30.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:30.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:04:30.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:04:30.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:30.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:04:30.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:04:30.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:30.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:04:30.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:04:30.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:30.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:04:30.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:04:30.457 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.457 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.457 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.457 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:30.458 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:30.458 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 111.108076139, 2023-07-02T14:04:30.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:04:30.458 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:30.458 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:30.458 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:30.458 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:30.458 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:30.459 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:04:30.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.459 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:30.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:30.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:30.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:30.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:30.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:30.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:30.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:30.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:04:30.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:04:30.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:30.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:04:30.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:04:30.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:30.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:04:30.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:04:30.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:30.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:04:30.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:04:30.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:30.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:04:30.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:04:30.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:30.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:04:30.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:04:30.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:30.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:04:30.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:04:30.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:30.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:04:30.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:04:30.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:30.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:04:30.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:04:30.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:30.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:04:30.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:04:30.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:30.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:04:30.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:04:30.467 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.467 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.467 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.468 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:30.468 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:30.468 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 111.10773669700001, 2023-07-02T14:04:30.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:04:30.468 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:30.469 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:30.469 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:30.469 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:30.469 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:30.469 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:04:30.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.469 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:30.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:30.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:30.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:30.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:30.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:30.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:30.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:30.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:04:30.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:04:30.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:30.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:04:30.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:04:30.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:30.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:04:30.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:04:30.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:30.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:04:30.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:04:30.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:30.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:04:30.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:04:30.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:30.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:04:30.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:04:30.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:30.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:04:30.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:04:30.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:30.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:04:30.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:04:30.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:30.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:04:30.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:04:30.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:30.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:04:30.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:04:30.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:30.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:04:30.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:04:30.478 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.478 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.478 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.478 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:30.478 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:30.479 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 111.107550644, 2023-07-02T14:04:30.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:04:30.479 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:30.479 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:30.479 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:30.479 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:30.479 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:30.480 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:04:30.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.480 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:30.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:30.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:30.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:30.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:30.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:30.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:30.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:30.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:30.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:04:30.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:30.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:04:30.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:04:30.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:30.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:04:30.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:04:30.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:30.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:04:30.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:04:30.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:30.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:04:30.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:04:30.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:30.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:04:30.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:04:30.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:30.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:04:30.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:04:30.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:30.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:04:30.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:04:30.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:30.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:04:30.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:04:30.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:30.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:04:30.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:04:30.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:30.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:04:30.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:04:30.488 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.488 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.488 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.489 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:30.489 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:30.489 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 111.107392842, 2023-07-02T14:04:30.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:04:30.489 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:30.489 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:30.489 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:30.490 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:30.490 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:30.490 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:04:30.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.490 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:30.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:30.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:30.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:30.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:30.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:30.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:30.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:30.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:04:30.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:04:30.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:30.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:04:30.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:04:30.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:30.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:04:30.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:04:30.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:30.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:04:30.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:04:30.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:30.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:04:30.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:04:30.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:30.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:04:30.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:04:30.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:30.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:04:30.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:04:30.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:30.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:04:30.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:04:30.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:30.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:04:30.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:04:30.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:30.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:04:30.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:04:30.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:30.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:04:30.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:04:30.498 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.498 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.499 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.499 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:30.499 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:30.499 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 111.10720105999999, 2023-07-02T14:04:30.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:04:30.499 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:30.500 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:30.500 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:30.500 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:30.500 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:30.500 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:04:30.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.500 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:30.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:30.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:30.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:30.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:30.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:30.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:30.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:30.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:04:30.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:04:30.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:30.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:04:30.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:04:30.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:30.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:04:30.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:04:30.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:30.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:04:30.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:04:30.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:30.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:04:30.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:04:30.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:30.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:04:30.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:04:30.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:30.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:04:30.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:04:30.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:30.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:04:30.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:04:30.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:30.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:04:30.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:04:30.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:30.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:04:30.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:04:30.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:30.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:04:30.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:04:30.509 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.509 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.509 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.510 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:30.510 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:30.510 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 111.1070254, 2023-07-02T14:04:30.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:04:30.510 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:30.510 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:30.510 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:30.510 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:30.511 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:30.511 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:04:30.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.511 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:30.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:30.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:30.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:30.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:30.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:30.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:30.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:30.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:30.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:04:30.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:30.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:04:30.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:04:30.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:30.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:04:30.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:04:30.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:30.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:04:30.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:04:30.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:30.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:04:30.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:04:30.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:30.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:04:30.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:04:30.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:30.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:04:30.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:04:30.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:30.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:04:30.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:04:30.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:30.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:04:30.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:04:30.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:30.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:04:30.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:04:30.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:30.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:04:30.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:04:30.519 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.519 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.520 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.520 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:30.520 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:30.520 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 111.10693121200001, 2023-07-02T14:04:30.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:04:30.520 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:30.521 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:30.521 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:30.521 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:30.521 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:30.521 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:04:30.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.521 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:30.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:30.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:30.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:30.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:30.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:30.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:30.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:30.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:04:30.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:04:30.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:30.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:04:30.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:04:30.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:30.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:04:30.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:04:30.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:30.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:04:30.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:04:30.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:30.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:04:30.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:04:30.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:30.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:04:30.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:04:30.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:30.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:04:30.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:04:30.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:30.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:04:30.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:04:30.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:30.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:04:30.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:04:30.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:30.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:04:30.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:04:30.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:30.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:04:30.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:04:30.530 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.530 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.530 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.530 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:30.531 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:30.531 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 111.106727826, 2023-07-02T14:04:30.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:04:30.531 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:30.531 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:30.531 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:30.531 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:30.531 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:30.532 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:04:30.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.532 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:30.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:30.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:30.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:30.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:30.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:30.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:30.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:30.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:04:30.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:04:30.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:30.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:04:30.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:04:30.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:30.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:04:30.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:04:30.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:30.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:04:30.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:04:30.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:30.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:04:30.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:04:30.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:30.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:04:30.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:04:30.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:30.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:04:30.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:04:30.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:30.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:04:30.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:04:30.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:30.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:04:30.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:04:30.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:30.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:04:30.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:04:30.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:30.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:04:30.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:04:30.540 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.540 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.540 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.541 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:30.541 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:30.541 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 111.10657716999999, 2023-07-02T14:04:30.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:04:30.541 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:30.541 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:30.542 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:30.542 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:30.542 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:30.542 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:04:30.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.542 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:30.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:30.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:30.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:30.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:30.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:30.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:30.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:30.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:04:30.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:04:30.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:30.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:04:30.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:04:30.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:30.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:04:30.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:04:30.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:30.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:04:30.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:04:30.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:30.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:04:30.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:04:30.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:30.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:04:30.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:04:30.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:30.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:04:30.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:04:30.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:30.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:04:30.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:04:30.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:30.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:04:30.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:04:30.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:30.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:04:30.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:04:30.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:30.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:04:30.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:04:30.551 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.551 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.551 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.551 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:30.551 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:30.551 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 111.10650446699999, 2023-07-02T14:04:30.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:04:30.552 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:30.552 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:30.552 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:30.552 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:30.552 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:30.552 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:04:30.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.553 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:30.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:30.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:30.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:30.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:30.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:30.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:30.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:30.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:30.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:04:30.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:30.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:04:30.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:04:30.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:30.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:04:30.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:04:30.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:30.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:04:30.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:04:30.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:30.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:04:30.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:04:30.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:30.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:04:30.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:04:30.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:30.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:04:30.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:04:30.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:30.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:04:30.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:04:30.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:30.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:04:30.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:04:30.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:30.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:04:30.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:04:30.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:30.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:04:30.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:04:30.561 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.561 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.561 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.562 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:30.562 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:30.562 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 111.106387477, 2023-07-02T14:04:30.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:04:30.562 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:30.562 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:30.562 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:30.562 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:30.563 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:30.563 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:04:30.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.563 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:30.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:30.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:30.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:30.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:30.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:30.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:30.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:30.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:30.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:04:30.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:30.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:04:30.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:04:30.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:30.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:04:30.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:04:30.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:30.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:04:30.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:04:30.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:30.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:04:30.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:04:30.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:30.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:04:30.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:04:30.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:30.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:04:30.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:04:30.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:30.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:04:30.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:04:30.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:30.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:04:30.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:04:30.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:30.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:04:30.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:04:30.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:30.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:04:30.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:04:30.571 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.571 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.572 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.572 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:30.572 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:30.572 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 111.106238026, 2023-07-02T14:04:30.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:04:30.573 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:30.573 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:30.573 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:30.573 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:30.573 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:30.573 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:04:30.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.573 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:30.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:30.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:30.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:30.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:30.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:30.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:30.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:30.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:04:30.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:04:30.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:30.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:04:30.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:04:30.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:30.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:04:30.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:04:30.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:30.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:04:30.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:04:30.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:30.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:04:30.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:04:30.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:30.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:04:30.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:04:30.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:30.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:04:30.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:04:30.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:30.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:04:30.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:04:30.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:30.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:04:30.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:04:30.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:30.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:04:30.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:04:30.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:30.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:04:30.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:04:30.582 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.582 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.582 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.582 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:30.582 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:30.583 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 111.08476205300001, 2023-07-02T14:04:30.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:04:30.583 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:30.583 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:30.583 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:30.583 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:30.583 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:30.583 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:04:30.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.584 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:30.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:30.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:30.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:30.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:30.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:30.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:30.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:30.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:04:30.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:04:30.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:30.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:04:30.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:04:30.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:30.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:04:30.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:04:30.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:30.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:04:30.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:04:30.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:30.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:04:30.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:04:30.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:30.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:04:30.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:04:30.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:30.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:04:30.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:04:30.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:30.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:04:30.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:04:30.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:30.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:04:30.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:04:30.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:30.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:04:30.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:04:30.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:30.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:04:30.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:04:30.592 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.592 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.592 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.593 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:30.593 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:30.593 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 111.084629517, 2023-07-02T14:04:30.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:04:30.593 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:30.593 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:30.593 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:30.594 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:30.594 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:30.594 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:04:30.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.594 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:30.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:30.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:30.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:30.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:30.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:30.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:30.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:30.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:04:30.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:04:30.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:30.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:04:30.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:04:30.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:30.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:04:30.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:04:30.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:30.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:04:30.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:04:30.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:30.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:04:30.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:04:30.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:30.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:04:30.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:04:30.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:30.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:04:30.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:04:30.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:30.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:04:30.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:04:30.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:30.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:04:30.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:04:30.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:30.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:04:30.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:04:30.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:30.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:04:30.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:04:30.602 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.603 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.603 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.603 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:30.603 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:30.603 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 111.023292856, 2023-07-02T14:04:30.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:04:30.604 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:30.604 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:30.604 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:30.604 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:30.604 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:30.604 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:04:30.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.605 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:30.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:30.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:30.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:30.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:30.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:30.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:30.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:30.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:04:30.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:04:30.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:30.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:04:30.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:04:30.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:30.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:04:30.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:04:30.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:30.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:04:30.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:04:30.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:30.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:04:30.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:04:30.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:30.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:04:30.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:04:30.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:30.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:04:30.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:04:30.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:30.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:04:30.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:04:30.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:30.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:04:30.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:04:30.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:30.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:04:30.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:04:30.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:30.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:30.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:30.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:04:30.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:04:30.613 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.613 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.613 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.613 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:30.613 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:30.613 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:30.615 INFO:tasks.workunit.client.0.smithi163.stderr:+ sync b3fbb028-e267-4d25-a750-93ba18a2316a-mnt 2023-07-02T14:04:30.615 INFO:tasks.workunit.client.0.smithi163.stderr:+ rm -rf b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4 b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4.tar.gz 2023-07-02T14:04:30.616 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:30.616 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:30.615+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:30.624 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:04:30.716 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:30.716+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:30.744 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:30.745 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:30.745 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:30.745 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:30.755 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:04:30.800 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:30.816 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:30.816+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:30.818 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:30.835 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:30.854 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:30.872 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:30.872 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:04:30.872 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:04:30.873 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:04:30.873 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:30.873 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:04:30.873 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:04:30.875 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:30.876 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:30.884 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:30.884 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:30.895 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:04:30.936 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:30.936+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:31.018 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:31.018+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:31.058 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:31.059 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:31.069 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:04:31.119 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:31.119+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:31.193 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:31.193 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:31.203 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:04:31.247 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:31.247+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:31.320 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:31.320+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:31.350 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:04:31.355 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:31.355 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:31.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:04:31.395 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i++ )) 2023-07-02T14:04:31.396 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i < 10 )) 2023-07-02T14:04:31.396 INFO:tasks.workunit.client.0.smithi163.stderr:+ pv -L 10M linux-5.4.tar.gz 2023-07-02T14:04:31.421 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:31.421+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:31.522 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:31.522+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:31.532 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:31.533 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:31.543 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:04:31.626 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:31.626+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:31.709 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:31.709 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:31.719 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:04:31.760 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:31.760+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:31.823 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:31.823+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:31.883 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:31.883 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:31.893 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:04:31.934 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:31.934+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:32.025 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:32.025+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:32.056 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:32.057 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:32.067 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:04:32.126 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:32.126+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:32.205 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:32.205 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:32.215 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:04:32.227 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:32.227+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:32.327 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:32.328+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:32.333 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:04:32.341 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:32.341 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:32.352 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:04:32.366 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap23 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap24 b3fbb028-e267-4d25-a750-93ba18a2316a@snap24.diff 2023-07-02T14:04:32.433 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:32.432+0000 7f24cee00700 0 -- 172.21.15.163:0/2482313815 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x55cd12341c50 msgr2=0x55cd12344080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:32.434 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:32.435+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:32.436 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:32.436+0000 7f24cf601700 0 -- 172.21.15.163:0/2482313815 >> [v2:172.21.15.22:6800/83351,v1:172.21.15.22:6801/83351] conn(0x7f24b4008780 msgr2=0x7f24b400abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:32.521 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:32.521 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:32.529 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:32.529+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:32.531 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:04:32.630 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:32.629+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:32.705 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:32.706 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:32.716 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:04:32.731 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:32.731+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:32.831 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:32.831+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:32.846 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:32.846 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:32.857 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:32.877 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:04:32.932 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:32.932+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:32.984 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:32.985 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:32.995 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:33.037 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:33.037+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:33.041 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:33.059 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:33.079 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:33.099 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:33.118 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:33.118 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:04:33.118 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:04:33.118 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:04:33.119 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:33.119 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:04:33.119 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:04:33.121 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:33.121 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:33.133 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:33.134 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:33.134 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:33.134 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:33.134 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:33.134+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:33.144 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:04:33.229 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete...2023-07-02T14:04:33.228+0000 7f24cfe02700 0 -- 172.21.15.163:0/2482313815 >> [v2:172.21.15.16:6800/83359,v1:172.21.15.16:6801/83359] conn(0x7f24a801e570 msgr2=0x7f24a8020960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:33.235 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:33.234+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:33.236 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:33.235+0000 7f24cee00700 0 -- 172.21.15.163:0/2482313815 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x55cd12346b80 msgr2=0x55cd12348fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:33.238 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:33.237+0000 7f24cee00700 0 -- 172.21.15.163:0/2482313815 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7f24a801e570 msgr2=0x7f24b003c190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-07-02T14:04:33.306 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:33.305+0000 7f24cf601700 0 -- 172.21.15.163:0/2482313815 >> [v2:172.21.15.22:6800/83351,v1:172.21.15.22:6801/83351] conn(0x55cd12341c50 msgr2=0x7f24b003b7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:33.335 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:33.335+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:33.362 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:33.363 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:33.363 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:33.363 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:33.363 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:33.377 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:04:33.461 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:33.461+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:33.537 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:33.537+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:33.549 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:33.549 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:33.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:33.549 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:33.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.550 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2620 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 196608~786432] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:33.550 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:33.550 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.222709894999994, 2023-07-02T14:04:33.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966, 2023-07-02T14:04:33.550 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:33.550 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:33.551 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:33.551 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:33.551 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:33.551 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2620 2023-07-02T14:04:33.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.551 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:33.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:33.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:33.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:33.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:33.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:33.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:33.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:33.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324571+0000", 2023-07-02T14:04:33.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016686649999999999 2023-07-02T14:04:33.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:33.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326240+0000", 2023-07-02T14:04:33.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1844e-05 2023-07-02T14:04:33.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:33.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326252+0000", 2023-07-02T14:04:33.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7760000000000003e-06 2023-07-02T14:04:33.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:33.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326258+0000", 2023-07-02T14:04:33.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0634000000000003e-05 2023-07-02T14:04:33.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326318+0000", 2023-07-02T14:04:33.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9629999999999993e-06 2023-07-02T14:04:33.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:33.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326328+0000", 2023-07-02T14:04:33.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.010877045 2023-07-02T14:04:33.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337205+0000", 2023-07-02T14:04:33.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022401300000000001 2023-07-02T14:04:33.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:33.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337429+0000", 2023-07-02T14:04:33.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000139822 2023-07-02T14:04:33.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:33.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337569+0000", 2023-07-02T14:04:33.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032942119999999999 2023-07-02T14:04:33.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:33.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340863+0000", 2023-07-02T14:04:33.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021302560000000001 2023-07-02T14:04:33.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:33.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342994+0000", 2023-07-02T14:04:33.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5634000000000002e-05 2023-07-02T14:04:33.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:33.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343059+0000", 2023-07-02T14:04:33.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143721 2023-07-02T14:04:33.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:33.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343203+0000", 2023-07-02T14:04:33.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966 2023-07-02T14:04:33.561 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.561 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.561 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.562 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2622 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 983040~36864] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:33.562 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:33.562 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.221017384000007, 2023-07-02T14:04:33.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664, 2023-07-02T14:04:33.563 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:33.563 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:33.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:33.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:33.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:33.563 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2622 2023-07-02T14:04:33.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.563 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:33.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:33.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:33.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:33.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:33.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:33.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:33.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:33.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326265+0000", 2023-07-02T14:04:33.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1282000000000001e-05 2023-07-02T14:04:33.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:33.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326286+0000", 2023-07-02T14:04:33.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9480000000000002e-06 2023-07-02T14:04:33.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:33.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326293+0000", 2023-07-02T14:04:33.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3610000000000003e-06 2023-07-02T14:04:33.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:33.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326297+0000", 2023-07-02T14:04:33.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7707999999999994e-05 2023-07-02T14:04:33.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326385+0000", 2023-07-02T14:04:33.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8070000000000006e-06 2023-07-02T14:04:33.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:33.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326394+0000", 2023-07-02T14:04:33.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.014535878 2023-07-02T14:04:33.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340930+0000", 2023-07-02T14:04:33.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000154551 2023-07-02T14:04:33.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:33.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341084+0000", 2023-07-02T14:04:33.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00032375900000000002 2023-07-02T14:04:33.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:33.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341408+0000", 2023-07-02T14:04:33.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012631066 2023-07-02T14:04:33.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:33.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354039+0000", 2023-07-02T14:04:33.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5741000000000001e-05 2023-07-02T14:04:33.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:33.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354055+0000", 2023-07-02T14:04:33.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8794000000000006e-05 2023-07-02T14:04:33.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:33.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354124+0000", 2023-07-02T14:04:33.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014667700000000001 2023-07-02T14:04:33.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:33.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354270+0000", 2023-07-02T14:04:33.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664 2023-07-02T14:04:33.573 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.573 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.574 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.574 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2727 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~12288] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:33.574 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:33.574 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.220981609000006, 2023-07-02T14:04:33.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999, 2023-07-02T14:04:33.575 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:33.575 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:33.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:33.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:33.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:33.575 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2727 2023-07-02T14:04:33.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.575 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:33.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:33.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:33.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:33.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:33.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:33.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:33.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:33.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:33.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3834e-05 2023-07-02T14:04:33.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:33.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326315+0000", 2023-07-02T14:04:33.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3029999999999996e-06 2023-07-02T14:04:33.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:33.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326321+0000", 2023-07-02T14:04:33.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7590000000000002e-06 2023-07-02T14:04:33.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:33.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326324+0000", 2023-07-02T14:04:33.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112872 2023-07-02T14:04:33.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326437+0000", 2023-07-02T14:04:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2290000000000002e-06 2023-07-02T14:04:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326442+0000", 2023-07-02T14:04:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0155896879999999 2023-07-02T14:04:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342031+0000", 2023-07-02T14:04:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016496100000000001 2023-07-02T14:04:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342196+0000", 2023-07-02T14:04:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021998699999999999 2023-07-02T14:04:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342416+0000", 2023-07-02T14:04:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011869744999999999 2023-07-02T14:04:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354286+0000", 2023-07-02T14:04:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0361e-05 2023-07-02T14:04:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354347+0000", 2023-07-02T14:04:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3312000000000005e-05 2023-07-02T14:04:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354430+0000", 2023-07-02T14:04:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126444 2023-07-02T14:04:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354556+0000", 2023-07-02T14:04:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999 2023-07-02T14:04:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2730 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [set-alloc-hint object_size 4194304 write_size 4194304,write 454656~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.220956599999994, 2023-07-02T14:04:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999, 2023-07-02T14:04:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2730 2023-07-02T14:04:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6049999999999997e-06 2023-07-02T14:04:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326332+0000", 2023-07-02T14:04:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3000000000000002e-06 2023-07-02T14:04:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326335+0000", 2023-07-02T14:04:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.719e-05 2023-07-02T14:04:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326352+0000", 2023-07-02T14:04:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011583599999999999 2023-07-02T14:04:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326468+0000", 2023-07-02T14:04:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.831e-06 2023-07-02T14:04:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326473+0000", 2023-07-02T14:04:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.016769915 2023-07-02T14:04:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343243+0000", 2023-07-02T14:04:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197352 2023-07-02T14:04:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343440+0000", 2023-07-02T14:04:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197581 2023-07-02T14:04:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343638+0000", 2023-07-02T14:04:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010980139 2023-07-02T14:04:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354618+0000", 2023-07-02T14:04:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056779530000000003 2023-07-02T14:04:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360296+0000", 2023-07-02T14:04:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101109 2023-07-02T14:04:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360397+0000", 2023-07-02T14:04:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000160273 2023-07-02T14:04:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360557+0000", 2023-07-02T14:04:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999 2023-07-02T14:04:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2752 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 1019904~3174400] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.220910376000006, 2023-07-02T14:04:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001, 2023-07-02T14:04:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2752 2023-07-02T14:04:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326372+0000", 2023-07-02T14:04:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052050380000000004 2023-07-02T14:04:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331577+0000", 2023-07-02T14:04:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6832000000000001e-05 2023-07-02T14:04:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331594+0000", 2023-07-02T14:04:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8220000000000007e-06 2023-07-02T14:04:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331602+0000", 2023-07-02T14:04:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9895999999999999e-05 2023-07-02T14:04:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331642+0000", 2023-07-02T14:04:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:04:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331649+0000", 2023-07-02T14:04:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0139141709999999 2023-07-02T14:04:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345563+0000", 2023-07-02T14:04:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113817 2023-07-02T14:04:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345677+0000", 2023-07-02T14:04:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018803400000000001 2023-07-02T14:04:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345865+0000", 2023-07-02T14:04:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087950359999999991 2023-07-02T14:04:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354660+0000", 2023-07-02T14:04:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059116669999999998 2023-07-02T14:04:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360572+0000", 2023-07-02T14:04:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0992000000000002e-05 2023-07-02T14:04:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360633+0000", 2023-07-02T14:04:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013601600000000001 2023-07-02T14:04:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360769+0000", 2023-07-02T14:04:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001 2023-07-02T14:04:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2759 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 0~454656] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.215670579000005, 2023-07-02T14:04:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999, 2023-07-02T14:04:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2759 2023-07-02T14:04:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331611+0000", 2023-07-02T14:04:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053795999999999998 2023-07-02T14:04:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332149+0000", 2023-07-02T14:04:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4970000000000007e-06 2023-07-02T14:04:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332158+0000", 2023-07-02T14:04:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:04:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332162+0000", 2023-07-02T14:04:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3526000000000001e-05 2023-07-02T14:04:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332185+0000", 2023-07-02T14:04:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:04:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332189+0000", 2023-07-02T14:04:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0186295910000001 2023-07-02T14:04:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350819+0000", 2023-07-02T14:04:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000111123 2023-07-02T14:04:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350930+0000", 2023-07-02T14:04:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019575300000000001 2023-07-02T14:04:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.351126+0000", 2023-07-02T14:04:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011503298 2023-07-02T14:04:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362629+0000", 2023-07-02T14:04:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085443989999999994 2023-07-02T14:04:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371173+0000", 2023-07-02T14:04:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3547999999999995e-05 2023-07-02T14:04:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371237+0000", 2023-07-02T14:04:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790300000000001 2023-07-02T14:04:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371375+0000", 2023-07-02T14:04:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999 2023-07-02T14:04:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2760 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 458752~131072] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.215114393999997, 2023-07-02T14:04:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001, 2023-07-02T14:04:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2760 2023-07-02T14:04:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332168+0000", 2023-07-02T14:04:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013281100000000001 2023-07-02T14:04:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332301+0000", 2023-07-02T14:04:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7389999999999999e-06 2023-07-02T14:04:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332308+0000", 2023-07-02T14:04:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0639999999999998e-06 2023-07-02T14:04:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332312+0000", 2023-07-02T14:04:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1735000000000001e-05 2023-07-02T14:04:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332333+0000", 2023-07-02T14:04:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8342000000000002e-05 2023-07-02T14:04:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:33.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332352+0000", 2023-07-02T14:04:33.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02234782 2023-07-02T14:04:33.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354699+0000", 2023-07-02T14:04:33.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110371 2023-07-02T14:04:33.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:33.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354810+0000", 2023-07-02T14:04:33.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022174900000000001 2023-07-02T14:04:33.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:33.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355032+0000", 2023-07-02T14:04:33.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076053329999999997 2023-07-02T14:04:33.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362637+0000", 2023-07-02T14:04:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087913609999999993 2023-07-02T14:04:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:33.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371428+0000", 2023-07-02T14:04:33.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8532999999999998e-05 2023-07-02T14:04:33.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:33.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371477+0000", 2023-07-02T14:04:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010990800000000001 2023-07-02T14:04:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371587+0000", 2023-07-02T14:04:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001 2023-07-02T14:04:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2766 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 589824~45056] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.214965769000003, 2023-07-02T14:04:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188, 2023-07-02T14:04:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2766 2023-07-02T14:04:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2247999999999998e-05 2023-07-02T14:04:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332369+0000", 2023-07-02T14:04:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4189999999999996e-06 2023-07-02T14:04:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332376+0000", 2023-07-02T14:04:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1049999999999999e-06 2023-07-02T14:04:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332379+0000", 2023-07-02T14:04:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6894000000000003e-05 2023-07-02T14:04:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332416+0000", 2023-07-02T14:04:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3970000000000002e-06 2023-07-02T14:04:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332422+0000", 2023-07-02T14:04:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0231382280000001 2023-07-02T14:04:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355560+0000", 2023-07-02T14:04:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9079000000000005e-05 2023-07-02T14:04:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355659+0000", 2023-07-02T14:04:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018005000000000001 2023-07-02T14:04:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355839+0000", 2023-07-02T14:04:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068044439999999998 2023-07-02T14:04:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362644+0000", 2023-07-02T14:04:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090155589999999994 2023-07-02T14:04:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371659+0000", 2023-07-02T14:04:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1297999999999994e-05 2023-07-02T14:04:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371730+0000", 2023-07-02T14:04:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013100699999999999 2023-07-02T14:04:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371861+0000", 2023-07-02T14:04:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188 2023-07-02T14:04:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2784 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 0~393216] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.214897316999995, 2023-07-02T14:04:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999, 2023-07-02T14:04:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2784 2023-07-02T14:04:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332385+0000", 2023-07-02T14:04:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040252400000000002 2023-07-02T14:04:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332787+0000", 2023-07-02T14:04:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0208e-05 2023-07-02T14:04:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332798+0000", 2023-07-02T14:04:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2023-07-02T14:04:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332801+0000", 2023-07-02T14:04:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4838000000000001e-05 2023-07-02T14:04:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332836+0000", 2023-07-02T14:04:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5319999999999998e-06 2023-07-02T14:04:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332839+0000", 2023-07-02T14:04:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0234290109999999 2023-07-02T14:04:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356268+0000", 2023-07-02T14:04:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011165700000000001 2023-07-02T14:04:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356380+0000", 2023-07-02T14:04:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022504799999999999 2023-07-02T14:04:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356605+0000", 2023-07-02T14:04:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015276201 2023-07-02T14:04:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371881+0000", 2023-07-02T14:04:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5496999999999999e-05 2023-07-02T14:04:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:33.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371957+0000", 2023-07-02T14:04:33.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4570000000000002e-05 2023-07-02T14:04:33.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:33.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372001+0000", 2023-07-02T14:04:33.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3880000000000003e-05 2023-07-02T14:04:33.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:33.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372085+0000", 2023-07-02T14:04:33.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999 2023-07-02T14:04:33.661 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.661 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.661 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.662 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2785 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 405504~446464] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:33.662 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:33.662 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.214475866000001, 2023-07-02T14:04:33.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999, 2023-07-02T14:04:33.663 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:33.663 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:33.663 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:33.664 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:33.664 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:33.664 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2785 2023-07-02T14:04:33.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.664 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:33.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:33.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:33.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:33.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:33.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:33.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:33.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:33.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:33.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059527500000000004 2023-07-02T14:04:33.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:33.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333402+0000", 2023-07-02T14:04:33.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7059999999999993e-06 2023-07-02T14:04:33.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:33.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333411+0000", 2023-07-02T14:04:33.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0450000000000001e-06 2023-07-02T14:04:33.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:33.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333414+0000", 2023-07-02T14:04:33.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5790999999999997e-05 2023-07-02T14:04:33.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:04:33.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3320000000000002e-06 2023-07-02T14:04:33.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:33.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333454+0000", 2023-07-02T14:04:33.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.025310269 2023-07-02T14:04:33.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358764+0000", 2023-07-02T14:04:33.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012414899999999999 2023-07-02T14:04:33.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:33.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358888+0000", 2023-07-02T14:04:33.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000217028 2023-07-02T14:04:33.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:33.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.359105+0000", 2023-07-02T14:04:33.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012788212 2023-07-02T14:04:33.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:33.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371893+0000", 2023-07-02T14:04:33.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012201569999999999 2023-07-02T14:04:33.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:33.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373114+0000", 2023-07-02T14:04:33.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4512999999999995e-05 2023-07-02T14:04:33.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:33.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373198+0000", 2023-07-02T14:04:33.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132373 2023-07-02T14:04:33.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:33.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373330+0000", 2023-07-02T14:04:33.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999 2023-07-02T14:04:33.675 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.675 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.675 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.675 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2786 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1310720~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:33.675 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:33.675 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.213863172999993, 2023-07-02T14:04:33.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001, 2023-07-02T14:04:33.676 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:33.676 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:33.676 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:33.676 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:33.676 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:33.677 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2786 2023-07-02T14:04:33.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.677 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:33.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:33.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:33.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:33.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:33.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:33.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:33.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:33.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333419+0000", 2023-07-02T14:04:33.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2009999999999999e-06 2023-07-02T14:04:33.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:33.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333427+0000", 2023-07-02T14:04:33.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0210000000000001e-06 2023-07-02T14:04:33.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:33.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333431+0000", 2023-07-02T14:04:33.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3769999999999999e-06 2023-07-02T14:04:33.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:33.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333434+0000", 2023-07-02T14:04:33.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1161e-05 2023-07-02T14:04:33.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333485+0000", 2023-07-02T14:04:33.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1549000000000001e-05 2023-07-02T14:04:33.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:33.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333496+0000", 2023-07-02T14:04:33.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0273116339999999 2023-07-02T14:04:33.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360808+0000", 2023-07-02T14:04:33.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000115128 2023-07-02T14:04:33.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:33.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360923+0000", 2023-07-02T14:04:33.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018367700000000001 2023-07-02T14:04:33.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:33.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361107+0000", 2023-07-02T14:04:33.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010796923999999999 2023-07-02T14:04:33.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:33.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371904+0000", 2023-07-02T14:04:33.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014983520000000001 2023-07-02T14:04:33.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:33.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373402+0000", 2023-07-02T14:04:33.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0520999999999999e-05 2023-07-02T14:04:33.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:33.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373463+0000", 2023-07-02T14:04:33.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000129095 2023-07-02T14:04:33.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:33.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373592+0000", 2023-07-02T14:04:33.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001 2023-07-02T14:04:33.687 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.687 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.688 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.688 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2787 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1433600~8192] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:33.688 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:33.688 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.213846372999996, 2023-07-02T14:04:33.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999, 2023-07-02T14:04:33.689 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:33.689 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:33.689 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:33.689 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:33.689 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:33.689 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2787 2023-07-02T14:04:33.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.689 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:33.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:33.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:33.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:33.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:33.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:33.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:33.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:33.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333436+0000", 2023-07-02T14:04:33.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2731999999999999e-05 2023-07-02T14:04:33.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:33.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:04:33.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7440000000000001e-06 2023-07-02T14:04:33.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:33.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333456+0000", 2023-07-02T14:04:33.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8650000000000003e-06 2023-07-02T14:04:33.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:33.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333460+0000", 2023-07-02T14:04:33.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0077e-05 2023-07-02T14:04:33.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333530+0000", 2023-07-02T14:04:33.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4180000000000001e-06 2023-07-02T14:04:33.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:33.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333533+0000", 2023-07-02T14:04:33.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0284002720000001 2023-07-02T14:04:33.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361933+0000", 2023-07-02T14:04:33.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6781999999999996e-05 2023-07-02T14:04:33.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:33.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362030+0000", 2023-07-02T14:04:33.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018305400000000001 2023-07-02T14:04:33.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:33.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362213+0000", 2023-07-02T14:04:33.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097015109999999995 2023-07-02T14:04:33.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:33.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371915+0000", 2023-07-02T14:04:33.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001742295 2023-07-02T14:04:33.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:33.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373657+0000", 2023-07-02T14:04:33.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5980000000000003e-05 2023-07-02T14:04:33.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:33.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373713+0000", 2023-07-02T14:04:33.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011783800000000001 2023-07-02T14:04:33.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:33.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373831+0000", 2023-07-02T14:04:33.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999 2023-07-02T14:04:33.699 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.699 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.699 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.700 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2803 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 851968~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:33.700 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:33.700 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.213804521, 2023-07-02T14:04:33.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514, 2023-07-02T14:04:33.700 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:33.700 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:33.701 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:33.701 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:33.701 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:33.701 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2803 2023-07-02T14:04:33.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.701 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:33.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:33.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:33.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:33.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:33.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:33.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:33.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:33.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:33.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4943000000000001e-05 2023-07-02T14:04:33.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:33.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333493+0000", 2023-07-02T14:04:33.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2629999999999997e-06 2023-07-02T14:04:33.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:33.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333500+0000", 2023-07-02T14:04:33.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4159999999999997e-06 2023-07-02T14:04:33.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:33.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333505+0000", 2023-07-02T14:04:33.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4518999999999998e-05 2023-07-02T14:04:33.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333559+0000", 2023-07-02T14:04:33.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5900000000000008e-06 2023-07-02T14:04:33.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:33.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333568+0000", 2023-07-02T14:04:33.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0291061779999999 2023-07-02T14:04:33.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362674+0000", 2023-07-02T14:04:33.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110412 2023-07-02T14:04:33.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:33.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362784+0000", 2023-07-02T14:04:33.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015384599999999999 2023-07-02T14:04:33.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:33.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362938+0000", 2023-07-02T14:04:33.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091682310000000006 2023-07-02T14:04:33.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:33.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372106+0000", 2023-07-02T14:04:33.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017703230000000001 2023-07-02T14:04:33.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:33.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373877+0000", 2023-07-02T14:04:33.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6387000000000002e-05 2023-07-02T14:04:33.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:33.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373923+0000", 2023-07-02T14:04:33.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014099199999999999 2023-07-02T14:04:33.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:33.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374064+0000", 2023-07-02T14:04:33.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514 2023-07-02T14:04:33.711 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.711 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.711 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.712 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2805 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 634880~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:33.712 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:33.712 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.213771346000001, 2023-07-02T14:04:33.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814, 2023-07-02T14:04:33.712 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:33.712 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:33.713 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:33.713 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:33.713 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:33.713 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2805 2023-07-02T14:04:33.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.713 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:33.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:33.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:33.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:33.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:33.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:33.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:33.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:33.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333511+0000", 2023-07-02T14:04:33.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207092 2023-07-02T14:04:33.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:33.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333718+0000", 2023-07-02T14:04:33.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5110000000000003e-06 2023-07-02T14:04:33.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:33.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333727+0000", 2023-07-02T14:04:33.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3270000000000007e-06 2023-07-02T14:04:33.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:33.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333736+0000", 2023-07-02T14:04:33.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0319999999999999e-05 2023-07-02T14:04:33.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333756+0000", 2023-07-02T14:04:33.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1695000000000001e-05 2023-07-02T14:04:33.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:33.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:04:33.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0295750560000001 2023-07-02T14:04:33.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363343+0000", 2023-07-02T14:04:33.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3387999999999994e-05 2023-07-02T14:04:33.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:33.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363437+0000", 2023-07-02T14:04:33.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018373 2023-07-02T14:04:33.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:33.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363620+0000", 2023-07-02T14:04:33.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0084948769999999996 2023-07-02T14:04:33.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:33.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372115+0000", 2023-07-02T14:04:33.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020082170000000001 2023-07-02T14:04:33.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:33.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374123+0000", 2023-07-02T14:04:33.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8875999999999997e-05 2023-07-02T14:04:33.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:33.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374182+0000", 2023-07-02T14:04:33.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0001272 2023-07-02T14:04:33.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:33.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374309+0000", 2023-07-02T14:04:33.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814 2023-07-02T14:04:33.723 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.723 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.723 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.724 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2806 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 1359872~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:33.724 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:33.724 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.213541015000004, 2023-07-02T14:04:33.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234, 2023-07-02T14:04:33.724 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:33.724 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:33.724 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:33.725 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:33.725 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:33.725 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2806 2023-07-02T14:04:33.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.725 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:33.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:33.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:33.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:33.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:33.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:33.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:33.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:33.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333741+0000", 2023-07-02T14:04:33.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3978999999999999e-05 2023-07-02T14:04:33.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:33.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333755+0000", 2023-07-02T14:04:33.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:04:33.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:33.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333761+0000", 2023-07-02T14:04:33.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7010000000000001e-06 2023-07-02T14:04:33.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:33.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333764+0000", 2023-07-02T14:04:33.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5086000000000003e-05 2023-07-02T14:04:33.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333799+0000", 2023-07-02T14:04:33.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6389999999999992e-06 2023-07-02T14:04:33.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:33.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333806+0000", 2023-07-02T14:04:33.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.030496675 2023-07-02T14:04:33.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364303+0000", 2023-07-02T14:04:33.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011382700000000001 2023-07-02T14:04:33.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:33.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364417+0000", 2023-07-02T14:04:33.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192058 2023-07-02T14:04:33.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:33.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364609+0000", 2023-07-02T14:04:33.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075132130000000004 2023-07-02T14:04:33.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:33.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372122+0000", 2023-07-02T14:04:33.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022731750000000001 2023-07-02T14:04:33.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:33.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374395+0000", 2023-07-02T14:04:33.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5277000000000005e-05 2023-07-02T14:04:33.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:33.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374460+0000", 2023-07-02T14:04:33.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017074599999999999 2023-07-02T14:04:33.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374631+0000", 2023-07-02T14:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234 2023-07-02T14:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.735 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 868352~49152] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.213514301000004, 2023-07-02T14:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001, 2023-07-02T14:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:33.736 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:33.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6355000000000001e-05 2023-07-02T14:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333794+0000", 2023-07-02T14:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8329999999999996e-06 2023-07-02T14:04:33.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333801+0000", 2023-07-02T14:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5419999999999996e-06 2023-07-02T14:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333807+0000", 2023-07-02T14:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2023-07-02T14:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333826+0000", 2023-07-02T14:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.303e-06 2023-07-02T14:04:33.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333830+0000", 2023-07-02T14:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.031137559 2023-07-02T14:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364967+0000", 2023-07-02T14:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1390000000000004e-05 2023-07-02T14:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365059+0000", 2023-07-02T14:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017597399999999999 2023-07-02T14:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365234+0000", 2023-07-02T14:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068943950000000002 2023-07-02T14:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:33.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372129+0000", 2023-07-02T14:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026189910000000002 2023-07-02T14:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374748+0000", 2023-07-02T14:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6613999999999996e-05 2023-07-02T14:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374814+0000", 2023-07-02T14:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502300000000001 2023-07-02T14:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374950+0000", 2023-07-02T14:04:33.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001 2023-07-02T14:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2815 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 917504~49152] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:33.747 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.209300068000005, 2023-07-02T14:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998, 2023-07-02T14:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:33.748 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2815 2023-07-02T14:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:33.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:33.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:04:33.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:33.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337980+0000", 2023-07-02T14:04:33.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017126700000000001 2023-07-02T14:04:33.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:33.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338152+0000", 2023-07-02T14:04:33.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2938999999999999e-05 2023-07-02T14:04:33.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:33.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338164+0000", 2023-07-02T14:04:33.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4448999999999999e-05 2023-07-02T14:04:33.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:33.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:04:33.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0281812 2023-07-02T14:04:33.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366370+0000", 2023-07-02T14:04:33.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1840000000000002e-05 2023-07-02T14:04:33.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:33.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366462+0000", 2023-07-02T14:04:33.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000186112 2023-07-02T14:04:33.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:33.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366648+0000", 2023-07-02T14:04:33.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054905709999999996 2023-07-02T14:04:33.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:33.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372139+0000", 2023-07-02T14:04:33.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044954790000000001 2023-07-02T14:04:33.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:33.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376634+0000", 2023-07-02T14:04:33.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4621000000000004e-05 2023-07-02T14:04:33.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:33.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376699+0000", 2023-07-02T14:04:33.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012862299999999999 2023-07-02T14:04:33.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:33.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376827+0000", 2023-07-02T14:04:33.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998 2023-07-02T14:04:33.757 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.757 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.757 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.758 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2816 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 917504~65536] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:33.758 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:33.758 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.209085689000005, 2023-07-02T14:04:33.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001, 2023-07-02T14:04:33.758 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:33.758 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:33.758 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:33.759 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:33.759 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:33.759 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2816 2023-07-02T14:04:33.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.759 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:33.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:33.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:33.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:33.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:33.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:33.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:33.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:33.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338196+0000", 2023-07-02T14:04:33.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6286999999999999e-05 2023-07-02T14:04:33.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:33.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338243+0000", 2023-07-02T14:04:33.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7030000000000001e-06 2023-07-02T14:04:33.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:33.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338252+0000", 2023-07-02T14:04:33.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:04:33.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:33.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338258+0000", 2023-07-02T14:04:33.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032085734999999997 2023-07-02T14:04:33.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370344+0000", 2023-07-02T14:04:33.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108621 2023-07-02T14:04:33.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:33.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370452+0000", 2023-07-02T14:04:33.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014936499999999999 2023-07-02T14:04:33.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:33.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370602+0000", 2023-07-02T14:04:33.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001778012 2023-07-02T14:04:33.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:33.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372380+0000", 2023-07-02T14:04:33.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044876380000000004 2023-07-02T14:04:33.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:33.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376867+0000", 2023-07-02T14:04:33.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9870999999999999e-05 2023-07-02T14:04:33.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:33.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376897+0000", 2023-07-02T14:04:33.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011401 2023-07-02T14:04:33.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:33.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.377011+0000", 2023-07-02T14:04:33.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001 2023-07-02T14:04:33.767 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.768 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.768 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:3878 2.1 e80/79 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head v 80'436, mlcod=80'429)", 2023-07-02T14:04:33.768 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:33.768 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.439832191999997, 2023-07-02T14:04:33.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037799472000000001, 2023-07-02T14:04:33.769 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:33.769 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:33.769 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:33.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:33.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:33.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:33.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:33.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:33.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:33.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:33.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107450+0000", 2023-07-02T14:04:33.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023914520000000001 2023-07-02T14:04:33.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:33.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109841+0000", 2023-07-02T14:04:33.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0823e-05 2023-07-02T14:04:33.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:33.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109862+0000", 2023-07-02T14:04:33.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2023-07-02T14:04:33.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:33.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109874+0000", 2023-07-02T14:04:33.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075009600000000001 2023-07-02T14:04:33.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.110624+0000", 2023-07-02T14:04:33.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5176000000000001e-05 2023-07-02T14:04:33.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:33.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.110649+0000", 2023-07-02T14:04:33.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034557310000000001 2023-07-02T14:04:33.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:33.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145206+0000", 2023-07-02T14:04:33.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4110999999999998e-05 2023-07-02T14:04:33.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:33.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145250+0000", 2023-07-02T14:04:33.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037799472000000001 2023-07-02T14:04:33.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.775 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.776 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.776 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:196 2.5 2:a6e4b3cf:::rbd_data.125de1449605.00000000000000c4:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:04:33.776 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172685+0000", 2023-07-02T14:04:33.776 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.374598033, 2023-07-02T14:04:33.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038025259999999998, 2023-07-02T14:04:33.777 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:33.777 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:33.777 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:33.777 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:04:33.777 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:04:33.777 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 196 2023-07-02T14:04:33.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.777 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:33.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:33.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172685+0000", 2023-07-02T14:04:33.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:33.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:33.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172685+0000", 2023-07-02T14:04:33.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:33.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:33.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172684+0000", 2023-07-02T14:04:33.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.693e-06 2023-07-02T14:04:33.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:33.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172688+0000", 2023-07-02T14:04:33.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4470000000000003e-06 2023-07-02T14:04:33.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:33.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172695+0000", 2023-07-02T14:04:33.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.014e-06 2023-07-02T14:04:33.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:33.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172699+0000", 2023-07-02T14:04:33.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025360667 2023-07-02T14:04:33.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:33.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.198059+0000", 2023-07-02T14:04:33.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011197900000000001 2023-07-02T14:04:33.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:33.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.198171+0000", 2023-07-02T14:04:33.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012538942000000001 2023-07-02T14:04:33.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:33.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:33.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:33.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.210710+0000", 2023-07-02T14:04:33.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038025259999999998 2023-07-02T14:04:33.783 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.783 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.783 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.784 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:33.784 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:33.784 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:33.786 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:04:33.786 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 100% complete...done. 2023-07-02T14:04:33.787 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:33.785+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:33.797 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:04:33.797 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=25 2023-07-02T14:04:33.797 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap25 2023-07-02T14:04:33.886 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:33.886+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:33.951 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:33.951 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:33.951 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:33.951 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:33.961 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:04:34.002 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:34.002+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:34.088 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:34.088+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:34.124 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:34.125 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:34.125 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:34.125 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:34.125 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:34.135 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:04:34.189 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:34.189+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:34.290 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:34.290+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:34.302 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:34.302 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:34.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:34.302 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:34.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.303 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:04:34.303 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:34.303 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 84.848258591000004, 2023-07-02T14:04:34.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:04:34.303 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:34.303 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:34.304 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:34.304 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:34.304 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:34.304 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:04:34.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.304 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:34.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:34.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:34.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:04:34.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:04:34.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:34.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:04:34.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:04:34.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:34.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:04:34.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:04:34.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:34.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:04:34.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:04:34.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:04:34.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:04:34.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:34.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:04:34.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:04:34.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:04:34.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:04:34.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:34.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:04:34.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:04:34.311 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.311 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.311 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.311 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:04:34.312 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:34.312 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.977565583000001, 2023-07-02T14:04:34.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:04:34.312 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:34.312 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:34.312 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:34.312 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:04:34.312 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:04:34.313 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:04:34.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.313 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:34.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:34.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:34.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:34.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:34.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:34.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:34.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:34.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:04:34.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:04:34.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:34.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:04:34.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:04:34.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:04:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:04:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:04:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:04:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:04:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:04:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:04:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:04:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:04:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:04:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:04:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:04:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:34.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:04:34.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:04:34.320 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.320 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.320 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.320 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:34.321 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:34.321 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.977440861999995, 2023-07-02T14:04:34.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:04:34.321 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:34.321 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:34.321 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:34.321 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:34.322 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:34.322 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:04:34.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.322 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:34.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:34.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:34.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:34.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:34.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:34.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:34.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:34.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:04:34.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:04:34.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:34.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:04:34.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:04:34.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:04:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:04:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:04:34.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:04:34.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:04:34.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:04:34.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:34.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:04:34.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:04:34.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:04:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:04:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:04:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:04:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.970590978000004, 2023-07-02T14:04:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:04:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:04:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:04:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:04:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:04:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:04:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:04:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:04:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:04:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:04:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:04:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:04:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:04:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:04:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:04:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:04:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:04:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:04:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:04:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.970528052999995, 2023-07-02T14:04:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:04:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:04:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:34.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:34.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:34.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:34.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:04:34.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:04:34.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:34.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:04:34.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:04:34.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:34.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:04:34.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:04:34.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:34.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:04:34.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:04:34.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:04:34.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:04:34.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:34.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:04:34.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:04:34.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:04:34.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:04:34.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:34.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:04:34.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:04:34.345 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.345 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.345 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.346 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:34.346 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:34.346 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.970227632000004, 2023-07-02T14:04:34.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:04:34.346 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:34.346 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:34.347 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:34.347 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:34.347 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:34.347 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:04:34.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.347 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:34.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:34.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:34.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:34.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:34.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:34.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:34.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:04:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:04:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:34.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:04:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:04:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:04:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:04:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:34.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:04:34.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:04:34.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:04:34.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:04:34.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:34.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:04:34.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:04:34.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:04:34.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:04:34.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:34.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:04:34.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:04:34.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:04:34.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:04:34.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:04:34.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:34.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:04:34.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:04:34.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:34.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:04:34.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:04:34.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:34.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:04:34.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:04:34.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:34.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:04:34.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:04:34.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:34.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:04:34.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:04:34.357 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.358 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.358 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.358 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:04:34.358 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:34.358 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.969703132000006, 2023-07-02T14:04:34.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:04:34.359 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:34.359 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:34.359 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:34.359 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:34.359 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:34.359 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:04:34.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.359 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:34.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:34.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:34.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:34.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:34.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:34.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:34.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:34.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:34.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:04:34.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:34.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:04:34.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:04:34.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:34.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:04:34.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:04:34.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:34.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:04:34.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:04:34.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:04:34.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:04:34.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:34.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:04:34.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:04:34.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:04:34.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:04:34.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:34.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:04:34.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:04:34.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:34.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:04:34.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:04:34.366 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.367 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.367 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.367 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:04:34.367 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:34.367 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.845684007000003, 2023-07-02T14:04:34.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:04:34.368 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:34.368 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:34.368 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:34.368 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:34.368 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:34.368 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:04:34.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.369 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:34.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:34.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:34.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:34.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:34.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:34.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:34.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:34.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:04:34.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:04:34.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:34.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:04:34.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:04:34.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:34.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:04:34.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:04:34.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:34.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:04:34.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:04:34.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:04:34.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:04:34.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:34.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:04:34.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:04:34.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:04:34.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:04:34.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:34.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:04:34.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:04:34.375 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.375 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.375 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.376 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:34.376 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:34.376 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.520201303, 2023-07-02T14:04:34.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:04:34.376 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:34.376 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:34.376 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:34.376 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:34.377 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:34.377 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:04:34.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.377 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:34.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:34.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:34.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:34.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:34.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:34.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:34.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:34.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:04:34.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:04:34.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:34.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:04:34.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:04:34.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:34.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:04:34.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:04:34.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:34.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:04:34.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:04:34.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:34.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:04:34.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:34.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:04:34.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:04:34.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:04:34.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:04:34.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:34.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:04:34.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:04:34.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:34.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:04:34.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:04:34.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:34.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:04:34.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:04:34.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:34.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:04:34.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:04:34.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:34.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:04:34.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:04:34.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:34.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:04:34.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:04:34.387 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.387 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.387 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.387 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3887 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:34.387 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:34.387 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.516338114, 2023-07-02T14:04:34.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998, 2023-07-02T14:04:34.388 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:34.388 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:34.388 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:34.388 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:34.388 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:34.388 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3887 2023-07-02T14:04:34.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.389 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:34.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:34.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:34.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:34.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:34.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:34.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:34.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:34.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784019+0000", 2023-07-02T14:04:34.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4073999999999998e-05 2023-07-02T14:04:34.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:34.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784063+0000", 2023-07-02T14:04:34.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6630000000000004e-06 2023-07-02T14:04:34.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:34.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784070+0000", 2023-07-02T14:04:34.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.286e-06 2023-07-02T14:04:34.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:34.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784073+0000", 2023-07-02T14:04:34.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4941999999999998e-05 2023-07-02T14:04:34.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784118+0000", 2023-07-02T14:04:34.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8120000000000008e-06 2023-07-02T14:04:34.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:34.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784126+0000", 2023-07-02T14:04:34.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32999709399999999 2023-07-02T14:04:34.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114123+0000", 2023-07-02T14:04:34.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000157257 2023-07-02T14:04:34.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:34.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114280+0000", 2023-07-02T14:04:34.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000341152 2023-07-02T14:04:34.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:34.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114622+0000", 2023-07-02T14:04:34.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019539766 2023-07-02T14:04:34.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:34.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.134161+0000", 2023-07-02T14:04:34.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015820796000000002 2023-07-02T14:04:34.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:34.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149982+0000", 2023-07-02T14:04:34.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.05e-05 2023-07-02T14:04:34.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:34.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150043+0000", 2023-07-02T14:04:34.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6453999999999999e-05 2023-07-02T14:04:34.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:34.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150089+0000", 2023-07-02T14:04:34.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998 2023-07-02T14:04:34.398 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.398 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.398 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.399 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3939 2.1 2:9ef34a83:::rbd_data.125de1449605.00000000000000b8:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:34.399 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:34.399 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.516280606000002, 2023-07-02T14:04:34.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001, 2023-07-02T14:04:34.399 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:34.399 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:34.400 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:34.400 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:34.400 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:34.400 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3939 2023-07-02T14:04:34.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.400 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:34.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:34.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:34.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:34.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:34.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:34.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:34.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:34.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:34.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046852830000000002 2023-07-02T14:04:34.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:34.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788763+0000", 2023-07-02T14:04:34.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3185e-05 2023-07-02T14:04:34.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:34.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788776+0000", 2023-07-02T14:04:34.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8470000000000003e-06 2023-07-02T14:04:34.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:34.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788781+0000", 2023-07-02T14:04:34.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7104000000000002e-05 2023-07-02T14:04:34.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788828+0000", 2023-07-02T14:04:34.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3516000000000001e-05 2023-07-02T14:04:34.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:34.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788841+0000", 2023-07-02T14:04:34.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33064513600000001 2023-07-02T14:04:34.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119487+0000", 2023-07-02T14:04:34.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9876999999999997e-05 2023-07-02T14:04:34.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:34.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119537+0000", 2023-07-02T14:04:34.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014160999999999999 2023-07-02T14:04:34.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:34.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119678+0000", 2023-07-02T14:04:34.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023054392 2023-07-02T14:04:34.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:34.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.142733+0000", 2023-07-02T14:04:34.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074065470000000003 2023-07-02T14:04:34.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:34.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150139+0000", 2023-07-02T14:04:34.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5025999999999997e-05 2023-07-02T14:04:34.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:34.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150204+0000", 2023-07-02T14:04:34.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147999999999997e-05 2023-07-02T14:04:34.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:34.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150241+0000", 2023-07-02T14:04:34.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001 2023-07-02T14:04:34.410 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.410 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.410 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.410 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3940 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:34.411 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:34.411 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.511569573999999, 2023-07-02T14:04:34.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001, 2023-07-02T14:04:34.411 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:34.411 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:34.411 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:34.411 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:34.411 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:34.412 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3940 2023-07-02T14:04:34.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.412 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:34.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:34.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:34.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:34.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:34.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:34.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:34.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:34.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788788+0000", 2023-07-02T14:04:34.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043166560000000003 2023-07-02T14:04:34.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:34.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793105+0000", 2023-07-02T14:04:34.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5970000000000001e-05 2023-07-02T14:04:34.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:34.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793121+0000", 2023-07-02T14:04:34.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0939999999999998e-06 2023-07-02T14:04:34.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:34.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793126+0000", 2023-07-02T14:04:34.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2438000000000002e-05 2023-07-02T14:04:34.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793178+0000", 2023-07-02T14:04:34.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2019e-05 2023-07-02T14:04:34.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:34.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793190+0000", 2023-07-02T14:04:34.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33208757799999999 2023-07-02T14:04:34.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125278+0000", 2023-07-02T14:04:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5255999999999997e-05 2023-07-02T14:04:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:34.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125373+0000", 2023-07-02T14:04:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015674999999999999 2023-07-02T14:04:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125530+0000", 2023-07-02T14:04:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018767308 2023-07-02T14:04:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:34.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144297+0000", 2023-07-02T14:04:34.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060291959999999997 2023-07-02T14:04:34.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:34.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150326+0000", 2023-07-02T14:04:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3886999999999993e-05 2023-07-02T14:04:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150390+0000", 2023-07-02T14:04:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010255 2023-07-02T14:04:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150493+0000", 2023-07-02T14:04:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001 2023-07-02T14:04:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.423 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3947 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:34.423 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:34.423 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.507224047000001, 2023-07-02T14:04:34.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999, 2023-07-02T14:04:34.423 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:34.423 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:34.423 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:34.423 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3947 2023-07-02T14:04:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:34.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:34.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:34.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:34.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:34.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793133+0000", 2023-07-02T14:04:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2519999999999999e-05 2023-07-02T14:04:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:34.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793176+0000", 2023-07-02T14:04:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:04:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793180+0000", 2023-07-02T14:04:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1709999999999998e-06 2023-07-02T14:04:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:34.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793184+0000", 2023-07-02T14:04:34.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1721999999999999e-05 2023-07-02T14:04:34.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793215+0000", 2023-07-02T14:04:34.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1280000000000008e-06 2023-07-02T14:04:34.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:34.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793224+0000", 2023-07-02T14:04:34.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34218549599999998 2023-07-02T14:04:34.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135409+0000", 2023-07-02T14:04:34.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117442 2023-07-02T14:04:34.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135526+0000", 2023-07-02T14:04:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000206257 2023-07-02T14:04:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135733+0000", 2023-07-02T14:04:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085700029999999996 2023-07-02T14:04:34.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:34.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144303+0000", 2023-07-02T14:04:34.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006278652 2023-07-02T14:04:34.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150581+0000", 2023-07-02T14:04:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4765999999999996e-05 2023-07-02T14:04:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150646+0000", 2023-07-02T14:04:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4777999999999995e-05 2023-07-02T14:04:34.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:34.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150711+0000", 2023-07-02T14:04:34.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999 2023-07-02T14:04:34.434 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.434 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.434 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3980 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 65536~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.507170164999998, 2023-07-02T14:04:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999, 2023-07-02T14:04:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:34.435 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:34.436 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:34.436 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:34.436 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:34.436 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3980 2023-07-02T14:04:34.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.436 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:34.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:34.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020244260000000002 2023-07-02T14:04:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:34.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795212+0000", 2023-07-02T14:04:34.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0005e-05 2023-07-02T14:04:34.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:34.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795222+0000", 2023-07-02T14:04:34.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9860000000000002e-06 2023-07-02T14:04:34.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:34.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795227+0000", 2023-07-02T14:04:34.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6369999999999999e-05 2023-07-02T14:04:34.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795264+0000", 2023-07-02T14:04:34.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1052e-05 2023-07-02T14:04:34.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:34.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795275+0000", 2023-07-02T14:04:34.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.343850459 2023-07-02T14:04:34.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139125+0000", 2023-07-02T14:04:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6760000000000004e-05 2023-07-02T14:04:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139202+0000", 2023-07-02T14:04:34.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015092799999999999 2023-07-02T14:04:34.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:34.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139353+0000", 2023-07-02T14:04:34.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011425940000000001 2023-07-02T14:04:34.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150779+0000", 2023-07-02T14:04:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033223060000000001 2023-07-02T14:04:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154101+0000", 2023-07-02T14:04:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9268999999999998e-05 2023-07-02T14:04:34.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:34.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154161+0000", 2023-07-02T14:04:34.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6499999999999999e-05 2023-07-02T14:04:34.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:34.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154207+0000", 2023-07-02T14:04:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999 2023-07-02T14:04:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3982 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1835008~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:34.446 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.505124877999997, 2023-07-02T14:04:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002, 2023-07-02T14:04:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:34.447 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3982 2023-07-02T14:04:34.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.448 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:34.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:34.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:34.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:34.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795233+0000", 2023-07-02T14:04:34.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8680999999999998e-05 2023-07-02T14:04:34.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:34.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795272+0000", 2023-07-02T14:04:34.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5629999999999999e-06 2023-07-02T14:04:34.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:34.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795277+0000", 2023-07-02T14:04:34.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8700000000000001e-06 2023-07-02T14:04:34.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:34.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795280+0000", 2023-07-02T14:04:34.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1964000000000001e-05 2023-07-02T14:04:34.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795312+0000", 2023-07-02T14:04:34.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.314e-06 2023-07-02T14:04:34.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:34.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795319+0000", 2023-07-02T14:04:34.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34828442599999998 2023-07-02T14:04:34.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143603+0000", 2023-07-02T14:04:34.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2438000000000001e-05 2023-07-02T14:04:34.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:34.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143665+0000", 2023-07-02T14:04:34.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3079999999999997e-05 2023-07-02T14:04:34.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:34.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143759+0000", 2023-07-02T14:04:34.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070633079999999999 2023-07-02T14:04:34.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:34.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150822+0000", 2023-07-02T14:04:34.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034430210000000001 2023-07-02T14:04:34.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:34.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154265+0000", 2023-07-02T14:04:34.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4249999999999998e-05 2023-07-02T14:04:34.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:34.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154309+0000", 2023-07-02T14:04:34.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2020000000000001e-05 2023-07-02T14:04:34.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:34.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154351+0000", 2023-07-02T14:04:34.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002 2023-07-02T14:04:34.458 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.458 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.458 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.459 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4054 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1900544~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:34.459 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:34.459 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.505073207000002, 2023-07-02T14:04:34.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002, 2023-07-02T14:04:34.459 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:34.459 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:34.460 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:34.460 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:34.460 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:34.460 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4054 2023-07-02T14:04:34.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.460 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:34.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:34.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:34.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:34.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:34.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:34.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:34.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:34.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:34.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018874860000000001 2023-07-02T14:04:34.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:34.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797172+0000", 2023-07-02T14:04:34.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3000000000000007e-06 2023-07-02T14:04:34.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:34.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797182+0000", 2023-07-02T14:04:34.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6260000000000003e-06 2023-07-02T14:04:34.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:34.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797186+0000", 2023-07-02T14:04:34.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9013e-05 2023-07-02T14:04:34.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797235+0000", 2023-07-02T14:04:34.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5990000000000008e-06 2023-07-02T14:04:34.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:34.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797245+0000", 2023-07-02T14:04:34.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34785621999999999 2023-07-02T14:04:34.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145101+0000", 2023-07-02T14:04:34.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7762000000000006e-05 2023-07-02T14:04:34.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:34.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145169+0000", 2023-07-02T14:04:34.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013469400000000001 2023-07-02T14:04:34.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:34.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145304+0000", 2023-07-02T14:04:34.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089538039999999992 2023-07-02T14:04:34.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:34.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154257+0000", 2023-07-02T14:04:34.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104218 2023-07-02T14:04:34.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:34.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154362+0000", 2023-07-02T14:04:34.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4168000000000001e-05 2023-07-02T14:04:34.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:34.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154406+0000", 2023-07-02T14:04:34.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3098000000000001e-05 2023-07-02T14:04:34.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:34.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154429+0000", 2023-07-02T14:04:34.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002 2023-07-02T14:04:34.470 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.470 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.470 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.471 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4056 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3670016~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:34.471 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:34.471 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.503166694000001, 2023-07-02T14:04:34.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997, 2023-07-02T14:04:34.472 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:34.472 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:34.472 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:34.472 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:34.472 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:34.472 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4056 2023-07-02T14:04:34.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.473 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:34.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:34.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:34.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:34.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:34.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:34.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:34.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:34.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797191+0000", 2023-07-02T14:04:34.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0438999999999997e-05 2023-07-02T14:04:34.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:34.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797232+0000", 2023-07-02T14:04:34.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6929999999999998e-06 2023-07-02T14:04:34.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:34.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797237+0000", 2023-07-02T14:04:34.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9739999999999998e-06 2023-07-02T14:04:34.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:34.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797240+0000", 2023-07-02T14:04:34.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0243000000000002e-05 2023-07-02T14:04:34.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797270+0000", 2023-07-02T14:04:34.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9889999999999997e-06 2023-07-02T14:04:34.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:34.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797276+0000", 2023-07-02T14:04:34.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35358319799999999 2023-07-02T14:04:34.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150859+0000", 2023-07-02T14:04:34.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1357e-05 2023-07-02T14:04:34.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:34.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150940+0000", 2023-07-02T14:04:34.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014903 2023-07-02T14:04:34.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:34.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151089+0000", 2023-07-02T14:04:34.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033512519999999999 2023-07-02T14:04:34.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:34.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154441+0000", 2023-07-02T14:04:34.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001050945 2023-07-02T14:04:34.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:34.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155492+0000", 2023-07-02T14:04:34.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2523e-05 2023-07-02T14:04:34.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:34.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155524+0000", 2023-07-02T14:04:34.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.057e-05 2023-07-02T14:04:34.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:34.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155555+0000", 2023-07-02T14:04:34.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997 2023-07-02T14:04:34.487 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.487 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.487 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.487 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4088 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3735552~458752] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:34.487 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:34.488 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.503114498999999, 2023-07-02T14:04:34.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998, 2023-07-02T14:04:34.488 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:34.488 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:34.488 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:34.488 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:34.488 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:34.489 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4088 2023-07-02T14:04:34.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.489 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:34.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:34.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:34.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:34.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:34.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:34.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:34.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:34.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:34.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00038503499999999999 2023-07-02T14:04:34.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:34.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797629+0000", 2023-07-02T14:04:34.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8739999999999995e-06 2023-07-02T14:04:34.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:34.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797637+0000", 2023-07-02T14:04:34.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7160000000000001e-06 2023-07-02T14:04:34.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:34.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797640+0000", 2023-07-02T14:04:34.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4342999999999998e-05 2023-07-02T14:04:34.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797695+0000", 2023-07-02T14:04:34.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0939999999999996e-06 2023-07-02T14:04:34.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:34.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797701+0000", 2023-07-02T14:04:34.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35408540799999999 2023-07-02T14:04:34.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151786+0000", 2023-07-02T14:04:34.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1953000000000006e-05 2023-07-02T14:04:34.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:34.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151868+0000", 2023-07-02T14:04:34.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015162800000000001 2023-07-02T14:04:34.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:34.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.152020+0000", 2023-07-02T14:04:34.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054561870000000004 2023-07-02T14:04:34.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:34.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.157476+0000", 2023-07-02T14:04:34.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059153199999999997 2023-07-02T14:04:34.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:34.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158068+0000", 2023-07-02T14:04:34.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2799e-05 2023-07-02T14:04:34.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:34.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158090+0000", 2023-07-02T14:04:34.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7235999999999999e-05 2023-07-02T14:04:34.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:34.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158108+0000", 2023-07-02T14:04:34.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998 2023-07-02T14:04:34.499 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.499 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.499 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.499 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4110 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:34.499 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:34.500 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.502713221999997, 2023-07-02T14:04:34.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864, 2023-07-02T14:04:34.500 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:34.500 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:34.500 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:34.500 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:34.500 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:34.501 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4110 2023-07-02T14:04:34.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.501 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:34.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:34.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:34.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:34.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:34.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:34.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:34.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:34.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:34.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9331e-05 2023-07-02T14:04:34.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:34.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797684+0000", 2023-07-02T14:04:34.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.147e-06 2023-07-02T14:04:34.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:34.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797689+0000", 2023-07-02T14:04:34.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5020000000000001e-06 2023-07-02T14:04:34.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:34.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797693+0000", 2023-07-02T14:04:34.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5254999999999999e-05 2023-07-02T14:04:34.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797748+0000", 2023-07-02T14:04:34.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5520000000000005e-06 2023-07-02T14:04:34.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:34.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797757+0000", 2023-07-02T14:04:34.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35674245700000001 2023-07-02T14:04:34.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154499+0000", 2023-07-02T14:04:34.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0090000000000001e-06 2023-07-02T14:04:34.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154506+0000", 2023-07-02T14:04:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864 2023-07-02T14:04:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.502661136, 2023-07-02T14:04:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998, 2023-07-02T14:04:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:34.509 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:04:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:34.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:34.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:34.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:34.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:34.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:34.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:34.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:34.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001714819 2023-07-02T14:04:34.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:34.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799412+0000", 2023-07-02T14:04:34.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7779999999999996e-06 2023-07-02T14:04:34.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:34.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799421+0000", 2023-07-02T14:04:34.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3510000000000002e-06 2023-07-02T14:04:34.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:34.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799424+0000", 2023-07-02T14:04:34.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0275999999999999e-05 2023-07-02T14:04:34.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799464+0000", 2023-07-02T14:04:34.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8389999999999996e-06 2023-07-02T14:04:34.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:34.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799473+0000", 2023-07-02T14:04:34.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35507638800000002 2023-07-02T14:04:34.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:34.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154550+0000", 2023-07-02T14:04:34.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6309999999999998e-06 2023-07-02T14:04:34.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:34.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:34.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:34.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154559+0000", 2023-07-02T14:04:34.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998 2023-07-02T14:04:34.518 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.518 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.519 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.519 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:34.519 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:34.519 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:34.520 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:04:34.522 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:04:34.522 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 2 2023-07-02T14:04:34.524 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:04:34.524 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:34.521+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:34.623 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:34.622+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:34.691 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:34.691 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:34.692 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:34.692 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:34.702 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:04:34.723 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:34.723+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:34.824 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:34.824+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:34.867 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:34.867 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:34.867 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:34.867 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:34.868 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:34.877 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:04:34.925 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:34.925+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:35.025 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:35.026+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:35.043 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:35.043 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:35.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:35.044 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:35.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.044 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:198 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:35.044 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:35.044 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.755446819, 2023-07-02T14:04:35.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999, 2023-07-02T14:04:35.044 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 198 2023-07-02T14:04:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.045 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:35.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286031+0000", 2023-07-02T14:04:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00049286700000000002 2023-07-02T14:04:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:35.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286524+0000", 2023-07-02T14:04:35.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4938e-05 2023-07-02T14:04:35.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:35.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286539+0000", 2023-07-02T14:04:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0680000000000002e-06 2023-07-02T14:04:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286545+0000", 2023-07-02T14:04:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026497718 2023-07-02T14:04:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313043+0000", 2023-07-02T14:04:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0884999999999998e-05 2023-07-02T14:04:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313094+0000", 2023-07-02T14:04:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014109999999999999 2023-07-02T14:04:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313235+0000", 2023-07-02T14:04:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048258980000000003 2023-07-02T14:04:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:35.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318061+0000", 2023-07-02T14:04:35.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014346350000000001 2023-07-02T14:04:35.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:35.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319495+0000", 2023-07-02T14:04:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1986e-05 2023-07-02T14:04:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319547+0000", 2023-07-02T14:04:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0457e-05 2023-07-02T14:04:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319578+0000", 2023-07-02T14:04:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999 2023-07-02T14:04:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:207 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.75463691100001, 2023-07-02T14:04:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003, 2023-07-02T14:04:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:35.055 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 207 2023-07-02T14:04:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:35.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286841+0000", 2023-07-02T14:04:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00068375100000000002 2023-07-02T14:04:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287525+0000", 2023-07-02T14:04:35.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1601e-05 2023-07-02T14:04:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287537+0000", 2023-07-02T14:04:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6409999999999996e-06 2023-07-02T14:04:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287543+0000", 2023-07-02T14:04:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027245844000000002 2023-07-02T14:04:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314789+0000", 2023-07-02T14:04:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8910000000000003e-05 2023-07-02T14:04:35.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314858+0000", 2023-07-02T14:04:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030992599999999997 2023-07-02T14:04:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:35.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315168+0000", 2023-07-02T14:04:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029122869999999999 2023-07-02T14:04:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318080+0000", 2023-07-02T14:04:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019963020000000001 2023-07-02T14:04:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320077+0000", 2023-07-02T14:04:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5116000000000005e-05 2023-07-02T14:04:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320142+0000", 2023-07-02T14:04:35.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3793000000000004e-05 2023-07-02T14:04:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320205+0000", 2023-07-02T14:04:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003 2023-07-02T14:04:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.064 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:236 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2424832~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.753924728, 2023-07-02T14:04:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003, 2023-07-02T14:04:35.065 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2023-07-02T14:04:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:35.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:35.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287553+0000", 2023-07-02T14:04:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0667e-05 2023-07-02T14:04:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:35.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287614+0000", 2023-07-02T14:04:35.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0454e-05 2023-07-02T14:04:35.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:35.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287624+0000", 2023-07-02T14:04:35.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8564999999999999e-05 2023-07-02T14:04:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287643+0000", 2023-07-02T14:04:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027977596 2023-07-02T14:04:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315621+0000", 2023-07-02T14:04:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000201728 2023-07-02T14:04:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315822+0000", 2023-07-02T14:04:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018436599999999999 2023-07-02T14:04:35.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316007+0000", 2023-07-02T14:04:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042423440000000003 2023-07-02T14:04:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:35.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320249+0000", 2023-07-02T14:04:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001452633 2023-07-02T14:04:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321702+0000", 2023-07-02T14:04:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3616000000000003e-05 2023-07-02T14:04:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321755+0000", 2023-07-02T14:04:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840999999999999e-05 2023-07-02T14:04:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321793+0000", 2023-07-02T14:04:35.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003 2023-07-02T14:04:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:251 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:35.075 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.75382919099999, 2023-07-02T14:04:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001, 2023-07-02T14:04:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:35.076 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 251 2023-07-02T14:04:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287649+0000", 2023-07-02T14:04:35.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3232000000000001e-05 2023-07-02T14:04:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287702+0000", 2023-07-02T14:04:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0010000000000007e-06 2023-07-02T14:04:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287711+0000", 2023-07-02T14:04:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3569999999999997e-06 2023-07-02T14:04:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287717+0000", 2023-07-02T14:04:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028686454 2023-07-02T14:04:35.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316403+0000", 2023-07-02T14:04:35.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7741000000000002e-05 2023-07-02T14:04:35.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:35.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316461+0000", 2023-07-02T14:04:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4313000000000004e-05 2023-07-02T14:04:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316545+0000", 2023-07-02T14:04:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003734032 2023-07-02T14:04:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320279+0000", 2023-07-02T14:04:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015237219999999999 2023-07-02T14:04:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321803+0000", 2023-07-02T14:04:35.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1927e-05 2023-07-02T14:04:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321845+0000", 2023-07-02T14:04:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5398999999999998e-05 2023-07-02T14:04:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321880+0000", 2023-07-02T14:04:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001 2023-07-02T14:04:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.085 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:255 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2555904~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.753414463, 2023-07-02T14:04:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002, 2023-07-02T14:04:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:35.086 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 255 2023-07-02T14:04:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:35.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288063+0000", 2023-07-02T14:04:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4459999999999997e-05 2023-07-02T14:04:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288118+0000", 2023-07-02T14:04:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0998999999999999e-05 2023-07-02T14:04:35.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288129+0000", 2023-07-02T14:04:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3219999999999998e-06 2023-07-02T14:04:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:35.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288135+0000", 2023-07-02T14:04:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029137386000000001 2023-07-02T14:04:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317273+0000", 2023-07-02T14:04:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7819999999999997e-05 2023-07-02T14:04:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317350+0000", 2023-07-02T14:04:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000174864 2023-07-02T14:04:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:35.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317525+0000", 2023-07-02T14:04:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027943519999999999 2023-07-02T14:04:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320320+0000", 2023-07-02T14:04:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015681460000000001 2023-07-02T14:04:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321888+0000", 2023-07-02T14:04:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9508999999999997e-05 2023-07-02T14:04:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321927+0000", 2023-07-02T14:04:35.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5428e-05 2023-07-02T14:04:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321963+0000", 2023-07-02T14:04:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002 2023-07-02T14:04:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:04:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 93.933803917000006, 2023-07-02T14:04:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:04:35.096 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:35.097 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:35.097 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:35.097 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:35.097 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:35.097 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:04:35.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.097 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:35.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:35.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:04:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:04:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:04:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:04:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:04:35.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:04:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:04:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:04:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:04:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:04:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:04:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:04:35.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:04:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:04:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:35.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:04:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:04:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:04:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:04:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.104 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.105 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.105 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:04:35.105 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:35.105 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 93.933743179000004, 2023-07-02T14:04:35.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:04:35.105 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:35.106 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:35.106 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:35.106 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:35.106 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:35.106 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:04:35.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.106 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:35.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:35.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:35.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:35.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:35.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:35.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:35.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:35.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:04:35.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:04:35.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:04:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:04:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:04:35.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:04:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:04:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:04:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:04:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:04:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:04:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:04:35.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:04:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:04:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:35.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:04:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:04:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:04:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:04:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:04:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:04:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:04:35.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:04:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:04:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:04:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:04:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:04:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.703288849000003, 2023-07-02T14:04:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:04:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:35.117 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:04:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:35.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:04:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:04:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:04:35.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:04:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:04:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:04:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:04:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:04:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:04:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:04:35.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:04:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:04:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:35.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:04:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:04:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:04:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:04:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:04:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:04:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:04:35.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:04:35.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:35.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:04:35.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:04:35.126 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.126 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.126 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.127 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2941 2.3 2:d3762e63:::rbd_data.125de1449605.00000000000000f8:head [write 65536~2031616] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:35.127 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:35.127 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.574185894999999, 2023-07-02T14:04:35.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002, 2023-07-02T14:04:35.127 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:35.127 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:35.128 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:35.128 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:35.128 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:35.128 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2941 2023-07-02T14:04:35.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.128 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:35.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:35.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:35.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:35.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:35.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:35.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:35.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:35.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467291+0000", 2023-07-02T14:04:35.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043022859999999998 2023-07-02T14:04:35.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:35.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471593+0000", 2023-07-02T14:04:35.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0999999999999999e-05 2023-07-02T14:04:35.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:35.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471614+0000", 2023-07-02T14:04:35.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0097e-05 2023-07-02T14:04:35.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:35.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471625+0000", 2023-07-02T14:04:35.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069808499999999998 2023-07-02T14:04:35.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472323+0000", 2023-07-02T14:04:35.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135635 2023-07-02T14:04:35.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:35.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472458+0000", 2023-07-02T14:04:35.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223365 2023-07-02T14:04:35.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:35.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472682+0000", 2023-07-02T14:04:35.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013761944999999999 2023-07-02T14:04:35.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:35.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.486444+0000", 2023-07-02T14:04:35.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016859747000000001 2023-07-02T14:04:35.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:35.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503303+0000", 2023-07-02T14:04:35.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6635000000000001e-05 2023-07-02T14:04:35.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:35.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503380+0000", 2023-07-02T14:04:35.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000161803 2023-07-02T14:04:35.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:35.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503542+0000", 2023-07-02T14:04:35.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002 2023-07-02T14:04:35.136 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.137 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.137 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.137 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2950 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 1507328~786432] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:35.137 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:35.137 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.566012587000003, 2023-07-02T14:04:35.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999, 2023-07-02T14:04:35.138 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:35.138 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:35.138 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:35.138 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:35.138 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:35.138 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2950 2023-07-02T14:04:35.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.139 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:35.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:35.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:35.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:35.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:35.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:35.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:35.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:35.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:35.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001147854 2023-07-02T14:04:35.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:35.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476614+0000", 2023-07-02T14:04:35.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.802e-05 2023-07-02T14:04:35.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:35.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476632+0000", 2023-07-02T14:04:35.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1860000000000003e-06 2023-07-02T14:04:35.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:35.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476639+0000", 2023-07-02T14:04:35.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061211240000000004 2023-07-02T14:04:35.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482760+0000", 2023-07-02T14:04:35.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8833999999999996e-05 2023-07-02T14:04:35.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:35.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482859+0000", 2023-07-02T14:04:35.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019270600000000001 2023-07-02T14:04:35.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:35.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.483051+0000", 2023-07-02T14:04:35.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060385630000000003 2023-07-02T14:04:35.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:35.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489090+0000", 2023-07-02T14:04:35.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024506918999999999 2023-07-02T14:04:35.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:35.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513597+0000", 2023-07-02T14:04:35.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9672999999999995e-05 2023-07-02T14:04:35.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:35.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513667+0000", 2023-07-02T14:04:35.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024329500000000001 2023-07-02T14:04:35.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:35.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513910+0000", 2023-07-02T14:04:35.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999 2023-07-02T14:04:35.147 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.147 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.147 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.147 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2951 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2686976~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:35.148 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:35.148 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.564831808999998, 2023-07-02T14:04:35.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999, 2023-07-02T14:04:35.148 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:35.148 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:35.148 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:35.148 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:35.148 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:35.149 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2951 2023-07-02T14:04:35.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.149 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:35.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:35.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:35.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:35.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:35.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:35.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:35.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:35.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476646+0000", 2023-07-02T14:04:35.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7639000000000003e-05 2023-07-02T14:04:35.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:35.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476684+0000", 2023-07-02T14:04:35.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3508000000000002e-05 2023-07-02T14:04:35.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:35.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476747+0000", 2023-07-02T14:04:35.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8499999999999994e-06 2023-07-02T14:04:35.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:35.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476755+0000", 2023-07-02T14:04:35.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078346160000000008 2023-07-02T14:04:35.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484590+0000", 2023-07-02T14:04:35.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7206000000000004e-05 2023-07-02T14:04:35.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:35.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484687+0000", 2023-07-02T14:04:35.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016243099999999999 2023-07-02T14:04:35.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:35.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484849+0000", 2023-07-02T14:04:35.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042482860000000004 2023-07-02T14:04:35.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:35.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489097+0000", 2023-07-02T14:04:35.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.02489504 2023-07-02T14:04:35.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:35.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513992+0000", 2023-07-02T14:04:35.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9972000000000001e-05 2023-07-02T14:04:35.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:35.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514052+0000", 2023-07-02T14:04:35.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149264 2023-07-02T14:04:35.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:35.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514202+0000", 2023-07-02T14:04:35.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999 2023-07-02T14:04:35.157 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.157 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.157 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.158 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2952 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2883584~12288] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:35.158 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:35.158 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.564717974000004, 2023-07-02T14:04:35.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651, 2023-07-02T14:04:35.158 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:35.158 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:35.159 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:35.159 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:35.159 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:35.159 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2952 2023-07-02T14:04:35.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.159 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:35.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:35.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:35.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:35.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:35.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:35.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:35.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:35.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476760+0000", 2023-07-02T14:04:35.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3664e-05 2023-07-02T14:04:35.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:35.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476774+0000", 2023-07-02T14:04:35.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.166e-06 2023-07-02T14:04:35.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:35.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476783+0000", 2023-07-02T14:04:35.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8720000000000001e-06 2023-07-02T14:04:35.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:35.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476788+0000", 2023-07-02T14:04:35.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087180529999999999 2023-07-02T14:04:35.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485506+0000", 2023-07-02T14:04:35.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5680999999999994e-05 2023-07-02T14:04:35.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:35.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485602+0000", 2023-07-02T14:04:35.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000222365 2023-07-02T14:04:35.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:35.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485824+0000", 2023-07-02T14:04:35.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003277379 2023-07-02T14:04:35.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:35.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489101+0000", 2023-07-02T14:04:35.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025151739999999999 2023-07-02T14:04:35.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:35.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514253+0000", 2023-07-02T14:04:35.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4749000000000002e-05 2023-07-02T14:04:35.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:35.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514288+0000", 2023-07-02T14:04:35.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130634 2023-07-02T14:04:35.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:35.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514419+0000", 2023-07-02T14:04:35.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651 2023-07-02T14:04:35.167 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.168 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.168 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.168 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9620 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 589824~65536] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:04:35.168 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:04:35.168 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.019274251000001, 2023-07-02T14:04:35.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001, 2023-07-02T14:04:35.169 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:35.169 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:35.169 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:35.169 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:35.169 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:35.169 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9620 2023-07-02T14:04:35.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.170 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:35.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:35.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:04:35.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:35.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:35.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:04:35.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:35.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:35.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022203+0000", 2023-07-02T14:04:35.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0346000000000001e-05 2023-07-02T14:04:35.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:35.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022263+0000", 2023-07-02T14:04:35.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4657999999999999e-05 2023-07-02T14:04:35.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:35.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022278+0000", 2023-07-02T14:04:35.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:04:35.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:35.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022285+0000", 2023-07-02T14:04:35.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0096000000000003e-05 2023-07-02T14:04:35.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022335+0000", 2023-07-02T14:04:35.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2334000000000001e-05 2023-07-02T14:04:35.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:35.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022347+0000", 2023-07-02T14:04:35.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095653701999999993 2023-07-02T14:04:35.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118001+0000", 2023-07-02T14:04:35.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000185797 2023-07-02T14:04:35.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:35.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118187+0000", 2023-07-02T14:04:35.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026397900000000001 2023-07-02T14:04:35.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:35.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118451+0000", 2023-07-02T14:04:35.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051832800000000002 2023-07-02T14:04:35.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:35.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.123634+0000", 2023-07-02T14:04:35.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001227332 2023-07-02T14:04:35.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:35.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124861+0000", 2023-07-02T14:04:35.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9493999999999998e-05 2023-07-02T14:04:35.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:35.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124901+0000", 2023-07-02T14:04:35.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0462000000000001e-05 2023-07-02T14:04:35.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:35.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124931+0000", 2023-07-02T14:04:35.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001 2023-07-02T14:04:35.179 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.179 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.179 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.180 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9621 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 655360~1966080] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:04:35.180 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:04:35.180 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.019086869000001, 2023-07-02T14:04:35.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999, 2023-07-02T14:04:35.180 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:35.180 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:35.180 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:35.181 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:35.181 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:35.181 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9621 2023-07-02T14:04:35.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.181 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:35.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:35.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:04:35.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:35.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:35.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:04:35.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:35.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:35.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022391+0000", 2023-07-02T14:04:35.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032053939999999999 2023-07-02T14:04:35.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:35.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025597+0000", 2023-07-02T14:04:35.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4307999999999999e-05 2023-07-02T14:04:35.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:35.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025611+0000", 2023-07-02T14:04:35.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.032e-06 2023-07-02T14:04:35.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:35.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025617+0000", 2023-07-02T14:04:35.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8977999999999998e-05 2023-07-02T14:04:35.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025656+0000", 2023-07-02T14:04:35.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:04:35.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:35.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025663+0000", 2023-07-02T14:04:35.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095220739999999998 2023-07-02T14:04:35.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120884+0000", 2023-07-02T14:04:35.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2525000000000001e-05 2023-07-02T14:04:35.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:35.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120956+0000", 2023-07-02T14:04:35.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016232400000000001 2023-07-02T14:04:35.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:35.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.121119+0000", 2023-07-02T14:04:35.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044007380000000004 2023-07-02T14:04:35.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:35.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.125519+0000", 2023-07-02T14:04:35.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000797209 2023-07-02T14:04:35.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:35.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126317+0000", 2023-07-02T14:04:35.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1228000000000004e-05 2023-07-02T14:04:35.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:35.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126378+0000", 2023-07-02T14:04:35.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978000000000003e-05 2023-07-02T14:04:35.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:35.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126430+0000", 2023-07-02T14:04:35.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999 2023-07-02T14:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:11434 2.1 2:8269d003:::rbd_data.125de1449605.00000000000000ce:head [write 65536~4128768] snapc 1a=[1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e107)", 2023-07-02T14:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 6.1236486000000001, 2023-07-02T14:04:35.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036409966000000002, 2023-07-02T14:04:35.192 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:35.192 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:35.192 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:35.192 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:35.192 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:35.192 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11434 2023-07-02T14:04:35.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.193 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:35.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:35.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:04:35.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:35.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:35.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:04:35.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:35.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:35.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917828+0000", 2023-07-02T14:04:35.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052044650000000001 2023-07-02T14:04:35.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:35.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923033+0000", 2023-07-02T14:04:35.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0097000000000001e-05 2023-07-02T14:04:35.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:35.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923053+0000", 2023-07-02T14:04:35.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.49e-06 2023-07-02T14:04:35.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:35.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923061+0000", 2023-07-02T14:04:35.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0090999999999998e-05 2023-07-02T14:04:35.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923121+0000", 2023-07-02T14:04:35.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000114095 2023-07-02T14:04:35.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:35.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923236+0000", 2023-07-02T14:04:35.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020094199999999999 2023-07-02T14:04:35.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:35.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923436+0000", 2023-07-02T14:04:35.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088693520000000005 2023-07-02T14:04:35.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:35.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.932306+0000", 2023-07-02T14:04:35.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021874112000000001 2023-07-02T14:04:35.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:35.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954180+0000", 2023-07-02T14:04:35.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4433e-05 2023-07-02T14:04:35.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:35.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954214+0000", 2023-07-02T14:04:35.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5884000000000002e-05 2023-07-02T14:04:35.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:35.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954240+0000", 2023-07-02T14:04:35.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036409966000000002 2023-07-02T14:04:35.201 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.201 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.201 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.201 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:189 2.3 2:c9621172:::rbd_data.125de1449605.00000000000000c0:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:04:35.202 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:04:35.202 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.8687995770000001, 2023-07-02T14:04:35.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001, 2023-07-02T14:04:35.202 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:35.202 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:35.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:35.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:04:35.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:04:35.203 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:04:35.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.203 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:35.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:35.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:04:35.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:35.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:35.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:04:35.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:35.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:35.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:04:35.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2023-07-02T14:04:35.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:35.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172682+0000", 2023-07-02T14:04:35.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0159999999999997e-06 2023-07-02T14:04:35.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:35.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172688+0000", 2023-07-02T14:04:35.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0010000000000003e-06 2023-07-02T14:04:35.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:35.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172692+0000", 2023-07-02T14:04:35.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026444333 2023-07-02T14:04:35.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199136+0000", 2023-07-02T14:04:35.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2230000000000004e-05 2023-07-02T14:04:35.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:35.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199218+0000", 2023-07-02T14:04:35.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012043595000000001 2023-07-02T14:04:35.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:35.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211262+0000", 2023-07-02T14:04:35.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001 2023-07-02T14:04:35.209 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.209 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.209 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.209 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:199 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:04:35.209 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:04:35.209 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.868756034, 2023-07-02T14:04:35.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002, 2023-07-02T14:04:35.210 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:35.210 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:35.210 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:35.210 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:04:35.210 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:04:35.210 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 199 2023-07-02T14:04:35.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.211 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:35.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:35.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:04:35.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:35.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:35.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:04:35.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:35.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:35.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172722+0000", 2023-07-02T14:04:35.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1250000000000001e-06 2023-07-02T14:04:35.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:35.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172725+0000", 2023-07-02T14:04:35.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5670000000000002e-06 2023-07-02T14:04:35.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:35.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172731+0000", 2023-07-02T14:04:35.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9310000000000001e-06 2023-07-02T14:04:35.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:35.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172735+0000", 2023-07-02T14:04:35.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030699852999999999 2023-07-02T14:04:35.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203435+0000", 2023-07-02T14:04:35.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6827e-05 2023-07-02T14:04:35.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:35.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203532+0000", 2023-07-02T14:04:35.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076955230000000001 2023-07-02T14:04:35.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:35.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211227+0000", 2023-07-02T14:04:35.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002 2023-07-02T14:04:35.216 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.217 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.217 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.217 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:213 2.1 2:8fbcc55d:::rbd_data.125de1449605.00000000000000d5:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e110)", 2023-07-02T14:04:35.217 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.241083+0000", 2023-07-02T14:04:35.217 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.800395448, 2023-07-02T14:04:35.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036164654999999997, 2023-07-02T14:04:35.218 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:35.218 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:35.218 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:35.218 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:04:35.218 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:04:35.218 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 213 2023-07-02T14:04:35.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.218 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:35.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:35.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.241083+0000", 2023-07-02T14:04:35.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:35.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:35.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.241083+0000", 2023-07-02T14:04:35.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:35.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:35.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.241083+0000", 2023-07-02T14:04:35.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.525e-06 2023-07-02T14:04:35.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:35.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.241086+0000", 2023-07-02T14:04:35.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0620000000000003e-06 2023-07-02T14:04:35.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:35.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.241093+0000", 2023-07-02T14:04:35.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.799e-06 2023-07-02T14:04:35.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:35.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.241096+0000", 2023-07-02T14:04:35.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030109975000000001 2023-07-02T14:04:35.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.271206+0000", 2023-07-02T14:04:35.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134172 2023-07-02T14:04:35.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:35.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.271341+0000", 2023-07-02T14:04:35.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059075309999999997 2023-07-02T14:04:35.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:35.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.277248+0000", 2023-07-02T14:04:35.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036164654999999997 2023-07-02T14:04:35.224 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.224 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.224 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.225 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:280 2.1 2:94df0216:::rbd_data.125de1449605.0000000000000117:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:04:35.225 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555448+0000", 2023-07-02T14:04:35.225 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.486030419, 2023-07-02T14:04:35.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989515, 2023-07-02T14:04:35.225 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:35.225 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:35.226 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:35.226 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:04:35.226 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:04:35.226 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 280 2023-07-02T14:04:35.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.226 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:35.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:35.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555448+0000", 2023-07-02T14:04:35.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:35.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:35.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555448+0000", 2023-07-02T14:04:35.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:35.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:35.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555448+0000", 2023-07-02T14:04:35.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1400000000000002e-06 2023-07-02T14:04:35.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:35.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555452+0000", 2023-07-02T14:04:35.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8199999999999999e-06 2023-07-02T14:04:35.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:35.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555459+0000", 2023-07-02T14:04:35.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7520000000000004e-06 2023-07-02T14:04:35.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:35.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555463+0000", 2023-07-02T14:04:35.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027814050999999999 2023-07-02T14:04:35.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.583277+0000", 2023-07-02T14:04:35.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192244 2023-07-02T14:04:35.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:35.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.583470+0000", 2023-07-02T14:04:35.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007968269 2023-07-02T14:04:35.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:35.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591438+0000", 2023-07-02T14:04:35.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989515 2023-07-02T14:04:35.232 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.232 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.232 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.233 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:281 2.1 2:901d5a95:::rbd_data.125de1449605.0000000000000118:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:04:35.233 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:04:35.233 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.4860107579999999, 2023-07-02T14:04:35.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002, 2023-07-02T14:04:35.233 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:35.233 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:35.233 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:35.233 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:04:35.234 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:04:35.234 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:04:35.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.234 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:35.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:35.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:04:35.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:35.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:35.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:04:35.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:35.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:35.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:04:35.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0469999999999998e-06 2023-07-02T14:04:35.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:35.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555471+0000", 2023-07-02T14:04:35.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8860000000000001e-06 2023-07-02T14:04:35.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:35.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555476+0000", 2023-07-02T14:04:35.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3160000000000002e-06 2023-07-02T14:04:35.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:35.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555483+0000", 2023-07-02T14:04:35.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989525000000001 2023-07-02T14:04:35.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:35.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591472+0000", 2023-07-02T14:04:35.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3708999999999996e-05 2023-07-02T14:04:35.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:35.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591536+0000", 2023-07-02T14:04:35.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055109720000000003 2023-07-02T14:04:35.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:35.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:35.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:35.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.597047+0000", 2023-07-02T14:04:35.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002 2023-07-02T14:04:35.240 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.240 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.240 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.240 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:35.240 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:35.240 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:35.241 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:04:35.242 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:04:35.243 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:35.242+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:35.342 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:04:35.342 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 2] out_osds: [5] dead_osds: [3] live_osds: [0, 1, 2, 4, 5] 2023-07-02T14:04:35.343 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:04:35.343 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2023-07-02T14:04:35.343 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-07-02T14:04:35.343 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-07-02T14:04:35.343 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2023-07-02T14:04:35.344 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:35.343+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:35.380 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:35.380 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:35.380 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:04:35.380 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:35.390 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:04:35.426 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:35.443 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:35.444 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:35.443+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:35.461 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:35.470 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:35.470 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "" 2023-07-02T14:04:35.470 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:35.479 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:35.497 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:35.497 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:04:35.497 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:04:35.498 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:04:35.498 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:35.498 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:04:35.498 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:04:35.499 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:35.500 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:35.544 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:35.545+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:35.555 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:35.555 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:35.565 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:04:35.646 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:35.645+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:35.728 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:35.728 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:35.738 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:04:35.780 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:35.781+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:35.848 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:35.848+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:35.861 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:35.861 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:35.871 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:04:35.948 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:35.948+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:36.035 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:36.035 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:36.045 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:36.049 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:36.049+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:36.150 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:36.150+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:36.169 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:36.169 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:36.179 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:36.251 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:36.251+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:36.322 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:36.322 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:36.338 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:04:36.377 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap24 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap25 b3fbb028-e267-4d25-a750-93ba18a2316a@snap25.diff 2023-07-02T14:04:36.379 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:36.379+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:36.434 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete...2023-07-02T14:04:36.434+0000 7fea3661e700 0 -- 172.21.15.163:0/2433849008 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x55c011293ed0 msgr2=0x55c011296300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-07-02T14:04:36.436 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:36.436+0000 7fea3661e700 0 -- 172.21.15.163:0/2433849008 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7fea1c036bb0 msgr2=0x7fea1c038fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:36.452 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:36.452+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:36.522 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:36.523 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:36.533 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:04:36.547 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 100% complete...done. 2023-07-02T14:04:36.550 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:04:36.550 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=26 2023-07-02T14:04:36.551 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap26 2023-07-02T14:04:36.554 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:36.554+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:36.654 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:36.654+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:36.697 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:36.697 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:36.707 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:04:36.790 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:36.790+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:36.855 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:36.856+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:36.871 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:36.871 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:36.881 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:04:36.956 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:36.956+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:37.045 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:37.045 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:37.055 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:04:37.057 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:37.057+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:37.157 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:37.157+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:37.178 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:37.178 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:37.189 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:04:37.257 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:37.257+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:37.313 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:37.313 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:37.323 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:04:37.355 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:04:37.358 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:37.358+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:37.371 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:04:37.376 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 2 2023-07-02T14:04:37.459 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:37.459+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:37.490 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:37.490 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:37.501 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:04:37.501 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:04:37.585 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:37.585+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:37.660 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:37.661+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:37.677 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:37.677 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:37.688 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:04:37.689 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:37.707 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:37.724 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:37.744 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:37.761 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:37.761+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:37.762 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:37.762 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:04:37.763 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:04:37.763 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:04:37.763 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:37.763 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:04:37.763 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:04:37.765 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:37.766 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:37.827 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:37.827 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:37.828 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:04:37.828 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:37.838 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:04:37.862 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:37.862+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:37.963 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:37.963+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:38.002 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:38.002 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:38.002 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:04:38.003 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:04:38.003 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:38.012 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:04:38.096 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:38.096+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:38.165 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:38.165+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:38.178 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:38.178 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:04:38.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:04:38.179 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:38.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.179 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:04:38.179 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:38.179 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 58.279524887999997, 2023-07-02T14:04:38.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:04:38.180 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.180 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.180 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:38.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:38.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:38.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:04:38.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:04:38.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:04:38.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:04:38.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:04:38.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:04:38.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:04:38.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:04:38.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:04:38.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:04:38.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:04:38.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:04:38.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:04:38.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:04:38.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:04:38.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:04:38.187 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.187 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.187 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.187 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:04:38.188 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:38.188 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 58.275439169999999, 2023-07-02T14:04:38.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:04:38.188 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.188 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.188 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:38.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:38.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:38.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:04:38.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:04:38.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:04:38.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:04:38.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:04:38.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:04:38.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:04:38.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:04:38.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:04:38.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:04:38.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:04:38.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:04:38.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:04:38.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:04:38.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:04:38.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:04:38.195 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.195 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.195 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.195 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:04:38.196 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:38.196 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 58.271574389000001, 2023-07-02T14:04:38.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:04:38.196 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.196 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.196 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:38.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:38.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:38.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:04:38.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:04:38.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:04:38.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:04:38.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.199 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:04:38.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:04:38.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:04:38.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:04:38.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:04:38.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:04:38.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:04:38.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:04:38.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:04:38.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:04:38.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:04:38.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:04:38.203 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.203 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.203 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.203 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:04:38.203 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:38.204 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 58.268050127999999, 2023-07-02T14:04:38.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:04:38.204 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.204 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.204 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:38.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:38.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:38.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:04:38.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:04:38.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:04:38.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:04:38.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:04:38.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:04:38.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:04:38.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:04:38.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:04:38.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:04:38.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:04:38.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:04:38.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:04:38.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:04:38.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:04:38.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:04:38.211 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.211 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.211 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.211 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6124 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1328, mlcod=81'1319)", 2023-07-02T14:04:38.211 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:38.212 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 58.264534744999999, 2023-07-02T14:04:38.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019096974999999999, 2023-07-02T14:04:38.212 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.212 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.212 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:38.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:38.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:38.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912602+0000", 2023-07-02T14:04:38.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6741999999999999e-05 2023-07-02T14:04:38.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912629+0000", 2023-07-02T14:04:38.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5163e-05 2023-07-02T14:04:38.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912644+0000", 2023-07-02T14:04:38.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4800000000000001e-06 2023-07-02T14:04:38.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912653+0000", 2023-07-02T14:04:38.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071988909999999998 2023-07-02T14:04:38.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.919851+0000", 2023-07-02T14:04:38.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2949999999999999e-05 2023-07-02T14:04:38.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.919874+0000", 2023-07-02T14:04:38.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011800804 2023-07-02T14:04:38.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931675+0000", 2023-07-02T14:04:38.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6033999999999998e-05 2023-07-02T14:04:38.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931701+0000", 2023-07-02T14:04:38.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019096974999999999 2023-07-02T14:04:38.219 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.219 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.219 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.219 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6139 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1337, mlcod=81'1333)", 2023-07-02T14:04:38.219 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:38.219 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 58.237844885000001, 2023-07-02T14:04:38.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018571681, 2023-07-02T14:04:38.220 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.220 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.220 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:38.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.221 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:38.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:38.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939293+0000", 2023-07-02T14:04:38.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003764613 2023-07-02T14:04:38.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943058+0000", 2023-07-02T14:04:38.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5797000000000001e-05 2023-07-02T14:04:38.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943074+0000", 2023-07-02T14:04:38.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5089999999999996e-06 2023-07-02T14:04:38.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943081+0000", 2023-07-02T14:04:38.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014727100000000001 2023-07-02T14:04:38.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943228+0000", 2023-07-02T14:04:38.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2246000000000003e-05 2023-07-02T14:04:38.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943261+0000", 2023-07-02T14:04:38.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014576317 2023-07-02T14:04:38.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957837+0000", 2023-07-02T14:04:38.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8962000000000002e-05 2023-07-02T14:04:38.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957866+0000", 2023-07-02T14:04:38.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018571681 2023-07-02T14:04:38.226 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.226 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.227 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.227 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:04:38.227 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:38.227 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 58.232345328999997, 2023-07-02T14:04:38.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:04:38.227 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.228 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.228 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:38.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:38.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:38.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.230 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:04:38.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:04:38.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.230 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.231 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:04:38.231 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:04:38.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.231 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:04:38.231 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:04:38.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:04:38.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:04:38.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:04:38.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:04:38.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:04:38.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:04:38.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:04:38.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:04:38.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:04:38.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:04:38.235 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.235 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.235 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.236 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:04:38.236 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:38.236 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 58.227687971000002, 2023-07-02T14:04:38.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:04:38.236 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.237 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.237 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:38.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:38.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:38.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:04:38.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:04:38.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:04:38.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:04:38.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:04:38.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:04:38.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.240 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:04:38.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:04:38.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:04:38.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:04:38.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:04:38.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:04:38.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:04:38.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:04:38.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:04:38.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:04:38.243 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.243 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.243 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.244 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:04:38.244 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:38.244 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 58.226748108000002, 2023-07-02T14:04:38.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:04:38.244 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.244 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.245 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.245 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:38.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:38.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:38.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.246 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.246 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:04:38.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:04:38.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:04:38.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:04:38.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:04:38.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:04:38.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:04:38.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:04:38.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:04:38.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:04:38.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:04:38.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:04:38.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.251 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.251 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:04:38.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:04:38.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.252 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.253 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:04:38.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:04:38.253 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.254 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.254 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.255 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:04:38.255 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:38.255 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 58.225350229999997, 2023-07-02T14:04:38.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:04:38.255 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.255 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.256 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:38.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:38.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:38.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:04:38.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:04:38.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:04:38.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:04:38.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:04:38.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:04:38.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:04:38.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:04:38.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:04:38.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:04:38.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.261 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:04:38.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:04:38.261 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.261 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.261 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:04:38.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:04:38.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.262 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.262 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.262 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:04:38.262 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:04:38.262 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.262 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.262 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.263 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6150 2.0 e81/46 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 81'1345, mlcod=81'1338)", 2023-07-02T14:04:38.263 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:38.263 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 58.217327283000003, 2023-07-02T14:04:38.263 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019172812000000001, 2023-07-02T14:04:38.263 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.263 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.264 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.264 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:38.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.265 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:38.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:38.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.265 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959810+0000", 2023-07-02T14:04:38.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048438650000000001 2023-07-02T14:04:38.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964653+0000", 2023-07-02T14:04:38.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2136e-05 2023-07-02T14:04:38.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964666+0000", 2023-07-02T14:04:38.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2169999999999999e-06 2023-07-02T14:04:38.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964672+0000", 2023-07-02T14:04:38.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00047561800000000003 2023-07-02T14:04:38.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.268 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965147+0000", 2023-07-02T14:04:38.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5891e-05 2023-07-02T14:04:38.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965173+0000", 2023-07-02T14:04:38.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013770973000000001 2023-07-02T14:04:38.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978944+0000", 2023-07-02T14:04:38.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0256999999999998e-05 2023-07-02T14:04:38.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978985+0000", 2023-07-02T14:04:38.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019172812000000001 2023-07-02T14:04:38.270 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.270 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.270 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.271 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:04:38.271 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:38.271 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 36.815159733999998, 2023-07-02T14:04:38.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:04:38.271 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.271 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.271 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:38.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:38.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:38.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.273 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:04:38.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:04:38.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:04:38.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:04:38.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:04:38.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:04:38.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.275 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.275 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:04:38.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:04:38.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:04:38.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:04:38.276 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:04:38.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:04:38.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.277 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.277 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:04:38.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:04:38.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:04:38.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:04:38.278 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.278 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.278 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.279 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:38.279 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:38.279 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.448810049999999, 2023-07-02T14:04:38.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:04:38.279 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.279 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.279 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.279 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.280 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.280 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:04:38.280 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.280 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:38.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:38.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:38.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:04:38.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:04:38.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.282 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:04:38.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:04:38.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.283 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:04:38.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:04:38.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:04:38.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:04:38.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:04:38.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:04:38.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:04:38.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:04:38.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:04:38.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:04:38.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:04:38.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:04:38.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:04:38.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:04:38.287 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.287 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.287 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.288 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7706 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 0~53248] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:38.288 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:38.288 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.444883302999997, 2023-07-02T14:04:38.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019507096000000002, 2023-07-02T14:04:38.288 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.288 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.289 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.289 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.289 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.289 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7706 2023-07-02T14:04:38.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.289 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:38.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:38.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:38.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732254+0000", 2023-07-02T14:04:38.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4764999999999997e-05 2023-07-02T14:04:38.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732289+0000", 2023-07-02T14:04:38.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0166999999999999e-05 2023-07-02T14:04:38.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732299+0000", 2023-07-02T14:04:38.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5310000000000001e-06 2023-07-02T14:04:38.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732305+0000", 2023-07-02T14:04:38.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060619599999999999 2023-07-02T14:04:38.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738367+0000", 2023-07-02T14:04:38.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011864 2023-07-02T14:04:38.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.294 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738486+0000", 2023-07-02T14:04:38.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013098291 2023-07-02T14:04:38.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.295 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751584+0000", 2023-07-02T14:04:38.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7800000000000002e-05 2023-07-02T14:04:38.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.295 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751642+0000", 2023-07-02T14:04:38.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000121198 2023-07-02T14:04:38.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751763+0000", 2023-07-02T14:04:38.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019507096000000002 2023-07-02T14:04:38.296 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.296 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.297 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.297 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7707 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 3145728~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:38.297 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:38.297 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.444826583000001, 2023-07-02T14:04:38.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999, 2023-07-02T14:04:38.297 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.298 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.298 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.298 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.298 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.298 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7707 2023-07-02T14:04:38.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.298 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:38.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:38.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:38.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:38.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5017e-05 2023-07-02T14:04:38.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732347+0000", 2023-07-02T14:04:38.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5821e-05 2023-07-02T14:04:38.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732363+0000", 2023-07-02T14:04:38.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1270000000000002e-06 2023-07-02T14:04:38.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732368+0000", 2023-07-02T14:04:38.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008444666 2023-07-02T14:04:38.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740812+0000", 2023-07-02T14:04:38.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011266599999999999 2023-07-02T14:04:38.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740925+0000", 2023-07-02T14:04:38.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010848808999999999 2023-07-02T14:04:38.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751774+0000", 2023-07-02T14:04:38.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8421999999999997e-05 2023-07-02T14:04:38.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751832+0000", 2023-07-02T14:04:38.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107453 2023-07-02T14:04:38.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751940+0000", 2023-07-02T14:04:38.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999 2023-07-02T14:04:38.306 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.306 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.306 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.306 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:38.306 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:38.306 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.444762705999999, 2023-07-02T14:04:38.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:04:38.307 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.307 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.307 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.307 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.307 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.307 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:04:38.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.308 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:38.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:38.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:38.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:38.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:04:38.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:04:38.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:04:38.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:04:38.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:04:38.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:04:38.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:04:38.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:04:38.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:04:38.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:04:38.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:04:38.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:04:38.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:04:38.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:04:38.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:04:38.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:04:38.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:04:38.315 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.315 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.315 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.315 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:38.315 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:38.316 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.440198881000001, 2023-07-02T14:04:38.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:04:38.316 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.316 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.316 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.316 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.316 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.316 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:04:38.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.317 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:38.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:38.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:38.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:04:38.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:04:38.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:04:38.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:04:38.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:04:38.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:04:38.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:04:38.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:04:38.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:04:38.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:04:38.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:04:38.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:04:38.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.322 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:04:38.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:04:38.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:04:38.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:04:38.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:04:38.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:04:38.324 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.324 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.324 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.324 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:38.324 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:38.325 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.440149015000003, 2023-07-02T14:04:38.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:04:38.325 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.325 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.325 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.325 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.325 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:04:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:04:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:04:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:04:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:04:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:04:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:04:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:04:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:04:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:04:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:04:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:04:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:04:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:04:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:04:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:04:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:04:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:04:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:04:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.439110022000001, 2023-07-02T14:04:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:04:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:04:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:04:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:04:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:04:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:04:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:04:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:04:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:04:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:04:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:04:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:04:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:04:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:04:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:04:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:04:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:04:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:04:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:04:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:04:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7993 2.7 2:fe72db53:::rbd_data.125de1449605.0000000000000072:head [write 65536~3276800] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.252151974, 2023-07-02T14:04:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919, 2023-07-02T14:04:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7993 2023-07-02T14:04:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924986+0000", 2023-07-02T14:04:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049542070000000004 2023-07-02T14:04:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929940+0000", 2023-07-02T14:04:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9740000000000001e-05 2023-07-02T14:04:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929960+0000", 2023-07-02T14:04:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.311e-05 2023-07-02T14:04:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929973+0000", 2023-07-02T14:04:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9629000000000002e-05 2023-07-02T14:04:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930023+0000", 2023-07-02T14:04:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012751799999999999 2023-07-02T14:04:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930150+0000", 2023-07-02T14:04:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016449042000000001 2023-07-02T14:04:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946599+0000", 2023-07-02T14:04:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3241e-05 2023-07-02T14:04:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946663+0000", 2023-07-02T14:04:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000102296 2023-07-02T14:04:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946765+0000", 2023-07-02T14:04:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919 2023-07-02T14:04:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.351 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:38.352 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:04:38.353 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:04:38.354 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:38.353+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:38.453 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:38.453+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:38.517 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:38.518 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:38.518 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:04:38.518 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:38.527 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:04:38.554 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:38.554+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:38.651 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:38.651 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:38.651 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:04:38.651 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:04:38.651 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:38.654 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:38.654+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:38.661 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:38.755 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:38.755+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:38.827 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:38.827 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:04:38.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:04:38.828 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:38.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.828 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:38.828 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:38.828 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.534609684, 2023-07-02T14:04:38.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:04:38.828 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.829 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.829 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.829 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.829 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.829 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:04:38.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.829 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:38.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:04:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:04:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:04:38.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:04:38.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:04:38.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:04:38.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:04:38.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:04:38.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:04:38.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:04:38.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:04:38.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:04:38.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:38.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:04:38.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:04:38.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:38.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:04:38.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:04:38.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:04:38.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:04:38.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:04:38.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:04:38.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:04:38.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:04:38.838 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.838 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.838 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.838 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:38.838 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:38.839 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.534374661, 2023-07-02T14:04:38.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:04:38.839 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.839 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.839 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.839 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.839 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.839 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:04:38.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.840 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:38.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:38.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:38.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:04:38.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:04:38.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:04:38.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:04:38.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:04:38.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:04:38.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:04:38.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:04:38.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:04:38.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:04:38.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:04:38.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:04:38.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:38.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:04:38.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:04:38.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:38.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:04:38.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:04:38.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:04:38.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:04:38.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:04:38.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:04:38.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:04:38.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:04:38.848 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.848 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.848 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.849 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:38.849 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:38.849 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.534175527, 2023-07-02T14:04:38.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:04:38.849 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.849 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.849 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.849 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.850 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.850 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:04:38.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.850 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:38.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:38.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:38.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:04:38.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:04:38.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:04:38.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:04:38.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:04:38.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:04:38.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:04:38.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:04:38.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:04:38.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:04:38.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:04:38.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:04:38.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:38.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:04:38.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:04:38.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:38.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:04:38.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:04:38.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:04:38.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:04:38.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:04:38.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:04:38.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:04:38.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:04:38.862 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.862 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.862 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.862 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:38.862 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:38.862 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.533810403, 2023-07-02T14:04:38.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:04:38.863 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.863 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.863 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.863 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.863 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.863 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:04:38.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.864 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:38.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:38.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:38.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:04:38.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:04:38.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:04:38.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:04:38.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:04:38.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:04:38.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:04:38.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:04:38.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:04:38.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:04:38.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:04:38.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:04:38.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:38.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:04:38.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:04:38.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:38.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:04:38.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:04:38.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:04:38.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:04:38.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:04:38.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:04:38.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:04:38.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:04:38.872 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.872 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.872 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.873 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:38.873 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:38.873 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.533729342, 2023-07-02T14:04:38.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:04:38.873 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.873 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.873 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.874 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.874 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.874 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:04:38.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.874 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:38.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:38.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:38.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:04:38.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:04:38.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:04:38.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:04:38.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:04:38.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:04:38.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:04:38.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:04:38.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:04:38.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:04:38.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:04:38.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:04:38.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:38.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:04:38.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:04:38.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:38.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:04:38.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:04:38.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:04:38.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:04:38.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:04:38.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:04:38.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:04:38.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:04:38.883 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.883 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.883 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.884 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:38.884 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:38.885 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.53363767099999, 2023-07-02T14:04:38.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:04:38.885 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.885 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.885 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.886 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.886 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.886 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:04:38.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.886 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:38.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:38.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:38.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:04:38.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:04:38.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:04:38.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:04:38.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:04:38.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:04:38.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:04:38.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:04:38.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.893 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:04:38.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:04:38.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:04:38.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:04:38.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:38.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:04:38.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:04:38.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:04:38.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:04:38.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:38.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:04:38.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:04:38.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:04:38.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:04:38.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:04:38.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:04:38.901 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.901 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.901 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.901 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:38.902 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:38.902 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.533298229, 2023-07-02T14:04:38.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:04:38.902 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.902 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.902 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.902 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.902 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.903 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:04:38.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.903 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:38.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:38.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:38.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:04:38.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:04:38.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:04:38.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:04:38.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:04:38.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:04:38.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:04:38.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:04:38.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:04:38.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:04:38.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:04:38.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:04:38.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:38.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:04:38.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:04:38.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:04:38.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:04:38.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:38.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:04:38.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:04:38.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:04:38.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:04:38.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:04:38.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:04:38.912 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.912 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.912 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.912 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:38.912 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:38.913 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.533112176, 2023-07-02T14:04:38.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:04:38.913 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.913 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.913 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.913 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.913 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.913 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:04:38.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.914 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:38.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:38.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:38.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:38.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:04:38.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:04:38.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:04:38.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:04:38.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:04:38.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:04:38.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:04:38.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:04:38.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:04:38.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:04:38.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:04:38.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:38.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:04:38.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:04:38.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:04:38.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:04:38.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:38.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:04:38.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:04:38.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:04:38.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:04:38.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.922 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:04:38.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:04:38.922 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.922 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.922 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.923 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:38.923 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:38.923 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.532954374, 2023-07-02T14:04:38.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:04:38.923 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.923 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.924 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.924 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.924 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.924 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:04:38.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.924 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:38.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:38.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:38.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:04:38.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:04:38.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:04:38.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:04:38.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:04:38.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:04:38.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:04:38.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:04:38.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:04:38.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:04:38.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:04:38.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:04:38.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:38.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:04:38.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:04:38.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:04:38.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:04:38.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:38.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:04:38.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:04:38.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:04:38.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:04:38.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:04:38.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:04:38.933 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.933 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.933 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.933 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:38.933 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:38.934 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.532762592, 2023-07-02T14:04:38.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:04:38.934 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.934 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.934 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.934 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.934 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.934 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:04:38.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.935 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:38.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:38.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:38.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:04:38.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:04:38.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:04:38.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:04:38.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:04:38.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:04:38.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:04:38.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:04:38.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:04:38.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:04:38.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:04:38.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:04:38.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:38.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:04:38.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:04:38.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:38.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:04:38.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:04:38.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:04:38.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:04:38.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:04:38.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:04:38.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:04:38.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:04:38.943 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.943 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.943 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.944 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:38.944 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:38.944 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.532586932, 2023-07-02T14:04:38.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:04:38.944 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.944 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.944 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.945 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.945 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.945 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:04:38.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.945 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:38.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:38.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:38.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:04:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:04:38.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:04:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:04:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:04:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:04:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:04:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:04:38.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:04:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:04:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:04:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:04:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:04:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:04:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:04:38.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:04:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:04:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:04:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:04:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:04:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:04:38.953 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.532492744, 2023-07-02T14:04:38.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:04:38.955 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.955 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.955 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.955 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.955 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.955 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:04:38.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.955 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:38.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:38.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:38.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:04:38.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:04:38.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:04:38.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:04:38.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:04:38.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:04:38.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:04:38.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:04:38.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:04:38.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:04:38.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:04:38.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:04:38.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:38.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:04:38.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:04:38.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:38.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:04:38.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:04:38.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:04:38.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:04:38.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:04:38.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:04:38.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:04:38.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:04:38.964 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.964 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.964 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.964 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:38.964 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:38.965 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.532289358, 2023-07-02T14:04:38.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:04:38.965 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.965 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.965 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.965 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.965 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.965 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:04:38.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.966 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:38.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:04:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:04:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:04:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:04:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:04:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:04:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:04:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:04:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:04:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:04:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:04:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:04:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:04:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:04:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:04:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:04:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:04:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:04:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:04:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:04:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:04:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:04:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.532138702, 2023-07-02T14:04:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:04:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:04:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:04:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:04:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:04:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:04:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:04:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:04:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:04:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:04:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:04:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:04:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:04:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:04:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:04:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:04:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:04:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:04:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:04:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:04:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:04:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:04:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:04:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:04:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.532065999, 2023-07-02T14:04:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:04:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:04:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:04:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:04:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:04:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:04:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:04:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:04:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:04:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:04:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:04:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:04:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:04:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:38.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:04:38.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:04:38.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:38.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:04:38.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:04:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:04:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:04:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:04:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:04:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:04:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:04:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.531949009, 2023-07-02T14:04:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:04:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:04:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:04:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:04:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:04:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:04:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:04:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:04:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:04:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:04:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:04:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:04:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:04:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:04:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:04:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:04:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:04:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:04:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:04:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:04:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:04:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:04:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:04:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.531799558, 2023-07-02T14:04:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:04:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:04:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:04:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:04:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:04:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:04:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:39.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:04:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:04:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:04:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:04:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:04:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:04:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:04:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:04:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:04:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:04:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:04:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:04:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:04:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:04:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:04:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:04:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:04:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:04:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.51032358499999, 2023-07-02T14:04:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:04:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:04:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:04:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:04:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:04:39.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:04:39.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:39.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:04:39.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:04:39.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:04:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:04:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:04:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:04:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:04:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:04:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:04:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:04:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:04:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:04:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:04:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:04:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:04:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:04:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:04:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:04:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.510191049, 2023-07-02T14:04:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:04:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:04:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:04:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:04:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:04:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:04:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:04:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:04:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:04:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:04:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:04:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:04:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:04:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:04:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:04:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:04:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:04:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:04:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:04:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:04:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:04:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:04:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:04:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:04:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.448854388, 2023-07-02T14:04:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:04:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:04:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:04:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:04:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:04:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:04:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:04:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:04:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:04:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:04:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:04:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:04:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:04:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:04:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:04:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:04:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:04:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:04:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:04:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:04:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:04:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:04:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:04:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:04:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:39.050 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:39.050 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:39.050 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:39.050 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:39.051 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:39.052 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:39.051+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:39.152 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:39.152+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:39.189 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:39.189 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:39.189 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:39.190 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:39.200 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:04:39.284 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:39.284+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:39.353 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:39.353+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:39.367 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:39.367 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:39.367 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:39.368 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:39.368 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:39.378 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:04:39.378 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap25 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap26 b3fbb028-e267-4d25-a750-93ba18a2316a@snap26.diff 2023-07-02T14:04:39.457 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:39.457+0000 7f4a93210700 0 -- 172.21.15.163:0/3147772387 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x55d8392f0e70 msgr2=0x55d8392f3260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:39.462 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:39.462+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:39.551 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:39.551 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2620 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 196608~786432] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 89.224854387999997, 2023-07-02T14:04:39.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966, 2023-07-02T14:04:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2620 2023-07-02T14:04:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.553 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:39.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324571+0000", 2023-07-02T14:04:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016686649999999999 2023-07-02T14:04:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326240+0000", 2023-07-02T14:04:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1844e-05 2023-07-02T14:04:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:39.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326252+0000", 2023-07-02T14:04:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7760000000000003e-06 2023-07-02T14:04:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326258+0000", 2023-07-02T14:04:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0634000000000003e-05 2023-07-02T14:04:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326318+0000", 2023-07-02T14:04:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9629999999999993e-06 2023-07-02T14:04:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:39.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326328+0000", 2023-07-02T14:04:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.010877045 2023-07-02T14:04:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337205+0000", 2023-07-02T14:04:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022401300000000001 2023-07-02T14:04:39.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337429+0000", 2023-07-02T14:04:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000139822 2023-07-02T14:04:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337569+0000", 2023-07-02T14:04:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032942119999999999 2023-07-02T14:04:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340863+0000", 2023-07-02T14:04:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021302560000000001 2023-07-02T14:04:39.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342994+0000", 2023-07-02T14:04:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5634000000000002e-05 2023-07-02T14:04:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:39.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343059+0000", 2023-07-02T14:04:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143721 2023-07-02T14:04:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343203+0000", 2023-07-02T14:04:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966 2023-07-02T14:04:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.563 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2622 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 983040~36864] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 89.223161876999995, 2023-07-02T14:04:39.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664, 2023-07-02T14:04:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2622 2023-07-02T14:04:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.565 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:39.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326265+0000", 2023-07-02T14:04:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1282000000000001e-05 2023-07-02T14:04:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326286+0000", 2023-07-02T14:04:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9480000000000002e-06 2023-07-02T14:04:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326293+0000", 2023-07-02T14:04:39.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3610000000000003e-06 2023-07-02T14:04:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326297+0000", 2023-07-02T14:04:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7707999999999994e-05 2023-07-02T14:04:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326385+0000", 2023-07-02T14:04:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8070000000000006e-06 2023-07-02T14:04:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326394+0000", 2023-07-02T14:04:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.014535878 2023-07-02T14:04:39.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340930+0000", 2023-07-02T14:04:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000154551 2023-07-02T14:04:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341084+0000", 2023-07-02T14:04:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00032375900000000002 2023-07-02T14:04:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341408+0000", 2023-07-02T14:04:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012631066 2023-07-02T14:04:39.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354039+0000", 2023-07-02T14:04:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5741000000000001e-05 2023-07-02T14:04:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:39.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354055+0000", 2023-07-02T14:04:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8794000000000006e-05 2023-07-02T14:04:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354124+0000", 2023-07-02T14:04:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014667700000000001 2023-07-02T14:04:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354270+0000", 2023-07-02T14:04:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664 2023-07-02T14:04:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2727 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~12288] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 89.223126101999995, 2023-07-02T14:04:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999, 2023-07-02T14:04:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.576 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2727 2023-07-02T14:04:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:39.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3834e-05 2023-07-02T14:04:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326315+0000", 2023-07-02T14:04:39.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3029999999999996e-06 2023-07-02T14:04:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326321+0000", 2023-07-02T14:04:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7590000000000002e-06 2023-07-02T14:04:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326324+0000", 2023-07-02T14:04:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112872 2023-07-02T14:04:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326437+0000", 2023-07-02T14:04:39.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2290000000000002e-06 2023-07-02T14:04:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326442+0000", 2023-07-02T14:04:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0155896879999999 2023-07-02T14:04:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342031+0000", 2023-07-02T14:04:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016496100000000001 2023-07-02T14:04:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342196+0000", 2023-07-02T14:04:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021998699999999999 2023-07-02T14:04:39.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342416+0000", 2023-07-02T14:04:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011869744999999999 2023-07-02T14:04:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:39.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354286+0000", 2023-07-02T14:04:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0361e-05 2023-07-02T14:04:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354347+0000", 2023-07-02T14:04:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3312000000000005e-05 2023-07-02T14:04:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354430+0000", 2023-07-02T14:04:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126444 2023-07-02T14:04:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354556+0000", 2023-07-02T14:04:39.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999 2023-07-02T14:04:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2730 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [set-alloc-hint object_size 4194304 write_size 4194304,write 454656~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:39.587 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 89.223101092999997, 2023-07-02T14:04:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999, 2023-07-02T14:04:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:39.588 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2730 2023-07-02T14:04:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:39.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:39.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6049999999999997e-06 2023-07-02T14:04:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326332+0000", 2023-07-02T14:04:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3000000000000002e-06 2023-07-02T14:04:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326335+0000", 2023-07-02T14:04:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.719e-05 2023-07-02T14:04:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326352+0000", 2023-07-02T14:04:39.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011583599999999999 2023-07-02T14:04:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326468+0000", 2023-07-02T14:04:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.831e-06 2023-07-02T14:04:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326473+0000", 2023-07-02T14:04:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.016769915 2023-07-02T14:04:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343243+0000", 2023-07-02T14:04:39.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197352 2023-07-02T14:04:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343440+0000", 2023-07-02T14:04:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197581 2023-07-02T14:04:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343638+0000", 2023-07-02T14:04:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010980139 2023-07-02T14:04:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354618+0000", 2023-07-02T14:04:39.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056779530000000003 2023-07-02T14:04:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360296+0000", 2023-07-02T14:04:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101109 2023-07-02T14:04:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360397+0000", 2023-07-02T14:04:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000160273 2023-07-02T14:04:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360557+0000", 2023-07-02T14:04:39.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999 2023-07-02T14:04:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2752 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 1019904~3174400] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:39.599 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 89.223054868999995, 2023-07-02T14:04:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001, 2023-07-02T14:04:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.600 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2752 2023-07-02T14:04:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:39.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:39.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326372+0000", 2023-07-02T14:04:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052050380000000004 2023-07-02T14:04:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331577+0000", 2023-07-02T14:04:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6832000000000001e-05 2023-07-02T14:04:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331594+0000", 2023-07-02T14:04:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8220000000000007e-06 2023-07-02T14:04:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331602+0000", 2023-07-02T14:04:39.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9895999999999999e-05 2023-07-02T14:04:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331642+0000", 2023-07-02T14:04:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:04:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331649+0000", 2023-07-02T14:04:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0139141709999999 2023-07-02T14:04:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345563+0000", 2023-07-02T14:04:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113817 2023-07-02T14:04:39.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345677+0000", 2023-07-02T14:04:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018803400000000001 2023-07-02T14:04:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:39.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345865+0000", 2023-07-02T14:04:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087950359999999991 2023-07-02T14:04:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354660+0000", 2023-07-02T14:04:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059116669999999998 2023-07-02T14:04:39.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360572+0000", 2023-07-02T14:04:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0992000000000002e-05 2023-07-02T14:04:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:39.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360633+0000", 2023-07-02T14:04:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013601600000000001 2023-07-02T14:04:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360769+0000", 2023-07-02T14:04:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001 2023-07-02T14:04:39.610 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2759 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 0~454656] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 89.217815071999993, 2023-07-02T14:04:39.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999, 2023-07-02T14:04:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2759 2023-07-02T14:04:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.612 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:39.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331611+0000", 2023-07-02T14:04:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053795999999999998 2023-07-02T14:04:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332149+0000", 2023-07-02T14:04:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4970000000000007e-06 2023-07-02T14:04:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332158+0000", 2023-07-02T14:04:39.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:04:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332162+0000", 2023-07-02T14:04:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3526000000000001e-05 2023-07-02T14:04:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332185+0000", 2023-07-02T14:04:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:04:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332189+0000", 2023-07-02T14:04:39.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0186295910000001 2023-07-02T14:04:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350819+0000", 2023-07-02T14:04:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000111123 2023-07-02T14:04:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350930+0000", 2023-07-02T14:04:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019575300000000001 2023-07-02T14:04:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.351126+0000", 2023-07-02T14:04:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011503298 2023-07-02T14:04:39.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362629+0000", 2023-07-02T14:04:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085443989999999994 2023-07-02T14:04:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371173+0000", 2023-07-02T14:04:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3547999999999995e-05 2023-07-02T14:04:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371237+0000", 2023-07-02T14:04:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790300000000001 2023-07-02T14:04:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371375+0000", 2023-07-02T14:04:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999 2023-07-02T14:04:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2760 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 458752~131072] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 89.217258887, 2023-07-02T14:04:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001, 2023-07-02T14:04:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:39.623 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2760 2023-07-02T14:04:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:39.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332168+0000", 2023-07-02T14:04:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013281100000000001 2023-07-02T14:04:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332301+0000", 2023-07-02T14:04:39.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7389999999999999e-06 2023-07-02T14:04:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332308+0000", 2023-07-02T14:04:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0639999999999998e-06 2023-07-02T14:04:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332312+0000", 2023-07-02T14:04:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1735000000000001e-05 2023-07-02T14:04:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332333+0000", 2023-07-02T14:04:39.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8342000000000002e-05 2023-07-02T14:04:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332352+0000", 2023-07-02T14:04:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02234782 2023-07-02T14:04:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354699+0000", 2023-07-02T14:04:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110371 2023-07-02T14:04:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354810+0000", 2023-07-02T14:04:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022174900000000001 2023-07-02T14:04:39.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355032+0000", 2023-07-02T14:04:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076053329999999997 2023-07-02T14:04:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:39.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362637+0000", 2023-07-02T14:04:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087913609999999993 2023-07-02T14:04:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371428+0000", 2023-07-02T14:04:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8532999999999998e-05 2023-07-02T14:04:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371477+0000", 2023-07-02T14:04:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010990800000000001 2023-07-02T14:04:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371587+0000", 2023-07-02T14:04:39.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001 2023-07-02T14:04:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2766 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 589824~45056] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:39.634 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 89.217110262000006, 2023-07-02T14:04:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188, 2023-07-02T14:04:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2766 2023-07-02T14:04:39.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:39.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2247999999999998e-05 2023-07-02T14:04:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332369+0000", 2023-07-02T14:04:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4189999999999996e-06 2023-07-02T14:04:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332376+0000", 2023-07-02T14:04:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1049999999999999e-06 2023-07-02T14:04:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332379+0000", 2023-07-02T14:04:39.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6894000000000003e-05 2023-07-02T14:04:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332416+0000", 2023-07-02T14:04:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3970000000000002e-06 2023-07-02T14:04:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332422+0000", 2023-07-02T14:04:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0231382280000001 2023-07-02T14:04:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355560+0000", 2023-07-02T14:04:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9079000000000005e-05 2023-07-02T14:04:39.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355659+0000", 2023-07-02T14:04:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018005000000000001 2023-07-02T14:04:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355839+0000", 2023-07-02T14:04:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068044439999999998 2023-07-02T14:04:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362644+0000", 2023-07-02T14:04:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090155589999999994 2023-07-02T14:04:39.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371659+0000", 2023-07-02T14:04:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1297999999999994e-05 2023-07-02T14:04:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:39.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371730+0000", 2023-07-02T14:04:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013100699999999999 2023-07-02T14:04:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371861+0000", 2023-07-02T14:04:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188 2023-07-02T14:04:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.645 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2784 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 0~393216] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 89.217041809999998, 2023-07-02T14:04:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999, 2023-07-02T14:04:39.646 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2784 2023-07-02T14:04:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:39.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:39.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332385+0000", 2023-07-02T14:04:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040252400000000002 2023-07-02T14:04:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332787+0000", 2023-07-02T14:04:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0208e-05 2023-07-02T14:04:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332798+0000", 2023-07-02T14:04:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2023-07-02T14:04:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332801+0000", 2023-07-02T14:04:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4838000000000001e-05 2023-07-02T14:04:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332836+0000", 2023-07-02T14:04:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5319999999999998e-06 2023-07-02T14:04:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332839+0000", 2023-07-02T14:04:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0234290109999999 2023-07-02T14:04:39.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356268+0000", 2023-07-02T14:04:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011165700000000001 2023-07-02T14:04:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:39.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356380+0000", 2023-07-02T14:04:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022504799999999999 2023-07-02T14:04:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356605+0000", 2023-07-02T14:04:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015276201 2023-07-02T14:04:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371881+0000", 2023-07-02T14:04:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5496999999999999e-05 2023-07-02T14:04:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371957+0000", 2023-07-02T14:04:39.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4570000000000002e-05 2023-07-02T14:04:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372001+0000", 2023-07-02T14:04:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3880000000000003e-05 2023-07-02T14:04:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372085+0000", 2023-07-02T14:04:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999 2023-07-02T14:04:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2785 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 405504~446464] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 89.216620359000004, 2023-07-02T14:04:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999, 2023-07-02T14:04:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.658 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2785 2023-07-02T14:04:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:39.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:39.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059527500000000004 2023-07-02T14:04:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:39.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333402+0000", 2023-07-02T14:04:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7059999999999993e-06 2023-07-02T14:04:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333411+0000", 2023-07-02T14:04:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0450000000000001e-06 2023-07-02T14:04:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333414+0000", 2023-07-02T14:04:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5790999999999997e-05 2023-07-02T14:04:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:04:39.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3320000000000002e-06 2023-07-02T14:04:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333454+0000", 2023-07-02T14:04:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.025310269 2023-07-02T14:04:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358764+0000", 2023-07-02T14:04:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012414899999999999 2023-07-02T14:04:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358888+0000", 2023-07-02T14:04:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000217028 2023-07-02T14:04:39.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.359105+0000", 2023-07-02T14:04:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012788212 2023-07-02T14:04:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:39.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371893+0000", 2023-07-02T14:04:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012201569999999999 2023-07-02T14:04:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373114+0000", 2023-07-02T14:04:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4512999999999995e-05 2023-07-02T14:04:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373198+0000", 2023-07-02T14:04:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132373 2023-07-02T14:04:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:39.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373330+0000", 2023-07-02T14:04:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999 2023-07-02T14:04:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2786 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1310720~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:39.669 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 89.216007665999996, 2023-07-02T14:04:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001, 2023-07-02T14:04:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.670 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2786 2023-07-02T14:04:39.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.671 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:39.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:39.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:39.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:39.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333419+0000", 2023-07-02T14:04:39.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2009999999999999e-06 2023-07-02T14:04:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333427+0000", 2023-07-02T14:04:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0210000000000001e-06 2023-07-02T14:04:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333431+0000", 2023-07-02T14:04:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3769999999999999e-06 2023-07-02T14:04:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333434+0000", 2023-07-02T14:04:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1161e-05 2023-07-02T14:04:39.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333485+0000", 2023-07-02T14:04:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1549000000000001e-05 2023-07-02T14:04:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:39.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333496+0000", 2023-07-02T14:04:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0273116339999999 2023-07-02T14:04:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360808+0000", 2023-07-02T14:04:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000115128 2023-07-02T14:04:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360923+0000", 2023-07-02T14:04:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018367700000000001 2023-07-02T14:04:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361107+0000", 2023-07-02T14:04:39.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010796923999999999 2023-07-02T14:04:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371904+0000", 2023-07-02T14:04:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014983520000000001 2023-07-02T14:04:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373402+0000", 2023-07-02T14:04:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0520999999999999e-05 2023-07-02T14:04:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373463+0000", 2023-07-02T14:04:39.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000129095 2023-07-02T14:04:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373592+0000", 2023-07-02T14:04:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001 2023-07-02T14:04:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.680 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2787 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1433600~8192] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 89.215990865999999, 2023-07-02T14:04:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999, 2023-07-02T14:04:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:39.681 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2787 2023-07-02T14:04:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:39.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333436+0000", 2023-07-02T14:04:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2731999999999999e-05 2023-07-02T14:04:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:39.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:04:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7440000000000001e-06 2023-07-02T14:04:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333456+0000", 2023-07-02T14:04:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8650000000000003e-06 2023-07-02T14:04:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333460+0000", 2023-07-02T14:04:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0077e-05 2023-07-02T14:04:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333530+0000", 2023-07-02T14:04:39.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4180000000000001e-06 2023-07-02T14:04:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333533+0000", 2023-07-02T14:04:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0284002720000001 2023-07-02T14:04:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361933+0000", 2023-07-02T14:04:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6781999999999996e-05 2023-07-02T14:04:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362030+0000", 2023-07-02T14:04:39.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018305400000000001 2023-07-02T14:04:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362213+0000", 2023-07-02T14:04:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097015109999999995 2023-07-02T14:04:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371915+0000", 2023-07-02T14:04:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001742295 2023-07-02T14:04:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373657+0000", 2023-07-02T14:04:39.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5980000000000003e-05 2023-07-02T14:04:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373713+0000", 2023-07-02T14:04:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011783800000000001 2023-07-02T14:04:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373831+0000", 2023-07-02T14:04:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999 2023-07-02T14:04:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2803 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 851968~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 89.215949014000003, 2023-07-02T14:04:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514, 2023-07-02T14:04:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:39.693 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2803 2023-07-02T14:04:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:39.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4943000000000001e-05 2023-07-02T14:04:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333493+0000", 2023-07-02T14:04:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2629999999999997e-06 2023-07-02T14:04:39.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333500+0000", 2023-07-02T14:04:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4159999999999997e-06 2023-07-02T14:04:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333505+0000", 2023-07-02T14:04:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4518999999999998e-05 2023-07-02T14:04:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333559+0000", 2023-07-02T14:04:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5900000000000008e-06 2023-07-02T14:04:39.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333568+0000", 2023-07-02T14:04:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0291061779999999 2023-07-02T14:04:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362674+0000", 2023-07-02T14:04:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110412 2023-07-02T14:04:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362784+0000", 2023-07-02T14:04:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015384599999999999 2023-07-02T14:04:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362938+0000", 2023-07-02T14:04:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091682310000000006 2023-07-02T14:04:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372106+0000", 2023-07-02T14:04:39.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017703230000000001 2023-07-02T14:04:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373877+0000", 2023-07-02T14:04:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6387000000000002e-05 2023-07-02T14:04:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373923+0000", 2023-07-02T14:04:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014099199999999999 2023-07-02T14:04:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374064+0000", 2023-07-02T14:04:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514 2023-07-02T14:04:39.703 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2805 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 634880~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 89.215915839000004, 2023-07-02T14:04:39.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814, 2023-07-02T14:04:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2805 2023-07-02T14:04:39.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333511+0000", 2023-07-02T14:04:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207092 2023-07-02T14:04:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333718+0000", 2023-07-02T14:04:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5110000000000003e-06 2023-07-02T14:04:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333727+0000", 2023-07-02T14:04:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3270000000000007e-06 2023-07-02T14:04:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333736+0000", 2023-07-02T14:04:39.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0319999999999999e-05 2023-07-02T14:04:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333756+0000", 2023-07-02T14:04:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1695000000000001e-05 2023-07-02T14:04:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:04:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0295750560000001 2023-07-02T14:04:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363343+0000", 2023-07-02T14:04:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3387999999999994e-05 2023-07-02T14:04:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:39.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363437+0000", 2023-07-02T14:04:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018373 2023-07-02T14:04:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363620+0000", 2023-07-02T14:04:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0084948769999999996 2023-07-02T14:04:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372115+0000", 2023-07-02T14:04:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020082170000000001 2023-07-02T14:04:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374123+0000", 2023-07-02T14:04:39.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8875999999999997e-05 2023-07-02T14:04:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374182+0000", 2023-07-02T14:04:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0001272 2023-07-02T14:04:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374309+0000", 2023-07-02T14:04:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814 2023-07-02T14:04:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.716 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2806 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 1359872~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 89.215685508000007, 2023-07-02T14:04:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234, 2023-07-02T14:04:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:39.717 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2806 2023-07-02T14:04:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:39.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333741+0000", 2023-07-02T14:04:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3978999999999999e-05 2023-07-02T14:04:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333755+0000", 2023-07-02T14:04:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:04:39.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333761+0000", 2023-07-02T14:04:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7010000000000001e-06 2023-07-02T14:04:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333764+0000", 2023-07-02T14:04:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5086000000000003e-05 2023-07-02T14:04:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333799+0000", 2023-07-02T14:04:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6389999999999992e-06 2023-07-02T14:04:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333806+0000", 2023-07-02T14:04:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.030496675 2023-07-02T14:04:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364303+0000", 2023-07-02T14:04:39.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011382700000000001 2023-07-02T14:04:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364417+0000", 2023-07-02T14:04:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192058 2023-07-02T14:04:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364609+0000", 2023-07-02T14:04:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075132130000000004 2023-07-02T14:04:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372122+0000", 2023-07-02T14:04:39.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022731750000000001 2023-07-02T14:04:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374395+0000", 2023-07-02T14:04:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5277000000000005e-05 2023-07-02T14:04:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374460+0000", 2023-07-02T14:04:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017074599999999999 2023-07-02T14:04:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374631+0000", 2023-07-02T14:04:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234 2023-07-02T14:04:39.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 868352~49152] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 89.215658794000007, 2023-07-02T14:04:39.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001, 2023-07-02T14:04:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:04:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.729 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:39.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:04:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6355000000000001e-05 2023-07-02T14:04:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333794+0000", 2023-07-02T14:04:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8329999999999996e-06 2023-07-02T14:04:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333801+0000", 2023-07-02T14:04:39.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5419999999999996e-06 2023-07-02T14:04:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333807+0000", 2023-07-02T14:04:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2023-07-02T14:04:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333826+0000", 2023-07-02T14:04:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.303e-06 2023-07-02T14:04:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333830+0000", 2023-07-02T14:04:39.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.031137559 2023-07-02T14:04:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364967+0000", 2023-07-02T14:04:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1390000000000004e-05 2023-07-02T14:04:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365059+0000", 2023-07-02T14:04:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017597399999999999 2023-07-02T14:04:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365234+0000", 2023-07-02T14:04:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068943950000000002 2023-07-02T14:04:39.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372129+0000", 2023-07-02T14:04:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026189910000000002 2023-07-02T14:04:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374748+0000", 2023-07-02T14:04:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6613999999999996e-05 2023-07-02T14:04:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374814+0000", 2023-07-02T14:04:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502300000000001 2023-07-02T14:04:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374950+0000", 2023-07-02T14:04:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001 2023-07-02T14:04:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2815 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 917504~49152] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.211444560999993, 2023-07-02T14:04:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998, 2023-07-02T14:04:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:39.740 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2815 2023-07-02T14:04:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:39.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:04:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337980+0000", 2023-07-02T14:04:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017126700000000001 2023-07-02T14:04:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338152+0000", 2023-07-02T14:04:39.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2938999999999999e-05 2023-07-02T14:04:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338164+0000", 2023-07-02T14:04:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4448999999999999e-05 2023-07-02T14:04:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:04:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0281812 2023-07-02T14:04:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366370+0000", 2023-07-02T14:04:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1840000000000002e-05 2023-07-02T14:04:39.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366462+0000", 2023-07-02T14:04:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000186112 2023-07-02T14:04:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:39.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366648+0000", 2023-07-02T14:04:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054905709999999996 2023-07-02T14:04:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372139+0000", 2023-07-02T14:04:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044954790000000001 2023-07-02T14:04:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376634+0000", 2023-07-02T14:04:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4621000000000004e-05 2023-07-02T14:04:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:39.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376699+0000", 2023-07-02T14:04:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012862299999999999 2023-07-02T14:04:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376827+0000", 2023-07-02T14:04:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998 2023-07-02T14:04:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.749 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2816 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 917504~65536] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.211230181999994, 2023-07-02T14:04:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001, 2023-07-02T14:04:39.750 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2816 2023-07-02T14:04:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:39.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:39.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338196+0000", 2023-07-02T14:04:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6286999999999999e-05 2023-07-02T14:04:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338243+0000", 2023-07-02T14:04:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7030000000000001e-06 2023-07-02T14:04:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338252+0000", 2023-07-02T14:04:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:04:39.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338258+0000", 2023-07-02T14:04:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032085734999999997 2023-07-02T14:04:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370344+0000", 2023-07-02T14:04:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108621 2023-07-02T14:04:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370452+0000", 2023-07-02T14:04:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014936499999999999 2023-07-02T14:04:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370602+0000", 2023-07-02T14:04:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001778012 2023-07-02T14:04:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372380+0000", 2023-07-02T14:04:39.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044876380000000004 2023-07-02T14:04:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376867+0000", 2023-07-02T14:04:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9870999999999999e-05 2023-07-02T14:04:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376897+0000", 2023-07-02T14:04:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011401 2023-07-02T14:04:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:39.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.377011+0000", 2023-07-02T14:04:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001 2023-07-02T14:04:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.760 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:3878 2.1 e80/79 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head v 80'436, mlcod=80'429)", 2023-07-02T14:04:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 61.441976685, 2023-07-02T14:04:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037799472000000001, 2023-07-02T14:04:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:39.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107450+0000", 2023-07-02T14:04:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023914520000000001 2023-07-02T14:04:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109841+0000", 2023-07-02T14:04:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0823e-05 2023-07-02T14:04:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109862+0000", 2023-07-02T14:04:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2023-07-02T14:04:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109874+0000", 2023-07-02T14:04:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075009600000000001 2023-07-02T14:04:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.110624+0000", 2023-07-02T14:04:39.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5176000000000001e-05 2023-07-02T14:04:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.110649+0000", 2023-07-02T14:04:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034557310000000001 2023-07-02T14:04:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145206+0000", 2023-07-02T14:04:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4110999999999998e-05 2023-07-02T14:04:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145250+0000", 2023-07-02T14:04:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037799472000000001 2023-07-02T14:04:39.767 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:196 2.5 2:a6e4b3cf:::rbd_data.125de1449605.00000000000000c4:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:04:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172685+0000", 2023-07-02T14:04:39.768 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 6.3767425260000001, 2023-07-02T14:04:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038025259999999998, 2023-07-02T14:04:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:04:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:04:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 196 2023-07-02T14:04:39.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172685+0000", 2023-07-02T14:04:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172685+0000", 2023-07-02T14:04:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172684+0000", 2023-07-02T14:04:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.693e-06 2023-07-02T14:04:39.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172688+0000", 2023-07-02T14:04:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4470000000000003e-06 2023-07-02T14:04:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:39.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172695+0000", 2023-07-02T14:04:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.014e-06 2023-07-02T14:04:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172699+0000", 2023-07-02T14:04:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025360667 2023-07-02T14:04:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.198059+0000", 2023-07-02T14:04:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011197900000000001 2023-07-02T14:04:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.198171+0000", 2023-07-02T14:04:39.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012538942000000001 2023-07-02T14:04:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.210710+0000", 2023-07-02T14:04:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038025259999999998 2023-07-02T14:04:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:39.776 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:39.776 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:39.776 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:04:39.778 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:04:39.779 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:39.777+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:39.779 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:39.677+0000 7f4a93210700 0 -- 172.21.15.163:0/3147772387 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f4a78036bb0 msgr2=0x7f4a78038fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:39.779 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 100% complete...done. 2023-07-02T14:04:39.779 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:04:39.779 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=27 2023-07-02T14:04:39.779 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap27 2023-07-02T14:04:39.877 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:39.877+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:39.946 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:39.946 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:39.946 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:39.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:04:39.959 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:39.977 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:39.977 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:39.977+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:39.995 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:40.013 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:40.032 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:40.032 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:04:40.033 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:04:40.033 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:04:40.033 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:40.033 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:04:40.033 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:04:40.036 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:40.037 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:40.079 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:40.078+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:40.083 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:40.084 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:40.094 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:04:40.180 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:40.180+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:40.217 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:40.218 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:40.228 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:04:40.281 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:40.280+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:40.367 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:04:40.381 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:04:40.387 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 4 2023-07-02T14:04:40.398 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:40.398 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:40.398 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:40.398+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:40.409 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:04:40.481 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 2] out_osds: [5] dead_osds: [3] live_osds: [0, 1, 2, 4, 5] 2023-07-02T14:04:40.482 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:04:40.482 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 3, 4, 2] 2023-07-02T14:04:40.482 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2023-07-02T14:04:40.492 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:40.492+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:40.577 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:40.578 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:40.584 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:40.584+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:40.588 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:04:40.685 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:40.684+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:40.754 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:40.754 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:40.764 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:04:40.785 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:40.785+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:40.812 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-07-02T14:04:40.812+0000 7ff490b05700 -1 mon.a@0(leader).osd e115 definitely_dead 0 2023-07-02T14:04:40.886 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:40.886+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:40.901 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:40.902 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:40.912 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:04:40.987 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:40.987+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:41.077 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:41.078 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:41.088 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:04:41.088 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:41.087+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:41.188 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:41.187+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:41.252 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:41.252 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:41.262 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:04:41.304 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:41.303+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:41.360 INFO:teuthology.orchestra.run.smithi016.stderr:marked out osd.4. 2023-07-02T14:04:41.360 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:04:41.414 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:41.414+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:41.428 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:41.429 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:41.439 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:04:41.490 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:41.490+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:41.561 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:41.561 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:41.571 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:41.591 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:41.591+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:41.691 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:41.691+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:41.694 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:41.694 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:41.704 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:41.792 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:41.792+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:41.833 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:41.833 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:41.843 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:04:41.893 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:41.893+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:41.967 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:41.967 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:41.977 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:04:41.995 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:41.994+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:42.038 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:04:42.095 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:42.094+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:42.146 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:42.147 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:42.157 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:04:42.200 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:42.200+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:42.205 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:42.223 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:42.240 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:42.259 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:42.278 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:42.278 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:04:42.278 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:04:42.279 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:04:42.279 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:42.279 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:04:42.279 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:04:42.282 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:42.282 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:42.287 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:42.287 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:42.287 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:42.287 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:42.296 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:42.296+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:42.297 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:04:42.397 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:42.396+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:42.461 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:42.462 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:42.462 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:42.462 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:42.462 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:42.472 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:04:42.497 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:42.497+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:42.598 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:42.598 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:42.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:42.598 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:42.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.598 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:04:42.599 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:42.599 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 93.143751562000006, 2023-07-02T14:04:42.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:04:42.599 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:42.599 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:42.599 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:42.599 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:42.599 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:42.600 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:04:42.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.600 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:42.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:42.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:42.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:42.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:42.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:42.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:42.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:42.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:04:42.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:04:42.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:42.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:04:42.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:04:42.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:42.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:04:42.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:04:42.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:42.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:04:42.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:04:42.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:04:42.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:04:42.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:42.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:04:42.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:04:42.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:04:42.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:04:42.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:42.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:04:42.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:04:42.606 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.606 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.607 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.607 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:04:42.607 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:42.607 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 92.273058554000002, 2023-07-02T14:04:42.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:04:42.607 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:42.608 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:42.608 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:42.608 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:04:42.608 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:04:42.608 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:04:42.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.608 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:42.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:42.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:42.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:42.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:42.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:42.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:42.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:42.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:04:42.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:04:42.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:42.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:04:42.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:04:42.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:42.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:04:42.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:04:42.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:42.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:04:42.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:04:42.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:04:42.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:04:42.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:42.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:04:42.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:04:42.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:04:42.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:04:42.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:42.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:04:42.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:04:42.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:42.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:04:42.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:04:42.615 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.615 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.616 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.616 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:42.616 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:42.616 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 92.272933832999996, 2023-07-02T14:04:42.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:04:42.617 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:42.617 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:42.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:42.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:42.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:42.617 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:04:42.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.617 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:42.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:42.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:42.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:42.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:42.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:42.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:42.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:42.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:04:42.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:04:42.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:42.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:04:42.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:04:42.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:42.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:04:42.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:04:42.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:42.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:04:42.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:04:42.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:04:42.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:04:42.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:42.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:04:42.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:04:42.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:04:42.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:04:42.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:42.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:04:42.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:04:42.624 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.624 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.624 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.625 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:42.625 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:42.625 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 92.266083949000006, 2023-07-02T14:04:42.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:04:42.625 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:42.625 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:42.625 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:42.625 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:42.626 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:42.626 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:04:42.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.626 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:42.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:42.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:42.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:42.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:42.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:42.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:04:42.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:42.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:04:42.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:04:42.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:42.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:04:42.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:04:42.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:42.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:04:42.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:04:42.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:42.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:04:42.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:04:42.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:04:42.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:04:42.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:42.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:04:42.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:04:42.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:04:42.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:04:42.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:42.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:04:42.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:04:42.632 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.632 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.633 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.633 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:42.633 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:42.633 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 92.266021023999997, 2023-07-02T14:04:42.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:04:42.633 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:42.634 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:42.634 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:42.634 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:42.634 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:42.634 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:04:42.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.634 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:42.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:42.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:42.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:42.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:42.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:42.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:42.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:42.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:04:42.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:04:42.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:42.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:04:42.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:04:42.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:42.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:04:42.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:04:42.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:42.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:04:42.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:04:42.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:04:42.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:04:42.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:42.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:04:42.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:04:42.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:04:42.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:04:42.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:42.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:04:42.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:04:42.641 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.641 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.641 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.641 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:42.641 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:42.642 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 92.265720603000005, 2023-07-02T14:04:42.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:04:42.642 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:42.642 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:42.642 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:42.642 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:42.642 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:42.642 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:04:42.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.643 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:42.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:42.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:42.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:42.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:42.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:42.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:42.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:42.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:04:42.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:04:42.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:42.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:04:42.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:04:42.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:42.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:04:42.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:04:42.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:42.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:04:42.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:04:42.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:04:42.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:04:42.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:42.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:04:42.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:04:42.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:04:42.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:04:42.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:42.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:04:42.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:04:42.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:04:42.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:04:42.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:04:42.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:42.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:04:42.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:04:42.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:42.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:04:42.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:04:42.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:42.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:04:42.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:04:42.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:42.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:04:42.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:04:42.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:42.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:04:42.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:04:42.653 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.653 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.653 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.654 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:04:42.654 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:42.654 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 92.265196102999994, 2023-07-02T14:04:42.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:04:42.654 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:42.654 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:42.654 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:42.654 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:42.655 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:42.655 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:04:42.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.655 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:42.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:42.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:42.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:42.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:42.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:42.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:42.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:42.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:42.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:04:42.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:42.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:04:42.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:04:42.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:42.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:04:42.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:04:42.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:42.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:04:42.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:04:42.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:04:42.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:04:42.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:42.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:04:42.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:04:42.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:04:42.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:04:42.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:42.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:04:42.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:04:42.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:42.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:04:42.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:04:42.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.663 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.663 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.663 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:04:42.663 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:42.663 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.141176978000004, 2023-07-02T14:04:42.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:04:42.664 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:42.664 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:42.664 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:42.664 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:42.664 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:42.664 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:04:42.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.664 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:42.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:42.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:42.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:42.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:42.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:42.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:42.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:42.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:04:42.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:04:42.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:42.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:04:42.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:04:42.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:42.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:04:42.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:04:42.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:42.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:04:42.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:04:42.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:04:42.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:04:42.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:42.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:04:42.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:04:42.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:04:42.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:04:42.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:42.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:04:42.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:04:42.671 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.671 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.671 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.671 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:42.672 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:42.672 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.815694273999995, 2023-07-02T14:04:42.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:04:42.672 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:42.672 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:42.672 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:42.672 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:42.673 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:42.673 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:04:42.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.673 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:42.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:42.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:42.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:42.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:42.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:42.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:42.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:42.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:04:42.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:04:42.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:42.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:04:42.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:04:42.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:42.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:04:42.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:04:42.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:42.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:04:42.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:04:42.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:42.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:04:42.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:42.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:04:42.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:04:42.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:04:42.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:04:42.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:42.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:04:42.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:04:42.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:42.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:04:42.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:04:42.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:42.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:04:42.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:04:42.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:42.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:04:42.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:04:42.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:42.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:04:42.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:04:42.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:42.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:04:42.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:04:42.683 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.683 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.683 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.683 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3887 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:42.684 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:42.684 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.811831084999994, 2023-07-02T14:04:42.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998, 2023-07-02T14:04:42.684 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:42.684 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:42.684 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:42.684 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:42.684 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:42.685 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3887 2023-07-02T14:04:42.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.685 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:42.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:42.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:42.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:42.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:42.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:42.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:42.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:42.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784019+0000", 2023-07-02T14:04:42.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4073999999999998e-05 2023-07-02T14:04:42.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:42.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784063+0000", 2023-07-02T14:04:42.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6630000000000004e-06 2023-07-02T14:04:42.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:42.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784070+0000", 2023-07-02T14:04:42.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.286e-06 2023-07-02T14:04:42.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:42.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784073+0000", 2023-07-02T14:04:42.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4941999999999998e-05 2023-07-02T14:04:42.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784118+0000", 2023-07-02T14:04:42.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8120000000000008e-06 2023-07-02T14:04:42.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:42.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784126+0000", 2023-07-02T14:04:42.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32999709399999999 2023-07-02T14:04:42.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114123+0000", 2023-07-02T14:04:42.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000157257 2023-07-02T14:04:42.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:42.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114280+0000", 2023-07-02T14:04:42.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000341152 2023-07-02T14:04:42.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:42.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114622+0000", 2023-07-02T14:04:42.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019539766 2023-07-02T14:04:42.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:42.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.134161+0000", 2023-07-02T14:04:42.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015820796000000002 2023-07-02T14:04:42.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:42.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149982+0000", 2023-07-02T14:04:42.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.05e-05 2023-07-02T14:04:42.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:42.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150043+0000", 2023-07-02T14:04:42.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6453999999999999e-05 2023-07-02T14:04:42.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:42.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150089+0000", 2023-07-02T14:04:42.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998 2023-07-02T14:04:42.695 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.695 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.695 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.695 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3939 2.1 2:9ef34a83:::rbd_data.125de1449605.00000000000000b8:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:42.695 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:42.696 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.811773576999997, 2023-07-02T14:04:42.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001, 2023-07-02T14:04:42.696 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:42.696 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:42.696 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:42.696 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:42.696 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:42.696 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3939 2023-07-02T14:04:42.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.697 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:42.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:42.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:42.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:42.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:42.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:42.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:42.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:42.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:42.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046852830000000002 2023-07-02T14:04:42.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:42.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788763+0000", 2023-07-02T14:04:42.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3185e-05 2023-07-02T14:04:42.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:42.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788776+0000", 2023-07-02T14:04:42.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8470000000000003e-06 2023-07-02T14:04:42.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:42.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788781+0000", 2023-07-02T14:04:42.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7104000000000002e-05 2023-07-02T14:04:42.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788828+0000", 2023-07-02T14:04:42.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3516000000000001e-05 2023-07-02T14:04:42.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:42.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788841+0000", 2023-07-02T14:04:42.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33064513600000001 2023-07-02T14:04:42.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119487+0000", 2023-07-02T14:04:42.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9876999999999997e-05 2023-07-02T14:04:42.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:42.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119537+0000", 2023-07-02T14:04:42.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014160999999999999 2023-07-02T14:04:42.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:42.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119678+0000", 2023-07-02T14:04:42.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023054392 2023-07-02T14:04:42.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:42.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.142733+0000", 2023-07-02T14:04:42.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074065470000000003 2023-07-02T14:04:42.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:42.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150139+0000", 2023-07-02T14:04:42.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5025999999999997e-05 2023-07-02T14:04:42.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:42.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150204+0000", 2023-07-02T14:04:42.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147999999999997e-05 2023-07-02T14:04:42.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:42.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150241+0000", 2023-07-02T14:04:42.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001 2023-07-02T14:04:42.706 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.707 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.707 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.707 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3940 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:42.707 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:42.707 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.807062544999994, 2023-07-02T14:04:42.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001, 2023-07-02T14:04:42.708 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:42.708 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:42.708 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:42.708 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:42.708 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:42.708 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3940 2023-07-02T14:04:42.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.709 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:42.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:42.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:42.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:42.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:42.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:42.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:42.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:42.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788788+0000", 2023-07-02T14:04:42.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043166560000000003 2023-07-02T14:04:42.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:42.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793105+0000", 2023-07-02T14:04:42.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5970000000000001e-05 2023-07-02T14:04:42.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:42.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793121+0000", 2023-07-02T14:04:42.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0939999999999998e-06 2023-07-02T14:04:42.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:42.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793126+0000", 2023-07-02T14:04:42.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2438000000000002e-05 2023-07-02T14:04:42.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793178+0000", 2023-07-02T14:04:42.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2019e-05 2023-07-02T14:04:42.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:42.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793190+0000", 2023-07-02T14:04:42.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33208757799999999 2023-07-02T14:04:42.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125278+0000", 2023-07-02T14:04:42.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5255999999999997e-05 2023-07-02T14:04:42.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:42.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125373+0000", 2023-07-02T14:04:42.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015674999999999999 2023-07-02T14:04:42.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:42.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125530+0000", 2023-07-02T14:04:42.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018767308 2023-07-02T14:04:42.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:42.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144297+0000", 2023-07-02T14:04:42.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060291959999999997 2023-07-02T14:04:42.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:42.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150326+0000", 2023-07-02T14:04:42.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3886999999999993e-05 2023-07-02T14:04:42.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:42.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150390+0000", 2023-07-02T14:04:42.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010255 2023-07-02T14:04:42.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:42.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150493+0000", 2023-07-02T14:04:42.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001 2023-07-02T14:04:42.718 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.718 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.719 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.719 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3947 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:42.719 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:42.719 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.802717017999996, 2023-07-02T14:04:42.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999, 2023-07-02T14:04:42.719 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:42.720 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:42.720 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:42.720 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:42.720 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:42.720 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3947 2023-07-02T14:04:42.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.720 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:42.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:42.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:42.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:42.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:42.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:42.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:42.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:42.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793133+0000", 2023-07-02T14:04:42.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2519999999999999e-05 2023-07-02T14:04:42.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:42.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793176+0000", 2023-07-02T14:04:42.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:04:42.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:42.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793180+0000", 2023-07-02T14:04:42.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1709999999999998e-06 2023-07-02T14:04:42.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:42.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793184+0000", 2023-07-02T14:04:42.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1721999999999999e-05 2023-07-02T14:04:42.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793215+0000", 2023-07-02T14:04:42.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1280000000000008e-06 2023-07-02T14:04:42.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:42.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793224+0000", 2023-07-02T14:04:42.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34218549599999998 2023-07-02T14:04:42.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135409+0000", 2023-07-02T14:04:42.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117442 2023-07-02T14:04:42.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:42.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135526+0000", 2023-07-02T14:04:42.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000206257 2023-07-02T14:04:42.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:42.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135733+0000", 2023-07-02T14:04:42.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085700029999999996 2023-07-02T14:04:42.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:42.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144303+0000", 2023-07-02T14:04:42.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006278652 2023-07-02T14:04:42.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:42.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150581+0000", 2023-07-02T14:04:42.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4765999999999996e-05 2023-07-02T14:04:42.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:42.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150646+0000", 2023-07-02T14:04:42.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4777999999999995e-05 2023-07-02T14:04:42.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:42.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150711+0000", 2023-07-02T14:04:42.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999 2023-07-02T14:04:42.730 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.730 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.730 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.731 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3980 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 65536~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:42.731 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:42.731 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.802663136000007, 2023-07-02T14:04:42.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999, 2023-07-02T14:04:42.731 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:42.731 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:42.731 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:42.731 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:42.732 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:42.732 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3980 2023-07-02T14:04:42.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.732 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:42.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:42.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:42.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:42.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:42.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:42.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:42.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:42.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:42.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020244260000000002 2023-07-02T14:04:42.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:42.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795212+0000", 2023-07-02T14:04:42.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0005e-05 2023-07-02T14:04:42.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:42.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795222+0000", 2023-07-02T14:04:42.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9860000000000002e-06 2023-07-02T14:04:42.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:42.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795227+0000", 2023-07-02T14:04:42.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6369999999999999e-05 2023-07-02T14:04:42.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795264+0000", 2023-07-02T14:04:42.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1052e-05 2023-07-02T14:04:42.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:42.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795275+0000", 2023-07-02T14:04:42.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.343850459 2023-07-02T14:04:42.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139125+0000", 2023-07-02T14:04:42.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6760000000000004e-05 2023-07-02T14:04:42.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:42.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139202+0000", 2023-07-02T14:04:42.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015092799999999999 2023-07-02T14:04:42.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:42.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139353+0000", 2023-07-02T14:04:42.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011425940000000001 2023-07-02T14:04:42.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:42.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150779+0000", 2023-07-02T14:04:42.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033223060000000001 2023-07-02T14:04:42.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:42.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154101+0000", 2023-07-02T14:04:42.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9268999999999998e-05 2023-07-02T14:04:42.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:42.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154161+0000", 2023-07-02T14:04:42.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6499999999999999e-05 2023-07-02T14:04:42.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:42.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154207+0000", 2023-07-02T14:04:42.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999 2023-07-02T14:04:42.741 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.742 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.742 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.742 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3982 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1835008~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:42.742 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:42.742 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.800617849000005, 2023-07-02T14:04:42.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002, 2023-07-02T14:04:42.743 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:42.743 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:42.743 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:42.743 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:42.743 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:42.743 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3982 2023-07-02T14:04:42.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.743 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:42.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:42.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:42.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:42.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:42.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:42.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:42.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:42.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795233+0000", 2023-07-02T14:04:42.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8680999999999998e-05 2023-07-02T14:04:42.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:42.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795272+0000", 2023-07-02T14:04:42.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5629999999999999e-06 2023-07-02T14:04:42.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:42.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795277+0000", 2023-07-02T14:04:42.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8700000000000001e-06 2023-07-02T14:04:42.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:42.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795280+0000", 2023-07-02T14:04:42.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1964000000000001e-05 2023-07-02T14:04:42.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795312+0000", 2023-07-02T14:04:42.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.314e-06 2023-07-02T14:04:42.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:42.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795319+0000", 2023-07-02T14:04:42.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34828442599999998 2023-07-02T14:04:42.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143603+0000", 2023-07-02T14:04:42.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2438000000000001e-05 2023-07-02T14:04:42.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:42.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143665+0000", 2023-07-02T14:04:42.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3079999999999997e-05 2023-07-02T14:04:42.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:42.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143759+0000", 2023-07-02T14:04:42.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070633079999999999 2023-07-02T14:04:42.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:42.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150822+0000", 2023-07-02T14:04:42.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034430210000000001 2023-07-02T14:04:42.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:42.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154265+0000", 2023-07-02T14:04:42.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4249999999999998e-05 2023-07-02T14:04:42.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:42.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154309+0000", 2023-07-02T14:04:42.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2020000000000001e-05 2023-07-02T14:04:42.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:42.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154351+0000", 2023-07-02T14:04:42.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002 2023-07-02T14:04:42.753 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.754 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.754 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.754 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4054 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1900544~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:42.754 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:42.754 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.800566177999997, 2023-07-02T14:04:42.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002, 2023-07-02T14:04:42.755 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:42.755 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:42.755 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:42.755 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:42.755 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:42.755 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4054 2023-07-02T14:04:42.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.756 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:42.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:42.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:42.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:42.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:42.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:42.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:42.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:42.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:42.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018874860000000001 2023-07-02T14:04:42.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:42.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797172+0000", 2023-07-02T14:04:42.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3000000000000007e-06 2023-07-02T14:04:42.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:42.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797182+0000", 2023-07-02T14:04:42.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6260000000000003e-06 2023-07-02T14:04:42.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:42.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797186+0000", 2023-07-02T14:04:42.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9013e-05 2023-07-02T14:04:42.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797235+0000", 2023-07-02T14:04:42.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5990000000000008e-06 2023-07-02T14:04:42.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:42.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797245+0000", 2023-07-02T14:04:42.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34785621999999999 2023-07-02T14:04:42.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145101+0000", 2023-07-02T14:04:42.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7762000000000006e-05 2023-07-02T14:04:42.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:42.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145169+0000", 2023-07-02T14:04:42.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013469400000000001 2023-07-02T14:04:42.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:42.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145304+0000", 2023-07-02T14:04:42.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089538039999999992 2023-07-02T14:04:42.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:42.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154257+0000", 2023-07-02T14:04:42.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104218 2023-07-02T14:04:42.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:42.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154362+0000", 2023-07-02T14:04:42.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4168000000000001e-05 2023-07-02T14:04:42.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:42.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154406+0000", 2023-07-02T14:04:42.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3098000000000001e-05 2023-07-02T14:04:42.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:42.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154429+0000", 2023-07-02T14:04:42.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002 2023-07-02T14:04:42.765 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.766 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.766 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.766 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4056 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3670016~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:42.766 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:42.766 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.798659665000002, 2023-07-02T14:04:42.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997, 2023-07-02T14:04:42.767 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:42.767 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:42.767 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:42.767 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:42.767 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:42.767 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4056 2023-07-02T14:04:42.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.767 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:42.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:42.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:42.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:42.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:42.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:42.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:42.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:42.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797191+0000", 2023-07-02T14:04:42.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0438999999999997e-05 2023-07-02T14:04:42.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:42.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797232+0000", 2023-07-02T14:04:42.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6929999999999998e-06 2023-07-02T14:04:42.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:42.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797237+0000", 2023-07-02T14:04:42.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9739999999999998e-06 2023-07-02T14:04:42.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:42.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797240+0000", 2023-07-02T14:04:42.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0243000000000002e-05 2023-07-02T14:04:42.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797270+0000", 2023-07-02T14:04:42.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9889999999999997e-06 2023-07-02T14:04:42.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:42.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797276+0000", 2023-07-02T14:04:42.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35358319799999999 2023-07-02T14:04:42.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150859+0000", 2023-07-02T14:04:42.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1357e-05 2023-07-02T14:04:42.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:42.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150940+0000", 2023-07-02T14:04:42.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014903 2023-07-02T14:04:42.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:42.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151089+0000", 2023-07-02T14:04:42.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033512519999999999 2023-07-02T14:04:42.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:42.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154441+0000", 2023-07-02T14:04:42.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001050945 2023-07-02T14:04:42.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:42.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155492+0000", 2023-07-02T14:04:42.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2523e-05 2023-07-02T14:04:42.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:42.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155524+0000", 2023-07-02T14:04:42.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.057e-05 2023-07-02T14:04:42.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:42.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155555+0000", 2023-07-02T14:04:42.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997 2023-07-02T14:04:42.777 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.777 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.777 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.778 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4088 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3735552~458752] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:42.778 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:42.778 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.798607469999993, 2023-07-02T14:04:42.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998, 2023-07-02T14:04:42.778 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:42.778 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:42.779 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:42.779 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:42.779 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:42.779 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4088 2023-07-02T14:04:42.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.779 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:42.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:42.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:42.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:42.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:42.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:42.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:42.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:42.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:42.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00038503499999999999 2023-07-02T14:04:42.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:42.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797629+0000", 2023-07-02T14:04:42.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8739999999999995e-06 2023-07-02T14:04:42.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:42.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797637+0000", 2023-07-02T14:04:42.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7160000000000001e-06 2023-07-02T14:04:42.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:42.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797640+0000", 2023-07-02T14:04:42.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4342999999999998e-05 2023-07-02T14:04:42.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797695+0000", 2023-07-02T14:04:42.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0939999999999996e-06 2023-07-02T14:04:42.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:42.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797701+0000", 2023-07-02T14:04:42.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35408540799999999 2023-07-02T14:04:42.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151786+0000", 2023-07-02T14:04:42.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1953000000000006e-05 2023-07-02T14:04:42.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:42.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151868+0000", 2023-07-02T14:04:42.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015162800000000001 2023-07-02T14:04:42.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:42.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.152020+0000", 2023-07-02T14:04:42.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054561870000000004 2023-07-02T14:04:42.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:42.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.157476+0000", 2023-07-02T14:04:42.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059153199999999997 2023-07-02T14:04:42.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:42.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158068+0000", 2023-07-02T14:04:42.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2799e-05 2023-07-02T14:04:42.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:42.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158090+0000", 2023-07-02T14:04:42.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7235999999999999e-05 2023-07-02T14:04:42.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:42.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158108+0000", 2023-07-02T14:04:42.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998 2023-07-02T14:04:42.789 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.789 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.789 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.789 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4110 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:42.790 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:42.790 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.798206192999999, 2023-07-02T14:04:42.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864, 2023-07-02T14:04:42.790 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:42.790 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:42.790 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:42.790 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:42.790 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:42.791 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4110 2023-07-02T14:04:42.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.791 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:42.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:42.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:42.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:42.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:42.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:42.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:42.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:42.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:42.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9331e-05 2023-07-02T14:04:42.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:42.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797684+0000", 2023-07-02T14:04:42.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.147e-06 2023-07-02T14:04:42.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:42.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797689+0000", 2023-07-02T14:04:42.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5020000000000001e-06 2023-07-02T14:04:42.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:42.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797693+0000", 2023-07-02T14:04:42.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5254999999999999e-05 2023-07-02T14:04:42.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797748+0000", 2023-07-02T14:04:42.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5520000000000005e-06 2023-07-02T14:04:42.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:42.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797757+0000", 2023-07-02T14:04:42.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35674245700000001 2023-07-02T14:04:42.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154499+0000", 2023-07-02T14:04:42.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0090000000000001e-06 2023-07-02T14:04:42.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:42.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154506+0000", 2023-07-02T14:04:42.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864 2023-07-02T14:04:42.797 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.797 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.797 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.798 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:42.798 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:42.798 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.798154107000002, 2023-07-02T14:04:42.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998, 2023-07-02T14:04:42.798 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:42.798 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:42.799 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:42.799 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:42.799 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:42.799 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:04:42.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.799 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:42.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:42.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:42.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:42.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:42.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:42.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:42.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:42.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:42.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001714819 2023-07-02T14:04:42.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:42.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799412+0000", 2023-07-02T14:04:42.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7779999999999996e-06 2023-07-02T14:04:42.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:42.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799421+0000", 2023-07-02T14:04:42.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3510000000000002e-06 2023-07-02T14:04:42.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:42.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799424+0000", 2023-07-02T14:04:42.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0275999999999999e-05 2023-07-02T14:04:42.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799464+0000", 2023-07-02T14:04:42.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8389999999999996e-06 2023-07-02T14:04:42.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:42.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799473+0000", 2023-07-02T14:04:42.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35507638800000002 2023-07-02T14:04:42.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:42.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154550+0000", 2023-07-02T14:04:42.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6309999999999998e-06 2023-07-02T14:04:42.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:42.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:42.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:42.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154559+0000", 2023-07-02T14:04:42.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998 2023-07-02T14:04:42.806 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.806 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.806 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.806 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:42.806 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:42.806 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:42.808 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:04:42.808 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:42.807+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:42.908 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:42.908+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:42.933 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:42.933 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:42.933 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:42.933 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:42.943 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:04:43.026 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:43.026+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:43.109 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:43.109+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:43.110 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:43.110 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:43.111 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:43.111 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:43.111 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:43.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:04:43.210 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:43.210+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:43.286 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:43.287 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:43.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:43.287 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:43.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.287 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:198 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:43.287 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:43.288 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 123.999019901, 2023-07-02T14:04:43.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999, 2023-07-02T14:04:43.288 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:43.288 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.288 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:43.288 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:43.288 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:43.288 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 198 2023-07-02T14:04:43.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.289 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:43.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:43.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:43.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:43.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:43.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286032+0000", 2023-07-02T14:04:43.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:43.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:43.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286031+0000", 2023-07-02T14:04:43.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00049286700000000002 2023-07-02T14:04:43.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:43.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286524+0000", 2023-07-02T14:04:43.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4938e-05 2023-07-02T14:04:43.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:43.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286539+0000", 2023-07-02T14:04:43.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0680000000000002e-06 2023-07-02T14:04:43.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286545+0000", 2023-07-02T14:04:43.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026497718 2023-07-02T14:04:43.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313043+0000", 2023-07-02T14:04:43.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0884999999999998e-05 2023-07-02T14:04:43.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:43.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313094+0000", 2023-07-02T14:04:43.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014109999999999999 2023-07-02T14:04:43.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:43.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.313235+0000", 2023-07-02T14:04:43.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048258980000000003 2023-07-02T14:04:43.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:43.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318061+0000", 2023-07-02T14:04:43.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014346350000000001 2023-07-02T14:04:43.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:43.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319495+0000", 2023-07-02T14:04:43.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1986e-05 2023-07-02T14:04:43.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:43.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319547+0000", 2023-07-02T14:04:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0457e-05 2023-07-02T14:04:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.319578+0000", 2023-07-02T14:04:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033545829999999999 2023-07-02T14:04:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.297 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:207 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 123.998209993, 2023-07-02T14:04:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003, 2023-07-02T14:04:43.298 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 207 2023-07-02T14:04:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:43.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:43.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286842+0000", 2023-07-02T14:04:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.286841+0000", 2023-07-02T14:04:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00068375100000000002 2023-07-02T14:04:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287525+0000", 2023-07-02T14:04:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1601e-05 2023-07-02T14:04:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:43.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287537+0000", 2023-07-02T14:04:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6409999999999996e-06 2023-07-02T14:04:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287543+0000", 2023-07-02T14:04:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027245844000000002 2023-07-02T14:04:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314789+0000", 2023-07-02T14:04:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8910000000000003e-05 2023-07-02T14:04:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.314858+0000", 2023-07-02T14:04:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00030992599999999997 2023-07-02T14:04:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315168+0000", 2023-07-02T14:04:43.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029122869999999999 2023-07-02T14:04:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.318080+0000", 2023-07-02T14:04:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019963020000000001 2023-07-02T14:04:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320077+0000", 2023-07-02T14:04:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5116000000000005e-05 2023-07-02T14:04:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320142+0000", 2023-07-02T14:04:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3793000000000004e-05 2023-07-02T14:04:43.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320205+0000", 2023-07-02T14:04:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033363449000000003 2023-07-02T14:04:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:236 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2424832~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 123.99749781, 2023-07-02T14:04:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003, 2023-07-02T14:04:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.309 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2023-07-02T14:04:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:43.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287553+0000", 2023-07-02T14:04:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0667e-05 2023-07-02T14:04:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:43.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287614+0000", 2023-07-02T14:04:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0454e-05 2023-07-02T14:04:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287624+0000", 2023-07-02T14:04:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8564999999999999e-05 2023-07-02T14:04:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287643+0000", 2023-07-02T14:04:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027977596 2023-07-02T14:04:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315621+0000", 2023-07-02T14:04:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000201728 2023-07-02T14:04:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315822+0000", 2023-07-02T14:04:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018436599999999999 2023-07-02T14:04:43.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316007+0000", 2023-07-02T14:04:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042423440000000003 2023-07-02T14:04:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320249+0000", 2023-07-02T14:04:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001452633 2023-07-02T14:04:43.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321702+0000", 2023-07-02T14:04:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3616000000000003e-05 2023-07-02T14:04:43.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321755+0000", 2023-07-02T14:04:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840999999999999e-05 2023-07-02T14:04:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:43.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321793+0000", 2023-07-02T14:04:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003 2023-07-02T14:04:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:251 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 123.99740227300001, 2023-07-02T14:04:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001, 2023-07-02T14:04:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:43.321 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 251 2023-07-02T14:04:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:43.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287649+0000", 2023-07-02T14:04:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3232000000000001e-05 2023-07-02T14:04:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287702+0000", 2023-07-02T14:04:43.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0010000000000007e-06 2023-07-02T14:04:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287711+0000", 2023-07-02T14:04:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3569999999999997e-06 2023-07-02T14:04:43.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287717+0000", 2023-07-02T14:04:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028686454 2023-07-02T14:04:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316403+0000", 2023-07-02T14:04:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7741000000000002e-05 2023-07-02T14:04:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316461+0000", 2023-07-02T14:04:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4313000000000004e-05 2023-07-02T14:04:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316545+0000", 2023-07-02T14:04:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003734032 2023-07-02T14:04:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320279+0000", 2023-07-02T14:04:43.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015237219999999999 2023-07-02T14:04:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321803+0000", 2023-07-02T14:04:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1927e-05 2023-07-02T14:04:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321845+0000", 2023-07-02T14:04:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5398999999999998e-05 2023-07-02T14:04:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321880+0000", 2023-07-02T14:04:43.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001 2023-07-02T14:04:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:255 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2555904~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:43.331 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 123.996987545, 2023-07-02T14:04:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002, 2023-07-02T14:04:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 255 2023-07-02T14:04:43.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288064+0000", 2023-07-02T14:04:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288063+0000", 2023-07-02T14:04:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4459999999999997e-05 2023-07-02T14:04:43.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288118+0000", 2023-07-02T14:04:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0998999999999999e-05 2023-07-02T14:04:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288129+0000", 2023-07-02T14:04:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3219999999999998e-06 2023-07-02T14:04:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.288135+0000", 2023-07-02T14:04:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029137386000000001 2023-07-02T14:04:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317273+0000", 2023-07-02T14:04:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7819999999999997e-05 2023-07-02T14:04:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317350+0000", 2023-07-02T14:04:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000174864 2023-07-02T14:04:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.317525+0000", 2023-07-02T14:04:43.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027943519999999999 2023-07-02T14:04:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320320+0000", 2023-07-02T14:04:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015681460000000001 2023-07-02T14:04:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321888+0000", 2023-07-02T14:04:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9508999999999997e-05 2023-07-02T14:04:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321927+0000", 2023-07-02T14:04:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5428e-05 2023-07-02T14:04:43.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321963+0000", 2023-07-02T14:04:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033898282000000002 2023-07-02T14:04:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:04:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 102.177376999, 2023-07-02T14:04:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:04:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:43.342 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:04:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:43.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:04:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:04:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:04:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:04:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:04:43.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:04:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:04:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:04:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:04:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:04:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:04:43.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:04:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:04:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:04:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:04:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:04:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:04:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:04:43.350 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:04:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 102.177316261, 2023-07-02T14:04:43.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:04:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:04:43.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:04:43.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:04:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:04:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:04:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:04:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:04:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:04:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:04:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:04:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:04:43.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:04:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:04:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:04:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:04:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:43.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:04:43.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:04:43.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:04:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:04:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:04:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:04:43.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:04:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:04:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:04:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:04:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:04:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:04:43.363 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.946861931000001, 2023-07-02T14:04:43.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:04:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:04:43.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:43.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:04:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:04:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:04:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:04:43.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:04:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:04:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:04:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:04:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:04:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:04:43.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:04:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:04:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:43.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:04:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:04:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:04:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:04:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:04:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:04:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:04:43.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:04:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:04:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:04:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.374 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2941 2.3 2:d3762e63:::rbd_data.125de1449605.00000000000000f8:head [write 65536~2031616] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.817758976999997, 2023-07-02T14:04:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002, 2023-07-02T14:04:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:43.375 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.376 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:43.376 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:43.376 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:43.376 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2941 2023-07-02T14:04:43.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.376 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:43.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:43.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:43.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:43.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467291+0000", 2023-07-02T14:04:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043022859999999998 2023-07-02T14:04:43.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471593+0000", 2023-07-02T14:04:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0999999999999999e-05 2023-07-02T14:04:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:43.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471614+0000", 2023-07-02T14:04:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0097e-05 2023-07-02T14:04:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471625+0000", 2023-07-02T14:04:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069808499999999998 2023-07-02T14:04:43.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472323+0000", 2023-07-02T14:04:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135635 2023-07-02T14:04:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:43.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472458+0000", 2023-07-02T14:04:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223365 2023-07-02T14:04:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472682+0000", 2023-07-02T14:04:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013761944999999999 2023-07-02T14:04:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.486444+0000", 2023-07-02T14:04:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016859747000000001 2023-07-02T14:04:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503303+0000", 2023-07-02T14:04:43.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6635000000000001e-05 2023-07-02T14:04:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503380+0000", 2023-07-02T14:04:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000161803 2023-07-02T14:04:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503542+0000", 2023-07-02T14:04:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002 2023-07-02T14:04:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2950 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 1507328~786432] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.809585669000001, 2023-07-02T14:04:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999, 2023-07-02T14:04:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:43.386 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2950 2023-07-02T14:04:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:43.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001147854 2023-07-02T14:04:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476614+0000", 2023-07-02T14:04:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.802e-05 2023-07-02T14:04:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476632+0000", 2023-07-02T14:04:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1860000000000003e-06 2023-07-02T14:04:43.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476639+0000", 2023-07-02T14:04:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061211240000000004 2023-07-02T14:04:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482760+0000", 2023-07-02T14:04:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8833999999999996e-05 2023-07-02T14:04:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482859+0000", 2023-07-02T14:04:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019270600000000001 2023-07-02T14:04:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.483051+0000", 2023-07-02T14:04:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060385630000000003 2023-07-02T14:04:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489090+0000", 2023-07-02T14:04:43.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024506918999999999 2023-07-02T14:04:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513597+0000", 2023-07-02T14:04:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9672999999999995e-05 2023-07-02T14:04:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513667+0000", 2023-07-02T14:04:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024329500000000001 2023-07-02T14:04:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513910+0000", 2023-07-02T14:04:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999 2023-07-02T14:04:43.395 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2951 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2686976~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:43.396 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.808404890999995, 2023-07-02T14:04:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999, 2023-07-02T14:04:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:43.397 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2951 2023-07-02T14:04:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:43.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:43.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476646+0000", 2023-07-02T14:04:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7639000000000003e-05 2023-07-02T14:04:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476684+0000", 2023-07-02T14:04:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3508000000000002e-05 2023-07-02T14:04:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476747+0000", 2023-07-02T14:04:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8499999999999994e-06 2023-07-02T14:04:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476755+0000", 2023-07-02T14:04:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078346160000000008 2023-07-02T14:04:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484590+0000", 2023-07-02T14:04:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7206000000000004e-05 2023-07-02T14:04:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484687+0000", 2023-07-02T14:04:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016243099999999999 2023-07-02T14:04:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484849+0000", 2023-07-02T14:04:43.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042482860000000004 2023-07-02T14:04:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489097+0000", 2023-07-02T14:04:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.02489504 2023-07-02T14:04:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513992+0000", 2023-07-02T14:04:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9972000000000001e-05 2023-07-02T14:04:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514052+0000", 2023-07-02T14:04:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149264 2023-07-02T14:04:43.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514202+0000", 2023-07-02T14:04:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999 2023-07-02T14:04:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2952 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2883584~12288] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.808291056000002, 2023-07-02T14:04:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651, 2023-07-02T14:04:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.407 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:43.408 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:43.408 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:43.408 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2952 2023-07-02T14:04:43.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:43.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476760+0000", 2023-07-02T14:04:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3664e-05 2023-07-02T14:04:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476774+0000", 2023-07-02T14:04:43.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.166e-06 2023-07-02T14:04:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476783+0000", 2023-07-02T14:04:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8720000000000001e-06 2023-07-02T14:04:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476788+0000", 2023-07-02T14:04:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087180529999999999 2023-07-02T14:04:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485506+0000", 2023-07-02T14:04:43.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5680999999999994e-05 2023-07-02T14:04:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485602+0000", 2023-07-02T14:04:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000222365 2023-07-02T14:04:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485824+0000", 2023-07-02T14:04:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003277379 2023-07-02T14:04:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489101+0000", 2023-07-02T14:04:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025151739999999999 2023-07-02T14:04:43.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514253+0000", 2023-07-02T14:04:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4749000000000002e-05 2023-07-02T14:04:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:43.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514288+0000", 2023-07-02T14:04:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130634 2023-07-02T14:04:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514419+0000", 2023-07-02T14:04:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651 2023-07-02T14:04:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.417 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9620 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 589824~65536] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:04:43.418 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:04:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.262847333, 2023-07-02T14:04:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001, 2023-07-02T14:04:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:43.419 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9620 2023-07-02T14:04:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:04:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:04:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022203+0000", 2023-07-02T14:04:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0346000000000001e-05 2023-07-02T14:04:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022263+0000", 2023-07-02T14:04:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4657999999999999e-05 2023-07-02T14:04:43.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022278+0000", 2023-07-02T14:04:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:04:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022285+0000", 2023-07-02T14:04:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0096000000000003e-05 2023-07-02T14:04:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022335+0000", 2023-07-02T14:04:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2334000000000001e-05 2023-07-02T14:04:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022347+0000", 2023-07-02T14:04:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095653701999999993 2023-07-02T14:04:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118001+0000", 2023-07-02T14:04:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000185797 2023-07-02T14:04:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118187+0000", 2023-07-02T14:04:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026397900000000001 2023-07-02T14:04:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118451+0000", 2023-07-02T14:04:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051832800000000002 2023-07-02T14:04:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.123634+0000", 2023-07-02T14:04:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001227332 2023-07-02T14:04:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124861+0000", 2023-07-02T14:04:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9493999999999998e-05 2023-07-02T14:04:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124901+0000", 2023-07-02T14:04:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0462000000000001e-05 2023-07-02T14:04:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124931+0000", 2023-07-02T14:04:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001 2023-07-02T14:04:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.430 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9621 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 655360~1966080] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:04:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:04:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.262659951, 2023-07-02T14:04:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999, 2023-07-02T14:04:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:43.431 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9621 2023-07-02T14:04:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:04:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:43.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:04:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:43.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022391+0000", 2023-07-02T14:04:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032053939999999999 2023-07-02T14:04:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025597+0000", 2023-07-02T14:04:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4307999999999999e-05 2023-07-02T14:04:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025611+0000", 2023-07-02T14:04:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.032e-06 2023-07-02T14:04:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025617+0000", 2023-07-02T14:04:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8977999999999998e-05 2023-07-02T14:04:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025656+0000", 2023-07-02T14:04:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:04:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025663+0000", 2023-07-02T14:04:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095220739999999998 2023-07-02T14:04:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120884+0000", 2023-07-02T14:04:43.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2525000000000001e-05 2023-07-02T14:04:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120956+0000", 2023-07-02T14:04:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016232400000000001 2023-07-02T14:04:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.121119+0000", 2023-07-02T14:04:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044007380000000004 2023-07-02T14:04:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:43.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.125519+0000", 2023-07-02T14:04:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000797209 2023-07-02T14:04:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126317+0000", 2023-07-02T14:04:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1228000000000004e-05 2023-07-02T14:04:43.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126378+0000", 2023-07-02T14:04:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978000000000003e-05 2023-07-02T14:04:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:43.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126430+0000", 2023-07-02T14:04:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999 2023-07-02T14:04:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:11434 2.1 2:8269d003:::rbd_data.125de1449605.00000000000000ce:head [write 65536~4128768] snapc 1a=[1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e107)", 2023-07-02T14:04:43.442 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:04:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.367221682, 2023-07-02T14:04:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036409966000000002, 2023-07-02T14:04:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:43.443 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11434 2023-07-02T14:04:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:04:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:04:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917828+0000", 2023-07-02T14:04:43.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052044650000000001 2023-07-02T14:04:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923033+0000", 2023-07-02T14:04:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0097000000000001e-05 2023-07-02T14:04:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923053+0000", 2023-07-02T14:04:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.49e-06 2023-07-02T14:04:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923061+0000", 2023-07-02T14:04:43.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0090999999999998e-05 2023-07-02T14:04:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923121+0000", 2023-07-02T14:04:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000114095 2023-07-02T14:04:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923236+0000", 2023-07-02T14:04:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020094199999999999 2023-07-02T14:04:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:43.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923436+0000", 2023-07-02T14:04:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088693520000000005 2023-07-02T14:04:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.932306+0000", 2023-07-02T14:04:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021874112000000001 2023-07-02T14:04:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954180+0000", 2023-07-02T14:04:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4433e-05 2023-07-02T14:04:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:43.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954214+0000", 2023-07-02T14:04:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5884000000000002e-05 2023-07-02T14:04:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954240+0000", 2023-07-02T14:04:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036409966000000002 2023-07-02T14:04:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.452 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:189 2.3 2:c9621172:::rbd_data.125de1449605.00000000000000c0:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:04:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:04:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.112372659, 2023-07-02T14:04:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001, 2023-07-02T14:04:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:04:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:04:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:04:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:43.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:04:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:04:43.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:04:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2023-07-02T14:04:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172682+0000", 2023-07-02T14:04:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0159999999999997e-06 2023-07-02T14:04:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172688+0000", 2023-07-02T14:04:43.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0010000000000003e-06 2023-07-02T14:04:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172692+0000", 2023-07-02T14:04:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026444333 2023-07-02T14:04:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199136+0000", 2023-07-02T14:04:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2230000000000004e-05 2023-07-02T14:04:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199218+0000", 2023-07-02T14:04:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012043595000000001 2023-07-02T14:04:43.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211262+0000", 2023-07-02T14:04:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001 2023-07-02T14:04:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.460 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:199 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:04:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:04:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.112329116, 2023-07-02T14:04:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002, 2023-07-02T14:04:43.461 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:04:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:04:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 199 2023-07-02T14:04:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:43.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:04:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:04:43.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172722+0000", 2023-07-02T14:04:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1250000000000001e-06 2023-07-02T14:04:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172725+0000", 2023-07-02T14:04:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5670000000000002e-06 2023-07-02T14:04:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172731+0000", 2023-07-02T14:04:43.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9310000000000001e-06 2023-07-02T14:04:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172735+0000", 2023-07-02T14:04:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030699852999999999 2023-07-02T14:04:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203435+0000", 2023-07-02T14:04:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6827e-05 2023-07-02T14:04:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203532+0000", 2023-07-02T14:04:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076955230000000001 2023-07-02T14:04:43.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211227+0000", 2023-07-02T14:04:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002 2023-07-02T14:04:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:213 2.1 2:8fbcc55d:::rbd_data.125de1449605.00000000000000d5:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e110)", 2023-07-02T14:04:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.241083+0000", 2023-07-02T14:04:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.043968530000001, 2023-07-02T14:04:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036164654999999997, 2023-07-02T14:04:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:43.469 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:04:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:04:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 213 2023-07-02T14:04:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.470 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.241083+0000", 2023-07-02T14:04:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:43.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.241083+0000", 2023-07-02T14:04:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.241083+0000", 2023-07-02T14:04:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.525e-06 2023-07-02T14:04:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.241086+0000", 2023-07-02T14:04:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0620000000000003e-06 2023-07-02T14:04:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.241093+0000", 2023-07-02T14:04:43.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.799e-06 2023-07-02T14:04:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.241096+0000", 2023-07-02T14:04:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030109975000000001 2023-07-02T14:04:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.271206+0000", 2023-07-02T14:04:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134172 2023-07-02T14:04:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.271341+0000", 2023-07-02T14:04:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059075309999999997 2023-07-02T14:04:43.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.277248+0000", 2023-07-02T14:04:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036164654999999997 2023-07-02T14:04:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.476 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:280 2.1 2:94df0216:::rbd_data.125de1449605.0000000000000117:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:04:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555448+0000", 2023-07-02T14:04:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.7296035009999997, 2023-07-02T14:04:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989515, 2023-07-02T14:04:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:43.477 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:04:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:04:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 280 2023-07-02T14:04:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555448+0000", 2023-07-02T14:04:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555448+0000", 2023-07-02T14:04:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:43.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555448+0000", 2023-07-02T14:04:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1400000000000002e-06 2023-07-02T14:04:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555452+0000", 2023-07-02T14:04:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8199999999999999e-06 2023-07-02T14:04:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555459+0000", 2023-07-02T14:04:43.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7520000000000004e-06 2023-07-02T14:04:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555463+0000", 2023-07-02T14:04:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027814050999999999 2023-07-02T14:04:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.583277+0000", 2023-07-02T14:04:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192244 2023-07-02T14:04:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.583470+0000", 2023-07-02T14:04:43.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007968269 2023-07-02T14:04:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591438+0000", 2023-07-02T14:04:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989515 2023-07-02T14:04:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.484 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:281 2.1 2:901d5a95:::rbd_data.125de1449605.0000000000000118:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:04:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:04:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.7295838400000001, 2023-07-02T14:04:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002, 2023-07-02T14:04:43.485 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:04:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:04:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:04:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:43.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:04:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:04:43.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:04:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0469999999999998e-06 2023-07-02T14:04:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555471+0000", 2023-07-02T14:04:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8860000000000001e-06 2023-07-02T14:04:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555476+0000", 2023-07-02T14:04:43.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3160000000000002e-06 2023-07-02T14:04:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555483+0000", 2023-07-02T14:04:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989525000000001 2023-07-02T14:04:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:43.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591472+0000", 2023-07-02T14:04:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3708999999999996e-05 2023-07-02T14:04:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591536+0000", 2023-07-02T14:04:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055109720000000003 2023-07-02T14:04:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:43.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.597047+0000", 2023-07-02T14:04:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002 2023-07-02T14:04:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:43.493 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:43.494 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:43.494 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:04:43.495 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:04:43.496 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:43.495+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:43.595 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:43.595+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:43.634 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:43.634 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:43.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.635 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPushReply(2.7 118/117 [PushReplyOp(2:e57d3f55:::rbd_data.125de1449605.0000000000000134:head)])", 2023-07-02T14:04:43.635 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:43.634397+0000", 2023-07-02T14:04:43.635 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.00014586499999999999, 2023-07-02T14:04:43.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000166047, 2023-07-02T14:04:43.635 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:43.635 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "reached pg", 2023-07-02T14:04:43.636 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:43.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:43.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:43.634397+0000", 2023-07-02T14:04:43.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:43.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:43.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:43.634397+0000", 2023-07-02T14:04:43.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:43.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:43.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:43.634396+0000", 2023-07-02T14:04:43.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0789999999999997e-06 2023-07-02T14:04:43.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:43.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:43.634403+0000", 2023-07-02T14:04:43.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1751e-05 2023-07-02T14:04:43.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:43.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:43.634414+0000", 2023-07-02T14:04:43.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0989999999999998e-06 2023-07-02T14:04:43.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:43.634419+0000", 2023-07-02T14:04:43.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6835999999999997e-05 2023-07-02T14:04:43.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:43.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:43.634466+0000", 2023-07-02T14:04:43.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8996000000000001e-05 2023-07-02T14:04:43.640 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:43.640 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:43.640 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:43.640 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:43.641 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2023-07-02T14:04:43.641 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 1 2023-07-02T14:04:43.641 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:43.645 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:04:43.696 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:43.696+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:43.782 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:43.782 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:43.782 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:04:43.782 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:04:43.783 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:43.793 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:04:43.797 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:43.797+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:43.898 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:43.898+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:43.921 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:43.921 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:04:43.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:04:43.922 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:43.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.922 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:04:43.922 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:43.922 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 64.022263761000005, 2023-07-02T14:04:43.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:04:43.922 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:43.922 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.923 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:43.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:43.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:43.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:43.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:43.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:43.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:43.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:43.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:04:43.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:04:43.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:43.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:04:43.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:04:43.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:43.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:04:43.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:04:43.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:04:43.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:04:43.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:43.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:04:43.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:04:43.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:43.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:04:43.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:04:43.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:43.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:04:43.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:04:43.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:43.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:04:43.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:04:43.930 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:43.931 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:43.931 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:43.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.931 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:04:43.931 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:43.931 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 64.018178043000006, 2023-07-02T14:04:43.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:04:43.932 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:43.932 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.932 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:43.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:43.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:43.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:43.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:43.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:43.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:43.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:43.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:04:43.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:04:43.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:43.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:04:43.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:04:43.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:43.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:04:43.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:04:43.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:04:43.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:04:43.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:43.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:04:43.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:04:43.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:43.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:04:43.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:04:43.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:43.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:04:43.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:04:43.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:43.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:04:43.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:04:43.939 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:43.939 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:43.939 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:43.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.939 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:04:43.939 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:43.940 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 64.014313262000002, 2023-07-02T14:04:43.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:04:43.940 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:43.940 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.940 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:43.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:43.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:43.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:43.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:43.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:43.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:43.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:43.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:04:43.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:04:43.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:43.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:04:43.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:04:43.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:43.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:04:43.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:04:43.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:04:43.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:04:43.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:43.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:04:43.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:04:43.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:43.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:04:43.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:04:43.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:43.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:04:43.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:04:43.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:43.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:04:43.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:04:43.947 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:43.947 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:43.947 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:43.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.948 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:04:43.948 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:43.948 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 64.010789001000006, 2023-07-02T14:04:43.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:04:43.948 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:43.948 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.949 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:43.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:43.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:43.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:43.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:43.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:43.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:43.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:43.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:04:43.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:04:43.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:43.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:04:43.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:04:43.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:43.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:04:43.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:04:43.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:04:43.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:04:43.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:43.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:04:43.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:04:43.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.954 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:43.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:04:43.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:04:43.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:43.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:04:43.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:04:43.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:43.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:04:43.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:04:43.956 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:43.956 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:43.956 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:43.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.956 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6124 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1328, mlcod=81'1319)", 2023-07-02T14:04:43.957 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:43.957 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 64.007273617999999, 2023-07-02T14:04:43.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019096974999999999, 2023-07-02T14:04:43.957 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:43.957 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.957 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:43.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:43.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:43.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:43.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:43.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912604+0000", 2023-07-02T14:04:43.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:43.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:43.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912602+0000", 2023-07-02T14:04:43.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6741999999999999e-05 2023-07-02T14:04:43.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:43.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912629+0000", 2023-07-02T14:04:43.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5163e-05 2023-07-02T14:04:43.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:43.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912644+0000", 2023-07-02T14:04:43.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4800000000000001e-06 2023-07-02T14:04:43.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.912653+0000", 2023-07-02T14:04:43.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071988909999999998 2023-07-02T14:04:43.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:43.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.919851+0000", 2023-07-02T14:04:43.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2949999999999999e-05 2023-07-02T14:04:43.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:43.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.919874+0000", 2023-07-02T14:04:43.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011800804 2023-07-02T14:04:43.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:43.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931675+0000", 2023-07-02T14:04:43.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6033999999999998e-05 2023-07-02T14:04:43.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:43.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.931701+0000", 2023-07-02T14:04:43.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019096974999999999 2023-07-02T14:04:43.965 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:43.965 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:43.966 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:43.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.966 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6139 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1337, mlcod=81'1333)", 2023-07-02T14:04:43.967 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:43.967 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.980583758000002, 2023-07-02T14:04:43.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018571681, 2023-07-02T14:04:43.967 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:43.967 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.968 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:43.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:43.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:43.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:43.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939294+0000", 2023-07-02T14:04:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.939293+0000", 2023-07-02T14:04:43.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003764613 2023-07-02T14:04:43.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:43.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943058+0000", 2023-07-02T14:04:43.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5797000000000001e-05 2023-07-02T14:04:43.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:43.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943074+0000", 2023-07-02T14:04:43.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5089999999999996e-06 2023-07-02T14:04:43.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943081+0000", 2023-07-02T14:04:43.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014727100000000001 2023-07-02T14:04:43.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:43.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943228+0000", 2023-07-02T14:04:43.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2246000000000003e-05 2023-07-02T14:04:43.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:43.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.943261+0000", 2023-07-02T14:04:43.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014576317 2023-07-02T14:04:43.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:43.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957837+0000", 2023-07-02T14:04:43.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8962000000000002e-05 2023-07-02T14:04:43.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:43.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957866+0000", 2023-07-02T14:04:43.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018571681 2023-07-02T14:04:43.975 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:43.975 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:43.975 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:43.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.975 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:04:43.975 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:43.976 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.975084201999998, 2023-07-02T14:04:43.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:04:43.976 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:43.976 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.976 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:43.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:43.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:43.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:43.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:43.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:43.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:43.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:43.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:04:43.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:04:43.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:43.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:04:43.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:04:43.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:43.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:04:43.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:04:43.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:04:43.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:04:43.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:43.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:04:43.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:04:43.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:43.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:04:43.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:04:43.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:43.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:04:43.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:04:43.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:43.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:04:43.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:04:43.984 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:43.984 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:43.984 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:43.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.984 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:04:43.984 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:43.985 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.970426844000002, 2023-07-02T14:04:43.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:04:43.985 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:43.985 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.985 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:43.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:43.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:43.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:43.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:43.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:43.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:43.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:43.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:04:43.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:04:43.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:43.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:04:43.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:04:43.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:43.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:04:43.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:04:43.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:04:43.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:04:43.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:43.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:04:43.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:04:43.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:43.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:04:43.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:04:43.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:43.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:04:43.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:04:43.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:43.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:04:43.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:04:43.992 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:43.992 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:43.992 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:43.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.992 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:04:43.992 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:43.993 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.969486981000003, 2023-07-02T14:04:43.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:04:43.993 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:43.993 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:43.993 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:43.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:43.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:43.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:43.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:43.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:43.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:43.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:43.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:04:43.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:04:43.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:43.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:04:43.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:04:43.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:43.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:04:43.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:04:43.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:43.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:04:43.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:04:43.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:43.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:04:43.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:04:43.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:43.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:04:43.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:04:43.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:43.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:04:43.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:04:43.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:43.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:43.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:43.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:04:43.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:04:44.000 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.000 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.000 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.000 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:04:44.000 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:44.000 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.968089102999997, 2023-07-02T14:04:44.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:04:44.001 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.001 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.001 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:44.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:44.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:44.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:04:44.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:04:44.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:04:44.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:04:44.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:04:44.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:04:44.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:04:44.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:04:44.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:04:44.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:04:44.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:04:44.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:04:44.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:04:44.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:04:44.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:04:44.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:04:44.007 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.008 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.008 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.008 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6150 2.0 e81/46 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 81'1345, mlcod=81'1338)", 2023-07-02T14:04:44.008 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:44.008 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 63.960066156000003, 2023-07-02T14:04:44.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019172812000000001, 2023-07-02T14:04:44.009 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.009 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.009 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:44.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959812+0000", 2023-07-02T14:04:44.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:44.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.959810+0000", 2023-07-02T14:04:44.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048438650000000001 2023-07-02T14:04:44.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964653+0000", 2023-07-02T14:04:44.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2136e-05 2023-07-02T14:04:44.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964666+0000", 2023-07-02T14:04:44.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2169999999999999e-06 2023-07-02T14:04:44.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.964672+0000", 2023-07-02T14:04:44.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00047561800000000003 2023-07-02T14:04:44.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965147+0000", 2023-07-02T14:04:44.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5891e-05 2023-07-02T14:04:44.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.965173+0000", 2023-07-02T14:04:44.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013770973000000001 2023-07-02T14:04:44.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978944+0000", 2023-07-02T14:04:44.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0256999999999998e-05 2023-07-02T14:04:44.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.978985+0000", 2023-07-02T14:04:44.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019172812000000001 2023-07-02T14:04:44.017 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.017 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.017 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.018 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:04:44.018 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:44.018 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 42.557898606999998, 2023-07-02T14:04:44.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:04:44.018 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.018 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.018 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:44.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:44.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:44.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:04:44.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:04:44.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:04:44.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:04:44.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:04:44.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:04:44.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:04:44.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:04:44.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:04:44.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:04:44.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:04:44.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:04:44.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:04:44.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:04:44.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:04:44.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:04:44.026 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.026 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.026 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.026 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:44.026 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:44.026 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 41.191548922999999, 2023-07-02T14:04:44.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:04:44.027 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.027 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.027 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.027 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.027 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.027 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:04:44.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.028 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:44.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:44.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:44.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:04:44.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:04:44.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:04:44.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:04:44.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:04:44.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:04:44.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:04:44.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:04:44.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:04:44.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:04:44.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:04:44.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:04:44.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:04:44.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:04:44.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:04:44.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:04:44.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:04:44.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:04:44.035 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.035 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.035 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.036 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7706 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 0~53248] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:44.036 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:44.036 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 41.187622175999998, 2023-07-02T14:04:44.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019507096000000002, 2023-07-02T14:04:44.036 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.037 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.037 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.037 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.037 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.037 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7706 2023-07-02T14:04:44.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.037 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:44.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:44.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:44.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732254+0000", 2023-07-02T14:04:44.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4764999999999997e-05 2023-07-02T14:04:44.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732289+0000", 2023-07-02T14:04:44.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0166999999999999e-05 2023-07-02T14:04:44.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732299+0000", 2023-07-02T14:04:44.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5310000000000001e-06 2023-07-02T14:04:44.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732305+0000", 2023-07-02T14:04:44.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060619599999999999 2023-07-02T14:04:44.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738367+0000", 2023-07-02T14:04:44.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011864 2023-07-02T14:04:44.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738486+0000", 2023-07-02T14:04:44.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013098291 2023-07-02T14:04:44.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751584+0000", 2023-07-02T14:04:44.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7800000000000002e-05 2023-07-02T14:04:44.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751642+0000", 2023-07-02T14:04:44.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000121198 2023-07-02T14:04:44.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751763+0000", 2023-07-02T14:04:44.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019507096000000002 2023-07-02T14:04:44.044 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7707 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 3145728~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:44.046 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 41.187565456000002, 2023-07-02T14:04:44.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999, 2023-07-02T14:04:44.046 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.046 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.046 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.047 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.047 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.047 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7707 2023-07-02T14:04:44.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.047 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:44.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:44.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:44.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:44.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5017e-05 2023-07-02T14:04:44.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732347+0000", 2023-07-02T14:04:44.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5821e-05 2023-07-02T14:04:44.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732363+0000", 2023-07-02T14:04:44.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1270000000000002e-06 2023-07-02T14:04:44.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732368+0000", 2023-07-02T14:04:44.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008444666 2023-07-02T14:04:44.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740812+0000", 2023-07-02T14:04:44.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011266599999999999 2023-07-02T14:04:44.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740925+0000", 2023-07-02T14:04:44.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010848808999999999 2023-07-02T14:04:44.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751774+0000", 2023-07-02T14:04:44.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8421999999999997e-05 2023-07-02T14:04:44.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751832+0000", 2023-07-02T14:04:44.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107453 2023-07-02T14:04:44.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751940+0000", 2023-07-02T14:04:44.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999 2023-07-02T14:04:44.054 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.055 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.055 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.055 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:44.055 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:44.055 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 41.187501578999999, 2023-07-02T14:04:44.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:04:44.056 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.056 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.056 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.056 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.057 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.057 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:04:44.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.057 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:44.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:44.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:44.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:44.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:04:44.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:04:44.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:04:44.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:04:44.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:04:44.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:04:44.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:04:44.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:04:44.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:04:44.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:04:44.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:04:44.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:04:44.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:04:44.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:04:44.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:04:44.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:04:44.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:04:44.064 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.064 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.065 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.065 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:44.065 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:44.065 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 41.182937754000001, 2023-07-02T14:04:44.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:04:44.065 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.066 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.066 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.066 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.066 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.066 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:04:44.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.066 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:44.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:44.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:44.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:04:44.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:04:44.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:04:44.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:04:44.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:04:44.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:04:44.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:04:44.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:04:44.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:04:44.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:04:44.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:04:44.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:04:44.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:04:44.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:04:44.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:04:44.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:04:44.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:04:44.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:04:44.074 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.074 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.075 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.075 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:44.075 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:44.075 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 41.182887888000003, 2023-07-02T14:04:44.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:04:44.075 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.076 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.076 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.076 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.076 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.076 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:04:44.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.076 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:44.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:44.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:44.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:04:44.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:04:44.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:04:44.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:04:44.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:04:44.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:04:44.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:04:44.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:04:44.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:04:44.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:04:44.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:04:44.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:04:44.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:04:44.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:04:44.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:04:44.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:04:44.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:04:44.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:04:44.084 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.084 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.084 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.085 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:44.085 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:44.085 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 41.181848895000002, 2023-07-02T14:04:44.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:04:44.085 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.085 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.086 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.086 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.086 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.086 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:04:44.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.086 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:44.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:44.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:44.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:04:44.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:04:44.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:04:44.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:04:44.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:04:44.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:04:44.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:04:44.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:04:44.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:04:44.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:04:44.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:04:44.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:04:44.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:04:44.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:04:44.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:04:44.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:04:44.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:04:44.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:04:44.094 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.094 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.094 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.095 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7993 2.7 2:fe72db53:::rbd_data.125de1449605.0000000000000072:head [write 65536~3276800] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:44.095 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:44.095 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.994890847000001, 2023-07-02T14:04:44.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919, 2023-07-02T14:04:44.095 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.095 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.096 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.096 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.096 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.096 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7993 2023-07-02T14:04:44.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.096 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:44.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:44.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:44.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924986+0000", 2023-07-02T14:04:44.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049542070000000004 2023-07-02T14:04:44.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929940+0000", 2023-07-02T14:04:44.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9740000000000001e-05 2023-07-02T14:04:44.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929960+0000", 2023-07-02T14:04:44.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.311e-05 2023-07-02T14:04:44.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929973+0000", 2023-07-02T14:04:44.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9629000000000002e-05 2023-07-02T14:04:44.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930023+0000", 2023-07-02T14:04:44.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012751799999999999 2023-07-02T14:04:44.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930150+0000", 2023-07-02T14:04:44.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016449042000000001 2023-07-02T14:04:44.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946599+0000", 2023-07-02T14:04:44.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3241e-05 2023-07-02T14:04:44.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946663+0000", 2023-07-02T14:04:44.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000102296 2023-07-02T14:04:44.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946765+0000", 2023-07-02T14:04:44.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919 2023-07-02T14:04:44.104 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.104 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.104 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.104 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.104 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.105 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:44.106 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:04:44.106 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:44.106+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:44.206 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:44.206+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:44.238 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:44.239 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:44.239 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:04:44.239 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:44.249 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:04:44.283 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:04:44.327 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:44.327+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:44.368 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:04:44.383 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:44.383 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:44.384 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:04:44.384 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:04:44.384 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:44.388 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap26 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap27 b3fbb028-e267-4d25-a750-93ba18a2316a@snap27.diff 2023-07-02T14:04:44.395 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:44.432 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:44.432+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:44.458 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 100% complete...done. 2023-07-02T14:04:44.462 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:04:44.462 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=28 2023-07-02T14:04:44.462 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap28 2023-07-02T14:04:44.485 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:44.509 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:44.509+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:44.519 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:44.543 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:44.543 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:04:44.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:04:44.543 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:44.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.544 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:44.544 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:44.544 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 125.25042446800001, 2023-07-02T14:04:44.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:04:44.544 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.544 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.545 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.545 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.545 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.545 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:04:44.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.545 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:44.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:44.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:44.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:04:44.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:04:44.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:04:44.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:04:44.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:04:44.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:04:44.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:04:44.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:04:44.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:04:44.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:04:44.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:04:44.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:04:44.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:44.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:04:44.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:04:44.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:44.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:04:44.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:04:44.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:04:44.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:04:44.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:04:44.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:04:44.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:04:44.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:04:44.554 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.554 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.554 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.555 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:44.555 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:44.555 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 125.250189445, 2023-07-02T14:04:44.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:04:44.555 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.555 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.555 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.556 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.556 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.556 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:04:44.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.556 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:44.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:44.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:44.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:04:44.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:04:44.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:04:44.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:04:44.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:04:44.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:04:44.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:04:44.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:04:44.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:04:44.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:04:44.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:04:44.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:04:44.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:44.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:04:44.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:04:44.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:44.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:04:44.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:04:44.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:04:44.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:04:44.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:04:44.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:04:44.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:04:44.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:04:44.569 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.569 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.569 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.569 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:44.569 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:44.569 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 125.249990311, 2023-07-02T14:04:44.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:04:44.570 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.570 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.570 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.570 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.570 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.571 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:04:44.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.571 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:44.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:44.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:44.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:04:44.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:04:44.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:04:44.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:04:44.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:04:44.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:04:44.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:04:44.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:04:44.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:04:44.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:04:44.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:04:44.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:04:44.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:44.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:04:44.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:04:44.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:44.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:04:44.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:04:44.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:04:44.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:04:44.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:04:44.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:04:44.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:04:44.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:04:44.582 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.582 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.582 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.583 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:44.583 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:44.583 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 125.24962518700001, 2023-07-02T14:04:44.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:04:44.583 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.583 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.583 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.584 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.584 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.584 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:04:44.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.584 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:44.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:44.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:44.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:04:44.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:04:44.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:04:44.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:04:44.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:04:44.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:04:44.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:04:44.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:04:44.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:04:44.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:04:44.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:04:44.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:04:44.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:44.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:04:44.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:04:44.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:44.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:04:44.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:04:44.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:04:44.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:04:44.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:04:44.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:04:44.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:04:44.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:04:44.593 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.593 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.593 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.594 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:44.594 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:44.594 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 125.249544126, 2023-07-02T14:04:44.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:04:44.594 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.595 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.595 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.595 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.595 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.595 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:04:44.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.595 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:44.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:44.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:44.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:04:44.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:04:44.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:04:44.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:04:44.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:04:44.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:04:44.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:04:44.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:04:44.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:04:44.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:04:44.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:04:44.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:04:44.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:44.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:04:44.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:04:44.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:44.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:04:44.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:04:44.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:04:44.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:04:44.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:04:44.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:04:44.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:04:44.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:04:44.604 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.604 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.604 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.604 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:44.604 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:44.605 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 125.249452455, 2023-07-02T14:04:44.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:04:44.605 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.605 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.605 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.605 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.605 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.606 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:04:44.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.606 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:44.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:44.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:44.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:04:44.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:04:44.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:04:44.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:04:44.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:04:44.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:04:44.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:04:44.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:04:44.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:04:44.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:04:44.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:04:44.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:04:44.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:44.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:04:44.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:04:44.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:04:44.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:04:44.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:44.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:04:44.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:04:44.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:04:44.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:04:44.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:04:44.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:04:44.614 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.615 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.615 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.615 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:44.615 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:44.615 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 125.249113013, 2023-07-02T14:04:44.615 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:04:44.616 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.616 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.616 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.616 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.616 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.616 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:04:44.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.616 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:44.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:44.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:44.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:04:44.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:04:44.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:04:44.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:04:44.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:04:44.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:04:44.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:04:44.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:04:44.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:04:44.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:04:44.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:04:44.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:04:44.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:44.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:04:44.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:04:44.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:04:44.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:04:44.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:44.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:04:44.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:04:44.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:04:44.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:04:44.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.625 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:04:44.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:04:44.625 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.625 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.625 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.626 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:44.626 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:44.626 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 125.24892696000001, 2023-07-02T14:04:44.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:04:44.626 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.626 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.626 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.627 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.627 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.627 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:04:44.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.627 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:44.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:44.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:44.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:44.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:04:44.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:04:44.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:04:44.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:04:44.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:04:44.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:04:44.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:04:44.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:04:44.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:04:44.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:04:44.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:04:44.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:44.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:04:44.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:04:44.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:04:44.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:04:44.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:44.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:04:44.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:04:44.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:04:44.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:04:44.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:04:44.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:04:44.636 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.636 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.636 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.636 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:44.636 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:44.637 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 125.248769158, 2023-07-02T14:04:44.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:04:44.637 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.637 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.637 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.637 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.637 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.637 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:04:44.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.638 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:44.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:44.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:44.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:04:44.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:04:44.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:04:44.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:04:44.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:04:44.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:04:44.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:04:44.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:04:44.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:04:44.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:04:44.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:04:44.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:04:44.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:44.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:04:44.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:04:44.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:04:44.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:04:44.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:44.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:04:44.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:04:44.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:04:44.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:04:44.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:04:44.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:04:44.646 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.646 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.646 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.647 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:44.647 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:44.647 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 125.248577376, 2023-07-02T14:04:44.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:04:44.647 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.647 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.648 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.648 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.648 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.648 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:04:44.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.648 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:44.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:44.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:44.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:04:44.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:04:44.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:04:44.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:04:44.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:04:44.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:04:44.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:04:44.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:04:44.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:04:44.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:04:44.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:04:44.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:04:44.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:44.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:04:44.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:04:44.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:44.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:04:44.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:04:44.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:04:44.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:04:44.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:04:44.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:04:44.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:04:44.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:04:44.657 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.657 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.657 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.657 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:44.657 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:44.657 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 125.248401716, 2023-07-02T14:04:44.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:04:44.658 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.658 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.658 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.658 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.658 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.658 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:04:44.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.659 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:44.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:44.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:44.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:44.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:04:44.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:04:44.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:04:44.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:04:44.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:04:44.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:04:44.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:04:44.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:04:44.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:04:44.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:04:44.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:04:44.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:44.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:04:44.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:04:44.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:44.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:04:44.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:04:44.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:04:44.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:04:44.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:04:44.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:04:44.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:04:44.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:04:44.667 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.668 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.668 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.669 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:44.669 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:44.669 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 125.248307528, 2023-07-02T14:04:44.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:04:44.669 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.669 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.670 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.670 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.670 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.670 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:04:44.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.670 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:44.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:44.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:44.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:04:44.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:04:44.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:04:44.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:04:44.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:04:44.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:04:44.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:04:44.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:04:44.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:04:44.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:04:44.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:04:44.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:04:44.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:44.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:04:44.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:04:44.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:44.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:04:44.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:04:44.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:04:44.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:04:44.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:04:44.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:04:44.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:04:44.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:04:44.679 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.679 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.679 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.679 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:44.680 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:44.680 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 125.248104142, 2023-07-02T14:04:44.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:04:44.680 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.680 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.680 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.680 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.680 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.681 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:04:44.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.681 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:44.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:44.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:44.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:04:44.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:04:44.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:04:44.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:04:44.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:04:44.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:04:44.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:04:44.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:04:44.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:04:44.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:04:44.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:04:44.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:04:44.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:44.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:04:44.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:04:44.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:44.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:04:44.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:04:44.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:04:44.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:04:44.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:04:44.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:04:44.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:04:44.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:04:44.691 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.691 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.691 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.692 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:44.692 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:44.692 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 125.247953486, 2023-07-02T14:04:44.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:04:44.693 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.693 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.693 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.693 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.694 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.694 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:04:44.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.694 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:44.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:44.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:44.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:04:44.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:04:44.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:04:44.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:04:44.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:04:44.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:04:44.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:04:44.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:04:44.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:04:44.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:04:44.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:04:44.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:04:44.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:44.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:04:44.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:04:44.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:44.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:04:44.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:04:44.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:04:44.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:04:44.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:04:44.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:04:44.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:04:44.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:04:44.703 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.703 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.703 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.703 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:44.703 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:44.704 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 125.247880783, 2023-07-02T14:04:44.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:04:44.704 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.704 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.704 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.704 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.704 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.704 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:04:44.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.705 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:44.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:44.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:44.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:44.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:04:44.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:04:44.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:04:44.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:04:44.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:04:44.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:04:44.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:04:44.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:04:44.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:04:44.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:04:44.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:04:44.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:44.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:04:44.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:04:44.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:44.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:04:44.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:04:44.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:04:44.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:04:44.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:04:44.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:04:44.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:04:44.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:04:44.713 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.713 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.713 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.714 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:44.714 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:44.714 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 125.247763793, 2023-07-02T14:04:44.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:04:44.714 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.714 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.714 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.715 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.715 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.715 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:04:44.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.715 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:44.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:44.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:44.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:44.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:04:44.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:04:44.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:04:44.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:04:44.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:04:44.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:04:44.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:04:44.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:04:44.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:04:44.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:04:44.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:04:44.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:44.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:04:44.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:04:44.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:44.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:04:44.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:04:44.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:04:44.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:04:44.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:04:44.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:04:44.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:04:44.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:04:44.724 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.724 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.724 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.724 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:44.725 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:44.725 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 125.24761434200001, 2023-07-02T14:04:44.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:04:44.725 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.725 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.725 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.725 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.725 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.726 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:04:44.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.726 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:44.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:44.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:44.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:04:44.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:04:44.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:04:44.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:04:44.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:04:44.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:04:44.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:04:44.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:04:44.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:04:44.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:04:44.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:04:44.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:04:44.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:44.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:04:44.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:04:44.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:44.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:04:44.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:04:44.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:04:44.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:04:44.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:04:44.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:04:44.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:04:44.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:04:44.734 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.734 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.734 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.735 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:44.735 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:44.735 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 125.226138369, 2023-07-02T14:04:44.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:04:44.735 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.735 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.736 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.736 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.736 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.736 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:04:44.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.736 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:44.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:44.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:44.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:04:44.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:04:44.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:04:44.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:04:44.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:04:44.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:04:44.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:04:44.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:04:44.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:04:44.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:04:44.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:04:44.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:04:44.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:44.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:04:44.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:04:44.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:04:44.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:04:44.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:44.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:04:44.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:04:44.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:04:44.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:04:44.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:04:44.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:04:44.745 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.745 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.745 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.745 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:44.745 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:44.745 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 125.226005833, 2023-07-02T14:04:44.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:04:44.746 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.746 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.746 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.746 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.746 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.746 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:04:44.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.747 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:44.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:44.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:44.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:04:44.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:04:44.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:04:44.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:04:44.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:04:44.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:04:44.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:04:44.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:04:44.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:04:44.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:04:44.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:04:44.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:04:44.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:44.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:04:44.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:04:44.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:04:44.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:04:44.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:44.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:04:44.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:04:44.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:04:44.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:04:44.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:04:44.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:04:44.755 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.755 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.756 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.756 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:44.756 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:44.757 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 125.164669172, 2023-07-02T14:04:44.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:04:44.757 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:44.757 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:44.757 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:44.757 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:44.757 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:44.757 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:04:44.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.758 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:44.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:44.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:44.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:44.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:44.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:44.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:44.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:44.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:04:44.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:04:44.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:44.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:04:44.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:04:44.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:44.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:04:44.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:04:44.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:44.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:04:44.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:04:44.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:44.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:04:44.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:04:44.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:44.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:04:44.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:04:44.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:44.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:04:44.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:04:44.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:44.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:04:44.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:04:44.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:44.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:04:44.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:04:44.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:44.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:04:44.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:04:44.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:44.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:44.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:44.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:04:44.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:04:44.766 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.766 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.766 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.767 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:44.767 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:44.767 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:44.768 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:44.769 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:44.769 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:44.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:44.770 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:44.771 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:04:44.771 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:04:44.771 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:04:44.771 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:44.771 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:04:44.771 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:04:44.772 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:44.772 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:44.768+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:44.868 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:44.868+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:44.946 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:44.946 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:44.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:04:44.999 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:44.999+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:45.070 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:45.069+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:45.127 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:45.128 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:45.139 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:04:45.182 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:45.182+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:45.272 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:45.271+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:45.276 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:45.276 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:45.288 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:04:45.373 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:45.373+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:45.406 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:04:45.417 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 2 2023-07-02T14:04:45.476 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:45.476+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:45.515 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:45.516 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:45.534 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:04:45.575 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:45.574+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:45.676 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:45.676+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:45.717 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:45.717 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:45.730 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:04:45.777 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:45.776+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:45.877 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:45.877+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:45.921 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:45.921 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:45.933 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:04:46.017 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:46.017+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:46.079 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:46.079+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:46.097 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:46.098 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:46.108 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:04:46.191 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:46.191+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:46.272 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:46.272 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:46.281 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:46.280+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:46.282 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:04:46.373 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 2] out_osds: [5, 4] dead_osds: [3] live_osds: [0, 1, 2, 4, 5] 2023-07-02T14:04:46.374 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:04:46.374 INFO:tasks.thrashosds.thrasher:Reweighting osd 1 to 0.9047304279977286 2023-07-02T14:04:46.375 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 0.9047304279977286 2023-07-02T14:04:46.390 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:04:46.419 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:46.419+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:46.452 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:46.452 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:46.464 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:04:46.507 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:46.507+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:46.583 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:46.583+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:46.591 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:46.591 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:46.601 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:04:46.684 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:46.684+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:46.765 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:46.766 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:46.772 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:04:46.776 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:04:46.818 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:46.818+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:46.886 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:46.885+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:46.900 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:46.900 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:46.910 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:04:46.911 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:46.928 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:46.945 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:46.963 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:46.981 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:46.981 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:04:46.982 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:04:46.982 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:04:46.982 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:46.982 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:04:46.982 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:04:46.984 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:46.984 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:46.986 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:46.986+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:47.034 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:47.035 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:47.035 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:04:47.035 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:47.044 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:04:47.087 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:47.086+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:47.188 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:47.188+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:47.208 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:47.208 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:47.209 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:04:47.209 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:04:47.209 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:47.218 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:47.288 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:47.288+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:47.344 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:47.344 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:04:47.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:04:47.345 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:47.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.345 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:47.345 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:47.345 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.051713509, 2023-07-02T14:04:47.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:04:47.346 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:47.346 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:47.346 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:47.346 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:47.346 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:47.346 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:04:47.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.347 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:47.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:47.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:47.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:47.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:47.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:47.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:47.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:47.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:04:47.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:04:47.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:47.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:04:47.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:04:47.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:47.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:04:47.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:04:47.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:47.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:04:47.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:04:47.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:47.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:04:47.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:04:47.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:47.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:04:47.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:04:47.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:47.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:04:47.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:04:47.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:47.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:04:47.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:04:47.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:47.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:04:47.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:04:47.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:47.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:04:47.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:04:47.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:47.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:04:47.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:04:47.356 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.356 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.356 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.357 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:47.357 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:47.357 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.05147848600001, 2023-07-02T14:04:47.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:04:47.357 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:47.357 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:47.357 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:47.358 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:47.358 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:47.358 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:04:47.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.359 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:47.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:47.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:47.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:47.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:47.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:47.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:47.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:47.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:04:47.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:04:47.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:47.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:04:47.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:04:47.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:47.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:04:47.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:04:47.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:47.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:04:47.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:04:47.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:47.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:04:47.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:04:47.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:47.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:04:47.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:04:47.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:47.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:04:47.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:04:47.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:47.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:04:47.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:04:47.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:47.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:04:47.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:04:47.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:47.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:04:47.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:04:47.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:47.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:04:47.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:04:47.367 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.367 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.367 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.368 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:47.368 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:47.368 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.05127935199999, 2023-07-02T14:04:47.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:04:47.368 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:47.368 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:47.369 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:47.369 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:47.369 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:47.369 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:04:47.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.369 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:47.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:47.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:47.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:47.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:47.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:47.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:47.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:47.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:04:47.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:04:47.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:47.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:04:47.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:04:47.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:47.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:04:47.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:04:47.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:47.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:04:47.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:04:47.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:47.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:04:47.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:04:47.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:47.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:04:47.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:04:47.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:47.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:04:47.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:04:47.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:47.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:04:47.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:04:47.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:47.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:04:47.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:04:47.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:47.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:04:47.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:04:47.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:47.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:04:47.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:04:47.378 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.378 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.378 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.378 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:47.378 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:47.378 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.05091422800001, 2023-07-02T14:04:47.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:04:47.379 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:47.379 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:47.379 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:47.379 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:47.379 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:47.379 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:04:47.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.380 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:47.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:47.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:47.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:47.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:47.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:47.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:47.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:47.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:04:47.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:04:47.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:47.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:04:47.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:04:47.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:47.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:04:47.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:04:47.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:47.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:04:47.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:04:47.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:47.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:04:47.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:04:47.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:47.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:04:47.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:04:47.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:47.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:04:47.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:04:47.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:47.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:04:47.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:04:47.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:47.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:04:47.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:04:47.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:47.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:04:47.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:04:47.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:47.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:04:47.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:04:47.388 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.388 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.388 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.389 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:47.389 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:47.389 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.05083316700001, 2023-07-02T14:04:47.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:04:47.389 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:47.389 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:47.389 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:47.389 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:47.390 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:47.390 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:04:47.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.390 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:47.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:47.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:47.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:47.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:47.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:47.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:47.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:47.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:04:47.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:04:47.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:47.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:04:47.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:04:47.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:47.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:04:47.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:04:47.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:47.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:04:47.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:04:47.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:47.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:04:47.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:04:47.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:47.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:04:47.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:04:47.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:47.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:04:47.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:04:47.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:47.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:04:47.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:04:47.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:47.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:04:47.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:04:47.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:47.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:04:47.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:04:47.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:47.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:04:47.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:04:47.399 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.399 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.399 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.399 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:47.399 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:47.399 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.050741496, 2023-07-02T14:04:47.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:04:47.400 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:47.400 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:47.400 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:47.400 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:47.400 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:47.400 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:04:47.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.401 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:47.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:47.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:47.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:47.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:47.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:47.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:47.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:47.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:04:47.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:04:47.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:47.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:04:47.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:04:47.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:47.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:04:47.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:04:47.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.404 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:47.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:04:47.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:04:47.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:47.405 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:04:47.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:04:47.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:47.405 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:04:47.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:04:47.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:47.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:04:47.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:04:47.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:47.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:04:47.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:04:47.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:47.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:04:47.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:04:47.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:47.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:04:47.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:04:47.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:47.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:04:47.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:04:47.409 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.409 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.409 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.410 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:47.410 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:47.410 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.05040205399999, 2023-07-02T14:04:47.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:04:47.410 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:47.410 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:47.410 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:47.411 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:47.411 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:47.411 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:04:47.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.411 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:47.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:47.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:47.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:47.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:47.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:47.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:47.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:47.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:04:47.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:04:47.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:47.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:04:47.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:04:47.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:47.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:04:47.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:04:47.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:47.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:04:47.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:04:47.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:47.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:04:47.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:04:47.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:47.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:04:47.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:04:47.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:47.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:04:47.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:04:47.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:47.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:04:47.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:04:47.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:47.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:04:47.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:04:47.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:47.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:04:47.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:04:47.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:47.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:04:47.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:04:47.419 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.419 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.420 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.420 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:47.420 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:47.420 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.050216001, 2023-07-02T14:04:47.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:04:47.421 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:47.421 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:47.421 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:47.421 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:47.421 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:47.421 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:04:47.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.421 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:47.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:47.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:47.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:47.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:47.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:47.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:47.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:47.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:47.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:04:47.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:47.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:04:47.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:04:47.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:47.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:04:47.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:04:47.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:47.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:04:47.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:04:47.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:47.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:04:47.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:04:47.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:47.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:04:47.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:04:47.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:47.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:04:47.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:04:47.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:47.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:04:47.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:04:47.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:47.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:04:47.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:04:47.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:47.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:04:47.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:04:47.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:47.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:04:47.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:04:47.431 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.431 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.431 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.431 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:47.431 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:47.432 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.05005819900001, 2023-07-02T14:04:47.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:04:47.432 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:47.432 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:47.432 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:47.432 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:47.432 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:47.432 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:04:47.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.433 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:47.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:47.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:47.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:47.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:47.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:47.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:47.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:47.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:04:47.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:04:47.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:47.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:04:47.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:04:47.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:47.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:04:47.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:04:47.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:47.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:04:47.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:04:47.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:47.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:04:47.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:04:47.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:47.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:04:47.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:04:47.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:47.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:04:47.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:04:47.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:47.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:04:47.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:04:47.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:47.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:04:47.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:04:47.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:47.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:04:47.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:04:47.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:47.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:04:47.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:04:47.441 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.441 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.441 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.442 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:47.442 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:47.442 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.049866417, 2023-07-02T14:04:47.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:04:47.442 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:47.442 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:47.442 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:47.442 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:47.443 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:47.443 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:04:47.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.443 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:47.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:47.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:47.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:47.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:47.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:47.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:47.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:47.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:04:47.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:04:47.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:47.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:04:47.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:04:47.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:47.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:04:47.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:04:47.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:47.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:04:47.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:04:47.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:47.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:04:47.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:04:47.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:47.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:04:47.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:04:47.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:47.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:04:47.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:04:47.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:47.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:04:47.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:04:47.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:47.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:04:47.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:04:47.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:47.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:04:47.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:04:47.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:47.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:04:47.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:04:47.452 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.452 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.452 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.452 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:47.452 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:47.453 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.04969075700001, 2023-07-02T14:04:47.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:04:47.453 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:47.453 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:47.453 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:47.453 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:47.453 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:47.453 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:04:47.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.454 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:47.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:47.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:47.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:47.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:47.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:47.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:47.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:47.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:47.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:04:47.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:47.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:04:47.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:04:47.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:47.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:04:47.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:04:47.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:47.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:04:47.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:04:47.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:47.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:04:47.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:04:47.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:47.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:04:47.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:04:47.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:47.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:04:47.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:04:47.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:47.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:04:47.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:04:47.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:47.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:04:47.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:04:47.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:47.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:04:47.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:04:47.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:47.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:04:47.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:04:47.462 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.462 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.462 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.463 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:47.463 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:47.463 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.04959656899999, 2023-07-02T14:04:47.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:04:47.463 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:47.463 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:47.464 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:47.464 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:47.464 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:47.464 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:04:47.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.464 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:47.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:47.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:47.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:47.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:47.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:47.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:47.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:47.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:04:47.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:04:47.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:47.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:04:47.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:04:47.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:47.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:04:47.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:04:47.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:47.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:04:47.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:04:47.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:47.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:04:47.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:04:47.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:47.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:04:47.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:04:47.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:47.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:04:47.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:04:47.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:47.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:04:47.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:04:47.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:47.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:04:47.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:04:47.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:47.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:04:47.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:04:47.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:47.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:04:47.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:04:47.473 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.473 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.473 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.473 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:47.473 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:47.474 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.04939318300001, 2023-07-02T14:04:47.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:04:47.474 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:47.474 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:47.474 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:47.474 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:47.474 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:47.474 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:04:47.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.475 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:47.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:47.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:47.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:47.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:47.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:47.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:47.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:47.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:04:47.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:04:47.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:47.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:04:47.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:04:47.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:47.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:04:47.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:04:47.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:47.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:04:47.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:04:47.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:47.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:04:47.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:04:47.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:47.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:04:47.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:04:47.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:47.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:04:47.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:04:47.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:47.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:04:47.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:04:47.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:47.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:04:47.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:04:47.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:47.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:04:47.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:04:47.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:47.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:04:47.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:04:47.483 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.483 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.483 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.484 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:47.484 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:47.484 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.04924252699999, 2023-07-02T14:04:47.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:04:47.484 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:47.485 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:47.485 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:47.485 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:47.485 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:47.485 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:04:47.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.485 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:47.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:47.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:47.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:47.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:47.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:47.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:47.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:47.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:04:47.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:04:47.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:47.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:04:47.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:04:47.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:47.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:04:47.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:04:47.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:47.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:04:47.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:04:47.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:47.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:04:47.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:04:47.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:47.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:04:47.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:04:47.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:47.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:04:47.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:04:47.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:47.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:04:47.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:04:47.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:47.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:04:47.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:04:47.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:47.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:04:47.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:04:47.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:47.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:04:47.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:04:47.494 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.494 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.494 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.495 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:47.495 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:47.495 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.04916982399999, 2023-07-02T14:04:47.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:04:47.495 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:47.496 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:47.496 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:47.496 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:47.496 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:47.496 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:04:47.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.496 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:47.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:47.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:47.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:47.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:47.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:47.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:47.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:47.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:47.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:04:47.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:47.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:04:47.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:04:47.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:47.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:04:47.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:04:47.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:47.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:04:47.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:04:47.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:47.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:04:47.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:04:47.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:47.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:04:47.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:04:47.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:47.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:04:47.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:04:47.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:47.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:04:47.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:04:47.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:47.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:04:47.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:04:47.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:47.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:04:47.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:04:47.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:47.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:04:47.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:04:47.505 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.506 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.506 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.506 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:47.506 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:47.506 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.04905283400001, 2023-07-02T14:04:47.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:04:47.507 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:47.507 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:47.507 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:47.507 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:47.507 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:47.507 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:04:47.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.508 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:47.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:47.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:47.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:47.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:47.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:47.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:47.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:47.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:47.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:04:47.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:47.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:04:47.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:04:47.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:47.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:04:47.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:04:47.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:47.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:04:47.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:04:47.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:47.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:04:47.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:04:47.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:47.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:04:47.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:04:47.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:47.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:04:47.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:04:47.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:47.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:04:47.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:04:47.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:47.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:04:47.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:04:47.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:47.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:04:47.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:04:47.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:47.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:04:47.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:04:47.517 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.517 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.517 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.518 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:47.518 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:47.518 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.04890338300001, 2023-07-02T14:04:47.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:04:47.518 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:47.519 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:47.519 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:47.519 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:47.519 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:47.519 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:04:47.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.519 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:47.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:47.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:47.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:47.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:47.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:47.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:47.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:47.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:04:47.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:04:47.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:47.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:04:47.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:04:47.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:47.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:04:47.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:04:47.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:47.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:04:47.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:04:47.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:47.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:04:47.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:04:47.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:47.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:04:47.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:04:47.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:47.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:04:47.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:04:47.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:47.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:04:47.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:04:47.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:47.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:04:47.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:04:47.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:47.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:04:47.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:04:47.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:47.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:04:47.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:04:47.528 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.528 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.528 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.528 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:47.529 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:47.529 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.02742741, 2023-07-02T14:04:47.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:04:47.529 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:47.529 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:47.529 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:47.529 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:47.530 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:47.530 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:04:47.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.530 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:47.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:47.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:47.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:47.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:47.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:47.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:47.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:47.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:04:47.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:04:47.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:47.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:04:47.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:04:47.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:47.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:04:47.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:04:47.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:47.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:04:47.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:04:47.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:47.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:04:47.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:04:47.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:47.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:04:47.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:04:47.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:47.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:04:47.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:04:47.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:47.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:04:47.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:04:47.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:47.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:04:47.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:04:47.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:47.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:04:47.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:04:47.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:47.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:04:47.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:04:47.538 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.538 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.538 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.539 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:47.539 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:47.539 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.02729487400001, 2023-07-02T14:04:47.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:04:47.540 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:47.540 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:47.540 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:47.540 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:47.540 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:47.540 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:04:47.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.540 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:47.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:47.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:47.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:47.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:47.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:47.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:47.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:47.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:04:47.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:04:47.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:47.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:04:47.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:04:47.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:47.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:04:47.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:04:47.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:47.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:04:47.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:04:47.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:47.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:04:47.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:04:47.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:47.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:04:47.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:04:47.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:47.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:04:47.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:04:47.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:47.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:04:47.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:04:47.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:47.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:04:47.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:04:47.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:47.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:04:47.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:04:47.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:47.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:04:47.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:04:47.549 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.549 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.549 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.550 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:47.550 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:47.550 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 127.96595821299999, 2023-07-02T14:04:47.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:04:47.550 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:47.550 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:47.550 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:47.551 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:47.551 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:47.551 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:04:47.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.551 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:47.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:47.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:47.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:47.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:47.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:47.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:47.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:47.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:04:47.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:04:47.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:47.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:04:47.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:04:47.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:47.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:04:47.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:04:47.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:47.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:04:47.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:04:47.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:47.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:04:47.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:04:47.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:47.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:04:47.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:04:47.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:47.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:04:47.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:04:47.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:47.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:04:47.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:04:47.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:47.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:04:47.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:04:47.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:47.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:04:47.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:04:47.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:47.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:47.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:47.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:04:47.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:04:47.559 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.560 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.560 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.560 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:47.560 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:47.560 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:47.561 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:04:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.1 to 0.90473 (e79c) 2023-07-02T14:04:47.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:47.564 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap27 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap28 b3fbb028-e267-4d25-a750-93ba18a2316a@snap28.diff 2023-07-02T14:04:47.564 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:47.488+0000 7fa515001700 0 -- 172.21.15.163:0/57475297 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7fa4f0008780 msgr2=0x7fa4f000abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:47.564 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:47.493+0000 7fa515001700 0 -- 172.21.15.163:0/57475297 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7fa4f8036c20 msgr2=0x7fa4f8039050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:47.565 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:47.561+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:47.662 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:47.662+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:47.706 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 100% complete...done. 2023-07-02T14:04:47.711 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:04:47.712 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=29 2023-07-02T14:04:47.712 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap29 2023-07-02T14:04:47.733 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:47.733 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:47.734 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:47.734 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:47.744 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:04:47.763 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:47.762+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:47.861 INFO:tasks.workunit.client.0.smithi163.stderr:+ tar -C b3fbb028-e267-4d25-a750-93ba18a2316a-mnt -xzf b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4.tar.gz 2023-07-02T14:04:47.863 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:47.863+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:47.907 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:47.908 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:47.908 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:47.908 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:47.908 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:47.918 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:04:47.964 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:47.963+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:48.065 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:48.064+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:48.085 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:48.085 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:48.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:48.085 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:48.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.085 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2620 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 196608~786432] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:48.086 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:48.086 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.758325275000004, 2023-07-02T14:04:48.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966, 2023-07-02T14:04:48.086 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.086 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.086 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.086 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.086 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.087 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2620 2023-07-02T14:04:48.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.087 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:48.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:48.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:48.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324571+0000", 2023-07-02T14:04:48.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016686649999999999 2023-07-02T14:04:48.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326240+0000", 2023-07-02T14:04:48.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1844e-05 2023-07-02T14:04:48.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326252+0000", 2023-07-02T14:04:48.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7760000000000003e-06 2023-07-02T14:04:48.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326258+0000", 2023-07-02T14:04:48.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0634000000000003e-05 2023-07-02T14:04:48.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326318+0000", 2023-07-02T14:04:48.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9629999999999993e-06 2023-07-02T14:04:48.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326328+0000", 2023-07-02T14:04:48.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.010877045 2023-07-02T14:04:48.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337205+0000", 2023-07-02T14:04:48.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022401300000000001 2023-07-02T14:04:48.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337429+0000", 2023-07-02T14:04:48.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000139822 2023-07-02T14:04:48.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:48.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337569+0000", 2023-07-02T14:04:48.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032942119999999999 2023-07-02T14:04:48.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340863+0000", 2023-07-02T14:04:48.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021302560000000001 2023-07-02T14:04:48.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342994+0000", 2023-07-02T14:04:48.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5634000000000002e-05 2023-07-02T14:04:48.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343059+0000", 2023-07-02T14:04:48.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143721 2023-07-02T14:04:48.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343203+0000", 2023-07-02T14:04:48.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966 2023-07-02T14:04:48.096 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.097 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.097 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.097 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2622 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 983040~36864] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:48.097 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:48.097 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.756632764000003, 2023-07-02T14:04:48.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664, 2023-07-02T14:04:48.098 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.098 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.098 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.098 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.098 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.098 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2622 2023-07-02T14:04:48.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.098 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:48.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:48.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:48.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326265+0000", 2023-07-02T14:04:48.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1282000000000001e-05 2023-07-02T14:04:48.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326286+0000", 2023-07-02T14:04:48.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9480000000000002e-06 2023-07-02T14:04:48.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326293+0000", 2023-07-02T14:04:48.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3610000000000003e-06 2023-07-02T14:04:48.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326297+0000", 2023-07-02T14:04:48.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7707999999999994e-05 2023-07-02T14:04:48.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326385+0000", 2023-07-02T14:04:48.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8070000000000006e-06 2023-07-02T14:04:48.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326394+0000", 2023-07-02T14:04:48.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.014535878 2023-07-02T14:04:48.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340930+0000", 2023-07-02T14:04:48.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000154551 2023-07-02T14:04:48.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341084+0000", 2023-07-02T14:04:48.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00032375900000000002 2023-07-02T14:04:48.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:48.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341408+0000", 2023-07-02T14:04:48.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012631066 2023-07-02T14:04:48.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354039+0000", 2023-07-02T14:04:48.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5741000000000001e-05 2023-07-02T14:04:48.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354055+0000", 2023-07-02T14:04:48.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8794000000000006e-05 2023-07-02T14:04:48.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354124+0000", 2023-07-02T14:04:48.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014667700000000001 2023-07-02T14:04:48.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354270+0000", 2023-07-02T14:04:48.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664 2023-07-02T14:04:48.108 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.108 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.108 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.109 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2727 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~12288] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:48.109 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:48.109 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.756596989000002, 2023-07-02T14:04:48.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999, 2023-07-02T14:04:48.109 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.109 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.109 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.109 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.110 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.110 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2727 2023-07-02T14:04:48.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.110 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:48.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:48.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:48.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:48.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3834e-05 2023-07-02T14:04:48.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326315+0000", 2023-07-02T14:04:48.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3029999999999996e-06 2023-07-02T14:04:48.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326321+0000", 2023-07-02T14:04:48.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7590000000000002e-06 2023-07-02T14:04:48.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326324+0000", 2023-07-02T14:04:48.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112872 2023-07-02T14:04:48.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326437+0000", 2023-07-02T14:04:48.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2290000000000002e-06 2023-07-02T14:04:48.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326442+0000", 2023-07-02T14:04:48.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0155896879999999 2023-07-02T14:04:48.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342031+0000", 2023-07-02T14:04:48.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016496100000000001 2023-07-02T14:04:48.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342196+0000", 2023-07-02T14:04:48.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021998699999999999 2023-07-02T14:04:48.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:48.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342416+0000", 2023-07-02T14:04:48.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011869744999999999 2023-07-02T14:04:48.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354286+0000", 2023-07-02T14:04:48.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0361e-05 2023-07-02T14:04:48.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354347+0000", 2023-07-02T14:04:48.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3312000000000005e-05 2023-07-02T14:04:48.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354430+0000", 2023-07-02T14:04:48.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126444 2023-07-02T14:04:48.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354556+0000", 2023-07-02T14:04:48.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999 2023-07-02T14:04:48.119 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.120 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.120 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.120 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2730 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [set-alloc-hint object_size 4194304 write_size 4194304,write 454656~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:48.120 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:48.120 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.756571980000004, 2023-07-02T14:04:48.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999, 2023-07-02T14:04:48.121 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.121 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.121 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.121 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.121 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.121 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2730 2023-07-02T14:04:48.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.121 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:48.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:48.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:48.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:48.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6049999999999997e-06 2023-07-02T14:04:48.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326332+0000", 2023-07-02T14:04:48.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3000000000000002e-06 2023-07-02T14:04:48.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326335+0000", 2023-07-02T14:04:48.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.719e-05 2023-07-02T14:04:48.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326352+0000", 2023-07-02T14:04:48.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011583599999999999 2023-07-02T14:04:48.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326468+0000", 2023-07-02T14:04:48.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.831e-06 2023-07-02T14:04:48.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326473+0000", 2023-07-02T14:04:48.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.016769915 2023-07-02T14:04:48.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343243+0000", 2023-07-02T14:04:48.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197352 2023-07-02T14:04:48.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343440+0000", 2023-07-02T14:04:48.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197581 2023-07-02T14:04:48.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:48.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343638+0000", 2023-07-02T14:04:48.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010980139 2023-07-02T14:04:48.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354618+0000", 2023-07-02T14:04:48.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056779530000000003 2023-07-02T14:04:48.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360296+0000", 2023-07-02T14:04:48.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101109 2023-07-02T14:04:48.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360397+0000", 2023-07-02T14:04:48.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000160273 2023-07-02T14:04:48.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360557+0000", 2023-07-02T14:04:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999 2023-07-02T14:04:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2752 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 1019904~3174400] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.756525756000002, 2023-07-02T14:04:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001, 2023-07-02T14:04:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2752 2023-07-02T14:04:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326372+0000", 2023-07-02T14:04:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052050380000000004 2023-07-02T14:04:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331577+0000", 2023-07-02T14:04:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6832000000000001e-05 2023-07-02T14:04:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331594+0000", 2023-07-02T14:04:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8220000000000007e-06 2023-07-02T14:04:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331602+0000", 2023-07-02T14:04:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9895999999999999e-05 2023-07-02T14:04:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331642+0000", 2023-07-02T14:04:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:04:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331649+0000", 2023-07-02T14:04:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0139141709999999 2023-07-02T14:04:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345563+0000", 2023-07-02T14:04:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113817 2023-07-02T14:04:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345677+0000", 2023-07-02T14:04:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018803400000000001 2023-07-02T14:04:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345865+0000", 2023-07-02T14:04:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087950359999999991 2023-07-02T14:04:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354660+0000", 2023-07-02T14:04:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059116669999999998 2023-07-02T14:04:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360572+0000", 2023-07-02T14:04:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0992000000000002e-05 2023-07-02T14:04:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360633+0000", 2023-07-02T14:04:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013601600000000001 2023-07-02T14:04:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360769+0000", 2023-07-02T14:04:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001 2023-07-02T14:04:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.143 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.143 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.143 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2759 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 0~454656] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:48.143 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:48.143 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.751285959000001, 2023-07-02T14:04:48.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999, 2023-07-02T14:04:48.144 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.144 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.144 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.144 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.144 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.144 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2759 2023-07-02T14:04:48.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.145 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:48.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:48.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:48.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331611+0000", 2023-07-02T14:04:48.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053795999999999998 2023-07-02T14:04:48.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332149+0000", 2023-07-02T14:04:48.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4970000000000007e-06 2023-07-02T14:04:48.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332158+0000", 2023-07-02T14:04:48.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:04:48.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332162+0000", 2023-07-02T14:04:48.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3526000000000001e-05 2023-07-02T14:04:48.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332185+0000", 2023-07-02T14:04:48.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:04:48.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332189+0000", 2023-07-02T14:04:48.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0186295910000001 2023-07-02T14:04:48.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350819+0000", 2023-07-02T14:04:48.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000111123 2023-07-02T14:04:48.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350930+0000", 2023-07-02T14:04:48.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019575300000000001 2023-07-02T14:04:48.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:48.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.351126+0000", 2023-07-02T14:04:48.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011503298 2023-07-02T14:04:48.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362629+0000", 2023-07-02T14:04:48.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085443989999999994 2023-07-02T14:04:48.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371173+0000", 2023-07-02T14:04:48.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3547999999999995e-05 2023-07-02T14:04:48.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371237+0000", 2023-07-02T14:04:48.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790300000000001 2023-07-02T14:04:48.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371375+0000", 2023-07-02T14:04:48.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999 2023-07-02T14:04:48.154 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.154 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.154 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.155 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2760 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 458752~131072] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:48.155 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:48.155 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.750729774000007, 2023-07-02T14:04:48.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001, 2023-07-02T14:04:48.155 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.155 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.155 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.156 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.156 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.156 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2760 2023-07-02T14:04:48.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.156 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:48.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:48.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:48.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332168+0000", 2023-07-02T14:04:48.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013281100000000001 2023-07-02T14:04:48.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332301+0000", 2023-07-02T14:04:48.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7389999999999999e-06 2023-07-02T14:04:48.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332308+0000", 2023-07-02T14:04:48.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0639999999999998e-06 2023-07-02T14:04:48.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332312+0000", 2023-07-02T14:04:48.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1735000000000001e-05 2023-07-02T14:04:48.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332333+0000", 2023-07-02T14:04:48.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8342000000000002e-05 2023-07-02T14:04:48.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332352+0000", 2023-07-02T14:04:48.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02234782 2023-07-02T14:04:48.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354699+0000", 2023-07-02T14:04:48.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110371 2023-07-02T14:04:48.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354810+0000", 2023-07-02T14:04:48.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022174900000000001 2023-07-02T14:04:48.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:48.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355032+0000", 2023-07-02T14:04:48.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076053329999999997 2023-07-02T14:04:48.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362637+0000", 2023-07-02T14:04:48.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087913609999999993 2023-07-02T14:04:48.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371428+0000", 2023-07-02T14:04:48.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8532999999999998e-05 2023-07-02T14:04:48.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371477+0000", 2023-07-02T14:04:48.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010990800000000001 2023-07-02T14:04:48.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371587+0000", 2023-07-02T14:04:48.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001 2023-07-02T14:04:48.166 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.166 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.166 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.166 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2766 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 589824~45056] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:48.166 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:48.166 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.750581148999999, 2023-07-02T14:04:48.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188, 2023-07-02T14:04:48.167 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.167 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.167 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.167 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.167 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.167 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2766 2023-07-02T14:04:48.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.168 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:48.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:48.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:48.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:48.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2247999999999998e-05 2023-07-02T14:04:48.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332369+0000", 2023-07-02T14:04:48.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4189999999999996e-06 2023-07-02T14:04:48.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332376+0000", 2023-07-02T14:04:48.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1049999999999999e-06 2023-07-02T14:04:48.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332379+0000", 2023-07-02T14:04:48.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6894000000000003e-05 2023-07-02T14:04:48.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332416+0000", 2023-07-02T14:04:48.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3970000000000002e-06 2023-07-02T14:04:48.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332422+0000", 2023-07-02T14:04:48.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0231382280000001 2023-07-02T14:04:48.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355560+0000", 2023-07-02T14:04:48.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9079000000000005e-05 2023-07-02T14:04:48.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355659+0000", 2023-07-02T14:04:48.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018005000000000001 2023-07-02T14:04:48.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:48.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355839+0000", 2023-07-02T14:04:48.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068044439999999998 2023-07-02T14:04:48.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362644+0000", 2023-07-02T14:04:48.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090155589999999994 2023-07-02T14:04:48.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371659+0000", 2023-07-02T14:04:48.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1297999999999994e-05 2023-07-02T14:04:48.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371730+0000", 2023-07-02T14:04:48.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013100699999999999 2023-07-02T14:04:48.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371861+0000", 2023-07-02T14:04:48.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188 2023-07-02T14:04:48.177 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.177 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.177 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2784 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 0~393216] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.750512697000005, 2023-07-02T14:04:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999, 2023-07-02T14:04:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2784 2023-07-02T14:04:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332385+0000", 2023-07-02T14:04:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040252400000000002 2023-07-02T14:04:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332787+0000", 2023-07-02T14:04:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0208e-05 2023-07-02T14:04:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332798+0000", 2023-07-02T14:04:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2023-07-02T14:04:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332801+0000", 2023-07-02T14:04:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4838000000000001e-05 2023-07-02T14:04:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332836+0000", 2023-07-02T14:04:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5319999999999998e-06 2023-07-02T14:04:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332839+0000", 2023-07-02T14:04:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0234290109999999 2023-07-02T14:04:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356268+0000", 2023-07-02T14:04:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011165700000000001 2023-07-02T14:04:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356380+0000", 2023-07-02T14:04:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022504799999999999 2023-07-02T14:04:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356605+0000", 2023-07-02T14:04:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015276201 2023-07-02T14:04:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371881+0000", 2023-07-02T14:04:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5496999999999999e-05 2023-07-02T14:04:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371957+0000", 2023-07-02T14:04:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4570000000000002e-05 2023-07-02T14:04:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372001+0000", 2023-07-02T14:04:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3880000000000003e-05 2023-07-02T14:04:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372085+0000", 2023-07-02T14:04:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999 2023-07-02T14:04:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2785 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 405504~446464] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.750091245999997, 2023-07-02T14:04:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999, 2023-07-02T14:04:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2785 2023-07-02T14:04:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059527500000000004 2023-07-02T14:04:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333402+0000", 2023-07-02T14:04:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7059999999999993e-06 2023-07-02T14:04:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333411+0000", 2023-07-02T14:04:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0450000000000001e-06 2023-07-02T14:04:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333414+0000", 2023-07-02T14:04:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5790999999999997e-05 2023-07-02T14:04:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:04:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3320000000000002e-06 2023-07-02T14:04:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333454+0000", 2023-07-02T14:04:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.025310269 2023-07-02T14:04:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358764+0000", 2023-07-02T14:04:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012414899999999999 2023-07-02T14:04:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358888+0000", 2023-07-02T14:04:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000217028 2023-07-02T14:04:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.359105+0000", 2023-07-02T14:04:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012788212 2023-07-02T14:04:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371893+0000", 2023-07-02T14:04:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012201569999999999 2023-07-02T14:04:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373114+0000", 2023-07-02T14:04:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4512999999999995e-05 2023-07-02T14:04:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373198+0000", 2023-07-02T14:04:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132373 2023-07-02T14:04:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373330+0000", 2023-07-02T14:04:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999 2023-07-02T14:04:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2786 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1310720~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.749478553000003, 2023-07-02T14:04:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001, 2023-07-02T14:04:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2786 2023-07-02T14:04:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333419+0000", 2023-07-02T14:04:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2009999999999999e-06 2023-07-02T14:04:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333427+0000", 2023-07-02T14:04:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0210000000000001e-06 2023-07-02T14:04:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333431+0000", 2023-07-02T14:04:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3769999999999999e-06 2023-07-02T14:04:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333434+0000", 2023-07-02T14:04:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1161e-05 2023-07-02T14:04:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333485+0000", 2023-07-02T14:04:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1549000000000001e-05 2023-07-02T14:04:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333496+0000", 2023-07-02T14:04:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0273116339999999 2023-07-02T14:04:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360808+0000", 2023-07-02T14:04:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000115128 2023-07-02T14:04:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360923+0000", 2023-07-02T14:04:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018367700000000001 2023-07-02T14:04:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361107+0000", 2023-07-02T14:04:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010796923999999999 2023-07-02T14:04:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371904+0000", 2023-07-02T14:04:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014983520000000001 2023-07-02T14:04:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373402+0000", 2023-07-02T14:04:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0520999999999999e-05 2023-07-02T14:04:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373463+0000", 2023-07-02T14:04:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000129095 2023-07-02T14:04:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373592+0000", 2023-07-02T14:04:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001 2023-07-02T14:04:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2787 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1433600~8192] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.749461753000006, 2023-07-02T14:04:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999, 2023-07-02T14:04:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2787 2023-07-02T14:04:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333436+0000", 2023-07-02T14:04:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2731999999999999e-05 2023-07-02T14:04:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:04:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7440000000000001e-06 2023-07-02T14:04:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333456+0000", 2023-07-02T14:04:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8650000000000003e-06 2023-07-02T14:04:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333460+0000", 2023-07-02T14:04:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0077e-05 2023-07-02T14:04:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333530+0000", 2023-07-02T14:04:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4180000000000001e-06 2023-07-02T14:04:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333533+0000", 2023-07-02T14:04:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0284002720000001 2023-07-02T14:04:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361933+0000", 2023-07-02T14:04:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6781999999999996e-05 2023-07-02T14:04:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362030+0000", 2023-07-02T14:04:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018305400000000001 2023-07-02T14:04:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362213+0000", 2023-07-02T14:04:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097015109999999995 2023-07-02T14:04:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371915+0000", 2023-07-02T14:04:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001742295 2023-07-02T14:04:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373657+0000", 2023-07-02T14:04:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5980000000000003e-05 2023-07-02T14:04:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373713+0000", 2023-07-02T14:04:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011783800000000001 2023-07-02T14:04:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373831+0000", 2023-07-02T14:04:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999 2023-07-02T14:04:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2803 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 851968~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.749419900999996, 2023-07-02T14:04:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514, 2023-07-02T14:04:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2803 2023-07-02T14:04:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4943000000000001e-05 2023-07-02T14:04:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333493+0000", 2023-07-02T14:04:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2629999999999997e-06 2023-07-02T14:04:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333500+0000", 2023-07-02T14:04:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4159999999999997e-06 2023-07-02T14:04:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333505+0000", 2023-07-02T14:04:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4518999999999998e-05 2023-07-02T14:04:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333559+0000", 2023-07-02T14:04:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5900000000000008e-06 2023-07-02T14:04:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333568+0000", 2023-07-02T14:04:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0291061779999999 2023-07-02T14:04:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362674+0000", 2023-07-02T14:04:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110412 2023-07-02T14:04:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362784+0000", 2023-07-02T14:04:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015384599999999999 2023-07-02T14:04:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362938+0000", 2023-07-02T14:04:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091682310000000006 2023-07-02T14:04:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372106+0000", 2023-07-02T14:04:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017703230000000001 2023-07-02T14:04:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373877+0000", 2023-07-02T14:04:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6387000000000002e-05 2023-07-02T14:04:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373923+0000", 2023-07-02T14:04:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014099199999999999 2023-07-02T14:04:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374064+0000", 2023-07-02T14:04:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514 2023-07-02T14:04:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2805 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 634880~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.749386725999997, 2023-07-02T14:04:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814, 2023-07-02T14:04:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2805 2023-07-02T14:04:48.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.237 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:48.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:48.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:48.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333511+0000", 2023-07-02T14:04:48.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207092 2023-07-02T14:04:48.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333718+0000", 2023-07-02T14:04:48.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5110000000000003e-06 2023-07-02T14:04:48.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333727+0000", 2023-07-02T14:04:48.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3270000000000007e-06 2023-07-02T14:04:48.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333736+0000", 2023-07-02T14:04:48.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0319999999999999e-05 2023-07-02T14:04:48.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333756+0000", 2023-07-02T14:04:48.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1695000000000001e-05 2023-07-02T14:04:48.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:04:48.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0295750560000001 2023-07-02T14:04:48.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363343+0000", 2023-07-02T14:04:48.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3387999999999994e-05 2023-07-02T14:04:48.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363437+0000", 2023-07-02T14:04:48.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018373 2023-07-02T14:04:48.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:48.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363620+0000", 2023-07-02T14:04:48.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0084948769999999996 2023-07-02T14:04:48.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372115+0000", 2023-07-02T14:04:48.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020082170000000001 2023-07-02T14:04:48.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374123+0000", 2023-07-02T14:04:48.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8875999999999997e-05 2023-07-02T14:04:48.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374182+0000", 2023-07-02T14:04:48.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0001272 2023-07-02T14:04:48.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374309+0000", 2023-07-02T14:04:48.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814 2023-07-02T14:04:48.246 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.246 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.246 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.247 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2806 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 1359872~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:48.247 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:48.247 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.749156395, 2023-07-02T14:04:48.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234, 2023-07-02T14:04:48.247 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.247 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.248 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.248 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.248 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.248 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2806 2023-07-02T14:04:48.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.248 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:48.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:48.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:48.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333741+0000", 2023-07-02T14:04:48.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3978999999999999e-05 2023-07-02T14:04:48.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333755+0000", 2023-07-02T14:04:48.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:04:48.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333761+0000", 2023-07-02T14:04:48.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7010000000000001e-06 2023-07-02T14:04:48.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333764+0000", 2023-07-02T14:04:48.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5086000000000003e-05 2023-07-02T14:04:48.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333799+0000", 2023-07-02T14:04:48.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6389999999999992e-06 2023-07-02T14:04:48.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333806+0000", 2023-07-02T14:04:48.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.030496675 2023-07-02T14:04:48.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364303+0000", 2023-07-02T14:04:48.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011382700000000001 2023-07-02T14:04:48.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364417+0000", 2023-07-02T14:04:48.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192058 2023-07-02T14:04:48.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:48.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364609+0000", 2023-07-02T14:04:48.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075132130000000004 2023-07-02T14:04:48.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372122+0000", 2023-07-02T14:04:48.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022731750000000001 2023-07-02T14:04:48.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374395+0000", 2023-07-02T14:04:48.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5277000000000005e-05 2023-07-02T14:04:48.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374460+0000", 2023-07-02T14:04:48.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017074599999999999 2023-07-02T14:04:48.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374631+0000", 2023-07-02T14:04:48.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234 2023-07-02T14:04:48.258 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.258 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.258 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.258 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 868352~49152] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:48.258 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:48.259 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.749129680999999, 2023-07-02T14:04:48.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001, 2023-07-02T14:04:48.259 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.259 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.259 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.259 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.259 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.259 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:04:48.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.260 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:48.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:48.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:48.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:04:48.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6355000000000001e-05 2023-07-02T14:04:48.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333794+0000", 2023-07-02T14:04:48.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8329999999999996e-06 2023-07-02T14:04:48.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333801+0000", 2023-07-02T14:04:48.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5419999999999996e-06 2023-07-02T14:04:48.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333807+0000", 2023-07-02T14:04:48.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2023-07-02T14:04:48.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333826+0000", 2023-07-02T14:04:48.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.303e-06 2023-07-02T14:04:48.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333830+0000", 2023-07-02T14:04:48.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.031137559 2023-07-02T14:04:48.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364967+0000", 2023-07-02T14:04:48.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1390000000000004e-05 2023-07-02T14:04:48.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365059+0000", 2023-07-02T14:04:48.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017597399999999999 2023-07-02T14:04:48.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:48.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365234+0000", 2023-07-02T14:04:48.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068943950000000002 2023-07-02T14:04:48.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372129+0000", 2023-07-02T14:04:48.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026189910000000002 2023-07-02T14:04:48.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374748+0000", 2023-07-02T14:04:48.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6613999999999996e-05 2023-07-02T14:04:48.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374814+0000", 2023-07-02T14:04:48.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502300000000001 2023-07-02T14:04:48.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374950+0000", 2023-07-02T14:04:48.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001 2023-07-02T14:04:48.269 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.269 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.269 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.270 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2815 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 917504~49152] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:48.270 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:48.270 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 96.744915448, 2023-07-02T14:04:48.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998, 2023-07-02T14:04:48.270 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.270 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.271 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.271 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.271 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.271 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2815 2023-07-02T14:04:48.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.271 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:48.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:48.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:04:48.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337980+0000", 2023-07-02T14:04:48.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017126700000000001 2023-07-02T14:04:48.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338152+0000", 2023-07-02T14:04:48.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2938999999999999e-05 2023-07-02T14:04:48.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338164+0000", 2023-07-02T14:04:48.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4448999999999999e-05 2023-07-02T14:04:48.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:04:48.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0281812 2023-07-02T14:04:48.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366370+0000", 2023-07-02T14:04:48.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1840000000000002e-05 2023-07-02T14:04:48.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366462+0000", 2023-07-02T14:04:48.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000186112 2023-07-02T14:04:48.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:48.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366648+0000", 2023-07-02T14:04:48.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054905709999999996 2023-07-02T14:04:48.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372139+0000", 2023-07-02T14:04:48.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044954790000000001 2023-07-02T14:04:48.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376634+0000", 2023-07-02T14:04:48.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4621000000000004e-05 2023-07-02T14:04:48.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376699+0000", 2023-07-02T14:04:48.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012862299999999999 2023-07-02T14:04:48.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376827+0000", 2023-07-02T14:04:48.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998 2023-07-02T14:04:48.279 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.280 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.280 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.280 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2816 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 917504~65536] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:48.280 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:48.280 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 96.744701069000001, 2023-07-02T14:04:48.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001, 2023-07-02T14:04:48.281 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.281 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.281 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.281 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.281 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.281 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2816 2023-07-02T14:04:48.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.282 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:48.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338197+0000", 2023-07-02T14:04:48.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:48.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338196+0000", 2023-07-02T14:04:48.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6286999999999999e-05 2023-07-02T14:04:48.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338243+0000", 2023-07-02T14:04:48.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7030000000000001e-06 2023-07-02T14:04:48.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338252+0000", 2023-07-02T14:04:48.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8089999999999998e-06 2023-07-02T14:04:48.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338258+0000", 2023-07-02T14:04:48.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032085734999999997 2023-07-02T14:04:48.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370344+0000", 2023-07-02T14:04:48.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108621 2023-07-02T14:04:48.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370452+0000", 2023-07-02T14:04:48.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014936499999999999 2023-07-02T14:04:48.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:48.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.370602+0000", 2023-07-02T14:04:48.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001778012 2023-07-02T14:04:48.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372380+0000", 2023-07-02T14:04:48.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044876380000000004 2023-07-02T14:04:48.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376867+0000", 2023-07-02T14:04:48.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9870999999999999e-05 2023-07-02T14:04:48.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376897+0000", 2023-07-02T14:04:48.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011401 2023-07-02T14:04:48.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.377011+0000", 2023-07-02T14:04:48.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038814000000000001 2023-07-02T14:04:48.290 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.290 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.290 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.290 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:3878 2.1 e80/79 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head v 80'436, mlcod=80'429)", 2023-07-02T14:04:48.290 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:48.291 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 69.975447571999993, 2023-07-02T14:04:48.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037799472000000001, 2023-07-02T14:04:48.291 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.291 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.291 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:48.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107451+0000", 2023-07-02T14:04:48.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:48.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.107450+0000", 2023-07-02T14:04:48.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0023914520000000001 2023-07-02T14:04:48.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109841+0000", 2023-07-02T14:04:48.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0823e-05 2023-07-02T14:04:48.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109862+0000", 2023-07-02T14:04:48.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1447000000000001e-05 2023-07-02T14:04:48.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.109874+0000", 2023-07-02T14:04:48.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00075009600000000001 2023-07-02T14:04:48.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.110624+0000", 2023-07-02T14:04:48.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5176000000000001e-05 2023-07-02T14:04:48.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.110649+0000", 2023-07-02T14:04:48.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034557310000000001 2023-07-02T14:04:48.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145206+0000", 2023-07-02T14:04:48.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4110999999999998e-05 2023-07-02T14:04:48.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145250+0000", 2023-07-02T14:04:48.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037799472000000001 2023-07-02T14:04:48.297 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.298 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.298 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.298 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:196 2.5 2:a6e4b3cf:::rbd_data.125de1449605.00000000000000c4:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:04:48.298 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172685+0000", 2023-07-02T14:04:48.298 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.910213412999999, 2023-07-02T14:04:48.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038025259999999998, 2023-07-02T14:04:48.299 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.299 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:48.299 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.299 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:04:48.299 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:04:48.299 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 196 2023-07-02T14:04:48.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.299 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172685+0000", 2023-07-02T14:04:48.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172685+0000", 2023-07-02T14:04:48.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:48.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172684+0000", 2023-07-02T14:04:48.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.693e-06 2023-07-02T14:04:48.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172688+0000", 2023-07-02T14:04:48.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4470000000000003e-06 2023-07-02T14:04:48.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172695+0000", 2023-07-02T14:04:48.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.014e-06 2023-07-02T14:04:48.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172699+0000", 2023-07-02T14:04:48.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025360667 2023-07-02T14:04:48.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.198059+0000", 2023-07-02T14:04:48.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011197900000000001 2023-07-02T14:04:48.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.198171+0000", 2023-07-02T14:04:48.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012538942000000001 2023-07-02T14:04:48.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.210710+0000", 2023-07-02T14:04:48.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038025259999999998 2023-07-02T14:04:48.305 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.305 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.305 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.306 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.306 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.306 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:48.307 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:04:48.307 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:48.307+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:48.408 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:48.407+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:48.474 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:48.474 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:48.474 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:48.474 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:48.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:04:48.530 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:48.526+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:48.607 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:48.608 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:48.608 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:48.608 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:48.608 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:48.609 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:48.608+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:48.617 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:04:48.710 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:48.709+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:48.783 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:48.783 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:48.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:48.784 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:48.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.784 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:04:48.784 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:48.784 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 99.329148567999994, 2023-07-02T14:04:48.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:04:48.784 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.785 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:48.785 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.785 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.785 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.785 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:04:48.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.785 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:48.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:48.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:48.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:04:48.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:04:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:04:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:04:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:04:48.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:04:48.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:04:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:04:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:04:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:04:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:04:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:04:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:04:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:04:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:04:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:04:48.792 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:04:48.793 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:48.794 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 98.458455560000004, 2023-07-02T14:04:48.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:04:48.794 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.794 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:48.794 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.794 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:04:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:04:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:04:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:48.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:04:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:04:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:04:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:04:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:04:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:04:48.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:04:48.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:04:48.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:04:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:04:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:04:48.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:04:48.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:04:48.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:04:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:04:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:04:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:04:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:04:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.804 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:48.804 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:48.804 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 98.458330838999998, 2023-07-02T14:04:48.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:04:48.804 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.805 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:48.805 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.805 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.805 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.805 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:04:48.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:04:48.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:04:48.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:04:48.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:04:48.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:04:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:04:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:04:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:04:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:04:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:04:48.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:04:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:04:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:04:48.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:04:48.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:04:48.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:04:48.812 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.813 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.813 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.813 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:48.813 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:48.813 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 98.451480954999994, 2023-07-02T14:04:48.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:04:48.814 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.814 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:48.814 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.814 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.814 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.815 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:04:48.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.815 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:48.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:48.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:04:48.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:04:48.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:04:48.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:04:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:04:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:04:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:04:48.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:04:48.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:04:48.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:04:48.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:04:48.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:04:48.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:04:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:04:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:04:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:04:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:04:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 98.451418029999999, 2023-07-02T14:04:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:04:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:48.823 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:04:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:48.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:48.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:48.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:04:48.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:04:48.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:04:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:04:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:04:48.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:04:48.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:04:48.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:04:48.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:04:48.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:04:48.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:04:48.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:04:48.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:04:48.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:04:48.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:04:48.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:04:48.832 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.832 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 98.451117608999994, 2023-07-02T14:04:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:04:48.833 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:04:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:48.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:04:48.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:04:48.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:04:48.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:04:48.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:04:48.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:04:48.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:04:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:04:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:04:48.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:04:48.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:04:48.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:04:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:04:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:04:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:04:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:04:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:04:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:04:48.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:04:48.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:04:48.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:04:48.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:04:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:04:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:04:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:04:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:04:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:04:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:04:48.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:04:48.846 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.846 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:04:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 98.450593108999996, 2023-07-02T14:04:48.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:04:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:04:48.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.849 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:48.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:48.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:04:48.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:04:48.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:04:48.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:04:48.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:04:48.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:04:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:04:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:04:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:04:48.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:04:48.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:04:48.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:04:48.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:04:48.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:04:48.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:04:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:04:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:04:48.856 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:04:48.857 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.326573984000007, 2023-07-02T14:04:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:04:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.858 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:04:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:48.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:04:48.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:04:48.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:04:48.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:04:48.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:04:48.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:04:48.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:04:48.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:04:48.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:04:48.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:04:48.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:04:48.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:04:48.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:04:48.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:04:48.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:04:48.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:04:48.866 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.866 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.866 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.867 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:48.867 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:48.867 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.001091279999997, 2023-07-02T14:04:48.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:04:48.867 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.868 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.868 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.868 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.868 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.868 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:04:48.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.868 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:48.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:48.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:48.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:04:48.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:04:48.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:04:48.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:04:48.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:04:48.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:04:48.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:04:48.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:04:48.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:48.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:04:48.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.874 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:04:48.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:04:48.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.874 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:04:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:04:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:04:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:04:48.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:48.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:04:48.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:04:48.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:04:48.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:04:48.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:04:48.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:04:48.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:04:48.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:04:48.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:04:48.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:04:48.879 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.879 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.880 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.880 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3887 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:48.880 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:48.880 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.997228090999997, 2023-07-02T14:04:48.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998, 2023-07-02T14:04:48.881 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.881 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.881 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.881 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.881 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.881 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3887 2023-07-02T14:04:48.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.882 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:48.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:48.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:48.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784019+0000", 2023-07-02T14:04:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4073999999999998e-05 2023-07-02T14:04:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784063+0000", 2023-07-02T14:04:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6630000000000004e-06 2023-07-02T14:04:48.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784070+0000", 2023-07-02T14:04:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.286e-06 2023-07-02T14:04:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.885 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784073+0000", 2023-07-02T14:04:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4941999999999998e-05 2023-07-02T14:04:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784118+0000", 2023-07-02T14:04:48.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8120000000000008e-06 2023-07-02T14:04:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784126+0000", 2023-07-02T14:04:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32999709399999999 2023-07-02T14:04:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114123+0000", 2023-07-02T14:04:48.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000157257 2023-07-02T14:04:48.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114280+0000", 2023-07-02T14:04:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000341152 2023-07-02T14:04:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:48.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114622+0000", 2023-07-02T14:04:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019539766 2023-07-02T14:04:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.134161+0000", 2023-07-02T14:04:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015820796000000002 2023-07-02T14:04:48.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149982+0000", 2023-07-02T14:04:48.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.05e-05 2023-07-02T14:04:48.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150043+0000", 2023-07-02T14:04:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6453999999999999e-05 2023-07-02T14:04:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150089+0000", 2023-07-02T14:04:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998 2023-07-02T14:04:48.892 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3939 2.1 2:9ef34a83:::rbd_data.125de1449605.00000000000000b8:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:48.893 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.997170582999999, 2023-07-02T14:04:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001, 2023-07-02T14:04:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.894 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3939 2023-07-02T14:04:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:48.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:48.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:48.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046852830000000002 2023-07-02T14:04:48.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788763+0000", 2023-07-02T14:04:48.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3185e-05 2023-07-02T14:04:48.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788776+0000", 2023-07-02T14:04:48.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8470000000000003e-06 2023-07-02T14:04:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788781+0000", 2023-07-02T14:04:48.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7104000000000002e-05 2023-07-02T14:04:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788828+0000", 2023-07-02T14:04:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3516000000000001e-05 2023-07-02T14:04:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788841+0000", 2023-07-02T14:04:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33064513600000001 2023-07-02T14:04:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119487+0000", 2023-07-02T14:04:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9876999999999997e-05 2023-07-02T14:04:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119537+0000", 2023-07-02T14:04:48.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014160999999999999 2023-07-02T14:04:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119678+0000", 2023-07-02T14:04:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023054392 2023-07-02T14:04:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.142733+0000", 2023-07-02T14:04:48.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074065470000000003 2023-07-02T14:04:48.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150139+0000", 2023-07-02T14:04:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5025999999999997e-05 2023-07-02T14:04:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150204+0000", 2023-07-02T14:04:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147999999999997e-05 2023-07-02T14:04:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150241+0000", 2023-07-02T14:04:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001 2023-07-02T14:04:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3940 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.992459550999996, 2023-07-02T14:04:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001, 2023-07-02T14:04:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.907 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3940 2023-07-02T14:04:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788788+0000", 2023-07-02T14:04:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043166560000000003 2023-07-02T14:04:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793105+0000", 2023-07-02T14:04:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5970000000000001e-05 2023-07-02T14:04:48.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793121+0000", 2023-07-02T14:04:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0939999999999998e-06 2023-07-02T14:04:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793126+0000", 2023-07-02T14:04:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2438000000000002e-05 2023-07-02T14:04:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793178+0000", 2023-07-02T14:04:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2019e-05 2023-07-02T14:04:48.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793190+0000", 2023-07-02T14:04:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33208757799999999 2023-07-02T14:04:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125278+0000", 2023-07-02T14:04:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5255999999999997e-05 2023-07-02T14:04:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125373+0000", 2023-07-02T14:04:48.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015674999999999999 2023-07-02T14:04:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125530+0000", 2023-07-02T14:04:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018767308 2023-07-02T14:04:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144297+0000", 2023-07-02T14:04:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060291959999999997 2023-07-02T14:04:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150326+0000", 2023-07-02T14:04:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3886999999999993e-05 2023-07-02T14:04:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150390+0000", 2023-07-02T14:04:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010255 2023-07-02T14:04:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150493+0000", 2023-07-02T14:04:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001 2023-07-02T14:04:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.919 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3947 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.988114023999998, 2023-07-02T14:04:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999, 2023-07-02T14:04:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3947 2023-07-02T14:04:48.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793133+0000", 2023-07-02T14:04:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2519999999999999e-05 2023-07-02T14:04:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793176+0000", 2023-07-02T14:04:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:04:48.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793180+0000", 2023-07-02T14:04:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1709999999999998e-06 2023-07-02T14:04:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793184+0000", 2023-07-02T14:04:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1721999999999999e-05 2023-07-02T14:04:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793215+0000", 2023-07-02T14:04:48.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1280000000000008e-06 2023-07-02T14:04:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793224+0000", 2023-07-02T14:04:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34218549599999998 2023-07-02T14:04:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135409+0000", 2023-07-02T14:04:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117442 2023-07-02T14:04:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135526+0000", 2023-07-02T14:04:48.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000206257 2023-07-02T14:04:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135733+0000", 2023-07-02T14:04:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085700029999999996 2023-07-02T14:04:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144303+0000", 2023-07-02T14:04:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006278652 2023-07-02T14:04:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150581+0000", 2023-07-02T14:04:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4765999999999996e-05 2023-07-02T14:04:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150646+0000", 2023-07-02T14:04:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4777999999999995e-05 2023-07-02T14:04:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150711+0000", 2023-07-02T14:04:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999 2023-07-02T14:04:48.932 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3980 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 65536~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:48.933 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.988060141999995, 2023-07-02T14:04:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999, 2023-07-02T14:04:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.934 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3980 2023-07-02T14:04:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020244260000000002 2023-07-02T14:04:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795212+0000", 2023-07-02T14:04:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0005e-05 2023-07-02T14:04:48.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795222+0000", 2023-07-02T14:04:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9860000000000002e-06 2023-07-02T14:04:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795227+0000", 2023-07-02T14:04:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6369999999999999e-05 2023-07-02T14:04:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795264+0000", 2023-07-02T14:04:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1052e-05 2023-07-02T14:04:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795275+0000", 2023-07-02T14:04:48.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.343850459 2023-07-02T14:04:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139125+0000", 2023-07-02T14:04:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6760000000000004e-05 2023-07-02T14:04:48.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139202+0000", 2023-07-02T14:04:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015092799999999999 2023-07-02T14:04:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139353+0000", 2023-07-02T14:04:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011425940000000001 2023-07-02T14:04:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150779+0000", 2023-07-02T14:04:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033223060000000001 2023-07-02T14:04:48.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154101+0000", 2023-07-02T14:04:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9268999999999998e-05 2023-07-02T14:04:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154161+0000", 2023-07-02T14:04:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6499999999999999e-05 2023-07-02T14:04:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154207+0000", 2023-07-02T14:04:48.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999 2023-07-02T14:04:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3982 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1835008~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:48.946 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.986014854999993, 2023-07-02T14:04:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002, 2023-07-02T14:04:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.947 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3982 2023-07-02T14:04:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:48.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795233+0000", 2023-07-02T14:04:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8680999999999998e-05 2023-07-02T14:04:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795272+0000", 2023-07-02T14:04:48.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5629999999999999e-06 2023-07-02T14:04:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795277+0000", 2023-07-02T14:04:48.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8700000000000001e-06 2023-07-02T14:04:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795280+0000", 2023-07-02T14:04:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1964000000000001e-05 2023-07-02T14:04:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795312+0000", 2023-07-02T14:04:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.314e-06 2023-07-02T14:04:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795319+0000", 2023-07-02T14:04:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34828442599999998 2023-07-02T14:04:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143603+0000", 2023-07-02T14:04:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2438000000000001e-05 2023-07-02T14:04:48.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143665+0000", 2023-07-02T14:04:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3079999999999997e-05 2023-07-02T14:04:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143759+0000", 2023-07-02T14:04:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070633079999999999 2023-07-02T14:04:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150822+0000", 2023-07-02T14:04:48.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034430210000000001 2023-07-02T14:04:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154265+0000", 2023-07-02T14:04:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4249999999999998e-05 2023-07-02T14:04:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154309+0000", 2023-07-02T14:04:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2020000000000001e-05 2023-07-02T14:04:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154351+0000", 2023-07-02T14:04:48.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002 2023-07-02T14:04:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4054 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1900544~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:48.959 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.985963183999999, 2023-07-02T14:04:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002, 2023-07-02T14:04:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.960 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.961 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.961 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4054 2023-07-02T14:04:48.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.961 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018874860000000001 2023-07-02T14:04:48.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797172+0000", 2023-07-02T14:04:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3000000000000007e-06 2023-07-02T14:04:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797182+0000", 2023-07-02T14:04:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6260000000000003e-06 2023-07-02T14:04:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797186+0000", 2023-07-02T14:04:48.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9013e-05 2023-07-02T14:04:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797235+0000", 2023-07-02T14:04:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5990000000000008e-06 2023-07-02T14:04:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797245+0000", 2023-07-02T14:04:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34785621999999999 2023-07-02T14:04:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145101+0000", 2023-07-02T14:04:48.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7762000000000006e-05 2023-07-02T14:04:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145169+0000", 2023-07-02T14:04:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013469400000000001 2023-07-02T14:04:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145304+0000", 2023-07-02T14:04:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089538039999999992 2023-07-02T14:04:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154257+0000", 2023-07-02T14:04:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104218 2023-07-02T14:04:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154362+0000", 2023-07-02T14:04:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4168000000000001e-05 2023-07-02T14:04:48.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154406+0000", 2023-07-02T14:04:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3098000000000001e-05 2023-07-02T14:04:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154429+0000", 2023-07-02T14:04:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002 2023-07-02T14:04:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4056 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3670016~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.984056671000005, 2023-07-02T14:04:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997, 2023-07-02T14:04:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.973 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4056 2023-07-02T14:04:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:48.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:48.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797191+0000", 2023-07-02T14:04:48.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0438999999999997e-05 2023-07-02T14:04:48.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797232+0000", 2023-07-02T14:04:48.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6929999999999998e-06 2023-07-02T14:04:48.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797237+0000", 2023-07-02T14:04:48.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9739999999999998e-06 2023-07-02T14:04:48.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797240+0000", 2023-07-02T14:04:48.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0243000000000002e-05 2023-07-02T14:04:48.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797270+0000", 2023-07-02T14:04:48.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9889999999999997e-06 2023-07-02T14:04:48.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797276+0000", 2023-07-02T14:04:48.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35358319799999999 2023-07-02T14:04:48.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150859+0000", 2023-07-02T14:04:48.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1357e-05 2023-07-02T14:04:48.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150940+0000", 2023-07-02T14:04:48.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014903 2023-07-02T14:04:48.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:48.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151089+0000", 2023-07-02T14:04:48.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033512519999999999 2023-07-02T14:04:48.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154441+0000", 2023-07-02T14:04:48.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001050945 2023-07-02T14:04:48.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155492+0000", 2023-07-02T14:04:48.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2523e-05 2023-07-02T14:04:48.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155524+0000", 2023-07-02T14:04:48.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.057e-05 2023-07-02T14:04:48.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155555+0000", 2023-07-02T14:04:48.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997 2023-07-02T14:04:48.985 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.986 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.986 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.986 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4088 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3735552~458752] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:48.986 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:48.986 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.984004475999996, 2023-07-02T14:04:48.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998, 2023-07-02T14:04:48.987 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:48.987 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:48.987 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:48.987 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:48.987 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:48.987 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4088 2023-07-02T14:04:48.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.988 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:48.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:48.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:48.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:48.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:48.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:48.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:48.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:48.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:48.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00038503499999999999 2023-07-02T14:04:48.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:48.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797629+0000", 2023-07-02T14:04:48.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8739999999999995e-06 2023-07-02T14:04:48.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:48.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797637+0000", 2023-07-02T14:04:48.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7160000000000001e-06 2023-07-02T14:04:48.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:48.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797640+0000", 2023-07-02T14:04:48.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4342999999999998e-05 2023-07-02T14:04:48.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797695+0000", 2023-07-02T14:04:48.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0939999999999996e-06 2023-07-02T14:04:48.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:48.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797701+0000", 2023-07-02T14:04:48.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35408540799999999 2023-07-02T14:04:48.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:48.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151786+0000", 2023-07-02T14:04:48.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1953000000000006e-05 2023-07-02T14:04:48.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:48.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151868+0000", 2023-07-02T14:04:48.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015162800000000001 2023-07-02T14:04:48.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:48.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.152020+0000", 2023-07-02T14:04:48.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054561870000000004 2023-07-02T14:04:48.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:48.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.157476+0000", 2023-07-02T14:04:48.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059153199999999997 2023-07-02T14:04:48.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:48.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158068+0000", 2023-07-02T14:04:48.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2799e-05 2023-07-02T14:04:48.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:48.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158090+0000", 2023-07-02T14:04:48.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7235999999999999e-05 2023-07-02T14:04:48.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:48.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158108+0000", 2023-07-02T14:04:48.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998 2023-07-02T14:04:48.998 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.999 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:48.999 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:48.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:48.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:48.999 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4110 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:48.999 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:48.999 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.983603199000001, 2023-07-02T14:04:48.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864, 2023-07-02T14:04:49.000 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:49.000 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:49.000 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:49.000 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:49.000 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:49.000 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4110 2023-07-02T14:04:49.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.001 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:49.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:49.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:49.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:49.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:49.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:49.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:49.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:49.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:49.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9331e-05 2023-07-02T14:04:49.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:49.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797684+0000", 2023-07-02T14:04:49.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.147e-06 2023-07-02T14:04:49.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:49.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797689+0000", 2023-07-02T14:04:49.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5020000000000001e-06 2023-07-02T14:04:49.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:49.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797693+0000", 2023-07-02T14:04:49.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5254999999999999e-05 2023-07-02T14:04:49.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:49.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797748+0000", 2023-07-02T14:04:49.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5520000000000005e-06 2023-07-02T14:04:49.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:49.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797757+0000", 2023-07-02T14:04:49.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35674245700000001 2023-07-02T14:04:49.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:49.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154499+0000", 2023-07-02T14:04:49.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0090000000000001e-06 2023-07-02T14:04:49.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:49.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154506+0000", 2023-07-02T14:04:49.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864 2023-07-02T14:04:49.008 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:49.008 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:49.009 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:49.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.009 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:49.009 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:49.009 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 70.983551113000004, 2023-07-02T14:04:49.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998, 2023-07-02T14:04:49.009 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:49.010 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:49.010 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:49.010 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:49.010 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:49.010 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:04:49.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.010 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:49.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:49.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:49.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:49.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:49.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:49.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:49.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:49.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:49.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001714819 2023-07-02T14:04:49.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:49.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799412+0000", 2023-07-02T14:04:49.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7779999999999996e-06 2023-07-02T14:04:49.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:49.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799421+0000", 2023-07-02T14:04:49.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3510000000000002e-06 2023-07-02T14:04:49.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:49.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799424+0000", 2023-07-02T14:04:49.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0275999999999999e-05 2023-07-02T14:04:49.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:49.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799464+0000", 2023-07-02T14:04:49.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8389999999999996e-06 2023-07-02T14:04:49.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:49.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799473+0000", 2023-07-02T14:04:49.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35507638800000002 2023-07-02T14:04:49.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:49.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154550+0000", 2023-07-02T14:04:49.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6309999999999998e-06 2023-07-02T14:04:49.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:49.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:49.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:49.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154559+0000", 2023-07-02T14:04:49.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998 2023-07-02T14:04:49.018 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:49.018 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:49.018 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:49.018 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:49.018 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:49.018 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:49.019 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:04:49.020 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:04:49.021 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:49.019+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:49.121 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:49.120+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:49.193 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:49.193 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:49.194 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:49.194 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:49.208 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:04:49.213 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:49.222 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:49.221+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:49.231 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:49.249 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:49.268 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:49.286 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:49.287 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:04:49.287 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:04:49.287 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:04:49.287 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:49.287 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:04:49.288 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:04:49.289 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:49.290 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:49.331 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:49.330+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:49.338 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:49.338 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:49.349 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:04:49.400 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:04:49.405 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 1 2023-07-02T14:04:49.423 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:49.422+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:49.513 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:49.514 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:49.524 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:04:49.524 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:49.523+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:49.566 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:04:49.625 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:49.624+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:49.671 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:49.671 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:49.681 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:04:49.764 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:49.764+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:49.826 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:49.825+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:49.845 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:49.846 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:49.856 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:04:49.939 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:49.938+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:50.019 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:50.020 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:50.028 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:50.027+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:50.029 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:04:50.129 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:50.128+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:50.193 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:50.193 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:50.203 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:04:50.230 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:50.229+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:50.326 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:50.326 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:50.331 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:50.330+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:50.336 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:50.386 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:04:50.406 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap28 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap29 b3fbb028-e267-4d25-a750-93ba18a2316a@snap29.diff 2023-07-02T14:04:50.441 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:50.440+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:50.470 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 51% complete...2023-07-02T14:04:50.463+0000 7fde475ca700 0 -- 172.21.15.163:0/2398305390 >> [v2:172.21.15.16:6800/83359,v1:172.21.15.16:6801/83359] conn(0x7fde24022940 msgr2=0x7fde24024da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-07-02T14:04:50.471 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:50.471 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:50.471 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:50.478 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:50.477+0000 7fde485cc700 0 -- 172.21.15.163:0/2398305390 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x564968745eb0 msgr2=0x5649687482a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:50.533 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:50.532+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:50.633 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:50.632+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:50.650 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:50.650 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:50.660 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:04:50.734 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:50.733+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:50.824 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:50.824 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:50.834 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:04:50.835 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:50.833+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:50.935 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:50.934+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:50.998 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:50.998 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:51.007 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:04:51.048 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:51.047+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:51.135 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 100% complete...done. 2023-07-02T14:04:51.137 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:51.136+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:51.138 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:04:51.138 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=30 2023-07-02T14:04:51.139 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap30 2023-07-02T14:04:51.173 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:51.174 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:51.184 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:04:51.268 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:51.267+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:51.291 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:04:51.339 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:51.338+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:51.350 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:51.350 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:51.361 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:04:51.410 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:04:51.416 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 4 2023-07-02T14:04:51.437 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:51.439 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:51.438+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:51.455 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:51.473 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:51.492 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:51.511 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:51.511 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:04:51.511 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:04:51.511 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:04:51.512 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:51.512 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:04:51.512 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:04:51.514 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:51.514 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:51.531 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:51.531 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:51.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:51.531 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:51.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.532 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:04:51.532 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:51.532 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 102.07724499699999, 2023-07-02T14:04:51.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:04:51.532 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:51.533 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:51.533 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:51.533 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:51.533 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:51.533 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:04:51.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.533 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:51.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:51.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:51.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:51.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:51.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:51.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:51.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:51.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:04:51.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:04:51.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:51.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:04:51.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:04:51.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:51.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:04:51.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:04:51.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:51.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:04:51.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:04:51.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:04:51.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:04:51.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:51.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:04:51.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:04:51.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:04:51.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:04:51.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:51.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:04:51.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:04:51.540 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.540 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.540 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.540 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:04:51.540 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:51.541 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 101.206551989, 2023-07-02T14:04:51.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:04:51.541 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:51.541 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:51.541 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:51.541 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:04:51.541 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:04:51.541 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:04:51.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.542 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:51.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:51.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:51.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:51.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:51.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:51.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:51.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:51.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:04:51.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:04:51.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:51.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:04:51.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:04:51.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:51.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:04:51.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:04:51.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:51.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:04:51.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:04:51.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:04:51.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:04:51.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:51.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:04:51.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:04:51.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:04:51.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:04:51.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:51.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:04:51.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:04:51.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:51.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:04:51.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:04:51.549 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.549 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.549 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.549 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 101.206427268, 2023-07-02T14:04:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:04:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:51.550 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:04:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:51.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:51.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:51.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:51.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:51.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:04:51.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:04:51.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:51.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:04:51.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:04:51.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:51.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:04:51.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:04:51.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:51.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:04:51.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:04:51.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:04:51.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:04:51.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:51.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:04:51.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:04:51.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:04:51.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:04:51.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:51.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:04:51.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:04:51.557 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.557 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.557 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.558 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:51.558 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:51.558 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 101.19957738399999, 2023-07-02T14:04:51.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:04:51.558 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:51.558 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:51.559 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:51.559 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:51.559 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:51.559 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:04:51.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.559 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:51.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:51.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:51.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:51.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:51.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:51.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:04:51.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:51.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:04:51.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:04:51.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:51.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:04:51.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:04:51.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:51.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:04:51.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:04:51.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:51.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:04:51.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:04:51.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:04:51.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:04:51.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:51.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:04:51.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:04:51.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:04:51.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:04:51.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:51.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:04:51.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:04:51.566 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.566 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.566 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.566 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:51.567 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:51.567 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 101.199514459, 2023-07-02T14:04:51.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:04:51.567 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:51.567 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:51.567 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:51.567 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:51.567 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:51.568 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:04:51.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.568 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:51.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:51.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:51.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:04:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:04:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:04:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:04:51.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:51.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:04:51.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:04:51.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:51.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:04:51.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:04:51.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:04:51.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:04:51.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:51.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:04:51.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:04:51.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:04:51.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:04:51.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:51.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:04:51.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:04:51.574 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.574 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.574 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.575 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:51.575 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:51.575 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 101.19921403799999, 2023-07-02T14:04:51.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:04:51.575 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:51.575 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:51.576 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:51.576 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:51.576 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:51.576 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:04:51.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.576 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:51.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:51.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:51.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:51.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:51.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:51.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:51.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:51.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:04:51.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:04:51.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:51.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:04:51.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:04:51.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:51.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:04:51.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:04:51.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:51.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:04:51.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:04:51.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:04:51.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:04:51.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:51.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:04:51.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:04:51.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:04:51.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:04:51.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:51.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:04:51.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:04:51.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:04:51.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:04:51.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:04:51.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:51.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:04:51.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:04:51.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:51.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:04:51.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:04:51.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:51.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:04:51.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:04:51.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:51.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:04:51.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:04:51.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:51.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:04:51.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:04:51.586 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.587 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.587 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.587 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:04:51.587 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:51.587 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 101.198689538, 2023-07-02T14:04:51.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:04:51.588 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:51.588 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:51.588 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:51.588 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:51.588 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:51.588 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:04:51.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.588 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:51.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:51.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:51.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:51.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:51.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:51.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:51.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:51.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:51.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:04:51.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:51.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:04:51.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:04:51.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:51.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:04:51.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:04:51.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:51.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:04:51.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:04:51.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:04:51.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:04:51.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:51.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:04:51.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:04:51.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:04:51.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:04:51.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:51.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:04:51.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:04:51.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:51.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:04:51.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:04:51.596 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.596 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.596 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.596 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:04:51.597 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:51.597 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.074670413000007, 2023-07-02T14:04:51.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:04:51.597 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:51.597 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:51.597 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:51.597 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:51.597 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:51.598 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:04:51.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.598 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:51.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:51.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:51.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:51.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:51.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:51.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:51.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:51.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:04:51.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:04:51.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:51.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:04:51.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:04:51.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:51.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:04:51.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:04:51.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:51.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:04:51.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:04:51.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:04:51.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:04:51.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:51.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:04:51.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:04:51.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:04:51.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:04:51.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:51.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:04:51.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:04:51.604 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.604 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.604 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.605 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:51.605 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:51.605 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.749187708999997, 2023-07-02T14:04:51.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:04:51.605 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:51.606 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:51.606 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:51.606 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:51.606 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:51.606 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:04:51.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.606 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:51.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:51.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:51.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:51.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:51.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:51.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:51.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:51.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:04:51.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:04:51.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:51.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:04:51.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:04:51.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:51.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:04:51.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:04:51.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:51.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:04:51.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:04:51.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:51.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:04:51.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:51.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:04:51.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:04:51.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:04:51.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:04:51.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:51.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:04:51.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:04:51.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:51.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:04:51.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:04:51.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:51.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:04:51.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:04:51.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:51.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:04:51.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:04:51.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:51.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:04:51.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:04:51.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:51.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:04:51.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:04:51.616 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.616 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.616 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.616 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3887 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:51.617 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:51.617 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.745324519999997, 2023-07-02T14:04:51.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998, 2023-07-02T14:04:51.617 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:51.617 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:51.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:51.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:51.618 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:51.618 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3887 2023-07-02T14:04:51.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.618 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:51.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:51.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:51.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:51.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:51.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:51.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:51.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:51.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784019+0000", 2023-07-02T14:04:51.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4073999999999998e-05 2023-07-02T14:04:51.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:51.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784063+0000", 2023-07-02T14:04:51.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6630000000000004e-06 2023-07-02T14:04:51.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:51.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784070+0000", 2023-07-02T14:04:51.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.286e-06 2023-07-02T14:04:51.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:51.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784073+0000", 2023-07-02T14:04:51.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4941999999999998e-05 2023-07-02T14:04:51.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784118+0000", 2023-07-02T14:04:51.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8120000000000008e-06 2023-07-02T14:04:51.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:51.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784126+0000", 2023-07-02T14:04:51.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32999709399999999 2023-07-02T14:04:51.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114123+0000", 2023-07-02T14:04:51.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000157257 2023-07-02T14:04:51.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:51.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114280+0000", 2023-07-02T14:04:51.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000341152 2023-07-02T14:04:51.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:51.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114622+0000", 2023-07-02T14:04:51.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019539766 2023-07-02T14:04:51.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:51.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.134161+0000", 2023-07-02T14:04:51.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015820796000000002 2023-07-02T14:04:51.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:51.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149982+0000", 2023-07-02T14:04:51.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.05e-05 2023-07-02T14:04:51.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:51.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150043+0000", 2023-07-02T14:04:51.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6453999999999999e-05 2023-07-02T14:04:51.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:51.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150089+0000", 2023-07-02T14:04:51.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998 2023-07-02T14:04:51.628 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.628 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.628 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.628 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3939 2.1 2:9ef34a83:::rbd_data.125de1449605.00000000000000b8:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:51.629 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:51.629 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.745267011999999, 2023-07-02T14:04:51.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001, 2023-07-02T14:04:51.629 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:51.629 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:51.629 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:51.629 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:51.629 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:51.630 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3939 2023-07-02T14:04:51.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.630 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:51.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:51.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:51.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:51.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:51.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:51.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:51.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:51.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:51.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046852830000000002 2023-07-02T14:04:51.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:51.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788763+0000", 2023-07-02T14:04:51.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3185e-05 2023-07-02T14:04:51.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:51.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788776+0000", 2023-07-02T14:04:51.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8470000000000003e-06 2023-07-02T14:04:51.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:51.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788781+0000", 2023-07-02T14:04:51.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7104000000000002e-05 2023-07-02T14:04:51.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788828+0000", 2023-07-02T14:04:51.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3516000000000001e-05 2023-07-02T14:04:51.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:51.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788841+0000", 2023-07-02T14:04:51.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33064513600000001 2023-07-02T14:04:51.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119487+0000", 2023-07-02T14:04:51.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9876999999999997e-05 2023-07-02T14:04:51.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:51.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119537+0000", 2023-07-02T14:04:51.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014160999999999999 2023-07-02T14:04:51.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:51.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119678+0000", 2023-07-02T14:04:51.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023054392 2023-07-02T14:04:51.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:51.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.142733+0000", 2023-07-02T14:04:51.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074065470000000003 2023-07-02T14:04:51.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:51.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150139+0000", 2023-07-02T14:04:51.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5025999999999997e-05 2023-07-02T14:04:51.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:51.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150204+0000", 2023-07-02T14:04:51.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147999999999997e-05 2023-07-02T14:04:51.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:51.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150241+0000", 2023-07-02T14:04:51.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001 2023-07-02T14:04:51.640 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.640 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.640 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.640 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3940 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:51.640 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:51.641 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.740555979999996, 2023-07-02T14:04:51.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001, 2023-07-02T14:04:51.641 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:51.641 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:51.641 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:51.641 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:51.641 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:51.641 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3940 2023-07-02T14:04:51.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.642 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:51.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:51.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:51.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:51.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:51.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:51.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:51.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:51.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788788+0000", 2023-07-02T14:04:51.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043166560000000003 2023-07-02T14:04:51.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:51.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793105+0000", 2023-07-02T14:04:51.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5970000000000001e-05 2023-07-02T14:04:51.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:51.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793121+0000", 2023-07-02T14:04:51.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0939999999999998e-06 2023-07-02T14:04:51.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:51.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793126+0000", 2023-07-02T14:04:51.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2438000000000002e-05 2023-07-02T14:04:51.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793178+0000", 2023-07-02T14:04:51.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2019e-05 2023-07-02T14:04:51.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:51.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793190+0000", 2023-07-02T14:04:51.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33208757799999999 2023-07-02T14:04:51.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125278+0000", 2023-07-02T14:04:51.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5255999999999997e-05 2023-07-02T14:04:51.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:51.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125373+0000", 2023-07-02T14:04:51.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015674999999999999 2023-07-02T14:04:51.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:51.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125530+0000", 2023-07-02T14:04:51.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018767308 2023-07-02T14:04:51.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:51.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144297+0000", 2023-07-02T14:04:51.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060291959999999997 2023-07-02T14:04:51.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:51.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150326+0000", 2023-07-02T14:04:51.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3886999999999993e-05 2023-07-02T14:04:51.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:51.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150390+0000", 2023-07-02T14:04:51.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010255 2023-07-02T14:04:51.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:51.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150493+0000", 2023-07-02T14:04:51.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001 2023-07-02T14:04:51.651 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.651 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.651 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.652 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3947 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:51.652 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:51.652 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.736210452999998, 2023-07-02T14:04:51.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999, 2023-07-02T14:04:51.652 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:51.653 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:51.653 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:51.653 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:51.653 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:51.653 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3947 2023-07-02T14:04:51.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.653 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:51.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:51.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:51.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:51.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:51.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:51.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:51.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:51.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793133+0000", 2023-07-02T14:04:51.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2519999999999999e-05 2023-07-02T14:04:51.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:51.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793176+0000", 2023-07-02T14:04:51.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:04:51.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:51.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793180+0000", 2023-07-02T14:04:51.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1709999999999998e-06 2023-07-02T14:04:51.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:51.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793184+0000", 2023-07-02T14:04:51.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1721999999999999e-05 2023-07-02T14:04:51.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793215+0000", 2023-07-02T14:04:51.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1280000000000008e-06 2023-07-02T14:04:51.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:51.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793224+0000", 2023-07-02T14:04:51.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34218549599999998 2023-07-02T14:04:51.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135409+0000", 2023-07-02T14:04:51.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117442 2023-07-02T14:04:51.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:51.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135526+0000", 2023-07-02T14:04:51.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000206257 2023-07-02T14:04:51.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:51.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135733+0000", 2023-07-02T14:04:51.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085700029999999996 2023-07-02T14:04:51.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:51.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144303+0000", 2023-07-02T14:04:51.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006278652 2023-07-02T14:04:51.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:51.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150581+0000", 2023-07-02T14:04:51.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4765999999999996e-05 2023-07-02T14:04:51.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:51.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150646+0000", 2023-07-02T14:04:51.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4777999999999995e-05 2023-07-02T14:04:51.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:51.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150711+0000", 2023-07-02T14:04:51.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999 2023-07-02T14:04:51.663 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.663 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.663 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.664 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3980 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 65536~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:51.664 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:51.664 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.736156570999995, 2023-07-02T14:04:51.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999, 2023-07-02T14:04:51.664 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:51.664 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:51.664 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:51.664 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:51.665 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:51.665 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3980 2023-07-02T14:04:51.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.665 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:51.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:51.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:51.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:51.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:51.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:51.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:51.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:51.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:51.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020244260000000002 2023-07-02T14:04:51.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:51.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795212+0000", 2023-07-02T14:04:51.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0005e-05 2023-07-02T14:04:51.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:51.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795222+0000", 2023-07-02T14:04:51.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9860000000000002e-06 2023-07-02T14:04:51.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:51.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795227+0000", 2023-07-02T14:04:51.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6369999999999999e-05 2023-07-02T14:04:51.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795264+0000", 2023-07-02T14:04:51.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1052e-05 2023-07-02T14:04:51.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:51.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795275+0000", 2023-07-02T14:04:51.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.343850459 2023-07-02T14:04:51.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139125+0000", 2023-07-02T14:04:51.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6760000000000004e-05 2023-07-02T14:04:51.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:51.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139202+0000", 2023-07-02T14:04:51.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015092799999999999 2023-07-02T14:04:51.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:51.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139353+0000", 2023-07-02T14:04:51.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011425940000000001 2023-07-02T14:04:51.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:51.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150779+0000", 2023-07-02T14:04:51.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033223060000000001 2023-07-02T14:04:51.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:51.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154101+0000", 2023-07-02T14:04:51.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9268999999999998e-05 2023-07-02T14:04:51.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:51.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154161+0000", 2023-07-02T14:04:51.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6499999999999999e-05 2023-07-02T14:04:51.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:51.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154207+0000", 2023-07-02T14:04:51.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999 2023-07-02T14:04:51.675 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.675 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.675 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.675 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3982 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1835008~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:51.675 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:51.676 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.734111283999994, 2023-07-02T14:04:51.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002, 2023-07-02T14:04:51.676 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:51.676 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:51.676 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:51.676 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:51.676 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:51.676 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3982 2023-07-02T14:04:51.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.677 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:51.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:51.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:51.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:51.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:51.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:51.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:51.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:51.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795233+0000", 2023-07-02T14:04:51.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8680999999999998e-05 2023-07-02T14:04:51.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:51.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795272+0000", 2023-07-02T14:04:51.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5629999999999999e-06 2023-07-02T14:04:51.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795277+0000", 2023-07-02T14:04:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8700000000000001e-06 2023-07-02T14:04:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795280+0000", 2023-07-02T14:04:51.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1964000000000001e-05 2023-07-02T14:04:51.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795312+0000", 2023-07-02T14:04:51.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.314e-06 2023-07-02T14:04:51.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:51.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795319+0000", 2023-07-02T14:04:51.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34828442599999998 2023-07-02T14:04:51.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143603+0000", 2023-07-02T14:04:51.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2438000000000001e-05 2023-07-02T14:04:51.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:51.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143665+0000", 2023-07-02T14:04:51.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3079999999999997e-05 2023-07-02T14:04:51.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:51.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143759+0000", 2023-07-02T14:04:51.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070633079999999999 2023-07-02T14:04:51.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:51.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150822+0000", 2023-07-02T14:04:51.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034430210000000001 2023-07-02T14:04:51.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:51.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154265+0000", 2023-07-02T14:04:51.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4249999999999998e-05 2023-07-02T14:04:51.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:51.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154309+0000", 2023-07-02T14:04:51.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2020000000000001e-05 2023-07-02T14:04:51.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:51.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154351+0000", 2023-07-02T14:04:51.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002 2023-07-02T14:04:51.686 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.686 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.687 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.687 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4054 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1900544~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:51.687 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:51.687 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.734059612999999, 2023-07-02T14:04:51.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002, 2023-07-02T14:04:51.687 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:51.688 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:51.688 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:51.688 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:51.688 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:51.688 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4054 2023-07-02T14:04:51.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.688 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:51.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:51.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:51.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:51.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:51.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:51.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:51.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:51.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:51.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018874860000000001 2023-07-02T14:04:51.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:51.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797172+0000", 2023-07-02T14:04:51.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3000000000000007e-06 2023-07-02T14:04:51.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:51.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797182+0000", 2023-07-02T14:04:51.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6260000000000003e-06 2023-07-02T14:04:51.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:51.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797186+0000", 2023-07-02T14:04:51.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9013e-05 2023-07-02T14:04:51.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797235+0000", 2023-07-02T14:04:51.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5990000000000008e-06 2023-07-02T14:04:51.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:51.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797245+0000", 2023-07-02T14:04:51.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34785621999999999 2023-07-02T14:04:51.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145101+0000", 2023-07-02T14:04:51.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7762000000000006e-05 2023-07-02T14:04:51.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:51.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145169+0000", 2023-07-02T14:04:51.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013469400000000001 2023-07-02T14:04:51.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:51.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145304+0000", 2023-07-02T14:04:51.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089538039999999992 2023-07-02T14:04:51.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:51.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154257+0000", 2023-07-02T14:04:51.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104218 2023-07-02T14:04:51.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:51.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154362+0000", 2023-07-02T14:04:51.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4168000000000001e-05 2023-07-02T14:04:51.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:51.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154406+0000", 2023-07-02T14:04:51.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3098000000000001e-05 2023-07-02T14:04:51.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:51.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154429+0000", 2023-07-02T14:04:51.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002 2023-07-02T14:04:51.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.698 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.699 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4056 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3670016~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:51.699 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:51.699 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.732153100000005, 2023-07-02T14:04:51.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997, 2023-07-02T14:04:51.699 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:51.699 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:51.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:51.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:51.700 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:51.700 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4056 2023-07-02T14:04:51.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.700 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:51.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:51.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:51.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:51.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:51.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:51.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:51.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:51.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797191+0000", 2023-07-02T14:04:51.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0438999999999997e-05 2023-07-02T14:04:51.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:51.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797232+0000", 2023-07-02T14:04:51.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6929999999999998e-06 2023-07-02T14:04:51.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:51.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797237+0000", 2023-07-02T14:04:51.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9739999999999998e-06 2023-07-02T14:04:51.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:51.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797240+0000", 2023-07-02T14:04:51.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0243000000000002e-05 2023-07-02T14:04:51.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797270+0000", 2023-07-02T14:04:51.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9889999999999997e-06 2023-07-02T14:04:51.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:51.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797276+0000", 2023-07-02T14:04:51.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35358319799999999 2023-07-02T14:04:51.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150859+0000", 2023-07-02T14:04:51.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1357e-05 2023-07-02T14:04:51.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:51.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150940+0000", 2023-07-02T14:04:51.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014903 2023-07-02T14:04:51.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:51.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151089+0000", 2023-07-02T14:04:51.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033512519999999999 2023-07-02T14:04:51.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:51.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154441+0000", 2023-07-02T14:04:51.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001050945 2023-07-02T14:04:51.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:51.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155492+0000", 2023-07-02T14:04:51.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2523e-05 2023-07-02T14:04:51.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:51.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155524+0000", 2023-07-02T14:04:51.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.057e-05 2023-07-02T14:04:51.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:51.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155555+0000", 2023-07-02T14:04:51.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997 2023-07-02T14:04:51.714 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.714 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.714 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.715 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4088 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3735552~458752] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:51.715 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:51.715 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.732100904999996, 2023-07-02T14:04:51.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998, 2023-07-02T14:04:51.715 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:51.715 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:51.715 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:51.716 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:51.716 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:51.716 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4088 2023-07-02T14:04:51.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.716 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:51.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:51.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:51.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:51.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:51.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:51.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:51.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:51.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:51.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00038503499999999999 2023-07-02T14:04:51.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:51.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797629+0000", 2023-07-02T14:04:51.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8739999999999995e-06 2023-07-02T14:04:51.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:51.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797637+0000", 2023-07-02T14:04:51.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7160000000000001e-06 2023-07-02T14:04:51.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:51.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797640+0000", 2023-07-02T14:04:51.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4342999999999998e-05 2023-07-02T14:04:51.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797695+0000", 2023-07-02T14:04:51.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0939999999999996e-06 2023-07-02T14:04:51.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:51.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797701+0000", 2023-07-02T14:04:51.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35408540799999999 2023-07-02T14:04:51.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151786+0000", 2023-07-02T14:04:51.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1953000000000006e-05 2023-07-02T14:04:51.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:51.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151868+0000", 2023-07-02T14:04:51.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015162800000000001 2023-07-02T14:04:51.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:51.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.152020+0000", 2023-07-02T14:04:51.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054561870000000004 2023-07-02T14:04:51.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:51.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.157476+0000", 2023-07-02T14:04:51.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059153199999999997 2023-07-02T14:04:51.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:51.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158068+0000", 2023-07-02T14:04:51.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2799e-05 2023-07-02T14:04:51.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:51.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158090+0000", 2023-07-02T14:04:51.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7235999999999999e-05 2023-07-02T14:04:51.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:51.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158108+0000", 2023-07-02T14:04:51.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998 2023-07-02T14:04:51.726 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.726 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.726 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.727 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4110 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:51.727 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:51.727 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.731699628000001, 2023-07-02T14:04:51.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864, 2023-07-02T14:04:51.727 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:51.727 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:51.728 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:51.728 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:51.728 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:51.728 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4110 2023-07-02T14:04:51.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.728 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:51.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:51.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:51.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:51.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:51.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:51.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:51.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:51.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:51.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9331e-05 2023-07-02T14:04:51.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:51.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797684+0000", 2023-07-02T14:04:51.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.147e-06 2023-07-02T14:04:51.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:51.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797689+0000", 2023-07-02T14:04:51.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5020000000000001e-06 2023-07-02T14:04:51.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:51.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797693+0000", 2023-07-02T14:04:51.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5254999999999999e-05 2023-07-02T14:04:51.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797748+0000", 2023-07-02T14:04:51.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5520000000000005e-06 2023-07-02T14:04:51.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:51.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797757+0000", 2023-07-02T14:04:51.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35674245700000001 2023-07-02T14:04:51.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154499+0000", 2023-07-02T14:04:51.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0090000000000001e-06 2023-07-02T14:04:51.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:51.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154506+0000", 2023-07-02T14:04:51.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864 2023-07-02T14:04:51.735 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.735 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.735 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.736 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:51.736 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:51.736 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.731647542000005, 2023-07-02T14:04:51.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998, 2023-07-02T14:04:51.736 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:51.736 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:51.736 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:51.737 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:51.737 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:51.737 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:04:51.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.737 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:51.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:51.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:51.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:51.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:51.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:51.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:51.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:51.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:51.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001714819 2023-07-02T14:04:51.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:51.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799412+0000", 2023-07-02T14:04:51.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7779999999999996e-06 2023-07-02T14:04:51.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:51.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799421+0000", 2023-07-02T14:04:51.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3510000000000002e-06 2023-07-02T14:04:51.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:51.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799424+0000", 2023-07-02T14:04:51.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0275999999999999e-05 2023-07-02T14:04:51.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799464+0000", 2023-07-02T14:04:51.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8389999999999996e-06 2023-07-02T14:04:51.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:51.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799473+0000", 2023-07-02T14:04:51.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35507638800000002 2023-07-02T14:04:51.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:51.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154550+0000", 2023-07-02T14:04:51.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6309999999999998e-06 2023-07-02T14:04:51.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:51.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:51.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:51.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154559+0000", 2023-07-02T14:04:51.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998 2023-07-02T14:04:51.743 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.744 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:51.744 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:51.744 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:51.745 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:04:51.746 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:51.744+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:51.846 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:51.845+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:51.947 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:51.946+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:51.989 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:51.989 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:51.989 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:51.989 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:52.013 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:04:52.055 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:52.054+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:52.150 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:52.148+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:52.254 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:52.249+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:52.287 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:52.287 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:52.287 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:52.287 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:52.288 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:52.307 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:04:52.350 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:52.349+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:52.400 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:04:52.457 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:52.451+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:52.551 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:52.550+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:52.566 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 2] out_osds: [5, 4] dead_osds: [3] live_osds: [0, 1, 2, 4, 5] 2023-07-02T14:04:52.566 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:04:52.566 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2023-07-02T14:04:52.566 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-07-02T14:04:52.566 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-07-02T14:04:52.567 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2023-07-02T14:04:52.577 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:52.577 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:52.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:52.577 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:52.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.578 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:236 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2424832~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:52.578 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:52.578 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 133.285354095, 2023-07-02T14:04:52.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003, 2023-07-02T14:04:52.578 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:52.578 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:52.578 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:52.579 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:52.579 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:52.579 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 236 2023-07-02T14:04:52.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.579 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:52.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:52.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:52.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:52.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:52.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287554+0000", 2023-07-02T14:04:52.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:52.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:52.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287553+0000", 2023-07-02T14:04:52.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0667e-05 2023-07-02T14:04:52.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:52.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287614+0000", 2023-07-02T14:04:52.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0454e-05 2023-07-02T14:04:52.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:52.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287624+0000", 2023-07-02T14:04:52.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8564999999999999e-05 2023-07-02T14:04:52.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287643+0000", 2023-07-02T14:04:52.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027977596 2023-07-02T14:04:52.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315621+0000", 2023-07-02T14:04:52.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000201728 2023-07-02T14:04:52.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:52.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.315822+0000", 2023-07-02T14:04:52.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018436599999999999 2023-07-02T14:04:52.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:52.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316007+0000", 2023-07-02T14:04:52.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042423440000000003 2023-07-02T14:04:52.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:52.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320249+0000", 2023-07-02T14:04:52.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001452633 2023-07-02T14:04:52.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:52.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321702+0000", 2023-07-02T14:04:52.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3616000000000003e-05 2023-07-02T14:04:52.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:52.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321755+0000", 2023-07-02T14:04:52.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840999999999999e-05 2023-07-02T14:04:52.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:52.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321793+0000", 2023-07-02T14:04:52.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034238979000000003 2023-07-02T14:04:52.588 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.588 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.588 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.588 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:251 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:52.588 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:52.589 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 133.28525855800001, 2023-07-02T14:04:52.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001, 2023-07-02T14:04:52.589 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:52.589 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:52.589 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:52.589 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:52.589 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:52.590 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 251 2023-07-02T14:04:52.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.590 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:52.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:52.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:52.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:52.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:52.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287650+0000", 2023-07-02T14:04:52.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:52.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:52.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287649+0000", 2023-07-02T14:04:52.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3232000000000001e-05 2023-07-02T14:04:52.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:52.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287702+0000", 2023-07-02T14:04:52.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0010000000000007e-06 2023-07-02T14:04:52.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:52.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287711+0000", 2023-07-02T14:04:52.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3569999999999997e-06 2023-07-02T14:04:52.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.287717+0000", 2023-07-02T14:04:52.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028686454 2023-07-02T14:04:52.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316403+0000", 2023-07-02T14:04:52.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7741000000000002e-05 2023-07-02T14:04:52.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:52.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316461+0000", 2023-07-02T14:04:52.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4313000000000004e-05 2023-07-02T14:04:52.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:52.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.316545+0000", 2023-07-02T14:04:52.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003734032 2023-07-02T14:04:52.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:52.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.320279+0000", 2023-07-02T14:04:52.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015237219999999999 2023-07-02T14:04:52.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:52.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321803+0000", 2023-07-02T14:04:52.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1927e-05 2023-07-02T14:04:52.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:52.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321845+0000", 2023-07-02T14:04:52.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5398999999999998e-05 2023-07-02T14:04:52.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:52.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:02:39.321880+0000", 2023-07-02T14:04:52.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.034230426000000001 2023-07-02T14:04:52.598 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.598 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.598 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.599 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:04:52.599 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:52.599 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 111.46523328400001, 2023-07-02T14:04:52.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:04:52.599 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:52.599 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:52.600 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:52.600 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:52.600 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:52.600 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:04:52.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.600 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:52.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:52.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:52.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:52.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:52.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:04:52.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:52.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:52.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:04:52.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:04:52.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:52.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:04:52.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:04:52.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:52.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:04:52.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:04:52.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:04:52.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:04:52.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:04:52.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:04:52.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:52.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:04:52.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:04:52.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:04:52.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:04:52.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:52.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:04:52.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:04:52.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:52.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:04:52.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:04:52.607 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.608 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.608 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.608 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:04:52.608 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:52.608 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 111.46517254600001, 2023-07-02T14:04:52.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:04:52.609 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:52.609 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:52.609 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:52.609 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:52.609 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:52.609 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:04:52.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.610 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:52.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:52.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:52.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:52.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:52.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:04:52.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:52.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:52.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:04:52.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:04:52.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:52.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:04:52.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:04:52.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:52.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:04:52.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:04:52.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:04:52.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:04:52.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:04:52.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:04:52.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:52.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:04:52.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:04:52.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:04:52.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:04:52.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:52.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:04:52.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:04:52.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:52.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:04:52.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:04:52.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:52.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:04:52.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:04:52.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:52.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:04:52.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:04:52.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:52.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:04:52.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:04:52.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:52.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:04:52.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:04:52.619 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.619 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.619 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.620 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:52.620 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:52.620 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 101.234718216, 2023-07-02T14:04:52.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:04:52.620 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:52.621 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:52.621 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:52.621 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:52.621 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:52.621 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:04:52.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.621 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:52.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:52.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:52.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:52.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:52.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:04:52.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:52.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:52.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:04:52.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:04:52.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:52.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:04:52.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:04:52.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:52.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:04:52.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:04:52.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:04:52.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:04:52.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:04:52.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:04:52.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:52.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:04:52.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:04:52.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:52.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:04:52.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:04:52.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:52.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:04:52.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:04:52.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:52.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:04:52.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:04:52.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:52.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:04:52.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:04:52.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:52.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:04:52.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:04:52.636 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.636 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.636 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.637 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2941 2.3 2:d3762e63:::rbd_data.125de1449605.00000000000000f8:head [write 65536~2031616] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:52.637 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:52.637 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 101.105615262, 2023-07-02T14:04:52.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002, 2023-07-02T14:04:52.637 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:52.637 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:52.638 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:52.638 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:52.638 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:52.638 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2941 2023-07-02T14:04:52.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.638 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:52.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:52.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:52.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:52.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:52.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467293+0000", 2023-07-02T14:04:52.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:52.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:52.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.467291+0000", 2023-07-02T14:04:52.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043022859999999998 2023-07-02T14:04:52.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:52.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471593+0000", 2023-07-02T14:04:52.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0999999999999999e-05 2023-07-02T14:04:52.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:52.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471614+0000", 2023-07-02T14:04:52.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0097e-05 2023-07-02T14:04:52.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.471625+0000", 2023-07-02T14:04:52.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00069808499999999998 2023-07-02T14:04:52.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472323+0000", 2023-07-02T14:04:52.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000135635 2023-07-02T14:04:52.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:52.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472458+0000", 2023-07-02T14:04:52.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223365 2023-07-02T14:04:52.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:52.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.472682+0000", 2023-07-02T14:04:52.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013761944999999999 2023-07-02T14:04:52.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:52.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.486444+0000", 2023-07-02T14:04:52.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016859747000000001 2023-07-02T14:04:52.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:52.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503303+0000", 2023-07-02T14:04:52.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6635000000000001e-05 2023-07-02T14:04:52.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:52.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503380+0000", 2023-07-02T14:04:52.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000161803 2023-07-02T14:04:52.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:52.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.503542+0000", 2023-07-02T14:04:52.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036248818000000002 2023-07-02T14:04:52.647 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.647 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.647 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.647 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2950 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 1507328~786432] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:52.647 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:52.648 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 101.097441954, 2023-07-02T14:04:52.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999, 2023-07-02T14:04:52.648 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:52.648 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:52.648 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:52.648 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:52.648 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:52.648 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2950 2023-07-02T14:04:52.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.649 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:52.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:52.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:52.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:52.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:52.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:52.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:52.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:52.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:04:52.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001147854 2023-07-02T14:04:52.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:52.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476614+0000", 2023-07-02T14:04:52.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.802e-05 2023-07-02T14:04:52.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:52.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476632+0000", 2023-07-02T14:04:52.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1860000000000003e-06 2023-07-02T14:04:52.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476639+0000", 2023-07-02T14:04:52.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061211240000000004 2023-07-02T14:04:52.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482760+0000", 2023-07-02T14:04:52.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8833999999999996e-05 2023-07-02T14:04:52.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:52.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482859+0000", 2023-07-02T14:04:52.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019270600000000001 2023-07-02T14:04:52.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:52.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.483051+0000", 2023-07-02T14:04:52.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060385630000000003 2023-07-02T14:04:52.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:52.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489090+0000", 2023-07-02T14:04:52.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024506918999999999 2023-07-02T14:04:52.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:52.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513597+0000", 2023-07-02T14:04:52.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9672999999999995e-05 2023-07-02T14:04:52.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:52.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513667+0000", 2023-07-02T14:04:52.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024329500000000001 2023-07-02T14:04:52.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:52.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513910+0000", 2023-07-02T14:04:52.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999 2023-07-02T14:04:52.657 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.657 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.657 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.658 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2951 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2686976~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:52.658 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:52.658 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 101.096261176, 2023-07-02T14:04:52.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999, 2023-07-02T14:04:52.658 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:52.658 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:52.659 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:52.659 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:52.659 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:52.659 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2951 2023-07-02T14:04:52.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.659 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:52.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:52.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:52.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:52.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:52.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:04:52.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:52.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:52.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476646+0000", 2023-07-02T14:04:52.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7639000000000003e-05 2023-07-02T14:04:52.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:52.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476684+0000", 2023-07-02T14:04:52.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3508000000000002e-05 2023-07-02T14:04:52.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:52.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476747+0000", 2023-07-02T14:04:52.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8499999999999994e-06 2023-07-02T14:04:52.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476755+0000", 2023-07-02T14:04:52.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078346160000000008 2023-07-02T14:04:52.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484590+0000", 2023-07-02T14:04:52.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7206000000000004e-05 2023-07-02T14:04:52.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:52.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484687+0000", 2023-07-02T14:04:52.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016243099999999999 2023-07-02T14:04:52.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:52.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484849+0000", 2023-07-02T14:04:52.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042482860000000004 2023-07-02T14:04:52.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:52.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489097+0000", 2023-07-02T14:04:52.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.02489504 2023-07-02T14:04:52.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:52.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513992+0000", 2023-07-02T14:04:52.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9972000000000001e-05 2023-07-02T14:04:52.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:52.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514052+0000", 2023-07-02T14:04:52.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149264 2023-07-02T14:04:52.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:52.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514202+0000", 2023-07-02T14:04:52.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999 2023-07-02T14:04:52.668 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.668 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.668 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.668 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2952 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2883584~12288] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:52.668 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:52.668 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 101.09614734100001, 2023-07-02T14:04:52.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651, 2023-07-02T14:04:52.669 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:52.669 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:52.669 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:52.669 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:52.669 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:52.669 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2952 2023-07-02T14:04:52.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.670 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:52.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:52.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:52.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:52.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:52.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:04:52.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:52.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:52.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476760+0000", 2023-07-02T14:04:52.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3664e-05 2023-07-02T14:04:52.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:52.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476774+0000", 2023-07-02T14:04:52.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.166e-06 2023-07-02T14:04:52.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:52.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476783+0000", 2023-07-02T14:04:52.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8720000000000001e-06 2023-07-02T14:04:52.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476788+0000", 2023-07-02T14:04:52.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087180529999999999 2023-07-02T14:04:52.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485506+0000", 2023-07-02T14:04:52.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5680999999999994e-05 2023-07-02T14:04:52.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:52.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485602+0000", 2023-07-02T14:04:52.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000222365 2023-07-02T14:04:52.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:52.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485824+0000", 2023-07-02T14:04:52.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003277379 2023-07-02T14:04:52.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:52.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489101+0000", 2023-07-02T14:04:52.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025151739999999999 2023-07-02T14:04:52.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:52.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514253+0000", 2023-07-02T14:04:52.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4749000000000002e-05 2023-07-02T14:04:52.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:52.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514288+0000", 2023-07-02T14:04:52.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130634 2023-07-02T14:04:52.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:52.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514419+0000", 2023-07-02T14:04:52.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651 2023-07-02T14:04:52.678 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.678 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.678 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.679 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9620 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 589824~65536] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:04:52.679 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:04:52.679 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.550703618, 2023-07-02T14:04:52.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001, 2023-07-02T14:04:52.679 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:52.679 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:52.680 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:52.680 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:52.680 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:52.680 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9620 2023-07-02T14:04:52.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.680 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:52.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:52.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:04:52.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:52.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:52.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:04:52.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:52.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:52.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022203+0000", 2023-07-02T14:04:52.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0346000000000001e-05 2023-07-02T14:04:52.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:52.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022263+0000", 2023-07-02T14:04:52.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4657999999999999e-05 2023-07-02T14:04:52.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:52.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022278+0000", 2023-07-02T14:04:52.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:04:52.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022285+0000", 2023-07-02T14:04:52.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0096000000000003e-05 2023-07-02T14:04:52.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022335+0000", 2023-07-02T14:04:52.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2334000000000001e-05 2023-07-02T14:04:52.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:52.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022347+0000", 2023-07-02T14:04:52.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095653701999999993 2023-07-02T14:04:52.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118001+0000", 2023-07-02T14:04:52.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000185797 2023-07-02T14:04:52.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:52.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118187+0000", 2023-07-02T14:04:52.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026397900000000001 2023-07-02T14:04:52.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:52.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118451+0000", 2023-07-02T14:04:52.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051832800000000002 2023-07-02T14:04:52.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:52.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.123634+0000", 2023-07-02T14:04:52.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001227332 2023-07-02T14:04:52.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:52.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124861+0000", 2023-07-02T14:04:52.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9493999999999998e-05 2023-07-02T14:04:52.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:52.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124901+0000", 2023-07-02T14:04:52.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0462000000000001e-05 2023-07-02T14:04:52.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:52.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124931+0000", 2023-07-02T14:04:52.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001 2023-07-02T14:04:52.690 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.690 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.690 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.691 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9621 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 655360~1966080] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:04:52.691 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:04:52.691 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.550516236, 2023-07-02T14:04:52.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999, 2023-07-02T14:04:52.691 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:52.691 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:52.691 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:52.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:52.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:52.692 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9621 2023-07-02T14:04:52.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.692 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:52.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:52.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:04:52.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:52.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:52.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:04:52.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:52.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:52.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022391+0000", 2023-07-02T14:04:52.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032053939999999999 2023-07-02T14:04:52.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:52.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025597+0000", 2023-07-02T14:04:52.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4307999999999999e-05 2023-07-02T14:04:52.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:52.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025611+0000", 2023-07-02T14:04:52.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.032e-06 2023-07-02T14:04:52.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025617+0000", 2023-07-02T14:04:52.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8977999999999998e-05 2023-07-02T14:04:52.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025656+0000", 2023-07-02T14:04:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:04:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025663+0000", 2023-07-02T14:04:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095220739999999998 2023-07-02T14:04:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120884+0000", 2023-07-02T14:04:52.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2525000000000001e-05 2023-07-02T14:04:52.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:52.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120956+0000", 2023-07-02T14:04:52.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016232400000000001 2023-07-02T14:04:52.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:52.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.121119+0000", 2023-07-02T14:04:52.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044007380000000004 2023-07-02T14:04:52.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:52.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.125519+0000", 2023-07-02T14:04:52.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000797209 2023-07-02T14:04:52.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:52.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126317+0000", 2023-07-02T14:04:52.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1228000000000004e-05 2023-07-02T14:04:52.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:52.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126378+0000", 2023-07-02T14:04:52.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978000000000003e-05 2023-07-02T14:04:52.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:52.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126430+0000", 2023-07-02T14:04:52.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999 2023-07-02T14:04:52.702 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.702 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.702 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.703 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:11434 2.1 2:8269d003:::rbd_data.125de1449605.00000000000000ce:head [write 65536~4128768] snapc 1a=[1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e107)", 2023-07-02T14:04:52.703 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:04:52.703 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.655077967, 2023-07-02T14:04:52.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036409966000000002, 2023-07-02T14:04:52.703 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:52.703 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:52.703 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:52.703 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:52.704 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:52.704 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11434 2023-07-02T14:04:52.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.704 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:52.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:52.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:04:52.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:52.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:52.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:04:52.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:04:52.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:52.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917828+0000", 2023-07-02T14:04:52.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052044650000000001 2023-07-02T14:04:52.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:52.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923033+0000", 2023-07-02T14:04:52.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0097000000000001e-05 2023-07-02T14:04:52.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:52.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923053+0000", 2023-07-02T14:04:52.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.49e-06 2023-07-02T14:04:52.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923061+0000", 2023-07-02T14:04:52.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0090999999999998e-05 2023-07-02T14:04:52.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923121+0000", 2023-07-02T14:04:52.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000114095 2023-07-02T14:04:52.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:52.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923236+0000", 2023-07-02T14:04:52.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020094199999999999 2023-07-02T14:04:52.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:52.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923436+0000", 2023-07-02T14:04:52.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088693520000000005 2023-07-02T14:04:52.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:52.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.932306+0000", 2023-07-02T14:04:52.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021874112000000001 2023-07-02T14:04:52.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:52.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954180+0000", 2023-07-02T14:04:52.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4433e-05 2023-07-02T14:04:52.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:52.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954214+0000", 2023-07-02T14:04:52.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5884000000000002e-05 2023-07-02T14:04:52.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:52.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954240+0000", 2023-07-02T14:04:52.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036409966000000002 2023-07-02T14:04:52.712 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.712 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.713 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.713 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:189 2.3 2:c9621172:::rbd_data.125de1449605.00000000000000c0:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:04:52.713 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:04:52.713 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.400228943999998, 2023-07-02T14:04:52.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001, 2023-07-02T14:04:52.713 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:52.714 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:52.714 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:52.714 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:04:52.714 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:04:52.714 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:04:52.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.714 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:52.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:52.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:04:52.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:52.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:52.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:04:52.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:52.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:52.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:04:52.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2023-07-02T14:04:52.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:52.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172682+0000", 2023-07-02T14:04:52.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0159999999999997e-06 2023-07-02T14:04:52.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:52.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172688+0000", 2023-07-02T14:04:52.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0010000000000003e-06 2023-07-02T14:04:52.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172692+0000", 2023-07-02T14:04:52.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026444333 2023-07-02T14:04:52.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199136+0000", 2023-07-02T14:04:52.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2230000000000004e-05 2023-07-02T14:04:52.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:52.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199218+0000", 2023-07-02T14:04:52.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012043595000000001 2023-07-02T14:04:52.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:52.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211262+0000", 2023-07-02T14:04:52.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001 2023-07-02T14:04:52.720 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.720 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.721 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.721 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:199 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:04:52.721 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:04:52.721 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.400185401000002, 2023-07-02T14:04:52.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002, 2023-07-02T14:04:52.721 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:52.722 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:52.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:52.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:04:52.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:04:52.722 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 199 2023-07-02T14:04:52.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.722 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:52.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:52.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:04:52.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:52.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:52.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:04:52.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:52.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:52.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172722+0000", 2023-07-02T14:04:52.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1250000000000001e-06 2023-07-02T14:04:52.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:52.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172725+0000", 2023-07-02T14:04:52.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5670000000000002e-06 2023-07-02T14:04:52.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:52.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172731+0000", 2023-07-02T14:04:52.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9310000000000001e-06 2023-07-02T14:04:52.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172735+0000", 2023-07-02T14:04:52.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030699852999999999 2023-07-02T14:04:52.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203435+0000", 2023-07-02T14:04:52.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6827e-05 2023-07-02T14:04:52.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:52.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203532+0000", 2023-07-02T14:04:52.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076955230000000001 2023-07-02T14:04:52.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:52.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211227+0000", 2023-07-02T14:04:52.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002 2023-07-02T14:04:52.728 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.728 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.729 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.729 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:213 2.1 2:8fbcc55d:::rbd_data.125de1449605.00000000000000d5:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e110)", 2023-07-02T14:04:52.729 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.241083+0000", 2023-07-02T14:04:52.730 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.331824815000001, 2023-07-02T14:04:52.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036164654999999997, 2023-07-02T14:04:52.730 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:52.730 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:52.730 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:52.730 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:04:52.730 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:04:52.731 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 213 2023-07-02T14:04:52.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.731 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:52.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:52.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.241083+0000", 2023-07-02T14:04:52.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:52.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:52.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.241083+0000", 2023-07-02T14:04:52.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:52.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:52.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.241083+0000", 2023-07-02T14:04:52.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.525e-06 2023-07-02T14:04:52.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:52.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.241086+0000", 2023-07-02T14:04:52.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0620000000000003e-06 2023-07-02T14:04:52.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:52.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.241093+0000", 2023-07-02T14:04:52.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.799e-06 2023-07-02T14:04:52.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.241096+0000", 2023-07-02T14:04:52.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030109975000000001 2023-07-02T14:04:52.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.271206+0000", 2023-07-02T14:04:52.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134172 2023-07-02T14:04:52.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:52.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.271341+0000", 2023-07-02T14:04:52.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059075309999999997 2023-07-02T14:04:52.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:52.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.277248+0000", 2023-07-02T14:04:52.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036164654999999997 2023-07-02T14:04:52.737 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.737 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.737 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.738 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:280 2.1 2:94df0216:::rbd_data.125de1449605.0000000000000117:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:04:52.738 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555448+0000", 2023-07-02T14:04:52.738 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.017459786, 2023-07-02T14:04:52.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989515, 2023-07-02T14:04:52.738 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:52.738 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:52.738 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:52.738 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:04:52.739 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:04:52.739 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 280 2023-07-02T14:04:52.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.739 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:52.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:52.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555448+0000", 2023-07-02T14:04:52.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:52.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:52.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555448+0000", 2023-07-02T14:04:52.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:52.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:52.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555448+0000", 2023-07-02T14:04:52.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1400000000000002e-06 2023-07-02T14:04:52.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:52.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555452+0000", 2023-07-02T14:04:52.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8199999999999999e-06 2023-07-02T14:04:52.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:52.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555459+0000", 2023-07-02T14:04:52.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7520000000000004e-06 2023-07-02T14:04:52.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555463+0000", 2023-07-02T14:04:52.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027814050999999999 2023-07-02T14:04:52.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.583277+0000", 2023-07-02T14:04:52.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192244 2023-07-02T14:04:52.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:52.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.583470+0000", 2023-07-02T14:04:52.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007968269 2023-07-02T14:04:52.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:52.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591438+0000", 2023-07-02T14:04:52.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989515 2023-07-02T14:04:52.745 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.745 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.745 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.745 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:281 2.1 2:901d5a95:::rbd_data.125de1449605.0000000000000118:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:04:52.746 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:04:52.746 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.017440125, 2023-07-02T14:04:52.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002, 2023-07-02T14:04:52.746 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:52.746 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:52.746 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:52.746 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:04:52.746 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:04:52.747 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:04:52.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.747 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:52.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:52.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:04:52.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:52.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:52.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:04:52.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:52.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:52.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:04:52.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0469999999999998e-06 2023-07-02T14:04:52.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:52.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555471+0000", 2023-07-02T14:04:52.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8860000000000001e-06 2023-07-02T14:04:52.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:52.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555476+0000", 2023-07-02T14:04:52.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3160000000000002e-06 2023-07-02T14:04:52.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555483+0000", 2023-07-02T14:04:52.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989525000000001 2023-07-02T14:04:52.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591472+0000", 2023-07-02T14:04:52.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3708999999999996e-05 2023-07-02T14:04:52.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:52.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591536+0000", 2023-07-02T14:04:52.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055109720000000003 2023-07-02T14:04:52.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:52.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.597047+0000", 2023-07-02T14:04:52.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002 2023-07-02T14:04:52.753 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.753 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.753 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.753 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5874.0:238 2.7 2:e57d3f55:::rbd_data.125de1449605.0000000000000134:1f [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e122)", 2023-07-02T14:04:52.753 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:47.557344+0000", 2023-07-02T14:04:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.0155643090000002, 2023-07-02T14:04:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035326628999999998, 2023-07-02T14:04:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5874", 2023-07-02T14:04:52.754 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/57475297", 2023-07-02T14:04:52.755 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 238 2023-07-02T14:04:52.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.755 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:52.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:52.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:47.557344+0000", 2023-07-02T14:04:52.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:52.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:52.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:47.557344+0000", 2023-07-02T14:04:52.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:52.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:52.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:47.557344+0000", 2023-07-02T14:04:52.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6359999999999998e-06 2023-07-02T14:04:52.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:52.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:47.557346+0000", 2023-07-02T14:04:52.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0000000000000002e-06 2023-07-02T14:04:52.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:52.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:47.557352+0000", 2023-07-02T14:04:52.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3249999999999998e-06 2023-07-02T14:04:52.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:47.557357+0000", 2023-07-02T14:04:52.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.031165531999999999 2023-07-02T14:04:52.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:47.588523+0000", 2023-07-02T14:04:52.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000125802 2023-07-02T14:04:52.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:52.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:47.588649+0000", 2023-07-02T14:04:52.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040217639999999997 2023-07-02T14:04:52.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:52.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:47.592671+0000", 2023-07-02T14:04:52.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035326628999999998 2023-07-02T14:04:52.761 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.761 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.761 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.761 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12534 2.1 2:8b20f686:::rbd_data.125de1449605.0000000000000056:head [write 3862528~4096] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:04:52.762 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.205652+0000", 2023-07-02T14:04:52.762 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.36725633600000002, 2023-07-02T14:04:52.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036222888000000002, 2023-07-02T14:04:52.762 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:52.762 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:52.762 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:52.762 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:52.762 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:52.763 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12534 2023-07-02T14:04:52.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.763 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:52.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:52.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205652+0000", 2023-07-02T14:04:52.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:52.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:52.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205652+0000", 2023-07-02T14:04:52.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:04:52.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:52.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205650+0000", 2023-07-02T14:04:52.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9589000000000002e-05 2023-07-02T14:04:52.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:52.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205669+0000", 2023-07-02T14:04:52.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4072e-05 2023-07-02T14:04:52.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:52.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205683+0000", 2023-07-02T14:04:52.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9360000000000002e-06 2023-07-02T14:04:52.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205690+0000", 2023-07-02T14:04:52.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015937382999999999 2023-07-02T14:04:52.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.221628+0000", 2023-07-02T14:04:52.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000118441 2023-07-02T14:04:52.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:52.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.221746+0000", 2023-07-02T14:04:52.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028993399999999999 2023-07-02T14:04:52.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:52.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.222036+0000", 2023-07-02T14:04:52.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046943469999999998 2023-07-02T14:04:52.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:52.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.226730+0000", 2023-07-02T14:04:52.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015041226 2023-07-02T14:04:52.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:52.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.241772+0000", 2023-07-02T14:04:52.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7695000000000002e-05 2023-07-02T14:04:52.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:52.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.241829+0000", 2023-07-02T14:04:52.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5509000000000001e-05 2023-07-02T14:04:52.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:52.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.241875+0000", 2023-07-02T14:04:52.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036222888000000002 2023-07-02T14:04:52.771 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.771 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.771 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.772 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12536 2.1 2:8b20f686:::rbd_data.125de1449605.0000000000000056:head [write 3670016~36864] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:04:52.772 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.205749+0000", 2023-07-02T14:04:52.772 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.36715916100000001, 2023-07-02T14:04:52.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036263036999999998, 2023-07-02T14:04:52.772 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:52.772 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:52.773 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:52.773 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:52.773 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:52.773 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12536 2023-07-02T14:04:52.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.773 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:52.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:52.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205749+0000", 2023-07-02T14:04:52.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:52.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:52.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205749+0000", 2023-07-02T14:04:52.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:52.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:52.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205748+0000", 2023-07-02T14:04:52.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1784999999999999e-05 2023-07-02T14:04:52.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:52.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205770+0000", 2023-07-02T14:04:52.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0659999999999993e-06 2023-07-02T14:04:52.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:52.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205779+0000", 2023-07-02T14:04:52.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.622e-06 2023-07-02T14:04:52.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205784+0000", 2023-07-02T14:04:52.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018333009000000001 2023-07-02T14:04:52.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:52.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.224117+0000", 2023-07-02T14:04:52.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018818400000000001 2023-07-02T14:04:52.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:52.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.224305+0000", 2023-07-02T14:04:52.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00032545700000000002 2023-07-02T14:04:52.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:52.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.224631+0000", 2023-07-02T14:04:52.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049349700000000003 2023-07-02T14:04:52.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:52.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.229566+0000", 2023-07-02T14:04:52.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012353153 2023-07-02T14:04:52.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:52.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.241919+0000", 2023-07-02T14:04:52.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2751999999999999e-05 2023-07-02T14:04:52.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:52.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.241971+0000", 2023-07-02T14:04:52.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0632e-05 2023-07-02T14:04:52.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:52.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:52.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:52.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.242012+0000", 2023-07-02T14:04:52.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036263036999999998 2023-07-02T14:04:52.782 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.782 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.782 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.782 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:52.782 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:52.782 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:52.784 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:04:52.784 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:52.785 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "" 2023-07-02T14:04:52.785 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:52.785 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:52.782+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:52.884 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:52.883+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:52.925 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:52.925 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:52.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:52.926 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:13060 2.7 e125/121 2:f3c0d8e8:::rbd_data.125de1449605.0000000000000093:head v 125'1999, mlcod=125'1996)", 2023-07-02T14:04:52.926 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:52.921229+0000", 2023-07-02T14:04:52.926 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0044573169999999997, 2023-07-02T14:04:52.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044757640000000001, 2023-07-02T14:04:52.926 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:52.926 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:04:52.927 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:52.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:52.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:52.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:52.921229+0000", 2023-07-02T14:04:52.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:52.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:52.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:52.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:52.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:52.921229+0000", 2023-07-02T14:04:52.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999938 2023-07-02T14:04:52.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:52.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:52.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:52.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:52.921222+0000", 2023-07-02T14:04:52.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00341246 2023-07-02T14:04:52.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:52.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:52.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:52.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:52.924635+0000", 2023-07-02T14:04:52.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.147e-05 2023-07-02T14:04:52.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:52.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:52.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:52.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:52.924646+0000", 2023-07-02T14:04:52.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6600000000000003e-06 2023-07-02T14:04:52.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:52.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:52.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:52.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:52.924651+0000", 2023-07-02T14:04:52.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1389e-05 2023-07-02T14:04:52.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:52.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:52.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:52.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:52.924692+0000", 2023-07-02T14:04:52.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4925e-05 2023-07-02T14:04:52.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:52.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:52.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:52.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:52.924707+0000", 2023-07-02T14:04:52.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0034785340000000001 2023-07-02T14:04:52.932 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:52.932 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:52.932 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:52.932 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:52.932 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2023-07-02T14:04:52.932 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 1 2023-07-02T14:04:52.933 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:52.939 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:04:53.022 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:53.021+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:53.086 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:53.085+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:53.124 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:53.124 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:53.124 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:04:53.124 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:04:53.124 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:53.138 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:04:53.186 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:53.185+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:53.287 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:53.287+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:53.312 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:53.313 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:04:53.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:04:53.313 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:53.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.313 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:04:53.314 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:53.314 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 73.413144586000001, 2023-07-02T14:04:53.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:04:53.314 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:53.314 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:53.314 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:53.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:53.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:53.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:53.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:53.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:04:53.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:53.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:53.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:04:53.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:04:53.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:53.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:04:53.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:04:53.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:53.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:04:53.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:04:53.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:53.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:04:53.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:04:53.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:53.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:04:53.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:04:53.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:53.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:04:53.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:04:53.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:53.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:04:53.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:04:53.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:53.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:04:53.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:04:53.321 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.322 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.322 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.322 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:04:53.322 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:53.322 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 73.409058868000002, 2023-07-02T14:04:53.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:04:53.323 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:53.323 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:53.323 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:53.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:53.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:53.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:53.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.324 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:53.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:04:53.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:53.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:53.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:04:53.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:04:53.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:53.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:04:53.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:04:53.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:53.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:04:53.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:04:53.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:53.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:04:53.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:04:53.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:53.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:04:53.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:04:53.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:53.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:04:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:04:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:04:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:04:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:53.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:04:53.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:04:53.330 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.330 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.330 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.331 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:04:53.331 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:53.331 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 73.405194086999998, 2023-07-02T14:04:53.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:04:53.331 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:53.331 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:53.332 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:53.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:53.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:53.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:53.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:53.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:04:53.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:53.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:53.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:04:53.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:04:53.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:53.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:04:53.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:04:53.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:53.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:04:53.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:04:53.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:53.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:04:53.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:04:53.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:53.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:04:53.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:04:53.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:53.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:04:53.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:04:53.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:53.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:04:53.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:04:53.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:53.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:04:53.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:04:53.339 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.339 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.339 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.339 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:04:53.339 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:53.340 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 73.401669826000003, 2023-07-02T14:04:53.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:04:53.340 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:53.340 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:53.340 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:53.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:53.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:53.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:53.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:53.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:04:53.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:53.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:53.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:04:53.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:04:53.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:53.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:04:53.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:04:53.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:53.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:04:53.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:04:53.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:53.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:04:53.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:04:53.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:53.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:04:53.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:04:53.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:53.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:04:53.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:04:53.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:53.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:04:53.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:04:53.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:53.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:04:53.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:04:53.347 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.347 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.347 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.348 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:04:53.348 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:53.348 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 73.365965027000001, 2023-07-02T14:04:53.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:04:53.348 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:53.349 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:53.349 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:53.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:53.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:53.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:53.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:53.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:04:53.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:53.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:53.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:04:53.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:04:53.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:53.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:04:53.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:04:53.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:53.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:04:53.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:04:53.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:53.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:04:53.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:04:53.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:53.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:04:53.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:04:53.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:53.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:04:53.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:04:53.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:53.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:04:53.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:04:53.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:53.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:04:53.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:04:53.356 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.356 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.356 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.356 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:04:53.357 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:53.357 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 73.361307668999999, 2023-07-02T14:04:53.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:04:53.357 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:53.357 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:53.357 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:53.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:53.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:53.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:53.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:53.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:04:53.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:53.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:53.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:04:53.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:04:53.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:53.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:04:53.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:04:53.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:53.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:04:53.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:04:53.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:53.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:04:53.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:04:53.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:53.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:04:53.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:04:53.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:53.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:04:53.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:04:53.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:53.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:04:53.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:04:53.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:53.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:04:53.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:04:53.365 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.365 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.365 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.366 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:04:53.366 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:53.366 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 73.360367805999999, 2023-07-02T14:04:53.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:04:53.366 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:53.366 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:53.366 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:53.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:53.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:53.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:53.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:53.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:04:53.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:53.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:53.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:04:53.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:04:53.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:53.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:04:53.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:04:53.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:53.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:04:53.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:04:53.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:53.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:04:53.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:04:53.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:53.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:04:53.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:04:53.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:53.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:04:53.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:04:53.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:53.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:04:53.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:04:53.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:53.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:04:53.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:04:53.374 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.374 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.374 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.374 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:04:53.374 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:53.375 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 73.358969927999993, 2023-07-02T14:04:53.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:04:53.375 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:53.375 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:53.375 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:53.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:53.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:53.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:53.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:53.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:04:53.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:53.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:53.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:04:53.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:04:53.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:53.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:04:53.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:04:53.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:53.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:04:53.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:04:53.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:53.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:04:53.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:04:53.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:53.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:04:53.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:04:53.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:53.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:04:53.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:04:53.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:53.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:04:53.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:04:53.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:53.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:04:53.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:04:53.382 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.382 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.382 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.383 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:04:53.383 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:53.383 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 51.948779432000002, 2023-07-02T14:04:53.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:04:53.383 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:53.383 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:53.384 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:53.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:53.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:53.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:53.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:04:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:53.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:04:53.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:04:53.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:53.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:04:53.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:04:53.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:53.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:04:53.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:04:53.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:53.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:04:53.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:04:53.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:53.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:04:53.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:04:53.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:53.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:04:53.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:04:53.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:53.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:04:53.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:04:53.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:53.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:04:53.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:04:53.391 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.391 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.391 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.392 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:53.392 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:53.392 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.582429748000003, 2023-07-02T14:04:53.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:04:53.392 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:53.392 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:53.392 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:53.393 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:53.393 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:53.393 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:04:53.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.393 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:53.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:53.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:53.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:53.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:53.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:04:53.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:53.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:53.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:04:53.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:04:53.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:53.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:04:53.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:04:53.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:53.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:04:53.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:04:53.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:53.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:04:53.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:04:53.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:53.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:04:53.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:04:53.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:53.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:04:53.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:04:53.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:53.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:04:53.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:04:53.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:53.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:04:53.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:04:53.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:53.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:04:53.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:04:53.401 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.401 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.401 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.401 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7706 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 0~53248] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:53.402 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:53.402 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.578503001000001, 2023-07-02T14:04:53.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019507096000000002, 2023-07-02T14:04:53.402 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:53.402 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:53.402 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:53.402 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:53.403 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:53.403 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7706 2023-07-02T14:04:53.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.403 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:53.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:53.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:53.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:53.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.404 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:53.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732256+0000", 2023-07-02T14:04:53.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:53.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:53.405 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732254+0000", 2023-07-02T14:04:53.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4764999999999997e-05 2023-07-02T14:04:53.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:53.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732289+0000", 2023-07-02T14:04:53.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0166999999999999e-05 2023-07-02T14:04:53.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:53.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732299+0000", 2023-07-02T14:04:53.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5310000000000001e-06 2023-07-02T14:04:53.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:53.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732305+0000", 2023-07-02T14:04:53.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060619599999999999 2023-07-02T14:04:53.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:53.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738367+0000", 2023-07-02T14:04:53.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011864 2023-07-02T14:04:53.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:53.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738486+0000", 2023-07-02T14:04:53.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013098291 2023-07-02T14:04:53.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:53.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751584+0000", 2023-07-02T14:04:53.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7800000000000002e-05 2023-07-02T14:04:53.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:53.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751642+0000", 2023-07-02T14:04:53.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000121198 2023-07-02T14:04:53.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:53.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751763+0000", 2023-07-02T14:04:53.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019507096000000002 2023-07-02T14:04:53.411 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.411 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.411 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.411 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7707 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 3145728~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:53.411 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:53.412 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.578446280999998, 2023-07-02T14:04:53.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999, 2023-07-02T14:04:53.412 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:53.412 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:53.412 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:53.412 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:53.412 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:53.413 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7707 2023-07-02T14:04:53.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.413 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:53.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:53.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:53.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:53.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:53.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:53.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:53.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:53.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:04:53.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5017e-05 2023-07-02T14:04:53.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:53.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732347+0000", 2023-07-02T14:04:53.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5821e-05 2023-07-02T14:04:53.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:53.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732363+0000", 2023-07-02T14:04:53.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1270000000000002e-06 2023-07-02T14:04:53.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:53.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732368+0000", 2023-07-02T14:04:53.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008444666 2023-07-02T14:04:53.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:53.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740812+0000", 2023-07-02T14:04:53.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011266599999999999 2023-07-02T14:04:53.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:53.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740925+0000", 2023-07-02T14:04:53.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010848808999999999 2023-07-02T14:04:53.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:53.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751774+0000", 2023-07-02T14:04:53.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8421999999999997e-05 2023-07-02T14:04:53.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:53.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751832+0000", 2023-07-02T14:04:53.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107453 2023-07-02T14:04:53.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:53.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751940+0000", 2023-07-02T14:04:53.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999 2023-07-02T14:04:53.421 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.421 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.421 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.421 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:53.421 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:53.422 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.578382404000003, 2023-07-02T14:04:53.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:04:53.422 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:53.422 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:53.422 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:53.422 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:53.422 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:53.422 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:04:53.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.423 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:53.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:53.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:53.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:53.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:53.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:53.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:53.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:53.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:04:53.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:04:53.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:53.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:04:53.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:04:53.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:53.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:04:53.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:04:53.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:53.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:04:53.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:04:53.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:53.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:04:53.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:04:53.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:53.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:04:53.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:04:53.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:53.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:04:53.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:04:53.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:53.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:04:53.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:04:53.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:53.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:04:53.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:04:53.430 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.431 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.431 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.431 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:53.431 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:53.431 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.573818578999997, 2023-07-02T14:04:53.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:04:53.432 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:53.432 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:53.432 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:53.433 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:53.433 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:53.433 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:04:53.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.433 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:53.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:53.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:53.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:53.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:53.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:04:53.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:53.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:53.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:04:53.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:04:53.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:53.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:04:53.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:04:53.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:53.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:04:53.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:04:53.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:53.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:04:53.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:04:53.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:53.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:04:53.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:04:53.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:53.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:04:53.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:04:53.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:53.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:04:53.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:04:53.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:53.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:04:53.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:04:53.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:53.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:04:53.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:04:53.441 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.441 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.441 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.442 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:53.442 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:53.442 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.573768713, 2023-07-02T14:04:53.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:04:53.442 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:53.442 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:53.443 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:53.443 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:53.443 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:53.443 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:04:53.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.444 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:53.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:53.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:53.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:53.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:53.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:04:53.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:53.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:53.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:04:53.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:04:53.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:53.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:04:53.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:04:53.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:53.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:04:53.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:04:53.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:53.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:04:53.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:04:53.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:53.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:04:53.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:04:53.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:53.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:04:53.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:04:53.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:53.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:04:53.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:04:53.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:53.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:04:53.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:04:53.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:53.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:04:53.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:04:53.452 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.452 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.452 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.453 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:53.453 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:53.453 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.572729719999998, 2023-07-02T14:04:53.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:04:53.453 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:53.453 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:53.454 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:53.454 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:53.454 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:53.454 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:04:53.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.454 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:53.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:53.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:53.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:53.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:53.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:04:53.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:53.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:53.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:04:53.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:04:53.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:04:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:04:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:04:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:04:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:04:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:04:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:04:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:04:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:04:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:04:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:04:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:04:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:04:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:04:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:04:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:04:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.463 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.463 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.463 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7993 2.7 2:fe72db53:::rbd_data.125de1449605.0000000000000072:head [write 65536~3276800] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:04:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.385771671999997, 2023-07-02T14:04:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919, 2023-07-02T14:04:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7993 2023-07-02T14:04:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:04:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924986+0000", 2023-07-02T14:04:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049542070000000004 2023-07-02T14:04:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929940+0000", 2023-07-02T14:04:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9740000000000001e-05 2023-07-02T14:04:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929960+0000", 2023-07-02T14:04:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.311e-05 2023-07-02T14:04:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:53.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929973+0000", 2023-07-02T14:04:53.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9629000000000002e-05 2023-07-02T14:04:53.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930023+0000", 2023-07-02T14:04:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012751799999999999 2023-07-02T14:04:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930150+0000", 2023-07-02T14:04:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016449042000000001 2023-07-02T14:04:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946599+0000", 2023-07-02T14:04:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3241e-05 2023-07-02T14:04:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946663+0000", 2023-07-02T14:04:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000102296 2023-07-02T14:04:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946765+0000", 2023-07-02T14:04:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919 2023-07-02T14:04:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11782 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 196608~2162688] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:04:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:04:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.9012517500000001, 2023-07-02T14:04:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365, 2023-07-02T14:04:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:53.475 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11782 2023-07-02T14:04:53.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.475 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:53.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:53.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:04:53.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:04:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409506+0000", 2023-07-02T14:04:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020172269999999999 2023-07-02T14:04:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411523+0000", 2023-07-02T14:04:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8901000000000001e-05 2023-07-02T14:04:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411542+0000", 2023-07-02T14:04:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5037e-05 2023-07-02T14:04:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411557+0000", 2023-07-02T14:04:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001931213 2023-07-02T14:04:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413489+0000", 2023-07-02T14:04:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000149207 2023-07-02T14:04:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413638+0000", 2023-07-02T14:04:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024752800000000001 2023-07-02T14:04:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413885+0000", 2023-07-02T14:04:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048516000000000002 2023-07-02T14:04:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.418737+0000", 2023-07-02T14:04:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010319259000000001 2023-07-02T14:04:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429056+0000", 2023-07-02T14:04:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2518000000000002e-05 2023-07-02T14:04:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429099+0000", 2023-07-02T14:04:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015275 2023-07-02T14:04:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:53.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429251+0000", 2023-07-02T14:04:53.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365 2023-07-02T14:04:53.484 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.484 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.484 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.485 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPushReply(2.7 119/117 [PushReplyOp(2:eb862bc0:::rbd_data.125de1449605.0000000000000040:13)])", 2023-07-02T14:04:53.485 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:04:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.9000884750000004, 2023-07-02T14:04:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999, 2023-07-02T14:04:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "reached pg", 2023-07-02T14:04:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:53.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:04:53.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:53.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:53.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:04:53.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:53.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:04:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7670000000000001e-06 2023-07-02T14:04:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410674+0000", 2023-07-02T14:04:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3973e-05 2023-07-02T14:04:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410688+0000", 2023-07-02T14:04:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0157e-05 2023-07-02T14:04:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410708+0000", 2023-07-02T14:04:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018578779 2023-07-02T14:04:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429287+0000", 2023-07-02T14:04:53.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013086549999999999 2023-07-02T14:04:53.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:53.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.430595+0000", 2023-07-02T14:04:53.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999 2023-07-02T14:04:53.492 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11861 2.7 2:e57d3f55:::rbd_data.125de1449605.0000000000000134:head [write 0~65536] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:04:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:04:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7659730979999999, 2023-07-02T14:04:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723, 2023-07-02T14:04:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11861 2023-07-02T14:04:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:04:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:53.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:04:53.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:53.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:53.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544785+0000", 2023-07-02T14:04:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010563700000000001 2023-07-02T14:04:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544890+0000", 2023-07-02T14:04:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8530000000000004e-06 2023-07-02T14:04:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544900+0000", 2023-07-02T14:04:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.665e-06 2023-07-02T14:04:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544906+0000", 2023-07-02T14:04:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000272575 2023-07-02T14:04:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545178+0000", 2023-07-02T14:04:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000111362 2023-07-02T14:04:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:53.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545290+0000", 2023-07-02T14:04:53.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016181100000000001 2023-07-02T14:04:53.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545452+0000", 2023-07-02T14:04:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044500750000000004 2023-07-02T14:04:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.549902+0000", 2023-07-02T14:04:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014850024 2023-07-02T14:04:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564752+0000", 2023-07-02T14:04:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6272000000000003e-05 2023-07-02T14:04:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564848+0000", 2023-07-02T14:04:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000162428 2023-07-02T14:04:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.565010+0000", 2023-07-02T14:04:53.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723 2023-07-02T14:04:53.504 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.504 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.504 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.504 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:53.504 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:53.504 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:53.506 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:04:53.507 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:04:53.507 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:53.505+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:53.515 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:04:53.607 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:53.605+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:53.679 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:53.679 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:53.679 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:04:53.679 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:53.690 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:04:53.733 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:53.733+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:53.792 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:53.808 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:53.807+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:53.819 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:53.823 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:53.823 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:53.823 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:04:53.823 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:04:53.824 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:53.834 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:53.848 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:53.909 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:53.908+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:54.001 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:54.001 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:04:54.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:04:54.002 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:54.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.002 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:54.002 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:54.002 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.708580434, 2023-07-02T14:04:54.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:04:54.003 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:54.003 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:54.003 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:54.003 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:54.003 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:54.003 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:04:54.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.004 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:54.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:54.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:54.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:54.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:54.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:54.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:54.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:54.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:04:54.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:04:54.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:54.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:04:54.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:04:54.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:54.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:04:54.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:04:54.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:54.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:04:54.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:04:54.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:54.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:04:54.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:04:54.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:54.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:04:54.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:04:54.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:54.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:04:54.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:04:54.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:54.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:04:54.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:04:54.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:54.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:04:54.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:04:54.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:54.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:04:54.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:04:54.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:54.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:04:54.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:04:54.013 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.013 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.013 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.014 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:54.014 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:54.014 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.70834541100001, 2023-07-02T14:04:54.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:04:54.014 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:54.014 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:54.015 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:54.015 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:54.015 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:54.015 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:04:54.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.015 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:54.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:54.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:54.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:54.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:54.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:54.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:54.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:54.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:04:54.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:04:54.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:54.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:04:54.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:04:54.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:54.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:04:54.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:04:54.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:54.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:04:54.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:04:54.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:54.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:04:54.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:04:54.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:54.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:04:54.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:04:54.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:54.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:04:54.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:04:54.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:54.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:04:54.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:04:54.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:54.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:04:54.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:04:54.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:54.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:04:54.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:04:54.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:54.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:04:54.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:04:54.025 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.025 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.025 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.025 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:54.025 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:54.026 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.708146277, 2023-07-02T14:04:54.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:04:54.026 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:54.026 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:54.026 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:54.026 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:54.026 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:54.027 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:04:54.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.027 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:54.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:54.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:54.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:54.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:54.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:54.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:54.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:54.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:04:54.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:04:54.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:54.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:04:54.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:04:54.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:54.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:04:54.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:04:54.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:54.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:04:54.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:04:54.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:54.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:04:54.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:04:54.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:54.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:04:54.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:04:54.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:54.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:04:54.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:04:54.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:54.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:04:54.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:04:54.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:54.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:04:54.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:04:54.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:54.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:04:54.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:04:54.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:54.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:04:54.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:04:54.039 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.039 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.039 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.039 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:54.039 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:54.040 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.70778115300001, 2023-07-02T14:04:54.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:04:54.040 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:54.040 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:54.040 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:54.040 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:54.041 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:54.041 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:04:54.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.041 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:54.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:54.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:54.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:54.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:54.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:54.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:54.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:54.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:04:54.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:04:54.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:54.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:04:54.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:04:54.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:54.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:04:54.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:04:54.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:54.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:04:54.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:04:54.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:54.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:04:54.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:04:54.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:54.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:04:54.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:04:54.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:54.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:04:54.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:04:54.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:54.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:04:54.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:04:54.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:54.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:04:54.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:04:54.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:54.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:04:54.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:04:54.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:54.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:04:54.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:04:54.050 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.050 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.051 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.051 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:54.051 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:54.051 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.70770009200001, 2023-07-02T14:04:54.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:04:54.052 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:54.052 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:54.052 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:54.052 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:54.052 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:54.052 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:04:54.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.053 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:54.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:54.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:54.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:54.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:54.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:54.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:54.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:54.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:04:54.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:04:54.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:54.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:04:54.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:04:54.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:54.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:04:54.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:04:54.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:54.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:04:54.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:04:54.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:54.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:04:54.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:04:54.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:54.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:04:54.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:04:54.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:54.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:04:54.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:04:54.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:54.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:04:54.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:04:54.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:54.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:04:54.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:04:54.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:54.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:04:54.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:04:54.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:54.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:04:54.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:04:54.067 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.067 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.068 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.068 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:54.069 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:54.069 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.707608421, 2023-07-02T14:04:54.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:04:54.069 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:54.070 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:54.070 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:54.070 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:54.070 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:54.071 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:04:54.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.071 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:54.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:54.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:54.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:54.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:54.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:54.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:54.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:54.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:04:54.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:04:54.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:54.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:04:54.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:04:54.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:54.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:04:54.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:04:54.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:54.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:04:54.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:04:54.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:54.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:04:54.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:04:54.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:54.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:04:54.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:04:54.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:54.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:04:54.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:04:54.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:54.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:04:54.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:04:54.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:54.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:04:54.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:04:54.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:54.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:04:54.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:04:54.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:54.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:04:54.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:04:54.081 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.081 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.081 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.082 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:54.082 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:54.082 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.70726897899999, 2023-07-02T14:04:54.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:04:54.082 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:54.082 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:54.083 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:54.083 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:54.083 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:54.083 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:04:54.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.083 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:54.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:54.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:54.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:54.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:54.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:54.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:54.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:54.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:04:54.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:04:54.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:54.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:04:54.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:04:54.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:54.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:04:54.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:04:54.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:54.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:04:54.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:04:54.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:54.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:04:54.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:04:54.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:54.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:04:54.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:04:54.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:54.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:04:54.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:04:54.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:54.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:04:54.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:04:54.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:54.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:04:54.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:04:54.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:54.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:04:54.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:04:54.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:54.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:04:54.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:04:54.093 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.093 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.093 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.093 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:54.094 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:54.094 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.707082926, 2023-07-02T14:04:54.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:04:54.094 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:54.094 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:54.094 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:54.094 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:54.095 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:54.095 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:04:54.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.095 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:54.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:54.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:54.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:54.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:54.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:54.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:54.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:54.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:54.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:04:54.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:54.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:04:54.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:04:54.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:54.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:04:54.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:04:54.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:54.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:04:54.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:04:54.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:54.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:04:54.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:04:54.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:54.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:04:54.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:04:54.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:54.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:04:54.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:04:54.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:54.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:04:54.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:04:54.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:54.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:04:54.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:04:54.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:54.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:04:54.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:04:54.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:54.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:04:54.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:04:54.104 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.104 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.104 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.105 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:54.105 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:54.105 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.70692512400001, 2023-07-02T14:04:54.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:04:54.105 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:54.105 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:54.106 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:54.106 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:54.106 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:54.106 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:04:54.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.106 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:54.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:54.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:54.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:54.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:54.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:54.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:54.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:54.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:04:54.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:04:54.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:54.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:04:54.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:04:54.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:54.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:04:54.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:04:54.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:54.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:04:54.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:04:54.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:54.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:04:54.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:04:54.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:54.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:04:54.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:04:54.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:54.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:04:54.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:04:54.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:54.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:04:54.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:04:54.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:54.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:04:54.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:04:54.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:54.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:04:54.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:04:54.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:54.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:04:54.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:04:54.115 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.115 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.115 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.115 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:54.115 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:54.115 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.70673334200001, 2023-07-02T14:04:54.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:04:54.116 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:54.116 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:54.116 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:54.116 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:54.116 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:54.116 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:04:54.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.117 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:54.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:54.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:54.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:54.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:54.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:54.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:54.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:54.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:04:54.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:04:54.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:54.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:04:54.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:04:54.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:54.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:04:54.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:04:54.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:54.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:04:54.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:04:54.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:54.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:04:54.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:04:54.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:54.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:04:54.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:04:54.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:54.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:04:54.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:04:54.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:54.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:04:54.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:04:54.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:54.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:04:54.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:04:54.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:54.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:04:54.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:04:54.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:54.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:04:54.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:04:54.125 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.125 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.125 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.126 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:54.126 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:54.126 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.70655768200001, 2023-07-02T14:04:54.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:04:54.126 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:54.126 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:54.127 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:54.127 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:54.127 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:54.127 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:04:54.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.127 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:54.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:54.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:54.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:54.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:54.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:54.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:54.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:54.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:54.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:04:54.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:54.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:04:54.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:04:54.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:54.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:04:54.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:04:54.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:54.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:04:54.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:04:54.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:54.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:04:54.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:04:54.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:54.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:04:54.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:04:54.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:54.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:04:54.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:04:54.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:54.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:04:54.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:04:54.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:54.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:04:54.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:04:54.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:54.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:04:54.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:04:54.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:54.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:04:54.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:04:54.135 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.136 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.136 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.136 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:54.136 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:54.136 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.70646349399999, 2023-07-02T14:04:54.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:04:54.137 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:54.137 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:54.137 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:54.137 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:54.137 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:54.137 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:04:54.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.138 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:54.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:54.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:54.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:54.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:54.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:54.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:54.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:54.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:04:54.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:04:54.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:54.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:04:54.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:04:54.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:54.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:04:54.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:04:54.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:54.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:04:54.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:04:54.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:54.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:04:54.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:04:54.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:54.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:04:54.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:04:54.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:54.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:04:54.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:04:54.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:54.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:04:54.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:04:54.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:54.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:04:54.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:04:54.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:54.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:04:54.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:04:54.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:54.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:04:54.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:04:54.146 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.146 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.146 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.147 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:54.147 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:54.147 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.70626010800001, 2023-07-02T14:04:54.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:04:54.147 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:54.147 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:54.147 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:54.147 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:54.148 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:54.148 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:04:54.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.148 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:54.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:54.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:54.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:54.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:54.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:54.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:54.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:54.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:04:54.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:04:54.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:54.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:04:54.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:04:54.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:54.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:04:54.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:04:54.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:54.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:04:54.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:04:54.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:54.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:04:54.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:04:54.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:54.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:04:54.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:04:54.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:54.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:04:54.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:04:54.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:54.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:04:54.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:04:54.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:54.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:04:54.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:04:54.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:54.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:04:54.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:04:54.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:54.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:04:54.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:04:54.156 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.156 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.156 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.157 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:54.157 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:54.157 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.70610945199999, 2023-07-02T14:04:54.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:04:54.157 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:54.157 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:54.158 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:54.158 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:54.158 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:54.158 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:04:54.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.158 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:54.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:54.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:54.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:54.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:54.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:54.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:54.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:54.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:04:54.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:04:54.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:54.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:04:54.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:04:54.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:54.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:04:54.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:04:54.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:54.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:04:54.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:04:54.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:54.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:04:54.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:04:54.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:54.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:04:54.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:04:54.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:54.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:04:54.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:04:54.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:54.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:04:54.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:04:54.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:54.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:04:54.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:04:54.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:54.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:04:54.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:04:54.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:54.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:04:54.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:04:54.166 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.167 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.167 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.167 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:54.167 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:54.167 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.70603674899999, 2023-07-02T14:04:54.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:04:54.168 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:54.168 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:54.168 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:54.168 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:54.168 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:54.168 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:04:54.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.168 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:54.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:54.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:54.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:54.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:54.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:54.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:54.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:54.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:54.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:04:54.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:54.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:04:54.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:04:54.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:54.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:04:54.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:04:54.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:54.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:04:54.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:04:54.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:54.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:04:54.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:04:54.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:54.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:04:54.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:04:54.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:54.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:04:54.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:04:54.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:54.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:04:54.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:04:54.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:54.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:04:54.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:04:54.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:54.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:04:54.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:04:54.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:54.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:04:54.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:04:54.177 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.177 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.177 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.177 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:54.178 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:54.178 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.70591975900001, 2023-07-02T14:04:54.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:04:54.178 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:54.178 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:54.178 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:54.178 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:54.178 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:54.179 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:04:54.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.179 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:54.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:54.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:54.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:54.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:54.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:54.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:54.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:54.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:54.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:04:54.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:54.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:04:54.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:04:54.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:54.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:04:54.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:04:54.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:54.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:04:54.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:04:54.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:54.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:04:54.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:04:54.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:54.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:04:54.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:04:54.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:54.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:04:54.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:04:54.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:54.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:04:54.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:04:54.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:54.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:04:54.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:04:54.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:54.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:04:54.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:04:54.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:54.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:04:54.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:04:54.187 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.187 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.187 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.188 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:54.188 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:54.188 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.70577030800001, 2023-07-02T14:04:54.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:04:54.188 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:54.188 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:54.188 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:54.189 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:54.189 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:54.189 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:04:54.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.189 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:54.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:54.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:54.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:54.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:54.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:54.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:54.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:54.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:04:54.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:04:54.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:54.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:04:54.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:04:54.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:54.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:04:54.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:04:54.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:54.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:04:54.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:04:54.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:54.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:04:54.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:04:54.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:54.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:04:54.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:04:54.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:54.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:04:54.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:04:54.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:54.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:04:54.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:04:54.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:54.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:04:54.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:04:54.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:54.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:04:54.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:04:54.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:54.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:04:54.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:04:54.197 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.197 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.198 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.198 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:54.198 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:54.198 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.684294335, 2023-07-02T14:04:54.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:04:54.198 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:54.199 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:54.199 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:54.199 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:54.199 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:54.199 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:04:54.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.199 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:54.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:54.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:54.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:54.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:54.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:54.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:54.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:54.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:04:54.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:04:54.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:54.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:04:54.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:04:54.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:54.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:04:54.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:04:54.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:54.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:04:54.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:04:54.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:54.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:04:54.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:04:54.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:54.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:04:54.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:04:54.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:54.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:04:54.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:04:54.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:54.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:04:54.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:04:54.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:54.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:04:54.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:04:54.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:54.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:04:54.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:04:54.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:54.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:04:54.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:04:54.208 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.208 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.208 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.208 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:54.208 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:54.208 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.68416179900001, 2023-07-02T14:04:54.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:04:54.209 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:54.209 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:54.209 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:54.209 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:54.209 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:54.209 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:04:54.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.210 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:54.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:54.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:54.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:54.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:54.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:54.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:54.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:54.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:04:54.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:04:54.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:54.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:04:54.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:04:54.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:54.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:04:54.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:04:54.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:54.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:04:54.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:04:54.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:54.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:04:54.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:04:54.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:54.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:04:54.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:04:54.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:54.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:04:54.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:04:54.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:54.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:04:54.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:04:54.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:54.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:04:54.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:04:54.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:54.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:04:54.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:04:54.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:54.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:04:54.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:04:54.218 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.218 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.218 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.219 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:54.219 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:54.219 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.622825138, 2023-07-02T14:04:54.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:04:54.219 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:54.219 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:54.219 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:54.219 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:54.220 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:54.220 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:04:54.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.220 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:54.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:54.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:54.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:54.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:54.221 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:54.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:54.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:54.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:04:54.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:04:54.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:54.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:04:54.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:04:54.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:54.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:04:54.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:04:54.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:54.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:04:54.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:04:54.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:54.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:04:54.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:04:54.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:54.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:04:54.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:04:54.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:54.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:04:54.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:04:54.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:54.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:04:54.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:04:54.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:54.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:04:54.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:04:54.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:54.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:04:54.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:04:54.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:54.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:54.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:54.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:04:54.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:04:54.228 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.228 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.228 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.229 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:54.229 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:54.229 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:54.230 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:54.230 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:54.231 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:54.229+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:54.231 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:54.232 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:54.232 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:04:54.233 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:04:54.233 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:04:54.233 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:54.233 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:04:54.233 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:04:54.233 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:54.331 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:54.330+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:54.364 INFO:tasks.workunit.client.0.smithi163.stderr:+ sync b3fbb028-e267-4d25-a750-93ba18a2316a-mnt 2023-07-02T14:04:54.410 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:54.411 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:54.424 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:04:54.452 INFO:tasks.workunit.client.0.smithi163.stderr:+ rm -rf b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4 b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4.tar.gz 2023-07-02T14:04:54.467 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:54.466+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:54.532 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:54.531+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:54.555 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:54.555 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:54.565 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:04:54.648 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:54.647+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:54.728 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:54.728 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:54.733 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:54.732+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:54.739 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:04:54.835 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:54.834+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:54.905 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:54.905 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:54.915 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:04:54.935 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:54.935+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:55.036 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:55.035+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:55.039 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:55.039 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:55.049 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:04:55.137 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:55.136+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:55.213 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:55.213 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:55.223 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:04:55.238 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:55.237+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:55.338 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:55.337+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:55.368 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:55.368 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:55.378 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:04:55.417 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap29 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap30 b3fbb028-e267-4d25-a750-93ba18a2316a@snap30.diff 2023-07-02T14:04:55.436 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i++ )) 2023-07-02T14:04:55.436 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i < 10 )) 2023-07-02T14:04:55.437 INFO:tasks.workunit.client.0.smithi163.stderr:+ pv -L 10M linux-5.4.tar.gz 2023-07-02T14:04:55.461 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:55.460+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:55.492 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:55.491+0000 7f3f8a88a700 0 -- 172.21.15.163:0/2744411176 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x563cc0af4610 msgr2=0x563cc0af6a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:55.500 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:55.499+0000 7f3f8a089700 0 -- 172.21.15.163:0/2744411176 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7f3f70008780 msgr2=0x7f3f7000abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:55.508 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:04:55.552 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:55.551+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:55.570 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:55.570 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:55.582 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:04:55.622 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 100% complete...done. 2023-07-02T14:04:55.626 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:04:55.627 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=31 2023-07-02T14:04:55.627 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap31 2023-07-02T14:04:55.666 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:55.665+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:55.741 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:55.740+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:55.753 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:55.753 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:55.764 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:04:55.769 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:04:55.775 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 1 2023-07-02T14:04:55.842 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:55.841+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:55.899 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:55.899 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:55.909 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:04:55.943 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:55.942+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:56.044 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:56.043+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:56.074 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:56.074 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:56.084 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:04:56.167 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:56.166+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:56.234 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:04:56.245 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:56.244+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:56.247 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:56.247 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:56.257 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:04:56.346 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:56.345+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:56.377 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:56.395 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:56.412 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:56.421 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:56.421 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:56.431 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:04:56.432 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:56.446 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:56.445+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:56.449 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:56.449 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:04:56.450 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:04:56.450 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:04:56.450 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:56.450 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:04:56.450 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:04:56.452 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:56.452 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:56.547 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:56.546+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:56.554 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:56.555 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:04:56.555 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:04:56.555 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:04:56.555 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:56.565 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:56.648 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:56.647+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:56.731 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:04:56.732 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:04:56.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:04:56.732 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:04:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.437951196, 2023-07-02T14:04:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:04:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:04:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:04:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:04:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:04:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:04:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:04:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:04:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:04:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:04:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:04:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:04:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:04:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:04:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:04:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:04:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:04:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:04:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:04:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:04:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:04:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:04:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:04:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:04:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:04:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.43771617300001, 2023-07-02T14:04:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:04:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:04:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:56.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:56.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:56.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:04:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:04:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:04:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:04:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:04:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:04:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:04:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:04:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:04:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:04:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:04:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:04:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:04:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:04:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:04:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:04:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:04:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:56.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:04:56.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:04:56.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:56.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:04:56.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:04:56.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:56.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:04:56.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:04:56.756 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.756 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.757 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.757 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:56.758 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:56.758 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.437517039, 2023-07-02T14:04:56.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:04:56.759 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:56.759 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:56.759 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:56.759 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:56.760 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:56.760 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:04:56.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.760 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:56.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:56.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:04:56.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:56.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:04:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:04:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:04:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:04:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:04:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:04:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:04:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:04:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:04:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:04:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:04:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:04:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:04:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:04:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:04:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:04:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:04:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:04:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:04:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:04:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:04:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:04:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.43715191499999, 2023-07-02T14:04:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:04:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:04:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:04:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:04:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:04:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:04:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:04:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:04:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:04:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:04:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:04:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:04:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:04:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:04:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:04:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:04:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:04:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:04:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:04:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:04:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:04:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:04:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:04:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:04:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:04:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.43707085400001, 2023-07-02T14:04:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:04:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:04:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:04:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:04:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:04:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:04:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:04:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:04:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:04:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:04:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:04:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:04:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:04:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:04:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:04:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:04:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:04:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:04:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:04:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:04:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:04:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:04:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:04:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:04:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:04:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.43697918300001, 2023-07-02T14:04:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:04:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:04:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:04:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:04:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:04:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:04:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:04:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:04:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:04:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:04:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:04:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:04:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:04:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:04:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:04:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:04:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:04:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:04:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:04:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:04:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:04:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:04:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:04:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:04:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:04:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.43663974099999, 2023-07-02T14:04:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:04:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:04:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:04:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:04:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:04:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:04:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:04:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:04:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:04:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:04:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:04:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:04:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:04:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:04:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:04:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:04:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:04:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:04:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:04:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:04:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:04:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:04:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:04:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:04:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:04:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.436453688, 2023-07-02T14:04:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:04:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:04:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:56.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:56.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:56.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:56.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:56.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:56.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:56.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:04:56.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:04:56.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:56.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:04:56.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:04:56.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:56.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:04:56.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:04:56.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:56.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:04:56.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:04:56.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:56.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:04:56.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:04:56.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:56.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:04:56.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:04:56.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:56.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:04:56.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:04:56.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:56.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:04:56.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:04:56.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:56.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:04:56.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:04:56.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:56.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:04:56.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:04:56.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:56.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:04:56.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:04:56.823 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.823 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.823 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.823 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:56.823 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:56.823 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.43629588600001, 2023-07-02T14:04:56.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:04:56.824 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:56.824 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:56.824 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:56.824 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:56.824 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:56.824 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:04:56.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.825 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:56.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:56.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:56.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:56.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:56.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:04:56.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:56.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:56.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:04:56.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:04:56.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:56.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:04:56.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:04:56.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:56.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:04:56.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:04:56.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:56.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:04:56.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:04:56.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:56.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:04:56.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:04:56.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:56.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:04:56.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:04:56.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:56.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:04:56.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:04:56.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:56.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:04:56.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:04:56.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:56.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:04:56.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:04:56.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:56.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:04:56.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:04:56.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:56.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:04:56.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:04:56.833 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.833 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.833 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.833 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:56.834 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:56.834 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.43610410400001, 2023-07-02T14:04:56.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:04:56.834 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:56.834 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:56.834 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:56.834 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:56.835 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:56.835 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:04:56.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.835 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:56.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:56.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:56.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:56.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:56.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:04:56.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:56.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:56.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:04:56.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:04:56.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:56.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:04:56.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:04:56.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:56.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:04:56.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:04:56.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:56.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:04:56.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:04:56.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:56.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:04:56.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:04:56.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:56.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:04:56.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:04:56.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:56.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:04:56.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:04:56.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:56.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:04:56.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:04:56.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:56.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:04:56.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:04:56.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:56.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:04:56.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:04:56.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:56.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:04:56.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:04:56.843 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.843 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.843 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.844 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:56.844 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:56.844 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.43592844400001, 2023-07-02T14:04:56.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:04:56.844 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:56.845 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:56.845 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:56.845 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:56.845 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:56.845 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:04:56.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.845 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:56.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:56.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:56.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:56.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:56.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:56.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:56.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:56.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:04:56.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:04:56.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:56.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:04:56.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:04:56.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:56.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:04:56.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:04:56.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:56.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:04:56.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:04:56.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:56.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:04:56.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:04:56.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:56.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:04:56.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:04:56.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:56.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:04:56.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:04:56.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:56.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:04:56.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:04:56.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:56.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:04:56.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:04:56.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:56.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:04:56.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:04:56.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:56.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:04:56.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:04:56.854 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.854 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.854 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.854 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:56.854 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:56.854 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.43583425599999, 2023-07-02T14:04:56.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:04:56.855 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:56.855 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:56.855 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:56.855 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:56.855 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:56.855 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:04:56.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.856 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:56.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:56.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:56.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:56.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:56.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:04:56.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:56.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:56.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:04:56.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:04:56.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:56.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:04:56.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:04:56.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:56.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:04:56.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:04:56.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:56.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:04:56.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:04:56.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:56.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:04:56.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:04:56.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:56.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:04:56.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:04:56.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:56.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:04:56.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:04:56.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:56.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:04:56.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:04:56.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:56.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:04:56.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:04:56.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:56.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:04:56.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:04:56.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:56.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:04:56.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:04:56.864 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.864 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.864 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.865 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:56.865 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:56.865 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.43563087000001, 2023-07-02T14:04:56.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:04:56.865 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:56.865 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:56.865 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:56.865 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:56.866 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:56.866 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:04:56.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.866 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:56.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:56.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:56.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:56.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:56.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:04:56.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:56.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:56.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:04:56.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:04:56.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:56.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:04:56.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:04:56.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:56.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:04:56.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:04:56.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:56.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:04:56.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:04:56.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:56.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:04:56.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:04:56.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:56.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:04:56.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:04:56.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:56.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:04:56.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:04:56.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:56.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:04:56.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:04:56.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:56.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:04:56.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:04:56.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:56.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:04:56.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:04:56.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:56.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:04:56.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:04:56.874 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.874 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.874 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.875 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:56.875 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:56.875 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.43548021399999, 2023-07-02T14:04:56.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:04:56.875 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:56.875 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:56.876 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:56.876 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:56.876 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:56.876 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:04:56.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.876 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:56.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:56.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:56.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:56.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:56.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:04:56.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:56.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:56.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:04:56.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:04:56.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:56.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:04:56.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:04:56.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:56.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:04:56.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:04:56.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:56.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:04:56.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:04:56.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:56.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:04:56.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:04:56.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:56.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:04:56.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:04:56.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:56.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:04:56.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:04:56.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:56.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:04:56.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:04:56.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:56.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:04:56.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:04:56.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:56.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:04:56.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:04:56.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:56.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:04:56.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:04:56.884 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.885 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.885 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.885 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:56.885 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:56.885 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.43540751099999, 2023-07-02T14:04:56.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:04:56.886 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:56.886 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:56.886 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:56.886 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:56.886 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:56.886 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:04:56.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.887 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:56.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:56.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:56.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:56.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:56.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:56.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:56.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:56.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:04:56.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:04:56.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:56.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:04:56.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:04:56.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:56.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:04:56.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:04:56.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:56.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:04:56.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:04:56.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:56.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:04:56.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:04:56.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:56.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:04:56.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:04:56.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:56.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:04:56.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:04:56.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:56.893 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:04:56.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:04:56.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.893 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:56.893 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:04:56.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:04:56.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:56.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:04:56.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:04:56.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:56.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:04:56.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:04:56.895 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.895 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.895 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.895 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:56.896 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:56.896 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.43529052100001, 2023-07-02T14:04:56.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:04:56.896 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:56.896 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:56.896 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:56.896 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:56.896 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:56.897 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:04:56.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.897 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:56.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:56.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:56.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:56.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:56.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:56.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:56.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:56.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:04:56.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:04:56.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:56.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:04:56.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:04:56.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:56.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:04:56.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:04:56.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:56.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:04:56.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:04:56.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:56.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:04:56.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:04:56.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:56.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:04:56.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:04:56.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:56.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:04:56.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:04:56.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:56.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:04:56.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:04:56.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:56.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:04:56.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:04:56.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:56.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:04:56.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:04:56.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:56.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:04:56.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:04:56.905 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.905 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.905 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.906 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:56.906 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:56.906 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.43514106999999, 2023-07-02T14:04:56.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:04:56.906 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:56.906 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:56.906 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:56.906 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:56.907 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:56.907 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:04:56.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.907 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:56.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:56.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:56.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:56.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:56.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:04:56.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:56.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:56.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:04:56.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:04:56.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:56.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:04:56.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:04:56.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:56.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:04:56.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:04:56.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:56.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:04:56.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:04:56.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:56.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:04:56.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:04:56.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:56.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:04:56.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:04:56.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:56.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:04:56.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:04:56.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:56.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:04:56.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:04:56.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:56.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:04:56.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:04:56.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:56.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:04:56.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:04:56.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:56.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:04:56.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:04:56.915 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.915 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.915 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.916 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:56.916 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:56.916 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.41366509700001, 2023-07-02T14:04:56.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:04:56.916 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:56.917 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:56.917 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:56.917 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:56.917 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:56.917 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:04:56.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.917 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:56.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:56.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:56.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:56.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:56.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:04:56.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:56.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:56.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:04:56.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:04:56.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:56.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:04:56.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:04:56.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:56.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:04:56.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:04:56.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:56.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:04:56.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:04:56.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:56.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:04:56.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:04:56.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:56.922 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:04:56.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:04:56.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:56.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:04:56.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:04:56.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:56.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:04:56.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:04:56.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:56.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:04:56.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:04:56.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:56.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:04:56.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:04:56.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:56.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:04:56.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:04:56.925 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.926 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.926 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.926 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:56.926 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:56.926 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.41353256100001, 2023-07-02T14:04:56.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:04:56.927 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:56.927 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:56.927 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:56.927 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:56.927 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:56.927 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:04:56.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.927 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:56.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:56.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:56.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:56.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:56.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:04:56.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:56.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:56.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:04:56.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:04:56.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:56.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:04:56.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:04:56.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:56.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:04:56.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:04:56.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:56.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:04:56.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:04:56.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:56.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:04:56.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:04:56.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:56.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:04:56.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:04:56.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:56.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:04:56.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:04:56.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:56.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:04:56.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:04:56.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:56.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:04:56.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:04:56.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:56.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:04:56.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:04:56.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:56.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:04:56.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:04:56.936 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.936 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.936 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.936 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:04:56.936 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:56.937 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.3521959, 2023-07-02T14:04:56.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:04:56.937 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:04:56.937 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:56.937 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:04:56.937 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:04:56.937 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:56.937 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:04:56.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.938 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:04:56.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:04:56.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:56.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:04:56.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:04:56.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:04:56.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:04:56.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:04:56.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:04:56.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:04:56.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:04:56.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:04:56.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:04:56.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:04:56.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:04:56.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:04:56.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:04:56.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:04:56.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:04:56.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:04:56.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:04:56.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:04:56.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:04:56.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:04:56.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:04:56.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:04:56.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:04:56.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:04:56.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:04:56.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:04:56.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:04:56.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:56.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:04:56.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:04:56.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:04:56.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:04:56.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:04:56.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:04:56.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:04:56.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:04:56.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:04:56.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:04:56.946 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.946 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.946 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.946 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:04:56.946 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:04:56.947 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:04:56.948 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap30 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap31 b3fbb028-e267-4d25-a750-93ba18a2316a@snap31.diff 2023-07-02T14:04:56.948 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:56.829+0000 7f0d6e6c7700 0 -- 172.21.15.163:0/654906395 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7f0d54008780 msgr2=0x7f0d5400abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-07-02T14:04:56.948 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:56.832+0000 7f0d6e6c7700 0 -- 172.21.15.163:0/654906395 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7f0d50036c20 msgr2=0x7f0d50039050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:56.949 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:04:56.949 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:56.946+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:56.950 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:57.049 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:57.048+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:57.143 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:57.144 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:57.144 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:57.144 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:57.149 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:57.148+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:57.155 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:04:57.250 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:57.249+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:57.346 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:57.346 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:57.346 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:57.346 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:57.346 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:57.351 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:57.350+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:57.360 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:04:57.452 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:57.451+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:57.549 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:57.550 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:57.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:57.550 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:57.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.550 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2620 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 196608~786432] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:57.550 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:57.551 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.22311574699999, 2023-07-02T14:04:57.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966, 2023-07-02T14:04:57.551 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:57.551 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:57.551 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:57.551 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:57.551 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:57.552 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2620 2023-07-02T14:04:57.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.552 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:57.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:57.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:57.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:57.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:57.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:04:57.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:57.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:57.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324571+0000", 2023-07-02T14:04:57.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016686649999999999 2023-07-02T14:04:57.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:57.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326240+0000", 2023-07-02T14:04:57.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1844e-05 2023-07-02T14:04:57.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:57.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326252+0000", 2023-07-02T14:04:57.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7760000000000003e-06 2023-07-02T14:04:57.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:57.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326258+0000", 2023-07-02T14:04:57.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0634000000000003e-05 2023-07-02T14:04:57.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326318+0000", 2023-07-02T14:04:57.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9629999999999993e-06 2023-07-02T14:04:57.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:57.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326328+0000", 2023-07-02T14:04:57.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.010877045 2023-07-02T14:04:57.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337205+0000", 2023-07-02T14:04:57.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022401300000000001 2023-07-02T14:04:57.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:57.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337429+0000", 2023-07-02T14:04:57.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000139822 2023-07-02T14:04:57.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:57.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337569+0000", 2023-07-02T14:04:57.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032942119999999999 2023-07-02T14:04:57.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:57.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340863+0000", 2023-07-02T14:04:57.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021302560000000001 2023-07-02T14:04:57.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:57.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342994+0000", 2023-07-02T14:04:57.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5634000000000002e-05 2023-07-02T14:04:57.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:57.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343059+0000", 2023-07-02T14:04:57.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143721 2023-07-02T14:04:57.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:57.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343203+0000", 2023-07-02T14:04:57.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966 2023-07-02T14:04:57.561 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.562 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.562 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2622 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 983040~36864] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:57.562 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:57.562 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.22142323600001, 2023-07-02T14:04:57.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664, 2023-07-02T14:04:57.563 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:57.563 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:57.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:57.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:57.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:57.563 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2622 2023-07-02T14:04:57.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:04:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326265+0000", 2023-07-02T14:04:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1282000000000001e-05 2023-07-02T14:04:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326286+0000", 2023-07-02T14:04:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9480000000000002e-06 2023-07-02T14:04:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326293+0000", 2023-07-02T14:04:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3610000000000003e-06 2023-07-02T14:04:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326297+0000", 2023-07-02T14:04:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7707999999999994e-05 2023-07-02T14:04:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326385+0000", 2023-07-02T14:04:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8070000000000006e-06 2023-07-02T14:04:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326394+0000", 2023-07-02T14:04:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.014535878 2023-07-02T14:04:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340930+0000", 2023-07-02T14:04:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000154551 2023-07-02T14:04:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341084+0000", 2023-07-02T14:04:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00032375900000000002 2023-07-02T14:04:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341408+0000", 2023-07-02T14:04:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012631066 2023-07-02T14:04:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354039+0000", 2023-07-02T14:04:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5741000000000001e-05 2023-07-02T14:04:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354055+0000", 2023-07-02T14:04:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8794000000000006e-05 2023-07-02T14:04:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354124+0000", 2023-07-02T14:04:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014667700000000001 2023-07-02T14:04:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354270+0000", 2023-07-02T14:04:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664 2023-07-02T14:04:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2727 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~12288] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.22138746100001, 2023-07-02T14:04:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999, 2023-07-02T14:04:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2727 2023-07-02T14:04:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:04:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3834e-05 2023-07-02T14:04:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326315+0000", 2023-07-02T14:04:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3029999999999996e-06 2023-07-02T14:04:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326321+0000", 2023-07-02T14:04:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7590000000000002e-06 2023-07-02T14:04:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326324+0000", 2023-07-02T14:04:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112872 2023-07-02T14:04:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326437+0000", 2023-07-02T14:04:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2290000000000002e-06 2023-07-02T14:04:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326442+0000", 2023-07-02T14:04:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0155896879999999 2023-07-02T14:04:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342031+0000", 2023-07-02T14:04:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016496100000000001 2023-07-02T14:04:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342196+0000", 2023-07-02T14:04:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021998699999999999 2023-07-02T14:04:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342416+0000", 2023-07-02T14:04:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011869744999999999 2023-07-02T14:04:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354286+0000", 2023-07-02T14:04:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0361e-05 2023-07-02T14:04:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354347+0000", 2023-07-02T14:04:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3312000000000005e-05 2023-07-02T14:04:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354430+0000", 2023-07-02T14:04:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126444 2023-07-02T14:04:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354556+0000", 2023-07-02T14:04:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999 2023-07-02T14:04:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2730 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [set-alloc-hint object_size 4194304 write_size 4194304,write 454656~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.22136245199999, 2023-07-02T14:04:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999, 2023-07-02T14:04:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2730 2023-07-02T14:04:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:04:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6049999999999997e-06 2023-07-02T14:04:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326332+0000", 2023-07-02T14:04:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3000000000000002e-06 2023-07-02T14:04:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326335+0000", 2023-07-02T14:04:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.719e-05 2023-07-02T14:04:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326352+0000", 2023-07-02T14:04:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011583599999999999 2023-07-02T14:04:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326468+0000", 2023-07-02T14:04:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.831e-06 2023-07-02T14:04:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326473+0000", 2023-07-02T14:04:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.016769915 2023-07-02T14:04:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343243+0000", 2023-07-02T14:04:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197352 2023-07-02T14:04:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343440+0000", 2023-07-02T14:04:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197581 2023-07-02T14:04:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343638+0000", 2023-07-02T14:04:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010980139 2023-07-02T14:04:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354618+0000", 2023-07-02T14:04:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056779530000000003 2023-07-02T14:04:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360296+0000", 2023-07-02T14:04:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101109 2023-07-02T14:04:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360397+0000", 2023-07-02T14:04:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000160273 2023-07-02T14:04:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360557+0000", 2023-07-02T14:04:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999 2023-07-02T14:04:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2752 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 1019904~3174400] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.22131622800001, 2023-07-02T14:04:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001, 2023-07-02T14:04:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2752 2023-07-02T14:04:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:57.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:04:57.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:57.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:57.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326372+0000", 2023-07-02T14:04:57.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052050380000000004 2023-07-02T14:04:57.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331577+0000", 2023-07-02T14:04:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6832000000000001e-05 2023-07-02T14:04:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331594+0000", 2023-07-02T14:04:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8220000000000007e-06 2023-07-02T14:04:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331602+0000", 2023-07-02T14:04:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9895999999999999e-05 2023-07-02T14:04:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331642+0000", 2023-07-02T14:04:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:04:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331649+0000", 2023-07-02T14:04:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0139141709999999 2023-07-02T14:04:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345563+0000", 2023-07-02T14:04:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113817 2023-07-02T14:04:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345677+0000", 2023-07-02T14:04:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018803400000000001 2023-07-02T14:04:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345865+0000", 2023-07-02T14:04:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087950359999999991 2023-07-02T14:04:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354660+0000", 2023-07-02T14:04:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059116669999999998 2023-07-02T14:04:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360572+0000", 2023-07-02T14:04:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0992000000000002e-05 2023-07-02T14:04:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360633+0000", 2023-07-02T14:04:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013601600000000001 2023-07-02T14:04:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360769+0000", 2023-07-02T14:04:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001 2023-07-02T14:04:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.609 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2759 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 0~454656] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:57.609 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:57.609 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.216076431, 2023-07-02T14:04:57.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999, 2023-07-02T14:04:57.609 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:57.609 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:57.610 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:57.610 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:57.610 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:57.610 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2759 2023-07-02T14:04:57.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.610 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:57.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:04:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331611+0000", 2023-07-02T14:04:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053795999999999998 2023-07-02T14:04:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332149+0000", 2023-07-02T14:04:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4970000000000007e-06 2023-07-02T14:04:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332158+0000", 2023-07-02T14:04:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:04:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332162+0000", 2023-07-02T14:04:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3526000000000001e-05 2023-07-02T14:04:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332185+0000", 2023-07-02T14:04:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:04:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332189+0000", 2023-07-02T14:04:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0186295910000001 2023-07-02T14:04:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350819+0000", 2023-07-02T14:04:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000111123 2023-07-02T14:04:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350930+0000", 2023-07-02T14:04:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019575300000000001 2023-07-02T14:04:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.351126+0000", 2023-07-02T14:04:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011503298 2023-07-02T14:04:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362629+0000", 2023-07-02T14:04:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085443989999999994 2023-07-02T14:04:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371173+0000", 2023-07-02T14:04:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3547999999999995e-05 2023-07-02T14:04:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371237+0000", 2023-07-02T14:04:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790300000000001 2023-07-02T14:04:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371375+0000", 2023-07-02T14:04:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999 2023-07-02T14:04:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2760 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 458752~131072] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.215520246, 2023-07-02T14:04:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001, 2023-07-02T14:04:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2760 2023-07-02T14:04:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:04:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332168+0000", 2023-07-02T14:04:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013281100000000001 2023-07-02T14:04:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332301+0000", 2023-07-02T14:04:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7389999999999999e-06 2023-07-02T14:04:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332308+0000", 2023-07-02T14:04:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0639999999999998e-06 2023-07-02T14:04:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332312+0000", 2023-07-02T14:04:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1735000000000001e-05 2023-07-02T14:04:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332333+0000", 2023-07-02T14:04:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8342000000000002e-05 2023-07-02T14:04:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332352+0000", 2023-07-02T14:04:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02234782 2023-07-02T14:04:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354699+0000", 2023-07-02T14:04:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110371 2023-07-02T14:04:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354810+0000", 2023-07-02T14:04:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022174900000000001 2023-07-02T14:04:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355032+0000", 2023-07-02T14:04:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076053329999999997 2023-07-02T14:04:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362637+0000", 2023-07-02T14:04:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087913609999999993 2023-07-02T14:04:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371428+0000", 2023-07-02T14:04:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8532999999999998e-05 2023-07-02T14:04:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371477+0000", 2023-07-02T14:04:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010990800000000001 2023-07-02T14:04:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371587+0000", 2023-07-02T14:04:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001 2023-07-02T14:04:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2766 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 589824~45056] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.215371621, 2023-07-02T14:04:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188, 2023-07-02T14:04:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2766 2023-07-02T14:04:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:04:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2247999999999998e-05 2023-07-02T14:04:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:57.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332369+0000", 2023-07-02T14:04:57.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4189999999999996e-06 2023-07-02T14:04:57.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:57.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332376+0000", 2023-07-02T14:04:57.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1049999999999999e-06 2023-07-02T14:04:57.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332379+0000", 2023-07-02T14:04:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6894000000000003e-05 2023-07-02T14:04:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332416+0000", 2023-07-02T14:04:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3970000000000002e-06 2023-07-02T14:04:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332422+0000", 2023-07-02T14:04:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0231382280000001 2023-07-02T14:04:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355560+0000", 2023-07-02T14:04:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9079000000000005e-05 2023-07-02T14:04:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355659+0000", 2023-07-02T14:04:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018005000000000001 2023-07-02T14:04:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355839+0000", 2023-07-02T14:04:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068044439999999998 2023-07-02T14:04:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362644+0000", 2023-07-02T14:04:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090155589999999994 2023-07-02T14:04:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371659+0000", 2023-07-02T14:04:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1297999999999994e-05 2023-07-02T14:04:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371730+0000", 2023-07-02T14:04:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013100699999999999 2023-07-02T14:04:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371861+0000", 2023-07-02T14:04:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188 2023-07-02T14:04:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2784 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 0~393216] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.21530316899999, 2023-07-02T14:04:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999, 2023-07-02T14:04:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2784 2023-07-02T14:04:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:04:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332385+0000", 2023-07-02T14:04:57.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040252400000000002 2023-07-02T14:04:57.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:57.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332787+0000", 2023-07-02T14:04:57.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0208e-05 2023-07-02T14:04:57.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:57.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332798+0000", 2023-07-02T14:04:57.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2023-07-02T14:04:57.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:57.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332801+0000", 2023-07-02T14:04:57.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4838000000000001e-05 2023-07-02T14:04:57.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332836+0000", 2023-07-02T14:04:57.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5319999999999998e-06 2023-07-02T14:04:57.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:57.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332839+0000", 2023-07-02T14:04:57.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0234290109999999 2023-07-02T14:04:57.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356268+0000", 2023-07-02T14:04:57.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011165700000000001 2023-07-02T14:04:57.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:57.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356380+0000", 2023-07-02T14:04:57.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022504799999999999 2023-07-02T14:04:57.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:57.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356605+0000", 2023-07-02T14:04:57.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015276201 2023-07-02T14:04:57.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:57.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371881+0000", 2023-07-02T14:04:57.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5496999999999999e-05 2023-07-02T14:04:57.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:57.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371957+0000", 2023-07-02T14:04:57.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4570000000000002e-05 2023-07-02T14:04:57.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:57.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372001+0000", 2023-07-02T14:04:57.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3880000000000003e-05 2023-07-02T14:04:57.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:57.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372085+0000", 2023-07-02T14:04:57.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999 2023-07-02T14:04:57.654 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.655 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.655 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2785 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 405504~446464] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:57.655 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:57.655 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.214881718, 2023-07-02T14:04:57.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999, 2023-07-02T14:04:57.656 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:57.656 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:57.656 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:57.656 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:57.656 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:57.656 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2785 2023-07-02T14:04:57.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.656 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:57.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:57.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:57.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:57.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:57.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:57.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:57.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:57.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:04:57.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059527500000000004 2023-07-02T14:04:57.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:57.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333402+0000", 2023-07-02T14:04:57.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7059999999999993e-06 2023-07-02T14:04:57.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:57.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333411+0000", 2023-07-02T14:04:57.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0450000000000001e-06 2023-07-02T14:04:57.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:57.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333414+0000", 2023-07-02T14:04:57.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5790999999999997e-05 2023-07-02T14:04:57.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:04:57.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3320000000000002e-06 2023-07-02T14:04:57.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:57.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333454+0000", 2023-07-02T14:04:57.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.025310269 2023-07-02T14:04:57.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358764+0000", 2023-07-02T14:04:57.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012414899999999999 2023-07-02T14:04:57.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:57.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358888+0000", 2023-07-02T14:04:57.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000217028 2023-07-02T14:04:57.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:57.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.359105+0000", 2023-07-02T14:04:57.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012788212 2023-07-02T14:04:57.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:57.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371893+0000", 2023-07-02T14:04:57.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012201569999999999 2023-07-02T14:04:57.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:57.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373114+0000", 2023-07-02T14:04:57.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4512999999999995e-05 2023-07-02T14:04:57.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:57.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373198+0000", 2023-07-02T14:04:57.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132373 2023-07-02T14:04:57.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:57.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373330+0000", 2023-07-02T14:04:57.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999 2023-07-02T14:04:57.666 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.666 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.666 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.667 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2786 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1310720~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:57.667 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:57.667 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.21426902499999, 2023-07-02T14:04:57.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001, 2023-07-02T14:04:57.667 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:57.667 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:57.667 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:57.667 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:57.668 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:57.668 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2786 2023-07-02T14:04:57.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.668 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:57.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:57.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:57.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:57.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:57.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:04:57.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:57.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:57.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333419+0000", 2023-07-02T14:04:57.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2009999999999999e-06 2023-07-02T14:04:57.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:57.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333427+0000", 2023-07-02T14:04:57.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0210000000000001e-06 2023-07-02T14:04:57.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:57.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333431+0000", 2023-07-02T14:04:57.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3769999999999999e-06 2023-07-02T14:04:57.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333434+0000", 2023-07-02T14:04:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1161e-05 2023-07-02T14:04:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333485+0000", 2023-07-02T14:04:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1549000000000001e-05 2023-07-02T14:04:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:57.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333496+0000", 2023-07-02T14:04:57.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0273116339999999 2023-07-02T14:04:57.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360808+0000", 2023-07-02T14:04:57.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000115128 2023-07-02T14:04:57.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:57.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360923+0000", 2023-07-02T14:04:57.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018367700000000001 2023-07-02T14:04:57.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:57.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361107+0000", 2023-07-02T14:04:57.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010796923999999999 2023-07-02T14:04:57.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:57.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371904+0000", 2023-07-02T14:04:57.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014983520000000001 2023-07-02T14:04:57.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:57.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373402+0000", 2023-07-02T14:04:57.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0520999999999999e-05 2023-07-02T14:04:57.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:57.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373463+0000", 2023-07-02T14:04:57.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000129095 2023-07-02T14:04:57.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:57.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373592+0000", 2023-07-02T14:04:57.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001 2023-07-02T14:04:57.678 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.678 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.678 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.678 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2787 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1433600~8192] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:57.678 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:57.678 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.214252225, 2023-07-02T14:04:57.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999, 2023-07-02T14:04:57.679 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:57.679 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:57.679 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:57.679 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:57.679 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:57.679 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2787 2023-07-02T14:04:57.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.680 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:57.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:57.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:57.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:57.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:57.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:04:57.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:57.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:57.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333436+0000", 2023-07-02T14:04:57.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2731999999999999e-05 2023-07-02T14:04:57.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:57.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:04:57.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7440000000000001e-06 2023-07-02T14:04:57.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:57.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333456+0000", 2023-07-02T14:04:57.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8650000000000003e-06 2023-07-02T14:04:57.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:57.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333460+0000", 2023-07-02T14:04:57.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0077e-05 2023-07-02T14:04:57.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333530+0000", 2023-07-02T14:04:57.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4180000000000001e-06 2023-07-02T14:04:57.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:57.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333533+0000", 2023-07-02T14:04:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0284002720000001 2023-07-02T14:04:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361933+0000", 2023-07-02T14:04:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6781999999999996e-05 2023-07-02T14:04:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362030+0000", 2023-07-02T14:04:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018305400000000001 2023-07-02T14:04:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362213+0000", 2023-07-02T14:04:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097015109999999995 2023-07-02T14:04:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371915+0000", 2023-07-02T14:04:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001742295 2023-07-02T14:04:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373657+0000", 2023-07-02T14:04:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5980000000000003e-05 2023-07-02T14:04:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373713+0000", 2023-07-02T14:04:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011783800000000001 2023-07-02T14:04:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373831+0000", 2023-07-02T14:04:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999 2023-07-02T14:04:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2803 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 851968~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.214210373, 2023-07-02T14:04:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514, 2023-07-02T14:04:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2803 2023-07-02T14:04:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:04:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4943000000000001e-05 2023-07-02T14:04:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333493+0000", 2023-07-02T14:04:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2629999999999997e-06 2023-07-02T14:04:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333500+0000", 2023-07-02T14:04:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4159999999999997e-06 2023-07-02T14:04:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333505+0000", 2023-07-02T14:04:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4518999999999998e-05 2023-07-02T14:04:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333559+0000", 2023-07-02T14:04:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5900000000000008e-06 2023-07-02T14:04:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333568+0000", 2023-07-02T14:04:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0291061779999999 2023-07-02T14:04:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362674+0000", 2023-07-02T14:04:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110412 2023-07-02T14:04:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362784+0000", 2023-07-02T14:04:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015384599999999999 2023-07-02T14:04:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:57.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362938+0000", 2023-07-02T14:04:57.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091682310000000006 2023-07-02T14:04:57.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:57.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372106+0000", 2023-07-02T14:04:57.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017703230000000001 2023-07-02T14:04:57.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:57.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373877+0000", 2023-07-02T14:04:57.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6387000000000002e-05 2023-07-02T14:04:57.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:57.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373923+0000", 2023-07-02T14:04:57.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014099199999999999 2023-07-02T14:04:57.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:57.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374064+0000", 2023-07-02T14:04:57.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514 2023-07-02T14:04:57.701 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.701 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.701 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.701 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2805 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 634880~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:57.701 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:57.701 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.214177198, 2023-07-02T14:04:57.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814, 2023-07-02T14:04:57.702 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:57.702 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:57.702 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:57.702 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:57.702 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:57.702 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2805 2023-07-02T14:04:57.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.703 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:57.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:57.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:57.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:57.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:57.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:04:57.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:57.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:57.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333511+0000", 2023-07-02T14:04:57.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207092 2023-07-02T14:04:57.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:57.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333718+0000", 2023-07-02T14:04:57.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5110000000000003e-06 2023-07-02T14:04:57.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:57.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333727+0000", 2023-07-02T14:04:57.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3270000000000007e-06 2023-07-02T14:04:57.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:57.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333736+0000", 2023-07-02T14:04:57.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0319999999999999e-05 2023-07-02T14:04:57.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333756+0000", 2023-07-02T14:04:57.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1695000000000001e-05 2023-07-02T14:04:57.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:57.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:04:57.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0295750560000001 2023-07-02T14:04:57.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363343+0000", 2023-07-02T14:04:57.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3387999999999994e-05 2023-07-02T14:04:57.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:57.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363437+0000", 2023-07-02T14:04:57.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018373 2023-07-02T14:04:57.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:57.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363620+0000", 2023-07-02T14:04:57.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0084948769999999996 2023-07-02T14:04:57.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:57.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372115+0000", 2023-07-02T14:04:57.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020082170000000001 2023-07-02T14:04:57.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:57.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374123+0000", 2023-07-02T14:04:57.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8875999999999997e-05 2023-07-02T14:04:57.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:57.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374182+0000", 2023-07-02T14:04:57.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0001272 2023-07-02T14:04:57.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:57.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374309+0000", 2023-07-02T14:04:57.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814 2023-07-02T14:04:57.712 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.712 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.712 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.713 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2806 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 1359872~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:57.713 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:57.713 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.213946867, 2023-07-02T14:04:57.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234, 2023-07-02T14:04:57.713 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:57.713 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:57.714 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:57.714 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:57.714 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:57.714 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2806 2023-07-02T14:04:57.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.714 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:57.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:04:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333741+0000", 2023-07-02T14:04:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3978999999999999e-05 2023-07-02T14:04:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333755+0000", 2023-07-02T14:04:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:04:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333761+0000", 2023-07-02T14:04:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7010000000000001e-06 2023-07-02T14:04:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333764+0000", 2023-07-02T14:04:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5086000000000003e-05 2023-07-02T14:04:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333799+0000", 2023-07-02T14:04:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6389999999999992e-06 2023-07-02T14:04:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333806+0000", 2023-07-02T14:04:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.030496675 2023-07-02T14:04:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364303+0000", 2023-07-02T14:04:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011382700000000001 2023-07-02T14:04:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364417+0000", 2023-07-02T14:04:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192058 2023-07-02T14:04:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364609+0000", 2023-07-02T14:04:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075132130000000004 2023-07-02T14:04:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:57.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372122+0000", 2023-07-02T14:04:57.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022731750000000001 2023-07-02T14:04:57.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:57.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374395+0000", 2023-07-02T14:04:57.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5277000000000005e-05 2023-07-02T14:04:57.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374460+0000", 2023-07-02T14:04:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017074599999999999 2023-07-02T14:04:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374631+0000", 2023-07-02T14:04:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234 2023-07-02T14:04:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 868352~49152] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.213920153, 2023-07-02T14:04:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001, 2023-07-02T14:04:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:04:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:57.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:04:57.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:57.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:57.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:04:57.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6355000000000001e-05 2023-07-02T14:04:57.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:57.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333794+0000", 2023-07-02T14:04:57.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8329999999999996e-06 2023-07-02T14:04:57.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:57.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333801+0000", 2023-07-02T14:04:57.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5419999999999996e-06 2023-07-02T14:04:57.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:57.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333807+0000", 2023-07-02T14:04:57.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2023-07-02T14:04:57.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333826+0000", 2023-07-02T14:04:57.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.303e-06 2023-07-02T14:04:57.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:57.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333830+0000", 2023-07-02T14:04:57.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.031137559 2023-07-02T14:04:57.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364967+0000", 2023-07-02T14:04:57.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1390000000000004e-05 2023-07-02T14:04:57.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:57.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365059+0000", 2023-07-02T14:04:57.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017597399999999999 2023-07-02T14:04:57.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:57.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365234+0000", 2023-07-02T14:04:57.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068943950000000002 2023-07-02T14:04:57.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:57.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372129+0000", 2023-07-02T14:04:57.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026189910000000002 2023-07-02T14:04:57.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:57.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374748+0000", 2023-07-02T14:04:57.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6613999999999996e-05 2023-07-02T14:04:57.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:57.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374814+0000", 2023-07-02T14:04:57.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502300000000001 2023-07-02T14:04:57.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:57.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374950+0000", 2023-07-02T14:04:57.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001 2023-07-02T14:04:57.735 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.735 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.735 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.736 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2815 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 917504~49152] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:04:57.736 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:57.736 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 106.20970592, 2023-07-02T14:04:57.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998, 2023-07-02T14:04:57.736 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:57.736 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:57.737 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:57.737 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:57.737 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:57.737 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2815 2023-07-02T14:04:57.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.737 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:57.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:57.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:57.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:57.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:57.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337983+0000", 2023-07-02T14:04:57.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:04:57.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:57.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337980+0000", 2023-07-02T14:04:57.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017126700000000001 2023-07-02T14:04:57.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:57.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338152+0000", 2023-07-02T14:04:57.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2938999999999999e-05 2023-07-02T14:04:57.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:57.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338164+0000", 2023-07-02T14:04:57.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4448999999999999e-05 2023-07-02T14:04:57.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:57.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:04:57.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0281812 2023-07-02T14:04:57.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366370+0000", 2023-07-02T14:04:57.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1840000000000002e-05 2023-07-02T14:04:57.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:57.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366462+0000", 2023-07-02T14:04:57.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000186112 2023-07-02T14:04:57.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:04:57.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366648+0000", 2023-07-02T14:04:57.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054905709999999996 2023-07-02T14:04:57.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:57.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372139+0000", 2023-07-02T14:04:57.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044954790000000001 2023-07-02T14:04:57.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:57.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376634+0000", 2023-07-02T14:04:57.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4621000000000004e-05 2023-07-02T14:04:57.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:57.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376699+0000", 2023-07-02T14:04:57.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012862299999999999 2023-07-02T14:04:57.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:57.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.376827+0000", 2023-07-02T14:04:57.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038844445999999998 2023-07-02T14:04:57.746 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.746 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.746 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.746 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:160 2.5 2:bbff50a9:::rbd_data.125de1449605.0000000000000027:22 [sparse-read 0~4063232 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:04:57.746 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.840744+0000", 2023-07-02T14:04:57.746 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.70694528899999998, 2023-07-02T14:04:57.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040371059000000001, 2023-07-02T14:04:57.747 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:57.747 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:57.747 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:57.747 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:04:57.747 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:04:57.747 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 160 2023-07-02T14:04:57.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.748 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:57.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:57.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.840744+0000", 2023-07-02T14:04:57.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:57.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:57.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.840744+0000", 2023-07-02T14:04:57.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:57.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:57.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.840743+0000", 2023-07-02T14:04:57.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0309999999999999e-06 2023-07-02T14:04:57.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:57.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.840746+0000", 2023-07-02T14:04:57.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.182e-06 2023-07-02T14:04:57.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:57.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.840753+0000", 2023-07-02T14:04:57.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2889999999999999e-06 2023-07-02T14:04:57.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:57.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.840757+0000", 2023-07-02T14:04:57.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000134416 2023-07-02T14:04:57.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.840892+0000", 2023-07-02T14:04:57.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0310000000000001e-06 2023-07-02T14:04:57.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2023-07-02T14:04:57.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.840899+0000", 2023-07-02T14:04:57.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032962518000000003 2023-07-02T14:04:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.873861+0000", 2023-07-02T14:04:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000140729 2023-07-02T14:04:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.874002+0000", 2023-07-02T14:04:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007112452 2023-07-02T14:04:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.881115+0000", 2023-07-02T14:04:57.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040371059000000001 2023-07-02T14:04:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:172 2.5 2:a79a9f1f:::rbd_data.125de1449605.0000000000000039:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:04:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.848672+0000", 2023-07-02T14:04:57.755 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.69901721800000005, 2023-07-02T14:04:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041540012000000001, 2023-07-02T14:04:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:04:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:04:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 172 2023-07-02T14:04:57.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848672+0000", 2023-07-02T14:04:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848672+0000", 2023-07-02T14:04:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:04:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848669+0000", 2023-07-02T14:04:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1610000000000001e-05 2023-07-02T14:04:57.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848690+0000", 2023-07-02T14:04:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0169999999999998e-05 2023-07-02T14:04:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:57.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848710+0000", 2023-07-02T14:04:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0960000000000007e-06 2023-07-02T14:04:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848720+0000", 2023-07-02T14:04:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0717000000000002e-05 2023-07-02T14:04:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848780+0000", 2023-07-02T14:04:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.303e-06 2023-07-02T14:04:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2023-07-02T14:04:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848789+0000", 2023-07-02T14:04:57.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032354921000000002 2023-07-02T14:04:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.881144+0000", 2023-07-02T14:04:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0827000000000005e-05 2023-07-02T14:04:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.881214+0000", 2023-07-02T14:04:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089973080000000007 2023-07-02T14:04:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.890212+0000", 2023-07-02T14:04:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041540012000000001 2023-07-02T14:04:57.763 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.764 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.764 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.764 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:174 2.5 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:04:57.764 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.851335+0000", 2023-07-02T14:04:57.764 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.69635401100000005, 2023-07-02T14:04:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.044329581999999999, 2023-07-02T14:04:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:04:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:04:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 174 2023-07-02T14:04:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851335+0000", 2023-07-02T14:04:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851335+0000", 2023-07-02T14:04:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2023-07-02T14:04:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851332+0000", 2023-07-02T14:04:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6269999999999998e-05 2023-07-02T14:04:57.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:57.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851348+0000", 2023-07-02T14:04:57.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3189000000000001e-05 2023-07-02T14:04:57.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851371+0000", 2023-07-02T14:04:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.132e-06 2023-07-02T14:04:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:57.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851380+0000", 2023-07-02T14:04:57.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7970999999999999e-05 2023-07-02T14:04:57.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851428+0000", 2023-07-02T14:04:57.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1050000000000003e-06 2023-07-02T14:04:57.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2023-07-02T14:04:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851433+0000", 2023-07-02T14:04:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038825011999999999 2023-07-02T14:04:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:57.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.890258+0000", 2023-07-02T14:04:57.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0666000000000002e-05 2023-07-02T14:04:57.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:57.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.890349+0000", 2023-07-02T14:04:57.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0053153460000000003 2023-07-02T14:04:57.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.895664+0000", 2023-07-02T14:04:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.044329581999999999 2023-07-02T14:04:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.773 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:57.774 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:57.774 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:57.775 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:04:57.776 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete...2023-07-02T14:04:57.719+0000 7f0d6dec6700 0 -- 172.21.15.163:0/654906395 >> [v2:172.21.15.16:6800/83359,v1:172.21.15.16:6801/83359] conn(0x7f0d480225e0 msgr2=0x7f0d480249d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:57.776 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:57.728+0000 7f0d6e6c7700 0 -- 172.21.15.163:0/654906395 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7f0d50036c20 msgr2=0x7f0d50037040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:57.776 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:57.732+0000 7f0d6dec6700 0 -- 172.21.15.163:0/654906395 >> [v2:172.21.15.16:6800/83359,v1:172.21.15.16:6801/83359] conn(0x7f0d54008780 msgr2=0x7f0d5003b680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:57.777 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:57.774+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:57.794 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:04:57.793+0000 7f0d6eec8700 0 -- 172.21.15.163:0/654906395 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x55de3d007240 msgr2=0x55de3d009630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:04:57.799 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 2] out_osds: [5, 4] dead_osds: [3] live_osds: [0, 1, 2, 4, 5] 2023-07-02T14:04:57.799 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2023-07-02T14:04:57.800 INFO:tasks.ceph.osd.3:Restarting daemon 2023-07-02T14:04:57.800 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-07-02T14:04:57.802 INFO:tasks.ceph.osd.3:Started 2023-07-02T14:04:57.802 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:04:57.876 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:57.874+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:57.922 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:57.922 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:57.923 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:57.923 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:57.933 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:04:57.934 INFO:teuthology.orchestra.run.smithi022.stderr:no valid command found; 10 closest matches: 2023-07-02T14:04:57.935 INFO:teuthology.orchestra.run.smithi022.stderr:0 2023-07-02T14:04:57.935 INFO:teuthology.orchestra.run.smithi022.stderr:1 2023-07-02T14:04:57.935 INFO:teuthology.orchestra.run.smithi022.stderr:2 2023-07-02T14:04:57.935 INFO:teuthology.orchestra.run.smithi022.stderr:abort 2023-07-02T14:04:57.935 INFO:teuthology.orchestra.run.smithi022.stderr:assert 2023-07-02T14:04:57.936 INFO:teuthology.orchestra.run.smithi022.stderr:config diff 2023-07-02T14:04:57.936 INFO:teuthology.orchestra.run.smithi022.stderr:config diff get 2023-07-02T14:04:57.936 INFO:teuthology.orchestra.run.smithi022.stderr:config get 2023-07-02T14:04:57.936 INFO:teuthology.orchestra.run.smithi022.stderr:config help [] 2023-07-02T14:04:57.936 INFO:teuthology.orchestra.run.smithi022.stderr:config set ... 2023-07-02T14:04:57.937 INFO:teuthology.orchestra.run.smithi022.stderr:admin_socket: invalid command 2023-07-02T14:04:57.937 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-07-02T14:04:57.938 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2023-07-02T14:04:57.976 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:57.975+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:58.040 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 100% complete...done. 2023-07-02T14:04:58.060 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:04:58.060 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=32 2023-07-02T14:04:58.060 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap32 2023-07-02T14:04:58.077 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:58.076+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:58.099 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:58.100 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:58.100 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:04:58.100 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:04:58.100 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:58.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:04:58.194 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:58.193+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:58.277 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:58.277 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:04:58.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:04:58.278 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:04:58.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.278 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:04:58.278 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:58.278 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 108.8222517, 2023-07-02T14:04:58.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:04:58.279 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:58.279 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:58.279 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:58.280 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:58.280 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:58.280 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:04:58.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.281 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:58.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:58.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:58.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:58.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:58.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:04:58.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:58.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:58.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:04:58.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:04:58.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:58.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:04:58.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:04:58.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:58.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:04:58.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:04:58.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:58.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:04:58.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:04:58.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:04:58.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:04:58.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:58.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:04:58.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:04:58.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:04:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:04:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:04:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:04:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.293 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.293 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.293 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:04:58.294 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:58.294 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.95155869200001, 2023-07-02T14:04:58.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:04:58.294 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:04:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:04:58.296 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:04:58.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.296 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:58.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:58.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:58.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:58.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:58.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:04:58.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:58.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:58.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:04:58.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:04:58.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:58.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:04:58.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:04:58.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:58.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:04:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:04:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:04:58.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:04:58.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:04:58.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:04:58.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:04:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:04:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:04:58.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:04:58.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:04:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:04:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:58.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:04:58.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:04:58.309 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.309 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.309 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.310 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:58.310 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:58.310 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.951433971, 2023-07-02T14:04:58.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:04:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:58.312 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:04:58.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.312 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:58.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:58.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:04:58.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:58.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:58.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:04:58.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:04:58.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:58.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:04:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:04:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:04:58.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:04:58.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:58.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:04:58.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:04:58.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:04:58.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:04:58.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:58.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:04:58.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:04:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:04:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:04:58.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:58.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:04:58.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:04:58.324 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.324 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.325 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.325 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.944584087, 2023-07-02T14:04:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:04:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:58.327 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:58.327 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:58.327 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:58.327 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:04:58.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.328 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:58.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:04:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:04:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:04:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:04:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:58.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:04:58.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:04:58.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:58.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:04:58.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:04:58.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:58.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:04:58.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:04:58.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:04:58.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:04:58.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:58.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:04:58.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:04:58.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:04:58.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:04:58.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:58.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:04:58.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:04:58.340 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.340 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.340 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.341 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:58.341 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:58.341 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.944521162, 2023-07-02T14:04:58.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:04:58.342 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:58.342 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:58.342 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:58.342 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:58.343 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:58.343 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:04:58.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.343 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:58.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:58.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:58.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:58.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:58.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:04:58.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:04:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:04:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:04:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:04:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:04:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:04:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:04:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:04:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:04:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:04:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:04:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:04:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:04:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:04:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:04:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:04:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:04:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.944220741, 2023-07-02T14:04:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:04:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:58.353 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:58.353 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:04:58.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.353 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:58.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:58.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:58.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:58.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:58.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:04:58.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:58.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:58.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:04:58.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:04:58.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:58.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:04:58.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:04:58.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:58.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:04:58.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:04:58.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:58.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:04:58.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:04:58.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:04:58.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:04:58.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:58.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:04:58.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:04:58.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:04:58.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:04:58.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:58.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:04:58.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:04:58.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:04:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:04:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:04:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:04:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:04:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:04:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:04:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:04:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:04:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:04:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:04:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:04:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:04:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:04:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.943696241, 2023-07-02T14:04:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:04:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:04:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:04:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:04:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:04:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:04:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:04:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:04:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:04:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:04:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:04:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:04:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:04:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:04:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:04:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:04:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:04:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:04:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:04:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:04:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:04:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:04:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 80.819677115999994, 2023-07-02T14:04:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:04:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:04:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:04:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:04:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:04:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:04:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:04:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:04:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:04:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:04:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:04:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:04:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:04:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:04:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:04:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:04:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:04:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:04:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:04:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 80.494194411999999, 2023-07-02T14:04:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:04:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:04:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:04:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:04:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:04:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:04:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:04:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:04:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:04:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:04:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:04:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:04:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:04:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:04:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:04:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:04:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:04:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:04:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:04:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:04:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:04:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:04:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:04:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:04:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:04:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:04:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:04:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:04:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3887 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 80.490331222999998, 2023-07-02T14:04:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998, 2023-07-02T14:04:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3887 2023-07-02T14:04:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:04:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784019+0000", 2023-07-02T14:04:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4073999999999998e-05 2023-07-02T14:04:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784063+0000", 2023-07-02T14:04:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6630000000000004e-06 2023-07-02T14:04:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784070+0000", 2023-07-02T14:04:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.286e-06 2023-07-02T14:04:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784073+0000", 2023-07-02T14:04:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4941999999999998e-05 2023-07-02T14:04:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784118+0000", 2023-07-02T14:04:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8120000000000008e-06 2023-07-02T14:04:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784126+0000", 2023-07-02T14:04:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32999709399999999 2023-07-02T14:04:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114123+0000", 2023-07-02T14:04:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000157257 2023-07-02T14:04:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114280+0000", 2023-07-02T14:04:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000341152 2023-07-02T14:04:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114622+0000", 2023-07-02T14:04:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019539766 2023-07-02T14:04:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.134161+0000", 2023-07-02T14:04:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015820796000000002 2023-07-02T14:04:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149982+0000", 2023-07-02T14:04:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.05e-05 2023-07-02T14:04:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150043+0000", 2023-07-02T14:04:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6453999999999999e-05 2023-07-02T14:04:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150089+0000", 2023-07-02T14:04:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998 2023-07-02T14:04:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3939 2.1 2:9ef34a83:::rbd_data.125de1449605.00000000000000b8:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 80.490273715000001, 2023-07-02T14:04:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001, 2023-07-02T14:04:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3939 2023-07-02T14:04:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:04:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046852830000000002 2023-07-02T14:04:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788763+0000", 2023-07-02T14:04:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3185e-05 2023-07-02T14:04:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788776+0000", 2023-07-02T14:04:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8470000000000003e-06 2023-07-02T14:04:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788781+0000", 2023-07-02T14:04:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7104000000000002e-05 2023-07-02T14:04:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788828+0000", 2023-07-02T14:04:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3516000000000001e-05 2023-07-02T14:04:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788841+0000", 2023-07-02T14:04:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33064513600000001 2023-07-02T14:04:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119487+0000", 2023-07-02T14:04:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9876999999999997e-05 2023-07-02T14:04:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119537+0000", 2023-07-02T14:04:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014160999999999999 2023-07-02T14:04:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119678+0000", 2023-07-02T14:04:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023054392 2023-07-02T14:04:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.142733+0000", 2023-07-02T14:04:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074065470000000003 2023-07-02T14:04:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150139+0000", 2023-07-02T14:04:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5025999999999997e-05 2023-07-02T14:04:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150204+0000", 2023-07-02T14:04:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147999999999997e-05 2023-07-02T14:04:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150241+0000", 2023-07-02T14:04:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001 2023-07-02T14:04:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3940 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 80.485562682999998, 2023-07-02T14:04:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001, 2023-07-02T14:04:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3940 2023-07-02T14:04:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:04:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:04:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788788+0000", 2023-07-02T14:04:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043166560000000003 2023-07-02T14:04:58.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793105+0000", 2023-07-02T14:04:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5970000000000001e-05 2023-07-02T14:04:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:58.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793121+0000", 2023-07-02T14:04:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0939999999999998e-06 2023-07-02T14:04:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793126+0000", 2023-07-02T14:04:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2438000000000002e-05 2023-07-02T14:04:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793178+0000", 2023-07-02T14:04:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2019e-05 2023-07-02T14:04:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:58.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793190+0000", 2023-07-02T14:04:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33208757799999999 2023-07-02T14:04:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125278+0000", 2023-07-02T14:04:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5255999999999997e-05 2023-07-02T14:04:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125373+0000", 2023-07-02T14:04:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015674999999999999 2023-07-02T14:04:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125530+0000", 2023-07-02T14:04:58.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018767308 2023-07-02T14:04:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144297+0000", 2023-07-02T14:04:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060291959999999997 2023-07-02T14:04:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150326+0000", 2023-07-02T14:04:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3886999999999993e-05 2023-07-02T14:04:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150390+0000", 2023-07-02T14:04:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010255 2023-07-02T14:04:58.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150493+0000", 2023-07-02T14:04:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001 2023-07-02T14:04:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3947 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 80.481217156, 2023-07-02T14:04:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999, 2023-07-02T14:04:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:58.428 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3947 2023-07-02T14:04:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:04:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:04:58.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793133+0000", 2023-07-02T14:04:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2519999999999999e-05 2023-07-02T14:04:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:58.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793176+0000", 2023-07-02T14:04:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:04:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793180+0000", 2023-07-02T14:04:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1709999999999998e-06 2023-07-02T14:04:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793184+0000", 2023-07-02T14:04:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1721999999999999e-05 2023-07-02T14:04:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793215+0000", 2023-07-02T14:04:58.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1280000000000008e-06 2023-07-02T14:04:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793224+0000", 2023-07-02T14:04:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34218549599999998 2023-07-02T14:04:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135409+0000", 2023-07-02T14:04:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117442 2023-07-02T14:04:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135526+0000", 2023-07-02T14:04:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000206257 2023-07-02T14:04:58.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135733+0000", 2023-07-02T14:04:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085700029999999996 2023-07-02T14:04:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144303+0000", 2023-07-02T14:04:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006278652 2023-07-02T14:04:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150581+0000", 2023-07-02T14:04:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4765999999999996e-05 2023-07-02T14:04:58.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150646+0000", 2023-07-02T14:04:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4777999999999995e-05 2023-07-02T14:04:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:58.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150711+0000", 2023-07-02T14:04:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999 2023-07-02T14:04:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3980 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 65536~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:58.439 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 80.481163273999996, 2023-07-02T14:04:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999, 2023-07-02T14:04:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:58.440 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3980 2023-07-02T14:04:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:04:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020244260000000002 2023-07-02T14:04:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795212+0000", 2023-07-02T14:04:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0005e-05 2023-07-02T14:04:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795222+0000", 2023-07-02T14:04:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9860000000000002e-06 2023-07-02T14:04:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795227+0000", 2023-07-02T14:04:58.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6369999999999999e-05 2023-07-02T14:04:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795264+0000", 2023-07-02T14:04:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1052e-05 2023-07-02T14:04:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795275+0000", 2023-07-02T14:04:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.343850459 2023-07-02T14:04:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139125+0000", 2023-07-02T14:04:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6760000000000004e-05 2023-07-02T14:04:58.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139202+0000", 2023-07-02T14:04:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015092799999999999 2023-07-02T14:04:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:58.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139353+0000", 2023-07-02T14:04:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011425940000000001 2023-07-02T14:04:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150779+0000", 2023-07-02T14:04:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033223060000000001 2023-07-02T14:04:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154101+0000", 2023-07-02T14:04:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9268999999999998e-05 2023-07-02T14:04:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154161+0000", 2023-07-02T14:04:58.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6499999999999999e-05 2023-07-02T14:04:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154207+0000", 2023-07-02T14:04:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999 2023-07-02T14:04:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3982 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1835008~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 80.479117986999995, 2023-07-02T14:04:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002, 2023-07-02T14:04:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:58.451 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3982 2023-07-02T14:04:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:58.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:04:58.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795233+0000", 2023-07-02T14:04:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8680999999999998e-05 2023-07-02T14:04:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795272+0000", 2023-07-02T14:04:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5629999999999999e-06 2023-07-02T14:04:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795277+0000", 2023-07-02T14:04:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8700000000000001e-06 2023-07-02T14:04:58.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795280+0000", 2023-07-02T14:04:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1964000000000001e-05 2023-07-02T14:04:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795312+0000", 2023-07-02T14:04:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.314e-06 2023-07-02T14:04:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795319+0000", 2023-07-02T14:04:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34828442599999998 2023-07-02T14:04:58.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143603+0000", 2023-07-02T14:04:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2438000000000001e-05 2023-07-02T14:04:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:58.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143665+0000", 2023-07-02T14:04:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3079999999999997e-05 2023-07-02T14:04:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143759+0000", 2023-07-02T14:04:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070633079999999999 2023-07-02T14:04:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150822+0000", 2023-07-02T14:04:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034430210000000001 2023-07-02T14:04:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154265+0000", 2023-07-02T14:04:58.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4249999999999998e-05 2023-07-02T14:04:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154309+0000", 2023-07-02T14:04:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2020000000000001e-05 2023-07-02T14:04:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154351+0000", 2023-07-02T14:04:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002 2023-07-02T14:04:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.462 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4054 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1900544~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 80.479066316000001, 2023-07-02T14:04:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002, 2023-07-02T14:04:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4054 2023-07-02T14:04:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:58.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:04:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018874860000000001 2023-07-02T14:04:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797172+0000", 2023-07-02T14:04:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3000000000000007e-06 2023-07-02T14:04:58.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797182+0000", 2023-07-02T14:04:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6260000000000003e-06 2023-07-02T14:04:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:58.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797186+0000", 2023-07-02T14:04:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9013e-05 2023-07-02T14:04:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797235+0000", 2023-07-02T14:04:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5990000000000008e-06 2023-07-02T14:04:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797245+0000", 2023-07-02T14:04:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34785621999999999 2023-07-02T14:04:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145101+0000", 2023-07-02T14:04:58.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7762000000000006e-05 2023-07-02T14:04:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145169+0000", 2023-07-02T14:04:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013469400000000001 2023-07-02T14:04:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145304+0000", 2023-07-02T14:04:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089538039999999992 2023-07-02T14:04:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154257+0000", 2023-07-02T14:04:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104218 2023-07-02T14:04:58.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:58.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154362+0000", 2023-07-02T14:04:58.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4168000000000001e-05 2023-07-02T14:04:58.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:58.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154406+0000", 2023-07-02T14:04:58.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3098000000000001e-05 2023-07-02T14:04:58.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:58.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154429+0000", 2023-07-02T14:04:58.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002 2023-07-02T14:04:58.473 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4056 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3670016~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 80.477159803000006, 2023-07-02T14:04:58.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997, 2023-07-02T14:04:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4056 2023-07-02T14:04:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.475 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:58.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:04:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797191+0000", 2023-07-02T14:04:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0438999999999997e-05 2023-07-02T14:04:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:58.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797232+0000", 2023-07-02T14:04:58.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6929999999999998e-06 2023-07-02T14:04:58.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:58.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797237+0000", 2023-07-02T14:04:58.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9739999999999998e-06 2023-07-02T14:04:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797240+0000", 2023-07-02T14:04:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0243000000000002e-05 2023-07-02T14:04:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797270+0000", 2023-07-02T14:04:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9889999999999997e-06 2023-07-02T14:04:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797276+0000", 2023-07-02T14:04:58.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35358319799999999 2023-07-02T14:04:58.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150859+0000", 2023-07-02T14:04:58.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1357e-05 2023-07-02T14:04:58.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150940+0000", 2023-07-02T14:04:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014903 2023-07-02T14:04:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151089+0000", 2023-07-02T14:04:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033512519999999999 2023-07-02T14:04:58.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154441+0000", 2023-07-02T14:04:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001050945 2023-07-02T14:04:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:58.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155492+0000", 2023-07-02T14:04:58.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2523e-05 2023-07-02T14:04:58.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:58.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155524+0000", 2023-07-02T14:04:58.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.057e-05 2023-07-02T14:04:58.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155555+0000", 2023-07-02T14:04:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997 2023-07-02T14:04:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4088 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3735552~458752] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 80.477107607999997, 2023-07-02T14:04:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998, 2023-07-02T14:04:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:04:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:58.487 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:58.487 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4088 2023-07-02T14:04:58.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.487 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:58.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:58.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:04:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00038503499999999999 2023-07-02T14:04:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797629+0000", 2023-07-02T14:04:58.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8739999999999995e-06 2023-07-02T14:04:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797637+0000", 2023-07-02T14:04:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7160000000000001e-06 2023-07-02T14:04:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797640+0000", 2023-07-02T14:04:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4342999999999998e-05 2023-07-02T14:04:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797695+0000", 2023-07-02T14:04:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0939999999999996e-06 2023-07-02T14:04:58.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:58.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797701+0000", 2023-07-02T14:04:58.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35408540799999999 2023-07-02T14:04:58.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151786+0000", 2023-07-02T14:04:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1953000000000006e-05 2023-07-02T14:04:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:04:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151868+0000", 2023-07-02T14:04:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015162800000000001 2023-07-02T14:04:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:04:58.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.152020+0000", 2023-07-02T14:04:58.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054561870000000004 2023-07-02T14:04:58.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:04:58.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.157476+0000", 2023-07-02T14:04:58.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059153199999999997 2023-07-02T14:04:58.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:04:58.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158068+0000", 2023-07-02T14:04:58.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2799e-05 2023-07-02T14:04:58.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:04:58.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158090+0000", 2023-07-02T14:04:58.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7235999999999999e-05 2023-07-02T14:04:58.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:58.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158108+0000", 2023-07-02T14:04:58.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998 2023-07-02T14:04:58.496 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.497 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.497 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.497 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4110 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:58.497 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:58.497 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 80.476706331000003, 2023-07-02T14:04:58.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864, 2023-07-02T14:04:58.498 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:58.498 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:58.498 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:58.498 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:58.498 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:58.498 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4110 2023-07-02T14:04:58.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.499 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:58.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:58.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:58.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:58.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:58.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:58.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:58.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:58.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:04:58.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9331e-05 2023-07-02T14:04:58.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:58.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797684+0000", 2023-07-02T14:04:58.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.147e-06 2023-07-02T14:04:58.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:58.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797689+0000", 2023-07-02T14:04:58.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5020000000000001e-06 2023-07-02T14:04:58.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:58.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797693+0000", 2023-07-02T14:04:58.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5254999999999999e-05 2023-07-02T14:04:58.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797748+0000", 2023-07-02T14:04:58.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5520000000000005e-06 2023-07-02T14:04:58.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:58.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797757+0000", 2023-07-02T14:04:58.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35674245700000001 2023-07-02T14:04:58.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154499+0000", 2023-07-02T14:04:58.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0090000000000001e-06 2023-07-02T14:04:58.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:58.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154506+0000", 2023-07-02T14:04:58.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864 2023-07-02T14:04:58.505 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.505 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.505 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.506 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:04:58.506 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:58.506 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 80.476654245000006, 2023-07-02T14:04:58.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998, 2023-07-02T14:04:58.506 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:04:58.506 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:04:58.506 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:04:58.506 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:04:58.507 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:04:58.507 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:04:58.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.507 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:04:58.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:04:58.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:58.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:04:58.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:04:58.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:58.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:04:58.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:04:58.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:04:58.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001714819 2023-07-02T14:04:58.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:04:58.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799412+0000", 2023-07-02T14:04:58.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7779999999999996e-06 2023-07-02T14:04:58.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:04:58.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799421+0000", 2023-07-02T14:04:58.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3510000000000002e-06 2023-07-02T14:04:58.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:04:58.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799424+0000", 2023-07-02T14:04:58.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0275999999999999e-05 2023-07-02T14:04:58.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799464+0000", 2023-07-02T14:04:58.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8389999999999996e-06 2023-07-02T14:04:58.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:04:58.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799473+0000", 2023-07-02T14:04:58.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35507638800000002 2023-07-02T14:04:58.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:04:58.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154550+0000", 2023-07-02T14:04:58.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6309999999999998e-06 2023-07-02T14:04:58.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:04:58.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:04:58.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:04:58.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154559+0000", 2023-07-02T14:04:58.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998 2023-07-02T14:04:58.513 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.513 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.514 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.514 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:04:58.514 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:04:58.514 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:58.514 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:04:58.515 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:04:58.400+0000 7fb3991ba200 -1 Falling back to public interface 2023-07-02T14:04:58.516 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:58.514+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:58.517 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:04:58.616 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:58.615+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:58.686 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:04:58.686 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:04:58.686 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:04:58.686 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:04:58.696 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:04:58.699 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:04:58.716 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:58.715+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:58.717 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:04:58.735 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:04:58.758 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:04:58.777 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:04:58.777 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:04:58.777 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:04:58.777 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:04:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:04:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:04:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:04:58.780 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:04:58.781 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:04:58.781 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:04:58.786 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 2 2023-07-02T14:04:58.823 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:58.822+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:58.829 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:58.829 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:58.839 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:04:58.917 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:58.916+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:58.950 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:04:59.006 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:59.007 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:59.017 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:04:59.018 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:59.017+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:59.118 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:59.117+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:59.155 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:59.155 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:59.166 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:04:59.218 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:04:59.217+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:04:59.289 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:59.290 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:59.300 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:04:59.342 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:04:59.342+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:04:59.419 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:04:59.419+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:04:59.423 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:59.423 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:59.433 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:04:59.520 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:59.519+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:59.597 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:59.597 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:59.607 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:04:59.621 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:59.620+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:59.722 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:59.721+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:04:59.730 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:59.730 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:59.740 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:04:59.745 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:04:59.822 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:04:59.821+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:04:59.863 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:04:59.863 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:04:59.873 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:04:59.923 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:04:59.922+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:00.001 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:00.001 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:00.011 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:05:00.052 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:00.051+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:00.124 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:00.123+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:00.174 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:00.174 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:00.184 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:05:00.225 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:00.224+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:00.326 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:00.325+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:00.348 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:00.348 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:00.360 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:05:00.443 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:00.442+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:00.524 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:00.524 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:00.527 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:00.526+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:00.535 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:05:00.628 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:00.627+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:00.699 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:00.699 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:00.709 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:05:00.750 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:00.749+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:00.782 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:05:00.788 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap31 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap32 b3fbb028-e267-4d25-a750-93ba18a2316a@snap32.diff 2023-07-02T14:05:00.828 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:00.828+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:00.842 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 100% complete...done. 2023-07-02T14:05:00.845 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:05:00.845 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=33 2023-07-02T14:05:00.845 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap33 2023-07-02T14:05:00.877 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:00.877 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:00.889 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:05:00.932 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:00.931+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:00.935 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:05:00.953 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:00.972 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:00.991 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:01.009 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:01.009 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:05:01.010 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:05:01.010 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:05:01.010 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:05:01.010 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:05:01.010 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:05:01.012 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:05:01.013 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:05:01.017 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:01.018 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:01.018 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:05:01.018 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:01.028 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:05:01.029 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:01.028+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:01.130 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:01.129+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:01.191 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:05:01.192 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:05:01.192 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:01.202 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:05:01.245 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:01.244+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:01.327 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:01.327 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:05:01.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:05:01.328 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:01.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.328 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:05:01.328 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:01.328 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 120.217909754, 2023-07-02T14:05:01.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:05:01.329 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:01.329 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:01.329 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:01.329 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:01.329 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:01.329 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:05:01.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.330 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:01.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:01.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:01.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:01.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:01.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:01.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:01.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:01.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:05:01.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:05:01.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:01.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:05:01.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:05:01.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:01.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:05:01.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:05:01.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:01.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:05:01.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:05:01.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:05:01.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:05:01.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:01.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:05:01.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:05:01.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:05:01.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:05:01.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:01.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:05:01.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:05:01.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:01.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:05:01.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:05:01.337 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.337 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.337 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.337 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:05:01.337 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:01.338 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 120.217849016, 2023-07-02T14:05:01.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:05:01.338 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:01.338 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:01.338 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:01.338 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:01.338 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:01.338 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:05:01.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.339 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:01.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:01.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:01.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:01.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:01.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:01.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:01.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:01.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:05:01.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:05:01.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:01.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:05:01.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:05:01.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:01.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:05:01.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:05:01.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:01.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:05:01.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:05:01.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:05:01.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:05:01.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:01.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:05:01.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:05:01.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:05:01.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:05:01.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:01.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:05:01.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:05:01.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:01.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:05:01.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:05:01.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:01.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:05:01.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:05:01.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:01.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:05:01.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:05:01.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:01.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:05:01.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:05:01.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:01.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:05:01.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:05:01.348 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.348 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.349 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.349 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:01.349 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:01.349 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 109.987394686, 2023-07-02T14:05:01.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:05:01.349 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:01.350 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:01.350 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:01.350 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:01.350 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:01.350 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:05:01.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.350 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:01.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:01.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:01.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:01.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:01.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:01.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:01.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:01.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:05:01.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:05:01.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:01.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:05:01.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:05:01.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:01.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:05:01.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:05:01.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:01.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:05:01.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:05:01.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:05:01.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:05:01.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:01.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:05:01.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:05:01.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:01.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:05:01.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:05:01.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:01.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:05:01.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:05:01.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:01.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:05:01.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:05:01.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:01.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:05:01.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:05:01.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:01.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:05:01.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:05:01.359 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.359 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.359 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.359 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2950 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 1507328~786432] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:01.359 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:01.360 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 109.850118424, 2023-07-02T14:05:01.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999, 2023-07-02T14:05:01.360 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:01.360 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:01.360 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:01.360 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:01.360 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:01.361 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2950 2023-07-02T14:05:01.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.361 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:01.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:01.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:01.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:01.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:01.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:01.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:01.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:01.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:01.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001147854 2023-07-02T14:05:01.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:01.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476614+0000", 2023-07-02T14:05:01.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.802e-05 2023-07-02T14:05:01.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:01.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476632+0000", 2023-07-02T14:05:01.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1860000000000003e-06 2023-07-02T14:05:01.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:01.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476639+0000", 2023-07-02T14:05:01.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061211240000000004 2023-07-02T14:05:01.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482760+0000", 2023-07-02T14:05:01.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8833999999999996e-05 2023-07-02T14:05:01.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:01.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482859+0000", 2023-07-02T14:05:01.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019270600000000001 2023-07-02T14:05:01.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:01.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.483051+0000", 2023-07-02T14:05:01.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060385630000000003 2023-07-02T14:05:01.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:01.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489090+0000", 2023-07-02T14:05:01.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024506918999999999 2023-07-02T14:05:01.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:01.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513597+0000", 2023-07-02T14:05:01.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9672999999999995e-05 2023-07-02T14:05:01.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:01.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513667+0000", 2023-07-02T14:05:01.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024329500000000001 2023-07-02T14:05:01.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:01.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513910+0000", 2023-07-02T14:05:01.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999 2023-07-02T14:05:01.369 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.369 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.369 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.370 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2951 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2686976~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:01.370 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:05:01.370 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 109.848937646, 2023-07-02T14:05:01.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999, 2023-07-02T14:05:01.370 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:01.370 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:01.370 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:01.371 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:01.371 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:01.371 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2951 2023-07-02T14:05:01.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.371 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:01.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:01.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:05:01.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:01.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:01.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:05:01.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:01.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:01.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476646+0000", 2023-07-02T14:05:01.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7639000000000003e-05 2023-07-02T14:05:01.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:01.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476684+0000", 2023-07-02T14:05:01.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3508000000000002e-05 2023-07-02T14:05:01.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:01.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476747+0000", 2023-07-02T14:05:01.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8499999999999994e-06 2023-07-02T14:05:01.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:01.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476755+0000", 2023-07-02T14:05:01.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078346160000000008 2023-07-02T14:05:01.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484590+0000", 2023-07-02T14:05:01.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7206000000000004e-05 2023-07-02T14:05:01.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:01.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484687+0000", 2023-07-02T14:05:01.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016243099999999999 2023-07-02T14:05:01.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:01.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484849+0000", 2023-07-02T14:05:01.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042482860000000004 2023-07-02T14:05:01.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:01.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489097+0000", 2023-07-02T14:05:01.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.02489504 2023-07-02T14:05:01.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:01.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513992+0000", 2023-07-02T14:05:01.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9972000000000001e-05 2023-07-02T14:05:01.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:01.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514052+0000", 2023-07-02T14:05:01.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149264 2023-07-02T14:05:01.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:01.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514202+0000", 2023-07-02T14:05:01.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999 2023-07-02T14:05:01.381 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.381 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.381 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.381 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2952 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2883584~12288] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:01.381 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:01.382 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 109.848823811, 2023-07-02T14:05:01.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651, 2023-07-02T14:05:01.382 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:01.382 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:01.382 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:01.382 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:01.382 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:01.382 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2952 2023-07-02T14:05:01.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.383 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:01.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:01.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:01.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:01.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:01.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:01.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:01.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:01.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476760+0000", 2023-07-02T14:05:01.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3664e-05 2023-07-02T14:05:01.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:01.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476774+0000", 2023-07-02T14:05:01.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.166e-06 2023-07-02T14:05:01.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:01.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476783+0000", 2023-07-02T14:05:01.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8720000000000001e-06 2023-07-02T14:05:01.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:01.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476788+0000", 2023-07-02T14:05:01.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087180529999999999 2023-07-02T14:05:01.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485506+0000", 2023-07-02T14:05:01.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5680999999999994e-05 2023-07-02T14:05:01.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:01.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485602+0000", 2023-07-02T14:05:01.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000222365 2023-07-02T14:05:01.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:01.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485824+0000", 2023-07-02T14:05:01.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003277379 2023-07-02T14:05:01.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:01.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489101+0000", 2023-07-02T14:05:01.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025151739999999999 2023-07-02T14:05:01.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:01.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514253+0000", 2023-07-02T14:05:01.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4749000000000002e-05 2023-07-02T14:05:01.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:01.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514288+0000", 2023-07-02T14:05:01.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130634 2023-07-02T14:05:01.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:01.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514419+0000", 2023-07-02T14:05:01.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651 2023-07-02T14:05:01.392 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.392 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.392 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.392 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9620 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 589824~65536] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:05:01.392 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:01.393 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.303380087999997, 2023-07-02T14:05:01.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001, 2023-07-02T14:05:01.393 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:01.393 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:01.393 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:01.393 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:01.393 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:01.393 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9620 2023-07-02T14:05:01.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.394 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:01.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:01.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:01.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:01.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:01.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:01.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:01.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:01.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022203+0000", 2023-07-02T14:05:01.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0346000000000001e-05 2023-07-02T14:05:01.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:01.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022263+0000", 2023-07-02T14:05:01.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4657999999999999e-05 2023-07-02T14:05:01.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:01.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022278+0000", 2023-07-02T14:05:01.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:05:01.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:01.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022285+0000", 2023-07-02T14:05:01.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0096000000000003e-05 2023-07-02T14:05:01.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022335+0000", 2023-07-02T14:05:01.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2334000000000001e-05 2023-07-02T14:05:01.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:01.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022347+0000", 2023-07-02T14:05:01.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095653701999999993 2023-07-02T14:05:01.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118001+0000", 2023-07-02T14:05:01.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000185797 2023-07-02T14:05:01.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:01.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118187+0000", 2023-07-02T14:05:01.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026397900000000001 2023-07-02T14:05:01.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:01.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118451+0000", 2023-07-02T14:05:01.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051832800000000002 2023-07-02T14:05:01.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:01.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.123634+0000", 2023-07-02T14:05:01.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001227332 2023-07-02T14:05:01.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:01.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124861+0000", 2023-07-02T14:05:01.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9493999999999998e-05 2023-07-02T14:05:01.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:01.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124901+0000", 2023-07-02T14:05:01.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0462000000000001e-05 2023-07-02T14:05:01.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:01.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124931+0000", 2023-07-02T14:05:01.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001 2023-07-02T14:05:01.404 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.404 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.404 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.405 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9621 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 655360~1966080] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:05:01.405 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:01.405 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.303192705999997, 2023-07-02T14:05:01.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999, 2023-07-02T14:05:01.405 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:01.405 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:01.405 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:01.406 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:01.406 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:01.406 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9621 2023-07-02T14:05:01.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.407 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:01.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:01.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:01.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:01.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:01.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:01.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:01.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:01.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022391+0000", 2023-07-02T14:05:01.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032053939999999999 2023-07-02T14:05:01.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:01.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025597+0000", 2023-07-02T14:05:01.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4307999999999999e-05 2023-07-02T14:05:01.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:01.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025611+0000", 2023-07-02T14:05:01.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.032e-06 2023-07-02T14:05:01.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:01.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025617+0000", 2023-07-02T14:05:01.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8977999999999998e-05 2023-07-02T14:05:01.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025656+0000", 2023-07-02T14:05:01.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:05:01.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:01.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025663+0000", 2023-07-02T14:05:01.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095220739999999998 2023-07-02T14:05:01.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120884+0000", 2023-07-02T14:05:01.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2525000000000001e-05 2023-07-02T14:05:01.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:01.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120956+0000", 2023-07-02T14:05:01.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016232400000000001 2023-07-02T14:05:01.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:01.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.121119+0000", 2023-07-02T14:05:01.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044007380000000004 2023-07-02T14:05:01.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:01.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.125519+0000", 2023-07-02T14:05:01.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000797209 2023-07-02T14:05:01.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:01.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126317+0000", 2023-07-02T14:05:01.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1228000000000004e-05 2023-07-02T14:05:01.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:01.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126378+0000", 2023-07-02T14:05:01.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978000000000003e-05 2023-07-02T14:05:01.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:01.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126430+0000", 2023-07-02T14:05:01.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999 2023-07-02T14:05:01.418 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.418 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.418 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.419 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:11434 2.1 2:8269d003:::rbd_data.125de1449605.00000000000000ce:head [write 65536~4128768] snapc 1a=[1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e107)", 2023-07-02T14:05:01.419 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:05:01.419 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.407754437000001, 2023-07-02T14:05:01.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036409966000000002, 2023-07-02T14:05:01.419 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:01.419 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:01.420 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:01.420 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:01.420 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:01.420 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11434 2023-07-02T14:05:01.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.420 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:01.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:01.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:05:01.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:01.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:01.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:05:01.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:01.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:01.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917828+0000", 2023-07-02T14:05:01.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052044650000000001 2023-07-02T14:05:01.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:01.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923033+0000", 2023-07-02T14:05:01.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0097000000000001e-05 2023-07-02T14:05:01.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:01.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923053+0000", 2023-07-02T14:05:01.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.49e-06 2023-07-02T14:05:01.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:01.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923061+0000", 2023-07-02T14:05:01.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0090999999999998e-05 2023-07-02T14:05:01.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923121+0000", 2023-07-02T14:05:01.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000114095 2023-07-02T14:05:01.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:01.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923236+0000", 2023-07-02T14:05:01.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020094199999999999 2023-07-02T14:05:01.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:01.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923436+0000", 2023-07-02T14:05:01.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088693520000000005 2023-07-02T14:05:01.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:01.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.932306+0000", 2023-07-02T14:05:01.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021874112000000001 2023-07-02T14:05:01.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:01.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954180+0000", 2023-07-02T14:05:01.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4433e-05 2023-07-02T14:05:01.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:01.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954214+0000", 2023-07-02T14:05:01.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5884000000000002e-05 2023-07-02T14:05:01.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:01.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954240+0000", 2023-07-02T14:05:01.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036409966000000002 2023-07-02T14:05:01.429 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.429 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.429 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.429 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:189 2.3 2:c9621172:::rbd_data.125de1449605.00000000000000c0:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:01.430 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:01.430 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 28.152905413999999, 2023-07-02T14:05:01.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001, 2023-07-02T14:05:01.430 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:01.430 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:01.430 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:01.430 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:01.431 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:01.431 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:05:01.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.431 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:01.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:01.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:01.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:01.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:01.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:01.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:01.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:01.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:01.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2023-07-02T14:05:01.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:01.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172682+0000", 2023-07-02T14:05:01.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0159999999999997e-06 2023-07-02T14:05:01.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:01.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172688+0000", 2023-07-02T14:05:01.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0010000000000003e-06 2023-07-02T14:05:01.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:01.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172692+0000", 2023-07-02T14:05:01.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026444333 2023-07-02T14:05:01.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199136+0000", 2023-07-02T14:05:01.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2230000000000004e-05 2023-07-02T14:05:01.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:01.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199218+0000", 2023-07-02T14:05:01.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012043595000000001 2023-07-02T14:05:01.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:01.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211262+0000", 2023-07-02T14:05:01.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001 2023-07-02T14:05:01.437 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.437 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.437 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.437 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:199 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:01.438 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:01.438 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 28.152861870999999, 2023-07-02T14:05:01.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002, 2023-07-02T14:05:01.438 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:01.438 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:01.438 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:01.438 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:01.438 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:01.439 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 199 2023-07-02T14:05:01.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.439 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:01.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:01.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:01.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:01.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:01.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:01.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:01.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:01.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172722+0000", 2023-07-02T14:05:01.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1250000000000001e-06 2023-07-02T14:05:01.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:01.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172725+0000", 2023-07-02T14:05:01.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5670000000000002e-06 2023-07-02T14:05:01.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:01.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172731+0000", 2023-07-02T14:05:01.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9310000000000001e-06 2023-07-02T14:05:01.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:01.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172735+0000", 2023-07-02T14:05:01.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030699852999999999 2023-07-02T14:05:01.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203435+0000", 2023-07-02T14:05:01.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6827e-05 2023-07-02T14:05:01.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:01.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203532+0000", 2023-07-02T14:05:01.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076955230000000001 2023-07-02T14:05:01.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:01.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211227+0000", 2023-07-02T14:05:01.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002 2023-07-02T14:05:01.445 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.445 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.445 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.445 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:281 2.1 2:901d5a95:::rbd_data.125de1449605.0000000000000118:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:01.445 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:01.445 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.770116595000001, 2023-07-02T14:05:01.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002, 2023-07-02T14:05:01.446 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:01.446 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:01.446 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:01.446 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:01.446 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:01.446 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:05:01.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.447 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:01.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:01.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:01.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:01.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:01.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:01.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:01.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:01.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:01.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0469999999999998e-06 2023-07-02T14:05:01.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555471+0000", 2023-07-02T14:05:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8860000000000001e-06 2023-07-02T14:05:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555476+0000", 2023-07-02T14:05:01.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3160000000000002e-06 2023-07-02T14:05:01.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:01.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555483+0000", 2023-07-02T14:05:01.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989525000000001 2023-07-02T14:05:01.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591472+0000", 2023-07-02T14:05:01.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3708999999999996e-05 2023-07-02T14:05:01.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:01.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591536+0000", 2023-07-02T14:05:01.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055109720000000003 2023-07-02T14:05:01.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:01.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.597047+0000", 2023-07-02T14:05:01.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002 2023-07-02T14:05:01.452 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.452 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.452 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.453 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12536 2.1 2:8b20f686:::rbd_data.125de1449605.0000000000000056:head [write 3670016~36864] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:05:01.453 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.205749+0000", 2023-07-02T14:05:01.453 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.1198356310000008, 2023-07-02T14:05:01.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036263036999999998, 2023-07-02T14:05:01.453 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:01.454 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:01.454 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:01.454 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:01.454 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:01.454 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12536 2023-07-02T14:05:01.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.454 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:01.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:01.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205749+0000", 2023-07-02T14:05:01.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:01.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:01.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205749+0000", 2023-07-02T14:05:01.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:01.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:01.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205748+0000", 2023-07-02T14:05:01.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1784999999999999e-05 2023-07-02T14:05:01.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:01.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205770+0000", 2023-07-02T14:05:01.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0659999999999993e-06 2023-07-02T14:05:01.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:01.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205779+0000", 2023-07-02T14:05:01.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.622e-06 2023-07-02T14:05:01.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:01.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205784+0000", 2023-07-02T14:05:01.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018333009000000001 2023-07-02T14:05:01.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.224117+0000", 2023-07-02T14:05:01.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018818400000000001 2023-07-02T14:05:01.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:01.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.224305+0000", 2023-07-02T14:05:01.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00032545700000000002 2023-07-02T14:05:01.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:01.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.224631+0000", 2023-07-02T14:05:01.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049349700000000003 2023-07-02T14:05:01.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:01.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.229566+0000", 2023-07-02T14:05:01.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012353153 2023-07-02T14:05:01.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:01.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.241919+0000", 2023-07-02T14:05:01.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2751999999999999e-05 2023-07-02T14:05:01.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:01.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.241971+0000", 2023-07-02T14:05:01.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0632e-05 2023-07-02T14:05:01.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:01.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.242012+0000", 2023-07-02T14:05:01.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036263036999999998 2023-07-02T14:05:01.463 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.463 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.463 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.463 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12883 2.1 2:9ffc48e1:::rbd_data.125de1449605.0000000000000074:head [write 4173824~20480] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:05:01.463 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:01.464 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.6795141220000005, 2023-07-02T14:05:01.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037871393000000003, 2023-07-02T14:05:01.464 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:01.464 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:01.464 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:01.464 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:01.464 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:01.464 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12883 2023-07-02T14:05:01.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.465 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:01.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:01.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:01.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:01.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:01.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:01.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:05:01.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:01.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646068+0000", 2023-07-02T14:05:01.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4935999999999999e-05 2023-07-02T14:05:01.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:01.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646103+0000", 2023-07-02T14:05:01.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.092e-05 2023-07-02T14:05:01.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:01.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646124+0000", 2023-07-02T14:05:01.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1519999999999998e-06 2023-07-02T14:05:01.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:01.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646133+0000", 2023-07-02T14:05:01.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012270200000000001 2023-07-02T14:05:01.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646255+0000", 2023-07-02T14:05:01.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014495400000000001 2023-07-02T14:05:01.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:01.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646400+0000", 2023-07-02T14:05:01.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000221626 2023-07-02T14:05:01.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:01.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646622+0000", 2023-07-02T14:05:01.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078929050000000004 2023-07-02T14:05:01.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:01.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.654515+0000", 2023-07-02T14:05:01.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029307343999999999 2023-07-02T14:05:01.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:01.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683822+0000", 2023-07-02T14:05:01.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8298000000000002e-05 2023-07-02T14:05:01.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:01.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683891+0000", 2023-07-02T14:05:01.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1381999999999997e-05 2023-07-02T14:05:01.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:01.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683942+0000", 2023-07-02T14:05:01.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037871393000000003 2023-07-02T14:05:01.473 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.473 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.473 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.474 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12985 2.7 2:eeddf24b:::rbd_data.125de1449605.000000000000007e:head [write 65536~4128768] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:05:01.474 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:01.474 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.542589649, 2023-07-02T14:05:01.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999, 2023-07-02T14:05:01.474 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:01.474 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:01.475 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:01.475 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:01.475 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:01.475 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12985 2023-07-02T14:05:01.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.475 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:01.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:01.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:01.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:01.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:01.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:01.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:01.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:01.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782994+0000", 2023-07-02T14:05:01.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019254741999999998 2023-07-02T14:05:01.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:01.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802248+0000", 2023-07-02T14:05:01.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0451e-05 2023-07-02T14:05:01.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:01.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802279+0000", 2023-07-02T14:05:01.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3318e-05 2023-07-02T14:05:01.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:01.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802292+0000", 2023-07-02T14:05:01.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109403 2023-07-02T14:05:01.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802402+0000", 2023-07-02T14:05:01.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012520599999999999 2023-07-02T14:05:01.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:01.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802527+0000", 2023-07-02T14:05:01.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187768 2023-07-02T14:05:01.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:05:01.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802715+0000", 2023-07-02T14:05:01.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013192822 2023-07-02T14:05:01.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:01.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.815907+0000", 2023-07-02T14:05:01.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0086451870000000004 2023-07-02T14:05:01.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:01.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824553+0000", 2023-07-02T14:05:01.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6759999999999999e-05 2023-07-02T14:05:01.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:01.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824609+0000", 2023-07-02T14:05:01.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011314700000000001 2023-07-02T14:05:01.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:01.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824722+0000", 2023-07-02T14:05:01.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999 2023-07-02T14:05:01.484 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.484 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.484 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.484 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:165 2.3 2:c9c6393a:::rbd_data.125de1449605.000000000000002c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:01.484 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:01.484 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.4858068319999997, 2023-07-02T14:05:01.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002, 2023-07-02T14:05:01.485 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:01.485 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:01.485 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:01.485 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:01.485 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:01.485 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 165 2023-07-02T14:05:01.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.486 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:01.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:01.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:01.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:01.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:01.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:01.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:01.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:01.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839777+0000", 2023-07-02T14:05:01.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6740000000000001e-06 2023-07-02T14:05:01.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:01.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839780+0000", 2023-07-02T14:05:01.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9030000000000004e-06 2023-07-02T14:05:01.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:01.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839788+0000", 2023-07-02T14:05:01.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1799999999999998e-06 2023-07-02T14:05:01.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:01.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839792+0000", 2023-07-02T14:05:01.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.031592227000000001 2023-07-02T14:05:01.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871384+0000", 2023-07-02T14:05:01.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013600099999999999 2023-07-02T14:05:01.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:01.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871520+0000", 2023-07-02T14:05:01.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082817089999999999 2023-07-02T14:05:01.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:01.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.879802+0000", 2023-07-02T14:05:01.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002 2023-07-02T14:05:01.491 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.491 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.491 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.492 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:252 2.7 2:f1d0abf0:::rbd_data.125de1449605.0000000000000083:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:01.492 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:01.492 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.0695218620000002, 2023-07-02T14:05:01.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997, 2023-07-02T14:05:01.492 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:01.493 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:01.493 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:01.493 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:01.493 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:01.493 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 252 2023-07-02T14:05:01.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.493 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:01.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:01.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:01.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:01.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:01.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:01.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:01.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:01.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256062+0000", 2023-07-02T14:05:01.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0149999999999999e-06 2023-07-02T14:05:01.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:01.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256065+0000", 2023-07-02T14:05:01.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5269999999999998e-06 2023-07-02T14:05:01.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:01.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256071+0000", 2023-07-02T14:05:01.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:05:01.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:01.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256074+0000", 2023-07-02T14:05:01.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028852909 2023-07-02T14:05:01.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.284927+0000", 2023-07-02T14:05:01.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168274 2023-07-02T14:05:01.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:01.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.285096+0000", 2023-07-02T14:05:01.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011036300000000001 2023-07-02T14:05:01.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:01.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296132+0000", 2023-07-02T14:05:01.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997 2023-07-02T14:05:01.499 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.499 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.499 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.500 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:261 2.1 2:971731cb:::rbd_data.125de1449605.000000000000008c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:01.500 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:01.500 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.069426387, 2023-07-02T14:05:01.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984, 2023-07-02T14:05:01.500 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:01.500 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:01.500 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:01.501 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:01.501 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:01.501 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 261 2023-07-02T14:05:01.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.501 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:01.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:01.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:01.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:01.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:01.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:01.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:01.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:01.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:01.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2819999999999999e-06 2023-07-02T14:05:01.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:01.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256160+0000", 2023-07-02T14:05:01.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2020000000000003e-06 2023-07-02T14:05:01.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:01.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256165+0000", 2023-07-02T14:05:01.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2100000000000003e-06 2023-07-02T14:05:01.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:01.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256170+0000", 2023-07-02T14:05:01.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025871715999999999 2023-07-02T14:05:01.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282041+0000", 2023-07-02T14:05:01.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132364 2023-07-02T14:05:01.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:01.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282174+0000", 2023-07-02T14:05:01.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014044566 2023-07-02T14:05:01.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:01.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296218+0000", 2023-07-02T14:05:01.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984 2023-07-02T14:05:01.507 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.507 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.507 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.507 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:264 2.3 2:dbf06c9d:::rbd_data.125de1449605.000000000000008f:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:01.508 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:01.508 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.0693922689999997, 2023-07-02T14:05:01.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003, 2023-07-02T14:05:01.508 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:01.508 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:01.508 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:01.508 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:01.508 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:01.509 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 264 2023-07-02T14:05:01.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.509 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:01.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:01.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:01.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:01.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:01.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:01.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:01.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:01.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:01.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5749999999999999e-06 2023-07-02T14:05:01.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:01.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256195+0000", 2023-07-02T14:05:01.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.327e-06 2023-07-02T14:05:01.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:01.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256200+0000", 2023-07-02T14:05:01.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0580000000000002e-06 2023-07-02T14:05:01.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:01.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256203+0000", 2023-07-02T14:05:01.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035707524999999997 2023-07-02T14:05:01.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.291911+0000", 2023-07-02T14:05:01.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014269799999999999 2023-07-02T14:05:01.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:01.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.292053+0000", 2023-07-02T14:05:01.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062760309999999996 2023-07-02T14:05:01.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:01.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.298329+0000", 2023-07-02T14:05:01.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003 2023-07-02T14:05:01.515 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.515 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.515 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.515 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:341 2.3 2:cee258bd:::rbd_data.125de1449605.00000000000000dc:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:01.515 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:01.515 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.6889049420000002, 2023-07-02T14:05:01.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997, 2023-07-02T14:05:01.516 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:01.516 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:01.516 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:01.516 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:01.516 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:01.516 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 341 2023-07-02T14:05:01.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.517 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:01.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:01.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:01.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:01.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:01.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:01.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:01.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:01.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636679+0000", 2023-07-02T14:05:01.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6240000000000002e-06 2023-07-02T14:05:01.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636682+0000", 2023-07-02T14:05:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6910000000000002e-06 2023-07-02T14:05:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636688+0000", 2023-07-02T14:05:01.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3469999999999999e-06 2023-07-02T14:05:01.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:01.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636691+0000", 2023-07-02T14:05:01.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035625560000000001 2023-07-02T14:05:01.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:01.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672317+0000", 2023-07-02T14:05:01.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108391 2023-07-02T14:05:01.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:01.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672425+0000", 2023-07-02T14:05:01.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012104976 2023-07-02T14:05:01.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:01.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:01.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:01.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.684530+0000", 2023-07-02T14:05:01.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997 2023-07-02T14:05:01.522 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.522 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.523 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.523 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:01.523 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:01.523 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:01.524 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:05:01.525 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:01.523+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:01.625 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:01.624+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:01.651 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:01.651 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:01.652 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:01.652 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:01.661 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:05:01.725 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:01.725+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:01.759 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:05:01.784 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:05:01.790 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 4 2023-07-02T14:05:01.826 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:01.826+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:01.827 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:01.827 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:01.827 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:01.828 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:01.828 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:01.837 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:05:01.927 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:01.926+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:02.003 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:02.003 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:02.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:02.004 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:02.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.004 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:05:02.004 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:02.004 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 82.104110473000006, 2023-07-02T14:05:02.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:05:02.005 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.005 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.005 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:02.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:02.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:05:02.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:05:02.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:05:02.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:05:02.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:05:02.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:05:02.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:05:02.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:05:02.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:05:02.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:05:02.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:05:02.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:05:02.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:05:02.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:05:02.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:05:02.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:05:02.011 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.011 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.012 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.012 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:05:02.012 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:02.012 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 82.100024755000007, 2023-07-02T14:05:02.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:05:02.012 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.013 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.013 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:02.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:02.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:05:02.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:05:02.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:05:02.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:05:02.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:05:02.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:05:02.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:05:02.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:05:02.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:05:02.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:05:02.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:05:02.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:05:02.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:05:02.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:05:02.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:05:02.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:05:02.019 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.019 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.019 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.020 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:05:02.020 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:02.020 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 82.096159974000003, 2023-07-02T14:05:02.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:05:02.020 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.020 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.021 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:02.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:02.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:02.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:05:02.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:05:02.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:05:02.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:05:02.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:05:02.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:05:02.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:05:02.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:05:02.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:05:02.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:05:02.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:05:02.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:05:02.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:05:02.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:05:02.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:05:02.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:05:02.027 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.027 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.027 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.027 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:05:02.028 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:02.028 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 82.092635713000007, 2023-07-02T14:05:02.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:05:02.028 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.028 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.028 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:02.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:02.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:05:02.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:05:02.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:05:02.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:05:02.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:05:02.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:05:02.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:05:02.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:05:02.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:05:02.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:05:02.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:05:02.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:05:02.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:05:02.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:05:02.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:05:02.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:05:02.035 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.035 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.035 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.035 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:05:02.036 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:02.036 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 82.056930914000006, 2023-07-02T14:05:02.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:05:02.036 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.036 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.036 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:02.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:02.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:02.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:05:02.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:05:02.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:05:02.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:05:02.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:05:02.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:05:02.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:05:02.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:05:02.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:05:02.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:05:02.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:05:02.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:05:02.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:05:02.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:05:02.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:05:02.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:05:02.042 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.043 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.043 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.043 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:05:02.043 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:02.043 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 82.052273556000003, 2023-07-02T14:05:02.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:05:02.044 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.044 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.044 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:02.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:02.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:05:02.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:05:02.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:05:02.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:05:02.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:05:02.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:05:02.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:05:02.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:05:02.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:05:02.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:05:02.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:05:02.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:05:02.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:05:02.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:05:02.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:05:02.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:05:02.050 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.051 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.051 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.051 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:05:02.051 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:02.051 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 82.051333693000004, 2023-07-02T14:05:02.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:05:02.052 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.052 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.052 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:02.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:02.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:05:02.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:05:02.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:05:02.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:05:02.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:05:02.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:05:02.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:05:02.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:05:02.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:05:02.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:05:02.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:05:02.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:05:02.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:05:02.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:05:02.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:05:02.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:05:02.058 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.058 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.058 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.059 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:05:02.059 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:02.059 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 82.049935814999998, 2023-07-02T14:05:02.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:05:02.059 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.059 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.060 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:02.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:02.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:02.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:05:02.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:05:02.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:05:02.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:05:02.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:05:02.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:05:02.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:05:02.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:05:02.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:05:02.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:05:02.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:05:02.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:05:02.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:05:02.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:05:02.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:05:02.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:05:02.066 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.066 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.066 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.067 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:05:02.067 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:02.067 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.639745318999999, 2023-07-02T14:05:02.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:05:02.067 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.067 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.067 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:02.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:02.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:05:02.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:05:02.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:05:02.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:05:02.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:05:02.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:05:02.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:05:02.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:05:02.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:05:02.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:05:02.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:05:02.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:05:02.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:05:02.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:05:02.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:05:02.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:05:02.074 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.074 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.074 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.074 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:02.074 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:02.075 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 59.273395635, 2023-07-02T14:05:02.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:05:02.075 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.075 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.075 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.075 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.075 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.075 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:05:02.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.076 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:02.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:02.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:05:02.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:05:02.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:05:02.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:05:02.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:05:02.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:05:02.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:05:02.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:05:02.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:05:02.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:05:02.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:05:02.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:05:02.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:05:02.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:05:02.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:05:02.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:05:02.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:05:02.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:05:02.083 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.083 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.083 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.083 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7707 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 3145728~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:02.084 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:02.084 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 59.269412168000002, 2023-07-02T14:05:02.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999, 2023-07-02T14:05:02.084 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.084 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.084 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.084 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.084 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.085 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7707 2023-07-02T14:05:02.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.085 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:02.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:02.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:02.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5017e-05 2023-07-02T14:05:02.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732347+0000", 2023-07-02T14:05:02.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5821e-05 2023-07-02T14:05:02.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732363+0000", 2023-07-02T14:05:02.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1270000000000002e-06 2023-07-02T14:05:02.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732368+0000", 2023-07-02T14:05:02.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008444666 2023-07-02T14:05:02.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740812+0000", 2023-07-02T14:05:02.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011266599999999999 2023-07-02T14:05:02.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740925+0000", 2023-07-02T14:05:02.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010848808999999999 2023-07-02T14:05:02.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751774+0000", 2023-07-02T14:05:02.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8421999999999997e-05 2023-07-02T14:05:02.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751832+0000", 2023-07-02T14:05:02.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107453 2023-07-02T14:05:02.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751940+0000", 2023-07-02T14:05:02.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999 2023-07-02T14:05:02.092 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.092 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.092 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.093 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:02.093 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:02.093 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 59.269348291, 2023-07-02T14:05:02.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:05:02.093 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.093 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.093 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.093 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.094 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.094 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:05:02.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.094 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:02.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:02.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:02.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:02.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:05:02.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:05:02.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:05:02.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:05:02.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:05:02.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:05:02.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:05:02.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:05:02.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:05:02.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:05:02.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:05:02.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:05:02.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:05:02.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:05:02.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:05:02.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:05:02.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:05:02.101 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.101 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.101 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.102 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:02.102 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:02.102 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 59.264784466000002, 2023-07-02T14:05:02.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:05:02.102 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.102 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.102 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.103 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.103 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.103 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:05:02.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.103 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:02.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:02.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:02.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:05:02.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:05:02.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:05:02.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:05:02.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:05:02.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:05:02.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:05:02.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:05:02.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:05:02.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:05:02.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:05:02.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:05:02.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:05:02.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:05:02.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:05:02.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:05:02.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:05:02.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:05:02.110 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.110 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.110 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.111 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:02.111 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:02.111 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 59.264734599999997, 2023-07-02T14:05:02.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:05:02.111 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.111 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.111 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.112 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.112 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.112 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:05:02.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.112 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:02.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:02.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:02.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:05:02.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:05:02.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:05:02.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:05:02.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:05:02.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:05:02.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:05:02.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:05:02.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:05:02.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:05:02.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:05:02.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:05:02.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:05:02.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:05:02.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:05:02.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:05:02.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:05:02.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:05:02.119 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.119 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.119 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.120 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:02.120 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:02.120 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 59.263695607000002, 2023-07-02T14:05:02.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:05:02.120 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.120 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.120 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.121 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.121 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.121 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:05:02.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.121 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:02.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:02.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:05:02.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:05:02.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:05:02.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:05:02.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:05:02.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:05:02.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:05:02.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:05:02.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:05:02.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:05:02.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:05:02.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:05:02.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:05:02.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:05:02.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:05:02.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:05:02.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:05:02.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:05:02.128 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.128 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.128 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.129 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7993 2.7 2:fe72db53:::rbd_data.125de1449605.0000000000000072:head [write 65536~3276800] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:02.129 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:02.129 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 59.076737559000001, 2023-07-02T14:05:02.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919, 2023-07-02T14:05:02.129 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.129 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.129 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.130 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.130 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.130 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7993 2023-07-02T14:05:02.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.130 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:02.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:02.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924986+0000", 2023-07-02T14:05:02.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049542070000000004 2023-07-02T14:05:02.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929940+0000", 2023-07-02T14:05:02.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9740000000000001e-05 2023-07-02T14:05:02.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929960+0000", 2023-07-02T14:05:02.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.311e-05 2023-07-02T14:05:02.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929973+0000", 2023-07-02T14:05:02.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9629000000000002e-05 2023-07-02T14:05:02.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930023+0000", 2023-07-02T14:05:02.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012751799999999999 2023-07-02T14:05:02.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930150+0000", 2023-07-02T14:05:02.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016449042000000001 2023-07-02T14:05:02.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946599+0000", 2023-07-02T14:05:02.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3241e-05 2023-07-02T14:05:02.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946663+0000", 2023-07-02T14:05:02.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000102296 2023-07-02T14:05:02.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946765+0000", 2023-07-02T14:05:02.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919 2023-07-02T14:05:02.137 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.137 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.137 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.138 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11782 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 196608~2162688] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:05:02.138 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:02.138 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.592217637000001, 2023-07-02T14:05:02.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365, 2023-07-02T14:05:02.138 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.138 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.138 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.138 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.139 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.139 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11782 2023-07-02T14:05:02.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.139 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:02.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:02.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409506+0000", 2023-07-02T14:05:02.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020172269999999999 2023-07-02T14:05:02.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411523+0000", 2023-07-02T14:05:02.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8901000000000001e-05 2023-07-02T14:05:02.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411542+0000", 2023-07-02T14:05:02.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5037e-05 2023-07-02T14:05:02.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411557+0000", 2023-07-02T14:05:02.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001931213 2023-07-02T14:05:02.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413489+0000", 2023-07-02T14:05:02.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000149207 2023-07-02T14:05:02.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413638+0000", 2023-07-02T14:05:02.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024752800000000001 2023-07-02T14:05:02.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:02.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413885+0000", 2023-07-02T14:05:02.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048516000000000002 2023-07-02T14:05:02.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.418737+0000", 2023-07-02T14:05:02.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010319259000000001 2023-07-02T14:05:02.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429056+0000", 2023-07-02T14:05:02.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2518000000000002e-05 2023-07-02T14:05:02.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429099+0000", 2023-07-02T14:05:02.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015275 2023-07-02T14:05:02.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429251+0000", 2023-07-02T14:05:02.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365 2023-07-02T14:05:02.147 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.147 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.147 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.148 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPushReply(2.7 119/117 [PushReplyOp(2:eb862bc0:::rbd_data.125de1449605.0000000000000040:13)])", 2023-07-02T14:05:02.148 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:02.148 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.591054362000001, 2023-07-02T14:05:02.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999, 2023-07-02T14:05:02.148 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.149 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "reached pg", 2023-07-02T14:05:02.149 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:02.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:02.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:02.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:02.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7670000000000001e-06 2023-07-02T14:05:02.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410674+0000", 2023-07-02T14:05:02.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3973e-05 2023-07-02T14:05:02.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410688+0000", 2023-07-02T14:05:02.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0157e-05 2023-07-02T14:05:02.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410708+0000", 2023-07-02T14:05:02.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018578779 2023-07-02T14:05:02.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429287+0000", 2023-07-02T14:05:02.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013086549999999999 2023-07-02T14:05:02.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.430595+0000", 2023-07-02T14:05:02.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999 2023-07-02T14:05:02.154 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.154 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.154 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.154 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11861 2.7 2:e57d3f55:::rbd_data.125de1449605.0000000000000134:head [write 0~65536] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:05:02.154 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:02.155 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.456938985000001, 2023-07-02T14:05:02.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723, 2023-07-02T14:05:02.155 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.155 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.155 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.155 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.155 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.155 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11861 2023-07-02T14:05:02.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.156 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:02.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:02.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544785+0000", 2023-07-02T14:05:02.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010563700000000001 2023-07-02T14:05:02.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544890+0000", 2023-07-02T14:05:02.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8530000000000004e-06 2023-07-02T14:05:02.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544900+0000", 2023-07-02T14:05:02.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.665e-06 2023-07-02T14:05:02.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544906+0000", 2023-07-02T14:05:02.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000272575 2023-07-02T14:05:02.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545178+0000", 2023-07-02T14:05:02.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000111362 2023-07-02T14:05:02.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545290+0000", 2023-07-02T14:05:02.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016181100000000001 2023-07-02T14:05:02.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:02.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545452+0000", 2023-07-02T14:05:02.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044500750000000004 2023-07-02T14:05:02.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.549902+0000", 2023-07-02T14:05:02.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014850024 2023-07-02T14:05:02.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564752+0000", 2023-07-02T14:05:02.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6272000000000003e-05 2023-07-02T14:05:02.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564848+0000", 2023-07-02T14:05:02.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000162428 2023-07-02T14:05:02.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.565010+0000", 2023-07-02T14:05:02.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723 2023-07-02T14:05:02.164 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.164 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.164 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.165 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:13262 2.7 e125/121 2:e0741dde:::rbd_data.125de1449605.00000000000000ad:head v 125'2024, mlcod=125'2021)", 2023-07-02T14:05:02.165 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:02.165 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7508612469999996, 2023-07-02T14:05:02.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999, 2023-07-02T14:05:02.165 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.165 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.165 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:02.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:02.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250862+0000", 2023-07-02T14:05:02.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080378800000000007 2023-07-02T14:05:02.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258900+0000", 2023-07-02T14:05:02.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8939999999999999e-06 2023-07-02T14:05:02.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258909+0000", 2023-07-02T14:05:02.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3560000000000003e-06 2023-07-02T14:05:02.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258914+0000", 2023-07-02T14:05:02.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2429000000000003e-05 2023-07-02T14:05:02.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258956+0000", 2023-07-02T14:05:02.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5094999999999999e-05 2023-07-02T14:05:02.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258981+0000", 2023-07-02T14:05:02.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012177307 2023-07-02T14:05:02.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271158+0000", 2023-07-02T14:05:02.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2104999999999999e-05 2023-07-02T14:05:02.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271181+0000", 2023-07-02T14:05:02.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999 2023-07-02T14:05:02.172 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.172 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.172 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.172 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.172 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.172 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:02.174 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:05:02.174 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:02.173+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:02.274 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:02.273+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:02.298 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:02.299 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:02.299 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:02.299 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:02.309 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:05:02.374 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:02.373+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:02.450 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:02.449+0000 7fb3991ba200 -1 osd.3 87 log_to_monitors {default=true} 2023-07-02T14:05:02.475 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:02.474+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:02.488 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:02.488 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:02.489 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:02.489 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:02.489 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:02.498 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:05:02.576 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:02.575+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:02.665 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:02.665 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:02.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:02.666 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:02.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.666 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:02.666 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:02.666 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 143.37267961699999, 2023-07-02T14:05:02.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:05:02.667 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.667 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.667 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.667 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.667 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.667 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:05:02.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.667 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:02.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:02.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:05:02.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:05:02.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:05:02.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:05:02.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:05:02.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:05:02.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:05:02.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:05:02.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:05:02.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:05:02.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:05:02.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:05:02.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:02.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:05:02.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:05:02.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:05:02.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:05:02.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:05:02.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:05:02.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:05:02.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:05:02.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:05:02.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:05:02.676 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.676 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.676 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.677 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:02.677 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:02.677 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 143.372444594, 2023-07-02T14:05:02.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:05:02.677 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.677 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.677 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.677 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.678 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.678 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:05:02.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.678 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:02.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:02.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:05:02.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:05:02.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:05:02.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:05:02.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:05:02.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:05:02.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:05:02.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:05:02.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:05:02.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:05:02.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:05:02.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:05:02.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:02.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:05:02.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:05:02.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:05:02.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:05:02.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:05:02.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:05:02.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:05:02.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:05:02.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:05:02.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:05:02.686 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.686 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.686 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.687 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:02.687 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:02.687 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 143.37224545999999, 2023-07-02T14:05:02.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:05:02.687 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.688 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.688 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.688 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.688 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.688 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:05:02.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.688 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:02.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:02.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:05:02.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:05:02.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:05:02.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:05:02.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:05:02.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:05:02.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:05:02.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:05:02.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:05:02.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:05:02.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:05:02.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:05:02.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:02.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:05:02.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:05:02.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:05:02.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:05:02.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:05:02.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:05:02.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:05:02.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:05:02.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:05:02.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:05:02.697 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.697 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.697 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.697 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:02.697 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:02.697 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 143.371880336, 2023-07-02T14:05:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:05:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:05:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:05:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:05:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:05:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:05:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:05:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:05:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:05:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:05:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:05:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:05:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:05:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:05:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:05:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:05:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:05:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:05:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:05:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:05:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:05:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:05:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:05:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:05:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:02.712 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 143.371799275, 2023-07-02T14:05:02.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:05:02.712 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.712 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.712 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.712 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.712 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.713 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:05:02.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.713 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:02.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:02.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:02.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:05:02.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:05:02.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:05:02.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:05:02.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:05:02.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:05:02.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:05:02.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:05:02.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:05:02.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:05:02.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:05:02.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:05:02.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:02.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:05:02.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:05:02.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:05:02.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:05:02.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:05:02.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:05:02.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:05:02.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:05:02.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:05:02.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:05:02.721 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.721 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.721 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.722 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:02.722 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:02.722 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 143.37170760399999, 2023-07-02T14:05:02.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:05:02.722 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.722 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.723 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.723 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.723 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.723 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:05:02.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.723 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:02.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:02.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:02.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:05:02.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:05:02.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:05:02.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:05:02.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:05:02.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:05:02.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:05:02.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:05:02.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:05:02.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:05:02.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:05:02.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:05:02.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:02.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:05:02.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:05:02.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:05:02.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:05:02.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:05:02.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:05:02.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:05:02.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:05:02.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:05:02.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:05:02.731 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.732 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.732 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.732 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:02.732 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:02.732 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 143.37136816200001, 2023-07-02T14:05:02.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:05:02.733 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.733 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.733 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.733 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.733 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.733 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:05:02.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.734 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:02.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:02.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:02.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:05:02.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:05:02.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:05:02.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:05:02.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:05:02.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:05:02.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:05:02.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:05:02.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:05:02.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:05:02.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:05:02.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:05:02.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:02.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:05:02.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:05:02.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:05:02.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:05:02.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:05:02.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:05:02.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:05:02.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:05:02.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:05:02.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:05:02.742 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.742 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.742 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.743 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:02.743 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:02.743 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 143.37118210899999, 2023-07-02T14:05:02.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:05:02.743 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.743 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.743 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.744 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.744 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.744 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:05:02.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.744 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:02.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:02.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:02.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:02.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:05:02.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:05:02.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:05:02.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:05:02.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:05:02.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:05:02.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:05:02.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:05:02.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:05:02.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:05:02.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:05:02.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:02.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:05:02.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:05:02.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:05:02.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:05:02.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:05:02.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:05:02.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:05:02.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:05:02.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:05:02.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:05:02.752 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.753 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.753 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.753 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:02.753 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:02.753 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 143.371024307, 2023-07-02T14:05:02.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:05:02.754 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.754 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.754 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.754 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.754 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.754 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:05:02.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.754 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:02.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:02.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:05:02.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:05:02.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:05:02.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:05:02.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:05:02.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:05:02.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:05:02.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:05:02.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:05:02.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:05:02.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:05:02.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:05:02.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:02.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:05:02.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:05:02.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:05:02.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:05:02.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:05:02.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:05:02.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:05:02.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:05:02.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:05:02.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:05:02.763 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.763 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.763 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.763 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:02.763 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:02.764 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 143.370832525, 2023-07-02T14:05:02.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:05:02.764 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.764 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.764 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.764 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.764 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.764 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:05:02.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.765 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:02.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:02.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:05:02.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:05:02.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:05:02.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:05:02.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:05:02.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:05:02.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:05:02.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:05:02.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:05:02.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:05:02.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:05:02.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:05:02.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:02.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:05:02.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:05:02.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:05:02.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:05:02.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:05:02.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:05:02.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:05:02.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:05:02.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:05:02.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:05:02.773 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.773 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.773 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.774 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:02.774 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:02.774 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 143.370656865, 2023-07-02T14:05:02.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:05:02.774 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.774 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.774 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.774 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.775 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.775 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:05:02.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.775 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:02.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:02.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:02.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:05:02.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:05:02.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:05:02.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:05:02.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:05:02.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:05:02.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:05:02.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:05:02.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:05:02.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:05:02.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:05:02.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:02.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:05:02.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:05:02.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:05:02.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:05:02.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:05:02.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:05:02.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:05:02.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:05:02.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:05:02.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:05:02.783 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.783 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.783 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.784 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:02.784 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:02.784 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 143.37056267700001, 2023-07-02T14:05:02.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:05:02.784 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.784 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.785 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.785 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.785 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.785 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:05:02.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.785 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:02.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:02.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:02.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:05:02.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:05:02.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:05:02.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:05:02.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:05:02.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:05:02.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:05:02.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:05:02.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:05:02.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:05:02.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:05:02.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:05:02.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:02.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:05:02.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:05:02.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:05:02.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:05:02.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:05:02.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:05:02.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:05:02.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:05:02.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:05:02.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:05:02.794 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.794 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.794 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.794 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:02.794 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:02.795 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 143.370359291, 2023-07-02T14:05:02.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:05:02.795 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.795 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.795 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.795 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.795 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.795 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:05:02.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.796 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:02.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:02.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:05:02.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:05:02.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:05:02.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:05:02.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:05:02.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:05:02.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:05:02.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:05:02.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:05:02.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:05:02.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:05:02.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:05:02.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:02.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:05:02.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:05:02.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:05:02.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:05:02.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:05:02.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:05:02.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:05:02.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:05:02.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:05:02.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:05:02.804 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.804 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.804 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.805 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:02.805 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:02.805 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 143.37020863500001, 2023-07-02T14:05:02.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:05:02.805 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.805 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.805 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.805 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.806 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.806 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:05:02.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.806 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:02.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:02.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:02.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:05:02.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:05:02.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:05:02.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:05:02.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:05:02.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:05:02.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:05:02.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:05:02.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:05:02.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:05:02.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:05:02.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:05:02.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:02.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:05:02.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:05:02.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:05:02.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:05:02.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:05:02.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:05:02.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:05:02.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:05:02.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:05:02.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:05:02.814 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.814 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.814 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.815 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:02.815 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:02.815 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 143.37013593200001, 2023-07-02T14:05:02.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:05:02.815 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.815 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.816 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.816 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.816 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.816 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:05:02.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.816 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:02.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:02.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:02.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:02.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:05:02.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:05:02.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:05:02.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:05:02.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:05:02.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:05:02.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:05:02.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:05:02.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:05:02.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:05:02.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:05:02.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:02.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:05:02.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:05:02.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:05:02.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:05:02.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:05:02.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:05:02.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:05:02.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:05:02.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:05:02.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:05:02.824 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.825 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.825 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.825 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:02.825 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:02.825 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 143.370018942, 2023-07-02T14:05:02.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:05:02.826 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.826 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.826 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.826 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.826 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.826 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:05:02.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.826 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:02.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:02.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:02.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:02.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:05:02.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:05:02.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:05:02.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:05:02.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:05:02.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:05:02.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:05:02.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:05:02.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:05:02.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:05:02.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:05:02.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:02.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:05:02.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:05:02.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:05:02.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:05:02.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:05:02.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:05:02.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:05:02.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:05:02.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:05:02.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:05:02.835 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.835 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.835 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.835 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:02.835 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:02.836 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 143.369869491, 2023-07-02T14:05:02.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:05:02.836 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.836 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.836 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.836 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.836 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.836 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:05:02.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.837 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:02.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:02.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:02.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:05:02.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:05:02.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:05:02.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:05:02.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:05:02.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:05:02.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:05:02.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:05:02.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:05:02.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:05:02.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:05:02.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:05:02.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:02.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:05:02.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:05:02.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:05:02.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:05:02.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:05:02.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:05:02.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:05:02.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:05:02.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:05:02.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:05:02.845 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.845 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.845 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.846 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:02.846 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:02.846 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 143.34839351799999, 2023-07-02T14:05:02.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:05:02.846 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.846 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.846 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.846 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.847 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.847 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:05:02.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.847 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:02.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:02.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:05:02.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:05:02.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:05:02.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:05:02.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:05:02.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:05:02.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:05:02.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:05:02.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:05:02.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:05:02.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:05:02.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:05:02.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:02.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:05:02.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:05:02.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:05:02.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:05:02.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:05:02.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:05:02.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:05:02.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:05:02.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:05:02.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:05:02.855 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.856 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.856 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.856 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:02.856 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:02.856 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 143.348260982, 2023-07-02T14:05:02.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:05:02.857 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.857 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.857 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.857 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.857 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.857 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:05:02.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.857 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:02.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:02.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:05:02.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:05:02.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:05:02.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:05:02.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:05:02.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:05:02.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:05:02.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:05:02.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:05:02.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:05:02.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:05:02.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:05:02.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:02.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:05:02.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:05:02.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:05:02.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:05:02.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:05:02.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:05:02.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:05:02.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:05:02.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:05:02.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:05:02.866 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.866 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.866 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.866 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:02.866 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:02.866 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 143.28692432099999, 2023-07-02T14:05:02.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:05:02.867 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:02.867 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:02.867 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:02.867 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:02.867 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:02.867 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:05:02.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.868 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:02.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:02.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:02.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:02.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:02.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:02.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:02.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:02.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:05:02.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:05:02.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:02.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:05:02.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:05:02.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:02.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:05:02.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:05:02.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:02.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:05:02.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:05:02.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:02.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:05:02.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:05:02.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:02.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:05:02.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:05:02.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:02.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:05:02.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:05:02.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:02.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:05:02.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:05:02.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:02.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:05:02.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:05:02.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:02.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:05:02.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:05:02.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:02.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:02.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:02.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:05:02.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:05:02.876 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.876 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.876 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.876 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:02.876 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:02.876 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:02.878 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:05:02.878 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:02.877+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:02.939 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:05:02.981 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:02.981+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:03.014 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:05:03.016 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:03.016 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:03.016 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:05:03.017 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:03.027 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:05:03.059 INFO:teuthology.orchestra.run.smithi022.stderr:no valid command found; 10 closest matches: 2023-07-02T14:05:03.059 INFO:teuthology.orchestra.run.smithi022.stderr:0 2023-07-02T14:05:03.060 INFO:teuthology.orchestra.run.smithi022.stderr:1 2023-07-02T14:05:03.060 INFO:teuthology.orchestra.run.smithi022.stderr:2 2023-07-02T14:05:03.060 INFO:teuthology.orchestra.run.smithi022.stderr:abort 2023-07-02T14:05:03.060 INFO:teuthology.orchestra.run.smithi022.stderr:assert 2023-07-02T14:05:03.060 INFO:teuthology.orchestra.run.smithi022.stderr:bluefs debug_inject_read_zeros 2023-07-02T14:05:03.060 INFO:teuthology.orchestra.run.smithi022.stderr:bluefs files list 2023-07-02T14:05:03.060 INFO:teuthology.orchestra.run.smithi022.stderr:bluefs stats 2023-07-02T14:05:03.061 INFO:teuthology.orchestra.run.smithi022.stderr:bluestore allocator dump block 2023-07-02T14:05:03.061 INFO:teuthology.orchestra.run.smithi022.stderr:bluestore allocator fragmentation block 2023-07-02T14:05:03.061 INFO:teuthology.orchestra.run.smithi022.stderr:admin_socket: invalid command 2023-07-02T14:05:03.061 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-07-02T14:05:03.062 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2023-07-02T14:05:03.084 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:05:03.085 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:03.083+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:03.109 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:03.108+0000 7fb38c2f3700 -1 osd.3 132 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-02T14:05:03.158 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:05:03.176 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:03.180 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:03.180+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:03.193 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:03.197 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:03.197 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:03.208 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:05:03.212 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:03.232 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:03.232 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:05:03.232 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:05:03.232 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:05:03.233 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.3 2023-07-02T14:05:03.233 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:05:03.233 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:05:03.235 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:05:03.235 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:05:03.280 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:03.280+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:03.334 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:03.334 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:03.344 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:05:03.387 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:03.386+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:03.468 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:03.469 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:03.479 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:05:03.520 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:03.519+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:03.582 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:03.582+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:03.643 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:03.643 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:03.653 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:05:03.683 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:03.683+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:03.777 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:03.777 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:03.784 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:03.784+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:03.787 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:05:03.885 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:03.884+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:03.910 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:03.911 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:03.921 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:05:03.986 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:03.985+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:04.086 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:04.085+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:04.089 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:04.089 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:04.100 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:05:04.187 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:04.186+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:04.264 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:04.265 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:04.275 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:05:04.316 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:04.315+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:04.388 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:04.388+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:04.407 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:04.407 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:04.418 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:05:04.489 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:04.488+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:04.541 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:04.541 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:04.551 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:05:04.594 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:04.594+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:04.675 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:04.675 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:04.685 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:05:04.691 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:04.690+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:04.791 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:04.790+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:04.849 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:04.850 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:04.860 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:05:04.901 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:04.900+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:04.993 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:04.992+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:05.023 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:05.023 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:05.033 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:05:05.094 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:05.093+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:05.097 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:05:05.163 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:05.164 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:05.175 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:05:05.194 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:05.194+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:05.237 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:05:05.295 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:05.294+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:05.307 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:05.307 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:05.318 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:05:05.380 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:05:05.396 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:05.396+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:05.398 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:05.416 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:05.435 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:05.453 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:05.471 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:05.472 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:05:05.472 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:05:05.472 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:05:05.472 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:05:05.473 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:05:05.473 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:05:05.485 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:05.485 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:05.485 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:05:05.485 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:05:05.486 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:05.496 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:05:05.497 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:05.496+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:05.598 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:05.597+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:05.662 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:05.662 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:05:05.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:05:05.662 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:05.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.663 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2620 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 196608~786432] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:05.663 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:05:05.663 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.33559173, 2023-07-02T14:05:05.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966, 2023-07-02T14:05:05.663 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:05.663 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:05.664 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:05.664 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:05.664 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:05.664 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2620 2023-07-02T14:05:05.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.664 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:05.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:05.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:05:05.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:05.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:05.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:05:05.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:05.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:05.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324571+0000", 2023-07-02T14:05:05.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016686649999999999 2023-07-02T14:05:05.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:05.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326240+0000", 2023-07-02T14:05:05.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1844e-05 2023-07-02T14:05:05.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:05.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326252+0000", 2023-07-02T14:05:05.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7760000000000003e-06 2023-07-02T14:05:05.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:05.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326258+0000", 2023-07-02T14:05:05.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0634000000000003e-05 2023-07-02T14:05:05.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326318+0000", 2023-07-02T14:05:05.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9629999999999993e-06 2023-07-02T14:05:05.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:05.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326328+0000", 2023-07-02T14:05:05.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.010877045 2023-07-02T14:05:05.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337205+0000", 2023-07-02T14:05:05.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022401300000000001 2023-07-02T14:05:05.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:05.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337429+0000", 2023-07-02T14:05:05.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000139822 2023-07-02T14:05:05.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:05.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337569+0000", 2023-07-02T14:05:05.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032942119999999999 2023-07-02T14:05:05.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:05.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340863+0000", 2023-07-02T14:05:05.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021302560000000001 2023-07-02T14:05:05.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:05.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342994+0000", 2023-07-02T14:05:05.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5634000000000002e-05 2023-07-02T14:05:05.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:05.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343059+0000", 2023-07-02T14:05:05.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143721 2023-07-02T14:05:05.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:05.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343203+0000", 2023-07-02T14:05:05.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966 2023-07-02T14:05:05.674 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.674 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.674 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.675 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2622 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 983040~36864] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:05.675 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:05:05.675 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.333899219, 2023-07-02T14:05:05.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664, 2023-07-02T14:05:05.675 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:05.675 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:05.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:05.676 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:05.676 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:05.676 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2622 2023-07-02T14:05:05.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.676 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:05.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:05.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:05:05.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:05.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:05.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:05:05.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:05.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:05.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326265+0000", 2023-07-02T14:05:05.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1282000000000001e-05 2023-07-02T14:05:05.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:05.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326286+0000", 2023-07-02T14:05:05.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9480000000000002e-06 2023-07-02T14:05:05.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:05.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326293+0000", 2023-07-02T14:05:05.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3610000000000003e-06 2023-07-02T14:05:05.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:05.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326297+0000", 2023-07-02T14:05:05.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7707999999999994e-05 2023-07-02T14:05:05.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326385+0000", 2023-07-02T14:05:05.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8070000000000006e-06 2023-07-02T14:05:05.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:05.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326394+0000", 2023-07-02T14:05:05.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.014535878 2023-07-02T14:05:05.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340930+0000", 2023-07-02T14:05:05.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000154551 2023-07-02T14:05:05.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:05.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341084+0000", 2023-07-02T14:05:05.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00032375900000000002 2023-07-02T14:05:05.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:05.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341408+0000", 2023-07-02T14:05:05.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012631066 2023-07-02T14:05:05.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:05.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354039+0000", 2023-07-02T14:05:05.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5741000000000001e-05 2023-07-02T14:05:05.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:05.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354055+0000", 2023-07-02T14:05:05.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8794000000000006e-05 2023-07-02T14:05:05.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:05.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354124+0000", 2023-07-02T14:05:05.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014667700000000001 2023-07-02T14:05:05.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:05.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354270+0000", 2023-07-02T14:05:05.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664 2023-07-02T14:05:05.686 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.686 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.686 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.686 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2727 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~12288] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:05.686 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:05:05.687 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.333863444, 2023-07-02T14:05:05.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999, 2023-07-02T14:05:05.687 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:05.687 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:05.687 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:05.687 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:05.687 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:05.687 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2727 2023-07-02T14:05:05.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.688 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:05.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:05.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:05:05.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:05.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:05.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:05:05.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:05.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:05.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:05:05.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3834e-05 2023-07-02T14:05:05.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:05.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326315+0000", 2023-07-02T14:05:05.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3029999999999996e-06 2023-07-02T14:05:05.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:05.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326321+0000", 2023-07-02T14:05:05.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7590000000000002e-06 2023-07-02T14:05:05.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:05.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326324+0000", 2023-07-02T14:05:05.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112872 2023-07-02T14:05:05.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326437+0000", 2023-07-02T14:05:05.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2290000000000002e-06 2023-07-02T14:05:05.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:05.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326442+0000", 2023-07-02T14:05:05.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0155896879999999 2023-07-02T14:05:05.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342031+0000", 2023-07-02T14:05:05.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016496100000000001 2023-07-02T14:05:05.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:05.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342196+0000", 2023-07-02T14:05:05.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021998699999999999 2023-07-02T14:05:05.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:05.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342416+0000", 2023-07-02T14:05:05.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011869744999999999 2023-07-02T14:05:05.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:05.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354286+0000", 2023-07-02T14:05:05.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0361e-05 2023-07-02T14:05:05.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:05.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354347+0000", 2023-07-02T14:05:05.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3312000000000005e-05 2023-07-02T14:05:05.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:05.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354430+0000", 2023-07-02T14:05:05.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126444 2023-07-02T14:05:05.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:05.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354556+0000", 2023-07-02T14:05:05.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999 2023-07-02T14:05:05.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.697 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.698 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2730 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [set-alloc-hint object_size 4194304 write_size 4194304,write 454656~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:05.698 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:05:05.698 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.333838435, 2023-07-02T14:05:05.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999, 2023-07-02T14:05:05.698 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:05.699 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:05.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:05.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:05.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:05.699 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2730 2023-07-02T14:05:05.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.699 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:05.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:05.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:05:05.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:05.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:05.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:05:05.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:05.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:05.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:05:05.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6049999999999997e-06 2023-07-02T14:05:05.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:05.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326332+0000", 2023-07-02T14:05:05.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3000000000000002e-06 2023-07-02T14:05:05.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:05.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326335+0000", 2023-07-02T14:05:05.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.719e-05 2023-07-02T14:05:05.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:05.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326352+0000", 2023-07-02T14:05:05.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011583599999999999 2023-07-02T14:05:05.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326468+0000", 2023-07-02T14:05:05.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.831e-06 2023-07-02T14:05:05.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:05.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326473+0000", 2023-07-02T14:05:05.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.016769915 2023-07-02T14:05:05.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343243+0000", 2023-07-02T14:05:05.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197352 2023-07-02T14:05:05.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:05.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343440+0000", 2023-07-02T14:05:05.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197581 2023-07-02T14:05:05.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:05.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343638+0000", 2023-07-02T14:05:05.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010980139 2023-07-02T14:05:05.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:05.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354618+0000", 2023-07-02T14:05:05.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056779530000000003 2023-07-02T14:05:05.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:05.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360296+0000", 2023-07-02T14:05:05.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101109 2023-07-02T14:05:05.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:05.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360397+0000", 2023-07-02T14:05:05.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000160273 2023-07-02T14:05:05.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:05.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360557+0000", 2023-07-02T14:05:05.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999 2023-07-02T14:05:05.709 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.709 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.709 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.709 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2752 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 1019904~3174400] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:05.710 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:05:05.710 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.333792211, 2023-07-02T14:05:05.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001, 2023-07-02T14:05:05.710 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:05.710 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:05.710 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:05.710 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:05.710 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:05.711 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2752 2023-07-02T14:05:05.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.711 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:05.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:05.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:05:05.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:05.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:05.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:05:05.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:05.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:05.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326372+0000", 2023-07-02T14:05:05.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052050380000000004 2023-07-02T14:05:05.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:05.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331577+0000", 2023-07-02T14:05:05.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6832000000000001e-05 2023-07-02T14:05:05.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:05.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331594+0000", 2023-07-02T14:05:05.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8220000000000007e-06 2023-07-02T14:05:05.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:05.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331602+0000", 2023-07-02T14:05:05.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9895999999999999e-05 2023-07-02T14:05:05.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331642+0000", 2023-07-02T14:05:05.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:05:05.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:05.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331649+0000", 2023-07-02T14:05:05.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0139141709999999 2023-07-02T14:05:05.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345563+0000", 2023-07-02T14:05:05.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113817 2023-07-02T14:05:05.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:05.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345677+0000", 2023-07-02T14:05:05.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018803400000000001 2023-07-02T14:05:05.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:05.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345865+0000", 2023-07-02T14:05:05.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087950359999999991 2023-07-02T14:05:05.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:05.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354660+0000", 2023-07-02T14:05:05.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059116669999999998 2023-07-02T14:05:05.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:05.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360572+0000", 2023-07-02T14:05:05.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0992000000000002e-05 2023-07-02T14:05:05.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:05.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360633+0000", 2023-07-02T14:05:05.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013601600000000001 2023-07-02T14:05:05.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:05.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360769+0000", 2023-07-02T14:05:05.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001 2023-07-02T14:05:05.720 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.721 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.721 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.721 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2759 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 0~454656] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:05.721 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:05:05.721 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.328552414, 2023-07-02T14:05:05.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999, 2023-07-02T14:05:05.722 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:05.722 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:05.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:05.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:05.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:05.722 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2759 2023-07-02T14:05:05.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.722 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:05.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:05.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:05:05.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:05.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:05.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:05:05.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:05.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:05.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331611+0000", 2023-07-02T14:05:05.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053795999999999998 2023-07-02T14:05:05.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:05.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332149+0000", 2023-07-02T14:05:05.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4970000000000007e-06 2023-07-02T14:05:05.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:05.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332158+0000", 2023-07-02T14:05:05.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:05:05.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:05.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332162+0000", 2023-07-02T14:05:05.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3526000000000001e-05 2023-07-02T14:05:05.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332185+0000", 2023-07-02T14:05:05.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:05:05.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:05.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332189+0000", 2023-07-02T14:05:05.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0186295910000001 2023-07-02T14:05:05.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350819+0000", 2023-07-02T14:05:05.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000111123 2023-07-02T14:05:05.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:05.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350930+0000", 2023-07-02T14:05:05.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019575300000000001 2023-07-02T14:05:05.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:05.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.351126+0000", 2023-07-02T14:05:05.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011503298 2023-07-02T14:05:05.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:05.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362629+0000", 2023-07-02T14:05:05.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085443989999999994 2023-07-02T14:05:05.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:05.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371173+0000", 2023-07-02T14:05:05.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3547999999999995e-05 2023-07-02T14:05:05.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:05.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371237+0000", 2023-07-02T14:05:05.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790300000000001 2023-07-02T14:05:05.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:05.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371375+0000", 2023-07-02T14:05:05.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999 2023-07-02T14:05:05.732 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.732 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.732 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.733 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2760 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 458752~131072] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:05.733 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:05:05.733 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.32799622899999, 2023-07-02T14:05:05.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001, 2023-07-02T14:05:05.733 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:05.733 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:05.733 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:05.734 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:05.734 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:05.734 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2760 2023-07-02T14:05:05.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.734 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:05.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:05.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:05:05.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:05.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:05.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:05:05.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:05.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:05.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332168+0000", 2023-07-02T14:05:05.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013281100000000001 2023-07-02T14:05:05.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:05.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332301+0000", 2023-07-02T14:05:05.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7389999999999999e-06 2023-07-02T14:05:05.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:05.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332308+0000", 2023-07-02T14:05:05.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0639999999999998e-06 2023-07-02T14:05:05.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:05.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332312+0000", 2023-07-02T14:05:05.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1735000000000001e-05 2023-07-02T14:05:05.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332333+0000", 2023-07-02T14:05:05.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8342000000000002e-05 2023-07-02T14:05:05.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:05.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332352+0000", 2023-07-02T14:05:05.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02234782 2023-07-02T14:05:05.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354699+0000", 2023-07-02T14:05:05.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110371 2023-07-02T14:05:05.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:05.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354810+0000", 2023-07-02T14:05:05.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022174900000000001 2023-07-02T14:05:05.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:05.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355032+0000", 2023-07-02T14:05:05.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076053329999999997 2023-07-02T14:05:05.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:05.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362637+0000", 2023-07-02T14:05:05.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087913609999999993 2023-07-02T14:05:05.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:05.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371428+0000", 2023-07-02T14:05:05.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8532999999999998e-05 2023-07-02T14:05:05.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:05.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371477+0000", 2023-07-02T14:05:05.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010990800000000001 2023-07-02T14:05:05.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:05.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371587+0000", 2023-07-02T14:05:05.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001 2023-07-02T14:05:05.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.744 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.744 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2766 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 589824~45056] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:05.745 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:05:05.745 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.327847604, 2023-07-02T14:05:05.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188, 2023-07-02T14:05:05.745 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:05.745 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:05.745 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:05.745 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:05.745 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:05.746 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2766 2023-07-02T14:05:05.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.746 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:05.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:05.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:05:05.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:05.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:05.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:05:05.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:05.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:05.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:05:05.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2247999999999998e-05 2023-07-02T14:05:05.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:05.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332369+0000", 2023-07-02T14:05:05.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4189999999999996e-06 2023-07-02T14:05:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332376+0000", 2023-07-02T14:05:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1049999999999999e-06 2023-07-02T14:05:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332379+0000", 2023-07-02T14:05:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6894000000000003e-05 2023-07-02T14:05:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332416+0000", 2023-07-02T14:05:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3970000000000002e-06 2023-07-02T14:05:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332422+0000", 2023-07-02T14:05:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0231382280000001 2023-07-02T14:05:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355560+0000", 2023-07-02T14:05:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9079000000000005e-05 2023-07-02T14:05:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355659+0000", 2023-07-02T14:05:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018005000000000001 2023-07-02T14:05:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:05.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355839+0000", 2023-07-02T14:05:05.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068044439999999998 2023-07-02T14:05:05.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:05.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362644+0000", 2023-07-02T14:05:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090155589999999994 2023-07-02T14:05:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371659+0000", 2023-07-02T14:05:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1297999999999994e-05 2023-07-02T14:05:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371730+0000", 2023-07-02T14:05:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013100699999999999 2023-07-02T14:05:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371861+0000", 2023-07-02T14:05:05.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188 2023-07-02T14:05:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2784 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 0~393216] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:05:05.756 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.32777915200001, 2023-07-02T14:05:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999, 2023-07-02T14:05:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:05.757 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2784 2023-07-02T14:05:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:05:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:05.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:05:05.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:05.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:05.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332385+0000", 2023-07-02T14:05:05.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040252400000000002 2023-07-02T14:05:05.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:05.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332787+0000", 2023-07-02T14:05:05.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0208e-05 2023-07-02T14:05:05.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:05.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332798+0000", 2023-07-02T14:05:05.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2023-07-02T14:05:05.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:05.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332801+0000", 2023-07-02T14:05:05.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4838000000000001e-05 2023-07-02T14:05:05.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332836+0000", 2023-07-02T14:05:05.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5319999999999998e-06 2023-07-02T14:05:05.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:05.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332839+0000", 2023-07-02T14:05:05.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0234290109999999 2023-07-02T14:05:05.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356268+0000", 2023-07-02T14:05:05.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011165700000000001 2023-07-02T14:05:05.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:05.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356380+0000", 2023-07-02T14:05:05.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022504799999999999 2023-07-02T14:05:05.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:05.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356605+0000", 2023-07-02T14:05:05.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015276201 2023-07-02T14:05:05.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:05.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371881+0000", 2023-07-02T14:05:05.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5496999999999999e-05 2023-07-02T14:05:05.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:05.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371957+0000", 2023-07-02T14:05:05.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4570000000000002e-05 2023-07-02T14:05:05.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:05.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372001+0000", 2023-07-02T14:05:05.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3880000000000003e-05 2023-07-02T14:05:05.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:05.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372085+0000", 2023-07-02T14:05:05.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999 2023-07-02T14:05:05.767 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.768 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.768 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2785 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 405504~446464] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:05.768 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:05:05.768 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.327357701, 2023-07-02T14:05:05.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999, 2023-07-02T14:05:05.769 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:05.769 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:05.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:05.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:05.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:05.769 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2785 2023-07-02T14:05:05.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.770 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:05.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:05.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:05:05.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:05.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:05.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:05:05.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:05.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:05.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:05:05.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059527500000000004 2023-07-02T14:05:05.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:05.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333402+0000", 2023-07-02T14:05:05.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7059999999999993e-06 2023-07-02T14:05:05.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:05.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333411+0000", 2023-07-02T14:05:05.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0450000000000001e-06 2023-07-02T14:05:05.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:05.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333414+0000", 2023-07-02T14:05:05.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5790999999999997e-05 2023-07-02T14:05:05.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:05:05.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3320000000000002e-06 2023-07-02T14:05:05.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:05.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333454+0000", 2023-07-02T14:05:05.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.025310269 2023-07-02T14:05:05.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358764+0000", 2023-07-02T14:05:05.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012414899999999999 2023-07-02T14:05:05.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:05.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358888+0000", 2023-07-02T14:05:05.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000217028 2023-07-02T14:05:05.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:05.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.359105+0000", 2023-07-02T14:05:05.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012788212 2023-07-02T14:05:05.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:05.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371893+0000", 2023-07-02T14:05:05.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012201569999999999 2023-07-02T14:05:05.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:05.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373114+0000", 2023-07-02T14:05:05.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4512999999999995e-05 2023-07-02T14:05:05.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:05.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373198+0000", 2023-07-02T14:05:05.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132373 2023-07-02T14:05:05.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:05.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373330+0000", 2023-07-02T14:05:05.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999 2023-07-02T14:05:05.779 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.779 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.779 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.780 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2786 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1310720~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:05.780 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:05:05.780 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.326745008, 2023-07-02T14:05:05.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001, 2023-07-02T14:05:05.780 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:05.780 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:05.781 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:05.781 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:05.781 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:05.781 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2786 2023-07-02T14:05:05.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.781 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:05.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:05.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:05:05.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:05.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:05.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:05:05.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:05.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:05.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333419+0000", 2023-07-02T14:05:05.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2009999999999999e-06 2023-07-02T14:05:05.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:05.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333427+0000", 2023-07-02T14:05:05.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0210000000000001e-06 2023-07-02T14:05:05.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:05.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333431+0000", 2023-07-02T14:05:05.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3769999999999999e-06 2023-07-02T14:05:05.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:05.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333434+0000", 2023-07-02T14:05:05.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1161e-05 2023-07-02T14:05:05.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333485+0000", 2023-07-02T14:05:05.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1549000000000001e-05 2023-07-02T14:05:05.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:05.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333496+0000", 2023-07-02T14:05:05.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0273116339999999 2023-07-02T14:05:05.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360808+0000", 2023-07-02T14:05:05.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000115128 2023-07-02T14:05:05.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:05.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360923+0000", 2023-07-02T14:05:05.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018367700000000001 2023-07-02T14:05:05.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:05.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361107+0000", 2023-07-02T14:05:05.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010796923999999999 2023-07-02T14:05:05.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:05.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371904+0000", 2023-07-02T14:05:05.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014983520000000001 2023-07-02T14:05:05.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:05.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373402+0000", 2023-07-02T14:05:05.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0520999999999999e-05 2023-07-02T14:05:05.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:05.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373463+0000", 2023-07-02T14:05:05.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000129095 2023-07-02T14:05:05.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:05.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373592+0000", 2023-07-02T14:05:05.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001 2023-07-02T14:05:05.791 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.791 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.791 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.791 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2787 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1433600~8192] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:05.792 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:05:05.792 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.32672820800001, 2023-07-02T14:05:05.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999, 2023-07-02T14:05:05.792 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:05.792 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:05.792 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:05.792 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:05.792 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:05.793 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2787 2023-07-02T14:05:05.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.793 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:05.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:05.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:05:05.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:05.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:05.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:05:05.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:05.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:05.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333436+0000", 2023-07-02T14:05:05.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2731999999999999e-05 2023-07-02T14:05:05.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:05.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:05:05.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7440000000000001e-06 2023-07-02T14:05:05.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:05.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333456+0000", 2023-07-02T14:05:05.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8650000000000003e-06 2023-07-02T14:05:05.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:05.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333460+0000", 2023-07-02T14:05:05.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0077e-05 2023-07-02T14:05:05.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333530+0000", 2023-07-02T14:05:05.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4180000000000001e-06 2023-07-02T14:05:05.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:05.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333533+0000", 2023-07-02T14:05:05.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0284002720000001 2023-07-02T14:05:05.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361933+0000", 2023-07-02T14:05:05.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6781999999999996e-05 2023-07-02T14:05:05.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:05.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362030+0000", 2023-07-02T14:05:05.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018305400000000001 2023-07-02T14:05:05.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:05.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362213+0000", 2023-07-02T14:05:05.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097015109999999995 2023-07-02T14:05:05.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:05.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371915+0000", 2023-07-02T14:05:05.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001742295 2023-07-02T14:05:05.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:05.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373657+0000", 2023-07-02T14:05:05.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5980000000000003e-05 2023-07-02T14:05:05.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:05.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373713+0000", 2023-07-02T14:05:05.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011783800000000001 2023-07-02T14:05:05.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:05.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373831+0000", 2023-07-02T14:05:05.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999 2023-07-02T14:05:05.802 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.803 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.803 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.803 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2803 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 851968~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:05.803 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:05:05.803 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.326686356, 2023-07-02T14:05:05.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514, 2023-07-02T14:05:05.804 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:05.804 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:05.804 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:05.804 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:05.804 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:05.804 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2803 2023-07-02T14:05:05.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.805 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:05.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:05.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:05:05.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:05.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:05.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:05:05.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:05.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:05.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:05:05.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4943000000000001e-05 2023-07-02T14:05:05.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:05.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333493+0000", 2023-07-02T14:05:05.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2629999999999997e-06 2023-07-02T14:05:05.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:05.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333500+0000", 2023-07-02T14:05:05.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4159999999999997e-06 2023-07-02T14:05:05.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:05.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333505+0000", 2023-07-02T14:05:05.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4518999999999998e-05 2023-07-02T14:05:05.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333559+0000", 2023-07-02T14:05:05.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5900000000000008e-06 2023-07-02T14:05:05.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:05.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333568+0000", 2023-07-02T14:05:05.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0291061779999999 2023-07-02T14:05:05.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362674+0000", 2023-07-02T14:05:05.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110412 2023-07-02T14:05:05.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:05.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362784+0000", 2023-07-02T14:05:05.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015384599999999999 2023-07-02T14:05:05.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:05.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362938+0000", 2023-07-02T14:05:05.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091682310000000006 2023-07-02T14:05:05.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:05.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372106+0000", 2023-07-02T14:05:05.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017703230000000001 2023-07-02T14:05:05.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:05.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373877+0000", 2023-07-02T14:05:05.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6387000000000002e-05 2023-07-02T14:05:05.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:05.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373923+0000", 2023-07-02T14:05:05.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014099199999999999 2023-07-02T14:05:05.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:05.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374064+0000", 2023-07-02T14:05:05.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514 2023-07-02T14:05:05.814 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.814 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.814 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.815 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2805 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 634880~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:05.815 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:05:05.815 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.326653181, 2023-07-02T14:05:05.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814, 2023-07-02T14:05:05.815 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:05.815 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:05.816 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:05.816 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:05.816 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:05.816 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2805 2023-07-02T14:05:05.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.816 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:05.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:05.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:05:05.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:05.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:05.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:05:05.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:05.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:05.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333511+0000", 2023-07-02T14:05:05.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207092 2023-07-02T14:05:05.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:05.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333718+0000", 2023-07-02T14:05:05.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5110000000000003e-06 2023-07-02T14:05:05.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:05.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333727+0000", 2023-07-02T14:05:05.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3270000000000007e-06 2023-07-02T14:05:05.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:05.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333736+0000", 2023-07-02T14:05:05.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0319999999999999e-05 2023-07-02T14:05:05.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333756+0000", 2023-07-02T14:05:05.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1695000000000001e-05 2023-07-02T14:05:05.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:05.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:05:05.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0295750560000001 2023-07-02T14:05:05.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363343+0000", 2023-07-02T14:05:05.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3387999999999994e-05 2023-07-02T14:05:05.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:05.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363437+0000", 2023-07-02T14:05:05.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018373 2023-07-02T14:05:05.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:05.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363620+0000", 2023-07-02T14:05:05.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0084948769999999996 2023-07-02T14:05:05.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:05.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372115+0000", 2023-07-02T14:05:05.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020082170000000001 2023-07-02T14:05:05.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:05.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374123+0000", 2023-07-02T14:05:05.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8875999999999997e-05 2023-07-02T14:05:05.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:05.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374182+0000", 2023-07-02T14:05:05.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0001272 2023-07-02T14:05:05.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:05.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374309+0000", 2023-07-02T14:05:05.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814 2023-07-02T14:05:05.826 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.826 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.826 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.826 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2806 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 1359872~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:05.826 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:05:05.827 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.32642285, 2023-07-02T14:05:05.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234, 2023-07-02T14:05:05.827 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:05.827 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:05.827 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:05.827 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:05.827 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:05.827 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2806 2023-07-02T14:05:05.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.828 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:05.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:05.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:05:05.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:05.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:05.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:05:05.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:05.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:05.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333741+0000", 2023-07-02T14:05:05.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3978999999999999e-05 2023-07-02T14:05:05.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:05.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333755+0000", 2023-07-02T14:05:05.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:05:05.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:05.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333761+0000", 2023-07-02T14:05:05.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7010000000000001e-06 2023-07-02T14:05:05.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:05.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333764+0000", 2023-07-02T14:05:05.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5086000000000003e-05 2023-07-02T14:05:05.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333799+0000", 2023-07-02T14:05:05.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6389999999999992e-06 2023-07-02T14:05:05.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:05.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333806+0000", 2023-07-02T14:05:05.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.030496675 2023-07-02T14:05:05.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364303+0000", 2023-07-02T14:05:05.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011382700000000001 2023-07-02T14:05:05.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:05.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364417+0000", 2023-07-02T14:05:05.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192058 2023-07-02T14:05:05.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:05.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364609+0000", 2023-07-02T14:05:05.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075132130000000004 2023-07-02T14:05:05.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:05.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372122+0000", 2023-07-02T14:05:05.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022731750000000001 2023-07-02T14:05:05.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:05.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374395+0000", 2023-07-02T14:05:05.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5277000000000005e-05 2023-07-02T14:05:05.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:05.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374460+0000", 2023-07-02T14:05:05.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017074599999999999 2023-07-02T14:05:05.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:05.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374631+0000", 2023-07-02T14:05:05.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234 2023-07-02T14:05:05.837 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.837 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.838 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.838 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 868352~49152] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:05.838 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:05:05.838 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.326396136, 2023-07-02T14:05:05.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001, 2023-07-02T14:05:05.838 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:05.839 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:05.839 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:05.839 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:05.839 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:05.839 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:05:05.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.839 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:05.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:05.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:05:05.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:05.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:05.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:05:05.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:05.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:05.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:05:05.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6355000000000001e-05 2023-07-02T14:05:05.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:05.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333794+0000", 2023-07-02T14:05:05.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8329999999999996e-06 2023-07-02T14:05:05.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:05.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333801+0000", 2023-07-02T14:05:05.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5419999999999996e-06 2023-07-02T14:05:05.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:05.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333807+0000", 2023-07-02T14:05:05.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2023-07-02T14:05:05.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333826+0000", 2023-07-02T14:05:05.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.303e-06 2023-07-02T14:05:05.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:05.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333830+0000", 2023-07-02T14:05:05.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.031137559 2023-07-02T14:05:05.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364967+0000", 2023-07-02T14:05:05.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1390000000000004e-05 2023-07-02T14:05:05.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:05.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365059+0000", 2023-07-02T14:05:05.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017597399999999999 2023-07-02T14:05:05.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:05.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365234+0000", 2023-07-02T14:05:05.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068943950000000002 2023-07-02T14:05:05.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:05.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372129+0000", 2023-07-02T14:05:05.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026189910000000002 2023-07-02T14:05:05.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:05.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374748+0000", 2023-07-02T14:05:05.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6613999999999996e-05 2023-07-02T14:05:05.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:05.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374814+0000", 2023-07-02T14:05:05.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502300000000001 2023-07-02T14:05:05.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:05.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374950+0000", 2023-07-02T14:05:05.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001 2023-07-02T14:05:05.849 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.849 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.849 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.850 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:172 2.5 2:a79a9f1f:::rbd_data.125de1449605.0000000000000039:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:05.850 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.848672+0000", 2023-07-02T14:05:05.850 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.8114932009999993, 2023-07-02T14:05:05.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041540012000000001, 2023-07-02T14:05:05.850 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:05.850 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:05.850 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:05.850 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:05.851 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:05.851 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 172 2023-07-02T14:05:05.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.851 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:05.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:05.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848672+0000", 2023-07-02T14:05:05.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:05.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:05.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848672+0000", 2023-07-02T14:05:05.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:05:05.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:05.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848669+0000", 2023-07-02T14:05:05.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1610000000000001e-05 2023-07-02T14:05:05.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:05.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848690+0000", 2023-07-02T14:05:05.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0169999999999998e-05 2023-07-02T14:05:05.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:05.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848710+0000", 2023-07-02T14:05:05.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0960000000000007e-06 2023-07-02T14:05:05.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:05.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848720+0000", 2023-07-02T14:05:05.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0717000000000002e-05 2023-07-02T14:05:05.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848780+0000", 2023-07-02T14:05:05.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.303e-06 2023-07-02T14:05:05.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2023-07-02T14:05:05.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848789+0000", 2023-07-02T14:05:05.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032354921000000002 2023-07-02T14:05:05.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.881144+0000", 2023-07-02T14:05:05.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0827000000000005e-05 2023-07-02T14:05:05.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:05.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.881214+0000", 2023-07-02T14:05:05.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089973080000000007 2023-07-02T14:05:05.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:05.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.890212+0000", 2023-07-02T14:05:05.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041540012000000001 2023-07-02T14:05:05.858 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.858 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.858 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.859 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:174 2.5 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:05.859 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.851335+0000", 2023-07-02T14:05:05.859 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.8088299939999999, 2023-07-02T14:05:05.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.044329581999999999, 2023-07-02T14:05:05.859 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:05.859 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:05.859 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:05.860 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:05.860 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:05.860 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 174 2023-07-02T14:05:05.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.860 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:05.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:05.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851335+0000", 2023-07-02T14:05:05.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:05.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:05.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851335+0000", 2023-07-02T14:05:05.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2023-07-02T14:05:05.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:05.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851332+0000", 2023-07-02T14:05:05.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6269999999999998e-05 2023-07-02T14:05:05.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:05.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851348+0000", 2023-07-02T14:05:05.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3189000000000001e-05 2023-07-02T14:05:05.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:05.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851371+0000", 2023-07-02T14:05:05.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.132e-06 2023-07-02T14:05:05.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:05.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851380+0000", 2023-07-02T14:05:05.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7970999999999999e-05 2023-07-02T14:05:05.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851428+0000", 2023-07-02T14:05:05.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1050000000000003e-06 2023-07-02T14:05:05.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2023-07-02T14:05:05.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851433+0000", 2023-07-02T14:05:05.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038825011999999999 2023-07-02T14:05:05.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.890258+0000", 2023-07-02T14:05:05.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0666000000000002e-05 2023-07-02T14:05:05.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:05.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.890349+0000", 2023-07-02T14:05:05.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0053153460000000003 2023-07-02T14:05:05.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:05.867 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.895664+0000", 2023-07-02T14:05:05.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.044329581999999999 2023-07-02T14:05:05.867 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.867 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.868 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.868 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:336 2.5 2:a8e921ad:::rbd_data.125de1449605.00000000000000d7:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:05.868 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636630+0000", 2023-07-02T14:05:05.868 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.0235350160000003, 2023-07-02T14:05:05.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041573730000000003, 2023-07-02T14:05:05.868 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:05.869 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:05.869 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:05.869 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:05.869 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:05.869 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 336 2023-07-02T14:05:05.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.869 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:05.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:05.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636630+0000", 2023-07-02T14:05:05.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:05.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:05.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636630+0000", 2023-07-02T14:05:05.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:05.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:05.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636629+0000", 2023-07-02T14:05:05.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.847e-06 2023-07-02T14:05:05.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:05.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636632+0000", 2023-07-02T14:05:05.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0449999999999997e-06 2023-07-02T14:05:05.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:05.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636638+0000", 2023-07-02T14:05:05.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5080000000000001e-06 2023-07-02T14:05:05.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:05.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636642+0000", 2023-07-02T14:05:05.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033756268999999998 2023-07-02T14:05:05.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.670398+0000", 2023-07-02T14:05:05.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8985000000000004e-05 2023-07-02T14:05:05.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.874 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:05.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.670467+0000", 2023-07-02T14:05:05.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077364360000000002 2023-07-02T14:05:05.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:05.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.678204+0000", 2023-07-02T14:05:05.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041573730000000003 2023-07-02T14:05:05.875 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.875 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.875 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.876 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:343 2.5 2:b3c82204:::rbd_data.125de1449605.00000000000000de:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:05.876 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636645+0000", 2023-07-02T14:05:05.876 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.0235195869999991, 2023-07-02T14:05:05.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.048581695000000001, 2023-07-02T14:05:05.876 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:05.876 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:05.877 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:05.877 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:05.877 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:05.877 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 343 2023-07-02T14:05:05.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.877 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:05.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:05.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636645+0000", 2023-07-02T14:05:05.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:05.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:05.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636645+0000", 2023-07-02T14:05:05.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:05.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:05.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636645+0000", 2023-07-02T14:05:05.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9190000000000004e-06 2023-07-02T14:05:05.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:05.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636651+0000", 2023-07-02T14:05:05.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4e-06 2023-07-02T14:05:05.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:05.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636656+0000", 2023-07-02T14:05:05.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5860000000000001e-06 2023-07-02T14:05:05.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:05.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636660+0000", 2023-07-02T14:05:05.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041586692000000001 2023-07-02T14:05:05.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:05.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.678246+0000", 2023-07-02T14:05:05.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113236 2023-07-02T14:05:05.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:05.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.678360+0000", 2023-07-02T14:05:05.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006867199 2023-07-02T14:05:05.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:05.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:05.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:05.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.685227+0000", 2023-07-02T14:05:05.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.048581695000000001 2023-07-02T14:05:05.883 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.883 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.883 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.883 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:05.883 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:05.884 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:05.884 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap32 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap33 b3fbb028-e267-4d25-a750-93ba18a2316a@snap33.diff 2023-07-02T14:05:05.885 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:05:05.886 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:05.884+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:05.986 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:05.985+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:06.050 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:06.050 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:06.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.050 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.6033.0:1 2.0 2.17b19478 (undecoded) ondisk+read+known_if_redirected e134)", 2023-07-02T14:05:06.051 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:05.825280+0000", 2023-07-02T14:05:06.051 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.22515085200000001, 2023-07-02T14:05:06.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.22517512000000001, 2023-07-02T14:05:06.051 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.051 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "delayed", 2023-07-02T14:05:06.051 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.051 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.6033", 2023-07-02T14:05:06.052 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3101648408", 2023-07-02T14:05:06.052 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 1 2023-07-02T14:05:06.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.052 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:05.825280+0000", 2023-07-02T14:05:06.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:05.825280+0000", 2023-07-02T14:05:06.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:06.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:05.825279+0000", 2023-07-02T14:05:06.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2110000000000003e-06 2023-07-02T14:05:06.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:05.825285+0000", 2023-07-02T14:05:06.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:05:06.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:05.825294+0000", 2023-07-02T14:05:06.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8339999999999999e-06 2023-07-02T14:05:06.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:05.825302+0000", 2023-07-02T14:05:06.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8751999999999995e-05 2023-07-02T14:05:06.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:05.825371+0000", 2023-07-02T14:05:06.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2187999999999999e-05 2023-07-02T14:05:06.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:05.825383+0000", 2023-07-02T14:05:06.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000103037 2023-07-02T14:05:06.057 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.057 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.057 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.058 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.058 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:05:06.058 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 1 2023-07-02T14:05:06.058 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:06.060 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:05:06.096 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:05:06.097 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:06.095+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:06.187 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:06.186+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:06.208 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:06.208 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:06.208 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:05:06.208 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:05:06.209 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:06.221 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:05:06.220+0000 7fac70b51700 0 -- 172.21.15.163:0/3101648408 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x558d9b7a5cd0 msgr2=0x558d9b7a80c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:06.222 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:05:06.239 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:05:06.239+0000 7fac70b51700 0 -- 172.21.15.163:0/3101648408 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7fac50036ca0 msgr2=0x7fac50039090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:06.285 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 100% complete...done. 2023-07-02T14:05:06.288 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:05:06.288 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=34 2023-07-02T14:05:06.289 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap34 2023-07-02T14:05:06.306 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:06.305+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:06.389 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:06.388+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:06.399 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:06.400 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:05:06.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:05:06.400 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:06.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.400 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:05:06.400 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:06.401 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 116.94579810499999, 2023-07-02T14:05:06.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:05:06.401 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.401 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:06.401 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.401 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:06.401 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:06.401 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:05:06.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.402 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:06.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:06.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:06.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:05:06.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:05:06.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:05:06.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:05:06.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:05:06.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:05:06.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:05:06.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:05:06.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:05:06.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:05:06.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:05:06.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:05:06.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:05:06.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:05:06.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:06.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:05:06.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:05:06.408 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.408 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.408 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.409 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:05:06.409 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:06.409 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 116.07510509700001, 2023-07-02T14:05:06.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:05:06.409 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.409 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:06.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:05:06.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:05:06.410 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:05:06.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.410 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:06.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:06.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:06.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:05:06.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:05:06.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:05:06.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:05:06.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:05:06.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:05:06.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:05:06.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:05:06.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:05:06.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:05:06.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:05:06.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:05:06.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:05:06.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:05:06.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:06.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:05:06.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:05:06.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:06.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:05:06.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:05:06.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.417 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.418 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:06.418 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:06.418 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 116.074980376, 2023-07-02T14:05:06.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:05:06.418 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.418 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:06.419 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.419 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:06.419 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:06.419 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:05:06.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.419 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:06.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:06.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:06.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:05:06.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:05:06.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:05:06.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:05:06.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:05:06.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:05:06.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:05:06.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:05:06.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:05:06.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:05:06.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:05:06.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:05:06.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:05:06.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:05:06.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:06.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:05:06.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:05:06.426 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.426 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.426 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.426 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:06.427 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:06.427 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 116.06813049199999, 2023-07-02T14:05:06.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:05:06.427 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.427 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:06.427 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.427 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:06.427 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:06.428 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:05:06.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.428 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:06.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:06.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:05:06.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:05:06.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:05:06.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:05:06.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:05:06.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:05:06.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:05:06.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:05:06.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:05:06.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:05:06.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:05:06.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:05:06.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:05:06.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:05:06.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:05:06.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:06.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:05:06.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:05:06.434 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.434 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.434 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.435 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:06.435 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:06.435 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 116.068067567, 2023-07-02T14:05:06.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:05:06.435 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.435 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:06.436 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.436 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:06.436 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:06.436 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:05:06.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.436 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:06.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:06.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:06.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:05:06.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:05:06.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:05:06.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:05:06.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:05:06.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:05:06.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:05:06.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:05:06.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:05:06.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:05:06.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:05:06.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:05:06.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:05:06.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:05:06.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:06.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:05:06.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:05:06.443 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.443 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.443 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.443 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:06.443 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:06.443 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 116.06776714599999, 2023-07-02T14:05:06.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:05:06.444 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.444 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:06.444 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.444 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:06.444 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:06.444 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:05:06.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.445 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:06.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:06.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:06.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:05:06.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:05:06.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:05:06.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:05:06.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:05:06.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:05:06.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:05:06.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:05:06.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:05:06.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:05:06.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:05:06.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:05:06.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:05:06.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:05:06.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:06.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:05:06.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:05:06.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:05:06.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:05:06.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:05:06.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:06.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:05:06.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:05:06.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:06.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:05:06.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:05:06.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:06.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:05:06.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:05:06.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:06.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:05:06.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:05:06.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:06.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:05:06.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:05:06.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.455 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.455 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:05:06.456 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:06.456 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 116.067242646, 2023-07-02T14:05:06.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:05:06.456 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.456 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:06.456 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.456 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:06.456 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:06.457 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:05:06.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.457 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:06.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:06.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:06.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:06.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:05:06.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:05:06.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:05:06.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:05:06.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:05:06.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:05:06.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:05:06.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:05:06.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:05:06.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:05:06.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:05:06.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:05:06.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:05:06.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:06.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:05:06.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:05:06.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:06.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:05:06.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:05:06.464 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.464 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.464 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.465 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:05:06.465 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:06.465 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.943223520999993, 2023-07-02T14:05:06.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:05:06.465 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.465 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:06.465 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.465 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:06.466 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:06.466 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:05:06.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.466 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:06.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:06.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:06.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:05:06.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:05:06.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:05:06.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:05:06.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:05:06.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:05:06.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:05:06.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:05:06.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:05:06.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:05:06.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:05:06.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:05:06.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:05:06.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:05:06.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:06.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:05:06.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:05:06.472 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.473 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.473 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.473 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:06.473 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:06.473 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.617740816999998, 2023-07-02T14:05:06.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:05:06.474 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.474 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:06.474 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.474 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:06.474 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:06.474 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:05:06.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.474 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:06.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:06.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:06.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:05:06.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:05:06.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:05:06.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:05:06.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:05:06.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:05:06.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:05:06.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:05:06.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:06.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:05:06.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:05:06.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:05:06.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:05:06.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:05:06.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:06.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:05:06.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:05:06.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:06.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:05:06.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:05:06.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:06.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:05:06.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:05:06.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:06.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:05:06.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:05:06.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:06.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:05:06.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:05:06.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:06.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:05:06.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:05:06.484 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.484 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.484 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.485 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3887 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:06.485 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:05:06.485 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.613877627999997, 2023-07-02T14:05:06.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998, 2023-07-02T14:05:06.485 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.485 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:06.486 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.486 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:06.486 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:06.486 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3887 2023-07-02T14:05:06.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.486 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:05:06.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:05:06.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:06.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784019+0000", 2023-07-02T14:05:06.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4073999999999998e-05 2023-07-02T14:05:06.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784063+0000", 2023-07-02T14:05:06.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6630000000000004e-06 2023-07-02T14:05:06.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784070+0000", 2023-07-02T14:05:06.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.286e-06 2023-07-02T14:05:06.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784073+0000", 2023-07-02T14:05:06.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4941999999999998e-05 2023-07-02T14:05:06.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784118+0000", 2023-07-02T14:05:06.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8120000000000008e-06 2023-07-02T14:05:06.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784126+0000", 2023-07-02T14:05:06.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32999709399999999 2023-07-02T14:05:06.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114123+0000", 2023-07-02T14:05:06.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000157257 2023-07-02T14:05:06.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:06.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114280+0000", 2023-07-02T14:05:06.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000341152 2023-07-02T14:05:06.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:06.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114622+0000", 2023-07-02T14:05:06.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019539766 2023-07-02T14:05:06.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:06.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.134161+0000", 2023-07-02T14:05:06.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015820796000000002 2023-07-02T14:05:06.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:06.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149982+0000", 2023-07-02T14:05:06.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.05e-05 2023-07-02T14:05:06.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:06.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150043+0000", 2023-07-02T14:05:06.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6453999999999999e-05 2023-07-02T14:05:06.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:06.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150089+0000", 2023-07-02T14:05:06.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998 2023-07-02T14:05:06.496 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.496 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.496 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.496 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3939 2.1 2:9ef34a83:::rbd_data.125de1449605.00000000000000b8:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:06.497 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:06.497 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.61382012, 2023-07-02T14:05:06.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001, 2023-07-02T14:05:06.497 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.497 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:06.497 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.497 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:06.497 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:06.498 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3939 2023-07-02T14:05:06.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.498 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:06.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:06.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:06.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:06.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046852830000000002 2023-07-02T14:05:06.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788763+0000", 2023-07-02T14:05:06.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3185e-05 2023-07-02T14:05:06.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788776+0000", 2023-07-02T14:05:06.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8470000000000003e-06 2023-07-02T14:05:06.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788781+0000", 2023-07-02T14:05:06.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7104000000000002e-05 2023-07-02T14:05:06.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788828+0000", 2023-07-02T14:05:06.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3516000000000001e-05 2023-07-02T14:05:06.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788841+0000", 2023-07-02T14:05:06.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33064513600000001 2023-07-02T14:05:06.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119487+0000", 2023-07-02T14:05:06.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9876999999999997e-05 2023-07-02T14:05:06.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:06.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119537+0000", 2023-07-02T14:05:06.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014160999999999999 2023-07-02T14:05:06.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:06.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119678+0000", 2023-07-02T14:05:06.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023054392 2023-07-02T14:05:06.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:06.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.142733+0000", 2023-07-02T14:05:06.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074065470000000003 2023-07-02T14:05:06.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:06.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150139+0000", 2023-07-02T14:05:06.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5025999999999997e-05 2023-07-02T14:05:06.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:06.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150204+0000", 2023-07-02T14:05:06.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147999999999997e-05 2023-07-02T14:05:06.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:06.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150241+0000", 2023-07-02T14:05:06.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001 2023-07-02T14:05:06.507 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.508 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.508 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.508 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3940 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:06.508 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:05:06.508 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.609109087999997, 2023-07-02T14:05:06.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001, 2023-07-02T14:05:06.509 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.509 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:06.509 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.509 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:06.509 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:06.509 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3940 2023-07-02T14:05:06.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.509 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:05:06.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:05:06.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:06.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788788+0000", 2023-07-02T14:05:06.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043166560000000003 2023-07-02T14:05:06.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793105+0000", 2023-07-02T14:05:06.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5970000000000001e-05 2023-07-02T14:05:06.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793121+0000", 2023-07-02T14:05:06.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0939999999999998e-06 2023-07-02T14:05:06.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793126+0000", 2023-07-02T14:05:06.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2438000000000002e-05 2023-07-02T14:05:06.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793178+0000", 2023-07-02T14:05:06.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2019e-05 2023-07-02T14:05:06.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793190+0000", 2023-07-02T14:05:06.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33208757799999999 2023-07-02T14:05:06.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125278+0000", 2023-07-02T14:05:06.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5255999999999997e-05 2023-07-02T14:05:06.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:06.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125373+0000", 2023-07-02T14:05:06.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015674999999999999 2023-07-02T14:05:06.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:06.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125530+0000", 2023-07-02T14:05:06.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018767308 2023-07-02T14:05:06.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:06.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144297+0000", 2023-07-02T14:05:06.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060291959999999997 2023-07-02T14:05:06.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:06.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150326+0000", 2023-07-02T14:05:06.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3886999999999993e-05 2023-07-02T14:05:06.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:06.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150390+0000", 2023-07-02T14:05:06.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010255 2023-07-02T14:05:06.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:06.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150493+0000", 2023-07-02T14:05:06.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001 2023-07-02T14:05:06.519 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.519 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.519 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.520 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3947 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:06.520 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:05:06.520 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.604763560999999, 2023-07-02T14:05:06.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999, 2023-07-02T14:05:06.520 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.520 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:06.521 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.521 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:06.521 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:06.521 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3947 2023-07-02T14:05:06.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.521 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:05:06.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:05:06.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:06.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793133+0000", 2023-07-02T14:05:06.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2519999999999999e-05 2023-07-02T14:05:06.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793176+0000", 2023-07-02T14:05:06.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:05:06.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793180+0000", 2023-07-02T14:05:06.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1709999999999998e-06 2023-07-02T14:05:06.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793184+0000", 2023-07-02T14:05:06.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1721999999999999e-05 2023-07-02T14:05:06.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793215+0000", 2023-07-02T14:05:06.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1280000000000008e-06 2023-07-02T14:05:06.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793224+0000", 2023-07-02T14:05:06.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34218549599999998 2023-07-02T14:05:06.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135409+0000", 2023-07-02T14:05:06.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117442 2023-07-02T14:05:06.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:06.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135526+0000", 2023-07-02T14:05:06.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000206257 2023-07-02T14:05:06.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:06.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135733+0000", 2023-07-02T14:05:06.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085700029999999996 2023-07-02T14:05:06.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:06.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144303+0000", 2023-07-02T14:05:06.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006278652 2023-07-02T14:05:06.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:06.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150581+0000", 2023-07-02T14:05:06.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4765999999999996e-05 2023-07-02T14:05:06.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:06.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150646+0000", 2023-07-02T14:05:06.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4777999999999995e-05 2023-07-02T14:05:06.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:06.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150711+0000", 2023-07-02T14:05:06.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999 2023-07-02T14:05:06.531 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.531 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.531 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.532 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3980 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 65536~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:06.532 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:05:06.532 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.604709678999996, 2023-07-02T14:05:06.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999, 2023-07-02T14:05:06.532 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.532 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:06.532 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.532 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:06.533 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:06.533 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3980 2023-07-02T14:05:06.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.533 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:05:06.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:05:06.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:06.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:05:06.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020244260000000002 2023-07-02T14:05:06.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795212+0000", 2023-07-02T14:05:06.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0005e-05 2023-07-02T14:05:06.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795222+0000", 2023-07-02T14:05:06.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9860000000000002e-06 2023-07-02T14:05:06.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795227+0000", 2023-07-02T14:05:06.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6369999999999999e-05 2023-07-02T14:05:06.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795264+0000", 2023-07-02T14:05:06.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1052e-05 2023-07-02T14:05:06.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795275+0000", 2023-07-02T14:05:06.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.343850459 2023-07-02T14:05:06.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139125+0000", 2023-07-02T14:05:06.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6760000000000004e-05 2023-07-02T14:05:06.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:06.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139202+0000", 2023-07-02T14:05:06.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015092799999999999 2023-07-02T14:05:06.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:06.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139353+0000", 2023-07-02T14:05:06.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011425940000000001 2023-07-02T14:05:06.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:06.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150779+0000", 2023-07-02T14:05:06.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033223060000000001 2023-07-02T14:05:06.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:06.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154101+0000", 2023-07-02T14:05:06.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9268999999999998e-05 2023-07-02T14:05:06.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:06.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154161+0000", 2023-07-02T14:05:06.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6499999999999999e-05 2023-07-02T14:05:06.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:06.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154207+0000", 2023-07-02T14:05:06.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999 2023-07-02T14:05:06.543 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.543 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.543 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.543 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3982 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1835008~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:06.543 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:05:06.543 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.602664391999994, 2023-07-02T14:05:06.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002, 2023-07-02T14:05:06.544 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.544 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:06.544 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.544 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:06.544 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:06.544 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3982 2023-07-02T14:05:06.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.545 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:05:06.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:05:06.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:06.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795233+0000", 2023-07-02T14:05:06.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8680999999999998e-05 2023-07-02T14:05:06.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795272+0000", 2023-07-02T14:05:06.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5629999999999999e-06 2023-07-02T14:05:06.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795277+0000", 2023-07-02T14:05:06.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8700000000000001e-06 2023-07-02T14:05:06.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795280+0000", 2023-07-02T14:05:06.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1964000000000001e-05 2023-07-02T14:05:06.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795312+0000", 2023-07-02T14:05:06.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.314e-06 2023-07-02T14:05:06.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795319+0000", 2023-07-02T14:05:06.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34828442599999998 2023-07-02T14:05:06.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143603+0000", 2023-07-02T14:05:06.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2438000000000001e-05 2023-07-02T14:05:06.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:06.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143665+0000", 2023-07-02T14:05:06.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3079999999999997e-05 2023-07-02T14:05:06.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:06.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143759+0000", 2023-07-02T14:05:06.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070633079999999999 2023-07-02T14:05:06.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:06.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150822+0000", 2023-07-02T14:05:06.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034430210000000001 2023-07-02T14:05:06.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:06.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154265+0000", 2023-07-02T14:05:06.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4249999999999998e-05 2023-07-02T14:05:06.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:06.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154309+0000", 2023-07-02T14:05:06.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2020000000000001e-05 2023-07-02T14:05:06.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:06.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154351+0000", 2023-07-02T14:05:06.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002 2023-07-02T14:05:06.554 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.554 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.554 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.555 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4054 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1900544~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:06.555 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:05:06.555 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.602612721, 2023-07-02T14:05:06.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002, 2023-07-02T14:05:06.555 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.555 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:06.556 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.556 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:06.556 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:06.556 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4054 2023-07-02T14:05:06.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.556 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:05:06.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:05:06.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:06.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:05:06.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018874860000000001 2023-07-02T14:05:06.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797172+0000", 2023-07-02T14:05:06.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3000000000000007e-06 2023-07-02T14:05:06.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797182+0000", 2023-07-02T14:05:06.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6260000000000003e-06 2023-07-02T14:05:06.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797186+0000", 2023-07-02T14:05:06.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9013e-05 2023-07-02T14:05:06.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797235+0000", 2023-07-02T14:05:06.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5990000000000008e-06 2023-07-02T14:05:06.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797245+0000", 2023-07-02T14:05:06.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34785621999999999 2023-07-02T14:05:06.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145101+0000", 2023-07-02T14:05:06.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7762000000000006e-05 2023-07-02T14:05:06.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:06.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145169+0000", 2023-07-02T14:05:06.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013469400000000001 2023-07-02T14:05:06.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:06.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145304+0000", 2023-07-02T14:05:06.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089538039999999992 2023-07-02T14:05:06.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:06.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154257+0000", 2023-07-02T14:05:06.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104218 2023-07-02T14:05:06.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:06.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154362+0000", 2023-07-02T14:05:06.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4168000000000001e-05 2023-07-02T14:05:06.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:06.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154406+0000", 2023-07-02T14:05:06.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3098000000000001e-05 2023-07-02T14:05:06.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:06.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154429+0000", 2023-07-02T14:05:06.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002 2023-07-02T14:05:06.566 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.566 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.566 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.567 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4056 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3670016~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:06.567 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:05:06.567 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.600706208000005, 2023-07-02T14:05:06.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997, 2023-07-02T14:05:06.567 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.567 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:06.567 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.568 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:06.568 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:06.568 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4056 2023-07-02T14:05:06.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.568 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:05:06.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:05:06.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:06.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797191+0000", 2023-07-02T14:05:06.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0438999999999997e-05 2023-07-02T14:05:06.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797232+0000", 2023-07-02T14:05:06.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6929999999999998e-06 2023-07-02T14:05:06.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797237+0000", 2023-07-02T14:05:06.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9739999999999998e-06 2023-07-02T14:05:06.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797240+0000", 2023-07-02T14:05:06.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0243000000000002e-05 2023-07-02T14:05:06.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797270+0000", 2023-07-02T14:05:06.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9889999999999997e-06 2023-07-02T14:05:06.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797276+0000", 2023-07-02T14:05:06.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35358319799999999 2023-07-02T14:05:06.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150859+0000", 2023-07-02T14:05:06.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1357e-05 2023-07-02T14:05:06.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:06.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150940+0000", 2023-07-02T14:05:06.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014903 2023-07-02T14:05:06.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:06.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151089+0000", 2023-07-02T14:05:06.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033512519999999999 2023-07-02T14:05:06.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:06.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154441+0000", 2023-07-02T14:05:06.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001050945 2023-07-02T14:05:06.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:06.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155492+0000", 2023-07-02T14:05:06.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2523e-05 2023-07-02T14:05:06.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:06.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155524+0000", 2023-07-02T14:05:06.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.057e-05 2023-07-02T14:05:06.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:06.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155555+0000", 2023-07-02T14:05:06.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997 2023-07-02T14:05:06.578 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.578 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.578 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.578 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4088 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3735552~458752] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:06.578 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:05:06.579 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.600654012999996, 2023-07-02T14:05:06.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998, 2023-07-02T14:05:06.579 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.579 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:06.579 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.579 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:06.579 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:06.579 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4088 2023-07-02T14:05:06.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.580 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:05:06.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:05:06.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:06.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:05:06.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00038503499999999999 2023-07-02T14:05:06.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797629+0000", 2023-07-02T14:05:06.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8739999999999995e-06 2023-07-02T14:05:06.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797637+0000", 2023-07-02T14:05:06.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7160000000000001e-06 2023-07-02T14:05:06.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797640+0000", 2023-07-02T14:05:06.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4342999999999998e-05 2023-07-02T14:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797695+0000", 2023-07-02T14:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0939999999999996e-06 2023-07-02T14:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797701+0000", 2023-07-02T14:05:06.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35408540799999999 2023-07-02T14:05:06.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151786+0000", 2023-07-02T14:05:06.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1953000000000006e-05 2023-07-02T14:05:06.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:06.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151868+0000", 2023-07-02T14:05:06.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015162800000000001 2023-07-02T14:05:06.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:06.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.152020+0000", 2023-07-02T14:05:06.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054561870000000004 2023-07-02T14:05:06.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:06.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.157476+0000", 2023-07-02T14:05:06.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059153199999999997 2023-07-02T14:05:06.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:06.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158068+0000", 2023-07-02T14:05:06.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2799e-05 2023-07-02T14:05:06.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:06.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158090+0000", 2023-07-02T14:05:06.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7235999999999999e-05 2023-07-02T14:05:06.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:06.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158108+0000", 2023-07-02T14:05:06.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998 2023-07-02T14:05:06.589 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.589 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.590 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.590 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4110 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:06.590 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:05:06.590 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.600252736000002, 2023-07-02T14:05:06.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864, 2023-07-02T14:05:06.590 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.591 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:06.591 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.591 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:06.591 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:06.591 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4110 2023-07-02T14:05:06.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.591 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:05:06.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:05:06.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:06.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:05:06.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9331e-05 2023-07-02T14:05:06.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797684+0000", 2023-07-02T14:05:06.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.147e-06 2023-07-02T14:05:06.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797689+0000", 2023-07-02T14:05:06.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5020000000000001e-06 2023-07-02T14:05:06.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797693+0000", 2023-07-02T14:05:06.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5254999999999999e-05 2023-07-02T14:05:06.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797748+0000", 2023-07-02T14:05:06.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5520000000000005e-06 2023-07-02T14:05:06.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797757+0000", 2023-07-02T14:05:06.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35674245700000001 2023-07-02T14:05:06.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154499+0000", 2023-07-02T14:05:06.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0090000000000001e-06 2023-07-02T14:05:06.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:06.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154506+0000", 2023-07-02T14:05:06.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864 2023-07-02T14:05:06.598 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.598 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.598 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.599 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:06.599 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:05:06.599 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 88.600200650000005, 2023-07-02T14:05:06.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998, 2023-07-02T14:05:06.599 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:06.599 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:06.599 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:06.599 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:06.600 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:06.600 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:05:06.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.600 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:06.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:06.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:05:06.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:06.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:06.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:05:06.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:06.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:06.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:05:06.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001714819 2023-07-02T14:05:06.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:06.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799412+0000", 2023-07-02T14:05:06.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7779999999999996e-06 2023-07-02T14:05:06.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:06.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799421+0000", 2023-07-02T14:05:06.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3510000000000002e-06 2023-07-02T14:05:06.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:06.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799424+0000", 2023-07-02T14:05:06.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0275999999999999e-05 2023-07-02T14:05:06.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799464+0000", 2023-07-02T14:05:06.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8389999999999996e-06 2023-07-02T14:05:06.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:06.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799473+0000", 2023-07-02T14:05:06.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35507638800000002 2023-07-02T14:05:06.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:06.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154550+0000", 2023-07-02T14:05:06.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6309999999999998e-06 2023-07-02T14:05:06.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:06.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:06.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:06.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154559+0000", 2023-07-02T14:05:06.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998 2023-07-02T14:05:06.606 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.606 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.607 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.607 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:06.607 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:06.607 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:06.608 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:05:06.609 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:06.607+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:06.709 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:06.708+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:06.779 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:06.779 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:06.779 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:05:06.779 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:06.790 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:05:06.810 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:06.809+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:06.911 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:06.910+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:06.956 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:06.956 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:06.956 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:05:06.956 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:05:06.957 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:06.967 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:05:07.011 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:07.010+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:07.102 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:07.103 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:05:07.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:05:07.103 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:07.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.103 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:05:07.104 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:07.104 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 125.99292084299999, 2023-07-02T14:05:07.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:05:07.104 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:07.104 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:07.104 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:07.104 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:07.104 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:07.105 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:05:07.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.105 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:07.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:07.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:07.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:07.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:07.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:07.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:07.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:07.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:05:07.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:05:07.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:07.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:05:07.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:05:07.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:07.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:05:07.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:05:07.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:07.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:05:07.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:05:07.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:05:07.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:05:07.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:07.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:05:07.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:05:07.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:05:07.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:05:07.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:07.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:05:07.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:05:07.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:07.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:05:07.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:05:07.112 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.112 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.112 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.113 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:05:07.113 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:07.113 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 125.99286010500001, 2023-07-02T14:05:07.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:05:07.113 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:07.113 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:07.113 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:07.114 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:07.114 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:07.114 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:05:07.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.114 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:07.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:07.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:07.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:07.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:07.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:07.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:07.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:07.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:05:07.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:05:07.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:07.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:05:07.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:05:07.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:07.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:05:07.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:05:07.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:07.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:05:07.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:05:07.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:05:07.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:05:07.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:07.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:05:07.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:05:07.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:05:07.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:05:07.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:07.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:05:07.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:05:07.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:07.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:05:07.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:05:07.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:07.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:05:07.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:05:07.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:07.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:05:07.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:05:07.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:07.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:05:07.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:05:07.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:07.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:05:07.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:05:07.124 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.124 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.124 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.124 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:07.124 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:07.124 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.762405775, 2023-07-02T14:05:07.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:05:07.125 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:07.125 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:07.125 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:07.125 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:07.125 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:07.125 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:05:07.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.126 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:07.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:07.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:07.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:07.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:07.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:07.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:07.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:07.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:05:07.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:05:07.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:07.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:05:07.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:05:07.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:07.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:05:07.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:05:07.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:07.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:05:07.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:05:07.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:05:07.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:05:07.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:07.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:05:07.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:05:07.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:07.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:05:07.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:05:07.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:07.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:05:07.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:05:07.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:07.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:05:07.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:05:07.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:07.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:05:07.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:05:07.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:07.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:05:07.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:05:07.134 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.134 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.134 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.135 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2950 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 1507328~786432] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:07.135 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:07.135 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.625129513, 2023-07-02T14:05:07.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999, 2023-07-02T14:05:07.135 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:07.135 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:07.135 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:07.135 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:07.136 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:07.136 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2950 2023-07-02T14:05:07.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.136 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:07.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:07.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:07.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:07.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:07.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:07.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:07.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:07.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:07.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001147854 2023-07-02T14:05:07.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:07.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476614+0000", 2023-07-02T14:05:07.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.802e-05 2023-07-02T14:05:07.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:07.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476632+0000", 2023-07-02T14:05:07.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1860000000000003e-06 2023-07-02T14:05:07.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:07.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476639+0000", 2023-07-02T14:05:07.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061211240000000004 2023-07-02T14:05:07.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482760+0000", 2023-07-02T14:05:07.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8833999999999996e-05 2023-07-02T14:05:07.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:07.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482859+0000", 2023-07-02T14:05:07.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019270600000000001 2023-07-02T14:05:07.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:07.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.483051+0000", 2023-07-02T14:05:07.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060385630000000003 2023-07-02T14:05:07.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:07.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489090+0000", 2023-07-02T14:05:07.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024506918999999999 2023-07-02T14:05:07.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:07.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513597+0000", 2023-07-02T14:05:07.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9672999999999995e-05 2023-07-02T14:05:07.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:07.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513667+0000", 2023-07-02T14:05:07.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024329500000000001 2023-07-02T14:05:07.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:07.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513910+0000", 2023-07-02T14:05:07.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999 2023-07-02T14:05:07.144 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.145 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.145 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.145 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2951 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2686976~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:07.145 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:05:07.145 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.623948735, 2023-07-02T14:05:07.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999, 2023-07-02T14:05:07.146 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:07.146 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:07.146 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:07.146 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:07.146 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:07.146 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2951 2023-07-02T14:05:07.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.146 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:07.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:07.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:05:07.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:07.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:07.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:05:07.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:07.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:07.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476646+0000", 2023-07-02T14:05:07.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7639000000000003e-05 2023-07-02T14:05:07.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476684+0000", 2023-07-02T14:05:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3508000000000002e-05 2023-07-02T14:05:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476747+0000", 2023-07-02T14:05:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8499999999999994e-06 2023-07-02T14:05:07.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:07.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476755+0000", 2023-07-02T14:05:07.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078346160000000008 2023-07-02T14:05:07.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484590+0000", 2023-07-02T14:05:07.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7206000000000004e-05 2023-07-02T14:05:07.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:07.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484687+0000", 2023-07-02T14:05:07.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016243099999999999 2023-07-02T14:05:07.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:07.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484849+0000", 2023-07-02T14:05:07.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042482860000000004 2023-07-02T14:05:07.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:07.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489097+0000", 2023-07-02T14:05:07.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.02489504 2023-07-02T14:05:07.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:07.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513992+0000", 2023-07-02T14:05:07.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9972000000000001e-05 2023-07-02T14:05:07.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:07.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514052+0000", 2023-07-02T14:05:07.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149264 2023-07-02T14:05:07.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:07.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514202+0000", 2023-07-02T14:05:07.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999 2023-07-02T14:05:07.155 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.155 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.155 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.155 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2952 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2883584~12288] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:07.155 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:07.156 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.62383490000001, 2023-07-02T14:05:07.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651, 2023-07-02T14:05:07.156 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:07.156 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:07.156 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:07.156 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:07.156 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:07.156 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2952 2023-07-02T14:05:07.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.157 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:07.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:07.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:07.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:07.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:07.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:07.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:07.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:07.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476760+0000", 2023-07-02T14:05:07.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3664e-05 2023-07-02T14:05:07.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:07.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476774+0000", 2023-07-02T14:05:07.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.166e-06 2023-07-02T14:05:07.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:07.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476783+0000", 2023-07-02T14:05:07.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8720000000000001e-06 2023-07-02T14:05:07.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:07.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476788+0000", 2023-07-02T14:05:07.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087180529999999999 2023-07-02T14:05:07.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485506+0000", 2023-07-02T14:05:07.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5680999999999994e-05 2023-07-02T14:05:07.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:07.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485602+0000", 2023-07-02T14:05:07.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000222365 2023-07-02T14:05:07.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:07.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485824+0000", 2023-07-02T14:05:07.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003277379 2023-07-02T14:05:07.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:07.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489101+0000", 2023-07-02T14:05:07.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025151739999999999 2023-07-02T14:05:07.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:07.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514253+0000", 2023-07-02T14:05:07.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4749000000000002e-05 2023-07-02T14:05:07.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:07.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514288+0000", 2023-07-02T14:05:07.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130634 2023-07-02T14:05:07.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:07.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514419+0000", 2023-07-02T14:05:07.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651 2023-07-02T14:05:07.165 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.165 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.165 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.166 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9620 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 589824~65536] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:05:07.166 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:07.166 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 42.078391177, 2023-07-02T14:05:07.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001, 2023-07-02T14:05:07.166 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:07.166 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:07.166 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:07.167 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:07.167 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:07.167 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9620 2023-07-02T14:05:07.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.167 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:07.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:07.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:07.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:07.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:07.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:07.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:07.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:07.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022203+0000", 2023-07-02T14:05:07.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0346000000000001e-05 2023-07-02T14:05:07.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:07.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022263+0000", 2023-07-02T14:05:07.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4657999999999999e-05 2023-07-02T14:05:07.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:07.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022278+0000", 2023-07-02T14:05:07.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:05:07.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:07.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022285+0000", 2023-07-02T14:05:07.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0096000000000003e-05 2023-07-02T14:05:07.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022335+0000", 2023-07-02T14:05:07.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2334000000000001e-05 2023-07-02T14:05:07.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:07.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022347+0000", 2023-07-02T14:05:07.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095653701999999993 2023-07-02T14:05:07.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118001+0000", 2023-07-02T14:05:07.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000185797 2023-07-02T14:05:07.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:07.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118187+0000", 2023-07-02T14:05:07.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026397900000000001 2023-07-02T14:05:07.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:07.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118451+0000", 2023-07-02T14:05:07.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051832800000000002 2023-07-02T14:05:07.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:07.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.123634+0000", 2023-07-02T14:05:07.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001227332 2023-07-02T14:05:07.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:07.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124861+0000", 2023-07-02T14:05:07.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9493999999999998e-05 2023-07-02T14:05:07.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:07.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124901+0000", 2023-07-02T14:05:07.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0462000000000001e-05 2023-07-02T14:05:07.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:07.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124931+0000", 2023-07-02T14:05:07.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001 2023-07-02T14:05:07.177 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.177 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.177 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.177 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9621 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 655360~1966080] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:05:07.178 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:07.178 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 42.078203795, 2023-07-02T14:05:07.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999, 2023-07-02T14:05:07.178 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:07.178 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:07.178 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:07.178 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:07.178 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:07.179 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9621 2023-07-02T14:05:07.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.179 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:07.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:07.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:07.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:07.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:07.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:07.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:07.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:07.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022391+0000", 2023-07-02T14:05:07.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032053939999999999 2023-07-02T14:05:07.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:07.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025597+0000", 2023-07-02T14:05:07.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4307999999999999e-05 2023-07-02T14:05:07.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:07.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025611+0000", 2023-07-02T14:05:07.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.032e-06 2023-07-02T14:05:07.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:07.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025617+0000", 2023-07-02T14:05:07.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8977999999999998e-05 2023-07-02T14:05:07.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025656+0000", 2023-07-02T14:05:07.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:05:07.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:07.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025663+0000", 2023-07-02T14:05:07.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095220739999999998 2023-07-02T14:05:07.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120884+0000", 2023-07-02T14:05:07.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2525000000000001e-05 2023-07-02T14:05:07.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:07.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120956+0000", 2023-07-02T14:05:07.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016232400000000001 2023-07-02T14:05:07.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:07.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.121119+0000", 2023-07-02T14:05:07.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044007380000000004 2023-07-02T14:05:07.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:07.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.125519+0000", 2023-07-02T14:05:07.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000797209 2023-07-02T14:05:07.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:07.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126317+0000", 2023-07-02T14:05:07.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1228000000000004e-05 2023-07-02T14:05:07.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:07.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126378+0000", 2023-07-02T14:05:07.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978000000000003e-05 2023-07-02T14:05:07.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:07.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126430+0000", 2023-07-02T14:05:07.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999 2023-07-02T14:05:07.188 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.188 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.189 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.189 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:11434 2.1 2:8269d003:::rbd_data.125de1449605.00000000000000ce:head [write 65536~4128768] snapc 1a=[1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e107)", 2023-07-02T14:05:07.189 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:05:07.189 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.182765525999997, 2023-07-02T14:05:07.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036409966000000002, 2023-07-02T14:05:07.189 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:07.190 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:07.190 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:07.190 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:07.190 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:07.190 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11434 2023-07-02T14:05:07.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.190 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:07.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:07.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:05:07.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:07.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:07.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:05:07.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:07.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:07.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917828+0000", 2023-07-02T14:05:07.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052044650000000001 2023-07-02T14:05:07.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:07.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923033+0000", 2023-07-02T14:05:07.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0097000000000001e-05 2023-07-02T14:05:07.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:07.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923053+0000", 2023-07-02T14:05:07.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.49e-06 2023-07-02T14:05:07.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:07.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923061+0000", 2023-07-02T14:05:07.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0090999999999998e-05 2023-07-02T14:05:07.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923121+0000", 2023-07-02T14:05:07.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000114095 2023-07-02T14:05:07.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:07.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923236+0000", 2023-07-02T14:05:07.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020094199999999999 2023-07-02T14:05:07.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:07.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923436+0000", 2023-07-02T14:05:07.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088693520000000005 2023-07-02T14:05:07.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:07.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.932306+0000", 2023-07-02T14:05:07.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021874112000000001 2023-07-02T14:05:07.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:07.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954180+0000", 2023-07-02T14:05:07.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4433e-05 2023-07-02T14:05:07.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:07.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954214+0000", 2023-07-02T14:05:07.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5884000000000002e-05 2023-07-02T14:05:07.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:07.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954240+0000", 2023-07-02T14:05:07.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036409966000000002 2023-07-02T14:05:07.199 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.199 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.199 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.199 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:189 2.3 2:c9621172:::rbd_data.125de1449605.00000000000000c0:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:07.199 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:07.199 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 33.927916502999999, 2023-07-02T14:05:07.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001, 2023-07-02T14:05:07.200 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:07.200 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:07.200 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:07.200 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:07.200 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:07.200 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:05:07.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.201 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:07.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:07.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:07.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:07.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:07.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:07.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:07.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:07.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:07.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2023-07-02T14:05:07.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:07.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172682+0000", 2023-07-02T14:05:07.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0159999999999997e-06 2023-07-02T14:05:07.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:07.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172688+0000", 2023-07-02T14:05:07.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0010000000000003e-06 2023-07-02T14:05:07.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:07.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172692+0000", 2023-07-02T14:05:07.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026444333 2023-07-02T14:05:07.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199136+0000", 2023-07-02T14:05:07.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2230000000000004e-05 2023-07-02T14:05:07.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:07.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199218+0000", 2023-07-02T14:05:07.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012043595000000001 2023-07-02T14:05:07.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:07.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211262+0000", 2023-07-02T14:05:07.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001 2023-07-02T14:05:07.206 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.206 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.207 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.207 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:199 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:07.207 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:07.207 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 33.927872960000002, 2023-07-02T14:05:07.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002, 2023-07-02T14:05:07.207 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:07.208 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:07.208 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:07.208 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:07.208 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:07.208 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 199 2023-07-02T14:05:07.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.208 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:07.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:07.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:07.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:07.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:07.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:07.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:07.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:07.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172722+0000", 2023-07-02T14:05:07.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1250000000000001e-06 2023-07-02T14:05:07.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:07.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172725+0000", 2023-07-02T14:05:07.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5670000000000002e-06 2023-07-02T14:05:07.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:07.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172731+0000", 2023-07-02T14:05:07.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9310000000000001e-06 2023-07-02T14:05:07.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:07.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172735+0000", 2023-07-02T14:05:07.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030699852999999999 2023-07-02T14:05:07.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203435+0000", 2023-07-02T14:05:07.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6827e-05 2023-07-02T14:05:07.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:07.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203532+0000", 2023-07-02T14:05:07.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076955230000000001 2023-07-02T14:05:07.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:07.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211227+0000", 2023-07-02T14:05:07.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002 2023-07-02T14:05:07.214 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.214 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.214 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.215 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:281 2.1 2:901d5a95:::rbd_data.125de1449605.0000000000000118:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:07.215 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:07.215 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 33.545127684000001, 2023-07-02T14:05:07.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002, 2023-07-02T14:05:07.215 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:07.215 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:05:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:07.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:07.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:07.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0469999999999998e-06 2023-07-02T14:05:07.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:07.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555471+0000", 2023-07-02T14:05:07.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8860000000000001e-06 2023-07-02T14:05:07.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:07.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555476+0000", 2023-07-02T14:05:07.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3160000000000002e-06 2023-07-02T14:05:07.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:07.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555483+0000", 2023-07-02T14:05:07.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989525000000001 2023-07-02T14:05:07.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591472+0000", 2023-07-02T14:05:07.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3708999999999996e-05 2023-07-02T14:05:07.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:07.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591536+0000", 2023-07-02T14:05:07.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055109720000000003 2023-07-02T14:05:07.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:07.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.597047+0000", 2023-07-02T14:05:07.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002 2023-07-02T14:05:07.222 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.222 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.222 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.222 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12536 2.1 2:8b20f686:::rbd_data.125de1449605.0000000000000056:head [write 3670016~36864] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:05:07.223 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.205749+0000", 2023-07-02T14:05:07.223 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.89484672, 2023-07-02T14:05:07.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036263036999999998, 2023-07-02T14:05:07.223 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:07.223 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:07.223 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:07.223 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:07.224 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:07.224 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12536 2023-07-02T14:05:07.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.224 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:07.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:07.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205749+0000", 2023-07-02T14:05:07.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:07.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:07.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205749+0000", 2023-07-02T14:05:07.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:07.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:07.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205748+0000", 2023-07-02T14:05:07.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1784999999999999e-05 2023-07-02T14:05:07.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:07.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205770+0000", 2023-07-02T14:05:07.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0659999999999993e-06 2023-07-02T14:05:07.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:07.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205779+0000", 2023-07-02T14:05:07.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.622e-06 2023-07-02T14:05:07.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:07.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205784+0000", 2023-07-02T14:05:07.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018333009000000001 2023-07-02T14:05:07.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.224117+0000", 2023-07-02T14:05:07.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018818400000000001 2023-07-02T14:05:07.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:07.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.224305+0000", 2023-07-02T14:05:07.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00032545700000000002 2023-07-02T14:05:07.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:07.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.224631+0000", 2023-07-02T14:05:07.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049349700000000003 2023-07-02T14:05:07.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:07.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.229566+0000", 2023-07-02T14:05:07.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012353153 2023-07-02T14:05:07.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:07.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.241919+0000", 2023-07-02T14:05:07.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2751999999999999e-05 2023-07-02T14:05:07.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:07.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.241971+0000", 2023-07-02T14:05:07.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0632e-05 2023-07-02T14:05:07.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:07.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.242012+0000", 2023-07-02T14:05:07.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036263036999999998 2023-07-02T14:05:07.232 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.232 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.233 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.233 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12883 2.1 2:9ffc48e1:::rbd_data.125de1449605.0000000000000074:head [write 4173824~20480] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:05:07.233 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:07.233 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.454525211, 2023-07-02T14:05:07.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037871393000000003, 2023-07-02T14:05:07.233 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:07.234 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:07.234 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:07.234 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:07.234 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:07.234 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12883 2023-07-02T14:05:07.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.234 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:07.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:07.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:07.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:07.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:07.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:07.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:05:07.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:07.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646068+0000", 2023-07-02T14:05:07.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4935999999999999e-05 2023-07-02T14:05:07.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:07.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646103+0000", 2023-07-02T14:05:07.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.092e-05 2023-07-02T14:05:07.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:07.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646124+0000", 2023-07-02T14:05:07.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1519999999999998e-06 2023-07-02T14:05:07.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:07.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646133+0000", 2023-07-02T14:05:07.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012270200000000001 2023-07-02T14:05:07.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646255+0000", 2023-07-02T14:05:07.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014495400000000001 2023-07-02T14:05:07.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:07.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646400+0000", 2023-07-02T14:05:07.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000221626 2023-07-02T14:05:07.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:07.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646622+0000", 2023-07-02T14:05:07.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078929050000000004 2023-07-02T14:05:07.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:07.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.654515+0000", 2023-07-02T14:05:07.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029307343999999999 2023-07-02T14:05:07.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:07.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683822+0000", 2023-07-02T14:05:07.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8298000000000002e-05 2023-07-02T14:05:07.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:07.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683891+0000", 2023-07-02T14:05:07.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1381999999999997e-05 2023-07-02T14:05:07.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:07.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683942+0000", 2023-07-02T14:05:07.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037871393000000003 2023-07-02T14:05:07.243 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.243 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.243 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.243 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12985 2.7 2:eeddf24b:::rbd_data.125de1449605.000000000000007e:head [write 65536~4128768] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:05:07.243 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:07.244 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.317600737999999, 2023-07-02T14:05:07.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999, 2023-07-02T14:05:07.244 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:07.244 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:07.244 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:07.244 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:07.244 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:07.244 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12985 2023-07-02T14:05:07.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.245 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:07.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:07.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:07.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:07.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:07.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:07.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:07.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:07.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782994+0000", 2023-07-02T14:05:07.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019254741999999998 2023-07-02T14:05:07.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:07.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802248+0000", 2023-07-02T14:05:07.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0451e-05 2023-07-02T14:05:07.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:07.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802279+0000", 2023-07-02T14:05:07.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3318e-05 2023-07-02T14:05:07.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:07.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802292+0000", 2023-07-02T14:05:07.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109403 2023-07-02T14:05:07.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802402+0000", 2023-07-02T14:05:07.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012520599999999999 2023-07-02T14:05:07.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:07.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802527+0000", 2023-07-02T14:05:07.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187768 2023-07-02T14:05:07.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:05:07.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802715+0000", 2023-07-02T14:05:07.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013192822 2023-07-02T14:05:07.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:07.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.815907+0000", 2023-07-02T14:05:07.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0086451870000000004 2023-07-02T14:05:07.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:07.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824553+0000", 2023-07-02T14:05:07.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6759999999999999e-05 2023-07-02T14:05:07.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:07.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824609+0000", 2023-07-02T14:05:07.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011314700000000001 2023-07-02T14:05:07.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:07.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824722+0000", 2023-07-02T14:05:07.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999 2023-07-02T14:05:07.253 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.253 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.253 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.254 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:165 2.3 2:c9c6393a:::rbd_data.125de1449605.000000000000002c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:07.254 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:07.254 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.260817920999999, 2023-07-02T14:05:07.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002, 2023-07-02T14:05:07.254 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:07.254 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:07.254 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:07.255 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:07.255 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:07.255 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 165 2023-07-02T14:05:07.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.255 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:07.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:07.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:07.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:07.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:07.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:07.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:07.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:07.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839777+0000", 2023-07-02T14:05:07.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6740000000000001e-06 2023-07-02T14:05:07.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:07.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839780+0000", 2023-07-02T14:05:07.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9030000000000004e-06 2023-07-02T14:05:07.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:07.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839788+0000", 2023-07-02T14:05:07.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1799999999999998e-06 2023-07-02T14:05:07.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:07.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839792+0000", 2023-07-02T14:05:07.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.031592227000000001 2023-07-02T14:05:07.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871384+0000", 2023-07-02T14:05:07.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013600099999999999 2023-07-02T14:05:07.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:07.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871520+0000", 2023-07-02T14:05:07.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082817089999999999 2023-07-02T14:05:07.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:07.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.879802+0000", 2023-07-02T14:05:07.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002 2023-07-02T14:05:07.261 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.261 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.261 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.261 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:252 2.7 2:f1d0abf0:::rbd_data.125de1449605.0000000000000083:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:07.262 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:07.262 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.8445329509999997, 2023-07-02T14:05:07.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997, 2023-07-02T14:05:07.262 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:07.262 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:07.262 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:07.262 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:07.262 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:07.263 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 252 2023-07-02T14:05:07.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.263 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:07.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:07.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:07.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:07.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:07.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:07.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:07.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:07.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256062+0000", 2023-07-02T14:05:07.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0149999999999999e-06 2023-07-02T14:05:07.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:07.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256065+0000", 2023-07-02T14:05:07.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5269999999999998e-06 2023-07-02T14:05:07.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:07.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256071+0000", 2023-07-02T14:05:07.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:05:07.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:07.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256074+0000", 2023-07-02T14:05:07.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028852909 2023-07-02T14:05:07.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.284927+0000", 2023-07-02T14:05:07.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168274 2023-07-02T14:05:07.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:07.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.285096+0000", 2023-07-02T14:05:07.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011036300000000001 2023-07-02T14:05:07.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:07.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296132+0000", 2023-07-02T14:05:07.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997 2023-07-02T14:05:07.269 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.269 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.269 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.270 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:261 2.1 2:971731cb:::rbd_data.125de1449605.000000000000008c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:07.270 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:07.270 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.8444374759999995, 2023-07-02T14:05:07.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984, 2023-07-02T14:05:07.270 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:07.270 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:07.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:07.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:07.271 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:07.271 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 261 2023-07-02T14:05:07.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.271 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:07.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:07.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:07.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:07.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:07.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:07.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:07.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:07.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:07.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2819999999999999e-06 2023-07-02T14:05:07.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:07.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256160+0000", 2023-07-02T14:05:07.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2020000000000003e-06 2023-07-02T14:05:07.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:07.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256165+0000", 2023-07-02T14:05:07.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2100000000000003e-06 2023-07-02T14:05:07.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:07.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256170+0000", 2023-07-02T14:05:07.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025871715999999999 2023-07-02T14:05:07.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282041+0000", 2023-07-02T14:05:07.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132364 2023-07-02T14:05:07.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:07.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282174+0000", 2023-07-02T14:05:07.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014044566 2023-07-02T14:05:07.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:07.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296218+0000", 2023-07-02T14:05:07.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984 2023-07-02T14:05:07.277 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.277 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.277 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.277 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:264 2.3 2:dbf06c9d:::rbd_data.125de1449605.000000000000008f:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:07.277 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:07.278 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.8444033579999992, 2023-07-02T14:05:07.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003, 2023-07-02T14:05:07.278 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:07.278 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:07.278 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:07.278 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:07.278 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:07.279 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 264 2023-07-02T14:05:07.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.279 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:07.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:07.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:07.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:07.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:07.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:07.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:07.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:07.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:07.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5749999999999999e-06 2023-07-02T14:05:07.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:07.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256195+0000", 2023-07-02T14:05:07.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.327e-06 2023-07-02T14:05:07.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:07.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256200+0000", 2023-07-02T14:05:07.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0580000000000002e-06 2023-07-02T14:05:07.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:07.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256203+0000", 2023-07-02T14:05:07.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035707524999999997 2023-07-02T14:05:07.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.291911+0000", 2023-07-02T14:05:07.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014269799999999999 2023-07-02T14:05:07.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:07.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.292053+0000", 2023-07-02T14:05:07.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062760309999999996 2023-07-02T14:05:07.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:07.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.298329+0000", 2023-07-02T14:05:07.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003 2023-07-02T14:05:07.285 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.285 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.285 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.285 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:341 2.3 2:cee258bd:::rbd_data.125de1449605.00000000000000dc:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:07.285 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:07.285 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.4639160310000001, 2023-07-02T14:05:07.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997, 2023-07-02T14:05:07.286 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:07.286 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:07.286 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:07.286 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:07.286 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:07.286 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 341 2023-07-02T14:05:07.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.287 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:07.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:07.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:07.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:07.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:07.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:07.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:07.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:07.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636679+0000", 2023-07-02T14:05:07.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6240000000000002e-06 2023-07-02T14:05:07.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:07.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636682+0000", 2023-07-02T14:05:07.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6910000000000002e-06 2023-07-02T14:05:07.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:07.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636688+0000", 2023-07-02T14:05:07.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3469999999999999e-06 2023-07-02T14:05:07.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:07.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636691+0000", 2023-07-02T14:05:07.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035625560000000001 2023-07-02T14:05:07.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:07.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672317+0000", 2023-07-02T14:05:07.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108391 2023-07-02T14:05:07.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:07.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672425+0000", 2023-07-02T14:05:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012104976 2023-07-02T14:05:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.684530+0000", 2023-07-02T14:05:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997 2023-07-02T14:05:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.292 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:07.293 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:07.293 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:07.294 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:05:07.294 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 4 2023-07-02T14:05:07.295 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:05:07.295 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:07.293+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:07.395 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:07.394+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:07.425 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:07.425 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:07.425 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:07.425 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:07.436 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:05:07.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:05:07.495 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:07.495+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:07.577 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:07.577 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:07.577 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:07.577 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:07.578 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:07.589 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:05:07.624 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:05:07.630 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:07.629+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:07.643 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:07.661 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:07.681 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:07.697 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:07.696+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:07.700 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:07.718 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:07.720 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:05:07.720 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:05:07.720 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:05:07.720 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:05:07.720 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:05:07.720 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:05:07.762 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:07.762 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:07.773 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:05:07.816 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:07.816+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:07.898 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:07.897+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:07.909 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:07.909 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:07.921 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:05:07.998 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:07.997+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:08.063 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:05:08.096 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:08.096 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:08.099 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:08.098+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:08.108 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:05:08.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:05:08.200 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:08.199+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:08.201 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:08.201 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:08.213 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-07-02T14:05:08.283 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:08.283 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:08.294 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:05:08.300 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:08.299+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:08.401 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:08.400+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:08.435 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:08.435 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:08.447 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:05:08.529 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:08.529+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:08.603 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:08.602+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:08.635 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:08.635 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:08.647 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:05:08.730 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:08.730+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:08.805 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:08.804+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:08.825 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:08.825 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:08.836 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:05:08.905 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:08.904+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:08.976 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:08.976 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:08.987 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:05:09.030 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:09.029+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:05:09.107 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:09.106+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.0 to 1 (10000) 2023-07-02T14:05:09.118 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:09.119 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:09.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-07-02T14:05:09.130 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:05:09.214 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:09.213+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:09.307 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:09.307 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:09.308 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:09.307+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:09.319 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:05:09.409 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:09.408+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:09.510 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:09.509+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:09.529 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:09.529 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:09.550 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:05:09.611 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:09.610+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:09.712 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:09.711+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:09.732 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:05:09.763 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:09.763 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:09.784 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:05:09.826 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:09.825+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:09.914 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:09.914+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:09.923 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:05:09.942 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:09.961 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:09.968 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:09.968 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:05:09.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:05:09.969 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:09.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.969 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:05:09.969 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:09.969 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 120.51388162400001, 2023-07-02T14:05:09.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:05:09.970 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:09.970 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:09.970 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:09.970 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:09.970 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:09.970 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:05:09.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.970 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:09.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:09.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:09.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:09.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:09.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:09.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:09.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:09.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:05:09.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:05:09.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:09.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:05:09.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:05:09.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:09.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:05:09.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:05:09.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:09.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:05:09.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:05:09.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:09.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:05:09.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:05:09.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:09.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:05:09.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:05:09.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:09.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:05:09.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:05:09.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:09.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:05:09.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:05:09.977 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:09.977 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:09.977 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:09.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.978 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:05:09.978 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:09.978 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 119.643188616, 2023-07-02T14:05:09.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:05:09.978 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:09.978 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:09.978 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:09.978 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:05:09.979 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:05:09.979 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:05:09.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.979 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:09.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:09.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:09.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:09.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:09.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:09.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:09.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:09.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:05:09.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:05:09.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:09.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:05:09.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:05:09.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:09.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:05:09.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:05:09.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:09.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:05:09.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:05:09.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:09.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:05:09.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:05:09.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:09.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:05:09.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:05:09.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:09.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:05:09.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:05:09.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:09.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:05:09.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:05:09.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:09.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:05:09.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:05:09.986 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:09.986 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:09.986 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:09.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.987 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:09.987 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:09.987 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 119.643063895, 2023-07-02T14:05:09.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:05:09.987 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:09.987 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:09.987 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:09.987 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:09.988 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:09.988 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:05:09.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.988 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:09.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:09.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:09.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:09.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:09.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:09.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:09.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:09.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:05:09.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:05:09.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:09.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:05:09.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:05:09.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:09.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:05:09.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:05:09.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:09.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:05:09.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:05:09.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:09.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:05:09.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:05:09.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:09.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:05:09.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:05:09.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:09.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:05:09.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:05:09.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:09.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:05:09.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:05:09.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:09.994 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:09.995 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:09.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.995 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:09.995 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:09.995 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 119.63621401100001, 2023-07-02T14:05:09.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:05:09.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:09.996 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:09.996 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:09.996 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:09.996 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:09.996 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:05:09.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.996 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:09.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:09.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:09.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:09.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:09.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:09.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:05:09.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:09.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:05:09.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:05:09.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:09.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:05:09.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:05:09.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:09.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:09.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:09.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:05:09.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:05:10.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:05:10.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:05:10.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:05:10.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:05:10.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:05:10.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:05:10.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:05:10.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:05:10.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:05:10.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:05:10.003 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.003 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.003 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.003 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:10.003 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:10.004 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 119.636151086, 2023-07-02T14:05:10.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:05:10.004 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.004 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:10.004 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.004 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.004 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.005 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:05:10.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.005 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:10.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:10.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:10.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:05:10.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:05:10.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:05:10.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:05:10.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:05:10.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:05:10.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:05:10.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:05:10.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:05:10.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:05:10.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:05:10.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:05:10.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:05:10.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:05:10.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:05:10.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:05:10.012 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.012 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.012 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.013 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:10.013 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:10.013 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 119.63585066500001, 2023-07-02T14:05:10.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:05:10.013 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.013 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.013 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.014 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.014 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.014 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:05:10.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.014 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:10.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:10.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:05:10.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:05:10.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:05:10.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:05:10.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:05:10.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:05:10.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:05:10.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:05:10.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:05:10.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:05:10.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:05:10.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:05:10.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:05:10.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:05:10.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:05:10.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:05:10.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:05:10.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:05:10.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:05:10.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:05:10.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:05:10.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:05:10.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:05:10.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:05:10.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:05:10.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:05:10.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:05:10.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:05:10.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:05:10.025 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.026 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.026 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.026 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:05:10.026 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:10.026 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 119.635326165, 2023-07-02T14:05:10.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:05:10.027 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.027 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:10.027 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.027 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.027 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.027 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:05:10.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.028 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:10.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:10.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:10.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:05:10.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:05:10.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:05:10.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:05:10.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:05:10.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:05:10.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:05:10.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:05:10.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:05:10.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:05:10.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:05:10.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:05:10.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:05:10.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:05:10.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:05:10.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:05:10.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:05:10.035 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.035 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.035 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.035 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:05:10.035 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:10.035 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 92.511307040000005, 2023-07-02T14:05:10.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:05:10.036 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.036 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:10.036 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.036 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.036 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.036 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:05:10.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.037 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:10.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:10.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:10.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:05:10.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:05:10.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:05:10.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:05:10.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:05:10.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:05:10.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:05:10.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:05:10.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:05:10.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:05:10.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:05:10.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:05:10.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:05:10.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:05:10.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:05:10.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:05:10.043 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.043 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.043 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.044 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:10.044 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:10.044 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 92.185824335999996, 2023-07-02T14:05:10.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:05:10.044 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.044 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.044 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.044 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.045 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.045 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:05:10.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.045 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:10.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:10.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:10.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:05:10.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:05:10.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:05:10.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:05:10.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:05:10.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:05:10.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:05:10.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:05:10.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:10.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:05:10.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:05:10.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:05:10.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:05:10.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:05:10.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:05:10.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:05:10.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:10.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:05:10.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:05:10.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:05:10.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:05:10.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:05:10.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:05:10.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:05:10.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:05:10.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:05:10.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:05:10.054 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.055 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.055 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.055 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3887 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:10.055 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:05:10.055 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 92.181961146999996, 2023-07-02T14:05:10.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998, 2023-07-02T14:05:10.056 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.056 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.056 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.056 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.056 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.056 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3887 2023-07-02T14:05:10.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.057 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:05:10.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784020+0000", 2023-07-02T14:05:10.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:10.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784019+0000", 2023-07-02T14:05:10.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4073999999999998e-05 2023-07-02T14:05:10.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784063+0000", 2023-07-02T14:05:10.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6630000000000004e-06 2023-07-02T14:05:10.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784070+0000", 2023-07-02T14:05:10.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.286e-06 2023-07-02T14:05:10.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784073+0000", 2023-07-02T14:05:10.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4941999999999998e-05 2023-07-02T14:05:10.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784118+0000", 2023-07-02T14:05:10.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8120000000000008e-06 2023-07-02T14:05:10.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784126+0000", 2023-07-02T14:05:10.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32999709399999999 2023-07-02T14:05:10.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114123+0000", 2023-07-02T14:05:10.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000157257 2023-07-02T14:05:10.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114280+0000", 2023-07-02T14:05:10.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000341152 2023-07-02T14:05:10.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:10.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.114622+0000", 2023-07-02T14:05:10.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019539766 2023-07-02T14:05:10.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.134161+0000", 2023-07-02T14:05:10.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015820796000000002 2023-07-02T14:05:10.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149982+0000", 2023-07-02T14:05:10.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.05e-05 2023-07-02T14:05:10.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150043+0000", 2023-07-02T14:05:10.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6453999999999999e-05 2023-07-02T14:05:10.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150089+0000", 2023-07-02T14:05:10.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36606863499999998 2023-07-02T14:05:10.067 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.068 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.068 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.068 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3939 2.1 2:9ef34a83:::rbd_data.125de1449605.00000000000000b8:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:10.068 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:10.069 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 92.181903638999998, 2023-07-02T14:05:10.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001, 2023-07-02T14:05:10.069 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.069 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.070 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.070 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.070 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.070 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3939 2023-07-02T14:05:10.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.070 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:10.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:10.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:10.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046852830000000002 2023-07-02T14:05:10.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788763+0000", 2023-07-02T14:05:10.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3185e-05 2023-07-02T14:05:10.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788776+0000", 2023-07-02T14:05:10.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8470000000000003e-06 2023-07-02T14:05:10.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788781+0000", 2023-07-02T14:05:10.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7104000000000002e-05 2023-07-02T14:05:10.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788828+0000", 2023-07-02T14:05:10.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3516000000000001e-05 2023-07-02T14:05:10.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788841+0000", 2023-07-02T14:05:10.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33064513600000001 2023-07-02T14:05:10.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119487+0000", 2023-07-02T14:05:10.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9876999999999997e-05 2023-07-02T14:05:10.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119537+0000", 2023-07-02T14:05:10.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014160999999999999 2023-07-02T14:05:10.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:10.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119678+0000", 2023-07-02T14:05:10.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023054392 2023-07-02T14:05:10.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.142733+0000", 2023-07-02T14:05:10.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074065470000000003 2023-07-02T14:05:10.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150139+0000", 2023-07-02T14:05:10.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5025999999999997e-05 2023-07-02T14:05:10.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150204+0000", 2023-07-02T14:05:10.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147999999999997e-05 2023-07-02T14:05:10.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150241+0000", 2023-07-02T14:05:10.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001 2023-07-02T14:05:10.081 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.081 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.081 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.082 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3940 2.1 2:92a6dd3f:::rbd_data.125de1449605.00000000000000be:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:10.082 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:05:10.082 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 92.177192606999995, 2023-07-02T14:05:10.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001, 2023-07-02T14:05:10.082 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.082 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.082 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.083 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.083 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.083 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3940 2023-07-02T14:05:10.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.083 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:05:10.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788789+0000", 2023-07-02T14:05:10.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:10.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788788+0000", 2023-07-02T14:05:10.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0043166560000000003 2023-07-02T14:05:10.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793105+0000", 2023-07-02T14:05:10.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5970000000000001e-05 2023-07-02T14:05:10.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793121+0000", 2023-07-02T14:05:10.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0939999999999998e-06 2023-07-02T14:05:10.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793126+0000", 2023-07-02T14:05:10.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2438000000000002e-05 2023-07-02T14:05:10.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793178+0000", 2023-07-02T14:05:10.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2019e-05 2023-07-02T14:05:10.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793190+0000", 2023-07-02T14:05:10.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33208757799999999 2023-07-02T14:05:10.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125278+0000", 2023-07-02T14:05:10.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5255999999999997e-05 2023-07-02T14:05:10.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125373+0000", 2023-07-02T14:05:10.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015674999999999999 2023-07-02T14:05:10.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:10.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125530+0000", 2023-07-02T14:05:10.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018767308 2023-07-02T14:05:10.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144297+0000", 2023-07-02T14:05:10.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060291959999999997 2023-07-02T14:05:10.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150326+0000", 2023-07-02T14:05:10.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3886999999999993e-05 2023-07-02T14:05:10.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150390+0000", 2023-07-02T14:05:10.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010255 2023-07-02T14:05:10.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150493+0000", 2023-07-02T14:05:10.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36170382000000001 2023-07-02T14:05:10.093 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.093 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.093 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.093 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3947 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 0~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:10.094 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:05:10.094 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 92.172847079999997, 2023-07-02T14:05:10.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999, 2023-07-02T14:05:10.094 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.094 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.094 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.094 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.094 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.095 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3947 2023-07-02T14:05:10.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.095 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:05:10.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793135+0000", 2023-07-02T14:05:10.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:10.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793133+0000", 2023-07-02T14:05:10.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2519999999999999e-05 2023-07-02T14:05:10.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793176+0000", 2023-07-02T14:05:10.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:05:10.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793180+0000", 2023-07-02T14:05:10.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1709999999999998e-06 2023-07-02T14:05:10.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793184+0000", 2023-07-02T14:05:10.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1721999999999999e-05 2023-07-02T14:05:10.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793215+0000", 2023-07-02T14:05:10.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1280000000000008e-06 2023-07-02T14:05:10.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793224+0000", 2023-07-02T14:05:10.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34218549599999998 2023-07-02T14:05:10.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135409+0000", 2023-07-02T14:05:10.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117442 2023-07-02T14:05:10.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135526+0000", 2023-07-02T14:05:10.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000206257 2023-07-02T14:05:10.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:10.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.135733+0000", 2023-07-02T14:05:10.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085700029999999996 2023-07-02T14:05:10.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.144303+0000", 2023-07-02T14:05:10.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006278652 2023-07-02T14:05:10.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150581+0000", 2023-07-02T14:05:10.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4765999999999996e-05 2023-07-02T14:05:10.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150646+0000", 2023-07-02T14:05:10.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4777999999999995e-05 2023-07-02T14:05:10.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150711+0000", 2023-07-02T14:05:10.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35757640099999999 2023-07-02T14:05:10.105 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.105 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.106 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.106 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3980 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 65536~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:10.106 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:05:10.106 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 92.172793197999994, 2023-07-02T14:05:10.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999, 2023-07-02T14:05:10.107 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.107 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.107 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.107 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.107 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.107 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3980 2023-07-02T14:05:10.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.107 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:05:10.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:05:10.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.793188+0000", 2023-07-02T14:05:10.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020244260000000002 2023-07-02T14:05:10.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795212+0000", 2023-07-02T14:05:10.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0005e-05 2023-07-02T14:05:10.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795222+0000", 2023-07-02T14:05:10.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9860000000000002e-06 2023-07-02T14:05:10.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795227+0000", 2023-07-02T14:05:10.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6369999999999999e-05 2023-07-02T14:05:10.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795264+0000", 2023-07-02T14:05:10.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1052e-05 2023-07-02T14:05:10.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795275+0000", 2023-07-02T14:05:10.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.343850459 2023-07-02T14:05:10.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139125+0000", 2023-07-02T14:05:10.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6760000000000004e-05 2023-07-02T14:05:10.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139202+0000", 2023-07-02T14:05:10.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015092799999999999 2023-07-02T14:05:10.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:10.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.139353+0000", 2023-07-02T14:05:10.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011425940000000001 2023-07-02T14:05:10.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150779+0000", 2023-07-02T14:05:10.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033223060000000001 2023-07-02T14:05:10.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154101+0000", 2023-07-02T14:05:10.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9268999999999998e-05 2023-07-02T14:05:10.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154161+0000", 2023-07-02T14:05:10.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6499999999999999e-05 2023-07-02T14:05:10.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154207+0000", 2023-07-02T14:05:10.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36101865999999999 2023-07-02T14:05:10.118 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.118 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.118 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.118 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3982 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1835008~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:10.119 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:05:10.119 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 92.170747911000007, 2023-07-02T14:05:10.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002, 2023-07-02T14:05:10.119 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.119 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.119 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.119 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.119 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.120 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3982 2023-07-02T14:05:10.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.120 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:05:10.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795234+0000", 2023-07-02T14:05:10.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795233+0000", 2023-07-02T14:05:10.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8680999999999998e-05 2023-07-02T14:05:10.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795272+0000", 2023-07-02T14:05:10.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5629999999999999e-06 2023-07-02T14:05:10.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795277+0000", 2023-07-02T14:05:10.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8700000000000001e-06 2023-07-02T14:05:10.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795280+0000", 2023-07-02T14:05:10.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1964000000000001e-05 2023-07-02T14:05:10.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795312+0000", 2023-07-02T14:05:10.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.314e-06 2023-07-02T14:05:10.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795319+0000", 2023-07-02T14:05:10.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34828442599999998 2023-07-02T14:05:10.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143603+0000", 2023-07-02T14:05:10.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2438000000000001e-05 2023-07-02T14:05:10.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143665+0000", 2023-07-02T14:05:10.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3079999999999997e-05 2023-07-02T14:05:10.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:10.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.143759+0000", 2023-07-02T14:05:10.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070633079999999999 2023-07-02T14:05:10.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150822+0000", 2023-07-02T14:05:10.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034430210000000001 2023-07-02T14:05:10.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154265+0000", 2023-07-02T14:05:10.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4249999999999998e-05 2023-07-02T14:05:10.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154309+0000", 2023-07-02T14:05:10.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2020000000000001e-05 2023-07-02T14:05:10.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154351+0000", 2023-07-02T14:05:10.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35911742400000002 2023-07-02T14:05:10.130 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.130 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.130 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.130 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4054 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 1900544~1769472] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:10.130 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:05:10.130 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 92.170696239999998, 2023-07-02T14:05:10.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002, 2023-07-02T14:05:10.131 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.131 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.131 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.131 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.131 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.131 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4054 2023-07-02T14:05:10.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.132 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:05:10.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:05:10.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.795285+0000", 2023-07-02T14:05:10.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018874860000000001 2023-07-02T14:05:10.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797172+0000", 2023-07-02T14:05:10.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3000000000000007e-06 2023-07-02T14:05:10.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797182+0000", 2023-07-02T14:05:10.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6260000000000003e-06 2023-07-02T14:05:10.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797186+0000", 2023-07-02T14:05:10.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9013e-05 2023-07-02T14:05:10.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797235+0000", 2023-07-02T14:05:10.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5990000000000008e-06 2023-07-02T14:05:10.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797245+0000", 2023-07-02T14:05:10.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.34785621999999999 2023-07-02T14:05:10.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145101+0000", 2023-07-02T14:05:10.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7762000000000006e-05 2023-07-02T14:05:10.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145169+0000", 2023-07-02T14:05:10.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013469400000000001 2023-07-02T14:05:10.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:10.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.145304+0000", 2023-07-02T14:05:10.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089538039999999992 2023-07-02T14:05:10.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154257+0000", 2023-07-02T14:05:10.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104218 2023-07-02T14:05:10.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154362+0000", 2023-07-02T14:05:10.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4168000000000001e-05 2023-07-02T14:05:10.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154406+0000", 2023-07-02T14:05:10.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3098000000000001e-05 2023-07-02T14:05:10.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154429+0000", 2023-07-02T14:05:10.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35914362500000002 2023-07-02T14:05:10.142 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.142 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.142 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.142 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4056 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3670016~65536] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:10.143 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:05:10.143 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 92.168789727000004, 2023-07-02T14:05:10.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997, 2023-07-02T14:05:10.143 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.143 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.143 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.143 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.143 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.144 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4056 2023-07-02T14:05:10.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.144 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:05:10.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797192+0000", 2023-07-02T14:05:10.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797191+0000", 2023-07-02T14:05:10.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0438999999999997e-05 2023-07-02T14:05:10.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797232+0000", 2023-07-02T14:05:10.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6929999999999998e-06 2023-07-02T14:05:10.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797237+0000", 2023-07-02T14:05:10.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9739999999999998e-06 2023-07-02T14:05:10.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797240+0000", 2023-07-02T14:05:10.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0243000000000002e-05 2023-07-02T14:05:10.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797270+0000", 2023-07-02T14:05:10.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9889999999999997e-06 2023-07-02T14:05:10.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797276+0000", 2023-07-02T14:05:10.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35358319799999999 2023-07-02T14:05:10.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150859+0000", 2023-07-02T14:05:10.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1357e-05 2023-07-02T14:05:10.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150940+0000", 2023-07-02T14:05:10.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014903 2023-07-02T14:05:10.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:10.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151089+0000", 2023-07-02T14:05:10.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033512519999999999 2023-07-02T14:05:10.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154441+0000", 2023-07-02T14:05:10.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001050945 2023-07-02T14:05:10.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155492+0000", 2023-07-02T14:05:10.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2523e-05 2023-07-02T14:05:10.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155524+0000", 2023-07-02T14:05:10.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.057e-05 2023-07-02T14:05:10.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.155555+0000", 2023-07-02T14:05:10.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35836276299999997 2023-07-02T14:05:10.155 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.155 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.156 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.156 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4088 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:head [write 3735552~458752] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:10.156 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:05:10.156 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 92.168737531999994, 2023-07-02T14:05:10.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998, 2023-07-02T14:05:10.157 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.157 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.157 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.157 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.157 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.157 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4088 2023-07-02T14:05:10.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.157 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:05:10.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:05:10.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797244+0000", 2023-07-02T14:05:10.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00038503499999999999 2023-07-02T14:05:10.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797629+0000", 2023-07-02T14:05:10.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8739999999999995e-06 2023-07-02T14:05:10.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797637+0000", 2023-07-02T14:05:10.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7160000000000001e-06 2023-07-02T14:05:10.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797640+0000", 2023-07-02T14:05:10.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4342999999999998e-05 2023-07-02T14:05:10.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797695+0000", 2023-07-02T14:05:10.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0939999999999996e-06 2023-07-02T14:05:10.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797701+0000", 2023-07-02T14:05:10.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35408540799999999 2023-07-02T14:05:10.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151786+0000", 2023-07-02T14:05:10.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1953000000000006e-05 2023-07-02T14:05:10.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.151868+0000", 2023-07-02T14:05:10.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015162800000000001 2023-07-02T14:05:10.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:10.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.152020+0000", 2023-07-02T14:05:10.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054561870000000004 2023-07-02T14:05:10.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.157476+0000", 2023-07-02T14:05:10.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059153199999999997 2023-07-02T14:05:10.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158068+0000", 2023-07-02T14:05:10.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2799e-05 2023-07-02T14:05:10.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158090+0000", 2023-07-02T14:05:10.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7235999999999999e-05 2023-07-02T14:05:10.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.158108+0000", 2023-07-02T14:05:10.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36086349099999998 2023-07-02T14:05:10.168 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.168 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.168 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.168 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4110 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:10.169 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:05:10.169 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 92.168336255, 2023-07-02T14:05:10.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864, 2023-07-02T14:05:10.169 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.169 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:10.169 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.169 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.170 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.170 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4110 2023-07-02T14:05:10.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.170 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:05:10.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:05:10.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797645+0000", 2023-07-02T14:05:10.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9331e-05 2023-07-02T14:05:10.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797684+0000", 2023-07-02T14:05:10.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.147e-06 2023-07-02T14:05:10.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797689+0000", 2023-07-02T14:05:10.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5020000000000001e-06 2023-07-02T14:05:10.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797693+0000", 2023-07-02T14:05:10.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5254999999999999e-05 2023-07-02T14:05:10.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797748+0000", 2023-07-02T14:05:10.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5520000000000005e-06 2023-07-02T14:05:10.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797757+0000", 2023-07-02T14:05:10.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35674245700000001 2023-07-02T14:05:10.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154499+0000", 2023-07-02T14:05:10.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0090000000000001e-06 2023-07-02T14:05:10.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154506+0000", 2023-07-02T14:05:10.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.356860864 2023-07-02T14:05:10.177 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.177 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.177 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.177 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:10.178 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:05:10.178 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 92.168284169000003, 2023-07-02T14:05:10.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998, 2023-07-02T14:05:10.178 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.178 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:10.178 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.178 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.178 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.179 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:05:10.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.179 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:05:10.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:05:10.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.797697+0000", 2023-07-02T14:05:10.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001714819 2023-07-02T14:05:10.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799412+0000", 2023-07-02T14:05:10.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7779999999999996e-06 2023-07-02T14:05:10.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799421+0000", 2023-07-02T14:05:10.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3510000000000002e-06 2023-07-02T14:05:10.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799424+0000", 2023-07-02T14:05:10.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0275999999999999e-05 2023-07-02T14:05:10.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799464+0000", 2023-07-02T14:05:10.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8389999999999996e-06 2023-07-02T14:05:10.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.799473+0000", 2023-07-02T14:05:10.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35507638800000002 2023-07-02T14:05:10.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154550+0000", 2023-07-02T14:05:10.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6309999999999998e-06 2023-07-02T14:05:10.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.154559+0000", 2023-07-02T14:05:10.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.35686173599999998 2023-07-02T14:05:10.187 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.187 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.187 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.187 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.187 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.187 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:10.189 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:05:10.189 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:10.189 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:10.190 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:10.190 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:05:10.190 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:05:10.191 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:05:10.191 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:05:10.191 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:05:10.191 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:05:10.192 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.1 to 1 (10000) 2023-07-02T14:05:10.192 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-07-02T14:05:10.193 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:10.187+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:10.289 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:10.287+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:10.368 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:10.369 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:10.369 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:05:10.369 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:10.384 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:05:10.425 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:10.424+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:10.490 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:10.489+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:10.554 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:10.555 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:10.555 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:05:10.555 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:05:10.555 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:10.566 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:05:10.607 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:10.606+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:10.691 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:10.690+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:10.739 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:10.740 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:05:10.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:05:10.740 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:10.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.740 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:05:10.740 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:10.741 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 129.630454958, 2023-07-02T14:05:10.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:05:10.741 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.741 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:10.741 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.741 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.741 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.741 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:05:10.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.742 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:10.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:10.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:10.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:05:10.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:05:10.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:05:10.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:05:10.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:05:10.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:05:10.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:05:10.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:05:10.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:05:10.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:05:10.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:05:10.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:05:10.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:05:10.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:05:10.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:05:10.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:05:10.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:05:10.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:05:10.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.749 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.749 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:05:10.750 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:10.750 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 129.63039422, 2023-07-02T14:05:10.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:05:10.750 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.750 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.750 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.750 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.750 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.751 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:05:10.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.751 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:10.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:10.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:10.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:05:10.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:05:10.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:05:10.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:05:10.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:05:10.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:05:10.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:05:10.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:05:10.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:05:10.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:05:10.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:05:10.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:05:10.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:05:10.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:05:10.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:05:10.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:05:10.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:10.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:05:10.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:05:10.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:05:10.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:05:10.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:05:10.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:05:10.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:05:10.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:05:10.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:05:10.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:05:10.761 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.761 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.761 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.762 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:10.762 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:10.762 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 119.39993989, 2023-07-02T14:05:10.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:05:10.762 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.762 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.763 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.763 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.763 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.763 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:05:10.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.763 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:10.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:10.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:10.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:05:10.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:05:10.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:05:10.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:05:10.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:05:10.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:05:10.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:05:10.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:05:10.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:05:10.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:05:10.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:05:10.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:05:10.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:10.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:05:10.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:05:10.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:05:10.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:05:10.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:05:10.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:05:10.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:05:10.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:05:10.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:05:10.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:05:10.774 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.774 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.774 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.774 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2950 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 1507328~786432] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:10.775 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:10.775 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 119.262663628, 2023-07-02T14:05:10.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999, 2023-07-02T14:05:10.775 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.775 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.775 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.775 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.776 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.776 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2950 2023-07-02T14:05:10.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.776 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:10.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:10.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:10.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001147854 2023-07-02T14:05:10.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476614+0000", 2023-07-02T14:05:10.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.802e-05 2023-07-02T14:05:10.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476632+0000", 2023-07-02T14:05:10.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1860000000000003e-06 2023-07-02T14:05:10.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476639+0000", 2023-07-02T14:05:10.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061211240000000004 2023-07-02T14:05:10.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482760+0000", 2023-07-02T14:05:10.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8833999999999996e-05 2023-07-02T14:05:10.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482859+0000", 2023-07-02T14:05:10.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019270600000000001 2023-07-02T14:05:10.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:10.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.483051+0000", 2023-07-02T14:05:10.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060385630000000003 2023-07-02T14:05:10.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489090+0000", 2023-07-02T14:05:10.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024506918999999999 2023-07-02T14:05:10.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513597+0000", 2023-07-02T14:05:10.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9672999999999995e-05 2023-07-02T14:05:10.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513667+0000", 2023-07-02T14:05:10.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024329500000000001 2023-07-02T14:05:10.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513910+0000", 2023-07-02T14:05:10.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999 2023-07-02T14:05:10.784 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.784 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.785 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.785 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2951 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2686976~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:10.785 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:05:10.785 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 119.26148284999999, 2023-07-02T14:05:10.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999, 2023-07-02T14:05:10.785 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.786 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.786 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.786 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.786 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.786 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2951 2023-07-02T14:05:10.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.786 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:05:10.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:05:10.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:10.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476646+0000", 2023-07-02T14:05:10.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7639000000000003e-05 2023-07-02T14:05:10.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476684+0000", 2023-07-02T14:05:10.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3508000000000002e-05 2023-07-02T14:05:10.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476747+0000", 2023-07-02T14:05:10.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8499999999999994e-06 2023-07-02T14:05:10.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476755+0000", 2023-07-02T14:05:10.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078346160000000008 2023-07-02T14:05:10.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484590+0000", 2023-07-02T14:05:10.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7206000000000004e-05 2023-07-02T14:05:10.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484687+0000", 2023-07-02T14:05:10.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016243099999999999 2023-07-02T14:05:10.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:10.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484849+0000", 2023-07-02T14:05:10.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042482860000000004 2023-07-02T14:05:10.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489097+0000", 2023-07-02T14:05:10.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.02489504 2023-07-02T14:05:10.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513992+0000", 2023-07-02T14:05:10.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9972000000000001e-05 2023-07-02T14:05:10.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514052+0000", 2023-07-02T14:05:10.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149264 2023-07-02T14:05:10.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514202+0000", 2023-07-02T14:05:10.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999 2023-07-02T14:05:10.795 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.795 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.795 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.795 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2952 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2883584~12288] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:10.796 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:10.796 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 119.261369015, 2023-07-02T14:05:10.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651, 2023-07-02T14:05:10.796 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.796 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.796 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.796 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.796 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.797 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2952 2023-07-02T14:05:10.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.797 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:10.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:10.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476760+0000", 2023-07-02T14:05:10.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3664e-05 2023-07-02T14:05:10.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476774+0000", 2023-07-02T14:05:10.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.166e-06 2023-07-02T14:05:10.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476783+0000", 2023-07-02T14:05:10.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8720000000000001e-06 2023-07-02T14:05:10.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476788+0000", 2023-07-02T14:05:10.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087180529999999999 2023-07-02T14:05:10.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485506+0000", 2023-07-02T14:05:10.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5680999999999994e-05 2023-07-02T14:05:10.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485602+0000", 2023-07-02T14:05:10.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000222365 2023-07-02T14:05:10.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:10.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485824+0000", 2023-07-02T14:05:10.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003277379 2023-07-02T14:05:10.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489101+0000", 2023-07-02T14:05:10.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025151739999999999 2023-07-02T14:05:10.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514253+0000", 2023-07-02T14:05:10.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4749000000000002e-05 2023-07-02T14:05:10.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514288+0000", 2023-07-02T14:05:10.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130634 2023-07-02T14:05:10.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514419+0000", 2023-07-02T14:05:10.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651 2023-07-02T14:05:10.805 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.805 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.805 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.806 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9620 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 589824~65536] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:05:10.806 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:10.806 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.715925292000001, 2023-07-02T14:05:10.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001, 2023-07-02T14:05:10.806 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.806 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.807 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.807 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.807 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.807 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9620 2023-07-02T14:05:10.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.807 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:10.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:10.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:10.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022203+0000", 2023-07-02T14:05:10.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0346000000000001e-05 2023-07-02T14:05:10.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022263+0000", 2023-07-02T14:05:10.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4657999999999999e-05 2023-07-02T14:05:10.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022278+0000", 2023-07-02T14:05:10.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:05:10.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022285+0000", 2023-07-02T14:05:10.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0096000000000003e-05 2023-07-02T14:05:10.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022335+0000", 2023-07-02T14:05:10.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2334000000000001e-05 2023-07-02T14:05:10.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022347+0000", 2023-07-02T14:05:10.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095653701999999993 2023-07-02T14:05:10.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118001+0000", 2023-07-02T14:05:10.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000185797 2023-07-02T14:05:10.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118187+0000", 2023-07-02T14:05:10.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026397900000000001 2023-07-02T14:05:10.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:10.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118451+0000", 2023-07-02T14:05:10.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051832800000000002 2023-07-02T14:05:10.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.123634+0000", 2023-07-02T14:05:10.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001227332 2023-07-02T14:05:10.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124861+0000", 2023-07-02T14:05:10.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9493999999999998e-05 2023-07-02T14:05:10.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124901+0000", 2023-07-02T14:05:10.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0462000000000001e-05 2023-07-02T14:05:10.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124931+0000", 2023-07-02T14:05:10.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001 2023-07-02T14:05:10.818 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.818 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.819 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.819 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9621 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 655360~1966080] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:05:10.819 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:10.819 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.715737910000001, 2023-07-02T14:05:10.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999, 2023-07-02T14:05:10.819 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.820 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.820 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.820 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.820 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.820 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9621 2023-07-02T14:05:10.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.820 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:10.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:10.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:10.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022391+0000", 2023-07-02T14:05:10.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032053939999999999 2023-07-02T14:05:10.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025597+0000", 2023-07-02T14:05:10.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4307999999999999e-05 2023-07-02T14:05:10.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025611+0000", 2023-07-02T14:05:10.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.032e-06 2023-07-02T14:05:10.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025617+0000", 2023-07-02T14:05:10.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8977999999999998e-05 2023-07-02T14:05:10.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025656+0000", 2023-07-02T14:05:10.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:05:10.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:10.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025663+0000", 2023-07-02T14:05:10.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095220739999999998 2023-07-02T14:05:10.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120884+0000", 2023-07-02T14:05:10.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2525000000000001e-05 2023-07-02T14:05:10.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120956+0000", 2023-07-02T14:05:10.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016232400000000001 2023-07-02T14:05:10.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:10.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.121119+0000", 2023-07-02T14:05:10.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044007380000000004 2023-07-02T14:05:10.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.125519+0000", 2023-07-02T14:05:10.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000797209 2023-07-02T14:05:10.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126317+0000", 2023-07-02T14:05:10.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1228000000000004e-05 2023-07-02T14:05:10.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126378+0000", 2023-07-02T14:05:10.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978000000000003e-05 2023-07-02T14:05:10.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126430+0000", 2023-07-02T14:05:10.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999 2023-07-02T14:05:10.830 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.830 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.830 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.831 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:11434 2.1 2:8269d003:::rbd_data.125de1449605.00000000000000ce:head [write 65536~4128768] snapc 1a=[1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e107)", 2023-07-02T14:05:10.831 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:05:10.831 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.820299640999998, 2023-07-02T14:05:10.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036409966000000002, 2023-07-02T14:05:10.831 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.831 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.832 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.832 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.832 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.832 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11434 2023-07-02T14:05:10.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.832 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:05:10.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:05:10.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:10.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917828+0000", 2023-07-02T14:05:10.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052044650000000001 2023-07-02T14:05:10.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923033+0000", 2023-07-02T14:05:10.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0097000000000001e-05 2023-07-02T14:05:10.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923053+0000", 2023-07-02T14:05:10.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.49e-06 2023-07-02T14:05:10.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923061+0000", 2023-07-02T14:05:10.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0090999999999998e-05 2023-07-02T14:05:10.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923121+0000", 2023-07-02T14:05:10.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000114095 2023-07-02T14:05:10.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923236+0000", 2023-07-02T14:05:10.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020094199999999999 2023-07-02T14:05:10.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:10.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923436+0000", 2023-07-02T14:05:10.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088693520000000005 2023-07-02T14:05:10.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.932306+0000", 2023-07-02T14:05:10.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021874112000000001 2023-07-02T14:05:10.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954180+0000", 2023-07-02T14:05:10.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4433e-05 2023-07-02T14:05:10.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954214+0000", 2023-07-02T14:05:10.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5884000000000002e-05 2023-07-02T14:05:10.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954240+0000", 2023-07-02T14:05:10.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036409966000000002 2023-07-02T14:05:10.841 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.841 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.841 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.841 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:189 2.3 2:c9621172:::rbd_data.125de1449605.00000000000000c0:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:10.841 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:10.841 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.565450618, 2023-07-02T14:05:10.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001, 2023-07-02T14:05:10.842 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.842 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:10.842 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.842 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:10.842 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:10.842 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:05:10.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.843 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:10.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:10.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:10.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2023-07-02T14:05:10.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172682+0000", 2023-07-02T14:05:10.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0159999999999997e-06 2023-07-02T14:05:10.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172688+0000", 2023-07-02T14:05:10.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0010000000000003e-06 2023-07-02T14:05:10.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172692+0000", 2023-07-02T14:05:10.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026444333 2023-07-02T14:05:10.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199136+0000", 2023-07-02T14:05:10.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2230000000000004e-05 2023-07-02T14:05:10.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199218+0000", 2023-07-02T14:05:10.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012043595000000001 2023-07-02T14:05:10.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211262+0000", 2023-07-02T14:05:10.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001 2023-07-02T14:05:10.848 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.849 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.849 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.849 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:199 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:10.849 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:10.849 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.565407075000003, 2023-07-02T14:05:10.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002, 2023-07-02T14:05:10.850 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.850 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:10.850 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.850 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:10.850 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:10.850 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 199 2023-07-02T14:05:10.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.850 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:10.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:10.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172722+0000", 2023-07-02T14:05:10.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1250000000000001e-06 2023-07-02T14:05:10.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172725+0000", 2023-07-02T14:05:10.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5670000000000002e-06 2023-07-02T14:05:10.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172731+0000", 2023-07-02T14:05:10.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9310000000000001e-06 2023-07-02T14:05:10.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172735+0000", 2023-07-02T14:05:10.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030699852999999999 2023-07-02T14:05:10.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203435+0000", 2023-07-02T14:05:10.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6827e-05 2023-07-02T14:05:10.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203532+0000", 2023-07-02T14:05:10.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076955230000000001 2023-07-02T14:05:10.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211227+0000", 2023-07-02T14:05:10.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002 2023-07-02T14:05:10.856 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.856 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.856 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.857 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:281 2.1 2:901d5a95:::rbd_data.125de1449605.0000000000000118:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:10.857 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:10.857 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.182661799000002, 2023-07-02T14:05:10.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002, 2023-07-02T14:05:10.857 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.857 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:10.858 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.858 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:10.858 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:10.858 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:05:10.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.858 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:10.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:10.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:10.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0469999999999998e-06 2023-07-02T14:05:10.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555471+0000", 2023-07-02T14:05:10.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8860000000000001e-06 2023-07-02T14:05:10.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555476+0000", 2023-07-02T14:05:10.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3160000000000002e-06 2023-07-02T14:05:10.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555483+0000", 2023-07-02T14:05:10.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989525000000001 2023-07-02T14:05:10.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591472+0000", 2023-07-02T14:05:10.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3708999999999996e-05 2023-07-02T14:05:10.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591536+0000", 2023-07-02T14:05:10.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055109720000000003 2023-07-02T14:05:10.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.597047+0000", 2023-07-02T14:05:10.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002 2023-07-02T14:05:10.864 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.864 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.864 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.865 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12536 2.1 2:8b20f686:::rbd_data.125de1449605.0000000000000056:head [write 3670016~36864] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:05:10.865 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.205749+0000", 2023-07-02T14:05:10.865 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.532380835000001, 2023-07-02T14:05:10.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036263036999999998, 2023-07-02T14:05:10.865 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.865 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.865 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.865 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.866 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.866 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12536 2023-07-02T14:05:10.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.866 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205749+0000", 2023-07-02T14:05:10.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.867 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205749+0000", 2023-07-02T14:05:10.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205748+0000", 2023-07-02T14:05:10.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1784999999999999e-05 2023-07-02T14:05:10.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205770+0000", 2023-07-02T14:05:10.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0659999999999993e-06 2023-07-02T14:05:10.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205779+0000", 2023-07-02T14:05:10.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.622e-06 2023-07-02T14:05:10.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.205784+0000", 2023-07-02T14:05:10.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018333009000000001 2023-07-02T14:05:10.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.224117+0000", 2023-07-02T14:05:10.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018818400000000001 2023-07-02T14:05:10.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.224305+0000", 2023-07-02T14:05:10.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00032545700000000002 2023-07-02T14:05:10.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:10.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.224631+0000", 2023-07-02T14:05:10.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049349700000000003 2023-07-02T14:05:10.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.229566+0000", 2023-07-02T14:05:10.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012353153 2023-07-02T14:05:10.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.241919+0000", 2023-07-02T14:05:10.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2751999999999999e-05 2023-07-02T14:05:10.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.241971+0000", 2023-07-02T14:05:10.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0632e-05 2023-07-02T14:05:10.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.874 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.242012+0000", 2023-07-02T14:05:10.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036263036999999998 2023-07-02T14:05:10.874 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.874 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.874 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.875 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12883 2.1 2:9ffc48e1:::rbd_data.125de1449605.0000000000000074:head [write 4173824~20480] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:05:10.875 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:10.876 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.092059326000001, 2023-07-02T14:05:10.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037871393000000003, 2023-07-02T14:05:10.876 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.876 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.876 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.876 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.877 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.877 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12883 2023-07-02T14:05:10.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.877 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:10.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:10.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:05:10.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646068+0000", 2023-07-02T14:05:10.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4935999999999999e-05 2023-07-02T14:05:10.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646103+0000", 2023-07-02T14:05:10.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.092e-05 2023-07-02T14:05:10.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646124+0000", 2023-07-02T14:05:10.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1519999999999998e-06 2023-07-02T14:05:10.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646133+0000", 2023-07-02T14:05:10.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012270200000000001 2023-07-02T14:05:10.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646255+0000", 2023-07-02T14:05:10.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014495400000000001 2023-07-02T14:05:10.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646400+0000", 2023-07-02T14:05:10.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000221626 2023-07-02T14:05:10.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:10.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646622+0000", 2023-07-02T14:05:10.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078929050000000004 2023-07-02T14:05:10.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.654515+0000", 2023-07-02T14:05:10.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029307343999999999 2023-07-02T14:05:10.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683822+0000", 2023-07-02T14:05:10.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8298000000000002e-05 2023-07-02T14:05:10.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683891+0000", 2023-07-02T14:05:10.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1381999999999997e-05 2023-07-02T14:05:10.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.885 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683942+0000", 2023-07-02T14:05:10.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037871393000000003 2023-07-02T14:05:10.885 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.885 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.885 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.886 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12985 2.7 2:eeddf24b:::rbd_data.125de1449605.000000000000007e:head [write 65536~4128768] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:05:10.886 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:10.886 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.955134853000001, 2023-07-02T14:05:10.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999, 2023-07-02T14:05:10.886 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.886 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:10.887 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.887 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:10.887 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:10.887 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12985 2023-07-02T14:05:10.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.887 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:10.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:10.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:10.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782994+0000", 2023-07-02T14:05:10.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019254741999999998 2023-07-02T14:05:10.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802248+0000", 2023-07-02T14:05:10.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0451e-05 2023-07-02T14:05:10.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802279+0000", 2023-07-02T14:05:10.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3318e-05 2023-07-02T14:05:10.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802292+0000", 2023-07-02T14:05:10.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109403 2023-07-02T14:05:10.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802402+0000", 2023-07-02T14:05:10.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012520599999999999 2023-07-02T14:05:10.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802527+0000", 2023-07-02T14:05:10.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187768 2023-07-02T14:05:10.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:05:10.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802715+0000", 2023-07-02T14:05:10.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013192822 2023-07-02T14:05:10.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:10.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.815907+0000", 2023-07-02T14:05:10.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0086451870000000004 2023-07-02T14:05:10.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:10.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824553+0000", 2023-07-02T14:05:10.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6759999999999999e-05 2023-07-02T14:05:10.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:10.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824609+0000", 2023-07-02T14:05:10.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011314700000000001 2023-07-02T14:05:10.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824722+0000", 2023-07-02T14:05:10.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999 2023-07-02T14:05:10.896 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.896 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.896 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.896 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:165 2.3 2:c9c6393a:::rbd_data.125de1449605.000000000000002c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:10.896 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:10.896 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.898352036, 2023-07-02T14:05:10.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002, 2023-07-02T14:05:10.897 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.897 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:10.897 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.897 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:10.897 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:10.897 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 165 2023-07-02T14:05:10.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.898 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:10.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:10.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839777+0000", 2023-07-02T14:05:10.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6740000000000001e-06 2023-07-02T14:05:10.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839780+0000", 2023-07-02T14:05:10.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9030000000000004e-06 2023-07-02T14:05:10.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839788+0000", 2023-07-02T14:05:10.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1799999999999998e-06 2023-07-02T14:05:10.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839792+0000", 2023-07-02T14:05:10.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.031592227000000001 2023-07-02T14:05:10.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871384+0000", 2023-07-02T14:05:10.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013600099999999999 2023-07-02T14:05:10.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871520+0000", 2023-07-02T14:05:10.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082817089999999999 2023-07-02T14:05:10.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.879802+0000", 2023-07-02T14:05:10.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002 2023-07-02T14:05:10.903 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.903 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.904 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.904 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:252 2.7 2:f1d0abf0:::rbd_data.125de1449605.0000000000000083:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:10.904 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:10.904 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.482067066000001, 2023-07-02T14:05:10.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997, 2023-07-02T14:05:10.904 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.905 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:10.905 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.905 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:10.905 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:10.905 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 252 2023-07-02T14:05:10.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.905 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:10.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:10.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256062+0000", 2023-07-02T14:05:10.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0149999999999999e-06 2023-07-02T14:05:10.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256065+0000", 2023-07-02T14:05:10.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5269999999999998e-06 2023-07-02T14:05:10.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256071+0000", 2023-07-02T14:05:10.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:05:10.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256074+0000", 2023-07-02T14:05:10.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028852909 2023-07-02T14:05:10.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.284927+0000", 2023-07-02T14:05:10.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168274 2023-07-02T14:05:10.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.285096+0000", 2023-07-02T14:05:10.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011036300000000001 2023-07-02T14:05:10.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296132+0000", 2023-07-02T14:05:10.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997 2023-07-02T14:05:10.911 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.911 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.912 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.912 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:261 2.1 2:971731cb:::rbd_data.125de1449605.000000000000008c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:10.912 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:10.912 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.481971591000001, 2023-07-02T14:05:10.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984, 2023-07-02T14:05:10.913 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.913 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:10.913 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.913 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:10.913 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:10.913 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 261 2023-07-02T14:05:10.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.913 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:10.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:10.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:10.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2819999999999999e-06 2023-07-02T14:05:10.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256160+0000", 2023-07-02T14:05:10.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2020000000000003e-06 2023-07-02T14:05:10.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256165+0000", 2023-07-02T14:05:10.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2100000000000003e-06 2023-07-02T14:05:10.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256170+0000", 2023-07-02T14:05:10.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025871715999999999 2023-07-02T14:05:10.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282041+0000", 2023-07-02T14:05:10.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132364 2023-07-02T14:05:10.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282174+0000", 2023-07-02T14:05:10.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014044566 2023-07-02T14:05:10.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296218+0000", 2023-07-02T14:05:10.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984 2023-07-02T14:05:10.919 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.919 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.919 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.920 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:264 2.3 2:dbf06c9d:::rbd_data.125de1449605.000000000000008f:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:10.920 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:10.920 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.481937473, 2023-07-02T14:05:10.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003, 2023-07-02T14:05:10.920 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.920 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 264 2023-07-02T14:05:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5749999999999999e-06 2023-07-02T14:05:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256195+0000", 2023-07-02T14:05:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.327e-06 2023-07-02T14:05:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256200+0000", 2023-07-02T14:05:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0580000000000002e-06 2023-07-02T14:05:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256203+0000", 2023-07-02T14:05:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035707524999999997 2023-07-02T14:05:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.291911+0000", 2023-07-02T14:05:10.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014269799999999999 2023-07-02T14:05:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.292053+0000", 2023-07-02T14:05:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062760309999999996 2023-07-02T14:05:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.298329+0000", 2023-07-02T14:05:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003 2023-07-02T14:05:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.927 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:341 2.3 2:cee258bd:::rbd_data.125de1449605.00000000000000dc:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.101450145999999, 2023-07-02T14:05:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997, 2023-07-02T14:05:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:10.928 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 341 2023-07-02T14:05:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636679+0000", 2023-07-02T14:05:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6240000000000002e-06 2023-07-02T14:05:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636682+0000", 2023-07-02T14:05:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6910000000000002e-06 2023-07-02T14:05:10.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636688+0000", 2023-07-02T14:05:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3469999999999999e-06 2023-07-02T14:05:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:10.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636691+0000", 2023-07-02T14:05:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035625560000000001 2023-07-02T14:05:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672317+0000", 2023-07-02T14:05:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108391 2023-07-02T14:05:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672425+0000", 2023-07-02T14:05:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012104976 2023-07-02T14:05:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.684530+0000", 2023-07-02T14:05:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997 2023-07-02T14:05:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:10.936 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:10.938 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:05:10.938 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:10.936+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:11.038 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:11.037+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:11.079 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:11.079 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:11.079 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:11.079 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:11.090 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:05:11.112 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.3 to 1 (10000) 2023-07-02T14:05:11.118 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:05:11.125 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-07-02T14:05:11.139 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap33 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap34 b3fbb028-e267-4d25-a750-93ba18a2316a@snap34.diff 2023-07-02T14:05:11.166 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:11.165+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:11.239 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:11.238+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:11.280 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:11.281 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:11.281 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:11.281 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:11.281 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:11.292 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:05:11.340 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:11.339+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:11.431 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:11.431 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:11.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:11.431 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:11.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.432 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:05:11.432 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:11.432 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 91.531764072000001, 2023-07-02T14:05:11.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:05:11.432 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:11.432 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:11.433 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:11.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:11.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:11.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:11.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:11.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:11.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:11.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:11.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:05:11.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:05:11.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:11.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:05:11.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:05:11.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:11.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:05:11.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:05:11.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:11.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:05:11.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:05:11.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:11.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:05:11.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:05:11.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:11.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:05:11.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:05:11.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:11.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:05:11.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:05:11.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:11.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:05:11.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:05:11.439 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.439 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.439 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.440 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:05:11.440 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:11.440 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 91.527678354000003, 2023-07-02T14:05:11.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:05:11.440 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:11.440 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:11.441 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:11.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:11.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:11.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:11.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:11.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:11.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:11.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:11.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:05:11.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:05:11.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:11.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:05:11.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:05:11.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:11.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:05:11.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:05:11.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:11.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:05:11.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:05:11.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:11.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:05:11.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:05:11.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:11.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:05:11.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:05:11.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:11.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:05:11.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:05:11.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:11.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:05:11.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:05:11.447 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.447 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.447 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.448 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:05:11.448 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:11.448 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 91.523813572999998, 2023-07-02T14:05:11.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:05:11.448 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:11.448 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:11.448 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:11.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:11.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:11.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:11.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:11.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:11.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:11.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:11.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:05:11.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:05:11.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:11.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:05:11.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:05:11.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:11.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:05:11.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:05:11.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:11.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:05:11.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:05:11.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:11.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:05:11.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:05:11.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:11.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:05:11.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:05:11.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:11.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:05:11.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:05:11.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:11.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:05:11.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:05:11.455 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.455 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.455 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.456 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:05:11.456 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:11.456 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 91.520289312000003, 2023-07-02T14:05:11.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:05:11.456 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:11.456 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:11.456 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:11.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:11.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:11.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:11.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:11.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:11.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:11.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:11.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:05:11.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:05:11.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:11.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:05:11.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:05:11.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:11.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:05:11.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:05:11.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:11.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:05:11.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:05:11.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:11.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:05:11.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:05:11.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:11.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:05:11.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:05:11.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:11.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:05:11.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:05:11.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:11.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:05:11.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:05:11.463 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.463 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.463 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.464 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:05:11.464 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:11.464 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 91.484584513000001, 2023-07-02T14:05:11.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:05:11.464 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:11.464 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:11.464 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:11.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:11.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:11.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:11.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:11.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:11.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:11.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:11.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:05:11.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:05:11.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:11.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:05:11.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:05:11.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:11.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:05:11.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:05:11.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:11.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:05:11.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:05:11.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:11.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:05:11.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:05:11.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:11.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:05:11.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:05:11.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:11.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:05:11.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:05:11.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:11.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:05:11.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:05:11.471 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.471 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.471 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.471 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:05:11.472 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:11.472 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 91.479927154999999, 2023-07-02T14:05:11.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:05:11.472 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:11.472 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:11.472 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:11.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:11.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:11.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:11.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:11.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:11.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:11.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:11.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:05:11.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:05:11.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:11.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:05:11.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:05:11.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:11.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:05:11.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:05:11.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:11.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:05:11.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:05:11.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:11.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:05:11.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:05:11.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:11.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:05:11.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:05:11.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:11.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:05:11.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:05:11.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:11.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:05:11.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:05:11.479 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.479 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.479 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.479 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:05:11.480 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:11.480 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 91.478987291999999, 2023-07-02T14:05:11.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:05:11.480 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:11.480 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:11.480 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:11.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:11.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:11.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:11.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:11.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:11.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:11.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:11.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:05:11.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:05:11.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:11.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:05:11.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:05:11.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:11.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:05:11.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:05:11.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:11.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:05:11.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:05:11.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:11.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:05:11.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:05:11.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:11.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:05:11.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:05:11.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:11.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:05:11.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:05:11.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:11.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:05:11.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:05:11.487 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.487 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.487 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.487 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:05:11.487 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:11.488 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 91.477589413999993, 2023-07-02T14:05:11.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:05:11.488 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:11.488 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:11.488 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:11.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:11.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:11.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:11.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:11.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:11.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:11.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:11.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:05:11.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:05:11.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:11.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:05:11.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:05:11.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:11.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:05:11.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:05:11.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:11.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:05:11.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:05:11.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:11.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:05:11.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:05:11.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:11.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:05:11.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:05:11.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:11.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:05:11.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:05:11.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:11.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:05:11.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:05:11.495 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.495 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.495 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.495 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:05:11.495 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:11.496 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 70.067398917999995, 2023-07-02T14:05:11.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:05:11.496 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:11.496 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:11.496 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:11.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:11.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:11.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:11.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:11.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:11.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:11.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:11.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:05:11.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:05:11.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:11.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:05:11.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:05:11.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:11.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:05:11.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:05:11.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:11.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:05:11.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:05:11.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:11.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:05:11.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:05:11.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:11.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:05:11.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:05:11.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:11.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:05:11.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:05:11.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:11.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:05:11.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:05:11.503 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.503 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.503 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.504 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:11.504 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:11.504 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.701049233999996, 2023-07-02T14:05:11.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:05:11.504 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:11.504 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:11.504 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:11.504 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:11.505 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:11.505 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:05:11.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.505 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:11.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:11.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:11.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:11.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:11.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:11.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:11.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:11.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:05:11.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:05:11.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:11.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:05:11.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:05:11.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:11.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:05:11.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:05:11.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:11.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:05:11.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:05:11.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:11.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:05:11.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:05:11.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:11.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:05:11.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:05:11.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:11.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:05:11.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:05:11.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:11.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:05:11.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:05:11.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:11.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:05:11.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:05:11.512 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.512 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.512 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.513 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7707 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 3145728~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:11.513 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:11.513 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.697065766999998, 2023-07-02T14:05:11.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999, 2023-07-02T14:05:11.513 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:11.513 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:11.513 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:11.514 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:11.514 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:11.514 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7707 2023-07-02T14:05:11.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.514 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:11.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:11.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:11.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:11.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:11.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:11.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:11.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:11.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:11.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5017e-05 2023-07-02T14:05:11.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:11.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732347+0000", 2023-07-02T14:05:11.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5821e-05 2023-07-02T14:05:11.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:11.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732363+0000", 2023-07-02T14:05:11.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1270000000000002e-06 2023-07-02T14:05:11.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:11.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732368+0000", 2023-07-02T14:05:11.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008444666 2023-07-02T14:05:11.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:11.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740812+0000", 2023-07-02T14:05:11.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011266599999999999 2023-07-02T14:05:11.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:11.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740925+0000", 2023-07-02T14:05:11.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010848808999999999 2023-07-02T14:05:11.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:11.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751774+0000", 2023-07-02T14:05:11.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8421999999999997e-05 2023-07-02T14:05:11.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:11.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751832+0000", 2023-07-02T14:05:11.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107453 2023-07-02T14:05:11.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:11.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751940+0000", 2023-07-02T14:05:11.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999 2023-07-02T14:05:11.521 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.521 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.521 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.522 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:11.522 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:11.522 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.697001889999996, 2023-07-02T14:05:11.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:05:11.522 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:11.522 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:11.522 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:11.523 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:11.523 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:11.523 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:05:11.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.523 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:11.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:11.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:11.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:11.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:11.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:11.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:11.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:11.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:11.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:05:11.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:11.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:05:11.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:05:11.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:11.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:05:11.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:05:11.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:11.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:05:11.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:05:11.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:11.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:05:11.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:05:11.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:11.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:05:11.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:05:11.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:11.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:05:11.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:05:11.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:11.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:05:11.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:05:11.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:11.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:05:11.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:05:11.530 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.530 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.530 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.531 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:11.531 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:11.531 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.692438065000005, 2023-07-02T14:05:11.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:05:11.531 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:11.531 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:11.532 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:11.532 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:11.532 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:11.532 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:05:11.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.532 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:11.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:11.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:11.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:11.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:11.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:11.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:11.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:11.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:05:11.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:05:11.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:11.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:05:11.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:05:11.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:11.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:05:11.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:05:11.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:11.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:05:11.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:05:11.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:11.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:05:11.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:05:11.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:11.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:05:11.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:05:11.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:11.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:05:11.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:05:11.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:11.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:05:11.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:05:11.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:11.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:05:11.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:05:11.539 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.539 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.540 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.540 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:11.540 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:11.540 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.692388199000007, 2023-07-02T14:05:11.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:05:11.541 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:11.541 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:11.541 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:11.541 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:11.541 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:11.541 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:05:11.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.542 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:11.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:11.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:11.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:11.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:11.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:11.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:11.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:11.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:05:11.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:05:11.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:11.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:05:11.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:05:11.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:11.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:05:11.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:05:11.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:11.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:05:11.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:05:11.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:11.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:05:11.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:05:11.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:11.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:05:11.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:05:11.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:11.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:05:11.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:05:11.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:11.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:05:11.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:05:11.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:11.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:05:11.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:05:11.549 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.549 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.549 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.549 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:11.549 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:11.550 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.691349205999998, 2023-07-02T14:05:11.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:05:11.550 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:11.550 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:11.550 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:11.550 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:11.550 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:11.550 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:05:11.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.551 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:11.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:11.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:11.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:11.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:11.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:11.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:11.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:11.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:05:11.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:05:11.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:11.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:05:11.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:05:11.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:11.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:05:11.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:05:11.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:11.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:05:11.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:05:11.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:11.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:05:11.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:05:11.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:11.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:05:11.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:05:11.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:11.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:05:11.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:05:11.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:11.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:05:11.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:05:11.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:11.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:05:11.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:05:11.558 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.558 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.558 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.558 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7993 2.7 2:fe72db53:::rbd_data.125de1449605.0000000000000072:head [write 65536~3276800] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:11.559 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:11.559 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 68.504391158000004, 2023-07-02T14:05:11.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919, 2023-07-02T14:05:11.559 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:11.559 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:11.559 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:11.559 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:11.559 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:11.560 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7993 2023-07-02T14:05:11.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.560 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:11.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:11.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:11.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:11.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:11.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:11.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:11.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:11.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924986+0000", 2023-07-02T14:05:11.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049542070000000004 2023-07-02T14:05:11.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:11.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929940+0000", 2023-07-02T14:05:11.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9740000000000001e-05 2023-07-02T14:05:11.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:11.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929960+0000", 2023-07-02T14:05:11.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.311e-05 2023-07-02T14:05:11.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:11.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929973+0000", 2023-07-02T14:05:11.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9629000000000002e-05 2023-07-02T14:05:11.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:11.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930023+0000", 2023-07-02T14:05:11.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012751799999999999 2023-07-02T14:05:11.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:11.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930150+0000", 2023-07-02T14:05:11.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016449042000000001 2023-07-02T14:05:11.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:11.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946599+0000", 2023-07-02T14:05:11.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3241e-05 2023-07-02T14:05:11.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:11.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946663+0000", 2023-07-02T14:05:11.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000102296 2023-07-02T14:05:11.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:11.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946765+0000", 2023-07-02T14:05:11.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919 2023-07-02T14:05:11.567 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.567 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.567 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.568 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11782 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 196608~2162688] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:05:11.568 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:11.568 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 27.019871236, 2023-07-02T14:05:11.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365, 2023-07-02T14:05:11.568 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:11.568 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:11.569 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:11.569 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:11.569 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:11.569 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11782 2023-07-02T14:05:11.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.569 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:11.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:11.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:11.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:11.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:11.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:11.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:11.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:11.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409506+0000", 2023-07-02T14:05:11.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020172269999999999 2023-07-02T14:05:11.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:11.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411523+0000", 2023-07-02T14:05:11.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8901000000000001e-05 2023-07-02T14:05:11.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:11.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411542+0000", 2023-07-02T14:05:11.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5037e-05 2023-07-02T14:05:11.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:11.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411557+0000", 2023-07-02T14:05:11.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001931213 2023-07-02T14:05:11.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:11.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413489+0000", 2023-07-02T14:05:11.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000149207 2023-07-02T14:05:11.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:11.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413638+0000", 2023-07-02T14:05:11.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024752800000000001 2023-07-02T14:05:11.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:11.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413885+0000", 2023-07-02T14:05:11.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048516000000000002 2023-07-02T14:05:11.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:11.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.418737+0000", 2023-07-02T14:05:11.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010319259000000001 2023-07-02T14:05:11.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:11.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429056+0000", 2023-07-02T14:05:11.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2518000000000002e-05 2023-07-02T14:05:11.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:11.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429099+0000", 2023-07-02T14:05:11.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015275 2023-07-02T14:05:11.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:11.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429251+0000", 2023-07-02T14:05:11.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365 2023-07-02T14:05:11.577 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.578 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.578 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.578 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPushReply(2.7 119/117 [PushReplyOp(2:eb862bc0:::rbd_data.125de1449605.0000000000000040:13)])", 2023-07-02T14:05:11.578 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:11.578 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 27.018707961, 2023-07-02T14:05:11.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999, 2023-07-02T14:05:11.579 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:11.579 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "reached pg", 2023-07-02T14:05:11.579 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:11.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:11.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:11.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:11.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:11.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:11.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:11.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:11.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:11.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7670000000000001e-06 2023-07-02T14:05:11.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:11.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410674+0000", 2023-07-02T14:05:11.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3973e-05 2023-07-02T14:05:11.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:11.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410688+0000", 2023-07-02T14:05:11.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0157e-05 2023-07-02T14:05:11.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:11.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410708+0000", 2023-07-02T14:05:11.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018578779 2023-07-02T14:05:11.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:11.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429287+0000", 2023-07-02T14:05:11.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013086549999999999 2023-07-02T14:05:11.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:11.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.430595+0000", 2023-07-02T14:05:11.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999 2023-07-02T14:05:11.584 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.584 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.584 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.585 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11861 2.7 2:e57d3f55:::rbd_data.125de1449605.0000000000000134:head [write 0~65536] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:05:11.585 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:11.585 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 26.884592584, 2023-07-02T14:05:11.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723, 2023-07-02T14:05:11.585 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:11.585 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:11.585 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:11.586 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:11.586 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:11.586 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11861 2023-07-02T14:05:11.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.586 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:11.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:11.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:11.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:11.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:11.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:11.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:11.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:11.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544785+0000", 2023-07-02T14:05:11.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010563700000000001 2023-07-02T14:05:11.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:11.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544890+0000", 2023-07-02T14:05:11.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8530000000000004e-06 2023-07-02T14:05:11.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:11.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544900+0000", 2023-07-02T14:05:11.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.665e-06 2023-07-02T14:05:11.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:11.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544906+0000", 2023-07-02T14:05:11.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000272575 2023-07-02T14:05:11.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:11.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545178+0000", 2023-07-02T14:05:11.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000111362 2023-07-02T14:05:11.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:11.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545290+0000", 2023-07-02T14:05:11.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016181100000000001 2023-07-02T14:05:11.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:11.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545452+0000", 2023-07-02T14:05:11.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044500750000000004 2023-07-02T14:05:11.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:11.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.549902+0000", 2023-07-02T14:05:11.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014850024 2023-07-02T14:05:11.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:11.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564752+0000", 2023-07-02T14:05:11.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6272000000000003e-05 2023-07-02T14:05:11.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:11.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564848+0000", 2023-07-02T14:05:11.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000162428 2023-07-02T14:05:11.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:11.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.565010+0000", 2023-07-02T14:05:11.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723 2023-07-02T14:05:11.594 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.595 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.595 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.595 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:13262 2.7 e125/121 2:e0741dde:::rbd_data.125de1449605.00000000000000ad:head v 125'2024, mlcod=125'2021)", 2023-07-02T14:05:11.595 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:11.595 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 18.178514845999999, 2023-07-02T14:05:11.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999, 2023-07-02T14:05:11.596 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:11.596 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:11.596 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:11.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:11.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:11.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:11.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:11.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:11.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:11.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:11.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250862+0000", 2023-07-02T14:05:11.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080378800000000007 2023-07-02T14:05:11.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:11.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258900+0000", 2023-07-02T14:05:11.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8939999999999999e-06 2023-07-02T14:05:11.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:11.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258909+0000", 2023-07-02T14:05:11.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3560000000000003e-06 2023-07-02T14:05:11.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:11.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258914+0000", 2023-07-02T14:05:11.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2429000000000003e-05 2023-07-02T14:05:11.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:11.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258956+0000", 2023-07-02T14:05:11.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5094999999999999e-05 2023-07-02T14:05:11.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:11.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258981+0000", 2023-07-02T14:05:11.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012177307 2023-07-02T14:05:11.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:11.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271158+0000", 2023-07-02T14:05:11.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2104999999999999e-05 2023-07-02T14:05:11.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:11.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:11.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:11.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271181+0000", 2023-07-02T14:05:11.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999 2023-07-02T14:05:11.602 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.602 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.602 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.603 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:11.603 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:11.603 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:11.604 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:05:11.605 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:11.603+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:11.704 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:11.704+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:11.776 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:11.776 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:11.776 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:11.776 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:11.787 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:05:11.806 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:11.804+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:11.899 INFO:tasks.workunit.client.0.smithi163.stderr:+ tar -C b3fbb028-e267-4d25-a750-93ba18a2316a-mnt -xzf b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4.tar.gz 2023-07-02T14:05:11.907 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:11.905+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:11.958 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:11.958 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:11.958 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:11.958 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:11.958 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:11.968 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:05:12.011 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:12.011+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:12.108 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:12.107+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:12.113 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:12.113 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:12.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:12.114 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:12.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.114 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:12.114 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:12.114 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.820776873, 2023-07-02T14:05:12.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:05:12.114 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.115 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:12.115 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:12.115 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:12.115 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:12.115 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:05:12.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.115 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:12.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:05:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:05:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:05:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:05:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:05:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:05:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:05:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:05:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:05:12.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:05:12.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:05:12.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:05:12.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:12.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:05:12.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:05:12.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:12.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:05:12.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:05:12.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:12.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:05:12.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:05:12.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:12.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:05:12.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:05:12.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:12.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:05:12.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:05:12.124 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.124 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.124 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.125 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:12.125 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:12.125 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.82054185000001, 2023-07-02T14:05:12.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:05:12.125 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.125 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:12.125 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:12.125 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:12.126 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:12.126 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:05:12.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.126 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:12.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:12.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:12.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:05:12.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:05:12.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:05:12.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:05:12.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:05:12.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:05:12.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:05:12.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:05:12.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:05:12.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:05:12.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:05:12.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:05:12.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:12.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:05:12.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:05:12.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:12.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:05:12.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:05:12.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:12.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:05:12.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:05:12.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:12.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:05:12.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:05:12.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:12.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:05:12.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:05:12.134 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.134 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.134 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.135 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:12.135 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:12.135 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.820342716, 2023-07-02T14:05:12.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:05:12.135 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.135 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:12.136 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:12.136 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:12.136 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:12.136 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:05:12.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.136 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:12.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:12.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:12.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:05:12.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:05:12.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:05:12.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:05:12.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:05:12.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:05:12.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:05:12.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:05:12.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:05:12.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:05:12.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:05:12.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:05:12.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:12.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:05:12.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:05:12.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:12.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:05:12.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:05:12.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:12.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:05:12.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:05:12.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:12.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:05:12.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:05:12.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:12.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:05:12.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:05:12.145 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.145 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.145 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.145 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:12.146 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:12.146 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.81997759199999, 2023-07-02T14:05:12.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:05:12.146 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.146 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:12.146 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:12.146 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:12.146 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:12.147 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:05:12.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.147 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:12.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:12.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:12.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:05:12.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:05:12.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:05:12.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:05:12.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:05:12.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:05:12.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:05:12.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:05:12.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:05:12.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:05:12.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:05:12.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:05:12.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:12.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:05:12.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:05:12.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:12.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:05:12.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:05:12.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:12.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:05:12.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:05:12.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:12.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:05:12.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:05:12.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:12.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:05:12.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:05:12.155 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.155 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.155 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.156 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:12.156 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:12.156 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.81989653100001, 2023-07-02T14:05:12.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:05:12.156 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.156 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:12.157 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:12.157 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:12.157 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:12.157 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:05:12.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.157 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:12.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:12.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:12.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:05:12.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:05:12.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:05:12.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:05:12.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:05:12.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:05:12.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:05:12.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:05:12.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:05:12.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:05:12.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:05:12.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:05:12.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:12.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:05:12.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:05:12.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:12.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:05:12.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:05:12.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:12.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:05:12.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:05:12.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:12.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:05:12.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:05:12.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:12.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:05:12.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:05:12.166 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.166 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.166 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.167 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:12.167 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:12.167 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.81980486, 2023-07-02T14:05:12.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:05:12.167 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.167 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:12.167 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:12.168 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:12.168 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:12.168 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:05:12.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.168 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:12.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:12.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:12.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:05:12.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:05:12.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:05:12.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:05:12.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:05:12.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:05:12.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:05:12.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:05:12.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:05:12.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:05:12.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:05:12.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:05:12.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:12.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:05:12.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:05:12.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:12.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:05:12.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:05:12.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:12.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:05:12.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:05:12.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:12.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:05:12.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:05:12.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:12.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:05:12.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:05:12.177 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.177 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.177 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.177 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:12.177 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:12.178 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.81946541799999, 2023-07-02T14:05:12.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:05:12.178 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.178 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:12.178 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:12.178 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:12.178 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:12.179 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:05:12.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.179 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:12.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:12.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:12.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:05:12.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:05:12.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:05:12.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:05:12.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:05:12.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:05:12.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:05:12.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:05:12.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:05:12.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:05:12.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:05:12.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:05:12.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:12.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:05:12.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:05:12.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:12.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:05:12.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:05:12.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:12.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:05:12.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:05:12.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:12.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:05:12.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:05:12.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:12.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:05:12.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:05:12.187 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.187 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.187 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.188 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:12.188 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:12.188 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.819279365, 2023-07-02T14:05:12.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:05:12.188 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.189 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:12.189 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:12.189 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:12.189 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:12.189 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:05:12.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.189 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:12.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:12.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:12.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:12.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:05:12.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:05:12.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:05:12.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:05:12.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:05:12.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:05:12.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:05:12.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:05:12.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:05:12.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:05:12.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:05:12.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:12.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:05:12.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:05:12.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:12.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:05:12.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:05:12.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:12.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:05:12.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:05:12.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:12.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:05:12.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:05:12.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:12.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:05:12.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:05:12.198 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.198 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.198 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.198 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:12.199 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:12.199 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.81912156300001, 2023-07-02T14:05:12.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:05:12.199 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.199 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:12.199 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:12.199 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:12.199 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:12.200 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:05:12.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.200 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:12.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:12.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:12.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:05:12.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:05:12.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:05:12.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:05:12.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:05:12.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:05:12.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:05:12.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:05:12.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:05:12.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:05:12.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:05:12.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:05:12.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:12.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:05:12.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:05:12.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:12.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:05:12.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:05:12.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:12.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:05:12.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:05:12.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:12.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:05:12.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:05:12.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:12.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:05:12.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:05:12.208 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.208 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.208 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.209 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:12.209 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:12.209 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.81892978100001, 2023-07-02T14:05:12.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:05:12.209 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.209 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:12.209 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:12.210 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:12.210 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:12.210 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:05:12.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.210 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:12.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:12.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:12.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:05:12.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:05:12.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:05:12.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:05:12.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:05:12.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:05:12.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:05:12.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:05:12.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:05:12.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:05:12.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:05:12.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:05:12.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:12.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:05:12.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:05:12.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:12.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:05:12.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:05:12.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:12.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:05:12.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:05:12.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:12.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:05:12.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:05:12.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:12.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:05:12.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:05:12.219 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.219 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.219 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.219 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:12.219 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:12.219 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.81875412100001, 2023-07-02T14:05:12.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:05:12.220 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.220 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:12.220 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:12.220 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:12.220 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:12.221 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:05:12.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.221 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.221 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:12.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:12.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:12.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:12.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:05:12.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:05:12.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:05:12.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:05:12.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:05:12.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:05:12.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:05:12.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:05:12.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:05:12.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:05:12.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:05:12.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:12.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:05:12.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:05:12.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:12.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:05:12.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:05:12.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:12.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:05:12.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:05:12.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:12.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:05:12.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:05:12.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:12.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:05:12.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:05:12.229 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.230 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.230 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.230 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:12.230 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:12.230 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.81865993299999, 2023-07-02T14:05:12.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:05:12.231 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.231 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:12.231 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:12.231 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:12.231 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:12.231 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:05:12.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.231 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:12.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:12.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:12.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:05:12.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:05:12.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:05:12.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:05:12.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:05:12.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:05:12.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:05:12.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:05:12.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:05:12.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:05:12.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:05:12.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:05:12.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:12.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:05:12.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:05:12.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:12.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:05:12.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:05:12.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:12.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:05:12.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:05:12.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:12.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:05:12.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:05:12.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:12.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:05:12.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:05:12.240 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.240 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.240 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.240 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:12.241 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:12.241 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.81845654700001, 2023-07-02T14:05:12.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:05:12.241 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.241 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:12.241 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:12.241 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:12.241 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:12.242 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:05:12.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.242 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:12.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:12.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:12.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:05:12.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:05:12.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:05:12.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:05:12.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.245 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:05:12.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:05:12.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:05:12.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:05:12.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.246 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.246 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:05:12.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:05:12.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:05:12.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:05:12.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:12.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:05:12.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:05:12.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:12.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:05:12.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:05:12.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:12.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:05:12.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:05:12.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:12.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:05:12.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:05:12.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:12.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:05:12.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:05:12.250 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.251 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.251 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.251 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:12.251 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:12.251 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.81830589099999, 2023-07-02T14:05:12.251 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:05:12.252 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.252 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:12.252 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:12.252 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:12.252 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:12.252 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:05:12.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.252 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.253 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:12.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:12.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:12.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:05:12.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:05:12.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:05:12.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:05:12.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:05:12.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:05:12.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:05:12.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:05:12.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:05:12.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:05:12.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:05:12.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:05:12.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:12.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:05:12.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:05:12.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:12.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:05:12.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:05:12.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:12.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:05:12.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:05:12.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:12.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:05:12.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:05:12.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.261 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:12.261 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:05:12.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:05:12.261 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.261 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.261 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.261 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.262 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:12.262 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:12.262 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.81823318799999, 2023-07-02T14:05:12.262 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:05:12.262 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.262 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:12.262 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:12.263 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:12.263 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:12.263 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:05:12.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.263 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.263 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:12.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.264 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:12.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:12.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.265 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:12.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:05:12.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.265 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:05:12.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:05:12.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:05:12.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:05:12.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:05:12.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:05:12.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:05:12.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:05:12.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.268 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:05:12.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:05:12.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:12.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:05:12.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:05:12.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:12.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:05:12.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:05:12.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:12.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:05:12.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:05:12.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:12.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:05:12.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:05:12.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.271 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:12.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:05:12.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:05:12.271 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.272 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.272 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.272 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:12.272 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:12.272 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.81811619800001, 2023-07-02T14:05:12.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:05:12.273 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.273 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:12.273 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:12.273 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:12.273 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:12.273 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:05:12.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.273 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:12.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:12.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:12.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.275 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.275 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:12.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:05:12.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:05:12.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:05:12.276 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:05:12.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:05:12.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.277 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.277 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:05:12.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:05:12.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:05:12.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:05:12.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:05:12.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:05:12.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:12.279 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:05:12.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:05:12.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:12.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:05:12.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:05:12.280 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:12.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:05:12.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:05:12.280 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:12.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:05:12.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:05:12.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:12.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:05:12.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:05:12.282 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.282 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.282 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.282 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.282 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:12.283 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:12.283 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.81796674700001, 2023-07-02T14:05:12.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:05:12.283 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.283 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:12.283 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:12.283 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:12.283 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:12.284 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:05:12.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.284 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:12.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:12.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:12.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:05:12.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:05:12.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:05:12.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:05:12.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:05:12.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:05:12.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:05:12.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:05:12.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:05:12.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:05:12.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:05:12.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:05:12.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:12.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:05:12.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:05:12.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:12.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:05:12.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:05:12.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:12.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:05:12.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:05:12.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:12.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:05:12.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:05:12.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:12.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:05:12.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:05:12.292 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.293 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.293 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.293 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:12.293 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:12.293 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.79649077400001, 2023-07-02T14:05:12.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:05:12.294 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.294 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:12.294 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:12.294 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:12.294 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:12.294 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:05:12.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.295 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.295 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:12.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.295 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:12.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:12.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:05:12.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:05:12.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:05:12.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:05:12.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:05:12.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:05:12.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:05:12.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:05:12.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:05:12.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:05:12.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:05:12.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:05:12.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:12.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:05:12.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:05:12.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:12.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:05:12.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:05:12.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:12.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:05:12.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:05:12.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:12.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:05:12.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:05:12.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:12.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:05:12.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:05:12.303 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.303 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.303 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.303 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:12.304 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:12.304 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.79635823800001, 2023-07-02T14:05:12.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:05:12.304 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.304 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:12.304 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:12.304 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:12.305 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:12.305 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:05:12.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.305 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:12.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:12.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:12.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.307 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:05:12.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:05:12.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.307 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:05:12.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:05:12.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:05:12.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:05:12.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:05:12.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:05:12.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:05:12.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:05:12.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:05:12.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:05:12.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:12.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:05:12.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:05:12.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:12.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:05:12.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:05:12.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:12.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:05:12.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:05:12.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:12.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:05:12.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:05:12.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:12.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:05:12.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:05:12.313 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.313 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.314 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.314 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:12.314 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:12.314 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.735021577, 2023-07-02T14:05:12.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:05:12.314 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.315 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:12.315 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:12.315 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:12.315 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:12.315 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:05:12.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.315 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:12.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:12.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:12.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:05:12.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:05:12.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:05:12.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:05:12.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:05:12.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:05:12.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:05:12.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:05:12.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:05:12.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:05:12.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:05:12.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:05:12.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:12.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:05:12.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:05:12.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:12.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:05:12.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:05:12.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.322 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:12.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:05:12.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:05:12.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:12.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:05:12.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:05:12.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:12.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:05:12.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:05:12.324 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.324 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.324 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.324 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.324 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.324 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:12.325 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:05:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:05:12.328 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:05:12.328 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:12.325+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:12.329 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.2 to 1 (10000) 2023-07-02T14:05:12.329 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 51% complete...2023-07-02T14:05:12.154+0000 7fd0b5de8700 0 -- 172.21.15.163:0/3262395932 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x561acd559280 msgr2=0x561acd55b670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:12.329 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 100% complete...done. 2023-07-02T14:05:12.329 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:05:12.329 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=35 2023-07-02T14:05:12.329 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap35 2023-07-02T14:05:12.330 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-07-02T14:05:12.330 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:05:12.427 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:12.426+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:12.515 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:12.515 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:12.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.515 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.7 140/134 [PushOp(2:f3c0d8e8:::rbd_data.125de1449605.0000000000000093:head, version: 125'2004, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:f3c0d8e8:::rbd_data.125de1449605.0000000000000093:head@125'2004, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-07-02T14:05:12.516 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:12.513714+0000", 2023-07-02T14:05:12.516 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0015906029999999999, 2023-07-02T14:05:12.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0016400379999999999, 2023-07-02T14:05:12.516 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:12.516 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:12.516 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:12.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:12.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.513714+0000", 2023-07-02T14:05:12.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:12.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:12.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.513714+0000", 2023-07-02T14:05:12.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:12.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:12.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.513713+0000", 2023-07-02T14:05:12.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00093494400000000005 2023-07-02T14:05:12.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:12.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.514648+0000", 2023-07-02T14:05:12.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7428e-05 2023-07-02T14:05:12.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:12.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.514666+0000", 2023-07-02T14:05:12.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2620000000000004e-06 2023-07-02T14:05:12.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:12.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.514675+0000", 2023-07-02T14:05:12.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1979999999999999e-05 2023-07-02T14:05:12.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:12.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.514747+0000", 2023-07-02T14:05:12.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6020000000000006e-06 2023-07-02T14:05:12.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:12.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:12.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:12.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.514756+0000", 2023-07-02T14:05:12.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001042339 2023-07-02T14:05:12.521 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.522 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:12.522 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.522 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:12.522 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2023-07-02T14:05:12.522 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 1 2023-07-02T14:05:12.522 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:12.527 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:05:12.528 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:05:12.528 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:12.527+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:12.547 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:12.567 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:12.587 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:12.607 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:12.627 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:12.628 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:05:12.628 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:05:12.629 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:05:12.629 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:05:12.629 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:05:12.629 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:05:12.629 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:12.628+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:12.673 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:12.674 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:12.686 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:05:12.691 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:05:12.693 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:05:12.705 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2023-07-02T14:05:12.268878+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5982788219,"num_objects":1791,"num_object_clones":1412,"num_object_copies":3582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1791,"num_whiteouts":0,"num_read":32457,"num_read_kb":7739706,"num_write":14927,"num_write_kb":5813212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":851,"num_bytes_recovered":2904401463,"num_keys_recovered":70,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14672,"ondisk_log_size":14672,"up":18,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":9766184,"kb_used_data":9594888,"kb_used_omap":229,"kb_used_meta":170010,"kb_avail":367721176,"statfs":{"total":386547056640,"available":376546484224,"internally_reserved":0,"allocated":9825165312,"data_stored":11540293821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":174090462},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":143868066,"num_objects":41,"num_object_clones":40,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":1175,"num_read_kb":62274,"num_write":253,"num_write_kb":140584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":440467780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000624"},"pg_stats":[{"pgid":"2.7","version":"137'2116","reported_seq":4673,"reported_epoch":137,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-07-02T14:05:09.833316+0000","last_change":"2023-07-02T14:05:06.140063+0000","last_active":"2023-07-02T14:05:09.833316+0000","last_peered":"2023-07-02T14:05:09.833316+0000","last_clean":"2023-07-02T14:05:03.081045+0000","last_became_active":"2023-07-02T14:05:06.138824+0000","last_became_peered":"2023-07-02T14:05:06.138824+0000","last_unstale":"2023-07-02T14:05:09.833316+0000","last_undegraded":"2023-07-02T14:05:06.091635+0000","last_fullsized":"2023-07-02T14:05:06.088179+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2116,"ondisk_log_size":2116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":784893934,"num_objects":236,"num_object_clones":185,"num_object_copies":472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":236,"num_whiteouts":0,"num_read":17270,"num_read_kb":986139,"num_write":1825,"num_write_kb":762999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":245,"num_bytes_recovered":786067668,"num_keys_recovered":66,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[2],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2","objects":55},{"shards":"2,4","objects":170}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"137'1620","reported_seq":2719,"reported_epoch":138,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-07-02T14:05:10.303839+0000","last_change":"2023-07-02T14:05:08.503259+0000","last_active":"2023-07-02T14:05:10.303839+0000","last_peered":"2023-07-02T14:05:10.303839+0000","last_clean":"2023-07-02T14:03:53.553905+0000","last_became_active":"2023-07-02T14:05:06.140061+0000","last_became_peered":"2023-07-02T14:05:06.140061+0000","last_unstale":"2023-07-02T14:05:10.303839+0000","last_undegraded":"2023-07-02T14:05:06.098034+0000","last_fullsized":"2023-07-02T14:05:06.092657+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1620,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":774988262,"num_objects":239,"num_object_clones":193,"num_object_copies":478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":239,"num_whiteouts":0,"num_read":881,"num_read_kb":1024149,"num_write":1427,"num_write_kb":753019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":440467780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":192}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"126'1747","reported_seq":2035,"reported_epoch":134,"state":"remapped+peering","last_fresh":"2023-07-02T14:05:05.087600+0000","last_change":"2023-07-02T14:05:05.087600+0000","last_active":"2023-07-02T14:05:05.087372+0000","last_peered":"2023-07-02T14:05:03.315649+0000","last_clean":"2023-07-02T14:03:52.550908+0000","last_became_active":"2023-07-02T14:03:55.559789+0000","last_became_peered":"2023-07-02T14:03:55.559789+0000","last_unstale":"2023-07-02T14:05:05.087600+0000","last_undegraded":"2023-07-02T14:05:05.087600+0000","last_fullsized":"2023-07-02T14:05:05.087600+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1747,"ondisk_log_size":1747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":737685990,"num_objects":206,"num_object_clones":162,"num_object_copies":412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":328,"num_read_kb":927502,"num_write":1585,"num_write_kb":710352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"137'3462","reported_seq":4111,"reported_epoch":137,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-07-02T14:05:09.848171+0000","last_change":"2023-07-02T14:05:06.171249+0000","last_active":"2023-07-02T14:05:09.848171+0000","last_peered":"2023-07-02T14:05:09.848171+0000","last_clean":"2023-07-02T14:05:03.082409+0000","last_became_active":"2023-07-02T14:05:06.167341+0000","last_became_peered":"2023-07-02T14:05:06.167341+0000","last_unstale":"2023-07-02T14:05:09.848171+0000","last_undegraded":"2023-07-02T14:05:06.102378+0000","last_fullsized":"2023-07-02T14:05:06.092644+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3462,"ondisk_log_size":3462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":847430620,"num_objects":251,"num_object_clones":197,"num_object_copies":502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":251,"num_whiteouts":0,"num_read":11526,"num_read_kb":1126547,"num_write":5038,"num_write_kb":827768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8523954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[1],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1","objects":11},{"shards":"1,4","objects":240}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":88,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-07-02T14:05:09.102391+0000","last_change":"2023-07-02T14:05:05.089421+0000","last_active":"2023-07-02T14:05:09.102391+0000","last_peered":"2023-07-02T14:05:09.102391+0000","last_clean":"2023-07-02T14:05:09.102391+0000","last_became_active":"2023-07-02T14:05:05.088812+0000","last_became_peered":"2023-07-02T14:05:05.088812+0000","last_unstale":"2023-07-02T14:05:09.102391+0000","last_undegraded":"2023-07-02T14:05:09.102391+0000","last_fullsized":"2023-07-02T14:05:09.102391+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"137'1609","reported_seq":2385,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-07-02T14:05:09.845246+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:05:09.845246+0000","last_peered":"2023-07-02T14:05:09.845246+0000","last_clean":"2023-07-02T14:05:09.845246+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:05:09.845246+0000","last_undegraded":"2023-07-02T14:05:09.845246+0000","last_fullsized":"2023-07-02T14:05:09.845246+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1609,"ondisk_log_size":1609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":824668986,"num_objects":243,"num_object_clones":191,"num_object_copies":486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":243,"num_whiteouts":0,"num_read":632,"num_read_kb":1051848,"num_write":1418,"num_write_kb":802854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"137'1590","reported_seq":2062,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-07-02T14:05:09.845230+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:05:09.845230+0000","last_peered":"2023-07-02T14:05:09.845230+0000","last_clean":"2023-07-02T14:05:09.845230+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:05:09.845230+0000","last_undegraded":"2023-07-02T14:05:09.845230+0000","last_fullsized":"2023-07-02T14:05:09.845230+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1590,"ondisk_log_size":1590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":820769578,"num_objects":238,"num_object_clones":186,"num_object_copies":476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":238,"num_whiteouts":0,"num_read":421,"num_read_kb":984584,"num_write":1404,"num_write_kb":798425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"137'1198","reported_seq":2203,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-07-02T14:05:09.846454+0000","last_change":"2023-07-02T14:03:13.073775+0000","last_active":"2023-07-02T14:05:09.846454+0000","last_peered":"2023-07-02T14:05:09.846454+0000","last_clean":"2023-07-02T14:05:09.846454+0000","last_became_active":"2023-07-02T14:03:13.073206+0000","last_became_peered":"2023-07-02T14:03:13.073206+0000","last_unstale":"2023-07-02T14:05:09.846454+0000","last_undegraded":"2023-07-02T14:05:09.846454+0000","last_fullsized":"2023-07-02T14:05:09.846454+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1198,"ondisk_log_size":1198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":579207835,"num_objects":191,"num_object_clones":151,"num_object_copies":382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":973,"num_read_kb":828824,"num_write":1047,"num_write_kb":562242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":96600083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"137'1330","reported_seq":1755,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-07-02T14:05:09.850002+0000","last_change":"2023-07-02T14:03:14.074306+0000","last_active":"2023-07-02T14:05:09.850002+0000","last_peered":"2023-07-02T14:05:09.850002+0000","last_clean":"2023-07-02T14:05:09.850002+0000","last_became_active":"2023-07-02T14:03:14.073735+0000","last_became_peered":"2023-07-02T14:03:14.073735+0000","last_unstale":"2023-07-02T14:05:09.850002+0000","last_undegraded":"2023-07-02T14:05:09.850002+0000","last_fullsized":"2023-07-02T14:05:09.850002+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1330,"ondisk_log_size":1330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":613143014,"num_objects":187,"num_object_clones":147,"num_object_copies":374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":426,"num_read_kb":810113,"num_write":1183,"num_write_kb":595553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":87949312,"num_keys_recovered":4,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5982788219,"num_objects":1791,"num_object_clones":1412,"num_object_copies":3582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1791,"num_whiteouts":0,"num_read":32457,"num_read_kb":7739706,"num_write":14927,"num_write_kb":5813212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":851,"num_bytes_recovered":2904401463,"num_keys_recovered":70,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11686809600,"data_stored":13674936600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},"log_size":14672,"ondisk_log_size":14672,"up":16,"acting":12,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640299,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":0,"seq":47244640299,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":133,"seq":571230650370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1214172,"kb_used_data":1188308,"kb_used_omap":229,"kb_used_meta":24602,"kb_avail":93157668,"statfs":{"total":96636764160,"available":95393452032,"internally_reserved":0,"allocated":1216827392,"data_stored":1405766805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":25192670},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640299,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2362588,"kb_used_data":2324948,"kb_used_omap":0,"kb_used_meta":37632,"kb_avail":92009252,"statfs":{"total":96636764160,"available":94217474048,"internally_reserved":0,"allocated":2380746752,"data_stored":2738075114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38535168},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.29399999999999998,"15min":0.29599999999999999},"min":{"1min":0.218,"5min":0.182,"15min":0.182},"max":{"1min":0.378,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.245},{"interface":"front","average":{"1min":0.317,"5min":0.315,"15min":0.314},"min":{"1min":0.23000000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.26100000000000001}]},{"osd":1,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34499999999999997,"15min":0.34699999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.48499999999999999,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.314,"15min":0.317},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.39700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.28799999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.45000000000000001,"15min":0.45100000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46999999999999997,"15min":0.46800000000000003},"min":{"1min":0.35299999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.41999999999999998}]},{"osd":4,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.443,"15min":0.44700000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.56799999999999995,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.45900000000000002,"15min":0.46000000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.57599999999999996,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.47399999999999998}]},{"osd":5,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47899999999999998,"15min":0.48299999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.65200000000000002,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.436},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.47599999999999998,"15min":0.48099999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.54600000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.35899999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1906968,"kb_used_data":1867600,"kb_used_omap":0,"kb_used_meta":39360,"kb_avail":92464872,"statfs":{"total":96636764160,"available":94684028928,"internally_reserved":0,"allocated":1912422400,"data_stored":2277166591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40304640},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.29099999999999998,"15min":0.29299999999999998},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.29699999999999999,"5min":0.30099999999999999,"15min":0.29999999999999999},"min":{"1min":0.216,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.42199999999999999,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.21099999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.34300000000000003,"15min":0.34799999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.48599999999999999,"5min":0.746,"15min":0.746},"last":0.247},{"interface":"front","average":{"1min":0.314,"5min":0.34100000000000003,"15min":0.34699999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.40000000000000002,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.32800000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.35599999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.49399999999999999,"15min":0.496},"min":{"1min":0.32400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.84899999999999998,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.28399999999999997}]},{"osd":4,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.47799999999999998,"15min":0.48499999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.77600000000000002,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.48399999999999999,"15min":0.49299999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.61099999999999999,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.307}]},{"osd":5,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.48099999999999998,"15min":0.495},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.59199999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.502,"15min":0.51200000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.38800000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640299,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4282456,"kb_used_data":4214032,"kb_used_omap":0,"kb_used_meta":68416,"kb_avail":90089384,"statfs":{"total":96636764160,"available":92251529216,"internally_reserved":0,"allocated":4315168768,"data_stored":5119285311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":70057984},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.20200000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.221},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.32200000000000001,"15min":0.33000000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.41099999999999998,"5min":0.496,"15min":0.496},"last":0.24399999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.33800000000000002,"15min":0.33200000000000002},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.222},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.36599999999999999,"15min":0.376},"min":{"1min":0.251,"5min":0.222,"15min":0.222},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.28499999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.497,"15min":0.501},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.624,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.498,"15min":0.499},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.64400000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.30199999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.48699999999999999,"15min":0.503},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.54300000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.498,"15min":0.504},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.38600000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51400000000000001,"15min":0.51500000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.5589999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.45100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1921400832,"data_stored":2287146599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4440342528,"data_stored":5258195111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2433634304,"data_stored":2791964242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1302298624,"data_stored":1494263790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1589133312,"data_stored":1843366858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:05:12.706 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:05:12.768 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:12.768+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:12.830 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:12.829+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:12.870 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:12.871 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:12.882 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:05:12.931 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:12.930+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:13.009 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:13.009 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:13.019 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:05:13.027 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:05:13.030 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:05:13.032 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:13.031+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:13.043 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2023-07-02T14:05:12.268878+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5982788219,"num_objects":1791,"num_object_clones":1412,"num_object_copies":3582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1791,"num_whiteouts":0,"num_read":32457,"num_read_kb":7739706,"num_write":14927,"num_write_kb":5813212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":851,"num_bytes_recovered":2904401463,"num_keys_recovered":70,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14672,"ondisk_log_size":14672,"up":18,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":9766184,"kb_used_data":9594888,"kb_used_omap":229,"kb_used_meta":170010,"kb_avail":367721176,"statfs":{"total":386547056640,"available":376546484224,"internally_reserved":0,"allocated":9825165312,"data_stored":11540293821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":174090462},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":143868066,"num_objects":41,"num_object_clones":40,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":1175,"num_read_kb":62274,"num_write":253,"num_write_kb":140584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":440467780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000624"},"pg_stats":[{"pgid":"2.7","version":"137'2116","reported_seq":4673,"reported_epoch":137,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-07-02T14:05:09.833316+0000","last_change":"2023-07-02T14:05:06.140063+0000","last_active":"2023-07-02T14:05:09.833316+0000","last_peered":"2023-07-02T14:05:09.833316+0000","last_clean":"2023-07-02T14:05:03.081045+0000","last_became_active":"2023-07-02T14:05:06.138824+0000","last_became_peered":"2023-07-02T14:05:06.138824+0000","last_unstale":"2023-07-02T14:05:09.833316+0000","last_undegraded":"2023-07-02T14:05:06.091635+0000","last_fullsized":"2023-07-02T14:05:06.088179+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2116,"ondisk_log_size":2116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":784893934,"num_objects":236,"num_object_clones":185,"num_object_copies":472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":236,"num_whiteouts":0,"num_read":17270,"num_read_kb":986139,"num_write":1825,"num_write_kb":762999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":245,"num_bytes_recovered":786067668,"num_keys_recovered":66,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[2],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2","objects":55},{"shards":"2,4","objects":170}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"137'1620","reported_seq":2719,"reported_epoch":138,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-07-02T14:05:10.303839+0000","last_change":"2023-07-02T14:05:08.503259+0000","last_active":"2023-07-02T14:05:10.303839+0000","last_peered":"2023-07-02T14:05:10.303839+0000","last_clean":"2023-07-02T14:03:53.553905+0000","last_became_active":"2023-07-02T14:05:06.140061+0000","last_became_peered":"2023-07-02T14:05:06.140061+0000","last_unstale":"2023-07-02T14:05:10.303839+0000","last_undegraded":"2023-07-02T14:05:06.098034+0000","last_fullsized":"2023-07-02T14:05:06.092657+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1620,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":774988262,"num_objects":239,"num_object_clones":193,"num_object_copies":478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":239,"num_whiteouts":0,"num_read":881,"num_read_kb":1024149,"num_write":1427,"num_write_kb":753019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":440467780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":192}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"126'1747","reported_seq":2035,"reported_epoch":134,"state":"remapped+peering","last_fresh":"2023-07-02T14:05:05.087600+0000","last_change":"2023-07-02T14:05:05.087600+0000","last_active":"2023-07-02T14:05:05.087372+0000","last_peered":"2023-07-02T14:05:03.315649+0000","last_clean":"2023-07-02T14:03:52.550908+0000","last_became_active":"2023-07-02T14:03:55.559789+0000","last_became_peered":"2023-07-02T14:03:55.559789+0000","last_unstale":"2023-07-02T14:05:05.087600+0000","last_undegraded":"2023-07-02T14:05:05.087600+0000","last_fullsized":"2023-07-02T14:05:05.087600+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1747,"ondisk_log_size":1747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":737685990,"num_objects":206,"num_object_clones":162,"num_object_copies":412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":328,"num_read_kb":927502,"num_write":1585,"num_write_kb":710352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"137'3462","reported_seq":4111,"reported_epoch":137,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-07-02T14:05:09.848171+0000","last_change":"2023-07-02T14:05:06.171249+0000","last_active":"2023-07-02T14:05:09.848171+0000","last_peered":"2023-07-02T14:05:09.848171+0000","last_clean":"2023-07-02T14:05:03.082409+0000","last_became_active":"2023-07-02T14:05:06.167341+0000","last_became_peered":"2023-07-02T14:05:06.167341+0000","last_unstale":"2023-07-02T14:05:09.848171+0000","last_undegraded":"2023-07-02T14:05:06.102378+0000","last_fullsized":"2023-07-02T14:05:06.092644+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3462,"ondisk_log_size":3462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":847430620,"num_objects":251,"num_object_clones":197,"num_object_copies":502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":251,"num_whiteouts":0,"num_read":11526,"num_read_kb":1126547,"num_write":5038,"num_write_kb":827768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8523954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[1],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1","objects":11},{"shards":"1,4","objects":240}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":88,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-07-02T14:05:09.102391+0000","last_change":"2023-07-02T14:05:05.089421+0000","last_active":"2023-07-02T14:05:09.102391+0000","last_peered":"2023-07-02T14:05:09.102391+0000","last_clean":"2023-07-02T14:05:09.102391+0000","last_became_active":"2023-07-02T14:05:05.088812+0000","last_became_peered":"2023-07-02T14:05:05.088812+0000","last_unstale":"2023-07-02T14:05:09.102391+0000","last_undegraded":"2023-07-02T14:05:09.102391+0000","last_fullsized":"2023-07-02T14:05:09.102391+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"137'1609","reported_seq":2385,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-07-02T14:05:09.845246+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:05:09.845246+0000","last_peered":"2023-07-02T14:05:09.845246+0000","last_clean":"2023-07-02T14:05:09.845246+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:05:09.845246+0000","last_undegraded":"2023-07-02T14:05:09.845246+0000","last_fullsized":"2023-07-02T14:05:09.845246+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1609,"ondisk_log_size":1609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":824668986,"num_objects":243,"num_object_clones":191,"num_object_copies":486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":243,"num_whiteouts":0,"num_read":632,"num_read_kb":1051848,"num_write":1418,"num_write_kb":802854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"137'1590","reported_seq":2062,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-07-02T14:05:09.845230+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:05:09.845230+0000","last_peered":"2023-07-02T14:05:09.845230+0000","last_clean":"2023-07-02T14:05:09.845230+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:05:09.845230+0000","last_undegraded":"2023-07-02T14:05:09.845230+0000","last_fullsized":"2023-07-02T14:05:09.845230+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1590,"ondisk_log_size":1590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":820769578,"num_objects":238,"num_object_clones":186,"num_object_copies":476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":238,"num_whiteouts":0,"num_read":421,"num_read_kb":984584,"num_write":1404,"num_write_kb":798425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"137'1198","reported_seq":2203,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-07-02T14:05:09.846454+0000","last_change":"2023-07-02T14:03:13.073775+0000","last_active":"2023-07-02T14:05:09.846454+0000","last_peered":"2023-07-02T14:05:09.846454+0000","last_clean":"2023-07-02T14:05:09.846454+0000","last_became_active":"2023-07-02T14:03:13.073206+0000","last_became_peered":"2023-07-02T14:03:13.073206+0000","last_unstale":"2023-07-02T14:05:09.846454+0000","last_undegraded":"2023-07-02T14:05:09.846454+0000","last_fullsized":"2023-07-02T14:05:09.846454+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1198,"ondisk_log_size":1198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":579207835,"num_objects":191,"num_object_clones":151,"num_object_copies":382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":973,"num_read_kb":828824,"num_write":1047,"num_write_kb":562242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":96600083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"137'1330","reported_seq":1755,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-07-02T14:05:09.850002+0000","last_change":"2023-07-02T14:03:14.074306+0000","last_active":"2023-07-02T14:05:09.850002+0000","last_peered":"2023-07-02T14:05:09.850002+0000","last_clean":"2023-07-02T14:05:09.850002+0000","last_became_active":"2023-07-02T14:03:14.073735+0000","last_became_peered":"2023-07-02T14:03:14.073735+0000","last_unstale":"2023-07-02T14:05:09.850002+0000","last_undegraded":"2023-07-02T14:05:09.850002+0000","last_fullsized":"2023-07-02T14:05:09.850002+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1330,"ondisk_log_size":1330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":613143014,"num_objects":187,"num_object_clones":147,"num_object_copies":374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":426,"num_read_kb":810113,"num_write":1183,"num_write_kb":595553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":87949312,"num_keys_recovered":4,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5982788219,"num_objects":1791,"num_object_clones":1412,"num_object_copies":3582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1791,"num_whiteouts":0,"num_read":32457,"num_read_kb":7739706,"num_write":14927,"num_write_kb":5813212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":851,"num_bytes_recovered":2904401463,"num_keys_recovered":70,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11686809600,"data_stored":13674936600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},"log_size":14672,"ondisk_log_size":14672,"up":16,"acting":12,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640299,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":0,"seq":47244640299,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":133,"seq":571230650370,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1214172,"kb_used_data":1188308,"kb_used_omap":229,"kb_used_meta":24602,"kb_avail":93157668,"statfs":{"total":96636764160,"available":95393452032,"internally_reserved":0,"allocated":1216827392,"data_stored":1405766805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":25192670},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640299,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2362588,"kb_used_data":2324948,"kb_used_omap":0,"kb_used_meta":37632,"kb_avail":92009252,"statfs":{"total":96636764160,"available":94217474048,"internally_reserved":0,"allocated":2380746752,"data_stored":2738075114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38535168},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.29399999999999998,"15min":0.29599999999999999},"min":{"1min":0.218,"5min":0.182,"15min":0.182},"max":{"1min":0.378,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.245},{"interface":"front","average":{"1min":0.317,"5min":0.315,"15min":0.314},"min":{"1min":0.23000000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.26100000000000001}]},{"osd":1,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34499999999999997,"15min":0.34699999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.48499999999999999,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.314,"15min":0.317},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.39700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.28799999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.45000000000000001,"15min":0.45100000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46999999999999997,"15min":0.46800000000000003},"min":{"1min":0.35299999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.41999999999999998}]},{"osd":4,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.443,"15min":0.44700000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.56799999999999995,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.45900000000000002,"15min":0.46000000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.57599999999999996,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.47399999999999998}]},{"osd":5,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47899999999999998,"15min":0.48299999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.65200000000000002,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.436},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.47599999999999998,"15min":0.48099999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.54600000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.35899999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1906968,"kb_used_data":1867600,"kb_used_omap":0,"kb_used_meta":39360,"kb_avail":92464872,"statfs":{"total":96636764160,"available":94684028928,"internally_reserved":0,"allocated":1912422400,"data_stored":2277166591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40304640},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.29099999999999998,"15min":0.29299999999999998},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.29699999999999999,"5min":0.30099999999999999,"15min":0.29999999999999999},"min":{"1min":0.216,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.42199999999999999,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.21099999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.34300000000000003,"15min":0.34799999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.48599999999999999,"5min":0.746,"15min":0.746},"last":0.247},{"interface":"front","average":{"1min":0.314,"5min":0.34100000000000003,"15min":0.34699999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.40000000000000002,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.32800000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.35599999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.49399999999999999,"15min":0.496},"min":{"1min":0.32400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.84899999999999998,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.28399999999999997}]},{"osd":4,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.47799999999999998,"15min":0.48499999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.77600000000000002,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.48399999999999999,"15min":0.49299999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.61099999999999999,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.307}]},{"osd":5,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.48099999999999998,"15min":0.495},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.59199999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.502,"15min":0.51200000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.38800000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640299,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4282456,"kb_used_data":4214032,"kb_used_omap":0,"kb_used_meta":68416,"kb_avail":90089384,"statfs":{"total":96636764160,"available":92251529216,"internally_reserved":0,"allocated":4315168768,"data_stored":5119285311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":70057984},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.20200000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.221},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.32200000000000001,"15min":0.33000000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.41099999999999998,"5min":0.496,"15min":0.496},"last":0.24399999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.33800000000000002,"15min":0.33200000000000002},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.222},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.36599999999999999,"15min":0.376},"min":{"1min":0.251,"5min":0.222,"15min":0.222},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.28499999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.497,"15min":0.501},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.624,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.498,"15min":0.499},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.64400000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.30199999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.48699999999999999,"15min":0.503},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.54300000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.498,"15min":0.504},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.38600000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51400000000000001,"15min":0.51500000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.5589999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.45100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1921400832,"data_stored":2287146599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4440342528,"data_stored":5258195111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2433634304,"data_stored":2791964242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1302298624,"data_stored":1494263790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1589133312,"data_stored":1843366858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:05:13.044 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-07-02T14:05:13.133 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:13.132+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:13.158 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:13.158 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:13.170 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:05:13.234 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:13.233+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:13.310 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:13.310 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:13.321 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:05:13.335 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:13.334+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:13.435 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:13.435+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:13.463 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:13.463 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:13.475 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:05:13.513 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:05:13.514 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 479/3582 objects degraded (13.372%), 3 pgs degraded","count":3},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":216,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":141,"num_osds":6,"num_up_osds":6,"osd_up_since":1688306704,"num_in_osds":4,"osd_in_since":1688306681,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"remapped+peering","count":1}],"num_pgs":9,"num_pools":2,"num_objects":1791,"data_bytes":5982788219,"bytes_used":10000572416,"bytes_avail":376546484224,"bytes_total":386547056640,"inactive_pgs_ratio":0.1111111119389534,"degraded_objects":479,"degraded_total":3582,"degraded_ratio":0.13372417643774429,"recovering_objects_per_sec":33,"recovering_bytes_per_sec":110099749,"recovering_keys_per_sec":0,"num_objects_recovered":135,"num_bytes_recovered":440467780,"num_keys_recovered":0,"read_bytes_sec":15939654,"write_bytes_sec":35983884,"read_op_per_sec":293,"write_op_per_sec":63},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2023-07-02T14:02:46.242493+0000","services":{}},"progress_events":{}} 2023-07-02T14:05:13.529 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2023-07-02T14:05:13.529 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:05:13.536 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:13.535+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:13.620 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:13.620 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:13.632 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:05:13.637 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:13.636+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:13.737 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:13.736+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:13.774 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:13.774 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:13.786 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:05:13.838 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:13.837+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:13.907 INFO:teuthology.orchestra.run.smithi016.stderr:2023-07-02T14:05:13.906+0000 7f4d27fff700 0 --2- 172.21.15.16:0/3277021241 >> [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] conn(0x7f4d28066830 0x7f4d28125900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-07-02T14:05:13.919 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:13.920 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:13.933 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:05:13.939 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:13.938+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:14.039 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:14.038+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:14.073 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:14.073 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:14.084 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:05:14.116 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:05:14.119 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:05:14.137 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2023-07-02T14:05:13.113582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5991176827,"num_objects":1793,"num_object_clones":1414,"num_object_copies":3586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1793,"num_whiteouts":0,"num_read":32457,"num_read_kb":7739706,"num_write":14931,"num_write_kb":5821404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1006,"num_bytes_recovered":3477133017,"num_keys_recovered":70,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14678,"ondisk_log_size":14678,"up":18,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":10708692,"kb_used_data":10543604,"kb_used_omap":229,"kb_used_meta":164826,"kb_avail":366778668,"statfs":{"total":386547056640,"available":375581356032,"internally_reserved":0,"allocated":10796650496,"data_stored":12690671439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":168782046},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":152256674,"num_objects":43,"num_object_clones":42,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":1175,"num_read_kb":62274,"num_write":257,"num_write_kb":148776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":1013199334,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.014915"},"pg_stats":[{"pgid":"2.7","version":"137'2116","reported_seq":4673,"reported_epoch":137,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-07-02T14:05:09.833316+0000","last_change":"2023-07-02T14:05:06.140063+0000","last_active":"2023-07-02T14:05:09.833316+0000","last_peered":"2023-07-02T14:05:09.833316+0000","last_clean":"2023-07-02T14:05:03.081045+0000","last_became_active":"2023-07-02T14:05:06.138824+0000","last_became_peered":"2023-07-02T14:05:06.138824+0000","last_unstale":"2023-07-02T14:05:09.833316+0000","last_undegraded":"2023-07-02T14:05:06.091635+0000","last_fullsized":"2023-07-02T14:05:06.088179+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2116,"ondisk_log_size":2116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":784893934,"num_objects":236,"num_object_clones":185,"num_object_copies":472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":236,"num_whiteouts":0,"num_read":17270,"num_read_kb":986139,"num_write":1825,"num_write_kb":762999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":245,"num_bytes_recovered":786067668,"num_keys_recovered":66,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[2],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2","objects":55},{"shards":"2,4","objects":170}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"137'1620","reported_seq":2719,"reported_epoch":138,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-07-02T14:05:10.303839+0000","last_change":"2023-07-02T14:05:08.503259+0000","last_active":"2023-07-02T14:05:10.303839+0000","last_peered":"2023-07-02T14:05:10.303839+0000","last_clean":"2023-07-02T14:03:53.553905+0000","last_became_active":"2023-07-02T14:05:06.140061+0000","last_became_peered":"2023-07-02T14:05:06.140061+0000","last_unstale":"2023-07-02T14:05:10.303839+0000","last_undegraded":"2023-07-02T14:05:06.098034+0000","last_fullsized":"2023-07-02T14:05:06.092657+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1620,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":774988262,"num_objects":239,"num_object_clones":193,"num_object_copies":478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":239,"num_whiteouts":0,"num_read":881,"num_read_kb":1024149,"num_write":1427,"num_write_kb":753019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":440467780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":192}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"138'1753","reported_seq":2053,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-07-02T14:05:11.107594+0000","last_change":"2023-07-02T14:05:10.106568+0000","last_active":"2023-07-02T14:05:11.107594+0000","last_peered":"2023-07-02T14:05:11.107594+0000","last_clean":"2023-07-02T14:05:11.107594+0000","last_became_active":"2023-07-02T14:05:10.103318+0000","last_became_peered":"2023-07-02T14:05:10.103318+0000","last_unstale":"2023-07-02T14:05:11.107594+0000","last_undegraded":"2023-07-02T14:05:11.107594+0000","last_fullsized":"2023-07-02T14:05:11.107594+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1753,"ondisk_log_size":1753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":746074598,"num_objects":208,"num_object_clones":164,"num_object_copies":416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":208,"num_whiteouts":0,"num_read":328,"num_read_kb":927502,"num_write":1589,"num_write_kb":718544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":572731554,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"137'3462","reported_seq":4111,"reported_epoch":137,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-07-02T14:05:09.848171+0000","last_change":"2023-07-02T14:05:06.171249+0000","last_active":"2023-07-02T14:05:09.848171+0000","last_peered":"2023-07-02T14:05:09.848171+0000","last_clean":"2023-07-02T14:05:03.082409+0000","last_became_active":"2023-07-02T14:05:06.167341+0000","last_became_peered":"2023-07-02T14:05:06.167341+0000","last_unstale":"2023-07-02T14:05:09.848171+0000","last_undegraded":"2023-07-02T14:05:06.102378+0000","last_fullsized":"2023-07-02T14:05:06.092644+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3462,"ondisk_log_size":3462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":847430620,"num_objects":251,"num_object_clones":197,"num_object_copies":502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":251,"num_whiteouts":0,"num_read":11526,"num_read_kb":1126547,"num_write":5038,"num_write_kb":827768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8523954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[1],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1","objects":11},{"shards":"1,4","objects":240}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":88,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-07-02T14:05:09.102391+0000","last_change":"2023-07-02T14:05:05.089421+0000","last_active":"2023-07-02T14:05:09.102391+0000","last_peered":"2023-07-02T14:05:09.102391+0000","last_clean":"2023-07-02T14:05:09.102391+0000","last_became_active":"2023-07-02T14:05:05.088812+0000","last_became_peered":"2023-07-02T14:05:05.088812+0000","last_unstale":"2023-07-02T14:05:09.102391+0000","last_undegraded":"2023-07-02T14:05:09.102391+0000","last_fullsized":"2023-07-02T14:05:09.102391+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"137'1609","reported_seq":2385,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-07-02T14:05:09.845246+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:05:09.845246+0000","last_peered":"2023-07-02T14:05:09.845246+0000","last_clean":"2023-07-02T14:05:09.845246+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:05:09.845246+0000","last_undegraded":"2023-07-02T14:05:09.845246+0000","last_fullsized":"2023-07-02T14:05:09.845246+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1609,"ondisk_log_size":1609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":824668986,"num_objects":243,"num_object_clones":191,"num_object_copies":486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":243,"num_whiteouts":0,"num_read":632,"num_read_kb":1051848,"num_write":1418,"num_write_kb":802854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"137'1590","reported_seq":2062,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-07-02T14:05:09.845230+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:05:09.845230+0000","last_peered":"2023-07-02T14:05:09.845230+0000","last_clean":"2023-07-02T14:05:09.845230+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:05:09.845230+0000","last_undegraded":"2023-07-02T14:05:09.845230+0000","last_fullsized":"2023-07-02T14:05:09.845230+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1590,"ondisk_log_size":1590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":820769578,"num_objects":238,"num_object_clones":186,"num_object_copies":476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":238,"num_whiteouts":0,"num_read":421,"num_read_kb":984584,"num_write":1404,"num_write_kb":798425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"137'1198","reported_seq":2203,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-07-02T14:05:09.846454+0000","last_change":"2023-07-02T14:03:13.073775+0000","last_active":"2023-07-02T14:05:09.846454+0000","last_peered":"2023-07-02T14:05:09.846454+0000","last_clean":"2023-07-02T14:05:09.846454+0000","last_became_active":"2023-07-02T14:03:13.073206+0000","last_became_peered":"2023-07-02T14:03:13.073206+0000","last_unstale":"2023-07-02T14:05:09.846454+0000","last_undegraded":"2023-07-02T14:05:09.846454+0000","last_fullsized":"2023-07-02T14:05:09.846454+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1198,"ondisk_log_size":1198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":579207835,"num_objects":191,"num_object_clones":151,"num_object_copies":382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":973,"num_read_kb":828824,"num_write":1047,"num_write_kb":562242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":96600083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"137'1330","reported_seq":1755,"reported_epoch":137,"state":"active+clean","last_fresh":"2023-07-02T14:05:09.850002+0000","last_change":"2023-07-02T14:03:14.074306+0000","last_active":"2023-07-02T14:05:09.850002+0000","last_peered":"2023-07-02T14:05:09.850002+0000","last_clean":"2023-07-02T14:05:09.850002+0000","last_became_active":"2023-07-02T14:03:14.073735+0000","last_became_peered":"2023-07-02T14:03:14.073735+0000","last_unstale":"2023-07-02T14:05:09.850002+0000","last_undegraded":"2023-07-02T14:05:09.850002+0000","last_fullsized":"2023-07-02T14:05:09.850002+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1330,"ondisk_log_size":1330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":613143014,"num_objects":187,"num_object_clones":147,"num_object_copies":374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":426,"num_read_kb":810113,"num_write":1183,"num_write_kb":595553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":87949312,"num_keys_recovered":4,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":5991176827,"num_objects":1793,"num_object_clones":1414,"num_object_copies":3586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1793,"num_whiteouts":0,"num_read":32457,"num_read_kb":7739706,"num_write":14931,"num_write_kb":5821404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1006,"num_bytes_recovered":3477133017,"num_keys_recovered":70,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12737937408,"data_stored":14923611704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},"log_size":14678,"ondisk_log_size":14678,"up":16,"acting":13,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640299,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":0,"seq":47244640299,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":133,"seq":571230650371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2156680,"kb_used_data":2137024,"kb_used_omap":229,"kb_used_meta":19418,"kb_avail":92215160,"statfs":{"total":96636764160,"available":94428323840,"internally_reserved":0,"allocated":2188312576,"data_stored":2556144423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":19884254},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640299,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2362588,"kb_used_data":2324948,"kb_used_omap":0,"kb_used_meta":37632,"kb_avail":92009252,"statfs":{"total":96636764160,"available":94217474048,"internally_reserved":0,"allocated":2380746752,"data_stored":2738075114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38535168},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.29399999999999998,"15min":0.29599999999999999},"min":{"1min":0.218,"5min":0.182,"15min":0.182},"max":{"1min":0.378,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.245},{"interface":"front","average":{"1min":0.317,"5min":0.315,"15min":0.314},"min":{"1min":0.23000000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.26100000000000001}]},{"osd":1,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34499999999999997,"15min":0.34699999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.48499999999999999,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.314,"15min":0.317},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.39700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.28799999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.45000000000000001,"15min":0.45100000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46999999999999997,"15min":0.46800000000000003},"min":{"1min":0.35299999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.41999999999999998}]},{"osd":4,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.443,"15min":0.44700000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.56799999999999995,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.45900000000000002,"15min":0.46000000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.57599999999999996,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.47399999999999998}]},{"osd":5,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47899999999999998,"15min":0.48299999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.65200000000000002,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.436},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.47599999999999998,"15min":0.48099999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.54600000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.35899999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1906968,"kb_used_data":1867600,"kb_used_omap":0,"kb_used_meta":39360,"kb_avail":92464872,"statfs":{"total":96636764160,"available":94684028928,"internally_reserved":0,"allocated":1912422400,"data_stored":2277166591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40304640},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.29099999999999998,"15min":0.29299999999999998},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.29699999999999999,"5min":0.30099999999999999,"15min":0.29999999999999999},"min":{"1min":0.216,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.42199999999999999,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.21099999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.34300000000000003,"15min":0.34799999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.48599999999999999,"5min":0.746,"15min":0.746},"last":0.247},{"interface":"front","average":{"1min":0.314,"5min":0.34100000000000003,"15min":0.34699999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.40000000000000002,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.32800000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.35599999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.49399999999999999,"15min":0.496},"min":{"1min":0.32400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.84899999999999998,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.28399999999999997}]},{"osd":4,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.47799999999999998,"15min":0.48499999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.77600000000000002,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.48399999999999999,"15min":0.49299999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.61099999999999999,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.307}]},{"osd":5,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.48099999999999998,"15min":0.495},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.59199999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.502,"15min":0.51200000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.38800000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640299,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4282456,"kb_used_data":4214032,"kb_used_omap":0,"kb_used_meta":68416,"kb_avail":90089384,"statfs":{"total":96636764160,"available":92251529216,"internally_reserved":0,"allocated":4315168768,"data_stored":5119285311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":70057984},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.20200000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.221},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.32200000000000001,"15min":0.33000000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.41099999999999998,"5min":0.496,"15min":0.496},"last":0.24399999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.33800000000000002,"15min":0.33200000000000002},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.222},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.36599999999999999,"15min":0.376},"min":{"1min":0.251,"5min":0.222,"15min":0.222},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.28499999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.497,"15min":0.501},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.624,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.498,"15min":0.499},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.64400000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.30199999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.48699999999999999,"15min":0.503},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.54300000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.498,"15min":0.504},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.38600000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51400000000000001,"15min":0.51500000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.5589999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.45100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1921400832,"data_stored":2287146599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4440342528,"data_stored":5258195111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2433634304,"data_stored":2791964242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2353426432,"data_stored":2742938894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1589133312,"data_stored":1843366858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:05:14.178 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:14.177+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:14.241 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:14.240+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:14.272 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:14.272 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:14.283 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:05:14.332 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:05:14.376 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:14.375+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:14.424 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:14.424 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:14.435 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:05:14.479 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:14.479+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:14.542 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:14.542+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:14.571 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:14.571 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:14.583 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:05:14.641 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:05:14.643 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:14.642+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:14.743 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:14.742+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:14.752 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:14.753 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:14.763 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:05:14.783 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:05:14.802 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:14.820 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:14.838 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:14.846 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:14.846+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:14.857 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:14.877 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:05:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:05:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:05:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:05:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:05:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:05:14.943 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:14.943 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:14.943 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:14.944 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:14.944 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:14.944+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:14.956 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:05:15.045 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:15.045+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:15.130 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:05:15.140 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:15.140 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:15.141 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:15.141 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:15.141 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:15.146 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:15.145+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:15.153 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:05:15.247 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:15.246+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:15.344 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:15.345 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:15.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:15.345 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:15.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.345 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:15.345 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:15.345 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.05178658299999, 2023-07-02T14:05:15.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:05:15.346 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:15.346 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:15.346 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:15.346 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:15.346 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:15.346 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:05:15.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.347 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:15.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:15.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:15.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:15.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:15.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:15.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:15.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:15.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:05:15.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:05:15.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:15.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:05:15.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:05:15.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:15.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:05:15.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:05:15.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:15.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:05:15.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:05:15.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:15.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:05:15.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:05:15.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:15.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:05:15.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:05:15.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:15.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:05:15.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:05:15.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:15.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:05:15.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:05:15.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:15.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:05:15.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:05:15.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:15.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:05:15.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:05:15.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:15.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:05:15.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:05:15.355 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.355 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.355 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.356 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:15.356 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:15.356 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.05155156000001, 2023-07-02T14:05:15.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:05:15.356 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:15.356 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:15.356 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:15.357 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:15.357 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:15.357 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:05:15.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.357 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:15.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:15.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:15.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:15.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:15.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:15.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:15.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:15.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:05:15.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:05:15.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:15.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:05:15.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:05:15.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:15.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:05:15.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:05:15.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:15.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:05:15.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:05:15.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:15.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:05:15.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:05:15.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:15.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:05:15.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:05:15.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:15.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:05:15.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:05:15.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:15.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:05:15.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:05:15.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:15.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:05:15.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:05:15.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:15.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:05:15.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:05:15.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:15.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:05:15.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:05:15.365 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.365 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.366 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.366 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:15.366 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:15.366 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.05135242599999, 2023-07-02T14:05:15.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:05:15.366 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:15.367 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:15.367 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:15.367 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:15.367 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:15.367 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:05:15.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.367 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:15.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:15.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:15.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:15.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:15.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:15.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:15.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:15.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:05:15.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:05:15.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:15.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:05:15.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:05:15.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:15.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:05:15.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:05:15.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:15.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:05:15.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:05:15.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:15.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:05:15.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:05:15.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:15.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:05:15.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:05:15.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:15.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:05:15.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:05:15.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:15.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:05:15.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:05:15.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:15.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:05:15.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:05:15.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:15.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:05:15.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:05:15.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:15.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:05:15.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:05:15.376 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.376 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.376 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.376 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:15.376 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:15.377 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.05098730200001, 2023-07-02T14:05:15.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:05:15.377 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:15.377 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:15.377 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:15.377 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:15.377 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:15.377 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:05:15.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.378 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:15.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:15.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:15.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:15.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:15.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:15.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:15.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:15.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:05:15.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:05:15.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:15.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:05:15.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:05:15.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:15.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:05:15.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:05:15.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:15.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:05:15.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:05:15.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:15.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:05:15.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:05:15.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:15.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:05:15.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:05:15.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:15.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:05:15.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:05:15.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:15.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:05:15.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:05:15.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:15.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:05:15.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:05:15.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:15.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:05:15.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:05:15.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:15.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:05:15.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:05:15.386 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.386 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.386 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.386 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:15.387 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:15.387 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.05090624100001, 2023-07-02T14:05:15.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:05:15.387 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:15.387 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:15.387 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:15.387 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:15.388 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:15.388 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:05:15.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.388 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:15.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:15.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:15.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:15.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:15.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:15.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:15.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:15.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:05:15.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:05:15.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:15.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:05:15.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:05:15.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:15.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:05:15.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:05:15.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:15.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:05:15.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:05:15.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:15.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:05:15.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:05:15.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:15.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:05:15.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:05:15.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:15.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:05:15.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:05:15.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:15.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:05:15.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:05:15.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:15.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:05:15.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:05:15.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:15.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:05:15.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:05:15.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:15.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:05:15.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:05:15.396 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.396 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.396 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.397 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:15.397 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:15.397 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.05081457, 2023-07-02T14:05:15.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:05:15.397 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:15.397 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:15.398 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:15.398 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:15.398 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:15.398 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:05:15.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.398 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:15.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:15.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:15.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:15.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:15.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:15.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:15.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:15.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:05:15.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:05:15.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:15.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:05:15.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:05:15.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:15.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:05:15.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:05:15.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:15.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:05:15.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:05:15.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:15.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:05:15.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:05:15.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:15.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:05:15.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:05:15.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.404 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:15.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:05:15.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:05:15.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.404 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:15.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:05:15.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:05:15.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:15.405 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:05:15.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:05:15.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:15.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:05:15.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:05:15.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:15.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:05:15.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:05:15.406 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.407 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.407 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.407 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:15.407 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:15.407 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.05047512799999, 2023-07-02T14:05:15.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:05:15.408 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:15.408 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:15.408 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:15.408 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:15.408 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:15.408 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:05:15.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.409 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:15.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:15.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:15.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:15.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:15.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:15.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:15.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:15.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:05:15.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:05:15.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:15.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:05:15.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:05:15.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:15.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:05:15.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:05:15.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:15.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:05:15.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:05:15.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:15.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:05:15.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:05:15.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:15.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:05:15.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:05:15.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:15.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:05:15.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:05:15.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:15.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:05:15.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:05:15.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:15.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:05:15.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:05:15.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:15.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:05:15.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:05:15.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:15.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:05:15.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:05:15.417 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.417 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.417 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.417 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:15.418 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:15.418 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.05028907499999, 2023-07-02T14:05:15.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:05:15.418 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:15.418 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:15.418 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:15.418 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:15.418 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:15.419 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:05:15.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.419 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:15.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:15.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:15.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:15.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:15.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:15.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:15.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:15.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:15.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:05:15.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:15.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:05:15.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:05:15.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:15.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:05:15.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:05:15.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:15.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:05:15.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:05:15.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:15.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:05:15.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:05:15.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:15.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:05:15.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:05:15.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:15.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:05:15.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:05:15.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:15.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:05:15.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:05:15.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:15.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:05:15.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:05:15.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:15.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:05:15.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:05:15.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:15.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:05:15.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:05:15.427 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.427 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.427 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.428 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:15.428 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:15.428 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.05013127300001, 2023-07-02T14:05:15.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:05:15.428 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:15.428 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:15.428 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:15.429 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:15.429 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:15.429 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:05:15.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.429 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:15.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:15.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:15.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:15.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:15.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:15.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:15.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:15.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:05:15.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:05:15.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:15.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:05:15.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:05:15.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:15.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:05:15.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:05:15.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:15.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:05:15.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:05:15.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:15.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:05:15.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:05:15.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:15.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:05:15.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:05:15.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:15.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:05:15.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:05:15.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:15.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:05:15.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:05:15.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:15.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:05:15.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:05:15.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:15.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:05:15.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:05:15.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:15.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:05:15.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:05:15.437 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.437 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.438 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.438 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:15.438 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:15.438 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.049939491, 2023-07-02T14:05:15.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:05:15.438 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:15.439 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:15.439 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:15.439 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:15.439 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:15.439 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:05:15.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.439 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:15.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:15.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:15.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:15.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:15.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:15.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:15.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:15.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:05:15.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:05:15.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:15.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:05:15.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:05:15.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:15.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:05:15.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:05:15.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:15.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:05:15.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:05:15.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:15.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:05:15.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:05:15.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:15.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:05:15.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:05:15.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:15.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:05:15.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:05:15.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:15.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:05:15.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:05:15.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:15.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:05:15.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:05:15.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:15.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:05:15.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:05:15.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:15.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:05:15.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:05:15.448 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.448 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.448 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.448 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:15.448 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:15.448 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.04976383100001, 2023-07-02T14:05:15.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:05:15.449 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:15.449 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:15.449 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:15.449 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:15.449 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:15.449 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:05:15.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.450 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:15.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:15.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:15.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:15.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:15.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:15.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:15.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:15.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:15.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:05:15.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:15.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:05:15.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:05:15.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:15.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:05:15.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:05:15.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:15.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:05:15.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:05:15.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:15.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:05:15.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:05:15.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:15.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:05:15.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:05:15.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:15.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:05:15.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:05:15.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:15.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:05:15.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:05:15.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:15.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:05:15.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:05:15.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:15.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:05:15.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:05:15.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:15.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:05:15.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:05:15.458 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.458 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.458 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.458 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:15.459 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:15.459 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.04966964299999, 2023-07-02T14:05:15.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:05:15.459 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:15.459 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:15.459 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:15.459 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:15.459 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:15.460 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:05:15.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.460 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:15.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:15.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:15.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:15.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:15.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:15.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:15.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:15.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:05:15.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:05:15.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:15.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:05:15.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:05:15.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:15.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:05:15.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:05:15.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:15.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:05:15.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:05:15.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:15.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:05:15.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:05:15.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:15.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:05:15.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:05:15.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:15.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:05:15.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:05:15.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:15.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:05:15.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:05:15.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:15.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:05:15.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:05:15.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:15.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:05:15.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:05:15.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:15.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:05:15.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:05:15.468 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.468 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.468 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.469 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:15.469 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:15.469 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.04946625700001, 2023-07-02T14:05:15.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:05:15.469 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:15.469 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:15.470 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:15.470 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:15.470 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:15.470 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:05:15.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.470 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:15.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:15.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:15.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:15.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:15.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:15.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:15.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:15.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:05:15.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:05:15.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:15.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:05:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:05:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:05:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:05:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:05:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:05:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:05:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:05:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:05:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:05:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:05:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:05:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:05:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:05:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:05:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:05:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:05:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:05:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:05:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:05:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.04931560099999, 2023-07-02T14:05:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:05:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:05:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:05:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:05:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:05:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:05:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:05:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:05:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:05:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:05:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:05:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:05:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:05:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:05:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:05:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:05:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:05:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:05:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:05:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:05:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:05:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:05:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:05:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:05:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.04924289799999, 2023-07-02T14:05:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:05:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:05:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:05:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:05:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:05:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:05:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:05:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:05:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:05:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:05:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:05:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:05:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:05:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:05:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:05:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:05:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:05:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:05:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:05:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:05:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:05:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:05:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:05:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.04912590800001, 2023-07-02T14:05:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:05:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:05:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:15.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:15.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:05:15.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:15.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:05:15.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:05:15.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:15.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:05:15.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:05:15.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:15.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:05:15.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:05:15.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:15.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:05:15.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:05:15.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:15.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:05:15.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:05:15.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:15.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:05:15.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:05:15.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:15.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:05:15.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:05:15.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:15.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:05:15.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:05:15.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:15.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:05:15.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:05:15.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:15.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:05:15.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:05:15.510 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.510 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.510 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.510 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:15.510 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:15.511 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.04897645700001, 2023-07-02T14:05:15.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:05:15.511 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:15.511 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:15.511 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:15.511 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:15.511 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:15.511 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:05:15.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.512 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:15.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:15.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:15.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:15.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:15.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:15.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:15.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:15.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:05:15.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:05:15.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:15.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:05:15.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:05:15.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:15.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:05:15.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:05:15.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:15.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:05:15.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:05:15.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:15.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:05:15.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:05:15.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:15.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:05:15.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:05:15.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:15.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:05:15.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:05:15.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:15.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:05:15.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:05:15.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:15.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:05:15.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:05:15.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:15.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:05:15.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:05:15.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:15.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:05:15.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:05:15.520 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.520 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.520 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.521 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:15.521 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:15.521 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.027500484, 2023-07-02T14:05:15.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:05:15.521 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:15.521 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:15.522 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:15.522 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:15.522 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:15.522 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:05:15.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.522 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:15.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:15.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:15.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:15.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:15.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:15.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:15.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:15.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:05:15.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:05:15.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:15.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:05:15.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:05:15.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:15.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:05:15.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:05:15.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:15.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:05:15.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:05:15.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:15.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:05:15.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:05:15.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:15.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:05:15.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:05:15.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:15.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:05:15.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:05:15.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:15.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:05:15.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:05:15.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:15.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:05:15.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:05:15.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:15.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:05:15.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:05:15.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:15.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:05:15.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:05:15.530 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.531 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.531 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.531 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:15.531 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:15.531 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.02736794800001, 2023-07-02T14:05:15.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:05:15.532 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:15.532 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:15.532 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:15.532 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:15.532 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:15.532 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:05:15.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.533 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:15.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:15.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:15.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:15.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:15.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:15.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:15.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:15.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:05:15.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:05:15.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:15.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:05:15.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:05:15.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:15.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:05:15.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:05:15.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:15.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:05:15.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:05:15.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:15.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:05:15.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:05:15.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:15.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:05:15.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:05:15.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:15.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:05:15.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:05:15.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:15.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:05:15.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:05:15.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:15.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:05:15.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:05:15.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:15.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:05:15.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:05:15.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:15.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:05:15.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:05:15.541 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.541 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.541 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.541 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:15.542 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:15.542 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 155.96603128699999, 2023-07-02T14:05:15.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:05:15.542 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:15.542 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:15.542 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:15.542 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:15.542 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:15.543 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:05:15.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.543 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:15.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:15.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:15.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:15.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:15.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:15.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:15.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:15.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:05:15.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:05:15.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:15.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:05:15.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:05:15.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:15.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:05:15.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:05:15.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:15.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:05:15.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:05:15.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:15.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:05:15.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:05:15.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:15.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:05:15.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:05:15.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:15.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:05:15.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:05:15.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:15.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:05:15.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:05:15.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:15.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:05:15.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:05:15.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:15.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:05:15.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:05:15.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:15.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:15.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:15.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:05:15.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:05:15.551 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.551 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.551 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.551 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:15.552 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:15.552 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:15.553 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:05:15.554 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:15.552+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:15.653 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:15.653+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:15.717 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:15.717 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:15.717 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:15.718 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:15.727 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:05:15.754 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:15.754+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:15.855 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:15.855+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:15.892 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:15.893 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:15.893 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:15.893 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:15.893 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:15.903 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:05:15.956 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:15.956+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:16.027 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:16.027 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:16.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:16.028 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:16.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.028 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.5 136/134 [PushOp(2:a5de3e1e:::rbd_data.125de1449605.0000000000000022:head, version: 125'1532, data_included: [3276800~917504], data_size: 917504, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:a5de3e1e:::rbd_data.125de1449605.0000000000000022:head@125'1532, size: 4194304, copy_subset: [131072~4063232], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3276800, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:05:16.028 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:07.478934+0000", 2023-07-02T14:05:16.028 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.5468958169999993, 2023-07-02T14:05:16.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051271370000000004, 2023-07-02T14:05:16.029 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:16.029 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:16.029 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:16.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:16.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:07.478934+0000", 2023-07-02T14:05:16.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:16.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:16.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:07.478934+0000", 2023-07-02T14:05:16.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:16.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:16.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:07.478934+0000", 2023-07-02T14:05:16.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00084429600000000002 2023-07-02T14:05:16.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:16.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:07.479778+0000", 2023-07-02T14:05:16.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4229999999999999e-05 2023-07-02T14:05:16.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:16.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:07.479792+0000", 2023-07-02T14:05:16.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.092e-06 2023-07-02T14:05:16.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:07.479799+0000", 2023-07-02T14:05:16.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6069999999999995e-05 2023-07-02T14:05:16.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:16.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:07.479875+0000", 2023-07-02T14:05:16.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1447000000000001e-05 2023-07-02T14:05:16.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:16.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:07.479887+0000", 2023-07-02T14:05:16.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041747319999999996 2023-07-02T14:05:16.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:16.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:07.484062+0000", 2023-07-02T14:05:16.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051271370000000004 2023-07-02T14:05:16.035 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.035 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.035 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.035 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.5 136/134 [PushOp(2:adf18789:::rbd_data.125de1449605.000000000000003c:head, version: 125'1559, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:adf18789:::rbd_data.125de1449605.000000000000003c:head@125'1559, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:05:16.035 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:07.645535+0000", 2023-07-02T14:05:16.035 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.3802949669999993, 2023-07-02T14:05:16.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049332029999999997, 2023-07-02T14:05:16.036 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:16.036 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:16.036 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:16.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:16.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:07.645535+0000", 2023-07-02T14:05:16.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:16.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:16.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:07.645535+0000", 2023-07-02T14:05:16.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:16.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:16.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:07.645534+0000", 2023-07-02T14:05:16.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00095168499999999997 2023-07-02T14:05:16.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:16.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:07.646486+0000", 2023-07-02T14:05:16.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5803000000000001e-05 2023-07-02T14:05:16.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:16.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:07.646502+0000", 2023-07-02T14:05:16.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1664999999999999e-05 2023-07-02T14:05:16.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:07.646514+0000", 2023-07-02T14:05:16.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9762e-05 2023-07-02T14:05:16.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:16.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:07.646593+0000", 2023-07-02T14:05:16.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0750999999999999e-05 2023-07-02T14:05:16.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:16.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:07.646604+0000", 2023-07-02T14:05:16.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003864363 2023-07-02T14:05:16.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:16.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:07.650469+0000", 2023-07-02T14:05:16.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049332029999999997 2023-07-02T14:05:16.042 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.042 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.042 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.042 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop_reply(client.4645.0:14084 2.5 e138/137 ondisk, result = 0)", 2023-07-02T14:05:16.042 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:10.108958+0000", 2023-07-02T14:05:16.043 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 5.9168719650000003, 2023-07-02T14:05:16.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0085271949999999996, 2023-07-02T14:05:16.043 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:16.043 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:16.043 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:16.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:16.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.108958+0000", 2023-07-02T14:05:16.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:16.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:16.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.108958+0000", 2023-07-02T14:05:16.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:16.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:16.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.108958+0000", 2023-07-02T14:05:16.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2310000000000001e-06 2023-07-02T14:05:16.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:16.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.108964+0000", 2023-07-02T14:05:16.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.17e-05 2023-07-02T14:05:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.108976+0000", 2023-07-02T14:05:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8040000000000002e-06 2023-07-02T14:05:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.108984+0000", 2023-07-02T14:05:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044563830000000004 2023-07-02T14:05:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.113440+0000", 2023-07-02T14:05:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7080000000000002e-06 2023-07-02T14:05:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.113446+0000", 2023-07-02T14:05:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040398209999999999 2023-07-02T14:05:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.117486+0000", 2023-07-02T14:05:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0085271949999999996 2023-07-02T14:05:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.6 138/134 [PushOp(2:71e595b4:::rbd_data.125de1449605.000000000000006a:head, version: 125'1480, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:71e595b4:::rbd_data.125de1449605.000000000000006a:head@125'1480, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:05:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:10.260427+0000", 2023-07-02T14:05:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 5.765402913, 2023-07-02T14:05:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046923210000000002, 2023-07-02T14:05:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.260427+0000", 2023-07-02T14:05:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.260427+0000", 2023-07-02T14:05:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.260427+0000", 2023-07-02T14:05:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00099835799999999997 2023-07-02T14:05:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.261425+0000", 2023-07-02T14:05:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7122999999999999e-05 2023-07-02T14:05:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.261442+0000", 2023-07-02T14:05:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7237000000000001e-05 2023-07-02T14:05:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.261459+0000", 2023-07-02T14:05:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013541639999999999 2023-07-02T14:05:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.262814+0000", 2023-07-02T14:05:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8590000000000004e-06 2023-07-02T14:05:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.262819+0000", 2023-07-02T14:05:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023011820000000001 2023-07-02T14:05:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.265120+0000", 2023-07-02T14:05:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046923210000000002 2023-07-02T14:05:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.6 138/134 [PushOp(2:73c51af9:::rbd_data.125de1449605.000000000000006d:head, version: 125'1482, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:73c51af9:::rbd_data.125de1449605.000000000000006d:head@125'1482, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:05:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:10.280284+0000", 2023-07-02T14:05:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 5.7455466299999998, 2023-07-02T14:05:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048528470000000004, 2023-07-02T14:05:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.280284+0000", 2023-07-02T14:05:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.280284+0000", 2023-07-02T14:05:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.280283+0000", 2023-07-02T14:05:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000966168 2023-07-02T14:05:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.281249+0000", 2023-07-02T14:05:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7489000000000001e-05 2023-07-02T14:05:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.281267+0000", 2023-07-02T14:05:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6330000000000008e-06 2023-07-02T14:05:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.281276+0000", 2023-07-02T14:05:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0015079710000000001 2023-07-02T14:05:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.282784+0000", 2023-07-02T14:05:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8949999999999993e-06 2023-07-02T14:05:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.282794+0000", 2023-07-02T14:05:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023423179999999999 2023-07-02T14:05:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.285137+0000", 2023-07-02T14:05:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048528470000000004 2023-07-02T14:05:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.6 138/134 [PushOp(2:7f0997df:::rbd_data.125de1449605.000000000000007b:head, version: 125'1490, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7f0997df:::rbd_data.125de1449605.000000000000007b:head@125'1490, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:05:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:10.318087+0000", 2023-07-02T14:05:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 5.7077436080000004, 2023-07-02T14:05:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047368769999999996, 2023-07-02T14:05:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.318087+0000", 2023-07-02T14:05:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.318087+0000", 2023-07-02T14:05:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.318086+0000", 2023-07-02T14:05:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00082252599999999996 2023-07-02T14:05:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.318909+0000", 2023-07-02T14:05:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1430000000000001e-05 2023-07-02T14:05:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.318920+0000", 2023-07-02T14:05:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.939e-06 2023-07-02T14:05:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.318927+0000", 2023-07-02T14:05:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0015256639999999999 2023-07-02T14:05:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.320453+0000", 2023-07-02T14:05:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9799999999999998e-06 2023-07-02T14:05:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.320458+0000", 2023-07-02T14:05:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023656829999999999 2023-07-02T14:05:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:10.322824+0000", 2023-07-02T14:05:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047368769999999996 2023-07-02T14:05:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.7 139/134 [PushOp(2:e3782f39:::rbd_data.125de1449605.0000000000000120:head, version: 119'1783, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:e3782f39:::rbd_data.125de1449605.0000000000000120:head@119'1783, size: 4194304, copy_subset: [1048576~3145728], clone_subset: {2:e3782f39:::rbd_data.125de1449605.0000000000000120:1e=[135168~913408]}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:05:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:11.997451+0000", 2023-07-02T14:05:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.0283790550000003, 2023-07-02T14:05:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052925849999999998, 2023-07-02T14:05:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:11.997451+0000", 2023-07-02T14:05:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:11.997451+0000", 2023-07-02T14:05:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999948 2023-07-02T14:05:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:11.997446+0000", 2023-07-02T14:05:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00092316000000000002 2023-07-02T14:05:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:11.998369+0000", 2023-07-02T14:05:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8343e-05 2023-07-02T14:05:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:11.998398+0000", 2023-07-02T14:05:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1489e-05 2023-07-02T14:05:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:11.998409+0000", 2023-07-02T14:05:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0458999999999996e-05 2023-07-02T14:05:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:11.998500+0000", 2023-07-02T14:05:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1858000000000001e-05 2023-07-02T14:05:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:11.998511+0000", 2023-07-02T14:05:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042323710000000004 2023-07-02T14:05:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.002744+0000", 2023-07-02T14:05:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052925849999999998 2023-07-02T14:05:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.7 139/134 [PushOp(2:e20dd252:::rbd_data.125de1449605.0000000000000121:head, version: 119'1784, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:e20dd252:::rbd_data.125de1449605.0000000000000121:head@119'1784, size: 4194304, copy_subset: [1048576~3145728], clone_subset: {2:e20dd252:::rbd_data.125de1449605.0000000000000121:1e=[131072~917504]}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:05:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:12.013286+0000", 2023-07-02T14:05:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.0125438679999998, 2023-07-02T14:05:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005273242, 2023-07-02T14:05:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.013286+0000", 2023-07-02T14:05:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.013286+0000", 2023-07-02T14:05:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.013286+0000", 2023-07-02T14:05:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00097144099999999997 2023-07-02T14:05:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.014257+0000", 2023-07-02T14:05:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4926000000000001e-05 2023-07-02T14:05:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.014272+0000", 2023-07-02T14:05:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2069999999999998e-06 2023-07-02T14:05:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.014279+0000", 2023-07-02T14:05:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6392999999999999e-05 2023-07-02T14:05:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.014356+0000", 2023-07-02T14:05:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9420000000000001e-06 2023-07-02T14:05:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.014365+0000", 2023-07-02T14:05:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041951530000000001 2023-07-02T14:05:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.018560+0000", 2023-07-02T14:05:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005273242 2023-07-02T14:05:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.7 139/134 [PushOp(2:e9a9fc91:::rbd_data.125de1449605.000000000000012c:head, version: 119'1792, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:e9a9fc91:::rbd_data.125de1449605.000000000000012c:head@119'1792, size: 4194304, copy_subset: [0~2097152,3145728~1048576], clone_subset: {2:e9a9fc91:::rbd_data.125de1449605.000000000000012c:1e=[2359296~786432]}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:05:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:12.063579+0000", 2023-07-02T14:05:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.9622509379999999, 2023-07-02T14:05:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054666009999999998, 2023-07-02T14:05:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.063579+0000", 2023-07-02T14:05:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.063579+0000", 2023-07-02T14:05:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.063578+0000", 2023-07-02T14:05:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00093561800000000004 2023-07-02T14:05:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.064514+0000", 2023-07-02T14:05:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7289e-05 2023-07-02T14:05:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.064531+0000", 2023-07-02T14:05:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3429999999999996e-06 2023-07-02T14:05:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.064540+0000", 2023-07-02T14:05:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1858e-05 2023-07-02T14:05:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.064632+0000", 2023-07-02T14:05:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2362e-05 2023-07-02T14:05:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:16.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.064644+0000", 2023-07-02T14:05:16.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044020530000000004 2023-07-02T14:05:16.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:16.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.069046+0000", 2023-07-02T14:05:16.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054666009999999998 2023-07-02T14:05:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.7 139/134 [PushOp(2:e57d3f55:::rbd_data.125de1449605.0000000000000134:head, version: 120'1801, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:e57d3f55:::rbd_data.125de1449605.0000000000000134:head@120'1801, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:4194304, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:05:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:12.099450+0000", 2023-07-02T14:05:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.9263798950000002, 2023-07-02T14:05:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053180579999999996, 2023-07-02T14:05:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.099450+0000", 2023-07-02T14:05:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.099450+0000", 2023-07-02T14:05:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.099450+0000", 2023-07-02T14:05:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00093281899999999999 2023-07-02T14:05:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.100382+0000", 2023-07-02T14:05:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5545000000000001e-05 2023-07-02T14:05:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.100398+0000", 2023-07-02T14:05:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6149999999999999e-06 2023-07-02T14:05:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.100406+0000", 2023-07-02T14:05:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5981000000000003e-05 2023-07-02T14:05:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.100482+0000", 2023-07-02T14:05:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8810000000000008e-06 2023-07-02T14:05:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.100492+0000", 2023-07-02T14:05:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042769369999999998 2023-07-02T14:05:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:12.104768+0000", 2023-07-02T14:05:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053180579999999996 2023-07-02T14:05:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:14225 2.6 e142/139 2:7013ec14:::rbd_data.125de1449605.0000000000000162:head v 142'1626, mlcod=142'1624)", 2023-07-02T14:05:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:15.077227+0000", 2023-07-02T14:05:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.94860354199999997, 2023-07-02T14:05:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012657399, 2023-07-02T14:05:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.077227+0000", 2023-07-02T14:05:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.077227+0000", 2023-07-02T14:05:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.077226+0000", 2023-07-02T14:05:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00560583 2023-07-02T14:05:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.082832+0000", 2023-07-02T14:05:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2310000000000002e-05 2023-07-02T14:05:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.082854+0000", 2023-07-02T14:05:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0416e-05 2023-07-02T14:05:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.082865+0000", 2023-07-02T14:05:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7460000000000003e-05 2023-07-02T14:05:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.082912+0000", 2023-07-02T14:05:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2274999999999999e-05 2023-07-02T14:05:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.082934+0000", 2023-07-02T14:05:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069184199999999998 2023-07-02T14:05:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.089853+0000", 2023-07-02T14:05:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1236999999999999e-05 2023-07-02T14:05:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.089884+0000", 2023-07-02T14:05:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012657399 2023-07-02T14:05:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:14228 2.7 2:fe1e2477:::rbd_data.125de1449605.0000000000000161:head [write 65536~3145728] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e142)", 2023-07-02T14:05:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:15.078543+0000", 2023-07-02T14:05:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.94728714899999999, 2023-07-02T14:05:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015849022000000001, 2023-07-02T14:05:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 14228 2023-07-02T14:05:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.078543+0000", 2023-07-02T14:05:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.078543+0000", 2023-07-02T14:05:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.078542+0000", 2023-07-02T14:05:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00446722 2023-07-02T14:05:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.083009+0000", 2023-07-02T14:05:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3627e-05 2023-07-02T14:05:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.083023+0000", 2023-07-02T14:05:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0939999999999996e-06 2023-07-02T14:05:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.083029+0000", 2023-07-02T14:05:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5158999999999998e-05 2023-07-02T14:05:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.083084+0000", 2023-07-02T14:05:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012269399999999999 2023-07-02T14:05:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.083207+0000", 2023-07-02T14:05:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019189700000000001 2023-07-02T14:05:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.083399+0000", 2023-07-02T14:05:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069438989999999999 2023-07-02T14:05:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.090343+0000", 2023-07-02T14:05:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039087230000000002 2023-07-02T14:05:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.094251+0000", 2023-07-02T14:05:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0945000000000005e-05 2023-07-02T14:05:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:16.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.094332+0000", 2023-07-02T14:05:16.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9976000000000003e-05 2023-07-02T14:05:16.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:16.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.094392+0000", 2023-07-02T14:05:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015849022000000001 2023-07-02T14:05:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.0 142/134 [PushOp(2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head, version: 125'3204, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head@125'3204, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-07-02T14:05:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:15.082883+0000", 2023-07-02T14:05:16.118 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.94294720399999998, 2023-07-02T14:05:16.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065801159999999996, 2023-07-02T14:05:16.118 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:16.118 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:16.118 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:16.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:16.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.082883+0000", 2023-07-02T14:05:16.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:16.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:16.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.082883+0000", 2023-07-02T14:05:16.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:16.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:16.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.082882+0000", 2023-07-02T14:05:16.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032597220000000001 2023-07-02T14:05:16.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:16.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.086142+0000", 2023-07-02T14:05:16.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9737000000000001e-05 2023-07-02T14:05:16.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:16.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.086161+0000", 2023-07-02T14:05:16.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0253e-05 2023-07-02T14:05:16.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.086172+0000", 2023-07-02T14:05:16.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8449999999999996e-05 2023-07-02T14:05:16.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:16.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.086250+0000", 2023-07-02T14:05:16.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0508e-05 2023-07-02T14:05:16.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:16.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.086261+0000", 2023-07-02T14:05:16.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032025349999999998 2023-07-02T14:05:16.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:16.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.089463+0000", 2023-07-02T14:05:16.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065801159999999996 2023-07-02T14:05:16.124 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.124 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.124 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.124 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:14217 2.0 e142/134 2:1c72c23d:::rbd_object_map.125de1449605:head v 142'3481, mlcod=125'3203)", 2023-07-02T14:05:16.125 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:15.086188+0000", 2023-07-02T14:05:16.125 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.93964254400000002, 2023-07-02T14:05:16.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048041270000000001, 2023-07-02T14:05:16.125 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:16.125 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.125 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:16.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.086188+0000", 2023-07-02T14:05:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.086188+0000", 2023-07-02T14:05:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.086187+0000", 2023-07-02T14:05:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5969999999999999e-06 2023-07-02T14:05:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:16.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.086192+0000", 2023-07-02T14:05:16.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0339999999999998e-06 2023-07-02T14:05:16.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:16.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.086196+0000", 2023-07-02T14:05:16.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6869999999999999e-06 2023-07-02T14:05:16.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.086199+0000", 2023-07-02T14:05:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003316982 2023-07-02T14:05:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.089516+0000", 2023-07-02T14:05:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3220000000000001e-05 2023-07-02T14:05:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.089539+0000", 2023-07-02T14:05:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014266400000000001 2023-07-02T14:05:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.090966+0000", 2023-07-02T14:05:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5871999999999999e-05 2023-07-02T14:05:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.090992+0000", 2023-07-02T14:05:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048041270000000001 2023-07-02T14:05:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:14230 2.7 2:fe1e2477:::rbd_data.125de1449605.0000000000000161:head [write 3211264~65536] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e142)", 2023-07-02T14:05:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:15.086314+0000", 2023-07-02T14:05:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.93951626899999996, 2023-07-02T14:05:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0082246090000000008, 2023-07-02T14:05:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 14230 2023-07-02T14:05:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.086314+0000", 2023-07-02T14:05:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.086314+0000", 2023-07-02T14:05:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.086312+0000", 2023-07-02T14:05:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011656 2023-07-02T14:05:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.086429+0000", 2023-07-02T14:05:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1260999999999999e-05 2023-07-02T14:05:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.086440+0000", 2023-07-02T14:05:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2649999999999998e-06 2023-07-02T14:05:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.086444+0000", 2023-07-02T14:05:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00085340500000000001 2023-07-02T14:05:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.087298+0000", 2023-07-02T14:05:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0252000000000004e-05 2023-07-02T14:05:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.087368+0000", 2023-07-02T14:05:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133299 2023-07-02T14:05:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.087501+0000", 2023-07-02T14:05:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002977726 2023-07-02T14:05:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.090479+0000", 2023-07-02T14:05:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039597670000000003 2023-07-02T14:05:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.094439+0000", 2023-07-02T14:05:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7043000000000003e-05 2023-07-02T14:05:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.094496+0000", 2023-07-02T14:05:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2638000000000001e-05 2023-07-02T14:05:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.094539+0000", 2023-07-02T14:05:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0082246090000000008 2023-07-02T14:05:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:14240 2.7 2:fe1e2477:::rbd_data.125de1449605.0000000000000161:head [write 3276800~917504] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e142)", 2023-07-02T14:05:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:15.094808+0000", 2023-07-02T14:05:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.93102228300000001, 2023-07-02T14:05:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099305820000000003, 2023-07-02T14:05:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 14240 2023-07-02T14:05:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.094808+0000", 2023-07-02T14:05:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:16.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:16.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.094808+0000", 2023-07-02T14:05:16.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:16.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.094807+0000", 2023-07-02T14:05:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001158538 2023-07-02T14:05:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.095965+0000", 2023-07-02T14:05:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3553000000000001e-05 2023-07-02T14:05:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.095979+0000", 2023-07-02T14:05:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8529999999999998e-06 2023-07-02T14:05:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.095988+0000", 2023-07-02T14:05:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7342999999999997e-05 2023-07-02T14:05:16.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:16.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.096035+0000", 2023-07-02T14:05:16.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000120455 2023-07-02T14:05:16.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.096156+0000", 2023-07-02T14:05:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00018319300000000001 2023-07-02T14:05:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.096339+0000", 2023-07-02T14:05:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023082530000000001 2023-07-02T14:05:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.098647+0000", 2023-07-02T14:05:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059775189999999997 2023-07-02T14:05:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.104625+0000", 2023-07-02T14:05:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7520000000000004e-05 2023-07-02T14:05:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.104692+0000", 2023-07-02T14:05:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6486000000000002e-05 2023-07-02T14:05:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.104739+0000", 2023-07-02T14:05:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099305820000000003 2023-07-02T14:05:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:14250 2.7 2:e90f6b02:::rbd_data.125de1449605.0000000000000166:head [write 65536~4128768] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e142)", 2023-07-02T14:05:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:15.107073+0000", 2023-07-02T14:05:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.91875711699999996, 2023-07-02T14:05:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016793801000000001, 2023-07-02T14:05:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 14250 2023-07-02T14:05:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.107073+0000", 2023-07-02T14:05:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.107073+0000", 2023-07-02T14:05:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.107072+0000", 2023-07-02T14:05:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050371499999999998 2023-07-02T14:05:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.112109+0000", 2023-07-02T14:05:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3864e-05 2023-07-02T14:05:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.112123+0000", 2023-07-02T14:05:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0278999999999999e-05 2023-07-02T14:05:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.112133+0000", 2023-07-02T14:05:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5973000000000001e-05 2023-07-02T14:05:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.112189+0000", 2023-07-02T14:05:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011774500000000001 2023-07-02T14:05:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.112307+0000", 2023-07-02T14:05:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020514799999999999 2023-07-02T14:05:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.112512+0000", 2023-07-02T14:05:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064637169999999999 2023-07-02T14:05:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.118976+0000", 2023-07-02T14:05:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047710929999999997 2023-07-02T14:05:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.123747+0000", 2023-07-02T14:05:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0846999999999994e-05 2023-07-02T14:05:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.123817+0000", 2023-07-02T14:05:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9518000000000003e-05 2023-07-02T14:05:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.123867+0000", 2023-07-02T14:05:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016793801000000001 2023-07-02T14:05:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:14254 2.5 2:ae059f7e:::rbd_data.125de1449605.0000000000000167:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~65536] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e142)", 2023-07-02T14:05:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:15.120212+0000", 2023-07-02T14:05:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.90561829000000005, 2023-07-02T14:05:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065608569999999998, 2023-07-02T14:05:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 14254 2023-07-02T14:05:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.120212+0000", 2023-07-02T14:05:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.120212+0000", 2023-07-02T14:05:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999962 2023-07-02T14:05:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.120208+0000", 2023-07-02T14:05:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000102321 2023-07-02T14:05:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.120310+0000", 2023-07-02T14:05:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4397e-05 2023-07-02T14:05:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.120325+0000", 2023-07-02T14:05:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6210000000000004e-06 2023-07-02T14:05:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.120332+0000", 2023-07-02T14:05:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8649000000000002e-05 2023-07-02T14:05:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.120371+0000", 2023-07-02T14:05:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000108558 2023-07-02T14:05:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.120480+0000", 2023-07-02T14:05:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0001114 2023-07-02T14:05:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.120591+0000", 2023-07-02T14:05:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0016044099999999999 2023-07-02T14:05:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.122195+0000", 2023-07-02T14:05:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044737400000000004 2023-07-02T14:05:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.126669+0000", 2023-07-02T14:05:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3516999999999995e-05 2023-07-02T14:05:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.126733+0000", 2023-07-02T14:05:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0138000000000001e-05 2023-07-02T14:05:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.126773+0000", 2023-07-02T14:05:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065608569999999998 2023-07-02T14:05:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:14257 2.5 2:ae059f7e:::rbd_data.125de1449605.0000000000000167:head [write 65536~4128768] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e143)", 2023-07-02T14:05:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:15.127465+0000", 2023-07-02T14:05:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.89836542900000005, 2023-07-02T14:05:16.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019550754, 2023-07-02T14:05:16.174 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:16.174 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.174 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:16.174 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:16.174 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.174 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 14257 2023-07-02T14:05:16.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.175 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:16.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:16.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.127465+0000", 2023-07-02T14:05:16.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:16.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:16.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.127465+0000", 2023-07-02T14:05:16.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:16.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:16.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.127463+0000", 2023-07-02T14:05:16.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046727089999999997 2023-07-02T14:05:16.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:16.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.132136+0000", 2023-07-02T14:05:16.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5492999999999999e-05 2023-07-02T14:05:16.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:16.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.132152+0000", 2023-07-02T14:05:16.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9029999999999993e-06 2023-07-02T14:05:16.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.132160+0000", 2023-07-02T14:05:16.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5988e-05 2023-07-02T14:05:16.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:16.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.132216+0000", 2023-07-02T14:05:16.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012380400000000001 2023-07-02T14:05:16.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:16.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.132340+0000", 2023-07-02T14:05:16.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000205157 2023-07-02T14:05:16.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:16.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.132545+0000", 2023-07-02T14:05:16.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065451229999999999 2023-07-02T14:05:16.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:16.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.139090+0000", 2023-07-02T14:05:16.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078566690000000002 2023-07-02T14:05:16.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.146947+0000", 2023-07-02T14:05:16.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0538999999999999e-05 2023-07-02T14:05:16.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:16.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.146988+0000", 2023-07-02T14:05:16.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.7925000000000001e-05 2023-07-02T14:05:16.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:16.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.147016+0000", 2023-07-02T14:05:16.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019550754 2023-07-02T14:05:16.183 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.183 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.183 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.183 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:14255 2.0 e143/134 2:045dc4d7:::rbd_data.125de1449605.0000000000000165:head v 143'3488, mlcod=125'3208)", 2023-07-02T14:05:16.183 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:15.131173+0000", 2023-07-02T14:05:16.184 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.89465747500000004, 2023-07-02T14:05:16.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012679177, 2023-07-02T14:05:16.184 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:16.184 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.184 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:16.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:16.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.131173+0000", 2023-07-02T14:05:16.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:16.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:16.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.131173+0000", 2023-07-02T14:05:16.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:16.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:16.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.131172+0000", 2023-07-02T14:05:16.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051576909999999998 2023-07-02T14:05:16.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:16.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.136330+0000", 2023-07-02T14:05:16.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6489000000000001e-05 2023-07-02T14:05:16.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:16.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.136346+0000", 2023-07-02T14:05:16.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1389999999999995e-06 2023-07-02T14:05:16.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.136355+0000", 2023-07-02T14:05:16.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7211000000000004e-05 2023-07-02T14:05:16.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:16.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.136422+0000", 2023-07-02T14:05:16.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3662000000000001e-05 2023-07-02T14:05:16.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:16.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.136446+0000", 2023-07-02T14:05:16.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073682820000000003 2023-07-02T14:05:16.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:16.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.143814+0000", 2023-07-02T14:05:16.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8201999999999998e-05 2023-07-02T14:05:16.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:16.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:16.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:16.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.143852+0000", 2023-07-02T14:05:16.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012679177 2023-07-02T14:05:16.190 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.191 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.191 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.191 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:16.191 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:16.191 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:16.192 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:16.192+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:16.193 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:05:16.293 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:16.292+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:16.323 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:16.324 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:16.324 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:05:16.324 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:16.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:05:16.394 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:16.393+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:16.495 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:16.495+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:16.498 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:16.498 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:16.498 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:05:16.498 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:05:16.498 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:16.508 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:05:16.596 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:16.595+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:16.633 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:16.633 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:05:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:05:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2620 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 196608~786432] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:05:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.306955839, 2023-07-02T14:05:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966, 2023-07-02T14:05:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2620 2023-07-02T14:05:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:05:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:05:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324571+0000", 2023-07-02T14:05:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016686649999999999 2023-07-02T14:05:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326240+0000", 2023-07-02T14:05:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1844e-05 2023-07-02T14:05:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326252+0000", 2023-07-02T14:05:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7760000000000003e-06 2023-07-02T14:05:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326258+0000", 2023-07-02T14:05:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0634000000000003e-05 2023-07-02T14:05:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326318+0000", 2023-07-02T14:05:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9629999999999993e-06 2023-07-02T14:05:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326328+0000", 2023-07-02T14:05:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.010877045 2023-07-02T14:05:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337205+0000", 2023-07-02T14:05:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022401300000000001 2023-07-02T14:05:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337429+0000", 2023-07-02T14:05:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000139822 2023-07-02T14:05:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337569+0000", 2023-07-02T14:05:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032942119999999999 2023-07-02T14:05:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340863+0000", 2023-07-02T14:05:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021302560000000001 2023-07-02T14:05:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342994+0000", 2023-07-02T14:05:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5634000000000002e-05 2023-07-02T14:05:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343059+0000", 2023-07-02T14:05:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143721 2023-07-02T14:05:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343203+0000", 2023-07-02T14:05:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966 2023-07-02T14:05:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2622 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 983040~36864] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:05:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.305263328, 2023-07-02T14:05:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664, 2023-07-02T14:05:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2622 2023-07-02T14:05:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:05:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:05:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326265+0000", 2023-07-02T14:05:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1282000000000001e-05 2023-07-02T14:05:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326286+0000", 2023-07-02T14:05:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9480000000000002e-06 2023-07-02T14:05:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326293+0000", 2023-07-02T14:05:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3610000000000003e-06 2023-07-02T14:05:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326297+0000", 2023-07-02T14:05:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7707999999999994e-05 2023-07-02T14:05:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326385+0000", 2023-07-02T14:05:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8070000000000006e-06 2023-07-02T14:05:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326394+0000", 2023-07-02T14:05:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.014535878 2023-07-02T14:05:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340930+0000", 2023-07-02T14:05:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000154551 2023-07-02T14:05:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341084+0000", 2023-07-02T14:05:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00032375900000000002 2023-07-02T14:05:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341408+0000", 2023-07-02T14:05:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012631066 2023-07-02T14:05:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354039+0000", 2023-07-02T14:05:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5741000000000001e-05 2023-07-02T14:05:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354055+0000", 2023-07-02T14:05:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8794000000000006e-05 2023-07-02T14:05:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354124+0000", 2023-07-02T14:05:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014667700000000001 2023-07-02T14:05:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354270+0000", 2023-07-02T14:05:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664 2023-07-02T14:05:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2727 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~12288] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:05:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.30522755299999, 2023-07-02T14:05:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999, 2023-07-02T14:05:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2727 2023-07-02T14:05:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:05:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:05:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:05:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3834e-05 2023-07-02T14:05:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326315+0000", 2023-07-02T14:05:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3029999999999996e-06 2023-07-02T14:05:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326321+0000", 2023-07-02T14:05:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7590000000000002e-06 2023-07-02T14:05:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326324+0000", 2023-07-02T14:05:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112872 2023-07-02T14:05:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326437+0000", 2023-07-02T14:05:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2290000000000002e-06 2023-07-02T14:05:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326442+0000", 2023-07-02T14:05:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0155896879999999 2023-07-02T14:05:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342031+0000", 2023-07-02T14:05:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016496100000000001 2023-07-02T14:05:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342196+0000", 2023-07-02T14:05:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021998699999999999 2023-07-02T14:05:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342416+0000", 2023-07-02T14:05:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011869744999999999 2023-07-02T14:05:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354286+0000", 2023-07-02T14:05:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0361e-05 2023-07-02T14:05:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354347+0000", 2023-07-02T14:05:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3312000000000005e-05 2023-07-02T14:05:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354430+0000", 2023-07-02T14:05:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126444 2023-07-02T14:05:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354556+0000", 2023-07-02T14:05:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999 2023-07-02T14:05:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2730 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [set-alloc-hint object_size 4194304 write_size 4194304,write 454656~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:05:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.305202544, 2023-07-02T14:05:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999, 2023-07-02T14:05:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2730 2023-07-02T14:05:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:05:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:05:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:05:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6049999999999997e-06 2023-07-02T14:05:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326332+0000", 2023-07-02T14:05:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3000000000000002e-06 2023-07-02T14:05:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326335+0000", 2023-07-02T14:05:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.719e-05 2023-07-02T14:05:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326352+0000", 2023-07-02T14:05:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011583599999999999 2023-07-02T14:05:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326468+0000", 2023-07-02T14:05:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.831e-06 2023-07-02T14:05:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326473+0000", 2023-07-02T14:05:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.016769915 2023-07-02T14:05:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343243+0000", 2023-07-02T14:05:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197352 2023-07-02T14:05:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343440+0000", 2023-07-02T14:05:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197581 2023-07-02T14:05:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343638+0000", 2023-07-02T14:05:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010980139 2023-07-02T14:05:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354618+0000", 2023-07-02T14:05:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056779530000000003 2023-07-02T14:05:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360296+0000", 2023-07-02T14:05:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101109 2023-07-02T14:05:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360397+0000", 2023-07-02T14:05:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000160273 2023-07-02T14:05:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360557+0000", 2023-07-02T14:05:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999 2023-07-02T14:05:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2752 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 1019904~3174400] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:05:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.30515631999999, 2023-07-02T14:05:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001, 2023-07-02T14:05:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2752 2023-07-02T14:05:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:05:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:05:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326372+0000", 2023-07-02T14:05:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052050380000000004 2023-07-02T14:05:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331577+0000", 2023-07-02T14:05:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6832000000000001e-05 2023-07-02T14:05:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331594+0000", 2023-07-02T14:05:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8220000000000007e-06 2023-07-02T14:05:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331602+0000", 2023-07-02T14:05:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9895999999999999e-05 2023-07-02T14:05:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331642+0000", 2023-07-02T14:05:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:05:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331649+0000", 2023-07-02T14:05:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0139141709999999 2023-07-02T14:05:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345563+0000", 2023-07-02T14:05:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113817 2023-07-02T14:05:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345677+0000", 2023-07-02T14:05:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018803400000000001 2023-07-02T14:05:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345865+0000", 2023-07-02T14:05:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087950359999999991 2023-07-02T14:05:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354660+0000", 2023-07-02T14:05:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059116669999999998 2023-07-02T14:05:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360572+0000", 2023-07-02T14:05:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0992000000000002e-05 2023-07-02T14:05:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360633+0000", 2023-07-02T14:05:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013601600000000001 2023-07-02T14:05:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360769+0000", 2023-07-02T14:05:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001 2023-07-02T14:05:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2759 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 0~454656] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:05:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.29991652299999, 2023-07-02T14:05:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999, 2023-07-02T14:05:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2759 2023-07-02T14:05:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:05:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:05:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331611+0000", 2023-07-02T14:05:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053795999999999998 2023-07-02T14:05:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332149+0000", 2023-07-02T14:05:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4970000000000007e-06 2023-07-02T14:05:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332158+0000", 2023-07-02T14:05:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:05:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332162+0000", 2023-07-02T14:05:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3526000000000001e-05 2023-07-02T14:05:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332185+0000", 2023-07-02T14:05:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:05:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332189+0000", 2023-07-02T14:05:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0186295910000001 2023-07-02T14:05:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350819+0000", 2023-07-02T14:05:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000111123 2023-07-02T14:05:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350930+0000", 2023-07-02T14:05:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019575300000000001 2023-07-02T14:05:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.351126+0000", 2023-07-02T14:05:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011503298 2023-07-02T14:05:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362629+0000", 2023-07-02T14:05:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085443989999999994 2023-07-02T14:05:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371173+0000", 2023-07-02T14:05:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3547999999999995e-05 2023-07-02T14:05:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371237+0000", 2023-07-02T14:05:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790300000000001 2023-07-02T14:05:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371375+0000", 2023-07-02T14:05:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999 2023-07-02T14:05:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2760 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 458752~131072] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:05:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.299360338, 2023-07-02T14:05:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001, 2023-07-02T14:05:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2760 2023-07-02T14:05:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:05:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:05:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332168+0000", 2023-07-02T14:05:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013281100000000001 2023-07-02T14:05:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332301+0000", 2023-07-02T14:05:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7389999999999999e-06 2023-07-02T14:05:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332308+0000", 2023-07-02T14:05:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0639999999999998e-06 2023-07-02T14:05:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332312+0000", 2023-07-02T14:05:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1735000000000001e-05 2023-07-02T14:05:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332333+0000", 2023-07-02T14:05:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8342000000000002e-05 2023-07-02T14:05:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332352+0000", 2023-07-02T14:05:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02234782 2023-07-02T14:05:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354699+0000", 2023-07-02T14:05:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110371 2023-07-02T14:05:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354810+0000", 2023-07-02T14:05:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022174900000000001 2023-07-02T14:05:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355032+0000", 2023-07-02T14:05:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076053329999999997 2023-07-02T14:05:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362637+0000", 2023-07-02T14:05:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087913609999999993 2023-07-02T14:05:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371428+0000", 2023-07-02T14:05:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8532999999999998e-05 2023-07-02T14:05:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371477+0000", 2023-07-02T14:05:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010990800000000001 2023-07-02T14:05:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371587+0000", 2023-07-02T14:05:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001 2023-07-02T14:05:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2766 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 589824~45056] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:05:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.29921171300001, 2023-07-02T14:05:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188, 2023-07-02T14:05:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2766 2023-07-02T14:05:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:05:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:05:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:05:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2247999999999998e-05 2023-07-02T14:05:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332369+0000", 2023-07-02T14:05:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4189999999999996e-06 2023-07-02T14:05:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332376+0000", 2023-07-02T14:05:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1049999999999999e-06 2023-07-02T14:05:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332379+0000", 2023-07-02T14:05:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6894000000000003e-05 2023-07-02T14:05:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332416+0000", 2023-07-02T14:05:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3970000000000002e-06 2023-07-02T14:05:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332422+0000", 2023-07-02T14:05:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0231382280000001 2023-07-02T14:05:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355560+0000", 2023-07-02T14:05:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9079000000000005e-05 2023-07-02T14:05:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355659+0000", 2023-07-02T14:05:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018005000000000001 2023-07-02T14:05:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355839+0000", 2023-07-02T14:05:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068044439999999998 2023-07-02T14:05:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362644+0000", 2023-07-02T14:05:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090155589999999994 2023-07-02T14:05:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371659+0000", 2023-07-02T14:05:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1297999999999994e-05 2023-07-02T14:05:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371730+0000", 2023-07-02T14:05:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013100699999999999 2023-07-02T14:05:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371861+0000", 2023-07-02T14:05:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188 2023-07-02T14:05:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2784 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 0~393216] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:05:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.299143261, 2023-07-02T14:05:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999, 2023-07-02T14:05:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2784 2023-07-02T14:05:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:05:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:05:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332385+0000", 2023-07-02T14:05:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040252400000000002 2023-07-02T14:05:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332787+0000", 2023-07-02T14:05:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0208e-05 2023-07-02T14:05:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332798+0000", 2023-07-02T14:05:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2023-07-02T14:05:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332801+0000", 2023-07-02T14:05:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4838000000000001e-05 2023-07-02T14:05:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332836+0000", 2023-07-02T14:05:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5319999999999998e-06 2023-07-02T14:05:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332839+0000", 2023-07-02T14:05:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0234290109999999 2023-07-02T14:05:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356268+0000", 2023-07-02T14:05:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011165700000000001 2023-07-02T14:05:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356380+0000", 2023-07-02T14:05:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022504799999999999 2023-07-02T14:05:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356605+0000", 2023-07-02T14:05:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015276201 2023-07-02T14:05:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371881+0000", 2023-07-02T14:05:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5496999999999999e-05 2023-07-02T14:05:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371957+0000", 2023-07-02T14:05:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4570000000000002e-05 2023-07-02T14:05:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372001+0000", 2023-07-02T14:05:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3880000000000003e-05 2023-07-02T14:05:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372085+0000", 2023-07-02T14:05:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999 2023-07-02T14:05:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2785 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 405504~446464] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:05:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.29872181, 2023-07-02T14:05:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999, 2023-07-02T14:05:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2785 2023-07-02T14:05:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:05:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:05:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:05:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059527500000000004 2023-07-02T14:05:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333402+0000", 2023-07-02T14:05:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7059999999999993e-06 2023-07-02T14:05:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333411+0000", 2023-07-02T14:05:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0450000000000001e-06 2023-07-02T14:05:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333414+0000", 2023-07-02T14:05:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5790999999999997e-05 2023-07-02T14:05:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:05:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3320000000000002e-06 2023-07-02T14:05:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333454+0000", 2023-07-02T14:05:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.025310269 2023-07-02T14:05:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358764+0000", 2023-07-02T14:05:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012414899999999999 2023-07-02T14:05:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358888+0000", 2023-07-02T14:05:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000217028 2023-07-02T14:05:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.359105+0000", 2023-07-02T14:05:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012788212 2023-07-02T14:05:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371893+0000", 2023-07-02T14:05:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012201569999999999 2023-07-02T14:05:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373114+0000", 2023-07-02T14:05:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4512999999999995e-05 2023-07-02T14:05:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373198+0000", 2023-07-02T14:05:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132373 2023-07-02T14:05:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373330+0000", 2023-07-02T14:05:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999 2023-07-02T14:05:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2786 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1310720~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:05:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.298109117, 2023-07-02T14:05:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001, 2023-07-02T14:05:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2786 2023-07-02T14:05:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:05:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:05:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333419+0000", 2023-07-02T14:05:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2009999999999999e-06 2023-07-02T14:05:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333427+0000", 2023-07-02T14:05:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0210000000000001e-06 2023-07-02T14:05:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333431+0000", 2023-07-02T14:05:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3769999999999999e-06 2023-07-02T14:05:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333434+0000", 2023-07-02T14:05:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1161e-05 2023-07-02T14:05:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333485+0000", 2023-07-02T14:05:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1549000000000001e-05 2023-07-02T14:05:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333496+0000", 2023-07-02T14:05:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0273116339999999 2023-07-02T14:05:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360808+0000", 2023-07-02T14:05:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000115128 2023-07-02T14:05:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360923+0000", 2023-07-02T14:05:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018367700000000001 2023-07-02T14:05:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361107+0000", 2023-07-02T14:05:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010796923999999999 2023-07-02T14:05:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371904+0000", 2023-07-02T14:05:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014983520000000001 2023-07-02T14:05:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373402+0000", 2023-07-02T14:05:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0520999999999999e-05 2023-07-02T14:05:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373463+0000", 2023-07-02T14:05:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000129095 2023-07-02T14:05:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373592+0000", 2023-07-02T14:05:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001 2023-07-02T14:05:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2787 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1433600~8192] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:05:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.298092317, 2023-07-02T14:05:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999, 2023-07-02T14:05:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2787 2023-07-02T14:05:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:05:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:05:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333436+0000", 2023-07-02T14:05:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2731999999999999e-05 2023-07-02T14:05:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:05:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7440000000000001e-06 2023-07-02T14:05:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333456+0000", 2023-07-02T14:05:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8650000000000003e-06 2023-07-02T14:05:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333460+0000", 2023-07-02T14:05:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0077e-05 2023-07-02T14:05:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333530+0000", 2023-07-02T14:05:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4180000000000001e-06 2023-07-02T14:05:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333533+0000", 2023-07-02T14:05:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0284002720000001 2023-07-02T14:05:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361933+0000", 2023-07-02T14:05:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6781999999999996e-05 2023-07-02T14:05:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362030+0000", 2023-07-02T14:05:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018305400000000001 2023-07-02T14:05:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362213+0000", 2023-07-02T14:05:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097015109999999995 2023-07-02T14:05:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371915+0000", 2023-07-02T14:05:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001742295 2023-07-02T14:05:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373657+0000", 2023-07-02T14:05:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5980000000000003e-05 2023-07-02T14:05:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373713+0000", 2023-07-02T14:05:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011783800000000001 2023-07-02T14:05:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373831+0000", 2023-07-02T14:05:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999 2023-07-02T14:05:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2803 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 851968~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:05:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.298050465, 2023-07-02T14:05:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514, 2023-07-02T14:05:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2803 2023-07-02T14:05:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:05:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:05:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:05:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4943000000000001e-05 2023-07-02T14:05:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333493+0000", 2023-07-02T14:05:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2629999999999997e-06 2023-07-02T14:05:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333500+0000", 2023-07-02T14:05:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4159999999999997e-06 2023-07-02T14:05:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333505+0000", 2023-07-02T14:05:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4518999999999998e-05 2023-07-02T14:05:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333559+0000", 2023-07-02T14:05:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5900000000000008e-06 2023-07-02T14:05:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333568+0000", 2023-07-02T14:05:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0291061779999999 2023-07-02T14:05:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362674+0000", 2023-07-02T14:05:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110412 2023-07-02T14:05:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362784+0000", 2023-07-02T14:05:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015384599999999999 2023-07-02T14:05:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362938+0000", 2023-07-02T14:05:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091682310000000006 2023-07-02T14:05:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372106+0000", 2023-07-02T14:05:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017703230000000001 2023-07-02T14:05:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373877+0000", 2023-07-02T14:05:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6387000000000002e-05 2023-07-02T14:05:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373923+0000", 2023-07-02T14:05:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014099199999999999 2023-07-02T14:05:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374064+0000", 2023-07-02T14:05:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514 2023-07-02T14:05:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2805 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 634880~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:05:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.29801729, 2023-07-02T14:05:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814, 2023-07-02T14:05:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2805 2023-07-02T14:05:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:05:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:05:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333511+0000", 2023-07-02T14:05:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207092 2023-07-02T14:05:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333718+0000", 2023-07-02T14:05:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5110000000000003e-06 2023-07-02T14:05:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333727+0000", 2023-07-02T14:05:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3270000000000007e-06 2023-07-02T14:05:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333736+0000", 2023-07-02T14:05:16.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0319999999999999e-05 2023-07-02T14:05:16.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333756+0000", 2023-07-02T14:05:16.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1695000000000001e-05 2023-07-02T14:05:16.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:16.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:05:16.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0295750560000001 2023-07-02T14:05:16.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363343+0000", 2023-07-02T14:05:16.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3387999999999994e-05 2023-07-02T14:05:16.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:16.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363437+0000", 2023-07-02T14:05:16.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018373 2023-07-02T14:05:16.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:16.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363620+0000", 2023-07-02T14:05:16.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0084948769999999996 2023-07-02T14:05:16.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:16.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372115+0000", 2023-07-02T14:05:16.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020082170000000001 2023-07-02T14:05:16.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374123+0000", 2023-07-02T14:05:16.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8875999999999997e-05 2023-07-02T14:05:16.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:16.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374182+0000", 2023-07-02T14:05:16.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0001272 2023-07-02T14:05:16.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374309+0000", 2023-07-02T14:05:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814 2023-07-02T14:05:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.798 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2806 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 1359872~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:16.798 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:05:16.798 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.29778695900001, 2023-07-02T14:05:16.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234, 2023-07-02T14:05:16.798 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:16.798 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.798 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:16.798 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2806 2023-07-02T14:05:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:05:16.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:16.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:16.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:05:16.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:16.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:16.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333741+0000", 2023-07-02T14:05:16.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3978999999999999e-05 2023-07-02T14:05:16.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:16.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333755+0000", 2023-07-02T14:05:16.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:05:16.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:16.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333761+0000", 2023-07-02T14:05:16.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7010000000000001e-06 2023-07-02T14:05:16.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333764+0000", 2023-07-02T14:05:16.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5086000000000003e-05 2023-07-02T14:05:16.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333799+0000", 2023-07-02T14:05:16.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6389999999999992e-06 2023-07-02T14:05:16.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:16.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333806+0000", 2023-07-02T14:05:16.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.030496675 2023-07-02T14:05:16.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364303+0000", 2023-07-02T14:05:16.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011382700000000001 2023-07-02T14:05:16.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:16.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364417+0000", 2023-07-02T14:05:16.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192058 2023-07-02T14:05:16.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:16.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364609+0000", 2023-07-02T14:05:16.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075132130000000004 2023-07-02T14:05:16.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:16.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372122+0000", 2023-07-02T14:05:16.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022731750000000001 2023-07-02T14:05:16.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374395+0000", 2023-07-02T14:05:16.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5277000000000005e-05 2023-07-02T14:05:16.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:16.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374460+0000", 2023-07-02T14:05:16.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017074599999999999 2023-07-02T14:05:16.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:16.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374631+0000", 2023-07-02T14:05:16.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234 2023-07-02T14:05:16.809 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.809 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.809 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.809 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 868352~49152] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:16.809 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:05:16.809 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.29776024500001, 2023-07-02T14:05:16.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001, 2023-07-02T14:05:16.810 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:16.810 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:16.810 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:16.810 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:16.810 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:16.810 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:05:16.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.811 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:16.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:16.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:05:16.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:16.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:16.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:05:16.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:16.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:16.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:05:16.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6355000000000001e-05 2023-07-02T14:05:16.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:16.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333794+0000", 2023-07-02T14:05:16.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8329999999999996e-06 2023-07-02T14:05:16.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:16.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333801+0000", 2023-07-02T14:05:16.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5419999999999996e-06 2023-07-02T14:05:16.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333807+0000", 2023-07-02T14:05:16.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2023-07-02T14:05:16.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333826+0000", 2023-07-02T14:05:16.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.303e-06 2023-07-02T14:05:16.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:16.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333830+0000", 2023-07-02T14:05:16.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.031137559 2023-07-02T14:05:16.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364967+0000", 2023-07-02T14:05:16.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1390000000000004e-05 2023-07-02T14:05:16.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:16.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365059+0000", 2023-07-02T14:05:16.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017597399999999999 2023-07-02T14:05:16.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:16.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365234+0000", 2023-07-02T14:05:16.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068943950000000002 2023-07-02T14:05:16.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:16.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372129+0000", 2023-07-02T14:05:16.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026189910000000002 2023-07-02T14:05:16.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:16.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374748+0000", 2023-07-02T14:05:16.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6613999999999996e-05 2023-07-02T14:05:16.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:16.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374814+0000", 2023-07-02T14:05:16.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502300000000001 2023-07-02T14:05:16.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:16.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374950+0000", 2023-07-02T14:05:16.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001 2023-07-02T14:05:16.820 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.820 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.821 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.821 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:172 2.5 2:a79a9f1f:::rbd_data.125de1449605.0000000000000039:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:16.821 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.848672+0000", 2023-07-02T14:05:16.821 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.782857310000001, 2023-07-02T14:05:16.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041540012000000001, 2023-07-02T14:05:16.821 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:16.822 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:16.822 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:16.822 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:16.822 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:16.822 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 172 2023-07-02T14:05:16.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.822 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:16.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:16.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848672+0000", 2023-07-02T14:05:16.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:16.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:16.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848672+0000", 2023-07-02T14:05:16.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:05:16.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:16.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848669+0000", 2023-07-02T14:05:16.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1610000000000001e-05 2023-07-02T14:05:16.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:16.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848690+0000", 2023-07-02T14:05:16.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0169999999999998e-05 2023-07-02T14:05:16.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:16.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848710+0000", 2023-07-02T14:05:16.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0960000000000007e-06 2023-07-02T14:05:16.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848720+0000", 2023-07-02T14:05:16.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0717000000000002e-05 2023-07-02T14:05:16.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848780+0000", 2023-07-02T14:05:16.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.303e-06 2023-07-02T14:05:16.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2023-07-02T14:05:16.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848789+0000", 2023-07-02T14:05:16.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032354921000000002 2023-07-02T14:05:16.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.881144+0000", 2023-07-02T14:05:16.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0827000000000005e-05 2023-07-02T14:05:16.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:16.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.881214+0000", 2023-07-02T14:05:16.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089973080000000007 2023-07-02T14:05:16.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:16.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.890212+0000", 2023-07-02T14:05:16.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041540012000000001 2023-07-02T14:05:16.829 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.830 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.830 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.830 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:174 2.5 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:16.830 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.851335+0000", 2023-07-02T14:05:16.830 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.780194102999999, 2023-07-02T14:05:16.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.044329581999999999, 2023-07-02T14:05:16.831 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:16.831 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:16.831 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:16.831 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:16.831 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:16.831 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 174 2023-07-02T14:05:16.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.831 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:16.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:16.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851335+0000", 2023-07-02T14:05:16.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:16.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:16.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851335+0000", 2023-07-02T14:05:16.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2023-07-02T14:05:16.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:16.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851332+0000", 2023-07-02T14:05:16.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6269999999999998e-05 2023-07-02T14:05:16.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:16.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851348+0000", 2023-07-02T14:05:16.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3189000000000001e-05 2023-07-02T14:05:16.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:16.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851371+0000", 2023-07-02T14:05:16.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.132e-06 2023-07-02T14:05:16.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851380+0000", 2023-07-02T14:05:16.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7970999999999999e-05 2023-07-02T14:05:16.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851428+0000", 2023-07-02T14:05:16.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1050000000000003e-06 2023-07-02T14:05:16.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2023-07-02T14:05:16.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851433+0000", 2023-07-02T14:05:16.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038825011999999999 2023-07-02T14:05:16.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.890258+0000", 2023-07-02T14:05:16.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0666000000000002e-05 2023-07-02T14:05:16.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:16.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.890349+0000", 2023-07-02T14:05:16.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0053153460000000003 2023-07-02T14:05:16.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:16.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.895664+0000", 2023-07-02T14:05:16.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.044329581999999999 2023-07-02T14:05:16.839 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.839 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.839 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.839 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:336 2.5 2:a8e921ad:::rbd_data.125de1449605.00000000000000d7:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:16.839 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636630+0000", 2023-07-02T14:05:16.839 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.994899125, 2023-07-02T14:05:16.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041573730000000003, 2023-07-02T14:05:16.840 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:16.840 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:16.840 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:16.840 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:16.840 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:16.840 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 336 2023-07-02T14:05:16.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.841 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:16.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:16.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636630+0000", 2023-07-02T14:05:16.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:16.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:16.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636630+0000", 2023-07-02T14:05:16.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:16.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:16.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636629+0000", 2023-07-02T14:05:16.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.847e-06 2023-07-02T14:05:16.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:16.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636632+0000", 2023-07-02T14:05:16.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0449999999999997e-06 2023-07-02T14:05:16.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:16.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636638+0000", 2023-07-02T14:05:16.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5080000000000001e-06 2023-07-02T14:05:16.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636642+0000", 2023-07-02T14:05:16.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033756268999999998 2023-07-02T14:05:16.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.670398+0000", 2023-07-02T14:05:16.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8985000000000004e-05 2023-07-02T14:05:16.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:16.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.670467+0000", 2023-07-02T14:05:16.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077364360000000002 2023-07-02T14:05:16.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:16.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.678204+0000", 2023-07-02T14:05:16.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041573730000000003 2023-07-02T14:05:16.846 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.846 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.847 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.847 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:343 2.5 2:b3c82204:::rbd_data.125de1449605.00000000000000de:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:16.847 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636645+0000", 2023-07-02T14:05:16.847 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.994883695999999, 2023-07-02T14:05:16.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.048581695000000001, 2023-07-02T14:05:16.847 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:16.848 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:16.848 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:16.848 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:16.848 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:16.848 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 343 2023-07-02T14:05:16.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.848 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:16.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:16.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636645+0000", 2023-07-02T14:05:16.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:16.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:16.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636645+0000", 2023-07-02T14:05:16.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:16.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:16.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636645+0000", 2023-07-02T14:05:16.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9190000000000004e-06 2023-07-02T14:05:16.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:16.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636651+0000", 2023-07-02T14:05:16.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4e-06 2023-07-02T14:05:16.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:16.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636656+0000", 2023-07-02T14:05:16.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5860000000000001e-06 2023-07-02T14:05:16.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:16.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636660+0000", 2023-07-02T14:05:16.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041586692000000001 2023-07-02T14:05:16.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:16.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.678246+0000", 2023-07-02T14:05:16.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113236 2023-07-02T14:05:16.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:16.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.678360+0000", 2023-07-02T14:05:16.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006867199 2023-07-02T14:05:16.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:16.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:16.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:16.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.685227+0000", 2023-07-02T14:05:16.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.048581695000000001 2023-07-02T14:05:16.854 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.854 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.854 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.854 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:16.855 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:16.855 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:16.856 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:05:16.856 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:16.855+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:16.890 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:05:16.956 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:16.955+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:17.025 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:17.026 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:17.026 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:05:17.026 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:17.036 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:05:17.037 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:05:17.055 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:17.057 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:17.057+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:17.074 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:17.093 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:17.111 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:17.130 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:17.132 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:05:17.132 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:05:17.132 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:05:17.132 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:05:17.133 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:05:17.133 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:05:17.139 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:05:17.150 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:05:17.185 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:17.184+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:17.188 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:17.189 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:17.211 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:05:17.212 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:05:17.218 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 4 2023-07-02T14:05:17.295 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:17.294+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:17.359 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:17.358+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:17.460 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:17.459+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:17.475 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:17.475 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:17.491 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:05:17.561 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:17.560+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:17.662 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:17.661+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:17.689 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:05:17.699 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:05:17.723 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2023-07-02T14:05:17.130488+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6029716253,"num_objects":1811,"num_object_clones":1421,"num_object_copies":3622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":194,"num_objects_misplaced":11,"num_objects_unfound":0,"num_objects_dirty":1811,"num_whiteouts":0,"num_read":32696,"num_read_kb":7744133,"num_write":15032,"num_write_kb":5859094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1282,"num_bytes_recovered":4428487187,"num_keys_recovered":70,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14760,"ondisk_log_size":14760,"up":18,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":10924812,"kb_used_data":10757612,"kb_used_omap":229,"kb_used_meta":166938,"kb_avail":366562548,"statfs":{"total":386547056640,"available":375360049152,"internally_reserved":0,"allocated":11015794688,"data_stored":12954154489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":170944734},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38539426,"num_objects":18,"num_object_clones":7,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-285,"num_objects_misplaced":11,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":239,"num_read_kb":4427,"num_write":101,"num_write_kb":37690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":276,"num_bytes_recovered":951354170,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.016906"},"pg_stats":[{"pgid":"2.7","version":"137'2116","reported_seq":4673,"reported_epoch":137,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-07-02T14:05:09.833316+0000","last_change":"2023-07-02T14:05:06.140063+0000","last_active":"2023-07-02T14:05:09.833316+0000","last_peered":"2023-07-02T14:05:09.833316+0000","last_clean":"2023-07-02T14:05:03.081045+0000","last_became_active":"2023-07-02T14:05:06.138824+0000","last_became_peered":"2023-07-02T14:05:06.138824+0000","last_unstale":"2023-07-02T14:05:09.833316+0000","last_undegraded":"2023-07-02T14:05:06.091635+0000","last_fullsized":"2023-07-02T14:05:06.088179+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2116,"ondisk_log_size":2116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":784893934,"num_objects":236,"num_object_clones":185,"num_object_copies":472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":236,"num_whiteouts":0,"num_read":17270,"num_read_kb":986139,"num_write":1825,"num_write_kb":762999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":245,"num_bytes_recovered":786067668,"num_keys_recovered":66,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[2],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2","objects":55},{"shards":"2,4","objects":170}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"142'1630","reported_seq":2905,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-07-02T14:05:15.112303+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:05:15.112303+0000","last_peered":"2023-07-02T14:05:15.112303+0000","last_clean":"2023-07-02T14:05:15.112303+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:05:15.112303+0000","last_undegraded":"2023-07-02T14:05:15.112303+0000","last_fullsized":"2023-07-02T14:05:15.112303+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1630,"ondisk_log_size":1630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":781222374,"num_objects":241,"num_object_clones":194,"num_object_copies":482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":241,"num_whiteouts":0,"num_read":948,"num_read_kb":1028437,"num_write":1436,"num_write_kb":759107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"138'1753","reported_seq":2053,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-07-02T14:05:11.107594+0000","last_change":"2023-07-02T14:05:10.106568+0000","last_active":"2023-07-02T14:05:11.107594+0000","last_peered":"2023-07-02T14:05:11.107594+0000","last_clean":"2023-07-02T14:05:11.107594+0000","last_became_active":"2023-07-02T14:05:10.103318+0000","last_became_peered":"2023-07-02T14:05:10.103318+0000","last_unstale":"2023-07-02T14:05:11.107594+0000","last_undegraded":"2023-07-02T14:05:11.107594+0000","last_fullsized":"2023-07-02T14:05:11.107594+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1753,"ondisk_log_size":1753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":746074598,"num_objects":208,"num_object_clones":164,"num_object_copies":416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":208,"num_whiteouts":0,"num_read":328,"num_read_kb":927502,"num_write":1589,"num_write_kb":718544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":572731554,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"143'3494","reported_seq":4609,"reported_epoch":143,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-07-02T14:05:15.307281+0000","last_change":"2023-07-02T14:05:12.813556+0000","last_active":"2023-07-02T14:05:15.307281+0000","last_peered":"2023-07-02T14:05:15.307281+0000","last_clean":"2023-07-02T14:05:03.082409+0000","last_became_active":"2023-07-02T14:05:06.167341+0000","last_became_peered":"2023-07-02T14:05:06.167341+0000","last_unstale":"2023-07-02T14:05:15.307281+0000","last_undegraded":"2023-07-02T14:05:06.102378+0000","last_fullsized":"2023-07-02T14:05:06.092644+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3494,"ondisk_log_size":3494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":853656540,"num_objects":253,"num_object_clones":198,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":11,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":11692,"num_read_kb":1126682,"num_write":5095,"num_write_kb":833901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":798122988,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[1],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1","objects":229},{"shards":"1,4","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":91,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-07-02T14:05:14.130460+0000","last_change":"2023-07-02T14:05:05.089421+0000","last_active":"2023-07-02T14:05:14.130460+0000","last_peered":"2023-07-02T14:05:14.130460+0000","last_clean":"2023-07-02T14:05:14.130460+0000","last_became_active":"2023-07-02T14:05:05.088812+0000","last_became_peered":"2023-07-02T14:05:05.088812+0000","last_unstale":"2023-07-02T14:05:14.130460+0000","last_undegraded":"2023-07-02T14:05:14.130460+0000","last_fullsized":"2023-07-02T14:05:14.130460+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"143'1620","reported_seq":2403,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-07-02T14:05:15.158329+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:05:15.158329+0000","last_peered":"2023-07-02T14:05:15.158329+0000","last_clean":"2023-07-02T14:05:15.158329+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:05:15.158329+0000","last_undegraded":"2023-07-02T14:05:15.158329+0000","last_fullsized":"2023-07-02T14:05:15.158329+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1620,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":831550266,"num_objects":246,"num_object_clones":192,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":246,"num_whiteouts":0,"num_read":636,"num_read_kb":1051850,"num_write":1428,"num_write_kb":809574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"143'1601","reported_seq":2078,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-07-02T14:05:15.174544+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:05:15.174544+0000","last_peered":"2023-07-02T14:05:15.174544+0000","last_clean":"2023-07-02T14:05:15.174544+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:05:15.174544+0000","last_undegraded":"2023-07-02T14:05:15.174544+0000","last_fullsized":"2023-07-02T14:05:15.174544+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1601,"ondisk_log_size":1601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":828306218,"num_objects":242,"num_object_clones":187,"num_object_copies":484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":242,"num_whiteouts":0,"num_read":421,"num_read_kb":984584,"num_write":1414,"num_write_kb":805785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"143'1210","reported_seq":2220,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-07-02T14:05:15.164734+0000","last_change":"2023-07-02T14:03:13.073775+0000","last_active":"2023-07-02T14:05:15.164734+0000","last_peered":"2023-07-02T14:05:15.164734+0000","last_clean":"2023-07-02T14:05:15.164734+0000","last_became_active":"2023-07-02T14:03:13.073206+0000","last_became_peered":"2023-07-02T14:03:13.073206+0000","last_unstale":"2023-07-02T14:05:15.164734+0000","last_undegraded":"2023-07-02T14:05:15.164734+0000","last_fullsized":"2023-07-02T14:05:15.164734+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1210,"ondisk_log_size":1210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":584643227,"num_objects":195,"num_object_clones":153,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":975,"num_read_kb":828826,"num_write":1057,"num_write_kb":567550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":96600083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"142'1336","reported_seq":1765,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-07-02T14:05:15.119878+0000","last_change":"2023-07-02T14:03:14.074306+0000","last_active":"2023-07-02T14:05:15.119878+0000","last_peered":"2023-07-02T14:05:15.119878+0000","last_clean":"2023-07-02T14:05:15.119878+0000","last_became_active":"2023-07-02T14:03:14.073735+0000","last_became_peered":"2023-07-02T14:03:14.073735+0000","last_unstale":"2023-07-02T14:05:15.119878+0000","last_undegraded":"2023-07-02T14:05:15.119878+0000","last_fullsized":"2023-07-02T14:05:15.119878+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1336,"ondisk_log_size":1336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":619369096,"num_objects":190,"num_object_clones":148,"num_object_copies":380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":0,"num_read":426,"num_read_kb":810113,"num_write":1188,"num_write_kb":601634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":87949312,"num_keys_recovered":4,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":6029716253,"num_objects":1811,"num_object_clones":1421,"num_object_copies":3622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":194,"num_objects_misplaced":11,"num_objects_unfound":0,"num_objects_dirty":1811,"num_whiteouts":0,"num_read":32696,"num_read_kb":7744133,"num_write":15032,"num_write_kb":5859094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1282,"num_bytes_recovered":4428487187,"num_keys_recovered":70,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12613038080,"data_stored":14805118682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},"log_size":14760,"ondisk_log_size":14760,"up":16,"acting":14,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640300,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":0,"seq":47244640300,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":133,"seq":571230650371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2156680,"kb_used_data":2137024,"kb_used_omap":229,"kb_used_meta":19418,"kb_avail":92215160,"statfs":{"total":96636764160,"available":94428323840,"internally_reserved":0,"allocated":2188312576,"data_stored":2556144423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":19884254},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640300,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2422476,"kb_used_data":2384388,"kb_used_omap":0,"kb_used_meta":38080,"kb_avail":91949364,"statfs":{"total":96636764160,"available":94156148736,"internally_reserved":0,"allocated":2441613312,"data_stored":2809402636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38993920},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.29399999999999998,"15min":0.29599999999999999},"min":{"1min":0.218,"5min":0.182,"15min":0.182},"max":{"1min":0.378,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.317,"5min":0.315,"15min":0.314},"min":{"1min":0.23000000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.28899999999999998}]},{"osd":1,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34499999999999997,"15min":0.34699999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.48499999999999999,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.254},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.314,"15min":0.317},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.39700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.432}]},{"osd":3,"last update":"Sun Jul 2 14:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.45000000000000001,"15min":0.45100000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46999999999999997,"15min":0.46800000000000003},"min":{"1min":0.35299999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.60399999999999998}]},{"osd":4,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.443,"15min":0.44700000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.56799999999999995,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.45900000000000002,"15min":0.46000000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.57599999999999996,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.68400000000000005}]},{"osd":5,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47899999999999998,"15min":0.48299999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.65200000000000002,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.47599999999999998,"15min":0.48099999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.54600000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.61099999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640300,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1923608,"kb_used_data":1884112,"kb_used_omap":0,"kb_used_meta":39488,"kb_avail":92448232,"statfs":{"total":96636764160,"available":94666989568,"internally_reserved":0,"allocated":1929330688,"data_stored":2300390851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40435712},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.29099999999999998,"15min":0.29299999999999998},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.29699999999999999,"5min":0.30099999999999999,"15min":0.29999999999999999},"min":{"1min":0.216,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.42199999999999999,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.33500000000000002}]},{"osd":2,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.34300000000000003,"15min":0.34799999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.48599999999999999,"5min":0.746,"15min":0.746},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.314,"5min":0.34100000000000003,"15min":0.34699999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.40000000000000002,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.29699999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.35599999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.49399999999999999,"15min":0.496},"min":{"1min":0.32400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.84899999999999998,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.36899999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.47799999999999998,"15min":0.48499999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.77600000000000002,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.48399999999999999,"15min":0.49299999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.61099999999999999,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.43099999999999999}]},{"osd":5,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.48099999999999998,"15min":0.495},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.59199999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.502,"15min":0.51200000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.54800000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640300,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4422048,"kb_used_data":4352088,"kb_used_omap":0,"kb_used_meta":69952,"kb_avail":89949792,"statfs":{"total":96636764160,"available":92108587008,"internally_reserved":0,"allocated":4456538112,"data_stored":5288216579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":71630848},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.20200000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.32200000000000001,"15min":0.33000000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.41099999999999998,"5min":0.496,"15min":0.496},"last":0.24299999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.33800000000000002,"15min":0.33200000000000002},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.222},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.36599999999999999,"15min":0.376},"min":{"1min":0.251,"5min":0.222,"15min":0.222},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.34200000000000003}]},{"osd":3,"last update":"Sun Jul 2 14:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.497,"15min":0.501},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.624,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.309},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.498,"15min":0.499},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.64400000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.314}]},{"osd":4,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.48699999999999999,"15min":0.503},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.54300000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.376},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.498,"15min":0.504},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.45600000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51400000000000001,"15min":0.51500000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.5589999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.48699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1938440192,"data_stored":2310539017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4478885888,"data_stored":5311603017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2459115520,"data_stored":2827943506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2353426432,"data_stored":2742938894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1383170048,"data_stored":1612094248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:05:17.724 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-07-02T14:05:17.763 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:17.763 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:17.764 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:17.762+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:17.784 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:05:17.867 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:17.866+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:17.965 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:17.964+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:18.058 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:18.058 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:18.066 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:18.065+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:18.070 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:05:18.165 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:05:18.167 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:18.166+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:18.267 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:18.266+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:18.341 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:18.341 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:18.351 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:05:18.394 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:18.394+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:18.418 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:05:18.418 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 194/3622 objects degraded (5.356%), 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":221,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":146,"num_osds":6,"num_up_osds":6,"osd_up_since":1688306704,"num_in_osds":4,"osd_in_since":1688306681,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":1811,"data_bytes":6029716253,"bytes_used":11187007488,"bytes_avail":375360049152,"bytes_total":386547056640,"degraded_objects":194,"degraded_total":3622,"degraded_ratio":0.053561568194367755,"misplaced_objects":11,"misplaced_total":3622,"misplaced_ratio":0.0030369961347321921,"recovering_objects_per_sec":107,"recovering_bytes_per_sec":380948176,"recovering_keys_per_sec":0,"num_objects_recovered":431,"num_bytes_recovered":1524085724,"num_keys_recovered":0,"read_bytes_sec":1133094,"write_bytes_sec":11743533,"read_op_per_sec":59,"write_op_per_sec":26},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2023-07-02T14:02:46.242493+0000","services":{}},"progress_events":{}} 2023-07-02T14:05:18.447 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2023-07-02T14:05:18.447 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:05:18.469 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:18.468+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:18.516 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:18.516 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:18.529 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:05:18.570 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:18.569+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:18.675 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:18.670+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:18.707 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:18.707 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:18.723 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:05:18.771 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:18.770+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:18.874 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:18.871+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:18.882 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:18.882 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:18.899 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:05:18.900 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:05:18.904 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:05:18.924 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2023-07-02T14:05:18.270021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6166891293,"num_objects":1851,"num_object_clones":1455,"num_object_copies":3702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1851,"num_whiteouts":0,"num_read":34120,"num_read_kb":7745189,"num_write":15437,"num_write_kb":5993226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1487,"num_bytes_recovered":5100088637,"num_keys_recovered":114,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15118,"ondisk_log_size":15118,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":11925144,"kb_used_data":11747768,"kb_used_omap":229,"kb_used_meta":177114,"kb_avail":365562216,"statfs":{"total":386547056640,"available":374335709184,"internally_reserved":0,"allocated":12029714432,"data_stored":14108776675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":181364958},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,0,0,0,0,8],"upper_bound":2048},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":175714466,"num_objects":58,"num_object_clones":41,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":1663,"num_read_kb":5483,"num_write":506,"num_write_kb":171822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":1622955620,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000764"},"pg_stats":[{"pgid":"2.7","version":"145'2242","reported_seq":5072,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-07-02T14:05:18.055142+0000","last_change":"2023-07-02T14:05:14.122260+0000","last_active":"2023-07-02T14:05:18.055142+0000","last_peered":"2023-07-02T14:05:18.055142+0000","last_clean":"2023-07-02T14:05:18.055142+0000","last_became_active":"2023-07-02T14:05:14.120593+0000","last_became_peered":"2023-07-02T14:05:14.120593+0000","last_unstale":"2023-07-02T14:05:18.055142+0000","last_undegraded":"2023-07-02T14:05:18.055142+0000","last_fullsized":"2023-07-02T14:05:18.055142+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2242,"ondisk_log_size":2242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":843876334,"num_objects":253,"num_object_clones":199,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":18184,"num_read_kb":986770,"num_write":1933,"num_write_kb":820601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":1395581788,"num_keys_recovered":110,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"142'1630","reported_seq":2905,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-07-02T14:05:15.112303+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:05:15.112303+0000","last_peered":"2023-07-02T14:05:15.112303+0000","last_clean":"2023-07-02T14:05:15.112303+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:05:15.112303+0000","last_undegraded":"2023-07-02T14:05:15.112303+0000","last_fullsized":"2023-07-02T14:05:15.112303+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1630,"ondisk_log_size":1630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":781222374,"num_objects":241,"num_object_clones":194,"num_object_copies":482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":241,"num_whiteouts":0,"num_read":948,"num_read_kb":1028437,"num_write":1436,"num_write_kb":759107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"145'1834","reported_seq":2129,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-07-02T14:05:18.008096+0000","last_change":"2023-07-02T14:05:10.106568+0000","last_active":"2023-07-02T14:05:18.008096+0000","last_peered":"2023-07-02T14:05:18.008096+0000","last_clean":"2023-07-02T14:05:18.008096+0000","last_became_active":"2023-07-02T14:05:10.103318+0000","last_became_peered":"2023-07-02T14:05:10.103318+0000","last_unstale":"2023-07-02T14:05:18.008096+0000","last_undegraded":"2023-07-02T14:05:18.008096+0000","last_fullsized":"2023-07-02T14:05:18.008096+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1834,"ondisk_log_size":1834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":786903526,"num_objects":221,"num_object_clones":174,"num_object_copies":442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":221,"num_whiteouts":0,"num_read":328,"num_read_kb":927502,"num_write":1660,"num_write_kb":758416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":572731554,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"145'3645","reported_seq":4732,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-07-02T14:05:18.087121+0000","last_change":"2023-07-02T14:05:17.141529+0000","last_active":"2023-07-02T14:05:18.087121+0000","last_peered":"2023-07-02T14:05:18.087121+0000","last_clean":"2023-07-02T14:05:18.087121+0000","last_became_active":"2023-07-02T14:05:17.138301+0000","last_became_peered":"2023-07-02T14:05:17.138301+0000","last_unstale":"2023-07-02T14:05:18.087121+0000","last_undegraded":"2023-07-02T14:05:18.087121+0000","last_fullsized":"2023-07-02T14:05:18.087121+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3645,"ondisk_log_size":3645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":891020252,"num_objects":263,"num_object_clones":208,"num_object_copies":526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":263,"num_whiteouts":0,"num_read":12202,"num_read_kb":1127107,"num_write":5321,"num_write_kb":870559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":860210318,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":91,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-07-02T14:05:14.130460+0000","last_change":"2023-07-02T14:05:05.089421+0000","last_active":"2023-07-02T14:05:14.130460+0000","last_peered":"2023-07-02T14:05:14.130460+0000","last_clean":"2023-07-02T14:05:14.130460+0000","last_became_active":"2023-07-02T14:05:05.088812+0000","last_became_peered":"2023-07-02T14:05:05.088812+0000","last_unstale":"2023-07-02T14:05:14.130460+0000","last_undegraded":"2023-07-02T14:05:14.130460+0000","last_fullsized":"2023-07-02T14:05:14.130460+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"143'1620","reported_seq":2403,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-07-02T14:05:15.158329+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:05:15.158329+0000","last_peered":"2023-07-02T14:05:15.158329+0000","last_clean":"2023-07-02T14:05:15.158329+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:05:15.158329+0000","last_undegraded":"2023-07-02T14:05:15.158329+0000","last_fullsized":"2023-07-02T14:05:15.158329+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1620,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":831550266,"num_objects":246,"num_object_clones":192,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":246,"num_whiteouts":0,"num_read":636,"num_read_kb":1051850,"num_write":1428,"num_write_kb":809574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"143'1601","reported_seq":2078,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-07-02T14:05:15.174544+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:05:15.174544+0000","last_peered":"2023-07-02T14:05:15.174544+0000","last_clean":"2023-07-02T14:05:15.174544+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:05:15.174544+0000","last_undegraded":"2023-07-02T14:05:15.174544+0000","last_fullsized":"2023-07-02T14:05:15.174544+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1601,"ondisk_log_size":1601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":828306218,"num_objects":242,"num_object_clones":187,"num_object_copies":484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":242,"num_whiteouts":0,"num_read":421,"num_read_kb":984584,"num_write":1414,"num_write_kb":805785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"143'1210","reported_seq":2220,"reported_epoch":143,"state":"active+clean","last_fresh":"2023-07-02T14:05:15.164734+0000","last_change":"2023-07-02T14:03:13.073775+0000","last_active":"2023-07-02T14:05:15.164734+0000","last_peered":"2023-07-02T14:05:15.164734+0000","last_clean":"2023-07-02T14:05:15.164734+0000","last_became_active":"2023-07-02T14:03:13.073206+0000","last_became_peered":"2023-07-02T14:03:13.073206+0000","last_unstale":"2023-07-02T14:05:15.164734+0000","last_undegraded":"2023-07-02T14:05:15.164734+0000","last_fullsized":"2023-07-02T14:05:15.164734+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1210,"ondisk_log_size":1210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":584643227,"num_objects":195,"num_object_clones":153,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":975,"num_read_kb":828826,"num_write":1057,"num_write_kb":567550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":96600083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"142'1336","reported_seq":1765,"reported_epoch":142,"state":"active+clean","last_fresh":"2023-07-02T14:05:15.119878+0000","last_change":"2023-07-02T14:03:14.074306+0000","last_active":"2023-07-02T14:05:15.119878+0000","last_peered":"2023-07-02T14:05:15.119878+0000","last_clean":"2023-07-02T14:05:15.119878+0000","last_became_active":"2023-07-02T14:03:14.073735+0000","last_became_peered":"2023-07-02T14:03:14.073735+0000","last_unstale":"2023-07-02T14:05:15.119878+0000","last_undegraded":"2023-07-02T14:05:15.119878+0000","last_fullsized":"2023-07-02T14:05:15.119878+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1336,"ondisk_log_size":1336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":619369096,"num_objects":190,"num_object_clones":148,"num_object_copies":380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":0,"num_read":426,"num_read_kb":810113,"num_write":1188,"num_write_kb":601634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":87949312,"num_keys_recovered":4,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":6166891293,"num_objects":1851,"num_object_clones":1455,"num_object_copies":3702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1851,"num_whiteouts":0,"num_read":34120,"num_read_kb":7745189,"num_write":15437,"num_write_kb":5993226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1487,"num_bytes_recovered":5100088637,"num_keys_recovered":114,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13588520960,"data_stored":15922483554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},"log_size":15118,"ondisk_log_size":15118,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640300,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":0,"seq":47244640300,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":133,"seq":571230650372,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3157012,"kb_used_data":3127180,"kb_used_omap":229,"kb_used_meta":29594,"kb_avail":91214828,"statfs":{"total":96636764160,"available":93403983872,"internally_reserved":0,"allocated":3202232320,"data_stored":3710766609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":30304478},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,8],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640300,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2422476,"kb_used_data":2384388,"kb_used_omap":0,"kb_used_meta":38080,"kb_avail":91949364,"statfs":{"total":96636764160,"available":94156148736,"internally_reserved":0,"allocated":2441613312,"data_stored":2809402636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38993920},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.29399999999999998,"15min":0.29599999999999999},"min":{"1min":0.218,"5min":0.182,"15min":0.182},"max":{"1min":0.378,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.317,"5min":0.315,"15min":0.314},"min":{"1min":0.23000000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.28899999999999998}]},{"osd":1,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34499999999999997,"15min":0.34699999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.48499999999999999,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.254},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.314,"15min":0.317},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.39700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.432}]},{"osd":3,"last update":"Sun Jul 2 14:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.45000000000000001,"15min":0.45100000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46999999999999997,"15min":0.46800000000000003},"min":{"1min":0.35299999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.60399999999999998}]},{"osd":4,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.443,"15min":0.44700000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.56799999999999995,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.45900000000000002,"15min":0.46000000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.57599999999999996,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.68400000000000005}]},{"osd":5,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47899999999999998,"15min":0.48299999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.65200000000000002,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.47599999999999998,"15min":0.48099999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.54600000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.61099999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640300,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1923608,"kb_used_data":1884112,"kb_used_omap":0,"kb_used_meta":39488,"kb_avail":92448232,"statfs":{"total":96636764160,"available":94666989568,"internally_reserved":0,"allocated":1929330688,"data_stored":2300390851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40435712},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.29099999999999998,"15min":0.29299999999999998},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.29699999999999999,"5min":0.30099999999999999,"15min":0.29999999999999999},"min":{"1min":0.216,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.42199999999999999,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.33500000000000002}]},{"osd":2,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.34300000000000003,"15min":0.34799999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.48599999999999999,"5min":0.746,"15min":0.746},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.314,"5min":0.34100000000000003,"15min":0.34699999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.40000000000000002,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.29699999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.35599999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.49399999999999999,"15min":0.496},"min":{"1min":0.32400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.84899999999999998,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.36899999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.47799999999999998,"15min":0.48499999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.77600000000000002,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.48399999999999999,"15min":0.49299999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.61099999999999999,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.43099999999999999}]},{"osd":5,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.48099999999999998,"15min":0.495},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.59199999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.502,"15min":0.51200000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.54800000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640300,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4422048,"kb_used_data":4352088,"kb_used_omap":0,"kb_used_meta":69952,"kb_avail":89949792,"statfs":{"total":96636764160,"available":92108587008,"internally_reserved":0,"allocated":4456538112,"data_stored":5288216579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":71630848},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.20200000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.32200000000000001,"15min":0.33000000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.41099999999999998,"5min":0.496,"15min":0.496},"last":0.24299999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.33800000000000002,"15min":0.33200000000000002},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.222},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.36599999999999999,"15min":0.376},"min":{"1min":0.251,"5min":0.222,"15min":0.222},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.34200000000000003}]},{"osd":3,"last update":"Sun Jul 2 14:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.497,"15min":0.501},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.624,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.309},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.498,"15min":0.499},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.64400000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.314}]},{"osd":4,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.48699999999999999,"15min":0.503},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.54300000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.376},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.498,"15min":0.504},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.45600000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51400000000000001,"15min":0.51500000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.5589999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.48699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1938440192,"data_stored":2310539017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4478885888,"data_stored":5311603017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2459115520,"data_stored":2827943506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3328909312,"data_stored":3860303766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1383170048,"data_stored":1612094248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:05:18.973 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:18.972+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:19.049 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:19.050 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:19.062 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:05:19.106 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:19.105+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:19.144 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:05:19.190 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:19.189+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:19.226 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:19.226 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:19.238 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:05:19.276 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:19.275+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:19.351 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:05:19.376 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:19.375+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:19.384 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:19.415 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:19.433 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:19.433 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:19.446 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:19.447 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:05:19.473 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:19.477 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:19.476+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:19.501 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:05:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:05:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:05:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:05:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:05:19.507 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:05:19.578 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:19.577+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:19.602 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:19.603 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:19.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.603 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15778 2.5 2:a18ce92e:::rbd_data.125de1449605.00000000000000f7:head [write 917504~1703936 in=1703936b] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:19.603 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.591432+0000", 2023-07-02T14:05:19.603 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.011450784, 2023-07-02T14:05:19.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011475097, 2023-07-02T14:05:19.604 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:19.604 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "waiting for sub ops", 2023-07-02T14:05:19.604 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:19.604 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:19.604 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:19.604 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15778 2023-07-02T14:05:19.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.604 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:19.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:19.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.591432+0000", 2023-07-02T14:05:19.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:19.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:19.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.591432+0000", 2023-07-02T14:05:19.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:19.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:19.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.591431+0000", 2023-07-02T14:05:19.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052303740000000003 2023-07-02T14:05:19.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:19.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.596661+0000", 2023-07-02T14:05:19.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7099e-05 2023-07-02T14:05:19.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:19.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.596678+0000", 2023-07-02T14:05:19.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0399999999999998e-06 2023-07-02T14:05:19.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:19.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.596687+0000", 2023-07-02T14:05:19.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3052999999999994e-05 2023-07-02T14:05:19.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:19.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.596750+0000", 2023-07-02T14:05:19.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3174000000000001e-05 2023-07-02T14:05:19.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:19.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.596824+0000", 2023-07-02T14:05:19.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011995899999999999 2023-07-02T14:05:19.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:19.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.596944+0000", 2023-07-02T14:05:19.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039184470000000002 2023-07-02T14:05:19.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:19.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.600862+0000", 2023-07-02T14:05:19.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009430028 2023-07-02T14:05:19.611 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.611 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:19.611 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.611 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.611 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2023-07-02T14:05:19.612 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 1 2023-07-02T14:05:19.612 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:19.613 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:05:19.679 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:19.678+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:19.736 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:19.736 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:19.737 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:19.737 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:19.737 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:19.746 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:05:19.780 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:19.779+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:19.873 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:19.873 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:19.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:19.874 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:19.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.874 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:14228 2.7 2:fe1e2477:::rbd_data.125de1449605.0000000000000161:head [write 65536~3145728] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e142)", 2023-07-02T14:05:19.874 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:15.078543+0000", 2023-07-02T14:05:19.874 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.7917958919999997, 2023-07-02T14:05:19.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015849022000000001, 2023-07-02T14:05:19.874 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:19.875 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:19.875 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:19.875 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:19.875 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:19.875 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 14228 2023-07-02T14:05:19.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.875 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:19.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:19.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.078543+0000", 2023-07-02T14:05:19.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:19.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:19.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.078543+0000", 2023-07-02T14:05:19.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:19.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:19.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.078542+0000", 2023-07-02T14:05:19.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00446722 2023-07-02T14:05:19.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:19.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.083009+0000", 2023-07-02T14:05:19.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3627e-05 2023-07-02T14:05:19.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:19.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.083023+0000", 2023-07-02T14:05:19.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0939999999999996e-06 2023-07-02T14:05:19.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:19.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.083029+0000", 2023-07-02T14:05:19.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5158999999999998e-05 2023-07-02T14:05:19.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:19.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.083084+0000", 2023-07-02T14:05:19.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012269399999999999 2023-07-02T14:05:19.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:19.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.083207+0000", 2023-07-02T14:05:19.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019189700000000001 2023-07-02T14:05:19.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:19.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.083399+0000", 2023-07-02T14:05:19.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069438989999999999 2023-07-02T14:05:19.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:19.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.090343+0000", 2023-07-02T14:05:19.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039087230000000002 2023-07-02T14:05:19.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:19.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.094251+0000", 2023-07-02T14:05:19.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0945000000000005e-05 2023-07-02T14:05:19.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:19.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.094332+0000", 2023-07-02T14:05:19.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9976000000000003e-05 2023-07-02T14:05:19.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:19.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.094392+0000", 2023-07-02T14:05:19.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015849022000000001 2023-07-02T14:05:19.884 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.884 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:19.885 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.885 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:14250 2.7 2:e90f6b02:::rbd_data.125de1449605.0000000000000166:head [write 65536~4128768] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e142)", 2023-07-02T14:05:19.885 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:15.107073+0000", 2023-07-02T14:05:19.885 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.7632658599999997, 2023-07-02T14:05:19.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016793801000000001, 2023-07-02T14:05:19.885 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:19.886 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:19.886 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:19.886 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:19.886 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:19.886 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 14250 2023-07-02T14:05:19.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.887 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:19.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:19.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.107073+0000", 2023-07-02T14:05:19.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:19.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:19.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.107073+0000", 2023-07-02T14:05:19.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:19.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:19.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.107072+0000", 2023-07-02T14:05:19.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050371499999999998 2023-07-02T14:05:19.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:19.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.112109+0000", 2023-07-02T14:05:19.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3864e-05 2023-07-02T14:05:19.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:19.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.112123+0000", 2023-07-02T14:05:19.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0278999999999999e-05 2023-07-02T14:05:19.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:19.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.112133+0000", 2023-07-02T14:05:19.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5973000000000001e-05 2023-07-02T14:05:19.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.893 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:19.893 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.112189+0000", 2023-07-02T14:05:19.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011774500000000001 2023-07-02T14:05:19.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.893 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:19.893 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.112307+0000", 2023-07-02T14:05:19.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020514799999999999 2023-07-02T14:05:19.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:19.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.112512+0000", 2023-07-02T14:05:19.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064637169999999999 2023-07-02T14:05:19.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:19.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.118976+0000", 2023-07-02T14:05:19.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047710929999999997 2023-07-02T14:05:19.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:19.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.123747+0000", 2023-07-02T14:05:19.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0846999999999994e-05 2023-07-02T14:05:19.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:19.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.123817+0000", 2023-07-02T14:05:19.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9518000000000003e-05 2023-07-02T14:05:19.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:19.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.123867+0000", 2023-07-02T14:05:19.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016793801000000001 2023-07-02T14:05:19.897 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.897 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:19.898 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.898 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:14257 2.5 2:ae059f7e:::rbd_data.125de1449605.0000000000000167:head [write 65536~4128768] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e143)", 2023-07-02T14:05:19.898 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:15.127465+0000", 2023-07-02T14:05:19.898 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.7428741719999996, 2023-07-02T14:05:19.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019550754, 2023-07-02T14:05:19.899 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:19.899 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:19.899 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:19.899 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:19.899 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:19.899 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 14257 2023-07-02T14:05:19.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.899 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:19.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:19.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.127465+0000", 2023-07-02T14:05:19.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:19.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:19.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.127465+0000", 2023-07-02T14:05:19.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:19.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:19.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.127463+0000", 2023-07-02T14:05:19.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046727089999999997 2023-07-02T14:05:19.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:19.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.132136+0000", 2023-07-02T14:05:19.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5492999999999999e-05 2023-07-02T14:05:19.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:19.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.132152+0000", 2023-07-02T14:05:19.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9029999999999993e-06 2023-07-02T14:05:19.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:19.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.132160+0000", 2023-07-02T14:05:19.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5988e-05 2023-07-02T14:05:19.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:19.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.132216+0000", 2023-07-02T14:05:19.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012380400000000001 2023-07-02T14:05:19.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:19.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.132340+0000", 2023-07-02T14:05:19.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000205157 2023-07-02T14:05:19.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:19.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.132545+0000", 2023-07-02T14:05:19.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065451229999999999 2023-07-02T14:05:19.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:19.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.139090+0000", 2023-07-02T14:05:19.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078566690000000002 2023-07-02T14:05:19.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:19.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.146947+0000", 2023-07-02T14:05:19.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0538999999999999e-05 2023-07-02T14:05:19.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:19.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.146988+0000", 2023-07-02T14:05:19.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.7925000000000001e-05 2023-07-02T14:05:19.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:19.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:15.147016+0000", 2023-07-02T14:05:19.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019550754 2023-07-02T14:05:19.908 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.908 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:19.908 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.909 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15698 2.5 2:bddac802:::rbd_data.125de1449605.00000000000000e6:head [write 524288~3670016] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:19.909 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.458647+0000", 2023-07-02T14:05:19.909 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.41169170199999999, 2023-07-02T14:05:19.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025272010000000001, 2023-07-02T14:05:19.909 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:19.909 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:19.909 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:19.909 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:19.910 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:19.910 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15698 2023-07-02T14:05:19.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.910 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:19.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:19.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.458647+0000", 2023-07-02T14:05:19.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:19.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:19.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.458647+0000", 2023-07-02T14:05:19.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:19.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:19.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.458646+0000", 2023-07-02T14:05:19.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045542569999999999 2023-07-02T14:05:19.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:19.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463201+0000", 2023-07-02T14:05:19.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8629999999999997e-06 2023-07-02T14:05:19.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:19.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463210+0000", 2023-07-02T14:05:19.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.515e-06 2023-07-02T14:05:19.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:19.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463214+0000", 2023-07-02T14:05:19.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013320000000000001 2023-07-02T14:05:19.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:19.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463347+0000", 2023-07-02T14:05:19.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016315500000000001 2023-07-02T14:05:19.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:19.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463510+0000", 2023-07-02T14:05:19.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028797499999999999 2023-07-02T14:05:19.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:19.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463798+0000", 2023-07-02T14:05:19.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013969006000000001 2023-07-02T14:05:19.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:19.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.477767+0000", 2023-07-02T14:05:19.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060527000000000003 2023-07-02T14:05:19.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:19.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.483820+0000", 2023-07-02T14:05:19.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5615000000000002e-05 2023-07-02T14:05:19.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:19.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.483876+0000", 2023-07-02T14:05:19.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3639e-05 2023-07-02T14:05:19.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:19.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.483919+0000", 2023-07-02T14:05:19.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025272010000000001 2023-07-02T14:05:19.919 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.919 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:19.919 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.920 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15699 2.5 2:b034946e:::rbd_data.125de1449605.00000000000000ec:head [write 315392~12288] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:19.920 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.463220+0000", 2023-07-02T14:05:19.920 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.40711920000000001, 2023-07-02T14:05:19.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020797308, 2023-07-02T14:05:19.920 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:19.920 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:19.920 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:19.921 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:19.921 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:19.921 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15699 2023-07-02T14:05:19.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.921 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:19.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:19.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463220+0000", 2023-07-02T14:05:19.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:19.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:19.922 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463220+0000", 2023-07-02T14:05:19.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:19.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:19.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463218+0000", 2023-07-02T14:05:19.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0812999999999999e-05 2023-07-02T14:05:19.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:19.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463239+0000", 2023-07-02T14:05:19.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0017000000000001e-05 2023-07-02T14:05:19.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:19.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463249+0000", 2023-07-02T14:05:19.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9479999999999999e-06 2023-07-02T14:05:19.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:19.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463254+0000", 2023-07-02T14:05:19.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088064400000000004 2023-07-02T14:05:19.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:19.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.472061+0000", 2023-07-02T14:05:19.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00018483799999999999 2023-07-02T14:05:19.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:19.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.472245+0000", 2023-07-02T14:05:19.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00027810699999999999 2023-07-02T14:05:19.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:19.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.472524+0000", 2023-07-02T14:05:19.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011244031999999999 2023-07-02T14:05:19.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:19.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.483768+0000", 2023-07-02T14:05:19.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00018423499999999999 2023-07-02T14:05:19.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:19.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.483952+0000", 2023-07-02T14:05:19.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9762999999999999e-05 2023-07-02T14:05:19.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:19.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.483992+0000", 2023-07-02T14:05:19.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.552e-05 2023-07-02T14:05:19.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:19.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.484017+0000", 2023-07-02T14:05:19.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020797308 2023-07-02T14:05:19.930 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.930 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:19.930 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.930 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15700 2.5 2:babe4c4a:::rbd_data.125de1449605.00000000000000e7:head [write 139264~122880] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:19.930 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.463260+0000", 2023-07-02T14:05:19.930 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.407079254, 2023-07-02T14:05:19.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020898702000000002, 2023-07-02T14:05:19.931 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:19.931 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:19.931 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:19.931 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:19.931 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:19.931 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15700 2023-07-02T14:05:19.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.932 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:19.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:19.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463260+0000", 2023-07-02T14:05:19.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:19.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:19.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463260+0000", 2023-07-02T14:05:19.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:19.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:19.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463259+0000", 2023-07-02T14:05:19.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0584e-05 2023-07-02T14:05:19.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:19.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463340+0000", 2023-07-02T14:05:19.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8790000000000003e-06 2023-07-02T14:05:19.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:19.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463348+0000", 2023-07-02T14:05:19.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.501e-06 2023-07-02T14:05:19.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:19.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463351+0000", 2023-07-02T14:05:19.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013077632 2023-07-02T14:05:19.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:19.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.476429+0000", 2023-07-02T14:05:19.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010442699999999999 2023-07-02T14:05:19.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:19.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.476533+0000", 2023-07-02T14:05:19.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013782000000000001 2023-07-02T14:05:19.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:19.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.476671+0000", 2023-07-02T14:05:19.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073543109999999997 2023-07-02T14:05:19.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:19.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.484025+0000", 2023-07-02T14:05:19.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3872999999999999e-05 2023-07-02T14:05:19.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:19.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.484059+0000", 2023-07-02T14:05:19.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5229999999999998e-05 2023-07-02T14:05:19.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:19.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.484114+0000", 2023-07-02T14:05:19.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3986999999999997e-05 2023-07-02T14:05:19.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:19.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.484158+0000", 2023-07-02T14:05:19.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020898702000000002 2023-07-02T14:05:19.940 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.940 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:19.940 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.941 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15701 2.5 2:babe4c4a:::rbd_data.125de1449605.00000000000000e7:head [write 524288~3670016] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:19.941 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.463355+0000", 2023-07-02T14:05:19.941 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.40698418600000003, 2023-07-02T14:05:19.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030948784, 2023-07-02T14:05:19.941 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:19.941 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:19.941 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:19.941 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:19.942 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:19.942 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15701 2023-07-02T14:05:19.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.942 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:19.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:19.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463355+0000", 2023-07-02T14:05:19.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:19.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:19.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463355+0000", 2023-07-02T14:05:19.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:19.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:19.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463354+0000", 2023-07-02T14:05:19.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005426246 2023-07-02T14:05:19.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:19.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.468781+0000", 2023-07-02T14:05:19.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6320000000000003e-06 2023-07-02T14:05:19.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:19.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.468790+0000", 2023-07-02T14:05:19.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8080000000000004e-06 2023-07-02T14:05:19.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:19.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.468795+0000", 2023-07-02T14:05:19.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089891880000000004 2023-07-02T14:05:19.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:19.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.477784+0000", 2023-07-02T14:05:19.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7979000000000002e-05 2023-07-02T14:05:19.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:19.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.477842+0000", 2023-07-02T14:05:19.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000112236 2023-07-02T14:05:19.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:19.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.477954+0000", 2023-07-02T14:05:19.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0085239600000000006 2023-07-02T14:05:19.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:19.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.486478+0000", 2023-07-02T14:05:19.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077413229999999996 2023-07-02T14:05:19.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:19.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.494220+0000", 2023-07-02T14:05:19.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3022999999999998e-05 2023-07-02T14:05:19.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:19.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.494263+0000", 2023-07-02T14:05:19.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0862000000000003e-05 2023-07-02T14:05:19.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:19.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.494304+0000", 2023-07-02T14:05:19.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030948784 2023-07-02T14:05:19.951 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.951 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:19.951 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.951 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15730 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:19.951 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503247+0000", 2023-07-02T14:05:19.951 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.36709198500000001, 2023-07-02T14:05:19.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024994411000000001, 2023-07-02T14:05:19.952 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:19.952 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:19.952 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:19.952 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:19.952 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:19.952 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15730 2023-07-02T14:05:19.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.953 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:19.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:19.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503247+0000", 2023-07-02T14:05:19.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:19.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:19.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503247+0000", 2023-07-02T14:05:19.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:19.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.954 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:19.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503246+0000", 2023-07-02T14:05:19.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6199999999999998e-06 2023-07-02T14:05:19.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:19.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503251+0000", 2023-07-02T14:05:19.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4030000000000007e-06 2023-07-02T14:05:19.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:19.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503260+0000", 2023-07-02T14:05:19.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0599999999999998e-06 2023-07-02T14:05:19.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:19.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503265+0000", 2023-07-02T14:05:19.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2533000000000001e-05 2023-07-02T14:05:19.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:19.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503307+0000", 2023-07-02T14:05:19.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2738999999999994e-05 2023-07-02T14:05:19.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:19.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503400+0000", 2023-07-02T14:05:19.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003609485 2023-07-02T14:05:19.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:19.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.507009+0000", 2023-07-02T14:05:19.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9531999999999998e-05 2023-07-02T14:05:19.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:19.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.507079+0000", 2023-07-02T14:05:19.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00041278899999999999 2023-07-02T14:05:19.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:19.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.507492+0000", 2023-07-02T14:05:19.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003118778 2023-07-02T14:05:19.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:19.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.510610+0000", 2023-07-02T14:05:19.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017377068999999998 2023-07-02T14:05:19.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:19.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.527987+0000", 2023-07-02T14:05:19.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2644000000000005e-05 2023-07-02T14:05:19.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:19.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.528050+0000", 2023-07-02T14:05:19.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019132400000000001 2023-07-02T14:05:19.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:19.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.528241+0000", 2023-07-02T14:05:19.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024994411000000001 2023-07-02T14:05:19.962 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.962 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:19.962 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.963 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15733 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:19.963 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503319+0000", 2023-07-02T14:05:19.963 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.36701982, 2023-07-02T14:05:19.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036594227999999999, 2023-07-02T14:05:19.963 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:19.963 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:19.964 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:19.964 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:19.964 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:19.964 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15733 2023-07-02T14:05:19.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.964 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:19.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:19.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503319+0000", 2023-07-02T14:05:19.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:19.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:19.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503319+0000", 2023-07-02T14:05:19.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:19.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:19.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503319+0000", 2023-07-02T14:05:19.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0300000000000004e-06 2023-07-02T14:05:19.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:19.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503323+0000", 2023-07-02T14:05:19.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1829e-05 2023-07-02T14:05:19.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:19.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503334+0000", 2023-07-02T14:05:19.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.511e-06 2023-07-02T14:05:19.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:19.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503341+0000", 2023-07-02T14:05:19.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7775000000000005e-05 2023-07-02T14:05:19.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:19.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503439+0000", 2023-07-02T14:05:19.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6311999999999998e-05 2023-07-02T14:05:19.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:19.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503505+0000", 2023-07-02T14:05:19.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057891940000000001 2023-07-02T14:05:19.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:19.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509294+0000", 2023-07-02T14:05:19.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0247999999999997e-05 2023-07-02T14:05:19.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:19.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509345+0000", 2023-07-02T14:05:19.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018957756999999999 2023-07-02T14:05:19.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:19.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.528302+0000", 2023-07-02T14:05:19.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5699000000000006e-05 2023-07-02T14:05:19.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:19.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.528398+0000", 2023-07-02T14:05:19.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00050004900000000002 2023-07-02T14:05:19.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:19.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.528898+0000", 2023-07-02T14:05:19.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023774949999999999 2023-07-02T14:05:19.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:19.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531276+0000", 2023-07-02T14:05:19.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084207899999999992 2023-07-02T14:05:19.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:19.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539696+0000", 2023-07-02T14:05:19.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8023000000000003e-05 2023-07-02T14:05:19.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:19.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539754+0000", 2023-07-02T14:05:19.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000159092 2023-07-02T14:05:19.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:19.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539913+0000", 2023-07-02T14:05:19.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036594227999999999 2023-07-02T14:05:19.975 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.975 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:19.975 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.976 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15734 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:19.976 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503349+0000", 2023-07-02T14:05:19.976 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.36699008500000002, 2023-07-02T14:05:19.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041221990999999999, 2023-07-02T14:05:19.976 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:19.976 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:19.977 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:19.977 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:19.977 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:19.977 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15734 2023-07-02T14:05:19.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.977 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:19.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:19.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503349+0000", 2023-07-02T14:05:19.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:19.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:19.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503349+0000", 2023-07-02T14:05:19.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:19.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:19.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503348+0000", 2023-07-02T14:05:19.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6969999999999999e-06 2023-07-02T14:05:19.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:19.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503352+0000", 2023-07-02T14:05:19.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4900000000000003e-06 2023-07-02T14:05:19.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:19.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503360+0000", 2023-07-02T14:05:19.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3630000000000004e-06 2023-07-02T14:05:19.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:19.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503364+0000", 2023-07-02T14:05:19.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017805100000000001 2023-07-02T14:05:19.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:19.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503542+0000", 2023-07-02T14:05:19.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4289999999999994e-05 2023-07-02T14:05:19.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:19.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503626+0000", 2023-07-02T14:05:19.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057627800000000003 2023-07-02T14:05:19.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:19.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509389+0000", 2023-07-02T14:05:19.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3256000000000004e-05 2023-07-02T14:05:19.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:19.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509472+0000", 2023-07-02T14:05:19.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021137590000000001 2023-07-02T14:05:19.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:19.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530610+0000", 2023-07-02T14:05:19.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7247e-05 2023-07-02T14:05:19.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:19.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530697+0000", 2023-07-02T14:05:19.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0092678020000000003 2023-07-02T14:05:19.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:19.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539965+0000", 2023-07-02T14:05:19.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8950999999999996e-05 2023-07-02T14:05:19.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:19.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.540054+0000", 2023-07-02T14:05:19.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00045742200000000003 2023-07-02T14:05:19.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:19.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.540511+0000", 2023-07-02T14:05:19.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003197653 2023-07-02T14:05:19.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:19.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.543709+0000", 2023-07-02T14:05:19.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00065262499999999999 2023-07-02T14:05:19.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:19.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.544362+0000", 2023-07-02T14:05:19.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9426000000000003e-05 2023-07-02T14:05:19.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:19.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.544421+0000", 2023-07-02T14:05:19.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014981099999999999 2023-07-02T14:05:19.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:19.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.544571+0000", 2023-07-02T14:05:19.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041221990999999999 2023-07-02T14:05:19.990 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.990 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:19.990 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:19.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.990 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15735 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:19.990 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503369+0000", 2023-07-02T14:05:19.990 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.36697046, 2023-07-02T14:05:19.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.049400686999999999, 2023-07-02T14:05:19.991 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:19.991 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:19.991 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:19.991 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:19.991 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:19.991 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15735 2023-07-02T14:05:19.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.992 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:19.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:19.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503369+0000", 2023-07-02T14:05:19.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:19.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:19.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503369+0000", 2023-07-02T14:05:19.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:19.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:19.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503368+0000", 2023-07-02T14:05:19.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3289999999999998e-06 2023-07-02T14:05:19.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:19.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503372+0000", 2023-07-02T14:05:19.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4719999999999998e-06 2023-07-02T14:05:19.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:19.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503379+0000", 2023-07-02T14:05:19.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2289999999999997e-06 2023-07-02T14:05:19.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:19.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503383+0000", 2023-07-02T14:05:19.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00030972399999999999 2023-07-02T14:05:19.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:19.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503693+0000", 2023-07-02T14:05:19.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000120279 2023-07-02T14:05:19.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:19.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503813+0000", 2023-07-02T14:05:19.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057248059999999998 2023-07-02T14:05:19.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:19.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509538+0000", 2023-07-02T14:05:19.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011721499999999999 2023-07-02T14:05:19.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:19.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509655+0000", 2023-07-02T14:05:19.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021108756999999999 2023-07-02T14:05:19.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:19.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530764+0000", 2023-07-02T14:05:19.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5428999999999997e-05 2023-07-02T14:05:19.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:19.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:19.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:19.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530859+0000", 2023-07-02T14:05:19.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010890082000000001 2023-07-02T14:05:19.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.541750+0000", 2023-07-02T14:05:20.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5916999999999997e-05 2023-07-02T14:05:20.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.541835+0000", 2023-07-02T14:05:20.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002791226 2023-07-02T14:05:20.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.544627+0000", 2023-07-02T14:05:20.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2878000000000002e-05 2023-07-02T14:05:20.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:20.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.544710+0000", 2023-07-02T14:05:20.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00044054400000000002 2023-07-02T14:05:20.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.545150+0000", 2023-07-02T14:05:20.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001421484 2023-07-02T14:05:20.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:20.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546572+0000", 2023-07-02T14:05:20.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060059900000000001 2023-07-02T14:05:20.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.552578+0000", 2023-07-02T14:05:20.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7685e-05 2023-07-02T14:05:20.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:20.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.552645+0000", 2023-07-02T14:05:20.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012397999999999999 2023-07-02T14:05:20.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:20.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.552769+0000", 2023-07-02T14:05:20.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.049400686999999999 2023-07-02T14:05:20.005 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.005 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:20.005 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.006 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15736 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:20.006 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503388+0000", 2023-07-02T14:05:20.006 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.36695123699999999, 2023-07-02T14:05:20.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.057425610000000002, 2023-07-02T14:05:20.006 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:20.006 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.007 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:20.007 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:20.007 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.007 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15736 2023-07-02T14:05:20.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.007 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:20.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:20.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503388+0000", 2023-07-02T14:05:20.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:20.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:20.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503388+0000", 2023-07-02T14:05:20.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:20.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:20.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503387+0000", 2023-07-02T14:05:20.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.382e-06 2023-07-02T14:05:20.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:20.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503391+0000", 2023-07-02T14:05:20.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7790000000000001e-06 2023-07-02T14:05:20.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:20.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503397+0000", 2023-07-02T14:05:20.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3110000000000001e-06 2023-07-02T14:05:20.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503400+0000", 2023-07-02T14:05:20.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00045478499999999998 2023-07-02T14:05:20.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503855+0000", 2023-07-02T14:05:20.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8894999999999995e-05 2023-07-02T14:05:20.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503944+0000", 2023-07-02T14:05:20.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057727849999999999 2023-07-02T14:05:20.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509716+0000", 2023-07-02T14:05:20.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.593e-05 2023-07-02T14:05:20.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509792+0000", 2023-07-02T14:05:20.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021127659 2023-07-02T14:05:20.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530920+0000", 2023-07-02T14:05:20.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0810999999999997e-05 2023-07-02T14:05:20.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530991+0000", 2023-07-02T14:05:20.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010903886999999999 2023-07-02T14:05:20.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.541895+0000", 2023-07-02T14:05:20.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4783000000000005e-05 2023-07-02T14:05:20.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.541979+0000", 2023-07-02T14:05:20.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041761189999999998 2023-07-02T14:05:20.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546155+0000", 2023-07-02T14:05:20.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2211000000000003e-05 2023-07-02T14:05:20.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546238+0000", 2023-07-02T14:05:20.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006603589 2023-07-02T14:05:20.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.552841+0000", 2023-07-02T14:05:20.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000108977 2023-07-02T14:05:20.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:20.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.552950+0000", 2023-07-02T14:05:20.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000543126 2023-07-02T14:05:20.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.553493+0000", 2023-07-02T14:05:20.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030365330000000001 2023-07-02T14:05:20.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:20.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.556530+0000", 2023-07-02T14:05:20.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041768609999999996 2023-07-02T14:05:20.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.560707+0000", 2023-07-02T14:05:20.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6613999999999999e-05 2023-07-02T14:05:20.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:20.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.560743+0000", 2023-07-02T14:05:20.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0005999999999993e-05 2023-07-02T14:05:20.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:20.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.560813+0000", 2023-07-02T14:05:20.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.057425610000000002 2023-07-02T14:05:20.022 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.022 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:20.022 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.023 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15737 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:20.023 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503667+0000", 2023-07-02T14:05:20.023 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.366672371, 2023-07-02T14:05:20.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.067804241000000001, 2023-07-02T14:05:20.023 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:20.023 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.023 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:20.024 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:20.024 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.024 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15737 2023-07-02T14:05:20.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.024 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:20.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:20.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503667+0000", 2023-07-02T14:05:20.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:20.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:20.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503667+0000", 2023-07-02T14:05:20.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:20.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:20.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503666+0000", 2023-07-02T14:05:20.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6499999999999999e-06 2023-07-02T14:05:20.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:20.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503672+0000", 2023-07-02T14:05:20.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9850000000000001e-06 2023-07-02T14:05:20.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:20.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503682+0000", 2023-07-02T14:05:20.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3580000000000002e-06 2023-07-02T14:05:20.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503688+0000", 2023-07-02T14:05:20.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00030985000000000002 2023-07-02T14:05:20.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503998+0000", 2023-07-02T14:05:20.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012238099999999999 2023-07-02T14:05:20.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.504120+0000", 2023-07-02T14:05:20.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057408620000000002 2023-07-02T14:05:20.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509861+0000", 2023-07-02T14:05:20.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2069000000000003e-05 2023-07-02T14:05:20.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509943+0000", 2023-07-02T14:05:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021112245000000002 2023-07-02T14:05:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531055+0000", 2023-07-02T14:05:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4781e-05 2023-07-02T14:05:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531140+0000", 2023-07-02T14:05:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010899282 2023-07-02T14:05:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.542039+0000", 2023-07-02T14:05:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3856999999999994e-05 2023-07-02T14:05:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.542123+0000", 2023-07-02T14:05:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041575969999999999 2023-07-02T14:05:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546281+0000", 2023-07-02T14:05:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8136000000000002e-05 2023-07-02T14:05:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546319+0000", 2023-07-02T14:05:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091989649999999999 2023-07-02T14:05:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.555518+0000", 2023-07-02T14:05:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3294000000000005e-05 2023-07-02T14:05:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.555611+0000", 2023-07-02T14:05:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052494760000000003 2023-07-02T14:05:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.560860+0000", 2023-07-02T14:05:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8725000000000002e-05 2023-07-02T14:05:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.560949+0000", 2023-07-02T14:05:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00046914100000000001 2023-07-02T14:05:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.561418+0000", 2023-07-02T14:05:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006959911 2023-07-02T14:05:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.568378+0000", 2023-07-02T14:05:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029392400000000001 2023-07-02T14:05:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.571317+0000", 2023-07-02T14:05:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8885999999999998e-05 2023-07-02T14:05:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.571376+0000", 2023-07-02T14:05:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4517000000000003e-05 2023-07-02T14:05:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.571471+0000", 2023-07-02T14:05:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.067804241000000001 2023-07-02T14:05:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15738 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503693+0000", 2023-07-02T14:05:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.36664584, 2023-07-02T14:05:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.080315304000000004, 2023-07-02T14:05:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15738 2023-07-02T14:05:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503693+0000", 2023-07-02T14:05:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503693+0000", 2023-07-02T14:05:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503693+0000", 2023-07-02T14:05:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3620000000000002e-06 2023-07-02T14:05:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503696+0000", 2023-07-02T14:05:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2850000000000002e-06 2023-07-02T14:05:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503701+0000", 2023-07-02T14:05:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.938e-06 2023-07-02T14:05:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503706+0000", 2023-07-02T14:05:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000447105 2023-07-02T14:05:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.504153+0000", 2023-07-02T14:05:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011312 2023-07-02T14:05:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.504267+0000", 2023-07-02T14:05:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057292469999999998 2023-07-02T14:05:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509996+0000", 2023-07-02T14:05:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7124000000000002e-05 2023-07-02T14:05:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.510073+0000", 2023-07-02T14:05:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021118604999999999 2023-07-02T14:05:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531192+0000", 2023-07-02T14:05:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4815999999999997e-05 2023-07-02T14:05:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531266+0000", 2023-07-02T14:05:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010915915 2023-07-02T14:05:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.542182+0000", 2023-07-02T14:05:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5304999999999998e-05 2023-07-02T14:05:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.542258+0000", 2023-07-02T14:05:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041042539999999999 2023-07-02T14:05:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546362+0000", 2023-07-02T14:05:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0340999999999995e-05 2023-07-02T14:05:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546442+0000", 2023-07-02T14:05:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0092239639999999994 2023-07-02T14:05:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.555666+0000", 2023-07-02T14:05:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0080999999999995e-05 2023-07-02T14:05:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.555736+0000", 2023-07-02T14:05:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069964290000000002 2023-07-02T14:05:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562733+0000", 2023-07-02T14:05:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7447000000000005e-05 2023-07-02T14:05:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562820+0000", 2023-07-02T14:05:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086771179999999993 2023-07-02T14:05:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.571497+0000", 2023-07-02T14:05:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7438000000000002e-05 2023-07-02T14:05:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.571545+0000", 2023-07-02T14:05:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000252651 2023-07-02T14:05:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.571797+0000", 2023-07-02T14:05:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001648369 2023-07-02T14:05:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.573446+0000", 2023-07-02T14:05:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010442987000000001 2023-07-02T14:05:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.583889+0000", 2023-07-02T14:05:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9243000000000002e-05 2023-07-02T14:05:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.583948+0000", 2023-07-02T14:05:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0590000000000001e-05 2023-07-02T14:05:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.584008+0000", 2023-07-02T14:05:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.080315304000000004 2023-07-02T14:05:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15739 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503710+0000", 2023-07-02T14:05:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.36662890300000001, 2023-07-02T14:05:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.084428051000000004, 2023-07-02T14:05:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15739 2023-07-02T14:05:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503710+0000", 2023-07-02T14:05:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503710+0000", 2023-07-02T14:05:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503710+0000", 2023-07-02T14:05:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9919999999999999e-06 2023-07-02T14:05:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503713+0000", 2023-07-02T14:05:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8439999999999997e-06 2023-07-02T14:05:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503718+0000", 2023-07-02T14:05:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.58e-06 2023-07-02T14:05:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503721+0000", 2023-07-02T14:05:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00059269800000000001 2023-07-02T14:05:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.504314+0000", 2023-07-02T14:05:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000121972 2023-07-02T14:05:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.504436+0000", 2023-07-02T14:05:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056719500000000003 2023-07-02T14:05:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.510108+0000", 2023-07-02T14:05:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8212000000000003e-05 2023-07-02T14:05:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.510186+0000", 2023-07-02T14:05:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021121849000000002 2023-07-02T14:05:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531308+0000", 2023-07-02T14:05:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2182999999999997e-05 2023-07-02T14:05:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531380+0000", 2023-07-02T14:05:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010927773999999999 2023-07-02T14:05:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.542308+0000", 2023-07-02T14:05:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7233999999999999e-05 2023-07-02T14:05:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.542385+0000", 2023-07-02T14:05:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041062779999999997 2023-07-02T14:05:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546491+0000", 2023-07-02T14:05:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7290000000000001e-05 2023-07-02T14:05:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546559+0000", 2023-07-02T14:05:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0092123529999999995 2023-07-02T14:05:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.555771+0000", 2023-07-02T14:05:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2665999999999998e-05 2023-07-02T14:05:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.555844+0000", 2023-07-02T14:05:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070310900000000003 2023-07-02T14:05:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562875+0000", 2023-07-02T14:05:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6711000000000001e-05 2023-07-02T14:05:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562921+0000", 2023-07-02T14:05:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010083190000000001 2023-07-02T14:05:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.573005+0000", 2023-07-02T14:05:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7387999999999997e-05 2023-07-02T14:05:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.573092+0000", 2023-07-02T14:05:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010965232 2023-07-02T14:05:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.584057+0000", 2023-07-02T14:05:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8296999999999998e-05 2023-07-02T14:05:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.584156+0000", 2023-07-02T14:05:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00048002599999999999 2023-07-02T14:05:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.584636+0000", 2023-07-02T14:05:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024421500000000001 2023-07-02T14:05:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.587078+0000", 2023-07-02T14:05:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000929487 2023-07-02T14:05:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.588007+0000", 2023-07-02T14:05:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0000999999999996e-05 2023-07-02T14:05:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.588077+0000", 2023-07-02T14:05:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0915999999999997e-05 2023-07-02T14:05:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.588138+0000", 2023-07-02T14:05:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.084428051000000004 2023-07-02T14:05:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15740 2.0 2:08f47745:::rbd_data.125de1449605.00000000000000eb:head [write 49152~4079616] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.507724+0000", 2023-07-02T14:05:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.36261520699999999, 2023-07-02T14:05:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031745934000000003, 2023-07-02T14:05:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15740 2023-07-02T14:05:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.507724+0000", 2023-07-02T14:05:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.507724+0000", 2023-07-02T14:05:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.507722+0000", 2023-07-02T14:05:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079849010000000008 2023-07-02T14:05:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515707+0000", 2023-07-02T14:05:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4861999999999999e-05 2023-07-02T14:05:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515722+0000", 2023-07-02T14:05:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7649999999999996e-06 2023-07-02T14:05:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515729+0000", 2023-07-02T14:05:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3593000000000001e-05 2023-07-02T14:05:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515772+0000", 2023-07-02T14:05:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011619499999999999 2023-07-02T14:05:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515888+0000", 2023-07-02T14:05:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020645599999999999 2023-07-02T14:05:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.516095+0000", 2023-07-02T14:05:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014613513999999999 2023-07-02T14:05:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530708+0000", 2023-07-02T14:05:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086383950000000001 2023-07-02T14:05:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539347+0000", 2023-07-02T14:05:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7710000000000001e-05 2023-07-02T14:05:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539414+0000", 2023-07-02T14:05:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5368999999999998e-05 2023-07-02T14:05:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539470+0000", 2023-07-02T14:05:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031745934000000003 2023-07-02T14:05:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15742 2.0 2:08f47745:::rbd_data.125de1449605.00000000000000eb:head [write 4128768~20480] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.515741+0000", 2023-07-02T14:05:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.35459850300000001, 2023-07-02T14:05:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023907014000000001, 2023-07-02T14:05:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15742 2023-07-02T14:05:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515741+0000", 2023-07-02T14:05:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515741+0000", 2023-07-02T14:05:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515739+0000", 2023-07-02T14:05:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1392000000000001e-05 2023-07-02T14:05:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515761+0000", 2023-07-02T14:05:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.588e-06 2023-07-02T14:05:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515768+0000", 2023-07-02T14:05:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.481e-06 2023-07-02T14:05:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515773+0000", 2023-07-02T14:05:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098114889999999996 2023-07-02T14:05:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.525584+0000", 2023-07-02T14:05:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012770899999999999 2023-07-02T14:05:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.525712+0000", 2023-07-02T14:05:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021492000000000001 2023-07-02T14:05:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.525927+0000", 2023-07-02T14:05:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050735060000000002 2023-07-02T14:05:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531001+0000", 2023-07-02T14:05:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0085373589999999996 2023-07-02T14:05:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539538+0000", 2023-07-02T14:05:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1940999999999995e-05 2023-07-02T14:05:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539600+0000", 2023-07-02T14:05:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7713000000000002e-05 2023-07-02T14:05:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539648+0000", 2023-07-02T14:05:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023907014000000001 2023-07-02T14:05:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15761 2.0 2:1c81f085:::rbd_data.125de1449605.00000000000000f0:head [write 491520~2129920] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.557597+0000", 2023-07-02T14:05:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.31274164799999998, 2023-07-02T14:05:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024902957999999999, 2023-07-02T14:05:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15761 2023-07-02T14:05:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.557597+0000", 2023-07-02T14:05:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.557597+0000", 2023-07-02T14:05:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.557597+0000", 2023-07-02T14:05:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046213230000000001 2023-07-02T14:05:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562218+0000", 2023-07-02T14:05:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0871e-05 2023-07-02T14:05:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562229+0000", 2023-07-02T14:05:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9510000000000003e-06 2023-07-02T14:05:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562235+0000", 2023-07-02T14:05:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00071883400000000001 2023-07-02T14:05:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562954+0000", 2023-07-02T14:05:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6080000000000002e-05 2023-07-02T14:05:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.563050+0000", 2023-07-02T14:05:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020347299999999999 2023-07-02T14:05:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.563253+0000", 2023-07-02T14:05:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065830150000000002 2023-07-02T14:05:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.569836+0000", 2023-07-02T14:05:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012531054 2023-07-02T14:05:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.582367+0000", 2023-07-02T14:05:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2144000000000006e-05 2023-07-02T14:05:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.582439+0000", 2023-07-02T14:05:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1063999999999997e-05 2023-07-02T14:05:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.582500+0000", 2023-07-02T14:05:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024902957999999999 2023-07-02T14:05:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15762 2.0 2:1c81f085:::rbd_data.125de1449605.00000000000000f0:head [write 2662400~24576] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.562241+0000", 2023-07-02T14:05:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.308098486, 2023-07-02T14:05:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020446751999999999, 2023-07-02T14:05:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15762 2023-07-02T14:05:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562241+0000", 2023-07-02T14:05:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562241+0000", 2023-07-02T14:05:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562240+0000", 2023-07-02T14:05:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6622999999999997e-05 2023-07-02T14:05:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562286+0000", 2023-07-02T14:05:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5429999999999994e-06 2023-07-02T14:05:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562295+0000", 2023-07-02T14:05:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2559999999999998e-06 2023-07-02T14:05:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562298+0000", 2023-07-02T14:05:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046232529999999999 2023-07-02T14:05:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.566921+0000", 2023-07-02T14:05:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011134800000000001 2023-07-02T14:05:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.567033+0000", 2023-07-02T14:05:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000214391 2023-07-02T14:05:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.567247+0000", 2023-07-02T14:05:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025927200000000002 2023-07-02T14:05:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.569840+0000", 2023-07-02T14:05:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012720309000000001 2023-07-02T14:05:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.582560+0000", 2023-07-02T14:05:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4462999999999996e-05 2023-07-02T14:05:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.582625+0000", 2023-07-02T14:05:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2766000000000002e-05 2023-07-02T14:05:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.582687+0000", 2023-07-02T14:05:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020446751999999999 2023-07-02T14:05:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15763 2.0 2:1c81f085:::rbd_data.125de1449605.00000000000000f0:head [write 2621440~40960] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.562302+0000", 2023-07-02T14:05:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.30803751499999998, 2023-07-02T14:05:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020546182, 2023-07-02T14:05:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15763 2023-07-02T14:05:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562302+0000", 2023-07-02T14:05:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562302+0000", 2023-07-02T14:05:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562301+0000", 2023-07-02T14:05:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.483e-05 2023-07-02T14:05:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562316+0000", 2023-07-02T14:05:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2349999999999999e-06 2023-07-02T14:05:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562321+0000", 2023-07-02T14:05:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6570000000000001e-06 2023-07-02T14:05:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562324+0000", 2023-07-02T14:05:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062584199999999998 2023-07-02T14:05:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.568582+0000", 2023-07-02T14:05:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.114e-05 2023-07-02T14:05:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.568653+0000", 2023-07-02T14:05:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115469 2023-07-02T14:05:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.568769+0000", 2023-07-02T14:05:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0015712600000000001 2023-07-02T14:05:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.570340+0000", 2023-07-02T14:05:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012404179 2023-07-02T14:05:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.582744+0000", 2023-07-02T14:05:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9110000000000002e-05 2023-07-02T14:05:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.582803+0000", 2023-07-02T14:05:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4283999999999998e-05 2023-07-02T14:05:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.582848+0000", 2023-07-02T14:05:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020546182 2023-07-02T14:05:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.135 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:20.135 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.135 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:20.135 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:20.135 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:20.136 INFO:tasks.workunit.client.0.smithi163.stderr:+ sync b3fbb028-e267-4d25-a750-93ba18a2316a-mnt 2023-07-02T14:05:20.137 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:20.135+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:20.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:05:20.141 INFO:tasks.workunit.client.0.smithi163.stderr:+ rm -rf b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4 b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4.tar.gz 2023-07-02T14:05:20.175 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:05:20.237 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:20.237+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:20.263 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:20.264 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:20.264 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:05:20.264 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:20.276 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:05:20.338 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:20.337+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:20.402 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:20.402 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:20.402 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:05:20.402 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:05:20.402 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:20.413 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:05:20.454 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:20.453+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:20.539 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:20.538+0000 7f121d370700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83347) UID: 0 2023-07-02T14:05:20.579 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:20.580 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:05:20.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:05:20.580 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:20.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.580 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2620 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 196608~786432] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:20.581 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:05:20.581 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 130.253233722, 2023-07-02T14:05:20.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966, 2023-07-02T14:05:20.581 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.581 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.581 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.581 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:20.581 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.582 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2620 2023-07-02T14:05:20.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.582 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:05:20.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324573+0000", 2023-07-02T14:05:20.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:20.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.324571+0000", 2023-07-02T14:05:20.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016686649999999999 2023-07-02T14:05:20.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326240+0000", 2023-07-02T14:05:20.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1844e-05 2023-07-02T14:05:20.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326252+0000", 2023-07-02T14:05:20.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7760000000000003e-06 2023-07-02T14:05:20.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326258+0000", 2023-07-02T14:05:20.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0634000000000003e-05 2023-07-02T14:05:20.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326318+0000", 2023-07-02T14:05:20.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9629999999999993e-06 2023-07-02T14:05:20.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:20.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326328+0000", 2023-07-02T14:05:20.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.010877045 2023-07-02T14:05:20.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337205+0000", 2023-07-02T14:05:20.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022401300000000001 2023-07-02T14:05:20.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337429+0000", 2023-07-02T14:05:20.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000139822 2023-07-02T14:05:20.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.337569+0000", 2023-07-02T14:05:20.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032942119999999999 2023-07-02T14:05:20.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340863+0000", 2023-07-02T14:05:20.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021302560000000001 2023-07-02T14:05:20.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:20.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342994+0000", 2023-07-02T14:05:20.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5634000000000002e-05 2023-07-02T14:05:20.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:20.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343059+0000", 2023-07-02T14:05:20.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143721 2023-07-02T14:05:20.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:20.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343203+0000", 2023-07-02T14:05:20.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.018629966 2023-07-02T14:05:20.592 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.592 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.592 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.592 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2622 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 983040~36864] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:20.593 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:05:20.593 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 130.25154121099999, 2023-07-02T14:05:20.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664, 2023-07-02T14:05:20.593 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.593 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.593 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.593 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:20.593 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.594 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2622 2023-07-02T14:05:20.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.594 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:05:20.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326266+0000", 2023-07-02T14:05:20.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:20.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326265+0000", 2023-07-02T14:05:20.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1282000000000001e-05 2023-07-02T14:05:20.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326286+0000", 2023-07-02T14:05:20.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9480000000000002e-06 2023-07-02T14:05:20.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326293+0000", 2023-07-02T14:05:20.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3610000000000003e-06 2023-07-02T14:05:20.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326297+0000", 2023-07-02T14:05:20.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7707999999999994e-05 2023-07-02T14:05:20.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326385+0000", 2023-07-02T14:05:20.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8070000000000006e-06 2023-07-02T14:05:20.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:20.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326394+0000", 2023-07-02T14:05:20.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.014535878 2023-07-02T14:05:20.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.340930+0000", 2023-07-02T14:05:20.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000154551 2023-07-02T14:05:20.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341084+0000", 2023-07-02T14:05:20.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00032375900000000002 2023-07-02T14:05:20.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.341408+0000", 2023-07-02T14:05:20.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012631066 2023-07-02T14:05:20.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354039+0000", 2023-07-02T14:05:20.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5741000000000001e-05 2023-07-02T14:05:20.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:20.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354055+0000", 2023-07-02T14:05:20.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8794000000000006e-05 2023-07-02T14:05:20.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:20.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354124+0000", 2023-07-02T14:05:20.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014667700000000001 2023-07-02T14:05:20.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:20.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354270+0000", 2023-07-02T14:05:20.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.028004664 2023-07-02T14:05:20.603 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.604 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.604 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.604 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2727 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~12288] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:20.604 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:05:20.604 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 130.251505436, 2023-07-02T14:05:20.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999, 2023-07-02T14:05:20.605 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.605 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.605 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.605 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:20.605 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.605 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2727 2023-07-02T14:05:20.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.606 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:05:20.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:05:20.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:20.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326301+0000", 2023-07-02T14:05:20.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3834e-05 2023-07-02T14:05:20.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326315+0000", 2023-07-02T14:05:20.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3029999999999996e-06 2023-07-02T14:05:20.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326321+0000", 2023-07-02T14:05:20.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7590000000000002e-06 2023-07-02T14:05:20.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326324+0000", 2023-07-02T14:05:20.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112872 2023-07-02T14:05:20.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326437+0000", 2023-07-02T14:05:20.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2290000000000002e-06 2023-07-02T14:05:20.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:20.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326442+0000", 2023-07-02T14:05:20.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0155896879999999 2023-07-02T14:05:20.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342031+0000", 2023-07-02T14:05:20.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016496100000000001 2023-07-02T14:05:20.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342196+0000", 2023-07-02T14:05:20.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021998699999999999 2023-07-02T14:05:20.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.342416+0000", 2023-07-02T14:05:20.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011869744999999999 2023-07-02T14:05:20.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:20.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354286+0000", 2023-07-02T14:05:20.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0361e-05 2023-07-02T14:05:20.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354347+0000", 2023-07-02T14:05:20.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3312000000000005e-05 2023-07-02T14:05:20.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:20.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354430+0000", 2023-07-02T14:05:20.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000126444 2023-07-02T14:05:20.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:20.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354556+0000", 2023-07-02T14:05:20.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0282548949999999 2023-07-02T14:05:20.615 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.615 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.615 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.616 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2730 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [set-alloc-hint object_size 4194304 write_size 4194304,write 454656~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:20.616 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:05:20.616 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 130.25148042699999, 2023-07-02T14:05:20.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999, 2023-07-02T14:05:20.616 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.616 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:20.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.617 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2730 2023-07-02T14:05:20.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.617 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:05:20.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:05:20.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:20.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326326+0000", 2023-07-02T14:05:20.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6049999999999997e-06 2023-07-02T14:05:20.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326332+0000", 2023-07-02T14:05:20.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3000000000000002e-06 2023-07-02T14:05:20.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326335+0000", 2023-07-02T14:05:20.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.719e-05 2023-07-02T14:05:20.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326352+0000", 2023-07-02T14:05:20.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011583599999999999 2023-07-02T14:05:20.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326468+0000", 2023-07-02T14:05:20.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.831e-06 2023-07-02T14:05:20.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:20.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326473+0000", 2023-07-02T14:05:20.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.016769915 2023-07-02T14:05:20.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343243+0000", 2023-07-02T14:05:20.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197352 2023-07-02T14:05:20.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343440+0000", 2023-07-02T14:05:20.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000197581 2023-07-02T14:05:20.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.343638+0000", 2023-07-02T14:05:20.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010980139 2023-07-02T14:05:20.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354618+0000", 2023-07-02T14:05:20.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056779530000000003 2023-07-02T14:05:20.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:20.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360296+0000", 2023-07-02T14:05:20.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000101109 2023-07-02T14:05:20.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:20.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360397+0000", 2023-07-02T14:05:20.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000160273 2023-07-02T14:05:20.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:20.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360557+0000", 2023-07-02T14:05:20.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0342307799999999 2023-07-02T14:05:20.627 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.627 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.627 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.627 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2752 2.4 2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head [write 1019904~3174400] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:20.628 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:05:20.628 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 130.251434203, 2023-07-02T14:05:20.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001, 2023-07-02T14:05:20.628 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.628 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.628 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.628 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:20.628 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.629 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2752 2023-07-02T14:05:20.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.629 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:05:20.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326373+0000", 2023-07-02T14:05:20.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:20.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.326372+0000", 2023-07-02T14:05:20.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052050380000000004 2023-07-02T14:05:20.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331577+0000", 2023-07-02T14:05:20.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6832000000000001e-05 2023-07-02T14:05:20.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331594+0000", 2023-07-02T14:05:20.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8220000000000007e-06 2023-07-02T14:05:20.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331602+0000", 2023-07-02T14:05:20.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9895999999999999e-05 2023-07-02T14:05:20.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331642+0000", 2023-07-02T14:05:20.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:05:20.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:20.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331649+0000", 2023-07-02T14:05:20.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0139141709999999 2023-07-02T14:05:20.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345563+0000", 2023-07-02T14:05:20.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113817 2023-07-02T14:05:20.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345677+0000", 2023-07-02T14:05:20.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018803400000000001 2023-07-02T14:05:20.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.345865+0000", 2023-07-02T14:05:20.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087950359999999991 2023-07-02T14:05:20.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354660+0000", 2023-07-02T14:05:20.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0059116669999999998 2023-07-02T14:05:20.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:20.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360572+0000", 2023-07-02T14:05:20.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0992000000000002e-05 2023-07-02T14:05:20.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:20.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360633+0000", 2023-07-02T14:05:20.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013601600000000001 2023-07-02T14:05:20.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:20.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360769+0000", 2023-07-02T14:05:20.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0343964370000001 2023-07-02T14:05:20.638 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.639 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.639 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.639 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2759 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 0~454656] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:20.639 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:05:20.639 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 130.246194406, 2023-07-02T14:05:20.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999, 2023-07-02T14:05:20.640 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.640 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.640 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.640 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:20.640 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.640 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2759 2023-07-02T14:05:20.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.641 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:05:20.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331612+0000", 2023-07-02T14:05:20.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:20.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.331611+0000", 2023-07-02T14:05:20.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00053795999999999998 2023-07-02T14:05:20.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332149+0000", 2023-07-02T14:05:20.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4970000000000007e-06 2023-07-02T14:05:20.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332158+0000", 2023-07-02T14:05:20.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:05:20.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332162+0000", 2023-07-02T14:05:20.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3526000000000001e-05 2023-07-02T14:05:20.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332185+0000", 2023-07-02T14:05:20.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7050000000000001e-06 2023-07-02T14:05:20.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:20.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332189+0000", 2023-07-02T14:05:20.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0186295910000001 2023-07-02T14:05:20.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350819+0000", 2023-07-02T14:05:20.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000111123 2023-07-02T14:05:20.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.350930+0000", 2023-07-02T14:05:20.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019575300000000001 2023-07-02T14:05:20.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.351126+0000", 2023-07-02T14:05:20.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011503298 2023-07-02T14:05:20.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:20.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362629+0000", 2023-07-02T14:05:20.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0085443989999999994 2023-07-02T14:05:20.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371173+0000", 2023-07-02T14:05:20.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3547999999999995e-05 2023-07-02T14:05:20.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:20.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371237+0000", 2023-07-02T14:05:20.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013790300000000001 2023-07-02T14:05:20.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:20.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371375+0000", 2023-07-02T14:05:20.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0397624009999999 2023-07-02T14:05:20.650 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.650 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.651 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.651 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2760 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 458752~131072] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:20.651 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:05:20.651 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 130.24563822100001, 2023-07-02T14:05:20.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001, 2023-07-02T14:05:20.651 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.652 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.652 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.652 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:20.652 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.652 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2760 2023-07-02T14:05:20.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.652 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:05:20.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332169+0000", 2023-07-02T14:05:20.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:20.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332168+0000", 2023-07-02T14:05:20.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013281100000000001 2023-07-02T14:05:20.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332301+0000", 2023-07-02T14:05:20.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7389999999999999e-06 2023-07-02T14:05:20.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332308+0000", 2023-07-02T14:05:20.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0639999999999998e-06 2023-07-02T14:05:20.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332312+0000", 2023-07-02T14:05:20.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1735000000000001e-05 2023-07-02T14:05:20.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332333+0000", 2023-07-02T14:05:20.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8342000000000002e-05 2023-07-02T14:05:20.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:20.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332352+0000", 2023-07-02T14:05:20.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02234782 2023-07-02T14:05:20.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354699+0000", 2023-07-02T14:05:20.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110371 2023-07-02T14:05:20.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354810+0000", 2023-07-02T14:05:20.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022174900000000001 2023-07-02T14:05:20.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355032+0000", 2023-07-02T14:05:20.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076053329999999997 2023-07-02T14:05:20.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:20.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362637+0000", 2023-07-02T14:05:20.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087913609999999993 2023-07-02T14:05:20.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371428+0000", 2023-07-02T14:05:20.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8532999999999998e-05 2023-07-02T14:05:20.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:20.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371477+0000", 2023-07-02T14:05:20.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010990800000000001 2023-07-02T14:05:20.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:20.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371587+0000", 2023-07-02T14:05:20.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0394180820000001 2023-07-02T14:05:20.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.662 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.663 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2766 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 589824~45056] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:20.663 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:05:20.663 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 130.245489596, 2023-07-02T14:05:20.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188, 2023-07-02T14:05:20.663 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.663 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.664 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.664 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:20.664 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.664 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2766 2023-07-02T14:05:20.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.664 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:05:20.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:05:20.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:20.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332317+0000", 2023-07-02T14:05:20.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2247999999999998e-05 2023-07-02T14:05:20.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332369+0000", 2023-07-02T14:05:20.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4189999999999996e-06 2023-07-02T14:05:20.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332376+0000", 2023-07-02T14:05:20.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1049999999999999e-06 2023-07-02T14:05:20.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332379+0000", 2023-07-02T14:05:20.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6894000000000003e-05 2023-07-02T14:05:20.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332416+0000", 2023-07-02T14:05:20.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3970000000000002e-06 2023-07-02T14:05:20.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:20.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332422+0000", 2023-07-02T14:05:20.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0231382280000001 2023-07-02T14:05:20.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355560+0000", 2023-07-02T14:05:20.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9079000000000005e-05 2023-07-02T14:05:20.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355659+0000", 2023-07-02T14:05:20.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018005000000000001 2023-07-02T14:05:20.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.355839+0000", 2023-07-02T14:05:20.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068044439999999998 2023-07-02T14:05:20.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:20.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362644+0000", 2023-07-02T14:05:20.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0090155589999999994 2023-07-02T14:05:20.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371659+0000", 2023-07-02T14:05:20.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1297999999999994e-05 2023-07-02T14:05:20.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:20.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371730+0000", 2023-07-02T14:05:20.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013100699999999999 2023-07-02T14:05:20.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:20.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371861+0000", 2023-07-02T14:05:20.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.039544188 2023-07-02T14:05:20.674 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.674 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.674 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.674 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2784 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 0~393216] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:20.675 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:05:20.675 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 130.24542114400001, 2023-07-02T14:05:20.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999, 2023-07-02T14:05:20.675 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.675 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:20.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.676 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2784 2023-07-02T14:05:20.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.676 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:05:20.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332386+0000", 2023-07-02T14:05:20.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:20.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332385+0000", 2023-07-02T14:05:20.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00040252400000000002 2023-07-02T14:05:20.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332787+0000", 2023-07-02T14:05:20.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0208e-05 2023-07-02T14:05:20.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332798+0000", 2023-07-02T14:05:20.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4400000000000001e-06 2023-07-02T14:05:20.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332801+0000", 2023-07-02T14:05:20.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4838000000000001e-05 2023-07-02T14:05:20.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332836+0000", 2023-07-02T14:05:20.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5319999999999998e-06 2023-07-02T14:05:20.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:20.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332839+0000", 2023-07-02T14:05:20.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0234290109999999 2023-07-02T14:05:20.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356268+0000", 2023-07-02T14:05:20.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011165700000000001 2023-07-02T14:05:20.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356380+0000", 2023-07-02T14:05:20.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022504799999999999 2023-07-02T14:05:20.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.356605+0000", 2023-07-02T14:05:20.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015276201 2023-07-02T14:05:20.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:20.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371881+0000", 2023-07-02T14:05:20.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5496999999999999e-05 2023-07-02T14:05:20.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371957+0000", 2023-07-02T14:05:20.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4570000000000002e-05 2023-07-02T14:05:20.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:20.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372001+0000", 2023-07-02T14:05:20.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3880000000000003e-05 2023-07-02T14:05:20.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:20.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372085+0000", 2023-07-02T14:05:20.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0396996249999999 2023-07-02T14:05:20.685 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2785 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 405504~446464] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:05:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 130.24499969300001, 2023-07-02T14:05:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999, 2023-07-02T14:05:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2785 2023-07-02T14:05:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.688 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:05:20.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:05:20.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:20.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.332807+0000", 2023-07-02T14:05:20.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059527500000000004 2023-07-02T14:05:20.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333402+0000", 2023-07-02T14:05:20.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7059999999999993e-06 2023-07-02T14:05:20.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333411+0000", 2023-07-02T14:05:20.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0450000000000001e-06 2023-07-02T14:05:20.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333414+0000", 2023-07-02T14:05:20.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5790999999999997e-05 2023-07-02T14:05:20.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:05:20.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3320000000000002e-06 2023-07-02T14:05:20.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:20.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333454+0000", 2023-07-02T14:05:20.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.025310269 2023-07-02T14:05:20.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358764+0000", 2023-07-02T14:05:20.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012414899999999999 2023-07-02T14:05:20.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358888+0000", 2023-07-02T14:05:20.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000217028 2023-07-02T14:05:20.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.359105+0000", 2023-07-02T14:05:20.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012788212 2023-07-02T14:05:20.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:20.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371893+0000", 2023-07-02T14:05:20.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0012201569999999999 2023-07-02T14:05:20.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373114+0000", 2023-07-02T14:05:20.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4512999999999995e-05 2023-07-02T14:05:20.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:20.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373198+0000", 2023-07-02T14:05:20.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132373 2023-07-02T14:05:20.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:20.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373330+0000", 2023-07-02T14:05:20.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0405232639999999 2023-07-02T14:05:20.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.697 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.698 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2786 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1310720~4096] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:20.698 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:05:20.698 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 130.24438699999999, 2023-07-02T14:05:20.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001, 2023-07-02T14:05:20.698 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.698 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:20.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.699 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2786 2023-07-02T14:05:20.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.699 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:05:20.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333420+0000", 2023-07-02T14:05:20.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:20.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333419+0000", 2023-07-02T14:05:20.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2009999999999999e-06 2023-07-02T14:05:20.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333427+0000", 2023-07-02T14:05:20.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0210000000000001e-06 2023-07-02T14:05:20.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333431+0000", 2023-07-02T14:05:20.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3769999999999999e-06 2023-07-02T14:05:20.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333434+0000", 2023-07-02T14:05:20.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1161e-05 2023-07-02T14:05:20.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333485+0000", 2023-07-02T14:05:20.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1549000000000001e-05 2023-07-02T14:05:20.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:20.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333496+0000", 2023-07-02T14:05:20.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0273116339999999 2023-07-02T14:05:20.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360808+0000", 2023-07-02T14:05:20.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000115128 2023-07-02T14:05:20.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.360923+0000", 2023-07-02T14:05:20.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018367700000000001 2023-07-02T14:05:20.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361107+0000", 2023-07-02T14:05:20.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010796923999999999 2023-07-02T14:05:20.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:20.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371904+0000", 2023-07-02T14:05:20.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014983520000000001 2023-07-02T14:05:20.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373402+0000", 2023-07-02T14:05:20.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0520999999999999e-05 2023-07-02T14:05:20.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:20.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373463+0000", 2023-07-02T14:05:20.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000129095 2023-07-02T14:05:20.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:20.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373592+0000", 2023-07-02T14:05:20.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0401719890000001 2023-07-02T14:05:20.709 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.709 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.709 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.709 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2787 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 1433600~8192] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:20.710 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:05:20.710 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 130.24437019999999, 2023-07-02T14:05:20.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999, 2023-07-02T14:05:20.710 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.710 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.710 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.710 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:20.710 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.711 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2787 2023-07-02T14:05:20.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.711 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:05:20.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333437+0000", 2023-07-02T14:05:20.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:20.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333436+0000", 2023-07-02T14:05:20.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2731999999999999e-05 2023-07-02T14:05:20.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333449+0000", 2023-07-02T14:05:20.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7440000000000001e-06 2023-07-02T14:05:20.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333456+0000", 2023-07-02T14:05:20.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8650000000000003e-06 2023-07-02T14:05:20.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333460+0000", 2023-07-02T14:05:20.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0077e-05 2023-07-02T14:05:20.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333530+0000", 2023-07-02T14:05:20.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4180000000000001e-06 2023-07-02T14:05:20.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:20.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333533+0000", 2023-07-02T14:05:20.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0284002720000001 2023-07-02T14:05:20.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.361933+0000", 2023-07-02T14:05:20.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6781999999999996e-05 2023-07-02T14:05:20.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362030+0000", 2023-07-02T14:05:20.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018305400000000001 2023-07-02T14:05:20.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362213+0000", 2023-07-02T14:05:20.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097015109999999995 2023-07-02T14:05:20.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:20.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.371915+0000", 2023-07-02T14:05:20.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001742295 2023-07-02T14:05:20.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373657+0000", 2023-07-02T14:05:20.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5980000000000003e-05 2023-07-02T14:05:20.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:20.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373713+0000", 2023-07-02T14:05:20.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011783800000000001 2023-07-02T14:05:20.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:20.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373831+0000", 2023-07-02T14:05:20.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0403941189999999 2023-07-02T14:05:20.720 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.721 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.721 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.721 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2803 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 851968~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:20.721 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:05:20.721 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 130.24432834800001, 2023-07-02T14:05:20.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514, 2023-07-02T14:05:20.722 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.722 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:20.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.722 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2803 2023-07-02T14:05:20.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.722 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:05:20.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:05:20.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:20.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333478+0000", 2023-07-02T14:05:20.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4943000000000001e-05 2023-07-02T14:05:20.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333493+0000", 2023-07-02T14:05:20.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2629999999999997e-06 2023-07-02T14:05:20.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333500+0000", 2023-07-02T14:05:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4159999999999997e-06 2023-07-02T14:05:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333505+0000", 2023-07-02T14:05:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4518999999999998e-05 2023-07-02T14:05:20.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333559+0000", 2023-07-02T14:05:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5900000000000008e-06 2023-07-02T14:05:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333568+0000", 2023-07-02T14:05:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0291061779999999 2023-07-02T14:05:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362674+0000", 2023-07-02T14:05:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000110412 2023-07-02T14:05:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362784+0000", 2023-07-02T14:05:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015384599999999999 2023-07-02T14:05:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.362938+0000", 2023-07-02T14:05:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091682310000000006 2023-07-02T14:05:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372106+0000", 2023-07-02T14:05:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017703230000000001 2023-07-02T14:05:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373877+0000", 2023-07-02T14:05:20.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6387000000000002e-05 2023-07-02T14:05:20.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:20.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.373923+0000", 2023-07-02T14:05:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014099199999999999 2023-07-02T14:05:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374064+0000", 2023-07-02T14:05:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040585514 2023-07-02T14:05:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.732 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2805 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 634880~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:05:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 130.24429517300001, 2023-07-02T14:05:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814, 2023-07-02T14:05:20.733 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2805 2023-07-02T14:05:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.734 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:05:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333512+0000", 2023-07-02T14:05:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333511+0000", 2023-07-02T14:05:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000207092 2023-07-02T14:05:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333718+0000", 2023-07-02T14:05:20.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5110000000000003e-06 2023-07-02T14:05:20.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333727+0000", 2023-07-02T14:05:20.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3270000000000007e-06 2023-07-02T14:05:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333736+0000", 2023-07-02T14:05:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0319999999999999e-05 2023-07-02T14:05:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333756+0000", 2023-07-02T14:05:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1695000000000001e-05 2023-07-02T14:05:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:05:20.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0295750560000001 2023-07-02T14:05:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363343+0000", 2023-07-02T14:05:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3387999999999994e-05 2023-07-02T14:05:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363437+0000", 2023-07-02T14:05:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00018373 2023-07-02T14:05:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.363620+0000", 2023-07-02T14:05:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0084948769999999996 2023-07-02T14:05:20.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372115+0000", 2023-07-02T14:05:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0020082170000000001 2023-07-02T14:05:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374123+0000", 2023-07-02T14:05:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8875999999999997e-05 2023-07-02T14:05:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374182+0000", 2023-07-02T14:05:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0001272 2023-07-02T14:05:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:20.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374309+0000", 2023-07-02T14:05:20.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040797814 2023-07-02T14:05:20.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.744 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2806 2.4 2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head [write 1359872~16384] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:05:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 130.244064842, 2023-07-02T14:05:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234, 2023-07-02T14:05:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.745 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2806 2023-07-02T14:05:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:05:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333742+0000", 2023-07-02T14:05:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333741+0000", 2023-07-02T14:05:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3978999999999999e-05 2023-07-02T14:05:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333755+0000", 2023-07-02T14:05:20.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:05:20.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333761+0000", 2023-07-02T14:05:20.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7010000000000001e-06 2023-07-02T14:05:20.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333764+0000", 2023-07-02T14:05:20.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5086000000000003e-05 2023-07-02T14:05:20.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333799+0000", 2023-07-02T14:05:20.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6389999999999992e-06 2023-07-02T14:05:20.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:20.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333806+0000", 2023-07-02T14:05:20.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.030496675 2023-07-02T14:05:20.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364303+0000", 2023-07-02T14:05:20.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011382700000000001 2023-07-02T14:05:20.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364417+0000", 2023-07-02T14:05:20.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192058 2023-07-02T14:05:20.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364609+0000", 2023-07-02T14:05:20.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075132130000000004 2023-07-02T14:05:20.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372122+0000", 2023-07-02T14:05:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022731750000000001 2023-07-02T14:05:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374395+0000", 2023-07-02T14:05:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5277000000000005e-05 2023-07-02T14:05:20.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374460+0000", 2023-07-02T14:05:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017074599999999999 2023-07-02T14:05:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:20.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374631+0000", 2023-07-02T14:05:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.040889234 2023-07-02T14:05:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head [write 868352~49152] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:20.756 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:05:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 130.244038128, 2023-07-02T14:05:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001, 2023-07-02T14:05:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:20.757 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:05:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:05:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333769+0000", 2023-07-02T14:05:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333768+0000", 2023-07-02T14:05:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6355000000000001e-05 2023-07-02T14:05:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333794+0000", 2023-07-02T14:05:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8329999999999996e-06 2023-07-02T14:05:20.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333801+0000", 2023-07-02T14:05:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5419999999999996e-06 2023-07-02T14:05:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333807+0000", 2023-07-02T14:05:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9456000000000001e-05 2023-07-02T14:05:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333826+0000", 2023-07-02T14:05:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.303e-06 2023-07-02T14:05:20.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.333830+0000", 2023-07-02T14:05:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.031137559 2023-07-02T14:05:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.364967+0000", 2023-07-02T14:05:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1390000000000004e-05 2023-07-02T14:05:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365059+0000", 2023-07-02T14:05:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017597399999999999 2023-07-02T14:05:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365234+0000", 2023-07-02T14:05:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068943950000000002 2023-07-02T14:05:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:20.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372129+0000", 2023-07-02T14:05:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0026189910000000002 2023-07-02T14:05:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374748+0000", 2023-07-02T14:05:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6613999999999996e-05 2023-07-02T14:05:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374814+0000", 2023-07-02T14:05:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013502300000000001 2023-07-02T14:05:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:20.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.374950+0000", 2023-07-02T14:05:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0411808300000001 2023-07-02T14:05:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.768 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:172 2.5 2:a79a9f1f:::rbd_data.125de1449605.0000000000000039:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.848672+0000", 2023-07-02T14:05:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.729135193000001, 2023-07-02T14:05:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041540012000000001, 2023-07-02T14:05:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:20.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:20.770 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 172 2023-07-02T14:05:20.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.770 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848672+0000", 2023-07-02T14:05:20.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848672+0000", 2023-07-02T14:05:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:05:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848669+0000", 2023-07-02T14:05:20.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1610000000000001e-05 2023-07-02T14:05:20.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848690+0000", 2023-07-02T14:05:20.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0169999999999998e-05 2023-07-02T14:05:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848710+0000", 2023-07-02T14:05:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0960000000000007e-06 2023-07-02T14:05:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848720+0000", 2023-07-02T14:05:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0717000000000002e-05 2023-07-02T14:05:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848780+0000", 2023-07-02T14:05:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.303e-06 2023-07-02T14:05:20.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2023-07-02T14:05:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.848789+0000", 2023-07-02T14:05:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.032354921000000002 2023-07-02T14:05:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.881144+0000", 2023-07-02T14:05:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0827000000000005e-05 2023-07-02T14:05:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.881214+0000", 2023-07-02T14:05:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089973080000000007 2023-07-02T14:05:20.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.890212+0000", 2023-07-02T14:05:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041540012000000001 2023-07-02T14:05:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:174 2.5 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.851335+0000", 2023-07-02T14:05:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.726471986, 2023-07-02T14:05:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.044329581999999999, 2023-07-02T14:05:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 174 2023-07-02T14:05:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851335+0000", 2023-07-02T14:05:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851335+0000", 2023-07-02T14:05:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999967 2023-07-02T14:05:20.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851332+0000", 2023-07-02T14:05:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6269999999999998e-05 2023-07-02T14:05:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851348+0000", 2023-07-02T14:05:20.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3189000000000001e-05 2023-07-02T14:05:20.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851371+0000", 2023-07-02T14:05:20.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.132e-06 2023-07-02T14:05:20.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851380+0000", 2023-07-02T14:05:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7970999999999999e-05 2023-07-02T14:05:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851428+0000", 2023-07-02T14:05:20.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1050000000000003e-06 2023-07-02T14:05:20.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting_for_map not empty", 2023-07-02T14:05:20.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.851433+0000", 2023-07-02T14:05:20.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038825011999999999 2023-07-02T14:05:20.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.890258+0000", 2023-07-02T14:05:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0666000000000002e-05 2023-07-02T14:05:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.890349+0000", 2023-07-02T14:05:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0053153460000000003 2023-07-02T14:05:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.895664+0000", 2023-07-02T14:05:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.044329581999999999 2023-07-02T14:05:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.786 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:336 2.5 2:a8e921ad:::rbd_data.125de1449605.00000000000000d7:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636630+0000", 2023-07-02T14:05:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.941177008, 2023-07-02T14:05:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041573730000000003, 2023-07-02T14:05:20.787 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 336 2023-07-02T14:05:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636630+0000", 2023-07-02T14:05:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636630+0000", 2023-07-02T14:05:20.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636629+0000", 2023-07-02T14:05:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.847e-06 2023-07-02T14:05:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636632+0000", 2023-07-02T14:05:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0449999999999997e-06 2023-07-02T14:05:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636638+0000", 2023-07-02T14:05:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5080000000000001e-06 2023-07-02T14:05:20.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636642+0000", 2023-07-02T14:05:20.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.033756268999999998 2023-07-02T14:05:20.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.670398+0000", 2023-07-02T14:05:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8985000000000004e-05 2023-07-02T14:05:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.670467+0000", 2023-07-02T14:05:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0077364360000000002 2023-07-02T14:05:20.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.678204+0000", 2023-07-02T14:05:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041573730000000003 2023-07-02T14:05:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:343 2.5 2:b3c82204:::rbd_data.125de1449605.00000000000000de:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636645+0000", 2023-07-02T14:05:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.941161578999999, 2023-07-02T14:05:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.048581695000000001, 2023-07-02T14:05:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.795 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:20.796 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.796 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:20.796 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:20.796 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 343 2023-07-02T14:05:20.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.796 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636645+0000", 2023-07-02T14:05:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636645+0000", 2023-07-02T14:05:20.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:20.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636645+0000", 2023-07-02T14:05:20.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9190000000000004e-06 2023-07-02T14:05:20.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636651+0000", 2023-07-02T14:05:20.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4e-06 2023-07-02T14:05:20.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636656+0000", 2023-07-02T14:05:20.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5860000000000001e-06 2023-07-02T14:05:20.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636660+0000", 2023-07-02T14:05:20.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041586692000000001 2023-07-02T14:05:20.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.678246+0000", 2023-07-02T14:05:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113236 2023-07-02T14:05:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.678360+0000", 2023-07-02T14:05:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006867199 2023-07-02T14:05:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.685227+0000", 2023-07-02T14:05:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.048581695000000001 2023-07-02T14:05:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.802 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.803 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.803 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.803 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.803 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:20.804 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:05:20.805 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:20.804+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:20.905 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:20.904+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:20.984 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:20.984 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:20.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.984 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:15893 2.1 e146/104 2:9d57bdf1:::rbd_data.125de1449605.0000000000000108:head v 146'1802, mlcod=146'1800)", 2023-07-02T14:05:20.984 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:20.973896+0000", 2023-07-02T14:05:20.985 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0064947499999999997, 2023-07-02T14:05:20.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065238420000000002, 2023-07-02T14:05:20.985 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.985 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:20.985 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.973896+0000", 2023-07-02T14:05:20.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.973896+0000", 2023-07-02T14:05:20.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:20.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.973894+0000", 2023-07-02T14:05:20.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00034426599999999997 2023-07-02T14:05:20.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.974238+0000", 2023-07-02T14:05:20.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2473999999999999e-05 2023-07-02T14:05:20.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.974251+0000", 2023-07-02T14:05:20.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9260000000000003e-06 2023-07-02T14:05:20.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.974258+0000", 2023-07-02T14:05:20.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2775000000000005e-05 2023-07-02T14:05:20.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.974351+0000", 2023-07-02T14:05:20.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5694000000000002e-05 2023-07-02T14:05:20.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.974376+0000", 2023-07-02T14:05:20.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048077999999999999 2023-07-02T14:05:20.990 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.990 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.990 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.991 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:15894 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:head [write 65536~1376256 in=1376256b] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:20.991 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:20.969924+0000", 2023-07-02T14:05:20.991 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.010466711, 2023-07-02T14:05:20.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010587809, 2023-07-02T14:05:20.991 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.991 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "waiting for sub ops", 2023-07-02T14:05:20.992 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:20.992 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:20.992 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:20.992 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 15894 2023-07-02T14:05:20.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.992 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:20.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:20.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.969924+0000", 2023-07-02T14:05:20.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:20.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:20.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.969924+0000", 2023-07-02T14:05:20.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:20.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:20.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.969922+0000", 2023-07-02T14:05:20.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078647560000000005 2023-07-02T14:05:20.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:20.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.977786+0000", 2023-07-02T14:05:20.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4528999999999999e-05 2023-07-02T14:05:20.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:20.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.977801+0000", 2023-07-02T14:05:20.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7725000000000001e-05 2023-07-02T14:05:20.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:20.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.977819+0000", 2023-07-02T14:05:20.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5792999999999998e-05 2023-07-02T14:05:20.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:20.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.977894+0000", 2023-07-02T14:05:20.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1032999999999995e-05 2023-07-02T14:05:20.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:20.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.977975+0000", 2023-07-02T14:05:20.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017952699999999999 2023-07-02T14:05:20.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3", 2023-07-02T14:05:20.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.978155+0000", 2023-07-02T14:05:20.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082313539999999998 2023-07-02T14:05:20.998 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.998 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:20.998 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:20.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:20.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:20.999 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:15900 2.0 e146/144 2:169af44a:::rbd_data.125de1449605.0000000000000106:head v 146'3936, mlcod=146'3933)", 2023-07-02T14:05:20.999 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:20.976762+0000", 2023-07-02T14:05:20.999 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0036286589999999998, 2023-07-02T14:05:20.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003847657, 2023-07-02T14:05:20.999 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:20.999 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:20.999 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.976762+0000", 2023-07-02T14:05:21.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.976762+0000", 2023-07-02T14:05:21.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:21.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.976761+0000", 2023-07-02T14:05:21.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0031803790000000001 2023-07-02T14:05:21.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.979942+0000", 2023-07-02T14:05:21.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5468000000000002e-05 2023-07-02T14:05:21.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.979957+0000", 2023-07-02T14:05:21.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6469999999999998e-06 2023-07-02T14:05:21.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.979965+0000", 2023-07-02T14:05:21.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9732000000000003e-05 2023-07-02T14:05:21.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.980034+0000", 2023-07-02T14:05:21.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4762000000000001e-05 2023-07-02T14:05:21.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:21.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:20.980059+0000", 2023-07-02T14:05:21.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032975579999999999 2023-07-02T14:05:21.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.005 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.005 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:05:21.005 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 3 2023-07-02T14:05:21.005 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:21.006 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:05:21.007 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:21.005+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:21.105 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i++ )) 2023-07-02T14:05:21.105 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i < 10 )) 2023-07-02T14:05:21.105 INFO:tasks.workunit.client.0.smithi163.stderr:+ pv -L 10M linux-5.4.tar.gz 2023-07-02T14:05:21.107 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:21.106+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:21.131 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:21.131 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:21.131 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:05:21.132 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:05:21.132 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:21.141 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:05:21.182 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:05:21.219 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap34 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap35 b3fbb028-e267-4d25-a750-93ba18a2316a@snap35.diff 2023-07-02T14:05:21.224 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:21.222+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:21.290 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:05:21.290+0000 7f7457c10700 0 -- 172.21.15.163:0/4228515631 >> [v2:172.21.15.22:6808/99977,v1:172.21.15.22:6809/99977] conn(0x55d78d658e50 msgr2=0x55d78d65b280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:21.297 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:05:21.297+0000 7f7456c0e700 0 -- 172.21.15.163:0/4228515631 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x55d78d65de20 msgr2=0x55d78d660270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:21.310 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:21.309+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:21.366 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:21.366 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:05:21.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:05:21.367 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:21.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.367 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:05:21.367 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:21.367 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 131.911569679, 2023-07-02T14:05:21.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:05:21.368 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:21.368 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:21.368 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:21.368 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:05:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:21.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:21.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:21.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:05:21.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:05:21.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:05:21.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:05:21.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:05:21.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:05:21.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:05:21.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:05:21.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:05:21.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:05:21.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:21.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:05:21.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:05:21.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:05:21.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:05:21.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:21.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:05:21.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:05:21.379 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.379 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.379 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.380 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:05:21.380 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:21.380 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 131.04087667100001, 2023-07-02T14:05:21.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:05:21.380 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:21.381 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:21.381 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:21.381 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:05:21.381 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:05:21.381 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:05:21.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.382 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:21.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:21.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:21.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:05:21.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:05:21.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:05:21.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:05:21.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:05:21.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:05:21.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:05:21.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:05:21.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:05:21.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:05:21.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:21.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:05:21.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:05:21.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:05:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:05:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:05:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:05:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:21.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:05:21.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:05:21.391 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.391 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.391 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.392 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:21.392 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:21.392 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 131.04075194999999, 2023-07-02T14:05:21.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:05:21.392 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:21.392 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:21.392 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:21.393 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:21.393 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:21.393 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:05:21.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.393 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:21.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:21.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:21.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:05:21.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:05:21.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:05:21.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:05:21.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:05:21.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:05:21.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:05:21.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:05:21.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:05:21.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:05:21.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:21.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:05:21.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:05:21.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:05:21.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:05:21.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:21.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:05:21.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:05:21.400 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.400 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.400 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.400 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:21.400 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:21.401 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 131.033902066, 2023-07-02T14:05:21.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:05:21.401 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:21.401 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:21.401 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:21.401 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:21.401 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:21.401 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:05:21.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.402 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:21.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:21.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:05:21.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:05:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:05:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:05:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:05:21.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:05:21.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:05:21.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:05:21.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:05:21.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:05:21.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:05:21.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:21.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:05:21.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:05:21.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:05:21.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:05:21.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:21.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:05:21.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:05:21.408 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.408 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.409 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.409 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:21.409 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:21.409 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 131.03383914099999, 2023-07-02T14:05:21.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:05:21.409 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:21.410 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:21.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:21.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:21.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:21.410 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:05:21.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.410 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:21.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:21.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:21.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:05:21.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:05:21.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:05:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:05:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:05:21.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:05:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:05:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:05:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:05:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:05:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:05:21.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:05:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:05:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:05:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:05:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:05:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.417 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 131.03353872, 2023-07-02T14:05:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:05:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:21.418 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:05:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:05:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:05:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:05:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:05:21.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:05:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:05:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:05:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:05:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:05:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:05:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:05:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:05:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:05:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:05:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:05:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:05:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:05:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:05:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:05:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:05:21.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:05:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:05:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:05:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:05:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:05:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:05:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:05:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:05:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:05:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.429 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:05:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 131.03301422000001, 2023-07-02T14:05:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:05:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:21.430 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:05:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:21.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:05:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:05:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:05:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:05:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:05:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:05:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:05:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:05:21.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:05:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:05:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:05:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:05:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:05:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:05:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:05:21.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:05:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:05:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.438 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:05:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 103.90899509499999, 2023-07-02T14:05:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:05:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:21.439 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:05:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:21.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:05:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:05:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:05:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:05:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:05:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:05:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:05:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:05:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:05:21.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:05:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:05:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:05:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:05:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:05:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:05:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:05:21.446 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 103.583512391, 2023-07-02T14:05:21.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:05:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:05:21.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:05:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:05:21.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:05:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:05:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:05:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:05:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:05:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:05:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:05:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:05:21.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:05:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:05:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:05:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:05:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:05:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:05:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:05:21.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:05:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:05:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:05:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:05:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:05:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:05:21.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:05:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:05:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3939 2.1 2:9ef34a83:::rbd_data.125de1449605.00000000000000b8:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 103.579591694, 2023-07-02T14:05:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001, 2023-07-02T14:05:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:21.459 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3939 2023-07-02T14:05:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046852830000000002 2023-07-02T14:05:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788763+0000", 2023-07-02T14:05:21.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3185e-05 2023-07-02T14:05:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788776+0000", 2023-07-02T14:05:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8470000000000003e-06 2023-07-02T14:05:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788781+0000", 2023-07-02T14:05:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7104000000000002e-05 2023-07-02T14:05:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788828+0000", 2023-07-02T14:05:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3516000000000001e-05 2023-07-02T14:05:21.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788841+0000", 2023-07-02T14:05:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33064513600000001 2023-07-02T14:05:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119487+0000", 2023-07-02T14:05:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9876999999999997e-05 2023-07-02T14:05:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119537+0000", 2023-07-02T14:05:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014160999999999999 2023-07-02T14:05:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119678+0000", 2023-07-02T14:05:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023054392 2023-07-02T14:05:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.142733+0000", 2023-07-02T14:05:21.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074065470000000003 2023-07-02T14:05:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150139+0000", 2023-07-02T14:05:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5025999999999997e-05 2023-07-02T14:05:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150204+0000", 2023-07-02T14:05:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147999999999997e-05 2023-07-02T14:05:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150241+0000", 2023-07-02T14:05:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001 2023-07-02T14:05:21.469 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:17 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 0~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:05:21.470 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.165236609000001, 2023-07-02T14:05:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997, 2023-07-02T14:05:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17 2023-07-02T14:05:21.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:05:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:05:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:05:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198431+0000", 2023-07-02T14:05:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2815e-05 2023-07-02T14:05:21.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198443+0000", 2023-07-02T14:05:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2569999999999999e-05 2023-07-02T14:05:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198466+0000", 2023-07-02T14:05:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0223e-05 2023-07-02T14:05:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198476+0000", 2023-07-02T14:05:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1792999999999997e-05 2023-07-02T14:05:21.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:05:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4249999999999998e-06 2023-07-02T14:05:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:21.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198536+0000", 2023-07-02T14:05:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91888663500000001 2023-07-02T14:05:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117423+0000", 2023-07-02T14:05:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223594 2023-07-02T14:05:21.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117647+0000", 2023-07-02T14:05:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033832900000000001 2023-07-02T14:05:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:21.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117985+0000", 2023-07-02T14:05:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997 2023-07-02T14:05:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.479 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:18 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 65536~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.165179397999999, 2023-07-02T14:05:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998, 2023-07-02T14:05:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:21.480 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18 2023-07-02T14:05:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7600000000000002e-06 2023-07-02T14:05:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198494+0000", 2023-07-02T14:05:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7840000000000004e-06 2023-07-02T14:05:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198502+0000", 2023-07-02T14:05:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6459999999999998e-06 2023-07-02T14:05:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198508+0000", 2023-07-02T14:05:21.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7395999999999997e-05 2023-07-02T14:05:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198565+0000", 2023-07-02T14:05:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6800000000000004e-06 2023-07-02T14:05:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198572+0000", 2023-07-02T14:05:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91965896800000002 2023-07-02T14:05:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118231+0000", 2023-07-02T14:05:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5212999999999998e-05 2023-07-02T14:05:21.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118326+0000", 2023-07-02T14:05:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000238538 2023-07-02T14:05:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:21.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118565+0000", 2023-07-02T14:05:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998 2023-07-02T14:05:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.488 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:19 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 131072~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.165155174000001, 2023-07-02T14:05:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002, 2023-07-02T14:05:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:21.489 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 19 2023-07-02T14:05:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1990000000000002e-06 2023-07-02T14:05:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198517+0000", 2023-07-02T14:05:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8789999999999997e-06 2023-07-02T14:05:21.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198524+0000", 2023-07-02T14:05:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:05:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:05:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5545999999999999e-05 2023-07-02T14:05:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2540000000000003e-06 2023-07-02T14:05:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198598+0000", 2023-07-02T14:05:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.919983263 2023-07-02T14:05:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118581+0000", 2023-07-02T14:05:21.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6878999999999994e-05 2023-07-02T14:05:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118648+0000", 2023-07-02T14:05:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149732 2023-07-02T14:05:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118798+0000", 2023-07-02T14:05:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002 2023-07-02T14:05:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:20 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 262144~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.16513621, 2023-07-02T14:05:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998, 2023-07-02T14:05:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:21.498 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20 2023-07-02T14:05:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:21.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.732e-06 2023-07-02T14:05:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198537+0000", 2023-07-02T14:05:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5259999999999999e-06 2023-07-02T14:05:21.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198543+0000", 2023-07-02T14:05:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.121e-06 2023-07-02T14:05:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198547+0000", 2023-07-02T14:05:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9591000000000005e-05 2023-07-02T14:05:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:05:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3120000000000004e-06 2023-07-02T14:05:21.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198621+0000", 2023-07-02T14:05:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92020789400000003 2023-07-02T14:05:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118829+0000", 2023-07-02T14:05:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022393386000000001 2023-07-02T14:05:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141223+0000", 2023-07-02T14:05:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000329866 2023-07-02T14:05:21.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141552+0000", 2023-07-02T14:05:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998 2023-07-02T14:05:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:21 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 196608~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.165108575, 2023-07-02T14:05:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995, 2023-07-02T14:05:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:21.507 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21 2023-07-02T14:05:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0169999999999999e-05 2023-07-02T14:05:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198571+0000", 2023-07-02T14:05:21.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3479999999999998e-06 2023-07-02T14:05:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198578+0000", 2023-07-02T14:05:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8439999999999997e-06 2023-07-02T14:05:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198583+0000", 2023-07-02T14:05:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8294999999999997e-05 2023-07-02T14:05:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198641+0000", 2023-07-02T14:05:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8120000000000001e-06 2023-07-02T14:05:21.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:05:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94293828499999999 2023-07-02T14:05:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141583+0000", 2023-07-02T14:05:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014607300000000001 2023-07-02T14:05:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141729+0000", 2023-07-02T14:05:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00044906600000000003 2023-07-02T14:05:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142178+0000", 2023-07-02T14:05:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995 2023-07-02T14:05:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:22 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 327680~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.165075144999999, 2023-07-02T14:05:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004, 2023-07-02T14:05:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:21.516 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 22 2023-07-02T14:05:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4850000000000001e-06 2023-07-02T14:05:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198597+0000", 2023-07-02T14:05:21.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:05:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198605+0000", 2023-07-02T14:05:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4839999999999998e-06 2023-07-02T14:05:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198609+0000", 2023-07-02T14:05:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3100999999999999e-05 2023-07-02T14:05:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198662+0000", 2023-07-02T14:05:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9099999999999998e-06 2023-07-02T14:05:21.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198666+0000", 2023-07-02T14:05:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.943539503 2023-07-02T14:05:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142206+0000", 2023-07-02T14:05:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012255999999999999 2023-07-02T14:05:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142328+0000", 2023-07-02T14:05:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208956 2023-07-02T14:05:21.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142537+0000", 2023-07-02T14:05:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004 2023-07-02T14:05:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:23 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 393216~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.165055165, 2023-07-02T14:05:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348, 2023-07-02T14:05:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:21.525 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23 2023-07-02T14:05:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8439999999999998e-06 2023-07-02T14:05:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:05:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0440000000000001e-06 2023-07-02T14:05:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198623+0000", 2023-07-02T14:05:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4480000000000002e-06 2023-07-02T14:05:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198628+0000", 2023-07-02T14:05:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5843e-05 2023-07-02T14:05:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:21.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.89e-06 2023-07-02T14:05:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198688+0000", 2023-07-02T14:05:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94389171999999999 2023-07-02T14:05:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142580+0000", 2023-07-02T14:05:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3892000000000004e-05 2023-07-02T14:05:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142644+0000", 2023-07-02T14:05:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017608000000000001 2023-07-02T14:05:21.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142820+0000", 2023-07-02T14:05:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348 2023-07-02T14:05:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.534 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:24 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 458752~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:21.534 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:05:21.534 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.165027091000001, 2023-07-02T14:05:21.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004, 2023-07-02T14:05:21.534 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:21.534 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:21.534 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:21.535 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:21.535 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:21.535 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 24 2023-07-02T14:05:21.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.535 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:05:21.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:05:21.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:21.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198642+0000", 2023-07-02T14:05:21.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.253e-06 2023-07-02T14:05:21.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:05:21.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1289999999999998e-06 2023-07-02T14:05:21.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198652+0000", 2023-07-02T14:05:21.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4510000000000002e-06 2023-07-02T14:05:21.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198657+0000", 2023-07-02T14:05:21.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9131000000000001e-05 2023-07-02T14:05:21.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198706+0000", 2023-07-02T14:05:21.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8890000000000004e-06 2023-07-02T14:05:21.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:21.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198710+0000", 2023-07-02T14:05:21.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94415412200000004 2023-07-02T14:05:21.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142864+0000", 2023-07-02T14:05:21.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8047000000000006e-05 2023-07-02T14:05:21.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:21.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142952+0000", 2023-07-02T14:05:21.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192967 2023-07-02T14:05:21.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:21.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143145+0000", 2023-07-02T14:05:21.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004 2023-07-02T14:05:21.542 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.542 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.542 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:25 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 524288~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.165008296, 2023-07-02T14:05:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001, 2023-07-02T14:05:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:21.543 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:21.544 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:21.544 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:21.544 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:21.544 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25 2023-07-02T14:05:21.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.544 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:21.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:21.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:21.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:21.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.615e-06 2023-07-02T14:05:21.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198664+0000", 2023-07-02T14:05:21.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4759999999999997e-06 2023-07-02T14:05:21.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198669+0000", 2023-07-02T14:05:21.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5999999999999998e-06 2023-07-02T14:05:21.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198673+0000", 2023-07-02T14:05:21.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4256999999999999e-05 2023-07-02T14:05:21.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198727+0000", 2023-07-02T14:05:21.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840000000000001e-06 2023-07-02T14:05:21.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:21.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198731+0000", 2023-07-02T14:05:21.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94444241900000003 2023-07-02T14:05:21.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143173+0000", 2023-07-02T14:05:21.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6839000000000002e-05 2023-07-02T14:05:21.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:21.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143240+0000", 2023-07-02T14:05:21.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017053399999999999 2023-07-02T14:05:21.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:21.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143411+0000", 2023-07-02T14:05:21.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001 2023-07-02T14:05:21.551 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.551 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.552 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.552 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:26 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 589824~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:21.552 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:21.552 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.164985144999999, 2023-07-02T14:05:21.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997, 2023-07-02T14:05:21.552 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:21.553 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:21.553 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:21.553 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:21.553 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:21.553 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 26 2023-07-02T14:05:21.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.553 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:21.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:21.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:21.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:21.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:21.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:21.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:21.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:21.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:21.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0520000000000002e-06 2023-07-02T14:05:21.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:21.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198687+0000", 2023-07-02T14:05:21.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1880000000000002e-06 2023-07-02T14:05:21.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:21.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198695+0000", 2023-07-02T14:05:21.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9960000000000004e-06 2023-07-02T14:05:21.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:21.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198699+0000", 2023-07-02T14:05:21.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2330999999999998e-05 2023-07-02T14:05:21.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198742+0000", 2023-07-02T14:05:21.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7950000000000001e-06 2023-07-02T14:05:21.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:21.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198745+0000", 2023-07-02T14:05:21.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94469398199999999 2023-07-02T14:05:21.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:21.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143439+0000", 2023-07-02T14:05:21.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6048e-05 2023-07-02T14:05:21.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:21.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143525+0000", 2023-07-02T14:05:21.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017016700000000001 2023-07-02T14:05:21.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:21.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:21.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:21.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143696+0000", 2023-07-02T14:05:21.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997 2023-07-02T14:05:21.560 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.560 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.561 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.561 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:21.561 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:21.561 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:21.561 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:05:21.562 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:21.561+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:21.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:05:21.564 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 100% complete...done. 2023-07-02T14:05:21.564 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:05:21.564 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=36 2023-07-02T14:05:21.565 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap36 2023-07-02T14:05:21.662 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:21.662+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:21.730 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:21.730 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:21.730 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:05:21.730 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:21.740 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:05:21.743 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:05:21.761 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:21.763 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:21.762+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:21.779 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:21.797 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:21.816 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:21.834 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:21.835 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:05:21.835 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:05:21.836 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:05:21.836 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:05:21.836 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:05:21.836 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:05:21.864 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:21.863+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:21.865 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:21.865 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:21.876 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:05:21.926 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:05:22.001 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:22.000+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:22.041 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:22.042 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:22.053 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:05:22.097 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:22.096+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:22.166 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:22.165+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:22.181 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:05:22.187 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 1 2023-07-02T14:05:22.194 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:22.194 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:22.205 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:05:22.262 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:05:22.264 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:05:22.275 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2023-07-02T14:05:22.147853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6743362333,"num_objects":2002,"num_object_clones":1598,"num_object_copies":4004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2002,"num_whiteouts":0,"num_read":34122,"num_read_kb":7745191,"num_write":16190,"num_write_kb":6549150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1487,"num_bytes_recovered":5100088637,"num_keys_recovered":114,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16014,"ondisk_log_size":16014,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":13273476,"kb_used_data":13086628,"kb_used_omap":229,"kb_used_meta":186586,"kb_avail":364213884,"statfs":{"total":386547056640,"available":372955017216,"internally_reserved":0,"allocated":13400707072,"data_stored":15570818985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":191064286},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,0,0,0,0,8],"upper_bound":2048},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":713646080,"num_objects":191,"num_object_clones":177,"num_object_copies":382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-194,"num_objects_misplaced":-11,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":1426,"num_read_kb":1058,"num_write":1158,"num_write_kb":690056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":671601450,"num_keys_recovered":44,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.878206"},"pg_stats":[{"pgid":"2.7","version":"145'2242","reported_seq":5072,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-07-02T14:05:18.055142+0000","last_change":"2023-07-02T14:05:14.122260+0000","last_active":"2023-07-02T14:05:18.055142+0000","last_peered":"2023-07-02T14:05:18.055142+0000","last_clean":"2023-07-02T14:05:18.055142+0000","last_became_active":"2023-07-02T14:05:14.120593+0000","last_became_peered":"2023-07-02T14:05:14.120593+0000","last_unstale":"2023-07-02T14:05:18.055142+0000","last_undegraded":"2023-07-02T14:05:18.055142+0000","last_fullsized":"2023-07-02T14:05:18.055142+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2242,"ondisk_log_size":2242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":843876334,"num_objects":253,"num_object_clones":199,"num_object_copies":506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":18184,"num_read_kb":986770,"num_write":1933,"num_write_kb":820601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":1395581788,"num_keys_recovered":110,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"146'1847","reported_seq":3092,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-07-02T14:05:20.109984+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:05:20.109984+0000","last_peered":"2023-07-02T14:05:20.109984+0000","last_clean":"2023-07-02T14:05:20.109984+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:05:20.109984+0000","last_undegraded":"2023-07-02T14:05:20.109984+0000","last_fullsized":"2023-07-02T14:05:20.109984+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1847,"ondisk_log_size":1847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":909992422,"num_objects":276,"num_object_clones":227,"num_object_copies":552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":276,"num_whiteouts":0,"num_read":948,"num_read_kb":1028437,"num_write":1620,"num_write_kb":884231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"145'1834","reported_seq":2129,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-07-02T14:05:18.008096+0000","last_change":"2023-07-02T14:05:10.106568+0000","last_active":"2023-07-02T14:05:18.008096+0000","last_peered":"2023-07-02T14:05:18.008096+0000","last_clean":"2023-07-02T14:05:18.008096+0000","last_became_active":"2023-07-02T14:05:10.103318+0000","last_became_peered":"2023-07-02T14:05:10.103318+0000","last_unstale":"2023-07-02T14:05:18.008096+0000","last_undegraded":"2023-07-02T14:05:18.008096+0000","last_fullsized":"2023-07-02T14:05:18.008096+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1834,"ondisk_log_size":1834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":786903526,"num_objects":221,"num_object_clones":174,"num_object_copies":442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":221,"num_whiteouts":0,"num_read":328,"num_read_kb":927502,"num_write":1660,"num_write_kb":758416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":572731554,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"145'3645","reported_seq":4732,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-07-02T14:05:18.087121+0000","last_change":"2023-07-02T14:05:17.141529+0000","last_active":"2023-07-02T14:05:18.087121+0000","last_peered":"2023-07-02T14:05:18.087121+0000","last_clean":"2023-07-02T14:05:18.087121+0000","last_became_active":"2023-07-02T14:05:17.138301+0000","last_became_peered":"2023-07-02T14:05:17.138301+0000","last_unstale":"2023-07-02T14:05:18.087121+0000","last_undegraded":"2023-07-02T14:05:18.087121+0000","last_fullsized":"2023-07-02T14:05:18.087121+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3645,"ondisk_log_size":3645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":891020252,"num_objects":263,"num_object_clones":208,"num_object_copies":526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":263,"num_whiteouts":0,"num_read":12202,"num_read_kb":1127107,"num_write":5321,"num_write_kb":870559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":860210318,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":94,"reported_epoch":145,"state":"active+clean","last_fresh":"2023-07-02T14:05:17.163229+0000","last_change":"2023-07-02T14:05:05.089421+0000","last_active":"2023-07-02T14:05:17.163229+0000","last_peered":"2023-07-02T14:05:17.163229+0000","last_clean":"2023-07-02T14:05:17.163229+0000","last_became_active":"2023-07-02T14:05:05.088812+0000","last_became_peered":"2023-07-02T14:05:05.088812+0000","last_unstale":"2023-07-02T14:05:17.163229+0000","last_undegraded":"2023-07-02T14:05:17.163229+0000","last_fullsized":"2023-07-02T14:05:17.163229+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"146'1813","reported_seq":2564,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-07-02T14:05:20.132788+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:05:20.132788+0000","last_peered":"2023-07-02T14:05:20.132788+0000","last_clean":"2023-07-02T14:05:20.132788+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:05:20.132788+0000","last_undegraded":"2023-07-02T14:05:20.132788+0000","last_fullsized":"2023-07-02T14:05:20.132788+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1813,"ondisk_log_size":1813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":967713594,"num_objects":282,"num_object_clones":227,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":636,"num_read_kb":1051850,"num_write":1586,"num_write_kb":941326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"146'1798","reported_seq":2247,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-07-02T14:05:20.066043+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:05:20.066043+0000","last_peered":"2023-07-02T14:05:20.066043+0000","last_clean":"2023-07-02T14:05:20.066043+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:05:20.066043+0000","last_undegraded":"2023-07-02T14:05:20.066043+0000","last_fullsized":"2023-07-02T14:05:20.066043+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1798,"ondisk_log_size":1798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":960074538,"num_objects":275,"num_object_clones":218,"num_object_copies":550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":421,"num_read_kb":984584,"num_write":1580,"num_write_kb":932033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"146'1361","reported_seq":2354,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-07-02T14:05:20.121118+0000","last_change":"2023-07-02T14:03:13.073775+0000","last_active":"2023-07-02T14:05:20.121118+0000","last_peered":"2023-07-02T14:05:20.121118+0000","last_clean":"2023-07-02T14:05:20.121118+0000","last_became_active":"2023-07-02T14:03:13.073206+0000","last_became_peered":"2023-07-02T14:03:13.073206+0000","last_unstale":"2023-07-02T14:05:20.121118+0000","last_undegraded":"2023-07-02T14:05:20.121118+0000","last_fullsized":"2023-07-02T14:05:20.121118+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1361,"ondisk_log_size":1361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":682164891,"num_objects":220,"num_object_clones":175,"num_object_copies":440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":220,"num_whiteouts":0,"num_read":977,"num_read_kb":828828,"num_write":1186,"num_write_kb":660030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":96600083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"146'1474","reported_seq":1884,"reported_epoch":146,"state":"active+clean","last_fresh":"2023-07-02T14:05:20.138725+0000","last_change":"2023-07-02T14:03:14.074306+0000","last_active":"2023-07-02T14:05:20.138725+0000","last_peered":"2023-07-02T14:05:20.138725+0000","last_clean":"2023-07-02T14:05:20.138725+0000","last_became_active":"2023-07-02T14:03:14.073735+0000","last_became_peered":"2023-07-02T14:03:14.073735+0000","last_unstale":"2023-07-02T14:05:20.138725+0000","last_undegraded":"2023-07-02T14:05:20.138725+0000","last_fullsized":"2023-07-02T14:05:20.138725+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1474,"ondisk_log_size":1474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":701616776,"num_objects":212,"num_object_clones":170,"num_object_copies":424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":212,"num_whiteouts":0,"num_read":426,"num_read_kb":810113,"num_write":1304,"num_write_kb":681954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":87949312,"num_keys_recovered":4,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":6743362333,"num_objects":2002,"num_object_clones":1598,"num_object_copies":4004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2002,"num_whiteouts":0,"num_read":34122,"num_read_kb":7745191,"num_write":16190,"num_write_kb":6549150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1487,"num_bytes_recovered":5100088637,"num_keys_recovered":114,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14220500992,"data_stored":16479854136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},"log_size":16014,"ondisk_log_size":16014,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640301,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":0,"seq":47244640301,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":133,"seq":571230650372,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3157012,"kb_used_data":3127180,"kb_used_omap":229,"kb_used_meta":29594,"kb_avail":91214828,"statfs":{"total":96636764160,"available":93403983872,"internally_reserved":0,"allocated":3202232320,"data_stored":3710766609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":30304478},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,8],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640301,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2807332,"kb_used_data":2762012,"kb_used_omap":0,"kb_used_meta":45312,"kb_avail":91564508,"statfs":{"total":96636764160,"available":93762056192,"internally_reserved":0,"allocated":2828300288,"data_stored":3214012238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46399488},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.29399999999999998,"15min":0.29599999999999999},"min":{"1min":0.218,"5min":0.182,"15min":0.182},"max":{"1min":0.378,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.317,"5min":0.315,"15min":0.314},"min":{"1min":0.23000000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.28899999999999998}]},{"osd":1,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34499999999999997,"15min":0.34699999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.48499999999999999,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.254},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.314,"15min":0.317},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.39700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.432}]},{"osd":3,"last update":"Sun Jul 2 14:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.45000000000000001,"15min":0.45100000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46999999999999997,"15min":0.46800000000000003},"min":{"1min":0.35299999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.60399999999999998}]},{"osd":4,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.443,"15min":0.44700000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.56799999999999995,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.45900000000000002,"15min":0.46000000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.57599999999999996,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.68400000000000005}]},{"osd":5,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47899999999999998,"15min":0.48299999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.65200000000000002,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.47599999999999998,"15min":0.48099999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.54600000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.61099999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640301,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2222344,"kb_used_data":2192320,"kb_used_omap":0,"kb_used_meta":30016,"kb_avail":92149496,"statfs":{"total":96636764160,"available":94361083904,"internally_reserved":0,"allocated":2244935680,"data_stored":2637785093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30736384},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.29099999999999998,"15min":0.29299999999999998},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.29699999999999999,"5min":0.30099999999999999,"15min":0.29999999999999999},"min":{"1min":0.216,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.42199999999999999,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.216}]},{"osd":2,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.34300000000000003,"15min":0.34799999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.48599999999999999,"5min":0.746,"15min":0.746},"last":0.435},{"interface":"front","average":{"1min":0.314,"5min":0.34100000000000003,"15min":0.34699999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.40000000000000002,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.31900000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.35599999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.49399999999999999,"15min":0.496},"min":{"1min":0.32400000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.84899999999999998,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.41899999999999998}]},{"osd":4,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.47799999999999998,"15min":0.48499999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.77600000000000002,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.48399999999999999,"15min":0.49299999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.61099999999999999,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.60599999999999998}]},{"osd":5,"last update":"Sun Jul 2 14:04:50 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.48099999999999998,"15min":0.495},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.59199999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.37},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.502,"15min":0.51200000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.51500000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640301,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5086788,"kb_used_data":5005116,"kb_used_omap":0,"kb_used_meta":81664,"kb_avail":89285052,"statfs":{"total":96636764160,"available":91427893248,"internally_reserved":0,"allocated":5125238784,"data_stored":6008255045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":83623936},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.20200000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.32200000000000001,"15min":0.33000000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.41099999999999998,"5min":0.496,"15min":0.496},"last":0.29599999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.33800000000000002,"15min":0.33200000000000002},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.222},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.36599999999999999,"15min":0.376},"min":{"1min":0.251,"5min":0.222,"15min":0.222},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.315}]},{"osd":3,"last update":"Sun Jul 2 14:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.497,"15min":0.501},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.624,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.498,"15min":0.499},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.64400000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.432}]},{"osd":4,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.48699999999999999,"15min":0.503},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.54300000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.498,"15min":0.504},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.46600000000000003}]},{"osd":5,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51400000000000001,"15min":0.51500000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.5589999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.5840000000000001,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.54400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2249867264,"data_stored":2639513353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5185765376,"data_stored":6066356553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2852573184,"data_stored":3242061394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3328909312,"data_stored":3860303766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":603385856,"data_stored":671619070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:05:22.276 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-07-02T14:05:22.277 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:05:22.277 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 2, 4, 5, 3] 2023-07-02T14:05:22.277 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-07-02T14:05:22.277 INFO:teuthology.orchestra.run:waiting for 300 2023-07-02T14:05:22.288 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:22.287+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:22.314 INFO:tasks.ceph.osd.0:Stopped 2023-07-02T14:05:22.315 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.0 2023-07-02T14:05:22.315 INFO:tasks.thrashosds.thrasher:remote for osd 0 is ubuntu@smithi016.front.sepia.ceph.com 2023-07-02T14:05:22.315 DEBUG:teuthology.orchestra.run.smithi016:> type ceph-objectstore-tool 2023-07-02T14:05:22.368 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:22.367+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:22.369 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-07-02T14:05:22.369 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op list-pgs 2023-07-02T14:05:22.386 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:22.387 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:22.397 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:05:22.481 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:22.480+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:22.563 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:22.563 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:22.570 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:22.569+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:22.573 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:05:22.670 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:22.670+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:22.737 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:22.737 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:22.747 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:05:22.771 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:22.770+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:22.872 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:22.871+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:22.910 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:22.911 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:22.921 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:05:23.003 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:23.003+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:23.074 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:23.073+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:23.084 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:23.085 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:23.095 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:05:23.175 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:23.174+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:23.183 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:05:23.188 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap35 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap36 b3fbb028-e267-4d25-a750-93ba18a2316a@snap36.diff 2023-07-02T14:05:23.227 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:23.227 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:23.238 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:05:23.287 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:23.286+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:23.372 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:23.372 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:23.376 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:23.376+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:23.383 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:05:23.477 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:23.477+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:23.548 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:23.549 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:23.559 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:05:23.578 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:23.578+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:23.678 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:23.677+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:23.684 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:23.684 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:23.695 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:05:23.779 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:23.778+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:23.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:05:23.861 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:23.862 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:23.872 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:05:23.879 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:23.879+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:23.980 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:23.979+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:23.988 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:24.006 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:24.025 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:24.041 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:24.041 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:05:24.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:05:24.041 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:24.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.041 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:05:24.042 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:24.042 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 142.931799754, 2023-07-02T14:05:24.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:05:24.042 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:24.042 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:24.042 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:24.042 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:24.042 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:24.043 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:05:24.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.043 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:24.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:24.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:24.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:24.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:24.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:24.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:24.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:24.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:05:24.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:05:24.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:24.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:05:24.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:05:24.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:24.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:05:24.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:05:24.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:05:24.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:05:24.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:05:24.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:05:24.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:24.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:05:24.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:05:24.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:05:24.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:05:24.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:24.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:05:24.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:05:24.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:24.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:05:24.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:05:24.050 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.050 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.050 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.051 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:05:24.051 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:24.051 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 142.93173901599999, 2023-07-02T14:05:24.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:05:24.051 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:24.051 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.051 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:24.051 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:24.052 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:24.052 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:05:24.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.052 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:24.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:24.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:24.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:24.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:24.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:24.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:24.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:24.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:05:24.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:05:24.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:24.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:05:24.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:05:24.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:24.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:05:24.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:05:24.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:05:24.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:05:24.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:05:24.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:05:24.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:24.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:05:24.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:05:24.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:05:24.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:05:24.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:24.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:05:24.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:05:24.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:05:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:05:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:05:24.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:05:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:05:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:05:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:05:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:05:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:05:24.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:05:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:24.062 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 132.70128468600001, 2023-07-02T14:05:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:05:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:05:24.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.064 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:24.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:24.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:24.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:24.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:24.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:24.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:24.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:24.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:05:24.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:05:24.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:05:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:05:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:24.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:05:24.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:05:24.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:05:24.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:05:24.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:05:24.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:05:24.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:24.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:05:24.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:05:24.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:24.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:05:24.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:05:24.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:24.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:05:24.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:05:24.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:24.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:05:24.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:05:24.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:24.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:05:24.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:05:24.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:24.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:05:24.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:05:24.072 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.072 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.072 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.073 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2950 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 1507328~786432] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:24.073 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:24.073 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 132.56400842400001, 2023-07-02T14:05:24.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999, 2023-07-02T14:05:24.073 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:24.073 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.073 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:24.073 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:24.074 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:24.074 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2950 2023-07-02T14:05:24.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.074 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:24.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:24.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:24.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:24.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:24.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:24.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:24.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:24.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:24.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001147854 2023-07-02T14:05:24.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:24.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476614+0000", 2023-07-02T14:05:24.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.802e-05 2023-07-02T14:05:24.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476632+0000", 2023-07-02T14:05:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1860000000000003e-06 2023-07-02T14:05:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476639+0000", 2023-07-02T14:05:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061211240000000004 2023-07-02T14:05:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482760+0000", 2023-07-02T14:05:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8833999999999996e-05 2023-07-02T14:05:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:24.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482859+0000", 2023-07-02T14:05:24.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019270600000000001 2023-07-02T14:05:24.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:24.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.483051+0000", 2023-07-02T14:05:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060385630000000003 2023-07-02T14:05:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489090+0000", 2023-07-02T14:05:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024506918999999999 2023-07-02T14:05:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:24.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513597+0000", 2023-07-02T14:05:24.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9672999999999995e-05 2023-07-02T14:05:24.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:24.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513667+0000", 2023-07-02T14:05:24.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024329500000000001 2023-07-02T14:05:24.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:24.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513910+0000", 2023-07-02T14:05:24.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999 2023-07-02T14:05:24.082 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.082 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.082 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2951 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2686976~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:05:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 132.56282764599999, 2023-07-02T14:05:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999, 2023-07-02T14:05:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:24.083 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2951 2023-07-02T14:05:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:05:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:05:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:24.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:24.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476646+0000", 2023-07-02T14:05:24.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7639000000000003e-05 2023-07-02T14:05:24.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:24.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476684+0000", 2023-07-02T14:05:24.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3508000000000002e-05 2023-07-02T14:05:24.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:24.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476747+0000", 2023-07-02T14:05:24.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8499999999999994e-06 2023-07-02T14:05:24.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476755+0000", 2023-07-02T14:05:24.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078346160000000008 2023-07-02T14:05:24.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484590+0000", 2023-07-02T14:05:24.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7206000000000004e-05 2023-07-02T14:05:24.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:24.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484687+0000", 2023-07-02T14:05:24.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016243099999999999 2023-07-02T14:05:24.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:24.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484849+0000", 2023-07-02T14:05:24.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042482860000000004 2023-07-02T14:05:24.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:24.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489097+0000", 2023-07-02T14:05:24.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.02489504 2023-07-02T14:05:24.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:24.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513992+0000", 2023-07-02T14:05:24.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9972000000000001e-05 2023-07-02T14:05:24.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:24.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514052+0000", 2023-07-02T14:05:24.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149264 2023-07-02T14:05:24.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:24.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514202+0000", 2023-07-02T14:05:24.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999 2023-07-02T14:05:24.092 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.093 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.093 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.093 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2952 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2883584~12288] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:24.093 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:24.093 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 132.56271381100001, 2023-07-02T14:05:24.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651, 2023-07-02T14:05:24.094 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:24.094 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.094 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:24.094 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:24.094 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:24.094 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2952 2023-07-02T14:05:24.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.094 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:24.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:24.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:24.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476760+0000", 2023-07-02T14:05:24.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3664e-05 2023-07-02T14:05:24.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476774+0000", 2023-07-02T14:05:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.166e-06 2023-07-02T14:05:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476783+0000", 2023-07-02T14:05:24.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8720000000000001e-06 2023-07-02T14:05:24.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476788+0000", 2023-07-02T14:05:24.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087180529999999999 2023-07-02T14:05:24.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485506+0000", 2023-07-02T14:05:24.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5680999999999994e-05 2023-07-02T14:05:24.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:24.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485602+0000", 2023-07-02T14:05:24.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000222365 2023-07-02T14:05:24.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:24.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485824+0000", 2023-07-02T14:05:24.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003277379 2023-07-02T14:05:24.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:24.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489101+0000", 2023-07-02T14:05:24.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025151739999999999 2023-07-02T14:05:24.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:24.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514253+0000", 2023-07-02T14:05:24.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4749000000000002e-05 2023-07-02T14:05:24.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:24.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514288+0000", 2023-07-02T14:05:24.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130634 2023-07-02T14:05:24.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:24.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514419+0000", 2023-07-02T14:05:24.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651 2023-07-02T14:05:24.103 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.103 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.103 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.103 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9620 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 589824~65536] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:05:24.103 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:24.104 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.017270087999997, 2023-07-02T14:05:24.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001, 2023-07-02T14:05:24.104 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:24.104 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.104 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:24.104 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:24.104 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:24.104 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9620 2023-07-02T14:05:24.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.105 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:24.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:24.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:24.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:24.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:24.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:24.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:24.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:24.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022203+0000", 2023-07-02T14:05:24.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0346000000000001e-05 2023-07-02T14:05:24.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:24.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022263+0000", 2023-07-02T14:05:24.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4657999999999999e-05 2023-07-02T14:05:24.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:24.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022278+0000", 2023-07-02T14:05:24.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:05:24.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022285+0000", 2023-07-02T14:05:24.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0096000000000003e-05 2023-07-02T14:05:24.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022335+0000", 2023-07-02T14:05:24.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2334000000000001e-05 2023-07-02T14:05:24.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:24.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022347+0000", 2023-07-02T14:05:24.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095653701999999993 2023-07-02T14:05:24.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118001+0000", 2023-07-02T14:05:24.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000185797 2023-07-02T14:05:24.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:24.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118187+0000", 2023-07-02T14:05:24.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026397900000000001 2023-07-02T14:05:24.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:24.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118451+0000", 2023-07-02T14:05:24.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051832800000000002 2023-07-02T14:05:24.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:24.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.123634+0000", 2023-07-02T14:05:24.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001227332 2023-07-02T14:05:24.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:24.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124861+0000", 2023-07-02T14:05:24.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9493999999999998e-05 2023-07-02T14:05:24.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:24.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124901+0000", 2023-07-02T14:05:24.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0462000000000001e-05 2023-07-02T14:05:24.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:24.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124931+0000", 2023-07-02T14:05:24.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001 2023-07-02T14:05:24.114 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.114 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.115 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.115 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9621 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 655360~1966080] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:05:24.115 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:24.115 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.017082705999997, 2023-07-02T14:05:24.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999, 2023-07-02T14:05:24.115 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:24.116 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.116 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:24.116 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:24.116 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:24.116 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9621 2023-07-02T14:05:24.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.116 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:24.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:24.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:24.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:24.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:24.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:24.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:24.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:24.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022391+0000", 2023-07-02T14:05:24.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032053939999999999 2023-07-02T14:05:24.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:24.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025597+0000", 2023-07-02T14:05:24.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4307999999999999e-05 2023-07-02T14:05:24.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:24.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025611+0000", 2023-07-02T14:05:24.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.032e-06 2023-07-02T14:05:24.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025617+0000", 2023-07-02T14:05:24.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8977999999999998e-05 2023-07-02T14:05:24.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025656+0000", 2023-07-02T14:05:24.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:05:24.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:24.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025663+0000", 2023-07-02T14:05:24.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095220739999999998 2023-07-02T14:05:24.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120884+0000", 2023-07-02T14:05:24.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2525000000000001e-05 2023-07-02T14:05:24.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:24.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120956+0000", 2023-07-02T14:05:24.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016232400000000001 2023-07-02T14:05:24.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:24.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.121119+0000", 2023-07-02T14:05:24.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044007380000000004 2023-07-02T14:05:24.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:24.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.125519+0000", 2023-07-02T14:05:24.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000797209 2023-07-02T14:05:24.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:24.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126317+0000", 2023-07-02T14:05:24.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1228000000000004e-05 2023-07-02T14:05:24.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:24.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126378+0000", 2023-07-02T14:05:24.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978000000000003e-05 2023-07-02T14:05:24.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:24.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126430+0000", 2023-07-02T14:05:24.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999 2023-07-02T14:05:24.126 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.126 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.126 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.127 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:11434 2.1 2:8269d003:::rbd_data.125de1449605.00000000000000ce:head [write 65536~4128768] snapc 1a=[1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e107)", 2023-07-02T14:05:24.127 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:05:24.127 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.121644437, 2023-07-02T14:05:24.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036409966000000002, 2023-07-02T14:05:24.127 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:24.127 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.127 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:24.127 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:24.128 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:24.128 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11434 2023-07-02T14:05:24.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.128 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:24.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:24.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:05:24.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:24.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:24.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:05:24.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:24.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:24.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917828+0000", 2023-07-02T14:05:24.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052044650000000001 2023-07-02T14:05:24.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:24.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923033+0000", 2023-07-02T14:05:24.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0097000000000001e-05 2023-07-02T14:05:24.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923053+0000", 2023-07-02T14:05:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.49e-06 2023-07-02T14:05:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923061+0000", 2023-07-02T14:05:24.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0090999999999998e-05 2023-07-02T14:05:24.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923121+0000", 2023-07-02T14:05:24.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000114095 2023-07-02T14:05:24.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:24.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923236+0000", 2023-07-02T14:05:24.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020094199999999999 2023-07-02T14:05:24.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:24.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923436+0000", 2023-07-02T14:05:24.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088693520000000005 2023-07-02T14:05:24.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:24.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.932306+0000", 2023-07-02T14:05:24.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021874112000000001 2023-07-02T14:05:24.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:24.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954180+0000", 2023-07-02T14:05:24.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4433e-05 2023-07-02T14:05:24.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:24.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954214+0000", 2023-07-02T14:05:24.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5884000000000002e-05 2023-07-02T14:05:24.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:24.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954240+0000", 2023-07-02T14:05:24.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036409966000000002 2023-07-02T14:05:24.136 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.136 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.136 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.137 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:189 2.3 2:c9621172:::rbd_data.125de1449605.00000000000000c0:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:24.137 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:24.137 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.866795414000002, 2023-07-02T14:05:24.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001, 2023-07-02T14:05:24.137 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:24.137 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:24.138 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:24.138 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:24.138 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:24.138 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:05:24.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.138 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:24.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:24.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:24.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:24.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:24.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:24.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:24.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:24.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:24.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2023-07-02T14:05:24.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:24.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172682+0000", 2023-07-02T14:05:24.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0159999999999997e-06 2023-07-02T14:05:24.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:24.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172688+0000", 2023-07-02T14:05:24.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0010000000000003e-06 2023-07-02T14:05:24.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172692+0000", 2023-07-02T14:05:24.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026444333 2023-07-02T14:05:24.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199136+0000", 2023-07-02T14:05:24.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2230000000000004e-05 2023-07-02T14:05:24.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:24.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199218+0000", 2023-07-02T14:05:24.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012043595000000001 2023-07-02T14:05:24.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:24.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211262+0000", 2023-07-02T14:05:24.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001 2023-07-02T14:05:24.144 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.144 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.144 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.145 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:199 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:24.145 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:24.145 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.866751870999998, 2023-07-02T14:05:24.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002, 2023-07-02T14:05:24.145 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:24.145 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:24.145 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:24.145 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:24.146 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:24.146 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 199 2023-07-02T14:05:24.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.146 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:24.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:24.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:24.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:24.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:24.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:24.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:24.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:24.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172722+0000", 2023-07-02T14:05:24.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1250000000000001e-06 2023-07-02T14:05:24.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:24.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172725+0000", 2023-07-02T14:05:24.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5670000000000002e-06 2023-07-02T14:05:24.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:24.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172731+0000", 2023-07-02T14:05:24.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9310000000000001e-06 2023-07-02T14:05:24.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172735+0000", 2023-07-02T14:05:24.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030699852999999999 2023-07-02T14:05:24.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203435+0000", 2023-07-02T14:05:24.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6827e-05 2023-07-02T14:05:24.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:24.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203532+0000", 2023-07-02T14:05:24.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076955230000000001 2023-07-02T14:05:24.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:24.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211227+0000", 2023-07-02T14:05:24.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002 2023-07-02T14:05:24.152 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.152 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.152 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.152 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:281 2.1 2:901d5a95:::rbd_data.125de1449605.0000000000000118:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:24.152 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:24.152 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.484006594999997, 2023-07-02T14:05:24.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002, 2023-07-02T14:05:24.153 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:24.153 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:24.153 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:24.153 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:24.153 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:24.153 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:05:24.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.154 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:24.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:24.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:24.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:24.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:24.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:24.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:24.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:24.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:24.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0469999999999998e-06 2023-07-02T14:05:24.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:24.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555471+0000", 2023-07-02T14:05:24.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8860000000000001e-06 2023-07-02T14:05:24.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:24.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555476+0000", 2023-07-02T14:05:24.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3160000000000002e-06 2023-07-02T14:05:24.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555483+0000", 2023-07-02T14:05:24.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989525000000001 2023-07-02T14:05:24.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591472+0000", 2023-07-02T14:05:24.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3708999999999996e-05 2023-07-02T14:05:24.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:24.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591536+0000", 2023-07-02T14:05:24.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055109720000000003 2023-07-02T14:05:24.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:24.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.597047+0000", 2023-07-02T14:05:24.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002 2023-07-02T14:05:24.159 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.159 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.159 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.160 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12883 2.1 2:9ffc48e1:::rbd_data.125de1449605.0000000000000074:head [write 4173824~20480] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:05:24.160 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:24.160 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.393404122, 2023-07-02T14:05:24.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037871393000000003, 2023-07-02T14:05:24.160 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:24.161 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.161 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:24.161 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:24.161 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:24.161 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12883 2023-07-02T14:05:24.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.161 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:24.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:24.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:24.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:24.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:24.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:24.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:05:24.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:24.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646068+0000", 2023-07-02T14:05:24.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4935999999999999e-05 2023-07-02T14:05:24.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:24.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646103+0000", 2023-07-02T14:05:24.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.092e-05 2023-07-02T14:05:24.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:24.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646124+0000", 2023-07-02T14:05:24.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1519999999999998e-06 2023-07-02T14:05:24.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646133+0000", 2023-07-02T14:05:24.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012270200000000001 2023-07-02T14:05:24.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646255+0000", 2023-07-02T14:05:24.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014495400000000001 2023-07-02T14:05:24.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:24.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646400+0000", 2023-07-02T14:05:24.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000221626 2023-07-02T14:05:24.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:24.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646622+0000", 2023-07-02T14:05:24.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078929050000000004 2023-07-02T14:05:24.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:24.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.654515+0000", 2023-07-02T14:05:24.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029307343999999999 2023-07-02T14:05:24.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:24.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683822+0000", 2023-07-02T14:05:24.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8298000000000002e-05 2023-07-02T14:05:24.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:24.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683891+0000", 2023-07-02T14:05:24.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1381999999999997e-05 2023-07-02T14:05:24.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:24.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683942+0000", 2023-07-02T14:05:24.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037871393000000003 2023-07-02T14:05:24.170 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.170 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.170 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.170 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12985 2.7 2:eeddf24b:::rbd_data.125de1449605.000000000000007e:head [write 65536~4128768] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:05:24.170 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:24.170 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.256479648999999, 2023-07-02T14:05:24.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999, 2023-07-02T14:05:24.171 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:24.171 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.171 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:24.171 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:24.171 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:24.171 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12985 2023-07-02T14:05:24.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.172 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:24.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:24.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:24.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:24.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:24.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:24.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:24.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:24.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782994+0000", 2023-07-02T14:05:24.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019254741999999998 2023-07-02T14:05:24.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:24.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802248+0000", 2023-07-02T14:05:24.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0451e-05 2023-07-02T14:05:24.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:24.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802279+0000", 2023-07-02T14:05:24.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3318e-05 2023-07-02T14:05:24.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802292+0000", 2023-07-02T14:05:24.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109403 2023-07-02T14:05:24.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802402+0000", 2023-07-02T14:05:24.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012520599999999999 2023-07-02T14:05:24.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:24.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802527+0000", 2023-07-02T14:05:24.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187768 2023-07-02T14:05:24.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:05:24.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802715+0000", 2023-07-02T14:05:24.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013192822 2023-07-02T14:05:24.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:24.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.815907+0000", 2023-07-02T14:05:24.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0086451870000000004 2023-07-02T14:05:24.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:24.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824553+0000", 2023-07-02T14:05:24.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6759999999999999e-05 2023-07-02T14:05:24.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:24.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824609+0000", 2023-07-02T14:05:24.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011314700000000001 2023-07-02T14:05:24.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:24.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824722+0000", 2023-07-02T14:05:24.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999 2023-07-02T14:05:24.180 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.180 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.180 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.180 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:165 2.3 2:c9c6393a:::rbd_data.125de1449605.000000000000002c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:24.181 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:24.181 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.199696832000001, 2023-07-02T14:05:24.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002, 2023-07-02T14:05:24.181 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:24.181 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:24.181 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:24.181 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:24.181 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:24.182 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 165 2023-07-02T14:05:24.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.182 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:24.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:24.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:24.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:24.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:24.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:24.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:24.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:24.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839777+0000", 2023-07-02T14:05:24.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6740000000000001e-06 2023-07-02T14:05:24.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:24.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839780+0000", 2023-07-02T14:05:24.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9030000000000004e-06 2023-07-02T14:05:24.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:24.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839788+0000", 2023-07-02T14:05:24.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1799999999999998e-06 2023-07-02T14:05:24.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839792+0000", 2023-07-02T14:05:24.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.031592227000000001 2023-07-02T14:05:24.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871384+0000", 2023-07-02T14:05:24.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013600099999999999 2023-07-02T14:05:24.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871520+0000", 2023-07-02T14:05:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082817089999999999 2023-07-02T14:05:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.879802+0000", 2023-07-02T14:05:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002 2023-07-02T14:05:24.187 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.188 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.188 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.188 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:252 2.7 2:f1d0abf0:::rbd_data.125de1449605.0000000000000083:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:24.188 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:24.188 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.783411862000001, 2023-07-02T14:05:24.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997, 2023-07-02T14:05:24.189 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:24.189 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:24.189 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:24.189 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:24.189 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:24.189 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 252 2023-07-02T14:05:24.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.189 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256062+0000", 2023-07-02T14:05:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0149999999999999e-06 2023-07-02T14:05:24.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256065+0000", 2023-07-02T14:05:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5269999999999998e-06 2023-07-02T14:05:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:24.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256071+0000", 2023-07-02T14:05:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:05:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256074+0000", 2023-07-02T14:05:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028852909 2023-07-02T14:05:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.284927+0000", 2023-07-02T14:05:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168274 2023-07-02T14:05:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.285096+0000", 2023-07-02T14:05:24.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011036300000000001 2023-07-02T14:05:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296132+0000", 2023-07-02T14:05:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997 2023-07-02T14:05:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.195 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:261 2.1 2:971731cb:::rbd_data.125de1449605.000000000000008c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.783316386999999, 2023-07-02T14:05:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984, 2023-07-02T14:05:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:24.196 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 261 2023-07-02T14:05:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:24.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2819999999999999e-06 2023-07-02T14:05:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:24.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256160+0000", 2023-07-02T14:05:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2020000000000003e-06 2023-07-02T14:05:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256165+0000", 2023-07-02T14:05:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2100000000000003e-06 2023-07-02T14:05:24.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256170+0000", 2023-07-02T14:05:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025871715999999999 2023-07-02T14:05:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282041+0000", 2023-07-02T14:05:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132364 2023-07-02T14:05:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282174+0000", 2023-07-02T14:05:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014044566 2023-07-02T14:05:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296218+0000", 2023-07-02T14:05:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984 2023-07-02T14:05:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:264 2.3 2:dbf06c9d:::rbd_data.125de1449605.000000000000008f:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.783282269000001, 2023-07-02T14:05:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003, 2023-07-02T14:05:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:24.204 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 264 2023-07-02T14:05:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:24.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5749999999999999e-06 2023-07-02T14:05:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256195+0000", 2023-07-02T14:05:24.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.327e-06 2023-07-02T14:05:24.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:24.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256200+0000", 2023-07-02T14:05:24.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0580000000000002e-06 2023-07-02T14:05:24.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256203+0000", 2023-07-02T14:05:24.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035707524999999997 2023-07-02T14:05:24.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.291911+0000", 2023-07-02T14:05:24.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014269799999999999 2023-07-02T14:05:24.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:24.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.292053+0000", 2023-07-02T14:05:24.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062760309999999996 2023-07-02T14:05:24.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:24.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.298329+0000", 2023-07-02T14:05:24.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003 2023-07-02T14:05:24.211 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.211 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.211 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.211 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:341 2.3 2:cee258bd:::rbd_data.125de1449605.00000000000000dc:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:24.212 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:24.212 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.402794942, 2023-07-02T14:05:24.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997, 2023-07-02T14:05:24.212 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:24.212 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:24.212 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:24.212 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:24.212 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:24.213 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 341 2023-07-02T14:05:24.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.213 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:24.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:24.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:24.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:24.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:24.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:24.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:24.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:24.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636679+0000", 2023-07-02T14:05:24.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6240000000000002e-06 2023-07-02T14:05:24.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:24.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636682+0000", 2023-07-02T14:05:24.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6910000000000002e-06 2023-07-02T14:05:24.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:24.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636688+0000", 2023-07-02T14:05:24.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3469999999999999e-06 2023-07-02T14:05:24.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636691+0000", 2023-07-02T14:05:24.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035625560000000001 2023-07-02T14:05:24.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672317+0000", 2023-07-02T14:05:24.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108391 2023-07-02T14:05:24.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:24.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672425+0000", 2023-07-02T14:05:24.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012104976 2023-07-02T14:05:24.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:24.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.684530+0000", 2023-07-02T14:05:24.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997 2023-07-02T14:05:24.218 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.219 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.219 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.219 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPushReply(2.7 139/134 [PushReplyOp(2:e3782f39:::rbd_data.125de1449605.0000000000000120:head)])", 2023-07-02T14:05:24.219 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:05:24.219 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 12.112262789000001, 2023-07-02T14:05:24.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998, 2023-07-02T14:05:24.220 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:24.220 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:24.220 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:24.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:24.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:05:24.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:24.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:24.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:05:24.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:24.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:24.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927211+0000", 2023-07-02T14:05:24.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3689999999999998e-06 2023-07-02T14:05:24.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:24.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927217+0000", 2023-07-02T14:05:24.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7269999999999996e-06 2023-07-02T14:05:24.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:24.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927225+0000", 2023-07-02T14:05:24.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:05:24.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927230+0000", 2023-07-02T14:05:24.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:05:24.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:24.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927266+0000", 2023-07-02T14:05:24.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.069972871000000006 2023-07-02T14:05:24.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:24.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:24.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:24.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.997239+0000", 2023-07-02T14:05:24.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998 2023-07-02T14:05:24.225 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.225 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.225 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.225 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:24.225 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:24.226 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:24.227 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:24.227 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:05:24.186+0000 7fab0b157700 0 -- 172.21.15.163:0/1954902621 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7faae8005540 msgr2=0x7faae8005970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:24.227 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:05:24.189+0000 7fab0b157700 0 -- 172.21.15.163:0/1954902621 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7faaf0036c10 msgr2=0x7faaf0039000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:24.228 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:05:24.228 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:24.229 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:05:24.229 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-07-02T14:05:24.230 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:05:24.230 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:05:24.230 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:05:24.230 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:05:24.230 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:05:24.230 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:05:24.231 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:05:24.232 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:24.226+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:24.327 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:24.326+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:24.373 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:24.373 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:24.373 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:24.373 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:24.385 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:05:24.467 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:24.467+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:24.528 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:24.527+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:24.554 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:24.554 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:24.554 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:24.554 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:24.554 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:24.564 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:05:24.628 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:24.627+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:24.676 INFO:teuthology.orchestra.run.smithi016.stdout:2.5 2023-07-02T14:05:24.676 INFO:teuthology.orchestra.run.smithi016.stdout:2.4 2023-07-02T14:05:24.676 INFO:teuthology.orchestra.run.smithi016.stdout:2.2 2023-07-02T14:05:24.676 INFO:teuthology.orchestra.run.smithi016.stdout:1.0 2023-07-02T14:05:24.729 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:24.728+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:24.757 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:24.757 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:24.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:24.757 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:24.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.758 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:05:24.758 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:24.758 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 104.85804221399999, 2023-07-02T14:05:24.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:05:24.758 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:24.758 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.759 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:24.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:24.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:24.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:24.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:24.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:24.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:24.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:24.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:05:24.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:05:24.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:24.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:05:24.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:05:24.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:24.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:05:24.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:05:24.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:05:24.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:05:24.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:24.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:05:24.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:05:24.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:24.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:05:24.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:05:24.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:24.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:05:24.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:05:24.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:24.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:05:24.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:05:24.765 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.765 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.765 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.766 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:05:24.766 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:24.766 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 104.853956496, 2023-07-02T14:05:24.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:05:24.766 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:24.766 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.766 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:24.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:24.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:24.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:24.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:24.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:24.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:24.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:24.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:05:24.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:05:24.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:24.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:05:24.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:05:24.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:24.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:05:24.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:05:24.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:05:24.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:05:24.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:24.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:05:24.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:05:24.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:24.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:05:24.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:05:24.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:24.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:05:24.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:05:24.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:24.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:05:24.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:05:24.773 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.773 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.773 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.773 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:05:24.774 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:24.774 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 104.850091715, 2023-07-02T14:05:24.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:05:24.774 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:24.774 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.774 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:24.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:24.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:24.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:24.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:24.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:24.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:24.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:24.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:05:24.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:05:24.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:24.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:05:24.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:05:24.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:24.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:05:24.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:05:24.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:05:24.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:05:24.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:24.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:05:24.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:05:24.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:24.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:05:24.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:05:24.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:24.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:05:24.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:05:24.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:24.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:05:24.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:05:24.781 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.781 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.781 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.781 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:05:24.781 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:24.782 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 104.846567454, 2023-07-02T14:05:24.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:05:24.782 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:24.782 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.782 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:24.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:24.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:24.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:24.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:24.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:24.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:24.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:24.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:05:24.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:05:24.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:24.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:05:24.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:05:24.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:24.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:05:24.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:05:24.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:05:24.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:05:24.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:24.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:05:24.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:05:24.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:24.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:05:24.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:05:24.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:24.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:05:24.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:05:24.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:24.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:05:24.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:05:24.789 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.789 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.789 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.789 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:05:24.789 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:24.789 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 104.81086265499999, 2023-07-02T14:05:24.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:05:24.790 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:24.790 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.790 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:24.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:24.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:24.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:24.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:24.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:24.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:24.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:24.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:05:24.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:05:24.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:24.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:05:24.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:05:24.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:24.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:05:24.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:05:24.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:05:24.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:05:24.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:24.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:05:24.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:05:24.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:24.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:05:24.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:05:24.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:24.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:05:24.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:05:24.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:24.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:05:24.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:05:24.796 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.797 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.797 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.797 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:05:24.797 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:24.797 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 104.80620529700001, 2023-07-02T14:05:24.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:05:24.798 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:24.798 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.798 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:24.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:24.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:24.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:24.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:24.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:24.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:24.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:24.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:05:24.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:05:24.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:24.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:05:24.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:05:24.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:24.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:05:24.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:05:24.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:05:24.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:05:24.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:24.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:05:24.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:05:24.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:24.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:05:24.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:05:24.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:24.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:05:24.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:05:24.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:24.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:05:24.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:05:24.804 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.804 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.804 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.805 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:05:24.805 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:24.805 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 104.80526543400001, 2023-07-02T14:05:24.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:05:24.805 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:24.806 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.806 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:24.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:24.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:24.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:24.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:24.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:24.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:24.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:24.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:05:24.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:05:24.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:24.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:05:24.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:05:24.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:24.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:05:24.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:05:24.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:05:24.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:05:24.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:24.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:05:24.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:05:24.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:24.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:05:24.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:05:24.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:24.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:05:24.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:05:24.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:24.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:05:24.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:05:24.812 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.812 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.812 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.813 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:05:24.813 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:24.813 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 104.803867556, 2023-07-02T14:05:24.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:05:24.813 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:24.813 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.814 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:24.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:24.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:24.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:24.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:24.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:24.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:24.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:24.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:05:24.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:05:24.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:24.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:05:24.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:05:24.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:24.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:05:24.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:05:24.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:05:24.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:05:24.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:24.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:05:24.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:05:24.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:24.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:05:24.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:05:24.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:24.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:05:24.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:05:24.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:24.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:05:24.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:05:24.820 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.820 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.820 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.821 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:05:24.821 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:24.821 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 83.393677060000002, 2023-07-02T14:05:24.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:05:24.821 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:24.821 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.821 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:24.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:24.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:24.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:24.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:24.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:24.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:24.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:24.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:05:24.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:05:24.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:24.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:05:24.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:05:24.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:24.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:05:24.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:05:24.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:05:24.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:05:24.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:24.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:05:24.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:05:24.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:24.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:05:24.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:05:24.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:24.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:05:24.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:05:24.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:24.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:05:24.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:05:24.828 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.828 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.828 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.828 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:24.829 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:24.829 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 82.027327376000002, 2023-07-02T14:05:24.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:05:24.829 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:24.829 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.829 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:24.829 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:24.829 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:24.830 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:05:24.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.830 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:24.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:24.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:24.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:24.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:24.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:24.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:24.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:24.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:05:24.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:05:24.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:24.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:05:24.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:05:24.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:24.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:05:24.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:05:24.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:05:24.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:05:24.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:24.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:05:24.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:05:24.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:24.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:05:24.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:05:24.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:24.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:05:24.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:05:24.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:24.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:05:24.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:05:24.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:24.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:05:24.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:05:24.837 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.837 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.837 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.838 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7707 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 3145728~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:24.838 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:24.838 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 82.023343909000005, 2023-07-02T14:05:24.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999, 2023-07-02T14:05:24.838 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:24.838 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.838 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:24.838 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:24.839 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:24.839 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7707 2023-07-02T14:05:24.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.839 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:24.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:24.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:24.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:24.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:24.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:24.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:24.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:24.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:24.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5017e-05 2023-07-02T14:05:24.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:24.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732347+0000", 2023-07-02T14:05:24.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5821e-05 2023-07-02T14:05:24.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:24.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732363+0000", 2023-07-02T14:05:24.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1270000000000002e-06 2023-07-02T14:05:24.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732368+0000", 2023-07-02T14:05:24.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008444666 2023-07-02T14:05:24.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:24.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740812+0000", 2023-07-02T14:05:24.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011266599999999999 2023-07-02T14:05:24.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:24.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740925+0000", 2023-07-02T14:05:24.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010848808999999999 2023-07-02T14:05:24.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:24.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751774+0000", 2023-07-02T14:05:24.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8421999999999997e-05 2023-07-02T14:05:24.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:24.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751832+0000", 2023-07-02T14:05:24.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107453 2023-07-02T14:05:24.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:24.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751940+0000", 2023-07-02T14:05:24.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999 2023-07-02T14:05:24.846 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.846 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.846 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.847 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:24.847 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:24.847 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 82.023280032000002, 2023-07-02T14:05:24.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:05:24.847 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:24.847 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.847 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:24.847 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:24.848 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:24.848 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:05:24.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.848 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:24.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:24.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:24.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:24.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:24.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:24.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:24.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:24.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:24.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:05:24.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:24.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:05:24.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:05:24.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:24.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:05:24.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:05:24.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:05:24.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:05:24.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:24.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:05:24.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:05:24.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:24.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:05:24.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:05:24.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:24.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:05:24.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:05:24.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:24.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:05:24.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:05:24.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:24.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:05:24.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:05:24.855 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.855 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.855 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.856 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:24.856 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:24.856 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 82.018716206999997, 2023-07-02T14:05:24.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:05:24.856 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:24.856 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.856 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:24.857 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:24.857 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:24.857 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:05:24.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.857 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:24.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:24.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:24.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:24.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:24.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:24.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:24.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:24.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:05:24.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:05:24.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:24.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:05:24.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:05:24.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:24.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:05:24.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:05:24.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:05:24.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:05:24.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:24.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:05:24.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:05:24.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:24.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:05:24.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:05:24.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:24.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:05:24.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:05:24.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:24.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:05:24.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:05:24.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:24.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:05:24.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:05:24.864 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.864 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.864 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.865 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:24.865 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:24.865 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 82.018666340999999, 2023-07-02T14:05:24.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:05:24.865 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:24.865 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.866 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:24.866 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:24.866 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:24.866 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:05:24.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.866 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:24.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:24.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:24.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:24.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:24.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:24.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:24.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:24.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:05:24.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:05:24.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:24.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:05:24.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:05:24.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:24.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:05:24.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:05:24.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:05:24.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:05:24.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:24.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:05:24.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:05:24.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:24.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:05:24.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:05:24.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:24.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:05:24.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:05:24.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:24.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:05:24.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:05:24.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:24.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:05:24.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:05:24.873 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.873 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.873 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.874 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:24.874 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:24.874 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 82.017627348000005, 2023-07-02T14:05:24.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:05:24.874 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:24.874 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.875 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:24.875 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:24.875 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:24.875 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:05:24.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.875 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:24.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:24.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:24.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:24.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:24.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:24.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:24.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:24.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:05:24.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:05:24.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:24.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:05:24.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:05:24.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:24.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:05:24.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:05:24.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:05:24.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:05:24.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:24.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:05:24.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:05:24.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:24.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:05:24.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:05:24.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:24.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:05:24.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:05:24.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:24.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:05:24.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:05:24.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:24.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:05:24.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:05:24.882 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.882 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.882 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.883 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7993 2.7 2:fe72db53:::rbd_data.125de1449605.0000000000000072:head [write 65536~3276800] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:24.883 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:24.883 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 81.830669299999997, 2023-07-02T14:05:24.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919, 2023-07-02T14:05:24.883 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:24.884 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.884 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:24.884 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:24.884 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:24.884 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7993 2023-07-02T14:05:24.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.884 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:24.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:24.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:24.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:24.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:24.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:24.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:24.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:24.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924986+0000", 2023-07-02T14:05:24.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049542070000000004 2023-07-02T14:05:24.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:24.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929940+0000", 2023-07-02T14:05:24.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9740000000000001e-05 2023-07-02T14:05:24.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:24.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929960+0000", 2023-07-02T14:05:24.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.311e-05 2023-07-02T14:05:24.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929973+0000", 2023-07-02T14:05:24.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9629000000000002e-05 2023-07-02T14:05:24.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:24.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930023+0000", 2023-07-02T14:05:24.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012751799999999999 2023-07-02T14:05:24.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:24.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930150+0000", 2023-07-02T14:05:24.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016449042000000001 2023-07-02T14:05:24.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:24.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946599+0000", 2023-07-02T14:05:24.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3241e-05 2023-07-02T14:05:24.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:24.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946663+0000", 2023-07-02T14:05:24.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000102296 2023-07-02T14:05:24.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:24.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946765+0000", 2023-07-02T14:05:24.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919 2023-07-02T14:05:24.891 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.891 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.892 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.892 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11782 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 196608~2162688] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:05:24.892 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:24.892 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.346149378, 2023-07-02T14:05:24.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365, 2023-07-02T14:05:24.892 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:24.893 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.893 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:24.893 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:24.893 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:24.893 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11782 2023-07-02T14:05:24.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.893 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:24.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:24.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:24.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:24.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:24.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:24.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:24.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:24.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409506+0000", 2023-07-02T14:05:24.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020172269999999999 2023-07-02T14:05:24.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:24.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411523+0000", 2023-07-02T14:05:24.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8901000000000001e-05 2023-07-02T14:05:24.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:24.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411542+0000", 2023-07-02T14:05:24.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5037e-05 2023-07-02T14:05:24.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411557+0000", 2023-07-02T14:05:24.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001931213 2023-07-02T14:05:24.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:24.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413489+0000", 2023-07-02T14:05:24.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000149207 2023-07-02T14:05:24.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:24.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413638+0000", 2023-07-02T14:05:24.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024752800000000001 2023-07-02T14:05:24.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:24.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413885+0000", 2023-07-02T14:05:24.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048516000000000002 2023-07-02T14:05:24.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:24.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.418737+0000", 2023-07-02T14:05:24.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010319259000000001 2023-07-02T14:05:24.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:24.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429056+0000", 2023-07-02T14:05:24.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2518000000000002e-05 2023-07-02T14:05:24.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:24.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429099+0000", 2023-07-02T14:05:24.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015275 2023-07-02T14:05:24.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:24.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429251+0000", 2023-07-02T14:05:24.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365 2023-07-02T14:05:24.902 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.902 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.902 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.902 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPushReply(2.7 119/117 [PushReplyOp(2:eb862bc0:::rbd_data.125de1449605.0000000000000040:13)])", 2023-07-02T14:05:24.902 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:24.902 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.344986102999997, 2023-07-02T14:05:24.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999, 2023-07-02T14:05:24.903 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:24.903 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "reached pg", 2023-07-02T14:05:24.903 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:24.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:24.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:24.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:24.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:24.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:24.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:24.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:24.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:24.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7670000000000001e-06 2023-07-02T14:05:24.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:24.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410674+0000", 2023-07-02T14:05:24.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3973e-05 2023-07-02T14:05:24.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:24.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410688+0000", 2023-07-02T14:05:24.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0157e-05 2023-07-02T14:05:24.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410708+0000", 2023-07-02T14:05:24.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018578779 2023-07-02T14:05:24.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:24.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429287+0000", 2023-07-02T14:05:24.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013086549999999999 2023-07-02T14:05:24.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:24.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.430595+0000", 2023-07-02T14:05:24.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999 2023-07-02T14:05:24.908 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.908 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.908 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.909 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11861 2.7 2:e57d3f55:::rbd_data.125de1449605.0000000000000134:head [write 0~65536] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:05:24.909 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:24.909 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.210870726000003, 2023-07-02T14:05:24.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723, 2023-07-02T14:05:24.909 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:24.910 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.910 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:24.910 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:24.910 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:24.910 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11861 2023-07-02T14:05:24.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.910 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:24.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:24.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:24.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:24.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:24.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:24.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:24.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:24.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544785+0000", 2023-07-02T14:05:24.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010563700000000001 2023-07-02T14:05:24.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:24.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544890+0000", 2023-07-02T14:05:24.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8530000000000004e-06 2023-07-02T14:05:24.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:24.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544900+0000", 2023-07-02T14:05:24.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.665e-06 2023-07-02T14:05:24.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544906+0000", 2023-07-02T14:05:24.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000272575 2023-07-02T14:05:24.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:24.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545178+0000", 2023-07-02T14:05:24.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000111362 2023-07-02T14:05:24.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:24.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545290+0000", 2023-07-02T14:05:24.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016181100000000001 2023-07-02T14:05:24.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:24.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545452+0000", 2023-07-02T14:05:24.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044500750000000004 2023-07-02T14:05:24.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:24.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.549902+0000", 2023-07-02T14:05:24.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014850024 2023-07-02T14:05:24.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:24.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564752+0000", 2023-07-02T14:05:24.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6272000000000003e-05 2023-07-02T14:05:24.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:24.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564848+0000", 2023-07-02T14:05:24.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000162428 2023-07-02T14:05:24.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:24.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.565010+0000", 2023-07-02T14:05:24.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723 2023-07-02T14:05:24.919 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.919 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.919 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.919 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:13262 2.7 e125/121 2:e0741dde:::rbd_data.125de1449605.00000000000000ad:head v 125'2024, mlcod=125'2021)", 2023-07-02T14:05:24.919 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:24.919 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.504792987999998, 2023-07-02T14:05:24.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999, 2023-07-02T14:05:24.920 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:24.920 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:24.920 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:24.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:24.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:24.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:24.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:24.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:24.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:24.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:24.922 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250862+0000", 2023-07-02T14:05:24.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080378800000000007 2023-07-02T14:05:24.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:24.922 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258900+0000", 2023-07-02T14:05:24.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8939999999999999e-06 2023-07-02T14:05:24.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:24.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258909+0000", 2023-07-02T14:05:24.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3560000000000003e-06 2023-07-02T14:05:24.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:24.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258914+0000", 2023-07-02T14:05:24.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2429000000000003e-05 2023-07-02T14:05:24.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:24.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258956+0000", 2023-07-02T14:05:24.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5094999999999999e-05 2023-07-02T14:05:24.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:24.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258981+0000", 2023-07-02T14:05:24.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012177307 2023-07-02T14:05:24.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:24.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271158+0000", 2023-07-02T14:05:24.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2104999999999999e-05 2023-07-02T14:05:24.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:24.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:24.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:24.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271181+0000", 2023-07-02T14:05:24.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999 2023-07-02T14:05:24.926 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.927 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.927 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.927 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:24.927 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:24.927 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:24.929 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:05:24.929 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:24.928+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:24.930 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete...2023-07-02T14:05:24.852+0000 7fab0b157700 0 -- 172.21.15.163:0/1954902621 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7faae8005540 msgr2=0x7faaf003b770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:24.930 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 28% complete...2023-07-02T14:05:24.861+0000 7fab0b157700 0 -- 172.21.15.163:0/1954902621 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7faaf0036c10 msgr2=0x7faaf0039000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:24.930 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:05:24.866+0000 7fab0b157700 0 -- 172.21.15.163:0/1954902621 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7faae8005540 msgr2=0x7faaf003b5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:25.009 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete...2023-07-02T14:05:25.009+0000 7fab0c159700 0 -- 172.21.15.163:0/1954902621 >> [v2:172.21.15.22:6808/99977,v1:172.21.15.22:6809/99977] conn(0x5613bac82380 msgr2=0x5613bac84770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:25.028 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:25.027+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:25.117 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:25.117 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:25.117 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:25.117 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:25.128 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:05:25.129 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:25.128+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:25.201 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op export --pgid 2.4 --file /var/log/ceph/exp.2.4.0 2023-07-02T14:05:25.230 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:25.229+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:25.267 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:25.267 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:25.267 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:25.268 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:25.268 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:25.280 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:05:25.362 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:25.362+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:25.388 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 100% complete...done. 2023-07-02T14:05:25.411 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:05:25.411 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=37 2023-07-02T14:05:25.412 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap37 2023-07-02T14:05:25.431 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:25.431+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:25.451 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:25.451 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:25.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:25.451 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:25.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.452 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:25.452 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:25.452 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.15805494, 2023-07-02T14:05:25.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:05:25.452 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:25.452 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:25.452 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:25.453 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:25.453 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:25.453 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:05:25.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.453 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:25.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:25.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:25.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:25.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:25.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:25.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:25.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:25.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:05:25.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:05:25.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:25.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:05:25.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:05:25.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:25.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:05:25.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:05:25.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:25.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:05:25.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:05:25.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:25.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:05:25.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:05:25.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:25.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:05:25.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:05:25.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:25.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:05:25.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:05:25.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:25.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:05:25.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:05:25.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:25.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:05:25.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:05:25.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:25.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:05:25.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:05:25.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:25.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:05:25.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:05:25.462 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.462 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.462 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.462 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:25.462 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:25.462 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.15781991700001, 2023-07-02T14:05:25.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:05:25.463 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:25.463 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:25.463 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:25.463 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:25.463 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:25.463 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:05:25.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.464 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:25.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:25.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:25.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:25.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:25.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:25.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:25.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:25.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:05:25.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:05:25.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:25.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:05:25.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:05:25.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:25.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:05:25.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:05:25.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:25.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:05:25.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:05:25.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:25.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:05:25.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:05:25.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:25.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:05:25.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:05:25.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:25.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:05:25.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:05:25.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:25.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:05:25.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:05:25.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:25.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:05:25.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:05:25.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:25.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:05:25.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:05:25.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:25.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:05:25.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:05:25.472 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.472 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.472 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.472 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:25.473 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:25.473 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.157620783, 2023-07-02T14:05:25.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:05:25.473 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:25.473 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:25.473 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:25.473 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:25.473 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:25.474 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:05:25.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.474 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:25.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:25.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:25.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:25.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:25.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:25.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:25.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:25.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:05:25.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:05:25.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:25.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:05:25.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:05:25.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:25.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:05:25.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:05:25.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:25.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:05:25.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:05:25.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:25.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:05:25.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:05:25.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:25.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:05:25.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:05:25.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:25.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:05:25.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:05:25.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:25.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:05:25.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:05:25.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:25.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:05:25.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:05:25.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:25.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:05:25.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:05:25.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:25.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:05:25.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:05:25.482 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.482 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.482 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.483 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:25.483 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:25.483 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.15725565899999, 2023-07-02T14:05:25.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:05:25.483 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:25.483 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:25.483 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:25.484 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:25.484 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:25.484 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:05:25.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.484 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:25.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:25.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:25.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:25.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:25.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:25.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:25.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:25.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:05:25.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:05:25.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:25.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:05:25.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:05:25.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:25.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:05:25.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:05:25.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:25.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:05:25.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:05:25.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:25.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:05:25.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:05:25.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:25.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:05:25.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:05:25.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:25.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:05:25.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:05:25.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:25.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:05:25.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:05:25.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:25.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:05:25.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:05:25.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:25.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:05:25.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:05:25.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:25.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:05:25.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:05:25.492 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.492 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.493 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.493 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:25.493 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:25.493 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.15717459800001, 2023-07-02T14:05:25.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:05:25.493 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:25.494 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:25.494 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:25.494 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:25.494 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:25.494 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:05:25.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.494 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:25.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:25.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:25.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:25.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:25.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:25.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:25.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:25.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:05:25.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:05:25.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:25.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:05:25.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:05:25.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:25.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:05:25.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:05:25.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:25.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:05:25.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:05:25.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:25.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:05:25.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:05:25.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:25.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:05:25.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:05:25.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:25.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:05:25.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:05:25.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:25.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:05:25.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:05:25.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:25.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:05:25.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:05:25.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:25.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:05:25.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:05:25.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:25.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:05:25.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:05:25.503 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.503 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.503 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.503 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:25.503 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:25.503 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.157082927, 2023-07-02T14:05:25.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:05:25.504 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:25.504 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:25.504 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:25.504 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:25.504 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:25.504 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:05:25.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.505 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:25.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:25.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:25.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:25.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:05:25.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:05:25.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:25.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:05:25.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:05:25.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:25.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:05:25.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:05:25.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:25.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:05:25.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:05:25.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:25.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:05:25.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:05:25.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:25.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:05:25.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:05:25.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:25.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:05:25.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:05:25.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:25.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:05:25.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:05:25.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:25.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:05:25.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:05:25.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:25.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:05:25.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:05:25.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:25.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:05:25.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:05:25.513 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.513 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.513 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.514 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:25.514 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:25.514 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.15674348499999, 2023-07-02T14:05:25.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:05:25.514 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:25.514 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:25.514 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:25.514 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:25.515 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:25.515 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:05:25.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.515 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:25.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:25.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:25.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:25.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:25.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:25.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:25.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:25.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:05:25.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:05:25.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:25.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:05:25.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:05:25.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:25.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:05:25.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:05:25.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:25.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:05:25.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:05:25.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:25.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:05:25.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:05:25.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:25.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:05:25.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:05:25.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:25.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:05:25.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:05:25.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:25.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:05:25.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:05:25.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:25.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:05:25.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:05:25.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:25.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:05:25.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:05:25.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:25.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:05:25.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:05:25.523 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.523 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.524 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.524 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:25.524 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:25.524 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.156557432, 2023-07-02T14:05:25.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:05:25.524 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:25.525 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:25.525 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:25.525 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:25.525 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:25.525 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:05:25.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.525 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:25.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:25.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:25.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:25.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:25.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:25.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:25.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:25.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:25.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:05:25.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:25.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:05:25.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:05:25.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:25.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:05:25.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:05:25.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:25.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:05:25.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:05:25.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:25.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:05:25.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:05:25.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:25.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:05:25.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:05:25.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:25.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:05:25.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:05:25.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:25.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:05:25.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:05:25.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:25.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:05:25.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:05:25.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:25.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:05:25.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:05:25.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:25.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:05:25.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:05:25.534 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.534 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.534 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.534 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:25.534 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:25.534 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.15639963000001, 2023-07-02T14:05:25.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:05:25.535 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:25.535 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:25.535 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:25.535 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:25.535 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:25.535 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:05:25.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.536 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:25.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:25.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:25.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:25.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:25.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:25.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:25.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:25.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:05:25.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:05:25.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:25.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:05:25.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:05:25.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:25.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:05:25.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:05:25.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:25.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:05:25.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:05:25.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:25.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:05:25.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:05:25.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:25.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:05:25.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:05:25.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:25.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:05:25.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:05:25.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:25.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:05:25.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:05:25.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:25.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:05:25.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:05:25.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:25.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:05:25.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:05:25.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:25.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:05:25.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:05:25.544 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.544 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.544 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.544 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:25.545 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:25.545 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.15620784800001, 2023-07-02T14:05:25.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:05:25.545 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:25.545 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:25.545 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:25.545 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:25.545 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:25.546 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:05:25.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.546 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:25.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:25.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:25.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:25.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:25.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:25.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:25.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:25.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:05:25.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:05:25.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:25.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:05:25.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:05:25.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:25.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:05:25.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:05:25.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:25.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:05:25.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:05:25.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:25.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:05:25.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:05:25.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:25.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:05:25.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:05:25.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:25.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:05:25.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:05:25.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:25.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:05:25.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:05:25.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:25.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:05:25.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:05:25.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:25.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:05:25.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:05:25.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:25.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:05:25.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:05:25.554 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.554 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.554 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.555 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:25.555 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:25.555 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.15603218800001, 2023-07-02T14:05:25.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:05:25.555 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:25.555 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:25.555 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:25.556 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:25.556 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:25.556 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:05:25.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.556 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:25.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:25.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:25.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:25.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:25.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:25.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:25.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:25.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:25.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:05:25.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:25.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:05:25.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:05:25.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:25.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:05:25.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:05:25.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:25.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:05:25.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:05:25.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:25.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:05:25.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:05:25.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:25.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:05:25.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:05:25.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:25.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:05:25.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:05:25.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:25.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:05:25.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:05:25.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:25.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:05:25.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:05:25.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:25.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:05:25.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:05:25.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:25.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:05:25.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:05:25.564 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.564 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.565 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.565 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:25.565 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:25.565 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.15593799999999, 2023-07-02T14:05:25.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:05:25.565 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:25.566 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:25.566 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:25.566 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:25.566 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:25.566 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:05:25.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.566 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:25.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:25.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:25.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:25.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:25.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:25.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:25.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:25.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:05:25.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:05:25.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:25.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:05:25.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:05:25.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:25.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:05:25.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:05:25.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:25.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:05:25.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:05:25.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:25.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:05:25.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:05:25.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:25.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:05:25.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:05:25.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:25.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:05:25.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:05:25.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:25.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:05:25.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:05:25.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:25.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:05:25.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:05:25.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:25.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:05:25.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:05:25.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:25.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:05:25.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:05:25.575 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.575 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.575 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.575 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:25.575 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:25.575 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.15573461400001, 2023-07-02T14:05:25.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:05:25.576 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:25.576 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:25.576 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:25.576 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:25.576 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:25.576 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:05:25.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.577 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:25.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:25.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:25.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:25.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:25.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:25.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:25.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:25.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:05:25.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:05:25.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:25.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:05:25.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:05:25.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:25.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:05:25.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:05:25.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:25.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:05:25.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:05:25.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:25.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:05:25.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:05:25.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:25.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:05:25.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:05:25.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:25.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:05:25.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:05:25.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:25.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:05:25.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:05:25.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:25.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:05:25.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:05:25.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:25.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:05:25.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:05:25.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:25.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:05:25.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:05:25.585 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.585 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.585 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.585 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:25.586 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:25.586 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.15558395799999, 2023-07-02T14:05:25.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:05:25.586 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:25.586 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:25.586 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:25.586 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:25.586 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:25.587 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:05:25.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.587 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:25.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:25.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:25.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:25.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:25.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:25.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:25.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:25.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:05:25.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:05:25.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:25.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:05:25.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:05:25.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:25.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:05:25.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:05:25.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:25.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:05:25.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:05:25.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:25.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:05:25.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:05:25.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:25.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:05:25.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:05:25.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:25.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:05:25.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:05:25.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:25.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:05:25.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:05:25.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:25.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:05:25.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:05:25.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:25.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:05:25.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:05:25.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:25.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:05:25.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:05:25.595 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.595 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.596 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.596 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:25.596 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:25.596 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.15551125499999, 2023-07-02T14:05:25.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:05:25.596 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:25.597 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:25.597 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:25.597 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:25.597 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:25.597 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:05:25.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.597 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:25.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:25.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:25.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:25.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:25.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:25.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:25.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:25.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:25.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:05:25.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:25.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:05:25.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:05:25.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:25.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:05:25.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:05:25.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:25.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:05:25.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:05:25.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:25.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:05:25.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:05:25.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:25.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:05:25.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:05:25.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:25.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:05:25.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:05:25.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:25.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:05:25.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:05:25.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.604 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:25.604 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:05:25.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:05:25.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:25.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:05:25.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:05:25.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:25.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:05:25.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:05:25.606 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.606 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.606 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.606 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:25.606 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:25.606 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.15539426500001, 2023-07-02T14:05:25.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:05:25.607 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:25.607 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:25.607 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:25.607 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:25.607 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:25.607 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:05:25.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.608 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:25.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:25.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:25.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:25.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:25.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:25.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:25.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:25.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:25.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:05:25.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:25.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:05:25.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:05:25.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:25.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:05:25.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:05:25.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:25.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:05:25.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:05:25.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:25.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:05:25.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:05:25.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:25.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:05:25.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:05:25.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:25.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:05:25.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:05:25.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:25.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:05:25.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:05:25.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:25.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:05:25.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:05:25.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:25.615 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:05:25.615 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:05:25.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:25.616 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:05:25.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:05:25.616 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.616 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.616 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.616 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.616 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:25.617 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:25.617 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.15524481400001, 2023-07-02T14:05:25.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:05:25.617 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:25.617 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:25.617 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:25.617 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:25.617 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:25.618 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:05:25.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.618 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:25.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:25.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:25.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:25.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:25.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:25.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:25.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:25.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:05:25.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:05:25.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:25.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:05:25.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:05:25.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:25.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:05:25.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:05:25.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:25.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:05:25.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:05:25.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:25.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:05:25.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:05:25.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:25.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:05:25.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:05:25.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:25.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:05:25.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:05:25.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:25.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:05:25.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:05:25.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:25.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:05:25.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:05:25.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.625 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:25.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:05:25.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:05:25.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:25.626 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:05:25.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:05:25.626 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.626 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.626 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.627 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:25.627 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:25.627 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.13376884100001, 2023-07-02T14:05:25.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:05:25.627 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:25.627 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:25.627 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:25.628 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:25.628 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:25.628 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:05:25.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.628 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:25.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:25.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:25.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:25.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:25.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:25.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:25.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:25.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:05:25.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:05:25.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:25.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:05:25.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:05:25.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:25.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:05:25.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:05:25.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:25.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:05:25.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:05:25.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:25.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:05:25.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:05:25.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:25.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:05:25.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:05:25.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:25.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:05:25.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:05:25.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:25.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:05:25.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:05:25.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:25.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:05:25.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:05:25.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:25.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:05:25.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:05:25.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:25.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:05:25.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:05:25.636 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.636 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.637 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.637 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:25.637 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:25.637 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.13363630500001, 2023-07-02T14:05:25.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:05:25.637 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:25.638 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:25.638 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:25.638 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:25.638 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:25.638 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:05:25.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.638 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:25.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:25.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:25.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:25.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:25.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:25.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:25.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:25.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:05:25.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:05:25.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:25.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:05:25.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:05:25.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:25.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:05:25.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:05:25.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:25.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:05:25.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:05:25.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:25.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:05:25.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:05:25.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:25.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:05:25.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:05:25.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:25.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:05:25.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:05:25.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:25.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:05:25.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:05:25.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:25.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:05:25.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:05:25.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:25.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:05:25.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:05:25.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:25.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:05:25.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:05:25.647 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.647 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.647 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.647 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:25.647 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:25.648 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.072299644, 2023-07-02T14:05:25.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:05:25.648 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:25.648 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:25.648 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:25.648 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:25.648 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:25.648 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:05:25.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.649 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:25.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:25.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:25.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:25.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:25.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:25.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:25.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:25.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:05:25.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:05:25.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:25.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:05:25.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:05:25.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:25.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:05:25.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:05:25.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:25.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:05:25.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:05:25.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:25.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:05:25.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:05:25.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:25.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:05:25.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:05:25.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:25.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:05:25.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:05:25.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:25.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:05:25.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:05:25.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:25.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:05:25.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:05:25.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:25.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:05:25.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:05:25.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:25.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:25.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:25.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:05:25.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:05:25.657 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.657 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.657 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.657 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:25.657 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:25.658 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:25.659 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:05:25.659 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:25.658+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:25.759 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:25.759+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:25.824 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:25.824 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:25.824 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:25.825 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:25.834 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:05:25.876 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:25.875+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:25.961 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:25.961+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:25.997 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:25.998 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:25.998 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:25.998 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:25.998 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:26.008 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:05:26.090 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:26.090+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:26.164 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:26.163+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:26.175 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:26.175 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:26.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:26.176 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:26.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.176 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15701 2.5 2:babe4c4a:::rbd_data.125de1449605.00000000000000e7:head [write 524288~3670016] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:26.176 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.463355+0000", 2023-07-02T14:05:26.176 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 6.7094796399999996, 2023-07-02T14:05:26.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030948784, 2023-07-02T14:05:26.177 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:26.177 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:26.177 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:26.177 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:26.177 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:26.177 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15701 2023-07-02T14:05:26.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.177 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:26.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:26.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463355+0000", 2023-07-02T14:05:26.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:26.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:26.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463355+0000", 2023-07-02T14:05:26.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:26.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:26.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463354+0000", 2023-07-02T14:05:26.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005426246 2023-07-02T14:05:26.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:26.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.468781+0000", 2023-07-02T14:05:26.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6320000000000003e-06 2023-07-02T14:05:26.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:26.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.468790+0000", 2023-07-02T14:05:26.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8080000000000004e-06 2023-07-02T14:05:26.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:26.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.468795+0000", 2023-07-02T14:05:26.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089891880000000004 2023-07-02T14:05:26.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.477784+0000", 2023-07-02T14:05:26.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7979000000000002e-05 2023-07-02T14:05:26.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:26.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.477842+0000", 2023-07-02T14:05:26.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000112236 2023-07-02T14:05:26.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:26.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.477954+0000", 2023-07-02T14:05:26.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0085239600000000006 2023-07-02T14:05:26.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:26.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.486478+0000", 2023-07-02T14:05:26.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077413229999999996 2023-07-02T14:05:26.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:26.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.494220+0000", 2023-07-02T14:05:26.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3022999999999998e-05 2023-07-02T14:05:26.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:26.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.494263+0000", 2023-07-02T14:05:26.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0862000000000003e-05 2023-07-02T14:05:26.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:26.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.494304+0000", 2023-07-02T14:05:26.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030948784 2023-07-02T14:05:26.186 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.186 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.186 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.186 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15733 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:26.186 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503319+0000", 2023-07-02T14:05:26.187 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 6.6695152740000001, 2023-07-02T14:05:26.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036594227999999999, 2023-07-02T14:05:26.187 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:26.187 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:26.187 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:26.187 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:26.187 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:26.187 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15733 2023-07-02T14:05:26.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.188 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:26.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:26.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503319+0000", 2023-07-02T14:05:26.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:26.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:26.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503319+0000", 2023-07-02T14:05:26.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:26.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:26.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503319+0000", 2023-07-02T14:05:26.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0300000000000004e-06 2023-07-02T14:05:26.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:26.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503323+0000", 2023-07-02T14:05:26.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1829e-05 2023-07-02T14:05:26.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:26.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503334+0000", 2023-07-02T14:05:26.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.511e-06 2023-07-02T14:05:26.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:26.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503341+0000", 2023-07-02T14:05:26.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7775000000000005e-05 2023-07-02T14:05:26.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503439+0000", 2023-07-02T14:05:26.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6311999999999998e-05 2023-07-02T14:05:26.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503505+0000", 2023-07-02T14:05:26.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057891940000000001 2023-07-02T14:05:26.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509294+0000", 2023-07-02T14:05:26.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0247999999999997e-05 2023-07-02T14:05:26.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509345+0000", 2023-07-02T14:05:26.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018957756999999999 2023-07-02T14:05:26.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.528302+0000", 2023-07-02T14:05:26.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5699000000000006e-05 2023-07-02T14:05:26.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:26.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.528398+0000", 2023-07-02T14:05:26.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00050004900000000002 2023-07-02T14:05:26.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:26.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.528898+0000", 2023-07-02T14:05:26.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023774949999999999 2023-07-02T14:05:26.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:26.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531276+0000", 2023-07-02T14:05:26.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084207899999999992 2023-07-02T14:05:26.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:26.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539696+0000", 2023-07-02T14:05:26.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8023000000000003e-05 2023-07-02T14:05:26.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:26.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539754+0000", 2023-07-02T14:05:26.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000159092 2023-07-02T14:05:26.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:26.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539913+0000", 2023-07-02T14:05:26.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036594227999999999 2023-07-02T14:05:26.198 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.199 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.199 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.199 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15734 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:26.199 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503349+0000", 2023-07-02T14:05:26.199 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 6.6694855390000001, 2023-07-02T14:05:26.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041221990999999999, 2023-07-02T14:05:26.200 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:26.200 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:26.200 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:26.200 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:26.200 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:26.200 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15734 2023-07-02T14:05:26.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.201 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:26.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:26.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503349+0000", 2023-07-02T14:05:26.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:26.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:26.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503349+0000", 2023-07-02T14:05:26.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:26.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:26.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503348+0000", 2023-07-02T14:05:26.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6969999999999999e-06 2023-07-02T14:05:26.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:26.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503352+0000", 2023-07-02T14:05:26.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4900000000000003e-06 2023-07-02T14:05:26.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:26.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503360+0000", 2023-07-02T14:05:26.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3630000000000004e-06 2023-07-02T14:05:26.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:26.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503364+0000", 2023-07-02T14:05:26.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017805100000000001 2023-07-02T14:05:26.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503542+0000", 2023-07-02T14:05:26.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4289999999999994e-05 2023-07-02T14:05:26.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503626+0000", 2023-07-02T14:05:26.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057627800000000003 2023-07-02T14:05:26.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509389+0000", 2023-07-02T14:05:26.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3256000000000004e-05 2023-07-02T14:05:26.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509472+0000", 2023-07-02T14:05:26.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021137590000000001 2023-07-02T14:05:26.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530610+0000", 2023-07-02T14:05:26.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7247e-05 2023-07-02T14:05:26.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530697+0000", 2023-07-02T14:05:26.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0092678020000000003 2023-07-02T14:05:26.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539965+0000", 2023-07-02T14:05:26.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8950999999999996e-05 2023-07-02T14:05:26.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:26.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.540054+0000", 2023-07-02T14:05:26.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00045742200000000003 2023-07-02T14:05:26.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:26.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.540511+0000", 2023-07-02T14:05:26.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003197653 2023-07-02T14:05:26.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:26.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.543709+0000", 2023-07-02T14:05:26.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00065262499999999999 2023-07-02T14:05:26.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:26.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.544362+0000", 2023-07-02T14:05:26.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9426000000000003e-05 2023-07-02T14:05:26.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:26.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.544421+0000", 2023-07-02T14:05:26.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014981099999999999 2023-07-02T14:05:26.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:26.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.544571+0000", 2023-07-02T14:05:26.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041221990999999999 2023-07-02T14:05:26.213 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.213 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.213 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.213 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15735 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:26.213 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503369+0000", 2023-07-02T14:05:26.214 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 6.6694659139999999, 2023-07-02T14:05:26.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.049400686999999999, 2023-07-02T14:05:26.214 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:26.214 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:26.214 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:26.214 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:26.214 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:26.214 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15735 2023-07-02T14:05:26.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.215 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:26.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:26.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503369+0000", 2023-07-02T14:05:26.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:26.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:26.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503369+0000", 2023-07-02T14:05:26.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:26.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:26.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503368+0000", 2023-07-02T14:05:26.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3289999999999998e-06 2023-07-02T14:05:26.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:26.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503372+0000", 2023-07-02T14:05:26.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4719999999999998e-06 2023-07-02T14:05:26.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:26.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503379+0000", 2023-07-02T14:05:26.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2289999999999997e-06 2023-07-02T14:05:26.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:26.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503383+0000", 2023-07-02T14:05:26.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00030972399999999999 2023-07-02T14:05:26.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503693+0000", 2023-07-02T14:05:26.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000120279 2023-07-02T14:05:26.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503813+0000", 2023-07-02T14:05:26.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057248059999999998 2023-07-02T14:05:26.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509538+0000", 2023-07-02T14:05:26.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011721499999999999 2023-07-02T14:05:26.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.221 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509655+0000", 2023-07-02T14:05:26.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021108756999999999 2023-07-02T14:05:26.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.221 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530764+0000", 2023-07-02T14:05:26.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5428999999999997e-05 2023-07-02T14:05:26.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530859+0000", 2023-07-02T14:05:26.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010890082000000001 2023-07-02T14:05:26.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.541750+0000", 2023-07-02T14:05:26.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5916999999999997e-05 2023-07-02T14:05:26.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.541835+0000", 2023-07-02T14:05:26.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002791226 2023-07-02T14:05:26.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.544627+0000", 2023-07-02T14:05:26.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2878000000000002e-05 2023-07-02T14:05:26.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:26.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.544710+0000", 2023-07-02T14:05:26.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00044054400000000002 2023-07-02T14:05:26.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:26.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.545150+0000", 2023-07-02T14:05:26.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001421484 2023-07-02T14:05:26.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:26.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546572+0000", 2023-07-02T14:05:26.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060059900000000001 2023-07-02T14:05:26.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:26.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.552578+0000", 2023-07-02T14:05:26.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7685e-05 2023-07-02T14:05:26.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:26.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.552645+0000", 2023-07-02T14:05:26.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012397999999999999 2023-07-02T14:05:26.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:26.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.552769+0000", 2023-07-02T14:05:26.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.049400686999999999 2023-07-02T14:05:26.228 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.228 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.229 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.229 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15736 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:26.229 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503388+0000", 2023-07-02T14:05:26.229 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 6.6694466910000001, 2023-07-02T14:05:26.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.057425610000000002, 2023-07-02T14:05:26.229 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:26.230 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:26.230 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:26.230 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:26.230 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:26.230 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15736 2023-07-02T14:05:26.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.230 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:26.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:26.231 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503388+0000", 2023-07-02T14:05:26.231 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:26.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:26.231 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503388+0000", 2023-07-02T14:05:26.231 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:26.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:26.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503387+0000", 2023-07-02T14:05:26.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.382e-06 2023-07-02T14:05:26.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:26.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503391+0000", 2023-07-02T14:05:26.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7790000000000001e-06 2023-07-02T14:05:26.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:26.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503397+0000", 2023-07-02T14:05:26.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3110000000000001e-06 2023-07-02T14:05:26.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:26.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503400+0000", 2023-07-02T14:05:26.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00045478499999999998 2023-07-02T14:05:26.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503855+0000", 2023-07-02T14:05:26.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8894999999999995e-05 2023-07-02T14:05:26.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503944+0000", 2023-07-02T14:05:26.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057727849999999999 2023-07-02T14:05:26.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509716+0000", 2023-07-02T14:05:26.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.593e-05 2023-07-02T14:05:26.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509792+0000", 2023-07-02T14:05:26.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021127659 2023-07-02T14:05:26.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530920+0000", 2023-07-02T14:05:26.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0810999999999997e-05 2023-07-02T14:05:26.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530991+0000", 2023-07-02T14:05:26.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010903886999999999 2023-07-02T14:05:26.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.541895+0000", 2023-07-02T14:05:26.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4783000000000005e-05 2023-07-02T14:05:26.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.541979+0000", 2023-07-02T14:05:26.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041761189999999998 2023-07-02T14:05:26.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546155+0000", 2023-07-02T14:05:26.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2211000000000003e-05 2023-07-02T14:05:26.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.240 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546238+0000", 2023-07-02T14:05:26.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006603589 2023-07-02T14:05:26.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.552841+0000", 2023-07-02T14:05:26.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000108977 2023-07-02T14:05:26.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:26.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.552950+0000", 2023-07-02T14:05:26.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000543126 2023-07-02T14:05:26.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:26.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.553493+0000", 2023-07-02T14:05:26.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030365330000000001 2023-07-02T14:05:26.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:26.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.556530+0000", 2023-07-02T14:05:26.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041768609999999996 2023-07-02T14:05:26.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:26.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.560707+0000", 2023-07-02T14:05:26.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6613999999999999e-05 2023-07-02T14:05:26.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:26.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.560743+0000", 2023-07-02T14:05:26.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0005999999999993e-05 2023-07-02T14:05:26.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:26.245 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.560813+0000", 2023-07-02T14:05:26.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.057425610000000002 2023-07-02T14:05:26.245 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.245 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.245 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.245 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15737 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:26.246 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503667+0000", 2023-07-02T14:05:26.246 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 6.6691678249999997, 2023-07-02T14:05:26.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.067804241000000001, 2023-07-02T14:05:26.246 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:26.246 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:26.246 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:26.246 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:26.246 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:26.247 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15737 2023-07-02T14:05:26.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.247 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:26.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:26.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503667+0000", 2023-07-02T14:05:26.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:26.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:26.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503667+0000", 2023-07-02T14:05:26.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:26.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:26.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503666+0000", 2023-07-02T14:05:26.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6499999999999999e-06 2023-07-02T14:05:26.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:26.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503672+0000", 2023-07-02T14:05:26.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9850000000000001e-06 2023-07-02T14:05:26.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:26.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503682+0000", 2023-07-02T14:05:26.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3580000000000002e-06 2023-07-02T14:05:26.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:26.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503688+0000", 2023-07-02T14:05:26.251 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00030985000000000002 2023-07-02T14:05:26.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.251 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.251 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503998+0000", 2023-07-02T14:05:26.251 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012238099999999999 2023-07-02T14:05:26.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.504120+0000", 2023-07-02T14:05:26.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057408620000000002 2023-07-02T14:05:26.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.252 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509861+0000", 2023-07-02T14:05:26.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2069000000000003e-05 2023-07-02T14:05:26.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.253 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509943+0000", 2023-07-02T14:05:26.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021112245000000002 2023-07-02T14:05:26.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531055+0000", 2023-07-02T14:05:26.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4781e-05 2023-07-02T14:05:26.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531140+0000", 2023-07-02T14:05:26.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010899282 2023-07-02T14:05:26.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.542039+0000", 2023-07-02T14:05:26.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3856999999999994e-05 2023-07-02T14:05:26.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.542123+0000", 2023-07-02T14:05:26.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041575969999999999 2023-07-02T14:05:26.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546281+0000", 2023-07-02T14:05:26.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8136000000000002e-05 2023-07-02T14:05:26.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546319+0000", 2023-07-02T14:05:26.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091989649999999999 2023-07-02T14:05:26.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.555518+0000", 2023-07-02T14:05:26.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3294000000000005e-05 2023-07-02T14:05:26.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.555611+0000", 2023-07-02T14:05:26.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052494760000000003 2023-07-02T14:05:26.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.560860+0000", 2023-07-02T14:05:26.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8725000000000002e-05 2023-07-02T14:05:26.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:26.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.560949+0000", 2023-07-02T14:05:26.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00046914100000000001 2023-07-02T14:05:26.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:26.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.561418+0000", 2023-07-02T14:05:26.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006959911 2023-07-02T14:05:26.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.261 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:26.261 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.568378+0000", 2023-07-02T14:05:26.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029392400000000001 2023-07-02T14:05:26.261 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.261 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:26.261 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.571317+0000", 2023-07-02T14:05:26.262 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8885999999999998e-05 2023-07-02T14:05:26.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.262 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.262 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:26.262 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.571376+0000", 2023-07-02T14:05:26.262 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4517000000000003e-05 2023-07-02T14:05:26.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.262 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:26.263 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.571471+0000", 2023-07-02T14:05:26.263 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.067804241000000001 2023-07-02T14:05:26.263 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.263 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.263 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.264 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15738 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:26.264 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503693+0000", 2023-07-02T14:05:26.264 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 6.6691412940000001, 2023-07-02T14:05:26.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.080315304000000004, 2023-07-02T14:05:26.264 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:26.264 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:26.264 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:26.264 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:26.265 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:26.265 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15738 2023-07-02T14:05:26.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.265 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:26.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:26.265 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503693+0000", 2023-07-02T14:05:26.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:26.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:26.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503693+0000", 2023-07-02T14:05:26.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:26.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:26.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503693+0000", 2023-07-02T14:05:26.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3620000000000002e-06 2023-07-02T14:05:26.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:26.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503696+0000", 2023-07-02T14:05:26.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2850000000000002e-06 2023-07-02T14:05:26.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:26.268 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503701+0000", 2023-07-02T14:05:26.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.938e-06 2023-07-02T14:05:26.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:26.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503706+0000", 2023-07-02T14:05:26.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000447105 2023-07-02T14:05:26.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.504153+0000", 2023-07-02T14:05:26.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011312 2023-07-02T14:05:26.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.504267+0000", 2023-07-02T14:05:26.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057292469999999998 2023-07-02T14:05:26.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509996+0000", 2023-07-02T14:05:26.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7124000000000002e-05 2023-07-02T14:05:26.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.271 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.510073+0000", 2023-07-02T14:05:26.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021118604999999999 2023-07-02T14:05:26.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531192+0000", 2023-07-02T14:05:26.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4815999999999997e-05 2023-07-02T14:05:26.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531266+0000", 2023-07-02T14:05:26.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010915915 2023-07-02T14:05:26.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.273 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.542182+0000", 2023-07-02T14:05:26.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5304999999999998e-05 2023-07-02T14:05:26.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.542258+0000", 2023-07-02T14:05:26.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041042539999999999 2023-07-02T14:05:26.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546362+0000", 2023-07-02T14:05:26.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0340999999999995e-05 2023-07-02T14:05:26.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.275 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.275 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546442+0000", 2023-07-02T14:05:26.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0092239639999999994 2023-07-02T14:05:26.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.275 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.555666+0000", 2023-07-02T14:05:26.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0080999999999995e-05 2023-07-02T14:05:26.276 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.555736+0000", 2023-07-02T14:05:26.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069964290000000002 2023-07-02T14:05:26.276 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.277 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.277 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562733+0000", 2023-07-02T14:05:26.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7447000000000005e-05 2023-07-02T14:05:26.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.277 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.277 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562820+0000", 2023-07-02T14:05:26.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086771179999999993 2023-07-02T14:05:26.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.571497+0000", 2023-07-02T14:05:26.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7438000000000002e-05 2023-07-02T14:05:26.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:26.279 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.571545+0000", 2023-07-02T14:05:26.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000252651 2023-07-02T14:05:26.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:26.279 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.571797+0000", 2023-07-02T14:05:26.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001648369 2023-07-02T14:05:26.280 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:26.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.573446+0000", 2023-07-02T14:05:26.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010442987000000001 2023-07-02T14:05:26.280 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:26.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.583889+0000", 2023-07-02T14:05:26.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9243000000000002e-05 2023-07-02T14:05:26.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:26.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.583948+0000", 2023-07-02T14:05:26.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0590000000000001e-05 2023-07-02T14:05:26.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.282 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:26.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.584008+0000", 2023-07-02T14:05:26.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.080315304000000004 2023-07-02T14:05:26.282 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.282 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.282 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.283 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15739 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:26.283 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503710+0000", 2023-07-02T14:05:26.283 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 6.6691243570000003, 2023-07-02T14:05:26.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.084428051000000004, 2023-07-02T14:05:26.283 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:26.283 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:26.284 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:26.284 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:26.284 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:26.284 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15739 2023-07-02T14:05:26.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.284 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:26.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:26.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503710+0000", 2023-07-02T14:05:26.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:26.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:26.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503710+0000", 2023-07-02T14:05:26.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:26.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:26.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503710+0000", 2023-07-02T14:05:26.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9919999999999999e-06 2023-07-02T14:05:26.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:26.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503713+0000", 2023-07-02T14:05:26.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8439999999999997e-06 2023-07-02T14:05:26.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:26.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503718+0000", 2023-07-02T14:05:26.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.58e-06 2023-07-02T14:05:26.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:26.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503721+0000", 2023-07-02T14:05:26.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00059269800000000001 2023-07-02T14:05:26.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.504314+0000", 2023-07-02T14:05:26.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000121972 2023-07-02T14:05:26.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.504436+0000", 2023-07-02T14:05:26.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056719500000000003 2023-07-02T14:05:26.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.510108+0000", 2023-07-02T14:05:26.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8212000000000003e-05 2023-07-02T14:05:26.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.510186+0000", 2023-07-02T14:05:26.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021121849000000002 2023-07-02T14:05:26.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531308+0000", 2023-07-02T14:05:26.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2182999999999997e-05 2023-07-02T14:05:26.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531380+0000", 2023-07-02T14:05:26.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010927773999999999 2023-07-02T14:05:26.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.542308+0000", 2023-07-02T14:05:26.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7233999999999999e-05 2023-07-02T14:05:26.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.542385+0000", 2023-07-02T14:05:26.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041062779999999997 2023-07-02T14:05:26.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546491+0000", 2023-07-02T14:05:26.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7290000000000001e-05 2023-07-02T14:05:26.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.294 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546559+0000", 2023-07-02T14:05:26.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0092123529999999995 2023-07-02T14:05:26.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.555771+0000", 2023-07-02T14:05:26.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2665999999999998e-05 2023-07-02T14:05:26.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.295 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.555844+0000", 2023-07-02T14:05:26.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070310900000000003 2023-07-02T14:05:26.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562875+0000", 2023-07-02T14:05:26.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6711000000000001e-05 2023-07-02T14:05:26.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562921+0000", 2023-07-02T14:05:26.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010083190000000001 2023-07-02T14:05:26.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.573005+0000", 2023-07-02T14:05:26.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7387999999999997e-05 2023-07-02T14:05:26.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.573092+0000", 2023-07-02T14:05:26.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010965232 2023-07-02T14:05:26.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.584057+0000", 2023-07-02T14:05:26.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8296999999999998e-05 2023-07-02T14:05:26.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:26.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.584156+0000", 2023-07-02T14:05:26.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00048002599999999999 2023-07-02T14:05:26.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:26.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.584636+0000", 2023-07-02T14:05:26.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024421500000000001 2023-07-02T14:05:26.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:26.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.587078+0000", 2023-07-02T14:05:26.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000929487 2023-07-02T14:05:26.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:26.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.588007+0000", 2023-07-02T14:05:26.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0000999999999996e-05 2023-07-02T14:05:26.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:26.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.588077+0000", 2023-07-02T14:05:26.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0915999999999997e-05 2023-07-02T14:05:26.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:26.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.588138+0000", 2023-07-02T14:05:26.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.084428051000000004 2023-07-02T14:05:26.302 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.303 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.303 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.303 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15740 2.0 2:08f47745:::rbd_data.125de1449605.00000000000000eb:head [write 49152~4079616] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:26.303 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.507724+0000", 2023-07-02T14:05:26.303 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 6.6651106609999999, 2023-07-02T14:05:26.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031745934000000003, 2023-07-02T14:05:26.304 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:26.304 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:26.304 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:26.304 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:26.304 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:26.304 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15740 2023-07-02T14:05:26.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.305 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:26.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:26.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.507724+0000", 2023-07-02T14:05:26.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:26.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:26.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.507724+0000", 2023-07-02T14:05:26.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:26.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:26.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.507722+0000", 2023-07-02T14:05:26.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079849010000000008 2023-07-02T14:05:26.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:26.307 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515707+0000", 2023-07-02T14:05:26.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4861999999999999e-05 2023-07-02T14:05:26.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:26.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515722+0000", 2023-07-02T14:05:26.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7649999999999996e-06 2023-07-02T14:05:26.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:26.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515729+0000", 2023-07-02T14:05:26.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3593000000000001e-05 2023-07-02T14:05:26.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515772+0000", 2023-07-02T14:05:26.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011619499999999999 2023-07-02T14:05:26.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:26.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515888+0000", 2023-07-02T14:05:26.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020645599999999999 2023-07-02T14:05:26.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:26.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.516095+0000", 2023-07-02T14:05:26.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014613513999999999 2023-07-02T14:05:26.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:26.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530708+0000", 2023-07-02T14:05:26.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086383950000000001 2023-07-02T14:05:26.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:26.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539347+0000", 2023-07-02T14:05:26.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7710000000000001e-05 2023-07-02T14:05:26.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:26.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539414+0000", 2023-07-02T14:05:26.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5368999999999998e-05 2023-07-02T14:05:26.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:26.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539470+0000", 2023-07-02T14:05:26.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031745934000000003 2023-07-02T14:05:26.313 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.313 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.313 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.314 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15841 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:26.314 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:20.082384+0000", 2023-07-02T14:05:26.314 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 6.0904506429999996, 2023-07-02T14:05:26.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026151851, 2023-07-02T14:05:26.314 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:26.314 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:26.314 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:26.314 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:26.315 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:26.315 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15841 2023-07-02T14:05:26.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.315 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:26.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:26.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082384+0000", 2023-07-02T14:05:26.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:26.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:26.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082384+0000", 2023-07-02T14:05:26.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:26.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:26.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082383+0000", 2023-07-02T14:05:26.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0970000000000002e-06 2023-07-02T14:05:26.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:26.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082386+0000", 2023-07-02T14:05:26.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2315e-05 2023-07-02T14:05:26.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:26.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082399+0000", 2023-07-02T14:05:26.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2179999999999996e-06 2023-07-02T14:05:26.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:26.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082405+0000", 2023-07-02T14:05:26.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001442444 2023-07-02T14:05:26.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.083847+0000", 2023-07-02T14:05:26.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3461000000000003e-05 2023-07-02T14:05:26.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.083891+0000", 2023-07-02T14:05:26.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003501512 2023-07-02T14:05:26.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.087392+0000", 2023-07-02T14:05:26.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2370000000000003e-05 2023-07-02T14:05:26.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.087435+0000", 2023-07-02T14:05:26.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048067229999999997 2023-07-02T14:05:26.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.322 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.092242+0000", 2023-07-02T14:05:26.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5259999999999997e-05 2023-07-02T14:05:26.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.322 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.092287+0000", 2023-07-02T14:05:26.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069701069999999997 2023-07-02T14:05:26.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.099257+0000", 2023-07-02T14:05:26.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3195000000000002e-05 2023-07-02T14:05:26.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.324 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.099300+0000", 2023-07-02T14:05:26.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003064612 2023-07-02T14:05:26.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.324 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.102365+0000", 2023-07-02T14:05:26.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6974e-05 2023-07-02T14:05:26.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:26.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.102422+0000", 2023-07-02T14:05:26.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026789799999999998 2023-07-02T14:05:26.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:26.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.102690+0000", 2023-07-02T14:05:26.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014865189999999999 2023-07-02T14:05:26.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:26.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.104176+0000", 2023-07-02T14:05:26.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042408189999999998 2023-07-02T14:05:26.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:26.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.108417+0000", 2023-07-02T14:05:26.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8080000000000001e-05 2023-07-02T14:05:26.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:26.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.108455+0000", 2023-07-02T14:05:26.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0704000000000005e-05 2023-07-02T14:05:26.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:26.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.108536+0000", 2023-07-02T14:05:26.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026151851 2023-07-02T14:05:26.329 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.329 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.329 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.330 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15842 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:26.330 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:20.082411+0000", 2023-07-02T14:05:26.330 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 6.0904232379999996, 2023-07-02T14:05:26.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032427978000000003, 2023-07-02T14:05:26.330 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:26.330 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:26.331 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:26.331 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:26.331 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:26.331 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15842 2023-07-02T14:05:26.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.331 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:26.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:26.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082411+0000", 2023-07-02T14:05:26.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:26.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:26.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082411+0000", 2023-07-02T14:05:26.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:26.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:26.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082411+0000", 2023-07-02T14:05:26.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6750000000000001e-06 2023-07-02T14:05:26.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:26.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082415+0000", 2023-07-02T14:05:26.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1348e-05 2023-07-02T14:05:26.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:26.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082427+0000", 2023-07-02T14:05:26.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5499999999999996e-06 2023-07-02T14:05:26.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:26.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082431+0000", 2023-07-02T14:05:26.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014813339999999999 2023-07-02T14:05:26.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.083913+0000", 2023-07-02T14:05:26.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9057000000000001e-05 2023-07-02T14:05:26.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.083962+0000", 2023-07-02T14:05:26.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003500862 2023-07-02T14:05:26.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.087463+0000", 2023-07-02T14:05:26.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5519000000000003e-05 2023-07-02T14:05:26.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.087498+0000", 2023-07-02T14:05:26.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048198030000000001 2023-07-02T14:05:26.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.092318+0000", 2023-07-02T14:05:26.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6455000000000002e-05 2023-07-02T14:05:26.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.092354+0000", 2023-07-02T14:05:26.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069876130000000002 2023-07-02T14:05:26.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.099342+0000", 2023-07-02T14:05:26.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4948000000000003e-05 2023-07-02T14:05:26.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.099397+0000", 2023-07-02T14:05:26.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043643629999999996 2023-07-02T14:05:26.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.103761+0000", 2023-07-02T14:05:26.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5099999999999995e-05 2023-07-02T14:05:26.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.103846+0000", 2023-07-02T14:05:26.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047369700000000001 2023-07-02T14:05:26.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.108583+0000", 2023-07-02T14:05:26.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8518999999999998e-05 2023-07-02T14:05:26.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:26.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.108662+0000", 2023-07-02T14:05:26.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00044123099999999999 2023-07-02T14:05:26.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:26.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.109103+0000", 2023-07-02T14:05:26.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055459309999999996 2023-07-02T14:05:26.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:26.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.114649+0000", 2023-07-02T14:05:26.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9299999999999999e-05 2023-07-02T14:05:26.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:26.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.114698+0000", 2023-07-02T14:05:26.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8134999999999998e-05 2023-07-02T14:05:26.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:26.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.114756+0000", 2023-07-02T14:05:26.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2806000000000006e-05 2023-07-02T14:05:26.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:26.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.114839+0000", 2023-07-02T14:05:26.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032427978000000003 2023-07-02T14:05:26.346 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.346 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.346 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.347 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15843 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:26.347 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:20.082443+0000", 2023-07-02T14:05:26.347 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 6.0903914710000002, 2023-07-02T14:05:26.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.038901416000000001, 2023-07-02T14:05:26.347 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:26.347 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:26.347 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:26.347 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:26.348 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:26.348 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15843 2023-07-02T14:05:26.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.348 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:26.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:26.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082443+0000", 2023-07-02T14:05:26.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:26.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:26.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082443+0000", 2023-07-02T14:05:26.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:26.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:26.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082442+0000", 2023-07-02T14:05:26.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2880000000000003e-06 2023-07-02T14:05:26.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:26.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082447+0000", 2023-07-02T14:05:26.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7710000000000001e-06 2023-07-02T14:05:26.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:26.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082453+0000", 2023-07-02T14:05:26.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.164e-06 2023-07-02T14:05:26.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:26.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082459+0000", 2023-07-02T14:05:26.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0015285089999999999 2023-07-02T14:05:26.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.083987+0000", 2023-07-02T14:05:26.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9825999999999999e-05 2023-07-02T14:05:26.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.084047+0000", 2023-07-02T14:05:26.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003478138 2023-07-02T14:05:26.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.087525+0000", 2023-07-02T14:05:26.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:05:26.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.087561+0000", 2023-07-02T14:05:26.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048258870000000001 2023-07-02T14:05:26.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.092387+0000", 2023-07-02T14:05:26.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6260000000000002e-05 2023-07-02T14:05:26.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.092423+0000", 2023-07-02T14:05:26.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070145980000000004 2023-07-02T14:05:26.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.099438+0000", 2023-07-02T14:05:26.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5892999999999997e-05 2023-07-02T14:05:26.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.099474+0000", 2023-07-02T14:05:26.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044162649999999999 2023-07-02T14:05:26.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.103890+0000", 2023-07-02T14:05:26.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3977000000000001e-05 2023-07-02T14:05:26.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.103954+0000", 2023-07-02T14:05:26.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0063264819999999996 2023-07-02T14:05:26.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.110281+0000", 2023-07-02T14:05:26.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7503999999999998e-05 2023-07-02T14:05:26.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.110328+0000", 2023-07-02T14:05:26.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045860379999999997 2023-07-02T14:05:26.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.114914+0000", 2023-07-02T14:05:26.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0939999999999994e-05 2023-07-02T14:05:26.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:26.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.114995+0000", 2023-07-02T14:05:26.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00046429400000000002 2023-07-02T14:05:26.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:26.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.115459+0000", 2023-07-02T14:05:26.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001388771 2023-07-02T14:05:26.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:26.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.116848+0000", 2023-07-02T14:05:26.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044324719999999998 2023-07-02T14:05:26.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:26.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.121281+0000", 2023-07-02T14:05:26.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8530999999999998e-05 2023-07-02T14:05:26.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:26.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.121309+0000", 2023-07-02T14:05:26.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5237999999999997e-05 2023-07-02T14:05:26.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:26.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.121344+0000", 2023-07-02T14:05:26.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.038901416000000001 2023-07-02T14:05:26.364 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.364 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.365 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.365 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15844 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:26.365 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:20.082463+0000", 2023-07-02T14:05:26.365 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 6.0903711840000003, 2023-07-02T14:05:26.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.043857647999999999, 2023-07-02T14:05:26.366 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:26.366 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:26.366 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:26.366 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:26.366 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:26.366 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15844 2023-07-02T14:05:26.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.366 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:26.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:26.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082463+0000", 2023-07-02T14:05:26.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:26.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:26.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082463+0000", 2023-07-02T14:05:26.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:26.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:26.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082463+0000", 2023-07-02T14:05:26.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.472e-06 2023-07-02T14:05:26.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:26.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082466+0000", 2023-07-02T14:05:26.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5299999999999999e-06 2023-07-02T14:05:26.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:26.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082474+0000", 2023-07-02T14:05:26.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.339e-06 2023-07-02T14:05:26.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:26.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082479+0000", 2023-07-02T14:05:26.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0015900230000000001 2023-07-02T14:05:26.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.084069+0000", 2023-07-02T14:05:26.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9060999999999998e-05 2023-07-02T14:05:26.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.084108+0000", 2023-07-02T14:05:26.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0034805980000000001 2023-07-02T14:05:26.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.087589+0000", 2023-07-02T14:05:26.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7870000000000001e-05 2023-07-02T14:05:26.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.087637+0000", 2023-07-02T14:05:26.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048162689999999998 2023-07-02T14:05:26.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.092453+0000", 2023-07-02T14:05:26.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7666000000000002e-05 2023-07-02T14:05:26.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.092491+0000", 2023-07-02T14:05:26.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070150239999999999 2023-07-02T14:05:26.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.099506+0000", 2023-07-02T14:05:26.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9339999999999999e-05 2023-07-02T14:05:26.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.099545+0000", 2023-07-02T14:05:26.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044636540000000001 2023-07-02T14:05:26.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.104009+0000", 2023-07-02T14:05:26.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9098999999999995e-05 2023-07-02T14:05:26.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.104098+0000", 2023-07-02T14:05:26.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006275821 2023-07-02T14:05:26.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.110374+0000", 2023-07-02T14:05:26.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2263999999999997e-05 2023-07-02T14:05:26.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.110456+0000", 2023-07-02T14:05:26.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060119539999999999 2023-07-02T14:05:26.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.116468+0000", 2023-07-02T14:05:26.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7787999999999995e-05 2023-07-02T14:05:26.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.116546+0000", 2023-07-02T14:05:26.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048420779999999997 2023-07-02T14:05:26.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.121388+0000", 2023-07-02T14:05:26.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2459000000000005e-05 2023-07-02T14:05:26.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:26.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.121470+0000", 2023-07-02T14:05:26.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000407697 2023-07-02T14:05:26.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:26.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.121878+0000", 2023-07-02T14:05:26.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0016891339999999999 2023-07-02T14:05:26.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:26.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.123567+0000", 2023-07-02T14:05:26.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002647983 2023-07-02T14:05:26.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:26.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.126215+0000", 2023-07-02T14:05:26.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1282000000000001e-05 2023-07-02T14:05:26.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:26.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.126266+0000", 2023-07-02T14:05:26.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4698999999999999e-05 2023-07-02T14:05:26.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:26.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.126321+0000", 2023-07-02T14:05:26.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.043857647999999999 2023-07-02T14:05:26.384 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.384 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.384 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.384 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15845 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:26.385 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:20.082484+0000", 2023-07-02T14:05:26.385 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 6.0903507169999997, 2023-07-02T14:05:26.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.048335177999999999, 2023-07-02T14:05:26.385 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:26.385 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:26.385 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:26.385 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:26.385 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:26.386 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15845 2023-07-02T14:05:26.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.386 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:26.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:26.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082484+0000", 2023-07-02T14:05:26.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:26.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:26.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082484+0000", 2023-07-02T14:05:26.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:26.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:26.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082483+0000", 2023-07-02T14:05:26.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.58e-06 2023-07-02T14:05:26.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:26.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082487+0000", 2023-07-02T14:05:26.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9920000000000004e-06 2023-07-02T14:05:26.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:26.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082493+0000", 2023-07-02T14:05:26.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3239999999999996e-06 2023-07-02T14:05:26.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:26.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082497+0000", 2023-07-02T14:05:26.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001622914 2023-07-02T14:05:26.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.084120+0000", 2023-07-02T14:05:26.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9756000000000003e-05 2023-07-02T14:05:26.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.084170+0000", 2023-07-02T14:05:26.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0034866229999999999 2023-07-02T14:05:26.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.087656+0000", 2023-07-02T14:05:26.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8263999999999998e-05 2023-07-02T14:05:26.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.087695+0000", 2023-07-02T14:05:26.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048144320000000004 2023-07-02T14:05:26.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.092509+0000", 2023-07-02T14:05:26.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7141999999999997e-05 2023-07-02T14:05:26.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.092556+0000", 2023-07-02T14:05:26.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070032920000000004 2023-07-02T14:05:26.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.099560+0000", 2023-07-02T14:05:26.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1183000000000002e-05 2023-07-02T14:05:26.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.099601+0000", 2023-07-02T14:05:26.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004518897 2023-07-02T14:05:26.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.104120+0000", 2023-07-02T14:05:26.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9988999999999999e-05 2023-07-02T14:05:26.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.104170+0000", 2023-07-02T14:05:26.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0063198350000000002 2023-07-02T14:05:26.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.110490+0000", 2023-07-02T14:05:26.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9635999999999998e-05 2023-07-02T14:05:26.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.110529+0000", 2023-07-02T14:05:26.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060621989999999999 2023-07-02T14:05:26.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.116591+0000", 2023-07-02T14:05:26.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1582999999999996e-05 2023-07-02T14:05:26.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.116653+0000", 2023-07-02T14:05:26.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064548440000000004 2023-07-02T14:05:26.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.123108+0000", 2023-07-02T14:05:26.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6099000000000005e-05 2023-07-02T14:05:26.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:26.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.123174+0000", 2023-07-02T14:05:26.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031684999999999999 2023-07-02T14:05:26.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.126342+0000", 2023-07-02T14:05:26.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1156999999999999e-05 2023-07-02T14:05:26.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:26.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.126384+0000", 2023-07-02T14:05:26.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021451400000000001 2023-07-02T14:05:26.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:26.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.126598+0000", 2023-07-02T14:05:26.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001432106 2023-07-02T14:05:26.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:26.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.128030+0000", 2023-07-02T14:05:26.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025489369999999998 2023-07-02T14:05:26.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:26.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.130579+0000", 2023-07-02T14:05:26.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015677200000000001 2023-07-02T14:05:26.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:26.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.130736+0000", 2023-07-02T14:05:26.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3071000000000005e-05 2023-07-02T14:05:26.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.404 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:26.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.130819+0000", 2023-07-02T14:05:26.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.048335177999999999 2023-07-02T14:05:26.404 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.404 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.405 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.405 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6249.0:252 2.7 2:f1d0abf0:::rbd_data.125de1449605.0000000000000083:27 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e150)", 2023-07-02T14:05:26.405 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:24.592830+0000", 2023-07-02T14:05:26.405 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 1.5800048289999999, 2023-07-02T14:05:26.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028734834000000001, 2023-07-02T14:05:26.406 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:26.406 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:26.406 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:26.406 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6249", 2023-07-02T14:05:26.406 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1954902621", 2023-07-02T14:05:26.406 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:05:26.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.406 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:26.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:26.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.592830+0000", 2023-07-02T14:05:26.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:26.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:26.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.592830+0000", 2023-07-02T14:05:26.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:26.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:26.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.592829+0000", 2023-07-02T14:05:26.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7859999999999999e-06 2023-07-02T14:05:26.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:26.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.592831+0000", 2023-07-02T14:05:26.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5470000000000001e-06 2023-07-02T14:05:26.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:26.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.592835+0000", 2023-07-02T14:05:26.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2060000000000001e-06 2023-07-02T14:05:26.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:26.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.592837+0000", 2023-07-02T14:05:26.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025517934999999999 2023-07-02T14:05:26.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.618355+0000", 2023-07-02T14:05:26.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5845000000000002e-05 2023-07-02T14:05:26.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:26.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.618441+0000", 2023-07-02T14:05:26.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031237679999999999 2023-07-02T14:05:26.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:26.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.621564+0000", 2023-07-02T14:05:26.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028734834000000001 2023-07-02T14:05:26.412 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.412 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.412 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.413 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6249.0:335 2.5 2:bf746601:::rbd_data.125de1449605.00000000000000d6:27 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e150)", 2023-07-02T14:05:26.413 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:24.965929+0000", 2023-07-02T14:05:26.413 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 1.206905688, 2023-07-02T14:05:26.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026512628999999999, 2023-07-02T14:05:26.413 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:26.413 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:26.414 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:26.414 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6249", 2023-07-02T14:05:26.414 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1954902621", 2023-07-02T14:05:26.414 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 335 2023-07-02T14:05:26.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.414 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:26.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:26.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.965929+0000", 2023-07-02T14:05:26.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:26.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:26.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.965929+0000", 2023-07-02T14:05:26.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:26.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:26.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.965928+0000", 2023-07-02T14:05:26.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.21e-06 2023-07-02T14:05:26.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:26.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.965931+0000", 2023-07-02T14:05:26.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6979999999999997e-06 2023-07-02T14:05:26.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:26.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.965935+0000", 2023-07-02T14:05:26.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3739999999999998e-06 2023-07-02T14:05:26.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:26.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.965939+0000", 2023-07-02T14:05:26.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022703899999999999 2023-07-02T14:05:26.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.988643+0000", 2023-07-02T14:05:26.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014639500000000001 2023-07-02T14:05:26.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:26.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.988789+0000", 2023-07-02T14:05:26.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036523609999999998 2023-07-02T14:05:26.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:26.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.992441+0000", 2023-07-02T14:05:26.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026512628999999999 2023-07-02T14:05:26.420 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.420 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.420 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.421 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6249.0:336 2.5 2:a8e921ad:::rbd_data.125de1449605.00000000000000d7:27 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e150)", 2023-07-02T14:05:26.421 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:24.965943+0000", 2023-07-02T14:05:26.421 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 1.2068917450000001, 2023-07-02T14:05:26.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029886167000000002, 2023-07-02T14:05:26.421 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:26.421 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:26.421 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:26.421 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6249", 2023-07-02T14:05:26.422 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1954902621", 2023-07-02T14:05:26.422 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 336 2023-07-02T14:05:26.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.422 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:26.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:26.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.965943+0000", 2023-07-02T14:05:26.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:26.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:26.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.965943+0000", 2023-07-02T14:05:26.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:26.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:26.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.965942+0000", 2023-07-02T14:05:26.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4329999999999998e-06 2023-07-02T14:05:26.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:26.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.965945+0000", 2023-07-02T14:05:26.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0049999999999998e-06 2023-07-02T14:05:26.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:26.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.965952+0000", 2023-07-02T14:05:26.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0080000000000004e-06 2023-07-02T14:05:26.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:26.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.965956+0000", 2023-07-02T14:05:26.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02651914 2023-07-02T14:05:26.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.992475+0000", 2023-07-02T14:05:26.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000111976 2023-07-02T14:05:26.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:26.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.992587+0000", 2023-07-02T14:05:26.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032419469999999998 2023-07-02T14:05:26.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:26.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.995829+0000", 2023-07-02T14:05:26.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029886167000000002 2023-07-02T14:05:26.428 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.428 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.428 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.428 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6249.0:341 2.5 2:ae55f2e2:::rbd_data.125de1449605.00000000000000e1:27 [sparse-read 0~4063232 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e150)", 2023-07-02T14:05:26.428 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:25.015119+0000", 2023-07-02T14:05:26.429 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 1.1577151480000001, 2023-07-02T14:05:26.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028784417, 2023-07-02T14:05:26.429 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:26.429 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:26.429 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:26.429 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6249", 2023-07-02T14:05:26.429 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1954902621", 2023-07-02T14:05:26.429 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 341 2023-07-02T14:05:26.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.430 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:26.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:26.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.015119+0000", 2023-07-02T14:05:26.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:26.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:26.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.015119+0000", 2023-07-02T14:05:26.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:26.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:26.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.015119+0000", 2023-07-02T14:05:26.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6599999999999999e-06 2023-07-02T14:05:26.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:26.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.015122+0000", 2023-07-02T14:05:26.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2506e-05 2023-07-02T14:05:26.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:26.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.015134+0000", 2023-07-02T14:05:26.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5199999999999997e-06 2023-07-02T14:05:26.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:26.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.015140+0000", 2023-07-02T14:05:26.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025739499999999998 2023-07-02T14:05:26.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.040879+0000", 2023-07-02T14:05:26.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4022000000000004e-05 2023-07-02T14:05:26.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:26.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.040953+0000", 2023-07-02T14:05:26.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002950659 2023-07-02T14:05:26.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:26.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.043904+0000", 2023-07-02T14:05:26.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028784417 2023-07-02T14:05:26.435 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.436 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.436 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.436 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6249.0:348 2.5 2:b3c82204:::rbd_data.125de1449605.00000000000000de:27 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e150)", 2023-07-02T14:05:26.436 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:25.017109+0000", 2023-07-02T14:05:26.436 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 1.1557255019999999, 2023-07-02T14:05:26.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031502175, 2023-07-02T14:05:26.437 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:26.437 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:26.437 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:26.437 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6249", 2023-07-02T14:05:26.437 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1954902621", 2023-07-02T14:05:26.437 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 348 2023-07-02T14:05:26.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.438 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:26.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:26.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.017109+0000", 2023-07-02T14:05:26.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:26.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:26.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.017109+0000", 2023-07-02T14:05:26.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:26.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:26.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.017108+0000", 2023-07-02T14:05:26.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0399999999999998e-06 2023-07-02T14:05:26.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:26.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.017117+0000", 2023-07-02T14:05:26.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2214000000000001e-05 2023-07-02T14:05:26.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:26.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.017139+0000", 2023-07-02T14:05:26.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7249999999999997e-06 2023-07-02T14:05:26.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:26.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.017146+0000", 2023-07-02T14:05:26.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026777637999999999 2023-07-02T14:05:26.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.043924+0000", 2023-07-02T14:05:26.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5428999999999998e-05 2023-07-02T14:05:26.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:26.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.044010+0000", 2023-07-02T14:05:26.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046015989999999996 2023-07-02T14:05:26.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:26.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.048611+0000", 2023-07-02T14:05:26.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031502175 2023-07-02T14:05:26.443 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.443 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.444 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.444 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6249.0:349 2.5 2:b1affa7f:::rbd_data.125de1449605.00000000000000e5:27 [sparse-read 0~4063232 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e150)", 2023-07-02T14:05:26.444 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:25.020253+0000", 2023-07-02T14:05:26.444 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 1.1525813789999999, 2023-07-02T14:05:26.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031754767000000003, 2023-07-02T14:05:26.444 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:26.445 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:26.445 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:26.445 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6249", 2023-07-02T14:05:26.445 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1954902621", 2023-07-02T14:05:26.445 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 349 2023-07-02T14:05:26.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.445 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:26.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:26.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.020253+0000", 2023-07-02T14:05:26.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:26.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:26.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.020253+0000", 2023-07-02T14:05:26.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:26.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:26.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.020252+0000", 2023-07-02T14:05:26.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7929999999999998e-06 2023-07-02T14:05:26.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:26.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.020259+0000", 2023-07-02T14:05:26.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6614000000000001e-05 2023-07-02T14:05:26.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:26.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.020276+0000", 2023-07-02T14:05:26.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:05:26.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:26.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.020284+0000", 2023-07-02T14:05:26.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028380684999999999 2023-07-02T14:05:26.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:26.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.048664+0000", 2023-07-02T14:05:26.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000187221 2023-07-02T14:05:26.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:26.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.048852+0000", 2023-07-02T14:05:26.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031562309999999998 2023-07-02T14:05:26.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:26.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:26.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:26.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.052008+0000", 2023-07-02T14:05:26.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031754767000000003 2023-07-02T14:05:26.451 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.451 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.451 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.451 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:26.451 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:26.452 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:26.452 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:05:26.453 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:05:26.454 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:05:26.454 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 1 2023-07-02T14:05:26.455 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:26.454+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:26.455 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:05:26.554 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:26.554+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:26.601 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:26.618 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:26.627 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:26.637 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:26.637 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:05:26.655 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:26.654+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:26.655 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:26.674 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:26.674 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-07-02T14:05:26.674 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:05:26.674 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:05:26.675 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:05:26.675 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:05:26.675 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:05:26.677 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:05:26.678 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:05:26.756 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:26.755+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:26.762 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:26.762 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:26.772 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:05:26.857 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:26.856+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:26.936 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:26.936 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:26.946 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:05:26.958 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:26.957+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:27.059 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:27.059+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:27.071 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:27.071 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:27.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:05:27.160 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:27.159+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:27.176 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:05:27.216 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:27.216 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:27.227 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:05:27.227 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap36 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap37 b3fbb028-e267-4d25-a750-93ba18a2316a@snap37.diff 2023-07-02T14:05:27.268 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:27.267+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:27.285 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 100% complete...done. 2023-07-02T14:05:27.288 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:05:27.289 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=38 2023-07-02T14:05:27.289 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap38 2023-07-02T14:05:27.351 INFO:teuthology.orchestra.run.smithi016.stderr:Exporting 2.4 info 2.4( v 146'1478 (0'0,146'1478] local-lis/les=63/64 n=212 ec=14/14 lis/c=63/63 les/c/f=64/64/0 sis=63) 2023-07-02T14:05:27.361 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:27.360+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:27.382 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2211c235:::rbd_data.125de1449605.0000000000000034:d# 2023-07-02T14:05:27.391 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2211c235:::rbd_data.125de1449605.0000000000000034:13# 2023-07-02T14:05:27.397 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2211c235:::rbd_data.125de1449605.0000000000000034:1a# 2023-07-02T14:05:27.403 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2211c235:::rbd_data.125de1449605.0000000000000034:21# 2023-07-02T14:05:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2211c235:::rbd_data.125de1449605.0000000000000034:26# 2023-07-02T14:05:27.409 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:27.409 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:27.414 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2211c235:::rbd_data.125de1449605.0000000000000034:head# 2023-07-02T14:05:27.420 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:23089833:::rbd_data.125de1449605.0000000000000092:e# 2023-07-02T14:05:27.421 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:05:27.426 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:23089833:::rbd_data.125de1449605.0000000000000092:13# 2023-07-02T14:05:27.431 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:23089833:::rbd_data.125de1449605.0000000000000092:1a# 2023-07-02T14:05:27.437 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:23089833:::rbd_data.125de1449605.0000000000000092:21# 2023-07-02T14:05:27.443 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:23089833:::rbd_data.125de1449605.0000000000000092:26# 2023-07-02T14:05:27.448 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:23089833:::rbd_data.125de1449605.0000000000000092:head# 2023-07-02T14:05:27.454 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:237c32fb:::rbd_object_map.125de1449605.000000000000000b:head# 2023-07-02T14:05:27.454 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:24849def:::rbd_data.125de1449605.0000000000000138:1f# 2023-07-02T14:05:27.458 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:24849def:::rbd_data.125de1449605.0000000000000138:head# 2023-07-02T14:05:27.462 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25261dc8:::rbd_data.125de1449605.00000000000000e3:e# 2023-07-02T14:05:27.463 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:27.463+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25261dc8:::rbd_data.125de1449605.00000000000000e3:13# 2023-07-02T14:05:27.473 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25261dc8:::rbd_data.125de1449605.00000000000000e3:14# 2023-07-02T14:05:27.478 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25261dc8:::rbd_data.125de1449605.00000000000000e3:1a# 2023-07-02T14:05:27.481 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25261dc8:::rbd_data.125de1449605.00000000000000e3:21# 2023-07-02T14:05:27.486 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25261dc8:::rbd_data.125de1449605.00000000000000e3:26# 2023-07-02T14:05:27.492 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head# 2023-07-02T14:05:27.497 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25643af7:::rbd_data.125de1449605.0000000000000052:e# 2023-07-02T14:05:27.503 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25643af7:::rbd_data.125de1449605.0000000000000052:13# 2023-07-02T14:05:27.508 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25643af7:::rbd_data.125de1449605.0000000000000052:1a# 2023-07-02T14:05:27.514 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25643af7:::rbd_data.125de1449605.0000000000000052:21# 2023-07-02T14:05:27.520 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25643af7:::rbd_data.125de1449605.0000000000000052:26# 2023-07-02T14:05:27.525 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25643af7:::rbd_data.125de1449605.0000000000000052:head# 2023-07-02T14:05:27.531 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25711f4d:::rbd_data.125de1449605.00000000000000fa:e# 2023-07-02T14:05:27.536 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25711f4d:::rbd_data.125de1449605.00000000000000fa:14# 2023-07-02T14:05:27.542 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25711f4d:::rbd_data.125de1449605.00000000000000fa:1a# 2023-07-02T14:05:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25711f4d:::rbd_data.125de1449605.00000000000000fa:head# 2023-07-02T14:05:27.553 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:27.553 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:27.554 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25e15f0e:::rbd_data.125de1449605.0000000000000087:e# 2023-07-02T14:05:27.559 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25e15f0e:::rbd_data.125de1449605.0000000000000087:13# 2023-07-02T14:05:27.563 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:27.562+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:27.564 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:05:27.565 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25e15f0e:::rbd_data.125de1449605.0000000000000087:1a# 2023-07-02T14:05:27.570 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25e15f0e:::rbd_data.125de1449605.0000000000000087:21# 2023-07-02T14:05:27.576 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25e15f0e:::rbd_data.125de1449605.0000000000000087:26# 2023-07-02T14:05:27.582 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:25e15f0e:::rbd_data.125de1449605.0000000000000087:head# 2023-07-02T14:05:27.588 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:261eca8d:::rbd_data.125de1449605.00000000000000ca:d# 2023-07-02T14:05:27.593 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:261eca8d:::rbd_data.125de1449605.00000000000000ca:e# 2023-07-02T14:05:27.597 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:261eca8d:::rbd_data.125de1449605.00000000000000ca:13# 2023-07-02T14:05:27.602 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:261eca8d:::rbd_data.125de1449605.00000000000000ca:1a# 2023-07-02T14:05:27.608 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:261eca8d:::rbd_data.125de1449605.00000000000000ca:21# 2023-07-02T14:05:27.613 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:261eca8d:::rbd_data.125de1449605.00000000000000ca:26# 2023-07-02T14:05:27.619 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:261eca8d:::rbd_data.125de1449605.00000000000000ca:head# 2023-07-02T14:05:27.625 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2628942f:::rbd_data.125de1449605.0000000000000005:d# 2023-07-02T14:05:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2628942f:::rbd_data.125de1449605.0000000000000005:13# 2023-07-02T14:05:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2628942f:::rbd_data.125de1449605.0000000000000005:19# 2023-07-02T14:05:27.638 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2628942f:::rbd_data.125de1449605.0000000000000005:1e# 2023-07-02T14:05:27.643 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2628942f:::rbd_data.125de1449605.0000000000000005:25# 2023-07-02T14:05:27.647 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2628942f:::rbd_data.125de1449605.0000000000000005:head# 2023-07-02T14:05:27.651 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:265ad6c3:::rbd_data.125de1449605.00000000000000fd:e# 2023-07-02T14:05:27.657 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:265ad6c3:::rbd_data.125de1449605.00000000000000fd:14# 2023-07-02T14:05:27.663 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:265ad6c3:::rbd_data.125de1449605.00000000000000fd:1a# 2023-07-02T14:05:27.663 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:27.662+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:27.669 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:265ad6c3:::rbd_data.125de1449605.00000000000000fd:head# 2023-07-02T14:05:27.674 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:26f20587:::rbd_data.125de1449605.0000000000000071:e# 2023-07-02T14:05:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:26f20587:::rbd_data.125de1449605.0000000000000071:13# 2023-07-02T14:05:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:26f20587:::rbd_data.125de1449605.0000000000000071:1a# 2023-07-02T14:05:27.692 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:26f20587:::rbd_data.125de1449605.0000000000000071:21# 2023-07-02T14:05:27.698 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:26f20587:::rbd_data.125de1449605.0000000000000071:26# 2023-07-02T14:05:27.703 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:26f20587:::rbd_data.125de1449605.0000000000000071:head# 2023-07-02T14:05:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:27cdd850:::rbd_data.125de1449605.0000000000000009:8# 2023-07-02T14:05:27.709 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:27cdd850:::rbd_data.125de1449605.0000000000000009:e# 2023-07-02T14:05:27.710 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:27cdd850:::rbd_data.125de1449605.0000000000000009:13# 2023-07-02T14:05:27.712 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:27cdd850:::rbd_data.125de1449605.0000000000000009:14# 2023-07-02T14:05:27.714 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:27cdd850:::rbd_data.125de1449605.0000000000000009:1a# 2023-07-02T14:05:27.715 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:27cdd850:::rbd_data.125de1449605.0000000000000009:21# 2023-07-02T14:05:27.717 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:27cdd850:::rbd_data.125de1449605.0000000000000009:26# 2023-07-02T14:05:27.719 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:27cdd850:::rbd_data.125de1449605.0000000000000009:head# 2023-07-02T14:05:27.721 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:28147dd3:::rbd_data.125de1449605.0000000000000112:e# 2023-07-02T14:05:27.726 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:28147dd3:::rbd_data.125de1449605.0000000000000112:14# 2023-07-02T14:05:27.727 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:27.727 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:27.732 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:28147dd3:::rbd_data.125de1449605.0000000000000112:1a# 2023-07-02T14:05:27.737 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:05:27.738 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:28147dd3:::rbd_data.125de1449605.0000000000000112:head# 2023-07-02T14:05:27.743 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:28376491:::rbd_data.125de1449605.000000000000011d:e# 2023-07-02T14:05:27.748 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:28376491:::rbd_data.125de1449605.000000000000011d:14# 2023-07-02T14:05:27.754 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:28376491:::rbd_data.125de1449605.000000000000011d:1a# 2023-07-02T14:05:27.760 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:28376491:::rbd_data.125de1449605.000000000000011d:head# 2023-07-02T14:05:27.764 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:28c6fc97:::rbd_data.125de1449605.000000000000014e:21# 2023-07-02T14:05:27.768 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:28c6fc97:::rbd_data.125de1449605.000000000000014e:25# 2023-07-02T14:05:27.772 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:28c6fc97:::rbd_data.125de1449605.000000000000014e:head# 2023-07-02T14:05:27.777 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2a14a29c:::rbd_object_map.125de1449605.0000000000000013:head# 2023-07-02T14:05:27.778 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2a8b9a55:::rbd_data.125de1449605.0000000000000088:e# 2023-07-02T14:05:27.778 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:27.777+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:27.783 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2a8b9a55:::rbd_data.125de1449605.0000000000000088:13# 2023-07-02T14:05:27.788 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2a8b9a55:::rbd_data.125de1449605.0000000000000088:1a# 2023-07-02T14:05:27.794 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2a8b9a55:::rbd_data.125de1449605.0000000000000088:21# 2023-07-02T14:05:27.800 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2a8b9a55:::rbd_data.125de1449605.0000000000000088:26# 2023-07-02T14:05:27.805 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2a8b9a55:::rbd_data.125de1449605.0000000000000088:head# 2023-07-02T14:05:27.811 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2aa9c62e:::rbd_data.125de1449605.0000000000000048:e# 2023-07-02T14:05:27.816 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2aa9c62e:::rbd_data.125de1449605.0000000000000048:13# 2023-07-02T14:05:27.822 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2aa9c62e:::rbd_data.125de1449605.0000000000000048:1a# 2023-07-02T14:05:27.827 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2aa9c62e:::rbd_data.125de1449605.0000000000000048:21# 2023-07-02T14:05:27.833 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2aa9c62e:::rbd_data.125de1449605.0000000000000048:26# 2023-07-02T14:05:27.838 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2aa9c62e:::rbd_data.125de1449605.0000000000000048:head# 2023-07-02T14:05:27.844 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2aa9ee50:::rbd_data.125de1449605.0000000000000055:e# 2023-07-02T14:05:27.850 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2aa9ee50:::rbd_data.125de1449605.0000000000000055:13# 2023-07-02T14:05:27.855 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2aa9ee50:::rbd_data.125de1449605.0000000000000055:1a# 2023-07-02T14:05:27.861 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2aa9ee50:::rbd_data.125de1449605.0000000000000055:21# 2023-07-02T14:05:27.864 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:27.863+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:27.867 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2aa9ee50:::rbd_data.125de1449605.0000000000000055:26# 2023-07-02T14:05:27.872 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2aa9ee50:::rbd_data.125de1449605.0000000000000055:head# 2023-07-02T14:05:27.878 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2c79e900:::rbd_data.125de1449605.000000000000004a:e# 2023-07-02T14:05:27.884 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2c79e900:::rbd_data.125de1449605.000000000000004a:13# 2023-07-02T14:05:27.890 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2c79e900:::rbd_data.125de1449605.000000000000004a:1a# 2023-07-02T14:05:27.895 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2c79e900:::rbd_data.125de1449605.000000000000004a:21# 2023-07-02T14:05:27.901 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:27.901 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:27.901 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2c79e900:::rbd_data.125de1449605.000000000000004a:26# 2023-07-02T14:05:27.907 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2c79e900:::rbd_data.125de1449605.000000000000004a:head# 2023-07-02T14:05:27.911 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:05:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:7# 2023-07-02T14:05:27.918 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:8# 2023-07-02T14:05:27.924 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:9# 2023-07-02T14:05:27.927 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:c# 2023-07-02T14:05:27.929 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:e# 2023-07-02T14:05:27.934 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:f# 2023-07-02T14:05:27.938 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:13# 2023-07-02T14:05:27.941 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:14# 2023-07-02T14:05:27.946 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:15# 2023-07-02T14:05:27.949 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:19# 2023-07-02T14:05:27.952 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:1a# 2023-07-02T14:05:27.957 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:1b# 2023-07-02T14:05:27.961 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:1c# 2023-07-02T14:05:27.964 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:20# 2023-07-02T14:05:27.964 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:27.963+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:27.967 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:21# 2023-07-02T14:05:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:23# 2023-07-02T14:05:27.975 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:25# 2023-07-02T14:05:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:26# 2023-07-02T14:05:27.983 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head# 2023-07-02T14:05:27.987 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2e32ef35:::rbd_data.125de1449605.0000000000000141:20# 2023-07-02T14:05:27.991 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2e32ef35:::rbd_data.125de1449605.0000000000000141:25# 2023-07-02T14:05:27.995 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2e32ef35:::rbd_data.125de1449605.0000000000000141:head# 2023-07-02T14:05:28.000 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2e908755:::rbd_data.125de1449605.00000000000000fb:e# 2023-07-02T14:05:28.006 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2e908755:::rbd_data.125de1449605.00000000000000fb:14# 2023-07-02T14:05:28.012 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2e908755:::rbd_data.125de1449605.00000000000000fb:1a# 2023-07-02T14:05:28.018 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2e908755:::rbd_data.125de1449605.00000000000000fb:head# 2023-07-02T14:05:28.023 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2f8e3c69:::rbd_data.125de1449605.0000000000000028:d# 2023-07-02T14:05:28.029 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2f8e3c69:::rbd_data.125de1449605.0000000000000028:e# 2023-07-02T14:05:28.034 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2f8e3c69:::rbd_data.125de1449605.0000000000000028:13# 2023-07-02T14:05:28.035 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:28.035 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:28.038 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2f8e3c69:::rbd_data.125de1449605.0000000000000028:1a# 2023-07-02T14:05:28.043 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2f8e3c69:::rbd_data.125de1449605.0000000000000028:21# 2023-07-02T14:05:28.046 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:05:28.049 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2f8e3c69:::rbd_data.125de1449605.0000000000000028:26# 2023-07-02T14:05:28.055 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:2f8e3c69:::rbd_data.125de1449605.0000000000000028:head# 2023-07-02T14:05:28.060 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:32e21197:::rbd_data.125de1449605.00000000000000cf:d# 2023-07-02T14:05:28.066 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:32e21197:::rbd_data.125de1449605.00000000000000cf:e# 2023-07-02T14:05:28.070 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:32e21197:::rbd_data.125de1449605.00000000000000cf:13# 2023-07-02T14:05:28.075 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:32e21197:::rbd_data.125de1449605.00000000000000cf:1a# 2023-07-02T14:05:28.080 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:32e21197:::rbd_data.125de1449605.00000000000000cf:21# 2023-07-02T14:05:28.086 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:32e21197:::rbd_data.125de1449605.00000000000000cf:26# 2023-07-02T14:05:28.087 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:28.086+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:28.092 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:32e21197:::rbd_data.125de1449605.00000000000000cf:head# 2023-07-02T14:05:28.098 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:34dd1278:::rbd_object_map.125de1449605.0000000000000020:head# 2023-07-02T14:05:28.098 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:35184cfb:::rbd_data.125de1449605.000000000000012a:e# 2023-07-02T14:05:28.101 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:35184cfb:::rbd_data.125de1449605.000000000000012a:14# 2023-07-02T14:05:28.103 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:35184cfb:::rbd_data.125de1449605.000000000000012a:1a# 2023-07-02T14:05:28.108 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:35184cfb:::rbd_data.125de1449605.000000000000012a:1e# 2023-07-02T14:05:28.112 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:35184cfb:::rbd_data.125de1449605.000000000000012a:head# 2023-07-02T14:05:28.117 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:357fce6c:::rbd_data.125de1449605.00000000000000c8:d# 2023-07-02T14:05:28.123 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:357fce6c:::rbd_data.125de1449605.00000000000000c8:e# 2023-07-02T14:05:28.127 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:357fce6c:::rbd_data.125de1449605.00000000000000c8:13# 2023-07-02T14:05:28.131 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:357fce6c:::rbd_data.125de1449605.00000000000000c8:1a# 2023-07-02T14:05:28.137 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:357fce6c:::rbd_data.125de1449605.00000000000000c8:21# 2023-07-02T14:05:28.142 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:357fce6c:::rbd_data.125de1449605.00000000000000c8:26# 2023-07-02T14:05:28.148 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:357fce6c:::rbd_data.125de1449605.00000000000000c8:head# 2023-07-02T14:05:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3625364f:::rbd_data.125de1449605.0000000000000026:4# 2023-07-02T14:05:28.157 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3625364f:::rbd_data.125de1449605.0000000000000026:7# 2023-07-02T14:05:28.160 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3625364f:::rbd_data.125de1449605.0000000000000026:d# 2023-07-02T14:05:28.164 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3625364f:::rbd_data.125de1449605.0000000000000026:13# 2023-07-02T14:05:28.166 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:28.165+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:28.170 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3625364f:::rbd_data.125de1449605.0000000000000026:1a# 2023-07-02T14:05:28.177 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3625364f:::rbd_data.125de1449605.0000000000000026:21# 2023-07-02T14:05:28.184 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3625364f:::rbd_data.125de1449605.0000000000000026:26# 2023-07-02T14:05:28.190 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3625364f:::rbd_data.125de1449605.0000000000000026:head# 2023-07-02T14:05:28.196 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3696af49:::rbd_data.125de1449605.000000000000015b:25# 2023-07-02T14:05:28.197 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3696af49:::rbd_data.125de1449605.000000000000015b:head# 2023-07-02T14:05:28.202 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3831950c:::rbd_directory:head# 2023-07-02T14:05:28.203 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:393f5a02:::rbd_data.125de1449605.0000000000000133:1e# 2023-07-02T14:05:28.207 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:05:28.208 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:393f5a02:::rbd_data.125de1449605.0000000000000133:head# 2023-07-02T14:05:28.214 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 1 2023-07-02T14:05:28.214 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3a23a925:::rbd_object_map.125de1449605.0000000000000026:head# 2023-07-02T14:05:28.214 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:e# 2023-07-02T14:05:28.218 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:28.218 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:28.220 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:13# 2023-07-02T14:05:28.226 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:14# 2023-07-02T14:05:28.229 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:05:28.231 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:1a# 2023-07-02T14:05:28.234 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:21# 2023-07-02T14:05:28.240 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:26# 2023-07-02T14:05:28.245 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head# 2023-07-02T14:05:28.249 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3d4830c1:::rbd_data.125de1449605.0000000000000140:20# 2023-07-02T14:05:28.253 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3d4830c1:::rbd_data.125de1449605.0000000000000140:25# 2023-07-02T14:05:28.257 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3d4830c1:::rbd_data.125de1449605.0000000000000140:head# 2023-07-02T14:05:28.263 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3e3fd82c:::rbd_data.125de1449605.0000000000000164:head# 2023-07-02T14:05:28.266 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:28.265+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:28.268 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3e650bab:::rbd_data.125de1449605.0000000000000031:d# 2023-07-02T14:05:28.274 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3e650bab:::rbd_data.125de1449605.0000000000000031:13# 2023-07-02T14:05:28.280 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3e650bab:::rbd_data.125de1449605.0000000000000031:1a# 2023-07-02T14:05:28.285 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3e650bab:::rbd_data.125de1449605.0000000000000031:21# 2023-07-02T14:05:28.291 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3e650bab:::rbd_data.125de1449605.0000000000000031:26# 2023-07-02T14:05:28.296 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3e650bab:::rbd_data.125de1449605.0000000000000031:head# 2023-07-02T14:05:28.302 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3e72164b:::rbd_data.125de1449605.0000000000000044:e# 2023-07-02T14:05:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3e72164b:::rbd_data.125de1449605.0000000000000044:13# 2023-07-02T14:05:28.314 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3e72164b:::rbd_data.125de1449605.0000000000000044:1a# 2023-07-02T14:05:28.320 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3e72164b:::rbd_data.125de1449605.0000000000000044:21# 2023-07-02T14:05:28.326 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3e72164b:::rbd_data.125de1449605.0000000000000044:26# 2023-07-02T14:05:28.331 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3e72164b:::rbd_data.125de1449605.0000000000000044:head# 2023-07-02T14:05:28.337 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3e76625b:::rbd_data.125de1449605.0000000000000038:e# 2023-07-02T14:05:28.343 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3e76625b:::rbd_data.125de1449605.0000000000000038:13# 2023-07-02T14:05:28.349 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3e76625b:::rbd_data.125de1449605.0000000000000038:1a# 2023-07-02T14:05:28.354 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3e76625b:::rbd_data.125de1449605.0000000000000038:21# 2023-07-02T14:05:28.360 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3e76625b:::rbd_data.125de1449605.0000000000000038:26# 2023-07-02T14:05:28.366 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3e76625b:::rbd_data.125de1449605.0000000000000038:head# 2023-07-02T14:05:28.367 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:28.366+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:28.371 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3ea366f9:::rbd_data.125de1449605.0000000000000115:e# 2023-07-02T14:05:28.377 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3ea366f9:::rbd_data.125de1449605.0000000000000115:14# 2023-07-02T14:05:28.383 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3ea366f9:::rbd_data.125de1449605.0000000000000115:1a# 2023-07-02T14:05:28.389 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3ea366f9:::rbd_data.125de1449605.0000000000000115:head# 2023-07-02T14:05:28.392 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:28.393 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:28.394 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:e# 2023-07-02T14:05:28.400 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:13# 2023-07-02T14:05:28.403 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:05:28.405 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:14# 2023-07-02T14:05:28.410 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:1a# 2023-07-02T14:05:28.414 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:21# 2023-07-02T14:05:28.420 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:26# 2023-07-02T14:05:28.424 INFO:teuthology.orchestra.run.smithi016.stderr:Read #2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head# 2023-07-02T14:05:28.429 INFO:teuthology.orchestra.run.smithi016.stderr:Export successful 2023-07-02T14:05:28.467 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:28.466+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:28.527 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:28.527 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:28.537 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:05:28.581 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:28.580+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:28.661 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:28.662 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:28.669 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:28.668+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:28.671 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:05:28.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:05:28.770 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:28.769+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:28.817 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:28.834 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:28.834 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:28.835 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:28.845 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:05:28.852 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:28.871 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:28.886 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:28.885+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:28.890 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:28.890 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-07-02T14:05:28.890 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:05:28.890 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:05:28.890 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:05:28.891 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:05:28.891 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:05:28.892 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:05:28.893 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:05:28.964 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --force --op remove --pgid 2.4 2023-07-02T14:05:28.971 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:28.971 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:28.972 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:05:28.972 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:28.972 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:28.971+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:28.982 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:05:29.072 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:29.072+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:29.146 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:29.146 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:29.146 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:05:29.146 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:05:29.146 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:29.156 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:05:29.173 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:29.172+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:29.192 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:05:29.215 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap37 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap38 b3fbb028-e267-4d25-a750-93ba18a2316a@snap38.diff 2023-07-02T14:05:29.274 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:29.273+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:29.294 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:29.294 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:05:29.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:05:29.294 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:05:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 139.84021297499999, 2023-07-02T14:05:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:05:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:05:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:05:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:05:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:05:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:05:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:05:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:05:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:05:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:05:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:05:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:05:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:05:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:05:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:05:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:05:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:05:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:05:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:05:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 138.969519967, 2023-07-02T14:05:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:05:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:05:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:05:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:05:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:05:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:05:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:05:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:05:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:05:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:05:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:05:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:05:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:05:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:05:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:05:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:05:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:05:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:05:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:05:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:05:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:05:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:05:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 138.969395246, 2023-07-02T14:05:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:05:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:05:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:05:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:05:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:05:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:05:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:05:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:05:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:05:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:05:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:05:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:05:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:05:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:05:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:05:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:05:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:05:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:05:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 138.96254536199999, 2023-07-02T14:05:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:05:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:05:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:05:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:05:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:05:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:05:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:05:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:05:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:05:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:05:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:05:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:05:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:05:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:05:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:05:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:05:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:05:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:05:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:05:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 138.96248243700001, 2023-07-02T14:05:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:05:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:05:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:05:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:05:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:05:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:05:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:05:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:05:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:05:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:05:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:05:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:05:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:05:29.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:05:29.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:05:29.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:05:29.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:05:29.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:05:29.337 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.337 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.337 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.338 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:29.338 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:29.338 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 138.96218201600001, 2023-07-02T14:05:29.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:05:29.338 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.338 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:29.338 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.339 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:29.339 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:29.339 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:05:29.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.339 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:29.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:29.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:29.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:05:29.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:05:29.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:05:29.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:05:29.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:05:29.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:05:29.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:05:29.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:05:29.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:05:29.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:05:29.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:05:29.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:05:29.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:05:29.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:05:29.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:29.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:05:29.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:05:29.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:05:29.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:05:29.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:05:29.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:29.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:05:29.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:05:29.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:29.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:05:29.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:05:29.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:29.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:05:29.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:05:29.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:29.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:05:29.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:05:29.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:05:29.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:05:29.349 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.350 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.350 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.350 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:05:29.350 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:29.350 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 138.961657516, 2023-07-02T14:05:29.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:05:29.351 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.351 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:29.351 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.351 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:29.351 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:29.351 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:05:29.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.351 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:29.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:29.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:29.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:29.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:05:29.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:05:29.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:05:29.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:05:29.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:05:29.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:05:29.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:05:29.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:05:29.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:05:29.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:05:29.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:05:29.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:05:29.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:05:29.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:29.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:05:29.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:05:29.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:05:29.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:05:29.358 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.359 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.359 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.359 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:05:29.359 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:29.359 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 111.837638391, 2023-07-02T14:05:29.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:05:29.360 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.360 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:29.360 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.360 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:29.360 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:29.360 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:05:29.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.361 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:29.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:29.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:29.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:05:29.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:05:29.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:05:29.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:05:29.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:05:29.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:05:29.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:05:29.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:05:29.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:05:29.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:05:29.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:05:29.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:05:29.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:05:29.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:05:29.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:05:29.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:05:29.367 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.367 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.367 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.367 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:29.368 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:29.368 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 111.512155687, 2023-07-02T14:05:29.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:05:29.368 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.368 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:29.368 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.368 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:29.368 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:29.369 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:05:29.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.369 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:29.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:29.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:29.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:05:29.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:05:29.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:05:29.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:05:29.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:05:29.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:05:29.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:05:29.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:05:29.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:29.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:05:29.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:05:29.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:05:29.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:05:29.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:05:29.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:29.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:05:29.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:05:29.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:29.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:05:29.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:05:29.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:29.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:05:29.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:05:29.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:29.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:05:29.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:05:29.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:29.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:05:29.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:05:29.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:05:29.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:05:29.378 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.378 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.379 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.379 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3939 2.1 2:9ef34a83:::rbd_data.125de1449605.00000000000000b8:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:29.379 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:29.379 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 111.50823499000001, 2023-07-02T14:05:29.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001, 2023-07-02T14:05:29.379 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.380 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:29.380 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.380 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:29.380 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:29.380 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3939 2023-07-02T14:05:29.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.380 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:29.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:29.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:29.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:29.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046852830000000002 2023-07-02T14:05:29.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788763+0000", 2023-07-02T14:05:29.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3185e-05 2023-07-02T14:05:29.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788776+0000", 2023-07-02T14:05:29.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8470000000000003e-06 2023-07-02T14:05:29.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788781+0000", 2023-07-02T14:05:29.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7104000000000002e-05 2023-07-02T14:05:29.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788828+0000", 2023-07-02T14:05:29.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3516000000000001e-05 2023-07-02T14:05:29.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788841+0000", 2023-07-02T14:05:29.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33064513600000001 2023-07-02T14:05:29.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119487+0000", 2023-07-02T14:05:29.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9876999999999997e-05 2023-07-02T14:05:29.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:29.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119537+0000", 2023-07-02T14:05:29.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014160999999999999 2023-07-02T14:05:29.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:29.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119678+0000", 2023-07-02T14:05:29.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023054392 2023-07-02T14:05:29.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:29.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.142733+0000", 2023-07-02T14:05:29.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074065470000000003 2023-07-02T14:05:29.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:29.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150139+0000", 2023-07-02T14:05:29.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5025999999999997e-05 2023-07-02T14:05:29.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:29.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150204+0000", 2023-07-02T14:05:29.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147999999999997e-05 2023-07-02T14:05:29.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150241+0000", 2023-07-02T14:05:29.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001 2023-07-02T14:05:29.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.390 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.391 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:17 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 0~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:29.391 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:05:29.391 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.093879905000001, 2023-07-02T14:05:29.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997, 2023-07-02T14:05:29.391 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.391 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:29.391 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.391 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:29.392 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:29.392 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17 2023-07-02T14:05:29.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.392 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:05:29.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:05:29.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:05:29.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198431+0000", 2023-07-02T14:05:29.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2815e-05 2023-07-02T14:05:29.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198443+0000", 2023-07-02T14:05:29.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2569999999999999e-05 2023-07-02T14:05:29.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198466+0000", 2023-07-02T14:05:29.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0223e-05 2023-07-02T14:05:29.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198476+0000", 2023-07-02T14:05:29.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1792999999999997e-05 2023-07-02T14:05:29.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:05:29.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4249999999999998e-06 2023-07-02T14:05:29.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198536+0000", 2023-07-02T14:05:29.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91888663500000001 2023-07-02T14:05:29.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117423+0000", 2023-07-02T14:05:29.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223594 2023-07-02T14:05:29.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:29.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117647+0000", 2023-07-02T14:05:29.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033832900000000001 2023-07-02T14:05:29.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117985+0000", 2023-07-02T14:05:29.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997 2023-07-02T14:05:29.399 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.399 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.399 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.400 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:18 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 65536~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:29.400 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:29.400 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.093822694, 2023-07-02T14:05:29.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998, 2023-07-02T14:05:29.400 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.400 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:29.400 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.401 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:29.401 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:29.401 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18 2023-07-02T14:05:29.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.401 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:29.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:29.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:29.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:29.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7600000000000002e-06 2023-07-02T14:05:29.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198494+0000", 2023-07-02T14:05:29.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7840000000000004e-06 2023-07-02T14:05:29.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198502+0000", 2023-07-02T14:05:29.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6459999999999998e-06 2023-07-02T14:05:29.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198508+0000", 2023-07-02T14:05:29.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7395999999999997e-05 2023-07-02T14:05:29.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198565+0000", 2023-07-02T14:05:29.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6800000000000004e-06 2023-07-02T14:05:29.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198572+0000", 2023-07-02T14:05:29.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91965896800000002 2023-07-02T14:05:29.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118231+0000", 2023-07-02T14:05:29.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5212999999999998e-05 2023-07-02T14:05:29.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:29.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118326+0000", 2023-07-02T14:05:29.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000238538 2023-07-02T14:05:29.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118565+0000", 2023-07-02T14:05:29.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998 2023-07-02T14:05:29.408 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.408 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.408 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.409 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:19 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 131072~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:29.409 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:29.409 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.093798469999999, 2023-07-02T14:05:29.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002, 2023-07-02T14:05:29.409 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.409 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:29.409 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:29.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:29.410 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 19 2023-07-02T14:05:29.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.410 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:29.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:29.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:29.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:29.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1990000000000002e-06 2023-07-02T14:05:29.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198517+0000", 2023-07-02T14:05:29.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8789999999999997e-06 2023-07-02T14:05:29.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198524+0000", 2023-07-02T14:05:29.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:05:29.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:05:29.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5545999999999999e-05 2023-07-02T14:05:29.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:29.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2540000000000003e-06 2023-07-02T14:05:29.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198598+0000", 2023-07-02T14:05:29.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.919983263 2023-07-02T14:05:29.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118581+0000", 2023-07-02T14:05:29.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6878999999999994e-05 2023-07-02T14:05:29.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:29.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118648+0000", 2023-07-02T14:05:29.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149732 2023-07-02T14:05:29.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118798+0000", 2023-07-02T14:05:29.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002 2023-07-02T14:05:29.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.417 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.418 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:20 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 262144~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:29.418 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:29.418 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.093779506000001, 2023-07-02T14:05:29.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998, 2023-07-02T14:05:29.418 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.418 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:29.418 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.419 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:29.419 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:29.419 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20 2023-07-02T14:05:29.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.419 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:29.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:29.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:29.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:29.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.732e-06 2023-07-02T14:05:29.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198537+0000", 2023-07-02T14:05:29.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5259999999999999e-06 2023-07-02T14:05:29.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198543+0000", 2023-07-02T14:05:29.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.121e-06 2023-07-02T14:05:29.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198547+0000", 2023-07-02T14:05:29.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9591000000000005e-05 2023-07-02T14:05:29.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:05:29.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3120000000000004e-06 2023-07-02T14:05:29.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198621+0000", 2023-07-02T14:05:29.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92020789400000003 2023-07-02T14:05:29.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118829+0000", 2023-07-02T14:05:29.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022393386000000001 2023-07-02T14:05:29.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:29.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141223+0000", 2023-07-02T14:05:29.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000329866 2023-07-02T14:05:29.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141552+0000", 2023-07-02T14:05:29.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998 2023-07-02T14:05:29.426 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.426 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.426 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.427 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:21 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 196608~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:29.427 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:29.427 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.093751870999998, 2023-07-02T14:05:29.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995, 2023-07-02T14:05:29.427 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.427 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:29.428 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.428 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:29.428 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:29.428 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21 2023-07-02T14:05:29.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.428 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:29.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:29.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:29.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:29.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0169999999999999e-05 2023-07-02T14:05:29.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198571+0000", 2023-07-02T14:05:29.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3479999999999998e-06 2023-07-02T14:05:29.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198578+0000", 2023-07-02T14:05:29.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8439999999999997e-06 2023-07-02T14:05:29.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198583+0000", 2023-07-02T14:05:29.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8294999999999997e-05 2023-07-02T14:05:29.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198641+0000", 2023-07-02T14:05:29.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8120000000000001e-06 2023-07-02T14:05:29.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:05:29.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94293828499999999 2023-07-02T14:05:29.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141583+0000", 2023-07-02T14:05:29.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014607300000000001 2023-07-02T14:05:29.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:29.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141729+0000", 2023-07-02T14:05:29.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00044906600000000003 2023-07-02T14:05:29.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142178+0000", 2023-07-02T14:05:29.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995 2023-07-02T14:05:29.435 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.435 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.436 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.436 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:22 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 327680~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:29.436 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:29.436 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.093718441, 2023-07-02T14:05:29.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004, 2023-07-02T14:05:29.436 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.437 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:29.437 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.437 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:29.437 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:29.437 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 22 2023-07-02T14:05:29.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.437 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:29.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:29.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:29.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:29.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4850000000000001e-06 2023-07-02T14:05:29.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198597+0000", 2023-07-02T14:05:29.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:05:29.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198605+0000", 2023-07-02T14:05:29.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4839999999999998e-06 2023-07-02T14:05:29.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198609+0000", 2023-07-02T14:05:29.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3100999999999999e-05 2023-07-02T14:05:29.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198662+0000", 2023-07-02T14:05:29.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9099999999999998e-06 2023-07-02T14:05:29.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198666+0000", 2023-07-02T14:05:29.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.943539503 2023-07-02T14:05:29.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142206+0000", 2023-07-02T14:05:29.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012255999999999999 2023-07-02T14:05:29.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:29.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142328+0000", 2023-07-02T14:05:29.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208956 2023-07-02T14:05:29.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142537+0000", 2023-07-02T14:05:29.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004 2023-07-02T14:05:29.444 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.444 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.445 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.445 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:23 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 393216~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:29.445 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:29.445 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.093698460999999, 2023-07-02T14:05:29.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348, 2023-07-02T14:05:29.446 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.446 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:29.446 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.446 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:29.446 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:29.446 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23 2023-07-02T14:05:29.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.446 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:29.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:29.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:29.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:29.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8439999999999998e-06 2023-07-02T14:05:29.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:05:29.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0440000000000001e-06 2023-07-02T14:05:29.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198623+0000", 2023-07-02T14:05:29.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4480000000000002e-06 2023-07-02T14:05:29.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198628+0000", 2023-07-02T14:05:29.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5843e-05 2023-07-02T14:05:29.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:29.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.89e-06 2023-07-02T14:05:29.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198688+0000", 2023-07-02T14:05:29.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94389171999999999 2023-07-02T14:05:29.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142580+0000", 2023-07-02T14:05:29.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3892000000000004e-05 2023-07-02T14:05:29.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:29.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142644+0000", 2023-07-02T14:05:29.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017608000000000001 2023-07-02T14:05:29.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142820+0000", 2023-07-02T14:05:29.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348 2023-07-02T14:05:29.453 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.454 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.454 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.454 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:24 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 458752~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:29.454 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:05:29.454 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.093670387, 2023-07-02T14:05:29.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004, 2023-07-02T14:05:29.455 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.455 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:29.455 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.455 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:29.455 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:29.455 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 24 2023-07-02T14:05:29.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.456 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:05:29.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:05:29.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:29.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198642+0000", 2023-07-02T14:05:29.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.253e-06 2023-07-02T14:05:29.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:05:29.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1289999999999998e-06 2023-07-02T14:05:29.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198652+0000", 2023-07-02T14:05:29.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4510000000000002e-06 2023-07-02T14:05:29.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198657+0000", 2023-07-02T14:05:29.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9131000000000001e-05 2023-07-02T14:05:29.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198706+0000", 2023-07-02T14:05:29.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8890000000000004e-06 2023-07-02T14:05:29.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198710+0000", 2023-07-02T14:05:29.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94415412200000004 2023-07-02T14:05:29.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142864+0000", 2023-07-02T14:05:29.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8047000000000006e-05 2023-07-02T14:05:29.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:29.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142952+0000", 2023-07-02T14:05:29.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192967 2023-07-02T14:05:29.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143145+0000", 2023-07-02T14:05:29.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004 2023-07-02T14:05:29.463 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.463 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.463 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.463 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:25 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 524288~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:29.463 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:29.463 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.093651592000001, 2023-07-02T14:05:29.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001, 2023-07-02T14:05:29.464 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.464 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:29.464 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.464 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:29.464 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:29.464 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25 2023-07-02T14:05:29.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.465 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:29.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:29.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:29.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:29.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.615e-06 2023-07-02T14:05:29.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198664+0000", 2023-07-02T14:05:29.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4759999999999997e-06 2023-07-02T14:05:29.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198669+0000", 2023-07-02T14:05:29.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5999999999999998e-06 2023-07-02T14:05:29.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198673+0000", 2023-07-02T14:05:29.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4256999999999999e-05 2023-07-02T14:05:29.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198727+0000", 2023-07-02T14:05:29.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840000000000001e-06 2023-07-02T14:05:29.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198731+0000", 2023-07-02T14:05:29.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94444241900000003 2023-07-02T14:05:29.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143173+0000", 2023-07-02T14:05:29.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6839000000000002e-05 2023-07-02T14:05:29.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:29.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143240+0000", 2023-07-02T14:05:29.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017053399999999999 2023-07-02T14:05:29.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143411+0000", 2023-07-02T14:05:29.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001 2023-07-02T14:05:29.472 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.472 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.472 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.472 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:26 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 589824~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:29.472 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:29.473 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.093628441, 2023-07-02T14:05:29.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997, 2023-07-02T14:05:29.473 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.473 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:29.473 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.473 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:29.473 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:29.473 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 26 2023-07-02T14:05:29.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.474 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:29.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:29.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:29.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:29.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0520000000000002e-06 2023-07-02T14:05:29.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198687+0000", 2023-07-02T14:05:29.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1880000000000002e-06 2023-07-02T14:05:29.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198695+0000", 2023-07-02T14:05:29.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9960000000000004e-06 2023-07-02T14:05:29.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198699+0000", 2023-07-02T14:05:29.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2330999999999998e-05 2023-07-02T14:05:29.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198742+0000", 2023-07-02T14:05:29.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7950000000000001e-06 2023-07-02T14:05:29.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198745+0000", 2023-07-02T14:05:29.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94469398199999999 2023-07-02T14:05:29.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143439+0000", 2023-07-02T14:05:29.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6048e-05 2023-07-02T14:05:29.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:29.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143525+0000", 2023-07-02T14:05:29.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017016700000000001 2023-07-02T14:05:29.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143696+0000", 2023-07-02T14:05:29.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997 2023-07-02T14:05:29.481 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.481 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.481 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.481 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.481 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.481 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:29.483 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:05:29.483 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:29.482+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:29.484 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:05:29.297+0000 7ff678c5b700 0 -- 172.21.15.163:0/1987498919 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7ff654004b10 msgr2=0x7ff654004f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:29.484 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:05:29.307+0000 7ff678c5b700 0 -- 172.21.15.163:0/1987498919 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7ff660036bb0 msgr2=0x7ff660038fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:29.484 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 100% complete...done. 2023-07-02T14:05:29.484 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:05:29.484 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=39 2023-07-02T14:05:29.484 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap39 2023-07-02T14:05:29.583 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:29.583+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:29.650 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:29.650 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:29.651 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:05:29.651 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:29.661 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:05:29.683 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:29.683+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:29.785 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:29.784+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:29.825 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:29.825 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:29.826 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:05:29.826 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:05:29.826 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:29.835 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:05:29.885 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:29.885+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:29.963 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:29.963 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:05:29.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:05:29.963 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:29.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.964 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:05:29.964 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:29.964 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 148.85304565600001, 2023-07-02T14:05:29.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:05:29.964 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.964 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:29.965 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.965 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:29.965 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:29.965 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:05:29.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.965 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:29.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:29.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:29.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:05:29.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:05:29.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:05:29.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:05:29.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:05:29.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:05:29.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:05:29.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:05:29.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:05:29.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:05:29.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:05:29.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:05:29.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:05:29.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:05:29.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:29.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:05:29.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:05:29.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:05:29.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:05:29.972 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.972 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.973 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.973 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:05:29.973 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:29.973 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 148.852984918, 2023-07-02T14:05:29.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:05:29.973 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.974 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:29.974 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.974 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:29.974 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:29.974 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:05:29.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.974 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:29.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:29.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:29.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:05:29.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:05:29.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:05:29.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:05:29.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:05:29.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:05:29.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:05:29.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:05:29.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:05:29.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:05:29.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:29.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:05:29.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:05:29.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:05:29.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:05:29.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:29.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:05:29.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:05:29.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:29.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:05:29.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:05:29.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:29.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:05:29.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:05:29.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:29.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:05:29.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:05:29.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:29.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:05:29.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:05:29.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:05:29.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:05:29.984 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.984 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.984 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.985 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:29.985 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:29.985 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 138.62253058799999, 2023-07-02T14:05:29.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:05:29.985 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.985 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:29.985 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.985 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:29.986 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:29.986 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:05:29.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.986 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:29.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:29.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:29.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:05:29.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:05:29.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:05:29.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:05:29.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:05:29.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:05:29.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:29.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:05:29.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:05:29.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:29.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:05:29.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:05:29.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:29.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:05:29.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:05:29.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:29.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:05:29.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:05:29.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:29.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:05:29.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:05:29.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:29.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:05:29.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:05:29.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:29.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:05:29.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:05:29.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:29.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:05:29.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:05:29.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.994 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:29.995 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:29.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.995 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2950 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 1507328~786432] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:29.995 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:29.995 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 138.48525432599999, 2023-07-02T14:05:29.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999, 2023-07-02T14:05:29.995 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:29.996 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:29.996 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:29.996 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:29.996 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:29.996 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2950 2023-07-02T14:05:29.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.996 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:29.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:29.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:29.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:29.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:29.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:29.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:29.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:29.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:29.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001147854 2023-07-02T14:05:29.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:29.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476614+0000", 2023-07-02T14:05:29.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.802e-05 2023-07-02T14:05:29.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:29.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:29.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:29.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476632+0000", 2023-07-02T14:05:29.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1860000000000003e-06 2023-07-02T14:05:29.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476639+0000", 2023-07-02T14:05:30.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061211240000000004 2023-07-02T14:05:30.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:30.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482760+0000", 2023-07-02T14:05:30.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8833999999999996e-05 2023-07-02T14:05:30.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:30.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482859+0000", 2023-07-02T14:05:30.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019270600000000001 2023-07-02T14:05:30.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:30.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.483051+0000", 2023-07-02T14:05:30.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060385630000000003 2023-07-02T14:05:30.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:30.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489090+0000", 2023-07-02T14:05:30.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024506918999999999 2023-07-02T14:05:30.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:30.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513597+0000", 2023-07-02T14:05:30.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9672999999999995e-05 2023-07-02T14:05:30.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:30.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513667+0000", 2023-07-02T14:05:30.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024329500000000001 2023-07-02T14:05:30.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:30.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513910+0000", 2023-07-02T14:05:30.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999 2023-07-02T14:05:30.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.005 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:30.005 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.005 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2951 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2686976~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:30.005 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:05:30.006 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 138.484073548, 2023-07-02T14:05:30.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999, 2023-07-02T14:05:30.006 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:30.006 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.006 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:30.006 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:30.006 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:30.007 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2951 2023-07-02T14:05:30.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.007 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:30.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:30.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:05:30.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:30.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:30.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:05:30.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:30.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:30.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476646+0000", 2023-07-02T14:05:30.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7639000000000003e-05 2023-07-02T14:05:30.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:30.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476684+0000", 2023-07-02T14:05:30.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3508000000000002e-05 2023-07-02T14:05:30.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:30.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476747+0000", 2023-07-02T14:05:30.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8499999999999994e-06 2023-07-02T14:05:30.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476755+0000", 2023-07-02T14:05:30.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078346160000000008 2023-07-02T14:05:30.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:30.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484590+0000", 2023-07-02T14:05:30.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7206000000000004e-05 2023-07-02T14:05:30.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:30.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484687+0000", 2023-07-02T14:05:30.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016243099999999999 2023-07-02T14:05:30.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:30.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484849+0000", 2023-07-02T14:05:30.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042482860000000004 2023-07-02T14:05:30.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:30.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489097+0000", 2023-07-02T14:05:30.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.02489504 2023-07-02T14:05:30.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:30.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513992+0000", 2023-07-02T14:05:30.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9972000000000001e-05 2023-07-02T14:05:30.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:30.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514052+0000", 2023-07-02T14:05:30.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149264 2023-07-02T14:05:30.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:30.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514202+0000", 2023-07-02T14:05:30.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999 2023-07-02T14:05:30.015 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.015 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:30.015 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.016 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2952 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2883584~12288] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:30.016 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:30.016 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 138.48395971299999, 2023-07-02T14:05:30.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651, 2023-07-02T14:05:30.016 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:30.016 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.017 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:30.017 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:30.017 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:30.017 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2952 2023-07-02T14:05:30.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.017 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:30.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:30.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:30.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:30.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:30.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:30.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:30.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:30.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476760+0000", 2023-07-02T14:05:30.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3664e-05 2023-07-02T14:05:30.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:30.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476774+0000", 2023-07-02T14:05:30.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.166e-06 2023-07-02T14:05:30.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:30.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476783+0000", 2023-07-02T14:05:30.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8720000000000001e-06 2023-07-02T14:05:30.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476788+0000", 2023-07-02T14:05:30.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087180529999999999 2023-07-02T14:05:30.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:30.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485506+0000", 2023-07-02T14:05:30.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5680999999999994e-05 2023-07-02T14:05:30.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:30.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485602+0000", 2023-07-02T14:05:30.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000222365 2023-07-02T14:05:30.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:30.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485824+0000", 2023-07-02T14:05:30.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003277379 2023-07-02T14:05:30.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:30.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489101+0000", 2023-07-02T14:05:30.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025151739999999999 2023-07-02T14:05:30.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:30.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514253+0000", 2023-07-02T14:05:30.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4749000000000002e-05 2023-07-02T14:05:30.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:30.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514288+0000", 2023-07-02T14:05:30.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130634 2023-07-02T14:05:30.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:30.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514419+0000", 2023-07-02T14:05:30.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651 2023-07-02T14:05:30.025 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.026 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:30.026 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.026 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9620 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 589824~65536] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:05:30.026 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:30.026 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.938515989999999, 2023-07-02T14:05:30.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001, 2023-07-02T14:05:30.027 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:30.027 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.027 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:30.027 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:30.027 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:30.027 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9620 2023-07-02T14:05:30.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.028 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:30.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:30.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:30.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:30.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:30.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:30.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:30.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:30.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022203+0000", 2023-07-02T14:05:30.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0346000000000001e-05 2023-07-02T14:05:30.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:30.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022263+0000", 2023-07-02T14:05:30.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4657999999999999e-05 2023-07-02T14:05:30.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:30.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022278+0000", 2023-07-02T14:05:30.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:05:30.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022285+0000", 2023-07-02T14:05:30.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0096000000000003e-05 2023-07-02T14:05:30.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:30.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022335+0000", 2023-07-02T14:05:30.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2334000000000001e-05 2023-07-02T14:05:30.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:30.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022347+0000", 2023-07-02T14:05:30.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095653701999999993 2023-07-02T14:05:30.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:30.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118001+0000", 2023-07-02T14:05:30.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000185797 2023-07-02T14:05:30.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:30.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118187+0000", 2023-07-02T14:05:30.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026397900000000001 2023-07-02T14:05:30.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:30.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118451+0000", 2023-07-02T14:05:30.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051832800000000002 2023-07-02T14:05:30.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:30.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.123634+0000", 2023-07-02T14:05:30.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001227332 2023-07-02T14:05:30.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:30.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124861+0000", 2023-07-02T14:05:30.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9493999999999998e-05 2023-07-02T14:05:30.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:30.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124901+0000", 2023-07-02T14:05:30.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0462000000000001e-05 2023-07-02T14:05:30.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:30.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124931+0000", 2023-07-02T14:05:30.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001 2023-07-02T14:05:30.037 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.037 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:30.037 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.038 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9621 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 655360~1966080] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:05:30.038 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:30.038 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.938328608000006, 2023-07-02T14:05:30.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999, 2023-07-02T14:05:30.038 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:30.038 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.039 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:30.039 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:30.039 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:30.039 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9621 2023-07-02T14:05:30.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.039 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:30.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:30.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:30.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:30.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:30.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:30.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:30.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:30.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022391+0000", 2023-07-02T14:05:30.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032053939999999999 2023-07-02T14:05:30.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:30.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025597+0000", 2023-07-02T14:05:30.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4307999999999999e-05 2023-07-02T14:05:30.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:30.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025611+0000", 2023-07-02T14:05:30.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.032e-06 2023-07-02T14:05:30.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025617+0000", 2023-07-02T14:05:30.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8977999999999998e-05 2023-07-02T14:05:30.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:30.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025656+0000", 2023-07-02T14:05:30.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:05:30.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:30.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025663+0000", 2023-07-02T14:05:30.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095220739999999998 2023-07-02T14:05:30.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:30.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120884+0000", 2023-07-02T14:05:30.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2525000000000001e-05 2023-07-02T14:05:30.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:30.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120956+0000", 2023-07-02T14:05:30.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016232400000000001 2023-07-02T14:05:30.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:30.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.121119+0000", 2023-07-02T14:05:30.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044007380000000004 2023-07-02T14:05:30.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:30.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.125519+0000", 2023-07-02T14:05:30.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000797209 2023-07-02T14:05:30.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:30.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126317+0000", 2023-07-02T14:05:30.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1228000000000004e-05 2023-07-02T14:05:30.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:30.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126378+0000", 2023-07-02T14:05:30.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978000000000003e-05 2023-07-02T14:05:30.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:30.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126430+0000", 2023-07-02T14:05:30.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999 2023-07-02T14:05:30.049 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.049 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:30.049 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.049 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:11434 2.1 2:8269d003:::rbd_data.125de1449605.00000000000000ce:head [write 65536~4128768] snapc 1a=[1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e107)", 2023-07-02T14:05:30.049 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:05:30.050 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 61.042890339000003, 2023-07-02T14:05:30.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036409966000000002, 2023-07-02T14:05:30.050 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:30.050 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.050 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:30.050 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:30.050 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:30.050 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11434 2023-07-02T14:05:30.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.051 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:30.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:30.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:05:30.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:30.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:30.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:05:30.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:30.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:30.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917828+0000", 2023-07-02T14:05:30.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052044650000000001 2023-07-02T14:05:30.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:30.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923033+0000", 2023-07-02T14:05:30.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0097000000000001e-05 2023-07-02T14:05:30.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:30.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923053+0000", 2023-07-02T14:05:30.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.49e-06 2023-07-02T14:05:30.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923061+0000", 2023-07-02T14:05:30.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0090999999999998e-05 2023-07-02T14:05:30.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:30.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923121+0000", 2023-07-02T14:05:30.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000114095 2023-07-02T14:05:30.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:30.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923236+0000", 2023-07-02T14:05:30.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020094199999999999 2023-07-02T14:05:30.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:30.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923436+0000", 2023-07-02T14:05:30.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088693520000000005 2023-07-02T14:05:30.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:30.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.932306+0000", 2023-07-02T14:05:30.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021874112000000001 2023-07-02T14:05:30.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:30.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954180+0000", 2023-07-02T14:05:30.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4433e-05 2023-07-02T14:05:30.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:30.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954214+0000", 2023-07-02T14:05:30.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5884000000000002e-05 2023-07-02T14:05:30.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:30.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954240+0000", 2023-07-02T14:05:30.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036409966000000002 2023-07-02T14:05:30.059 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.059 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:30.060 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.060 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:189 2.3 2:c9621172:::rbd_data.125de1449605.00000000000000c0:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:30.060 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:30.060 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.788041315999997, 2023-07-02T14:05:30.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001, 2023-07-02T14:05:30.060 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:30.061 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:30.061 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:30.061 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:30.061 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:30.061 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:05:30.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.061 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:30.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:30.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:30.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:30.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:30.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:30.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:30.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:30.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:30.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2023-07-02T14:05:30.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:30.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172682+0000", 2023-07-02T14:05:30.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0159999999999997e-06 2023-07-02T14:05:30.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:30.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172688+0000", 2023-07-02T14:05:30.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0010000000000003e-06 2023-07-02T14:05:30.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172692+0000", 2023-07-02T14:05:30.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026444333 2023-07-02T14:05:30.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:30.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199136+0000", 2023-07-02T14:05:30.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2230000000000004e-05 2023-07-02T14:05:30.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:30.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199218+0000", 2023-07-02T14:05:30.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012043595000000001 2023-07-02T14:05:30.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:30.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211262+0000", 2023-07-02T14:05:30.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001 2023-07-02T14:05:30.067 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.067 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:30.067 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.068 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:199 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:30.068 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:30.068 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.787997773000001, 2023-07-02T14:05:30.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002, 2023-07-02T14:05:30.068 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:30.068 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:30.069 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:30.069 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:30.069 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:30.069 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 199 2023-07-02T14:05:30.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.069 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:30.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:30.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:30.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:30.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:30.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:30.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:30.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:30.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172722+0000", 2023-07-02T14:05:30.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1250000000000001e-06 2023-07-02T14:05:30.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:30.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172725+0000", 2023-07-02T14:05:30.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5670000000000002e-06 2023-07-02T14:05:30.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:30.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172731+0000", 2023-07-02T14:05:30.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9310000000000001e-06 2023-07-02T14:05:30.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172735+0000", 2023-07-02T14:05:30.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030699852999999999 2023-07-02T14:05:30.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:30.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203435+0000", 2023-07-02T14:05:30.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6827e-05 2023-07-02T14:05:30.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:30.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203532+0000", 2023-07-02T14:05:30.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076955230000000001 2023-07-02T14:05:30.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:30.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211227+0000", 2023-07-02T14:05:30.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002 2023-07-02T14:05:30.075 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.075 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:30.075 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.076 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:281 2.1 2:901d5a95:::rbd_data.125de1449605.0000000000000118:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:30.076 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:30.076 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.405252496999999, 2023-07-02T14:05:30.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002, 2023-07-02T14:05:30.076 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:30.076 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:30.076 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:30.077 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:30.077 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:30.077 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:05:30.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.077 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:30.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:30.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:30.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:30.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:30.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:30.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:30.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:30.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:30.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0469999999999998e-06 2023-07-02T14:05:30.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:30.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555471+0000", 2023-07-02T14:05:30.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8860000000000001e-06 2023-07-02T14:05:30.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:30.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555476+0000", 2023-07-02T14:05:30.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3160000000000002e-06 2023-07-02T14:05:30.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555483+0000", 2023-07-02T14:05:30.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989525000000001 2023-07-02T14:05:30.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:30.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591472+0000", 2023-07-02T14:05:30.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3708999999999996e-05 2023-07-02T14:05:30.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:30.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591536+0000", 2023-07-02T14:05:30.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055109720000000003 2023-07-02T14:05:30.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:30.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.597047+0000", 2023-07-02T14:05:30.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002 2023-07-02T14:05:30.083 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.083 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:30.083 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.083 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12883 2.1 2:9ffc48e1:::rbd_data.125de1449605.0000000000000074:head [write 4173824~20480] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:05:30.083 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:30.084 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.314650024000002, 2023-07-02T14:05:30.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037871393000000003, 2023-07-02T14:05:30.084 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:30.084 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.084 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:30.084 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:30.084 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:30.085 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12883 2023-07-02T14:05:30.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.085 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:30.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:30.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:30.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:30.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:30.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:30.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:05:30.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:30.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646068+0000", 2023-07-02T14:05:30.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4935999999999999e-05 2023-07-02T14:05:30.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:30.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646103+0000", 2023-07-02T14:05:30.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.092e-05 2023-07-02T14:05:30.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:30.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646124+0000", 2023-07-02T14:05:30.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1519999999999998e-06 2023-07-02T14:05:30.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646133+0000", 2023-07-02T14:05:30.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012270200000000001 2023-07-02T14:05:30.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:30.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646255+0000", 2023-07-02T14:05:30.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014495400000000001 2023-07-02T14:05:30.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:30.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646400+0000", 2023-07-02T14:05:30.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000221626 2023-07-02T14:05:30.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:30.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646622+0000", 2023-07-02T14:05:30.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078929050000000004 2023-07-02T14:05:30.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:30.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.654515+0000", 2023-07-02T14:05:30.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029307343999999999 2023-07-02T14:05:30.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:30.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683822+0000", 2023-07-02T14:05:30.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8298000000000002e-05 2023-07-02T14:05:30.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:30.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683891+0000", 2023-07-02T14:05:30.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1381999999999997e-05 2023-07-02T14:05:30.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:30.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683942+0000", 2023-07-02T14:05:30.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037871393000000003 2023-07-02T14:05:30.093 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.093 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:30.093 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.094 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12985 2.7 2:eeddf24b:::rbd_data.125de1449605.000000000000007e:head [write 65536~4128768] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:05:30.094 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:30.094 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 37.177725551000002, 2023-07-02T14:05:30.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999, 2023-07-02T14:05:30.094 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:30.095 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.095 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:30.095 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:30.095 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:30.095 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12985 2023-07-02T14:05:30.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.095 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:30.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:30.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:30.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:30.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:30.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:30.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:30.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:30.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782994+0000", 2023-07-02T14:05:30.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019254741999999998 2023-07-02T14:05:30.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:30.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802248+0000", 2023-07-02T14:05:30.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0451e-05 2023-07-02T14:05:30.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:30.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802279+0000", 2023-07-02T14:05:30.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3318e-05 2023-07-02T14:05:30.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802292+0000", 2023-07-02T14:05:30.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109403 2023-07-02T14:05:30.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:30.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802402+0000", 2023-07-02T14:05:30.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012520599999999999 2023-07-02T14:05:30.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:30.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802527+0000", 2023-07-02T14:05:30.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187768 2023-07-02T14:05:30.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:05:30.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802715+0000", 2023-07-02T14:05:30.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013192822 2023-07-02T14:05:30.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:30.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.815907+0000", 2023-07-02T14:05:30.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0086451870000000004 2023-07-02T14:05:30.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:30.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824553+0000", 2023-07-02T14:05:30.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6759999999999999e-05 2023-07-02T14:05:30.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:30.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824609+0000", 2023-07-02T14:05:30.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011314700000000001 2023-07-02T14:05:30.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:30.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824722+0000", 2023-07-02T14:05:30.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999 2023-07-02T14:05:30.104 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.104 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:30.104 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.105 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:165 2.3 2:c9c6393a:::rbd_data.125de1449605.000000000000002c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:30.105 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:30.105 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 33.120942734000003, 2023-07-02T14:05:30.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002, 2023-07-02T14:05:30.105 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:30.105 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:30.105 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:30.105 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:30.106 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:30.106 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 165 2023-07-02T14:05:30.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.106 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:30.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:30.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:30.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:30.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:30.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:30.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:30.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:30.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839777+0000", 2023-07-02T14:05:30.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6740000000000001e-06 2023-07-02T14:05:30.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:30.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839780+0000", 2023-07-02T14:05:30.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9030000000000004e-06 2023-07-02T14:05:30.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:30.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839788+0000", 2023-07-02T14:05:30.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1799999999999998e-06 2023-07-02T14:05:30.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839792+0000", 2023-07-02T14:05:30.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.031592227000000001 2023-07-02T14:05:30.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:30.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871384+0000", 2023-07-02T14:05:30.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013600099999999999 2023-07-02T14:05:30.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:30.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871520+0000", 2023-07-02T14:05:30.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082817089999999999 2023-07-02T14:05:30.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:30.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.879802+0000", 2023-07-02T14:05:30.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002 2023-07-02T14:05:30.112 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.112 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:30.112 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.112 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:252 2.7 2:f1d0abf0:::rbd_data.125de1449605.0000000000000083:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:30.112 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:30.113 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.704657763999997, 2023-07-02T14:05:30.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997, 2023-07-02T14:05:30.113 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:30.113 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:30.113 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:30.113 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:30.113 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:30.113 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 252 2023-07-02T14:05:30.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.114 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:30.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:30.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:30.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:30.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:30.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:30.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:30.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:30.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256062+0000", 2023-07-02T14:05:30.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0149999999999999e-06 2023-07-02T14:05:30.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:30.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256065+0000", 2023-07-02T14:05:30.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5269999999999998e-06 2023-07-02T14:05:30.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:30.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256071+0000", 2023-07-02T14:05:30.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:05:30.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256074+0000", 2023-07-02T14:05:30.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028852909 2023-07-02T14:05:30.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:30.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.284927+0000", 2023-07-02T14:05:30.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168274 2023-07-02T14:05:30.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:30.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.285096+0000", 2023-07-02T14:05:30.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011036300000000001 2023-07-02T14:05:30.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:30.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296132+0000", 2023-07-02T14:05:30.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997 2023-07-02T14:05:30.119 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.120 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:30.120 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.120 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:261 2.1 2:971731cb:::rbd_data.125de1449605.000000000000008c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:30.120 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:30.120 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.704562289000002, 2023-07-02T14:05:30.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984, 2023-07-02T14:05:30.121 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:30.121 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:30.121 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:30.121 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:30.121 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:30.121 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 261 2023-07-02T14:05:30.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.122 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:30.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:30.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:30.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:30.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:30.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:30.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:30.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:30.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:30.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2819999999999999e-06 2023-07-02T14:05:30.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:30.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256160+0000", 2023-07-02T14:05:30.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2020000000000003e-06 2023-07-02T14:05:30.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:30.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256165+0000", 2023-07-02T14:05:30.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2100000000000003e-06 2023-07-02T14:05:30.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256170+0000", 2023-07-02T14:05:30.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025871715999999999 2023-07-02T14:05:30.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:30.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282041+0000", 2023-07-02T14:05:30.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132364 2023-07-02T14:05:30.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:30.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282174+0000", 2023-07-02T14:05:30.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014044566 2023-07-02T14:05:30.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:30.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296218+0000", 2023-07-02T14:05:30.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984 2023-07-02T14:05:30.127 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.127 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:30.128 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.128 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:264 2.3 2:dbf06c9d:::rbd_data.125de1449605.000000000000008f:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:30.128 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:30.128 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.704528171, 2023-07-02T14:05:30.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003, 2023-07-02T14:05:30.129 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:30.129 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:30.129 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:30.129 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:30.129 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:30.129 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 264 2023-07-02T14:05:30.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.130 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:30.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:30.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:30.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:30.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:30.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:30.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:30.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:30.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:30.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5749999999999999e-06 2023-07-02T14:05:30.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:30.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256195+0000", 2023-07-02T14:05:30.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.327e-06 2023-07-02T14:05:30.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:30.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256200+0000", 2023-07-02T14:05:30.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0580000000000002e-06 2023-07-02T14:05:30.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256203+0000", 2023-07-02T14:05:30.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035707524999999997 2023-07-02T14:05:30.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:30.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.291911+0000", 2023-07-02T14:05:30.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014269799999999999 2023-07-02T14:05:30.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:30.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.292053+0000", 2023-07-02T14:05:30.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062760309999999996 2023-07-02T14:05:30.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:30.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.298329+0000", 2023-07-02T14:05:30.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003 2023-07-02T14:05:30.135 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.135 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:30.136 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.136 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:341 2.3 2:cee258bd:::rbd_data.125de1449605.00000000000000dc:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:30.136 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:30.136 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.324040844000002, 2023-07-02T14:05:30.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997, 2023-07-02T14:05:30.137 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:30.137 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:30.137 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:30.137 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:30.137 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:30.137 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 341 2023-07-02T14:05:30.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.137 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:30.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:30.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:30.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:30.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:30.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:30.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:30.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:30.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636679+0000", 2023-07-02T14:05:30.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6240000000000002e-06 2023-07-02T14:05:30.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:30.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636682+0000", 2023-07-02T14:05:30.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6910000000000002e-06 2023-07-02T14:05:30.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:30.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636688+0000", 2023-07-02T14:05:30.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3469999999999999e-06 2023-07-02T14:05:30.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636691+0000", 2023-07-02T14:05:30.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035625560000000001 2023-07-02T14:05:30.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:30.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672317+0000", 2023-07-02T14:05:30.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108391 2023-07-02T14:05:30.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:30.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672425+0000", 2023-07-02T14:05:30.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012104976 2023-07-02T14:05:30.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:30.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.684530+0000", 2023-07-02T14:05:30.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997 2023-07-02T14:05:30.143 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.143 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:30.143 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.144 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPushReply(2.7 139/134 [PushReplyOp(2:e3782f39:::rbd_data.125de1449605.0000000000000120:head)])", 2023-07-02T14:05:30.144 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:05:30.144 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.033508691000002, 2023-07-02T14:05:30.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998, 2023-07-02T14:05:30.144 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:30.145 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:30.145 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:30.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:30.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:05:30.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:30.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:30.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:05:30.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:30.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:30.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927211+0000", 2023-07-02T14:05:30.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3689999999999998e-06 2023-07-02T14:05:30.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:30.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927217+0000", 2023-07-02T14:05:30.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7269999999999996e-06 2023-07-02T14:05:30.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:30.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927225+0000", 2023-07-02T14:05:30.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:05:30.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927230+0000", 2023-07-02T14:05:30.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:05:30.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:30.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927266+0000", 2023-07-02T14:05:30.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.069972871000000006 2023-07-02T14:05:30.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:30.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:30.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:30.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.997239+0000", 2023-07-02T14:05:30.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998 2023-07-02T14:05:30.150 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.150 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:30.150 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.150 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:30.150 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:30.150 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:30.152 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:05:30.152 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:30.151+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:30.185 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:05:30.221 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:05:30.227 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 4 2023-07-02T14:05:30.253 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:30.252+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:30.295 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:30.295 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:30.295 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:30.296 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:30.306 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:05:30.388 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:30.388+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:30.454 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:30.453+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:30.470 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:30.470 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:30.470 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:30.470 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:30.470 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:30.480 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:05:30.564 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:30.564+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:30.646 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:30.647 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:30.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:30.647 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:30.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.647 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:05:30.647 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:30.647 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 110.74756942, 2023-07-02T14:05:30.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:05:30.648 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:30.648 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.648 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:30.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:30.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:30.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:30.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:30.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:30.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:30.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:30.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:05:30.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:05:30.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:30.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:05:30.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:05:30.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:30.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:05:30.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:05:30.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:05:30.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:05:30.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:30.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:05:30.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:05:30.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:30.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:05:30.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:05:30.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:30.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:05:30.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:05:30.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:30.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:05:30.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:05:30.654 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.655 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.655 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.655 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:05:30.655 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:30.655 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 110.74348370200001, 2023-07-02T14:05:30.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:05:30.656 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:30.656 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.656 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:30.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:30.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:30.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:30.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:30.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:30.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:30.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:30.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:05:30.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:05:30.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:30.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:05:30.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:05:30.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:30.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:05:30.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:05:30.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:05:30.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:05:30.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:30.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:05:30.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:05:30.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:30.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:05:30.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:05:30.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:30.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:05:30.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:05:30.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:30.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:05:30.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:05:30.662 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.663 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.663 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.663 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:05:30.663 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:30.663 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 110.739618921, 2023-07-02T14:05:30.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:05:30.664 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:30.664 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.664 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:30.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:30.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:30.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:30.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:30.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:30.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:30.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:30.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:05:30.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:05:30.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:30.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:05:30.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:05:30.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:30.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:05:30.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:05:30.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:05:30.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:05:30.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:30.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:05:30.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:05:30.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:30.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:05:30.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:05:30.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:30.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:05:30.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:05:30.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:30.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:05:30.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:05:30.670 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.670 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.670 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.671 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:05:30.671 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:30.671 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 110.73609466000001, 2023-07-02T14:05:30.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:05:30.671 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:30.671 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.672 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:30.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:30.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:30.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:30.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:30.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:30.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:30.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:30.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:05:30.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:05:30.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:30.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:05:30.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:05:30.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:30.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:05:30.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:05:30.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:05:30.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:05:30.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:30.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:05:30.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:05:30.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:30.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:05:30.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:05:30.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:30.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:05:30.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:05:30.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:30.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:05:30.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:05:30.678 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.678 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.678 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.679 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:05:30.679 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:30.679 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 110.700389861, 2023-07-02T14:05:30.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:05:30.679 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:30.679 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.679 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:30.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:30.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:30.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:30.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:30.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:30.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:30.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:30.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:05:30.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:05:30.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:30.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:05:30.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:05:30.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:30.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:05:30.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:05:30.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:05:30.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:05:30.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:30.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:05:30.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:05:30.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:30.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:05:30.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:05:30.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:30.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:05:30.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:05:30.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:30.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:05:30.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:05:30.686 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.686 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.686 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.686 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:05:30.686 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:30.687 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 110.695732503, 2023-07-02T14:05:30.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:05:30.687 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:30.687 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.687 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:30.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:30.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:30.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:30.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:30.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:30.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:30.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:30.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:05:30.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:05:30.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:30.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:05:30.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:05:30.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:30.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:05:30.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:05:30.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:05:30.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:05:30.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:30.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:05:30.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:05:30.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:30.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:05:30.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:05:30.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:30.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:05:30.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:05:30.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:30.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:05:30.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:05:30.693 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.694 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.694 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.694 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:05:30.694 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:30.694 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 110.69479264, 2023-07-02T14:05:30.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:05:30.695 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:30.695 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.695 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:30.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:30.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:30.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:30.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:30.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:30.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:30.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:30.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:05:30.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:05:30.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:30.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:05:30.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:05:30.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:30.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:05:30.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:05:30.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:05:30.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:05:30.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:30.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:05:30.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:05:30.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:30.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:05:30.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:05:30.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:30.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:05:30.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:05:30.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:30.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:05:30.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:05:30.701 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.701 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.702 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.702 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:05:30.702 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:30.702 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 110.693394762, 2023-07-02T14:05:30.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:05:30.702 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:30.703 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.703 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:30.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:30.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:30.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:30.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:30.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:30.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:30.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:30.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:05:30.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:05:30.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:30.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:05:30.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:05:30.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:30.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:05:30.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:05:30.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:05:30.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:05:30.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:30.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:05:30.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:05:30.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:30.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:05:30.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:05:30.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:30.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:05:30.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:05:30.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:30.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:05:30.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:05:30.709 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.709 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.709 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.710 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:05:30.710 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:30.710 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 89.283204265999998, 2023-07-02T14:05:30.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:05:30.710 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:30.710 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.711 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:30.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:30.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:30.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:30.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:30.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:30.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:30.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:30.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:05:30.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:05:30.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:30.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:05:30.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:05:30.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:30.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:05:30.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:05:30.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:05:30.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:05:30.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:30.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:05:30.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:05:30.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:30.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:05:30.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:05:30.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:30.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:05:30.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:05:30.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:30.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:05:30.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:05:30.717 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.717 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.717 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.717 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:30.718 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:30.718 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.916854581999999, 2023-07-02T14:05:30.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:05:30.718 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:30.718 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.718 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:30.718 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:30.718 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:30.719 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:05:30.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.719 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:30.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:30.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:30.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:30.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:30.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:30.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:30.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:30.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:05:30.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:05:30.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:30.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:05:30.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:05:30.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:30.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:05:30.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:05:30.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:05:30.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:05:30.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:30.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:05:30.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:05:30.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:30.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:05:30.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:05:30.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:30.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:05:30.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:05:30.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:30.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:05:30.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:05:30.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:30.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:05:30.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:05:30.726 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.726 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.726 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.727 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7707 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 3145728~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:30.727 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:30.727 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.912871115000002, 2023-07-02T14:05:30.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999, 2023-07-02T14:05:30.727 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:30.727 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.727 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:30.727 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:30.728 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:30.728 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7707 2023-07-02T14:05:30.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.728 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:30.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:30.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:30.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:30.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:30.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:30.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:30.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:30.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:30.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5017e-05 2023-07-02T14:05:30.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:30.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732347+0000", 2023-07-02T14:05:30.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5821e-05 2023-07-02T14:05:30.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:30.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732363+0000", 2023-07-02T14:05:30.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1270000000000002e-06 2023-07-02T14:05:30.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732368+0000", 2023-07-02T14:05:30.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008444666 2023-07-02T14:05:30.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:30.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740812+0000", 2023-07-02T14:05:30.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011266599999999999 2023-07-02T14:05:30.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:30.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740925+0000", 2023-07-02T14:05:30.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010848808999999999 2023-07-02T14:05:30.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:30.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751774+0000", 2023-07-02T14:05:30.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8421999999999997e-05 2023-07-02T14:05:30.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:30.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751832+0000", 2023-07-02T14:05:30.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107453 2023-07-02T14:05:30.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:30.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751940+0000", 2023-07-02T14:05:30.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999 2023-07-02T14:05:30.735 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.735 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.735 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.736 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:30.736 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:30.736 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.912807237999999, 2023-07-02T14:05:30.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:05:30.736 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:30.736 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.736 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:30.736 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:30.737 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:30.737 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:05:30.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.737 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:30.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:30.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:30.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:30.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:30.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:30.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:30.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:30.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:30.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:05:30.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:30.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:05:30.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:05:30.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:30.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:05:30.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:05:30.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:05:30.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:05:30.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:30.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:05:30.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:05:30.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:30.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:05:30.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:05:30.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:30.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:05:30.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:05:30.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:30.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:05:30.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:05:30.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:30.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:05:30.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:05:30.744 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.744 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.744 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.745 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:30.745 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:30.745 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.908243412999994, 2023-07-02T14:05:30.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:05:30.745 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:30.745 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.746 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:30.746 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:30.746 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:30.746 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:05:30.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.746 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:30.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:30.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:30.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:30.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:30.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:30.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:30.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:30.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:05:30.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:05:30.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:30.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:05:30.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:05:30.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:30.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:05:30.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:05:30.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:05:30.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:05:30.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:30.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:05:30.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:05:30.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:30.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:05:30.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:05:30.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:30.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:05:30.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:05:30.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:30.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:05:30.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:05:30.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:30.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:05:30.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:05:30.753 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.753 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.754 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.754 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:30.754 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:30.754 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.908193546999996, 2023-07-02T14:05:30.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:05:30.754 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:30.755 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.755 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:30.755 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:30.755 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:30.755 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:05:30.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.755 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:30.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:30.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:30.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:30.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:30.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:30.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:30.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:30.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:05:30.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:05:30.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:30.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:05:30.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:05:30.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:30.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:05:30.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:05:30.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:05:30.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:05:30.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:30.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:05:30.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:05:30.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:30.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:05:30.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:05:30.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:30.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:05:30.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:05:30.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:30.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:05:30.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:05:30.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:30.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:05:30.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:05:30.762 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.762 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.763 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.763 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:30.763 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:30.763 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.907154554000002, 2023-07-02T14:05:30.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:05:30.763 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:30.764 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.764 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:30.764 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:30.764 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:30.764 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:05:30.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.764 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:30.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:30.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:30.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:30.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:30.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:30.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:30.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:30.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:05:30.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:05:30.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:30.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:05:30.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:05:30.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:30.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:05:30.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:05:30.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:05:30.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:05:30.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:30.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:05:30.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:05:30.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:30.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:05:30.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:05:30.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:30.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:05:30.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:05:30.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:30.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:05:30.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:05:30.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:30.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:05:30.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:05:30.771 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.771 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.772 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.772 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7993 2.7 2:fe72db53:::rbd_data.125de1449605.0000000000000072:head [write 65536~3276800] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:30.772 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:30.772 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 87.720196505999994, 2023-07-02T14:05:30.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919, 2023-07-02T14:05:30.773 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:30.773 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.773 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:30.773 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:30.773 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:30.773 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7993 2023-07-02T14:05:30.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.773 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:30.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:30.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:30.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:30.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:30.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:30.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:30.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:30.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924986+0000", 2023-07-02T14:05:30.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049542070000000004 2023-07-02T14:05:30.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:30.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929940+0000", 2023-07-02T14:05:30.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9740000000000001e-05 2023-07-02T14:05:30.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:30.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929960+0000", 2023-07-02T14:05:30.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.311e-05 2023-07-02T14:05:30.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929973+0000", 2023-07-02T14:05:30.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9629000000000002e-05 2023-07-02T14:05:30.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:30.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930023+0000", 2023-07-02T14:05:30.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012751799999999999 2023-07-02T14:05:30.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:30.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930150+0000", 2023-07-02T14:05:30.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016449042000000001 2023-07-02T14:05:30.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:30.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946599+0000", 2023-07-02T14:05:30.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3241e-05 2023-07-02T14:05:30.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:30.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946663+0000", 2023-07-02T14:05:30.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000102296 2023-07-02T14:05:30.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:30.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946765+0000", 2023-07-02T14:05:30.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919 2023-07-02T14:05:30.780 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.780 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.781 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.781 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11782 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 196608~2162688] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:05:30.781 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:30.781 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.235676583999997, 2023-07-02T14:05:30.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365, 2023-07-02T14:05:30.782 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:30.782 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.782 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:30.782 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:30.782 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:30.782 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11782 2023-07-02T14:05:30.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.782 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:30.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:30.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:30.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:30.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:30.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:30.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:30.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:30.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409506+0000", 2023-07-02T14:05:30.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020172269999999999 2023-07-02T14:05:30.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:30.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411523+0000", 2023-07-02T14:05:30.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8901000000000001e-05 2023-07-02T14:05:30.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:30.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411542+0000", 2023-07-02T14:05:30.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5037e-05 2023-07-02T14:05:30.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411557+0000", 2023-07-02T14:05:30.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001931213 2023-07-02T14:05:30.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:30.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413489+0000", 2023-07-02T14:05:30.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000149207 2023-07-02T14:05:30.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:30.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413638+0000", 2023-07-02T14:05:30.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024752800000000001 2023-07-02T14:05:30.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:30.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413885+0000", 2023-07-02T14:05:30.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048516000000000002 2023-07-02T14:05:30.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:30.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.418737+0000", 2023-07-02T14:05:30.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010319259000000001 2023-07-02T14:05:30.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:30.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429056+0000", 2023-07-02T14:05:30.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2518000000000002e-05 2023-07-02T14:05:30.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:30.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429099+0000", 2023-07-02T14:05:30.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015275 2023-07-02T14:05:30.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:30.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429251+0000", 2023-07-02T14:05:30.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365 2023-07-02T14:05:30.791 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.791 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.791 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.791 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPushReply(2.7 119/117 [PushReplyOp(2:eb862bc0:::rbd_data.125de1449605.0000000000000040:13)])", 2023-07-02T14:05:30.791 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:30.791 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.234513309, 2023-07-02T14:05:30.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999, 2023-07-02T14:05:30.792 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:30.792 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "reached pg", 2023-07-02T14:05:30.792 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:30.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:30.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:30.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:30.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:30.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:30.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:30.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:30.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:30.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7670000000000001e-06 2023-07-02T14:05:30.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:30.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410674+0000", 2023-07-02T14:05:30.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3973e-05 2023-07-02T14:05:30.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:30.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410688+0000", 2023-07-02T14:05:30.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0157e-05 2023-07-02T14:05:30.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410708+0000", 2023-07-02T14:05:30.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018578779 2023-07-02T14:05:30.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:30.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429287+0000", 2023-07-02T14:05:30.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013086549999999999 2023-07-02T14:05:30.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:30.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.430595+0000", 2023-07-02T14:05:30.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999 2023-07-02T14:05:30.797 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.797 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.797 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.798 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11861 2.7 2:e57d3f55:::rbd_data.125de1449605.0000000000000134:head [write 0~65536] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:05:30.798 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:30.798 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.100397932, 2023-07-02T14:05:30.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723, 2023-07-02T14:05:30.798 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:30.798 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.798 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:30.799 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:30.799 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:30.799 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11861 2023-07-02T14:05:30.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.799 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:30.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:30.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:30.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:30.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:30.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:30.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:30.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:30.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544785+0000", 2023-07-02T14:05:30.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010563700000000001 2023-07-02T14:05:30.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:30.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544890+0000", 2023-07-02T14:05:30.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8530000000000004e-06 2023-07-02T14:05:30.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:30.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544900+0000", 2023-07-02T14:05:30.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.665e-06 2023-07-02T14:05:30.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544906+0000", 2023-07-02T14:05:30.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000272575 2023-07-02T14:05:30.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:30.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545178+0000", 2023-07-02T14:05:30.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000111362 2023-07-02T14:05:30.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:30.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545290+0000", 2023-07-02T14:05:30.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016181100000000001 2023-07-02T14:05:30.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:30.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545452+0000", 2023-07-02T14:05:30.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044500750000000004 2023-07-02T14:05:30.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:30.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.549902+0000", 2023-07-02T14:05:30.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014850024 2023-07-02T14:05:30.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:30.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564752+0000", 2023-07-02T14:05:30.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6272000000000003e-05 2023-07-02T14:05:30.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:30.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564848+0000", 2023-07-02T14:05:30.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000162428 2023-07-02T14:05:30.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:30.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.565010+0000", 2023-07-02T14:05:30.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723 2023-07-02T14:05:30.807 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.808 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.808 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.808 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:13262 2.7 e125/121 2:e0741dde:::rbd_data.125de1449605.00000000000000ad:head v 125'2024, mlcod=125'2021)", 2023-07-02T14:05:30.808 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:30.808 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.394320194000002, 2023-07-02T14:05:30.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999, 2023-07-02T14:05:30.809 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:30.809 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:30.809 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:30.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:30.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:30.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:30.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:30.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:30.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:30.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:30.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250862+0000", 2023-07-02T14:05:30.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080378800000000007 2023-07-02T14:05:30.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:30.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258900+0000", 2023-07-02T14:05:30.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8939999999999999e-06 2023-07-02T14:05:30.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:30.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258909+0000", 2023-07-02T14:05:30.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3560000000000003e-06 2023-07-02T14:05:30.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:30.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258914+0000", 2023-07-02T14:05:30.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2429000000000003e-05 2023-07-02T14:05:30.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:30.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258956+0000", 2023-07-02T14:05:30.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5094999999999999e-05 2023-07-02T14:05:30.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:30.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258981+0000", 2023-07-02T14:05:30.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012177307 2023-07-02T14:05:30.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:30.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271158+0000", 2023-07-02T14:05:30.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2104999999999999e-05 2023-07-02T14:05:30.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:30.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:30.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:30.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271181+0000", 2023-07-02T14:05:30.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999 2023-07-02T14:05:30.815 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.815 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.815 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.815 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:30.816 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:30.816 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:30.817 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:05:30.817 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:30.816+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:30.894 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:05:30.917 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:30.917+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:30.982 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:30.982 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:30.982 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:30.983 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:30.992 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:05:31.034 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:31.033+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:31.035 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:31.054 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:31.055 INFO:teuthology.orchestra.run.smithi016.stdout: marking collection for removal 2023-07-02T14:05:31.056 INFO:teuthology.orchestra.run.smithi016.stdout:setting '_remove' omap key 2023-07-02T14:05:31.056 INFO:teuthology.orchestra.run.smithi016.stdout:finish_remove_pgs 2.4_head removing 2.4 2023-07-02T14:05:31.083 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:31.117 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:31.119 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:31.119+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:31.150 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:31.150 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-07-02T14:05:31.150 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:05:31.151 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:05:31.151 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:05:31.151 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:05:31.151 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:05:31.153 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:05:31.154 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:05:31.157 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:31.157 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:31.167 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:05:31.220 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:31.220+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:31.291 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:31.291 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:31.301 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:05:31.343 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:31.342+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:31.415 INFO:teuthology.orchestra.run.smithi016.stdout:Remove successful 2023-07-02T14:05:31.422 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:31.422+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:31.466 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:31.466 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:31.476 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:05:31.523 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:31.522+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:31.600 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:31.600 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:31.611 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:05:31.652 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:31.651+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:31.725 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:31.724+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:31.775 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:31.775 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:31.785 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:05:31.826 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:31.825+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:31.836 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.4.0 2023-07-02T14:05:31.916 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:31.917 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:31.927 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:05:31.928 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:31.927+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:32.027 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:32.027+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:32.051 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:32.051 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:32.061 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:05:32.128 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:32.128+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:32.198 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:05:32.229 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:32.229+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:32.230 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:32.230 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:32.245 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:05:32.330 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:32.330+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:32.413 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:32.413 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:32.424 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:05:32.467 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:32.467+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:32.532 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:32.532+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:32.548 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:32.549 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:32.559 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:05:32.632 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:32.632+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:32.723 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:32.723 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:32.734 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:05:32.734 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:32.733+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:32.834 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:32.833+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:32.872 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:32.872 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:32.883 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:05:32.966 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:32.965+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:33.036 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:33.035+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:33.047 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:33.048 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:33.058 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:05:33.087 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:05:33.141 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:33.140+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:33.155 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:05:33.221 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:33.221 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:33.231 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:05:33.237 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:33.237+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:33.294 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:33.311 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:33.329 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:33.338 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:33.337+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:33.347 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:33.365 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:33.365 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-07-02T14:05:33.366 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:05:33.366 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:05:33.366 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:05:33.366 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:05:33.366 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:05:33.368 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:05:33.368 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:05:33.396 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:33.396 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:33.396 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:33.396 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:33.406 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:05:33.447 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:33.446+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:33.539 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:33.539+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:33.569 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:33.569 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:33.569 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:33.569 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:33.570 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:33.579 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:05:33.661 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:33.661+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:33.699 INFO:teuthology.orchestra.run.smithi016.stdout:get_pg_num_history pg_num_history pg_num_history(e148 pg_nums {1={12=1},2={14=8}} deleted_pools ) 2023-07-02T14:05:33.700 INFO:teuthology.orchestra.run.smithi016.stdout:Importing pgid 2.4 2023-07-02T14:05:33.707 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2211c235:::rbd_data.125de1449605.0000000000000034:d# 2023-07-02T14:05:33.715 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2211c235:::rbd_data.125de1449605.0000000000000034:13# 2023-07-02T14:05:33.720 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2211c235:::rbd_data.125de1449605.0000000000000034:1a# 2023-07-02T14:05:33.725 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2211c235:::rbd_data.125de1449605.0000000000000034:21# 2023-07-02T14:05:33.730 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2211c235:::rbd_data.125de1449605.0000000000000034:26# 2023-07-02T14:05:33.735 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2211c235:::rbd_data.125de1449605.0000000000000034:head# 2023-07-02T14:05:33.735 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{d=[d,c,b,a,9,8],13=[13,12,11,10,f,e],1a=[1a,19,18,17,16,15,14],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:33.740 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:23089833:::rbd_data.125de1449605.0000000000000092:e# 2023-07-02T14:05:33.741 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:33.741+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:33.744 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:33.745 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:33.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:33.745 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:33.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.745 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:33.745 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:33.746 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.45237829199999, 2023-07-02T14:05:33.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:05:33.746 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:33.746 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:33.746 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:33.746 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:33.746 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:33.747 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:05:33.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.747 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:33.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:33.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:33.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:33.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:33.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:33.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:33.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:33.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:05:33.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:05:33.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:33.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:05:33.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:05:33.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:33.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:05:33.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:05:33.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:33.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:05:33.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:05:33.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:33.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:05:33.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:05:33.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:33.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:05:33.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:05:33.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:33.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:05:33.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:05:33.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:33.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:05:33.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:05:33.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:33.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:05:33.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:05:33.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:33.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:05:33.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:05:33.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:33.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:05:33.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:05:33.755 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.755 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.755 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.756 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:33.756 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:33.756 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.452143269, 2023-07-02T14:05:33.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:05:33.756 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:33.757 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:33.757 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:33.757 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:33.757 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:33.757 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:05:33.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.757 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:33.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:33.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:33.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:33.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:33.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:33.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:33.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:33.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:05:33.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:05:33.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:33.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:05:33.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:05:33.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:33.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:05:33.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:05:33.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:33.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:05:33.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:05:33.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:33.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:05:33.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:05:33.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:33.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:05:33.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:05:33.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:33.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:05:33.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:05:33.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:33.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:05:33.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:05:33.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:33.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:05:33.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:05:33.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:33.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:05:33.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:05:33.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:33.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:05:33.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:05:33.766 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.766 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.766 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.766 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:33.767 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:33.767 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.45194413499999, 2023-07-02T14:05:33.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:05:33.767 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:33.767 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:33.767 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:33.767 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:33.767 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:33.768 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:05:33.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.768 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:33.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:33.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:33.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:33.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:33.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:33.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:33.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:33.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:05:33.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:05:33.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:33.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:05:33.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:05:33.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:33.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:05:33.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:05:33.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:33.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:05:33.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:05:33.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:33.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:05:33.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:05:33.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:33.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:05:33.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:05:33.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:33.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:05:33.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:05:33.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:33.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:05:33.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:05:33.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:33.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:05:33.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:05:33.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:33.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:05:33.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:05:33.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:33.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:05:33.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:05:33.776 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.776 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.776 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.777 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:33.777 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:33.777 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.45157901100001, 2023-07-02T14:05:33.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:05:33.777 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:33.778 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:33.778 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:33.778 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:33.778 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:33.778 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:05:33.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.778 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:33.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:33.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:33.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:33.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:33.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:33.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:33.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:33.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:05:33.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:05:33.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:33.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:05:33.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:05:33.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:33.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:05:33.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:05:33.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:33.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:05:33.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:05:33.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:33.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:05:33.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:05:33.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:33.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:05:33.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:05:33.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:33.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:05:33.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:05:33.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:33.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:05:33.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:05:33.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:33.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:05:33.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:05:33.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:33.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:05:33.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:05:33.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:33.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:05:33.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:05:33.787 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.787 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.787 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.787 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:33.787 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:33.787 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.45149795, 2023-07-02T14:05:33.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:05:33.788 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:33.788 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:33.788 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:33.788 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:33.788 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:33.788 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:05:33.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.789 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:33.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:33.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:33.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:33.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:33.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:33.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:33.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:33.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:05:33.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:05:33.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:33.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:05:33.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:05:33.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:33.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:05:33.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:05:33.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:33.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:05:33.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:05:33.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:33.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:05:33.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:05:33.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:33.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:05:33.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:05:33.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:33.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:05:33.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:05:33.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:33.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:05:33.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:05:33.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:33.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:05:33.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:05:33.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:33.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:05:33.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:05:33.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:33.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:05:33.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:05:33.797 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.797 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.797 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.798 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:33.798 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:33.798 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.451406279, 2023-07-02T14:05:33.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:05:33.798 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:33.798 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:33.798 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:33.799 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:33.799 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:33.799 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:05:33.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.799 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:33.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:33.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:33.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:33.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:33.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:33.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:33.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:33.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:05:33.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:05:33.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:33.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:05:33.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:05:33.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:33.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:05:33.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:05:33.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:33.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:05:33.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:05:33.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:33.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:05:33.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:05:33.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:33.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:05:33.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:05:33.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:33.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:05:33.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:05:33.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:33.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:05:33.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:05:33.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:33.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:05:33.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:05:33.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:33.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:05:33.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:05:33.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:33.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:05:33.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:05:33.807 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.808 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.808 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.808 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:33.808 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:33.808 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.45106683700001, 2023-07-02T14:05:33.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:05:33.809 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:33.809 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:33.809 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:33.809 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:33.809 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:33.809 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:05:33.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.810 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:33.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:33.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:33.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:33.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:33.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:33.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:33.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:33.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:05:33.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:05:33.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:33.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:05:33.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:05:33.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:33.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:05:33.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:05:33.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:33.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:05:33.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:05:33.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:33.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:05:33.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:05:33.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:33.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:05:33.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:05:33.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:33.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:05:33.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:05:33.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:33.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:05:33.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:05:33.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:33.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:05:33.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:05:33.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:33.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:05:33.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:05:33.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:33.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:05:33.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:05:33.818 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.818 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.818 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.818 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:33.819 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:33.819 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.45088078399999, 2023-07-02T14:05:33.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:05:33.819 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:33.819 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:33.819 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:33.819 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:33.820 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:33.820 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:05:33.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.820 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:33.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:33.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:33.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:33.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:33.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:33.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:33.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:33.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:33.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:05:33.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:33.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:05:33.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:05:33.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:33.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:05:33.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:05:33.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:33.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:05:33.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:05:33.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:33.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:05:33.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:05:33.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:33.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:05:33.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:05:33.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:33.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:05:33.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:05:33.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:33.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:05:33.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:05:33.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:33.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:05:33.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:05:33.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:33.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:05:33.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:05:33.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:33.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:05:33.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:05:33.828 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.828 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.828 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.829 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:33.829 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:33.829 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.450722982, 2023-07-02T14:05:33.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:05:33.829 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:33.829 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:33.830 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:33.830 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:33.830 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:33.830 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:05:33.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.830 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:33.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:33.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:33.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:33.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:33.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:33.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:33.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:33.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:05:33.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:05:33.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:33.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:05:33.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:05:33.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:33.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:05:33.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:05:33.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:33.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:05:33.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:05:33.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:33.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:05:33.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:05:33.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:33.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:05:33.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:05:33.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:33.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:05:33.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:05:33.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:33.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:05:33.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:05:33.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:33.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:05:33.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:05:33.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:33.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:05:33.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:05:33.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:33.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:05:33.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:05:33.838 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.839 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.839 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.839 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:33.839 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:33.839 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.4505312, 2023-07-02T14:05:33.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:05:33.840 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:33.840 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:33.840 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:33.840 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:33.840 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:33.840 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:05:33.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.840 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:33.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:33.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:33.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:33.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:33.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:33.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:33.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:33.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:05:33.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:05:33.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:33.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:05:33.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:05:33.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:33.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:05:33.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:05:33.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:33.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:05:33.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:05:33.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:33.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:05:33.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:05:33.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:33.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:05:33.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:05:33.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:33.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:05:33.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:05:33.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:33.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:05:33.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:05:33.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:33.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:05:33.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:05:33.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:33.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:05:33.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:05:33.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:33.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:05:33.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:05:33.849 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.849 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.849 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.849 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:33.850 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:33.850 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.45035554, 2023-07-02T14:05:33.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:05:33.850 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:33.850 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:33.850 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:33.850 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:33.851 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:33.851 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:05:33.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.851 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:33.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:33.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:33.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:33.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:33.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:33.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:33.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:33.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:33.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:05:33.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:33.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:05:33.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:05:33.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:33.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:05:33.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:05:33.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:33.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:05:33.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:05:33.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:33.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:05:33.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:05:33.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:33.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:05:33.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:05:33.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:33.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:05:33.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:05:33.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:33.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:05:33.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:05:33.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:33.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:05:33.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:05:33.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:33.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:05:33.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:05:33.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:33.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:05:33.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:05:33.859 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.859 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.859 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.860 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:33.860 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:33.860 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.45026135200001, 2023-07-02T14:05:33.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:05:33.860 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:33.860 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:33.861 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:33.861 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:33.861 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:33.861 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:05:33.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.861 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:33.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:33.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:33.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:33.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:33.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:33.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:33.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:33.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:05:33.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:05:33.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:33.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:05:33.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:05:33.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:33.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:05:33.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:05:33.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:33.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:05:33.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:05:33.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:33.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:05:33.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:05:33.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:33.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:05:33.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:05:33.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:33.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:05:33.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:05:33.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:33.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:05:33.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:05:33.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:33.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:05:33.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:05:33.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:33.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:05:33.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:05:33.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:33.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:05:33.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:05:33.869 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.870 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.870 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.870 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:33.870 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:33.870 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.450057966, 2023-07-02T14:05:33.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:05:33.871 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:33.871 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:33.871 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:33.871 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:33.871 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:33.871 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:05:33.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.872 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:33.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:33.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:33.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:33.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:33.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:33.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:33.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:33.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:05:33.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:05:33.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:33.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:05:33.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:05:33.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:33.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:05:33.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:05:33.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:33.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:05:33.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:05:33.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:33.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:05:33.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:05:33.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:33.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:05:33.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:05:33.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:33.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:05:33.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:05:33.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:33.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:05:33.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:05:33.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:33.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:05:33.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:05:33.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:33.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:05:33.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:05:33.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:33.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:05:33.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:05:33.880 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.880 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.880 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.880 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:33.881 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:33.881 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.44990730999999, 2023-07-02T14:05:33.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:05:33.881 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:33.881 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:33.881 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:33.881 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:33.881 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:33.882 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:05:33.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.882 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:33.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:33.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:33.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:33.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:33.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:33.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:33.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:33.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:05:33.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:05:33.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:33.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:05:33.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:05:33.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:33.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:05:33.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:05:33.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:33.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:05:33.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:05:33.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:33.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:05:33.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:05:33.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:33.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:05:33.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:05:33.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:33.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:05:33.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:05:33.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:33.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:05:33.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:05:33.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:33.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:05:33.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:05:33.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:33.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:05:33.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:05:33.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:33.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:05:33.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:05:33.890 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.890 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.891 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.891 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:33.891 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:33.891 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.44983460700001, 2023-07-02T14:05:33.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:05:33.891 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:33.892 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:33.892 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:33.892 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:33.892 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:33.892 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:05:33.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.892 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:33.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.893 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:33.893 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:33.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:33.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.893 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:33.893 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:33.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:33.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:33.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:33.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:05:33.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:33.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:05:33.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:05:33.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:33.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:05:33.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:05:33.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:33.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:05:33.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:05:33.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:33.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:05:33.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:05:33.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:33.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:05:33.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:05:33.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:33.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:05:33.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:05:33.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:33.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:05:33.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:05:33.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:33.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:05:33.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:05:33.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:33.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:05:33.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:05:33.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:33.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:05:33.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:05:33.901 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.901 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.901 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.901 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:33.901 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:33.902 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.449717617, 2023-07-02T14:05:33.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:05:33.902 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:33.902 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:33.902 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:33.902 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:33.902 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:33.902 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:05:33.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.903 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:33.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:33.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:33.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:33.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:33.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:33.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:33.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:33.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:33.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:05:33.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:33.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:05:33.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:05:33.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:33.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:05:33.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:05:33.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:33.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:05:33.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:05:33.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:33.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:05:33.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:05:33.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:33.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:05:33.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:05:33.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:33.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:05:33.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:05:33.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:33.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:05:33.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:05:33.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:33.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:05:33.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:05:33.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:33.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:05:33.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:05:33.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:33.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:05:33.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:05:33.911 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.911 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.911 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.912 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:33.912 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:33.912 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.44956816600001, 2023-07-02T14:05:33.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:05:33.912 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:33.912 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:33.912 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:33.913 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:33.913 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:33.913 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:05:33.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.913 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:33.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:33.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:33.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:33.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:33.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:33.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:33.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:33.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:05:33.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:05:33.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:33.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:05:33.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:05:33.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:33.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:05:33.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:05:33.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:33.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:05:33.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:05:33.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:33.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:05:33.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:05:33.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:33.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:05:33.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:05:33.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:33.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:05:33.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:05:33.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:33.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:05:33.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:05:33.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:33.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:05:33.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:05:33.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:33.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:05:33.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:05:33.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:33.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:05:33.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:05:33.921 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.921 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.922 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.922 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:33.922 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:33.922 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.428092193, 2023-07-02T14:05:33.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:05:33.923 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:33.923 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:33.923 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:33.923 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:33.923 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:33.923 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:05:33.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.923 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:33.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:33.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:33.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:33.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:33.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:33.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:33.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:33.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:05:33.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:05:33.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:33.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:05:33.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:05:33.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:33.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:05:33.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:05:33.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:33.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:05:33.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:05:33.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:33.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:05:33.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:05:33.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:33.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:05:33.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:05:33.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:33.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:05:33.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:05:33.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:33.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:05:33.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:05:33.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:33.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:05:33.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:05:33.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:33.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:05:33.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:05:33.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:33.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:05:33.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:05:33.932 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.932 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.932 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.932 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:33.932 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:33.933 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.427959657, 2023-07-02T14:05:33.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:05:33.933 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:33.933 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:33.933 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:33.933 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:33.933 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:33.933 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:05:33.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.934 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:33.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:33.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:33.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:33.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:33.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:33.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:33.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:33.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:05:33.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:05:33.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:33.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:05:33.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:05:33.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:33.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:05:33.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:05:33.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:33.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:05:33.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:05:33.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:33.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:05:33.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:05:33.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:33.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:05:33.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:05:33.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:33.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:05:33.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:05:33.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:33.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:05:33.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:05:33.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:33.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:05:33.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:05:33.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:33.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:05:33.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:05:33.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:33.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:05:33.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:05:33.942 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.942 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.942 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.943 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:33.943 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:33.943 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.36662299599999, 2023-07-02T14:05:33.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:05:33.943 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:33.943 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:33.944 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:33.944 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:33.944 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:33.944 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:05:33.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.944 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:33.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:33.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:33.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:33.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:33.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:33.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:33.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:33.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:05:33.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:05:33.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:33.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:05:33.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:05:33.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:33.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:05:33.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:05:33.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:33.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:05:33.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:05:33.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:33.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:05:33.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:05:33.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:33.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:05:33.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:05:33.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:33.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:05:33.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:05:33.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:33.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:05:33.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:05:33.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:33.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:05:33.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:05:33.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:33.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:05:33.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:05:33.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:33.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:33.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:33.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:05:33.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:05:33.953 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.953 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.953 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.953 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:33.953 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:33.953 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:33.955 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:05:33.958 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:23089833:::rbd_data.125de1449605.0000000000000092:13# 2023-07-02T14:05:33.958 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:23089833:::rbd_data.125de1449605.0000000000000092:1a# 2023-07-02T14:05:33.958 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:23089833:::rbd_data.125de1449605.0000000000000092:21# 2023-07-02T14:05:33.959 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:23089833:::rbd_data.125de1449605.0000000000000092:26# 2023-07-02T14:05:33.959 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:23089833:::rbd_data.125de1449605.0000000000000092:head# 2023-07-02T14:05:33.959 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{e=[e,d,c,b,a,9,8],13=[13,12,11,10,f],1a=[1a,19,18,17,16,15,14],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:33.959 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:237c32fb:::rbd_object_map.125de1449605.000000000000000b:head# 2023-07-02T14:05:33.959 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 0=[]:{} 2023-07-02T14:05:33.959 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:24849def:::rbd_data.125de1449605.0000000000000138:1f# 2023-07-02T14:05:33.959 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:24849def:::rbd_data.125de1449605.0000000000000138:head# 2023-07-02T14:05:33.960 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 1f=[]:{1f=[1f]} 2023-07-02T14:05:33.960 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25261dc8:::rbd_data.125de1449605.00000000000000e3:e# 2023-07-02T14:05:33.960 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25261dc8:::rbd_data.125de1449605.00000000000000e3:13# 2023-07-02T14:05:33.960 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25261dc8:::rbd_data.125de1449605.00000000000000e3:14# 2023-07-02T14:05:33.960 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25261dc8:::rbd_data.125de1449605.00000000000000e3:1a# 2023-07-02T14:05:33.960 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25261dc8:::rbd_data.125de1449605.00000000000000e3:21# 2023-07-02T14:05:33.960 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25261dc8:::rbd_data.125de1449605.00000000000000e3:26# 2023-07-02T14:05:33.960 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25261dc8:::rbd_data.125de1449605.00000000000000e3:head# 2023-07-02T14:05:33.961 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{e=[e,d,c,b,a,9,8],13=[13,12,11,10,f],14=[14],1a=[1a,19,18,17,16,15],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:33.961 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25643af7:::rbd_data.125de1449605.0000000000000052:e# 2023-07-02T14:05:33.961 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25643af7:::rbd_data.125de1449605.0000000000000052:13# 2023-07-02T14:05:33.961 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25643af7:::rbd_data.125de1449605.0000000000000052:1a# 2023-07-02T14:05:33.961 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25643af7:::rbd_data.125de1449605.0000000000000052:21# 2023-07-02T14:05:33.961 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25643af7:::rbd_data.125de1449605.0000000000000052:26# 2023-07-02T14:05:33.961 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25643af7:::rbd_data.125de1449605.0000000000000052:head# 2023-07-02T14:05:33.961 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{e=[e,d,c,b,a,9,8],13=[13,12,11,10,f],1a=[1a,19,18,17,16,15,14],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:33.962 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25711f4d:::rbd_data.125de1449605.00000000000000fa:e# 2023-07-02T14:05:33.962 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25711f4d:::rbd_data.125de1449605.00000000000000fa:14# 2023-07-02T14:05:33.962 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25711f4d:::rbd_data.125de1449605.00000000000000fa:1a# 2023-07-02T14:05:33.962 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25711f4d:::rbd_data.125de1449605.00000000000000fa:head# 2023-07-02T14:05:33.962 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 1a=[]:{e=[e,d,c,b,a,9,8],14=[14,13,12,11,10,f],1a=[1a,19,18,17,16,15]} 2023-07-02T14:05:33.962 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25e15f0e:::rbd_data.125de1449605.0000000000000087:e# 2023-07-02T14:05:33.962 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25e15f0e:::rbd_data.125de1449605.0000000000000087:13# 2023-07-02T14:05:33.963 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25e15f0e:::rbd_data.125de1449605.0000000000000087:1a# 2023-07-02T14:05:33.963 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25e15f0e:::rbd_data.125de1449605.0000000000000087:21# 2023-07-02T14:05:33.963 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25e15f0e:::rbd_data.125de1449605.0000000000000087:26# 2023-07-02T14:05:33.963 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:25e15f0e:::rbd_data.125de1449605.0000000000000087:head# 2023-07-02T14:05:33.963 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{e=[e,d,c,b,a,9,8],13=[13,12,11,10,f],1a=[1a,19,18,17,16,15,14],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:33.963 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:261eca8d:::rbd_data.125de1449605.00000000000000ca:d# 2023-07-02T14:05:33.963 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:261eca8d:::rbd_data.125de1449605.00000000000000ca:e# 2023-07-02T14:05:33.963 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:261eca8d:::rbd_data.125de1449605.00000000000000ca:13# 2023-07-02T14:05:33.964 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:261eca8d:::rbd_data.125de1449605.00000000000000ca:1a# 2023-07-02T14:05:33.964 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:261eca8d:::rbd_data.125de1449605.00000000000000ca:21# 2023-07-02T14:05:33.964 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:261eca8d:::rbd_data.125de1449605.00000000000000ca:26# 2023-07-02T14:05:33.964 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:261eca8d:::rbd_data.125de1449605.00000000000000ca:head# 2023-07-02T14:05:33.964 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{d=[d,c,b,a,9,8],e=[e],13=[13,12,11,10,f],1a=[1a,19,18,17,16,15,14],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:33.964 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2628942f:::rbd_data.125de1449605.0000000000000005:d# 2023-07-02T14:05:33.964 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2628942f:::rbd_data.125de1449605.0000000000000005:13# 2023-07-02T14:05:33.964 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2628942f:::rbd_data.125de1449605.0000000000000005:19# 2023-07-02T14:05:33.965 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2628942f:::rbd_data.125de1449605.0000000000000005:1e# 2023-07-02T14:05:33.965 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2628942f:::rbd_data.125de1449605.0000000000000005:25# 2023-07-02T14:05:33.965 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2628942f:::rbd_data.125de1449605.0000000000000005:head# 2023-07-02T14:05:33.965 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 25=[]:{d=[d,c,b,a,9,8,7,6,5,4],13=[13,12,11,10,f,e],19=[19,18,17,16,15,14],1e=[1e,1d,1c,1b,1a],25=[25,24,23,22,21,20,1f]} 2023-07-02T14:05:33.965 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:33.954+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:33.966 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:265ad6c3:::rbd_data.125de1449605.00000000000000fd:e# 2023-07-02T14:05:33.966 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:265ad6c3:::rbd_data.125de1449605.00000000000000fd:14# 2023-07-02T14:05:33.966 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:265ad6c3:::rbd_data.125de1449605.00000000000000fd:1a# 2023-07-02T14:05:33.969 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:265ad6c3:::rbd_data.125de1449605.00000000000000fd:head# 2023-07-02T14:05:33.969 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 1a=[]:{e=[e,d,c,b,a,9,8],14=[14,13,12,11,10,f],1a=[1a,19,18,17,16,15]} 2023-07-02T14:05:33.974 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:26f20587:::rbd_data.125de1449605.0000000000000071:e# 2023-07-02T14:05:33.978 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:26f20587:::rbd_data.125de1449605.0000000000000071:13# 2023-07-02T14:05:33.983 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:26f20587:::rbd_data.125de1449605.0000000000000071:1a# 2023-07-02T14:05:33.988 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:26f20587:::rbd_data.125de1449605.0000000000000071:21# 2023-07-02T14:05:33.992 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:26f20587:::rbd_data.125de1449605.0000000000000071:26# 2023-07-02T14:05:33.997 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:26f20587:::rbd_data.125de1449605.0000000000000071:head# 2023-07-02T14:05:33.998 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{e=[e,d,c,b,a,9,8],13=[13,12,11,10,f],1a=[1a,19,18,17,16,15,14],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:34.002 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:27cdd850:::rbd_data.125de1449605.0000000000000009:8# 2023-07-02T14:05:34.003 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:27cdd850:::rbd_data.125de1449605.0000000000000009:e# 2023-07-02T14:05:34.004 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:27cdd850:::rbd_data.125de1449605.0000000000000009:13# 2023-07-02T14:05:34.005 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:27cdd850:::rbd_data.125de1449605.0000000000000009:14# 2023-07-02T14:05:34.007 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:27cdd850:::rbd_data.125de1449605.0000000000000009:1a# 2023-07-02T14:05:34.009 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:27cdd850:::rbd_data.125de1449605.0000000000000009:21# 2023-07-02T14:05:34.011 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:27cdd850:::rbd_data.125de1449605.0000000000000009:26# 2023-07-02T14:05:34.013 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:27cdd850:::rbd_data.125de1449605.0000000000000009:head# 2023-07-02T14:05:34.014 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{8=[8,7,6,5,4],e=[e,d,c,b,a,9],13=[13,12,11,10,f],14=[14],1a=[1a,19,18,17,16,15],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:34.016 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:28147dd3:::rbd_data.125de1449605.0000000000000112:e# 2023-07-02T14:05:34.020 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:28147dd3:::rbd_data.125de1449605.0000000000000112:14# 2023-07-02T14:05:34.025 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:28147dd3:::rbd_data.125de1449605.0000000000000112:1a# 2023-07-02T14:05:34.030 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:28147dd3:::rbd_data.125de1449605.0000000000000112:head# 2023-07-02T14:05:34.031 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 1a=[]:{e=[e,d,c,b,a,9,8],14=[14,13,12,11,10,f],1a=[1a,19,18,17,16,15]} 2023-07-02T14:05:34.035 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:28376491:::rbd_data.125de1449605.000000000000011d:e# 2023-07-02T14:05:34.040 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:28376491:::rbd_data.125de1449605.000000000000011d:14# 2023-07-02T14:05:34.045 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:28376491:::rbd_data.125de1449605.000000000000011d:1a# 2023-07-02T14:05:34.050 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:28376491:::rbd_data.125de1449605.000000000000011d:head# 2023-07-02T14:05:34.050 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 1a=[]:{e=[e,d,c,b,a,9],14=[14,13,12,11,10,f],1a=[1a,19,18,17,16,15]} 2023-07-02T14:05:34.055 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:28c6fc97:::rbd_data.125de1449605.000000000000014e:21# 2023-07-02T14:05:34.055 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:34.054+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:34.058 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:28c6fc97:::rbd_data.125de1449605.000000000000014e:25# 2023-07-02T14:05:34.062 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:28c6fc97:::rbd_data.125de1449605.000000000000014e:head# 2023-07-02T14:05:34.063 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 25=[]:{21=[21],25=[25,24,23,22]} 2023-07-02T14:05:34.067 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2a14a29c:::rbd_object_map.125de1449605.0000000000000013:head# 2023-07-02T14:05:34.067 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 0=[]:{} 2023-07-02T14:05:34.068 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2a8b9a55:::rbd_data.125de1449605.0000000000000088:e# 2023-07-02T14:05:34.072 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2a8b9a55:::rbd_data.125de1449605.0000000000000088:13# 2023-07-02T14:05:34.077 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2a8b9a55:::rbd_data.125de1449605.0000000000000088:1a# 2023-07-02T14:05:34.082 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2a8b9a55:::rbd_data.125de1449605.0000000000000088:21# 2023-07-02T14:05:34.087 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2a8b9a55:::rbd_data.125de1449605.0000000000000088:26# 2023-07-02T14:05:34.092 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2a8b9a55:::rbd_data.125de1449605.0000000000000088:head# 2023-07-02T14:05:34.093 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{e=[e,d,c,b,a,9,8],13=[13,12,11,10,f],1a=[1a,19,18,17,16,15,14],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:34.097 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2aa9c62e:::rbd_data.125de1449605.0000000000000048:e# 2023-07-02T14:05:34.102 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2aa9c62e:::rbd_data.125de1449605.0000000000000048:13# 2023-07-02T14:05:34.107 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2aa9c62e:::rbd_data.125de1449605.0000000000000048:1a# 2023-07-02T14:05:34.112 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2aa9c62e:::rbd_data.125de1449605.0000000000000048:21# 2023-07-02T14:05:34.116 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2aa9c62e:::rbd_data.125de1449605.0000000000000048:26# 2023-07-02T14:05:34.120 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:34.120 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:34.120 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:34.121 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:34.121 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2aa9c62e:::rbd_data.125de1449605.0000000000000048:head# 2023-07-02T14:05:34.122 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{e=[e,d,c,b,a,9,8],13=[13,12,11,10,f],1a=[1a,19,18,17,16,15,14],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:34.126 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2aa9ee50:::rbd_data.125de1449605.0000000000000055:e# 2023-07-02T14:05:34.130 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:05:34.131 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2aa9ee50:::rbd_data.125de1449605.0000000000000055:13# 2023-07-02T14:05:34.136 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2aa9ee50:::rbd_data.125de1449605.0000000000000055:1a# 2023-07-02T14:05:34.141 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2aa9ee50:::rbd_data.125de1449605.0000000000000055:21# 2023-07-02T14:05:34.146 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2aa9ee50:::rbd_data.125de1449605.0000000000000055:26# 2023-07-02T14:05:34.151 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2aa9ee50:::rbd_data.125de1449605.0000000000000055:head# 2023-07-02T14:05:34.151 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{e=[e,d,c,b,a,9,8],13=[13,12,11,10,f],1a=[1a,19,18,17,16,15,14],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:34.156 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2c79e900:::rbd_data.125de1449605.000000000000004a:e# 2023-07-02T14:05:34.160 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2c79e900:::rbd_data.125de1449605.000000000000004a:13# 2023-07-02T14:05:34.165 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2c79e900:::rbd_data.125de1449605.000000000000004a:1a# 2023-07-02T14:05:34.170 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2c79e900:::rbd_data.125de1449605.000000000000004a:21# 2023-07-02T14:05:34.172 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:34.171+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:34.174 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2c79e900:::rbd_data.125de1449605.000000000000004a:26# 2023-07-02T14:05:34.179 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2c79e900:::rbd_data.125de1449605.000000000000004a:head# 2023-07-02T14:05:34.180 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{e=[e,d,c,b,a,9,8],13=[13,12,11,10,f],1a=[1a,19,18,17,16,15,14],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:34.184 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:7# 2023-07-02T14:05:34.189 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:8# 2023-07-02T14:05:34.194 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:9# 2023-07-02T14:05:34.198 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:c# 2023-07-02T14:05:34.203 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:e# 2023-07-02T14:05:34.208 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:f# 2023-07-02T14:05:34.212 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:13# 2023-07-02T14:05:34.217 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:14# 2023-07-02T14:05:34.222 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:15# 2023-07-02T14:05:34.227 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:19# 2023-07-02T14:05:34.229 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap38 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap39 b3fbb028-e267-4d25-a750-93ba18a2316a@snap39.diff 2023-07-02T14:05:34.232 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:1a# 2023-07-02T14:05:34.236 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:1b# 2023-07-02T14:05:34.241 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:1c# 2023-07-02T14:05:34.246 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:20# 2023-07-02T14:05:34.250 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:21# 2023-07-02T14:05:34.255 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:23# 2023-07-02T14:05:34.257 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:34.256+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:34.260 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:25# 2023-07-02T14:05:34.265 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:26# 2023-07-02T14:05:34.270 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head# 2023-07-02T14:05:34.270 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{7=[7,6,5,4],8=[8],9=[9],c=[c,b,a],e=[e,d],f=[f],13=[13,12,11,10],14=[14],15=[15],19=[19,18,17,16],1a=[1a],1b=[1b],1c=[1c],20=[20,1f,1e,1d],21=[21],23=[23,22],25=[25,24],26=[26]} 2023-07-02T14:05:34.275 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2e32ef35:::rbd_data.125de1449605.0000000000000141:20# 2023-07-02T14:05:34.278 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2e32ef35:::rbd_data.125de1449605.0000000000000141:25# 2023-07-02T14:05:34.279 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 100% complete...done. 2023-07-02T14:05:34.283 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:05:34.283 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=40 2023-07-02T14:05:34.283 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap40 2023-07-02T14:05:34.283 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2e32ef35:::rbd_data.125de1449605.0000000000000141:head# 2023-07-02T14:05:34.284 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 25=[]:{20=[20],25=[25,24,23,22,21]} 2023-07-02T14:05:34.288 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2e908755:::rbd_data.125de1449605.00000000000000fb:e# 2023-07-02T14:05:34.293 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2e908755:::rbd_data.125de1449605.00000000000000fb:14# 2023-07-02T14:05:34.294 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:34.295 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:34.295 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:34.295 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:34.295 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:34.298 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2e908755:::rbd_data.125de1449605.00000000000000fb:1a# 2023-07-02T14:05:34.302 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2e908755:::rbd_data.125de1449605.00000000000000fb:head# 2023-07-02T14:05:34.303 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 1a=[]:{e=[e,d,c,b,a,9,8],14=[14,13,12,11,10,f],1a=[1a,19,18,17,16,15]} 2023-07-02T14:05:34.305 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:05:34.308 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2f8e3c69:::rbd_data.125de1449605.0000000000000028:d# 2023-07-02T14:05:34.313 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2f8e3c69:::rbd_data.125de1449605.0000000000000028:e# 2023-07-02T14:05:34.317 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2f8e3c69:::rbd_data.125de1449605.0000000000000028:13# 2023-07-02T14:05:34.322 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2f8e3c69:::rbd_data.125de1449605.0000000000000028:1a# 2023-07-02T14:05:34.327 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2f8e3c69:::rbd_data.125de1449605.0000000000000028:21# 2023-07-02T14:05:34.332 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2f8e3c69:::rbd_data.125de1449605.0000000000000028:26# 2023-07-02T14:05:34.337 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:2f8e3c69:::rbd_data.125de1449605.0000000000000028:head# 2023-07-02T14:05:34.338 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{d=[d,c,b,a,9,8],e=[e],13=[13,12,11,10,f],1a=[1a,19,18,17,16,15,14],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:34.342 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:32e21197:::rbd_data.125de1449605.00000000000000cf:d# 2023-07-02T14:05:34.347 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:32e21197:::rbd_data.125de1449605.00000000000000cf:e# 2023-07-02T14:05:34.352 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:32e21197:::rbd_data.125de1449605.00000000000000cf:13# 2023-07-02T14:05:34.356 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:32e21197:::rbd_data.125de1449605.00000000000000cf:1a# 2023-07-02T14:05:34.362 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:32e21197:::rbd_data.125de1449605.00000000000000cf:21# 2023-07-02T14:05:34.366 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:32e21197:::rbd_data.125de1449605.00000000000000cf:26# 2023-07-02T14:05:34.371 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:32e21197:::rbd_data.125de1449605.00000000000000cf:head# 2023-07-02T14:05:34.372 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{d=[d,c,b,a,9,8],e=[e],13=[13,12,11,10,f],1a=[1a,19,18,17,16,15,14],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:34.376 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:34dd1278:::rbd_object_map.125de1449605.0000000000000020:head# 2023-07-02T14:05:34.376 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 0=[]:{} 2023-07-02T14:05:34.377 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:35184cfb:::rbd_data.125de1449605.000000000000012a:e# 2023-07-02T14:05:34.379 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:35184cfb:::rbd_data.125de1449605.000000000000012a:14# 2023-07-02T14:05:34.383 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:35184cfb:::rbd_data.125de1449605.000000000000012a:1a# 2023-07-02T14:05:34.387 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:34.387+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:34.388 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:35184cfb:::rbd_data.125de1449605.000000000000012a:1e# 2023-07-02T14:05:34.393 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:35184cfb:::rbd_data.125de1449605.000000000000012a:head# 2023-07-02T14:05:34.393 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 1e=[]:{e=[e,d,c,b,a,9],14=[14,13,12,11,10,f],1a=[1a,19,18,17,16,15],1e=[1e,1d,1c,1b]} 2023-07-02T14:05:34.398 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:357fce6c:::rbd_data.125de1449605.00000000000000c8:d# 2023-07-02T14:05:34.402 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:357fce6c:::rbd_data.125de1449605.00000000000000c8:e# 2023-07-02T14:05:34.408 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:357fce6c:::rbd_data.125de1449605.00000000000000c8:13# 2023-07-02T14:05:34.413 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:357fce6c:::rbd_data.125de1449605.00000000000000c8:1a# 2023-07-02T14:05:34.417 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:357fce6c:::rbd_data.125de1449605.00000000000000c8:21# 2023-07-02T14:05:34.422 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:357fce6c:::rbd_data.125de1449605.00000000000000c8:26# 2023-07-02T14:05:34.427 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:357fce6c:::rbd_data.125de1449605.00000000000000c8:head# 2023-07-02T14:05:34.428 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{d=[d,c,b,a,9,8],e=[e],13=[13,12,11,10,f],1a=[1a,19,18,17,16,15,14],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:34.433 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3625364f:::rbd_data.125de1449605.0000000000000026:4# 2023-07-02T14:05:34.436 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3625364f:::rbd_data.125de1449605.0000000000000026:7# 2023-07-02T14:05:34.440 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3625364f:::rbd_data.125de1449605.0000000000000026:d# 2023-07-02T14:05:34.445 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3625364f:::rbd_data.125de1449605.0000000000000026:13# 2023-07-02T14:05:34.450 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3625364f:::rbd_data.125de1449605.0000000000000026:1a# 2023-07-02T14:05:34.455 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3625364f:::rbd_data.125de1449605.0000000000000026:21# 2023-07-02T14:05:34.459 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3625364f:::rbd_data.125de1449605.0000000000000026:26# 2023-07-02T14:05:34.460 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:34.460+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:34.464 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3625364f:::rbd_data.125de1449605.0000000000000026:head# 2023-07-02T14:05:34.465 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{4=[4],7=[7,6,5],d=[d,c,b,a,9,8],13=[13,12,11,10,f,e],1a=[1a,19,18,17,16,15,14],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:34.470 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3696af49:::rbd_data.125de1449605.000000000000015b:25# 2023-07-02T14:05:34.471 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3696af49:::rbd_data.125de1449605.000000000000015b:head# 2023-07-02T14:05:34.472 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:34.473 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:34.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:34.473 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:34.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.473 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15701 2.5 2:babe4c4a:::rbd_data.125de1449605.00000000000000e7:head [write 524288~3670016] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:34.473 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.463355+0000", 2023-07-02T14:05:34.473 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 15.005718696000001, 2023-07-02T14:05:34.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030948784, 2023-07-02T14:05:34.474 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:34.474 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:34.474 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:34.474 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:34.474 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:34.474 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15701 2023-07-02T14:05:34.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.475 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:34.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:34.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463355+0000", 2023-07-02T14:05:34.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:34.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:34.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463355+0000", 2023-07-02T14:05:34.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:34.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:34.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.463354+0000", 2023-07-02T14:05:34.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005426246 2023-07-02T14:05:34.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:34.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.468781+0000", 2023-07-02T14:05:34.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6320000000000003e-06 2023-07-02T14:05:34.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:34.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.468790+0000", 2023-07-02T14:05:34.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8080000000000004e-06 2023-07-02T14:05:34.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:34.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.468795+0000", 2023-07-02T14:05:34.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089891880000000004 2023-07-02T14:05:34.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.477784+0000", 2023-07-02T14:05:34.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7979000000000002e-05 2023-07-02T14:05:34.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:34.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.477842+0000", 2023-07-02T14:05:34.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000112236 2023-07-02T14:05:34.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:34.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.477954+0000", 2023-07-02T14:05:34.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0085239600000000006 2023-07-02T14:05:34.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:34.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.486478+0000", 2023-07-02T14:05:34.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077413229999999996 2023-07-02T14:05:34.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:34.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.494220+0000", 2023-07-02T14:05:34.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3022999999999998e-05 2023-07-02T14:05:34.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:34.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.494263+0000", 2023-07-02T14:05:34.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0862000000000003e-05 2023-07-02T14:05:34.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:34.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.494304+0000", 2023-07-02T14:05:34.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030948784 2023-07-02T14:05:34.483 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.484 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.484 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.484 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15733 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:34.484 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503319+0000", 2023-07-02T14:05:34.484 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.965754329999999, 2023-07-02T14:05:34.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036594227999999999, 2023-07-02T14:05:34.485 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:34.485 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:34.485 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:34.485 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:34.485 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:34.485 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15733 2023-07-02T14:05:34.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.485 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:34.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:34.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503319+0000", 2023-07-02T14:05:34.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:34.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:34.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503319+0000", 2023-07-02T14:05:34.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:34.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:34.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503319+0000", 2023-07-02T14:05:34.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0300000000000004e-06 2023-07-02T14:05:34.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:34.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503323+0000", 2023-07-02T14:05:34.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1829e-05 2023-07-02T14:05:34.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:34.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503334+0000", 2023-07-02T14:05:34.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.511e-06 2023-07-02T14:05:34.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:34.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503341+0000", 2023-07-02T14:05:34.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7775000000000005e-05 2023-07-02T14:05:34.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503439+0000", 2023-07-02T14:05:34.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6311999999999998e-05 2023-07-02T14:05:34.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503505+0000", 2023-07-02T14:05:34.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057891940000000001 2023-07-02T14:05:34.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509294+0000", 2023-07-02T14:05:34.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0247999999999997e-05 2023-07-02T14:05:34.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509345+0000", 2023-07-02T14:05:34.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018957756999999999 2023-07-02T14:05:34.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.528302+0000", 2023-07-02T14:05:34.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5699000000000006e-05 2023-07-02T14:05:34.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:34.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.528398+0000", 2023-07-02T14:05:34.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00050004900000000002 2023-07-02T14:05:34.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:34.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.528898+0000", 2023-07-02T14:05:34.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023774949999999999 2023-07-02T14:05:34.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:34.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531276+0000", 2023-07-02T14:05:34.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084207899999999992 2023-07-02T14:05:34.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:34.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539696+0000", 2023-07-02T14:05:34.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8023000000000003e-05 2023-07-02T14:05:34.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:34.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539754+0000", 2023-07-02T14:05:34.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000159092 2023-07-02T14:05:34.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:34.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539913+0000", 2023-07-02T14:05:34.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.036594227999999999 2023-07-02T14:05:34.496 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.497 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.497 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.497 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15734 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:34.497 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503349+0000", 2023-07-02T14:05:34.497 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.965724594999999, 2023-07-02T14:05:34.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041221990999999999, 2023-07-02T14:05:34.498 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:34.498 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:34.498 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:34.498 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:34.498 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:34.498 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15734 2023-07-02T14:05:34.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.499 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:34.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:34.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503349+0000", 2023-07-02T14:05:34.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:34.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:34.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503349+0000", 2023-07-02T14:05:34.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:34.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:34.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503348+0000", 2023-07-02T14:05:34.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6969999999999999e-06 2023-07-02T14:05:34.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:34.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503352+0000", 2023-07-02T14:05:34.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4900000000000003e-06 2023-07-02T14:05:34.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:34.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503360+0000", 2023-07-02T14:05:34.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3630000000000004e-06 2023-07-02T14:05:34.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:34.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503364+0000", 2023-07-02T14:05:34.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017805100000000001 2023-07-02T14:05:34.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503542+0000", 2023-07-02T14:05:34.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4289999999999994e-05 2023-07-02T14:05:34.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503626+0000", 2023-07-02T14:05:34.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057627800000000003 2023-07-02T14:05:34.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509389+0000", 2023-07-02T14:05:34.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3256000000000004e-05 2023-07-02T14:05:34.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509472+0000", 2023-07-02T14:05:34.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021137590000000001 2023-07-02T14:05:34.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530610+0000", 2023-07-02T14:05:34.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7247e-05 2023-07-02T14:05:34.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530697+0000", 2023-07-02T14:05:34.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0092678020000000003 2023-07-02T14:05:34.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539965+0000", 2023-07-02T14:05:34.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8950999999999996e-05 2023-07-02T14:05:34.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:34.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.540054+0000", 2023-07-02T14:05:34.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00045742200000000003 2023-07-02T14:05:34.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:34.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.540511+0000", 2023-07-02T14:05:34.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003197653 2023-07-02T14:05:34.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:34.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.543709+0000", 2023-07-02T14:05:34.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00065262499999999999 2023-07-02T14:05:34.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:34.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.544362+0000", 2023-07-02T14:05:34.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9426000000000003e-05 2023-07-02T14:05:34.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:34.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.544421+0000", 2023-07-02T14:05:34.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014981099999999999 2023-07-02T14:05:34.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:34.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.544571+0000", 2023-07-02T14:05:34.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041221990999999999 2023-07-02T14:05:34.511 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.511 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.511 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.512 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15735 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:34.512 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503369+0000", 2023-07-02T14:05:34.512 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.965704970000001, 2023-07-02T14:05:34.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.049400686999999999, 2023-07-02T14:05:34.512 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:34.512 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:34.512 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:34.513 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:34.513 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:34.513 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15735 2023-07-02T14:05:34.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.513 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:34.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:34.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503369+0000", 2023-07-02T14:05:34.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:34.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:34.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503369+0000", 2023-07-02T14:05:34.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:34.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:34.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503368+0000", 2023-07-02T14:05:34.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3289999999999998e-06 2023-07-02T14:05:34.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:34.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503372+0000", 2023-07-02T14:05:34.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4719999999999998e-06 2023-07-02T14:05:34.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:34.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503379+0000", 2023-07-02T14:05:34.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2289999999999997e-06 2023-07-02T14:05:34.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:34.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503383+0000", 2023-07-02T14:05:34.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00030972399999999999 2023-07-02T14:05:34.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503693+0000", 2023-07-02T14:05:34.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000120279 2023-07-02T14:05:34.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503813+0000", 2023-07-02T14:05:34.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057248059999999998 2023-07-02T14:05:34.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509538+0000", 2023-07-02T14:05:34.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011721499999999999 2023-07-02T14:05:34.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509655+0000", 2023-07-02T14:05:34.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021108756999999999 2023-07-02T14:05:34.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530764+0000", 2023-07-02T14:05:34.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5428999999999997e-05 2023-07-02T14:05:34.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530859+0000", 2023-07-02T14:05:34.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010890082000000001 2023-07-02T14:05:34.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.541750+0000", 2023-07-02T14:05:34.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5916999999999997e-05 2023-07-02T14:05:34.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.541835+0000", 2023-07-02T14:05:34.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002791226 2023-07-02T14:05:34.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.544627+0000", 2023-07-02T14:05:34.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2878000000000002e-05 2023-07-02T14:05:34.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:34.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.544710+0000", 2023-07-02T14:05:34.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00044054400000000002 2023-07-02T14:05:34.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:34.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.545150+0000", 2023-07-02T14:05:34.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001421484 2023-07-02T14:05:34.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:34.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546572+0000", 2023-07-02T14:05:34.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060059900000000001 2023-07-02T14:05:34.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:34.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.552578+0000", 2023-07-02T14:05:34.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7685e-05 2023-07-02T14:05:34.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:34.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.552645+0000", 2023-07-02T14:05:34.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012397999999999999 2023-07-02T14:05:34.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:34.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.552769+0000", 2023-07-02T14:05:34.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.049400686999999999 2023-07-02T14:05:34.527 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.527 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.527 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.527 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15736 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:34.528 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503388+0000", 2023-07-02T14:05:34.528 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.965685747, 2023-07-02T14:05:34.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.057425610000000002, 2023-07-02T14:05:34.528 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:34.528 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:34.528 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:34.528 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:34.528 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:34.529 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15736 2023-07-02T14:05:34.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.529 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:34.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:34.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503388+0000", 2023-07-02T14:05:34.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:34.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:34.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503388+0000", 2023-07-02T14:05:34.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:34.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:34.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503387+0000", 2023-07-02T14:05:34.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.382e-06 2023-07-02T14:05:34.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:34.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503391+0000", 2023-07-02T14:05:34.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7790000000000001e-06 2023-07-02T14:05:34.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:34.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503397+0000", 2023-07-02T14:05:34.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3110000000000001e-06 2023-07-02T14:05:34.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:34.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503400+0000", 2023-07-02T14:05:34.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00045478499999999998 2023-07-02T14:05:34.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503855+0000", 2023-07-02T14:05:34.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8894999999999995e-05 2023-07-02T14:05:34.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503944+0000", 2023-07-02T14:05:34.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057727849999999999 2023-07-02T14:05:34.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509716+0000", 2023-07-02T14:05:34.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.593e-05 2023-07-02T14:05:34.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509792+0000", 2023-07-02T14:05:34.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021127659 2023-07-02T14:05:34.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530920+0000", 2023-07-02T14:05:34.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0810999999999997e-05 2023-07-02T14:05:34.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530991+0000", 2023-07-02T14:05:34.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010903886999999999 2023-07-02T14:05:34.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.541895+0000", 2023-07-02T14:05:34.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4783000000000005e-05 2023-07-02T14:05:34.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.541979+0000", 2023-07-02T14:05:34.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041761189999999998 2023-07-02T14:05:34.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546155+0000", 2023-07-02T14:05:34.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2211000000000003e-05 2023-07-02T14:05:34.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546238+0000", 2023-07-02T14:05:34.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006603589 2023-07-02T14:05:34.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.552841+0000", 2023-07-02T14:05:34.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000108977 2023-07-02T14:05:34.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:34.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.552950+0000", 2023-07-02T14:05:34.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000543126 2023-07-02T14:05:34.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:34.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.553493+0000", 2023-07-02T14:05:34.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030365330000000001 2023-07-02T14:05:34.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:34.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.556530+0000", 2023-07-02T14:05:34.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041768609999999996 2023-07-02T14:05:34.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:34.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.560707+0000", 2023-07-02T14:05:34.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6613999999999999e-05 2023-07-02T14:05:34.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:34.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.560743+0000", 2023-07-02T14:05:34.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0005999999999993e-05 2023-07-02T14:05:34.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:34.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.560813+0000", 2023-07-02T14:05:34.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.057425610000000002 2023-07-02T14:05:34.544 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.545 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.545 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.545 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15737 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:34.545 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503667+0000", 2023-07-02T14:05:34.545 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.965406881, 2023-07-02T14:05:34.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.067804241000000001, 2023-07-02T14:05:34.546 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:34.546 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:34.546 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:34.546 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:34.546 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:34.546 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15737 2023-07-02T14:05:34.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.546 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:34.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:34.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503667+0000", 2023-07-02T14:05:34.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:34.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:34.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503667+0000", 2023-07-02T14:05:34.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:34.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:34.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503666+0000", 2023-07-02T14:05:34.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6499999999999999e-06 2023-07-02T14:05:34.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:34.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503672+0000", 2023-07-02T14:05:34.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9850000000000001e-06 2023-07-02T14:05:34.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:34.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503682+0000", 2023-07-02T14:05:34.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3580000000000002e-06 2023-07-02T14:05:34.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:34.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503688+0000", 2023-07-02T14:05:34.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00030985000000000002 2023-07-02T14:05:34.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503998+0000", 2023-07-02T14:05:34.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012238099999999999 2023-07-02T14:05:34.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.504120+0000", 2023-07-02T14:05:34.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057408620000000002 2023-07-02T14:05:34.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509861+0000", 2023-07-02T14:05:34.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2069000000000003e-05 2023-07-02T14:05:34.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509943+0000", 2023-07-02T14:05:34.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021112245000000002 2023-07-02T14:05:34.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531055+0000", 2023-07-02T14:05:34.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4781e-05 2023-07-02T14:05:34.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531140+0000", 2023-07-02T14:05:34.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010899282 2023-07-02T14:05:34.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.542039+0000", 2023-07-02T14:05:34.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3856999999999994e-05 2023-07-02T14:05:34.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.542123+0000", 2023-07-02T14:05:34.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041575969999999999 2023-07-02T14:05:34.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546281+0000", 2023-07-02T14:05:34.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8136000000000002e-05 2023-07-02T14:05:34.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546319+0000", 2023-07-02T14:05:34.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091989649999999999 2023-07-02T14:05:34.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.555518+0000", 2023-07-02T14:05:34.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3294000000000005e-05 2023-07-02T14:05:34.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.555611+0000", 2023-07-02T14:05:34.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052494760000000003 2023-07-02T14:05:34.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.560860+0000", 2023-07-02T14:05:34.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8725000000000002e-05 2023-07-02T14:05:34.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:34.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.560949+0000", 2023-07-02T14:05:34.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00046914100000000001 2023-07-02T14:05:34.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:34.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.561418+0000", 2023-07-02T14:05:34.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006959911 2023-07-02T14:05:34.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:34.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.568378+0000", 2023-07-02T14:05:34.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029392400000000001 2023-07-02T14:05:34.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:34.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.571317+0000", 2023-07-02T14:05:34.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8885999999999998e-05 2023-07-02T14:05:34.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:34.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.571376+0000", 2023-07-02T14:05:34.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4517000000000003e-05 2023-07-02T14:05:34.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:34.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.571471+0000", 2023-07-02T14:05:34.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.067804241000000001 2023-07-02T14:05:34.563 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.563 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.563 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.563 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15738 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:34.564 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503693+0000", 2023-07-02T14:05:34.564 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.96538035, 2023-07-02T14:05:34.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.080315304000000004, 2023-07-02T14:05:34.564 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:34.564 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:34.564 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:34.564 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:34.564 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:34.565 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15738 2023-07-02T14:05:34.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.565 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:34.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:34.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503693+0000", 2023-07-02T14:05:34.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:34.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:34.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503693+0000", 2023-07-02T14:05:34.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:34.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:34.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503693+0000", 2023-07-02T14:05:34.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3620000000000002e-06 2023-07-02T14:05:34.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:34.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503696+0000", 2023-07-02T14:05:34.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2850000000000002e-06 2023-07-02T14:05:34.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:34.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503701+0000", 2023-07-02T14:05:34.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.938e-06 2023-07-02T14:05:34.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:34.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503706+0000", 2023-07-02T14:05:34.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000447105 2023-07-02T14:05:34.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.504153+0000", 2023-07-02T14:05:34.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011312 2023-07-02T14:05:34.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.504267+0000", 2023-07-02T14:05:34.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057292469999999998 2023-07-02T14:05:34.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.509996+0000", 2023-07-02T14:05:34.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7124000000000002e-05 2023-07-02T14:05:34.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.510073+0000", 2023-07-02T14:05:34.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021118604999999999 2023-07-02T14:05:34.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531192+0000", 2023-07-02T14:05:34.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4815999999999997e-05 2023-07-02T14:05:34.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531266+0000", 2023-07-02T14:05:34.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010915915 2023-07-02T14:05:34.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.542182+0000", 2023-07-02T14:05:34.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5304999999999998e-05 2023-07-02T14:05:34.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.542258+0000", 2023-07-02T14:05:34.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041042539999999999 2023-07-02T14:05:34.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546362+0000", 2023-07-02T14:05:34.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0340999999999995e-05 2023-07-02T14:05:34.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546442+0000", 2023-07-02T14:05:34.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0092239639999999994 2023-07-02T14:05:34.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.555666+0000", 2023-07-02T14:05:34.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0080999999999995e-05 2023-07-02T14:05:34.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.555736+0000", 2023-07-02T14:05:34.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069964290000000002 2023-07-02T14:05:34.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562733+0000", 2023-07-02T14:05:34.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7447000000000005e-05 2023-07-02T14:05:34.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562820+0000", 2023-07-02T14:05:34.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086771179999999993 2023-07-02T14:05:34.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.571497+0000", 2023-07-02T14:05:34.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7438000000000002e-05 2023-07-02T14:05:34.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:34.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.571545+0000", 2023-07-02T14:05:34.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000252651 2023-07-02T14:05:34.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:34.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.571797+0000", 2023-07-02T14:05:34.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001648369 2023-07-02T14:05:34.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:34.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.573446+0000", 2023-07-02T14:05:34.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010442987000000001 2023-07-02T14:05:34.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:34.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.583889+0000", 2023-07-02T14:05:34.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9243000000000002e-05 2023-07-02T14:05:34.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:34.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.583948+0000", 2023-07-02T14:05:34.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0590000000000001e-05 2023-07-02T14:05:34.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:34.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.584008+0000", 2023-07-02T14:05:34.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.080315304000000004 2023-07-02T14:05:34.583 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.583 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.583 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.583 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15739 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:34.583 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.503710+0000", 2023-07-02T14:05:34.584 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.965363413, 2023-07-02T14:05:34.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.084428051000000004, 2023-07-02T14:05:34.584 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:34.584 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:34.584 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:34.584 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:34.584 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:34.584 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15739 2023-07-02T14:05:34.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.585 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:34.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:34.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503710+0000", 2023-07-02T14:05:34.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:34.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:34.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503710+0000", 2023-07-02T14:05:34.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:34.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:34.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503710+0000", 2023-07-02T14:05:34.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9919999999999999e-06 2023-07-02T14:05:34.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:34.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503713+0000", 2023-07-02T14:05:34.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8439999999999997e-06 2023-07-02T14:05:34.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:34.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503718+0000", 2023-07-02T14:05:34.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.58e-06 2023-07-02T14:05:34.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:34.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.503721+0000", 2023-07-02T14:05:34.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00059269800000000001 2023-07-02T14:05:34.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.504314+0000", 2023-07-02T14:05:34.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000121972 2023-07-02T14:05:34.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.504436+0000", 2023-07-02T14:05:34.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056719500000000003 2023-07-02T14:05:34.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.510108+0000", 2023-07-02T14:05:34.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8212000000000003e-05 2023-07-02T14:05:34.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.510186+0000", 2023-07-02T14:05:34.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021121849000000002 2023-07-02T14:05:34.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531308+0000", 2023-07-02T14:05:34.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2182999999999997e-05 2023-07-02T14:05:34.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.531380+0000", 2023-07-02T14:05:34.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010927773999999999 2023-07-02T14:05:34.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.542308+0000", 2023-07-02T14:05:34.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7233999999999999e-05 2023-07-02T14:05:34.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.542385+0000", 2023-07-02T14:05:34.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041062779999999997 2023-07-02T14:05:34.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546491+0000", 2023-07-02T14:05:34.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7290000000000001e-05 2023-07-02T14:05:34.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.546559+0000", 2023-07-02T14:05:34.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0092123529999999995 2023-07-02T14:05:34.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.555771+0000", 2023-07-02T14:05:34.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2665999999999998e-05 2023-07-02T14:05:34.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.555844+0000", 2023-07-02T14:05:34.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070310900000000003 2023-07-02T14:05:34.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562875+0000", 2023-07-02T14:05:34.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6711000000000001e-05 2023-07-02T14:05:34.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.562921+0000", 2023-07-02T14:05:34.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010083190000000001 2023-07-02T14:05:34.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.573005+0000", 2023-07-02T14:05:34.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7387999999999997e-05 2023-07-02T14:05:34.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.573092+0000", 2023-07-02T14:05:34.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010965232 2023-07-02T14:05:34.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.584057+0000", 2023-07-02T14:05:34.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8296999999999998e-05 2023-07-02T14:05:34.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:34.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.584156+0000", 2023-07-02T14:05:34.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00048002599999999999 2023-07-02T14:05:34.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:34.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.584636+0000", 2023-07-02T14:05:34.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024421500000000001 2023-07-02T14:05:34.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:34.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.587078+0000", 2023-07-02T14:05:34.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000929487 2023-07-02T14:05:34.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:34.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.588007+0000", 2023-07-02T14:05:34.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0000999999999996e-05 2023-07-02T14:05:34.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:34.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.588077+0000", 2023-07-02T14:05:34.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0915999999999997e-05 2023-07-02T14:05:34.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:34.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.588138+0000", 2023-07-02T14:05:34.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.084428051000000004 2023-07-02T14:05:34.604 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.604 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.604 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.604 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15740 2.0 2:08f47745:::rbd_data.125de1449605.00000000000000eb:head [write 49152~4079616] snapc 26=[26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:34.604 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:19.507724+0000", 2023-07-02T14:05:34.605 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.961349716999999, 2023-07-02T14:05:34.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031745934000000003, 2023-07-02T14:05:34.605 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:34.605 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:34.605 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:34.605 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:34.605 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:34.606 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15740 2023-07-02T14:05:34.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.606 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:34.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:34.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.507724+0000", 2023-07-02T14:05:34.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:34.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:34.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.507724+0000", 2023-07-02T14:05:34.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:34.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:34.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.507722+0000", 2023-07-02T14:05:34.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079849010000000008 2023-07-02T14:05:34.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:34.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515707+0000", 2023-07-02T14:05:34.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4861999999999999e-05 2023-07-02T14:05:34.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:34.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515722+0000", 2023-07-02T14:05:34.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7649999999999996e-06 2023-07-02T14:05:34.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:34.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515729+0000", 2023-07-02T14:05:34.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3593000000000001e-05 2023-07-02T14:05:34.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515772+0000", 2023-07-02T14:05:34.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011619499999999999 2023-07-02T14:05:34.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:34.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.515888+0000", 2023-07-02T14:05:34.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020645599999999999 2023-07-02T14:05:34.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:34.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.516095+0000", 2023-07-02T14:05:34.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014613513999999999 2023-07-02T14:05:34.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:34.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.530708+0000", 2023-07-02T14:05:34.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086383950000000001 2023-07-02T14:05:34.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:34.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539347+0000", 2023-07-02T14:05:34.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7710000000000001e-05 2023-07-02T14:05:34.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:34.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539414+0000", 2023-07-02T14:05:34.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5368999999999998e-05 2023-07-02T14:05:34.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:34.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:19.539470+0000", 2023-07-02T14:05:34.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031745934000000003 2023-07-02T14:05:34.614 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.614 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.615 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.615 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15842 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:34.615 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:20.082411+0000", 2023-07-02T14:05:34.615 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.386662294000001, 2023-07-02T14:05:34.615 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032427978000000003, 2023-07-02T14:05:34.615 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:34.616 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:34.616 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:34.616 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:34.616 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:34.616 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15842 2023-07-02T14:05:34.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.616 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:34.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:34.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082411+0000", 2023-07-02T14:05:34.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:34.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:34.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082411+0000", 2023-07-02T14:05:34.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:34.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:34.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082411+0000", 2023-07-02T14:05:34.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6750000000000001e-06 2023-07-02T14:05:34.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:34.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082415+0000", 2023-07-02T14:05:34.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1348e-05 2023-07-02T14:05:34.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:34.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082427+0000", 2023-07-02T14:05:34.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5499999999999996e-06 2023-07-02T14:05:34.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:34.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082431+0000", 2023-07-02T14:05:34.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014813339999999999 2023-07-02T14:05:34.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.083913+0000", 2023-07-02T14:05:34.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9057000000000001e-05 2023-07-02T14:05:34.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.083962+0000", 2023-07-02T14:05:34.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003500862 2023-07-02T14:05:34.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.087463+0000", 2023-07-02T14:05:34.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5519000000000003e-05 2023-07-02T14:05:34.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.087498+0000", 2023-07-02T14:05:34.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048198030000000001 2023-07-02T14:05:34.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.092318+0000", 2023-07-02T14:05:34.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6455000000000002e-05 2023-07-02T14:05:34.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.092354+0000", 2023-07-02T14:05:34.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069876130000000002 2023-07-02T14:05:34.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.625 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.099342+0000", 2023-07-02T14:05:34.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4948000000000003e-05 2023-07-02T14:05:34.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.625 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.099397+0000", 2023-07-02T14:05:34.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043643629999999996 2023-07-02T14:05:34.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.626 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.103761+0000", 2023-07-02T14:05:34.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5099999999999995e-05 2023-07-02T14:05:34.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.103846+0000", 2023-07-02T14:05:34.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047369700000000001 2023-07-02T14:05:34.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.108583+0000", 2023-07-02T14:05:34.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8518999999999998e-05 2023-07-02T14:05:34.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:34.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.108662+0000", 2023-07-02T14:05:34.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00044123099999999999 2023-07-02T14:05:34.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:34.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.109103+0000", 2023-07-02T14:05:34.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055459309999999996 2023-07-02T14:05:34.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:34.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.114649+0000", 2023-07-02T14:05:34.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9299999999999999e-05 2023-07-02T14:05:34.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:34.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.114698+0000", 2023-07-02T14:05:34.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8134999999999998e-05 2023-07-02T14:05:34.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:34.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.114756+0000", 2023-07-02T14:05:34.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2806000000000006e-05 2023-07-02T14:05:34.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:34.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.114839+0000", 2023-07-02T14:05:34.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032427978000000003 2023-07-02T14:05:34.632 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.632 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.632 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.632 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15843 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:34.632 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:20.082443+0000", 2023-07-02T14:05:34.632 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.386630526999999, 2023-07-02T14:05:34.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.038901416000000001, 2023-07-02T14:05:34.633 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:34.633 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:34.633 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:34.633 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:34.633 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:34.633 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15843 2023-07-02T14:05:34.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.634 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:34.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:34.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082443+0000", 2023-07-02T14:05:34.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:34.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:34.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082443+0000", 2023-07-02T14:05:34.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:34.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:34.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082442+0000", 2023-07-02T14:05:34.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2880000000000003e-06 2023-07-02T14:05:34.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:34.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082447+0000", 2023-07-02T14:05:34.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7710000000000001e-06 2023-07-02T14:05:34.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:34.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082453+0000", 2023-07-02T14:05:34.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.164e-06 2023-07-02T14:05:34.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:34.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082459+0000", 2023-07-02T14:05:34.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0015285089999999999 2023-07-02T14:05:34.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.083987+0000", 2023-07-02T14:05:34.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9825999999999999e-05 2023-07-02T14:05:34.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.084047+0000", 2023-07-02T14:05:34.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003478138 2023-07-02T14:05:34.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.087525+0000", 2023-07-02T14:05:34.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:05:34.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.087561+0000", 2023-07-02T14:05:34.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048258870000000001 2023-07-02T14:05:34.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.092387+0000", 2023-07-02T14:05:34.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6260000000000002e-05 2023-07-02T14:05:34.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.092423+0000", 2023-07-02T14:05:34.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070145980000000004 2023-07-02T14:05:34.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.099438+0000", 2023-07-02T14:05:34.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5892999999999997e-05 2023-07-02T14:05:34.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.099474+0000", 2023-07-02T14:05:34.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044162649999999999 2023-07-02T14:05:34.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.103890+0000", 2023-07-02T14:05:34.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3977000000000001e-05 2023-07-02T14:05:34.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.103954+0000", 2023-07-02T14:05:34.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0063264819999999996 2023-07-02T14:05:34.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.110281+0000", 2023-07-02T14:05:34.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7503999999999998e-05 2023-07-02T14:05:34.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.110328+0000", 2023-07-02T14:05:34.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045860379999999997 2023-07-02T14:05:34.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.114914+0000", 2023-07-02T14:05:34.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0939999999999994e-05 2023-07-02T14:05:34.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:34.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.114995+0000", 2023-07-02T14:05:34.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00046429400000000002 2023-07-02T14:05:34.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:34.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.115459+0000", 2023-07-02T14:05:34.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001388771 2023-07-02T14:05:34.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:34.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.116848+0000", 2023-07-02T14:05:34.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044324719999999998 2023-07-02T14:05:34.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:34.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.121281+0000", 2023-07-02T14:05:34.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8530999999999998e-05 2023-07-02T14:05:34.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:34.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.121309+0000", 2023-07-02T14:05:34.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5237999999999997e-05 2023-07-02T14:05:34.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:34.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.121344+0000", 2023-07-02T14:05:34.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.038901416000000001 2023-07-02T14:05:34.650 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.650 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.650 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.651 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15844 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:34.651 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:20.082463+0000", 2023-07-02T14:05:34.651 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.38661024, 2023-07-02T14:05:34.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.043857647999999999, 2023-07-02T14:05:34.651 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:34.651 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:34.651 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:34.651 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:34.652 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:34.652 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15844 2023-07-02T14:05:34.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.652 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:34.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:34.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082463+0000", 2023-07-02T14:05:34.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:34.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:34.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082463+0000", 2023-07-02T14:05:34.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:34.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:34.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082463+0000", 2023-07-02T14:05:34.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.472e-06 2023-07-02T14:05:34.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:34.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082466+0000", 2023-07-02T14:05:34.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5299999999999999e-06 2023-07-02T14:05:34.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:34.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082474+0000", 2023-07-02T14:05:34.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.339e-06 2023-07-02T14:05:34.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:34.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082479+0000", 2023-07-02T14:05:34.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0015900230000000001 2023-07-02T14:05:34.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.084069+0000", 2023-07-02T14:05:34.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9060999999999998e-05 2023-07-02T14:05:34.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.084108+0000", 2023-07-02T14:05:34.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0034805980000000001 2023-07-02T14:05:34.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.087589+0000", 2023-07-02T14:05:34.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7870000000000001e-05 2023-07-02T14:05:34.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.087637+0000", 2023-07-02T14:05:34.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048162689999999998 2023-07-02T14:05:34.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.092453+0000", 2023-07-02T14:05:34.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7666000000000002e-05 2023-07-02T14:05:34.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.092491+0000", 2023-07-02T14:05:34.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070150239999999999 2023-07-02T14:05:34.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.099506+0000", 2023-07-02T14:05:34.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9339999999999999e-05 2023-07-02T14:05:34.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.099545+0000", 2023-07-02T14:05:34.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044636540000000001 2023-07-02T14:05:34.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.104009+0000", 2023-07-02T14:05:34.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9098999999999995e-05 2023-07-02T14:05:34.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.104098+0000", 2023-07-02T14:05:34.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006275821 2023-07-02T14:05:34.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.110374+0000", 2023-07-02T14:05:34.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2263999999999997e-05 2023-07-02T14:05:34.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.110456+0000", 2023-07-02T14:05:34.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060119539999999999 2023-07-02T14:05:34.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.116468+0000", 2023-07-02T14:05:34.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7787999999999995e-05 2023-07-02T14:05:34.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.116546+0000", 2023-07-02T14:05:34.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048420779999999997 2023-07-02T14:05:34.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.121388+0000", 2023-07-02T14:05:34.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2459000000000005e-05 2023-07-02T14:05:34.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:34.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.121470+0000", 2023-07-02T14:05:34.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000407697 2023-07-02T14:05:34.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:34.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.121878+0000", 2023-07-02T14:05:34.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0016891339999999999 2023-07-02T14:05:34.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:34.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.123567+0000", 2023-07-02T14:05:34.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002647983 2023-07-02T14:05:34.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:34.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.126215+0000", 2023-07-02T14:05:34.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1282000000000001e-05 2023-07-02T14:05:34.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:34.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.126266+0000", 2023-07-02T14:05:34.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4698999999999999e-05 2023-07-02T14:05:34.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:34.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.126321+0000", 2023-07-02T14:05:34.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.043857647999999999 2023-07-02T14:05:34.670 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.670 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.670 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.670 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15845 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e146)", 2023-07-02T14:05:34.671 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:20.082484+0000", 2023-07-02T14:05:34.671 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.386589773000001, 2023-07-02T14:05:34.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.048335177999999999, 2023-07-02T14:05:34.671 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:34.671 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:34.671 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:34.671 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:34.672 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:34.672 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15845 2023-07-02T14:05:34.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.672 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:34.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:34.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082484+0000", 2023-07-02T14:05:34.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:34.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:34.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082484+0000", 2023-07-02T14:05:34.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:34.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:34.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082483+0000", 2023-07-02T14:05:34.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.58e-06 2023-07-02T14:05:34.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:34.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082487+0000", 2023-07-02T14:05:34.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9920000000000004e-06 2023-07-02T14:05:34.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:34.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082493+0000", 2023-07-02T14:05:34.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3239999999999996e-06 2023-07-02T14:05:34.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:34.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.082497+0000", 2023-07-02T14:05:34.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001622914 2023-07-02T14:05:34.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.084120+0000", 2023-07-02T14:05:34.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9756000000000003e-05 2023-07-02T14:05:34.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.084170+0000", 2023-07-02T14:05:34.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0034866229999999999 2023-07-02T14:05:34.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.087656+0000", 2023-07-02T14:05:34.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8263999999999998e-05 2023-07-02T14:05:34.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.087695+0000", 2023-07-02T14:05:34.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048144320000000004 2023-07-02T14:05:34.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.092509+0000", 2023-07-02T14:05:34.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7141999999999997e-05 2023-07-02T14:05:34.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.092556+0000", 2023-07-02T14:05:34.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070032920000000004 2023-07-02T14:05:34.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.099560+0000", 2023-07-02T14:05:34.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1183000000000002e-05 2023-07-02T14:05:34.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.099601+0000", 2023-07-02T14:05:34.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004518897 2023-07-02T14:05:34.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.104120+0000", 2023-07-02T14:05:34.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9988999999999999e-05 2023-07-02T14:05:34.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.104170+0000", 2023-07-02T14:05:34.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0063198350000000002 2023-07-02T14:05:34.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.110490+0000", 2023-07-02T14:05:34.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9635999999999998e-05 2023-07-02T14:05:34.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.110529+0000", 2023-07-02T14:05:34.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060621989999999999 2023-07-02T14:05:34.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.116591+0000", 2023-07-02T14:05:34.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1582999999999996e-05 2023-07-02T14:05:34.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.116653+0000", 2023-07-02T14:05:34.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064548440000000004 2023-07-02T14:05:34.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.123108+0000", 2023-07-02T14:05:34.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6099000000000005e-05 2023-07-02T14:05:34.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.123174+0000", 2023-07-02T14:05:34.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031684999999999999 2023-07-02T14:05:34.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.126342+0000", 2023-07-02T14:05:34.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1156999999999999e-05 2023-07-02T14:05:34.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:34.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.126384+0000", 2023-07-02T14:05:34.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021451400000000001 2023-07-02T14:05:34.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:34.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.126598+0000", 2023-07-02T14:05:34.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001432106 2023-07-02T14:05:34.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:34.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.128030+0000", 2023-07-02T14:05:34.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025489369999999998 2023-07-02T14:05:34.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:34.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.130579+0000", 2023-07-02T14:05:34.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015677200000000001 2023-07-02T14:05:34.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:34.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.130736+0000", 2023-07-02T14:05:34.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3071000000000005e-05 2023-07-02T14:05:34.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:34.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:20.130819+0000", 2023-07-02T14:05:34.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.048335177999999999 2023-07-02T14:05:34.691 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.691 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.691 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.691 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6249.0:336 2.5 2:a8e921ad:::rbd_data.125de1449605.00000000000000d7:27 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e150)", 2023-07-02T14:05:34.692 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:24.965943+0000", 2023-07-02T14:05:34.692 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.5031308009999993, 2023-07-02T14:05:34.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029886167000000002, 2023-07-02T14:05:34.692 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:34.692 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:34.692 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:34.692 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6249", 2023-07-02T14:05:34.692 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1954902621", 2023-07-02T14:05:34.693 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 336 2023-07-02T14:05:34.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.693 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:34.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:34.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.965943+0000", 2023-07-02T14:05:34.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:34.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:34.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.965943+0000", 2023-07-02T14:05:34.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:34.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:34.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.965942+0000", 2023-07-02T14:05:34.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4329999999999998e-06 2023-07-02T14:05:34.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:34.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.965945+0000", 2023-07-02T14:05:34.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0049999999999998e-06 2023-07-02T14:05:34.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:34.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.965952+0000", 2023-07-02T14:05:34.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0080000000000004e-06 2023-07-02T14:05:34.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:34.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.965956+0000", 2023-07-02T14:05:34.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02651914 2023-07-02T14:05:34.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.992475+0000", 2023-07-02T14:05:34.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000111976 2023-07-02T14:05:34.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:34.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.992587+0000", 2023-07-02T14:05:34.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032419469999999998 2023-07-02T14:05:34.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:34.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:24.995829+0000", 2023-07-02T14:05:34.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029886167000000002 2023-07-02T14:05:34.699 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.699 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.699 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.699 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6249.0:341 2.5 2:ae55f2e2:::rbd_data.125de1449605.00000000000000e1:27 [sparse-read 0~4063232 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e150)", 2023-07-02T14:05:34.699 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:25.015119+0000", 2023-07-02T14:05:34.699 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.4539542040000004, 2023-07-02T14:05:34.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028784417, 2023-07-02T14:05:34.700 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:34.700 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:34.700 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:34.700 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6249", 2023-07-02T14:05:34.700 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1954902621", 2023-07-02T14:05:34.700 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 341 2023-07-02T14:05:34.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.701 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:34.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:34.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.015119+0000", 2023-07-02T14:05:34.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:34.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:34.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.015119+0000", 2023-07-02T14:05:34.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:34.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:34.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.015119+0000", 2023-07-02T14:05:34.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6599999999999999e-06 2023-07-02T14:05:34.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:34.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.015122+0000", 2023-07-02T14:05:34.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2506e-05 2023-07-02T14:05:34.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:34.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.015134+0000", 2023-07-02T14:05:34.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5199999999999997e-06 2023-07-02T14:05:34.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:34.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.015140+0000", 2023-07-02T14:05:34.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025739499999999998 2023-07-02T14:05:34.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.040879+0000", 2023-07-02T14:05:34.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4022000000000004e-05 2023-07-02T14:05:34.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:34.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.040953+0000", 2023-07-02T14:05:34.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002950659 2023-07-02T14:05:34.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:34.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.043904+0000", 2023-07-02T14:05:34.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028784417 2023-07-02T14:05:34.707 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.707 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.707 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.707 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6249.0:348 2.5 2:b3c82204:::rbd_data.125de1449605.00000000000000de:27 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e150)", 2023-07-02T14:05:34.707 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:25.017109+0000", 2023-07-02T14:05:34.708 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.4519645580000002, 2023-07-02T14:05:34.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031502175, 2023-07-02T14:05:34.708 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:34.708 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:34.708 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:34.708 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6249", 2023-07-02T14:05:34.708 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1954902621", 2023-07-02T14:05:34.709 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 348 2023-07-02T14:05:34.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.709 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:34.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:34.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.017109+0000", 2023-07-02T14:05:34.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:34.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:34.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.017109+0000", 2023-07-02T14:05:34.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:34.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:34.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.017108+0000", 2023-07-02T14:05:34.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0399999999999998e-06 2023-07-02T14:05:34.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:34.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.017117+0000", 2023-07-02T14:05:34.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2214000000000001e-05 2023-07-02T14:05:34.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:34.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.017139+0000", 2023-07-02T14:05:34.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7249999999999997e-06 2023-07-02T14:05:34.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:34.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.017146+0000", 2023-07-02T14:05:34.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026777637999999999 2023-07-02T14:05:34.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.043924+0000", 2023-07-02T14:05:34.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5428999999999998e-05 2023-07-02T14:05:34.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:34.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.044010+0000", 2023-07-02T14:05:34.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046015989999999996 2023-07-02T14:05:34.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:34.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.048611+0000", 2023-07-02T14:05:34.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031502175 2023-07-02T14:05:34.715 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.715 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.715 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.715 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6249.0:349 2.5 2:b1affa7f:::rbd_data.125de1449605.00000000000000e5:27 [sparse-read 0~4063232 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e150)", 2023-07-02T14:05:34.715 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:25.020253+0000", 2023-07-02T14:05:34.716 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.448820435, 2023-07-02T14:05:34.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031754767000000003, 2023-07-02T14:05:34.716 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:34.716 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:05:34.716 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:34.716 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6249", 2023-07-02T14:05:34.716 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1954902621", 2023-07-02T14:05:34.716 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 349 2023-07-02T14:05:34.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.717 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:34.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:34.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.020253+0000", 2023-07-02T14:05:34.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:34.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:34.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.020253+0000", 2023-07-02T14:05:34.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:34.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:34.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.020252+0000", 2023-07-02T14:05:34.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7929999999999998e-06 2023-07-02T14:05:34.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:34.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.020259+0000", 2023-07-02T14:05:34.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6614000000000001e-05 2023-07-02T14:05:34.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:34.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.020276+0000", 2023-07-02T14:05:34.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:05:34.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:34.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.020284+0000", 2023-07-02T14:05:34.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028380684999999999 2023-07-02T14:05:34.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.048664+0000", 2023-07-02T14:05:34.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000187221 2023-07-02T14:05:34.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:34.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.048852+0000", 2023-07-02T14:05:34.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031562309999999998 2023-07-02T14:05:34.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:34.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:25.052008+0000", 2023-07-02T14:05:34.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031754767000000003 2023-07-02T14:05:34.723 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.723 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.723 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.723 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15964 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e151)", 2023-07-02T14:05:34.723 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:26.185626+0000", 2023-07-02T14:05:34.723 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.2834472370000007, 2023-07-02T14:05:34.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031245408999999998, 2023-07-02T14:05:34.724 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:34.724 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:34.724 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:34.724 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:34.724 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:34.724 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15964 2023-07-02T14:05:34.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.725 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:34.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:34.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.185626+0000", 2023-07-02T14:05:34.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:34.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:34.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.185626+0000", 2023-07-02T14:05:34.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:34.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:34.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.185626+0000", 2023-07-02T14:05:34.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0820000000000001e-06 2023-07-02T14:05:34.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:34.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.185628+0000", 2023-07-02T14:05:34.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.915e-06 2023-07-02T14:05:34.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:34.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.185631+0000", 2023-07-02T14:05:34.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9739999999999998e-06 2023-07-02T14:05:34.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:34.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.185634+0000", 2023-07-02T14:05:34.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018254600000000001 2023-07-02T14:05:34.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.187459+0000", 2023-07-02T14:05:34.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9926999999999999e-05 2023-07-02T14:05:34.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.187499+0000", 2023-07-02T14:05:34.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0035085139999999999 2023-07-02T14:05:34.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.191008+0000", 2023-07-02T14:05:34.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3294999999999999e-05 2023-07-02T14:05:34.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.191041+0000", 2023-07-02T14:05:34.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045127580000000004 2023-07-02T14:05:34.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.195554+0000", 2023-07-02T14:05:34.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0448000000000001e-05 2023-07-02T14:05:34.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.195584+0000", 2023-07-02T14:05:34.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055268870000000003 2023-07-02T14:05:34.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.201111+0000", 2023-07-02T14:05:34.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5851000000000002e-05 2023-07-02T14:05:34.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.201197+0000", 2023-07-02T14:05:34.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054014340000000001 2023-07-02T14:05:34.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.206599+0000", 2023-07-02T14:05:34.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8350999999999999e-05 2023-07-02T14:05:34.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.206637+0000", 2023-07-02T14:05:34.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043349679999999998 2023-07-02T14:05:34.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.210972+0000", 2023-07-02T14:05:34.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8902000000000001e-05 2023-07-02T14:05:34.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:34.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.211011+0000", 2023-07-02T14:05:34.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000234546 2023-07-02T14:05:34.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:34.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.211245+0000", 2023-07-02T14:05:34.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00336396 2023-07-02T14:05:34.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:34.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.214609+0000", 2023-07-02T14:05:34.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0021768310000000002 2023-07-02T14:05:34.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:34.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.216786+0000", 2023-07-02T14:05:34.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1900000000000003e-05 2023-07-02T14:05:34.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:34.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.216818+0000", 2023-07-02T14:05:34.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3671000000000002e-05 2023-07-02T14:05:34.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:34.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.216872+0000", 2023-07-02T14:05:34.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031245408999999998 2023-07-02T14:05:34.739 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.739 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.740 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.740 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15965 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e151)", 2023-07-02T14:05:34.740 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:26.185650+0000", 2023-07-02T14:05:34.740 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.2834237749999993, 2023-07-02T14:05:34.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039012667000000001, 2023-07-02T14:05:34.741 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:34.741 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:34.741 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:34.741 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:34.741 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:34.741 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15965 2023-07-02T14:05:34.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.741 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:34.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:34.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.185650+0000", 2023-07-02T14:05:34.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:34.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:34.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.185650+0000", 2023-07-02T14:05:34.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:34.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:34.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.185649+0000", 2023-07-02T14:05:34.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3349999999999998e-06 2023-07-02T14:05:34.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:34.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.185653+0000", 2023-07-02T14:05:34.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.252e-05 2023-07-02T14:05:34.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:34.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.185665+0000", 2023-07-02T14:05:34.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9170000000000002e-06 2023-07-02T14:05:34.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:34.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.185668+0000", 2023-07-02T14:05:34.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018611249999999999 2023-07-02T14:05:34.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.187529+0000", 2023-07-02T14:05:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8660999999999999e-05 2023-07-02T14:05:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.187598+0000", 2023-07-02T14:05:34.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0034832999999999999 2023-07-02T14:05:34.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.191081+0000", 2023-07-02T14:05:34.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7349999999999997e-05 2023-07-02T14:05:34.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.191149+0000", 2023-07-02T14:05:34.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044751349999999999 2023-07-02T14:05:34.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.195624+0000", 2023-07-02T14:05:34.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8989000000000001e-05 2023-07-02T14:05:34.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.195683+0000", 2023-07-02T14:05:34.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055595139999999998 2023-07-02T14:05:34.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.201242+0000", 2023-07-02T14:05:34.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8012999999999997e-05 2023-07-02T14:05:34.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.201280+0000", 2023-07-02T14:05:34.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054002149999999999 2023-07-02T14:05:34.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.206680+0000", 2023-07-02T14:05:34.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6339000000000003e-05 2023-07-02T14:05:34.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.206747+0000", 2023-07-02T14:05:34.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005353544 2023-07-02T14:05:34.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.212100+0000", 2023-07-02T14:05:34.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7359000000000003e-05 2023-07-02T14:05:34.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.212178+0000", 2023-07-02T14:05:34.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047343489999999997 2023-07-02T14:05:34.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.216912+0000", 2023-07-02T14:05:34.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1339000000000002e-05 2023-07-02T14:05:34.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:34.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.216993+0000", 2023-07-02T14:05:34.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00037516999999999997 2023-07-02T14:05:34.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:34.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.217368+0000", 2023-07-02T14:05:34.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014849240000000001 2023-07-02T14:05:34.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:34.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.218853+0000", 2023-07-02T14:05:34.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005735609 2023-07-02T14:05:34.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:34.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.224589+0000", 2023-07-02T14:05:34.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0202e-05 2023-07-02T14:05:34.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:34.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.224619+0000", 2023-07-02T14:05:34.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3204000000000002e-05 2023-07-02T14:05:34.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:34.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.224662+0000", 2023-07-02T14:05:34.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.039012667000000001 2023-07-02T14:05:34.758 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.758 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.758 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.758 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:15966 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e151)", 2023-07-02T14:05:34.759 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:26.185690+0000", 2023-07-02T14:05:34.759 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.2833837789999993, 2023-07-02T14:05:34.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.044656933000000003, 2023-07-02T14:05:34.759 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:34.759 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:34.759 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:34.759 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:34.760 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:34.760 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 15966 2023-07-02T14:05:34.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.760 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:34.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:34.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.185690+0000", 2023-07-02T14:05:34.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:34.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:34.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.185690+0000", 2023-07-02T14:05:34.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:34.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:34.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.185689+0000", 2023-07-02T14:05:34.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.269e-06 2023-07-02T14:05:34.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:34.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.185693+0000", 2023-07-02T14:05:34.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8589999999999998e-06 2023-07-02T14:05:34.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:34.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.185696+0000", 2023-07-02T14:05:34.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3329999999999999e-06 2023-07-02T14:05:34.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:34.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.185699+0000", 2023-07-02T14:05:34.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001923162 2023-07-02T14:05:34.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.187622+0000", 2023-07-02T14:05:34.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6918e-05 2023-07-02T14:05:34.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.187669+0000", 2023-07-02T14:05:34.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003500693 2023-07-02T14:05:34.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.191170+0000", 2023-07-02T14:05:34.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2897e-05 2023-07-02T14:05:34.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.191202+0000", 2023-07-02T14:05:34.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045023700000000003 2023-07-02T14:05:34.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.195705+0000", 2023-07-02T14:05:34.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2509999999999999e-05 2023-07-02T14:05:34.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.195737+0000", 2023-07-02T14:05:34.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055699310000000002 2023-07-02T14:05:34.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.201307+0000", 2023-07-02T14:05:34.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4431999999999996e-05 2023-07-02T14:05:34.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.201372+0000", 2023-07-02T14:05:34.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053948349999999997 2023-07-02T14:05:34.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.206767+0000", 2023-07-02T14:05:34.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2135999999999999e-05 2023-07-02T14:05:34.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.206799+0000", 2023-07-02T14:05:34.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054208060000000002 2023-07-02T14:05:34.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.212219+0000", 2023-07-02T14:05:34.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9267999999999997e-05 2023-07-02T14:05:34.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.212259+0000", 2023-07-02T14:05:34.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00598045 2023-07-02T14:05:34.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.218239+0000", 2023-07-02T14:05:34.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8990000000000002e-05 2023-07-02T14:05:34.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:34.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.218308+0000", 2023-07-02T14:05:34.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0063867359999999996 2023-07-02T14:05:34.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:34.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.224695+0000", 2023-07-02T14:05:34.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6881000000000001e-05 2023-07-02T14:05:34.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:34.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.224742+0000", 2023-07-02T14:05:34.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00023006799999999999 2023-07-02T14:05:34.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:34.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.224972+0000", 2023-07-02T14:05:34.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0016091670000000001 2023-07-02T14:05:34.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:34.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.226581+0000", 2023-07-02T14:05:34.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037026060000000002 2023-07-02T14:05:34.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:34.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.230284+0000", 2023-07-02T14:05:34.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9311000000000001e-05 2023-07-02T14:05:34.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:34.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.230313+0000", 2023-07-02T14:05:34.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3723000000000002e-05 2023-07-02T14:05:34.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:34.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:34.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:34.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:26.230347+0000", 2023-07-02T14:05:34.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.044656933000000003 2023-07-02T14:05:34.778 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.778 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.778 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.778 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:34.778 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:34.778 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:34.780 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:05:34.784 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 25=[]:{25=[25,24,23,22]} 2023-07-02T14:05:34.784 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3831950c:::rbd_directory:head# 2023-07-02T14:05:34.784 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 0=[]:{} 2023-07-02T14:05:34.784 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:393f5a02:::rbd_data.125de1449605.0000000000000133:1e# 2023-07-02T14:05:34.785 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:393f5a02:::rbd_data.125de1449605.0000000000000133:head# 2023-07-02T14:05:34.785 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 1e=[]:{1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9]} 2023-07-02T14:05:34.785 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3a23a925:::rbd_object_map.125de1449605.0000000000000026:head# 2023-07-02T14:05:34.785 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 0=[]:{} 2023-07-02T14:05:34.785 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:e# 2023-07-02T14:05:34.785 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:13# 2023-07-02T14:05:34.785 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:14# 2023-07-02T14:05:34.785 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:1a# 2023-07-02T14:05:34.786 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:21# 2023-07-02T14:05:34.786 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:26# 2023-07-02T14:05:34.786 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3a831e6f:::rbd_data.125de1449605.00000000000000f1:head# 2023-07-02T14:05:34.786 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{e=[e,d,c,b,a,9,8],13=[13,12,11,10,f],14=[14],1a=[1a,19,18,17,16,15],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:34.786 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3d4830c1:::rbd_data.125de1449605.0000000000000140:20# 2023-07-02T14:05:34.786 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3d4830c1:::rbd_data.125de1449605.0000000000000140:25# 2023-07-02T14:05:34.786 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3d4830c1:::rbd_data.125de1449605.0000000000000140:head# 2023-07-02T14:05:34.787 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 25=[]:{20=[20],25=[25,24,23,22,21]} 2023-07-02T14:05:34.787 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3e3fd82c:::rbd_data.125de1449605.0000000000000164:head# 2023-07-02T14:05:34.787 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{} 2023-07-02T14:05:34.787 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3e650bab:::rbd_data.125de1449605.0000000000000031:d# 2023-07-02T14:05:34.787 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3e650bab:::rbd_data.125de1449605.0000000000000031:13# 2023-07-02T14:05:34.787 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3e650bab:::rbd_data.125de1449605.0000000000000031:1a# 2023-07-02T14:05:34.787 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3e650bab:::rbd_data.125de1449605.0000000000000031:21# 2023-07-02T14:05:34.788 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3e650bab:::rbd_data.125de1449605.0000000000000031:26# 2023-07-02T14:05:34.788 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3e650bab:::rbd_data.125de1449605.0000000000000031:head# 2023-07-02T14:05:34.788 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{d=[d,c,b,a,9,8],13=[13,12,11,10,f,e],1a=[1a,19,18,17,16,15,14],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:34.788 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3e72164b:::rbd_data.125de1449605.0000000000000044:e# 2023-07-02T14:05:34.788 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3e72164b:::rbd_data.125de1449605.0000000000000044:13# 2023-07-02T14:05:34.788 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3e72164b:::rbd_data.125de1449605.0000000000000044:1a# 2023-07-02T14:05:34.788 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3e72164b:::rbd_data.125de1449605.0000000000000044:21# 2023-07-02T14:05:34.789 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3e72164b:::rbd_data.125de1449605.0000000000000044:26# 2023-07-02T14:05:34.789 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3e72164b:::rbd_data.125de1449605.0000000000000044:head# 2023-07-02T14:05:34.789 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{e=[e,d,c,b,a,9,8],13=[13,12,11,10,f],1a=[1a,19,18,17,16,15,14],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:34.789 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3e76625b:::rbd_data.125de1449605.0000000000000038:e# 2023-07-02T14:05:34.789 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3e76625b:::rbd_data.125de1449605.0000000000000038:13# 2023-07-02T14:05:34.789 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3e76625b:::rbd_data.125de1449605.0000000000000038:1a# 2023-07-02T14:05:34.789 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3e76625b:::rbd_data.125de1449605.0000000000000038:21# 2023-07-02T14:05:34.790 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3e76625b:::rbd_data.125de1449605.0000000000000038:26# 2023-07-02T14:05:34.790 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3e76625b:::rbd_data.125de1449605.0000000000000038:head# 2023-07-02T14:05:34.790 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{e=[e,d,c,b,a,9,8],13=[13,12,11,10,f],1a=[1a,19,18,17,16,15,14],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:34.790 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3ea366f9:::rbd_data.125de1449605.0000000000000115:e# 2023-07-02T14:05:34.790 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3ea366f9:::rbd_data.125de1449605.0000000000000115:14# 2023-07-02T14:05:34.790 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3ea366f9:::rbd_data.125de1449605.0000000000000115:1a# 2023-07-02T14:05:34.790 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3ea366f9:::rbd_data.125de1449605.0000000000000115:head# 2023-07-02T14:05:34.790 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 1a=[]:{e=[e,d,c,b,a,9,8],14=[14,13,12,11,10,f],1a=[1a,19,18,17,16,15]} 2023-07-02T14:05:34.791 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:e# 2023-07-02T14:05:34.791 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:13# 2023-07-02T14:05:34.791 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:14# 2023-07-02T14:05:34.791 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:1a# 2023-07-02T14:05:34.791 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:21# 2023-07-02T14:05:34.791 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:26# 2023-07-02T14:05:34.791 INFO:teuthology.orchestra.run.smithi016.stdout:Write #2:3ffa3b56:::rbd_data.125de1449605.00000000000000f2:head# 2023-07-02T14:05:34.791 INFO:teuthology.orchestra.run.smithi016.stdout:snapset 26=[]:{e=[e,d,c,b,a,9,8],13=[13,12,11,10,f],14=[14],1a=[1a,19,18,17,16,15],21=[21,20,1f,1e,1d,1c,1b],26=[26,25,24,23,22]} 2023-07-02T14:05:34.792 INFO:teuthology.orchestra.run.smithi016.stdout:write_pg epoch 146 info 2.4( v 146'1478 (0'0,146'1478] local-lis/les=63/64 n=212 ec=14/14 lis/c=63/63 les/c/f=64/64/0 sis=63) 2023-07-02T14:05:34.792 INFO:teuthology.orchestra.run.smithi016.stdout:Import successful 2023-07-02T14:05:34.792 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:34.779+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:34.880 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:34.880+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:34.916 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:34.916 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:34.916 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:05:34.916 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:34.926 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:05:34.980 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:34.980+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:35.049 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:05:35.050 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:05:35.050 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:35.060 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:05:35.099 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:05:35.101 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:35.101+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:35.121 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:05:35.128 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 1 2023-07-02T14:05:35.182 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:35.182+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:35.207 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /var/log/ceph/exp.2.4.0 2023-07-02T14:05:35.233 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:35.234 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:05:35.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:05:35.234 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:35.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.234 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:05:35.234 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:35.234 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 145.778672734, 2023-07-02T14:05:35.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:05:35.235 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:35.235 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:35.235 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:35.235 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:35.235 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:35.235 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:05:35.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.236 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:35.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:35.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:35.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:35.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:35.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:35.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:35.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:35.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:05:35.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:05:35.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:35.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:05:35.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:05:35.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:35.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:05:35.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:05:35.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:35.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:05:35.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:05:35.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:05:35.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:05:35.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:35.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:05:35.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:05:35.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:05:35.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:05:35.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:35.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:05:35.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:05:35.242 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.242 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.242 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.243 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:05:35.243 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:35.243 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 144.90797972600001, 2023-07-02T14:05:35.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:05:35.243 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:35.243 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:35.243 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:35.244 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:05:35.244 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:05:35.244 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:05:35.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.244 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:35.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:35.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:35.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:35.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:35.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:35.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:35.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:35.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:05:35.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:05:35.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:35.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:05:35.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:05:35.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:35.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:05:35.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:05:35.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:35.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:05:35.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:05:35.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:05:35.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:05:35.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:35.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:05:35.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:05:35.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:05:35.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:05:35.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:35.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:05:35.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:05:35.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:35.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:05:35.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:05:35.251 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.251 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.251 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.252 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:35.252 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:35.252 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 144.90785500499999, 2023-07-02T14:05:35.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:05:35.252 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:35.253 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:35.253 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:35.253 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:35.253 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:35.253 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:05:35.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.253 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:35.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:35.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:35.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:35.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:35.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:35.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:35.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:35.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:05:35.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:05:35.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:35.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:05:35.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:05:35.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:35.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:05:35.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:05:35.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:35.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:05:35.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:05:35.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:05:35.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:05:35.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:35.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:05:35.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:05:35.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:05:35.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:05:35.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:35.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:05:35.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:05:35.260 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.260 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.260 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.260 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:35.261 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:35.261 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 144.901005121, 2023-07-02T14:05:35.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:05:35.261 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:35.261 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:35.261 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:35.261 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:35.261 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:35.262 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:05:35.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.262 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:35.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:35.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:35.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:35.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:35.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:35.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:05:35.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:35.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:05:35.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:05:35.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:35.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:05:35.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:05:35.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:35.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:05:35.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:05:35.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:35.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:05:35.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:05:35.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:05:35.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:05:35.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:35.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:05:35.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:05:35.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:05:35.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:05:35.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:35.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:05:35.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:05:35.268 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.268 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.269 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.269 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:35.269 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:35.269 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 144.90094219599999, 2023-07-02T14:05:35.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:05:35.269 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:35.270 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:35.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:35.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:35.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:35.270 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:05:35.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.270 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:35.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:35.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:35.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:35.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:35.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:35.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:35.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:35.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:05:35.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:05:35.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:35.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:05:35.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:05:35.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:35.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:05:35.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:05:35.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:35.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:05:35.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:05:35.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:05:35.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:05:35.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:35.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:05:35.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:05:35.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:05:35.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:05:35.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:35.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:05:35.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:05:35.277 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.277 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.277 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.277 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:35.278 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:35.278 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 144.900641775, 2023-07-02T14:05:35.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:05:35.278 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:35.278 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:35.278 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:35.278 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:35.278 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:35.279 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:05:35.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.279 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:35.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:35.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:35.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:35.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:35.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:35.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:35.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:35.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:05:35.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:05:35.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:35.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:05:35.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:05:35.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:35.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:05:35.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:05:35.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:35.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:05:35.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:05:35.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:05:35.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:05:35.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:35.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:05:35.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:05:35.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:05:35.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:05:35.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:35.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:05:35.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:05:35.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:05:35.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:05:35.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:05:35.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:35.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:05:35.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:05:35.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:35.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:05:35.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:05:35.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:35.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:05:35.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:05:35.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:35.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:05:35.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:05:35.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:35.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:05:35.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:05:35.289 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.289 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.289 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.290 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:05:35.290 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:35.290 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 144.90011727500001, 2023-07-02T14:05:35.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:05:35.290 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:35.290 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:35.290 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:35.291 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:35.291 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:35.291 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:05:35.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.291 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:35.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:35.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:35.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:35.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:35.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:35.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:35.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:35.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:35.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:05:35.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:35.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:05:35.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:05:35.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:35.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:05:35.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:05:35.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:35.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:05:35.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:05:35.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:05:35.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:05:35.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:35.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:05:35.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:05:35.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:05:35.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:05:35.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:35.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:05:35.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:05:35.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:35.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:05:35.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:05:35.298 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.298 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.298 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.299 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:05:35.299 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:35.299 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 117.77609815, 2023-07-02T14:05:35.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:05:35.299 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:35.299 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:35.300 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:35.300 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:35.300 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:35.300 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:05:35.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.300 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:35.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:35.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:35.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:35.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:35.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:35.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:35.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:35.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:05:35.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:05:35.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:35.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:05:35.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:05:35.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:35.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:05:35.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:05:35.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:35.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:05:35.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:05:35.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:05:35.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:05:35.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:35.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:05:35.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:05:35.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:05:35.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:05:35.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:35.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:05:35.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:05:35.307 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.307 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.307 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.307 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:35.307 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:35.307 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 117.450615446, 2023-07-02T14:05:35.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:05:35.308 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:35.308 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:35.308 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:35.308 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:35.308 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:35.308 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:05:35.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.309 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:35.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:35.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:35.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:35.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:35.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:35.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:35.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:35.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:05:35.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:05:35.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:35.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:05:35.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:05:35.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:35.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:05:35.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:05:35.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:35.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:05:35.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:05:35.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:35.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:05:35.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:35.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:05:35.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:05:35.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:05:35.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:05:35.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:35.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:05:35.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:05:35.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:35.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:05:35.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:05:35.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:35.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:05:35.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:05:35.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:35.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:05:35.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:05:35.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:35.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:05:35.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:05:35.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:35.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:05:35.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:05:35.318 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.318 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.318 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.319 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3939 2.1 2:9ef34a83:::rbd_data.125de1449605.00000000000000b8:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:35.319 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:35.319 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 117.446694749, 2023-07-02T14:05:35.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001, 2023-07-02T14:05:35.319 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:35.320 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:35.320 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:35.320 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:35.320 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:35.320 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3939 2023-07-02T14:05:35.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.320 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:35.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:35.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:35.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:35.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:35.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:35.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:35.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:35.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:35.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046852830000000002 2023-07-02T14:05:35.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:35.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788763+0000", 2023-07-02T14:05:35.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3185e-05 2023-07-02T14:05:35.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:35.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788776+0000", 2023-07-02T14:05:35.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8470000000000003e-06 2023-07-02T14:05:35.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:35.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788781+0000", 2023-07-02T14:05:35.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7104000000000002e-05 2023-07-02T14:05:35.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788828+0000", 2023-07-02T14:05:35.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3516000000000001e-05 2023-07-02T14:05:35.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:35.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788841+0000", 2023-07-02T14:05:35.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33064513600000001 2023-07-02T14:05:35.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119487+0000", 2023-07-02T14:05:35.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9876999999999997e-05 2023-07-02T14:05:35.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:35.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119537+0000", 2023-07-02T14:05:35.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014160999999999999 2023-07-02T14:05:35.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:35.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119678+0000", 2023-07-02T14:05:35.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023054392 2023-07-02T14:05:35.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:35.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.142733+0000", 2023-07-02T14:05:35.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074065470000000003 2023-07-02T14:05:35.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:35.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150139+0000", 2023-07-02T14:05:35.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5025999999999997e-05 2023-07-02T14:05:35.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:35.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150204+0000", 2023-07-02T14:05:35.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147999999999997e-05 2023-07-02T14:05:35.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:35.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150241+0000", 2023-07-02T14:05:35.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001 2023-07-02T14:05:35.330 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.330 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.330 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.331 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:17 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 0~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:35.331 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:05:35.331 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.032339663999998, 2023-07-02T14:05:35.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997, 2023-07-02T14:05:35.331 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:35.331 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:35.331 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:35.331 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:35.332 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:35.332 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17 2023-07-02T14:05:35.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.332 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:35.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:35.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:05:35.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:35.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:35.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:05:35.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:05:35.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:35.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198431+0000", 2023-07-02T14:05:35.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2815e-05 2023-07-02T14:05:35.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:35.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198443+0000", 2023-07-02T14:05:35.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2569999999999999e-05 2023-07-02T14:05:35.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:35.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198466+0000", 2023-07-02T14:05:35.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0223e-05 2023-07-02T14:05:35.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:35.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198476+0000", 2023-07-02T14:05:35.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1792999999999997e-05 2023-07-02T14:05:35.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:05:35.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4249999999999998e-06 2023-07-02T14:05:35.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:35.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198536+0000", 2023-07-02T14:05:35.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91888663500000001 2023-07-02T14:05:35.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117423+0000", 2023-07-02T14:05:35.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223594 2023-07-02T14:05:35.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:35.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117647+0000", 2023-07-02T14:05:35.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033832900000000001 2023-07-02T14:05:35.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:35.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117985+0000", 2023-07-02T14:05:35.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997 2023-07-02T14:05:35.339 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.339 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.339 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.340 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:18 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 65536~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:35.340 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:35.340 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.032282453000001, 2023-07-02T14:05:35.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998, 2023-07-02T14:05:35.340 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:35.340 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:35.340 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:35.341 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:35.341 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:35.341 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18 2023-07-02T14:05:35.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.341 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:35.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:35.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:35.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:35.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:35.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:35.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:35.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:35.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:35.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7600000000000002e-06 2023-07-02T14:05:35.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:35.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198494+0000", 2023-07-02T14:05:35.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7840000000000004e-06 2023-07-02T14:05:35.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:35.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198502+0000", 2023-07-02T14:05:35.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6459999999999998e-06 2023-07-02T14:05:35.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:35.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198508+0000", 2023-07-02T14:05:35.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7395999999999997e-05 2023-07-02T14:05:35.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198565+0000", 2023-07-02T14:05:35.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6800000000000004e-06 2023-07-02T14:05:35.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:35.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198572+0000", 2023-07-02T14:05:35.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91965896800000002 2023-07-02T14:05:35.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118231+0000", 2023-07-02T14:05:35.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5212999999999998e-05 2023-07-02T14:05:35.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:35.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118326+0000", 2023-07-02T14:05:35.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000238538 2023-07-02T14:05:35.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:35.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118565+0000", 2023-07-02T14:05:35.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998 2023-07-02T14:05:35.348 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.348 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.348 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.349 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:19 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 131072~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:35.349 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:35.349 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.032258229, 2023-07-02T14:05:35.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002, 2023-07-02T14:05:35.349 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:35.349 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:35.350 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:35.350 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:35.350 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:35.350 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 19 2023-07-02T14:05:35.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.350 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:35.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:35.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:35.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:35.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:35.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:35.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:35.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:35.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:35.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1990000000000002e-06 2023-07-02T14:05:35.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:35.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198517+0000", 2023-07-02T14:05:35.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8789999999999997e-06 2023-07-02T14:05:35.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:35.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198524+0000", 2023-07-02T14:05:35.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:05:35.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:35.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:05:35.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5545999999999999e-05 2023-07-02T14:05:35.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:35.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2540000000000003e-06 2023-07-02T14:05:35.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:35.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198598+0000", 2023-07-02T14:05:35.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.919983263 2023-07-02T14:05:35.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118581+0000", 2023-07-02T14:05:35.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6878999999999994e-05 2023-07-02T14:05:35.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:35.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118648+0000", 2023-07-02T14:05:35.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149732 2023-07-02T14:05:35.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:35.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118798+0000", 2023-07-02T14:05:35.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002 2023-07-02T14:05:35.357 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.357 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.357 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.358 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:20 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 262144~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:35.358 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:35.358 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.032239265000001, 2023-07-02T14:05:35.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998, 2023-07-02T14:05:35.358 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:35.359 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:35.359 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:35.359 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:35.359 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:35.359 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20 2023-07-02T14:05:35.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.359 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:35.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:35.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:35.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:35.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:35.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:35.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:35.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:35.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:35.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.732e-06 2023-07-02T14:05:35.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:35.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198537+0000", 2023-07-02T14:05:35.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5259999999999999e-06 2023-07-02T14:05:35.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:35.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198543+0000", 2023-07-02T14:05:35.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.121e-06 2023-07-02T14:05:35.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:35.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198547+0000", 2023-07-02T14:05:35.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9591000000000005e-05 2023-07-02T14:05:35.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:05:35.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3120000000000004e-06 2023-07-02T14:05:35.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:35.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198621+0000", 2023-07-02T14:05:35.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92020789400000003 2023-07-02T14:05:35.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118829+0000", 2023-07-02T14:05:35.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022393386000000001 2023-07-02T14:05:35.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:35.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141223+0000", 2023-07-02T14:05:35.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000329866 2023-07-02T14:05:35.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:35.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141552+0000", 2023-07-02T14:05:35.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998 2023-07-02T14:05:35.366 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.367 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.367 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.367 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:21 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 196608~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:35.367 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:35.367 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.032211629999999, 2023-07-02T14:05:35.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995, 2023-07-02T14:05:35.368 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:35.368 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:35.368 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:35.368 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:35.368 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:35.368 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21 2023-07-02T14:05:35.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.368 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:35.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:35.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:35.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:35.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:35.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:35.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:35.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:35.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:35.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0169999999999999e-05 2023-07-02T14:05:35.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:35.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198571+0000", 2023-07-02T14:05:35.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3479999999999998e-06 2023-07-02T14:05:35.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:35.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198578+0000", 2023-07-02T14:05:35.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8439999999999997e-06 2023-07-02T14:05:35.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:35.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198583+0000", 2023-07-02T14:05:35.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8294999999999997e-05 2023-07-02T14:05:35.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198641+0000", 2023-07-02T14:05:35.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8120000000000001e-06 2023-07-02T14:05:35.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:35.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:05:35.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94293828499999999 2023-07-02T14:05:35.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141583+0000", 2023-07-02T14:05:35.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014607300000000001 2023-07-02T14:05:35.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:35.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141729+0000", 2023-07-02T14:05:35.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00044906600000000003 2023-07-02T14:05:35.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:35.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142178+0000", 2023-07-02T14:05:35.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995 2023-07-02T14:05:35.375 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.376 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.376 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.376 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:22 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 327680~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:35.376 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:35.376 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.032178200000001, 2023-07-02T14:05:35.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004, 2023-07-02T14:05:35.377 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:35.377 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:35.377 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:35.377 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:35.377 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:35.377 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 22 2023-07-02T14:05:35.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.377 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:35.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:35.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:35.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:35.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:35.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:35.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:35.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:35.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:35.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4850000000000001e-06 2023-07-02T14:05:35.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:35.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198597+0000", 2023-07-02T14:05:35.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:05:35.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:35.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198605+0000", 2023-07-02T14:05:35.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4839999999999998e-06 2023-07-02T14:05:35.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:35.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198609+0000", 2023-07-02T14:05:35.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3100999999999999e-05 2023-07-02T14:05:35.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198662+0000", 2023-07-02T14:05:35.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9099999999999998e-06 2023-07-02T14:05:35.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:35.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198666+0000", 2023-07-02T14:05:35.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.943539503 2023-07-02T14:05:35.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142206+0000", 2023-07-02T14:05:35.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012255999999999999 2023-07-02T14:05:35.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:35.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142328+0000", 2023-07-02T14:05:35.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208956 2023-07-02T14:05:35.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:35.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142537+0000", 2023-07-02T14:05:35.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004 2023-07-02T14:05:35.384 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.385 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.385 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.385 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:23 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 393216~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:35.385 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:35.385 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.032158219999999, 2023-07-02T14:05:35.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348, 2023-07-02T14:05:35.386 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:35.386 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:35.386 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:35.386 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:35.386 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:35.386 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23 2023-07-02T14:05:35.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.387 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:35.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:35.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:35.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:35.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:35.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:35.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:35.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:35.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:35.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8439999999999998e-06 2023-07-02T14:05:35.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:35.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:05:35.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0440000000000001e-06 2023-07-02T14:05:35.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:35.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198623+0000", 2023-07-02T14:05:35.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4480000000000002e-06 2023-07-02T14:05:35.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:35.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198628+0000", 2023-07-02T14:05:35.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5843e-05 2023-07-02T14:05:35.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:35.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.89e-06 2023-07-02T14:05:35.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:35.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198688+0000", 2023-07-02T14:05:35.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94389171999999999 2023-07-02T14:05:35.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142580+0000", 2023-07-02T14:05:35.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3892000000000004e-05 2023-07-02T14:05:35.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:35.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142644+0000", 2023-07-02T14:05:35.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017608000000000001 2023-07-02T14:05:35.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:35.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142820+0000", 2023-07-02T14:05:35.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348 2023-07-02T14:05:35.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.394 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.394 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:24 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 458752~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:35.394 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:05:35.394 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.032130146, 2023-07-02T14:05:35.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004, 2023-07-02T14:05:35.395 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:35.395 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:35.395 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:35.395 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:35.395 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:35.395 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 24 2023-07-02T14:05:35.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.396 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:35.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:35.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:05:35.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:35.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:35.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:05:35.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:35.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:35.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198642+0000", 2023-07-02T14:05:35.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.253e-06 2023-07-02T14:05:35.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:35.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:05:35.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1289999999999998e-06 2023-07-02T14:05:35.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:35.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198652+0000", 2023-07-02T14:05:35.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4510000000000002e-06 2023-07-02T14:05:35.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:35.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198657+0000", 2023-07-02T14:05:35.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9131000000000001e-05 2023-07-02T14:05:35.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198706+0000", 2023-07-02T14:05:35.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8890000000000004e-06 2023-07-02T14:05:35.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:35.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198710+0000", 2023-07-02T14:05:35.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94415412200000004 2023-07-02T14:05:35.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142864+0000", 2023-07-02T14:05:35.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8047000000000006e-05 2023-07-02T14:05:35.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:35.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142952+0000", 2023-07-02T14:05:35.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192967 2023-07-02T14:05:35.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:35.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143145+0000", 2023-07-02T14:05:35.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004 2023-07-02T14:05:35.403 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.403 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.403 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.403 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:25 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 524288~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:35.403 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:35.404 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.032111351000001, 2023-07-02T14:05:35.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001, 2023-07-02T14:05:35.404 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:35.404 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:35.404 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:35.404 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:35.404 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:35.405 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25 2023-07-02T14:05:35.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.405 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:35.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:35.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:35.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:35.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:35.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:35.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:35.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:35.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:35.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.615e-06 2023-07-02T14:05:35.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:35.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198664+0000", 2023-07-02T14:05:35.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4759999999999997e-06 2023-07-02T14:05:35.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:35.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198669+0000", 2023-07-02T14:05:35.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5999999999999998e-06 2023-07-02T14:05:35.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:35.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198673+0000", 2023-07-02T14:05:35.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4256999999999999e-05 2023-07-02T14:05:35.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198727+0000", 2023-07-02T14:05:35.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840000000000001e-06 2023-07-02T14:05:35.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:35.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198731+0000", 2023-07-02T14:05:35.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94444241900000003 2023-07-02T14:05:35.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143173+0000", 2023-07-02T14:05:35.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6839000000000002e-05 2023-07-02T14:05:35.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:35.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143240+0000", 2023-07-02T14:05:35.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017053399999999999 2023-07-02T14:05:35.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:35.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143411+0000", 2023-07-02T14:05:35.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001 2023-07-02T14:05:35.412 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.412 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.412 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.413 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:26 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 589824~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:35.413 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:35.413 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 24.0320882, 2023-07-02T14:05:35.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997, 2023-07-02T14:05:35.413 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:35.413 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:35.413 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:35.414 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:35.414 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:35.414 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 26 2023-07-02T14:05:35.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.414 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:35.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:35.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:35.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:35.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:35.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:35.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:35.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:35.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:35.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0520000000000002e-06 2023-07-02T14:05:35.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:35.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198687+0000", 2023-07-02T14:05:35.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1880000000000002e-06 2023-07-02T14:05:35.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:35.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198695+0000", 2023-07-02T14:05:35.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9960000000000004e-06 2023-07-02T14:05:35.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:35.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198699+0000", 2023-07-02T14:05:35.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2330999999999998e-05 2023-07-02T14:05:35.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198742+0000", 2023-07-02T14:05:35.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7950000000000001e-06 2023-07-02T14:05:35.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:35.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198745+0000", 2023-07-02T14:05:35.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94469398199999999 2023-07-02T14:05:35.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:35.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143439+0000", 2023-07-02T14:05:35.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6048e-05 2023-07-02T14:05:35.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:35.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143525+0000", 2023-07-02T14:05:35.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017016700000000001 2023-07-02T14:05:35.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:35.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:35.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:35.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143696+0000", 2023-07-02T14:05:35.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997 2023-07-02T14:05:35.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.421 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:35.422 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:35.422 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:35.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:05:35.423 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:05:35.425 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-02T14:05:35.425 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:35.423+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:35.524 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:35.524+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:35.602 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:35.603 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:35.603 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:05:35.603 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:35.610 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:35.615 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:05:35.629 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:35.630 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:35.629+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:35.648 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:35.667 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:35.686 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:35.686 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-07-02T14:05:35.686 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:05:35.687 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:05:35.687 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:05:35.687 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:05:35.687 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:05:35.690 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:05:35.690 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:05:35.731 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:35.731+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:35.754 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:35.754 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:35.765 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:05:35.781 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:05:35.781 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":156,"fsid":"9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2","created":"2023-07-02T14:01:37.298060+0000","modified":"2023-07-02T14:05:35.084528+0000","last_up_change":"2023-07-02T14:05:23.150734+0000","last_in_change":"2023-07-02T14:04:41.351678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-07-02T14:01:45.323455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-02T14:01:47.448897+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"156","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":43,"snap_epoch":156,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"43c446fe-c811-4ee3-99e8-d7452edfc1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":136,"down_at":149,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":83359},{"type":"v1","addr":"172.21.15.16:6801","nonce":83359}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":83359},{"type":"v1","addr":"172.21.15.16:6803","nonce":83359}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":83359},{"type":"v1","addr":"172.21.15.16:6807","nonce":83359}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":83359},{"type":"v1","addr":"172.21.15.16:6805","nonce":83359}]},"public_addr":"172.21.15.16:6801/83359","cluster_addr":"172.21.15.16:6803/83359","heartbeat_back_addr":"172.21.15.16:6807/83359","heartbeat_front_addr":"172.21.15.16:6805/83359","state":["exists"]},{"osd":1,"uuid":"4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":83360},{"type":"v1","addr":"172.21.15.16:6809","nonce":83360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":83360},{"type":"v1","addr":"172.21.15.16:6811","nonce":83360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":83360},{"type":"v1","addr":"172.21.15.16:6815","nonce":83360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":83360},{"type":"v1","addr":"172.21.15.16:6813","nonce":83360}]},"public_addr":"172.21.15.16:6809/83360","cluster_addr":"172.21.15.16:6811/83360","heartbeat_back_addr":"172.21.15.16:6815/83360","heartbeat_front_addr":"172.21.15.16:6813/83360","state":["exists","up"]},{"osd":2,"uuid":"99f8ea0a-5a71-4abe-b087-8e891fd11d1b","up":1,"in":1,"weight":1,"primary_affinity":0.9522247314453125,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":83361},{"type":"v1","addr":"172.21.15.16:6817","nonce":83361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":83361},{"type":"v1","addr":"172.21.15.16:6819","nonce":83361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":83361},{"type":"v1","addr":"172.21.15.16:6823","nonce":83361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":83361},{"type":"v1","addr":"172.21.15.16:6821","nonce":83361}]},"public_addr":"172.21.15.16:6817/83361","cluster_addr":"172.21.15.16:6819/83361","heartbeat_back_addr":"172.21.15.16:6823/83361","heartbeat_front_addr":"172.21.15.16:6821/83361","state":["exists","up"]},{"osd":3,"uuid":"d285b4d5-0bef-421a-b84a-943e485b10c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":87,"up_from":133,"up_thru":149,"down_at":88,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":99977},{"type":"v1","addr":"172.21.15.22:6809","nonce":99977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":99977},{"type":"v1","addr":"172.21.15.22:6811","nonce":99977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":99977},{"type":"v1","addr":"172.21.15.22:6815","nonce":99977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":99977},{"type":"v1","addr":"172.21.15.22:6813","nonce":99977}]},"public_addr":"172.21.15.22:6809/99977","cluster_addr":"172.21.15.22:6811/99977","heartbeat_back_addr":"172.21.15.22:6815/99977","heartbeat_front_addr":"172.21.15.22:6813/99977","state":["exists","up"]},{"osd":4,"uuid":"7ba0b0db-e407-4c63-8278-aab09f3082c4","up":1,"in":0,"weight":0,"primary_affinity":0.23297119140625,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":83351},{"type":"v1","addr":"172.21.15.22:6801","nonce":83351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":83351},{"type":"v1","addr":"172.21.15.22:6803","nonce":83351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":83351},{"type":"v1","addr":"172.21.15.22:6807","nonce":83351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":83351},{"type":"v1","addr":"172.21.15.22:6805","nonce":83351}]},"public_addr":"172.21.15.22:6801/83351","cluster_addr":"172.21.15.22:6803/83351","heartbeat_back_addr":"172.21.15.22:6807/83351","heartbeat_front_addr":"172.21.15.22:6805/83351","state":["exists","up"]},{"osd":5,"uuid":"29663bfa-f6fe-4ce7-b45a-a9b7b4424c42","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":83352},{"type":"v1","addr":"172.21.15.22:6817","nonce":83352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":83352},{"type":"v1","addr":"172.21.15.22:6819","nonce":83352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":83352},{"type":"v1","addr":"172.21.15.22:6823","nonce":83352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":83352},{"type":"v1","addr":"172.21.15.22:6821","nonce":83352}]},"public_addr":"172.21.15.22:6817/83352","cluster_addr":"172.21.15.22:6819/83352","heartbeat_back_addr":"172.21.15.22:6823/83352","heartbeat_front_addr":"172.21.15.22:6821/83352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.001178+0000","dead_epoch":149},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.184181+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.274854+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.186217+0000","dead_epoch":88},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":65536,"last_purged_snaps_scrub":"2023-07-02T14:01:44.247442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":65536,"last_purged_snaps_scrub":"2023-07-02T14:01:44.220075+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[0,3]},{"pgid":"2.0","osds":[4,1]},{"pgid":"2.2","osds":[1,5]},{"pgid":"2.4","osds":[5,0]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-02T14:05:35.795 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-07-02T14:01:45.323455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-02T14:01:47.448897+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '156', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 43, 'snap_epoch': 156, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-07-02T14:05:35.795 DEBUG:teuthology.orchestra.run.smithi016:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op apply-layout-settings --pool device_health_metrics 2023-07-02T14:05:35.826 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:35.825+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:35.909 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:35.910 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:35.920 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:05:35.963 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:35.963+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:36.027 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:36.027+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:36.058 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:36.058 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:36.069 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:05:36.095 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:05:36.129 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap39 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap40 b3fbb028-e267-4d25-a750-93ba18a2316a@snap40.diff 2023-07-02T14:05:36.147 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:36.147+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:36.192 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete...2023-07-02T14:05:36.191+0000 7f926c97d700 0 -- 172.21.15.163:0/152257757 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f9248006500 msgr2=0x7f9248004110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:36.195 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:05:36.195+0000 7f926c97d700 0 -- 172.21.15.163:0/152257757 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f9250036bb0 msgr2=0x7f9250038fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:36.205 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 51% complete... Exporting image: 100% complete...done. 2023-07-02T14:05:36.209 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:05:36.209 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=41 2023-07-02T14:05:36.210 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap41 2023-07-02T14:05:36.228 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:36.227+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:36.234 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:36.234 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:36.244 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:05:36.329 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:36.328+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:36.409 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:36.409 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:36.419 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:05:36.429 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:36.429+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:36.529 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:36.529+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:36.583 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:36.583 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:36.593 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:05:36.634 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:36.633+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:36.731 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:36.730+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:36.756 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:36.756 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:36.766 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:05:36.849 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:36.848+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:36.931 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:36.932 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:36.932 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:36.932+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:36.941 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:05:37.033 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:37.032+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:37.110 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:37.111 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:37.123 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:05:37.126 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:05:37.132 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 2 2023-07-02T14:05:37.165 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:37.164+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:37.235 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:37.234+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:37.288 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:37.288 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:37.298 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:05:37.340 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:37.339+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:37.436 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:37.435+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:37.463 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:37.464 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:37.474 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:05:37.537 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:37.537+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:37.571 INFO:tasks.workunit.client.0.smithi163.stderr:+ tar -C b3fbb028-e267-4d25-a750-93ba18a2316a-mnt -xzf b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4.tar.gz 2023-07-02T14:05:37.598 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:37.598 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:37.608 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:05:37.651 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:37.651+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:37.691 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:05:37.699 INFO:teuthology.orchestra.run.smithi016.stderr:Nothing to do for non-filestore backend 2023-07-02T14:05:37.736 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:37.736 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:37.738 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:37.738+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:37.747 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:05:37.834 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:37.839 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:37.839+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:37.852 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:37.870 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:37.888 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:37.907 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:37.907 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-07-02T14:05:37.908 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:05:37.908 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:05:37.908 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:05:37.908 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:05:37.908 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:05:37.910 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:05:37.911 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:05:37.916 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:37.917 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:05:37.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:05:37.917 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:37.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.917 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:05:37.917 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:37.917 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 148.46293134699999, 2023-07-02T14:05:37.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:05:37.918 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:37.918 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:37.918 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:37.918 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:37.918 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:37.918 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:05:37.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.919 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:37.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:37.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:37.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:37.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:37.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:37.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:37.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:37.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:05:37.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:05:37.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:37.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:05:37.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:05:37.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:37.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:05:37.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:05:37.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:37.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:05:37.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:05:37.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:37.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:05:37.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:05:37.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:37.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:05:37.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:05:37.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:37.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:05:37.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:05:37.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:37.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:05:37.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:05:37.925 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:37.925 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:37.925 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:37.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.926 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:05:37.926 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:37.926 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 147.592238339, 2023-07-02T14:05:37.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:05:37.926 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:37.926 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:37.927 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:37.927 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:05:37.927 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:05:37.927 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:05:37.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.927 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:37.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:37.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:37.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:37.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:37.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:37.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:37.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:37.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:05:37.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:05:37.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:37.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:05:37.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:05:37.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:37.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:05:37.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:05:37.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:37.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:05:37.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:05:37.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:37.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:05:37.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:05:37.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:37.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:05:37.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:05:37.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:37.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:05:37.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:05:37.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:37.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:05:37.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:05:37.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:37.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:05:37.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:05:37.934 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:37.934 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:37.934 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:37.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.935 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:37.935 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:37.935 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 147.59211361800001, 2023-07-02T14:05:37.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:05:37.935 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:37.936 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:37.936 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:37.936 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:37.936 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:37.936 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:05:37.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.936 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:37.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:37.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:37.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:37.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:37.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:37.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:37.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:37.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:05:37.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:05:37.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:37.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:05:37.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:05:37.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:37.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:05:37.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:05:37.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:37.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:05:37.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:05:37.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:37.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:05:37.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:05:37.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:37.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:05:37.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:05:37.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:37.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:05:37.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:05:37.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:37.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:05:37.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:05:37.943 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:37.943 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:37.943 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:37.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.943 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:37.944 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:37.944 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 147.58526373399999, 2023-07-02T14:05:37.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:05:37.944 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:37.944 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:37.944 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:37.944 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:37.945 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:37.945 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:05:37.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.945 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:37.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:37.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:37.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:37.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:37.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:37.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:05:37.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:37.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:05:37.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:05:37.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:37.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:05:37.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:05:37.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:37.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:05:37.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:05:37.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:37.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:05:37.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:05:37.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:37.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:05:37.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:05:37.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:37.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:05:37.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:05:37.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:37.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:05:37.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:05:37.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:37.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:05:37.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:05:37.951 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:37.951 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:37.952 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:37.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.952 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:37.952 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:37.952 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 147.58520080900001, 2023-07-02T14:05:37.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:05:37.953 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:37.953 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:37.953 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:37.953 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:37.953 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:37.953 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:05:37.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.953 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:37.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:37.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:37.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:37.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:37.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:37.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:37.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:37.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:05:37.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:05:37.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:37.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:05:37.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:05:37.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:37.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:05:37.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:05:37.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:37.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:05:37.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:05:37.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:37.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:05:37.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:05:37.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:37.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:05:37.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:05:37.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:37.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:05:37.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:05:37.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:37.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:05:37.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:05:37.960 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:37.960 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:37.960 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:37.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.960 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:37.961 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:37.961 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 147.58490038799999, 2023-07-02T14:05:37.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:05:37.961 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:37.961 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:37.961 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:37.961 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:37.961 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:37.962 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:05:37.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.962 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:37.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:37.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:37.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:37.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:37.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:37.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:37.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:37.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:05:37.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:05:37.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:37.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:05:37.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:05:37.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:37.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:05:37.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:05:37.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:37.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:05:37.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:05:37.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:37.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:05:37.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:05:37.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:37.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:05:37.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:05:37.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:37.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:05:37.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:05:37.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:37.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:05:37.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:05:37.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:05:37.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:05:37.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:05:37.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:37.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:05:37.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:05:37.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:37.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:05:37.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:05:37.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:37.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:05:37.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:05:37.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:37.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:05:37.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:05:37.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:37.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:05:37.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:05:37.972 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:37.972 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:37.972 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:37.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.973 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:05:37.973 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:37.973 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 147.58437588800001, 2023-07-02T14:05:37.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:05:37.973 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:37.973 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:37.974 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:37.974 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:37.974 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:37.974 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:05:37.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.974 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:37.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:37.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:37.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:37.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:37.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:37.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:37.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:37.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:37.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:05:37.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:05:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:05:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:05:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:05:37.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:05:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:05:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:37.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:05:37.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:05:37.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:37.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:05:37.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:05:37.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:37.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:05:37.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:05:37.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:37.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:05:37.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:05:37.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:37.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:05:37.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:05:37.981 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:37.981 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:37.981 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:37.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.982 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:05:37.982 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:37.982 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 120.46035676299999, 2023-07-02T14:05:37.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:05:37.982 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:37.983 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:37.983 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:37.983 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:37.983 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:37.983 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:05:37.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.983 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:37.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:37.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:37.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:37.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:37.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:37.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:37.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:37.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:05:37.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:05:37.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:37.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:05:37.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:05:37.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:37.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:05:37.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:05:37.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:37.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:05:37.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:05:37.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:37.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:05:37.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:05:37.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:37.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:05:37.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:05:37.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:37.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:05:37.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:05:37.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:37.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:05:37.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:05:37.990 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:37.990 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:37.990 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:37.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.990 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:37.990 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:37.991 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 120.134874059, 2023-07-02T14:05:37.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:05:37.991 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:37.991 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:37.991 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:37.991 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:37.991 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:37.992 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:05:37.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.992 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:37.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:37.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:37.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:37.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:37.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:37.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:37.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:37.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:05:37.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:05:37.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:37.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:05:37.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:05:37.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:37.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:05:37.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:05:37.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:37.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:05:37.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:05:37.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:37.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:37.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:05:37.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:37.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:05:37.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:05:37.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:37.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:05:37.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:05:37.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:37.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:05:37.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:05:37.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:37.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:05:37.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:05:37.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:37.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:37.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:37.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:05:38.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:05:38.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:38.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:05:38.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:05:38.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:38.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:05:38.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:05:38.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:05:38.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:05:38.002 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.002 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.002 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.002 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3939 2.1 2:9ef34a83:::rbd_data.125de1449605.00000000000000b8:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:38.002 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:38.002 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 120.130953362, 2023-07-02T14:05:38.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001, 2023-07-02T14:05:38.003 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.003 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:38.003 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.003 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:38.003 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:38.003 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3939 2023-07-02T14:05:38.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.004 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:38.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:38.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:38.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:38.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046852830000000002 2023-07-02T14:05:38.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788763+0000", 2023-07-02T14:05:38.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3185e-05 2023-07-02T14:05:38.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788776+0000", 2023-07-02T14:05:38.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8470000000000003e-06 2023-07-02T14:05:38.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788781+0000", 2023-07-02T14:05:38.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7104000000000002e-05 2023-07-02T14:05:38.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788828+0000", 2023-07-02T14:05:38.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3516000000000001e-05 2023-07-02T14:05:38.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:38.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788841+0000", 2023-07-02T14:05:38.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33064513600000001 2023-07-02T14:05:38.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119487+0000", 2023-07-02T14:05:38.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9876999999999997e-05 2023-07-02T14:05:38.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119537+0000", 2023-07-02T14:05:38.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014160999999999999 2023-07-02T14:05:38.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:38.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119678+0000", 2023-07-02T14:05:38.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023054392 2023-07-02T14:05:38.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:38.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.142733+0000", 2023-07-02T14:05:38.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074065470000000003 2023-07-02T14:05:38.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:38.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150139+0000", 2023-07-02T14:05:38.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5025999999999997e-05 2023-07-02T14:05:38.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:38.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150204+0000", 2023-07-02T14:05:38.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147999999999997e-05 2023-07-02T14:05:38.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150241+0000", 2023-07-02T14:05:38.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001 2023-07-02T14:05:38.013 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.013 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.013 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.014 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:17 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 0~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:38.014 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:05:38.014 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.716598276999999, 2023-07-02T14:05:38.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997, 2023-07-02T14:05:38.014 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.014 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:38.015 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.015 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:38.015 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:38.015 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17 2023-07-02T14:05:38.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.015 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:05:38.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:05:38.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:05:38.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198431+0000", 2023-07-02T14:05:38.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2815e-05 2023-07-02T14:05:38.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198443+0000", 2023-07-02T14:05:38.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2569999999999999e-05 2023-07-02T14:05:38.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198466+0000", 2023-07-02T14:05:38.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0223e-05 2023-07-02T14:05:38.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198476+0000", 2023-07-02T14:05:38.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1792999999999997e-05 2023-07-02T14:05:38.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:05:38.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4249999999999998e-06 2023-07-02T14:05:38.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:38.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198536+0000", 2023-07-02T14:05:38.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91888663500000001 2023-07-02T14:05:38.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117423+0000", 2023-07-02T14:05:38.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223594 2023-07-02T14:05:38.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117647+0000", 2023-07-02T14:05:38.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033832900000000001 2023-07-02T14:05:38.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117985+0000", 2023-07-02T14:05:38.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997 2023-07-02T14:05:38.022 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.022 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.023 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.023 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:18 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 65536~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:38.023 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:38.023 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.716541066000001, 2023-07-02T14:05:38.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998, 2023-07-02T14:05:38.023 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.024 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:38.024 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.024 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:38.024 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:38.024 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18 2023-07-02T14:05:38.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.024 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:38.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:38.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:38.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:38.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7600000000000002e-06 2023-07-02T14:05:38.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198494+0000", 2023-07-02T14:05:38.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7840000000000004e-06 2023-07-02T14:05:38.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198502+0000", 2023-07-02T14:05:38.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6459999999999998e-06 2023-07-02T14:05:38.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198508+0000", 2023-07-02T14:05:38.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7395999999999997e-05 2023-07-02T14:05:38.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198565+0000", 2023-07-02T14:05:38.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6800000000000004e-06 2023-07-02T14:05:38.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:38.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198572+0000", 2023-07-02T14:05:38.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91965896800000002 2023-07-02T14:05:38.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118231+0000", 2023-07-02T14:05:38.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5212999999999998e-05 2023-07-02T14:05:38.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118326+0000", 2023-07-02T14:05:38.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000238538 2023-07-02T14:05:38.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118565+0000", 2023-07-02T14:05:38.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998 2023-07-02T14:05:38.031 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.032 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.032 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.032 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:19 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 131072~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:38.032 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:38.032 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.716516842000001, 2023-07-02T14:05:38.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002, 2023-07-02T14:05:38.033 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.033 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:38.033 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.033 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:38.033 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:38.033 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 19 2023-07-02T14:05:38.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.033 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:38.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:38.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:38.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:38.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1990000000000002e-06 2023-07-02T14:05:38.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198517+0000", 2023-07-02T14:05:38.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8789999999999997e-06 2023-07-02T14:05:38.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198524+0000", 2023-07-02T14:05:38.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:05:38.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:05:38.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5545999999999999e-05 2023-07-02T14:05:38.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:38.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2540000000000003e-06 2023-07-02T14:05:38.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:38.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198598+0000", 2023-07-02T14:05:38.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.919983263 2023-07-02T14:05:38.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118581+0000", 2023-07-02T14:05:38.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6878999999999994e-05 2023-07-02T14:05:38.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118648+0000", 2023-07-02T14:05:38.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149732 2023-07-02T14:05:38.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118798+0000", 2023-07-02T14:05:38.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002 2023-07-02T14:05:38.041 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.041 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.041 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.041 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:20 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 262144~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:38.041 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:38.041 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.716497877999998, 2023-07-02T14:05:38.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998, 2023-07-02T14:05:38.042 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.042 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:38.042 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.042 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:38.042 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:38.042 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20 2023-07-02T14:05:38.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.043 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:38.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:38.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:38.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:38.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.732e-06 2023-07-02T14:05:38.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198537+0000", 2023-07-02T14:05:38.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5259999999999999e-06 2023-07-02T14:05:38.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198543+0000", 2023-07-02T14:05:38.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.121e-06 2023-07-02T14:05:38.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198547+0000", 2023-07-02T14:05:38.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9591000000000005e-05 2023-07-02T14:05:38.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:05:38.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3120000000000004e-06 2023-07-02T14:05:38.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:38.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198621+0000", 2023-07-02T14:05:38.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92020789400000003 2023-07-02T14:05:38.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118829+0000", 2023-07-02T14:05:38.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022393386000000001 2023-07-02T14:05:38.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141223+0000", 2023-07-02T14:05:38.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000329866 2023-07-02T14:05:38.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141552+0000", 2023-07-02T14:05:38.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998 2023-07-02T14:05:38.050 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.050 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.050 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.050 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:21 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 196608~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:38.050 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:38.051 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.716470243, 2023-07-02T14:05:38.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995, 2023-07-02T14:05:38.051 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.051 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:38.051 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.051 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:38.051 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:38.051 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21 2023-07-02T14:05:38.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.052 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:38.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:38.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:38.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:38.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0169999999999999e-05 2023-07-02T14:05:38.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198571+0000", 2023-07-02T14:05:38.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3479999999999998e-06 2023-07-02T14:05:38.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198578+0000", 2023-07-02T14:05:38.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8439999999999997e-06 2023-07-02T14:05:38.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198583+0000", 2023-07-02T14:05:38.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8294999999999997e-05 2023-07-02T14:05:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198641+0000", 2023-07-02T14:05:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8120000000000001e-06 2023-07-02T14:05:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:38.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:05:38.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94293828499999999 2023-07-02T14:05:38.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141583+0000", 2023-07-02T14:05:38.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014607300000000001 2023-07-02T14:05:38.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141729+0000", 2023-07-02T14:05:38.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00044906600000000003 2023-07-02T14:05:38.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142178+0000", 2023-07-02T14:05:38.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995 2023-07-02T14:05:38.059 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.059 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.059 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.059 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:22 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 327680~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:38.060 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:38.060 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.716436813000001, 2023-07-02T14:05:38.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004, 2023-07-02T14:05:38.060 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.060 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:38.060 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.060 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:38.060 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:38.061 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 22 2023-07-02T14:05:38.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.061 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:38.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:38.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:38.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:38.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4850000000000001e-06 2023-07-02T14:05:38.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198597+0000", 2023-07-02T14:05:38.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:05:38.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198605+0000", 2023-07-02T14:05:38.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4839999999999998e-06 2023-07-02T14:05:38.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198609+0000", 2023-07-02T14:05:38.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3100999999999999e-05 2023-07-02T14:05:38.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198662+0000", 2023-07-02T14:05:38.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9099999999999998e-06 2023-07-02T14:05:38.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:38.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198666+0000", 2023-07-02T14:05:38.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.943539503 2023-07-02T14:05:38.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142206+0000", 2023-07-02T14:05:38.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012255999999999999 2023-07-02T14:05:38.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142328+0000", 2023-07-02T14:05:38.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208956 2023-07-02T14:05:38.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142537+0000", 2023-07-02T14:05:38.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004 2023-07-02T14:05:38.068 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.068 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.068 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.069 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:23 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 393216~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:38.069 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:38.069 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.716416833, 2023-07-02T14:05:38.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348, 2023-07-02T14:05:38.069 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.069 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:38.069 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.069 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:38.070 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:38.070 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23 2023-07-02T14:05:38.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.070 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:38.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:38.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:38.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.071 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:38.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8439999999999998e-06 2023-07-02T14:05:38.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:05:38.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0440000000000001e-06 2023-07-02T14:05:38.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198623+0000", 2023-07-02T14:05:38.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4480000000000002e-06 2023-07-02T14:05:38.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198628+0000", 2023-07-02T14:05:38.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5843e-05 2023-07-02T14:05:38.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:38.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.89e-06 2023-07-02T14:05:38.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:38.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198688+0000", 2023-07-02T14:05:38.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94389171999999999 2023-07-02T14:05:38.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142580+0000", 2023-07-02T14:05:38.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3892000000000004e-05 2023-07-02T14:05:38.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142644+0000", 2023-07-02T14:05:38.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017608000000000001 2023-07-02T14:05:38.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142820+0000", 2023-07-02T14:05:38.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348 2023-07-02T14:05:38.077 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.077 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.077 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.078 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:24 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 458752~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:38.078 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:05:38.078 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.716388759000001, 2023-07-02T14:05:38.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004, 2023-07-02T14:05:38.078 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.078 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:38.078 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.079 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:38.079 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:38.079 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 24 2023-07-02T14:05:38.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.079 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:05:38.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:05:38.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:38.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198642+0000", 2023-07-02T14:05:38.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.253e-06 2023-07-02T14:05:38.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:05:38.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1289999999999998e-06 2023-07-02T14:05:38.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198652+0000", 2023-07-02T14:05:38.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4510000000000002e-06 2023-07-02T14:05:38.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198657+0000", 2023-07-02T14:05:38.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9131000000000001e-05 2023-07-02T14:05:38.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198706+0000", 2023-07-02T14:05:38.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8890000000000004e-06 2023-07-02T14:05:38.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:38.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198710+0000", 2023-07-02T14:05:38.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94415412200000004 2023-07-02T14:05:38.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142864+0000", 2023-07-02T14:05:38.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8047000000000006e-05 2023-07-02T14:05:38.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142952+0000", 2023-07-02T14:05:38.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192967 2023-07-02T14:05:38.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143145+0000", 2023-07-02T14:05:38.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004 2023-07-02T14:05:38.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.086 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:25 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 524288~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.716369963999998, 2023-07-02T14:05:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001, 2023-07-02T14:05:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.088 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:38.088 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:38.088 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25 2023-07-02T14:05:38.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.088 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:38.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:38.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:38.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:38.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.615e-06 2023-07-02T14:05:38.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198664+0000", 2023-07-02T14:05:38.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4759999999999997e-06 2023-07-02T14:05:38.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198669+0000", 2023-07-02T14:05:38.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5999999999999998e-06 2023-07-02T14:05:38.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198673+0000", 2023-07-02T14:05:38.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4256999999999999e-05 2023-07-02T14:05:38.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198727+0000", 2023-07-02T14:05:38.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840000000000001e-06 2023-07-02T14:05:38.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:38.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198731+0000", 2023-07-02T14:05:38.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94444241900000003 2023-07-02T14:05:38.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143173+0000", 2023-07-02T14:05:38.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6839000000000002e-05 2023-07-02T14:05:38.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143240+0000", 2023-07-02T14:05:38.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017053399999999999 2023-07-02T14:05:38.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143411+0000", 2023-07-02T14:05:38.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001 2023-07-02T14:05:38.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.095 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.095 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.096 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:26 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 589824~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:38.096 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:38.096 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.716346813000001, 2023-07-02T14:05:38.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997, 2023-07-02T14:05:38.096 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.096 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:38.097 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.097 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:38.097 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:38.097 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 26 2023-07-02T14:05:38.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.097 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:38.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:38.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:38.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:38.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0520000000000002e-06 2023-07-02T14:05:38.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198687+0000", 2023-07-02T14:05:38.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1880000000000002e-06 2023-07-02T14:05:38.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198695+0000", 2023-07-02T14:05:38.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9960000000000004e-06 2023-07-02T14:05:38.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198699+0000", 2023-07-02T14:05:38.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2330999999999998e-05 2023-07-02T14:05:38.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198742+0000", 2023-07-02T14:05:38.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7950000000000001e-06 2023-07-02T14:05:38.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:38.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198745+0000", 2023-07-02T14:05:38.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94469398199999999 2023-07-02T14:05:38.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143439+0000", 2023-07-02T14:05:38.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6048e-05 2023-07-02T14:05:38.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143525+0000", 2023-07-02T14:05:38.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017016700000000001 2023-07-02T14:05:38.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143696+0000", 2023-07-02T14:05:38.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997 2023-07-02T14:05:38.104 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.104 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.105 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.105 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.105 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.105 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:38.106 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:05:38.107 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:05:38.107 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:38.106+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:38.207 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:38.207+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:38.218 DEBUG:teuthology.orchestra.run.smithi016:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op apply-layout-settings --pool rbd 2023-07-02T14:05:38.278 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:38.278 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:38.278 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:05:38.279 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:38.293 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:05:38.334 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:38.334+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:38.408 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:38.408+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:38.459 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:38.459 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:38.459 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:05:38.459 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:05:38.459 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:38.469 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:05:38.509 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:38.509+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:38.594 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:38.595 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:05:38.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:05:38.595 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:38.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.595 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:05:38.595 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:38.595 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 157.48531056100001, 2023-07-02T14:05:38.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:05:38.596 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.596 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:38.596 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.596 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:38.596 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:38.596 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:05:38.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.597 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:38.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:38.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:38.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:05:38.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:05:38.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:05:38.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:05:38.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:05:38.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:05:38.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:05:38.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:05:38.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:05:38.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:05:38.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:38.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:05:38.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:05:38.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:05:38.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:05:38.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:05:38.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:05:38.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:05:38.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:05:38.604 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.604 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.604 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.604 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:05:38.605 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:38.605 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 157.485249823, 2023-07-02T14:05:38.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:05:38.605 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.605 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:38.605 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.605 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:38.605 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:38.606 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:05:38.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.606 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:38.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:38.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:38.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:05:38.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:05:38.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:05:38.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:05:38.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:05:38.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:05:38.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:05:38.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:05:38.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:05:38.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:05:38.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:38.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:05:38.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:05:38.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:05:38.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:05:38.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:05:38.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:05:38.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:38.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:05:38.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:05:38.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:38.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:05:38.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:05:38.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:38.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:05:38.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:05:38.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:38.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:05:38.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:05:38.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:05:38.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:05:38.615 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.615 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.616 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.616 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:38.616 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:38.616 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 147.25479549299999, 2023-07-02T14:05:38.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:05:38.617 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.617 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:38.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:38.617 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:38.617 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:05:38.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.617 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:38.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:38.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:38.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:05:38.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:05:38.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:05:38.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:05:38.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:05:38.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:05:38.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:05:38.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:05:38.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:05:38.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:05:38.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:05:38.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:05:38.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:38.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:05:38.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:05:38.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:38.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:05:38.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:05:38.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:38.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:05:38.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:05:38.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:38.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:05:38.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:05:38.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:05:38.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:05:38.626 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.626 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.626 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.626 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2950 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 1507328~786432] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:38.627 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:38.627 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 147.11751923099999, 2023-07-02T14:05:38.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999, 2023-07-02T14:05:38.627 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.627 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:38.627 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.627 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:38.627 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:38.628 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2950 2023-07-02T14:05:38.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.628 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:38.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:38.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:38.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:38.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001147854 2023-07-02T14:05:38.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476614+0000", 2023-07-02T14:05:38.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.802e-05 2023-07-02T14:05:38.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476632+0000", 2023-07-02T14:05:38.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1860000000000003e-06 2023-07-02T14:05:38.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476639+0000", 2023-07-02T14:05:38.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061211240000000004 2023-07-02T14:05:38.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482760+0000", 2023-07-02T14:05:38.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8833999999999996e-05 2023-07-02T14:05:38.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482859+0000", 2023-07-02T14:05:38.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019270600000000001 2023-07-02T14:05:38.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:38.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.483051+0000", 2023-07-02T14:05:38.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060385630000000003 2023-07-02T14:05:38.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:38.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489090+0000", 2023-07-02T14:05:38.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024506918999999999 2023-07-02T14:05:38.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:38.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513597+0000", 2023-07-02T14:05:38.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9672999999999995e-05 2023-07-02T14:05:38.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:38.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513667+0000", 2023-07-02T14:05:38.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024329500000000001 2023-07-02T14:05:38.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513910+0000", 2023-07-02T14:05:38.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999 2023-07-02T14:05:38.636 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.636 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.636 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.637 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2951 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2686976~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:38.637 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:05:38.637 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 147.116338453, 2023-07-02T14:05:38.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999, 2023-07-02T14:05:38.637 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.637 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:38.637 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.638 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:38.638 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:38.638 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2951 2023-07-02T14:05:38.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.638 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:05:38.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476647+0000", 2023-07-02T14:05:38.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:38.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476646+0000", 2023-07-02T14:05:38.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7639000000000003e-05 2023-07-02T14:05:38.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476684+0000", 2023-07-02T14:05:38.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3508000000000002e-05 2023-07-02T14:05:38.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476747+0000", 2023-07-02T14:05:38.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8499999999999994e-06 2023-07-02T14:05:38.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476755+0000", 2023-07-02T14:05:38.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078346160000000008 2023-07-02T14:05:38.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484590+0000", 2023-07-02T14:05:38.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7206000000000004e-05 2023-07-02T14:05:38.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484687+0000", 2023-07-02T14:05:38.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016243099999999999 2023-07-02T14:05:38.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:38.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.484849+0000", 2023-07-02T14:05:38.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042482860000000004 2023-07-02T14:05:38.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:38.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489097+0000", 2023-07-02T14:05:38.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.02489504 2023-07-02T14:05:38.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:38.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513992+0000", 2023-07-02T14:05:38.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9972000000000001e-05 2023-07-02T14:05:38.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:38.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514052+0000", 2023-07-02T14:05:38.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149264 2023-07-02T14:05:38.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514202+0000", 2023-07-02T14:05:38.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037554628999999999 2023-07-02T14:05:38.646 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.646 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.647 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.647 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2952 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2883584~12288] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:38.647 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:38.647 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 147.11622461799999, 2023-07-02T14:05:38.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651, 2023-07-02T14:05:38.648 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.648 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:38.648 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.648 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:38.648 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:38.648 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2952 2023-07-02T14:05:38.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.648 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:38.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:38.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:38.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476760+0000", 2023-07-02T14:05:38.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3664e-05 2023-07-02T14:05:38.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476774+0000", 2023-07-02T14:05:38.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.166e-06 2023-07-02T14:05:38.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476783+0000", 2023-07-02T14:05:38.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8720000000000001e-06 2023-07-02T14:05:38.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476788+0000", 2023-07-02T14:05:38.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087180529999999999 2023-07-02T14:05:38.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485506+0000", 2023-07-02T14:05:38.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5680999999999994e-05 2023-07-02T14:05:38.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485602+0000", 2023-07-02T14:05:38.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000222365 2023-07-02T14:05:38.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:38.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485824+0000", 2023-07-02T14:05:38.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003277379 2023-07-02T14:05:38.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:38.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489101+0000", 2023-07-02T14:05:38.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025151739999999999 2023-07-02T14:05:38.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:38.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514253+0000", 2023-07-02T14:05:38.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4749000000000002e-05 2023-07-02T14:05:38.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:38.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514288+0000", 2023-07-02T14:05:38.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130634 2023-07-02T14:05:38.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514419+0000", 2023-07-02T14:05:38.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651 2023-07-02T14:05:38.657 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.657 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.657 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.657 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9620 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 589824~65536] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:05:38.657 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:38.658 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.570780894999999, 2023-07-02T14:05:38.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001, 2023-07-02T14:05:38.658 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.658 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:38.658 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.658 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:38.658 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:38.658 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9620 2023-07-02T14:05:38.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.659 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:38.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:38.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:38.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022203+0000", 2023-07-02T14:05:38.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0346000000000001e-05 2023-07-02T14:05:38.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022263+0000", 2023-07-02T14:05:38.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4657999999999999e-05 2023-07-02T14:05:38.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022278+0000", 2023-07-02T14:05:38.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:05:38.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022285+0000", 2023-07-02T14:05:38.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0096000000000003e-05 2023-07-02T14:05:38.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022335+0000", 2023-07-02T14:05:38.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2334000000000001e-05 2023-07-02T14:05:38.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:38.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022347+0000", 2023-07-02T14:05:38.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095653701999999993 2023-07-02T14:05:38.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118001+0000", 2023-07-02T14:05:38.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000185797 2023-07-02T14:05:38.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118187+0000", 2023-07-02T14:05:38.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026397900000000001 2023-07-02T14:05:38.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:38.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118451+0000", 2023-07-02T14:05:38.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051832800000000002 2023-07-02T14:05:38.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:38.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.123634+0000", 2023-07-02T14:05:38.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001227332 2023-07-02T14:05:38.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:38.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124861+0000", 2023-07-02T14:05:38.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9493999999999998e-05 2023-07-02T14:05:38.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:38.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124901+0000", 2023-07-02T14:05:38.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0462000000000001e-05 2023-07-02T14:05:38.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124931+0000", 2023-07-02T14:05:38.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001 2023-07-02T14:05:38.668 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.668 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.669 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.669 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9621 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 655360~1966080] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:05:38.669 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:38.669 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.570593513000006, 2023-07-02T14:05:38.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999, 2023-07-02T14:05:38.669 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.670 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:38.670 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.670 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:38.670 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:38.670 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9621 2023-07-02T14:05:38.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.670 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:38.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:38.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:38.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022391+0000", 2023-07-02T14:05:38.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032053939999999999 2023-07-02T14:05:38.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025597+0000", 2023-07-02T14:05:38.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4307999999999999e-05 2023-07-02T14:05:38.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025611+0000", 2023-07-02T14:05:38.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.032e-06 2023-07-02T14:05:38.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025617+0000", 2023-07-02T14:05:38.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8977999999999998e-05 2023-07-02T14:05:38.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025656+0000", 2023-07-02T14:05:38.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:05:38.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:38.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025663+0000", 2023-07-02T14:05:38.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095220739999999998 2023-07-02T14:05:38.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120884+0000", 2023-07-02T14:05:38.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2525000000000001e-05 2023-07-02T14:05:38.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120956+0000", 2023-07-02T14:05:38.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016232400000000001 2023-07-02T14:05:38.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:38.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.121119+0000", 2023-07-02T14:05:38.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044007380000000004 2023-07-02T14:05:38.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:38.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.125519+0000", 2023-07-02T14:05:38.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000797209 2023-07-02T14:05:38.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:38.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126317+0000", 2023-07-02T14:05:38.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1228000000000004e-05 2023-07-02T14:05:38.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:38.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126378+0000", 2023-07-02T14:05:38.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978000000000003e-05 2023-07-02T14:05:38.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126430+0000", 2023-07-02T14:05:38.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999 2023-07-02T14:05:38.680 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.680 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.680 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.681 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:11434 2.1 2:8269d003:::rbd_data.125de1449605.00000000000000ce:head [write 65536~4128768] snapc 1a=[1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e107)", 2023-07-02T14:05:38.681 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:05:38.681 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 69.675155243999995, 2023-07-02T14:05:38.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036409966000000002, 2023-07-02T14:05:38.681 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.681 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:38.681 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.682 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:38.682 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:38.682 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11434 2023-07-02T14:05:38.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.682 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:05:38.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917830+0000", 2023-07-02T14:05:38.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:38.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.917828+0000", 2023-07-02T14:05:38.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0052044650000000001 2023-07-02T14:05:38.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923033+0000", 2023-07-02T14:05:38.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0097000000000001e-05 2023-07-02T14:05:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923053+0000", 2023-07-02T14:05:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.49e-06 2023-07-02T14:05:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923061+0000", 2023-07-02T14:05:38.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0090999999999998e-05 2023-07-02T14:05:38.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923121+0000", 2023-07-02T14:05:38.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000114095 2023-07-02T14:05:38.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923236+0000", 2023-07-02T14:05:38.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020094199999999999 2023-07-02T14:05:38.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:38.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.923436+0000", 2023-07-02T14:05:38.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088693520000000005 2023-07-02T14:05:38.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:38.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.932306+0000", 2023-07-02T14:05:38.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021874112000000001 2023-07-02T14:05:38.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:38.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954180+0000", 2023-07-02T14:05:38.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4433e-05 2023-07-02T14:05:38.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:38.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954214+0000", 2023-07-02T14:05:38.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5884000000000002e-05 2023-07-02T14:05:38.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:28.954240+0000", 2023-07-02T14:05:38.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.036409966000000002 2023-07-02T14:05:38.690 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.690 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.691 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.691 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:189 2.3 2:c9621172:::rbd_data.125de1449605.00000000000000c0:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:38.691 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:38.691 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.420306221000004, 2023-07-02T14:05:38.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001, 2023-07-02T14:05:38.691 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.692 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:38.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:38.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:38.692 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:05:38.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.692 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:38.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:38.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:38.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:38.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2023-07-02T14:05:38.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172682+0000", 2023-07-02T14:05:38.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0159999999999997e-06 2023-07-02T14:05:38.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172688+0000", 2023-07-02T14:05:38.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0010000000000003e-06 2023-07-02T14:05:38.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172692+0000", 2023-07-02T14:05:38.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026444333 2023-07-02T14:05:38.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199136+0000", 2023-07-02T14:05:38.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2230000000000004e-05 2023-07-02T14:05:38.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199218+0000", 2023-07-02T14:05:38.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012043595000000001 2023-07-02T14:05:38.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211262+0000", 2023-07-02T14:05:38.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001 2023-07-02T14:05:38.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.698 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.698 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.699 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:199 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:38.699 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:38.699 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.420262678, 2023-07-02T14:05:38.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002, 2023-07-02T14:05:38.699 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.699 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:38.699 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.700 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:38.700 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:38.700 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 199 2023-07-02T14:05:38.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.700 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:38.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:38.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:38.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172722+0000", 2023-07-02T14:05:38.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1250000000000001e-06 2023-07-02T14:05:38.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172725+0000", 2023-07-02T14:05:38.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5670000000000002e-06 2023-07-02T14:05:38.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172731+0000", 2023-07-02T14:05:38.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9310000000000001e-06 2023-07-02T14:05:38.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172735+0000", 2023-07-02T14:05:38.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030699852999999999 2023-07-02T14:05:38.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203435+0000", 2023-07-02T14:05:38.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6827e-05 2023-07-02T14:05:38.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203532+0000", 2023-07-02T14:05:38.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076955230000000001 2023-07-02T14:05:38.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211227+0000", 2023-07-02T14:05:38.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002 2023-07-02T14:05:38.706 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.706 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.706 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.706 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:281 2.1 2:901d5a95:::rbd_data.125de1449605.0000000000000118:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:38.707 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:38.707 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 65.037517402000006, 2023-07-02T14:05:38.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002, 2023-07-02T14:05:38.707 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.707 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:38.707 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.707 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:38.707 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:38.708 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:05:38.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.708 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:38.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:38.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:38.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:38.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0469999999999998e-06 2023-07-02T14:05:38.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555471+0000", 2023-07-02T14:05:38.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8860000000000001e-06 2023-07-02T14:05:38.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555476+0000", 2023-07-02T14:05:38.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3160000000000002e-06 2023-07-02T14:05:38.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555483+0000", 2023-07-02T14:05:38.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989525000000001 2023-07-02T14:05:38.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591472+0000", 2023-07-02T14:05:38.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3708999999999996e-05 2023-07-02T14:05:38.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591536+0000", 2023-07-02T14:05:38.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055109720000000003 2023-07-02T14:05:38.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.597047+0000", 2023-07-02T14:05:38.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002 2023-07-02T14:05:38.714 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.714 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.714 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.714 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12883 2.1 2:9ffc48e1:::rbd_data.125de1449605.0000000000000074:head [write 4173824~20480] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:05:38.714 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:38.714 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.946914929000002, 2023-07-02T14:05:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037871393000000003, 2023-07-02T14:05:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12883 2023-07-02T14:05:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:05:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646068+0000", 2023-07-02T14:05:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4935999999999999e-05 2023-07-02T14:05:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646103+0000", 2023-07-02T14:05:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.092e-05 2023-07-02T14:05:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646124+0000", 2023-07-02T14:05:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1519999999999998e-06 2023-07-02T14:05:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646133+0000", 2023-07-02T14:05:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012270200000000001 2023-07-02T14:05:38.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646255+0000", 2023-07-02T14:05:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014495400000000001 2023-07-02T14:05:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646400+0000", 2023-07-02T14:05:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000221626 2023-07-02T14:05:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646622+0000", 2023-07-02T14:05:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078929050000000004 2023-07-02T14:05:38.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.654515+0000", 2023-07-02T14:05:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029307343999999999 2023-07-02T14:05:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:38.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683822+0000", 2023-07-02T14:05:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8298000000000002e-05 2023-07-02T14:05:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683891+0000", 2023-07-02T14:05:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1381999999999997e-05 2023-07-02T14:05:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683942+0000", 2023-07-02T14:05:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037871393000000003 2023-07-02T14:05:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12985 2.7 2:eeddf24b:::rbd_data.125de1449605.000000000000007e:head [write 65536~4128768] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:05:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.809990456000001, 2023-07-02T14:05:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999, 2023-07-02T14:05:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.725 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12985 2023-07-02T14:05:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782994+0000", 2023-07-02T14:05:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019254741999999998 2023-07-02T14:05:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802248+0000", 2023-07-02T14:05:38.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0451e-05 2023-07-02T14:05:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802279+0000", 2023-07-02T14:05:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3318e-05 2023-07-02T14:05:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802292+0000", 2023-07-02T14:05:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109403 2023-07-02T14:05:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802402+0000", 2023-07-02T14:05:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012520599999999999 2023-07-02T14:05:38.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802527+0000", 2023-07-02T14:05:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187768 2023-07-02T14:05:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:05:38.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802715+0000", 2023-07-02T14:05:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013192822 2023-07-02T14:05:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.815907+0000", 2023-07-02T14:05:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0086451870000000004 2023-07-02T14:05:38.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824553+0000", 2023-07-02T14:05:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6759999999999999e-05 2023-07-02T14:05:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:38.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824609+0000", 2023-07-02T14:05:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011314700000000001 2023-07-02T14:05:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824722+0000", 2023-07-02T14:05:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999 2023-07-02T14:05:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.734 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:165 2.3 2:c9c6393a:::rbd_data.125de1449605.000000000000002c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.753207639000003, 2023-07-02T14:05:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002, 2023-07-02T14:05:38.735 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 165 2023-07-02T14:05:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:38.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839777+0000", 2023-07-02T14:05:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6740000000000001e-06 2023-07-02T14:05:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839780+0000", 2023-07-02T14:05:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9030000000000004e-06 2023-07-02T14:05:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839788+0000", 2023-07-02T14:05:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1799999999999998e-06 2023-07-02T14:05:38.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839792+0000", 2023-07-02T14:05:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.031592227000000001 2023-07-02T14:05:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871384+0000", 2023-07-02T14:05:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013600099999999999 2023-07-02T14:05:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871520+0000", 2023-07-02T14:05:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082817089999999999 2023-07-02T14:05:38.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.879802+0000", 2023-07-02T14:05:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002 2023-07-02T14:05:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:252 2.7 2:f1d0abf0:::rbd_data.125de1449605.0000000000000083:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.336922669000003, 2023-07-02T14:05:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997, 2023-07-02T14:05:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:38.743 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 252 2023-07-02T14:05:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256062+0000", 2023-07-02T14:05:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0149999999999999e-06 2023-07-02T14:05:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256065+0000", 2023-07-02T14:05:38.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5269999999999998e-06 2023-07-02T14:05:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256071+0000", 2023-07-02T14:05:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:05:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256074+0000", 2023-07-02T14:05:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028852909 2023-07-02T14:05:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.284927+0000", 2023-07-02T14:05:38.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168274 2023-07-02T14:05:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.285096+0000", 2023-07-02T14:05:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011036300000000001 2023-07-02T14:05:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296132+0000", 2023-07-02T14:05:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997 2023-07-02T14:05:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.750 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:261 2.1 2:971731cb:::rbd_data.125de1449605.000000000000008c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.336827194000001, 2023-07-02T14:05:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984, 2023-07-02T14:05:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.751 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 261 2023-07-02T14:05:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:38.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2819999999999999e-06 2023-07-02T14:05:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256160+0000", 2023-07-02T14:05:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2020000000000003e-06 2023-07-02T14:05:38.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256165+0000", 2023-07-02T14:05:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2100000000000003e-06 2023-07-02T14:05:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256170+0000", 2023-07-02T14:05:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025871715999999999 2023-07-02T14:05:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282041+0000", 2023-07-02T14:05:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132364 2023-07-02T14:05:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282174+0000", 2023-07-02T14:05:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014044566 2023-07-02T14:05:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296218+0000", 2023-07-02T14:05:38.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984 2023-07-02T14:05:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:264 2.3 2:dbf06c9d:::rbd_data.125de1449605.000000000000008f:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:38.758 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.336793075999999, 2023-07-02T14:05:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003, 2023-07-02T14:05:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:38.759 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 264 2023-07-02T14:05:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:38.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5749999999999999e-06 2023-07-02T14:05:38.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256195+0000", 2023-07-02T14:05:38.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.327e-06 2023-07-02T14:05:38.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256200+0000", 2023-07-02T14:05:38.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0580000000000002e-06 2023-07-02T14:05:38.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256203+0000", 2023-07-02T14:05:38.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035707524999999997 2023-07-02T14:05:38.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.291911+0000", 2023-07-02T14:05:38.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014269799999999999 2023-07-02T14:05:38.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.292053+0000", 2023-07-02T14:05:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062760309999999996 2023-07-02T14:05:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.298329+0000", 2023-07-02T14:05:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003 2023-07-02T14:05:38.765 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:341 2.3 2:cee258bd:::rbd_data.125de1449605.00000000000000dc:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 40.956305749000002, 2023-07-02T14:05:38.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997, 2023-07-02T14:05:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 341 2023-07-02T14:05:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.767 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636679+0000", 2023-07-02T14:05:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6240000000000002e-06 2023-07-02T14:05:38.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636682+0000", 2023-07-02T14:05:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6910000000000002e-06 2023-07-02T14:05:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636688+0000", 2023-07-02T14:05:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3469999999999999e-06 2023-07-02T14:05:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636691+0000", 2023-07-02T14:05:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035625560000000001 2023-07-02T14:05:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672317+0000", 2023-07-02T14:05:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108391 2023-07-02T14:05:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:38.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672425+0000", 2023-07-02T14:05:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012104976 2023-07-02T14:05:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.684530+0000", 2023-07-02T14:05:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997 2023-07-02T14:05:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.773 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPushReply(2.7 139/134 [PushReplyOp(2:e3782f39:::rbd_data.125de1449605.0000000000000120:head)])", 2023-07-02T14:05:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:05:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.665773596000001, 2023-07-02T14:05:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998, 2023-07-02T14:05:38.774 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:05:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:05:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927211+0000", 2023-07-02T14:05:38.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3689999999999998e-06 2023-07-02T14:05:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927217+0000", 2023-07-02T14:05:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7269999999999996e-06 2023-07-02T14:05:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927225+0000", 2023-07-02T14:05:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:05:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927230+0000", 2023-07-02T14:05:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:05:38.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927266+0000", 2023-07-02T14:05:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.069972871000000006 2023-07-02T14:05:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:38.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.997239+0000", 2023-07-02T14:05:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998 2023-07-02T14:05:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:38.780 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:38.780 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:38.782 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:05:38.782 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:38.781+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:38.882 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:38.881+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:38.916 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:38.916 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:38.916 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:38.916 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:38.926 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:05:39.009 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:39.008+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:39.084 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:39.084+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:39.089 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:39.090 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:39.100 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:05:39.102 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:05:39.133 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap40 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap41 b3fbb028-e267-4d25-a750-93ba18a2316a@snap41.diff 2023-07-02T14:05:39.185 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:39.185+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:39.193 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 51% complete...2023-07-02T14:05:39.192+0000 7fc2257a9700 0 -- 172.21.15.163:0/978436059 >> [v2:172.21.15.22:6808/99977,v1:172.21.15.22:6809/99977] conn(0x55770d114d70 msgr2=0x55770d117160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:39.195 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 100% complete...done. 2023-07-02T14:05:39.198 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:05:39.198 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=42 2023-07-02T14:05:39.198 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap42 2023-07-02T14:05:39.269 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:39.269 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:39.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:39.270 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:39.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.270 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:05:39.270 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:39.270 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.370338617, 2023-07-02T14:05:39.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:05:39.270 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.271 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.271 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.271 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:39.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:39.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:39.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:05:39.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:05:39.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.273 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:05:39.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:05:39.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:05:39.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:05:39.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:05:39.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:05:39.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.275 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.275 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:05:39.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:05:39.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:05:39.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:05:39.276 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:05:39.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:05:39.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.277 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.277 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:05:39.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:05:39.277 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.277 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.278 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.278 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:05:39.278 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:39.278 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.366252899, 2023-07-02T14:05:39.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:05:39.278 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.279 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.279 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.279 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:39.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:39.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:39.280 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:05:39.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:05:39.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:05:39.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:05:39.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:05:39.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:05:39.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.282 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:05:39.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:05:39.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.283 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:05:39.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:05:39.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:05:39.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:05:39.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:05:39.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:05:39.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:05:39.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:05:39.285 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.285 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.285 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.286 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:05:39.286 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:39.286 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.362388118, 2023-07-02T14:05:39.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:05:39.286 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.286 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.287 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:39.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:39.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:39.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:05:39.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:05:39.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:05:39.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:05:39.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:05:39.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:05:39.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:05:39.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:05:39.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:05:39.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:05:39.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:05:39.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:05:39.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:05:39.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:05:39.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:05:39.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:05:39.293 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.293 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.293 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.294 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:05:39.294 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:39.294 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.358863857, 2023-07-02T14:05:39.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:05:39.294 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.294 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.294 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.295 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:39.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.295 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:39.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:39.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:05:39.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:05:39.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:05:39.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:05:39.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:05:39.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:05:39.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:05:39.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:05:39.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:05:39.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:05:39.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:05:39.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:05:39.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:05:39.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:05:39.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:05:39.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:05:39.301 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.301 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.301 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.301 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:05:39.302 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:39.302 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.323159058, 2023-07-02T14:05:39.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:05:39.302 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.302 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.302 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:39.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:39.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:39.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:05:39.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:05:39.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:05:39.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:05:39.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:05:39.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:05:39.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:05:39.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:05:39.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:05:39.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:05:39.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.307 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:05:39.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:05:39.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:05:39.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:05:39.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:05:39.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:05:39.309 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.309 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.309 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.309 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:05:39.309 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:39.309 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.3185017, 2023-07-02T14:05:39.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:05:39.310 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.310 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.310 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:39.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:39.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:39.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:05:39.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:05:39.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:05:39.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:05:39.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:05:39.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:05:39.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:05:39.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:05:39.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:05:39.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:05:39.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:05:39.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:05:39.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:05:39.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:05:39.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:05:39.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:05:39.317 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.317 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.317 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.317 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:05:39.317 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:39.317 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.317561837, 2023-07-02T14:05:39.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:05:39.318 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.318 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.318 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:39.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:39.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:39.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:05:39.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:05:39.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:05:39.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:05:39.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:05:39.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:05:39.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:05:39.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:05:39.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.322 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:05:39.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:05:39.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:05:39.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:05:39.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:05:39.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:05:39.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.324 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:05:39.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:05:39.324 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.325 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.325 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.325 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:05:39.325 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:39.325 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.31616395899999, 2023-07-02T14:05:39.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:05:39.326 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.326 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.326 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:39.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:39.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:39.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:05:39.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:05:39.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:05:39.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:05:39.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:05:39.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:05:39.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:05:39.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:05:39.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:05:39.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:05:39.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:05:39.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:05:39.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:05:39.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:05:39.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:05:39.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:05:39.332 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.332 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.332 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.333 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:05:39.333 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:39.333 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 97.905973462999995, 2023-07-02T14:05:39.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:05:39.333 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.333 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.334 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:39.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:39.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:39.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:05:39.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:05:39.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:05:39.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:05:39.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:05:39.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:05:39.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:05:39.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:05:39.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:05:39.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:05:39.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:05:39.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:05:39.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:05:39.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:05:39.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:05:39.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:05:39.340 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.340 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.340 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.341 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:39.341 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:39.341 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 96.539623778999996, 2023-07-02T14:05:39.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:05:39.341 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.341 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.341 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:39.342 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:39.342 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:39.342 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:05:39.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.342 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:39.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:39.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:39.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:05:39.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:05:39.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:05:39.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:05:39.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:05:39.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:05:39.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:05:39.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:05:39.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:05:39.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:05:39.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:05:39.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:05:39.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:39.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:05:39.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:05:39.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:05:39.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:05:39.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:05:39.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:05:39.349 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.349 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.349 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.350 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7707 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 3145728~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:39.350 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:39.350 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 96.535640311999998, 2023-07-02T14:05:39.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999, 2023-07-02T14:05:39.350 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.350 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.351 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:39.351 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:39.351 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:39.351 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7707 2023-07-02T14:05:39.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.351 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:39.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:39.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:39.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:39.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5017e-05 2023-07-02T14:05:39.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732347+0000", 2023-07-02T14:05:39.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5821e-05 2023-07-02T14:05:39.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732363+0000", 2023-07-02T14:05:39.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1270000000000002e-06 2023-07-02T14:05:39.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732368+0000", 2023-07-02T14:05:39.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008444666 2023-07-02T14:05:39.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740812+0000", 2023-07-02T14:05:39.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011266599999999999 2023-07-02T14:05:39.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740925+0000", 2023-07-02T14:05:39.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010848808999999999 2023-07-02T14:05:39.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:39.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751774+0000", 2023-07-02T14:05:39.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8421999999999997e-05 2023-07-02T14:05:39.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751832+0000", 2023-07-02T14:05:39.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107453 2023-07-02T14:05:39.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751940+0000", 2023-07-02T14:05:39.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999 2023-07-02T14:05:39.358 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.358 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.358 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.359 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:39.359 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:39.359 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 96.535576434999996, 2023-07-02T14:05:39.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:05:39.359 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.360 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.360 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:39.360 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:39.360 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:39.360 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:05:39.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.360 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:39.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:39.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:39.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:39.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:05:39.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:05:39.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:05:39.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:05:39.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:05:39.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:05:39.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:05:39.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:05:39.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:05:39.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:05:39.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:05:39.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:39.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:05:39.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:05:39.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:05:39.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:05:39.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:05:39.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:05:39.367 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.367 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.368 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.368 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:39.368 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:39.368 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 96.531012610000005, 2023-07-02T14:05:39.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:05:39.369 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.369 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.369 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:39.369 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:39.369 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:39.369 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:05:39.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.369 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:39.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:39.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:39.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:05:39.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:05:39.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:05:39.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:05:39.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:05:39.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:05:39.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:05:39.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:05:39.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:05:39.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:05:39.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:05:39.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:05:39.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:39.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:05:39.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:05:39.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:05:39.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:05:39.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:05:39.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:05:39.377 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.377 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.377 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.377 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:39.377 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:39.377 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 96.530962744000007, 2023-07-02T14:05:39.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:05:39.378 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.378 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.378 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:39.378 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:39.378 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:39.378 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:05:39.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.379 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:39.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:39.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:39.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:05:39.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:05:39.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:05:39.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:05:39.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:05:39.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:05:39.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:05:39.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:05:39.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:05:39.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:05:39.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:05:39.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:05:39.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:39.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:05:39.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:05:39.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:05:39.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:05:39.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:05:39.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:05:39.386 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.386 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.386 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.386 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:39.386 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:39.386 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 96.529923750999998, 2023-07-02T14:05:39.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:05:39.387 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.387 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.387 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:39.387 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:39.387 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:39.387 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:05:39.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.388 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:39.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:39.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:39.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:05:39.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:05:39.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:05:39.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:05:39.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:05:39.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:05:39.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:05:39.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:05:39.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:05:39.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:05:39.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:05:39.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:05:39.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:39.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:05:39.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:05:39.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:05:39.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:05:39.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:05:39.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:05:39.395 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.395 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.395 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.395 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7993 2.7 2:fe72db53:::rbd_data.125de1449605.0000000000000072:head [write 65536~3276800] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:39.395 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:39.395 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 96.342965703000004, 2023-07-02T14:05:39.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919, 2023-07-02T14:05:39.396 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.396 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.396 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:39.396 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:39.396 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:39.396 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7993 2023-07-02T14:05:39.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.397 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:39.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:39.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:39.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924986+0000", 2023-07-02T14:05:39.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049542070000000004 2023-07-02T14:05:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929940+0000", 2023-07-02T14:05:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9740000000000001e-05 2023-07-02T14:05:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929960+0000", 2023-07-02T14:05:39.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.311e-05 2023-07-02T14:05:39.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929973+0000", 2023-07-02T14:05:39.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9629000000000002e-05 2023-07-02T14:05:39.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930023+0000", 2023-07-02T14:05:39.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012751799999999999 2023-07-02T14:05:39.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930150+0000", 2023-07-02T14:05:39.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016449042000000001 2023-07-02T14:05:39.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:39.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946599+0000", 2023-07-02T14:05:39.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3241e-05 2023-07-02T14:05:39.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946663+0000", 2023-07-02T14:05:39.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000102296 2023-07-02T14:05:39.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946765+0000", 2023-07-02T14:05:39.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919 2023-07-02T14:05:39.404 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.404 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.404 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.404 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11782 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 196608~2162688] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:05:39.404 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:39.405 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 54.858445781, 2023-07-02T14:05:39.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365, 2023-07-02T14:05:39.405 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.405 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.405 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:39.405 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:39.405 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:39.405 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11782 2023-07-02T14:05:39.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.406 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:39.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:39.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:39.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409506+0000", 2023-07-02T14:05:39.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020172269999999999 2023-07-02T14:05:39.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411523+0000", 2023-07-02T14:05:39.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8901000000000001e-05 2023-07-02T14:05:39.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411542+0000", 2023-07-02T14:05:39.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5037e-05 2023-07-02T14:05:39.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411557+0000", 2023-07-02T14:05:39.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001931213 2023-07-02T14:05:39.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413489+0000", 2023-07-02T14:05:39.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000149207 2023-07-02T14:05:39.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413638+0000", 2023-07-02T14:05:39.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024752800000000001 2023-07-02T14:05:39.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:39.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413885+0000", 2023-07-02T14:05:39.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048516000000000002 2023-07-02T14:05:39.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:39.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.418737+0000", 2023-07-02T14:05:39.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010319259000000001 2023-07-02T14:05:39.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:39.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429056+0000", 2023-07-02T14:05:39.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2518000000000002e-05 2023-07-02T14:05:39.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429099+0000", 2023-07-02T14:05:39.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015275 2023-07-02T14:05:39.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429251+0000", 2023-07-02T14:05:39.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365 2023-07-02T14:05:39.414 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.414 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.414 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.415 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPushReply(2.7 119/117 [PushReplyOp(2:eb862bc0:::rbd_data.125de1449605.0000000000000040:13)])", 2023-07-02T14:05:39.415 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:39.415 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 54.857282505999997, 2023-07-02T14:05:39.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999, 2023-07-02T14:05:39.415 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.415 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "reached pg", 2023-07-02T14:05:39.415 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:39.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:39.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:39.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:39.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7670000000000001e-06 2023-07-02T14:05:39.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410674+0000", 2023-07-02T14:05:39.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3973e-05 2023-07-02T14:05:39.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410688+0000", 2023-07-02T14:05:39.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0157e-05 2023-07-02T14:05:39.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410708+0000", 2023-07-02T14:05:39.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018578779 2023-07-02T14:05:39.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429287+0000", 2023-07-02T14:05:39.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013086549999999999 2023-07-02T14:05:39.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.430595+0000", 2023-07-02T14:05:39.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999 2023-07-02T14:05:39.421 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.421 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.421 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.421 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11861 2.7 2:e57d3f55:::rbd_data.125de1449605.0000000000000134:head [write 0~65536] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:05:39.421 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:39.422 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 54.723167128999997, 2023-07-02T14:05:39.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723, 2023-07-02T14:05:39.422 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.422 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.422 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:39.422 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:39.422 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:39.422 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11861 2023-07-02T14:05:39.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.423 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:39.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:39.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:39.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544785+0000", 2023-07-02T14:05:39.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010563700000000001 2023-07-02T14:05:39.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544890+0000", 2023-07-02T14:05:39.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8530000000000004e-06 2023-07-02T14:05:39.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544900+0000", 2023-07-02T14:05:39.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.665e-06 2023-07-02T14:05:39.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544906+0000", 2023-07-02T14:05:39.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000272575 2023-07-02T14:05:39.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545178+0000", 2023-07-02T14:05:39.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000111362 2023-07-02T14:05:39.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545290+0000", 2023-07-02T14:05:39.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016181100000000001 2023-07-02T14:05:39.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:39.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545452+0000", 2023-07-02T14:05:39.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044500750000000004 2023-07-02T14:05:39.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:39.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.549902+0000", 2023-07-02T14:05:39.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014850024 2023-07-02T14:05:39.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:39.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564752+0000", 2023-07-02T14:05:39.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6272000000000003e-05 2023-07-02T14:05:39.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564848+0000", 2023-07-02T14:05:39.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000162428 2023-07-02T14:05:39.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.565010+0000", 2023-07-02T14:05:39.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723 2023-07-02T14:05:39.431 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.431 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.431 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.432 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:13262 2.7 e125/121 2:e0741dde:::rbd_data.125de1449605.00000000000000ad:head v 125'2024, mlcod=125'2021)", 2023-07-02T14:05:39.432 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:39.432 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.017089390999999, 2023-07-02T14:05:39.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999, 2023-07-02T14:05:39.432 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.432 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.432 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:39.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:39.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:39.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250862+0000", 2023-07-02T14:05:39.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080378800000000007 2023-07-02T14:05:39.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258900+0000", 2023-07-02T14:05:39.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8939999999999999e-06 2023-07-02T14:05:39.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258909+0000", 2023-07-02T14:05:39.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3560000000000003e-06 2023-07-02T14:05:39.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258914+0000", 2023-07-02T14:05:39.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2429000000000003e-05 2023-07-02T14:05:39.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258956+0000", 2023-07-02T14:05:39.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5094999999999999e-05 2023-07-02T14:05:39.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258981+0000", 2023-07-02T14:05:39.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012177307 2023-07-02T14:05:39.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271158+0000", 2023-07-02T14:05:39.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2104999999999999e-05 2023-07-02T14:05:39.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271181+0000", 2023-07-02T14:05:39.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999 2023-07-02T14:05:39.439 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.439 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.439 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.439 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.439 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.439 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:39.441 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:05:39.441 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:39.439+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:39.541 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:39.541+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:39.606 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:39.606 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:39.607 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:39.607 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:39.616 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:05:39.659 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:39.659+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:39.740 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:39.740 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:39.741 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:39.740+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:39.750 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:05:39.841 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:39.841+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:39.912 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:05:39.915 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:39.915 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.623185753, 2023-07-02T14:05:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:05:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:05:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:05:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:05:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:05:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:05:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:05:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:05:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:05:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:05:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:05:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:05:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:05:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:05:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:05:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:05:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:05:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:05:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:05:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:05:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:05:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:05:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:05:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:05:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.62295073000001, 2023-07-02T14:05:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:05:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:05:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:05:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:05:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:05:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:05:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:05:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:05:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:05:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:05:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:05:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:05:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:05:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:05:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:05:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:05:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:05:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:05:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:05:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:05:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:05:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:05:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:05:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:05:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.622751596, 2023-07-02T14:05:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:05:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:05:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:05:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:05:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:05:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:05:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:05:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:05:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:05:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:05:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:05:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:05:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:05:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:05:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:05:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:05:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:05:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:05:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:05:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:05:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:05:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:05:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:05:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:05:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.62238647199999, 2023-07-02T14:05:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:05:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:05:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:05:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:05:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:05:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:05:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:05:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:05:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:05:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:05:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:05:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:05:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:05:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:05:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:05:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:05:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:05:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:05:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:05:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:05:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:05:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:05:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:05:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:05:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.62230541100001, 2023-07-02T14:05:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:05:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:05:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:05:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:05:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:05:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:05:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:05:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:05:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:05:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:05:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:05:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:05:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:05:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:05:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:05:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:05:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:05:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:05:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:05:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:05:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:05:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:05:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:05:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:05:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.62221374000001, 2023-07-02T14:05:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:05:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:05:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:05:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:05:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:05:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:05:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:05:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:05:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:05:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:05:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:05:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:05:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:05:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:05:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:05:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:05:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:05:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:05:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:05:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:05:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:05:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:05:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:05:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:05:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.62187429799999, 2023-07-02T14:05:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:05:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:05:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:05:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:05:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:05:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:05:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:05:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:05:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:05:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:05:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:05:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:05:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:05:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:05:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:05:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:05:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:05:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:05:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:05:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:05:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:05:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:05:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:05:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:05:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.621688245, 2023-07-02T14:05:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:05:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:05:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:05:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:05:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:05:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:05:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:05:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:05:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:05:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:05:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:05:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:05:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:05:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:05:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:05:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:05:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:05:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:05:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:05:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:05:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:05:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:05:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:05:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.62153044300001, 2023-07-02T14:05:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:05:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:05:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:05:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:05:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:05:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:05:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:05:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:05:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:05:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:05:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:05:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:05:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:05:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:05:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:05:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:05:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:05:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:05:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:05:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:05:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:05:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:05:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:05:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:05:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.62133866100001, 2023-07-02T14:05:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:05:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:05:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:05:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:05:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:05:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:05:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:05:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:05:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:05:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:05:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:05:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:05:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:05:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:05:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:05:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:05:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:05:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:05:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:05:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:05:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:05:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:05:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:05:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:05:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.62116300100001, 2023-07-02T14:05:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:05:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:05:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:05:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:05:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:05:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:05:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:05:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:05:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:05:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:05:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:05:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:05:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:05:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:05:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:05:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:05:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:05:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:05:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:05:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:05:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:05:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:05:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:05:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.62106881299999, 2023-07-02T14:05:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:05:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:05:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:05:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:05:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:05:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:05:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:05:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:05:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:05:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:05:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:05:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:05:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:05:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:05:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:05:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:05:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:05:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:05:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:05:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:05:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:05:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:05:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:05:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:05:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.62086542700001, 2023-07-02T14:05:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:05:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:05:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:05:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:05:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:05:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:05:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:05:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:05:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:05:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:05:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:05:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:05:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:05:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:05:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:05:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:05:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:05:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:05:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:05:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:05:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:05:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:05:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:05:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:05:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.620714771, 2023-07-02T14:05:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:05:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:05:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:05:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:05:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:05:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:05:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:05:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:05:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:05:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:05:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:40.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:05:40.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:05:40.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:40.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:05:40.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:05:40.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:05:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:05:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:40.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:05:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:05:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:05:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:05:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:05:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:05:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:40.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:05:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:05:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:40.060 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.620642068, 2023-07-02T14:05:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:05:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:40.061 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:05:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:40.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:05:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:05:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:05:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:05:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:05:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:05:40.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:05:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:05:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:05:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:05:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:05:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:05:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:05:40.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:05:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:05:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:40.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:05:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:05:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:05:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:05:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:05:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:05:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.62052507799999, 2023-07-02T14:05:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:05:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:40.071 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:05:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:40.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:05:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:05:40.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:05:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:05:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:05:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:05:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:05:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:05:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:05:40.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:05:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:05:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:40.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:05:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:05:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:05:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:05:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:05:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:05:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:05:40.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:05:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:05:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:05:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:40.080 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.62037562699999, 2023-07-02T14:05:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:05:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:40.081 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:05:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:40.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:40.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:05:40.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:05:40.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:40.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:05:40.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:05:40.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:40.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:05:40.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:05:40.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:40.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:05:40.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:05:40.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:40.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:05:40.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:05:40.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:40.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:05:40.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:05:40.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:40.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:05:40.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:05:40.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:40.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:05:40.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:05:40.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:40.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:05:40.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:05:40.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:40.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:05:40.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:05:40.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:40.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:05:40.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:05:40.090 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.091 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:40.091 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.091 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:40.091 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:40.091 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.59889965400001, 2023-07-02T14:05:40.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:05:40.092 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:40.092 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:40.092 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:40.092 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:40.092 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:40.092 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:05:40.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.093 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:40.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:40.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:40.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:40.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:40.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:40.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:40.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:40.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:05:40.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:05:40.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:40.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:05:40.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:05:40.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:40.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:05:40.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:05:40.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:40.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:05:40.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:05:40.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:40.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:05:40.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:05:40.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:40.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:05:40.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:05:40.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:40.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:05:40.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:05:40.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:40.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:05:40.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:05:40.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:40.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:05:40.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:05:40.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:40.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:05:40.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:05:40.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:40.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:05:40.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:05:40.101 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.101 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:40.101 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.101 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:40.102 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:40.102 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.59876711800001, 2023-07-02T14:05:40.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:05:40.102 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:40.102 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:40.102 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:40.102 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:40.102 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:40.103 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:05:40.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.103 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:40.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:40.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:40.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:40.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:40.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:40.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:40.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:40.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:05:40.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:05:40.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:40.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:05:40.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:05:40.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:40.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:05:40.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:05:40.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:40.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:05:40.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:05:40.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:40.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:05:40.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:05:40.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:40.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:05:40.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:05:40.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:40.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:05:40.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:05:40.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:40.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:05:40.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:05:40.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:40.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:05:40.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:05:40.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:40.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:05:40.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:05:40.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:40.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:05:40.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:05:40.111 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.111 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:40.111 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.112 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:40.112 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:40.112 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.537430457, 2023-07-02T14:05:40.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:05:40.112 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:40.112 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:40.113 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:40.113 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:40.113 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:40.113 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:05:40.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.113 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:40.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:40.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:40.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:40.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:40.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:40.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:40.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:40.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:05:40.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:05:40.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:40.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:05:40.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:05:40.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:40.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:05:40.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:05:40.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:40.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:05:40.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:05:40.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:40.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:05:40.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:05:40.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:40.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:05:40.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:05:40.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:40.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:05:40.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:05:40.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:40.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:05:40.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:05:40.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:40.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:05:40.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:05:40.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:40.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:05:40.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:05:40.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:40.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:40.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:40.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:05:40.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:05:40.121 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.122 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:40.122 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.122 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:40.122 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:40.122 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:40.123 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:40.123 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:40.124 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:40.124 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:40.124 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:05:40.125 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:40.122+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:40.141 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:40.141 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-07-02T14:05:40.142 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:05:40.142 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:05:40.142 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:05:40.142 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:05:40.143 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:05:40.143 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:05:40.145 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:05:40.146 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:05:40.150 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 1 2023-07-02T14:05:40.223 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:40.223+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:40.319 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:40.319 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:40.324 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:40.324+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:40.332 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:05:40.388 INFO:teuthology.orchestra.run.smithi016.stderr:Nothing to do for non-filestore backend 2023-07-02T14:05:40.425 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:40.424+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:40.508 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:40.508 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:40.518 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:05:40.526 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:40.526+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:40.626 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:40.626+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:40.681 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:40.681 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:40.691 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:05:40.734 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:40.734+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:40.816 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:40.816 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:40.826 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:05:40.867 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:40.867+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:40.928 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:40.928+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:40.991 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:40.991 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:41.002 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:05:41.045 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:41.045+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:41.114 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:05:41.126 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:41.126 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:41.129 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:41.129+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:41.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:05:41.152 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap41 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap42 b3fbb028-e267-4d25-a750-93ba18a2316a@snap42.diff 2023-07-02T14:05:41.198 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 100% complete...done. 2023-07-02T14:05:41.202 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:05:41.202 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=43 2023-07-02T14:05:41.203 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap43 2023-07-02T14:05:41.230 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:41.230+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:41.306 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:41.306 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:41.317 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:05:41.358 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:41.358+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:41.431 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:41.431+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:41.482 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:41.482 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:41.492 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:05:41.532 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:41.531+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:41.632 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:41.632+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:41.658 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:41.659 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:41.669 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:05:41.733 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:41.732+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:41.807 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:41.807 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:41.818 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:05:41.834 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:41.834+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:41.935 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:41.935+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:41.982 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:41.982 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:41.992 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:05:42.036 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:42.035+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:42.111 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:05:42.118 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:42.118 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:42.132 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:05:42.147 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:05:42.160 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:05:42.166 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:42.166+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:42.170 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 3 2023-07-02T14:05:42.236 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:42.236+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:42.327 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:42.327 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:42.337 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:42.336+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:42.338 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:05:42.383 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:42.410 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:42.433 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:42.438 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:42.438+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:42.460 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:42.489 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:42.489 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-07-02T14:05:42.490 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:05:42.490 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:05:42.490 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:05:42.490 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:05:42.490 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:05:42.495 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:05:42.496 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:05:42.525 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:42.525 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:42.525 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:42.525 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:42.525 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:42.538 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:42.538+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:42.541 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:05:42.639 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:42.638+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:42.735 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:42.735 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:42.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:42.735 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:42.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.736 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:42.736 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:42.736 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 183.442557556, 2023-07-02T14:05:42.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:05:42.736 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:42.736 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:42.737 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:42.737 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:42.737 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:42.737 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:05:42.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.737 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:42.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:42.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:42.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:42.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:42.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:42.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:42.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:42.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:05:42.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:05:42.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:42.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:05:42.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:05:42.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:42.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:05:42.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:05:42.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:42.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:05:42.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:05:42.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:42.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:05:42.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:05:42.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:42.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:05:42.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:05:42.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:42.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:05:42.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:05:42.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:42.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:05:42.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:05:42.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:42.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:05:42.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:05:42.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:42.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:05:42.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:05:42.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:42.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:05:42.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:05:42.746 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.746 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.746 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.746 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:42.747 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:42.747 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 183.44232253300001, 2023-07-02T14:05:42.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:05:42.747 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:42.747 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:42.747 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:42.747 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:42.747 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:42.748 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:05:42.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.748 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:42.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:42.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:42.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:42.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:42.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:42.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:42.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:42.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:05:42.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:05:42.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:42.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:05:42.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:05:42.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:42.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:05:42.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:05:42.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:42.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:05:42.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:05:42.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:42.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:05:42.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:05:42.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:42.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:05:42.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:05:42.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:42.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:05:42.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:05:42.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:42.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:05:42.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:05:42.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:42.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:05:42.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:05:42.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:42.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:05:42.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:05:42.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:42.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:05:42.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:05:42.756 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.756 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.756 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.757 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:42.757 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:42.757 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 183.442123399, 2023-07-02T14:05:42.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:05:42.757 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:42.757 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:42.757 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:42.758 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:42.758 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:42.758 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:05:42.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.758 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:42.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:42.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:42.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:42.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:42.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:42.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:42.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:42.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:05:42.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:05:42.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:42.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:05:42.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:05:42.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:42.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:05:42.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:05:42.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:42.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:05:42.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:05:42.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:42.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:05:42.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:05:42.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:42.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:05:42.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:05:42.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:42.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:05:42.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:05:42.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:42.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:05:42.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:05:42.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:42.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:05:42.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:05:42.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:42.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:05:42.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:05:42.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:42.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:05:42.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:05:42.767 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.767 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.767 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.767 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:42.767 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:42.767 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 183.44175827500001, 2023-07-02T14:05:42.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:05:42.768 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:42.768 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:42.768 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:42.768 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:42.768 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:42.768 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:05:42.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.769 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:42.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:42.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:42.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:42.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:42.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:42.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:42.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:42.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:05:42.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:05:42.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:42.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:05:42.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:05:42.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:42.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:05:42.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:05:42.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:42.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:05:42.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:05:42.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:42.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:05:42.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:05:42.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:42.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:05:42.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:05:42.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:42.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:05:42.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:05:42.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:42.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:05:42.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:05:42.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:42.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:05:42.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:05:42.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:42.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:05:42.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:05:42.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:42.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:05:42.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:05:42.777 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.777 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.777 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.778 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:42.778 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:42.778 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 183.44167721400001, 2023-07-02T14:05:42.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:05:42.778 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:42.778 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:42.778 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:42.779 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:42.779 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:42.779 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:05:42.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.779 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:42.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:42.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:42.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:42.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:42.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:42.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:42.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:42.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:05:42.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:05:42.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:42.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:05:42.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:05:42.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:42.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:05:42.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:05:42.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:42.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:05:42.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:05:42.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:42.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:05:42.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:05:42.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:42.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:05:42.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:05:42.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:42.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:05:42.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:05:42.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:42.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:05:42.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:05:42.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:42.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:05:42.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:05:42.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:42.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:05:42.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:05:42.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:42.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:05:42.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:05:42.788 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.788 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.788 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.788 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:42.788 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:42.789 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 183.441585543, 2023-07-02T14:05:42.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:05:42.789 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:42.789 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:42.789 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:42.789 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:42.789 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:42.790 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:05:42.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.790 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:42.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:42.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:42.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:42.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:42.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:42.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:42.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:42.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:05:42.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:05:42.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:42.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:05:42.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:05:42.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:42.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:05:42.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:05:42.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:42.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:05:42.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:05:42.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:42.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:05:42.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:05:42.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:42.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:05:42.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:05:42.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:42.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:05:42.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:05:42.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:42.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:05:42.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:05:42.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:42.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:05:42.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:05:42.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:42.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:05:42.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:05:42.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:42.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:05:42.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:05:42.798 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.798 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.798 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.799 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:42.799 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:42.799 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 183.44124610099999, 2023-07-02T14:05:42.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:05:42.799 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:42.800 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:42.800 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:42.800 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:42.800 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:42.800 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:05:42.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.800 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:42.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:42.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:42.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:42.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:42.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:42.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:42.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:42.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:05:42.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:05:42.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:42.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:05:42.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:05:42.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:42.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:05:42.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:05:42.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:42.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:05:42.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:05:42.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:42.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:05:42.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:05:42.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:42.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:05:42.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:05:42.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:42.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:05:42.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:05:42.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:42.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:05:42.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:05:42.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:42.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:05:42.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:05:42.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:42.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:05:42.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:05:42.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:42.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:05:42.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:05:42.809 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.809 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.809 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.809 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:42.810 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:42.810 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 183.441060048, 2023-07-02T14:05:42.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:05:42.810 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:42.810 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:42.810 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:42.810 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:42.811 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:42.811 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:05:42.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.811 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:42.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:42.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:42.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:42.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:42.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:42.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:42.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:42.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:42.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:05:42.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:42.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:05:42.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:05:42.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:42.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:05:42.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:05:42.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:42.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:05:42.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:05:42.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:42.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:05:42.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:05:42.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:42.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:05:42.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:05:42.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:42.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:05:42.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:05:42.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:42.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:05:42.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:05:42.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:42.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:05:42.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:05:42.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:42.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:05:42.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:05:42.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:42.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:05:42.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:05:42.819 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.819 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.820 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.820 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:42.820 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:42.820 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 183.44090224600001, 2023-07-02T14:05:42.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:05:42.821 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:42.821 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:42.821 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:42.821 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:42.821 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:42.821 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:05:42.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.821 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:42.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:42.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:42.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:42.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:42.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:42.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:42.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:42.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:05:42.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:05:42.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:42.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:05:42.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:05:42.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:42.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:05:42.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:05:42.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:42.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:05:42.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:05:42.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:42.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:05:42.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:05:42.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:42.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:05:42.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:05:42.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:42.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:05:42.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:05:42.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:42.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:05:42.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:05:42.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:42.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:05:42.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:05:42.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:42.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:05:42.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:05:42.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:42.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:05:42.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:05:42.830 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.830 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.830 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.830 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:42.830 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:42.831 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 183.44071046400001, 2023-07-02T14:05:42.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:05:42.831 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:42.831 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:42.831 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:42.831 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:42.831 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:42.831 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:05:42.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.832 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:42.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:42.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:42.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:42.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:42.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:42.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:42.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:42.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:05:42.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:05:42.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:42.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:05:42.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:05:42.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:42.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:05:42.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:05:42.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:42.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:05:42.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:05:42.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:42.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:05:42.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:05:42.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:42.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:05:42.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:05:42.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:42.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:05:42.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:05:42.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:42.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:05:42.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:05:42.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:42.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:05:42.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:05:42.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:42.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:05:42.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:05:42.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:42.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:05:42.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:05:42.840 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.840 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.840 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.841 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:42.841 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:42.841 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 183.44053480400001, 2023-07-02T14:05:42.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:05:42.841 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:42.841 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:42.842 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:42.842 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:42.842 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:42.842 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:05:42.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.842 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:42.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:42.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:42.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:42.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:42.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:42.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:42.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:42.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:42.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:05:42.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:42.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:05:42.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:05:42.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:42.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:05:42.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:05:42.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:42.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:05:42.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:05:42.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:42.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:05:42.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:05:42.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:42.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:05:42.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:05:42.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:42.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:05:42.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:05:42.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:42.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:05:42.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:05:42.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:42.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:05:42.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:05:42.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:42.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:05:42.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:05:42.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:42.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:05:42.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:05:42.851 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.851 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.851 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.851 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:42.851 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:42.852 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 183.44044061599999, 2023-07-02T14:05:42.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:05:42.852 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:42.852 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:42.852 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:42.852 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:42.852 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:42.852 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:05:42.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.853 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:42.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:42.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:42.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:42.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:42.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:42.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:42.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:42.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:05:42.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:05:42.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:42.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:05:42.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:05:42.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:42.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:05:42.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:05:42.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:42.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:05:42.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:05:42.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:42.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:05:42.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:05:42.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:42.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:05:42.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:05:42.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:42.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:05:42.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:05:42.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:42.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:05:42.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:05:42.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:42.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:05:42.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:05:42.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:42.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:05:42.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:05:42.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:42.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:05:42.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:05:42.861 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.861 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.861 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.862 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:42.862 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:42.862 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 183.44023723000001, 2023-07-02T14:05:42.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:05:42.862 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:42.862 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:42.863 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:42.863 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:42.863 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:42.863 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:05:42.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.863 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:42.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:42.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:42.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:42.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:42.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:42.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:42.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:42.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:05:42.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:05:42.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:42.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:05:42.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:05:42.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:42.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:05:42.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:05:42.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:42.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:05:42.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:05:42.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:42.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:05:42.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:05:42.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:42.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:05:42.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:05:42.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:42.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:05:42.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:05:42.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:42.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:05:42.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:05:42.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:42.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:05:42.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:05:42.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:42.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:05:42.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:05:42.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:42.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:05:42.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:05:42.872 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.872 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.872 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.872 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:42.873 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:42.873 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 183.44008657399999, 2023-07-02T14:05:42.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:05:42.873 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:42.873 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:42.873 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:42.873 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:42.873 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:42.874 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:05:42.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.874 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:42.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:42.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:42.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:42.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:42.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:42.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:42.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:42.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:05:42.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:05:42.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:42.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:05:42.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:05:42.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:42.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:05:42.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:05:42.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:42.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:05:42.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:05:42.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:42.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:05:42.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:05:42.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:42.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:05:42.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:05:42.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:42.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:05:42.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:05:42.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:42.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:05:42.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:05:42.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:42.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:05:42.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:05:42.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:42.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:05:42.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:05:42.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:42.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:05:42.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:05:42.882 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.882 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.882 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.883 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:42.883 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:42.883 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 183.44001387099999, 2023-07-02T14:05:42.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:05:42.883 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:42.883 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:42.884 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:42.884 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:42.884 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:42.884 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:05:42.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.884 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:42.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:42.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:42.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:42.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:42.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:42.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:42.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:42.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:42.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:05:42.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:42.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:05:42.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:05:42.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:42.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:05:42.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:05:42.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:42.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:05:42.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:05:42.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:42.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:05:42.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:05:42.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:42.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:05:42.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:05:42.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:42.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:05:42.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:05:42.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:42.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:05:42.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:05:42.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:42.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:05:42.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:05:42.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:42.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:05:42.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:05:42.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:42.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:05:42.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:05:42.892 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.893 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.893 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.893 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:42.893 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:42.893 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 183.43989688100001, 2023-07-02T14:05:42.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:05:42.894 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:42.894 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:42.894 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:42.894 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:42.894 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:42.894 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:05:42.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.894 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:42.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:42.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:42.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:42.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:42.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:42.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:42.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:42.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:42.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:05:42.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:42.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:05:42.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:05:42.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:42.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:05:42.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:05:42.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:42.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:05:42.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:05:42.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:42.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:05:42.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:05:42.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:42.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:05:42.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:05:42.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:42.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:05:42.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:05:42.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:42.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:05:42.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:05:42.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:42.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:05:42.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:05:42.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:42.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:05:42.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:05:42.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:42.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:05:42.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:05:42.903 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.903 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.903 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.904 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:42.904 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:42.904 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 183.43974743000001, 2023-07-02T14:05:42.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:05:42.904 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:42.904 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:42.905 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:42.905 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:42.905 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:42.905 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:05:42.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.905 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:42.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:42.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:42.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:42.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:42.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:42.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:42.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:42.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:05:42.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:05:42.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:42.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:05:42.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:05:42.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:42.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:05:42.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:05:42.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:42.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:05:42.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:05:42.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:42.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:05:42.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:05:42.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:42.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:05:42.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:05:42.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:42.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:05:42.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:05:42.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:42.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:05:42.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:05:42.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:42.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:05:42.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:05:42.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:42.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:05:42.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:05:42.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:42.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:05:42.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:05:42.914 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.914 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.914 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.914 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:42.914 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:42.914 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 183.418271457, 2023-07-02T14:05:42.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:05:42.915 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:42.915 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:42.915 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:42.915 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:42.915 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:42.915 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:05:42.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.916 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:42.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:42.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:42.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:42.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:42.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:42.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:42.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:42.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:05:42.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:05:42.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:42.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:05:42.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:05:42.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:42.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:05:42.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:05:42.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:42.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:05:42.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:05:42.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:42.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:05:42.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:05:42.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:42.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:05:42.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:05:42.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:42.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:05:42.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:05:42.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:42.922 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:05:42.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:05:42.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:42.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:05:42.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:05:42.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:42.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:05:42.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:05:42.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:42.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:05:42.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:05:42.924 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.924 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.924 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.925 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:42.925 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:42.925 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 183.41813892100001, 2023-07-02T14:05:42.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:05:42.925 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:05:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:42.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:42.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:05:42.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:05:42.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:42.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:05:42.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:05:42.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:42.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:05:42.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:05:42.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:42.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:05:42.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:05:42.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:42.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:05:42.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:05:42.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:42.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:05:42.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:05:42.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:42.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:05:42.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:05:42.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:42.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:05:42.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:05:42.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:42.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:05:42.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:05:42.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:42.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:05:42.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:05:42.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:42.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:05:42.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:05:42.935 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.935 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.935 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.935 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:42.935 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:42.936 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 183.35680225999999, 2023-07-02T14:05:42.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:05:42.936 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:42.936 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:42.936 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:42.936 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:42.936 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:42.936 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:05:42.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.937 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:42.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:42.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:42.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:42.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:42.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:42.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:42.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:42.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:05:42.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:05:42.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:42.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:05:42.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:05:42.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:42.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:05:42.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:05:42.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:42.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:05:42.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:05:42.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:42.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:05:42.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:05:42.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:42.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:05:42.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:05:42.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:42.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:05:42.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:05:42.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:42.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:05:42.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:05:42.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:42.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:05:42.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:05:42.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:42.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:05:42.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:05:42.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:42.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:42.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:42.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:05:42.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:05:42.945 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.945 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.945 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.946 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:42.946 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:42.946 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:42.947 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:05:42.948 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:42.947+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:43.047 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:43.047+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:43.137 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17199 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.130827+0000", 2023-07-02T14:05:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0052332890000000003, 2023-07-02T14:05:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052532239999999999, 2023-07-02T14:05:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "delayed", 2023-07-02T14:05:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17199 2023-07-02T14:05:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130827+0000", 2023-07-02T14:05:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130827+0000", 2023-07-02T14:05:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130827+0000", 2023-07-02T14:05:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.517e-06 2023-07-02T14:05:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130830+0000", 2023-07-02T14:05:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5080000000000002e-06 2023-07-02T14:05:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130837+0000", 2023-07-02T14:05:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6266e-05 2023-07-02T14:05:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130853+0000", 2023-07-02T14:05:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000197383 2023-07-02T14:05:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.131050+0000", 2023-07-02T14:05:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1548999999999997e-05 2023-07-02T14:05:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.131102+0000", 2023-07-02T14:05:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029049750000000002 2023-07-02T14:05:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.134007+0000", 2023-07-02T14:05:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5704000000000005e-05 2023-07-02T14:05:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.134093+0000", 2023-07-02T14:05:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032654070000000001 2023-07-02T14:05:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17200 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.130867+0000", 2023-07-02T14:05:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0051935230000000002, 2023-07-02T14:05:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052785970000000003, 2023-07-02T14:05:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "delayed", 2023-07-02T14:05:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17200 2023-07-02T14:05:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130867+0000", 2023-07-02T14:05:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130867+0000", 2023-07-02T14:05:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130866+0000", 2023-07-02T14:05:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.887e-06 2023-07-02T14:05:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130871+0000", 2023-07-02T14:05:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0139999999999998e-06 2023-07-02T14:05:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130879+0000", 2023-07-02T14:05:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0980000000000001e-06 2023-07-02T14:05:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130884+0000", 2023-07-02T14:05:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000246939 2023-07-02T14:05:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.131131+0000", 2023-07-02T14:05:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0562000000000002e-05 2023-07-02T14:05:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.131172+0000", 2023-07-02T14:05:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029853420000000002 2023-07-02T14:05:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.134157+0000", 2023-07-02T14:05:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8777999999999999e-05 2023-07-02T14:05:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.134216+0000", 2023-07-02T14:05:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033488260000000001 2023-07-02T14:05:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17201 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.130889+0000", 2023-07-02T14:05:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0051714159999999999, 2023-07-02T14:05:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053068129999999996, 2023-07-02T14:05:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "delayed", 2023-07-02T14:05:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17201 2023-07-02T14:05:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130889+0000", 2023-07-02T14:05:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130889+0000", 2023-07-02T14:05:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130889+0000", 2023-07-02T14:05:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5370000000000002e-06 2023-07-02T14:05:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130892+0000", 2023-07-02T14:05:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7769999999999999e-06 2023-07-02T14:05:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130898+0000", 2023-07-02T14:05:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.079e-06 2023-07-02T14:05:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130903+0000", 2023-07-02T14:05:43.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000294049 2023-07-02T14:05:43.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.131197+0000", 2023-07-02T14:05:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7262999999999999e-05 2023-07-02T14:05:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.131234+0000", 2023-07-02T14:05:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030141450000000002 2023-07-02T14:05:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.134248+0000", 2023-07-02T14:05:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7478000000000003e-05 2023-07-02T14:05:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.134286+0000", 2023-07-02T14:05:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033968610000000002 2023-07-02T14:05:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17202 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.130908+0000", 2023-07-02T14:05:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0051528759999999998, 2023-07-02T14:05:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053377650000000004, 2023-07-02T14:05:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "delayed", 2023-07-02T14:05:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17202 2023-07-02T14:05:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130908+0000", 2023-07-02T14:05:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130908+0000", 2023-07-02T14:05:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130907+0000", 2023-07-02T14:05:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6710000000000001e-06 2023-07-02T14:05:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130911+0000", 2023-07-02T14:05:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.331e-06 2023-07-02T14:05:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130917+0000", 2023-07-02T14:05:43.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3529999999999997e-06 2023-07-02T14:05:43.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130921+0000", 2023-07-02T14:05:43.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035358699999999999 2023-07-02T14:05:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.131275+0000", 2023-07-02T14:05:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5117000000000005e-05 2023-07-02T14:05:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.131360+0000", 2023-07-02T14:05:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00295334 2023-07-02T14:05:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.134313+0000", 2023-07-02T14:05:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3500000000000001e-05 2023-07-02T14:05:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.134347+0000", 2023-07-02T14:05:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0034394260000000002 2023-07-02T14:05:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17203 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.130932+0000", 2023-07-02T14:05:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0051279530000000002, 2023-07-02T14:05:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053609909999999998, 2023-07-02T14:05:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "delayed", 2023-07-02T14:05:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17203 2023-07-02T14:05:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130932+0000", 2023-07-02T14:05:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130932+0000", 2023-07-02T14:05:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130932+0000", 2023-07-02T14:05:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1130000000000003e-06 2023-07-02T14:05:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130936+0000", 2023-07-02T14:05:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1509999999999998e-06 2023-07-02T14:05:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130943+0000", 2023-07-02T14:05:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4800000000000003e-06 2023-07-02T14:05:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130948+0000", 2023-07-02T14:05:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00044417000000000002 2023-07-02T14:05:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.131392+0000", 2023-07-02T14:05:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0191999999999997e-05 2023-07-02T14:05:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.131432+0000", 2023-07-02T14:05:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029408149999999998 2023-07-02T14:05:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.134373+0000", 2023-07-02T14:05:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3498999999999999e-05 2023-07-02T14:05:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.134406+0000", 2023-07-02T14:05:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0034738939999999999 2023-07-02T14:05:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17204 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.130952+0000", 2023-07-02T14:05:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0051082039999999999, 2023-07-02T14:05:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053902170000000001, 2023-07-02T14:05:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "delayed", 2023-07-02T14:05:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17204 2023-07-02T14:05:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130952+0000", 2023-07-02T14:05:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130952+0000", 2023-07-02T14:05:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130952+0000", 2023-07-02T14:05:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7139999999999999e-06 2023-07-02T14:05:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130955+0000", 2023-07-02T14:05:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.054e-06 2023-07-02T14:05:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130962+0000", 2023-07-02T14:05:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7610000000000001e-06 2023-07-02T14:05:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130965+0000", 2023-07-02T14:05:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0034574929999999999 2023-07-02T14:05:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.134423+0000", 2023-07-02T14:05:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9406999999999997e-05 2023-07-02T14:05:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.134462+0000", 2023-07-02T14:05:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003510024 2023-07-02T14:05:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17092 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.060073+0000", 2023-07-02T14:05:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.075987536999999994, 2023-07-02T14:05:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.076313905000000001, 2023-07-02T14:05:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "waiting for sub ops", 2023-07-02T14:05:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17092 2023-07-02T14:05:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060073+0000", 2023-07-02T14:05:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060073+0000", 2023-07-02T14:05:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060072+0000", 2023-07-02T14:05:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8319999999999999e-06 2023-07-02T14:05:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060076+0000", 2023-07-02T14:05:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8969999999999996e-06 2023-07-02T14:05:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060082+0000", 2023-07-02T14:05:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0500000000000002e-06 2023-07-02T14:05:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060086+0000", 2023-07-02T14:05:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026385699999999999 2023-07-02T14:05:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060350+0000", 2023-07-02T14:05:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2800999999999996e-05 2023-07-02T14:05:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060423+0000", 2023-07-02T14:05:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073658539999999998 2023-07-02T14:05:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067789+0000", 2023-07-02T14:05:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8631000000000003e-05 2023-07-02T14:05:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067827+0000", 2023-07-02T14:05:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091377279999999995 2023-07-02T14:05:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076965+0000", 2023-07-02T14:05:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1624e-05 2023-07-02T14:05:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077007+0000", 2023-07-02T14:05:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070729699999999996 2023-07-02T14:05:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084080+0000", 2023-07-02T14:05:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6180000000000003e-05 2023-07-02T14:05:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084116+0000", 2023-07-02T14:05:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00938195 2023-07-02T14:05:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093498+0000", 2023-07-02T14:05:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2301000000000001e-05 2023-07-02T14:05:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093530+0000", 2023-07-02T14:05:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0085737460000000001 2023-07-02T14:05:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102104+0000", 2023-07-02T14:05:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4241999999999997e-05 2023-07-02T14:05:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102188+0000", 2023-07-02T14:05:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01019401 2023-07-02T14:05:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112382+0000", 2023-07-02T14:05:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9167e-05 2023-07-02T14:05:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112421+0000", 2023-07-02T14:05:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0094647619999999998 2023-07-02T14:05:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.121886+0000", 2023-07-02T14:05:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7998000000000006e-05 2023-07-02T14:05:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.121984+0000", 2023-07-02T14:05:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098062940000000001 2023-07-02T14:05:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.131790+0000", 2023-07-02T14:05:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0334999999999999e-05 2023-07-02T14:05:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.131831+0000", 2023-07-02T14:05:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024656500000000002 2023-07-02T14:05:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.132077+0000", 2023-07-02T14:05:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002335087 2023-07-02T14:05:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.134412+0000", 2023-07-02T14:05:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.074339379999999997 2023-07-02T14:05:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17093 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.060090+0000", 2023-07-02T14:05:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.075970215999999993, 2023-07-02T14:05:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.076398600999999997, 2023-07-02T14:05:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "delayed", 2023-07-02T14:05:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17093 2023-07-02T14:05:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060090+0000", 2023-07-02T14:05:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060090+0000", 2023-07-02T14:05:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060090+0000", 2023-07-02T14:05:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0960000000000001e-06 2023-07-02T14:05:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060093+0000", 2023-07-02T14:05:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5849999999999999e-06 2023-07-02T14:05:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060099+0000", 2023-07-02T14:05:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.737e-06 2023-07-02T14:05:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060102+0000", 2023-07-02T14:05:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00034522300000000003 2023-07-02T14:05:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060447+0000", 2023-07-02T14:05:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0445999999999999e-05 2023-07-02T14:05:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060488+0000", 2023-07-02T14:05:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073722299999999996 2023-07-02T14:05:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067860+0000", 2023-07-02T14:05:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9782e-05 2023-07-02T14:05:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067900+0000", 2023-07-02T14:05:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091379500000000006 2023-07-02T14:05:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077038+0000", 2023-07-02T14:05:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3115999999999999e-05 2023-07-02T14:05:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077071+0000", 2023-07-02T14:05:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070828310000000004 2023-07-02T14:05:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084154+0000", 2023-07-02T14:05:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3495999999999999e-05 2023-07-02T14:05:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084187+0000", 2023-07-02T14:05:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093731779999999994 2023-07-02T14:05:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093561+0000", 2023-07-02T14:05:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2091999999999997e-05 2023-07-02T14:05:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093593+0000", 2023-07-02T14:05:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086525769999999998 2023-07-02T14:05:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102245+0000", 2023-07-02T14:05:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2937999999999998e-05 2023-07-02T14:05:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102328+0000", 2023-07-02T14:05:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010122403 2023-07-02T14:05:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112451+0000", 2023-07-02T14:05:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3404000000000001e-05 2023-07-02T14:05:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112484+0000", 2023-07-02T14:05:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0095671820000000005 2023-07-02T14:05:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122051+0000", 2023-07-02T14:05:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8579999999999995e-05 2023-07-02T14:05:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122150+0000", 2023-07-02T14:05:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011234401999999999 2023-07-02T14:05:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133384+0000", 2023-07-02T14:05:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010675099999999999 2023-07-02T14:05:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133491+0000", 2023-07-02T14:05:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.073400660000000006 2023-07-02T14:05:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17094 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.060106+0000", 2023-07-02T14:05:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.075954246000000003, 2023-07-02T14:05:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.076478371000000003, 2023-07-02T14:05:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "delayed", 2023-07-02T14:05:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17094 2023-07-02T14:05:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060106+0000", 2023-07-02T14:05:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060106+0000", 2023-07-02T14:05:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060106+0000", 2023-07-02T14:05:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0280000000000001e-06 2023-07-02T14:05:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060109+0000", 2023-07-02T14:05:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8779999999999997e-06 2023-07-02T14:05:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060114+0000", 2023-07-02T14:05:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5810000000000001e-06 2023-07-02T14:05:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060117+0000", 2023-07-02T14:05:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000400636 2023-07-02T14:05:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060518+0000", 2023-07-02T14:05:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3895999999999999e-05 2023-07-02T14:05:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060592+0000", 2023-07-02T14:05:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073529210000000001 2023-07-02T14:05:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067945+0000", 2023-07-02T14:05:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1876000000000001e-05 2023-07-02T14:05:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068017+0000", 2023-07-02T14:05:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090869430000000001 2023-07-02T14:05:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077104+0000", 2023-07-02T14:05:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2429000000000003e-05 2023-07-02T14:05:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077146+0000", 2023-07-02T14:05:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070717699999999998 2023-07-02T14:05:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084218+0000", 2023-07-02T14:05:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2542000000000001e-05 2023-07-02T14:05:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084250+0000", 2023-07-02T14:05:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009381927 2023-07-02T14:05:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093632+0000", 2023-07-02T14:05:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3652999999999998e-05 2023-07-02T14:05:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093666+0000", 2023-07-02T14:05:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0087224720000000002 2023-07-02T14:05:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102388+0000", 2023-07-02T14:05:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9060000000000005e-05 2023-07-02T14:05:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102477+0000", 2023-07-02T14:05:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010033492999999999 2023-07-02T14:05:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112511+0000", 2023-07-02T14:05:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4153000000000002e-05 2023-07-02T14:05:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112555+0000", 2023-07-02T14:05:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096431239999999994 2023-07-02T14:05:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122198+0000", 2023-07-02T14:05:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2417999999999996e-05 2023-07-02T14:05:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122281+0000", 2023-07-02T14:05:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011250624000000001 2023-07-02T14:05:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133531+0000", 2023-07-02T14:05:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3800000000000001e-05 2023-07-02T14:05:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133575+0000", 2023-07-02T14:05:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.073468885999999997 2023-07-02T14:05:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17095 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.060121+0000", 2023-07-02T14:05:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.075938955000000002, 2023-07-02T14:05:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.076566757999999999, 2023-07-02T14:05:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "delayed", 2023-07-02T14:05:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17095 2023-07-02T14:05:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060121+0000", 2023-07-02T14:05:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060121+0000", 2023-07-02T14:05:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060121+0000", 2023-07-02T14:05:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5140000000000004e-06 2023-07-02T14:05:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060129+0000", 2023-07-02T14:05:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4520000000000002e-06 2023-07-02T14:05:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060135+0000", 2023-07-02T14:05:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2960000000000004e-06 2023-07-02T14:05:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060141+0000", 2023-07-02T14:05:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00047506399999999998 2023-07-02T14:05:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060616+0000", 2023-07-02T14:05:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8182999999999997e-05 2023-07-02T14:05:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060655+0000", 2023-07-02T14:05:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074016079999999996 2023-07-02T14:05:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068056+0000", 2023-07-02T14:05:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6019000000000005e-05 2023-07-02T14:05:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068132+0000", 2023-07-02T14:05:43.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090475429999999999 2023-07-02T14:05:43.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077180+0000", 2023-07-02T14:05:43.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7527999999999997e-05 2023-07-02T14:05:43.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077217+0000", 2023-07-02T14:05:43.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070671400000000004 2023-07-02T14:05:43.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084284+0000", 2023-07-02T14:05:43.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3818e-05 2023-07-02T14:05:43.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084318+0000", 2023-07-02T14:05:43.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093793799999999997 2023-07-02T14:05:43.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.261 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.261 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093698+0000", 2023-07-02T14:05:43.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2876999999999997e-05 2023-07-02T14:05:43.261 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.261 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.261 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093730+0000", 2023-07-02T14:05:43.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088179079999999993 2023-07-02T14:05:43.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.262 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.262 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.262 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102548+0000", 2023-07-02T14:05:43.262 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5802000000000003e-05 2023-07-02T14:05:43.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.262 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.262 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.263 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102634+0000", 2023-07-02T14:05:43.263 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099570119999999995 2023-07-02T14:05:43.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.263 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112591+0000", 2023-07-02T14:05:43.263 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6594000000000001e-05 2023-07-02T14:05:43.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.264 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112648+0000", 2023-07-02T14:05:43.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009671921 2023-07-02T14:05:43.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.264 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122320+0000", 2023-07-02T14:05:43.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2005000000000002e-05 2023-07-02T14:05:43.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.265 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122362+0000", 2023-07-02T14:05:43.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011261546000000001 2023-07-02T14:05:43.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133623+0000", 2023-07-02T14:05:43.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9981000000000006e-05 2023-07-02T14:05:43.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133693+0000", 2023-07-02T14:05:43.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.073571745999999993 2023-07-02T14:05:43.267 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.267 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.267 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.267 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17197 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.267 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.130765+0000", 2023-07-02T14:05:43.268 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0052955160000000001, 2023-07-02T14:05:43.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060188189999999999, 2023-07-02T14:05:43.268 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.268 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "delayed", 2023-07-02T14:05:43.268 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.268 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.268 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.268 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17197 2023-07-02T14:05:43.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.269 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130765+0000", 2023-07-02T14:05:43.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130765+0000", 2023-07-02T14:05:43.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:43.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130764+0000", 2023-07-02T14:05:43.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.128e-05 2023-07-02T14:05:43.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.271 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130775+0000", 2023-07-02T14:05:43.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4500000000000004e-06 2023-07-02T14:05:43.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130784+0000", 2023-07-02T14:05:43.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4349999999999996e-06 2023-07-02T14:05:43.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130789+0000", 2023-07-02T14:05:43.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6774999999999997e-05 2023-07-02T14:05:43.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.273 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130856+0000", 2023-07-02T14:05:43.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6754999999999994e-05 2023-07-02T14:05:43.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.273 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130923+0000", 2023-07-02T14:05:43.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028855460000000001 2023-07-02T14:05:43.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133808+0000", 2023-07-02T14:05:43.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6143000000000006e-05 2023-07-02T14:05:43.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.275 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.275 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133884+0000", 2023-07-02T14:05:43.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031195960000000001 2023-07-02T14:05:43.275 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.275 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.275 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.276 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17096 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.276 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:05:43.276 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.075895136000000002, 2023-07-02T14:05:43.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.076666877999999994, 2023-07-02T14:05:43.276 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.276 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "delayed", 2023-07-02T14:05:43.277 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.277 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.277 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.277 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17096 2023-07-02T14:05:43.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.277 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.277 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:05:43.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:05:43.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:43.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.279 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:05:43.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3400000000000002e-06 2023-07-02T14:05:43.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.279 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060168+0000", 2023-07-02T14:05:43.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4020000000000004e-06 2023-07-02T14:05:43.280 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060173+0000", 2023-07-02T14:05:43.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4259999999999999e-06 2023-07-02T14:05:43.280 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060176+0000", 2023-07-02T14:05:43.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00049978399999999997 2023-07-02T14:05:43.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060676+0000", 2023-07-02T14:05:43.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0135000000000004e-05 2023-07-02T14:05:43.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.282 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060756+0000", 2023-07-02T14:05:43.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074338060000000003 2023-07-02T14:05:43.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.283 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068190+0000", 2023-07-02T14:05:43.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7941000000000004e-05 2023-07-02T14:05:43.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068278+0000", 2023-07-02T14:05:43.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089611870000000007 2023-07-02T14:05:43.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077239+0000", 2023-07-02T14:05:43.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6349e-05 2023-07-02T14:05:43.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077275+0000", 2023-07-02T14:05:43.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007072142 2023-07-02T14:05:43.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084347+0000", 2023-07-02T14:05:43.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.591e-05 2023-07-02T14:05:43.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084383+0000", 2023-07-02T14:05:43.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093660119999999999 2023-07-02T14:05:43.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093749+0000", 2023-07-02T14:05:43.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1538000000000002e-05 2023-07-02T14:05:43.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093781+0000", 2023-07-02T14:05:43.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088884629999999992 2023-07-02T14:05:43.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102669+0000", 2023-07-02T14:05:43.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6032999999999997e-05 2023-07-02T14:05:43.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102725+0000", 2023-07-02T14:05:43.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009945522 2023-07-02T14:05:43.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112671+0000", 2023-07-02T14:05:43.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5741000000000002e-05 2023-07-02T14:05:43.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112707+0000", 2023-07-02T14:05:43.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096867940000000003 2023-07-02T14:05:43.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122393+0000", 2023-07-02T14:05:43.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6244999999999998e-05 2023-07-02T14:05:43.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122480+0000", 2023-07-02T14:05:43.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011251107999999999 2023-07-02T14:05:43.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133731+0000", 2023-07-02T14:05:43.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6400999999999999e-05 2023-07-02T14:05:43.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133767+0000", 2023-07-02T14:05:43.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.073601816 2023-07-02T14:05:43.293 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.293 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.294 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.294 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17198 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.294 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.130801+0000", 2023-07-02T14:05:43.294 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0052596780000000003, 2023-07-02T14:05:43.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061262540000000002, 2023-07-02T14:05:43.294 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.295 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "delayed", 2023-07-02T14:05:43.295 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.295 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.295 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.295 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17198 2023-07-02T14:05:43.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.295 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130801+0000", 2023-07-02T14:05:43.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130801+0000", 2023-07-02T14:05:43.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:43.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130800+0000", 2023-07-02T14:05:43.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5709999999999999e-06 2023-07-02T14:05:43.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130805+0000", 2023-07-02T14:05:43.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0590000000000003e-06 2023-07-02T14:05:43.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130813+0000", 2023-07-02T14:05:43.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3520000000000003e-06 2023-07-02T14:05:43.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130819+0000", 2023-07-02T14:05:43.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013274200000000001 2023-07-02T14:05:43.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130952+0000", 2023-07-02T14:05:43.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6854999999999997e-05 2023-07-02T14:05:43.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.131009+0000", 2023-07-02T14:05:43.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029174700000000001 2023-07-02T14:05:43.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133926+0000", 2023-07-02T14:05:43.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5785999999999999e-05 2023-07-02T14:05:43.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133962+0000", 2023-07-02T14:05:43.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031610789999999998 2023-07-02T14:05:43.302 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.302 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.302 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.302 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.302 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2023-07-02T14:05:43.302 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 13 2023-07-02T14:05:43.302 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:43.304 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:05:43.305 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:43.304+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:43.405 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:43.404+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:43.506 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:43.505+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:43.510 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:43.510 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:43.510 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:43.510 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:43.511 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:43.523 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:05:43.606 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:43.606+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:43.708 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:43.708+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:43.725 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:43.725 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:43.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:43.726 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:43.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.726 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:16940 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.726 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.855862+0000", 2023-07-02T14:05:43.726 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.86377644899999995, 2023-07-02T14:05:43.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.103600949, 2023-07-02T14:05:43.726 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.726 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:43.727 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.727 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.727 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.727 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 16940 2023-07-02T14:05:43.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.727 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855862+0000", 2023-07-02T14:05:43.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855862+0000", 2023-07-02T14:05:43.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:43.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855862+0000", 2023-07-02T14:05:43.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.353e-06 2023-07-02T14:05:43.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855868+0000", 2023-07-02T14:05:43.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.746e-06 2023-07-02T14:05:43.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855873+0000", 2023-07-02T14:05:43.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4030000000000001e-06 2023-07-02T14:05:43.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855875+0000", 2023-07-02T14:05:43.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050000019999999999 2023-07-02T14:05:43.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.860875+0000", 2023-07-02T14:05:43.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3556000000000001e-05 2023-07-02T14:05:43.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.860919+0000", 2023-07-02T14:05:43.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055944549999999999 2023-07-02T14:05:43.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.866513+0000", 2023-07-02T14:05:43.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2478999999999999e-05 2023-07-02T14:05:43.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.866546+0000", 2023-07-02T14:05:43.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050944609999999998 2023-07-02T14:05:43.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.871640+0000", 2023-07-02T14:05:43.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2138000000000003e-05 2023-07-02T14:05:43.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.871672+0000", 2023-07-02T14:05:43.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014725571999999999 2023-07-02T14:05:43.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.886398+0000", 2023-07-02T14:05:43.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6910999999999999e-05 2023-07-02T14:05:43.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.886435+0000", 2023-07-02T14:05:43.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015334054 2023-07-02T14:05:43.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.901769+0000", 2023-07-02T14:05:43.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0036000000000001e-05 2023-07-02T14:05:43.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.901809+0000", 2023-07-02T14:05:43.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014746522999999999 2023-07-02T14:05:43.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.916555+0000", 2023-07-02T14:05:43.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3622999999999999e-05 2023-07-02T14:05:43.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.916599+0000", 2023-07-02T14:05:43.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099546679999999999 2023-07-02T14:05:43.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.926554+0000", 2023-07-02T14:05:43.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7073000000000002e-05 2023-07-02T14:05:43.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.926591+0000", 2023-07-02T14:05:43.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045803249999999997 2023-07-02T14:05:43.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.931171+0000", 2023-07-02T14:05:43.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9468000000000004e-05 2023-07-02T14:05:43.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.931271+0000", 2023-07-02T14:05:43.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078347380000000008 2023-07-02T14:05:43.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.939105+0000", 2023-07-02T14:05:43.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1427999999999994e-05 2023-07-02T14:05:43.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.939177+0000", 2023-07-02T14:05:43.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093866379999999992 2023-07-02T14:05:43.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.948563+0000", 2023-07-02T14:05:43.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3999999999999995e-05 2023-07-02T14:05:43.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:43.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.948647+0000", 2023-07-02T14:05:43.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00034591999999999999 2023-07-02T14:05:43.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:43.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.948993+0000", 2023-07-02T14:05:43.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056008810000000003 2023-07-02T14:05:43.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:43.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.954594+0000", 2023-07-02T14:05:43.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047713010000000004 2023-07-02T14:05:43.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:43.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959365+0000", 2023-07-02T14:05:43.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4107999999999998e-05 2023-07-02T14:05:43.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:43.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959410+0000", 2023-07-02T14:05:43.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3331999999999997e-05 2023-07-02T14:05:43.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:43.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959463+0000", 2023-07-02T14:05:43.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.103600949 2023-07-02T14:05:43.747 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.747 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.747 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.748 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:16941 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.748 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:05:43.748 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.86376080399999999, 2023-07-02T14:05:43.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11449124500000001, 2023-07-02T14:05:43.748 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.748 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:43.748 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.749 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.749 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.749 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 16941 2023-07-02T14:05:43.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.749 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:05:43.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:05:43.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:05:43.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855877+0000", 2023-07-02T14:05:43.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3209999999999999e-06 2023-07-02T14:05:43.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855880+0000", 2023-07-02T14:05:43.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9299999999999999e-06 2023-07-02T14:05:43.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855883+0000", 2023-07-02T14:05:43.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6309999999999999e-06 2023-07-02T14:05:43.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855885+0000", 2023-07-02T14:05:43.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050516850000000002 2023-07-02T14:05:43.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.860937+0000", 2023-07-02T14:05:43.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1844999999999997e-05 2023-07-02T14:05:43.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.860979+0000", 2023-07-02T14:05:43.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055853480000000004 2023-07-02T14:05:43.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.866564+0000", 2023-07-02T14:05:43.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2821999999999998e-05 2023-07-02T14:05:43.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.866597+0000", 2023-07-02T14:05:43.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050938900000000002 2023-07-02T14:05:43.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.871691+0000", 2023-07-02T14:05:43.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1210000000000001e-05 2023-07-02T14:05:43.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.871722+0000", 2023-07-02T14:05:43.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014768687000000001 2023-07-02T14:05:43.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.886491+0000", 2023-07-02T14:05:43.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7248000000000003e-05 2023-07-02T14:05:43.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.886568+0000", 2023-07-02T14:05:43.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015403168 2023-07-02T14:05:43.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.901971+0000", 2023-07-02T14:05:43.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4035000000000002e-05 2023-07-02T14:05:43.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.902005+0000", 2023-07-02T14:05:43.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014618580000000001 2023-07-02T14:05:43.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.916624+0000", 2023-07-02T14:05:43.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.443e-05 2023-07-02T14:05:43.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.916668+0000", 2023-07-02T14:05:43.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099598989999999995 2023-07-02T14:05:43.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.926628+0000", 2023-07-02T14:05:43.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3123999999999999e-05 2023-07-02T14:05:43.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.926711+0000", 2023-07-02T14:05:43.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046031930000000002 2023-07-02T14:05:43.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.931314+0000", 2023-07-02T14:05:43.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3257000000000003e-05 2023-07-02T14:05:43.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.931358+0000", 2023-07-02T14:05:43.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078370969999999995 2023-07-02T14:05:43.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.939195+0000", 2023-07-02T14:05:43.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2522000000000003e-05 2023-07-02T14:05:43.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.939247+0000", 2023-07-02T14:05:43.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01118103 2023-07-02T14:05:43.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.950428+0000", 2023-07-02T14:05:43.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5001999999999996e-05 2023-07-02T14:05:43.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.950523+0000", 2023-07-02T14:05:43.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089648390000000005 2023-07-02T14:05:43.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959488+0000", 2023-07-02T14:05:43.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4569e-05 2023-07-02T14:05:43.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:43.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959533+0000", 2023-07-02T14:05:43.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026104299999999997 2023-07-02T14:05:43.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:43.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959794+0000", 2023-07-02T14:05:43.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025221789999999998 2023-07-02T14:05:43.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:43.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.962316+0000", 2023-07-02T14:05:43.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079080520000000005 2023-07-02T14:05:43.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:43.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970224+0000", 2023-07-02T14:05:43.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1265e-05 2023-07-02T14:05:43.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:43.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970255+0000", 2023-07-02T14:05:43.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000113581 2023-07-02T14:05:43.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:43.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970369+0000", 2023-07-02T14:05:43.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11449124500000001 2023-07-02T14:05:43.770 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.770 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.770 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.771 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17047 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.771 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.979346+0000", 2023-07-02T14:05:43.771 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.74029239800000002, 2023-07-02T14:05:43.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.095586451000000003, 2023-07-02T14:05:43.771 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.771 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:43.772 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.772 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.772 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.772 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17047 2023-07-02T14:05:43.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.772 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979346+0000", 2023-07-02T14:05:43.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979346+0000", 2023-07-02T14:05:43.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:43.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979345+0000", 2023-07-02T14:05:43.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.775e-06 2023-07-02T14:05:43.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979350+0000", 2023-07-02T14:05:43.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8170000000000001e-06 2023-07-02T14:05:43.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979357+0000", 2023-07-02T14:05:43.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9539999999999998e-06 2023-07-02T14:05:43.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979361+0000", 2023-07-02T14:05:43.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01301737 2023-07-02T14:05:43.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992378+0000", 2023-07-02T14:05:43.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4696999999999997e-05 2023-07-02T14:05:43.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992413+0000", 2023-07-02T14:05:43.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061779579999999999 2023-07-02T14:05:43.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998591+0000", 2023-07-02T14:05:43.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3654999999999999e-05 2023-07-02T14:05:43.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998664+0000", 2023-07-02T14:05:43.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091576609999999992 2023-07-02T14:05:43.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.007822+0000", 2023-07-02T14:05:43.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7914000000000002e-05 2023-07-02T14:05:43.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.007870+0000", 2023-07-02T14:05:43.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059512150000000002 2023-07-02T14:05:43.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.013821+0000", 2023-07-02T14:05:43.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1442999999999999e-05 2023-07-02T14:05:43.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.013902+0000", 2023-07-02T14:05:43.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075954430000000003 2023-07-02T14:05:43.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021498+0000", 2023-07-02T14:05:43.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0065000000000001e-05 2023-07-02T14:05:43.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021528+0000", 2023-07-02T14:05:43.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056511080000000002 2023-07-02T14:05:43.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027179+0000", 2023-07-02T14:05:43.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8032999999999998e-05 2023-07-02T14:05:43.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027237+0000", 2023-07-02T14:05:43.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048459089999999998 2023-07-02T14:05:43.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032083+0000", 2023-07-02T14:05:43.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1186000000000003e-05 2023-07-02T14:05:43.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032114+0000", 2023-07-02T14:05:43.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055940670000000003 2023-07-02T14:05:43.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.037708+0000", 2023-07-02T14:05:43.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0913000000000003e-05 2023-07-02T14:05:43.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.037789+0000", 2023-07-02T14:05:43.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043937489999999997 2023-07-02T14:05:43.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042183+0000", 2023-07-02T14:05:43.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0321000000000007e-05 2023-07-02T14:05:43.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042253+0000", 2023-07-02T14:05:43.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073046439999999999 2023-07-02T14:05:43.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049558+0000", 2023-07-02T14:05:43.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0810000000000006e-05 2023-07-02T14:05:43.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049649+0000", 2023-07-02T14:05:43.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0094162250000000003 2023-07-02T14:05:43.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059065+0000", 2023-07-02T14:05:43.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2486999999999997e-05 2023-07-02T14:05:43.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059157+0000", 2023-07-02T14:05:43.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059730709999999999 2023-07-02T14:05:43.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.065131+0000", 2023-07-02T14:05:43.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9991999999999998e-05 2023-07-02T14:05:43.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:43.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.065180+0000", 2023-07-02T14:05:43.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00037239199999999998 2023-07-02T14:05:43.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:43.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.065553+0000", 2023-07-02T14:05:43.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037123600000000001 2023-07-02T14:05:43.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:43.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.069265+0000", 2023-07-02T14:05:43.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055193860000000003 2023-07-02T14:05:43.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:43.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.074785+0000", 2023-07-02T14:05:43.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9505e-05 2023-07-02T14:05:43.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:43.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.074814+0000", 2023-07-02T14:05:43.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000118282 2023-07-02T14:05:43.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:43.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.074932+0000", 2023-07-02T14:05:43.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.095586451000000003 2023-07-02T14:05:43.794 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.795 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.795 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.795 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17048 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.795 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.979365+0000", 2023-07-02T14:05:43.795 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.74027317699999995, 2023-07-02T14:05:43.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.102723355, 2023-07-02T14:05:43.796 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.796 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:43.796 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.796 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.796 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.796 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17048 2023-07-02T14:05:43.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.797 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979365+0000", 2023-07-02T14:05:43.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979365+0000", 2023-07-02T14:05:43.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:43.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979365+0000", 2023-07-02T14:05:43.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4489999999999999e-06 2023-07-02T14:05:43.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979368+0000", 2023-07-02T14:05:43.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.178e-06 2023-07-02T14:05:43.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979374+0000", 2023-07-02T14:05:43.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0029999999999996e-06 2023-07-02T14:05:43.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979378+0000", 2023-07-02T14:05:43.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013053530000000001 2023-07-02T14:05:43.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992432+0000", 2023-07-02T14:05:43.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4631000000000001e-05 2023-07-02T14:05:43.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992467+0000", 2023-07-02T14:05:43.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062315440000000003 2023-07-02T14:05:43.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998698+0000", 2023-07-02T14:05:43.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2839999999999997e-05 2023-07-02T14:05:43.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998731+0000", 2023-07-02T14:05:43.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091807750000000004 2023-07-02T14:05:43.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.007912+0000", 2023-07-02T14:05:43.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0945999999999995e-05 2023-07-02T14:05:43.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.007983+0000", 2023-07-02T14:05:43.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059545220000000003 2023-07-02T14:05:43.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.013937+0000", 2023-07-02T14:05:43.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2709000000000001e-05 2023-07-02T14:05:43.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.013970+0000", 2023-07-02T14:05:43.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075831359999999999 2023-07-02T14:05:43.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021553+0000", 2023-07-02T14:05:43.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0125e-05 2023-07-02T14:05:43.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021583+0000", 2023-07-02T14:05:43.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056843680000000004 2023-07-02T14:05:43.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027268+0000", 2023-07-02T14:05:43.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0429e-05 2023-07-02T14:05:43.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027298+0000", 2023-07-02T14:05:43.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048443660000000001 2023-07-02T14:05:43.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032142+0000", 2023-07-02T14:05:43.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1180000000000003e-05 2023-07-02T14:05:43.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032174+0000", 2023-07-02T14:05:43.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005665944 2023-07-02T14:05:43.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.037839+0000", 2023-07-02T14:05:43.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6186000000000001e-05 2023-07-02T14:05:43.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.037886+0000", 2023-07-02T14:05:43.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044234330000000001 2023-07-02T14:05:43.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042309+0000", 2023-07-02T14:05:43.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6038999999999997e-05 2023-07-02T14:05:43.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042355+0000", 2023-07-02T14:05:43.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073303919999999998 2023-07-02T14:05:43.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049686+0000", 2023-07-02T14:05:43.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5858000000000001e-05 2023-07-02T14:05:43.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049721+0000", 2023-07-02T14:05:43.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0094923960000000002 2023-07-02T14:05:43.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059214+0000", 2023-07-02T14:05:43.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3883999999999998e-05 2023-07-02T14:05:43.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059298+0000", 2023-07-02T14:05:43.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080288240000000004 2023-07-02T14:05:43.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067326+0000", 2023-07-02T14:05:43.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0906999999999998e-05 2023-07-02T14:05:43.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067387+0000", 2023-07-02T14:05:43.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075801130000000003 2023-07-02T14:05:43.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.074968+0000", 2023-07-02T14:05:43.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1454e-05 2023-07-02T14:05:43.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:43.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.075009+0000", 2023-07-02T14:05:43.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025346800000000002 2023-07-02T14:05:43.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:43.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.075262+0000", 2023-07-02T14:05:43.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020188070000000001 2023-07-02T14:05:43.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:43.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077281+0000", 2023-07-02T14:05:43.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046643250000000004 2023-07-02T14:05:43.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:43.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.081946+0000", 2023-07-02T14:05:43.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.7495999999999999e-05 2023-07-02T14:05:43.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:43.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.081973+0000", 2023-07-02T14:05:43.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115494 2023-07-02T14:05:43.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:43.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082089+0000", 2023-07-02T14:05:43.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.102723355 2023-07-02T14:05:43.820 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.820 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.820 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.821 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17049 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.821 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:05:43.821 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.74025581299999998, 2023-07-02T14:05:43.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.112277108, 2023-07-02T14:05:43.821 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.822 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:43.822 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.822 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.822 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.822 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17049 2023-07-02T14:05:43.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.822 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:05:43.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:05:43.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:43.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979382+0000", 2023-07-02T14:05:43.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3450000000000002e-06 2023-07-02T14:05:43.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979385+0000", 2023-07-02T14:05:43.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9000000000000003e-06 2023-07-02T14:05:43.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979391+0000", 2023-07-02T14:05:43.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9999999999999998e-06 2023-07-02T14:05:43.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979395+0000", 2023-07-02T14:05:43.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013089412 2023-07-02T14:05:43.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992485+0000", 2023-07-02T14:05:43.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3828999999999997e-05 2023-07-02T14:05:43.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992519+0000", 2023-07-02T14:05:43.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062549889999999999 2023-07-02T14:05:43.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998774+0000", 2023-07-02T14:05:43.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4870999999999995e-05 2023-07-02T14:05:43.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998848+0000", 2023-07-02T14:05:43.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091715140000000004 2023-07-02T14:05:43.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008020+0000", 2023-07-02T14:05:43.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8659999999999999e-05 2023-07-02T14:05:43.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008059+0000", 2023-07-02T14:05:43.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059565110000000003 2023-07-02T14:05:43.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014015+0000", 2023-07-02T14:05:43.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8088000000000002e-05 2023-07-02T14:05:43.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014093+0000", 2023-07-02T14:05:43.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007513395 2023-07-02T14:05:43.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021607+0000", 2023-07-02T14:05:43.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9777e-05 2023-07-02T14:05:43.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021636+0000", 2023-07-02T14:05:43.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056999549999999996 2023-07-02T14:05:43.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027336+0000", 2023-07-02T14:05:43.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8356999999999996e-05 2023-07-02T14:05:43.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027405+0000", 2023-07-02T14:05:43.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047942130000000003 2023-07-02T14:05:43.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032199+0000", 2023-07-02T14:05:43.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1137999999999999e-05 2023-07-02T14:05:43.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032230+0000", 2023-07-02T14:05:43.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057056759999999998 2023-07-02T14:05:43.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.037936+0000", 2023-07-02T14:05:43.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6334999999999993e-05 2023-07-02T14:05:43.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038012+0000", 2023-07-02T14:05:43.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043985609999999996 2023-07-02T14:05:43.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042411+0000", 2023-07-02T14:05:43.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6857e-05 2023-07-02T14:05:43.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042487+0000", 2023-07-02T14:05:43.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007284446 2023-07-02T14:05:43.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049772+0000", 2023-07-02T14:05:43.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3757000000000006e-05 2023-07-02T14:05:43.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049846+0000", 2023-07-02T14:05:43.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0095186679999999992 2023-07-02T14:05:43.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059364+0000", 2023-07-02T14:05:43.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9931000000000005e-05 2023-07-02T14:05:43.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059434+0000", 2023-07-02T14:05:43.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079904920000000001 2023-07-02T14:05:43.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067425+0000", 2023-07-02T14:05:43.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5583000000000001e-05 2023-07-02T14:05:43.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067460+0000", 2023-07-02T14:05:43.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009114278 2023-07-02T14:05:43.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076575+0000", 2023-07-02T14:05:43.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5880000000000001e-05 2023-07-02T14:05:43.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076620+0000", 2023-07-02T14:05:43.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055118470000000003 2023-07-02T14:05:43.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082132+0000", 2023-07-02T14:05:43.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8306999999999997e-05 2023-07-02T14:05:43.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:43.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082181+0000", 2023-07-02T14:05:43.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026259599999999998 2023-07-02T14:05:43.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:43.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082443+0000", 2023-07-02T14:05:43.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018855339999999999 2023-07-02T14:05:43.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:43.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084329+0000", 2023-07-02T14:05:43.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071877729999999997 2023-07-02T14:05:43.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:43.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091517+0000", 2023-07-02T14:05:43.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3424999999999999e-05 2023-07-02T14:05:43.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:43.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091550+0000", 2023-07-02T14:05:43.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109724 2023-07-02T14:05:43.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:43.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091660+0000", 2023-07-02T14:05:43.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.112277108 2023-07-02T14:05:43.847 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.847 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.847 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.848 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17050 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.848 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:05:43.848 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.74023888800000004, 2023-07-02T14:05:43.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12076701300000001, 2023-07-02T14:05:43.848 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.848 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:43.849 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.849 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.849 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.849 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17050 2023-07-02T14:05:43.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.849 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:05:43.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:05:43.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:43.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:05:43.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0759999999999999e-06 2023-07-02T14:05:43.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979402+0000", 2023-07-02T14:05:43.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0480000000000004e-06 2023-07-02T14:05:43.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979408+0000", 2023-07-02T14:05:43.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9559999999999999e-06 2023-07-02T14:05:43.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979412+0000", 2023-07-02T14:05:43.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013117658000000001 2023-07-02T14:05:43.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992530+0000", 2023-07-02T14:05:43.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4464000000000001e-05 2023-07-02T14:05:43.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992564+0000", 2023-07-02T14:05:43.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0063143640000000003 2023-07-02T14:05:43.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998879+0000", 2023-07-02T14:05:43.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2842000000000002e-05 2023-07-02T14:05:43.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998911+0000", 2023-07-02T14:05:43.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091863190000000001 2023-07-02T14:05:43.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008098+0000", 2023-07-02T14:05:43.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1448999999999999e-05 2023-07-02T14:05:43.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008179+0000", 2023-07-02T14:05:43.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059353720000000004 2023-07-02T14:05:43.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014115+0000", 2023-07-02T14:05:43.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1928000000000002e-05 2023-07-02T14:05:43.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014157+0000", 2023-07-02T14:05:43.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074935210000000004 2023-07-02T14:05:43.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021650+0000", 2023-07-02T14:05:43.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0199999999999999e-05 2023-07-02T14:05:43.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021680+0000", 2023-07-02T14:05:43.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057425109999999996 2023-07-02T14:05:43.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027423+0000", 2023-07-02T14:05:43.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1498999999999998e-05 2023-07-02T14:05:43.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027454+0000", 2023-07-02T14:05:43.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047903420000000004 2023-07-02T14:05:43.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032245+0000", 2023-07-02T14:05:43.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2552999999999998e-05 2023-07-02T14:05:43.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032277+0000", 2023-07-02T14:05:43.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057638079999999996 2023-07-02T14:05:43.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038041+0000", 2023-07-02T14:05:43.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9168000000000002e-05 2023-07-02T14:05:43.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038080+0000", 2023-07-02T14:05:43.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044291799999999996 2023-07-02T14:05:43.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042509+0000", 2023-07-02T14:05:43.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3528999999999997e-05 2023-07-02T14:05:43.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042543+0000", 2023-07-02T14:05:43.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073244729999999997 2023-07-02T14:05:43.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049867+0000", 2023-07-02T14:05:43.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2873999999999997e-05 2023-07-02T14:05:43.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049900+0000", 2023-07-02T14:05:43.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0095670149999999999 2023-07-02T14:05:43.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059467+0000", 2023-07-02T14:05:43.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0406000000000003e-05 2023-07-02T14:05:43.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059538+0000", 2023-07-02T14:05:43.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079635079999999994 2023-07-02T14:05:43.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067501+0000", 2023-07-02T14:05:43.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4795999999999997e-05 2023-07-02T14:05:43.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067536+0000", 2023-07-02T14:05:43.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091231969999999996 2023-07-02T14:05:43.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076659+0000", 2023-07-02T14:05:43.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3208999999999999e-05 2023-07-02T14:05:43.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076702+0000", 2023-07-02T14:05:43.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070660480000000001 2023-07-02T14:05:43.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.083768+0000", 2023-07-02T14:05:43.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5546000000000001e-05 2023-07-02T14:05:43.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.083814+0000", 2023-07-02T14:05:43.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078830659999999993 2023-07-02T14:05:43.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091697+0000", 2023-07-02T14:05:43.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5921000000000003e-05 2023-07-02T14:05:43.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:43.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091743+0000", 2023-07-02T14:05:43.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026571400000000001 2023-07-02T14:05:43.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:43.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.092009+0000", 2023-07-02T14:05:43.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022058429999999999 2023-07-02T14:05:43.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:43.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.094214+0000", 2023-07-02T14:05:43.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057115869999999997 2023-07-02T14:05:43.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:43.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.099926+0000", 2023-07-02T14:05:43.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0287999999999997e-05 2023-07-02T14:05:43.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:43.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.099976+0000", 2023-07-02T14:05:43.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019014999999999999 2023-07-02T14:05:43.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:43.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.100166+0000", 2023-07-02T14:05:43.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12076701300000001 2023-07-02T14:05:43.875 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.875 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.875 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.876 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17095 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.876 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.060121+0000", 2023-07-02T14:05:43.876 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.65951693099999997, 2023-07-02T14:05:43.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.099578634999999999, 2023-07-02T14:05:43.876 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.876 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:43.877 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.877 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.877 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.877 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17095 2023-07-02T14:05:43.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.877 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060121+0000", 2023-07-02T14:05:43.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060121+0000", 2023-07-02T14:05:43.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:43.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060121+0000", 2023-07-02T14:05:43.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5140000000000004e-06 2023-07-02T14:05:43.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060129+0000", 2023-07-02T14:05:43.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4520000000000002e-06 2023-07-02T14:05:43.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060135+0000", 2023-07-02T14:05:43.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2960000000000004e-06 2023-07-02T14:05:43.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060141+0000", 2023-07-02T14:05:43.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00047506399999999998 2023-07-02T14:05:43.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060616+0000", 2023-07-02T14:05:43.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8182999999999997e-05 2023-07-02T14:05:43.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060655+0000", 2023-07-02T14:05:43.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074016079999999996 2023-07-02T14:05:43.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068056+0000", 2023-07-02T14:05:43.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6019000000000005e-05 2023-07-02T14:05:43.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068132+0000", 2023-07-02T14:05:43.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090475429999999999 2023-07-02T14:05:43.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077180+0000", 2023-07-02T14:05:43.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7527999999999997e-05 2023-07-02T14:05:43.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077217+0000", 2023-07-02T14:05:43.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070671400000000004 2023-07-02T14:05:43.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084284+0000", 2023-07-02T14:05:43.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3818e-05 2023-07-02T14:05:43.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084318+0000", 2023-07-02T14:05:43.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093793799999999997 2023-07-02T14:05:43.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093698+0000", 2023-07-02T14:05:43.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2876999999999997e-05 2023-07-02T14:05:43.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093730+0000", 2023-07-02T14:05:43.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088179079999999993 2023-07-02T14:05:43.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102548+0000", 2023-07-02T14:05:43.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5802000000000003e-05 2023-07-02T14:05:43.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102634+0000", 2023-07-02T14:05:43.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099570119999999995 2023-07-02T14:05:43.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112591+0000", 2023-07-02T14:05:43.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6594000000000001e-05 2023-07-02T14:05:43.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112648+0000", 2023-07-02T14:05:43.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009671921 2023-07-02T14:05:43.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122320+0000", 2023-07-02T14:05:43.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2005000000000002e-05 2023-07-02T14:05:43.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122362+0000", 2023-07-02T14:05:43.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011261546000000001 2023-07-02T14:05:43.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133623+0000", 2023-07-02T14:05:43.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9981000000000006e-05 2023-07-02T14:05:43.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133693+0000", 2023-07-02T14:05:43.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093884069999999997 2023-07-02T14:05:43.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.893 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.893 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.143082+0000", 2023-07-02T14:05:43.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103371 2023-07-02T14:05:43.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.143185+0000", 2023-07-02T14:05:43.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051199679999999999 2023-07-02T14:05:43.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.148305+0000", 2023-07-02T14:05:43.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6837999999999998e-05 2023-07-02T14:05:43.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.148392+0000", 2023-07-02T14:05:43.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046966489999999998 2023-07-02T14:05:43.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.153088+0000", 2023-07-02T14:05:43.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5444000000000001e-05 2023-07-02T14:05:43.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:43.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.153134+0000", 2023-07-02T14:05:43.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000238259 2023-07-02T14:05:43.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:43.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.153372+0000", 2023-07-02T14:05:43.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023790500000000002 2023-07-02T14:05:43.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:43.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.155751+0000", 2023-07-02T14:05:43.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038028099999999998 2023-07-02T14:05:43.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:43.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.159554+0000", 2023-07-02T14:05:43.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3145000000000002e-05 2023-07-02T14:05:43.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:43.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.159587+0000", 2023-07-02T14:05:43.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000112948 2023-07-02T14:05:43.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:43.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.159700+0000", 2023-07-02T14:05:43.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.099578634999999999 2023-07-02T14:05:43.900 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.900 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.900 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.900 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17096 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.900 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:05:43.901 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.65947311200000003, 2023-07-02T14:05:43.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.10620154, 2023-07-02T14:05:43.901 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.901 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:43.901 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.901 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.901 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.901 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17096 2023-07-02T14:05:43.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.902 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:05:43.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:05:43.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:43.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:05:43.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3400000000000002e-06 2023-07-02T14:05:43.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060168+0000", 2023-07-02T14:05:43.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4020000000000004e-06 2023-07-02T14:05:43.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060173+0000", 2023-07-02T14:05:43.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4259999999999999e-06 2023-07-02T14:05:43.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060176+0000", 2023-07-02T14:05:43.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00049978399999999997 2023-07-02T14:05:43.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060676+0000", 2023-07-02T14:05:43.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0135000000000004e-05 2023-07-02T14:05:43.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060756+0000", 2023-07-02T14:05:43.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074338060000000003 2023-07-02T14:05:43.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068190+0000", 2023-07-02T14:05:43.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7941000000000004e-05 2023-07-02T14:05:43.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068278+0000", 2023-07-02T14:05:43.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089611870000000007 2023-07-02T14:05:43.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077239+0000", 2023-07-02T14:05:43.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6349e-05 2023-07-02T14:05:43.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077275+0000", 2023-07-02T14:05:43.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007072142 2023-07-02T14:05:43.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084347+0000", 2023-07-02T14:05:43.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.591e-05 2023-07-02T14:05:43.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084383+0000", 2023-07-02T14:05:43.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093660119999999999 2023-07-02T14:05:43.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093749+0000", 2023-07-02T14:05:43.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1538000000000002e-05 2023-07-02T14:05:43.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093781+0000", 2023-07-02T14:05:43.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088884629999999992 2023-07-02T14:05:43.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102669+0000", 2023-07-02T14:05:43.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6032999999999997e-05 2023-07-02T14:05:43.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102725+0000", 2023-07-02T14:05:43.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009945522 2023-07-02T14:05:43.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112671+0000", 2023-07-02T14:05:43.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5741000000000002e-05 2023-07-02T14:05:43.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112707+0000", 2023-07-02T14:05:43.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096867940000000003 2023-07-02T14:05:43.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122393+0000", 2023-07-02T14:05:43.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6244999999999998e-05 2023-07-02T14:05:43.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122480+0000", 2023-07-02T14:05:43.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011251107999999999 2023-07-02T14:05:43.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133731+0000", 2023-07-02T14:05:43.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6400999999999999e-05 2023-07-02T14:05:43.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133767+0000", 2023-07-02T14:05:43.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0094677070000000006 2023-07-02T14:05:43.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.143235+0000", 2023-07-02T14:05:43.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8721999999999996e-05 2023-07-02T14:05:43.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.143314+0000", 2023-07-02T14:05:43.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051196050000000002 2023-07-02T14:05:43.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.148433+0000", 2023-07-02T14:05:43.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2484999999999997e-05 2023-07-02T14:05:43.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.148476+0000", 2023-07-02T14:05:43.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006202234 2023-07-02T14:05:43.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.154678+0000", 2023-07-02T14:05:43.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2699000000000002e-05 2023-07-02T14:05:43.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.154761+0000", 2023-07-02T14:05:43.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049653809999999996 2023-07-02T14:05:43.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.159726+0000", 2023-07-02T14:05:43.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6862e-05 2023-07-02T14:05:43.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:43.922 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.159773+0000", 2023-07-02T14:05:43.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024618099999999999 2023-07-02T14:05:43.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:43.922 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.160019+0000", 2023-07-02T14:05:43.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0019043059999999999 2023-07-02T14:05:43.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:43.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.161923+0000", 2023-07-02T14:05:43.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004316884 2023-07-02T14:05:43.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:43.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.166240+0000", 2023-07-02T14:05:43.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2611000000000003e-05 2023-07-02T14:05:43.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:43.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.166273+0000", 2023-07-02T14:05:43.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4047000000000003e-05 2023-07-02T14:05:43.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:43.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.166367+0000", 2023-07-02T14:05:43.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.10620154 2023-07-02T14:05:43.925 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.925 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.925 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.926 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17204 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.926 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.130952+0000", 2023-07-02T14:05:43.926 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.58868617999999995, 2023-07-02T14:05:43.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.101008762, 2023-07-02T14:05:43.926 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.926 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:43.927 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.927 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.927 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.927 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17204 2023-07-02T14:05:43.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.927 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130952+0000", 2023-07-02T14:05:43.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130952+0000", 2023-07-02T14:05:43.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:43.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130952+0000", 2023-07-02T14:05:43.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7139999999999999e-06 2023-07-02T14:05:43.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130955+0000", 2023-07-02T14:05:43.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.054e-06 2023-07-02T14:05:43.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130962+0000", 2023-07-02T14:05:43.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7610000000000001e-06 2023-07-02T14:05:43.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130965+0000", 2023-07-02T14:05:43.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0034574929999999999 2023-07-02T14:05:43.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.134423+0000", 2023-07-02T14:05:43.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9406999999999997e-05 2023-07-02T14:05:43.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.134462+0000", 2023-07-02T14:05:43.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096105089999999997 2023-07-02T14:05:43.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.144073+0000", 2023-07-02T14:05:43.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3737000000000002e-05 2023-07-02T14:05:43.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.144146+0000", 2023-07-02T14:05:43.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050853909999999999 2023-07-02T14:05:43.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.149232+0000", 2023-07-02T14:05:43.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1114999999999999e-05 2023-07-02T14:05:43.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.149303+0000", 2023-07-02T14:05:43.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060490919999999998 2023-07-02T14:05:43.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.155352+0000", 2023-07-02T14:05:43.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5466000000000002e-05 2023-07-02T14:05:43.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.155388+0000", 2023-07-02T14:05:43.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064891990000000002 2023-07-02T14:05:43.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.161877+0000", 2023-07-02T14:05:43.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9493000000000003e-05 2023-07-02T14:05:43.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.161916+0000", 2023-07-02T14:05:43.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062878689999999998 2023-07-02T14:05:43.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.168204+0000", 2023-07-02T14:05:43.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4718000000000002e-05 2023-07-02T14:05:43.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.168239+0000", 2023-07-02T14:05:43.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043609909999999998 2023-07-02T14:05:43.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.172600+0000", 2023-07-02T14:05:43.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4609e-05 2023-07-02T14:05:43.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.172634+0000", 2023-07-02T14:05:43.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050403130000000003 2023-07-02T14:05:43.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.177675+0000", 2023-07-02T14:05:43.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.523e-05 2023-07-02T14:05:43.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.177710+0000", 2023-07-02T14:05:43.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043389429999999996 2023-07-02T14:05:43.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.182049+0000", 2023-07-02T14:05:43.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3048999999999998e-05 2023-07-02T14:05:43.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.182122+0000", 2023-07-02T14:05:43.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041995360000000002 2023-07-02T14:05:43.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.186321+0000", 2023-07-02T14:05:43.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2482000000000001e-05 2023-07-02T14:05:43.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.186404+0000", 2023-07-02T14:05:43.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013084261 2023-07-02T14:05:43.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.199488+0000", 2023-07-02T14:05:43.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013297600000000001 2023-07-02T14:05:43.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.199621+0000", 2023-07-02T14:05:43.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014611084999999999 2023-07-02T14:05:43.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.214232+0000", 2023-07-02T14:05:43.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7821000000000001e-05 2023-07-02T14:05:43.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.214290+0000", 2023-07-02T14:05:43.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089499660000000002 2023-07-02T14:05:43.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.223240+0000", 2023-07-02T14:05:43.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010005000000000001 2023-07-02T14:05:43.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:43.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.223340+0000", 2023-07-02T14:05:43.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00049127800000000005 2023-07-02T14:05:43.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:43.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.223831+0000", 2023-07-02T14:05:43.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064706440000000002 2023-07-02T14:05:43.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:43.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.230302+0000", 2023-07-02T14:05:43.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0015080759999999999 2023-07-02T14:05:43.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:43.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.231810+0000", 2023-07-02T14:05:43.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6409000000000003e-05 2023-07-02T14:05:43.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:43.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.231847+0000", 2023-07-02T14:05:43.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011443 2023-07-02T14:05:43.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:43.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.231961+0000", 2023-07-02T14:05:43.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.101008762 2023-07-02T14:05:43.951 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.951 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.951 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.951 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17289 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.951 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.253143+0000", 2023-07-02T14:05:43.952 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.46649542100000002, 2023-07-02T14:05:43.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.093423744000000003, 2023-07-02T14:05:43.952 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.952 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:43.952 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.952 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.952 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.952 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17289 2023-07-02T14:05:43.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.953 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.253143+0000", 2023-07-02T14:05:43.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.954 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.253143+0000", 2023-07-02T14:05:43.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:43.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.954 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.253142+0000", 2023-07-02T14:05:43.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.651e-06 2023-07-02T14:05:43.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.253147+0000", 2023-07-02T14:05:43.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7580000000000004e-06 2023-07-02T14:05:43.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.253157+0000", 2023-07-02T14:05:43.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1620000000000001e-06 2023-07-02T14:05:43.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.253161+0000", 2023-07-02T14:05:43.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9446999999999997e-05 2023-07-02T14:05:43.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.253200+0000", 2023-07-02T14:05:43.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1219999999999998e-05 2023-07-02T14:05:43.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.253281+0000", 2023-07-02T14:05:43.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004960379 2023-07-02T14:05:43.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.258242+0000", 2023-07-02T14:05:43.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2092000000000001e-05 2023-07-02T14:05:43.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.258304+0000", 2023-07-02T14:05:43.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066845969999999996 2023-07-02T14:05:43.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.264988+0000", 2023-07-02T14:05:43.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2184000000000002e-05 2023-07-02T14:05:43.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.265021+0000", 2023-07-02T14:05:43.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089431939999999998 2023-07-02T14:05:43.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.273964+0000", 2023-07-02T14:05:43.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3935e-05 2023-07-02T14:05:43.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.273998+0000", 2023-07-02T14:05:43.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013312146 2023-07-02T14:05:43.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.287310+0000", 2023-07-02T14:05:43.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8985000000000004e-05 2023-07-02T14:05:43.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.287389+0000", 2023-07-02T14:05:43.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069356799999999996 2023-07-02T14:05:43.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.294325+0000", 2023-07-02T14:05:43.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1613000000000005e-05 2023-07-02T14:05:43.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.294416+0000", 2023-07-02T14:05:43.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077370850000000003 2023-07-02T14:05:43.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302153+0000", 2023-07-02T14:05:43.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1329999999999999e-05 2023-07-02T14:05:43.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302195+0000", 2023-07-02T14:05:43.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0095656539999999998 2023-07-02T14:05:43.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.311760+0000", 2023-07-02T14:05:43.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1889000000000002e-05 2023-07-02T14:05:43.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.311842+0000", 2023-07-02T14:05:43.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098551019999999993 2023-07-02T14:05:43.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.321697+0000", 2023-07-02T14:05:43.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3013000000000003e-05 2023-07-02T14:05:43.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.321740+0000", 2023-07-02T14:05:43.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099913020000000005 2023-07-02T14:05:43.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.331732+0000", 2023-07-02T14:05:43.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7336999999999995e-05 2023-07-02T14:05:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.331809+0000", 2023-07-02T14:05:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071936229999999997 2023-07-02T14:05:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.339003+0000", 2023-07-02T14:05:43.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4049999999999997e-05 2023-07-02T14:05:43.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:43.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.339087+0000", 2023-07-02T14:05:43.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00029404499999999999 2023-07-02T14:05:43.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:43.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.339381+0000", 2023-07-02T14:05:43.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024404240000000001 2023-07-02T14:05:43.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:43.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341821+0000", 2023-07-02T14:05:43.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00460305 2023-07-02T14:05:43.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:43.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.346424+0000", 2023-07-02T14:05:43.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0695999999999999e-05 2023-07-02T14:05:43.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:43.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.346455+0000", 2023-07-02T14:05:43.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000111871 2023-07-02T14:05:43.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:43.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.346567+0000", 2023-07-02T14:05:43.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.093423744000000003 2023-07-02T14:05:43.974 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.974 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.974 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.974 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17294 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.974 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.256047+0000", 2023-07-02T14:05:43.975 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.46359146400000001, 2023-07-02T14:05:43.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.099803734000000005, 2023-07-02T14:05:43.975 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.975 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:43.975 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.975 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.975 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.975 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17294 2023-07-02T14:05:43.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.976 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256047+0000", 2023-07-02T14:05:43.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:43.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256047+0000", 2023-07-02T14:05:43.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:43.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:43.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256047+0000", 2023-07-02T14:05:43.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.367e-06 2023-07-02T14:05:43.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:43.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256049+0000", 2023-07-02T14:05:43.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0019999999999998e-06 2023-07-02T14:05:43.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:43.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256052+0000", 2023-07-02T14:05:43.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9450000000000002e-06 2023-07-02T14:05:43.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:43.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256055+0000", 2023-07-02T14:05:43.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022819160000000002 2023-07-02T14:05:43.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.258337+0000", 2023-07-02T14:05:43.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1145999999999999e-05 2023-07-02T14:05:43.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.258418+0000", 2023-07-02T14:05:43.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066309109999999997 2023-07-02T14:05:43.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.265049+0000", 2023-07-02T14:05:43.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6449000000000002e-05 2023-07-02T14:05:43.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.265095+0000", 2023-07-02T14:05:43.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089284169999999993 2023-07-02T14:05:43.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.274024+0000", 2023-07-02T14:05:43.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7626999999999997e-05 2023-07-02T14:05:43.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.274061+0000", 2023-07-02T14:05:43.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013396880999999999 2023-07-02T14:05:43.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.287458+0000", 2023-07-02T14:05:43.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1865000000000003e-05 2023-07-02T14:05:43.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.287540+0000", 2023-07-02T14:05:43.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069236419999999998 2023-07-02T14:05:43.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.294464+0000", 2023-07-02T14:05:43.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7681999999999993e-05 2023-07-02T14:05:43.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.294531+0000", 2023-07-02T14:05:43.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076921070000000001 2023-07-02T14:05:43.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302223+0000", 2023-07-02T14:05:43.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5528999999999998e-05 2023-07-02T14:05:43.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302259+0000", 2023-07-02T14:05:43.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096256480000000005 2023-07-02T14:05:43.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.311885+0000", 2023-07-02T14:05:43.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6389e-05 2023-07-02T14:05:43.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.311921+0000", 2023-07-02T14:05:43.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098685439999999999 2023-07-02T14:05:43.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.321790+0000", 2023-07-02T14:05:43.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1817999999999995e-05 2023-07-02T14:05:43.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.321871+0000", 2023-07-02T14:05:43.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009974966 2023-07-02T14:05:43.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.331846+0000", 2023-07-02T14:05:43.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6520000000000002e-05 2023-07-02T14:05:43.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.331893+0000", 2023-07-02T14:05:43.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086651149999999993 2023-07-02T14:05:43.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.340558+0000", 2023-07-02T14:05:43.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5918000000000001e-05 2023-07-02T14:05:43.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:43.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.340634+0000", 2023-07-02T14:05:43.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059983099999999998 2023-07-02T14:05:43.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:43.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.346632+0000", 2023-07-02T14:05:43.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7887999999999997e-05 2023-07-02T14:05:43.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:43.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.346710+0000", 2023-07-02T14:05:43.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00046673100000000001 2023-07-02T14:05:43.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:43.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.347177+0000", 2023-07-02T14:05:43.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027484829999999999 2023-07-02T14:05:43.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:43.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349925+0000", 2023-07-02T14:05:43.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056857970000000002 2023-07-02T14:05:43.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:43.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.355611+0000", 2023-07-02T14:05:43.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9406000000000001e-05 2023-07-02T14:05:43.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:43.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.355661+0000", 2023-07-02T14:05:43.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019010399999999999 2023-07-02T14:05:43.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:43.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.355851+0000", 2023-07-02T14:05:43.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.099803734000000005 2023-07-02T14:05:43.996 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.997 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:43.997 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:43.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.997 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17295 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:43.997 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.256057+0000", 2023-07-02T14:05:43.997 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.463581244, 2023-07-02T14:05:43.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.107964279, 2023-07-02T14:05:43.998 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:43.998 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:43.998 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:43.998 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:43.998 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:43.998 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17295 2023-07-02T14:05:43.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.998 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:43.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:43.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256057+0000", 2023-07-02T14:05:43.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:43.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:43.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:43.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:44.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256057+0000", 2023-07-02T14:05:44.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:44.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:44.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256057+0000", 2023-07-02T14:05:44.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.458e-06 2023-07-02T14:05:44.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:44.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256059+0000", 2023-07-02T14:05:44.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9019999999999999e-06 2023-07-02T14:05:44.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:44.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256062+0000", 2023-07-02T14:05:44.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9929999999999998e-06 2023-07-02T14:05:44.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256064+0000", 2023-07-02T14:05:44.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023724789999999998 2023-07-02T14:05:44.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.258437+0000", 2023-07-02T14:05:44.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7698000000000002e-05 2023-07-02T14:05:44.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.258494+0000", 2023-07-02T14:05:44.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066424140000000001 2023-07-02T14:05:44.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.265137+0000", 2023-07-02T14:05:44.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2169000000000002e-05 2023-07-02T14:05:44.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.265179+0000", 2023-07-02T14:05:44.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089047970000000008 2023-07-02T14:05:44.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.274084+0000", 2023-07-02T14:05:44.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7489e-05 2023-07-02T14:05:44.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.274121+0000", 2023-07-02T14:05:44.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013452426999999999 2023-07-02T14:05:44.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.287574+0000", 2023-07-02T14:05:44.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5149999999999996e-05 2023-07-02T14:05:44.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.287659+0000", 2023-07-02T14:05:44.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069046059999999998 2023-07-02T14:05:44.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.294563+0000", 2023-07-02T14:05:44.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7073000000000001e-05 2023-07-02T14:05:44.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.294611+0000", 2023-07-02T14:05:44.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076769480000000003 2023-07-02T14:05:44.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302287+0000", 2023-07-02T14:05:44.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4381000000000002e-05 2023-07-02T14:05:44.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302322+0000", 2023-07-02T14:05:44.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009637863 2023-07-02T14:05:44.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.311960+0000", 2023-07-02T14:05:44.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4847999999999999e-05 2023-07-02T14:05:44.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312035+0000", 2023-07-02T14:05:44.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098741579999999992 2023-07-02T14:05:44.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.321909+0000", 2023-07-02T14:05:44.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6455999999999998e-05 2023-07-02T14:05:44.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.321945+0000", 2023-07-02T14:05:44.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099894690000000008 2023-07-02T14:05:44.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.331935+0000", 2023-07-02T14:05:44.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2550000000000002e-05 2023-07-02T14:05:44.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332007+0000", 2023-07-02T14:05:44.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086760090000000002 2023-07-02T14:05:44.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.340683+0000", 2023-07-02T14:05:44.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8462999999999998e-05 2023-07-02T14:05:44.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.340752+0000", 2023-07-02T14:05:44.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00830381 2023-07-02T14:05:44.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349055+0000", 2023-07-02T14:05:44.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9289999999999994e-05 2023-07-02T14:05:44.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349145+0000", 2023-07-02T14:05:44.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067717250000000001 2023-07-02T14:05:44.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.355917+0000", 2023-07-02T14:05:44.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4431999999999995e-05 2023-07-02T14:05:44.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:44.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.355991+0000", 2023-07-02T14:05:44.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00044773399999999998 2023-07-02T14:05:44.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:44.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.356439+0000", 2023-07-02T14:05:44.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028403690000000001 2023-07-02T14:05:44.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:44.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.359279+0000", 2023-07-02T14:05:44.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045352450000000002 2023-07-02T14:05:44.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:44.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.363814+0000", 2023-07-02T14:05:44.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6167e-05 2023-07-02T14:05:44.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:44.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.363860+0000", 2023-07-02T14:05:44.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000160951 2023-07-02T14:05:44.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:44.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.364021+0000", 2023-07-02T14:05:44.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.107964279 2023-07-02T14:05:44.021 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:44.021 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:44.021 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:44.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.021 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17351 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:44.022 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302537+0000", 2023-07-02T14:05:44.022 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.41710151000000001, 2023-07-02T14:05:44.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.099134853999999994, 2023-07-02T14:05:44.022 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:44.022 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:44.022 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:44.022 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:44.022 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:44.023 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17351 2023-07-02T14:05:44.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.023 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:44.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:44.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302537+0000", 2023-07-02T14:05:44.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:44.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:44.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302537+0000", 2023-07-02T14:05:44.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:44.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:44.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302536+0000", 2023-07-02T14:05:44.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3329999999999996e-06 2023-07-02T14:05:44.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:44.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302541+0000", 2023-07-02T14:05:44.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6600000000000001e-06 2023-07-02T14:05:44.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:44.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302550+0000", 2023-07-02T14:05:44.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9e-06 2023-07-02T14:05:44.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302557+0000", 2023-07-02T14:05:44.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8735999999999998e-05 2023-07-02T14:05:44.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302656+0000", 2023-07-02T14:05:44.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7564000000000002e-05 2023-07-02T14:05:44.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302693+0000", 2023-07-02T14:05:44.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096663749999999996 2023-07-02T14:05:44.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312360+0000", 2023-07-02T14:05:44.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6055000000000003e-05 2023-07-02T14:05:44.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312436+0000", 2023-07-02T14:05:44.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098653899999999999 2023-07-02T14:05:44.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322301+0000", 2023-07-02T14:05:44.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7258999999999997e-05 2023-07-02T14:05:44.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322338+0000", 2023-07-02T14:05:44.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099812860000000007 2023-07-02T14:05:44.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332320+0000", 2023-07-02T14:05:44.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3201999999999995e-05 2023-07-02T14:05:44.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332393+0000", 2023-07-02T14:05:44.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088690169999999999 2023-07-02T14:05:44.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341262+0000", 2023-07-02T14:05:44.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1718999999999995e-05 2023-07-02T14:05:44.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341344+0000", 2023-07-02T14:05:44.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008184574 2023-07-02T14:05:44.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349528+0000", 2023-07-02T14:05:44.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2019000000000001e-05 2023-07-02T14:05:44.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349620+0000", 2023-07-02T14:05:44.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089525679999999993 2023-07-02T14:05:44.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358573+0000", 2023-07-02T14:05:44.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5639e-05 2023-07-02T14:05:44.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358638+0000", 2023-07-02T14:05:44.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079895899999999995 2023-07-02T14:05:44.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366628+0000", 2023-07-02T14:05:44.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5162999999999997e-05 2023-07-02T14:05:44.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366723+0000", 2023-07-02T14:05:44.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074560440000000002 2023-07-02T14:05:44.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374179+0000", 2023-07-02T14:05:44.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9793999999999999e-05 2023-07-02T14:05:44.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374219+0000", 2023-07-02T14:05:44.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091487210000000003 2023-07-02T14:05:44.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383368+0000", 2023-07-02T14:05:44.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7316000000000004e-05 2023-07-02T14:05:44.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383435+0000", 2023-07-02T14:05:44.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010649080999999999 2023-07-02T14:05:44.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.394084+0000", 2023-07-02T14:05:44.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8598999999999997e-05 2023-07-02T14:05:44.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:44.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.394173+0000", 2023-07-02T14:05:44.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042554099999999998 2023-07-02T14:05:44.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:44.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.394598+0000", 2023-07-02T14:05:44.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048553529999999998 2023-07-02T14:05:44.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:44.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.399454+0000", 2023-07-02T14:05:44.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020927350000000001 2023-07-02T14:05:44.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:44.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401546+0000", 2023-07-02T14:05:44.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.7813999999999998e-05 2023-07-02T14:05:44.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:44.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401574+0000", 2023-07-02T14:05:44.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7508000000000002e-05 2023-07-02T14:05:44.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:44.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401672+0000", 2023-07-02T14:05:44.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.099134853999999994 2023-07-02T14:05:44.044 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:44.044 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:44.044 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:44.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.044 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17352 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:44.044 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302565+0000", 2023-07-02T14:05:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.41707302800000001, 2023-07-02T14:05:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.105667996, 2023-07-02T14:05:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:44.045 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17352 2023-07-02T14:05:44.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.046 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:44.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:44.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302565+0000", 2023-07-02T14:05:44.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:44.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:44.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302565+0000", 2023-07-02T14:05:44.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:44.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:44.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302565+0000", 2023-07-02T14:05:44.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7000000000000002e-06 2023-07-02T14:05:44.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:44.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302568+0000", 2023-07-02T14:05:44.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2140000000000001e-06 2023-07-02T14:05:44.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:44.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302575+0000", 2023-07-02T14:05:44.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3250000000000001e-06 2023-07-02T14:05:44.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302579+0000", 2023-07-02T14:05:44.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014498700000000001 2023-07-02T14:05:44.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302724+0000", 2023-07-02T14:05:44.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5444999999999993e-05 2023-07-02T14:05:44.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302799+0000", 2023-07-02T14:05:44.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096710000000000008 2023-07-02T14:05:44.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312470+0000", 2023-07-02T14:05:44.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4798999999999997e-05 2023-07-02T14:05:44.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312505+0000", 2023-07-02T14:05:44.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098663940000000006 2023-07-02T14:05:44.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322372+0000", 2023-07-02T14:05:44.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1996000000000002e-05 2023-07-02T14:05:44.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322414+0000", 2023-07-02T14:05:44.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010015735 2023-07-02T14:05:44.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332429+0000", 2023-07-02T14:05:44.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2929000000000003e-05 2023-07-02T14:05:44.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332462+0000", 2023-07-02T14:05:44.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089318550000000007 2023-07-02T14:05:44.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341394+0000", 2023-07-02T14:05:44.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7248e-05 2023-07-02T14:05:44.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341431+0000", 2023-07-02T14:05:44.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0082266739999999998 2023-07-02T14:05:44.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349658+0000", 2023-07-02T14:05:44.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3188999999999997e-05 2023-07-02T14:05:44.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349691+0000", 2023-07-02T14:05:44.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089857329999999992 2023-07-02T14:05:44.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358677+0000", 2023-07-02T14:05:44.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6436000000000001e-05 2023-07-02T14:05:44.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358713+0000", 2023-07-02T14:05:44.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080528400000000003 2023-07-02T14:05:44.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366766+0000", 2023-07-02T14:05:44.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3217999999999999e-05 2023-07-02T14:05:44.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366799+0000", 2023-07-02T14:05:44.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074649970000000001 2023-07-02T14:05:44.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374264+0000", 2023-07-02T14:05:44.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9482999999999999e-05 2023-07-02T14:05:44.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374334+0000", 2023-07-02T14:05:44.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091483889999999998 2023-07-02T14:05:44.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383482+0000", 2023-07-02T14:05:44.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3095000000000005e-05 2023-07-02T14:05:44.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383545+0000", 2023-07-02T14:05:44.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013359414 2023-07-02T14:05:44.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.396905+0000", 2023-07-02T14:05:44.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012269900000000001 2023-07-02T14:05:44.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397028+0000", 2023-07-02T14:05:44.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046629660000000002 2023-07-02T14:05:44.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401691+0000", 2023-07-02T14:05:44.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5111999999999999e-05 2023-07-02T14:05:44.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:44.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401726+0000", 2023-07-02T14:05:44.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024867800000000001 2023-07-02T14:05:44.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:44.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401974+0000", 2023-07-02T14:05:44.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028409360000000001 2023-07-02T14:05:44.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:44.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404815+0000", 2023-07-02T14:05:44.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031148529999999999 2023-07-02T14:05:44.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:44.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.407930+0000", 2023-07-02T14:05:44.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6836000000000003e-05 2023-07-02T14:05:44.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:44.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.407987+0000", 2023-07-02T14:05:44.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024641799999999998 2023-07-02T14:05:44.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:44.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408233+0000", 2023-07-02T14:05:44.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.105667996 2023-07-02T14:05:44.068 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:44.068 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:44.068 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:44.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.069 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17353 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:44.069 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:05:44.069 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.41705421199999998, 2023-07-02T14:05:44.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.115724379, 2023-07-02T14:05:44.069 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:44.069 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:44.069 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:44.069 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:44.070 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:44.070 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17353 2023-07-02T14:05:44.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.070 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:44.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:44.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:05:44.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:44.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:44.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:05:44.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:44.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:44.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:05:44.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.41e-06 2023-07-02T14:05:44.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:44.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302587+0000", 2023-07-02T14:05:44.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.015e-06 2023-07-02T14:05:44.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:44.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302593+0000", 2023-07-02T14:05:44.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8019999999999997e-06 2023-07-02T14:05:44.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302599+0000", 2023-07-02T14:05:44.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022701199999999999 2023-07-02T14:05:44.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302826+0000", 2023-07-02T14:05:44.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.426e-05 2023-07-02T14:05:44.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302870+0000", 2023-07-02T14:05:44.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096759489999999997 2023-07-02T14:05:44.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312546+0000", 2023-07-02T14:05:44.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5785000000000006e-05 2023-07-02T14:05:44.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312632+0000", 2023-07-02T14:05:44.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098106970000000002 2023-07-02T14:05:44.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322443+0000", 2023-07-02T14:05:44.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3161999999999999e-05 2023-07-02T14:05:44.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322476+0000", 2023-07-02T14:05:44.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010015823 2023-07-02T14:05:44.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332492+0000", 2023-07-02T14:05:44.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1165999999999999e-05 2023-07-02T14:05:44.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332533+0000", 2023-07-02T14:05:44.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089435269999999997 2023-07-02T14:05:44.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341476+0000", 2023-07-02T14:05:44.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000112691 2023-07-02T14:05:44.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341589+0000", 2023-07-02T14:05:44.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008147362 2023-07-02T14:05:44.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349736+0000", 2023-07-02T14:05:44.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5149000000000007e-05 2023-07-02T14:05:44.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349832+0000", 2023-07-02T14:05:44.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089226519999999997 2023-07-02T14:05:44.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358754+0000", 2023-07-02T14:05:44.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6091999999999998e-05 2023-07-02T14:05:44.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358810+0000", 2023-07-02T14:05:44.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080318000000000004 2023-07-02T14:05:44.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366842+0000", 2023-07-02T14:05:44.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4321999999999998e-05 2023-07-02T14:05:44.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366916+0000", 2023-07-02T14:05:44.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074664049999999997 2023-07-02T14:05:44.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374383+0000", 2023-07-02T14:05:44.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5460000000000001e-05 2023-07-02T14:05:44.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374438+0000", 2023-07-02T14:05:44.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091562450000000004 2023-07-02T14:05:44.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383594+0000", 2023-07-02T14:05:44.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2545000000000004e-05 2023-07-02T14:05:44.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383667+0000", 2023-07-02T14:05:44.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013424551 2023-07-02T14:05:44.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397092+0000", 2023-07-02T14:05:44.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6991999999999997e-05 2023-07-02T14:05:44.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397169+0000", 2023-07-02T14:05:44.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064399139999999997 2023-07-02T14:05:44.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403608+0000", 2023-07-02T14:05:44.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2614000000000006e-05 2023-07-02T14:05:44.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403691+0000", 2023-07-02T14:05:44.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004611693 2023-07-02T14:05:44.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408303+0000", 2023-07-02T14:05:44.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1935999999999995e-05 2023-07-02T14:05:44.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:44.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408395+0000", 2023-07-02T14:05:44.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00051077700000000002 2023-07-02T14:05:44.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:44.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408906+0000", 2023-07-02T14:05:44.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090767460000000001 2023-07-02T14:05:44.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:44.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.417982+0000", 2023-07-02T14:05:44.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4283999999999997e-05 2023-07-02T14:05:44.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:44.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418037+0000", 2023-07-02T14:05:44.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9667999999999999e-05 2023-07-02T14:05:44.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:44.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418096+0000", 2023-07-02T14:05:44.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021233800000000001 2023-07-02T14:05:44.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:44.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418309+0000", 2023-07-02T14:05:44.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.115724379 2023-07-02T14:05:44.093 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:44.093 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:44.094 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:44.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.094 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17354 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:44.094 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:05:44.094 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.41703525200000002, 2023-07-02T14:05:44.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493, 2023-07-02T14:05:44.094 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:44.095 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:44.095 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:44.095 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:44.095 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:44.095 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17354 2023-07-02T14:05:44.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.095 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:44.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:44.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:05:44.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:44.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:44.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:05:44.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:44.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:44.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:05:44.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7440000000000001e-06 2023-07-02T14:05:44.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:44.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302606+0000", 2023-07-02T14:05:44.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:05:44.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:44.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302613+0000", 2023-07-02T14:05:44.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8810000000000004e-06 2023-07-02T14:05:44.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302618+0000", 2023-07-02T14:05:44.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028613100000000002 2023-07-02T14:05:44.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302904+0000", 2023-07-02T14:05:44.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5278999999999995e-05 2023-07-02T14:05:44.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302979+0000", 2023-07-02T14:05:44.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096904750000000005 2023-07-02T14:05:44.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312669+0000", 2023-07-02T14:05:44.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0328000000000002e-05 2023-07-02T14:05:44.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312700+0000", 2023-07-02T14:05:44.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098019319999999993 2023-07-02T14:05:44.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322502+0000", 2023-07-02T14:05:44.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4669999999999998e-05 2023-07-02T14:05:44.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322536+0000", 2023-07-02T14:05:44.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010023839 2023-07-02T14:05:44.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332560+0000", 2023-07-02T14:05:44.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5559999999999998e-05 2023-07-02T14:05:44.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332596+0000", 2023-07-02T14:05:44.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090401319999999993 2023-07-02T14:05:44.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341636+0000", 2023-07-02T14:05:44.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5373000000000001e-05 2023-07-02T14:05:44.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341681+0000", 2023-07-02T14:05:44.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081994059999999994 2023-07-02T14:05:44.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349881+0000", 2023-07-02T14:05:44.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4319000000000001e-05 2023-07-02T14:05:44.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349915+0000", 2023-07-02T14:05:44.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089388339999999997 2023-07-02T14:05:44.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358854+0000", 2023-07-02T14:05:44.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4319000000000005e-05 2023-07-02T14:05:44.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358928+0000", 2023-07-02T14:05:44.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080234690000000001 2023-07-02T14:05:44.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366952+0000", 2023-07-02T14:05:44.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2082999999999998e-05 2023-07-02T14:05:44.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366984+0000", 2023-07-02T14:05:44.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075015790000000004 2023-07-02T14:05:44.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374485+0000", 2023-07-02T14:05:44.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4480999999999995e-05 2023-07-02T14:05:44.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374550+0000", 2023-07-02T14:05:44.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091666620000000008 2023-07-02T14:05:44.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383716+0000", 2023-07-02T14:05:44.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9318000000000003e-05 2023-07-02T14:05:44.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383786+0000", 2023-07-02T14:05:44.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013434115 2023-07-02T14:05:44.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397220+0000", 2023-07-02T14:05:44.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4573999999999997e-05 2023-07-02T14:05:44.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397264+0000", 2023-07-02T14:05:44.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064824649999999998 2023-07-02T14:05:44.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403747+0000", 2023-07-02T14:05:44.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1087999999999993e-05 2023-07-02T14:05:44.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403828+0000", 2023-07-02T14:05:44.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072854449999999998 2023-07-02T14:05:44.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411113+0000", 2023-07-02T14:05:44.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5120000000000001e-05 2023-07-02T14:05:44.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411169+0000", 2023-07-02T14:05:44.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007211505 2023-07-02T14:05:44.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418380+0000", 2023-07-02T14:05:44.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011289399999999999 2023-07-02T14:05:44.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:44.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418493+0000", 2023-07-02T14:05:44.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00051217500000000002 2023-07-02T14:05:44.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:44.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.419005+0000", 2023-07-02T14:05:44.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00327393 2023-07-02T14:05:44.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:44.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422279+0000", 2023-07-02T14:05:44.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041432350000000003 2023-07-02T14:05:44.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:44.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426422+0000", 2023-07-02T14:05:44.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7021000000000003e-05 2023-07-02T14:05:44.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:44.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426469+0000", 2023-07-02T14:05:44.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000190256 2023-07-02T14:05:44.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:44.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426660+0000", 2023-07-02T14:05:44.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493 2023-07-02T14:05:44.120 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:44.120 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:44.120 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:44.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.121 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17355 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:44.121 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:05:44.121 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.41701659099999999, 2023-07-02T14:05:44.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001, 2023-07-02T14:05:44.121 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:44.121 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:44.121 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:44.122 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:44.122 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:44.122 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17355 2023-07-02T14:05:44.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.122 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:44.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:44.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:05:44.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:44.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:44.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:05:44.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:44.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:44.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302621+0000", 2023-07-02T14:05:44.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2260000000000001e-06 2023-07-02T14:05:44.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:44.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302625+0000", 2023-07-02T14:05:44.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8599999999999998e-06 2023-07-02T14:05:44.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:44.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302630+0000", 2023-07-02T14:05:44.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0629999999999999e-06 2023-07-02T14:05:44.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302635+0000", 2023-07-02T14:05:44.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035731199999999998 2023-07-02T14:05:44.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302992+0000", 2023-07-02T14:05:44.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4164e-05 2023-07-02T14:05:44.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.303026+0000", 2023-07-02T14:05:44.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097115900000000008 2023-07-02T14:05:44.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312738+0000", 2023-07-02T14:05:44.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1701999999999998e-05 2023-07-02T14:05:44.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312819+0000", 2023-07-02T14:05:44.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097363640000000008 2023-07-02T14:05:44.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322556+0000", 2023-07-02T14:05:44.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2517000000000001e-05 2023-07-02T14:05:44.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322588+0000", 2023-07-02T14:05:44.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01002393 2023-07-02T14:05:44.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332612+0000", 2023-07-02T14:05:44.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2960999999999998e-05 2023-07-02T14:05:44.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332645+0000", 2023-07-02T14:05:44.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090669819999999995 2023-07-02T14:05:44.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341712+0000", 2023-07-02T14:05:44.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9288000000000002e-05 2023-07-02T14:05:44.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341811+0000", 2023-07-02T14:05:44.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081504460000000004 2023-07-02T14:05:44.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349962+0000", 2023-07-02T14:05:44.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107321 2023-07-02T14:05:44.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.350069+0000", 2023-07-02T14:05:44.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088790999999999991 2023-07-02T14:05:44.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358948+0000", 2023-07-02T14:05:44.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1711000000000002e-05 2023-07-02T14:05:44.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358980+0000", 2023-07-02T14:05:44.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080367579999999997 2023-07-02T14:05:44.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367017+0000", 2023-07-02T14:05:44.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6329000000000002e-05 2023-07-02T14:05:44.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367073+0000", 2023-07-02T14:05:44.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075097300000000001 2023-07-02T14:05:44.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374583+0000", 2023-07-02T14:05:44.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5710999999999995e-05 2023-07-02T14:05:44.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374648+0000", 2023-07-02T14:05:44.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009165912 2023-07-02T14:05:44.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383814+0000", 2023-07-02T14:05:44.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3916000000000003e-05 2023-07-02T14:05:44.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383878+0000", 2023-07-02T14:05:44.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013440532999999999 2023-07-02T14:05:44.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397319+0000", 2023-07-02T14:05:44.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7218999999999995e-05 2023-07-02T14:05:44.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397386+0000", 2023-07-02T14:05:44.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065165279999999997 2023-07-02T14:05:44.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403903+0000", 2023-07-02T14:05:44.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8152000000000005e-05 2023-07-02T14:05:44.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404001+0000", 2023-07-02T14:05:44.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072185360000000002 2023-07-02T14:05:44.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411219+0000", 2023-07-02T14:05:44.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:05:44.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411310+0000", 2023-07-02T14:05:44.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010021663 2023-07-02T14:05:44.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421331+0000", 2023-07-02T14:05:44.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0148000000000006e-05 2023-07-02T14:05:44.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421411+0000", 2023-07-02T14:05:44.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053056170000000003 2023-07-02T14:05:44.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426717+0000", 2023-07-02T14:05:44.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8242000000000001e-05 2023-07-02T14:05:44.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:44.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426795+0000", 2023-07-02T14:05:44.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00041308399999999998 2023-07-02T14:05:44.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:44.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.427208+0000", 2023-07-02T14:05:44.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028330849999999999 2023-07-02T14:05:44.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:44.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430041+0000", 2023-07-02T14:05:44.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00072642300000000002 2023-07-02T14:05:44.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:44.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430768+0000", 2023-07-02T14:05:44.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1847000000000002e-05 2023-07-02T14:05:44.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:44.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430800+0000", 2023-07-02T14:05:44.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9308999999999994e-05 2023-07-02T14:05:44.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:44.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430899+0000", 2023-07-02T14:05:44.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001 2023-07-02T14:05:44.148 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:44.148 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:44.148 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:44.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.149 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17489 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:44.149 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.389780+0000", 2023-07-02T14:05:44.149 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.32985810100000001, 2023-07-02T14:05:44.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.093494121999999999, 2023-07-02T14:05:44.149 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:44.149 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:44.149 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:44.149 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:44.150 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:44.150 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17489 2023-07-02T14:05:44.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.150 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:44.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:44.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.389780+0000", 2023-07-02T14:05:44.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:44.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:44.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.389780+0000", 2023-07-02T14:05:44.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:44.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:44.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.389779+0000", 2023-07-02T14:05:44.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9250000000000003e-06 2023-07-02T14:05:44.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:44.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.389784+0000", 2023-07-02T14:05:44.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2860000000000002e-06 2023-07-02T14:05:44.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:44.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.389792+0000", 2023-07-02T14:05:44.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6720000000000003e-06 2023-07-02T14:05:44.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.389796+0000", 2023-07-02T14:05:44.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5188000000000003e-05 2023-07-02T14:05:44.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.389831+0000", 2023-07-02T14:05:44.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5021999999999997e-05 2023-07-02T14:05:44.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.389877+0000", 2023-07-02T14:05:44.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079433630000000002 2023-07-02T14:05:44.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397820+0000", 2023-07-02T14:05:44.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4178999999999999e-05 2023-07-02T14:05:44.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397854+0000", 2023-07-02T14:05:44.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006876966 2023-07-02T14:05:44.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404731+0000", 2023-07-02T14:05:44.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5018999999999994e-05 2023-07-02T14:05:44.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404806+0000", 2023-07-02T14:05:44.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007040892 2023-07-02T14:05:44.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411847+0000", 2023-07-02T14:05:44.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6153000000000001e-05 2023-07-02T14:05:44.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411923+0000", 2023-07-02T14:05:44.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010256862 2023-07-02T14:05:44.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422180+0000", 2023-07-02T14:05:44.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9196999999999994e-05 2023-07-02T14:05:44.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422259+0000", 2023-07-02T14:05:44.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006858121 2023-07-02T14:05:44.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.429117+0000", 2023-07-02T14:05:44.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7781999999999998e-05 2023-07-02T14:05:44.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.429155+0000", 2023-07-02T14:05:44.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037400139999999998 2023-07-02T14:05:44.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.432895+0000", 2023-07-02T14:05:44.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2409000000000001e-05 2023-07-02T14:05:44.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.432927+0000", 2023-07-02T14:05:44.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041665080000000002 2023-07-02T14:05:44.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.437094+0000", 2023-07-02T14:05:44.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8099000000000001e-05 2023-07-02T14:05:44.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.437142+0000", 2023-07-02T14:05:44.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0035492800000000001 2023-07-02T14:05:44.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.440691+0000", 2023-07-02T14:05:44.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1769999999999999e-05 2023-07-02T14:05:44.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.440763+0000", 2023-07-02T14:05:44.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051038780000000001 2023-07-02T14:05:44.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.445867+0000", 2023-07-02T14:05:44.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1784000000000002e-05 2023-07-02T14:05:44.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.445949+0000", 2023-07-02T14:05:44.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091922770000000004 2023-07-02T14:05:44.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.455141+0000", 2023-07-02T14:05:44.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5089e-05 2023-07-02T14:05:44.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.455206+0000", 2023-07-02T14:05:44.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013848835 2023-07-02T14:05:44.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.469055+0000", 2023-07-02T14:05:44.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7831999999999994e-05 2023-07-02T14:05:44.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:44.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.469153+0000", 2023-07-02T14:05:44.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0004929 2023-07-02T14:05:44.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:44.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.469646+0000", 2023-07-02T14:05:44.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011213887000000001 2023-07-02T14:05:44.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:44.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.480860+0000", 2023-07-02T14:05:44.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023170170000000002 2023-07-02T14:05:44.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:44.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.483177+0000", 2023-07-02T14:05:44.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8994999999999999e-05 2023-07-02T14:05:44.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:44.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.483206+0000", 2023-07-02T14:05:44.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8773e-05 2023-07-02T14:05:44.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:44.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.483274+0000", 2023-07-02T14:05:44.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.093494121999999999 2023-07-02T14:05:44.172 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:44.172 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:44.172 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:44.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.173 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17503 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:44.173 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.400832+0000", 2023-07-02T14:05:44.173 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.318806584, 2023-07-02T14:05:44.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.102096161, 2023-07-02T14:05:44.173 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:44.173 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:44.174 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:44.174 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:44.174 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:44.174 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17503 2023-07-02T14:05:44.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.174 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:44.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:44.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400832+0000", 2023-07-02T14:05:44.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:44.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:44.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400832+0000", 2023-07-02T14:05:44.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:44.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:44.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400831+0000", 2023-07-02T14:05:44.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8929999999999998e-06 2023-07-02T14:05:44.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:44.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400835+0000", 2023-07-02T14:05:44.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1750000000000005e-06 2023-07-02T14:05:44.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:44.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400843+0000", 2023-07-02T14:05:44.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0000000000000004e-06 2023-07-02T14:05:44.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400848+0000", 2023-07-02T14:05:44.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011499 2023-07-02T14:05:44.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400963+0000", 2023-07-02T14:05:44.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3274000000000004e-05 2023-07-02T14:05:44.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401027+0000", 2023-07-02T14:05:44.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039469880000000002 2023-07-02T14:05:44.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404974+0000", 2023-07-02T14:05:44.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1142000000000004e-05 2023-07-02T14:05:44.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.405045+0000", 2023-07-02T14:05:44.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069901540000000002 2023-07-02T14:05:44.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.412035+0000", 2023-07-02T14:05:44.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4048999999999994e-05 2023-07-02T14:05:44.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.412119+0000", 2023-07-02T14:05:44.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010323508 2023-07-02T14:05:44.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422442+0000", 2023-07-02T14:05:44.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7278000000000001e-05 2023-07-02T14:05:44.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422520+0000", 2023-07-02T14:05:44.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067846620000000003 2023-07-02T14:05:44.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.429304+0000", 2023-07-02T14:05:44.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2307000000000001e-05 2023-07-02T14:05:44.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.429337+0000", 2023-07-02T14:05:44.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036949779999999998 2023-07-02T14:05:44.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.433032+0000", 2023-07-02T14:05:44.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6023999999999998e-05 2023-07-02T14:05:44.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.433088+0000", 2023-07-02T14:05:44.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041466389999999997 2023-07-02T14:05:44.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.437234+0000", 2023-07-02T14:05:44.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7540000000000003e-05 2023-07-02T14:05:44.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.437272+0000", 2023-07-02T14:05:44.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0035993079999999998 2023-07-02T14:05:44.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.440871+0000", 2023-07-02T14:05:44.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5723000000000007e-05 2023-07-02T14:05:44.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.440947+0000", 2023-07-02T14:05:44.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051923990000000003 2023-07-02T14:05:44.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.446139+0000", 2023-07-02T14:05:44.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7749000000000004e-05 2023-07-02T14:05:44.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.446217+0000", 2023-07-02T14:05:44.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093492230000000003 2023-07-02T14:05:44.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.455566+0000", 2023-07-02T14:05:44.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1462e-05 2023-07-02T14:05:44.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.455648+0000", 2023-07-02T14:05:44.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016420769000000002 2023-07-02T14:05:44.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.472068+0000", 2023-07-02T14:05:44.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7961999999999996e-05 2023-07-02T14:05:44.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.472156+0000", 2023-07-02T14:05:44.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012805568 2023-07-02T14:05:44.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.484962+0000", 2023-07-02T14:05:44.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5397999999999998e-05 2023-07-02T14:05:44.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.485047+0000", 2023-07-02T14:05:44.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080975649999999993 2023-07-02T14:05:44.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.493145+0000", 2023-07-02T14:05:44.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00018218600000000001 2023-07-02T14:05:44.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:44.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.493327+0000", 2023-07-02T14:05:44.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00055431199999999995 2023-07-02T14:05:44.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:44.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.493881+0000", 2023-07-02T14:05:44.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003367676 2023-07-02T14:05:44.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:44.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.497249+0000", 2023-07-02T14:05:44.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054194930000000001 2023-07-02T14:05:44.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:44.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.502669+0000", 2023-07-02T14:05:44.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4320999999999996e-05 2023-07-02T14:05:44.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:44.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.502733+0000", 2023-07-02T14:05:44.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000194966 2023-07-02T14:05:44.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:44.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.502928+0000", 2023-07-02T14:05:44.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.102096161 2023-07-02T14:05:44.198 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:44.198 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:44.198 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:44.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.198 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17562 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:44.199 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.481218+0000", 2023-07-02T14:05:44.199 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.23842037299999999, 2023-07-02T14:05:44.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.094429647000000005, 2023-07-02T14:05:44.199 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:44.199 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:44.199 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:44.199 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:44.199 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:44.200 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17562 2023-07-02T14:05:44.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.200 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:44.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:44.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.481218+0000", 2023-07-02T14:05:44.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:44.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:44.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.481218+0000", 2023-07-02T14:05:44.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:44.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:44.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.481218+0000", 2023-07-02T14:05:44.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1770000000000001e-06 2023-07-02T14:05:44.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:44.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.481220+0000", 2023-07-02T14:05:44.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3380000000000001e-06 2023-07-02T14:05:44.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:44.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.481223+0000", 2023-07-02T14:05:44.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.215e-06 2023-07-02T14:05:44.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.481225+0000", 2023-07-02T14:05:44.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048269530000000001 2023-07-02T14:05:44.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.486052+0000", 2023-07-02T14:05:44.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4368000000000003e-05 2023-07-02T14:05:44.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.486137+0000", 2023-07-02T14:05:44.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011071688 2023-07-02T14:05:44.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.497208+0000", 2023-07-02T14:05:44.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4938000000000003e-05 2023-07-02T14:05:44.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.497243+0000", 2023-07-02T14:05:44.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084346579999999994 2023-07-02T14:05:44.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.505678+0000", 2023-07-02T14:05:44.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3590000000000002e-05 2023-07-02T14:05:44.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.505712+0000", 2023-07-02T14:05:44.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013633684 2023-07-02T14:05:44.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.519345+0000", 2023-07-02T14:05:44.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2536000000000002e-05 2023-07-02T14:05:44.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.519378+0000", 2023-07-02T14:05:44.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059358880000000003 2023-07-02T14:05:44.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.525314+0000", 2023-07-02T14:05:44.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5751e-05 2023-07-02T14:05:44.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.525390+0000", 2023-07-02T14:05:44.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045313630000000001 2023-07-02T14:05:44.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.529921+0000", 2023-07-02T14:05:44.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8229e-05 2023-07-02T14:05:44.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.529989+0000", 2023-07-02T14:05:44.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096487320000000001 2023-07-02T14:05:44.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.539638+0000", 2023-07-02T14:05:44.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7615999999999997e-05 2023-07-02T14:05:44.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.539715+0000", 2023-07-02T14:05:44.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068960460000000003 2023-07-02T14:05:44.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.546612+0000", 2023-07-02T14:05:44.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9788000000000002e-05 2023-07-02T14:05:44.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.546691+0000", 2023-07-02T14:05:44.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073086089999999998 2023-07-02T14:05:44.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.554000+0000", 2023-07-02T14:05:44.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5504000000000001e-05 2023-07-02T14:05:44.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:44.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.554075+0000", 2023-07-02T14:05:44.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062212200000000004 2023-07-02T14:05:44.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:44.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.560297+0000", 2023-07-02T14:05:44.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2172999999999998e-05 2023-07-02T14:05:44.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:44.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.560339+0000", 2023-07-02T14:05:44.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022935499999999999 2023-07-02T14:05:44.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:44.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.560568+0000", 2023-07-02T14:05:44.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058391780000000004 2023-07-02T14:05:44.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:44.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.566407+0000", 2023-07-02T14:05:44.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090585809999999996 2023-07-02T14:05:44.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:44.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.575466+0000", 2023-07-02T14:05:44.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6462000000000002e-05 2023-07-02T14:05:44.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:44.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.575532+0000", 2023-07-02T14:05:44.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011524800000000001 2023-07-02T14:05:44.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:44.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:44.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:44.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.575648+0000", 2023-07-02T14:05:44.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.094429647000000005 2023-07-02T14:05:44.220 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:44.220 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:44.220 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:44.220 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:44.220 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:44.220 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:44.221 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:05:44.222 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:44.221+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:44.223 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:05:44.323 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:44.323+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:44.415 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:44.415 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:44.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.416 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:18141 2.6 2:62d289a6:::rbd_data.125de1449605.00000000000000ef:head [write 131072~2490368 in=2490368b] snapc 2e=[2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:44.416 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:44.384789+0000", 2023-07-02T14:05:44.416 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.028961023999999998, 2023-07-02T14:05:44.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029014532999999999, 2023-07-02T14:05:44.416 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:44.416 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "waiting for sub ops", 2023-07-02T14:05:44.416 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:44.417 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:44.417 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:44.417 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18141 2023-07-02T14:05:44.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.417 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:44.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:44.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384789+0000", 2023-07-02T14:05:44.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:44.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:44.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384789+0000", 2023-07-02T14:05:44.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:44.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:44.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384788+0000", 2023-07-02T14:05:44.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033777249999999998 2023-07-02T14:05:44.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:44.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.388166+0000", 2023-07-02T14:05:44.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4002000000000001e-05 2023-07-02T14:05:44.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:44.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.388180+0000", 2023-07-02T14:05:44.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4810000000000003e-06 2023-07-02T14:05:44.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.388186+0000", 2023-07-02T14:05:44.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023569515999999999 2023-07-02T14:05:44.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:44.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.411756+0000", 2023-07-02T14:05:44.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113736 2023-07-02T14:05:44.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:44.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.411869+0000", 2023-07-02T14:05:44.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000213067 2023-07-02T14:05:44.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3", 2023-07-02T14:05:44.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.412083+0000", 2023-07-02T14:05:44.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027293934999999998 2023-07-02T14:05:44.423 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.423 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:44.423 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.423 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:18142 2.6 2.65914b46 (undecoded) ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:44.424 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:44.388193+0000", 2023-07-02T14:05:44.424 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.025556194000000001, 2023-07-02T14:05:44.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025712217999999998, 2023-07-02T14:05:44.424 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:44.424 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "queued for pg", 2023-07-02T14:05:44.424 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:44.424 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:44.425 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:44.425 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18142 2023-07-02T14:05:44.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.425 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:44.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:44.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.388193+0000", 2023-07-02T14:05:44.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:44.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:44.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.388193+0000", 2023-07-02T14:05:44.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:44.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:44.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.388192+0000", 2023-07-02T14:05:44.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7855e-05 2023-07-02T14:05:44.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:44.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.388220+0000", 2023-07-02T14:05:44.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0202e-05 2023-07-02T14:05:44.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:44.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.388230+0000", 2023-07-02T14:05:44.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0780000000000003e-06 2023-07-02T14:05:44.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.388235+0000", 2023-07-02T14:05:44.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1721000000000003e-05 2023-07-02T14:05:44.429 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.429 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:44.429 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.429 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18156 2.1 e161/104 2:85c3ea39:::rbd_data.125de1449605.00000000000000e8:head v 161'2069, mlcod=161'2063)", 2023-07-02T14:05:44.429 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:44.404081+0000", 2023-07-02T14:05:44.430 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0096686729999999992, 2023-07-02T14:05:44.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0098975859999999999, 2023-07-02T14:05:44.430 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:44.430 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:44.430 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:44.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:44.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.404081+0000", 2023-07-02T14:05:44.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:44.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:44.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.404081+0000", 2023-07-02T14:05:44.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:44.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:44.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.404080+0000", 2023-07-02T14:05:44.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7320000000000002e-05 2023-07-02T14:05:44.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:44.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.404118+0000", 2023-07-02T14:05:44.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.847e-05 2023-07-02T14:05:44.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:44.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.404146+0000", 2023-07-02T14:05:44.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3989999999999998e-06 2023-07-02T14:05:44.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.404153+0000", 2023-07-02T14:05:44.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000744828 2023-07-02T14:05:44.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:44.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.404898+0000", 2023-07-02T14:05:44.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.883e-05 2023-07-02T14:05:44.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:44.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.404927+0000", 2023-07-02T14:05:44.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00084610600000000001 2023-07-02T14:05:44.435 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.435 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:44.435 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.436 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18159 2.0 e161/144 2:08f47745:::rbd_data.125de1449605.00000000000000eb:head v 161'4476, mlcod=161'4472)", 2023-07-02T14:05:44.436 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:44.406258+0000", 2023-07-02T14:05:44.436 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.00749148, 2023-07-02T14:05:44.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007794624, 2023-07-02T14:05:44.436 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:44.436 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:44.437 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:44.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:44.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.406258+0000", 2023-07-02T14:05:44.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:44.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:44.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.406258+0000", 2023-07-02T14:05:44.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:44.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:44.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.406257+0000", 2023-07-02T14:05:44.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0920000000000002e-05 2023-07-02T14:05:44.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:44.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.406288+0000", 2023-07-02T14:05:44.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2699999999999993e-06 2023-07-02T14:05:44.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:44.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.406298+0000", 2023-07-02T14:05:44.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5090000000000001e-06 2023-07-02T14:05:44.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.406303+0000", 2023-07-02T14:05:44.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1342999999999996e-05 2023-07-02T14:05:44.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:44.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.406394+0000", 2023-07-02T14:05:44.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0329000000000001e-05 2023-07-02T14:05:44.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:44.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.406415+0000", 2023-07-02T14:05:44.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015660399999999999 2023-07-02T14:05:44.442 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.442 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:44.442 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.442 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18157 2.1 e161/104 2:85c3ea39:::rbd_data.125de1449605.00000000000000e8:head v 161'2070, mlcod=161'2063)", 2023-07-02T14:05:44.442 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:44.407456+0000", 2023-07-02T14:05:44.443 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0062936119999999996, 2023-07-02T14:05:44.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066714019999999999, 2023-07-02T14:05:44.443 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:44.443 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:44.443 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:44.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:44.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.407456+0000", 2023-07-02T14:05:44.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:44.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:44.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.407456+0000", 2023-07-02T14:05:44.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:44.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:44.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.407455+0000", 2023-07-02T14:05:44.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6325e-05 2023-07-02T14:05:44.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:44.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.407471+0000", 2023-07-02T14:05:44.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0737000000000001e-05 2023-07-02T14:05:44.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:44.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.407482+0000", 2023-07-02T14:05:44.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8329999999999996e-06 2023-07-02T14:05:44.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.407489+0000", 2023-07-02T14:05:44.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010694529999999999 2023-07-02T14:05:44.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:44.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.408559+0000", 2023-07-02T14:05:44.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1999999999999999e-05 2023-07-02T14:05:44.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:44.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.408581+0000", 2023-07-02T14:05:44.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001124484 2023-07-02T14:05:44.448 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.448 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:44.448 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.449 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18158 2.1 e161/104 2:85c3ea39:::rbd_data.125de1449605.00000000000000e8:head v 161'2071, mlcod=161'2063)", 2023-07-02T14:05:44.449 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:44.407509+0000", 2023-07-02T14:05:44.449 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0062405009999999999, 2023-07-02T14:05:44.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066981310000000004, 2023-07-02T14:05:44.449 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:44.449 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:44.450 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:44.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:44.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.407509+0000", 2023-07-02T14:05:44.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:44.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:44.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.407509+0000", 2023-07-02T14:05:44.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:44.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:44.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.407509+0000", 2023-07-02T14:05:44.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4671000000000001e-05 2023-07-02T14:05:44.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:44.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.407533+0000", 2023-07-02T14:05:44.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1540000000000002e-06 2023-07-02T14:05:44.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:44.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.407542+0000", 2023-07-02T14:05:44.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0509999999999996e-06 2023-07-02T14:05:44.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.407547+0000", 2023-07-02T14:05:44.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025251290000000001 2023-07-02T14:05:44.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:44.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.410072+0000", 2023-07-02T14:05:44.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1814000000000002e-05 2023-07-02T14:05:44.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:44.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.410094+0000", 2023-07-02T14:05:44.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0025843860000000001 2023-07-02T14:05:44.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.455 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:44.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.455 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18160 2.0 e161/144 2:08f47745:::rbd_data.125de1449605.00000000000000eb:head v 161'4477, mlcod=161'4472)", 2023-07-02T14:05:44.455 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:44.408097+0000", 2023-07-02T14:05:44.456 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.005652781, 2023-07-02T14:05:44.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061823679999999997, 2023-07-02T14:05:44.456 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:44.456 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:44.456 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:44.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:44.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.408097+0000", 2023-07-02T14:05:44.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:44.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:44.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.408097+0000", 2023-07-02T14:05:44.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:44.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:44.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.408096+0000", 2023-07-02T14:05:44.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028154930000000001 2023-07-02T14:05:44.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:44.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.410911+0000", 2023-07-02T14:05:44.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3108999999999999e-05 2023-07-02T14:05:44.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:44.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.410925+0000", 2023-07-02T14:05:44.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6539999999999999e-06 2023-07-02T14:05:44.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.410932+0000", 2023-07-02T14:05:44.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001067588 2023-07-02T14:05:44.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:44.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.412000+0000", 2023-07-02T14:05:44.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3553e-05 2023-07-02T14:05:44.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:44.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.412023+0000", 2023-07-02T14:05:44.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039265020000000001 2023-07-02T14:05:44.461 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.461 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:44.461 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.462 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4645.0:18140 2.6 e161/139)", 2023-07-02T14:05:44.462 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:44.411854+0000", 2023-07-02T14:05:44.462 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0018952330000000001, 2023-07-02T14:05:44.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002494942, 2023-07-02T14:05:44.462 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:44.462 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "queued for pg", 2023-07-02T14:05:44.463 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:44.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:44.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.411854+0000", 2023-07-02T14:05:44.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:44.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:44.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.411854+0000", 2023-07-02T14:05:44.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:44.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:44.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.411854+0000", 2023-07-02T14:05:44.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1290000000000001e-06 2023-07-02T14:05:44.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:44.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.411860+0000", 2023-07-02T14:05:44.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.276e-06 2023-07-02T14:05:44.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:44.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.411868+0000", 2023-07-02T14:05:44.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7880000000000003e-06 2023-07-02T14:05:44.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.411874+0000", 2023-07-02T14:05:44.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9327e-05 2023-07-02T14:05:44.466 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.467 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:44.467 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.467 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:18162 2.6 2.fb859036 (undecoded) ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:44.467 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:44.412339+0000", 2023-07-02T14:05:44.467 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0014110640000000001, 2023-07-02T14:05:44.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002079642, 2023-07-02T14:05:44.468 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:44.468 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "queued for pg", 2023-07-02T14:05:44.468 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:44.468 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:44.468 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:44.468 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18162 2023-07-02T14:05:44.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.468 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:44.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:44.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.412339+0000", 2023-07-02T14:05:44.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:44.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:44.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.412339+0000", 2023-07-02T14:05:44.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:44.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:44.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.412337+0000", 2023-07-02T14:05:44.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00041782100000000002 2023-07-02T14:05:44.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:44.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.412755+0000", 2023-07-02T14:05:44.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9908e-05 2023-07-02T14:05:44.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:44.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.412775+0000", 2023-07-02T14:05:44.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2e-06 2023-07-02T14:05:44.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.412784+0000", 2023-07-02T14:05:44.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00044526100000000002 2023-07-02T14:05:44.472 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.472 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:44.473 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.473 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18161 2.0 e161/144)", 2023-07-02T14:05:44.473 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:44.413241+0000", 2023-07-02T14:05:44.473 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.00050835500000000005, 2023-07-02T14:05:44.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001242078, 2023-07-02T14:05:44.474 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:44.474 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "queued for pg", 2023-07-02T14:05:44.474 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:44.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:44.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.413241+0000", 2023-07-02T14:05:44.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:44.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:44.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.413241+0000", 2023-07-02T14:05:44.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:44.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:44.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.413241+0000", 2023-07-02T14:05:44.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.879e-05 2023-07-02T14:05:44.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:44.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.413279+0000", 2023-07-02T14:05:44.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1911000000000001e-05 2023-07-02T14:05:44.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:44.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.413301+0000", 2023-07-02T14:05:44.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.781e-06 2023-07-02T14:05:44.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.413308+0000", 2023-07-02T14:05:44.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6721e-05 2023-07-02T14:05:44.478 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.478 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:44.478 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.478 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:18163 2.6 2.fb859036 (undecoded) ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:44.478 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:44.412795+0000", 2023-07-02T14:05:44.478 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.00095474700000000004, 2023-07-02T14:05:44.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017445049999999999, 2023-07-02T14:05:44.479 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:44.479 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "queued for pg", 2023-07-02T14:05:44.479 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:44.479 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:44.479 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:44.479 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18163 2023-07-02T14:05:44.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.480 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:44.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:44.480 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.412795+0000", 2023-07-02T14:05:44.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:44.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:44.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.412795+0000", 2023-07-02T14:05:44.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:44.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:44.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.412794+0000", 2023-07-02T14:05:44.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010008560000000001 2023-07-02T14:05:44.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:44.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.413794+0000", 2023-07-02T14:05:44.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5739e-05 2023-07-02T14:05:44.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:44.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.413810+0000", 2023-07-02T14:05:44.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1493e-05 2023-07-02T14:05:44.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.413822+0000", 2023-07-02T14:05:44.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001026774 2023-07-02T14:05:44.483 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.484 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:44.484 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.484 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:18137 2.6 2:6889ea74:::rbd_data.125de1449605.00000000000000ea:head [write 3801088~20480 in=20480b] snapc 2e=[2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:44.484 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:44.384653+0000", 2023-07-02T14:05:44.484 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.029096431999999998, 2023-07-02T14:05:44.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029966278999999998, 2023-07-02T14:05:44.485 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:44.485 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "waiting for sub ops", 2023-07-02T14:05:44.485 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:44.485 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:44.485 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:44.485 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18137 2023-07-02T14:05:44.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.485 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:44.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:44.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384653+0000", 2023-07-02T14:05:44.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:44.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:44.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384653+0000", 2023-07-02T14:05:44.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:44.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:44.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384653+0000", 2023-07-02T14:05:44.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5637e-05 2023-07-02T14:05:44.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:44.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384668+0000", 2023-07-02T14:05:44.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9009999999999997e-06 2023-07-02T14:05:44.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:44.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384675+0000", 2023-07-02T14:05:44.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5769999999999996e-06 2023-07-02T14:05:44.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384683+0000", 2023-07-02T14:05:44.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01772897 2023-07-02T14:05:44.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:44.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.402412+0000", 2023-07-02T14:05:44.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013336700000000001 2023-07-02T14:05:44.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:44.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.402545+0000", 2023-07-02T14:05:44.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023799100000000001 2023-07-02T14:05:44.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3", 2023-07-02T14:05:44.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.402783+0000", 2023-07-02T14:05:44.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0088917530000000005 2023-07-02T14:05:44.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:44.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.411675+0000", 2023-07-02T14:05:44.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027021738 2023-07-02T14:05:44.492 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.492 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:44.492 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.493 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:18139 2.6 2:62d289a6:::rbd_data.125de1449605.00000000000000ef:head [write 126976~4096 in=4096b] snapc 2e=[2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:44.493 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:44.384688+0000", 2023-07-02T14:05:44.493 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.029061828000000001, 2023-07-02T14:05:44.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030039120999999998, 2023-07-02T14:05:44.493 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:44.493 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "waiting for sub ops", 2023-07-02T14:05:44.493 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:44.494 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:44.494 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:44.494 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18139 2023-07-02T14:05:44.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.494 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:44.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:44.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384688+0000", 2023-07-02T14:05:44.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:44.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:44.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384688+0000", 2023-07-02T14:05:44.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:44.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:44.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384687+0000", 2023-07-02T14:05:44.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6790000000000005e-06 2023-07-02T14:05:44.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:44.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384695+0000", 2023-07-02T14:05:44.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4309999999999999e-06 2023-07-02T14:05:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384701+0000", 2023-07-02T14:05:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9929999999999997e-06 2023-07-02T14:05:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384705+0000", 2023-07-02T14:05:44.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020196425 2023-07-02T14:05:44.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:44.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.404902+0000", 2023-07-02T14:05:44.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000159751 2023-07-02T14:05:44.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:44.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.405062+0000", 2023-07-02T14:05:44.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000231556 2023-07-02T14:05:44.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3", 2023-07-02T14:05:44.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.405293+0000", 2023-07-02T14:05:44.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064289940000000004 2023-07-02T14:05:44.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:44.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.411722+0000", 2023-07-02T14:05:44.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027034325000000001 2023-07-02T14:05:44.500 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.501 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:44.501 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.501 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:18140 2.6 2:62d289a6:::rbd_data.125de1449605.00000000000000ef:head [write 0~126976 in=126976b] snapc 2e=[2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:44.501 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:44.384711+0000", 2023-07-02T14:05:44.501 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.029038820999999999, 2023-07-02T14:05:44.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030124429000000001, 2023-07-02T14:05:44.502 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:44.502 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "waiting for sub ops", 2023-07-02T14:05:44.502 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:44.502 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:44.502 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:44.502 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18140 2023-07-02T14:05:44.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.502 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:44.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:44.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384711+0000", 2023-07-02T14:05:44.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:44.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:44.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384711+0000", 2023-07-02T14:05:44.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:44.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:44.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384710+0000", 2023-07-02T14:05:44.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8934999999999997e-05 2023-07-02T14:05:44.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:44.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384769+0000", 2023-07-02T14:05:44.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1130000000000007e-06 2023-07-02T14:05:44.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:44.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384778+0000", 2023-07-02T14:05:44.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4549999999999997e-06 2023-07-02T14:05:44.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:44.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.384783+0000", 2023-07-02T14:05:44.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024403831000000001 2023-07-02T14:05:44.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:44.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.409187+0000", 2023-07-02T14:05:44.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113693 2023-07-02T14:05:44.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:44.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.409300+0000", 2023-07-02T14:05:44.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000224625 2023-07-02T14:05:44.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:44.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:44.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3", 2023-07-02T14:05:44.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:44.409525+0000", 2023-07-02T14:05:44.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024814238999999998 2023-07-02T14:05:44.508 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.508 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:44.508 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.508 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:44.509 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:05:44.509 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 14 2023-07-02T14:05:44.509 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:44.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:05:44.510 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:44.510+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:44.511 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:05:44.611 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:44.611+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:44.679 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:05:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:05:44.680 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:44.690 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:05:44.693 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:44.711 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:44.711+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:44.713 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:44.732 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:44.750 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:44.769 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:44.770 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-07-02T14:05:44.770 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:05:44.770 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:05:44.770 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:05:44.770 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:05:44.770 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:05:44.772 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:05:44.773 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:05:44.814 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:44.814+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:44.816 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:44.816 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:44.826 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:05:44.913 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:44.913+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:44.994 INFO:tasks.workunit.client.0.smithi163.stderr:+ sync b3fbb028-e267-4d25-a750-93ba18a2316a-mnt 2023-07-02T14:05:45.007 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:45.007 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:45.014 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:45.014+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:45.019 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:05:45.080 INFO:tasks.workunit.client.0.smithi163.stderr:+ rm -rf b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4 b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4.tar.gz 2023-07-02T14:05:45.114 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:45.114+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:45.171 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap42 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap43 b3fbb028-e267-4d25-a750-93ba18a2316a@snap43.diff 2023-07-02T14:05:45.185 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:45.185 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:45.195 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:05:45.236 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:45.236+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:45.238 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:05:45.238+0000 7f96e29e2700 0 -- 172.21.15.163:0/2793512156 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f96c0004b10 msgr2=0x7f96c0004f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:45.244 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:05:45.244+0000 7f96e31e3700 0 -- 172.21.15.163:0/2793512156 >> [v2:172.21.15.22:6808/99977,v1:172.21.15.22:6809/99977] conn(0x5584979f9f30 msgr2=0x5584979fc320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:45.316 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:45.316+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:45.386 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:45.386 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:45.390 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:05:45.399 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:05:45.442 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:45.443+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:45.466 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 100% complete...done. 2023-07-02T14:05:45.471 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:05:45.471 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=44 2023-07-02T14:05:45.471 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap44 2023-07-02T14:05:45.517 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:45.518+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:45.527 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:45.527 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:45.537 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:05:45.620 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:45.621+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:45.701 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:45.701 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:45.711 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:05:45.753 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:45.753+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:45.820 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:45.821+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:45.875 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:45.875 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:45.885 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:05:45.917 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 2] out_osds: [5, 4] dead_osds: [0] live_osds: [1, 2, 4, 5, 3] 2023-07-02T14:05:45.917 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2023-07-02T14:05:45.917 INFO:tasks.ceph.osd.0:Restarting daemon 2023-07-02T14:05:45.918 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-07-02T14:05:45.920 INFO:tasks.ceph.osd.0:Started 2023-07-02T14:05:45.920 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:05:45.921 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:45.921+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:46.021 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:46.022+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:46.045 INFO:teuthology.orchestra.run.smithi016.stderr:no valid command found; 10 closest matches: 2023-07-02T14:05:46.045 INFO:teuthology.orchestra.run.smithi016.stderr:0 2023-07-02T14:05:46.045 INFO:teuthology.orchestra.run.smithi016.stderr:1 2023-07-02T14:05:46.045 INFO:teuthology.orchestra.run.smithi016.stderr:2 2023-07-02T14:05:46.045 INFO:teuthology.orchestra.run.smithi016.stderr:abort 2023-07-02T14:05:46.045 INFO:teuthology.orchestra.run.smithi016.stderr:assert 2023-07-02T14:05:46.045 INFO:teuthology.orchestra.run.smithi016.stderr:config diff 2023-07-02T14:05:46.046 INFO:teuthology.orchestra.run.smithi016.stderr:config diff get 2023-07-02T14:05:46.046 INFO:teuthology.orchestra.run.smithi016.stderr:config get 2023-07-02T14:05:46.046 INFO:teuthology.orchestra.run.smithi016.stderr:config help [] 2023-07-02T14:05:46.046 INFO:teuthology.orchestra.run.smithi016.stderr:config set ... 2023-07-02T14:05:46.046 INFO:teuthology.orchestra.run.smithi016.stderr:admin_socket: invalid command 2023-07-02T14:05:46.047 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-07-02T14:05:46.047 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2023-07-02T14:05:46.050 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:46.050 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:46.061 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:05:46.122 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:46.122+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:46.223 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:46.223+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:46.225 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:46.225 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:46.236 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:05:46.323 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:46.323+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:46.403 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:46.403 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:46.416 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:05:46.431 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:05:46.436 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 3 2023-07-02T14:05:46.457 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:46.457+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:46.513 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:46.513+0000 7fd57ef6e200 -1 Falling back to public interface 2023-07-02T14:05:46.524 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:46.524+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:46.587 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:46.587 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:46.597 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:05:46.639 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:46.640+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:46.642 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i++ )) 2023-07-02T14:05:46.642 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i < 10 )) 2023-07-02T14:05:46.642 INFO:tasks.workunit.client.0.smithi163.stderr:+ pv -L 10M linux-5.4.tar.gz 2023-07-02T14:05:46.721 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:46.721 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:46.725 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:46.725+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:46.731 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:05:46.773 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:05:46.826 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:46.827+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:46.894 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:46.895 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:46.905 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:05:46.913 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:46.933 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:46.945 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:46.946+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:46.951 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:46.969 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:46.987 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:46.988 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-07-02T14:05:46.988 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:05:46.988 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:05:46.988 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:05:46.988 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:05:46.989 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:05:46.990 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:05:46.991 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:05:47.028 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:47.028+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:47.030 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:47.031 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:47.031 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:05:47.031 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:47.041 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:05:47.129 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:47.129+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:47.205 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:47.205 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:47.206 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:05:47.206 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:05:47.206 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:47.215 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:05:47.256 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:47.256+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:47.331 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:47.331+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:47.381 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:47.381 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:05:47.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:05:47.382 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:47.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.382 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:05:47.382 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:47.382 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 157.927665601, 2023-07-02T14:05:47.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:05:47.383 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:47.383 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:47.383 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:47.383 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:47.383 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:47.383 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:05:47.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.383 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:47.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:47.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:47.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:47.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:47.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:47.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:47.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:47.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:05:47.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:05:47.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:47.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:05:47.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:05:47.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:47.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:05:47.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:05:47.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:47.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:05:47.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:05:47.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:05:47.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:05:47.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:47.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:05:47.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:05:47.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:05:47.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:05:47.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:47.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:05:47.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:05:47.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.390 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.391 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:05:47.391 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:47.391 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 157.05697259300001, 2023-07-02T14:05:47.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:05:47.391 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:47.391 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:47.391 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:47.392 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:05:47.392 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:05:47.392 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:05:47.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.392 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:47.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:47.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:47.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:47.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:47.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:47.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:47.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:47.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:05:47.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:05:47.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:47.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:05:47.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:05:47.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:47.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:05:47.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:05:47.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:47.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:05:47.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:05:47.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:05:47.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:05:47.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:47.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:05:47.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:05:47.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:05:47.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:05:47.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:47.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:05:47.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:05:47.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:47.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:05:47.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:05:47.399 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.399 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.400 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.400 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:47.400 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:47.400 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 157.05684787199999, 2023-07-02T14:05:47.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:05:47.400 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:47.401 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:47.401 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:47.401 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:47.401 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:47.401 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:05:47.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.401 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:47.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:47.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:47.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:47.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:47.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:47.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:47.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:47.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:05:47.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:05:47.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:47.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:05:47.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:05:47.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:47.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:05:47.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:05:47.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:47.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:05:47.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:05:47.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:05:47.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:05:47.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:47.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:05:47.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:05:47.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:05:47.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:05:47.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:47.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:05:47.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:05:47.408 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.408 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.408 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.408 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:47.409 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:47.409 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 157.049997988, 2023-07-02T14:05:47.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:05:47.409 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:47.409 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:47.409 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:47.409 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:47.409 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:47.410 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:05:47.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.410 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:47.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:47.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:47.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:47.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:47.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:47.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:05:47.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:47.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:05:47.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:05:47.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:47.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:05:47.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:05:47.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:47.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:05:47.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:05:47.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:47.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:05:47.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:05:47.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:05:47.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:05:47.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:47.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:05:47.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:05:47.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:05:47.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:05:47.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:47.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:05:47.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:05:47.416 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.417 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.417 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:47.417 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:47.417 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 157.04993506299999, 2023-07-02T14:05:47.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:05:47.418 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:47.418 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:47.418 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:47.418 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:47.418 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:47.418 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:05:47.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.418 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:47.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:47.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:47.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:47.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:47.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:47.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:47.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:47.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:05:47.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:05:47.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:47.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:05:47.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:05:47.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:47.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:05:47.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:05:47.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:47.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:05:47.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:05:47.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:05:47.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:05:47.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:47.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:05:47.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:05:47.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:05:47.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:05:47.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:47.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:05:47.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:05:47.425 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.425 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.425 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.426 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:47.426 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:47.426 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 157.049634642, 2023-07-02T14:05:47.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:05:47.426 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:47.426 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:47.426 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:47.427 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:47.427 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:47.427 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:05:47.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.427 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:47.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:47.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:47.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:47.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:47.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:47.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:47.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:47.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:05:47.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:05:47.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:47.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:05:47.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:05:47.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:47.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:05:47.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:05:47.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:47.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:05:47.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:05:47.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:05:47.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:05:47.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:47.432 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:05:47.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:05:47.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:05:47.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:05:47.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:47.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:05:47.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:05:47.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:05:47.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:05:47.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:05:47.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:47.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:05:47.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:05:47.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:47.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:05:47.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:05:47.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:47.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:05:47.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:05:47.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:47.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:05:47.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:05:47.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:47.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:05:47.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:05:47.437 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.437 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.438 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.438 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:05:47.438 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:47.438 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 157.04911014199999, 2023-07-02T14:05:47.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:05:47.439 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:47.439 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:47.439 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:47.439 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:47.439 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:47.439 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:05:47.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.439 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:47.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:47.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:47.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:47.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:47.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:47.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:47.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:47.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:47.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:05:47.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:47.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:05:47.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:05:47.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:47.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:05:47.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:05:47.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:47.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:05:47.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:05:47.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:05:47.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:05:47.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:47.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:05:47.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:05:47.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:05:47.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:05:47.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:47.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:05:47.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:05:47.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:47.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:05:47.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:05:47.446 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.447 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.447 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.447 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:05:47.447 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:47.447 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 129.925091017, 2023-07-02T14:05:47.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:05:47.448 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:47.448 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:47.448 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:47.448 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:47.448 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:47.448 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:05:47.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.448 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:47.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:47.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:47.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:47.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:47.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:47.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:47.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:47.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:05:47.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:05:47.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:47.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:05:47.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:05:47.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:47.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:05:47.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:05:47.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:47.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:05:47.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:05:47.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:05:47.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:05:47.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:47.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:05:47.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:05:47.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:05:47.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:05:47.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:47.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:05:47.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:05:47.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.455 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.455 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:47.456 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:47.456 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 129.599608313, 2023-07-02T14:05:47.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:05:47.456 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:47.456 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:47.456 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:47.456 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:47.457 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:47.457 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:05:47.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.457 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:47.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:47.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:47.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:47.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:47.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:47.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:47.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:47.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:05:47.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:05:47.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:47.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:05:47.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:05:47.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:47.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:05:47.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:05:47.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:47.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:05:47.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:05:47.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:47.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:05:47.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:47.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:05:47.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:05:47.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:05:47.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:05:47.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:47.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:05:47.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:05:47.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:47.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:05:47.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:05:47.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:47.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:05:47.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:05:47.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:47.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:05:47.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:05:47.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:47.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:05:47.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:05:47.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:47.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:05:47.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:05:47.467 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.467 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.467 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.467 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3939 2.1 2:9ef34a83:::rbd_data.125de1449605.00000000000000b8:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:47.467 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:47.467 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 129.59568761599999, 2023-07-02T14:05:47.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001, 2023-07-02T14:05:47.468 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:47.468 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:47.468 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:47.468 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:47.468 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:47.468 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3939 2023-07-02T14:05:47.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.469 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:47.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:47.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:47.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:47.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:47.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:47.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:47.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:47.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:47.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046852830000000002 2023-07-02T14:05:47.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:47.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788763+0000", 2023-07-02T14:05:47.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3185e-05 2023-07-02T14:05:47.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:47.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788776+0000", 2023-07-02T14:05:47.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8470000000000003e-06 2023-07-02T14:05:47.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:47.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788781+0000", 2023-07-02T14:05:47.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7104000000000002e-05 2023-07-02T14:05:47.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788828+0000", 2023-07-02T14:05:47.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3516000000000001e-05 2023-07-02T14:05:47.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:47.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788841+0000", 2023-07-02T14:05:47.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33064513600000001 2023-07-02T14:05:47.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119487+0000", 2023-07-02T14:05:47.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9876999999999997e-05 2023-07-02T14:05:47.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:47.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119537+0000", 2023-07-02T14:05:47.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014160999999999999 2023-07-02T14:05:47.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:47.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119678+0000", 2023-07-02T14:05:47.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023054392 2023-07-02T14:05:47.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:47.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.142733+0000", 2023-07-02T14:05:47.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074065470000000003 2023-07-02T14:05:47.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:47.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150139+0000", 2023-07-02T14:05:47.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5025999999999997e-05 2023-07-02T14:05:47.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:47.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150204+0000", 2023-07-02T14:05:47.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147999999999997e-05 2023-07-02T14:05:47.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:47.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150241+0000", 2023-07-02T14:05:47.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001 2023-07-02T14:05:47.478 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.478 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.478 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.479 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:17 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 0~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:47.479 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:05:47.479 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.181332531000002, 2023-07-02T14:05:47.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997, 2023-07-02T14:05:47.479 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:47.479 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:47.480 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:47.480 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:47.480 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:47.480 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17 2023-07-02T14:05:47.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.480 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:47.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.480 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:47.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:05:47.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:47.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:47.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:05:47.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:05:47.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:47.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198431+0000", 2023-07-02T14:05:47.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2815e-05 2023-07-02T14:05:47.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:47.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198443+0000", 2023-07-02T14:05:47.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2569999999999999e-05 2023-07-02T14:05:47.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:47.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198466+0000", 2023-07-02T14:05:47.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0223e-05 2023-07-02T14:05:47.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:47.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198476+0000", 2023-07-02T14:05:47.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1792999999999997e-05 2023-07-02T14:05:47.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:05:47.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4249999999999998e-06 2023-07-02T14:05:47.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:47.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198536+0000", 2023-07-02T14:05:47.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91888663500000001 2023-07-02T14:05:47.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117423+0000", 2023-07-02T14:05:47.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223594 2023-07-02T14:05:47.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:47.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117647+0000", 2023-07-02T14:05:47.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033832900000000001 2023-07-02T14:05:47.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:47.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117985+0000", 2023-07-02T14:05:47.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997 2023-07-02T14:05:47.487 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.487 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.488 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.488 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:18 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 65536~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:47.488 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:47.488 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.181275319999997, 2023-07-02T14:05:47.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998, 2023-07-02T14:05:47.489 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:47.489 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:47.489 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:47.489 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:47.489 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:47.489 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18 2023-07-02T14:05:47.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.489 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:47.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:47.490 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:47.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:47.490 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.490 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:47.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:47.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:47.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:47.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:47.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7600000000000002e-06 2023-07-02T14:05:47.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:47.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198494+0000", 2023-07-02T14:05:47.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7840000000000004e-06 2023-07-02T14:05:47.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:47.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198502+0000", 2023-07-02T14:05:47.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6459999999999998e-06 2023-07-02T14:05:47.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:47.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198508+0000", 2023-07-02T14:05:47.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7395999999999997e-05 2023-07-02T14:05:47.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198565+0000", 2023-07-02T14:05:47.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6800000000000004e-06 2023-07-02T14:05:47.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:47.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198572+0000", 2023-07-02T14:05:47.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91965896800000002 2023-07-02T14:05:47.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118231+0000", 2023-07-02T14:05:47.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5212999999999998e-05 2023-07-02T14:05:47.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:47.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118326+0000", 2023-07-02T14:05:47.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000238538 2023-07-02T14:05:47.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:47.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118565+0000", 2023-07-02T14:05:47.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998 2023-07-02T14:05:47.497 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.497 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.497 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.497 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:19 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 131072~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:47.498 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:47.498 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.181251095999997, 2023-07-02T14:05:47.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002, 2023-07-02T14:05:47.498 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:47.498 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:47.498 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:47.498 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:47.498 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:47.499 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 19 2023-07-02T14:05:47.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.499 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:47.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.499 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:47.499 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:47.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:47.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:47.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:47.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:47.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:47.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:47.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1990000000000002e-06 2023-07-02T14:05:47.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:47.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198517+0000", 2023-07-02T14:05:47.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8789999999999997e-06 2023-07-02T14:05:47.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:47.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198524+0000", 2023-07-02T14:05:47.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:05:47.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:47.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:05:47.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5545999999999999e-05 2023-07-02T14:05:47.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:47.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2540000000000003e-06 2023-07-02T14:05:47.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:47.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198598+0000", 2023-07-02T14:05:47.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.919983263 2023-07-02T14:05:47.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118581+0000", 2023-07-02T14:05:47.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6878999999999994e-05 2023-07-02T14:05:47.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:47.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118648+0000", 2023-07-02T14:05:47.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149732 2023-07-02T14:05:47.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:47.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118798+0000", 2023-07-02T14:05:47.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002 2023-07-02T14:05:47.506 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.506 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.506 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.507 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:20 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 262144~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:47.507 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:47.507 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.181232131999998, 2023-07-02T14:05:47.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998, 2023-07-02T14:05:47.507 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:47.507 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:47.507 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:47.508 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:47.508 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:47.508 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20 2023-07-02T14:05:47.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.508 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:47.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:47.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:47.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:47.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:47.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:47.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:47.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:47.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:47.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.732e-06 2023-07-02T14:05:47.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:47.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198537+0000", 2023-07-02T14:05:47.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5259999999999999e-06 2023-07-02T14:05:47.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:47.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198543+0000", 2023-07-02T14:05:47.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.121e-06 2023-07-02T14:05:47.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:47.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198547+0000", 2023-07-02T14:05:47.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9591000000000005e-05 2023-07-02T14:05:47.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:05:47.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3120000000000004e-06 2023-07-02T14:05:47.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:47.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198621+0000", 2023-07-02T14:05:47.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92020789400000003 2023-07-02T14:05:47.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118829+0000", 2023-07-02T14:05:47.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022393386000000001 2023-07-02T14:05:47.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:47.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141223+0000", 2023-07-02T14:05:47.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000329866 2023-07-02T14:05:47.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:47.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141552+0000", 2023-07-02T14:05:47.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998 2023-07-02T14:05:47.515 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.515 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.515 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.516 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:21 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 196608~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:47.516 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:47.516 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.181204497000003, 2023-07-02T14:05:47.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995, 2023-07-02T14:05:47.516 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:47.516 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:47.517 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:47.517 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:47.517 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:47.517 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21 2023-07-02T14:05:47.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.517 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:47.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:47.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:47.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:47.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:47.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:47.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:47.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:47.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:47.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0169999999999999e-05 2023-07-02T14:05:47.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:47.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198571+0000", 2023-07-02T14:05:47.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3479999999999998e-06 2023-07-02T14:05:47.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:47.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198578+0000", 2023-07-02T14:05:47.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8439999999999997e-06 2023-07-02T14:05:47.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:47.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198583+0000", 2023-07-02T14:05:47.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8294999999999997e-05 2023-07-02T14:05:47.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198641+0000", 2023-07-02T14:05:47.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8120000000000001e-06 2023-07-02T14:05:47.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:47.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:05:47.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94293828499999999 2023-07-02T14:05:47.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141583+0000", 2023-07-02T14:05:47.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014607300000000001 2023-07-02T14:05:47.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:47.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141729+0000", 2023-07-02T14:05:47.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00044906600000000003 2023-07-02T14:05:47.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:47.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142178+0000", 2023-07-02T14:05:47.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995 2023-07-02T14:05:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:22 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 327680~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:47.525 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.181171067000001, 2023-07-02T14:05:47.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004, 2023-07-02T14:05:47.526 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:47.526 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:47.526 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:47.526 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:47.526 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:47.526 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 22 2023-07-02T14:05:47.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.527 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:47.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:47.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:47.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:47.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:47.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:47.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:47.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:47.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:47.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4850000000000001e-06 2023-07-02T14:05:47.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:47.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198597+0000", 2023-07-02T14:05:47.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:05:47.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:47.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198605+0000", 2023-07-02T14:05:47.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4839999999999998e-06 2023-07-02T14:05:47.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:47.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198609+0000", 2023-07-02T14:05:47.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3100999999999999e-05 2023-07-02T14:05:47.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198662+0000", 2023-07-02T14:05:47.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9099999999999998e-06 2023-07-02T14:05:47.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:47.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198666+0000", 2023-07-02T14:05:47.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.943539503 2023-07-02T14:05:47.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142206+0000", 2023-07-02T14:05:47.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012255999999999999 2023-07-02T14:05:47.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:47.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142328+0000", 2023-07-02T14:05:47.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208956 2023-07-02T14:05:47.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:47.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142537+0000", 2023-07-02T14:05:47.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004 2023-07-02T14:05:47.534 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.534 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.534 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.534 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:23 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 393216~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:47.535 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:47.535 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.181151087000003, 2023-07-02T14:05:47.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348, 2023-07-02T14:05:47.535 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:47.535 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:47.535 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:47.535 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:47.535 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:47.536 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23 2023-07-02T14:05:47.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.536 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:47.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:47.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:47.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:47.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:47.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:47.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:47.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:47.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:47.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8439999999999998e-06 2023-07-02T14:05:47.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:47.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:05:47.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0440000000000001e-06 2023-07-02T14:05:47.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:47.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198623+0000", 2023-07-02T14:05:47.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4480000000000002e-06 2023-07-02T14:05:47.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:47.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198628+0000", 2023-07-02T14:05:47.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5843e-05 2023-07-02T14:05:47.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:47.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.89e-06 2023-07-02T14:05:47.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:47.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198688+0000", 2023-07-02T14:05:47.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94389171999999999 2023-07-02T14:05:47.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142580+0000", 2023-07-02T14:05:47.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3892000000000004e-05 2023-07-02T14:05:47.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:47.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142644+0000", 2023-07-02T14:05:47.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017608000000000001 2023-07-02T14:05:47.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:47.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142820+0000", 2023-07-02T14:05:47.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348 2023-07-02T14:05:47.543 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.543 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.543 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.544 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:24 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 458752~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:47.544 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:05:47.544 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.181123012999997, 2023-07-02T14:05:47.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004, 2023-07-02T14:05:47.544 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:47.544 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:47.545 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:47.545 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:47.545 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:47.545 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 24 2023-07-02T14:05:47.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.545 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:47.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:47.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:05:47.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:47.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:47.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:05:47.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:47.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:47.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198642+0000", 2023-07-02T14:05:47.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.253e-06 2023-07-02T14:05:47.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:47.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:05:47.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1289999999999998e-06 2023-07-02T14:05:47.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:47.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198652+0000", 2023-07-02T14:05:47.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4510000000000002e-06 2023-07-02T14:05:47.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:47.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198657+0000", 2023-07-02T14:05:47.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9131000000000001e-05 2023-07-02T14:05:47.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198706+0000", 2023-07-02T14:05:47.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8890000000000004e-06 2023-07-02T14:05:47.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:47.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198710+0000", 2023-07-02T14:05:47.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94415412200000004 2023-07-02T14:05:47.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142864+0000", 2023-07-02T14:05:47.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8047000000000006e-05 2023-07-02T14:05:47.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:47.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142952+0000", 2023-07-02T14:05:47.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192967 2023-07-02T14:05:47.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:47.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143145+0000", 2023-07-02T14:05:47.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004 2023-07-02T14:05:47.552 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.552 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.552 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.553 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:25 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 524288~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:47.553 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:47.553 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.181104218000002, 2023-07-02T14:05:47.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001, 2023-07-02T14:05:47.553 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:47.553 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:47.554 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:47.554 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:47.554 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:47.554 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25 2023-07-02T14:05:47.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.554 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:47.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:47.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:47.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:47.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:47.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:47.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:47.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:47.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:47.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.615e-06 2023-07-02T14:05:47.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:47.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198664+0000", 2023-07-02T14:05:47.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4759999999999997e-06 2023-07-02T14:05:47.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:47.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198669+0000", 2023-07-02T14:05:47.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5999999999999998e-06 2023-07-02T14:05:47.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198673+0000", 2023-07-02T14:05:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4256999999999999e-05 2023-07-02T14:05:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198727+0000", 2023-07-02T14:05:47.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840000000000001e-06 2023-07-02T14:05:47.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:47.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198731+0000", 2023-07-02T14:05:47.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94444241900000003 2023-07-02T14:05:47.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143173+0000", 2023-07-02T14:05:47.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6839000000000002e-05 2023-07-02T14:05:47.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:47.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143240+0000", 2023-07-02T14:05:47.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017053399999999999 2023-07-02T14:05:47.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:47.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143411+0000", 2023-07-02T14:05:47.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001 2023-07-02T14:05:47.561 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.561 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.562 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:26 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 589824~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:47.562 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:47.562 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.181081067000001, 2023-07-02T14:05:47.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997, 2023-07-02T14:05:47.562 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:47.563 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:47.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:47.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:47.563 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:47.563 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 26 2023-07-02T14:05:47.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.563 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:47.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:47.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:47.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:47.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:47.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:47.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:47.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:47.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:47.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0520000000000002e-06 2023-07-02T14:05:47.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:47.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198687+0000", 2023-07-02T14:05:47.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1880000000000002e-06 2023-07-02T14:05:47.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:47.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198695+0000", 2023-07-02T14:05:47.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9960000000000004e-06 2023-07-02T14:05:47.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:47.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198699+0000", 2023-07-02T14:05:47.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2330999999999998e-05 2023-07-02T14:05:47.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198742+0000", 2023-07-02T14:05:47.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7950000000000001e-06 2023-07-02T14:05:47.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:47.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198745+0000", 2023-07-02T14:05:47.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94469398199999999 2023-07-02T14:05:47.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:47.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143439+0000", 2023-07-02T14:05:47.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6048e-05 2023-07-02T14:05:47.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:47.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143525+0000", 2023-07-02T14:05:47.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017016700000000001 2023-07-02T14:05:47.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:47.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:47.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:47.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143696+0000", 2023-07-02T14:05:47.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997 2023-07-02T14:05:47.570 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.571 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.571 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.571 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:47.571 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:47.571 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:47.571 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:05:47.573 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:05:47.574 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:47.572+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:47.673 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:47.674+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:47.740 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:47.740 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:47.740 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:05:47.741 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:47.751 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:05:47.795 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:47.795+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:47.875 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:47.875 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:47.876 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:05:47.876 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:05:47.876 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:47.876 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:47.876+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:47.885 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:05:47.977 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:47.977+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:48.050 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:48.050 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:05:48.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:05:48.050 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:48.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.051 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:05:48.051 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:48.051 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 166.940718914, 2023-07-02T14:05:48.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:05:48.051 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:48.051 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:48.051 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:48.052 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:48.052 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:48.052 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:05:48.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.052 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:48.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:48.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:48.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:48.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:48.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:48.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:48.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:48.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:05:48.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:05:48.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:48.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:05:48.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:05:48.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:48.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:05:48.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:05:48.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:05:48.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:05:48.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:05:48.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:05:48.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:48.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:05:48.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:05:48.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:05:48.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:05:48.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:48.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:05:48.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:05:48.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:48.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:05:48.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:05:48.059 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.059 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.059 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.060 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:05:48.060 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:48.060 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 166.940658176, 2023-07-02T14:05:48.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:05:48.060 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:48.061 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.061 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:48.061 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:48.061 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:48.061 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:05:48.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.061 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:48.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:48.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:48.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:48.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:48.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:48.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:48.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:48.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:05:48.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:05:48.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:48.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:05:48.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:05:48.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:48.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:05:48.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:05:48.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:05:48.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:05:48.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:05:48.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:05:48.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:48.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:05:48.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:05:48.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:05:48.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:05:48.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:48.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:05:48.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:05:48.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:48.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:05:48.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:05:48.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:48.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:05:48.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:05:48.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:48.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:05:48.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:05:48.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:48.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:05:48.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:05:48.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:48.071 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:05:48.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:05:48.071 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.071 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.071 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.071 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.071 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:48.072 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:48.072 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 156.71020384600001, 2023-07-02T14:05:48.072 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:05:48.072 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:48.072 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.072 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:48.072 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:48.072 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:48.073 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:05:48.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.073 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:48.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:48.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:48.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:48.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:48.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:48.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:48.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:48.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:05:48.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:05:48.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:48.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:05:48.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:05:48.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:48.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:05:48.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:05:48.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:05:48.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:05:48.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:05:48.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:05:48.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:48.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:05:48.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:05:48.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.078 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:48.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:05:48.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:05:48.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.079 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:48.079 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:05:48.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:05:48.079 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.079 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:48.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:05:48.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:05:48.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:48.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:05:48.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:05:48.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:48.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:05:48.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:05:48.081 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.081 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.081 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.082 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2950 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 1507328~786432] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:48.082 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:48.082 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 156.57292758400001, 2023-07-02T14:05:48.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999, 2023-07-02T14:05:48.082 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:48.082 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.083 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:48.083 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:48.083 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:48.083 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2950 2023-07-02T14:05:48.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.083 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:48.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:48.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:48.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:48.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:48.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:48.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:48.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:48.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:48.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001147854 2023-07-02T14:05:48.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:48.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476614+0000", 2023-07-02T14:05:48.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.802e-05 2023-07-02T14:05:48.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.086 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:48.086 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476632+0000", 2023-07-02T14:05:48.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1860000000000003e-06 2023-07-02T14:05:48.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476639+0000", 2023-07-02T14:05:48.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061211240000000004 2023-07-02T14:05:48.087 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.087 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.087 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.087 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482760+0000", 2023-07-02T14:05:48.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8833999999999996e-05 2023-07-02T14:05:48.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:48.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482859+0000", 2023-07-02T14:05:48.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019270600000000001 2023-07-02T14:05:48.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:48.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.483051+0000", 2023-07-02T14:05:48.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060385630000000003 2023-07-02T14:05:48.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:48.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489090+0000", 2023-07-02T14:05:48.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024506918999999999 2023-07-02T14:05:48.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:48.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513597+0000", 2023-07-02T14:05:48.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9672999999999995e-05 2023-07-02T14:05:48.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:48.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513667+0000", 2023-07-02T14:05:48.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024329500000000001 2023-07-02T14:05:48.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:48.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513910+0000", 2023-07-02T14:05:48.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999 2023-07-02T14:05:48.092 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.092 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.092 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.092 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2952 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2883584~12288] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:48.092 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:48.092 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 156.57163297100001, 2023-07-02T14:05:48.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651, 2023-07-02T14:05:48.093 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:48.093 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.093 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:48.093 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:48.093 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:48.093 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2952 2023-07-02T14:05:48.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.094 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:48.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:48.094 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:48.094 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:48.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.094 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:48.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:48.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:48.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.095 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.095 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:48.095 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476760+0000", 2023-07-02T14:05:48.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3664e-05 2023-07-02T14:05:48.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:48.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476774+0000", 2023-07-02T14:05:48.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.166e-06 2023-07-02T14:05:48.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:48.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476783+0000", 2023-07-02T14:05:48.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8720000000000001e-06 2023-07-02T14:05:48.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476788+0000", 2023-07-02T14:05:48.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087180529999999999 2023-07-02T14:05:48.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485506+0000", 2023-07-02T14:05:48.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5680999999999994e-05 2023-07-02T14:05:48.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:48.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485602+0000", 2023-07-02T14:05:48.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000222365 2023-07-02T14:05:48.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:48.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485824+0000", 2023-07-02T14:05:48.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003277379 2023-07-02T14:05:48.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:48.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489101+0000", 2023-07-02T14:05:48.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025151739999999999 2023-07-02T14:05:48.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:48.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514253+0000", 2023-07-02T14:05:48.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4749000000000002e-05 2023-07-02T14:05:48.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:48.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514288+0000", 2023-07-02T14:05:48.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130634 2023-07-02T14:05:48.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:48.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514419+0000", 2023-07-02T14:05:48.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651 2023-07-02T14:05:48.102 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.102 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.102 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.103 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9620 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 589824~65536] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:05:48.103 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:48.103 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.026189247999994, 2023-07-02T14:05:48.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001, 2023-07-02T14:05:48.103 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:48.103 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.103 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:48.103 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:48.104 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:48.104 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9620 2023-07-02T14:05:48.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.104 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:48.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:48.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:48.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:48.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:48.105 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:48.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:48.105 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.105 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.105 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:48.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022203+0000", 2023-07-02T14:05:48.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0346000000000001e-05 2023-07-02T14:05:48.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:48.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022263+0000", 2023-07-02T14:05:48.106 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4657999999999999e-05 2023-07-02T14:05:48.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:48.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022278+0000", 2023-07-02T14:05:48.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:05:48.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022285+0000", 2023-07-02T14:05:48.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0096000000000003e-05 2023-07-02T14:05:48.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022335+0000", 2023-07-02T14:05:48.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2334000000000001e-05 2023-07-02T14:05:48.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:48.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022347+0000", 2023-07-02T14:05:48.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095653701999999993 2023-07-02T14:05:48.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118001+0000", 2023-07-02T14:05:48.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000185797 2023-07-02T14:05:48.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:48.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118187+0000", 2023-07-02T14:05:48.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026397900000000001 2023-07-02T14:05:48.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:48.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118451+0000", 2023-07-02T14:05:48.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051832800000000002 2023-07-02T14:05:48.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:48.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.123634+0000", 2023-07-02T14:05:48.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001227332 2023-07-02T14:05:48.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:48.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124861+0000", 2023-07-02T14:05:48.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9493999999999998e-05 2023-07-02T14:05:48.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:48.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124901+0000", 2023-07-02T14:05:48.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0462000000000001e-05 2023-07-02T14:05:48.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:48.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124931+0000", 2023-07-02T14:05:48.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001 2023-07-02T14:05:48.114 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.114 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.114 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.114 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9621 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 655360~1966080] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:05:48.114 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:48.115 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.026001866000001, 2023-07-02T14:05:48.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999, 2023-07-02T14:05:48.115 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:48.115 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.115 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:48.115 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:48.115 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:48.115 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9621 2023-07-02T14:05:48.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.116 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:48.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:48.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:48.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:48.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:48.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:48.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:48.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:48.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022391+0000", 2023-07-02T14:05:48.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032053939999999999 2023-07-02T14:05:48.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:48.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025597+0000", 2023-07-02T14:05:48.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4307999999999999e-05 2023-07-02T14:05:48.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:48.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025611+0000", 2023-07-02T14:05:48.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.032e-06 2023-07-02T14:05:48.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025617+0000", 2023-07-02T14:05:48.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8977999999999998e-05 2023-07-02T14:05:48.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025656+0000", 2023-07-02T14:05:48.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:05:48.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:48.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025663+0000", 2023-07-02T14:05:48.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095220739999999998 2023-07-02T14:05:48.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120884+0000", 2023-07-02T14:05:48.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2525000000000001e-05 2023-07-02T14:05:48.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:48.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120956+0000", 2023-07-02T14:05:48.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016232400000000001 2023-07-02T14:05:48.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:48.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.121119+0000", 2023-07-02T14:05:48.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044007380000000004 2023-07-02T14:05:48.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:48.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.125519+0000", 2023-07-02T14:05:48.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000797209 2023-07-02T14:05:48.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:48.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126317+0000", 2023-07-02T14:05:48.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1228000000000004e-05 2023-07-02T14:05:48.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:48.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126378+0000", 2023-07-02T14:05:48.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978000000000003e-05 2023-07-02T14:05:48.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:48.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126430+0000", 2023-07-02T14:05:48.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999 2023-07-02T14:05:48.125 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.125 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.126 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.126 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:189 2.3 2:c9621172:::rbd_data.125de1449605.00000000000000c0:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:48.126 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:48.126 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.875714574, 2023-07-02T14:05:48.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001, 2023-07-02T14:05:48.127 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:48.127 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:48.127 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:48.127 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:48.127 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:48.127 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:05:48.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.127 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:48.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:48.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:48.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:48.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:48.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:48.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:48.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:48.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:48.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2023-07-02T14:05:48.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:48.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172682+0000", 2023-07-02T14:05:48.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0159999999999997e-06 2023-07-02T14:05:48.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:48.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172688+0000", 2023-07-02T14:05:48.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0010000000000003e-06 2023-07-02T14:05:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172692+0000", 2023-07-02T14:05:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026444333 2023-07-02T14:05:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199136+0000", 2023-07-02T14:05:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2230000000000004e-05 2023-07-02T14:05:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199218+0000", 2023-07-02T14:05:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012043595000000001 2023-07-02T14:05:48.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211262+0000", 2023-07-02T14:05:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001 2023-07-02T14:05:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:199 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.875671030999996, 2023-07-02T14:05:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002, 2023-07-02T14:05:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:48.134 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 199 2023-07-02T14:05:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:48.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172722+0000", 2023-07-02T14:05:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1250000000000001e-06 2023-07-02T14:05:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172725+0000", 2023-07-02T14:05:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5670000000000002e-06 2023-07-02T14:05:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172731+0000", 2023-07-02T14:05:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9310000000000001e-06 2023-07-02T14:05:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172735+0000", 2023-07-02T14:05:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030699852999999999 2023-07-02T14:05:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203435+0000", 2023-07-02T14:05:48.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6827e-05 2023-07-02T14:05:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203532+0000", 2023-07-02T14:05:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076955230000000001 2023-07-02T14:05:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211227+0000", 2023-07-02T14:05:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002 2023-07-02T14:05:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.141 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:281 2.1 2:901d5a95:::rbd_data.125de1449605.0000000000000118:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.492925755000002, 2023-07-02T14:05:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002, 2023-07-02T14:05:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:48.143 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:05:48.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.143 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:48.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:48.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:48.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:48.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:48.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:48.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:48.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:48.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:48.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0469999999999998e-06 2023-07-02T14:05:48.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:48.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555471+0000", 2023-07-02T14:05:48.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8860000000000001e-06 2023-07-02T14:05:48.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:48.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555476+0000", 2023-07-02T14:05:48.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3160000000000002e-06 2023-07-02T14:05:48.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555483+0000", 2023-07-02T14:05:48.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989525000000001 2023-07-02T14:05:48.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591472+0000", 2023-07-02T14:05:48.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3708999999999996e-05 2023-07-02T14:05:48.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:48.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591536+0000", 2023-07-02T14:05:48.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055109720000000003 2023-07-02T14:05:48.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:48.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.597047+0000", 2023-07-02T14:05:48.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002 2023-07-02T14:05:48.149 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.149 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.149 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.149 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12883 2.1 2:9ffc48e1:::rbd_data.125de1449605.0000000000000074:head [write 4173824~20480] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:05:48.149 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:48.150 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.402323281999998, 2023-07-02T14:05:48.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037871393000000003, 2023-07-02T14:05:48.150 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:48.150 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.150 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:48.150 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:48.150 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:48.151 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12883 2023-07-02T14:05:48.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.151 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:48.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:48.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:48.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:48.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:48.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:48.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:05:48.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:48.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646068+0000", 2023-07-02T14:05:48.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4935999999999999e-05 2023-07-02T14:05:48.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:48.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646103+0000", 2023-07-02T14:05:48.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.092e-05 2023-07-02T14:05:48.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:48.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646124+0000", 2023-07-02T14:05:48.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1519999999999998e-06 2023-07-02T14:05:48.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646133+0000", 2023-07-02T14:05:48.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012270200000000001 2023-07-02T14:05:48.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646255+0000", 2023-07-02T14:05:48.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014495400000000001 2023-07-02T14:05:48.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:48.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646400+0000", 2023-07-02T14:05:48.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000221626 2023-07-02T14:05:48.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:48.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646622+0000", 2023-07-02T14:05:48.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078929050000000004 2023-07-02T14:05:48.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:48.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.654515+0000", 2023-07-02T14:05:48.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029307343999999999 2023-07-02T14:05:48.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:48.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683822+0000", 2023-07-02T14:05:48.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8298000000000002e-05 2023-07-02T14:05:48.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:48.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683891+0000", 2023-07-02T14:05:48.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1381999999999997e-05 2023-07-02T14:05:48.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:48.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683942+0000", 2023-07-02T14:05:48.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037871393000000003 2023-07-02T14:05:48.159 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.159 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.159 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.160 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12985 2.7 2:eeddf24b:::rbd_data.125de1449605.000000000000007e:head [write 65536~4128768] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:05:48.160 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:48.160 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.265398808999997, 2023-07-02T14:05:48.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999, 2023-07-02T14:05:48.160 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:48.160 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.161 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:48.161 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:48.161 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:48.161 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12985 2023-07-02T14:05:48.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.161 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:48.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:48.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:48.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:48.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:48.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:48.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:48.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:48.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782994+0000", 2023-07-02T14:05:48.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019254741999999998 2023-07-02T14:05:48.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:48.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802248+0000", 2023-07-02T14:05:48.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0451e-05 2023-07-02T14:05:48.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:48.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802279+0000", 2023-07-02T14:05:48.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3318e-05 2023-07-02T14:05:48.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802292+0000", 2023-07-02T14:05:48.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109403 2023-07-02T14:05:48.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802402+0000", 2023-07-02T14:05:48.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012520599999999999 2023-07-02T14:05:48.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:48.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802527+0000", 2023-07-02T14:05:48.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187768 2023-07-02T14:05:48.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:05:48.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802715+0000", 2023-07-02T14:05:48.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013192822 2023-07-02T14:05:48.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:48.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.815907+0000", 2023-07-02T14:05:48.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0086451870000000004 2023-07-02T14:05:48.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:48.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824553+0000", 2023-07-02T14:05:48.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6759999999999999e-05 2023-07-02T14:05:48.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:48.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824609+0000", 2023-07-02T14:05:48.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011314700000000001 2023-07-02T14:05:48.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:48.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824722+0000", 2023-07-02T14:05:48.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999 2023-07-02T14:05:48.170 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.170 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.170 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.170 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:165 2.3 2:c9c6393a:::rbd_data.125de1449605.000000000000002c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:48.170 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:48.171 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 51.208615991999999, 2023-07-02T14:05:48.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002, 2023-07-02T14:05:48.171 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:48.171 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:48.171 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:48.171 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:48.171 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:48.171 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 165 2023-07-02T14:05:48.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.172 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:48.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:48.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:48.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:48.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:48.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:48.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:48.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:48.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839777+0000", 2023-07-02T14:05:48.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6740000000000001e-06 2023-07-02T14:05:48.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:48.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839780+0000", 2023-07-02T14:05:48.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9030000000000004e-06 2023-07-02T14:05:48.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:48.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839788+0000", 2023-07-02T14:05:48.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1799999999999998e-06 2023-07-02T14:05:48.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839792+0000", 2023-07-02T14:05:48.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.031592227000000001 2023-07-02T14:05:48.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871384+0000", 2023-07-02T14:05:48.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013600099999999999 2023-07-02T14:05:48.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:48.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871520+0000", 2023-07-02T14:05:48.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082817089999999999 2023-07-02T14:05:48.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:48.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.879802+0000", 2023-07-02T14:05:48.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002 2023-07-02T14:05:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:252 2.7 2:f1d0abf0:::rbd_data.125de1449605.0000000000000083:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:48.178 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.792331021999999, 2023-07-02T14:05:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997, 2023-07-02T14:05:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 252 2023-07-02T14:05:48.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256062+0000", 2023-07-02T14:05:48.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0149999999999999e-06 2023-07-02T14:05:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256065+0000", 2023-07-02T14:05:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5269999999999998e-06 2023-07-02T14:05:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256071+0000", 2023-07-02T14:05:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:05:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256074+0000", 2023-07-02T14:05:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028852909 2023-07-02T14:05:48.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.284927+0000", 2023-07-02T14:05:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168274 2023-07-02T14:05:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.285096+0000", 2023-07-02T14:05:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011036300000000001 2023-07-02T14:05:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296132+0000", 2023-07-02T14:05:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997 2023-07-02T14:05:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.185 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:261 2.1 2:971731cb:::rbd_data.125de1449605.000000000000008c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.792235546999997, 2023-07-02T14:05:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984, 2023-07-02T14:05:48.186 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 261 2023-07-02T14:05:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:48.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:48.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2819999999999999e-06 2023-07-02T14:05:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256160+0000", 2023-07-02T14:05:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2020000000000003e-06 2023-07-02T14:05:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256165+0000", 2023-07-02T14:05:48.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2100000000000003e-06 2023-07-02T14:05:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256170+0000", 2023-07-02T14:05:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025871715999999999 2023-07-02T14:05:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282041+0000", 2023-07-02T14:05:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132364 2023-07-02T14:05:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282174+0000", 2023-07-02T14:05:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014044566 2023-07-02T14:05:48.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296218+0000", 2023-07-02T14:05:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984 2023-07-02T14:05:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.193 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:264 2.3 2:dbf06c9d:::rbd_data.125de1449605.000000000000008f:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.792201429000002, 2023-07-02T14:05:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003, 2023-07-02T14:05:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:48.194 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 264 2023-07-02T14:05:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:48.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5749999999999999e-06 2023-07-02T14:05:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256195+0000", 2023-07-02T14:05:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.327e-06 2023-07-02T14:05:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256200+0000", 2023-07-02T14:05:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0580000000000002e-06 2023-07-02T14:05:48.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256203+0000", 2023-07-02T14:05:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035707524999999997 2023-07-02T14:05:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.291911+0000", 2023-07-02T14:05:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014269799999999999 2023-07-02T14:05:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.292053+0000", 2023-07-02T14:05:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062760309999999996 2023-07-02T14:05:48.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.298329+0000", 2023-07-02T14:05:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003 2023-07-02T14:05:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:341 2.3 2:cee258bd:::rbd_data.125de1449605.00000000000000dc:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.411714101999998, 2023-07-02T14:05:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997, 2023-07-02T14:05:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:48.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 341 2023-07-02T14:05:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:48.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636679+0000", 2023-07-02T14:05:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6240000000000002e-06 2023-07-02T14:05:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636682+0000", 2023-07-02T14:05:48.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6910000000000002e-06 2023-07-02T14:05:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636688+0000", 2023-07-02T14:05:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3469999999999999e-06 2023-07-02T14:05:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636691+0000", 2023-07-02T14:05:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035625560000000001 2023-07-02T14:05:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672317+0000", 2023-07-02T14:05:48.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108391 2023-07-02T14:05:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672425+0000", 2023-07-02T14:05:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012104976 2023-07-02T14:05:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.684530+0000", 2023-07-02T14:05:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997 2023-07-02T14:05:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.209 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPushReply(2.7 139/134 [PushReplyOp(2:e3782f39:::rbd_data.125de1449605.0000000000000120:head)])", 2023-07-02T14:05:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:05:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.121181948999997, 2023-07-02T14:05:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998, 2023-07-02T14:05:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:05:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:05:48.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927211+0000", 2023-07-02T14:05:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3689999999999998e-06 2023-07-02T14:05:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927217+0000", 2023-07-02T14:05:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7269999999999996e-06 2023-07-02T14:05:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927225+0000", 2023-07-02T14:05:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:05:48.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927230+0000", 2023-07-02T14:05:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:05:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927266+0000", 2023-07-02T14:05:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.069972871000000006 2023-07-02T14:05:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.997239+0000", 2023-07-02T14:05:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998 2023-07-02T14:05:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.215 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:17593 2.3 2:dd618512:::rbd_data.125de1449605.000000000000008b:head [write 65536~3932160] snapc 2e=[2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:43.549480+0000", 2023-07-02T14:05:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.4989143580000004, 2023-07-02T14:05:48.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.03787782, 2023-07-02T14:05:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17593 2023-07-02T14:05:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.217 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.549480+0000", 2023-07-02T14:05:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:48.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.549480+0000", 2023-07-02T14:05:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999943 2023-07-02T14:05:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.549474+0000", 2023-07-02T14:05:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049631190000000002 2023-07-02T14:05:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.554437+0000", 2023-07-02T14:05:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7652e-05 2023-07-02T14:05:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.554455+0000", 2023-07-02T14:05:48.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4089999999999997e-06 2023-07-02T14:05:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.554462+0000", 2023-07-02T14:05:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1801000000000002e-05 2023-07-02T14:05:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.554504+0000", 2023-07-02T14:05:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000125489 2023-07-02T14:05:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.554629+0000", 2023-07-02T14:05:48.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000178903 2023-07-02T14:05:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.554808+0000", 2023-07-02T14:05:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0098879320000000003 2023-07-02T14:05:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.564696+0000", 2023-07-02T14:05:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022554149999999999 2023-07-02T14:05:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.587250+0000", 2023-07-02T14:05:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7098000000000001e-05 2023-07-02T14:05:48.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.587308+0000", 2023-07-02T14:05:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9795e-05 2023-07-02T14:05:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:48.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.587357+0000", 2023-07-02T14:05:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.03787782 2023-07-02T14:05:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.226 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:17596 2.1 2:965e04ab:::rbd_data.125de1449605.0000000000000089:head [write 65536~3997696] snapc 2e=[2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:43.554508+0000", 2023-07-02T14:05:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.4938854389999996, 2023-07-02T14:05:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037965229000000003, 2023-07-02T14:05:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:48.227 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17596 2023-07-02T14:05:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.554508+0000", 2023-07-02T14:05:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:48.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.554508+0000", 2023-07-02T14:05:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:48.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.554508+0000", 2023-07-02T14:05:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054068659999999998 2023-07-02T14:05:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.559915+0000", 2023-07-02T14:05:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1087999999999999e-05 2023-07-02T14:05:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.559936+0000", 2023-07-02T14:05:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0188e-05 2023-07-02T14:05:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.559946+0000", 2023-07-02T14:05:48.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9580999999999998e-05 2023-07-02T14:05:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.559996+0000", 2023-07-02T14:05:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0260999999999995e-05 2023-07-02T14:05:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.560086+0000", 2023-07-02T14:05:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012091699999999999 2023-07-02T14:05:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.560207+0000", 2023-07-02T14:05:48.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021177420999999998 2023-07-02T14:05:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.581384+0000", 2023-07-02T14:05:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010973284 2023-07-02T14:05:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.592357+0000", 2023-07-02T14:05:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1208e-05 2023-07-02T14:05:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.592429+0000", 2023-07-02T14:05:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4962000000000001e-05 2023-07-02T14:05:48.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.592474+0000", 2023-07-02T14:05:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037965229000000003 2023-07-02T14:05:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:48.237 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:48.237 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:48.238 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:05:48.239 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:48.238+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:48.339 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:48.339+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:48.376 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:48.376 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:48.377 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:48.377 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:48.387 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:05:48.396 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:05:48.469 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:48.470+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:48.541 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:48.541+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:48.552 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:48.552 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:48.552 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:48.552 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:48.552 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:48.562 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:05:48.645 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:48.646+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:48.728 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:48.729 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:48.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:48.729 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:48.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.729 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:05:48.729 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:48.729 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.829117604, 2023-07-02T14:05:48.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:05:48.730 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:48.730 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.730 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:48.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:48.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:48.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:48.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:48.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:48.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:48.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:48.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:05:48.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:05:48.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:48.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:05:48.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:05:48.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:48.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:05:48.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:05:48.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:05:48.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:05:48.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:48.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:05:48.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:05:48.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:48.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:05:48.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:05:48.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:48.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:05:48.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:05:48.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:48.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:05:48.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:05:48.737 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.737 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.737 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.737 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:05:48.738 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:48.738 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.825031886, 2023-07-02T14:05:48.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:05:48.738 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:48.738 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.738 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:48.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:48.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:48.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:48.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:48.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:48.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:48.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:48.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:05:48.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:05:48.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:48.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:05:48.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:05:48.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:48.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:05:48.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:05:48.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:05:48.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:05:48.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:48.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:05:48.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:05:48.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:48.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:05:48.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:05:48.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:48.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:05:48.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:05:48.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:48.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:05:48.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:05:48.745 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.745 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.745 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.745 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:05:48.745 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:48.745 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.821167105, 2023-07-02T14:05:48.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:05:48.746 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:48.746 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.746 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:48.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:48.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:48.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:48.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:48.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:48.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:48.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:48.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:05:48.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:05:48.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:48.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:05:48.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:05:48.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:48.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:05:48.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:05:48.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:05:48.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:05:48.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:48.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:05:48.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:05:48.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:48.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:05:48.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:05:48.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:48.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:05:48.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:05:48.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:48.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:05:48.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:05:48.752 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.753 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.753 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.753 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:05:48.753 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:48.753 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.81764284400001, 2023-07-02T14:05:48.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:05:48.754 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:48.754 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.754 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:48.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:48.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:48.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:48.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:48.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:48.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:48.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:48.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:05:48.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:05:48.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:48.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:05:48.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:05:48.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:48.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:05:48.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:05:48.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:05:48.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:05:48.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:48.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:05:48.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:05:48.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:48.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:05:48.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:05:48.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:48.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:05:48.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:05:48.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:48.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:05:48.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:05:48.760 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.760 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.761 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.761 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:05:48.761 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:48.761 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.781938045, 2023-07-02T14:05:48.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:05:48.761 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:48.762 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.762 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:48.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:48.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:48.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:48.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:48.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:48.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:48.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:48.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:05:48.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:05:48.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:48.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:05:48.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:05:48.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:48.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:05:48.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:05:48.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:05:48.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:05:48.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:48.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:05:48.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:05:48.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:48.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:05:48.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:05:48.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:48.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:05:48.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:05:48.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:48.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:05:48.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:05:48.768 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.768 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.768 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.769 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:05:48.769 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:48.769 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.777280687, 2023-07-02T14:05:48.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:05:48.769 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:48.769 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.770 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:48.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:48.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:48.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:48.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:48.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:48.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:48.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:48.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:05:48.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:05:48.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:48.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:05:48.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:05:48.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:48.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:05:48.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:05:48.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:05:48.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:05:48.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:48.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:05:48.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:05:48.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:48.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:05:48.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:05:48.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:48.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:05:48.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:05:48.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:48.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:05:48.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:05:48.776 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.776 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.776 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.777 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:05:48.777 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:48.777 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.77634082399999, 2023-07-02T14:05:48.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:05:48.777 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:48.777 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.777 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:48.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:48.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:48.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:48.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:48.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:48.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:48.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:48.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:05:48.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:05:48.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:48.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:05:48.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:05:48.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:48.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:05:48.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:05:48.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:05:48.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:05:48.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:48.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:05:48.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:05:48.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:48.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:05:48.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:05:48.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:48.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:05:48.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:05:48.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:48.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:05:48.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:05:48.784 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.784 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.784 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.784 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:05:48.784 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:48.785 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 128.77494294600001, 2023-07-02T14:05:48.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:05:48.785 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:48.785 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.785 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:48.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:48.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:48.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:48.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:48.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:48.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:48.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:48.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:05:48.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:05:48.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:48.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:05:48.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:05:48.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:48.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:05:48.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:05:48.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:05:48.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:05:48.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:48.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:05:48.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:05:48.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:48.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:05:48.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:05:48.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:48.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:05:48.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:05:48.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:48.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:05:48.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:05:48.791 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.792 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.792 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.792 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:05:48.792 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:48.792 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 107.36475245, 2023-07-02T14:05:48.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:05:48.793 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:48.793 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.793 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:48.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:48.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:48.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:48.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:48.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:48.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:48.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:48.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:05:48.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:05:48.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:48.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:05:48.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:05:48.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:48.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:05:48.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:05:48.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:05:48.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:05:48.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:48.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:05:48.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:05:48.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:48.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:05:48.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:05:48.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:48.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:05:48.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:05:48.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:48.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:05:48.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:05:48.799 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.799 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.800 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.800 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:48.800 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:48.800 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 105.998402766, 2023-07-02T14:05:48.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:05:48.800 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:48.801 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.801 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:48.801 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:48.801 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:48.801 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:05:48.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.801 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:48.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:48.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:48.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:48.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:48.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:48.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:48.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:48.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:05:48.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:05:48.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:48.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:05:48.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:05:48.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:48.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:05:48.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:05:48.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:05:48.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:05:48.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:48.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:05:48.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:05:48.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:48.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:05:48.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:05:48.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:48.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:05:48.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:05:48.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:48.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:05:48.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:05:48.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:48.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:05:48.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:05:48.808 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.809 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.809 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.809 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7707 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 3145728~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:48.809 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:48.809 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 105.994419299, 2023-07-02T14:05:48.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999, 2023-07-02T14:05:48.810 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:48.810 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.810 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:48.810 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:48.810 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:48.810 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7707 2023-07-02T14:05:48.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.810 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:48.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:48.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:48.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:48.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:48.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:48.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:48.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:48.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:48.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5017e-05 2023-07-02T14:05:48.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:48.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732347+0000", 2023-07-02T14:05:48.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5821e-05 2023-07-02T14:05:48.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:48.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732363+0000", 2023-07-02T14:05:48.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1270000000000002e-06 2023-07-02T14:05:48.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732368+0000", 2023-07-02T14:05:48.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008444666 2023-07-02T14:05:48.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:48.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740812+0000", 2023-07-02T14:05:48.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011266599999999999 2023-07-02T14:05:48.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:48.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740925+0000", 2023-07-02T14:05:48.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010848808999999999 2023-07-02T14:05:48.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:48.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751774+0000", 2023-07-02T14:05:48.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8421999999999997e-05 2023-07-02T14:05:48.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:48.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751832+0000", 2023-07-02T14:05:48.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107453 2023-07-02T14:05:48.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:48.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751940+0000", 2023-07-02T14:05:48.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999 2023-07-02T14:05:48.817 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.818 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.818 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.818 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:48.818 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:48.818 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 105.994355422, 2023-07-02T14:05:48.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:05:48.819 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:48.819 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.819 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:48.819 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:48.819 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:48.819 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:05:48.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.820 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:48.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:48.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:48.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:48.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:48.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:48.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:48.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:48.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:48.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:05:48.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:48.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:05:48.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:05:48.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:48.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:05:48.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:05:48.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:05:48.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:05:48.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:48.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:05:48.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:05:48.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:48.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:05:48.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:05:48.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:48.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:05:48.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:05:48.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:48.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:05:48.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:05:48.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:48.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:05:48.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:05:48.827 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.827 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.827 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.827 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:48.827 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:48.827 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 105.98979159700001, 2023-07-02T14:05:48.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:05:48.828 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:48.828 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.828 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:48.828 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:48.828 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:48.828 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:05:48.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.829 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:48.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:48.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:48.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:48.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:48.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:48.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:48.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:48.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:05:48.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:05:48.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:48.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:05:48.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:05:48.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:48.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:05:48.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:05:48.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:05:48.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:05:48.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:48.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:05:48.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:05:48.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:48.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:05:48.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:05:48.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:48.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:05:48.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:05:48.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:48.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:05:48.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:05:48.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:48.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:05:48.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:05:48.836 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.836 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.836 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.836 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:48.837 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:48.837 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 105.989741731, 2023-07-02T14:05:48.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:05:48.837 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:48.837 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.837 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:48.837 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:48.837 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:48.838 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:05:48.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.838 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:48.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:48.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:48.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:48.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:48.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:48.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:48.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:48.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:05:48.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:05:48.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:48.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:05:48.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:05:48.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:48.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:05:48.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:05:48.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:05:48.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:05:48.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:48.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:05:48.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:05:48.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:48.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:05:48.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:05:48.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:48.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:05:48.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:05:48.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:48.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:05:48.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:05:48.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:48.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:05:48.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:05:48.845 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.845 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.845 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.845 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:48.846 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:48.846 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 105.988702738, 2023-07-02T14:05:48.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:05:48.846 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:48.846 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.846 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:48.846 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:48.846 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:48.847 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:05:48.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.847 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:48.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:48.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:48.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:48.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:48.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:48.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:48.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:48.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:05:48.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:05:48.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:48.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:05:48.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:05:48.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:48.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:05:48.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:05:48.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:05:48.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:05:48.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:48.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:05:48.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:05:48.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:48.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:05:48.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:05:48.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:48.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:05:48.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:05:48.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:48.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:05:48.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:05:48.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:48.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:05:48.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:05:48.854 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.854 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.854 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.855 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7993 2.7 2:fe72db53:::rbd_data.125de1449605.0000000000000072:head [write 65536~3276800] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:48.855 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:48.855 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 105.80174469000001, 2023-07-02T14:05:48.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919, 2023-07-02T14:05:48.855 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:48.855 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.855 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:48.855 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:48.856 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:48.856 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7993 2023-07-02T14:05:48.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.856 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:48.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:48.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:48.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:48.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:48.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:48.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:48.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:48.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924986+0000", 2023-07-02T14:05:48.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049542070000000004 2023-07-02T14:05:48.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:48.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929940+0000", 2023-07-02T14:05:48.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9740000000000001e-05 2023-07-02T14:05:48.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:48.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929960+0000", 2023-07-02T14:05:48.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.311e-05 2023-07-02T14:05:48.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929973+0000", 2023-07-02T14:05:48.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9629000000000002e-05 2023-07-02T14:05:48.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:48.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930023+0000", 2023-07-02T14:05:48.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012751799999999999 2023-07-02T14:05:48.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:48.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930150+0000", 2023-07-02T14:05:48.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016449042000000001 2023-07-02T14:05:48.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:48.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946599+0000", 2023-07-02T14:05:48.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3241e-05 2023-07-02T14:05:48.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:48.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946663+0000", 2023-07-02T14:05:48.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000102296 2023-07-02T14:05:48.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:48.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946765+0000", 2023-07-02T14:05:48.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919 2023-07-02T14:05:48.863 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.863 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.863 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.864 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11782 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 196608~2162688] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:05:48.864 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:48.864 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 64.317224768000003, 2023-07-02T14:05:48.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365, 2023-07-02T14:05:48.864 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:48.864 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.864 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:48.865 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:48.865 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:48.865 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11782 2023-07-02T14:05:48.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.865 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:48.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:48.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:48.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:48.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:48.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:48.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:48.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:48.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409506+0000", 2023-07-02T14:05:48.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020172269999999999 2023-07-02T14:05:48.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:48.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411523+0000", 2023-07-02T14:05:48.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8901000000000001e-05 2023-07-02T14:05:48.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:48.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411542+0000", 2023-07-02T14:05:48.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5037e-05 2023-07-02T14:05:48.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411557+0000", 2023-07-02T14:05:48.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001931213 2023-07-02T14:05:48.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:48.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413489+0000", 2023-07-02T14:05:48.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000149207 2023-07-02T14:05:48.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:48.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413638+0000", 2023-07-02T14:05:48.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024752800000000001 2023-07-02T14:05:48.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:48.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413885+0000", 2023-07-02T14:05:48.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048516000000000002 2023-07-02T14:05:48.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:48.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.418737+0000", 2023-07-02T14:05:48.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010319259000000001 2023-07-02T14:05:48.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:48.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429056+0000", 2023-07-02T14:05:48.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2518000000000002e-05 2023-07-02T14:05:48.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:48.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429099+0000", 2023-07-02T14:05:48.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015275 2023-07-02T14:05:48.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:48.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429251+0000", 2023-07-02T14:05:48.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365 2023-07-02T14:05:48.873 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.873 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.874 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.874 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPushReply(2.7 119/117 [PushReplyOp(2:eb862bc0:::rbd_data.125de1449605.0000000000000040:13)])", 2023-07-02T14:05:48.874 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:48.874 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 64.316061493000007, 2023-07-02T14:05:48.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999, 2023-07-02T14:05:48.874 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:48.875 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "reached pg", 2023-07-02T14:05:48.875 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:48.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:48.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:48.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:48.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:48.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:48.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:48.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:48.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:48.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7670000000000001e-06 2023-07-02T14:05:48.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:48.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410674+0000", 2023-07-02T14:05:48.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3973e-05 2023-07-02T14:05:48.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:48.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410688+0000", 2023-07-02T14:05:48.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0157e-05 2023-07-02T14:05:48.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410708+0000", 2023-07-02T14:05:48.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018578779 2023-07-02T14:05:48.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:48.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429287+0000", 2023-07-02T14:05:48.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013086549999999999 2023-07-02T14:05:48.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:48.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.430595+0000", 2023-07-02T14:05:48.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999 2023-07-02T14:05:48.880 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.880 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.880 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.881 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11861 2.7 2:e57d3f55:::rbd_data.125de1449605.0000000000000134:head [write 0~65536] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:05:48.881 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:48.881 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 64.181946116000006, 2023-07-02T14:05:48.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723, 2023-07-02T14:05:48.881 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:48.881 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.881 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:48.881 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:48.882 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:48.882 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11861 2023-07-02T14:05:48.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.882 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:48.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:48.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:48.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:48.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:48.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:48.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:48.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:48.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544785+0000", 2023-07-02T14:05:48.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010563700000000001 2023-07-02T14:05:48.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:48.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544890+0000", 2023-07-02T14:05:48.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8530000000000004e-06 2023-07-02T14:05:48.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:48.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544900+0000", 2023-07-02T14:05:48.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.665e-06 2023-07-02T14:05:48.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544906+0000", 2023-07-02T14:05:48.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000272575 2023-07-02T14:05:48.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:48.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545178+0000", 2023-07-02T14:05:48.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000111362 2023-07-02T14:05:48.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:48.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545290+0000", 2023-07-02T14:05:48.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016181100000000001 2023-07-02T14:05:48.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:48.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545452+0000", 2023-07-02T14:05:48.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044500750000000004 2023-07-02T14:05:48.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:48.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.549902+0000", 2023-07-02T14:05:48.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014850024 2023-07-02T14:05:48.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:48.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564752+0000", 2023-07-02T14:05:48.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6272000000000003e-05 2023-07-02T14:05:48.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:48.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564848+0000", 2023-07-02T14:05:48.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000162428 2023-07-02T14:05:48.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:48.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.565010+0000", 2023-07-02T14:05:48.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723 2023-07-02T14:05:48.890 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.890 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.891 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.891 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:13262 2.7 e125/121 2:e0741dde:::rbd_data.125de1449605.00000000000000ad:head v 125'2024, mlcod=125'2021)", 2023-07-02T14:05:48.891 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:48.891 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 55.475868378000001, 2023-07-02T14:05:48.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999, 2023-07-02T14:05:48.891 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:48.892 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:48.892 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:48.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:48.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:48.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:48.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.893 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:48.893 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:48.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:48.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.893 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:48.893 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250862+0000", 2023-07-02T14:05:48.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080378800000000007 2023-07-02T14:05:48.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:48.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258900+0000", 2023-07-02T14:05:48.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8939999999999999e-06 2023-07-02T14:05:48.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:48.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258909+0000", 2023-07-02T14:05:48.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3560000000000003e-06 2023-07-02T14:05:48.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:48.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258914+0000", 2023-07-02T14:05:48.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2429000000000003e-05 2023-07-02T14:05:48.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:48.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258956+0000", 2023-07-02T14:05:48.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5094999999999999e-05 2023-07-02T14:05:48.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:48.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258981+0000", 2023-07-02T14:05:48.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012177307 2023-07-02T14:05:48.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:48.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271158+0000", 2023-07-02T14:05:48.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2104999999999999e-05 2023-07-02T14:05:48.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:48.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:48.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:48.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271181+0000", 2023-07-02T14:05:48.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999 2023-07-02T14:05:48.898 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.898 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.898 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.899 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:48.899 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:48.899 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:48.900 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:05:48.901 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:48.900+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:48.992 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:05:49.000 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:49.001+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:49.064 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:49.064 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:49.065 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:49.065 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:49.075 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:05:49.117 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:49.117+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:49.138 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:49.155 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:49.173 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:49.191 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:49.202 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:49.202+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:49.208 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:49.209 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-07-02T14:05:49.209 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:05:49.209 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:05:49.209 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:05:49.209 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:05:49.209 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:05:49.211 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:05:49.212 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:05:49.240 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:49.240 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:49.250 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:05:49.303 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:49.303+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:49.374 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:49.374 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:49.384 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:05:49.425 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:49.425+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:49.437 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap43 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap44 b3fbb028-e267-4d25-a750-93ba18a2316a@snap44.diff 2023-07-02T14:05:49.504 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete...2023-07-02T14:05:49.503+0000 7f86b59d9700 0 -- 172.21.15.163:0/52776445 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7f869c029de0 msgr2=0x7f869c006240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:49.504 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:49.505+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:49.508 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:05:49.507+0000 7f86b61da700 0 -- 172.21.15.163:0/52776445 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f869c004b10 msgr2=0x7f869c004f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:49.561 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:49.561 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:49.572 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:05:49.613 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:49.613+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:49.706 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:49.706+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:49.748 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:49.748 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:49.759 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:05:49.844 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:49.844+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:49.908 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:49.908+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:49.928 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:49.928 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:49.941 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:05:50.008 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:50.008+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:50.096 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:50.097 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:50.109 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:50.109+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:50.112 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:05:50.210 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:50.210+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:50.278 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:50.278 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:50.298 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:05:50.339 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:50.339+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:50.344 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete...2023-07-02T14:05:50.343+0000 7f86b61da700 0 -- 172.21.15.163:0/52776445 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f869c029de0 msgr2=0x7f869803b770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:50.349 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:05:50.348+0000 7f86b59d9700 0 -- 172.21.15.163:0/52776445 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7f8698036bb0 msgr2=0x7f8698038fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:50.353 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:05:50.352+0000 7f86b61da700 0 -- 172.21.15.163:0/52776445 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f869c004b10 msgr2=0x7f869803e160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:50.409 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:05:50.411 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:50.412+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:50.412 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete...2023-07-02T14:05:50.412+0000 7f86b69db700 0 -- 172.21.15.163:0/52776445 >> [v2:172.21.15.22:6808/99977,v1:172.21.15.22:6809/99977] conn(0x55e8717edcd0 msgr2=0x55e8717f0140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:50.478 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:50.478 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:50.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:05:50.534 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:50.534+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:50.613 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:50.613+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:50.646 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:50.647 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:50.659 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:05:50.714 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:50.715+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:50.759 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 100% complete...done. 2023-07-02T14:05:50.782 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:05:50.782 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=45 2023-07-02T14:05:50.782 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap45 2023-07-02T14:05:50.815 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:50.815+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:50.836 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:50.836+0000 7fd57ef6e200 -1 osd.0 148 log_to_monitors {default=true} 2023-07-02T14:05:50.846 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:50.846 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:50.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:05:50.916 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:50.917+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:50.980 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:50.981 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:50.991 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:05:51.033 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:51.034+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:51.048 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:05:51.116 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:51.116 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:51.117 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:51.117+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:51.126 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:05:51.168 INFO:teuthology.orchestra.run.smithi016.stderr:no valid command found; 10 closest matches: 2023-07-02T14:05:51.168 INFO:teuthology.orchestra.run.smithi016.stderr:0 2023-07-02T14:05:51.169 INFO:teuthology.orchestra.run.smithi016.stderr:1 2023-07-02T14:05:51.169 INFO:teuthology.orchestra.run.smithi016.stderr:2 2023-07-02T14:05:51.169 INFO:teuthology.orchestra.run.smithi016.stderr:abort 2023-07-02T14:05:51.169 INFO:teuthology.orchestra.run.smithi016.stderr:assert 2023-07-02T14:05:51.169 INFO:teuthology.orchestra.run.smithi016.stderr:bluefs debug_inject_read_zeros 2023-07-02T14:05:51.169 INFO:teuthology.orchestra.run.smithi016.stderr:bluefs files list 2023-07-02T14:05:51.169 INFO:teuthology.orchestra.run.smithi016.stderr:bluefs stats 2023-07-02T14:05:51.169 INFO:teuthology.orchestra.run.smithi016.stderr:bluestore allocator dump block 2023-07-02T14:05:51.170 INFO:teuthology.orchestra.run.smithi016.stderr:bluestore allocator fragmentation block 2023-07-02T14:05:51.170 INFO:teuthology.orchestra.run.smithi016.stderr:admin_socket: invalid command 2023-07-02T14:05:51.170 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-07-02T14:05:51.171 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2023-07-02T14:05:51.213 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:05:51.257 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:51.257+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:51.289 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:51.290 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:51.299 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:05:51.318 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:51.318+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:51.353 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:51.370 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:51.388 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:51.404 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:05:51.411 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:51.414 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:51.414+0000 7fd5720a7700 -1 osd.0 148 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-02T14:05:51.419 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:51.419+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:51.430 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:51.431 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:51.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:51.431 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:51.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.431 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:05:51.431 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:51.432 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 131.53170058200001, 2023-07-02T14:05:51.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:05:51.432 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:51.432 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:51.432 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:51.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:51.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:51.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:51.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:51.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:51.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:51.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:51.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:05:51.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:05:51.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:51.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:05:51.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:05:51.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:51.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:05:51.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:05:51.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:51.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:05:51.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:05:51.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:51.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:05:51.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:05:51.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:51.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:05:51.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:05:51.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:51.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:05:51.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:05:51.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:51.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:05:51.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:05:51.439 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.439 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.439 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.439 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:05:51.439 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:51.440 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 131.52761486399999, 2023-07-02T14:05:51.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:05:51.440 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:51.440 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:51.440 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:51.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:51.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:51.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:51.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:51.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:51.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:51.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:51.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:05:51.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:05:51.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:51.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:05:51.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:05:51.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:51.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:05:51.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:05:51.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:51.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:05:51.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:05:51.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:51.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:05:51.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:05:51.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:51.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:05:51.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:05:51.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:51.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:05:51.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:05:51.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:51.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:05:51.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:05:51.447 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.447 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.447 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.448 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:05:51.448 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:51.448 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 131.52375008300001, 2023-07-02T14:05:51.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:05:51.448 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:51.448 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:51.448 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:51.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:51.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:51.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:51.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:51.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:51.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:51.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:51.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:05:51.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:05:51.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:51.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:05:51.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:05:51.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:51.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:05:51.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:05:51.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:51.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:05:51.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:05:51.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:51.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:05:51.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:05:51.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:51.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:05:51.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:05:51.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:51.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:05:51.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:05:51.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:51.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:05:51.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:05:51.455 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.455 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.455 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.456 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:05:51.456 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:51.456 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 131.52022582199999, 2023-07-02T14:05:51.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:05:51.456 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:51.456 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:51.456 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:51.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:51.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:51.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:51.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:51.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:51.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:51.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:51.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:05:51.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:05:51.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:51.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:05:51.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:05:51.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:51.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:05:51.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:05:51.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:51.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:05:51.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:05:51.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:51.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:05:51.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:05:51.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:51.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:05:51.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:05:51.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:51.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:05:51.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:05:51.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:51.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:05:51.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:05:51.463 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.463 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.463 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.463 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:05:51.464 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:51.464 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 131.48452102300001, 2023-07-02T14:05:51.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:05:51.464 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:51.464 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:51.464 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:51.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:51.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:51.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:51.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:51.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:51.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:51.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:51.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:05:51.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:05:51.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:51.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:05:51.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:05:51.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:51.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:05:51.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:05:51.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:51.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:05:51.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:05:51.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:51.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:05:51.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:05:51.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:51.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:05:51.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:05:51.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:51.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:05:51.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:05:51.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:51.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:05:51.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:05:51.471 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.471 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.471 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.472 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:05:51.472 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:51.472 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 131.47986366500001, 2023-07-02T14:05:51.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:05:51.472 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:51.472 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:51.473 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:51.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:51.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:51.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:51.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:51.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:51.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:51.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:51.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:05:51.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:05:51.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:51.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:05:51.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:05:51.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:51.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:05:51.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:05:51.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:51.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:05:51.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:05:51.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:51.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:05:51.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:05:51.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:51.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:05:51.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:05:51.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:51.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:05:51.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:05:51.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:51.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:05:51.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:05:51.479 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.479 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.479 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.480 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:05:51.480 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:51.480 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 131.478923802, 2023-07-02T14:05:51.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:05:51.480 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:51.480 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:51.480 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:51.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:51.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:51.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:51.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:51.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:51.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:51.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:51.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:05:51.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:05:51.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:51.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:05:51.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:05:51.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:51.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:05:51.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:05:51.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:51.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:05:51.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:05:51.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:51.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:05:51.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:05:51.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:51.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:05:51.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:05:51.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:51.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:05:51.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:05:51.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:51.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:05:51.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:05:51.488 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.488 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.488 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.489 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:05:51.489 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:51.489 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 131.47752592399999, 2023-07-02T14:05:51.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:05:51.489 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:51.489 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:51.489 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:51.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:51.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:51.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:51.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:51.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:51.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:51.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:51.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:05:51.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:05:51.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:51.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:05:51.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:05:51.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:51.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:05:51.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:05:51.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:51.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:05:51.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:05:51.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:51.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:05:51.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:05:51.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:51.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:05:51.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:05:51.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:51.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:05:51.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:05:51.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:51.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:05:51.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:05:51.496 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.496 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.496 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.497 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:05:51.497 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:51.497 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 110.06733542800001, 2023-07-02T14:05:51.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:05:51.498 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:51.498 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:51.498 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:51.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:51.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:51.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:51.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:51.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:51.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:51.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:51.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:05:51.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:05:51.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:51.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:05:51.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:05:51.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:51.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:05:51.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:05:51.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:51.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:05:51.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:05:51.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:51.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:05:51.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:05:51.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:51.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:05:51.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:05:51.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:51.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:05:51.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:05:51.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:51.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:05:51.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:05:51.506 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.506 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.507 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.507 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:51.507 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:51.507 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 108.70098574399999, 2023-07-02T14:05:51.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:05:51.508 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:51.508 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:51.508 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:51.508 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:51.508 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:51.508 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:05:51.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.508 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:51.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:51.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:51.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:51.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:51.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:51.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:51.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:51.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:05:51.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:05:51.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:51.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:05:51.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:05:51.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:51.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:05:51.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:05:51.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:51.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:05:51.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:05:51.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:51.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:05:51.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:05:51.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:51.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:05:51.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:05:51.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:51.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:05:51.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:05:51.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:51.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:05:51.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:05:51.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:51.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:05:51.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:05:51.517 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.518 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.518 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.518 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7707 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 3145728~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:51.518 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:51.518 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 108.697002277, 2023-07-02T14:05:51.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999, 2023-07-02T14:05:51.519 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:51.519 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:51.519 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:51.519 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:51.519 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:51.519 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7707 2023-07-02T14:05:51.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.520 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:51.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:51.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:51.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:51.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:51.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:51.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:51.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:51.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:51.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5017e-05 2023-07-02T14:05:51.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:51.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732347+0000", 2023-07-02T14:05:51.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5821e-05 2023-07-02T14:05:51.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:51.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732363+0000", 2023-07-02T14:05:51.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1270000000000002e-06 2023-07-02T14:05:51.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:51.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732368+0000", 2023-07-02T14:05:51.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008444666 2023-07-02T14:05:51.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:51.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740812+0000", 2023-07-02T14:05:51.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011266599999999999 2023-07-02T14:05:51.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:51.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740925+0000", 2023-07-02T14:05:51.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010848808999999999 2023-07-02T14:05:51.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:51.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751774+0000", 2023-07-02T14:05:51.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8421999999999997e-05 2023-07-02T14:05:51.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:51.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751832+0000", 2023-07-02T14:05:51.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107453 2023-07-02T14:05:51.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:51.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751940+0000", 2023-07-02T14:05:51.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999 2023-07-02T14:05:51.527 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.527 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.527 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.528 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:51.528 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:51.528 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 108.69693839999999, 2023-07-02T14:05:51.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:05:51.528 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:51.528 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:51.529 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:51.529 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:51.529 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:51.529 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:05:51.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.529 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:51.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:51.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:51.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:51.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:51.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:51.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:51.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:51.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:51.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:05:51.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:51.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:05:51.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:05:51.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:51.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:05:51.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:05:51.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:51.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:05:51.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:05:51.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:51.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:05:51.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:05:51.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:51.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:05:51.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:05:51.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:51.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:05:51.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:05:51.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:51.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:05:51.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:05:51.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:51.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:05:51.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:05:51.537 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.537 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.537 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.537 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:51.537 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:51.537 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 108.692374575, 2023-07-02T14:05:51.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:05:51.538 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:51.538 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:51.538 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:51.538 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:51.538 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:51.538 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:05:51.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.539 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:51.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:51.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:51.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:51.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:51.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:51.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:51.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:51.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:05:51.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:05:51.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:51.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:05:51.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:05:51.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:51.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:05:51.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:05:51.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:51.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:05:51.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:05:51.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:51.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:05:51.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:05:51.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:51.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:05:51.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:05:51.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:51.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:05:51.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:05:51.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:51.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:05:51.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:05:51.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:51.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:05:51.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:05:51.546 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.546 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.546 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.546 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:51.546 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:51.547 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 108.692324709, 2023-07-02T14:05:51.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:05:51.547 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:51.547 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:51.547 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:51.547 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:51.547 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:51.547 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:05:51.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.548 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:51.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:51.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:51.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:51.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:51.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:51.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:51.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:51.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:05:51.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:05:51.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:51.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:05:51.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:05:51.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:51.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:05:51.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:05:51.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:51.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:05:51.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:05:51.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:51.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:05:51.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:05:51.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:51.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:05:51.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:05:51.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:51.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:05:51.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:05:51.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:51.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:05:51.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:05:51.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:51.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:05:51.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:05:51.555 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.555 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.555 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.555 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:51.556 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:51.556 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 108.691285716, 2023-07-02T14:05:51.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:05:51.556 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:51.556 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:51.556 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:51.556 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:51.557 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:51.557 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:05:51.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.557 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:51.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:51.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:51.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:51.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:51.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:51.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:51.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:51.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:05:51.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:05:51.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:51.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:05:51.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:05:51.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:51.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:05:51.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:05:51.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:51.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:05:51.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:05:51.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:51.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:05:51.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:05:51.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:51.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:05:51.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:05:51.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:51.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:05:51.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:05:51.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:51.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:05:51.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:05:51.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:51.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:05:51.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:05:51.564 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.564 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.564 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.565 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7993 2.7 2:fe72db53:::rbd_data.125de1449605.0000000000000072:head [write 65536~3276800] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:51.565 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:51.565 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 108.504327668, 2023-07-02T14:05:51.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919, 2023-07-02T14:05:51.565 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:51.565 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:51.565 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:51.565 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:51.566 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:51.566 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7993 2023-07-02T14:05:51.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.566 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:51.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:51.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:51.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:51.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:51.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:51.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:51.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:51.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924986+0000", 2023-07-02T14:05:51.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049542070000000004 2023-07-02T14:05:51.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:51.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929940+0000", 2023-07-02T14:05:51.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9740000000000001e-05 2023-07-02T14:05:51.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:51.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929960+0000", 2023-07-02T14:05:51.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.311e-05 2023-07-02T14:05:51.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:51.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929973+0000", 2023-07-02T14:05:51.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9629000000000002e-05 2023-07-02T14:05:51.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:51.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930023+0000", 2023-07-02T14:05:51.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012751799999999999 2023-07-02T14:05:51.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:51.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930150+0000", 2023-07-02T14:05:51.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016449042000000001 2023-07-02T14:05:51.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:51.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946599+0000", 2023-07-02T14:05:51.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3241e-05 2023-07-02T14:05:51.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:51.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946663+0000", 2023-07-02T14:05:51.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000102296 2023-07-02T14:05:51.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:51.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946765+0000", 2023-07-02T14:05:51.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919 2023-07-02T14:05:51.573 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.573 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.573 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.574 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11782 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 196608~2162688] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:05:51.574 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:51.574 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 67.019807745999998, 2023-07-02T14:05:51.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365, 2023-07-02T14:05:51.574 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:51.574 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:51.574 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:51.575 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:51.575 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:51.575 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11782 2023-07-02T14:05:51.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.575 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:51.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:51.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:51.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:51.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:51.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:51.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:51.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:51.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409506+0000", 2023-07-02T14:05:51.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020172269999999999 2023-07-02T14:05:51.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:51.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411523+0000", 2023-07-02T14:05:51.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8901000000000001e-05 2023-07-02T14:05:51.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:51.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411542+0000", 2023-07-02T14:05:51.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5037e-05 2023-07-02T14:05:51.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:51.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411557+0000", 2023-07-02T14:05:51.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001931213 2023-07-02T14:05:51.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:51.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413489+0000", 2023-07-02T14:05:51.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000149207 2023-07-02T14:05:51.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:51.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413638+0000", 2023-07-02T14:05:51.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024752800000000001 2023-07-02T14:05:51.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:51.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413885+0000", 2023-07-02T14:05:51.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048516000000000002 2023-07-02T14:05:51.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:51.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.418737+0000", 2023-07-02T14:05:51.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010319259000000001 2023-07-02T14:05:51.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:51.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429056+0000", 2023-07-02T14:05:51.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2518000000000002e-05 2023-07-02T14:05:51.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:51.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429099+0000", 2023-07-02T14:05:51.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015275 2023-07-02T14:05:51.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:51.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429251+0000", 2023-07-02T14:05:51.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365 2023-07-02T14:05:51.583 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.583 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.584 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.584 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPushReply(2.7 119/117 [PushReplyOp(2:eb862bc0:::rbd_data.125de1449605.0000000000000040:13)])", 2023-07-02T14:05:51.584 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:51.584 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 67.018644471000002, 2023-07-02T14:05:51.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999, 2023-07-02T14:05:51.584 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:51.585 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "reached pg", 2023-07-02T14:05:51.585 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:51.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:51.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:51.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:51.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:51.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:51.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:51.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:51.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:51.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7670000000000001e-06 2023-07-02T14:05:51.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:51.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410674+0000", 2023-07-02T14:05:51.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3973e-05 2023-07-02T14:05:51.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:51.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410688+0000", 2023-07-02T14:05:51.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0157e-05 2023-07-02T14:05:51.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:51.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410708+0000", 2023-07-02T14:05:51.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018578779 2023-07-02T14:05:51.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:51.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429287+0000", 2023-07-02T14:05:51.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013086549999999999 2023-07-02T14:05:51.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:51.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.430595+0000", 2023-07-02T14:05:51.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999 2023-07-02T14:05:51.590 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.590 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.590 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.591 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11861 2.7 2:e57d3f55:::rbd_data.125de1449605.0000000000000134:head [write 0~65536] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:05:51.591 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:51.591 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 66.884529094000001, 2023-07-02T14:05:51.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723, 2023-07-02T14:05:51.591 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:51.591 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:51.591 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:51.591 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:51.592 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:51.592 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11861 2023-07-02T14:05:51.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.592 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:51.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:51.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:51.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:51.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:51.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:51.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:51.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:51.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544785+0000", 2023-07-02T14:05:51.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010563700000000001 2023-07-02T14:05:51.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:51.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544890+0000", 2023-07-02T14:05:51.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8530000000000004e-06 2023-07-02T14:05:51.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:51.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544900+0000", 2023-07-02T14:05:51.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.665e-06 2023-07-02T14:05:51.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:51.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544906+0000", 2023-07-02T14:05:51.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000272575 2023-07-02T14:05:51.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:51.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545178+0000", 2023-07-02T14:05:51.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000111362 2023-07-02T14:05:51.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:51.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545290+0000", 2023-07-02T14:05:51.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016181100000000001 2023-07-02T14:05:51.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:51.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545452+0000", 2023-07-02T14:05:51.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044500750000000004 2023-07-02T14:05:51.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:51.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.549902+0000", 2023-07-02T14:05:51.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014850024 2023-07-02T14:05:51.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:51.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564752+0000", 2023-07-02T14:05:51.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6272000000000003e-05 2023-07-02T14:05:51.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:51.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564848+0000", 2023-07-02T14:05:51.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000162428 2023-07-02T14:05:51.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:51.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.565010+0000", 2023-07-02T14:05:51.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723 2023-07-02T14:05:51.600 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.600 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.601 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.601 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:13262 2.7 e125/121 2:e0741dde:::rbd_data.125de1449605.00000000000000ad:head v 125'2024, mlcod=125'2021)", 2023-07-02T14:05:51.601 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:51.601 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 58.178451355999997, 2023-07-02T14:05:51.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999, 2023-07-02T14:05:51.602 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:51.602 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:51.602 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:51.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:51.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:51.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:51.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:51.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:51.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:51.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:51.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250862+0000", 2023-07-02T14:05:51.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080378800000000007 2023-07-02T14:05:51.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.604 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:51.604 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258900+0000", 2023-07-02T14:05:51.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8939999999999999e-06 2023-07-02T14:05:51.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:51.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258909+0000", 2023-07-02T14:05:51.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3560000000000003e-06 2023-07-02T14:05:51.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:51.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258914+0000", 2023-07-02T14:05:51.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2429000000000003e-05 2023-07-02T14:05:51.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:51.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258956+0000", 2023-07-02T14:05:51.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5094999999999999e-05 2023-07-02T14:05:51.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:51.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258981+0000", 2023-07-02T14:05:51.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012177307 2023-07-02T14:05:51.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:51.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271158+0000", 2023-07-02T14:05:51.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2104999999999999e-05 2023-07-02T14:05:51.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:51.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:51.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:51.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271181+0000", 2023-07-02T14:05:51.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999 2023-07-02T14:05:51.609 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.609 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.609 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.609 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:51.609 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:51.610 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:51.610 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:51.611 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:05:51.611 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 4 2023-07-02T14:05:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-07-02T14:05:51.612 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:05:51.613 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:51.611+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:Error ENXIO: problem getting command descriptions from osd.0 2023-07-02T14:05:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:05:51.613 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:05:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:05:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:05:51.614 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:05:51.614 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-07-02T14:05:51.712 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:51.712+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:51.777 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:51.778 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:51.778 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:51.778 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:51.788 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:05:51.813 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:51.813+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:51.914 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:51.914+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:51.951 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:51.951 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:51.951 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:51.952 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:51.952 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:51.961 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:05:52.045 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:52.045+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:52.116 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:52.116+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:52.126 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:52.127 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:52.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:52.127 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:52.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.127 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:52.127 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:52.128 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.834327503, 2023-07-02T14:05:52.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:05:52.128 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.128 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.128 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.128 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.128 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.129 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:05:52.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.129 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:52.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:52.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:52.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:05:52.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:05:52.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:05:52.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:05:52.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:05:52.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:05:52.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:05:52.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:05:52.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:05:52.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:05:52.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:05:52.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:05:52.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:52.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:05:52.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:05:52.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:05:52.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:05:52.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:05:52.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:05:52.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:05:52.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:05:52.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:05:52.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:05:52.137 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.138 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.138 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.138 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:52.138 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:52.138 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.83409248000001, 2023-07-02T14:05:52.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:05:52.139 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.139 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.139 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.139 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.139 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.139 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:05:52.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.140 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:52.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:52.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:52.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:05:52.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:05:52.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:05:52.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:05:52.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:05:52.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:05:52.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:05:52.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:05:52.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:05:52.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:05:52.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:05:52.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:05:52.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:52.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:05:52.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:05:52.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:05:52.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:05:52.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:05:52.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:05:52.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:05:52.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:05:52.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:05:52.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:05:52.148 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.148 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.148 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.149 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:52.149 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:52.149 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.833893346, 2023-07-02T14:05:52.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:05:52.149 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.149 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.149 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.150 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.150 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.150 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:05:52.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.150 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:52.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:52.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:52.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:05:52.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:05:52.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:05:52.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:05:52.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:05:52.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:05:52.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:05:52.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:05:52.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:05:52.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:05:52.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:05:52.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:05:52.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:52.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:05:52.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:05:52.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:05:52.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:05:52.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:05:52.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:05:52.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:05:52.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:05:52.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:05:52.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:05:52.159 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.159 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.159 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.160 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:52.160 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:52.160 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.83352822200001, 2023-07-02T14:05:52.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:05:52.160 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.161 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.161 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.161 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.161 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.161 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:05:52.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.161 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:52.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:52.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:52.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:05:52.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:05:52.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:05:52.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:05:52.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:05:52.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:05:52.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:05:52.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:05:52.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:05:52.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:05:52.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:05:52.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:05:52.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:52.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:05:52.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:05:52.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:05:52.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:05:52.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:05:52.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:05:52.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:05:52.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:05:52.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:05:52.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:05:52.170 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.170 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.170 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.170 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:52.171 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:52.171 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.83344716100001, 2023-07-02T14:05:52.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:05:52.171 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.171 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.171 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.171 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.171 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.172 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:05:52.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.172 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:52.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:52.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:52.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:05:52.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:05:52.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:05:52.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:05:52.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:05:52.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:05:52.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:05:52.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:05:52.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:05:52.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:05:52.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:05:52.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:05:52.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:52.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:05:52.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:05:52.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:05:52.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:05:52.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:05:52.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:05:52.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:05:52.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:05:52.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:05:52.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:05:52.181 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.181 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.181 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.181 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:52.181 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:52.181 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.83335549, 2023-07-02T14:05:52.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:05:52.182 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.182 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.182 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.182 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.182 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.182 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:05:52.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.183 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:52.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:52.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:52.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:05:52.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:05:52.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:05:52.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:05:52.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:05:52.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:05:52.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:05:52.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:05:52.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:05:52.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:05:52.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:05:52.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:05:52.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:52.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:05:52.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:05:52.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:05:52.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:05:52.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:05:52.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:05:52.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:05:52.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:05:52.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:05:52.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:05:52.191 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.191 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.191 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.192 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:52.192 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:52.192 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.83301604799999, 2023-07-02T14:05:52.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:05:52.192 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.192 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.192 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.193 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.193 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.193 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:05:52.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.193 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:52.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:52.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:52.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:05:52.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:05:52.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:05:52.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:05:52.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:05:52.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:05:52.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:05:52.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:05:52.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:05:52.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:05:52.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:05:52.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:05:52.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:52.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:05:52.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:05:52.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.199 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:05:52.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:05:52.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:05:52.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:05:52.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:05:52.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:05:52.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:05:52.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:05:52.201 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.202 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.202 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.202 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:52.202 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:52.202 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.832829995, 2023-07-02T14:05:52.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:05:52.203 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.203 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.203 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.203 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.203 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.203 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:05:52.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.203 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:52.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:52.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:52.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:52.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:05:52.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:05:52.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:05:52.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:05:52.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:05:52.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:05:52.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:05:52.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:05:52.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:05:52.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:05:52.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:05:52.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:52.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:05:52.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:05:52.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:05:52.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:05:52.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:05:52.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:05:52.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:05:52.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:05:52.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:05:52.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:05:52.212 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.212 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.212 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.213 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:52.213 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:52.213 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.83267219300001, 2023-07-02T14:05:52.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:05:52.213 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.213 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.214 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.214 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.214 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.214 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:05:52.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.214 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:52.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:52.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:52.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:05:52.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:05:52.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:05:52.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:05:52.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:05:52.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:05:52.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:05:52.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:05:52.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:05:52.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:05:52.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:05:52.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:05:52.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:52.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:05:52.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:05:52.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:05:52.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:05:52.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.221 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:05:52.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:05:52.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:05:52.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:05:52.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:05:52.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:05:52.223 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.223 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.223 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.223 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:52.223 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:52.223 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.83248041100001, 2023-07-02T14:05:52.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:05:52.224 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.224 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.224 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.224 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.224 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.224 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:05:52.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.225 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:52.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:52.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:52.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:05:52.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:05:52.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:05:52.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:05:52.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:05:52.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:05:52.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:05:52.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:05:52.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:05:52.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:05:52.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.230 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:05:52.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:05:52.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.230 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:52.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:05:52.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:05:52.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.231 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:05:52.231 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:05:52.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:05:52.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:05:52.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:05:52.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:05:52.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:05:52.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:05:52.233 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.233 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.233 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.234 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:52.234 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:52.234 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.83230475100001, 2023-07-02T14:05:52.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:05:52.234 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.234 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.235 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.235 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.235 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.235 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:05:52.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.235 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:52.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:52.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:52.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:52.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:05:52.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:05:52.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:05:52.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:05:52.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:05:52.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:05:52.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:05:52.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.240 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:05:52.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:05:52.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.240 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:05:52.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:05:52.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:52.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:05:52.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:05:52.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:05:52.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:05:52.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:05:52.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:05:52.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:05:52.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:05:52.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:05:52.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:05:52.244 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.244 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.244 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.245 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:52.245 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:52.245 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.83221056299999, 2023-07-02T14:05:52.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:05:52.245 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.245 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.245 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.246 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.246 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.246 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:05:52.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.246 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.246 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.246 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:52.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:52.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:52.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:05:52.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:05:52.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:05:52.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:05:52.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:05:52.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:05:52.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:05:52.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:05:52.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:05:52.251 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:05:52.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.251 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.251 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:05:52.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:05:52.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.252 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:52.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:05:52.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:05:52.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.253 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:05:52.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:05:52.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:05:52.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:05:52.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:05:52.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:05:52.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:05:52.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:05:52.255 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.255 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.256 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.256 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:52.256 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:52.256 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.83200717700001, 2023-07-02T14:05:52.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:05:52.257 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.257 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.257 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.257 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.257 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.257 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:05:52.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.257 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:52.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:52.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:52.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:05:52.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:05:52.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:05:52.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:05:52.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:05:52.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:05:52.261 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.261 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.261 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:05:52.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:05:52.261 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.262 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.262 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:05:52.262 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:05:52.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.262 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.262 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.262 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:05:52.263 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:05:52.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:52.263 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:05:52.263 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:05:52.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.264 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:05:52.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:05:52.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:05:52.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:05:52.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:05:52.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:05:52.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.268 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:05:52.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:05:52.269 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.269 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.269 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.270 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:52.270 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:52.270 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.83185652099999, 2023-07-02T14:05:52.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:05:52.271 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.271 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.271 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.272 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.272 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.272 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:05:52.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.273 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.273 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:52.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.275 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.275 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:52.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:52.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:05:52.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:05:52.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.277 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.277 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:05:52.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:05:52.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.279 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:05:52.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:05:52.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:05:52.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:05:52.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:05:52.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:05:52.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.282 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:05:52.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:05:52.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.282 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:52.283 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:05:52.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:05:52.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.283 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:05:52.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:05:52.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:05:52.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:05:52.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:05:52.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:05:52.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:05:52.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:05:52.285 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.286 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.286 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.286 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:52.286 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:52.286 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.83178381799999, 2023-07-02T14:05:52.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:05:52.287 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.287 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.287 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.287 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.287 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.287 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:05:52.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.288 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:52.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:52.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:52.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:52.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:05:52.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:05:52.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:05:52.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:05:52.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:05:52.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:05:52.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:05:52.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:05:52.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:05:52.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:05:52.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:05:52.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:52.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:05:52.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:05:52.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.294 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:05:52.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:05:52.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.294 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:05:52.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:05:52.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.295 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:05:52.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:05:52.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:05:52.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:05:52.297 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.297 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.297 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.298 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:52.298 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:52.298 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.83166682800001, 2023-07-02T14:05:52.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:05:52.298 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.298 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.298 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.299 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.299 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.299 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:05:52.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.299 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:52.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:52.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:52.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:52.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:05:52.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:05:52.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:05:52.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:05:52.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:05:52.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:05:52.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:05:52.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:05:52.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:05:52.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:05:52.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:05:52.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:52.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:05:52.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:05:52.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:05:52.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:05:52.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:05:52.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:05:52.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.307 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:05:52.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:05:52.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.307 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:05:52.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:05:52.308 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.308 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.308 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.308 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:52.309 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:52.309 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.83151737700001, 2023-07-02T14:05:52.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:05:52.309 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.309 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.309 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.309 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.309 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.310 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:05:52.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.310 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:52.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:52.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:52.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:05:52.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:05:52.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:05:52.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:05:52.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:05:52.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:05:52.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:05:52.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:05:52.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:05:52.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:05:52.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:05:52.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:05:52.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:52.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:05:52.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:05:52.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:05:52.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:05:52.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:05:52.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:05:52.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:05:52.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:05:52.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:05:52.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:05:52.319 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.319 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.319 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.319 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:52.320 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:52.320 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.810041404, 2023-07-02T14:05:52.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:05:52.320 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.320 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.320 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.320 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.321 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.321 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:05:52.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.321 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:52.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.322 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:52.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:52.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:05:52.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:05:52.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:05:52.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:05:52.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.324 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:05:52.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:05:52.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:05:52.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:05:52.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:05:52.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:05:52.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:05:52.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:05:52.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:52.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:05:52.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:05:52.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:05:52.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:05:52.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:05:52.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:05:52.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:05:52.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:05:52.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:05:52.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:05:52.329 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.329 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.330 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.330 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:52.330 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:52.330 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.80990886800001, 2023-07-02T14:05:52.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:05:52.331 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.331 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.331 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.331 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.331 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.331 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:05:52.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.331 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:52.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:52.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:52.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:05:52.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:05:52.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:05:52.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:05:52.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:05:52.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:05:52.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:05:52.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:05:52.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:05:52.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:05:52.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:05:52.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:05:52.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:52.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:05:52.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:05:52.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:05:52.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:05:52.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:05:52.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:05:52.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:05:52.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:05:52.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:05:52.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:05:52.340 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.340 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.340 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.340 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:52.341 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:52.341 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.748572207, 2023-07-02T14:05:52.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:05:52.341 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.341 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.341 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.341 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.341 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.342 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:05:52.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.342 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:52.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:52.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:52.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:05:52.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:05:52.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:05:52.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:05:52.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:05:52.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:05:52.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:05:52.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:05:52.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:05:52.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:05:52.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:05:52.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:05:52.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:52.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:05:52.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:05:52.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:05:52.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:05:52.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:05:52.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:05:52.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:05:52.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:05:52.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:05:52.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:05:52.351 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.351 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.351 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.351 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.351 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.352 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:52.353 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:05:52.354 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:52.353+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:52.454 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:52.454+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:52.520 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:52.520 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:52.521 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:52.521 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:52.530 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:05:52.572 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:52.572+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:52.656 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:52.655+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:52.694 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:52.694 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:52.694 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:52.694 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:52.695 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:52.704 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:05:52.757 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:52.757+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:52.832 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:52.833 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:52.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:52.833 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:52.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.833 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:16940 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:52.833 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.855862+0000", 2023-07-02T14:05:52.833 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.9720266590000008, 2023-07-02T14:05:52.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.103600949, 2023-07-02T14:05:52.834 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.834 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.834 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.834 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.834 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.834 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 16940 2023-07-02T14:05:52.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.835 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855862+0000", 2023-07-02T14:05:52.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855862+0000", 2023-07-02T14:05:52.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:52.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855862+0000", 2023-07-02T14:05:52.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.353e-06 2023-07-02T14:05:52.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855868+0000", 2023-07-02T14:05:52.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.746e-06 2023-07-02T14:05:52.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855873+0000", 2023-07-02T14:05:52.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4030000000000001e-06 2023-07-02T14:05:52.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855875+0000", 2023-07-02T14:05:52.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050000019999999999 2023-07-02T14:05:52.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.860875+0000", 2023-07-02T14:05:52.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3556000000000001e-05 2023-07-02T14:05:52.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.860919+0000", 2023-07-02T14:05:52.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055944549999999999 2023-07-02T14:05:52.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.866513+0000", 2023-07-02T14:05:52.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2478999999999999e-05 2023-07-02T14:05:52.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.866546+0000", 2023-07-02T14:05:52.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050944609999999998 2023-07-02T14:05:52.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.871640+0000", 2023-07-02T14:05:52.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2138000000000003e-05 2023-07-02T14:05:52.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.871672+0000", 2023-07-02T14:05:52.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014725571999999999 2023-07-02T14:05:52.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.886398+0000", 2023-07-02T14:05:52.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6910999999999999e-05 2023-07-02T14:05:52.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.886435+0000", 2023-07-02T14:05:52.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015334054 2023-07-02T14:05:52.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.901769+0000", 2023-07-02T14:05:52.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0036000000000001e-05 2023-07-02T14:05:52.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.901809+0000", 2023-07-02T14:05:52.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014746522999999999 2023-07-02T14:05:52.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.916555+0000", 2023-07-02T14:05:52.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3622999999999999e-05 2023-07-02T14:05:52.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.916599+0000", 2023-07-02T14:05:52.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099546679999999999 2023-07-02T14:05:52.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.926554+0000", 2023-07-02T14:05:52.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7073000000000002e-05 2023-07-02T14:05:52.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.926591+0000", 2023-07-02T14:05:52.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045803249999999997 2023-07-02T14:05:52.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.931171+0000", 2023-07-02T14:05:52.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9468000000000004e-05 2023-07-02T14:05:52.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.931271+0000", 2023-07-02T14:05:52.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078347380000000008 2023-07-02T14:05:52.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.939105+0000", 2023-07-02T14:05:52.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1427999999999994e-05 2023-07-02T14:05:52.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.939177+0000", 2023-07-02T14:05:52.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093866379999999992 2023-07-02T14:05:52.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.948563+0000", 2023-07-02T14:05:52.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3999999999999995e-05 2023-07-02T14:05:52.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.948647+0000", 2023-07-02T14:05:52.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00034591999999999999 2023-07-02T14:05:52.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:52.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.948993+0000", 2023-07-02T14:05:52.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056008810000000003 2023-07-02T14:05:52.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.954594+0000", 2023-07-02T14:05:52.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047713010000000004 2023-07-02T14:05:52.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959365+0000", 2023-07-02T14:05:52.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4107999999999998e-05 2023-07-02T14:05:52.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959410+0000", 2023-07-02T14:05:52.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3331999999999997e-05 2023-07-02T14:05:52.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959463+0000", 2023-07-02T14:05:52.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.103600949 2023-07-02T14:05:52.855 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.855 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.855 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.856 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:16941 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:52.856 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:05:52.856 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.9720110139999996, 2023-07-02T14:05:52.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11449124500000001, 2023-07-02T14:05:52.856 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.857 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.857 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.857 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.857 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.857 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 16941 2023-07-02T14:05:52.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.858 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:05:52.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:05:52.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:05:52.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855877+0000", 2023-07-02T14:05:52.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3209999999999999e-06 2023-07-02T14:05:52.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855880+0000", 2023-07-02T14:05:52.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9299999999999999e-06 2023-07-02T14:05:52.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855883+0000", 2023-07-02T14:05:52.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6309999999999999e-06 2023-07-02T14:05:52.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855885+0000", 2023-07-02T14:05:52.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050516850000000002 2023-07-02T14:05:52.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.860937+0000", 2023-07-02T14:05:52.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1844999999999997e-05 2023-07-02T14:05:52.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.860979+0000", 2023-07-02T14:05:52.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055853480000000004 2023-07-02T14:05:52.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.866564+0000", 2023-07-02T14:05:52.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2821999999999998e-05 2023-07-02T14:05:52.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.866597+0000", 2023-07-02T14:05:52.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050938900000000002 2023-07-02T14:05:52.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.871691+0000", 2023-07-02T14:05:52.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1210000000000001e-05 2023-07-02T14:05:52.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.871722+0000", 2023-07-02T14:05:52.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014768687000000001 2023-07-02T14:05:52.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.886491+0000", 2023-07-02T14:05:52.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7248000000000003e-05 2023-07-02T14:05:52.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.886568+0000", 2023-07-02T14:05:52.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015403168 2023-07-02T14:05:52.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.901971+0000", 2023-07-02T14:05:52.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4035000000000002e-05 2023-07-02T14:05:52.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.902005+0000", 2023-07-02T14:05:52.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014618580000000001 2023-07-02T14:05:52.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.916624+0000", 2023-07-02T14:05:52.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.443e-05 2023-07-02T14:05:52.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.916668+0000", 2023-07-02T14:05:52.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099598989999999995 2023-07-02T14:05:52.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.926628+0000", 2023-07-02T14:05:52.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3123999999999999e-05 2023-07-02T14:05:52.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.926711+0000", 2023-07-02T14:05:52.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046031930000000002 2023-07-02T14:05:52.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.931314+0000", 2023-07-02T14:05:52.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3257000000000003e-05 2023-07-02T14:05:52.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.931358+0000", 2023-07-02T14:05:52.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078370969999999995 2023-07-02T14:05:52.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.939195+0000", 2023-07-02T14:05:52.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2522000000000003e-05 2023-07-02T14:05:52.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.939247+0000", 2023-07-02T14:05:52.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01118103 2023-07-02T14:05:52.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.950428+0000", 2023-07-02T14:05:52.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5001999999999996e-05 2023-07-02T14:05:52.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.950523+0000", 2023-07-02T14:05:52.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089648390000000005 2023-07-02T14:05:52.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959488+0000", 2023-07-02T14:05:52.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4569e-05 2023-07-02T14:05:52.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959533+0000", 2023-07-02T14:05:52.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026104299999999997 2023-07-02T14:05:52.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:52.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959794+0000", 2023-07-02T14:05:52.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025221789999999998 2023-07-02T14:05:52.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.962316+0000", 2023-07-02T14:05:52.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079080520000000005 2023-07-02T14:05:52.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970224+0000", 2023-07-02T14:05:52.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1265e-05 2023-07-02T14:05:52.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970255+0000", 2023-07-02T14:05:52.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000113581 2023-07-02T14:05:52.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970369+0000", 2023-07-02T14:05:52.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11449124500000001 2023-07-02T14:05:52.879 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.880 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.880 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.880 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17047 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:52.880 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.979346+0000", 2023-07-02T14:05:52.880 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.8485426080000007, 2023-07-02T14:05:52.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.095586451000000003, 2023-07-02T14:05:52.881 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.881 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.881 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.881 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.881 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.881 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17047 2023-07-02T14:05:52.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.881 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979346+0000", 2023-07-02T14:05:52.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979346+0000", 2023-07-02T14:05:52.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:52.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979345+0000", 2023-07-02T14:05:52.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.775e-06 2023-07-02T14:05:52.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979350+0000", 2023-07-02T14:05:52.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8170000000000001e-06 2023-07-02T14:05:52.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979357+0000", 2023-07-02T14:05:52.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9539999999999998e-06 2023-07-02T14:05:52.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979361+0000", 2023-07-02T14:05:52.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01301737 2023-07-02T14:05:52.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992378+0000", 2023-07-02T14:05:52.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4696999999999997e-05 2023-07-02T14:05:52.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992413+0000", 2023-07-02T14:05:52.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061779579999999999 2023-07-02T14:05:52.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998591+0000", 2023-07-02T14:05:52.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3654999999999999e-05 2023-07-02T14:05:52.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998664+0000", 2023-07-02T14:05:52.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091576609999999992 2023-07-02T14:05:52.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.007822+0000", 2023-07-02T14:05:52.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7914000000000002e-05 2023-07-02T14:05:52.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.007870+0000", 2023-07-02T14:05:52.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059512150000000002 2023-07-02T14:05:52.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.013821+0000", 2023-07-02T14:05:52.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1442999999999999e-05 2023-07-02T14:05:52.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.013902+0000", 2023-07-02T14:05:52.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075954430000000003 2023-07-02T14:05:52.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021498+0000", 2023-07-02T14:05:52.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0065000000000001e-05 2023-07-02T14:05:52.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021528+0000", 2023-07-02T14:05:52.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056511080000000002 2023-07-02T14:05:52.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.893 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027179+0000", 2023-07-02T14:05:52.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8032999999999998e-05 2023-07-02T14:05:52.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.893 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.893 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027237+0000", 2023-07-02T14:05:52.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048459089999999998 2023-07-02T14:05:52.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032083+0000", 2023-07-02T14:05:52.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1186000000000003e-05 2023-07-02T14:05:52.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032114+0000", 2023-07-02T14:05:52.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055940670000000003 2023-07-02T14:05:52.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.037708+0000", 2023-07-02T14:05:52.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0913000000000003e-05 2023-07-02T14:05:52.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.037789+0000", 2023-07-02T14:05:52.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043937489999999997 2023-07-02T14:05:52.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042183+0000", 2023-07-02T14:05:52.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0321000000000007e-05 2023-07-02T14:05:52.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042253+0000", 2023-07-02T14:05:52.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073046439999999999 2023-07-02T14:05:52.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049558+0000", 2023-07-02T14:05:52.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0810000000000006e-05 2023-07-02T14:05:52.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049649+0000", 2023-07-02T14:05:52.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0094162250000000003 2023-07-02T14:05:52.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059065+0000", 2023-07-02T14:05:52.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2486999999999997e-05 2023-07-02T14:05:52.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059157+0000", 2023-07-02T14:05:52.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059730709999999999 2023-07-02T14:05:52.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.065131+0000", 2023-07-02T14:05:52.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9991999999999998e-05 2023-07-02T14:05:52.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.065180+0000", 2023-07-02T14:05:52.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00037239199999999998 2023-07-02T14:05:52.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:52.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.065553+0000", 2023-07-02T14:05:52.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037123600000000001 2023-07-02T14:05:52.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.069265+0000", 2023-07-02T14:05:52.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055193860000000003 2023-07-02T14:05:52.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.074785+0000", 2023-07-02T14:05:52.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9505e-05 2023-07-02T14:05:52.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.074814+0000", 2023-07-02T14:05:52.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000118282 2023-07-02T14:05:52.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.074932+0000", 2023-07-02T14:05:52.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.095586451000000003 2023-07-02T14:05:52.905 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.905 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.905 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.905 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17048 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:52.905 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.979365+0000", 2023-07-02T14:05:52.905 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.8485233870000002, 2023-07-02T14:05:52.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.102723355, 2023-07-02T14:05:52.906 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.906 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.906 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.906 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.906 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.906 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17048 2023-07-02T14:05:52.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.907 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979365+0000", 2023-07-02T14:05:52.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979365+0000", 2023-07-02T14:05:52.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:52.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979365+0000", 2023-07-02T14:05:52.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4489999999999999e-06 2023-07-02T14:05:52.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979368+0000", 2023-07-02T14:05:52.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.178e-06 2023-07-02T14:05:52.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979374+0000", 2023-07-02T14:05:52.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0029999999999996e-06 2023-07-02T14:05:52.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979378+0000", 2023-07-02T14:05:52.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013053530000000001 2023-07-02T14:05:52.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992432+0000", 2023-07-02T14:05:52.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4631000000000001e-05 2023-07-02T14:05:52.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992467+0000", 2023-07-02T14:05:52.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062315440000000003 2023-07-02T14:05:52.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998698+0000", 2023-07-02T14:05:52.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2839999999999997e-05 2023-07-02T14:05:52.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998731+0000", 2023-07-02T14:05:52.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091807750000000004 2023-07-02T14:05:52.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.007912+0000", 2023-07-02T14:05:52.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0945999999999995e-05 2023-07-02T14:05:52.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.007983+0000", 2023-07-02T14:05:52.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059545220000000003 2023-07-02T14:05:52.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.013937+0000", 2023-07-02T14:05:52.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2709000000000001e-05 2023-07-02T14:05:52.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.013970+0000", 2023-07-02T14:05:52.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075831359999999999 2023-07-02T14:05:52.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021553+0000", 2023-07-02T14:05:52.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0125e-05 2023-07-02T14:05:52.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021583+0000", 2023-07-02T14:05:52.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056843680000000004 2023-07-02T14:05:52.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027268+0000", 2023-07-02T14:05:52.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0429e-05 2023-07-02T14:05:52.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027298+0000", 2023-07-02T14:05:52.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048443660000000001 2023-07-02T14:05:52.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032142+0000", 2023-07-02T14:05:52.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1180000000000003e-05 2023-07-02T14:05:52.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032174+0000", 2023-07-02T14:05:52.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005665944 2023-07-02T14:05:52.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.037839+0000", 2023-07-02T14:05:52.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6186000000000001e-05 2023-07-02T14:05:52.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.037886+0000", 2023-07-02T14:05:52.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044234330000000001 2023-07-02T14:05:52.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.922 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042309+0000", 2023-07-02T14:05:52.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6038999999999997e-05 2023-07-02T14:05:52.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042355+0000", 2023-07-02T14:05:52.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073303919999999998 2023-07-02T14:05:52.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049686+0000", 2023-07-02T14:05:52.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5858000000000001e-05 2023-07-02T14:05:52.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049721+0000", 2023-07-02T14:05:52.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0094923960000000002 2023-07-02T14:05:52.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059214+0000", 2023-07-02T14:05:52.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3883999999999998e-05 2023-07-02T14:05:52.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059298+0000", 2023-07-02T14:05:52.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080288240000000004 2023-07-02T14:05:52.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067326+0000", 2023-07-02T14:05:52.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0906999999999998e-05 2023-07-02T14:05:52.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067387+0000", 2023-07-02T14:05:52.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075801130000000003 2023-07-02T14:05:52.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.074968+0000", 2023-07-02T14:05:52.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1454e-05 2023-07-02T14:05:52.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.075009+0000", 2023-07-02T14:05:52.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025346800000000002 2023-07-02T14:05:52.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:52.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.075262+0000", 2023-07-02T14:05:52.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020188070000000001 2023-07-02T14:05:52.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077281+0000", 2023-07-02T14:05:52.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046643250000000004 2023-07-02T14:05:52.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.081946+0000", 2023-07-02T14:05:52.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.7495999999999999e-05 2023-07-02T14:05:52.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.081973+0000", 2023-07-02T14:05:52.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115494 2023-07-02T14:05:52.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082089+0000", 2023-07-02T14:05:52.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.102723355 2023-07-02T14:05:52.931 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.931 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.931 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.932 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17049 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:52.932 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:05:52.932 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.8485060230000006, 2023-07-02T14:05:52.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.112277108, 2023-07-02T14:05:52.933 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.933 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.933 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.933 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.933 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.934 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17049 2023-07-02T14:05:52.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.934 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:05:52.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:05:52.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:52.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979382+0000", 2023-07-02T14:05:52.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3450000000000002e-06 2023-07-02T14:05:52.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979385+0000", 2023-07-02T14:05:52.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9000000000000003e-06 2023-07-02T14:05:52.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979391+0000", 2023-07-02T14:05:52.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9999999999999998e-06 2023-07-02T14:05:52.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979395+0000", 2023-07-02T14:05:52.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013089412 2023-07-02T14:05:52.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992485+0000", 2023-07-02T14:05:52.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3828999999999997e-05 2023-07-02T14:05:52.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992519+0000", 2023-07-02T14:05:52.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062549889999999999 2023-07-02T14:05:52.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998774+0000", 2023-07-02T14:05:52.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4870999999999995e-05 2023-07-02T14:05:52.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998848+0000", 2023-07-02T14:05:52.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091715140000000004 2023-07-02T14:05:52.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008020+0000", 2023-07-02T14:05:52.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8659999999999999e-05 2023-07-02T14:05:52.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008059+0000", 2023-07-02T14:05:52.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059565110000000003 2023-07-02T14:05:52.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014015+0000", 2023-07-02T14:05:52.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8088000000000002e-05 2023-07-02T14:05:52.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014093+0000", 2023-07-02T14:05:52.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007513395 2023-07-02T14:05:52.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021607+0000", 2023-07-02T14:05:52.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9777e-05 2023-07-02T14:05:52.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021636+0000", 2023-07-02T14:05:52.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056999549999999996 2023-07-02T14:05:52.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027336+0000", 2023-07-02T14:05:52.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8356999999999996e-05 2023-07-02T14:05:52.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027405+0000", 2023-07-02T14:05:52.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047942130000000003 2023-07-02T14:05:52.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032199+0000", 2023-07-02T14:05:52.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1137999999999999e-05 2023-07-02T14:05:52.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032230+0000", 2023-07-02T14:05:52.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057056759999999998 2023-07-02T14:05:52.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.037936+0000", 2023-07-02T14:05:52.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6334999999999993e-05 2023-07-02T14:05:52.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038012+0000", 2023-07-02T14:05:52.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043985609999999996 2023-07-02T14:05:52.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042411+0000", 2023-07-02T14:05:52.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6857e-05 2023-07-02T14:05:52.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042487+0000", 2023-07-02T14:05:52.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007284446 2023-07-02T14:05:52.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049772+0000", 2023-07-02T14:05:52.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3757000000000006e-05 2023-07-02T14:05:52.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049846+0000", 2023-07-02T14:05:52.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0095186679999999992 2023-07-02T14:05:52.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059364+0000", 2023-07-02T14:05:52.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9931000000000005e-05 2023-07-02T14:05:52.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059434+0000", 2023-07-02T14:05:52.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079904920000000001 2023-07-02T14:05:52.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067425+0000", 2023-07-02T14:05:52.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5583000000000001e-05 2023-07-02T14:05:52.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067460+0000", 2023-07-02T14:05:52.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009114278 2023-07-02T14:05:52.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076575+0000", 2023-07-02T14:05:52.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5880000000000001e-05 2023-07-02T14:05:52.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.954 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076620+0000", 2023-07-02T14:05:52.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055118470000000003 2023-07-02T14:05:52.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082132+0000", 2023-07-02T14:05:52.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8306999999999997e-05 2023-07-02T14:05:52.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082181+0000", 2023-07-02T14:05:52.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026259599999999998 2023-07-02T14:05:52.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:52.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082443+0000", 2023-07-02T14:05:52.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018855339999999999 2023-07-02T14:05:52.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084329+0000", 2023-07-02T14:05:52.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071877729999999997 2023-07-02T14:05:52.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091517+0000", 2023-07-02T14:05:52.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3424999999999999e-05 2023-07-02T14:05:52.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091550+0000", 2023-07-02T14:05:52.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109724 2023-07-02T14:05:52.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091660+0000", 2023-07-02T14:05:52.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.112277108 2023-07-02T14:05:52.960 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.960 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.960 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.961 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17050 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:52.961 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:05:52.961 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.8484890979999999, 2023-07-02T14:05:52.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12076701300000001, 2023-07-02T14:05:52.961 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.961 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.961 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.962 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.962 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.962 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17050 2023-07-02T14:05:52.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.962 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:05:52.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:05:52.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:52.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:05:52.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0759999999999999e-06 2023-07-02T14:05:52.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979402+0000", 2023-07-02T14:05:52.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0480000000000004e-06 2023-07-02T14:05:52.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979408+0000", 2023-07-02T14:05:52.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9559999999999999e-06 2023-07-02T14:05:52.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979412+0000", 2023-07-02T14:05:52.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013117658000000001 2023-07-02T14:05:52.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992530+0000", 2023-07-02T14:05:52.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4464000000000001e-05 2023-07-02T14:05:52.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992564+0000", 2023-07-02T14:05:52.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0063143640000000003 2023-07-02T14:05:52.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998879+0000", 2023-07-02T14:05:52.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2842000000000002e-05 2023-07-02T14:05:52.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998911+0000", 2023-07-02T14:05:52.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091863190000000001 2023-07-02T14:05:52.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008098+0000", 2023-07-02T14:05:52.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1448999999999999e-05 2023-07-02T14:05:52.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008179+0000", 2023-07-02T14:05:52.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059353720000000004 2023-07-02T14:05:52.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014115+0000", 2023-07-02T14:05:52.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1928000000000002e-05 2023-07-02T14:05:52.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014157+0000", 2023-07-02T14:05:52.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074935210000000004 2023-07-02T14:05:52.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021650+0000", 2023-07-02T14:05:52.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0199999999999999e-05 2023-07-02T14:05:52.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021680+0000", 2023-07-02T14:05:52.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057425109999999996 2023-07-02T14:05:52.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027423+0000", 2023-07-02T14:05:52.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1498999999999998e-05 2023-07-02T14:05:52.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027454+0000", 2023-07-02T14:05:52.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047903420000000004 2023-07-02T14:05:52.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032245+0000", 2023-07-02T14:05:52.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2552999999999998e-05 2023-07-02T14:05:52.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032277+0000", 2023-07-02T14:05:52.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057638079999999996 2023-07-02T14:05:52.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038041+0000", 2023-07-02T14:05:52.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9168000000000002e-05 2023-07-02T14:05:52.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038080+0000", 2023-07-02T14:05:52.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044291799999999996 2023-07-02T14:05:52.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042509+0000", 2023-07-02T14:05:52.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3528999999999997e-05 2023-07-02T14:05:52.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042543+0000", 2023-07-02T14:05:52.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073244729999999997 2023-07-02T14:05:52.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049867+0000", 2023-07-02T14:05:52.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2873999999999997e-05 2023-07-02T14:05:52.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049900+0000", 2023-07-02T14:05:52.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0095670149999999999 2023-07-02T14:05:52.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059467+0000", 2023-07-02T14:05:52.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0406000000000003e-05 2023-07-02T14:05:52.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059538+0000", 2023-07-02T14:05:52.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079635079999999994 2023-07-02T14:05:52.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067501+0000", 2023-07-02T14:05:52.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4795999999999997e-05 2023-07-02T14:05:52.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067536+0000", 2023-07-02T14:05:52.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091231969999999996 2023-07-02T14:05:52.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076659+0000", 2023-07-02T14:05:52.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3208999999999999e-05 2023-07-02T14:05:52.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076702+0000", 2023-07-02T14:05:52.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070660480000000001 2023-07-02T14:05:52.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.083768+0000", 2023-07-02T14:05:52.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5546000000000001e-05 2023-07-02T14:05:52.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.083814+0000", 2023-07-02T14:05:52.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078830659999999993 2023-07-02T14:05:52.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091697+0000", 2023-07-02T14:05:52.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5921000000000003e-05 2023-07-02T14:05:52.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:52.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091743+0000", 2023-07-02T14:05:52.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026571400000000001 2023-07-02T14:05:52.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:52.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.092009+0000", 2023-07-02T14:05:52.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022058429999999999 2023-07-02T14:05:52.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:52.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.094214+0000", 2023-07-02T14:05:52.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057115869999999997 2023-07-02T14:05:52.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:52.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.099926+0000", 2023-07-02T14:05:52.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0287999999999997e-05 2023-07-02T14:05:52.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:52.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.099976+0000", 2023-07-02T14:05:52.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019014999999999999 2023-07-02T14:05:52.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:52.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.100166+0000", 2023-07-02T14:05:52.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12076701300000001 2023-07-02T14:05:52.991 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.991 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:52.991 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:52.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.991 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17095 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:52.991 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.060121+0000", 2023-07-02T14:05:52.991 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.7677671410000002, 2023-07-02T14:05:52.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.099578634999999999, 2023-07-02T14:05:52.992 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:52.992 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:52.992 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:52.992 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:52.992 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:52.992 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17095 2023-07-02T14:05:52.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.993 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:52.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:52.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060121+0000", 2023-07-02T14:05:52.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:52.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:52.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060121+0000", 2023-07-02T14:05:52.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:52.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:52.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060121+0000", 2023-07-02T14:05:52.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5140000000000004e-06 2023-07-02T14:05:52.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:52.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060129+0000", 2023-07-02T14:05:52.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4520000000000002e-06 2023-07-02T14:05:52.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:52.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060135+0000", 2023-07-02T14:05:52.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2960000000000004e-06 2023-07-02T14:05:52.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:52.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060141+0000", 2023-07-02T14:05:52.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00047506399999999998 2023-07-02T14:05:52.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060616+0000", 2023-07-02T14:05:52.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8182999999999997e-05 2023-07-02T14:05:52.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060655+0000", 2023-07-02T14:05:52.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074016079999999996 2023-07-02T14:05:52.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:52.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068056+0000", 2023-07-02T14:05:52.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6019000000000005e-05 2023-07-02T14:05:52.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:52.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068132+0000", 2023-07-02T14:05:52.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090475429999999999 2023-07-02T14:05:52.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:52.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:52.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077180+0000", 2023-07-02T14:05:53.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7527999999999997e-05 2023-07-02T14:05:53.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077217+0000", 2023-07-02T14:05:53.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070671400000000004 2023-07-02T14:05:53.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084284+0000", 2023-07-02T14:05:53.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3818e-05 2023-07-02T14:05:53.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084318+0000", 2023-07-02T14:05:53.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093793799999999997 2023-07-02T14:05:53.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093698+0000", 2023-07-02T14:05:53.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2876999999999997e-05 2023-07-02T14:05:53.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093730+0000", 2023-07-02T14:05:53.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088179079999999993 2023-07-02T14:05:53.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102548+0000", 2023-07-02T14:05:53.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5802000000000003e-05 2023-07-02T14:05:53.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102634+0000", 2023-07-02T14:05:53.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099570119999999995 2023-07-02T14:05:53.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112591+0000", 2023-07-02T14:05:53.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6594000000000001e-05 2023-07-02T14:05:53.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112648+0000", 2023-07-02T14:05:53.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009671921 2023-07-02T14:05:53.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122320+0000", 2023-07-02T14:05:53.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2005000000000002e-05 2023-07-02T14:05:53.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122362+0000", 2023-07-02T14:05:53.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011261546000000001 2023-07-02T14:05:53.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133623+0000", 2023-07-02T14:05:53.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9981000000000006e-05 2023-07-02T14:05:53.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133693+0000", 2023-07-02T14:05:53.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093884069999999997 2023-07-02T14:05:53.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.143082+0000", 2023-07-02T14:05:53.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103371 2023-07-02T14:05:53.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.143185+0000", 2023-07-02T14:05:53.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051199679999999999 2023-07-02T14:05:53.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.148305+0000", 2023-07-02T14:05:53.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6837999999999998e-05 2023-07-02T14:05:53.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.148392+0000", 2023-07-02T14:05:53.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046966489999999998 2023-07-02T14:05:53.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.153088+0000", 2023-07-02T14:05:53.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5444000000000001e-05 2023-07-02T14:05:53.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:53.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.153134+0000", 2023-07-02T14:05:53.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000238259 2023-07-02T14:05:53.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:53.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.153372+0000", 2023-07-02T14:05:53.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023790500000000002 2023-07-02T14:05:53.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:53.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.155751+0000", 2023-07-02T14:05:53.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038028099999999998 2023-07-02T14:05:53.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:53.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.159554+0000", 2023-07-02T14:05:53.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3145000000000002e-05 2023-07-02T14:05:53.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:53.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.159587+0000", 2023-07-02T14:05:53.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000112948 2023-07-02T14:05:53.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:53.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.159700+0000", 2023-07-02T14:05:53.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.099578634999999999 2023-07-02T14:05:53.015 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.015 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:53.015 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.016 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17096 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:53.016 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:05:53.016 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.7677233220000002, 2023-07-02T14:05:53.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.10620154, 2023-07-02T14:05:53.016 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:53.017 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:53.017 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:53.017 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:53.017 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:53.017 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17096 2023-07-02T14:05:53.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.017 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:53.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:53.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:05:53.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:53.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:53.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:05:53.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:53.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:53.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:05:53.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3400000000000002e-06 2023-07-02T14:05:53.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:53.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060168+0000", 2023-07-02T14:05:53.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4020000000000004e-06 2023-07-02T14:05:53.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:53.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060173+0000", 2023-07-02T14:05:53.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4259999999999999e-06 2023-07-02T14:05:53.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:53.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060176+0000", 2023-07-02T14:05:53.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00049978399999999997 2023-07-02T14:05:53.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060676+0000", 2023-07-02T14:05:53.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0135000000000004e-05 2023-07-02T14:05:53.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060756+0000", 2023-07-02T14:05:53.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074338060000000003 2023-07-02T14:05:53.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068190+0000", 2023-07-02T14:05:53.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7941000000000004e-05 2023-07-02T14:05:53.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068278+0000", 2023-07-02T14:05:53.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089611870000000007 2023-07-02T14:05:53.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077239+0000", 2023-07-02T14:05:53.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6349e-05 2023-07-02T14:05:53.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077275+0000", 2023-07-02T14:05:53.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007072142 2023-07-02T14:05:53.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084347+0000", 2023-07-02T14:05:53.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.591e-05 2023-07-02T14:05:53.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084383+0000", 2023-07-02T14:05:53.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093660119999999999 2023-07-02T14:05:53.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093749+0000", 2023-07-02T14:05:53.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1538000000000002e-05 2023-07-02T14:05:53.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093781+0000", 2023-07-02T14:05:53.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088884629999999992 2023-07-02T14:05:53.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102669+0000", 2023-07-02T14:05:53.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6032999999999997e-05 2023-07-02T14:05:53.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102725+0000", 2023-07-02T14:05:53.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009945522 2023-07-02T14:05:53.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112671+0000", 2023-07-02T14:05:53.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5741000000000002e-05 2023-07-02T14:05:53.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112707+0000", 2023-07-02T14:05:53.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096867940000000003 2023-07-02T14:05:53.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122393+0000", 2023-07-02T14:05:53.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6244999999999998e-05 2023-07-02T14:05:53.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122480+0000", 2023-07-02T14:05:53.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011251107999999999 2023-07-02T14:05:53.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133731+0000", 2023-07-02T14:05:53.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6400999999999999e-05 2023-07-02T14:05:53.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133767+0000", 2023-07-02T14:05:53.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0094677070000000006 2023-07-02T14:05:53.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.143235+0000", 2023-07-02T14:05:53.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8721999999999996e-05 2023-07-02T14:05:53.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.143314+0000", 2023-07-02T14:05:53.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051196050000000002 2023-07-02T14:05:53.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.148433+0000", 2023-07-02T14:05:53.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2484999999999997e-05 2023-07-02T14:05:53.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.148476+0000", 2023-07-02T14:05:53.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006202234 2023-07-02T14:05:53.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.154678+0000", 2023-07-02T14:05:53.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2699000000000002e-05 2023-07-02T14:05:53.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.154761+0000", 2023-07-02T14:05:53.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049653809999999996 2023-07-02T14:05:53.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.159726+0000", 2023-07-02T14:05:53.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6862e-05 2023-07-02T14:05:53.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:53.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.159773+0000", 2023-07-02T14:05:53.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024618099999999999 2023-07-02T14:05:53.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:53.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.160019+0000", 2023-07-02T14:05:53.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0019043059999999999 2023-07-02T14:05:53.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:53.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.161923+0000", 2023-07-02T14:05:53.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004316884 2023-07-02T14:05:53.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:53.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.166240+0000", 2023-07-02T14:05:53.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2611000000000003e-05 2023-07-02T14:05:53.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:53.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.166273+0000", 2023-07-02T14:05:53.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4047000000000003e-05 2023-07-02T14:05:53.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:53.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.166367+0000", 2023-07-02T14:05:53.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.10620154 2023-07-02T14:05:53.042 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.042 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:53.043 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.043 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17204 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:53.043 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.130952+0000", 2023-07-02T14:05:53.043 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.6969363899999994, 2023-07-02T14:05:53.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.101008762, 2023-07-02T14:05:53.043 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:53.044 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:53.044 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:53.044 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:53.044 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:53.044 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17204 2023-07-02T14:05:53.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.044 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:53.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:53.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130952+0000", 2023-07-02T14:05:53.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:53.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:53.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130952+0000", 2023-07-02T14:05:53.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:53.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:53.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130952+0000", 2023-07-02T14:05:53.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7139999999999999e-06 2023-07-02T14:05:53.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:53.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130955+0000", 2023-07-02T14:05:53.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.054e-06 2023-07-02T14:05:53.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:53.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130962+0000", 2023-07-02T14:05:53.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7610000000000001e-06 2023-07-02T14:05:53.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:53.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130965+0000", 2023-07-02T14:05:53.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0034574929999999999 2023-07-02T14:05:53.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.134423+0000", 2023-07-02T14:05:53.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9406999999999997e-05 2023-07-02T14:05:53.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.134462+0000", 2023-07-02T14:05:53.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096105089999999997 2023-07-02T14:05:53.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.144073+0000", 2023-07-02T14:05:53.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3737000000000002e-05 2023-07-02T14:05:53.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.144146+0000", 2023-07-02T14:05:53.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050853909999999999 2023-07-02T14:05:53.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.149232+0000", 2023-07-02T14:05:53.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1114999999999999e-05 2023-07-02T14:05:53.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.149303+0000", 2023-07-02T14:05:53.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060490919999999998 2023-07-02T14:05:53.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.155352+0000", 2023-07-02T14:05:53.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5466000000000002e-05 2023-07-02T14:05:53.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.155388+0000", 2023-07-02T14:05:53.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064891990000000002 2023-07-02T14:05:53.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.161877+0000", 2023-07-02T14:05:53.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9493000000000003e-05 2023-07-02T14:05:53.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.161916+0000", 2023-07-02T14:05:53.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062878689999999998 2023-07-02T14:05:53.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.168204+0000", 2023-07-02T14:05:53.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4718000000000002e-05 2023-07-02T14:05:53.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.168239+0000", 2023-07-02T14:05:53.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043609909999999998 2023-07-02T14:05:53.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.172600+0000", 2023-07-02T14:05:53.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4609e-05 2023-07-02T14:05:53.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.172634+0000", 2023-07-02T14:05:53.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050403130000000003 2023-07-02T14:05:53.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.177675+0000", 2023-07-02T14:05:53.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.523e-05 2023-07-02T14:05:53.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.177710+0000", 2023-07-02T14:05:53.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043389429999999996 2023-07-02T14:05:53.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.182049+0000", 2023-07-02T14:05:53.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3048999999999998e-05 2023-07-02T14:05:53.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.182122+0000", 2023-07-02T14:05:53.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041995360000000002 2023-07-02T14:05:53.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.186321+0000", 2023-07-02T14:05:53.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2482000000000001e-05 2023-07-02T14:05:53.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.186404+0000", 2023-07-02T14:05:53.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013084261 2023-07-02T14:05:53.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.199488+0000", 2023-07-02T14:05:53.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013297600000000001 2023-07-02T14:05:53.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.199621+0000", 2023-07-02T14:05:53.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014611084999999999 2023-07-02T14:05:53.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.214232+0000", 2023-07-02T14:05:53.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7821000000000001e-05 2023-07-02T14:05:53.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.214290+0000", 2023-07-02T14:05:53.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089499660000000002 2023-07-02T14:05:53.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.223240+0000", 2023-07-02T14:05:53.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010005000000000001 2023-07-02T14:05:53.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:53.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.223340+0000", 2023-07-02T14:05:53.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00049127800000000005 2023-07-02T14:05:53.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:53.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.223831+0000", 2023-07-02T14:05:53.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064706440000000002 2023-07-02T14:05:53.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:53.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.230302+0000", 2023-07-02T14:05:53.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0015080759999999999 2023-07-02T14:05:53.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:53.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.231810+0000", 2023-07-02T14:05:53.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6409000000000003e-05 2023-07-02T14:05:53.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:53.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.231847+0000", 2023-07-02T14:05:53.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011443 2023-07-02T14:05:53.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:53.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.231961+0000", 2023-07-02T14:05:53.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.101008762 2023-07-02T14:05:53.069 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.069 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:53.069 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.070 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17294 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:53.070 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.256047+0000", 2023-07-02T14:05:53.070 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.5718416739999999, 2023-07-02T14:05:53.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.099803734000000005, 2023-07-02T14:05:53.070 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:53.070 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:53.070 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:53.071 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:53.071 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:53.071 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17294 2023-07-02T14:05:53.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.071 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:53.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:53.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256047+0000", 2023-07-02T14:05:53.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:53.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:53.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256047+0000", 2023-07-02T14:05:53.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:53.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:53.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256047+0000", 2023-07-02T14:05:53.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.367e-06 2023-07-02T14:05:53.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:53.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256049+0000", 2023-07-02T14:05:53.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0019999999999998e-06 2023-07-02T14:05:53.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:53.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256052+0000", 2023-07-02T14:05:53.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9450000000000002e-06 2023-07-02T14:05:53.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:53.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256055+0000", 2023-07-02T14:05:53.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022819160000000002 2023-07-02T14:05:53.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.258337+0000", 2023-07-02T14:05:53.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1145999999999999e-05 2023-07-02T14:05:53.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.258418+0000", 2023-07-02T14:05:53.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066309109999999997 2023-07-02T14:05:53.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.265049+0000", 2023-07-02T14:05:53.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6449000000000002e-05 2023-07-02T14:05:53.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.265095+0000", 2023-07-02T14:05:53.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089284169999999993 2023-07-02T14:05:53.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.274024+0000", 2023-07-02T14:05:53.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7626999999999997e-05 2023-07-02T14:05:53.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.274061+0000", 2023-07-02T14:05:53.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013396880999999999 2023-07-02T14:05:53.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.287458+0000", 2023-07-02T14:05:53.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1865000000000003e-05 2023-07-02T14:05:53.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.287540+0000", 2023-07-02T14:05:53.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069236419999999998 2023-07-02T14:05:53.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.294464+0000", 2023-07-02T14:05:53.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7681999999999993e-05 2023-07-02T14:05:53.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.294531+0000", 2023-07-02T14:05:53.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076921070000000001 2023-07-02T14:05:53.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302223+0000", 2023-07-02T14:05:53.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5528999999999998e-05 2023-07-02T14:05:53.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302259+0000", 2023-07-02T14:05:53.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096256480000000005 2023-07-02T14:05:53.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.311885+0000", 2023-07-02T14:05:53.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6389e-05 2023-07-02T14:05:53.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.311921+0000", 2023-07-02T14:05:53.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098685439999999999 2023-07-02T14:05:53.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.321790+0000", 2023-07-02T14:05:53.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1817999999999995e-05 2023-07-02T14:05:53.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.321871+0000", 2023-07-02T14:05:53.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009974966 2023-07-02T14:05:53.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.331846+0000", 2023-07-02T14:05:53.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6520000000000002e-05 2023-07-02T14:05:53.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.331893+0000", 2023-07-02T14:05:53.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086651149999999993 2023-07-02T14:05:53.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.340558+0000", 2023-07-02T14:05:53.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5918000000000001e-05 2023-07-02T14:05:53.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.340634+0000", 2023-07-02T14:05:53.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059983099999999998 2023-07-02T14:05:53.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.346632+0000", 2023-07-02T14:05:53.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7887999999999997e-05 2023-07-02T14:05:53.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:53.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.346710+0000", 2023-07-02T14:05:53.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00046673100000000001 2023-07-02T14:05:53.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:53.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.347177+0000", 2023-07-02T14:05:53.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027484829999999999 2023-07-02T14:05:53.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:53.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349925+0000", 2023-07-02T14:05:53.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056857970000000002 2023-07-02T14:05:53.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:53.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.355611+0000", 2023-07-02T14:05:53.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9406000000000001e-05 2023-07-02T14:05:53.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:53.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.355661+0000", 2023-07-02T14:05:53.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019010399999999999 2023-07-02T14:05:53.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:53.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.355851+0000", 2023-07-02T14:05:53.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.099803734000000005 2023-07-02T14:05:53.093 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.093 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:53.093 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.093 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17295 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:53.093 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.256057+0000", 2023-07-02T14:05:53.094 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.5718314539999998, 2023-07-02T14:05:53.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.107964279, 2023-07-02T14:05:53.094 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:53.094 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:53.094 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:53.094 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:53.094 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:53.095 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17295 2023-07-02T14:05:53.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.095 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:53.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:53.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256057+0000", 2023-07-02T14:05:53.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:53.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:53.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256057+0000", 2023-07-02T14:05:53.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:53.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:53.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256057+0000", 2023-07-02T14:05:53.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.458e-06 2023-07-02T14:05:53.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:53.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256059+0000", 2023-07-02T14:05:53.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9019999999999999e-06 2023-07-02T14:05:53.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:53.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256062+0000", 2023-07-02T14:05:53.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9929999999999998e-06 2023-07-02T14:05:53.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:53.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256064+0000", 2023-07-02T14:05:53.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023724789999999998 2023-07-02T14:05:53.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.258437+0000", 2023-07-02T14:05:53.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7698000000000002e-05 2023-07-02T14:05:53.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.258494+0000", 2023-07-02T14:05:53.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066424140000000001 2023-07-02T14:05:53.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.265137+0000", 2023-07-02T14:05:53.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2169000000000002e-05 2023-07-02T14:05:53.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.265179+0000", 2023-07-02T14:05:53.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089047970000000008 2023-07-02T14:05:53.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.274084+0000", 2023-07-02T14:05:53.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7489e-05 2023-07-02T14:05:53.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.274121+0000", 2023-07-02T14:05:53.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013452426999999999 2023-07-02T14:05:53.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.287574+0000", 2023-07-02T14:05:53.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5149999999999996e-05 2023-07-02T14:05:53.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.287659+0000", 2023-07-02T14:05:53.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069046059999999998 2023-07-02T14:05:53.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.294563+0000", 2023-07-02T14:05:53.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7073000000000001e-05 2023-07-02T14:05:53.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.294611+0000", 2023-07-02T14:05:53.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076769480000000003 2023-07-02T14:05:53.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302287+0000", 2023-07-02T14:05:53.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4381000000000002e-05 2023-07-02T14:05:53.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302322+0000", 2023-07-02T14:05:53.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009637863 2023-07-02T14:05:53.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.311960+0000", 2023-07-02T14:05:53.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4847999999999999e-05 2023-07-02T14:05:53.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312035+0000", 2023-07-02T14:05:53.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098741579999999992 2023-07-02T14:05:53.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.321909+0000", 2023-07-02T14:05:53.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6455999999999998e-05 2023-07-02T14:05:53.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.321945+0000", 2023-07-02T14:05:53.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099894690000000008 2023-07-02T14:05:53.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.331935+0000", 2023-07-02T14:05:53.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2550000000000002e-05 2023-07-02T14:05:53.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332007+0000", 2023-07-02T14:05:53.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086760090000000002 2023-07-02T14:05:53.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.340683+0000", 2023-07-02T14:05:53.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8462999999999998e-05 2023-07-02T14:05:53.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.340752+0000", 2023-07-02T14:05:53.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00830381 2023-07-02T14:05:53.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349055+0000", 2023-07-02T14:05:53.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9289999999999994e-05 2023-07-02T14:05:53.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349145+0000", 2023-07-02T14:05:53.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067717250000000001 2023-07-02T14:05:53.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.355917+0000", 2023-07-02T14:05:53.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4431999999999995e-05 2023-07-02T14:05:53.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:53.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.355991+0000", 2023-07-02T14:05:53.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00044773399999999998 2023-07-02T14:05:53.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:53.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.356439+0000", 2023-07-02T14:05:53.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028403690000000001 2023-07-02T14:05:53.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:53.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.359279+0000", 2023-07-02T14:05:53.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045352450000000002 2023-07-02T14:05:53.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:53.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.363814+0000", 2023-07-02T14:05:53.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6167e-05 2023-07-02T14:05:53.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:53.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.363860+0000", 2023-07-02T14:05:53.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000160951 2023-07-02T14:05:53.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:53.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.364021+0000", 2023-07-02T14:05:53.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.107964279 2023-07-02T14:05:53.117 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.117 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:53.118 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.118 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17351 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:53.118 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302537+0000", 2023-07-02T14:05:53.118 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.5253517199999997, 2023-07-02T14:05:53.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.099134853999999994, 2023-07-02T14:05:53.119 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:53.119 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:53.119 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:53.119 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:53.119 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:53.119 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17351 2023-07-02T14:05:53.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.119 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:53.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:53.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302537+0000", 2023-07-02T14:05:53.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:53.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:53.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302537+0000", 2023-07-02T14:05:53.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:53.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:53.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302536+0000", 2023-07-02T14:05:53.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3329999999999996e-06 2023-07-02T14:05:53.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:53.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302541+0000", 2023-07-02T14:05:53.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6600000000000001e-06 2023-07-02T14:05:53.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:53.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302550+0000", 2023-07-02T14:05:53.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9e-06 2023-07-02T14:05:53.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:53.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302557+0000", 2023-07-02T14:05:53.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8735999999999998e-05 2023-07-02T14:05:53.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302656+0000", 2023-07-02T14:05:53.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7564000000000002e-05 2023-07-02T14:05:53.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302693+0000", 2023-07-02T14:05:53.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096663749999999996 2023-07-02T14:05:53.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312360+0000", 2023-07-02T14:05:53.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6055000000000003e-05 2023-07-02T14:05:53.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312436+0000", 2023-07-02T14:05:53.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098653899999999999 2023-07-02T14:05:53.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322301+0000", 2023-07-02T14:05:53.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7258999999999997e-05 2023-07-02T14:05:53.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322338+0000", 2023-07-02T14:05:53.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099812860000000007 2023-07-02T14:05:53.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332320+0000", 2023-07-02T14:05:53.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3201999999999995e-05 2023-07-02T14:05:53.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332393+0000", 2023-07-02T14:05:53.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088690169999999999 2023-07-02T14:05:53.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341262+0000", 2023-07-02T14:05:53.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1718999999999995e-05 2023-07-02T14:05:53.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341344+0000", 2023-07-02T14:05:53.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008184574 2023-07-02T14:05:53.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349528+0000", 2023-07-02T14:05:53.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2019000000000001e-05 2023-07-02T14:05:53.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349620+0000", 2023-07-02T14:05:53.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089525679999999993 2023-07-02T14:05:53.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358573+0000", 2023-07-02T14:05:53.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5639e-05 2023-07-02T14:05:53.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358638+0000", 2023-07-02T14:05:53.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079895899999999995 2023-07-02T14:05:53.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366628+0000", 2023-07-02T14:05:53.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5162999999999997e-05 2023-07-02T14:05:53.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366723+0000", 2023-07-02T14:05:53.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074560440000000002 2023-07-02T14:05:53.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374179+0000", 2023-07-02T14:05:53.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9793999999999999e-05 2023-07-02T14:05:53.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374219+0000", 2023-07-02T14:05:53.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091487210000000003 2023-07-02T14:05:53.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383368+0000", 2023-07-02T14:05:53.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7316000000000004e-05 2023-07-02T14:05:53.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383435+0000", 2023-07-02T14:05:53.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010649080999999999 2023-07-02T14:05:53.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.394084+0000", 2023-07-02T14:05:53.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8598999999999997e-05 2023-07-02T14:05:53.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:53.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.394173+0000", 2023-07-02T14:05:53.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042554099999999998 2023-07-02T14:05:53.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:53.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.394598+0000", 2023-07-02T14:05:53.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048553529999999998 2023-07-02T14:05:53.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:53.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.399454+0000", 2023-07-02T14:05:53.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020927350000000001 2023-07-02T14:05:53.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:53.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401546+0000", 2023-07-02T14:05:53.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.7813999999999998e-05 2023-07-02T14:05:53.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:53.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401574+0000", 2023-07-02T14:05:53.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7508000000000002e-05 2023-07-02T14:05:53.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:53.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401672+0000", 2023-07-02T14:05:53.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.099134853999999994 2023-07-02T14:05:53.141 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.141 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:53.141 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.142 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17352 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:53.142 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302565+0000", 2023-07-02T14:05:53.142 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.5253232380000004, 2023-07-02T14:05:53.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.105667996, 2023-07-02T14:05:53.142 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:53.142 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:53.142 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17352 2023-07-02T14:05:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302565+0000", 2023-07-02T14:05:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302565+0000", 2023-07-02T14:05:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:53.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302565+0000", 2023-07-02T14:05:53.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7000000000000002e-06 2023-07-02T14:05:53.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:53.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302568+0000", 2023-07-02T14:05:53.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2140000000000001e-06 2023-07-02T14:05:53.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:53.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302575+0000", 2023-07-02T14:05:53.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3250000000000001e-06 2023-07-02T14:05:53.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:53.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302579+0000", 2023-07-02T14:05:53.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014498700000000001 2023-07-02T14:05:53.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302724+0000", 2023-07-02T14:05:53.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5444999999999993e-05 2023-07-02T14:05:53.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302799+0000", 2023-07-02T14:05:53.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096710000000000008 2023-07-02T14:05:53.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312470+0000", 2023-07-02T14:05:53.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4798999999999997e-05 2023-07-02T14:05:53.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312505+0000", 2023-07-02T14:05:53.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098663940000000006 2023-07-02T14:05:53.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322372+0000", 2023-07-02T14:05:53.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1996000000000002e-05 2023-07-02T14:05:53.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322414+0000", 2023-07-02T14:05:53.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010015735 2023-07-02T14:05:53.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332429+0000", 2023-07-02T14:05:53.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2929000000000003e-05 2023-07-02T14:05:53.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332462+0000", 2023-07-02T14:05:53.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089318550000000007 2023-07-02T14:05:53.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341394+0000", 2023-07-02T14:05:53.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7248e-05 2023-07-02T14:05:53.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341431+0000", 2023-07-02T14:05:53.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0082266739999999998 2023-07-02T14:05:53.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349658+0000", 2023-07-02T14:05:53.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3188999999999997e-05 2023-07-02T14:05:53.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349691+0000", 2023-07-02T14:05:53.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089857329999999992 2023-07-02T14:05:53.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358677+0000", 2023-07-02T14:05:53.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6436000000000001e-05 2023-07-02T14:05:53.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358713+0000", 2023-07-02T14:05:53.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080528400000000003 2023-07-02T14:05:53.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366766+0000", 2023-07-02T14:05:53.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3217999999999999e-05 2023-07-02T14:05:53.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366799+0000", 2023-07-02T14:05:53.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074649970000000001 2023-07-02T14:05:53.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374264+0000", 2023-07-02T14:05:53.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9482999999999999e-05 2023-07-02T14:05:53.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374334+0000", 2023-07-02T14:05:53.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091483889999999998 2023-07-02T14:05:53.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383482+0000", 2023-07-02T14:05:53.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3095000000000005e-05 2023-07-02T14:05:53.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383545+0000", 2023-07-02T14:05:53.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013359414 2023-07-02T14:05:53.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.396905+0000", 2023-07-02T14:05:53.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012269900000000001 2023-07-02T14:05:53.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397028+0000", 2023-07-02T14:05:53.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046629660000000002 2023-07-02T14:05:53.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401691+0000", 2023-07-02T14:05:53.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5111999999999999e-05 2023-07-02T14:05:53.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:53.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401726+0000", 2023-07-02T14:05:53.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024867800000000001 2023-07-02T14:05:53.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:53.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401974+0000", 2023-07-02T14:05:53.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028409360000000001 2023-07-02T14:05:53.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:53.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404815+0000", 2023-07-02T14:05:53.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031148529999999999 2023-07-02T14:05:53.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:53.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.407930+0000", 2023-07-02T14:05:53.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6836000000000003e-05 2023-07-02T14:05:53.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:53.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.407987+0000", 2023-07-02T14:05:53.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024641799999999998 2023-07-02T14:05:53.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:53.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408233+0000", 2023-07-02T14:05:53.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.105667996 2023-07-02T14:05:53.166 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.167 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:53.167 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.167 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17353 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:53.167 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:05:53.167 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.5253044219999996, 2023-07-02T14:05:53.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.115724379, 2023-07-02T14:05:53.168 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:53.168 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:53.168 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:53.168 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:53.168 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:53.168 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17353 2023-07-02T14:05:53.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.169 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:53.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:53.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:05:53.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:53.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:53.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:05:53.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:53.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:53.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:05:53.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.41e-06 2023-07-02T14:05:53.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:53.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302587+0000", 2023-07-02T14:05:53.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.015e-06 2023-07-02T14:05:53.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:53.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302593+0000", 2023-07-02T14:05:53.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8019999999999997e-06 2023-07-02T14:05:53.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:53.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302599+0000", 2023-07-02T14:05:53.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022701199999999999 2023-07-02T14:05:53.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302826+0000", 2023-07-02T14:05:53.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.426e-05 2023-07-02T14:05:53.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302870+0000", 2023-07-02T14:05:53.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096759489999999997 2023-07-02T14:05:53.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312546+0000", 2023-07-02T14:05:53.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5785000000000006e-05 2023-07-02T14:05:53.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312632+0000", 2023-07-02T14:05:53.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098106970000000002 2023-07-02T14:05:53.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322443+0000", 2023-07-02T14:05:53.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3161999999999999e-05 2023-07-02T14:05:53.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322476+0000", 2023-07-02T14:05:53.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010015823 2023-07-02T14:05:53.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332492+0000", 2023-07-02T14:05:53.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1165999999999999e-05 2023-07-02T14:05:53.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332533+0000", 2023-07-02T14:05:53.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089435269999999997 2023-07-02T14:05:53.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341476+0000", 2023-07-02T14:05:53.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000112691 2023-07-02T14:05:53.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341589+0000", 2023-07-02T14:05:53.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008147362 2023-07-02T14:05:53.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349736+0000", 2023-07-02T14:05:53.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5149000000000007e-05 2023-07-02T14:05:53.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349832+0000", 2023-07-02T14:05:53.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089226519999999997 2023-07-02T14:05:53.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358754+0000", 2023-07-02T14:05:53.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6091999999999998e-05 2023-07-02T14:05:53.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358810+0000", 2023-07-02T14:05:53.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080318000000000004 2023-07-02T14:05:53.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366842+0000", 2023-07-02T14:05:53.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4321999999999998e-05 2023-07-02T14:05:53.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366916+0000", 2023-07-02T14:05:53.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074664049999999997 2023-07-02T14:05:53.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374383+0000", 2023-07-02T14:05:53.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5460000000000001e-05 2023-07-02T14:05:53.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374438+0000", 2023-07-02T14:05:53.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091562450000000004 2023-07-02T14:05:53.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383594+0000", 2023-07-02T14:05:53.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2545000000000004e-05 2023-07-02T14:05:53.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383667+0000", 2023-07-02T14:05:53.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013424551 2023-07-02T14:05:53.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397092+0000", 2023-07-02T14:05:53.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6991999999999997e-05 2023-07-02T14:05:53.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397169+0000", 2023-07-02T14:05:53.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064399139999999997 2023-07-02T14:05:53.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403608+0000", 2023-07-02T14:05:53.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2614000000000006e-05 2023-07-02T14:05:53.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403691+0000", 2023-07-02T14:05:53.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004611693 2023-07-02T14:05:53.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408303+0000", 2023-07-02T14:05:53.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1935999999999995e-05 2023-07-02T14:05:53.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:53.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408395+0000", 2023-07-02T14:05:53.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00051077700000000002 2023-07-02T14:05:53.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:53.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408906+0000", 2023-07-02T14:05:53.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090767460000000001 2023-07-02T14:05:53.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:53.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.417982+0000", 2023-07-02T14:05:53.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4283999999999997e-05 2023-07-02T14:05:53.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:53.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418037+0000", 2023-07-02T14:05:53.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9667999999999999e-05 2023-07-02T14:05:53.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:53.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418096+0000", 2023-07-02T14:05:53.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021233800000000001 2023-07-02T14:05:53.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:53.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418309+0000", 2023-07-02T14:05:53.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.115724379 2023-07-02T14:05:53.193 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.194 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:53.194 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.194 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17354 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:53.194 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:05:53.194 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.5252854619999994, 2023-07-02T14:05:53.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493, 2023-07-02T14:05:53.195 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:53.195 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:53.195 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:53.195 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:53.195 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:53.195 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17354 2023-07-02T14:05:53.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.195 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:53.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:53.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:05:53.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:53.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:53.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:05:53.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:53.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:53.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:05:53.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7440000000000001e-06 2023-07-02T14:05:53.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:53.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302606+0000", 2023-07-02T14:05:53.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:05:53.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:53.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302613+0000", 2023-07-02T14:05:53.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8810000000000004e-06 2023-07-02T14:05:53.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.199 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:53.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302618+0000", 2023-07-02T14:05:53.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028613100000000002 2023-07-02T14:05:53.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302904+0000", 2023-07-02T14:05:53.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5278999999999995e-05 2023-07-02T14:05:53.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302979+0000", 2023-07-02T14:05:53.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096904750000000005 2023-07-02T14:05:53.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312669+0000", 2023-07-02T14:05:53.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0328000000000002e-05 2023-07-02T14:05:53.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312700+0000", 2023-07-02T14:05:53.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098019319999999993 2023-07-02T14:05:53.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322502+0000", 2023-07-02T14:05:53.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4669999999999998e-05 2023-07-02T14:05:53.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322536+0000", 2023-07-02T14:05:53.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010023839 2023-07-02T14:05:53.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332560+0000", 2023-07-02T14:05:53.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5559999999999998e-05 2023-07-02T14:05:53.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332596+0000", 2023-07-02T14:05:53.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090401319999999993 2023-07-02T14:05:53.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341636+0000", 2023-07-02T14:05:53.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5373000000000001e-05 2023-07-02T14:05:53.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341681+0000", 2023-07-02T14:05:53.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081994059999999994 2023-07-02T14:05:53.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349881+0000", 2023-07-02T14:05:53.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4319000000000001e-05 2023-07-02T14:05:53.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349915+0000", 2023-07-02T14:05:53.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089388339999999997 2023-07-02T14:05:53.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358854+0000", 2023-07-02T14:05:53.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4319000000000005e-05 2023-07-02T14:05:53.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358928+0000", 2023-07-02T14:05:53.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080234690000000001 2023-07-02T14:05:53.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366952+0000", 2023-07-02T14:05:53.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2082999999999998e-05 2023-07-02T14:05:53.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366984+0000", 2023-07-02T14:05:53.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075015790000000004 2023-07-02T14:05:53.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374485+0000", 2023-07-02T14:05:53.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4480999999999995e-05 2023-07-02T14:05:53.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374550+0000", 2023-07-02T14:05:53.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091666620000000008 2023-07-02T14:05:53.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383716+0000", 2023-07-02T14:05:53.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9318000000000003e-05 2023-07-02T14:05:53.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383786+0000", 2023-07-02T14:05:53.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013434115 2023-07-02T14:05:53.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397220+0000", 2023-07-02T14:05:53.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4573999999999997e-05 2023-07-02T14:05:53.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397264+0000", 2023-07-02T14:05:53.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064824649999999998 2023-07-02T14:05:53.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403747+0000", 2023-07-02T14:05:53.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1087999999999993e-05 2023-07-02T14:05:53.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403828+0000", 2023-07-02T14:05:53.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072854449999999998 2023-07-02T14:05:53.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411113+0000", 2023-07-02T14:05:53.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5120000000000001e-05 2023-07-02T14:05:53.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411169+0000", 2023-07-02T14:05:53.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007211505 2023-07-02T14:05:53.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418380+0000", 2023-07-02T14:05:53.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011289399999999999 2023-07-02T14:05:53.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:53.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418493+0000", 2023-07-02T14:05:53.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00051217500000000002 2023-07-02T14:05:53.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:53.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.419005+0000", 2023-07-02T14:05:53.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00327393 2023-07-02T14:05:53.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:53.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422279+0000", 2023-07-02T14:05:53.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041432350000000003 2023-07-02T14:05:53.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:53.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426422+0000", 2023-07-02T14:05:53.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7021000000000003e-05 2023-07-02T14:05:53.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:53.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426469+0000", 2023-07-02T14:05:53.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000190256 2023-07-02T14:05:53.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:53.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426660+0000", 2023-07-02T14:05:53.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493 2023-07-02T14:05:53.221 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.221 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:53.221 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.221 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17355 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:53.221 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:05:53.222 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.5252668010000008, 2023-07-02T14:05:53.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001, 2023-07-02T14:05:53.222 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:53.222 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:53.222 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:53.222 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:53.222 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:53.222 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17355 2023-07-02T14:05:53.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.223 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:53.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:53.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:05:53.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:53.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:53.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:05:53.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:53.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:53.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302621+0000", 2023-07-02T14:05:53.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2260000000000001e-06 2023-07-02T14:05:53.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:53.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302625+0000", 2023-07-02T14:05:53.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8599999999999998e-06 2023-07-02T14:05:53.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:53.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302630+0000", 2023-07-02T14:05:53.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0629999999999999e-06 2023-07-02T14:05:53.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:53.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302635+0000", 2023-07-02T14:05:53.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035731199999999998 2023-07-02T14:05:53.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302992+0000", 2023-07-02T14:05:53.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4164e-05 2023-07-02T14:05:53.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.303026+0000", 2023-07-02T14:05:53.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097115900000000008 2023-07-02T14:05:53.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312738+0000", 2023-07-02T14:05:53.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1701999999999998e-05 2023-07-02T14:05:53.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312819+0000", 2023-07-02T14:05:53.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097363640000000008 2023-07-02T14:05:53.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322556+0000", 2023-07-02T14:05:53.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2517000000000001e-05 2023-07-02T14:05:53.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.230 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322588+0000", 2023-07-02T14:05:53.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01002393 2023-07-02T14:05:53.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.231 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332612+0000", 2023-07-02T14:05:53.231 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2960999999999998e-05 2023-07-02T14:05:53.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.231 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332645+0000", 2023-07-02T14:05:53.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090669819999999995 2023-07-02T14:05:53.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341712+0000", 2023-07-02T14:05:53.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9288000000000002e-05 2023-07-02T14:05:53.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341811+0000", 2023-07-02T14:05:53.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081504460000000004 2023-07-02T14:05:53.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349962+0000", 2023-07-02T14:05:53.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107321 2023-07-02T14:05:53.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.350069+0000", 2023-07-02T14:05:53.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088790999999999991 2023-07-02T14:05:53.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358948+0000", 2023-07-02T14:05:53.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1711000000000002e-05 2023-07-02T14:05:53.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358980+0000", 2023-07-02T14:05:53.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080367579999999997 2023-07-02T14:05:53.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367017+0000", 2023-07-02T14:05:53.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6329000000000002e-05 2023-07-02T14:05:53.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367073+0000", 2023-07-02T14:05:53.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075097300000000001 2023-07-02T14:05:53.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374583+0000", 2023-07-02T14:05:53.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5710999999999995e-05 2023-07-02T14:05:53.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374648+0000", 2023-07-02T14:05:53.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009165912 2023-07-02T14:05:53.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383814+0000", 2023-07-02T14:05:53.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3916000000000003e-05 2023-07-02T14:05:53.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383878+0000", 2023-07-02T14:05:53.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013440532999999999 2023-07-02T14:05:53.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.240 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397319+0000", 2023-07-02T14:05:53.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7218999999999995e-05 2023-07-02T14:05:53.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.240 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397386+0000", 2023-07-02T14:05:53.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065165279999999997 2023-07-02T14:05:53.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403903+0000", 2023-07-02T14:05:53.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8152000000000005e-05 2023-07-02T14:05:53.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404001+0000", 2023-07-02T14:05:53.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072185360000000002 2023-07-02T14:05:53.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411219+0000", 2023-07-02T14:05:53.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:05:53.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411310+0000", 2023-07-02T14:05:53.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010021663 2023-07-02T14:05:53.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421331+0000", 2023-07-02T14:05:53.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0148000000000006e-05 2023-07-02T14:05:53.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421411+0000", 2023-07-02T14:05:53.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053056170000000003 2023-07-02T14:05:53.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.245 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426717+0000", 2023-07-02T14:05:53.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8242000000000001e-05 2023-07-02T14:05:53.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:53.245 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426795+0000", 2023-07-02T14:05:53.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00041308399999999998 2023-07-02T14:05:53.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.246 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.246 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:53.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.427208+0000", 2023-07-02T14:05:53.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028330849999999999 2023-07-02T14:05:53.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:53.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430041+0000", 2023-07-02T14:05:53.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00072642300000000002 2023-07-02T14:05:53.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:53.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430768+0000", 2023-07-02T14:05:53.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1847000000000002e-05 2023-07-02T14:05:53.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:53.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430800+0000", 2023-07-02T14:05:53.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9308999999999994e-05 2023-07-02T14:05:53.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:53.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430899+0000", 2023-07-02T14:05:53.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001 2023-07-02T14:05:53.249 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.249 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:53.249 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.250 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17489 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:53.250 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.389780+0000", 2023-07-02T14:05:53.250 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.4381083110000006, 2023-07-02T14:05:53.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.093494121999999999, 2023-07-02T14:05:53.250 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:53.250 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:53.250 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:53.251 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:53.251 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:53.251 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17489 2023-07-02T14:05:53.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.251 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:53.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.251 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:53.251 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.389780+0000", 2023-07-02T14:05:53.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:53.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.252 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:53.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.389780+0000", 2023-07-02T14:05:53.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:53.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.252 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:53.253 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.389779+0000", 2023-07-02T14:05:53.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9250000000000003e-06 2023-07-02T14:05:53.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:53.253 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.389784+0000", 2023-07-02T14:05:53.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2860000000000002e-06 2023-07-02T14:05:53.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:53.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.389792+0000", 2023-07-02T14:05:53.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6720000000000003e-06 2023-07-02T14:05:53.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:53.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.389796+0000", 2023-07-02T14:05:53.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5188000000000003e-05 2023-07-02T14:05:53.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.389831+0000", 2023-07-02T14:05:53.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5021999999999997e-05 2023-07-02T14:05:53.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.389877+0000", 2023-07-02T14:05:53.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079433630000000002 2023-07-02T14:05:53.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397820+0000", 2023-07-02T14:05:53.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4178999999999999e-05 2023-07-02T14:05:53.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397854+0000", 2023-07-02T14:05:53.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006876966 2023-07-02T14:05:53.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404731+0000", 2023-07-02T14:05:53.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5018999999999994e-05 2023-07-02T14:05:53.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404806+0000", 2023-07-02T14:05:53.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007040892 2023-07-02T14:05:53.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411847+0000", 2023-07-02T14:05:53.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6153000000000001e-05 2023-07-02T14:05:53.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411923+0000", 2023-07-02T14:05:53.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010256862 2023-07-02T14:05:53.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422180+0000", 2023-07-02T14:05:53.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9196999999999994e-05 2023-07-02T14:05:53.261 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.261 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.261 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422259+0000", 2023-07-02T14:05:53.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006858121 2023-07-02T14:05:53.261 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.262 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.262 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.429117+0000", 2023-07-02T14:05:53.262 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7781999999999998e-05 2023-07-02T14:05:53.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.262 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.262 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.262 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.429155+0000", 2023-07-02T14:05:53.262 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037400139999999998 2023-07-02T14:05:53.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.263 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.432895+0000", 2023-07-02T14:05:53.263 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2409000000000001e-05 2023-07-02T14:05:53.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.432927+0000", 2023-07-02T14:05:53.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041665080000000002 2023-07-02T14:05:53.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.264 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.437094+0000", 2023-07-02T14:05:53.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8099000000000001e-05 2023-07-02T14:05:53.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.265 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.437142+0000", 2023-07-02T14:05:53.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0035492800000000001 2023-07-02T14:05:53.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.440691+0000", 2023-07-02T14:05:53.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1769999999999999e-05 2023-07-02T14:05:53.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.440763+0000", 2023-07-02T14:05:53.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051038780000000001 2023-07-02T14:05:53.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.445867+0000", 2023-07-02T14:05:53.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1784000000000002e-05 2023-07-02T14:05:53.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.445949+0000", 2023-07-02T14:05:53.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091922770000000004 2023-07-02T14:05:53.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.268 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.455141+0000", 2023-07-02T14:05:53.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5089e-05 2023-07-02T14:05:53.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.455206+0000", 2023-07-02T14:05:53.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013848835 2023-07-02T14:05:53.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.469055+0000", 2023-07-02T14:05:53.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7831999999999994e-05 2023-07-02T14:05:53.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:53.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.469153+0000", 2023-07-02T14:05:53.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0004929 2023-07-02T14:05:53.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:53.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.469646+0000", 2023-07-02T14:05:53.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011213887000000001 2023-07-02T14:05:53.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.271 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:53.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.480860+0000", 2023-07-02T14:05:53.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023170170000000002 2023-07-02T14:05:53.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:53.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.483177+0000", 2023-07-02T14:05:53.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8994999999999999e-05 2023-07-02T14:05:53.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:53.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.483206+0000", 2023-07-02T14:05:53.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8773e-05 2023-07-02T14:05:53.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.273 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:53.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.483274+0000", 2023-07-02T14:05:53.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.093494121999999999 2023-07-02T14:05:53.273 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.274 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:53.274 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.274 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17503 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:53.274 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.400832+0000", 2023-07-02T14:05:53.274 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.4270567940000003, 2023-07-02T14:05:53.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.102096161, 2023-07-02T14:05:53.275 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:53.275 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:53.275 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:53.275 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:53.275 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:53.275 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17503 2023-07-02T14:05:53.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.275 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:53.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:53.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400832+0000", 2023-07-02T14:05:53.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:53.276 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:53.277 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400832+0000", 2023-07-02T14:05:53.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:53.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.277 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:53.277 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400831+0000", 2023-07-02T14:05:53.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8929999999999998e-06 2023-07-02T14:05:53.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:53.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400835+0000", 2023-07-02T14:05:53.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1750000000000005e-06 2023-07-02T14:05:53.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:53.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400843+0000", 2023-07-02T14:05:53.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0000000000000004e-06 2023-07-02T14:05:53.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:53.279 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400848+0000", 2023-07-02T14:05:53.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011499 2023-07-02T14:05:53.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400963+0000", 2023-07-02T14:05:53.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3274000000000004e-05 2023-07-02T14:05:53.280 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401027+0000", 2023-07-02T14:05:53.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039469880000000002 2023-07-02T14:05:53.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404974+0000", 2023-07-02T14:05:53.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1142000000000004e-05 2023-07-02T14:05:53.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.405045+0000", 2023-07-02T14:05:53.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069901540000000002 2023-07-02T14:05:53.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.282 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.412035+0000", 2023-07-02T14:05:53.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4048999999999994e-05 2023-07-02T14:05:53.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.283 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.412119+0000", 2023-07-02T14:05:53.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010323508 2023-07-02T14:05:53.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422442+0000", 2023-07-02T14:05:53.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7278000000000001e-05 2023-07-02T14:05:53.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422520+0000", 2023-07-02T14:05:53.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067846620000000003 2023-07-02T14:05:53.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.429304+0000", 2023-07-02T14:05:53.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2307000000000001e-05 2023-07-02T14:05:53.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.429337+0000", 2023-07-02T14:05:53.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036949779999999998 2023-07-02T14:05:53.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.433032+0000", 2023-07-02T14:05:53.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6023999999999998e-05 2023-07-02T14:05:53.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.433088+0000", 2023-07-02T14:05:53.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041466389999999997 2023-07-02T14:05:53.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.437234+0000", 2023-07-02T14:05:53.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7540000000000003e-05 2023-07-02T14:05:53.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.437272+0000", 2023-07-02T14:05:53.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0035993079999999998 2023-07-02T14:05:53.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.440871+0000", 2023-07-02T14:05:53.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5723000000000007e-05 2023-07-02T14:05:53.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.440947+0000", 2023-07-02T14:05:53.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051923990000000003 2023-07-02T14:05:53.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.446139+0000", 2023-07-02T14:05:53.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7749000000000004e-05 2023-07-02T14:05:53.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.446217+0000", 2023-07-02T14:05:53.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093492230000000003 2023-07-02T14:05:53.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.455566+0000", 2023-07-02T14:05:53.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1462e-05 2023-07-02T14:05:53.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.455648+0000", 2023-07-02T14:05:53.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016420769000000002 2023-07-02T14:05:53.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.472068+0000", 2023-07-02T14:05:53.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7961999999999996e-05 2023-07-02T14:05:53.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.472156+0000", 2023-07-02T14:05:53.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012805568 2023-07-02T14:05:53.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.294 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.484962+0000", 2023-07-02T14:05:53.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5397999999999998e-05 2023-07-02T14:05:53.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.485047+0000", 2023-07-02T14:05:53.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080975649999999993 2023-07-02T14:05:53.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.295 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.493145+0000", 2023-07-02T14:05:53.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00018218600000000001 2023-07-02T14:05:53.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:53.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.493327+0000", 2023-07-02T14:05:53.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00055431199999999995 2023-07-02T14:05:53.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:53.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.493881+0000", 2023-07-02T14:05:53.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003367676 2023-07-02T14:05:53.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:53.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.497249+0000", 2023-07-02T14:05:53.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054194930000000001 2023-07-02T14:05:53.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:53.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.502669+0000", 2023-07-02T14:05:53.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4320999999999996e-05 2023-07-02T14:05:53.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:53.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.502733+0000", 2023-07-02T14:05:53.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000194966 2023-07-02T14:05:53.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:53.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.502928+0000", 2023-07-02T14:05:53.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.102096161 2023-07-02T14:05:53.299 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.299 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:53.299 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.300 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17562 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:53.300 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.481218+0000", 2023-07-02T14:05:53.300 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.3466705829999999, 2023-07-02T14:05:53.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.094429647000000005, 2023-07-02T14:05:53.300 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:53.301 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:53.301 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:53.301 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:53.301 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:53.301 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17562 2023-07-02T14:05:53.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.301 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:53.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:53.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.481218+0000", 2023-07-02T14:05:53.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:53.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:53.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.481218+0000", 2023-07-02T14:05:53.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:53.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:53.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.481218+0000", 2023-07-02T14:05:53.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1770000000000001e-06 2023-07-02T14:05:53.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:53.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.481220+0000", 2023-07-02T14:05:53.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3380000000000001e-06 2023-07-02T14:05:53.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:53.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.481223+0000", 2023-07-02T14:05:53.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.215e-06 2023-07-02T14:05:53.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:53.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.481225+0000", 2023-07-02T14:05:53.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048269530000000001 2023-07-02T14:05:53.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.486052+0000", 2023-07-02T14:05:53.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4368000000000003e-05 2023-07-02T14:05:53.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.486137+0000", 2023-07-02T14:05:53.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011071688 2023-07-02T14:05:53.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.307 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.497208+0000", 2023-07-02T14:05:53.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4938000000000003e-05 2023-07-02T14:05:53.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.497243+0000", 2023-07-02T14:05:53.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084346579999999994 2023-07-02T14:05:53.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.505678+0000", 2023-07-02T14:05:53.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3590000000000002e-05 2023-07-02T14:05:53.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.505712+0000", 2023-07-02T14:05:53.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013633684 2023-07-02T14:05:53.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.519345+0000", 2023-07-02T14:05:53.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2536000000000002e-05 2023-07-02T14:05:53.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.519378+0000", 2023-07-02T14:05:53.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059358880000000003 2023-07-02T14:05:53.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.525314+0000", 2023-07-02T14:05:53.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5751e-05 2023-07-02T14:05:53.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.525390+0000", 2023-07-02T14:05:53.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045313630000000001 2023-07-02T14:05:53.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.529921+0000", 2023-07-02T14:05:53.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8229e-05 2023-07-02T14:05:53.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.529989+0000", 2023-07-02T14:05:53.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096487320000000001 2023-07-02T14:05:53.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.539638+0000", 2023-07-02T14:05:53.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7615999999999997e-05 2023-07-02T14:05:53.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.539715+0000", 2023-07-02T14:05:53.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068960460000000003 2023-07-02T14:05:53.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.546612+0000", 2023-07-02T14:05:53.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9788000000000002e-05 2023-07-02T14:05:53.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.546691+0000", 2023-07-02T14:05:53.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073086089999999998 2023-07-02T14:05:53.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.554000+0000", 2023-07-02T14:05:53.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5504000000000001e-05 2023-07-02T14:05:53.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.554075+0000", 2023-07-02T14:05:53.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062212200000000004 2023-07-02T14:05:53.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.560297+0000", 2023-07-02T14:05:53.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2172999999999998e-05 2023-07-02T14:05:53.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:53.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.560339+0000", 2023-07-02T14:05:53.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022935499999999999 2023-07-02T14:05:53.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:53.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.560568+0000", 2023-07-02T14:05:53.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058391780000000004 2023-07-02T14:05:53.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:53.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.566407+0000", 2023-07-02T14:05:53.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090585809999999996 2023-07-02T14:05:53.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:53.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.575466+0000", 2023-07-02T14:05:53.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6462000000000002e-05 2023-07-02T14:05:53.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:53.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.575532+0000", 2023-07-02T14:05:53.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011524800000000001 2023-07-02T14:05:53.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:53.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.575648+0000", 2023-07-02T14:05:53.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.094429647000000005 2023-07-02T14:05:53.321 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.321 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:53.321 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.322 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18093 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:53.322 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:44.314438+0000", 2023-07-02T14:05:53.322 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.5134501149999995, 2023-07-02T14:05:53.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.100684336, 2023-07-02T14:05:53.322 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:53.322 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:53.323 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:53.323 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:53.323 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:53.323 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18093 2023-07-02T14:05:53.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.323 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:53.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:53.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.314438+0000", 2023-07-02T14:05:53.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:53.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.324 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:53.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.314438+0000", 2023-07-02T14:05:53.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:53.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:53.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.314438+0000", 2023-07-02T14:05:53.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9950000000000001e-06 2023-07-02T14:05:53.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:53.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.314441+0000", 2023-07-02T14:05:53.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5540000000000002e-06 2023-07-02T14:05:53.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:53.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.314445+0000", 2023-07-02T14:05:53.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4939999999999999e-06 2023-07-02T14:05:53.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:53.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.314448+0000", 2023-07-02T14:05:53.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0092987369999999996 2023-07-02T14:05:53.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.323747+0000", 2023-07-02T14:05:53.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9320000000000003e-05 2023-07-02T14:05:53.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.323786+0000", 2023-07-02T14:05:53.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017813928 2023-07-02T14:05:53.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.341600+0000", 2023-07-02T14:05:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4443000000000002e-05 2023-07-02T14:05:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.341635+0000", 2023-07-02T14:05:53.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057434189999999996 2023-07-02T14:05:53.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.347378+0000", 2023-07-02T14:05:53.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0620999999999997e-05 2023-07-02T14:05:53.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.347409+0000", 2023-07-02T14:05:53.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056030760000000002 2023-07-02T14:05:53.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.353012+0000", 2023-07-02T14:05:53.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9368000000000003e-05 2023-07-02T14:05:53.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.353091+0000", 2023-07-02T14:05:53.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041744800000000004 2023-07-02T14:05:53.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.357265+0000", 2023-07-02T14:05:53.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2137000000000001e-05 2023-07-02T14:05:53.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.357298+0000", 2023-07-02T14:05:53.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011258729 2023-07-02T14:05:53.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.368556+0000", 2023-07-02T14:05:53.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1092000000000003e-05 2023-07-02T14:05:53.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.368627+0000", 2023-07-02T14:05:53.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086375040000000007 2023-07-02T14:05:53.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.377265+0000", 2023-07-02T14:05:53.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3410000000000004e-05 2023-07-02T14:05:53.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.377338+0000", 2023-07-02T14:05:53.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077396879999999998 2023-07-02T14:05:53.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.385078+0000", 2023-07-02T14:05:53.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000100339 2023-07-02T14:05:53.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.385178+0000", 2023-07-02T14:05:53.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080754450000000005 2023-07-02T14:05:53.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.393254+0000", 2023-07-02T14:05:53.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012267699999999999 2023-07-02T14:05:53.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:05:53.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.393377+0000", 2023-07-02T14:05:53.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089193870000000008 2023-07-02T14:05:53.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:53.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.402296+0000", 2023-07-02T14:05:53.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6298000000000003e-05 2023-07-02T14:05:53.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:53.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.402342+0000", 2023-07-02T14:05:53.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024651799999999999 2023-07-02T14:05:53.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:53.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.402589+0000", 2023-07-02T14:05:53.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012327615 2023-07-02T14:05:53.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:53.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.414916+0000", 2023-07-02T14:05:53.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8810000000000002e-05 2023-07-02T14:05:53.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:53.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.414965+0000", 2023-07-02T14:05:53.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8016999999999997e-05 2023-07-02T14:05:53.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:53.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.415023+0000", 2023-07-02T14:05:53.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9636000000000006e-05 2023-07-02T14:05:53.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:53.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:53.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:53.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.415123+0000", 2023-07-02T14:05:53.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.100684336 2023-07-02T14:05:53.343 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.343 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:53.343 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.344 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:53.344 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:53.344 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:53.345 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:53.344+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:53.346 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:05:53.418 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:05:53.462 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:53.461+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:53.489 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:53.489 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:53.489 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:05:53.489 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:53.500 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:05:53.546 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:53.545+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:53.615 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:05:53.646 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:53.645+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:53.667 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:53.667 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:53.667 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:05:53.667 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:05:53.667 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:53.678 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:05:53.747 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:53.746+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:53.758 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:05:53.776 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:53.794 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:53.805 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:53.806 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:53.812 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:53.816 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:05:53.830 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:53.847 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:53.846+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:53.849 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:05:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:05:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:05:53.851 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:05:53.851 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:05:53.851 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:05:53.941 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:53.941 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:53.948 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:53.948+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:53.951 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:05:54.049 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:54.048+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:54.074 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:54.074 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:54.084 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:05:54.150 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:54.149+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:54.210 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:54.210 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:54.220 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:05:54.263 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:54.263+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:54.352 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:54.351+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:54.357 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:54.358 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:54.368 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:05:54.416 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:05:54.474 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:54.473+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:54.533 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:54.533 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:54.543 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:05:54.586 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:54.586+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:54.654 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:54.654+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:54.667 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:54.667 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:54.677 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:05:54.755 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:54.754+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:54.841 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:54.841 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:54.851 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:05:54.856 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:54.855+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:54.957 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:54.956+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:54.974 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:54.974 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:54.984 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:05:55.058 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:55.057+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:55.115 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:55.116 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:55.127 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:05:55.159 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:55.158+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:55.260 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:55.259+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:55.294 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:55.294 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:55.304 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:05:55.388 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:55.387+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:55.449 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap44 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap45 b3fbb028-e267-4d25-a750-93ba18a2316a@snap45.diff 2023-07-02T14:05:55.461 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:55.460+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:55.481 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:55.482 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:55.494 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:05:55.576 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:55.576+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:55.663 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:55.662+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:55.666 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:55.666 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:55.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:05:55.763 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:55.762+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:55.827 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:55.827 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:55.839 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:05:55.862 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:05:55.880 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:55.879+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:55.965 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:55.964+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:56.033 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:56.033 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:56.037 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:05:56.046 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:05:56.057 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:56.065 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:56.065+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:56.077 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:56.100 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:56.120 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:56.143 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:56.145 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:05:56.145 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:05:56.145 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:05:56.146 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:05:56.146 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:05:56.146 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:05:56.172 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:05:56.199 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:56.199 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:05:56.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:05:56.200 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:56.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.200 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:05:56.201 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:56.201 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 166.744395346, 2023-07-02T14:05:56.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:05:56.201 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.202 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:56.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:56.202 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:56.203 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:05:56.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.203 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:56.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:05:56.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:56.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:05:56.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:05:56.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:05:56.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:05:56.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:56.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:05:56.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:05:56.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:56.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:05:56.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:05:56.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:05:56.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:05:56.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:56.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:05:56.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:05:56.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:05:56.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:05:56.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:56.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:05:56.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:05:56.216 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.216 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.216 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.217 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:05:56.217 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:56.218 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 165.87370233799999, 2023-07-02T14:05:56.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:05:56.218 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.218 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:56.219 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.219 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:05:56.219 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:05:56.219 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:05:56.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.220 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:56.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:05:56.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:56.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:05:56.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:05:56.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:05:56.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:05:56.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:56.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:05:56.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:05:56.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:56.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:05:56.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:05:56.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:05:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:05:56.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:56.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:05:56.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:05:56.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:05:56.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:05:56.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:56.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:05:56.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:05:56.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:56.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:05:56.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:05:56.234 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.234 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.234 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.235 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:56.235 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:56.235 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 165.873577617, 2023-07-02T14:05:56.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:05:56.236 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.236 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:56.236 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.237 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:56.237 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:56.237 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:05:56.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.238 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:56.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:05:56.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:56.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:05:56.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:05:56.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:05:56.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:05:56.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:56.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:05:56.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:05:56.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:56.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:05:56.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:05:56.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:05:56.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:05:56.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:56.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:05:56.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:05:56.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:05:56.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:05:56.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:56.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:05:56.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:05:56.250 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.250 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.251 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.251 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:56.252 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:56.252 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 165.866727733, 2023-07-02T14:05:56.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:05:56.252 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.253 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:56.253 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.253 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:56.253 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:56.254 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:05:56.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.254 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:56.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:05:56.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:05:56.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:05:56.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:05:56.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:05:56.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:05:56.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:56.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:05:56.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:05:56.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:56.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:05:56.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:05:56.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:05:56.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:05:56.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:56.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:05:56.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:05:56.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:05:56.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:05:56.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:56.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:05:56.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:05:56.263 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.263 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.263 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.264 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:56.264 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:56.264 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 165.866664808, 2023-07-02T14:05:56.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:05:56.264 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.264 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:56.264 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.264 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:56.265 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:56.265 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:05:56.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.265 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:56.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:05:56.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:56.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:05:56.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:05:56.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:05:56.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:05:56.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:56.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:05:56.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:05:56.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:56.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:05:56.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:05:56.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:05:56.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:05:56.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:56.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:05:56.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:05:56.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:05:56.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:05:56.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:56.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:05:56.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:05:56.271 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.271 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.272 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.272 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:05:56.272 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:56.272 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 165.866364387, 2023-07-02T14:05:56.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:05:56.273 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.273 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:56.273 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.273 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:56.273 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:56.273 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:05:56.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.273 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:56.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:05:56.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:56.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:05:56.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:05:56.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:05:56.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:05:56.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:56.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:05:56.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:05:56.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:56.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:05:56.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:05:56.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:05:56.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:05:56.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:56.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:05:56.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:05:56.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:05:56.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:05:56.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:56.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:05:56.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:05:56.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:05:56.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:05:56.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:05:56.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:56.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:05:56.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:05:56.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:56.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:05:56.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:05:56.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:56.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:05:56.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:05:56.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:56.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:05:56.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:05:56.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:56.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:05:56.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:05:56.284 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.284 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.284 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.285 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:05:56.285 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:56.285 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 165.86583988699999, 2023-07-02T14:05:56.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:05:56.285 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.285 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:56.285 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.285 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:56.286 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:56.286 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:05:56.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.286 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:56.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:56.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:56.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:05:56.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:05:56.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:05:56.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:05:56.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:56.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:05:56.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:05:56.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:56.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:05:56.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:05:56.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:05:56.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:05:56.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:56.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:05:56.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:05:56.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:05:56.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:05:56.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:56.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:05:56.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:05:56.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:56.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:05:56.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:05:56.293 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.293 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.293 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.294 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:05:56.294 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:56.294 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 138.741820762, 2023-07-02T14:05:56.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:05:56.294 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.294 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:56.294 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.295 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:56.295 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:56.295 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:05:56.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.295 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:56.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:05:56.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:56.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:05:56.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:05:56.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:05:56.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:05:56.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:56.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:05:56.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:05:56.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:56.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:05:56.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:05:56.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:05:56.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:05:56.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:56.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:05:56.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:05:56.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:05:56.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:05:56.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:56.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:05:56.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:05:56.302 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.302 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.302 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.302 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:56.302 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:56.302 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 138.41633805800001, 2023-07-02T14:05:56.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:05:56.303 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.303 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:56.303 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.303 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:56.303 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:56.303 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:05:56.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.304 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:56.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:05:56.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:56.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:05:56.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:05:56.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:05:56.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:05:56.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:56.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:05:56.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:05:56.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:56.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:05:56.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:05:56.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:56.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:05:56.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:56.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:05:56.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:05:56.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:05:56.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:05:56.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:56.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:05:56.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:05:56.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:56.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:05:56.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:05:56.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:56.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:05:56.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:05:56.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:56.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:05:56.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:05:56.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:56.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:05:56.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:05:56.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:56.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:05:56.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:05:56.313 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.313 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.314 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.314 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3939 2.1 2:9ef34a83:::rbd_data.125de1449605.00000000000000b8:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:05:56.314 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:56.314 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 138.412417361, 2023-07-02T14:05:56.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001, 2023-07-02T14:05:56.314 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.315 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:56.315 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.315 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:56.315 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:56.315 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3939 2023-07-02T14:05:56.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.315 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:56.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:56.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:56.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:05:56.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046852830000000002 2023-07-02T14:05:56.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788763+0000", 2023-07-02T14:05:56.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3185e-05 2023-07-02T14:05:56.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:56.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788776+0000", 2023-07-02T14:05:56.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8470000000000003e-06 2023-07-02T14:05:56.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:56.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788781+0000", 2023-07-02T14:05:56.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7104000000000002e-05 2023-07-02T14:05:56.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788828+0000", 2023-07-02T14:05:56.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3516000000000001e-05 2023-07-02T14:05:56.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:56.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.788841+0000", 2023-07-02T14:05:56.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.33064513600000001 2023-07-02T14:05:56.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119487+0000", 2023-07-02T14:05:56.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9876999999999997e-05 2023-07-02T14:05:56.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:56.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119537+0000", 2023-07-02T14:05:56.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014160999999999999 2023-07-02T14:05:56.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:56.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.119678+0000", 2023-07-02T14:05:56.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023054392 2023-07-02T14:05:56.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:56.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.142733+0000", 2023-07-02T14:05:56.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0074065470000000003 2023-07-02T14:05:56.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:56.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150139+0000", 2023-07-02T14:05:56.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5025999999999997e-05 2023-07-02T14:05:56.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:56.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150204+0000", 2023-07-02T14:05:56.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7147999999999997e-05 2023-07-02T14:05:56.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:56.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.150241+0000", 2023-07-02T14:05:56.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36616328500000001 2023-07-02T14:05:56.325 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.325 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.325 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.326 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:17 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 0~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:56.326 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:05:56.326 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.998062275999999, 2023-07-02T14:05:56.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997, 2023-07-02T14:05:56.326 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.326 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:56.326 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.327 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:56.327 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:56.327 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17 2023-07-02T14:05:56.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.327 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:05:56.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:05:56.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:05:56.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198431+0000", 2023-07-02T14:05:56.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2815e-05 2023-07-02T14:05:56.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198443+0000", 2023-07-02T14:05:56.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2569999999999999e-05 2023-07-02T14:05:56.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:56.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198466+0000", 2023-07-02T14:05:56.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0223e-05 2023-07-02T14:05:56.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:56.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198476+0000", 2023-07-02T14:05:56.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1792999999999997e-05 2023-07-02T14:05:56.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:05:56.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4249999999999998e-06 2023-07-02T14:05:56.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:56.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198536+0000", 2023-07-02T14:05:56.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91888663500000001 2023-07-02T14:05:56.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117423+0000", 2023-07-02T14:05:56.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223594 2023-07-02T14:05:56.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:56.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117647+0000", 2023-07-02T14:05:56.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033832900000000001 2023-07-02T14:05:56.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:56.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117985+0000", 2023-07-02T14:05:56.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997 2023-07-02T14:05:56.334 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.334 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.334 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.335 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:18 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 65536~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:56.335 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:56.335 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.998005065000001, 2023-07-02T14:05:56.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998, 2023-07-02T14:05:56.335 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.336 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:56.336 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.336 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:56.336 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:56.336 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18 2023-07-02T14:05:56.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.336 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:56.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:56.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:56.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:05:56.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7600000000000002e-06 2023-07-02T14:05:56.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198494+0000", 2023-07-02T14:05:56.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7840000000000004e-06 2023-07-02T14:05:56.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:56.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198502+0000", 2023-07-02T14:05:56.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6459999999999998e-06 2023-07-02T14:05:56.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:56.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198508+0000", 2023-07-02T14:05:56.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7395999999999997e-05 2023-07-02T14:05:56.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198565+0000", 2023-07-02T14:05:56.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6800000000000004e-06 2023-07-02T14:05:56.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:56.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198572+0000", 2023-07-02T14:05:56.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91965896800000002 2023-07-02T14:05:56.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118231+0000", 2023-07-02T14:05:56.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5212999999999998e-05 2023-07-02T14:05:56.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:56.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118326+0000", 2023-07-02T14:05:56.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000238538 2023-07-02T14:05:56.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:56.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118565+0000", 2023-07-02T14:05:56.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998 2023-07-02T14:05:56.343 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.343 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.344 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.344 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:19 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 131072~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:56.344 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:56.344 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.997980841, 2023-07-02T14:05:56.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002, 2023-07-02T14:05:56.345 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.345 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:56.345 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.345 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:56.345 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:56.345 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 19 2023-07-02T14:05:56.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.345 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:56.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:56.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:56.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:05:56.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1990000000000002e-06 2023-07-02T14:05:56.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198517+0000", 2023-07-02T14:05:56.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8789999999999997e-06 2023-07-02T14:05:56.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:56.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198524+0000", 2023-07-02T14:05:56.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:05:56.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:56.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:05:56.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5545999999999999e-05 2023-07-02T14:05:56.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:56.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2540000000000003e-06 2023-07-02T14:05:56.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:56.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198598+0000", 2023-07-02T14:05:56.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.919983263 2023-07-02T14:05:56.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118581+0000", 2023-07-02T14:05:56.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6878999999999994e-05 2023-07-02T14:05:56.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:56.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118648+0000", 2023-07-02T14:05:56.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149732 2023-07-02T14:05:56.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:56.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118798+0000", 2023-07-02T14:05:56.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002 2023-07-02T14:05:56.353 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.353 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.353 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.353 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:20 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 262144~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:56.353 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:56.354 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.997961877000002, 2023-07-02T14:05:56.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998, 2023-07-02T14:05:56.354 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.354 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:56.354 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.354 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:56.354 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:56.354 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20 2023-07-02T14:05:56.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.355 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:56.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:56.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:56.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:05:56.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.732e-06 2023-07-02T14:05:56.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198537+0000", 2023-07-02T14:05:56.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5259999999999999e-06 2023-07-02T14:05:56.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:56.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198543+0000", 2023-07-02T14:05:56.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.121e-06 2023-07-02T14:05:56.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:56.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198547+0000", 2023-07-02T14:05:56.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9591000000000005e-05 2023-07-02T14:05:56.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:05:56.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3120000000000004e-06 2023-07-02T14:05:56.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:56.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198621+0000", 2023-07-02T14:05:56.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92020789400000003 2023-07-02T14:05:56.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118829+0000", 2023-07-02T14:05:56.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022393386000000001 2023-07-02T14:05:56.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:56.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141223+0000", 2023-07-02T14:05:56.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000329866 2023-07-02T14:05:56.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:56.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141552+0000", 2023-07-02T14:05:56.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998 2023-07-02T14:05:56.362 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.362 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.362 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.363 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:21 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 196608~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:56.363 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:56.363 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.997934241999999, 2023-07-02T14:05:56.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995, 2023-07-02T14:05:56.363 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.363 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:56.363 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.363 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:56.364 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:56.364 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21 2023-07-02T14:05:56.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.364 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:56.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:56.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:56.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:05:56.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0169999999999999e-05 2023-07-02T14:05:56.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198571+0000", 2023-07-02T14:05:56.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3479999999999998e-06 2023-07-02T14:05:56.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:56.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198578+0000", 2023-07-02T14:05:56.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8439999999999997e-06 2023-07-02T14:05:56.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:56.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198583+0000", 2023-07-02T14:05:56.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8294999999999997e-05 2023-07-02T14:05:56.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198641+0000", 2023-07-02T14:05:56.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8120000000000001e-06 2023-07-02T14:05:56.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:56.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:05:56.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94293828499999999 2023-07-02T14:05:56.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141583+0000", 2023-07-02T14:05:56.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014607300000000001 2023-07-02T14:05:56.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:56.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141729+0000", 2023-07-02T14:05:56.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00044906600000000003 2023-07-02T14:05:56.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:56.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142178+0000", 2023-07-02T14:05:56.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995 2023-07-02T14:05:56.371 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.371 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.371 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.372 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:22 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 327680~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:56.372 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:56.372 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.997900811999997, 2023-07-02T14:05:56.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004, 2023-07-02T14:05:56.372 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.373 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:56.373 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.373 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:56.373 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:56.373 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 22 2023-07-02T14:05:56.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.373 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:56.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:56.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:56.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:05:56.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4850000000000001e-06 2023-07-02T14:05:56.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198597+0000", 2023-07-02T14:05:56.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:05:56.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:56.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198605+0000", 2023-07-02T14:05:56.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4839999999999998e-06 2023-07-02T14:05:56.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:56.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198609+0000", 2023-07-02T14:05:56.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3100999999999999e-05 2023-07-02T14:05:56.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198662+0000", 2023-07-02T14:05:56.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9099999999999998e-06 2023-07-02T14:05:56.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:56.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198666+0000", 2023-07-02T14:05:56.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.943539503 2023-07-02T14:05:56.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142206+0000", 2023-07-02T14:05:56.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012255999999999999 2023-07-02T14:05:56.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:56.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142328+0000", 2023-07-02T14:05:56.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208956 2023-07-02T14:05:56.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:56.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142537+0000", 2023-07-02T14:05:56.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004 2023-07-02T14:05:56.381 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.381 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.381 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.381 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:23 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 393216~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:56.381 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:56.381 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.997880832, 2023-07-02T14:05:56.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348, 2023-07-02T14:05:56.382 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.382 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:56.382 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.382 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:56.382 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:56.382 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23 2023-07-02T14:05:56.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.383 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:56.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:56.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:56.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:05:56.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8439999999999998e-06 2023-07-02T14:05:56.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:05:56.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0440000000000001e-06 2023-07-02T14:05:56.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:56.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198623+0000", 2023-07-02T14:05:56.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4480000000000002e-06 2023-07-02T14:05:56.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:56.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198628+0000", 2023-07-02T14:05:56.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5843e-05 2023-07-02T14:05:56.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:56.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.89e-06 2023-07-02T14:05:56.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:56.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198688+0000", 2023-07-02T14:05:56.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94389171999999999 2023-07-02T14:05:56.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142580+0000", 2023-07-02T14:05:56.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3892000000000004e-05 2023-07-02T14:05:56.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:56.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142644+0000", 2023-07-02T14:05:56.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017608000000000001 2023-07-02T14:05:56.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:56.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142820+0000", 2023-07-02T14:05:56.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348 2023-07-02T14:05:56.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.390 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.390 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:24 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 458752~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:56.391 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:05:56.391 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.997852758000001, 2023-07-02T14:05:56.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004, 2023-07-02T14:05:56.391 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.391 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:56.391 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.391 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:56.391 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:56.392 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 24 2023-07-02T14:05:56.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.392 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:05:56.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:05:56.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:56.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198642+0000", 2023-07-02T14:05:56.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.253e-06 2023-07-02T14:05:56.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:05:56.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1289999999999998e-06 2023-07-02T14:05:56.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:56.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198652+0000", 2023-07-02T14:05:56.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4510000000000002e-06 2023-07-02T14:05:56.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:56.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198657+0000", 2023-07-02T14:05:56.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9131000000000001e-05 2023-07-02T14:05:56.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198706+0000", 2023-07-02T14:05:56.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8890000000000004e-06 2023-07-02T14:05:56.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:56.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198710+0000", 2023-07-02T14:05:56.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94415412200000004 2023-07-02T14:05:56.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142864+0000", 2023-07-02T14:05:56.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8047000000000006e-05 2023-07-02T14:05:56.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:56.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142952+0000", 2023-07-02T14:05:56.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192967 2023-07-02T14:05:56.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143145+0000", 2023-07-02T14:05:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004 2023-07-02T14:05:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.400 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:25 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 524288~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:56.400 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:56.400 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.997833962999998, 2023-07-02T14:05:56.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001, 2023-07-02T14:05:56.400 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.400 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:56.400 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.400 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:56.401 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:56.401 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25 2023-07-02T14:05:56.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.401 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:56.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:56.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:56.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:05:56.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.615e-06 2023-07-02T14:05:56.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198664+0000", 2023-07-02T14:05:56.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4759999999999997e-06 2023-07-02T14:05:56.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:56.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198669+0000", 2023-07-02T14:05:56.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5999999999999998e-06 2023-07-02T14:05:56.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:56.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198673+0000", 2023-07-02T14:05:56.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4256999999999999e-05 2023-07-02T14:05:56.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198727+0000", 2023-07-02T14:05:56.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840000000000001e-06 2023-07-02T14:05:56.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:56.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198731+0000", 2023-07-02T14:05:56.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94444241900000003 2023-07-02T14:05:56.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143173+0000", 2023-07-02T14:05:56.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6839000000000002e-05 2023-07-02T14:05:56.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:56.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143240+0000", 2023-07-02T14:05:56.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017053399999999999 2023-07-02T14:05:56.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:56.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143411+0000", 2023-07-02T14:05:56.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001 2023-07-02T14:05:56.408 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.408 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.408 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.409 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:26 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 589824~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:05:56.409 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:56.409 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.997810811999997, 2023-07-02T14:05:56.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997, 2023-07-02T14:05:56.409 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.409 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:56.409 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:05:56.410 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:05:56.410 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 26 2023-07-02T14:05:56.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.410 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:56.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:05:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0520000000000002e-06 2023-07-02T14:05:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198687+0000", 2023-07-02T14:05:56.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1880000000000002e-06 2023-07-02T14:05:56.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:56.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198695+0000", 2023-07-02T14:05:56.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9960000000000004e-06 2023-07-02T14:05:56.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:56.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198699+0000", 2023-07-02T14:05:56.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2330999999999998e-05 2023-07-02T14:05:56.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198742+0000", 2023-07-02T14:05:56.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7950000000000001e-06 2023-07-02T14:05:56.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:56.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198745+0000", 2023-07-02T14:05:56.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94469398199999999 2023-07-02T14:05:56.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:56.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143439+0000", 2023-07-02T14:05:56.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6048e-05 2023-07-02T14:05:56.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:56.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143525+0000", 2023-07-02T14:05:56.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017016700000000001 2023-07-02T14:05:56.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:56.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143696+0000", 2023-07-02T14:05:56.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997 2023-07-02T14:05:56.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.417 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.417 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.418 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:56.418 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:56.418 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:56.418 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:56.197+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:56.419 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:05:56.420 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:56.420 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:56.421 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:05:56.421 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:56.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-07-02T14:05:56.422 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:56.421+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:56.430 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete...2023-07-02T14:05:56.429+0000 7f07add5a700 0 -- 172.21.15.163:0/1999128455 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f0794006500 msgr2=0x7f0794004110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:56.433 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:05:56.432+0000 7f07add5a700 0 -- 172.21.15.163:0/1999128455 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f0790037d00 msgr2=0x7f079003a0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:05:56.434 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:05:56.523 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:56.522+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:56.624 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:56.623+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:56.638 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:56.638 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:56.638 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:05:56.638 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:56.651 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:05:56.735 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:56.734+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:05:56.819 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:56.819 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:05:56.819 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:05:56.820 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:05:56.820 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:56.826 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:56.825+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:56.829 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:05:56.927 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:56.926+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:56.994 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:05:56.995 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:05:56.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:05:56.995 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:05:56.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.995 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:05:56.995 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:56.995 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 175.88545612499999, 2023-07-02T14:05:56.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:05:56.996 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:56.996 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:56.996 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:56.996 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:56.996 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:56.996 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:05:56.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.997 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:56.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:56.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:56.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:56.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:56.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:05:56.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:56.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:56.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:05:56.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:05:56.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:56.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:05:56.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:05:56.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:56.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:56.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:57.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:05:57.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:05:57.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:05:57.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:05:57.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:05:57.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:05:57.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:57.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:05:57.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:05:57.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:05:57.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:05:57.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:57.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:05:57.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:05:57.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:57.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:05:57.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:05:57.004 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.004 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.004 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.004 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:05:57.005 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:57.005 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 175.88539538699999, 2023-07-02T14:05:57.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:05:57.005 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:57.005 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.005 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:57.005 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:57.005 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:57.006 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:05:57.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.006 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:57.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:57.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:57.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:57.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:57.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:05:57.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:57.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:57.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:05:57.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:05:57.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:57.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:05:57.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:05:57.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:57.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:05:57.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:05:57.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:05:57.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:05:57.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:05:57.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:05:57.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:57.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:05:57.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:05:57.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:05:57.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:05:57.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:57.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:05:57.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:05:57.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:57.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:05:57.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:05:57.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:57.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:05:57.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:05:57.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:57.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:05:57.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:05:57.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:57.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:05:57.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:05:57.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:57.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:05:57.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:05:57.015 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.016 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.016 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.016 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:57.016 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:57.016 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 165.654941057, 2023-07-02T14:05:57.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:05:57.017 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:57.017 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.017 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:57.017 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:57.017 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:57.017 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:05:57.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.017 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:57.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:57.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:57.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:57.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:57.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:05:57.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:57.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:57.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:05:57.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:05:57.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:57.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:05:57.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:05:57.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:57.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:05:57.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:05:57.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:05:57.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:05:57.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:05:57.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:05:57.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:57.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:05:57.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:05:57.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:57.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:05:57.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:05:57.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:57.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:05:57.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:05:57.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:57.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:05:57.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:05:57.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:57.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:05:57.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:05:57.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:57.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:05:57.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:05:57.026 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.026 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.026 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.027 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2950 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 1507328~786432] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:57.027 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:57.027 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 165.517664795, 2023-07-02T14:05:57.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999, 2023-07-02T14:05:57.027 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:57.027 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.027 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:57.027 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:57.028 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:57.028 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2950 2023-07-02T14:05:57.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.028 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:57.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:57.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:57.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:57.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:57.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:57.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:57.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:57.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.475466+0000", 2023-07-02T14:05:57.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001147854 2023-07-02T14:05:57.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:57.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476614+0000", 2023-07-02T14:05:57.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.802e-05 2023-07-02T14:05:57.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:57.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476632+0000", 2023-07-02T14:05:57.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1860000000000003e-06 2023-07-02T14:05:57.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476639+0000", 2023-07-02T14:05:57.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061211240000000004 2023-07-02T14:05:57.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482760+0000", 2023-07-02T14:05:57.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8833999999999996e-05 2023-07-02T14:05:57.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:57.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.482859+0000", 2023-07-02T14:05:57.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019270600000000001 2023-07-02T14:05:57.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:57.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.483051+0000", 2023-07-02T14:05:57.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060385630000000003 2023-07-02T14:05:57.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:57.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489090+0000", 2023-07-02T14:05:57.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024506918999999999 2023-07-02T14:05:57.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:57.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513597+0000", 2023-07-02T14:05:57.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9672999999999995e-05 2023-07-02T14:05:57.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:57.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513667+0000", 2023-07-02T14:05:57.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024329500000000001 2023-07-02T14:05:57.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:57.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.513910+0000", 2023-07-02T14:05:57.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038443604999999999 2023-07-02T14:05:57.036 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.036 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.037 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.037 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2952 2.3 2:ca3aa014:::rbd_data.125de1449605.00000000000000fe:head [write 2883584~12288] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:05:57.037 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:57.037 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 165.516370182, 2023-07-02T14:05:57.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651, 2023-07-02T14:05:57.037 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:57.038 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.038 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:57.038 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:57.038 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:57.038 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2952 2023-07-02T14:05:57.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.038 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:57.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:57.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:57.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:57.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:57.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476761+0000", 2023-07-02T14:05:57.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:57.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:57.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476760+0000", 2023-07-02T14:05:57.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3664e-05 2023-07-02T14:05:57.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:57.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476774+0000", 2023-07-02T14:05:57.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.166e-06 2023-07-02T14:05:57.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:57.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476783+0000", 2023-07-02T14:05:57.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8720000000000001e-06 2023-07-02T14:05:57.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.476788+0000", 2023-07-02T14:05:57.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087180529999999999 2023-07-02T14:05:57.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485506+0000", 2023-07-02T14:05:57.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5680999999999994e-05 2023-07-02T14:05:57.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:57.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485602+0000", 2023-07-02T14:05:57.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000222365 2023-07-02T14:05:57.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:57.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.485824+0000", 2023-07-02T14:05:57.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003277379 2023-07-02T14:05:57.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:57.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.489101+0000", 2023-07-02T14:05:57.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025151739999999999 2023-07-02T14:05:57.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:57.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514253+0000", 2023-07-02T14:05:57.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4749000000000002e-05 2023-07-02T14:05:57.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:57.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514288+0000", 2023-07-02T14:05:57.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000130634 2023-07-02T14:05:57.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:57.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.514419+0000", 2023-07-02T14:05:57.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037657651 2023-07-02T14:05:57.047 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.047 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.047 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.047 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9620 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 589824~65536] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:05:57.048 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:57.048 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.970926458999998, 2023-07-02T14:05:57.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001, 2023-07-02T14:05:57.048 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:57.048 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.048 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:57.048 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:57.048 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:57.049 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9620 2023-07-02T14:05:57.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.049 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:57.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.049 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:57.049 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:57.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:57.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:57.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:05:57.050 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:05:57.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.050 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.050 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:57.050 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022203+0000", 2023-07-02T14:05:57.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0346000000000001e-05 2023-07-02T14:05:57.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:57.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022263+0000", 2023-07-02T14:05:57.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4657999999999999e-05 2023-07-02T14:05:57.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:57.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022278+0000", 2023-07-02T14:05:57.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:05:57.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022285+0000", 2023-07-02T14:05:57.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0096000000000003e-05 2023-07-02T14:05:57.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022335+0000", 2023-07-02T14:05:57.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2334000000000001e-05 2023-07-02T14:05:57.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:57.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022347+0000", 2023-07-02T14:05:57.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095653701999999993 2023-07-02T14:05:57.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118001+0000", 2023-07-02T14:05:57.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000185797 2023-07-02T14:05:57.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:57.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118187+0000", 2023-07-02T14:05:57.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026397900000000001 2023-07-02T14:05:57.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:57.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118451+0000", 2023-07-02T14:05:57.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051832800000000002 2023-07-02T14:05:57.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:57.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.123634+0000", 2023-07-02T14:05:57.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001227332 2023-07-02T14:05:57.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:57.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124861+0000", 2023-07-02T14:05:57.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9493999999999998e-05 2023-07-02T14:05:57.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.057 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:57.057 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124901+0000", 2023-07-02T14:05:57.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0462000000000001e-05 2023-07-02T14:05:57.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:57.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124931+0000", 2023-07-02T14:05:57.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001 2023-07-02T14:05:57.058 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.058 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.059 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.059 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9621 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 655360~1966080] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:05:57.059 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:57.059 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 91.970739077000005, 2023-07-02T14:05:57.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999, 2023-07-02T14:05:57.060 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:57.060 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.060 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:57.060 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:57.060 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:57.060 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9621 2023-07-02T14:05:57.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.060 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:57.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:57.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:57.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:57.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:57.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:05:57.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:05:57.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:57.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022391+0000", 2023-07-02T14:05:57.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032053939999999999 2023-07-02T14:05:57.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:57.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025597+0000", 2023-07-02T14:05:57.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4307999999999999e-05 2023-07-02T14:05:57.063 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.063 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.063 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:57.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025611+0000", 2023-07-02T14:05:57.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.032e-06 2023-07-02T14:05:57.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.064 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.064 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025617+0000", 2023-07-02T14:05:57.064 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8977999999999998e-05 2023-07-02T14:05:57.064 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.064 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025656+0000", 2023-07-02T14:05:57.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:05:57.065 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.065 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.065 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:05:57.065 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025663+0000", 2023-07-02T14:05:57.065 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095220739999999998 2023-07-02T14:05:57.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.066 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120884+0000", 2023-07-02T14:05:57.066 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2525000000000001e-05 2023-07-02T14:05:57.066 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.066 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.066 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:57.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120956+0000", 2023-07-02T14:05:57.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016232400000000001 2023-07-02T14:05:57.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.067 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.067 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:57.067 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.121119+0000", 2023-07-02T14:05:57.067 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044007380000000004 2023-07-02T14:05:57.067 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:57.068 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.125519+0000", 2023-07-02T14:05:57.068 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000797209 2023-07-02T14:05:57.068 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.068 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.068 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:57.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126317+0000", 2023-07-02T14:05:57.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1228000000000004e-05 2023-07-02T14:05:57.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.069 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.069 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:57.069 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126378+0000", 2023-07-02T14:05:57.069 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978000000000003e-05 2023-07-02T14:05:57.069 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.070 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.070 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:57.070 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126430+0000", 2023-07-02T14:05:57.070 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999 2023-07-02T14:05:57.070 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.070 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.070 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.070 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.071 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.071 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:189 2.3 2:c9621172:::rbd_data.125de1449605.00000000000000c0:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:57.071 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:57.071 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.820451785000003, 2023-07-02T14:05:57.071 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001, 2023-07-02T14:05:57.071 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:57.071 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:57.071 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:57.072 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:57.072 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:57.072 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:05:57.072 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.072 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:57.072 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.072 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:57.072 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:57.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:57.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.073 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:57.073 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:57.073 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:57.073 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.073 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:57.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:05:57.074 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2023-07-02T14:05:57.074 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.074 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.074 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:57.074 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172682+0000", 2023-07-02T14:05:57.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0159999999999997e-06 2023-07-02T14:05:57.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.075 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:57.075 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172688+0000", 2023-07-02T14:05:57.075 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0010000000000003e-06 2023-07-02T14:05:57.075 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.075 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172692+0000", 2023-07-02T14:05:57.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026444333 2023-07-02T14:05:57.076 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.076 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.076 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.076 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199136+0000", 2023-07-02T14:05:57.076 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2230000000000004e-05 2023-07-02T14:05:57.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:57.077 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199218+0000", 2023-07-02T14:05:57.077 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012043595000000001 2023-07-02T14:05:57.077 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.077 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.077 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:57.078 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211262+0000", 2023-07-02T14:05:57.078 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001 2023-07-02T14:05:57.078 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.078 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.078 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.078 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.078 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.078 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:199 2.1 2:985b1fe8:::rbd_data.125de1449605.00000000000000c6:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:57.079 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:57.079 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.820408241999999, 2023-07-02T14:05:57.079 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002, 2023-07-02T14:05:57.079 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:57.079 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:57.079 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:57.079 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:57.079 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:57.080 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 199 2023-07-02T14:05:57.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.080 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:57.080 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.080 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:57.080 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:57.080 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:57.080 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:57.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172723+0000", 2023-07-02T14:05:57.081 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:57.081 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.081 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.081 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:57.081 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172722+0000", 2023-07-02T14:05:57.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1250000000000001e-06 2023-07-02T14:05:57.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.082 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.082 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:57.082 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172725+0000", 2023-07-02T14:05:57.082 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5670000000000002e-06 2023-07-02T14:05:57.082 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:57.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172731+0000", 2023-07-02T14:05:57.083 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9310000000000001e-06 2023-07-02T14:05:57.083 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.083 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.083 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.083 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172735+0000", 2023-07-02T14:05:57.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030699852999999999 2023-07-02T14:05:57.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.084 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.084 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203435+0000", 2023-07-02T14:05:57.084 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6827e-05 2023-07-02T14:05:57.084 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.084 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:57.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.203532+0000", 2023-07-02T14:05:57.085 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076955230000000001 2023-07-02T14:05:57.085 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.085 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.085 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:57.085 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211227+0000", 2023-07-02T14:05:57.086 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038504280000000002 2023-07-02T14:05:57.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.086 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.086 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.086 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.086 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.086 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:281 2.1 2:901d5a95:::rbd_data.125de1449605.0000000000000118:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:05:57.086 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:57.087 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.437662966000005, 2023-07-02T14:05:57.087 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002, 2023-07-02T14:05:57.087 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:57.087 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:57.087 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:57.087 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:05:57.087 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:05:57.087 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:05:57.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.088 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:57.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.088 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:57.088 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:57.088 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:57.088 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.088 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:57.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:57.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:57.089 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.089 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.089 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:57.089 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:05:57.089 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0469999999999998e-06 2023-07-02T14:05:57.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:57.090 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555471+0000", 2023-07-02T14:05:57.090 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8860000000000001e-06 2023-07-02T14:05:57.090 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.090 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.090 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:57.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555476+0000", 2023-07-02T14:05:57.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3160000000000002e-06 2023-07-02T14:05:57.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.091 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.091 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.091 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555483+0000", 2023-07-02T14:05:57.091 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989525000000001 2023-07-02T14:05:57.091 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.092 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591472+0000", 2023-07-02T14:05:57.092 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3708999999999996e-05 2023-07-02T14:05:57.092 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.092 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.092 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:57.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591536+0000", 2023-07-02T14:05:57.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055109720000000003 2023-07-02T14:05:57.093 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.093 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.093 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:57.093 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.597047+0000", 2023-07-02T14:05:57.093 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002 2023-07-02T14:05:57.094 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.094 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.094 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.094 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.094 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.094 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12883 2.1 2:9ffc48e1:::rbd_data.125de1449605.0000000000000074:head [write 4173824~20480] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:05:57.094 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:57.094 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.347060493000001, 2023-07-02T14:05:57.095 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037871393000000003, 2023-07-02T14:05:57.095 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:57.095 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.095 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:57.095 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:57.095 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:57.095 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12883 2023-07-02T14:05:57.095 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.096 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:57.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:57.096 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:57.096 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:57.096 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.096 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.096 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:57.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646071+0000", 2023-07-02T14:05:57.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999971 2023-07-02T14:05:57.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.097 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.097 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:57.097 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646068+0000", 2023-07-02T14:05:57.097 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4935999999999999e-05 2023-07-02T14:05:57.097 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:57.098 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646103+0000", 2023-07-02T14:05:57.098 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.092e-05 2023-07-02T14:05:57.098 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.098 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.098 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:57.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646124+0000", 2023-07-02T14:05:57.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1519999999999998e-06 2023-07-02T14:05:57.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.099 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.099 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.099 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646133+0000", 2023-07-02T14:05:57.099 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012270200000000001 2023-07-02T14:05:57.099 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646255+0000", 2023-07-02T14:05:57.100 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014495400000000001 2023-07-02T14:05:57.100 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.100 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.100 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:57.100 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646400+0000", 2023-07-02T14:05:57.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000221626 2023-07-02T14:05:57.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.101 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:57.101 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.646622+0000", 2023-07-02T14:05:57.101 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0078929050000000004 2023-07-02T14:05:57.101 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.101 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:57.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.654515+0000", 2023-07-02T14:05:57.102 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.029307343999999999 2023-07-02T14:05:57.102 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.102 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.102 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:57.102 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683822+0000", 2023-07-02T14:05:57.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8298000000000002e-05 2023-07-02T14:05:57.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.103 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:57.103 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683891+0000", 2023-07-02T14:05:57.103 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1381999999999997e-05 2023-07-02T14:05:57.103 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.103 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.104 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:57.104 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.683942+0000", 2023-07-02T14:05:57.104 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037871393000000003 2023-07-02T14:05:57.104 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.104 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.104 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.104 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.104 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.105 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12985 2.7 2:eeddf24b:::rbd_data.125de1449605.000000000000007e:head [write 65536~4128768] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:05:57.105 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:57.105 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.210136019999993, 2023-07-02T14:05:57.105 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999, 2023-07-02T14:05:57.105 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:57.105 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.105 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:57.105 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:57.106 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:57.106 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12985 2023-07-02T14:05:57.106 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.106 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:57.106 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.106 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:57.106 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:57.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:57.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.107 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:57.107 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:05:57.107 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:57.107 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.107 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:57.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782994+0000", 2023-07-02T14:05:57.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019254741999999998 2023-07-02T14:05:57.108 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.108 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.108 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:57.108 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802248+0000", 2023-07-02T14:05:57.108 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0451e-05 2023-07-02T14:05:57.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:57.109 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802279+0000", 2023-07-02T14:05:57.109 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3318e-05 2023-07-02T14:05:57.109 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.109 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.109 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802292+0000", 2023-07-02T14:05:57.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109403 2023-07-02T14:05:57.110 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.110 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.110 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.110 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802402+0000", 2023-07-02T14:05:57.110 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012520599999999999 2023-07-02T14:05:57.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:57.111 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802527+0000", 2023-07-02T14:05:57.111 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187768 2023-07-02T14:05:57.111 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.111 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.111 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:05:57.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802715+0000", 2023-07-02T14:05:57.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013192822 2023-07-02T14:05:57.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.112 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.112 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:57.112 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.815907+0000", 2023-07-02T14:05:57.112 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0086451870000000004 2023-07-02T14:05:57.112 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:57.113 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824553+0000", 2023-07-02T14:05:57.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6759999999999999e-05 2023-07-02T14:05:57.113 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.113 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:57.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824609+0000", 2023-07-02T14:05:57.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011314700000000001 2023-07-02T14:05:57.114 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:57.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824722+0000", 2023-07-02T14:05:57.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999 2023-07-02T14:05:57.114 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.115 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.115 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.115 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:165 2.3 2:c9c6393a:::rbd_data.125de1449605.000000000000002c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:57.115 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:57.115 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 60.153353203000002, 2023-07-02T14:05:57.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002, 2023-07-02T14:05:57.116 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:57.116 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:57.116 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:57.116 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:57.116 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:57.116 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 165 2023-07-02T14:05:57.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.116 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:57.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:57.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:57.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:57.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:57.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:05:57.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:57.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:57.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839777+0000", 2023-07-02T14:05:57.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6740000000000001e-06 2023-07-02T14:05:57.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:57.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839780+0000", 2023-07-02T14:05:57.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9030000000000004e-06 2023-07-02T14:05:57.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:57.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839788+0000", 2023-07-02T14:05:57.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1799999999999998e-06 2023-07-02T14:05:57.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839792+0000", 2023-07-02T14:05:57.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.031592227000000001 2023-07-02T14:05:57.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871384+0000", 2023-07-02T14:05:57.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013600099999999999 2023-07-02T14:05:57.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:57.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871520+0000", 2023-07-02T14:05:57.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082817089999999999 2023-07-02T14:05:57.122 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:57.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.879802+0000", 2023-07-02T14:05:57.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002 2023-07-02T14:05:57.122 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.123 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.123 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.123 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:252 2.7 2:f1d0abf0:::rbd_data.125de1449605.0000000000000083:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:57.123 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:57.123 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.737068233000002, 2023-07-02T14:05:57.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997, 2023-07-02T14:05:57.124 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:57.124 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:57.124 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:57.124 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:57.124 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:57.124 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 252 2023-07-02T14:05:57.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.125 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:57.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:57.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:57.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:57.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:57.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:05:57.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:57.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:57.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256062+0000", 2023-07-02T14:05:57.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0149999999999999e-06 2023-07-02T14:05:57.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:57.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256065+0000", 2023-07-02T14:05:57.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5269999999999998e-06 2023-07-02T14:05:57.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:57.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256071+0000", 2023-07-02T14:05:57.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:05:57.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256074+0000", 2023-07-02T14:05:57.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028852909 2023-07-02T14:05:57.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.284927+0000", 2023-07-02T14:05:57.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168274 2023-07-02T14:05:57.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:57.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.285096+0000", 2023-07-02T14:05:57.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011036300000000001 2023-07-02T14:05:57.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:57.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296132+0000", 2023-07-02T14:05:57.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997 2023-07-02T14:05:57.130 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.130 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.131 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.131 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:261 2.1 2:971731cb:::rbd_data.125de1449605.000000000000008c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:57.131 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:57.131 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.736972758, 2023-07-02T14:05:57.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984, 2023-07-02T14:05:57.131 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:57.132 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:57.132 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:57.132 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:57.132 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:57.132 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 261 2023-07-02T14:05:57.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.132 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:57.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:57.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:57.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:57.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:57.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:57.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:57.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:57.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:05:57.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2819999999999999e-06 2023-07-02T14:05:57.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:57.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256160+0000", 2023-07-02T14:05:57.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2020000000000003e-06 2023-07-02T14:05:57.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:57.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256165+0000", 2023-07-02T14:05:57.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2100000000000003e-06 2023-07-02T14:05:57.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256170+0000", 2023-07-02T14:05:57.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025871715999999999 2023-07-02T14:05:57.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282041+0000", 2023-07-02T14:05:57.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132364 2023-07-02T14:05:57.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:57.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282174+0000", 2023-07-02T14:05:57.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014044566 2023-07-02T14:05:57.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:57.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296218+0000", 2023-07-02T14:05:57.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984 2023-07-02T14:05:57.140 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.140 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.140 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.141 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:264 2.3 2:dbf06c9d:::rbd_data.125de1449605.000000000000008f:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:57.141 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:57.141 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.736938639999998, 2023-07-02T14:05:57.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003, 2023-07-02T14:05:57.142 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:57.142 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:57.142 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:57.142 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:57.142 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:57.142 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 264 2023-07-02T14:05:57.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.143 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:57.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:57.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:57.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:57.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:57.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:57.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:57.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:57.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:05:57.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5749999999999999e-06 2023-07-02T14:05:57.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:57.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256195+0000", 2023-07-02T14:05:57.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.327e-06 2023-07-02T14:05:57.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:57.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256200+0000", 2023-07-02T14:05:57.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0580000000000002e-06 2023-07-02T14:05:57.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256203+0000", 2023-07-02T14:05:57.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035707524999999997 2023-07-02T14:05:57.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.291911+0000", 2023-07-02T14:05:57.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014269799999999999 2023-07-02T14:05:57.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:57.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.292053+0000", 2023-07-02T14:05:57.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062760309999999996 2023-07-02T14:05:57.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:57.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.298329+0000", 2023-07-02T14:05:57.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003 2023-07-02T14:05:57.149 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.149 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.149 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.149 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:341 2.3 2:cee258bd:::rbd_data.125de1449605.00000000000000dc:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:05:57.149 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:57.150 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 59.356451313000001, 2023-07-02T14:05:57.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997, 2023-07-02T14:05:57.150 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:57.150 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:05:57.150 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:57.150 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:05:57.150 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:05:57.150 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 341 2023-07-02T14:05:57.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.151 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:57.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:57.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:57.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:57.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:57.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:05:57.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:57.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:57.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636679+0000", 2023-07-02T14:05:57.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6240000000000002e-06 2023-07-02T14:05:57.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:57.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636682+0000", 2023-07-02T14:05:57.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6910000000000002e-06 2023-07-02T14:05:57.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:57.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636688+0000", 2023-07-02T14:05:57.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3469999999999999e-06 2023-07-02T14:05:57.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636691+0000", 2023-07-02T14:05:57.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035625560000000001 2023-07-02T14:05:57.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672317+0000", 2023-07-02T14:05:57.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108391 2023-07-02T14:05:57.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:57.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672425+0000", 2023-07-02T14:05:57.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012104976 2023-07-02T14:05:57.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:57.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.684530+0000", 2023-07-02T14:05:57.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997 2023-07-02T14:05:57.157 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.157 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.157 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.157 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPushReply(2.7 139/134 [PushReplyOp(2:e3782f39:::rbd_data.125de1449605.0000000000000120:head)])", 2023-07-02T14:05:57.157 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:05:57.157 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.06591916, 2023-07-02T14:05:57.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998, 2023-07-02T14:05:57.158 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:57.158 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:05:57.158 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:57.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:57.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:05:57.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:57.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:57.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:05:57.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:57.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:57.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927211+0000", 2023-07-02T14:05:57.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3689999999999998e-06 2023-07-02T14:05:57.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:57.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927217+0000", 2023-07-02T14:05:57.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7269999999999996e-06 2023-07-02T14:05:57.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:57.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927225+0000", 2023-07-02T14:05:57.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:05:57.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927230+0000", 2023-07-02T14:05:57.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:05:57.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927266+0000", 2023-07-02T14:05:57.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.069972871000000006 2023-07-02T14:05:57.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:57.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.997239+0000", 2023-07-02T14:05:57.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998 2023-07-02T14:05:57.163 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.163 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.163 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.164 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:17593 2.3 2:dd618512:::rbd_data.125de1449605.000000000000008b:head [write 65536~3932160] snapc 2e=[2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:57.164 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:43.549480+0000", 2023-07-02T14:05:57.164 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.443651569, 2023-07-02T14:05:57.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.03787782, 2023-07-02T14:05:57.164 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:57.164 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.164 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:57.165 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:57.165 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:57.165 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17593 2023-07-02T14:05:57.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.165 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:57.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:57.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.549480+0000", 2023-07-02T14:05:57.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:57.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:57.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.549480+0000", 2023-07-02T14:05:57.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999943 2023-07-02T14:05:57.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:57.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.549474+0000", 2023-07-02T14:05:57.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049631190000000002 2023-07-02T14:05:57.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:57.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.554437+0000", 2023-07-02T14:05:57.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7652e-05 2023-07-02T14:05:57.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:57.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.554455+0000", 2023-07-02T14:05:57.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4089999999999997e-06 2023-07-02T14:05:57.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.554462+0000", 2023-07-02T14:05:57.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1801000000000002e-05 2023-07-02T14:05:57.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.554504+0000", 2023-07-02T14:05:57.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000125489 2023-07-02T14:05:57.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:57.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.554629+0000", 2023-07-02T14:05:57.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000178903 2023-07-02T14:05:57.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:57.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.554808+0000", 2023-07-02T14:05:57.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0098879320000000003 2023-07-02T14:05:57.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:57.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.564696+0000", 2023-07-02T14:05:57.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022554149999999999 2023-07-02T14:05:57.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:57.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.587250+0000", 2023-07-02T14:05:57.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7098000000000001e-05 2023-07-02T14:05:57.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:57.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.587308+0000", 2023-07-02T14:05:57.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9795e-05 2023-07-02T14:05:57.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:57.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.587357+0000", 2023-07-02T14:05:57.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.03787782 2023-07-02T14:05:57.173 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.174 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.174 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.174 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:17596 2.1 2:965e04ab:::rbd_data.125de1449605.0000000000000089:head [write 65536~3997696] snapc 2e=[2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e161)", 2023-07-02T14:05:57.174 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:43.554508+0000", 2023-07-02T14:05:57.174 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.438622649999999, 2023-07-02T14:05:57.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037965229000000003, 2023-07-02T14:05:57.175 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:05:57.175 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.175 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:05:57.175 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:05:57.175 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:57.175 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17596 2023-07-02T14:05:57.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.176 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:05:57.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:05:57.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.554508+0000", 2023-07-02T14:05:57.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:05:57.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:05:57.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.554508+0000", 2023-07-02T14:05:57.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:57.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:05:57.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.554508+0000", 2023-07-02T14:05:57.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0054068659999999998 2023-07-02T14:05:57.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:05:57.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.559915+0000", 2023-07-02T14:05:57.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1087999999999999e-05 2023-07-02T14:05:57.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:05:57.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.559936+0000", 2023-07-02T14:05:57.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0188e-05 2023-07-02T14:05:57.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.559946+0000", 2023-07-02T14:05:57.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9580999999999998e-05 2023-07-02T14:05:57.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:05:57.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.559996+0000", 2023-07-02T14:05:57.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0260999999999995e-05 2023-07-02T14:05:57.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:05:57.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.560086+0000", 2023-07-02T14:05:57.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012091699999999999 2023-07-02T14:05:57.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:05:57.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.560207+0000", 2023-07-02T14:05:57.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021177420999999998 2023-07-02T14:05:57.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:05:57.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.581384+0000", 2023-07-02T14:05:57.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010973284 2023-07-02T14:05:57.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:57.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.592357+0000", 2023-07-02T14:05:57.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1208e-05 2023-07-02T14:05:57.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:05:57.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.592429+0000", 2023-07-02T14:05:57.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4962000000000001e-05 2023-07-02T14:05:57.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:05:57.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:05:57.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:05:57.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:43.592474+0000", 2023-07-02T14:05:57.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.037965229000000003 2023-07-02T14:05:57.184 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.184 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.184 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.184 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:05:57.185 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:05:57.185 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:05:57.186 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 100% complete...done. 2023-07-02T14:05:57.187 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:05:57.187 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=46 2023-07-02T14:05:57.187 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap46 2023-07-02T14:05:57.187 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:05:57.188 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.0 to 1 (10000) 2023-07-02T14:05:57.188 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:05:57.189 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-07-02T14:05:57.190 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:57.185+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:57.287 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:57.286+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:57.325 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:57.326 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:57.326 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:57.326 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:57.336 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:05:57.388 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:57.387+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:05:57.489 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:05:57.488+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:05:57.500 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:57.500 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:57.500 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:57.501 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:57.501 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:57.510 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:05:57.590 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:57.589+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:05:57.676 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:57.677 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:57.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:57.677 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:57.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.677 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:05:57.677 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:57.678 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.77776442800001, 2023-07-02T14:05:57.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:05:57.678 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:57.678 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.678 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:57.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:57.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:57.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:57.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:57.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:05:57.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:57.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:57.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:05:57.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:05:57.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:57.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:05:57.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:05:57.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:57.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:05:57.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:05:57.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:05:57.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:05:57.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:57.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:05:57.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:05:57.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:57.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:05:57.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:05:57.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:57.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:05:57.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:05:57.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:57.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:05:57.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:05:57.685 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.685 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.685 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.686 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:05:57.686 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:57.686 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.77367871000001, 2023-07-02T14:05:57.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:05:57.686 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:57.686 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.686 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:57.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:57.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:57.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:57.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:57.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:05:57.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:57.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:57.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:05:57.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:05:57.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:57.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:05:57.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:05:57.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:57.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:05:57.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:05:57.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:05:57.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:05:57.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:57.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:05:57.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:05:57.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:57.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:05:57.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:05:57.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:57.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:05:57.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:05:57.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:57.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:05:57.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:05:57.693 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.693 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.693 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.694 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:05:57.694 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:57.694 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.76981392900001, 2023-07-02T14:05:57.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:05:57.694 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:57.694 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.694 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:57.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:57.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:57.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:57.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:57.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:05:57.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:57.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:57.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:05:57.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:05:57.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:57.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:05:57.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:05:57.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:57.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:05:57.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:05:57.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:05:57.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:05:57.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:57.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:05:57.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:05:57.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:57.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:05:57.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:05:57.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:57.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:05:57.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:05:57.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:57.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:05:57.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:05:57.701 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.701 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.701 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.702 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:05:57.702 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:57.702 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.76628966800001, 2023-07-02T14:05:57.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:05:57.702 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:57.702 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.703 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:57.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:57.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:57.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:57.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:57.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:05:57.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:57.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:57.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:05:57.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:05:57.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:57.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:05:57.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:05:57.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:57.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:05:57.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:05:57.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:05:57.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:05:57.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:57.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:05:57.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:05:57.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:57.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:05:57.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:05:57.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:57.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:05:57.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:05:57.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:57.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:05:57.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:05:57.709 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.709 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.709 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.710 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:05:57.710 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:57.710 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.73058486900001, 2023-07-02T14:05:57.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:05:57.710 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:57.710 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.710 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:57.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:57.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:57.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:57.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:57.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:05:57.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:57.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:57.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:05:57.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:05:57.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:57.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:05:57.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:05:57.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:57.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:05:57.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:05:57.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:05:57.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:05:57.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:57.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:05:57.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:05:57.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:57.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:05:57.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:05:57.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:57.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:05:57.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:05:57.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:57.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:05:57.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:05:57.717 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.717 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.717 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.718 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:05:57.718 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:57.718 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.72592751100001, 2023-07-02T14:05:57.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:05:57.718 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:57.718 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.718 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:57.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:57.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:57.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:57.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:57.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:05:57.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:57.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:57.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:05:57.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:05:57.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:57.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:05:57.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:05:57.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:57.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:05:57.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:05:57.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:05:57.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:05:57.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:57.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:05:57.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:05:57.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:57.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:05:57.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:05:57.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:57.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:05:57.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:05:57.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:57.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:05:57.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:05:57.725 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.725 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.725 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.726 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:05:57.726 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:57.726 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.724987648, 2023-07-02T14:05:57.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:05:57.726 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:57.726 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.726 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:57.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:57.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:57.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:57.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:57.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:05:57.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:57.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:57.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:05:57.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:05:57.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:57.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:05:57.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:05:57.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:57.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:05:57.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:05:57.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:05:57.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:05:57.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:57.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:05:57.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:05:57.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:57.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:05:57.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:05:57.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:57.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:05:57.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:05:57.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:57.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:05:57.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:05:57.733 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.733 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.733 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.733 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:05:57.734 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:57.734 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 137.72358976999999, 2023-07-02T14:05:57.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:05:57.734 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:57.734 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.734 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:57.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:57.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:57.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:57.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:57.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:05:57.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:57.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:57.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:05:57.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:05:57.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:57.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:05:57.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:05:57.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:57.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:05:57.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:05:57.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:05:57.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:05:57.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:57.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:05:57.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:05:57.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:57.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:05:57.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:05:57.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:57.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:05:57.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:05:57.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:57.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:05:57.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:05:57.741 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.741 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.741 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.741 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:05:57.741 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:57.741 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 116.31339927400001, 2023-07-02T14:05:57.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:05:57.742 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:57.742 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.742 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:57.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:57.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:57.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:57.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:57.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:05:57.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:57.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:57.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:05:57.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:05:57.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:57.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:05:57.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:05:57.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:57.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:05:57.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:05:57.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:05:57.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:05:57.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:57.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:05:57.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:05:57.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:57.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:05:57.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:05:57.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:57.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:05:57.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:05:57.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:57.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:05:57.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:05:57.749 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.749 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.749 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.749 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:57.749 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:57.749 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 114.94704959000001, 2023-07-02T14:05:57.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:05:57.750 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:57.750 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.750 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:57.750 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:57.750 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:57.750 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:05:57.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.751 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:57.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:57.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:57.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:57.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:57.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:05:57.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:57.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:57.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:05:57.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:05:57.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:57.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:05:57.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:05:57.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:57.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:05:57.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:05:57.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:05:57.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:05:57.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:57.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:05:57.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:05:57.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:57.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:05:57.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:05:57.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:57.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:05:57.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:05:57.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:57.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:05:57.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:05:57.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:57.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:05:57.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:05:57.758 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.758 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.758 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.759 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7707 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 3145728~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:57.759 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:57.759 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 114.94306612299999, 2023-07-02T14:05:57.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999, 2023-07-02T14:05:57.759 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:57.759 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.759 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:57.759 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:57.760 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:57.760 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7707 2023-07-02T14:05:57.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.760 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:57.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:57.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:57.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:57.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:57.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:57.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:57.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:57.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:05:57.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5017e-05 2023-07-02T14:05:57.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:57.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732347+0000", 2023-07-02T14:05:57.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5821e-05 2023-07-02T14:05:57.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:57.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732363+0000", 2023-07-02T14:05:57.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1270000000000002e-06 2023-07-02T14:05:57.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732368+0000", 2023-07-02T14:05:57.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008444666 2023-07-02T14:05:57.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:57.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740812+0000", 2023-07-02T14:05:57.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011266599999999999 2023-07-02T14:05:57.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:57.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740925+0000", 2023-07-02T14:05:57.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010848808999999999 2023-07-02T14:05:57.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:57.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751774+0000", 2023-07-02T14:05:57.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8421999999999997e-05 2023-07-02T14:05:57.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:57.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751832+0000", 2023-07-02T14:05:57.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107453 2023-07-02T14:05:57.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:57.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751940+0000", 2023-07-02T14:05:57.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999 2023-07-02T14:05:57.767 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.767 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.767 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.768 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:57.768 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:57.768 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 114.94300224600001, 2023-07-02T14:05:57.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:05:57.768 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:57.768 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.769 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:57.769 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:57.769 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:57.769 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:05:57.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.769 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:57.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:57.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:57.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:57.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:57.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:57.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:57.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:57.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:05:57.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:05:57.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:57.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:05:57.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:05:57.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:57.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:05:57.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:05:57.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:05:57.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:05:57.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:57.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:05:57.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:05:57.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:57.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:05:57.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:05:57.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:57.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:05:57.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:05:57.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:57.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:05:57.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:05:57.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:57.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:05:57.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:05:57.776 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.776 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.777 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.777 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:57.777 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:57.777 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 114.938438421, 2023-07-02T14:05:57.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:05:57.777 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:57.778 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.778 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:57.778 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:57.778 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:57.778 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:05:57.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.778 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:57.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:57.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:57.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:57.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:57.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:05:57.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:05:57.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:57.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:05:57.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:05:57.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:57.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:05:57.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:05:57.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:57.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:05:57.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:05:57.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:05:57.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:05:57.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:57.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:05:57.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:05:57.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:57.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:05:57.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:05:57.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:57.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:05:57.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:05:57.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:57.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:05:57.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:05:57.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:57.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:05:57.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:05:57.786 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.786 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.786 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.786 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:57.787 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:57.787 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 114.938388555, 2023-07-02T14:05:57.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:05:57.787 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:57.787 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.787 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:57.787 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:57.787 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:57.788 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:05:57.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.788 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:57.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:57.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:57.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:57.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:57.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:05:57.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:57.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:57.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:05:57.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:05:57.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:57.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:05:57.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:05:57.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:57.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:05:57.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:05:57.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:05:57.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:05:57.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:57.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:05:57.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:05:57.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:57.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:05:57.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:05:57.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:57.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:05:57.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:05:57.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:57.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:05:57.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:05:57.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:57.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:05:57.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:05:57.795 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.795 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.795 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.796 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:57.796 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:57.796 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 114.93734956199999, 2023-07-02T14:05:57.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:05:57.796 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:57.796 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.796 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:57.796 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:57.797 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:57.797 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:05:57.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.797 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:57.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:57.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:57.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:57.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:57.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:05:57.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:57.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:57.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:05:57.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:05:57.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:57.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:05:57.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:05:57.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:57.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:05:57.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:05:57.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:05:57.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:05:57.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:57.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:05:57.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:05:57.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:57.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:05:57.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:05:57.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:57.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:05:57.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:05:57.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:57.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:05:57.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:05:57.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:57.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:05:57.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:05:57.804 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.804 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.804 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.805 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7993 2.7 2:fe72db53:::rbd_data.125de1449605.0000000000000072:head [write 65536~3276800] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:05:57.805 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:57.805 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 114.750391514, 2023-07-02T14:05:57.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919, 2023-07-02T14:05:57.805 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:57.805 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.805 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:57.806 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:57.806 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:57.806 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7993 2023-07-02T14:05:57.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.806 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:57.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:57.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:57.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:57.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:57.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:05:57.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:57.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:57.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924986+0000", 2023-07-02T14:05:57.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049542070000000004 2023-07-02T14:05:57.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:57.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929940+0000", 2023-07-02T14:05:57.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9740000000000001e-05 2023-07-02T14:05:57.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:57.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929960+0000", 2023-07-02T14:05:57.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.311e-05 2023-07-02T14:05:57.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929973+0000", 2023-07-02T14:05:57.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9629000000000002e-05 2023-07-02T14:05:57.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:57.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930023+0000", 2023-07-02T14:05:57.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012751799999999999 2023-07-02T14:05:57.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:57.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930150+0000", 2023-07-02T14:05:57.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016449042000000001 2023-07-02T14:05:57.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:57.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946599+0000", 2023-07-02T14:05:57.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3241e-05 2023-07-02T14:05:57.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:57.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946663+0000", 2023-07-02T14:05:57.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000102296 2023-07-02T14:05:57.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:57.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946765+0000", 2023-07-02T14:05:57.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919 2023-07-02T14:05:57.813 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.813 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.813 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.814 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11782 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 196608~2162688] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:05:57.814 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:57.814 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 73.265871591999996, 2023-07-02T14:05:57.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365, 2023-07-02T14:05:57.814 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:57.814 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.815 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:57.815 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:57.815 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:57.815 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11782 2023-07-02T14:05:57.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.815 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:57.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:57.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:57.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:57.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:57.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:05:57.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:57.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:57.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409506+0000", 2023-07-02T14:05:57.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020172269999999999 2023-07-02T14:05:57.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:57.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411523+0000", 2023-07-02T14:05:57.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8901000000000001e-05 2023-07-02T14:05:57.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:57.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411542+0000", 2023-07-02T14:05:57.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5037e-05 2023-07-02T14:05:57.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411557+0000", 2023-07-02T14:05:57.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001931213 2023-07-02T14:05:57.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:57.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413489+0000", 2023-07-02T14:05:57.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000149207 2023-07-02T14:05:57.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:57.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413638+0000", 2023-07-02T14:05:57.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024752800000000001 2023-07-02T14:05:57.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:57.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413885+0000", 2023-07-02T14:05:57.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048516000000000002 2023-07-02T14:05:57.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:57.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.418737+0000", 2023-07-02T14:05:57.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010319259000000001 2023-07-02T14:05:57.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:57.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429056+0000", 2023-07-02T14:05:57.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2518000000000002e-05 2023-07-02T14:05:57.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:57.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429099+0000", 2023-07-02T14:05:57.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015275 2023-07-02T14:05:57.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:57.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429251+0000", 2023-07-02T14:05:57.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365 2023-07-02T14:05:57.824 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.824 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.824 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.824 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPushReply(2.7 119/117 [PushReplyOp(2:eb862bc0:::rbd_data.125de1449605.0000000000000040:13)])", 2023-07-02T14:05:57.824 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:57.825 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 73.264708317, 2023-07-02T14:05:57.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999, 2023-07-02T14:05:57.825 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:57.825 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "reached pg", 2023-07-02T14:05:57.825 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:57.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:57.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:57.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:57.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:57.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:57.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:57.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:57.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:05:57.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7670000000000001e-06 2023-07-02T14:05:57.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:57.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410674+0000", 2023-07-02T14:05:57.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3973e-05 2023-07-02T14:05:57.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:57.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410688+0000", 2023-07-02T14:05:57.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0157e-05 2023-07-02T14:05:57.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410708+0000", 2023-07-02T14:05:57.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018578779 2023-07-02T14:05:57.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:57.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429287+0000", 2023-07-02T14:05:57.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013086549999999999 2023-07-02T14:05:57.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:57.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.430595+0000", 2023-07-02T14:05:57.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999 2023-07-02T14:05:57.830 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.831 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.831 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.831 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11861 2.7 2:e57d3f55:::rbd_data.125de1449605.0000000000000134:head [write 0~65536] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:05:57.831 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:57.831 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 73.13059294, 2023-07-02T14:05:57.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723, 2023-07-02T14:05:57.832 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:57.832 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.832 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:57.832 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:57.832 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:57.832 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11861 2023-07-02T14:05:57.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.832 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:57.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:57.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:57.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:57.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:57.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:05:57.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:57.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:57.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544785+0000", 2023-07-02T14:05:57.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010563700000000001 2023-07-02T14:05:57.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:57.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544890+0000", 2023-07-02T14:05:57.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8530000000000004e-06 2023-07-02T14:05:57.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:57.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544900+0000", 2023-07-02T14:05:57.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.665e-06 2023-07-02T14:05:57.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544906+0000", 2023-07-02T14:05:57.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000272575 2023-07-02T14:05:57.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:57.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545178+0000", 2023-07-02T14:05:57.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000111362 2023-07-02T14:05:57.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:57.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545290+0000", 2023-07-02T14:05:57.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016181100000000001 2023-07-02T14:05:57.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:57.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545452+0000", 2023-07-02T14:05:57.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044500750000000004 2023-07-02T14:05:57.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:57.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.549902+0000", 2023-07-02T14:05:57.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014850024 2023-07-02T14:05:57.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:57.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564752+0000", 2023-07-02T14:05:57.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6272000000000003e-05 2023-07-02T14:05:57.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:57.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564848+0000", 2023-07-02T14:05:57.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000162428 2023-07-02T14:05:57.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:57.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.565010+0000", 2023-07-02T14:05:57.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723 2023-07-02T14:05:57.841 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.841 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.841 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.841 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:13262 2.7 e125/121 2:e0741dde:::rbd_data.125de1449605.00000000000000ad:head v 125'2024, mlcod=125'2021)", 2023-07-02T14:05:57.842 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:57.842 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 64.424515201999995, 2023-07-02T14:05:57.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999, 2023-07-02T14:05:57.842 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:57.842 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:57.842 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:57.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:57.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:57.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:57.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:57.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:05:57.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:57.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:57.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250862+0000", 2023-07-02T14:05:57.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080378800000000007 2023-07-02T14:05:57.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:57.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258900+0000", 2023-07-02T14:05:57.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8939999999999999e-06 2023-07-02T14:05:57.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:57.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258909+0000", 2023-07-02T14:05:57.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3560000000000003e-06 2023-07-02T14:05:57.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:57.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258914+0000", 2023-07-02T14:05:57.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2429000000000003e-05 2023-07-02T14:05:57.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:57.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258956+0000", 2023-07-02T14:05:57.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5094999999999999e-05 2023-07-02T14:05:57.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:57.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258981+0000", 2023-07-02T14:05:57.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012177307 2023-07-02T14:05:57.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:57.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271158+0000", 2023-07-02T14:05:57.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2104999999999999e-05 2023-07-02T14:05:57.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:57.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:57.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:57.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271181+0000", 2023-07-02T14:05:57.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999 2023-07-02T14:05:57.849 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.849 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.849 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.849 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:57.849 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:57.849 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:57.850 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:05:57.851 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:57.849+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:57.951 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:57.950+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:57.976 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:57.976 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:57.976 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:05:57.976 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:57.986 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:05:58.052 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:58.051+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:58.093 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.1 to 1 (10000) 2023-07-02T14:05:58.107 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-07-02T14:05:58.110 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:58.111 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:05:58.111 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:05:58.111 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:05:58.111 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:58.122 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:05:58.159 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:05:58.163 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:58.162+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:58.253 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:58.253+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:58.289 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:05:58.290 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:05:58.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:05:58.290 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:05:58.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.290 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:58.290 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:58.290 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 198.99704135499999, 2023-07-02T14:05:58.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:05:58.291 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:58.291 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:58.291 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:58.291 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:58.291 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:58.291 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:05:58.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.292 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:58.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:58.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:58.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:58.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:58.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:05:58.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:58.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:58.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:05:58.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:05:58.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:58.294 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:05:58.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:05:58.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:58.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:05:58.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:05:58.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.295 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:58.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:05:58.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:05:58.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:58.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:05:58.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:05:58.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:58.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:05:58.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:05:58.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:58.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:05:58.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:05:58.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:58.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:05:58.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:05:58.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:58.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:05:58.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:05:58.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:58.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:05:58.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:05:58.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:58.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:05:58.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:05:58.300 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.300 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.300 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.301 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:58.301 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:58.301 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 198.99680633200001, 2023-07-02T14:05:58.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:05:58.301 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:58.301 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:58.301 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:58.302 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:58.302 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:58.302 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:05:58.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.302 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:58.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:58.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:58.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:58.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:58.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:05:58.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:58.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:58.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:05:58.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:05:58.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:58.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:05:58.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:05:58.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:58.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:05:58.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:05:58.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:58.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:05:58.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:05:58.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:58.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:05:58.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:05:58.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:58.307 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:05:58.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:05:58.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:58.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:05:58.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:05:58.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:58.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:05:58.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:05:58.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:58.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:05:58.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:05:58.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:58.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:05:58.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:05:58.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:58.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:05:58.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:05:58.310 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.311 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.311 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.311 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:58.311 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:58.311 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 198.99660719799999, 2023-07-02T14:05:58.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:05:58.312 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:58.312 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:58.312 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:58.312 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:58.312 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:58.312 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:05:58.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.313 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:58.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:58.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:58.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:58.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:58.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:05:58.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:58.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:58.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:05:58.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:05:58.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:58.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:05:58.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:05:58.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:58.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:05:58.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:05:58.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:58.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:05:58.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:05:58.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:58.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:05:58.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:05:58.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:58.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:05:58.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:05:58.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:58.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:05:58.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:05:58.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:58.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:05:58.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:05:58.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:58.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:05:58.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:05:58.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:58.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:05:58.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:05:58.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:58.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:05:58.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:05:58.321 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.321 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.321 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.322 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:58.322 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:58.322 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 198.99624207400001, 2023-07-02T14:05:58.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:05:58.322 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:58.322 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:58.322 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:58.322 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:58.323 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:58.323 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:05:58.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.323 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:58.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:58.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:58.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:58.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.324 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:58.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:05:58.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:58.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:58.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:05:58.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:05:58.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:58.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:05:58.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:05:58.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:58.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:05:58.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:05:58.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:58.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:05:58.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:05:58.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:58.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:05:58.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:05:58.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:58.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:05:58.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:05:58.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:58.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:05:58.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:05:58.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:58.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:05:58.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:05:58.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:58.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:05:58.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:05:58.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:58.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:05:58.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:05:58.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:58.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:05:58.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:05:58.331 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.331 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.332 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.332 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:58.332 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:58.332 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 198.99616101300001, 2023-07-02T14:05:58.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:05:58.333 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:58.333 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:58.333 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:58.333 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:58.333 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:58.333 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:05:58.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.333 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:58.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:58.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:58.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:58.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:58.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:05:58.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:58.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:58.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:05:58.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:05:58.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:58.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:05:58.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:05:58.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:58.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:05:58.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:05:58.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:58.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:05:58.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:05:58.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:58.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:05:58.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:05:58.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:58.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:05:58.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:05:58.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:58.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:05:58.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:05:58.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:58.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:05:58.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:05:58.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:58.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:05:58.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:05:58.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:58.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:05:58.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:05:58.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:58.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:05:58.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:05:58.342 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.342 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.342 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.342 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:58.343 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:58.343 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 198.996069342, 2023-07-02T14:05:58.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:05:58.343 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:58.343 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:58.343 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:58.343 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:58.343 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:58.344 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:05:58.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.344 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:58.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:58.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:58.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:58.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:58.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:05:58.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:58.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:58.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:05:58.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:05:58.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:58.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:05:58.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:05:58.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:58.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:05:58.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:05:58.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:58.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:05:58.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:05:58.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:58.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:05:58.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:05:58.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:58.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:05:58.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:05:58.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:58.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:05:58.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:05:58.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:58.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:05:58.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:05:58.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:58.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:05:58.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:05:58.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:58.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:05:58.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:05:58.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:58.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:05:58.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:05:58.352 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.352 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.352 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.353 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:58.353 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:58.353 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 198.99572989999999, 2023-07-02T14:05:58.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:05:58.353 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:58.353 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:58.354 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:58.354 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:58.354 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:58.354 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:05:58.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.354 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:58.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:58.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:58.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:58.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:58.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:05:58.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:58.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:58.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:05:58.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:05:58.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:58.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:05:58.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:05:58.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:58.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:05:58.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:05:58.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:58.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:05:58.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:05:58.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:58.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:05:58.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:05:58.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:58.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:05:58.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:05:58.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:58.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:05:58.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:05:58.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:58.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:05:58.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:05:58.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:58.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:05:58.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:05:58.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:58.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:05:58.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:05:58.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:58.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:05:58.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:05:58.363 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.363 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.363 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.363 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:58.363 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:58.364 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 198.99554384699999, 2023-07-02T14:05:58.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:05:58.364 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:58.364 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:58.364 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:58.364 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:58.364 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:58.364 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:05:58.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.365 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:58.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:58.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:58.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:58.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:58.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:58.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:58.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:58.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:05:58.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:05:58.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:58.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:05:58.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:05:58.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:58.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:05:58.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:05:58.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:58.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:05:58.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:05:58.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:58.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:05:58.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:05:58.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:58.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:05:58.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:05:58.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:58.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:05:58.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:05:58.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:58.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:05:58.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:05:58.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:58.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:05:58.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:05:58.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:58.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:05:58.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:05:58.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:58.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:05:58.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:05:58.373 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.373 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.373 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.374 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:58.374 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:58.374 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 198.995386045, 2023-07-02T14:05:58.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:05:58.374 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:58.374 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:58.374 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:58.375 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:58.375 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:58.375 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:05:58.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.375 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:58.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:58.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:58.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:58.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:58.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:05:58.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:58.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:58.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:05:58.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:05:58.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:58.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:05:58.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:05:58.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:58.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:05:58.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:05:58.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:58.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:05:58.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:05:58.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:58.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:05:58.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:05:58.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:58.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:05:58.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:05:58.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:58.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:05:58.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:05:58.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:58.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:05:58.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:05:58.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:58.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:05:58.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:05:58.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:58.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:05:58.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:05:58.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:58.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:05:58.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:05:58.383 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.383 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.384 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.384 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:58.384 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:58.384 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 198.995194263, 2023-07-02T14:05:58.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:05:58.385 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:58.385 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:58.385 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:58.385 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:58.385 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:58.385 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:05:58.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.385 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:58.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:58.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:58.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:58.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:58.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:05:58.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:58.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:58.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:05:58.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:05:58.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:58.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:05:58.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:05:58.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:58.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:05:58.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:05:58.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:58.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:05:58.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:05:58.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:58.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:05:58.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:05:58.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:58.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:05:58.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:05:58.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:58.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:05:58.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:05:58.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:58.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:05:58.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:05:58.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:58.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:05:58.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:05:58.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:58.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:05:58.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:05:58.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:58.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:05:58.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:05:58.394 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.394 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.394 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.394 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:58.394 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:58.395 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 198.99501860300001, 2023-07-02T14:05:58.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:05:58.395 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:58.395 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:58.395 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:58.395 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:58.395 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:58.395 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:05:58.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.396 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:58.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:58.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:58.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:58.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:58.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:58.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:58.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:58.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:05:58.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:05:58.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:58.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:05:58.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:05:58.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:58.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:05:58.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:05:58.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:58.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:05:58.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:05:58.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:58.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:05:58.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:05:58.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:58.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:05:58.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:05:58.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:58.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:05:58.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:05:58.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:58.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:05:58.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:05:58.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:58.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:05:58.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:05:58.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:58.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:05:58.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:05:58.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.404 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:58.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:05:58.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:05:58.404 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.404 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.404 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.405 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:58.405 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:58.405 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 198.99492441500001, 2023-07-02T14:05:58.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:05:58.405 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:58.405 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:58.406 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:58.406 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:58.406 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:58.406 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:05:58.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.406 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:58.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:58.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:58.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:58.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:58.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:05:58.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:58.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:58.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:05:58.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:05:58.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:58.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:05:58.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:05:58.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:58.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:05:58.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:05:58.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:58.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:05:58.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:05:58.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:58.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:05:58.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:05:58.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:58.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:05:58.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:05:58.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:58.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:05:58.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:05:58.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:58.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:05:58.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:05:58.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:58.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:05:58.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:05:58.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:58.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:05:58.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:05:58.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:58.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:05:58.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:05:58.415 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.415 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.415 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.415 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:58.415 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:58.415 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 198.994721029, 2023-07-02T14:05:58.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:05:58.416 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:58.416 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:58.416 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:58.416 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:58.416 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:58.416 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:05:58.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.417 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:58.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:58.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:58.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:58.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:58.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:05:58.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:58.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:58.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:05:58.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:05:58.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:58.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:05:58.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:05:58.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:58.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:05:58.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:05:58.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:58.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:05:58.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:05:58.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:58.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:05:58.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:05:58.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:58.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:05:58.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:05:58.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:58.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:05:58.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:05:58.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:58.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:05:58.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:05:58.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:58.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:05:58.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:05:58.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:58.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:05:58.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:05:58.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:58.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:05:58.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:05:58.425 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.425 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.425 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.426 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:58.426 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:58.426 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 198.99457037299999, 2023-07-02T14:05:58.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:05:58.426 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:58.426 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:58.426 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:58.426 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:58.427 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:58.427 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:05:58.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.427 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:58.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:58.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:58.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:58.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:58.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:05:58.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:58.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:58.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:05:58.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:05:58.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:58.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:05:58.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:05:58.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:58.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:05:58.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:05:58.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:58.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:05:58.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:05:58.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:58.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:05:58.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:05:58.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:58.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:05:58.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:05:58.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:58.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:05:58.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:05:58.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:58.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:05:58.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:05:58.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:58.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:05:58.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:05:58.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:58.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:05:58.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:05:58.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:58.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:05:58.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:05:58.435 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.435 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.436 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.436 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:58.436 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:58.436 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 198.99449766999999, 2023-07-02T14:05:58.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:05:58.436 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:58.437 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:58.437 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:58.437 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:58.437 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:58.437 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:05:58.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.437 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:58.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:58.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:58.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:58.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:58.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:58.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:58.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:58.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:05:58.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:05:58.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:58.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:05:58.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:05:58.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:58.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:05:58.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:05:58.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:58.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:05:58.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:05:58.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:58.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:05:58.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:05:58.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:58.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:05:58.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:05:58.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:58.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:05:58.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:05:58.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:58.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:05:58.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:05:58.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:58.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:05:58.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:05:58.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:58.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:05:58.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:05:58.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:58.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:05:58.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:05:58.446 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.446 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.446 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.446 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:58.446 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:58.447 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 198.99438068000001, 2023-07-02T14:05:58.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:05:58.447 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:58.447 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:58.447 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:58.447 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:58.447 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:58.447 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:05:58.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.448 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:58.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:58.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:58.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:58.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:58.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:58.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:58.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:58.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:05:58.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:05:58.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:58.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:05:58.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:05:58.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:58.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:05:58.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:05:58.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:58.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:05:58.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:05:58.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:58.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:05:58.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:05:58.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:58.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:05:58.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:05:58.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:58.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:05:58.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:05:58.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:58.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:05:58.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:05:58.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:58.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:05:58.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:05:58.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:58.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:05:58.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:05:58.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:58.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:05:58.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:05:58.456 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.456 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.456 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.457 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:58.457 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:58.457 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 198.99423122900001, 2023-07-02T14:05:58.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:05:58.457 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:58.457 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:58.457 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:58.458 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:58.458 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:58.458 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:05:58.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.458 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:58.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:58.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:58.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:58.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:58.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:05:58.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:05:58.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:58.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:05:58.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:05:58.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:58.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:05:58.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:05:58.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:58.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:05:58.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:05:58.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:58.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:05:58.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:05:58.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:58.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:05:58.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:05:58.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:58.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:05:58.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:05:58.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:58.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:05:58.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:05:58.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:58.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:05:58.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:05:58.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:58.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:05:58.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:05:58.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:58.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:05:58.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:05:58.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:58.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:05:58.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:05:58.466 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.466 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.467 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.467 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:58.467 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:58.467 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 198.972755256, 2023-07-02T14:05:58.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:05:58.467 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:58.468 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:58.468 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:58.468 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:58.468 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:58.468 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:05:58.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.468 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:58.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:58.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:58.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:58.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:58.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:05:58.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:58.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:58.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:05:58.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:05:58.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:58.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:05:58.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:05:58.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:58.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:05:58.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:05:58.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:58.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:05:58.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:05:58.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:58.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:05:58.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:05:58.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:58.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:05:58.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:05:58.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:58.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:05:58.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:05:58.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:58.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:05:58.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:05:58.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:58.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:05:58.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:05:58.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:58.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:05:58.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:05:58.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:58.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:05:58.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:05:58.477 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.477 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.477 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.477 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:58.477 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:58.478 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 198.97262272, 2023-07-02T14:05:58.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:05:58.478 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:58.478 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:58.478 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:58.478 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:58.478 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:58.478 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:05:58.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.479 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:58.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:58.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:58.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:58.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:58.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:05:58.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:58.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:58.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:05:58.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:05:58.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:58.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:05:58.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:05:58.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:58.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:05:58.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:05:58.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:58.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:05:58.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:05:58.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:58.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:05:58.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:05:58.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:58.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:05:58.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:05:58.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:05:58.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:05:58.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:05:58.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:58.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:05:58.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:05:58.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:58.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:05:58.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:05:58.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:58.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:05:58.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:05:58.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:58.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:05:58.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:05:58.487 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.487 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.487 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.488 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:05:58.488 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:58.488 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 198.91128605899999, 2023-07-02T14:05:58.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:05:58.488 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:05:58.488 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:05:58.488 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:05:58.489 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:05:58.489 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:05:58.489 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:05:58.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.489 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:05:58.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:05:58.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:58.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:05:58.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:05:58.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:05:58.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:05:58.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:05:58.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:05:58.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:05:58.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:05:58.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:05:58.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:05:58.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:05:58.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:05:58.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:05:58.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:05:58.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:05:58.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:05:58.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:05:58.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:05:58.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:05:58.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:05:58.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:05:58.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:05:58.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:05:58.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:05:58.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:05:58.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:05:58.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:05:58.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:05:58.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:05:58.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:05:58.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:05:58.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:05:58.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:05:58.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:05:58.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:05:58.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:05:58.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:05:58.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:05:58.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:05:58.498 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.498 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.498 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.498 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:05:58.498 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:05:58.498 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:05:58.500 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:05:58.500 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:05:58.500 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:05:58.500 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:05:58.500 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:05:58.500 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:05:58.501 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:05:58.502 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:05:58.502 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:05:58.502 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:05:58.502 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:05:58.502 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:05:58.502 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:05:58.503 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:58.498+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:58.599 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:58.599+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:05:58.663 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:58.664 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:58.674 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:05:58.700 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:58.700+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:58.796 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:58.796 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:58.801 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:58.801+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:58.806 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:05:58.902 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:58.901+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:05:58.929 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:58.930 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:58.939 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:05:59.004 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:59.003+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:59.069 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:59.069 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:59.079 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:05:59.096 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.3 to 1 (10000) 2023-07-02T14:05:59.105 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:59.104+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:59.109 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-07-02T14:05:59.137 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:05:59.143 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 2 2023-07-02T14:05:59.191 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:05:59.206 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:59.205+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:59.208 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:59.208 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:59.219 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:05:59.307 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:05:59.306+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:05:59.388 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:59.389 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:59.399 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:05:59.441 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:59.440+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:05:59.508 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:05:59.507+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:05:59.566 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:59.566 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:59.576 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:05:59.608 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:59.608+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:59.699 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:59.700 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:59.709 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:05:59.708+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:05:59.710 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:05:59.810 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:05:59.809+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:05:59.873 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:05:59.873 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:05:59.883 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:05:59.927 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:05:59.926+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:00.007 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:00.007 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:00.012 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:00.011+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:00.017 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:06:00.113 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:00.112+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:00.181 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:00.181 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:00.191 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:06:00.214 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:00.213+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:00.314 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:00.314+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:00.315 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:00.315 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:00.326 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:06:00.415 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:00.414+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.2 to 1 (10000) 2023-07-02T14:06:00.430 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:06:00.443 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-07-02T14:06:00.443 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:06:00.464 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:00.465 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:00.475 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:06:00.504 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:06:00.516 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:00.515+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:00.617 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:00.616+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:00.640 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:00.640 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:00.650 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:06:00.677 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:00.695 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:00.712 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:00.731 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:00.732 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:00.732+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:00.750 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:00.768 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:00.769 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:06:00.769 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:06:00.769 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:06:00.769 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:06:00.770 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:06:00.800 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:06:00.802 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:06:00.813 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2023-07-02T14:06:00.424889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8220586353,"num_objects":2431,"num_object_clones":1985,"num_object_copies":4862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":50081,"num_read_kb":10611683,"num_write":19521,"num_write_kb":7982237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1692,"num_bytes_recovered":5815889213,"num_keys_recovered":114,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19247,"ondisk_log_size":19247,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":16178080,"kb_used_data":15966784,"kb_used_omap":1114,"kb_used_meta":210149,"kb_avail":361309280,"statfs":{"total":386547056640,"available":369980702720,"internally_reserved":0,"allocated":16349986816,"data_stored":18794695597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1141261,"internal_metadata":215193075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16384,"num_objects":1,"num_object_clones":1,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-75,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":77518,"num_write":3,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":531869696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.004883"},"pg_stats":[{"pgid":"2.7","version":"165'2725","reported_seq":6274,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-07-02T14:05:57.157549+0000","last_change":"2023-07-02T14:05:14.122260+0000","last_active":"2023-07-02T14:05:57.157549+0000","last_peered":"2023-07-02T14:05:57.157549+0000","last_clean":"2023-07-02T14:05:57.157549+0000","last_became_active":"2023-07-02T14:05:14.120593+0000","last_became_peered":"2023-07-02T14:05:14.120593+0000","last_unstale":"2023-07-02T14:05:57.157549+0000","last_undegraded":"2023-07-02T14:05:57.157549+0000","last_fullsized":"2023-07-02T14:05:57.157549+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2725,"ondisk_log_size":2725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1089489040,"num_objects":322,"num_object_clones":263,"num_object_copies":644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":29178,"num_read_kb":1386208,"num_write":2326,"num_write_kb":1059072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":1395581788,"num_keys_recovered":110,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"169'2173","reported_seq":3700,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-07-02T14:05:58.093727+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:05:58.093727+0000","last_peered":"2023-07-02T14:05:58.093727+0000","last_clean":"2023-07-02T14:05:58.093727+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:05:58.093727+0000","last_undegraded":"2023-07-02T14:05:58.093727+0000","last_fullsized":"2023-07-02T14:05:58.093727+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2173,"ondisk_log_size":2173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1065304874,"num_objects":324,"num_object_clones":269,"num_object_copies":648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":1251,"num_read_kb":1389391,"num_write":1904,"num_write_kb":1034481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"163'2310","reported_seq":2834,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-07-02T14:05:57.096599+0000","last_change":"2023-07-02T14:05:57.093448+0000","last_active":"2023-07-02T14:05:57.096599+0000","last_peered":"2023-07-02T14:05:57.096599+0000","last_clean":"2023-07-02T14:05:57.096599+0000","last_became_active":"2023-07-02T14:05:57.089226+0000","last_became_peered":"2023-07-02T14:05:57.089226+0000","last_unstale":"2023-07-02T14:05:57.096599+0000","last_undegraded":"2023-07-02T14:05:57.096599+0000","last_fullsized":"2023-07-02T14:05:57.096599+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2310,"ondisk_log_size":2310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038954982,"num_objects":283,"num_object_clones":233,"num_object_copies":566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":0,"num_read":418,"num_read_kb":1287950,"num_write":2077,"num_write_kb":1002192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":860565666,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"165'4536","reported_seq":5871,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-07-02T14:05:57.152834+0000","last_change":"2023-07-02T14:05:17.141529+0000","last_active":"2023-07-02T14:05:57.152834+0000","last_peered":"2023-07-02T14:05:57.152834+0000","last_clean":"2023-07-02T14:05:57.152834+0000","last_became_active":"2023-07-02T14:05:17.138301+0000","last_became_peered":"2023-07-02T14:05:17.138301+0000","last_unstale":"2023-07-02T14:05:57.152834+0000","last_undegraded":"2023-07-02T14:05:57.152834+0000","last_fullsized":"2023-07-02T14:05:57.152834+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":4536,"ondisk_log_size":4536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1143891394,"num_objects":336,"num_object_clones":276,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":0,"num_read":15634,"num_read_kb":1502710,"num_write":6655,"num_write_kb":1118541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":860210318,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":98,"reported_epoch":168,"state":"active+clean","last_fresh":"2023-07-02T14:05:54.412749+0000","last_change":"2023-07-02T14:05:53.410292+0000","last_active":"2023-07-02T14:05:54.412749+0000","last_peered":"2023-07-02T14:05:54.412749+0000","last_clean":"2023-07-02T14:05:54.412749+0000","last_became_active":"2023-07-02T14:05:53.409913+0000","last_became_peered":"2023-07-02T14:05:53.409913+0000","last_unstale":"2023-07-02T14:05:54.412749+0000","last_undegraded":"2023-07-02T14:05:54.412749+0000","last_fullsized":"2023-07-02T14:05:54.412749+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"165'2109","reported_seq":3209,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-07-02T14:05:58.090416+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:05:58.090416+0000","last_peered":"2023-07-02T14:05:58.090416+0000","last_clean":"2023-07-02T14:05:58.090416+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:05:58.090416+0000","last_undegraded":"2023-07-02T14:05:58.090416+0000","last_fullsized":"2023-07-02T14:05:58.090416+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2109,"ondisk_log_size":2109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1145177052,"num_objects":335,"num_object_clones":271,"num_object_copies":670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1006,"num_read_kb":1437730,"num_write":1838,"num_write_kb":1112259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"165'2097","reported_seq":2702,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-07-02T14:05:58.090414+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:05:58.090414+0000","last_peered":"2023-07-02T14:05:58.090414+0000","last_clean":"2023-07-02T14:05:58.090414+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:05:58.090414+0000","last_undegraded":"2023-07-02T14:05:58.090414+0000","last_fullsized":"2023-07-02T14:05:58.090414+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2097,"ondisk_log_size":2097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1112269934,"num_objects":318,"num_object_clones":258,"num_object_copies":636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":595,"num_read_kb":1375429,"num_write":1839,"num_write_kb":1080663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"169'1610","reported_seq":3091,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-07-02T14:05:59.113316+0000","last_change":"2023-07-02T14:05:58.098047+0000","last_active":"2023-07-02T14:05:59.113316+0000","last_peered":"2023-07-02T14:05:59.113316+0000","last_clean":"2023-07-02T14:05:59.113316+0000","last_became_active":"2023-07-02T14:05:58.096421+0000","last_became_peered":"2023-07-02T14:05:58.096421+0000","last_unstale":"2023-07-02T14:05:59.113316+0000","last_undegraded":"2023-07-02T14:05:59.113316+0000","last_fullsized":"2023-07-02T14:05:59.113316+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1610,"ondisk_log_size":1610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":828097341,"num_objects":269,"num_object_clones":215,"num_object_copies":538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":1306,"num_read_kb":1161093,"num_write":1395,"num_write_kb":800047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":340635667,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"165'1687","reported_seq":2493,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-07-02T14:05:58.093625+0000","last_change":"2023-07-02T14:05:56.429093+0000","last_active":"2023-07-02T14:05:58.093625+0000","last_peered":"2023-07-02T14:05:58.093625+0000","last_clean":"2023-07-02T14:05:58.093625+0000","last_became_active":"2023-07-02T14:05:56.427016+0000","last_became_peered":"2023-07-02T14:05:56.427016+0000","last_unstale":"2023-07-02T14:05:58.093625+0000","last_undegraded":"2023-07-02T14:05:58.093625+0000","last_fullsized":"2023-07-02T14:05:58.093625+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1687,"ondisk_log_size":1687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":797401736,"num_objects":244,"num_object_clones":200,"num_object_copies":488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":244,"num_whiteouts":0,"num_read":693,"num_read_kb":1071172,"num_write":1487,"num_write_kb":774982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":271880192,"num_keys_recovered":4,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":8220586353,"num_objects":2431,"num_object_clones":1985,"num_object_copies":4862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":50081,"num_read_kb":10611683,"num_write":19521,"num_write_kb":7982237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1692,"num_bytes_recovered":5815889213,"num_keys_recovered":114,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16474308608,"data_stored":18953343876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":19247,"ondisk_log_size":19247,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640309,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":0,"seq":47244640309,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":133,"seq":571230650380,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4280136,"kb_used_data":4222016,"kb_used_omap":229,"kb_used_meta":57882,"kb_avail":90091704,"statfs":{"total":96636764160,"available":92253904896,"internally_reserved":0,"allocated":4323344384,"data_stored":4943460282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":59271390},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640309,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3300452,"kb_used_data":3266332,"kb_used_omap":0,"kb_used_meta":34112,"kb_avail":91071388,"statfs":{"total":96636764160,"available":93257101312,"internally_reserved":0,"allocated":3344723968,"data_stored":3811138462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34930688},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.29399999999999998,"15min":0.29599999999999999},"min":{"1min":0.218,"5min":0.182,"15min":0.182},"max":{"1min":0.378,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.252},{"interface":"front","average":{"1min":0.317,"5min":0.315,"15min":0.314},"min":{"1min":0.23000000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.20200000000000001}]},{"osd":1,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.38800000000000001,"15min":0.36199999999999999},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.2720000000000002,"5min":4.2720000000000002,"15min":4.2720000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.32300000000000001,"15min":0.32000000000000001},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.35399999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.45000000000000001,"15min":0.45100000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46999999999999997,"15min":0.46800000000000003},"min":{"1min":0.35299999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.378}]},{"osd":4,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.45400000000000001,"15min":0.45100000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.46899999999999997,"15min":0.46300000000000002},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.55400000000000005}]},{"osd":5,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.34699999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.68899999999999995}]}]},{"osd":0,"up_from":166,"seq":712964571138,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2580888,"kb_used_data":2560272,"kb_used_omap":884,"kb_used_meta":19723,"kb_avail":91790952,"statfs":{"total":96636764160,"available":93993934848,"internally_reserved":0,"allocated":2621718528,"data_stored":2924164758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":905963,"internal_metadata":20196629},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640309,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6016604,"kb_used_data":5918164,"kb_used_omap":0,"kb_used_meta":98432,"kb_avail":88355236,"statfs":{"total":96636764160,"available":90475761664,"internally_reserved":0,"allocated":6060199936,"data_stored":7115932095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":100794368},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.20200000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.255},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.32200000000000001,"15min":0.33000000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.41099999999999998,"5min":0.496,"15min":0.496},"last":0.24099999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.33500000000000002,"15min":0.33100000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.47899999999999998,"5min":1.615,"15min":1.615},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.53100000000000003,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.22900000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.497,"15min":0.501},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.624,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.498,"15min":0.499},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.64400000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.36299999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47899999999999998,"15min":0.5},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.495,"15min":0.503},"min":{"1min":0.308,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71999999999999997,"5min":1.524,"15min":1.524},"last":0.39600000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.51500000000000001},"min":{"1min":0.33400000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.73899999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54800000000000004,"15min":0.54600000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.77600000000000002,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.45700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2727088128,"data_stored":3060078507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6058119168,"data_stored":7115118331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3342643200,"data_stored":3810324698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4321288192,"data_stored":4942656354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25169920,"data_stored":25165986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:06:00.815 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:06:00.817 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:00.817 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:00.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:00.818 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:00.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.818 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:06:00.818 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:00.818 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 140.91862565100001, 2023-07-02T14:06:00.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:06:00.819 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:00.819 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:00.819 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:00.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:00.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:00.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:00.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:00.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:00.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:00.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:00.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:06:00.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:06:00.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:00.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:06:00.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:06:00.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:00.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:06:00.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:06:00.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:00.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:06:00.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:06:00.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:00.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:06:00.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:06:00.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:00.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:06:00.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:06:00.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:00.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:06:00.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:06:00.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:00.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:06:00.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:06:00.826 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.826 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.826 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.826 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:06:00.826 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:00.827 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 140.91453993299999, 2023-07-02T14:06:00.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:06:00.827 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:00.827 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:00.827 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:00.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:00.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:00.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:00.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:00.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:00.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:00.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:00.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:06:00.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:06:00.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:00.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:06:00.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:06:00.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:00.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:06:00.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:06:00.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:00.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:06:00.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:06:00.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:00.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:06:00.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:06:00.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:00.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:06:00.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:06:00.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:00.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:06:00.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:06:00.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:00.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:06:00.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:06:00.834 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.834 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.834 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.834 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:06:00.834 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:00.835 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 140.91067515200001, 2023-07-02T14:06:00.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:06:00.835 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:00.835 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:00.835 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:00.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:00.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:00.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:00.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:00.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:00.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:00.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:00.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:06:00.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:06:00.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:00.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:06:00.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:06:00.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:00.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:06:00.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:06:00.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:00.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:06:00.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:06:00.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:00.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:06:00.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:06:00.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:00.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:06:00.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:06:00.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:00.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:06:00.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:06:00.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:00.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:06:00.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:06:00.841 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.842 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.842 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.842 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:06:00.842 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:06:00.842 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 140.90715089099999, 2023-07-02T14:06:00.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:06:00.843 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:00.843 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:00.843 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:00.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:00.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:06:00.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:00.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:00.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:06:00.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:00.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:00.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:06:00.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:06:00.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:00.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:06:00.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:06:00.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:00.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:06:00.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:06:00.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:00.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:06:00.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:06:00.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:00.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:06:00.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:06:00.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:00.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:06:00.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:06:00.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:00.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:06:00.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:06:00.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:00.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:06:00.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:06:00.849 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.850 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.850 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.850 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:06:00.850 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:00.850 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 140.87144609200001, 2023-07-02T14:06:00.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:06:00.851 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:00.851 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:00.851 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:00.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:00.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:00.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:00.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:00.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:00.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:00.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:00.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:06:00.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:06:00.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:00.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:06:00.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:06:00.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:00.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:06:00.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:06:00.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:00.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:06:00.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:06:00.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:00.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:06:00.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:06:00.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:00.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:06:00.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:06:00.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:00.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:06:00.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:06:00.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:00.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:06:00.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:06:00.857 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.857 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.857 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.858 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:06:00.858 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:00.858 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 140.86678873400001, 2023-07-02T14:06:00.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:06:00.858 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:00.858 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:00.859 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:00.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:00.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:00.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:00.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:00.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:00.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:00.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:00.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:06:00.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:06:00.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:00.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:06:00.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:06:00.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:00.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:06:00.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:06:00.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:00.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:06:00.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:06:00.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:00.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:06:00.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:06:00.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:00.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:06:00.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:06:00.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:00.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:06:00.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:06:00.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:00.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:06:00.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:06:00.865 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.865 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.865 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.866 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:06:00.866 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:00.866 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 140.865848871, 2023-07-02T14:06:00.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:06:00.866 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:00.866 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:00.866 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:00.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:00.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:00.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:00.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:00.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:00.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:00.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:00.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:06:00.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:06:00.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:00.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:06:00.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:06:00.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:00.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:06:00.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:06:00.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:00.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:06:00.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:06:00.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:00.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:06:00.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:06:00.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:00.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:06:00.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:06:00.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:00.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:06:00.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:06:00.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:00.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:06:00.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:06:00.873 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.873 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.873 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.874 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:06:00.874 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:00.874 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 140.86445099299999, 2023-07-02T14:06:00.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:06:00.874 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:00.874 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:00.874 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:00.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:00.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:00.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:00.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:00.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:00.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:00.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:00.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:06:00.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:06:00.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:00.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:06:00.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:06:00.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:00.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:06:00.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:06:00.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:00.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:06:00.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:06:00.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:00.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:06:00.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:06:00.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:00.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:06:00.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:06:00.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:00.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:06:00.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:06:00.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:00.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:06:00.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:06:00.881 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.881 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.881 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.882 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:06:00.882 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:00.882 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 119.45426049700001, 2023-07-02T14:06:00.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:06:00.882 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:00.882 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:00.882 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:00.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:00.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:00.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:00.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:00.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:00.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:00.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:00.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:06:00.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:06:00.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:00.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:06:00.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:06:00.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:00.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:06:00.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:06:00.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:00.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:06:00.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:06:00.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:00.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:06:00.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:06:00.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:00.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:06:00.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:06:00.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:00.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:06:00.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:06:00.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:00.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:06:00.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:06:00.889 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.889 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.889 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.889 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:00.890 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:00.890 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 118.08791081299999, 2023-07-02T14:06:00.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:06:00.890 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:00.890 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:00.890 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:00.890 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:00.890 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:00.891 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:06:00.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.891 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:00.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:00.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:00.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:00.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:00.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:00.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:00.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:00.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:06:00.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:06:00.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.893 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:00.893 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:06:00.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:06:00.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:00.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:06:00.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:06:00.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:00.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:06:00.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:06:00.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:00.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:06:00.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:06:00.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:00.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:06:00.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:06:00.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:00.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:06:00.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:06:00.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:00.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:06:00.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:06:00.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:00.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:06:00.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:06:00.898 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.898 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.898 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.899 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7707 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 3145728~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:00.899 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:06:00.899 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 118.083927346, 2023-07-02T14:06:00.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999, 2023-07-02T14:06:00.899 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:00.899 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:00.899 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:00.899 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:00.900 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:00.900 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7707 2023-07-02T14:06:00.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.900 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:00.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:00.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:06:00.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:00.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:00.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:06:00.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:00.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:00.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:06:00.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5017e-05 2023-07-02T14:06:00.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:00.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732347+0000", 2023-07-02T14:06:00.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5821e-05 2023-07-02T14:06:00.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:00.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732363+0000", 2023-07-02T14:06:00.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1270000000000002e-06 2023-07-02T14:06:00.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:00.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732368+0000", 2023-07-02T14:06:00.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008444666 2023-07-02T14:06:00.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:00.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740812+0000", 2023-07-02T14:06:00.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011266599999999999 2023-07-02T14:06:00.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:00.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740925+0000", 2023-07-02T14:06:00.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010848808999999999 2023-07-02T14:06:00.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:00.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751774+0000", 2023-07-02T14:06:00.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8421999999999997e-05 2023-07-02T14:06:00.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:00.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751832+0000", 2023-07-02T14:06:00.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107453 2023-07-02T14:06:00.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:00.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751940+0000", 2023-07-02T14:06:00.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999 2023-07-02T14:06:00.907 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.907 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.907 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.908 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:00.908 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:00.908 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 118.08386346899999, 2023-07-02T14:06:00.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:06:00.908 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:00.908 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:00.908 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:00.909 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:00.909 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:00.909 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:06:00.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.909 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:00.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:00.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:00.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:00.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:00.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:00.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:00.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:00.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:00.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:06:00.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:00.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:06:00.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:06:00.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:00.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:06:00.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:06:00.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:00.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:06:00.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:06:00.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:00.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:06:00.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:06:00.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:00.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:06:00.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:06:00.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:00.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:06:00.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:06:00.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:00.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:06:00.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:06:00.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:00.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:06:00.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:06:00.916 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.916 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.916 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.917 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:00.917 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:00.917 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 118.079299644, 2023-07-02T14:06:00.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:06:00.917 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:00.917 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:00.917 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:00.918 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:00.918 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:00.918 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:06:00.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.918 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:00.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:00.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:00.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:00.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:00.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:00.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:00.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:00.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:06:00.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:06:00.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:00.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:06:00.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:06:00.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:00.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:06:00.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:06:00.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:00.922 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:06:00.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:06:00.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:00.922 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:06:00.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:06:00.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:00.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:06:00.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:06:00.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:00.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:06:00.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:06:00.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:00.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:06:00.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:06:00.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:00.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:06:00.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:06:00.925 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.925 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.925 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.926 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:00.926 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:00.927 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 118.079249778, 2023-07-02T14:06:00.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:06:00.927 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:00.927 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:00.927 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:00.927 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:00.927 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:00.928 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:06:00.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.928 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:00.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:00.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:00.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:00.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:00.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:00.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:00.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:00.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:06:00.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:06:00.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:00.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:06:00.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:06:00.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:00.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:06:00.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:06:00.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:00.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:06:00.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:06:00.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:00.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:06:00.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:06:00.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:00.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:06:00.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:06:00.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:00.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:06:00.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:06:00.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:00.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:06:00.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:06:00.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:00.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:06:00.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:06:00.935 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.935 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.936 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.936 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:00.936 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:00.936 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 118.078210785, 2023-07-02T14:06:00.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:06:00.937 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:00.937 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:00.937 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:00.937 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:00.937 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:00.937 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:06:00.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.937 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:00.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:00.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:00.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:00.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:00.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:00.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:00.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:00.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:06:00.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:06:00.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:00.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:06:00.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:06:00.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:00.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:06:00.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:06:00.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:00.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:06:00.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:06:00.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:00.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:06:00.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:06:00.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:00.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:06:00.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:06:00.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:00.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:06:00.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:06:00.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:00.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:06:00.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:06:00.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:00.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:06:00.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:06:00.945 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.945 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.945 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.945 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7993 2.7 2:fe72db53:::rbd_data.125de1449605.0000000000000072:head [write 65536~3276800] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:00.945 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:06:00.946 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 117.891252737, 2023-07-02T14:06:00.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919, 2023-07-02T14:06:00.946 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:00.946 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:00.946 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:00.946 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:00.946 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:00.946 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7993 2023-07-02T14:06:00.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.947 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:00.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:00.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:06:00.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:00.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:00.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:06:00.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:00.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:00.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924986+0000", 2023-07-02T14:06:00.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049542070000000004 2023-07-02T14:06:00.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:00.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929940+0000", 2023-07-02T14:06:00.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9740000000000001e-05 2023-07-02T14:06:00.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:00.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929960+0000", 2023-07-02T14:06:00.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.311e-05 2023-07-02T14:06:00.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:00.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929973+0000", 2023-07-02T14:06:00.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9629000000000002e-05 2023-07-02T14:06:00.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:00.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930023+0000", 2023-07-02T14:06:00.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012751799999999999 2023-07-02T14:06:00.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:00.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930150+0000", 2023-07-02T14:06:00.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016449042000000001 2023-07-02T14:06:00.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:00.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946599+0000", 2023-07-02T14:06:00.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3241e-05 2023-07-02T14:06:00.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:00.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946663+0000", 2023-07-02T14:06:00.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000102296 2023-07-02T14:06:00.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:00.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946765+0000", 2023-07-02T14:06:00.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919 2023-07-02T14:06:00.954 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.954 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.954 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.955 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11782 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 196608~2162688] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:06:00.955 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:06:00.955 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 76.406732814999998, 2023-07-02T14:06:00.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365, 2023-07-02T14:06:00.955 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:00.955 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:00.955 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:00.955 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:00.956 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:00.956 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11782 2023-07-02T14:06:00.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.956 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:00.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:00.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:06:00.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:00.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:00.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:06:00.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:00.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:00.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409506+0000", 2023-07-02T14:06:00.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020172269999999999 2023-07-02T14:06:00.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:00.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411523+0000", 2023-07-02T14:06:00.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8901000000000001e-05 2023-07-02T14:06:00.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:00.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411542+0000", 2023-07-02T14:06:00.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5037e-05 2023-07-02T14:06:00.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:00.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411557+0000", 2023-07-02T14:06:00.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001931213 2023-07-02T14:06:00.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:00.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413489+0000", 2023-07-02T14:06:00.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000149207 2023-07-02T14:06:00.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:00.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413638+0000", 2023-07-02T14:06:00.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024752800000000001 2023-07-02T14:06:00.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:00.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413885+0000", 2023-07-02T14:06:00.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048516000000000002 2023-07-02T14:06:00.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:00.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.418737+0000", 2023-07-02T14:06:00.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010319259000000001 2023-07-02T14:06:00.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:00.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429056+0000", 2023-07-02T14:06:00.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2518000000000002e-05 2023-07-02T14:06:00.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:00.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429099+0000", 2023-07-02T14:06:00.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015275 2023-07-02T14:06:00.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:00.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429251+0000", 2023-07-02T14:06:00.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365 2023-07-02T14:06:00.964 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.965 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.965 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.965 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPushReply(2.7 119/117 [PushReplyOp(2:eb862bc0:::rbd_data.125de1449605.0000000000000040:13)])", 2023-07-02T14:06:00.965 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:06:00.965 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 76.405569540000002, 2023-07-02T14:06:00.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999, 2023-07-02T14:06:00.966 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:00.966 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "reached pg", 2023-07-02T14:06:00.966 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:00.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:00.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:06:00.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:00.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:00.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:06:00.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:00.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:00.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:06:00.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7670000000000001e-06 2023-07-02T14:06:00.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:00.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410674+0000", 2023-07-02T14:06:00.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3973e-05 2023-07-02T14:06:00.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:00.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410688+0000", 2023-07-02T14:06:00.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0157e-05 2023-07-02T14:06:00.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:00.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410708+0000", 2023-07-02T14:06:00.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018578779 2023-07-02T14:06:00.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:00.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429287+0000", 2023-07-02T14:06:00.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013086549999999999 2023-07-02T14:06:00.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:00.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.430595+0000", 2023-07-02T14:06:00.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999 2023-07-02T14:06:00.971 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.971 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.971 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.972 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11861 2.7 2:e57d3f55:::rbd_data.125de1449605.0000000000000134:head [write 0~65536] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:06:00.972 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:06:00.972 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 76.271454163000001, 2023-07-02T14:06:00.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723, 2023-07-02T14:06:00.972 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:00.972 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:00.972 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:00.973 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:00.973 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:00.973 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11861 2023-07-02T14:06:00.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.973 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:00.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:00.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:06:00.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:00.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:00.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:06:00.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:00.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:00.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544785+0000", 2023-07-02T14:06:00.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010563700000000001 2023-07-02T14:06:00.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:00.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544890+0000", 2023-07-02T14:06:00.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8530000000000004e-06 2023-07-02T14:06:00.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:00.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544900+0000", 2023-07-02T14:06:00.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.665e-06 2023-07-02T14:06:00.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:00.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544906+0000", 2023-07-02T14:06:00.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000272575 2023-07-02T14:06:00.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:00.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545178+0000", 2023-07-02T14:06:00.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000111362 2023-07-02T14:06:00.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:00.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545290+0000", 2023-07-02T14:06:00.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016181100000000001 2023-07-02T14:06:00.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:00.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545452+0000", 2023-07-02T14:06:00.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044500750000000004 2023-07-02T14:06:00.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:00.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.549902+0000", 2023-07-02T14:06:00.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014850024 2023-07-02T14:06:00.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:00.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564752+0000", 2023-07-02T14:06:00.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6272000000000003e-05 2023-07-02T14:06:00.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:00.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564848+0000", 2023-07-02T14:06:00.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000162428 2023-07-02T14:06:00.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:00.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.565010+0000", 2023-07-02T14:06:00.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723 2023-07-02T14:06:00.982 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.982 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.982 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.982 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:13262 2.7 e125/121 2:e0741dde:::rbd_data.125de1449605.00000000000000ad:head v 125'2024, mlcod=125'2021)", 2023-07-02T14:06:00.982 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:06:00.983 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 67.565376424999997, 2023-07-02T14:06:00.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999, 2023-07-02T14:06:00.983 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:00.983 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:00.983 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:00.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:00.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:06:00.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:00.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:00.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:06:00.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:00.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:00.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250862+0000", 2023-07-02T14:06:00.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080378800000000007 2023-07-02T14:06:00.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:00.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258900+0000", 2023-07-02T14:06:00.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8939999999999999e-06 2023-07-02T14:06:00.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:00.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258909+0000", 2023-07-02T14:06:00.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3560000000000003e-06 2023-07-02T14:06:00.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:00.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258914+0000", 2023-07-02T14:06:00.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2429000000000003e-05 2023-07-02T14:06:00.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:00.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258956+0000", 2023-07-02T14:06:00.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5094999999999999e-05 2023-07-02T14:06:00.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:00.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258981+0000", 2023-07-02T14:06:00.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012177307 2023-07-02T14:06:00.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:00.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271158+0000", 2023-07-02T14:06:00.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2104999999999999e-05 2023-07-02T14:06:00.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:00.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:00.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:00.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271181+0000", 2023-07-02T14:06:00.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999 2023-07-02T14:06:00.990 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.990 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.990 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.990 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:00.990 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:00.990 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:00.992 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:06:00.993 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:00.991+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:01.092 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:01.091+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:01.145 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap45 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap46 b3fbb028-e267-4d25-a750-93ba18a2316a@snap46.diff 2023-07-02T14:06:01.159 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:01.159 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:01.159 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:06:01.159 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:01.169 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:06:01.192 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:01.191+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:01.214 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:01.213+0000 7f89f2c74700 0 -- 172.21.15.163:0/2321094239 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f89d00035d0 msgr2=0x7f89d0003a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:01.222 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:01.221+0000 7f89f2c74700 0 -- 172.21.15.163:0/2321094239 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f89d8036bb0 msgr2=0x7f89d8038fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:01.261 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 100% complete...done. 2023-07-02T14:06:01.264 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:06:01.264 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=47 2023-07-02T14:06:01.264 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap47 2023-07-02T14:06:01.293 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:01.292+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:01.302 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:06:01.305 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:06:01.317 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2023-07-02T14:06:00.424889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8220586353,"num_objects":2431,"num_object_clones":1985,"num_object_copies":4862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":50081,"num_read_kb":10611683,"num_write":19521,"num_write_kb":7982237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1692,"num_bytes_recovered":5815889213,"num_keys_recovered":114,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19247,"ondisk_log_size":19247,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":16178080,"kb_used_data":15966784,"kb_used_omap":1114,"kb_used_meta":210149,"kb_avail":361309280,"statfs":{"total":386547056640,"available":369980702720,"internally_reserved":0,"allocated":16349986816,"data_stored":18794695597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1141261,"internal_metadata":215193075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16384,"num_objects":1,"num_object_clones":1,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-75,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":743,"num_read_kb":77518,"num_write":3,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":531869696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.004883"},"pg_stats":[{"pgid":"2.7","version":"165'2725","reported_seq":6274,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-07-02T14:05:57.157549+0000","last_change":"2023-07-02T14:05:14.122260+0000","last_active":"2023-07-02T14:05:57.157549+0000","last_peered":"2023-07-02T14:05:57.157549+0000","last_clean":"2023-07-02T14:05:57.157549+0000","last_became_active":"2023-07-02T14:05:14.120593+0000","last_became_peered":"2023-07-02T14:05:14.120593+0000","last_unstale":"2023-07-02T14:05:57.157549+0000","last_undegraded":"2023-07-02T14:05:57.157549+0000","last_fullsized":"2023-07-02T14:05:57.157549+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2725,"ondisk_log_size":2725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1089489040,"num_objects":322,"num_object_clones":263,"num_object_copies":644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":29178,"num_read_kb":1386208,"num_write":2326,"num_write_kb":1059072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":1395581788,"num_keys_recovered":110,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"169'2173","reported_seq":3700,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-07-02T14:05:58.093727+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:05:58.093727+0000","last_peered":"2023-07-02T14:05:58.093727+0000","last_clean":"2023-07-02T14:05:58.093727+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:05:58.093727+0000","last_undegraded":"2023-07-02T14:05:58.093727+0000","last_fullsized":"2023-07-02T14:05:58.093727+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2173,"ondisk_log_size":2173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1065304874,"num_objects":324,"num_object_clones":269,"num_object_copies":648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":1251,"num_read_kb":1389391,"num_write":1904,"num_write_kb":1034481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"163'2310","reported_seq":2834,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-07-02T14:05:57.096599+0000","last_change":"2023-07-02T14:05:57.093448+0000","last_active":"2023-07-02T14:05:57.096599+0000","last_peered":"2023-07-02T14:05:57.096599+0000","last_clean":"2023-07-02T14:05:57.096599+0000","last_became_active":"2023-07-02T14:05:57.089226+0000","last_became_peered":"2023-07-02T14:05:57.089226+0000","last_unstale":"2023-07-02T14:05:57.096599+0000","last_undegraded":"2023-07-02T14:05:57.096599+0000","last_fullsized":"2023-07-02T14:05:57.096599+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2310,"ondisk_log_size":2310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038954982,"num_objects":283,"num_object_clones":233,"num_object_copies":566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":0,"num_read":418,"num_read_kb":1287950,"num_write":2077,"num_write_kb":1002192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":860565666,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"165'4536","reported_seq":5871,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-07-02T14:05:57.152834+0000","last_change":"2023-07-02T14:05:17.141529+0000","last_active":"2023-07-02T14:05:57.152834+0000","last_peered":"2023-07-02T14:05:57.152834+0000","last_clean":"2023-07-02T14:05:57.152834+0000","last_became_active":"2023-07-02T14:05:17.138301+0000","last_became_peered":"2023-07-02T14:05:17.138301+0000","last_unstale":"2023-07-02T14:05:57.152834+0000","last_undegraded":"2023-07-02T14:05:57.152834+0000","last_fullsized":"2023-07-02T14:05:57.152834+0000","mapping_epoch":144,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":4536,"ondisk_log_size":4536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1143891394,"num_objects":336,"num_object_clones":276,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":336,"num_whiteouts":0,"num_read":15634,"num_read_kb":1502710,"num_write":6655,"num_write_kb":1118541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":860210318,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":98,"reported_epoch":168,"state":"active+clean","last_fresh":"2023-07-02T14:05:54.412749+0000","last_change":"2023-07-02T14:05:53.410292+0000","last_active":"2023-07-02T14:05:54.412749+0000","last_peered":"2023-07-02T14:05:54.412749+0000","last_clean":"2023-07-02T14:05:54.412749+0000","last_became_active":"2023-07-02T14:05:53.409913+0000","last_became_peered":"2023-07-02T14:05:53.409913+0000","last_unstale":"2023-07-02T14:05:54.412749+0000","last_undegraded":"2023-07-02T14:05:54.412749+0000","last_fullsized":"2023-07-02T14:05:54.412749+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"165'2109","reported_seq":3209,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-07-02T14:05:58.090416+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:05:58.090416+0000","last_peered":"2023-07-02T14:05:58.090416+0000","last_clean":"2023-07-02T14:05:58.090416+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:05:58.090416+0000","last_undegraded":"2023-07-02T14:05:58.090416+0000","last_fullsized":"2023-07-02T14:05:58.090416+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2109,"ondisk_log_size":2109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1145177052,"num_objects":335,"num_object_clones":271,"num_object_copies":670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1006,"num_read_kb":1437730,"num_write":1838,"num_write_kb":1112259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"165'2097","reported_seq":2702,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-07-02T14:05:58.090414+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:05:58.090414+0000","last_peered":"2023-07-02T14:05:58.090414+0000","last_clean":"2023-07-02T14:05:58.090414+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:05:58.090414+0000","last_undegraded":"2023-07-02T14:05:58.090414+0000","last_fullsized":"2023-07-02T14:05:58.090414+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2097,"ondisk_log_size":2097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1112269934,"num_objects":318,"num_object_clones":258,"num_object_copies":636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":0,"num_read":595,"num_read_kb":1375429,"num_write":1839,"num_write_kb":1080663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"169'1610","reported_seq":3091,"reported_epoch":173,"state":"active+clean","last_fresh":"2023-07-02T14:05:59.113316+0000","last_change":"2023-07-02T14:05:58.098047+0000","last_active":"2023-07-02T14:05:59.113316+0000","last_peered":"2023-07-02T14:05:59.113316+0000","last_clean":"2023-07-02T14:05:59.113316+0000","last_became_active":"2023-07-02T14:05:58.096421+0000","last_became_peered":"2023-07-02T14:05:58.096421+0000","last_unstale":"2023-07-02T14:05:59.113316+0000","last_undegraded":"2023-07-02T14:05:59.113316+0000","last_fullsized":"2023-07-02T14:05:59.113316+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1610,"ondisk_log_size":1610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":828097341,"num_objects":269,"num_object_clones":215,"num_object_copies":538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":1306,"num_read_kb":1161093,"num_write":1395,"num_write_kb":800047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":340635667,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"165'1687","reported_seq":2493,"reported_epoch":172,"state":"active+clean","last_fresh":"2023-07-02T14:05:58.093625+0000","last_change":"2023-07-02T14:05:56.429093+0000","last_active":"2023-07-02T14:05:58.093625+0000","last_peered":"2023-07-02T14:05:58.093625+0000","last_clean":"2023-07-02T14:05:58.093625+0000","last_became_active":"2023-07-02T14:05:56.427016+0000","last_became_peered":"2023-07-02T14:05:56.427016+0000","last_unstale":"2023-07-02T14:05:58.093625+0000","last_undegraded":"2023-07-02T14:05:58.093625+0000","last_fullsized":"2023-07-02T14:05:58.093625+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1687,"ondisk_log_size":1687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":797401736,"num_objects":244,"num_object_clones":200,"num_object_copies":488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":244,"num_whiteouts":0,"num_read":693,"num_read_kb":1071172,"num_write":1487,"num_write_kb":774982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":271880192,"num_keys_recovered":4,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":8220586353,"num_objects":2431,"num_object_clones":1985,"num_object_copies":4862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":50081,"num_read_kb":10611683,"num_write":19521,"num_write_kb":7982237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1692,"num_bytes_recovered":5815889213,"num_keys_recovered":114,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16474308608,"data_stored":18953343876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":19247,"ondisk_log_size":19247,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640309,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":0,"seq":47244640309,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":133,"seq":571230650380,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4280136,"kb_used_data":4222016,"kb_used_omap":229,"kb_used_meta":57882,"kb_avail":90091704,"statfs":{"total":96636764160,"available":92253904896,"internally_reserved":0,"allocated":4323344384,"data_stored":4943460282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":59271390},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640309,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3300452,"kb_used_data":3266332,"kb_used_omap":0,"kb_used_meta":34112,"kb_avail":91071388,"statfs":{"total":96636764160,"available":93257101312,"internally_reserved":0,"allocated":3344723968,"data_stored":3811138462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34930688},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.29399999999999998,"15min":0.29599999999999999},"min":{"1min":0.218,"5min":0.182,"15min":0.182},"max":{"1min":0.378,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.252},{"interface":"front","average":{"1min":0.317,"5min":0.315,"15min":0.314},"min":{"1min":0.23000000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.20200000000000001}]},{"osd":1,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.38800000000000001,"15min":0.36199999999999999},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.2720000000000002,"5min":4.2720000000000002,"15min":4.2720000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.32300000000000001,"15min":0.32000000000000001},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.35399999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:03:48 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.45000000000000001,"15min":0.45100000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.46999999999999997,"15min":0.46800000000000003},"min":{"1min":0.35299999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.378}]},{"osd":4,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.45400000000000001,"15min":0.45100000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.46899999999999997,"15min":0.46300000000000002},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.55400000000000005}]},{"osd":5,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.34699999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.68899999999999995}]}]},{"osd":0,"up_from":166,"seq":712964571138,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2580888,"kb_used_data":2560272,"kb_used_omap":884,"kb_used_meta":19723,"kb_avail":91790952,"statfs":{"total":96636764160,"available":93993934848,"internally_reserved":0,"allocated":2621718528,"data_stored":2924164758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":905963,"internal_metadata":20196629},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640309,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6016604,"kb_used_data":5918164,"kb_used_omap":0,"kb_used_meta":98432,"kb_avail":88355236,"statfs":{"total":96636764160,"available":90475761664,"internally_reserved":0,"allocated":6060199936,"data_stored":7115932095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":100794368},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.20200000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.255},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.32200000000000001,"15min":0.33000000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.41099999999999998,"5min":0.496,"15min":0.496},"last":0.24099999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.33500000000000002,"15min":0.33100000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.47899999999999998,"5min":1.615,"15min":1.615},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.53100000000000003,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.22900000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:03:51 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.497,"15min":0.501},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.624,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.498,"15min":0.499},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.64400000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.36299999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47899999999999998,"15min":0.5},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.495,"15min":0.503},"min":{"1min":0.308,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71999999999999997,"5min":1.524,"15min":1.524},"last":0.39600000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.51500000000000001},"min":{"1min":0.33400000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.73899999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54800000000000004,"15min":0.54600000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.77600000000000002,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.45700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2727088128,"data_stored":3060078507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6058119168,"data_stored":7115118331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3342643200,"data_stored":3810324698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4321288192,"data_stored":4942656354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25169920,"data_stored":25165986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:06:01.318 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-07-02T14:06:01.318 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:06:01.318 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2023-07-02T14:06:01.319 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-07-02T14:06:01.334 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:01.334 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:01.335 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:01.335 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:01.335 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:01.344 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:06:01.427 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:01.427+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:01.467 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:06:01.472 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 1 2023-07-02T14:06:01.494 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:01.494+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:01.515 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:01.515 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:01.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:01.515 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:01.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.516 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:01.516 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:06:01.516 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 202.22270670399999, 2023-07-02T14:06:01.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:06:01.516 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:01.516 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:01.516 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:01.517 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:01.517 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:01.517 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:06:01.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.517 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:01.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:01.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:06:01.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:01.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:01.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:06:01.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:01.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:01.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:06:01.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:06:01.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:01.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:06:01.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:06:01.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:01.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:06:01.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:06:01.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:01.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:06:01.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:06:01.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:01.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:06:01.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:06:01.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:01.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:06:01.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:06:01.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:01.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:06:01.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:06:01.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:01.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:06:01.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:06:01.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:01.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:06:01.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:06:01.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:01.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:06:01.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:06:01.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:01.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:06:01.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:06:01.525 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.526 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.526 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.526 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:01.526 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:06:01.526 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 202.222471681, 2023-07-02T14:06:01.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:06:01.527 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:01.527 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:01.527 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:01.527 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:01.527 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:01.527 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:06:01.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.528 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:01.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:01.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:06:01.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:01.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:01.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:06:01.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:01.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:01.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:06:01.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:06:01.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:01.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:06:01.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:06:01.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:01.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:06:01.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:06:01.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:01.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:06:01.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:06:01.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:01.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:06:01.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:06:01.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:01.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:06:01.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:06:01.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:01.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:06:01.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:06:01.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:01.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:06:01.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:06:01.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:01.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:06:01.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:06:01.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:01.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:06:01.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:06:01.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:01.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:06:01.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:06:01.536 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.536 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.536 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.537 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:01.537 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:06:01.537 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 202.22227254699999, 2023-07-02T14:06:01.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:06:01.537 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:01.537 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:01.537 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:01.537 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:01.538 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:01.538 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:06:01.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.538 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:01.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:01.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:06:01.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:01.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:01.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:06:01.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:01.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:01.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:06:01.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:06:01.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:01.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:06:01.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:06:01.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:01.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:06:01.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:06:01.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:01.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:06:01.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:06:01.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:01.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:06:01.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:06:01.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:01.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:06:01.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:06:01.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:01.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:06:01.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:06:01.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:01.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:06:01.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:06:01.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:01.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:06:01.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:06:01.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:01.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:06:01.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:06:01.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:01.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:06:01.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:06:01.546 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.546 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.547 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.547 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:01.547 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:06:01.547 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 202.221907423, 2023-07-02T14:06:01.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:06:01.547 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:01.548 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:01.548 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:01.548 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:01.548 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:01.548 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:06:01.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.548 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:01.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:01.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:06:01.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:01.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:01.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:06:01.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:01.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:01.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:06:01.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:06:01.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:01.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:06:01.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:06:01.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:01.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:06:01.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:06:01.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:01.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:06:01.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:06:01.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:01.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:06:01.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:06:01.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:01.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:06:01.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:06:01.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:01.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:06:01.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:06:01.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:01.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:06:01.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:06:01.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:01.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:06:01.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:06:01.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:01.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:06:01.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:06:01.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:01.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:06:01.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:06:01.557 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.557 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.557 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.557 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:01.557 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:06:01.558 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 202.221826362, 2023-07-02T14:06:01.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:06:01.558 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:01.558 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:01.558 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:01.558 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:01.558 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:01.558 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:06:01.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.559 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:01.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:01.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:06:01.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:01.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:01.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:06:01.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:01.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:01.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:06:01.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:06:01.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:01.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:06:01.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:06:01.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:01.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:06:01.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:06:01.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:01.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:06:01.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:06:01.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:01.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:06:01.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:06:01.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:01.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:06:01.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:06:01.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:01.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:06:01.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:06:01.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:01.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:06:01.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:06:01.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:01.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:06:01.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:06:01.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:01.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:06:01.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:06:01.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:01.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:06:01.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:06:01.567 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.567 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.567 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.568 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:01.568 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:06:01.568 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 202.22173469099999, 2023-07-02T14:06:01.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:06:01.568 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:01.568 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:01.568 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:01.569 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:01.569 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:01.569 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:06:01.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.569 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:01.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:01.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:06:01.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:01.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:01.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:06:01.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:01.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:01.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:06:01.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:06:01.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:01.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:06:01.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:06:01.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:01.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:06:01.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:06:01.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:01.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:06:01.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:06:01.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:01.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:06:01.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:06:01.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:01.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:06:01.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:06:01.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:01.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:06:01.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:06:01.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:01.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:06:01.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:06:01.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:01.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:06:01.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:06:01.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:01.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:06:01.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:06:01.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:01.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:06:01.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:06:01.577 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.578 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.578 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.578 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:01.578 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:06:01.578 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 202.22139524900001, 2023-07-02T14:06:01.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:06:01.579 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:01.579 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:01.579 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:01.579 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:01.579 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:01.579 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:06:01.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.579 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:01.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:01.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:06:01.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:01.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:01.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:06:01.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:01.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:01.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:06:01.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:06:01.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:01.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:06:01.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:06:01.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:01.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:06:01.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:06:01.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:01.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:06:01.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:06:01.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:01.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:06:01.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:06:01.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:01.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:06:01.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:06:01.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:01.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:06:01.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:06:01.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:01.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:06:01.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:06:01.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:01.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:06:01.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:06:01.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:01.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:06:01.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:06:01.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:01.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:06:01.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:06:01.588 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.588 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.588 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.588 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:01.589 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:01.589 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 202.22120919599999, 2023-07-02T14:06:01.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:06:01.589 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:01.589 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:01.589 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:01.589 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:01.589 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:01.590 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:06:01.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.590 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:01.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:01.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:01.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:01.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:01.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:01.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:01.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:01.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:01.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:06:01.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:01.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:06:01.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:06:01.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:01.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:06:01.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:06:01.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:01.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:06:01.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:06:01.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:01.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:06:01.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:06:01.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:01.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:06:01.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:06:01.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:01.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:06:01.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:06:01.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:01.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:06:01.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:06:01.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:01.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:06:01.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:06:01.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:01.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:06:01.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:06:01.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:01.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:06:01.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:06:01.598 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.598 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.598 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.599 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:01.599 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:06:01.599 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 202.221051394, 2023-07-02T14:06:01.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:06:01.599 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:01.599 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:01.600 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:01.600 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:01.600 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:01.600 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:06:01.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.600 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:01.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:01.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:06:01.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:01.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:01.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:06:01.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:01.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:01.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:06:01.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:06:01.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:01.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:06:01.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:06:01.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:01.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:06:01.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:06:01.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.604 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:01.604 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:06:01.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:06:01.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.604 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:01.604 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:06:01.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:06:01.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:01.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:06:01.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:06:01.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:01.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:06:01.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:06:01.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:01.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:06:01.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:06:01.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:01.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:06:01.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:06:01.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:01.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:06:01.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:06:01.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:01.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:06:01.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:06:01.608 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.609 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.609 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.609 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:01.609 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:06:01.609 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 202.220859612, 2023-07-02T14:06:01.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:06:01.610 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:01.610 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:01.610 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:01.610 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:01.610 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:01.610 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:06:01.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.610 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:01.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:01.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:06:01.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:01.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:01.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:06:01.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:01.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:01.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:06:01.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:06:01.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:01.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:06:01.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:06:01.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:01.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:06:01.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:06:01.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:01.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:06:01.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:06:01.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:01.615 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:06:01.615 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:06:01.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:01.615 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:06:01.615 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:06:01.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.616 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.616 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:01.616 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:06:01.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:06:01.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.616 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:01.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:06:01.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:06:01.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:01.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:06:01.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:06:01.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:01.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:06:01.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:06:01.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:01.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:06:01.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:06:01.619 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.619 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.619 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.619 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:01.620 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:01.620 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 202.220683952, 2023-07-02T14:06:01.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:06:01.620 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:01.620 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:01.620 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:01.620 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:01.620 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:01.621 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:06:01.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.621 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:01.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:01.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:01.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:01.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:01.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:01.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:01.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:01.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:01.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:06:01.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:01.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:06:01.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:06:01.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:01.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:06:01.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:06:01.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:01.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:06:01.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:06:01.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.625 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:01.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:06:01.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:06:01.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:01.626 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:06:01.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:06:01.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:01.626 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:06:01.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:06:01.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:01.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:06:01.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:06:01.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:01.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:06:01.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:06:01.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:01.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:06:01.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:06:01.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:01.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:06:01.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:06:01.629 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.629 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.629 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.630 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:01.630 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:06:01.630 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 202.22058976400001, 2023-07-02T14:06:01.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:06:01.630 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:01.630 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:01.631 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:01.631 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:01.631 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:01.631 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:06:01.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.631 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:01.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:01.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:06:01.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:01.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:01.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:06:01.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:01.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:01.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:06:01.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:06:01.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:01.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:06:01.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:06:01.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:01.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:06:01.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:06:01.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:01.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:06:01.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:06:01.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:01.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:06:01.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:06:01.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:01.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:06:01.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:06:01.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:01.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:06:01.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:06:01.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:01.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:06:01.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:06:01.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:01.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:06:01.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:06:01.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:01.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:06:01.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:06:01.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:01.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:06:01.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:06:01.639 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.640 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.640 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.640 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:01.640 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:06:01.640 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 202.220386378, 2023-07-02T14:06:01.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:06:01.641 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:01.641 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:01.641 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:01.641 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:01.641 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:01.641 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:06:01.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.642 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:01.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:01.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:06:01.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:01.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:01.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:06:01.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:01.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:01.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:06:01.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:06:01.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:01.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:06:01.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:06:01.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:01.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:06:01.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:06:01.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:01.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:06:01.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:06:01.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:01.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:06:01.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:06:01.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:01.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:06:01.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:06:01.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:01.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:06:01.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:06:01.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:01.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:06:01.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:06:01.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:01.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:06:01.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:06:01.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:01.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:06:01.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:06:01.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:01.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:06:01.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:06:01.650 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.650 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.650 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.651 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:01.651 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:06:01.651 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 202.22023572200001, 2023-07-02T14:06:01.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:06:01.651 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:01.651 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:01.651 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:01.651 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:01.652 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:01.652 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:06:01.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.652 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:01.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:01.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:06:01.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:01.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:01.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:06:01.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:01.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:01.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:06:01.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:06:01.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:01.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:06:01.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:06:01.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:01.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:06:01.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:06:01.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:01.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:06:01.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:06:01.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:01.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:06:01.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:06:01.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:01.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:06:01.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:06:01.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:01.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:06:01.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:06:01.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:01.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:06:01.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:06:01.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:01.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:06:01.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:06:01.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:01.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:06:01.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:06:01.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:01.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:06:01.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:06:01.660 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.660 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.661 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.661 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:01.661 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:01.661 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 202.22016301900001, 2023-07-02T14:06:01.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:06:01.661 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:01.662 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:01.662 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:01.662 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:01.662 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:01.662 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:06:01.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.662 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:01.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:01.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:01.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:01.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:01.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:01.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:01.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:01.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:01.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:06:01.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:01.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:06:01.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:06:01.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:01.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:06:01.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:06:01.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:01.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:06:01.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:06:01.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:01.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:06:01.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:06:01.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:01.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:06:01.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:06:01.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:01.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:06:01.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:06:01.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:01.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:06:01.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:06:01.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:01.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:06:01.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:06:01.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:01.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:06:01.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:06:01.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:01.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:06:01.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:06:01.671 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.671 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.671 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.671 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:01.671 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:01.672 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 202.220046029, 2023-07-02T14:06:01.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:06:01.672 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:01.672 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:01.672 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:01.672 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:01.672 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:01.672 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:06:01.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.673 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:01.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:01.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:01.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:01.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:01.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:01.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:01.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:01.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:01.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:06:01.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:01.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:06:01.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:06:01.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:01.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:06:01.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:06:01.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:01.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:06:01.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:06:01.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:01.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:06:01.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:06:01.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:01.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:06:01.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:06:01.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:01.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:06:01.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:06:01.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:01.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:06:01.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:06:01.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:01.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:06:01.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:06:01.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:01.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:06:01.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:06:01.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:01.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:06:01.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:06:01.681 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.681 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.681 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.682 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:01.682 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:06:01.682 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 202.219896578, 2023-07-02T14:06:01.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:06:01.682 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:01.682 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:01.682 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:01.682 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:01.683 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:01.683 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:06:01.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.683 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:01.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:01.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:06:01.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:01.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:01.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:06:01.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:01.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:01.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:06:01.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:06:01.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:01.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:06:01.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:06:01.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:01.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:06:01.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:06:01.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:01.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:06:01.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:06:01.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:01.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:06:01.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:06:01.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:01.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:06:01.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:06:01.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:01.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:06:01.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:06:01.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:01.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:06:01.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:06:01.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:01.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:06:01.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:06:01.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:01.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:06:01.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:06:01.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:01.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:06:01.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:06:01.691 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.691 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.692 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.692 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:01.692 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:06:01.692 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 202.198420605, 2023-07-02T14:06:01.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:06:01.692 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:01.693 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:01.693 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:01.693 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:01.693 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:01.693 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:06:01.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.693 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:01.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:01.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:06:01.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:01.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:01.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:06:01.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:01.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:01.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:06:01.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:06:01.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:01.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:06:01.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:06:01.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:01.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:06:01.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:06:01.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:01.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:06:01.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:06:01.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:01.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:06:01.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:06:01.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:01.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:06:01.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:06:01.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:01.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:06:01.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:06:01.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:01.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:06:01.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:06:01.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:01.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:06:01.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:06:01.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:01.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:06:01.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:06:01.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:01.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:06:01.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:06:01.702 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.702 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.702 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.702 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:01.702 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:06:01.703 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 202.198288069, 2023-07-02T14:06:01.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:06:01.703 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:01.703 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:01.703 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:01.703 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:01.703 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:01.703 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:06:01.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.704 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:01.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:01.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:06:01.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:01.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:01.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:06:01.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:01.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:01.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:06:01.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:06:01.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:01.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:06:01.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:06:01.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:01.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:06:01.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:06:01.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:01.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:06:01.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:06:01.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:01.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:06:01.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:06:01.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:01.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:06:01.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:06:01.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:01.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:06:01.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:06:01.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:01.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:06:01.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:06:01.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:01.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:06:01.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:06:01.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:01.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:06:01.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:06:01.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:01.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:06:01.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:06:01.712 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.712 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.712 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.713 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:01.713 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:06:01.713 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 202.13695140799999, 2023-07-02T14:06:01.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:06:01.713 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:01.713 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:01.714 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:01.714 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:01.714 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:01.714 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:06:01.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.714 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:01.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:01.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:06:01.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:01.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:01.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:06:01.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:01.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:01.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:06:01.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:06:01.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:01.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:06:01.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:06:01.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:01.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:06:01.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:06:01.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:01.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:06:01.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:06:01.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:01.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:06:01.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:06:01.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:01.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:06:01.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:06:01.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:01.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:06:01.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:06:01.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:01.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:06:01.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:06:01.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:01.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:06:01.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:06:01.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:01.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:06:01.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:06:01.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:01.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:01.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:01.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:06:01.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:06:01.722 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.723 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.723 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.723 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:01.723 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:01.723 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:01.725 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:06:01.725 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:06:01.726 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:01.726 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 175, 2023-07-02T14:06:01.726 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2", 2023-07-02T14:06:01.726 INFO:teuthology.orchestra.run.smithi016.stdout: "created": "2023-07-02T14:01:37.298060+0000", 2023-07-02T14:06:01.726 INFO:teuthology.orchestra.run.smithi016.stdout: "modified": "2023-07-02T14:06:01.427348+0000", 2023-07-02T14:06:01.726 INFO:teuthology.orchestra.run.smithi016.stdout: "last_up_change": "2023-07-02T14:05:52.397969+0000", 2023-07-02T14:06:01.726 INFO:teuthology.orchestra.run.smithi016.stdout: "last_in_change": "2023-07-02T14:04:41.351678+0000", 2023-07-02T14:06:01.726 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-07-02T14:06:01.727 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_num": 5801984, 2023-07-02T14:06:01.727 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_set": [ 2023-07-02T14:06:01.727 INFO:teuthology.orchestra.run.smithi016.stdout: "noscrub", 2023-07-02T14:06:01.727 INFO:teuthology.orchestra.run.smithi016.stdout: "pglog_hardlimit", 2023-07-02T14:06:01.727 INFO:teuthology.orchestra.run.smithi016.stdout: "purged_snapdirs", 2023-07-02T14:06:01.727 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_deletes", 2023-07-02T14:06:01.727 INFO:teuthology.orchestra.run.smithi016.stdout: "sortbitwise" 2023-07-02T14:06:01.728 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:06:01.728 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_version": 6, 2023-07-02T14:06:01.728 INFO:teuthology.orchestra.run.smithi016.stdout: "full_ratio": 0.94999998807907104, 2023-07-02T14:06:01.728 INFO:teuthology.orchestra.run.smithi016.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-07-02T14:06:01.728 INFO:teuthology.orchestra.run.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2023-07-02T14:06:01.728 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_snapshot": "", 2023-07-02T14:06:01.728 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_max": 2, 2023-07-02T14:06:01.728 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd": 6, 2023-07-02T14:06:01.729 INFO:teuthology.orchestra.run.smithi016.stdout: "require_min_compat_client": "luminous", 2023-07-02T14:06:01.729 INFO:teuthology.orchestra.run.smithi016.stdout: "min_compat_client": "luminous", 2023-07-02T14:06:01.729 INFO:teuthology.orchestra.run.smithi016.stdout: "require_osd_release": "pacific", 2023-07-02T14:06:01.729 INFO:teuthology.orchestra.run.smithi016.stdout: "pools": [ 2023-07-02T14:06:01.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.729 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 1, 2023-07-02T14:06:01.729 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "device_health_metrics", 2023-07-02T14:06:01.729 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:45.323455+0000", 2023-07-02T14:06:01.730 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2023-07-02T14:06:01.730 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2023-07-02T14:06:01.730 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:06:01.730 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:06:01.730 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:06:01.730 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:06:01.730 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:06:01.731 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:06:01.731 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:06:01.731 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:06:01.731 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:06:01.731 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:06:01.731 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 1, 2023-07-02T14:06:01.731 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 1, 2023-07-02T14:06:01.731 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 1, 2023-07-02T14:06:01.732 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 1, 2023-07-02T14:06:01.732 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 1, 2023-07-02T14:06:01.732 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:06:01.732 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:06:01.732 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:06:01.732 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:06:01.732 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:06:01.732 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:06:01.733 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:06:01.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.733 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "14", 2023-07-02T14:06:01.733 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:06:01.733 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:06:01.733 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:06:01.733 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:06:01.733 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:06:01.734 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2023-07-02T14:06:01.734 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2023-07-02T14:06:01.734 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:06:01.734 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:06:01.734 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:06:01.734 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:06:01.734 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:06:01.734 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:06:01.735 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:06:01.735 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:06:01.735 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:06:01.735 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:06:01.735 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:06:01.735 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:06:01.735 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:06:01.735 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:06:01.736 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:06:01.736 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:06:01.736 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:06:01.736 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:06:01.736 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:06:01.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.736 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:06:01.737 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:06:01.737 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:06:01.737 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:06:01.737 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:06:01.737 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:06:01.737 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:06:01.737 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:06:01.737 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:06:01.738 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:06:01.738 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:06:01.738 INFO:teuthology.orchestra.run.smithi016.stdout: "options": { 2023-07-02T14:06:01.738 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_max": 32, 2023-07-02T14:06:01.738 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_min": 1 2023-07-02T14:06:01.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.738 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:06:01.738 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_devicehealth": {} 2023-07-02T14:06:01.739 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.739 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 2, 2023-07-02T14:06:01.739 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "rbd", 2023-07-02T14:06:01.739 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:47.448897+0000", 2023-07-02T14:06:01.739 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 8193, 2023-07-02T14:06:01.739 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2023-07-02T14:06:01.740 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:06:01.740 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:06:01.740 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:06:01.740 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:06:01.740 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:06:01.740 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:06:01.740 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:06:01.740 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:06:01.741 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:06:01.741 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:06:01.741 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2023-07-02T14:06:01.741 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2023-07-02T14:06:01.741 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2023-07-02T14:06:01.741 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2023-07-02T14:06:01.741 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2023-07-02T14:06:01.741 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:06:01.742 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:06:01.742 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:06:01.742 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:06:01.742 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:06:01.742 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:06:01.742 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:06:01.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.743 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "175", 2023-07-02T14:06:01.743 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:06:01.743 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:06:01.743 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:06:01.743 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:06:01.743 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:06:01.743 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 50, 2023-07-02T14:06:01.743 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 175, 2023-07-02T14:06:01.744 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:06:01.744 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:06:01.744 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:06:01.744 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:06:01.744 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:06:01.744 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:06:01.744 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:06:01.744 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:06:01.745 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:06:01.745 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:06:01.745 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:06:01.745 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:06:01.745 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:06:01.745 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:06:01.745 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:06:01.745 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:06:01.746 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:06:01.746 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:06:01.746 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:06:01.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.746 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:06:01.746 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:06:01.746 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:06:01.746 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:06:01.747 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:06:01.747 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:06:01.747 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:06:01.747 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:06:01.747 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:06:01.747 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:06:01.747 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:06:01.747 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2023-07-02T14:06:01.748 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:06:01.748 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2023-07-02T14:06:01.748 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.748 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.748 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:06:01.748 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:06:01.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.749 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:06:01.749 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "43c446fe-c811-4ee3-99e8-d7452edfc1c1", 2023-07-02T14:06:01.749 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:06:01.749 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:06:01.749 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:06:01.749 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:06:01.749 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 11, 2023-07-02T14:06:01.749 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 148, 2023-07-02T14:06:01.750 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 166, 2023-07-02T14:06:01.750 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 166, 2023-07-02T14:06:01.750 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 149, 2023-07-02T14:06:01.750 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:06:01.750 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:06:01.750 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.750 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.751 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2023-07-02T14:06:01.751 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 115688 2023-07-02T14:06:01.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.751 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.751 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2023-07-02T14:06:01.751 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 115688 2023-07-02T14:06:01.751 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.752 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.752 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:06:01.752 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.752 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.752 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6802", 2023-07-02T14:06:01.752 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 115688 2023-07-02T14:06:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6803", 2023-07-02T14:06:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 115688 2023-07-02T14:06:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.754 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:06:01.754 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.754 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.754 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6806", 2023-07-02T14:06:01.754 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 115688 2023-07-02T14:06:01.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.755 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.755 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6807", 2023-07-02T14:06:01.755 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 115688 2023-07-02T14:06:01.755 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.755 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.755 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:06:01.755 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.756 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.756 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6804", 2023-07-02T14:06:01.756 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 115688 2023-07-02T14:06:01.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.757 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.757 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6805", 2023-07-02T14:06:01.757 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 115688 2023-07-02T14:06:01.757 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.757 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.758 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6801/115688", 2023-07-02T14:06:01.758 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6803/115688", 2023-07-02T14:06:01.758 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6807/115688", 2023-07-02T14:06:01.758 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6805/115688", 2023-07-02T14:06:01.758 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:06:01.758 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:06:01.759 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:06:01.759 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.759 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:06:01.759 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b", 2023-07-02T14:06:01.759 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:06:01.759 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:06:01.760 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:06:01.760 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:06:01.760 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:06:01.760 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:06:01.760 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:06:01.760 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 171, 2023-07-02T14:06:01.760 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:06:01.761 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:06:01.761 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:06:01.761 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.761 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.761 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6808", 2023-07-02T14:06:01.761 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:06:01.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.762 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.762 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6809", 2023-07-02T14:06:01.762 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:06:01.762 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.762 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.763 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:06:01.763 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.763 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.763 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6810", 2023-07-02T14:06:01.763 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:06:01.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.764 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.764 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6811", 2023-07-02T14:06:01.764 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:06:01.764 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.764 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.765 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:06:01.765 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.765 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.765 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6814", 2023-07-02T14:06:01.765 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:06:01.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.766 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.766 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6815", 2023-07-02T14:06:01.766 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:06:01.766 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.766 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.767 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:06:01.767 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.767 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.767 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6812", 2023-07-02T14:06:01.768 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:06:01.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.768 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.768 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6813", 2023-07-02T14:06:01.768 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:06:01.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.769 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.769 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6809/83360", 2023-07-02T14:06:01.769 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6811/83360", 2023-07-02T14:06:01.769 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6815/83360", 2023-07-02T14:06:01.769 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6813/83360", 2023-07-02T14:06:01.769 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:06:01.770 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:06:01.770 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:06:01.770 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.770 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:06:01.770 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "99f8ea0a-5a71-4abe-b087-8e891fd11d1b", 2023-07-02T14:06:01.770 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:06:01.771 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:06:01.771 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:06:01.771 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 0.9522247314453125, 2023-07-02T14:06:01.771 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:06:01.771 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:06:01.771 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:06:01.771 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 140, 2023-07-02T14:06:01.772 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:06:01.772 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:06:01.772 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:06:01.772 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.772 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.772 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6816", 2023-07-02T14:06:01.772 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:06:01.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.773 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.773 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6817", 2023-07-02T14:06:01.773 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:06:01.773 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.773 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.774 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:06:01.774 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.774 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.774 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6818", 2023-07-02T14:06:01.774 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:06:01.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.775 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.775 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6819", 2023-07-02T14:06:01.775 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:06:01.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.775 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.775 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:06:01.776 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.776 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6822", 2023-07-02T14:06:01.776 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:06:01.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.777 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6823", 2023-07-02T14:06:01.777 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:06:01.777 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.777 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.777 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:06:01.777 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.778 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.778 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6820", 2023-07-02T14:06:01.778 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:06:01.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.778 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.778 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6821", 2023-07-02T14:06:01.779 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:06:01.779 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.779 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.779 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6817/83361", 2023-07-02T14:06:01.779 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6819/83361", 2023-07-02T14:06:01.779 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6823/83361", 2023-07-02T14:06:01.779 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6821/83361", 2023-07-02T14:06:01.780 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:06:01.780 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:06:01.780 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:06:01.780 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.780 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:06:01.781 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "d285b4d5-0bef-421a-b84a-943e485b10c2", 2023-07-02T14:06:01.781 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:06:01.781 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:06:01.781 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:06:01.781 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:06:01.781 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 11, 2023-07-02T14:06:01.781 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 87, 2023-07-02T14:06:01.781 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 133, 2023-07-02T14:06:01.782 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 170, 2023-07-02T14:06:01.782 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 88, 2023-07-02T14:06:01.782 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:06:01.782 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:06:01.782 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.782 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.782 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6808", 2023-07-02T14:06:01.783 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 99977 2023-07-02T14:06:01.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.783 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.783 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6809", 2023-07-02T14:06:01.783 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 99977 2023-07-02T14:06:01.783 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.783 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.784 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:06:01.784 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.784 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.784 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6810", 2023-07-02T14:06:01.784 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 99977 2023-07-02T14:06:01.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.785 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.785 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6811", 2023-07-02T14:06:01.785 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 99977 2023-07-02T14:06:01.785 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.785 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.786 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:06:01.786 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.786 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.786 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6814", 2023-07-02T14:06:01.786 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 99977 2023-07-02T14:06:01.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.787 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.787 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6815", 2023-07-02T14:06:01.787 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 99977 2023-07-02T14:06:01.787 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.787 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.787 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:06:01.788 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.788 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.788 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6812", 2023-07-02T14:06:01.788 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 99977 2023-07-02T14:06:01.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.789 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.789 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6813", 2023-07-02T14:06:01.789 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 99977 2023-07-02T14:06:01.789 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.789 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.789 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6809/99977", 2023-07-02T14:06:01.790 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6811/99977", 2023-07-02T14:06:01.790 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6815/99977", 2023-07-02T14:06:01.790 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6813/99977", 2023-07-02T14:06:01.790 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:06:01.790 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:06:01.790 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:06:01.790 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.791 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:06:01.791 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "7ba0b0db-e407-4c63-8278-aab09f3082c4", 2023-07-02T14:06:01.791 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:06:01.791 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 0, 2023-07-02T14:06:01.791 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 0, 2023-07-02T14:06:01.791 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 0.23297119140625, 2023-07-02T14:06:01.791 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:06:01.792 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:06:01.792 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:06:01.792 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 120, 2023-07-02T14:06:01.792 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:06:01.792 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:06:01.792 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:06:01.792 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.793 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.793 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6800", 2023-07-02T14:06:01.793 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:06:01.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.793 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.793 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6801", 2023-07-02T14:06:01.794 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:06:01.794 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.794 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.794 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:06:01.794 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.795 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.795 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6802", 2023-07-02T14:06:01.795 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:06:01.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.795 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.795 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6803", 2023-07-02T14:06:01.796 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:06:01.796 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.796 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.796 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:06:01.796 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.797 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.797 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6806", 2023-07-02T14:06:01.797 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:06:01.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.797 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.797 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6807", 2023-07-02T14:06:01.798 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:06:01.798 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.798 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.798 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:06:01.798 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.799 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.799 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6804", 2023-07-02T14:06:01.799 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:06:01.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.799 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.799 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6805", 2023-07-02T14:06:01.799 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:06:01.800 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.800 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.800 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6801/83351", 2023-07-02T14:06:01.800 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6803/83351", 2023-07-02T14:06:01.800 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6807/83351", 2023-07-02T14:06:01.800 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6805/83351", 2023-07-02T14:06:01.800 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:06:01.801 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:06:01.801 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:06:01.801 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.801 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:06:01.801 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "29663bfa-f6fe-4ce7-b45a-a9b7b4424c42", 2023-07-02T14:06:01.801 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:06:01.802 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 0, 2023-07-02T14:06:01.802 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 0, 2023-07-02T14:06:01.802 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:06:01.802 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:06:01.802 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:06:01.802 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:06:01.802 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 34, 2023-07-02T14:06:01.802 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:06:01.803 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:06:01.803 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:06:01.803 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.803 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.803 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6816", 2023-07-02T14:06:01.803 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:06:01.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.804 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.804 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6817", 2023-07-02T14:06:01.804 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:06:01.804 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.804 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.805 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:06:01.805 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.805 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.805 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6818", 2023-07-02T14:06:01.805 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:06:01.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.806 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.806 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6819", 2023-07-02T14:06:01.806 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:06:01.806 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.806 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.806 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:06:01.806 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.807 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.807 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6822", 2023-07-02T14:06:01.807 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:06:01.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.808 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.808 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6823", 2023-07-02T14:06:01.808 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:06:01.808 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.808 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.808 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:06:01.809 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:06:01.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.809 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:06:01.809 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6820", 2023-07-02T14:06:01.809 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:06:01.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.809 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:06:01.810 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6821", 2023-07-02T14:06:01.810 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:06:01.810 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.810 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.810 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6817/83352", 2023-07-02T14:06:01.810 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6819/83352", 2023-07-02T14:06:01.810 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6823/83352", 2023-07-02T14:06:01.811 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6821/83352", 2023-07-02T14:06:01.811 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:06:01.811 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:06:01.811 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:06:01.811 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.811 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.811 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:06:01.811 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_xinfo": [ 2023-07-02T14:06:01.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.812 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:06:01.812 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:06:01.812 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:06:01.812 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:06:01.812 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:06:01.812 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:06:01.813 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.001178+0000", 2023-07-02T14:06:01.813 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 149 2023-07-02T14:06:01.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.813 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:06:01.813 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:06:01.813 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:06:01.813 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:06:01.814 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:06:01.814 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:06:01.814 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.184181+0000", 2023-07-02T14:06:01.814 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:06:01.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.814 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:06:01.814 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:06:01.815 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:06:01.815 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:06:01.815 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:06:01.815 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:06:01.815 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.274854+0000", 2023-07-02T14:06:01.815 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:06:01.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.816 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:06:01.816 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:06:01.816 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:06:01.816 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:06:01.816 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:06:01.816 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:06:01.816 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.186217+0000", 2023-07-02T14:06:01.817 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 88 2023-07-02T14:06:01.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.817 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:06:01.817 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:06:01.817 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:06:01.817 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:06:01.817 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:06:01.818 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 65536, 2023-07-02T14:06:01.818 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.247442+0000", 2023-07-02T14:06:01.818 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:06:01.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.818 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:06:01.818 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:06:01.818 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:06:01.819 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:06:01.819 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:06:01.819 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 65536, 2023-07-02T14:06:01.819 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.220075+0000", 2023-07-02T14:06:01.819 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:06:01.819 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.819 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:06:01.819 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap": [ 2023-07-02T14:06:01.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.820 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "1.0", 2023-07-02T14:06:01.820 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:06:01.820 INFO:teuthology.orchestra.run.smithi016.stdout: 0, 2023-07-02T14:06:01.820 INFO:teuthology.orchestra.run.smithi016.stdout: 3 2023-07-02T14:06:01.820 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.821 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.0", 2023-07-02T14:06:01.821 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:06:01.821 INFO:teuthology.orchestra.run.smithi016.stdout: 4, 2023-07-02T14:06:01.821 INFO:teuthology.orchestra.run.smithi016.stdout: 1 2023-07-02T14:06:01.821 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.822 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.2", 2023-07-02T14:06:01.822 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:06:01.822 INFO:teuthology.orchestra.run.smithi016.stdout: 1, 2023-07-02T14:06:01.822 INFO:teuthology.orchestra.run.smithi016.stdout: 5 2023-07-02T14:06:01.822 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:01.822 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.4", 2023-07-02T14:06:01.823 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:06:01.823 INFO:teuthology.orchestra.run.smithi016.stdout: 5, 2023-07-02T14:06:01.823 INFO:teuthology.orchestra.run.smithi016.stdout: 0 2023-07-02T14:06:01.823 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:01.823 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.823 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:06:01.823 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_items": [], 2023-07-02T14:06:01.823 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_temp": [], 2023-07-02T14:06:01.824 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_temp": [], 2023-07-02T14:06:01.824 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklist": {}, 2023-07-02T14:06:01.824 INFO:teuthology.orchestra.run.smithi016.stdout: "range_blocklist": {}, 2023-07-02T14:06:01.824 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profiles": { 2023-07-02T14:06:01.824 INFO:teuthology.orchestra.run.smithi016.stdout: "default": { 2023-07-02T14:06:01.824 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2023-07-02T14:06:01.825 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "2", 2023-07-02T14:06:01.825 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "1", 2023-07-02T14:06:01.825 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "jerasure", 2023-07-02T14:06:01.825 INFO:teuthology.orchestra.run.smithi016.stdout: "ruleset-failure-domain": "osd", 2023-07-02T14:06:01.825 INFO:teuthology.orchestra.run.smithi016.stdout: "technique": "reed_sol_van" 2023-07-02T14:06:01.825 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:01.825 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps_queue": [], 2023-07-02T14:06:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "new_removed_snaps": [], 2023-07-02T14:06:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "new_purged_snaps": [], 2023-07-02T14:06:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_node_flags": {}, 2023-07-02T14:06:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "device_class_flags": {}, 2023-07-02T14:06:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode": { 2023-07-02T14:06:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_enabled": false, 2023-07-02T14:06:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_bucket_count": 0, 2023-07-02T14:06:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "degraded_stretch_mode": 0, 2023-07-02T14:06:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: "recovering_stretch_mode": 0, 2023-07-02T14:06:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_bucket": 0 2023-07-02T14:06:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:01.827 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:01.828 INFO:tasks.thrashosds.thrasher:j is {'epoch': 175, 'fsid': '9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2', 'created': '2023-07-02T14:01:37.298060+0000', 'modified': '2023-07-02T14:06:01.427348+0000', 'last_up_change': '2023-07-02T14:05:52.397969+0000', 'last_in_change': '2023-07-02T14:04:41.351678+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-07-02T14:01:45.323455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-02T14:01:47.448897+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '175', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 50, 'snap_epoch': 175, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '43c446fe-c811-4ee3-99e8-d7452edfc1c1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 11, 'last_clean_end': 148, 'up_from': 166, 'up_thru': 166, 'down_at': 149, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6800', 'nonce': 115688}, {'type': 'v1', 'addr': '172.21.15.16:6801', 'nonce': 115688}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6802', 'nonce': 115688}, {'type': 'v1', 'addr': '172.21.15.16:6803', 'nonce': 115688}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6806', 'nonce': 115688}, {'type': 'v1', 'addr': '172.21.15.16:6807', 'nonce': 115688}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6804', 'nonce': 115688}, {'type': 'v1', 'addr': '172.21.15.16:6805', 'nonce': 115688}]}, 'public_addr': '172.21.15.16:6801/115688', 'cluster_addr': '172.21.15.16:6803/115688', 'heartbeat_back_addr': '172.21.15.16:6807/115688', 'heartbeat_front_addr': '172.21.15.16:6805/115688', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 171, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6808', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6809', 'nonce': 83360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6810', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6811', 'nonce': 83360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6814', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6815', 'nonce': 83360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6812', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6813', 'nonce': 83360}]}, 'public_addr': '172.21.15.16:6809/83360', 'cluster_addr': '172.21.15.16:6811/83360', 'heartbeat_back_addr': '172.21.15.16:6815/83360', 'heartbeat_front_addr': '172.21.15.16:6813/83360', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '99f8ea0a-5a71-4abe-b087-8e891fd11d1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9522247314453125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 140, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6816', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6817', 'nonce': 83361}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6818', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6819', 'nonce': 83361}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6822', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6823', 'nonce': 83361}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6820', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6821', 'nonce': 83361}]}, 'public_addr': '172.21.15.16:6817/83361', 'cluster_addr': '172.21.15.16:6819/83361', 'heartbeat_back_addr': '172.21.15.16:6823/83361', 'heartbeat_front_addr': '172.21.15.16:6821/83361', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd285b4d5-0bef-421a-b84a-943e485b10c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 11, 'last_clean_end': 87, 'up_from': 133, 'up_thru': 170, 'down_at': 88, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6808', 'nonce': 99977}, {'type': 'v1', 'addr': '172.21.15.22:6809', 'nonce': 99977}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6810', 'nonce': 99977}, {'type': 'v1', 'addr': '172.21.15.22:6811', 'nonce': 99977}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6814', 'nonce': 99977}, {'type': 'v1', 'addr': '172.21.15.22:6815', 'nonce': 99977}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6812', 'nonce': 99977}, {'type': 'v1', 'addr': '172.21.15.22:6813', 'nonce': 99977}]}, 'public_addr': '172.21.15.22:6809/99977', 'cluster_addr': '172.21.15.22:6811/99977', 'heartbeat_back_addr': '172.21.15.22:6815/99977', 'heartbeat_front_addr': '172.21.15.22:6813/99977', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7ba0b0db-e407-4c63-8278-aab09f3082c4', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.23297119140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 120, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6800', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6801', 'nonce': 83351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6802', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6803', 'nonce': 83351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6806', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6807', 'nonce': 83351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6804', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6805', 'nonce': 83351}]}, 'public_addr': '172.21.15.22:6801/83351', 'cluster_addr': '172.21.15.22:6803/83351', 'heartbeat_back_addr': '172.21.15.22:6807/83351', 'heartbeat_front_addr': '172.21.15.22:6805/83351', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '29663bfa-f6fe-4ce7-b45a-a9b7b4424c42', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6816', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6817', 'nonce': 83352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6818', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6819', 'nonce': 83352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6822', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6823', 'nonce': 83352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6820', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6821', 'nonce': 83352}]}, 'public_addr': '172.21.15.22:6817/83352', 'cluster_addr': '172.21.15.22:6819/83352', 'heartbeat_back_addr': '172.21.15.22:6823/83352', 'heartbeat_front_addr': '172.21.15.22:6821/83352', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.001178+0000', 'dead_epoch': 149}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.184181+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.274854+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.186217+0000', 'dead_epoch': 88}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.247442+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.220075+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [0, 3]}, {'pgid': '2.0', 'osds': [4, 1]}, {'pgid': '2.2', 'osds': [1, 5]}, {'pgid': '2.4', 'osds': [5, 0]}], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-07-02T14:06:01.828 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2023-07-02T14:06:01.828 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:01.723+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:01.830 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:01.829+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:01.931 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:01.930+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:01.953 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:01.954 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:01.954 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:06:01.954 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:01.964 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:06:02.032 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:02.031+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:02.087 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:02.087 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:02.087 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:02.088 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:02.088 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:02.097 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:06:02.139 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:02.138+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:02.234 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:02.233+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:02.266 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:02.267 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:02.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:02.267 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:02.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.267 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:16940 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:02.267 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.855862+0000", 2023-07-02T14:06:02.268 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 19.406258081000001, 2023-07-02T14:06:02.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.103600949, 2023-07-02T14:06:02.268 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:02.268 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:02.268 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:02.268 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:02.268 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:02.269 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 16940 2023-07-02T14:06:02.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.269 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:02.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:02.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855862+0000", 2023-07-02T14:06:02.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:02.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:02.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855862+0000", 2023-07-02T14:06:02.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:02.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:02.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855862+0000", 2023-07-02T14:06:02.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.353e-06 2023-07-02T14:06:02.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.271 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:02.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855868+0000", 2023-07-02T14:06:02.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.746e-06 2023-07-02T14:06:02.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:02.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855873+0000", 2023-07-02T14:06:02.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4030000000000001e-06 2023-07-02T14:06:02.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:02.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855875+0000", 2023-07-02T14:06:02.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050000019999999999 2023-07-02T14:06:02.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.273 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.860875+0000", 2023-07-02T14:06:02.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3556000000000001e-05 2023-07-02T14:06:02.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.860919+0000", 2023-07-02T14:06:02.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055944549999999999 2023-07-02T14:06:02.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.866513+0000", 2023-07-02T14:06:02.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2478999999999999e-05 2023-07-02T14:06:02.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.275 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.275 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.866546+0000", 2023-07-02T14:06:02.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050944609999999998 2023-07-02T14:06:02.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.871640+0000", 2023-07-02T14:06:02.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2138000000000003e-05 2023-07-02T14:06:02.276 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.871672+0000", 2023-07-02T14:06:02.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014725571999999999 2023-07-02T14:06:02.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.277 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.277 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.886398+0000", 2023-07-02T14:06:02.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6910999999999999e-05 2023-07-02T14:06:02.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.277 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.886435+0000", 2023-07-02T14:06:02.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015334054 2023-07-02T14:06:02.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.901769+0000", 2023-07-02T14:06:02.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0036000000000001e-05 2023-07-02T14:06:02.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.279 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.901809+0000", 2023-07-02T14:06:02.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014746522999999999 2023-07-02T14:06:02.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.279 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.916555+0000", 2023-07-02T14:06:02.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3622999999999999e-05 2023-07-02T14:06:02.280 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.916599+0000", 2023-07-02T14:06:02.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099546679999999999 2023-07-02T14:06:02.280 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.926554+0000", 2023-07-02T14:06:02.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7073000000000002e-05 2023-07-02T14:06:02.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.926591+0000", 2023-07-02T14:06:02.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045803249999999997 2023-07-02T14:06:02.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.282 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.931171+0000", 2023-07-02T14:06:02.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9468000000000004e-05 2023-07-02T14:06:02.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.282 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.283 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.931271+0000", 2023-07-02T14:06:02.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078347380000000008 2023-07-02T14:06:02.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.283 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.939105+0000", 2023-07-02T14:06:02.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1427999999999994e-05 2023-07-02T14:06:02.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.939177+0000", 2023-07-02T14:06:02.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093866379999999992 2023-07-02T14:06:02.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.948563+0000", 2023-07-02T14:06:02.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3999999999999995e-05 2023-07-02T14:06:02.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:02.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.948647+0000", 2023-07-02T14:06:02.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00034591999999999999 2023-07-02T14:06:02.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:02.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.948993+0000", 2023-07-02T14:06:02.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056008810000000003 2023-07-02T14:06:02.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:02.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.954594+0000", 2023-07-02T14:06:02.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047713010000000004 2023-07-02T14:06:02.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:02.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959365+0000", 2023-07-02T14:06:02.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4107999999999998e-05 2023-07-02T14:06:02.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:02.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959410+0000", 2023-07-02T14:06:02.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3331999999999997e-05 2023-07-02T14:06:02.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:02.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959463+0000", 2023-07-02T14:06:02.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.103600949 2023-07-02T14:06:02.289 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.289 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.289 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.289 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:16941 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:02.289 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:06:02.290 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 19.406242435999999, 2023-07-02T14:06:02.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11449124500000001, 2023-07-02T14:06:02.290 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:02.290 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:02.290 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:02.290 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:02.290 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:02.290 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 16941 2023-07-02T14:06:02.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.291 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:02.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:02.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:06:02.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:02.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:02.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:06:02.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:06:02.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:02.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855877+0000", 2023-07-02T14:06:02.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3209999999999999e-06 2023-07-02T14:06:02.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:02.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855880+0000", 2023-07-02T14:06:02.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9299999999999999e-06 2023-07-02T14:06:02.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:02.294 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855883+0000", 2023-07-02T14:06:02.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6309999999999999e-06 2023-07-02T14:06:02.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:02.294 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855885+0000", 2023-07-02T14:06:02.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050516850000000002 2023-07-02T14:06:02.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.295 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.860937+0000", 2023-07-02T14:06:02.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1844999999999997e-05 2023-07-02T14:06:02.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.295 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.860979+0000", 2023-07-02T14:06:02.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055853480000000004 2023-07-02T14:06:02.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.866564+0000", 2023-07-02T14:06:02.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2821999999999998e-05 2023-07-02T14:06:02.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.866597+0000", 2023-07-02T14:06:02.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050938900000000002 2023-07-02T14:06:02.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.871691+0000", 2023-07-02T14:06:02.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1210000000000001e-05 2023-07-02T14:06:02.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.871722+0000", 2023-07-02T14:06:02.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014768687000000001 2023-07-02T14:06:02.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.886491+0000", 2023-07-02T14:06:02.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7248000000000003e-05 2023-07-02T14:06:02.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.886568+0000", 2023-07-02T14:06:02.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015403168 2023-07-02T14:06:02.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.901971+0000", 2023-07-02T14:06:02.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4035000000000002e-05 2023-07-02T14:06:02.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.902005+0000", 2023-07-02T14:06:02.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014618580000000001 2023-07-02T14:06:02.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.916624+0000", 2023-07-02T14:06:02.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.443e-05 2023-07-02T14:06:02.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.916668+0000", 2023-07-02T14:06:02.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099598989999999995 2023-07-02T14:06:02.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.926628+0000", 2023-07-02T14:06:02.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3123999999999999e-05 2023-07-02T14:06:02.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.926711+0000", 2023-07-02T14:06:02.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046031930000000002 2023-07-02T14:06:02.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.931314+0000", 2023-07-02T14:06:02.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3257000000000003e-05 2023-07-02T14:06:02.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.931358+0000", 2023-07-02T14:06:02.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078370969999999995 2023-07-02T14:06:02.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.939195+0000", 2023-07-02T14:06:02.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2522000000000003e-05 2023-07-02T14:06:02.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.939247+0000", 2023-07-02T14:06:02.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01118103 2023-07-02T14:06:02.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.950428+0000", 2023-07-02T14:06:02.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5001999999999996e-05 2023-07-02T14:06:02.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.307 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.950523+0000", 2023-07-02T14:06:02.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089648390000000005 2023-07-02T14:06:02.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959488+0000", 2023-07-02T14:06:02.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4569e-05 2023-07-02T14:06:02.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:02.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959533+0000", 2023-07-02T14:06:02.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026104299999999997 2023-07-02T14:06:02.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:02.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959794+0000", 2023-07-02T14:06:02.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025221789999999998 2023-07-02T14:06:02.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:02.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.962316+0000", 2023-07-02T14:06:02.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079080520000000005 2023-07-02T14:06:02.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:02.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970224+0000", 2023-07-02T14:06:02.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1265e-05 2023-07-02T14:06:02.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:02.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970255+0000", 2023-07-02T14:06:02.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000113581 2023-07-02T14:06:02.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:02.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970369+0000", 2023-07-02T14:06:02.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11449124500000001 2023-07-02T14:06:02.312 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.312 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.312 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.312 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17047 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:02.312 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.979346+0000", 2023-07-02T14:06:02.313 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 19.282774029999999, 2023-07-02T14:06:02.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.095586451000000003, 2023-07-02T14:06:02.313 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:02.313 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:02.313 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:02.313 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:02.313 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:02.313 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17047 2023-07-02T14:06:02.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.314 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:02.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:02.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979346+0000", 2023-07-02T14:06:02.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:02.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:02.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979346+0000", 2023-07-02T14:06:02.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:02.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:02.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979345+0000", 2023-07-02T14:06:02.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.775e-06 2023-07-02T14:06:02.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:02.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979350+0000", 2023-07-02T14:06:02.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8170000000000001e-06 2023-07-02T14:06:02.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:02.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979357+0000", 2023-07-02T14:06:02.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9539999999999998e-06 2023-07-02T14:06:02.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:02.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979361+0000", 2023-07-02T14:06:02.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01301737 2023-07-02T14:06:02.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992378+0000", 2023-07-02T14:06:02.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4696999999999997e-05 2023-07-02T14:06:02.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992413+0000", 2023-07-02T14:06:02.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061779579999999999 2023-07-02T14:06:02.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998591+0000", 2023-07-02T14:06:02.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3654999999999999e-05 2023-07-02T14:06:02.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998664+0000", 2023-07-02T14:06:02.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091576609999999992 2023-07-02T14:06:02.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.007822+0000", 2023-07-02T14:06:02.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7914000000000002e-05 2023-07-02T14:06:02.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.007870+0000", 2023-07-02T14:06:02.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059512150000000002 2023-07-02T14:06:02.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.322 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.013821+0000", 2023-07-02T14:06:02.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1442999999999999e-05 2023-07-02T14:06:02.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.322 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.013902+0000", 2023-07-02T14:06:02.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075954430000000003 2023-07-02T14:06:02.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021498+0000", 2023-07-02T14:06:02.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0065000000000001e-05 2023-07-02T14:06:02.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.324 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021528+0000", 2023-07-02T14:06:02.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056511080000000002 2023-07-02T14:06:02.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.324 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027179+0000", 2023-07-02T14:06:02.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8032999999999998e-05 2023-07-02T14:06:02.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027237+0000", 2023-07-02T14:06:02.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048459089999999998 2023-07-02T14:06:02.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032083+0000", 2023-07-02T14:06:02.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1186000000000003e-05 2023-07-02T14:06:02.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032114+0000", 2023-07-02T14:06:02.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055940670000000003 2023-07-02T14:06:02.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.037708+0000", 2023-07-02T14:06:02.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0913000000000003e-05 2023-07-02T14:06:02.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.037789+0000", 2023-07-02T14:06:02.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043937489999999997 2023-07-02T14:06:02.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042183+0000", 2023-07-02T14:06:02.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0321000000000007e-05 2023-07-02T14:06:02.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042253+0000", 2023-07-02T14:06:02.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073046439999999999 2023-07-02T14:06:02.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049558+0000", 2023-07-02T14:06:02.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0810000000000006e-05 2023-07-02T14:06:02.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049649+0000", 2023-07-02T14:06:02.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0094162250000000003 2023-07-02T14:06:02.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059065+0000", 2023-07-02T14:06:02.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2486999999999997e-05 2023-07-02T14:06:02.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059157+0000", 2023-07-02T14:06:02.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059730709999999999 2023-07-02T14:06:02.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.065131+0000", 2023-07-02T14:06:02.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9991999999999998e-05 2023-07-02T14:06:02.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:02.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.065180+0000", 2023-07-02T14:06:02.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00037239199999999998 2023-07-02T14:06:02.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:02.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.065553+0000", 2023-07-02T14:06:02.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037123600000000001 2023-07-02T14:06:02.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:02.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.069265+0000", 2023-07-02T14:06:02.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055193860000000003 2023-07-02T14:06:02.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:02.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.074785+0000", 2023-07-02T14:06:02.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9505e-05 2023-07-02T14:06:02.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:02.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.074814+0000", 2023-07-02T14:06:02.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000118282 2023-07-02T14:06:02.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:02.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.074932+0000", 2023-07-02T14:06:02.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.095586451000000003 2023-07-02T14:06:02.336 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.336 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.336 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.337 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17048 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:02.337 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.979365+0000", 2023-07-02T14:06:02.337 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 19.282754809, 2023-07-02T14:06:02.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.102723355, 2023-07-02T14:06:02.337 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:02.337 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:02.337 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:02.338 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:02.338 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:02.338 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17048 2023-07-02T14:06:02.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.338 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:02.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:02.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979365+0000", 2023-07-02T14:06:02.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:02.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:02.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979365+0000", 2023-07-02T14:06:02.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:02.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:02.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979365+0000", 2023-07-02T14:06:02.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4489999999999999e-06 2023-07-02T14:06:02.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:02.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979368+0000", 2023-07-02T14:06:02.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.178e-06 2023-07-02T14:06:02.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:02.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979374+0000", 2023-07-02T14:06:02.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0029999999999996e-06 2023-07-02T14:06:02.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:02.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979378+0000", 2023-07-02T14:06:02.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013053530000000001 2023-07-02T14:06:02.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992432+0000", 2023-07-02T14:06:02.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4631000000000001e-05 2023-07-02T14:06:02.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992467+0000", 2023-07-02T14:06:02.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062315440000000003 2023-07-02T14:06:02.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998698+0000", 2023-07-02T14:06:02.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2839999999999997e-05 2023-07-02T14:06:02.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998731+0000", 2023-07-02T14:06:02.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091807750000000004 2023-07-02T14:06:02.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.007912+0000", 2023-07-02T14:06:02.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0945999999999995e-05 2023-07-02T14:06:02.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.007983+0000", 2023-07-02T14:06:02.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059545220000000003 2023-07-02T14:06:02.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.013937+0000", 2023-07-02T14:06:02.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2709000000000001e-05 2023-07-02T14:06:02.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.013970+0000", 2023-07-02T14:06:02.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075831359999999999 2023-07-02T14:06:02.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021553+0000", 2023-07-02T14:06:02.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0125e-05 2023-07-02T14:06:02.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021583+0000", 2023-07-02T14:06:02.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056843680000000004 2023-07-02T14:06:02.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027268+0000", 2023-07-02T14:06:02.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0429e-05 2023-07-02T14:06:02.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027298+0000", 2023-07-02T14:06:02.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048443660000000001 2023-07-02T14:06:02.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032142+0000", 2023-07-02T14:06:02.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1180000000000003e-05 2023-07-02T14:06:02.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032174+0000", 2023-07-02T14:06:02.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005665944 2023-07-02T14:06:02.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.037839+0000", 2023-07-02T14:06:02.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6186000000000001e-05 2023-07-02T14:06:02.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.037886+0000", 2023-07-02T14:06:02.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044234330000000001 2023-07-02T14:06:02.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042309+0000", 2023-07-02T14:06:02.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6038999999999997e-05 2023-07-02T14:06:02.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042355+0000", 2023-07-02T14:06:02.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073303919999999998 2023-07-02T14:06:02.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049686+0000", 2023-07-02T14:06:02.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5858000000000001e-05 2023-07-02T14:06:02.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049721+0000", 2023-07-02T14:06:02.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0094923960000000002 2023-07-02T14:06:02.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059214+0000", 2023-07-02T14:06:02.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3883999999999998e-05 2023-07-02T14:06:02.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059298+0000", 2023-07-02T14:06:02.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080288240000000004 2023-07-02T14:06:02.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067326+0000", 2023-07-02T14:06:02.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0906999999999998e-05 2023-07-02T14:06:02.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067387+0000", 2023-07-02T14:06:02.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075801130000000003 2023-07-02T14:06:02.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.074968+0000", 2023-07-02T14:06:02.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1454e-05 2023-07-02T14:06:02.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:02.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.075009+0000", 2023-07-02T14:06:02.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025346800000000002 2023-07-02T14:06:02.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:02.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.075262+0000", 2023-07-02T14:06:02.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020188070000000001 2023-07-02T14:06:02.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:02.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077281+0000", 2023-07-02T14:06:02.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046643250000000004 2023-07-02T14:06:02.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:02.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.081946+0000", 2023-07-02T14:06:02.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.7495999999999999e-05 2023-07-02T14:06:02.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:02.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.081973+0000", 2023-07-02T14:06:02.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115494 2023-07-02T14:06:02.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:02.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082089+0000", 2023-07-02T14:06:02.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.102723355 2023-07-02T14:06:02.362 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.362 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.362 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.362 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17049 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:02.362 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:06:02.363 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 19.282737444999999, 2023-07-02T14:06:02.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.112277108, 2023-07-02T14:06:02.363 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:02.363 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:02.363 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:02.363 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:02.363 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:02.363 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17049 2023-07-02T14:06:02.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.364 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:02.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:02.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:06:02.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:02.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:02.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:06:02.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:02.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:02.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979382+0000", 2023-07-02T14:06:02.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3450000000000002e-06 2023-07-02T14:06:02.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:02.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979385+0000", 2023-07-02T14:06:02.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9000000000000003e-06 2023-07-02T14:06:02.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:02.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979391+0000", 2023-07-02T14:06:02.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9999999999999998e-06 2023-07-02T14:06:02.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:02.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979395+0000", 2023-07-02T14:06:02.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013089412 2023-07-02T14:06:02.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992485+0000", 2023-07-02T14:06:02.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3828999999999997e-05 2023-07-02T14:06:02.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992519+0000", 2023-07-02T14:06:02.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062549889999999999 2023-07-02T14:06:02.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998774+0000", 2023-07-02T14:06:02.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4870999999999995e-05 2023-07-02T14:06:02.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998848+0000", 2023-07-02T14:06:02.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091715140000000004 2023-07-02T14:06:02.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008020+0000", 2023-07-02T14:06:02.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8659999999999999e-05 2023-07-02T14:06:02.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008059+0000", 2023-07-02T14:06:02.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059565110000000003 2023-07-02T14:06:02.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014015+0000", 2023-07-02T14:06:02.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8088000000000002e-05 2023-07-02T14:06:02.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014093+0000", 2023-07-02T14:06:02.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007513395 2023-07-02T14:06:02.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021607+0000", 2023-07-02T14:06:02.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9777e-05 2023-07-02T14:06:02.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021636+0000", 2023-07-02T14:06:02.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056999549999999996 2023-07-02T14:06:02.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027336+0000", 2023-07-02T14:06:02.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8356999999999996e-05 2023-07-02T14:06:02.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027405+0000", 2023-07-02T14:06:02.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047942130000000003 2023-07-02T14:06:02.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032199+0000", 2023-07-02T14:06:02.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1137999999999999e-05 2023-07-02T14:06:02.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032230+0000", 2023-07-02T14:06:02.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057056759999999998 2023-07-02T14:06:02.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.037936+0000", 2023-07-02T14:06:02.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6334999999999993e-05 2023-07-02T14:06:02.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038012+0000", 2023-07-02T14:06:02.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043985609999999996 2023-07-02T14:06:02.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042411+0000", 2023-07-02T14:06:02.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6857e-05 2023-07-02T14:06:02.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042487+0000", 2023-07-02T14:06:02.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007284446 2023-07-02T14:06:02.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049772+0000", 2023-07-02T14:06:02.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3757000000000006e-05 2023-07-02T14:06:02.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049846+0000", 2023-07-02T14:06:02.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0095186679999999992 2023-07-02T14:06:02.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059364+0000", 2023-07-02T14:06:02.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9931000000000005e-05 2023-07-02T14:06:02.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059434+0000", 2023-07-02T14:06:02.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079904920000000001 2023-07-02T14:06:02.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067425+0000", 2023-07-02T14:06:02.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5583000000000001e-05 2023-07-02T14:06:02.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067460+0000", 2023-07-02T14:06:02.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009114278 2023-07-02T14:06:02.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076575+0000", 2023-07-02T14:06:02.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5880000000000001e-05 2023-07-02T14:06:02.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076620+0000", 2023-07-02T14:06:02.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055118470000000003 2023-07-02T14:06:02.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082132+0000", 2023-07-02T14:06:02.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8306999999999997e-05 2023-07-02T14:06:02.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:02.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082181+0000", 2023-07-02T14:06:02.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026259599999999998 2023-07-02T14:06:02.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:02.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082443+0000", 2023-07-02T14:06:02.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018855339999999999 2023-07-02T14:06:02.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:02.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084329+0000", 2023-07-02T14:06:02.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071877729999999997 2023-07-02T14:06:02.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:02.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091517+0000", 2023-07-02T14:06:02.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3424999999999999e-05 2023-07-02T14:06:02.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:02.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091550+0000", 2023-07-02T14:06:02.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109724 2023-07-02T14:06:02.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:02.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091660+0000", 2023-07-02T14:06:02.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.112277108 2023-07-02T14:06:02.388 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.389 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.389 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.389 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17050 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:02.389 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:06:02.389 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 19.282720520000002, 2023-07-02T14:06:02.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12076701300000001, 2023-07-02T14:06:02.390 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:02.390 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:02.390 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:02.390 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:02.390 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:02.390 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17050 2023-07-02T14:06:02.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.391 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:02.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:02.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:06:02.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:02.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:02.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:06:02.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:02.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:02.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:06:02.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0759999999999999e-06 2023-07-02T14:06:02.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:02.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979402+0000", 2023-07-02T14:06:02.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0480000000000004e-06 2023-07-02T14:06:02.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:02.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979408+0000", 2023-07-02T14:06:02.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9559999999999999e-06 2023-07-02T14:06:02.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:02.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979412+0000", 2023-07-02T14:06:02.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013117658000000001 2023-07-02T14:06:02.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992530+0000", 2023-07-02T14:06:02.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4464000000000001e-05 2023-07-02T14:06:02.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992564+0000", 2023-07-02T14:06:02.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0063143640000000003 2023-07-02T14:06:02.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998879+0000", 2023-07-02T14:06:02.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2842000000000002e-05 2023-07-02T14:06:02.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998911+0000", 2023-07-02T14:06:02.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091863190000000001 2023-07-02T14:06:02.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008098+0000", 2023-07-02T14:06:02.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1448999999999999e-05 2023-07-02T14:06:02.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008179+0000", 2023-07-02T14:06:02.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059353720000000004 2023-07-02T14:06:02.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014115+0000", 2023-07-02T14:06:02.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1928000000000002e-05 2023-07-02T14:06:02.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014157+0000", 2023-07-02T14:06:02.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074935210000000004 2023-07-02T14:06:02.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021650+0000", 2023-07-02T14:06:02.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0199999999999999e-05 2023-07-02T14:06:02.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021680+0000", 2023-07-02T14:06:02.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057425109999999996 2023-07-02T14:06:02.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027423+0000", 2023-07-02T14:06:02.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1498999999999998e-05 2023-07-02T14:06:02.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027454+0000", 2023-07-02T14:06:02.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047903420000000004 2023-07-02T14:06:02.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032245+0000", 2023-07-02T14:06:02.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2552999999999998e-05 2023-07-02T14:06:02.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032277+0000", 2023-07-02T14:06:02.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057638079999999996 2023-07-02T14:06:02.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038041+0000", 2023-07-02T14:06:02.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9168000000000002e-05 2023-07-02T14:06:02.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.404 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038080+0000", 2023-07-02T14:06:02.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044291799999999996 2023-07-02T14:06:02.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.405 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042509+0000", 2023-07-02T14:06:02.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3528999999999997e-05 2023-07-02T14:06:02.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042543+0000", 2023-07-02T14:06:02.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073244729999999997 2023-07-02T14:06:02.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049867+0000", 2023-07-02T14:06:02.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2873999999999997e-05 2023-07-02T14:06:02.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049900+0000", 2023-07-02T14:06:02.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0095670149999999999 2023-07-02T14:06:02.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059467+0000", 2023-07-02T14:06:02.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0406000000000003e-05 2023-07-02T14:06:02.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059538+0000", 2023-07-02T14:06:02.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079635079999999994 2023-07-02T14:06:02.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067501+0000", 2023-07-02T14:06:02.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4795999999999997e-05 2023-07-02T14:06:02.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067536+0000", 2023-07-02T14:06:02.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091231969999999996 2023-07-02T14:06:02.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076659+0000", 2023-07-02T14:06:02.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3208999999999999e-05 2023-07-02T14:06:02.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076702+0000", 2023-07-02T14:06:02.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070660480000000001 2023-07-02T14:06:02.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.083768+0000", 2023-07-02T14:06:02.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5546000000000001e-05 2023-07-02T14:06:02.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.083814+0000", 2023-07-02T14:06:02.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078830659999999993 2023-07-02T14:06:02.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091697+0000", 2023-07-02T14:06:02.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5921000000000003e-05 2023-07-02T14:06:02.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:02.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091743+0000", 2023-07-02T14:06:02.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026571400000000001 2023-07-02T14:06:02.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:02.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.092009+0000", 2023-07-02T14:06:02.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022058429999999999 2023-07-02T14:06:02.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:02.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.094214+0000", 2023-07-02T14:06:02.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057115869999999997 2023-07-02T14:06:02.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:02.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.099926+0000", 2023-07-02T14:06:02.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0287999999999997e-05 2023-07-02T14:06:02.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:02.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.099976+0000", 2023-07-02T14:06:02.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019014999999999999 2023-07-02T14:06:02.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:02.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.100166+0000", 2023-07-02T14:06:02.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12076701300000001 2023-07-02T14:06:02.417 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.417 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.417 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.417 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17095 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:02.417 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.060121+0000", 2023-07-02T14:06:02.417 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 19.201998563, 2023-07-02T14:06:02.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.099578634999999999, 2023-07-02T14:06:02.418 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:02.418 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:02.418 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:02.418 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:02.418 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:02.418 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17095 2023-07-02T14:06:02.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.419 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:02.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:02.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060121+0000", 2023-07-02T14:06:02.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:02.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:02.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060121+0000", 2023-07-02T14:06:02.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:02.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:02.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060121+0000", 2023-07-02T14:06:02.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5140000000000004e-06 2023-07-02T14:06:02.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:02.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060129+0000", 2023-07-02T14:06:02.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4520000000000002e-06 2023-07-02T14:06:02.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:02.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060135+0000", 2023-07-02T14:06:02.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2960000000000004e-06 2023-07-02T14:06:02.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:02.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060141+0000", 2023-07-02T14:06:02.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00047506399999999998 2023-07-02T14:06:02.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060616+0000", 2023-07-02T14:06:02.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8182999999999997e-05 2023-07-02T14:06:02.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060655+0000", 2023-07-02T14:06:02.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074016079999999996 2023-07-02T14:06:02.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068056+0000", 2023-07-02T14:06:02.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6019000000000005e-05 2023-07-02T14:06:02.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068132+0000", 2023-07-02T14:06:02.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090475429999999999 2023-07-02T14:06:02.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077180+0000", 2023-07-02T14:06:02.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7527999999999997e-05 2023-07-02T14:06:02.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077217+0000", 2023-07-02T14:06:02.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070671400000000004 2023-07-02T14:06:02.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084284+0000", 2023-07-02T14:06:02.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3818e-05 2023-07-02T14:06:02.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084318+0000", 2023-07-02T14:06:02.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093793799999999997 2023-07-02T14:06:02.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093698+0000", 2023-07-02T14:06:02.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2876999999999997e-05 2023-07-02T14:06:02.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093730+0000", 2023-07-02T14:06:02.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088179079999999993 2023-07-02T14:06:02.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102548+0000", 2023-07-02T14:06:02.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5802000000000003e-05 2023-07-02T14:06:02.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102634+0000", 2023-07-02T14:06:02.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099570119999999995 2023-07-02T14:06:02.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112591+0000", 2023-07-02T14:06:02.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6594000000000001e-05 2023-07-02T14:06:02.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112648+0000", 2023-07-02T14:06:02.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009671921 2023-07-02T14:06:02.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122320+0000", 2023-07-02T14:06:02.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2005000000000002e-05 2023-07-02T14:06:02.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122362+0000", 2023-07-02T14:06:02.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011261546000000001 2023-07-02T14:06:02.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133623+0000", 2023-07-02T14:06:02.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9981000000000006e-05 2023-07-02T14:06:02.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133693+0000", 2023-07-02T14:06:02.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093884069999999997 2023-07-02T14:06:02.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.143082+0000", 2023-07-02T14:06:02.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103371 2023-07-02T14:06:02.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.143185+0000", 2023-07-02T14:06:02.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051199679999999999 2023-07-02T14:06:02.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.148305+0000", 2023-07-02T14:06:02.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6837999999999998e-05 2023-07-02T14:06:02.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.148392+0000", 2023-07-02T14:06:02.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046966489999999998 2023-07-02T14:06:02.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.153088+0000", 2023-07-02T14:06:02.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5444000000000001e-05 2023-07-02T14:06:02.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:02.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.153134+0000", 2023-07-02T14:06:02.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000238259 2023-07-02T14:06:02.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:02.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.153372+0000", 2023-07-02T14:06:02.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023790500000000002 2023-07-02T14:06:02.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:02.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.155751+0000", 2023-07-02T14:06:02.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038028099999999998 2023-07-02T14:06:02.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:02.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.159554+0000", 2023-07-02T14:06:02.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3145000000000002e-05 2023-07-02T14:06:02.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:02.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.159587+0000", 2023-07-02T14:06:02.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000112948 2023-07-02T14:06:02.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:02.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.159700+0000", 2023-07-02T14:06:02.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.099578634999999999 2023-07-02T14:06:02.441 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.441 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.441 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.441 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17096 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:02.442 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:06:02.442 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 19.201954743999998, 2023-07-02T14:06:02.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.10620154, 2023-07-02T14:06:02.442 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:02.442 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:02.442 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:02.442 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:02.442 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:02.443 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17096 2023-07-02T14:06:02.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.443 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:02.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:02.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:06:02.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:02.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:02.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:06:02.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:02.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:02.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:06:02.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3400000000000002e-06 2023-07-02T14:06:02.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:02.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060168+0000", 2023-07-02T14:06:02.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4020000000000004e-06 2023-07-02T14:06:02.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:02.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060173+0000", 2023-07-02T14:06:02.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4259999999999999e-06 2023-07-02T14:06:02.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:02.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060176+0000", 2023-07-02T14:06:02.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00049978399999999997 2023-07-02T14:06:02.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060676+0000", 2023-07-02T14:06:02.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0135000000000004e-05 2023-07-02T14:06:02.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060756+0000", 2023-07-02T14:06:02.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074338060000000003 2023-07-02T14:06:02.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068190+0000", 2023-07-02T14:06:02.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7941000000000004e-05 2023-07-02T14:06:02.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068278+0000", 2023-07-02T14:06:02.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089611870000000007 2023-07-02T14:06:02.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077239+0000", 2023-07-02T14:06:02.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6349e-05 2023-07-02T14:06:02.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077275+0000", 2023-07-02T14:06:02.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007072142 2023-07-02T14:06:02.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084347+0000", 2023-07-02T14:06:02.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.591e-05 2023-07-02T14:06:02.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084383+0000", 2023-07-02T14:06:02.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093660119999999999 2023-07-02T14:06:02.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093749+0000", 2023-07-02T14:06:02.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1538000000000002e-05 2023-07-02T14:06:02.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093781+0000", 2023-07-02T14:06:02.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088884629999999992 2023-07-02T14:06:02.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102669+0000", 2023-07-02T14:06:02.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6032999999999997e-05 2023-07-02T14:06:02.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102725+0000", 2023-07-02T14:06:02.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009945522 2023-07-02T14:06:02.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112671+0000", 2023-07-02T14:06:02.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5741000000000002e-05 2023-07-02T14:06:02.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112707+0000", 2023-07-02T14:06:02.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096867940000000003 2023-07-02T14:06:02.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122393+0000", 2023-07-02T14:06:02.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6244999999999998e-05 2023-07-02T14:06:02.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122480+0000", 2023-07-02T14:06:02.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011251107999999999 2023-07-02T14:06:02.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133731+0000", 2023-07-02T14:06:02.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6400999999999999e-05 2023-07-02T14:06:02.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133767+0000", 2023-07-02T14:06:02.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0094677070000000006 2023-07-02T14:06:02.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.143235+0000", 2023-07-02T14:06:02.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8721999999999996e-05 2023-07-02T14:06:02.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.143314+0000", 2023-07-02T14:06:02.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051196050000000002 2023-07-02T14:06:02.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.148433+0000", 2023-07-02T14:06:02.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2484999999999997e-05 2023-07-02T14:06:02.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.148476+0000", 2023-07-02T14:06:02.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006202234 2023-07-02T14:06:02.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.154678+0000", 2023-07-02T14:06:02.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2699000000000002e-05 2023-07-02T14:06:02.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.154761+0000", 2023-07-02T14:06:02.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049653809999999996 2023-07-02T14:06:02.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.159726+0000", 2023-07-02T14:06:02.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6862e-05 2023-07-02T14:06:02.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:02.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.159773+0000", 2023-07-02T14:06:02.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024618099999999999 2023-07-02T14:06:02.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:02.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.160019+0000", 2023-07-02T14:06:02.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0019043059999999999 2023-07-02T14:06:02.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:02.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.161923+0000", 2023-07-02T14:06:02.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004316884 2023-07-02T14:06:02.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:02.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.166240+0000", 2023-07-02T14:06:02.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2611000000000003e-05 2023-07-02T14:06:02.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:02.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.166273+0000", 2023-07-02T14:06:02.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4047000000000003e-05 2023-07-02T14:06:02.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:02.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.166367+0000", 2023-07-02T14:06:02.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.10620154 2023-07-02T14:06:02.466 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.467 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.467 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.467 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17204 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:02.467 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.130952+0000", 2023-07-02T14:06:02.467 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 19.131167812000001, 2023-07-02T14:06:02.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.101008762, 2023-07-02T14:06:02.468 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:02.468 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:02.468 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:02.468 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:02.468 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:02.468 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17204 2023-07-02T14:06:02.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.469 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:02.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:02.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130952+0000", 2023-07-02T14:06:02.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:02.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:02.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130952+0000", 2023-07-02T14:06:02.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:02.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:02.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130952+0000", 2023-07-02T14:06:02.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7139999999999999e-06 2023-07-02T14:06:02.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:02.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130955+0000", 2023-07-02T14:06:02.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.054e-06 2023-07-02T14:06:02.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:02.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130962+0000", 2023-07-02T14:06:02.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7610000000000001e-06 2023-07-02T14:06:02.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:02.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.130965+0000", 2023-07-02T14:06:02.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0034574929999999999 2023-07-02T14:06:02.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.134423+0000", 2023-07-02T14:06:02.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9406999999999997e-05 2023-07-02T14:06:02.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.134462+0000", 2023-07-02T14:06:02.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096105089999999997 2023-07-02T14:06:02.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.144073+0000", 2023-07-02T14:06:02.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3737000000000002e-05 2023-07-02T14:06:02.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.144146+0000", 2023-07-02T14:06:02.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050853909999999999 2023-07-02T14:06:02.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.149232+0000", 2023-07-02T14:06:02.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1114999999999999e-05 2023-07-02T14:06:02.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.149303+0000", 2023-07-02T14:06:02.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060490919999999998 2023-07-02T14:06:02.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.155352+0000", 2023-07-02T14:06:02.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5466000000000002e-05 2023-07-02T14:06:02.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.155388+0000", 2023-07-02T14:06:02.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064891990000000002 2023-07-02T14:06:02.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.161877+0000", 2023-07-02T14:06:02.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9493000000000003e-05 2023-07-02T14:06:02.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.161916+0000", 2023-07-02T14:06:02.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062878689999999998 2023-07-02T14:06:02.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.168204+0000", 2023-07-02T14:06:02.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4718000000000002e-05 2023-07-02T14:06:02.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.168239+0000", 2023-07-02T14:06:02.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043609909999999998 2023-07-02T14:06:02.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.172600+0000", 2023-07-02T14:06:02.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4609e-05 2023-07-02T14:06:02.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.172634+0000", 2023-07-02T14:06:02.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050403130000000003 2023-07-02T14:06:02.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.177675+0000", 2023-07-02T14:06:02.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.523e-05 2023-07-02T14:06:02.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.177710+0000", 2023-07-02T14:06:02.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043389429999999996 2023-07-02T14:06:02.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.182049+0000", 2023-07-02T14:06:02.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3048999999999998e-05 2023-07-02T14:06:02.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.182122+0000", 2023-07-02T14:06:02.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041995360000000002 2023-07-02T14:06:02.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.186321+0000", 2023-07-02T14:06:02.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2482000000000001e-05 2023-07-02T14:06:02.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.186404+0000", 2023-07-02T14:06:02.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013084261 2023-07-02T14:06:02.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.199488+0000", 2023-07-02T14:06:02.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013297600000000001 2023-07-02T14:06:02.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.199621+0000", 2023-07-02T14:06:02.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014611084999999999 2023-07-02T14:06:02.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.214232+0000", 2023-07-02T14:06:02.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7821000000000001e-05 2023-07-02T14:06:02.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.214290+0000", 2023-07-02T14:06:02.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089499660000000002 2023-07-02T14:06:02.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.223240+0000", 2023-07-02T14:06:02.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010005000000000001 2023-07-02T14:06:02.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:02.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.223340+0000", 2023-07-02T14:06:02.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00049127800000000005 2023-07-02T14:06:02.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:02.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.223831+0000", 2023-07-02T14:06:02.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064706440000000002 2023-07-02T14:06:02.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:02.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.230302+0000", 2023-07-02T14:06:02.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0015080759999999999 2023-07-02T14:06:02.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:02.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.231810+0000", 2023-07-02T14:06:02.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6409000000000003e-05 2023-07-02T14:06:02.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:02.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.231847+0000", 2023-07-02T14:06:02.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011443 2023-07-02T14:06:02.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:02.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.231961+0000", 2023-07-02T14:06:02.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.101008762 2023-07-02T14:06:02.492 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.492 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.492 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.493 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17294 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:02.493 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.256047+0000", 2023-07-02T14:06:02.493 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 19.006073096000001, 2023-07-02T14:06:02.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.099803734000000005, 2023-07-02T14:06:02.493 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:02.493 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:02.493 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:02.494 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:02.494 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:02.494 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17294 2023-07-02T14:06:02.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.494 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:02.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:02.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256047+0000", 2023-07-02T14:06:02.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:02.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:02.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256047+0000", 2023-07-02T14:06:02.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:02.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:02.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256047+0000", 2023-07-02T14:06:02.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.367e-06 2023-07-02T14:06:02.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:02.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256049+0000", 2023-07-02T14:06:02.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0019999999999998e-06 2023-07-02T14:06:02.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:02.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256052+0000", 2023-07-02T14:06:02.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9450000000000002e-06 2023-07-02T14:06:02.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:02.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256055+0000", 2023-07-02T14:06:02.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022819160000000002 2023-07-02T14:06:02.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.258337+0000", 2023-07-02T14:06:02.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1145999999999999e-05 2023-07-02T14:06:02.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.258418+0000", 2023-07-02T14:06:02.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066309109999999997 2023-07-02T14:06:02.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.265049+0000", 2023-07-02T14:06:02.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6449000000000002e-05 2023-07-02T14:06:02.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.265095+0000", 2023-07-02T14:06:02.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089284169999999993 2023-07-02T14:06:02.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.274024+0000", 2023-07-02T14:06:02.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7626999999999997e-05 2023-07-02T14:06:02.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.274061+0000", 2023-07-02T14:06:02.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013396880999999999 2023-07-02T14:06:02.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.287458+0000", 2023-07-02T14:06:02.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1865000000000003e-05 2023-07-02T14:06:02.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.287540+0000", 2023-07-02T14:06:02.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069236419999999998 2023-07-02T14:06:02.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.294464+0000", 2023-07-02T14:06:02.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7681999999999993e-05 2023-07-02T14:06:02.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.294531+0000", 2023-07-02T14:06:02.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076921070000000001 2023-07-02T14:06:02.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302223+0000", 2023-07-02T14:06:02.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5528999999999998e-05 2023-07-02T14:06:02.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302259+0000", 2023-07-02T14:06:02.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096256480000000005 2023-07-02T14:06:02.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.311885+0000", 2023-07-02T14:06:02.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6389e-05 2023-07-02T14:06:02.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.311921+0000", 2023-07-02T14:06:02.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098685439999999999 2023-07-02T14:06:02.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.321790+0000", 2023-07-02T14:06:02.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1817999999999995e-05 2023-07-02T14:06:02.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.321871+0000", 2023-07-02T14:06:02.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009974966 2023-07-02T14:06:02.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.331846+0000", 2023-07-02T14:06:02.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6520000000000002e-05 2023-07-02T14:06:02.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.331893+0000", 2023-07-02T14:06:02.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086651149999999993 2023-07-02T14:06:02.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.340558+0000", 2023-07-02T14:06:02.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5918000000000001e-05 2023-07-02T14:06:02.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.340634+0000", 2023-07-02T14:06:02.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059983099999999998 2023-07-02T14:06:02.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.346632+0000", 2023-07-02T14:06:02.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7887999999999997e-05 2023-07-02T14:06:02.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:02.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.346710+0000", 2023-07-02T14:06:02.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00046673100000000001 2023-07-02T14:06:02.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:02.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.347177+0000", 2023-07-02T14:06:02.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027484829999999999 2023-07-02T14:06:02.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:02.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349925+0000", 2023-07-02T14:06:02.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056857970000000002 2023-07-02T14:06:02.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:02.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.355611+0000", 2023-07-02T14:06:02.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9406000000000001e-05 2023-07-02T14:06:02.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:02.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.355661+0000", 2023-07-02T14:06:02.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019010399999999999 2023-07-02T14:06:02.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:02.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.355851+0000", 2023-07-02T14:06:02.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.099803734000000005 2023-07-02T14:06:02.515 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.515 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.515 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.516 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17295 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:02.516 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.256057+0000", 2023-07-02T14:06:02.516 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 19.006062876000001, 2023-07-02T14:06:02.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.107964279, 2023-07-02T14:06:02.516 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:02.516 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:02.517 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:02.517 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:02.517 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:02.517 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17295 2023-07-02T14:06:02.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.517 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:02.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:02.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256057+0000", 2023-07-02T14:06:02.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:02.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:02.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256057+0000", 2023-07-02T14:06:02.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:02.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:02.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256057+0000", 2023-07-02T14:06:02.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.458e-06 2023-07-02T14:06:02.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:02.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256059+0000", 2023-07-02T14:06:02.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9019999999999999e-06 2023-07-02T14:06:02.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:02.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256062+0000", 2023-07-02T14:06:02.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9929999999999998e-06 2023-07-02T14:06:02.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:02.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256064+0000", 2023-07-02T14:06:02.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023724789999999998 2023-07-02T14:06:02.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.258437+0000", 2023-07-02T14:06:02.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7698000000000002e-05 2023-07-02T14:06:02.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.258494+0000", 2023-07-02T14:06:02.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066424140000000001 2023-07-02T14:06:02.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.265137+0000", 2023-07-02T14:06:02.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2169000000000002e-05 2023-07-02T14:06:02.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.265179+0000", 2023-07-02T14:06:02.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089047970000000008 2023-07-02T14:06:02.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.274084+0000", 2023-07-02T14:06:02.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7489e-05 2023-07-02T14:06:02.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.274121+0000", 2023-07-02T14:06:02.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013452426999999999 2023-07-02T14:06:02.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.287574+0000", 2023-07-02T14:06:02.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5149999999999996e-05 2023-07-02T14:06:02.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.287659+0000", 2023-07-02T14:06:02.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069046059999999998 2023-07-02T14:06:02.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.294563+0000", 2023-07-02T14:06:02.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7073000000000001e-05 2023-07-02T14:06:02.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.294611+0000", 2023-07-02T14:06:02.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076769480000000003 2023-07-02T14:06:02.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302287+0000", 2023-07-02T14:06:02.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4381000000000002e-05 2023-07-02T14:06:02.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302322+0000", 2023-07-02T14:06:02.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009637863 2023-07-02T14:06:02.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.311960+0000", 2023-07-02T14:06:02.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4847999999999999e-05 2023-07-02T14:06:02.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312035+0000", 2023-07-02T14:06:02.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098741579999999992 2023-07-02T14:06:02.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.321909+0000", 2023-07-02T14:06:02.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6455999999999998e-05 2023-07-02T14:06:02.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.321945+0000", 2023-07-02T14:06:02.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099894690000000008 2023-07-02T14:06:02.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.331935+0000", 2023-07-02T14:06:02.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2550000000000002e-05 2023-07-02T14:06:02.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332007+0000", 2023-07-02T14:06:02.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086760090000000002 2023-07-02T14:06:02.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.340683+0000", 2023-07-02T14:06:02.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8462999999999998e-05 2023-07-02T14:06:02.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.340752+0000", 2023-07-02T14:06:02.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00830381 2023-07-02T14:06:02.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349055+0000", 2023-07-02T14:06:02.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9289999999999994e-05 2023-07-02T14:06:02.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349145+0000", 2023-07-02T14:06:02.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067717250000000001 2023-07-02T14:06:02.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.355917+0000", 2023-07-02T14:06:02.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4431999999999995e-05 2023-07-02T14:06:02.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:02.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.355991+0000", 2023-07-02T14:06:02.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00044773399999999998 2023-07-02T14:06:02.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:02.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.356439+0000", 2023-07-02T14:06:02.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028403690000000001 2023-07-02T14:06:02.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:02.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.359279+0000", 2023-07-02T14:06:02.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045352450000000002 2023-07-02T14:06:02.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:02.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.363814+0000", 2023-07-02T14:06:02.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6167e-05 2023-07-02T14:06:02.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:02.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.363860+0000", 2023-07-02T14:06:02.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000160951 2023-07-02T14:06:02.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:02.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.364021+0000", 2023-07-02T14:06:02.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.107964279 2023-07-02T14:06:02.539 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.540 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.540 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.540 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17351 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:02.540 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302537+0000", 2023-07-02T14:06:02.540 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 18.959583142, 2023-07-02T14:06:02.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.099134853999999994, 2023-07-02T14:06:02.541 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:02.541 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:02.541 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:02.541 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:02.541 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:02.541 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17351 2023-07-02T14:06:02.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.541 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:02.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:02.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302537+0000", 2023-07-02T14:06:02.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:02.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:02.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302537+0000", 2023-07-02T14:06:02.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:02.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:02.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302536+0000", 2023-07-02T14:06:02.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3329999999999996e-06 2023-07-02T14:06:02.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:02.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302541+0000", 2023-07-02T14:06:02.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6600000000000001e-06 2023-07-02T14:06:02.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:02.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302550+0000", 2023-07-02T14:06:02.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9e-06 2023-07-02T14:06:02.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:02.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302557+0000", 2023-07-02T14:06:02.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8735999999999998e-05 2023-07-02T14:06:02.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302656+0000", 2023-07-02T14:06:02.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7564000000000002e-05 2023-07-02T14:06:02.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302693+0000", 2023-07-02T14:06:02.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096663749999999996 2023-07-02T14:06:02.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312360+0000", 2023-07-02T14:06:02.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6055000000000003e-05 2023-07-02T14:06:02.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312436+0000", 2023-07-02T14:06:02.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098653899999999999 2023-07-02T14:06:02.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322301+0000", 2023-07-02T14:06:02.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7258999999999997e-05 2023-07-02T14:06:02.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322338+0000", 2023-07-02T14:06:02.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099812860000000007 2023-07-02T14:06:02.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332320+0000", 2023-07-02T14:06:02.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3201999999999995e-05 2023-07-02T14:06:02.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332393+0000", 2023-07-02T14:06:02.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088690169999999999 2023-07-02T14:06:02.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341262+0000", 2023-07-02T14:06:02.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1718999999999995e-05 2023-07-02T14:06:02.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341344+0000", 2023-07-02T14:06:02.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008184574 2023-07-02T14:06:02.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349528+0000", 2023-07-02T14:06:02.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2019000000000001e-05 2023-07-02T14:06:02.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349620+0000", 2023-07-02T14:06:02.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089525679999999993 2023-07-02T14:06:02.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358573+0000", 2023-07-02T14:06:02.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5639e-05 2023-07-02T14:06:02.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358638+0000", 2023-07-02T14:06:02.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079895899999999995 2023-07-02T14:06:02.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366628+0000", 2023-07-02T14:06:02.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5162999999999997e-05 2023-07-02T14:06:02.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366723+0000", 2023-07-02T14:06:02.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074560440000000002 2023-07-02T14:06:02.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374179+0000", 2023-07-02T14:06:02.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9793999999999999e-05 2023-07-02T14:06:02.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374219+0000", 2023-07-02T14:06:02.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091487210000000003 2023-07-02T14:06:02.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383368+0000", 2023-07-02T14:06:02.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7316000000000004e-05 2023-07-02T14:06:02.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383435+0000", 2023-07-02T14:06:02.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010649080999999999 2023-07-02T14:06:02.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.394084+0000", 2023-07-02T14:06:02.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8598999999999997e-05 2023-07-02T14:06:02.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:02.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.394173+0000", 2023-07-02T14:06:02.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042554099999999998 2023-07-02T14:06:02.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:02.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.394598+0000", 2023-07-02T14:06:02.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048553529999999998 2023-07-02T14:06:02.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:02.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.399454+0000", 2023-07-02T14:06:02.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020927350000000001 2023-07-02T14:06:02.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:02.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401546+0000", 2023-07-02T14:06:02.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.7813999999999998e-05 2023-07-02T14:06:02.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:02.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401574+0000", 2023-07-02T14:06:02.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7508000000000002e-05 2023-07-02T14:06:02.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:02.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401672+0000", 2023-07-02T14:06:02.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.099134853999999994 2023-07-02T14:06:02.562 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.563 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.563 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.563 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17352 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:02.563 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302565+0000", 2023-07-02T14:06:02.563 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 18.959554659999998, 2023-07-02T14:06:02.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.105667996, 2023-07-02T14:06:02.564 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:02.564 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:02.564 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:02.564 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:02.564 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:02.564 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17352 2023-07-02T14:06:02.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.565 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:02.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:02.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302565+0000", 2023-07-02T14:06:02.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:02.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:02.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302565+0000", 2023-07-02T14:06:02.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:02.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:02.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302565+0000", 2023-07-02T14:06:02.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7000000000000002e-06 2023-07-02T14:06:02.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:02.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302568+0000", 2023-07-02T14:06:02.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2140000000000001e-06 2023-07-02T14:06:02.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:02.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302575+0000", 2023-07-02T14:06:02.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3250000000000001e-06 2023-07-02T14:06:02.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:02.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302579+0000", 2023-07-02T14:06:02.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014498700000000001 2023-07-02T14:06:02.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302724+0000", 2023-07-02T14:06:02.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5444999999999993e-05 2023-07-02T14:06:02.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302799+0000", 2023-07-02T14:06:02.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096710000000000008 2023-07-02T14:06:02.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312470+0000", 2023-07-02T14:06:02.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4798999999999997e-05 2023-07-02T14:06:02.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312505+0000", 2023-07-02T14:06:02.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098663940000000006 2023-07-02T14:06:02.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322372+0000", 2023-07-02T14:06:02.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1996000000000002e-05 2023-07-02T14:06:02.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322414+0000", 2023-07-02T14:06:02.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010015735 2023-07-02T14:06:02.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332429+0000", 2023-07-02T14:06:02.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2929000000000003e-05 2023-07-02T14:06:02.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332462+0000", 2023-07-02T14:06:02.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089318550000000007 2023-07-02T14:06:02.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341394+0000", 2023-07-02T14:06:02.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7248e-05 2023-07-02T14:06:02.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341431+0000", 2023-07-02T14:06:02.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0082266739999999998 2023-07-02T14:06:02.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349658+0000", 2023-07-02T14:06:02.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3188999999999997e-05 2023-07-02T14:06:02.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349691+0000", 2023-07-02T14:06:02.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089857329999999992 2023-07-02T14:06:02.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358677+0000", 2023-07-02T14:06:02.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6436000000000001e-05 2023-07-02T14:06:02.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358713+0000", 2023-07-02T14:06:02.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080528400000000003 2023-07-02T14:06:02.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366766+0000", 2023-07-02T14:06:02.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3217999999999999e-05 2023-07-02T14:06:02.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366799+0000", 2023-07-02T14:06:02.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074649970000000001 2023-07-02T14:06:02.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374264+0000", 2023-07-02T14:06:02.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9482999999999999e-05 2023-07-02T14:06:02.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374334+0000", 2023-07-02T14:06:02.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091483889999999998 2023-07-02T14:06:02.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383482+0000", 2023-07-02T14:06:02.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3095000000000005e-05 2023-07-02T14:06:02.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383545+0000", 2023-07-02T14:06:02.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013359414 2023-07-02T14:06:02.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.396905+0000", 2023-07-02T14:06:02.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012269900000000001 2023-07-02T14:06:02.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397028+0000", 2023-07-02T14:06:02.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046629660000000002 2023-07-02T14:06:02.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401691+0000", 2023-07-02T14:06:02.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5111999999999999e-05 2023-07-02T14:06:02.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:02.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401726+0000", 2023-07-02T14:06:02.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024867800000000001 2023-07-02T14:06:02.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:02.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401974+0000", 2023-07-02T14:06:02.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028409360000000001 2023-07-02T14:06:02.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:02.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404815+0000", 2023-07-02T14:06:02.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031148529999999999 2023-07-02T14:06:02.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:02.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.407930+0000", 2023-07-02T14:06:02.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6836000000000003e-05 2023-07-02T14:06:02.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:02.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.407987+0000", 2023-07-02T14:06:02.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024641799999999998 2023-07-02T14:06:02.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:02.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408233+0000", 2023-07-02T14:06:02.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.105667996 2023-07-02T14:06:02.587 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.587 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.587 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.587 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17353 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:02.588 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:06:02.588 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 18.959535844000001, 2023-07-02T14:06:02.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.115724379, 2023-07-02T14:06:02.588 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:02.588 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:02.588 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:02.588 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:02.589 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:02.589 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17353 2023-07-02T14:06:02.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.589 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:02.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:02.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:06:02.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:02.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:02.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:06:02.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:02.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:02.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:06:02.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.41e-06 2023-07-02T14:06:02.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:02.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302587+0000", 2023-07-02T14:06:02.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.015e-06 2023-07-02T14:06:02.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:02.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302593+0000", 2023-07-02T14:06:02.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8019999999999997e-06 2023-07-02T14:06:02.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:02.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302599+0000", 2023-07-02T14:06:02.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022701199999999999 2023-07-02T14:06:02.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302826+0000", 2023-07-02T14:06:02.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.426e-05 2023-07-02T14:06:02.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302870+0000", 2023-07-02T14:06:02.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096759489999999997 2023-07-02T14:06:02.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312546+0000", 2023-07-02T14:06:02.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5785000000000006e-05 2023-07-02T14:06:02.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312632+0000", 2023-07-02T14:06:02.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098106970000000002 2023-07-02T14:06:02.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322443+0000", 2023-07-02T14:06:02.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3161999999999999e-05 2023-07-02T14:06:02.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322476+0000", 2023-07-02T14:06:02.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010015823 2023-07-02T14:06:02.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332492+0000", 2023-07-02T14:06:02.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1165999999999999e-05 2023-07-02T14:06:02.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332533+0000", 2023-07-02T14:06:02.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089435269999999997 2023-07-02T14:06:02.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341476+0000", 2023-07-02T14:06:02.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000112691 2023-07-02T14:06:02.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341589+0000", 2023-07-02T14:06:02.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008147362 2023-07-02T14:06:02.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349736+0000", 2023-07-02T14:06:02.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5149000000000007e-05 2023-07-02T14:06:02.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349832+0000", 2023-07-02T14:06:02.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089226519999999997 2023-07-02T14:06:02.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358754+0000", 2023-07-02T14:06:02.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6091999999999998e-05 2023-07-02T14:06:02.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358810+0000", 2023-07-02T14:06:02.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080318000000000004 2023-07-02T14:06:02.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366842+0000", 2023-07-02T14:06:02.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4321999999999998e-05 2023-07-02T14:06:02.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366916+0000", 2023-07-02T14:06:02.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074664049999999997 2023-07-02T14:06:02.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374383+0000", 2023-07-02T14:06:02.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5460000000000001e-05 2023-07-02T14:06:02.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.604 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.604 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374438+0000", 2023-07-02T14:06:02.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091562450000000004 2023-07-02T14:06:02.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.604 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383594+0000", 2023-07-02T14:06:02.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2545000000000004e-05 2023-07-02T14:06:02.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383667+0000", 2023-07-02T14:06:02.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013424551 2023-07-02T14:06:02.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397092+0000", 2023-07-02T14:06:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6991999999999997e-05 2023-07-02T14:06:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397169+0000", 2023-07-02T14:06:02.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064399139999999997 2023-07-02T14:06:02.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403608+0000", 2023-07-02T14:06:02.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2614000000000006e-05 2023-07-02T14:06:02.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403691+0000", 2023-07-02T14:06:02.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004611693 2023-07-02T14:06:02.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408303+0000", 2023-07-02T14:06:02.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1935999999999995e-05 2023-07-02T14:06:02.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:02.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408395+0000", 2023-07-02T14:06:02.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00051077700000000002 2023-07-02T14:06:02.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:02.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408906+0000", 2023-07-02T14:06:02.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090767460000000001 2023-07-02T14:06:02.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:02.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.417982+0000", 2023-07-02T14:06:02.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4283999999999997e-05 2023-07-02T14:06:02.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:02.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418037+0000", 2023-07-02T14:06:02.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9667999999999999e-05 2023-07-02T14:06:02.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:02.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418096+0000", 2023-07-02T14:06:02.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021233800000000001 2023-07-02T14:06:02.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:02.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418309+0000", 2023-07-02T14:06:02.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.115724379 2023-07-02T14:06:02.612 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.613 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.613 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.613 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17354 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:02.613 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:02.613 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 18.959516883999999, 2023-07-02T14:06:02.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493, 2023-07-02T14:06:02.614 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:02.614 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:02.614 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:02.614 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:02.614 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:02.614 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17354 2023-07-02T14:06:02.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.615 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:02.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:02.615 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:02.615 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:02.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:02.616 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:02.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:02.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.616 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.616 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:02.616 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:02.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7440000000000001e-06 2023-07-02T14:06:02.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:02.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302606+0000", 2023-07-02T14:06:02.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:06:02.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:02.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302613+0000", 2023-07-02T14:06:02.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8810000000000004e-06 2023-07-02T14:06:02.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:02.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302618+0000", 2023-07-02T14:06:02.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028613100000000002 2023-07-02T14:06:02.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302904+0000", 2023-07-02T14:06:02.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5278999999999995e-05 2023-07-02T14:06:02.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302979+0000", 2023-07-02T14:06:02.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096904750000000005 2023-07-02T14:06:02.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312669+0000", 2023-07-02T14:06:02.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0328000000000002e-05 2023-07-02T14:06:02.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312700+0000", 2023-07-02T14:06:02.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098019319999999993 2023-07-02T14:06:02.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322502+0000", 2023-07-02T14:06:02.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4669999999999998e-05 2023-07-02T14:06:02.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322536+0000", 2023-07-02T14:06:02.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010023839 2023-07-02T14:06:02.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332560+0000", 2023-07-02T14:06:02.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5559999999999998e-05 2023-07-02T14:06:02.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332596+0000", 2023-07-02T14:06:02.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090401319999999993 2023-07-02T14:06:02.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341636+0000", 2023-07-02T14:06:02.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5373000000000001e-05 2023-07-02T14:06:02.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341681+0000", 2023-07-02T14:06:02.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081994059999999994 2023-07-02T14:06:02.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.625 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349881+0000", 2023-07-02T14:06:02.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4319000000000001e-05 2023-07-02T14:06:02.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.626 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349915+0000", 2023-07-02T14:06:02.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089388339999999997 2023-07-02T14:06:02.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.626 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358854+0000", 2023-07-02T14:06:02.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4319000000000005e-05 2023-07-02T14:06:02.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358928+0000", 2023-07-02T14:06:02.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080234690000000001 2023-07-02T14:06:02.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366952+0000", 2023-07-02T14:06:02.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2082999999999998e-05 2023-07-02T14:06:02.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366984+0000", 2023-07-02T14:06:02.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075015790000000004 2023-07-02T14:06:02.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374485+0000", 2023-07-02T14:06:02.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4480999999999995e-05 2023-07-02T14:06:02.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374550+0000", 2023-07-02T14:06:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091666620000000008 2023-07-02T14:06:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383716+0000", 2023-07-02T14:06:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9318000000000003e-05 2023-07-02T14:06:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383786+0000", 2023-07-02T14:06:02.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013434115 2023-07-02T14:06:02.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397220+0000", 2023-07-02T14:06:02.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4573999999999997e-05 2023-07-02T14:06:02.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397264+0000", 2023-07-02T14:06:02.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064824649999999998 2023-07-02T14:06:02.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403747+0000", 2023-07-02T14:06:02.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1087999999999993e-05 2023-07-02T14:06:02.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403828+0000", 2023-07-02T14:06:02.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072854449999999998 2023-07-02T14:06:02.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411113+0000", 2023-07-02T14:06:02.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5120000000000001e-05 2023-07-02T14:06:02.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411169+0000", 2023-07-02T14:06:02.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007211505 2023-07-02T14:06:02.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418380+0000", 2023-07-02T14:06:02.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011289399999999999 2023-07-02T14:06:02.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:02.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418493+0000", 2023-07-02T14:06:02.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00051217500000000002 2023-07-02T14:06:02.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:02.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.419005+0000", 2023-07-02T14:06:02.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00327393 2023-07-02T14:06:02.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:02.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422279+0000", 2023-07-02T14:06:02.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041432350000000003 2023-07-02T14:06:02.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:02.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426422+0000", 2023-07-02T14:06:02.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7021000000000003e-05 2023-07-02T14:06:02.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:02.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426469+0000", 2023-07-02T14:06:02.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000190256 2023-07-02T14:06:02.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:02.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426660+0000", 2023-07-02T14:06:02.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493 2023-07-02T14:06:02.639 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.639 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.640 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.640 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17355 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:02.640 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:02.640 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 18.959498223000001, 2023-07-02T14:06:02.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001, 2023-07-02T14:06:02.640 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:02.641 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:02.641 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:02.641 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:02.641 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:02.641 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17355 2023-07-02T14:06:02.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.641 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:02.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:02.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:02.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:02.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:02.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:02.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:02.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:02.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302621+0000", 2023-07-02T14:06:02.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2260000000000001e-06 2023-07-02T14:06:02.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:02.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302625+0000", 2023-07-02T14:06:02.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8599999999999998e-06 2023-07-02T14:06:02.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:02.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302630+0000", 2023-07-02T14:06:02.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0629999999999999e-06 2023-07-02T14:06:02.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:02.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302635+0000", 2023-07-02T14:06:02.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035731199999999998 2023-07-02T14:06:02.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302992+0000", 2023-07-02T14:06:02.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4164e-05 2023-07-02T14:06:02.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.303026+0000", 2023-07-02T14:06:02.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097115900000000008 2023-07-02T14:06:02.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312738+0000", 2023-07-02T14:06:02.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1701999999999998e-05 2023-07-02T14:06:02.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312819+0000", 2023-07-02T14:06:02.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097363640000000008 2023-07-02T14:06:02.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322556+0000", 2023-07-02T14:06:02.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2517000000000001e-05 2023-07-02T14:06:02.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322588+0000", 2023-07-02T14:06:02.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01002393 2023-07-02T14:06:02.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332612+0000", 2023-07-02T14:06:02.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2960999999999998e-05 2023-07-02T14:06:02.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332645+0000", 2023-07-02T14:06:02.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090669819999999995 2023-07-02T14:06:02.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341712+0000", 2023-07-02T14:06:02.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9288000000000002e-05 2023-07-02T14:06:02.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341811+0000", 2023-07-02T14:06:02.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081504460000000004 2023-07-02T14:06:02.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349962+0000", 2023-07-02T14:06:02.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107321 2023-07-02T14:06:02.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.350069+0000", 2023-07-02T14:06:02.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088790999999999991 2023-07-02T14:06:02.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358948+0000", 2023-07-02T14:06:02.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1711000000000002e-05 2023-07-02T14:06:02.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358980+0000", 2023-07-02T14:06:02.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080367579999999997 2023-07-02T14:06:02.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367017+0000", 2023-07-02T14:06:02.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6329000000000002e-05 2023-07-02T14:06:02.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367073+0000", 2023-07-02T14:06:02.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075097300000000001 2023-07-02T14:06:02.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374583+0000", 2023-07-02T14:06:02.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5710999999999995e-05 2023-07-02T14:06:02.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374648+0000", 2023-07-02T14:06:02.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009165912 2023-07-02T14:06:02.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383814+0000", 2023-07-02T14:06:02.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3916000000000003e-05 2023-07-02T14:06:02.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383878+0000", 2023-07-02T14:06:02.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013440532999999999 2023-07-02T14:06:02.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397319+0000", 2023-07-02T14:06:02.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7218999999999995e-05 2023-07-02T14:06:02.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397386+0000", 2023-07-02T14:06:02.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065165279999999997 2023-07-02T14:06:02.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403903+0000", 2023-07-02T14:06:02.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8152000000000005e-05 2023-07-02T14:06:02.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404001+0000", 2023-07-02T14:06:02.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072185360000000002 2023-07-02T14:06:02.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411219+0000", 2023-07-02T14:06:02.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:06:02.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411310+0000", 2023-07-02T14:06:02.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010021663 2023-07-02T14:06:02.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421331+0000", 2023-07-02T14:06:02.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0148000000000006e-05 2023-07-02T14:06:02.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421411+0000", 2023-07-02T14:06:02.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053056170000000003 2023-07-02T14:06:02.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426717+0000", 2023-07-02T14:06:02.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8242000000000001e-05 2023-07-02T14:06:02.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:02.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426795+0000", 2023-07-02T14:06:02.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00041308399999999998 2023-07-02T14:06:02.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:02.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.427208+0000", 2023-07-02T14:06:02.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028330849999999999 2023-07-02T14:06:02.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:02.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430041+0000", 2023-07-02T14:06:02.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00072642300000000002 2023-07-02T14:06:02.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:02.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430768+0000", 2023-07-02T14:06:02.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1847000000000002e-05 2023-07-02T14:06:02.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:02.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430800+0000", 2023-07-02T14:06:02.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9308999999999994e-05 2023-07-02T14:06:02.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:02.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430899+0000", 2023-07-02T14:06:02.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001 2023-07-02T14:06:02.668 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.668 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.668 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.668 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17503 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:02.668 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.400832+0000", 2023-07-02T14:06:02.669 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 18.861288215999998, 2023-07-02T14:06:02.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.102096161, 2023-07-02T14:06:02.669 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:02.669 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:02.669 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:02.669 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:02.669 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:02.669 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17503 2023-07-02T14:06:02.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.670 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:02.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:02.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400832+0000", 2023-07-02T14:06:02.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:02.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:02.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400832+0000", 2023-07-02T14:06:02.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:02.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:02.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400831+0000", 2023-07-02T14:06:02.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8929999999999998e-06 2023-07-02T14:06:02.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:02.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400835+0000", 2023-07-02T14:06:02.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1750000000000005e-06 2023-07-02T14:06:02.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:02.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400843+0000", 2023-07-02T14:06:02.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0000000000000004e-06 2023-07-02T14:06:02.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:02.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400848+0000", 2023-07-02T14:06:02.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011499 2023-07-02T14:06:02.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.400963+0000", 2023-07-02T14:06:02.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3274000000000004e-05 2023-07-02T14:06:02.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401027+0000", 2023-07-02T14:06:02.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039469880000000002 2023-07-02T14:06:02.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404974+0000", 2023-07-02T14:06:02.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1142000000000004e-05 2023-07-02T14:06:02.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.405045+0000", 2023-07-02T14:06:02.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069901540000000002 2023-07-02T14:06:02.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.412035+0000", 2023-07-02T14:06:02.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4048999999999994e-05 2023-07-02T14:06:02.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.412119+0000", 2023-07-02T14:06:02.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010323508 2023-07-02T14:06:02.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422442+0000", 2023-07-02T14:06:02.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7278000000000001e-05 2023-07-02T14:06:02.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422520+0000", 2023-07-02T14:06:02.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067846620000000003 2023-07-02T14:06:02.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.429304+0000", 2023-07-02T14:06:02.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2307000000000001e-05 2023-07-02T14:06:02.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.429337+0000", 2023-07-02T14:06:02.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036949779999999998 2023-07-02T14:06:02.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.433032+0000", 2023-07-02T14:06:02.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6023999999999998e-05 2023-07-02T14:06:02.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.433088+0000", 2023-07-02T14:06:02.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041466389999999997 2023-07-02T14:06:02.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.437234+0000", 2023-07-02T14:06:02.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7540000000000003e-05 2023-07-02T14:06:02.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.437272+0000", 2023-07-02T14:06:02.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0035993079999999998 2023-07-02T14:06:02.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.440871+0000", 2023-07-02T14:06:02.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5723000000000007e-05 2023-07-02T14:06:02.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.440947+0000", 2023-07-02T14:06:02.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051923990000000003 2023-07-02T14:06:02.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.446139+0000", 2023-07-02T14:06:02.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7749000000000004e-05 2023-07-02T14:06:02.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.446217+0000", 2023-07-02T14:06:02.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093492230000000003 2023-07-02T14:06:02.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.455566+0000", 2023-07-02T14:06:02.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1462e-05 2023-07-02T14:06:02.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.455648+0000", 2023-07-02T14:06:02.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016420769000000002 2023-07-02T14:06:02.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.472068+0000", 2023-07-02T14:06:02.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7961999999999996e-05 2023-07-02T14:06:02.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.472156+0000", 2023-07-02T14:06:02.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012805568 2023-07-02T14:06:02.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.484962+0000", 2023-07-02T14:06:02.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5397999999999998e-05 2023-07-02T14:06:02.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.485047+0000", 2023-07-02T14:06:02.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080975649999999993 2023-07-02T14:06:02.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.493145+0000", 2023-07-02T14:06:02.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00018218600000000001 2023-07-02T14:06:02.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:02.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.493327+0000", 2023-07-02T14:06:02.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00055431199999999995 2023-07-02T14:06:02.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:02.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.493881+0000", 2023-07-02T14:06:02.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003367676 2023-07-02T14:06:02.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:02.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.497249+0000", 2023-07-02T14:06:02.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054194930000000001 2023-07-02T14:06:02.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:02.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.502669+0000", 2023-07-02T14:06:02.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4320999999999996e-05 2023-07-02T14:06:02.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:02.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.502733+0000", 2023-07-02T14:06:02.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000194966 2023-07-02T14:06:02.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:02.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.502928+0000", 2023-07-02T14:06:02.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.102096161 2023-07-02T14:06:02.693 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.693 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.694 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.694 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18093 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:02.694 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:44.314438+0000", 2023-07-02T14:06:02.694 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.947681537000001, 2023-07-02T14:06:02.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.100684336, 2023-07-02T14:06:02.694 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:02.695 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:02.695 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:02.695 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:02.695 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:02.695 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18093 2023-07-02T14:06:02.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.695 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:02.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:02.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.314438+0000", 2023-07-02T14:06:02.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:02.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:02.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.314438+0000", 2023-07-02T14:06:02.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:02.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:02.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.314438+0000", 2023-07-02T14:06:02.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9950000000000001e-06 2023-07-02T14:06:02.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.314441+0000", 2023-07-02T14:06:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5540000000000002e-06 2023-07-02T14:06:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.314445+0000", 2023-07-02T14:06:02.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4939999999999999e-06 2023-07-02T14:06:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.314448+0000", 2023-07-02T14:06:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0092987369999999996 2023-07-02T14:06:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.323747+0000", 2023-07-02T14:06:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9320000000000003e-05 2023-07-02T14:06:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.323786+0000", 2023-07-02T14:06:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017813928 2023-07-02T14:06:02.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.341600+0000", 2023-07-02T14:06:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4443000000000002e-05 2023-07-02T14:06:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.341635+0000", 2023-07-02T14:06:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057434189999999996 2023-07-02T14:06:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.347378+0000", 2023-07-02T14:06:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0620999999999997e-05 2023-07-02T14:06:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.347409+0000", 2023-07-02T14:06:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056030760000000002 2023-07-02T14:06:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.353012+0000", 2023-07-02T14:06:02.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9368000000000003e-05 2023-07-02T14:06:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.353091+0000", 2023-07-02T14:06:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041744800000000004 2023-07-02T14:06:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.357265+0000", 2023-07-02T14:06:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2137000000000001e-05 2023-07-02T14:06:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.357298+0000", 2023-07-02T14:06:02.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011258729 2023-07-02T14:06:02.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.368556+0000", 2023-07-02T14:06:02.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1092000000000003e-05 2023-07-02T14:06:02.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.368627+0000", 2023-07-02T14:06:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086375040000000007 2023-07-02T14:06:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.377265+0000", 2023-07-02T14:06:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3410000000000004e-05 2023-07-02T14:06:02.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.377338+0000", 2023-07-02T14:06:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077396879999999998 2023-07-02T14:06:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.385078+0000", 2023-07-02T14:06:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000100339 2023-07-02T14:06:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.385178+0000", 2023-07-02T14:06:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080754450000000005 2023-07-02T14:06:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.393254+0000", 2023-07-02T14:06:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012267699999999999 2023-07-02T14:06:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.393377+0000", 2023-07-02T14:06:02.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089193870000000008 2023-07-02T14:06:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.402296+0000", 2023-07-02T14:06:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6298000000000003e-05 2023-07-02T14:06:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:02.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.402342+0000", 2023-07-02T14:06:02.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024651799999999999 2023-07-02T14:06:02.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:02.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.402589+0000", 2023-07-02T14:06:02.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012327615 2023-07-02T14:06:02.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:02.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.414916+0000", 2023-07-02T14:06:02.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8810000000000002e-05 2023-07-02T14:06:02.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:02.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.414965+0000", 2023-07-02T14:06:02.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8016999999999997e-05 2023-07-02T14:06:02.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:02.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.415023+0000", 2023-07-02T14:06:02.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9636000000000006e-05 2023-07-02T14:06:02.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:02.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:44.415123+0000", 2023-07-02T14:06:02.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.100684336 2023-07-02T14:06:02.715 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.715 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.715 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.716 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:410 2.5 2:bbaed619:::rbd_data.125de1449605.0000000000000109:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:06:02.716 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:02.716 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 5.811552958, 2023-07-02T14:06:02.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003, 2023-07-02T14:06:02.716 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:02.716 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:02.716 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:02.717 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:06:02.717 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:06:02.717 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:06:02.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.717 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:02.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:02.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:02.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:02.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:02.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:02.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:02.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:02.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:02.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.635e-06 2023-07-02T14:06:02.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:02.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450568+0000", 2023-07-02T14:06:02.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5970000000000002e-06 2023-07-02T14:06:02.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:02.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450573+0000", 2023-07-02T14:06:02.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3379999999999999e-06 2023-07-02T14:06:02.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:02.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450575+0000", 2023-07-02T14:06:02.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9458000000000001e-05 2023-07-02T14:06:02.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450615+0000", 2023-07-02T14:06:02.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0569999999999993e-06 2023-07-02T14:06:02.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:02.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450623+0000", 2023-07-02T14:06:02.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.63968709199999996 2023-07-02T14:06:02.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090310+0000", 2023-07-02T14:06:02.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020157399999999999 2023-07-02T14:06:02.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:02.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090512+0000", 2023-07-02T14:06:02.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027739190000000001 2023-07-02T14:06:02.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:02.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093285+0000", 2023-07-02T14:06:02.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003 2023-07-02T14:06:02.724 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.724 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.724 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.725 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:416 2.5 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:06:02.725 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:02.725 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 5.8115235250000001, 2023-07-02T14:06:02.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002, 2023-07-02T14:06:02.725 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:02.725 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:02.725 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:02.726 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:06:02.726 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:06:02.726 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:06:02.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.726 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:02.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:02.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:02.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:02.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:02.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:02.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:02.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:02.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:02.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.34e-06 2023-07-02T14:06:02.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:02.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450599+0000", 2023-07-02T14:06:02.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1549999999999998e-06 2023-07-02T14:06:02.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:02.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450604+0000", 2023-07-02T14:06:02.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:06:02.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:02.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450607+0000", 2023-07-02T14:06:02.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5717999999999999e-05 2023-07-02T14:06:02.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450643+0000", 2023-07-02T14:06:02.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1619999999999999e-06 2023-07-02T14:06:02.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:02.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450648+0000", 2023-07-02T14:06:02.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64292291800000001 2023-07-02T14:06:02.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:02.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093571+0000", 2023-07-02T14:06:02.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115628 2023-07-02T14:06:02.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:02.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093687+0000", 2023-07-02T14:06:02.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002885552 2023-07-02T14:06:02.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:02.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:02.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:02.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.096572+0000", 2023-07-02T14:06:02.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002 2023-07-02T14:06:02.733 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.733 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.733 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.733 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:02.734 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:02.734 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:02.734 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:06:02.736 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:02.734+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:02.737 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:06:02.737 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap46 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap47 b3fbb028-e267-4d25-a750-93ba18a2316a@snap47.diff 2023-07-02T14:06:02.738 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 51% complete...2023-07-02T14:06:02.536+0000 7f809135b700 0 -- 172.21.15.163:0/3184071355 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f806c004b10 msgr2=0x7f806c004f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:02.738 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 100% complete...done. 2023-07-02T14:06:02.738 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:06:02.738 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=48 2023-07-02T14:06:02.738 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap48 2023-07-02T14:06:02.780 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:06:02.835 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:02.835+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:02.887 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:02.888 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:02.888 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:02.888 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:02.899 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:06:02.937 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:02.937 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:02.936+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:02.955 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:02.973 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:02.991 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:03.010 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:03.028 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:03.029 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:03.029 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:06:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:06:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:06:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:06:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:06:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:06:03.036 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:03.036+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:03.046 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:06:03.110 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete...+ tar -C b3fbb028-e267-4d25-a750-93ba18a2316a-mnt -xzf b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4.tar.gz 2023-07-02T14:06:03.138 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:03.137+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:03.208 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:03.208 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:03.218 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:06:03.259 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:03.258+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:03.339 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:03.338+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:03.382 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:03.382 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:03.392 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:06:03.439 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:06:03.439 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:03.438+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:03.478 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 100% complete...done. 2023-07-02T14:06:03.484 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 2 2023-07-02T14:06:03.522 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:03.522 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:03.532 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:06:03.540 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:03.540+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:03.641 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:03.640+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:03.657 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:03.657 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:03.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:06:03.742 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:03.741+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:03.791 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:03.791 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:03.801 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:06:03.843 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:03.842+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:03.944 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:03.943+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:03.965 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:03.965 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:03.975 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:06:04.045 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:04.044+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:04.139 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:04.139 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:04.146 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:04.145+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:04.149 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:06:04.247 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:04.246+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:04.287 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:04.288 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:04.298 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:06:04.347 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:04.347+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:04.422 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:04.423 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:04.433 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:06:04.477 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:04.476+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:04.549 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:04.548+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:04.556 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:04.556 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:04.566 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:06:04.650 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:04.649+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:04.730 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:04.730 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:04.740 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:06:04.750 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:04.750+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:04.851 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:04.850+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:04.863 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:04.863 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:04.878 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:06:04.961 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:04.960+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:05.042 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:05.042 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:05.046 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:06:05.053 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:06:05.090 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:05.089+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:05.153 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:05.152+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:05.190 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:05.208 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:05.218 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:05.218 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:05.225 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:05.228 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:06:05.243 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:05.269 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:05.268+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:05.270 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:05.295 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:05.296 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:06:05.297 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:06:05.297 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:06:05.297 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:06:05.297 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:06:05.297 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:06:05.354 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:05.353+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:05.421 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:05.421 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:05.421 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:05.421 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:05.422 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:05.434 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:06:05.454 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:06:05.476 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:05.475+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:05.485 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap47 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap48 b3fbb028-e267-4d25-a750-93ba18a2316a@snap48.diff 2023-07-02T14:06:05.552 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 100% complete...done. 2023-07-02T14:06:05.555 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:05.554+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:05.556 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:06:05.556 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=49 2023-07-02T14:06:05.556 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap49 2023-07-02T14:06:05.634 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:05.634 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:05.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:05.634 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:05.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.635 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:16941 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:05.635 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:06:05.635 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.771992832999999, 2023-07-02T14:06:05.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11449124500000001, 2023-07-02T14:06:05.635 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:05.635 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:05.635 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:05.635 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:05.636 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:05.636 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 16941 2023-07-02T14:06:05.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.636 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:05.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:05.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:06:05.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:05.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:05.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:06:05.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:06:05.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:05.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855877+0000", 2023-07-02T14:06:05.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3209999999999999e-06 2023-07-02T14:06:05.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:05.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855880+0000", 2023-07-02T14:06:05.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9299999999999999e-06 2023-07-02T14:06:05.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:05.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855883+0000", 2023-07-02T14:06:05.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6309999999999999e-06 2023-07-02T14:06:05.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:05.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855885+0000", 2023-07-02T14:06:05.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050516850000000002 2023-07-02T14:06:05.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.860937+0000", 2023-07-02T14:06:05.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1844999999999997e-05 2023-07-02T14:06:05.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.860979+0000", 2023-07-02T14:06:05.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055853480000000004 2023-07-02T14:06:05.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.866564+0000", 2023-07-02T14:06:05.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2821999999999998e-05 2023-07-02T14:06:05.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.866597+0000", 2023-07-02T14:06:05.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050938900000000002 2023-07-02T14:06:05.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.871691+0000", 2023-07-02T14:06:05.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1210000000000001e-05 2023-07-02T14:06:05.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.871722+0000", 2023-07-02T14:06:05.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014768687000000001 2023-07-02T14:06:05.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.886491+0000", 2023-07-02T14:06:05.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7248000000000003e-05 2023-07-02T14:06:05.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.886568+0000", 2023-07-02T14:06:05.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015403168 2023-07-02T14:06:05.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.901971+0000", 2023-07-02T14:06:05.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4035000000000002e-05 2023-07-02T14:06:05.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.902005+0000", 2023-07-02T14:06:05.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014618580000000001 2023-07-02T14:06:05.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.916624+0000", 2023-07-02T14:06:05.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.443e-05 2023-07-02T14:06:05.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.916668+0000", 2023-07-02T14:06:05.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099598989999999995 2023-07-02T14:06:05.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.926628+0000", 2023-07-02T14:06:05.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3123999999999999e-05 2023-07-02T14:06:05.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.926711+0000", 2023-07-02T14:06:05.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046031930000000002 2023-07-02T14:06:05.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.931314+0000", 2023-07-02T14:06:05.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3257000000000003e-05 2023-07-02T14:06:05.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.931358+0000", 2023-07-02T14:06:05.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078370969999999995 2023-07-02T14:06:05.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.939195+0000", 2023-07-02T14:06:05.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2522000000000003e-05 2023-07-02T14:06:05.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.939247+0000", 2023-07-02T14:06:05.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01118103 2023-07-02T14:06:05.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.950428+0000", 2023-07-02T14:06:05.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5001999999999996e-05 2023-07-02T14:06:05.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.950523+0000", 2023-07-02T14:06:05.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089648390000000005 2023-07-02T14:06:05.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959488+0000", 2023-07-02T14:06:05.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4569e-05 2023-07-02T14:06:05.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:05.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959533+0000", 2023-07-02T14:06:05.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026104299999999997 2023-07-02T14:06:05.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:05.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959794+0000", 2023-07-02T14:06:05.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025221789999999998 2023-07-02T14:06:05.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:05.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.962316+0000", 2023-07-02T14:06:05.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079080520000000005 2023-07-02T14:06:05.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:05.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970224+0000", 2023-07-02T14:06:05.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1265e-05 2023-07-02T14:06:05.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:05.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970255+0000", 2023-07-02T14:06:05.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000113581 2023-07-02T14:06:05.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:05.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970369+0000", 2023-07-02T14:06:05.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11449124500000001 2023-07-02T14:06:05.657 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.657 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:05.657 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.658 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17049 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:05.658 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:06:05.658 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.648487842000002, 2023-07-02T14:06:05.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.112277108, 2023-07-02T14:06:05.658 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:05.659 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:05.659 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:05.659 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:05.659 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:05.659 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17049 2023-07-02T14:06:05.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.659 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:05.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:05.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:06:05.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:05.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:05.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:06:05.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:05.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:05.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979382+0000", 2023-07-02T14:06:05.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3450000000000002e-06 2023-07-02T14:06:05.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:05.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979385+0000", 2023-07-02T14:06:05.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9000000000000003e-06 2023-07-02T14:06:05.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:05.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979391+0000", 2023-07-02T14:06:05.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9999999999999998e-06 2023-07-02T14:06:05.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:05.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979395+0000", 2023-07-02T14:06:05.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013089412 2023-07-02T14:06:05.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992485+0000", 2023-07-02T14:06:05.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3828999999999997e-05 2023-07-02T14:06:05.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992519+0000", 2023-07-02T14:06:05.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062549889999999999 2023-07-02T14:06:05.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998774+0000", 2023-07-02T14:06:05.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4870999999999995e-05 2023-07-02T14:06:05.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998848+0000", 2023-07-02T14:06:05.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091715140000000004 2023-07-02T14:06:05.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008020+0000", 2023-07-02T14:06:05.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8659999999999999e-05 2023-07-02T14:06:05.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008059+0000", 2023-07-02T14:06:05.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059565110000000003 2023-07-02T14:06:05.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014015+0000", 2023-07-02T14:06:05.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8088000000000002e-05 2023-07-02T14:06:05.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014093+0000", 2023-07-02T14:06:05.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007513395 2023-07-02T14:06:05.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021607+0000", 2023-07-02T14:06:05.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9777e-05 2023-07-02T14:06:05.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021636+0000", 2023-07-02T14:06:05.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056999549999999996 2023-07-02T14:06:05.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027336+0000", 2023-07-02T14:06:05.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8356999999999996e-05 2023-07-02T14:06:05.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027405+0000", 2023-07-02T14:06:05.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047942130000000003 2023-07-02T14:06:05.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032199+0000", 2023-07-02T14:06:05.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1137999999999999e-05 2023-07-02T14:06:05.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032230+0000", 2023-07-02T14:06:05.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057056759999999998 2023-07-02T14:06:05.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.037936+0000", 2023-07-02T14:06:05.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6334999999999993e-05 2023-07-02T14:06:05.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038012+0000", 2023-07-02T14:06:05.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043985609999999996 2023-07-02T14:06:05.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042411+0000", 2023-07-02T14:06:05.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6857e-05 2023-07-02T14:06:05.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042487+0000", 2023-07-02T14:06:05.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007284446 2023-07-02T14:06:05.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049772+0000", 2023-07-02T14:06:05.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3757000000000006e-05 2023-07-02T14:06:05.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049846+0000", 2023-07-02T14:06:05.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0095186679999999992 2023-07-02T14:06:05.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059364+0000", 2023-07-02T14:06:05.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9931000000000005e-05 2023-07-02T14:06:05.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059434+0000", 2023-07-02T14:06:05.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079904920000000001 2023-07-02T14:06:05.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067425+0000", 2023-07-02T14:06:05.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5583000000000001e-05 2023-07-02T14:06:05.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067460+0000", 2023-07-02T14:06:05.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009114278 2023-07-02T14:06:05.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076575+0000", 2023-07-02T14:06:05.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5880000000000001e-05 2023-07-02T14:06:05.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076620+0000", 2023-07-02T14:06:05.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055118470000000003 2023-07-02T14:06:05.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082132+0000", 2023-07-02T14:06:05.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8306999999999997e-05 2023-07-02T14:06:05.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:05.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082181+0000", 2023-07-02T14:06:05.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026259599999999998 2023-07-02T14:06:05.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:05.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082443+0000", 2023-07-02T14:06:05.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018855339999999999 2023-07-02T14:06:05.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:05.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084329+0000", 2023-07-02T14:06:05.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071877729999999997 2023-07-02T14:06:05.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:05.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091517+0000", 2023-07-02T14:06:05.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3424999999999999e-05 2023-07-02T14:06:05.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:05.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091550+0000", 2023-07-02T14:06:05.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109724 2023-07-02T14:06:05.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:05.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091660+0000", 2023-07-02T14:06:05.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.112277108 2023-07-02T14:06:05.684 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.684 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:05.685 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.685 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17050 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:05.685 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:06:05.685 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.648470917000001, 2023-07-02T14:06:05.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12076701300000001, 2023-07-02T14:06:05.685 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:05.686 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:05.686 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:05.686 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:05.686 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:05.686 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17050 2023-07-02T14:06:05.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.686 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:05.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:05.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:06:05.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:05.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:05.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:06:05.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:05.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:05.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:06:05.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0759999999999999e-06 2023-07-02T14:06:05.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:05.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979402+0000", 2023-07-02T14:06:05.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0480000000000004e-06 2023-07-02T14:06:05.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:05.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979408+0000", 2023-07-02T14:06:05.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9559999999999999e-06 2023-07-02T14:06:05.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:05.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979412+0000", 2023-07-02T14:06:05.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013117658000000001 2023-07-02T14:06:05.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992530+0000", 2023-07-02T14:06:05.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4464000000000001e-05 2023-07-02T14:06:05.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992564+0000", 2023-07-02T14:06:05.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0063143640000000003 2023-07-02T14:06:05.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998879+0000", 2023-07-02T14:06:05.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2842000000000002e-05 2023-07-02T14:06:05.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998911+0000", 2023-07-02T14:06:05.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091863190000000001 2023-07-02T14:06:05.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008098+0000", 2023-07-02T14:06:05.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1448999999999999e-05 2023-07-02T14:06:05.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008179+0000", 2023-07-02T14:06:05.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059353720000000004 2023-07-02T14:06:05.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014115+0000", 2023-07-02T14:06:05.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1928000000000002e-05 2023-07-02T14:06:05.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014157+0000", 2023-07-02T14:06:05.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074935210000000004 2023-07-02T14:06:05.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021650+0000", 2023-07-02T14:06:05.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0199999999999999e-05 2023-07-02T14:06:05.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021680+0000", 2023-07-02T14:06:05.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057425109999999996 2023-07-02T14:06:05.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027423+0000", 2023-07-02T14:06:05.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1498999999999998e-05 2023-07-02T14:06:05.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027454+0000", 2023-07-02T14:06:05.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047903420000000004 2023-07-02T14:06:05.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032245+0000", 2023-07-02T14:06:05.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2552999999999998e-05 2023-07-02T14:06:05.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032277+0000", 2023-07-02T14:06:05.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057638079999999996 2023-07-02T14:06:05.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038041+0000", 2023-07-02T14:06:05.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9168000000000002e-05 2023-07-02T14:06:05.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038080+0000", 2023-07-02T14:06:05.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044291799999999996 2023-07-02T14:06:05.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042509+0000", 2023-07-02T14:06:05.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3528999999999997e-05 2023-07-02T14:06:05.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042543+0000", 2023-07-02T14:06:05.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073244729999999997 2023-07-02T14:06:05.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049867+0000", 2023-07-02T14:06:05.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2873999999999997e-05 2023-07-02T14:06:05.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049900+0000", 2023-07-02T14:06:05.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0095670149999999999 2023-07-02T14:06:05.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059467+0000", 2023-07-02T14:06:05.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0406000000000003e-05 2023-07-02T14:06:05.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059538+0000", 2023-07-02T14:06:05.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079635079999999994 2023-07-02T14:06:05.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067501+0000", 2023-07-02T14:06:05.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4795999999999997e-05 2023-07-02T14:06:05.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067536+0000", 2023-07-02T14:06:05.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091231969999999996 2023-07-02T14:06:05.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076659+0000", 2023-07-02T14:06:05.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3208999999999999e-05 2023-07-02T14:06:05.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076702+0000", 2023-07-02T14:06:05.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070660480000000001 2023-07-02T14:06:05.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.083768+0000", 2023-07-02T14:06:05.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5546000000000001e-05 2023-07-02T14:06:05.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.083814+0000", 2023-07-02T14:06:05.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078830659999999993 2023-07-02T14:06:05.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091697+0000", 2023-07-02T14:06:05.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5921000000000003e-05 2023-07-02T14:06:05.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:05.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091743+0000", 2023-07-02T14:06:05.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026571400000000001 2023-07-02T14:06:05.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:05.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.092009+0000", 2023-07-02T14:06:05.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022058429999999999 2023-07-02T14:06:05.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:05.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.094214+0000", 2023-07-02T14:06:05.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057115869999999997 2023-07-02T14:06:05.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:05.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.099926+0000", 2023-07-02T14:06:05.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0287999999999997e-05 2023-07-02T14:06:05.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:05.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.099976+0000", 2023-07-02T14:06:05.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019014999999999999 2023-07-02T14:06:05.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:05.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.100166+0000", 2023-07-02T14:06:05.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12076701300000001 2023-07-02T14:06:05.713 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.713 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:05.713 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.713 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17096 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:05.713 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:06:05.714 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.567705141000001, 2023-07-02T14:06:05.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.10620154, 2023-07-02T14:06:05.714 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:05.714 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:05.714 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:05.714 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:05.714 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:05.714 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17096 2023-07-02T14:06:05.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.715 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:05.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:05.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:06:05.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:05.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:05.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:06:05.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:05.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:05.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:06:05.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3400000000000002e-06 2023-07-02T14:06:05.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:05.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060168+0000", 2023-07-02T14:06:05.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4020000000000004e-06 2023-07-02T14:06:05.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:05.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060173+0000", 2023-07-02T14:06:05.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4259999999999999e-06 2023-07-02T14:06:05.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:05.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060176+0000", 2023-07-02T14:06:05.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00049978399999999997 2023-07-02T14:06:05.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060676+0000", 2023-07-02T14:06:05.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0135000000000004e-05 2023-07-02T14:06:05.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060756+0000", 2023-07-02T14:06:05.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074338060000000003 2023-07-02T14:06:05.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068190+0000", 2023-07-02T14:06:05.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7941000000000004e-05 2023-07-02T14:06:05.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068278+0000", 2023-07-02T14:06:05.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089611870000000007 2023-07-02T14:06:05.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077239+0000", 2023-07-02T14:06:05.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6349e-05 2023-07-02T14:06:05.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077275+0000", 2023-07-02T14:06:05.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007072142 2023-07-02T14:06:05.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084347+0000", 2023-07-02T14:06:05.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.591e-05 2023-07-02T14:06:05.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084383+0000", 2023-07-02T14:06:05.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093660119999999999 2023-07-02T14:06:05.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093749+0000", 2023-07-02T14:06:05.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1538000000000002e-05 2023-07-02T14:06:05.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093781+0000", 2023-07-02T14:06:05.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088884629999999992 2023-07-02T14:06:05.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102669+0000", 2023-07-02T14:06:05.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6032999999999997e-05 2023-07-02T14:06:05.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102725+0000", 2023-07-02T14:06:05.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009945522 2023-07-02T14:06:05.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112671+0000", 2023-07-02T14:06:05.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5741000000000002e-05 2023-07-02T14:06:05.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112707+0000", 2023-07-02T14:06:05.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096867940000000003 2023-07-02T14:06:05.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122393+0000", 2023-07-02T14:06:05.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6244999999999998e-05 2023-07-02T14:06:05.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122480+0000", 2023-07-02T14:06:05.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011251107999999999 2023-07-02T14:06:05.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133731+0000", 2023-07-02T14:06:05.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6400999999999999e-05 2023-07-02T14:06:05.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133767+0000", 2023-07-02T14:06:05.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0094677070000000006 2023-07-02T14:06:05.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.143235+0000", 2023-07-02T14:06:05.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8721999999999996e-05 2023-07-02T14:06:05.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.143314+0000", 2023-07-02T14:06:05.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051196050000000002 2023-07-02T14:06:05.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.148433+0000", 2023-07-02T14:06:05.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2484999999999997e-05 2023-07-02T14:06:05.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.148476+0000", 2023-07-02T14:06:05.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006202234 2023-07-02T14:06:05.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.154678+0000", 2023-07-02T14:06:05.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2699000000000002e-05 2023-07-02T14:06:05.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.154761+0000", 2023-07-02T14:06:05.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049653809999999996 2023-07-02T14:06:05.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.159726+0000", 2023-07-02T14:06:05.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6862e-05 2023-07-02T14:06:05.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:05.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.159773+0000", 2023-07-02T14:06:05.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024618099999999999 2023-07-02T14:06:05.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:05.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.160019+0000", 2023-07-02T14:06:05.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0019043059999999999 2023-07-02T14:06:05.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:05.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.161923+0000", 2023-07-02T14:06:05.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004316884 2023-07-02T14:06:05.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:05.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.166240+0000", 2023-07-02T14:06:05.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2611000000000003e-05 2023-07-02T14:06:05.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:05.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.166273+0000", 2023-07-02T14:06:05.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4047000000000003e-05 2023-07-02T14:06:05.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:05.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.166367+0000", 2023-07-02T14:06:05.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.10620154 2023-07-02T14:06:05.738 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.739 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:05.739 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.739 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17295 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:05.739 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.256057+0000", 2023-07-02T14:06:05.739 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.371813273000001, 2023-07-02T14:06:05.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.107964279, 2023-07-02T14:06:05.740 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:05.740 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:05.740 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:05.740 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:05.740 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:05.740 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17295 2023-07-02T14:06:05.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.740 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:05.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:05.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256057+0000", 2023-07-02T14:06:05.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:05.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:05.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256057+0000", 2023-07-02T14:06:05.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:05.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:05.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256057+0000", 2023-07-02T14:06:05.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.458e-06 2023-07-02T14:06:05.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:05.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256059+0000", 2023-07-02T14:06:05.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9019999999999999e-06 2023-07-02T14:06:05.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:05.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256062+0000", 2023-07-02T14:06:05.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9929999999999998e-06 2023-07-02T14:06:05.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:05.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256064+0000", 2023-07-02T14:06:05.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023724789999999998 2023-07-02T14:06:05.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.258437+0000", 2023-07-02T14:06:05.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7698000000000002e-05 2023-07-02T14:06:05.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.258494+0000", 2023-07-02T14:06:05.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066424140000000001 2023-07-02T14:06:05.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.265137+0000", 2023-07-02T14:06:05.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2169000000000002e-05 2023-07-02T14:06:05.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.265179+0000", 2023-07-02T14:06:05.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089047970000000008 2023-07-02T14:06:05.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.274084+0000", 2023-07-02T14:06:05.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7489e-05 2023-07-02T14:06:05.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.274121+0000", 2023-07-02T14:06:05.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013452426999999999 2023-07-02T14:06:05.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.287574+0000", 2023-07-02T14:06:05.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5149999999999996e-05 2023-07-02T14:06:05.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.287659+0000", 2023-07-02T14:06:05.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069046059999999998 2023-07-02T14:06:05.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.294563+0000", 2023-07-02T14:06:05.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7073000000000001e-05 2023-07-02T14:06:05.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.294611+0000", 2023-07-02T14:06:05.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076769480000000003 2023-07-02T14:06:05.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302287+0000", 2023-07-02T14:06:05.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4381000000000002e-05 2023-07-02T14:06:05.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302322+0000", 2023-07-02T14:06:05.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009637863 2023-07-02T14:06:05.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.311960+0000", 2023-07-02T14:06:05.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4847999999999999e-05 2023-07-02T14:06:05.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312035+0000", 2023-07-02T14:06:05.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098741579999999992 2023-07-02T14:06:05.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.321909+0000", 2023-07-02T14:06:05.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6455999999999998e-05 2023-07-02T14:06:05.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.321945+0000", 2023-07-02T14:06:05.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099894690000000008 2023-07-02T14:06:05.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.331935+0000", 2023-07-02T14:06:05.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2550000000000002e-05 2023-07-02T14:06:05.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332007+0000", 2023-07-02T14:06:05.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086760090000000002 2023-07-02T14:06:05.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.340683+0000", 2023-07-02T14:06:05.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8462999999999998e-05 2023-07-02T14:06:05.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.340752+0000", 2023-07-02T14:06:05.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00830381 2023-07-02T14:06:05.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349055+0000", 2023-07-02T14:06:05.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9289999999999994e-05 2023-07-02T14:06:05.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349145+0000", 2023-07-02T14:06:05.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067717250000000001 2023-07-02T14:06:05.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.355917+0000", 2023-07-02T14:06:05.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4431999999999995e-05 2023-07-02T14:06:05.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:05.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.355991+0000", 2023-07-02T14:06:05.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00044773399999999998 2023-07-02T14:06:05.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:05.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.356439+0000", 2023-07-02T14:06:05.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028403690000000001 2023-07-02T14:06:05.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:05.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.359279+0000", 2023-07-02T14:06:05.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045352450000000002 2023-07-02T14:06:05.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:05.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.363814+0000", 2023-07-02T14:06:05.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6167e-05 2023-07-02T14:06:05.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:05.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.363860+0000", 2023-07-02T14:06:05.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000160951 2023-07-02T14:06:05.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:05.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.364021+0000", 2023-07-02T14:06:05.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.107964279 2023-07-02T14:06:05.763 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.763 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:05.763 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.763 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17352 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:05.764 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302565+0000", 2023-07-02T14:06:05.764 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.325305057000001, 2023-07-02T14:06:05.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.105667996, 2023-07-02T14:06:05.764 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:05.764 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:05.764 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:05.764 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:05.765 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:05.765 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17352 2023-07-02T14:06:05.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.765 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:05.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:05.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302565+0000", 2023-07-02T14:06:05.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:05.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:05.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302565+0000", 2023-07-02T14:06:05.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:05.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:05.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302565+0000", 2023-07-02T14:06:05.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7000000000000002e-06 2023-07-02T14:06:05.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:05.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302568+0000", 2023-07-02T14:06:05.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2140000000000001e-06 2023-07-02T14:06:05.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:05.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302575+0000", 2023-07-02T14:06:05.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3250000000000001e-06 2023-07-02T14:06:05.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:05.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302579+0000", 2023-07-02T14:06:05.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014498700000000001 2023-07-02T14:06:05.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302724+0000", 2023-07-02T14:06:05.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5444999999999993e-05 2023-07-02T14:06:05.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302799+0000", 2023-07-02T14:06:05.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096710000000000008 2023-07-02T14:06:05.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312470+0000", 2023-07-02T14:06:05.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4798999999999997e-05 2023-07-02T14:06:05.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312505+0000", 2023-07-02T14:06:05.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098663940000000006 2023-07-02T14:06:05.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322372+0000", 2023-07-02T14:06:05.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1996000000000002e-05 2023-07-02T14:06:05.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322414+0000", 2023-07-02T14:06:05.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010015735 2023-07-02T14:06:05.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332429+0000", 2023-07-02T14:06:05.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2929000000000003e-05 2023-07-02T14:06:05.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332462+0000", 2023-07-02T14:06:05.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089318550000000007 2023-07-02T14:06:05.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341394+0000", 2023-07-02T14:06:05.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7248e-05 2023-07-02T14:06:05.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341431+0000", 2023-07-02T14:06:05.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0082266739999999998 2023-07-02T14:06:05.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349658+0000", 2023-07-02T14:06:05.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3188999999999997e-05 2023-07-02T14:06:05.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349691+0000", 2023-07-02T14:06:05.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089857329999999992 2023-07-02T14:06:05.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358677+0000", 2023-07-02T14:06:05.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6436000000000001e-05 2023-07-02T14:06:05.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358713+0000", 2023-07-02T14:06:05.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080528400000000003 2023-07-02T14:06:05.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366766+0000", 2023-07-02T14:06:05.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3217999999999999e-05 2023-07-02T14:06:05.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366799+0000", 2023-07-02T14:06:05.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074649970000000001 2023-07-02T14:06:05.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374264+0000", 2023-07-02T14:06:05.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9482999999999999e-05 2023-07-02T14:06:05.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374334+0000", 2023-07-02T14:06:05.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091483889999999998 2023-07-02T14:06:05.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383482+0000", 2023-07-02T14:06:05.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3095000000000005e-05 2023-07-02T14:06:05.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383545+0000", 2023-07-02T14:06:05.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013359414 2023-07-02T14:06:05.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.396905+0000", 2023-07-02T14:06:05.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012269900000000001 2023-07-02T14:06:05.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397028+0000", 2023-07-02T14:06:05.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046629660000000002 2023-07-02T14:06:05.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401691+0000", 2023-07-02T14:06:05.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5111999999999999e-05 2023-07-02T14:06:05.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:05.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401726+0000", 2023-07-02T14:06:05.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024867800000000001 2023-07-02T14:06:05.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:05.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.401974+0000", 2023-07-02T14:06:05.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028409360000000001 2023-07-02T14:06:05.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:05.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404815+0000", 2023-07-02T14:06:05.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031148529999999999 2023-07-02T14:06:05.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:05.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.407930+0000", 2023-07-02T14:06:05.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6836000000000003e-05 2023-07-02T14:06:05.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:05.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.407987+0000", 2023-07-02T14:06:05.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024641799999999998 2023-07-02T14:06:05.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:05.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408233+0000", 2023-07-02T14:06:05.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.105667996 2023-07-02T14:06:05.787 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.787 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:05.788 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.788 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17353 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:05.788 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:06:05.788 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.325286241000001, 2023-07-02T14:06:05.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.115724379, 2023-07-02T14:06:05.789 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:05.789 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:05.789 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:05.789 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:05.789 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:05.789 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17353 2023-07-02T14:06:05.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.789 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:05.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:05.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:06:05.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:05.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:05.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:06:05.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:05.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:05.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:06:05.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.41e-06 2023-07-02T14:06:05.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:05.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302587+0000", 2023-07-02T14:06:05.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.015e-06 2023-07-02T14:06:05.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:05.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302593+0000", 2023-07-02T14:06:05.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8019999999999997e-06 2023-07-02T14:06:05.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:05.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302599+0000", 2023-07-02T14:06:05.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022701199999999999 2023-07-02T14:06:05.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302826+0000", 2023-07-02T14:06:05.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.426e-05 2023-07-02T14:06:05.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302870+0000", 2023-07-02T14:06:05.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096759489999999997 2023-07-02T14:06:05.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312546+0000", 2023-07-02T14:06:05.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5785000000000006e-05 2023-07-02T14:06:05.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312632+0000", 2023-07-02T14:06:05.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098106970000000002 2023-07-02T14:06:05.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322443+0000", 2023-07-02T14:06:05.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3161999999999999e-05 2023-07-02T14:06:05.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322476+0000", 2023-07-02T14:06:05.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010015823 2023-07-02T14:06:05.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332492+0000", 2023-07-02T14:06:05.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1165999999999999e-05 2023-07-02T14:06:05.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332533+0000", 2023-07-02T14:06:05.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089435269999999997 2023-07-02T14:06:05.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341476+0000", 2023-07-02T14:06:05.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000112691 2023-07-02T14:06:05.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341589+0000", 2023-07-02T14:06:05.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008147362 2023-07-02T14:06:05.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349736+0000", 2023-07-02T14:06:05.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5149000000000007e-05 2023-07-02T14:06:05.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349832+0000", 2023-07-02T14:06:05.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089226519999999997 2023-07-02T14:06:05.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358754+0000", 2023-07-02T14:06:05.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6091999999999998e-05 2023-07-02T14:06:05.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358810+0000", 2023-07-02T14:06:05.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080318000000000004 2023-07-02T14:06:05.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366842+0000", 2023-07-02T14:06:05.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4321999999999998e-05 2023-07-02T14:06:05.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366916+0000", 2023-07-02T14:06:05.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074664049999999997 2023-07-02T14:06:05.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374383+0000", 2023-07-02T14:06:05.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5460000000000001e-05 2023-07-02T14:06:05.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374438+0000", 2023-07-02T14:06:05.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091562450000000004 2023-07-02T14:06:05.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383594+0000", 2023-07-02T14:06:05.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2545000000000004e-05 2023-07-02T14:06:05.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383667+0000", 2023-07-02T14:06:05.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013424551 2023-07-02T14:06:05.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397092+0000", 2023-07-02T14:06:05.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6991999999999997e-05 2023-07-02T14:06:05.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397169+0000", 2023-07-02T14:06:05.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064399139999999997 2023-07-02T14:06:05.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403608+0000", 2023-07-02T14:06:05.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2614000000000006e-05 2023-07-02T14:06:05.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403691+0000", 2023-07-02T14:06:05.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004611693 2023-07-02T14:06:05.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408303+0000", 2023-07-02T14:06:05.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1935999999999995e-05 2023-07-02T14:06:05.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:05.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408395+0000", 2023-07-02T14:06:05.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00051077700000000002 2023-07-02T14:06:05.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:05.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408906+0000", 2023-07-02T14:06:05.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090767460000000001 2023-07-02T14:06:05.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:05.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.417982+0000", 2023-07-02T14:06:05.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4283999999999997e-05 2023-07-02T14:06:05.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:05.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418037+0000", 2023-07-02T14:06:05.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9667999999999999e-05 2023-07-02T14:06:05.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:05.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418096+0000", 2023-07-02T14:06:05.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021233800000000001 2023-07-02T14:06:05.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:05.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418309+0000", 2023-07-02T14:06:05.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.115724379 2023-07-02T14:06:05.813 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.813 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:05.813 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.814 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17354 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:05.814 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:05.814 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.325267280999999, 2023-07-02T14:06:05.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493, 2023-07-02T14:06:05.814 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:05.814 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:05.815 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:05.815 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:05.815 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:05.815 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17354 2023-07-02T14:06:05.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.815 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:05.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:05.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:05.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:05.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:05.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:05.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:05.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:05.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:05.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7440000000000001e-06 2023-07-02T14:06:05.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:05.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302606+0000", 2023-07-02T14:06:05.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:06:05.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:05.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302613+0000", 2023-07-02T14:06:05.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8810000000000004e-06 2023-07-02T14:06:05.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:05.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302618+0000", 2023-07-02T14:06:05.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028613100000000002 2023-07-02T14:06:05.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302904+0000", 2023-07-02T14:06:05.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5278999999999995e-05 2023-07-02T14:06:05.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302979+0000", 2023-07-02T14:06:05.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096904750000000005 2023-07-02T14:06:05.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312669+0000", 2023-07-02T14:06:05.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0328000000000002e-05 2023-07-02T14:06:05.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312700+0000", 2023-07-02T14:06:05.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098019319999999993 2023-07-02T14:06:05.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322502+0000", 2023-07-02T14:06:05.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4669999999999998e-05 2023-07-02T14:06:05.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322536+0000", 2023-07-02T14:06:05.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010023839 2023-07-02T14:06:05.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332560+0000", 2023-07-02T14:06:05.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5559999999999998e-05 2023-07-02T14:06:05.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332596+0000", 2023-07-02T14:06:05.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090401319999999993 2023-07-02T14:06:05.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341636+0000", 2023-07-02T14:06:05.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5373000000000001e-05 2023-07-02T14:06:05.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341681+0000", 2023-07-02T14:06:05.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081994059999999994 2023-07-02T14:06:05.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349881+0000", 2023-07-02T14:06:05.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4319000000000001e-05 2023-07-02T14:06:05.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349915+0000", 2023-07-02T14:06:05.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089388339999999997 2023-07-02T14:06:05.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358854+0000", 2023-07-02T14:06:05.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4319000000000005e-05 2023-07-02T14:06:05.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358928+0000", 2023-07-02T14:06:05.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080234690000000001 2023-07-02T14:06:05.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366952+0000", 2023-07-02T14:06:05.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2082999999999998e-05 2023-07-02T14:06:05.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366984+0000", 2023-07-02T14:06:05.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075015790000000004 2023-07-02T14:06:05.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374485+0000", 2023-07-02T14:06:05.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4480999999999995e-05 2023-07-02T14:06:05.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374550+0000", 2023-07-02T14:06:05.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091666620000000008 2023-07-02T14:06:05.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383716+0000", 2023-07-02T14:06:05.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9318000000000003e-05 2023-07-02T14:06:05.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383786+0000", 2023-07-02T14:06:05.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013434115 2023-07-02T14:06:05.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397220+0000", 2023-07-02T14:06:05.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4573999999999997e-05 2023-07-02T14:06:05.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397264+0000", 2023-07-02T14:06:05.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064824649999999998 2023-07-02T14:06:05.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403747+0000", 2023-07-02T14:06:05.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1087999999999993e-05 2023-07-02T14:06:05.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403828+0000", 2023-07-02T14:06:05.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072854449999999998 2023-07-02T14:06:05.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411113+0000", 2023-07-02T14:06:05.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5120000000000001e-05 2023-07-02T14:06:05.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411169+0000", 2023-07-02T14:06:05.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007211505 2023-07-02T14:06:05.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418380+0000", 2023-07-02T14:06:05.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011289399999999999 2023-07-02T14:06:05.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:05.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418493+0000", 2023-07-02T14:06:05.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00051217500000000002 2023-07-02T14:06:05.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:05.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.419005+0000", 2023-07-02T14:06:05.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00327393 2023-07-02T14:06:05.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:05.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422279+0000", 2023-07-02T14:06:05.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041432350000000003 2023-07-02T14:06:05.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:05.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426422+0000", 2023-07-02T14:06:05.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7021000000000003e-05 2023-07-02T14:06:05.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:05.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426469+0000", 2023-07-02T14:06:05.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000190256 2023-07-02T14:06:05.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:05.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426660+0000", 2023-07-02T14:06:05.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493 2023-07-02T14:06:05.840 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.840 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:05.841 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.841 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17355 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:05.841 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:05.841 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 22.32524862, 2023-07-02T14:06:05.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001, 2023-07-02T14:06:05.841 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:05.842 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:05.842 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:05.842 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:05.842 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:05.842 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17355 2023-07-02T14:06:05.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.842 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:05.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:05.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:05.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:05.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:05.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:05.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:05.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:05.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302621+0000", 2023-07-02T14:06:05.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2260000000000001e-06 2023-07-02T14:06:05.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:05.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302625+0000", 2023-07-02T14:06:05.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8599999999999998e-06 2023-07-02T14:06:05.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:05.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302630+0000", 2023-07-02T14:06:05.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0629999999999999e-06 2023-07-02T14:06:05.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:05.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302635+0000", 2023-07-02T14:06:05.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035731199999999998 2023-07-02T14:06:05.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302992+0000", 2023-07-02T14:06:05.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4164e-05 2023-07-02T14:06:05.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.303026+0000", 2023-07-02T14:06:05.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097115900000000008 2023-07-02T14:06:05.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312738+0000", 2023-07-02T14:06:05.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1701999999999998e-05 2023-07-02T14:06:05.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312819+0000", 2023-07-02T14:06:05.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097363640000000008 2023-07-02T14:06:05.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322556+0000", 2023-07-02T14:06:05.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2517000000000001e-05 2023-07-02T14:06:05.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322588+0000", 2023-07-02T14:06:05.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01002393 2023-07-02T14:06:05.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332612+0000", 2023-07-02T14:06:05.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2960999999999998e-05 2023-07-02T14:06:05.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332645+0000", 2023-07-02T14:06:05.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090669819999999995 2023-07-02T14:06:05.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341712+0000", 2023-07-02T14:06:05.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9288000000000002e-05 2023-07-02T14:06:05.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341811+0000", 2023-07-02T14:06:05.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081504460000000004 2023-07-02T14:06:05.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349962+0000", 2023-07-02T14:06:05.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107321 2023-07-02T14:06:05.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.350069+0000", 2023-07-02T14:06:05.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088790999999999991 2023-07-02T14:06:05.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358948+0000", 2023-07-02T14:06:05.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1711000000000002e-05 2023-07-02T14:06:05.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358980+0000", 2023-07-02T14:06:05.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080367579999999997 2023-07-02T14:06:05.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367017+0000", 2023-07-02T14:06:05.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6329000000000002e-05 2023-07-02T14:06:05.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367073+0000", 2023-07-02T14:06:05.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075097300000000001 2023-07-02T14:06:05.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374583+0000", 2023-07-02T14:06:05.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5710999999999995e-05 2023-07-02T14:06:05.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374648+0000", 2023-07-02T14:06:05.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009165912 2023-07-02T14:06:05.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383814+0000", 2023-07-02T14:06:05.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3916000000000003e-05 2023-07-02T14:06:05.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383878+0000", 2023-07-02T14:06:05.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013440532999999999 2023-07-02T14:06:05.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397319+0000", 2023-07-02T14:06:05.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7218999999999995e-05 2023-07-02T14:06:05.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397386+0000", 2023-07-02T14:06:05.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065165279999999997 2023-07-02T14:06:05.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403903+0000", 2023-07-02T14:06:05.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8152000000000005e-05 2023-07-02T14:06:05.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404001+0000", 2023-07-02T14:06:05.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072185360000000002 2023-07-02T14:06:05.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411219+0000", 2023-07-02T14:06:05.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:06:05.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411310+0000", 2023-07-02T14:06:05.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010021663 2023-07-02T14:06:05.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421331+0000", 2023-07-02T14:06:05.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0148000000000006e-05 2023-07-02T14:06:05.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421411+0000", 2023-07-02T14:06:05.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053056170000000003 2023-07-02T14:06:05.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426717+0000", 2023-07-02T14:06:05.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8242000000000001e-05 2023-07-02T14:06:05.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:05.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426795+0000", 2023-07-02T14:06:05.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00041308399999999998 2023-07-02T14:06:05.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:05.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.427208+0000", 2023-07-02T14:06:05.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028330849999999999 2023-07-02T14:06:05.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:05.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430041+0000", 2023-07-02T14:06:05.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00072642300000000002 2023-07-02T14:06:05.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:05.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430768+0000", 2023-07-02T14:06:05.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1847000000000002e-05 2023-07-02T14:06:05.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:05.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430800+0000", 2023-07-02T14:06:05.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9308999999999994e-05 2023-07-02T14:06:05.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:05.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430899+0000", 2023-07-02T14:06:05.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001 2023-07-02T14:06:05.869 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.869 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:05.869 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.869 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:410 2.5 2:bbaed619:::rbd_data.125de1449605.0000000000000109:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:06:05.869 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:05.869 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.1773033549999994, 2023-07-02T14:06:05.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003, 2023-07-02T14:06:05.870 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:05.870 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:05.870 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:05.870 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:06:05.870 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:06:05.870 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:06:05.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.871 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:05.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:05.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:05.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:05.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:05.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:05.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:05.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:05.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:05.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.635e-06 2023-07-02T14:06:05.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:05.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450568+0000", 2023-07-02T14:06:05.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5970000000000002e-06 2023-07-02T14:06:05.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:05.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450573+0000", 2023-07-02T14:06:05.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3379999999999999e-06 2023-07-02T14:06:05.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:05.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450575+0000", 2023-07-02T14:06:05.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9458000000000001e-05 2023-07-02T14:06:05.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450615+0000", 2023-07-02T14:06:05.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0569999999999993e-06 2023-07-02T14:06:05.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:05.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450623+0000", 2023-07-02T14:06:05.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.63968709199999996 2023-07-02T14:06:05.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090310+0000", 2023-07-02T14:06:05.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020157399999999999 2023-07-02T14:06:05.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:05.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090512+0000", 2023-07-02T14:06:05.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027739190000000001 2023-07-02T14:06:05.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:05.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093285+0000", 2023-07-02T14:06:05.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003 2023-07-02T14:06:05.878 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.878 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:05.878 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.878 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:416 2.5 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:06:05.878 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:05.879 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.1772739219999995, 2023-07-02T14:06:05.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002, 2023-07-02T14:06:05.879 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:05.879 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:05.879 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:05.879 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:06:05.879 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:06:05.880 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:06:05.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.880 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:05.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:05.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:05.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:05.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:05.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:05.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:05.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:05.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:05.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.34e-06 2023-07-02T14:06:05.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:05.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450599+0000", 2023-07-02T14:06:05.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1549999999999998e-06 2023-07-02T14:06:05.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:05.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450604+0000", 2023-07-02T14:06:05.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:06:05.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:05.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450607+0000", 2023-07-02T14:06:05.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5717999999999999e-05 2023-07-02T14:06:05.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450643+0000", 2023-07-02T14:06:05.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1619999999999999e-06 2023-07-02T14:06:05.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:05.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450648+0000", 2023-07-02T14:06:05.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64292291800000001 2023-07-02T14:06:05.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093571+0000", 2023-07-02T14:06:05.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115628 2023-07-02T14:06:05.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:05.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093687+0000", 2023-07-02T14:06:05.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002885552 2023-07-02T14:06:05.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:05.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.096572+0000", 2023-07-02T14:06:05.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002 2023-07-02T14:06:05.887 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.887 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:05.887 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.887 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18552 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:05.888 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.244993+0000", 2023-07-02T14:06:05.888 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.382877782, 2023-07-02T14:06:05.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.107684349, 2023-07-02T14:06:05.888 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:05.888 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:05.888 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:05.888 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:05.889 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:05.889 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18552 2023-07-02T14:06:05.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.889 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:05.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:05.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.244993+0000", 2023-07-02T14:06:05.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:05.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:05.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.244993+0000", 2023-07-02T14:06:05.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:05.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:05.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.244992+0000", 2023-07-02T14:06:05.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.613e-06 2023-07-02T14:06:05.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:05.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.244996+0000", 2023-07-02T14:06:05.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7490000000000004e-06 2023-07-02T14:06:05.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:05.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.245002+0000", 2023-07-02T14:06:05.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3120000000000004e-06 2023-07-02T14:06:05.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:05.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.245006+0000", 2023-07-02T14:06:05.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000359706 2023-07-02T14:06:05.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.893 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.893 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.245366+0000", 2023-07-02T14:06:05.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0433999999999999e-05 2023-07-02T14:06:05.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.245406+0000", 2023-07-02T14:06:05.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041032769999999998 2023-07-02T14:06:05.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249509+0000", 2023-07-02T14:06:05.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3030000000000001e-05 2023-07-02T14:06:05.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249542+0000", 2023-07-02T14:06:05.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066355299999999997 2023-07-02T14:06:05.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256178+0000", 2023-07-02T14:06:05.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5154999999999998e-05 2023-07-02T14:06:05.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256213+0000", 2023-07-02T14:06:05.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012170596000000001 2023-07-02T14:06:05.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268384+0000", 2023-07-02T14:06:05.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3485000000000006e-05 2023-07-02T14:06:05.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268447+0000", 2023-07-02T14:06:05.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011136736 2023-07-02T14:06:05.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279584+0000", 2023-07-02T14:06:05.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3447e-05 2023-07-02T14:06:05.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279617+0000", 2023-07-02T14:06:05.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011815668 2023-07-02T14:06:05.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291433+0000", 2023-07-02T14:06:05.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0329e-05 2023-07-02T14:06:05.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291463+0000", 2023-07-02T14:06:05.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079621299999999996 2023-07-02T14:06:05.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299425+0000", 2023-07-02T14:06:05.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6808999999999999e-05 2023-07-02T14:06:05.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299462+0000", 2023-07-02T14:06:05.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072025659999999997 2023-07-02T14:06:05.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306665+0000", 2023-07-02T14:06:05.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8457e-05 2023-07-02T14:06:05.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306723+0000", 2023-07-02T14:06:05.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056322869999999997 2023-07-02T14:06:05.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312355+0000", 2023-07-02T14:06:05.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4682999999999998e-05 2023-07-02T14:06:05.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312410+0000", 2023-07-02T14:06:05.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004796453 2023-07-02T14:06:05.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317207+0000", 2023-07-02T14:06:05.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6775999999999999e-05 2023-07-02T14:06:05.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317273+0000", 2023-07-02T14:06:05.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049760610000000004 2023-07-02T14:06:05.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322249+0000", 2023-07-02T14:06:05.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7802999999999999e-05 2023-07-02T14:06:05.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322337+0000", 2023-07-02T14:06:05.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066115770000000004 2023-07-02T14:06:05.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.328949+0000", 2023-07-02T14:06:05.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6105e-05 2023-07-02T14:06:05.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.328995+0000", 2023-07-02T14:06:05.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062196949999999999 2023-07-02T14:06:05.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335215+0000", 2023-07-02T14:06:05.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0904e-05 2023-07-02T14:06:05.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335256+0000", 2023-07-02T14:06:05.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096283380000000002 2023-07-02T14:06:05.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.344884+0000", 2023-07-02T14:06:05.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7175999999999994e-05 2023-07-02T14:06:05.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.344961+0000", 2023-07-02T14:06:05.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029952550000000001 2023-07-02T14:06:05.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.347956+0000", 2023-07-02T14:06:05.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8840000000000001e-05 2023-07-02T14:06:05.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:05.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.347995+0000", 2023-07-02T14:06:05.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00023967300000000001 2023-07-02T14:06:05.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:05.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.348235+0000", 2023-07-02T14:06:05.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023177219999999999 2023-07-02T14:06:05.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:05.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350553+0000", 2023-07-02T14:06:05.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0019716069999999998 2023-07-02T14:06:05.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:05.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.352524+0000", 2023-07-02T14:06:05.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0142e-05 2023-07-02T14:06:05.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:05.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.352554+0000", 2023-07-02T14:06:05.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012266600000000001 2023-07-02T14:06:05.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:05.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.352677+0000", 2023-07-02T14:06:05.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.107684349 2023-07-02T14:06:05.915 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.915 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:05.915 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.916 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18554 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:05.916 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248917+0000", 2023-07-02T14:06:05.916 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.37895310599999998, 2023-07-02T14:06:05.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11109714599999999, 2023-07-02T14:06:05.916 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:05.916 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:05.917 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:05.917 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:05.917 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:05.917 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18554 2023-07-02T14:06:05.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.917 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:05.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:05.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248917+0000", 2023-07-02T14:06:05.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:05.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:05.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248917+0000", 2023-07-02T14:06:05.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:05.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:05.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248916+0000", 2023-07-02T14:06:05.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1690000000000003e-06 2023-07-02T14:06:05.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:05.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248923+0000", 2023-07-02T14:06:05.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2769999999999994e-06 2023-07-02T14:06:05.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:05.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248933+0000", 2023-07-02T14:06:05.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1860000000000002e-06 2023-07-02T14:06:05.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:05.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248937+0000", 2023-07-02T14:06:05.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00063144000000000004 2023-07-02T14:06:05.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249568+0000", 2023-07-02T14:06:05.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1486999999999998e-05 2023-07-02T14:06:05.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.922 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249610+0000", 2023-07-02T14:06:05.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066288199999999997 2023-07-02T14:06:05.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256239+0000", 2023-07-02T14:06:05.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3636000000000001e-05 2023-07-02T14:06:05.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256272+0000", 2023-07-02T14:06:05.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01221675 2023-07-02T14:06:05.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268489+0000", 2023-07-02T14:06:05.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0283000000000004e-05 2023-07-02T14:06:05.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268569+0000", 2023-07-02T14:06:05.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011087254 2023-07-02T14:06:05.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279657+0000", 2023-07-02T14:06:05.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2872e-05 2023-07-02T14:06:05.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279689+0000", 2023-07-02T14:06:05.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011797277 2023-07-02T14:06:05.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291487+0000", 2023-07-02T14:06:05.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0111e-05 2023-07-02T14:06:05.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291517+0000", 2023-07-02T14:06:05.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079886279999999994 2023-07-02T14:06:05.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299505+0000", 2023-07-02T14:06:05.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2787000000000006e-05 2023-07-02T14:06:05.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299578+0000", 2023-07-02T14:06:05.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071806889999999996 2023-07-02T14:06:05.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306759+0000", 2023-07-02T14:06:05.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1510000000000002e-05 2023-07-02T14:06:05.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306790+0000", 2023-07-02T14:06:05.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056651269999999998 2023-07-02T14:06:05.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312456+0000", 2023-07-02T14:06:05.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8580999999999999e-05 2023-07-02T14:06:05.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312534+0000", 2023-07-02T14:06:05.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047950680000000004 2023-07-02T14:06:05.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317329+0000", 2023-07-02T14:06:05.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4718000000000002e-05 2023-07-02T14:06:05.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317364+0000", 2023-07-02T14:06:05.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050140469999999998 2023-07-02T14:06:05.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322378+0000", 2023-07-02T14:06:05.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1049e-05 2023-07-02T14:06:05.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322409+0000", 2023-07-02T14:06:05.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006616612 2023-07-02T14:06:05.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329026+0000", 2023-07-02T14:06:05.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4612e-05 2023-07-02T14:06:05.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329060+0000", 2023-07-02T14:06:05.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062225020000000004 2023-07-02T14:06:05.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335283+0000", 2023-07-02T14:06:05.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0385e-05 2023-07-02T14:06:05.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335323+0000", 2023-07-02T14:06:05.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009678262 2023-07-02T14:06:05.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345001+0000", 2023-07-02T14:06:05.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0340999999999999e-05 2023-07-02T14:06:05.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345042+0000", 2023-07-02T14:06:05.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045838650000000003 2023-07-02T14:06:05.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349626+0000", 2023-07-02T14:06:05.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010221999999999999 2023-07-02T14:06:05.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349728+0000", 2023-07-02T14:06:05.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029884349999999998 2023-07-02T14:06:05.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.352716+0000", 2023-07-02T14:06:05.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3821e-05 2023-07-02T14:06:05.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:05.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.352750+0000", 2023-07-02T14:06:05.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021565399999999999 2023-07-02T14:06:05.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:05.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.352966+0000", 2023-07-02T14:06:05.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018362960000000001 2023-07-02T14:06:05.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:05.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354802+0000", 2023-07-02T14:06:05.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051013409999999997 2023-07-02T14:06:05.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:05.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.359903+0000", 2023-07-02T14:06:05.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9068999999999999e-05 2023-07-02T14:06:05.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:05.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.359932+0000", 2023-07-02T14:06:05.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2008000000000005e-05 2023-07-02T14:06:05.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:05.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.360014+0000", 2023-07-02T14:06:05.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11109714599999999 2023-07-02T14:06:05.943 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.944 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:05.944 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.944 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18555 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:05.944 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248941+0000", 2023-07-02T14:06:05.944 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.37892950800000003, 2023-07-02T14:06:05.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11809699899999999, 2023-07-02T14:06:05.945 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:05.945 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:05.945 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:05.945 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:05.945 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:05.945 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18555 2023-07-02T14:06:05.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.945 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:05.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:05.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248941+0000", 2023-07-02T14:06:05.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:05.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:05.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248941+0000", 2023-07-02T14:06:05.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:05.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:05.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248940+0000", 2023-07-02T14:06:05.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.7159999999999999e-06 2023-07-02T14:06:05.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:05.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248943+0000", 2023-07-02T14:06:05.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5949999999999999e-06 2023-07-02T14:06:05.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:05.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248947+0000", 2023-07-02T14:06:05.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4480000000000001e-06 2023-07-02T14:06:05.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:05.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248949+0000", 2023-07-02T14:06:05.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00067899499999999995 2023-07-02T14:06:05.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249628+0000", 2023-07-02T14:06:05.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1100000000000003e-05 2023-07-02T14:06:05.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249669+0000", 2023-07-02T14:06:05.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066349690000000001 2023-07-02T14:06:05.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256304+0000", 2023-07-02T14:06:05.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2691999999999998e-05 2023-07-02T14:06:05.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256337+0000", 2023-07-02T14:06:05.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012284679999999999 2023-07-02T14:06:05.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268622+0000", 2023-07-02T14:06:05.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6550000000000001e-05 2023-07-02T14:06:05.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268658+0000", 2023-07-02T14:06:05.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011061109 2023-07-02T14:06:05.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279719+0000", 2023-07-02T14:06:05.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2134000000000001e-05 2023-07-02T14:06:05.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.954 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279751+0000", 2023-07-02T14:06:05.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011798329999999999 2023-07-02T14:06:05.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291550+0000", 2023-07-02T14:06:05.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9839999999999999e-05 2023-07-02T14:06:05.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291580+0000", 2023-07-02T14:06:05.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080386550000000005 2023-07-02T14:06:05.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299618+0000", 2023-07-02T14:06:05.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2632999999999997e-05 2023-07-02T14:06:05.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299651+0000", 2023-07-02T14:06:05.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071769859999999998 2023-07-02T14:06:05.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306828+0000", 2023-07-02T14:06:05.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1123000000000004e-05 2023-07-02T14:06:05.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306889+0000", 2023-07-02T14:06:05.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056835710000000001 2023-07-02T14:06:05.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312573+0000", 2023-07-02T14:06:05.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0115000000000002e-05 2023-07-02T14:06:05.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312643+0000", 2023-07-02T14:06:05.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004754538 2023-07-02T14:06:05.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317397+0000", 2023-07-02T14:06:05.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4385000000000002e-05 2023-07-02T14:06:05.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317452+0000", 2023-07-02T14:06:05.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049988350000000001 2023-07-02T14:06:05.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322450+0000", 2023-07-02T14:06:05.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4749e-05 2023-07-02T14:06:05.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322515+0000", 2023-07-02T14:06:05.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065710810000000003 2023-07-02T14:06:05.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329086+0000", 2023-07-02T14:06:05.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0982000000000003e-05 2023-07-02T14:06:05.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329117+0000", 2023-07-02T14:06:05.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062301259999999999 2023-07-02T14:06:05.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335347+0000", 2023-07-02T14:06:05.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2830999999999998e-05 2023-07-02T14:06:05.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335380+0000", 2023-07-02T14:06:05.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097041909999999992 2023-07-02T14:06:05.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345084+0000", 2023-07-02T14:06:05.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2044000000000003e-05 2023-07-02T14:06:05.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345156+0000", 2023-07-02T14:06:05.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004610711 2023-07-02T14:06:05.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349767+0000", 2023-07-02T14:06:05.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8640000000000002e-05 2023-07-02T14:06:05.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349816+0000", 2023-07-02T14:06:05.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043231160000000001 2023-07-02T14:06:05.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354139+0000", 2023-07-02T14:06:05.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8073e-05 2023-07-02T14:06:05.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354177+0000", 2023-07-02T14:06:05.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005870681 2023-07-02T14:06:05.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.360048+0000", 2023-07-02T14:06:05.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9535e-05 2023-07-02T14:06:05.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:05.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.360087+0000", 2023-07-02T14:06:05.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021778100000000001 2023-07-02T14:06:05.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:05.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.360305+0000", 2023-07-02T14:06:05.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001720708 2023-07-02T14:06:05.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:05.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.362026+0000", 2023-07-02T14:06:05.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049001009999999996 2023-07-02T14:06:05.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:05.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.366926+0000", 2023-07-02T14:06:05.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3682e-05 2023-07-02T14:06:05.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:05.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.366959+0000", 2023-07-02T14:06:05.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8419999999999998e-05 2023-07-02T14:06:05.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:05.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.367038+0000", 2023-07-02T14:06:05.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11809699899999999 2023-07-02T14:06:05.973 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.973 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:05.973 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:05.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.974 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18556 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:05.974 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248957+0000", 2023-07-02T14:06:05.974 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.37891353900000002, 2023-07-02T14:06:05.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12347108800000001, 2023-07-02T14:06:05.974 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:05.974 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:05.974 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:05.975 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:05.975 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:05.975 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18556 2023-07-02T14:06:05.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.975 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:05.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:05.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248957+0000", 2023-07-02T14:06:05.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:05.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:05.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248957+0000", 2023-07-02T14:06:05.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:05.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:05.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248957+0000", 2023-07-02T14:06:05.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4119999999999999e-06 2023-07-02T14:06:05.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:05.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248959+0000", 2023-07-02T14:06:05.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5669999999999999e-06 2023-07-02T14:06:05.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:05.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248962+0000", 2023-07-02T14:06:05.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3989999999999998e-06 2023-07-02T14:06:05.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:05.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248965+0000", 2023-07-02T14:06:05.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00072489500000000003 2023-07-02T14:06:05.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249690+0000", 2023-07-02T14:06:05.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7876999999999997e-05 2023-07-02T14:06:05.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249728+0000", 2023-07-02T14:06:05.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066341250000000003 2023-07-02T14:06:05.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256362+0000", 2023-07-02T14:06:05.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2715000000000001e-05 2023-07-02T14:06:05.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256395+0000", 2023-07-02T14:06:05.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012305508 2023-07-02T14:06:05.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268700+0000", 2023-07-02T14:06:05.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7537e-05 2023-07-02T14:06:05.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268778+0000", 2023-07-02T14:06:05.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011004300999999999 2023-07-02T14:06:05.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279782+0000", 2023-07-02T14:06:05.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1996000000000003e-05 2023-07-02T14:06:05.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279814+0000", 2023-07-02T14:06:05.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011789104 2023-07-02T14:06:05.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291603+0000", 2023-07-02T14:06:05.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9825999999999999e-05 2023-07-02T14:06:05.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291633+0000", 2023-07-02T14:06:05.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080637 2023-07-02T14:06:05.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299696+0000", 2023-07-02T14:06:05.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8535999999999996e-05 2023-07-02T14:06:05.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299765+0000", 2023-07-02T14:06:05.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071538169999999998 2023-07-02T14:06:05.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306919+0000", 2023-07-02T14:06:05.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0337000000000001e-05 2023-07-02T14:06:05.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306949+0000", 2023-07-02T14:06:05.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057379919999999999 2023-07-02T14:06:05.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312687+0000", 2023-07-02T14:06:05.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8463000000000001e-05 2023-07-02T14:06:05.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312726+0000", 2023-07-02T14:06:05.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004766251 2023-07-02T14:06:05.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317492+0000", 2023-07-02T14:06:05.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1788000000000001e-05 2023-07-02T14:06:05.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317524+0000", 2023-07-02T14:06:05.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050322910000000004 2023-07-02T14:06:05.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322556+0000", 2023-07-02T14:06:05.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9141999999999999e-05 2023-07-02T14:06:05.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322595+0000", 2023-07-02T14:06:05.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006553354 2023-07-02T14:06:05.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329148+0000", 2023-07-02T14:06:05.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2458e-05 2023-07-02T14:06:05.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329181+0000", 2023-07-02T14:06:05.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006222483 2023-07-02T14:06:05.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335403+0000", 2023-07-02T14:06:05.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0419000000000001e-05 2023-07-02T14:06:05.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335434+0000", 2023-07-02T14:06:05.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097556550000000002 2023-07-02T14:06:05.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345189+0000", 2023-07-02T14:06:05.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2968e-05 2023-07-02T14:06:05.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345222+0000", 2023-07-02T14:06:05.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046402270000000002 2023-07-02T14:06:05.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349863+0000", 2023-07-02T14:06:05.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7782999999999997e-05 2023-07-02T14:06:05.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349940+0000", 2023-07-02T14:06:05.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042690699999999998 2023-07-02T14:06:05.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354210+0000", 2023-07-02T14:06:05.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5648000000000001e-05 2023-07-02T14:06:05.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354255+0000", 2023-07-02T14:06:05.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071341269999999997 2023-07-02T14:06:05.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:05.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361389+0000", 2023-07-02T14:06:05.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9140000000000001e-05 2023-07-02T14:06:05.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:05.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:05.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361428+0000", 2023-07-02T14:06:05.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056418470000000002 2023-07-02T14:06:05.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:05.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.367070+0000", 2023-07-02T14:06:06.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1863000000000002e-05 2023-07-02T14:06:06.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:06.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.367112+0000", 2023-07-02T14:06:06.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00027357599999999998 2023-07-02T14:06:06.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:06.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.367386+0000", 2023-07-02T14:06:06.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020277680000000001 2023-07-02T14:06:06.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:06.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369413+0000", 2023-07-02T14:06:06.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029075699999999999 2023-07-02T14:06:06.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:06.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372321+0000", 2023-07-02T14:06:06.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1955999999999997e-05 2023-07-02T14:06:06.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:06.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372353+0000", 2023-07-02T14:06:06.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4938999999999995e-05 2023-07-02T14:06:06.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:06.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372428+0000", 2023-07-02T14:06:06.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12347108800000001 2023-07-02T14:06:06.004 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:06.004 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:06.004 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:06.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.004 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18557 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:06.005 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:06.005 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.37890154300000001, 2023-07-02T14:06:06.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13120991400000001, 2023-07-02T14:06:06.005 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:06.005 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.005 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:06.005 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:06.006 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:06.006 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18557 2023-07-02T14:06:06.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.006 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:06.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:06.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:06.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:06.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:06.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:06.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:06:06.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:06.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:06.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8870000000000001e-06 2023-07-02T14:06:06.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:06.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248971+0000", 2023-07-02T14:06:06.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9009999999999996e-06 2023-07-02T14:06:06.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:06.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248974+0000", 2023-07-02T14:06:06.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9809999999999997e-06 2023-07-02T14:06:06.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248978+0000", 2023-07-02T14:06:06.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00076989199999999999 2023-07-02T14:06:06.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249748+0000", 2023-07-02T14:06:06.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8245999999999999e-05 2023-07-02T14:06:06.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249787+0000", 2023-07-02T14:06:06.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066332969999999998 2023-07-02T14:06:06.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256420+0000", 2023-07-02T14:06:06.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2566e-05 2023-07-02T14:06:06.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256452+0000", 2023-07-02T14:06:06.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012364714000000001 2023-07-02T14:06:06.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268817+0000", 2023-07-02T14:06:06.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5113000000000001e-05 2023-07-02T14:06:06.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268852+0000", 2023-07-02T14:06:06.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010993899999999999 2023-07-02T14:06:06.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279846+0000", 2023-07-02T14:06:06.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2144999999999998e-05 2023-07-02T14:06:06.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279878+0000", 2023-07-02T14:06:06.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011778958000000001 2023-07-02T14:06:06.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291657+0000", 2023-07-02T14:06:06.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9592000000000001e-05 2023-07-02T14:06:06.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291687+0000", 2023-07-02T14:06:06.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081107220000000008 2023-07-02T14:06:06.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299798+0000", 2023-07-02T14:06:06.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0290999999999998e-05 2023-07-02T14:06:06.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299828+0000", 2023-07-02T14:06:06.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071584489999999999 2023-07-02T14:06:06.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306986+0000", 2023-07-02T14:06:06.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0755999999999999e-05 2023-07-02T14:06:06.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307047+0000", 2023-07-02T14:06:06.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057110260000000001 2023-07-02T14:06:06.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312758+0000", 2023-07-02T14:06:06.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5145999999999999e-05 2023-07-02T14:06:06.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312793+0000", 2023-07-02T14:06:06.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004758361 2023-07-02T14:06:06.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317552+0000", 2023-07-02T14:06:06.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9190000000000002e-05 2023-07-02T14:06:06.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317601+0000", 2023-07-02T14:06:06.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050329420000000003 2023-07-02T14:06:06.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322634+0000", 2023-07-02T14:06:06.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5737999999999998e-05 2023-07-02T14:06:06.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322719+0000", 2023-07-02T14:06:06.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064995679999999998 2023-07-02T14:06:06.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329219+0000", 2023-07-02T14:06:06.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0908000000000003e-05 2023-07-02T14:06:06.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329250+0000", 2023-07-02T14:06:06.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062072029999999997 2023-07-02T14:06:06.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335457+0000", 2023-07-02T14:06:06.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9471999999999998e-05 2023-07-02T14:06:06.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335487+0000", 2023-07-02T14:06:06.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097780239999999997 2023-07-02T14:06:06.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345265+0000", 2023-07-02T14:06:06.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0982999999999994e-05 2023-07-02T14:06:06.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345336+0000", 2023-07-02T14:06:06.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046378799999999996 2023-07-02T14:06:06.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349973+0000", 2023-07-02T14:06:06.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1069999999999999e-05 2023-07-02T14:06:06.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350005+0000", 2023-07-02T14:06:06.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042766870000000004 2023-07-02T14:06:06.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354281+0000", 2023-07-02T14:06:06.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1155000000000002e-05 2023-07-02T14:06:06.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354312+0000", 2023-07-02T14:06:06.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071436820000000002 2023-07-02T14:06:06.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361456+0000", 2023-07-02T14:06:06.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9054000000000002e-05 2023-07-02T14:06:06.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361505+0000", 2023-07-02T14:06:06.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072301320000000002 2023-07-02T14:06:06.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368735+0000", 2023-07-02T14:06:06.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1013999999999994e-05 2023-07-02T14:06:06.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368816+0000", 2023-07-02T14:06:06.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036582490000000001 2023-07-02T14:06:06.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372475+0000", 2023-07-02T14:06:06.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4652000000000004e-05 2023-07-02T14:06:06.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:06.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372539+0000", 2023-07-02T14:06:06.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00031374199999999999 2023-07-02T14:06:06.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:06.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372853+0000", 2023-07-02T14:06:06.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030071299999999998 2023-07-02T14:06:06.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:06.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.375860+0000", 2023-07-02T14:06:06.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041897749999999997 2023-07-02T14:06:06.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:06.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380050+0000", 2023-07-02T14:06:06.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8331000000000003e-05 2023-07-02T14:06:06.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:06.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380088+0000", 2023-07-02T14:06:06.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0626999999999998e-05 2023-07-02T14:06:06.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:06.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380179+0000", 2023-07-02T14:06:06.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13120991400000001 2023-07-02T14:06:06.036 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:06.036 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:06.036 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:06.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.037 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18558 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:06.037 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:06.037 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.37888484099999997, 2023-07-02T14:06:06.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13755567899999999, 2023-07-02T14:06:06.037 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:06.037 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.037 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:06.037 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:06.038 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:06.038 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18558 2023-07-02T14:06:06.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.038 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:06.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:06.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:06.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:06.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:06.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:06.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:06.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:06.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248985+0000", 2023-07-02T14:06:06.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8630000000000001e-06 2023-07-02T14:06:06.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:06.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248989+0000", 2023-07-02T14:06:06.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0750000000000003e-06 2023-07-02T14:06:06.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:06.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248995+0000", 2023-07-02T14:06:06.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3290000000000004e-06 2023-07-02T14:06:06.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248999+0000", 2023-07-02T14:06:06.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00080656300000000005 2023-07-02T14:06:06.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249806+0000", 2023-07-02T14:06:06.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2040999999999999e-05 2023-07-02T14:06:06.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249848+0000", 2023-07-02T14:06:06.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066312419999999999 2023-07-02T14:06:06.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256479+0000", 2023-07-02T14:06:06.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3348e-05 2023-07-02T14:06:06.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256513+0000", 2023-07-02T14:06:06.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012391531000000001 2023-07-02T14:06:06.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268904+0000", 2023-07-02T14:06:06.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.347e-05 2023-07-02T14:06:06.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268978+0000", 2023-07-02T14:06:06.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010929772000000001 2023-07-02T14:06:06.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279907+0000", 2023-07-02T14:06:06.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2669999999999997e-05 2023-07-02T14:06:06.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279940+0000", 2023-07-02T14:06:06.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011770332999999999 2023-07-02T14:06:06.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291710+0000", 2023-07-02T14:06:06.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1207000000000001e-05 2023-07-02T14:06:06.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291742+0000", 2023-07-02T14:06:06.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081242199999999997 2023-07-02T14:06:06.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299866+0000", 2023-07-02T14:06:06.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4650000000000002e-05 2023-07-02T14:06:06.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299900+0000", 2023-07-02T14:06:06.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071810270000000004 2023-07-02T14:06:06.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307081+0000", 2023-07-02T14:06:06.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4378000000000002e-05 2023-07-02T14:06:06.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307116+0000", 2023-07-02T14:06:06.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057031909999999998 2023-07-02T14:06:06.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312819+0000", 2023-07-02T14:06:06.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1612999999999997e-05 2023-07-02T14:06:06.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312851+0000", 2023-07-02T14:06:06.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048172470000000002 2023-07-02T14:06:06.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317668+0000", 2023-07-02T14:06:06.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3328e-05 2023-07-02T14:06:06.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317741+0000", 2023-07-02T14:06:06.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050117290000000004 2023-07-02T14:06:06.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322753+0000", 2023-07-02T14:06:06.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9872000000000001e-05 2023-07-02T14:06:06.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322783+0000", 2023-07-02T14:06:06.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064910610000000002 2023-07-02T14:06:06.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329274+0000", 2023-07-02T14:06:06.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0097e-05 2023-07-02T14:06:06.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329304+0000", 2023-07-02T14:06:06.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062163879999999998 2023-07-02T14:06:06.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335520+0000", 2023-07-02T14:06:06.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9513e-05 2023-07-02T14:06:06.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335550+0000", 2023-07-02T14:06:06.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098209969999999997 2023-07-02T14:06:06.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345371+0000", 2023-07-02T14:06:06.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9862999999999999e-05 2023-07-02T14:06:06.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345401+0000", 2023-07-02T14:06:06.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046552030000000001 2023-07-02T14:06:06.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350056+0000", 2023-07-02T14:06:06.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8751999999999993e-05 2023-07-02T14:06:06.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350145+0000", 2023-07-02T14:06:06.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041981450000000003 2023-07-02T14:06:06.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354343+0000", 2023-07-02T14:06:06.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1501000000000002e-05 2023-07-02T14:06:06.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354374+0000", 2023-07-02T14:06:06.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071551740000000003 2023-07-02T14:06:06.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361530+0000", 2023-07-02T14:06:06.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5156e-05 2023-07-02T14:06:06.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361565+0000", 2023-07-02T14:06:06.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072918569999999997 2023-07-02T14:06:06.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368857+0000", 2023-07-02T14:06:06.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5770999999999999e-05 2023-07-02T14:06:06.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368902+0000", 2023-07-02T14:06:06.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057210569999999999 2023-07-02T14:06:06.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374623+0000", 2023-07-02T14:06:06.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3266999999999998e-05 2023-07-02T14:06:06.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374667+0000", 2023-07-02T14:06:06.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055479570000000001 2023-07-02T14:06:06.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380215+0000", 2023-07-02T14:06:06.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0067e-05 2023-07-02T14:06:06.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:06.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380265+0000", 2023-07-02T14:06:06.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00043082299999999999 2023-07-02T14:06:06.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:06.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380696+0000", 2023-07-02T14:06:06.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024853240000000001 2023-07-02T14:06:06.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:06.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.383181+0000", 2023-07-02T14:06:06.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032601280000000002 2023-07-02T14:06:06.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:06.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386441+0000", 2023-07-02T14:06:06.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9748e-05 2023-07-02T14:06:06.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:06.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386471+0000", 2023-07-02T14:06:06.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0531000000000006e-05 2023-07-02T14:06:06.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:06.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386541+0000", 2023-07-02T14:06:06.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13755567899999999 2023-07-02T14:06:06.069 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:06.070 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:06.070 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:06.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.070 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18559 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:06.070 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:06.070 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.37886223699999999, 2023-07-02T14:06:06.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.146184591, 2023-07-02T14:06:06.071 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:06.071 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.071 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:06.071 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:06.071 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:06.071 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18559 2023-07-02T14:06:06.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.071 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:06.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:06.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:06.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:06.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:06.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:06.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:06.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:06.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:06.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.726e-06 2023-07-02T14:06:06.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:06.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249011+0000", 2023-07-02T14:06:06.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7939999999999997e-06 2023-07-02T14:06:06.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:06.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249018+0000", 2023-07-02T14:06:06.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3470000000000001e-06 2023-07-02T14:06:06.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249023+0000", 2023-07-02T14:06:06.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00084402900000000005 2023-07-02T14:06:06.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249867+0000", 2023-07-02T14:06:06.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9050000000000001e-05 2023-07-02T14:06:06.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249906+0000", 2023-07-02T14:06:06.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066311039999999996 2023-07-02T14:06:06.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256537+0000", 2023-07-02T14:06:06.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2790000000000003e-05 2023-07-02T14:06:06.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256570+0000", 2023-07-02T14:06:06.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012436129000000001 2023-07-02T14:06:06.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269006+0000", 2023-07-02T14:06:06.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0173999999999999e-05 2023-07-02T14:06:06.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269046+0000", 2023-07-02T14:06:06.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010924698 2023-07-02T14:06:06.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279971+0000", 2023-07-02T14:06:06.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3831000000000002e-05 2023-07-02T14:06:06.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280004+0000", 2023-07-02T14:06:06.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011795912 2023-07-02T14:06:06.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291800+0000", 2023-07-02T14:06:06.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5613999999999997e-05 2023-07-02T14:06:06.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291886+0000", 2023-07-02T14:06:06.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080395999999999992 2023-07-02T14:06:06.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299925+0000", 2023-07-02T14:06:06.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0694999999999997e-05 2023-07-02T14:06:06.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299956+0000", 2023-07-02T14:06:06.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072049449999999999 2023-07-02T14:06:06.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307161+0000", 2023-07-02T14:06:06.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5523000000000001e-05 2023-07-02T14:06:06.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307247+0000", 2023-07-02T14:06:06.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056331929999999999 2023-07-02T14:06:06.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312880+0000", 2023-07-02T14:06:06.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1646000000000002e-05 2023-07-02T14:06:06.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312911+0000", 2023-07-02T14:06:06.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048627540000000004 2023-07-02T14:06:06.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317774+0000", 2023-07-02T14:06:06.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2008000000000003e-05 2023-07-02T14:06:06.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317806+0000", 2023-07-02T14:06:06.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050055689999999996 2023-07-02T14:06:06.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322812+0000", 2023-07-02T14:06:06.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3442999999999997e-05 2023-07-02T14:06:06.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322855+0000", 2023-07-02T14:06:06.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006472813 2023-07-02T14:06:06.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329328+0000", 2023-07-02T14:06:06.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0128e-05 2023-07-02T14:06:06.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329358+0000", 2023-07-02T14:06:06.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062156490000000002 2023-07-02T14:06:06.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335574+0000", 2023-07-02T14:06:06.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2357999999999997e-05 2023-07-02T14:06:06.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335606+0000", 2023-07-02T14:06:06.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098318409999999992 2023-07-02T14:06:06.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345438+0000", 2023-07-02T14:06:06.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1174000000000007e-05 2023-07-02T14:06:06.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345509+0000", 2023-07-02T14:06:06.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004669851 2023-07-02T14:06:06.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350179+0000", 2023-07-02T14:06:06.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1559000000000001e-05 2023-07-02T14:06:06.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350211+0000", 2023-07-02T14:06:06.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041865740000000002 2023-07-02T14:06:06.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354397+0000", 2023-07-02T14:06:06.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9850000000000001e-05 2023-07-02T14:06:06.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354427+0000", 2023-07-02T14:06:06.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071608449999999999 2023-07-02T14:06:06.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361588+0000", 2023-07-02T14:06:06.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8346000000000002e-05 2023-07-02T14:06:06.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361626+0000", 2023-07-02T14:06:06.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073175009999999997 2023-07-02T14:06:06.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368944+0000", 2023-07-02T14:06:06.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2879000000000005e-05 2023-07-02T14:06:06.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369017+0000", 2023-07-02T14:06:06.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056850690000000001 2023-07-02T14:06:06.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374702+0000", 2023-07-02T14:06:06.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7693e-05 2023-07-02T14:06:06.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374739+0000", 2023-07-02T14:06:06.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076925149999999996 2023-07-02T14:06:06.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382432+0000", 2023-07-02T14:06:06.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3259e-05 2023-07-02T14:06:06.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382475+0000", 2023-07-02T14:06:06.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041005110000000003 2023-07-02T14:06:06.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386576+0000", 2023-07-02T14:06:06.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2074999999999999e-05 2023-07-02T14:06:06.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:06.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386618+0000", 2023-07-02T14:06:06.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000233692 2023-07-02T14:06:06.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:06.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386851+0000", 2023-07-02T14:06:06.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024022269999999998 2023-07-02T14:06:06.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:06.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.389254+0000", 2023-07-02T14:06:06.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058506469999999996 2023-07-02T14:06:06.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:06.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395104+0000", 2023-07-02T14:06:06.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8807000000000003e-05 2023-07-02T14:06:06.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:06.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395143+0000", 2023-07-02T14:06:06.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9616000000000001e-05 2023-07-02T14:06:06.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:06.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395193+0000", 2023-07-02T14:06:06.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.146184591 2023-07-02T14:06:06.104 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:06.104 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:06.104 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:06.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.105 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18560 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:06.105 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:06.105 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.378843614, 2023-07-02T14:06:06.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.15647551900000001, 2023-07-02T14:06:06.105 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:06.105 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.106 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:06.106 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:06.106 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:06.106 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18560 2023-07-02T14:06:06.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.106 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:06.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:06.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:06.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:06.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:06.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:06.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:06.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:06.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249026+0000", 2023-07-02T14:06:06.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3419999999999999e-06 2023-07-02T14:06:06.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:06.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249030+0000", 2023-07-02T14:06:06.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7869999999999998e-06 2023-07-02T14:06:06.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:06.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249035+0000", 2023-07-02T14:06:06.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9639999999999997e-06 2023-07-02T14:06:06.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249039+0000", 2023-07-02T14:06:06.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00088477599999999999 2023-07-02T14:06:06.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249924+0000", 2023-07-02T14:06:06.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7333999999999999e-05 2023-07-02T14:06:06.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249962+0000", 2023-07-02T14:06:06.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066323409999999999 2023-07-02T14:06:06.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256594+0000", 2023-07-02T14:06:06.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2595999999999998e-05 2023-07-02T14:06:06.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256627+0000", 2023-07-02T14:06:06.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012449894 2023-07-02T14:06:06.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269076+0000", 2023-07-02T14:06:06.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3942999999999997e-05 2023-07-02T14:06:06.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269110+0000", 2023-07-02T14:06:06.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010923884 2023-07-02T14:06:06.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280034+0000", 2023-07-02T14:06:06.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7774000000000001e-05 2023-07-02T14:06:06.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280072+0000", 2023-07-02T14:06:06.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011850027000000001 2023-07-02T14:06:06.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291922+0000", 2023-07-02T14:06:06.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2128000000000001e-05 2023-07-02T14:06:06.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291954+0000", 2023-07-02T14:06:06.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080261150000000003 2023-07-02T14:06:06.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299980+0000", 2023-07-02T14:06:06.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6322999999999998e-05 2023-07-02T14:06:06.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300017+0000", 2023-07-02T14:06:06.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072683039999999997 2023-07-02T14:06:06.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307285+0000", 2023-07-02T14:06:06.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3309000000000002e-05 2023-07-02T14:06:06.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307318+0000", 2023-07-02T14:06:06.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056178649999999997 2023-07-02T14:06:06.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312936+0000", 2023-07-02T14:06:06.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1866000000000003e-05 2023-07-02T14:06:06.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312968+0000", 2023-07-02T14:06:06.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048789970000000004 2023-07-02T14:06:06.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317847+0000", 2023-07-02T14:06:06.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0223999999999996e-05 2023-07-02T14:06:06.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317927+0000", 2023-07-02T14:06:06.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049543649999999996 2023-07-02T14:06:06.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322882+0000", 2023-07-02T14:06:06.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1399000000000002e-05 2023-07-02T14:06:06.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322913+0000", 2023-07-02T14:06:06.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006534214 2023-07-02T14:06:06.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329447+0000", 2023-07-02T14:06:06.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9649999999999998e-05 2023-07-02T14:06:06.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329527+0000", 2023-07-02T14:06:06.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061026839999999997 2023-07-02T14:06:06.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335629+0000", 2023-07-02T14:06:06.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9611000000000002e-05 2023-07-02T14:06:06.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335659+0000", 2023-07-02T14:06:06.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098851549999999996 2023-07-02T14:06:06.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345544+0000", 2023-07-02T14:06:06.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0587999999999999e-05 2023-07-02T14:06:06.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345575+0000", 2023-07-02T14:06:06.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046790859999999998 2023-07-02T14:06:06.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350254+0000", 2023-07-02T14:06:06.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0284000000000002e-05 2023-07-02T14:06:06.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350304+0000", 2023-07-02T14:06:06.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041560540000000002 2023-07-02T14:06:06.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354460+0000", 2023-07-02T14:06:06.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2427000000000004e-05 2023-07-02T14:06:06.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354523+0000", 2023-07-02T14:06:06.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071268770000000002 2023-07-02T14:06:06.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361650+0000", 2023-07-02T14:06:06.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0026999999999999e-05 2023-07-02T14:06:06.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361680+0000", 2023-07-02T14:06:06.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073733080000000003 2023-07-02T14:06:06.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369053+0000", 2023-07-02T14:06:06.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7635999999999997e-05 2023-07-02T14:06:06.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369091+0000", 2023-07-02T14:06:06.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056908059999999996 2023-07-02T14:06:06.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374781+0000", 2023-07-02T14:06:06.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4656000000000002e-05 2023-07-02T14:06:06.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374816+0000", 2023-07-02T14:06:06.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077037809999999998 2023-07-02T14:06:06.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382520+0000", 2023-07-02T14:06:06.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7858e-05 2023-07-02T14:06:06.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382588+0000", 2023-07-02T14:06:06.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057884019999999998 2023-07-02T14:06:06.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388376+0000", 2023-07-02T14:06:06.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4831000000000001e-05 2023-07-02T14:06:06.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388461+0000", 2023-07-02T14:06:06.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067708480000000003 2023-07-02T14:06:06.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395232+0000", 2023-07-02T14:06:06.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7948000000000002e-05 2023-07-02T14:06:06.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:06.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395290+0000", 2023-07-02T14:06:06.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00027672599999999998 2023-07-02T14:06:06.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:06.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395566+0000", 2023-07-02T14:06:06.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025705569999999998 2023-07-02T14:06:06.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:06.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.398137+0000", 2023-07-02T14:06:06.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072257099999999998 2023-07-02T14:06:06.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:06.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405363+0000", 2023-07-02T14:06:06.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3887999999999998e-05 2023-07-02T14:06:06.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:06.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405397+0000", 2023-07-02T14:06:06.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000105769 2023-07-02T14:06:06.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:06.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405502+0000", 2023-07-02T14:06:06.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.15647551900000001 2023-07-02T14:06:06.140 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:06.140 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:06.140 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:06.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.141 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18561 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:06.141 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:06.141 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.37882325, 2023-07-02T14:06:06.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744, 2023-07-02T14:06:06.141 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:06.141 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.141 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:06.142 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:06.142 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:06.142 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18561 2023-07-02T14:06:06.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.142 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:06.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:06.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:06.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:06.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:06.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:06.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:06.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:06.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:06.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.563e-06 2023-07-02T14:06:06.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:06.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249050+0000", 2023-07-02T14:06:06.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4500000000000001e-06 2023-07-02T14:06:06.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:06.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249057+0000", 2023-07-02T14:06:06.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1860000000000002e-06 2023-07-02T14:06:06.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249061+0000", 2023-07-02T14:06:06.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00091153399999999998 2023-07-02T14:06:06.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249972+0000", 2023-07-02T14:06:06.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7156999999999997e-05 2023-07-02T14:06:06.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.250010+0000", 2023-07-02T14:06:06.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006641816 2023-07-02T14:06:06.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256651+0000", 2023-07-02T14:06:06.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1968999999999998e-05 2023-07-02T14:06:06.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256683+0000", 2023-07-02T14:06:06.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012463952 2023-07-02T14:06:06.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269147+0000", 2023-07-02T14:06:06.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2855000000000006e-05 2023-07-02T14:06:06.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269220+0000", 2023-07-02T14:06:06.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010870418999999999 2023-07-02T14:06:06.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280091+0000", 2023-07-02T14:06:06.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1921999999999997e-05 2023-07-02T14:06:06.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280123+0000", 2023-07-02T14:06:06.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011861682 2023-07-02T14:06:06.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291984+0000", 2023-07-02T14:06:06.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5259000000000006e-05 2023-07-02T14:06:06.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.292059+0000", 2023-07-02T14:06:06.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079709359999999996 2023-07-02T14:06:06.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300030+0000", 2023-07-02T14:06:06.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0729999999999998e-05 2023-07-02T14:06:06.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300071+0000", 2023-07-02T14:06:06.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072785690000000004 2023-07-02T14:06:06.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307350+0000", 2023-07-02T14:06:06.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1948999999999999e-05 2023-07-02T14:06:06.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307412+0000", 2023-07-02T14:06:06.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055785779999999998 2023-07-02T14:06:06.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312990+0000", 2023-07-02T14:06:06.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0957000000000002e-05 2023-07-02T14:06:06.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.313021+0000", 2023-07-02T14:06:06.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049246400000000001 2023-07-02T14:06:06.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317946+0000", 2023-07-02T14:06:06.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9346e-05 2023-07-02T14:06:06.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317975+0000", 2023-07-02T14:06:06.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049649469999999999 2023-07-02T14:06:06.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322940+0000", 2023-07-02T14:06:06.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9017000000000002e-05 2023-07-02T14:06:06.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322989+0000", 2023-07-02T14:06:06.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065907029999999998 2023-07-02T14:06:06.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329580+0000", 2023-07-02T14:06:06.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5879999999999999e-05 2023-07-02T14:06:06.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329656+0000", 2023-07-02T14:06:06.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060166960000000002 2023-07-02T14:06:06.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335672+0000", 2023-07-02T14:06:06.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8963e-05 2023-07-02T14:06:06.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335701+0000", 2023-07-02T14:06:06.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098989560000000004 2023-07-02T14:06:06.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345600+0000", 2023-07-02T14:06:06.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5395999999999995e-05 2023-07-02T14:06:06.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345666+0000", 2023-07-02T14:06:06.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046589530000000004 2023-07-02T14:06:06.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350325+0000", 2023-07-02T14:06:06.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2459000000000002e-05 2023-07-02T14:06:06.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350357+0000", 2023-07-02T14:06:06.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042054290000000001 2023-07-02T14:06:06.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354563+0000", 2023-07-02T14:06:06.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5612999999999999e-05 2023-07-02T14:06:06.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354598+0000", 2023-07-02T14:06:06.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070967449999999998 2023-07-02T14:06:06.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361695+0000", 2023-07-02T14:06:06.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9314000000000001e-05 2023-07-02T14:06:06.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361724+0000", 2023-07-02T14:06:06.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007391028 2023-07-02T14:06:06.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369115+0000", 2023-07-02T14:06:06.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9163000000000002e-05 2023-07-02T14:06:06.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369184+0000", 2023-07-02T14:06:06.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056532500000000003 2023-07-02T14:06:06.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374838+0000", 2023-07-02T14:06:06.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5297999999999999e-05 2023-07-02T14:06:06.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374883+0000", 2023-07-02T14:06:06.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077292189999999998 2023-07-02T14:06:06.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382612+0000", 2023-07-02T14:06:06.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1102000000000001e-05 2023-07-02T14:06:06.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382653+0000", 2023-07-02T14:06:06.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058515989999999999 2023-07-02T14:06:06.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388505+0000", 2023-07-02T14:06:06.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9196000000000006e-05 2023-07-02T14:06:06.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388574+0000", 2023-07-02T14:06:06.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084245339999999991 2023-07-02T14:06:06.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.396999+0000", 2023-07-02T14:06:06.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0596999999999998e-05 2023-07-02T14:06:06.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:06.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.397039+0000", 2023-07-02T14:06:06.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084911339999999991 2023-07-02T14:06:06.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:06.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405530+0000", 2023-07-02T14:06:06.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7555000000000001e-05 2023-07-02T14:06:06.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:06.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405578+0000", 2023-07-02T14:06:06.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000229607 2023-07-02T14:06:06.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:06.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405808+0000", 2023-07-02T14:06:06.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026919940000000001 2023-07-02T14:06:06.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:06.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.408500+0000", 2023-07-02T14:06:06.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053065059999999999 2023-07-02T14:06:06.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:06.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413806+0000", 2023-07-02T14:06:06.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0490000000000001e-05 2023-07-02T14:06:06.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:06.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413837+0000", 2023-07-02T14:06:06.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000100386 2023-07-02T14:06:06.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:06.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:06.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:06.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413937+0000", 2023-07-02T14:06:06.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744 2023-07-02T14:06:06.177 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:06.177 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:06.177 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:06.178 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:06.178 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:06.178 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:06.179 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:06:06.180 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:06.178+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:06.280 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:06.279+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:06.306 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:06.307 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:06.307 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:06.307 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:06.317 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:06:06.381 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:06.380+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:06.441 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:06.441 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:06.441 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:06:06.441 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:06:06.441 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:06.452 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:06:06.456 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:06:06.496 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:06:06.504 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 3 2023-07-02T14:06:06.507 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:06.505+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:06.583 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:06.582+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:06.634 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:06.634 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:06:06.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:06:06.634 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:06.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.634 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18607 2.4 e176/169 2:27cdd850:::rbd_data.125de1449605.0000000000000009:head v 176'1697, mlcod=176'1694)", 2023-07-02T14:06:06.635 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.358892+0000", 2023-07-02T14:06:06.635 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.2736284410000001, 2023-07-02T14:06:06.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060718129999999997, 2023-07-02T14:06:06.635 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:06.635 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.635 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:06.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:06.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.358892+0000", 2023-07-02T14:06:06.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:06.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:06.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.358892+0000", 2023-07-02T14:06:06.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:06.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:06.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.358890+0000", 2023-07-02T14:06:06.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019239999999999999 2023-07-02T14:06:06.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:06.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.359083+0000", 2023-07-02T14:06:06.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4688999999999999e-05 2023-07-02T14:06:06.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:06.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.359098+0000", 2023-07-02T14:06:06.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3610000000000003e-06 2023-07-02T14:06:06.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.359105+0000", 2023-07-02T14:06:06.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9206999999999998e-05 2023-07-02T14:06:06.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:06.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.359154+0000", 2023-07-02T14:06:06.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6939000000000001e-05 2023-07-02T14:06:06.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:06.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.359181+0000", 2023-07-02T14:06:06.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0057403690000000004 2023-07-02T14:06:06.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:06.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.364921+0000", 2023-07-02T14:06:06.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2417999999999999e-05 2023-07-02T14:06:06.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:06.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.364964+0000", 2023-07-02T14:06:06.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060718129999999997 2023-07-02T14:06:06.642 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.642 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.642 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.642 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18641 2.4 e176/169 2:2223f49a:::rbd_data.125de1449605.0000000000000186:head v 176'1701, mlcod=176'1698)", 2023-07-02T14:06:06.643 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.417060+0000", 2023-07-02T14:06:06.643 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.2154603660000001, 2023-07-02T14:06:06.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0094924280000000007, 2023-07-02T14:06:06.643 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:06.643 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.643 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:06.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:06.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.417060+0000", 2023-07-02T14:06:06.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:06.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:06.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.417060+0000", 2023-07-02T14:06:06.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:06.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:06.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.417058+0000", 2023-07-02T14:06:06.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022442769999999998 2023-07-02T14:06:06.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:06.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.419302+0000", 2023-07-02T14:06:06.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4287999999999999e-05 2023-07-02T14:06:06.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:06.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.419317+0000", 2023-07-02T14:06:06.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3890000000000002e-06 2023-07-02T14:06:06.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.419323+0000", 2023-07-02T14:06:06.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5989000000000002e-05 2023-07-02T14:06:06.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:06.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.419379+0000", 2023-07-02T14:06:06.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7104e-05 2023-07-02T14:06:06.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:06.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.419406+0000", 2023-07-02T14:06:06.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0070985570000000001 2023-07-02T14:06:06.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:06.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426505+0000", 2023-07-02T14:06:06.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.791e-05 2023-07-02T14:06:06.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:06.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426553+0000", 2023-07-02T14:06:06.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0094924280000000007 2023-07-02T14:06:06.650 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.650 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.650 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.650 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18643 2.5 e176/170 2:a9ace6f8:::rbd_data.125de1449605.0000000000000187:head v 176'2314, mlcod=173'2311)", 2023-07-02T14:06:06.650 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:06:06.651 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.2059719259999999, 2023-07-02T14:06:06.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015027297, 2023-07-02T14:06:06.651 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:06.651 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.651 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:06.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:06.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:06:06.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:06.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:06.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:06:06.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:06.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:06.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426547+0000", 2023-07-02T14:06:06.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056864430000000002 2023-07-02T14:06:06.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:06.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432234+0000", 2023-07-02T14:06:06.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.415e-05 2023-07-02T14:06:06.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:06.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432248+0000", 2023-07-02T14:06:06.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7490000000000001e-06 2023-07-02T14:06:06.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432255+0000", 2023-07-02T14:06:06.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3915000000000002e-05 2023-07-02T14:06:06.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:06.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432299+0000", 2023-07-02T14:06:06.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1214000000000001e-05 2023-07-02T14:06:06.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:06.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432320+0000", 2023-07-02T14:06:06.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0092114950000000001 2023-07-02T14:06:06.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:06.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.441531+0000", 2023-07-02T14:06:06.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4422999999999998e-05 2023-07-02T14:06:06.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:06.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.441576+0000", 2023-07-02T14:06:06.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015027297 2023-07-02T14:06:06.658 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.658 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.658 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.658 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18644 2.2 e176/171 2:44e6ed94:::rbd_data.125de1449605.0000000000000185:head v 176'1626, mlcod=176'1623)", 2023-07-02T14:06:06.658 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.428712+0000", 2023-07-02T14:06:06.658 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.203808185, 2023-07-02T14:06:06.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012273601, 2023-07-02T14:06:06.659 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:06.659 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.659 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:06.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:06.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.428712+0000", 2023-07-02T14:06:06.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:06.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:06.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.428712+0000", 2023-07-02T14:06:06.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:06.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:06.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.428711+0000", 2023-07-02T14:06:06.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017963639999999999 2023-07-02T14:06:06.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:06.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.430507+0000", 2023-07-02T14:06:06.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0897e-05 2023-07-02T14:06:06.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:06.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.430528+0000", 2023-07-02T14:06:06.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7940000000000008e-06 2023-07-02T14:06:06.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.430537+0000", 2023-07-02T14:06:06.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00034051099999999999 2023-07-02T14:06:06.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:06.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.430878+0000", 2023-07-02T14:06:06.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5058000000000001e-05 2023-07-02T14:06:06.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:06.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.430893+0000", 2023-07-02T14:06:06.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010060174 2023-07-02T14:06:06.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:06.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.440953+0000", 2023-07-02T14:06:06.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3064000000000001e-05 2023-07-02T14:06:06.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:06.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.440986+0000", 2023-07-02T14:06:06.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012273601 2023-07-02T14:06:06.665 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.666 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.666 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.666 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18650 2.2 e176/171 2:52fb801b:::rbd_data.125de1449605.0000000000000188:head v 176'1629, mlcod=176'1626)", 2023-07-02T14:06:06.666 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.447115+0000", 2023-07-02T14:06:06.666 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.1854054279999999, 2023-07-02T14:06:06.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010075483, 2023-07-02T14:06:06.667 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:06.667 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.667 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:06.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:06.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.447115+0000", 2023-07-02T14:06:06.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:06.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:06.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.447115+0000", 2023-07-02T14:06:06.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:06.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:06.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.447114+0000", 2023-07-02T14:06:06.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015636440000000001 2023-07-02T14:06:06.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:06.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.448677+0000", 2023-07-02T14:06:06.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1951e-05 2023-07-02T14:06:06.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:06.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.448699+0000", 2023-07-02T14:06:06.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6589999999999996e-06 2023-07-02T14:06:06.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.448708+0000", 2023-07-02T14:06:06.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3374999999999995e-05 2023-07-02T14:06:06.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:06.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.448771+0000", 2023-07-02T14:06:06.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7480000000000001e-05 2023-07-02T14:06:06.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:06.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.448799+0000", 2023-07-02T14:06:06.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0083377360000000001 2023-07-02T14:06:06.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:06.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.457137+0000", 2023-07-02T14:06:06.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4017999999999997e-05 2023-07-02T14:06:06.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:06.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.457191+0000", 2023-07-02T14:06:06.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010075483 2023-07-02T14:06:06.673 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.673 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.673 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.674 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18663 2.2 e176/171 2:5ecc17e8:::rbd_data.125de1449605.000000000000018d:head v 176'1636, mlcod=176'1631)", 2023-07-02T14:06:06.674 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.472111+0000", 2023-07-02T14:06:06.674 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.1604090389999999, 2023-07-02T14:06:06.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068624360000000004, 2023-07-02T14:06:06.674 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:06.675 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.675 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:06.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:06.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.472111+0000", 2023-07-02T14:06:06.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:06.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:06.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.472111+0000", 2023-07-02T14:06:06.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:06.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:06.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.472110+0000", 2023-07-02T14:06:06.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3269000000000003e-05 2023-07-02T14:06:06.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:06.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.472143+0000", 2023-07-02T14:06:06.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1117000000000001e-05 2023-07-02T14:06:06.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:06.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.472164+0000", 2023-07-02T14:06:06.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6560000000000004e-06 2023-07-02T14:06:06.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.472174+0000", 2023-07-02T14:06:06.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021704499999999999 2023-07-02T14:06:06.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:06.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.472391+0000", 2023-07-02T14:06:06.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4508999999999999e-05 2023-07-02T14:06:06.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:06.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.472415+0000", 2023-07-02T14:06:06.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065112809999999998 2023-07-02T14:06:06.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:06.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.478927+0000", 2023-07-02T14:06:06.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7142999999999999e-05 2023-07-02T14:06:06.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:06.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.478974+0000", 2023-07-02T14:06:06.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068624360000000004 2023-07-02T14:06:06.681 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.681 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.681 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.682 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18680 2.2 e176/171 2:5fd2923a:::rbd_data.125de1449605.000000000000018c:head v 176'1639, mlcod=176'1637)", 2023-07-02T14:06:06.682 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.489711+0000", 2023-07-02T14:06:06.682 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.142809484, 2023-07-02T14:06:06.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014407402, 2023-07-02T14:06:06.682 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:06.682 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.682 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:06.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:06.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.489711+0000", 2023-07-02T14:06:06.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:06.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:06.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.489711+0000", 2023-07-02T14:06:06.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:06.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:06.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.489709+0000", 2023-07-02T14:06:06.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001991792 2023-07-02T14:06:06.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:06.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491701+0000", 2023-07-02T14:06:06.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7323000000000002e-05 2023-07-02T14:06:06.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:06.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491738+0000", 2023-07-02T14:06:06.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2483999999999999e-05 2023-07-02T14:06:06.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491751+0000", 2023-07-02T14:06:06.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7896000000000003e-05 2023-07-02T14:06:06.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:06.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491809+0000", 2023-07-02T14:06:06.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0185000000000001e-05 2023-07-02T14:06:06.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:06.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491829+0000", 2023-07-02T14:06:06.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012250901999999999 2023-07-02T14:06:06.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:06.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.504080+0000", 2023-07-02T14:06:06.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:06:06.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:06.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.504118+0000", 2023-07-02T14:06:06.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014407402 2023-07-02T14:06:06.689 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.689 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.689 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.690 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18681 2.2 e176/171 2:5ecc17e8:::rbd_data.125de1449605.000000000000018d:head v 176'1640, mlcod=176'1637)", 2023-07-02T14:06:06.690 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.497048+0000", 2023-07-02T14:06:06.690 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.1354723440000001, 2023-07-02T14:06:06.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012067603999999999, 2023-07-02T14:06:06.690 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:06.690 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.690 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:06.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:06.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.497048+0000", 2023-07-02T14:06:06.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:06.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:06.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.497048+0000", 2023-07-02T14:06:06.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:06.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:06.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.497047+0000", 2023-07-02T14:06:06.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021639139999999999 2023-07-02T14:06:06.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:06.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.499211+0000", 2023-07-02T14:06:06.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7983e-05 2023-07-02T14:06:06.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:06.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.499239+0000", 2023-07-02T14:06:06.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6110999999999999e-05 2023-07-02T14:06:06.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.499255+0000", 2023-07-02T14:06:06.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0260999999999998e-05 2023-07-02T14:06:06.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:06.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.499315+0000", 2023-07-02T14:06:06.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0349e-05 2023-07-02T14:06:06.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:06.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.499345+0000", 2023-07-02T14:06:06.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097133659999999993 2023-07-02T14:06:06.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:06.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.509059+0000", 2023-07-02T14:06:06.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7092999999999997e-05 2023-07-02T14:06:06.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:06.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.509116+0000", 2023-07-02T14:06:06.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012067603999999999 2023-07-02T14:06:06.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.697 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.697 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.697 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18690 2.2 e176/171 2:416a1700:::rbd_data.125de1449605.000000000000018f:head v 176'1644, mlcod=176'1641)", 2023-07-02T14:06:06.698 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.523233+0000", 2023-07-02T14:06:06.698 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.109287353, 2023-07-02T14:06:06.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01191801, 2023-07-02T14:06:06.698 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:06.698 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.698 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:06.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:06.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.523233+0000", 2023-07-02T14:06:06.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:06.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:06.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.523233+0000", 2023-07-02T14:06:06.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:06.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:06.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.523231+0000", 2023-07-02T14:06:06.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014368580000000001 2023-07-02T14:06:06.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:06.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.524668+0000", 2023-07-02T14:06:06.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.929e-05 2023-07-02T14:06:06.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:06.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.524688+0000", 2023-07-02T14:06:06.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9709999999999999e-06 2023-07-02T14:06:06.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.524698+0000", 2023-07-02T14:06:06.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112815 2023-07-02T14:06:06.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:06.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.524810+0000", 2023-07-02T14:06:06.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7217999999999999e-05 2023-07-02T14:06:06.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:06.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.524838+0000", 2023-07-02T14:06:06.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010267129 2023-07-02T14:06:06.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:06.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.535105+0000", 2023-07-02T14:06:06.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6399000000000001e-05 2023-07-02T14:06:06.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:06.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.535151+0000", 2023-07-02T14:06:06.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01191801 2023-07-02T14:06:06.705 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.705 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.705 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.705 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18694 2.5 e176/170 2:ad0b6442:::rbd_data.125de1449605.0000000000000192:head v 176'2317, mlcod=176'2314)", 2023-07-02T14:06:06.705 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.529519+0000", 2023-07-02T14:06:06.706 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.1030009860000001, 2023-07-02T14:06:06.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011113384, 2023-07-02T14:06:06.706 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:06.706 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.706 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:06.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:06.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.529519+0000", 2023-07-02T14:06:06.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:06.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:06.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.529519+0000", 2023-07-02T14:06:06.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:06.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:06.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.529518+0000", 2023-07-02T14:06:06.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033832699999999999 2023-07-02T14:06:06.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:06.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532902+0000", 2023-07-02T14:06:06.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1599e-05 2023-07-02T14:06:06.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:06.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532913+0000", 2023-07-02T14:06:06.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3319999999999999e-06 2023-07-02T14:06:06.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532919+0000", 2023-07-02T14:06:06.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2502000000000001e-05 2023-07-02T14:06:06.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:06.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532961+0000", 2023-07-02T14:06:06.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5031000000000001e-05 2023-07-02T14:06:06.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:06.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532976+0000", 2023-07-02T14:06:06.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076043589999999998 2023-07-02T14:06:06.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:06.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.540580+0000", 2023-07-02T14:06:06.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2391e-05 2023-07-02T14:06:06.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:06.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.540633+0000", 2023-07-02T14:06:06.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011113384 2023-07-02T14:06:06.713 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.713 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.713 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.713 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18695 2.2 e176/171 2:5d95ffc4:::rbd_data.125de1449605.0000000000000193:head v 176'1646, mlcod=176'1641)", 2023-07-02T14:06:06.713 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.529768+0000", 2023-07-02T14:06:06.713 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.102752409, 2023-07-02T14:06:06.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005953687, 2023-07-02T14:06:06.714 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:06.714 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.714 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:06.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:06.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.529768+0000", 2023-07-02T14:06:06.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:06.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:06.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.529768+0000", 2023-07-02T14:06:06.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:06.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:06.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.529768+0000", 2023-07-02T14:06:06.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5627000000000003e-05 2023-07-02T14:06:06.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:06.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.529803+0000", 2023-07-02T14:06:06.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4433999999999999e-05 2023-07-02T14:06:06.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:06.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.529818+0000", 2023-07-02T14:06:06.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1719999999999996e-06 2023-07-02T14:06:06.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.529827+0000", 2023-07-02T14:06:06.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00045031299999999998 2023-07-02T14:06:06.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:06.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.530277+0000", 2023-07-02T14:06:06.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3538000000000003e-05 2023-07-02T14:06:06.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:06.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.530311+0000", 2023-07-02T14:06:06.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0053702919999999996 2023-07-02T14:06:06.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:06.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.535681+0000", 2023-07-02T14:06:06.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0861000000000001e-05 2023-07-02T14:06:06.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:06.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.535722+0000", 2023-07-02T14:06:06.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.005953687 2023-07-02T14:06:06.720 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.720 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.721 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.721 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18697 2.5 e176/170 2:ad0b6442:::rbd_data.125de1449605.0000000000000192:head v 176'2318, mlcod=176'2314)", 2023-07-02T14:06:06.721 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.532930+0000", 2023-07-02T14:06:06.721 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.0995903359999999, 2023-07-02T14:06:06.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079038089999999995, 2023-07-02T14:06:06.722 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:06.722 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.722 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:06.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:06.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532930+0000", 2023-07-02T14:06:06.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:06.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:06.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532930+0000", 2023-07-02T14:06:06.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:06.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:06.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532930+0000", 2023-07-02T14:06:06.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0482000000000001e-05 2023-07-02T14:06:06.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:06.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532960+0000", 2023-07-02T14:06:06.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0919999999999997e-06 2023-07-02T14:06:06.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:06.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532965+0000", 2023-07-02T14:06:06.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.858e-06 2023-07-02T14:06:06.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532968+0000", 2023-07-02T14:06:06.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0040023130000000004 2023-07-02T14:06:06.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:06.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.536970+0000", 2023-07-02T14:06:06.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7188000000000001e-05 2023-07-02T14:06:06.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:06.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.536997+0000", 2023-07-02T14:06:06.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003776616 2023-07-02T14:06:06.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:06.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.540774+0000", 2023-07-02T14:06:06.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9849999999999998e-05 2023-07-02T14:06:06.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:06.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.540834+0000", 2023-07-02T14:06:06.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079038089999999995 2023-07-02T14:06:06.728 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.728 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.728 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.729 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18701 2.2 e176/171 2:5d95ffc4:::rbd_data.125de1449605.0000000000000193:head v 176'1647, mlcod=176'1644)", 2023-07-02T14:06:06.729 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.547419+0000", 2023-07-02T14:06:06.729 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.0851012550000001, 2023-07-02T14:06:06.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055955479999999997, 2023-07-02T14:06:06.729 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:06.729 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.730 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:06.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:06.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.547419+0000", 2023-07-02T14:06:06.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:06.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:06.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.547419+0000", 2023-07-02T14:06:06.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:06.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:06.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.547418+0000", 2023-07-02T14:06:06.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00063843199999999998 2023-07-02T14:06:06.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:06.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.548057+0000", 2023-07-02T14:06:06.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7458000000000001e-05 2023-07-02T14:06:06.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:06.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.548074+0000", 2023-07-02T14:06:06.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2420000000000003e-06 2023-07-02T14:06:06.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.548081+0000", 2023-07-02T14:06:06.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8817999999999999e-05 2023-07-02T14:06:06.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:06.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.548130+0000", 2023-07-02T14:06:06.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3779000000000001e-05 2023-07-02T14:06:06.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:06.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.548154+0000", 2023-07-02T14:06:06.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0048068329999999999 2023-07-02T14:06:06.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:06.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.552961+0000", 2023-07-02T14:06:06.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.401e-05 2023-07-02T14:06:06.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:06.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.553015+0000", 2023-07-02T14:06:06.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055955479999999997 2023-07-02T14:06:06.736 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.736 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.736 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.737 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18713 2.5 e176/170 2:ad0b6442:::rbd_data.125de1449605.0000000000000192:head v 176'2321, mlcod=176'2319)", 2023-07-02T14:06:06.737 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.551020+0000", 2023-07-02T14:06:06.737 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.0815007729999999, 2023-07-02T14:06:06.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068408030000000003, 2023-07-02T14:06:06.737 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:06.737 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.737 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:06.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:06.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.551020+0000", 2023-07-02T14:06:06.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:06.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:06.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.551020+0000", 2023-07-02T14:06:06.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:06.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:06.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.551018+0000", 2023-07-02T14:06:06.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039175080000000001 2023-07-02T14:06:06.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:06.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.554936+0000", 2023-07-02T14:06:06.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1069e-05 2023-07-02T14:06:06.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:06.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.554947+0000", 2023-07-02T14:06:06.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4169999999999997e-06 2023-07-02T14:06:06.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.554953+0000", 2023-07-02T14:06:06.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7589999999999998e-05 2023-07-02T14:06:06.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:06.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.554991+0000", 2023-07-02T14:06:06.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8248999999999998e-05 2023-07-02T14:06:06.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:06.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.555039+0000", 2023-07-02T14:06:06.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027839990000000001 2023-07-02T14:06:06.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:06.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.557823+0000", 2023-07-02T14:06:06.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7456999999999997e-05 2023-07-02T14:06:06.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:06.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.557861+0000", 2023-07-02T14:06:06.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068408030000000003 2023-07-02T14:06:06.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.744 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.744 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18723 2.5 e176/170 2:a564ac9c:::rbd_data.125de1449605.0000000000000196:head v 176'2324, mlcod=176'2321)", 2023-07-02T14:06:06.745 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.568716+0000", 2023-07-02T14:06:06.745 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.063804177, 2023-07-02T14:06:06.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013584191000000001, 2023-07-02T14:06:06.745 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:06.745 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.745 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:06.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:06.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.568716+0000", 2023-07-02T14:06:06.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:06.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:06.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.568716+0000", 2023-07-02T14:06:06.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:06.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:06.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.568715+0000", 2023-07-02T14:06:06.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039052919999999999 2023-07-02T14:06:06.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:06.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.572620+0000", 2023-07-02T14:06:06.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5108000000000001e-05 2023-07-02T14:06:06.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:06.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.572645+0000", 2023-07-02T14:06:06.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0495e-05 2023-07-02T14:06:06.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.572656+0000", 2023-07-02T14:06:06.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6385999999999999e-05 2023-07-02T14:06:06.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:06.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.572702+0000", 2023-07-02T14:06:06.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8578e-05 2023-07-02T14:06:06.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:06.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.572731+0000", 2023-07-02T14:06:06.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095207799999999995 2023-07-02T14:06:06.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:06.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.582251+0000", 2023-07-02T14:06:06.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9021999999999999e-05 2023-07-02T14:06:06.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:06.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.582300+0000", 2023-07-02T14:06:06.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013584191000000001 2023-07-02T14:06:06.752 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.752 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.752 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.752 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18730 2.2 e176/171 2:5828630b:::rbd_data.125de1449605.000000000000019a:head v 176'1651, mlcod=176'1649)", 2023-07-02T14:06:06.752 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.602615+0000", 2023-07-02T14:06:06.753 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.0299055130000001, 2023-07-02T14:06:06.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079390020000000006, 2023-07-02T14:06:06.753 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:06.753 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.753 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:06.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:06.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.602615+0000", 2023-07-02T14:06:06.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:06.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:06.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.602615+0000", 2023-07-02T14:06:06.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:06.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:06.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.602613+0000", 2023-07-02T14:06:06.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00082471499999999995 2023-07-02T14:06:06.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:06.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.603438+0000", 2023-07-02T14:06:06.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2739999999999999e-05 2023-07-02T14:06:06.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:06.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.603461+0000", 2023-07-02T14:06:06.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0336999999999999e-05 2023-07-02T14:06:06.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.603471+0000", 2023-07-02T14:06:06.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6564999999999998e-05 2023-07-02T14:06:06.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:06.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.603527+0000", 2023-07-02T14:06:06.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7444e-05 2023-07-02T14:06:06.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:06.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.603555+0000", 2023-07-02T14:06:06.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0069596850000000002 2023-07-02T14:06:06.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:06.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.610515+0000", 2023-07-02T14:06:06.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9409999999999997e-05 2023-07-02T14:06:06.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:06.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.610554+0000", 2023-07-02T14:06:06.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0079390020000000006 2023-07-02T14:06:06.759 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.760 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.760 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.760 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18739 2.2 e176/171 2:57b58924:::rbd_data.125de1449605.0000000000000199:head v 176'1655, mlcod=176'1653)", 2023-07-02T14:06:06.760 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.625088+0000", 2023-07-02T14:06:06.760 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 1.007432286, 2023-07-02T14:06:06.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00970676, 2023-07-02T14:06:06.761 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:06.761 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.761 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:06.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:06.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.625088+0000", 2023-07-02T14:06:06.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:06.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:06.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.625088+0000", 2023-07-02T14:06:06.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:06.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:06.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.625087+0000", 2023-07-02T14:06:06.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0013589730000000001 2023-07-02T14:06:06.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:06.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.626445+0000", 2023-07-02T14:06:06.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7889e-05 2023-07-02T14:06:06.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:06.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.626473+0000", 2023-07-02T14:06:06.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1839999999999999e-06 2023-07-02T14:06:06.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.626482+0000", 2023-07-02T14:06:06.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7895999999999997e-05 2023-07-02T14:06:06.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:06.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.626529+0000", 2023-07-02T14:06:06.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7637999999999999e-05 2023-07-02T14:06:06.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:06.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.626557+0000", 2023-07-02T14:06:06.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082062690000000004 2023-07-02T14:06:06.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:06.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.634763+0000", 2023-07-02T14:06:06.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1597000000000003e-05 2023-07-02T14:06:06.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:06.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.634795+0000", 2023-07-02T14:06:06.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00970676 2023-07-02T14:06:06.767 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.767 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.768 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18819 2.5 e177/170 2:bb7db89c:::rbd_data.125de1449605.00000000000001a5:head v 177'2333, mlcod=177'2331)", 2023-07-02T14:06:06.768 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.492715+0000", 2023-07-02T14:06:06.768 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.13980503799999999, 2023-07-02T14:06:06.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0094826010000000002, 2023-07-02T14:06:06.768 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:06.769 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.769 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:06.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:06.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.492715+0000", 2023-07-02T14:06:06.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:06.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:06.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.492715+0000", 2023-07-02T14:06:06.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:06.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:06.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.492715+0000", 2023-07-02T14:06:06.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019720390000000001 2023-07-02T14:06:06.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:06.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.494687+0000", 2023-07-02T14:06:06.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3229e-05 2023-07-02T14:06:06.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:06.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.494700+0000", 2023-07-02T14:06:06.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0730000000000004e-06 2023-07-02T14:06:06.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.494705+0000", 2023-07-02T14:06:06.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8112999999999998e-05 2023-07-02T14:06:06.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:06.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.494753+0000", 2023-07-02T14:06:06.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4848999999999999e-05 2023-07-02T14:06:06.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:06.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.494778+0000", 2023-07-02T14:06:06.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007384013 2023-07-02T14:06:06.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:06.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.502162+0000", 2023-07-02T14:06:06.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6047999999999998e-05 2023-07-02T14:06:06.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:06.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.502198+0000", 2023-07-02T14:06:06.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0094826010000000002 2023-07-02T14:06:06.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.775 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.776 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18823 2.5 e177/170 2:ad4691f5:::rbd_data.125de1449605.00000000000001a4:head v 177'2335, mlcod=177'2331)", 2023-07-02T14:06:06.776 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.494855+0000", 2023-07-02T14:06:06.776 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.137665553, 2023-07-02T14:06:06.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073688850000000004, 2023-07-02T14:06:06.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:06.776 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.777 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:06.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:06.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.494855+0000", 2023-07-02T14:06:06.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:06.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:06.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.494855+0000", 2023-07-02T14:06:06.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:06.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:06.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.494854+0000", 2023-07-02T14:06:06.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011165799999999999 2023-07-02T14:06:06.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:06.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.494966+0000", 2023-07-02T14:06:06.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5960000000000004e-06 2023-07-02T14:06:06.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:06.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.494975+0000", 2023-07-02T14:06:06.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.546e-06 2023-07-02T14:06:06.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.494978+0000", 2023-07-02T14:06:06.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002287884 2023-07-02T14:06:06.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:06.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.497266+0000", 2023-07-02T14:06:06.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3606999999999999e-05 2023-07-02T14:06:06.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:06.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.497290+0000", 2023-07-02T14:06:06.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049140829999999996 2023-07-02T14:06:06.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:06.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.502204+0000", 2023-07-02T14:06:06.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0068000000000002e-05 2023-07-02T14:06:06.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:06.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.502224+0000", 2023-07-02T14:06:06.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073688850000000004 2023-07-02T14:06:06.783 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.783 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.783 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.784 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18818 2.4 e177/169 2:21299eda:::rbd_data.125de1449605.00000000000001a6:head v 177'1707, mlcod=176'1705)", 2023-07-02T14:06:06.784 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.496945+0000", 2023-07-02T14:06:06.784 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.13557502199999999, 2023-07-02T14:06:06.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006225703, 2023-07-02T14:06:06.784 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:06.784 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:06.784 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:06.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:06.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.496945+0000", 2023-07-02T14:06:06.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:06.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:06.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.496945+0000", 2023-07-02T14:06:06.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:06.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:06.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.496944+0000", 2023-07-02T14:06:06.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000946488 2023-07-02T14:06:06.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.497890+0000", 2023-07-02T14:06:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2551000000000001e-05 2023-07-02T14:06:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.497903+0000", 2023-07-02T14:06:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6699999999999997e-06 2023-07-02T14:06:06.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:06.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.497909+0000", 2023-07-02T14:06:06.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6807000000000001e-05 2023-07-02T14:06:06.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:06.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.497956+0000", 2023-07-02T14:06:06.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3496999999999999e-05 2023-07-02T14:06:06.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:06.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.497980+0000", 2023-07-02T14:06:06.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051539059999999998 2023-07-02T14:06:06.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:06.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.503134+0000", 2023-07-02T14:06:06.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7586999999999998e-05 2023-07-02T14:06:06.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:06.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:06.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:06.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.503171+0000", 2023-07-02T14:06:06.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.006225703 2023-07-02T14:06:06.791 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.791 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.791 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.791 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:06.791 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:06.791 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:06.793 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:06:06.793 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:06.792+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:06.829 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 2] out_osds: [5, 4] dead_osds: [] live_osds: [1, 2, 4, 5, 3, 0] 2023-07-02T14:06:06.830 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:06:06.830 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2023-07-02T14:06:06.830 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-07-02T14:06:06.830 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-07-02T14:06:06.831 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2023-07-02T14:06:06.894 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:06.893+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:06.994 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:06.993+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:06.995 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:06.996 INFO:teuthology.orchestra.run.smithi022.stdout: "success": "" 2023-07-02T14:06:06.996 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:07.020 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:07.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18943 2.1 e177/104 2:9a51482a:::rbd_data.125de1449605.0000000000000035:head v 177'2120, mlcod=177'2117)", 2023-07-02T14:06:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.008131+0000", 2023-07-02T14:06:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.011902595, 2023-07-02T14:06:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011922746, 2023-07-02T14:06:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:07.021 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.008131+0000", 2023-07-02T14:06:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.008131+0000", 2023-07-02T14:06:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.008131+0000", 2023-07-02T14:06:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5948999999999998e-05 2023-07-02T14:06:07.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.008157+0000", 2023-07-02T14:06:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7283999999999999e-05 2023-07-02T14:06:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.008174+0000", 2023-07-02T14:06:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3899999999999998e-06 2023-07-02T14:06:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.008180+0000", 2023-07-02T14:06:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0723000000000001e-05 2023-07-02T14:06:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.008241+0000", 2023-07-02T14:06:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8046e-05 2023-07-02T14:06:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.008259+0000", 2023-07-02T14:06:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000127603 2023-07-02T14:06:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:18945 2.4 2:3e650bab:::rbd_data.125de1449605.0000000000000031:head [write 4063232~131072 in=131072b] snapc 34=[34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e177)", 2023-07-02T14:06:07.027 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.009216+0000", 2023-07-02T14:06:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.010817771, 2023-07-02T14:06:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010891214999999999, 2023-07-02T14:06:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "waiting for sub ops", 2023-07-02T14:06:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:07.028 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18945 2023-07-02T14:06:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.009216+0000", 2023-07-02T14:06:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.009216+0000", 2023-07-02T14:06:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.030 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.009215+0000", 2023-07-02T14:06:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8205000000000003e-05 2023-07-02T14:06:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.009273+0000", 2023-07-02T14:06:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6005999999999999e-05 2023-07-02T14:06:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.009289+0000", 2023-07-02T14:06:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.435e-06 2023-07-02T14:06:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.009296+0000", 2023-07-02T14:06:07.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9953000000000002e-05 2023-07-02T14:06:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.009336+0000", 2023-07-02T14:06:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000114485 2023-07-02T14:06:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.009451+0000", 2023-07-02T14:06:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00019888600000000001 2023-07-02T14:06:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.009650+0000", 2023-07-02T14:06:07.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0060073929999999998 2023-07-02T14:06:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.015657+0000", 2023-07-02T14:06:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064407609999999997 2023-07-02T14:06:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.035 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18942 2.3 e177/105 2:d4be914f:::rbd_data.125de1449605.0000000000000033:head v 177'2176, mlcod=177'2171)", 2023-07-02T14:06:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.012068+0000", 2023-07-02T14:06:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0079658290000000007, 2023-07-02T14:06:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0080938120000000006, 2023-07-02T14:06:07.036 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.012068+0000", 2023-07-02T14:06:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.012068+0000", 2023-07-02T14:06:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.012067+0000", 2023-07-02T14:06:07.038 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5601000000000001e-05 2023-07-02T14:06:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.012092+0000", 2023-07-02T14:06:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5747999999999999e-05 2023-07-02T14:06:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.012108+0000", 2023-07-02T14:06:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9750000000000007e-06 2023-07-02T14:06:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.012116+0000", 2023-07-02T14:06:07.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0027836950000000001 2023-07-02T14:06:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.014900+0000", 2023-07-02T14:06:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7606999999999998e-05 2023-07-02T14:06:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.014927+0000", 2023-07-02T14:06:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0028592880000000002 2023-07-02T14:06:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.042 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18944 2.1 e177/104 2:9a51482a:::rbd_data.125de1449605.0000000000000035:head v 177'2121, mlcod=177'2117)", 2023-07-02T14:06:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.015202+0000", 2023-07-02T14:06:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0048316410000000002, 2023-07-02T14:06:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0049976860000000003, 2023-07-02T14:06:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.015202+0000", 2023-07-02T14:06:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.015202+0000", 2023-07-02T14:06:07.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.015201+0000", 2023-07-02T14:06:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011725279999999999 2023-07-02T14:06:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.016374+0000", 2023-07-02T14:06:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3287e-05 2023-07-02T14:06:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.016387+0000", 2023-07-02T14:06:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.046e-06 2023-07-02T14:06:07.046 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.016394+0000", 2023-07-02T14:06:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2598000000000001e-05 2023-07-02T14:06:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.047 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.016436+0000", 2023-07-02T14:06:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6011e-05 2023-07-02T14:06:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.016452+0000", 2023-07-02T14:06:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001250199 2023-07-02T14:06:07.048 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:18946 2.4 2:2f8e3c69:::rbd_data.125de1449605.0000000000000028:head [write 0~65536 in=65536b] snapc 34=[34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e177)", 2023-07-02T14:06:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.018690+0000", 2023-07-02T14:06:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0013440419999999999, 2023-07-02T14:06:07.049 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0015534170000000001, 2023-07-02T14:06:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "waiting for sub ops", 2023-07-02T14:06:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18946 2023-07-02T14:06:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.050 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.018690+0000", 2023-07-02T14:06:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.051 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.018690+0000", 2023-07-02T14:06:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:06:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.018688+0000", 2023-07-02T14:06:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6716999999999999e-05 2023-07-02T14:06:07.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.018744+0000", 2023-07-02T14:06:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9131999999999999e-05 2023-07-02T14:06:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.053 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.018763+0000", 2023-07-02T14:06:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3549999999999994e-06 2023-07-02T14:06:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.018773+0000", 2023-07-02T14:06:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3786000000000003e-05 2023-07-02T14:06:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.018827+0000", 2023-07-02T14:06:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000117645 2023-07-02T14:06:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.018944+0000", 2023-07-02T14:06:07.055 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028888999999999998 2023-07-02T14:06:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.019233+0000", 2023-07-02T14:06:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00054317199999999997 2023-07-02T14:06:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.056 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18924 2.0 e177/144 2:1c72c23d:::rbd_object_map.125de1449605:head v 177'4658, mlcod=177'4656)", 2023-07-02T14:06:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.019209+0000", 2023-07-02T14:06:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.00082512099999999995, 2023-07-02T14:06:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001079644, 2023-07-02T14:06:07.057 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.019209+0000", 2023-07-02T14:06:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.058 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.019209+0000", 2023-07-02T14:06:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.019208+0000", 2023-07-02T14:06:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6629999999999999e-06 2023-07-02T14:06:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.019216+0000", 2023-07-02T14:06:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4209000000000001e-05 2023-07-02T14:06:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.019231+0000", 2023-07-02T14:06:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0170000000000002e-06 2023-07-02T14:06:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.019237+0000", 2023-07-02T14:06:07.061 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8553000000000001e-05 2023-07-02T14:06:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.019285+0000", 2023-07-02T14:06:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7693999999999999e-05 2023-07-02T14:06:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.062 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.019313+0000", 2023-07-02T14:06:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000104106 2023-07-02T14:06:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:06:07.063 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 6 2023-07-02T14:06:07.064 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:07.064 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:06:07.095 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:07.094+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:07.199 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:07.195+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:07.296 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:07.297 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:07.297 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:06:07.297 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:06:07.297 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:07.298 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:07.296+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:07.314 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:06:07.317 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:06:07.400 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:07.399+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:07.498 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:07.497+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:07.547 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:07.572 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:07.572 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:06:07.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:06:07.573 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:07.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.573 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:06:07.573 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:07.573 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 178.117338574, 2023-07-02T14:06:07.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:06:07.573 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.574 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:07.574 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.574 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:07.574 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:07.574 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:06:07.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.574 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:07.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:07.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:07.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:06:07.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:06:07.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:06:07.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:06:07.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:06:07.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:06:07.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:06:07.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:06:07.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:06:07.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:06:07.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:07.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:06:07.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:06:07.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:06:07.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:06:07.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:07.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:06:07.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:06:07.582 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.582 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.582 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.582 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:06:07.582 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:07.583 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 177.24664556600001, 2023-07-02T14:06:07.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:06:07.583 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.583 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:07.583 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.583 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:06:07.583 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:06:07.584 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:06:07.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.584 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:07.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:06:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:06:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:06:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:06:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:06:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:06:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:06:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:06:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:06:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:06:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:06:07.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:06:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:06:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:06:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:06:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:06:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:07.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:06:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:06:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.592 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:07.593 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:07.593 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 177.24652084499999, 2023-07-02T14:06:07.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:06:07.593 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.593 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:07.593 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.593 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:06:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:07.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:06:07.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:06:07.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:06:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:06:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:06:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:06:07.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:06:07.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:06:07.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:06:07.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:06:07.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:07.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:06:07.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:06:07.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:06:07.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:06:07.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:06:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:06:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.602 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:07.602 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:07.602 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 177.239670961, 2023-07-02T14:06:07.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:06:07.602 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.602 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:07.603 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.603 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:07.603 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:07.603 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:06:07.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.603 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:07.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:07.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:06:07.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:06:07.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:06:07.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:06:07.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:06:07.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:06:07.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:06:07.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:06:07.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:06:07.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:06:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:06:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:06:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:06:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:06:07.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:06:07.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:07.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:06:07.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:06:07.610 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.610 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.611 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.611 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:07.611 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:07.611 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 177.23960803599999, 2023-07-02T14:06:07.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:06:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:06:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.612 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:07.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:07.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:06:07.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:06:07.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:06:07.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:06:07.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:06:07.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:06:07.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:06:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:06:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:06:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:06:07.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:06:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:06:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:06:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:06:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:06:07.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:06:07.620 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.620 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.620 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.620 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:07.620 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 177.239307615, 2023-07-02T14:06:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:06:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:07.621 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:06:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:06:07.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:06:07.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:06:07.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:06:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:06:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:06:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:06:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:06:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:06:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:06:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:06:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:06:07.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:06:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:06:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:06:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:06:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:06:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:06:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:06:07.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:06:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:06:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:06:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:06:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:06:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:06:07.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:07.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:06:07.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:06:07.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:07.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:06:07.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:06:07.633 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.633 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.633 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:06:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 177.23878311499999, 2023-07-02T14:06:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:06:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.634 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:06:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:07.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:07.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:06:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:06:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:06:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:06:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:06:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:06:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:06:07.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:06:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:06:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:07.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:06:07.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:06:07.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:06:07.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:06:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:06:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:06:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:06:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:06:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:06:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 150.11476399, 2023-07-02T14:06:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:06:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:07.644 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:06:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:07.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:07.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:06:07.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:06:07.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:06:07.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:06:07.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:06:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:06:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:06:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:06:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:06:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:06:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:06:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:06:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:06:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:06:07.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:07.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:06:07.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:06:07.652 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.652 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.652 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.653 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:06:07.653 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:07.653 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 149.789281286, 2023-07-02T14:06:07.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:06:07.653 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:06:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.654 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:06:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:06:07.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:06:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:06:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:06:07.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:06:07.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:06:07.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:06:07.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:06:07.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:06:07.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:06:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:06:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:06:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:06:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:06:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:06:07.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:06:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:06:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:07.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:06:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:06:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:06:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:06:07.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:07.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:06:07.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:06:07.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:07.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:06:07.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:06:07.665 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.665 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.665 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:17 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 0~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.371005504000003, 2023-07-02T14:06:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997, 2023-07-02T14:06:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:07.666 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17 2023-07-02T14:06:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:07.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:07.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:06:07.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198431+0000", 2023-07-02T14:06:07.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2815e-05 2023-07-02T14:06:07.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198443+0000", 2023-07-02T14:06:07.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2569999999999999e-05 2023-07-02T14:06:07.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198466+0000", 2023-07-02T14:06:07.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0223e-05 2023-07-02T14:06:07.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198476+0000", 2023-07-02T14:06:07.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1792999999999997e-05 2023-07-02T14:06:07.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:06:07.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4249999999999998e-06 2023-07-02T14:06:07.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:07.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198536+0000", 2023-07-02T14:06:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91888663500000001 2023-07-02T14:06:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117423+0000", 2023-07-02T14:06:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223594 2023-07-02T14:06:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117647+0000", 2023-07-02T14:06:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033832900000000001 2023-07-02T14:06:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:07.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117985+0000", 2023-07-02T14:06:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997 2023-07-02T14:06:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:18 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 65536~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:07.676 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:07.676 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.370948292999998, 2023-07-02T14:06:07.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998, 2023-07-02T14:06:07.676 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.676 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:07.676 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.676 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:07.677 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:07.677 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18 2023-07-02T14:06:07.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.677 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:07.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:07.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:07.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7600000000000002e-06 2023-07-02T14:06:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198494+0000", 2023-07-02T14:06:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7840000000000004e-06 2023-07-02T14:06:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198502+0000", 2023-07-02T14:06:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6459999999999998e-06 2023-07-02T14:06:07.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198508+0000", 2023-07-02T14:06:07.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7395999999999997e-05 2023-07-02T14:06:07.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198565+0000", 2023-07-02T14:06:07.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6800000000000004e-06 2023-07-02T14:06:07.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:07.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198572+0000", 2023-07-02T14:06:07.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91965896800000002 2023-07-02T14:06:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118231+0000", 2023-07-02T14:06:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5212999999999998e-05 2023-07-02T14:06:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118326+0000", 2023-07-02T14:06:07.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000238538 2023-07-02T14:06:07.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:07.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118565+0000", 2023-07-02T14:06:07.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998 2023-07-02T14:06:07.685 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.685 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.685 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.685 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:19 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 131072~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.370924068999997, 2023-07-02T14:06:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002, 2023-07-02T14:06:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:07.686 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 19 2023-07-02T14:06:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:07.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1990000000000002e-06 2023-07-02T14:06:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198517+0000", 2023-07-02T14:06:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8789999999999997e-06 2023-07-02T14:06:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198524+0000", 2023-07-02T14:06:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:06:07.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:06:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5545999999999999e-05 2023-07-02T14:06:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2540000000000003e-06 2023-07-02T14:06:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198598+0000", 2023-07-02T14:06:07.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.919983263 2023-07-02T14:06:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118581+0000", 2023-07-02T14:06:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6878999999999994e-05 2023-07-02T14:06:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118648+0000", 2023-07-02T14:06:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149732 2023-07-02T14:06:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118798+0000", 2023-07-02T14:06:07.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002 2023-07-02T14:06:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:20 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 262144~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:07.695 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:07.696 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.370905104999999, 2023-07-02T14:06:07.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998, 2023-07-02T14:06:07.696 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.696 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:07.696 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.696 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:07.696 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:07.697 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20 2023-07-02T14:06:07.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.697 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:07.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:07.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.732e-06 2023-07-02T14:06:07.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198537+0000", 2023-07-02T14:06:07.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5259999999999999e-06 2023-07-02T14:06:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198543+0000", 2023-07-02T14:06:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.121e-06 2023-07-02T14:06:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198547+0000", 2023-07-02T14:06:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9591000000000005e-05 2023-07-02T14:06:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:06:07.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3120000000000004e-06 2023-07-02T14:06:07.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:07.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198621+0000", 2023-07-02T14:06:07.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92020789400000003 2023-07-02T14:06:07.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118829+0000", 2023-07-02T14:06:07.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022393386000000001 2023-07-02T14:06:07.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141223+0000", 2023-07-02T14:06:07.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000329866 2023-07-02T14:06:07.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:07.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141552+0000", 2023-07-02T14:06:07.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998 2023-07-02T14:06:07.704 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.705 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.705 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.705 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:21 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 196608~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:07.705 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:07.705 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.370877470000003, 2023-07-02T14:06:07.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995, 2023-07-02T14:06:07.706 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.706 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:07.706 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.706 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:07.706 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:07.706 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21 2023-07-02T14:06:07.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.707 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:07.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:07.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:07.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:07.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0169999999999999e-05 2023-07-02T14:06:07.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198571+0000", 2023-07-02T14:06:07.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3479999999999998e-06 2023-07-02T14:06:07.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198578+0000", 2023-07-02T14:06:07.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8439999999999997e-06 2023-07-02T14:06:07.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198583+0000", 2023-07-02T14:06:07.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8294999999999997e-05 2023-07-02T14:06:07.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198641+0000", 2023-07-02T14:06:07.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8120000000000001e-06 2023-07-02T14:06:07.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:07.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:06:07.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94293828499999999 2023-07-02T14:06:07.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141583+0000", 2023-07-02T14:06:07.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014607300000000001 2023-07-02T14:06:07.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141729+0000", 2023-07-02T14:06:07.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00044906600000000003 2023-07-02T14:06:07.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:07.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142178+0000", 2023-07-02T14:06:07.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995 2023-07-02T14:06:07.715 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.715 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.715 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.715 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:22 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 327680~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:07.715 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:07.715 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.370844040000001, 2023-07-02T14:06:07.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004, 2023-07-02T14:06:07.716 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.716 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:07.716 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.716 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:07.716 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:07.717 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 22 2023-07-02T14:06:07.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.717 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:07.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:07.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:07.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:07.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4850000000000001e-06 2023-07-02T14:06:07.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198597+0000", 2023-07-02T14:06:07.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:06:07.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198605+0000", 2023-07-02T14:06:07.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4839999999999998e-06 2023-07-02T14:06:07.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198609+0000", 2023-07-02T14:06:07.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3100999999999999e-05 2023-07-02T14:06:07.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198662+0000", 2023-07-02T14:06:07.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9099999999999998e-06 2023-07-02T14:06:07.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:07.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198666+0000", 2023-07-02T14:06:07.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.943539503 2023-07-02T14:06:07.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142206+0000", 2023-07-02T14:06:07.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012255999999999999 2023-07-02T14:06:07.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142328+0000", 2023-07-02T14:06:07.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208956 2023-07-02T14:06:07.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:07.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142537+0000", 2023-07-02T14:06:07.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004 2023-07-02T14:06:07.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.725 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.725 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.725 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:23 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 393216~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:07.725 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:07.725 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.370824059999997, 2023-07-02T14:06:07.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348, 2023-07-02T14:06:07.726 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.726 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:07.726 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.726 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:07.726 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:07.726 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23 2023-07-02T14:06:07.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.727 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:07.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:07.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:07.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:07.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8439999999999998e-06 2023-07-02T14:06:07.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:06:07.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0440000000000001e-06 2023-07-02T14:06:07.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198623+0000", 2023-07-02T14:06:07.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4480000000000002e-06 2023-07-02T14:06:07.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198628+0000", 2023-07-02T14:06:07.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5843e-05 2023-07-02T14:06:07.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:07.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.89e-06 2023-07-02T14:06:07.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:07.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198688+0000", 2023-07-02T14:06:07.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94389171999999999 2023-07-02T14:06:07.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142580+0000", 2023-07-02T14:06:07.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3892000000000004e-05 2023-07-02T14:06:07.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142644+0000", 2023-07-02T14:06:07.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017608000000000001 2023-07-02T14:06:07.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:07.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142820+0000", 2023-07-02T14:06:07.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348 2023-07-02T14:06:07.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.735 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.735 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.735 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:24 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 458752~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:07.735 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:07.735 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.370795985999997, 2023-07-02T14:06:07.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004, 2023-07-02T14:06:07.736 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.736 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:07.736 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.736 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:07.736 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:07.736 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 24 2023-07-02T14:06:07.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.737 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:07.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:07.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:07.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198642+0000", 2023-07-02T14:06:07.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.253e-06 2023-07-02T14:06:07.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:06:07.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1289999999999998e-06 2023-07-02T14:06:07.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198652+0000", 2023-07-02T14:06:07.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4510000000000002e-06 2023-07-02T14:06:07.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198657+0000", 2023-07-02T14:06:07.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9131000000000001e-05 2023-07-02T14:06:07.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198706+0000", 2023-07-02T14:06:07.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8890000000000004e-06 2023-07-02T14:06:07.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:07.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198710+0000", 2023-07-02T14:06:07.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94415412200000004 2023-07-02T14:06:07.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142864+0000", 2023-07-02T14:06:07.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8047000000000006e-05 2023-07-02T14:06:07.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142952+0000", 2023-07-02T14:06:07.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192967 2023-07-02T14:06:07.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:07.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143145+0000", 2023-07-02T14:06:07.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004 2023-07-02T14:06:07.744 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.744 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.745 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.745 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:25 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 524288~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:07.745 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:07.745 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.370777191000002, 2023-07-02T14:06:07.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001, 2023-07-02T14:06:07.746 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.746 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:07.746 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.746 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:07.746 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:07.746 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25 2023-07-02T14:06:07.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.747 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:07.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:07.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:07.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:07.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.615e-06 2023-07-02T14:06:07.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198664+0000", 2023-07-02T14:06:07.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4759999999999997e-06 2023-07-02T14:06:07.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198669+0000", 2023-07-02T14:06:07.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5999999999999998e-06 2023-07-02T14:06:07.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198673+0000", 2023-07-02T14:06:07.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4256999999999999e-05 2023-07-02T14:06:07.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198727+0000", 2023-07-02T14:06:07.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840000000000001e-06 2023-07-02T14:06:07.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:07.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198731+0000", 2023-07-02T14:06:07.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94444241900000003 2023-07-02T14:06:07.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143173+0000", 2023-07-02T14:06:07.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6839000000000002e-05 2023-07-02T14:06:07.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143240+0000", 2023-07-02T14:06:07.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017053399999999999 2023-07-02T14:06:07.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:07.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143411+0000", 2023-07-02T14:06:07.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001 2023-07-02T14:06:07.754 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.754 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.755 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.755 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:26 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 589824~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:07.755 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:07.755 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.370754040000001, 2023-07-02T14:06:07.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997, 2023-07-02T14:06:07.756 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.756 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:07.756 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.756 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:07.756 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:07.756 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 26 2023-07-02T14:06:07.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.756 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:07.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:07.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:07.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:07.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0520000000000002e-06 2023-07-02T14:06:07.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198687+0000", 2023-07-02T14:06:07.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1880000000000002e-06 2023-07-02T14:06:07.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198695+0000", 2023-07-02T14:06:07.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9960000000000004e-06 2023-07-02T14:06:07.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198699+0000", 2023-07-02T14:06:07.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2330999999999998e-05 2023-07-02T14:06:07.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198742+0000", 2023-07-02T14:06:07.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7950000000000001e-06 2023-07-02T14:06:07.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:07.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198745+0000", 2023-07-02T14:06:07.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94469398199999999 2023-07-02T14:06:07.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143439+0000", 2023-07-02T14:06:07.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6048e-05 2023-07-02T14:06:07.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143525+0000", 2023-07-02T14:06:07.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017016700000000001 2023-07-02T14:06:07.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:07.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143696+0000", 2023-07-02T14:06:07.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997 2023-07-02T14:06:07.764 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.764 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.765 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.765 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.6579.0:11 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected e171)", 2023-07-02T14:06:07.765 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:07.765 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 10.410955553999999, 2023-07-02T14:06:07.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998, 2023-07-02T14:06:07.766 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:07.766 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:07.766 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:07.766 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.6579", 2023-07-02T14:06:07.766 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/554403467", 2023-07-02T14:06:07.766 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11 2023-07-02T14:06:07.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.767 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:07.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:07.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:07.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:07.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:07.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:07.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:07.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:07.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158481+0000", 2023-07-02T14:06:07.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:06:07.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:07.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158487+0000", 2023-07-02T14:06:07.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7630000000000006e-06 2023-07-02T14:06:07.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:07.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158496+0000", 2023-07-02T14:06:07.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5100000000000001e-06 2023-07-02T14:06:07.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:07.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158504+0000", 2023-07-02T14:06:07.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9397000000000002e-05 2023-07-02T14:06:07.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158553+0000", 2023-07-02T14:06:07.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5810000000000003e-06 2023-07-02T14:06:07.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:07.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158562+0000", 2023-07-02T14:06:07.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93893079499999998 2023-07-02T14:06:07.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:07.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097493+0000", 2023-07-02T14:06:07.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015799299999999999 2023-07-02T14:06:07.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:07.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097651+0000", 2023-07-02T14:06:07.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187746 2023-07-02T14:06:07.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:07.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:07.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:07.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097839+0000", 2023-07-02T14:06:07.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998 2023-07-02T14:06:07.774 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.774 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.774 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:07.775 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:07.775 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:07.776 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:07.777 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:06:07.777 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:07.778 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:07.778 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:07.778 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:07.779 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:06:07.779 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:06:07.779 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:06:07.779 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:06:07.779 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:06:07.779 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:06:07.780 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:07.775+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:07.877 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:07.876+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:07.978 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:07.977+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:08.017 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:08.017 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:08.037 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:06:08.079 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:08.078+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:08.179 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:08.179+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:08.256 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:08.256 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:08.269 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:06:08.312 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:08.311+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:08.381 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:08.380+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:08.466 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:06:08.471 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:08.472 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:08.483 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:08.481+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:08.495 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:06:08.583 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:08.582+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:08.674 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:08.674 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:08.684 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:06:08.684 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:08.683+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:08.785 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:08.784+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:08.871 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:08.871 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:08.882 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:06:08.924 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:08.924+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:08.988 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:08.985+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:09.038 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:09.038 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:09.052 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:06:09.093 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:09.092+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:09.186 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:09.185+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:09.218 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:09.218 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:09.228 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:06:09.287 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:09.287+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:09.356 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:09.356 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:09.367 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:06:09.408 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:09.407+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:06:09.489 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:09.487+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:09.505 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap48 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap49 b3fbb028-e267-4d25-a750-93ba18a2316a@snap49.diff 2023-07-02T14:06:09.557 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:09.557 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:09.572 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:06:09.576 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:09.576+0000 7fec4ea75700 0 -- 172.21.15.163:0/1948225002 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x56112c87acc0 msgr2=0x56112c87d110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:09.582 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:09.582+0000 7fec4f276700 0 -- 172.21.15.163:0/1948225002 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7fec34004630 msgr2=0x7fec34004a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:09.589 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:09.588+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:09.690 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:09.689+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:09.744 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:09.744 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:09.755 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:06:09.780 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:06:09.796 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:09.795+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:09.853 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete... Exporting image: 50% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 100% complete...done. 2023-07-02T14:06:09.860 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:06:09.860 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=50 2023-07-02T14:06:09.861 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap50 2023-07-02T14:06:09.892 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:09.891+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:09.940 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:09.940 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:09.946 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:09.950 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:06:09.963 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:09.980 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:09.992 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:09.991+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:09.998 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:10.016 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:10.033 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:10.034 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:06:10.034 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:06:10.034 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:06:10.035 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:06:10.035 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:06:10.035 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:06:10.093 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:10.092+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:10.119 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:10.119 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:10.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:10.119 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:10.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.120 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:16941 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:10.120 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:06:10.120 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 27.25823407, 2023-07-02T14:06:10.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11449124500000001, 2023-07-02T14:06:10.120 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:10.120 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:10.121 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:10.121 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:10.121 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:10.121 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 16941 2023-07-02T14:06:10.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.121 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:10.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:10.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:06:10.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:10.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:10.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:06:10.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:06:10.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:10.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855877+0000", 2023-07-02T14:06:10.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3209999999999999e-06 2023-07-02T14:06:10.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:10.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855880+0000", 2023-07-02T14:06:10.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9299999999999999e-06 2023-07-02T14:06:10.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:10.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855883+0000", 2023-07-02T14:06:10.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6309999999999999e-06 2023-07-02T14:06:10.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:10.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855885+0000", 2023-07-02T14:06:10.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050516850000000002 2023-07-02T14:06:10.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.860937+0000", 2023-07-02T14:06:10.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1844999999999997e-05 2023-07-02T14:06:10.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.860979+0000", 2023-07-02T14:06:10.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055853480000000004 2023-07-02T14:06:10.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.866564+0000", 2023-07-02T14:06:10.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2821999999999998e-05 2023-07-02T14:06:10.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.866597+0000", 2023-07-02T14:06:10.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050938900000000002 2023-07-02T14:06:10.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.871691+0000", 2023-07-02T14:06:10.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1210000000000001e-05 2023-07-02T14:06:10.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.871722+0000", 2023-07-02T14:06:10.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014768687000000001 2023-07-02T14:06:10.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.886491+0000", 2023-07-02T14:06:10.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7248000000000003e-05 2023-07-02T14:06:10.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.886568+0000", 2023-07-02T14:06:10.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015403168 2023-07-02T14:06:10.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.901971+0000", 2023-07-02T14:06:10.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4035000000000002e-05 2023-07-02T14:06:10.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.902005+0000", 2023-07-02T14:06:10.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014618580000000001 2023-07-02T14:06:10.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.916624+0000", 2023-07-02T14:06:10.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.443e-05 2023-07-02T14:06:10.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.916668+0000", 2023-07-02T14:06:10.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099598989999999995 2023-07-02T14:06:10.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.926628+0000", 2023-07-02T14:06:10.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3123999999999999e-05 2023-07-02T14:06:10.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.926711+0000", 2023-07-02T14:06:10.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046031930000000002 2023-07-02T14:06:10.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.931314+0000", 2023-07-02T14:06:10.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3257000000000003e-05 2023-07-02T14:06:10.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.931358+0000", 2023-07-02T14:06:10.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078370969999999995 2023-07-02T14:06:10.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.939195+0000", 2023-07-02T14:06:10.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2522000000000003e-05 2023-07-02T14:06:10.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.939247+0000", 2023-07-02T14:06:10.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01118103 2023-07-02T14:06:10.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.950428+0000", 2023-07-02T14:06:10.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5001999999999996e-05 2023-07-02T14:06:10.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.950523+0000", 2023-07-02T14:06:10.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089648390000000005 2023-07-02T14:06:10.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959488+0000", 2023-07-02T14:06:10.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4569e-05 2023-07-02T14:06:10.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:10.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959533+0000", 2023-07-02T14:06:10.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026104299999999997 2023-07-02T14:06:10.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:10.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959794+0000", 2023-07-02T14:06:10.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025221789999999998 2023-07-02T14:06:10.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:10.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.962316+0000", 2023-07-02T14:06:10.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079080520000000005 2023-07-02T14:06:10.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:10.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970224+0000", 2023-07-02T14:06:10.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1265e-05 2023-07-02T14:06:10.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:10.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970255+0000", 2023-07-02T14:06:10.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000113581 2023-07-02T14:06:10.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:10.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970369+0000", 2023-07-02T14:06:10.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11449124500000001 2023-07-02T14:06:10.142 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.142 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.143 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.143 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17049 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:10.143 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:06:10.143 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 27.134729079, 2023-07-02T14:06:10.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.112277108, 2023-07-02T14:06:10.143 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:10.144 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:10.144 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:10.144 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:10.144 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:10.144 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17049 2023-07-02T14:06:10.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.144 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:10.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:10.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:06:10.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:10.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:10.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:06:10.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:10.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:10.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979382+0000", 2023-07-02T14:06:10.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3450000000000002e-06 2023-07-02T14:06:10.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:10.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979385+0000", 2023-07-02T14:06:10.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9000000000000003e-06 2023-07-02T14:06:10.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:10.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979391+0000", 2023-07-02T14:06:10.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9999999999999998e-06 2023-07-02T14:06:10.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:10.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979395+0000", 2023-07-02T14:06:10.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013089412 2023-07-02T14:06:10.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992485+0000", 2023-07-02T14:06:10.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3828999999999997e-05 2023-07-02T14:06:10.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992519+0000", 2023-07-02T14:06:10.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062549889999999999 2023-07-02T14:06:10.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998774+0000", 2023-07-02T14:06:10.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4870999999999995e-05 2023-07-02T14:06:10.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998848+0000", 2023-07-02T14:06:10.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091715140000000004 2023-07-02T14:06:10.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008020+0000", 2023-07-02T14:06:10.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8659999999999999e-05 2023-07-02T14:06:10.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008059+0000", 2023-07-02T14:06:10.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059565110000000003 2023-07-02T14:06:10.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014015+0000", 2023-07-02T14:06:10.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8088000000000002e-05 2023-07-02T14:06:10.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014093+0000", 2023-07-02T14:06:10.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007513395 2023-07-02T14:06:10.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021607+0000", 2023-07-02T14:06:10.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9777e-05 2023-07-02T14:06:10.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021636+0000", 2023-07-02T14:06:10.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056999549999999996 2023-07-02T14:06:10.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027336+0000", 2023-07-02T14:06:10.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8356999999999996e-05 2023-07-02T14:06:10.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027405+0000", 2023-07-02T14:06:10.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047942130000000003 2023-07-02T14:06:10.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032199+0000", 2023-07-02T14:06:10.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1137999999999999e-05 2023-07-02T14:06:10.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032230+0000", 2023-07-02T14:06:10.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057056759999999998 2023-07-02T14:06:10.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.037936+0000", 2023-07-02T14:06:10.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6334999999999993e-05 2023-07-02T14:06:10.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038012+0000", 2023-07-02T14:06:10.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043985609999999996 2023-07-02T14:06:10.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042411+0000", 2023-07-02T14:06:10.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6857e-05 2023-07-02T14:06:10.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042487+0000", 2023-07-02T14:06:10.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007284446 2023-07-02T14:06:10.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049772+0000", 2023-07-02T14:06:10.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3757000000000006e-05 2023-07-02T14:06:10.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049846+0000", 2023-07-02T14:06:10.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0095186679999999992 2023-07-02T14:06:10.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059364+0000", 2023-07-02T14:06:10.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9931000000000005e-05 2023-07-02T14:06:10.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059434+0000", 2023-07-02T14:06:10.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079904920000000001 2023-07-02T14:06:10.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067425+0000", 2023-07-02T14:06:10.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5583000000000001e-05 2023-07-02T14:06:10.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067460+0000", 2023-07-02T14:06:10.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009114278 2023-07-02T14:06:10.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076575+0000", 2023-07-02T14:06:10.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5880000000000001e-05 2023-07-02T14:06:10.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076620+0000", 2023-07-02T14:06:10.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055118470000000003 2023-07-02T14:06:10.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082132+0000", 2023-07-02T14:06:10.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8306999999999997e-05 2023-07-02T14:06:10.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:10.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082181+0000", 2023-07-02T14:06:10.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026259599999999998 2023-07-02T14:06:10.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:10.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082443+0000", 2023-07-02T14:06:10.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018855339999999999 2023-07-02T14:06:10.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:10.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084329+0000", 2023-07-02T14:06:10.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071877729999999997 2023-07-02T14:06:10.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:10.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091517+0000", 2023-07-02T14:06:10.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3424999999999999e-05 2023-07-02T14:06:10.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:10.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091550+0000", 2023-07-02T14:06:10.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109724 2023-07-02T14:06:10.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:10.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091660+0000", 2023-07-02T14:06:10.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.112277108 2023-07-02T14:06:10.169 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.169 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.169 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.170 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17050 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:10.170 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:06:10.170 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 27.134712153999999, 2023-07-02T14:06:10.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12076701300000001, 2023-07-02T14:06:10.170 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:10.171 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:10.171 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:10.171 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:10.171 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:10.171 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17050 2023-07-02T14:06:10.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.171 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:10.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:10.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:06:10.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:10.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:10.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:06:10.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:10.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:10.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:06:10.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0759999999999999e-06 2023-07-02T14:06:10.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:10.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979402+0000", 2023-07-02T14:06:10.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0480000000000004e-06 2023-07-02T14:06:10.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:10.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979408+0000", 2023-07-02T14:06:10.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9559999999999999e-06 2023-07-02T14:06:10.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:10.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979412+0000", 2023-07-02T14:06:10.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013117658000000001 2023-07-02T14:06:10.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992530+0000", 2023-07-02T14:06:10.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4464000000000001e-05 2023-07-02T14:06:10.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992564+0000", 2023-07-02T14:06:10.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0063143640000000003 2023-07-02T14:06:10.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998879+0000", 2023-07-02T14:06:10.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2842000000000002e-05 2023-07-02T14:06:10.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998911+0000", 2023-07-02T14:06:10.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091863190000000001 2023-07-02T14:06:10.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008098+0000", 2023-07-02T14:06:10.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1448999999999999e-05 2023-07-02T14:06:10.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008179+0000", 2023-07-02T14:06:10.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059353720000000004 2023-07-02T14:06:10.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014115+0000", 2023-07-02T14:06:10.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1928000000000002e-05 2023-07-02T14:06:10.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014157+0000", 2023-07-02T14:06:10.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074935210000000004 2023-07-02T14:06:10.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021650+0000", 2023-07-02T14:06:10.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0199999999999999e-05 2023-07-02T14:06:10.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021680+0000", 2023-07-02T14:06:10.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057425109999999996 2023-07-02T14:06:10.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027423+0000", 2023-07-02T14:06:10.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1498999999999998e-05 2023-07-02T14:06:10.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027454+0000", 2023-07-02T14:06:10.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047903420000000004 2023-07-02T14:06:10.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032245+0000", 2023-07-02T14:06:10.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2552999999999998e-05 2023-07-02T14:06:10.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032277+0000", 2023-07-02T14:06:10.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057638079999999996 2023-07-02T14:06:10.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038041+0000", 2023-07-02T14:06:10.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9168000000000002e-05 2023-07-02T14:06:10.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038080+0000", 2023-07-02T14:06:10.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044291799999999996 2023-07-02T14:06:10.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042509+0000", 2023-07-02T14:06:10.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3528999999999997e-05 2023-07-02T14:06:10.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042543+0000", 2023-07-02T14:06:10.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073244729999999997 2023-07-02T14:06:10.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049867+0000", 2023-07-02T14:06:10.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2873999999999997e-05 2023-07-02T14:06:10.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049900+0000", 2023-07-02T14:06:10.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0095670149999999999 2023-07-02T14:06:10.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059467+0000", 2023-07-02T14:06:10.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0406000000000003e-05 2023-07-02T14:06:10.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059538+0000", 2023-07-02T14:06:10.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079635079999999994 2023-07-02T14:06:10.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067501+0000", 2023-07-02T14:06:10.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4795999999999997e-05 2023-07-02T14:06:10.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067536+0000", 2023-07-02T14:06:10.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091231969999999996 2023-07-02T14:06:10.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076659+0000", 2023-07-02T14:06:10.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3208999999999999e-05 2023-07-02T14:06:10.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076702+0000", 2023-07-02T14:06:10.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070660480000000001 2023-07-02T14:06:10.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.083768+0000", 2023-07-02T14:06:10.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5546000000000001e-05 2023-07-02T14:06:10.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.083814+0000", 2023-07-02T14:06:10.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078830659999999993 2023-07-02T14:06:10.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091697+0000", 2023-07-02T14:06:10.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5921000000000003e-05 2023-07-02T14:06:10.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:10.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091743+0000", 2023-07-02T14:06:10.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026571400000000001 2023-07-02T14:06:10.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:10.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.092009+0000", 2023-07-02T14:06:10.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022058429999999999 2023-07-02T14:06:10.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:10.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.094214+0000", 2023-07-02T14:06:10.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057115869999999997 2023-07-02T14:06:10.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:10.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.099926+0000", 2023-07-02T14:06:10.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0287999999999997e-05 2023-07-02T14:06:10.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:10.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.099976+0000", 2023-07-02T14:06:10.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019014999999999999 2023-07-02T14:06:10.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:10.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.100166+0000", 2023-07-02T14:06:10.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12076701300000001 2023-07-02T14:06:10.197 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.198 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.198 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.198 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17096 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:10.198 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:06:10.198 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 27.053946377999999, 2023-07-02T14:06:10.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.10620154, 2023-07-02T14:06:10.199 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:10.199 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:10.199 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:10.199 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:10.199 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:10.199 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17096 2023-07-02T14:06:10.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.200 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:10.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:10.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:06:10.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:10.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:10.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:06:10.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:10.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:10.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060165+0000", 2023-07-02T14:06:10.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3400000000000002e-06 2023-07-02T14:06:10.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:10.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060168+0000", 2023-07-02T14:06:10.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4020000000000004e-06 2023-07-02T14:06:10.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:10.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060173+0000", 2023-07-02T14:06:10.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4259999999999999e-06 2023-07-02T14:06:10.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:10.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060176+0000", 2023-07-02T14:06:10.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00049978399999999997 2023-07-02T14:06:10.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060676+0000", 2023-07-02T14:06:10.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0135000000000004e-05 2023-07-02T14:06:10.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.060756+0000", 2023-07-02T14:06:10.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074338060000000003 2023-07-02T14:06:10.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068190+0000", 2023-07-02T14:06:10.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7941000000000004e-05 2023-07-02T14:06:10.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.068278+0000", 2023-07-02T14:06:10.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089611870000000007 2023-07-02T14:06:10.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077239+0000", 2023-07-02T14:06:10.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6349e-05 2023-07-02T14:06:10.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.077275+0000", 2023-07-02T14:06:10.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007072142 2023-07-02T14:06:10.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084347+0000", 2023-07-02T14:06:10.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.591e-05 2023-07-02T14:06:10.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084383+0000", 2023-07-02T14:06:10.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093660119999999999 2023-07-02T14:06:10.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093749+0000", 2023-07-02T14:06:10.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1538000000000002e-05 2023-07-02T14:06:10.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.093781+0000", 2023-07-02T14:06:10.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088884629999999992 2023-07-02T14:06:10.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102669+0000", 2023-07-02T14:06:10.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6032999999999997e-05 2023-07-02T14:06:10.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.102725+0000", 2023-07-02T14:06:10.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009945522 2023-07-02T14:06:10.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112671+0000", 2023-07-02T14:06:10.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5741000000000002e-05 2023-07-02T14:06:10.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.112707+0000", 2023-07-02T14:06:10.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096867940000000003 2023-07-02T14:06:10.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122393+0000", 2023-07-02T14:06:10.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6244999999999998e-05 2023-07-02T14:06:10.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.122480+0000", 2023-07-02T14:06:10.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011251107999999999 2023-07-02T14:06:10.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133731+0000", 2023-07-02T14:06:10.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6400999999999999e-05 2023-07-02T14:06:10.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.133767+0000", 2023-07-02T14:06:10.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0094677070000000006 2023-07-02T14:06:10.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.143235+0000", 2023-07-02T14:06:10.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8721999999999996e-05 2023-07-02T14:06:10.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.143314+0000", 2023-07-02T14:06:10.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051196050000000002 2023-07-02T14:06:10.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.148433+0000", 2023-07-02T14:06:10.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2484999999999997e-05 2023-07-02T14:06:10.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.148476+0000", 2023-07-02T14:06:10.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006202234 2023-07-02T14:06:10.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.154678+0000", 2023-07-02T14:06:10.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2699000000000002e-05 2023-07-02T14:06:10.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.154761+0000", 2023-07-02T14:06:10.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049653809999999996 2023-07-02T14:06:10.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.159726+0000", 2023-07-02T14:06:10.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6862e-05 2023-07-02T14:06:10.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:10.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.159773+0000", 2023-07-02T14:06:10.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024618099999999999 2023-07-02T14:06:10.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:10.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.160019+0000", 2023-07-02T14:06:10.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0019043059999999999 2023-07-02T14:06:10.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:10.221 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.161923+0000", 2023-07-02T14:06:10.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004316884 2023-07-02T14:06:10.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:10.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.166240+0000", 2023-07-02T14:06:10.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2611000000000003e-05 2023-07-02T14:06:10.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:10.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.166273+0000", 2023-07-02T14:06:10.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4047000000000003e-05 2023-07-02T14:06:10.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:10.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.166367+0000", 2023-07-02T14:06:10.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.10620154 2023-07-02T14:06:10.223 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.223 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.223 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.224 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17295 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:10.224 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.256057+0000", 2023-07-02T14:06:10.224 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 26.858054509999999, 2023-07-02T14:06:10.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.107964279, 2023-07-02T14:06:10.224 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:10.225 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:10.225 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:10.225 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:10.225 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:10.225 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17295 2023-07-02T14:06:10.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.225 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:10.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:10.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256057+0000", 2023-07-02T14:06:10.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:10.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:10.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256057+0000", 2023-07-02T14:06:10.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:10.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:10.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256057+0000", 2023-07-02T14:06:10.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.458e-06 2023-07-02T14:06:10.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:10.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256059+0000", 2023-07-02T14:06:10.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9019999999999999e-06 2023-07-02T14:06:10.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:10.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256062+0000", 2023-07-02T14:06:10.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9929999999999998e-06 2023-07-02T14:06:10.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:10.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.256064+0000", 2023-07-02T14:06:10.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023724789999999998 2023-07-02T14:06:10.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.258437+0000", 2023-07-02T14:06:10.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7698000000000002e-05 2023-07-02T14:06:10.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.230 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.258494+0000", 2023-07-02T14:06:10.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066424140000000001 2023-07-02T14:06:10.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.231 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.265137+0000", 2023-07-02T14:06:10.231 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2169000000000002e-05 2023-07-02T14:06:10.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.231 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.265179+0000", 2023-07-02T14:06:10.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089047970000000008 2023-07-02T14:06:10.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.274084+0000", 2023-07-02T14:06:10.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7489e-05 2023-07-02T14:06:10.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.274121+0000", 2023-07-02T14:06:10.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013452426999999999 2023-07-02T14:06:10.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.287574+0000", 2023-07-02T14:06:10.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5149999999999996e-05 2023-07-02T14:06:10.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.287659+0000", 2023-07-02T14:06:10.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069046059999999998 2023-07-02T14:06:10.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.294563+0000", 2023-07-02T14:06:10.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7073000000000001e-05 2023-07-02T14:06:10.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.294611+0000", 2023-07-02T14:06:10.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076769480000000003 2023-07-02T14:06:10.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302287+0000", 2023-07-02T14:06:10.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4381000000000002e-05 2023-07-02T14:06:10.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302322+0000", 2023-07-02T14:06:10.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009637863 2023-07-02T14:06:10.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.311960+0000", 2023-07-02T14:06:10.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4847999999999999e-05 2023-07-02T14:06:10.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312035+0000", 2023-07-02T14:06:10.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098741579999999992 2023-07-02T14:06:10.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.321909+0000", 2023-07-02T14:06:10.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6455999999999998e-05 2023-07-02T14:06:10.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.321945+0000", 2023-07-02T14:06:10.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099894690000000008 2023-07-02T14:06:10.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.240 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.331935+0000", 2023-07-02T14:06:10.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2550000000000002e-05 2023-07-02T14:06:10.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.240 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332007+0000", 2023-07-02T14:06:10.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0086760090000000002 2023-07-02T14:06:10.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.340683+0000", 2023-07-02T14:06:10.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8462999999999998e-05 2023-07-02T14:06:10.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.340752+0000", 2023-07-02T14:06:10.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00830381 2023-07-02T14:06:10.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349055+0000", 2023-07-02T14:06:10.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9289999999999994e-05 2023-07-02T14:06:10.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349145+0000", 2023-07-02T14:06:10.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067717250000000001 2023-07-02T14:06:10.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.355917+0000", 2023-07-02T14:06:10.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4431999999999995e-05 2023-07-02T14:06:10.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:10.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.355991+0000", 2023-07-02T14:06:10.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00044773399999999998 2023-07-02T14:06:10.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:10.245 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.356439+0000", 2023-07-02T14:06:10.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028403690000000001 2023-07-02T14:06:10.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:10.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.359279+0000", 2023-07-02T14:06:10.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045352450000000002 2023-07-02T14:06:10.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.246 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.246 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:10.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.363814+0000", 2023-07-02T14:06:10.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6167e-05 2023-07-02T14:06:10.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:10.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.363860+0000", 2023-07-02T14:06:10.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000160951 2023-07-02T14:06:10.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:10.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.364021+0000", 2023-07-02T14:06:10.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.107964279 2023-07-02T14:06:10.248 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.248 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.248 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.248 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17353 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:10.249 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:06:10.249 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 26.811527477999999, 2023-07-02T14:06:10.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.115724379, 2023-07-02T14:06:10.249 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:10.249 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:10.249 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:10.249 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:10.249 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:10.250 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17353 2023-07-02T14:06:10.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.250 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:10.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:10.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:06:10.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:10.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.251 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:10.251 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:06:10.251 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:10.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.251 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:10.251 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:06:10.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.41e-06 2023-07-02T14:06:10.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.252 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:10.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302587+0000", 2023-07-02T14:06:10.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.015e-06 2023-07-02T14:06:10.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:10.253 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302593+0000", 2023-07-02T14:06:10.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8019999999999997e-06 2023-07-02T14:06:10.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:10.253 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302599+0000", 2023-07-02T14:06:10.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022701199999999999 2023-07-02T14:06:10.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302826+0000", 2023-07-02T14:06:10.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.426e-05 2023-07-02T14:06:10.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302870+0000", 2023-07-02T14:06:10.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096759489999999997 2023-07-02T14:06:10.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312546+0000", 2023-07-02T14:06:10.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5785000000000006e-05 2023-07-02T14:06:10.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312632+0000", 2023-07-02T14:06:10.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098106970000000002 2023-07-02T14:06:10.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322443+0000", 2023-07-02T14:06:10.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3161999999999999e-05 2023-07-02T14:06:10.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322476+0000", 2023-07-02T14:06:10.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010015823 2023-07-02T14:06:10.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332492+0000", 2023-07-02T14:06:10.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1165999999999999e-05 2023-07-02T14:06:10.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332533+0000", 2023-07-02T14:06:10.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089435269999999997 2023-07-02T14:06:10.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341476+0000", 2023-07-02T14:06:10.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000112691 2023-07-02T14:06:10.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341589+0000", 2023-07-02T14:06:10.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008147362 2023-07-02T14:06:10.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349736+0000", 2023-07-02T14:06:10.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5149000000000007e-05 2023-07-02T14:06:10.261 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.261 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.261 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349832+0000", 2023-07-02T14:06:10.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089226519999999997 2023-07-02T14:06:10.261 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.262 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.262 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358754+0000", 2023-07-02T14:06:10.262 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6091999999999998e-05 2023-07-02T14:06:10.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.262 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.262 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.262 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358810+0000", 2023-07-02T14:06:10.262 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080318000000000004 2023-07-02T14:06:10.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.263 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366842+0000", 2023-07-02T14:06:10.263 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4321999999999998e-05 2023-07-02T14:06:10.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366916+0000", 2023-07-02T14:06:10.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074664049999999997 2023-07-02T14:06:10.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.264 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374383+0000", 2023-07-02T14:06:10.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5460000000000001e-05 2023-07-02T14:06:10.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.265 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374438+0000", 2023-07-02T14:06:10.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091562450000000004 2023-07-02T14:06:10.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383594+0000", 2023-07-02T14:06:10.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2545000000000004e-05 2023-07-02T14:06:10.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383667+0000", 2023-07-02T14:06:10.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013424551 2023-07-02T14:06:10.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397092+0000", 2023-07-02T14:06:10.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6991999999999997e-05 2023-07-02T14:06:10.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397169+0000", 2023-07-02T14:06:10.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064399139999999997 2023-07-02T14:06:10.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.268 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403608+0000", 2023-07-02T14:06:10.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2614000000000006e-05 2023-07-02T14:06:10.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403691+0000", 2023-07-02T14:06:10.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004611693 2023-07-02T14:06:10.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408303+0000", 2023-07-02T14:06:10.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1935999999999995e-05 2023-07-02T14:06:10.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:10.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408395+0000", 2023-07-02T14:06:10.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00051077700000000002 2023-07-02T14:06:10.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:10.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408906+0000", 2023-07-02T14:06:10.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090767460000000001 2023-07-02T14:06:10.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.271 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:10.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.417982+0000", 2023-07-02T14:06:10.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4283999999999997e-05 2023-07-02T14:06:10.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:10.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418037+0000", 2023-07-02T14:06:10.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9667999999999999e-05 2023-07-02T14:06:10.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:10.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418096+0000", 2023-07-02T14:06:10.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021233800000000001 2023-07-02T14:06:10.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.273 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:10.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418309+0000", 2023-07-02T14:06:10.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.115724379 2023-07-02T14:06:10.273 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.274 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.274 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.274 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17354 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:10.274 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:10.274 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 26.811508518, 2023-07-02T14:06:10.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493, 2023-07-02T14:06:10.275 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:10.275 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:10.275 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:10.275 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:10.275 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:10.275 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17354 2023-07-02T14:06:10.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.275 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:10.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:10.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:10.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:10.276 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:10.277 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:10.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:10.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.277 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:10.277 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:10.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7440000000000001e-06 2023-07-02T14:06:10.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:10.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302606+0000", 2023-07-02T14:06:10.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:06:10.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:10.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302613+0000", 2023-07-02T14:06:10.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8810000000000004e-06 2023-07-02T14:06:10.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:10.279 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302618+0000", 2023-07-02T14:06:10.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028613100000000002 2023-07-02T14:06:10.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302904+0000", 2023-07-02T14:06:10.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5278999999999995e-05 2023-07-02T14:06:10.280 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302979+0000", 2023-07-02T14:06:10.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096904750000000005 2023-07-02T14:06:10.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312669+0000", 2023-07-02T14:06:10.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0328000000000002e-05 2023-07-02T14:06:10.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312700+0000", 2023-07-02T14:06:10.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098019319999999993 2023-07-02T14:06:10.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.282 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322502+0000", 2023-07-02T14:06:10.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4669999999999998e-05 2023-07-02T14:06:10.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.283 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322536+0000", 2023-07-02T14:06:10.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010023839 2023-07-02T14:06:10.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332560+0000", 2023-07-02T14:06:10.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5559999999999998e-05 2023-07-02T14:06:10.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332596+0000", 2023-07-02T14:06:10.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090401319999999993 2023-07-02T14:06:10.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341636+0000", 2023-07-02T14:06:10.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5373000000000001e-05 2023-07-02T14:06:10.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341681+0000", 2023-07-02T14:06:10.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081994059999999994 2023-07-02T14:06:10.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349881+0000", 2023-07-02T14:06:10.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4319000000000001e-05 2023-07-02T14:06:10.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349915+0000", 2023-07-02T14:06:10.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089388339999999997 2023-07-02T14:06:10.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358854+0000", 2023-07-02T14:06:10.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4319000000000005e-05 2023-07-02T14:06:10.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358928+0000", 2023-07-02T14:06:10.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080234690000000001 2023-07-02T14:06:10.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366952+0000", 2023-07-02T14:06:10.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2082999999999998e-05 2023-07-02T14:06:10.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366984+0000", 2023-07-02T14:06:10.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075015790000000004 2023-07-02T14:06:10.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374485+0000", 2023-07-02T14:06:10.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4480999999999995e-05 2023-07-02T14:06:10.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374550+0000", 2023-07-02T14:06:10.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091666620000000008 2023-07-02T14:06:10.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383716+0000", 2023-07-02T14:06:10.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9318000000000003e-05 2023-07-02T14:06:10.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383786+0000", 2023-07-02T14:06:10.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013434115 2023-07-02T14:06:10.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397220+0000", 2023-07-02T14:06:10.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4573999999999997e-05 2023-07-02T14:06:10.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397264+0000", 2023-07-02T14:06:10.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064824649999999998 2023-07-02T14:06:10.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.294 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403747+0000", 2023-07-02T14:06:10.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1087999999999993e-05 2023-07-02T14:06:10.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.294 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403828+0000", 2023-07-02T14:06:10.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072854449999999998 2023-07-02T14:06:10.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.295 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411113+0000", 2023-07-02T14:06:10.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5120000000000001e-05 2023-07-02T14:06:10.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411169+0000", 2023-07-02T14:06:10.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007211505 2023-07-02T14:06:10.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418380+0000", 2023-07-02T14:06:10.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011289399999999999 2023-07-02T14:06:10.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:10.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418493+0000", 2023-07-02T14:06:10.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00051217500000000002 2023-07-02T14:06:10.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:10.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.419005+0000", 2023-07-02T14:06:10.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00327393 2023-07-02T14:06:10.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:10.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422279+0000", 2023-07-02T14:06:10.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041432350000000003 2023-07-02T14:06:10.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:10.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426422+0000", 2023-07-02T14:06:10.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7021000000000003e-05 2023-07-02T14:06:10.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:10.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426469+0000", 2023-07-02T14:06:10.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000190256 2023-07-02T14:06:10.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:10.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426660+0000", 2023-07-02T14:06:10.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493 2023-07-02T14:06:10.300 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.301 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.301 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.301 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17355 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:10.301 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:10.301 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 26.811489857000002, 2023-07-02T14:06:10.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001, 2023-07-02T14:06:10.302 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:10.302 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:10.302 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:10.302 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:10.302 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:10.302 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17355 2023-07-02T14:06:10.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.303 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:10.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:10.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:10.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:10.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:10.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:10.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:10.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:10.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302621+0000", 2023-07-02T14:06:10.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2260000000000001e-06 2023-07-02T14:06:10.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:10.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302625+0000", 2023-07-02T14:06:10.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8599999999999998e-06 2023-07-02T14:06:10.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:10.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302630+0000", 2023-07-02T14:06:10.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0629999999999999e-06 2023-07-02T14:06:10.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:10.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302635+0000", 2023-07-02T14:06:10.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035731199999999998 2023-07-02T14:06:10.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.307 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302992+0000", 2023-07-02T14:06:10.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4164e-05 2023-07-02T14:06:10.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.307 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.303026+0000", 2023-07-02T14:06:10.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097115900000000008 2023-07-02T14:06:10.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312738+0000", 2023-07-02T14:06:10.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1701999999999998e-05 2023-07-02T14:06:10.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312819+0000", 2023-07-02T14:06:10.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097363640000000008 2023-07-02T14:06:10.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322556+0000", 2023-07-02T14:06:10.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2517000000000001e-05 2023-07-02T14:06:10.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322588+0000", 2023-07-02T14:06:10.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01002393 2023-07-02T14:06:10.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332612+0000", 2023-07-02T14:06:10.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2960999999999998e-05 2023-07-02T14:06:10.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332645+0000", 2023-07-02T14:06:10.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090669819999999995 2023-07-02T14:06:10.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341712+0000", 2023-07-02T14:06:10.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9288000000000002e-05 2023-07-02T14:06:10.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341811+0000", 2023-07-02T14:06:10.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081504460000000004 2023-07-02T14:06:10.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349962+0000", 2023-07-02T14:06:10.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107321 2023-07-02T14:06:10.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.350069+0000", 2023-07-02T14:06:10.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088790999999999991 2023-07-02T14:06:10.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358948+0000", 2023-07-02T14:06:10.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1711000000000002e-05 2023-07-02T14:06:10.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358980+0000", 2023-07-02T14:06:10.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080367579999999997 2023-07-02T14:06:10.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367017+0000", 2023-07-02T14:06:10.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6329000000000002e-05 2023-07-02T14:06:10.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367073+0000", 2023-07-02T14:06:10.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075097300000000001 2023-07-02T14:06:10.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374583+0000", 2023-07-02T14:06:10.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5710999999999995e-05 2023-07-02T14:06:10.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374648+0000", 2023-07-02T14:06:10.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009165912 2023-07-02T14:06:10.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383814+0000", 2023-07-02T14:06:10.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3916000000000003e-05 2023-07-02T14:06:10.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383878+0000", 2023-07-02T14:06:10.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013440532999999999 2023-07-02T14:06:10.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397319+0000", 2023-07-02T14:06:10.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7218999999999995e-05 2023-07-02T14:06:10.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397386+0000", 2023-07-02T14:06:10.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065165279999999997 2023-07-02T14:06:10.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403903+0000", 2023-07-02T14:06:10.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8152000000000005e-05 2023-07-02T14:06:10.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404001+0000", 2023-07-02T14:06:10.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072185360000000002 2023-07-02T14:06:10.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.322 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411219+0000", 2023-07-02T14:06:10.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:06:10.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411310+0000", 2023-07-02T14:06:10.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010021663 2023-07-02T14:06:10.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421331+0000", 2023-07-02T14:06:10.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0148000000000006e-05 2023-07-02T14:06:10.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.324 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421411+0000", 2023-07-02T14:06:10.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053056170000000003 2023-07-02T14:06:10.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426717+0000", 2023-07-02T14:06:10.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8242000000000001e-05 2023-07-02T14:06:10.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:10.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426795+0000", 2023-07-02T14:06:10.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00041308399999999998 2023-07-02T14:06:10.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:10.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.427208+0000", 2023-07-02T14:06:10.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028330849999999999 2023-07-02T14:06:10.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:10.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430041+0000", 2023-07-02T14:06:10.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00072642300000000002 2023-07-02T14:06:10.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:10.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430768+0000", 2023-07-02T14:06:10.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1847000000000002e-05 2023-07-02T14:06:10.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:10.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430800+0000", 2023-07-02T14:06:10.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9308999999999994e-05 2023-07-02T14:06:10.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:10.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430899+0000", 2023-07-02T14:06:10.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001 2023-07-02T14:06:10.329 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.329 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.329 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.329 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:410 2.5 2:bbaed619:::rbd_data.125de1449605.0000000000000109:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:06:10.330 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:10.330 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.663544591999999, 2023-07-02T14:06:10.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003, 2023-07-02T14:06:10.330 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:10.330 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:10.330 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:10.330 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:06:10.330 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:06:10.331 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:06:10.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.331 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:10.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:10.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:10.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:10.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:10.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:10.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:10.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:10.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:10.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.635e-06 2023-07-02T14:06:10.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:10.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450568+0000", 2023-07-02T14:06:10.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5970000000000002e-06 2023-07-02T14:06:10.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:10.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450573+0000", 2023-07-02T14:06:10.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3379999999999999e-06 2023-07-02T14:06:10.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:10.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450575+0000", 2023-07-02T14:06:10.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9458000000000001e-05 2023-07-02T14:06:10.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450615+0000", 2023-07-02T14:06:10.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0569999999999993e-06 2023-07-02T14:06:10.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:10.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450623+0000", 2023-07-02T14:06:10.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.63968709199999996 2023-07-02T14:06:10.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090310+0000", 2023-07-02T14:06:10.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020157399999999999 2023-07-02T14:06:10.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:10.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090512+0000", 2023-07-02T14:06:10.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027739190000000001 2023-07-02T14:06:10.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:10.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093285+0000", 2023-07-02T14:06:10.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003 2023-07-02T14:06:10.338 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.338 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.338 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.339 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:416 2.5 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:06:10.339 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:10.339 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.663515158999999, 2023-07-02T14:06:10.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002, 2023-07-02T14:06:10.339 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:10.339 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:10.339 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:10.339 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:06:10.340 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:06:10.340 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:06:10.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.340 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:10.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:10.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:10.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:10.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:10.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:10.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:10.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:10.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:10.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.34e-06 2023-07-02T14:06:10.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:10.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450599+0000", 2023-07-02T14:06:10.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1549999999999998e-06 2023-07-02T14:06:10.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:10.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450604+0000", 2023-07-02T14:06:10.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:06:10.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:10.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450607+0000", 2023-07-02T14:06:10.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5717999999999999e-05 2023-07-02T14:06:10.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450643+0000", 2023-07-02T14:06:10.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1619999999999999e-06 2023-07-02T14:06:10.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:10.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450648+0000", 2023-07-02T14:06:10.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64292291800000001 2023-07-02T14:06:10.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093571+0000", 2023-07-02T14:06:10.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115628 2023-07-02T14:06:10.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:10.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093687+0000", 2023-07-02T14:06:10.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002885552 2023-07-02T14:06:10.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:10.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.096572+0000", 2023-07-02T14:06:10.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002 2023-07-02T14:06:10.347 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.347 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.347 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.348 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18552 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:10.348 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.244993+0000", 2023-07-02T14:06:10.348 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.8691190190000002, 2023-07-02T14:06:10.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.107684349, 2023-07-02T14:06:10.348 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:10.348 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:10.348 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:10.349 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:10.349 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:10.349 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18552 2023-07-02T14:06:10.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.349 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:10.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:10.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.244993+0000", 2023-07-02T14:06:10.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:10.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:10.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.244993+0000", 2023-07-02T14:06:10.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:10.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:10.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.244992+0000", 2023-07-02T14:06:10.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.613e-06 2023-07-02T14:06:10.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:10.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.244996+0000", 2023-07-02T14:06:10.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7490000000000004e-06 2023-07-02T14:06:10.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:10.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.245002+0000", 2023-07-02T14:06:10.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3120000000000004e-06 2023-07-02T14:06:10.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:10.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.245006+0000", 2023-07-02T14:06:10.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000359706 2023-07-02T14:06:10.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.245366+0000", 2023-07-02T14:06:10.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0433999999999999e-05 2023-07-02T14:06:10.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.245406+0000", 2023-07-02T14:06:10.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041032769999999998 2023-07-02T14:06:10.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249509+0000", 2023-07-02T14:06:10.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3030000000000001e-05 2023-07-02T14:06:10.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249542+0000", 2023-07-02T14:06:10.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066355299999999997 2023-07-02T14:06:10.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256178+0000", 2023-07-02T14:06:10.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5154999999999998e-05 2023-07-02T14:06:10.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256213+0000", 2023-07-02T14:06:10.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012170596000000001 2023-07-02T14:06:10.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268384+0000", 2023-07-02T14:06:10.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3485000000000006e-05 2023-07-02T14:06:10.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268447+0000", 2023-07-02T14:06:10.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011136736 2023-07-02T14:06:10.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279584+0000", 2023-07-02T14:06:10.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3447e-05 2023-07-02T14:06:10.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279617+0000", 2023-07-02T14:06:10.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011815668 2023-07-02T14:06:10.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291433+0000", 2023-07-02T14:06:10.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0329e-05 2023-07-02T14:06:10.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291463+0000", 2023-07-02T14:06:10.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079621299999999996 2023-07-02T14:06:10.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299425+0000", 2023-07-02T14:06:10.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6808999999999999e-05 2023-07-02T14:06:10.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299462+0000", 2023-07-02T14:06:10.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072025659999999997 2023-07-02T14:06:10.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306665+0000", 2023-07-02T14:06:10.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8457e-05 2023-07-02T14:06:10.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306723+0000", 2023-07-02T14:06:10.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056322869999999997 2023-07-02T14:06:10.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312355+0000", 2023-07-02T14:06:10.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4682999999999998e-05 2023-07-02T14:06:10.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312410+0000", 2023-07-02T14:06:10.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004796453 2023-07-02T14:06:10.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317207+0000", 2023-07-02T14:06:10.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6775999999999999e-05 2023-07-02T14:06:10.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317273+0000", 2023-07-02T14:06:10.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049760610000000004 2023-07-02T14:06:10.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322249+0000", 2023-07-02T14:06:10.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7802999999999999e-05 2023-07-02T14:06:10.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322337+0000", 2023-07-02T14:06:10.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066115770000000004 2023-07-02T14:06:10.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.328949+0000", 2023-07-02T14:06:10.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6105e-05 2023-07-02T14:06:10.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.328995+0000", 2023-07-02T14:06:10.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062196949999999999 2023-07-02T14:06:10.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335215+0000", 2023-07-02T14:06:10.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0904e-05 2023-07-02T14:06:10.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335256+0000", 2023-07-02T14:06:10.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096283380000000002 2023-07-02T14:06:10.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.344884+0000", 2023-07-02T14:06:10.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7175999999999994e-05 2023-07-02T14:06:10.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.344961+0000", 2023-07-02T14:06:10.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029952550000000001 2023-07-02T14:06:10.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.347956+0000", 2023-07-02T14:06:10.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8840000000000001e-05 2023-07-02T14:06:10.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:10.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.347995+0000", 2023-07-02T14:06:10.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00023967300000000001 2023-07-02T14:06:10.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:10.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.348235+0000", 2023-07-02T14:06:10.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0023177219999999999 2023-07-02T14:06:10.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:10.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350553+0000", 2023-07-02T14:06:10.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0019716069999999998 2023-07-02T14:06:10.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:10.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.352524+0000", 2023-07-02T14:06:10.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0142e-05 2023-07-02T14:06:10.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:10.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.352554+0000", 2023-07-02T14:06:10.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012266600000000001 2023-07-02T14:06:10.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:10.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.352677+0000", 2023-07-02T14:06:10.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.107684349 2023-07-02T14:06:10.375 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.376 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.376 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.376 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18554 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:10.376 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248917+0000", 2023-07-02T14:06:10.376 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.8651943429999998, 2023-07-02T14:06:10.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11109714599999999, 2023-07-02T14:06:10.377 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:10.377 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:10.377 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:10.377 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:10.377 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:10.377 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18554 2023-07-02T14:06:10.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.377 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:10.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:10.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248917+0000", 2023-07-02T14:06:10.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:10.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:10.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248917+0000", 2023-07-02T14:06:10.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:10.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:10.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248916+0000", 2023-07-02T14:06:10.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1690000000000003e-06 2023-07-02T14:06:10.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:10.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248923+0000", 2023-07-02T14:06:10.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2769999999999994e-06 2023-07-02T14:06:10.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:10.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248933+0000", 2023-07-02T14:06:10.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1860000000000002e-06 2023-07-02T14:06:10.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:10.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248937+0000", 2023-07-02T14:06:10.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00063144000000000004 2023-07-02T14:06:10.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249568+0000", 2023-07-02T14:06:10.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1486999999999998e-05 2023-07-02T14:06:10.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249610+0000", 2023-07-02T14:06:10.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066288199999999997 2023-07-02T14:06:10.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256239+0000", 2023-07-02T14:06:10.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3636000000000001e-05 2023-07-02T14:06:10.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256272+0000", 2023-07-02T14:06:10.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01221675 2023-07-02T14:06:10.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268489+0000", 2023-07-02T14:06:10.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0283000000000004e-05 2023-07-02T14:06:10.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268569+0000", 2023-07-02T14:06:10.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011087254 2023-07-02T14:06:10.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279657+0000", 2023-07-02T14:06:10.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2872e-05 2023-07-02T14:06:10.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279689+0000", 2023-07-02T14:06:10.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011797277 2023-07-02T14:06:10.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291487+0000", 2023-07-02T14:06:10.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0111e-05 2023-07-02T14:06:10.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291517+0000", 2023-07-02T14:06:10.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079886279999999994 2023-07-02T14:06:10.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299505+0000", 2023-07-02T14:06:10.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2787000000000006e-05 2023-07-02T14:06:10.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299578+0000", 2023-07-02T14:06:10.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071806889999999996 2023-07-02T14:06:10.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306759+0000", 2023-07-02T14:06:10.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1510000000000002e-05 2023-07-02T14:06:10.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306790+0000", 2023-07-02T14:06:10.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056651269999999998 2023-07-02T14:06:10.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312456+0000", 2023-07-02T14:06:10.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8580999999999999e-05 2023-07-02T14:06:10.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312534+0000", 2023-07-02T14:06:10.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047950680000000004 2023-07-02T14:06:10.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317329+0000", 2023-07-02T14:06:10.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4718000000000002e-05 2023-07-02T14:06:10.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317364+0000", 2023-07-02T14:06:10.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050140469999999998 2023-07-02T14:06:10.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322378+0000", 2023-07-02T14:06:10.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1049e-05 2023-07-02T14:06:10.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322409+0000", 2023-07-02T14:06:10.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006616612 2023-07-02T14:06:10.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329026+0000", 2023-07-02T14:06:10.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4612e-05 2023-07-02T14:06:10.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329060+0000", 2023-07-02T14:06:10.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062225020000000004 2023-07-02T14:06:10.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335283+0000", 2023-07-02T14:06:10.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0385e-05 2023-07-02T14:06:10.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335323+0000", 2023-07-02T14:06:10.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009678262 2023-07-02T14:06:10.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345001+0000", 2023-07-02T14:06:10.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0340999999999999e-05 2023-07-02T14:06:10.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345042+0000", 2023-07-02T14:06:10.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045838650000000003 2023-07-02T14:06:10.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349626+0000", 2023-07-02T14:06:10.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010221999999999999 2023-07-02T14:06:10.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349728+0000", 2023-07-02T14:06:10.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029884349999999998 2023-07-02T14:06:10.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.352716+0000", 2023-07-02T14:06:10.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3821e-05 2023-07-02T14:06:10.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:10.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.352750+0000", 2023-07-02T14:06:10.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021565399999999999 2023-07-02T14:06:10.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:10.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.352966+0000", 2023-07-02T14:06:10.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018362960000000001 2023-07-02T14:06:10.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:10.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354802+0000", 2023-07-02T14:06:10.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051013409999999997 2023-07-02T14:06:10.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:10.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.359903+0000", 2023-07-02T14:06:10.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9068999999999999e-05 2023-07-02T14:06:10.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:10.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.359932+0000", 2023-07-02T14:06:10.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2008000000000005e-05 2023-07-02T14:06:10.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:10.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.360014+0000", 2023-07-02T14:06:10.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11109714599999999 2023-07-02T14:06:10.404 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.404 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.404 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.404 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18555 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:10.404 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248941+0000", 2023-07-02T14:06:10.405 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.8651707450000004, 2023-07-02T14:06:10.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11809699899999999, 2023-07-02T14:06:10.405 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:10.405 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:10.405 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:10.405 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:10.405 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:10.406 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18555 2023-07-02T14:06:10.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.406 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:10.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:10.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248941+0000", 2023-07-02T14:06:10.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:10.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:10.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248941+0000", 2023-07-02T14:06:10.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:10.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:10.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248940+0000", 2023-07-02T14:06:10.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.7159999999999999e-06 2023-07-02T14:06:10.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:10.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248943+0000", 2023-07-02T14:06:10.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5949999999999999e-06 2023-07-02T14:06:10.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:10.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248947+0000", 2023-07-02T14:06:10.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4480000000000001e-06 2023-07-02T14:06:10.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:10.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248949+0000", 2023-07-02T14:06:10.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00067899499999999995 2023-07-02T14:06:10.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249628+0000", 2023-07-02T14:06:10.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1100000000000003e-05 2023-07-02T14:06:10.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249669+0000", 2023-07-02T14:06:10.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066349690000000001 2023-07-02T14:06:10.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256304+0000", 2023-07-02T14:06:10.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2691999999999998e-05 2023-07-02T14:06:10.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256337+0000", 2023-07-02T14:06:10.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012284679999999999 2023-07-02T14:06:10.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268622+0000", 2023-07-02T14:06:10.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6550000000000001e-05 2023-07-02T14:06:10.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268658+0000", 2023-07-02T14:06:10.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011061109 2023-07-02T14:06:10.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279719+0000", 2023-07-02T14:06:10.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2134000000000001e-05 2023-07-02T14:06:10.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279751+0000", 2023-07-02T14:06:10.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011798329999999999 2023-07-02T14:06:10.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291550+0000", 2023-07-02T14:06:10.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9839999999999999e-05 2023-07-02T14:06:10.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291580+0000", 2023-07-02T14:06:10.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080386550000000005 2023-07-02T14:06:10.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299618+0000", 2023-07-02T14:06:10.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2632999999999997e-05 2023-07-02T14:06:10.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299651+0000", 2023-07-02T14:06:10.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071769859999999998 2023-07-02T14:06:10.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306828+0000", 2023-07-02T14:06:10.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1123000000000004e-05 2023-07-02T14:06:10.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306889+0000", 2023-07-02T14:06:10.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056835710000000001 2023-07-02T14:06:10.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312573+0000", 2023-07-02T14:06:10.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0115000000000002e-05 2023-07-02T14:06:10.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312643+0000", 2023-07-02T14:06:10.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004754538 2023-07-02T14:06:10.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317397+0000", 2023-07-02T14:06:10.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4385000000000002e-05 2023-07-02T14:06:10.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317452+0000", 2023-07-02T14:06:10.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049988350000000001 2023-07-02T14:06:10.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322450+0000", 2023-07-02T14:06:10.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4749e-05 2023-07-02T14:06:10.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322515+0000", 2023-07-02T14:06:10.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065710810000000003 2023-07-02T14:06:10.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329086+0000", 2023-07-02T14:06:10.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0982000000000003e-05 2023-07-02T14:06:10.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329117+0000", 2023-07-02T14:06:10.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062301259999999999 2023-07-02T14:06:10.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335347+0000", 2023-07-02T14:06:10.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2830999999999998e-05 2023-07-02T14:06:10.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335380+0000", 2023-07-02T14:06:10.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097041909999999992 2023-07-02T14:06:10.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345084+0000", 2023-07-02T14:06:10.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2044000000000003e-05 2023-07-02T14:06:10.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345156+0000", 2023-07-02T14:06:10.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004610711 2023-07-02T14:06:10.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349767+0000", 2023-07-02T14:06:10.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8640000000000002e-05 2023-07-02T14:06:10.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349816+0000", 2023-07-02T14:06:10.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043231160000000001 2023-07-02T14:06:10.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354139+0000", 2023-07-02T14:06:10.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8073e-05 2023-07-02T14:06:10.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354177+0000", 2023-07-02T14:06:10.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005870681 2023-07-02T14:06:10.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.360048+0000", 2023-07-02T14:06:10.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9535e-05 2023-07-02T14:06:10.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:10.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.360087+0000", 2023-07-02T14:06:10.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021778100000000001 2023-07-02T14:06:10.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:10.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.360305+0000", 2023-07-02T14:06:10.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001720708 2023-07-02T14:06:10.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:10.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.362026+0000", 2023-07-02T14:06:10.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049001009999999996 2023-07-02T14:06:10.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:10.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.366926+0000", 2023-07-02T14:06:10.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3682e-05 2023-07-02T14:06:10.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:10.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.366959+0000", 2023-07-02T14:06:10.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8419999999999998e-05 2023-07-02T14:06:10.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:10.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.367038+0000", 2023-07-02T14:06:10.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11809699899999999 2023-07-02T14:06:10.433 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.433 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.434 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.434 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18556 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:10.434 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248957+0000", 2023-07-02T14:06:10.434 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.8651547759999998, 2023-07-02T14:06:10.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12347108800000001, 2023-07-02T14:06:10.434 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:10.435 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:10.435 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:10.435 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:10.435 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:10.435 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18556 2023-07-02T14:06:10.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.435 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:10.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:10.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248957+0000", 2023-07-02T14:06:10.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:10.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:10.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248957+0000", 2023-07-02T14:06:10.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:10.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:10.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248957+0000", 2023-07-02T14:06:10.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4119999999999999e-06 2023-07-02T14:06:10.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:10.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248959+0000", 2023-07-02T14:06:10.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5669999999999999e-06 2023-07-02T14:06:10.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:10.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248962+0000", 2023-07-02T14:06:10.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3989999999999998e-06 2023-07-02T14:06:10.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:10.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248965+0000", 2023-07-02T14:06:10.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00072489500000000003 2023-07-02T14:06:10.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249690+0000", 2023-07-02T14:06:10.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7876999999999997e-05 2023-07-02T14:06:10.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249728+0000", 2023-07-02T14:06:10.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066341250000000003 2023-07-02T14:06:10.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256362+0000", 2023-07-02T14:06:10.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2715000000000001e-05 2023-07-02T14:06:10.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256395+0000", 2023-07-02T14:06:10.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012305508 2023-07-02T14:06:10.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268700+0000", 2023-07-02T14:06:10.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7537e-05 2023-07-02T14:06:10.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268778+0000", 2023-07-02T14:06:10.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011004300999999999 2023-07-02T14:06:10.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279782+0000", 2023-07-02T14:06:10.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1996000000000003e-05 2023-07-02T14:06:10.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279814+0000", 2023-07-02T14:06:10.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011789104 2023-07-02T14:06:10.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291603+0000", 2023-07-02T14:06:10.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9825999999999999e-05 2023-07-02T14:06:10.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291633+0000", 2023-07-02T14:06:10.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080637 2023-07-02T14:06:10.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299696+0000", 2023-07-02T14:06:10.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8535999999999996e-05 2023-07-02T14:06:10.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299765+0000", 2023-07-02T14:06:10.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071538169999999998 2023-07-02T14:06:10.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306919+0000", 2023-07-02T14:06:10.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0337000000000001e-05 2023-07-02T14:06:10.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306949+0000", 2023-07-02T14:06:10.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057379919999999999 2023-07-02T14:06:10.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312687+0000", 2023-07-02T14:06:10.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8463000000000001e-05 2023-07-02T14:06:10.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312726+0000", 2023-07-02T14:06:10.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004766251 2023-07-02T14:06:10.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317492+0000", 2023-07-02T14:06:10.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1788000000000001e-05 2023-07-02T14:06:10.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317524+0000", 2023-07-02T14:06:10.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050322910000000004 2023-07-02T14:06:10.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322556+0000", 2023-07-02T14:06:10.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9141999999999999e-05 2023-07-02T14:06:10.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322595+0000", 2023-07-02T14:06:10.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006553354 2023-07-02T14:06:10.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329148+0000", 2023-07-02T14:06:10.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2458e-05 2023-07-02T14:06:10.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329181+0000", 2023-07-02T14:06:10.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006222483 2023-07-02T14:06:10.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335403+0000", 2023-07-02T14:06:10.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0419000000000001e-05 2023-07-02T14:06:10.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335434+0000", 2023-07-02T14:06:10.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097556550000000002 2023-07-02T14:06:10.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345189+0000", 2023-07-02T14:06:10.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2968e-05 2023-07-02T14:06:10.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345222+0000", 2023-07-02T14:06:10.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046402270000000002 2023-07-02T14:06:10.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349863+0000", 2023-07-02T14:06:10.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7782999999999997e-05 2023-07-02T14:06:10.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349940+0000", 2023-07-02T14:06:10.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042690699999999998 2023-07-02T14:06:10.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354210+0000", 2023-07-02T14:06:10.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5648000000000001e-05 2023-07-02T14:06:10.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354255+0000", 2023-07-02T14:06:10.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071341269999999997 2023-07-02T14:06:10.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361389+0000", 2023-07-02T14:06:10.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9140000000000001e-05 2023-07-02T14:06:10.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361428+0000", 2023-07-02T14:06:10.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056418470000000002 2023-07-02T14:06:10.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.367070+0000", 2023-07-02T14:06:10.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1863000000000002e-05 2023-07-02T14:06:10.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:10.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.367112+0000", 2023-07-02T14:06:10.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00027357599999999998 2023-07-02T14:06:10.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:10.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.367386+0000", 2023-07-02T14:06:10.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020277680000000001 2023-07-02T14:06:10.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:10.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369413+0000", 2023-07-02T14:06:10.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029075699999999999 2023-07-02T14:06:10.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:10.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372321+0000", 2023-07-02T14:06:10.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1955999999999997e-05 2023-07-02T14:06:10.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:10.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372353+0000", 2023-07-02T14:06:10.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4938999999999995e-05 2023-07-02T14:06:10.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:10.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372428+0000", 2023-07-02T14:06:10.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12347108800000001 2023-07-02T14:06:10.464 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.464 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.465 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.465 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18557 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:10.465 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:10.465 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.8651427800000002, 2023-07-02T14:06:10.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13120991400000001, 2023-07-02T14:06:10.465 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:10.466 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:10.466 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:10.466 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:10.466 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:10.466 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18557 2023-07-02T14:06:10.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.466 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:10.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:10.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:10.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:10.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:10.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:10.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:06:10.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:10.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:10.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8870000000000001e-06 2023-07-02T14:06:10.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:10.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248971+0000", 2023-07-02T14:06:10.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9009999999999996e-06 2023-07-02T14:06:10.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:10.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248974+0000", 2023-07-02T14:06:10.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9809999999999997e-06 2023-07-02T14:06:10.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:10.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248978+0000", 2023-07-02T14:06:10.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00076989199999999999 2023-07-02T14:06:10.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249748+0000", 2023-07-02T14:06:10.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8245999999999999e-05 2023-07-02T14:06:10.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249787+0000", 2023-07-02T14:06:10.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066332969999999998 2023-07-02T14:06:10.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256420+0000", 2023-07-02T14:06:10.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2566e-05 2023-07-02T14:06:10.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256452+0000", 2023-07-02T14:06:10.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012364714000000001 2023-07-02T14:06:10.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268817+0000", 2023-07-02T14:06:10.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5113000000000001e-05 2023-07-02T14:06:10.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268852+0000", 2023-07-02T14:06:10.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010993899999999999 2023-07-02T14:06:10.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279846+0000", 2023-07-02T14:06:10.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2144999999999998e-05 2023-07-02T14:06:10.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279878+0000", 2023-07-02T14:06:10.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011778958000000001 2023-07-02T14:06:10.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291657+0000", 2023-07-02T14:06:10.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9592000000000001e-05 2023-07-02T14:06:10.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291687+0000", 2023-07-02T14:06:10.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081107220000000008 2023-07-02T14:06:10.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299798+0000", 2023-07-02T14:06:10.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0290999999999998e-05 2023-07-02T14:06:10.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299828+0000", 2023-07-02T14:06:10.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071584489999999999 2023-07-02T14:06:10.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306986+0000", 2023-07-02T14:06:10.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0755999999999999e-05 2023-07-02T14:06:10.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307047+0000", 2023-07-02T14:06:10.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057110260000000001 2023-07-02T14:06:10.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312758+0000", 2023-07-02T14:06:10.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5145999999999999e-05 2023-07-02T14:06:10.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312793+0000", 2023-07-02T14:06:10.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004758361 2023-07-02T14:06:10.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317552+0000", 2023-07-02T14:06:10.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9190000000000002e-05 2023-07-02T14:06:10.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317601+0000", 2023-07-02T14:06:10.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050329420000000003 2023-07-02T14:06:10.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322634+0000", 2023-07-02T14:06:10.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5737999999999998e-05 2023-07-02T14:06:10.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322719+0000", 2023-07-02T14:06:10.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064995679999999998 2023-07-02T14:06:10.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329219+0000", 2023-07-02T14:06:10.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0908000000000003e-05 2023-07-02T14:06:10.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329250+0000", 2023-07-02T14:06:10.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062072029999999997 2023-07-02T14:06:10.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335457+0000", 2023-07-02T14:06:10.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9471999999999998e-05 2023-07-02T14:06:10.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335487+0000", 2023-07-02T14:06:10.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097780239999999997 2023-07-02T14:06:10.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345265+0000", 2023-07-02T14:06:10.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0982999999999994e-05 2023-07-02T14:06:10.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345336+0000", 2023-07-02T14:06:10.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046378799999999996 2023-07-02T14:06:10.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349973+0000", 2023-07-02T14:06:10.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1069999999999999e-05 2023-07-02T14:06:10.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350005+0000", 2023-07-02T14:06:10.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042766870000000004 2023-07-02T14:06:10.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354281+0000", 2023-07-02T14:06:10.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1155000000000002e-05 2023-07-02T14:06:10.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354312+0000", 2023-07-02T14:06:10.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071436820000000002 2023-07-02T14:06:10.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361456+0000", 2023-07-02T14:06:10.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9054000000000002e-05 2023-07-02T14:06:10.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361505+0000", 2023-07-02T14:06:10.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072301320000000002 2023-07-02T14:06:10.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368735+0000", 2023-07-02T14:06:10.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1013999999999994e-05 2023-07-02T14:06:10.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368816+0000", 2023-07-02T14:06:10.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036582490000000001 2023-07-02T14:06:10.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372475+0000", 2023-07-02T14:06:10.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4652000000000004e-05 2023-07-02T14:06:10.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:10.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372539+0000", 2023-07-02T14:06:10.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00031374199999999999 2023-07-02T14:06:10.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:10.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372853+0000", 2023-07-02T14:06:10.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030071299999999998 2023-07-02T14:06:10.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:10.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.375860+0000", 2023-07-02T14:06:10.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041897749999999997 2023-07-02T14:06:10.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:10.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380050+0000", 2023-07-02T14:06:10.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8331000000000003e-05 2023-07-02T14:06:10.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:10.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380088+0000", 2023-07-02T14:06:10.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0626999999999998e-05 2023-07-02T14:06:10.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:10.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380179+0000", 2023-07-02T14:06:10.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13120991400000001 2023-07-02T14:06:10.496 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.497 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.497 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.497 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18558 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:10.497 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:10.497 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.8651260780000003, 2023-07-02T14:06:10.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13755567899999999, 2023-07-02T14:06:10.498 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:10.498 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:10.498 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:10.498 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:10.498 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:10.498 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18558 2023-07-02T14:06:10.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.498 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:10.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:10.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:10.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:10.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:10.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:10.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:10.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:10.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248985+0000", 2023-07-02T14:06:10.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8630000000000001e-06 2023-07-02T14:06:10.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:10.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248989+0000", 2023-07-02T14:06:10.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0750000000000003e-06 2023-07-02T14:06:10.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:10.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248995+0000", 2023-07-02T14:06:10.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3290000000000004e-06 2023-07-02T14:06:10.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:10.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248999+0000", 2023-07-02T14:06:10.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00080656300000000005 2023-07-02T14:06:10.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249806+0000", 2023-07-02T14:06:10.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2040999999999999e-05 2023-07-02T14:06:10.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249848+0000", 2023-07-02T14:06:10.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066312419999999999 2023-07-02T14:06:10.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256479+0000", 2023-07-02T14:06:10.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3348e-05 2023-07-02T14:06:10.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256513+0000", 2023-07-02T14:06:10.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012391531000000001 2023-07-02T14:06:10.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268904+0000", 2023-07-02T14:06:10.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.347e-05 2023-07-02T14:06:10.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268978+0000", 2023-07-02T14:06:10.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010929772000000001 2023-07-02T14:06:10.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279907+0000", 2023-07-02T14:06:10.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2669999999999997e-05 2023-07-02T14:06:10.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279940+0000", 2023-07-02T14:06:10.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011770332999999999 2023-07-02T14:06:10.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291710+0000", 2023-07-02T14:06:10.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1207000000000001e-05 2023-07-02T14:06:10.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291742+0000", 2023-07-02T14:06:10.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081242199999999997 2023-07-02T14:06:10.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299866+0000", 2023-07-02T14:06:10.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4650000000000002e-05 2023-07-02T14:06:10.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299900+0000", 2023-07-02T14:06:10.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071810270000000004 2023-07-02T14:06:10.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307081+0000", 2023-07-02T14:06:10.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4378000000000002e-05 2023-07-02T14:06:10.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307116+0000", 2023-07-02T14:06:10.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057031909999999998 2023-07-02T14:06:10.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312819+0000", 2023-07-02T14:06:10.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1612999999999997e-05 2023-07-02T14:06:10.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312851+0000", 2023-07-02T14:06:10.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048172470000000002 2023-07-02T14:06:10.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317668+0000", 2023-07-02T14:06:10.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3328e-05 2023-07-02T14:06:10.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317741+0000", 2023-07-02T14:06:10.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050117290000000004 2023-07-02T14:06:10.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322753+0000", 2023-07-02T14:06:10.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9872000000000001e-05 2023-07-02T14:06:10.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322783+0000", 2023-07-02T14:06:10.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064910610000000002 2023-07-02T14:06:10.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329274+0000", 2023-07-02T14:06:10.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0097e-05 2023-07-02T14:06:10.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329304+0000", 2023-07-02T14:06:10.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062163879999999998 2023-07-02T14:06:10.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335520+0000", 2023-07-02T14:06:10.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9513e-05 2023-07-02T14:06:10.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335550+0000", 2023-07-02T14:06:10.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098209969999999997 2023-07-02T14:06:10.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345371+0000", 2023-07-02T14:06:10.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9862999999999999e-05 2023-07-02T14:06:10.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345401+0000", 2023-07-02T14:06:10.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046552030000000001 2023-07-02T14:06:10.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350056+0000", 2023-07-02T14:06:10.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8751999999999993e-05 2023-07-02T14:06:10.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350145+0000", 2023-07-02T14:06:10.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041981450000000003 2023-07-02T14:06:10.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354343+0000", 2023-07-02T14:06:10.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1501000000000002e-05 2023-07-02T14:06:10.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354374+0000", 2023-07-02T14:06:10.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071551740000000003 2023-07-02T14:06:10.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361530+0000", 2023-07-02T14:06:10.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5156e-05 2023-07-02T14:06:10.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361565+0000", 2023-07-02T14:06:10.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072918569999999997 2023-07-02T14:06:10.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368857+0000", 2023-07-02T14:06:10.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5770999999999999e-05 2023-07-02T14:06:10.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368902+0000", 2023-07-02T14:06:10.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057210569999999999 2023-07-02T14:06:10.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374623+0000", 2023-07-02T14:06:10.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3266999999999998e-05 2023-07-02T14:06:10.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374667+0000", 2023-07-02T14:06:10.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055479570000000001 2023-07-02T14:06:10.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380215+0000", 2023-07-02T14:06:10.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0067e-05 2023-07-02T14:06:10.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:10.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380265+0000", 2023-07-02T14:06:10.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00043082299999999999 2023-07-02T14:06:10.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:10.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380696+0000", 2023-07-02T14:06:10.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024853240000000001 2023-07-02T14:06:10.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:10.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.383181+0000", 2023-07-02T14:06:10.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032601280000000002 2023-07-02T14:06:10.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:10.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386441+0000", 2023-07-02T14:06:10.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9748e-05 2023-07-02T14:06:10.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:10.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386471+0000", 2023-07-02T14:06:10.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0531000000000006e-05 2023-07-02T14:06:10.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:10.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386541+0000", 2023-07-02T14:06:10.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13755567899999999 2023-07-02T14:06:10.530 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.530 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.530 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.530 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18559 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:10.530 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:10.531 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.8651034739999997, 2023-07-02T14:06:10.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.146184591, 2023-07-02T14:06:10.531 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:10.531 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:10.531 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:10.531 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:10.531 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:10.531 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18559 2023-07-02T14:06:10.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.532 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:10.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:10.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:10.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:10.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:10.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:10.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:10.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:10.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:10.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.726e-06 2023-07-02T14:06:10.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:10.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249011+0000", 2023-07-02T14:06:10.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7939999999999997e-06 2023-07-02T14:06:10.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:10.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249018+0000", 2023-07-02T14:06:10.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3470000000000001e-06 2023-07-02T14:06:10.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:10.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249023+0000", 2023-07-02T14:06:10.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00084402900000000005 2023-07-02T14:06:10.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249867+0000", 2023-07-02T14:06:10.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9050000000000001e-05 2023-07-02T14:06:10.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249906+0000", 2023-07-02T14:06:10.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066311039999999996 2023-07-02T14:06:10.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256537+0000", 2023-07-02T14:06:10.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2790000000000003e-05 2023-07-02T14:06:10.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256570+0000", 2023-07-02T14:06:10.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012436129000000001 2023-07-02T14:06:10.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269006+0000", 2023-07-02T14:06:10.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0173999999999999e-05 2023-07-02T14:06:10.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269046+0000", 2023-07-02T14:06:10.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010924698 2023-07-02T14:06:10.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279971+0000", 2023-07-02T14:06:10.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3831000000000002e-05 2023-07-02T14:06:10.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280004+0000", 2023-07-02T14:06:10.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011795912 2023-07-02T14:06:10.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291800+0000", 2023-07-02T14:06:10.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5613999999999997e-05 2023-07-02T14:06:10.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291886+0000", 2023-07-02T14:06:10.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080395999999999992 2023-07-02T14:06:10.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299925+0000", 2023-07-02T14:06:10.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0694999999999997e-05 2023-07-02T14:06:10.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299956+0000", 2023-07-02T14:06:10.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072049449999999999 2023-07-02T14:06:10.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307161+0000", 2023-07-02T14:06:10.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5523000000000001e-05 2023-07-02T14:06:10.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307247+0000", 2023-07-02T14:06:10.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056331929999999999 2023-07-02T14:06:10.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312880+0000", 2023-07-02T14:06:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1646000000000002e-05 2023-07-02T14:06:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312911+0000", 2023-07-02T14:06:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048627540000000004 2023-07-02T14:06:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317774+0000", 2023-07-02T14:06:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2008000000000003e-05 2023-07-02T14:06:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317806+0000", 2023-07-02T14:06:10.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050055689999999996 2023-07-02T14:06:10.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322812+0000", 2023-07-02T14:06:10.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3442999999999997e-05 2023-07-02T14:06:10.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322855+0000", 2023-07-02T14:06:10.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006472813 2023-07-02T14:06:10.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329328+0000", 2023-07-02T14:06:10.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0128e-05 2023-07-02T14:06:10.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329358+0000", 2023-07-02T14:06:10.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062156490000000002 2023-07-02T14:06:10.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335574+0000", 2023-07-02T14:06:10.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2357999999999997e-05 2023-07-02T14:06:10.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335606+0000", 2023-07-02T14:06:10.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098318409999999992 2023-07-02T14:06:10.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345438+0000", 2023-07-02T14:06:10.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1174000000000007e-05 2023-07-02T14:06:10.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345509+0000", 2023-07-02T14:06:10.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004669851 2023-07-02T14:06:10.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350179+0000", 2023-07-02T14:06:10.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1559000000000001e-05 2023-07-02T14:06:10.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350211+0000", 2023-07-02T14:06:10.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041865740000000002 2023-07-02T14:06:10.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354397+0000", 2023-07-02T14:06:10.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9850000000000001e-05 2023-07-02T14:06:10.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354427+0000", 2023-07-02T14:06:10.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071608449999999999 2023-07-02T14:06:10.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361588+0000", 2023-07-02T14:06:10.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8346000000000002e-05 2023-07-02T14:06:10.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361626+0000", 2023-07-02T14:06:10.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073175009999999997 2023-07-02T14:06:10.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368944+0000", 2023-07-02T14:06:10.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2879000000000005e-05 2023-07-02T14:06:10.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369017+0000", 2023-07-02T14:06:10.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056850690000000001 2023-07-02T14:06:10.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374702+0000", 2023-07-02T14:06:10.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7693e-05 2023-07-02T14:06:10.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374739+0000", 2023-07-02T14:06:10.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076925149999999996 2023-07-02T14:06:10.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382432+0000", 2023-07-02T14:06:10.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3259e-05 2023-07-02T14:06:10.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382475+0000", 2023-07-02T14:06:10.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041005110000000003 2023-07-02T14:06:10.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386576+0000", 2023-07-02T14:06:10.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2074999999999999e-05 2023-07-02T14:06:10.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:10.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386618+0000", 2023-07-02T14:06:10.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000233692 2023-07-02T14:06:10.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:10.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386851+0000", 2023-07-02T14:06:10.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024022269999999998 2023-07-02T14:06:10.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:10.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.389254+0000", 2023-07-02T14:06:10.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058506469999999996 2023-07-02T14:06:10.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:10.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395104+0000", 2023-07-02T14:06:10.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8807000000000003e-05 2023-07-02T14:06:10.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:10.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395143+0000", 2023-07-02T14:06:10.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9616000000000001e-05 2023-07-02T14:06:10.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:10.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395193+0000", 2023-07-02T14:06:10.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.146184591 2023-07-02T14:06:10.564 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.564 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.565 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.565 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18560 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:10.565 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:10.565 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.8650848509999998, 2023-07-02T14:06:10.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.15647551900000001, 2023-07-02T14:06:10.565 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:10.566 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:10.566 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:10.566 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:10.566 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:10.566 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18560 2023-07-02T14:06:10.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.566 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:10.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:10.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:10.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:10.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:10.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:10.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:10.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:10.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249026+0000", 2023-07-02T14:06:10.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3419999999999999e-06 2023-07-02T14:06:10.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:10.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249030+0000", 2023-07-02T14:06:10.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7869999999999998e-06 2023-07-02T14:06:10.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:10.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249035+0000", 2023-07-02T14:06:10.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9639999999999997e-06 2023-07-02T14:06:10.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:10.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249039+0000", 2023-07-02T14:06:10.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00088477599999999999 2023-07-02T14:06:10.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249924+0000", 2023-07-02T14:06:10.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7333999999999999e-05 2023-07-02T14:06:10.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249962+0000", 2023-07-02T14:06:10.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066323409999999999 2023-07-02T14:06:10.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256594+0000", 2023-07-02T14:06:10.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2595999999999998e-05 2023-07-02T14:06:10.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256627+0000", 2023-07-02T14:06:10.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012449894 2023-07-02T14:06:10.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269076+0000", 2023-07-02T14:06:10.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3942999999999997e-05 2023-07-02T14:06:10.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269110+0000", 2023-07-02T14:06:10.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010923884 2023-07-02T14:06:10.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280034+0000", 2023-07-02T14:06:10.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7774000000000001e-05 2023-07-02T14:06:10.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280072+0000", 2023-07-02T14:06:10.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011850027000000001 2023-07-02T14:06:10.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291922+0000", 2023-07-02T14:06:10.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2128000000000001e-05 2023-07-02T14:06:10.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291954+0000", 2023-07-02T14:06:10.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080261150000000003 2023-07-02T14:06:10.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299980+0000", 2023-07-02T14:06:10.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6322999999999998e-05 2023-07-02T14:06:10.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300017+0000", 2023-07-02T14:06:10.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072683039999999997 2023-07-02T14:06:10.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307285+0000", 2023-07-02T14:06:10.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3309000000000002e-05 2023-07-02T14:06:10.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307318+0000", 2023-07-02T14:06:10.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056178649999999997 2023-07-02T14:06:10.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312936+0000", 2023-07-02T14:06:10.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1866000000000003e-05 2023-07-02T14:06:10.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312968+0000", 2023-07-02T14:06:10.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048789970000000004 2023-07-02T14:06:10.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317847+0000", 2023-07-02T14:06:10.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0223999999999996e-05 2023-07-02T14:06:10.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317927+0000", 2023-07-02T14:06:10.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049543649999999996 2023-07-02T14:06:10.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322882+0000", 2023-07-02T14:06:10.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1399000000000002e-05 2023-07-02T14:06:10.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322913+0000", 2023-07-02T14:06:10.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006534214 2023-07-02T14:06:10.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329447+0000", 2023-07-02T14:06:10.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9649999999999998e-05 2023-07-02T14:06:10.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329527+0000", 2023-07-02T14:06:10.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061026839999999997 2023-07-02T14:06:10.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335629+0000", 2023-07-02T14:06:10.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9611000000000002e-05 2023-07-02T14:06:10.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335659+0000", 2023-07-02T14:06:10.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098851549999999996 2023-07-02T14:06:10.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345544+0000", 2023-07-02T14:06:10.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0587999999999999e-05 2023-07-02T14:06:10.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345575+0000", 2023-07-02T14:06:10.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046790859999999998 2023-07-02T14:06:10.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350254+0000", 2023-07-02T14:06:10.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0284000000000002e-05 2023-07-02T14:06:10.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350304+0000", 2023-07-02T14:06:10.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041560540000000002 2023-07-02T14:06:10.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354460+0000", 2023-07-02T14:06:10.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2427000000000004e-05 2023-07-02T14:06:10.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354523+0000", 2023-07-02T14:06:10.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071268770000000002 2023-07-02T14:06:10.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361650+0000", 2023-07-02T14:06:10.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0026999999999999e-05 2023-07-02T14:06:10.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361680+0000", 2023-07-02T14:06:10.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073733080000000003 2023-07-02T14:06:10.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369053+0000", 2023-07-02T14:06:10.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7635999999999997e-05 2023-07-02T14:06:10.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369091+0000", 2023-07-02T14:06:10.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056908059999999996 2023-07-02T14:06:10.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374781+0000", 2023-07-02T14:06:10.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4656000000000002e-05 2023-07-02T14:06:10.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374816+0000", 2023-07-02T14:06:10.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077037809999999998 2023-07-02T14:06:10.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382520+0000", 2023-07-02T14:06:10.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7858e-05 2023-07-02T14:06:10.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382588+0000", 2023-07-02T14:06:10.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057884019999999998 2023-07-02T14:06:10.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388376+0000", 2023-07-02T14:06:10.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4831000000000001e-05 2023-07-02T14:06:10.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388461+0000", 2023-07-02T14:06:10.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067708480000000003 2023-07-02T14:06:10.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395232+0000", 2023-07-02T14:06:10.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7948000000000002e-05 2023-07-02T14:06:10.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:10.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395290+0000", 2023-07-02T14:06:10.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00027672599999999998 2023-07-02T14:06:10.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:10.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395566+0000", 2023-07-02T14:06:10.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025705569999999998 2023-07-02T14:06:10.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:10.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.398137+0000", 2023-07-02T14:06:10.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072257099999999998 2023-07-02T14:06:10.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:10.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405363+0000", 2023-07-02T14:06:10.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3887999999999998e-05 2023-07-02T14:06:10.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:10.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405397+0000", 2023-07-02T14:06:10.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000105769 2023-07-02T14:06:10.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:10.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405502+0000", 2023-07-02T14:06:10.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.15647551900000001 2023-07-02T14:06:10.600 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.600 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.600 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.601 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18561 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:10.601 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:10.601 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.8650644869999997, 2023-07-02T14:06:10.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744, 2023-07-02T14:06:10.601 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:10.601 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:10.602 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:10.602 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:10.602 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:10.602 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18561 2023-07-02T14:06:10.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.602 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:10.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:10.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:10.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:10.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:10.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:10.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:10.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.604 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:10.604 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:10.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.563e-06 2023-07-02T14:06:10.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.604 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:10.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249050+0000", 2023-07-02T14:06:10.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4500000000000001e-06 2023-07-02T14:06:10.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:10.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249057+0000", 2023-07-02T14:06:10.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1860000000000002e-06 2023-07-02T14:06:10.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:10.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249061+0000", 2023-07-02T14:06:10.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00091153399999999998 2023-07-02T14:06:10.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249972+0000", 2023-07-02T14:06:10.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7156999999999997e-05 2023-07-02T14:06:10.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.250010+0000", 2023-07-02T14:06:10.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006641816 2023-07-02T14:06:10.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256651+0000", 2023-07-02T14:06:10.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1968999999999998e-05 2023-07-02T14:06:10.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256683+0000", 2023-07-02T14:06:10.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012463952 2023-07-02T14:06:10.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269147+0000", 2023-07-02T14:06:10.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2855000000000006e-05 2023-07-02T14:06:10.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269220+0000", 2023-07-02T14:06:10.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010870418999999999 2023-07-02T14:06:10.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280091+0000", 2023-07-02T14:06:10.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1921999999999997e-05 2023-07-02T14:06:10.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280123+0000", 2023-07-02T14:06:10.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011861682 2023-07-02T14:06:10.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291984+0000", 2023-07-02T14:06:10.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5259000000000006e-05 2023-07-02T14:06:10.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.292059+0000", 2023-07-02T14:06:10.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079709359999999996 2023-07-02T14:06:10.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300030+0000", 2023-07-02T14:06:10.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0729999999999998e-05 2023-07-02T14:06:10.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300071+0000", 2023-07-02T14:06:10.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072785690000000004 2023-07-02T14:06:10.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307350+0000", 2023-07-02T14:06:10.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1948999999999999e-05 2023-07-02T14:06:10.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.615 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307412+0000", 2023-07-02T14:06:10.615 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055785779999999998 2023-07-02T14:06:10.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.615 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312990+0000", 2023-07-02T14:06:10.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0957000000000002e-05 2023-07-02T14:06:10.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.616 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.616 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.616 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.313021+0000", 2023-07-02T14:06:10.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049246400000000001 2023-07-02T14:06:10.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.616 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317946+0000", 2023-07-02T14:06:10.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9346e-05 2023-07-02T14:06:10.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317975+0000", 2023-07-02T14:06:10.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049649469999999999 2023-07-02T14:06:10.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322940+0000", 2023-07-02T14:06:10.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9017000000000002e-05 2023-07-02T14:06:10.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322989+0000", 2023-07-02T14:06:10.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065907029999999998 2023-07-02T14:06:10.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329580+0000", 2023-07-02T14:06:10.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5879999999999999e-05 2023-07-02T14:06:10.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329656+0000", 2023-07-02T14:06:10.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060166960000000002 2023-07-02T14:06:10.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335672+0000", 2023-07-02T14:06:10.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8963e-05 2023-07-02T14:06:10.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335701+0000", 2023-07-02T14:06:10.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098989560000000004 2023-07-02T14:06:10.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345600+0000", 2023-07-02T14:06:10.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5395999999999995e-05 2023-07-02T14:06:10.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345666+0000", 2023-07-02T14:06:10.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046589530000000004 2023-07-02T14:06:10.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350325+0000", 2023-07-02T14:06:10.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2459000000000002e-05 2023-07-02T14:06:10.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350357+0000", 2023-07-02T14:06:10.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042054290000000001 2023-07-02T14:06:10.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354563+0000", 2023-07-02T14:06:10.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5612999999999999e-05 2023-07-02T14:06:10.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.625 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354598+0000", 2023-07-02T14:06:10.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070967449999999998 2023-07-02T14:06:10.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.626 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361695+0000", 2023-07-02T14:06:10.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9314000000000001e-05 2023-07-02T14:06:10.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.626 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361724+0000", 2023-07-02T14:06:10.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007391028 2023-07-02T14:06:10.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369115+0000", 2023-07-02T14:06:10.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9163000000000002e-05 2023-07-02T14:06:10.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369184+0000", 2023-07-02T14:06:10.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056532500000000003 2023-07-02T14:06:10.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374838+0000", 2023-07-02T14:06:10.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5297999999999999e-05 2023-07-02T14:06:10.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374883+0000", 2023-07-02T14:06:10.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077292189999999998 2023-07-02T14:06:10.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382612+0000", 2023-07-02T14:06:10.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1102000000000001e-05 2023-07-02T14:06:10.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382653+0000", 2023-07-02T14:06:10.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058515989999999999 2023-07-02T14:06:10.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388505+0000", 2023-07-02T14:06:10.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9196000000000006e-05 2023-07-02T14:06:10.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388574+0000", 2023-07-02T14:06:10.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084245339999999991 2023-07-02T14:06:10.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.396999+0000", 2023-07-02T14:06:10.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0596999999999998e-05 2023-07-02T14:06:10.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.397039+0000", 2023-07-02T14:06:10.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084911339999999991 2023-07-02T14:06:10.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405530+0000", 2023-07-02T14:06:10.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7555000000000001e-05 2023-07-02T14:06:10.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:10.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405578+0000", 2023-07-02T14:06:10.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000229607 2023-07-02T14:06:10.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:10.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405808+0000", 2023-07-02T14:06:10.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026919940000000001 2023-07-02T14:06:10.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:10.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.408500+0000", 2023-07-02T14:06:10.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053065059999999999 2023-07-02T14:06:10.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:10.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413806+0000", 2023-07-02T14:06:10.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0490000000000001e-05 2023-07-02T14:06:10.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:10.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413837+0000", 2023-07-02T14:06:10.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000100386 2023-07-02T14:06:10.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:10.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413937+0000", 2023-07-02T14:06:10.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744 2023-07-02T14:06:10.637 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.638 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.638 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.638 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18879 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e177)", 2023-07-02T14:06:10.638 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:06.898926+0000", 2023-07-02T14:06:10.638 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.2151854470000001, 2023-07-02T14:06:10.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.108448455, 2023-07-02T14:06:10.639 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:10.639 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:10.639 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:10.639 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:10.639 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:10.639 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18879 2023-07-02T14:06:10.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.639 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:10.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:10.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.898926+0000", 2023-07-02T14:06:10.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:10.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:10.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.898926+0000", 2023-07-02T14:06:10.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:10.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:10.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.898926+0000", 2023-07-02T14:06:10.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.45e-06 2023-07-02T14:06:10.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:10.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.898929+0000", 2023-07-02T14:06:10.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4110000000000004e-06 2023-07-02T14:06:10.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:10.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.898934+0000", 2023-07-02T14:06:10.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.168e-06 2023-07-02T14:06:10.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:10.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.898938+0000", 2023-07-02T14:06:10.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015109999999999999 2023-07-02T14:06:10.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.899089+0000", 2023-07-02T14:06:10.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3013000000000003e-05 2023-07-02T14:06:10.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.899132+0000", 2023-07-02T14:06:10.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033983960000000001 2023-07-02T14:06:10.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.902530+0000", 2023-07-02T14:06:10.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2097000000000001e-05 2023-07-02T14:06:10.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.902562+0000", 2023-07-02T14:06:10.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013082125 2023-07-02T14:06:10.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.915644+0000", 2023-07-02T14:06:10.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1718999999999998e-05 2023-07-02T14:06:10.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.915686+0000", 2023-07-02T14:06:10.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068310189999999998 2023-07-02T14:06:10.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.922517+0000", 2023-07-02T14:06:10.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1585999999999999e-05 2023-07-02T14:06:10.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.922549+0000", 2023-07-02T14:06:10.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014397978000000001 2023-07-02T14:06:10.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.936947+0000", 2023-07-02T14:06:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2971000000000002e-05 2023-07-02T14:06:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.937030+0000", 2023-07-02T14:06:10.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070590799999999997 2023-07-02T14:06:10.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.944089+0000", 2023-07-02T14:06:10.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1683000000000001e-05 2023-07-02T14:06:10.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.944120+0000", 2023-07-02T14:06:10.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089819119999999999 2023-07-02T14:06:10.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.953102+0000", 2023-07-02T14:06:10.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7537999999999999e-05 2023-07-02T14:06:10.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.953140+0000", 2023-07-02T14:06:10.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016871374000000001 2023-07-02T14:06:10.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.970011+0000", 2023-07-02T14:06:10.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0485000000000002e-05 2023-07-02T14:06:10.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.970062+0000", 2023-07-02T14:06:10.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013052505000000001 2023-07-02T14:06:10.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.983114+0000", 2023-07-02T14:06:10.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8034999999999997e-05 2023-07-02T14:06:10.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:10.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.983162+0000", 2023-07-02T14:06:10.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013268611 2023-07-02T14:06:10.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:10.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.996431+0000", 2023-07-02T14:06:10.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0791000000000001e-05 2023-07-02T14:06:10.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:10.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.996492+0000", 2023-07-02T14:06:10.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00036019099999999999 2023-07-02T14:06:10.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:10.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.996852+0000", 2023-07-02T14:06:10.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004127917 2023-07-02T14:06:10.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:10.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:07.000980+0000", 2023-07-02T14:06:10.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061381760000000004 2023-07-02T14:06:10.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:10.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:07.007118+0000", 2023-07-02T14:06:10.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1156999999999997e-05 2023-07-02T14:06:10.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:10.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:07.007189+0000", 2023-07-02T14:06:10.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000185639 2023-07-02T14:06:10.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:10.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:10.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:10.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:07.007375+0000", 2023-07-02T14:06:10.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.108448455 2023-07-02T14:06:10.659 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.659 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.660 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.660 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:10.660 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:10.660 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:10.661 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:06:10.661 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 1 2023-07-02T14:06:10.662 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:06:10.662 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:10.660+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:10.663 INFO:tasks.workunit.client.0.smithi163.stderr:+ sync b3fbb028-e267-4d25-a750-93ba18a2316a-mnt 2023-07-02T14:06:10.735 INFO:tasks.workunit.client.0.smithi163.stderr:+ rm -rf b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4 b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4.tar.gz 2023-07-02T14:06:10.762 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:10.762+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:10.801 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:10.802 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:10.802 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:10.802 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:10.812 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:06:10.863 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:10.862+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:10.935 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:06:10.936 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:06:10.936 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:10.946 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:06:10.987 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:10.986+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:11.065 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:11.064+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:11.112 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:11.112 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:06:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:06:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18643 2.5 e176/170 2:a9ace6f8:::rbd_data.125de1449605.0000000000000187:head v 176'2314, mlcod=173'2311)", 2023-07-02T14:06:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:06:11.113 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.6840700689999997, 2023-07-02T14:06:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015027297, 2023-07-02T14:06:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:06:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:06:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.115 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426547+0000", 2023-07-02T14:06:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056864430000000002 2023-07-02T14:06:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432234+0000", 2023-07-02T14:06:11.116 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.415e-05 2023-07-02T14:06:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432248+0000", 2023-07-02T14:06:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7490000000000001e-06 2023-07-02T14:06:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432255+0000", 2023-07-02T14:06:11.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3915000000000002e-05 2023-07-02T14:06:11.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432299+0000", 2023-07-02T14:06:11.118 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1214000000000001e-05 2023-07-02T14:06:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432320+0000", 2023-07-02T14:06:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0092114950000000001 2023-07-02T14:06:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.441531+0000", 2023-07-02T14:06:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4422999999999998e-05 2023-07-02T14:06:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.441576+0000", 2023-07-02T14:06:11.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015027297 2023-07-02T14:06:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18644 2.2 e176/171 2:44e6ed94:::rbd_data.125de1449605.0000000000000185:head v 176'1626, mlcod=176'1623)", 2023-07-02T14:06:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.428712+0000", 2023-07-02T14:06:11.121 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.6819063280000002, 2023-07-02T14:06:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012273601, 2023-07-02T14:06:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.428712+0000", 2023-07-02T14:06:11.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.428712+0000", 2023-07-02T14:06:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.428711+0000", 2023-07-02T14:06:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017963639999999999 2023-07-02T14:06:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.430507+0000", 2023-07-02T14:06:11.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0897e-05 2023-07-02T14:06:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.430528+0000", 2023-07-02T14:06:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7940000000000008e-06 2023-07-02T14:06:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.125 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.430537+0000", 2023-07-02T14:06:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00034051099999999999 2023-07-02T14:06:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.430878+0000", 2023-07-02T14:06:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5058000000000001e-05 2023-07-02T14:06:11.126 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.430893+0000", 2023-07-02T14:06:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010060174 2023-07-02T14:06:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.440953+0000", 2023-07-02T14:06:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3064000000000001e-05 2023-07-02T14:06:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.440986+0000", 2023-07-02T14:06:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012273601 2023-07-02T14:06:11.128 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18680 2.2 e176/171 2:5fd2923a:::rbd_data.125de1449605.000000000000018c:head v 176'1639, mlcod=176'1637)", 2023-07-02T14:06:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.489711+0000", 2023-07-02T14:06:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.6209076270000002, 2023-07-02T14:06:11.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014407402, 2023-07-02T14:06:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.489711+0000", 2023-07-02T14:06:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.489711+0000", 2023-07-02T14:06:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.131 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.489709+0000", 2023-07-02T14:06:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001991792 2023-07-02T14:06:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491701+0000", 2023-07-02T14:06:11.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7323000000000002e-05 2023-07-02T14:06:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491738+0000", 2023-07-02T14:06:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2483999999999999e-05 2023-07-02T14:06:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491751+0000", 2023-07-02T14:06:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7896000000000003e-05 2023-07-02T14:06:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491809+0000", 2023-07-02T14:06:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0185000000000001e-05 2023-07-02T14:06:11.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491829+0000", 2023-07-02T14:06:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012250901999999999 2023-07-02T14:06:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.504080+0000", 2023-07-02T14:06:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:06:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.504118+0000", 2023-07-02T14:06:11.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014407402 2023-07-02T14:06:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18681 2.2 e176/171 2:5ecc17e8:::rbd_data.125de1449605.000000000000018d:head v 176'1640, mlcod=176'1637)", 2023-07-02T14:06:11.137 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.497048+0000", 2023-07-02T14:06:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.6135704869999996, 2023-07-02T14:06:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012067603999999999, 2023-07-02T14:06:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.497048+0000", 2023-07-02T14:06:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.497048+0000", 2023-07-02T14:06:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:11.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.497047+0000", 2023-07-02T14:06:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021639139999999999 2023-07-02T14:06:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.499211+0000", 2023-07-02T14:06:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7983e-05 2023-07-02T14:06:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.499239+0000", 2023-07-02T14:06:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6110999999999999e-05 2023-07-02T14:06:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.499255+0000", 2023-07-02T14:06:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0260999999999998e-05 2023-07-02T14:06:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.499315+0000", 2023-07-02T14:06:11.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0349e-05 2023-07-02T14:06:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.499345+0000", 2023-07-02T14:06:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097133659999999993 2023-07-02T14:06:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.509059+0000", 2023-07-02T14:06:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7092999999999997e-05 2023-07-02T14:06:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.509116+0000", 2023-07-02T14:06:11.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012067603999999999 2023-07-02T14:06:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18690 2.2 e176/171 2:416a1700:::rbd_data.125de1449605.000000000000018f:head v 176'1644, mlcod=176'1641)", 2023-07-02T14:06:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.523233+0000", 2023-07-02T14:06:11.145 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.5873854959999996, 2023-07-02T14:06:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01191801, 2023-07-02T14:06:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.523233+0000", 2023-07-02T14:06:11.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.523233+0000", 2023-07-02T14:06:11.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:11.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.523231+0000", 2023-07-02T14:06:11.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014368580000000001 2023-07-02T14:06:11.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.524668+0000", 2023-07-02T14:06:11.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.929e-05 2023-07-02T14:06:11.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.524688+0000", 2023-07-02T14:06:11.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9709999999999999e-06 2023-07-02T14:06:11.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.524698+0000", 2023-07-02T14:06:11.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112815 2023-07-02T14:06:11.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.524810+0000", 2023-07-02T14:06:11.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7217999999999999e-05 2023-07-02T14:06:11.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.524838+0000", 2023-07-02T14:06:11.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010267129 2023-07-02T14:06:11.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.535105+0000", 2023-07-02T14:06:11.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6399000000000001e-05 2023-07-02T14:06:11.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.535151+0000", 2023-07-02T14:06:11.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01191801 2023-07-02T14:06:11.153 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.153 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.153 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.153 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18694 2.5 e176/170 2:ad0b6442:::rbd_data.125de1449605.0000000000000192:head v 176'2317, mlcod=176'2314)", 2023-07-02T14:06:11.153 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.529519+0000", 2023-07-02T14:06:11.154 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.581099129, 2023-07-02T14:06:11.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011113384, 2023-07-02T14:06:11.154 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.154 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.154 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.529519+0000", 2023-07-02T14:06:11.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.529519+0000", 2023-07-02T14:06:11.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:11.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.529518+0000", 2023-07-02T14:06:11.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033832699999999999 2023-07-02T14:06:11.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532902+0000", 2023-07-02T14:06:11.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1599e-05 2023-07-02T14:06:11.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532913+0000", 2023-07-02T14:06:11.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3319999999999999e-06 2023-07-02T14:06:11.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532919+0000", 2023-07-02T14:06:11.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2502000000000001e-05 2023-07-02T14:06:11.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532961+0000", 2023-07-02T14:06:11.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5031000000000001e-05 2023-07-02T14:06:11.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532976+0000", 2023-07-02T14:06:11.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076043589999999998 2023-07-02T14:06:11.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.540580+0000", 2023-07-02T14:06:11.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2391e-05 2023-07-02T14:06:11.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.540633+0000", 2023-07-02T14:06:11.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011113384 2023-07-02T14:06:11.160 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.161 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.161 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.161 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18723 2.5 e176/170 2:a564ac9c:::rbd_data.125de1449605.0000000000000196:head v 176'2324, mlcod=176'2321)", 2023-07-02T14:06:11.161 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.568716+0000", 2023-07-02T14:06:11.161 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.5419023200000002, 2023-07-02T14:06:11.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013584191000000001, 2023-07-02T14:06:11.162 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.162 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.162 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.568716+0000", 2023-07-02T14:06:11.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.568716+0000", 2023-07-02T14:06:11.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:11.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.568715+0000", 2023-07-02T14:06:11.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039052919999999999 2023-07-02T14:06:11.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.572620+0000", 2023-07-02T14:06:11.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5108000000000001e-05 2023-07-02T14:06:11.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.572645+0000", 2023-07-02T14:06:11.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0495e-05 2023-07-02T14:06:11.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.572656+0000", 2023-07-02T14:06:11.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6385999999999999e-05 2023-07-02T14:06:11.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.572702+0000", 2023-07-02T14:06:11.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8578e-05 2023-07-02T14:06:11.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.572731+0000", 2023-07-02T14:06:11.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095207799999999995 2023-07-02T14:06:11.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.582251+0000", 2023-07-02T14:06:11.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9021999999999999e-05 2023-07-02T14:06:11.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.582300+0000", 2023-07-02T14:06:11.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013584191000000001 2023-07-02T14:06:11.169 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.169 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.169 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.169 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18859 2.5 e177/170 2:a5de3e1e:::rbd_data.125de1449605.0000000000000022:head v 177'2343, mlcod=177'2341)", 2023-07-02T14:06:11.169 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.883954+0000", 2023-07-02T14:06:11.169 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.2266648419999999, 2023-07-02T14:06:11.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013529319999999999, 2023-07-02T14:06:11.170 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.170 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.170 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.883954+0000", 2023-07-02T14:06:11.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.883954+0000", 2023-07-02T14:06:11.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:11.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.883953+0000", 2023-07-02T14:06:11.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003004048 2023-07-02T14:06:11.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.886957+0000", 2023-07-02T14:06:11.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0279999999999999e-05 2023-07-02T14:06:11.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.886977+0000", 2023-07-02T14:06:11.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2190000000000004e-06 2023-07-02T14:06:11.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.886985+0000", 2023-07-02T14:06:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7524e-05 2023-07-02T14:06:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.887043+0000", 2023-07-02T14:06:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6378000000000001e-05 2023-07-02T14:06:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.887069+0000", 2023-07-02T14:06:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010370551 2023-07-02T14:06:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.897440+0000", 2023-07-02T14:06:11.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3152999999999998e-05 2023-07-02T14:06:11.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.897483+0000", 2023-07-02T14:06:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013529319999999999 2023-07-02T14:06:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.177 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18881 2.5 e177/170 2:b1731d2f:::rbd_data.125de1449605.0000000000000024:head v 177'2350, mlcod=177'2347)", 2023-07-02T14:06:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:06:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.2047042450000003, 2023-07-02T14:06:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020820709999999999, 2023-07-02T14:06:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:06:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:06:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905913+0000", 2023-07-02T14:06:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0080191519999999999 2023-07-02T14:06:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913932+0000", 2023-07-02T14:06:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3757999999999999e-05 2023-07-02T14:06:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913956+0000", 2023-07-02T14:06:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4009999999999997e-06 2023-07-02T14:06:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913965+0000", 2023-07-02T14:06:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3383e-05 2023-07-02T14:06:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914018+0000", 2023-07-02T14:06:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7674e-05 2023-07-02T14:06:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914046+0000", 2023-07-02T14:06:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012656555999999999 2023-07-02T14:06:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.926702+0000", 2023-07-02T14:06:11.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2891000000000001e-05 2023-07-02T14:06:11.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.926735+0000", 2023-07-02T14:06:11.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020820709999999999 2023-07-02T14:06:11.185 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.185 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.185 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.185 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18885 2.4 e177/169 2:3625364f:::rbd_data.125de1449605.0000000000000026:head v 177'1716, mlcod=177'1714)", 2023-07-02T14:06:11.185 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.914022+0000", 2023-07-02T14:06:11.186 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.1965964419999997, 2023-07-02T14:06:11.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013176468, 2023-07-02T14:06:11.186 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.186 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.186 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914022+0000", 2023-07-02T14:06:11.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914022+0000", 2023-07-02T14:06:11.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:11.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914021+0000", 2023-07-02T14:06:11.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002349889 2023-07-02T14:06:11.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916371+0000", 2023-07-02T14:06:11.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5821e-05 2023-07-02T14:06:11.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916387+0000", 2023-07-02T14:06:11.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3200000000000006e-06 2023-07-02T14:06:11.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916396+0000", 2023-07-02T14:06:11.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010131820000000001 2023-07-02T14:06:11.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.917409+0000", 2023-07-02T14:06:11.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3589000000000001e-05 2023-07-02T14:06:11.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.917433+0000", 2023-07-02T14:06:11.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097323819999999995 2023-07-02T14:06:11.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.927165+0000", 2023-07-02T14:06:11.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3422e-05 2023-07-02T14:06:11.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.927199+0000", 2023-07-02T14:06:11.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013176468 2023-07-02T14:06:11.193 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.193 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.193 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.193 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18883 2.5 e177/170 2:a5de3e1e:::rbd_data.125de1449605.0000000000000022:head v 177'2351, mlcod=177'2347)", 2023-07-02T14:06:11.193 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.916456+0000", 2023-07-02T14:06:11.193 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.1941624800000001, 2023-07-02T14:06:11.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010715634, 2023-07-02T14:06:11.194 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.194 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.194 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916456+0000", 2023-07-02T14:06:11.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916456+0000", 2023-07-02T14:06:11.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:11.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916455+0000", 2023-07-02T14:06:11.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2898000000000006e-05 2023-07-02T14:06:11.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916528+0000", 2023-07-02T14:06:11.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2599999999999994e-06 2023-07-02T14:06:11.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916537+0000", 2023-07-02T14:06:11.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1209999999999998e-06 2023-07-02T14:06:11.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916542+0000", 2023-07-02T14:06:11.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032458040000000001 2023-07-02T14:06:11.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.919788+0000", 2023-07-02T14:06:11.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5001999999999999e-05 2023-07-02T14:06:11.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.919813+0000", 2023-07-02T14:06:11.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073193750000000004 2023-07-02T14:06:11.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.927132+0000", 2023-07-02T14:06:11.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9486000000000001e-05 2023-07-02T14:06:11.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.927172+0000", 2023-07-02T14:06:11.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010715634 2023-07-02T14:06:11.200 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.200 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.201 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.201 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18893 2.5 e177/170 2:bbff50a9:::rbd_data.125de1449605.0000000000000027:head v 177'2354, mlcod=177'2351)", 2023-07-02T14:06:11.201 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.933879+0000", 2023-07-02T14:06:11.201 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.1767394859999998, 2023-07-02T14:06:11.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012696278, 2023-07-02T14:06:11.202 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.202 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.202 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.933879+0000", 2023-07-02T14:06:11.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.933879+0000", 2023-07-02T14:06:11.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:11.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.933878+0000", 2023-07-02T14:06:11.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035422819999999999 2023-07-02T14:06:11.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.937421+0000", 2023-07-02T14:06:11.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7059999999999999e-06 2023-07-02T14:06:11.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.937430+0000", 2023-07-02T14:06:11.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5379999999999996e-06 2023-07-02T14:06:11.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.937435+0000", 2023-07-02T14:06:11.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8078999999999999e-05 2023-07-02T14:06:11.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.937473+0000", 2023-07-02T14:06:11.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4100999999999998e-05 2023-07-02T14:06:11.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.937497+0000", 2023-07-02T14:06:11.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.009041248 2023-07-02T14:06:11.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.946538+0000", 2023-07-02T14:06:11.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6986999999999997e-05 2023-07-02T14:06:11.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.946575+0000", 2023-07-02T14:06:11.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012696278 2023-07-02T14:06:11.208 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.208 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.209 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.209 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18935 2.4 e177/169 2:3e650bab:::rbd_data.125de1449605.0000000000000031:head v 177'1727, mlcod=177'1721)", 2023-07-02T14:06:11.209 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.994084+0000", 2023-07-02T14:06:11.209 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.1165347800000003, 2023-07-02T14:06:11.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011588876, 2023-07-02T14:06:11.210 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.210 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.210 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.994084+0000", 2023-07-02T14:06:11.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.994084+0000", 2023-07-02T14:06:11.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:11.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.994083+0000", 2023-07-02T14:06:11.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011146350000000001 2023-07-02T14:06:11.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.995197+0000", 2023-07-02T14:06:11.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4032999999999998e-05 2023-07-02T14:06:11.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.995231+0000", 2023-07-02T14:06:11.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9760000000000001e-06 2023-07-02T14:06:11.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.995240+0000", 2023-07-02T14:06:11.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00029382799999999999 2023-07-02T14:06:11.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.995534+0000", 2023-07-02T14:06:11.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6772999999999999e-05 2023-07-02T14:06:11.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.995561+0000", 2023-07-02T14:06:11.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010068725000000001 2023-07-02T14:06:11.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.005630+0000", 2023-07-02T14:06:11.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2930999999999999e-05 2023-07-02T14:06:11.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.005673+0000", 2023-07-02T14:06:11.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011588876 2023-07-02T14:06:11.216 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.216 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.217 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.217 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19061 2.5 e177/170 2:a79a9f1f:::rbd_data.125de1449605.0000000000000039:head v 177'2369, mlcod=177'2367)", 2023-07-02T14:06:11.217 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.252771+0000", 2023-07-02T14:06:11.217 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.8578480420000001, 2023-07-02T14:06:11.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010293825, 2023-07-02T14:06:11.218 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.218 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.218 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.252771+0000", 2023-07-02T14:06:11.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.252771+0000", 2023-07-02T14:06:11.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:11.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.252770+0000", 2023-07-02T14:06:11.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002567518 2023-07-02T14:06:11.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.255337+0000", 2023-07-02T14:06:11.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1004000000000003e-05 2023-07-02T14:06:11.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.255368+0000", 2023-07-02T14:06:11.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5399999999999996e-06 2023-07-02T14:06:11.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.255377+0000", 2023-07-02T14:06:11.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4950000000000001e-05 2023-07-02T14:06:11.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.255432+0000", 2023-07-02T14:06:11.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6458e-05 2023-07-02T14:06:11.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.255458+0000", 2023-07-02T14:06:11.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0075801430000000001 2023-07-02T14:06:11.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.263038+0000", 2023-07-02T14:06:11.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6211e-05 2023-07-02T14:06:11.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.263064+0000", 2023-07-02T14:06:11.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010293825 2023-07-02T14:06:11.225 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.225 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.225 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.225 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19073 2.5 e177/170 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:head v 177'2378, mlcod=177'2376)", 2023-07-02T14:06:11.225 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:06:11.225 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.8369910620000001, 2023-07-02T14:06:11.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018435625000000001, 2023-07-02T14:06:11.226 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.226 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.226 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:06:11.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:06:11.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:11.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273626+0000", 2023-07-02T14:06:11.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046035659999999999 2023-07-02T14:06:11.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278230+0000", 2023-07-02T14:06:11.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9386e-05 2023-07-02T14:06:11.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278259+0000", 2023-07-02T14:06:11.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1972e-05 2023-07-02T14:06:11.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278271+0000", 2023-07-02T14:06:11.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017532680000000001 2023-07-02T14:06:11.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280025+0000", 2023-07-02T14:06:11.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2367000000000001e-05 2023-07-02T14:06:11.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280047+0000", 2023-07-02T14:06:11.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011987028 2023-07-02T14:06:11.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.292034+0000", 2023-07-02T14:06:11.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9266000000000001e-05 2023-07-02T14:06:11.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.292063+0000", 2023-07-02T14:06:11.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018435625000000001 2023-07-02T14:06:11.232 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.233 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.233 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.233 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19075 2.5 e177/170 2:adf18789:::rbd_data.125de1449605.000000000000003c:head v 177'2379, mlcod=177'2376)", 2023-07-02T14:06:11.233 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.280379+0000", 2023-07-02T14:06:11.233 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.8302399949999999, 2023-07-02T14:06:11.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014230578000000001, 2023-07-02T14:06:11.234 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.234 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.234 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280379+0000", 2023-07-02T14:06:11.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280379+0000", 2023-07-02T14:06:11.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999804 2023-07-02T14:06:11.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280359+0000", 2023-07-02T14:06:11.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047641209999999996 2023-07-02T14:06:11.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.285123+0000", 2023-07-02T14:06:11.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3711e-05 2023-07-02T14:06:11.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.285137+0000", 2023-07-02T14:06:11.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.277e-06 2023-07-02T14:06:11.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.285144+0000", 2023-07-02T14:06:11.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00041800499999999999 2023-07-02T14:06:11.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.285562+0000", 2023-07-02T14:06:11.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6143000000000001e-05 2023-07-02T14:06:11.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.285578+0000", 2023-07-02T14:06:11.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089876420000000005 2023-07-02T14:06:11.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.294566+0000", 2023-07-02T14:06:11.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3466e-05 2023-07-02T14:06:11.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.294609+0000", 2023-07-02T14:06:11.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014230578000000001 2023-07-02T14:06:11.240 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.241 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.241 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.241 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19118 2.5 e177/170 2:be65b5f1:::rbd_data.125de1449605.0000000000000043:head v 177'2389, mlcod=177'2386)", 2023-07-02T14:06:11.241 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.342749+0000", 2023-07-02T14:06:11.241 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.767869658, 2023-07-02T14:06:11.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011707482, 2023-07-02T14:06:11.242 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.242 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.242 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.342749+0000", 2023-07-02T14:06:11.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.342749+0000", 2023-07-02T14:06:11.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:11.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.342748+0000", 2023-07-02T14:06:11.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045726380000000004 2023-07-02T14:06:11.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.347320+0000", 2023-07-02T14:06:11.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1033e-05 2023-07-02T14:06:11.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.347341+0000", 2023-07-02T14:06:11.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1297e-05 2023-07-02T14:06:11.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.347353+0000", 2023-07-02T14:06:11.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9991000000000002e-05 2023-07-02T14:06:11.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.347413+0000", 2023-07-02T14:06:11.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2135000000000001e-05 2023-07-02T14:06:11.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.347435+0000", 2023-07-02T14:06:11.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0069823029999999996 2023-07-02T14:06:11.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.354417+0000", 2023-07-02T14:06:11.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9447999999999999e-05 2023-07-02T14:06:11.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.354456+0000", 2023-07-02T14:06:11.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011707482 2023-07-02T14:06:11.249 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.249 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.249 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.249 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19193 2.2 e177/171 2:5382aedc:::rbd_data.125de1449605.000000000000004e:head v 177'1704, mlcod=177'1701)", 2023-07-02T14:06:11.249 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.473454+0000", 2023-07-02T14:06:11.249 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.6371641779999999, 2023-07-02T14:06:11.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012446847, 2023-07-02T14:06:11.250 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.250 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.250 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.473454+0000", 2023-07-02T14:06:11.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.473454+0000", 2023-07-02T14:06:11.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:11.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.473452+0000", 2023-07-02T14:06:11.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001241418 2023-07-02T14:06:11.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.474694+0000", 2023-07-02T14:06:11.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6421999999999999e-05 2023-07-02T14:06:11.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.474720+0000", 2023-07-02T14:06:11.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9219999999999992e-06 2023-07-02T14:06:11.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.474730+0000", 2023-07-02T14:06:11.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000926395 2023-07-02T14:06:11.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475656+0000", 2023-07-02T14:06:11.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5822000000000001e-05 2023-07-02T14:06:11.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475682+0000", 2023-07-02T14:06:11.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01016774 2023-07-02T14:06:11.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485850+0000", 2023-07-02T14:06:11.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1266e-05 2023-07-02T14:06:11.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485901+0000", 2023-07-02T14:06:11.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012446847 2023-07-02T14:06:11.257 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.257 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.257 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.258 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19196 2.4 e177/169 2:2c79e900:::rbd_data.125de1449605.000000000000004a:head v 177'1774, mlcod=177'1772)", 2023-07-02T14:06:11.258 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.475437+0000", 2023-07-02T14:06:11.258 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.635181904, 2023-07-02T14:06:11.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013623751, 2023-07-02T14:06:11.258 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.258 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.258 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475437+0000", 2023-07-02T14:06:11.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475437+0000", 2023-07-02T14:06:11.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:06:11.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475434+0000", 2023-07-02T14:06:11.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3578999999999994e-05 2023-07-02T14:06:11.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475518+0000", 2023-07-02T14:06:11.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1090999999999998e-05 2023-07-02T14:06:11.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475539+0000", 2023-07-02T14:06:11.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4860000000000003e-06 2023-07-02T14:06:11.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475549+0000", 2023-07-02T14:06:11.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000107015 2023-07-02T14:06:11.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475656+0000", 2023-07-02T14:06:11.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6625e-05 2023-07-02T14:06:11.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475682+0000", 2023-07-02T14:06:11.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013334037 2023-07-02T14:06:11.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489016+0000", 2023-07-02T14:06:11.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4141999999999999e-05 2023-07-02T14:06:11.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489060+0000", 2023-07-02T14:06:11.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013623751 2023-07-02T14:06:11.265 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.265 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.265 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.266 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19221 2.4 e177/169 2:25643af7:::rbd_data.125de1449605.0000000000000052:head v 177'1782, mlcod=177'1777)", 2023-07-02T14:06:11.266 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.527928+0000", 2023-07-02T14:06:11.266 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.5826905660000001, 2023-07-02T14:06:11.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010683292, 2023-07-02T14:06:11.266 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.266 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.266 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.527928+0000", 2023-07-02T14:06:11.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.527928+0000", 2023-07-02T14:06:11.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:11.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.527927+0000", 2023-07-02T14:06:11.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5786999999999999e-05 2023-07-02T14:06:11.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.527953+0000", 2023-07-02T14:06:11.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4833e-05 2023-07-02T14:06:11.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.527967+0000", 2023-07-02T14:06:11.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.32e-06 2023-07-02T14:06:11.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.527976+0000", 2023-07-02T14:06:11.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0053743699999999998 2023-07-02T14:06:11.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.533350+0000", 2023-07-02T14:06:11.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1956999999999999e-05 2023-07-02T14:06:11.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.533382+0000", 2023-07-02T14:06:11.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051965850000000001 2023-07-02T14:06:11.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.538579+0000", 2023-07-02T14:06:11.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2743999999999997e-05 2023-07-02T14:06:11.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.538611+0000", 2023-07-02T14:06:11.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010683292 2023-07-02T14:06:11.273 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.273 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.273 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.273 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.273 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.273 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:11.275 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:11.274+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:11.275 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:06:11.375 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:11.374+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:11.440 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:11.441 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:11.441 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:11.441 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:11.451 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:06:11.476 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:11.475+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:11.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:06:11.537 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap49 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap50 b3fbb028-e267-4d25-a750-93ba18a2316a@snap50.diff 2023-07-02T14:06:11.576 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:11.575+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:11.626 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:11.626+0000 7f4d605e8700 0 -- 172.21.15.163:0/3443768441 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f4d44004b10 msgr2=0x7f4d44004f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:11.630 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:11.630+0000 7f4d5fde7700 0 -- 172.21.15.163:0/3443768441 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7f4d4400b200 msgr2=0x7f4d4400d650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:11.651 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:11.651 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:11.651 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:06:11.651 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:06:11.651 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:11.671 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:06:11.677 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:11.676+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:11.778 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:11.777+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:11.829 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:11.829 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:06:11.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:06:11.829 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:11.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.829 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:06:11.830 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:11.830 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 182.37519362500001, 2023-07-02T14:06:11.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:06:11.830 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.830 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:11.830 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:11.830 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:11.831 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:11.831 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:06:11.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.831 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:11.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:11.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:11.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:06:11.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:06:11.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:06:11.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:06:11.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:06:11.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:06:11.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:06:11.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:06:11.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:06:11.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:06:11.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:11.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:06:11.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:06:11.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:06:11.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:06:11.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:06:11.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:06:11.838 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.838 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.838 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.838 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:06:11.838 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:11.838 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 181.50450061699999, 2023-07-02T14:06:11.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:06:11.839 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.839 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:11.839 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:11.839 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:06:11.839 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:06:11.839 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:06:11.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.840 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:11.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:11.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:11.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:06:11.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:06:11.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:06:11.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:06:11.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:06:11.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:06:11.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:06:11.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:06:11.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:06:11.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:06:11.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:11.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:06:11.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:06:11.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:06:11.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:06:11.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:06:11.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:06:11.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:06:11.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:06:11.847 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.847 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.847 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.847 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:11.847 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:11.848 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 181.504375896, 2023-07-02T14:06:11.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:06:11.848 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.848 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:11.848 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:11.848 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:11.848 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:11.848 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:06:11.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.849 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:11.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:11.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:11.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:06:11.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:06:11.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:06:11.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:06:11.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:06:11.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:06:11.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:06:11.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:06:11.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:06:11.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:06:11.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:11.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:06:11.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:06:11.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:06:11.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:06:11.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:06:11.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:06:11.855 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.855 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.855 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.856 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:11.856 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:11.856 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 181.49752601200001, 2023-07-02T14:06:11.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:06:11.856 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.856 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:11.857 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:11.857 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:11.857 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:11.857 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:06:11.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.857 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:11.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:11.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:06:11.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:06:11.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:06:11.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:06:11.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:06:11.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:06:11.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:06:11.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:06:11.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:06:11.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:06:11.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:06:11.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:11.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:06:11.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:06:11.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:06:11.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:06:11.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:06:11.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:06:11.864 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.864 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.864 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.864 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:11.865 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:11.865 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 181.497463087, 2023-07-02T14:06:11.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:06:11.865 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.865 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:11.865 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:11.865 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:11.865 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:11.866 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:06:11.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.866 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:11.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.867 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:11.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:11.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.867 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:06:11.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:06:11.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:06:11.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:06:11.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:06:11.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:06:11.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:06:11.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:06:11.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:06:11.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:06:11.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:11.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:06:11.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:06:11.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:06:11.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:06:11.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:06:11.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:06:11.872 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.872 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.873 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.873 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:11.873 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:11.873 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 181.49716266600001, 2023-07-02T14:06:11.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:06:11.873 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.874 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.874 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:11.874 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:11.874 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:11.874 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:06:11.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.874 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:11.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:11.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:11.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:06:11.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:06:11.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:06:11.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:06:11.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:06:11.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:06:11.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:06:11.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:06:11.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:06:11.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:06:11.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:11.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:06:11.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:06:11.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:06:11.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:06:11.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:06:11.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:06:11.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:06:11.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:06:11.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:06:11.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:11.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:06:11.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:06:11.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:11.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:06:11.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:06:11.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:11.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:06:11.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:06:11.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:06:11.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:06:11.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:06:11.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:06:11.885 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.885 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.885 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.885 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:06:11.885 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:11.886 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 181.496638166, 2023-07-02T14:06:11.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:06:11.886 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.886 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:11.886 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:11.886 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:11.886 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:11.886 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:06:11.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.887 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:11.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:11.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:11.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:11.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:06:11.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:06:11.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:06:11.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:06:11.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:06:11.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:06:11.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:06:11.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:06:11.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:06:11.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:11.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:06:11.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:06:11.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:06:11.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:06:11.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:06:11.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:06:11.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:06:11.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:06:11.894 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.895 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.895 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.895 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:06:11.895 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:11.895 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 154.37261904100001, 2023-07-02T14:06:11.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:06:11.896 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.896 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:11.896 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:11.896 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:11.896 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:11.896 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:06:11.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.897 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:11.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:11.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:11.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:06:11.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:06:11.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:06:11.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:06:11.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:06:11.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:06:11.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:06:11.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:06:11.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:06:11.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:06:11.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:11.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:06:11.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:06:11.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:06:11.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:06:11.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:06:11.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:06:11.903 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.903 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.903 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.904 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:06:11.904 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:11.904 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 154.04713633700001, 2023-07-02T14:06:11.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:06:11.904 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.904 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:11.905 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:11.905 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:11.905 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:11.905 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:06:11.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.905 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:11.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:11.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:11.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:06:11.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:06:11.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:06:11.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:06:11.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:06:11.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:06:11.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:06:11.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:06:11.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:06:11.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:06:11.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:11.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:06:11.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:06:11.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:06:11.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:06:11.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:06:11.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:06:11.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:11.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:06:11.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:06:11.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:11.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:06:11.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:06:11.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:11.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:06:11.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:06:11.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:11.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:06:11.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:06:11.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:06:11.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:06:11.915 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.915 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.915 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.916 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:17 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 0~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:11.916 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:11.916 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 60.628860555000003, 2023-07-02T14:06:11.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997, 2023-07-02T14:06:11.916 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.916 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:11.916 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:11.917 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:11.917 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:11.917 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17 2023-07-02T14:06:11.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.917 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:11.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:11.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:06:11.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198431+0000", 2023-07-02T14:06:11.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2815e-05 2023-07-02T14:06:11.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198443+0000", 2023-07-02T14:06:11.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2569999999999999e-05 2023-07-02T14:06:11.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198466+0000", 2023-07-02T14:06:11.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0223e-05 2023-07-02T14:06:11.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198476+0000", 2023-07-02T14:06:11.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1792999999999997e-05 2023-07-02T14:06:11.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:06:11.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4249999999999998e-06 2023-07-02T14:06:11.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:11.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198536+0000", 2023-07-02T14:06:11.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91888663500000001 2023-07-02T14:06:11.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117423+0000", 2023-07-02T14:06:11.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223594 2023-07-02T14:06:11.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117647+0000", 2023-07-02T14:06:11.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033832900000000001 2023-07-02T14:06:11.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117985+0000", 2023-07-02T14:06:11.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997 2023-07-02T14:06:11.924 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.924 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.924 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.925 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:18 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 65536~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:11.925 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:11.925 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 60.628803343999998, 2023-07-02T14:06:11.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998, 2023-07-02T14:06:11.925 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.925 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:11.926 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:11.926 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:11.926 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:11.926 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18 2023-07-02T14:06:11.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.926 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:11.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:11.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:11.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:11.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7600000000000002e-06 2023-07-02T14:06:11.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198494+0000", 2023-07-02T14:06:11.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7840000000000004e-06 2023-07-02T14:06:11.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198502+0000", 2023-07-02T14:06:11.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6459999999999998e-06 2023-07-02T14:06:11.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198508+0000", 2023-07-02T14:06:11.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7395999999999997e-05 2023-07-02T14:06:11.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198565+0000", 2023-07-02T14:06:11.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6800000000000004e-06 2023-07-02T14:06:11.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:11.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198572+0000", 2023-07-02T14:06:11.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91965896800000002 2023-07-02T14:06:11.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118231+0000", 2023-07-02T14:06:11.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5212999999999998e-05 2023-07-02T14:06:11.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118326+0000", 2023-07-02T14:06:11.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000238538 2023-07-02T14:06:11.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118565+0000", 2023-07-02T14:06:11.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998 2023-07-02T14:06:11.933 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.933 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.934 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.934 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:19 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 131072~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:11.934 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:11.934 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 60.628779119999997, 2023-07-02T14:06:11.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002, 2023-07-02T14:06:11.935 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.935 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:11.935 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:11.935 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:11.935 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:11.935 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 19 2023-07-02T14:06:11.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.935 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:11.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:11.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:11.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:11.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1990000000000002e-06 2023-07-02T14:06:11.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198517+0000", 2023-07-02T14:06:11.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8789999999999997e-06 2023-07-02T14:06:11.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198524+0000", 2023-07-02T14:06:11.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:06:11.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:06:11.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5545999999999999e-05 2023-07-02T14:06:11.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:11.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2540000000000003e-06 2023-07-02T14:06:11.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:11.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198598+0000", 2023-07-02T14:06:11.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.919983263 2023-07-02T14:06:11.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118581+0000", 2023-07-02T14:06:11.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6878999999999994e-05 2023-07-02T14:06:11.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118648+0000", 2023-07-02T14:06:11.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149732 2023-07-02T14:06:11.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118798+0000", 2023-07-02T14:06:11.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002 2023-07-02T14:06:11.942 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.943 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.943 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.943 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:20 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 262144~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:11.943 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:11.943 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 60.628760155999998, 2023-07-02T14:06:11.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998, 2023-07-02T14:06:11.944 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.944 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:11.944 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:11.944 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:11.944 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:11.944 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20 2023-07-02T14:06:11.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.945 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:11.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:11.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:11.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:11.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.732e-06 2023-07-02T14:06:11.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198537+0000", 2023-07-02T14:06:11.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5259999999999999e-06 2023-07-02T14:06:11.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198543+0000", 2023-07-02T14:06:11.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.121e-06 2023-07-02T14:06:11.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198547+0000", 2023-07-02T14:06:11.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9591000000000005e-05 2023-07-02T14:06:11.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:06:11.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3120000000000004e-06 2023-07-02T14:06:11.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:11.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198621+0000", 2023-07-02T14:06:11.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92020789400000003 2023-07-02T14:06:11.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118829+0000", 2023-07-02T14:06:11.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022393386000000001 2023-07-02T14:06:11.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141223+0000", 2023-07-02T14:06:11.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000329866 2023-07-02T14:06:11.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141552+0000", 2023-07-02T14:06:11.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998 2023-07-02T14:06:11.952 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.952 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.952 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.952 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:21 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 196608~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:11.952 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:11.953 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 60.628732521000003, 2023-07-02T14:06:11.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995, 2023-07-02T14:06:11.953 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.953 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:11.953 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:11.953 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:11.953 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:11.953 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21 2023-07-02T14:06:11.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.954 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:11.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:11.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:11.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:11.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0169999999999999e-05 2023-07-02T14:06:11.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198571+0000", 2023-07-02T14:06:11.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3479999999999998e-06 2023-07-02T14:06:11.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198578+0000", 2023-07-02T14:06:11.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8439999999999997e-06 2023-07-02T14:06:11.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198583+0000", 2023-07-02T14:06:11.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8294999999999997e-05 2023-07-02T14:06:11.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198641+0000", 2023-07-02T14:06:11.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8120000000000001e-06 2023-07-02T14:06:11.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:11.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:06:11.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94293828499999999 2023-07-02T14:06:11.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141583+0000", 2023-07-02T14:06:11.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014607300000000001 2023-07-02T14:06:11.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141729+0000", 2023-07-02T14:06:11.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00044906600000000003 2023-07-02T14:06:11.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142178+0000", 2023-07-02T14:06:11.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995 2023-07-02T14:06:11.961 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.961 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.961 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.961 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:22 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 327680~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:11.962 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:11.962 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 60.628699091000001, 2023-07-02T14:06:11.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004, 2023-07-02T14:06:11.962 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.962 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:11.962 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:11.962 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:11.963 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:11.963 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 22 2023-07-02T14:06:11.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.963 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:11.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:11.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:11.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:11.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4850000000000001e-06 2023-07-02T14:06:11.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198597+0000", 2023-07-02T14:06:11.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:06:11.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198605+0000", 2023-07-02T14:06:11.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4839999999999998e-06 2023-07-02T14:06:11.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198609+0000", 2023-07-02T14:06:11.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3100999999999999e-05 2023-07-02T14:06:11.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198662+0000", 2023-07-02T14:06:11.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9099999999999998e-06 2023-07-02T14:06:11.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:11.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198666+0000", 2023-07-02T14:06:11.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.943539503 2023-07-02T14:06:11.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142206+0000", 2023-07-02T14:06:11.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012255999999999999 2023-07-02T14:06:11.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142328+0000", 2023-07-02T14:06:11.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208956 2023-07-02T14:06:11.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142537+0000", 2023-07-02T14:06:11.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004 2023-07-02T14:06:11.970 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.970 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.970 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.971 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:23 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 393216~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:11.971 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:11.971 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 60.628679110999997, 2023-07-02T14:06:11.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348, 2023-07-02T14:06:11.971 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.971 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:11.972 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:11.972 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:11.972 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:11.972 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23 2023-07-02T14:06:11.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.972 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:11.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:11.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:11.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:11.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8439999999999998e-06 2023-07-02T14:06:11.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:06:11.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0440000000000001e-06 2023-07-02T14:06:11.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198623+0000", 2023-07-02T14:06:11.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4480000000000002e-06 2023-07-02T14:06:11.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198628+0000", 2023-07-02T14:06:11.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5843e-05 2023-07-02T14:06:11.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:11.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.89e-06 2023-07-02T14:06:11.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:11.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198688+0000", 2023-07-02T14:06:11.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94389171999999999 2023-07-02T14:06:11.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142580+0000", 2023-07-02T14:06:11.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3892000000000004e-05 2023-07-02T14:06:11.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142644+0000", 2023-07-02T14:06:11.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017608000000000001 2023-07-02T14:06:11.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142820+0000", 2023-07-02T14:06:11.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348 2023-07-02T14:06:11.979 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.979 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.980 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.980 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:24 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 458752~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:11.980 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:11.980 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 60.628651036999997, 2023-07-02T14:06:11.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004, 2023-07-02T14:06:11.981 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.981 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:11.981 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:11.981 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:11.981 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:11.981 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 24 2023-07-02T14:06:11.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.981 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:11.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:11.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:11.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198642+0000", 2023-07-02T14:06:11.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.253e-06 2023-07-02T14:06:11.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:06:11.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1289999999999998e-06 2023-07-02T14:06:11.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198652+0000", 2023-07-02T14:06:11.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4510000000000002e-06 2023-07-02T14:06:11.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198657+0000", 2023-07-02T14:06:11.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9131000000000001e-05 2023-07-02T14:06:11.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198706+0000", 2023-07-02T14:06:11.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8890000000000004e-06 2023-07-02T14:06:11.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:11.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198710+0000", 2023-07-02T14:06:11.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94415412200000004 2023-07-02T14:06:11.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142864+0000", 2023-07-02T14:06:11.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8047000000000006e-05 2023-07-02T14:06:11.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142952+0000", 2023-07-02T14:06:11.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192967 2023-07-02T14:06:11.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143145+0000", 2023-07-02T14:06:11.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004 2023-07-02T14:06:11.989 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.989 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.989 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.989 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:25 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 524288~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:11.989 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:11.989 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 60.628632242000002, 2023-07-02T14:06:11.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001, 2023-07-02T14:06:11.990 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.990 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:11.990 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:11.990 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:11.990 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:11.990 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25 2023-07-02T14:06:11.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.991 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:11.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:11.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:11.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:11.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:11.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:11.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:11.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:11.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:11.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.615e-06 2023-07-02T14:06:11.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:11.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198664+0000", 2023-07-02T14:06:11.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4759999999999997e-06 2023-07-02T14:06:11.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:11.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198669+0000", 2023-07-02T14:06:11.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5999999999999998e-06 2023-07-02T14:06:11.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:11.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198673+0000", 2023-07-02T14:06:11.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4256999999999999e-05 2023-07-02T14:06:11.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198727+0000", 2023-07-02T14:06:11.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840000000000001e-06 2023-07-02T14:06:11.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:11.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198731+0000", 2023-07-02T14:06:11.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94444241900000003 2023-07-02T14:06:11.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:11.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143173+0000", 2023-07-02T14:06:11.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6839000000000002e-05 2023-07-02T14:06:11.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:11.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143240+0000", 2023-07-02T14:06:11.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017053399999999999 2023-07-02T14:06:11.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:11.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143411+0000", 2023-07-02T14:06:11.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001 2023-07-02T14:06:11.998 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.998 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:11.998 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:11.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:11.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:11.998 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:26 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 589824~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:11.999 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:11.999 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 60.628609091000001, 2023-07-02T14:06:11.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997, 2023-07-02T14:06:11.999 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:11.999 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:11.999 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:11.999 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:12.000 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:12.000 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 26 2023-07-02T14:06:12.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.000 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:12.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:12.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:12.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:12.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:12.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:12.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:12.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:12.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:12.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0520000000000002e-06 2023-07-02T14:06:12.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:12.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198687+0000", 2023-07-02T14:06:12.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1880000000000002e-06 2023-07-02T14:06:12.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:12.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198695+0000", 2023-07-02T14:06:12.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9960000000000004e-06 2023-07-02T14:06:12.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:12.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198699+0000", 2023-07-02T14:06:12.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2330999999999998e-05 2023-07-02T14:06:12.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:12.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198742+0000", 2023-07-02T14:06:12.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7950000000000001e-06 2023-07-02T14:06:12.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:12.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198745+0000", 2023-07-02T14:06:12.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94469398199999999 2023-07-02T14:06:12.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:12.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143439+0000", 2023-07-02T14:06:12.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6048e-05 2023-07-02T14:06:12.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:12.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143525+0000", 2023-07-02T14:06:12.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017016700000000001 2023-07-02T14:06:12.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:12.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143696+0000", 2023-07-02T14:06:12.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997 2023-07-02T14:06:12.007 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:12.007 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:12.007 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:12.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.008 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.6579.0:11 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected e171)", 2023-07-02T14:06:12.008 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:12.008 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 14.668810604999999, 2023-07-02T14:06:12.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998, 2023-07-02T14:06:12.008 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:12.008 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:12.008 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:12.009 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.6579", 2023-07-02T14:06:12.009 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/554403467", 2023-07-02T14:06:12.009 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11 2023-07-02T14:06:12.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.009 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:12.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:12.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:12.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:12.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:12.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:12.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:12.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:12.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158481+0000", 2023-07-02T14:06:12.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:06:12.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:12.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158487+0000", 2023-07-02T14:06:12.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7630000000000006e-06 2023-07-02T14:06:12.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:12.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158496+0000", 2023-07-02T14:06:12.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5100000000000001e-06 2023-07-02T14:06:12.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:12.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158504+0000", 2023-07-02T14:06:12.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9397000000000002e-05 2023-07-02T14:06:12.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:12.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158553+0000", 2023-07-02T14:06:12.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5810000000000003e-06 2023-07-02T14:06:12.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:12.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158562+0000", 2023-07-02T14:06:12.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93893079499999998 2023-07-02T14:06:12.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:12.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097493+0000", 2023-07-02T14:06:12.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015799299999999999 2023-07-02T14:06:12.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:12.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097651+0000", 2023-07-02T14:06:12.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187746 2023-07-02T14:06:12.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:12.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:12.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:12.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097839+0000", 2023-07-02T14:06:12.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998 2023-07-02T14:06:12.016 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:12.016 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:12.016 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:12.017 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:12.017 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:12.017 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:12.017 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 2] out_osds: [5, 4] dead_osds: [] live_osds: [1, 2, 4, 5, 3, 0] 2023-07-02T14:06:12.018 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:06:12.018 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2023-07-02T14:06:12.018 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-07-02T14:06:12.018 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-07-02T14:06:12.019 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2023-07-02T14:06:12.021 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:06:12.021 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:12.017+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:12.022 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete...+ (( i++ )) 2023-07-02T14:06:12.022 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i < 10 )) 2023-07-02T14:06:12.022 INFO:tasks.workunit.client.0.smithi163.stderr:+ pv -L 10M linux-5.4.tar.gz 2023-07-02T14:06:12.045 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:06:12.118 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:12.117+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:12.217 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:12.218 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:12.218 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:12.218 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:12.219 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:12.218+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:12.222 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:12.222 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "" 2023-07-02T14:06:12.222 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:12.229 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:06:12.232 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:12.250 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:12.267 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:12.286 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:12.306 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:12.319 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:12.318+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:12.324 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:12.325 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:06:12.325 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:06:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:06:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:06:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:06:12.326 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:06:12.357 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:12.357 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:12.368 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:06:12.420 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:12.419+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:12.476 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete...2023-07-02T14:06:12.476+0000 7f4d60de9700 0 -- 172.21.15.163:0/3443768441 >> [v2:172.21.15.22:6808/99977,v1:172.21.15.22:6809/99977] conn(0x5614c6062c30 msgr2=0x5614c6065020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:12.479 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:12.479+0000 7f4d5fde7700 0 -- 172.21.15.163:0/3443768441 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7f4d44004b10 msgr2=0x7f4d40039660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:12.483 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:12.483+0000 7f4d605e8700 0 -- 172.21.15.163:0/3443768441 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f4d40036bb0 msgr2=0x7f4d40038fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:06:12.521 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:12.520+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:12.533 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:12.533+0000 7f4d60de9700 0 -- 172.21.15.163:0/3443768441 >> [v2:172.21.15.22:6808/99977,v1:172.21.15.22:6809/99977] conn(0x7f4d4400b200 msgr2=0x7f4d40039b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:12.558 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:12.558 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:12.571 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:06:12.621 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:12.621+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:12.698 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:12.698 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:12.708 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:06:12.749 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:12.748+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:12.822 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:12.821+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:12.839 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 100% complete...done. 2023-07-02T14:06:12.846 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:06:12.846 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=51 2023-07-02T14:06:12.846 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap51 2023-07-02T14:06:12.875 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:12.875 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:12.885 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:06:12.923 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:12.922+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:13.009 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:13.009 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:13.019 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:06:13.024 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:13.023+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:13.125 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:13.124+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:13.143 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:13.144 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:13.154 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:06:13.237 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:13.236+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:13.318 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:13.318 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:13.326 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:13.325+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:13.328 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:06:13.426 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:13.426+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:13.492 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:13.493 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:13.504 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:06:13.544 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:06:13.546 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:13.545+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:13.551 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 3 2023-07-02T14:06:13.628 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:13.628+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:13.668 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:13.669 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:13.679 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:06:13.729 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:13.728+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:13.802 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:13.803 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:13.814 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:06:13.856 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:13.856+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:13.931 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:13.930+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:13.937 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:13.937 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:13.947 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:06:14.031 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:14.030+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:14.085 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:14.086 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:14.096 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:06:14.132 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:14.132+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:14.233 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:14.233+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:14.260 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:14.260 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:14.270 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:06:14.334 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:14.333+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:14.337 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:06:14.395 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:14.396 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:14.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:06:14.435 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:14.435+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:14.478 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:14.496 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:14.514 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:14.528 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:06:14.533 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:14.535 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:14.535 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:14.535 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:14.535 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:14.536 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:14.535+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:14.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:06:14.553 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:14.571 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:14.573 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:06:14.573 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:06:14.574 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:06:14.574 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:06:14.574 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:06:14.574 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:06:14.637 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:14.637+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:14.679 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:14.680 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:14.680 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:06:14.680 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:06:14.680 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:14.690 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:06:14.738 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:14.737+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:14.816 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:14.816 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:06:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:06:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:06:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 185.36274372400001, 2023-07-02T14:06:14.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:06:14.818 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:14.818 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:14.818 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:14.818 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:14.818 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:14.818 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:06:14.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.819 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:14.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:14.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:14.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:14.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:14.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:14.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:14.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:14.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:06:14.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:06:14.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:14.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:06:14.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:06:14.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:14.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:06:14.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:06:14.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:14.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:06:14.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:06:14.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:06:14.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:06:14.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:14.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:06:14.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:06:14.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:06:14.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:06:14.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:14.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:06:14.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:06:14.826 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.826 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:14.826 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.826 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:06:14.826 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:14.827 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 184.49205071599999, 2023-07-02T14:06:14.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:06:14.827 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:14.827 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:14.827 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:14.827 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:06:14.827 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:06:14.827 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:06:14.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.828 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:14.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:14.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:14.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:14.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:14.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:14.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:14.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:14.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:06:14.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:06:14.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:14.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:06:14.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:06:14.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:06:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:06:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:06:14.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:06:14.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:06:14.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:06:14.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:14.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:06:14.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:06:14.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:06:14.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:06:14.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:14.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:06:14.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:06:14.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:14.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:06:14.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:06:14.835 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.835 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:14.835 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.836 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:14.836 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:14.836 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 184.491925995, 2023-07-02T14:06:14.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:06:14.836 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:14.836 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:14.837 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:14.837 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:14.837 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:14.837 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:06:14.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.837 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:14.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:14.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:14.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:14.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:14.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:14.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:14.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:14.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:06:14.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:06:14.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:14.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:06:14.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:06:14.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:14.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:06:14.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:06:14.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:14.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:06:14.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:06:14.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:06:14.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:06:14.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:14.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:06:14.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:06:14.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:06:14.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:06:14.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:14.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:06:14.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:06:14.844 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.844 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:14.844 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.844 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:14.844 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:14.845 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 184.48507611100001, 2023-07-02T14:06:14.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:06:14.845 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:14.845 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:14.845 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:14.845 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:14.845 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:14.846 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:06:14.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.846 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:14.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.846 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:14.846 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:14.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:14.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:14.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:14.847 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:06:14.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:14.847 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:06:14.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:06:14.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:14.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:06:14.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:06:14.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:14.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:06:14.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:06:14.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:14.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:06:14.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:06:14.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:06:14.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:06:14.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:14.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:06:14.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:06:14.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:06:14.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:06:14.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:14.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:06:14.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:06:14.852 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.852 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:14.852 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.853 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:14.853 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:14.853 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 184.485013186, 2023-07-02T14:06:14.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:06:14.853 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:14.853 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:14.854 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:14.854 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:14.854 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:14.854 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:06:14.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.854 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:14.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:14.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:14.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:14.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:14.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:14.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:14.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:14.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:06:14.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:06:14.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:14.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:06:14.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:06:14.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:14.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:06:14.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:06:14.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:14.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:06:14.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:06:14.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:06:14.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:06:14.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:14.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:06:14.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:06:14.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:06:14.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:06:14.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:14.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:06:14.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:06:14.861 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.861 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:14.861 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.861 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:14.861 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:14.862 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 184.48471276500001, 2023-07-02T14:06:14.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:06:14.862 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:14.862 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:14.862 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:14.862 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:14.862 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:14.862 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:06:14.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.863 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:14.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:14.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:14.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:14.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:14.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:14.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:14.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:14.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:06:14.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:06:14.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:14.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:06:14.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:06:14.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:14.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:06:14.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:06:14.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:14.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:06:14.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:06:14.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.867 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:06:14.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:06:14.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:14.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:06:14.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:06:14.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:06:14.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:06:14.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:14.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:06:14.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:06:14.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:06:14.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:06:14.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:06:14.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:14.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:06:14.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:06:14.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:14.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:06:14.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:06:14.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:14.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:06:14.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:06:14.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:14.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:06:14.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:06:14.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:14.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:06:14.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:06:14.873 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.874 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:14.874 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.874 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:06:14.874 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:14.874 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 184.484188265, 2023-07-02T14:06:14.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:06:14.875 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:14.875 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:14.875 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:14.875 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:14.875 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:14.875 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:06:14.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.876 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:14.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:14.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:14.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:14.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:14.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:14.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:14.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:14.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:14.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:06:14.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:14.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:06:14.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:06:14.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:14.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:06:14.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:06:14.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:14.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:06:14.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:06:14.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:06:14.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:06:14.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:14.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:06:14.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:06:14.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:06:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:06:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:14.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:06:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:06:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:06:14.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:06:14.884 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.884 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:14.884 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.884 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:06:14.884 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:14.884 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 157.36016914000001, 2023-07-02T14:06:14.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:06:14.885 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:14.885 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:14.885 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:14.885 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:14.885 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:14.885 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:06:14.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.886 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:14.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:14.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:14.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:14.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:14.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:14.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:14.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:14.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:06:14.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:06:14.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:14.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:06:14.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:06:14.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:14.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:06:14.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:06:14.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:14.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:06:14.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:06:14.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:06:14.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:06:14.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:14.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:06:14.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:06:14.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:06:14.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:06:14.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:14.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:06:14.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:06:14.892 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.892 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:14.892 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.893 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:06:14.893 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:14.893 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 157.03468643599999, 2023-07-02T14:06:14.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:06:14.893 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:14.893 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:14.894 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:14.894 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:14.894 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:14.894 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:06:14.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.894 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:14.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:14.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:14.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:14.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:14.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:14.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:14.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:14.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:06:14.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:06:14.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:14.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:06:14.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:06:14.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:14.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:06:14.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:06:14.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:14.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:06:14.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:06:14.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:06:14.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:06:14.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:14.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:06:14.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:06:14.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:06:14.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:06:14.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:14.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:06:14.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:06:14.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:14.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:06:14.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:06:14.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:14.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:06:14.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:06:14.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:14.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:06:14.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:06:14.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:14.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:06:14.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:06:14.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:14.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:06:14.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:06:14.907 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.907 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:14.907 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.907 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:17 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 0~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:14.908 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:14.908 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.616410653999999, 2023-07-02T14:06:14.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997, 2023-07-02T14:06:14.908 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:14.908 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:14.909 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:14.909 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:14.909 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:14.909 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17 2023-07-02T14:06:14.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.909 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:14.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:14.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:14.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:14.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:14.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:14.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:06:14.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:14.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198431+0000", 2023-07-02T14:06:14.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2815e-05 2023-07-02T14:06:14.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:14.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198443+0000", 2023-07-02T14:06:14.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2569999999999999e-05 2023-07-02T14:06:14.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:14.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198466+0000", 2023-07-02T14:06:14.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0223e-05 2023-07-02T14:06:14.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:14.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198476+0000", 2023-07-02T14:06:14.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1792999999999997e-05 2023-07-02T14:06:14.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:06:14.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4249999999999998e-06 2023-07-02T14:06:14.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:14.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198536+0000", 2023-07-02T14:06:14.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91888663500000001 2023-07-02T14:06:14.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117423+0000", 2023-07-02T14:06:14.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223594 2023-07-02T14:06:14.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:14.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117647+0000", 2023-07-02T14:06:14.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033832900000000001 2023-07-02T14:06:14.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:14.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117985+0000", 2023-07-02T14:06:14.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997 2023-07-02T14:06:14.916 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.917 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:14.917 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.917 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:18 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 65536~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:14.917 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:14.917 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.616353443000001, 2023-07-02T14:06:14.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998, 2023-07-02T14:06:14.918 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:14.918 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:14.918 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:14.918 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:14.918 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:14.918 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18 2023-07-02T14:06:14.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.918 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:14.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:14.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:14.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:14.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:14.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:14.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:14.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:14.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:14.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7600000000000002e-06 2023-07-02T14:06:14.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:14.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198494+0000", 2023-07-02T14:06:14.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7840000000000004e-06 2023-07-02T14:06:14.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:14.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198502+0000", 2023-07-02T14:06:14.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6459999999999998e-06 2023-07-02T14:06:14.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:14.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198508+0000", 2023-07-02T14:06:14.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7395999999999997e-05 2023-07-02T14:06:14.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198565+0000", 2023-07-02T14:06:14.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6800000000000004e-06 2023-07-02T14:06:14.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:14.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198572+0000", 2023-07-02T14:06:14.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91965896800000002 2023-07-02T14:06:14.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118231+0000", 2023-07-02T14:06:14.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5212999999999998e-05 2023-07-02T14:06:14.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:14.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118326+0000", 2023-07-02T14:06:14.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000238538 2023-07-02T14:06:14.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:14.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118565+0000", 2023-07-02T14:06:14.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998 2023-07-02T14:06:14.926 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.926 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:14.926 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.927 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:19 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 131072~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:14.927 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:14.927 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.616329219000001, 2023-07-02T14:06:14.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002, 2023-07-02T14:06:14.927 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:14.927 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:14.927 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:14.927 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:14.928 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:14.928 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 19 2023-07-02T14:06:14.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.928 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:14.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:14.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:14.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:14.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:14.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:14.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:14.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:14.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:14.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1990000000000002e-06 2023-07-02T14:06:14.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:14.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198517+0000", 2023-07-02T14:06:14.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8789999999999997e-06 2023-07-02T14:06:14.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:14.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198524+0000", 2023-07-02T14:06:14.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:06:14.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:14.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:06:14.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5545999999999999e-05 2023-07-02T14:06:14.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:14.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2540000000000003e-06 2023-07-02T14:06:14.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:14.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198598+0000", 2023-07-02T14:06:14.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.919983263 2023-07-02T14:06:14.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118581+0000", 2023-07-02T14:06:14.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6878999999999994e-05 2023-07-02T14:06:14.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:14.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118648+0000", 2023-07-02T14:06:14.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149732 2023-07-02T14:06:14.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:14.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118798+0000", 2023-07-02T14:06:14.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002 2023-07-02T14:06:14.935 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.935 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:14.935 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.936 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:20 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 262144~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:14.936 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:14.936 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.616310255000002, 2023-07-02T14:06:14.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998, 2023-07-02T14:06:14.936 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:14.936 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:14.937 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:14.937 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:14.937 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:14.937 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20 2023-07-02T14:06:14.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.937 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:14.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:14.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:14.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:14.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:14.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:14.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:14.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:14.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:14.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.732e-06 2023-07-02T14:06:14.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:14.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198537+0000", 2023-07-02T14:06:14.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5259999999999999e-06 2023-07-02T14:06:14.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:14.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198543+0000", 2023-07-02T14:06:14.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.121e-06 2023-07-02T14:06:14.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:14.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198547+0000", 2023-07-02T14:06:14.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9591000000000005e-05 2023-07-02T14:06:14.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:06:14.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3120000000000004e-06 2023-07-02T14:06:14.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:14.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198621+0000", 2023-07-02T14:06:14.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92020789400000003 2023-07-02T14:06:14.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118829+0000", 2023-07-02T14:06:14.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022393386000000001 2023-07-02T14:06:14.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:14.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141223+0000", 2023-07-02T14:06:14.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000329866 2023-07-02T14:06:14.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:14.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141552+0000", 2023-07-02T14:06:14.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998 2023-07-02T14:06:14.945 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.945 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:14.945 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.946 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:21 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 196608~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:14.946 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:14.946 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.61628262, 2023-07-02T14:06:14.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995, 2023-07-02T14:06:14.946 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:14.946 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:14.946 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:14.946 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:14.947 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:14.947 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21 2023-07-02T14:06:14.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.947 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:14.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:14.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:14.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:14.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:14.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:14.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:14.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:14.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:14.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0169999999999999e-05 2023-07-02T14:06:14.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:14.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198571+0000", 2023-07-02T14:06:14.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3479999999999998e-06 2023-07-02T14:06:14.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:14.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198578+0000", 2023-07-02T14:06:14.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8439999999999997e-06 2023-07-02T14:06:14.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:14.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198583+0000", 2023-07-02T14:06:14.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8294999999999997e-05 2023-07-02T14:06:14.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198641+0000", 2023-07-02T14:06:14.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8120000000000001e-06 2023-07-02T14:06:14.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:14.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:06:14.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94293828499999999 2023-07-02T14:06:14.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141583+0000", 2023-07-02T14:06:14.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014607300000000001 2023-07-02T14:06:14.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:14.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141729+0000", 2023-07-02T14:06:14.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00044906600000000003 2023-07-02T14:06:14.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:14.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142178+0000", 2023-07-02T14:06:14.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995 2023-07-02T14:06:14.954 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.954 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:14.954 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.955 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:22 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 327680~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:14.955 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:14.955 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.616249189999998, 2023-07-02T14:06:14.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004, 2023-07-02T14:06:14.955 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:14.955 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:14.955 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:14.956 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:14.956 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:14.956 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 22 2023-07-02T14:06:14.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.956 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:14.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:14.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:14.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:14.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:14.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:14.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:14.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:14.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:14.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4850000000000001e-06 2023-07-02T14:06:14.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:14.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198597+0000", 2023-07-02T14:06:14.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:06:14.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:14.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198605+0000", 2023-07-02T14:06:14.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4839999999999998e-06 2023-07-02T14:06:14.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:14.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198609+0000", 2023-07-02T14:06:14.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3100999999999999e-05 2023-07-02T14:06:14.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198662+0000", 2023-07-02T14:06:14.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9099999999999998e-06 2023-07-02T14:06:14.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:14.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198666+0000", 2023-07-02T14:06:14.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.943539503 2023-07-02T14:06:14.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142206+0000", 2023-07-02T14:06:14.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012255999999999999 2023-07-02T14:06:14.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:14.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142328+0000", 2023-07-02T14:06:14.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208956 2023-07-02T14:06:14.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:14.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142537+0000", 2023-07-02T14:06:14.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004 2023-07-02T14:06:14.963 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.963 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:14.963 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.964 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:23 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 393216~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:14.964 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:14.964 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.61622921, 2023-07-02T14:06:14.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348, 2023-07-02T14:06:14.964 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:14.965 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:14.965 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:14.965 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:14.965 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:14.965 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23 2023-07-02T14:06:14.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.965 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:14.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:14.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:14.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:14.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:14.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:14.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:14.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:14.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:14.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8439999999999998e-06 2023-07-02T14:06:14.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:14.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:06:14.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0440000000000001e-06 2023-07-02T14:06:14.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:14.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198623+0000", 2023-07-02T14:06:14.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4480000000000002e-06 2023-07-02T14:06:14.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:14.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198628+0000", 2023-07-02T14:06:14.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5843e-05 2023-07-02T14:06:14.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:14.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.89e-06 2023-07-02T14:06:14.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:14.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198688+0000", 2023-07-02T14:06:14.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94389171999999999 2023-07-02T14:06:14.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142580+0000", 2023-07-02T14:06:14.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3892000000000004e-05 2023-07-02T14:06:14.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:14.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142644+0000", 2023-07-02T14:06:14.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017608000000000001 2023-07-02T14:06:14.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:14.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142820+0000", 2023-07-02T14:06:14.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348 2023-07-02T14:06:14.973 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.973 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:14.973 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.973 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:24 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 458752~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:14.974 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:14.974 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.616201136000001, 2023-07-02T14:06:14.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004, 2023-07-02T14:06:14.974 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:14.974 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:14.975 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:14.975 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:14.975 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:14.975 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 24 2023-07-02T14:06:14.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.975 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:14.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:14.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:14.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:14.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:14.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:14.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:14.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:14.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198642+0000", 2023-07-02T14:06:14.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.253e-06 2023-07-02T14:06:14.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:14.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:06:14.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1289999999999998e-06 2023-07-02T14:06:14.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:14.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198652+0000", 2023-07-02T14:06:14.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4510000000000002e-06 2023-07-02T14:06:14.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:14.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198657+0000", 2023-07-02T14:06:14.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9131000000000001e-05 2023-07-02T14:06:14.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198706+0000", 2023-07-02T14:06:14.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8890000000000004e-06 2023-07-02T14:06:14.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:14.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198710+0000", 2023-07-02T14:06:14.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94415412200000004 2023-07-02T14:06:14.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142864+0000", 2023-07-02T14:06:14.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8047000000000006e-05 2023-07-02T14:06:14.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:14.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142952+0000", 2023-07-02T14:06:14.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192967 2023-07-02T14:06:14.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:14.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143145+0000", 2023-07-02T14:06:14.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004 2023-07-02T14:06:14.983 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.983 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:14.983 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.983 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:25 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 524288~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:14.983 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:14.984 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.616182340999998, 2023-07-02T14:06:14.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001, 2023-07-02T14:06:14.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:14.984 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:14.984 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:14.984 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:14.984 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:14.985 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25 2023-07-02T14:06:14.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.985 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:14.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:14.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:14.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:14.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:14.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:14.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:14.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:14.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:14.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.615e-06 2023-07-02T14:06:14.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:14.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198664+0000", 2023-07-02T14:06:14.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4759999999999997e-06 2023-07-02T14:06:14.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:14.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198669+0000", 2023-07-02T14:06:14.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5999999999999998e-06 2023-07-02T14:06:14.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:14.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198673+0000", 2023-07-02T14:06:14.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4256999999999999e-05 2023-07-02T14:06:14.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198727+0000", 2023-07-02T14:06:14.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840000000000001e-06 2023-07-02T14:06:14.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:14.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198731+0000", 2023-07-02T14:06:14.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94444241900000003 2023-07-02T14:06:14.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143173+0000", 2023-07-02T14:06:14.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6839000000000002e-05 2023-07-02T14:06:14.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:14.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143240+0000", 2023-07-02T14:06:14.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017053399999999999 2023-07-02T14:06:14.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:14.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143411+0000", 2023-07-02T14:06:14.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001 2023-07-02T14:06:14.992 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.992 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:14.992 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:14.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:26 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 589824~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.616159189999998, 2023-07-02T14:06:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997, 2023-07-02T14:06:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:14.993 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 26 2023-07-02T14:06:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:14.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:14.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:14.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:14.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:14.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:14.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:14.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0520000000000002e-06 2023-07-02T14:06:14.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:14.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198687+0000", 2023-07-02T14:06:14.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1880000000000002e-06 2023-07-02T14:06:14.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:14.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198695+0000", 2023-07-02T14:06:14.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9960000000000004e-06 2023-07-02T14:06:14.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:14.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198699+0000", 2023-07-02T14:06:14.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2330999999999998e-05 2023-07-02T14:06:14.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:14.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198742+0000", 2023-07-02T14:06:14.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7950000000000001e-06 2023-07-02T14:06:14.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:14.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:14.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198745+0000", 2023-07-02T14:06:14.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94469398199999999 2023-07-02T14:06:14.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:14.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143439+0000", 2023-07-02T14:06:15.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6048e-05 2023-07-02T14:06:15.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143525+0000", 2023-07-02T14:06:15.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017016700000000001 2023-07-02T14:06:15.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143696+0000", 2023-07-02T14:06:15.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997 2023-07-02T14:06:15.001 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.002 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.002 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.002 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.6579.0:11 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected e171)", 2023-07-02T14:06:15.002 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:15.002 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.656360704000001, 2023-07-02T14:06:15.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998, 2023-07-02T14:06:15.003 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.003 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:15.003 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:15.003 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.6579", 2023-07-02T14:06:15.003 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/554403467", 2023-07-02T14:06:15.003 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11 2023-07-02T14:06:15.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.003 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:15.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:15.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:15.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158481+0000", 2023-07-02T14:06:15.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:06:15.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158487+0000", 2023-07-02T14:06:15.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7630000000000006e-06 2023-07-02T14:06:15.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158496+0000", 2023-07-02T14:06:15.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5100000000000001e-06 2023-07-02T14:06:15.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158504+0000", 2023-07-02T14:06:15.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9397000000000002e-05 2023-07-02T14:06:15.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158553+0000", 2023-07-02T14:06:15.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5810000000000003e-06 2023-07-02T14:06:15.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:15.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158562+0000", 2023-07-02T14:06:15.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93893079499999998 2023-07-02T14:06:15.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097493+0000", 2023-07-02T14:06:15.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015799299999999999 2023-07-02T14:06:15.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097651+0000", 2023-07-02T14:06:15.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187746 2023-07-02T14:06:15.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097839+0000", 2023-07-02T14:06:15.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998 2023-07-02T14:06:15.011 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.012 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.012 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.012 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.012 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.012 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:15.013 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:06:15.014 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:15.013+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:15.114 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:15.113+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:15.179 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:15.179 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:15.179 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:15.179 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:15.189 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:06:15.215 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:15.214+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:15.312 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:15.313 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:15.313 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:06:15.313 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:06:15.313 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:15.315 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:15.314+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:15.323 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:06:15.416 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:15.415+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:15.489 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:15.489 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:06:15.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:06:15.489 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:15.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.490 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:06:15.490 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:06:15.490 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 194.37995400599999, 2023-07-02T14:06:15.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:06:15.490 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.490 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:15.491 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:15.491 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:15.491 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:15.491 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:06:15.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.491 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:06:15.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:06:15.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:06:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:06:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:06:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:06:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:06:15.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:06:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:06:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:06:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:06:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:06:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:06:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:06:15.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:06:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:06:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:06:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:06:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:06:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:06:15.498 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:06:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:06:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 194.37989326799999, 2023-07-02T14:06:15.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:06:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:06:15.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:06:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:06:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:06:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:06:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:06:15.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:06:15.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:06:15.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:06:15.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:06:15.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:06:15.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:06:15.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:06:15.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:15.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:06:15.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:06:15.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:06:15.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:06:15.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:06:15.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:06:15.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:15.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:06:15.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:06:15.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:15.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:06:15.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:06:15.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:15.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:06:15.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:06:15.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:15.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:06:15.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:06:15.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:06:15.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:06:15.511 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.511 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.511 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.512 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:06:15.512 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:06:15.512 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 184.149438938, 2023-07-02T14:06:15.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:06:15.512 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.512 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:15.512 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:15.513 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:15.513 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:15.513 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:06:15.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.513 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:06:15.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:06:15.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:15.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:06:15.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:06:15.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:06:15.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:06:15.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:06:15.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:06:15.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:06:15.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:06:15.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:06:15.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:06:15.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:06:15.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:06:15.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:15.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:06:15.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:06:15.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:15.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:06:15.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:06:15.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:15.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:06:15.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:06:15.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:15.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:06:15.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:06:15.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:06:15.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:06:15.522 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.522 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.522 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.522 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9620 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 589824~65536] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:06:15.522 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:06:15.523 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 110.46542434, 2023-07-02T14:06:15.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001, 2023-07-02T14:06:15.523 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.523 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:15.523 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:15.523 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:15.523 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:15.523 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9620 2023-07-02T14:06:15.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.524 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:06:15.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:06:15.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:15.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022203+0000", 2023-07-02T14:06:15.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0346000000000001e-05 2023-07-02T14:06:15.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022263+0000", 2023-07-02T14:06:15.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4657999999999999e-05 2023-07-02T14:06:15.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022278+0000", 2023-07-02T14:06:15.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:06:15.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022285+0000", 2023-07-02T14:06:15.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0096000000000003e-05 2023-07-02T14:06:15.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022335+0000", 2023-07-02T14:06:15.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2334000000000001e-05 2023-07-02T14:06:15.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:15.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022347+0000", 2023-07-02T14:06:15.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095653701999999993 2023-07-02T14:06:15.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118001+0000", 2023-07-02T14:06:15.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000185797 2023-07-02T14:06:15.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118187+0000", 2023-07-02T14:06:15.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026397900000000001 2023-07-02T14:06:15.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:15.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118451+0000", 2023-07-02T14:06:15.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051832800000000002 2023-07-02T14:06:15.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:15.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.123634+0000", 2023-07-02T14:06:15.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001227332 2023-07-02T14:06:15.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:15.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124861+0000", 2023-07-02T14:06:15.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9493999999999998e-05 2023-07-02T14:06:15.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:15.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124901+0000", 2023-07-02T14:06:15.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0462000000000001e-05 2023-07-02T14:06:15.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124931+0000", 2023-07-02T14:06:15.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001 2023-07-02T14:06:15.535 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.535 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.535 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.535 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9621 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 655360~1966080] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:06:15.535 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:06:15.535 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 110.46523695800001, 2023-07-02T14:06:15.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999, 2023-07-02T14:06:15.536 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.536 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:15.536 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:15.536 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:15.536 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:15.536 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9621 2023-07-02T14:06:15.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.537 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:06:15.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:06:15.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:15.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022391+0000", 2023-07-02T14:06:15.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032053939999999999 2023-07-02T14:06:15.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025597+0000", 2023-07-02T14:06:15.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4307999999999999e-05 2023-07-02T14:06:15.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025611+0000", 2023-07-02T14:06:15.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.032e-06 2023-07-02T14:06:15.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025617+0000", 2023-07-02T14:06:15.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8977999999999998e-05 2023-07-02T14:06:15.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025656+0000", 2023-07-02T14:06:15.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:06:15.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:15.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025663+0000", 2023-07-02T14:06:15.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095220739999999998 2023-07-02T14:06:15.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120884+0000", 2023-07-02T14:06:15.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2525000000000001e-05 2023-07-02T14:06:15.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120956+0000", 2023-07-02T14:06:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016232400000000001 2023-07-02T14:06:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.121119+0000", 2023-07-02T14:06:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044007380000000004 2023-07-02T14:06:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.125519+0000", 2023-07-02T14:06:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000797209 2023-07-02T14:06:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126317+0000", 2023-07-02T14:06:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1228000000000004e-05 2023-07-02T14:06:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126378+0000", 2023-07-02T14:06:15.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978000000000003e-05 2023-07-02T14:06:15.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126430+0000", 2023-07-02T14:06:15.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999 2023-07-02T14:06:15.546 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.546 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.547 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.547 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:189 2.3 2:c9621172:::rbd_data.125de1449605.00000000000000c0:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:06:15.547 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:06:15.547 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 102.314949666, 2023-07-02T14:06:15.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001, 2023-07-02T14:06:15.547 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.548 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:15.548 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:15.548 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:06:15.548 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:06:15.548 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:06:15.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.548 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:06:15.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:06:15.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:15.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:06:15.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2023-07-02T14:06:15.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172682+0000", 2023-07-02T14:06:15.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0159999999999997e-06 2023-07-02T14:06:15.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172688+0000", 2023-07-02T14:06:15.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0010000000000003e-06 2023-07-02T14:06:15.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172692+0000", 2023-07-02T14:06:15.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026444333 2023-07-02T14:06:15.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199136+0000", 2023-07-02T14:06:15.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2230000000000004e-05 2023-07-02T14:06:15.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199218+0000", 2023-07-02T14:06:15.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012043595000000001 2023-07-02T14:06:15.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211262+0000", 2023-07-02T14:06:15.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001 2023-07-02T14:06:15.555 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.555 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.555 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.555 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:281 2.1 2:901d5a95:::rbd_data.125de1449605.0000000000000118:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:06:15.555 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:06:15.556 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 101.93216084700001, 2023-07-02T14:06:15.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002, 2023-07-02T14:06:15.556 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.556 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:15.556 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:15.556 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:06:15.556 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:06:15.557 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:06:15.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.557 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:06:15.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:06:15.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:15.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:06:15.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0469999999999998e-06 2023-07-02T14:06:15.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555471+0000", 2023-07-02T14:06:15.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8860000000000001e-06 2023-07-02T14:06:15.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555476+0000", 2023-07-02T14:06:15.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3160000000000002e-06 2023-07-02T14:06:15.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555483+0000", 2023-07-02T14:06:15.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989525000000001 2023-07-02T14:06:15.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591472+0000", 2023-07-02T14:06:15.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3708999999999996e-05 2023-07-02T14:06:15.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591536+0000", 2023-07-02T14:06:15.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055109720000000003 2023-07-02T14:06:15.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.597047+0000", 2023-07-02T14:06:15.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002 2023-07-02T14:06:15.563 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.563 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.563 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.563 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12985 2.7 2:eeddf24b:::rbd_data.125de1449605.000000000000007e:head [write 65536~4128768] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:06:15.563 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:06:15.563 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.704633900999994, 2023-07-02T14:06:15.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999, 2023-07-02T14:06:15.564 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.564 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:15.564 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:15.564 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:15.564 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:15.564 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12985 2023-07-02T14:06:15.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.565 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:06:15.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:06:15.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:15.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782994+0000", 2023-07-02T14:06:15.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019254741999999998 2023-07-02T14:06:15.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802248+0000", 2023-07-02T14:06:15.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0451e-05 2023-07-02T14:06:15.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802279+0000", 2023-07-02T14:06:15.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3318e-05 2023-07-02T14:06:15.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802292+0000", 2023-07-02T14:06:15.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109403 2023-07-02T14:06:15.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802402+0000", 2023-07-02T14:06:15.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012520599999999999 2023-07-02T14:06:15.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802527+0000", 2023-07-02T14:06:15.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187768 2023-07-02T14:06:15.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:15.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802715+0000", 2023-07-02T14:06:15.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013192822 2023-07-02T14:06:15.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:15.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.815907+0000", 2023-07-02T14:06:15.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0086451870000000004 2023-07-02T14:06:15.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:15.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824553+0000", 2023-07-02T14:06:15.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6759999999999999e-05 2023-07-02T14:06:15.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:15.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824609+0000", 2023-07-02T14:06:15.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011314700000000001 2023-07-02T14:06:15.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824722+0000", 2023-07-02T14:06:15.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999 2023-07-02T14:06:15.573 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.573 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.573 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.574 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:165 2.3 2:c9c6393a:::rbd_data.125de1449605.000000000000002c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:06:15.574 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:06:15.574 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.647851083999996, 2023-07-02T14:06:15.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002, 2023-07-02T14:06:15.574 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.575 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:15.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:15.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:06:15.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:06:15.575 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 165 2023-07-02T14:06:15.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.575 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:06:15.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:06:15.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:15.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839777+0000", 2023-07-02T14:06:15.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6740000000000001e-06 2023-07-02T14:06:15.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839780+0000", 2023-07-02T14:06:15.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9030000000000004e-06 2023-07-02T14:06:15.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839788+0000", 2023-07-02T14:06:15.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1799999999999998e-06 2023-07-02T14:06:15.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839792+0000", 2023-07-02T14:06:15.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.031592227000000001 2023-07-02T14:06:15.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871384+0000", 2023-07-02T14:06:15.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013600099999999999 2023-07-02T14:06:15.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871520+0000", 2023-07-02T14:06:15.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082817089999999999 2023-07-02T14:06:15.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.879802+0000", 2023-07-02T14:06:15.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002 2023-07-02T14:06:15.581 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.581 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.581 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.582 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:252 2.7 2:f1d0abf0:::rbd_data.125de1449605.0000000000000083:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:06:15.582 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:06:15.582 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.231566114000003, 2023-07-02T14:06:15.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997, 2023-07-02T14:06:15.582 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.582 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:15.583 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:15.583 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:06:15.583 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:06:15.583 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 252 2023-07-02T14:06:15.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.583 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:06:15.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:06:15.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:15.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256062+0000", 2023-07-02T14:06:15.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0149999999999999e-06 2023-07-02T14:06:15.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256065+0000", 2023-07-02T14:06:15.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5269999999999998e-06 2023-07-02T14:06:15.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256071+0000", 2023-07-02T14:06:15.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:06:15.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256074+0000", 2023-07-02T14:06:15.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028852909 2023-07-02T14:06:15.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.284927+0000", 2023-07-02T14:06:15.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168274 2023-07-02T14:06:15.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.285096+0000", 2023-07-02T14:06:15.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011036300000000001 2023-07-02T14:06:15.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296132+0000", 2023-07-02T14:06:15.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997 2023-07-02T14:06:15.590 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.590 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.590 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.590 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:261 2.1 2:971731cb:::rbd_data.125de1449605.000000000000008c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:06:15.590 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:06:15.590 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.231470638999994, 2023-07-02T14:06:15.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984, 2023-07-02T14:06:15.591 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.591 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:15.591 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:15.591 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:06:15.591 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:06:15.591 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 261 2023-07-02T14:06:15.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.592 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:06:15.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:06:15.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:15.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:06:15.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2819999999999999e-06 2023-07-02T14:06:15.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256160+0000", 2023-07-02T14:06:15.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2020000000000003e-06 2023-07-02T14:06:15.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256165+0000", 2023-07-02T14:06:15.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2100000000000003e-06 2023-07-02T14:06:15.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256170+0000", 2023-07-02T14:06:15.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025871715999999999 2023-07-02T14:06:15.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282041+0000", 2023-07-02T14:06:15.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132364 2023-07-02T14:06:15.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282174+0000", 2023-07-02T14:06:15.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014044566 2023-07-02T14:06:15.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296218+0000", 2023-07-02T14:06:15.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984 2023-07-02T14:06:15.597 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.598 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.598 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.598 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:264 2.3 2:dbf06c9d:::rbd_data.125de1449605.000000000000008f:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:06:15.598 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:06:15.598 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.231436521000006, 2023-07-02T14:06:15.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003, 2023-07-02T14:06:15.599 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.599 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:15.599 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:15.599 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:06:15.599 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:06:15.599 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 264 2023-07-02T14:06:15.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.600 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:06:15.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:06:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:06:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5749999999999999e-06 2023-07-02T14:06:15.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256195+0000", 2023-07-02T14:06:15.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.327e-06 2023-07-02T14:06:15.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256200+0000", 2023-07-02T14:06:15.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0580000000000002e-06 2023-07-02T14:06:15.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256203+0000", 2023-07-02T14:06:15.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035707524999999997 2023-07-02T14:06:15.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.291911+0000", 2023-07-02T14:06:15.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014269799999999999 2023-07-02T14:06:15.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.292053+0000", 2023-07-02T14:06:15.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062760309999999996 2023-07-02T14:06:15.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.298329+0000", 2023-07-02T14:06:15.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003 2023-07-02T14:06:15.606 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.606 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.606 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.606 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:341 2.3 2:cee258bd:::rbd_data.125de1449605.00000000000000dc:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:06:15.606 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:06:15.606 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 77.850949193999995, 2023-07-02T14:06:15.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997, 2023-07-02T14:06:15.607 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.607 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:15.607 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:15.607 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:06:15.607 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:06:15.607 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 341 2023-07-02T14:06:15.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.608 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:06:15.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:06:15.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:15.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636679+0000", 2023-07-02T14:06:15.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6240000000000002e-06 2023-07-02T14:06:15.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636682+0000", 2023-07-02T14:06:15.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6910000000000002e-06 2023-07-02T14:06:15.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636688+0000", 2023-07-02T14:06:15.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3469999999999999e-06 2023-07-02T14:06:15.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636691+0000", 2023-07-02T14:06:15.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035625560000000001 2023-07-02T14:06:15.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672317+0000", 2023-07-02T14:06:15.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108391 2023-07-02T14:06:15.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672425+0000", 2023-07-02T14:06:15.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012104976 2023-07-02T14:06:15.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.684530+0000", 2023-07-02T14:06:15.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997 2023-07-02T14:06:15.614 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.615 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.615 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.615 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPushReply(2.7 139/134 [PushReplyOp(2:e3782f39:::rbd_data.125de1449605.0000000000000120:head)])", 2023-07-02T14:06:15.615 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:06:15.615 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 63.560417041000001, 2023-07-02T14:06:15.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998, 2023-07-02T14:06:15.616 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.616 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:15.616 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:06:15.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:06:15.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:15.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927211+0000", 2023-07-02T14:06:15.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3689999999999998e-06 2023-07-02T14:06:15.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927217+0000", 2023-07-02T14:06:15.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7269999999999996e-06 2023-07-02T14:06:15.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927225+0000", 2023-07-02T14:06:15.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:06:15.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927230+0000", 2023-07-02T14:06:15.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:06:15.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927266+0000", 2023-07-02T14:06:15.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.069972871000000006 2023-07-02T14:06:15.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.997239+0000", 2023-07-02T14:06:15.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998 2023-07-02T14:06:15.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.621 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19179 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2832, mlcod=177'2827)", 2023-07-02T14:06:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:06:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.0288872110000007, 2023-07-02T14:06:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051928864999999998, 2023-07-02T14:06:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:15.622 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:06:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:06:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458741+0000", 2023-07-02T14:06:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01233852 2023-07-02T14:06:15.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471079+0000", 2023-07-02T14:06:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8131e-05 2023-07-02T14:06:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471097+0000", 2023-07-02T14:06:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.19e-06 2023-07-02T14:06:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471104+0000", 2023-07-02T14:06:15.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4463999999999999e-05 2023-07-02T14:06:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471158+0000", 2023-07-02T14:06:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1723000000000001e-05 2023-07-02T14:06:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471190+0000", 2023-07-02T14:06:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039438622999999999 2023-07-02T14:06:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510628+0000", 2023-07-02T14:06:15.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2280000000000002e-05 2023-07-02T14:06:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510671+0000", 2023-07-02T14:06:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051928864999999998 2023-07-02T14:06:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.629 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19180 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2833, mlcod=177'2827)", 2023-07-02T14:06:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:06:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.0165099709999996, 2023-07-02T14:06:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039574109000000003, 2023-07-02T14:06:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.630 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:06:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:06:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471118+0000", 2023-07-02T14:06:15.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4799000000000001e-05 2023-07-02T14:06:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471143+0000", 2023-07-02T14:06:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3960000000000003e-06 2023-07-02T14:06:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471149+0000", 2023-07-02T14:06:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9230000000000004e-06 2023-07-02T14:06:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471153+0000", 2023-07-02T14:06:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015076050000000001 2023-07-02T14:06:15.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.486229+0000", 2023-07-02T14:06:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8466000000000002e-05 2023-07-02T14:06:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.486257+0000", 2023-07-02T14:06:15.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024419020999999999 2023-07-02T14:06:15.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510676+0000", 2023-07-02T14:06:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6693000000000001e-05 2023-07-02T14:06:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510693+0000", 2023-07-02T14:06:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039574109000000003 2023-07-02T14:06:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19181 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2834, mlcod=177'2827)", 2023-07-02T14:06:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:06:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.0164687099999998, 2023-07-02T14:06:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039548481000000003, 2023-07-02T14:06:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:06:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:06:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:06:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8320999999999999e-05 2023-07-02T14:06:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471188+0000", 2023-07-02T14:06:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9669999999999998e-06 2023-07-02T14:06:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471194+0000", 2023-07-02T14:06:15.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8709999999999999e-06 2023-07-02T14:06:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471198+0000", 2023-07-02T14:06:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016760634999999999 2023-07-02T14:06:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.487959+0000", 2023-07-02T14:06:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8398000000000002e-05 2023-07-02T14:06:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.487987+0000", 2023-07-02T14:06:15.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022707627000000001 2023-07-02T14:06:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510695+0000", 2023-07-02T14:06:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3991999999999999e-05 2023-07-02T14:06:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510709+0000", 2023-07-02T14:06:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039548481000000003 2023-07-02T14:06:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19192 2.7 e177/141 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head v 177'2835, mlcod=177'2827)", 2023-07-02T14:06:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:06:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.0164257380000006, 2023-07-02T14:06:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039521186999999999, 2023-07-02T14:06:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:06:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:06:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:06:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014525079 2023-07-02T14:06:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485728+0000", 2023-07-02T14:06:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8372e-05 2023-07-02T14:06:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485746+0000", 2023-07-02T14:06:15.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6149999999999997e-06 2023-07-02T14:06:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485755+0000", 2023-07-02T14:06:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036949270000000002 2023-07-02T14:06:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489450+0000", 2023-07-02T14:06:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9415e-05 2023-07-02T14:06:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489469+0000", 2023-07-02T14:06:15.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021241488999999999 2023-07-02T14:06:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510711+0000", 2023-07-02T14:06:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3604999999999999e-05 2023-07-02T14:06:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510724+0000", 2023-07-02T14:06:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039521186999999999 2023-07-02T14:06:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:20659 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2134016~94208] snapc 35=[35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e179)", 2023-07-02T14:06:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:06:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.897242356, 2023-07-02T14:06:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041695300999999997, 2023-07-02T14:06:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:15.655 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20659 2023-07-02T14:06:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:06:15.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:06:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590386+0000", 2023-07-02T14:06:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1415999999999994e-05 2023-07-02T14:06:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590467+0000", 2023-07-02T14:06:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8029999999999999e-06 2023-07-02T14:06:15.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590474+0000", 2023-07-02T14:06:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5379999999999998e-06 2023-07-02T14:06:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590478+0000", 2023-07-02T14:06:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073612449999999998 2023-07-02T14:06:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.597839+0000", 2023-07-02T14:06:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013387800000000001 2023-07-02T14:06:15.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.597973+0000", 2023-07-02T14:06:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022472799999999999 2023-07-02T14:06:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.598197+0000", 2023-07-02T14:06:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065294259999999996 2023-07-02T14:06:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:15.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.604727+0000", 2023-07-02T14:06:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027239217999999999 2023-07-02T14:06:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.631966+0000", 2023-07-02T14:06:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3194999999999994e-05 2023-07-02T14:06:15.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632029+0000", 2023-07-02T14:06:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2639999999999997e-05 2023-07-02T14:06:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632082+0000", 2023-07-02T14:06:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041695300999999997 2023-07-02T14:06:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.665 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:20660 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2031616~196608] snapc 35=[35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e179)", 2023-07-02T14:06:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:06:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.8971473360000002, 2023-07-02T14:06:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041610001000000001, 2023-07-02T14:06:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:15.666 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20660 2023-07-02T14:06:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:06:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:15.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:06:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590481+0000", 2023-07-02T14:06:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012730599999999999 2023-07-02T14:06:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590608+0000", 2023-07-02T14:06:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6319999999999997e-06 2023-07-02T14:06:15.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:15.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590617+0000", 2023-07-02T14:06:15.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5209999999999998e-06 2023-07-02T14:06:15.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:15.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590621+0000", 2023-07-02T14:06:15.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018367430000000001 2023-07-02T14:06:15.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:15.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592457+0000", 2023-07-02T14:06:15.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016640599999999999 2023-07-02T14:06:15.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:15.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592624+0000", 2023-07-02T14:06:15.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028927400000000001 2023-07-02T14:06:15.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:15.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592913+0000", 2023-07-02T14:06:15.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011797813000000001 2023-07-02T14:06:15.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:15.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.604711+0000", 2023-07-02T14:06:15.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027285997999999999 2023-07-02T14:06:15.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:15.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.631997+0000", 2023-07-02T14:06:15.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8890000000000001e-05 2023-07-02T14:06:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632046+0000", 2023-07-02T14:06:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5867e-05 2023-07-02T14:06:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:15.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632092+0000", 2023-07-02T14:06:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041610001000000001 2023-07-02T14:06:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:15.676 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:15.677 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:15.678 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:06:15.679 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:06:15.679 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:15.677+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:15.778 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:15.777+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:15.817 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:15.817 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:15.817 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:06:15.817 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:15.827 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:06:15.879 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:15.878+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:15.951 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:15.951 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:15.961 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:06:15.979 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:15.979+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:16.081 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:16.080+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:16.086 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:06:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.18717963500001, 2023-07-02T14:06:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:06:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:06:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:06:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:06:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:06:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:06:16.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:06:16.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:06:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:06:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:06:16.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:06:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:06:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:06:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:06:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:06:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:06:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:06:16.094 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:06:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:16.095 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.18309391700001, 2023-07-02T14:06:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:06:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:06:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:06:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:06:16.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:06:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:06:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:06:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:06:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:06:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:06:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:06:16.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:06:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:06:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:06:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:06:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:06:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:06:16.102 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:06:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.179229136, 2023-07-02T14:06:16.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:06:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:06:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:06:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:06:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:06:16.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:06:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:06:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:06:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:06:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:06:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:06:16.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:06:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:06:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:06:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:06:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:06:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:06:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.110 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:06:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:06:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.17570487500001, 2023-07-02T14:06:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:06:16.111 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:06:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:06:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:06:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:06:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:06:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:06:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:06:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:06:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:06:16.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:06:16.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:06:16.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:06:16.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:06:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:06:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:06:16.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:06:16.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:06:16.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:06:16.119 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.120 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.120 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.121 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:06:16.121 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:16.121 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.14000007600001, 2023-07-02T14:06:16.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:06:16.122 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.122 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.123 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:16.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:16.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:06:16.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:06:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:06:16.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:06:16.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:06:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:06:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:06:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:06:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:06:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:06:16.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:06:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:06:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:06:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:06:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:06:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:06:16.132 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:06:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:16.133 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.135342718, 2023-07-02T14:06:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:06:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:16.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:06:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:06:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:06:16.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:06:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:06:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:06:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:06:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:06:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:06:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:06:16.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:06:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:06:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:06:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:06:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:06:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:06:16.140 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:06:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.13440285499999, 2023-07-02T14:06:16.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:06:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:06:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:06:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:06:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:06:16.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:06:16.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:06:16.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:06:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:06:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:06:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:06:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:06:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:06:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:06:16.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:06:16.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:06:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:06:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.149 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:06:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.13300497700001, 2023-07-02T14:06:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:06:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:16.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:06:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:06:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:06:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:06:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:06:16.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:06:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:06:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:06:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:06:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:06:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:06:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:06:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:06:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:06:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:06:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:06:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.157 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:06:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 134.722814481, 2023-07-02T14:06:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:06:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:06:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:06:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:06:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:06:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:06:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:06:16.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:06:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:06:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:06:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:06:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:06:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:06:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:06:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:06:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:06:16.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:06:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:16.165 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 133.356464797, 2023-07-02T14:06:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:06:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.166 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:06:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:06:16.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:06:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:06:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:06:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:06:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:06:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:06:16.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:06:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:06:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:06:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:06:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:06:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:06:16.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:06:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:06:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:06:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:06:16.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:06:16.174 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.174 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.174 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.175 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7707 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 3145728~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:16.175 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:06:16.175 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 133.35248132999999, 2023-07-02T14:06:16.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999, 2023-07-02T14:06:16.175 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.175 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.175 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.176 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.176 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.176 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7707 2023-07-02T14:06:16.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.176 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:06:16.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:06:16.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:06:16.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5017e-05 2023-07-02T14:06:16.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732347+0000", 2023-07-02T14:06:16.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5821e-05 2023-07-02T14:06:16.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732363+0000", 2023-07-02T14:06:16.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1270000000000002e-06 2023-07-02T14:06:16.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732368+0000", 2023-07-02T14:06:16.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008444666 2023-07-02T14:06:16.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740812+0000", 2023-07-02T14:06:16.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011266599999999999 2023-07-02T14:06:16.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740925+0000", 2023-07-02T14:06:16.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010848808999999999 2023-07-02T14:06:16.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751774+0000", 2023-07-02T14:06:16.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8421999999999997e-05 2023-07-02T14:06:16.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751832+0000", 2023-07-02T14:06:16.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107453 2023-07-02T14:06:16.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751940+0000", 2023-07-02T14:06:16.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999 2023-07-02T14:06:16.183 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.183 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.184 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.184 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:16.184 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:16.184 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 133.35241745299999, 2023-07-02T14:06:16.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:06:16.185 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.185 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.185 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.185 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.185 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.185 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:06:16.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.185 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:16.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:16.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:16.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:16.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:06:16.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:06:16.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:06:16.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:06:16.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:06:16.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:06:16.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:06:16.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:06:16.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:06:16.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:06:16.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:06:16.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:06:16.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:06:16.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:06:16.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:06:16.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:06:16.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:06:16.193 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.193 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.193 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.193 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:16.193 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:16.193 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 133.347853628, 2023-07-02T14:06:16.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:06:16.194 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.194 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.194 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.194 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.194 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.194 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:06:16.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.195 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:16.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:16.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:16.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:06:16.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:06:16.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:06:16.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:06:16.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:06:16.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:06:16.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:06:16.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:06:16.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.199 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:06:16.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:06:16.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.199 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:06:16.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:06:16.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:06:16.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:06:16.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:06:16.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:06:16.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:06:16.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:06:16.202 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.202 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.202 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.202 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:16.203 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:16.203 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 133.34780376200001, 2023-07-02T14:06:16.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:06:16.203 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.203 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.203 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.203 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.203 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.204 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:06:16.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.204 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:16.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:16.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:16.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:06:16.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:06:16.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:06:16.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:06:16.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:06:16.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:06:16.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:06:16.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:06:16.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:06:16.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:06:16.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:06:16.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:06:16.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:06:16.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:06:16.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:06:16.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:06:16.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:06:16.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:06:16.211 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.211 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.211 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.212 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:16.212 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:16.212 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 133.346764769, 2023-07-02T14:06:16.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:06:16.212 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.212 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.212 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.213 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.213 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.213 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:06:16.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.213 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:16.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:16.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:06:16.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:06:16.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:06:16.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:06:16.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:06:16.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:06:16.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:06:16.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:06:16.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:06:16.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:06:16.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:06:16.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:06:16.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:06:16.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:06:16.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:06:16.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:06:16.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:06:16.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:06:16.221 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.221 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.222 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.222 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7993 2.7 2:fe72db53:::rbd_data.125de1449605.0000000000000072:head [write 65536~3276800] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:16.223 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:06:16.223 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 133.159806721, 2023-07-02T14:06:16.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919, 2023-07-02T14:06:16.224 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.224 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.224 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.224 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.225 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.225 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7993 2023-07-02T14:06:16.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.226 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:06:16.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:06:16.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924986+0000", 2023-07-02T14:06:16.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049542070000000004 2023-07-02T14:06:16.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929940+0000", 2023-07-02T14:06:16.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9740000000000001e-05 2023-07-02T14:06:16.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929960+0000", 2023-07-02T14:06:16.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.311e-05 2023-07-02T14:06:16.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.230 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929973+0000", 2023-07-02T14:06:16.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9629000000000002e-05 2023-07-02T14:06:16.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.231 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930023+0000", 2023-07-02T14:06:16.231 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012751799999999999 2023-07-02T14:06:16.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930150+0000", 2023-07-02T14:06:16.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016449042000000001 2023-07-02T14:06:16.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946599+0000", 2023-07-02T14:06:16.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3241e-05 2023-07-02T14:06:16.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946663+0000", 2023-07-02T14:06:16.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000102296 2023-07-02T14:06:16.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946765+0000", 2023-07-02T14:06:16.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919 2023-07-02T14:06:16.234 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.234 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.234 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.234 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11782 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 196608~2162688] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:06:16.234 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:06:16.235 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 91.675286799000006, 2023-07-02T14:06:16.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365, 2023-07-02T14:06:16.235 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.235 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.235 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.235 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.235 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.236 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11782 2023-07-02T14:06:16.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.236 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:06:16.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:06:16.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409506+0000", 2023-07-02T14:06:16.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020172269999999999 2023-07-02T14:06:16.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411523+0000", 2023-07-02T14:06:16.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8901000000000001e-05 2023-07-02T14:06:16.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411542+0000", 2023-07-02T14:06:16.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5037e-05 2023-07-02T14:06:16.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411557+0000", 2023-07-02T14:06:16.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001931213 2023-07-02T14:06:16.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.240 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413489+0000", 2023-07-02T14:06:16.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000149207 2023-07-02T14:06:16.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413638+0000", 2023-07-02T14:06:16.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024752800000000001 2023-07-02T14:06:16.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:16.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413885+0000", 2023-07-02T14:06:16.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048516000000000002 2023-07-02T14:06:16.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.418737+0000", 2023-07-02T14:06:16.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010319259000000001 2023-07-02T14:06:16.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429056+0000", 2023-07-02T14:06:16.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2518000000000002e-05 2023-07-02T14:06:16.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429099+0000", 2023-07-02T14:06:16.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015275 2023-07-02T14:06:16.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.245 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429251+0000", 2023-07-02T14:06:16.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365 2023-07-02T14:06:16.245 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.245 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.245 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.245 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPushReply(2.7 119/117 [PushReplyOp(2:eb862bc0:::rbd_data.125de1449605.0000000000000040:13)])", 2023-07-02T14:06:16.246 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:06:16.246 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 91.674123523999995, 2023-07-02T14:06:16.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999, 2023-07-02T14:06:16.246 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.246 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "reached pg", 2023-07-02T14:06:16.246 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.246 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:06:16.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:06:16.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:16.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:06:16.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7670000000000001e-06 2023-07-02T14:06:16.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410674+0000", 2023-07-02T14:06:16.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3973e-05 2023-07-02T14:06:16.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410688+0000", 2023-07-02T14:06:16.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0157e-05 2023-07-02T14:06:16.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410708+0000", 2023-07-02T14:06:16.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018578779 2023-07-02T14:06:16.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.251 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429287+0000", 2023-07-02T14:06:16.251 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013086549999999999 2023-07-02T14:06:16.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.251 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.251 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.430595+0000", 2023-07-02T14:06:16.251 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999 2023-07-02T14:06:16.251 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.252 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.252 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.252 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.252 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11861 2.7 2:e57d3f55:::rbd_data.125de1449605.0000000000000134:head [write 0~65536] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:06:16.252 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:06:16.252 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 91.540008146999995, 2023-07-02T14:06:16.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723, 2023-07-02T14:06:16.253 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.253 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.253 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.253 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.253 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.253 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11861 2023-07-02T14:06:16.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.254 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:06:16.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:06:16.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544785+0000", 2023-07-02T14:06:16.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010563700000000001 2023-07-02T14:06:16.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544890+0000", 2023-07-02T14:06:16.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8530000000000004e-06 2023-07-02T14:06:16.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544900+0000", 2023-07-02T14:06:16.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.665e-06 2023-07-02T14:06:16.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544906+0000", 2023-07-02T14:06:16.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000272575 2023-07-02T14:06:16.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545178+0000", 2023-07-02T14:06:16.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000111362 2023-07-02T14:06:16.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545290+0000", 2023-07-02T14:06:16.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016181100000000001 2023-07-02T14:06:16.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:16.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545452+0000", 2023-07-02T14:06:16.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044500750000000004 2023-07-02T14:06:16.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.549902+0000", 2023-07-02T14:06:16.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014850024 2023-07-02T14:06:16.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564752+0000", 2023-07-02T14:06:16.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6272000000000003e-05 2023-07-02T14:06:16.261 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.261 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.261 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564848+0000", 2023-07-02T14:06:16.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000162428 2023-07-02T14:06:16.261 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.262 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.262 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.262 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.565010+0000", 2023-07-02T14:06:16.262 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723 2023-07-02T14:06:16.262 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.262 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.262 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.263 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:13262 2.7 e125/121 2:e0741dde:::rbd_data.125de1449605.00000000000000ad:head v 125'2024, mlcod=125'2021)", 2023-07-02T14:06:16.263 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:06:16.263 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 82.833930409000004, 2023-07-02T14:06:16.263 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999, 2023-07-02T14:06:16.263 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.263 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.263 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.264 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:06:16.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.264 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:06:16.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.265 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250862+0000", 2023-07-02T14:06:16.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080378800000000007 2023-07-02T14:06:16.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258900+0000", 2023-07-02T14:06:16.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8939999999999999e-06 2023-07-02T14:06:16.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258909+0000", 2023-07-02T14:06:16.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3560000000000003e-06 2023-07-02T14:06:16.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258914+0000", 2023-07-02T14:06:16.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2429000000000003e-05 2023-07-02T14:06:16.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.268 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258956+0000", 2023-07-02T14:06:16.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5094999999999999e-05 2023-07-02T14:06:16.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.268 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258981+0000", 2023-07-02T14:06:16.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012177307 2023-07-02T14:06:16.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271158+0000", 2023-07-02T14:06:16.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2104999999999999e-05 2023-07-02T14:06:16.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271181+0000", 2023-07-02T14:06:16.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999 2023-07-02T14:06:16.270 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.270 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.270 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.270 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.270 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.271 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:16.272 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:06:16.272 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:16.271+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:16.372 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:16.372+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:16.437 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:16.437 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:16.437 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:06:16.437 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:16.447 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:06:16.473 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:16.472+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:16.553 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap50 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap51 b3fbb028-e267-4d25-a750-93ba18a2316a@snap51.diff 2023-07-02T14:06:16.574 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:16.573+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:16.587 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:06:16.620 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:16.620 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:16.620 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:16.621 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:16.621 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:16.622 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete...2023-07-02T14:06:16.622+0000 7ff3233e2700 0 -- 172.21.15.163:0/4253608877 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7ff304019ca0 msgr2=0x7ff30401c090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:16.626 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:16.625+0000 7ff323be3700 0 -- 172.21.15.163:0/4253608877 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7ff300004b10 msgr2=0x7ff300004f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-07-02T14:06:16.632 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:06:16.674 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:16.674+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:16.755 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 100% complete...done. 2023-07-02T14:06:16.758 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:06:16.758 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=52 2023-07-02T14:06:16.759 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap52 2023-07-02T14:06:16.764 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:16.764 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:16.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:16.764 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:16.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.765 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:16.765 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:06:16.765 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 217.47166589700001, 2023-07-02T14:06:16.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:06:16.765 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.765 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.765 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.766 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.766 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.766 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:06:16.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.766 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:06:16.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:06:16.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:06:16.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:06:16.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:06:16.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:06:16.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:06:16.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:06:16.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:06:16.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:06:16.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:06:16.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:06:16.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:06:16.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:06:16.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:16.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:06:16.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:06:16.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:06:16.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:06:16.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:06:16.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:06:16.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:06:16.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:06:16.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:06:16.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:06:16.775 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.775 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.775 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.776 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:16.776 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:06:16.776 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 217.47143087399999, 2023-07-02T14:06:16.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:06:16.776 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.776 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.776 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.777 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.777 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.777 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:06:16.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.777 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:06:16.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:06:16.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:06:16.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:06:16.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:06:16.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:06:16.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:06:16.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:06:16.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:06:16.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:06:16.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:06:16.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:06:16.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:06:16.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:06:16.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:16.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:06:16.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:06:16.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:06:16.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:06:16.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:06:16.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:06:16.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:06:16.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:06:16.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:06:16.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:06:16.786 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.786 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.786 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.786 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:16.786 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:06:16.787 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 217.47123174000001, 2023-07-02T14:06:16.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:06:16.787 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.787 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.787 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.787 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.787 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.787 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:06:16.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.788 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:06:16.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:06:16.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:06:16.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:06:16.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:06:16.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:06:16.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:06:16.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:06:16.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:06:16.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:06:16.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:06:16.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:06:16.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:06:16.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:06:16.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:16.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:06:16.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:06:16.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:06:16.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:06:16.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:06:16.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:06:16.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:06:16.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:06:16.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:06:16.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:06:16.796 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.796 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.796 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.797 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:16.797 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:06:16.797 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 217.470866616, 2023-07-02T14:06:16.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:06:16.797 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.798 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.798 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.798 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.798 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.798 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:06:16.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.798 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:06:16.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:06:16.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:16.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:06:16.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:06:16.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:06:16.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:06:16.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:06:16.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:06:16.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:06:16.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:06:16.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:06:16.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:06:16.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:06:16.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:06:16.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:16.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:06:16.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:06:16.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:06:16.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:06:16.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:06:16.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:06:16.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:06:16.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:06:16.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:06:16.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:06:16.807 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.807 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.807 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.808 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:16.808 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:06:16.808 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 217.47078555499999, 2023-07-02T14:06:16.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:06:16.808 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.808 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.809 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.809 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.809 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.809 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:06:16.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.809 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:06:16.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:06:16.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:16.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:06:16.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:06:16.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:06:16.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:06:16.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:06:16.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:06:16.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:06:16.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:06:16.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:06:16.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:06:16.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:06:16.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:06:16.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:16.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:06:16.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:06:16.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:06:16.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:06:16.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:06:16.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:06:16.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:06:16.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:06:16.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:06:16.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:06:16.818 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.818 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.818 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.818 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:16.819 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:06:16.819 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 217.47069388400001, 2023-07-02T14:06:16.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:06:16.819 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.819 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.819 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.819 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.820 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.820 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:06:16.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.820 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:06:16.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:06:16.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:16.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:06:16.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:06:16.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:06:16.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:06:16.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:06:16.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:06:16.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:06:16.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:06:16.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:06:16.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:06:16.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:06:16.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:06:16.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:16.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:06:16.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:06:16.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:06:16.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:06:16.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:06:16.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:06:16.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:06:16.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:06:16.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:06:16.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:06:16.829 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.829 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.829 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.829 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:16.829 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:06:16.829 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 217.470354442, 2023-07-02T14:06:16.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:06:16.830 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.830 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.830 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.830 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.830 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.830 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:06:16.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.831 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:06:16.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:06:16.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:16.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:06:16.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:06:16.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:06:16.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:06:16.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:06:16.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:06:16.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:06:16.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:06:16.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:06:16.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:06:16.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:06:16.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:06:16.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:16.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:06:16.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:06:16.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:06:16.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:06:16.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:06:16.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:06:16.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:06:16.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:06:16.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:06:16.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:06:16.839 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.839 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.839 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.840 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:16.840 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:16.840 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 217.47016838900001, 2023-07-02T14:06:16.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:06:16.840 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.841 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.841 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.841 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.841 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.841 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:06:16.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.841 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:16.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:16.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:16.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:16.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:06:16.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:06:16.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:06:16.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:06:16.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:06:16.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:06:16.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:06:16.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:06:16.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:06:16.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:06:16.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:06:16.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:16.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:06:16.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:06:16.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:06:16.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:06:16.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:06:16.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:06:16.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:06:16.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:06:16.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:06:16.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:06:16.850 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.850 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.850 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.850 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:16.850 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:06:16.850 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 217.47001058699999, 2023-07-02T14:06:16.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:06:16.851 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.851 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.851 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.851 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.851 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.851 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:06:16.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.852 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:06:16.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:06:16.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:06:16.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:06:16.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:06:16.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:06:16.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:06:16.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:06:16.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:06:16.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:06:16.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:06:16.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:06:16.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:06:16.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:06:16.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:16.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:06:16.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:06:16.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:06:16.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:06:16.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:06:16.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:06:16.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:06:16.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:06:16.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:06:16.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:06:16.860 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.860 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.861 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.861 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:16.861 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:06:16.861 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 217.46981880499999, 2023-07-02T14:06:16.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:06:16.861 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.862 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.862 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.862 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.862 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.862 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:06:16.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.862 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:06:16.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:06:16.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:06:16.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:06:16.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:06:16.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:06:16.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:06:16.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:06:16.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:06:16.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:06:16.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:06:16.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:06:16.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:06:16.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:06:16.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:16.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:06:16.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:06:16.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:06:16.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:06:16.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:06:16.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:06:16.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:06:16.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:06:16.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:06:16.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:06:16.871 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.871 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.871 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.872 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:16.872 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:16.872 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 217.46964314499999, 2023-07-02T14:06:16.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:06:16.872 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.872 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.872 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.873 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.873 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.873 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:06:16.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.873 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:16.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:16.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:16.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:06:16.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:06:16.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:06:16.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:06:16.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:06:16.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:06:16.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:06:16.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:06:16.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:06:16.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:06:16.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:06:16.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:16.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:06:16.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:06:16.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:06:16.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:06:16.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:06:16.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:06:16.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:06:16.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:06:16.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:06:16.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:06:16.882 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.882 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.882 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.882 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:16.883 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:06:16.883 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 217.469548957, 2023-07-02T14:06:16.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:06:16.883 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.883 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.883 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.883 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.884 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.884 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:06:16.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.884 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:06:16.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:06:16.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:16.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:06:16.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:06:16.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:06:16.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:06:16.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:06:16.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:06:16.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:06:16.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:06:16.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:06:16.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:06:16.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:06:16.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:06:16.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:16.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:06:16.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:06:16.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:06:16.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:06:16.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:06:16.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:06:16.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:06:16.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:06:16.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:06:16.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:06:16.892 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.893 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.893 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.893 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:16.893 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:06:16.893 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 217.46934557099999, 2023-07-02T14:06:16.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:06:16.894 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.894 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.894 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.894 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.894 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.894 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:06:16.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.894 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:06:16.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:06:16.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:06:16.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:06:16.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:06:16.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:06:16.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:06:16.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:06:16.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:06:16.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:06:16.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:06:16.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:06:16.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:06:16.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:06:16.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:16.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:06:16.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:06:16.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:06:16.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:06:16.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:06:16.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:06:16.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:06:16.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:06:16.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:06:16.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:06:16.903 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.903 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.903 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.904 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:16.904 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:06:16.904 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 217.469194915, 2023-07-02T14:06:16.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:06:16.904 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.904 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.904 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.905 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.905 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.905 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:06:16.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.905 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:06:16.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:06:16.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:16.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:06:16.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:06:16.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:06:16.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:06:16.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:06:16.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:06:16.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:06:16.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:06:16.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:06:16.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:06:16.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:06:16.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:06:16.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:16.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:06:16.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:06:16.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:06:16.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:06:16.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:06:16.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:06:16.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:06:16.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:06:16.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:06:16.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:06:16.914 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.914 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.914 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.914 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:16.915 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:16.915 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 217.469122212, 2023-07-02T14:06:16.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:06:16.915 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.915 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.915 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.915 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.916 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.916 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:06:16.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.916 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:16.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:16.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:16.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:16.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:06:16.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:06:16.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:06:16.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:06:16.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:06:16.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:06:16.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:06:16.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:06:16.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:06:16.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:06:16.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:06:16.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:16.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:06:16.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:06:16.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.922 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:06:16.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:06:16.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:06:16.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:06:16.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:06:16.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:06:16.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:06:16.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:06:16.924 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.924 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.925 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.925 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:16.925 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:16.925 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 217.46900522199999, 2023-07-02T14:06:16.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:06:16.926 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.926 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.926 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.926 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.926 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.926 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:06:16.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.926 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:16.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:16.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:16.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:16.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:06:16.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:06:16.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:06:16.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:06:16.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:06:16.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:06:16.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:06:16.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:06:16.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:06:16.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:06:16.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:06:16.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:16.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:06:16.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:06:16.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:06:16.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:06:16.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:06:16.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:06:16.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:06:16.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:06:16.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:06:16.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:06:16.935 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.935 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.935 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.936 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:16.936 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:06:16.936 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 217.46885577099999, 2023-07-02T14:06:16.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:06:16.936 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.936 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.936 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.937 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.937 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.937 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:06:16.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.937 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:06:16.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:06:16.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:16.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:06:16.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:06:16.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:06:16.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:06:16.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:06:16.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:06:16.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:06:16.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:06:16.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:06:16.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:06:16.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:06:16.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:06:16.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:16.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:06:16.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:06:16.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:06:16.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:06:16.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:06:16.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:06:16.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:06:16.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:06:16.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:06:16.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:06:16.946 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.946 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.946 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.946 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:16.947 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:06:16.947 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 217.44737979799999, 2023-07-02T14:06:16.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:06:16.947 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.947 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.947 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.947 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.947 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.948 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:06:16.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.948 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:06:16.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:06:16.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:06:16.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:06:16.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:06:16.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:06:16.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:06:16.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:06:16.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:06:16.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:06:16.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:06:16.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:06:16.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:06:16.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:06:16.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:16.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:06:16.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:06:16.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.954 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:06:16.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:06:16.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:06:16.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:06:16.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:06:16.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:06:16.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:06:16.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:06:16.956 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.957 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.957 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.957 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:16.957 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:06:16.957 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 217.44724726199999, 2023-07-02T14:06:16.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:06:16.958 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.958 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.958 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.958 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.958 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.958 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:06:16.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.958 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:06:16.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:06:16.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:06:16.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:06:16.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:06:16.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:06:16.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:06:16.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:06:16.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:06:16.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:06:16.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:06:16.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:06:16.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:06:16.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:06:16.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:16.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:06:16.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:06:16.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:06:16.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:06:16.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:06:16.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:06:16.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:06:16.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:06:16.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:06:16.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:06:16.967 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.967 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.967 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.968 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:16.968 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:06:16.968 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 217.38591060100001, 2023-07-02T14:06:16.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:06:16.968 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:16.968 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:16.969 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:16.969 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:16.969 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:16.969 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:06:16.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.969 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:16.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:16.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:06:16.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:16.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:16.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:06:16.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:16.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:16.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:06:16.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:06:16.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:16.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:06:16.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:06:16.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:16.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:06:16.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:06:16.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:16.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:06:16.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:06:16.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:16.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:06:16.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:06:16.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:16.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:06:16.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:06:16.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:16.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:06:16.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:06:16.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:16.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:06:16.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:06:16.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:16.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:06:16.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:06:16.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:16.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:06:16.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:06:16.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:16.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:16.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:16.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:06:16.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:06:16.978 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.978 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.978 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.978 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:16.978 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:16.979 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:16.980 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:16.981 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:16.981 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:16.981 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:16.981 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:16.981 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:16.982 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:06:16.982 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:16.979+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:16.983 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:06:16.983 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:06:16.983 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:06:16.983 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:06:16.983 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:06:16.983 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:06:17.081 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:17.080+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:17.146 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:17.146 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:17.156 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:06:17.197 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:17.196+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:17.235 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 2] out_osds: [5, 4] dead_osds: [] live_osds: [1, 2, 4, 5, 3, 0] 2023-07-02T14:06:17.235 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:06:17.235 INFO:tasks.thrashosds.thrasher:Adding osd 4 2023-07-02T14:06:17.236 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2023-07-02T14:06:17.282 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:17.281+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:17.320 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:17.320 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:17.330 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:06:17.412 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:17.412+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:17.483 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:17.482+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:17.493 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:17.494 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:17.503 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:06:17.552 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-07-02T14:06:17.551+0000 7ff490b05700 -1 mon.a@0(leader).osd e183 definitely_dead 0 2023-07-02T14:06:17.554 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:06:17.560 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 1 2023-07-02T14:06:17.584 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:17.583+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:17.632 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:17.633 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:17.643 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:06:17.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:06:17.684 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:17.683+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:17.785 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:17.783+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:17.811 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:17.812 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:17.822 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:06:17.905 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:17.904+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:17.986 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:17.986 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:17.986 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:17.986+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:17.996 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:06:18.087 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:18.086+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:18.160 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:18.160 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:18.170 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:06:18.214 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:18.213+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:18.288 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:18.287+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:18.293 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:18.294 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:18.304 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:06:18.389 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:18.388+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:18.468 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:18.468 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:18.478 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:06:18.489 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:18.489+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:18.520 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:06:18.521 INFO:teuthology.orchestra.run.smithi016.stderr:marked in osd.4. 2023-07-02T14:06:18.535 INFO:tasks.thrashosds.thrasher:Added osd 4 2023-07-02T14:06:18.561 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap51 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap52 b3fbb028-e267-4d25-a750-93ba18a2316a@snap52.diff 2023-07-02T14:06:18.590 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:18.589+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:18.607 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:18.607 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:18.617 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:06:18.691 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:18.691+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:18.780 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:18.781 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:18.791 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:06:18.792 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:18.791+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:18.893 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:18.892+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:18.914 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:18.914 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:18.925 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:06:18.985 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:06:18.994 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:18.993+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:19.062 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:19.062 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:19.072 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:06:19.094 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:19.093+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:19.124 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:19.142 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:19.159 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:19.178 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:19.195 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:19.194+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:19.197 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:19.214 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:06:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:06:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:06:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:06:19.217 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:06:19.217 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:06:19.238 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:19.239 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:19.239 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:19.239 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:19.239 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:19.249 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:06:19.296 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:19.296+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:19.374 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:19.375 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:19.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:19.375 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:19.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.375 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:06:19.375 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:19.375 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 159.475445931, 2023-07-02T14:06:19.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:06:19.376 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:19.376 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:19.376 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:19.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:19.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:19.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:19.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:19.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:19.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:19.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:19.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:06:19.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:06:19.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:19.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:06:19.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:06:19.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:19.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:06:19.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:06:19.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:19.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:06:19.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:06:19.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:19.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:06:19.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:06:19.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:19.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:06:19.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:06:19.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:19.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:06:19.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:06:19.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:19.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:06:19.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:06:19.383 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.383 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.383 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.383 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:06:19.383 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:19.384 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 159.471360213, 2023-07-02T14:06:19.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:06:19.384 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:19.384 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:19.384 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:19.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:19.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:19.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:19.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:19.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:19.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:19.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:19.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:06:19.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:06:19.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:19.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:06:19.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:06:19.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:19.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:06:19.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:06:19.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:19.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:06:19.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:06:19.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:19.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:06:19.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:06:19.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:19.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:06:19.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:06:19.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:19.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:06:19.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:06:19.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:19.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:06:19.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:06:19.391 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.391 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.391 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.391 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:06:19.392 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:19.392 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 159.46749543199999, 2023-07-02T14:06:19.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:06:19.392 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:19.392 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:19.392 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:19.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:19.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:19.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:19.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:19.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:19.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:19.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:19.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:06:19.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:06:19.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:19.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:06:19.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:06:19.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:19.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:06:19.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:06:19.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:19.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:06:19.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:06:19.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:19.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:06:19.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:06:19.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:19.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:06:19.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:06:19.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:19.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:06:19.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:06:19.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:19.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:06:19.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:06:19.399 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.399 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.399 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.399 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6121 2.0 e81/46 2:18e1eab5:::rbd_data.125de1449605.0000000000000099:head v 81'1326, mlcod=81'1319)", 2023-07-02T14:06:19.399 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:06:19.400 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 159.463971171, 2023-07-02T14:06:19.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001, 2023-07-02T14:06:19.400 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:19.400 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:19.400 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:19.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:19.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:06:19.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:19.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:19.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909089+0000", 2023-07-02T14:06:19.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:19.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:19.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909088+0000", 2023-07-02T14:06:19.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9641999999999994e-05 2023-07-02T14:06:19.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:19.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909167+0000", 2023-07-02T14:06:19.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.182e-05 2023-07-02T14:06:19.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:19.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909179+0000", 2023-07-02T14:06:19.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9180000000000002e-06 2023-07-02T14:06:19.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.404 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:19.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909184+0000", 2023-07-02T14:06:19.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043633259999999998 2023-07-02T14:06:19.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.404 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:19.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913547+0000", 2023-07-02T14:06:19.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2974999999999999e-05 2023-07-02T14:06:19.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:19.405 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.913570+0000", 2023-07-02T14:06:19.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015457077 2023-07-02T14:06:19.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:19.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929028+0000", 2023-07-02T14:06:19.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6677000000000001e-05 2023-07-02T14:06:19.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:19.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.929064+0000", 2023-07-02T14:06:19.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019975281000000001 2023-07-02T14:06:19.407 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.407 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.407 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.408 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:06:19.408 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:19.408 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 159.428266372, 2023-07-02T14:06:19.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:06:19.409 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:19.409 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:19.410 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:19.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:19.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:19.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:19.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:19.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:19.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:19.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:19.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:06:19.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:06:19.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:19.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:06:19.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:06:19.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:19.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:06:19.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:06:19.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:19.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:06:19.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:06:19.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:19.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:06:19.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:06:19.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:19.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:06:19.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:06:19.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:19.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:06:19.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:06:19.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:19.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:06:19.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:06:19.418 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.418 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.418 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.418 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:06:19.419 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:19.419 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 159.42360901399999, 2023-07-02T14:06:19.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:06:19.419 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:19.419 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:19.419 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:19.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:19.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:19.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:19.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:19.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:19.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:19.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:19.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:06:19.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:06:19.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:19.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:06:19.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:06:19.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:19.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:06:19.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:06:19.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:19.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:06:19.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:06:19.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:19.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:06:19.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:06:19.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:19.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:06:19.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:06:19.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:19.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:06:19.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:06:19.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:19.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:06:19.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:06:19.426 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.426 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.426 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.426 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:06:19.427 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:19.427 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 159.42266915100001, 2023-07-02T14:06:19.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:06:19.427 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:19.427 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:19.427 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:19.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:19.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:19.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:19.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:19.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:19.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:19.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:19.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:06:19.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:06:19.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:19.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:06:19.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:06:19.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:19.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:06:19.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:06:19.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:19.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:06:19.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:06:19.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:19.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:06:19.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:06:19.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:19.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:06:19.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:06:19.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:19.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:06:19.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:06:19.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:19.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:06:19.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:06:19.434 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.434 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.434 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.435 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:06:19.435 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:19.435 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 159.421271273, 2023-07-02T14:06:19.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:06:19.435 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:19.435 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:19.435 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:19.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:19.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:19.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:19.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:19.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:19.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:19.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:19.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:06:19.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:06:19.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:19.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:06:19.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:06:19.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:19.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:06:19.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:06:19.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:19.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:06:19.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:06:19.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:19.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:06:19.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:06:19.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:19.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:06:19.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:06:19.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:19.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:06:19.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:06:19.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:19.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:06:19.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:06:19.443 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.443 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.443 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.443 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:06:19.443 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:19.444 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 138.01108077699999, 2023-07-02T14:06:19.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:06:19.444 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:19.444 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:19.444 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:19.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:19.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:19.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:19.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:19.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:19.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:19.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:19.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:06:19.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:06:19.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:19.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:06:19.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:06:19.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:19.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:06:19.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:06:19.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:19.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:06:19.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:06:19.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:19.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:06:19.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:06:19.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:19.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:06:19.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:06:19.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:19.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:06:19.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:06:19.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:19.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:06:19.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:06:19.451 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.451 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.451 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.452 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:19.452 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:19.452 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 136.64473109299999, 2023-07-02T14:06:19.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:06:19.452 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:19.452 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:19.452 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:19.453 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:19.453 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:19.453 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:06:19.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.453 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:19.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:19.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:19.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:19.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:19.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:19.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:19.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:19.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:06:19.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:06:19.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:19.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:06:19.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:06:19.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:19.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:06:19.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:06:19.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:19.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:06:19.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:06:19.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:19.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:06:19.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:06:19.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:19.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:06:19.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:06:19.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:19.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:06:19.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:06:19.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:19.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:06:19.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:06:19.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:19.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:06:19.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:06:19.460 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.461 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.461 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.461 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7707 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 3145728~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:19.462 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:06:19.462 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 136.64074762600001, 2023-07-02T14:06:19.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999, 2023-07-02T14:06:19.462 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:19.462 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:19.462 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:19.462 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:19.463 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:19.463 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7707 2023-07-02T14:06:19.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.463 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:19.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:19.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:06:19.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:19.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:19.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:06:19.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:19.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:19.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732312+0000", 2023-07-02T14:06:19.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5017e-05 2023-07-02T14:06:19.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:19.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732347+0000", 2023-07-02T14:06:19.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5821e-05 2023-07-02T14:06:19.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:19.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732363+0000", 2023-07-02T14:06:19.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1270000000000002e-06 2023-07-02T14:06:19.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:19.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732368+0000", 2023-07-02T14:06:19.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008444666 2023-07-02T14:06:19.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:19.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740812+0000", 2023-07-02T14:06:19.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011266599999999999 2023-07-02T14:06:19.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:19.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.740925+0000", 2023-07-02T14:06:19.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010848808999999999 2023-07-02T14:06:19.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:19.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751774+0000", 2023-07-02T14:06:19.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8421999999999997e-05 2023-07-02T14:06:19.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:19.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751832+0000", 2023-07-02T14:06:19.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107453 2023-07-02T14:06:19.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:19.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751940+0000", 2023-07-02T14:06:19.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019627209999999999 2023-07-02T14:06:19.470 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.470 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.470 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.471 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:19.471 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:19.471 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 136.640683749, 2023-07-02T14:06:19.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:06:19.471 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:19.472 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:19.472 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:19.472 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:19.472 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:19.472 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:06:19.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.473 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:19.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:19.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:19.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:19.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:19.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:19.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:19.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:19.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:19.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:06:19.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:19.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:06:19.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:06:19.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:19.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:06:19.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:06:19.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:19.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:06:19.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:06:19.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:19.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:06:19.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:06:19.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:19.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:06:19.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:06:19.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:19.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:06:19.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:06:19.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:19.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:06:19.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:06:19.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:19.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:06:19.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:06:19.480 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.480 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.480 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.480 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:19.480 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:19.481 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 136.63611992400001, 2023-07-02T14:06:19.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:06:19.481 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:19.481 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:19.481 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:19.481 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:19.481 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:19.481 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:06:19.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.482 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:19.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:19.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:19.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:19.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:19.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:19.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:19.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:19.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:06:19.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:06:19.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:19.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:06:19.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:06:19.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:19.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:06:19.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:06:19.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:19.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:06:19.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:06:19.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:19.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:06:19.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:06:19.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:19.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:06:19.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:06:19.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:19.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:06:19.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:06:19.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:19.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:06:19.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:06:19.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:19.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:06:19.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:06:19.489 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.489 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.490 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.490 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:19.490 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:19.490 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 136.636070058, 2023-07-02T14:06:19.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:06:19.491 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:19.491 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:19.491 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:19.491 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:19.491 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:19.491 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:06:19.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.491 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:19.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:19.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:19.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:19.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:19.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:19.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:19.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:19.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:06:19.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:06:19.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:19.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:06:19.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:06:19.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:19.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:06:19.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:06:19.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:19.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:06:19.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:06:19.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:19.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:06:19.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:06:19.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:19.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:06:19.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:06:19.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:19.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:06:19.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:06:19.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:19.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:06:19.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:06:19.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:19.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:06:19.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:06:19.499 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.499 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.499 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.500 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:19.500 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:19.500 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 136.63503106499999, 2023-07-02T14:06:19.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:06:19.500 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:19.500 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:19.500 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:19.501 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:19.501 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:19.501 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:06:19.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.501 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:19.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:19.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:19.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:19.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:19.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:19.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:19.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:19.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:06:19.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:06:19.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:19.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:06:19.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:06:19.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:19.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:06:19.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:06:19.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:19.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:06:19.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:06:19.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:19.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:06:19.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:06:19.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:19.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:06:19.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:06:19.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:19.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:06:19.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:06:19.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:19.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:06:19.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:06:19.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:19.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:06:19.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:06:19.508 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.509 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.509 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.509 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7993 2.7 2:fe72db53:::rbd_data.125de1449605.0000000000000072:head [write 65536~3276800] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:19.509 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:06:19.509 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 136.44807301700001, 2023-07-02T14:06:19.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919, 2023-07-02T14:06:19.510 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:19.510 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:19.510 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:19.510 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:19.510 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:19.510 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7993 2023-07-02T14:06:19.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.511 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:19.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:19.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:06:19.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:19.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:19.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924987+0000", 2023-07-02T14:06:19.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:19.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:19.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.924986+0000", 2023-07-02T14:06:19.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049542070000000004 2023-07-02T14:06:19.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:19.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929940+0000", 2023-07-02T14:06:19.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9740000000000001e-05 2023-07-02T14:06:19.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:19.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929960+0000", 2023-07-02T14:06:19.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.311e-05 2023-07-02T14:06:19.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:19.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.929973+0000", 2023-07-02T14:06:19.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9629000000000002e-05 2023-07-02T14:06:19.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:19.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930023+0000", 2023-07-02T14:06:19.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012751799999999999 2023-07-02T14:06:19.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:19.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.930150+0000", 2023-07-02T14:06:19.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016449042000000001 2023-07-02T14:06:19.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:19.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946599+0000", 2023-07-02T14:06:19.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3241e-05 2023-07-02T14:06:19.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:19.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946663+0000", 2023-07-02T14:06:19.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000102296 2023-07-02T14:06:19.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:19.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.946765+0000", 2023-07-02T14:06:19.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021777919 2023-07-02T14:06:19.518 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.518 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.518 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.519 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11782 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 196608~2162688] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:06:19.519 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:06:19.519 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 94.963553094999995, 2023-07-02T14:06:19.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365, 2023-07-02T14:06:19.519 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:19.519 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:19.519 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:19.520 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:19.520 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:19.520 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11782 2023-07-02T14:06:19.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.520 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:19.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:19.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:06:19.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:19.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:19.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409507+0000", 2023-07-02T14:06:19.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:19.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:19.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.409506+0000", 2023-07-02T14:06:19.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020172269999999999 2023-07-02T14:06:19.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:19.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411523+0000", 2023-07-02T14:06:19.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8901000000000001e-05 2023-07-02T14:06:19.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:19.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411542+0000", 2023-07-02T14:06:19.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5037e-05 2023-07-02T14:06:19.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:19.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.411557+0000", 2023-07-02T14:06:19.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001931213 2023-07-02T14:06:19.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:19.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413489+0000", 2023-07-02T14:06:19.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000149207 2023-07-02T14:06:19.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:19.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413638+0000", 2023-07-02T14:06:19.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024752800000000001 2023-07-02T14:06:19.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:19.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.413885+0000", 2023-07-02T14:06:19.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048516000000000002 2023-07-02T14:06:19.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:19.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.418737+0000", 2023-07-02T14:06:19.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010319259000000001 2023-07-02T14:06:19.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:19.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429056+0000", 2023-07-02T14:06:19.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2518000000000002e-05 2023-07-02T14:06:19.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:19.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429099+0000", 2023-07-02T14:06:19.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015275 2023-07-02T14:06:19.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:19.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429251+0000", 2023-07-02T14:06:19.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019744365 2023-07-02T14:06:19.529 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.529 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.529 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.530 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPushReply(2.7 119/117 [PushReplyOp(2:eb862bc0:::rbd_data.125de1449605.0000000000000040:13)])", 2023-07-02T14:06:19.530 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:06:19.530 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 94.962389819999999, 2023-07-02T14:06:19.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999, 2023-07-02T14:06:19.530 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:19.530 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "reached pg", 2023-07-02T14:06:19.530 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:19.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:19.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:06:19.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:19.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:19.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:06:19.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:19.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:19.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410670+0000", 2023-07-02T14:06:19.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7670000000000001e-06 2023-07-02T14:06:19.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:19.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410674+0000", 2023-07-02T14:06:19.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3973e-05 2023-07-02T14:06:19.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:19.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410688+0000", 2023-07-02T14:06:19.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0157e-05 2023-07-02T14:06:19.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:19.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.410708+0000", 2023-07-02T14:06:19.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018578779 2023-07-02T14:06:19.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:19.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.429287+0000", 2023-07-02T14:06:19.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0013086549999999999 2023-07-02T14:06:19.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:19.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.430595+0000", 2023-07-02T14:06:19.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019924931999999999 2023-07-02T14:06:19.536 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.536 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.536 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.537 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:11861 2.7 2:e57d3f55:::rbd_data.125de1449605.0000000000000134:head [write 0~65536] snapc 1e=[1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e119)", 2023-07-02T14:06:19.537 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:06:19.537 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 94.828274442999998, 2023-07-02T14:06:19.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723, 2023-07-02T14:06:19.537 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:19.537 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:19.537 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:19.537 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:19.538 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:19.538 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 11861 2023-07-02T14:06:19.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.538 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:19.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:19.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:06:19.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:19.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:19.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544786+0000", 2023-07-02T14:06:19.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:19.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:19.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544785+0000", 2023-07-02T14:06:19.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010563700000000001 2023-07-02T14:06:19.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:19.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544890+0000", 2023-07-02T14:06:19.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8530000000000004e-06 2023-07-02T14:06:19.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:19.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544900+0000", 2023-07-02T14:06:19.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.665e-06 2023-07-02T14:06:19.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:19.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.544906+0000", 2023-07-02T14:06:19.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000272575 2023-07-02T14:06:19.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:19.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545178+0000", 2023-07-02T14:06:19.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000111362 2023-07-02T14:06:19.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:19.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545290+0000", 2023-07-02T14:06:19.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016181100000000001 2023-07-02T14:06:19.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:19.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.545452+0000", 2023-07-02T14:06:19.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044500750000000004 2023-07-02T14:06:19.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:19.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.549902+0000", 2023-07-02T14:06:19.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014850024 2023-07-02T14:06:19.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:19.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564752+0000", 2023-07-02T14:06:19.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6272000000000003e-05 2023-07-02T14:06:19.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:19.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.564848+0000", 2023-07-02T14:06:19.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000162428 2023-07-02T14:06:19.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:19.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:44.565010+0000", 2023-07-02T14:06:19.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020224723 2023-07-02T14:06:19.547 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.547 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.547 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.548 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:13262 2.7 e125/121 2:e0741dde:::rbd_data.125de1449605.00000000000000ad:head v 125'2024, mlcod=125'2021)", 2023-07-02T14:06:19.548 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:06:19.548 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 86.122196704999993, 2023-07-02T14:06:19.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999, 2023-07-02T14:06:19.548 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:19.548 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:19.548 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:19.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:19.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:06:19.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:19.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:19.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250863+0000", 2023-07-02T14:06:19.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:19.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:19.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.250862+0000", 2023-07-02T14:06:19.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080378800000000007 2023-07-02T14:06:19.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:19.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258900+0000", 2023-07-02T14:06:19.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8939999999999999e-06 2023-07-02T14:06:19.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:19.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258909+0000", 2023-07-02T14:06:19.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3560000000000003e-06 2023-07-02T14:06:19.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:19.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258914+0000", 2023-07-02T14:06:19.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2429000000000003e-05 2023-07-02T14:06:19.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:19.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258956+0000", 2023-07-02T14:06:19.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5094999999999999e-05 2023-07-02T14:06:19.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:19.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.258981+0000", 2023-07-02T14:06:19.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012177307 2023-07-02T14:06:19.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:19.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271158+0000", 2023-07-02T14:06:19.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2104999999999999e-05 2023-07-02T14:06:19.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:19.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:19.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:19.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:53.271181+0000", 2023-07-02T14:06:19.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020317116999999999 2023-07-02T14:06:19.555 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.555 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.555 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.555 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:19.555 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:19.555 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:19.557 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:19.555+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:19.557 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:06:19.657 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:19.656+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:19.681 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:19.682 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:19.682 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:06:19.682 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:19.692 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:06:19.775 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:19.774+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:19.855 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:19.855 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:19.855 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:19.855 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:19.856 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:19.858 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:19.858+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:19.865 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:06:19.960 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:19.959+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:20.030 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:06:20.031 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 220.73806505600001, 2023-07-02T14:06:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:06:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:06:20.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:06:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:06:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:06:20.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:06:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:06:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:06:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:06:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:06:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:06:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:06:20.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:06:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:06:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:06:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:06:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:06:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:06:20.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:06:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:06:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:06:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:06:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:06:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:06:20.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:06:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:06:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:06:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 220.73783003299999, 2023-07-02T14:06:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:06:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.042 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:06:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:06:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:06:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:06:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:06:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:06:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:06:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:06:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:06:20.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:06:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:06:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:06:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:06:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:06:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:06:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:06:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:06:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:06:20.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:06:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:06:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:06:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:06:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:06:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:06:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:06:20.051 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:06:20.052 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 220.73763089900001, 2023-07-02T14:06:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:06:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:06:20.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:06:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:06:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:06:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:06:20.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:06:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:06:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:06:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:06:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:06:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:06:20.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:06:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:06:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:06:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:06:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:06:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:06:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:06:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:06:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:06:20.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:06:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:06:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:06:20.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:06:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:06:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:06:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 220.737265775, 2023-07-02T14:06:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:06:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.063 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:06:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:06:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:06:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:06:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:06:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:06:20.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:06:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:06:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:06:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:06:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:06:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:06:20.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:06:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:06:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:06:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:06:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:06:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:06:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:06:20.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:06:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:06:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:06:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:06:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:06:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:06:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:06:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 220.73718471399999, 2023-07-02T14:06:20.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:06:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:06:20.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:06:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:06:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:06:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:06:20.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:06:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:06:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:06:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:06:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:06:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:06:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:06:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:06:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:06:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:06:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:06:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:06:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:06:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:06:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:06:20.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:06:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:06:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:06:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:06:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:06:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.083 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:06:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 220.73709304299999, 2023-07-02T14:06:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:06:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.084 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:06:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:06:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:06:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:06:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:06:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:06:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:06:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:06:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:06:20.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:06:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:06:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:06:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:06:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:06:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:06:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:06:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:06:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:06:20.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:06:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:06:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:06:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:06:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:06:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:06:20.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:06:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:20.094 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:06:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 220.736753601, 2023-07-02T14:06:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:06:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.095 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:06:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:06:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:06:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:06:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:06:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:06:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:06:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:06:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:06:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:06:20.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:06:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:06:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:06:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:06:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:06:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:06:20.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:06:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:06:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:06:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:06:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:06:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:06:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:06:20.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:06:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:06:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 220.73656754800001, 2023-07-02T14:06:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:06:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.105 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:06:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:06:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:06:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:06:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:06:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:06:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:06:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:06:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:06:20.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:06:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:06:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:06:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:06:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:06:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:06:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:06:20.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:06:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:06:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:06:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:06:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:06:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:06:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.114 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:06:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 220.73640974599999, 2023-07-02T14:06:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:06:20.115 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:06:20.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:06:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:06:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:06:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:06:20.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:06:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:06:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:06:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:06:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:06:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:06:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:06:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:06:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:06:20.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:06:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:06:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:06:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:06:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:06:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:06:20.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:06:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:06:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:06:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:06:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:06:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.125 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:06:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 220.73621796399999, 2023-07-02T14:06:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:06:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.126 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:06:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:06:20.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:06:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:20.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:06:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:06:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:06:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:06:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:06:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:06:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:06:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:06:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:06:20.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:06:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:06:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:06:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:06:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:06:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:06:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:06:20.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:06:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:06:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:06:20.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:06:20.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:06:20.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:06:20.135 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.136 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.136 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.136 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:20.136 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:20.136 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 220.73604230399999, 2023-07-02T14:06:20.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:06:20.137 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.137 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.137 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.137 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.137 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.137 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:06:20.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.137 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:20.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:20.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:20.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:20.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:06:20.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:06:20.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:06:20.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:06:20.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:06:20.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:06:20.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:06:20.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:06:20.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:06:20.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:06:20.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:06:20.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:20.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:06:20.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:06:20.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:06:20.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:06:20.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:06:20.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:06:20.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:06:20.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:06:20.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:06:20.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:06:20.146 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.146 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.146 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.146 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:20.147 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:06:20.147 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 220.735948116, 2023-07-02T14:06:20.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:06:20.147 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.147 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.147 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.147 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.147 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.148 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:06:20.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.148 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:06:20.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:06:20.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:20.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:06:20.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:06:20.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:06:20.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:06:20.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:06:20.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:06:20.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:06:20.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:06:20.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:06:20.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:06:20.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:06:20.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:06:20.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:20.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:06:20.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:06:20.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:06:20.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:06:20.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:06:20.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:06:20.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:06:20.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:06:20.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:06:20.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:06:20.156 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.156 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.156 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.157 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:20.157 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:06:20.157 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 220.73574472999999, 2023-07-02T14:06:20.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:06:20.157 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.157 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.158 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.158 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.158 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.158 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:06:20.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.158 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:06:20.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:06:20.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:20.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:06:20.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:06:20.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:06:20.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:06:20.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:06:20.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:06:20.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:06:20.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:06:20.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:06:20.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:06:20.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:06:20.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:06:20.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:20.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:06:20.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:06:20.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:06:20.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:06:20.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:06:20.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:06:20.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:06:20.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:06:20.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:06:20.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:06:20.167 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.167 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.167 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.167 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:20.168 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:06:20.168 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 220.73559407400001, 2023-07-02T14:06:20.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:06:20.168 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.168 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.168 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.168 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.168 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.169 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:06:20.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.169 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:06:20.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:06:20.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:20.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:06:20.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:06:20.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:06:20.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:06:20.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:06:20.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:06:20.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:06:20.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:06:20.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:06:20.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:06:20.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:06:20.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:06:20.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:20.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:06:20.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:06:20.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:06:20.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:06:20.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:06:20.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:06:20.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:06:20.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:06:20.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:06:20.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:06:20.177 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.177 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.177 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.178 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:20.178 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:20.178 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 220.735521371, 2023-07-02T14:06:20.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:06:20.178 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.178 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.179 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.179 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.179 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.179 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:06:20.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.179 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:20.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:20.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:20.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:20.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:06:20.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:06:20.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:06:20.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:06:20.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:06:20.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:06:20.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:06:20.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:06:20.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:06:20.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:06:20.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:06:20.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:20.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:06:20.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:06:20.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:06:20.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:06:20.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:06:20.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:06:20.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:06:20.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:06:20.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:06:20.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:06:20.188 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.188 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.188 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.188 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:20.188 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:20.189 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 220.735404381, 2023-07-02T14:06:20.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:06:20.189 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.189 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.189 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.189 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.189 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.190 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:06:20.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.190 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:20.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:20.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:20.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:20.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:06:20.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:06:20.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:06:20.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:06:20.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:06:20.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:06:20.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:06:20.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:06:20.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:06:20.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:06:20.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:06:20.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:20.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:06:20.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:06:20.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:06:20.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:06:20.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:06:20.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:06:20.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:06:20.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:06:20.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:06:20.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:06:20.198 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.198 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.199 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.199 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:20.199 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:06:20.199 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 220.73525493, 2023-07-02T14:06:20.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:06:20.199 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.200 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.200 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.200 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.200 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.200 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:06:20.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.200 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:06:20.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:06:20.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:20.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:06:20.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:06:20.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:06:20.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:06:20.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:06:20.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:06:20.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:06:20.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:06:20.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:06:20.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:06:20.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:06:20.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:06:20.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:20.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:06:20.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:06:20.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:06:20.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:06:20.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:06:20.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:06:20.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:06:20.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:06:20.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:06:20.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:06:20.209 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.209 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.209 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.209 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:20.209 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:06:20.209 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 220.71377895699999, 2023-07-02T14:06:20.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:06:20.210 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.210 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.210 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.210 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.210 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.210 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:06:20.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.211 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:06:20.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:06:20.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:20.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:06:20.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:06:20.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:06:20.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:06:20.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:06:20.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:06:20.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:06:20.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:06:20.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:06:20.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:06:20.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:06:20.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:06:20.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:20.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:06:20.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:06:20.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:06:20.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:06:20.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:06:20.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:06:20.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:06:20.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:06:20.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:06:20.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:06:20.219 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.219 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.219 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.220 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:20.220 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:06:20.220 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 220.71364642099999, 2023-07-02T14:06:20.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:06:20.220 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.220 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.220 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.221 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.221 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.221 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:06:20.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.221 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.221 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:06:20.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:06:20.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:20.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:06:20.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:06:20.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:06:20.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:06:20.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:06:20.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:06:20.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:06:20.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:06:20.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:06:20.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:06:20.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:06:20.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:06:20.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:20.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:06:20.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:06:20.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:06:20.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:06:20.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:06:20.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:06:20.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:06:20.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:06:20.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:06:20.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:06:20.230 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.230 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.230 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.230 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:20.230 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:06:20.230 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 220.65230976000001, 2023-07-02T14:06:20.231 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:06:20.231 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.231 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.231 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.231 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.231 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.231 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:06:20.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.232 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:06:20.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:06:20.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:20.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:06:20.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:06:20.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:06:20.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:06:20.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:06:20.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:06:20.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:06:20.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:06:20.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:06:20.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:06:20.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:06:20.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:06:20.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:20.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:06:20.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:06:20.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:06:20.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:06:20.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:06:20.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:06:20.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:06:20.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:06:20.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:06:20.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:06:20.240 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.240 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.240 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.240 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.240 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.241 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:20.242 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:06:20.242 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:20.240+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:20.342 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:20.341+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:20.368 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:20.368 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:20.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.368 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20768 2.7 2.1292458f (undecoded) ondisk+write+known_if_redirected e183)", 2023-07-02T14:06:20.368 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:20.368 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 1.1455629540000001, 2023-07-02T14:06:20.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1455846970000001, 2023-07-02T14:06:20.369 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.369 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "delayed", 2023-07-02T14:06:20.369 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.369 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.369 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.369 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20768 2023-07-02T14:06:20.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.370 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:20.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:20.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:20.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:20.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7829999999999998e-06 2023-07-02T14:06:20.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222419+0000", 2023-07-02T14:06:20.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1380000000000004e-06 2023-07-02T14:06:20.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222425+0000", 2023-07-02T14:06:20.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2519999999999996e-06 2023-07-02T14:06:20.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222431+0000", 2023-07-02T14:06:20.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2157999999999994e-05 2023-07-02T14:06:20.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222523+0000", 2023-07-02T14:06:20.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5147000000000001e-05 2023-07-02T14:06:20.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:20.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222538+0000", 2023-07-02T14:06:20.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30105419700000002 2023-07-02T14:06:20.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523592+0000", 2023-07-02T14:06:20.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2316e-05 2023-07-02T14:06:20.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:20.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523605+0000", 2023-07-02T14:06:20.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30119029400000003 2023-07-02T14:06:20.376 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.376 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20769 2.7 2.1292458f (undecoded) ondisk+write+known_if_redirected e184)", 2023-07-02T14:06:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 1.145200499, 2023-07-02T14:06:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.145278207, 2023-07-02T14:06:20.377 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.378 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "delayed", 2023-07-02T14:06:20.378 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.378 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.378 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.378 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20769 2023-07-02T14:06:20.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.378 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:20.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:20.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:20.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222776+0000", 2023-07-02T14:06:20.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9710000000000003e-06 2023-07-02T14:06:20.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222781+0000", 2023-07-02T14:06:20.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9899999999999992e-06 2023-07-02T14:06:20.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222791+0000", 2023-07-02T14:06:20.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4240000000000003e-06 2023-07-02T14:06:20.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222795+0000", 2023-07-02T14:06:20.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2366000000000001e-05 2023-07-02T14:06:20.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222828+0000", 2023-07-02T14:06:20.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4790000000000001e-06 2023-07-02T14:06:20.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:20.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222834+0000", 2023-07-02T14:06:20.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30084589299999998 2023-07-02T14:06:20.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523680+0000", 2023-07-02T14:06:20.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.87e-06 2023-07-02T14:06:20.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:20.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523685+0000", 2023-07-02T14:06:20.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30090815199999998 2023-07-02T14:06:20.385 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.385 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.385 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.385 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20770 2.7 2.1292458f (undecoded) ondisk+write+known_if_redirected e185)", 2023-07-02T14:06:20.386 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:20.386 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.84398338900000003, 2023-07-02T14:06:20.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.84410907099999999, 2023-07-02T14:06:20.386 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.386 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "delayed", 2023-07-02T14:06:20.386 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.386 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.386 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.387 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20770 2023-07-02T14:06:20.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.387 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:20.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:20.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:20.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523992+0000", 2023-07-02T14:06:20.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6720000000000004e-06 2023-07-02T14:06:20.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524000+0000", 2023-07-02T14:06:20.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3711e-05 2023-07-02T14:06:20.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524014+0000", 2023-07-02T14:06:20.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7849999999999992e-06 2023-07-02T14:06:20.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524022+0000", 2023-07-02T14:06:20.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2366000000000001e-05 2023-07-02T14:06:20.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524054+0000", 2023-07-02T14:06:20.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8169999999999998e-06 2023-07-02T14:06:20.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:20.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524059+0000", 2023-07-02T14:06:20.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4745000000000004e-05 2023-07-02T14:06:20.392 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.392 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.392 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.393 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.393 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2023-07-02T14:06:20.393 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 3 2023-07-02T14:06:20.393 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:20.394 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:06:20.443 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:20.442+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:20.517 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:20.517 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:20.517 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:20.518 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:20.518 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:20.528 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:06:20.536 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:06:20.570 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:20.569+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:20.645 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:20.644+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:20.684 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:20.683+0000 7fca04ca1700 0 -- 172.21.15.163:0/3583344905 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x55a3438be310 msgr2=0x55a3438c0700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:20.698 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:20.697+0000 7fca04ca1700 0 -- 172.21.15.163:0/3583344905 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7fc9e8037df0 msgr2=0x7fc9e803a1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:20.715 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:20.715 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:20.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:20.716 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:20.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.716 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:16941 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:20.716 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:06:20.716 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.853615617999999, 2023-07-02T14:06:20.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11449124500000001, 2023-07-02T14:06:20.716 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.717 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.717 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.717 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.717 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.717 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 16941 2023-07-02T14:06:20.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.717 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:06:20.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:06:20.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:06:20.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855877+0000", 2023-07-02T14:06:20.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3209999999999999e-06 2023-07-02T14:06:20.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855880+0000", 2023-07-02T14:06:20.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9299999999999999e-06 2023-07-02T14:06:20.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855883+0000", 2023-07-02T14:06:20.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6309999999999999e-06 2023-07-02T14:06:20.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855885+0000", 2023-07-02T14:06:20.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050516850000000002 2023-07-02T14:06:20.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.860937+0000", 2023-07-02T14:06:20.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1844999999999997e-05 2023-07-02T14:06:20.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.860979+0000", 2023-07-02T14:06:20.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055853480000000004 2023-07-02T14:06:20.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.866564+0000", 2023-07-02T14:06:20.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2821999999999998e-05 2023-07-02T14:06:20.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.866597+0000", 2023-07-02T14:06:20.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050938900000000002 2023-07-02T14:06:20.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.871691+0000", 2023-07-02T14:06:20.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1210000000000001e-05 2023-07-02T14:06:20.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.871722+0000", 2023-07-02T14:06:20.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014768687000000001 2023-07-02T14:06:20.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.886491+0000", 2023-07-02T14:06:20.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7248000000000003e-05 2023-07-02T14:06:20.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.886568+0000", 2023-07-02T14:06:20.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015403168 2023-07-02T14:06:20.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.901971+0000", 2023-07-02T14:06:20.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4035000000000002e-05 2023-07-02T14:06:20.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.902005+0000", 2023-07-02T14:06:20.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014618580000000001 2023-07-02T14:06:20.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.916624+0000", 2023-07-02T14:06:20.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.443e-05 2023-07-02T14:06:20.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.916668+0000", 2023-07-02T14:06:20.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099598989999999995 2023-07-02T14:06:20.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.926628+0000", 2023-07-02T14:06:20.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3123999999999999e-05 2023-07-02T14:06:20.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.926711+0000", 2023-07-02T14:06:20.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046031930000000002 2023-07-02T14:06:20.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.931314+0000", 2023-07-02T14:06:20.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3257000000000003e-05 2023-07-02T14:06:20.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.931358+0000", 2023-07-02T14:06:20.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078370969999999995 2023-07-02T14:06:20.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.939195+0000", 2023-07-02T14:06:20.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2522000000000003e-05 2023-07-02T14:06:20.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.939247+0000", 2023-07-02T14:06:20.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01118103 2023-07-02T14:06:20.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.950428+0000", 2023-07-02T14:06:20.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5001999999999996e-05 2023-07-02T14:06:20.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.950523+0000", 2023-07-02T14:06:20.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089648390000000005 2023-07-02T14:06:20.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959488+0000", 2023-07-02T14:06:20.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4569e-05 2023-07-02T14:06:20.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959533+0000", 2023-07-02T14:06:20.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026104299999999997 2023-07-02T14:06:20.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:20.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959794+0000", 2023-07-02T14:06:20.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025221789999999998 2023-07-02T14:06:20.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.962316+0000", 2023-07-02T14:06:20.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079080520000000005 2023-07-02T14:06:20.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970224+0000", 2023-07-02T14:06:20.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1265e-05 2023-07-02T14:06:20.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970255+0000", 2023-07-02T14:06:20.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000113581 2023-07-02T14:06:20.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970369+0000", 2023-07-02T14:06:20.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11449124500000001 2023-07-02T14:06:20.739 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.739 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.739 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.739 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17049 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:20.740 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:06:20.740 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.730110627000002, 2023-07-02T14:06:20.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.112277108, 2023-07-02T14:06:20.740 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.740 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.740 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.740 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.740 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.741 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17049 2023-07-02T14:06:20.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.741 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:06:20.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:06:20.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:20.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979382+0000", 2023-07-02T14:06:20.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3450000000000002e-06 2023-07-02T14:06:20.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979385+0000", 2023-07-02T14:06:20.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9000000000000003e-06 2023-07-02T14:06:20.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979391+0000", 2023-07-02T14:06:20.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9999999999999998e-06 2023-07-02T14:06:20.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979395+0000", 2023-07-02T14:06:20.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013089412 2023-07-02T14:06:20.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992485+0000", 2023-07-02T14:06:20.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3828999999999997e-05 2023-07-02T14:06:20.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992519+0000", 2023-07-02T14:06:20.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062549889999999999 2023-07-02T14:06:20.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998774+0000", 2023-07-02T14:06:20.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4870999999999995e-05 2023-07-02T14:06:20.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998848+0000", 2023-07-02T14:06:20.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091715140000000004 2023-07-02T14:06:20.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008020+0000", 2023-07-02T14:06:20.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8659999999999999e-05 2023-07-02T14:06:20.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008059+0000", 2023-07-02T14:06:20.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059565110000000003 2023-07-02T14:06:20.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014015+0000", 2023-07-02T14:06:20.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8088000000000002e-05 2023-07-02T14:06:20.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014093+0000", 2023-07-02T14:06:20.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007513395 2023-07-02T14:06:20.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021607+0000", 2023-07-02T14:06:20.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9777e-05 2023-07-02T14:06:20.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021636+0000", 2023-07-02T14:06:20.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056999549999999996 2023-07-02T14:06:20.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027336+0000", 2023-07-02T14:06:20.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8356999999999996e-05 2023-07-02T14:06:20.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027405+0000", 2023-07-02T14:06:20.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047942130000000003 2023-07-02T14:06:20.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032199+0000", 2023-07-02T14:06:20.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1137999999999999e-05 2023-07-02T14:06:20.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032230+0000", 2023-07-02T14:06:20.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057056759999999998 2023-07-02T14:06:20.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.037936+0000", 2023-07-02T14:06:20.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6334999999999993e-05 2023-07-02T14:06:20.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038012+0000", 2023-07-02T14:06:20.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043985609999999996 2023-07-02T14:06:20.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042411+0000", 2023-07-02T14:06:20.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6857e-05 2023-07-02T14:06:20.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042487+0000", 2023-07-02T14:06:20.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007284446 2023-07-02T14:06:20.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049772+0000", 2023-07-02T14:06:20.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3757000000000006e-05 2023-07-02T14:06:20.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049846+0000", 2023-07-02T14:06:20.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0095186679999999992 2023-07-02T14:06:20.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059364+0000", 2023-07-02T14:06:20.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9931000000000005e-05 2023-07-02T14:06:20.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059434+0000", 2023-07-02T14:06:20.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079904920000000001 2023-07-02T14:06:20.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067425+0000", 2023-07-02T14:06:20.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5583000000000001e-05 2023-07-02T14:06:20.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067460+0000", 2023-07-02T14:06:20.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009114278 2023-07-02T14:06:20.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076575+0000", 2023-07-02T14:06:20.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5880000000000001e-05 2023-07-02T14:06:20.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076620+0000", 2023-07-02T14:06:20.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055118470000000003 2023-07-02T14:06:20.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082132+0000", 2023-07-02T14:06:20.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8306999999999997e-05 2023-07-02T14:06:20.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082181+0000", 2023-07-02T14:06:20.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026259599999999998 2023-07-02T14:06:20.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:20.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082443+0000", 2023-07-02T14:06:20.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018855339999999999 2023-07-02T14:06:20.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084329+0000", 2023-07-02T14:06:20.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071877729999999997 2023-07-02T14:06:20.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091517+0000", 2023-07-02T14:06:20.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3424999999999999e-05 2023-07-02T14:06:20.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091550+0000", 2023-07-02T14:06:20.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109724 2023-07-02T14:06:20.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091660+0000", 2023-07-02T14:06:20.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.112277108 2023-07-02T14:06:20.766 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.766 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.766 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.767 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17050 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:20.767 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:06:20.767 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.730093701999998, 2023-07-02T14:06:20.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12076701300000001, 2023-07-02T14:06:20.767 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.767 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.768 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.768 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.768 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.768 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17050 2023-07-02T14:06:20.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.768 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:06:20.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:06:20.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:20.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:06:20.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0759999999999999e-06 2023-07-02T14:06:20.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979402+0000", 2023-07-02T14:06:20.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0480000000000004e-06 2023-07-02T14:06:20.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979408+0000", 2023-07-02T14:06:20.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9559999999999999e-06 2023-07-02T14:06:20.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979412+0000", 2023-07-02T14:06:20.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013117658000000001 2023-07-02T14:06:20.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992530+0000", 2023-07-02T14:06:20.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4464000000000001e-05 2023-07-02T14:06:20.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992564+0000", 2023-07-02T14:06:20.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0063143640000000003 2023-07-02T14:06:20.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998879+0000", 2023-07-02T14:06:20.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2842000000000002e-05 2023-07-02T14:06:20.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998911+0000", 2023-07-02T14:06:20.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091863190000000001 2023-07-02T14:06:20.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008098+0000", 2023-07-02T14:06:20.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1448999999999999e-05 2023-07-02T14:06:20.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008179+0000", 2023-07-02T14:06:20.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059353720000000004 2023-07-02T14:06:20.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014115+0000", 2023-07-02T14:06:20.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1928000000000002e-05 2023-07-02T14:06:20.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014157+0000", 2023-07-02T14:06:20.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074935210000000004 2023-07-02T14:06:20.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021650+0000", 2023-07-02T14:06:20.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0199999999999999e-05 2023-07-02T14:06:20.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021680+0000", 2023-07-02T14:06:20.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057425109999999996 2023-07-02T14:06:20.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027423+0000", 2023-07-02T14:06:20.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1498999999999998e-05 2023-07-02T14:06:20.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027454+0000", 2023-07-02T14:06:20.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047903420000000004 2023-07-02T14:06:20.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032245+0000", 2023-07-02T14:06:20.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2552999999999998e-05 2023-07-02T14:06:20.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032277+0000", 2023-07-02T14:06:20.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057638079999999996 2023-07-02T14:06:20.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038041+0000", 2023-07-02T14:06:20.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9168000000000002e-05 2023-07-02T14:06:20.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038080+0000", 2023-07-02T14:06:20.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044291799999999996 2023-07-02T14:06:20.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042509+0000", 2023-07-02T14:06:20.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3528999999999997e-05 2023-07-02T14:06:20.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042543+0000", 2023-07-02T14:06:20.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073244729999999997 2023-07-02T14:06:20.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049867+0000", 2023-07-02T14:06:20.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2873999999999997e-05 2023-07-02T14:06:20.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049900+0000", 2023-07-02T14:06:20.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0095670149999999999 2023-07-02T14:06:20.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059467+0000", 2023-07-02T14:06:20.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0406000000000003e-05 2023-07-02T14:06:20.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059538+0000", 2023-07-02T14:06:20.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079635079999999994 2023-07-02T14:06:20.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067501+0000", 2023-07-02T14:06:20.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4795999999999997e-05 2023-07-02T14:06:20.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067536+0000", 2023-07-02T14:06:20.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091231969999999996 2023-07-02T14:06:20.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076659+0000", 2023-07-02T14:06:20.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3208999999999999e-05 2023-07-02T14:06:20.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076702+0000", 2023-07-02T14:06:20.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070660480000000001 2023-07-02T14:06:20.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.083768+0000", 2023-07-02T14:06:20.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5546000000000001e-05 2023-07-02T14:06:20.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.083814+0000", 2023-07-02T14:06:20.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078830659999999993 2023-07-02T14:06:20.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091697+0000", 2023-07-02T14:06:20.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5921000000000003e-05 2023-07-02T14:06:20.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091743+0000", 2023-07-02T14:06:20.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026571400000000001 2023-07-02T14:06:20.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:20.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.092009+0000", 2023-07-02T14:06:20.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022058429999999999 2023-07-02T14:06:20.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.094214+0000", 2023-07-02T14:06:20.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057115869999999997 2023-07-02T14:06:20.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.099926+0000", 2023-07-02T14:06:20.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0287999999999997e-05 2023-07-02T14:06:20.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.099976+0000", 2023-07-02T14:06:20.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019014999999999999 2023-07-02T14:06:20.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.100166+0000", 2023-07-02T14:06:20.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12076701300000001 2023-07-02T14:06:20.794 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.794 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.795 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.795 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17353 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:20.795 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:06:20.795 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.406909026000001, 2023-07-02T14:06:20.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.115724379, 2023-07-02T14:06:20.795 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.796 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.796 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.796 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.796 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.796 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17353 2023-07-02T14:06:20.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.796 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:06:20.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:06:20.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:20.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:06:20.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.41e-06 2023-07-02T14:06:20.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302587+0000", 2023-07-02T14:06:20.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.015e-06 2023-07-02T14:06:20.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302593+0000", 2023-07-02T14:06:20.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8019999999999997e-06 2023-07-02T14:06:20.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302599+0000", 2023-07-02T14:06:20.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022701199999999999 2023-07-02T14:06:20.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302826+0000", 2023-07-02T14:06:20.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.426e-05 2023-07-02T14:06:20.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302870+0000", 2023-07-02T14:06:20.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096759489999999997 2023-07-02T14:06:20.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312546+0000", 2023-07-02T14:06:20.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5785000000000006e-05 2023-07-02T14:06:20.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312632+0000", 2023-07-02T14:06:20.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098106970000000002 2023-07-02T14:06:20.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322443+0000", 2023-07-02T14:06:20.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3161999999999999e-05 2023-07-02T14:06:20.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322476+0000", 2023-07-02T14:06:20.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010015823 2023-07-02T14:06:20.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332492+0000", 2023-07-02T14:06:20.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1165999999999999e-05 2023-07-02T14:06:20.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332533+0000", 2023-07-02T14:06:20.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089435269999999997 2023-07-02T14:06:20.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341476+0000", 2023-07-02T14:06:20.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000112691 2023-07-02T14:06:20.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341589+0000", 2023-07-02T14:06:20.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008147362 2023-07-02T14:06:20.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349736+0000", 2023-07-02T14:06:20.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5149000000000007e-05 2023-07-02T14:06:20.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349832+0000", 2023-07-02T14:06:20.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089226519999999997 2023-07-02T14:06:20.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358754+0000", 2023-07-02T14:06:20.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6091999999999998e-05 2023-07-02T14:06:20.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358810+0000", 2023-07-02T14:06:20.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080318000000000004 2023-07-02T14:06:20.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366842+0000", 2023-07-02T14:06:20.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4321999999999998e-05 2023-07-02T14:06:20.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366916+0000", 2023-07-02T14:06:20.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074664049999999997 2023-07-02T14:06:20.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374383+0000", 2023-07-02T14:06:20.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5460000000000001e-05 2023-07-02T14:06:20.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374438+0000", 2023-07-02T14:06:20.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091562450000000004 2023-07-02T14:06:20.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383594+0000", 2023-07-02T14:06:20.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2545000000000004e-05 2023-07-02T14:06:20.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383667+0000", 2023-07-02T14:06:20.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013424551 2023-07-02T14:06:20.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397092+0000", 2023-07-02T14:06:20.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6991999999999997e-05 2023-07-02T14:06:20.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397169+0000", 2023-07-02T14:06:20.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064399139999999997 2023-07-02T14:06:20.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403608+0000", 2023-07-02T14:06:20.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2614000000000006e-05 2023-07-02T14:06:20.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403691+0000", 2023-07-02T14:06:20.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004611693 2023-07-02T14:06:20.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408303+0000", 2023-07-02T14:06:20.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1935999999999995e-05 2023-07-02T14:06:20.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408395+0000", 2023-07-02T14:06:20.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00051077700000000002 2023-07-02T14:06:20.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:20.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408906+0000", 2023-07-02T14:06:20.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090767460000000001 2023-07-02T14:06:20.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.417982+0000", 2023-07-02T14:06:20.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4283999999999997e-05 2023-07-02T14:06:20.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418037+0000", 2023-07-02T14:06:20.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9667999999999999e-05 2023-07-02T14:06:20.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418096+0000", 2023-07-02T14:06:20.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021233800000000001 2023-07-02T14:06:20.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418309+0000", 2023-07-02T14:06:20.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.115724379 2023-07-02T14:06:20.820 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.820 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.820 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.821 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17354 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:20.821 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:20.821 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.406890066000003, 2023-07-02T14:06:20.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493, 2023-07-02T14:06:20.821 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.822 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.822 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.822 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.822 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.822 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17354 2023-07-02T14:06:20.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.822 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:20.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:20.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:20.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:20.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7440000000000001e-06 2023-07-02T14:06:20.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302606+0000", 2023-07-02T14:06:20.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:06:20.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302613+0000", 2023-07-02T14:06:20.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8810000000000004e-06 2023-07-02T14:06:20.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302618+0000", 2023-07-02T14:06:20.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028613100000000002 2023-07-02T14:06:20.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302904+0000", 2023-07-02T14:06:20.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5278999999999995e-05 2023-07-02T14:06:20.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302979+0000", 2023-07-02T14:06:20.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096904750000000005 2023-07-02T14:06:20.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312669+0000", 2023-07-02T14:06:20.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0328000000000002e-05 2023-07-02T14:06:20.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312700+0000", 2023-07-02T14:06:20.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098019319999999993 2023-07-02T14:06:20.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322502+0000", 2023-07-02T14:06:20.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4669999999999998e-05 2023-07-02T14:06:20.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322536+0000", 2023-07-02T14:06:20.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010023839 2023-07-02T14:06:20.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332560+0000", 2023-07-02T14:06:20.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5559999999999998e-05 2023-07-02T14:06:20.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332596+0000", 2023-07-02T14:06:20.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090401319999999993 2023-07-02T14:06:20.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341636+0000", 2023-07-02T14:06:20.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5373000000000001e-05 2023-07-02T14:06:20.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341681+0000", 2023-07-02T14:06:20.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081994059999999994 2023-07-02T14:06:20.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349881+0000", 2023-07-02T14:06:20.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4319000000000001e-05 2023-07-02T14:06:20.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349915+0000", 2023-07-02T14:06:20.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089388339999999997 2023-07-02T14:06:20.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358854+0000", 2023-07-02T14:06:20.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4319000000000005e-05 2023-07-02T14:06:20.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358928+0000", 2023-07-02T14:06:20.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080234690000000001 2023-07-02T14:06:20.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366952+0000", 2023-07-02T14:06:20.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2082999999999998e-05 2023-07-02T14:06:20.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366984+0000", 2023-07-02T14:06:20.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075015790000000004 2023-07-02T14:06:20.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374485+0000", 2023-07-02T14:06:20.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4480999999999995e-05 2023-07-02T14:06:20.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374550+0000", 2023-07-02T14:06:20.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091666620000000008 2023-07-02T14:06:20.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383716+0000", 2023-07-02T14:06:20.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9318000000000003e-05 2023-07-02T14:06:20.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383786+0000", 2023-07-02T14:06:20.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013434115 2023-07-02T14:06:20.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397220+0000", 2023-07-02T14:06:20.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4573999999999997e-05 2023-07-02T14:06:20.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397264+0000", 2023-07-02T14:06:20.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064824649999999998 2023-07-02T14:06:20.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403747+0000", 2023-07-02T14:06:20.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1087999999999993e-05 2023-07-02T14:06:20.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403828+0000", 2023-07-02T14:06:20.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072854449999999998 2023-07-02T14:06:20.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411113+0000", 2023-07-02T14:06:20.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5120000000000001e-05 2023-07-02T14:06:20.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411169+0000", 2023-07-02T14:06:20.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007211505 2023-07-02T14:06:20.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418380+0000", 2023-07-02T14:06:20.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011289399999999999 2023-07-02T14:06:20.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418493+0000", 2023-07-02T14:06:20.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00051217500000000002 2023-07-02T14:06:20.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:20.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.419005+0000", 2023-07-02T14:06:20.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00327393 2023-07-02T14:06:20.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422279+0000", 2023-07-02T14:06:20.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041432350000000003 2023-07-02T14:06:20.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426422+0000", 2023-07-02T14:06:20.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7021000000000003e-05 2023-07-02T14:06:20.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426469+0000", 2023-07-02T14:06:20.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000190256 2023-07-02T14:06:20.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426660+0000", 2023-07-02T14:06:20.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493 2023-07-02T14:06:20.847 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.847 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.847 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.848 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17355 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:20.848 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:20.848 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.406871404999997, 2023-07-02T14:06:20.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001, 2023-07-02T14:06:20.848 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.848 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.848 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.849 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.849 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.849 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17355 2023-07-02T14:06:20.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.849 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:20.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:20.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:20.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302621+0000", 2023-07-02T14:06:20.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2260000000000001e-06 2023-07-02T14:06:20.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302625+0000", 2023-07-02T14:06:20.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8599999999999998e-06 2023-07-02T14:06:20.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302630+0000", 2023-07-02T14:06:20.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0629999999999999e-06 2023-07-02T14:06:20.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302635+0000", 2023-07-02T14:06:20.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035731199999999998 2023-07-02T14:06:20.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302992+0000", 2023-07-02T14:06:20.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4164e-05 2023-07-02T14:06:20.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.303026+0000", 2023-07-02T14:06:20.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097115900000000008 2023-07-02T14:06:20.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312738+0000", 2023-07-02T14:06:20.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1701999999999998e-05 2023-07-02T14:06:20.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312819+0000", 2023-07-02T14:06:20.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097363640000000008 2023-07-02T14:06:20.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322556+0000", 2023-07-02T14:06:20.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2517000000000001e-05 2023-07-02T14:06:20.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322588+0000", 2023-07-02T14:06:20.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01002393 2023-07-02T14:06:20.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332612+0000", 2023-07-02T14:06:20.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2960999999999998e-05 2023-07-02T14:06:20.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332645+0000", 2023-07-02T14:06:20.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090669819999999995 2023-07-02T14:06:20.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341712+0000", 2023-07-02T14:06:20.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9288000000000002e-05 2023-07-02T14:06:20.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341811+0000", 2023-07-02T14:06:20.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081504460000000004 2023-07-02T14:06:20.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349962+0000", 2023-07-02T14:06:20.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107321 2023-07-02T14:06:20.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.350069+0000", 2023-07-02T14:06:20.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088790999999999991 2023-07-02T14:06:20.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358948+0000", 2023-07-02T14:06:20.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1711000000000002e-05 2023-07-02T14:06:20.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358980+0000", 2023-07-02T14:06:20.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080367579999999997 2023-07-02T14:06:20.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367017+0000", 2023-07-02T14:06:20.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6329000000000002e-05 2023-07-02T14:06:20.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367073+0000", 2023-07-02T14:06:20.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075097300000000001 2023-07-02T14:06:20.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374583+0000", 2023-07-02T14:06:20.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5710999999999995e-05 2023-07-02T14:06:20.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374648+0000", 2023-07-02T14:06:20.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009165912 2023-07-02T14:06:20.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383814+0000", 2023-07-02T14:06:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3916000000000003e-05 2023-07-02T14:06:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383878+0000", 2023-07-02T14:06:20.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013440532999999999 2023-07-02T14:06:20.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397319+0000", 2023-07-02T14:06:20.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7218999999999995e-05 2023-07-02T14:06:20.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397386+0000", 2023-07-02T14:06:20.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065165279999999997 2023-07-02T14:06:20.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403903+0000", 2023-07-02T14:06:20.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8152000000000005e-05 2023-07-02T14:06:20.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404001+0000", 2023-07-02T14:06:20.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072185360000000002 2023-07-02T14:06:20.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411219+0000", 2023-07-02T14:06:20.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:06:20.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411310+0000", 2023-07-02T14:06:20.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010021663 2023-07-02T14:06:20.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421331+0000", 2023-07-02T14:06:20.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0148000000000006e-05 2023-07-02T14:06:20.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421411+0000", 2023-07-02T14:06:20.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053056170000000003 2023-07-02T14:06:20.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426717+0000", 2023-07-02T14:06:20.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8242000000000001e-05 2023-07-02T14:06:20.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426795+0000", 2023-07-02T14:06:20.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00041308399999999998 2023-07-02T14:06:20.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:20.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.427208+0000", 2023-07-02T14:06:20.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028330849999999999 2023-07-02T14:06:20.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430041+0000", 2023-07-02T14:06:20.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00072642300000000002 2023-07-02T14:06:20.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430768+0000", 2023-07-02T14:06:20.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1847000000000002e-05 2023-07-02T14:06:20.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430800+0000", 2023-07-02T14:06:20.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9308999999999994e-05 2023-07-02T14:06:20.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430899+0000", 2023-07-02T14:06:20.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001 2023-07-02T14:06:20.875 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.875 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.875 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.876 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:410 2.5 2:bbaed619:::rbd_data.125de1449605.0000000000000109:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:06:20.876 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:20.876 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 24.25892614, 2023-07-02T14:06:20.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003, 2023-07-02T14:06:20.876 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.876 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:20.877 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.877 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:06:20.877 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:06:20.877 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:06:20.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.877 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:20.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:20.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:20.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:20.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.635e-06 2023-07-02T14:06:20.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450568+0000", 2023-07-02T14:06:20.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5970000000000002e-06 2023-07-02T14:06:20.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450573+0000", 2023-07-02T14:06:20.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3379999999999999e-06 2023-07-02T14:06:20.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450575+0000", 2023-07-02T14:06:20.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9458000000000001e-05 2023-07-02T14:06:20.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450615+0000", 2023-07-02T14:06:20.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0569999999999993e-06 2023-07-02T14:06:20.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:20.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450623+0000", 2023-07-02T14:06:20.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.63968709199999996 2023-07-02T14:06:20.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090310+0000", 2023-07-02T14:06:20.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020157399999999999 2023-07-02T14:06:20.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090512+0000", 2023-07-02T14:06:20.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027739190000000001 2023-07-02T14:06:20.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093285+0000", 2023-07-02T14:06:20.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003 2023-07-02T14:06:20.884 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.884 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.884 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.885 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:416 2.5 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:06:20.885 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:20.885 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 24.258896707000002, 2023-07-02T14:06:20.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002, 2023-07-02T14:06:20.885 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.885 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:20.886 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.886 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:06:20.886 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:06:20.886 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:06:20.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.886 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:20.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:20.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:20.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:20.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.34e-06 2023-07-02T14:06:20.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450599+0000", 2023-07-02T14:06:20.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1549999999999998e-06 2023-07-02T14:06:20.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450604+0000", 2023-07-02T14:06:20.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:06:20.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450607+0000", 2023-07-02T14:06:20.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5717999999999999e-05 2023-07-02T14:06:20.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450643+0000", 2023-07-02T14:06:20.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1619999999999999e-06 2023-07-02T14:06:20.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:20.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450648+0000", 2023-07-02T14:06:20.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64292291800000001 2023-07-02T14:06:20.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093571+0000", 2023-07-02T14:06:20.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115628 2023-07-02T14:06:20.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093687+0000", 2023-07-02T14:06:20.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002885552 2023-07-02T14:06:20.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.893 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.893 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.096572+0000", 2023-07-02T14:06:20.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002 2023-07-02T14:06:20.893 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.893 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.893 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.894 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18554 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:20.894 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248917+0000", 2023-07-02T14:06:20.894 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 15.460575891, 2023-07-02T14:06:20.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11109714599999999, 2023-07-02T14:06:20.894 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.894 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.895 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.895 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.895 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.895 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18554 2023-07-02T14:06:20.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.895 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248917+0000", 2023-07-02T14:06:20.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248917+0000", 2023-07-02T14:06:20.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:20.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248916+0000", 2023-07-02T14:06:20.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1690000000000003e-06 2023-07-02T14:06:20.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248923+0000", 2023-07-02T14:06:20.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2769999999999994e-06 2023-07-02T14:06:20.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248933+0000", 2023-07-02T14:06:20.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1860000000000002e-06 2023-07-02T14:06:20.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248937+0000", 2023-07-02T14:06:20.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00063144000000000004 2023-07-02T14:06:20.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249568+0000", 2023-07-02T14:06:20.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1486999999999998e-05 2023-07-02T14:06:20.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249610+0000", 2023-07-02T14:06:20.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066288199999999997 2023-07-02T14:06:20.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256239+0000", 2023-07-02T14:06:20.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3636000000000001e-05 2023-07-02T14:06:20.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256272+0000", 2023-07-02T14:06:20.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01221675 2023-07-02T14:06:20.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268489+0000", 2023-07-02T14:06:20.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0283000000000004e-05 2023-07-02T14:06:20.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268569+0000", 2023-07-02T14:06:20.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011087254 2023-07-02T14:06:20.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279657+0000", 2023-07-02T14:06:20.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2872e-05 2023-07-02T14:06:20.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279689+0000", 2023-07-02T14:06:20.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011797277 2023-07-02T14:06:20.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291487+0000", 2023-07-02T14:06:20.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0111e-05 2023-07-02T14:06:20.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291517+0000", 2023-07-02T14:06:20.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079886279999999994 2023-07-02T14:06:20.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299505+0000", 2023-07-02T14:06:20.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2787000000000006e-05 2023-07-02T14:06:20.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299578+0000", 2023-07-02T14:06:20.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071806889999999996 2023-07-02T14:06:20.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306759+0000", 2023-07-02T14:06:20.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1510000000000002e-05 2023-07-02T14:06:20.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306790+0000", 2023-07-02T14:06:20.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056651269999999998 2023-07-02T14:06:20.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312456+0000", 2023-07-02T14:06:20.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8580999999999999e-05 2023-07-02T14:06:20.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312534+0000", 2023-07-02T14:06:20.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047950680000000004 2023-07-02T14:06:20.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317329+0000", 2023-07-02T14:06:20.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4718000000000002e-05 2023-07-02T14:06:20.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317364+0000", 2023-07-02T14:06:20.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050140469999999998 2023-07-02T14:06:20.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322378+0000", 2023-07-02T14:06:20.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1049e-05 2023-07-02T14:06:20.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322409+0000", 2023-07-02T14:06:20.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006616612 2023-07-02T14:06:20.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329026+0000", 2023-07-02T14:06:20.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4612e-05 2023-07-02T14:06:20.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329060+0000", 2023-07-02T14:06:20.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062225020000000004 2023-07-02T14:06:20.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335283+0000", 2023-07-02T14:06:20.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0385e-05 2023-07-02T14:06:20.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335323+0000", 2023-07-02T14:06:20.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009678262 2023-07-02T14:06:20.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345001+0000", 2023-07-02T14:06:20.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0340999999999999e-05 2023-07-02T14:06:20.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345042+0000", 2023-07-02T14:06:20.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045838650000000003 2023-07-02T14:06:20.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349626+0000", 2023-07-02T14:06:20.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010221999999999999 2023-07-02T14:06:20.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349728+0000", 2023-07-02T14:06:20.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029884349999999998 2023-07-02T14:06:20.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.352716+0000", 2023-07-02T14:06:20.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3821e-05 2023-07-02T14:06:20.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.352750+0000", 2023-07-02T14:06:20.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021565399999999999 2023-07-02T14:06:20.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:20.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.352966+0000", 2023-07-02T14:06:20.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018362960000000001 2023-07-02T14:06:20.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354802+0000", 2023-07-02T14:06:20.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051013409999999997 2023-07-02T14:06:20.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.359903+0000", 2023-07-02T14:06:20.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9068999999999999e-05 2023-07-02T14:06:20.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.359932+0000", 2023-07-02T14:06:20.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2008000000000005e-05 2023-07-02T14:06:20.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.360014+0000", 2023-07-02T14:06:20.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11109714599999999 2023-07-02T14:06:20.921 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.921 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.921 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.922 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18555 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:20.922 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248941+0000", 2023-07-02T14:06:20.922 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 15.460552292999999, 2023-07-02T14:06:20.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11809699899999999, 2023-07-02T14:06:20.922 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.922 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.923 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.923 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.923 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.923 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18555 2023-07-02T14:06:20.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.923 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248941+0000", 2023-07-02T14:06:20.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248941+0000", 2023-07-02T14:06:20.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:20.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248940+0000", 2023-07-02T14:06:20.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.7159999999999999e-06 2023-07-02T14:06:20.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248943+0000", 2023-07-02T14:06:20.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5949999999999999e-06 2023-07-02T14:06:20.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248947+0000", 2023-07-02T14:06:20.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4480000000000001e-06 2023-07-02T14:06:20.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248949+0000", 2023-07-02T14:06:20.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00067899499999999995 2023-07-02T14:06:20.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249628+0000", 2023-07-02T14:06:20.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1100000000000003e-05 2023-07-02T14:06:20.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249669+0000", 2023-07-02T14:06:20.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066349690000000001 2023-07-02T14:06:20.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256304+0000", 2023-07-02T14:06:20.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2691999999999998e-05 2023-07-02T14:06:20.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256337+0000", 2023-07-02T14:06:20.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012284679999999999 2023-07-02T14:06:20.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268622+0000", 2023-07-02T14:06:20.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6550000000000001e-05 2023-07-02T14:06:20.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268658+0000", 2023-07-02T14:06:20.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011061109 2023-07-02T14:06:20.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279719+0000", 2023-07-02T14:06:20.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2134000000000001e-05 2023-07-02T14:06:20.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279751+0000", 2023-07-02T14:06:20.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011798329999999999 2023-07-02T14:06:20.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291550+0000", 2023-07-02T14:06:20.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9839999999999999e-05 2023-07-02T14:06:20.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291580+0000", 2023-07-02T14:06:20.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080386550000000005 2023-07-02T14:06:20.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299618+0000", 2023-07-02T14:06:20.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2632999999999997e-05 2023-07-02T14:06:20.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299651+0000", 2023-07-02T14:06:20.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071769859999999998 2023-07-02T14:06:20.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306828+0000", 2023-07-02T14:06:20.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1123000000000004e-05 2023-07-02T14:06:20.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306889+0000", 2023-07-02T14:06:20.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056835710000000001 2023-07-02T14:06:20.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312573+0000", 2023-07-02T14:06:20.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0115000000000002e-05 2023-07-02T14:06:20.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312643+0000", 2023-07-02T14:06:20.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004754538 2023-07-02T14:06:20.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317397+0000", 2023-07-02T14:06:20.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4385000000000002e-05 2023-07-02T14:06:20.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317452+0000", 2023-07-02T14:06:20.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049988350000000001 2023-07-02T14:06:20.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322450+0000", 2023-07-02T14:06:20.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4749e-05 2023-07-02T14:06:20.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322515+0000", 2023-07-02T14:06:20.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065710810000000003 2023-07-02T14:06:20.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329086+0000", 2023-07-02T14:06:20.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0982000000000003e-05 2023-07-02T14:06:20.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329117+0000", 2023-07-02T14:06:20.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062301259999999999 2023-07-02T14:06:20.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335347+0000", 2023-07-02T14:06:20.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2830999999999998e-05 2023-07-02T14:06:20.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335380+0000", 2023-07-02T14:06:20.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097041909999999992 2023-07-02T14:06:20.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345084+0000", 2023-07-02T14:06:20.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2044000000000003e-05 2023-07-02T14:06:20.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345156+0000", 2023-07-02T14:06:20.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004610711 2023-07-02T14:06:20.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349767+0000", 2023-07-02T14:06:20.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8640000000000002e-05 2023-07-02T14:06:20.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349816+0000", 2023-07-02T14:06:20.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043231160000000001 2023-07-02T14:06:20.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354139+0000", 2023-07-02T14:06:20.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8073e-05 2023-07-02T14:06:20.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354177+0000", 2023-07-02T14:06:20.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005870681 2023-07-02T14:06:20.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.360048+0000", 2023-07-02T14:06:20.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9535e-05 2023-07-02T14:06:20.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.360087+0000", 2023-07-02T14:06:20.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021778100000000001 2023-07-02T14:06:20.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:20.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.360305+0000", 2023-07-02T14:06:20.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001720708 2023-07-02T14:06:20.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.362026+0000", 2023-07-02T14:06:20.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049001009999999996 2023-07-02T14:06:20.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.366926+0000", 2023-07-02T14:06:20.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3682e-05 2023-07-02T14:06:20.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.366959+0000", 2023-07-02T14:06:20.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8419999999999998e-05 2023-07-02T14:06:20.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.367038+0000", 2023-07-02T14:06:20.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11809699899999999 2023-07-02T14:06:20.950 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.951 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.951 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.951 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18556 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:20.951 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248957+0000", 2023-07-02T14:06:20.951 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 15.460536324, 2023-07-02T14:06:20.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12347108800000001, 2023-07-02T14:06:20.952 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.952 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.952 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.952 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.952 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.952 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18556 2023-07-02T14:06:20.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.952 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248957+0000", 2023-07-02T14:06:20.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248957+0000", 2023-07-02T14:06:20.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:20.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.954 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248957+0000", 2023-07-02T14:06:20.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4119999999999999e-06 2023-07-02T14:06:20.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248959+0000", 2023-07-02T14:06:20.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5669999999999999e-06 2023-07-02T14:06:20.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248962+0000", 2023-07-02T14:06:20.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3989999999999998e-06 2023-07-02T14:06:20.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248965+0000", 2023-07-02T14:06:20.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00072489500000000003 2023-07-02T14:06:20.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249690+0000", 2023-07-02T14:06:20.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7876999999999997e-05 2023-07-02T14:06:20.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249728+0000", 2023-07-02T14:06:20.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066341250000000003 2023-07-02T14:06:20.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256362+0000", 2023-07-02T14:06:20.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2715000000000001e-05 2023-07-02T14:06:20.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256395+0000", 2023-07-02T14:06:20.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012305508 2023-07-02T14:06:20.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268700+0000", 2023-07-02T14:06:20.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7537e-05 2023-07-02T14:06:20.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268778+0000", 2023-07-02T14:06:20.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011004300999999999 2023-07-02T14:06:20.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279782+0000", 2023-07-02T14:06:20.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1996000000000003e-05 2023-07-02T14:06:20.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279814+0000", 2023-07-02T14:06:20.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011789104 2023-07-02T14:06:20.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291603+0000", 2023-07-02T14:06:20.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9825999999999999e-05 2023-07-02T14:06:20.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291633+0000", 2023-07-02T14:06:20.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080637 2023-07-02T14:06:20.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299696+0000", 2023-07-02T14:06:20.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8535999999999996e-05 2023-07-02T14:06:20.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299765+0000", 2023-07-02T14:06:20.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071538169999999998 2023-07-02T14:06:20.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306919+0000", 2023-07-02T14:06:20.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0337000000000001e-05 2023-07-02T14:06:20.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306949+0000", 2023-07-02T14:06:20.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057379919999999999 2023-07-02T14:06:20.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312687+0000", 2023-07-02T14:06:20.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8463000000000001e-05 2023-07-02T14:06:20.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312726+0000", 2023-07-02T14:06:20.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004766251 2023-07-02T14:06:20.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317492+0000", 2023-07-02T14:06:20.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1788000000000001e-05 2023-07-02T14:06:20.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317524+0000", 2023-07-02T14:06:20.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050322910000000004 2023-07-02T14:06:20.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322556+0000", 2023-07-02T14:06:20.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9141999999999999e-05 2023-07-02T14:06:20.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322595+0000", 2023-07-02T14:06:20.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006553354 2023-07-02T14:06:20.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329148+0000", 2023-07-02T14:06:20.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2458e-05 2023-07-02T14:06:20.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329181+0000", 2023-07-02T14:06:20.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006222483 2023-07-02T14:06:20.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335403+0000", 2023-07-02T14:06:20.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0419000000000001e-05 2023-07-02T14:06:20.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335434+0000", 2023-07-02T14:06:20.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097556550000000002 2023-07-02T14:06:20.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345189+0000", 2023-07-02T14:06:20.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2968e-05 2023-07-02T14:06:20.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345222+0000", 2023-07-02T14:06:20.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046402270000000002 2023-07-02T14:06:20.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349863+0000", 2023-07-02T14:06:20.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7782999999999997e-05 2023-07-02T14:06:20.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349940+0000", 2023-07-02T14:06:20.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042690699999999998 2023-07-02T14:06:20.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354210+0000", 2023-07-02T14:06:20.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5648000000000001e-05 2023-07-02T14:06:20.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354255+0000", 2023-07-02T14:06:20.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071341269999999997 2023-07-02T14:06:20.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361389+0000", 2023-07-02T14:06:20.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9140000000000001e-05 2023-07-02T14:06:20.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361428+0000", 2023-07-02T14:06:20.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056418470000000002 2023-07-02T14:06:20.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.367070+0000", 2023-07-02T14:06:20.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1863000000000002e-05 2023-07-02T14:06:20.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:20.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.367112+0000", 2023-07-02T14:06:20.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00027357599999999998 2023-07-02T14:06:20.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:20.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.367386+0000", 2023-07-02T14:06:20.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020277680000000001 2023-07-02T14:06:20.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:20.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369413+0000", 2023-07-02T14:06:20.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029075699999999999 2023-07-02T14:06:20.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:20.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372321+0000", 2023-07-02T14:06:20.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1955999999999997e-05 2023-07-02T14:06:20.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:20.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372353+0000", 2023-07-02T14:06:20.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4938999999999995e-05 2023-07-02T14:06:20.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:20.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372428+0000", 2023-07-02T14:06:20.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12347108800000001 2023-07-02T14:06:20.981 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.981 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:20.981 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:20.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.982 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18557 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:20.982 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:20.982 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 15.460524328, 2023-07-02T14:06:20.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13120991400000001, 2023-07-02T14:06:20.982 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:20.982 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:20.982 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:20.982 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:20.983 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:20.983 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18557 2023-07-02T14:06:20.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.983 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:20.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:20.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:20.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:20.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:20.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:20.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:06:20.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:20.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:20.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8870000000000001e-06 2023-07-02T14:06:20.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:20.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248971+0000", 2023-07-02T14:06:20.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9009999999999996e-06 2023-07-02T14:06:20.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:20.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248974+0000", 2023-07-02T14:06:20.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9809999999999997e-06 2023-07-02T14:06:20.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:20.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248978+0000", 2023-07-02T14:06:20.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00076989199999999999 2023-07-02T14:06:20.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249748+0000", 2023-07-02T14:06:20.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8245999999999999e-05 2023-07-02T14:06:20.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249787+0000", 2023-07-02T14:06:20.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066332969999999998 2023-07-02T14:06:20.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256420+0000", 2023-07-02T14:06:20.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2566e-05 2023-07-02T14:06:20.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256452+0000", 2023-07-02T14:06:20.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012364714000000001 2023-07-02T14:06:20.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268817+0000", 2023-07-02T14:06:20.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5113000000000001e-05 2023-07-02T14:06:20.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268852+0000", 2023-07-02T14:06:20.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010993899999999999 2023-07-02T14:06:20.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279846+0000", 2023-07-02T14:06:20.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2144999999999998e-05 2023-07-02T14:06:20.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279878+0000", 2023-07-02T14:06:20.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011778958000000001 2023-07-02T14:06:20.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291657+0000", 2023-07-02T14:06:20.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9592000000000001e-05 2023-07-02T14:06:20.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291687+0000", 2023-07-02T14:06:20.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081107220000000008 2023-07-02T14:06:20.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299798+0000", 2023-07-02T14:06:20.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0290999999999998e-05 2023-07-02T14:06:20.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299828+0000", 2023-07-02T14:06:20.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071584489999999999 2023-07-02T14:06:20.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306986+0000", 2023-07-02T14:06:20.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0755999999999999e-05 2023-07-02T14:06:20.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307047+0000", 2023-07-02T14:06:20.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057110260000000001 2023-07-02T14:06:20.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312758+0000", 2023-07-02T14:06:20.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5145999999999999e-05 2023-07-02T14:06:20.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312793+0000", 2023-07-02T14:06:20.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004758361 2023-07-02T14:06:20.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317552+0000", 2023-07-02T14:06:20.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9190000000000002e-05 2023-07-02T14:06:20.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317601+0000", 2023-07-02T14:06:20.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050329420000000003 2023-07-02T14:06:20.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:20.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322634+0000", 2023-07-02T14:06:20.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5737999999999998e-05 2023-07-02T14:06:20.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:20.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:20.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322719+0000", 2023-07-02T14:06:20.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064995679999999998 2023-07-02T14:06:20.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:20.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329219+0000", 2023-07-02T14:06:21.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0908000000000003e-05 2023-07-02T14:06:21.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329250+0000", 2023-07-02T14:06:21.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062072029999999997 2023-07-02T14:06:21.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335457+0000", 2023-07-02T14:06:21.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9471999999999998e-05 2023-07-02T14:06:21.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335487+0000", 2023-07-02T14:06:21.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097780239999999997 2023-07-02T14:06:21.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345265+0000", 2023-07-02T14:06:21.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0982999999999994e-05 2023-07-02T14:06:21.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345336+0000", 2023-07-02T14:06:21.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046378799999999996 2023-07-02T14:06:21.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349973+0000", 2023-07-02T14:06:21.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1069999999999999e-05 2023-07-02T14:06:21.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350005+0000", 2023-07-02T14:06:21.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042766870000000004 2023-07-02T14:06:21.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354281+0000", 2023-07-02T14:06:21.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1155000000000002e-05 2023-07-02T14:06:21.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354312+0000", 2023-07-02T14:06:21.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071436820000000002 2023-07-02T14:06:21.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361456+0000", 2023-07-02T14:06:21.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9054000000000002e-05 2023-07-02T14:06:21.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361505+0000", 2023-07-02T14:06:21.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072301320000000002 2023-07-02T14:06:21.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368735+0000", 2023-07-02T14:06:21.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1013999999999994e-05 2023-07-02T14:06:21.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368816+0000", 2023-07-02T14:06:21.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036582490000000001 2023-07-02T14:06:21.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372475+0000", 2023-07-02T14:06:21.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4652000000000004e-05 2023-07-02T14:06:21.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:21.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372539+0000", 2023-07-02T14:06:21.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00031374199999999999 2023-07-02T14:06:21.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:21.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372853+0000", 2023-07-02T14:06:21.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030071299999999998 2023-07-02T14:06:21.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:21.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.375860+0000", 2023-07-02T14:06:21.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041897749999999997 2023-07-02T14:06:21.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:21.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380050+0000", 2023-07-02T14:06:21.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8331000000000003e-05 2023-07-02T14:06:21.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:21.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380088+0000", 2023-07-02T14:06:21.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0626999999999998e-05 2023-07-02T14:06:21.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:21.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380179+0000", 2023-07-02T14:06:21.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13120991400000001 2023-07-02T14:06:21.013 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:21.013 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:21.013 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:21.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.013 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18558 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:21.013 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:21.014 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 15.460507626, 2023-07-02T14:06:21.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13755567899999999, 2023-07-02T14:06:21.014 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:21.014 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:21.014 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:21.014 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:21.014 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:21.014 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18558 2023-07-02T14:06:21.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.015 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:21.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:21.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:21.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:21.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:21.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:21.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:21.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:21.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248985+0000", 2023-07-02T14:06:21.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8630000000000001e-06 2023-07-02T14:06:21.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:21.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248989+0000", 2023-07-02T14:06:21.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0750000000000003e-06 2023-07-02T14:06:21.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:21.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248995+0000", 2023-07-02T14:06:21.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3290000000000004e-06 2023-07-02T14:06:21.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:21.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248999+0000", 2023-07-02T14:06:21.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00080656300000000005 2023-07-02T14:06:21.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249806+0000", 2023-07-02T14:06:21.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2040999999999999e-05 2023-07-02T14:06:21.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249848+0000", 2023-07-02T14:06:21.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066312419999999999 2023-07-02T14:06:21.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256479+0000", 2023-07-02T14:06:21.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3348e-05 2023-07-02T14:06:21.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256513+0000", 2023-07-02T14:06:21.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012391531000000001 2023-07-02T14:06:21.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268904+0000", 2023-07-02T14:06:21.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.347e-05 2023-07-02T14:06:21.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268978+0000", 2023-07-02T14:06:21.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010929772000000001 2023-07-02T14:06:21.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279907+0000", 2023-07-02T14:06:21.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2669999999999997e-05 2023-07-02T14:06:21.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279940+0000", 2023-07-02T14:06:21.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011770332999999999 2023-07-02T14:06:21.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291710+0000", 2023-07-02T14:06:21.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1207000000000001e-05 2023-07-02T14:06:21.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291742+0000", 2023-07-02T14:06:21.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081242199999999997 2023-07-02T14:06:21.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299866+0000", 2023-07-02T14:06:21.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4650000000000002e-05 2023-07-02T14:06:21.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299900+0000", 2023-07-02T14:06:21.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071810270000000004 2023-07-02T14:06:21.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307081+0000", 2023-07-02T14:06:21.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4378000000000002e-05 2023-07-02T14:06:21.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307116+0000", 2023-07-02T14:06:21.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057031909999999998 2023-07-02T14:06:21.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312819+0000", 2023-07-02T14:06:21.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1612999999999997e-05 2023-07-02T14:06:21.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312851+0000", 2023-07-02T14:06:21.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048172470000000002 2023-07-02T14:06:21.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317668+0000", 2023-07-02T14:06:21.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3328e-05 2023-07-02T14:06:21.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317741+0000", 2023-07-02T14:06:21.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050117290000000004 2023-07-02T14:06:21.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322753+0000", 2023-07-02T14:06:21.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9872000000000001e-05 2023-07-02T14:06:21.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322783+0000", 2023-07-02T14:06:21.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064910610000000002 2023-07-02T14:06:21.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329274+0000", 2023-07-02T14:06:21.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0097e-05 2023-07-02T14:06:21.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329304+0000", 2023-07-02T14:06:21.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062163879999999998 2023-07-02T14:06:21.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335520+0000", 2023-07-02T14:06:21.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9513e-05 2023-07-02T14:06:21.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335550+0000", 2023-07-02T14:06:21.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098209969999999997 2023-07-02T14:06:21.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345371+0000", 2023-07-02T14:06:21.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9862999999999999e-05 2023-07-02T14:06:21.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345401+0000", 2023-07-02T14:06:21.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046552030000000001 2023-07-02T14:06:21.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350056+0000", 2023-07-02T14:06:21.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8751999999999993e-05 2023-07-02T14:06:21.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350145+0000", 2023-07-02T14:06:21.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041981450000000003 2023-07-02T14:06:21.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354343+0000", 2023-07-02T14:06:21.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1501000000000002e-05 2023-07-02T14:06:21.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354374+0000", 2023-07-02T14:06:21.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071551740000000003 2023-07-02T14:06:21.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361530+0000", 2023-07-02T14:06:21.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5156e-05 2023-07-02T14:06:21.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361565+0000", 2023-07-02T14:06:21.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072918569999999997 2023-07-02T14:06:21.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368857+0000", 2023-07-02T14:06:21.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5770999999999999e-05 2023-07-02T14:06:21.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368902+0000", 2023-07-02T14:06:21.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057210569999999999 2023-07-02T14:06:21.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374623+0000", 2023-07-02T14:06:21.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3266999999999998e-05 2023-07-02T14:06:21.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374667+0000", 2023-07-02T14:06:21.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055479570000000001 2023-07-02T14:06:21.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380215+0000", 2023-07-02T14:06:21.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0067e-05 2023-07-02T14:06:21.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:21.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380265+0000", 2023-07-02T14:06:21.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00043082299999999999 2023-07-02T14:06:21.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:21.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380696+0000", 2023-07-02T14:06:21.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024853240000000001 2023-07-02T14:06:21.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:21.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.383181+0000", 2023-07-02T14:06:21.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032601280000000002 2023-07-02T14:06:21.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:21.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386441+0000", 2023-07-02T14:06:21.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9748e-05 2023-07-02T14:06:21.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:21.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386471+0000", 2023-07-02T14:06:21.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0531000000000006e-05 2023-07-02T14:06:21.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:21.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386541+0000", 2023-07-02T14:06:21.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13755567899999999 2023-07-02T14:06:21.046 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:21.046 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:21.046 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:21.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.046 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18559 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:21.047 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:21.047 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 15.460485022, 2023-07-02T14:06:21.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.146184591, 2023-07-02T14:06:21.047 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:21.047 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:21.047 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:21.047 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:21.047 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:21.048 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18559 2023-07-02T14:06:21.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.048 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:21.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:21.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:21.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:21.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:21.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:21.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:21.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:21.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:21.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.726e-06 2023-07-02T14:06:21.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:21.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249011+0000", 2023-07-02T14:06:21.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7939999999999997e-06 2023-07-02T14:06:21.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:21.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249018+0000", 2023-07-02T14:06:21.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3470000000000001e-06 2023-07-02T14:06:21.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:21.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249023+0000", 2023-07-02T14:06:21.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00084402900000000005 2023-07-02T14:06:21.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249867+0000", 2023-07-02T14:06:21.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9050000000000001e-05 2023-07-02T14:06:21.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249906+0000", 2023-07-02T14:06:21.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066311039999999996 2023-07-02T14:06:21.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256537+0000", 2023-07-02T14:06:21.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2790000000000003e-05 2023-07-02T14:06:21.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256570+0000", 2023-07-02T14:06:21.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012436129000000001 2023-07-02T14:06:21.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269006+0000", 2023-07-02T14:06:21.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0173999999999999e-05 2023-07-02T14:06:21.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269046+0000", 2023-07-02T14:06:21.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010924698 2023-07-02T14:06:21.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279971+0000", 2023-07-02T14:06:21.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3831000000000002e-05 2023-07-02T14:06:21.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280004+0000", 2023-07-02T14:06:21.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011795912 2023-07-02T14:06:21.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291800+0000", 2023-07-02T14:06:21.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5613999999999997e-05 2023-07-02T14:06:21.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291886+0000", 2023-07-02T14:06:21.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080395999999999992 2023-07-02T14:06:21.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299925+0000", 2023-07-02T14:06:21.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0694999999999997e-05 2023-07-02T14:06:21.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299956+0000", 2023-07-02T14:06:21.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072049449999999999 2023-07-02T14:06:21.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307161+0000", 2023-07-02T14:06:21.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5523000000000001e-05 2023-07-02T14:06:21.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307247+0000", 2023-07-02T14:06:21.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056331929999999999 2023-07-02T14:06:21.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312880+0000", 2023-07-02T14:06:21.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1646000000000002e-05 2023-07-02T14:06:21.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312911+0000", 2023-07-02T14:06:21.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048627540000000004 2023-07-02T14:06:21.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317774+0000", 2023-07-02T14:06:21.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2008000000000003e-05 2023-07-02T14:06:21.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317806+0000", 2023-07-02T14:06:21.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050055689999999996 2023-07-02T14:06:21.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322812+0000", 2023-07-02T14:06:21.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3442999999999997e-05 2023-07-02T14:06:21.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322855+0000", 2023-07-02T14:06:21.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006472813 2023-07-02T14:06:21.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329328+0000", 2023-07-02T14:06:21.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0128e-05 2023-07-02T14:06:21.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329358+0000", 2023-07-02T14:06:21.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062156490000000002 2023-07-02T14:06:21.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335574+0000", 2023-07-02T14:06:21.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2357999999999997e-05 2023-07-02T14:06:21.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335606+0000", 2023-07-02T14:06:21.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098318409999999992 2023-07-02T14:06:21.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345438+0000", 2023-07-02T14:06:21.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1174000000000007e-05 2023-07-02T14:06:21.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345509+0000", 2023-07-02T14:06:21.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004669851 2023-07-02T14:06:21.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350179+0000", 2023-07-02T14:06:21.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1559000000000001e-05 2023-07-02T14:06:21.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350211+0000", 2023-07-02T14:06:21.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041865740000000002 2023-07-02T14:06:21.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354397+0000", 2023-07-02T14:06:21.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9850000000000001e-05 2023-07-02T14:06:21.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354427+0000", 2023-07-02T14:06:21.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071608449999999999 2023-07-02T14:06:21.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361588+0000", 2023-07-02T14:06:21.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8346000000000002e-05 2023-07-02T14:06:21.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361626+0000", 2023-07-02T14:06:21.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073175009999999997 2023-07-02T14:06:21.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368944+0000", 2023-07-02T14:06:21.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2879000000000005e-05 2023-07-02T14:06:21.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369017+0000", 2023-07-02T14:06:21.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056850690000000001 2023-07-02T14:06:21.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374702+0000", 2023-07-02T14:06:21.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7693e-05 2023-07-02T14:06:21.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374739+0000", 2023-07-02T14:06:21.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076925149999999996 2023-07-02T14:06:21.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382432+0000", 2023-07-02T14:06:21.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3259e-05 2023-07-02T14:06:21.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382475+0000", 2023-07-02T14:06:21.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041005110000000003 2023-07-02T14:06:21.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386576+0000", 2023-07-02T14:06:21.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2074999999999999e-05 2023-07-02T14:06:21.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:21.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386618+0000", 2023-07-02T14:06:21.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000233692 2023-07-02T14:06:21.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:21.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386851+0000", 2023-07-02T14:06:21.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024022269999999998 2023-07-02T14:06:21.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:21.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.389254+0000", 2023-07-02T14:06:21.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058506469999999996 2023-07-02T14:06:21.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:21.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395104+0000", 2023-07-02T14:06:21.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8807000000000003e-05 2023-07-02T14:06:21.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:21.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395143+0000", 2023-07-02T14:06:21.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9616000000000001e-05 2023-07-02T14:06:21.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:21.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395193+0000", 2023-07-02T14:06:21.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.146184591 2023-07-02T14:06:21.080 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:21.080 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:21.080 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:21.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.081 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18560 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:21.081 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:21.081 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 15.460466399, 2023-07-02T14:06:21.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.15647551900000001, 2023-07-02T14:06:21.081 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:21.081 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:21.081 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:21.082 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:21.082 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:21.082 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18560 2023-07-02T14:06:21.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.082 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:21.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:21.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:21.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:21.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:21.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:21.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:21.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:21.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249026+0000", 2023-07-02T14:06:21.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3419999999999999e-06 2023-07-02T14:06:21.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:21.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249030+0000", 2023-07-02T14:06:21.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7869999999999998e-06 2023-07-02T14:06:21.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:21.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249035+0000", 2023-07-02T14:06:21.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9639999999999997e-06 2023-07-02T14:06:21.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:21.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249039+0000", 2023-07-02T14:06:21.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00088477599999999999 2023-07-02T14:06:21.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249924+0000", 2023-07-02T14:06:21.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7333999999999999e-05 2023-07-02T14:06:21.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249962+0000", 2023-07-02T14:06:21.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066323409999999999 2023-07-02T14:06:21.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256594+0000", 2023-07-02T14:06:21.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2595999999999998e-05 2023-07-02T14:06:21.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256627+0000", 2023-07-02T14:06:21.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012449894 2023-07-02T14:06:21.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269076+0000", 2023-07-02T14:06:21.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3942999999999997e-05 2023-07-02T14:06:21.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269110+0000", 2023-07-02T14:06:21.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010923884 2023-07-02T14:06:21.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280034+0000", 2023-07-02T14:06:21.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7774000000000001e-05 2023-07-02T14:06:21.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280072+0000", 2023-07-02T14:06:21.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011850027000000001 2023-07-02T14:06:21.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291922+0000", 2023-07-02T14:06:21.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2128000000000001e-05 2023-07-02T14:06:21.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291954+0000", 2023-07-02T14:06:21.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080261150000000003 2023-07-02T14:06:21.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299980+0000", 2023-07-02T14:06:21.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6322999999999998e-05 2023-07-02T14:06:21.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300017+0000", 2023-07-02T14:06:21.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072683039999999997 2023-07-02T14:06:21.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307285+0000", 2023-07-02T14:06:21.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3309000000000002e-05 2023-07-02T14:06:21.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307318+0000", 2023-07-02T14:06:21.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056178649999999997 2023-07-02T14:06:21.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312936+0000", 2023-07-02T14:06:21.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1866000000000003e-05 2023-07-02T14:06:21.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312968+0000", 2023-07-02T14:06:21.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048789970000000004 2023-07-02T14:06:21.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317847+0000", 2023-07-02T14:06:21.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0223999999999996e-05 2023-07-02T14:06:21.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317927+0000", 2023-07-02T14:06:21.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049543649999999996 2023-07-02T14:06:21.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322882+0000", 2023-07-02T14:06:21.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1399000000000002e-05 2023-07-02T14:06:21.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322913+0000", 2023-07-02T14:06:21.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006534214 2023-07-02T14:06:21.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329447+0000", 2023-07-02T14:06:21.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9649999999999998e-05 2023-07-02T14:06:21.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329527+0000", 2023-07-02T14:06:21.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061026839999999997 2023-07-02T14:06:21.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335629+0000", 2023-07-02T14:06:21.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9611000000000002e-05 2023-07-02T14:06:21.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335659+0000", 2023-07-02T14:06:21.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098851549999999996 2023-07-02T14:06:21.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345544+0000", 2023-07-02T14:06:21.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0587999999999999e-05 2023-07-02T14:06:21.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345575+0000", 2023-07-02T14:06:21.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046790859999999998 2023-07-02T14:06:21.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350254+0000", 2023-07-02T14:06:21.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0284000000000002e-05 2023-07-02T14:06:21.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350304+0000", 2023-07-02T14:06:21.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041560540000000002 2023-07-02T14:06:21.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354460+0000", 2023-07-02T14:06:21.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2427000000000004e-05 2023-07-02T14:06:21.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354523+0000", 2023-07-02T14:06:21.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071268770000000002 2023-07-02T14:06:21.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361650+0000", 2023-07-02T14:06:21.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0026999999999999e-05 2023-07-02T14:06:21.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361680+0000", 2023-07-02T14:06:21.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073733080000000003 2023-07-02T14:06:21.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369053+0000", 2023-07-02T14:06:21.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7635999999999997e-05 2023-07-02T14:06:21.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369091+0000", 2023-07-02T14:06:21.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056908059999999996 2023-07-02T14:06:21.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374781+0000", 2023-07-02T14:06:21.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4656000000000002e-05 2023-07-02T14:06:21.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374816+0000", 2023-07-02T14:06:21.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077037809999999998 2023-07-02T14:06:21.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382520+0000", 2023-07-02T14:06:21.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7858e-05 2023-07-02T14:06:21.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382588+0000", 2023-07-02T14:06:21.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057884019999999998 2023-07-02T14:06:21.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388376+0000", 2023-07-02T14:06:21.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4831000000000001e-05 2023-07-02T14:06:21.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388461+0000", 2023-07-02T14:06:21.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067708480000000003 2023-07-02T14:06:21.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395232+0000", 2023-07-02T14:06:21.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7948000000000002e-05 2023-07-02T14:06:21.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:21.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395290+0000", 2023-07-02T14:06:21.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00027672599999999998 2023-07-02T14:06:21.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:21.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395566+0000", 2023-07-02T14:06:21.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025705569999999998 2023-07-02T14:06:21.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:21.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.398137+0000", 2023-07-02T14:06:21.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072257099999999998 2023-07-02T14:06:21.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:21.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405363+0000", 2023-07-02T14:06:21.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3887999999999998e-05 2023-07-02T14:06:21.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:21.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405397+0000", 2023-07-02T14:06:21.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000105769 2023-07-02T14:06:21.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:21.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405502+0000", 2023-07-02T14:06:21.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.15647551900000001 2023-07-02T14:06:21.116 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:21.116 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:21.116 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:21.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.116 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18561 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:21.117 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:21.117 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 15.460446035, 2023-07-02T14:06:21.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744, 2023-07-02T14:06:21.117 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:21.117 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:21.117 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:21.117 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:21.117 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:21.118 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18561 2023-07-02T14:06:21.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.118 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:21.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:21.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:21.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:21.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:21.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:21.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:21.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:21.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:21.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.563e-06 2023-07-02T14:06:21.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:21.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249050+0000", 2023-07-02T14:06:21.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4500000000000001e-06 2023-07-02T14:06:21.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:21.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249057+0000", 2023-07-02T14:06:21.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1860000000000002e-06 2023-07-02T14:06:21.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:21.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249061+0000", 2023-07-02T14:06:21.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00091153399999999998 2023-07-02T14:06:21.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249972+0000", 2023-07-02T14:06:21.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7156999999999997e-05 2023-07-02T14:06:21.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.250010+0000", 2023-07-02T14:06:21.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006641816 2023-07-02T14:06:21.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256651+0000", 2023-07-02T14:06:21.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1968999999999998e-05 2023-07-02T14:06:21.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256683+0000", 2023-07-02T14:06:21.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012463952 2023-07-02T14:06:21.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269147+0000", 2023-07-02T14:06:21.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2855000000000006e-05 2023-07-02T14:06:21.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269220+0000", 2023-07-02T14:06:21.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010870418999999999 2023-07-02T14:06:21.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280091+0000", 2023-07-02T14:06:21.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1921999999999997e-05 2023-07-02T14:06:21.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280123+0000", 2023-07-02T14:06:21.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011861682 2023-07-02T14:06:21.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291984+0000", 2023-07-02T14:06:21.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5259000000000006e-05 2023-07-02T14:06:21.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.292059+0000", 2023-07-02T14:06:21.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079709359999999996 2023-07-02T14:06:21.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300030+0000", 2023-07-02T14:06:21.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0729999999999998e-05 2023-07-02T14:06:21.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300071+0000", 2023-07-02T14:06:21.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072785690000000004 2023-07-02T14:06:21.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307350+0000", 2023-07-02T14:06:21.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1948999999999999e-05 2023-07-02T14:06:21.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307412+0000", 2023-07-02T14:06:21.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055785779999999998 2023-07-02T14:06:21.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312990+0000", 2023-07-02T14:06:21.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0957000000000002e-05 2023-07-02T14:06:21.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.313021+0000", 2023-07-02T14:06:21.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049246400000000001 2023-07-02T14:06:21.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317946+0000", 2023-07-02T14:06:21.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9346e-05 2023-07-02T14:06:21.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317975+0000", 2023-07-02T14:06:21.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049649469999999999 2023-07-02T14:06:21.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322940+0000", 2023-07-02T14:06:21.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9017000000000002e-05 2023-07-02T14:06:21.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322989+0000", 2023-07-02T14:06:21.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065907029999999998 2023-07-02T14:06:21.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329580+0000", 2023-07-02T14:06:21.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5879999999999999e-05 2023-07-02T14:06:21.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329656+0000", 2023-07-02T14:06:21.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060166960000000002 2023-07-02T14:06:21.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335672+0000", 2023-07-02T14:06:21.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8963e-05 2023-07-02T14:06:21.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335701+0000", 2023-07-02T14:06:21.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098989560000000004 2023-07-02T14:06:21.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345600+0000", 2023-07-02T14:06:21.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5395999999999995e-05 2023-07-02T14:06:21.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345666+0000", 2023-07-02T14:06:21.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046589530000000004 2023-07-02T14:06:21.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350325+0000", 2023-07-02T14:06:21.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2459000000000002e-05 2023-07-02T14:06:21.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350357+0000", 2023-07-02T14:06:21.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042054290000000001 2023-07-02T14:06:21.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354563+0000", 2023-07-02T14:06:21.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5612999999999999e-05 2023-07-02T14:06:21.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354598+0000", 2023-07-02T14:06:21.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070967449999999998 2023-07-02T14:06:21.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361695+0000", 2023-07-02T14:06:21.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9314000000000001e-05 2023-07-02T14:06:21.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361724+0000", 2023-07-02T14:06:21.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007391028 2023-07-02T14:06:21.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369115+0000", 2023-07-02T14:06:21.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9163000000000002e-05 2023-07-02T14:06:21.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369184+0000", 2023-07-02T14:06:21.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056532500000000003 2023-07-02T14:06:21.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374838+0000", 2023-07-02T14:06:21.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5297999999999999e-05 2023-07-02T14:06:21.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374883+0000", 2023-07-02T14:06:21.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077292189999999998 2023-07-02T14:06:21.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382612+0000", 2023-07-02T14:06:21.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1102000000000001e-05 2023-07-02T14:06:21.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382653+0000", 2023-07-02T14:06:21.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058515989999999999 2023-07-02T14:06:21.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388505+0000", 2023-07-02T14:06:21.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9196000000000006e-05 2023-07-02T14:06:21.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388574+0000", 2023-07-02T14:06:21.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084245339999999991 2023-07-02T14:06:21.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.396999+0000", 2023-07-02T14:06:21.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0596999999999998e-05 2023-07-02T14:06:21.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.397039+0000", 2023-07-02T14:06:21.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084911339999999991 2023-07-02T14:06:21.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405530+0000", 2023-07-02T14:06:21.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7555000000000001e-05 2023-07-02T14:06:21.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:21.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405578+0000", 2023-07-02T14:06:21.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000229607 2023-07-02T14:06:21.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:21.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405808+0000", 2023-07-02T14:06:21.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026919940000000001 2023-07-02T14:06:21.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:21.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.408500+0000", 2023-07-02T14:06:21.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053065059999999999 2023-07-02T14:06:21.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:21.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413806+0000", 2023-07-02T14:06:21.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0490000000000001e-05 2023-07-02T14:06:21.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:21.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413837+0000", 2023-07-02T14:06:21.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000100386 2023-07-02T14:06:21.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:21.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413937+0000", 2023-07-02T14:06:21.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744 2023-07-02T14:06:21.153 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:21.153 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:21.153 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:21.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.153 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18879 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e177)", 2023-07-02T14:06:21.153 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:06.898926+0000", 2023-07-02T14:06:21.154 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 13.810566995, 2023-07-02T14:06:21.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.108448455, 2023-07-02T14:06:21.154 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:21.154 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:21.154 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:21.154 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:21.154 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:21.154 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18879 2023-07-02T14:06:21.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.155 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:21.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:21.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.898926+0000", 2023-07-02T14:06:21.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:21.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:21.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.898926+0000", 2023-07-02T14:06:21.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:21.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:21.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.898926+0000", 2023-07-02T14:06:21.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.45e-06 2023-07-02T14:06:21.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:21.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.898929+0000", 2023-07-02T14:06:21.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4110000000000004e-06 2023-07-02T14:06:21.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:21.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.898934+0000", 2023-07-02T14:06:21.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.168e-06 2023-07-02T14:06:21.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:21.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.898938+0000", 2023-07-02T14:06:21.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015109999999999999 2023-07-02T14:06:21.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.899089+0000", 2023-07-02T14:06:21.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3013000000000003e-05 2023-07-02T14:06:21.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.899132+0000", 2023-07-02T14:06:21.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033983960000000001 2023-07-02T14:06:21.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.902530+0000", 2023-07-02T14:06:21.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2097000000000001e-05 2023-07-02T14:06:21.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.902562+0000", 2023-07-02T14:06:21.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013082125 2023-07-02T14:06:21.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.915644+0000", 2023-07-02T14:06:21.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1718999999999998e-05 2023-07-02T14:06:21.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.915686+0000", 2023-07-02T14:06:21.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068310189999999998 2023-07-02T14:06:21.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.922517+0000", 2023-07-02T14:06:21.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1585999999999999e-05 2023-07-02T14:06:21.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.922549+0000", 2023-07-02T14:06:21.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014397978000000001 2023-07-02T14:06:21.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.936947+0000", 2023-07-02T14:06:21.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2971000000000002e-05 2023-07-02T14:06:21.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.937030+0000", 2023-07-02T14:06:21.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070590799999999997 2023-07-02T14:06:21.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.944089+0000", 2023-07-02T14:06:21.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1683000000000001e-05 2023-07-02T14:06:21.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.944120+0000", 2023-07-02T14:06:21.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089819119999999999 2023-07-02T14:06:21.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.953102+0000", 2023-07-02T14:06:21.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7537999999999999e-05 2023-07-02T14:06:21.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.953140+0000", 2023-07-02T14:06:21.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016871374000000001 2023-07-02T14:06:21.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.970011+0000", 2023-07-02T14:06:21.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0485000000000002e-05 2023-07-02T14:06:21.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.970062+0000", 2023-07-02T14:06:21.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013052505000000001 2023-07-02T14:06:21.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.983114+0000", 2023-07-02T14:06:21.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8034999999999997e-05 2023-07-02T14:06:21.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:21.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.983162+0000", 2023-07-02T14:06:21.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013268611 2023-07-02T14:06:21.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.996431+0000", 2023-07-02T14:06:21.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0791000000000001e-05 2023-07-02T14:06:21.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:21.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.996492+0000", 2023-07-02T14:06:21.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00036019099999999999 2023-07-02T14:06:21.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:21.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.996852+0000", 2023-07-02T14:06:21.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004127917 2023-07-02T14:06:21.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:21.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:07.000980+0000", 2023-07-02T14:06:21.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061381760000000004 2023-07-02T14:06:21.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:21.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:07.007118+0000", 2023-07-02T14:06:21.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1156999999999997e-05 2023-07-02T14:06:21.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:21.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:07.007189+0000", 2023-07-02T14:06:21.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000185639 2023-07-02T14:06:21.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:21.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:07.007375+0000", 2023-07-02T14:06:21.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.108448455 2023-07-02T14:06:21.174 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:21.175 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:21.175 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:21.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.175 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20768 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch ping cookie 140544751726928 gen 52] snapc 0=[] ondisk+write+known_if_redirected e183)", 2023-07-02T14:06:21.175 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:21.175 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 1.4870787670000001, 2023-07-02T14:06:21.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3751256970000001, 2023-07-02T14:06:21.176 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:21.176 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:21.176 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:21.176 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:21.176 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:21.176 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20768 2023-07-02T14:06:21.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.176 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:21.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:21.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:21.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:21.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:21.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:21.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:21.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:21.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:21.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7829999999999998e-06 2023-07-02T14:06:21.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:21.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222419+0000", 2023-07-02T14:06:21.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1380000000000004e-06 2023-07-02T14:06:21.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:21.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222425+0000", 2023-07-02T14:06:21.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2519999999999996e-06 2023-07-02T14:06:21.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:21.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222431+0000", 2023-07-02T14:06:21.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2157999999999994e-05 2023-07-02T14:06:21.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222523+0000", 2023-07-02T14:06:21.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5147000000000001e-05 2023-07-02T14:06:21.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:21.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222538+0000", 2023-07-02T14:06:21.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30105419700000002 2023-07-02T14:06:21.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523592+0000", 2023-07-02T14:06:21.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2316e-05 2023-07-02T14:06:21.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:21.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523605+0000", 2023-07-02T14:06:21.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.073658953 2023-07-02T14:06:21.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597264+0000", 2023-07-02T14:06:21.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000213415 2023-07-02T14:06:21.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:21.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597477+0000", 2023-07-02T14:06:21.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3034999999999995e-05 2023-07-02T14:06:21.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:21.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597540+0000", 2023-07-02T14:06:21.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3751256970000001 2023-07-02T14:06:21.185 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:21.185 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:21.185 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:21.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.185 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20769 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 53] snapc 0=[] ondisk+write+known_if_redirected e184)", 2023-07-02T14:06:21.185 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:21.185 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 1.486716312, 2023-07-02T14:06:21.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.380884139, 2023-07-02T14:06:21.186 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:21.186 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:21.186 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:21.186 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:21.186 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:21.186 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20769 2023-07-02T14:06:21.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.187 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:21.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:21.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:21.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:21.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:21.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:21.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:21.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:21.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222776+0000", 2023-07-02T14:06:21.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9710000000000003e-06 2023-07-02T14:06:21.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:21.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222781+0000", 2023-07-02T14:06:21.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9899999999999992e-06 2023-07-02T14:06:21.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:21.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222791+0000", 2023-07-02T14:06:21.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4240000000000003e-06 2023-07-02T14:06:21.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:21.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222795+0000", 2023-07-02T14:06:21.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2366000000000001e-05 2023-07-02T14:06:21.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222828+0000", 2023-07-02T14:06:21.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4790000000000001e-06 2023-07-02T14:06:21.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:21.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222834+0000", 2023-07-02T14:06:21.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30084589299999998 2023-07-02T14:06:21.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523680+0000", 2023-07-02T14:06:21.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.87e-06 2023-07-02T14:06:21.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:21.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523685+0000", 2023-07-02T14:06:21.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.074111042 2023-07-02T14:06:21.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597796+0000", 2023-07-02T14:06:21.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0411000000000001e-05 2023-07-02T14:06:21.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:21.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597856+0000", 2023-07-02T14:06:21.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058045340000000001 2023-07-02T14:06:21.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:21.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.603661+0000", 2023-07-02T14:06:21.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.380884139 2023-07-02T14:06:21.195 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:21.195 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:21.195 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:21.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.195 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20770 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 54] snapc 0=[] ondisk+write+known_if_redirected e185)", 2023-07-02T14:06:21.196 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:21.196 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 1.1854992019999999, 2023-07-02T14:06:21.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0799819349999999, 2023-07-02T14:06:21.196 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:21.196 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:21.196 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:21.196 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:21.197 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:21.197 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20770 2023-07-02T14:06:21.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.197 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:21.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:21.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:21.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:21.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:21.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:21.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:21.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:21.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523992+0000", 2023-07-02T14:06:21.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6720000000000004e-06 2023-07-02T14:06:21.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.199 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:21.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524000+0000", 2023-07-02T14:06:21.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3711e-05 2023-07-02T14:06:21.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:21.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524014+0000", 2023-07-02T14:06:21.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7849999999999992e-06 2023-07-02T14:06:21.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:21.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524022+0000", 2023-07-02T14:06:21.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2366000000000001e-05 2023-07-02T14:06:21.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524054+0000", 2023-07-02T14:06:21.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8169999999999998e-06 2023-07-02T14:06:21.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:21.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524059+0000", 2023-07-02T14:06:21.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0748338850000001 2023-07-02T14:06:21.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:21.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.598893+0000", 2023-07-02T14:06:21.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116968 2023-07-02T14:06:21.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:21.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.599010+0000", 2023-07-02T14:06:21.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049663370000000004 2023-07-02T14:06:21.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:21.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:21.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:21.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.603976+0000", 2023-07-02T14:06:21.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0799819349999999 2023-07-02T14:06:21.204 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:21.204 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:21.204 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:21.204 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:21.204 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:21.205 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:21.206 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 100% complete...done. 2023-07-02T14:06:21.206 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:06:21.206 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=53 2023-07-02T14:06:21.207 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap53 2023-07-02T14:06:21.207 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:06:21.207 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:21.205+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:21.227 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:06:21.306 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:21.305+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:21.334 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:21.335 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:21.345 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:06:21.390 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:21.407 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:21.406+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:21.409 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:21.426 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:21.444 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:21.462 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:21.480 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:21.482 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:06:21.482 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:06:21.482 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:06:21.482 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:06:21.482 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:06:21.482 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:06:21.512 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:21.512 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:21.513 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:21.511+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:21.523 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:06:21.529 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:06:21.592 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:06:21.601 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 1 2023-07-02T14:06:21.607 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:21.607+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:21.658 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:21.658 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:21.668 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:06:21.708 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:21.708+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:21.792 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:21.793 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:21.804 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:06:21.809 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:21.808+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:21.910 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:21.909+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:21.927 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:21.927 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:21.937 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:06:22.010 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:22.009+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:22.061 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:22.061 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:22.071 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:06:22.111 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:22.110+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:22.195 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:22.195 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:22.205 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:06:22.212 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:22.211+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:22.312 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:22.312+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:22.329 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:22.329 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:22.339 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:06:22.413 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:22.412+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:22.463 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:22.463 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:22.474 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:06:22.516 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:22.516+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:22.603 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap52 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap53 b3fbb028-e267-4d25-a750-93ba18a2316a@snap53.diff 2023-07-02T14:06:22.656 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:22.656 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:22.657 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:06:22.657 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:22.655+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:22.665 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 100% complete...done. 2023-07-02T14:06:22.668 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:06:22.669 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=54 2023-07-02T14:06:22.669 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap54 2023-07-02T14:06:22.757 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:22.756+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:22.847 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:22.847 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:22.858 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:22.858+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:22.859 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:06:22.958 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:22.957+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:23.009 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:23.009 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:23.022 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:06:23.059 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:23.058+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:23.160 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:23.159+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:23.215 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:23.216 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:23.229 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:06:23.261 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:23.261+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:23.362 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:23.362+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:23.424 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:23.424 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:23.438 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:06:23.463 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:23.463+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:23.494 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:06:23.536 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 2, 4] out_osds: [5] dead_osds: [] live_osds: [1, 2, 4, 5, 3, 0] 2023-07-02T14:06:23.536 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-07-02T14:06:23.541 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:06:23.581 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:23.580+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:23.595 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:06:23.599 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:23.599 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:23.606 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 2 2023-07-02T14:06:23.613 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:06:23.664 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:23.663+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:23.666 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:23.686 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:23.704 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:23.724 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:23.743 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:23.762 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:23.763 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:06:23.763 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:06:23.763 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:06:23.764 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:06:23.764 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:06:23.764 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:06:23.765 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:23.764+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:23.782 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:23.782 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:23.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:23.783 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPushReply(2.7 188/185 [PushReplyOp(2:f1d0abf0:::rbd_data.125de1449605.0000000000000083:head)])", 2023-07-02T14:06:23.783 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:23.781647+0000", 2023-07-02T14:06:23.783 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.00081871899999999998, 2023-07-02T14:06:23.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00083620700000000005, 2023-07-02T14:06:23.783 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:23.783 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "reached pg", 2023-07-02T14:06:23.783 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:23.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:23.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:23.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:23.781647+0000", 2023-07-02T14:06:23.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:23.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:23.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:23.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:23.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:23.781647+0000", 2023-07-02T14:06:23.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:06:23.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:23.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:23.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:23.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:23.781647+0000", 2023-07-02T14:06:23.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.084e-06 2023-07-02T14:06:23.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:23.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:23.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:23.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:23.781650+0000", 2023-07-02T14:06:23.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.392e-06 2023-07-02T14:06:23.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:23.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:23.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:23.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:23.781656+0000", 2023-07-02T14:06:23.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8870000000000002e-06 2023-07-02T14:06:23.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:23.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:23.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:23.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:23.781660+0000", 2023-07-02T14:06:23.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024888599999999997 2023-07-02T14:06:23.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:23.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:23.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:23.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:23.781909+0000", 2023-07-02T14:06:23.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026201499999999999 2023-07-02T14:06:23.788 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:23.788 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:23.788 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:23.788 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:23.789 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2023-07-02T14:06:23.789 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 1 2023-07-02T14:06:23.789 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:23.797 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:06:23.880 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:23.880+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:23.967 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:23.966+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:24.001 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:24.001 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:24.001 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:24.001 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:24.002 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:24.015 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:06:24.067 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:24.066+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:24.168 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:24.167+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:24.198 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:24.198 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:24.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:24.198 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:24.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.199 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:16941 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:24.199 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:06:24.199 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 41.337322892000003, 2023-07-02T14:06:24.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11449124500000001, 2023-07-02T14:06:24.199 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:24.199 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:24.200 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:24.200 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:24.200 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:24.200 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 16941 2023-07-02T14:06:24.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.200 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:24.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:24.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:06:24.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:24.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:24.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855878+0000", 2023-07-02T14:06:24.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:06:24.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:24.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855877+0000", 2023-07-02T14:06:24.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3209999999999999e-06 2023-07-02T14:06:24.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:24.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855880+0000", 2023-07-02T14:06:24.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9299999999999999e-06 2023-07-02T14:06:24.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:24.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855883+0000", 2023-07-02T14:06:24.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6309999999999999e-06 2023-07-02T14:06:24.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:24.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.855885+0000", 2023-07-02T14:06:24.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050516850000000002 2023-07-02T14:06:24.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.860937+0000", 2023-07-02T14:06:24.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1844999999999997e-05 2023-07-02T14:06:24.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.860979+0000", 2023-07-02T14:06:24.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055853480000000004 2023-07-02T14:06:24.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.866564+0000", 2023-07-02T14:06:24.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2821999999999998e-05 2023-07-02T14:06:24.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.866597+0000", 2023-07-02T14:06:24.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050938900000000002 2023-07-02T14:06:24.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.871691+0000", 2023-07-02T14:06:24.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1210000000000001e-05 2023-07-02T14:06:24.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.871722+0000", 2023-07-02T14:06:24.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014768687000000001 2023-07-02T14:06:24.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.886491+0000", 2023-07-02T14:06:24.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7248000000000003e-05 2023-07-02T14:06:24.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.886568+0000", 2023-07-02T14:06:24.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015403168 2023-07-02T14:06:24.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.901971+0000", 2023-07-02T14:06:24.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4035000000000002e-05 2023-07-02T14:06:24.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.902005+0000", 2023-07-02T14:06:24.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014618580000000001 2023-07-02T14:06:24.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.916624+0000", 2023-07-02T14:06:24.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.443e-05 2023-07-02T14:06:24.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.916668+0000", 2023-07-02T14:06:24.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099598989999999995 2023-07-02T14:06:24.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.926628+0000", 2023-07-02T14:06:24.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3123999999999999e-05 2023-07-02T14:06:24.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.926711+0000", 2023-07-02T14:06:24.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046031930000000002 2023-07-02T14:06:24.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.931314+0000", 2023-07-02T14:06:24.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3257000000000003e-05 2023-07-02T14:06:24.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.931358+0000", 2023-07-02T14:06:24.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078370969999999995 2023-07-02T14:06:24.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.939195+0000", 2023-07-02T14:06:24.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2522000000000003e-05 2023-07-02T14:06:24.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.939247+0000", 2023-07-02T14:06:24.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01118103 2023-07-02T14:06:24.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.950428+0000", 2023-07-02T14:06:24.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5001999999999996e-05 2023-07-02T14:06:24.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.950523+0000", 2023-07-02T14:06:24.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089648390000000005 2023-07-02T14:06:24.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959488+0000", 2023-07-02T14:06:24.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4569e-05 2023-07-02T14:06:24.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:24.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959533+0000", 2023-07-02T14:06:24.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026104299999999997 2023-07-02T14:06:24.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:24.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.959794+0000", 2023-07-02T14:06:24.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025221789999999998 2023-07-02T14:06:24.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:24.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.962316+0000", 2023-07-02T14:06:24.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079080520000000005 2023-07-02T14:06:24.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:24.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970224+0000", 2023-07-02T14:06:24.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1265e-05 2023-07-02T14:06:24.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:24.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970255+0000", 2023-07-02T14:06:24.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000113581 2023-07-02T14:06:24.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:24.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.970369+0000", 2023-07-02T14:06:24.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11449124500000001 2023-07-02T14:06:24.230 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.230 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.230 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.231 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17049 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:24.231 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:06:24.231 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 41.213817900999999, 2023-07-02T14:06:24.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.112277108, 2023-07-02T14:06:24.232 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:24.232 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:24.233 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:24.233 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:24.233 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:24.233 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17049 2023-07-02T14:06:24.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.234 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:24.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:24.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:06:24.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:24.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:24.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979383+0000", 2023-07-02T14:06:24.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:24.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:24.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979382+0000", 2023-07-02T14:06:24.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3450000000000002e-06 2023-07-02T14:06:24.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:24.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979385+0000", 2023-07-02T14:06:24.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9000000000000003e-06 2023-07-02T14:06:24.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:24.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979391+0000", 2023-07-02T14:06:24.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9999999999999998e-06 2023-07-02T14:06:24.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:24.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979395+0000", 2023-07-02T14:06:24.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013089412 2023-07-02T14:06:24.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992485+0000", 2023-07-02T14:06:24.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3828999999999997e-05 2023-07-02T14:06:24.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992519+0000", 2023-07-02T14:06:24.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062549889999999999 2023-07-02T14:06:24.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.245 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998774+0000", 2023-07-02T14:06:24.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4870999999999995e-05 2023-07-02T14:06:24.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.246 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998848+0000", 2023-07-02T14:06:24.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091715140000000004 2023-07-02T14:06:24.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008020+0000", 2023-07-02T14:06:24.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8659999999999999e-05 2023-07-02T14:06:24.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008059+0000", 2023-07-02T14:06:24.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059565110000000003 2023-07-02T14:06:24.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014015+0000", 2023-07-02T14:06:24.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8088000000000002e-05 2023-07-02T14:06:24.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.251 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.251 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014093+0000", 2023-07-02T14:06:24.251 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007513395 2023-07-02T14:06:24.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.252 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021607+0000", 2023-07-02T14:06:24.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9777e-05 2023-07-02T14:06:24.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021636+0000", 2023-07-02T14:06:24.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056999549999999996 2023-07-02T14:06:24.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027336+0000", 2023-07-02T14:06:24.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8356999999999996e-05 2023-07-02T14:06:24.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027405+0000", 2023-07-02T14:06:24.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047942130000000003 2023-07-02T14:06:24.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032199+0000", 2023-07-02T14:06:24.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1137999999999999e-05 2023-07-02T14:06:24.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032230+0000", 2023-07-02T14:06:24.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057056759999999998 2023-07-02T14:06:24.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.037936+0000", 2023-07-02T14:06:24.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6334999999999993e-05 2023-07-02T14:06:24.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.261 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.261 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038012+0000", 2023-07-02T14:06:24.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043985609999999996 2023-07-02T14:06:24.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.262 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.262 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.262 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042411+0000", 2023-07-02T14:06:24.263 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6857e-05 2023-07-02T14:06:24.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042487+0000", 2023-07-02T14:06:24.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007284446 2023-07-02T14:06:24.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.265 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049772+0000", 2023-07-02T14:06:24.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3757000000000006e-05 2023-07-02T14:06:24.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049846+0000", 2023-07-02T14:06:24.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0095186679999999992 2023-07-02T14:06:24.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059364+0000", 2023-07-02T14:06:24.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9931000000000005e-05 2023-07-02T14:06:24.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.268 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059434+0000", 2023-07-02T14:06:24.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079904920000000001 2023-07-02T14:06:24.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067425+0000", 2023-07-02T14:06:24.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5583000000000001e-05 2023-07-02T14:06:24.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067460+0000", 2023-07-02T14:06:24.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009114278 2023-07-02T14:06:24.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076575+0000", 2023-07-02T14:06:24.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5880000000000001e-05 2023-07-02T14:06:24.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.271 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076620+0000", 2023-07-02T14:06:24.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055118470000000003 2023-07-02T14:06:24.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.271 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082132+0000", 2023-07-02T14:06:24.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8306999999999997e-05 2023-07-02T14:06:24.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:24.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082181+0000", 2023-07-02T14:06:24.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026259599999999998 2023-07-02T14:06:24.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.273 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:24.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.082443+0000", 2023-07-02T14:06:24.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018855339999999999 2023-07-02T14:06:24.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:24.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.084329+0000", 2023-07-02T14:06:24.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071877729999999997 2023-07-02T14:06:24.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:24.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091517+0000", 2023-07-02T14:06:24.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3424999999999999e-05 2023-07-02T14:06:24.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.275 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:24.275 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091550+0000", 2023-07-02T14:06:24.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109724 2023-07-02T14:06:24.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.275 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:24.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091660+0000", 2023-07-02T14:06:24.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.112277108 2023-07-02T14:06:24.276 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.276 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.276 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.276 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.277 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17050 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:24.277 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:06:24.277 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 41.213800976000002, 2023-07-02T14:06:24.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12076701300000001, 2023-07-02T14:06:24.277 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:24.277 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:24.277 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:24.278 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:24.278 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:24.278 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17050 2023-07-02T14:06:24.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.278 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:24.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:24.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:06:24.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:24.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:24.279 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:06:24.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:24.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:24.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979399+0000", 2023-07-02T14:06:24.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0759999999999999e-06 2023-07-02T14:06:24.280 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:24.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979402+0000", 2023-07-02T14:06:24.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0480000000000004e-06 2023-07-02T14:06:24.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:24.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979408+0000", 2023-07-02T14:06:24.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9559999999999999e-06 2023-07-02T14:06:24.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:24.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.979412+0000", 2023-07-02T14:06:24.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013117658000000001 2023-07-02T14:06:24.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.282 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992530+0000", 2023-07-02T14:06:24.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4464000000000001e-05 2023-07-02T14:06:24.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.283 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.992564+0000", 2023-07-02T14:06:24.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0063143640000000003 2023-07-02T14:06:24.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998879+0000", 2023-07-02T14:06:24.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2842000000000002e-05 2023-07-02T14:06:24.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:42.998911+0000", 2023-07-02T14:06:24.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091863190000000001 2023-07-02T14:06:24.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008098+0000", 2023-07-02T14:06:24.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1448999999999999e-05 2023-07-02T14:06:24.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.008179+0000", 2023-07-02T14:06:24.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059353720000000004 2023-07-02T14:06:24.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014115+0000", 2023-07-02T14:06:24.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1928000000000002e-05 2023-07-02T14:06:24.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.014157+0000", 2023-07-02T14:06:24.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074935210000000004 2023-07-02T14:06:24.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021650+0000", 2023-07-02T14:06:24.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0199999999999999e-05 2023-07-02T14:06:24.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.021680+0000", 2023-07-02T14:06:24.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057425109999999996 2023-07-02T14:06:24.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027423+0000", 2023-07-02T14:06:24.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1498999999999998e-05 2023-07-02T14:06:24.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.027454+0000", 2023-07-02T14:06:24.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047903420000000004 2023-07-02T14:06:24.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032245+0000", 2023-07-02T14:06:24.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2552999999999998e-05 2023-07-02T14:06:24.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.032277+0000", 2023-07-02T14:06:24.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057638079999999996 2023-07-02T14:06:24.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038041+0000", 2023-07-02T14:06:24.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9168000000000002e-05 2023-07-02T14:06:24.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.038080+0000", 2023-07-02T14:06:24.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044291799999999996 2023-07-02T14:06:24.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042509+0000", 2023-07-02T14:06:24.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3528999999999997e-05 2023-07-02T14:06:24.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.042543+0000", 2023-07-02T14:06:24.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073244729999999997 2023-07-02T14:06:24.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.294 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049867+0000", 2023-07-02T14:06:24.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2873999999999997e-05 2023-07-02T14:06:24.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.049900+0000", 2023-07-02T14:06:24.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0095670149999999999 2023-07-02T14:06:24.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.295 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059467+0000", 2023-07-02T14:06:24.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0406000000000003e-05 2023-07-02T14:06:24.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.059538+0000", 2023-07-02T14:06:24.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079635079999999994 2023-07-02T14:06:24.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067501+0000", 2023-07-02T14:06:24.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4795999999999997e-05 2023-07-02T14:06:24.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.067536+0000", 2023-07-02T14:06:24.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091231969999999996 2023-07-02T14:06:24.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076659+0000", 2023-07-02T14:06:24.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3208999999999999e-05 2023-07-02T14:06:24.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.076702+0000", 2023-07-02T14:06:24.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070660480000000001 2023-07-02T14:06:24.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.083768+0000", 2023-07-02T14:06:24.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5546000000000001e-05 2023-07-02T14:06:24.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.083814+0000", 2023-07-02T14:06:24.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078830659999999993 2023-07-02T14:06:24.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091697+0000", 2023-07-02T14:06:24.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5921000000000003e-05 2023-07-02T14:06:24.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:24.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.091743+0000", 2023-07-02T14:06:24.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026571400000000001 2023-07-02T14:06:24.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:24.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.092009+0000", 2023-07-02T14:06:24.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022058429999999999 2023-07-02T14:06:24.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:24.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.094214+0000", 2023-07-02T14:06:24.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057115869999999997 2023-07-02T14:06:24.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:24.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.099926+0000", 2023-07-02T14:06:24.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0287999999999997e-05 2023-07-02T14:06:24.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:24.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.099976+0000", 2023-07-02T14:06:24.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00019014999999999999 2023-07-02T14:06:24.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:24.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.100166+0000", 2023-07-02T14:06:24.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12076701300000001 2023-07-02T14:06:24.305 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.305 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.305 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.305 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17353 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:24.305 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:06:24.306 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.890616299999998, 2023-07-02T14:06:24.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.115724379, 2023-07-02T14:06:24.306 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:24.306 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:24.306 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:24.306 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:24.306 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:24.307 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17353 2023-07-02T14:06:24.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.307 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:24.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:24.307 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:06:24.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:24.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:24.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:06:24.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:24.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:24.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302584+0000", 2023-07-02T14:06:24.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.41e-06 2023-07-02T14:06:24.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:24.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302587+0000", 2023-07-02T14:06:24.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.015e-06 2023-07-02T14:06:24.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:24.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302593+0000", 2023-07-02T14:06:24.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8019999999999997e-06 2023-07-02T14:06:24.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:24.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302599+0000", 2023-07-02T14:06:24.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00022701199999999999 2023-07-02T14:06:24.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302826+0000", 2023-07-02T14:06:24.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.426e-05 2023-07-02T14:06:24.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302870+0000", 2023-07-02T14:06:24.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096759489999999997 2023-07-02T14:06:24.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312546+0000", 2023-07-02T14:06:24.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5785000000000006e-05 2023-07-02T14:06:24.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312632+0000", 2023-07-02T14:06:24.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098106970000000002 2023-07-02T14:06:24.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322443+0000", 2023-07-02T14:06:24.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3161999999999999e-05 2023-07-02T14:06:24.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322476+0000", 2023-07-02T14:06:24.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010015823 2023-07-02T14:06:24.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332492+0000", 2023-07-02T14:06:24.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1165999999999999e-05 2023-07-02T14:06:24.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332533+0000", 2023-07-02T14:06:24.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089435269999999997 2023-07-02T14:06:24.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341476+0000", 2023-07-02T14:06:24.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000112691 2023-07-02T14:06:24.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341589+0000", 2023-07-02T14:06:24.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.008147362 2023-07-02T14:06:24.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349736+0000", 2023-07-02T14:06:24.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5149000000000007e-05 2023-07-02T14:06:24.318 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.318 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.318 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349832+0000", 2023-07-02T14:06:24.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089226519999999997 2023-07-02T14:06:24.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.319 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358754+0000", 2023-07-02T14:06:24.319 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6091999999999998e-05 2023-07-02T14:06:24.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.319 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358810+0000", 2023-07-02T14:06:24.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080318000000000004 2023-07-02T14:06:24.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366842+0000", 2023-07-02T14:06:24.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4321999999999998e-05 2023-07-02T14:06:24.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366916+0000", 2023-07-02T14:06:24.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074664049999999997 2023-07-02T14:06:24.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374383+0000", 2023-07-02T14:06:24.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5460000000000001e-05 2023-07-02T14:06:24.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.322 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374438+0000", 2023-07-02T14:06:24.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091562450000000004 2023-07-02T14:06:24.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383594+0000", 2023-07-02T14:06:24.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2545000000000004e-05 2023-07-02T14:06:24.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383667+0000", 2023-07-02T14:06:24.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013424551 2023-07-02T14:06:24.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.324 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397092+0000", 2023-07-02T14:06:24.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6991999999999997e-05 2023-07-02T14:06:24.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397169+0000", 2023-07-02T14:06:24.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064399139999999997 2023-07-02T14:06:24.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403608+0000", 2023-07-02T14:06:24.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2614000000000006e-05 2023-07-02T14:06:24.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403691+0000", 2023-07-02T14:06:24.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004611693 2023-07-02T14:06:24.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408303+0000", 2023-07-02T14:06:24.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1935999999999995e-05 2023-07-02T14:06:24.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:24.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408395+0000", 2023-07-02T14:06:24.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00051077700000000002 2023-07-02T14:06:24.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:24.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.408906+0000", 2023-07-02T14:06:24.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090767460000000001 2023-07-02T14:06:24.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:24.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.417982+0000", 2023-07-02T14:06:24.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4283999999999997e-05 2023-07-02T14:06:24.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:24.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418037+0000", 2023-07-02T14:06:24.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9667999999999999e-05 2023-07-02T14:06:24.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:24.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418096+0000", 2023-07-02T14:06:24.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021233800000000001 2023-07-02T14:06:24.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:24.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418309+0000", 2023-07-02T14:06:24.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.115724379 2023-07-02T14:06:24.331 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.331 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.331 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.332 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17354 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:24.332 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:24.332 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.890597339999999, 2023-07-02T14:06:24.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493, 2023-07-02T14:06:24.332 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:24.332 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:24.333 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:24.333 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:24.333 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:24.333 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17354 2023-07-02T14:06:24.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.333 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:24.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:24.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:24.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:24.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:24.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:24.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:24.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:24.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:24.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7440000000000001e-06 2023-07-02T14:06:24.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:24.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302606+0000", 2023-07-02T14:06:24.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:06:24.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:24.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302613+0000", 2023-07-02T14:06:24.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8810000000000004e-06 2023-07-02T14:06:24.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:24.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302618+0000", 2023-07-02T14:06:24.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028613100000000002 2023-07-02T14:06:24.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302904+0000", 2023-07-02T14:06:24.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5278999999999995e-05 2023-07-02T14:06:24.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302979+0000", 2023-07-02T14:06:24.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096904750000000005 2023-07-02T14:06:24.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312669+0000", 2023-07-02T14:06:24.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0328000000000002e-05 2023-07-02T14:06:24.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312700+0000", 2023-07-02T14:06:24.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098019319999999993 2023-07-02T14:06:24.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322502+0000", 2023-07-02T14:06:24.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4669999999999998e-05 2023-07-02T14:06:24.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322536+0000", 2023-07-02T14:06:24.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010023839 2023-07-02T14:06:24.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332560+0000", 2023-07-02T14:06:24.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5559999999999998e-05 2023-07-02T14:06:24.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332596+0000", 2023-07-02T14:06:24.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090401319999999993 2023-07-02T14:06:24.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341636+0000", 2023-07-02T14:06:24.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5373000000000001e-05 2023-07-02T14:06:24.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341681+0000", 2023-07-02T14:06:24.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081994059999999994 2023-07-02T14:06:24.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349881+0000", 2023-07-02T14:06:24.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4319000000000001e-05 2023-07-02T14:06:24.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349915+0000", 2023-07-02T14:06:24.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089388339999999997 2023-07-02T14:06:24.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358854+0000", 2023-07-02T14:06:24.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4319000000000005e-05 2023-07-02T14:06:24.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358928+0000", 2023-07-02T14:06:24.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080234690000000001 2023-07-02T14:06:24.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366952+0000", 2023-07-02T14:06:24.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2082999999999998e-05 2023-07-02T14:06:24.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366984+0000", 2023-07-02T14:06:24.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075015790000000004 2023-07-02T14:06:24.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374485+0000", 2023-07-02T14:06:24.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4480999999999995e-05 2023-07-02T14:06:24.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374550+0000", 2023-07-02T14:06:24.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091666620000000008 2023-07-02T14:06:24.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383716+0000", 2023-07-02T14:06:24.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9318000000000003e-05 2023-07-02T14:06:24.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383786+0000", 2023-07-02T14:06:24.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013434115 2023-07-02T14:06:24.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397220+0000", 2023-07-02T14:06:24.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4573999999999997e-05 2023-07-02T14:06:24.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397264+0000", 2023-07-02T14:06:24.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064824649999999998 2023-07-02T14:06:24.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403747+0000", 2023-07-02T14:06:24.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1087999999999993e-05 2023-07-02T14:06:24.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403828+0000", 2023-07-02T14:06:24.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072854449999999998 2023-07-02T14:06:24.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411113+0000", 2023-07-02T14:06:24.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5120000000000001e-05 2023-07-02T14:06:24.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411169+0000", 2023-07-02T14:06:24.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007211505 2023-07-02T14:06:24.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418380+0000", 2023-07-02T14:06:24.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011289399999999999 2023-07-02T14:06:24.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:24.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418493+0000", 2023-07-02T14:06:24.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00051217500000000002 2023-07-02T14:06:24.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:24.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.419005+0000", 2023-07-02T14:06:24.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00327393 2023-07-02T14:06:24.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:24.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422279+0000", 2023-07-02T14:06:24.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041432350000000003 2023-07-02T14:06:24.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:24.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426422+0000", 2023-07-02T14:06:24.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7021000000000003e-05 2023-07-02T14:06:24.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:24.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426469+0000", 2023-07-02T14:06:24.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000190256 2023-07-02T14:06:24.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:24.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426660+0000", 2023-07-02T14:06:24.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493 2023-07-02T14:06:24.359 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.359 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.359 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.359 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17355 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:24.359 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:24.360 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 40.890578679000001, 2023-07-02T14:06:24.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001, 2023-07-02T14:06:24.360 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:24.360 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:24.360 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:24.360 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:24.360 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:24.360 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17355 2023-07-02T14:06:24.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.361 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:24.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:24.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:24.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:24.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:24.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:24.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:24.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:24.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302621+0000", 2023-07-02T14:06:24.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2260000000000001e-06 2023-07-02T14:06:24.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:24.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302625+0000", 2023-07-02T14:06:24.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8599999999999998e-06 2023-07-02T14:06:24.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:24.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302630+0000", 2023-07-02T14:06:24.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0629999999999999e-06 2023-07-02T14:06:24.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:24.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302635+0000", 2023-07-02T14:06:24.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035731199999999998 2023-07-02T14:06:24.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302992+0000", 2023-07-02T14:06:24.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4164e-05 2023-07-02T14:06:24.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.303026+0000", 2023-07-02T14:06:24.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097115900000000008 2023-07-02T14:06:24.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312738+0000", 2023-07-02T14:06:24.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1701999999999998e-05 2023-07-02T14:06:24.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312819+0000", 2023-07-02T14:06:24.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097363640000000008 2023-07-02T14:06:24.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322556+0000", 2023-07-02T14:06:24.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2517000000000001e-05 2023-07-02T14:06:24.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322588+0000", 2023-07-02T14:06:24.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01002393 2023-07-02T14:06:24.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332612+0000", 2023-07-02T14:06:24.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2960999999999998e-05 2023-07-02T14:06:24.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332645+0000", 2023-07-02T14:06:24.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090669819999999995 2023-07-02T14:06:24.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341712+0000", 2023-07-02T14:06:24.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9288000000000002e-05 2023-07-02T14:06:24.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341811+0000", 2023-07-02T14:06:24.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081504460000000004 2023-07-02T14:06:24.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349962+0000", 2023-07-02T14:06:24.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107321 2023-07-02T14:06:24.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.350069+0000", 2023-07-02T14:06:24.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088790999999999991 2023-07-02T14:06:24.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358948+0000", 2023-07-02T14:06:24.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1711000000000002e-05 2023-07-02T14:06:24.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358980+0000", 2023-07-02T14:06:24.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080367579999999997 2023-07-02T14:06:24.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367017+0000", 2023-07-02T14:06:24.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6329000000000002e-05 2023-07-02T14:06:24.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367073+0000", 2023-07-02T14:06:24.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075097300000000001 2023-07-02T14:06:24.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374583+0000", 2023-07-02T14:06:24.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5710999999999995e-05 2023-07-02T14:06:24.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374648+0000", 2023-07-02T14:06:24.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009165912 2023-07-02T14:06:24.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383814+0000", 2023-07-02T14:06:24.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3916000000000003e-05 2023-07-02T14:06:24.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383878+0000", 2023-07-02T14:06:24.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013440532999999999 2023-07-02T14:06:24.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397319+0000", 2023-07-02T14:06:24.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7218999999999995e-05 2023-07-02T14:06:24.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397386+0000", 2023-07-02T14:06:24.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065165279999999997 2023-07-02T14:06:24.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403903+0000", 2023-07-02T14:06:24.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8152000000000005e-05 2023-07-02T14:06:24.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404001+0000", 2023-07-02T14:06:24.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072185360000000002 2023-07-02T14:06:24.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411219+0000", 2023-07-02T14:06:24.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:06:24.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411310+0000", 2023-07-02T14:06:24.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010021663 2023-07-02T14:06:24.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421331+0000", 2023-07-02T14:06:24.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0148000000000006e-05 2023-07-02T14:06:24.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421411+0000", 2023-07-02T14:06:24.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053056170000000003 2023-07-02T14:06:24.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426717+0000", 2023-07-02T14:06:24.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8242000000000001e-05 2023-07-02T14:06:24.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:24.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426795+0000", 2023-07-02T14:06:24.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00041308399999999998 2023-07-02T14:06:24.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:24.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.427208+0000", 2023-07-02T14:06:24.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028330849999999999 2023-07-02T14:06:24.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:24.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430041+0000", 2023-07-02T14:06:24.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00072642300000000002 2023-07-02T14:06:24.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:24.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430768+0000", 2023-07-02T14:06:24.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1847000000000002e-05 2023-07-02T14:06:24.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:24.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430800+0000", 2023-07-02T14:06:24.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9308999999999994e-05 2023-07-02T14:06:24.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:24.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430899+0000", 2023-07-02T14:06:24.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001 2023-07-02T14:06:24.387 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.387 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.387 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.387 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:410 2.5 2:bbaed619:::rbd_data.125de1449605.0000000000000109:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:06:24.388 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:24.388 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 27.742633414, 2023-07-02T14:06:24.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003, 2023-07-02T14:06:24.388 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:24.388 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:24.388 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:24.388 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:06:24.389 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:06:24.389 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:06:24.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.389 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:24.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:24.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:24.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:24.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:24.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:24.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:24.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:24.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:24.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.635e-06 2023-07-02T14:06:24.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:24.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450568+0000", 2023-07-02T14:06:24.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5970000000000002e-06 2023-07-02T14:06:24.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:24.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450573+0000", 2023-07-02T14:06:24.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3379999999999999e-06 2023-07-02T14:06:24.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:24.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450575+0000", 2023-07-02T14:06:24.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9458000000000001e-05 2023-07-02T14:06:24.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450615+0000", 2023-07-02T14:06:24.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0569999999999993e-06 2023-07-02T14:06:24.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:24.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450623+0000", 2023-07-02T14:06:24.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.63968709199999996 2023-07-02T14:06:24.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090310+0000", 2023-07-02T14:06:24.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020157399999999999 2023-07-02T14:06:24.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:24.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090512+0000", 2023-07-02T14:06:24.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027739190000000001 2023-07-02T14:06:24.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:24.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093285+0000", 2023-07-02T14:06:24.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003 2023-07-02T14:06:24.396 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.396 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.396 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.397 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:416 2.5 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:06:24.397 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:24.397 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 27.742603980999998, 2023-07-02T14:06:24.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002, 2023-07-02T14:06:24.397 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:24.397 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:24.397 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:24.397 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:06:24.398 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:06:24.398 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:06:24.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.398 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:24.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:24.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:24.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:24.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:24.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:24.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:24.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:24.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:24.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.34e-06 2023-07-02T14:06:24.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:24.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450599+0000", 2023-07-02T14:06:24.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1549999999999998e-06 2023-07-02T14:06:24.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:24.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450604+0000", 2023-07-02T14:06:24.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:06:24.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:24.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450607+0000", 2023-07-02T14:06:24.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5717999999999999e-05 2023-07-02T14:06:24.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450643+0000", 2023-07-02T14:06:24.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1619999999999999e-06 2023-07-02T14:06:24.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:24.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450648+0000", 2023-07-02T14:06:24.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64292291800000001 2023-07-02T14:06:24.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093571+0000", 2023-07-02T14:06:24.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115628 2023-07-02T14:06:24.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.404 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:24.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093687+0000", 2023-07-02T14:06:24.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002885552 2023-07-02T14:06:24.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:24.405 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.096572+0000", 2023-07-02T14:06:24.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002 2023-07-02T14:06:24.405 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.405 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.405 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.406 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18554 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:24.406 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248917+0000", 2023-07-02T14:06:24.406 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 18.944283165000002, 2023-07-02T14:06:24.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11109714599999999, 2023-07-02T14:06:24.406 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:24.406 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:24.406 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:24.407 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:24.407 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:24.407 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18554 2023-07-02T14:06:24.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.407 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:24.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:24.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248917+0000", 2023-07-02T14:06:24.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:24.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:24.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248917+0000", 2023-07-02T14:06:24.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:24.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:24.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248916+0000", 2023-07-02T14:06:24.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1690000000000003e-06 2023-07-02T14:06:24.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:24.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248923+0000", 2023-07-02T14:06:24.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2769999999999994e-06 2023-07-02T14:06:24.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:24.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248933+0000", 2023-07-02T14:06:24.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1860000000000002e-06 2023-07-02T14:06:24.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:24.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248937+0000", 2023-07-02T14:06:24.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00063144000000000004 2023-07-02T14:06:24.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249568+0000", 2023-07-02T14:06:24.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1486999999999998e-05 2023-07-02T14:06:24.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249610+0000", 2023-07-02T14:06:24.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066288199999999997 2023-07-02T14:06:24.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256239+0000", 2023-07-02T14:06:24.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3636000000000001e-05 2023-07-02T14:06:24.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256272+0000", 2023-07-02T14:06:24.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01221675 2023-07-02T14:06:24.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268489+0000", 2023-07-02T14:06:24.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0283000000000004e-05 2023-07-02T14:06:24.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268569+0000", 2023-07-02T14:06:24.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011087254 2023-07-02T14:06:24.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279657+0000", 2023-07-02T14:06:24.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2872e-05 2023-07-02T14:06:24.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279689+0000", 2023-07-02T14:06:24.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011797277 2023-07-02T14:06:24.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291487+0000", 2023-07-02T14:06:24.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0111e-05 2023-07-02T14:06:24.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291517+0000", 2023-07-02T14:06:24.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079886279999999994 2023-07-02T14:06:24.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299505+0000", 2023-07-02T14:06:24.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2787000000000006e-05 2023-07-02T14:06:24.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299578+0000", 2023-07-02T14:06:24.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071806889999999996 2023-07-02T14:06:24.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306759+0000", 2023-07-02T14:06:24.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1510000000000002e-05 2023-07-02T14:06:24.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306790+0000", 2023-07-02T14:06:24.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056651269999999998 2023-07-02T14:06:24.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312456+0000", 2023-07-02T14:06:24.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8580999999999999e-05 2023-07-02T14:06:24.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312534+0000", 2023-07-02T14:06:24.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047950680000000004 2023-07-02T14:06:24.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317329+0000", 2023-07-02T14:06:24.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4718000000000002e-05 2023-07-02T14:06:24.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317364+0000", 2023-07-02T14:06:24.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050140469999999998 2023-07-02T14:06:24.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322378+0000", 2023-07-02T14:06:24.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1049e-05 2023-07-02T14:06:24.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322409+0000", 2023-07-02T14:06:24.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006616612 2023-07-02T14:06:24.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329026+0000", 2023-07-02T14:06:24.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4612e-05 2023-07-02T14:06:24.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329060+0000", 2023-07-02T14:06:24.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062225020000000004 2023-07-02T14:06:24.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335283+0000", 2023-07-02T14:06:24.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0385e-05 2023-07-02T14:06:24.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335323+0000", 2023-07-02T14:06:24.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009678262 2023-07-02T14:06:24.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345001+0000", 2023-07-02T14:06:24.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0340999999999999e-05 2023-07-02T14:06:24.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345042+0000", 2023-07-02T14:06:24.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045838650000000003 2023-07-02T14:06:24.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349626+0000", 2023-07-02T14:06:24.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010221999999999999 2023-07-02T14:06:24.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349728+0000", 2023-07-02T14:06:24.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029884349999999998 2023-07-02T14:06:24.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.352716+0000", 2023-07-02T14:06:24.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3821e-05 2023-07-02T14:06:24.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:24.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.352750+0000", 2023-07-02T14:06:24.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021565399999999999 2023-07-02T14:06:24.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:24.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.352966+0000", 2023-07-02T14:06:24.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018362960000000001 2023-07-02T14:06:24.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:24.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354802+0000", 2023-07-02T14:06:24.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051013409999999997 2023-07-02T14:06:24.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:24.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.359903+0000", 2023-07-02T14:06:24.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9068999999999999e-05 2023-07-02T14:06:24.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:24.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.359932+0000", 2023-07-02T14:06:24.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2008000000000005e-05 2023-07-02T14:06:24.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:24.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.360014+0000", 2023-07-02T14:06:24.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11109714599999999 2023-07-02T14:06:24.434 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.434 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.434 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.434 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18555 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:24.434 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248941+0000", 2023-07-02T14:06:24.434 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 18.944259567, 2023-07-02T14:06:24.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11809699899999999, 2023-07-02T14:06:24.435 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:24.435 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:24.435 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:24.435 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:24.435 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:24.435 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18555 2023-07-02T14:06:24.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.436 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:24.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:24.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248941+0000", 2023-07-02T14:06:24.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:24.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:24.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248941+0000", 2023-07-02T14:06:24.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:24.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:24.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248940+0000", 2023-07-02T14:06:24.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.7159999999999999e-06 2023-07-02T14:06:24.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:24.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248943+0000", 2023-07-02T14:06:24.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5949999999999999e-06 2023-07-02T14:06:24.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:24.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248947+0000", 2023-07-02T14:06:24.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4480000000000001e-06 2023-07-02T14:06:24.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:24.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248949+0000", 2023-07-02T14:06:24.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00067899499999999995 2023-07-02T14:06:24.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249628+0000", 2023-07-02T14:06:24.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1100000000000003e-05 2023-07-02T14:06:24.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249669+0000", 2023-07-02T14:06:24.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066349690000000001 2023-07-02T14:06:24.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256304+0000", 2023-07-02T14:06:24.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2691999999999998e-05 2023-07-02T14:06:24.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256337+0000", 2023-07-02T14:06:24.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012284679999999999 2023-07-02T14:06:24.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268622+0000", 2023-07-02T14:06:24.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6550000000000001e-05 2023-07-02T14:06:24.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268658+0000", 2023-07-02T14:06:24.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011061109 2023-07-02T14:06:24.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279719+0000", 2023-07-02T14:06:24.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2134000000000001e-05 2023-07-02T14:06:24.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279751+0000", 2023-07-02T14:06:24.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011798329999999999 2023-07-02T14:06:24.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291550+0000", 2023-07-02T14:06:24.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9839999999999999e-05 2023-07-02T14:06:24.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291580+0000", 2023-07-02T14:06:24.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080386550000000005 2023-07-02T14:06:24.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299618+0000", 2023-07-02T14:06:24.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2632999999999997e-05 2023-07-02T14:06:24.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299651+0000", 2023-07-02T14:06:24.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071769859999999998 2023-07-02T14:06:24.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306828+0000", 2023-07-02T14:06:24.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1123000000000004e-05 2023-07-02T14:06:24.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306889+0000", 2023-07-02T14:06:24.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056835710000000001 2023-07-02T14:06:24.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312573+0000", 2023-07-02T14:06:24.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0115000000000002e-05 2023-07-02T14:06:24.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312643+0000", 2023-07-02T14:06:24.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004754538 2023-07-02T14:06:24.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317397+0000", 2023-07-02T14:06:24.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4385000000000002e-05 2023-07-02T14:06:24.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317452+0000", 2023-07-02T14:06:24.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049988350000000001 2023-07-02T14:06:24.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322450+0000", 2023-07-02T14:06:24.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4749e-05 2023-07-02T14:06:24.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322515+0000", 2023-07-02T14:06:24.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065710810000000003 2023-07-02T14:06:24.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329086+0000", 2023-07-02T14:06:24.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0982000000000003e-05 2023-07-02T14:06:24.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329117+0000", 2023-07-02T14:06:24.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062301259999999999 2023-07-02T14:06:24.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335347+0000", 2023-07-02T14:06:24.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2830999999999998e-05 2023-07-02T14:06:24.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335380+0000", 2023-07-02T14:06:24.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097041909999999992 2023-07-02T14:06:24.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345084+0000", 2023-07-02T14:06:24.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2044000000000003e-05 2023-07-02T14:06:24.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345156+0000", 2023-07-02T14:06:24.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004610711 2023-07-02T14:06:24.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349767+0000", 2023-07-02T14:06:24.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8640000000000002e-05 2023-07-02T14:06:24.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349816+0000", 2023-07-02T14:06:24.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043231160000000001 2023-07-02T14:06:24.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354139+0000", 2023-07-02T14:06:24.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8073e-05 2023-07-02T14:06:24.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354177+0000", 2023-07-02T14:06:24.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005870681 2023-07-02T14:06:24.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.360048+0000", 2023-07-02T14:06:24.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9535e-05 2023-07-02T14:06:24.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:24.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.360087+0000", 2023-07-02T14:06:24.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021778100000000001 2023-07-02T14:06:24.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:24.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.360305+0000", 2023-07-02T14:06:24.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001720708 2023-07-02T14:06:24.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:24.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.362026+0000", 2023-07-02T14:06:24.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049001009999999996 2023-07-02T14:06:24.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:24.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.366926+0000", 2023-07-02T14:06:24.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3682e-05 2023-07-02T14:06:24.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:24.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.366959+0000", 2023-07-02T14:06:24.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8419999999999998e-05 2023-07-02T14:06:24.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:24.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.367038+0000", 2023-07-02T14:06:24.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.11809699899999999 2023-07-02T14:06:24.463 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.464 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.464 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.464 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18556 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:24.464 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248957+0000", 2023-07-02T14:06:24.464 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 18.944243598, 2023-07-02T14:06:24.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12347108800000001, 2023-07-02T14:06:24.465 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:24.465 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:24.465 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:24.465 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:24.465 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:24.465 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18556 2023-07-02T14:06:24.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.465 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:24.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:24.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248957+0000", 2023-07-02T14:06:24.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:24.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:24.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248957+0000", 2023-07-02T14:06:24.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:24.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:24.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248957+0000", 2023-07-02T14:06:24.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4119999999999999e-06 2023-07-02T14:06:24.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:24.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248959+0000", 2023-07-02T14:06:24.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5669999999999999e-06 2023-07-02T14:06:24.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:24.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248962+0000", 2023-07-02T14:06:24.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3989999999999998e-06 2023-07-02T14:06:24.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:24.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248965+0000", 2023-07-02T14:06:24.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00072489500000000003 2023-07-02T14:06:24.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249690+0000", 2023-07-02T14:06:24.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7876999999999997e-05 2023-07-02T14:06:24.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249728+0000", 2023-07-02T14:06:24.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066341250000000003 2023-07-02T14:06:24.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256362+0000", 2023-07-02T14:06:24.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2715000000000001e-05 2023-07-02T14:06:24.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256395+0000", 2023-07-02T14:06:24.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012305508 2023-07-02T14:06:24.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268700+0000", 2023-07-02T14:06:24.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7537e-05 2023-07-02T14:06:24.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268778+0000", 2023-07-02T14:06:24.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011004300999999999 2023-07-02T14:06:24.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279782+0000", 2023-07-02T14:06:24.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1996000000000003e-05 2023-07-02T14:06:24.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279814+0000", 2023-07-02T14:06:24.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011789104 2023-07-02T14:06:24.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291603+0000", 2023-07-02T14:06:24.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9825999999999999e-05 2023-07-02T14:06:24.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291633+0000", 2023-07-02T14:06:24.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080637 2023-07-02T14:06:24.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299696+0000", 2023-07-02T14:06:24.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8535999999999996e-05 2023-07-02T14:06:24.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299765+0000", 2023-07-02T14:06:24.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071538169999999998 2023-07-02T14:06:24.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306919+0000", 2023-07-02T14:06:24.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0337000000000001e-05 2023-07-02T14:06:24.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306949+0000", 2023-07-02T14:06:24.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057379919999999999 2023-07-02T14:06:24.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312687+0000", 2023-07-02T14:06:24.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8463000000000001e-05 2023-07-02T14:06:24.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312726+0000", 2023-07-02T14:06:24.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004766251 2023-07-02T14:06:24.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317492+0000", 2023-07-02T14:06:24.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1788000000000001e-05 2023-07-02T14:06:24.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317524+0000", 2023-07-02T14:06:24.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050322910000000004 2023-07-02T14:06:24.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322556+0000", 2023-07-02T14:06:24.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9141999999999999e-05 2023-07-02T14:06:24.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322595+0000", 2023-07-02T14:06:24.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006553354 2023-07-02T14:06:24.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329148+0000", 2023-07-02T14:06:24.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2458e-05 2023-07-02T14:06:24.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329181+0000", 2023-07-02T14:06:24.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006222483 2023-07-02T14:06:24.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335403+0000", 2023-07-02T14:06:24.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0419000000000001e-05 2023-07-02T14:06:24.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335434+0000", 2023-07-02T14:06:24.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097556550000000002 2023-07-02T14:06:24.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345189+0000", 2023-07-02T14:06:24.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2968e-05 2023-07-02T14:06:24.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345222+0000", 2023-07-02T14:06:24.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046402270000000002 2023-07-02T14:06:24.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349863+0000", 2023-07-02T14:06:24.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7782999999999997e-05 2023-07-02T14:06:24.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349940+0000", 2023-07-02T14:06:24.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042690699999999998 2023-07-02T14:06:24.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354210+0000", 2023-07-02T14:06:24.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5648000000000001e-05 2023-07-02T14:06:24.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354255+0000", 2023-07-02T14:06:24.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071341269999999997 2023-07-02T14:06:24.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361389+0000", 2023-07-02T14:06:24.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9140000000000001e-05 2023-07-02T14:06:24.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361428+0000", 2023-07-02T14:06:24.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056418470000000002 2023-07-02T14:06:24.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.367070+0000", 2023-07-02T14:06:24.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1863000000000002e-05 2023-07-02T14:06:24.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:24.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.367112+0000", 2023-07-02T14:06:24.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00027357599999999998 2023-07-02T14:06:24.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:24.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.367386+0000", 2023-07-02T14:06:24.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0020277680000000001 2023-07-02T14:06:24.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:24.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369413+0000", 2023-07-02T14:06:24.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029075699999999999 2023-07-02T14:06:24.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:24.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372321+0000", 2023-07-02T14:06:24.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1955999999999997e-05 2023-07-02T14:06:24.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:24.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372353+0000", 2023-07-02T14:06:24.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4938999999999995e-05 2023-07-02T14:06:24.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:24.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372428+0000", 2023-07-02T14:06:24.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12347108800000001 2023-07-02T14:06:24.494 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.495 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.495 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.495 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18557 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:24.495 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:24.495 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 18.944231601999999, 2023-07-02T14:06:24.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13120991400000001, 2023-07-02T14:06:24.496 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:24.496 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:24.496 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:24.496 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:24.496 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:24.496 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18557 2023-07-02T14:06:24.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.496 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:24.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:24.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:24.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:24.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:24.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:24.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:06:24.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:24.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:24.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8870000000000001e-06 2023-07-02T14:06:24.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:24.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248971+0000", 2023-07-02T14:06:24.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9009999999999996e-06 2023-07-02T14:06:24.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:24.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248974+0000", 2023-07-02T14:06:24.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9809999999999997e-06 2023-07-02T14:06:24.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:24.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248978+0000", 2023-07-02T14:06:24.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00076989199999999999 2023-07-02T14:06:24.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249748+0000", 2023-07-02T14:06:24.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8245999999999999e-05 2023-07-02T14:06:24.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249787+0000", 2023-07-02T14:06:24.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066332969999999998 2023-07-02T14:06:24.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256420+0000", 2023-07-02T14:06:24.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2566e-05 2023-07-02T14:06:24.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256452+0000", 2023-07-02T14:06:24.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012364714000000001 2023-07-02T14:06:24.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268817+0000", 2023-07-02T14:06:24.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5113000000000001e-05 2023-07-02T14:06:24.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268852+0000", 2023-07-02T14:06:24.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010993899999999999 2023-07-02T14:06:24.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279846+0000", 2023-07-02T14:06:24.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2144999999999998e-05 2023-07-02T14:06:24.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279878+0000", 2023-07-02T14:06:24.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011778958000000001 2023-07-02T14:06:24.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291657+0000", 2023-07-02T14:06:24.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9592000000000001e-05 2023-07-02T14:06:24.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291687+0000", 2023-07-02T14:06:24.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081107220000000008 2023-07-02T14:06:24.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299798+0000", 2023-07-02T14:06:24.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0290999999999998e-05 2023-07-02T14:06:24.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299828+0000", 2023-07-02T14:06:24.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071584489999999999 2023-07-02T14:06:24.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306986+0000", 2023-07-02T14:06:24.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0755999999999999e-05 2023-07-02T14:06:24.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307047+0000", 2023-07-02T14:06:24.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057110260000000001 2023-07-02T14:06:24.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312758+0000", 2023-07-02T14:06:24.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5145999999999999e-05 2023-07-02T14:06:24.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312793+0000", 2023-07-02T14:06:24.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004758361 2023-07-02T14:06:24.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317552+0000", 2023-07-02T14:06:24.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9190000000000002e-05 2023-07-02T14:06:24.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317601+0000", 2023-07-02T14:06:24.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050329420000000003 2023-07-02T14:06:24.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322634+0000", 2023-07-02T14:06:24.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5737999999999998e-05 2023-07-02T14:06:24.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322719+0000", 2023-07-02T14:06:24.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064995679999999998 2023-07-02T14:06:24.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329219+0000", 2023-07-02T14:06:24.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0908000000000003e-05 2023-07-02T14:06:24.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329250+0000", 2023-07-02T14:06:24.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062072029999999997 2023-07-02T14:06:24.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335457+0000", 2023-07-02T14:06:24.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9471999999999998e-05 2023-07-02T14:06:24.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335487+0000", 2023-07-02T14:06:24.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097780239999999997 2023-07-02T14:06:24.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345265+0000", 2023-07-02T14:06:24.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0982999999999994e-05 2023-07-02T14:06:24.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345336+0000", 2023-07-02T14:06:24.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046378799999999996 2023-07-02T14:06:24.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349973+0000", 2023-07-02T14:06:24.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1069999999999999e-05 2023-07-02T14:06:24.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350005+0000", 2023-07-02T14:06:24.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042766870000000004 2023-07-02T14:06:24.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354281+0000", 2023-07-02T14:06:24.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1155000000000002e-05 2023-07-02T14:06:24.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354312+0000", 2023-07-02T14:06:24.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071436820000000002 2023-07-02T14:06:24.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361456+0000", 2023-07-02T14:06:24.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9054000000000002e-05 2023-07-02T14:06:24.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361505+0000", 2023-07-02T14:06:24.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072301320000000002 2023-07-02T14:06:24.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368735+0000", 2023-07-02T14:06:24.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1013999999999994e-05 2023-07-02T14:06:24.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368816+0000", 2023-07-02T14:06:24.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036582490000000001 2023-07-02T14:06:24.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372475+0000", 2023-07-02T14:06:24.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4652000000000004e-05 2023-07-02T14:06:24.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:24.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372539+0000", 2023-07-02T14:06:24.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00031374199999999999 2023-07-02T14:06:24.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:24.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372853+0000", 2023-07-02T14:06:24.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030071299999999998 2023-07-02T14:06:24.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:24.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.375860+0000", 2023-07-02T14:06:24.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041897749999999997 2023-07-02T14:06:24.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:24.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380050+0000", 2023-07-02T14:06:24.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8331000000000003e-05 2023-07-02T14:06:24.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:24.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380088+0000", 2023-07-02T14:06:24.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0626999999999998e-05 2023-07-02T14:06:24.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:24.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380179+0000", 2023-07-02T14:06:24.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13120991400000001 2023-07-02T14:06:24.527 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.527 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.527 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.527 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18558 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:24.527 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:24.528 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 18.944214899999999, 2023-07-02T14:06:24.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13755567899999999, 2023-07-02T14:06:24.528 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:24.528 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:24.528 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:24.528 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:24.528 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:24.529 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18558 2023-07-02T14:06:24.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.529 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:24.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:24.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:24.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:24.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:24.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:24.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:24.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:24.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248985+0000", 2023-07-02T14:06:24.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8630000000000001e-06 2023-07-02T14:06:24.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:24.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248989+0000", 2023-07-02T14:06:24.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0750000000000003e-06 2023-07-02T14:06:24.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:24.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248995+0000", 2023-07-02T14:06:24.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3290000000000004e-06 2023-07-02T14:06:24.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:24.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248999+0000", 2023-07-02T14:06:24.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00080656300000000005 2023-07-02T14:06:24.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249806+0000", 2023-07-02T14:06:24.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2040999999999999e-05 2023-07-02T14:06:24.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249848+0000", 2023-07-02T14:06:24.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066312419999999999 2023-07-02T14:06:24.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256479+0000", 2023-07-02T14:06:24.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3348e-05 2023-07-02T14:06:24.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256513+0000", 2023-07-02T14:06:24.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012391531000000001 2023-07-02T14:06:24.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268904+0000", 2023-07-02T14:06:24.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.347e-05 2023-07-02T14:06:24.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268978+0000", 2023-07-02T14:06:24.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010929772000000001 2023-07-02T14:06:24.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279907+0000", 2023-07-02T14:06:24.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2669999999999997e-05 2023-07-02T14:06:24.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279940+0000", 2023-07-02T14:06:24.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011770332999999999 2023-07-02T14:06:24.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291710+0000", 2023-07-02T14:06:24.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1207000000000001e-05 2023-07-02T14:06:24.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291742+0000", 2023-07-02T14:06:24.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081242199999999997 2023-07-02T14:06:24.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299866+0000", 2023-07-02T14:06:24.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4650000000000002e-05 2023-07-02T14:06:24.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299900+0000", 2023-07-02T14:06:24.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071810270000000004 2023-07-02T14:06:24.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307081+0000", 2023-07-02T14:06:24.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4378000000000002e-05 2023-07-02T14:06:24.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307116+0000", 2023-07-02T14:06:24.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057031909999999998 2023-07-02T14:06:24.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312819+0000", 2023-07-02T14:06:24.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1612999999999997e-05 2023-07-02T14:06:24.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312851+0000", 2023-07-02T14:06:24.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048172470000000002 2023-07-02T14:06:24.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317668+0000", 2023-07-02T14:06:24.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3328e-05 2023-07-02T14:06:24.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317741+0000", 2023-07-02T14:06:24.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050117290000000004 2023-07-02T14:06:24.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322753+0000", 2023-07-02T14:06:24.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9872000000000001e-05 2023-07-02T14:06:24.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322783+0000", 2023-07-02T14:06:24.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064910610000000002 2023-07-02T14:06:24.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329274+0000", 2023-07-02T14:06:24.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0097e-05 2023-07-02T14:06:24.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329304+0000", 2023-07-02T14:06:24.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062163879999999998 2023-07-02T14:06:24.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335520+0000", 2023-07-02T14:06:24.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9513e-05 2023-07-02T14:06:24.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335550+0000", 2023-07-02T14:06:24.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098209969999999997 2023-07-02T14:06:24.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345371+0000", 2023-07-02T14:06:24.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9862999999999999e-05 2023-07-02T14:06:24.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345401+0000", 2023-07-02T14:06:24.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046552030000000001 2023-07-02T14:06:24.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350056+0000", 2023-07-02T14:06:24.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8751999999999993e-05 2023-07-02T14:06:24.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350145+0000", 2023-07-02T14:06:24.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041981450000000003 2023-07-02T14:06:24.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354343+0000", 2023-07-02T14:06:24.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1501000000000002e-05 2023-07-02T14:06:24.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354374+0000", 2023-07-02T14:06:24.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071551740000000003 2023-07-02T14:06:24.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361530+0000", 2023-07-02T14:06:24.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5156e-05 2023-07-02T14:06:24.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361565+0000", 2023-07-02T14:06:24.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072918569999999997 2023-07-02T14:06:24.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368857+0000", 2023-07-02T14:06:24.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5770999999999999e-05 2023-07-02T14:06:24.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368902+0000", 2023-07-02T14:06:24.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057210569999999999 2023-07-02T14:06:24.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374623+0000", 2023-07-02T14:06:24.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3266999999999998e-05 2023-07-02T14:06:24.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374667+0000", 2023-07-02T14:06:24.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055479570000000001 2023-07-02T14:06:24.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380215+0000", 2023-07-02T14:06:24.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0067e-05 2023-07-02T14:06:24.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:24.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380265+0000", 2023-07-02T14:06:24.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00043082299999999999 2023-07-02T14:06:24.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:24.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380696+0000", 2023-07-02T14:06:24.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024853240000000001 2023-07-02T14:06:24.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:24.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.383181+0000", 2023-07-02T14:06:24.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032601280000000002 2023-07-02T14:06:24.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:24.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386441+0000", 2023-07-02T14:06:24.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9748e-05 2023-07-02T14:06:24.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:24.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386471+0000", 2023-07-02T14:06:24.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0531000000000006e-05 2023-07-02T14:06:24.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:24.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386541+0000", 2023-07-02T14:06:24.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13755567899999999 2023-07-02T14:06:24.561 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.561 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.561 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.561 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18559 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:24.561 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:24.562 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 18.944192296000001, 2023-07-02T14:06:24.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.146184591, 2023-07-02T14:06:24.562 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:24.562 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:24.562 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:24.562 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:24.562 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:24.563 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18559 2023-07-02T14:06:24.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.563 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:24.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:24.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:24.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:24.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:24.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:24.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:24.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:24.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:24.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.726e-06 2023-07-02T14:06:24.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:24.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249011+0000", 2023-07-02T14:06:24.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7939999999999997e-06 2023-07-02T14:06:24.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:24.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249018+0000", 2023-07-02T14:06:24.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3470000000000001e-06 2023-07-02T14:06:24.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:24.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249023+0000", 2023-07-02T14:06:24.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00084402900000000005 2023-07-02T14:06:24.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249867+0000", 2023-07-02T14:06:24.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9050000000000001e-05 2023-07-02T14:06:24.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249906+0000", 2023-07-02T14:06:24.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066311039999999996 2023-07-02T14:06:24.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256537+0000", 2023-07-02T14:06:24.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2790000000000003e-05 2023-07-02T14:06:24.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256570+0000", 2023-07-02T14:06:24.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012436129000000001 2023-07-02T14:06:24.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269006+0000", 2023-07-02T14:06:24.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0173999999999999e-05 2023-07-02T14:06:24.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269046+0000", 2023-07-02T14:06:24.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010924698 2023-07-02T14:06:24.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279971+0000", 2023-07-02T14:06:24.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3831000000000002e-05 2023-07-02T14:06:24.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280004+0000", 2023-07-02T14:06:24.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011795912 2023-07-02T14:06:24.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291800+0000", 2023-07-02T14:06:24.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5613999999999997e-05 2023-07-02T14:06:24.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291886+0000", 2023-07-02T14:06:24.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080395999999999992 2023-07-02T14:06:24.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299925+0000", 2023-07-02T14:06:24.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0694999999999997e-05 2023-07-02T14:06:24.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299956+0000", 2023-07-02T14:06:24.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072049449999999999 2023-07-02T14:06:24.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307161+0000", 2023-07-02T14:06:24.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5523000000000001e-05 2023-07-02T14:06:24.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307247+0000", 2023-07-02T14:06:24.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056331929999999999 2023-07-02T14:06:24.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312880+0000", 2023-07-02T14:06:24.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1646000000000002e-05 2023-07-02T14:06:24.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312911+0000", 2023-07-02T14:06:24.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048627540000000004 2023-07-02T14:06:24.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317774+0000", 2023-07-02T14:06:24.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2008000000000003e-05 2023-07-02T14:06:24.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317806+0000", 2023-07-02T14:06:24.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050055689999999996 2023-07-02T14:06:24.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322812+0000", 2023-07-02T14:06:24.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3442999999999997e-05 2023-07-02T14:06:24.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322855+0000", 2023-07-02T14:06:24.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006472813 2023-07-02T14:06:24.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329328+0000", 2023-07-02T14:06:24.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0128e-05 2023-07-02T14:06:24.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329358+0000", 2023-07-02T14:06:24.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062156490000000002 2023-07-02T14:06:24.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335574+0000", 2023-07-02T14:06:24.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2357999999999997e-05 2023-07-02T14:06:24.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335606+0000", 2023-07-02T14:06:24.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098318409999999992 2023-07-02T14:06:24.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345438+0000", 2023-07-02T14:06:24.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1174000000000007e-05 2023-07-02T14:06:24.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345509+0000", 2023-07-02T14:06:24.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004669851 2023-07-02T14:06:24.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350179+0000", 2023-07-02T14:06:24.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1559000000000001e-05 2023-07-02T14:06:24.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350211+0000", 2023-07-02T14:06:24.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041865740000000002 2023-07-02T14:06:24.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354397+0000", 2023-07-02T14:06:24.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9850000000000001e-05 2023-07-02T14:06:24.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354427+0000", 2023-07-02T14:06:24.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071608449999999999 2023-07-02T14:06:24.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361588+0000", 2023-07-02T14:06:24.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8346000000000002e-05 2023-07-02T14:06:24.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361626+0000", 2023-07-02T14:06:24.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073175009999999997 2023-07-02T14:06:24.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368944+0000", 2023-07-02T14:06:24.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2879000000000005e-05 2023-07-02T14:06:24.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369017+0000", 2023-07-02T14:06:24.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056850690000000001 2023-07-02T14:06:24.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374702+0000", 2023-07-02T14:06:24.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7693e-05 2023-07-02T14:06:24.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374739+0000", 2023-07-02T14:06:24.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076925149999999996 2023-07-02T14:06:24.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382432+0000", 2023-07-02T14:06:24.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3259e-05 2023-07-02T14:06:24.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382475+0000", 2023-07-02T14:06:24.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041005110000000003 2023-07-02T14:06:24.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386576+0000", 2023-07-02T14:06:24.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2074999999999999e-05 2023-07-02T14:06:24.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:24.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386618+0000", 2023-07-02T14:06:24.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000233692 2023-07-02T14:06:24.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:24.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386851+0000", 2023-07-02T14:06:24.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024022269999999998 2023-07-02T14:06:24.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:24.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.389254+0000", 2023-07-02T14:06:24.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058506469999999996 2023-07-02T14:06:24.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:24.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395104+0000", 2023-07-02T14:06:24.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8807000000000003e-05 2023-07-02T14:06:24.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:24.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395143+0000", 2023-07-02T14:06:24.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9616000000000001e-05 2023-07-02T14:06:24.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:24.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395193+0000", 2023-07-02T14:06:24.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.146184591 2023-07-02T14:06:24.596 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.596 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.596 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.596 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18560 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:24.596 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:24.597 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 18.944173673000002, 2023-07-02T14:06:24.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.15647551900000001, 2023-07-02T14:06:24.597 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:24.597 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:24.597 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:24.597 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:24.597 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:24.597 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18560 2023-07-02T14:06:24.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.598 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:24.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:24.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:24.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:24.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:24.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:24.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:24.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:24.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249026+0000", 2023-07-02T14:06:24.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3419999999999999e-06 2023-07-02T14:06:24.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:24.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249030+0000", 2023-07-02T14:06:24.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7869999999999998e-06 2023-07-02T14:06:24.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:24.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249035+0000", 2023-07-02T14:06:24.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9639999999999997e-06 2023-07-02T14:06:24.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:24.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249039+0000", 2023-07-02T14:06:24.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00088477599999999999 2023-07-02T14:06:24.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249924+0000", 2023-07-02T14:06:24.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7333999999999999e-05 2023-07-02T14:06:24.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249962+0000", 2023-07-02T14:06:24.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066323409999999999 2023-07-02T14:06:24.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256594+0000", 2023-07-02T14:06:24.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2595999999999998e-05 2023-07-02T14:06:24.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.604 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.604 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256627+0000", 2023-07-02T14:06:24.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012449894 2023-07-02T14:06:24.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.604 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269076+0000", 2023-07-02T14:06:24.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3942999999999997e-05 2023-07-02T14:06:24.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269110+0000", 2023-07-02T14:06:24.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010923884 2023-07-02T14:06:24.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280034+0000", 2023-07-02T14:06:24.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7774000000000001e-05 2023-07-02T14:06:24.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280072+0000", 2023-07-02T14:06:24.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011850027000000001 2023-07-02T14:06:24.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291922+0000", 2023-07-02T14:06:24.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2128000000000001e-05 2023-07-02T14:06:24.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291954+0000", 2023-07-02T14:06:24.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080261150000000003 2023-07-02T14:06:24.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299980+0000", 2023-07-02T14:06:24.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6322999999999998e-05 2023-07-02T14:06:24.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300017+0000", 2023-07-02T14:06:24.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072683039999999997 2023-07-02T14:06:24.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307285+0000", 2023-07-02T14:06:24.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3309000000000002e-05 2023-07-02T14:06:24.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307318+0000", 2023-07-02T14:06:24.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056178649999999997 2023-07-02T14:06:24.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312936+0000", 2023-07-02T14:06:24.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1866000000000003e-05 2023-07-02T14:06:24.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312968+0000", 2023-07-02T14:06:24.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048789970000000004 2023-07-02T14:06:24.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317847+0000", 2023-07-02T14:06:24.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0223999999999996e-05 2023-07-02T14:06:24.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317927+0000", 2023-07-02T14:06:24.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049543649999999996 2023-07-02T14:06:24.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322882+0000", 2023-07-02T14:06:24.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1399000000000002e-05 2023-07-02T14:06:24.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322913+0000", 2023-07-02T14:06:24.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006534214 2023-07-02T14:06:24.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.615 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329447+0000", 2023-07-02T14:06:24.615 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9649999999999998e-05 2023-07-02T14:06:24.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.616 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329527+0000", 2023-07-02T14:06:24.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061026839999999997 2023-07-02T14:06:24.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.616 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.616 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.616 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335629+0000", 2023-07-02T14:06:24.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9611000000000002e-05 2023-07-02T14:06:24.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335659+0000", 2023-07-02T14:06:24.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098851549999999996 2023-07-02T14:06:24.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345544+0000", 2023-07-02T14:06:24.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0587999999999999e-05 2023-07-02T14:06:24.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345575+0000", 2023-07-02T14:06:24.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046790859999999998 2023-07-02T14:06:24.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350254+0000", 2023-07-02T14:06:24.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0284000000000002e-05 2023-07-02T14:06:24.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350304+0000", 2023-07-02T14:06:24.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041560540000000002 2023-07-02T14:06:24.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354460+0000", 2023-07-02T14:06:24.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2427000000000004e-05 2023-07-02T14:06:24.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354523+0000", 2023-07-02T14:06:24.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071268770000000002 2023-07-02T14:06:24.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361650+0000", 2023-07-02T14:06:24.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0026999999999999e-05 2023-07-02T14:06:24.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361680+0000", 2023-07-02T14:06:24.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073733080000000003 2023-07-02T14:06:24.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369053+0000", 2023-07-02T14:06:24.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7635999999999997e-05 2023-07-02T14:06:24.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369091+0000", 2023-07-02T14:06:24.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056908059999999996 2023-07-02T14:06:24.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374781+0000", 2023-07-02T14:06:24.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4656000000000002e-05 2023-07-02T14:06:24.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374816+0000", 2023-07-02T14:06:24.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077037809999999998 2023-07-02T14:06:24.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.625 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382520+0000", 2023-07-02T14:06:24.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7858e-05 2023-07-02T14:06:24.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.626 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382588+0000", 2023-07-02T14:06:24.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057884019999999998 2023-07-02T14:06:24.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388376+0000", 2023-07-02T14:06:24.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4831000000000001e-05 2023-07-02T14:06:24.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388461+0000", 2023-07-02T14:06:24.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067708480000000003 2023-07-02T14:06:24.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395232+0000", 2023-07-02T14:06:24.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7948000000000002e-05 2023-07-02T14:06:24.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:24.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395290+0000", 2023-07-02T14:06:24.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00027672599999999998 2023-07-02T14:06:24.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:24.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395566+0000", 2023-07-02T14:06:24.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025705569999999998 2023-07-02T14:06:24.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:24.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.398137+0000", 2023-07-02T14:06:24.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072257099999999998 2023-07-02T14:06:24.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:24.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405363+0000", 2023-07-02T14:06:24.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3887999999999998e-05 2023-07-02T14:06:24.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:24.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405397+0000", 2023-07-02T14:06:24.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000105769 2023-07-02T14:06:24.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:24.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405502+0000", 2023-07-02T14:06:24.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.15647551900000001 2023-07-02T14:06:24.632 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.632 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.632 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.633 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18561 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:24.633 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:24.633 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 18.944153309000001, 2023-07-02T14:06:24.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744, 2023-07-02T14:06:24.633 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:24.633 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:24.634 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:24.634 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:24.634 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:24.634 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18561 2023-07-02T14:06:24.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.634 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:24.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:24.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:24.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:24.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:24.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:24.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:24.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:24.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:24.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.563e-06 2023-07-02T14:06:24.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:24.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249050+0000", 2023-07-02T14:06:24.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4500000000000001e-06 2023-07-02T14:06:24.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:24.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249057+0000", 2023-07-02T14:06:24.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1860000000000002e-06 2023-07-02T14:06:24.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:24.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249061+0000", 2023-07-02T14:06:24.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00091153399999999998 2023-07-02T14:06:24.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249972+0000", 2023-07-02T14:06:24.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7156999999999997e-05 2023-07-02T14:06:24.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.250010+0000", 2023-07-02T14:06:24.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006641816 2023-07-02T14:06:24.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256651+0000", 2023-07-02T14:06:24.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1968999999999998e-05 2023-07-02T14:06:24.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256683+0000", 2023-07-02T14:06:24.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012463952 2023-07-02T14:06:24.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269147+0000", 2023-07-02T14:06:24.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2855000000000006e-05 2023-07-02T14:06:24.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269220+0000", 2023-07-02T14:06:24.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010870418999999999 2023-07-02T14:06:24.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280091+0000", 2023-07-02T14:06:24.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1921999999999997e-05 2023-07-02T14:06:24.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280123+0000", 2023-07-02T14:06:24.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011861682 2023-07-02T14:06:24.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291984+0000", 2023-07-02T14:06:24.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5259000000000006e-05 2023-07-02T14:06:24.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.292059+0000", 2023-07-02T14:06:24.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079709359999999996 2023-07-02T14:06:24.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300030+0000", 2023-07-02T14:06:24.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0729999999999998e-05 2023-07-02T14:06:24.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300071+0000", 2023-07-02T14:06:24.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072785690000000004 2023-07-02T14:06:24.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307350+0000", 2023-07-02T14:06:24.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1948999999999999e-05 2023-07-02T14:06:24.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307412+0000", 2023-07-02T14:06:24.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055785779999999998 2023-07-02T14:06:24.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312990+0000", 2023-07-02T14:06:24.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0957000000000002e-05 2023-07-02T14:06:24.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.313021+0000", 2023-07-02T14:06:24.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049246400000000001 2023-07-02T14:06:24.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317946+0000", 2023-07-02T14:06:24.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9346e-05 2023-07-02T14:06:24.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317975+0000", 2023-07-02T14:06:24.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049649469999999999 2023-07-02T14:06:24.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322940+0000", 2023-07-02T14:06:24.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9017000000000002e-05 2023-07-02T14:06:24.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322989+0000", 2023-07-02T14:06:24.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065907029999999998 2023-07-02T14:06:24.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329580+0000", 2023-07-02T14:06:24.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5879999999999999e-05 2023-07-02T14:06:24.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329656+0000", 2023-07-02T14:06:24.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060166960000000002 2023-07-02T14:06:24.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335672+0000", 2023-07-02T14:06:24.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8963e-05 2023-07-02T14:06:24.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335701+0000", 2023-07-02T14:06:24.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098989560000000004 2023-07-02T14:06:24.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345600+0000", 2023-07-02T14:06:24.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5395999999999995e-05 2023-07-02T14:06:24.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345666+0000", 2023-07-02T14:06:24.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046589530000000004 2023-07-02T14:06:24.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350325+0000", 2023-07-02T14:06:24.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2459000000000002e-05 2023-07-02T14:06:24.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350357+0000", 2023-07-02T14:06:24.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042054290000000001 2023-07-02T14:06:24.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354563+0000", 2023-07-02T14:06:24.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5612999999999999e-05 2023-07-02T14:06:24.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354598+0000", 2023-07-02T14:06:24.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070967449999999998 2023-07-02T14:06:24.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361695+0000", 2023-07-02T14:06:24.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9314000000000001e-05 2023-07-02T14:06:24.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361724+0000", 2023-07-02T14:06:24.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007391028 2023-07-02T14:06:24.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369115+0000", 2023-07-02T14:06:24.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9163000000000002e-05 2023-07-02T14:06:24.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369184+0000", 2023-07-02T14:06:24.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056532500000000003 2023-07-02T14:06:24.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374838+0000", 2023-07-02T14:06:24.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5297999999999999e-05 2023-07-02T14:06:24.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374883+0000", 2023-07-02T14:06:24.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077292189999999998 2023-07-02T14:06:24.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382612+0000", 2023-07-02T14:06:24.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1102000000000001e-05 2023-07-02T14:06:24.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382653+0000", 2023-07-02T14:06:24.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058515989999999999 2023-07-02T14:06:24.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388505+0000", 2023-07-02T14:06:24.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9196000000000006e-05 2023-07-02T14:06:24.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388574+0000", 2023-07-02T14:06:24.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084245339999999991 2023-07-02T14:06:24.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.396999+0000", 2023-07-02T14:06:24.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0596999999999998e-05 2023-07-02T14:06:24.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.397039+0000", 2023-07-02T14:06:24.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084911339999999991 2023-07-02T14:06:24.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405530+0000", 2023-07-02T14:06:24.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7555000000000001e-05 2023-07-02T14:06:24.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:24.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405578+0000", 2023-07-02T14:06:24.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000229607 2023-07-02T14:06:24.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:24.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405808+0000", 2023-07-02T14:06:24.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026919940000000001 2023-07-02T14:06:24.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:24.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.408500+0000", 2023-07-02T14:06:24.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053065059999999999 2023-07-02T14:06:24.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:24.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413806+0000", 2023-07-02T14:06:24.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0490000000000001e-05 2023-07-02T14:06:24.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:24.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413837+0000", 2023-07-02T14:06:24.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000100386 2023-07-02T14:06:24.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:24.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413937+0000", 2023-07-02T14:06:24.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744 2023-07-02T14:06:24.670 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.670 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.670 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.671 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18879 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e177)", 2023-07-02T14:06:24.671 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:06.898926+0000", 2023-07-02T14:06:24.671 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.294274268999999, 2023-07-02T14:06:24.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.108448455, 2023-07-02T14:06:24.671 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:24.671 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:24.671 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:24.672 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:24.672 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:24.672 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18879 2023-07-02T14:06:24.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.672 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:24.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:24.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.898926+0000", 2023-07-02T14:06:24.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:24.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:24.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.898926+0000", 2023-07-02T14:06:24.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:24.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:24.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.898926+0000", 2023-07-02T14:06:24.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.45e-06 2023-07-02T14:06:24.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:24.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.898929+0000", 2023-07-02T14:06:24.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4110000000000004e-06 2023-07-02T14:06:24.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:24.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.898934+0000", 2023-07-02T14:06:24.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.168e-06 2023-07-02T14:06:24.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:24.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.898938+0000", 2023-07-02T14:06:24.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015109999999999999 2023-07-02T14:06:24.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.899089+0000", 2023-07-02T14:06:24.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3013000000000003e-05 2023-07-02T14:06:24.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.899132+0000", 2023-07-02T14:06:24.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0033983960000000001 2023-07-02T14:06:24.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.902530+0000", 2023-07-02T14:06:24.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2097000000000001e-05 2023-07-02T14:06:24.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.902562+0000", 2023-07-02T14:06:24.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013082125 2023-07-02T14:06:24.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.915644+0000", 2023-07-02T14:06:24.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1718999999999998e-05 2023-07-02T14:06:24.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.915686+0000", 2023-07-02T14:06:24.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068310189999999998 2023-07-02T14:06:24.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.922517+0000", 2023-07-02T14:06:24.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1585999999999999e-05 2023-07-02T14:06:24.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.922549+0000", 2023-07-02T14:06:24.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014397978000000001 2023-07-02T14:06:24.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.936947+0000", 2023-07-02T14:06:24.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2971000000000002e-05 2023-07-02T14:06:24.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.937030+0000", 2023-07-02T14:06:24.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070590799999999997 2023-07-02T14:06:24.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.944089+0000", 2023-07-02T14:06:24.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1683000000000001e-05 2023-07-02T14:06:24.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.944120+0000", 2023-07-02T14:06:24.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089819119999999999 2023-07-02T14:06:24.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.953102+0000", 2023-07-02T14:06:24.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7537999999999999e-05 2023-07-02T14:06:24.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.953140+0000", 2023-07-02T14:06:24.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016871374000000001 2023-07-02T14:06:24.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.970011+0000", 2023-07-02T14:06:24.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0485000000000002e-05 2023-07-02T14:06:24.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.970062+0000", 2023-07-02T14:06:24.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013052505000000001 2023-07-02T14:06:24.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.983114+0000", 2023-07-02T14:06:24.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8034999999999997e-05 2023-07-02T14:06:24.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:24.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.983162+0000", 2023-07-02T14:06:24.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013268611 2023-07-02T14:06:24.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.996431+0000", 2023-07-02T14:06:24.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0791000000000001e-05 2023-07-02T14:06:24.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:24.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.996492+0000", 2023-07-02T14:06:24.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00036019099999999999 2023-07-02T14:06:24.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:24.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:06.996852+0000", 2023-07-02T14:06:24.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004127917 2023-07-02T14:06:24.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:24.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:07.000980+0000", 2023-07-02T14:06:24.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061381760000000004 2023-07-02T14:06:24.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:24.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:07.007118+0000", 2023-07-02T14:06:24.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1156999999999997e-05 2023-07-02T14:06:24.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:24.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:07.007189+0000", 2023-07-02T14:06:24.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000185639 2023-07-02T14:06:24.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:24.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:07.007375+0000", 2023-07-02T14:06:24.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.108448455 2023-07-02T14:06:24.692 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.692 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.692 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.693 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20768 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch ping cookie 140544751726928 gen 52] snapc 0=[] ondisk+write+known_if_redirected e183)", 2023-07-02T14:06:24.693 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:24.693 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.9707860410000002, 2023-07-02T14:06:24.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3751256970000001, 2023-07-02T14:06:24.693 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:24.693 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:24.693 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:24.694 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:24.694 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:24.694 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20768 2023-07-02T14:06:24.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.694 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:24.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:24.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:24.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:24.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:24.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:24.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:24.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:24.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:24.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7829999999999998e-06 2023-07-02T14:06:24.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:24.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222419+0000", 2023-07-02T14:06:24.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1380000000000004e-06 2023-07-02T14:06:24.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:24.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222425+0000", 2023-07-02T14:06:24.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2519999999999996e-06 2023-07-02T14:06:24.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:24.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222431+0000", 2023-07-02T14:06:24.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2157999999999994e-05 2023-07-02T14:06:24.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222523+0000", 2023-07-02T14:06:24.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5147000000000001e-05 2023-07-02T14:06:24.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:24.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222538+0000", 2023-07-02T14:06:24.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30105419700000002 2023-07-02T14:06:24.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523592+0000", 2023-07-02T14:06:24.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2316e-05 2023-07-02T14:06:24.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:24.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523605+0000", 2023-07-02T14:06:24.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.073658953 2023-07-02T14:06:24.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597264+0000", 2023-07-02T14:06:24.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000213415 2023-07-02T14:06:24.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:24.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597477+0000", 2023-07-02T14:06:24.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3034999999999995e-05 2023-07-02T14:06:24.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:24.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597540+0000", 2023-07-02T14:06:24.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3751256970000001 2023-07-02T14:06:24.702 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.703 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.703 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.703 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20769 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 53] snapc 0=[] ondisk+write+known_if_redirected e184)", 2023-07-02T14:06:24.703 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:24.703 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.9704235859999999, 2023-07-02T14:06:24.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.380884139, 2023-07-02T14:06:24.704 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:24.704 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:24.704 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:24.704 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:24.704 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:24.704 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20769 2023-07-02T14:06:24.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.705 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:24.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:24.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:24.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:24.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:24.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:24.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:24.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:24.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222776+0000", 2023-07-02T14:06:24.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9710000000000003e-06 2023-07-02T14:06:24.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:24.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222781+0000", 2023-07-02T14:06:24.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9899999999999992e-06 2023-07-02T14:06:24.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:24.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222791+0000", 2023-07-02T14:06:24.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4240000000000003e-06 2023-07-02T14:06:24.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:24.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222795+0000", 2023-07-02T14:06:24.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2366000000000001e-05 2023-07-02T14:06:24.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222828+0000", 2023-07-02T14:06:24.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4790000000000001e-06 2023-07-02T14:06:24.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:24.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222834+0000", 2023-07-02T14:06:24.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30084589299999998 2023-07-02T14:06:24.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523680+0000", 2023-07-02T14:06:24.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.87e-06 2023-07-02T14:06:24.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:24.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523685+0000", 2023-07-02T14:06:24.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.074111042 2023-07-02T14:06:24.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597796+0000", 2023-07-02T14:06:24.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0411000000000001e-05 2023-07-02T14:06:24.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:24.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597856+0000", 2023-07-02T14:06:24.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058045340000000001 2023-07-02T14:06:24.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:24.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.603661+0000", 2023-07-02T14:06:24.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.380884139 2023-07-02T14:06:24.713 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.713 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.713 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.714 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20770 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 54] snapc 0=[] ondisk+write+known_if_redirected e185)", 2023-07-02T14:06:24.714 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:24.714 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 4.6692064760000003, 2023-07-02T14:06:24.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0799819349999999, 2023-07-02T14:06:24.714 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:24.714 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:24.714 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:24.714 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:24.715 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:24.715 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20770 2023-07-02T14:06:24.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.715 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:24.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:24.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:24.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:24.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:24.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:24.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:24.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:24.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523992+0000", 2023-07-02T14:06:24.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6720000000000004e-06 2023-07-02T14:06:24.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:24.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524000+0000", 2023-07-02T14:06:24.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3711e-05 2023-07-02T14:06:24.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:24.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524014+0000", 2023-07-02T14:06:24.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7849999999999992e-06 2023-07-02T14:06:24.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:24.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524022+0000", 2023-07-02T14:06:24.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2366000000000001e-05 2023-07-02T14:06:24.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524054+0000", 2023-07-02T14:06:24.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8169999999999998e-06 2023-07-02T14:06:24.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:24.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524059+0000", 2023-07-02T14:06:24.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0748338850000001 2023-07-02T14:06:24.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:24.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.598893+0000", 2023-07-02T14:06:24.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116968 2023-07-02T14:06:24.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:24.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.599010+0000", 2023-07-02T14:06:24.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049663370000000004 2023-07-02T14:06:24.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:24.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:24.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:24.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.603976+0000", 2023-07-02T14:06:24.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0799819349999999 2023-07-02T14:06:24.722 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.722 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.722 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.722 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:24.723 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:24.723 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:24.724 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:06:24.725 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:06:24.725 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.0 to 1 (10000) 2023-07-02T14:06:24.726 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-07-02T14:06:24.726 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:24.723+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:24.825 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:24.824+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:24.906 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:24.907 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:24.918 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:06:24.925 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:24.924+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:25.025 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:25.024+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:25.057 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:25.058 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:25.058 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:06:25.058 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:06:25.058 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:25.069 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:06:25.153 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:25.152+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:25.227 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:25.226+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:25.260 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:25.261 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:06:25.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:06:25.261 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:25.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.261 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18643 2.5 e176/170 2:a9ace6f8:::rbd_data.125de1449605.0000000000000187:head v 176'2314, mlcod=173'2311)", 2023-07-02T14:06:25.261 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:06:25.261 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.832047575000001, 2023-07-02T14:06:25.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015027297, 2023-07-02T14:06:25.262 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.262 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:25.262 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:06:25.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:06:25.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:25.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426547+0000", 2023-07-02T14:06:25.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056864430000000002 2023-07-02T14:06:25.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432234+0000", 2023-07-02T14:06:25.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.415e-05 2023-07-02T14:06:25.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432248+0000", 2023-07-02T14:06:25.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7490000000000001e-06 2023-07-02T14:06:25.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432255+0000", 2023-07-02T14:06:25.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3915000000000002e-05 2023-07-02T14:06:25.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432299+0000", 2023-07-02T14:06:25.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1214000000000001e-05 2023-07-02T14:06:25.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432320+0000", 2023-07-02T14:06:25.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0092114950000000001 2023-07-02T14:06:25.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:25.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.441531+0000", 2023-07-02T14:06:25.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4422999999999998e-05 2023-07-02T14:06:25.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.441576+0000", 2023-07-02T14:06:25.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015027297 2023-07-02T14:06:25.268 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.269 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.269 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.269 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18644 2.2 e176/171 2:44e6ed94:::rbd_data.125de1449605.0000000000000185:head v 176'1626, mlcod=176'1623)", 2023-07-02T14:06:25.269 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.428712+0000", 2023-07-02T14:06:25.269 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.829883834, 2023-07-02T14:06:25.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012273601, 2023-07-02T14:06:25.270 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.270 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:25.270 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.428712+0000", 2023-07-02T14:06:25.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.428712+0000", 2023-07-02T14:06:25.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:25.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.428711+0000", 2023-07-02T14:06:25.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017963639999999999 2023-07-02T14:06:25.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.430507+0000", 2023-07-02T14:06:25.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0897e-05 2023-07-02T14:06:25.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.430528+0000", 2023-07-02T14:06:25.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7940000000000008e-06 2023-07-02T14:06:25.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.430537+0000", 2023-07-02T14:06:25.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00034051099999999999 2023-07-02T14:06:25.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.430878+0000", 2023-07-02T14:06:25.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5058000000000001e-05 2023-07-02T14:06:25.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.430893+0000", 2023-07-02T14:06:25.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010060174 2023-07-02T14:06:25.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:25.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.440953+0000", 2023-07-02T14:06:25.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3064000000000001e-05 2023-07-02T14:06:25.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.440986+0000", 2023-07-02T14:06:25.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012273601 2023-07-02T14:06:25.276 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.276 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.277 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.277 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18680 2.2 e176/171 2:5fd2923a:::rbd_data.125de1449605.000000000000018c:head v 176'1639, mlcod=176'1637)", 2023-07-02T14:06:25.277 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.489711+0000", 2023-07-02T14:06:25.277 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.768885133000001, 2023-07-02T14:06:25.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014407402, 2023-07-02T14:06:25.277 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.278 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:25.278 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.489711+0000", 2023-07-02T14:06:25.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.489711+0000", 2023-07-02T14:06:25.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:25.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.489709+0000", 2023-07-02T14:06:25.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001991792 2023-07-02T14:06:25.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491701+0000", 2023-07-02T14:06:25.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7323000000000002e-05 2023-07-02T14:06:25.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491738+0000", 2023-07-02T14:06:25.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2483999999999999e-05 2023-07-02T14:06:25.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491751+0000", 2023-07-02T14:06:25.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7896000000000003e-05 2023-07-02T14:06:25.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491809+0000", 2023-07-02T14:06:25.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0185000000000001e-05 2023-07-02T14:06:25.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491829+0000", 2023-07-02T14:06:25.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012250901999999999 2023-07-02T14:06:25.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:25.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.504080+0000", 2023-07-02T14:06:25.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:06:25.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.504118+0000", 2023-07-02T14:06:25.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014407402 2023-07-02T14:06:25.284 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.284 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.284 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.285 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18681 2.2 e176/171 2:5ecc17e8:::rbd_data.125de1449605.000000000000018d:head v 176'1640, mlcod=176'1637)", 2023-07-02T14:06:25.285 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.497048+0000", 2023-07-02T14:06:25.285 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.761547993000001, 2023-07-02T14:06:25.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012067603999999999, 2023-07-02T14:06:25.285 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.285 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:25.285 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.497048+0000", 2023-07-02T14:06:25.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.497048+0000", 2023-07-02T14:06:25.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:25.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.497047+0000", 2023-07-02T14:06:25.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021639139999999999 2023-07-02T14:06:25.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.499211+0000", 2023-07-02T14:06:25.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7983e-05 2023-07-02T14:06:25.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.499239+0000", 2023-07-02T14:06:25.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6110999999999999e-05 2023-07-02T14:06:25.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.499255+0000", 2023-07-02T14:06:25.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0260999999999998e-05 2023-07-02T14:06:25.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.499315+0000", 2023-07-02T14:06:25.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0349e-05 2023-07-02T14:06:25.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.499345+0000", 2023-07-02T14:06:25.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097133659999999993 2023-07-02T14:06:25.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:25.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.509059+0000", 2023-07-02T14:06:25.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7092999999999997e-05 2023-07-02T14:06:25.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.509116+0000", 2023-07-02T14:06:25.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012067603999999999 2023-07-02T14:06:25.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.292 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.292 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18690 2.2 e176/171 2:416a1700:::rbd_data.125de1449605.000000000000018f:head v 176'1644, mlcod=176'1641)", 2023-07-02T14:06:25.293 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.523233+0000", 2023-07-02T14:06:25.293 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.735363002, 2023-07-02T14:06:25.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01191801, 2023-07-02T14:06:25.293 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.293 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:25.293 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.523233+0000", 2023-07-02T14:06:25.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.523233+0000", 2023-07-02T14:06:25.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:25.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.523231+0000", 2023-07-02T14:06:25.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014368580000000001 2023-07-02T14:06:25.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.524668+0000", 2023-07-02T14:06:25.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.929e-05 2023-07-02T14:06:25.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.524688+0000", 2023-07-02T14:06:25.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9709999999999999e-06 2023-07-02T14:06:25.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.524698+0000", 2023-07-02T14:06:25.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112815 2023-07-02T14:06:25.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.524810+0000", 2023-07-02T14:06:25.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7217999999999999e-05 2023-07-02T14:06:25.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.524838+0000", 2023-07-02T14:06:25.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010267129 2023-07-02T14:06:25.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:25.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.535105+0000", 2023-07-02T14:06:25.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6399000000000001e-05 2023-07-02T14:06:25.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.535151+0000", 2023-07-02T14:06:25.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01191801 2023-07-02T14:06:25.300 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.300 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.300 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.300 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18694 2.5 e176/170 2:ad0b6442:::rbd_data.125de1449605.0000000000000192:head v 176'2317, mlcod=176'2314)", 2023-07-02T14:06:25.300 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.529519+0000", 2023-07-02T14:06:25.301 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.729076634999998, 2023-07-02T14:06:25.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011113384, 2023-07-02T14:06:25.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.301 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:25.301 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.529519+0000", 2023-07-02T14:06:25.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.529519+0000", 2023-07-02T14:06:25.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:25.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.529518+0000", 2023-07-02T14:06:25.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0033832699999999999 2023-07-02T14:06:25.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532902+0000", 2023-07-02T14:06:25.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1599e-05 2023-07-02T14:06:25.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532913+0000", 2023-07-02T14:06:25.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3319999999999999e-06 2023-07-02T14:06:25.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532919+0000", 2023-07-02T14:06:25.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2502000000000001e-05 2023-07-02T14:06:25.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532961+0000", 2023-07-02T14:06:25.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5031000000000001e-05 2023-07-02T14:06:25.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.532976+0000", 2023-07-02T14:06:25.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076043589999999998 2023-07-02T14:06:25.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:25.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.540580+0000", 2023-07-02T14:06:25.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2391e-05 2023-07-02T14:06:25.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.540633+0000", 2023-07-02T14:06:25.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011113384 2023-07-02T14:06:25.307 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.308 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.308 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.308 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18723 2.5 e176/170 2:a564ac9c:::rbd_data.125de1449605.0000000000000196:head v 176'2324, mlcod=176'2321)", 2023-07-02T14:06:25.308 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.568716+0000", 2023-07-02T14:06:25.308 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.689879825999999, 2023-07-02T14:06:25.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013584191000000001, 2023-07-02T14:06:25.309 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.309 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:25.309 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.568716+0000", 2023-07-02T14:06:25.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.568716+0000", 2023-07-02T14:06:25.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:25.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.568715+0000", 2023-07-02T14:06:25.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039052919999999999 2023-07-02T14:06:25.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.572620+0000", 2023-07-02T14:06:25.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5108000000000001e-05 2023-07-02T14:06:25.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.572645+0000", 2023-07-02T14:06:25.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0495e-05 2023-07-02T14:06:25.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.572656+0000", 2023-07-02T14:06:25.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6385999999999999e-05 2023-07-02T14:06:25.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.572702+0000", 2023-07-02T14:06:25.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8578e-05 2023-07-02T14:06:25.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.572731+0000", 2023-07-02T14:06:25.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095207799999999995 2023-07-02T14:06:25.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:25.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.582251+0000", 2023-07-02T14:06:25.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9021999999999999e-05 2023-07-02T14:06:25.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.582300+0000", 2023-07-02T14:06:25.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013584191000000001 2023-07-02T14:06:25.315 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.315 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.315 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.316 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18859 2.5 e177/170 2:a5de3e1e:::rbd_data.125de1449605.0000000000000022:head v 177'2343, mlcod=177'2341)", 2023-07-02T14:06:25.316 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.883954+0000", 2023-07-02T14:06:25.316 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.374642347999998, 2023-07-02T14:06:25.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013529319999999999, 2023-07-02T14:06:25.316 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.316 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:25.317 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.883954+0000", 2023-07-02T14:06:25.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.883954+0000", 2023-07-02T14:06:25.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:25.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.883953+0000", 2023-07-02T14:06:25.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003004048 2023-07-02T14:06:25.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.886957+0000", 2023-07-02T14:06:25.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0279999999999999e-05 2023-07-02T14:06:25.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.886977+0000", 2023-07-02T14:06:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2190000000000004e-06 2023-07-02T14:06:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.886985+0000", 2023-07-02T14:06:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7524e-05 2023-07-02T14:06:25.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.887043+0000", 2023-07-02T14:06:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6378000000000001e-05 2023-07-02T14:06:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.887069+0000", 2023-07-02T14:06:25.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010370551 2023-07-02T14:06:25.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:25.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.897440+0000", 2023-07-02T14:06:25.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3152999999999998e-05 2023-07-02T14:06:25.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.897483+0000", 2023-07-02T14:06:25.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013529319999999999 2023-07-02T14:06:25.323 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.323 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.323 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.324 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18881 2.5 e177/170 2:b1731d2f:::rbd_data.125de1449605.0000000000000024:head v 177'2350, mlcod=177'2347)", 2023-07-02T14:06:25.324 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:06:25.324 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.352681750999999, 2023-07-02T14:06:25.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020820709999999999, 2023-07-02T14:06:25.324 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.324 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:25.324 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:06:25.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:06:25.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:25.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905913+0000", 2023-07-02T14:06:25.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0080191519999999999 2023-07-02T14:06:25.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913932+0000", 2023-07-02T14:06:25.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3757999999999999e-05 2023-07-02T14:06:25.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913956+0000", 2023-07-02T14:06:25.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4009999999999997e-06 2023-07-02T14:06:25.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913965+0000", 2023-07-02T14:06:25.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3383e-05 2023-07-02T14:06:25.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914018+0000", 2023-07-02T14:06:25.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7674e-05 2023-07-02T14:06:25.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914046+0000", 2023-07-02T14:06:25.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012656555999999999 2023-07-02T14:06:25.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:25.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.926702+0000", 2023-07-02T14:06:25.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2891000000000001e-05 2023-07-02T14:06:25.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.926735+0000", 2023-07-02T14:06:25.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020820709999999999 2023-07-02T14:06:25.331 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.331 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.331 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.331 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18885 2.4 e177/169 2:3625364f:::rbd_data.125de1449605.0000000000000026:head v 177'1716, mlcod=177'1714)", 2023-07-02T14:06:25.331 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.914022+0000", 2023-07-02T14:06:25.332 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.344573948000001, 2023-07-02T14:06:25.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013176468, 2023-07-02T14:06:25.332 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.332 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:25.332 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914022+0000", 2023-07-02T14:06:25.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914022+0000", 2023-07-02T14:06:25.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:25.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914021+0000", 2023-07-02T14:06:25.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002349889 2023-07-02T14:06:25.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916371+0000", 2023-07-02T14:06:25.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5821e-05 2023-07-02T14:06:25.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916387+0000", 2023-07-02T14:06:25.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3200000000000006e-06 2023-07-02T14:06:25.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916396+0000", 2023-07-02T14:06:25.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010131820000000001 2023-07-02T14:06:25.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.917409+0000", 2023-07-02T14:06:25.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3589000000000001e-05 2023-07-02T14:06:25.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.917433+0000", 2023-07-02T14:06:25.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097323819999999995 2023-07-02T14:06:25.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:25.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.927165+0000", 2023-07-02T14:06:25.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3422e-05 2023-07-02T14:06:25.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.927199+0000", 2023-07-02T14:06:25.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013176468 2023-07-02T14:06:25.339 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.339 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.339 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.339 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18883 2.5 e177/170 2:a5de3e1e:::rbd_data.125de1449605.0000000000000022:head v 177'2351, mlcod=177'2347)", 2023-07-02T14:06:25.339 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.916456+0000", 2023-07-02T14:06:25.339 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.342139985999999, 2023-07-02T14:06:25.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010715634, 2023-07-02T14:06:25.340 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.340 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:25.340 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916456+0000", 2023-07-02T14:06:25.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916456+0000", 2023-07-02T14:06:25.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:25.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916455+0000", 2023-07-02T14:06:25.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2898000000000006e-05 2023-07-02T14:06:25.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916528+0000", 2023-07-02T14:06:25.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2599999999999994e-06 2023-07-02T14:06:25.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916537+0000", 2023-07-02T14:06:25.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1209999999999998e-06 2023-07-02T14:06:25.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916542+0000", 2023-07-02T14:06:25.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032458040000000001 2023-07-02T14:06:25.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.919788+0000", 2023-07-02T14:06:25.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5001999999999999e-05 2023-07-02T14:06:25.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.919813+0000", 2023-07-02T14:06:25.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073193750000000004 2023-07-02T14:06:25.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:25.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.927132+0000", 2023-07-02T14:06:25.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9486000000000001e-05 2023-07-02T14:06:25.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.927172+0000", 2023-07-02T14:06:25.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010715634 2023-07-02T14:06:25.346 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.346 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.347 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.347 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18893 2.5 e177/170 2:bbff50a9:::rbd_data.125de1449605.0000000000000027:head v 177'2354, mlcod=177'2351)", 2023-07-02T14:06:25.347 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.933879+0000", 2023-07-02T14:06:25.347 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.324716991999999, 2023-07-02T14:06:25.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012696278, 2023-07-02T14:06:25.347 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.348 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:25.348 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.933879+0000", 2023-07-02T14:06:25.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.933879+0000", 2023-07-02T14:06:25.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:25.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.933878+0000", 2023-07-02T14:06:25.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035422819999999999 2023-07-02T14:06:25.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.937421+0000", 2023-07-02T14:06:25.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7059999999999999e-06 2023-07-02T14:06:25.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.937430+0000", 2023-07-02T14:06:25.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5379999999999996e-06 2023-07-02T14:06:25.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.937435+0000", 2023-07-02T14:06:25.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8078999999999999e-05 2023-07-02T14:06:25.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.937473+0000", 2023-07-02T14:06:25.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4100999999999998e-05 2023-07-02T14:06:25.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.937497+0000", 2023-07-02T14:06:25.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.009041248 2023-07-02T14:06:25.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:25.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.946538+0000", 2023-07-02T14:06:25.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6986999999999997e-05 2023-07-02T14:06:25.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.946575+0000", 2023-07-02T14:06:25.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012696278 2023-07-02T14:06:25.354 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.354 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.354 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.355 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18935 2.4 e177/169 2:3e650bab:::rbd_data.125de1449605.0000000000000031:head v 177'1727, mlcod=177'1721)", 2023-07-02T14:06:25.355 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.994084+0000", 2023-07-02T14:06:25.355 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.264512285999999, 2023-07-02T14:06:25.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011588876, 2023-07-02T14:06:25.355 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.355 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:25.355 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.994084+0000", 2023-07-02T14:06:25.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.994084+0000", 2023-07-02T14:06:25.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:25.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.994083+0000", 2023-07-02T14:06:25.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011146350000000001 2023-07-02T14:06:25.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.995197+0000", 2023-07-02T14:06:25.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4032999999999998e-05 2023-07-02T14:06:25.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.995231+0000", 2023-07-02T14:06:25.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9760000000000001e-06 2023-07-02T14:06:25.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.995240+0000", 2023-07-02T14:06:25.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00029382799999999999 2023-07-02T14:06:25.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.995534+0000", 2023-07-02T14:06:25.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6772999999999999e-05 2023-07-02T14:06:25.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.995561+0000", 2023-07-02T14:06:25.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010068725000000001 2023-07-02T14:06:25.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:25.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.005630+0000", 2023-07-02T14:06:25.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2930999999999999e-05 2023-07-02T14:06:25.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.005673+0000", 2023-07-02T14:06:25.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011588876 2023-07-02T14:06:25.363 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.363 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.364 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.364 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19073 2.5 e177/170 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:head v 177'2378, mlcod=177'2376)", 2023-07-02T14:06:25.364 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:06:25.364 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.984968567999999, 2023-07-02T14:06:25.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018435625000000001, 2023-07-02T14:06:25.364 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.365 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:25.365 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:06:25.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:06:25.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:25.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273626+0000", 2023-07-02T14:06:25.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046035659999999999 2023-07-02T14:06:25.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278230+0000", 2023-07-02T14:06:25.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9386e-05 2023-07-02T14:06:25.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278259+0000", 2023-07-02T14:06:25.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1972e-05 2023-07-02T14:06:25.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278271+0000", 2023-07-02T14:06:25.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017532680000000001 2023-07-02T14:06:25.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280025+0000", 2023-07-02T14:06:25.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2367000000000001e-05 2023-07-02T14:06:25.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280047+0000", 2023-07-02T14:06:25.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011987028 2023-07-02T14:06:25.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:25.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.292034+0000", 2023-07-02T14:06:25.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9266000000000001e-05 2023-07-02T14:06:25.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.292063+0000", 2023-07-02T14:06:25.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018435625000000001 2023-07-02T14:06:25.372 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.372 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.372 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.373 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19075 2.5 e177/170 2:adf18789:::rbd_data.125de1449605.000000000000003c:head v 177'2379, mlcod=177'2376)", 2023-07-02T14:06:25.373 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.280379+0000", 2023-07-02T14:06:25.373 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.978217501, 2023-07-02T14:06:25.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014230578000000001, 2023-07-02T14:06:25.373 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.374 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:25.374 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280379+0000", 2023-07-02T14:06:25.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280379+0000", 2023-07-02T14:06:25.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999804 2023-07-02T14:06:25.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280359+0000", 2023-07-02T14:06:25.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047641209999999996 2023-07-02T14:06:25.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.285123+0000", 2023-07-02T14:06:25.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3711e-05 2023-07-02T14:06:25.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.285137+0000", 2023-07-02T14:06:25.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.277e-06 2023-07-02T14:06:25.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.285144+0000", 2023-07-02T14:06:25.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00041800499999999999 2023-07-02T14:06:25.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.285562+0000", 2023-07-02T14:06:25.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6143000000000001e-05 2023-07-02T14:06:25.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.285578+0000", 2023-07-02T14:06:25.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089876420000000005 2023-07-02T14:06:25.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:25.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.294566+0000", 2023-07-02T14:06:25.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3466e-05 2023-07-02T14:06:25.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.294609+0000", 2023-07-02T14:06:25.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014230578000000001 2023-07-02T14:06:25.381 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.381 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.381 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.381 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19118 2.5 e177/170 2:be65b5f1:::rbd_data.125de1449605.0000000000000043:head v 177'2389, mlcod=177'2386)", 2023-07-02T14:06:25.381 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.342749+0000", 2023-07-02T14:06:25.382 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.915847163999999, 2023-07-02T14:06:25.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011707482, 2023-07-02T14:06:25.382 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.382 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:25.382 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.342749+0000", 2023-07-02T14:06:25.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.342749+0000", 2023-07-02T14:06:25.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:25.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.342748+0000", 2023-07-02T14:06:25.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045726380000000004 2023-07-02T14:06:25.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.347320+0000", 2023-07-02T14:06:25.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1033e-05 2023-07-02T14:06:25.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.347341+0000", 2023-07-02T14:06:25.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1297e-05 2023-07-02T14:06:25.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.347353+0000", 2023-07-02T14:06:25.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9991000000000002e-05 2023-07-02T14:06:25.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.347413+0000", 2023-07-02T14:06:25.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2135000000000001e-05 2023-07-02T14:06:25.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.347435+0000", 2023-07-02T14:06:25.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0069823029999999996 2023-07-02T14:06:25.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:25.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.354417+0000", 2023-07-02T14:06:25.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9447999999999999e-05 2023-07-02T14:06:25.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.354456+0000", 2023-07-02T14:06:25.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011707482 2023-07-02T14:06:25.389 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.389 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.389 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.389 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19193 2.2 e177/171 2:5382aedc:::rbd_data.125de1449605.000000000000004e:head v 177'1704, mlcod=177'1701)", 2023-07-02T14:06:25.389 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.473454+0000", 2023-07-02T14:06:25.390 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.785141683999999, 2023-07-02T14:06:25.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012446847, 2023-07-02T14:06:25.390 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.390 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:25.390 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.473454+0000", 2023-07-02T14:06:25.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.473454+0000", 2023-07-02T14:06:25.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:25.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.473452+0000", 2023-07-02T14:06:25.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001241418 2023-07-02T14:06:25.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.474694+0000", 2023-07-02T14:06:25.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6421999999999999e-05 2023-07-02T14:06:25.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.474720+0000", 2023-07-02T14:06:25.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9219999999999992e-06 2023-07-02T14:06:25.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.474730+0000", 2023-07-02T14:06:25.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000926395 2023-07-02T14:06:25.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475656+0000", 2023-07-02T14:06:25.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5822000000000001e-05 2023-07-02T14:06:25.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475682+0000", 2023-07-02T14:06:25.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01016774 2023-07-02T14:06:25.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:25.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485850+0000", 2023-07-02T14:06:25.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1266e-05 2023-07-02T14:06:25.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485901+0000", 2023-07-02T14:06:25.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012446847 2023-07-02T14:06:25.397 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.397 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.397 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.397 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19196 2.4 e177/169 2:2c79e900:::rbd_data.125de1449605.000000000000004a:head v 177'1774, mlcod=177'1772)", 2023-07-02T14:06:25.398 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.475437+0000", 2023-07-02T14:06:25.398 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.78315941, 2023-07-02T14:06:25.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013623751, 2023-07-02T14:06:25.398 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.398 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:25.398 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475437+0000", 2023-07-02T14:06:25.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475437+0000", 2023-07-02T14:06:25.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:06:25.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475434+0000", 2023-07-02T14:06:25.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3578999999999994e-05 2023-07-02T14:06:25.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475518+0000", 2023-07-02T14:06:25.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1090999999999998e-05 2023-07-02T14:06:25.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475539+0000", 2023-07-02T14:06:25.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4860000000000003e-06 2023-07-02T14:06:25.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475549+0000", 2023-07-02T14:06:25.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000107015 2023-07-02T14:06:25.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475656+0000", 2023-07-02T14:06:25.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6625e-05 2023-07-02T14:06:25.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475682+0000", 2023-07-02T14:06:25.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013334037 2023-07-02T14:06:25.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:25.404 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489016+0000", 2023-07-02T14:06:25.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4141999999999999e-05 2023-07-02T14:06:25.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489060+0000", 2023-07-02T14:06:25.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013623751 2023-07-02T14:06:25.405 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.405 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.405 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.406 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:20649 2.2 e179/171 2:50a48cb3:::rbd_data.125de1449605.0000000000000104:head v 179'1841, mlcod=179'1836)", 2023-07-02T14:06:25.406 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:11.610880+0000", 2023-07-02T14:06:25.406 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.647715666, 2023-07-02T14:06:25.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010906325999999999, 2023-07-02T14:06:25.406 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.406 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:25.406 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.610880+0000", 2023-07-02T14:06:25.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.610880+0000", 2023-07-02T14:06:25.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:25.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.610879+0000", 2023-07-02T14:06:25.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2001000000000002e-05 2023-07-02T14:06:25.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.610901+0000", 2023-07-02T14:06:25.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9700000000000001e-05 2023-07-02T14:06:25.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.610920+0000", 2023-07-02T14:06:25.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:06:25.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.610930+0000", 2023-07-02T14:06:25.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001156722 2023-07-02T14:06:25.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.612087+0000", 2023-07-02T14:06:25.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5507000000000002e-05 2023-07-02T14:06:25.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.612112+0000", 2023-07-02T14:06:25.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0096299330000000002 2023-07-02T14:06:25.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:25.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.621742+0000", 2023-07-02T14:06:25.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4503999999999999e-05 2023-07-02T14:06:25.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.621787+0000", 2023-07-02T14:06:25.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010906325999999999 2023-07-02T14:06:25.413 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.413 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.413 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.414 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:20658 2.2 e179/171 2:44c63967:::rbd_data.125de1449605.0000000000000101:head v 179'1842, mlcod=179'1836)", 2023-07-02T14:06:25.414 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:11.612985+0000", 2023-07-02T14:06:25.414 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 13.645611090999999, 2023-07-02T14:06:25.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010831500000000001, 2023-07-02T14:06:25.414 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.414 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:25.414 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.612985+0000", 2023-07-02T14:06:25.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.612985+0000", 2023-07-02T14:06:25.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:06:25.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.612983+0000", 2023-07-02T14:06:25.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016891000000000001 2023-07-02T14:06:25.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.613152+0000", 2023-07-02T14:06:25.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5787000000000001e-05 2023-07-02T14:06:25.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.613187+0000", 2023-07-02T14:06:25.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9750000000000002e-06 2023-07-02T14:06:25.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.613197+0000", 2023-07-02T14:06:25.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000204926 2023-07-02T14:06:25.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.613402+0000", 2023-07-02T14:06:25.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5429999999999999e-05 2023-07-02T14:06:25.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.613428+0000", 2023-07-02T14:06:25.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010339263 2023-07-02T14:06:25.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:25.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.623767+0000", 2023-07-02T14:06:25.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9497999999999999e-05 2023-07-02T14:06:25.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.623817+0000", 2023-07-02T14:06:25.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010831500000000001 2023-07-02T14:06:25.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.421 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.421 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.421 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.422 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:25.423 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:25.422+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:25.424 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:06:25.523 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:25.523+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.1 to 1 (10000) 2023-07-02T14:06:25.583 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-07-02T14:06:25.607 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap53 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap54 b3fbb028-e267-4d25-a750-93ba18a2316a@snap54.diff 2023-07-02T14:06:25.624 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:25.623+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:25.675 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:25.675 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:25.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.676 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:21076 2.4 2:3f532a0a:::rbd_data.125de1449605.00000000000001b9:head [write 3276800~655360 in=655360b] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e190)", 2023-07-02T14:06:25.676 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.660438+0000", 2023-07-02T14:06:25.676 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0042510730000000002, 2023-07-02T14:06:25.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0042869910000000004, 2023-07-02T14:06:25.676 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.676 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "waiting for sub ops", 2023-07-02T14:06:25.677 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:25.677 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:25.677 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:25.677 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21076 2023-07-02T14:06:25.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.677 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.660438+0000", 2023-07-02T14:06:25.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.660438+0000", 2023-07-02T14:06:25.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:25.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.660437+0000", 2023-07-02T14:06:25.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00058913800000000001 2023-07-02T14:06:25.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.661026+0000", 2023-07-02T14:06:25.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1141999999999999e-05 2023-07-02T14:06:25.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.661037+0000", 2023-07-02T14:06:25.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8670000000000002e-06 2023-07-02T14:06:25.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.661042+0000", 2023-07-02T14:06:25.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2103000000000001e-05 2023-07-02T14:06:25.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.661074+0000", 2023-07-02T14:06:25.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5553999999999997e-05 2023-07-02T14:06:25.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.661129+0000", 2023-07-02T14:06:25.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010481500000000001 2023-07-02T14:06:25.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:25.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.661234+0000", 2023-07-02T14:06:25.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079638500000000004 2023-07-02T14:06:25.683 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.683 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.683 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.684 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:21077 2.4 2:3f532a0a:::rbd_data.125de1449605.00000000000001b9:head [write 3932160~65536 in=65536b] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e190)", 2023-07-02T14:06:25.684 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.662639+0000", 2023-07-02T14:06:25.684 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.002050372, 2023-07-02T14:06:25.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0022278160000000001, 2023-07-02T14:06:25.684 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.684 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "waiting for sub ops", 2023-07-02T14:06:25.684 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:25.685 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:25.685 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:25.685 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21077 2023-07-02T14:06:25.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.685 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.662639+0000", 2023-07-02T14:06:25.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.662639+0000", 2023-07-02T14:06:25.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:25.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.662637+0000", 2023-07-02T14:06:25.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4165999999999994e-05 2023-07-02T14:06:25.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.662711+0000", 2023-07-02T14:06:25.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6429000000000001e-05 2023-07-02T14:06:25.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.662728+0000", 2023-07-02T14:06:25.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7765999999999999e-05 2023-07-02T14:06:25.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.662745+0000", 2023-07-02T14:06:25.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00052114499999999998 2023-07-02T14:06:25.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.663267+0000", 2023-07-02T14:06:25.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1711000000000006e-05 2023-07-02T14:06:25.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.663328+0000", 2023-07-02T14:06:25.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000113997 2023-07-02T14:06:25.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:25.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.663442+0000", 2023-07-02T14:06:25.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00080378800000000005 2023-07-02T14:06:25.691 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.691 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.691 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.692 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:21078 2.2 2:55d439e0:::rbd_data.125de1449605.00000000000001b8:head [write 3538944~524288 in=524288b] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e190)", 2023-07-02T14:06:25.692 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.662855+0000", 2023-07-02T14:06:25.692 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.001833774, 2023-07-02T14:06:25.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021673679999999998, 2023-07-02T14:06:25.692 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.692 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "waiting for sub ops", 2023-07-02T14:06:25.693 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:25.693 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:25.693 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:25.693 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21078 2023-07-02T14:06:25.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.693 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.662855+0000", 2023-07-02T14:06:25.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.662855+0000", 2023-07-02T14:06:25.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:25.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.662854+0000", 2023-07-02T14:06:25.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050183300000000003 2023-07-02T14:06:25.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.663356+0000", 2023-07-02T14:06:25.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6169999999999999e-05 2023-07-02T14:06:25.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.663372+0000", 2023-07-02T14:06:25.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9500000000000004e-06 2023-07-02T14:06:25.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.663379+0000", 2023-07-02T14:06:25.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7967999999999999e-05 2023-07-02T14:06:25.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.663427+0000", 2023-07-02T14:06:25.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4731000000000001e-05 2023-07-02T14:06:25.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.663492+0000", 2023-07-02T14:06:25.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109419 2023-07-02T14:06:25.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:25.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.663601+0000", 2023-07-02T14:06:25.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00074592799999999998 2023-07-02T14:06:25.699 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.699 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.699 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.700 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.6499.0:14 2.2 2.16c67442 (undecoded) ondisk+read+known_if_redirected e190)", 2023-07-02T14:06:25.700 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.663432+0000", 2023-07-02T14:06:25.700 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.0012564609999999999, 2023-07-02T14:06:25.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0016737060000000001, 2023-07-02T14:06:25.700 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.700 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "queued for pg", 2023-07-02T14:06:25.700 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:25.701 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.6499", 2023-07-02T14:06:25.701 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2253736305", 2023-07-02T14:06:25.701 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 14 2023-07-02T14:06:25.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.701 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.663432+0000", 2023-07-02T14:06:25.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.663432+0000", 2023-07-02T14:06:25.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:25.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.663431+0000", 2023-07-02T14:06:25.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6599999999999998e-06 2023-07-02T14:06:25.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.663438+0000", 2023-07-02T14:06:25.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0219e-05 2023-07-02T14:06:25.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.663448+0000", 2023-07-02T14:06:25.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6439999999999997e-06 2023-07-02T14:06:25.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.663454+0000", 2023-07-02T14:06:25.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1324000000000001e-05 2023-07-02T14:06:25.705 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.705 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.705 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.706 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:21048 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e190)", 2023-07-02T14:06:25.706 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.594312+0000", 2023-07-02T14:06:25.706 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.070377347000000007, 2023-07-02T14:06:25.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070867679000000003, 2023-07-02T14:06:25.706 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.706 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "waiting for sub ops", 2023-07-02T14:06:25.707 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:25.707 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:25.707 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:25.707 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21048 2023-07-02T14:06:25.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.707 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594312+0000", 2023-07-02T14:06:25.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594312+0000", 2023-07-02T14:06:25.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:25.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594311+0000", 2023-07-02T14:06:25.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.05e-06 2023-07-02T14:06:25.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594314+0000", 2023-07-02T14:06:25.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9520000000000002e-06 2023-07-02T14:06:25.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594319+0000", 2023-07-02T14:06:25.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1839999999999999e-06 2023-07-02T14:06:25.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594322+0000", 2023-07-02T14:06:25.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010158032000000001 2023-07-02T14:06:25.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.604480+0000", 2023-07-02T14:06:25.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000131777 2023-07-02T14:06:25.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:25.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.604612+0000", 2023-07-02T14:06:25.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089699840000000003 2023-07-02T14:06:25.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.613582+0000", 2023-07-02T14:06:25.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0668999999999999e-05 2023-07-02T14:06:25.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:25.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.613633+0000", 2023-07-02T14:06:25.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010697506000000001 2023-07-02T14:06:25.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.624330+0000", 2023-07-02T14:06:25.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014519899999999999 2023-07-02T14:06:25.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:25.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.624476+0000", 2023-07-02T14:06:25.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015660500000000001 2023-07-02T14:06:25.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.640136+0000", 2023-07-02T14:06:25.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013172800000000001 2023-07-02T14:06:25.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:25.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.640268+0000", 2023-07-02T14:06:25.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015574856 2023-07-02T14:06:25.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.655843+0000", 2023-07-02T14:06:25.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7578999999999998e-05 2023-07-02T14:06:25.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.655910+0000", 2023-07-02T14:06:25.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00046436300000000003 2023-07-02T14:06:25.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:25.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.656375+0000", 2023-07-02T14:06:25.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0061109620000000002 2023-07-02T14:06:25.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:25.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.662486+0000", 2023-07-02T14:06:25.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.068174034999999994 2023-07-02T14:06:25.719 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.719 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.719 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.719 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4645.0:21070 2.0 e190/185 ondisk, result = 0)", 2023-07-02T14:06:25.719 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.664701+0000", 2023-07-02T14:06:25.720 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4294967295.9999876, 2023-07-02T14:06:25.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079985899999999999, 2023-07-02T14:06:25.720 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.720 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:25.720 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.664701+0000", 2023-07-02T14:06:25.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.664701+0000", 2023-07-02T14:06:25.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:25.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.664701+0000", 2023-07-02T14:06:25.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9230000000000004e-06 2023-07-02T14:06:25.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.664705+0000", 2023-07-02T14:06:25.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2589999999999997e-06 2023-07-02T14:06:25.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.664711+0000", 2023-07-02T14:06:25.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9160000000000003e-06 2023-07-02T14:06:25.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.664715+0000", 2023-07-02T14:06:25.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00046260700000000002 2023-07-02T14:06:25.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.665178+0000", 2023-07-02T14:06:25.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2179e-05 2023-07-02T14:06:25.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.665190+0000", 2023-07-02T14:06:25.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00031138599999999998 2023-07-02T14:06:25.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:25.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.665501+0000", 2023-07-02T14:06:25.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079985899999999999 2023-07-02T14:06:25.726 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.726 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.726 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.727 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:21049 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e190)", 2023-07-02T14:06:25.727 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.594326+0000", 2023-07-02T14:06:25.727 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.070362970999999996, 2023-07-02T14:06:25.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.072485954000000005, 2023-07-02T14:06:25.727 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.727 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "delayed", 2023-07-02T14:06:25.727 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:25.727 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:25.728 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:25.728 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21049 2023-07-02T14:06:25.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.728 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594326+0000", 2023-07-02T14:06:25.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594326+0000", 2023-07-02T14:06:25.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:25.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594326+0000", 2023-07-02T14:06:25.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6279999999999998e-06 2023-07-02T14:06:25.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594329+0000", 2023-07-02T14:06:25.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1390000000000003e-06 2023-07-02T14:06:25.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594334+0000", 2023-07-02T14:06:25.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4000000000000001e-06 2023-07-02T14:06:25.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594338+0000", 2023-07-02T14:06:25.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01030987 2023-07-02T14:06:25.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.604648+0000", 2023-07-02T14:06:25.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108546 2023-07-02T14:06:25.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:25.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.604756+0000", 2023-07-02T14:06:25.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089037769999999999 2023-07-02T14:06:25.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.613660+0000", 2023-07-02T14:06:25.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6557000000000001e-05 2023-07-02T14:06:25.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:25.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.613717+0000", 2023-07-02T14:06:25.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011297208 2023-07-02T14:06:25.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.625014+0000", 2023-07-02T14:06:25.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000138539 2023-07-02T14:06:25.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:25.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.625152+0000", 2023-07-02T14:06:25.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015157522 2023-07-02T14:06:25.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.640310+0000", 2023-07-02T14:06:25.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010977200000000001 2023-07-02T14:06:25.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:25.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.640420+0000", 2023-07-02T14:06:25.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018872832999999999 2023-07-02T14:06:25.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.659292+0000", 2023-07-02T14:06:25.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011888900000000001 2023-07-02T14:06:25.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:25.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.659411+0000", 2023-07-02T14:06:25.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.065085355999999997 2023-07-02T14:06:25.738 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.738 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.739 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.739 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop_reply(client.4645.0:21048 2.0 e190/185 ondisk, result = 0)", 2023-07-02T14:06:25.739 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.664772+0000", 2023-07-02T14:06:25.739 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4294967295.999917, 2023-07-02T14:06:25.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021915620000000002, 2023-07-02T14:06:25.739 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.740 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:25.740 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.664772+0000", 2023-07-02T14:06:25.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.664772+0000", 2023-07-02T14:06:25.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:25.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.664771+0000", 2023-07-02T14:06:25.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.152e-06 2023-07-02T14:06:25.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.664776+0000", 2023-07-02T14:06:25.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5907000000000001e-05 2023-07-02T14:06:25.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.664802+0000", 2023-07-02T14:06:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7909999999999999e-06 2023-07-02T14:06:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.664809+0000", 2023-07-02T14:06:25.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00071888900000000001 2023-07-02T14:06:25.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.665528+0000", 2023-07-02T14:06:25.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5923999999999997e-05 2023-07-02T14:06:25.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:25.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.665564+0000", 2023-07-02T14:06:25.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00079212399999999995 2023-07-02T14:06:25.745 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.745 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.745 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.746 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:21050 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e190)", 2023-07-02T14:06:25.746 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.594341+0000", 2023-07-02T14:06:25.746 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.070347794000000005, 2023-07-02T14:06:25.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.072745097999999994, 2023-07-02T14:06:25.746 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.746 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "delayed", 2023-07-02T14:06:25.746 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:25.746 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:25.747 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:25.747 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21050 2023-07-02T14:06:25.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.747 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594341+0000", 2023-07-02T14:06:25.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594341+0000", 2023-07-02T14:06:25.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:25.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594341+0000", 2023-07-02T14:06:25.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0580000000000002e-06 2023-07-02T14:06:25.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594344+0000", 2023-07-02T14:06:25.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1270000000000002e-06 2023-07-02T14:06:25.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594349+0000", 2023-07-02T14:06:25.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3620000000000002e-06 2023-07-02T14:06:25.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594352+0000", 2023-07-02T14:06:25.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010434524000000001 2023-07-02T14:06:25.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.751 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.751 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.604787+0000", 2023-07-02T14:06:25.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017082299999999999 2023-07-02T14:06:25.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:25.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.604958+0000", 2023-07-02T14:06:25.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0099093840000000002 2023-07-02T14:06:25.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.614867+0000", 2023-07-02T14:06:25.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7389000000000001e-05 2023-07-02T14:06:25.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:25.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.614934+0000", 2023-07-02T14:06:25.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010277787 2023-07-02T14:06:25.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.625212+0000", 2023-07-02T14:06:25.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106523 2023-07-02T14:06:25.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:25.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.625319+0000", 2023-07-02T14:06:25.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015144019 2023-07-02T14:06:25.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.640463+0000", 2023-07-02T14:06:25.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7225999999999997e-05 2023-07-02T14:06:25.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:25.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.640520+0000", 2023-07-02T14:06:25.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018942843000000001 2023-07-02T14:06:25.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.659463+0000", 2023-07-02T14:06:25.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7238999999999995e-05 2023-07-02T14:06:25.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:25.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.659560+0000", 2023-07-02T14:06:25.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.065219004999999997 2023-07-02T14:06:25.757 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.757 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.758 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.758 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPushReply(2.0 190/185 [PushReplyOp(2:1f6e35fd:::rbd_data.125de1449605.0000000000000111:head)])", 2023-07-02T14:06:25.758 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.664820+0000", 2023-07-02T14:06:25.758 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4294967295.9998689, 2023-07-02T14:06:25.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0024972369999999998, 2023-07-02T14:06:25.758 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.759 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "queued for pg", 2023-07-02T14:06:25.759 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.759 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.664820+0000", 2023-07-02T14:06:25.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.664820+0000", 2023-07-02T14:06:25.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:25.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.664820+0000", 2023-07-02T14:06:25.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7819999999999999e-06 2023-07-02T14:06:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.664825+0000", 2023-07-02T14:06:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8259999999999999e-06 2023-07-02T14:06:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.664834+0000", 2023-07-02T14:06:25.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3329999999999996e-06 2023-07-02T14:06:25.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.664839+0000", 2023-07-02T14:06:25.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9474e-05 2023-07-02T14:06:25.763 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.763 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.763 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.763 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:21051 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=44b,call rbd.object_map_update in=38b] snapc 0=[] ondisk+write+known_if_redirected e190)", 2023-07-02T14:06:25.763 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.594356+0000", 2023-07-02T14:06:25.763 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.070333145, 2023-07-02T14:06:25.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.073045228000000004, 2023-07-02T14:06:25.764 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.764 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "delayed", 2023-07-02T14:06:25.764 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:25.764 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:25.764 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:25.764 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21051 2023-07-02T14:06:25.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.765 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594356+0000", 2023-07-02T14:06:25.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594356+0000", 2023-07-02T14:06:25.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:25.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594355+0000", 2023-07-02T14:06:25.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0060000000000001e-06 2023-07-02T14:06:25.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594358+0000", 2023-07-02T14:06:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1533e-05 2023-07-02T14:06:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594370+0000", 2023-07-02T14:06:25.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1860000000000002e-06 2023-07-02T14:06:25.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.594374+0000", 2023-07-02T14:06:25.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010621821 2023-07-02T14:06:25.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.604996+0000", 2023-07-02T14:06:25.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015216200000000001 2023-07-02T14:06:25.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:25.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.605148+0000", 2023-07-02T14:06:25.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0098872209999999999 2023-07-02T14:06:25.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.615035+0000", 2023-07-02T14:06:25.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8943999999999998e-05 2023-07-02T14:06:25.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:25.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.615084+0000", 2023-07-02T14:06:25.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010311753999999999 2023-07-02T14:06:25.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.625396+0000", 2023-07-02T14:06:25.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000105746 2023-07-02T14:06:25.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:25.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.625502+0000", 2023-07-02T14:06:25.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015613691000000001 2023-07-02T14:06:25.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.641116+0000", 2023-07-02T14:06:25.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5194e-05 2023-07-02T14:06:25.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:25.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.641171+0000", 2023-07-02T14:06:25.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018629100999999999 2023-07-02T14:06:25.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:25.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.659800+0000", 2023-07-02T14:06:25.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100257 2023-07-02T14:06:25.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:25.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.659900+0000", 2023-07-02T14:06:25.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.065544313000000007 2023-07-02T14:06:25.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.775 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.775 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.775 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPushReply(2.0 190/185 [PushReplyOp(2:014344e8:::rbd_data.125de1449605.0000000000000114:head)])", 2023-07-02T14:06:25.775 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.665556+0000", 2023-07-02T14:06:25.776 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4294967295.9991331, 2023-07-02T14:06:25.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0019914020000000002, 2023-07-02T14:06:25.776 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.776 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "queued for pg", 2023-07-02T14:06:25.776 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.665556+0000", 2023-07-02T14:06:25.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.665556+0000", 2023-07-02T14:06:25.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:25.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.665555+0000", 2023-07-02T14:06:25.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3110000000000002e-06 2023-07-02T14:06:25.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.665561+0000", 2023-07-02T14:06:25.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.06e-05 2023-07-02T14:06:25.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.665572+0000", 2023-07-02T14:06:25.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.2929999999999997e-06 2023-07-02T14:06:25.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.665578+0000", 2023-07-02T14:06:25.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2461999999999999e-05 2023-07-02T14:06:25.780 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.780 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.780 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.781 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:21079 2.4 2.9b17a8bc (undecoded) ondisk+write+known_if_redirected e190)", 2023-07-02T14:06:25.781 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.663748+0000", 2023-07-02T14:06:25.781 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.00094124400000000004, 2023-07-02T14:06:25.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038608290000000001, 2023-07-02T14:06:25.781 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:25.781 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "queued for pg", 2023-07-02T14:06:25.781 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:25.782 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:25.782 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:25.782 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21079 2023-07-02T14:06:25.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.782 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:25.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:25.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.663748+0000", 2023-07-02T14:06:25.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:25.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:25.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.663748+0000", 2023-07-02T14:06:25.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:25.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.663747+0000", 2023-07-02T14:06:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003734837 2023-07-02T14:06:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.667482+0000", 2023-07-02T14:06:25.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1878999999999999e-05 2023-07-02T14:06:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.667493+0000", 2023-07-02T14:06:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4699999999999999e-06 2023-07-02T14:06:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:25.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.667500+0000", 2023-07-02T14:06:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0037522139999999998 2023-07-02T14:06:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:25.786 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:06:25.787 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 13 2023-07-02T14:06:25.787 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:25.787 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:06:25.789 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:06:25.790 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:25.787+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:25.790 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete...2023-07-02T14:06:25.706+0000 7f745a875700 0 -- 172.21.15.163:0/2253736305 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x55e4007f7e30 msgr2=0x55e4007fa220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-07-02T14:06:25.790 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:25.713+0000 7f745a875700 0 -- 172.21.15.163:0/2253736305 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f7440036c10 msgr2=0x7f7440039040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:25.791 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 51% complete... Exporting image: 100% complete...done. 2023-07-02T14:06:25.791 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:06:25.791 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=55 2023-07-02T14:06:25.791 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap55 2023-07-02T14:06:25.888 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:25.887+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:25.975 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:25.976 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:25.976 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:06:25.976 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:06:25.976 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:25.984 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:25.987 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:06:25.989 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:25.988+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:26.005 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:26.026 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:26.047 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:26.066 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:26.085 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:26.087 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:06:26.087 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:06:26.087 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:06:26.087 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:06:26.087 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:06:26.087 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:06:26.090 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:26.090+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:26.144 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:26.144 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:26.155 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:06:26.192 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:26.190+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:26.291 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:26.291 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:26.292 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:26.291+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:26.303 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:06:26.393 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:26.392+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:26.485 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:26.486 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:26.493 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:26.492+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:26.498 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:06:26.556 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.3 to 1 (10000) 2023-07-02T14:06:26.572 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-07-02T14:06:26.605 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:06:26.612 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 4 2023-07-02T14:06:26.613 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:26.612+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:26.695 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:26.694+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:26.700 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:26.701 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:26.714 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:06:26.727 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:06:26.796 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:26.795+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:26.851 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:26.851 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:26.864 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:06:26.897 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:26.896+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:26.998 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:26.997+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:27.037 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:27.037 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:27.049 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:06:27.099 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:27.098+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:27.200 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:27.198+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:27.203 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:27.203 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:27.217 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:06:27.300 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:27.299+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:27.362 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:27.363 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:27.375 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:06:27.402 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:27.400+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:27.502 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:27.501+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:27.521 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:27.521 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:27.533 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:06:27.558 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.2 to 1 (10000) 2023-07-02T14:06:27.559 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:06:27.578 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2023-07-02T14:06:27.619 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:27.617+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:27.704 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:27.703+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:27.712 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:27.713 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:27.726 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:06:27.805 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:27.804+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:27.906 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:27.905+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:27.923 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:27.923 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:27.936 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:06:28.022 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:28.021+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:28.099 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:06:28.108 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:28.107+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:28.123 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:28.123 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:28.136 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:06:28.219 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:28.218+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:28.254 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:28.273 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:28.291 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:28.310 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:28.309+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:28.311 INFO:tasks.workunit.client.0.smithi163.stderr:+ tar -C b3fbb028-e267-4d25-a750-93ba18a2316a-mnt -xzf b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4.tar.gz 2023-07-02T14:06:28.312 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:28.325 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:28.325 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:28.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:28.326 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:28.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.327 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17354 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:28.327 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:28.328 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 45.016043901000003, 2023-07-02T14:06:28.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493, 2023-07-02T14:06:28.329 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:28.329 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:28.329 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:28.330 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:28.330 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:28.331 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17354 2023-07-02T14:06:28.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.331 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:28.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:28.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:28.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:28.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:28.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:28.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:28.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:28.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:28.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7440000000000001e-06 2023-07-02T14:06:28.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:28.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302606+0000", 2023-07-02T14:06:28.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:06:28.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:28.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302613+0000", 2023-07-02T14:06:28.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8810000000000004e-06 2023-07-02T14:06:28.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:28.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302618+0000", 2023-07-02T14:06:28.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028613100000000002 2023-07-02T14:06:28.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302904+0000", 2023-07-02T14:06:28.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5278999999999995e-05 2023-07-02T14:06:28.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302979+0000", 2023-07-02T14:06:28.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096904750000000005 2023-07-02T14:06:28.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312669+0000", 2023-07-02T14:06:28.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0328000000000002e-05 2023-07-02T14:06:28.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312700+0000", 2023-07-02T14:06:28.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098019319999999993 2023-07-02T14:06:28.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322502+0000", 2023-07-02T14:06:28.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4669999999999998e-05 2023-07-02T14:06:28.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322536+0000", 2023-07-02T14:06:28.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010023839 2023-07-02T14:06:28.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332560+0000", 2023-07-02T14:06:28.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5559999999999998e-05 2023-07-02T14:06:28.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332596+0000", 2023-07-02T14:06:28.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090401319999999993 2023-07-02T14:06:28.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341636+0000", 2023-07-02T14:06:28.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5373000000000001e-05 2023-07-02T14:06:28.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341681+0000", 2023-07-02T14:06:28.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081994059999999994 2023-07-02T14:06:28.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.342 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.342 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349881+0000", 2023-07-02T14:06:28.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4319000000000001e-05 2023-07-02T14:06:28.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349915+0000", 2023-07-02T14:06:28.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089388339999999997 2023-07-02T14:06:28.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358854+0000", 2023-07-02T14:06:28.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4319000000000005e-05 2023-07-02T14:06:28.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358928+0000", 2023-07-02T14:06:28.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080234690000000001 2023-07-02T14:06:28.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366952+0000", 2023-07-02T14:06:28.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2082999999999998e-05 2023-07-02T14:06:28.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366984+0000", 2023-07-02T14:06:28.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075015790000000004 2023-07-02T14:06:28.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374485+0000", 2023-07-02T14:06:28.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4480999999999995e-05 2023-07-02T14:06:28.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374550+0000", 2023-07-02T14:06:28.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091666620000000008 2023-07-02T14:06:28.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383716+0000", 2023-07-02T14:06:28.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9318000000000003e-05 2023-07-02T14:06:28.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383786+0000", 2023-07-02T14:06:28.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013434115 2023-07-02T14:06:28.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397220+0000", 2023-07-02T14:06:28.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4573999999999997e-05 2023-07-02T14:06:28.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397264+0000", 2023-07-02T14:06:28.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064824649999999998 2023-07-02T14:06:28.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403747+0000", 2023-07-02T14:06:28.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1087999999999993e-05 2023-07-02T14:06:28.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403828+0000", 2023-07-02T14:06:28.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072854449999999998 2023-07-02T14:06:28.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411113+0000", 2023-07-02T14:06:28.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5120000000000001e-05 2023-07-02T14:06:28.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411169+0000", 2023-07-02T14:06:28.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007211505 2023-07-02T14:06:28.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418380+0000", 2023-07-02T14:06:28.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011289399999999999 2023-07-02T14:06:28.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:28.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418493+0000", 2023-07-02T14:06:28.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00051217500000000002 2023-07-02T14:06:28.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:28.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.419005+0000", 2023-07-02T14:06:28.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00327393 2023-07-02T14:06:28.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:28.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422279+0000", 2023-07-02T14:06:28.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041432350000000003 2023-07-02T14:06:28.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:28.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426422+0000", 2023-07-02T14:06:28.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7021000000000003e-05 2023-07-02T14:06:28.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:28.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426469+0000", 2023-07-02T14:06:28.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000190256 2023-07-02T14:06:28.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:28.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426660+0000", 2023-07-02T14:06:28.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493 2023-07-02T14:06:28.357 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.357 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.357 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.357 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17355 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:28.357 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:28.358 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 45.016025239999998, 2023-07-02T14:06:28.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001, 2023-07-02T14:06:28.358 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:28.358 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:28.358 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:28.358 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:28.358 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:28.358 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17355 2023-07-02T14:06:28.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.359 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:28.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:28.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:28.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:28.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:28.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:28.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:28.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:28.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302621+0000", 2023-07-02T14:06:28.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2260000000000001e-06 2023-07-02T14:06:28.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:28.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302625+0000", 2023-07-02T14:06:28.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8599999999999998e-06 2023-07-02T14:06:28.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:28.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302630+0000", 2023-07-02T14:06:28.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0629999999999999e-06 2023-07-02T14:06:28.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:28.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302635+0000", 2023-07-02T14:06:28.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035731199999999998 2023-07-02T14:06:28.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302992+0000", 2023-07-02T14:06:28.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4164e-05 2023-07-02T14:06:28.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.303026+0000", 2023-07-02T14:06:28.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097115900000000008 2023-07-02T14:06:28.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312738+0000", 2023-07-02T14:06:28.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1701999999999998e-05 2023-07-02T14:06:28.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312819+0000", 2023-07-02T14:06:28.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097363640000000008 2023-07-02T14:06:28.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322556+0000", 2023-07-02T14:06:28.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2517000000000001e-05 2023-07-02T14:06:28.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322588+0000", 2023-07-02T14:06:28.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01002393 2023-07-02T14:06:28.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332612+0000", 2023-07-02T14:06:28.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2960999999999998e-05 2023-07-02T14:06:28.367 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332645+0000", 2023-07-02T14:06:28.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090669819999999995 2023-07-02T14:06:28.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341712+0000", 2023-07-02T14:06:28.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9288000000000002e-05 2023-07-02T14:06:28.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341811+0000", 2023-07-02T14:06:28.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081504460000000004 2023-07-02T14:06:28.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349962+0000", 2023-07-02T14:06:28.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107321 2023-07-02T14:06:28.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.350069+0000", 2023-07-02T14:06:28.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088790999999999991 2023-07-02T14:06:28.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358948+0000", 2023-07-02T14:06:28.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1711000000000002e-05 2023-07-02T14:06:28.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358980+0000", 2023-07-02T14:06:28.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080367579999999997 2023-07-02T14:06:28.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367017+0000", 2023-07-02T14:06:28.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6329000000000002e-05 2023-07-02T14:06:28.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367073+0000", 2023-07-02T14:06:28.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075097300000000001 2023-07-02T14:06:28.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374583+0000", 2023-07-02T14:06:28.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5710999999999995e-05 2023-07-02T14:06:28.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374648+0000", 2023-07-02T14:06:28.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009165912 2023-07-02T14:06:28.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383814+0000", 2023-07-02T14:06:28.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3916000000000003e-05 2023-07-02T14:06:28.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383878+0000", 2023-07-02T14:06:28.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013440532999999999 2023-07-02T14:06:28.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397319+0000", 2023-07-02T14:06:28.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7218999999999995e-05 2023-07-02T14:06:28.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397386+0000", 2023-07-02T14:06:28.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065165279999999997 2023-07-02T14:06:28.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403903+0000", 2023-07-02T14:06:28.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8152000000000005e-05 2023-07-02T14:06:28.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404001+0000", 2023-07-02T14:06:28.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072185360000000002 2023-07-02T14:06:28.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411219+0000", 2023-07-02T14:06:28.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:06:28.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411310+0000", 2023-07-02T14:06:28.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010021663 2023-07-02T14:06:28.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421331+0000", 2023-07-02T14:06:28.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0148000000000006e-05 2023-07-02T14:06:28.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421411+0000", 2023-07-02T14:06:28.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053056170000000003 2023-07-02T14:06:28.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426717+0000", 2023-07-02T14:06:28.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8242000000000001e-05 2023-07-02T14:06:28.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:28.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426795+0000", 2023-07-02T14:06:28.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00041308399999999998 2023-07-02T14:06:28.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:28.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.427208+0000", 2023-07-02T14:06:28.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028330849999999999 2023-07-02T14:06:28.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:28.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430041+0000", 2023-07-02T14:06:28.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00072642300000000002 2023-07-02T14:06:28.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:28.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430768+0000", 2023-07-02T14:06:28.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1847000000000002e-05 2023-07-02T14:06:28.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:28.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430800+0000", 2023-07-02T14:06:28.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9308999999999994e-05 2023-07-02T14:06:28.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:28.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430899+0000", 2023-07-02T14:06:28.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001 2023-07-02T14:06:28.385 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.385 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.385 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.386 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:410 2.5 2:bbaed619:::rbd_data.125de1449605.0000000000000109:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:06:28.386 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:28.386 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.868079975000001, 2023-07-02T14:06:28.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003, 2023-07-02T14:06:28.386 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:28.386 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:28.386 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:28.387 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:06:28.387 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:06:28.387 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:06:28.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.387 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:28.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:28.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:28.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:28.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:28.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:28.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:28.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:28.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:28.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.635e-06 2023-07-02T14:06:28.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:28.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450568+0000", 2023-07-02T14:06:28.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5970000000000002e-06 2023-07-02T14:06:28.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:28.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450573+0000", 2023-07-02T14:06:28.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3379999999999999e-06 2023-07-02T14:06:28.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:28.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450575+0000", 2023-07-02T14:06:28.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9458000000000001e-05 2023-07-02T14:06:28.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450615+0000", 2023-07-02T14:06:28.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0569999999999993e-06 2023-07-02T14:06:28.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:28.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450623+0000", 2023-07-02T14:06:28.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.63968709199999996 2023-07-02T14:06:28.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090310+0000", 2023-07-02T14:06:28.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020157399999999999 2023-07-02T14:06:28.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:28.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090512+0000", 2023-07-02T14:06:28.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027739190000000001 2023-07-02T14:06:28.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:28.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093285+0000", 2023-07-02T14:06:28.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003 2023-07-02T14:06:28.394 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.394 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.394 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.395 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:416 2.5 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:06:28.395 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:28.395 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.868050541999999, 2023-07-02T14:06:28.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002, 2023-07-02T14:06:28.395 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:28.395 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:28.396 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:28.396 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:06:28.396 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:06:28.396 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:06:28.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.396 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:28.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:28.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:28.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:28.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:28.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:28.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:28.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:28.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:28.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.34e-06 2023-07-02T14:06:28.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:28.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450599+0000", 2023-07-02T14:06:28.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1549999999999998e-06 2023-07-02T14:06:28.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:28.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450604+0000", 2023-07-02T14:06:28.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:06:28.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:28.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450607+0000", 2023-07-02T14:06:28.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5717999999999999e-05 2023-07-02T14:06:28.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450643+0000", 2023-07-02T14:06:28.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1619999999999999e-06 2023-07-02T14:06:28.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:28.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450648+0000", 2023-07-02T14:06:28.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64292291800000001 2023-07-02T14:06:28.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093571+0000", 2023-07-02T14:06:28.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115628 2023-07-02T14:06:28.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:28.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093687+0000", 2023-07-02T14:06:28.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002885552 2023-07-02T14:06:28.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:28.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.096572+0000", 2023-07-02T14:06:28.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002 2023-07-02T14:06:28.403 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.403 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.404 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.404 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18557 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:28.404 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:28.404 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 23.069678162999999, 2023-07-02T14:06:28.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13120991400000001, 2023-07-02T14:06:28.404 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:28.405 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:28.405 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:28.405 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:28.405 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:28.405 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18557 2023-07-02T14:06:28.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.405 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:28.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:28.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:28.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:28.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:28.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:28.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:06:28.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:28.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:28.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8870000000000001e-06 2023-07-02T14:06:28.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:28.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248971+0000", 2023-07-02T14:06:28.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9009999999999996e-06 2023-07-02T14:06:28.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:28.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248974+0000", 2023-07-02T14:06:28.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9809999999999997e-06 2023-07-02T14:06:28.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:28.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248978+0000", 2023-07-02T14:06:28.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00076989199999999999 2023-07-02T14:06:28.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249748+0000", 2023-07-02T14:06:28.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8245999999999999e-05 2023-07-02T14:06:28.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249787+0000", 2023-07-02T14:06:28.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066332969999999998 2023-07-02T14:06:28.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256420+0000", 2023-07-02T14:06:28.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2566e-05 2023-07-02T14:06:28.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256452+0000", 2023-07-02T14:06:28.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012364714000000001 2023-07-02T14:06:28.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268817+0000", 2023-07-02T14:06:28.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5113000000000001e-05 2023-07-02T14:06:28.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268852+0000", 2023-07-02T14:06:28.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010993899999999999 2023-07-02T14:06:28.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279846+0000", 2023-07-02T14:06:28.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2144999999999998e-05 2023-07-02T14:06:28.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279878+0000", 2023-07-02T14:06:28.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011778958000000001 2023-07-02T14:06:28.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291657+0000", 2023-07-02T14:06:28.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9592000000000001e-05 2023-07-02T14:06:28.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291687+0000", 2023-07-02T14:06:28.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081107220000000008 2023-07-02T14:06:28.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299798+0000", 2023-07-02T14:06:28.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0290999999999998e-05 2023-07-02T14:06:28.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299828+0000", 2023-07-02T14:06:28.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071584489999999999 2023-07-02T14:06:28.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306986+0000", 2023-07-02T14:06:28.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0755999999999999e-05 2023-07-02T14:06:28.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307047+0000", 2023-07-02T14:06:28.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057110260000000001 2023-07-02T14:06:28.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312758+0000", 2023-07-02T14:06:28.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5145999999999999e-05 2023-07-02T14:06:28.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312793+0000", 2023-07-02T14:06:28.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004758361 2023-07-02T14:06:28.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317552+0000", 2023-07-02T14:06:28.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9190000000000002e-05 2023-07-02T14:06:28.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317601+0000", 2023-07-02T14:06:28.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050329420000000003 2023-07-02T14:06:28.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322634+0000", 2023-07-02T14:06:28.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5737999999999998e-05 2023-07-02T14:06:28.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322719+0000", 2023-07-02T14:06:28.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064995679999999998 2023-07-02T14:06:28.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329219+0000", 2023-07-02T14:06:28.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0908000000000003e-05 2023-07-02T14:06:28.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329250+0000", 2023-07-02T14:06:28.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062072029999999997 2023-07-02T14:06:28.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335457+0000", 2023-07-02T14:06:28.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9471999999999998e-05 2023-07-02T14:06:28.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335487+0000", 2023-07-02T14:06:28.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097780239999999997 2023-07-02T14:06:28.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345265+0000", 2023-07-02T14:06:28.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0982999999999994e-05 2023-07-02T14:06:28.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345336+0000", 2023-07-02T14:06:28.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046378799999999996 2023-07-02T14:06:28.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349973+0000", 2023-07-02T14:06:28.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1069999999999999e-05 2023-07-02T14:06:28.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350005+0000", 2023-07-02T14:06:28.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042766870000000004 2023-07-02T14:06:28.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354281+0000", 2023-07-02T14:06:28.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1155000000000002e-05 2023-07-02T14:06:28.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354312+0000", 2023-07-02T14:06:28.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071436820000000002 2023-07-02T14:06:28.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361456+0000", 2023-07-02T14:06:28.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9054000000000002e-05 2023-07-02T14:06:28.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361505+0000", 2023-07-02T14:06:28.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072301320000000002 2023-07-02T14:06:28.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368735+0000", 2023-07-02T14:06:28.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1013999999999994e-05 2023-07-02T14:06:28.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368816+0000", 2023-07-02T14:06:28.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036582490000000001 2023-07-02T14:06:28.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372475+0000", 2023-07-02T14:06:28.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4652000000000004e-05 2023-07-02T14:06:28.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:28.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372539+0000", 2023-07-02T14:06:28.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00031374199999999999 2023-07-02T14:06:28.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:28.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372853+0000", 2023-07-02T14:06:28.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030071299999999998 2023-07-02T14:06:28.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:28.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.375860+0000", 2023-07-02T14:06:28.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041897749999999997 2023-07-02T14:06:28.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:28.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380050+0000", 2023-07-02T14:06:28.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8331000000000003e-05 2023-07-02T14:06:28.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:28.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380088+0000", 2023-07-02T14:06:28.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0626999999999998e-05 2023-07-02T14:06:28.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:28.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380179+0000", 2023-07-02T14:06:28.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13120991400000001 2023-07-02T14:06:28.436 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.436 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.436 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.436 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18558 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:28.436 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:28.436 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 23.069661460999999, 2023-07-02T14:06:28.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13755567899999999, 2023-07-02T14:06:28.437 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:28.437 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:28.437 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:28.437 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:28.437 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:28.437 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18558 2023-07-02T14:06:28.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.438 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:28.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:28.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:28.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:28.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:28.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:28.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:28.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:28.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248985+0000", 2023-07-02T14:06:28.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8630000000000001e-06 2023-07-02T14:06:28.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:28.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248989+0000", 2023-07-02T14:06:28.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0750000000000003e-06 2023-07-02T14:06:28.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:28.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248995+0000", 2023-07-02T14:06:28.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3290000000000004e-06 2023-07-02T14:06:28.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:28.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248999+0000", 2023-07-02T14:06:28.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00080656300000000005 2023-07-02T14:06:28.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249806+0000", 2023-07-02T14:06:28.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2040999999999999e-05 2023-07-02T14:06:28.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249848+0000", 2023-07-02T14:06:28.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066312419999999999 2023-07-02T14:06:28.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256479+0000", 2023-07-02T14:06:28.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3348e-05 2023-07-02T14:06:28.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256513+0000", 2023-07-02T14:06:28.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012391531000000001 2023-07-02T14:06:28.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268904+0000", 2023-07-02T14:06:28.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.347e-05 2023-07-02T14:06:28.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268978+0000", 2023-07-02T14:06:28.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010929772000000001 2023-07-02T14:06:28.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279907+0000", 2023-07-02T14:06:28.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2669999999999997e-05 2023-07-02T14:06:28.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279940+0000", 2023-07-02T14:06:28.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011770332999999999 2023-07-02T14:06:28.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291710+0000", 2023-07-02T14:06:28.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1207000000000001e-05 2023-07-02T14:06:28.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291742+0000", 2023-07-02T14:06:28.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081242199999999997 2023-07-02T14:06:28.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299866+0000", 2023-07-02T14:06:28.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4650000000000002e-05 2023-07-02T14:06:28.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299900+0000", 2023-07-02T14:06:28.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071810270000000004 2023-07-02T14:06:28.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307081+0000", 2023-07-02T14:06:28.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4378000000000002e-05 2023-07-02T14:06:28.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307116+0000", 2023-07-02T14:06:28.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057031909999999998 2023-07-02T14:06:28.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312819+0000", 2023-07-02T14:06:28.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1612999999999997e-05 2023-07-02T14:06:28.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312851+0000", 2023-07-02T14:06:28.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048172470000000002 2023-07-02T14:06:28.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317668+0000", 2023-07-02T14:06:28.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3328e-05 2023-07-02T14:06:28.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317741+0000", 2023-07-02T14:06:28.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050117290000000004 2023-07-02T14:06:28.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322753+0000", 2023-07-02T14:06:28.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9872000000000001e-05 2023-07-02T14:06:28.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322783+0000", 2023-07-02T14:06:28.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064910610000000002 2023-07-02T14:06:28.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329274+0000", 2023-07-02T14:06:28.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0097e-05 2023-07-02T14:06:28.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329304+0000", 2023-07-02T14:06:28.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062163879999999998 2023-07-02T14:06:28.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335520+0000", 2023-07-02T14:06:28.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9513e-05 2023-07-02T14:06:28.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335550+0000", 2023-07-02T14:06:28.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098209969999999997 2023-07-02T14:06:28.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345371+0000", 2023-07-02T14:06:28.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9862999999999999e-05 2023-07-02T14:06:28.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345401+0000", 2023-07-02T14:06:28.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046552030000000001 2023-07-02T14:06:28.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350056+0000", 2023-07-02T14:06:28.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8751999999999993e-05 2023-07-02T14:06:28.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350145+0000", 2023-07-02T14:06:28.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041981450000000003 2023-07-02T14:06:28.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354343+0000", 2023-07-02T14:06:28.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1501000000000002e-05 2023-07-02T14:06:28.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354374+0000", 2023-07-02T14:06:28.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071551740000000003 2023-07-02T14:06:28.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361530+0000", 2023-07-02T14:06:28.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5156e-05 2023-07-02T14:06:28.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361565+0000", 2023-07-02T14:06:28.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072918569999999997 2023-07-02T14:06:28.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368857+0000", 2023-07-02T14:06:28.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5770999999999999e-05 2023-07-02T14:06:28.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368902+0000", 2023-07-02T14:06:28.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057210569999999999 2023-07-02T14:06:28.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374623+0000", 2023-07-02T14:06:28.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3266999999999998e-05 2023-07-02T14:06:28.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374667+0000", 2023-07-02T14:06:28.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055479570000000001 2023-07-02T14:06:28.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380215+0000", 2023-07-02T14:06:28.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0067e-05 2023-07-02T14:06:28.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:28.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380265+0000", 2023-07-02T14:06:28.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00043082299999999999 2023-07-02T14:06:28.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:28.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380696+0000", 2023-07-02T14:06:28.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024853240000000001 2023-07-02T14:06:28.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:28.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.383181+0000", 2023-07-02T14:06:28.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032601280000000002 2023-07-02T14:06:28.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:28.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386441+0000", 2023-07-02T14:06:28.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9748e-05 2023-07-02T14:06:28.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:28.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386471+0000", 2023-07-02T14:06:28.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0531000000000006e-05 2023-07-02T14:06:28.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:28.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386541+0000", 2023-07-02T14:06:28.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13755567899999999 2023-07-02T14:06:28.469 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.469 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.469 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.470 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18559 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:28.470 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:28.470 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 23.069638857000001, 2023-07-02T14:06:28.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.146184591, 2023-07-02T14:06:28.470 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:28.470 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:28.470 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:28.471 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:28.471 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:28.471 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18559 2023-07-02T14:06:28.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.471 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:28.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:28.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:28.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:28.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:28.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:28.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:28.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:28.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:28.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.726e-06 2023-07-02T14:06:28.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:28.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249011+0000", 2023-07-02T14:06:28.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7939999999999997e-06 2023-07-02T14:06:28.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:28.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249018+0000", 2023-07-02T14:06:28.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3470000000000001e-06 2023-07-02T14:06:28.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:28.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249023+0000", 2023-07-02T14:06:28.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00084402900000000005 2023-07-02T14:06:28.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249867+0000", 2023-07-02T14:06:28.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9050000000000001e-05 2023-07-02T14:06:28.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249906+0000", 2023-07-02T14:06:28.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066311039999999996 2023-07-02T14:06:28.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256537+0000", 2023-07-02T14:06:28.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2790000000000003e-05 2023-07-02T14:06:28.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256570+0000", 2023-07-02T14:06:28.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012436129000000001 2023-07-02T14:06:28.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269006+0000", 2023-07-02T14:06:28.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0173999999999999e-05 2023-07-02T14:06:28.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269046+0000", 2023-07-02T14:06:28.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010924698 2023-07-02T14:06:28.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279971+0000", 2023-07-02T14:06:28.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3831000000000002e-05 2023-07-02T14:06:28.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280004+0000", 2023-07-02T14:06:28.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011795912 2023-07-02T14:06:28.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291800+0000", 2023-07-02T14:06:28.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5613999999999997e-05 2023-07-02T14:06:28.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291886+0000", 2023-07-02T14:06:28.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080395999999999992 2023-07-02T14:06:28.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299925+0000", 2023-07-02T14:06:28.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0694999999999997e-05 2023-07-02T14:06:28.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299956+0000", 2023-07-02T14:06:28.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072049449999999999 2023-07-02T14:06:28.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307161+0000", 2023-07-02T14:06:28.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5523000000000001e-05 2023-07-02T14:06:28.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307247+0000", 2023-07-02T14:06:28.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056331929999999999 2023-07-02T14:06:28.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312880+0000", 2023-07-02T14:06:28.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1646000000000002e-05 2023-07-02T14:06:28.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312911+0000", 2023-07-02T14:06:28.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048627540000000004 2023-07-02T14:06:28.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317774+0000", 2023-07-02T14:06:28.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2008000000000003e-05 2023-07-02T14:06:28.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317806+0000", 2023-07-02T14:06:28.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050055689999999996 2023-07-02T14:06:28.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322812+0000", 2023-07-02T14:06:28.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3442999999999997e-05 2023-07-02T14:06:28.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322855+0000", 2023-07-02T14:06:28.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006472813 2023-07-02T14:06:28.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329328+0000", 2023-07-02T14:06:28.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0128e-05 2023-07-02T14:06:28.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329358+0000", 2023-07-02T14:06:28.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062156490000000002 2023-07-02T14:06:28.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335574+0000", 2023-07-02T14:06:28.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2357999999999997e-05 2023-07-02T14:06:28.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335606+0000", 2023-07-02T14:06:28.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098318409999999992 2023-07-02T14:06:28.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345438+0000", 2023-07-02T14:06:28.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1174000000000007e-05 2023-07-02T14:06:28.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345509+0000", 2023-07-02T14:06:28.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004669851 2023-07-02T14:06:28.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350179+0000", 2023-07-02T14:06:28.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1559000000000001e-05 2023-07-02T14:06:28.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350211+0000", 2023-07-02T14:06:28.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041865740000000002 2023-07-02T14:06:28.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354397+0000", 2023-07-02T14:06:28.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9850000000000001e-05 2023-07-02T14:06:28.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354427+0000", 2023-07-02T14:06:28.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071608449999999999 2023-07-02T14:06:28.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361588+0000", 2023-07-02T14:06:28.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8346000000000002e-05 2023-07-02T14:06:28.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361626+0000", 2023-07-02T14:06:28.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073175009999999997 2023-07-02T14:06:28.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368944+0000", 2023-07-02T14:06:28.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2879000000000005e-05 2023-07-02T14:06:28.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369017+0000", 2023-07-02T14:06:28.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056850690000000001 2023-07-02T14:06:28.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374702+0000", 2023-07-02T14:06:28.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7693e-05 2023-07-02T14:06:28.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374739+0000", 2023-07-02T14:06:28.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076925149999999996 2023-07-02T14:06:28.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382432+0000", 2023-07-02T14:06:28.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3259e-05 2023-07-02T14:06:28.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382475+0000", 2023-07-02T14:06:28.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041005110000000003 2023-07-02T14:06:28.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386576+0000", 2023-07-02T14:06:28.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2074999999999999e-05 2023-07-02T14:06:28.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:28.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386618+0000", 2023-07-02T14:06:28.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000233692 2023-07-02T14:06:28.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:28.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386851+0000", 2023-07-02T14:06:28.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024022269999999998 2023-07-02T14:06:28.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:28.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.389254+0000", 2023-07-02T14:06:28.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058506469999999996 2023-07-02T14:06:28.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:28.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395104+0000", 2023-07-02T14:06:28.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8807000000000003e-05 2023-07-02T14:06:28.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:28.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395143+0000", 2023-07-02T14:06:28.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9616000000000001e-05 2023-07-02T14:06:28.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:28.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395193+0000", 2023-07-02T14:06:28.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.146184591 2023-07-02T14:06:28.504 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.504 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.504 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.504 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18560 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:28.505 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:28.505 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 23.069620233999999, 2023-07-02T14:06:28.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.15647551900000001, 2023-07-02T14:06:28.505 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:28.505 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:28.505 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:28.505 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:28.505 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:28.506 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18560 2023-07-02T14:06:28.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.506 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:28.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:28.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:28.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:28.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:28.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:28.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:28.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:28.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249026+0000", 2023-07-02T14:06:28.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3419999999999999e-06 2023-07-02T14:06:28.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:28.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249030+0000", 2023-07-02T14:06:28.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7869999999999998e-06 2023-07-02T14:06:28.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:28.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249035+0000", 2023-07-02T14:06:28.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9639999999999997e-06 2023-07-02T14:06:28.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:28.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249039+0000", 2023-07-02T14:06:28.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00088477599999999999 2023-07-02T14:06:28.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249924+0000", 2023-07-02T14:06:28.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7333999999999999e-05 2023-07-02T14:06:28.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249962+0000", 2023-07-02T14:06:28.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066323409999999999 2023-07-02T14:06:28.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256594+0000", 2023-07-02T14:06:28.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2595999999999998e-05 2023-07-02T14:06:28.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256627+0000", 2023-07-02T14:06:28.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012449894 2023-07-02T14:06:28.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269076+0000", 2023-07-02T14:06:28.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3942999999999997e-05 2023-07-02T14:06:28.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269110+0000", 2023-07-02T14:06:28.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010923884 2023-07-02T14:06:28.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280034+0000", 2023-07-02T14:06:28.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7774000000000001e-05 2023-07-02T14:06:28.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280072+0000", 2023-07-02T14:06:28.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011850027000000001 2023-07-02T14:06:28.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291922+0000", 2023-07-02T14:06:28.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2128000000000001e-05 2023-07-02T14:06:28.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291954+0000", 2023-07-02T14:06:28.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080261150000000003 2023-07-02T14:06:28.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299980+0000", 2023-07-02T14:06:28.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6322999999999998e-05 2023-07-02T14:06:28.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300017+0000", 2023-07-02T14:06:28.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072683039999999997 2023-07-02T14:06:28.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307285+0000", 2023-07-02T14:06:28.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3309000000000002e-05 2023-07-02T14:06:28.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307318+0000", 2023-07-02T14:06:28.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056178649999999997 2023-07-02T14:06:28.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312936+0000", 2023-07-02T14:06:28.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1866000000000003e-05 2023-07-02T14:06:28.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312968+0000", 2023-07-02T14:06:28.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048789970000000004 2023-07-02T14:06:28.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317847+0000", 2023-07-02T14:06:28.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0223999999999996e-05 2023-07-02T14:06:28.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317927+0000", 2023-07-02T14:06:28.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049543649999999996 2023-07-02T14:06:28.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322882+0000", 2023-07-02T14:06:28.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1399000000000002e-05 2023-07-02T14:06:28.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322913+0000", 2023-07-02T14:06:28.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006534214 2023-07-02T14:06:28.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329447+0000", 2023-07-02T14:06:28.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9649999999999998e-05 2023-07-02T14:06:28.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329527+0000", 2023-07-02T14:06:28.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061026839999999997 2023-07-02T14:06:28.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335629+0000", 2023-07-02T14:06:28.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9611000000000002e-05 2023-07-02T14:06:28.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335659+0000", 2023-07-02T14:06:28.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098851549999999996 2023-07-02T14:06:28.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345544+0000", 2023-07-02T14:06:28.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0587999999999999e-05 2023-07-02T14:06:28.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345575+0000", 2023-07-02T14:06:28.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046790859999999998 2023-07-02T14:06:28.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350254+0000", 2023-07-02T14:06:28.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0284000000000002e-05 2023-07-02T14:06:28.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350304+0000", 2023-07-02T14:06:28.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041560540000000002 2023-07-02T14:06:28.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354460+0000", 2023-07-02T14:06:28.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2427000000000004e-05 2023-07-02T14:06:28.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354523+0000", 2023-07-02T14:06:28.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071268770000000002 2023-07-02T14:06:28.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361650+0000", 2023-07-02T14:06:28.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0026999999999999e-05 2023-07-02T14:06:28.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361680+0000", 2023-07-02T14:06:28.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073733080000000003 2023-07-02T14:06:28.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369053+0000", 2023-07-02T14:06:28.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7635999999999997e-05 2023-07-02T14:06:28.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369091+0000", 2023-07-02T14:06:28.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056908059999999996 2023-07-02T14:06:28.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374781+0000", 2023-07-02T14:06:28.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4656000000000002e-05 2023-07-02T14:06:28.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374816+0000", 2023-07-02T14:06:28.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077037809999999998 2023-07-02T14:06:28.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382520+0000", 2023-07-02T14:06:28.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7858e-05 2023-07-02T14:06:28.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382588+0000", 2023-07-02T14:06:28.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057884019999999998 2023-07-02T14:06:28.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388376+0000", 2023-07-02T14:06:28.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4831000000000001e-05 2023-07-02T14:06:28.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388461+0000", 2023-07-02T14:06:28.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067708480000000003 2023-07-02T14:06:28.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395232+0000", 2023-07-02T14:06:28.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7948000000000002e-05 2023-07-02T14:06:28.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:28.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395290+0000", 2023-07-02T14:06:28.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00027672599999999998 2023-07-02T14:06:28.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:28.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395566+0000", 2023-07-02T14:06:28.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025705569999999998 2023-07-02T14:06:28.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:28.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.398137+0000", 2023-07-02T14:06:28.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072257099999999998 2023-07-02T14:06:28.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:28.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405363+0000", 2023-07-02T14:06:28.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3887999999999998e-05 2023-07-02T14:06:28.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:28.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405397+0000", 2023-07-02T14:06:28.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000105769 2023-07-02T14:06:28.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:28.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405502+0000", 2023-07-02T14:06:28.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.15647551900000001 2023-07-02T14:06:28.540 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.540 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.540 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.540 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18561 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:28.541 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:28.541 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 23.069599870000001, 2023-07-02T14:06:28.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744, 2023-07-02T14:06:28.541 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:28.541 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:28.541 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:28.541 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:28.542 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:28.542 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18561 2023-07-02T14:06:28.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.542 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:28.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:28.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:28.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:28.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:28.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:28.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:28.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:28.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:28.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.563e-06 2023-07-02T14:06:28.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:28.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249050+0000", 2023-07-02T14:06:28.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4500000000000001e-06 2023-07-02T14:06:28.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:28.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249057+0000", 2023-07-02T14:06:28.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1860000000000002e-06 2023-07-02T14:06:28.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:28.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249061+0000", 2023-07-02T14:06:28.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00091153399999999998 2023-07-02T14:06:28.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249972+0000", 2023-07-02T14:06:28.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7156999999999997e-05 2023-07-02T14:06:28.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.250010+0000", 2023-07-02T14:06:28.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006641816 2023-07-02T14:06:28.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256651+0000", 2023-07-02T14:06:28.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1968999999999998e-05 2023-07-02T14:06:28.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256683+0000", 2023-07-02T14:06:28.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012463952 2023-07-02T14:06:28.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269147+0000", 2023-07-02T14:06:28.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2855000000000006e-05 2023-07-02T14:06:28.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269220+0000", 2023-07-02T14:06:28.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010870418999999999 2023-07-02T14:06:28.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280091+0000", 2023-07-02T14:06:28.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1921999999999997e-05 2023-07-02T14:06:28.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280123+0000", 2023-07-02T14:06:28.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011861682 2023-07-02T14:06:28.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291984+0000", 2023-07-02T14:06:28.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5259000000000006e-05 2023-07-02T14:06:28.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.292059+0000", 2023-07-02T14:06:28.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079709359999999996 2023-07-02T14:06:28.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300030+0000", 2023-07-02T14:06:28.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0729999999999998e-05 2023-07-02T14:06:28.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300071+0000", 2023-07-02T14:06:28.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072785690000000004 2023-07-02T14:06:28.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307350+0000", 2023-07-02T14:06:28.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1948999999999999e-05 2023-07-02T14:06:28.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307412+0000", 2023-07-02T14:06:28.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055785779999999998 2023-07-02T14:06:28.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312990+0000", 2023-07-02T14:06:28.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0957000000000002e-05 2023-07-02T14:06:28.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.313021+0000", 2023-07-02T14:06:28.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049246400000000001 2023-07-02T14:06:28.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317946+0000", 2023-07-02T14:06:28.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9346e-05 2023-07-02T14:06:28.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317975+0000", 2023-07-02T14:06:28.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049649469999999999 2023-07-02T14:06:28.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322940+0000", 2023-07-02T14:06:28.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9017000000000002e-05 2023-07-02T14:06:28.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322989+0000", 2023-07-02T14:06:28.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065907029999999998 2023-07-02T14:06:28.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329580+0000", 2023-07-02T14:06:28.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5879999999999999e-05 2023-07-02T14:06:28.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329656+0000", 2023-07-02T14:06:28.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060166960000000002 2023-07-02T14:06:28.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335672+0000", 2023-07-02T14:06:28.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8963e-05 2023-07-02T14:06:28.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335701+0000", 2023-07-02T14:06:28.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098989560000000004 2023-07-02T14:06:28.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345600+0000", 2023-07-02T14:06:28.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5395999999999995e-05 2023-07-02T14:06:28.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345666+0000", 2023-07-02T14:06:28.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046589530000000004 2023-07-02T14:06:28.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350325+0000", 2023-07-02T14:06:28.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2459000000000002e-05 2023-07-02T14:06:28.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350357+0000", 2023-07-02T14:06:28.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042054290000000001 2023-07-02T14:06:28.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354563+0000", 2023-07-02T14:06:28.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5612999999999999e-05 2023-07-02T14:06:28.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354598+0000", 2023-07-02T14:06:28.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070967449999999998 2023-07-02T14:06:28.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361695+0000", 2023-07-02T14:06:28.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9314000000000001e-05 2023-07-02T14:06:28.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361724+0000", 2023-07-02T14:06:28.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007391028 2023-07-02T14:06:28.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369115+0000", 2023-07-02T14:06:28.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9163000000000002e-05 2023-07-02T14:06:28.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369184+0000", 2023-07-02T14:06:28.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056532500000000003 2023-07-02T14:06:28.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374838+0000", 2023-07-02T14:06:28.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5297999999999999e-05 2023-07-02T14:06:28.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374883+0000", 2023-07-02T14:06:28.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077292189999999998 2023-07-02T14:06:28.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382612+0000", 2023-07-02T14:06:28.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1102000000000001e-05 2023-07-02T14:06:28.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382653+0000", 2023-07-02T14:06:28.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058515989999999999 2023-07-02T14:06:28.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388505+0000", 2023-07-02T14:06:28.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9196000000000006e-05 2023-07-02T14:06:28.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388574+0000", 2023-07-02T14:06:28.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084245339999999991 2023-07-02T14:06:28.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.396999+0000", 2023-07-02T14:06:28.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0596999999999998e-05 2023-07-02T14:06:28.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:28.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.397039+0000", 2023-07-02T14:06:28.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084911339999999991 2023-07-02T14:06:28.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405530+0000", 2023-07-02T14:06:28.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7555000000000001e-05 2023-07-02T14:06:28.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:28.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405578+0000", 2023-07-02T14:06:28.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000229607 2023-07-02T14:06:28.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:28.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405808+0000", 2023-07-02T14:06:28.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026919940000000001 2023-07-02T14:06:28.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:28.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.408500+0000", 2023-07-02T14:06:28.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053065059999999999 2023-07-02T14:06:28.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:28.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413806+0000", 2023-07-02T14:06:28.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0490000000000001e-05 2023-07-02T14:06:28.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:28.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413837+0000", 2023-07-02T14:06:28.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000100386 2023-07-02T14:06:28.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:28.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413937+0000", 2023-07-02T14:06:28.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744 2023-07-02T14:06:28.577 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.577 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.577 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.578 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20768 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch ping cookie 140544751726928 gen 52] snapc 0=[] ondisk+write+known_if_redirected e183)", 2023-07-02T14:06:28.578 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:28.578 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.0962326020000006, 2023-07-02T14:06:28.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3751256970000001, 2023-07-02T14:06:28.578 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:28.579 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:28.579 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:28.579 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:28.579 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:28.579 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20768 2023-07-02T14:06:28.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.579 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:28.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:28.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:28.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:28.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:28.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:28.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:28.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:28.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:28.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7829999999999998e-06 2023-07-02T14:06:28.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:28.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222419+0000", 2023-07-02T14:06:28.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1380000000000004e-06 2023-07-02T14:06:28.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:28.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222425+0000", 2023-07-02T14:06:28.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2519999999999996e-06 2023-07-02T14:06:28.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:28.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222431+0000", 2023-07-02T14:06:28.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2157999999999994e-05 2023-07-02T14:06:28.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222523+0000", 2023-07-02T14:06:28.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5147000000000001e-05 2023-07-02T14:06:28.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:28.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222538+0000", 2023-07-02T14:06:28.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30105419700000002 2023-07-02T14:06:28.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523592+0000", 2023-07-02T14:06:28.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2316e-05 2023-07-02T14:06:28.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:28.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523605+0000", 2023-07-02T14:06:28.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.073658953 2023-07-02T14:06:28.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597264+0000", 2023-07-02T14:06:28.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000213415 2023-07-02T14:06:28.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:28.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597477+0000", 2023-07-02T14:06:28.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3034999999999995e-05 2023-07-02T14:06:28.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:28.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597540+0000", 2023-07-02T14:06:28.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3751256970000001 2023-07-02T14:06:28.588 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.588 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.588 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.588 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20769 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 53] snapc 0=[] ondisk+write+known_if_redirected e184)", 2023-07-02T14:06:28.588 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:28.589 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 9.0958701469999994, 2023-07-02T14:06:28.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.380884139, 2023-07-02T14:06:28.589 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:28.589 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:28.589 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:28.589 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:28.589 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:28.589 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20769 2023-07-02T14:06:28.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.590 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:28.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:28.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:28.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:28.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:28.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:28.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:28.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:28.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222776+0000", 2023-07-02T14:06:28.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9710000000000003e-06 2023-07-02T14:06:28.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:28.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222781+0000", 2023-07-02T14:06:28.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9899999999999992e-06 2023-07-02T14:06:28.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:28.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222791+0000", 2023-07-02T14:06:28.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4240000000000003e-06 2023-07-02T14:06:28.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:28.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222795+0000", 2023-07-02T14:06:28.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2366000000000001e-05 2023-07-02T14:06:28.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222828+0000", 2023-07-02T14:06:28.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4790000000000001e-06 2023-07-02T14:06:28.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:28.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222834+0000", 2023-07-02T14:06:28.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30084589299999998 2023-07-02T14:06:28.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523680+0000", 2023-07-02T14:06:28.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.87e-06 2023-07-02T14:06:28.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:28.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523685+0000", 2023-07-02T14:06:28.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.074111042 2023-07-02T14:06:28.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597796+0000", 2023-07-02T14:06:28.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0411000000000001e-05 2023-07-02T14:06:28.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:28.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597856+0000", 2023-07-02T14:06:28.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058045340000000001 2023-07-02T14:06:28.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:28.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.603661+0000", 2023-07-02T14:06:28.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.380884139 2023-07-02T14:06:28.598 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.598 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.598 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.599 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20770 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 54] snapc 0=[] ondisk+write+known_if_redirected e185)", 2023-07-02T14:06:28.599 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:28.599 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 8.7946530369999998, 2023-07-02T14:06:28.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0799819349999999, 2023-07-02T14:06:28.599 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:28.599 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:28.599 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:28.600 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:28.600 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:28.600 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20770 2023-07-02T14:06:28.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.600 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:28.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:28.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:28.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:28.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:28.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:28.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:28.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:28.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523992+0000", 2023-07-02T14:06:28.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6720000000000004e-06 2023-07-02T14:06:28.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:28.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524000+0000", 2023-07-02T14:06:28.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3711e-05 2023-07-02T14:06:28.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:28.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524014+0000", 2023-07-02T14:06:28.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7849999999999992e-06 2023-07-02T14:06:28.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.604 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:28.604 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524022+0000", 2023-07-02T14:06:28.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2366000000000001e-05 2023-07-02T14:06:28.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.604 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.604 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524054+0000", 2023-07-02T14:06:28.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8169999999999998e-06 2023-07-02T14:06:28.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:28.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524059+0000", 2023-07-02T14:06:28.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0748338850000001 2023-07-02T14:06:28.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.598893+0000", 2023-07-02T14:06:28.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116968 2023-07-02T14:06:28.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:28.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.599010+0000", 2023-07-02T14:06:28.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049663370000000004 2023-07-02T14:06:28.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:28.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.603976+0000", 2023-07-02T14:06:28.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0799819349999999 2023-07-02T14:06:28.607 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.607 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.607 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.608 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20833 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 55] snapc 0=[] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:28.608 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:06:28.608 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.7865337320000001, 2023-07-02T14:06:28.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.029349206, 2023-07-02T14:06:28.608 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:28.609 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:28.609 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:28.609 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:28.609 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:28.609 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20833 2023-07-02T14:06:28.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.609 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:28.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:28.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:06:28.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:28.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:28.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:06:28.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:28.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:28.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532112+0000", 2023-07-02T14:06:28.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7590000000000003e-06 2023-07-02T14:06:28.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:28.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532120+0000", 2023-07-02T14:06:28.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8130000000000001e-05 2023-07-02T14:06:28.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:28.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532139+0000", 2023-07-02T14:06:28.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3299999999999999e-06 2023-07-02T14:06:28.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:28.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532147+0000", 2023-07-02T14:06:28.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9679999999999998e-05 2023-07-02T14:06:28.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532207+0000", 2023-07-02T14:06:28.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0648e-05 2023-07-02T14:06:28.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op must wait for map", 2023-07-02T14:06:28.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532217+0000", 2023-07-02T14:06:28.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002082455 2023-07-02T14:06:28.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.615 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.534300+0000", 2023-07-02T14:06:28.615 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0710000000000001e-05 2023-07-02T14:06:28.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:28.615 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.534310+0000", 2023-07-02T14:06:28.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.013227235 2023-07-02T14:06:28.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.616 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.616 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.616 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.547538+0000", 2023-07-02T14:06:28.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021904300000000001 2023-07-02T14:06:28.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.616 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:28.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.547757+0000", 2023-07-02T14:06:28.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013705784 2023-07-02T14:06:28.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:28.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561462+0000", 2023-07-02T14:06:28.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.029349206 2023-07-02T14:06:28.618 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.618 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.618 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.618 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20865 2.7 2:ff0f5f99:::rbd_data.125de1449605.0000000000000006:head [write 131072~8192] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:28.618 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:06:28.619 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.0419106600000001, 2023-07-02T14:06:28.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.288115762, 2023-07-02T14:06:28.619 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:28.619 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:28.619 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:28.619 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:28.619 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:28.619 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20865 2023-07-02T14:06:28.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.620 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:28.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:28.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:06:28.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:28.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:28.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:06:28.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:28.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:28.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276735+0000", 2023-07-02T14:06:28.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3273e-05 2023-07-02T14:06:28.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:28.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276748+0000", 2023-07-02T14:06:28.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3895e-05 2023-07-02T14:06:28.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:28.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276762+0000", 2023-07-02T14:06:28.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.896e-06 2023-07-02T14:06:28.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:28.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276768+0000", 2023-07-02T14:06:28.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5850999999999999e-05 2023-07-02T14:06:28.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276814+0000", 2023-07-02T14:06:28.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5290000000000003e-06 2023-07-02T14:06:28.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:28.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276821+0000", 2023-07-02T14:06:28.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.27225312600000001 2023-07-02T14:06:28.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.625 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549074+0000", 2023-07-02T14:06:28.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000151223 2023-07-02T14:06:28.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:28.626 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549226+0000", 2023-07-02T14:06:28.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024865400000000001 2023-07-02T14:06:28.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:28.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549474+0000", 2023-07-02T14:06:28.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012039185000000001 2023-07-02T14:06:28.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:28.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561513+0000", 2023-07-02T14:06:28.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003221327 2023-07-02T14:06:28.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:28.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564735+0000", 2023-07-02T14:06:28.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2026000000000005e-05 2023-07-02T14:06:28.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:28.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564807+0000", 2023-07-02T14:06:28.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5303000000000003e-05 2023-07-02T14:06:28.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:28.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564852+0000", 2023-07-02T14:06:28.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.288115762 2023-07-02T14:06:28.629 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.630 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.630 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.630 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20866 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 131072~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:28.630 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:06:28.630 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.0362173389999998, 2023-07-02T14:06:28.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.28773701200000001, 2023-07-02T14:06:28.631 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:28.631 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:28.631 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:28.631 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:28.631 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:28.631 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20866 2023-07-02T14:06:28.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.631 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:28.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:28.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:06:28.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:28.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:28.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:06:28.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:28.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:28.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282428+0000", 2023-07-02T14:06:28.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8469999999999998e-05 2023-07-02T14:06:28.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:28.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282517+0000", 2023-07-02T14:06:28.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2974e-05 2023-07-02T14:06:28.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:28.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282530+0000", 2023-07-02T14:06:28.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4530000000000001e-06 2023-07-02T14:06:28.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:28.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282535+0000", 2023-07-02T14:06:28.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1926999999999999e-05 2023-07-02T14:06:28.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282587+0000", 2023-07-02T14:06:28.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3083e-05 2023-07-02T14:06:28.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:28.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282600+0000", 2023-07-02T14:06:28.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.270692971 2023-07-02T14:06:28.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553293+0000", 2023-07-02T14:06:28.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017065599999999999 2023-07-02T14:06:28.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:28.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553464+0000", 2023-07-02T14:06:28.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026504399999999998 2023-07-02T14:06:28.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:28.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553729+0000", 2023-07-02T14:06:28.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011173008 2023-07-02T14:06:28.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:28.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564902+0000", 2023-07-02T14:06:28.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051242600000000003 2023-07-02T14:06:28.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:28.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570026+0000", 2023-07-02T14:06:28.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8098999999999999e-05 2023-07-02T14:06:28.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:28.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570104+0000", 2023-07-02T14:06:28.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2323000000000006e-05 2023-07-02T14:06:28.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:28.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570167+0000", 2023-07-02T14:06:28.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.28773701200000001 2023-07-02T14:06:28.641 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.641 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.641 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.642 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20875 2.7 2:e3210266:::rbd_data.125de1449605.00000000000001a3:head [write 0~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:28.642 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:06:28.642 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.9971665230000002, 2023-07-02T14:06:28.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.25297039599999999, 2023-07-02T14:06:28.642 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:28.642 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:28.643 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:28.643 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:28.643 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:28.643 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20875 2023-07-02T14:06:28.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.643 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:28.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:28.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:06:28.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:28.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:28.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:06:28.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:28.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:28.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321479+0000", 2023-07-02T14:06:28.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015194399999999999 2023-07-02T14:06:28.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:28.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321631+0000", 2023-07-02T14:06:28.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4236999999999999e-05 2023-07-02T14:06:28.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:28.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321645+0000", 2023-07-02T14:06:28.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2160000000000003e-06 2023-07-02T14:06:28.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:28.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321651+0000", 2023-07-02T14:06:28.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3609999999999998e-05 2023-07-02T14:06:28.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321695+0000", 2023-07-02T14:06:28.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1989999999999995e-06 2023-07-02T14:06:28.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:28.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321704+0000", 2023-07-02T14:06:28.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.23542881500000001 2023-07-02T14:06:28.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557133+0000", 2023-07-02T14:06:28.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017359600000000001 2023-07-02T14:06:28.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:28.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557307+0000", 2023-07-02T14:06:28.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00023367599999999999 2023-07-02T14:06:28.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:28.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557540+0000", 2023-07-02T14:06:28.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016744963000000002 2023-07-02T14:06:28.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:28.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574285+0000", 2023-07-02T14:06:28.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1330999999999999e-05 2023-07-02T14:06:28.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:28.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574346+0000", 2023-07-02T14:06:28.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9750000000000002e-05 2023-07-02T14:06:28.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:28.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574406+0000", 2023-07-02T14:06:28.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4647999999999997e-05 2023-07-02T14:06:28.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:28.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574451+0000", 2023-07-02T14:06:28.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.25297039599999999 2023-07-02T14:06:28.653 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.653 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.653 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.653 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20915 2.7 2:e6e64067:::rbd_data.125de1449605.00000000000001a9:head [write 0~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:28.654 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:06:28.654 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.9339127280000001, 2023-07-02T14:06:28.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.19966078500000001, 2023-07-02T14:06:28.654 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:28.654 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:28.654 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:28.654 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:28.655 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:28.655 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20915 2023-07-02T14:06:28.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.655 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:28.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:28.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:06:28.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:28.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:28.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:06:28.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:28.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:28.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384732+0000", 2023-07-02T14:06:28.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9439999999999999e-05 2023-07-02T14:06:28.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:28.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384802+0000", 2023-07-02T14:06:28.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7572999999999999e-05 2023-07-02T14:06:28.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:28.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384819+0000", 2023-07-02T14:06:28.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1670000000000004e-06 2023-07-02T14:06:28.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:28.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384825+0000", 2023-07-02T14:06:28.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6471999999999998e-05 2023-07-02T14:06:28.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384872+0000", 2023-07-02T14:06:28.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0028e-05 2023-07-02T14:06:28.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:28.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384882+0000", 2023-07-02T14:06:28.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.17674214099999999 2023-07-02T14:06:28.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561624+0000", 2023-07-02T14:06:28.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000135809 2023-07-02T14:06:28.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:28.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561760+0000", 2023-07-02T14:06:28.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020468 2023-07-02T14:06:28.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:28.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561965+0000", 2023-07-02T14:06:28.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012498198 2023-07-02T14:06:28.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:28.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574463+0000", 2023-07-02T14:06:28.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098197509999999998 2023-07-02T14:06:28.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:28.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584283+0000", 2023-07-02T14:06:28.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2056000000000004e-05 2023-07-02T14:06:28.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:28.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584345+0000", 2023-07-02T14:06:28.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0513000000000002e-05 2023-07-02T14:06:28.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:28.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584395+0000", 2023-07-02T14:06:28.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.19966078500000001 2023-07-02T14:06:28.665 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.665 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.665 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.665 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20948 2.7 2:ff0f5f99:::rbd_data.125de1449605.0000000000000006:head [write 139264~2220032] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:28.665 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.450499+0000", 2023-07-02T14:06:28.666 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.8681480009999998, 2023-07-02T14:06:28.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13400999399999999, 2023-07-02T14:06:28.666 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:28.666 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:28.666 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:28.666 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:28.666 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:28.666 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20948 2023-07-02T14:06:28.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.667 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:28.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:28.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.450499+0000", 2023-07-02T14:06:28.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:28.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:28.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.450499+0000", 2023-07-02T14:06:28.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:28.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:28.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.450497+0000", 2023-07-02T14:06:28.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043010360000000003 2023-07-02T14:06:28.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:28.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454798+0000", 2023-07-02T14:06:28.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8073000000000001e-05 2023-07-02T14:06:28.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:28.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454816+0000", 2023-07-02T14:06:28.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3769999999999996e-06 2023-07-02T14:06:28.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:28.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454825+0000", 2023-07-02T14:06:28.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2797999999999998e-05 2023-07-02T14:06:28.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454878+0000", 2023-07-02T14:06:28.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0020999999999999e-05 2023-07-02T14:06:28.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:28.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454888+0000", 2023-07-02T14:06:28.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.110043313 2023-07-02T14:06:28.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564931+0000", 2023-07-02T14:06:28.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7185e-05 2023-07-02T14:06:28.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:28.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.565008+0000", 2023-07-02T14:06:28.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014058199999999999 2023-07-02T14:06:28.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:28.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.565149+0000", 2023-07-02T14:06:28.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093221559999999998 2023-07-02T14:06:28.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:28.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574471+0000", 2023-07-02T14:06:28.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099660410000000001 2023-07-02T14:06:28.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:28.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584437+0000", 2023-07-02T14:06:28.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2805000000000001e-05 2023-07-02T14:06:28.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:28.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584480+0000", 2023-07-02T14:06:28.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9393000000000001e-05 2023-07-02T14:06:28.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:28.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584509+0000", 2023-07-02T14:06:28.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13400999399999999 2023-07-02T14:06:28.676 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.676 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.677 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.677 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20949 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 196608~2162688] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:28.677 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.454838+0000", 2023-07-02T14:06:28.677 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.8638088420000001, 2023-07-02T14:06:28.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12983145400000001, 2023-07-02T14:06:28.678 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:28.678 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:28.678 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:28.678 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:28.678 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:28.678 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20949 2023-07-02T14:06:28.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.678 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:28.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:28.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454838+0000", 2023-07-02T14:06:28.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:28.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:28.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454838+0000", 2023-07-02T14:06:28.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:28.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:28.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454837+0000", 2023-07-02T14:06:28.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003772821 2023-07-02T14:06:28.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:28.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458610+0000", 2023-07-02T14:06:28.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3013999999999999e-05 2023-07-02T14:06:28.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:28.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458623+0000", 2023-07-02T14:06:28.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2789999999999998e-06 2023-07-02T14:06:28.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:28.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458628+0000", 2023-07-02T14:06:28.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7630999999999998e-05 2023-07-02T14:06:28.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458676+0000", 2023-07-02T14:06:28.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1579999999999994e-06 2023-07-02T14:06:28.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:28.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458685+0000", 2023-07-02T14:06:28.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.111536444 2023-07-02T14:06:28.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570221+0000", 2023-07-02T14:06:28.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000120844 2023-07-02T14:06:28.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:28.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570342+0000", 2023-07-02T14:06:28.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000149965 2023-07-02T14:06:28.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:28.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570492+0000", 2023-07-02T14:06:28.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014027843999999999 2023-07-02T14:06:28.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:28.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584520+0000", 2023-07-02T14:06:28.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8446000000000003e-05 2023-07-02T14:06:28.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:28.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584568+0000", 2023-07-02T14:06:28.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6771999999999998e-05 2023-07-02T14:06:28.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:28.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584625+0000", 2023-07-02T14:06:28.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4401999999999999e-05 2023-07-02T14:06:28.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:28.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584670+0000", 2023-07-02T14:06:28.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12983145400000001 2023-07-02T14:06:28.688 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.688 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.688 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.689 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20972 2.7 2:e3210266:::rbd_data.125de1449605.00000000000001a3:head [write 65536~4128768] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:28.689 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.463174+0000", 2023-07-02T14:06:28.689 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 2.8554730020000001, 2023-07-02T14:06:28.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12719914800000001, 2023-07-02T14:06:28.689 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:28.689 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:28.689 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:28.690 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:28.690 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:28.690 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20972 2023-07-02T14:06:28.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.690 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:28.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:28.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.463174+0000", 2023-07-02T14:06:28.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:28.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:28.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.463174+0000", 2023-07-02T14:06:28.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:28.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:28.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.463173+0000", 2023-07-02T14:06:28.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069882069999999998 2023-07-02T14:06:28.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:28.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470161+0000", 2023-07-02T14:06:28.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0103999999999999e-05 2023-07-02T14:06:28.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:28.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470181+0000", 2023-07-02T14:06:28.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:06:28.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:28.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470189+0000", 2023-07-02T14:06:28.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0970999999999994e-05 2023-07-02T14:06:28.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470270+0000", 2023-07-02T14:06:28.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4389000000000001e-05 2023-07-02T14:06:28.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:28.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470284+0000", 2023-07-02T14:06:28.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.10421862799999999 2023-07-02T14:06:28.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:28.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574503+0000", 2023-07-02T14:06:28.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2457e-05 2023-07-02T14:06:28.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:28.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574585+0000", 2023-07-02T14:06:28.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013870899999999999 2023-07-02T14:06:28.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:28.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574724+0000", 2023-07-02T14:06:28.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012886246000000001 2023-07-02T14:06:28.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:28.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.587610+0000", 2023-07-02T14:06:28.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026883599999999999 2023-07-02T14:06:28.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:28.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.590298+0000", 2023-07-02T14:06:28.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1850000000000001e-05 2023-07-02T14:06:28.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:28.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.590340+0000", 2023-07-02T14:06:28.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2901999999999998e-05 2023-07-02T14:06:28.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:28.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:28.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:28.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.590373+0000", 2023-07-02T14:06:28.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12719914800000001 2023-07-02T14:06:28.700 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.700 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.700 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.700 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:28.700 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:28.700 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:28.701 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:28.702 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:28.703 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:06:28.703 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:28.701+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:06:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:06:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:06:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:06:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:06:28.704 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:06:28.705 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.4 to 1 (10000) 2023-07-02T14:06:28.706 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-07-02T14:06:28.706 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:06:28.803 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:28.802+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:28.873 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:28.873 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:28.873 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:28.873 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:28.884 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:06:28.927 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:28.926+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:29.004 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:29.003+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:29.008 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:29.008 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:29.009 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:06:29.009 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:06:29.009 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:29.019 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:06:29.049 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:06:29.051 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:06:29.063 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2023-07-02T14:06:28.558999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9472296739,"num_objects":2816,"num_object_clones":2325,"num_object_copies":5632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":275,"num_objects_unfound":0,"num_objects_dirty":2816,"num_whiteouts":0,"num_read":65946,"num_read_kb":12384683,"num_write":22134,"num_write_kb":9195051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2165,"num_bytes_recovered":7440234273,"num_keys_recovered":178,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21867,"ondisk_log_size":21867,"up":18,"acting":17,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":20,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":20427012,"kb_used_data":20139804,"kb_used_omap":1114,"kb_used_meta":282021,"kb_avail":451432188,"statfs":{"total":483183820800,"available":462266560512,"internally_reserved":0,"allocated":20623159296,"data_stored":23760669307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1141261,"internal_metadata":288790003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":54542660,"num_objects":26,"num_object_clones":20,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":163,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":2211,"num_read_kb":46575,"num_write":90,"num_write_kb":53301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":216,"num_bytes_recovered":761573716,"num_keys_recovered":64,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.027215"},"pg_stats":[{"pgid":"2.7","version":"191'3091","reported_seq":7976,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-07-02T14:06:26.608992+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:06:26.608992+0000","last_peered":"2023-07-02T14:06:26.608992+0000","last_clean":"2023-07-02T14:06:26.608992+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:06:26.608992+0000","last_undegraded":"2023-07-02T14:06:26.608992+0000","last_fullsized":"2023-07-02T14:06:26.608992+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3091,"ondisk_log_size":3091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1276541072,"num_objects":376,"num_object_clones":311,"num_object_copies":752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":41821,"num_read_kb":1595255,"num_write":2612,"num_write_kb":1240032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"189'2473","reported_seq":4112,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-07-02T14:06:25.271517+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:06:25.271517+0000","last_peered":"2023-07-02T14:06:25.271517+0000","last_clean":"2023-07-02T14:06:25.271517+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:06:25.271517+0000","last_undegraded":"2023-07-02T14:06:25.271517+0000","last_fullsized":"2023-07-02T14:06:25.271517+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2473,"ondisk_log_size":2473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1227764846,"num_objects":380,"num_object_clones":318,"num_object_copies":760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":380,"num_whiteouts":0,"num_read":1401,"num_read_kb":1671896,"num_write":2155,"num_write_kb":1191863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"190'2629","reported_seq":3194,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-07-02T14:06:26.555573+0000","last_change":"2023-07-02T14:05:57.093448+0000","last_active":"2023-07-02T14:06:26.555573+0000","last_peered":"2023-07-02T14:06:26.555573+0000","last_clean":"2023-07-02T14:06:26.555573+0000","last_became_active":"2023-07-02T14:05:57.089226+0000","last_became_peered":"2023-07-02T14:05:57.089226+0000","last_unstale":"2023-07-02T14:06:26.555573+0000","last_undegraded":"2023-07-02T14:06:26.555573+0000","last_fullsized":"2023-07-02T14:06:26.555573+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2629,"ondisk_log_size":2629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206334412,"num_objects":333,"num_object_clones":274,"num_object_copies":666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":489,"num_read_kb":1499543,"num_write":2355,"num_write_kb":1165651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":860565666,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"189'5157","reported_seq":6836,"reported_epoch":189,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-07-02T14:06:25.318100+0000","last_change":"2023-07-02T14:06:24.032319+0000","last_active":"2023-07-02T14:06:25.318100+0000","last_peered":"2023-07-02T14:06:25.318100+0000","last_clean":"2023-07-02T14:06:17.520145+0000","last_became_active":"2023-07-02T14:06:20.625414+0000","last_became_peered":"2023-07-02T14:06:20.625414+0000","last_unstale":"2023-07-02T14:06:25.318100+0000","last_undegraded":"2023-07-02T14:06:20.540308+0000","last_fullsized":"2023-07-02T14:06:20.528554+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":5157,"ondisk_log_size":5157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1303791042,"num_objects":381,"num_object_clones":318,"num_object_copies":762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":275,"num_objects_unfound":0,"num_objects_dirty":381,"num_whiteouts":0,"num_read":18034,"num_read_kb":1708239,"num_write":7599,"num_write_kb":1274150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":1210013154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1","objects":103},{"shards":"1,3","objects":275}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":119,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-07-02T14:06:25.259326+0000","last_change":"2023-07-02T14:05:53.410292+0000","last_active":"2023-07-02T14:06:25.259326+0000","last_peered":"2023-07-02T14:06:25.259326+0000","last_clean":"2023-07-02T14:06:25.259326+0000","last_became_active":"2023-07-02T14:05:53.409913+0000","last_became_peered":"2023-07-02T14:05:53.409913+0000","last_unstale":"2023-07-02T14:06:25.259326+0000","last_undegraded":"2023-07-02T14:06:25.259326+0000","last_fullsized":"2023-07-02T14:06:25.259326+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"189'2395","reported_seq":3595,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-07-02T14:06:25.242949+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:06:25.242949+0000","last_peered":"2023-07-02T14:06:25.242949+0000","last_clean":"2023-07-02T14:06:25.242949+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:06:25.242949+0000","last_undegraded":"2023-07-02T14:06:25.242949+0000","last_fullsized":"2023-07-02T14:06:25.242949+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2395,"ondisk_log_size":2395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1320547454,"num_objects":387,"num_object_clones":320,"num_object_copies":774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":387,"num_whiteouts":0,"num_read":1142,"num_read_kb":1676149,"num_write":2075,"num_write_kb":1283520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"189'2344","reported_seq":3041,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-07-02T14:06:25.245105+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:06:25.245105+0000","last_peered":"2023-07-02T14:06:25.245105+0000","last_clean":"2023-07-02T14:06:25.245105+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:06:25.245105+0000","last_undegraded":"2023-07-02T14:06:25.245105+0000","last_fullsized":"2023-07-02T14:06:25.245105+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2344,"ondisk_log_size":2344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1258112272,"num_objects":363,"num_object_clones":298,"num_object_copies":726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":715,"num_read_kb":1566165,"num_write":2046,"num_write_kb":1220500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"189'1857","reported_seq":3546,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-07-02T14:06:24.545222+0000","last_change":"2023-07-02T14:05:58.098047+0000","last_active":"2023-07-02T14:06:24.545222+0000","last_peered":"2023-07-02T14:06:24.545222+0000","last_clean":"2023-07-02T14:06:24.545222+0000","last_became_active":"2023-07-02T14:05:58.096421+0000","last_became_peered":"2023-07-02T14:05:58.096421+0000","last_unstale":"2023-07-02T14:06:24.545222+0000","last_undegraded":"2023-07-02T14:06:24.545222+0000","last_fullsized":"2023-07-02T14:06:24.545222+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1857,"ondisk_log_size":1857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":969606273,"num_objects":316,"num_object_clones":253,"num_object_copies":632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":1539,"num_read_kb":1421160,"num_write":1604,"num_write_kb":935617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":340635667,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"189'1921","reported_seq":2822,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-07-02T14:06:25.316705+0000","last_change":"2023-07-02T14:05:56.429093+0000","last_active":"2023-07-02T14:06:25.316705+0000","last_peered":"2023-07-02T14:06:25.316705+0000","last_clean":"2023-07-02T14:06:25.316705+0000","last_became_active":"2023-07-02T14:05:56.427016+0000","last_became_peered":"2023-07-02T14:05:56.427016+0000","last_unstale":"2023-07-02T14:06:25.316705+0000","last_undegraded":"2023-07-02T14:06:25.316705+0000","last_fullsized":"2023-07-02T14:06:25.316705+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1921,"ondisk_log_size":1921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":909599368,"num_objects":280,"num_object_clones":233,"num_object_copies":560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":280,"num_whiteouts":0,"num_read":805,"num_read_kb":1246276,"num_write":1688,"num_write_kb":883718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":271880192,"num_keys_recovered":4,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9472296739,"num_objects":2816,"num_object_clones":2325,"num_object_copies":5632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":275,"num_objects_unfound":0,"num_objects_dirty":2816,"num_whiteouts":0,"num_read":65946,"num_read_kb":12384683,"num_write":22134,"num_write_kb":9195051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2165,"num_bytes_recovered":7440234273,"num_keys_recovered":178,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20655927296,"data_stored":23833925674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":21867,"ondisk_log_size":21867,"up":16,"acting":15,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640314,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640314,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1639684,"kb_used_data":1583100,"kb_used_omap":0,"kb_used_meta":56576,"kb_avail":92732156,"statfs":{"total":96636764160,"available":94957727744,"internally_reserved":0,"allocated":1621094400,"data_stored":1827858700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":57933824},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.44,"15min":0.44400000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.64400000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47399999999999998,"15min":0.47999999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.69999999999999996,"5min":0.877,"15min":0.877},"last":0.40400000000000003}]},{"osd":1,"last update":"Sun Jul 2 14:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.44500000000000001,"15min":0.46600000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.436,"15min":0.44700000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.48699999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.44,"15min":0.45000000000000001},"min":{"1min":0.26500000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.47699999999999998,"15min":0.48399999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.38900000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.36599999999999999}]},{"osd":5,"last update":"Sun Jul 2 14:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.40899999999999997,"15min":0.45300000000000001},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.65400000000000003,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.44500000000000001,"15min":0.497},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.54800000000000004}]}]},{"osd":3,"up_from":133,"seq":571230650386,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4948560,"kb_used_data":4893000,"kb_used_omap":229,"kb_used_meta":55322,"kb_avail":89423280,"statfs":{"total":96636764160,"available":91569438720,"internally_reserved":0,"allocated":5010432000,"data_stored":5739736146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":56649950},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":1,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.63},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.54000000000000004}]},{"osd":4,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.59699999999999998}]},{"osd":5,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.67400000000000004}]}]},{"osd":2,"up_from":11,"seq":47244640314,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3773864,"kb_used_data":3731168,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":90597976,"statfs":{"total":96636764160,"available":92772327424,"internally_reserved":0,"allocated":3820716032,"data_stored":4365717830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.29399999999999998,"15min":0.29599999999999999},"min":{"1min":0.218,"5min":0.182,"15min":0.182},"max":{"1min":0.378,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.317,"5min":0.315,"15min":0.314},"min":{"1min":0.23000000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.39200000000000002}]},{"osd":1,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.38800000000000001,"15min":0.36199999999999999},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.2720000000000002,"5min":4.2720000000000002,"15min":4.2720000000000002},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.32300000000000001,"15min":0.32000000000000001},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.34699999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2,"5min":2,"15min":2},"last":0.377}]},{"osd":4,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.45400000000000001,"15min":0.45100000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.46899999999999997,"15min":0.46300000000000002},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.52400000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.34699999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.438}]}]},{"osd":0,"up_from":166,"seq":712964571144,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3197288,"kb_used_data":3162528,"kb_used_omap":884,"kb_used_meta":29835,"kb_avail":91174552,"statfs":{"total":96636764160,"available":93362741248,"internally_reserved":0,"allocated":3238428672,"data_stored":3664538791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":905963,"internal_metadata":30551317},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]}]},{"osd":1,"up_from":11,"seq":47244640314,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6867616,"kb_used_data":6770008,"kb_used_omap":0,"kb_used_meta":97600,"kb_avail":87504224,"statfs":{"total":96636764160,"available":89604325376,"internally_reserved":0,"allocated":6932488192,"data_stored":8162817840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":99942400},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.20200000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.32200000000000001,"15min":0.33000000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.41099999999999998,"5min":0.496,"15min":0.496},"last":0.25600000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.33500000000000002,"15min":0.33100000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.47899999999999998,"5min":1.615,"15min":1.615},"last":0.318},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.53100000000000003,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.28499999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.35599999999999998}]},{"osd":4,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47899999999999998,"15min":0.5},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.495,"15min":0.503},"min":{"1min":0.308,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71999999999999997,"5min":1.524,"15min":1.524},"last":0.39000000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.51500000000000001},"min":{"1min":0.33400000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.73899999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54800000000000004,"15min":0.54600000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.77600000000000002,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.46600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3248918528,"data_stored":3676420821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6949228544,"data_stored":8200825543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3825262592,"data_stored":4378559006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5008130048,"data_stored":5738833548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1624387584,"data_stored":1839286756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:06:29.065 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:06:29.105 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:29.104+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:29.176 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:29.176 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:06:29.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:06:29.176 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:29.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.177 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18643 2.5 e176/170 2:a9ace6f8:::rbd_data.125de1449605.0000000000000187:head v 176'2314, mlcod=173'2311)", 2023-07-02T14:06:29.177 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:06:29.177 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.748094244000001, 2023-07-02T14:06:29.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015027297, 2023-07-02T14:06:29.178 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.178 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.178 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:06:29.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:06:29.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:29.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426547+0000", 2023-07-02T14:06:29.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056864430000000002 2023-07-02T14:06:29.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432234+0000", 2023-07-02T14:06:29.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.415e-05 2023-07-02T14:06:29.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432248+0000", 2023-07-02T14:06:29.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7490000000000001e-06 2023-07-02T14:06:29.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432255+0000", 2023-07-02T14:06:29.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3915000000000002e-05 2023-07-02T14:06:29.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432299+0000", 2023-07-02T14:06:29.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1214000000000001e-05 2023-07-02T14:06:29.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432320+0000", 2023-07-02T14:06:29.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0092114950000000001 2023-07-02T14:06:29.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.441531+0000", 2023-07-02T14:06:29.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4422999999999998e-05 2023-07-02T14:06:29.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.441576+0000", 2023-07-02T14:06:29.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015027297 2023-07-02T14:06:29.184 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.185 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.185 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.185 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18644 2.2 e176/171 2:44e6ed94:::rbd_data.125de1449605.0000000000000185:head v 176'1626, mlcod=176'1623)", 2023-07-02T14:06:29.185 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.428712+0000", 2023-07-02T14:06:29.185 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.745930503, 2023-07-02T14:06:29.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012273601, 2023-07-02T14:06:29.186 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.186 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.186 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.428712+0000", 2023-07-02T14:06:29.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.428712+0000", 2023-07-02T14:06:29.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:29.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.428711+0000", 2023-07-02T14:06:29.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017963639999999999 2023-07-02T14:06:29.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.430507+0000", 2023-07-02T14:06:29.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0897e-05 2023-07-02T14:06:29.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.430528+0000", 2023-07-02T14:06:29.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7940000000000008e-06 2023-07-02T14:06:29.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.430537+0000", 2023-07-02T14:06:29.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00034051099999999999 2023-07-02T14:06:29.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.430878+0000", 2023-07-02T14:06:29.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5058000000000001e-05 2023-07-02T14:06:29.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.430893+0000", 2023-07-02T14:06:29.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010060174 2023-07-02T14:06:29.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.440953+0000", 2023-07-02T14:06:29.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3064000000000001e-05 2023-07-02T14:06:29.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.440986+0000", 2023-07-02T14:06:29.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012273601 2023-07-02T14:06:29.192 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.192 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.193 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.193 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18680 2.2 e176/171 2:5fd2923a:::rbd_data.125de1449605.000000000000018c:head v 176'1639, mlcod=176'1637)", 2023-07-02T14:06:29.193 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.489711+0000", 2023-07-02T14:06:29.193 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.684931802000001, 2023-07-02T14:06:29.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014407402, 2023-07-02T14:06:29.194 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.194 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.194 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.489711+0000", 2023-07-02T14:06:29.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.489711+0000", 2023-07-02T14:06:29.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:29.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.489709+0000", 2023-07-02T14:06:29.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001991792 2023-07-02T14:06:29.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491701+0000", 2023-07-02T14:06:29.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7323000000000002e-05 2023-07-02T14:06:29.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491738+0000", 2023-07-02T14:06:29.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2483999999999999e-05 2023-07-02T14:06:29.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491751+0000", 2023-07-02T14:06:29.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7896000000000003e-05 2023-07-02T14:06:29.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491809+0000", 2023-07-02T14:06:29.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0185000000000001e-05 2023-07-02T14:06:29.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491829+0000", 2023-07-02T14:06:29.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012250901999999999 2023-07-02T14:06:29.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.504080+0000", 2023-07-02T14:06:29.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:06:29.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.504118+0000", 2023-07-02T14:06:29.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014407402 2023-07-02T14:06:29.200 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.200 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.201 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.201 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18681 2.2 e176/171 2:5ecc17e8:::rbd_data.125de1449605.000000000000018d:head v 176'1640, mlcod=176'1637)", 2023-07-02T14:06:29.201 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.497048+0000", 2023-07-02T14:06:29.201 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.677594662000001, 2023-07-02T14:06:29.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012067603999999999, 2023-07-02T14:06:29.201 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.202 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.202 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.497048+0000", 2023-07-02T14:06:29.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.497048+0000", 2023-07-02T14:06:29.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:29.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.497047+0000", 2023-07-02T14:06:29.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0021639139999999999 2023-07-02T14:06:29.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.499211+0000", 2023-07-02T14:06:29.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7983e-05 2023-07-02T14:06:29.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.499239+0000", 2023-07-02T14:06:29.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6110999999999999e-05 2023-07-02T14:06:29.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.499255+0000", 2023-07-02T14:06:29.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0260999999999998e-05 2023-07-02T14:06:29.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.499315+0000", 2023-07-02T14:06:29.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0349e-05 2023-07-02T14:06:29.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.499345+0000", 2023-07-02T14:06:29.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097133659999999993 2023-07-02T14:06:29.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.509059+0000", 2023-07-02T14:06:29.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7092999999999997e-05 2023-07-02T14:06:29.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.509116+0000", 2023-07-02T14:06:29.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012067603999999999 2023-07-02T14:06:29.208 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.208 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.208 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.209 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18690 2.2 e176/171 2:416a1700:::rbd_data.125de1449605.000000000000018f:head v 176'1644, mlcod=176'1641)", 2023-07-02T14:06:29.209 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.523233+0000", 2023-07-02T14:06:29.209 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.651409671, 2023-07-02T14:06:29.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01191801, 2023-07-02T14:06:29.209 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.209 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.210 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.523233+0000", 2023-07-02T14:06:29.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.523233+0000", 2023-07-02T14:06:29.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:29.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.523231+0000", 2023-07-02T14:06:29.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0014368580000000001 2023-07-02T14:06:29.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.524668+0000", 2023-07-02T14:06:29.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.929e-05 2023-07-02T14:06:29.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.524688+0000", 2023-07-02T14:06:29.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9709999999999999e-06 2023-07-02T14:06:29.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.524698+0000", 2023-07-02T14:06:29.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000112815 2023-07-02T14:06:29.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.524810+0000", 2023-07-02T14:06:29.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7217999999999999e-05 2023-07-02T14:06:29.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.524838+0000", 2023-07-02T14:06:29.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010267129 2023-07-02T14:06:29.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.535105+0000", 2023-07-02T14:06:29.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6399000000000001e-05 2023-07-02T14:06:29.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.535151+0000", 2023-07-02T14:06:29.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01191801 2023-07-02T14:06:29.216 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.216 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.216 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.217 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18723 2.5 e176/170 2:a564ac9c:::rbd_data.125de1449605.0000000000000196:head v 176'2324, mlcod=176'2321)", 2023-07-02T14:06:29.217 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.568716+0000", 2023-07-02T14:06:29.217 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.605926494999999, 2023-07-02T14:06:29.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013584191000000001, 2023-07-02T14:06:29.217 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.217 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.217 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.568716+0000", 2023-07-02T14:06:29.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.568716+0000", 2023-07-02T14:06:29.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:29.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.568715+0000", 2023-07-02T14:06:29.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039052919999999999 2023-07-02T14:06:29.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.572620+0000", 2023-07-02T14:06:29.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5108000000000001e-05 2023-07-02T14:06:29.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.572645+0000", 2023-07-02T14:06:29.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0495e-05 2023-07-02T14:06:29.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.572656+0000", 2023-07-02T14:06:29.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6385999999999999e-05 2023-07-02T14:06:29.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.572702+0000", 2023-07-02T14:06:29.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8578e-05 2023-07-02T14:06:29.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.572731+0000", 2023-07-02T14:06:29.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0095207799999999995 2023-07-02T14:06:29.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.582251+0000", 2023-07-02T14:06:29.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9021999999999999e-05 2023-07-02T14:06:29.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.582300+0000", 2023-07-02T14:06:29.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013584191000000001 2023-07-02T14:06:29.224 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.224 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.224 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.225 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18859 2.5 e177/170 2:a5de3e1e:::rbd_data.125de1449605.0000000000000022:head v 177'2343, mlcod=177'2341)", 2023-07-02T14:06:29.225 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.883954+0000", 2023-07-02T14:06:29.225 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.290689016999998, 2023-07-02T14:06:29.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013529319999999999, 2023-07-02T14:06:29.225 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.225 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.225 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.883954+0000", 2023-07-02T14:06:29.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.883954+0000", 2023-07-02T14:06:29.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:29.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.883953+0000", 2023-07-02T14:06:29.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.003004048 2023-07-02T14:06:29.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.886957+0000", 2023-07-02T14:06:29.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0279999999999999e-05 2023-07-02T14:06:29.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.886977+0000", 2023-07-02T14:06:29.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2190000000000004e-06 2023-07-02T14:06:29.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.886985+0000", 2023-07-02T14:06:29.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7524e-05 2023-07-02T14:06:29.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.887043+0000", 2023-07-02T14:06:29.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6378000000000001e-05 2023-07-02T14:06:29.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.887069+0000", 2023-07-02T14:06:29.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010370551 2023-07-02T14:06:29.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.897440+0000", 2023-07-02T14:06:29.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3152999999999998e-05 2023-07-02T14:06:29.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.897483+0000", 2023-07-02T14:06:29.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013529319999999999 2023-07-02T14:06:29.232 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.232 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.232 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.232 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18881 2.5 e177/170 2:b1731d2f:::rbd_data.125de1449605.0000000000000024:head v 177'2350, mlcod=177'2347)", 2023-07-02T14:06:29.233 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:06:29.233 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.268728419999999, 2023-07-02T14:06:29.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020820709999999999, 2023-07-02T14:06:29.233 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.233 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.233 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:06:29.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:06:29.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:29.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905913+0000", 2023-07-02T14:06:29.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0080191519999999999 2023-07-02T14:06:29.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913932+0000", 2023-07-02T14:06:29.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3757999999999999e-05 2023-07-02T14:06:29.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913956+0000", 2023-07-02T14:06:29.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4009999999999997e-06 2023-07-02T14:06:29.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913965+0000", 2023-07-02T14:06:29.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3383e-05 2023-07-02T14:06:29.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914018+0000", 2023-07-02T14:06:29.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7674e-05 2023-07-02T14:06:29.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914046+0000", 2023-07-02T14:06:29.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012656555999999999 2023-07-02T14:06:29.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.926702+0000", 2023-07-02T14:06:29.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2891000000000001e-05 2023-07-02T14:06:29.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.926735+0000", 2023-07-02T14:06:29.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020820709999999999 2023-07-02T14:06:29.240 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.240 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.241 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.241 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18885 2.4 e177/169 2:3625364f:::rbd_data.125de1449605.0000000000000026:head v 177'1716, mlcod=177'1714)", 2023-07-02T14:06:29.241 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.914022+0000", 2023-07-02T14:06:29.241 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.260620617000001, 2023-07-02T14:06:29.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013176468, 2023-07-02T14:06:29.242 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.242 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.242 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914022+0000", 2023-07-02T14:06:29.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914022+0000", 2023-07-02T14:06:29.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:29.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914021+0000", 2023-07-02T14:06:29.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.002349889 2023-07-02T14:06:29.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916371+0000", 2023-07-02T14:06:29.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5821e-05 2023-07-02T14:06:29.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916387+0000", 2023-07-02T14:06:29.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3200000000000006e-06 2023-07-02T14:06:29.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.916396+0000", 2023-07-02T14:06:29.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0010131820000000001 2023-07-02T14:06:29.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.917409+0000", 2023-07-02T14:06:29.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3589000000000001e-05 2023-07-02T14:06:29.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.917433+0000", 2023-07-02T14:06:29.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097323819999999995 2023-07-02T14:06:29.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.927165+0000", 2023-07-02T14:06:29.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3422e-05 2023-07-02T14:06:29.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.927199+0000", 2023-07-02T14:06:29.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013176468 2023-07-02T14:06:29.248 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.248 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.249 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.249 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18893 2.5 e177/170 2:bbff50a9:::rbd_data.125de1449605.0000000000000027:head v 177'2354, mlcod=177'2351)", 2023-07-02T14:06:29.249 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.933879+0000", 2023-07-02T14:06:29.249 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.240763660999999, 2023-07-02T14:06:29.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012696278, 2023-07-02T14:06:29.249 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.250 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.250 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.933879+0000", 2023-07-02T14:06:29.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.933879+0000", 2023-07-02T14:06:29.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:29.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.933878+0000", 2023-07-02T14:06:29.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0035422819999999999 2023-07-02T14:06:29.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.937421+0000", 2023-07-02T14:06:29.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.7059999999999999e-06 2023-07-02T14:06:29.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.937430+0000", 2023-07-02T14:06:29.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5379999999999996e-06 2023-07-02T14:06:29.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.937435+0000", 2023-07-02T14:06:29.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8078999999999999e-05 2023-07-02T14:06:29.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.937473+0000", 2023-07-02T14:06:29.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4100999999999998e-05 2023-07-02T14:06:29.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.937497+0000", 2023-07-02T14:06:29.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.009041248 2023-07-02T14:06:29.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.946538+0000", 2023-07-02T14:06:29.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6986999999999997e-05 2023-07-02T14:06:29.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.946575+0000", 2023-07-02T14:06:29.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012696278 2023-07-02T14:06:29.256 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.256 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.256 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.257 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18935 2.4 e177/169 2:3e650bab:::rbd_data.125de1449605.0000000000000031:head v 177'1727, mlcod=177'1721)", 2023-07-02T14:06:29.257 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.994084+0000", 2023-07-02T14:06:29.257 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.180558954999999, 2023-07-02T14:06:29.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011588876, 2023-07-02T14:06:29.257 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.258 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.258 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.994084+0000", 2023-07-02T14:06:29.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.994084+0000", 2023-07-02T14:06:29.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:29.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.994083+0000", 2023-07-02T14:06:29.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0011146350000000001 2023-07-02T14:06:29.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.995197+0000", 2023-07-02T14:06:29.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4032999999999998e-05 2023-07-02T14:06:29.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.995231+0000", 2023-07-02T14:06:29.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9760000000000001e-06 2023-07-02T14:06:29.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.995240+0000", 2023-07-02T14:06:29.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00029382799999999999 2023-07-02T14:06:29.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.995534+0000", 2023-07-02T14:06:29.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6772999999999999e-05 2023-07-02T14:06:29.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.995561+0000", 2023-07-02T14:06:29.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010068725000000001 2023-07-02T14:06:29.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.005630+0000", 2023-07-02T14:06:29.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2930999999999999e-05 2023-07-02T14:06:29.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.005673+0000", 2023-07-02T14:06:29.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011588876 2023-07-02T14:06:29.264 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.264 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.264 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.265 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19073 2.5 e177/170 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:head v 177'2378, mlcod=177'2376)", 2023-07-02T14:06:29.265 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:06:29.265 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.901015236999999, 2023-07-02T14:06:29.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018435625000000001, 2023-07-02T14:06:29.265 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.266 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.266 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:06:29.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:06:29.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:29.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273626+0000", 2023-07-02T14:06:29.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046035659999999999 2023-07-02T14:06:29.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278230+0000", 2023-07-02T14:06:29.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9386e-05 2023-07-02T14:06:29.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278259+0000", 2023-07-02T14:06:29.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1972e-05 2023-07-02T14:06:29.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278271+0000", 2023-07-02T14:06:29.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017532680000000001 2023-07-02T14:06:29.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280025+0000", 2023-07-02T14:06:29.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2367000000000001e-05 2023-07-02T14:06:29.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280047+0000", 2023-07-02T14:06:29.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011987028 2023-07-02T14:06:29.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.292034+0000", 2023-07-02T14:06:29.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9266000000000001e-05 2023-07-02T14:06:29.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.292063+0000", 2023-07-02T14:06:29.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018435625000000001 2023-07-02T14:06:29.272 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.272 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.272 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.273 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19075 2.5 e177/170 2:adf18789:::rbd_data.125de1449605.000000000000003c:head v 177'2379, mlcod=177'2376)", 2023-07-02T14:06:29.273 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.280379+0000", 2023-07-02T14:06:29.273 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.89426417, 2023-07-02T14:06:29.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014230578000000001, 2023-07-02T14:06:29.273 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.273 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.274 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280379+0000", 2023-07-02T14:06:29.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280379+0000", 2023-07-02T14:06:29.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999804 2023-07-02T14:06:29.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280359+0000", 2023-07-02T14:06:29.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047641209999999996 2023-07-02T14:06:29.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.285123+0000", 2023-07-02T14:06:29.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3711e-05 2023-07-02T14:06:29.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.285137+0000", 2023-07-02T14:06:29.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.277e-06 2023-07-02T14:06:29.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.285144+0000", 2023-07-02T14:06:29.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00041800499999999999 2023-07-02T14:06:29.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.285562+0000", 2023-07-02T14:06:29.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6143000000000001e-05 2023-07-02T14:06:29.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.285578+0000", 2023-07-02T14:06:29.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089876420000000005 2023-07-02T14:06:29.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.294566+0000", 2023-07-02T14:06:29.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3466e-05 2023-07-02T14:06:29.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.294609+0000", 2023-07-02T14:06:29.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014230578000000001 2023-07-02T14:06:29.280 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.280 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.280 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.281 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19118 2.5 e177/170 2:be65b5f1:::rbd_data.125de1449605.0000000000000043:head v 177'2389, mlcod=177'2386)", 2023-07-02T14:06:29.281 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.342749+0000", 2023-07-02T14:06:29.281 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.831893832999999, 2023-07-02T14:06:29.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011707482, 2023-07-02T14:06:29.281 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.281 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.282 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.342749+0000", 2023-07-02T14:06:29.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.342749+0000", 2023-07-02T14:06:29.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:29.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.342748+0000", 2023-07-02T14:06:29.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045726380000000004 2023-07-02T14:06:29.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.347320+0000", 2023-07-02T14:06:29.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1033e-05 2023-07-02T14:06:29.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.347341+0000", 2023-07-02T14:06:29.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1297e-05 2023-07-02T14:06:29.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.347353+0000", 2023-07-02T14:06:29.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9991000000000002e-05 2023-07-02T14:06:29.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.347413+0000", 2023-07-02T14:06:29.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2135000000000001e-05 2023-07-02T14:06:29.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.347435+0000", 2023-07-02T14:06:29.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0069823029999999996 2023-07-02T14:06:29.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.354417+0000", 2023-07-02T14:06:29.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9447999999999999e-05 2023-07-02T14:06:29.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.354456+0000", 2023-07-02T14:06:29.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011707482 2023-07-02T14:06:29.288 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.288 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.288 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.289 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19193 2.2 e177/171 2:5382aedc:::rbd_data.125de1449605.000000000000004e:head v 177'1704, mlcod=177'1701)", 2023-07-02T14:06:29.289 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.473454+0000", 2023-07-02T14:06:29.289 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.701188352999999, 2023-07-02T14:06:29.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012446847, 2023-07-02T14:06:29.289 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.289 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.289 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.473454+0000", 2023-07-02T14:06:29.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.473454+0000", 2023-07-02T14:06:29.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:29.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.473452+0000", 2023-07-02T14:06:29.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001241418 2023-07-02T14:06:29.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.474694+0000", 2023-07-02T14:06:29.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6421999999999999e-05 2023-07-02T14:06:29.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.474720+0000", 2023-07-02T14:06:29.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.9219999999999992e-06 2023-07-02T14:06:29.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.474730+0000", 2023-07-02T14:06:29.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000926395 2023-07-02T14:06:29.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475656+0000", 2023-07-02T14:06:29.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5822000000000001e-05 2023-07-02T14:06:29.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475682+0000", 2023-07-02T14:06:29.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01016774 2023-07-02T14:06:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485850+0000", 2023-07-02T14:06:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1266e-05 2023-07-02T14:06:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485901+0000", 2023-07-02T14:06:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012446847 2023-07-02T14:06:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19196 2.4 e177/169 2:2c79e900:::rbd_data.125de1449605.000000000000004a:head v 177'1774, mlcod=177'1772)", 2023-07-02T14:06:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.475437+0000", 2023-07-02T14:06:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.699206079, 2023-07-02T14:06:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013623751, 2023-07-02T14:06:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475437+0000", 2023-07-02T14:06:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475437+0000", 2023-07-02T14:06:29.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:06:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475434+0000", 2023-07-02T14:06:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3578999999999994e-05 2023-07-02T14:06:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475518+0000", 2023-07-02T14:06:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1090999999999998e-05 2023-07-02T14:06:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475539+0000", 2023-07-02T14:06:29.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4860000000000003e-06 2023-07-02T14:06:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475549+0000", 2023-07-02T14:06:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000107015 2023-07-02T14:06:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475656+0000", 2023-07-02T14:06:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6625e-05 2023-07-02T14:06:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475682+0000", 2023-07-02T14:06:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013334037 2023-07-02T14:06:29.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489016+0000", 2023-07-02T14:06:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4141999999999999e-05 2023-07-02T14:06:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489060+0000", 2023-07-02T14:06:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013623751 2023-07-02T14:06:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:20883 2.5 e189/170 2:ad4691f5:::rbd_data.125de1449605.00000000000001a4:head v 189'2616, mlcod=181'2613)", 2023-07-02T14:06:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.342662+0000", 2023-07-02T14:06:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.8319811929999998, 2023-07-02T14:06:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012357348000000001, 2023-07-02T14:06:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.342662+0000", 2023-07-02T14:06:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.342662+0000", 2023-07-02T14:06:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.342661+0000", 2023-07-02T14:06:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0030281850000000001 2023-07-02T14:06:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.345689+0000", 2023-07-02T14:06:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7099999999999999e-05 2023-07-02T14:06:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.345706+0000", 2023-07-02T14:06:29.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4359999999999999e-06 2023-07-02T14:06:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.345712+0000", 2023-07-02T14:06:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1153999999999999e-05 2023-07-02T14:06:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.345754+0000", 2023-07-02T14:06:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3956e-05 2023-07-02T14:06:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.345767+0000", 2023-07-02T14:06:29.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0092101100000000005 2023-07-02T14:06:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.354978+0000", 2023-07-02T14:06:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1350999999999998e-05 2023-07-02T14:06:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.355019+0000", 2023-07-02T14:06:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012357348000000001 2023-07-02T14:06:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.312 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:20898 2.4 e189/169 2:21299eda:::rbd_data.125de1449605.00000000000001a6:head v 189'1930, mlcod=189'1928)", 2023-07-02T14:06:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.362271+0000", 2023-07-02T14:06:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.8123721380000002, 2023-07-02T14:06:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011644125, 2023-07-02T14:06:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.362271+0000", 2023-07-02T14:06:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.362271+0000", 2023-07-02T14:06:29.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.362269+0000", 2023-07-02T14:06:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001168936 2023-07-02T14:06:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.363438+0000", 2023-07-02T14:06:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1375000000000001e-05 2023-07-02T14:06:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.363459+0000", 2023-07-02T14:06:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0129999999999999e-06 2023-07-02T14:06:29.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.363468+0000", 2023-07-02T14:06:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9339999999999999e-05 2023-07-02T14:06:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.363518+0000", 2023-07-02T14:06:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6098999999999999e-05 2023-07-02T14:06:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.363544+0000", 2023-07-02T14:06:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010335578999999999 2023-07-02T14:06:29.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.373880+0000", 2023-07-02T14:06:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5281999999999999e-05 2023-07-02T14:06:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.373915+0000", 2023-07-02T14:06:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011644125 2023-07-02T14:06:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:20909 2.5 e189/170 2:bb7db89c:::rbd_data.125de1449605.00000000000001a5:head v 189'2620, mlcod=189'2618)", 2023-07-02T14:06:29.320 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.368184+0000", 2023-07-02T14:06:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.8064591600000002, 2023-07-02T14:06:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016727725999999998, 2023-07-02T14:06:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.368184+0000", 2023-07-02T14:06:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.368184+0000", 2023-07-02T14:06:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.368182+0000", 2023-07-02T14:06:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067720749999999998 2023-07-02T14:06:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.374954+0000", 2023-07-02T14:06:29.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4501000000000001e-05 2023-07-02T14:06:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.374969+0000", 2023-07-02T14:06:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4100000000000002e-06 2023-07-02T14:06:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.374976+0000", 2023-07-02T14:06:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.736e-05 2023-07-02T14:06:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.375034+0000", 2023-07-02T14:06:29.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4544000000000001e-05 2023-07-02T14:06:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.375058+0000", 2023-07-02T14:06:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0098134910000000006 2023-07-02T14:06:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.384872+0000", 2023-07-02T14:06:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9687000000000001e-05 2023-07-02T14:06:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.384911+0000", 2023-07-02T14:06:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016727725999999998 2023-07-02T14:06:29.327 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:21096 2.5 e190/170 2:a0acb434:::rbd_data.125de1449605.00000000000001bd:head v 190'2623, mlcod=189'2621)", 2023-07-02T14:06:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.705165+0000", 2023-07-02T14:06:29.328 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 3.4694773240000001, 2023-07-02T14:06:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027008152000000001, 2023-07-02T14:06:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.705165+0000", 2023-07-02T14:06:29.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.705165+0000", 2023-07-02T14:06:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.705164+0000", 2023-07-02T14:06:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017192187000000001 2023-07-02T14:06:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722356+0000", 2023-07-02T14:06:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0043e-05 2023-07-02T14:06:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722387+0000", 2023-07-02T14:06:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8230000000000001e-06 2023-07-02T14:06:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722395+0000", 2023-07-02T14:06:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9427e-05 2023-07-02T14:06:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722445+0000", 2023-07-02T14:06:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4321999999999999e-05 2023-07-02T14:06:29.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722469+0000", 2023-07-02T14:06:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0096570649999999994 2023-07-02T14:06:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.732126+0000", 2023-07-02T14:06:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7490000000000001e-05 2023-07-02T14:06:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.732174+0000", 2023-07-02T14:06:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027008152000000001 2023-07-02T14:06:29.335 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.336 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.336 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.336 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.336 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.336 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:29.338 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:06:29.338 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:29.337+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:29.405 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:06:29.407 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:06:29.418 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2023-07-02T14:06:28.558999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9472296739,"num_objects":2816,"num_object_clones":2325,"num_object_copies":5632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":275,"num_objects_unfound":0,"num_objects_dirty":2816,"num_whiteouts":0,"num_read":65946,"num_read_kb":12384683,"num_write":22134,"num_write_kb":9195051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2165,"num_bytes_recovered":7440234273,"num_keys_recovered":178,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21867,"ondisk_log_size":21867,"up":18,"acting":17,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":20,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":20427012,"kb_used_data":20139804,"kb_used_omap":1114,"kb_used_meta":282021,"kb_avail":451432188,"statfs":{"total":483183820800,"available":462266560512,"internally_reserved":0,"allocated":20623159296,"data_stored":23760669307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1141261,"internal_metadata":288790003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":54542660,"num_objects":26,"num_object_clones":20,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":163,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":2211,"num_read_kb":46575,"num_write":90,"num_write_kb":53301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":216,"num_bytes_recovered":761573716,"num_keys_recovered":64,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.027215"},"pg_stats":[{"pgid":"2.7","version":"191'3091","reported_seq":7976,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-07-02T14:06:26.608992+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:06:26.608992+0000","last_peered":"2023-07-02T14:06:26.608992+0000","last_clean":"2023-07-02T14:06:26.608992+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:06:26.608992+0000","last_undegraded":"2023-07-02T14:06:26.608992+0000","last_fullsized":"2023-07-02T14:06:26.608992+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3091,"ondisk_log_size":3091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1276541072,"num_objects":376,"num_object_clones":311,"num_object_copies":752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":41821,"num_read_kb":1595255,"num_write":2612,"num_write_kb":1240032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"189'2473","reported_seq":4112,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-07-02T14:06:25.271517+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:06:25.271517+0000","last_peered":"2023-07-02T14:06:25.271517+0000","last_clean":"2023-07-02T14:06:25.271517+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:06:25.271517+0000","last_undegraded":"2023-07-02T14:06:25.271517+0000","last_fullsized":"2023-07-02T14:06:25.271517+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2473,"ondisk_log_size":2473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1227764846,"num_objects":380,"num_object_clones":318,"num_object_copies":760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":380,"num_whiteouts":0,"num_read":1401,"num_read_kb":1671896,"num_write":2155,"num_write_kb":1191863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"190'2629","reported_seq":3194,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-07-02T14:06:26.555573+0000","last_change":"2023-07-02T14:05:57.093448+0000","last_active":"2023-07-02T14:06:26.555573+0000","last_peered":"2023-07-02T14:06:26.555573+0000","last_clean":"2023-07-02T14:06:26.555573+0000","last_became_active":"2023-07-02T14:05:57.089226+0000","last_became_peered":"2023-07-02T14:05:57.089226+0000","last_unstale":"2023-07-02T14:06:26.555573+0000","last_undegraded":"2023-07-02T14:06:26.555573+0000","last_fullsized":"2023-07-02T14:06:26.555573+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2629,"ondisk_log_size":2629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206334412,"num_objects":333,"num_object_clones":274,"num_object_copies":666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":489,"num_read_kb":1499543,"num_write":2355,"num_write_kb":1165651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":860565666,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"189'5157","reported_seq":6836,"reported_epoch":189,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-07-02T14:06:25.318100+0000","last_change":"2023-07-02T14:06:24.032319+0000","last_active":"2023-07-02T14:06:25.318100+0000","last_peered":"2023-07-02T14:06:25.318100+0000","last_clean":"2023-07-02T14:06:17.520145+0000","last_became_active":"2023-07-02T14:06:20.625414+0000","last_became_peered":"2023-07-02T14:06:20.625414+0000","last_unstale":"2023-07-02T14:06:25.318100+0000","last_undegraded":"2023-07-02T14:06:20.540308+0000","last_fullsized":"2023-07-02T14:06:20.528554+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":5157,"ondisk_log_size":5157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1303791042,"num_objects":381,"num_object_clones":318,"num_object_copies":762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":275,"num_objects_unfound":0,"num_objects_dirty":381,"num_whiteouts":0,"num_read":18034,"num_read_kb":1708239,"num_write":7599,"num_write_kb":1274150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":1210013154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1","objects":103},{"shards":"1,3","objects":275}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":119,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-07-02T14:06:25.259326+0000","last_change":"2023-07-02T14:05:53.410292+0000","last_active":"2023-07-02T14:06:25.259326+0000","last_peered":"2023-07-02T14:06:25.259326+0000","last_clean":"2023-07-02T14:06:25.259326+0000","last_became_active":"2023-07-02T14:05:53.409913+0000","last_became_peered":"2023-07-02T14:05:53.409913+0000","last_unstale":"2023-07-02T14:06:25.259326+0000","last_undegraded":"2023-07-02T14:06:25.259326+0000","last_fullsized":"2023-07-02T14:06:25.259326+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"189'2395","reported_seq":3595,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-07-02T14:06:25.242949+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:06:25.242949+0000","last_peered":"2023-07-02T14:06:25.242949+0000","last_clean":"2023-07-02T14:06:25.242949+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:06:25.242949+0000","last_undegraded":"2023-07-02T14:06:25.242949+0000","last_fullsized":"2023-07-02T14:06:25.242949+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2395,"ondisk_log_size":2395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1320547454,"num_objects":387,"num_object_clones":320,"num_object_copies":774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":387,"num_whiteouts":0,"num_read":1142,"num_read_kb":1676149,"num_write":2075,"num_write_kb":1283520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"189'2344","reported_seq":3041,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-07-02T14:06:25.245105+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:06:25.245105+0000","last_peered":"2023-07-02T14:06:25.245105+0000","last_clean":"2023-07-02T14:06:25.245105+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:06:25.245105+0000","last_undegraded":"2023-07-02T14:06:25.245105+0000","last_fullsized":"2023-07-02T14:06:25.245105+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2344,"ondisk_log_size":2344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1258112272,"num_objects":363,"num_object_clones":298,"num_object_copies":726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":715,"num_read_kb":1566165,"num_write":2046,"num_write_kb":1220500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"189'1857","reported_seq":3546,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-07-02T14:06:24.545222+0000","last_change":"2023-07-02T14:05:58.098047+0000","last_active":"2023-07-02T14:06:24.545222+0000","last_peered":"2023-07-02T14:06:24.545222+0000","last_clean":"2023-07-02T14:06:24.545222+0000","last_became_active":"2023-07-02T14:05:58.096421+0000","last_became_peered":"2023-07-02T14:05:58.096421+0000","last_unstale":"2023-07-02T14:06:24.545222+0000","last_undegraded":"2023-07-02T14:06:24.545222+0000","last_fullsized":"2023-07-02T14:06:24.545222+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1857,"ondisk_log_size":1857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":969606273,"num_objects":316,"num_object_clones":253,"num_object_copies":632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":1539,"num_read_kb":1421160,"num_write":1604,"num_write_kb":935617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":340635667,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"189'1921","reported_seq":2822,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-07-02T14:06:25.316705+0000","last_change":"2023-07-02T14:05:56.429093+0000","last_active":"2023-07-02T14:06:25.316705+0000","last_peered":"2023-07-02T14:06:25.316705+0000","last_clean":"2023-07-02T14:06:25.316705+0000","last_became_active":"2023-07-02T14:05:56.427016+0000","last_became_peered":"2023-07-02T14:05:56.427016+0000","last_unstale":"2023-07-02T14:06:25.316705+0000","last_undegraded":"2023-07-02T14:06:25.316705+0000","last_fullsized":"2023-07-02T14:06:25.316705+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1921,"ondisk_log_size":1921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":909599368,"num_objects":280,"num_object_clones":233,"num_object_copies":560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":280,"num_whiteouts":0,"num_read":805,"num_read_kb":1246276,"num_write":1688,"num_write_kb":883718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":271880192,"num_keys_recovered":4,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9472296739,"num_objects":2816,"num_object_clones":2325,"num_object_copies":5632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":275,"num_objects_unfound":0,"num_objects_dirty":2816,"num_whiteouts":0,"num_read":65946,"num_read_kb":12384683,"num_write":22134,"num_write_kb":9195051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2165,"num_bytes_recovered":7440234273,"num_keys_recovered":178,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20655927296,"data_stored":23833925674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":21867,"ondisk_log_size":21867,"up":16,"acting":15,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640314,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640314,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1639684,"kb_used_data":1583100,"kb_used_omap":0,"kb_used_meta":56576,"kb_avail":92732156,"statfs":{"total":96636764160,"available":94957727744,"internally_reserved":0,"allocated":1621094400,"data_stored":1827858700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":57933824},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.44,"15min":0.44400000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.64400000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47399999999999998,"15min":0.47999999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.69999999999999996,"5min":0.877,"15min":0.877},"last":0.40400000000000003}]},{"osd":1,"last update":"Sun Jul 2 14:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.44500000000000001,"15min":0.46600000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.436,"15min":0.44700000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.48699999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.44,"15min":0.45000000000000001},"min":{"1min":0.26500000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.47699999999999998,"15min":0.48399999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.38900000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.36599999999999999}]},{"osd":5,"last update":"Sun Jul 2 14:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.40899999999999997,"15min":0.45300000000000001},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.65400000000000003,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.44500000000000001,"15min":0.497},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.54800000000000004}]}]},{"osd":3,"up_from":133,"seq":571230650386,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4948560,"kb_used_data":4893000,"kb_used_omap":229,"kb_used_meta":55322,"kb_avail":89423280,"statfs":{"total":96636764160,"available":91569438720,"internally_reserved":0,"allocated":5010432000,"data_stored":5739736146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":56649950},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":1,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.63},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.54000000000000004}]},{"osd":4,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.59699999999999998}]},{"osd":5,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.67400000000000004}]}]},{"osd":2,"up_from":11,"seq":47244640314,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3773864,"kb_used_data":3731168,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":90597976,"statfs":{"total":96636764160,"available":92772327424,"internally_reserved":0,"allocated":3820716032,"data_stored":4365717830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.29399999999999998,"15min":0.29599999999999999},"min":{"1min":0.218,"5min":0.182,"15min":0.182},"max":{"1min":0.378,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.317,"5min":0.315,"15min":0.314},"min":{"1min":0.23000000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.39200000000000002}]},{"osd":1,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.38800000000000001,"15min":0.36199999999999999},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.2720000000000002,"5min":4.2720000000000002,"15min":4.2720000000000002},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.32300000000000001,"15min":0.32000000000000001},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.34699999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2,"5min":2,"15min":2},"last":0.377}]},{"osd":4,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.45400000000000001,"15min":0.45100000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.46899999999999997,"15min":0.46300000000000002},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.52400000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.34699999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.438}]}]},{"osd":0,"up_from":166,"seq":712964571144,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3197288,"kb_used_data":3162528,"kb_used_omap":884,"kb_used_meta":29835,"kb_avail":91174552,"statfs":{"total":96636764160,"available":93362741248,"internally_reserved":0,"allocated":3238428672,"data_stored":3664538791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":905963,"internal_metadata":30551317},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]}]},{"osd":1,"up_from":11,"seq":47244640314,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6867616,"kb_used_data":6770008,"kb_used_omap":0,"kb_used_meta":97600,"kb_avail":87504224,"statfs":{"total":96636764160,"available":89604325376,"internally_reserved":0,"allocated":6932488192,"data_stored":8162817840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":99942400},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.20200000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.32200000000000001,"15min":0.33000000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.41099999999999998,"5min":0.496,"15min":0.496},"last":0.25600000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.33500000000000002,"15min":0.33100000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.47899999999999998,"5min":1.615,"15min":1.615},"last":0.318},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.53100000000000003,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.28499999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.35599999999999998}]},{"osd":4,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47899999999999998,"15min":0.5},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.495,"15min":0.503},"min":{"1min":0.308,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71999999999999997,"5min":1.524,"15min":1.524},"last":0.39000000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.51500000000000001},"min":{"1min":0.33400000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.73899999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54800000000000004,"15min":0.54600000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.77600000000000002,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.46600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3248918528,"data_stored":3676420821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6949228544,"data_stored":8200825543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3825262592,"data_stored":4378559006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5008130048,"data_stored":5738833548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1624387584,"data_stored":1839286756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:06:29.420 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-07-02T14:06:29.463 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:29.462+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:29.464 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:29.465 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:29.465 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:29.465 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:29.476 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:06:29.539 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:29.538+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:29.578 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:06:29.640 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:29.639+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:29.647 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:29.647 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:29.648 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:06:29.648 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:06:29.648 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:29.658 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:06:29.740 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:29.740+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:29.807 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:06:29.808 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3/5632 objects degraded (0.053%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":292,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":194,"num_osds":6,"num_up_osds":6,"osd_up_since":1688306752,"num_in_osds":5,"osd_in_since":1688306778,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2816,"data_bytes":9472296739,"bytes_used":20917260288,"bytes_avail":462266560512,"bytes_total":483183820800,"degraded_objects":3,"degraded_total":5632,"degraded_ratio":0.00053267045454545451,"misplaced_objects":275,"misplaced_total":5632,"misplaced_ratio":0.048828125,"recovering_objects_per_sec":53,"recovering_bytes_per_sec":190364809,"recovering_keys_per_sec":15,"num_objects_recovered":216,"num_bytes_recovered":761573716,"num_keys_recovered":64,"read_bytes_sec":11921407,"write_bytes_sec":13643004,"read_op_per_sec":552,"write_op_per_sec":22},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2023-07-02T14:06:04.278714+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-07-02T14:06:29.822 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2023-07-02T14:06:29.822 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:06:29.841 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:29.840+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:29.845 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:29.845 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:06:29.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:06:29.845 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:29.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.846 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:06:29.846 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:29.846 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 200.39154826199999, 2023-07-02T14:06:29.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:06:29.846 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.846 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:29.846 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:29.847 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:29.847 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:29.847 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:06:29.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.847 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.847 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:29.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:29.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:29.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:06:29.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:06:29.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:06:29.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:06:29.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:06:29.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:06:29.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:06:29.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:06:29.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:06:29.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:06:29.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:29.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:06:29.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:06:29.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:06:29.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:06:29.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:06:29.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:06:29.854 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.854 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.854 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.854 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:06:29.854 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:29.854 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 199.520855254, 2023-07-02T14:06:29.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:06:29.855 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.855 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:29.855 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:29.855 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:06:29.855 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:06:29.855 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:06:29.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.856 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:29.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:29.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:29.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:06:29.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:06:29.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:06:29.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:06:29.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:06:29.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:06:29.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:06:29.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:06:29.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:06:29.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:06:29.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:29.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:06:29.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:06:29.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:06:29.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:06:29.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:06:29.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:06:29.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:06:29.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:06:29.863 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.863 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.863 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.863 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:29.863 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:29.863 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 199.52073053300001, 2023-07-02T14:06:29.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:06:29.864 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.864 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:29.864 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:29.864 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:29.864 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:29.864 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:06:29.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.865 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:29.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:29.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:29.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:06:29.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:06:29.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.867 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:06:29.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:06:29.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:06:29.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:06:29.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:06:29.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:06:29.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:06:29.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:06:29.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:29.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:06:29.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:06:29.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:06:29.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:06:29.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:06:29.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:06:29.871 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.871 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.871 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.872 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:29.872 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:29.872 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 199.51388064899999, 2023-07-02T14:06:29.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:06:29.872 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.872 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:29.872 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:29.873 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:29.873 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:29.873 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:06:29.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.873 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:29.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.874 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:29.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:06:29.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:06:29.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:06:29.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:06:29.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:06:29.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:06:29.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:06:29.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:06:29.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:06:29.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:06:29.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:06:29.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:29.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:06:29.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:06:29.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:06:29.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:06:29.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:06:29.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:06:29.879 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.879 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.880 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.880 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:29.880 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:29.880 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 199.51381772400001, 2023-07-02T14:06:29.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:06:29.881 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.881 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:29.881 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:29.881 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:29.881 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:29.881 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:06:29.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.881 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:29.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:29.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:29.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:06:29.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:06:29.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:06:29.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:06:29.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:06:29.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:06:29.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.885 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:06:29.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:06:29.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.885 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:06:29.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:06:29.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:29.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:06:29.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:06:29.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:06:29.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:06:29.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:06:29.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:06:29.888 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.888 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.888 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.888 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:29.889 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:29.889 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 199.51351730299999, 2023-07-02T14:06:29.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:06:29.889 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.889 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.889 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:29.889 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:29.889 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:29.890 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:06:29.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.890 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:29.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:29.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:29.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:06:29.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:06:29.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:06:29.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:06:29.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:06:29.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:06:29.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:06:29.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:06:29.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:06:29.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:06:29.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:29.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:06:29.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:06:29.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:06:29.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:06:29.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:06:29.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:06:29.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:06:29.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:06:29.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:06:29.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:29.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:06:29.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:06:29.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:29.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:06:29.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:06:29.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:29.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:06:29.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:06:29.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:06:29.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:06:29.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:06:29.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:06:29.900 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.900 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.900 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.901 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:06:29.901 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:29.901 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 199.512992803, 2023-07-02T14:06:29.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:06:29.901 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.901 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:29.901 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:29.902 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:29.902 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:29.902 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:06:29.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.902 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:29.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:29.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:29.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:29.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:06:29.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:06:29.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:06:29.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:06:29.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:06:29.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:06:29.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:06:29.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:06:29.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:06:29.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:29.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:06:29.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:06:29.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:06:29.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:06:29.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:06:29.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:06:29.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:06:29.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:06:29.909 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.909 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.909 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.910 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:06:29.910 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:29.910 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 172.38897367800001, 2023-07-02T14:06:29.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:06:29.910 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.910 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:29.910 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:29.911 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:29.911 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:29.911 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:06:29.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.911 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:29.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:29.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:29.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:06:29.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:06:29.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:06:29.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:06:29.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:06:29.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:06:29.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:06:29.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:06:29.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:06:29.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:06:29.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:29.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:06:29.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:06:29.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:06:29.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:06:29.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:06:29.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:06:29.917 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.917 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.918 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.918 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:06:29.918 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:29.918 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 172.06349097399999, 2023-07-02T14:06:29.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:06:29.919 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.919 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:29.919 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:29.919 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:29.919 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:29.919 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:06:29.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.919 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:29.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:29.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:29.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:06:29.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:06:29.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:06:29.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:06:29.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:06:29.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:06:29.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:06:29.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:06:29.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:06:29.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:06:29.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:29.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:06:29.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:06:29.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:06:29.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:06:29.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:06:29.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:06:29.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:29.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:06:29.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:06:29.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:29.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:06:29.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:06:29.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:29.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:06:29.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:06:29.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:29.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:06:29.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:06:29.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:06:29.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:06:29.929 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.929 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.929 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.930 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:17 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 0~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:29.930 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:29.930 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.645215191999995, 2023-07-02T14:06:29.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997, 2023-07-02T14:06:29.930 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.930 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:29.930 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:29.930 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:29.931 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:29.931 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17 2023-07-02T14:06:29.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.931 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:29.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:29.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:06:29.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198431+0000", 2023-07-02T14:06:29.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2815e-05 2023-07-02T14:06:29.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198443+0000", 2023-07-02T14:06:29.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2569999999999999e-05 2023-07-02T14:06:29.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198466+0000", 2023-07-02T14:06:29.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0223e-05 2023-07-02T14:06:29.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198476+0000", 2023-07-02T14:06:29.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1792999999999997e-05 2023-07-02T14:06:29.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:06:29.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4249999999999998e-06 2023-07-02T14:06:29.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:29.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198536+0000", 2023-07-02T14:06:29.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91888663500000001 2023-07-02T14:06:29.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117423+0000", 2023-07-02T14:06:29.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223594 2023-07-02T14:06:29.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117647+0000", 2023-07-02T14:06:29.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033832900000000001 2023-07-02T14:06:29.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117985+0000", 2023-07-02T14:06:29.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997 2023-07-02T14:06:29.938 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.938 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.938 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.938 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:18 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 65536~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:29.939 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:29.939 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.645157980999997, 2023-07-02T14:06:29.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998, 2023-07-02T14:06:29.939 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.939 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:29.939 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:29.939 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:29.940 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:29.940 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18 2023-07-02T14:06:29.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.940 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:29.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:29.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:29.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:29.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7600000000000002e-06 2023-07-02T14:06:29.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198494+0000", 2023-07-02T14:06:29.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7840000000000004e-06 2023-07-02T14:06:29.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198502+0000", 2023-07-02T14:06:29.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6459999999999998e-06 2023-07-02T14:06:29.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198508+0000", 2023-07-02T14:06:29.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7395999999999997e-05 2023-07-02T14:06:29.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198565+0000", 2023-07-02T14:06:29.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6800000000000004e-06 2023-07-02T14:06:29.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:29.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198572+0000", 2023-07-02T14:06:29.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91965896800000002 2023-07-02T14:06:29.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118231+0000", 2023-07-02T14:06:29.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5212999999999998e-05 2023-07-02T14:06:29.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118326+0000", 2023-07-02T14:06:29.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000238538 2023-07-02T14:06:29.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118565+0000", 2023-07-02T14:06:29.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998 2023-07-02T14:06:29.947 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.947 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.947 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.948 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:19 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 131072~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:29.948 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:29.948 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.645133756999996, 2023-07-02T14:06:29.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002, 2023-07-02T14:06:29.948 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.948 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:29.948 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:29.948 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:29.949 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:29.949 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 19 2023-07-02T14:06:29.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.949 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:29.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:29.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:29.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:29.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1990000000000002e-06 2023-07-02T14:06:29.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198517+0000", 2023-07-02T14:06:29.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8789999999999997e-06 2023-07-02T14:06:29.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198524+0000", 2023-07-02T14:06:29.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:06:29.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:06:29.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5545999999999999e-05 2023-07-02T14:06:29.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:29.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2540000000000003e-06 2023-07-02T14:06:29.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:29.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198598+0000", 2023-07-02T14:06:29.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.919983263 2023-07-02T14:06:29.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118581+0000", 2023-07-02T14:06:29.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6878999999999994e-05 2023-07-02T14:06:29.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118648+0000", 2023-07-02T14:06:29.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149732 2023-07-02T14:06:29.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118798+0000", 2023-07-02T14:06:29.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002 2023-07-02T14:06:29.956 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.956 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.956 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.957 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:20 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 262144~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:29.957 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:29.957 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.645114793000005, 2023-07-02T14:06:29.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998, 2023-07-02T14:06:29.957 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.957 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:29.957 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:29.957 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:29.958 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:29.958 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20 2023-07-02T14:06:29.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.958 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:29.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:29.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:29.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:29.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.732e-06 2023-07-02T14:06:29.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198537+0000", 2023-07-02T14:06:29.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5259999999999999e-06 2023-07-02T14:06:29.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198543+0000", 2023-07-02T14:06:29.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.121e-06 2023-07-02T14:06:29.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198547+0000", 2023-07-02T14:06:29.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9591000000000005e-05 2023-07-02T14:06:29.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:06:29.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3120000000000004e-06 2023-07-02T14:06:29.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:29.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198621+0000", 2023-07-02T14:06:29.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92020789400000003 2023-07-02T14:06:29.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118829+0000", 2023-07-02T14:06:29.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022393386000000001 2023-07-02T14:06:29.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141223+0000", 2023-07-02T14:06:29.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000329866 2023-07-02T14:06:29.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141552+0000", 2023-07-02T14:06:29.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998 2023-07-02T14:06:29.965 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.965 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.965 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.965 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:21 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 196608~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:29.966 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:29.966 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.645087157999995, 2023-07-02T14:06:29.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995, 2023-07-02T14:06:29.966 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.966 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:29.966 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:29.966 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:29.966 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:29.967 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21 2023-07-02T14:06:29.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.967 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:29.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:29.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:29.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:29.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0169999999999999e-05 2023-07-02T14:06:29.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198571+0000", 2023-07-02T14:06:29.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3479999999999998e-06 2023-07-02T14:06:29.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198578+0000", 2023-07-02T14:06:29.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8439999999999997e-06 2023-07-02T14:06:29.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198583+0000", 2023-07-02T14:06:29.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8294999999999997e-05 2023-07-02T14:06:29.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198641+0000", 2023-07-02T14:06:29.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8120000000000001e-06 2023-07-02T14:06:29.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:29.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:06:29.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94293828499999999 2023-07-02T14:06:29.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141583+0000", 2023-07-02T14:06:29.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014607300000000001 2023-07-02T14:06:29.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141729+0000", 2023-07-02T14:06:29.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00044906600000000003 2023-07-02T14:06:29.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142178+0000", 2023-07-02T14:06:29.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995 2023-07-02T14:06:29.974 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.974 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.974 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.974 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:22 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 327680~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:29.975 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:29.975 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.645053727999993, 2023-07-02T14:06:29.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004, 2023-07-02T14:06:29.975 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.975 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:29.975 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:29.975 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:29.976 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:29.976 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 22 2023-07-02T14:06:29.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.976 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:29.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.977 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:29.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:29.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:29.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4850000000000001e-06 2023-07-02T14:06:29.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198597+0000", 2023-07-02T14:06:29.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:06:29.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198605+0000", 2023-07-02T14:06:29.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4839999999999998e-06 2023-07-02T14:06:29.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198609+0000", 2023-07-02T14:06:29.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3100999999999999e-05 2023-07-02T14:06:29.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198662+0000", 2023-07-02T14:06:29.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9099999999999998e-06 2023-07-02T14:06:29.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:29.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198666+0000", 2023-07-02T14:06:29.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.943539503 2023-07-02T14:06:29.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142206+0000", 2023-07-02T14:06:29.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012255999999999999 2023-07-02T14:06:29.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142328+0000", 2023-07-02T14:06:29.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208956 2023-07-02T14:06:29.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142537+0000", 2023-07-02T14:06:29.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004 2023-07-02T14:06:29.983 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.983 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.983 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.983 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:23 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 393216~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:29.984 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:29.984 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.645033748000003, 2023-07-02T14:06:29.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348, 2023-07-02T14:06:29.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.984 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:29.984 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:29.984 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:29.984 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:29.985 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23 2023-07-02T14:06:29.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.985 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:29.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:29.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:29.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:29.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8439999999999998e-06 2023-07-02T14:06:29.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:06:29.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0440000000000001e-06 2023-07-02T14:06:29.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198623+0000", 2023-07-02T14:06:29.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4480000000000002e-06 2023-07-02T14:06:29.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198628+0000", 2023-07-02T14:06:29.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5843e-05 2023-07-02T14:06:29.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:29.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.89e-06 2023-07-02T14:06:29.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:29.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198688+0000", 2023-07-02T14:06:29.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94389171999999999 2023-07-02T14:06:29.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142580+0000", 2023-07-02T14:06:29.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3892000000000004e-05 2023-07-02T14:06:29.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:29.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142644+0000", 2023-07-02T14:06:29.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017608000000000001 2023-07-02T14:06:29.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:29.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142820+0000", 2023-07-02T14:06:29.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348 2023-07-02T14:06:29.992 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.992 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:29.992 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:29.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.992 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:24 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 458752~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:29.993 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:29.993 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.645005674000004, 2023-07-02T14:06:29.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004, 2023-07-02T14:06:29.993 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:29.993 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:29.993 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:29.993 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:29.993 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:29.994 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 24 2023-07-02T14:06:29.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.994 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:29.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:29.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:29.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:29.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:29.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:29.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:29.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:29.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198642+0000", 2023-07-02T14:06:29.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.253e-06 2023-07-02T14:06:29.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:29.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:06:29.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1289999999999998e-06 2023-07-02T14:06:29.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:29.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198652+0000", 2023-07-02T14:06:29.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4510000000000002e-06 2023-07-02T14:06:29.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:29.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198657+0000", 2023-07-02T14:06:29.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9131000000000001e-05 2023-07-02T14:06:29.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198706+0000", 2023-07-02T14:06:29.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8890000000000004e-06 2023-07-02T14:06:29.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:29.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198710+0000", 2023-07-02T14:06:29.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94415412200000004 2023-07-02T14:06:29.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:29.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:29.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:29.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142864+0000", 2023-07-02T14:06:29.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8047000000000006e-05 2023-07-02T14:06:30.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142952+0000", 2023-07-02T14:06:30.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192967 2023-07-02T14:06:30.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143145+0000", 2023-07-02T14:06:30.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004 2023-07-02T14:06:30.001 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.001 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.001 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.001 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:25 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 524288~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:30.002 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:30.002 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.644986879000001, 2023-07-02T14:06:30.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001, 2023-07-02T14:06:30.002 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.002 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:30.002 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:30.002 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:30.002 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:30.003 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25 2023-07-02T14:06:30.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.003 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:30.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:30.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:30.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:30.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.615e-06 2023-07-02T14:06:30.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198664+0000", 2023-07-02T14:06:30.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4759999999999997e-06 2023-07-02T14:06:30.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198669+0000", 2023-07-02T14:06:30.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5999999999999998e-06 2023-07-02T14:06:30.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198673+0000", 2023-07-02T14:06:30.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4256999999999999e-05 2023-07-02T14:06:30.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198727+0000", 2023-07-02T14:06:30.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840000000000001e-06 2023-07-02T14:06:30.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:30.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198731+0000", 2023-07-02T14:06:30.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94444241900000003 2023-07-02T14:06:30.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143173+0000", 2023-07-02T14:06:30.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6839000000000002e-05 2023-07-02T14:06:30.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143240+0000", 2023-07-02T14:06:30.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017053399999999999 2023-07-02T14:06:30.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143411+0000", 2023-07-02T14:06:30.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001 2023-07-02T14:06:30.010 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.010 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.010 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.010 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:26 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 589824~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:30.011 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:30.011 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.644963727999993, 2023-07-02T14:06:30.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997, 2023-07-02T14:06:30.011 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.011 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:30.011 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:30.011 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:30.011 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:30.012 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 26 2023-07-02T14:06:30.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.012 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:30.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:30.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:30.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:30.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0520000000000002e-06 2023-07-02T14:06:30.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198687+0000", 2023-07-02T14:06:30.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1880000000000002e-06 2023-07-02T14:06:30.014 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198695+0000", 2023-07-02T14:06:30.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9960000000000004e-06 2023-07-02T14:06:30.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.015 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.015 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198699+0000", 2023-07-02T14:06:30.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2330999999999998e-05 2023-07-02T14:06:30.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198742+0000", 2023-07-02T14:06:30.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7950000000000001e-06 2023-07-02T14:06:30.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:30.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198745+0000", 2023-07-02T14:06:30.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94469398199999999 2023-07-02T14:06:30.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143439+0000", 2023-07-02T14:06:30.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6048e-05 2023-07-02T14:06:30.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143525+0000", 2023-07-02T14:06:30.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017016700000000001 2023-07-02T14:06:30.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143696+0000", 2023-07-02T14:06:30.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997 2023-07-02T14:06:30.019 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.019 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.019 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.019 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.6579.0:11 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected e171)", 2023-07-02T14:06:30.020 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:30.020 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.685165241999997, 2023-07-02T14:06:30.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998, 2023-07-02T14:06:30.020 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.020 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:30.020 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:30.020 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.6579", 2023-07-02T14:06:30.020 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/554403467", 2023-07-02T14:06:30.021 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11 2023-07-02T14:06:30.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.021 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:30.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:30.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:30.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158481+0000", 2023-07-02T14:06:30.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:06:30.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.023 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.023 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158487+0000", 2023-07-02T14:06:30.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7630000000000006e-06 2023-07-02T14:06:30.023 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158496+0000", 2023-07-02T14:06:30.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5100000000000001e-06 2023-07-02T14:06:30.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158504+0000", 2023-07-02T14:06:30.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9397000000000002e-05 2023-07-02T14:06:30.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158553+0000", 2023-07-02T14:06:30.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5810000000000003e-06 2023-07-02T14:06:30.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:30.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158562+0000", 2023-07-02T14:06:30.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93893079499999998 2023-07-02T14:06:30.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097493+0000", 2023-07-02T14:06:30.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015799299999999999 2023-07-02T14:06:30.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097651+0000", 2023-07-02T14:06:30.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187746 2023-07-02T14:06:30.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097839+0000", 2023-07-02T14:06:30.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998 2023-07-02T14:06:30.028 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.028 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.028 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.028 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.028 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.028 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:30.030 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:06:30.030 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:30.028+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:30.130 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:30.129+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:30.145 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:06:30.149 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:06:30.164 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2023-07-02T14:06:29.563518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9472296739,"num_objects":2816,"num_object_clones":2325,"num_object_copies":5632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":275,"num_objects_unfound":0,"num_objects_dirty":2816,"num_whiteouts":0,"num_read":65946,"num_read_kb":12384683,"num_write":22134,"num_write_kb":9195051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2165,"num_bytes_recovered":7440234273,"num_keys_recovered":178,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21867,"ondisk_log_size":21867,"up":18,"acting":17,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":20,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":20427012,"kb_used_data":20139804,"kb_used_omap":1114,"kb_used_meta":282021,"kb_avail":451432188,"statfs":{"total":483183820800,"available":462266560512,"internally_reserved":0,"allocated":20623159296,"data_stored":23760669307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1141261,"internal_metadata":288790003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38141952,"num_objects":12,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1,"num_objects_misplaced":-112,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2072,"num_read_kb":1421,"num_write":34,"num_write_kb":37252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":411770880,"num_keys_recovered":64,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.024215"},"pg_stats":[{"pgid":"2.7","version":"191'3091","reported_seq":7976,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-07-02T14:06:26.608992+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:06:26.608992+0000","last_peered":"2023-07-02T14:06:26.608992+0000","last_clean":"2023-07-02T14:06:26.608992+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:06:26.608992+0000","last_undegraded":"2023-07-02T14:06:26.608992+0000","last_fullsized":"2023-07-02T14:06:26.608992+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3091,"ondisk_log_size":3091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1276541072,"num_objects":376,"num_object_clones":311,"num_object_copies":752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":41821,"num_read_kb":1595255,"num_write":2612,"num_write_kb":1240032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"189'2473","reported_seq":4112,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-07-02T14:06:25.271517+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:06:25.271517+0000","last_peered":"2023-07-02T14:06:25.271517+0000","last_clean":"2023-07-02T14:06:25.271517+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:06:25.271517+0000","last_undegraded":"2023-07-02T14:06:25.271517+0000","last_fullsized":"2023-07-02T14:06:25.271517+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2473,"ondisk_log_size":2473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1227764846,"num_objects":380,"num_object_clones":318,"num_object_copies":760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":380,"num_whiteouts":0,"num_read":1401,"num_read_kb":1671896,"num_write":2155,"num_write_kb":1191863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"190'2629","reported_seq":3194,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-07-02T14:06:26.555573+0000","last_change":"2023-07-02T14:05:57.093448+0000","last_active":"2023-07-02T14:06:26.555573+0000","last_peered":"2023-07-02T14:06:26.555573+0000","last_clean":"2023-07-02T14:06:26.555573+0000","last_became_active":"2023-07-02T14:05:57.089226+0000","last_became_peered":"2023-07-02T14:05:57.089226+0000","last_unstale":"2023-07-02T14:06:26.555573+0000","last_undegraded":"2023-07-02T14:06:26.555573+0000","last_fullsized":"2023-07-02T14:06:26.555573+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2629,"ondisk_log_size":2629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206334412,"num_objects":333,"num_object_clones":274,"num_object_copies":666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":489,"num_read_kb":1499543,"num_write":2355,"num_write_kb":1165651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":860565666,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"189'5157","reported_seq":6836,"reported_epoch":189,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-07-02T14:06:25.318100+0000","last_change":"2023-07-02T14:06:24.032319+0000","last_active":"2023-07-02T14:06:25.318100+0000","last_peered":"2023-07-02T14:06:25.318100+0000","last_clean":"2023-07-02T14:06:17.520145+0000","last_became_active":"2023-07-02T14:06:20.625414+0000","last_became_peered":"2023-07-02T14:06:20.625414+0000","last_unstale":"2023-07-02T14:06:25.318100+0000","last_undegraded":"2023-07-02T14:06:20.540308+0000","last_fullsized":"2023-07-02T14:06:20.528554+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":5157,"ondisk_log_size":5157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1303791042,"num_objects":381,"num_object_clones":318,"num_object_copies":762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":275,"num_objects_unfound":0,"num_objects_dirty":381,"num_whiteouts":0,"num_read":18034,"num_read_kb":1708239,"num_write":7599,"num_write_kb":1274150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":1210013154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1","objects":103},{"shards":"1,3","objects":275}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":119,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-07-02T14:06:25.259326+0000","last_change":"2023-07-02T14:05:53.410292+0000","last_active":"2023-07-02T14:06:25.259326+0000","last_peered":"2023-07-02T14:06:25.259326+0000","last_clean":"2023-07-02T14:06:25.259326+0000","last_became_active":"2023-07-02T14:05:53.409913+0000","last_became_peered":"2023-07-02T14:05:53.409913+0000","last_unstale":"2023-07-02T14:06:25.259326+0000","last_undegraded":"2023-07-02T14:06:25.259326+0000","last_fullsized":"2023-07-02T14:06:25.259326+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"189'2395","reported_seq":3595,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-07-02T14:06:25.242949+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:06:25.242949+0000","last_peered":"2023-07-02T14:06:25.242949+0000","last_clean":"2023-07-02T14:06:25.242949+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:06:25.242949+0000","last_undegraded":"2023-07-02T14:06:25.242949+0000","last_fullsized":"2023-07-02T14:06:25.242949+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2395,"ondisk_log_size":2395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1320547454,"num_objects":387,"num_object_clones":320,"num_object_copies":774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":387,"num_whiteouts":0,"num_read":1142,"num_read_kb":1676149,"num_write":2075,"num_write_kb":1283520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"189'2344","reported_seq":3041,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-07-02T14:06:25.245105+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:06:25.245105+0000","last_peered":"2023-07-02T14:06:25.245105+0000","last_clean":"2023-07-02T14:06:25.245105+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:06:25.245105+0000","last_undegraded":"2023-07-02T14:06:25.245105+0000","last_fullsized":"2023-07-02T14:06:25.245105+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2344,"ondisk_log_size":2344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1258112272,"num_objects":363,"num_object_clones":298,"num_object_copies":726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":715,"num_read_kb":1566165,"num_write":2046,"num_write_kb":1220500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"189'1857","reported_seq":3546,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-07-02T14:06:24.545222+0000","last_change":"2023-07-02T14:05:58.098047+0000","last_active":"2023-07-02T14:06:24.545222+0000","last_peered":"2023-07-02T14:06:24.545222+0000","last_clean":"2023-07-02T14:06:24.545222+0000","last_became_active":"2023-07-02T14:05:58.096421+0000","last_became_peered":"2023-07-02T14:05:58.096421+0000","last_unstale":"2023-07-02T14:06:24.545222+0000","last_undegraded":"2023-07-02T14:06:24.545222+0000","last_fullsized":"2023-07-02T14:06:24.545222+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1857,"ondisk_log_size":1857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":969606273,"num_objects":316,"num_object_clones":253,"num_object_copies":632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":1539,"num_read_kb":1421160,"num_write":1604,"num_write_kb":935617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":340635667,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"189'1921","reported_seq":2822,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-07-02T14:06:25.316705+0000","last_change":"2023-07-02T14:05:56.429093+0000","last_active":"2023-07-02T14:06:25.316705+0000","last_peered":"2023-07-02T14:06:25.316705+0000","last_clean":"2023-07-02T14:06:25.316705+0000","last_became_active":"2023-07-02T14:05:56.427016+0000","last_became_peered":"2023-07-02T14:05:56.427016+0000","last_unstale":"2023-07-02T14:06:25.316705+0000","last_undegraded":"2023-07-02T14:06:25.316705+0000","last_fullsized":"2023-07-02T14:06:25.316705+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1921,"ondisk_log_size":1921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":909599368,"num_objects":280,"num_object_clones":233,"num_object_copies":560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":280,"num_whiteouts":0,"num_read":805,"num_read_kb":1246276,"num_write":1688,"num_write_kb":883718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":271880192,"num_keys_recovered":4,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9472296739,"num_objects":2816,"num_object_clones":2325,"num_object_copies":5632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":275,"num_objects_unfound":0,"num_objects_dirty":2816,"num_whiteouts":0,"num_read":65946,"num_read_kb":12384683,"num_write":22134,"num_write_kb":9195051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2165,"num_bytes_recovered":7440234273,"num_keys_recovered":178,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20655927296,"data_stored":23833925674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":21867,"ondisk_log_size":21867,"up":16,"acting":15,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640314,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640314,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1639684,"kb_used_data":1583100,"kb_used_omap":0,"kb_used_meta":56576,"kb_avail":92732156,"statfs":{"total":96636764160,"available":94957727744,"internally_reserved":0,"allocated":1621094400,"data_stored":1827858700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":57933824},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.44,"15min":0.44400000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.64400000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47399999999999998,"15min":0.47999999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.69999999999999996,"5min":0.877,"15min":0.877},"last":0.40400000000000003}]},{"osd":1,"last update":"Sun Jul 2 14:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.44500000000000001,"15min":0.46600000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.436,"15min":0.44700000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.48699999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.44,"15min":0.45000000000000001},"min":{"1min":0.26500000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.47699999999999998,"15min":0.48399999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.38900000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.36599999999999999}]},{"osd":5,"last update":"Sun Jul 2 14:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.40899999999999997,"15min":0.45300000000000001},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.65400000000000003,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.44500000000000001,"15min":0.497},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.54800000000000004}]}]},{"osd":3,"up_from":133,"seq":571230650386,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4948560,"kb_used_data":4893000,"kb_used_omap":229,"kb_used_meta":55322,"kb_avail":89423280,"statfs":{"total":96636764160,"available":91569438720,"internally_reserved":0,"allocated":5010432000,"data_stored":5739736146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":56649950},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":1,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.63},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.54000000000000004}]},{"osd":4,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.59699999999999998}]},{"osd":5,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.67400000000000004}]}]},{"osd":2,"up_from":11,"seq":47244640314,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3773864,"kb_used_data":3731168,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":90597976,"statfs":{"total":96636764160,"available":92772327424,"internally_reserved":0,"allocated":3820716032,"data_stored":4365717830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.29399999999999998,"15min":0.29599999999999999},"min":{"1min":0.218,"5min":0.182,"15min":0.182},"max":{"1min":0.378,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.317,"5min":0.315,"15min":0.314},"min":{"1min":0.23000000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.39200000000000002}]},{"osd":1,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.38800000000000001,"15min":0.36199999999999999},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.2720000000000002,"5min":4.2720000000000002,"15min":4.2720000000000002},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.32300000000000001,"15min":0.32000000000000001},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.34699999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2,"5min":2,"15min":2},"last":0.377}]},{"osd":4,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.45400000000000001,"15min":0.45100000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.46899999999999997,"15min":0.46300000000000002},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.52400000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.34699999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.438}]}]},{"osd":0,"up_from":166,"seq":712964571144,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3197288,"kb_used_data":3162528,"kb_used_omap":884,"kb_used_meta":29835,"kb_avail":91174552,"statfs":{"total":96636764160,"available":93362741248,"internally_reserved":0,"allocated":3238428672,"data_stored":3664538791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":905963,"internal_metadata":30551317},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]}]},{"osd":1,"up_from":11,"seq":47244640314,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6867616,"kb_used_data":6770008,"kb_used_omap":0,"kb_used_meta":97600,"kb_avail":87504224,"statfs":{"total":96636764160,"available":89604325376,"internally_reserved":0,"allocated":6932488192,"data_stored":8162817840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":99942400},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.20200000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.32200000000000001,"15min":0.33000000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.41099999999999998,"5min":0.496,"15min":0.496},"last":0.25600000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.33500000000000002,"15min":0.33100000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.47899999999999998,"5min":1.615,"15min":1.615},"last":0.318},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.53100000000000003,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.28499999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.35599999999999998}]},{"osd":4,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47899999999999998,"15min":0.5},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.495,"15min":0.503},"min":{"1min":0.308,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71999999999999997,"5min":1.524,"15min":1.524},"last":0.39000000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.51500000000000001},"min":{"1min":0.33400000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.73899999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54800000000000004,"15min":0.54600000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.77600000000000002,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.46600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3248918528,"data_stored":3676420821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6949228544,"data_stored":8200825543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3825262592,"data_stored":4378559006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5008130048,"data_stored":5738833548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1624387584,"data_stored":1839286756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:06:30.228 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:30.229 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:30.229 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:30.229 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:30.231 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:30.231+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:30.239 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:06:30.332 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:30.331+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:30.363 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:30.364 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:30.364 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:06:30.364 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:06:30.365 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:30.374 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:06:30.433 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:30.432+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:30.533 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:30.533+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:30.540 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:30.540 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:06:30.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:06:30.540 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:30.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.541 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:06:30.541 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:06:30.541 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 209.43076630900001, 2023-07-02T14:06:30.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:06:30.541 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.541 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:30.542 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:30.542 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:30.542 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:30.542 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:06:30.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.542 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:06:30.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:06:30.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:30.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:06:30.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:06:30.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:06:30.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:06:30.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:06:30.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:06:30.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:06:30.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:06:30.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:06:30.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:06:30.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:30.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:06:30.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:06:30.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:06:30.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:06:30.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:06:30.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:06:30.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:06:30.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:06:30.551 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.551 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.551 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.551 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:06:30.551 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:06:30.551 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 209.430705571, 2023-07-02T14:06:30.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:06:30.552 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.552 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:30.552 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:30.552 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:30.552 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:30.552 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:06:30.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.553 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:06:30.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:06:30.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:30.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:06:30.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:06:30.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:06:30.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:06:30.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:06:30.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:06:30.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:06:30.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:06:30.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:06:30.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:06:30.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:30.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:06:30.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:06:30.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:06:30.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:06:30.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:06:30.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:06:30.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:30.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:06:30.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:06:30.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:30.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:06:30.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:06:30.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:30.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:06:30.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:06:30.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:30.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:06:30.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:06:30.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:06:30.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:06:30.563 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.563 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.563 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.563 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:06:30.563 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:06:30.564 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 199.20025124099999, 2023-07-02T14:06:30.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:06:30.564 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.564 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:30.564 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:30.564 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:30.564 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:30.564 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:06:30.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.565 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:06:30.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:06:30.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:30.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:06:30.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:06:30.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:06:30.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:06:30.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:06:30.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:06:30.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:06:30.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:06:30.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:06:30.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:06:30.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:06:30.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:06:30.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:30.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:06:30.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:06:30.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:30.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:06:30.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:06:30.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:30.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:06:30.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:06:30.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:30.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:06:30.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:06:30.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:06:30.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:06:30.573 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.574 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.574 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.574 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9620 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 589824~65536] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:06:30.574 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:06:30.574 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 125.516236643, 2023-07-02T14:06:30.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001, 2023-07-02T14:06:30.575 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.575 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:30.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:30.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:30.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:30.575 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9620 2023-07-02T14:06:30.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.575 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:06:30.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:06:30.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:30.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022203+0000", 2023-07-02T14:06:30.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0346000000000001e-05 2023-07-02T14:06:30.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022263+0000", 2023-07-02T14:06:30.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4657999999999999e-05 2023-07-02T14:06:30.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022278+0000", 2023-07-02T14:06:30.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:06:30.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022285+0000", 2023-07-02T14:06:30.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0096000000000003e-05 2023-07-02T14:06:30.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022335+0000", 2023-07-02T14:06:30.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2334000000000001e-05 2023-07-02T14:06:30.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:30.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022347+0000", 2023-07-02T14:06:30.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095653701999999993 2023-07-02T14:06:30.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118001+0000", 2023-07-02T14:06:30.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000185797 2023-07-02T14:06:30.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118187+0000", 2023-07-02T14:06:30.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026397900000000001 2023-07-02T14:06:30.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:30.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118451+0000", 2023-07-02T14:06:30.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051832800000000002 2023-07-02T14:06:30.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:30.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.123634+0000", 2023-07-02T14:06:30.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001227332 2023-07-02T14:06:30.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:30.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124861+0000", 2023-07-02T14:06:30.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9493999999999998e-05 2023-07-02T14:06:30.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:30.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124901+0000", 2023-07-02T14:06:30.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0462000000000001e-05 2023-07-02T14:06:30.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124931+0000", 2023-07-02T14:06:30.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001 2023-07-02T14:06:30.586 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.586 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.586 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.586 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9621 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 655360~1966080] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:06:30.587 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:06:30.587 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 125.51604926100001, 2023-07-02T14:06:30.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999, 2023-07-02T14:06:30.587 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.587 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:30.587 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:30.587 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:30.587 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:30.588 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9621 2023-07-02T14:06:30.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.588 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:06:30.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:06:30.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:30.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022391+0000", 2023-07-02T14:06:30.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032053939999999999 2023-07-02T14:06:30.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025597+0000", 2023-07-02T14:06:30.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4307999999999999e-05 2023-07-02T14:06:30.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025611+0000", 2023-07-02T14:06:30.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.032e-06 2023-07-02T14:06:30.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025617+0000", 2023-07-02T14:06:30.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8977999999999998e-05 2023-07-02T14:06:30.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025656+0000", 2023-07-02T14:06:30.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:06:30.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:30.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025663+0000", 2023-07-02T14:06:30.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095220739999999998 2023-07-02T14:06:30.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120884+0000", 2023-07-02T14:06:30.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2525000000000001e-05 2023-07-02T14:06:30.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120956+0000", 2023-07-02T14:06:30.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016232400000000001 2023-07-02T14:06:30.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:30.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.121119+0000", 2023-07-02T14:06:30.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044007380000000004 2023-07-02T14:06:30.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:30.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.125519+0000", 2023-07-02T14:06:30.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000797209 2023-07-02T14:06:30.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:30.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126317+0000", 2023-07-02T14:06:30.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1228000000000004e-05 2023-07-02T14:06:30.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:30.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126378+0000", 2023-07-02T14:06:30.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978000000000003e-05 2023-07-02T14:06:30.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126430+0000", 2023-07-02T14:06:30.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999 2023-07-02T14:06:30.597 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.598 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.598 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.598 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:189 2.3 2:c9621172:::rbd_data.125de1449605.00000000000000c0:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:06:30.598 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:06:30.598 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 117.365761969, 2023-07-02T14:06:30.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001, 2023-07-02T14:06:30.599 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.599 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:30.599 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:30.599 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:06:30.599 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:06:30.599 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:06:30.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.600 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:06:30.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:06:30.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:30.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:06:30.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2023-07-02T14:06:30.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172682+0000", 2023-07-02T14:06:30.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0159999999999997e-06 2023-07-02T14:06:30.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172688+0000", 2023-07-02T14:06:30.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0010000000000003e-06 2023-07-02T14:06:30.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172692+0000", 2023-07-02T14:06:30.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026444333 2023-07-02T14:06:30.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199136+0000", 2023-07-02T14:06:30.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2230000000000004e-05 2023-07-02T14:06:30.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199218+0000", 2023-07-02T14:06:30.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012043595000000001 2023-07-02T14:06:30.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211262+0000", 2023-07-02T14:06:30.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001 2023-07-02T14:06:30.605 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.605 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.606 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.606 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:281 2.1 2:901d5a95:::rbd_data.125de1449605.0000000000000118:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:06:30.606 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:06:30.606 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 116.98297315000001, 2023-07-02T14:06:30.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002, 2023-07-02T14:06:30.606 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.607 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:30.607 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:30.607 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:06:30.607 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:06:30.607 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:06:30.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.607 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:06:30.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:06:30.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:30.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:06:30.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0469999999999998e-06 2023-07-02T14:06:30.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555471+0000", 2023-07-02T14:06:30.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8860000000000001e-06 2023-07-02T14:06:30.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555476+0000", 2023-07-02T14:06:30.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3160000000000002e-06 2023-07-02T14:06:30.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555483+0000", 2023-07-02T14:06:30.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989525000000001 2023-07-02T14:06:30.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591472+0000", 2023-07-02T14:06:30.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3708999999999996e-05 2023-07-02T14:06:30.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591536+0000", 2023-07-02T14:06:30.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055109720000000003 2023-07-02T14:06:30.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.597047+0000", 2023-07-02T14:06:30.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002 2023-07-02T14:06:30.613 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.613 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.613 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.614 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12985 2.7 2:eeddf24b:::rbd_data.125de1449605.000000000000007e:head [write 65536~4128768] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:06:30.614 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:06:30.614 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.755446203999995, 2023-07-02T14:06:30.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999, 2023-07-02T14:06:30.614 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.614 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:30.615 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:30.615 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:30.615 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:30.615 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12985 2023-07-02T14:06:30.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.615 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:06:30.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:06:30.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:30.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782994+0000", 2023-07-02T14:06:30.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019254741999999998 2023-07-02T14:06:30.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802248+0000", 2023-07-02T14:06:30.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0451e-05 2023-07-02T14:06:30.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802279+0000", 2023-07-02T14:06:30.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3318e-05 2023-07-02T14:06:30.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802292+0000", 2023-07-02T14:06:30.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109403 2023-07-02T14:06:30.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802402+0000", 2023-07-02T14:06:30.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012520599999999999 2023-07-02T14:06:30.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802527+0000", 2023-07-02T14:06:30.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187768 2023-07-02T14:06:30.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:30.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802715+0000", 2023-07-02T14:06:30.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013192822 2023-07-02T14:06:30.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:30.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.815907+0000", 2023-07-02T14:06:30.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0086451870000000004 2023-07-02T14:06:30.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:30.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824553+0000", 2023-07-02T14:06:30.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6759999999999999e-05 2023-07-02T14:06:30.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:30.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824609+0000", 2023-07-02T14:06:30.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011314700000000001 2023-07-02T14:06:30.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824722+0000", 2023-07-02T14:06:30.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999 2023-07-02T14:06:30.624 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.624 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.624 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.625 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:165 2.3 2:c9c6393a:::rbd_data.125de1449605.000000000000002c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:06:30.625 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:06:30.625 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 93.698663386999996, 2023-07-02T14:06:30.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002, 2023-07-02T14:06:30.625 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.625 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:30.625 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:30.625 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:06:30.626 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:06:30.626 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 165 2023-07-02T14:06:30.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.626 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:06:30.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:06:30.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:30.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839777+0000", 2023-07-02T14:06:30.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6740000000000001e-06 2023-07-02T14:06:30.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839780+0000", 2023-07-02T14:06:30.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9030000000000004e-06 2023-07-02T14:06:30.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839788+0000", 2023-07-02T14:06:30.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1799999999999998e-06 2023-07-02T14:06:30.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839792+0000", 2023-07-02T14:06:30.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.031592227000000001 2023-07-02T14:06:30.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871384+0000", 2023-07-02T14:06:30.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013600099999999999 2023-07-02T14:06:30.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871520+0000", 2023-07-02T14:06:30.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082817089999999999 2023-07-02T14:06:30.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.879802+0000", 2023-07-02T14:06:30.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002 2023-07-02T14:06:30.632 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.632 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.632 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.633 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:252 2.7 2:f1d0abf0:::rbd_data.125de1449605.0000000000000083:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:06:30.633 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:06:30.633 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 93.282378417000004, 2023-07-02T14:06:30.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997, 2023-07-02T14:06:30.633 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.633 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:30.633 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:30.633 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:06:30.634 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:06:30.634 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 252 2023-07-02T14:06:30.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.634 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:06:30.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:06:30.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:30.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256062+0000", 2023-07-02T14:06:30.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0149999999999999e-06 2023-07-02T14:06:30.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256065+0000", 2023-07-02T14:06:30.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5269999999999998e-06 2023-07-02T14:06:30.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256071+0000", 2023-07-02T14:06:30.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:06:30.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256074+0000", 2023-07-02T14:06:30.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028852909 2023-07-02T14:06:30.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.284927+0000", 2023-07-02T14:06:30.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168274 2023-07-02T14:06:30.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.285096+0000", 2023-07-02T14:06:30.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011036300000000001 2023-07-02T14:06:30.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296132+0000", 2023-07-02T14:06:30.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997 2023-07-02T14:06:30.640 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.640 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.640 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.640 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:261 2.1 2:971731cb:::rbd_data.125de1449605.000000000000008c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:06:30.640 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:06:30.641 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 93.282282941999995, 2023-07-02T14:06:30.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984, 2023-07-02T14:06:30.641 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.641 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:30.641 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:30.641 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:06:30.641 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:06:30.641 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 261 2023-07-02T14:06:30.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.642 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:06:30.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:06:30.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:30.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:06:30.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2819999999999999e-06 2023-07-02T14:06:30.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256160+0000", 2023-07-02T14:06:30.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2020000000000003e-06 2023-07-02T14:06:30.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256165+0000", 2023-07-02T14:06:30.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2100000000000003e-06 2023-07-02T14:06:30.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256170+0000", 2023-07-02T14:06:30.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025871715999999999 2023-07-02T14:06:30.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282041+0000", 2023-07-02T14:06:30.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132364 2023-07-02T14:06:30.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282174+0000", 2023-07-02T14:06:30.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014044566 2023-07-02T14:06:30.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296218+0000", 2023-07-02T14:06:30.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984 2023-07-02T14:06:30.648 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.648 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.649 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.649 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:264 2.3 2:dbf06c9d:::rbd_data.125de1449605.000000000000008f:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:06:30.649 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:06:30.649 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 93.282248824000007, 2023-07-02T14:06:30.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003, 2023-07-02T14:06:30.649 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.650 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:30.650 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:30.650 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:06:30.650 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:06:30.650 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 264 2023-07-02T14:06:30.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.650 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:06:30.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:06:30.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:30.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:06:30.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5749999999999999e-06 2023-07-02T14:06:30.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256195+0000", 2023-07-02T14:06:30.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.327e-06 2023-07-02T14:06:30.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256200+0000", 2023-07-02T14:06:30.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0580000000000002e-06 2023-07-02T14:06:30.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256203+0000", 2023-07-02T14:06:30.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035707524999999997 2023-07-02T14:06:30.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.291911+0000", 2023-07-02T14:06:30.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014269799999999999 2023-07-02T14:06:30.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.292053+0000", 2023-07-02T14:06:30.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062760309999999996 2023-07-02T14:06:30.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.298329+0000", 2023-07-02T14:06:30.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003 2023-07-02T14:06:30.657 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.657 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.657 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.657 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:341 2.3 2:cee258bd:::rbd_data.125de1449605.00000000000000dc:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:06:30.658 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:06:30.658 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 92.901761496999995, 2023-07-02T14:06:30.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997, 2023-07-02T14:06:30.658 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.658 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:30.658 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:30.658 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:06:30.658 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:06:30.659 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 341 2023-07-02T14:06:30.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.659 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:06:30.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:06:30.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:30.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636679+0000", 2023-07-02T14:06:30.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6240000000000002e-06 2023-07-02T14:06:30.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636682+0000", 2023-07-02T14:06:30.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6910000000000002e-06 2023-07-02T14:06:30.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636688+0000", 2023-07-02T14:06:30.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3469999999999999e-06 2023-07-02T14:06:30.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636691+0000", 2023-07-02T14:06:30.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035625560000000001 2023-07-02T14:06:30.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672317+0000", 2023-07-02T14:06:30.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108391 2023-07-02T14:06:30.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672425+0000", 2023-07-02T14:06:30.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012104976 2023-07-02T14:06:30.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.684530+0000", 2023-07-02T14:06:30.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997 2023-07-02T14:06:30.665 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.665 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.665 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.665 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPushReply(2.7 139/134 [PushReplyOp(2:e3782f39:::rbd_data.125de1449605.0000000000000120:head)])", 2023-07-02T14:06:30.665 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:06:30.666 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 78.611229343999995, 2023-07-02T14:06:30.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998, 2023-07-02T14:06:30.666 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.666 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:30.666 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:06:30.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:06:30.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:30.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927211+0000", 2023-07-02T14:06:30.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3689999999999998e-06 2023-07-02T14:06:30.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927217+0000", 2023-07-02T14:06:30.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7269999999999996e-06 2023-07-02T14:06:30.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927225+0000", 2023-07-02T14:06:30.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:06:30.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927230+0000", 2023-07-02T14:06:30.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:06:30.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927266+0000", 2023-07-02T14:06:30.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.069972871000000006 2023-07-02T14:06:30.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.997239+0000", 2023-07-02T14:06:30.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998 2023-07-02T14:06:30.671 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.671 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.671 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.672 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19179 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2832, mlcod=177'2827)", 2023-07-02T14:06:30.672 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:06:30.672 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.079699514000001, 2023-07-02T14:06:30.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051928864999999998, 2023-07-02T14:06:30.672 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.673 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:30.673 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:06:30.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:06:30.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:30.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458741+0000", 2023-07-02T14:06:30.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01233852 2023-07-02T14:06:30.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471079+0000", 2023-07-02T14:06:30.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8131e-05 2023-07-02T14:06:30.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471097+0000", 2023-07-02T14:06:30.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.19e-06 2023-07-02T14:06:30.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471104+0000", 2023-07-02T14:06:30.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4463999999999999e-05 2023-07-02T14:06:30.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471158+0000", 2023-07-02T14:06:30.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1723000000000001e-05 2023-07-02T14:06:30.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471190+0000", 2023-07-02T14:06:30.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039438622999999999 2023-07-02T14:06:30.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:30.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510628+0000", 2023-07-02T14:06:30.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2280000000000002e-05 2023-07-02T14:06:30.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510671+0000", 2023-07-02T14:06:30.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051928864999999998 2023-07-02T14:06:30.679 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.679 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.679 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.680 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19180 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2833, mlcod=177'2827)", 2023-07-02T14:06:30.680 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:06:30.680 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.067322273999999, 2023-07-02T14:06:30.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039574109000000003, 2023-07-02T14:06:30.680 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.680 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:30.680 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:06:30.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:06:30.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:30.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471118+0000", 2023-07-02T14:06:30.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4799000000000001e-05 2023-07-02T14:06:30.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471143+0000", 2023-07-02T14:06:30.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3960000000000003e-06 2023-07-02T14:06:30.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471149+0000", 2023-07-02T14:06:30.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9230000000000004e-06 2023-07-02T14:06:30.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471153+0000", 2023-07-02T14:06:30.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015076050000000001 2023-07-02T14:06:30.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.486229+0000", 2023-07-02T14:06:30.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8466000000000002e-05 2023-07-02T14:06:30.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.486257+0000", 2023-07-02T14:06:30.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024419020999999999 2023-07-02T14:06:30.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:30.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510676+0000", 2023-07-02T14:06:30.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6693000000000001e-05 2023-07-02T14:06:30.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510693+0000", 2023-07-02T14:06:30.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039574109000000003 2023-07-02T14:06:30.687 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.687 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.687 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.687 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19181 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2834, mlcod=177'2827)", 2023-07-02T14:06:30.688 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:06:30.688 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.067281012999999, 2023-07-02T14:06:30.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039548481000000003, 2023-07-02T14:06:30.688 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.688 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:30.688 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:06:30.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:06:30.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:30.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:06:30.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8320999999999999e-05 2023-07-02T14:06:30.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471188+0000", 2023-07-02T14:06:30.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9669999999999998e-06 2023-07-02T14:06:30.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471194+0000", 2023-07-02T14:06:30.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8709999999999999e-06 2023-07-02T14:06:30.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471198+0000", 2023-07-02T14:06:30.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016760634999999999 2023-07-02T14:06:30.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.487959+0000", 2023-07-02T14:06:30.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8398000000000002e-05 2023-07-02T14:06:30.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.487987+0000", 2023-07-02T14:06:30.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022707627000000001 2023-07-02T14:06:30.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:30.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510695+0000", 2023-07-02T14:06:30.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3991999999999999e-05 2023-07-02T14:06:30.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510709+0000", 2023-07-02T14:06:30.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039548481000000003 2023-07-02T14:06:30.695 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.695 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.695 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.695 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19192 2.7 e177/141 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head v 177'2835, mlcod=177'2827)", 2023-07-02T14:06:30.695 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:06:30.696 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.067238041, 2023-07-02T14:06:30.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039521186999999999, 2023-07-02T14:06:30.696 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.696 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:30.696 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:06:30.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:06:30.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:30.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:06:30.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014525079 2023-07-02T14:06:30.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485728+0000", 2023-07-02T14:06:30.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8372e-05 2023-07-02T14:06:30.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485746+0000", 2023-07-02T14:06:30.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6149999999999997e-06 2023-07-02T14:06:30.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485755+0000", 2023-07-02T14:06:30.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036949270000000002 2023-07-02T14:06:30.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489450+0000", 2023-07-02T14:06:30.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9415e-05 2023-07-02T14:06:30.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489469+0000", 2023-07-02T14:06:30.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021241488999999999 2023-07-02T14:06:30.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:30.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510711+0000", 2023-07-02T14:06:30.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3604999999999999e-05 2023-07-02T14:06:30.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510724+0000", 2023-07-02T14:06:30.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039521186999999999 2023-07-02T14:06:30.702 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.703 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.703 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.703 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:20659 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2134016~94208] snapc 35=[35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e179)", 2023-07-02T14:06:30.703 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:06:30.703 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.948054659, 2023-07-02T14:06:30.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041695300999999997, 2023-07-02T14:06:30.704 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.704 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:30.704 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:30.704 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:30.704 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:30.704 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20659 2023-07-02T14:06:30.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.704 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:06:30.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:06:30.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:30.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590386+0000", 2023-07-02T14:06:30.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1415999999999994e-05 2023-07-02T14:06:30.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590467+0000", 2023-07-02T14:06:30.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8029999999999999e-06 2023-07-02T14:06:30.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590474+0000", 2023-07-02T14:06:30.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5379999999999998e-06 2023-07-02T14:06:30.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590478+0000", 2023-07-02T14:06:30.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073612449999999998 2023-07-02T14:06:30.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.597839+0000", 2023-07-02T14:06:30.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013387800000000001 2023-07-02T14:06:30.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.597973+0000", 2023-07-02T14:06:30.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022472799999999999 2023-07-02T14:06:30.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:30.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.598197+0000", 2023-07-02T14:06:30.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065294259999999996 2023-07-02T14:06:30.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:30.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.604727+0000", 2023-07-02T14:06:30.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027239217999999999 2023-07-02T14:06:30.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:30.712 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.631966+0000", 2023-07-02T14:06:30.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3194999999999994e-05 2023-07-02T14:06:30.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:30.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632029+0000", 2023-07-02T14:06:30.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2639999999999997e-05 2023-07-02T14:06:30.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632082+0000", 2023-07-02T14:06:30.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041695300999999997 2023-07-02T14:06:30.713 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.714 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.714 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.714 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:20660 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2031616~196608] snapc 35=[35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e179)", 2023-07-02T14:06:30.714 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:06:30.714 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.947959639, 2023-07-02T14:06:30.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041610001000000001, 2023-07-02T14:06:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20660 2023-07-02T14:06:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.715 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:30.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:30.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:06:30.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:30.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:30.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:06:30.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:30.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:30.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590481+0000", 2023-07-02T14:06:30.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012730599999999999 2023-07-02T14:06:30.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:30.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590608+0000", 2023-07-02T14:06:30.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6319999999999997e-06 2023-07-02T14:06:30.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:30.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590617+0000", 2023-07-02T14:06:30.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5209999999999998e-06 2023-07-02T14:06:30.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590621+0000", 2023-07-02T14:06:30.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018367430000000001 2023-07-02T14:06:30.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:30.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592457+0000", 2023-07-02T14:06:30.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016640599999999999 2023-07-02T14:06:30.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:30.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592624+0000", 2023-07-02T14:06:30.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028927400000000001 2023-07-02T14:06:30.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:30.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592913+0000", 2023-07-02T14:06:30.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011797813000000001 2023-07-02T14:06:30.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:30.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.604711+0000", 2023-07-02T14:06:30.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027285997999999999 2023-07-02T14:06:30.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:30.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.631997+0000", 2023-07-02T14:06:30.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8890000000000001e-05 2023-07-02T14:06:30.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:30.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632046+0000", 2023-07-02T14:06:30.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5867e-05 2023-07-02T14:06:30.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:30.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:30.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:30.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632092+0000", 2023-07-02T14:06:30.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041610001000000001 2023-07-02T14:06:30.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.724 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.724 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:30.724 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:30.724 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:30.725 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:06:30.726 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap54 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap55 b3fbb028-e267-4d25-a750-93ba18a2316a@snap55.diff 2023-07-02T14:06:30.726 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:30.689+0000 7f53f27bb700 0 -- 172.21.15.163:0/632350369 >> [v2:172.21.15.22:6808/99977,v1:172.21.15.22:6809/99977] conn(0x7f53d8008780 msgr2=0x7f53d800abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:30.727 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:30.694+0000 7f53f1fba700 0 -- 172.21.15.163:0/632350369 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7f53d8021870 msgr2=0x7f53d8023cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:30.727 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:06:30.728 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:06:30.728 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:30.725+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:30.826 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:30.825+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:30.868 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:30.868 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:30.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:30.869 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:21208 2.0 e195/193 2:1c72c23d:::rbd_object_map.125de1449605:head v 195'5244, mlcod=195'5242)", 2023-07-02T14:06:30.869 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:30.868036+0000", 2023-07-02T14:06:30.869 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.00036434599999999998, 2023-07-02T14:06:30.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00038398600000000001, 2023-07-02T14:06:30.869 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:30.869 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:30.869 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:30.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:30.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:30.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:30.868036+0000", 2023-07-02T14:06:30.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:30.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:30.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:30.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:30.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:30.868036+0000", 2023-07-02T14:06:30.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:30.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:30.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:30.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:30.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:30.868035+0000", 2023-07-02T14:06:30.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2391999999999999e-05 2023-07-02T14:06:30.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:30.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:30.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:30.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:30.868047+0000", 2023-07-02T14:06:30.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8080999999999998e-05 2023-07-02T14:06:30.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:30.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:30.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:30.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:30.868065+0000", 2023-07-02T14:06:30.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7949999999999996e-06 2023-07-02T14:06:30.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:30.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:30.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:30.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:30.868074+0000", 2023-07-02T14:06:30.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5954e-05 2023-07-02T14:06:30.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:30.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:30.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:30.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:30.868130+0000", 2023-07-02T14:06:30.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2997999999999999e-05 2023-07-02T14:06:30.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:30.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:30.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:30.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:30.868153+0000", 2023-07-02T14:06:30.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011684 2023-07-02T14:06:30.875 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:30.875 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:30.875 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:30.875 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:30.875 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2023-07-02T14:06:30.875 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 1 2023-07-02T14:06:30.875 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:30.881 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:06:30.926 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:30.926+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:30.946 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:30.947 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 100% complete...done. 2023-07-02T14:06:30.952 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:06:30.952 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=56 2023-07-02T14:06:30.953 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap56 2023-07-02T14:06:30.972 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:31.004 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:31.027 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:31.026+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:31.028 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:31.029 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:31.029 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:31.029 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:31.029 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:31.032 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:31.043 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:06:31.061 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:31.082 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:31.087 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:06:31.087 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:06:31.087 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:06:31.087 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:06:31.087 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:06:31.087 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:06:31.143 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:31.142+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:31.220 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:31.220 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:31.229 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:31.228+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:31.230 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:06:31.330 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:31.329+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:31.394 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:31.394 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:31.404 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:06:31.445 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:31.444+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:31.532 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:31.531+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:31.567 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:31.568 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:31.579 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:06:31.625 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:06:31.633 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 1 2023-07-02T14:06:31.634 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:31.632+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:31.733 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:31.732+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:31.764 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:31.764 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:31.779 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:06:31.835 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:31.834+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:31.916 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:31.916 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:31.927 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:06:31.936 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:31.935+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:32.036 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:32.035+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:32.050 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:32.051 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:32.061 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:06:32.146 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:32.145+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:32.229 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:32.229 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:32.237 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:32.237+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:32.239 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:06:32.338 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:32.337+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:32.364 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:32.364 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:32.374 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:06:32.458 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:32.457+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:32.540 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:32.539+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:32.615 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:32.615 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:32.632 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:06:32.634 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap55 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap56 b3fbb028-e267-4d25-a750-93ba18a2316a@snap56.diff 2023-07-02T14:06:32.673 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:32.672+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:32.721 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:32.720+0000 7fa18b929700 0 -- 172.21.15.163:0/3807422572 >> [v2:172.21.15.22:6808/99977,v1:172.21.15.22:6809/99977] conn(0x7fa168008780 msgr2=0x7fa16800abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:32.727 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:32.726+0000 7fa18b128700 0 -- 172.21.15.163:0/3807422572 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x563a9b6e2280 msgr2=0x563a9b6e4670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:32.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:06:32.741 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:32.740+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:32.842 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:32.841+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:32.887 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:32.887 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:32.905 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:06:32.943 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:32.942+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:32.974 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete... Exporting image: 6% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 100% complete...done. 2023-07-02T14:06:32.979 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:06:32.980 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=57 2023-07-02T14:06:32.980 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap57 2023-07-02T14:06:33.043 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:33.043+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:33.103 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:06:33.107 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:33.107 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:33.126 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:06:33.167 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:06:33.169 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:33.168+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:33.245 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:33.245+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:33.305 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:33.323 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:33.329 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:33.329 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:33.330 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:06:33.330 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:06:33.330 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:33.341 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:06:33.342 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:33.345 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:33.345+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:33.360 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:33.379 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:33.397 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:33.398 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:06:33.398 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:06:33.398 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:06:33.399 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:06:33.399 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:06:33.399 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:06:33.451 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:33.451+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:33.469 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:33.469 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:06:33.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:06:33.469 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:33.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.470 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:06:33.470 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:33.470 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 204.01510463100001, 2023-07-02T14:06:33.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:06:33.470 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:33.470 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:33.471 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:33.471 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:33.471 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:33.471 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:06:33.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.471 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:33.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:33.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:33.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:33.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.472 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.472 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:33.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:33.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:33.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:33.473 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:06:33.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:06:33.473 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.473 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:33.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:06:33.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:06:33.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:33.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:06:33.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:06:33.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:33.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:06:33.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:06:33.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:06:33.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:06:33.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:33.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:06:33.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:06:33.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:06:33.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:06:33.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:33.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:06:33.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:06:33.478 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.478 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.479 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.479 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:06:33.479 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:33.480 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 203.144411623, 2023-07-02T14:06:33.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:06:33.480 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:33.480 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:33.480 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:33.480 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:06:33.480 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:06:33.480 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:06:33.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.481 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:33.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:33.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:33.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:33.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:33.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:33.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:33.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:33.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:06:33.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:06:33.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:33.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:06:33.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:06:33.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:33.484 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:06:33.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:06:33.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.485 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:33.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:06:33.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:06:33.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:06:33.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:06:33.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:33.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:06:33.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:06:33.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:06:33.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:06:33.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:33.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:06:33.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:06:33.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:33.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:06:33.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:06:33.489 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.489 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.489 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.490 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.490 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:33.490 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:33.490 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 203.144286902, 2023-07-02T14:06:33.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:06:33.490 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:33.490 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:33.490 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:33.491 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:33.491 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:33.491 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:06:33.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.491 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:33.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:33.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:33.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:33.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:33.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:33.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:33.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:33.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:06:33.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:06:33.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:33.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:06:33.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:06:33.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:33.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:06:33.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:06:33.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:33.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:06:33.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:06:33.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:06:33.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:06:33.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:33.496 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:06:33.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:06:33.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:06:33.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:06:33.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.497 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:33.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:06:33.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:06:33.498 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.498 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.498 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.498 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.498 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.498 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:33.498 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:33.499 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 203.13743701800001, 2023-07-02T14:06:33.499 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:06:33.499 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:33.499 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:33.499 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:33.499 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:33.499 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:33.499 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:06:33.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.500 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:33.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.500 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:33.500 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:33.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:33.500 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:33.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:33.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:06:33.501 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.501 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:33.501 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:06:33.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:06:33.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:33.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:06:33.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:06:33.502 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:33.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:06:33.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:06:33.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:33.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:06:33.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:06:33.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:06:33.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:06:33.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:33.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:06:33.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:06:33.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:06:33.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:06:33.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:33.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:06:33.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:06:33.506 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.506 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.507 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.507 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:33.507 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:33.507 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 203.13737409300001, 2023-07-02T14:06:33.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:06:33.507 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:33.508 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:33.508 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:33.508 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:33.508 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:33.508 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:06:33.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.508 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:33.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:33.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:33.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:33.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:33.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:33.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:33.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:33.510 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:06:33.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:06:33.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:33.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:06:33.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:06:33.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:33.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:06:33.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:06:33.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:33.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:06:33.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:06:33.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:06:33.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:06:33.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:33.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:06:33.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:06:33.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:06:33.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:06:33.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:33.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:06:33.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:06:33.515 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.515 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.515 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.515 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:33.516 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:33.516 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 203.13707367200001, 2023-07-02T14:06:33.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:06:33.516 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:33.516 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:33.516 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:33.516 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:33.516 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:33.517 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:06:33.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.517 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:33.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:33.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:33.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:33.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:33.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:33.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:33.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:33.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:06:33.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:06:33.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:33.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:06:33.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:06:33.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:33.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:06:33.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:06:33.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:33.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:06:33.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:06:33.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:06:33.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:06:33.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:33.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:06:33.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:06:33.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:06:33.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:06:33.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:33.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:06:33.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:06:33.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:06:33.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:06:33.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:06:33.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:33.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:06:33.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:06:33.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:33.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:06:33.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:06:33.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:33.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:06:33.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:06:33.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:33.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:06:33.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:06:33.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:33.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:06:33.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:06:33.527 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.527 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.527 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.528 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:06:33.528 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:33.528 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 203.136549172, 2023-07-02T14:06:33.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:06:33.528 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:33.528 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:33.529 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:33.529 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:33.529 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:33.529 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:06:33.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.529 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:33.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:33.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:33.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:33.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:33.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:33.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:33.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:33.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:33.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:06:33.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:33.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:06:33.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:06:33.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:33.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:06:33.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:06:33.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:33.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:06:33.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:06:33.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:06:33.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:06:33.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:33.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:06:33.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:06:33.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:06:33.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:06:33.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:33.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:06:33.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:06:33.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:33.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:06:33.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:06:33.536 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.536 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.537 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.537 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:06:33.537 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:33.537 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 176.01253004700001, 2023-07-02T14:06:33.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:06:33.537 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:06:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.538 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:33.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:06:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:06:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:06:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:06:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:06:33.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:06:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:06:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:06:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:06:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:06:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:06:33.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:06:33.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:06:33.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:06:33.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:06:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:06:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.545 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:06:33.546 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:33.546 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 175.68704734299999, 2023-07-02T14:06:33.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:06:33.546 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:33.546 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:33.546 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:33.546 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:33.547 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:33.547 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:06:33.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.547 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:33.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:33.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:33.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:33.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:33.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:33.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:33.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:33.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:06:33.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:06:33.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:33.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:06:33.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:06:33.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:33.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:06:33.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:06:33.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:33.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:06:33.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:06:33.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:06:33.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:06:33.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:33.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:06:33.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:06:33.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:06:33.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:06:33.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:33.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:06:33.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:06:33.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:33.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:06:33.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:06:33.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:33.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:06:33.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:06:33.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:33.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:06:33.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:06:33.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:33.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:06:33.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:06:33.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:33.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:06:33.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:06:33.557 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.557 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.557 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.557 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:17 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 0~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:33.557 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:33.558 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.268771560999994, 2023-07-02T14:06:33.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997, 2023-07-02T14:06:33.558 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:33.558 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:33.558 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:33.558 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:33.558 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:33.559 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17 2023-07-02T14:06:33.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.559 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:33.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:33.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:33.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:33.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:33.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:33.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:06:33.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:33.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198431+0000", 2023-07-02T14:06:33.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2815e-05 2023-07-02T14:06:33.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:33.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198443+0000", 2023-07-02T14:06:33.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2569999999999999e-05 2023-07-02T14:06:33.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:33.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198466+0000", 2023-07-02T14:06:33.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0223e-05 2023-07-02T14:06:33.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:33.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198476+0000", 2023-07-02T14:06:33.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1792999999999997e-05 2023-07-02T14:06:33.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:06:33.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4249999999999998e-06 2023-07-02T14:06:33.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:33.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198536+0000", 2023-07-02T14:06:33.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91888663500000001 2023-07-02T14:06:33.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117423+0000", 2023-07-02T14:06:33.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223594 2023-07-02T14:06:33.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:33.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117647+0000", 2023-07-02T14:06:33.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033832900000000001 2023-07-02T14:06:33.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:33.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117985+0000", 2023-07-02T14:06:33.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997 2023-07-02T14:06:33.566 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.566 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.566 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.567 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:18 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 65536~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:33.567 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:33.567 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.268714349999996, 2023-07-02T14:06:33.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998, 2023-07-02T14:06:33.567 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:33.567 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:33.567 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:33.567 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:33.568 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:33.568 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18 2023-07-02T14:06:33.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.568 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:33.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:33.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:33.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:33.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:33.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:33.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:33.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:33.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:33.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7600000000000002e-06 2023-07-02T14:06:33.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:33.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198494+0000", 2023-07-02T14:06:33.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7840000000000004e-06 2023-07-02T14:06:33.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:33.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198502+0000", 2023-07-02T14:06:33.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6459999999999998e-06 2023-07-02T14:06:33.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:33.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198508+0000", 2023-07-02T14:06:33.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7395999999999997e-05 2023-07-02T14:06:33.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198565+0000", 2023-07-02T14:06:33.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6800000000000004e-06 2023-07-02T14:06:33.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:33.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198572+0000", 2023-07-02T14:06:33.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91965896800000002 2023-07-02T14:06:33.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118231+0000", 2023-07-02T14:06:33.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5212999999999998e-05 2023-07-02T14:06:33.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:33.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118326+0000", 2023-07-02T14:06:33.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000238538 2023-07-02T14:06:33.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:33.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118565+0000", 2023-07-02T14:06:33.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998 2023-07-02T14:06:33.575 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.575 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.575 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.576 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:19 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 131072~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:33.576 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:33.576 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.268690125999996, 2023-07-02T14:06:33.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002, 2023-07-02T14:06:33.576 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:33.576 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:33.577 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:33.577 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:33.577 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:33.577 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 19 2023-07-02T14:06:33.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.577 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:33.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:33.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:33.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:33.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:33.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:33.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:33.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:33.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:33.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1990000000000002e-06 2023-07-02T14:06:33.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198517+0000", 2023-07-02T14:06:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8789999999999997e-06 2023-07-02T14:06:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198524+0000", 2023-07-02T14:06:33.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:06:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:06:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5545999999999999e-05 2023-07-02T14:06:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2540000000000003e-06 2023-07-02T14:06:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198598+0000", 2023-07-02T14:06:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.919983263 2023-07-02T14:06:33.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118581+0000", 2023-07-02T14:06:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6878999999999994e-05 2023-07-02T14:06:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:33.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118648+0000", 2023-07-02T14:06:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149732 2023-07-02T14:06:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118798+0000", 2023-07-02T14:06:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002 2023-07-02T14:06:33.584 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:20 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 262144~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.268671162000004, 2023-07-02T14:06:33.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998, 2023-07-02T14:06:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20 2023-07-02T14:06:33.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.732e-06 2023-07-02T14:06:33.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198537+0000", 2023-07-02T14:06:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5259999999999999e-06 2023-07-02T14:06:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:33.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198543+0000", 2023-07-02T14:06:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.121e-06 2023-07-02T14:06:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198547+0000", 2023-07-02T14:06:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9591000000000005e-05 2023-07-02T14:06:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:06:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3120000000000004e-06 2023-07-02T14:06:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198621+0000", 2023-07-02T14:06:33.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92020789400000003 2023-07-02T14:06:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118829+0000", 2023-07-02T14:06:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022393386000000001 2023-07-02T14:06:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141223+0000", 2023-07-02T14:06:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000329866 2023-07-02T14:06:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141552+0000", 2023-07-02T14:06:33.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998 2023-07-02T14:06:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:21 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 196608~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:33.594 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.268643526999995, 2023-07-02T14:06:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995, 2023-07-02T14:06:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:33.595 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21 2023-07-02T14:06:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:33.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0169999999999999e-05 2023-07-02T14:06:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198571+0000", 2023-07-02T14:06:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3479999999999998e-06 2023-07-02T14:06:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198578+0000", 2023-07-02T14:06:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8439999999999997e-06 2023-07-02T14:06:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198583+0000", 2023-07-02T14:06:33.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8294999999999997e-05 2023-07-02T14:06:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198641+0000", 2023-07-02T14:06:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8120000000000001e-06 2023-07-02T14:06:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:06:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94293828499999999 2023-07-02T14:06:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141583+0000", 2023-07-02T14:06:33.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014607300000000001 2023-07-02T14:06:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141729+0000", 2023-07-02T14:06:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00044906600000000003 2023-07-02T14:06:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142178+0000", 2023-07-02T14:06:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995 2023-07-02T14:06:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:22 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 327680~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.268610097000007, 2023-07-02T14:06:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004, 2023-07-02T14:06:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:33.604 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 22 2023-07-02T14:06:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:33.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4850000000000001e-06 2023-07-02T14:06:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198597+0000", 2023-07-02T14:06:33.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:06:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198605+0000", 2023-07-02T14:06:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4839999999999998e-06 2023-07-02T14:06:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198609+0000", 2023-07-02T14:06:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3100999999999999e-05 2023-07-02T14:06:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198662+0000", 2023-07-02T14:06:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9099999999999998e-06 2023-07-02T14:06:33.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198666+0000", 2023-07-02T14:06:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.943539503 2023-07-02T14:06:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142206+0000", 2023-07-02T14:06:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012255999999999999 2023-07-02T14:06:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142328+0000", 2023-07-02T14:06:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208956 2023-07-02T14:06:33.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142537+0000", 2023-07-02T14:06:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004 2023-07-02T14:06:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:23 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 393216~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.268590117000002, 2023-07-02T14:06:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348, 2023-07-02T14:06:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:33.613 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23 2023-07-02T14:06:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:33.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:33.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8439999999999998e-06 2023-07-02T14:06:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:33.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:06:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0440000000000001e-06 2023-07-02T14:06:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198623+0000", 2023-07-02T14:06:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4480000000000002e-06 2023-07-02T14:06:33.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198628+0000", 2023-07-02T14:06:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5843e-05 2023-07-02T14:06:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.89e-06 2023-07-02T14:06:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198688+0000", 2023-07-02T14:06:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94389171999999999 2023-07-02T14:06:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142580+0000", 2023-07-02T14:06:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3892000000000004e-05 2023-07-02T14:06:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142644+0000", 2023-07-02T14:06:33.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017608000000000001 2023-07-02T14:06:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142820+0000", 2023-07-02T14:06:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348 2023-07-02T14:06:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.621 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:24 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 458752~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.268562043000003, 2023-07-02T14:06:33.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004, 2023-07-02T14:06:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 24 2023-07-02T14:06:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.623 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198642+0000", 2023-07-02T14:06:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.253e-06 2023-07-02T14:06:33.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:06:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1289999999999998e-06 2023-07-02T14:06:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198652+0000", 2023-07-02T14:06:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4510000000000002e-06 2023-07-02T14:06:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198657+0000", 2023-07-02T14:06:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9131000000000001e-05 2023-07-02T14:06:33.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198706+0000", 2023-07-02T14:06:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8890000000000004e-06 2023-07-02T14:06:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:33.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198710+0000", 2023-07-02T14:06:33.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94415412200000004 2023-07-02T14:06:33.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142864+0000", 2023-07-02T14:06:33.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8047000000000006e-05 2023-07-02T14:06:33.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:33.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142952+0000", 2023-07-02T14:06:33.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192967 2023-07-02T14:06:33.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:33.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143145+0000", 2023-07-02T14:06:33.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004 2023-07-02T14:06:33.631 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.631 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.631 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.631 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:25 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 524288~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:33.631 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:33.632 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.268543248, 2023-07-02T14:06:33.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001, 2023-07-02T14:06:33.632 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:33.632 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:33.632 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:33.632 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:33.632 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:33.633 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25 2023-07-02T14:06:33.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.633 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:33.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:33.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:33.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:33.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:33.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.615e-06 2023-07-02T14:06:33.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:33.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198664+0000", 2023-07-02T14:06:33.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4759999999999997e-06 2023-07-02T14:06:33.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:33.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198669+0000", 2023-07-02T14:06:33.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5999999999999998e-06 2023-07-02T14:06:33.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198673+0000", 2023-07-02T14:06:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4256999999999999e-05 2023-07-02T14:06:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198727+0000", 2023-07-02T14:06:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840000000000001e-06 2023-07-02T14:06:33.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198731+0000", 2023-07-02T14:06:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94444241900000003 2023-07-02T14:06:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143173+0000", 2023-07-02T14:06:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6839000000000002e-05 2023-07-02T14:06:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143240+0000", 2023-07-02T14:06:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017053399999999999 2023-07-02T14:06:33.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143411+0000", 2023-07-02T14:06:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001 2023-07-02T14:06:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:26 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 589824~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.268520097000007, 2023-07-02T14:06:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997, 2023-07-02T14:06:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:33.641 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 26 2023-07-02T14:06:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:33.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0520000000000002e-06 2023-07-02T14:06:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:33.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198687+0000", 2023-07-02T14:06:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1880000000000002e-06 2023-07-02T14:06:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198695+0000", 2023-07-02T14:06:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9960000000000004e-06 2023-07-02T14:06:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198699+0000", 2023-07-02T14:06:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2330999999999998e-05 2023-07-02T14:06:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198742+0000", 2023-07-02T14:06:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7950000000000001e-06 2023-07-02T14:06:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198745+0000", 2023-07-02T14:06:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94469398199999999 2023-07-02T14:06:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143439+0000", 2023-07-02T14:06:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6048e-05 2023-07-02T14:06:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:33.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143525+0000", 2023-07-02T14:06:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017016700000000001 2023-07-02T14:06:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143696+0000", 2023-07-02T14:06:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997 2023-07-02T14:06:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.649 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.6579.0:11 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected e171)", 2023-07-02T14:06:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.308721611000003, 2023-07-02T14:06:33.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998, 2023-07-02T14:06:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.6579", 2023-07-02T14:06:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/554403467", 2023-07-02T14:06:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11 2023-07-02T14:06:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.651 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:33.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158481+0000", 2023-07-02T14:06:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:06:33.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158487+0000", 2023-07-02T14:06:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7630000000000006e-06 2023-07-02T14:06:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:33.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158496+0000", 2023-07-02T14:06:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5100000000000001e-06 2023-07-02T14:06:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158504+0000", 2023-07-02T14:06:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9397000000000002e-05 2023-07-02T14:06:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158553+0000", 2023-07-02T14:06:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5810000000000003e-06 2023-07-02T14:06:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:33.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158562+0000", 2023-07-02T14:06:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93893079499999998 2023-07-02T14:06:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097493+0000", 2023-07-02T14:06:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015799299999999999 2023-07-02T14:06:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097651+0000", 2023-07-02T14:06:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187746 2023-07-02T14:06:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:33.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097839+0000", 2023-07-02T14:06:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998 2023-07-02T14:06:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:33.659 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:33.659 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:33.661 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:06:33.662 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:06:33.663 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:06:33.663 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2023-07-02T14:06:32.283567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9595488197,"num_objects":2857,"num_object_clones":2353,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":66424,"num_read_kb":12394299,"num_write":22453,"num_write_kb":9315461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2445,"num_bytes_recovered":8416021391,"num_keys_recovered":178,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22162,"ondisk_log_size":22162,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":20,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":21106972,"kb_used_data":20811828,"kb_used_omap":1114,"kb_used_meta":293989,"kb_avail":450752228,"statfs":{"total":483183820800,"available":461570281472,"internally_reserved":0,"allocated":21311311872,"data_stored":24553096753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1141261,"internal_metadata":301045235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":123191458,"num_objects":41,"num_object_clones":28,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3,"num_objects_misplaced":-275,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":478,"num_read_kb":9616,"num_write":319,"num_write_kb":120410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":975787118,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000779"},"pg_stats":[{"pgid":"2.7","version":"191'3091","reported_seq":7976,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-07-02T14:06:26.608992+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:06:26.608992+0000","last_peered":"2023-07-02T14:06:26.608992+0000","last_clean":"2023-07-02T14:06:26.608992+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:06:26.608992+0000","last_undegraded":"2023-07-02T14:06:26.608992+0000","last_fullsized":"2023-07-02T14:06:26.608992+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3091,"ondisk_log_size":3091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1276541072,"num_objects":376,"num_object_clones":311,"num_object_copies":752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":41821,"num_read_kb":1595255,"num_write":2612,"num_write_kb":1240032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"194'2510","reported_seq":4220,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-07-02T14:06:30.191821+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:06:30.191821+0000","last_peered":"2023-07-02T14:06:30.191821+0000","last_clean":"2023-07-02T14:06:30.191821+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:06:30.191821+0000","last_undegraded":"2023-07-02T14:06:30.191821+0000","last_fullsized":"2023-07-02T14:06:30.191821+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2510,"ondisk_log_size":2510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1246770286,"num_objects":386,"num_object_clones":324,"num_object_copies":772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":386,"num_whiteouts":0,"num_read":1474,"num_read_kb":1676441,"num_write":2186,"num_write_kb":1210423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"190'2629","reported_seq":3194,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-07-02T14:06:26.555573+0000","last_change":"2023-07-02T14:05:57.093448+0000","last_active":"2023-07-02T14:06:26.555573+0000","last_peered":"2023-07-02T14:06:26.555573+0000","last_clean":"2023-07-02T14:06:26.555573+0000","last_became_active":"2023-07-02T14:05:57.089226+0000","last_became_peered":"2023-07-02T14:05:57.089226+0000","last_unstale":"2023-07-02T14:06:26.555573+0000","last_undegraded":"2023-07-02T14:06:26.555573+0000","last_fullsized":"2023-07-02T14:06:26.555573+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2629,"ondisk_log_size":2629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206334412,"num_objects":333,"num_object_clones":274,"num_object_copies":666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":333,"num_whiteouts":0,"num_read":489,"num_read_kb":1499543,"num_write":2355,"num_write_kb":1165651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":860565666,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"194'5239","reported_seq":7494,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-07-02T14:06:30.186624+0000","last_change":"2023-07-02T14:06:29.572710+0000","last_active":"2023-07-02T14:06:30.186624+0000","last_peered":"2023-07-02T14:06:30.186624+0000","last_clean":"2023-07-02T14:06:30.186624+0000","last_became_active":"2023-07-02T14:06:29.569824+0000","last_became_peered":"2023-07-02T14:06:29.569824+0000","last_unstale":"2023-07-02T14:06:30.186624+0000","last_undegraded":"2023-07-02T14:06:30.186624+0000","last_fullsized":"2023-07-02T14:06:30.186624+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":5239,"ondisk_log_size":5239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1321027010,"num_objects":387,"num_object_clones":321,"num_object_copies":774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":387,"num_whiteouts":0,"num_read":18356,"num_read_kb":1708504,"num_write":7730,"num_write_kb":1291087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":123,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-07-02T14:06:29.582047+0000","last_change":"2023-07-02T14:05:53.410292+0000","last_active":"2023-07-02T14:06:29.582047+0000","last_peered":"2023-07-02T14:06:29.582047+0000","last_clean":"2023-07-02T14:06:29.582047+0000","last_became_active":"2023-07-02T14:05:53.409913+0000","last_became_peered":"2023-07-02T14:05:53.409913+0000","last_unstale":"2023-07-02T14:06:29.582047+0000","last_undegraded":"2023-07-02T14:06:29.582047+0000","last_fullsized":"2023-07-02T14:06:29.582047+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"194'2401","reported_seq":3604,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-07-02T14:06:30.172903+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:06:30.172903+0000","last_peered":"2023-07-02T14:06:30.172903+0000","last_clean":"2023-07-02T14:06:30.172903+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:06:30.172903+0000","last_undegraded":"2023-07-02T14:06:30.172903+0000","last_fullsized":"2023-07-02T14:06:30.172903+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2401,"ondisk_log_size":2401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1322644606,"num_objects":389,"num_object_clones":322,"num_object_copies":778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":389,"num_whiteouts":0,"num_read":1142,"num_read_kb":1676149,"num_write":2079,"num_write_kb":1285568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"194'2399","reported_seq":3094,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-07-02T14:06:30.167658+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:06:30.167658+0000","last_peered":"2023-07-02T14:06:30.167658+0000","last_clean":"2023-07-02T14:06:30.167658+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:06:30.167658+0000","last_undegraded":"2023-07-02T14:06:30.167658+0000","last_fullsized":"2023-07-02T14:06:30.167658+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2399,"ondisk_log_size":2399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1285702928,"num_objects":372,"num_object_clones":305,"num_object_copies":744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":372,"num_whiteouts":0,"num_read":715,"num_read_kb":1566165,"num_write":2094,"num_write_kb":1247444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"194'1930","reported_seq":3624,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-07-02T14:06:30.126570+0000","last_change":"2023-07-02T14:05:58.098047+0000","last_active":"2023-07-02T14:06:30.126570+0000","last_peered":"2023-07-02T14:06:30.126570+0000","last_clean":"2023-07-02T14:06:30.126570+0000","last_became_active":"2023-07-02T14:05:58.096421+0000","last_became_peered":"2023-07-02T14:05:58.096421+0000","last_unstale":"2023-07-02T14:06:30.126570+0000","last_undegraded":"2023-07-02T14:06:30.126570+0000","last_fullsized":"2023-07-02T14:06:30.126570+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1930,"ondisk_log_size":1930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1005323555,"num_objects":327,"num_object_clones":259,"num_object_copies":654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":1547,"num_read_kb":1421166,"num_write":1671,"num_write_kb":970498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":340635667,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"194'1963","reported_seq":2940,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-07-02T14:06:30.087229+0000","last_change":"2023-07-02T14:05:56.429093+0000","last_active":"2023-07-02T14:06:30.087229+0000","last_peered":"2023-07-02T14:06:30.087229+0000","last_clean":"2023-07-02T14:06:30.087229+0000","last_became_active":"2023-07-02T14:05:56.427016+0000","last_became_peered":"2023-07-02T14:05:56.427016+0000","last_unstale":"2023-07-02T14:06:30.087229+0000","last_undegraded":"2023-07-02T14:06:30.087229+0000","last_fullsized":"2023-07-02T14:06:30.087229+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":1963,"ondisk_log_size":1963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":931144328,"num_objects":287,"num_object_clones":237,"num_object_copies":574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":287,"num_whiteouts":0,"num_read":880,"num_read_kb":1251076,"num_write":1726,"num_write_kb":904758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":271880192,"num_keys_recovered":4,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":9595488197,"num_objects":2857,"num_object_clones":2353,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":66424,"num_read_kb":12394299,"num_write":22453,"num_write_kb":9315461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2445,"num_bytes_recovered":8416021391,"num_keys_recovered":178,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21345234944,"data_stored":24661433496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":22162,"ondisk_log_size":22162,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640315,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640315,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2639228,"kb_used_data":2572916,"kb_used_omap":0,"kb_used_meta":66304,"kb_avail":91732612,"statfs":{"total":96636764160,"available":93934194688,"internally_reserved":0,"allocated":2634665984,"data_stored":2981129193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":67895296},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.44,"15min":0.44400000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.64400000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.501},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47399999999999998,"15min":0.47999999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.69999999999999996,"5min":0.877,"15min":0.877},"last":0.53700000000000003}]},{"osd":1,"last update":"Sun Jul 2 14:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.44500000000000001,"15min":0.46600000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.436,"15min":0.44700000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.629}]},{"osd":2,"last update":"Sun Jul 2 14:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.44,"15min":0.45000000000000001},"min":{"1min":0.26500000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.47699999999999998,"15min":0.48399999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.66800000000000004}]},{"osd":3,"last update":"Sun Jul 2 14:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.48199999999999998}]},{"osd":5,"last update":"Sun Jul 2 14:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.40899999999999997,"15min":0.45300000000000001},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.65400000000000003,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.44500000000000001,"15min":0.497},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.59099999999999997}]}]},{"osd":3,"up_from":133,"seq":571230650386,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4948560,"kb_used_data":4893000,"kb_used_omap":229,"kb_used_meta":55322,"kb_avail":89423280,"statfs":{"total":96636764160,"available":91569438720,"internally_reserved":0,"allocated":5010432000,"data_stored":5739736146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":56649950},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":1,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.63},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.54000000000000004}]},{"osd":4,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.59699999999999998}]},{"osd":5,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.67400000000000004}]}]},{"osd":2,"up_from":11,"seq":47244640315,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3315532,"kb_used_data":3272260,"kb_used_omap":0,"kb_used_meta":43264,"kb_avail":91056308,"statfs":{"total":96636764160,"available":93241659392,"internally_reserved":0,"allocated":3350794240,"data_stored":3820117617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":44302336},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.29399999999999998,"15min":0.29599999999999999},"min":{"1min":0.218,"5min":0.182,"15min":0.182},"max":{"1min":0.378,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.317,"5min":0.315,"15min":0.314},"min":{"1min":0.23000000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.317}]},{"osd":1,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.38800000000000001,"15min":0.36199999999999999},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.2720000000000002,"5min":4.2720000000000002,"15min":4.2720000000000002},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.32300000000000001,"15min":0.32000000000000001},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.26000000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2,"5min":2,"15min":2},"last":0.502}]},{"osd":4,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.45400000000000001,"15min":0.45100000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.46899999999999997,"15min":0.46300000000000002},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.51800000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.34699999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.39100000000000001}]}]},{"osd":0,"up_from":166,"seq":712964571145,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3211788,"kb_used_data":3180868,"kb_used_omap":884,"kb_used_meta":30027,"kb_avail":91160052,"statfs":{"total":96636764160,"available":93347893248,"internally_reserved":0,"allocated":3257208832,"data_stored":3702503694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":905963,"internal_metadata":30747925},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640315,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6991864,"kb_used_data":6892784,"kb_used_omap":0,"kb_used_meta":99072,"kb_avail":87379976,"statfs":{"total":96636764160,"available":89477095424,"internally_reserved":0,"allocated":7058210816,"data_stored":8309610103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":101449728},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.20200000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.32200000000000001,"15min":0.33000000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.41099999999999998,"5min":0.496,"15min":0.496},"last":0.36599999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.33500000000000002,"15min":0.33100000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.47899999999999998,"5min":1.615,"15min":1.615},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.53100000000000003,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.26200000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.73699999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47899999999999998,"15min":0.5},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.625},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.495,"15min":0.503},"min":{"1min":0.308,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71999999999999997,"5min":1.524,"15min":1.524},"last":0.61599999999999999}]},{"osd":5,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.51500000000000001},"min":{"1min":0.33400000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.73899999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54800000000000004,"15min":0.54600000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.77600000000000002,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.78000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3283763200,"data_stored":3751574391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7069802496,"data_stored":8354835305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3351207936,"data_stored":3835977946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5008130048,"data_stored":5738833548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2632331264,"data_stored":2980212306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:06:33.664 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-07-02T14:06:33.665 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:06:33.665 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2023-07-02T14:06:33.665 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-07-02T14:06:33.665 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-07-02T14:06:33.666 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2023-07-02T14:06:33.666 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:33.660+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:33.761 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:33.760+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:33.796 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:33.796 INFO:teuthology.orchestra.run.smithi022.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-07-02T14:06:33.797 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:33.826 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:33.826 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:33.827 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:33.827 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:33.836 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:06:33.862 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:33.862+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:33.961 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:33.961 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:33.961 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:06:33.961 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:06:33.961 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:33.962 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:33.962+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:33.971 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:06:34.063 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:34.063+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:34.098 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:06:34.136 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:06:34.143 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 2 2023-07-02T14:06:34.159 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:34.160 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:06:34.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:06:34.160 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:34.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.160 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:06:34.160 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:06:34.161 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 213.05029841000001, 2023-07-02T14:06:34.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:06:34.161 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:34.161 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:34.161 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:34.161 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:34.161 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:34.161 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:06:34.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.162 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:34.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:34.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:06:34.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:34.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:34.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:06:34.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:34.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:34.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:06:34.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:06:34.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:34.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:06:34.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:06:34.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:34.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:06:34.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:06:34.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:06:34.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:06:34.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:06:34.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:06:34.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:34.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:06:34.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:06:34.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:06:34.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:06:34.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:34.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:06:34.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:06:34.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:34.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:06:34.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:06:34.169 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.169 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.169 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.169 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:06:34.170 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:06:34.170 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 213.05023767200001, 2023-07-02T14:06:34.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:06:34.170 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:34.170 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:34.170 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:34.170 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:34.171 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:34.171 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:06:34.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.171 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:34.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:34.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:06:34.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:34.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:34.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:06:34.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:34.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:34.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:06:34.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:06:34.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.173 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:34.173 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:06:34.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:06:34.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:34.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:06:34.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:06:34.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:06:34.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:06:34.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:06:34.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:06:34.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:34.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:06:34.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:06:34.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:06:34.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:06:34.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:34.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:06:34.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:06:34.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:34.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:06:34.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:06:34.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:34.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:06:34.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:06:34.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:34.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:06:34.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:06:34.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:34.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:06:34.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:06:34.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:34.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:06:34.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:06:34.181 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.181 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.181 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.181 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:06:34.181 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:06:34.182 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 202.81978334199999, 2023-07-02T14:06:34.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:06:34.182 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:34.182 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:34.182 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:34.182 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:34.182 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:34.182 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:06:34.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.183 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:34.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:34.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:06:34.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:34.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:34.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:06:34.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:34.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:34.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:06:34.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:06:34.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.185 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:34.185 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:06:34.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:06:34.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:34.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:06:34.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:06:34.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:06:34.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:06:34.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:06:34.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:06:34.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:34.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:06:34.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:06:34.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:34.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:06:34.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:06:34.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:34.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:06:34.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:06:34.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:34.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:06:34.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:06:34.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:34.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:06:34.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:06:34.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:34.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:06:34.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:06:34.191 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.191 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.191 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.192 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9620 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 589824~65536] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:06:34.192 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:06:34.192 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 129.13576874399999, 2023-07-02T14:06:34.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001, 2023-07-02T14:06:34.192 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:34.192 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:34.193 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:34.193 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:34.193 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:34.193 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9620 2023-07-02T14:06:34.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.193 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:34.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:34.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:06:34.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:34.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:34.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:06:34.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:34.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:34.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022203+0000", 2023-07-02T14:06:34.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0346000000000001e-05 2023-07-02T14:06:34.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:34.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022263+0000", 2023-07-02T14:06:34.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4657999999999999e-05 2023-07-02T14:06:34.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:34.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022278+0000", 2023-07-02T14:06:34.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:06:34.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022285+0000", 2023-07-02T14:06:34.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0096000000000003e-05 2023-07-02T14:06:34.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022335+0000", 2023-07-02T14:06:34.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2334000000000001e-05 2023-07-02T14:06:34.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:34.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022347+0000", 2023-07-02T14:06:34.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095653701999999993 2023-07-02T14:06:34.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118001+0000", 2023-07-02T14:06:34.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000185797 2023-07-02T14:06:34.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:34.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118187+0000", 2023-07-02T14:06:34.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026397900000000001 2023-07-02T14:06:34.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:34.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118451+0000", 2023-07-02T14:06:34.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051832800000000002 2023-07-02T14:06:34.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:34.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.123634+0000", 2023-07-02T14:06:34.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001227332 2023-07-02T14:06:34.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:34.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124861+0000", 2023-07-02T14:06:34.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9493999999999998e-05 2023-07-02T14:06:34.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:34.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124901+0000", 2023-07-02T14:06:34.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0462000000000001e-05 2023-07-02T14:06:34.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:34.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124931+0000", 2023-07-02T14:06:34.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001 2023-07-02T14:06:34.203 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.203 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.203 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.203 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9621 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 655360~1966080] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:06:34.204 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:06:34.204 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 129.13558136200001, 2023-07-02T14:06:34.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999, 2023-07-02T14:06:34.204 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:34.204 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:34.204 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:34.204 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:34.205 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:34.205 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9621 2023-07-02T14:06:34.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.205 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:34.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:34.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:06:34.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:34.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:34.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:06:34.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:34.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:34.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022391+0000", 2023-07-02T14:06:34.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032053939999999999 2023-07-02T14:06:34.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:34.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025597+0000", 2023-07-02T14:06:34.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4307999999999999e-05 2023-07-02T14:06:34.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:34.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025611+0000", 2023-07-02T14:06:34.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.032e-06 2023-07-02T14:06:34.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025617+0000", 2023-07-02T14:06:34.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8977999999999998e-05 2023-07-02T14:06:34.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025656+0000", 2023-07-02T14:06:34.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:06:34.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:34.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025663+0000", 2023-07-02T14:06:34.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095220739999999998 2023-07-02T14:06:34.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120884+0000", 2023-07-02T14:06:34.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2525000000000001e-05 2023-07-02T14:06:34.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:34.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120956+0000", 2023-07-02T14:06:34.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016232400000000001 2023-07-02T14:06:34.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:34.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.121119+0000", 2023-07-02T14:06:34.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044007380000000004 2023-07-02T14:06:34.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:34.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.125519+0000", 2023-07-02T14:06:34.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000797209 2023-07-02T14:06:34.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:34.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126317+0000", 2023-07-02T14:06:34.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1228000000000004e-05 2023-07-02T14:06:34.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:34.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126378+0000", 2023-07-02T14:06:34.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978000000000003e-05 2023-07-02T14:06:34.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:34.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126430+0000", 2023-07-02T14:06:34.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999 2023-07-02T14:06:34.215 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.215 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.215 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.215 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:189 2.3 2:c9621172:::rbd_data.125de1449605.00000000000000c0:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:06:34.215 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:06:34.215 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 120.98529406999999, 2023-07-02T14:06:34.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001, 2023-07-02T14:06:34.216 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:34.216 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:34.216 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:34.216 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:06:34.216 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:06:34.216 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:06:34.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.217 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:34.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:34.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:06:34.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:34.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:34.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:06:34.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:34.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:34.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:06:34.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2023-07-02T14:06:34.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:34.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172682+0000", 2023-07-02T14:06:34.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0159999999999997e-06 2023-07-02T14:06:34.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:34.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172688+0000", 2023-07-02T14:06:34.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0010000000000003e-06 2023-07-02T14:06:34.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172692+0000", 2023-07-02T14:06:34.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026444333 2023-07-02T14:06:34.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199136+0000", 2023-07-02T14:06:34.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2230000000000004e-05 2023-07-02T14:06:34.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:34.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199218+0000", 2023-07-02T14:06:34.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012043595000000001 2023-07-02T14:06:34.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:34.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211262+0000", 2023-07-02T14:06:34.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001 2023-07-02T14:06:34.222 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.223 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.223 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.223 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:281 2.1 2:901d5a95:::rbd_data.125de1449605.0000000000000118:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:06:34.223 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:06:34.223 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 120.602505251, 2023-07-02T14:06:34.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002, 2023-07-02T14:06:34.224 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:34.224 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:34.224 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:34.224 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:06:34.224 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:06:34.224 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:06:34.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.224 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:34.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:34.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:06:34.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:34.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:34.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:06:34.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:34.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:34.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:06:34.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0469999999999998e-06 2023-07-02T14:06:34.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:34.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555471+0000", 2023-07-02T14:06:34.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8860000000000001e-06 2023-07-02T14:06:34.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:34.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555476+0000", 2023-07-02T14:06:34.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3160000000000002e-06 2023-07-02T14:06:34.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555483+0000", 2023-07-02T14:06:34.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989525000000001 2023-07-02T14:06:34.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591472+0000", 2023-07-02T14:06:34.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3708999999999996e-05 2023-07-02T14:06:34.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:34.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591536+0000", 2023-07-02T14:06:34.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055109720000000003 2023-07-02T14:06:34.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:34.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.597047+0000", 2023-07-02T14:06:34.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002 2023-07-02T14:06:34.230 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.230 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.230 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.231 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12985 2.7 2:eeddf24b:::rbd_data.125de1449605.000000000000007e:head [write 65536~4128768] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:06:34.231 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:06:34.231 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 101.374978305, 2023-07-02T14:06:34.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999, 2023-07-02T14:06:34.231 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:34.232 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:34.232 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:34.232 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:34.232 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:34.232 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12985 2023-07-02T14:06:34.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.232 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:34.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:34.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:06:34.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:34.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:34.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:06:34.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:34.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:34.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782994+0000", 2023-07-02T14:06:34.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019254741999999998 2023-07-02T14:06:34.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:34.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802248+0000", 2023-07-02T14:06:34.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0451e-05 2023-07-02T14:06:34.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:34.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802279+0000", 2023-07-02T14:06:34.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3318e-05 2023-07-02T14:06:34.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802292+0000", 2023-07-02T14:06:34.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109403 2023-07-02T14:06:34.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802402+0000", 2023-07-02T14:06:34.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012520599999999999 2023-07-02T14:06:34.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:34.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802527+0000", 2023-07-02T14:06:34.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187768 2023-07-02T14:06:34.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:34.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802715+0000", 2023-07-02T14:06:34.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013192822 2023-07-02T14:06:34.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:34.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.815907+0000", 2023-07-02T14:06:34.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0086451870000000004 2023-07-02T14:06:34.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:34.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824553+0000", 2023-07-02T14:06:34.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6759999999999999e-05 2023-07-02T14:06:34.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:34.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824609+0000", 2023-07-02T14:06:34.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011314700000000001 2023-07-02T14:06:34.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:34.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824722+0000", 2023-07-02T14:06:34.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999 2023-07-02T14:06:34.241 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.241 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.241 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.241 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:165 2.3 2:c9c6393a:::rbd_data.125de1449605.000000000000002c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:06:34.242 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:06:34.242 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 97.318195488000001, 2023-07-02T14:06:34.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002, 2023-07-02T14:06:34.242 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:34.242 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:34.242 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:34.242 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:06:34.242 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:06:34.243 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 165 2023-07-02T14:06:34.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.243 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:34.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:34.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:06:34.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:34.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:34.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:06:34.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:34.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:34.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839777+0000", 2023-07-02T14:06:34.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6740000000000001e-06 2023-07-02T14:06:34.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:34.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839780+0000", 2023-07-02T14:06:34.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9030000000000004e-06 2023-07-02T14:06:34.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:34.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839788+0000", 2023-07-02T14:06:34.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1799999999999998e-06 2023-07-02T14:06:34.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839792+0000", 2023-07-02T14:06:34.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.031592227000000001 2023-07-02T14:06:34.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871384+0000", 2023-07-02T14:06:34.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013600099999999999 2023-07-02T14:06:34.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:34.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871520+0000", 2023-07-02T14:06:34.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082817089999999999 2023-07-02T14:06:34.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:34.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.879802+0000", 2023-07-02T14:06:34.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002 2023-07-02T14:06:34.249 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.249 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.249 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.249 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:252 2.7 2:f1d0abf0:::rbd_data.125de1449605.0000000000000083:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:06:34.249 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:06:34.249 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 96.901910517999994, 2023-07-02T14:06:34.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997, 2023-07-02T14:06:34.250 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:34.250 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:34.250 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:34.250 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:06:34.250 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:06:34.250 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 252 2023-07-02T14:06:34.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.251 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:34.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:34.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:06:34.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:34.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:34.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:06:34.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:34.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:34.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256062+0000", 2023-07-02T14:06:34.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0149999999999999e-06 2023-07-02T14:06:34.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:34.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256065+0000", 2023-07-02T14:06:34.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5269999999999998e-06 2023-07-02T14:06:34.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:34.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256071+0000", 2023-07-02T14:06:34.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:06:34.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256074+0000", 2023-07-02T14:06:34.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028852909 2023-07-02T14:06:34.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.284927+0000", 2023-07-02T14:06:34.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168274 2023-07-02T14:06:34.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:34.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.285096+0000", 2023-07-02T14:06:34.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011036300000000001 2023-07-02T14:06:34.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:34.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296132+0000", 2023-07-02T14:06:34.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997 2023-07-02T14:06:34.256 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.257 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.257 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.257 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:261 2.1 2:971731cb:::rbd_data.125de1449605.000000000000008c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:06:34.257 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:06:34.257 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 96.901815042999999, 2023-07-02T14:06:34.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984, 2023-07-02T14:06:34.258 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:34.258 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:34.258 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:34.258 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:06:34.258 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:06:34.258 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 261 2023-07-02T14:06:34.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.258 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:34.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:34.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:06:34.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:34.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:34.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:06:34.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:34.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:34.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:06:34.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2819999999999999e-06 2023-07-02T14:06:34.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:34.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256160+0000", 2023-07-02T14:06:34.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2020000000000003e-06 2023-07-02T14:06:34.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:34.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256165+0000", 2023-07-02T14:06:34.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2100000000000003e-06 2023-07-02T14:06:34.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256170+0000", 2023-07-02T14:06:34.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025871715999999999 2023-07-02T14:06:34.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282041+0000", 2023-07-02T14:06:34.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132364 2023-07-02T14:06:34.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:34.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282174+0000", 2023-07-02T14:06:34.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014044566 2023-07-02T14:06:34.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:34.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296218+0000", 2023-07-02T14:06:34.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984 2023-07-02T14:06:34.264 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.264 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.265 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.265 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:264 2.3 2:dbf06c9d:::rbd_data.125de1449605.000000000000008f:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:06:34.265 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:06:34.265 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 96.901780924999997, 2023-07-02T14:06:34.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003, 2023-07-02T14:06:34.265 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:34.266 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:34.266 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:34.266 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:06:34.266 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:06:34.266 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 264 2023-07-02T14:06:34.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.266 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:34.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:34.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:06:34.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:34.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:34.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:06:34.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:34.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:34.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:06:34.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5749999999999999e-06 2023-07-02T14:06:34.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:34.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256195+0000", 2023-07-02T14:06:34.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.327e-06 2023-07-02T14:06:34.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:34.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256200+0000", 2023-07-02T14:06:34.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0580000000000002e-06 2023-07-02T14:06:34.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256203+0000", 2023-07-02T14:06:34.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035707524999999997 2023-07-02T14:06:34.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.291911+0000", 2023-07-02T14:06:34.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014269799999999999 2023-07-02T14:06:34.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:34.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.292053+0000", 2023-07-02T14:06:34.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062760309999999996 2023-07-02T14:06:34.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:34.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.298329+0000", 2023-07-02T14:06:34.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003 2023-07-02T14:06:34.272 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.272 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.272 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.273 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:341 2.3 2:cee258bd:::rbd_data.125de1449605.00000000000000dc:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:06:34.273 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:06:34.273 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 96.521293598, 2023-07-02T14:06:34.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997, 2023-07-02T14:06:34.273 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:34.273 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:34.274 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:34.274 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:06:34.274 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:06:34.274 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 341 2023-07-02T14:06:34.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.274 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:34.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:34.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:06:34.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:34.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:34.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:06:34.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:34.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:34.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636679+0000", 2023-07-02T14:06:34.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6240000000000002e-06 2023-07-02T14:06:34.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:34.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636682+0000", 2023-07-02T14:06:34.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6910000000000002e-06 2023-07-02T14:06:34.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:34.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636688+0000", 2023-07-02T14:06:34.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3469999999999999e-06 2023-07-02T14:06:34.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636691+0000", 2023-07-02T14:06:34.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035625560000000001 2023-07-02T14:06:34.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672317+0000", 2023-07-02T14:06:34.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108391 2023-07-02T14:06:34.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:34.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672425+0000", 2023-07-02T14:06:34.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012104976 2023-07-02T14:06:34.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:34.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.684530+0000", 2023-07-02T14:06:34.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997 2023-07-02T14:06:34.280 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.280 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.280 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.281 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPushReply(2.7 139/134 [PushReplyOp(2:e3782f39:::rbd_data.125de1449605.0000000000000120:head)])", 2023-07-02T14:06:34.281 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:06:34.281 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 82.230761444999999, 2023-07-02T14:06:34.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998, 2023-07-02T14:06:34.281 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:34.281 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:34.281 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:34.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:34.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:06:34.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:34.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:34.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:06:34.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:34.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:34.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927211+0000", 2023-07-02T14:06:34.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3689999999999998e-06 2023-07-02T14:06:34.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:34.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927217+0000", 2023-07-02T14:06:34.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7269999999999996e-06 2023-07-02T14:06:34.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:34.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927225+0000", 2023-07-02T14:06:34.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:06:34.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927230+0000", 2023-07-02T14:06:34.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:06:34.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927266+0000", 2023-07-02T14:06:34.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.069972871000000006 2023-07-02T14:06:34.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:34.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.997239+0000", 2023-07-02T14:06:34.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998 2023-07-02T14:06:34.287 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.287 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.287 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.287 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19179 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2832, mlcod=177'2827)", 2023-07-02T14:06:34.287 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:06:34.287 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.699231614999999, 2023-07-02T14:06:34.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051928864999999998, 2023-07-02T14:06:34.288 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:34.288 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:34.288 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:34.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:34.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:06:34.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:34.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:34.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:06:34.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:34.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:34.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458741+0000", 2023-07-02T14:06:34.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01233852 2023-07-02T14:06:34.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:34.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471079+0000", 2023-07-02T14:06:34.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8131e-05 2023-07-02T14:06:34.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:34.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471097+0000", 2023-07-02T14:06:34.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.19e-06 2023-07-02T14:06:34.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471104+0000", 2023-07-02T14:06:34.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4463999999999999e-05 2023-07-02T14:06:34.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471158+0000", 2023-07-02T14:06:34.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1723000000000001e-05 2023-07-02T14:06:34.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:34.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471190+0000", 2023-07-02T14:06:34.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039438622999999999 2023-07-02T14:06:34.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:34.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510628+0000", 2023-07-02T14:06:34.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2280000000000002e-05 2023-07-02T14:06:34.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:34.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510671+0000", 2023-07-02T14:06:34.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051928864999999998 2023-07-02T14:06:34.294 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.294 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.295 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.295 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19180 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2833, mlcod=177'2827)", 2023-07-02T14:06:34.295 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:06:34.295 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.686854374999999, 2023-07-02T14:06:34.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039574109000000003, 2023-07-02T14:06:34.295 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:34.296 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:34.296 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:34.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:34.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:06:34.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:34.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:34.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:06:34.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:34.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:34.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471118+0000", 2023-07-02T14:06:34.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4799000000000001e-05 2023-07-02T14:06:34.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:34.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471143+0000", 2023-07-02T14:06:34.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3960000000000003e-06 2023-07-02T14:06:34.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:34.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471149+0000", 2023-07-02T14:06:34.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9230000000000004e-06 2023-07-02T14:06:34.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471153+0000", 2023-07-02T14:06:34.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015076050000000001 2023-07-02T14:06:34.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.486229+0000", 2023-07-02T14:06:34.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8466000000000002e-05 2023-07-02T14:06:34.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:34.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.486257+0000", 2023-07-02T14:06:34.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024419020999999999 2023-07-02T14:06:34.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:34.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510676+0000", 2023-07-02T14:06:34.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6693000000000001e-05 2023-07-02T14:06:34.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:34.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510693+0000", 2023-07-02T14:06:34.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039574109000000003 2023-07-02T14:06:34.302 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.302 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.302 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.303 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19181 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2834, mlcod=177'2827)", 2023-07-02T14:06:34.303 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:06:34.303 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.686813114, 2023-07-02T14:06:34.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039548481000000003, 2023-07-02T14:06:34.303 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:34.303 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:34.304 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:34.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:34.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:06:34.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:34.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:06:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:06:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8320999999999999e-05 2023-07-02T14:06:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:34.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471188+0000", 2023-07-02T14:06:34.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9669999999999998e-06 2023-07-02T14:06:34.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:34.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471194+0000", 2023-07-02T14:06:34.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8709999999999999e-06 2023-07-02T14:06:34.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471198+0000", 2023-07-02T14:06:34.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016760634999999999 2023-07-02T14:06:34.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.487959+0000", 2023-07-02T14:06:34.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8398000000000002e-05 2023-07-02T14:06:34.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:34.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.487987+0000", 2023-07-02T14:06:34.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022707627000000001 2023-07-02T14:06:34.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:34.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510695+0000", 2023-07-02T14:06:34.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3991999999999999e-05 2023-07-02T14:06:34.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:34.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510709+0000", 2023-07-02T14:06:34.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039548481000000003 2023-07-02T14:06:34.310 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.310 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.310 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.311 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19192 2.7 e177/141 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head v 177'2835, mlcod=177'2827)", 2023-07-02T14:06:34.311 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:06:34.311 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.686770142, 2023-07-02T14:06:34.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039521186999999999, 2023-07-02T14:06:34.311 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:34.311 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:34.311 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:34.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:34.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:06:34.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:34.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:34.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:06:34.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:34.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:34.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:06:34.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014525079 2023-07-02T14:06:34.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:34.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485728+0000", 2023-07-02T14:06:34.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8372e-05 2023-07-02T14:06:34.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:34.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485746+0000", 2023-07-02T14:06:34.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6149999999999997e-06 2023-07-02T14:06:34.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485755+0000", 2023-07-02T14:06:34.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036949270000000002 2023-07-02T14:06:34.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489450+0000", 2023-07-02T14:06:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9415e-05 2023-07-02T14:06:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489469+0000", 2023-07-02T14:06:34.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021241488999999999 2023-07-02T14:06:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510711+0000", 2023-07-02T14:06:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3604999999999999e-05 2023-07-02T14:06:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510724+0000", 2023-07-02T14:06:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039521186999999999 2023-07-02T14:06:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.318 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:20659 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2134016~94208] snapc 35=[35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e179)", 2023-07-02T14:06:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:06:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.567586760000001, 2023-07-02T14:06:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041695300999999997, 2023-07-02T14:06:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:34.319 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:34.320 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20659 2023-07-02T14:06:34.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.320 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:34.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:34.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:06:34.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:34.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:34.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:06:34.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:34.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:34.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590386+0000", 2023-07-02T14:06:34.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1415999999999994e-05 2023-07-02T14:06:34.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:34.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590467+0000", 2023-07-02T14:06:34.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8029999999999999e-06 2023-07-02T14:06:34.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:34.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590474+0000", 2023-07-02T14:06:34.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5379999999999998e-06 2023-07-02T14:06:34.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590478+0000", 2023-07-02T14:06:34.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073612449999999998 2023-07-02T14:06:34.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.597839+0000", 2023-07-02T14:06:34.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013387800000000001 2023-07-02T14:06:34.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.597973+0000", 2023-07-02T14:06:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022472799999999999 2023-07-02T14:06:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.598197+0000", 2023-07-02T14:06:34.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065294259999999996 2023-07-02T14:06:34.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:34.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.604727+0000", 2023-07-02T14:06:34.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027239217999999999 2023-07-02T14:06:34.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:34.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.631966+0000", 2023-07-02T14:06:34.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3194999999999994e-05 2023-07-02T14:06:34.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:34.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632029+0000", 2023-07-02T14:06:34.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2639999999999997e-05 2023-07-02T14:06:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632082+0000", 2023-07-02T14:06:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041695300999999997 2023-07-02T14:06:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.328 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:20660 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2031616~196608] snapc 35=[35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e179)", 2023-07-02T14:06:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:06:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.567491740000001, 2023-07-02T14:06:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041610001000000001, 2023-07-02T14:06:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:34.329 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20660 2023-07-02T14:06:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:06:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:06:34.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590481+0000", 2023-07-02T14:06:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012730599999999999 2023-07-02T14:06:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590608+0000", 2023-07-02T14:06:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6319999999999997e-06 2023-07-02T14:06:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590617+0000", 2023-07-02T14:06:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5209999999999998e-06 2023-07-02T14:06:34.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590621+0000", 2023-07-02T14:06:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018367430000000001 2023-07-02T14:06:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:34.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592457+0000", 2023-07-02T14:06:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016640599999999999 2023-07-02T14:06:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592624+0000", 2023-07-02T14:06:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028927400000000001 2023-07-02T14:06:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592913+0000", 2023-07-02T14:06:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011797813000000001 2023-07-02T14:06:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:34.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.604711+0000", 2023-07-02T14:06:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027285997999999999 2023-07-02T14:06:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.631997+0000", 2023-07-02T14:06:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8890000000000001e-05 2023-07-02T14:06:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632046+0000", 2023-07-02T14:06:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5867e-05 2023-07-02T14:06:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632092+0000", 2023-07-02T14:06:34.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041610001000000001 2023-07-02T14:06:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:34.339 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:34.339 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:34.341 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:06:34.341 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:34.340+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:34.441 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:34.441+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:34.516 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:34.517 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:34.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.517 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22209 2.7 e197/189 2:e55aff39:::rbd_data.125de1449605.0000000000000080:head v 197'3280, mlcod=197'3273)", 2023-07-02T14:06:34.517 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.508230+0000", 2023-07-02T14:06:34.517 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0081600639999999999, 2023-07-02T14:06:34.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081866390000000008, 2023-07-02T14:06:34.518 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:34.518 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:34.518 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:34.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:34.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.508230+0000", 2023-07-02T14:06:34.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:34.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:34.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.508230+0000", 2023-07-02T14:06:34.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:34.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:34.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.508229+0000", 2023-07-02T14:06:34.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9364000000000006e-05 2023-07-02T14:06:34.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:34.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.508328+0000", 2023-07-02T14:06:34.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3164e-05 2023-07-02T14:06:34.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:34.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.508341+0000", 2023-07-02T14:06:34.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8769999999999998e-06 2023-07-02T14:06:34.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.508347+0000", 2023-07-02T14:06:34.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00074076399999999998 2023-07-02T14:06:34.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:34.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.509088+0000", 2023-07-02T14:06:34.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2846000000000001e-05 2023-07-02T14:06:34.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:34.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.509111+0000", 2023-07-02T14:06:34.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00088124799999999995 2023-07-02T14:06:34.523 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.523 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:34.523 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.524 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22213 2.7 e197/189 2:f1d0abf0:::rbd_data.125de1449605.0000000000000083:head v 197'3281, mlcod=197'3276)", 2023-07-02T14:06:34.524 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.510553+0000", 2023-07-02T14:06:34.524 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0058364330000000002, 2023-07-02T14:06:34.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059381269999999996, 2023-07-02T14:06:34.524 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:34.524 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:34.525 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:34.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:34.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.510553+0000", 2023-07-02T14:06:34.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:34.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:34.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.510553+0000", 2023-07-02T14:06:34.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:34.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:34.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.510553+0000", 2023-07-02T14:06:34.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.064e-05 2023-07-02T14:06:34.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:34.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.510563+0000", 2023-07-02T14:06:34.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2859e-05 2023-07-02T14:06:34.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:34.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.510576+0000", 2023-07-02T14:06:34.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5489999999999997e-06 2023-07-02T14:06:34.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.510582+0000", 2023-07-02T14:06:34.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0001894 2023-07-02T14:06:34.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:34.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.510771+0000", 2023-07-02T14:06:34.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0024e-05 2023-07-02T14:06:34.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:34.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.510791+0000", 2023-07-02T14:06:34.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000237641 2023-07-02T14:06:34.530 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.530 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:34.530 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.530 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22214 2.7 e197/189 2:f1d0abf0:::rbd_data.125de1449605.0000000000000083:head v 197'3282, mlcod=197'3276)", 2023-07-02T14:06:34.530 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.512093+0000", 2023-07-02T14:06:34.531 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0042972280000000002, 2023-07-02T14:06:34.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044707820000000004, 2023-07-02T14:06:34.531 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:34.531 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:34.531 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:34.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:34.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.512093+0000", 2023-07-02T14:06:34.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:34.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:34.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.512093+0000", 2023-07-02T14:06:34.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:34.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:34.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.512092+0000", 2023-07-02T14:06:34.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8649999999999997e-05 2023-07-02T14:06:34.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:34.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.512140+0000", 2023-07-02T14:06:34.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4204e-05 2023-07-02T14:06:34.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:34.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.512155+0000", 2023-07-02T14:06:34.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0260000000000002e-06 2023-07-02T14:06:34.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.512162+0000", 2023-07-02T14:06:34.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2156000000000001e-05 2023-07-02T14:06:34.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:34.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.512204+0000", 2023-07-02T14:06:34.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1004000000000001e-05 2023-07-02T14:06:34.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:34.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.512225+0000", 2023-07-02T14:06:34.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013213 2023-07-02T14:06:34.536 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.536 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:34.536 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.537 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22216 2.0 e197/193 2:04ef6660:::rbd_data.125de1449605.0000000000000095:head v 197'5536, mlcod=197'5532)", 2023-07-02T14:06:34.537 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.512649+0000", 2023-07-02T14:06:34.537 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.003740625, 2023-07-02T14:06:34.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039834529999999996, 2023-07-02T14:06:34.537 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:34.537 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:34.538 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:34.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:34.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.512649+0000", 2023-07-02T14:06:34.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:34.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:34.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.512649+0000", 2023-07-02T14:06:34.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:34.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:34.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.512648+0000", 2023-07-02T14:06:34.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4213999999999999e-05 2023-07-02T14:06:34.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:34.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.512673+0000", 2023-07-02T14:06:34.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4646e-05 2023-07-02T14:06:34.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:34.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.512687+0000", 2023-07-02T14:06:34.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1299999999999998e-06 2023-07-02T14:06:34.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.512693+0000", 2023-07-02T14:06:34.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1356999999999997e-05 2023-07-02T14:06:34.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:34.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.512735+0000", 2023-07-02T14:06:34.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2541e-05 2023-07-02T14:06:34.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:34.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.512757+0000", 2023-07-02T14:06:34.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000108119 2023-07-02T14:06:34.543 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.543 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:34.543 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.543 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.543 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2023-07-02T14:06:34.543 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 4 2023-07-02T14:06:34.544 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:34.545 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:06:34.545 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:34.544+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:34.649 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:34.646+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:34.729 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:34.729 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:34.730 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:34.730 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:34.730 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:34.747 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:06:34.748 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:34.747+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:34.848 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:34.847+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:34.928 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:34.928 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:34.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:34.929 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:34.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.929 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:06:34.929 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:34.929 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 175.02896893100001, 2023-07-02T14:06:34.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:06:34.930 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:34.930 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:34.930 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:34.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:34.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:34.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:34.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:34.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:34.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:34.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:34.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:06:34.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:06:34.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:34.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:06:34.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:06:34.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:34.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:06:34.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:06:34.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:06:34.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:06:34.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:34.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:06:34.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:06:34.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:34.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:06:34.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:06:34.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:34.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:06:34.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:06:34.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:34.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:06:34.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:06:34.937 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.937 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:34.937 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.938 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:06:34.938 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:34.938 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 175.02488321300001, 2023-07-02T14:06:34.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:06:34.938 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:34.938 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:34.938 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:34.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:34.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:34.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:34.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:34.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:34.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:34.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:34.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:06:34.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:06:34.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:34.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:06:34.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:06:34.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:34.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:06:34.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:06:34.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:06:34.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:06:34.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:34.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:06:34.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:06:34.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:34.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:06:34.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:06:34.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:34.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:06:34.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:06:34.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:34.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:06:34.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:06:34.945 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.945 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:34.945 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.946 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:06:34.946 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:34.946 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 175.02101843200001, 2023-07-02T14:06:34.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:06:34.946 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:34.946 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:34.946 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:34.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:34.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:34.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:34.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:34.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:34.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:34.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:34.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:06:34.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:06:34.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:34.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:06:34.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:06:34.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:34.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:06:34.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:06:34.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:06:34.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:06:34.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:34.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:06:34.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:06:34.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:34.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:06:34.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:06:34.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:34.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:06:34.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:06:34.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:34.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:06:34.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:06:34.953 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.953 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:34.953 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.954 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:06:34.954 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:34.954 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.98178937200001, 2023-07-02T14:06:34.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:06:34.954 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:34.954 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:34.954 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:34.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:34.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:34.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:34.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:34.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:34.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:34.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:34.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:06:34.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:06:34.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:34.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:06:34.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:06:34.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:34.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:06:34.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:06:34.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:06:34.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:06:34.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:34.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:06:34.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:06:34.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:34.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:06:34.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:06:34.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:34.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:06:34.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:06:34.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:34.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:06:34.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:06:34.961 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.961 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:34.961 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.961 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:06:34.962 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:34.962 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.97713201400001, 2023-07-02T14:06:34.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:06:34.962 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:34.962 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:34.962 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:34.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:34.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:34.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:34.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:34.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:34.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:34.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:34.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:06:34.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:06:34.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:34.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:06:34.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:06:34.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:34.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:06:34.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:06:34.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:06:34.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:06:34.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:34.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:06:34.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:06:34.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:34.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:06:34.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:06:34.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:34.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:06:34.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:06:34.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:34.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:06:34.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:06:34.969 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.969 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:34.969 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.970 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:06:34.970 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:34.970 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.97619215099999, 2023-07-02T14:06:34.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:06:34.970 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:34.970 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:34.971 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:34.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:34.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:34.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:34.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:34.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:34.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:34.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:34.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:06:34.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:06:34.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:34.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:06:34.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:06:34.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:34.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:06:34.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:06:34.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:06:34.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:06:34.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:34.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:06:34.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:06:34.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:34.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:06:34.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:06:34.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:34.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:06:34.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:06:34.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:34.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:06:34.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:06:34.977 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.977 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:34.977 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.978 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:06:34.978 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:34.978 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 174.97479427299999, 2023-07-02T14:06:34.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:06:34.978 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:34.978 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:34.979 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:34.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:34.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:34.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:34.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:34.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:34.981 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:34.981+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:34.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:34.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:34.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:06:34.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:06:34.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:34.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:06:34.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:06:34.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:34.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:06:34.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:06:34.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:06:34.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:06:34.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:34.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:06:34.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:06:34.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:34.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:06:34.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:06:34.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:34.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:06:34.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:06:34.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:34.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:06:34.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:06:34.987 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.987 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:34.988 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.988 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:06:34.988 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:34.988 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 153.564603777, 2023-07-02T14:06:34.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:06:34.988 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:34.989 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:34.989 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:34.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:34.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:34.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:34.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:34.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:34.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:34.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:34.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:06:34.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:06:34.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:34.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:06:34.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:06:34.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:34.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:06:34.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:06:34.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:34.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:06:34.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:06:34.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:34.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:06:34.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:06:34.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:34.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:06:34.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:06:34.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:34.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:06:34.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:06:34.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:34.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:06:34.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:06:34.995 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.995 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:34.995 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:34.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.996 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:34.996 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:34.996 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.198254093, 2023-07-02T14:06:34.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:06:34.996 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:34.996 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:34.996 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:34.997 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:34.997 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:34.997 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:06:34.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.997 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:34.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:34.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:34.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:34.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:34.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:34.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:34.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:34.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:06:34.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:06:34.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:34.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:34.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:34.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:06:35.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:06:35.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:06:35.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:06:35.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:06:35.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:06:35.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:06:35.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:06:35.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:06:35.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:06:35.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:06:35.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:06:35.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:06:35.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:06:35.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:06:35.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:06:35.004 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.004 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.004 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.005 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:35.005 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:35.005 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.19420674899999, 2023-07-02T14:06:35.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:06:35.005 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.005 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.006 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.006 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.006 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.006 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:06:35.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.006 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:35.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:35.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:35.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:35.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:06:35.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:06:35.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:06:35.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:06:35.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:06:35.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:06:35.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:06:35.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:06:35.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:06:35.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:06:35.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:06:35.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:06:35.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:06:35.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:06:35.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:06:35.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:06:35.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:06:35.013 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.014 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.014 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.014 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:35.014 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:35.014 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.189642924, 2023-07-02T14:06:35.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:06:35.015 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.015 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.015 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.015 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.015 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.015 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:06:35.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.016 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:35.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:35.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:35.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:06:35.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:06:35.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:06:35.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:06:35.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:06:35.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:06:35.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:06:35.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:06:35.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:06:35.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:06:35.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:06:35.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:06:35.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:06:35.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:06:35.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:06:35.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:06:35.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:06:35.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:06:35.023 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.023 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.023 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.023 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:35.023 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:35.024 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.18959305800001, 2023-07-02T14:06:35.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:06:35.024 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.024 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.024 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.024 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.024 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.025 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:06:35.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.025 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:35.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:35.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:35.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:06:35.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:06:35.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:06:35.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:06:35.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:06:35.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:06:35.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:06:35.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:06:35.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:06:35.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:06:35.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:06:35.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:06:35.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:06:35.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:06:35.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:06:35.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:06:35.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:06:35.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:06:35.032 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.032 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.032 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.033 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:35.033 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:35.033 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 152.18855406500001, 2023-07-02T14:06:35.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:06:35.033 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.033 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.033 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.034 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.034 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.034 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:06:35.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.034 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:35.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:35.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:35.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:06:35.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:06:35.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:06:35.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:06:35.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:06:35.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:06:35.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:06:35.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:06:35.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:06:35.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:06:35.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:06:35.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:06:35.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:06:35.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:06:35.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:06:35.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:06:35.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:06:35.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:06:35.041 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.041 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.041 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.042 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22312 2.0 e197/193 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 197'5590, mlcod=197'5582)", 2023-07-02T14:06:35.042 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:06:35.042 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.17810936599999999, 2023-07-02T14:06:35.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021943201999999998, 2023-07-02T14:06:35.042 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.042 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.043 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:06:35.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:06:35.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:35.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748473+0000", 2023-07-02T14:06:35.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00099631099999999994 2023-07-02T14:06:35.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749469+0000", 2023-07-02T14:06:35.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4279e-05 2023-07-02T14:06:35.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749483+0000", 2023-07-02T14:06:35.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5289999999999997e-06 2023-07-02T14:06:35.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749490+0000", 2023-07-02T14:06:35.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047909650000000003 2023-07-02T14:06:35.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.754281+0000", 2023-07-02T14:06:35.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1999000000000001e-05 2023-07-02T14:06:35.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.754303+0000", 2023-07-02T14:06:35.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016063747 2023-07-02T14:06:35.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.770367+0000", 2023-07-02T14:06:35.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0231000000000001e-05 2023-07-02T14:06:35.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.770417+0000", 2023-07-02T14:06:35.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021943201999999998 2023-07-02T14:06:35.049 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.049 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.049 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.050 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22313 2.0 e197/193 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 197'5592, mlcod=197'5587)", 2023-07-02T14:06:35.050 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:06:35.050 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.172855963, 2023-07-02T14:06:35.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026016164000000001, 2023-07-02T14:06:35.050 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.050 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.050 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:06:35.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:06:35.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999933 2023-07-02T14:06:35.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753720+0000", 2023-07-02T14:06:35.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4851999999999995e-05 2023-07-02T14:06:35.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753785+0000", 2023-07-02T14:06:35.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0216e-05 2023-07-02T14:06:35.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753796+0000", 2023-07-02T14:06:35.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4249999999999997e-06 2023-07-02T14:06:35.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753801+0000", 2023-07-02T14:06:35.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070274359999999998 2023-07-02T14:06:35.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760828+0000", 2023-07-02T14:06:35.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0975000000000002e-05 2023-07-02T14:06:35.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760849+0000", 2023-07-02T14:06:35.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018855916 2023-07-02T14:06:35.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.779705+0000", 2023-07-02T14:06:35.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8037999999999997e-05 2023-07-02T14:06:35.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.779743+0000", 2023-07-02T14:06:35.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026016164000000001 2023-07-02T14:06:35.057 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.057 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.057 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.057 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22314 2.0 e197/193 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 197'5594, mlcod=197'5587)", 2023-07-02T14:06:35.058 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:06:35.058 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.17025130899999999, 2023-07-02T14:06:35.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027004523999999999, 2023-07-02T14:06:35.058 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.058 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.058 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:06:35.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:06:35.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:35.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756331+0000", 2023-07-02T14:06:35.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4697999999999999e-05 2023-07-02T14:06:35.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756365+0000", 2023-07-02T14:06:35.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3902e-05 2023-07-02T14:06:35.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756379+0000", 2023-07-02T14:06:35.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7329999999999997e-06 2023-07-02T14:06:35.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756386+0000", 2023-07-02T14:06:35.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062861929999999998 2023-07-02T14:06:35.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.762672+0000", 2023-07-02T14:06:35.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9031e-05 2023-07-02T14:06:35.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.762711+0000", 2023-07-02T14:06:35.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020589275000000001 2023-07-02T14:06:35.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783301+0000", 2023-07-02T14:06:35.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5720999999999999e-05 2023-07-02T14:06:35.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783336+0000", 2023-07-02T14:06:35.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027004523999999999 2023-07-02T14:06:35.065 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.065 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.065 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.065 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22315 2.0 e197/193 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 197'5595, mlcod=197'5587)", 2023-07-02T14:06:35.065 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:06:35.065 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.16628203799999999, 2023-07-02T14:06:35.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023064345999999999, 2023-07-02T14:06:35.066 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.066 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.066 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:06:35.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:06:35.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:35.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760300+0000", 2023-07-02T14:06:35.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3056000000000002e-05 2023-07-02T14:06:35.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760353+0000", 2023-07-02T14:06:35.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2825e-05 2023-07-02T14:06:35.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760366+0000", 2023-07-02T14:06:35.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8290000000000002e-06 2023-07-02T14:06:35.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760373+0000", 2023-07-02T14:06:35.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075982089999999999 2023-07-02T14:06:35.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.767971+0000", 2023-07-02T14:06:35.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4096000000000001e-05 2023-07-02T14:06:35.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.767995+0000", 2023-07-02T14:06:35.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015345908 2023-07-02T14:06:35.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783341+0000", 2023-07-02T14:06:35.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4273e-05 2023-07-02T14:06:35.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783365+0000", 2023-07-02T14:06:35.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023064345999999999 2023-07-02T14:06:35.072 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.073 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.073 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.073 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22319 2.0 e197/193 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 197'5597, mlcod=197'5587)", 2023-07-02T14:06:35.073 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:06:35.073 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.162779218, 2023-07-02T14:06:35.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022862501, 2023-07-02T14:06:35.074 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.074 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.074 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:06:35.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:06:35.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:35.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763803+0000", 2023-07-02T14:06:35.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052067220000000004 2023-07-02T14:06:35.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769010+0000", 2023-07-02T14:06:35.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3249e-05 2023-07-02T14:06:35.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769023+0000", 2023-07-02T14:06:35.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1790000000000008e-06 2023-07-02T14:06:35.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769031+0000", 2023-07-02T14:06:35.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030611079999999999 2023-07-02T14:06:35.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.772092+0000", 2023-07-02T14:06:35.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3958e-05 2023-07-02T14:06:35.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.772116+0000", 2023-07-02T14:06:35.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014504064000000001 2023-07-02T14:06:35.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.786620+0000", 2023-07-02T14:06:35.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5952999999999999e-05 2023-07-02T14:06:35.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.786666+0000", 2023-07-02T14:06:35.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022862501 2023-07-02T14:06:35.080 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.080 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.080 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.081 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22335 2.0 e197/193 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 197'5602, mlcod=197'5599)", 2023-07-02T14:06:35.081 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:06:35.081 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.135708941, 2023-07-02T14:06:35.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024437743000000001, 2023-07-02T14:06:35.081 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.081 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.082 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:06:35.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:06:35.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:35.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790873+0000", 2023-07-02T14:06:35.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081672399999999992 2023-07-02T14:06:35.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799040+0000", 2023-07-02T14:06:35.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5712999999999999e-05 2023-07-02T14:06:35.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799066+0000", 2023-07-02T14:06:35.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1413999999999999e-05 2023-07-02T14:06:35.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799077+0000", 2023-07-02T14:06:35.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3102000000000007e-05 2023-07-02T14:06:35.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799140+0000", 2023-07-02T14:06:35.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.883e-05 2023-07-02T14:06:35.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799169+0000", 2023-07-02T14:06:35.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016086700999999998 2023-07-02T14:06:35.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.815256+0000", 2023-07-02T14:06:35.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5967000000000001e-05 2023-07-02T14:06:35.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.815312+0000", 2023-07-02T14:06:35.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024437743000000001 2023-07-02T14:06:35.088 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.088 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.088 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.089 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22336 2.0 e197/193 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 197'5603, mlcod=197'5599)", 2023-07-02T14:06:35.089 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:06:35.089 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.12747365499999999, 2023-07-02T14:06:35.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022713011000000002, 2023-07-02T14:06:35.089 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.089 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.090 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:06:35.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:06:35.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:35.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799108+0000", 2023-07-02T14:06:35.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00058943600000000004 2023-07-02T14:06:35.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799698+0000", 2023-07-02T14:06:35.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6205000000000002e-05 2023-07-02T14:06:35.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799714+0000", 2023-07-02T14:06:35.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9630000000000001e-06 2023-07-02T14:06:35.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799721+0000", 2023-07-02T14:06:35.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076846579999999996 2023-07-02T14:06:35.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.807405+0000", 2023-07-02T14:06:35.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8191000000000002e-05 2023-07-02T14:06:35.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.807434+0000", 2023-07-02T14:06:35.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014338020999999999 2023-07-02T14:06:35.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.821772+0000", 2023-07-02T14:06:35.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0868999999999997e-05 2023-07-02T14:06:35.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.821822+0000", 2023-07-02T14:06:35.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022713011000000002 2023-07-02T14:06:35.096 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.096 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.096 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.096 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.097 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.097 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:35.097 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:06:35.098 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:35.098+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:35.200 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:35.199+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:35.242 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:35.242 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:35.242 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:06:35.242 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:35.256 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:06:35.341 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:35.340+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:35.401 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:35.401+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:35.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:06:35.433 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:35.433 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:35.433 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:35.434 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:35.434 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:35.446 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:06:35.502 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:35.502+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:35.602 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:35.602+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:35.614 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:35.637 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:35.637 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:35.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:35.637 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:35.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.638 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:35.638 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:06:35.638 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 236.344281102, 2023-07-02T14:06:35.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:06:35.638 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.638 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.638 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.639 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.639 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.639 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:06:35.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.639 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:06:35.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:06:35.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:35.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:06:35.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:06:35.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:06:35.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:06:35.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:06:35.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:06:35.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:06:35.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:06:35.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:06:35.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:06:35.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:06:35.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:06:35.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:35.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:06:35.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:06:35.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:35.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:06:35.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:06:35.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:06:35.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:06:35.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:06:35.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:06:35.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:06:35.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:06:35.648 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.648 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.648 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.648 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:35.648 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:06:35.648 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 236.34404607900001, 2023-07-02T14:06:35.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:06:35.649 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.649 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.649 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.649 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.649 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.649 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:06:35.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.650 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:06:35.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:06:35.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:35.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:06:35.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:06:35.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:06:35.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:06:35.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:06:35.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:06:35.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:06:35.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:06:35.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:06:35.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:06:35.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:06:35.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:06:35.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:35.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:06:35.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:06:35.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:35.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:06:35.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:06:35.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:06:35.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:06:35.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:06:35.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:06:35.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:06:35.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:06:35.658 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.658 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.658 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.659 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:35.659 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:06:35.659 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 236.343846945, 2023-07-02T14:06:35.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:06:35.659 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.659 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.660 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.660 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.660 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.660 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:06:35.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.660 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:06:35.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:06:35.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:35.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:06:35.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:06:35.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:06:35.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:06:35.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:06:35.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:06:35.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:06:35.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:06:35.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:06:35.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:06:35.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:06:35.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:06:35.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:35.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:06:35.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:06:35.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:35.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:06:35.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:06:35.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:06:35.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:06:35.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:06:35.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:06:35.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:06:35.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:06:35.669 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.669 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.669 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.669 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:35.670 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:06:35.670 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 236.34348182100001, 2023-07-02T14:06:35.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:06:35.670 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.670 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.670 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.670 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.670 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.671 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:06:35.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.671 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:06:35.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:06:35.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:35.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:06:35.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:06:35.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:06:35.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:06:35.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:06:35.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:06:35.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:06:35.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:06:35.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:06:35.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:06:35.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:06:35.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:06:35.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:35.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:06:35.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:06:35.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:35.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:06:35.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:06:35.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:06:35.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:06:35.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:06:35.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:06:35.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:06:35.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:06:35.679 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.679 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.679 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.680 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:35.680 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:06:35.680 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 236.34340076000001, 2023-07-02T14:06:35.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:06:35.680 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.681 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.681 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.681 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.681 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.681 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:06:35.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.681 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:06:35.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:06:35.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:35.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:06:35.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:06:35.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:06:35.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:06:35.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:06:35.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:06:35.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:06:35.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:06:35.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:06:35.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:06:35.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:06:35.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:06:35.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:35.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:06:35.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:06:35.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:35.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:06:35.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:06:35.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:06:35.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:06:35.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:06:35.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:06:35.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:06:35.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:06:35.690 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.690 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.690 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.691 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:35.691 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:06:35.691 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 236.343309089, 2023-07-02T14:06:35.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:06:35.691 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.691 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.691 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.692 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.692 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.692 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:06:35.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.692 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:06:35.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:06:35.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:35.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:06:35.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:06:35.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:06:35.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:06:35.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:06:35.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:06:35.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:06:35.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:06:35.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:06:35.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:06:35.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:06:35.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:06:35.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:35.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:06:35.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:06:35.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:06:35.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:06:35.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:35.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:06:35.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:06:35.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:06:35.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:06:35.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:06:35.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:06:35.700 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.701 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.701 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.701 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:35.701 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:06:35.701 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 236.34296964699999, 2023-07-02T14:06:35.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:06:35.702 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.702 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.702 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.702 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.702 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.702 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:06:35.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.702 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:06:35.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:06:35.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:35.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:06:35.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:06:35.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:06:35.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:06:35.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:06:35.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:06:35.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:06:35.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:06:35.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:06:35.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:06:35.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:06:35.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:06:35.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:35.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:06:35.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:06:35.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:06:35.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:06:35.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:35.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:06:35.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:06:35.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:06:35.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:06:35.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:06:35.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:06:35.711 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.711 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.711 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.712 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:35.712 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:35.712 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 236.342783594, 2023-07-02T14:06:35.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:06:35.712 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.712 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.712 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.713 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.713 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.713 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:06:35.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.713 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:35.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:35.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:35.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:35.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:06:35.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:06:35.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:06:35.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:06:35.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:06:35.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:06:35.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:06:35.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:06:35.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:06:35.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:06:35.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:06:35.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:35.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:06:35.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:06:35.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:06:35.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:06:35.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:35.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:06:35.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:06:35.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:06:35.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:06:35.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:06:35.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:06:35.721 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.722 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.722 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.722 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:35.722 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:06:35.722 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 236.34262579200001, 2023-07-02T14:06:35.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:06:35.723 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.723 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.723 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.723 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.723 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.723 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:06:35.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.724 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:06:35.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:06:35.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:35.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:06:35.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:06:35.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:06:35.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:06:35.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:06:35.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:06:35.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:06:35.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:06:35.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:06:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:06:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:06:35.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:06:35.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:35.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:06:35.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:06:35.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:06:35.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:06:35.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:35.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:06:35.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:06:35.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:06:35.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:06:35.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:06:35.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:06:35.732 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.732 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.732 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.732 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:35.733 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:06:35.733 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 236.34243401000001, 2023-07-02T14:06:35.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:06:35.733 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.733 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.733 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.733 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.734 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.734 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:06:35.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.734 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:06:35.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:06:35.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:35.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:06:35.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:06:35.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:06:35.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:06:35.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:06:35.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:06:35.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:06:35.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:06:35.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:06:35.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:06:35.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:06:35.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:06:35.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:35.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:06:35.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:06:35.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:35.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:06:35.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:06:35.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:06:35.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:06:35.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:06:35.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:06:35.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:06:35.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:06:35.742 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.743 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.743 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.743 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:35.743 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:35.743 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 236.34225835000001, 2023-07-02T14:06:35.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:06:35.744 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.744 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.744 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.744 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.744 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.744 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:06:35.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.744 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:35.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:35.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:35.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:35.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:06:35.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:06:35.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:06:35.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:06:35.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:06:35.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:06:35.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:06:35.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:06:35.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:06:35.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:06:35.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:06:35.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:35.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:06:35.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:06:35.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:35.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:06:35.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:06:35.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:06:35.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:06:35.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:06:35.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:06:35.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:06:35.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:06:35.753 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.753 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.753 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.753 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:35.754 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:06:35.754 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 236.34216416199999, 2023-07-02T14:06:35.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:06:35.754 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.754 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.754 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.754 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.755 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.755 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:06:35.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.755 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:06:35.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:06:35.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:35.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:06:35.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:06:35.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:06:35.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:06:35.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:06:35.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:06:35.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:06:35.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:06:35.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:06:35.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:06:35.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:06:35.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:06:35.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:35.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:06:35.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:06:35.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:35.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:06:35.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:06:35.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:06:35.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:06:35.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:06:35.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:06:35.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:06:35.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:06:35.763 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.763 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.764 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.764 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:35.764 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:06:35.764 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 236.34196077600001, 2023-07-02T14:06:35.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:06:35.764 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.765 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.765 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.765 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.765 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.765 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:06:35.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.765 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:06:35.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:06:35.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:35.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:06:35.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:06:35.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:06:35.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:06:35.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:06:35.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:06:35.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:06:35.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:06:35.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:06:35.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:06:35.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:06:35.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:06:35.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:35.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:06:35.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:06:35.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:35.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:06:35.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:06:35.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:06:35.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:06:35.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:06:35.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:06:35.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:06:35.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:06:35.774 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.774 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.774 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.774 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:35.775 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:06:35.775 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 236.34181011999999, 2023-07-02T14:06:35.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:06:35.775 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.775 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.775 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.775 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.776 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.776 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:06:35.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.776 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:06:35.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:06:35.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:35.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:06:35.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:06:35.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:06:35.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:06:35.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:06:35.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:06:35.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:06:35.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:06:35.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:06:35.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:06:35.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:06:35.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:06:35.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:35.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:06:35.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:06:35.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:35.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:06:35.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:06:35.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:06:35.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:06:35.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:06:35.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:06:35.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:06:35.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:06:35.784 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.784 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.785 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.785 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:35.785 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:35.785 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 236.34173741699999, 2023-07-02T14:06:35.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:06:35.786 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.786 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.786 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.786 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.786 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.786 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:06:35.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.786 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:35.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:35.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:35.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:35.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:06:35.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:06:35.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:06:35.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:06:35.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:06:35.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:06:35.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:06:35.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:06:35.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:06:35.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:06:35.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:06:35.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:35.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:06:35.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:06:35.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:35.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:06:35.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:06:35.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:06:35.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:06:35.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:06:35.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:06:35.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:06:35.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:06:35.795 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.795 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.795 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.795 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:35.795 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:35.796 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 236.34162042700001, 2023-07-02T14:06:35.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:06:35.796 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.796 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.796 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.796 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.796 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.797 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:06:35.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.797 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:35.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:35.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:35.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:35.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:06:35.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:06:35.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:06:35.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:06:35.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:06:35.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:06:35.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:06:35.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:06:35.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:06:35.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:06:35.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:06:35.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:35.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:06:35.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:06:35.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:35.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:06:35.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:06:35.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:06:35.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:06:35.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:06:35.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:06:35.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:06:35.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:06:35.805 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.805 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.805 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.806 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:35.806 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:06:35.806 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 236.34147097600001, 2023-07-02T14:06:35.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:06:35.806 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.806 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.807 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.807 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.807 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.807 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:06:35.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.807 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:06:35.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:06:35.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:35.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:06:35.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:06:35.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:06:35.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:06:35.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:06:35.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:06:35.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:06:35.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:06:35.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:06:35.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:06:35.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:06:35.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:06:35.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:35.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:06:35.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:06:35.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:35.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:06:35.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:06:35.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:06:35.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:06:35.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:06:35.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:06:35.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:06:35.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:06:35.815 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.816 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.816 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.816 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:35.816 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:06:35.816 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 236.319995003, 2023-07-02T14:06:35.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:06:35.817 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.817 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.817 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.817 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.817 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.817 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:06:35.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.818 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:06:35.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:06:35.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:35.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:06:35.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:06:35.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:06:35.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:06:35.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:06:35.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:06:35.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:06:35.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:06:35.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:06:35.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:06:35.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:06:35.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:06:35.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:35.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:06:35.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:06:35.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:06:35.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:06:35.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:35.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:06:35.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:06:35.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:06:35.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:06:35.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:06:35.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:06:35.826 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.826 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.826 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.827 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:35.827 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:06:35.827 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 236.31986246700001, 2023-07-02T14:06:35.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:06:35.827 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.827 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.827 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.827 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.828 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.828 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:06:35.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.828 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:06:35.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:06:35.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:35.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:06:35.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:06:35.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:06:35.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:06:35.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:06:35.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:06:35.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:06:35.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:06:35.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:06:35.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:06:35.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:06:35.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:06:35.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:35.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:06:35.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:06:35.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:06:35.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:06:35.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:35.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:06:35.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:06:35.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:06:35.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:06:35.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:06:35.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:06:35.837 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.837 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.837 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.838 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:35.838 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:06:35.838 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 236.25852580599999, 2023-07-02T14:06:35.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:06:35.838 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:35.838 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:35.838 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:35.838 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:35.839 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:35.839 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:06:35.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.839 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:35.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:35.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:06:35.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:35.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:35.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:06:35.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:35.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:35.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:06:35.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:06:35.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:35.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:06:35.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:06:35.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:35.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:06:35.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:06:35.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:35.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:06:35.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:06:35.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:35.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:06:35.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:06:35.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:35.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:06:35.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:06:35.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:35.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:06:35.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:06:35.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:35.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:06:35.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:06:35.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:35.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:06:35.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:06:35.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:35.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:06:35.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:06:35.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:35.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:35.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:35.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:06:35.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:06:35.847 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.847 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.848 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.848 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:35.848 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:35.848 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:35.850 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:35.850 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:35.850 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:35.850 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:35.850 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:35.851 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:06:35.851 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:35.849+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:35.852 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:06:35.852 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:06:35.852 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:06:35.853 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:06:35.853 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:06:35.853 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:06:35.950 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:35.950+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:36.021 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:36.022 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:36.032 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:06:36.077 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:36.076+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:36.111 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:06:36.144 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap56 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap57 b3fbb028-e267-4d25-a750-93ba18a2316a@snap57.diff 2023-07-02T14:06:36.155 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:36.155+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:36.178 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:36.178 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:36.189 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:06:36.208 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:36.207+0000 7f6d93aa5700 0 -- 172.21.15.163:0/1218119754 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7f6d74026340 msgr2=0x7f6d74028730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:36.214 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:36.213+0000 7f6d94aa7700 0 -- 172.21.15.163:0/1218119754 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x561abd642f80 msgr2=0x561abd645370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-07-02T14:06:36.252 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:36.252+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:36.264 INFO:tasks.workunit.client.0.smithi163.stderr:+ sync b3fbb028-e267-4d25-a750-93ba18a2316a-mnt 2023-07-02T14:06:36.311 INFO:tasks.workunit.client.0.smithi163.stderr:+ rm -rf b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4 b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4.tar.gz 2023-07-02T14:06:36.353 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:36.353+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:36.370 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:36.370 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:36.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:06:36.454 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:36.453+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:36.523 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:36.523 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:36.533 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:06:36.577 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:36.577+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:36.656 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:36.655+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:36.664 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:36.664 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:36.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:06:36.748 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 89% complete... Exporting image: 100% complete...done. 2023-07-02T14:06:36.754 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:06:36.754 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=58 2023-07-02T14:06:36.754 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap58 2023-07-02T14:06:36.757 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:36.756+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:36.844 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:36.844 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:36.854 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:06:36.897 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:36.897+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:36.959 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:36.959+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:36.989 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:36.990 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:37.002 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:06:37.086 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:37.086+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:37.103 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:06:37.142 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:06:37.148 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 4 2023-07-02T14:06:37.160 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:37.159+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:37.171 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:37.171 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:37.181 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:06:37.261 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:37.261+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:37.337 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:37.337 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:37.348 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:06:37.361 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:37.361+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:37.431 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i++ )) 2023-07-02T14:06:37.431 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i < 10 )) 2023-07-02T14:06:37.431 INFO:tasks.workunit.client.0.smithi163.stderr:+ pv -L 10M linux-5.4.tar.gz 2023-07-02T14:06:37.462 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:37.462+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:37.512 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:37.512 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:37.522 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:06:37.566 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:37.566+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:37.647 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:37.647 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:37.657 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:06:37.663 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:37.663+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:37.764 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:37.764+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:37.821 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:37.822 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:37.832 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:06:37.853 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:06:37.877 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:37.876+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:37.966 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:37.966+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:37.974 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:37.974 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:37.984 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:06:37.990 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:38.007 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:38.025 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:38.042 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:38.061 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:38.067 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:38.066+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:38.079 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:38.080 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:06:38.080 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:06:38.080 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:06:38.080 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:06:38.081 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:06:38.081 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:06:38.149 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:38.149 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:38.150 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:38.150 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:38.150 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:38.159 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:06:38.168 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:38.168+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:38.268 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:38.268+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:38.325 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:38.325 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:38.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:06:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 178.42586542399999, 2023-07-02T14:06:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:06:38.326 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:06:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:06:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:06:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:06:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:06:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:06:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:06:38.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:06:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:06:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:06:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:06:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:06:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:06:38.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:06:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:06:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:06:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.333 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:06:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 178.421779706, 2023-07-02T14:06:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:06:38.334 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:06:38.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:06:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:06:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:06:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:06:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:06:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:06:38.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:06:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:06:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:06:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:06:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:06:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:06:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:06:38.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:06:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:06:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.341 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:06:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 178.41791492499999, 2023-07-02T14:06:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:06:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.342 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:06:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:06:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:06:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:06:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:06:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:06:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:06:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:06:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:06:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:06:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:06:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:06:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:06:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:06:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:06:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:06:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:06:38.350 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 178.37868586499999, 2023-07-02T14:06:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:06:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:38.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:38.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:38.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:06:38.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:06:38.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:06:38.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:06:38.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.354 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.354 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:06:38.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:06:38.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:06:38.355 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:06:38.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.355 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.355 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:06:38.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:06:38.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:06:38.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:06:38.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:06:38.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:06:38.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:06:38.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:06:38.358 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.358 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.358 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.358 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:06:38.358 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:38.359 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 178.37402850699999, 2023-07-02T14:06:38.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:06:38.359 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.359 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.359 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:38.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:38.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:38.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:06:38.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:06:38.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:06:38.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:06:38.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:06:38.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:06:38.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:06:38.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:06:38.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:06:38.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:06:38.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:06:38.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:06:38.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:06:38.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:06:38.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:06:38.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:06:38.365 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.366 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.366 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.366 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.366 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:06:38.366 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:38.366 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 178.37308864400001, 2023-07-02T14:06:38.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:06:38.367 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.367 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.367 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.367 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.367 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.367 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:38.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:38.368 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:38.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:06:38.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:06:38.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:06:38.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:06:38.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:06:38.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:06:38.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:06:38.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:06:38.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:06:38.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:06:38.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:06:38.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:06:38.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:06:38.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:06:38.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:06:38.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:06:38.373 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.373 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.374 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.374 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:06:38.374 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:38.374 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 178.371690766, 2023-07-02T14:06:38.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:06:38.374 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.375 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.375 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:38.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:38.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:38.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:06:38.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:06:38.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:06:38.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:06:38.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:06:38.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:06:38.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:06:38.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:06:38.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:06:38.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:06:38.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:06:38.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:06:38.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:06:38.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:06:38.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:06:38.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:06:38.381 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.381 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.382 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.382 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:06:38.382 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:38.382 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 156.96150026999999, 2023-07-02T14:06:38.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:06:38.382 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.383 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.383 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:38.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:38.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:38.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:06:38.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:06:38.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:06:38.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:06:38.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:06:38.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:06:38.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:06:38.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:06:38.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:06:38.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:06:38.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:06:38.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:06:38.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:06:38.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:06:38.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:06:38.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:06:38.389 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.389 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.389 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.390 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:38.390 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:38.390 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 155.59515058599999, 2023-07-02T14:06:38.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:06:38.390 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.390 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.391 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:38.391 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:38.391 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:38.391 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:06:38.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.391 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:38.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:38.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:38.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:06:38.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:06:38.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:06:38.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:06:38.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:06:38.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:06:38.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:06:38.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:06:38.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:06:38.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:06:38.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:06:38.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:06:38.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:38.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:06:38.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:06:38.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:06:38.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:06:38.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:06:38.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:06:38.398 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.398 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.399 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.399 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:38.399 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:38.399 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 155.591103242, 2023-07-02T14:06:38.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:06:38.399 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.400 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.400 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:38.400 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:38.400 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:38.400 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:06:38.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.400 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:38.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:38.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:38.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:38.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:06:38.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:06:38.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:06:38.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:06:38.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:06:38.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.404 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:06:38.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:06:38.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.405 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:06:38.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:06:38.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.405 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:06:38.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:06:38.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:38.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:06:38.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:06:38.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:06:38.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:06:38.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:06:38.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:06:38.407 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.408 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.408 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.408 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:38.408 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:38.408 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 155.58653941700001, 2023-07-02T14:06:38.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:06:38.409 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.409 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.409 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:38.409 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:38.409 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:38.409 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:06:38.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.410 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:38.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:38.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:38.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:06:38.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:06:38.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:06:38.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:06:38.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:06:38.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:06:38.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:06:38.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:06:38.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:06:38.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:06:38.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:06:38.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:06:38.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:38.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:06:38.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:06:38.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:06:38.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:06:38.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:06:38.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:06:38.417 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.417 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.417 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.417 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:38.417 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:38.418 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 155.586489551, 2023-07-02T14:06:38.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:06:38.418 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.418 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.418 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:38.418 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:38.418 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:38.418 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:06:38.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.419 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:38.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:38.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:38.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:06:38.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:06:38.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:06:38.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:06:38.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:06:38.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:06:38.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:06:38.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:06:38.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:06:38.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:06:38.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:06:38.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:06:38.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:38.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:06:38.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:06:38.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:06:38.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:06:38.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:06:38.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:06:38.426 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.426 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.426 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.426 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:38.427 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:38.427 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 155.58545055799999, 2023-07-02T14:06:38.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:06:38.427 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.427 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.427 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:38.427 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:38.427 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:38.428 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:06:38.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.428 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:38.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:38.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:38.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:06:38.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:06:38.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:06:38.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:06:38.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:06:38.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:06:38.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:06:38.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:06:38.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:06:38.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:06:38.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:06:38.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:06:38.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:38.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:06:38.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:06:38.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:06:38.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:06:38.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:06:38.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:06:38.435 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.435 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.435 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.436 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22312 2.0 e197/193 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 197'5590, mlcod=197'5582)", 2023-07-02T14:06:38.436 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:06:38.436 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.575005859, 2023-07-02T14:06:38.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021943201999999998, 2023-07-02T14:06:38.436 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.436 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.436 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:06:38.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:06:38.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:38.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748473+0000", 2023-07-02T14:06:38.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00099631099999999994 2023-07-02T14:06:38.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749469+0000", 2023-07-02T14:06:38.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4279e-05 2023-07-02T14:06:38.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749483+0000", 2023-07-02T14:06:38.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5289999999999997e-06 2023-07-02T14:06:38.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749490+0000", 2023-07-02T14:06:38.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047909650000000003 2023-07-02T14:06:38.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.754281+0000", 2023-07-02T14:06:38.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1999000000000001e-05 2023-07-02T14:06:38.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.754303+0000", 2023-07-02T14:06:38.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016063747 2023-07-02T14:06:38.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.770367+0000", 2023-07-02T14:06:38.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0231000000000001e-05 2023-07-02T14:06:38.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.770417+0000", 2023-07-02T14:06:38.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021943201999999998 2023-07-02T14:06:38.443 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.443 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.443 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.443 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22313 2.0 e197/193 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 197'5592, mlcod=197'5587)", 2023-07-02T14:06:38.444 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:06:38.444 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.5697524559999998, 2023-07-02T14:06:38.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026016164000000001, 2023-07-02T14:06:38.444 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.444 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.444 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:06:38.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:06:38.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999933 2023-07-02T14:06:38.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753720+0000", 2023-07-02T14:06:38.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4851999999999995e-05 2023-07-02T14:06:38.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753785+0000", 2023-07-02T14:06:38.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0216e-05 2023-07-02T14:06:38.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753796+0000", 2023-07-02T14:06:38.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4249999999999997e-06 2023-07-02T14:06:38.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753801+0000", 2023-07-02T14:06:38.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070274359999999998 2023-07-02T14:06:38.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760828+0000", 2023-07-02T14:06:38.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0975000000000002e-05 2023-07-02T14:06:38.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760849+0000", 2023-07-02T14:06:38.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018855916 2023-07-02T14:06:38.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.779705+0000", 2023-07-02T14:06:38.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8037999999999997e-05 2023-07-02T14:06:38.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.779743+0000", 2023-07-02T14:06:38.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026016164000000001 2023-07-02T14:06:38.451 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.451 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.451 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.451 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22314 2.0 e197/193 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 197'5594, mlcod=197'5587)", 2023-07-02T14:06:38.451 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:06:38.452 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.567147802, 2023-07-02T14:06:38.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027004523999999999, 2023-07-02T14:06:38.452 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.452 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.452 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:06:38.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:06:38.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:38.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756331+0000", 2023-07-02T14:06:38.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4697999999999999e-05 2023-07-02T14:06:38.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756365+0000", 2023-07-02T14:06:38.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3902e-05 2023-07-02T14:06:38.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756379+0000", 2023-07-02T14:06:38.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7329999999999997e-06 2023-07-02T14:06:38.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756386+0000", 2023-07-02T14:06:38.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062861929999999998 2023-07-02T14:06:38.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.762672+0000", 2023-07-02T14:06:38.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9031e-05 2023-07-02T14:06:38.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.762711+0000", 2023-07-02T14:06:38.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020589275000000001 2023-07-02T14:06:38.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783301+0000", 2023-07-02T14:06:38.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5720999999999999e-05 2023-07-02T14:06:38.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783336+0000", 2023-07-02T14:06:38.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027004523999999999 2023-07-02T14:06:38.459 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.459 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.459 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.459 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22315 2.0 e197/193 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 197'5595, mlcod=197'5587)", 2023-07-02T14:06:38.459 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:06:38.459 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.5631785310000001, 2023-07-02T14:06:38.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023064345999999999, 2023-07-02T14:06:38.460 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.460 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.460 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:06:38.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:06:38.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:38.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760300+0000", 2023-07-02T14:06:38.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3056000000000002e-05 2023-07-02T14:06:38.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760353+0000", 2023-07-02T14:06:38.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2825e-05 2023-07-02T14:06:38.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760366+0000", 2023-07-02T14:06:38.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8290000000000002e-06 2023-07-02T14:06:38.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760373+0000", 2023-07-02T14:06:38.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075982089999999999 2023-07-02T14:06:38.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.767971+0000", 2023-07-02T14:06:38.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4096000000000001e-05 2023-07-02T14:06:38.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.767995+0000", 2023-07-02T14:06:38.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015345908 2023-07-02T14:06:38.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783341+0000", 2023-07-02T14:06:38.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4273e-05 2023-07-02T14:06:38.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783365+0000", 2023-07-02T14:06:38.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023064345999999999 2023-07-02T14:06:38.466 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.467 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.467 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.467 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22319 2.0 e197/193 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 197'5597, mlcod=197'5587)", 2023-07-02T14:06:38.467 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:06:38.467 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.5596757110000001, 2023-07-02T14:06:38.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022862501, 2023-07-02T14:06:38.468 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.468 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.468 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:06:38.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:06:38.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:38.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763803+0000", 2023-07-02T14:06:38.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052067220000000004 2023-07-02T14:06:38.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769010+0000", 2023-07-02T14:06:38.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3249e-05 2023-07-02T14:06:38.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769023+0000", 2023-07-02T14:06:38.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1790000000000008e-06 2023-07-02T14:06:38.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769031+0000", 2023-07-02T14:06:38.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030611079999999999 2023-07-02T14:06:38.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.772092+0000", 2023-07-02T14:06:38.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3958e-05 2023-07-02T14:06:38.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.772116+0000", 2023-07-02T14:06:38.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014504064000000001 2023-07-02T14:06:38.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.786620+0000", 2023-07-02T14:06:38.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5952999999999999e-05 2023-07-02T14:06:38.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.786666+0000", 2023-07-02T14:06:38.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022862501 2023-07-02T14:06:38.474 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.474 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.475 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.475 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22335 2.0 e197/193 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 197'5602, mlcod=197'5599)", 2023-07-02T14:06:38.475 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:06:38.475 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.5326054340000002, 2023-07-02T14:06:38.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024437743000000001, 2023-07-02T14:06:38.475 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.476 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.476 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:06:38.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:06:38.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:38.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790873+0000", 2023-07-02T14:06:38.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081672399999999992 2023-07-02T14:06:38.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799040+0000", 2023-07-02T14:06:38.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5712999999999999e-05 2023-07-02T14:06:38.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799066+0000", 2023-07-02T14:06:38.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1413999999999999e-05 2023-07-02T14:06:38.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799077+0000", 2023-07-02T14:06:38.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3102000000000007e-05 2023-07-02T14:06:38.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799140+0000", 2023-07-02T14:06:38.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.883e-05 2023-07-02T14:06:38.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799169+0000", 2023-07-02T14:06:38.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016086700999999998 2023-07-02T14:06:38.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.815256+0000", 2023-07-02T14:06:38.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5967000000000001e-05 2023-07-02T14:06:38.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.815312+0000", 2023-07-02T14:06:38.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024437743000000001 2023-07-02T14:06:38.482 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.482 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.482 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.483 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22336 2.0 e197/193 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 197'5603, mlcod=197'5599)", 2023-07-02T14:06:38.483 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:06:38.483 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.524370148, 2023-07-02T14:06:38.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022713011000000002, 2023-07-02T14:06:38.483 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.483 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.484 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:06:38.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:06:38.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:38.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799108+0000", 2023-07-02T14:06:38.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00058943600000000004 2023-07-02T14:06:38.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799698+0000", 2023-07-02T14:06:38.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6205000000000002e-05 2023-07-02T14:06:38.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799714+0000", 2023-07-02T14:06:38.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9630000000000001e-06 2023-07-02T14:06:38.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799721+0000", 2023-07-02T14:06:38.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076846579999999996 2023-07-02T14:06:38.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.807405+0000", 2023-07-02T14:06:38.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8191000000000002e-05 2023-07-02T14:06:38.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.807434+0000", 2023-07-02T14:06:38.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014338020999999999 2023-07-02T14:06:38.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.821772+0000", 2023-07-02T14:06:38.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0868999999999997e-05 2023-07-02T14:06:38.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.821822+0000", 2023-07-02T14:06:38.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022713011000000002 2023-07-02T14:06:38.490 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.490 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.490 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.490 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.491 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.491 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:38.492 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:06:38.493 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:38.491+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:38.593 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:38.592+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:38.656 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:38.656 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:38.657 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:06:38.657 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:38.666 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:06:38.693 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:38.693+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:38.794 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:38.794+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:38.806 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 2, 4] out_osds: [5] dead_osds: [] live_osds: [1, 2, 4, 5, 3, 0] 2023-07-02T14:06:38.807 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:06:38.807 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2023-07-02T14:06:38.807 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-07-02T14:06:38.807 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-07-02T14:06:38.808 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2023-07-02T14:06:38.829 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:38.829 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:38.829 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:38.829 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:38.829 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:38.839 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:06:38.895 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:38.894+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:38.940 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:38.940 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "" 2023-07-02T14:06:38.940 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:38.966 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.967 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:06:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 239.674001901, 2023-07-02T14:06:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:06:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.968 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:06:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:06:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:06:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:06:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:06:38.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:06:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:06:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:06:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:06:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:06:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:06:38.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:06:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:06:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:06:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:06:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:06:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:06:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:06:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:06:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:06:38.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:06:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:06:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:06:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:06:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:06:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:06:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 239.67376687800001, 2023-07-02T14:06:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:06:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:38.979 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:06:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:06:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:06:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:06:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:06:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:06:38.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:06:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:06:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:06:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:06:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:06:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:06:38.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:06:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:06:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:06:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:06:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:06:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:06:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:06:38.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:06:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:06:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:06:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:06:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:06:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:06:38.988 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:06:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 239.673567744, 2023-07-02T14:06:38.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:06:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:06:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.990 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:06:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:38.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:06:38.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:38.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:38.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:06:38.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:06:38.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:06:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:06:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:06:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:06:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:06:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:06:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:06:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:06:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:06:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:06:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:06:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:06:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:06:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:06:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:06:38.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:06:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:06:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:06:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:38.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:06:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:06:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:38.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:06:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 239.67320262000001, 2023-07-02T14:06:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:06:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.000 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:06:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:06:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:06:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:06:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:06:39.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:06:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:06:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:06:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:06:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:06:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:06:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:06:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:06:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:06:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:06:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:06:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:06:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:06:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:06:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:06:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:06:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:06:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:06:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:06:39.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:06:39.010 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.010 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.010 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:06:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 239.67312155900001, 2023-07-02T14:06:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:06:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.011 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:06:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:06:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:06:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:39.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:06:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:06:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:06:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:06:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:06:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:06:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:06:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:06:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:06:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:06:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:06:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:06:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:06:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:06:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:06:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:06:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:06:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:06:39.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:06:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:06:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:06:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:06:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.021 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.022 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:39.022 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:06:39.022 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 239.673029888, 2023-07-02T14:06:39.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:06:39.022 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.023 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.023 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.023 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.023 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.023 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:06:39.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:06:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:06:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:06:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:06:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:06:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:06:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:06:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:06:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:06:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:06:39.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:06:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:06:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:06:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:06:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:06:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:06:39.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:06:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:06:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:06:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:06:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:06:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:06:39.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:06:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:06:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:06:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 239.67269044599999, 2023-07-02T14:06:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:06:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.034 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:06:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:06:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:06:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:39.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:06:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:06:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:06:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:06:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:06:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:06:39.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:06:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:06:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:06:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:06:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:06:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:06:39.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:06:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:06:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:06:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:06:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:06:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:06:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:06:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:06:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:06:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:06:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:39.044 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 239.672504393, 2023-07-02T14:06:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:06:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.045 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:06:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:06:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:06:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:06:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:06:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:06:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:06:39.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:06:39.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:06:39.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:06:39.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:06:39.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:06:39.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:39.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:06:39.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:06:39.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:06:39.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:06:39.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:06:39.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:06:39.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:06:39.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:06:39.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:06:39.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:06:39.054 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.054 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.054 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.055 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:39.055 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:06:39.055 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 239.67234659100001, 2023-07-02T14:06:39.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:06:39.055 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.056 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.056 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.056 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.056 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.056 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:06:39.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.056 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:06:39.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:06:39.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:39.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:06:39.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:06:39.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:06:39.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:06:39.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:06:39.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:06:39.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:06:39.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:06:39.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:06:39.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:06:39.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:06:39.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:06:39.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:39.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:06:39.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:06:39.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:06:39.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:06:39.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:06:39.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:06:39.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:06:39.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:06:39.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:06:39.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:06:39.065 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.065 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.065 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.065 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:39.066 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:06:39.066 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 239.67215480900001, 2023-07-02T14:06:39.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:06:39.066 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.066 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.066 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.066 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.067 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.067 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:06:39.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.067 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:06:39.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:06:39.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:39.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:06:39.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:06:39.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:06:39.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:06:39.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:06:39.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:06:39.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:06:39.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:06:39.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:06:39.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:06:39.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:06:39.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:06:39.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:39.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:06:39.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:06:39.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:06:39.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:06:39.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:06:39.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:06:39.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:06:39.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:06:39.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:06:39.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:06:39.076 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.076 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.076 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.077 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:39.077 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:39.077 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 239.67197914900001, 2023-07-02T14:06:39.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:06:39.077 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.077 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.077 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.078 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.078 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.078 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:06:39.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.078 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:39.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:06:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:06:39.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:06:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:06:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:06:39.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:06:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:06:39.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:06:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:06:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:06:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:06:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:06:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:06:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:06:39.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:06:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:06:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:06:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:06:39.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:06:39.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:06:39.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:06:39.089 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.089 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.089 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.089 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:39.089 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:06:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 239.67188496099999, 2023-07-02T14:06:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:06:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.090 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:06:39.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.091 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:06:39.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:06:39.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:39.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:06:39.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:06:39.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:06:39.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:06:39.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:06:39.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:06:39.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:06:39.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:06:39.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:06:39.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:06:39.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:06:39.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:06:39.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:39.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:06:39.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:06:39.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:06:39.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:06:39.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:06:39.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:06:39.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:06:39.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:06:39.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:06:39.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:06:39.099 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.099 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.100 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.100 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:39.100 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:06:39.100 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 239.67168157500001, 2023-07-02T14:06:39.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:06:39.101 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.101 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.101 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.101 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.101 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.101 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:06:39.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:06:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:06:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:06:39.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:06:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:06:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:06:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:06:39.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:06:39.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:06:39.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:06:39.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:06:39.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:06:39.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:06:39.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:06:39.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:39.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:06:39.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:06:39.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:06:39.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:06:39.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:06:39.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:06:39.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:06:39.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:06:39.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:06:39.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:06:39.110 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.111 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.111 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.111 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:39.111 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:06:39.111 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 239.67153091899999, 2023-07-02T14:06:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:06:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:06:39.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.113 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:06:39.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:06:39.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:39.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:06:39.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:06:39.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:06:39.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:06:39.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:06:39.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:06:39.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:06:39.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:06:39.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:06:39.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:06:39.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:06:39.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:06:39.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:39.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:06:39.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:06:39.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:06:39.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:06:39.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:06:39.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:06:39.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:06:39.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:06:39.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:06:39.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:06:39.122 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.122 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.122 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.123 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:39.123 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:39.123 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 239.67145821599999, 2023-07-02T14:06:39.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:06:39.123 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.123 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.123 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.124 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.124 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.124 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:06:39.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.124 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:39.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:39.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:39.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:39.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:06:39.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:06:39.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:06:39.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:06:39.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:06:39.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:06:39.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:06:39.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:06:39.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:06:39.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:06:39.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:06:39.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:39.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:06:39.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:06:39.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:06:39.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:06:39.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.131 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.131 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:06:39.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:06:39.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:06:39.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:06:39.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:06:39.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:06:39.133 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.133 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.133 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.134 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:39.134 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:39.134 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 239.67134122600001, 2023-07-02T14:06:39.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:06:39.134 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.134 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.134 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.134 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.135 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.135 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:06:39.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.135 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:39.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:39.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:39.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:39.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:06:39.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:06:39.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:06:39.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:06:39.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:06:39.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:06:39.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:06:39.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:06:39.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:06:39.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:06:39.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:06:39.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:39.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:06:39.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:06:39.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:06:39.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:06:39.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:06:39.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:06:39.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:06:39.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:06:39.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:06:39.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:06:39.146 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.146 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.146 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.147 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:39.147 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:06:39.147 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 239.67119177500001, 2023-07-02T14:06:39.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:06:39.147 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.147 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.148 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.148 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.148 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.148 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:06:39.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.148 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:06:39.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:06:39.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:39.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:06:39.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:06:39.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:06:39.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:06:39.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:06:39.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:06:39.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:06:39.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:06:39.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:06:39.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:06:39.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:06:39.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:06:39.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:39.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:06:39.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:06:39.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:06:39.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:06:39.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:06:39.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:06:39.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:06:39.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:06:39.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:06:39.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:06:39.157 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.157 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.157 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.157 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:39.157 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:06:39.158 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 239.649715802, 2023-07-02T14:06:39.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:06:39.158 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.158 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.158 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.158 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.158 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.159 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:06:39.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.159 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:06:39.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:06:39.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:39.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:06:39.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:06:39.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:06:39.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:06:39.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:06:39.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:06:39.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:06:39.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:06:39.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:06:39.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:06:39.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:06:39.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:06:39.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:39.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:06:39.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:06:39.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:06:39.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:06:39.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:06:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:06:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:06:39.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:06:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:06:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:06:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.167 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.168 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:39.168 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:06:39.168 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 239.64958326600001, 2023-07-02T14:06:39.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:06:39.168 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.169 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.169 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.169 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.169 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.169 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:06:39.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.169 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:06:39.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:06:39.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:39.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:06:39.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:06:39.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:06:39.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:06:39.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:06:39.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:06:39.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:06:39.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:06:39.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:06:39.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:06:39.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:06:39.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:06:39.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:39.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:06:39.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:06:39.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:06:39.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:06:39.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:06:39.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:06:39.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:06:39.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:06:39.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:06:39.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:06:39.178 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.178 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.178 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.178 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:39.178 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:06:39.179 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 239.58824660499999, 2023-07-02T14:06:39.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:06:39.179 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.179 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.179 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.179 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.179 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.179 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:06:39.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.180 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:06:39.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:06:39.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:39.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:06:39.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:06:39.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:06:39.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:06:39.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:06:39.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:06:39.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:06:39.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:06:39.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:06:39.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:06:39.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:06:39.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:06:39.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:39.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:06:39.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:06:39.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:06:39.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:06:39.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:06:39.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:06:39.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:06:39.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:06:39.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:06:39.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:06:39.188 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.188 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.189 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.189 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.189 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.189 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:39.190 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:06:39.191 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:39.190+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:39.192 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:06:39.291 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:39.291+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:39.358 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:39.358 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:39.358 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:06:39.359 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:39.368 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:06:39.412 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:39.411+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:39.492 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:39.491+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:39.493 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:39.493 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:39.493 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:39.493 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:39.493 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:39.503 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:06:39.593 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:39.593+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:39.671 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:39.671 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:39.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17354 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 56.364629149999999, 2023-07-02T14:06:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493, 2023-07-02T14:06:39.672 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17354 2023-07-02T14:06:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.673 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7440000000000001e-06 2023-07-02T14:06:39.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302606+0000", 2023-07-02T14:06:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:06:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302613+0000", 2023-07-02T14:06:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8810000000000004e-06 2023-07-02T14:06:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302618+0000", 2023-07-02T14:06:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028613100000000002 2023-07-02T14:06:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302904+0000", 2023-07-02T14:06:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5278999999999995e-05 2023-07-02T14:06:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302979+0000", 2023-07-02T14:06:39.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096904750000000005 2023-07-02T14:06:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312669+0000", 2023-07-02T14:06:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0328000000000002e-05 2023-07-02T14:06:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312700+0000", 2023-07-02T14:06:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098019319999999993 2023-07-02T14:06:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322502+0000", 2023-07-02T14:06:39.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4669999999999998e-05 2023-07-02T14:06:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322536+0000", 2023-07-02T14:06:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010023839 2023-07-02T14:06:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332560+0000", 2023-07-02T14:06:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5559999999999998e-05 2023-07-02T14:06:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332596+0000", 2023-07-02T14:06:39.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090401319999999993 2023-07-02T14:06:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341636+0000", 2023-07-02T14:06:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5373000000000001e-05 2023-07-02T14:06:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341681+0000", 2023-07-02T14:06:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081994059999999994 2023-07-02T14:06:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349881+0000", 2023-07-02T14:06:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4319000000000001e-05 2023-07-02T14:06:39.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349915+0000", 2023-07-02T14:06:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089388339999999997 2023-07-02T14:06:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358854+0000", 2023-07-02T14:06:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4319000000000005e-05 2023-07-02T14:06:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358928+0000", 2023-07-02T14:06:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080234690000000001 2023-07-02T14:06:39.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366952+0000", 2023-07-02T14:06:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2082999999999998e-05 2023-07-02T14:06:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366984+0000", 2023-07-02T14:06:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075015790000000004 2023-07-02T14:06:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374485+0000", 2023-07-02T14:06:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4480999999999995e-05 2023-07-02T14:06:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374550+0000", 2023-07-02T14:06:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091666620000000008 2023-07-02T14:06:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383716+0000", 2023-07-02T14:06:39.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9318000000000003e-05 2023-07-02T14:06:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383786+0000", 2023-07-02T14:06:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013434115 2023-07-02T14:06:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397220+0000", 2023-07-02T14:06:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4573999999999997e-05 2023-07-02T14:06:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397264+0000", 2023-07-02T14:06:39.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064824649999999998 2023-07-02T14:06:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403747+0000", 2023-07-02T14:06:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1087999999999993e-05 2023-07-02T14:06:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403828+0000", 2023-07-02T14:06:39.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072854449999999998 2023-07-02T14:06:39.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411113+0000", 2023-07-02T14:06:39.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5120000000000001e-05 2023-07-02T14:06:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411169+0000", 2023-07-02T14:06:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007211505 2023-07-02T14:06:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418380+0000", 2023-07-02T14:06:39.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011289399999999999 2023-07-02T14:06:39.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418493+0000", 2023-07-02T14:06:39.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00051217500000000002 2023-07-02T14:06:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.419005+0000", 2023-07-02T14:06:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00327393 2023-07-02T14:06:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422279+0000", 2023-07-02T14:06:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041432350000000003 2023-07-02T14:06:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426422+0000", 2023-07-02T14:06:39.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7021000000000003e-05 2023-07-02T14:06:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426469+0000", 2023-07-02T14:06:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000190256 2023-07-02T14:06:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426660+0000", 2023-07-02T14:06:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493 2023-07-02T14:06:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.699 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17355 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:39.700 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:39.700 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 56.364610489, 2023-07-02T14:06:39.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001, 2023-07-02T14:06:39.700 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.700 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.700 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.700 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.701 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.701 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17355 2023-07-02T14:06:39.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.701 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:39.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302621+0000", 2023-07-02T14:06:39.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2260000000000001e-06 2023-07-02T14:06:39.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302625+0000", 2023-07-02T14:06:39.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8599999999999998e-06 2023-07-02T14:06:39.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302630+0000", 2023-07-02T14:06:39.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0629999999999999e-06 2023-07-02T14:06:39.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302635+0000", 2023-07-02T14:06:39.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035731199999999998 2023-07-02T14:06:39.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302992+0000", 2023-07-02T14:06:39.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4164e-05 2023-07-02T14:06:39.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.303026+0000", 2023-07-02T14:06:39.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097115900000000008 2023-07-02T14:06:39.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312738+0000", 2023-07-02T14:06:39.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1701999999999998e-05 2023-07-02T14:06:39.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312819+0000", 2023-07-02T14:06:39.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097363640000000008 2023-07-02T14:06:39.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322556+0000", 2023-07-02T14:06:39.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2517000000000001e-05 2023-07-02T14:06:39.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322588+0000", 2023-07-02T14:06:39.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01002393 2023-07-02T14:06:39.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332612+0000", 2023-07-02T14:06:39.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2960999999999998e-05 2023-07-02T14:06:39.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332645+0000", 2023-07-02T14:06:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090669819999999995 2023-07-02T14:06:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341712+0000", 2023-07-02T14:06:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9288000000000002e-05 2023-07-02T14:06:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341811+0000", 2023-07-02T14:06:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081504460000000004 2023-07-02T14:06:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349962+0000", 2023-07-02T14:06:39.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107321 2023-07-02T14:06:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.350069+0000", 2023-07-02T14:06:39.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088790999999999991 2023-07-02T14:06:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358948+0000", 2023-07-02T14:06:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1711000000000002e-05 2023-07-02T14:06:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358980+0000", 2023-07-02T14:06:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080367579999999997 2023-07-02T14:06:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367017+0000", 2023-07-02T14:06:39.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6329000000000002e-05 2023-07-02T14:06:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367073+0000", 2023-07-02T14:06:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075097300000000001 2023-07-02T14:06:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374583+0000", 2023-07-02T14:06:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5710999999999995e-05 2023-07-02T14:06:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374648+0000", 2023-07-02T14:06:39.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009165912 2023-07-02T14:06:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383814+0000", 2023-07-02T14:06:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3916000000000003e-05 2023-07-02T14:06:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383878+0000", 2023-07-02T14:06:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013440532999999999 2023-07-02T14:06:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397319+0000", 2023-07-02T14:06:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7218999999999995e-05 2023-07-02T14:06:39.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397386+0000", 2023-07-02T14:06:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065165279999999997 2023-07-02T14:06:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403903+0000", 2023-07-02T14:06:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8152000000000005e-05 2023-07-02T14:06:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404001+0000", 2023-07-02T14:06:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072185360000000002 2023-07-02T14:06:39.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411219+0000", 2023-07-02T14:06:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:06:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411310+0000", 2023-07-02T14:06:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010021663 2023-07-02T14:06:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421331+0000", 2023-07-02T14:06:39.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0148000000000006e-05 2023-07-02T14:06:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421411+0000", 2023-07-02T14:06:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053056170000000003 2023-07-02T14:06:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426717+0000", 2023-07-02T14:06:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8242000000000001e-05 2023-07-02T14:06:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426795+0000", 2023-07-02T14:06:39.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00041308399999999998 2023-07-02T14:06:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.427208+0000", 2023-07-02T14:06:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028330849999999999 2023-07-02T14:06:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430041+0000", 2023-07-02T14:06:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00072642300000000002 2023-07-02T14:06:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430768+0000", 2023-07-02T14:06:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1847000000000002e-05 2023-07-02T14:06:39.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430800+0000", 2023-07-02T14:06:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9308999999999994e-05 2023-07-02T14:06:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430899+0000", 2023-07-02T14:06:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001 2023-07-02T14:06:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.730 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:410 2.5 2:bbaed619:::rbd_data.125de1449605.0000000000000109:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:06:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 43.216665224000003, 2023-07-02T14:06:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003, 2023-07-02T14:06:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:06:39.731 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:06:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:06:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.635e-06 2023-07-02T14:06:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450568+0000", 2023-07-02T14:06:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5970000000000002e-06 2023-07-02T14:06:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450573+0000", 2023-07-02T14:06:39.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3379999999999999e-06 2023-07-02T14:06:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450575+0000", 2023-07-02T14:06:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9458000000000001e-05 2023-07-02T14:06:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450615+0000", 2023-07-02T14:06:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0569999999999993e-06 2023-07-02T14:06:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450623+0000", 2023-07-02T14:06:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.63968709199999996 2023-07-02T14:06:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090310+0000", 2023-07-02T14:06:39.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020157399999999999 2023-07-02T14:06:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090512+0000", 2023-07-02T14:06:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027739190000000001 2023-07-02T14:06:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093285+0000", 2023-07-02T14:06:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003 2023-07-02T14:06:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:416 2.5 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:06:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 43.216635791000002, 2023-07-02T14:06:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002, 2023-07-02T14:06:39.741 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:06:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:06:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:06:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.742 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.34e-06 2023-07-02T14:06:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450599+0000", 2023-07-02T14:06:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1549999999999998e-06 2023-07-02T14:06:39.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450604+0000", 2023-07-02T14:06:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:06:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450607+0000", 2023-07-02T14:06:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5717999999999999e-05 2023-07-02T14:06:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450643+0000", 2023-07-02T14:06:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1619999999999999e-06 2023-07-02T14:06:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450648+0000", 2023-07-02T14:06:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64292291800000001 2023-07-02T14:06:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093571+0000", 2023-07-02T14:06:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115628 2023-07-02T14:06:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093687+0000", 2023-07-02T14:06:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002885552 2023-07-02T14:06:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.096572+0000", 2023-07-02T14:06:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002 2023-07-02T14:06:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.750 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18557 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 34.418263412000002, 2023-07-02T14:06:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13120991400000001, 2023-07-02T14:06:39.751 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18557 2023-07-02T14:06:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.752 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:06:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8870000000000001e-06 2023-07-02T14:06:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248971+0000", 2023-07-02T14:06:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9009999999999996e-06 2023-07-02T14:06:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248974+0000", 2023-07-02T14:06:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9809999999999997e-06 2023-07-02T14:06:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248978+0000", 2023-07-02T14:06:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00076989199999999999 2023-07-02T14:06:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249748+0000", 2023-07-02T14:06:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8245999999999999e-05 2023-07-02T14:06:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249787+0000", 2023-07-02T14:06:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066332969999999998 2023-07-02T14:06:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256420+0000", 2023-07-02T14:06:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2566e-05 2023-07-02T14:06:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256452+0000", 2023-07-02T14:06:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012364714000000001 2023-07-02T14:06:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268817+0000", 2023-07-02T14:06:39.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5113000000000001e-05 2023-07-02T14:06:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268852+0000", 2023-07-02T14:06:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010993899999999999 2023-07-02T14:06:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279846+0000", 2023-07-02T14:06:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2144999999999998e-05 2023-07-02T14:06:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279878+0000", 2023-07-02T14:06:39.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011778958000000001 2023-07-02T14:06:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291657+0000", 2023-07-02T14:06:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9592000000000001e-05 2023-07-02T14:06:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291687+0000", 2023-07-02T14:06:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081107220000000008 2023-07-02T14:06:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299798+0000", 2023-07-02T14:06:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0290999999999998e-05 2023-07-02T14:06:39.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299828+0000", 2023-07-02T14:06:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071584489999999999 2023-07-02T14:06:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306986+0000", 2023-07-02T14:06:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0755999999999999e-05 2023-07-02T14:06:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307047+0000", 2023-07-02T14:06:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057110260000000001 2023-07-02T14:06:39.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312758+0000", 2023-07-02T14:06:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5145999999999999e-05 2023-07-02T14:06:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312793+0000", 2023-07-02T14:06:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004758361 2023-07-02T14:06:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317552+0000", 2023-07-02T14:06:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9190000000000002e-05 2023-07-02T14:06:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317601+0000", 2023-07-02T14:06:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050329420000000003 2023-07-02T14:06:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322634+0000", 2023-07-02T14:06:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5737999999999998e-05 2023-07-02T14:06:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322719+0000", 2023-07-02T14:06:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064995679999999998 2023-07-02T14:06:39.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329219+0000", 2023-07-02T14:06:39.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0908000000000003e-05 2023-07-02T14:06:39.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329250+0000", 2023-07-02T14:06:39.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062072029999999997 2023-07-02T14:06:39.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335457+0000", 2023-07-02T14:06:39.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9471999999999998e-05 2023-07-02T14:06:39.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335487+0000", 2023-07-02T14:06:39.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097780239999999997 2023-07-02T14:06:39.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345265+0000", 2023-07-02T14:06:39.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0982999999999994e-05 2023-07-02T14:06:39.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345336+0000", 2023-07-02T14:06:39.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046378799999999996 2023-07-02T14:06:39.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349973+0000", 2023-07-02T14:06:39.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1069999999999999e-05 2023-07-02T14:06:39.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350005+0000", 2023-07-02T14:06:39.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042766870000000004 2023-07-02T14:06:39.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354281+0000", 2023-07-02T14:06:39.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1155000000000002e-05 2023-07-02T14:06:39.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354312+0000", 2023-07-02T14:06:39.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071436820000000002 2023-07-02T14:06:39.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361456+0000", 2023-07-02T14:06:39.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9054000000000002e-05 2023-07-02T14:06:39.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361505+0000", 2023-07-02T14:06:39.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072301320000000002 2023-07-02T14:06:39.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368735+0000", 2023-07-02T14:06:39.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1013999999999994e-05 2023-07-02T14:06:39.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368816+0000", 2023-07-02T14:06:39.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036582490000000001 2023-07-02T14:06:39.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372475+0000", 2023-07-02T14:06:39.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4652000000000004e-05 2023-07-02T14:06:39.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372539+0000", 2023-07-02T14:06:39.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00031374199999999999 2023-07-02T14:06:39.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:39.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372853+0000", 2023-07-02T14:06:39.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030071299999999998 2023-07-02T14:06:39.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.375860+0000", 2023-07-02T14:06:39.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041897749999999997 2023-07-02T14:06:39.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380050+0000", 2023-07-02T14:06:39.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8331000000000003e-05 2023-07-02T14:06:39.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380088+0000", 2023-07-02T14:06:39.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0626999999999998e-05 2023-07-02T14:06:39.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380179+0000", 2023-07-02T14:06:39.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13120991400000001 2023-07-02T14:06:39.784 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.784 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.784 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.784 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18558 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:39.784 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:39.785 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 34.418246709999998, 2023-07-02T14:06:39.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13755567899999999, 2023-07-02T14:06:39.785 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.785 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.785 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.785 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.785 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.786 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18558 2023-07-02T14:06:39.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.786 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:39.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:39.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:39.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248985+0000", 2023-07-02T14:06:39.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8630000000000001e-06 2023-07-02T14:06:39.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248989+0000", 2023-07-02T14:06:39.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0750000000000003e-06 2023-07-02T14:06:39.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248995+0000", 2023-07-02T14:06:39.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3290000000000004e-06 2023-07-02T14:06:39.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248999+0000", 2023-07-02T14:06:39.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00080656300000000005 2023-07-02T14:06:39.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249806+0000", 2023-07-02T14:06:39.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2040999999999999e-05 2023-07-02T14:06:39.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249848+0000", 2023-07-02T14:06:39.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066312419999999999 2023-07-02T14:06:39.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256479+0000", 2023-07-02T14:06:39.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3348e-05 2023-07-02T14:06:39.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256513+0000", 2023-07-02T14:06:39.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012391531000000001 2023-07-02T14:06:39.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268904+0000", 2023-07-02T14:06:39.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.347e-05 2023-07-02T14:06:39.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268978+0000", 2023-07-02T14:06:39.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010929772000000001 2023-07-02T14:06:39.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279907+0000", 2023-07-02T14:06:39.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2669999999999997e-05 2023-07-02T14:06:39.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279940+0000", 2023-07-02T14:06:39.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011770332999999999 2023-07-02T14:06:39.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291710+0000", 2023-07-02T14:06:39.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1207000000000001e-05 2023-07-02T14:06:39.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291742+0000", 2023-07-02T14:06:39.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081242199999999997 2023-07-02T14:06:39.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299866+0000", 2023-07-02T14:06:39.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4650000000000002e-05 2023-07-02T14:06:39.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299900+0000", 2023-07-02T14:06:39.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071810270000000004 2023-07-02T14:06:39.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307081+0000", 2023-07-02T14:06:39.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4378000000000002e-05 2023-07-02T14:06:39.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307116+0000", 2023-07-02T14:06:39.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057031909999999998 2023-07-02T14:06:39.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312819+0000", 2023-07-02T14:06:39.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1612999999999997e-05 2023-07-02T14:06:39.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312851+0000", 2023-07-02T14:06:39.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048172470000000002 2023-07-02T14:06:39.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317668+0000", 2023-07-02T14:06:39.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3328e-05 2023-07-02T14:06:39.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317741+0000", 2023-07-02T14:06:39.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050117290000000004 2023-07-02T14:06:39.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322753+0000", 2023-07-02T14:06:39.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9872000000000001e-05 2023-07-02T14:06:39.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322783+0000", 2023-07-02T14:06:39.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064910610000000002 2023-07-02T14:06:39.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329274+0000", 2023-07-02T14:06:39.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0097e-05 2023-07-02T14:06:39.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329304+0000", 2023-07-02T14:06:39.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062163879999999998 2023-07-02T14:06:39.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335520+0000", 2023-07-02T14:06:39.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9513e-05 2023-07-02T14:06:39.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335550+0000", 2023-07-02T14:06:39.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098209969999999997 2023-07-02T14:06:39.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345371+0000", 2023-07-02T14:06:39.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9862999999999999e-05 2023-07-02T14:06:39.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345401+0000", 2023-07-02T14:06:39.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046552030000000001 2023-07-02T14:06:39.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350056+0000", 2023-07-02T14:06:39.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8751999999999993e-05 2023-07-02T14:06:39.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350145+0000", 2023-07-02T14:06:39.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041981450000000003 2023-07-02T14:06:39.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354343+0000", 2023-07-02T14:06:39.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1501000000000002e-05 2023-07-02T14:06:39.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354374+0000", 2023-07-02T14:06:39.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071551740000000003 2023-07-02T14:06:39.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361530+0000", 2023-07-02T14:06:39.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5156e-05 2023-07-02T14:06:39.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361565+0000", 2023-07-02T14:06:39.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072918569999999997 2023-07-02T14:06:39.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368857+0000", 2023-07-02T14:06:39.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5770999999999999e-05 2023-07-02T14:06:39.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368902+0000", 2023-07-02T14:06:39.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057210569999999999 2023-07-02T14:06:39.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374623+0000", 2023-07-02T14:06:39.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3266999999999998e-05 2023-07-02T14:06:39.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374667+0000", 2023-07-02T14:06:39.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055479570000000001 2023-07-02T14:06:39.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380215+0000", 2023-07-02T14:06:39.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0067e-05 2023-07-02T14:06:39.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380265+0000", 2023-07-02T14:06:39.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00043082299999999999 2023-07-02T14:06:39.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:39.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380696+0000", 2023-07-02T14:06:39.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024853240000000001 2023-07-02T14:06:39.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.383181+0000", 2023-07-02T14:06:39.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032601280000000002 2023-07-02T14:06:39.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386441+0000", 2023-07-02T14:06:39.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9748e-05 2023-07-02T14:06:39.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386471+0000", 2023-07-02T14:06:39.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0531000000000006e-05 2023-07-02T14:06:39.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386541+0000", 2023-07-02T14:06:39.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13755567899999999 2023-07-02T14:06:39.818 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.819 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.819 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.819 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18559 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:39.819 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:39.819 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 34.418224105999997, 2023-07-02T14:06:39.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.146184591, 2023-07-02T14:06:39.820 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.820 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.820 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.820 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.820 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.820 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18559 2023-07-02T14:06:39.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.820 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:39.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:39.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:39.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:39.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.726e-06 2023-07-02T14:06:39.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249011+0000", 2023-07-02T14:06:39.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7939999999999997e-06 2023-07-02T14:06:39.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249018+0000", 2023-07-02T14:06:39.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3470000000000001e-06 2023-07-02T14:06:39.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249023+0000", 2023-07-02T14:06:39.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00084402900000000005 2023-07-02T14:06:39.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249867+0000", 2023-07-02T14:06:39.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9050000000000001e-05 2023-07-02T14:06:39.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249906+0000", 2023-07-02T14:06:39.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066311039999999996 2023-07-02T14:06:39.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256537+0000", 2023-07-02T14:06:39.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2790000000000003e-05 2023-07-02T14:06:39.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256570+0000", 2023-07-02T14:06:39.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012436129000000001 2023-07-02T14:06:39.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269006+0000", 2023-07-02T14:06:39.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0173999999999999e-05 2023-07-02T14:06:39.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269046+0000", 2023-07-02T14:06:39.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010924698 2023-07-02T14:06:39.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279971+0000", 2023-07-02T14:06:39.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3831000000000002e-05 2023-07-02T14:06:39.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280004+0000", 2023-07-02T14:06:39.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011795912 2023-07-02T14:06:39.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291800+0000", 2023-07-02T14:06:39.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5613999999999997e-05 2023-07-02T14:06:39.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291886+0000", 2023-07-02T14:06:39.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080395999999999992 2023-07-02T14:06:39.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299925+0000", 2023-07-02T14:06:39.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0694999999999997e-05 2023-07-02T14:06:39.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299956+0000", 2023-07-02T14:06:39.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072049449999999999 2023-07-02T14:06:39.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307161+0000", 2023-07-02T14:06:39.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5523000000000001e-05 2023-07-02T14:06:39.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307247+0000", 2023-07-02T14:06:39.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056331929999999999 2023-07-02T14:06:39.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312880+0000", 2023-07-02T14:06:39.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1646000000000002e-05 2023-07-02T14:06:39.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312911+0000", 2023-07-02T14:06:39.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048627540000000004 2023-07-02T14:06:39.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317774+0000", 2023-07-02T14:06:39.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2008000000000003e-05 2023-07-02T14:06:39.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317806+0000", 2023-07-02T14:06:39.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050055689999999996 2023-07-02T14:06:39.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322812+0000", 2023-07-02T14:06:39.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3442999999999997e-05 2023-07-02T14:06:39.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322855+0000", 2023-07-02T14:06:39.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006472813 2023-07-02T14:06:39.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329328+0000", 2023-07-02T14:06:39.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0128e-05 2023-07-02T14:06:39.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329358+0000", 2023-07-02T14:06:39.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062156490000000002 2023-07-02T14:06:39.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335574+0000", 2023-07-02T14:06:39.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2357999999999997e-05 2023-07-02T14:06:39.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335606+0000", 2023-07-02T14:06:39.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098318409999999992 2023-07-02T14:06:39.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345438+0000", 2023-07-02T14:06:39.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1174000000000007e-05 2023-07-02T14:06:39.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345509+0000", 2023-07-02T14:06:39.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004669851 2023-07-02T14:06:39.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350179+0000", 2023-07-02T14:06:39.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1559000000000001e-05 2023-07-02T14:06:39.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350211+0000", 2023-07-02T14:06:39.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041865740000000002 2023-07-02T14:06:39.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354397+0000", 2023-07-02T14:06:39.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9850000000000001e-05 2023-07-02T14:06:39.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354427+0000", 2023-07-02T14:06:39.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071608449999999999 2023-07-02T14:06:39.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361588+0000", 2023-07-02T14:06:39.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8346000000000002e-05 2023-07-02T14:06:39.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361626+0000", 2023-07-02T14:06:39.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073175009999999997 2023-07-02T14:06:39.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368944+0000", 2023-07-02T14:06:39.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2879000000000005e-05 2023-07-02T14:06:39.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369017+0000", 2023-07-02T14:06:39.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056850690000000001 2023-07-02T14:06:39.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374702+0000", 2023-07-02T14:06:39.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7693e-05 2023-07-02T14:06:39.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374739+0000", 2023-07-02T14:06:39.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076925149999999996 2023-07-02T14:06:39.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382432+0000", 2023-07-02T14:06:39.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3259e-05 2023-07-02T14:06:39.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382475+0000", 2023-07-02T14:06:39.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041005110000000003 2023-07-02T14:06:39.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386576+0000", 2023-07-02T14:06:39.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2074999999999999e-05 2023-07-02T14:06:39.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386618+0000", 2023-07-02T14:06:39.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000233692 2023-07-02T14:06:39.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:39.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386851+0000", 2023-07-02T14:06:39.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024022269999999998 2023-07-02T14:06:39.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.389254+0000", 2023-07-02T14:06:39.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058506469999999996 2023-07-02T14:06:39.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395104+0000", 2023-07-02T14:06:39.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8807000000000003e-05 2023-07-02T14:06:39.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395143+0000", 2023-07-02T14:06:39.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9616000000000001e-05 2023-07-02T14:06:39.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395193+0000", 2023-07-02T14:06:39.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.146184591 2023-07-02T14:06:39.855 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.855 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.855 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.855 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18560 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:39.855 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:39.855 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 34.418205483000001, 2023-07-02T14:06:39.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.15647551900000001, 2023-07-02T14:06:39.856 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.856 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.856 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.856 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.856 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.856 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18560 2023-07-02T14:06:39.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.857 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:39.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:39.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:39.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249026+0000", 2023-07-02T14:06:39.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3419999999999999e-06 2023-07-02T14:06:39.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249030+0000", 2023-07-02T14:06:39.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7869999999999998e-06 2023-07-02T14:06:39.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249035+0000", 2023-07-02T14:06:39.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9639999999999997e-06 2023-07-02T14:06:39.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249039+0000", 2023-07-02T14:06:39.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00088477599999999999 2023-07-02T14:06:39.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249924+0000", 2023-07-02T14:06:39.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7333999999999999e-05 2023-07-02T14:06:39.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249962+0000", 2023-07-02T14:06:39.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066323409999999999 2023-07-02T14:06:39.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256594+0000", 2023-07-02T14:06:39.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2595999999999998e-05 2023-07-02T14:06:39.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256627+0000", 2023-07-02T14:06:39.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012449894 2023-07-02T14:06:39.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269076+0000", 2023-07-02T14:06:39.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3942999999999997e-05 2023-07-02T14:06:39.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269110+0000", 2023-07-02T14:06:39.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010923884 2023-07-02T14:06:39.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280034+0000", 2023-07-02T14:06:39.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7774000000000001e-05 2023-07-02T14:06:39.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280072+0000", 2023-07-02T14:06:39.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011850027000000001 2023-07-02T14:06:39.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291922+0000", 2023-07-02T14:06:39.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2128000000000001e-05 2023-07-02T14:06:39.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291954+0000", 2023-07-02T14:06:39.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080261150000000003 2023-07-02T14:06:39.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299980+0000", 2023-07-02T14:06:39.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6322999999999998e-05 2023-07-02T14:06:39.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300017+0000", 2023-07-02T14:06:39.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072683039999999997 2023-07-02T14:06:39.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307285+0000", 2023-07-02T14:06:39.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3309000000000002e-05 2023-07-02T14:06:39.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307318+0000", 2023-07-02T14:06:39.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056178649999999997 2023-07-02T14:06:39.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312936+0000", 2023-07-02T14:06:39.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1866000000000003e-05 2023-07-02T14:06:39.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312968+0000", 2023-07-02T14:06:39.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048789970000000004 2023-07-02T14:06:39.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317847+0000", 2023-07-02T14:06:39.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0223999999999996e-05 2023-07-02T14:06:39.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317927+0000", 2023-07-02T14:06:39.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049543649999999996 2023-07-02T14:06:39.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322882+0000", 2023-07-02T14:06:39.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1399000000000002e-05 2023-07-02T14:06:39.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322913+0000", 2023-07-02T14:06:39.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006534214 2023-07-02T14:06:39.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329447+0000", 2023-07-02T14:06:39.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9649999999999998e-05 2023-07-02T14:06:39.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329527+0000", 2023-07-02T14:06:39.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061026839999999997 2023-07-02T14:06:39.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335629+0000", 2023-07-02T14:06:39.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9611000000000002e-05 2023-07-02T14:06:39.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335659+0000", 2023-07-02T14:06:39.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098851549999999996 2023-07-02T14:06:39.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345544+0000", 2023-07-02T14:06:39.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0587999999999999e-05 2023-07-02T14:06:39.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345575+0000", 2023-07-02T14:06:39.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046790859999999998 2023-07-02T14:06:39.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350254+0000", 2023-07-02T14:06:39.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0284000000000002e-05 2023-07-02T14:06:39.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350304+0000", 2023-07-02T14:06:39.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041560540000000002 2023-07-02T14:06:39.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354460+0000", 2023-07-02T14:06:39.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2427000000000004e-05 2023-07-02T14:06:39.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354523+0000", 2023-07-02T14:06:39.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071268770000000002 2023-07-02T14:06:39.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361650+0000", 2023-07-02T14:06:39.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0026999999999999e-05 2023-07-02T14:06:39.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361680+0000", 2023-07-02T14:06:39.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073733080000000003 2023-07-02T14:06:39.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369053+0000", 2023-07-02T14:06:39.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7635999999999997e-05 2023-07-02T14:06:39.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369091+0000", 2023-07-02T14:06:39.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056908059999999996 2023-07-02T14:06:39.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374781+0000", 2023-07-02T14:06:39.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4656000000000002e-05 2023-07-02T14:06:39.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374816+0000", 2023-07-02T14:06:39.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077037809999999998 2023-07-02T14:06:39.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382520+0000", 2023-07-02T14:06:39.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7858e-05 2023-07-02T14:06:39.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382588+0000", 2023-07-02T14:06:39.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057884019999999998 2023-07-02T14:06:39.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388376+0000", 2023-07-02T14:06:39.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4831000000000001e-05 2023-07-02T14:06:39.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388461+0000", 2023-07-02T14:06:39.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067708480000000003 2023-07-02T14:06:39.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395232+0000", 2023-07-02T14:06:39.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7948000000000002e-05 2023-07-02T14:06:39.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395290+0000", 2023-07-02T14:06:39.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00027672599999999998 2023-07-02T14:06:39.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:39.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395566+0000", 2023-07-02T14:06:39.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025705569999999998 2023-07-02T14:06:39.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.398137+0000", 2023-07-02T14:06:39.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072257099999999998 2023-07-02T14:06:39.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405363+0000", 2023-07-02T14:06:39.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3887999999999998e-05 2023-07-02T14:06:39.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405397+0000", 2023-07-02T14:06:39.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000105769 2023-07-02T14:06:39.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405502+0000", 2023-07-02T14:06:39.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.15647551900000001 2023-07-02T14:06:39.892 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.892 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.892 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.892 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18561 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:39.892 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:39.892 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 34.418185119, 2023-07-02T14:06:39.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744, 2023-07-02T14:06:39.893 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.893 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.893 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.893 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.893 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.893 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18561 2023-07-02T14:06:39.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.894 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:39.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:39.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:39.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:39.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.563e-06 2023-07-02T14:06:39.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249050+0000", 2023-07-02T14:06:39.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4500000000000001e-06 2023-07-02T14:06:39.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249057+0000", 2023-07-02T14:06:39.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1860000000000002e-06 2023-07-02T14:06:39.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249061+0000", 2023-07-02T14:06:39.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00091153399999999998 2023-07-02T14:06:39.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249972+0000", 2023-07-02T14:06:39.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7156999999999997e-05 2023-07-02T14:06:39.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.250010+0000", 2023-07-02T14:06:39.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006641816 2023-07-02T14:06:39.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256651+0000", 2023-07-02T14:06:39.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1968999999999998e-05 2023-07-02T14:06:39.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256683+0000", 2023-07-02T14:06:39.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012463952 2023-07-02T14:06:39.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269147+0000", 2023-07-02T14:06:39.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2855000000000006e-05 2023-07-02T14:06:39.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269220+0000", 2023-07-02T14:06:39.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010870418999999999 2023-07-02T14:06:39.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280091+0000", 2023-07-02T14:06:39.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1921999999999997e-05 2023-07-02T14:06:39.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280123+0000", 2023-07-02T14:06:39.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011861682 2023-07-02T14:06:39.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291984+0000", 2023-07-02T14:06:39.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5259000000000006e-05 2023-07-02T14:06:39.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.292059+0000", 2023-07-02T14:06:39.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079709359999999996 2023-07-02T14:06:39.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300030+0000", 2023-07-02T14:06:39.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0729999999999998e-05 2023-07-02T14:06:39.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300071+0000", 2023-07-02T14:06:39.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072785690000000004 2023-07-02T14:06:39.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307350+0000", 2023-07-02T14:06:39.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1948999999999999e-05 2023-07-02T14:06:39.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307412+0000", 2023-07-02T14:06:39.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055785779999999998 2023-07-02T14:06:39.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312990+0000", 2023-07-02T14:06:39.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0957000000000002e-05 2023-07-02T14:06:39.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.313021+0000", 2023-07-02T14:06:39.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049246400000000001 2023-07-02T14:06:39.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317946+0000", 2023-07-02T14:06:39.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9346e-05 2023-07-02T14:06:39.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317975+0000", 2023-07-02T14:06:39.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049649469999999999 2023-07-02T14:06:39.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322940+0000", 2023-07-02T14:06:39.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9017000000000002e-05 2023-07-02T14:06:39.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322989+0000", 2023-07-02T14:06:39.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065907029999999998 2023-07-02T14:06:39.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329580+0000", 2023-07-02T14:06:39.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5879999999999999e-05 2023-07-02T14:06:39.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329656+0000", 2023-07-02T14:06:39.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060166960000000002 2023-07-02T14:06:39.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335672+0000", 2023-07-02T14:06:39.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8963e-05 2023-07-02T14:06:39.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335701+0000", 2023-07-02T14:06:39.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098989560000000004 2023-07-02T14:06:39.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345600+0000", 2023-07-02T14:06:39.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5395999999999995e-05 2023-07-02T14:06:39.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345666+0000", 2023-07-02T14:06:39.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046589530000000004 2023-07-02T14:06:39.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350325+0000", 2023-07-02T14:06:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2459000000000002e-05 2023-07-02T14:06:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350357+0000", 2023-07-02T14:06:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042054290000000001 2023-07-02T14:06:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354563+0000", 2023-07-02T14:06:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5612999999999999e-05 2023-07-02T14:06:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354598+0000", 2023-07-02T14:06:39.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070967449999999998 2023-07-02T14:06:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361695+0000", 2023-07-02T14:06:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9314000000000001e-05 2023-07-02T14:06:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361724+0000", 2023-07-02T14:06:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007391028 2023-07-02T14:06:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369115+0000", 2023-07-02T14:06:39.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9163000000000002e-05 2023-07-02T14:06:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369184+0000", 2023-07-02T14:06:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056532500000000003 2023-07-02T14:06:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374838+0000", 2023-07-02T14:06:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5297999999999999e-05 2023-07-02T14:06:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374883+0000", 2023-07-02T14:06:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077292189999999998 2023-07-02T14:06:39.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382612+0000", 2023-07-02T14:06:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1102000000000001e-05 2023-07-02T14:06:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382653+0000", 2023-07-02T14:06:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058515989999999999 2023-07-02T14:06:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388505+0000", 2023-07-02T14:06:39.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9196000000000006e-05 2023-07-02T14:06:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388574+0000", 2023-07-02T14:06:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084245339999999991 2023-07-02T14:06:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.396999+0000", 2023-07-02T14:06:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0596999999999998e-05 2023-07-02T14:06:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.397039+0000", 2023-07-02T14:06:39.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084911339999999991 2023-07-02T14:06:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405530+0000", 2023-07-02T14:06:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7555000000000001e-05 2023-07-02T14:06:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405578+0000", 2023-07-02T14:06:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000229607 2023-07-02T14:06:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405808+0000", 2023-07-02T14:06:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026919940000000001 2023-07-02T14:06:39.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.408500+0000", 2023-07-02T14:06:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053065059999999999 2023-07-02T14:06:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413806+0000", 2023-07-02T14:06:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0490000000000001e-05 2023-07-02T14:06:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413837+0000", 2023-07-02T14:06:39.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000100386 2023-07-02T14:06:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413937+0000", 2023-07-02T14:06:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744 2023-07-02T14:06:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.930 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20768 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch ping cookie 140544751726928 gen 52] snapc 0=[] ondisk+write+known_if_redirected e183)", 2023-07-02T14:06:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 20.444817851, 2023-07-02T14:06:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3751256970000001, 2023-07-02T14:06:39.931 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20768 2023-07-02T14:06:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:39.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7829999999999998e-06 2023-07-02T14:06:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222419+0000", 2023-07-02T14:06:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1380000000000004e-06 2023-07-02T14:06:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222425+0000", 2023-07-02T14:06:39.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2519999999999996e-06 2023-07-02T14:06:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222431+0000", 2023-07-02T14:06:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2157999999999994e-05 2023-07-02T14:06:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222523+0000", 2023-07-02T14:06:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5147000000000001e-05 2023-07-02T14:06:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:39.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222538+0000", 2023-07-02T14:06:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30105419700000002 2023-07-02T14:06:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523592+0000", 2023-07-02T14:06:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2316e-05 2023-07-02T14:06:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523605+0000", 2023-07-02T14:06:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.073658953 2023-07-02T14:06:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597264+0000", 2023-07-02T14:06:39.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000213415 2023-07-02T14:06:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597477+0000", 2023-07-02T14:06:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3034999999999995e-05 2023-07-02T14:06:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597540+0000", 2023-07-02T14:06:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3751256970000001 2023-07-02T14:06:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.941 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20769 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 53] snapc 0=[] ondisk+write+known_if_redirected e184)", 2023-07-02T14:06:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 20.444455395999999, 2023-07-02T14:06:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.380884139, 2023-07-02T14:06:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.942 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20769 2023-07-02T14:06:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:39.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222776+0000", 2023-07-02T14:06:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9710000000000003e-06 2023-07-02T14:06:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222781+0000", 2023-07-02T14:06:39.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9899999999999992e-06 2023-07-02T14:06:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222791+0000", 2023-07-02T14:06:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4240000000000003e-06 2023-07-02T14:06:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222795+0000", 2023-07-02T14:06:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2366000000000001e-05 2023-07-02T14:06:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222828+0000", 2023-07-02T14:06:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4790000000000001e-06 2023-07-02T14:06:39.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222834+0000", 2023-07-02T14:06:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30084589299999998 2023-07-02T14:06:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523680+0000", 2023-07-02T14:06:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.87e-06 2023-07-02T14:06:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523685+0000", 2023-07-02T14:06:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.074111042 2023-07-02T14:06:39.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597796+0000", 2023-07-02T14:06:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0411000000000001e-05 2023-07-02T14:06:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597856+0000", 2023-07-02T14:06:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058045340000000001 2023-07-02T14:06:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.603661+0000", 2023-07-02T14:06:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.380884139 2023-07-02T14:06:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.951 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20770 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 54] snapc 0=[] ondisk+write+known_if_redirected e185)", 2023-07-02T14:06:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 20.143238285999999, 2023-07-02T14:06:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0799819349999999, 2023-07-02T14:06:39.952 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20770 2023-07-02T14:06:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523992+0000", 2023-07-02T14:06:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6720000000000004e-06 2023-07-02T14:06:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524000+0000", 2023-07-02T14:06:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3711e-05 2023-07-02T14:06:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524014+0000", 2023-07-02T14:06:39.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7849999999999992e-06 2023-07-02T14:06:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524022+0000", 2023-07-02T14:06:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2366000000000001e-05 2023-07-02T14:06:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524054+0000", 2023-07-02T14:06:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8169999999999998e-06 2023-07-02T14:06:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524059+0000", 2023-07-02T14:06:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0748338850000001 2023-07-02T14:06:39.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.598893+0000", 2023-07-02T14:06:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116968 2023-07-02T14:06:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.599010+0000", 2023-07-02T14:06:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049663370000000004 2023-07-02T14:06:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.603976+0000", 2023-07-02T14:06:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0799819349999999 2023-07-02T14:06:39.960 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20833 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 55] snapc 0=[] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:06:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 15.135118981, 2023-07-02T14:06:39.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.029349206, 2023-07-02T14:06:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20833 2023-07-02T14:06:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.962 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:06:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:06:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532112+0000", 2023-07-02T14:06:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7590000000000003e-06 2023-07-02T14:06:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532120+0000", 2023-07-02T14:06:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8130000000000001e-05 2023-07-02T14:06:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532139+0000", 2023-07-02T14:06:39.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3299999999999999e-06 2023-07-02T14:06:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532147+0000", 2023-07-02T14:06:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9679999999999998e-05 2023-07-02T14:06:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532207+0000", 2023-07-02T14:06:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0648e-05 2023-07-02T14:06:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op must wait for map", 2023-07-02T14:06:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532217+0000", 2023-07-02T14:06:39.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002082455 2023-07-02T14:06:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.534300+0000", 2023-07-02T14:06:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0710000000000001e-05 2023-07-02T14:06:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.534310+0000", 2023-07-02T14:06:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.013227235 2023-07-02T14:06:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.547538+0000", 2023-07-02T14:06:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021904300000000001 2023-07-02T14:06:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.547757+0000", 2023-07-02T14:06:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013705784 2023-07-02T14:06:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561462+0000", 2023-07-02T14:06:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.029349206 2023-07-02T14:06:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.971 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20865 2.7 2:ff0f5f99:::rbd_data.125de1449605.0000000000000006:head [write 131072~8192] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:06:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.390495909, 2023-07-02T14:06:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.288115762, 2023-07-02T14:06:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.972 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20865 2023-07-02T14:06:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:06:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:06:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276735+0000", 2023-07-02T14:06:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3273e-05 2023-07-02T14:06:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276748+0000", 2023-07-02T14:06:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3895e-05 2023-07-02T14:06:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276762+0000", 2023-07-02T14:06:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.896e-06 2023-07-02T14:06:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276768+0000", 2023-07-02T14:06:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5850999999999999e-05 2023-07-02T14:06:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276814+0000", 2023-07-02T14:06:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5290000000000003e-06 2023-07-02T14:06:39.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276821+0000", 2023-07-02T14:06:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.27225312600000001 2023-07-02T14:06:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549074+0000", 2023-07-02T14:06:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000151223 2023-07-02T14:06:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549226+0000", 2023-07-02T14:06:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024865400000000001 2023-07-02T14:06:39.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549474+0000", 2023-07-02T14:06:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012039185000000001 2023-07-02T14:06:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561513+0000", 2023-07-02T14:06:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003221327 2023-07-02T14:06:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564735+0000", 2023-07-02T14:06:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2026000000000005e-05 2023-07-02T14:06:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564807+0000", 2023-07-02T14:06:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5303000000000003e-05 2023-07-02T14:06:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564852+0000", 2023-07-02T14:06:39.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.288115762 2023-07-02T14:06:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20866 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 131072~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:39.983 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:06:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.384802587999999, 2023-07-02T14:06:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.28773701200000001, 2023-07-02T14:06:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.984 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20866 2023-07-02T14:06:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:06:39.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:06:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282428+0000", 2023-07-02T14:06:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8469999999999998e-05 2023-07-02T14:06:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282517+0000", 2023-07-02T14:06:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2974e-05 2023-07-02T14:06:39.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282530+0000", 2023-07-02T14:06:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4530000000000001e-06 2023-07-02T14:06:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282535+0000", 2023-07-02T14:06:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1926999999999999e-05 2023-07-02T14:06:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282587+0000", 2023-07-02T14:06:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3083e-05 2023-07-02T14:06:39.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282600+0000", 2023-07-02T14:06:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.270692971 2023-07-02T14:06:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:39.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553293+0000", 2023-07-02T14:06:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017065599999999999 2023-07-02T14:06:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553464+0000", 2023-07-02T14:06:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026504399999999998 2023-07-02T14:06:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553729+0000", 2023-07-02T14:06:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011173008 2023-07-02T14:06:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564902+0000", 2023-07-02T14:06:39.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051242600000000003 2023-07-02T14:06:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570026+0000", 2023-07-02T14:06:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8098999999999999e-05 2023-07-02T14:06:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570104+0000", 2023-07-02T14:06:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2323000000000006e-05 2023-07-02T14:06:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570167+0000", 2023-07-02T14:06:39.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.28773701200000001 2023-07-02T14:06:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20875 2.7 2:e3210266:::rbd_data.125de1449605.00000000000001a3:head [write 0~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:06:39.995 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.345751772, 2023-07-02T14:06:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.25297039599999999, 2023-07-02T14:06:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20875 2023-07-02T14:06:39.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:06:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:06:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:39.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321479+0000", 2023-07-02T14:06:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015194399999999999 2023-07-02T14:06:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321631+0000", 2023-07-02T14:06:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4236999999999999e-05 2023-07-02T14:06:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:39.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321645+0000", 2023-07-02T14:06:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2160000000000003e-06 2023-07-02T14:06:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321651+0000", 2023-07-02T14:06:40.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3609999999999998e-05 2023-07-02T14:06:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321695+0000", 2023-07-02T14:06:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1989999999999995e-06 2023-07-02T14:06:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321704+0000", 2023-07-02T14:06:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.23542881500000001 2023-07-02T14:06:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557133+0000", 2023-07-02T14:06:40.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017359600000000001 2023-07-02T14:06:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557307+0000", 2023-07-02T14:06:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00023367599999999999 2023-07-02T14:06:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557540+0000", 2023-07-02T14:06:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016744963000000002 2023-07-02T14:06:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574285+0000", 2023-07-02T14:06:40.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1330999999999999e-05 2023-07-02T14:06:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574346+0000", 2023-07-02T14:06:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9750000000000002e-05 2023-07-02T14:06:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574406+0000", 2023-07-02T14:06:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4647999999999997e-05 2023-07-02T14:06:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574451+0000", 2023-07-02T14:06:40.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.25297039599999999 2023-07-02T14:06:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20915 2.7 2:e6e64067:::rbd_data.125de1449605.00000000000001a9:head [write 0~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:40.007 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:06:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.282497977, 2023-07-02T14:06:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.19966078500000001, 2023-07-02T14:06:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:40.008 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20915 2023-07-02T14:06:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:06:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:06:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384732+0000", 2023-07-02T14:06:40.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9439999999999999e-05 2023-07-02T14:06:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384802+0000", 2023-07-02T14:06:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7572999999999999e-05 2023-07-02T14:06:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384819+0000", 2023-07-02T14:06:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1670000000000004e-06 2023-07-02T14:06:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384825+0000", 2023-07-02T14:06:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6471999999999998e-05 2023-07-02T14:06:40.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384872+0000", 2023-07-02T14:06:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0028e-05 2023-07-02T14:06:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384882+0000", 2023-07-02T14:06:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.17674214099999999 2023-07-02T14:06:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561624+0000", 2023-07-02T14:06:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000135809 2023-07-02T14:06:40.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561760+0000", 2023-07-02T14:06:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020468 2023-07-02T14:06:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:40.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561965+0000", 2023-07-02T14:06:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012498198 2023-07-02T14:06:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574463+0000", 2023-07-02T14:06:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098197509999999998 2023-07-02T14:06:40.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584283+0000", 2023-07-02T14:06:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2056000000000004e-05 2023-07-02T14:06:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:40.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584345+0000", 2023-07-02T14:06:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0513000000000002e-05 2023-07-02T14:06:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584395+0000", 2023-07-02T14:06:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.19966078500000001 2023-07-02T14:06:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:40.018 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20948 2.7 2:ff0f5f99:::rbd_data.125de1449605.0000000000000006:head [write 139264~2220032] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.450499+0000", 2023-07-02T14:06:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.216733250000001, 2023-07-02T14:06:40.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13400999399999999, 2023-07-02T14:06:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20948 2023-07-02T14:06:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.020 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.450499+0000", 2023-07-02T14:06:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:40.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.450499+0000", 2023-07-02T14:06:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.450497+0000", 2023-07-02T14:06:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043010360000000003 2023-07-02T14:06:40.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454798+0000", 2023-07-02T14:06:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8073000000000001e-05 2023-07-02T14:06:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:40.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454816+0000", 2023-07-02T14:06:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3769999999999996e-06 2023-07-02T14:06:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454825+0000", 2023-07-02T14:06:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2797999999999998e-05 2023-07-02T14:06:40.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454878+0000", 2023-07-02T14:06:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0020999999999999e-05 2023-07-02T14:06:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:40.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454888+0000", 2023-07-02T14:06:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.110043313 2023-07-02T14:06:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564931+0000", 2023-07-02T14:06:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7185e-05 2023-07-02T14:06:40.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.565008+0000", 2023-07-02T14:06:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014058199999999999 2023-07-02T14:06:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:40.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.565149+0000", 2023-07-02T14:06:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093221559999999998 2023-07-02T14:06:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574471+0000", 2023-07-02T14:06:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099660410000000001 2023-07-02T14:06:40.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584437+0000", 2023-07-02T14:06:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2805000000000001e-05 2023-07-02T14:06:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:40.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584480+0000", 2023-07-02T14:06:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9393000000000001e-05 2023-07-02T14:06:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584509+0000", 2023-07-02T14:06:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13400999399999999 2023-07-02T14:06:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:40.030 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20949 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 196608~2162688] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.454838+0000", 2023-07-02T14:06:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.212394091, 2023-07-02T14:06:40.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12983145400000001, 2023-07-02T14:06:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20949 2023-07-02T14:06:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.032 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454838+0000", 2023-07-02T14:06:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454838+0000", 2023-07-02T14:06:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454837+0000", 2023-07-02T14:06:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003772821 2023-07-02T14:06:40.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458610+0000", 2023-07-02T14:06:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3013999999999999e-05 2023-07-02T14:06:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:40.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458623+0000", 2023-07-02T14:06:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2789999999999998e-06 2023-07-02T14:06:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458628+0000", 2023-07-02T14:06:40.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7630999999999998e-05 2023-07-02T14:06:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458676+0000", 2023-07-02T14:06:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1579999999999994e-06 2023-07-02T14:06:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458685+0000", 2023-07-02T14:06:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.111536444 2023-07-02T14:06:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570221+0000", 2023-07-02T14:06:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000120844 2023-07-02T14:06:40.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570342+0000", 2023-07-02T14:06:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000149965 2023-07-02T14:06:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570492+0000", 2023-07-02T14:06:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014027843999999999 2023-07-02T14:06:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584520+0000", 2023-07-02T14:06:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8446000000000003e-05 2023-07-02T14:06:40.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584568+0000", 2023-07-02T14:06:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6771999999999998e-05 2023-07-02T14:06:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:40.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584625+0000", 2023-07-02T14:06:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4401999999999999e-05 2023-07-02T14:06:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584670+0000", 2023-07-02T14:06:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12983145400000001 2023-07-02T14:06:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:40.042 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20972 2.7 2:e3210266:::rbd_data.125de1449605.00000000000001a3:head [write 65536~4128768] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.463174+0000", 2023-07-02T14:06:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.204058250999999, 2023-07-02T14:06:40.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12719914800000001, 2023-07-02T14:06:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20972 2023-07-02T14:06:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.044 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.463174+0000", 2023-07-02T14:06:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.463174+0000", 2023-07-02T14:06:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.463173+0000", 2023-07-02T14:06:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069882069999999998 2023-07-02T14:06:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470161+0000", 2023-07-02T14:06:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0103999999999999e-05 2023-07-02T14:06:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470181+0000", 2023-07-02T14:06:40.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:06:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470189+0000", 2023-07-02T14:06:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0970999999999994e-05 2023-07-02T14:06:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470270+0000", 2023-07-02T14:06:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4389000000000001e-05 2023-07-02T14:06:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470284+0000", 2023-07-02T14:06:40.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.10421862799999999 2023-07-02T14:06:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574503+0000", 2023-07-02T14:06:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2457e-05 2023-07-02T14:06:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574585+0000", 2023-07-02T14:06:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013870899999999999 2023-07-02T14:06:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574724+0000", 2023-07-02T14:06:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012886246000000001 2023-07-02T14:06:40.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.587610+0000", 2023-07-02T14:06:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026883599999999999 2023-07-02T14:06:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.590298+0000", 2023-07-02T14:06:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1850000000000001e-05 2023-07-02T14:06:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.590340+0000", 2023-07-02T14:06:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2901999999999998e-05 2023-07-02T14:06:40.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.590373+0000", 2023-07-02T14:06:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12719914800000001 2023-07-02T14:06:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:40.055 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:40.056 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:06:40.056 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:40.055+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:40.091 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:06:40.112 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:06:40.168 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:40.168+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:40.198 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:40.199 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:40.199 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:40.210 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:06:40.250 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:40.257 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:40.256+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:40.270 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:40.288 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:40.307 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:40.325 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:40.343 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:06:40.344 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:06:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:06:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:06:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:06:40.345 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:06:40.357 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:40.357+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:40.377 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:40.377 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:40.388 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:06:40.458 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:40.457+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:40.553 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:40.553 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:40.558 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:40.557+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:40.563 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:06:40.660 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:40.659+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:40.689 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:40.689 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:40.700 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:06:40.761 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:40.760+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:40.824 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:40.824 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:40.834 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:06:40.861 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:40.861+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:40.958 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:40.959 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:40.962 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:40.962+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:40.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:06:41.062 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:41.062+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:41.135 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:41.136 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:41.147 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:06:41.150 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap57 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap58 b3fbb028-e267-4d25-a750-93ba18a2316a@snap58.diff 2023-07-02T14:06:41.190 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:41.190+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:41.232 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 17% complete... Exporting image: 18% complete...2023-07-02T14:06:41.232+0000 7feb03fff700 0 -- 172.21.15.163:0/217018737 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7feaf0007920 msgr2=0x7feaf001a010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:41.236 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:41.235+0000 7feb0ae9c700 0 -- 172.21.15.163:0/217018737 >> [v2:172.21.15.22:6808/99977,v1:172.21.15.22:6809/99977] conn(0x7feaf0008780 msgr2=0x7feaf000abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:41.264 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:41.263+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:41.289 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:41.289 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:41.300 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:06:41.365 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:41.364+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:41.466 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:41.466+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:41.506 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:41.506 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:41.520 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:06:41.566 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:41.566+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:41.650 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:41.650 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:41.661 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:06:41.667 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:41.667+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:41.768 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:41.768+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:41.777 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete...2023-07-02T14:06:41.776+0000 7feb0b69d700 0 -- 172.21.15.163:0/217018737 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x557c7172b040 msgr2=0x557c7172d430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:41.781 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:41.780+0000 7feb0ae9c700 0 -- 172.21.15.163:0/217018737 >> [v2:172.21.15.22:6808/99977,v1:172.21.15.22:6809/99977] conn(0x7feaf0007920 msgr2=0x7feaec03b610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:41.786 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 32% complete...2023-07-02T14:06:41.785+0000 7feb0b69d700 0 -- 172.21.15.163:0/217018737 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7feaec036bb0 msgr2=0x7feaec038fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:41.798 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:41.798 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:41.811 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:06:41.860 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete...2023-07-02T14:06:41.859+0000 7feb0ae9c700 0 -- 172.21.15.163:0/217018737 >> [v2:172.21.15.22:6808/99977,v1:172.21.15.22:6809/99977] conn(0x557c7172b040 msgr2=0x7feaec040340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:41.869 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:41.869+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:41.945 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:41.945 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:41.957 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:06:41.998 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:41.997+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:42.071 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:42.071+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:42.123 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:42.123 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:42.128 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:06:42.133 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:06:42.173 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:42.173+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:42.252 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 100% complete...done. 2023-07-02T14:06:42.259 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:06:42.259 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=59 2023-07-02T14:06:42.259 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap59 2023-07-02T14:06:42.260 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:42.260 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:42.271 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:06:42.273 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:42.273+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:42.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:06:42.374 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:42.373+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:42.398 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:42.398 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:42.408 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:06:42.492 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:42.491+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:42.495 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:42.512 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:42.529 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:42.547 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:42.565 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:42.573 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:42.573 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:42.573 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:06:42.574 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:42.575 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:42.575+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:42.583 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:42.584 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:06:42.585 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:06:42.585 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:06:42.586 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:06:42.586 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:06:42.586 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:06:42.586 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:06:42.676 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:42.675+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:42.748 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:42.748 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:42.748 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:42.748 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:42.748 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:42.758 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:06:42.776 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:42.776+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:42.877 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:42.877+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:42.925 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:42.925 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17354 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 59.619342289999999, 2023-07-02T14:06:42.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493, 2023-07-02T14:06:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17354 2023-07-02T14:06:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.927 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:42.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:42.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:42.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:42.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:42.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:42.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:42.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:42.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:42.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7440000000000001e-06 2023-07-02T14:06:42.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:42.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302606+0000", 2023-07-02T14:06:42.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:06:42.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:42.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302613+0000", 2023-07-02T14:06:42.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8810000000000004e-06 2023-07-02T14:06:42.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:42.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302618+0000", 2023-07-02T14:06:42.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028613100000000002 2023-07-02T14:06:42.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302904+0000", 2023-07-02T14:06:42.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5278999999999995e-05 2023-07-02T14:06:42.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302979+0000", 2023-07-02T14:06:42.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096904750000000005 2023-07-02T14:06:42.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312669+0000", 2023-07-02T14:06:42.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0328000000000002e-05 2023-07-02T14:06:42.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312700+0000", 2023-07-02T14:06:42.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098019319999999993 2023-07-02T14:06:42.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322502+0000", 2023-07-02T14:06:42.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4669999999999998e-05 2023-07-02T14:06:42.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322536+0000", 2023-07-02T14:06:42.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010023839 2023-07-02T14:06:42.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332560+0000", 2023-07-02T14:06:42.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5559999999999998e-05 2023-07-02T14:06:42.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332596+0000", 2023-07-02T14:06:42.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090401319999999993 2023-07-02T14:06:42.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341636+0000", 2023-07-02T14:06:42.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5373000000000001e-05 2023-07-02T14:06:42.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341681+0000", 2023-07-02T14:06:42.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081994059999999994 2023-07-02T14:06:42.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349881+0000", 2023-07-02T14:06:42.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4319000000000001e-05 2023-07-02T14:06:42.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349915+0000", 2023-07-02T14:06:42.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089388339999999997 2023-07-02T14:06:42.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358854+0000", 2023-07-02T14:06:42.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4319000000000005e-05 2023-07-02T14:06:42.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358928+0000", 2023-07-02T14:06:42.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080234690000000001 2023-07-02T14:06:42.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366952+0000", 2023-07-02T14:06:42.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2082999999999998e-05 2023-07-02T14:06:42.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366984+0000", 2023-07-02T14:06:42.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075015790000000004 2023-07-02T14:06:42.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374485+0000", 2023-07-02T14:06:42.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4480999999999995e-05 2023-07-02T14:06:42.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374550+0000", 2023-07-02T14:06:42.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091666620000000008 2023-07-02T14:06:42.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383716+0000", 2023-07-02T14:06:42.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9318000000000003e-05 2023-07-02T14:06:42.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383786+0000", 2023-07-02T14:06:42.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013434115 2023-07-02T14:06:42.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397220+0000", 2023-07-02T14:06:42.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4573999999999997e-05 2023-07-02T14:06:42.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397264+0000", 2023-07-02T14:06:42.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064824649999999998 2023-07-02T14:06:42.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403747+0000", 2023-07-02T14:06:42.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1087999999999993e-05 2023-07-02T14:06:42.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403828+0000", 2023-07-02T14:06:42.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072854449999999998 2023-07-02T14:06:42.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411113+0000", 2023-07-02T14:06:42.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5120000000000001e-05 2023-07-02T14:06:42.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411169+0000", 2023-07-02T14:06:42.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007211505 2023-07-02T14:06:42.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418380+0000", 2023-07-02T14:06:42.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011289399999999999 2023-07-02T14:06:42.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:42.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418493+0000", 2023-07-02T14:06:42.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00051217500000000002 2023-07-02T14:06:42.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:42.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.419005+0000", 2023-07-02T14:06:42.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00327393 2023-07-02T14:06:42.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:42.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422279+0000", 2023-07-02T14:06:42.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041432350000000003 2023-07-02T14:06:42.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:42.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426422+0000", 2023-07-02T14:06:42.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7021000000000003e-05 2023-07-02T14:06:42.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:42.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426469+0000", 2023-07-02T14:06:42.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000190256 2023-07-02T14:06:42.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:42.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426660+0000", 2023-07-02T14:06:42.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493 2023-07-02T14:06:42.953 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:42.953 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:42.953 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:42.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.953 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17355 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:42.953 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:42.953 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 59.619323629, 2023-07-02T14:06:42.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001, 2023-07-02T14:06:42.954 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:42.954 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:42.954 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:42.954 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:42.954 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:42.954 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17355 2023-07-02T14:06:42.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.955 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:42.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:42.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:42.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:42.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:42.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:42.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:42.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:42.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302621+0000", 2023-07-02T14:06:42.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2260000000000001e-06 2023-07-02T14:06:42.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:42.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302625+0000", 2023-07-02T14:06:42.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8599999999999998e-06 2023-07-02T14:06:42.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:42.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302630+0000", 2023-07-02T14:06:42.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0629999999999999e-06 2023-07-02T14:06:42.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:42.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302635+0000", 2023-07-02T14:06:42.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035731199999999998 2023-07-02T14:06:42.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302992+0000", 2023-07-02T14:06:42.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4164e-05 2023-07-02T14:06:42.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.303026+0000", 2023-07-02T14:06:42.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097115900000000008 2023-07-02T14:06:42.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312738+0000", 2023-07-02T14:06:42.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1701999999999998e-05 2023-07-02T14:06:42.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312819+0000", 2023-07-02T14:06:42.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097363640000000008 2023-07-02T14:06:42.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322556+0000", 2023-07-02T14:06:42.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2517000000000001e-05 2023-07-02T14:06:42.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322588+0000", 2023-07-02T14:06:42.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01002393 2023-07-02T14:06:42.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332612+0000", 2023-07-02T14:06:42.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2960999999999998e-05 2023-07-02T14:06:42.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332645+0000", 2023-07-02T14:06:42.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090669819999999995 2023-07-02T14:06:42.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341712+0000", 2023-07-02T14:06:42.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9288000000000002e-05 2023-07-02T14:06:42.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341811+0000", 2023-07-02T14:06:42.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081504460000000004 2023-07-02T14:06:42.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349962+0000", 2023-07-02T14:06:42.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107321 2023-07-02T14:06:42.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.350069+0000", 2023-07-02T14:06:42.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088790999999999991 2023-07-02T14:06:42.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358948+0000", 2023-07-02T14:06:42.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1711000000000002e-05 2023-07-02T14:06:42.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358980+0000", 2023-07-02T14:06:42.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080367579999999997 2023-07-02T14:06:42.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367017+0000", 2023-07-02T14:06:42.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6329000000000002e-05 2023-07-02T14:06:42.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367073+0000", 2023-07-02T14:06:42.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075097300000000001 2023-07-02T14:06:42.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374583+0000", 2023-07-02T14:06:42.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5710999999999995e-05 2023-07-02T14:06:42.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374648+0000", 2023-07-02T14:06:42.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009165912 2023-07-02T14:06:42.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383814+0000", 2023-07-02T14:06:42.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3916000000000003e-05 2023-07-02T14:06:42.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383878+0000", 2023-07-02T14:06:42.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013440532999999999 2023-07-02T14:06:42.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397319+0000", 2023-07-02T14:06:42.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7218999999999995e-05 2023-07-02T14:06:42.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397386+0000", 2023-07-02T14:06:42.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065165279999999997 2023-07-02T14:06:42.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403903+0000", 2023-07-02T14:06:42.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8152000000000005e-05 2023-07-02T14:06:42.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404001+0000", 2023-07-02T14:06:42.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072185360000000002 2023-07-02T14:06:42.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411219+0000", 2023-07-02T14:06:42.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:06:42.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411310+0000", 2023-07-02T14:06:42.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010021663 2023-07-02T14:06:42.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421331+0000", 2023-07-02T14:06:42.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0148000000000006e-05 2023-07-02T14:06:42.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:42.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421411+0000", 2023-07-02T14:06:42.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053056170000000003 2023-07-02T14:06:42.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426717+0000", 2023-07-02T14:06:42.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8242000000000001e-05 2023-07-02T14:06:42.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:42.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426795+0000", 2023-07-02T14:06:42.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00041308399999999998 2023-07-02T14:06:42.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:42.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.427208+0000", 2023-07-02T14:06:42.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028330849999999999 2023-07-02T14:06:42.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:42.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430041+0000", 2023-07-02T14:06:42.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00072642300000000002 2023-07-02T14:06:42.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:42.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430768+0000", 2023-07-02T14:06:42.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1847000000000002e-05 2023-07-02T14:06:42.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:42.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430800+0000", 2023-07-02T14:06:42.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9308999999999994e-05 2023-07-02T14:06:42.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:42.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430899+0000", 2023-07-02T14:06:42.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001 2023-07-02T14:06:42.981 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:42.981 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:42.981 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:42.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.981 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:410 2.5 2:bbaed619:::rbd_data.125de1449605.0000000000000109:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:06:42.981 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:42.982 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.471378364000003, 2023-07-02T14:06:42.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003, 2023-07-02T14:06:42.982 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:42.982 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:42.982 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:42.982 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:06:42.982 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:06:42.982 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:06:42.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.983 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:42.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:42.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:42.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:42.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:42.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:42.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:42.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:42.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:42.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.635e-06 2023-07-02T14:06:42.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:42.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450568+0000", 2023-07-02T14:06:42.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5970000000000002e-06 2023-07-02T14:06:42.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:42.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450573+0000", 2023-07-02T14:06:42.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3379999999999999e-06 2023-07-02T14:06:42.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:42.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450575+0000", 2023-07-02T14:06:42.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9458000000000001e-05 2023-07-02T14:06:42.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450615+0000", 2023-07-02T14:06:42.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0569999999999993e-06 2023-07-02T14:06:42.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:42.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450623+0000", 2023-07-02T14:06:42.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.63968709199999996 2023-07-02T14:06:42.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090310+0000", 2023-07-02T14:06:42.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020157399999999999 2023-07-02T14:06:42.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:42.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090512+0000", 2023-07-02T14:06:42.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027739190000000001 2023-07-02T14:06:42.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:42.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093285+0000", 2023-07-02T14:06:42.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003 2023-07-02T14:06:42.990 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:42.990 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:42.990 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:42.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.991 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:416 2.5 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:06:42.991 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:42.991 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.471348931000001, 2023-07-02T14:06:42.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002, 2023-07-02T14:06:42.991 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:42.992 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:42.992 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:42.992 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:06:42.992 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:06:42.992 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:06:42.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.992 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:42.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:42.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:42.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:42.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:42.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:42.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:42.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:42.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:42.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.34e-06 2023-07-02T14:06:42.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:42.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450599+0000", 2023-07-02T14:06:42.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1549999999999998e-06 2023-07-02T14:06:42.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:42.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450604+0000", 2023-07-02T14:06:42.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:06:42.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:42.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450607+0000", 2023-07-02T14:06:42.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5717999999999999e-05 2023-07-02T14:06:42.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450643+0000", 2023-07-02T14:06:42.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1619999999999999e-06 2023-07-02T14:06:42.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:42.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450648+0000", 2023-07-02T14:06:42.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64292291800000001 2023-07-02T14:06:42.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:42.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093571+0000", 2023-07-02T14:06:42.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115628 2023-07-02T14:06:42.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:42.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093687+0000", 2023-07-02T14:06:42.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002885552 2023-07-02T14:06:42.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:42.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:42.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:42.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.096572+0000", 2023-07-02T14:06:43.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002 2023-07-02T14:06:43.000 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.000 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:43.000 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.000 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18557 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:43.001 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:43.001 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.672976552000002, 2023-07-02T14:06:43.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13120991400000001, 2023-07-02T14:06:43.001 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:43.001 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.001 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:43.001 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:43.001 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:43.002 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18557 2023-07-02T14:06:43.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.002 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:43.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:43.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:43.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:43.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:43.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:43.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:06:43.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:43.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:43.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8870000000000001e-06 2023-07-02T14:06:43.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:43.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248971+0000", 2023-07-02T14:06:43.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9009999999999996e-06 2023-07-02T14:06:43.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:43.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248974+0000", 2023-07-02T14:06:43.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9809999999999997e-06 2023-07-02T14:06:43.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248978+0000", 2023-07-02T14:06:43.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00076989199999999999 2023-07-02T14:06:43.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249748+0000", 2023-07-02T14:06:43.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8245999999999999e-05 2023-07-02T14:06:43.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249787+0000", 2023-07-02T14:06:43.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066332969999999998 2023-07-02T14:06:43.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256420+0000", 2023-07-02T14:06:43.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2566e-05 2023-07-02T14:06:43.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256452+0000", 2023-07-02T14:06:43.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012364714000000001 2023-07-02T14:06:43.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268817+0000", 2023-07-02T14:06:43.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5113000000000001e-05 2023-07-02T14:06:43.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268852+0000", 2023-07-02T14:06:43.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010993899999999999 2023-07-02T14:06:43.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279846+0000", 2023-07-02T14:06:43.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2144999999999998e-05 2023-07-02T14:06:43.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279878+0000", 2023-07-02T14:06:43.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011778958000000001 2023-07-02T14:06:43.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291657+0000", 2023-07-02T14:06:43.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9592000000000001e-05 2023-07-02T14:06:43.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291687+0000", 2023-07-02T14:06:43.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081107220000000008 2023-07-02T14:06:43.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299798+0000", 2023-07-02T14:06:43.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0290999999999998e-05 2023-07-02T14:06:43.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299828+0000", 2023-07-02T14:06:43.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071584489999999999 2023-07-02T14:06:43.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306986+0000", 2023-07-02T14:06:43.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0755999999999999e-05 2023-07-02T14:06:43.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307047+0000", 2023-07-02T14:06:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057110260000000001 2023-07-02T14:06:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312758+0000", 2023-07-02T14:06:43.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5145999999999999e-05 2023-07-02T14:06:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312793+0000", 2023-07-02T14:06:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004758361 2023-07-02T14:06:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317552+0000", 2023-07-02T14:06:43.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9190000000000002e-05 2023-07-02T14:06:43.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317601+0000", 2023-07-02T14:06:43.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050329420000000003 2023-07-02T14:06:43.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322634+0000", 2023-07-02T14:06:43.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5737999999999998e-05 2023-07-02T14:06:43.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322719+0000", 2023-07-02T14:06:43.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064995679999999998 2023-07-02T14:06:43.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329219+0000", 2023-07-02T14:06:43.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0908000000000003e-05 2023-07-02T14:06:43.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329250+0000", 2023-07-02T14:06:43.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062072029999999997 2023-07-02T14:06:43.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335457+0000", 2023-07-02T14:06:43.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9471999999999998e-05 2023-07-02T14:06:43.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335487+0000", 2023-07-02T14:06:43.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097780239999999997 2023-07-02T14:06:43.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345265+0000", 2023-07-02T14:06:43.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0982999999999994e-05 2023-07-02T14:06:43.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345336+0000", 2023-07-02T14:06:43.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046378799999999996 2023-07-02T14:06:43.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349973+0000", 2023-07-02T14:06:43.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1069999999999999e-05 2023-07-02T14:06:43.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350005+0000", 2023-07-02T14:06:43.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042766870000000004 2023-07-02T14:06:43.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354281+0000", 2023-07-02T14:06:43.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1155000000000002e-05 2023-07-02T14:06:43.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354312+0000", 2023-07-02T14:06:43.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071436820000000002 2023-07-02T14:06:43.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361456+0000", 2023-07-02T14:06:43.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9054000000000002e-05 2023-07-02T14:06:43.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361505+0000", 2023-07-02T14:06:43.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072301320000000002 2023-07-02T14:06:43.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368735+0000", 2023-07-02T14:06:43.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1013999999999994e-05 2023-07-02T14:06:43.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368816+0000", 2023-07-02T14:06:43.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036582490000000001 2023-07-02T14:06:43.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372475+0000", 2023-07-02T14:06:43.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4652000000000004e-05 2023-07-02T14:06:43.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:43.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372539+0000", 2023-07-02T14:06:43.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00031374199999999999 2023-07-02T14:06:43.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:43.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372853+0000", 2023-07-02T14:06:43.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030071299999999998 2023-07-02T14:06:43.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:43.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.375860+0000", 2023-07-02T14:06:43.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041897749999999997 2023-07-02T14:06:43.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:43.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380050+0000", 2023-07-02T14:06:43.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8331000000000003e-05 2023-07-02T14:06:43.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:43.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380088+0000", 2023-07-02T14:06:43.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0626999999999998e-05 2023-07-02T14:06:43.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:43.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380179+0000", 2023-07-02T14:06:43.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13120991400000001 2023-07-02T14:06:43.033 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.033 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:43.033 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.033 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18558 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:43.033 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:43.033 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.672959849999998, 2023-07-02T14:06:43.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13755567899999999, 2023-07-02T14:06:43.034 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:43.034 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.034 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:43.034 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:43.034 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:43.034 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18558 2023-07-02T14:06:43.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.035 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:43.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:43.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:43.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:43.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:43.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:43.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:43.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:43.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248985+0000", 2023-07-02T14:06:43.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8630000000000001e-06 2023-07-02T14:06:43.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:43.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248989+0000", 2023-07-02T14:06:43.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0750000000000003e-06 2023-07-02T14:06:43.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:43.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248995+0000", 2023-07-02T14:06:43.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3290000000000004e-06 2023-07-02T14:06:43.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248999+0000", 2023-07-02T14:06:43.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00080656300000000005 2023-07-02T14:06:43.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249806+0000", 2023-07-02T14:06:43.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2040999999999999e-05 2023-07-02T14:06:43.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249848+0000", 2023-07-02T14:06:43.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066312419999999999 2023-07-02T14:06:43.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256479+0000", 2023-07-02T14:06:43.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3348e-05 2023-07-02T14:06:43.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256513+0000", 2023-07-02T14:06:43.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012391531000000001 2023-07-02T14:06:43.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268904+0000", 2023-07-02T14:06:43.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.347e-05 2023-07-02T14:06:43.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268978+0000", 2023-07-02T14:06:43.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010929772000000001 2023-07-02T14:06:43.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279907+0000", 2023-07-02T14:06:43.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2669999999999997e-05 2023-07-02T14:06:43.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279940+0000", 2023-07-02T14:06:43.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011770332999999999 2023-07-02T14:06:43.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291710+0000", 2023-07-02T14:06:43.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1207000000000001e-05 2023-07-02T14:06:43.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291742+0000", 2023-07-02T14:06:43.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081242199999999997 2023-07-02T14:06:43.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299866+0000", 2023-07-02T14:06:43.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4650000000000002e-05 2023-07-02T14:06:43.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299900+0000", 2023-07-02T14:06:43.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071810270000000004 2023-07-02T14:06:43.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307081+0000", 2023-07-02T14:06:43.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4378000000000002e-05 2023-07-02T14:06:43.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307116+0000", 2023-07-02T14:06:43.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057031909999999998 2023-07-02T14:06:43.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312819+0000", 2023-07-02T14:06:43.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1612999999999997e-05 2023-07-02T14:06:43.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312851+0000", 2023-07-02T14:06:43.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048172470000000002 2023-07-02T14:06:43.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317668+0000", 2023-07-02T14:06:43.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3328e-05 2023-07-02T14:06:43.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317741+0000", 2023-07-02T14:06:43.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050117290000000004 2023-07-02T14:06:43.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322753+0000", 2023-07-02T14:06:43.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9872000000000001e-05 2023-07-02T14:06:43.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322783+0000", 2023-07-02T14:06:43.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064910610000000002 2023-07-02T14:06:43.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329274+0000", 2023-07-02T14:06:43.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0097e-05 2023-07-02T14:06:43.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329304+0000", 2023-07-02T14:06:43.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062163879999999998 2023-07-02T14:06:43.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335520+0000", 2023-07-02T14:06:43.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9513e-05 2023-07-02T14:06:43.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335550+0000", 2023-07-02T14:06:43.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098209969999999997 2023-07-02T14:06:43.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345371+0000", 2023-07-02T14:06:43.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9862999999999999e-05 2023-07-02T14:06:43.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345401+0000", 2023-07-02T14:06:43.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046552030000000001 2023-07-02T14:06:43.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350056+0000", 2023-07-02T14:06:43.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8751999999999993e-05 2023-07-02T14:06:43.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350145+0000", 2023-07-02T14:06:43.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041981450000000003 2023-07-02T14:06:43.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354343+0000", 2023-07-02T14:06:43.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1501000000000002e-05 2023-07-02T14:06:43.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354374+0000", 2023-07-02T14:06:43.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071551740000000003 2023-07-02T14:06:43.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361530+0000", 2023-07-02T14:06:43.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5156e-05 2023-07-02T14:06:43.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361565+0000", 2023-07-02T14:06:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072918569999999997 2023-07-02T14:06:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368857+0000", 2023-07-02T14:06:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5770999999999999e-05 2023-07-02T14:06:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368902+0000", 2023-07-02T14:06:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057210569999999999 2023-07-02T14:06:43.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374623+0000", 2023-07-02T14:06:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3266999999999998e-05 2023-07-02T14:06:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374667+0000", 2023-07-02T14:06:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055479570000000001 2023-07-02T14:06:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380215+0000", 2023-07-02T14:06:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0067e-05 2023-07-02T14:06:43.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380265+0000", 2023-07-02T14:06:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00043082299999999999 2023-07-02T14:06:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:43.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380696+0000", 2023-07-02T14:06:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024853240000000001 2023-07-02T14:06:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.383181+0000", 2023-07-02T14:06:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032601280000000002 2023-07-02T14:06:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386441+0000", 2023-07-02T14:06:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9748e-05 2023-07-02T14:06:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:43.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386471+0000", 2023-07-02T14:06:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0531000000000006e-05 2023-07-02T14:06:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386541+0000", 2023-07-02T14:06:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13755567899999999 2023-07-02T14:06:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.129 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18559 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.672937245999996, 2023-07-02T14:06:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.146184591, 2023-07-02T14:06:43.130 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18559 2023-07-02T14:06:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.131 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:43.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.726e-06 2023-07-02T14:06:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249011+0000", 2023-07-02T14:06:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7939999999999997e-06 2023-07-02T14:06:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:43.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249018+0000", 2023-07-02T14:06:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3470000000000001e-06 2023-07-02T14:06:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249023+0000", 2023-07-02T14:06:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00084402900000000005 2023-07-02T14:06:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249867+0000", 2023-07-02T14:06:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9050000000000001e-05 2023-07-02T14:06:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249906+0000", 2023-07-02T14:06:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066311039999999996 2023-07-02T14:06:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256537+0000", 2023-07-02T14:06:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2790000000000003e-05 2023-07-02T14:06:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256570+0000", 2023-07-02T14:06:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012436129000000001 2023-07-02T14:06:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269006+0000", 2023-07-02T14:06:43.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0173999999999999e-05 2023-07-02T14:06:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269046+0000", 2023-07-02T14:06:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010924698 2023-07-02T14:06:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279971+0000", 2023-07-02T14:06:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3831000000000002e-05 2023-07-02T14:06:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280004+0000", 2023-07-02T14:06:43.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011795912 2023-07-02T14:06:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291800+0000", 2023-07-02T14:06:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5613999999999997e-05 2023-07-02T14:06:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291886+0000", 2023-07-02T14:06:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080395999999999992 2023-07-02T14:06:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299925+0000", 2023-07-02T14:06:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0694999999999997e-05 2023-07-02T14:06:43.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299956+0000", 2023-07-02T14:06:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072049449999999999 2023-07-02T14:06:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307161+0000", 2023-07-02T14:06:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5523000000000001e-05 2023-07-02T14:06:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307247+0000", 2023-07-02T14:06:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056331929999999999 2023-07-02T14:06:43.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312880+0000", 2023-07-02T14:06:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1646000000000002e-05 2023-07-02T14:06:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312911+0000", 2023-07-02T14:06:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048627540000000004 2023-07-02T14:06:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317774+0000", 2023-07-02T14:06:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2008000000000003e-05 2023-07-02T14:06:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317806+0000", 2023-07-02T14:06:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050055689999999996 2023-07-02T14:06:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322812+0000", 2023-07-02T14:06:43.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3442999999999997e-05 2023-07-02T14:06:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322855+0000", 2023-07-02T14:06:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006472813 2023-07-02T14:06:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329328+0000", 2023-07-02T14:06:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0128e-05 2023-07-02T14:06:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329358+0000", 2023-07-02T14:06:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062156490000000002 2023-07-02T14:06:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335574+0000", 2023-07-02T14:06:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2357999999999997e-05 2023-07-02T14:06:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335606+0000", 2023-07-02T14:06:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098318409999999992 2023-07-02T14:06:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345438+0000", 2023-07-02T14:06:43.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1174000000000007e-05 2023-07-02T14:06:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345509+0000", 2023-07-02T14:06:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004669851 2023-07-02T14:06:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350179+0000", 2023-07-02T14:06:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1559000000000001e-05 2023-07-02T14:06:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350211+0000", 2023-07-02T14:06:43.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041865740000000002 2023-07-02T14:06:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354397+0000", 2023-07-02T14:06:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9850000000000001e-05 2023-07-02T14:06:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354427+0000", 2023-07-02T14:06:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071608449999999999 2023-07-02T14:06:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361588+0000", 2023-07-02T14:06:43.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8346000000000002e-05 2023-07-02T14:06:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361626+0000", 2023-07-02T14:06:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073175009999999997 2023-07-02T14:06:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368944+0000", 2023-07-02T14:06:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2879000000000005e-05 2023-07-02T14:06:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369017+0000", 2023-07-02T14:06:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056850690000000001 2023-07-02T14:06:43.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374702+0000", 2023-07-02T14:06:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7693e-05 2023-07-02T14:06:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374739+0000", 2023-07-02T14:06:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076925149999999996 2023-07-02T14:06:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382432+0000", 2023-07-02T14:06:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3259e-05 2023-07-02T14:06:43.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382475+0000", 2023-07-02T14:06:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041005110000000003 2023-07-02T14:06:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386576+0000", 2023-07-02T14:06:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2074999999999999e-05 2023-07-02T14:06:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386618+0000", 2023-07-02T14:06:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000233692 2023-07-02T14:06:43.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386851+0000", 2023-07-02T14:06:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024022269999999998 2023-07-02T14:06:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.389254+0000", 2023-07-02T14:06:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058506469999999996 2023-07-02T14:06:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395104+0000", 2023-07-02T14:06:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8807000000000003e-05 2023-07-02T14:06:43.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395143+0000", 2023-07-02T14:06:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9616000000000001e-05 2023-07-02T14:06:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:43.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395193+0000", 2023-07-02T14:06:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.146184591 2023-07-02T14:06:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18560 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:43.165 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:43.166 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.672918623000001, 2023-07-02T14:06:43.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.15647551900000001, 2023-07-02T14:06:43.166 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:43.166 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.166 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:43.166 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:43.166 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18560 2023-07-02T14:06:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:43.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:43.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249026+0000", 2023-07-02T14:06:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3419999999999999e-06 2023-07-02T14:06:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249030+0000", 2023-07-02T14:06:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7869999999999998e-06 2023-07-02T14:06:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249035+0000", 2023-07-02T14:06:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9639999999999997e-06 2023-07-02T14:06:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249039+0000", 2023-07-02T14:06:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00088477599999999999 2023-07-02T14:06:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249924+0000", 2023-07-02T14:06:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7333999999999999e-05 2023-07-02T14:06:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249962+0000", 2023-07-02T14:06:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066323409999999999 2023-07-02T14:06:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256594+0000", 2023-07-02T14:06:43.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2595999999999998e-05 2023-07-02T14:06:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256627+0000", 2023-07-02T14:06:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012449894 2023-07-02T14:06:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269076+0000", 2023-07-02T14:06:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3942999999999997e-05 2023-07-02T14:06:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269110+0000", 2023-07-02T14:06:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010923884 2023-07-02T14:06:43.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280034+0000", 2023-07-02T14:06:43.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7774000000000001e-05 2023-07-02T14:06:43.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280072+0000", 2023-07-02T14:06:43.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011850027000000001 2023-07-02T14:06:43.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291922+0000", 2023-07-02T14:06:43.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2128000000000001e-05 2023-07-02T14:06:43.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291954+0000", 2023-07-02T14:06:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080261150000000003 2023-07-02T14:06:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299980+0000", 2023-07-02T14:06:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6322999999999998e-05 2023-07-02T14:06:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300017+0000", 2023-07-02T14:06:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072683039999999997 2023-07-02T14:06:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307285+0000", 2023-07-02T14:06:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3309000000000002e-05 2023-07-02T14:06:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307318+0000", 2023-07-02T14:06:43.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056178649999999997 2023-07-02T14:06:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312936+0000", 2023-07-02T14:06:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1866000000000003e-05 2023-07-02T14:06:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312968+0000", 2023-07-02T14:06:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048789970000000004 2023-07-02T14:06:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317847+0000", 2023-07-02T14:06:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0223999999999996e-05 2023-07-02T14:06:43.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317927+0000", 2023-07-02T14:06:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049543649999999996 2023-07-02T14:06:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322882+0000", 2023-07-02T14:06:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1399000000000002e-05 2023-07-02T14:06:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322913+0000", 2023-07-02T14:06:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006534214 2023-07-02T14:06:43.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329447+0000", 2023-07-02T14:06:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9649999999999998e-05 2023-07-02T14:06:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329527+0000", 2023-07-02T14:06:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061026839999999997 2023-07-02T14:06:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335629+0000", 2023-07-02T14:06:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9611000000000002e-05 2023-07-02T14:06:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335659+0000", 2023-07-02T14:06:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098851549999999996 2023-07-02T14:06:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345544+0000", 2023-07-02T14:06:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0587999999999999e-05 2023-07-02T14:06:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345575+0000", 2023-07-02T14:06:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046790859999999998 2023-07-02T14:06:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350254+0000", 2023-07-02T14:06:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0284000000000002e-05 2023-07-02T14:06:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350304+0000", 2023-07-02T14:06:43.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041560540000000002 2023-07-02T14:06:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354460+0000", 2023-07-02T14:06:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2427000000000004e-05 2023-07-02T14:06:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354523+0000", 2023-07-02T14:06:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071268770000000002 2023-07-02T14:06:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361650+0000", 2023-07-02T14:06:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0026999999999999e-05 2023-07-02T14:06:43.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361680+0000", 2023-07-02T14:06:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073733080000000003 2023-07-02T14:06:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369053+0000", 2023-07-02T14:06:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7635999999999997e-05 2023-07-02T14:06:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369091+0000", 2023-07-02T14:06:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056908059999999996 2023-07-02T14:06:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374781+0000", 2023-07-02T14:06:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4656000000000002e-05 2023-07-02T14:06:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374816+0000", 2023-07-02T14:06:43.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077037809999999998 2023-07-02T14:06:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382520+0000", 2023-07-02T14:06:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7858e-05 2023-07-02T14:06:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382588+0000", 2023-07-02T14:06:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057884019999999998 2023-07-02T14:06:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388376+0000", 2023-07-02T14:06:43.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4831000000000001e-05 2023-07-02T14:06:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388461+0000", 2023-07-02T14:06:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067708480000000003 2023-07-02T14:06:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395232+0000", 2023-07-02T14:06:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7948000000000002e-05 2023-07-02T14:06:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395290+0000", 2023-07-02T14:06:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00027672599999999998 2023-07-02T14:06:43.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395566+0000", 2023-07-02T14:06:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025705569999999998 2023-07-02T14:06:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.398137+0000", 2023-07-02T14:06:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072257099999999998 2023-07-02T14:06:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405363+0000", 2023-07-02T14:06:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3887999999999998e-05 2023-07-02T14:06:43.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405397+0000", 2023-07-02T14:06:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000105769 2023-07-02T14:06:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:43.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405502+0000", 2023-07-02T14:06:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.15647551900000001 2023-07-02T14:06:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18561 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:43.201 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.672898259, 2023-07-02T14:06:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744, 2023-07-02T14:06:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:43.202 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18561 2023-07-02T14:06:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:43.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.563e-06 2023-07-02T14:06:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249050+0000", 2023-07-02T14:06:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4500000000000001e-06 2023-07-02T14:06:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249057+0000", 2023-07-02T14:06:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1860000000000002e-06 2023-07-02T14:06:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249061+0000", 2023-07-02T14:06:43.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00091153399999999998 2023-07-02T14:06:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249972+0000", 2023-07-02T14:06:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7156999999999997e-05 2023-07-02T14:06:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.250010+0000", 2023-07-02T14:06:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006641816 2023-07-02T14:06:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256651+0000", 2023-07-02T14:06:43.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1968999999999998e-05 2023-07-02T14:06:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256683+0000", 2023-07-02T14:06:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012463952 2023-07-02T14:06:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269147+0000", 2023-07-02T14:06:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2855000000000006e-05 2023-07-02T14:06:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269220+0000", 2023-07-02T14:06:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010870418999999999 2023-07-02T14:06:43.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280091+0000", 2023-07-02T14:06:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1921999999999997e-05 2023-07-02T14:06:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280123+0000", 2023-07-02T14:06:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011861682 2023-07-02T14:06:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291984+0000", 2023-07-02T14:06:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5259000000000006e-05 2023-07-02T14:06:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.292059+0000", 2023-07-02T14:06:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079709359999999996 2023-07-02T14:06:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300030+0000", 2023-07-02T14:06:43.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0729999999999998e-05 2023-07-02T14:06:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300071+0000", 2023-07-02T14:06:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072785690000000004 2023-07-02T14:06:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307350+0000", 2023-07-02T14:06:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1948999999999999e-05 2023-07-02T14:06:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307412+0000", 2023-07-02T14:06:43.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055785779999999998 2023-07-02T14:06:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312990+0000", 2023-07-02T14:06:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0957000000000002e-05 2023-07-02T14:06:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.313021+0000", 2023-07-02T14:06:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049246400000000001 2023-07-02T14:06:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317946+0000", 2023-07-02T14:06:43.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9346e-05 2023-07-02T14:06:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317975+0000", 2023-07-02T14:06:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049649469999999999 2023-07-02T14:06:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322940+0000", 2023-07-02T14:06:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9017000000000002e-05 2023-07-02T14:06:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322989+0000", 2023-07-02T14:06:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065907029999999998 2023-07-02T14:06:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329580+0000", 2023-07-02T14:06:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5879999999999999e-05 2023-07-02T14:06:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329656+0000", 2023-07-02T14:06:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060166960000000002 2023-07-02T14:06:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335672+0000", 2023-07-02T14:06:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8963e-05 2023-07-02T14:06:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335701+0000", 2023-07-02T14:06:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098989560000000004 2023-07-02T14:06:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345600+0000", 2023-07-02T14:06:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5395999999999995e-05 2023-07-02T14:06:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345666+0000", 2023-07-02T14:06:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046589530000000004 2023-07-02T14:06:43.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350325+0000", 2023-07-02T14:06:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2459000000000002e-05 2023-07-02T14:06:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350357+0000", 2023-07-02T14:06:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042054290000000001 2023-07-02T14:06:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354563+0000", 2023-07-02T14:06:43.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5612999999999999e-05 2023-07-02T14:06:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354598+0000", 2023-07-02T14:06:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070967449999999998 2023-07-02T14:06:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361695+0000", 2023-07-02T14:06:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9314000000000001e-05 2023-07-02T14:06:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361724+0000", 2023-07-02T14:06:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007391028 2023-07-02T14:06:43.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369115+0000", 2023-07-02T14:06:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9163000000000002e-05 2023-07-02T14:06:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369184+0000", 2023-07-02T14:06:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056532500000000003 2023-07-02T14:06:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374838+0000", 2023-07-02T14:06:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5297999999999999e-05 2023-07-02T14:06:43.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374883+0000", 2023-07-02T14:06:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077292189999999998 2023-07-02T14:06:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382612+0000", 2023-07-02T14:06:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1102000000000001e-05 2023-07-02T14:06:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382653+0000", 2023-07-02T14:06:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058515989999999999 2023-07-02T14:06:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388505+0000", 2023-07-02T14:06:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9196000000000006e-05 2023-07-02T14:06:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388574+0000", 2023-07-02T14:06:43.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084245339999999991 2023-07-02T14:06:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.396999+0000", 2023-07-02T14:06:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0596999999999998e-05 2023-07-02T14:06:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.397039+0000", 2023-07-02T14:06:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084911339999999991 2023-07-02T14:06:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405530+0000", 2023-07-02T14:06:43.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7555000000000001e-05 2023-07-02T14:06:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405578+0000", 2023-07-02T14:06:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000229607 2023-07-02T14:06:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405808+0000", 2023-07-02T14:06:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026919940000000001 2023-07-02T14:06:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.408500+0000", 2023-07-02T14:06:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053065059999999999 2023-07-02T14:06:43.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413806+0000", 2023-07-02T14:06:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0490000000000001e-05 2023-07-02T14:06:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:43.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413837+0000", 2023-07-02T14:06:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000100386 2023-07-02T14:06:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413937+0000", 2023-07-02T14:06:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744 2023-07-02T14:06:43.238 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20768 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch ping cookie 140544751726928 gen 52] snapc 0=[] ondisk+write+known_if_redirected e183)", 2023-07-02T14:06:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 23.699530991, 2023-07-02T14:06:43.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3751256970000001, 2023-07-02T14:06:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20768 2023-07-02T14:06:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.240 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:43.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7829999999999998e-06 2023-07-02T14:06:43.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222419+0000", 2023-07-02T14:06:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1380000000000004e-06 2023-07-02T14:06:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:43.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222425+0000", 2023-07-02T14:06:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2519999999999996e-06 2023-07-02T14:06:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222431+0000", 2023-07-02T14:06:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2157999999999994e-05 2023-07-02T14:06:43.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222523+0000", 2023-07-02T14:06:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5147000000000001e-05 2023-07-02T14:06:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222538+0000", 2023-07-02T14:06:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30105419700000002 2023-07-02T14:06:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523592+0000", 2023-07-02T14:06:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2316e-05 2023-07-02T14:06:43.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523605+0000", 2023-07-02T14:06:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.073658953 2023-07-02T14:06:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597264+0000", 2023-07-02T14:06:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000213415 2023-07-02T14:06:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597477+0000", 2023-07-02T14:06:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3034999999999995e-05 2023-07-02T14:06:43.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597540+0000", 2023-07-02T14:06:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3751256970000001 2023-07-02T14:06:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:43.249 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20769 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 53] snapc 0=[] ondisk+write+known_if_redirected e184)", 2023-07-02T14:06:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 23.699168535999998, 2023-07-02T14:06:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.380884139, 2023-07-02T14:06:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:43.250 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20769 2023-07-02T14:06:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:43.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:43.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222776+0000", 2023-07-02T14:06:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9710000000000003e-06 2023-07-02T14:06:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222781+0000", 2023-07-02T14:06:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9899999999999992e-06 2023-07-02T14:06:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222791+0000", 2023-07-02T14:06:43.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4240000000000003e-06 2023-07-02T14:06:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222795+0000", 2023-07-02T14:06:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2366000000000001e-05 2023-07-02T14:06:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222828+0000", 2023-07-02T14:06:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4790000000000001e-06 2023-07-02T14:06:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222834+0000", 2023-07-02T14:06:43.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30084589299999998 2023-07-02T14:06:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523680+0000", 2023-07-02T14:06:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.87e-06 2023-07-02T14:06:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:43.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523685+0000", 2023-07-02T14:06:43.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.074111042 2023-07-02T14:06:43.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597796+0000", 2023-07-02T14:06:43.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0411000000000001e-05 2023-07-02T14:06:43.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:43.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597856+0000", 2023-07-02T14:06:43.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058045340000000001 2023-07-02T14:06:43.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:43.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.603661+0000", 2023-07-02T14:06:43.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.380884139 2023-07-02T14:06:43.260 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.260 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:43.260 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.261 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20770 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 54] snapc 0=[] ondisk+write+known_if_redirected e185)", 2023-07-02T14:06:43.261 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:43.261 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 23.397951425999999, 2023-07-02T14:06:43.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0799819349999999, 2023-07-02T14:06:43.261 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:43.261 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:43.261 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:43.262 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:43.262 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:43.262 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20770 2023-07-02T14:06:43.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.262 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:43.262 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.262 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:43.262 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:43.263 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:43.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:43.263 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:43.263 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:43.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.264 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:43.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523992+0000", 2023-07-02T14:06:43.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6720000000000004e-06 2023-07-02T14:06:43.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.264 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:43.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524000+0000", 2023-07-02T14:06:43.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3711e-05 2023-07-02T14:06:43.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:43.265 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524014+0000", 2023-07-02T14:06:43.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7849999999999992e-06 2023-07-02T14:06:43.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.266 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.266 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.266 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524022+0000", 2023-07-02T14:06:43.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2366000000000001e-05 2023-07-02T14:06:43.266 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524054+0000", 2023-07-02T14:06:43.267 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8169999999999998e-06 2023-07-02T14:06:43.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:43.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524059+0000", 2023-07-02T14:06:43.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0748338850000001 2023-07-02T14:06:43.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.268 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.598893+0000", 2023-07-02T14:06:43.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116968 2023-07-02T14:06:43.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:43.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.599010+0000", 2023-07-02T14:06:43.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049663370000000004 2023-07-02T14:06:43.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:43.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.603976+0000", 2023-07-02T14:06:43.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0799819349999999 2023-07-02T14:06:43.270 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.270 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:43.270 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.270 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20833 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 55] snapc 0=[] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:43.271 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:06:43.271 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 18.389832121000001, 2023-07-02T14:06:43.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.029349206, 2023-07-02T14:06:43.271 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:43.271 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:43.271 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:43.271 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:43.272 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:43.272 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20833 2023-07-02T14:06:43.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.272 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:43.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:43.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:06:43.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:43.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.273 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:43.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:06:43.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:43.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:43.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532112+0000", 2023-07-02T14:06:43.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7590000000000003e-06 2023-07-02T14:06:43.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:43.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532120+0000", 2023-07-02T14:06:43.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8130000000000001e-05 2023-07-02T14:06:43.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.275 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:43.275 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532139+0000", 2023-07-02T14:06:43.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3299999999999999e-06 2023-07-02T14:06:43.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532147+0000", 2023-07-02T14:06:43.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9679999999999998e-05 2023-07-02T14:06:43.276 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.276 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.276 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532207+0000", 2023-07-02T14:06:43.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0648e-05 2023-07-02T14:06:43.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.277 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op must wait for map", 2023-07-02T14:06:43.277 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532217+0000", 2023-07-02T14:06:43.277 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002082455 2023-07-02T14:06:43.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.277 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.534300+0000", 2023-07-02T14:06:43.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0710000000000001e-05 2023-07-02T14:06:43.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:43.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.534310+0000", 2023-07-02T14:06:43.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.013227235 2023-07-02T14:06:43.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.279 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.547538+0000", 2023-07-02T14:06:43.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021904300000000001 2023-07-02T14:06:43.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:43.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.547757+0000", 2023-07-02T14:06:43.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013705784 2023-07-02T14:06:43.280 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:43.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561462+0000", 2023-07-02T14:06:43.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.029349206 2023-07-02T14:06:43.281 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.281 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:43.281 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.281 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20865 2.7 2:ff0f5f99:::rbd_data.125de1449605.0000000000000006:head [write 131072~8192] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:43.282 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:06:43.282 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.645209049000002, 2023-07-02T14:06:43.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.288115762, 2023-07-02T14:06:43.282 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:43.282 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.282 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:43.282 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:43.283 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:43.283 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20865 2023-07-02T14:06:43.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.283 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:43.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:43.283 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:06:43.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:43.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:43.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:06:43.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:43.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:43.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276735+0000", 2023-07-02T14:06:43.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3273e-05 2023-07-02T14:06:43.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:43.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276748+0000", 2023-07-02T14:06:43.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3895e-05 2023-07-02T14:06:43.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:43.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276762+0000", 2023-07-02T14:06:43.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.896e-06 2023-07-02T14:06:43.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.286 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.286 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276768+0000", 2023-07-02T14:06:43.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5850999999999999e-05 2023-07-02T14:06:43.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.287 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.287 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276814+0000", 2023-07-02T14:06:43.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5290000000000003e-06 2023-07-02T14:06:43.287 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.287 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:43.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276821+0000", 2023-07-02T14:06:43.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.27225312600000001 2023-07-02T14:06:43.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549074+0000", 2023-07-02T14:06:43.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000151223 2023-07-02T14:06:43.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:43.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549226+0000", 2023-07-02T14:06:43.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024865400000000001 2023-07-02T14:06:43.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:43.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549474+0000", 2023-07-02T14:06:43.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012039185000000001 2023-07-02T14:06:43.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:43.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561513+0000", 2023-07-02T14:06:43.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003221327 2023-07-02T14:06:43.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:43.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564735+0000", 2023-07-02T14:06:43.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2026000000000005e-05 2023-07-02T14:06:43.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:43.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564807+0000", 2023-07-02T14:06:43.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5303000000000003e-05 2023-07-02T14:06:43.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:43.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564852+0000", 2023-07-02T14:06:43.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.288115762 2023-07-02T14:06:43.293 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.293 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:43.293 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.293 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20866 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 131072~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:43.293 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:06:43.293 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.639515727999999, 2023-07-02T14:06:43.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.28773701200000001, 2023-07-02T14:06:43.294 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:43.294 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.294 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:43.294 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:43.294 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:43.294 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20866 2023-07-02T14:06:43.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.295 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:43.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.295 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:43.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:06:43.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:43.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.295 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:43.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:06:43.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:43.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:43.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282428+0000", 2023-07-02T14:06:43.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8469999999999998e-05 2023-07-02T14:06:43.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:43.297 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282517+0000", 2023-07-02T14:06:43.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2974e-05 2023-07-02T14:06:43.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.297 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:43.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282530+0000", 2023-07-02T14:06:43.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4530000000000001e-06 2023-07-02T14:06:43.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.298 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282535+0000", 2023-07-02T14:06:43.298 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1926999999999999e-05 2023-07-02T14:06:43.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282587+0000", 2023-07-02T14:06:43.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3083e-05 2023-07-02T14:06:43.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:43.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282600+0000", 2023-07-02T14:06:43.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.270692971 2023-07-02T14:06:43.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553293+0000", 2023-07-02T14:06:43.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017065599999999999 2023-07-02T14:06:43.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:43.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553464+0000", 2023-07-02T14:06:43.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026504399999999998 2023-07-02T14:06:43.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:43.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553729+0000", 2023-07-02T14:06:43.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011173008 2023-07-02T14:06:43.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:43.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564902+0000", 2023-07-02T14:06:43.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051242600000000003 2023-07-02T14:06:43.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:43.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570026+0000", 2023-07-02T14:06:43.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8098999999999999e-05 2023-07-02T14:06:43.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:43.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570104+0000", 2023-07-02T14:06:43.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2323000000000006e-05 2023-07-02T14:06:43.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:43.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570167+0000", 2023-07-02T14:06:43.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.28773701200000001 2023-07-02T14:06:43.304 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.305 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:43.305 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.305 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20875 2.7 2:e3210266:::rbd_data.125de1449605.00000000000001a3:head [write 0~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:43.305 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:06:43.305 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.600464912, 2023-07-02T14:06:43.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.25297039599999999, 2023-07-02T14:06:43.306 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:43.306 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.306 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:43.306 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:43.306 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:43.306 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20875 2023-07-02T14:06:43.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.306 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:43.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:43.307 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:06:43.307 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:43.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.307 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:43.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:06:43.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:43.308 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.308 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.308 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:43.308 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321479+0000", 2023-07-02T14:06:43.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015194399999999999 2023-07-02T14:06:43.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:43.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321631+0000", 2023-07-02T14:06:43.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4236999999999999e-05 2023-07-02T14:06:43.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:43.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321645+0000", 2023-07-02T14:06:43.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2160000000000003e-06 2023-07-02T14:06:43.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321651+0000", 2023-07-02T14:06:43.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3609999999999998e-05 2023-07-02T14:06:43.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321695+0000", 2023-07-02T14:06:43.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1989999999999995e-06 2023-07-02T14:06:43.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:43.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321704+0000", 2023-07-02T14:06:43.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.23542881500000001 2023-07-02T14:06:43.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557133+0000", 2023-07-02T14:06:43.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017359600000000001 2023-07-02T14:06:43.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:43.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557307+0000", 2023-07-02T14:06:43.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00023367599999999999 2023-07-02T14:06:43.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:43.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557540+0000", 2023-07-02T14:06:43.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016744963000000002 2023-07-02T14:06:43.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:43.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574285+0000", 2023-07-02T14:06:43.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1330999999999999e-05 2023-07-02T14:06:43.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:43.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574346+0000", 2023-07-02T14:06:43.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9750000000000002e-05 2023-07-02T14:06:43.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:43.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574406+0000", 2023-07-02T14:06:43.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4647999999999997e-05 2023-07-02T14:06:43.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.317 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:43.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574451+0000", 2023-07-02T14:06:43.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.25297039599999999 2023-07-02T14:06:43.317 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.317 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:43.317 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.317 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.318 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.318 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20915 2.7 2:e6e64067:::rbd_data.125de1449605.00000000000001a9:head [write 0~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:43.318 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:06:43.318 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.537211116999998, 2023-07-02T14:06:43.318 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.19966078500000001, 2023-07-02T14:06:43.318 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:43.319 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.319 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:43.319 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:43.319 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:43.319 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20915 2023-07-02T14:06:43.319 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.319 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:43.319 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:43.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:06:43.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:43.320 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.320 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.320 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:43.320 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:06:43.320 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:43.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:43.321 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384732+0000", 2023-07-02T14:06:43.321 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9439999999999999e-05 2023-07-02T14:06:43.321 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.321 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.321 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:43.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384802+0000", 2023-07-02T14:06:43.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7572999999999999e-05 2023-07-02T14:06:43.322 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.322 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.322 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:43.322 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384819+0000", 2023-07-02T14:06:43.322 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1670000000000004e-06 2023-07-02T14:06:43.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.323 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384825+0000", 2023-07-02T14:06:43.323 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6471999999999998e-05 2023-07-02T14:06:43.323 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.323 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.323 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384872+0000", 2023-07-02T14:06:43.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0028e-05 2023-07-02T14:06:43.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.324 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.324 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:43.324 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384882+0000", 2023-07-02T14:06:43.324 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.17674214099999999 2023-07-02T14:06:43.324 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561624+0000", 2023-07-02T14:06:43.325 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000135809 2023-07-02T14:06:43.325 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.325 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.325 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:43.325 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561760+0000", 2023-07-02T14:06:43.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020468 2023-07-02T14:06:43.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.326 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:43.326 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561965+0000", 2023-07-02T14:06:43.326 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012498198 2023-07-02T14:06:43.326 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.326 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:43.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574463+0000", 2023-07-02T14:06:43.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098197509999999998 2023-07-02T14:06:43.327 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.327 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.327 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:43.327 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584283+0000", 2023-07-02T14:06:43.327 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2056000000000004e-05 2023-07-02T14:06:43.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.328 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:43.328 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584345+0000", 2023-07-02T14:06:43.328 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0513000000000002e-05 2023-07-02T14:06:43.328 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.328 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.329 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:43.329 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584395+0000", 2023-07-02T14:06:43.329 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.19966078500000001 2023-07-02T14:06:43.329 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.329 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:43.329 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.329 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.329 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.330 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20948 2.7 2:ff0f5f99:::rbd_data.125de1449605.0000000000000006:head [write 139264~2220032] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:43.330 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.450499+0000", 2023-07-02T14:06:43.330 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.471446390000001, 2023-07-02T14:06:43.330 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13400999399999999, 2023-07-02T14:06:43.330 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:43.330 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.330 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:43.331 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:43.331 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:43.331 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20948 2023-07-02T14:06:43.331 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.331 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:43.331 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.331 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:43.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.450499+0000", 2023-07-02T14:06:43.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:43.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.332 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.332 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:43.332 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.450499+0000", 2023-07-02T14:06:43.332 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:43.332 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:43.333 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.450497+0000", 2023-07-02T14:06:43.333 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043010360000000003 2023-07-02T14:06:43.333 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.333 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.333 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:43.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454798+0000", 2023-07-02T14:06:43.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8073000000000001e-05 2023-07-02T14:06:43.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.334 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.334 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:43.334 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454816+0000", 2023-07-02T14:06:43.334 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3769999999999996e-06 2023-07-02T14:06:43.334 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454825+0000", 2023-07-02T14:06:43.335 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2797999999999998e-05 2023-07-02T14:06:43.335 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.335 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.335 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.335 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454878+0000", 2023-07-02T14:06:43.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0020999999999999e-05 2023-07-02T14:06:43.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.336 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:43.336 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454888+0000", 2023-07-02T14:06:43.336 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.110043313 2023-07-02T14:06:43.336 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.336 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564931+0000", 2023-07-02T14:06:43.337 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7185e-05 2023-07-02T14:06:43.337 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.337 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.337 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:43.337 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.565008+0000", 2023-07-02T14:06:43.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014058199999999999 2023-07-02T14:06:43.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.338 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:43.338 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.565149+0000", 2023-07-02T14:06:43.338 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093221559999999998 2023-07-02T14:06:43.338 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.338 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:43.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574471+0000", 2023-07-02T14:06:43.339 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099660410000000001 2023-07-02T14:06:43.339 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.339 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.339 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:43.339 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584437+0000", 2023-07-02T14:06:43.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2805000000000001e-05 2023-07-02T14:06:43.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.340 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:43.340 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584480+0000", 2023-07-02T14:06:43.340 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9393000000000001e-05 2023-07-02T14:06:43.340 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.340 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.341 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:43.341 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584509+0000", 2023-07-02T14:06:43.341 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13400999399999999 2023-07-02T14:06:43.341 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.341 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:43.341 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.341 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.341 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.342 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20949 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 196608~2162688] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:43.342 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.454838+0000", 2023-07-02T14:06:43.342 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.467107231, 2023-07-02T14:06:43.342 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12983145400000001, 2023-07-02T14:06:43.342 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:43.342 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.342 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:43.342 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:43.343 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:43.343 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20949 2023-07-02T14:06:43.343 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.343 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:43.343 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.343 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:43.343 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454838+0000", 2023-07-02T14:06:43.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:43.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.344 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.344 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:43.344 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454838+0000", 2023-07-02T14:06:43.344 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:43.344 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:43.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454837+0000", 2023-07-02T14:06:43.345 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003772821 2023-07-02T14:06:43.345 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.345 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.345 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:43.345 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458610+0000", 2023-07-02T14:06:43.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3013999999999999e-05 2023-07-02T14:06:43.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.346 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.346 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:43.346 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458623+0000", 2023-07-02T14:06:43.346 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2789999999999998e-06 2023-07-02T14:06:43.346 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458628+0000", 2023-07-02T14:06:43.347 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7630999999999998e-05 2023-07-02T14:06:43.347 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.347 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.347 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.347 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458676+0000", 2023-07-02T14:06:43.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1579999999999994e-06 2023-07-02T14:06:43.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.348 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.348 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:43.348 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458685+0000", 2023-07-02T14:06:43.348 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.111536444 2023-07-02T14:06:43.348 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570221+0000", 2023-07-02T14:06:43.349 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000120844 2023-07-02T14:06:43.349 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.349 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.349 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:43.349 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570342+0000", 2023-07-02T14:06:43.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000149965 2023-07-02T14:06:43.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.350 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.350 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:43.350 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570492+0000", 2023-07-02T14:06:43.350 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014027843999999999 2023-07-02T14:06:43.350 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:43.351 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584520+0000", 2023-07-02T14:06:43.351 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8446000000000003e-05 2023-07-02T14:06:43.351 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.351 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.351 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:43.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584568+0000", 2023-07-02T14:06:43.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6771999999999998e-05 2023-07-02T14:06:43.352 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.352 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.352 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:43.352 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584625+0000", 2023-07-02T14:06:43.352 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4401999999999999e-05 2023-07-02T14:06:43.353 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.353 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.353 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:43.353 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584670+0000", 2023-07-02T14:06:43.353 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12983145400000001 2023-07-02T14:06:43.353 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.353 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:43.353 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.354 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.354 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.354 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20972 2.7 2:e3210266:::rbd_data.125de1449605.00000000000001a3:head [write 65536~4128768] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:43.354 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.463174+0000", 2023-07-02T14:06:43.354 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.458771390999999, 2023-07-02T14:06:43.354 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12719914800000001, 2023-07-02T14:06:43.354 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:43.354 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.355 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:43.355 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:43.355 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:43.355 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20972 2023-07-02T14:06:43.355 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.355 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:43.355 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:43.356 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.463174+0000", 2023-07-02T14:06:43.356 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:43.356 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.356 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.356 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:43.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.463174+0000", 2023-07-02T14:06:43.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:43.357 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.357 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.357 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:43.357 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.463173+0000", 2023-07-02T14:06:43.357 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069882069999999998 2023-07-02T14:06:43.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:43.358 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470161+0000", 2023-07-02T14:06:43.358 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0103999999999999e-05 2023-07-02T14:06:43.358 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.358 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.358 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:43.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470181+0000", 2023-07-02T14:06:43.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:06:43.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.359 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.359 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.359 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470189+0000", 2023-07-02T14:06:43.359 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0970999999999994e-05 2023-07-02T14:06:43.359 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.360 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470270+0000", 2023-07-02T14:06:43.360 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4389000000000001e-05 2023-07-02T14:06:43.360 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.360 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.360 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:43.361 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470284+0000", 2023-07-02T14:06:43.361 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.10421862799999999 2023-07-02T14:06:43.361 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.361 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.361 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:43.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574503+0000", 2023-07-02T14:06:43.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2457e-05 2023-07-02T14:06:43.362 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.362 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.362 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:43.362 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574585+0000", 2023-07-02T14:06:43.362 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013870899999999999 2023-07-02T14:06:43.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.363 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:43.363 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574724+0000", 2023-07-02T14:06:43.363 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012886246000000001 2023-07-02T14:06:43.363 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.363 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:43.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.587610+0000", 2023-07-02T14:06:43.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026883599999999999 2023-07-02T14:06:43.364 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.364 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.364 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:43.364 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.590298+0000", 2023-07-02T14:06:43.364 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1850000000000001e-05 2023-07-02T14:06:43.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.365 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:43.365 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.590340+0000", 2023-07-02T14:06:43.365 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2901999999999998e-05 2023-07-02T14:06:43.365 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:43.365 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:43.366 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:43.366 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.590373+0000", 2023-07-02T14:06:43.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12719914800000001 2023-07-02T14:06:43.366 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.366 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:43.366 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.366 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:43.366 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:43.367 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:43.368 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:06:43.368 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 1 2023-07-02T14:06:43.369 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:06:43.370 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:06:43.370 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:43.368+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:43.468 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:43.468+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:43.540 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:43.540 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:43.540 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:43.541 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:43.550 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:06:43.591 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:43.591+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:43.669 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:43.668+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:43.714 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:43.714 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:43.714 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:06:43.714 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:06:43.714 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:43.724 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:06:43.770 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:43.770+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:43.849 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:43.850 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:06:43.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:06:43.850 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:43.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.850 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18643 2.5 e176/170 2:a9ace6f8:::rbd_data.125de1449605.0000000000000187:head v 176'2314, mlcod=173'2311)", 2023-07-02T14:06:43.850 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:06:43.851 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.421484188999997, 2023-07-02T14:06:43.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015027297, 2023-07-02T14:06:43.851 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:43.851 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.851 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:43.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:43.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:06:43.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:43.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:43.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:06:43.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:43.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:43.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426547+0000", 2023-07-02T14:06:43.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056864430000000002 2023-07-02T14:06:43.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:43.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432234+0000", 2023-07-02T14:06:43.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.415e-05 2023-07-02T14:06:43.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.854 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:43.854 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432248+0000", 2023-07-02T14:06:43.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7490000000000001e-06 2023-07-02T14:06:43.854 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432255+0000", 2023-07-02T14:06:43.855 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3915000000000002e-05 2023-07-02T14:06:43.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:43.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432299+0000", 2023-07-02T14:06:43.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1214000000000001e-05 2023-07-02T14:06:43.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:43.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432320+0000", 2023-07-02T14:06:43.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0092114950000000001 2023-07-02T14:06:43.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:43.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.441531+0000", 2023-07-02T14:06:43.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4422999999999998e-05 2023-07-02T14:06:43.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:43.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.441576+0000", 2023-07-02T14:06:43.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015027297 2023-07-02T14:06:43.858 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.858 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:43.858 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.858 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18680 2.2 e176/171 2:5fd2923a:::rbd_data.125de1449605.000000000000018c:head v 176'1639, mlcod=176'1637)", 2023-07-02T14:06:43.858 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.489711+0000", 2023-07-02T14:06:43.859 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 38.358321746999998, 2023-07-02T14:06:43.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014407402, 2023-07-02T14:06:43.859 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:43.859 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.859 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:43.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:43.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.489711+0000", 2023-07-02T14:06:43.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:43.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:43.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.489711+0000", 2023-07-02T14:06:43.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:43.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:43.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.489709+0000", 2023-07-02T14:06:43.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001991792 2023-07-02T14:06:43.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:43.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491701+0000", 2023-07-02T14:06:43.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7323000000000002e-05 2023-07-02T14:06:43.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:43.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491738+0000", 2023-07-02T14:06:43.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2483999999999999e-05 2023-07-02T14:06:43.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491751+0000", 2023-07-02T14:06:43.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7896000000000003e-05 2023-07-02T14:06:43.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:43.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491809+0000", 2023-07-02T14:06:43.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0185000000000001e-05 2023-07-02T14:06:43.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.864 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:43.864 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491829+0000", 2023-07-02T14:06:43.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012250901999999999 2023-07-02T14:06:43.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:43.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.504080+0000", 2023-07-02T14:06:43.865 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:06:43.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.865 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.865 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:43.865 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.504118+0000", 2023-07-02T14:06:43.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014407402 2023-07-02T14:06:43.866 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.866 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:43.866 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.866 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18881 2.5 e177/170 2:b1731d2f:::rbd_data.125de1449605.0000000000000024:head v 177'2350, mlcod=177'2347)", 2023-07-02T14:06:43.866 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:06:43.867 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.942118364999999, 2023-07-02T14:06:43.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020820709999999999, 2023-07-02T14:06:43.867 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:43.867 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.867 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:43.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:43.867 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:06:43.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:43.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:43.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:06:43.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:43.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:43.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905913+0000", 2023-07-02T14:06:43.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0080191519999999999 2023-07-02T14:06:43.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:43.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913932+0000", 2023-07-02T14:06:43.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3757999999999999e-05 2023-07-02T14:06:43.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:43.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913956+0000", 2023-07-02T14:06:43.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4009999999999997e-06 2023-07-02T14:06:43.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913965+0000", 2023-07-02T14:06:43.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3383e-05 2023-07-02T14:06:43.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:43.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914018+0000", 2023-07-02T14:06:43.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7674e-05 2023-07-02T14:06:43.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:43.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914046+0000", 2023-07-02T14:06:43.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012656555999999999 2023-07-02T14:06:43.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:43.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.926702+0000", 2023-07-02T14:06:43.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2891000000000001e-05 2023-07-02T14:06:43.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:43.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.926735+0000", 2023-07-02T14:06:43.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020820709999999999 2023-07-02T14:06:43.874 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.874 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:43.874 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.874 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19073 2.5 e177/170 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:head v 177'2378, mlcod=177'2376)", 2023-07-02T14:06:43.874 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:06:43.875 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.574405182, 2023-07-02T14:06:43.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018435625000000001, 2023-07-02T14:06:43.875 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:43.875 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.876 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:43.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:43.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:06:43.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:43.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:43.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:06:43.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:43.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:43.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273626+0000", 2023-07-02T14:06:43.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046035659999999999 2023-07-02T14:06:43.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:43.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278230+0000", 2023-07-02T14:06:43.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9386e-05 2023-07-02T14:06:43.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:43.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278259+0000", 2023-07-02T14:06:43.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1972e-05 2023-07-02T14:06:43.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278271+0000", 2023-07-02T14:06:43.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017532680000000001 2023-07-02T14:06:43.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:43.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280025+0000", 2023-07-02T14:06:43.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2367000000000001e-05 2023-07-02T14:06:43.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:43.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280047+0000", 2023-07-02T14:06:43.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011987028 2023-07-02T14:06:43.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:43.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.292034+0000", 2023-07-02T14:06:43.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9266000000000001e-05 2023-07-02T14:06:43.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:43.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.292063+0000", 2023-07-02T14:06:43.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018435625000000001 2023-07-02T14:06:43.882 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.882 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:43.882 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.883 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19075 2.5 e177/170 2:adf18789:::rbd_data.125de1449605.000000000000003c:head v 177'2379, mlcod=177'2376)", 2023-07-02T14:06:43.883 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.280379+0000", 2023-07-02T14:06:43.883 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.567654115000003, 2023-07-02T14:06:43.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014230578000000001, 2023-07-02T14:06:43.883 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:43.883 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.884 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:43.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:43.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280379+0000", 2023-07-02T14:06:43.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:43.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.885 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:43.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280379+0000", 2023-07-02T14:06:43.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999804 2023-07-02T14:06:43.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.885 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:43.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280359+0000", 2023-07-02T14:06:43.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0047641209999999996 2023-07-02T14:06:43.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:43.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.285123+0000", 2023-07-02T14:06:43.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3711e-05 2023-07-02T14:06:43.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:43.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.285137+0000", 2023-07-02T14:06:43.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.277e-06 2023-07-02T14:06:43.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.285144+0000", 2023-07-02T14:06:43.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00041800499999999999 2023-07-02T14:06:43.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:43.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.285562+0000", 2023-07-02T14:06:43.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6143000000000001e-05 2023-07-02T14:06:43.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:43.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.285578+0000", 2023-07-02T14:06:43.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0089876420000000005 2023-07-02T14:06:43.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:43.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.294566+0000", 2023-07-02T14:06:43.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3466e-05 2023-07-02T14:06:43.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:43.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.294609+0000", 2023-07-02T14:06:43.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014230578000000001 2023-07-02T14:06:43.890 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.890 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:43.890 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.891 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19196 2.4 e177/169 2:2c79e900:::rbd_data.125de1449605.000000000000004a:head v 177'1774, mlcod=177'1772)", 2023-07-02T14:06:43.891 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.475437+0000", 2023-07-02T14:06:43.891 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.372596024000003, 2023-07-02T14:06:43.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013623751, 2023-07-02T14:06:43.891 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:43.891 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.891 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:43.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:43.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475437+0000", 2023-07-02T14:06:43.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:43.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:43.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475437+0000", 2023-07-02T14:06:43.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:06:43.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:43.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475434+0000", 2023-07-02T14:06:43.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3578999999999994e-05 2023-07-02T14:06:43.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:43.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475518+0000", 2023-07-02T14:06:43.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1090999999999998e-05 2023-07-02T14:06:43.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:43.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475539+0000", 2023-07-02T14:06:43.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4860000000000003e-06 2023-07-02T14:06:43.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475549+0000", 2023-07-02T14:06:43.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000107015 2023-07-02T14:06:43.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:43.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475656+0000", 2023-07-02T14:06:43.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6625e-05 2023-07-02T14:06:43.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:43.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.475682+0000", 2023-07-02T14:06:43.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013334037 2023-07-02T14:06:43.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:43.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489016+0000", 2023-07-02T14:06:43.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4141999999999999e-05 2023-07-02T14:06:43.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:43.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489060+0000", 2023-07-02T14:06:43.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013623751 2023-07-02T14:06:43.898 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.898 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:43.898 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.899 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:20909 2.5 e189/170 2:bb7db89c:::rbd_data.125de1449605.00000000000001a5:head v 189'2620, mlcod=189'2618)", 2023-07-02T14:06:43.899 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.368184+0000", 2023-07-02T14:06:43.899 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.479849105, 2023-07-02T14:06:43.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016727725999999998, 2023-07-02T14:06:43.899 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:43.899 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.899 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:43.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:43.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.368184+0000", 2023-07-02T14:06:43.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:43.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:43.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.368184+0000", 2023-07-02T14:06:43.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:43.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:43.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.368182+0000", 2023-07-02T14:06:43.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067720749999999998 2023-07-02T14:06:43.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:43.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.374954+0000", 2023-07-02T14:06:43.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4501000000000001e-05 2023-07-02T14:06:43.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:43.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.374969+0000", 2023-07-02T14:06:43.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4100000000000002e-06 2023-07-02T14:06:43.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.374976+0000", 2023-07-02T14:06:43.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.736e-05 2023-07-02T14:06:43.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:43.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.375034+0000", 2023-07-02T14:06:43.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4544000000000001e-05 2023-07-02T14:06:43.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:43.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.375058+0000", 2023-07-02T14:06:43.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0098134910000000006 2023-07-02T14:06:43.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:43.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.384872+0000", 2023-07-02T14:06:43.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9687000000000001e-05 2023-07-02T14:06:43.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.384911+0000", 2023-07-02T14:06:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016727725999999998 2023-07-02T14:06:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.907 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:21096 2.5 e190/170 2:a0acb434:::rbd_data.125de1449605.00000000000001bd:head v 190'2623, mlcod=189'2621)", 2023-07-02T14:06:43.907 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.705165+0000", 2023-07-02T14:06:43.907 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.142867269, 2023-07-02T14:06:43.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027008152000000001, 2023-07-02T14:06:43.907 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:43.907 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.907 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:43.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:43.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.705165+0000", 2023-07-02T14:06:43.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:43.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:43.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.705165+0000", 2023-07-02T14:06:43.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:43.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:43.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.705164+0000", 2023-07-02T14:06:43.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017192187000000001 2023-07-02T14:06:43.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:43.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722356+0000", 2023-07-02T14:06:43.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0043e-05 2023-07-02T14:06:43.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:43.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722387+0000", 2023-07-02T14:06:43.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8230000000000001e-06 2023-07-02T14:06:43.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722395+0000", 2023-07-02T14:06:43.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9427e-05 2023-07-02T14:06:43.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:43.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722445+0000", 2023-07-02T14:06:43.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4321999999999999e-05 2023-07-02T14:06:43.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:43.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722469+0000", 2023-07-02T14:06:43.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0096570649999999994 2023-07-02T14:06:43.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:43.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.732126+0000", 2023-07-02T14:06:43.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7490000000000001e-05 2023-07-02T14:06:43.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:43.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.732174+0000", 2023-07-02T14:06:43.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027008152000000001 2023-07-02T14:06:43.914 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.914 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:43.914 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.914 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22204 2.4 e197/169 2:25e15f0e:::rbd_data.125de1449605.0000000000000087:head v 197'2046, mlcod=197'2043)", 2023-07-02T14:06:43.915 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.493042+0000", 2023-07-02T14:06:43.915 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.3549910060000006, 2023-07-02T14:06:43.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016388874000000001, 2023-07-02T14:06:43.915 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:43.915 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.915 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:43.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:43.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493042+0000", 2023-07-02T14:06:43.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:43.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:43.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493042+0000", 2023-07-02T14:06:43.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:43.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:43.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493041+0000", 2023-07-02T14:06:43.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9621999999999999e-05 2023-07-02T14:06:43.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:43.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493070+0000", 2023-07-02T14:06:43.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3439e-05 2023-07-02T14:06:43.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:43.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493084+0000", 2023-07-02T14:06:43.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6720000000000004e-06 2023-07-02T14:06:43.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493092+0000", 2023-07-02T14:06:43.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4440000000000001e-05 2023-07-02T14:06:43.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:43.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493136+0000", 2023-07-02T14:06:43.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3710000000000002e-05 2023-07-02T14:06:43.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:43.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493160+0000", 2023-07-02T14:06:43.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016220526999999998 2023-07-02T14:06:43.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:43.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.509380+0000", 2023-07-02T14:06:43.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0417000000000002e-05 2023-07-02T14:06:43.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:43.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.509431+0000", 2023-07-02T14:06:43.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016388874000000001 2023-07-02T14:06:43.922 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.922 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:43.922 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.922 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22261 2.4 e197/169 2:2a8b9a55:::rbd_data.125de1449605.0000000000000088:head v 197'2053, mlcod=197'2050)", 2023-07-02T14:06:43.922 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.616181+0000", 2023-07-02T14:06:43.923 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.2318518580000006, 2023-07-02T14:06:43.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014816169000000001, 2023-07-02T14:06:43.923 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:43.923 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.923 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:43.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:43.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.616181+0000", 2023-07-02T14:06:43.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:43.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:43.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.616181+0000", 2023-07-02T14:06:43.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:43.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:43.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.616180+0000", 2023-07-02T14:06:43.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00089455100000000005 2023-07-02T14:06:43.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:43.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617074+0000", 2023-07-02T14:06:43.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9324e-05 2023-07-02T14:06:43.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:43.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617093+0000", 2023-07-02T14:06:43.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2199999999999998e-06 2023-07-02T14:06:43.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617103+0000", 2023-07-02T14:06:43.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013166799999999999 2023-07-02T14:06:43.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:43.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617234+0000", 2023-07-02T14:06:43.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9300000000000001e-05 2023-07-02T14:06:43.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:43.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617264+0000", 2023-07-02T14:06:43.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013684961000000001 2023-07-02T14:06:43.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:43.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.630949+0000", 2023-07-02T14:06:43.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8435e-05 2023-07-02T14:06:43.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:43.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.630997+0000", 2023-07-02T14:06:43.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014816169000000001 2023-07-02T14:06:43.930 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.930 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:43.930 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.930 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22265 2.5 e197/170 2:b033b7a2:::rbd_data.125de1449605.000000000000008a:head v 197'2737, mlcod=197'2734)", 2023-07-02T14:06:43.930 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.620697+0000", 2023-07-02T14:06:43.930 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.2273361020000007, 2023-07-02T14:06:43.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016340187999999999, 2023-07-02T14:06:43.931 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:43.931 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.931 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:43.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:43.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.620697+0000", 2023-07-02T14:06:43.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:43.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:43.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.620697+0000", 2023-07-02T14:06:43.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:43.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:43.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.620695+0000", 2023-07-02T14:06:43.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029799269999999998 2023-07-02T14:06:43.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:43.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623675+0000", 2023-07-02T14:06:43.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6534000000000001e-05 2023-07-02T14:06:43.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:43.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623692+0000", 2023-07-02T14:06:43.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0400000000000004e-06 2023-07-02T14:06:43.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623699+0000", 2023-07-02T14:06:43.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3189999999999997e-05 2023-07-02T14:06:43.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:43.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623752+0000", 2023-07-02T14:06:43.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2078000000000001e-05 2023-07-02T14:06:43.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:43.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623774+0000", 2023-07-02T14:06:43.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013226343999999999 2023-07-02T14:06:43.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:43.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.637001+0000", 2023-07-02T14:06:43.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6279000000000003e-05 2023-07-02T14:06:43.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:43.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.637037+0000", 2023-07-02T14:06:43.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016340187999999999 2023-07-02T14:06:43.937 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.937 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:43.938 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.938 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22352 2.5 e197/170 2:ac0b3e68:::rbd_data.125de1449605.00000000000000a2:head v 197'2748, mlcod=197'2745)", 2023-07-02T14:06:43.938 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.836234+0000", 2023-07-02T14:06:43.938 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 9.0117987050000004, 2023-07-02T14:06:43.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015976187999999999, 2023-07-02T14:06:43.938 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:43.939 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.939 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:43.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:43.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.836234+0000", 2023-07-02T14:06:43.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:43.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:43.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.836234+0000", 2023-07-02T14:06:43.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:43.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:43.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.836233+0000", 2023-07-02T14:06:43.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066101179999999999 2023-07-02T14:06:43.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:43.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842843+0000", 2023-07-02T14:06:43.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9471e-05 2023-07-02T14:06:43.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:43.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842862+0000", 2023-07-02T14:06:43.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3270000000000007e-06 2023-07-02T14:06:43.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842872+0000", 2023-07-02T14:06:43.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0770999999999998e-05 2023-07-02T14:06:43.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:43.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842933+0000", 2023-07-02T14:06:43.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0375999999999999e-05 2023-07-02T14:06:43.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:43.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842963+0000", 2023-07-02T14:06:43.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091962380000000007 2023-07-02T14:06:43.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:43.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.852159+0000", 2023-07-02T14:06:43.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1100999999999997e-05 2023-07-02T14:06:43.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:43.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.852210+0000", 2023-07-02T14:06:43.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015976187999999999 2023-07-02T14:06:43.945 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.945 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:43.945 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.946 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22416 2.5 e197/170 2:aa88fbae:::rbd_data.125de1449605.00000000000000af:head v 197'2769, mlcod=197'2766)", 2023-07-02T14:06:43.946 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.954925+0000", 2023-07-02T14:06:43.946 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.8931080920000003, 2023-07-02T14:06:43.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017286838999999998, 2023-07-02T14:06:43.946 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:43.946 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.947 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:43.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:43.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.954925+0000", 2023-07-02T14:06:43.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:43.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:43.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.954925+0000", 2023-07-02T14:06:43.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:43.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:43.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.954924+0000", 2023-07-02T14:06:43.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007392755 2023-07-02T14:06:43.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:43.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962316+0000", 2023-07-02T14:06:43.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9266000000000002e-05 2023-07-02T14:06:43.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:43.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962336+0000", 2023-07-02T14:06:43.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0961000000000001e-05 2023-07-02T14:06:43.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962347+0000", 2023-07-02T14:06:43.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6159999999999998e-05 2023-07-02T14:06:43.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:43.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962403+0000", 2023-07-02T14:06:43.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7605000000000003e-05 2023-07-02T14:06:43.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:43.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962440+0000", 2023-07-02T14:06:43.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097334670000000009 2023-07-02T14:06:43.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:43.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.972174+0000", 2023-07-02T14:06:43.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7759000000000002e-05 2023-07-02T14:06:43.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:43.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.972211+0000", 2023-07-02T14:06:43.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017286838999999998 2023-07-02T14:06:43.953 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.953 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:43.953 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.954 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22470 2.5 e197/170 2:af9d8f16:::rbd_data.125de1449605.00000000000000b2:head v 197'2772, mlcod=197'2769)", 2023-07-02T14:06:43.954 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.062494+0000", 2023-07-02T14:06:43.954 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.785538893, 2023-07-02T14:06:43.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018691016000000001, 2023-07-02T14:06:43.954 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:43.954 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.954 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:43.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:43.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.062494+0000", 2023-07-02T14:06:43.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:43.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:43.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.062494+0000", 2023-07-02T14:06:43.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:43.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:43.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.062492+0000", 2023-07-02T14:06:43.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010932252999999999 2023-07-02T14:06:43.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:43.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073424+0000", 2023-07-02T14:06:43.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3642000000000001e-05 2023-07-02T14:06:43.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:43.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073438+0000", 2023-07-02T14:06:43.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6270000000000002e-06 2023-07-02T14:06:43.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073445+0000", 2023-07-02T14:06:43.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0265999999999997e-05 2023-07-02T14:06:43.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:43.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073485+0000", 2023-07-02T14:06:43.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5221000000000001e-05 2023-07-02T14:06:43.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:43.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073510+0000", 2023-07-02T14:06:43.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076289959999999999 2023-07-02T14:06:43.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:43.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.081139+0000", 2023-07-02T14:06:43.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5618999999999998e-05 2023-07-02T14:06:43.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.961 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:43.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.081185+0000", 2023-07-02T14:06:43.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018691016000000001 2023-07-02T14:06:43.961 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.961 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:43.961 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.962 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22516 2.5 e197/170 2:a545e9c7:::rbd_data.125de1449605.00000000000000c3:head v 197'2781, mlcod=197'2778)", 2023-07-02T14:06:43.962 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.146937+0000", 2023-07-02T14:06:43.962 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.7010955729999999, 2023-07-02T14:06:43.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015143108000000001, 2023-07-02T14:06:43.962 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:43.962 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.962 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:43.962 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:43.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.146937+0000", 2023-07-02T14:06:43.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:43.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:43.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.146937+0000", 2023-07-02T14:06:43.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:43.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:43.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.146937+0000", 2023-07-02T14:06:43.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8502000000000001e-05 2023-07-02T14:06:43.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:43.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.147025+0000", 2023-07-02T14:06:43.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4660000000000005e-06 2023-07-02T14:06:43.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:43.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.147035+0000", 2023-07-02T14:06:43.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3360000000000002e-06 2023-07-02T14:06:43.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.147040+0000", 2023-07-02T14:06:43.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004086124 2023-07-02T14:06:43.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:43.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.151126+0000", 2023-07-02T14:06:43.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9261999999999999e-05 2023-07-02T14:06:43.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:43.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.151155+0000", 2023-07-02T14:06:43.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010884783 2023-07-02T14:06:43.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:43.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.162040+0000", 2023-07-02T14:06:43.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0268000000000001e-05 2023-07-02T14:06:43.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:43.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.162080+0000", 2023-07-02T14:06:43.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015143108000000001 2023-07-02T14:06:43.969 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.969 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:43.969 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.969 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22517 2.5 e197/170 2:b58a101b:::rbd_data.125de1449605.00000000000000c2:head v 197'2783, mlcod=197'2778)", 2023-07-02T14:06:43.970 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.148357+0000", 2023-07-02T14:06:43.970 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.6996753590000004, 2023-07-02T14:06:43.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013758195000000001, 2023-07-02T14:06:43.970 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:43.970 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.970 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:43.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:43.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.148357+0000", 2023-07-02T14:06:43.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:43.971 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:43.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.148357+0000", 2023-07-02T14:06:43.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:43.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:43.972 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.148357+0000", 2023-07-02T14:06:43.972 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016233799999999999 2023-07-02T14:06:43.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:43.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.148519+0000", 2023-07-02T14:06:43.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.5375000000000001e-05 2023-07-02T14:06:43.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:43.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.148534+0000", 2023-07-02T14:06:43.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9410000000000004e-06 2023-07-02T14:06:43.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.148540+0000", 2023-07-02T14:06:43.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0041588290000000002 2023-07-02T14:06:43.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.974 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:43.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.152699+0000", 2023-07-02T14:06:43.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3327000000000001e-05 2023-07-02T14:06:43.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.975 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:43.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.152733+0000", 2023-07-02T14:06:43.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093613399999999992 2023-07-02T14:06:43.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:43.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.162094+0000", 2023-07-02T14:06:43.976 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1719e-05 2023-07-02T14:06:43.976 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:43.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.162116+0000", 2023-07-02T14:06:43.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013758195000000001 2023-07-02T14:06:43.977 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.977 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:43.977 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.977 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22534 2.5 e197/170 2:b58a101b:::rbd_data.125de1449605.00000000000000c2:head v 197'2788, mlcod=197'2783)", 2023-07-02T14:06:43.978 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.173613+0000", 2023-07-02T14:06:43.978 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.6744200760000005, 2023-07-02T14:06:43.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023211177999999999, 2023-07-02T14:06:43.978 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:43.978 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.978 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:43.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.978 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:43.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.173613+0000", 2023-07-02T14:06:43.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:43.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:43.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.173613+0000", 2023-07-02T14:06:43.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:43.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:43.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.173612+0000", 2023-07-02T14:06:43.980 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055730149999999997 2023-07-02T14:06:43.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:43.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179185+0000", 2023-07-02T14:06:43.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1826e-05 2023-07-02T14:06:43.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:43.981 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179197+0000", 2023-07-02T14:06:43.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9130000000000002e-06 2023-07-02T14:06:43.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179202+0000", 2023-07-02T14:06:43.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1668999999999997e-05 2023-07-02T14:06:43.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:43.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179244+0000", 2023-07-02T14:06:43.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6171999999999999e-05 2023-07-02T14:06:43.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.983 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:43.983 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179270+0000", 2023-07-02T14:06:43.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017512570000000002 2023-07-02T14:06:43.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:43.984 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.196782+0000", 2023-07-02T14:06:43.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1576999999999999e-05 2023-07-02T14:06:43.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:43.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.196824+0000", 2023-07-02T14:06:43.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023211177999999999 2023-07-02T14:06:43.985 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.985 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:43.985 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.986 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22539 2.5 e197/170 2:a6e4b3cf:::rbd_data.125de1449605.00000000000000c4:head v 197'2789, mlcod=197'2786)", 2023-07-02T14:06:43.986 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.179446+0000", 2023-07-02T14:06:43.986 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.6685862720000006, 2023-07-02T14:06:43.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017409688999999999, 2023-07-02T14:06:43.986 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:43.986 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.986 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:43.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:43.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179446+0000", 2023-07-02T14:06:43.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:43.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:43.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179446+0000", 2023-07-02T14:06:43.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:43.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:43.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179446+0000", 2023-07-02T14:06:43.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082263240000000001 2023-07-02T14:06:43.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:43.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187672+0000", 2023-07-02T14:06:43.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7371e-05 2023-07-02T14:06:43.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:43.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187689+0000", 2023-07-02T14:06:43.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0820000000000005e-06 2023-07-02T14:06:43.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187698+0000", 2023-07-02T14:06:43.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0510999999999998e-05 2023-07-02T14:06:43.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:43.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187749+0000", 2023-07-02T14:06:43.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4805000000000001e-05 2023-07-02T14:06:43.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.991 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:43.991 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187774+0000", 2023-07-02T14:06:43.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.009056049 2023-07-02T14:06:43.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:43.992 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.196830+0000", 2023-07-02T14:06:43.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6347e-05 2023-07-02T14:06:43.992 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:43.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.196856+0000", 2023-07-02T14:06:43.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017409688999999999 2023-07-02T14:06:43.993 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.993 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:43.993 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:43.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.993 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22591 2.5 e197/170 2:bf746601:::rbd_data.125de1449605.00000000000000d6:head v 197'2799, mlcod=197'2795)", 2023-07-02T14:06:43.994 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.286442+0000", 2023-07-02T14:06:43.994 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.5615905039999998, 2023-07-02T14:06:43.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016215869000000001, 2023-07-02T14:06:43.994 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:43.994 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:43.994 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:43.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:43.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.286442+0000", 2023-07-02T14:06:43.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:43.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:43.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.286442+0000", 2023-07-02T14:06:43.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:43.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:43.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.286441+0000", 2023-07-02T14:06:43.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00513759 2023-07-02T14:06:43.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:43.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291579+0000", 2023-07-02T14:06:43.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5048000000000001e-05 2023-07-02T14:06:43.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:43.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291604+0000", 2023-07-02T14:06:43.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1429e-05 2023-07-02T14:06:43.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:43.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291615+0000", 2023-07-02T14:06:43.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6360999999999994e-05 2023-07-02T14:06:43.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:43.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291712+0000", 2023-07-02T14:06:43.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9581999999999999e-05 2023-07-02T14:06:43.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:43.999 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:43.999 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291741+0000", 2023-07-02T14:06:43.999 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010882032999999999 2023-07-02T14:06:43.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:43.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:44.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.302623+0000", 2023-07-02T14:06:44.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4688999999999999e-05 2023-07-02T14:06:44.000 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.302658+0000", 2023-07-02T14:06:44.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016215869000000001 2023-07-02T14:06:44.001 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.001 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.001 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.001 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22597 2.5 e197/170 2:a8e921ad:::rbd_data.125de1449605.00000000000000d7:head v 197'2802, mlcod=197'2799)", 2023-07-02T14:06:44.001 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.307993+0000", 2023-07-02T14:06:44.002 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.5400392919999994, 2023-07-02T14:06:44.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014767641, 2023-07-02T14:06:44.002 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.002 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:44.002 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.307993+0000", 2023-07-02T14:06:44.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.307993+0000", 2023-07-02T14:06:44.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:44.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.307993+0000", 2023-07-02T14:06:44.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004799523 2023-07-02T14:06:44.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.312792+0000", 2023-07-02T14:06:44.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0421e-05 2023-07-02T14:06:44.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.312813+0000", 2023-07-02T14:06:44.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7560000000000005e-06 2023-07-02T14:06:44.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.312821+0000", 2023-07-02T14:06:44.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4166999999999998e-05 2023-07-02T14:06:44.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.312876+0000", 2023-07-02T14:06:44.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7472999999999999e-05 2023-07-02T14:06:44.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:44.007 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.312903+0000", 2023-07-02T14:06:44.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0098211700000000006 2023-07-02T14:06:44.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.007 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:44.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.322724+0000", 2023-07-02T14:06:44.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6928999999999998e-05 2023-07-02T14:06:44.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.322761+0000", 2023-07-02T14:06:44.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014767641 2023-07-02T14:06:44.008 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.009 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.009 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.009 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.009 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.009 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:44.010 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 2, 4] out_osds: [5] dead_osds: [] live_osds: [1, 2, 4, 5, 3, 0] 2023-07-02T14:06:44.010 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:06:44.010 INFO:tasks.thrashosds.thrasher:primary_affinity 2023-07-02T14:06:44.010 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.551824 2023-07-02T14:06:44.011 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 0.5518238136214928 2023-07-02T14:06:44.012 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:44.010+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:44.013 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:06:44.111 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:44.110+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:44.149 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap58 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap59 b3fbb028-e267-4d25-a750-93ba18a2316a@snap59.diff 2023-07-02T14:06:44.188 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:44.189 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:44.189 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:44.189 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:44.200 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:06:44.211 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:44.211+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:44.227 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete...2023-07-02T14:06:44.227+0000 7f1de36cd700 0 -- 172.21.15.163:0/2287446329 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x55fa9e121010 msgr2=0x55fa9e123440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:44.230 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:44.229+0000 7f1de36cd700 0 -- 172.21.15.163:0/2287446329 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f1dc8036bb0 msgr2=0x7f1dc8038fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:44.313 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:44.313+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:44.345 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:44.345 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:44.346 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:06:44.346 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:06:44.346 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:44.354 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:06:44.364 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 50% complete... Exporting image: 100% complete...done. 2023-07-02T14:06:44.368 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:06:44.368 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=60 2023-07-02T14:06:44.368 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap60 2023-07-02T14:06:44.437 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:44.437+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:44.515 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:44.514+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:44.520 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:44.520 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:06:44.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:06:44.521 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:44.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.521 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:06:44.521 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:44.521 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 215.06638971199999, 2023-07-02T14:06:44.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:06:44.522 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.522 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:44.522 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:44.522 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:44.522 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:44.523 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:06:44.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.523 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:44.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:44.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:44.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:06:44.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:06:44.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:06:44.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:06:44.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:06:44.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:06:44.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:06:44.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:06:44.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:06:44.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:06:44.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:44.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:06:44.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:06:44.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:06:44.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:06:44.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:06:44.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:06:44.530 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.530 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.531 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.531 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:06:44.531 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:44.532 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 214.195696704, 2023-07-02T14:06:44.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:06:44.532 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.532 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:44.532 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:44.532 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:06:44.533 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:06:44.533 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:06:44.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.533 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:44.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:44.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:44.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:06:44.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:06:44.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:06:44.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:06:44.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:06:44.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:06:44.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:06:44.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:06:44.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:06:44.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:06:44.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:44.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:06:44.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:06:44.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:06:44.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:06:44.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:44.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:06:44.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:06:44.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:06:44.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:06:44.540 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.540 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.540 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.541 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:44.541 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:44.541 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 214.19557198300001, 2023-07-02T14:06:44.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:06:44.541 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.541 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:44.541 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:44.542 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:44.542 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:44.542 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:06:44.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.542 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:44.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:44.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:44.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:06:44.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:06:44.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:06:44.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:06:44.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:06:44.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:06:44.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:06:44.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:06:44.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:06:44.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:06:44.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:44.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:06:44.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:06:44.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:06:44.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:06:44.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:06:44.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:06:44.549 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.549 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.549 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.549 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:44.549 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:44.550 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 214.18872209899999, 2023-07-02T14:06:44.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:06:44.550 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.550 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:44.550 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:44.550 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:44.550 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:44.550 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:06:44.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.551 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:44.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:44.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:06:44.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.552 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:06:44.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:06:44.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:06:44.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:06:44.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.553 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:06:44.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:06:44.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:06:44.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:06:44.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:06:44.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:06:44.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:44.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:06:44.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:06:44.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:06:44.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:06:44.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:06:44.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:06:44.557 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.557 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.557 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.558 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:44.558 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:44.558 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 214.18865917400001, 2023-07-02T14:06:44.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:06:44.558 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.558 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:44.558 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:44.559 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:44.559 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:44.559 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:06:44.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.559 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:44.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:44.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:44.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:06:44.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:06:44.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:06:44.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:06:44.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:06:44.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:06:44.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:06:44.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:06:44.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:06:44.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:06:44.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:44.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:06:44.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:06:44.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:06:44.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:06:44.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:06:44.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:06:44.566 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.566 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.566 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.566 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:44.566 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:44.567 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 214.18835875299999, 2023-07-02T14:06:44.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:06:44.567 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.567 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:44.567 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:44.567 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:44.567 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:44.567 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:06:44.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.568 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:44.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:44.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:44.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:06:44.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:06:44.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:06:44.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:06:44.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:06:44.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:06:44.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:06:44.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:06:44.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:06:44.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:06:44.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:44.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:06:44.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:06:44.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:06:44.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:06:44.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:44.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:06:44.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:06:44.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:06:44.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:06:44.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:06:44.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:44.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:06:44.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:06:44.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:44.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:06:44.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:06:44.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:44.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:06:44.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:06:44.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:44.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:06:44.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:06:44.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:06:44.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:06:44.578 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.578 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.578 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.579 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:06:44.579 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:44.579 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 214.18783425300001, 2023-07-02T14:06:44.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:06:44.579 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.579 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:44.579 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:44.580 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:44.580 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:44.580 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:06:44.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.580 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:44.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:44.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:44.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:44.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:06:44.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:06:44.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:06:44.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:06:44.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:06:44.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:06:44.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:06:44.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:06:44.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:06:44.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:44.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:06:44.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:06:44.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:06:44.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:06:44.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:44.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:06:44.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:06:44.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:06:44.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:06:44.587 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.587 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.587 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.588 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:06:44.588 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:44.588 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 187.06381512799999, 2023-07-02T14:06:44.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:06:44.588 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.589 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:44.589 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:44.589 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:44.589 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:44.589 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:06:44.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.589 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:44.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:44.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:44.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:06:44.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:06:44.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:06:44.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:06:44.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:06:44.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:06:44.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:06:44.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:06:44.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:06:44.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:06:44.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:44.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:06:44.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:06:44.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:06:44.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:06:44.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:06:44.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:06:44.596 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.596 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.596 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.597 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:06:44.597 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:44.597 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 186.73833242399999, 2023-07-02T14:06:44.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:06:44.597 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.597 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:44.597 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:44.597 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:44.598 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:44.598 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:06:44.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.598 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:44.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:44.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:44.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:06:44.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:06:44.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:06:44.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:06:44.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:06:44.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:06:44.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:06:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:06:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:06:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:06:44.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:06:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:06:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:06:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:06:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:06:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:06:44.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:06:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:06:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:44.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:06:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:06:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:06:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:06:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:06:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:06:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:06:44.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:06:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:17 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 0~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:44.608 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 93.320056641999997, 2023-07-02T14:06:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997, 2023-07-02T14:06:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:44.609 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17 2023-07-02T14:06:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:06:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198431+0000", 2023-07-02T14:06:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2815e-05 2023-07-02T14:06:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198443+0000", 2023-07-02T14:06:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2569999999999999e-05 2023-07-02T14:06:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198466+0000", 2023-07-02T14:06:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0223e-05 2023-07-02T14:06:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198476+0000", 2023-07-02T14:06:44.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1792999999999997e-05 2023-07-02T14:06:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:06:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4249999999999998e-06 2023-07-02T14:06:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198536+0000", 2023-07-02T14:06:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91888663500000001 2023-07-02T14:06:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117423+0000", 2023-07-02T14:06:44.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223594 2023-07-02T14:06:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117647+0000", 2023-07-02T14:06:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033832900000000001 2023-07-02T14:06:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117985+0000", 2023-07-02T14:06:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997 2023-07-02T14:06:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.617 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:18 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 65536~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 93.319999430999999, 2023-07-02T14:06:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998, 2023-07-02T14:06:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:44.618 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18 2023-07-02T14:06:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:44.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7600000000000002e-06 2023-07-02T14:06:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198494+0000", 2023-07-02T14:06:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7840000000000004e-06 2023-07-02T14:06:44.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198502+0000", 2023-07-02T14:06:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6459999999999998e-06 2023-07-02T14:06:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198508+0000", 2023-07-02T14:06:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7395999999999997e-05 2023-07-02T14:06:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198565+0000", 2023-07-02T14:06:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6800000000000004e-06 2023-07-02T14:06:44.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198572+0000", 2023-07-02T14:06:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91965896800000002 2023-07-02T14:06:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118231+0000", 2023-07-02T14:06:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5212999999999998e-05 2023-07-02T14:06:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118326+0000", 2023-07-02T14:06:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000238538 2023-07-02T14:06:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118565+0000", 2023-07-02T14:06:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998 2023-07-02T14:06:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:19 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 131072~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 93.319975206999999, 2023-07-02T14:06:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002, 2023-07-02T14:06:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:44.627 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 19 2023-07-02T14:06:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1990000000000002e-06 2023-07-02T14:06:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198517+0000", 2023-07-02T14:06:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8789999999999997e-06 2023-07-02T14:06:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198524+0000", 2023-07-02T14:06:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:06:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:06:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5545999999999999e-05 2023-07-02T14:06:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:44.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2540000000000003e-06 2023-07-02T14:06:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198598+0000", 2023-07-02T14:06:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.919983263 2023-07-02T14:06:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118581+0000", 2023-07-02T14:06:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6878999999999994e-05 2023-07-02T14:06:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118648+0000", 2023-07-02T14:06:44.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149732 2023-07-02T14:06:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118798+0000", 2023-07-02T14:06:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002 2023-07-02T14:06:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.635 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:20 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 262144~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 93.319956242999993, 2023-07-02T14:06:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998, 2023-07-02T14:06:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.636 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20 2023-07-02T14:06:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:44.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.732e-06 2023-07-02T14:06:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198537+0000", 2023-07-02T14:06:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5259999999999999e-06 2023-07-02T14:06:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198543+0000", 2023-07-02T14:06:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.121e-06 2023-07-02T14:06:44.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198547+0000", 2023-07-02T14:06:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9591000000000005e-05 2023-07-02T14:06:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:06:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3120000000000004e-06 2023-07-02T14:06:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198621+0000", 2023-07-02T14:06:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92020789400000003 2023-07-02T14:06:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118829+0000", 2023-07-02T14:06:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022393386000000001 2023-07-02T14:06:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:44.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141223+0000", 2023-07-02T14:06:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000329866 2023-07-02T14:06:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141552+0000", 2023-07-02T14:06:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998 2023-07-02T14:06:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.644 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:21 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 196608~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 93.319928607999998, 2023-07-02T14:06:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995, 2023-07-02T14:06:44.645 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21 2023-07-02T14:06:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0169999999999999e-05 2023-07-02T14:06:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198571+0000", 2023-07-02T14:06:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3479999999999998e-06 2023-07-02T14:06:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198578+0000", 2023-07-02T14:06:44.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8439999999999997e-06 2023-07-02T14:06:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198583+0000", 2023-07-02T14:06:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8294999999999997e-05 2023-07-02T14:06:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198641+0000", 2023-07-02T14:06:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8120000000000001e-06 2023-07-02T14:06:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:06:44.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94293828499999999 2023-07-02T14:06:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141583+0000", 2023-07-02T14:06:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014607300000000001 2023-07-02T14:06:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141729+0000", 2023-07-02T14:06:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00044906600000000003 2023-07-02T14:06:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142178+0000", 2023-07-02T14:06:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995 2023-07-02T14:06:44.653 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:22 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 327680~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 93.319895177999996, 2023-07-02T14:06:44.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004, 2023-07-02T14:06:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 22 2023-07-02T14:06:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.655 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4850000000000001e-06 2023-07-02T14:06:44.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198597+0000", 2023-07-02T14:06:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:06:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198605+0000", 2023-07-02T14:06:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4839999999999998e-06 2023-07-02T14:06:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198609+0000", 2023-07-02T14:06:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3100999999999999e-05 2023-07-02T14:06:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198662+0000", 2023-07-02T14:06:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9099999999999998e-06 2023-07-02T14:06:44.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198666+0000", 2023-07-02T14:06:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.943539503 2023-07-02T14:06:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142206+0000", 2023-07-02T14:06:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012255999999999999 2023-07-02T14:06:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142328+0000", 2023-07-02T14:06:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208956 2023-07-02T14:06:44.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142537+0000", 2023-07-02T14:06:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004 2023-07-02T14:06:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:23 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 393216~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 93.319875198000005, 2023-07-02T14:06:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348, 2023-07-02T14:06:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:44.664 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23 2023-07-02T14:06:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8439999999999998e-06 2023-07-02T14:06:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:06:44.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0440000000000001e-06 2023-07-02T14:06:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198623+0000", 2023-07-02T14:06:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4480000000000002e-06 2023-07-02T14:06:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198628+0000", 2023-07-02T14:06:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5843e-05 2023-07-02T14:06:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:44.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.89e-06 2023-07-02T14:06:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198688+0000", 2023-07-02T14:06:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94389171999999999 2023-07-02T14:06:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142580+0000", 2023-07-02T14:06:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3892000000000004e-05 2023-07-02T14:06:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142644+0000", 2023-07-02T14:06:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017608000000000001 2023-07-02T14:06:44.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142820+0000", 2023-07-02T14:06:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348 2023-07-02T14:06:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:24 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 458752~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 93.319847124000006, 2023-07-02T14:06:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004, 2023-07-02T14:06:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.673 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 24 2023-07-02T14:06:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:44.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198642+0000", 2023-07-02T14:06:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.253e-06 2023-07-02T14:06:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:06:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1289999999999998e-06 2023-07-02T14:06:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198652+0000", 2023-07-02T14:06:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4510000000000002e-06 2023-07-02T14:06:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198657+0000", 2023-07-02T14:06:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9131000000000001e-05 2023-07-02T14:06:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198706+0000", 2023-07-02T14:06:44.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8890000000000004e-06 2023-07-02T14:06:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198710+0000", 2023-07-02T14:06:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94415412200000004 2023-07-02T14:06:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142864+0000", 2023-07-02T14:06:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8047000000000006e-05 2023-07-02T14:06:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142952+0000", 2023-07-02T14:06:44.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192967 2023-07-02T14:06:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143145+0000", 2023-07-02T14:06:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004 2023-07-02T14:06:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.681 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:25 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 524288~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 93.319828329000003, 2023-07-02T14:06:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001, 2023-07-02T14:06:44.682 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25 2023-07-02T14:06:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:44.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.615e-06 2023-07-02T14:06:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198664+0000", 2023-07-02T14:06:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4759999999999997e-06 2023-07-02T14:06:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198669+0000", 2023-07-02T14:06:44.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5999999999999998e-06 2023-07-02T14:06:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198673+0000", 2023-07-02T14:06:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4256999999999999e-05 2023-07-02T14:06:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198727+0000", 2023-07-02T14:06:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840000000000001e-06 2023-07-02T14:06:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198731+0000", 2023-07-02T14:06:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94444241900000003 2023-07-02T14:06:44.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143173+0000", 2023-07-02T14:06:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6839000000000002e-05 2023-07-02T14:06:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:44.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143240+0000", 2023-07-02T14:06:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017053399999999999 2023-07-02T14:06:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143411+0000", 2023-07-02T14:06:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001 2023-07-02T14:06:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.690 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:26 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 589824~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 93.319805177999996, 2023-07-02T14:06:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997, 2023-07-02T14:06:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 26 2023-07-02T14:06:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:44.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0520000000000002e-06 2023-07-02T14:06:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198687+0000", 2023-07-02T14:06:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1880000000000002e-06 2023-07-02T14:06:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198695+0000", 2023-07-02T14:06:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9960000000000004e-06 2023-07-02T14:06:44.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198699+0000", 2023-07-02T14:06:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2330999999999998e-05 2023-07-02T14:06:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198742+0000", 2023-07-02T14:06:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7950000000000001e-06 2023-07-02T14:06:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198745+0000", 2023-07-02T14:06:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94469398199999999 2023-07-02T14:06:44.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143439+0000", 2023-07-02T14:06:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6048e-05 2023-07-02T14:06:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:44.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143525+0000", 2023-07-02T14:06:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017016700000000001 2023-07-02T14:06:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143696+0000", 2023-07-02T14:06:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997 2023-07-02T14:06:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.699 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.6579.0:11 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected e171)", 2023-07-02T14:06:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 47.360006691999999, 2023-07-02T14:06:44.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998, 2023-07-02T14:06:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.6579", 2023-07-02T14:06:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/554403467", 2023-07-02T14:06:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11 2023-07-02T14:06:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.701 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:44.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158481+0000", 2023-07-02T14:06:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:06:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158487+0000", 2023-07-02T14:06:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7630000000000006e-06 2023-07-02T14:06:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:44.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158496+0000", 2023-07-02T14:06:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5100000000000001e-06 2023-07-02T14:06:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158504+0000", 2023-07-02T14:06:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9397000000000002e-05 2023-07-02T14:06:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158553+0000", 2023-07-02T14:06:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5810000000000003e-06 2023-07-02T14:06:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158562+0000", 2023-07-02T14:06:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93893079499999998 2023-07-02T14:06:44.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:44.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097493+0000", 2023-07-02T14:06:44.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015799299999999999 2023-07-02T14:06:44.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:44.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097651+0000", 2023-07-02T14:06:44.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187746 2023-07-02T14:06:44.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:44.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:44.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:44.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097839+0000", 2023-07-02T14:06:44.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998 2023-07-02T14:06:44.708 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.709 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.709 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.709 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:44.709 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:44.709 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:44.710 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:06:44.711 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:06:44.712 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:44.711+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:44.812 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:44.812+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:44.879 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:44.879 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:44.879 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:44.879 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:44.890 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:06:44.892 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:44.910 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:44.913 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:44.912+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:44.928 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:44.946 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:44.964 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:44.983 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:44.984 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:06:44.984 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:06:44.984 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:06:44.984 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:06:44.984 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:06:44.985 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:06:45.016 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:45.017 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:45.017 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:45.016+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:45.027 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:06:45.103 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.0 primary-affinity to 0.551824 (8361642) 2023-07-02T14:06:45.115 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:45.115+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:45.168 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:06:45.174 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 1 2023-07-02T14:06:45.195 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:45.195 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:45.205 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:06:45.249 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:45.248+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:45.316 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:45.316+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:45.340 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:45.340 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:45.351 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:06:45.371 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:06:45.434 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:45.433+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:45.515 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:45.516 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:45.517 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:45.517+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:45.526 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:06:45.617 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:45.617+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:45.690 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:45.690 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:45.701 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:06:45.719 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:45.719+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:45.820 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:45.819+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:45.826 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:45.827 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:45.837 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:06:45.922 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:45.921+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:46.002 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:46.003 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:46.012 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:06:46.021 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:46.021+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:46.107 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:06:46.122 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:46.121+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:46.176 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap59 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap60 b3fbb028-e267-4d25-a750-93ba18a2316a@snap60.diff 2023-07-02T14:06:46.177 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:46.177 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:46.188 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:06:46.229 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:46.228+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:46.257 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 100% complete...done. 2023-07-02T14:06:46.260 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:06:46.260 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=61 2023-07-02T14:06:46.260 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap61 2023-07-02T14:06:46.323 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:46.323+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:46.354 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:46.355 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:46.365 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:06:46.424 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:46.423+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:46.489 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:46.490 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:46.500 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:06:46.525 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:46.525+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:46.626 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:46.626+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:46.664 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:46.664 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:46.674 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:06:46.728 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:46.727+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:46.802 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:46.803 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:46.813 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:06:46.829 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:46.828+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:46.929 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:46.929+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:46.936 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:46.937 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:46.947 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:06:46.995 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:06:47.031 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:47.031+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:47.121 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:47.122 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:47.131 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:47.131+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:47.133 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:06:47.153 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:06:47.160 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 4 2023-07-02T14:06:47.176 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:47.194 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:47.212 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:47.232 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:47.232 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:47.231+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:47.248 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:47.271 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:06:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:06:47.273 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:06:47.274 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:06:47.274 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:06:47.274 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:06:47.304 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:47.305 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:47.305 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:47.305 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:47.316 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:06:47.334 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:47.334+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:47.434 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:47.434+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:47.480 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:47.480 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:47.480 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:06:47.480 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:06:47.481 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:47.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:06:47.573 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:47.573+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:47.636 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:47.636+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:47.655 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:47.656 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:06:47.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:06:47.656 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:47.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.656 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:06:47.656 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:47.657 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 218.201977167, 2023-07-02T14:06:47.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:06:47.657 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:47.657 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:47.657 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:47.657 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:47.657 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:47.657 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:06:47.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.658 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:47.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:47.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:47.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:47.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:47.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:47.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:47.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:47.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:06:47.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:06:47.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:47.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:06:47.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:06:47.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:47.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:06:47.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:06:47.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:47.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:06:47.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:06:47.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:06:47.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:06:47.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:47.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:06:47.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:06:47.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:06:47.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:06:47.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:47.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:06:47.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:06:47.664 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.664 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.664 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.665 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:06:47.665 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:47.665 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 217.33128415900001, 2023-07-02T14:06:47.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:06:47.665 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:47.665 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:47.666 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:47.666 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:06:47.666 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:06:47.666 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:06:47.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.666 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:47.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:47.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:47.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:47.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:47.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:47.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:47.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:47.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:06:47.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:06:47.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:47.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:06:47.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:06:47.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:47.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:06:47.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:06:47.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:47.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:06:47.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:06:47.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:06:47.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:06:47.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:47.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:06:47.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:06:47.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:06:47.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:06:47.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:47.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:06:47.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:06:47.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:47.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:06:47.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:06:47.673 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.673 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.674 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.674 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:47.674 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:47.674 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 217.33115943799999, 2023-07-02T14:06:47.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:06:47.674 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:47.675 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:47.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:47.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:47.675 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:47.675 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:06:47.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.675 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:47.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:47.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:47.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:47.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:47.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:47.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:47.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:47.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:06:47.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:06:47.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:47.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:06:47.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:06:47.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:47.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:06:47.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:06:47.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:47.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:06:47.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:06:47.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:06:47.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:06:47.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:47.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:06:47.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:06:47.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:06:47.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:06:47.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:47.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:06:47.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:06:47.682 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.682 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.682 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.683 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:47.683 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:47.683 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 217.324309554, 2023-07-02T14:06:47.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:06:47.683 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:47.683 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:47.683 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:47.684 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:47.684 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:47.684 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:06:47.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.684 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:47.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:47.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:47.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:47.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:47.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:47.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:06:47.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:47.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:06:47.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:06:47.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:47.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:06:47.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:06:47.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:47.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:06:47.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:06:47.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:47.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:06:47.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:06:47.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.688 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.688 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:06:47.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:06:47.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:47.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:06:47.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:06:47.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:06:47.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:06:47.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:47.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:06:47.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:06:47.691 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.691 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.691 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.691 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:47.691 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:47.691 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 217.32424662899999, 2023-07-02T14:06:47.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:06:47.692 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:47.692 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:47.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:47.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:47.692 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:47.692 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:06:47.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.693 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:47.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:47.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:47.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:47.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:47.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:47.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:47.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:47.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:06:47.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:06:47.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:47.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:06:47.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:06:47.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:47.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:06:47.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:06:47.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.696 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:47.696 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:06:47.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:06:47.696 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:06:47.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:06:47.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:47.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:06:47.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:06:47.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:06:47.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:06:47.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:47.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:06:47.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:06:47.699 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.699 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.699 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.700 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:47.700 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:47.700 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 217.323946208, 2023-07-02T14:06:47.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:06:47.700 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:47.700 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:47.700 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:47.700 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:47.701 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:47.701 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:06:47.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.701 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:47.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:47.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:47.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:47.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:47.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:47.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:47.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:47.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:06:47.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:06:47.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:47.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:06:47.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:06:47.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:47.704 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:06:47.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:06:47.704 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.704 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:47.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:06:47.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:06:47.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:06:47.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:06:47.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:47.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:06:47.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:06:47.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:06:47.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:06:47.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:47.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:06:47.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:06:47.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:06:47.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:06:47.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:06:47.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:47.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:06:47.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:06:47.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:47.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:06:47.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:06:47.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:47.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:06:47.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:06:47.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:47.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:06:47.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:06:47.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.711 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:47.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:06:47.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:06:47.711 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.711 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.711 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.712 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.712 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:06:47.712 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:47.712 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 217.32342170800001, 2023-07-02T14:06:47.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:06:47.712 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:47.712 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:47.713 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:47.713 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:47.713 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:47.713 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:06:47.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.713 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:47.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:47.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:47.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:47.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:47.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:47.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:47.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:47.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:47.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:06:47.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:47.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:06:47.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:06:47.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:47.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:06:47.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:06:47.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:47.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:06:47.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:06:47.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:06:47.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:06:47.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:47.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:06:47.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:06:47.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:06:47.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:06:47.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.719 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:47.719 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:06:47.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:06:47.720 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:47.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:06:47.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:06:47.720 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.720 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.721 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.721 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:06:47.721 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:47.721 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 190.19940258299999, 2023-07-02T14:06:47.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:06:47.721 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:47.722 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:47.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:47.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:47.722 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:47.722 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:06:47.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.722 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:47.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:47.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:47.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:47.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:47.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:47.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:47.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:47.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:06:47.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:06:47.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:47.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:06:47.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:06:47.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:47.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:06:47.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:06:47.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:47.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:06:47.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:06:47.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:06:47.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:06:47.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.727 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:47.727 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:06:47.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:06:47.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:06:47.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:06:47.728 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:47.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:06:47.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:06:47.729 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.729 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.729 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.729 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:06:47.729 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:47.730 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 189.873919879, 2023-07-02T14:06:47.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:06:47.730 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:47.730 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:47.730 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:47.730 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:47.730 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:47.730 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:06:47.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.731 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:47.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:47.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:47.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:47.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:47.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:47.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:47.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:47.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:06:47.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:06:47.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:47.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:06:47.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:06:47.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:47.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:06:47.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:06:47.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:47.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:06:47.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:06:47.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.735 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.735 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:06:47.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:06:47.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:47.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:06:47.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:06:47.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:06:47.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:06:47.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:47.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:06:47.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:06:47.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:47.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:06:47.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:06:47.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:47.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:06:47.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:06:47.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:47.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:06:47.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:06:47.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:47.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:06:47.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:06:47.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:47.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:06:47.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:06:47.740 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.741 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.741 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.741 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:17 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 0~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:47.741 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:47.741 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 96.455644097000004, 2023-07-02T14:06:47.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997, 2023-07-02T14:06:47.742 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:47.742 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:47.742 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:47.742 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:47.742 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:47.742 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17 2023-07-02T14:06:47.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.742 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:47.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:47.743 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:47.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:47.743 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.743 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:47.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:47.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:06:47.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:47.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198431+0000", 2023-07-02T14:06:47.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2815e-05 2023-07-02T14:06:47.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:47.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198443+0000", 2023-07-02T14:06:47.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2569999999999999e-05 2023-07-02T14:06:47.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:47.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198466+0000", 2023-07-02T14:06:47.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0223e-05 2023-07-02T14:06:47.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:47.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198476+0000", 2023-07-02T14:06:47.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1792999999999997e-05 2023-07-02T14:06:47.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:06:47.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4249999999999998e-06 2023-07-02T14:06:47.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:47.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198536+0000", 2023-07-02T14:06:47.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91888663500000001 2023-07-02T14:06:47.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117423+0000", 2023-07-02T14:06:47.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223594 2023-07-02T14:06:47.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:47.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117647+0000", 2023-07-02T14:06:47.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033832900000000001 2023-07-02T14:06:47.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:47.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117985+0000", 2023-07-02T14:06:47.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997 2023-07-02T14:06:47.749 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.750 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.750 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.750 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:18 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 65536~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:47.750 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:47.750 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 96.455586886000006, 2023-07-02T14:06:47.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998, 2023-07-02T14:06:47.751 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:47.751 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:47.751 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:47.751 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:47.751 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:47.751 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18 2023-07-02T14:06:47.751 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.752 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:47.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:47.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:47.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:47.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:47.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:47.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:47.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:47.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:47.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7600000000000002e-06 2023-07-02T14:06:47.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:47.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198494+0000", 2023-07-02T14:06:47.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7840000000000004e-06 2023-07-02T14:06:47.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:47.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198502+0000", 2023-07-02T14:06:47.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6459999999999998e-06 2023-07-02T14:06:47.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:47.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198508+0000", 2023-07-02T14:06:47.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7395999999999997e-05 2023-07-02T14:06:47.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198565+0000", 2023-07-02T14:06:47.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6800000000000004e-06 2023-07-02T14:06:47.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:47.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198572+0000", 2023-07-02T14:06:47.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91965896800000002 2023-07-02T14:06:47.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118231+0000", 2023-07-02T14:06:47.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5212999999999998e-05 2023-07-02T14:06:47.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:47.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118326+0000", 2023-07-02T14:06:47.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000238538 2023-07-02T14:06:47.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.758 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:47.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118565+0000", 2023-07-02T14:06:47.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998 2023-07-02T14:06:47.759 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.759 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.759 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.759 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.759 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:19 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 131072~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:47.759 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:47.759 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 96.455562662000006, 2023-07-02T14:06:47.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002, 2023-07-02T14:06:47.760 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:47.760 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:47.760 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:47.760 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:47.760 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:47.760 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 19 2023-07-02T14:06:47.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.761 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:47.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:47.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:47.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:47.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:47.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:47.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:47.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:47.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:47.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1990000000000002e-06 2023-07-02T14:06:47.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:47.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198517+0000", 2023-07-02T14:06:47.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8789999999999997e-06 2023-07-02T14:06:47.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:47.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198524+0000", 2023-07-02T14:06:47.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:06:47.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:47.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:06:47.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5545999999999999e-05 2023-07-02T14:06:47.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:47.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2540000000000003e-06 2023-07-02T14:06:47.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:47.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198598+0000", 2023-07-02T14:06:47.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.919983263 2023-07-02T14:06:47.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.766 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.766 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118581+0000", 2023-07-02T14:06:47.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6878999999999994e-05 2023-07-02T14:06:47.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:47.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118648+0000", 2023-07-02T14:06:47.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149732 2023-07-02T14:06:47.767 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:47.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118798+0000", 2023-07-02T14:06:47.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002 2023-07-02T14:06:47.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.768 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.768 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.768 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:20 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 262144~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:47.768 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:47.768 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 96.455543698, 2023-07-02T14:06:47.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998, 2023-07-02T14:06:47.769 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:47.769 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:47.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:47.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:47.769 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:47.769 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20 2023-07-02T14:06:47.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.770 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:47.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:47.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:47.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:47.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:47.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:47.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:47.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:47.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:47.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.732e-06 2023-07-02T14:06:47.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:47.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198537+0000", 2023-07-02T14:06:47.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5259999999999999e-06 2023-07-02T14:06:47.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:47.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198543+0000", 2023-07-02T14:06:47.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.121e-06 2023-07-02T14:06:47.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:47.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198547+0000", 2023-07-02T14:06:47.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9591000000000005e-05 2023-07-02T14:06:47.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:06:47.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3120000000000004e-06 2023-07-02T14:06:47.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.774 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:47.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198621+0000", 2023-07-02T14:06:47.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92020789400000003 2023-07-02T14:06:47.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.775 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118829+0000", 2023-07-02T14:06:47.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022393386000000001 2023-07-02T14:06:47.775 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:47.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141223+0000", 2023-07-02T14:06:47.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000329866 2023-07-02T14:06:47.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:47.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141552+0000", 2023-07-02T14:06:47.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998 2023-07-02T14:06:47.777 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.777 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.777 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.777 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:21 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 196608~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:47.777 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:47.777 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 96.455516063000005, 2023-07-02T14:06:47.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995, 2023-07-02T14:06:47.778 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:47.778 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:47.778 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:47.778 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:47.778 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:47.778 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21 2023-07-02T14:06:47.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.779 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:47.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:47.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:47.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:47.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:47.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:47.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:47.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:47.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:47.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0169999999999999e-05 2023-07-02T14:06:47.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:47.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198571+0000", 2023-07-02T14:06:47.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3479999999999998e-06 2023-07-02T14:06:47.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:47.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198578+0000", 2023-07-02T14:06:47.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8439999999999997e-06 2023-07-02T14:06:47.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.782 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:47.782 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198583+0000", 2023-07-02T14:06:47.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8294999999999997e-05 2023-07-02T14:06:47.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198641+0000", 2023-07-02T14:06:47.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8120000000000001e-06 2023-07-02T14:06:47.783 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:47.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:06:47.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94293828499999999 2023-07-02T14:06:47.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141583+0000", 2023-07-02T14:06:47.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014607300000000001 2023-07-02T14:06:47.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:47.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141729+0000", 2023-07-02T14:06:47.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00044906600000000003 2023-07-02T14:06:47.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:47.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142178+0000", 2023-07-02T14:06:47.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995 2023-07-02T14:06:47.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.786 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.786 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.786 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:22 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 327680~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:47.786 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:47.787 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 96.455482633000003, 2023-07-02T14:06:47.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004, 2023-07-02T14:06:47.787 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:47.787 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:47.787 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:47.787 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:47.787 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:47.787 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 22 2023-07-02T14:06:47.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.788 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:47.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:47.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:47.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:47.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:47.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:47.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:47.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:47.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:47.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4850000000000001e-06 2023-07-02T14:06:47.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:47.790 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198597+0000", 2023-07-02T14:06:47.790 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:06:47.790 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.790 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:47.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198605+0000", 2023-07-02T14:06:47.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4839999999999998e-06 2023-07-02T14:06:47.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.791 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.791 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:47.791 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198609+0000", 2023-07-02T14:06:47.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3100999999999999e-05 2023-07-02T14:06:47.791 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198662+0000", 2023-07-02T14:06:47.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9099999999999998e-06 2023-07-02T14:06:47.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:47.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198666+0000", 2023-07-02T14:06:47.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.943539503 2023-07-02T14:06:47.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142206+0000", 2023-07-02T14:06:47.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012255999999999999 2023-07-02T14:06:47.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:47.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142328+0000", 2023-07-02T14:06:47.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208956 2023-07-02T14:06:47.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:47.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142537+0000", 2023-07-02T14:06:47.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004 2023-07-02T14:06:47.795 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.795 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.795 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.795 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:23 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 393216~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:47.795 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:47.795 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 96.455462652999998, 2023-07-02T14:06:47.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348, 2023-07-02T14:06:47.796 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:47.796 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:47.796 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:47.796 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:47.796 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:47.796 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23 2023-07-02T14:06:47.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.797 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:47.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:47.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:47.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:47.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:47.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:47.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:47.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:47.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:47.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8439999999999998e-06 2023-07-02T14:06:47.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:47.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:06:47.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0440000000000001e-06 2023-07-02T14:06:47.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:47.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198623+0000", 2023-07-02T14:06:47.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4480000000000002e-06 2023-07-02T14:06:47.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:47.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198628+0000", 2023-07-02T14:06:47.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5843e-05 2023-07-02T14:06:47.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:47.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.89e-06 2023-07-02T14:06:47.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:47.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198688+0000", 2023-07-02T14:06:47.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94389171999999999 2023-07-02T14:06:47.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142580+0000", 2023-07-02T14:06:47.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3892000000000004e-05 2023-07-02T14:06:47.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:47.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142644+0000", 2023-07-02T14:06:47.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017608000000000001 2023-07-02T14:06:47.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:47.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142820+0000", 2023-07-02T14:06:47.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348 2023-07-02T14:06:47.804 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.804 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.804 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.804 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:24 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 458752~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:47.804 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:47.805 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 96.455434578999999, 2023-07-02T14:06:47.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004, 2023-07-02T14:06:47.805 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:47.805 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:47.805 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:47.805 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:47.805 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:47.805 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 24 2023-07-02T14:06:47.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.806 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:47.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:47.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:47.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:47.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:47.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:47.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:47.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:47.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198642+0000", 2023-07-02T14:06:47.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.253e-06 2023-07-02T14:06:47.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:47.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:06:47.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1289999999999998e-06 2023-07-02T14:06:47.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:47.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198652+0000", 2023-07-02T14:06:47.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4510000000000002e-06 2023-07-02T14:06:47.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:47.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198657+0000", 2023-07-02T14:06:47.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9131000000000001e-05 2023-07-02T14:06:47.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198706+0000", 2023-07-02T14:06:47.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8890000000000004e-06 2023-07-02T14:06:47.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:47.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198710+0000", 2023-07-02T14:06:47.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94415412200000004 2023-07-02T14:06:47.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142864+0000", 2023-07-02T14:06:47.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8047000000000006e-05 2023-07-02T14:06:47.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:47.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142952+0000", 2023-07-02T14:06:47.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192967 2023-07-02T14:06:47.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:47.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143145+0000", 2023-07-02T14:06:47.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004 2023-07-02T14:06:47.813 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.813 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.813 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.813 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:25 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 524288~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:47.813 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:47.813 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 96.455415783999996, 2023-07-02T14:06:47.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001, 2023-07-02T14:06:47.814 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:47.814 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:47.814 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:47.814 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:47.814 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:47.814 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25 2023-07-02T14:06:47.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.815 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:47.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:47.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:47.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:47.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:47.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:47.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:47.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:47.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:47.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.615e-06 2023-07-02T14:06:47.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:47.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198664+0000", 2023-07-02T14:06:47.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4759999999999997e-06 2023-07-02T14:06:47.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:47.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198669+0000", 2023-07-02T14:06:47.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5999999999999998e-06 2023-07-02T14:06:47.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:47.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198673+0000", 2023-07-02T14:06:47.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4256999999999999e-05 2023-07-02T14:06:47.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198727+0000", 2023-07-02T14:06:47.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840000000000001e-06 2023-07-02T14:06:47.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:47.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198731+0000", 2023-07-02T14:06:47.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94444241900000003 2023-07-02T14:06:47.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143173+0000", 2023-07-02T14:06:47.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6839000000000002e-05 2023-07-02T14:06:47.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:47.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143240+0000", 2023-07-02T14:06:47.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017053399999999999 2023-07-02T14:06:47.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:47.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143411+0000", 2023-07-02T14:06:47.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001 2023-07-02T14:06:47.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.822 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.822 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:26 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 589824~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:47.822 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:47.822 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 96.455392633000002, 2023-07-02T14:06:47.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997, 2023-07-02T14:06:47.823 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:47.823 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:47.823 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:47.823 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:47.823 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:47.823 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 26 2023-07-02T14:06:47.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.824 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:47.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.824 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:47.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:47.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:47.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:47.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:47.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:47.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:47.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:47.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0520000000000002e-06 2023-07-02T14:06:47.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:47.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198687+0000", 2023-07-02T14:06:47.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1880000000000002e-06 2023-07-02T14:06:47.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:47.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198695+0000", 2023-07-02T14:06:47.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9960000000000004e-06 2023-07-02T14:06:47.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:47.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198699+0000", 2023-07-02T14:06:47.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2330999999999998e-05 2023-07-02T14:06:47.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198742+0000", 2023-07-02T14:06:47.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7950000000000001e-06 2023-07-02T14:06:47.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:47.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198745+0000", 2023-07-02T14:06:47.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94469398199999999 2023-07-02T14:06:47.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143439+0000", 2023-07-02T14:06:47.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6048e-05 2023-07-02T14:06:47.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:47.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143525+0000", 2023-07-02T14:06:47.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017016700000000001 2023-07-02T14:06:47.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:47.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143696+0000", 2023-07-02T14:06:47.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997 2023-07-02T14:06:47.831 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.831 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.831 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.831 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.6579.0:11 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected e171)", 2023-07-02T14:06:47.831 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:47.832 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.495594146999998, 2023-07-02T14:06:47.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998, 2023-07-02T14:06:47.832 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:47.832 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:47.832 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:47.832 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.6579", 2023-07-02T14:06:47.832 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/554403467", 2023-07-02T14:06:47.832 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11 2023-07-02T14:06:47.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.833 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:47.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:47.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:47.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:47.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:47.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:47.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:47.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:47.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158481+0000", 2023-07-02T14:06:47.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:06:47.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:47.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158487+0000", 2023-07-02T14:06:47.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7630000000000006e-06 2023-07-02T14:06:47.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:47.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158496+0000", 2023-07-02T14:06:47.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5100000000000001e-06 2023-07-02T14:06:47.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:47.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158504+0000", 2023-07-02T14:06:47.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9397000000000002e-05 2023-07-02T14:06:47.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158553+0000", 2023-07-02T14:06:47.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5810000000000003e-06 2023-07-02T14:06:47.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:47.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158562+0000", 2023-07-02T14:06:47.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93893079499999998 2023-07-02T14:06:47.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:47.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097493+0000", 2023-07-02T14:06:47.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015799299999999999 2023-07-02T14:06:47.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:47.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097651+0000", 2023-07-02T14:06:47.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187746 2023-07-02T14:06:47.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:47.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:47.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:47.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097839+0000", 2023-07-02T14:06:47.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998 2023-07-02T14:06:47.840 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.840 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.840 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.840 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:47.840 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:47.840 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:47.842 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:06:47.842 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:47.841+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:47.942 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:47.942+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:48.008 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:48.008 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:48.008 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:48.008 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:48.018 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:06:48.043 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:48.042+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:48.120 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:06:48.143 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:48.144 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:48.144 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:06:48.144 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:06:48.144 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:48.144 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:48.143+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:48.154 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:06:48.244 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:48.243+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:48.341 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:48.341 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:06:48.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:06:48.342 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:48.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.342 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:06:48.342 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:06:48.342 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 227.23199838900001, 2023-07-02T14:06:48.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:06:48.343 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:48.343 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:48.343 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:48.343 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:48.343 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:48.343 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:06:48.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.344 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:48.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:48.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:06:48.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:48.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:48.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:06:48.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:48.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:48.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:06:48.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:06:48.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:48.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:06:48.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:06:48.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:48.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:06:48.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:06:48.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:48.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:06:48.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:06:48.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:06:48.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:06:48.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:48.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:06:48.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:06:48.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:06:48.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:06:48.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:48.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:06:48.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:06:48.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:48.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:06:48.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:06:48.351 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.352 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.352 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.352 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:06:48.352 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:06:48.352 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 227.23193765100001, 2023-07-02T14:06:48.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:06:48.353 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:48.353 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:48.353 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:48.353 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:48.353 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:48.353 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:06:48.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.354 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:48.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:48.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:06:48.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:48.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:48.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:06:48.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:48.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:48.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:06:48.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:06:48.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:48.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:06:48.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:06:48.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:48.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:06:48.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:06:48.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:48.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:06:48.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:06:48.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:06:48.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:06:48.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:48.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:06:48.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:06:48.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:06:48.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:06:48.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:48.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:06:48.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:06:48.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:48.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:06:48.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:06:48.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:48.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:06:48.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:06:48.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:48.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:06:48.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:06:48.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:48.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:06:48.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:06:48.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:48.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:06:48.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:06:48.365 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.365 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.365 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.365 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:06:48.365 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:06:48.366 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 217.00148332099999, 2023-07-02T14:06:48.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:06:48.366 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:48.366 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:48.366 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:48.366 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:48.366 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:48.367 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:06:48.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.367 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:48.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:48.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:06:48.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:48.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:48.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:06:48.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:48.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:48.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:06:48.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:06:48.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:48.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:06:48.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:06:48.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:48.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:06:48.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:06:48.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:48.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:06:48.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:06:48.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:06:48.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:06:48.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:48.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:06:48.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:06:48.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:48.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:06:48.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:06:48.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:48.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:06:48.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:06:48.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:48.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:06:48.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:06:48.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:48.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:06:48.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:06:48.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:48.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:06:48.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:06:48.376 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.377 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.377 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.377 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9620 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 589824~65536] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:06:48.377 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:06:48.377 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 143.31746872299999, 2023-07-02T14:06:48.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001, 2023-07-02T14:06:48.378 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:48.378 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:48.378 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:48.378 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:48.379 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:48.379 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9620 2023-07-02T14:06:48.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.379 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:48.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:48.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:06:48.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:48.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:48.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:06:48.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:48.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:48.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022203+0000", 2023-07-02T14:06:48.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0346000000000001e-05 2023-07-02T14:06:48.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:48.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022263+0000", 2023-07-02T14:06:48.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4657999999999999e-05 2023-07-02T14:06:48.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:48.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022278+0000", 2023-07-02T14:06:48.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:06:48.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:48.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022285+0000", 2023-07-02T14:06:48.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0096000000000003e-05 2023-07-02T14:06:48.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022335+0000", 2023-07-02T14:06:48.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2334000000000001e-05 2023-07-02T14:06:48.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:48.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022347+0000", 2023-07-02T14:06:48.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095653701999999993 2023-07-02T14:06:48.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118001+0000", 2023-07-02T14:06:48.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000185797 2023-07-02T14:06:48.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:48.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118187+0000", 2023-07-02T14:06:48.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026397900000000001 2023-07-02T14:06:48.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:48.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118451+0000", 2023-07-02T14:06:48.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051832800000000002 2023-07-02T14:06:48.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:48.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.123634+0000", 2023-07-02T14:06:48.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001227332 2023-07-02T14:06:48.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:48.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124861+0000", 2023-07-02T14:06:48.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9493999999999998e-05 2023-07-02T14:06:48.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:48.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124901+0000", 2023-07-02T14:06:48.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0462000000000001e-05 2023-07-02T14:06:48.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:48.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124931+0000", 2023-07-02T14:06:48.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001 2023-07-02T14:06:48.389 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.390 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.390 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.390 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9621 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 655360~1966080] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:06:48.390 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:06:48.391 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 143.31728134100001, 2023-07-02T14:06:48.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999, 2023-07-02T14:06:48.391 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:48.391 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:48.391 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:48.391 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:48.391 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:48.391 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9621 2023-07-02T14:06:48.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.392 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:48.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:48.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:06:48.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:48.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:48.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:06:48.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:48.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:48.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022391+0000", 2023-07-02T14:06:48.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032053939999999999 2023-07-02T14:06:48.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:48.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025597+0000", 2023-07-02T14:06:48.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4307999999999999e-05 2023-07-02T14:06:48.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:48.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025611+0000", 2023-07-02T14:06:48.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.032e-06 2023-07-02T14:06:48.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.395 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:48.395 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025617+0000", 2023-07-02T14:06:48.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8977999999999998e-05 2023-07-02T14:06:48.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025656+0000", 2023-07-02T14:06:48.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:06:48.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:48.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025663+0000", 2023-07-02T14:06:48.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095220739999999998 2023-07-02T14:06:48.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120884+0000", 2023-07-02T14:06:48.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2525000000000001e-05 2023-07-02T14:06:48.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:48.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120956+0000", 2023-07-02T14:06:48.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016232400000000001 2023-07-02T14:06:48.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:48.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.121119+0000", 2023-07-02T14:06:48.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044007380000000004 2023-07-02T14:06:48.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:48.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.125519+0000", 2023-07-02T14:06:48.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000797209 2023-07-02T14:06:48.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:48.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126317+0000", 2023-07-02T14:06:48.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1228000000000004e-05 2023-07-02T14:06:48.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:48.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126378+0000", 2023-07-02T14:06:48.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978000000000003e-05 2023-07-02T14:06:48.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:48.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126430+0000", 2023-07-02T14:06:48.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999 2023-07-02T14:06:48.402 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.402 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.402 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.403 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:189 2.3 2:c9621172:::rbd_data.125de1449605.00000000000000c0:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:06:48.403 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:06:48.403 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 135.16699404900001, 2023-07-02T14:06:48.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001, 2023-07-02T14:06:48.403 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:48.403 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:48.404 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:48.404 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:06:48.404 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:06:48.404 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 189 2023-07-02T14:06:48.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.404 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:48.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.404 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:48.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:06:48.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:48.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:48.405 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:06:48.405 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:48.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:48.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172679+0000", 2023-07-02T14:06:48.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9220000000000001e-06 2023-07-02T14:06:48.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:48.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172682+0000", 2023-07-02T14:06:48.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0159999999999997e-06 2023-07-02T14:06:48.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:48.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172688+0000", 2023-07-02T14:06:48.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0010000000000003e-06 2023-07-02T14:06:48.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:48.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.172692+0000", 2023-07-02T14:06:48.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026444333 2023-07-02T14:06:48.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199136+0000", 2023-07-02T14:06:48.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2230000000000004e-05 2023-07-02T14:06:48.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:48.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.199218+0000", 2023-07-02T14:06:48.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012043595000000001 2023-07-02T14:06:48.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:48.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.211262+0000", 2023-07-02T14:06:48.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.038582668000000001 2023-07-02T14:06:48.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.411 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.411 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.411 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:281 2.1 2:901d5a95:::rbd_data.125de1449605.0000000000000118:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:06:48.411 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:06:48.411 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 134.78420523, 2023-07-02T14:06:48.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002, 2023-07-02T14:06:48.412 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:48.412 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:48.412 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:48.412 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:06:48.412 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:06:48.412 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:06:48.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.413 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:48.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.413 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:48.413 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:06:48.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:48.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:48.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:06:48.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:48.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.414 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.414 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:48.414 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:06:48.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0469999999999998e-06 2023-07-02T14:06:48.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:48.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555471+0000", 2023-07-02T14:06:48.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8860000000000001e-06 2023-07-02T14:06:48.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:48.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555476+0000", 2023-07-02T14:06:48.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3160000000000002e-06 2023-07-02T14:06:48.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:48.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555483+0000", 2023-07-02T14:06:48.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989525000000001 2023-07-02T14:06:48.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591472+0000", 2023-07-02T14:06:48.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3708999999999996e-05 2023-07-02T14:06:48.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:48.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591536+0000", 2023-07-02T14:06:48.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055109720000000003 2023-07-02T14:06:48.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:48.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.597047+0000", 2023-07-02T14:06:48.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002 2023-07-02T14:06:48.419 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.419 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.419 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.419 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12985 2.7 2:eeddf24b:::rbd_data.125de1449605.000000000000007e:head [write 65536~4128768] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:06:48.420 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:06:48.420 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 115.556678284, 2023-07-02T14:06:48.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999, 2023-07-02T14:06:48.420 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:48.420 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:48.421 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:48.421 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:48.421 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:48.421 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12985 2023-07-02T14:06:48.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.421 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:48.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:48.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:06:48.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:48.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.422 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:48.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:06:48.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:48.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:48.423 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782994+0000", 2023-07-02T14:06:48.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019254741999999998 2023-07-02T14:06:48.423 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.423 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.423 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:48.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802248+0000", 2023-07-02T14:06:48.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0451e-05 2023-07-02T14:06:48.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:48.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802279+0000", 2023-07-02T14:06:48.424 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3318e-05 2023-07-02T14:06:48.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:48.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802292+0000", 2023-07-02T14:06:48.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109403 2023-07-02T14:06:48.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802402+0000", 2023-07-02T14:06:48.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012520599999999999 2023-07-02T14:06:48.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:48.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802527+0000", 2023-07-02T14:06:48.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187768 2023-07-02T14:06:48.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:48.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802715+0000", 2023-07-02T14:06:48.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013192822 2023-07-02T14:06:48.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:48.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.815907+0000", 2023-07-02T14:06:48.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0086451870000000004 2023-07-02T14:06:48.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:48.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824553+0000", 2023-07-02T14:06:48.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6759999999999999e-05 2023-07-02T14:06:48.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:48.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824609+0000", 2023-07-02T14:06:48.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011314700000000001 2023-07-02T14:06:48.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:48.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824722+0000", 2023-07-02T14:06:48.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999 2023-07-02T14:06:48.430 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.430 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.430 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.431 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:165 2.3 2:c9c6393a:::rbd_data.125de1449605.000000000000002c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:06:48.431 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:06:48.431 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 111.499895467, 2023-07-02T14:06:48.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002, 2023-07-02T14:06:48.431 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:48.431 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:48.432 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:48.432 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:06:48.432 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:06:48.432 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 165 2023-07-02T14:06:48.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.432 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:48.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.432 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:48.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:06:48.433 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:48.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.433 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.433 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:48.433 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:06:48.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:48.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:48.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839777+0000", 2023-07-02T14:06:48.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6740000000000001e-06 2023-07-02T14:06:48.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:48.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839780+0000", 2023-07-02T14:06:48.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9030000000000004e-06 2023-07-02T14:06:48.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:48.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839788+0000", 2023-07-02T14:06:48.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1799999999999998e-06 2023-07-02T14:06:48.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:48.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839792+0000", 2023-07-02T14:06:48.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.031592227000000001 2023-07-02T14:06:48.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871384+0000", 2023-07-02T14:06:48.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013600099999999999 2023-07-02T14:06:48.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:48.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871520+0000", 2023-07-02T14:06:48.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082817089999999999 2023-07-02T14:06:48.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:48.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.879802+0000", 2023-07-02T14:06:48.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002 2023-07-02T14:06:48.438 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.438 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.439 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.439 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:252 2.7 2:f1d0abf0:::rbd_data.125de1449605.0000000000000083:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:06:48.439 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:06:48.439 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 111.083610497, 2023-07-02T14:06:48.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997, 2023-07-02T14:06:48.439 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:48.440 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:48.440 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:48.440 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:06:48.440 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:06:48.440 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 252 2023-07-02T14:06:48.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.440 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:48.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:48.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:06:48.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:48.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:48.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:06:48.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:48.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.442 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:48.442 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256062+0000", 2023-07-02T14:06:48.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0149999999999999e-06 2023-07-02T14:06:48.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:48.443 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256065+0000", 2023-07-02T14:06:48.443 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5269999999999998e-06 2023-07-02T14:06:48.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:48.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256071+0000", 2023-07-02T14:06:48.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:06:48.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:48.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256074+0000", 2023-07-02T14:06:48.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028852909 2023-07-02T14:06:48.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.284927+0000", 2023-07-02T14:06:48.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168274 2023-07-02T14:06:48.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:48.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.285096+0000", 2023-07-02T14:06:48.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011036300000000001 2023-07-02T14:06:48.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:48.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296132+0000", 2023-07-02T14:06:48.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997 2023-07-02T14:06:48.447 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.447 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.447 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.447 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:261 2.1 2:971731cb:::rbd_data.125de1449605.000000000000008c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:06:48.447 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:06:48.448 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 111.083515022, 2023-07-02T14:06:48.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984, 2023-07-02T14:06:48.448 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:48.448 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:48.448 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:48.448 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:06:48.448 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:06:48.449 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 261 2023-07-02T14:06:48.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.449 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:48.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:48.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:06:48.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:48.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:48.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:06:48.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:48.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:48.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:06:48.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2819999999999999e-06 2023-07-02T14:06:48.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.451 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:48.451 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256160+0000", 2023-07-02T14:06:48.451 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2020000000000003e-06 2023-07-02T14:06:48.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.452 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:48.452 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256165+0000", 2023-07-02T14:06:48.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2100000000000003e-06 2023-07-02T14:06:48.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:48.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256170+0000", 2023-07-02T14:06:48.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025871715999999999 2023-07-02T14:06:48.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282041+0000", 2023-07-02T14:06:48.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132364 2023-07-02T14:06:48.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:48.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282174+0000", 2023-07-02T14:06:48.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014044566 2023-07-02T14:06:48.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:48.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296218+0000", 2023-07-02T14:06:48.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984 2023-07-02T14:06:48.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.455 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.456 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:264 2.3 2:dbf06c9d:::rbd_data.125de1449605.000000000000008f:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:06:48.456 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:06:48.456 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 111.083480904, 2023-07-02T14:06:48.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003, 2023-07-02T14:06:48.456 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:48.457 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:48.457 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:48.457 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:06:48.457 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:06:48.457 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 264 2023-07-02T14:06:48.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.457 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:48.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:48.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:06:48.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:48.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:48.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:06:48.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:48.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:48.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:06:48.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5749999999999999e-06 2023-07-02T14:06:48.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:48.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256195+0000", 2023-07-02T14:06:48.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.327e-06 2023-07-02T14:06:48.460 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:48.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256200+0000", 2023-07-02T14:06:48.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0580000000000002e-06 2023-07-02T14:06:48.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.461 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:48.461 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256203+0000", 2023-07-02T14:06:48.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035707524999999997 2023-07-02T14:06:48.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.462 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.291911+0000", 2023-07-02T14:06:48.462 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014269799999999999 2023-07-02T14:06:48.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:48.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.292053+0000", 2023-07-02T14:06:48.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062760309999999996 2023-07-02T14:06:48.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:48.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.298329+0000", 2023-07-02T14:06:48.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003 2023-07-02T14:06:48.464 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.464 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.464 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.464 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:341 2.3 2:cee258bd:::rbd_data.125de1449605.00000000000000dc:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:06:48.465 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:06:48.465 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 110.702993577, 2023-07-02T14:06:48.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997, 2023-07-02T14:06:48.465 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:48.465 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:48.465 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:48.465 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:06:48.465 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:06:48.466 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 341 2023-07-02T14:06:48.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.466 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:48.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:48.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:06:48.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:48.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:48.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:06:48.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:48.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:48.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636679+0000", 2023-07-02T14:06:48.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6240000000000002e-06 2023-07-02T14:06:48.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:48.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636682+0000", 2023-07-02T14:06:48.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6910000000000002e-06 2023-07-02T14:06:48.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:48.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636688+0000", 2023-07-02T14:06:48.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3469999999999999e-06 2023-07-02T14:06:48.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:48.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636691+0000", 2023-07-02T14:06:48.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035625560000000001 2023-07-02T14:06:48.470 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.470 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.470 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.470 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672317+0000", 2023-07-02T14:06:48.470 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108391 2023-07-02T14:06:48.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:48.471 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672425+0000", 2023-07-02T14:06:48.471 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012104976 2023-07-02T14:06:48.471 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.471 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.471 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:48.472 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.684530+0000", 2023-07-02T14:06:48.472 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997 2023-07-02T14:06:48.472 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.472 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.472 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.472 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.473 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.473 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPushReply(2.7 139/134 [PushReplyOp(2:e3782f39:::rbd_data.125de1449605.0000000000000120:head)])", 2023-07-02T14:06:48.473 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:06:48.473 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 96.412461424, 2023-07-02T14:06:48.473 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998, 2023-07-02T14:06:48.473 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:48.474 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:48.474 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:48.474 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.474 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:48.474 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:06:48.474 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:48.474 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:48.475 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:06:48.475 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:48.475 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.475 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.475 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:48.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927211+0000", 2023-07-02T14:06:48.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3689999999999998e-06 2023-07-02T14:06:48.476 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.476 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.476 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:48.476 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927217+0000", 2023-07-02T14:06:48.476 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7269999999999996e-06 2023-07-02T14:06:48.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.477 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:48.477 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927225+0000", 2023-07-02T14:06:48.477 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:06:48.477 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.477 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:48.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927230+0000", 2023-07-02T14:06:48.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:06:48.478 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.478 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.478 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.478 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927266+0000", 2023-07-02T14:06:48.478 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.069972871000000006 2023-07-02T14:06:48.479 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.479 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.479 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:48.479 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.997239+0000", 2023-07-02T14:06:48.479 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998 2023-07-02T14:06:48.479 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.479 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.479 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.480 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.480 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.480 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19179 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2832, mlcod=177'2827)", 2023-07-02T14:06:48.480 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:06:48.480 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 40.880931594000003, 2023-07-02T14:06:48.480 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051928864999999998, 2023-07-02T14:06:48.480 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:48.481 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:48.481 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:48.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.481 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:48.481 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:06:48.481 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:48.481 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.481 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:48.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:06:48.482 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:48.482 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.482 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.482 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:48.482 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458741+0000", 2023-07-02T14:06:48.483 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01233852 2023-07-02T14:06:48.483 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.483 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.483 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:48.483 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471079+0000", 2023-07-02T14:06:48.484 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8131e-05 2023-07-02T14:06:48.484 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.484 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.484 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:48.485 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471097+0000", 2023-07-02T14:06:48.485 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.19e-06 2023-07-02T14:06:48.485 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.485 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:48.486 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471104+0000", 2023-07-02T14:06:48.486 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4463999999999999e-05 2023-07-02T14:06:48.486 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.486 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.486 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471158+0000", 2023-07-02T14:06:48.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1723000000000001e-05 2023-07-02T14:06:48.487 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.487 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.487 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:48.487 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471190+0000", 2023-07-02T14:06:48.487 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039438622999999999 2023-07-02T14:06:48.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.488 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:48.488 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510628+0000", 2023-07-02T14:06:48.488 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2280000000000002e-05 2023-07-02T14:06:48.488 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.488 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.489 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:48.489 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510671+0000", 2023-07-02T14:06:48.489 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051928864999999998 2023-07-02T14:06:48.489 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.489 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.489 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.489 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.489 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.490 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19180 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2833, mlcod=177'2827)", 2023-07-02T14:06:48.490 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:06:48.490 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 40.868554353999997, 2023-07-02T14:06:48.490 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039574109000000003, 2023-07-02T14:06:48.490 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:48.490 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:48.490 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:48.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:48.491 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:06:48.491 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:48.491 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.491 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.491 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:48.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:06:48.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:48.492 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.492 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.492 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:48.492 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471118+0000", 2023-07-02T14:06:48.492 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4799000000000001e-05 2023-07-02T14:06:48.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:48.493 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471143+0000", 2023-07-02T14:06:48.493 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3960000000000003e-06 2023-07-02T14:06:48.493 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.493 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.493 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:48.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471149+0000", 2023-07-02T14:06:48.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9230000000000004e-06 2023-07-02T14:06:48.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.494 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.494 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:48.494 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471153+0000", 2023-07-02T14:06:48.494 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015076050000000001 2023-07-02T14:06:48.494 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.486229+0000", 2023-07-02T14:06:48.495 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8466000000000002e-05 2023-07-02T14:06:48.495 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.495 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.495 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:48.495 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.486257+0000", 2023-07-02T14:06:48.496 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024419020999999999 2023-07-02T14:06:48.496 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.496 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.496 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:48.497 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510676+0000", 2023-07-02T14:06:48.497 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6693000000000001e-05 2023-07-02T14:06:48.497 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.497 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.498 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:48.498 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510693+0000", 2023-07-02T14:06:48.498 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039574109000000003 2023-07-02T14:06:48.498 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.499 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.499 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.499 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.500 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.500 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19181 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2834, mlcod=177'2827)", 2023-07-02T14:06:48.500 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:06:48.500 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 40.868513092999997, 2023-07-02T14:06:48.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039548481000000003, 2023-07-02T14:06:48.501 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:48.501 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:48.501 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:48.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:48.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:06:48.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:48.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:48.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:06:48.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:48.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:48.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:06:48.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8320999999999999e-05 2023-07-02T14:06:48.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:48.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471188+0000", 2023-07-02T14:06:48.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9669999999999998e-06 2023-07-02T14:06:48.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:48.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471194+0000", 2023-07-02T14:06:48.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8709999999999999e-06 2023-07-02T14:06:48.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:48.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471198+0000", 2023-07-02T14:06:48.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016760634999999999 2023-07-02T14:06:48.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.487959+0000", 2023-07-02T14:06:48.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8398000000000002e-05 2023-07-02T14:06:48.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:48.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.487987+0000", 2023-07-02T14:06:48.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022707627000000001 2023-07-02T14:06:48.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:48.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510695+0000", 2023-07-02T14:06:48.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3991999999999999e-05 2023-07-02T14:06:48.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:48.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510709+0000", 2023-07-02T14:06:48.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039548481000000003 2023-07-02T14:06:48.514 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.514 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.515 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.515 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19192 2.7 e177/141 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head v 177'2835, mlcod=177'2827)", 2023-07-02T14:06:48.516 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:06:48.516 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 40.868470121000001, 2023-07-02T14:06:48.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039521186999999999, 2023-07-02T14:06:48.517 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:48.517 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:48.517 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:48.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:48.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:06:48.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:48.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:48.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:06:48.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:48.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:48.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:06:48.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014525079 2023-07-02T14:06:48.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:48.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485728+0000", 2023-07-02T14:06:48.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8372e-05 2023-07-02T14:06:48.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:48.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485746+0000", 2023-07-02T14:06:48.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6149999999999997e-06 2023-07-02T14:06:48.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:48.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485755+0000", 2023-07-02T14:06:48.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036949270000000002 2023-07-02T14:06:48.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489450+0000", 2023-07-02T14:06:48.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9415e-05 2023-07-02T14:06:48.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:48.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489469+0000", 2023-07-02T14:06:48.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021241488999999999 2023-07-02T14:06:48.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:48.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510711+0000", 2023-07-02T14:06:48.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3604999999999999e-05 2023-07-02T14:06:48.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:48.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510724+0000", 2023-07-02T14:06:48.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039521186999999999 2023-07-02T14:06:48.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.524 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.524 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.525 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:20659 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2134016~94208] snapc 35=[35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e179)", 2023-07-02T14:06:48.525 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:06:48.525 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.749286738999999, 2023-07-02T14:06:48.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041695300999999997, 2023-07-02T14:06:48.525 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:48.526 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:48.526 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:48.526 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:48.526 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:48.526 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20659 2023-07-02T14:06:48.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.526 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:48.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:48.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:06:48.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:48.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:48.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:06:48.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:48.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:48.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590386+0000", 2023-07-02T14:06:48.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1415999999999994e-05 2023-07-02T14:06:48.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:48.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590467+0000", 2023-07-02T14:06:48.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8029999999999999e-06 2023-07-02T14:06:48.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:48.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590474+0000", 2023-07-02T14:06:48.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5379999999999998e-06 2023-07-02T14:06:48.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:48.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590478+0000", 2023-07-02T14:06:48.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073612449999999998 2023-07-02T14:06:48.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.597839+0000", 2023-07-02T14:06:48.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013387800000000001 2023-07-02T14:06:48.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:48.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.597973+0000", 2023-07-02T14:06:48.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022472799999999999 2023-07-02T14:06:48.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:48.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.598197+0000", 2023-07-02T14:06:48.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065294259999999996 2023-07-02T14:06:48.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:48.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.604727+0000", 2023-07-02T14:06:48.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027239217999999999 2023-07-02T14:06:48.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:48.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.631966+0000", 2023-07-02T14:06:48.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3194999999999994e-05 2023-07-02T14:06:48.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:48.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632029+0000", 2023-07-02T14:06:48.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2639999999999997e-05 2023-07-02T14:06:48.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:48.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632082+0000", 2023-07-02T14:06:48.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041695300999999997 2023-07-02T14:06:48.536 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.536 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.536 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.537 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:20660 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2031616~196608] snapc 35=[35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e179)", 2023-07-02T14:06:48.537 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:06:48.537 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.749191719000002, 2023-07-02T14:06:48.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041610001000000001, 2023-07-02T14:06:48.537 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:48.537 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:48.537 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:48.538 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:48.538 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:48.538 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20660 2023-07-02T14:06:48.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.538 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:48.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:48.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:06:48.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:48.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:48.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:06:48.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:48.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:48.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590481+0000", 2023-07-02T14:06:48.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012730599999999999 2023-07-02T14:06:48.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:48.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590608+0000", 2023-07-02T14:06:48.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6319999999999997e-06 2023-07-02T14:06:48.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:48.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590617+0000", 2023-07-02T14:06:48.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5209999999999998e-06 2023-07-02T14:06:48.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:48.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590621+0000", 2023-07-02T14:06:48.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018367430000000001 2023-07-02T14:06:48.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:48.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592457+0000", 2023-07-02T14:06:48.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016640599999999999 2023-07-02T14:06:48.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:48.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592624+0000", 2023-07-02T14:06:48.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028927400000000001 2023-07-02T14:06:48.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:48.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592913+0000", 2023-07-02T14:06:48.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011797813000000001 2023-07-02T14:06:48.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:48.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.604711+0000", 2023-07-02T14:06:48.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027285997999999999 2023-07-02T14:06:48.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:48.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.631997+0000", 2023-07-02T14:06:48.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8890000000000001e-05 2023-07-02T14:06:48.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:48.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632046+0000", 2023-07-02T14:06:48.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5867e-05 2023-07-02T14:06:48.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:48.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:48.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:48.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632092+0000", 2023-07-02T14:06:48.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041610001000000001 2023-07-02T14:06:48.547 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.547 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.547 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.547 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:48.547 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:48.548 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:48.549 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:06:48.549 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:48.548+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:48.650 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:48.650+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:48.687 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:48.688 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:48.688 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:06:48.688 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:48.698 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:06:48.751 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:48.750+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:48.822 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:48.822 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:48.823 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:48.823 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:48.823 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:48.833 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:06:48.874 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:48.873+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:48.952 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:48.951+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:48.999 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:48.999 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:48.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:48.999 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:48.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:48.999 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:06:49.000 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:49.000 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 189.099758229, 2023-07-02T14:06:49.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:06:49.000 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:49.000 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:49.000 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:49.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:49.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:49.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:49.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:49.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:49.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:49.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:49.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:06:49.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:06:49.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:49.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:06:49.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:06:49.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:49.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:06:49.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:06:49.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:49.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:06:49.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:06:49.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:49.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:06:49.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:06:49.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:49.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:06:49.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:06:49.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:49.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:06:49.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:06:49.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:49.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:06:49.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:06:49.007 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.007 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.007 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.007 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:06:49.008 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:49.008 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 189.095672511, 2023-07-02T14:06:49.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:06:49.008 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:49.008 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:49.008 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:49.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:49.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:49.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:49.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:49.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:49.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:49.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:49.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:06:49.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:06:49.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:49.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:06:49.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:06:49.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:49.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:06:49.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:06:49.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:49.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:06:49.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:06:49.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:49.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:06:49.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:06:49.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:49.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:06:49.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:06:49.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:49.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:06:49.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:06:49.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:49.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:06:49.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:06:49.015 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.015 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.015 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.015 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:06:49.015 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:49.015 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 189.09180773, 2023-07-02T14:06:49.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:06:49.016 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:49.016 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:49.016 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:49.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:49.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:49.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:49.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:49.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:49.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:49.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:49.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:06:49.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:06:49.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:49.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:06:49.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:06:49.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:49.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:06:49.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:06:49.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:49.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:06:49.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:06:49.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:49.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:06:49.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:06:49.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:49.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:06:49.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:06:49.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:49.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:06:49.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:06:49.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:49.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:06:49.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:06:49.022 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.022 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.023 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.023 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:06:49.023 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:49.023 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 189.05257867, 2023-07-02T14:06:49.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:06:49.023 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:49.024 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:49.024 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:49.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:49.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:49.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:49.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:49.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:49.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:49.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:49.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:06:49.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:06:49.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:49.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:06:49.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:06:49.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:49.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:06:49.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:06:49.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:49.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:06:49.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:06:49.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:49.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:06:49.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:06:49.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:49.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:06:49.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:06:49.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:49.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:06:49.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:06:49.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:49.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:06:49.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:06:49.030 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.030 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.030 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.031 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:06:49.031 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:49.031 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 189.047921312, 2023-07-02T14:06:49.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:06:49.031 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:49.031 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:49.032 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:49.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:49.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:49.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:49.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:49.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:49.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:49.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:49.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:06:49.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:06:49.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:49.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:06:49.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:06:49.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:49.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:06:49.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:06:49.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:49.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:06:49.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:06:49.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:49.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:06:49.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:06:49.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:49.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:06:49.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:06:49.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:49.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:06:49.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:06:49.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:49.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:06:49.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:06:49.038 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.038 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.038 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.039 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:06:49.039 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:49.039 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 189.04698144899999, 2023-07-02T14:06:49.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:06:49.039 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:49.039 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:49.039 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:49.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:49.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:49.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:49.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:49.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:49.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:49.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:49.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:06:49.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:06:49.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:49.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:06:49.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:06:49.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:49.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:06:49.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:06:49.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:49.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:06:49.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:06:49.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:49.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:06:49.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:06:49.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:49.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:06:49.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:06:49.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:49.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:06:49.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:06:49.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:49.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:06:49.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:06:49.046 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.046 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.046 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.046 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:06:49.046 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:49.046 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 189.04558357100001, 2023-07-02T14:06:49.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:06:49.047 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:49.047 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:49.047 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:49.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:49.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:49.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:49.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:49.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:49.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:49.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:49.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:06:49.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:06:49.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:49.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:06:49.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:06:49.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:49.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:06:49.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:06:49.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:49.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:06:49.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:06:49.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:49.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:06:49.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:06:49.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:49.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:06:49.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:06:49.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:49.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:06:49.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:06:49.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:49.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:06:49.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:06:49.053 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.054 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.054 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.054 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:06:49.054 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:49.054 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 167.635393075, 2023-07-02T14:06:49.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:06:49.055 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:49.055 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:49.055 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:49.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:49.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:49.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:49.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:49.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:49.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:49.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:49.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:06:49.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:06:49.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:49.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:06:49.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:06:49.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:49.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:06:49.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:06:49.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:49.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:06:49.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:06:49.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:49.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:06:49.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:06:49.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:49.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:06:49.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:06:49.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:49.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:06:49.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:06:49.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:49.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:06:49.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:06:49.061 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.061 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.061 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.062 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:49.062 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:49.062 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.269043391, 2023-07-02T14:06:49.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:06:49.062 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:49.062 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:49.063 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:49.063 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:49.063 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:49.063 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:06:49.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.063 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:49.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:49.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:49.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:49.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:49.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:49.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:49.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:49.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:06:49.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:06:49.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:49.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:06:49.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:06:49.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:49.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:06:49.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:06:49.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:49.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:06:49.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:06:49.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:49.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:06:49.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:06:49.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:49.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:06:49.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:06:49.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:49.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:06:49.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:06:49.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:49.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:06:49.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:06:49.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:49.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:06:49.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:06:49.070 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.070 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.071 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.071 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:49.071 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:49.071 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.26499604700001, 2023-07-02T14:06:49.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:06:49.071 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:49.072 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:49.072 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:49.072 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:49.072 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:49.072 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:06:49.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.072 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:49.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:49.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:49.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:49.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:49.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:49.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:49.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:49.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:49.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:06:49.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:49.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:06:49.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:06:49.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:49.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:06:49.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:06:49.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:49.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:06:49.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:06:49.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:49.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:06:49.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:06:49.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:49.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:06:49.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:06:49.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:49.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:06:49.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:06:49.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:49.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:06:49.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:06:49.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:49.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:06:49.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:06:49.079 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.080 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.080 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.080 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:49.080 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:49.080 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.26043222199999, 2023-07-02T14:06:49.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:06:49.081 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:49.081 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:49.081 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:49.081 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:49.081 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:49.081 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:06:49.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.081 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:49.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:49.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:49.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:49.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:49.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:49.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:49.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:49.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:06:49.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:06:49.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:49.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:06:49.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:06:49.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:49.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:06:49.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:06:49.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:49.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:06:49.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:06:49.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:49.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:06:49.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:06:49.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:49.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:06:49.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:06:49.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:49.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:06:49.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:06:49.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:49.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:06:49.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:06:49.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:49.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:06:49.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:06:49.088 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.089 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.089 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.089 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:49.089 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:49.089 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.26038235600001, 2023-07-02T14:06:49.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:06:49.090 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:49.090 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:49.090 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:49.090 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:49.090 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:49.090 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:06:49.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.090 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:49.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:49.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:49.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:49.091 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.091 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:49.091 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:49.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:49.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.092 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.092 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:49.092 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:06:49.092 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:06:49.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:49.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:06:49.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:06:49.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:49.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:06:49.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:06:49.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:49.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:06:49.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:06:49.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:49.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:06:49.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:06:49.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:49.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:06:49.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:06:49.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:49.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:06:49.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:06:49.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:49.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:06:49.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:06:49.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:49.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:06:49.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:06:49.098 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.098 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.098 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.098 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:49.098 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:49.098 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 166.259343363, 2023-07-02T14:06:49.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:06:49.099 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:49.099 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:49.099 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:49.099 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:49.099 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:49.099 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:06:49.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.100 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:49.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:49.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:49.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:49.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:49.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:49.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:49.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:49.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:06:49.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:06:49.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:49.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:06:49.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:06:49.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:49.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:06:49.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:06:49.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:49.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:06:49.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:06:49.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:49.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:06:49.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:06:49.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:49.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:06:49.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:06:49.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:49.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:06:49.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:06:49.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:49.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:06:49.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:06:49.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:49.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:06:49.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:06:49.107 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.107 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.107 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.107 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22312 2.0 e197/193 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 197'5590, mlcod=197'5582)", 2023-07-02T14:06:49.107 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:06:49.107 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.248898664, 2023-07-02T14:06:49.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021943201999999998, 2023-07-02T14:06:49.108 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:49.108 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:49.108 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:49.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:49.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:06:49.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:49.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:49.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:06:49.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:49.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:49.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748473+0000", 2023-07-02T14:06:49.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00099631099999999994 2023-07-02T14:06:49.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:49.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749469+0000", 2023-07-02T14:06:49.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4279e-05 2023-07-02T14:06:49.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:49.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749483+0000", 2023-07-02T14:06:49.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5289999999999997e-06 2023-07-02T14:06:49.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:49.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749490+0000", 2023-07-02T14:06:49.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047909650000000003 2023-07-02T14:06:49.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:49.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.754281+0000", 2023-07-02T14:06:49.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1999000000000001e-05 2023-07-02T14:06:49.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:49.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.754303+0000", 2023-07-02T14:06:49.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016063747 2023-07-02T14:06:49.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:49.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.770367+0000", 2023-07-02T14:06:49.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0231000000000001e-05 2023-07-02T14:06:49.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:49.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.770417+0000", 2023-07-02T14:06:49.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021943201999999998 2023-07-02T14:06:49.114 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.114 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.115 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.115 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22313 2.0 e197/193 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 197'5592, mlcod=197'5587)", 2023-07-02T14:06:49.115 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:06:49.115 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.243645260999999, 2023-07-02T14:06:49.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026016164000000001, 2023-07-02T14:06:49.115 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:49.116 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:49.116 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:49.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:49.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:06:49.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:49.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:49.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:06:49.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999933 2023-07-02T14:06:49.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:49.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753720+0000", 2023-07-02T14:06:49.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4851999999999995e-05 2023-07-02T14:06:49.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:49.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753785+0000", 2023-07-02T14:06:49.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0216e-05 2023-07-02T14:06:49.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:49.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753796+0000", 2023-07-02T14:06:49.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4249999999999997e-06 2023-07-02T14:06:49.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:49.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753801+0000", 2023-07-02T14:06:49.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070274359999999998 2023-07-02T14:06:49.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:49.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760828+0000", 2023-07-02T14:06:49.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0975000000000002e-05 2023-07-02T14:06:49.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:49.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760849+0000", 2023-07-02T14:06:49.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018855916 2023-07-02T14:06:49.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:49.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.779705+0000", 2023-07-02T14:06:49.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8037999999999997e-05 2023-07-02T14:06:49.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:49.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.779743+0000", 2023-07-02T14:06:49.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026016164000000001 2023-07-02T14:06:49.123 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.123 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.123 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.123 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22314 2.0 e197/193 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 197'5594, mlcod=197'5587)", 2023-07-02T14:06:49.123 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:06:49.124 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.241040607, 2023-07-02T14:06:49.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027004523999999999, 2023-07-02T14:06:49.124 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:49.124 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:49.124 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:49.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:49.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:06:49.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:49.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:49.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:06:49.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:49.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:49.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756331+0000", 2023-07-02T14:06:49.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4697999999999999e-05 2023-07-02T14:06:49.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:49.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756365+0000", 2023-07-02T14:06:49.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3902e-05 2023-07-02T14:06:49.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:49.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756379+0000", 2023-07-02T14:06:49.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7329999999999997e-06 2023-07-02T14:06:49.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:49.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756386+0000", 2023-07-02T14:06:49.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062861929999999998 2023-07-02T14:06:49.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:49.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.762672+0000", 2023-07-02T14:06:49.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9031e-05 2023-07-02T14:06:49.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:49.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.762711+0000", 2023-07-02T14:06:49.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020589275000000001 2023-07-02T14:06:49.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:49.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783301+0000", 2023-07-02T14:06:49.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5720999999999999e-05 2023-07-02T14:06:49.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:49.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783336+0000", 2023-07-02T14:06:49.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027004523999999999 2023-07-02T14:06:49.130 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.131 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.131 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.131 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22315 2.0 e197/193 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 197'5595, mlcod=197'5587)", 2023-07-02T14:06:49.131 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:06:49.131 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.237071336, 2023-07-02T14:06:49.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023064345999999999, 2023-07-02T14:06:49.132 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:49.132 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:49.132 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:49.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:49.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:06:49.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:49.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:49.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:06:49.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:49.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:49.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760300+0000", 2023-07-02T14:06:49.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3056000000000002e-05 2023-07-02T14:06:49.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:49.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760353+0000", 2023-07-02T14:06:49.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2825e-05 2023-07-02T14:06:49.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:49.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760366+0000", 2023-07-02T14:06:49.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8290000000000002e-06 2023-07-02T14:06:49.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:49.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760373+0000", 2023-07-02T14:06:49.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075982089999999999 2023-07-02T14:06:49.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:49.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.767971+0000", 2023-07-02T14:06:49.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4096000000000001e-05 2023-07-02T14:06:49.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:49.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.767995+0000", 2023-07-02T14:06:49.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015345908 2023-07-02T14:06:49.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:49.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783341+0000", 2023-07-02T14:06:49.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4273e-05 2023-07-02T14:06:49.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:49.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783365+0000", 2023-07-02T14:06:49.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023064345999999999 2023-07-02T14:06:49.138 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.138 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.138 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.139 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22319 2.0 e197/193 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 197'5597, mlcod=197'5587)", 2023-07-02T14:06:49.139 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:06:49.139 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.233568516, 2023-07-02T14:06:49.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022862501, 2023-07-02T14:06:49.139 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:49.139 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:49.140 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:49.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:49.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:06:49.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:49.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:49.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:06:49.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:49.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:49.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763803+0000", 2023-07-02T14:06:49.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052067220000000004 2023-07-02T14:06:49.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:49.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769010+0000", 2023-07-02T14:06:49.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3249e-05 2023-07-02T14:06:49.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:49.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769023+0000", 2023-07-02T14:06:49.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1790000000000008e-06 2023-07-02T14:06:49.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:49.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769031+0000", 2023-07-02T14:06:49.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030611079999999999 2023-07-02T14:06:49.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:49.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.772092+0000", 2023-07-02T14:06:49.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3958e-05 2023-07-02T14:06:49.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:49.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.772116+0000", 2023-07-02T14:06:49.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014504064000000001 2023-07-02T14:06:49.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:49.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.786620+0000", 2023-07-02T14:06:49.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5952999999999999e-05 2023-07-02T14:06:49.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:49.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.786666+0000", 2023-07-02T14:06:49.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022862501 2023-07-02T14:06:49.146 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.146 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.146 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.147 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22335 2.0 e197/193 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 197'5602, mlcod=197'5599)", 2023-07-02T14:06:49.147 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:06:49.147 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.206498239, 2023-07-02T14:06:49.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024437743000000001, 2023-07-02T14:06:49.147 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:49.147 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:49.147 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:49.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:49.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:06:49.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:49.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:49.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:06:49.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:49.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:49.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790873+0000", 2023-07-02T14:06:49.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081672399999999992 2023-07-02T14:06:49.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:49.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799040+0000", 2023-07-02T14:06:49.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5712999999999999e-05 2023-07-02T14:06:49.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:49.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799066+0000", 2023-07-02T14:06:49.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1413999999999999e-05 2023-07-02T14:06:49.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:49.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799077+0000", 2023-07-02T14:06:49.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3102000000000007e-05 2023-07-02T14:06:49.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:49.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799140+0000", 2023-07-02T14:06:49.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.883e-05 2023-07-02T14:06:49.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:49.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799169+0000", 2023-07-02T14:06:49.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016086700999999998 2023-07-02T14:06:49.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:49.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.815256+0000", 2023-07-02T14:06:49.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5967000000000001e-05 2023-07-02T14:06:49.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:49.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.815312+0000", 2023-07-02T14:06:49.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024437743000000001 2023-07-02T14:06:49.154 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.154 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.154 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.154 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22336 2.0 e197/193 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 197'5603, mlcod=197'5599)", 2023-07-02T14:06:49.154 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:06:49.155 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 14.198262953, 2023-07-02T14:06:49.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022713011000000002, 2023-07-02T14:06:49.155 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:49.155 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:49.155 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:49.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:49.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:06:49.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:49.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:49.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:06:49.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:49.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:49.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799108+0000", 2023-07-02T14:06:49.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00058943600000000004 2023-07-02T14:06:49.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:49.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799698+0000", 2023-07-02T14:06:49.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6205000000000002e-05 2023-07-02T14:06:49.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:49.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799714+0000", 2023-07-02T14:06:49.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9630000000000001e-06 2023-07-02T14:06:49.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:49.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799721+0000", 2023-07-02T14:06:49.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076846579999999996 2023-07-02T14:06:49.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:49.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.807405+0000", 2023-07-02T14:06:49.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8191000000000002e-05 2023-07-02T14:06:49.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:49.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.807434+0000", 2023-07-02T14:06:49.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014338020999999999 2023-07-02T14:06:49.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:49.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.821772+0000", 2023-07-02T14:06:49.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0868999999999997e-05 2023-07-02T14:06:49.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:49.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:49.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:49.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.821822+0000", 2023-07-02T14:06:49.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022713011000000002 2023-07-02T14:06:49.161 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.161 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.162 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.162 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:49.162 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:49.162 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:49.163 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:06:49.164 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:06:49.164 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:49.163+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:49.264 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:49.264+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:49.286 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:06:49.329 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:49.330 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:49.330 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:06:49.330 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:49.340 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:06:49.365 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:49.365+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:49.437 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:49.455 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:49.463 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:49.464 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:49.464 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:49.464 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:49.464 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:49.466 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:49.465+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:49.472 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:49.473 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:06:49.490 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:49.509 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:49.528 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:49.529 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:06:49.529 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:06:49.529 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:06:49.529 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:06:49.529 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:06:49.530 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:06:49.581 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:49.581+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:49.599 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:49.599 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:49.609 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:06:49.667 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:49.667+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:49.733 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:49.733 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:49.743 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:06:49.768 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:49.768+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:49.869 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:49.869+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:49.906 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:49.907 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:49.917 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:06:49.970 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:49.970+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:50.040 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:50.041 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:50.051 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:06:50.094 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:50.094+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:50.115 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 2, 4] out_osds: [5] dead_osds: [] live_osds: [1, 2, 4, 5, 3, 0] 2023-07-02T14:06:50.116 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:06:50.116 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2023-07-02T14:06:50.116 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-07-02T14:06:50.116 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-07-02T14:06:50.117 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2023-07-02T14:06:50.172 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:50.172+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:50.189 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:50.189 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:50.200 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:06:50.247 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:50.248 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "" 2023-07-02T14:06:50.248 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:50.273 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:50.272+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:50.366 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:50.366 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:50.374 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:50.373+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:50.377 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:06:50.475 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:50.475+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:50.500 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:50.501 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:50.511 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:06:50.576 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:50.576+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:50.676 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:50.676 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:50.677 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:50.677+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:50.686 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:06:50.778 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:50.778+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:50.850 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:50.850 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:50.861 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:06:50.878 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:50.878+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:50.979 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:50.979+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:50.986 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:50.987 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:50.997 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:06:51.080 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:51.080+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:51.161 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap60 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap61 b3fbb028-e267-4d25-a750-93ba18a2316a@snap61.diff 2023-07-02T14:06:51.162 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:51.163 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:51.165 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:06:51.173 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:06:51.181 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:51.181+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:51.250 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 100% complete...done. 2023-07-02T14:06:51.253 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:06:51.253 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=62 2023-07-02T14:06:51.254 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap62 2023-07-02T14:06:51.282 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:51.281+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:51.366 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:51.366 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:51.380 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:06:51.382 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:51.382+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:51.483 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:51.483+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:51.504 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:51.504 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:51.514 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:06:51.541 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:06:51.597 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:51.597+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:51.642 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:51.642 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:51.653 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:06:51.684 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:51.684+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:51.692 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:51.710 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:51.727 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:51.745 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:51.764 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:51.776 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:51.776 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:51.777 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:51.777 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:51.777 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:51.782 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:51.783 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:06:51.783 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:06:51.783 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:06:51.784 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:06:51.784 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:06:51.784 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:06:51.785 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:51.785+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:51.786 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:06:51.886 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:51.885+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:51.912 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:51.912 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:51.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:51.913 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:51.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.913 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:06:51.913 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:51.913 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.013148148, 2023-07-02T14:06:51.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:06:51.913 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:51.914 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:51.914 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:51.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:51.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:51.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:51.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:51.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:06:51.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:51.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:51.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:06:51.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:06:51.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:51.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:06:51.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:06:51.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:51.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:06:51.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:06:51.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:51.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:06:51.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:06:51.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:51.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:06:51.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:06:51.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:51.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:06:51.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:06:51.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:51.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:06:51.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:06:51.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:51.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:06:51.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:06:51.921 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:51.921 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:51.921 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:51.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.921 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:06:51.922 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:51.922 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.00906243, 2023-07-02T14:06:51.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:06:51.922 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:51.922 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:51.922 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:51.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:51.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:51.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:51.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:51.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:06:51.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:51.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:51.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:06:51.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:06:51.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:51.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:06:51.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:06:51.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:51.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:06:51.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:06:51.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:51.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:06:51.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:06:51.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:51.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:06:51.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:06:51.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:51.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:06:51.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:06:51.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:51.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:06:51.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:06:51.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:51.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:06:51.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:06:51.929 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:51.929 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:51.929 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:51.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.930 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:06:51.930 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:51.930 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 192.005197649, 2023-07-02T14:06:51.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:06:51.931 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:51.931 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:51.931 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:51.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:51.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:51.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:51.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:51.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:06:51.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:51.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:51.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:06:51.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:06:51.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:51.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:06:51.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:06:51.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:51.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:06:51.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:06:51.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:51.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:06:51.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:06:51.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:51.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:06:51.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:06:51.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:51.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:06:51.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:06:51.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:51.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:06:51.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:06:51.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:51.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:06:51.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:06:51.938 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:51.938 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:51.938 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:51.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.938 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:06:51.938 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:51.939 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 191.965968589, 2023-07-02T14:06:51.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:06:51.939 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:51.939 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:51.939 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:51.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:51.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:51.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:51.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:51.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:06:51.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:51.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:51.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:06:51.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:06:51.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:51.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:06:51.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:06:51.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:51.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:06:51.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:06:51.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:51.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:06:51.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:06:51.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:51.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:06:51.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:06:51.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:51.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:06:51.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:06:51.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:51.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:06:51.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:06:51.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:51.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:06:51.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:06:51.946 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:51.946 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:51.946 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:51.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.947 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:06:51.947 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:51.947 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 191.961311231, 2023-07-02T14:06:51.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:06:51.947 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:51.947 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:51.947 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:51.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:51.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:51.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:51.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:51.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:06:51.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:51.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:51.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:06:51.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:06:51.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:51.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:06:51.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:06:51.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:51.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:06:51.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:06:51.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:51.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:06:51.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:06:51.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:51.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:06:51.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:06:51.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:51.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:06:51.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:06:51.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:51.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:06:51.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:06:51.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.954 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:51.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:06:51.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:06:51.954 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:51.954 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:51.955 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:51.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.955 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:06:51.955 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:51.955 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 191.96037136800001, 2023-07-02T14:06:51.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:06:51.956 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:51.956 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:51.956 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:51.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:51.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:51.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:51.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:51.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:06:51.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:51.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:51.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:06:51.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:06:51.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:51.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:06:51.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:06:51.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:51.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:06:51.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:06:51.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:51.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:06:51.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:06:51.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:51.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:06:51.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:06:51.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:51.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:06:51.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:06:51.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:51.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:06:51.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:06:51.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:51.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:06:51.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:06:51.963 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:51.963 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:51.963 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:51.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.964 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:06:51.964 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:51.964 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 191.95897349000001, 2023-07-02T14:06:51.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:06:51.964 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:51.964 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:51.965 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:51.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:51.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:51.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:51.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:51.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:06:51.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:51.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:51.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:06:51.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:06:51.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:51.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:06:51.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:06:51.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:51.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:06:51.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:06:51.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:51.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:06:51.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:06:51.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:51.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:06:51.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:06:51.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:51.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:06:51.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:06:51.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:51.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:06:51.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:06:51.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:51.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:06:51.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:06:51.972 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:51.972 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:51.972 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:51.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.973 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:06:51.973 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:51.973 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 170.54878299399999, 2023-07-02T14:06:51.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:06:51.973 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:51.973 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:51.974 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:51.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:51.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:51.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:51.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:51.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:06:51.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:51.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:51.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:06:51.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:06:51.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:51.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:06:51.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:06:51.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:51.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:06:51.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:06:51.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:51.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:06:51.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:06:51.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:51.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:06:51.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:06:51.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:51.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:06:51.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:06:51.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:51.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:06:51.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:06:51.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:51.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:06:51.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:06:51.981 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:51.981 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:51.981 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:51.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.982 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:51.982 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:51.982 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 169.18243330999999, 2023-07-02T14:06:51.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:06:51.982 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:51.982 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:51.982 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:51.983 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:51.983 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:51.983 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:06:51.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.983 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:51.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:51.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:51.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:51.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:51.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:06:51.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:51.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:51.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:06:51.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:06:51.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:51.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:06:51.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:06:51.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:51.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:06:51.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:06:51.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:51.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:06:51.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:06:51.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:51.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:06:51.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:06:51.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:51.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:06:51.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:06:51.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:51.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:06:51.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:06:51.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:51.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:06:51.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:06:51.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:51.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:06:51.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:06:51.991 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:51.991 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:51.991 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:51.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.991 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:51.992 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:51.992 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 169.17838596600001, 2023-07-02T14:06:51.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:06:51.992 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:51.992 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:51.992 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:51.993 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:51.993 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:51.993 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:06:51.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.993 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:51.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:51.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:51.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:51.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:51.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:51.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:51.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:51.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:06:51.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:06:51.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:51.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:06:51.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:06:51.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:51.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:06:51.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:06:51.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:51.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:06:51.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:06:51.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:51.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:06:51.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:06:51.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:51.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:06:51.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:06:51.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:51.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:51.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:51.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:06:52.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:06:52.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:06:52.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:06:52.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:06:52.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:06:52.001 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.001 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.001 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.002 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:52.002 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:52.002 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 169.17382214099999, 2023-07-02T14:06:52.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:06:52.002 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.002 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.003 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.003 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.003 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.003 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:06:52.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.003 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:52.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:06:52.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:52.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:06:52.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:06:52.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:06:52.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:06:52.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:06:52.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:06:52.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:06:52.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:06:52.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:06:52.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:06:52.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:06:52.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:06:52.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:06:52.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:06:52.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:06:52.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:06:52.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:06:52.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:06:52.011 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.011 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.011 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.011 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:52.012 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:52.012 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 169.173772275, 2023-07-02T14:06:52.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:06:52.012 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.012 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.012 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.012 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.013 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.013 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:06:52.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.013 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:52.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:06:52.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:52.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:06:52.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:06:52.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:06:52.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:06:52.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:06:52.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:06:52.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:06:52.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:06:52.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:06:52.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:06:52.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:06:52.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:06:52.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:06:52.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:06:52.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:06:52.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:06:52.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:06:52.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:06:52.021 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.021 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.021 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.021 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:06:52.022 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:52.022 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 169.172733282, 2023-07-02T14:06:52.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:06:52.022 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.022 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.022 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.022 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.023 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.023 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:06:52.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.023 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:52.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:06:52.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:52.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:06:52.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:06:52.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:06:52.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:06:52.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:06:52.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:06:52.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:06:52.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:06:52.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:06:52.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:06:52.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:06:52.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:06:52.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:06:52.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:06:52.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:06:52.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:06:52.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:06:52.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:06:52.031 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.031 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.031 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.031 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22312 2.0 e197/193 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 197'5590, mlcod=197'5582)", 2023-07-02T14:06:52.032 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:06:52.032 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.162288582999999, 2023-07-02T14:06:52.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021943201999999998, 2023-07-02T14:06:52.032 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.032 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.032 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:06:52.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:06:52.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:52.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748473+0000", 2023-07-02T14:06:52.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00099631099999999994 2023-07-02T14:06:52.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749469+0000", 2023-07-02T14:06:52.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4279e-05 2023-07-02T14:06:52.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749483+0000", 2023-07-02T14:06:52.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5289999999999997e-06 2023-07-02T14:06:52.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749490+0000", 2023-07-02T14:06:52.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047909650000000003 2023-07-02T14:06:52.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.754281+0000", 2023-07-02T14:06:52.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1999000000000001e-05 2023-07-02T14:06:52.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.754303+0000", 2023-07-02T14:06:52.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016063747 2023-07-02T14:06:52.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.770367+0000", 2023-07-02T14:06:52.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0231000000000001e-05 2023-07-02T14:06:52.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.770417+0000", 2023-07-02T14:06:52.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021943201999999998 2023-07-02T14:06:52.039 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.040 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.040 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.040 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22313 2.0 e197/193 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 197'5592, mlcod=197'5587)", 2023-07-02T14:06:52.040 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:06:52.040 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.157035180000001, 2023-07-02T14:06:52.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026016164000000001, 2023-07-02T14:06:52.041 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.041 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.041 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:06:52.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:06:52.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999933 2023-07-02T14:06:52.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753720+0000", 2023-07-02T14:06:52.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4851999999999995e-05 2023-07-02T14:06:52.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753785+0000", 2023-07-02T14:06:52.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0216e-05 2023-07-02T14:06:52.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753796+0000", 2023-07-02T14:06:52.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4249999999999997e-06 2023-07-02T14:06:52.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753801+0000", 2023-07-02T14:06:52.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070274359999999998 2023-07-02T14:06:52.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760828+0000", 2023-07-02T14:06:52.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0975000000000002e-05 2023-07-02T14:06:52.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760849+0000", 2023-07-02T14:06:52.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018855916 2023-07-02T14:06:52.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.779705+0000", 2023-07-02T14:06:52.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8037999999999997e-05 2023-07-02T14:06:52.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.779743+0000", 2023-07-02T14:06:52.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026016164000000001 2023-07-02T14:06:52.049 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.049 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.049 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.049 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22314 2.0 e197/193 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 197'5594, mlcod=197'5587)", 2023-07-02T14:06:52.049 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:06:52.050 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.154430525999999, 2023-07-02T14:06:52.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027004523999999999, 2023-07-02T14:06:52.050 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.050 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.050 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:06:52.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:06:52.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:52.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756331+0000", 2023-07-02T14:06:52.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4697999999999999e-05 2023-07-02T14:06:52.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756365+0000", 2023-07-02T14:06:52.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3902e-05 2023-07-02T14:06:52.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756379+0000", 2023-07-02T14:06:52.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7329999999999997e-06 2023-07-02T14:06:52.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756386+0000", 2023-07-02T14:06:52.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062861929999999998 2023-07-02T14:06:52.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.762672+0000", 2023-07-02T14:06:52.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9031e-05 2023-07-02T14:06:52.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.762711+0000", 2023-07-02T14:06:52.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020589275000000001 2023-07-02T14:06:52.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783301+0000", 2023-07-02T14:06:52.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5720999999999999e-05 2023-07-02T14:06:52.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783336+0000", 2023-07-02T14:06:52.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027004523999999999 2023-07-02T14:06:52.057 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.057 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.058 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.058 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22315 2.0 e197/193 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 197'5595, mlcod=197'5587)", 2023-07-02T14:06:52.058 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:06:52.058 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.150461255, 2023-07-02T14:06:52.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023064345999999999, 2023-07-02T14:06:52.059 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.059 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.059 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:06:52.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:06:52.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:52.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760300+0000", 2023-07-02T14:06:52.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3056000000000002e-05 2023-07-02T14:06:52.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760353+0000", 2023-07-02T14:06:52.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2825e-05 2023-07-02T14:06:52.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760366+0000", 2023-07-02T14:06:52.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8290000000000002e-06 2023-07-02T14:06:52.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760373+0000", 2023-07-02T14:06:52.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075982089999999999 2023-07-02T14:06:52.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.767971+0000", 2023-07-02T14:06:52.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4096000000000001e-05 2023-07-02T14:06:52.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.767995+0000", 2023-07-02T14:06:52.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015345908 2023-07-02T14:06:52.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783341+0000", 2023-07-02T14:06:52.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4273e-05 2023-07-02T14:06:52.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783365+0000", 2023-07-02T14:06:52.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023064345999999999 2023-07-02T14:06:52.066 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.066 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.066 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.066 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22319 2.0 e197/193 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 197'5597, mlcod=197'5587)", 2023-07-02T14:06:52.067 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:06:52.067 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.146958434999998, 2023-07-02T14:06:52.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022862501, 2023-07-02T14:06:52.067 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.067 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.067 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:06:52.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:06:52.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:52.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763803+0000", 2023-07-02T14:06:52.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052067220000000004 2023-07-02T14:06:52.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769010+0000", 2023-07-02T14:06:52.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3249e-05 2023-07-02T14:06:52.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769023+0000", 2023-07-02T14:06:52.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1790000000000008e-06 2023-07-02T14:06:52.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769031+0000", 2023-07-02T14:06:52.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030611079999999999 2023-07-02T14:06:52.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.772092+0000", 2023-07-02T14:06:52.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3958e-05 2023-07-02T14:06:52.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.772116+0000", 2023-07-02T14:06:52.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014504064000000001 2023-07-02T14:06:52.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.786620+0000", 2023-07-02T14:06:52.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5952999999999999e-05 2023-07-02T14:06:52.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.786666+0000", 2023-07-02T14:06:52.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022862501 2023-07-02T14:06:52.074 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.074 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.074 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.074 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22335 2.0 e197/193 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 197'5602, mlcod=197'5599)", 2023-07-02T14:06:52.075 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:06:52.075 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.119888157999998, 2023-07-02T14:06:52.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024437743000000001, 2023-07-02T14:06:52.075 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.075 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.075 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:06:52.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:06:52.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:52.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790873+0000", 2023-07-02T14:06:52.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081672399999999992 2023-07-02T14:06:52.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799040+0000", 2023-07-02T14:06:52.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5712999999999999e-05 2023-07-02T14:06:52.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799066+0000", 2023-07-02T14:06:52.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1413999999999999e-05 2023-07-02T14:06:52.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799077+0000", 2023-07-02T14:06:52.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3102000000000007e-05 2023-07-02T14:06:52.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799140+0000", 2023-07-02T14:06:52.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.883e-05 2023-07-02T14:06:52.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799169+0000", 2023-07-02T14:06:52.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016086700999999998 2023-07-02T14:06:52.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.815256+0000", 2023-07-02T14:06:52.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5967000000000001e-05 2023-07-02T14:06:52.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.815312+0000", 2023-07-02T14:06:52.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024437743000000001 2023-07-02T14:06:52.082 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.082 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.082 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.083 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22336 2.0 e197/193 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 197'5603, mlcod=197'5599)", 2023-07-02T14:06:52.083 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:06:52.083 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 17.111652872000001, 2023-07-02T14:06:52.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022713011000000002, 2023-07-02T14:06:52.083 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.083 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.083 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:06:52.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:06:52.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:52.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799108+0000", 2023-07-02T14:06:52.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00058943600000000004 2023-07-02T14:06:52.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799698+0000", 2023-07-02T14:06:52.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6205000000000002e-05 2023-07-02T14:06:52.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799714+0000", 2023-07-02T14:06:52.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9630000000000001e-06 2023-07-02T14:06:52.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799721+0000", 2023-07-02T14:06:52.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076846579999999996 2023-07-02T14:06:52.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.807405+0000", 2023-07-02T14:06:52.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8191000000000002e-05 2023-07-02T14:06:52.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.807434+0000", 2023-07-02T14:06:52.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014338020999999999 2023-07-02T14:06:52.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.821772+0000", 2023-07-02T14:06:52.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0868999999999997e-05 2023-07-02T14:06:52.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.090 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.821822+0000", 2023-07-02T14:06:52.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022713011000000002 2023-07-02T14:06:52.090 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.090 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.090 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.090 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.091 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.091 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:52.092 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:52.092+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:52.092 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:06:52.123 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:06:52.166 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:06:52.173 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 2 2023-07-02T14:06:52.192 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:52.192+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:52.267 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:52.267 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:52.267 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:06:52.267 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:52.277 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:06:52.293 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:52.294+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:52.394 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:52.393+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:52.443 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:52.443 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:52.443 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:52.443 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:52.444 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:52.453 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:06:52.495 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:52.494+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:52.596 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:52.596+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:52.618 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:52.618 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:52.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:52.619 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:52.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.619 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:52.619 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:06:52.619 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 253.32556136100001, 2023-07-02T14:06:52.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:06:52.620 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.620 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.620 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.620 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.620 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.620 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:06:52.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.621 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:06:52.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:06:52.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:52.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:06:52.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:06:52.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:06:52.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:06:52.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:06:52.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:06:52.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:06:52.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:06:52.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.625 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:06:52.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:06:52.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.626 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:06:52.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:06:52.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:52.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:06:52.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:06:52.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:52.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:06:52.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:06:52.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:06:52.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:06:52.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:06:52.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:06:52.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:06:52.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:06:52.630 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.630 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.630 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.630 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:52.631 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:06:52.631 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 253.325326338, 2023-07-02T14:06:52.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:06:52.631 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.631 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.631 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.631 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.632 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.632 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:06:52.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.632 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:06:52.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:06:52.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:52.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:06:52.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:06:52.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:06:52.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:06:52.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:06:52.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:06:52.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:06:52.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:06:52.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:06:52.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:06:52.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:06:52.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:06:52.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:52.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:06:52.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:06:52.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:52.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:06:52.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:06:52.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:06:52.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:06:52.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:06:52.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:06:52.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:06:52.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:06:52.642 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.642 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.642 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.643 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:52.643 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:06:52.643 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 253.32512720400001, 2023-07-02T14:06:52.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:06:52.643 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.643 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.644 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.644 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.644 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.644 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:06:52.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.644 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:06:52.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:06:52.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:52.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:06:52.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:06:52.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:06:52.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:06:52.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:06:52.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:06:52.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:06:52.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:06:52.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:06:52.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:06:52.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:06:52.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:06:52.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:52.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:06:52.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:06:52.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:52.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:06:52.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:06:52.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:06:52.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:06:52.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:06:52.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:06:52.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:06:52.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:06:52.654 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.654 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.654 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.654 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:52.654 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:06:52.655 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 253.32476208, 2023-07-02T14:06:52.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:06:52.655 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.655 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.655 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.655 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.655 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.656 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:06:52.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.656 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:06:52.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:06:52.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:52.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:06:52.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:06:52.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:06:52.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:06:52.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:06:52.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:06:52.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:06:52.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:06:52.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:06:52.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:06:52.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:06:52.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:06:52.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:52.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:06:52.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:06:52.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:52.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:06:52.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:06:52.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:06:52.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:06:52.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:06:52.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:06:52.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:06:52.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:06:52.665 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.665 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.665 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.666 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:52.666 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:06:52.666 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 253.324681019, 2023-07-02T14:06:52.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:06:52.666 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.666 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.667 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.667 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.667 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.667 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:06:52.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.667 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:06:52.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:06:52.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:52.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:06:52.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:06:52.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:06:52.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:06:52.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:06:52.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:06:52.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:06:52.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:06:52.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:06:52.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:06:52.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:06:52.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:06:52.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:52.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:06:52.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:06:52.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:52.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:06:52.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:06:52.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:06:52.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:06:52.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:06:52.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:06:52.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:06:52.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:06:52.676 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.677 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.677 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.677 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:52.677 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:06:52.677 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 253.32458934799999, 2023-07-02T14:06:52.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:06:52.678 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.678 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.678 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.678 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.678 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.678 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:06:52.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.679 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:06:52.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:06:52.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:52.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:06:52.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:06:52.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:06:52.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:06:52.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:06:52.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:06:52.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:06:52.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:06:52.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:06:52.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:06:52.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:06:52.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:06:52.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:52.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:06:52.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:06:52.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:06:52.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:06:52.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:52.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:06:52.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:06:52.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:06:52.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:06:52.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:06:52.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:06:52.688 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.688 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.688 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.689 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:52.689 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:06:52.689 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 253.32424990600001, 2023-07-02T14:06:52.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:06:52.689 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.689 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.689 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.690 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.690 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.690 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:06:52.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.690 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:06:52.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:06:52.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:52.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:06:52.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:06:52.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:06:52.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:06:52.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:06:52.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:06:52.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:06:52.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:06:52.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:06:52.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:06:52.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:06:52.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:06:52.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:52.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:06:52.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:06:52.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:06:52.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:06:52.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:52.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:06:52.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:06:52.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:06:52.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:06:52.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:06:52.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:06:52.699 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.699 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.700 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.700 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:52.700 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:52.700 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 253.32406385300001, 2023-07-02T14:06:52.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:06:52.701 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.701 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.701 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.701 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.701 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.701 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:06:52.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.702 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:52.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:52.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:52.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:06:52.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:06:52.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:06:52.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:06:52.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:06:52.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:06:52.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:06:52.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:06:52.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:06:52.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:06:52.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:06:52.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:06:52.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:52.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:06:52.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:06:52.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:06:52.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:06:52.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:52.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:06:52.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:06:52.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:06:52.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:06:52.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:06:52.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:06:52.711 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.711 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.711 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.711 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:52.711 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:06:52.712 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 253.32390605099999, 2023-07-02T14:06:52.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:06:52.712 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.712 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.712 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.712 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.712 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.713 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:06:52.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.713 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:06:52.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:06:52.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:52.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:06:52.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:06:52.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:06:52.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:06:52.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:06:52.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:06:52.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:06:52.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:06:52.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:06:52.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:06:52.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:06:52.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:06:52.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:52.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:06:52.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:06:52.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:06:52.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:06:52.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:52.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:06:52.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:06:52.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:06:52.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:06:52.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:06:52.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:06:52.722 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.722 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.722 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.723 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:52.723 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:06:52.723 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 253.32371426899999, 2023-07-02T14:06:52.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:06:52.723 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.723 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.723 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.724 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.724 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.724 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:06:52.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.724 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:06:52.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:06:52.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:52.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:06:52.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:06:52.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:06:52.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:06:52.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:06:52.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:06:52.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:06:52.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:06:52.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:06:52.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:06:52.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:06:52.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:06:52.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:52.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:06:52.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:06:52.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:52.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:06:52.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:06:52.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:06:52.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:06:52.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:06:52.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:06:52.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:06:52.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:06:52.733 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.733 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.734 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.734 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:52.734 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:52.734 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 253.323538609, 2023-07-02T14:06:52.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:06:52.735 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.735 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.735 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.735 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.735 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.735 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:06:52.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.736 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:52.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:52.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:52.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:06:52.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:06:52.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:06:52.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:06:52.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:06:52.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:06:52.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:06:52.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:06:52.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:06:52.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:06:52.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:06:52.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:06:52.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:52.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:06:52.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:06:52.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:52.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:06:52.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:06:52.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:06:52.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:06:52.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:06:52.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:06:52.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:06:52.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:06:52.745 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.745 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.745 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.745 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:52.745 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:06:52.746 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 253.323444421, 2023-07-02T14:06:52.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:06:52.746 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.746 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.746 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.746 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.746 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.747 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:06:52.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.747 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:06:52.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:06:52.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:52.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:06:52.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:06:52.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:06:52.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:06:52.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:06:52.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:06:52.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:06:52.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:06:52.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:06:52.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:06:52.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:06:52.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:06:52.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:52.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:06:52.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:06:52.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:52.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:06:52.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:06:52.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:06:52.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:06:52.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:06:52.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:06:52.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:06:52.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:06:52.756 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.756 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.756 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.757 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:52.757 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:06:52.757 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 253.323241035, 2023-07-02T14:06:52.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:06:52.757 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.757 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.757 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.758 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.758 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.758 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:06:52.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.758 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:06:52.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:06:52.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:52.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:06:52.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:06:52.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:06:52.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:06:52.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:06:52.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:06:52.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:06:52.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:06:52.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:06:52.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:06:52.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:06:52.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:06:52.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:52.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:06:52.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:06:52.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:52.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:06:52.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:06:52.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:06:52.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:06:52.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:06:52.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:06:52.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:06:52.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:06:52.767 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.767 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.768 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.768 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:52.768 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:06:52.768 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 253.32309037900001, 2023-07-02T14:06:52.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:06:52.769 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.769 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.769 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.769 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.769 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.769 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:06:52.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.770 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:06:52.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:06:52.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:52.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:06:52.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:06:52.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:06:52.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:06:52.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:06:52.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:06:52.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:06:52.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:06:52.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:06:52.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:06:52.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:06:52.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:06:52.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:52.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:06:52.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:06:52.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:52.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:06:52.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:06:52.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:06:52.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:06:52.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:06:52.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:06:52.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:06:52.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:06:52.779 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.779 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.779 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.779 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:52.780 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:52.780 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 253.32301767600001, 2023-07-02T14:06:52.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:06:52.780 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.780 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.780 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.780 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.781 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.781 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:06:52.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.781 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:52.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:52.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:52.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:06:52.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:06:52.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:06:52.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:06:52.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:06:52.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:06:52.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:06:52.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:06:52.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:06:52.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:06:52.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:06:52.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:06:52.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:52.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:06:52.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:06:52.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:52.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:06:52.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:06:52.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:06:52.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:06:52.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:06:52.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:06:52.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:06:52.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:06:52.790 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.790 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.790 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.791 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:52.791 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:52.791 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 253.322900686, 2023-07-02T14:06:52.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:06:52.791 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.792 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.792 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.792 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.792 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.792 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:06:52.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.792 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:52.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:52.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:52.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:06:52.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:06:52.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:06:52.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:06:52.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:06:52.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:06:52.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.796 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:06:52.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:06:52.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:06:52.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:06:52.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:06:52.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:06:52.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:52.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:06:52.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:06:52.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:52.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:06:52.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:06:52.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:06:52.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:06:52.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:06:52.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:06:52.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:06:52.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:06:52.802 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.802 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.802 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.802 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:52.802 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:06:52.802 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 253.322751235, 2023-07-02T14:06:52.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:06:52.803 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.803 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.803 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.803 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.803 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.803 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:06:52.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.804 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:06:52.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:06:52.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:52.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:06:52.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:06:52.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:06:52.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:06:52.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:06:52.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:06:52.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:06:52.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:06:52.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:06:52.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:06:52.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:06:52.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:06:52.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:52.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:06:52.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:06:52.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:52.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:06:52.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:06:52.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:06:52.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:06:52.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:06:52.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:06:52.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:06:52.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:06:52.813 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.813 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.813 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.814 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:52.814 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:06:52.814 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 253.30127526199999, 2023-07-02T14:06:52.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:06:52.814 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.814 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.814 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.815 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.815 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.815 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:06:52.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.815 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:06:52.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:06:52.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:52.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:06:52.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:06:52.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:06:52.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:06:52.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:06:52.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:06:52.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:06:52.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:06:52.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:06:52.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:06:52.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:06:52.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:06:52.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:52.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:06:52.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:06:52.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:06:52.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:06:52.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:52.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:06:52.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:06:52.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:06:52.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:06:52.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:06:52.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:06:52.824 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.824 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.824 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.825 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:52.825 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:06:52.825 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 253.30114272599999, 2023-07-02T14:06:52.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:06:52.825 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.825 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.826 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.826 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.826 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.826 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:06:52.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.826 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:06:52.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:06:52.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:52.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:06:52.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:06:52.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:06:52.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:06:52.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:06:52.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:06:52.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:06:52.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:06:52.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:06:52.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:06:52.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:06:52.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:06:52.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:52.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:06:52.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:06:52.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:06:52.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:06:52.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:52.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:06:52.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:06:52.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:06:52.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:06:52.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:06:52.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:06:52.835 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.836 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.836 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.836 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:06:52.836 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:06:52.836 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 253.23980606500001, 2023-07-02T14:06:52.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:06:52.837 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:52.837 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:52.837 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:52.837 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:52.837 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:52.837 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:06:52.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.838 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:52.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:52.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:06:52.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:52.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:52.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:06:52.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:52.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:52.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:06:52.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:06:52.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:52.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:06:52.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:06:52.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:52.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:06:52.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:06:52.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:52.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:06:52.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:06:52.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:52.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:06:52.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:06:52.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:52.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:06:52.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:06:52.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:06:52.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:06:52.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:06:52.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:52.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:06:52.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:06:52.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:52.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:06:52.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:06:52.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:52.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:06:52.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:06:52.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:52.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:52.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:52.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:06:52.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:06:52.847 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.847 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.847 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.847 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:52.847 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:52.847 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:52.849 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:06:52.849 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:52.848+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:52.949 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:52.948+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:53.013 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:53.014 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:53.014 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:06:53.014 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:53.023 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:06:53.050 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:53.050+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:53.151 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:53.150+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:53.188 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:53.188 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:53.188 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:53.188 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:53.188 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:53.198 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:06:53.252 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:53.252+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:53.352 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:53.352+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:53.366 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:53.366 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:53.366 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:53.366 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:53.366 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.366 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17354 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:53.367 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:53.367 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 70.059993294999998, 2023-07-02T14:06:53.367 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493, 2023-07-02T14:06:53.367 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:53.367 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:53.367 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:53.367 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:53.368 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:53.368 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17354 2023-07-02T14:06:53.368 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.368 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:53.368 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.368 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:53.368 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:53.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:53.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.369 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:53.369 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:53.369 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:53.369 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.369 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:53.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:53.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7440000000000001e-06 2023-07-02T14:06:53.370 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.370 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.370 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:53.370 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302606+0000", 2023-07-02T14:06:53.370 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:06:53.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:53.371 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302613+0000", 2023-07-02T14:06:53.371 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8810000000000004e-06 2023-07-02T14:06:53.371 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.371 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.371 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:53.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302618+0000", 2023-07-02T14:06:53.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028613100000000002 2023-07-02T14:06:53.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.372 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.372 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.372 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302904+0000", 2023-07-02T14:06:53.372 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5278999999999995e-05 2023-07-02T14:06:53.372 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.373 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302979+0000", 2023-07-02T14:06:53.373 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096904750000000005 2023-07-02T14:06:53.373 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.373 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.373 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312669+0000", 2023-07-02T14:06:53.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0328000000000002e-05 2023-07-02T14:06:53.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.374 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.374 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.374 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312700+0000", 2023-07-02T14:06:53.374 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098019319999999993 2023-07-02T14:06:53.374 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322502+0000", 2023-07-02T14:06:53.375 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4669999999999998e-05 2023-07-02T14:06:53.375 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.375 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.375 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.375 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322536+0000", 2023-07-02T14:06:53.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010023839 2023-07-02T14:06:53.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.376 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.376 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332560+0000", 2023-07-02T14:06:53.376 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5559999999999998e-05 2023-07-02T14:06:53.376 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.376 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332596+0000", 2023-07-02T14:06:53.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090401319999999993 2023-07-02T14:06:53.377 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.377 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.377 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.377 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341636+0000", 2023-07-02T14:06:53.377 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5373000000000001e-05 2023-07-02T14:06:53.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.378 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341681+0000", 2023-07-02T14:06:53.378 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081994059999999994 2023-07-02T14:06:53.378 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.378 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.378 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349881+0000", 2023-07-02T14:06:53.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4319000000000001e-05 2023-07-02T14:06:53.379 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.379 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.379 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.379 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349915+0000", 2023-07-02T14:06:53.379 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089388339999999997 2023-07-02T14:06:53.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.380 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358854+0000", 2023-07-02T14:06:53.380 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4319000000000005e-05 2023-07-02T14:06:53.380 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.380 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.380 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358928+0000", 2023-07-02T14:06:53.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080234690000000001 2023-07-02T14:06:53.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.381 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.381 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.381 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366952+0000", 2023-07-02T14:06:53.381 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2082999999999998e-05 2023-07-02T14:06:53.381 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366984+0000", 2023-07-02T14:06:53.382 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075015790000000004 2023-07-02T14:06:53.382 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.382 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.382 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.382 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374485+0000", 2023-07-02T14:06:53.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4480999999999995e-05 2023-07-02T14:06:53.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.383 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.383 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.383 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374550+0000", 2023-07-02T14:06:53.383 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091666620000000008 2023-07-02T14:06:53.383 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383716+0000", 2023-07-02T14:06:53.384 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9318000000000003e-05 2023-07-02T14:06:53.384 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.384 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.384 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.384 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383786+0000", 2023-07-02T14:06:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013434115 2023-07-02T14:06:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397220+0000", 2023-07-02T14:06:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4573999999999997e-05 2023-07-02T14:06:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397264+0000", 2023-07-02T14:06:53.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064824649999999998 2023-07-02T14:06:53.386 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.386 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.386 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.386 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403747+0000", 2023-07-02T14:06:53.386 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1087999999999993e-05 2023-07-02T14:06:53.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.387 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403828+0000", 2023-07-02T14:06:53.387 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072854449999999998 2023-07-02T14:06:53.387 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.387 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.387 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411113+0000", 2023-07-02T14:06:53.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5120000000000001e-05 2023-07-02T14:06:53.388 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.388 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.388 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.388 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411169+0000", 2023-07-02T14:06:53.388 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007211505 2023-07-02T14:06:53.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.389 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418380+0000", 2023-07-02T14:06:53.389 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011289399999999999 2023-07-02T14:06:53.389 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.389 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.389 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:53.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418493+0000", 2023-07-02T14:06:53.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00051217500000000002 2023-07-02T14:06:53.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.390 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:53.390 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.419005+0000", 2023-07-02T14:06:53.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00327393 2023-07-02T14:06:53.390 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:53.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422279+0000", 2023-07-02T14:06:53.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041432350000000003 2023-07-02T14:06:53.391 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.391 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.391 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:53.391 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426422+0000", 2023-07-02T14:06:53.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7021000000000003e-05 2023-07-02T14:06:53.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:53.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426469+0000", 2023-07-02T14:06:53.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000190256 2023-07-02T14:06:53.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:53.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426660+0000", 2023-07-02T14:06:53.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493 2023-07-02T14:06:53.393 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.393 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.393 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.394 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17355 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:53.394 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:53.394 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 70.059974634, 2023-07-02T14:06:53.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001, 2023-07-02T14:06:53.394 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:53.394 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:53.394 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:53.395 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:53.395 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:53.395 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17355 2023-07-02T14:06:53.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.395 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:53.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:53.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:53.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:53.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:53.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:53.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:53.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:53.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302621+0000", 2023-07-02T14:06:53.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2260000000000001e-06 2023-07-02T14:06:53.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:53.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302625+0000", 2023-07-02T14:06:53.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8599999999999998e-06 2023-07-02T14:06:53.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:53.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302630+0000", 2023-07-02T14:06:53.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0629999999999999e-06 2023-07-02T14:06:53.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:53.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302635+0000", 2023-07-02T14:06:53.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035731199999999998 2023-07-02T14:06:53.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.399 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.399 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302992+0000", 2023-07-02T14:06:53.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4164e-05 2023-07-02T14:06:53.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.400 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.303026+0000", 2023-07-02T14:06:53.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097115900000000008 2023-07-02T14:06:53.400 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312738+0000", 2023-07-02T14:06:53.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1701999999999998e-05 2023-07-02T14:06:53.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312819+0000", 2023-07-02T14:06:53.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097363640000000008 2023-07-02T14:06:53.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322556+0000", 2023-07-02T14:06:53.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2517000000000001e-05 2023-07-02T14:06:53.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322588+0000", 2023-07-02T14:06:53.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01002393 2023-07-02T14:06:53.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332612+0000", 2023-07-02T14:06:53.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2960999999999998e-05 2023-07-02T14:06:53.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.404 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332645+0000", 2023-07-02T14:06:53.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090669819999999995 2023-07-02T14:06:53.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.404 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341712+0000", 2023-07-02T14:06:53.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9288000000000002e-05 2023-07-02T14:06:53.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.405 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341811+0000", 2023-07-02T14:06:53.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081504460000000004 2023-07-02T14:06:53.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349962+0000", 2023-07-02T14:06:53.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107321 2023-07-02T14:06:53.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.350069+0000", 2023-07-02T14:06:53.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088790999999999991 2023-07-02T14:06:53.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358948+0000", 2023-07-02T14:06:53.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1711000000000002e-05 2023-07-02T14:06:53.407 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358980+0000", 2023-07-02T14:06:53.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080367579999999997 2023-07-02T14:06:53.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.408 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.408 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367017+0000", 2023-07-02T14:06:53.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6329000000000002e-05 2023-07-02T14:06:53.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367073+0000", 2023-07-02T14:06:53.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075097300000000001 2023-07-02T14:06:53.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374583+0000", 2023-07-02T14:06:53.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5710999999999995e-05 2023-07-02T14:06:53.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374648+0000", 2023-07-02T14:06:53.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009165912 2023-07-02T14:06:53.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383814+0000", 2023-07-02T14:06:53.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3916000000000003e-05 2023-07-02T14:06:53.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383878+0000", 2023-07-02T14:06:53.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013440532999999999 2023-07-02T14:06:53.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397319+0000", 2023-07-02T14:06:53.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7218999999999995e-05 2023-07-02T14:06:53.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397386+0000", 2023-07-02T14:06:53.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065165279999999997 2023-07-02T14:06:53.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403903+0000", 2023-07-02T14:06:53.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8152000000000005e-05 2023-07-02T14:06:53.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404001+0000", 2023-07-02T14:06:53.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072185360000000002 2023-07-02T14:06:53.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411219+0000", 2023-07-02T14:06:53.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:06:53.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411310+0000", 2023-07-02T14:06:53.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010021663 2023-07-02T14:06:53.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421331+0000", 2023-07-02T14:06:53.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0148000000000006e-05 2023-07-02T14:06:53.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.416 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421411+0000", 2023-07-02T14:06:53.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053056170000000003 2023-07-02T14:06:53.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.417 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.417 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.417 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426717+0000", 2023-07-02T14:06:53.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8242000000000001e-05 2023-07-02T14:06:53.417 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:53.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426795+0000", 2023-07-02T14:06:53.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00041308399999999998 2023-07-02T14:06:53.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:53.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.427208+0000", 2023-07-02T14:06:53.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028330849999999999 2023-07-02T14:06:53.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:53.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430041+0000", 2023-07-02T14:06:53.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00072642300000000002 2023-07-02T14:06:53.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:53.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430768+0000", 2023-07-02T14:06:53.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1847000000000002e-05 2023-07-02T14:06:53.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:53.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430800+0000", 2023-07-02T14:06:53.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9308999999999994e-05 2023-07-02T14:06:53.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:53.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430899+0000", 2023-07-02T14:06:53.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001 2023-07-02T14:06:53.421 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.421 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.422 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.422 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:410 2.5 2:bbaed619:::rbd_data.125de1449605.0000000000000109:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:06:53.422 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:53.422 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 56.912029369000003, 2023-07-02T14:06:53.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003, 2023-07-02T14:06:53.422 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:53.423 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:53.423 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:53.423 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:06:53.423 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:06:53.423 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:06:53.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.423 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:53.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:53.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:53.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:53.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:53.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:53.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:53.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.425 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:53.425 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:53.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.635e-06 2023-07-02T14:06:53.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:53.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450568+0000", 2023-07-02T14:06:53.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5970000000000002e-06 2023-07-02T14:06:53.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:53.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450573+0000", 2023-07-02T14:06:53.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3379999999999999e-06 2023-07-02T14:06:53.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:53.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450575+0000", 2023-07-02T14:06:53.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9458000000000001e-05 2023-07-02T14:06:53.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450615+0000", 2023-07-02T14:06:53.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0569999999999993e-06 2023-07-02T14:06:53.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:53.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450623+0000", 2023-07-02T14:06:53.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.63968709199999996 2023-07-02T14:06:53.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090310+0000", 2023-07-02T14:06:53.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020157399999999999 2023-07-02T14:06:53.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:53.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090512+0000", 2023-07-02T14:06:53.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027739190000000001 2023-07-02T14:06:53.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:53.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093285+0000", 2023-07-02T14:06:53.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003 2023-07-02T14:06:53.430 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.431 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.431 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.431 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:416 2.5 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:06:53.431 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:53.431 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 56.911999936000001, 2023-07-02T14:06:53.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002, 2023-07-02T14:06:53.432 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:53.432 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:53.432 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:53.432 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:06:53.432 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:06:53.432 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:06:53.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.432 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:53.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:53.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:53.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:53.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.433 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:53.433 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:53.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:53.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:53.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:53.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.34e-06 2023-07-02T14:06:53.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:53.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450599+0000", 2023-07-02T14:06:53.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1549999999999998e-06 2023-07-02T14:06:53.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:53.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450604+0000", 2023-07-02T14:06:53.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:06:53.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:53.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450607+0000", 2023-07-02T14:06:53.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5717999999999999e-05 2023-07-02T14:06:53.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450643+0000", 2023-07-02T14:06:53.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1619999999999999e-06 2023-07-02T14:06:53.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:53.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450648+0000", 2023-07-02T14:06:53.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64292291800000001 2023-07-02T14:06:53.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093571+0000", 2023-07-02T14:06:53.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115628 2023-07-02T14:06:53.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:53.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093687+0000", 2023-07-02T14:06:53.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002885552 2023-07-02T14:06:53.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:53.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.096572+0000", 2023-07-02T14:06:53.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002 2023-07-02T14:06:53.440 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.440 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.440 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.440 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18557 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:53.440 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:53.440 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 48.113627557000001, 2023-07-02T14:06:53.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13120991400000001, 2023-07-02T14:06:53.441 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:53.441 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:53.441 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:53.441 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:53.441 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:53.441 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18557 2023-07-02T14:06:53.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.442 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:53.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:53.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:53.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:53.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:53.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:53.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:06:53.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:53.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:53.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8870000000000001e-06 2023-07-02T14:06:53.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:53.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248971+0000", 2023-07-02T14:06:53.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9009999999999996e-06 2023-07-02T14:06:53.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:53.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248974+0000", 2023-07-02T14:06:53.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9809999999999997e-06 2023-07-02T14:06:53.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:53.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248978+0000", 2023-07-02T14:06:53.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00076989199999999999 2023-07-02T14:06:53.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249748+0000", 2023-07-02T14:06:53.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8245999999999999e-05 2023-07-02T14:06:53.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249787+0000", 2023-07-02T14:06:53.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066332969999999998 2023-07-02T14:06:53.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256420+0000", 2023-07-02T14:06:53.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2566e-05 2023-07-02T14:06:53.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256452+0000", 2023-07-02T14:06:53.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012364714000000001 2023-07-02T14:06:53.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268817+0000", 2023-07-02T14:06:53.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5113000000000001e-05 2023-07-02T14:06:53.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268852+0000", 2023-07-02T14:06:53.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010993899999999999 2023-07-02T14:06:53.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279846+0000", 2023-07-02T14:06:53.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2144999999999998e-05 2023-07-02T14:06:53.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279878+0000", 2023-07-02T14:06:53.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011778958000000001 2023-07-02T14:06:53.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291657+0000", 2023-07-02T14:06:53.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9592000000000001e-05 2023-07-02T14:06:53.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291687+0000", 2023-07-02T14:06:53.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081107220000000008 2023-07-02T14:06:53.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299798+0000", 2023-07-02T14:06:53.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0290999999999998e-05 2023-07-02T14:06:53.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299828+0000", 2023-07-02T14:06:53.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071584489999999999 2023-07-02T14:06:53.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306986+0000", 2023-07-02T14:06:53.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0755999999999999e-05 2023-07-02T14:06:53.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307047+0000", 2023-07-02T14:06:53.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057110260000000001 2023-07-02T14:06:53.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312758+0000", 2023-07-02T14:06:53.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5145999999999999e-05 2023-07-02T14:06:53.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312793+0000", 2023-07-02T14:06:53.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004758361 2023-07-02T14:06:53.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317552+0000", 2023-07-02T14:06:53.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9190000000000002e-05 2023-07-02T14:06:53.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317601+0000", 2023-07-02T14:06:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050329420000000003 2023-07-02T14:06:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322634+0000", 2023-07-02T14:06:53.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5737999999999998e-05 2023-07-02T14:06:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322719+0000", 2023-07-02T14:06:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064995679999999998 2023-07-02T14:06:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329219+0000", 2023-07-02T14:06:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0908000000000003e-05 2023-07-02T14:06:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329250+0000", 2023-07-02T14:06:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062072029999999997 2023-07-02T14:06:53.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335457+0000", 2023-07-02T14:06:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9471999999999998e-05 2023-07-02T14:06:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335487+0000", 2023-07-02T14:06:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097780239999999997 2023-07-02T14:06:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345265+0000", 2023-07-02T14:06:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0982999999999994e-05 2023-07-02T14:06:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345336+0000", 2023-07-02T14:06:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046378799999999996 2023-07-02T14:06:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349973+0000", 2023-07-02T14:06:53.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1069999999999999e-05 2023-07-02T14:06:53.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350005+0000", 2023-07-02T14:06:53.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042766870000000004 2023-07-02T14:06:53.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354281+0000", 2023-07-02T14:06:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1155000000000002e-05 2023-07-02T14:06:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354312+0000", 2023-07-02T14:06:53.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071436820000000002 2023-07-02T14:06:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361456+0000", 2023-07-02T14:06:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9054000000000002e-05 2023-07-02T14:06:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361505+0000", 2023-07-02T14:06:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072301320000000002 2023-07-02T14:06:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368735+0000", 2023-07-02T14:06:53.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1013999999999994e-05 2023-07-02T14:06:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368816+0000", 2023-07-02T14:06:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036582490000000001 2023-07-02T14:06:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372475+0000", 2023-07-02T14:06:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4652000000000004e-05 2023-07-02T14:06:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372539+0000", 2023-07-02T14:06:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00031374199999999999 2023-07-02T14:06:53.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:53.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372853+0000", 2023-07-02T14:06:53.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030071299999999998 2023-07-02T14:06:53.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:53.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.375860+0000", 2023-07-02T14:06:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041897749999999997 2023-07-02T14:06:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380050+0000", 2023-07-02T14:06:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8331000000000003e-05 2023-07-02T14:06:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380088+0000", 2023-07-02T14:06:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0626999999999998e-05 2023-07-02T14:06:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380179+0000", 2023-07-02T14:06:53.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13120991400000001 2023-07-02T14:06:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18558 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:53.472 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 48.113610854999997, 2023-07-02T14:06:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13755567899999999, 2023-07-02T14:06:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:53.473 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18558 2023-07-02T14:06:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:53.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:53.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:53.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:53.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248985+0000", 2023-07-02T14:06:53.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8630000000000001e-06 2023-07-02T14:06:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248989+0000", 2023-07-02T14:06:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0750000000000003e-06 2023-07-02T14:06:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248995+0000", 2023-07-02T14:06:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3290000000000004e-06 2023-07-02T14:06:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248999+0000", 2023-07-02T14:06:53.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00080656300000000005 2023-07-02T14:06:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249806+0000", 2023-07-02T14:06:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2040999999999999e-05 2023-07-02T14:06:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249848+0000", 2023-07-02T14:06:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066312419999999999 2023-07-02T14:06:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256479+0000", 2023-07-02T14:06:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3348e-05 2023-07-02T14:06:53.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256513+0000", 2023-07-02T14:06:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012391531000000001 2023-07-02T14:06:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268904+0000", 2023-07-02T14:06:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.347e-05 2023-07-02T14:06:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268978+0000", 2023-07-02T14:06:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010929772000000001 2023-07-02T14:06:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279907+0000", 2023-07-02T14:06:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2669999999999997e-05 2023-07-02T14:06:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279940+0000", 2023-07-02T14:06:53.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011770332999999999 2023-07-02T14:06:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291710+0000", 2023-07-02T14:06:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1207000000000001e-05 2023-07-02T14:06:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291742+0000", 2023-07-02T14:06:53.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081242199999999997 2023-07-02T14:06:53.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299866+0000", 2023-07-02T14:06:53.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4650000000000002e-05 2023-07-02T14:06:53.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299900+0000", 2023-07-02T14:06:53.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071810270000000004 2023-07-02T14:06:53.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307081+0000", 2023-07-02T14:06:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4378000000000002e-05 2023-07-02T14:06:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307116+0000", 2023-07-02T14:06:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057031909999999998 2023-07-02T14:06:53.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312819+0000", 2023-07-02T14:06:53.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1612999999999997e-05 2023-07-02T14:06:53.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312851+0000", 2023-07-02T14:06:53.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048172470000000002 2023-07-02T14:06:53.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317668+0000", 2023-07-02T14:06:53.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3328e-05 2023-07-02T14:06:53.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317741+0000", 2023-07-02T14:06:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050117290000000004 2023-07-02T14:06:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322753+0000", 2023-07-02T14:06:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9872000000000001e-05 2023-07-02T14:06:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322783+0000", 2023-07-02T14:06:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064910610000000002 2023-07-02T14:06:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329274+0000", 2023-07-02T14:06:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0097e-05 2023-07-02T14:06:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329304+0000", 2023-07-02T14:06:53.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062163879999999998 2023-07-02T14:06:53.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335520+0000", 2023-07-02T14:06:53.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9513e-05 2023-07-02T14:06:53.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335550+0000", 2023-07-02T14:06:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098209969999999997 2023-07-02T14:06:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345371+0000", 2023-07-02T14:06:53.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9862999999999999e-05 2023-07-02T14:06:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345401+0000", 2023-07-02T14:06:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046552030000000001 2023-07-02T14:06:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350056+0000", 2023-07-02T14:06:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8751999999999993e-05 2023-07-02T14:06:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350145+0000", 2023-07-02T14:06:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041981450000000003 2023-07-02T14:06:53.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354343+0000", 2023-07-02T14:06:53.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1501000000000002e-05 2023-07-02T14:06:53.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354374+0000", 2023-07-02T14:06:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071551740000000003 2023-07-02T14:06:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361530+0000", 2023-07-02T14:06:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5156e-05 2023-07-02T14:06:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361565+0000", 2023-07-02T14:06:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072918569999999997 2023-07-02T14:06:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368857+0000", 2023-07-02T14:06:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5770999999999999e-05 2023-07-02T14:06:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368902+0000", 2023-07-02T14:06:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057210569999999999 2023-07-02T14:06:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374623+0000", 2023-07-02T14:06:53.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3266999999999998e-05 2023-07-02T14:06:53.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374667+0000", 2023-07-02T14:06:53.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055479570000000001 2023-07-02T14:06:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380215+0000", 2023-07-02T14:06:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0067e-05 2023-07-02T14:06:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380265+0000", 2023-07-02T14:06:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00043082299999999999 2023-07-02T14:06:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380696+0000", 2023-07-02T14:06:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024853240000000001 2023-07-02T14:06:53.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.383181+0000", 2023-07-02T14:06:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032601280000000002 2023-07-02T14:06:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:53.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386441+0000", 2023-07-02T14:06:53.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9748e-05 2023-07-02T14:06:53.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:53.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386471+0000", 2023-07-02T14:06:53.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0531000000000006e-05 2023-07-02T14:06:53.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:53.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386541+0000", 2023-07-02T14:06:53.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13755567899999999 2023-07-02T14:06:53.505 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.505 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.505 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.506 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18559 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:53.506 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:53.506 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 48.113588251000003, 2023-07-02T14:06:53.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.146184591, 2023-07-02T14:06:53.506 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:53.506 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:53.506 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:53.507 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:53.507 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:53.507 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18559 2023-07-02T14:06:53.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.507 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:53.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:53.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:53.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:53.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:53.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:53.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:53.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:53.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:53.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.726e-06 2023-07-02T14:06:53.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:53.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249011+0000", 2023-07-02T14:06:53.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7939999999999997e-06 2023-07-02T14:06:53.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:53.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249018+0000", 2023-07-02T14:06:53.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3470000000000001e-06 2023-07-02T14:06:53.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:53.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249023+0000", 2023-07-02T14:06:53.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00084402900000000005 2023-07-02T14:06:53.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249867+0000", 2023-07-02T14:06:53.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9050000000000001e-05 2023-07-02T14:06:53.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249906+0000", 2023-07-02T14:06:53.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066311039999999996 2023-07-02T14:06:53.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256537+0000", 2023-07-02T14:06:53.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2790000000000003e-05 2023-07-02T14:06:53.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256570+0000", 2023-07-02T14:06:53.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012436129000000001 2023-07-02T14:06:53.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269006+0000", 2023-07-02T14:06:53.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0173999999999999e-05 2023-07-02T14:06:53.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269046+0000", 2023-07-02T14:06:53.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010924698 2023-07-02T14:06:53.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279971+0000", 2023-07-02T14:06:53.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3831000000000002e-05 2023-07-02T14:06:53.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280004+0000", 2023-07-02T14:06:53.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011795912 2023-07-02T14:06:53.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291800+0000", 2023-07-02T14:06:53.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5613999999999997e-05 2023-07-02T14:06:53.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291886+0000", 2023-07-02T14:06:53.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080395999999999992 2023-07-02T14:06:53.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299925+0000", 2023-07-02T14:06:53.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0694999999999997e-05 2023-07-02T14:06:53.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299956+0000", 2023-07-02T14:06:53.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072049449999999999 2023-07-02T14:06:53.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307161+0000", 2023-07-02T14:06:53.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5523000000000001e-05 2023-07-02T14:06:53.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307247+0000", 2023-07-02T14:06:53.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056331929999999999 2023-07-02T14:06:53.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312880+0000", 2023-07-02T14:06:53.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1646000000000002e-05 2023-07-02T14:06:53.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312911+0000", 2023-07-02T14:06:53.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048627540000000004 2023-07-02T14:06:53.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317774+0000", 2023-07-02T14:06:53.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2008000000000003e-05 2023-07-02T14:06:53.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317806+0000", 2023-07-02T14:06:53.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050055689999999996 2023-07-02T14:06:53.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322812+0000", 2023-07-02T14:06:53.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3442999999999997e-05 2023-07-02T14:06:53.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322855+0000", 2023-07-02T14:06:53.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006472813 2023-07-02T14:06:53.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329328+0000", 2023-07-02T14:06:53.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0128e-05 2023-07-02T14:06:53.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329358+0000", 2023-07-02T14:06:53.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062156490000000002 2023-07-02T14:06:53.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335574+0000", 2023-07-02T14:06:53.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2357999999999997e-05 2023-07-02T14:06:53.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335606+0000", 2023-07-02T14:06:53.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098318409999999992 2023-07-02T14:06:53.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345438+0000", 2023-07-02T14:06:53.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1174000000000007e-05 2023-07-02T14:06:53.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345509+0000", 2023-07-02T14:06:53.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004669851 2023-07-02T14:06:53.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350179+0000", 2023-07-02T14:06:53.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1559000000000001e-05 2023-07-02T14:06:53.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350211+0000", 2023-07-02T14:06:53.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041865740000000002 2023-07-02T14:06:53.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354397+0000", 2023-07-02T14:06:53.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9850000000000001e-05 2023-07-02T14:06:53.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354427+0000", 2023-07-02T14:06:53.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071608449999999999 2023-07-02T14:06:53.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361588+0000", 2023-07-02T14:06:53.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8346000000000002e-05 2023-07-02T14:06:53.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361626+0000", 2023-07-02T14:06:53.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073175009999999997 2023-07-02T14:06:53.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368944+0000", 2023-07-02T14:06:53.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2879000000000005e-05 2023-07-02T14:06:53.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369017+0000", 2023-07-02T14:06:53.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056850690000000001 2023-07-02T14:06:53.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374702+0000", 2023-07-02T14:06:53.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7693e-05 2023-07-02T14:06:53.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374739+0000", 2023-07-02T14:06:53.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076925149999999996 2023-07-02T14:06:53.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382432+0000", 2023-07-02T14:06:53.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3259e-05 2023-07-02T14:06:53.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382475+0000", 2023-07-02T14:06:53.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041005110000000003 2023-07-02T14:06:53.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386576+0000", 2023-07-02T14:06:53.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2074999999999999e-05 2023-07-02T14:06:53.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:53.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386618+0000", 2023-07-02T14:06:53.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000233692 2023-07-02T14:06:53.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:53.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386851+0000", 2023-07-02T14:06:53.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024022269999999998 2023-07-02T14:06:53.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:53.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.389254+0000", 2023-07-02T14:06:53.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058506469999999996 2023-07-02T14:06:53.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:53.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395104+0000", 2023-07-02T14:06:53.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8807000000000003e-05 2023-07-02T14:06:53.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:53.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395143+0000", 2023-07-02T14:06:53.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9616000000000001e-05 2023-07-02T14:06:53.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:53.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395193+0000", 2023-07-02T14:06:53.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.146184591 2023-07-02T14:06:53.540 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.540 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.540 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.540 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18560 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:53.540 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:53.541 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 48.113569628, 2023-07-02T14:06:53.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.15647551900000001, 2023-07-02T14:06:53.541 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:53.541 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:53.541 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:53.541 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:53.541 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:53.541 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18560 2023-07-02T14:06:53.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.542 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:53.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:53.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:53.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:53.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:53.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:53.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:53.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:53.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249026+0000", 2023-07-02T14:06:53.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3419999999999999e-06 2023-07-02T14:06:53.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:53.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249030+0000", 2023-07-02T14:06:53.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7869999999999998e-06 2023-07-02T14:06:53.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:53.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249035+0000", 2023-07-02T14:06:53.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9639999999999997e-06 2023-07-02T14:06:53.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:53.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249039+0000", 2023-07-02T14:06:53.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00088477599999999999 2023-07-02T14:06:53.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249924+0000", 2023-07-02T14:06:53.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7333999999999999e-05 2023-07-02T14:06:53.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249962+0000", 2023-07-02T14:06:53.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066323409999999999 2023-07-02T14:06:53.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256594+0000", 2023-07-02T14:06:53.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2595999999999998e-05 2023-07-02T14:06:53.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256627+0000", 2023-07-02T14:06:53.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012449894 2023-07-02T14:06:53.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269076+0000", 2023-07-02T14:06:53.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3942999999999997e-05 2023-07-02T14:06:53.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269110+0000", 2023-07-02T14:06:53.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010923884 2023-07-02T14:06:53.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280034+0000", 2023-07-02T14:06:53.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7774000000000001e-05 2023-07-02T14:06:53.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280072+0000", 2023-07-02T14:06:53.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011850027000000001 2023-07-02T14:06:53.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291922+0000", 2023-07-02T14:06:53.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2128000000000001e-05 2023-07-02T14:06:53.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291954+0000", 2023-07-02T14:06:53.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080261150000000003 2023-07-02T14:06:53.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299980+0000", 2023-07-02T14:06:53.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6322999999999998e-05 2023-07-02T14:06:53.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300017+0000", 2023-07-02T14:06:53.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072683039999999997 2023-07-02T14:06:53.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307285+0000", 2023-07-02T14:06:53.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3309000000000002e-05 2023-07-02T14:06:53.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307318+0000", 2023-07-02T14:06:53.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056178649999999997 2023-07-02T14:06:53.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312936+0000", 2023-07-02T14:06:53.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1866000000000003e-05 2023-07-02T14:06:53.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312968+0000", 2023-07-02T14:06:53.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048789970000000004 2023-07-02T14:06:53.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317847+0000", 2023-07-02T14:06:53.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0223999999999996e-05 2023-07-02T14:06:53.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317927+0000", 2023-07-02T14:06:53.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049543649999999996 2023-07-02T14:06:53.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322882+0000", 2023-07-02T14:06:53.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1399000000000002e-05 2023-07-02T14:06:53.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322913+0000", 2023-07-02T14:06:53.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006534214 2023-07-02T14:06:53.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329447+0000", 2023-07-02T14:06:53.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9649999999999998e-05 2023-07-02T14:06:53.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329527+0000", 2023-07-02T14:06:53.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061026839999999997 2023-07-02T14:06:53.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335629+0000", 2023-07-02T14:06:53.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9611000000000002e-05 2023-07-02T14:06:53.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335659+0000", 2023-07-02T14:06:53.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098851549999999996 2023-07-02T14:06:53.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345544+0000", 2023-07-02T14:06:53.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0587999999999999e-05 2023-07-02T14:06:53.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345575+0000", 2023-07-02T14:06:53.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046790859999999998 2023-07-02T14:06:53.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350254+0000", 2023-07-02T14:06:53.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0284000000000002e-05 2023-07-02T14:06:53.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350304+0000", 2023-07-02T14:06:53.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041560540000000002 2023-07-02T14:06:53.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354460+0000", 2023-07-02T14:06:53.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2427000000000004e-05 2023-07-02T14:06:53.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354523+0000", 2023-07-02T14:06:53.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071268770000000002 2023-07-02T14:06:53.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361650+0000", 2023-07-02T14:06:53.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0026999999999999e-05 2023-07-02T14:06:53.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361680+0000", 2023-07-02T14:06:53.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073733080000000003 2023-07-02T14:06:53.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369053+0000", 2023-07-02T14:06:53.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7635999999999997e-05 2023-07-02T14:06:53.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369091+0000", 2023-07-02T14:06:53.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056908059999999996 2023-07-02T14:06:53.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374781+0000", 2023-07-02T14:06:53.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4656000000000002e-05 2023-07-02T14:06:53.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374816+0000", 2023-07-02T14:06:53.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077037809999999998 2023-07-02T14:06:53.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382520+0000", 2023-07-02T14:06:53.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7858e-05 2023-07-02T14:06:53.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382588+0000", 2023-07-02T14:06:53.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057884019999999998 2023-07-02T14:06:53.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388376+0000", 2023-07-02T14:06:53.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4831000000000001e-05 2023-07-02T14:06:53.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388461+0000", 2023-07-02T14:06:53.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067708480000000003 2023-07-02T14:06:53.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395232+0000", 2023-07-02T14:06:53.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7948000000000002e-05 2023-07-02T14:06:53.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:53.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395290+0000", 2023-07-02T14:06:53.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00027672599999999998 2023-07-02T14:06:53.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:53.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395566+0000", 2023-07-02T14:06:53.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025705569999999998 2023-07-02T14:06:53.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:53.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.398137+0000", 2023-07-02T14:06:53.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072257099999999998 2023-07-02T14:06:53.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:53.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405363+0000", 2023-07-02T14:06:53.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3887999999999998e-05 2023-07-02T14:06:53.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:53.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405397+0000", 2023-07-02T14:06:53.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000105769 2023-07-02T14:06:53.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:53.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405502+0000", 2023-07-02T14:06:53.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.15647551900000001 2023-07-02T14:06:53.576 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.576 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.576 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.576 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18561 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:53.576 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:53.576 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 48.113549264, 2023-07-02T14:06:53.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744, 2023-07-02T14:06:53.577 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:53.577 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:53.577 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:53.577 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:53.577 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:53.577 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18561 2023-07-02T14:06:53.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.578 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:53.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:53.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:53.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:53.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:53.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:53.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:53.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:53.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:53.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.563e-06 2023-07-02T14:06:53.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:53.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249050+0000", 2023-07-02T14:06:53.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4500000000000001e-06 2023-07-02T14:06:53.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:53.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249057+0000", 2023-07-02T14:06:53.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1860000000000002e-06 2023-07-02T14:06:53.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:53.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249061+0000", 2023-07-02T14:06:53.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00091153399999999998 2023-07-02T14:06:53.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249972+0000", 2023-07-02T14:06:53.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7156999999999997e-05 2023-07-02T14:06:53.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.250010+0000", 2023-07-02T14:06:53.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006641816 2023-07-02T14:06:53.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256651+0000", 2023-07-02T14:06:53.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1968999999999998e-05 2023-07-02T14:06:53.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256683+0000", 2023-07-02T14:06:53.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012463952 2023-07-02T14:06:53.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269147+0000", 2023-07-02T14:06:53.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2855000000000006e-05 2023-07-02T14:06:53.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269220+0000", 2023-07-02T14:06:53.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010870418999999999 2023-07-02T14:06:53.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280091+0000", 2023-07-02T14:06:53.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1921999999999997e-05 2023-07-02T14:06:53.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280123+0000", 2023-07-02T14:06:53.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011861682 2023-07-02T14:06:53.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291984+0000", 2023-07-02T14:06:53.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5259000000000006e-05 2023-07-02T14:06:53.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.292059+0000", 2023-07-02T14:06:53.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079709359999999996 2023-07-02T14:06:53.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300030+0000", 2023-07-02T14:06:53.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0729999999999998e-05 2023-07-02T14:06:53.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300071+0000", 2023-07-02T14:06:53.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072785690000000004 2023-07-02T14:06:53.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307350+0000", 2023-07-02T14:06:53.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1948999999999999e-05 2023-07-02T14:06:53.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307412+0000", 2023-07-02T14:06:53.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055785779999999998 2023-07-02T14:06:53.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312990+0000", 2023-07-02T14:06:53.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0957000000000002e-05 2023-07-02T14:06:53.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.313021+0000", 2023-07-02T14:06:53.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049246400000000001 2023-07-02T14:06:53.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317946+0000", 2023-07-02T14:06:53.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9346e-05 2023-07-02T14:06:53.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317975+0000", 2023-07-02T14:06:53.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049649469999999999 2023-07-02T14:06:53.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322940+0000", 2023-07-02T14:06:53.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9017000000000002e-05 2023-07-02T14:06:53.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322989+0000", 2023-07-02T14:06:53.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065907029999999998 2023-07-02T14:06:53.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329580+0000", 2023-07-02T14:06:53.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5879999999999999e-05 2023-07-02T14:06:53.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329656+0000", 2023-07-02T14:06:53.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060166960000000002 2023-07-02T14:06:53.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335672+0000", 2023-07-02T14:06:53.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8963e-05 2023-07-02T14:06:53.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335701+0000", 2023-07-02T14:06:53.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098989560000000004 2023-07-02T14:06:53.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345600+0000", 2023-07-02T14:06:53.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5395999999999995e-05 2023-07-02T14:06:53.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345666+0000", 2023-07-02T14:06:53.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046589530000000004 2023-07-02T14:06:53.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350325+0000", 2023-07-02T14:06:53.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2459000000000002e-05 2023-07-02T14:06:53.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350357+0000", 2023-07-02T14:06:53.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042054290000000001 2023-07-02T14:06:53.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354563+0000", 2023-07-02T14:06:53.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5612999999999999e-05 2023-07-02T14:06:53.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354598+0000", 2023-07-02T14:06:53.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070967449999999998 2023-07-02T14:06:53.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361695+0000", 2023-07-02T14:06:53.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9314000000000001e-05 2023-07-02T14:06:53.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361724+0000", 2023-07-02T14:06:53.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007391028 2023-07-02T14:06:53.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369115+0000", 2023-07-02T14:06:53.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9163000000000002e-05 2023-07-02T14:06:53.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.604 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369184+0000", 2023-07-02T14:06:53.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056532500000000003 2023-07-02T14:06:53.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.604 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.604 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374838+0000", 2023-07-02T14:06:53.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5297999999999999e-05 2023-07-02T14:06:53.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374883+0000", 2023-07-02T14:06:53.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077292189999999998 2023-07-02T14:06:53.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382612+0000", 2023-07-02T14:06:53.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1102000000000001e-05 2023-07-02T14:06:53.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382653+0000", 2023-07-02T14:06:53.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058515989999999999 2023-07-02T14:06:53.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388505+0000", 2023-07-02T14:06:53.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9196000000000006e-05 2023-07-02T14:06:53.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388574+0000", 2023-07-02T14:06:53.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084245339999999991 2023-07-02T14:06:53.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.396999+0000", 2023-07-02T14:06:53.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0596999999999998e-05 2023-07-02T14:06:53.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:53.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.397039+0000", 2023-07-02T14:06:53.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084911339999999991 2023-07-02T14:06:53.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405530+0000", 2023-07-02T14:06:53.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7555000000000001e-05 2023-07-02T14:06:53.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:53.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405578+0000", 2023-07-02T14:06:53.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000229607 2023-07-02T14:06:53.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:53.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405808+0000", 2023-07-02T14:06:53.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026919940000000001 2023-07-02T14:06:53.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:53.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.408500+0000", 2023-07-02T14:06:53.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053065059999999999 2023-07-02T14:06:53.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:53.613 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413806+0000", 2023-07-02T14:06:53.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0490000000000001e-05 2023-07-02T14:06:53.613 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.613 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.613 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:53.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413837+0000", 2023-07-02T14:06:53.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000100386 2023-07-02T14:06:53.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:53.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413937+0000", 2023-07-02T14:06:53.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744 2023-07-02T14:06:53.614 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.615 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.615 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.615 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20768 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch ping cookie 140544751726928 gen 52] snapc 0=[] ondisk+write+known_if_redirected e183)", 2023-07-02T14:06:53.615 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:53.615 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 34.140181996000003, 2023-07-02T14:06:53.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3751256970000001, 2023-07-02T14:06:53.616 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:53.616 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:53.616 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:53.616 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:53.616 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:53.616 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20768 2023-07-02T14:06:53.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.617 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:53.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:53.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:53.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:53.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:53.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:53.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:53.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:53.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:53.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7829999999999998e-06 2023-07-02T14:06:53.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:53.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222419+0000", 2023-07-02T14:06:53.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1380000000000004e-06 2023-07-02T14:06:53.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:53.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222425+0000", 2023-07-02T14:06:53.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2519999999999996e-06 2023-07-02T14:06:53.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:53.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222431+0000", 2023-07-02T14:06:53.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2157999999999994e-05 2023-07-02T14:06:53.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222523+0000", 2023-07-02T14:06:53.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5147000000000001e-05 2023-07-02T14:06:53.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:53.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222538+0000", 2023-07-02T14:06:53.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30105419700000002 2023-07-02T14:06:53.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523592+0000", 2023-07-02T14:06:53.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2316e-05 2023-07-02T14:06:53.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:53.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523605+0000", 2023-07-02T14:06:53.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.073658953 2023-07-02T14:06:53.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597264+0000", 2023-07-02T14:06:53.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000213415 2023-07-02T14:06:53.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:53.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597477+0000", 2023-07-02T14:06:53.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3034999999999995e-05 2023-07-02T14:06:53.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.625 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:53.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597540+0000", 2023-07-02T14:06:53.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3751256970000001 2023-07-02T14:06:53.625 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.625 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.625 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.626 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20769 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 53] snapc 0=[] ondisk+write+known_if_redirected e184)", 2023-07-02T14:06:53.626 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:53.626 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 34.139819541000001, 2023-07-02T14:06:53.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.380884139, 2023-07-02T14:06:53.626 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:53.626 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:53.627 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:53.627 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:53.627 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:53.627 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20769 2023-07-02T14:06:53.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.627 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:53.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:53.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:53.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:53.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:53.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:53.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:53.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:53.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222776+0000", 2023-07-02T14:06:53.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9710000000000003e-06 2023-07-02T14:06:53.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:53.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222781+0000", 2023-07-02T14:06:53.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9899999999999992e-06 2023-07-02T14:06:53.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:53.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222791+0000", 2023-07-02T14:06:53.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4240000000000003e-06 2023-07-02T14:06:53.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:53.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222795+0000", 2023-07-02T14:06:53.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2366000000000001e-05 2023-07-02T14:06:53.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222828+0000", 2023-07-02T14:06:53.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4790000000000001e-06 2023-07-02T14:06:53.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:53.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222834+0000", 2023-07-02T14:06:53.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30084589299999998 2023-07-02T14:06:53.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523680+0000", 2023-07-02T14:06:53.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.87e-06 2023-07-02T14:06:53.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:53.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523685+0000", 2023-07-02T14:06:53.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.074111042 2023-07-02T14:06:53.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597796+0000", 2023-07-02T14:06:53.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0411000000000001e-05 2023-07-02T14:06:53.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:53.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597856+0000", 2023-07-02T14:06:53.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058045340000000001 2023-07-02T14:06:53.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:53.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.603661+0000", 2023-07-02T14:06:53.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.380884139 2023-07-02T14:06:53.637 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.637 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.637 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.638 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20770 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 54] snapc 0=[] ondisk+write+known_if_redirected e185)", 2023-07-02T14:06:53.638 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:53.638 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 33.838602430999998, 2023-07-02T14:06:53.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0799819349999999, 2023-07-02T14:06:53.638 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:53.638 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:53.638 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:53.639 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:53.639 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:53.639 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20770 2023-07-02T14:06:53.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.639 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:53.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:53.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:53.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:53.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:53.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:53.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:53.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:53.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523992+0000", 2023-07-02T14:06:53.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6720000000000004e-06 2023-07-02T14:06:53.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:53.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524000+0000", 2023-07-02T14:06:53.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3711e-05 2023-07-02T14:06:53.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:53.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524014+0000", 2023-07-02T14:06:53.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7849999999999992e-06 2023-07-02T14:06:53.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:53.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524022+0000", 2023-07-02T14:06:53.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2366000000000001e-05 2023-07-02T14:06:53.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524054+0000", 2023-07-02T14:06:53.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8169999999999998e-06 2023-07-02T14:06:53.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:53.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524059+0000", 2023-07-02T14:06:53.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0748338850000001 2023-07-02T14:06:53.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.598893+0000", 2023-07-02T14:06:53.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116968 2023-07-02T14:06:53.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:53.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.599010+0000", 2023-07-02T14:06:53.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049663370000000004 2023-07-02T14:06:53.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:53.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.603976+0000", 2023-07-02T14:06:53.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0799819349999999 2023-07-02T14:06:53.647 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.647 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.647 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.647 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20833 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 55] snapc 0=[] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:53.647 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:06:53.647 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 28.830483126000001, 2023-07-02T14:06:53.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.029349206, 2023-07-02T14:06:53.648 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:53.648 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:53.648 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:53.648 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:53.648 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:53.648 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20833 2023-07-02T14:06:53.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.649 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:53.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:53.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:06:53.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:53.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:53.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:06:53.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:53.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:53.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532112+0000", 2023-07-02T14:06:53.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7590000000000003e-06 2023-07-02T14:06:53.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:53.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532120+0000", 2023-07-02T14:06:53.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8130000000000001e-05 2023-07-02T14:06:53.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:53.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532139+0000", 2023-07-02T14:06:53.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3299999999999999e-06 2023-07-02T14:06:53.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:53.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532147+0000", 2023-07-02T14:06:53.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9679999999999998e-05 2023-07-02T14:06:53.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532207+0000", 2023-07-02T14:06:53.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0648e-05 2023-07-02T14:06:53.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op must wait for map", 2023-07-02T14:06:53.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532217+0000", 2023-07-02T14:06:53.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002082455 2023-07-02T14:06:53.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.534300+0000", 2023-07-02T14:06:53.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0710000000000001e-05 2023-07-02T14:06:53.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:53.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.534310+0000", 2023-07-02T14:06:53.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.013227235 2023-07-02T14:06:53.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.547538+0000", 2023-07-02T14:06:53.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021904300000000001 2023-07-02T14:06:53.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:53.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.547757+0000", 2023-07-02T14:06:53.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013705784 2023-07-02T14:06:53.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:53.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561462+0000", 2023-07-02T14:06:53.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.029349206 2023-07-02T14:06:53.657 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.657 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.657 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.658 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20865 2.7 2:ff0f5f99:::rbd_data.125de1449605.0000000000000006:head [write 131072~8192] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:53.658 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:06:53.658 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 28.085860054000001, 2023-07-02T14:06:53.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.288115762, 2023-07-02T14:06:53.658 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:53.658 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20865 2023-07-02T14:06:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:06:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:06:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:53.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:53.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276735+0000", 2023-07-02T14:06:53.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3273e-05 2023-07-02T14:06:53.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:53.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276748+0000", 2023-07-02T14:06:53.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3895e-05 2023-07-02T14:06:53.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:53.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276762+0000", 2023-07-02T14:06:53.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.896e-06 2023-07-02T14:06:53.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:53.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276768+0000", 2023-07-02T14:06:53.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5850999999999999e-05 2023-07-02T14:06:53.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276814+0000", 2023-07-02T14:06:53.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5290000000000003e-06 2023-07-02T14:06:53.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:53.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276821+0000", 2023-07-02T14:06:53.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.27225312600000001 2023-07-02T14:06:53.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549074+0000", 2023-07-02T14:06:53.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000151223 2023-07-02T14:06:53.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:53.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549226+0000", 2023-07-02T14:06:53.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024865400000000001 2023-07-02T14:06:53.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:53.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549474+0000", 2023-07-02T14:06:53.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012039185000000001 2023-07-02T14:06:53.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:53.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561513+0000", 2023-07-02T14:06:53.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003221327 2023-07-02T14:06:53.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:53.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564735+0000", 2023-07-02T14:06:53.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2026000000000005e-05 2023-07-02T14:06:53.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:53.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564807+0000", 2023-07-02T14:06:53.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5303000000000003e-05 2023-07-02T14:06:53.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:53.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564852+0000", 2023-07-02T14:06:53.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.288115762 2023-07-02T14:06:53.669 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.669 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.669 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.669 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20866 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 131072~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:53.670 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:06:53.670 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 28.080166732999999, 2023-07-02T14:06:53.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.28773701200000001, 2023-07-02T14:06:53.670 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:53.670 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:53.670 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:53.670 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:53.670 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:53.671 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20866 2023-07-02T14:06:53.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.671 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:53.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:53.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:06:53.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:53.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:53.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:06:53.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:53.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:53.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282428+0000", 2023-07-02T14:06:53.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8469999999999998e-05 2023-07-02T14:06:53.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:53.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282517+0000", 2023-07-02T14:06:53.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2974e-05 2023-07-02T14:06:53.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:53.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282530+0000", 2023-07-02T14:06:53.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4530000000000001e-06 2023-07-02T14:06:53.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:53.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282535+0000", 2023-07-02T14:06:53.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1926999999999999e-05 2023-07-02T14:06:53.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282587+0000", 2023-07-02T14:06:53.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3083e-05 2023-07-02T14:06:53.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:53.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282600+0000", 2023-07-02T14:06:53.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.270692971 2023-07-02T14:06:53.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553293+0000", 2023-07-02T14:06:53.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017065599999999999 2023-07-02T14:06:53.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:53.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553464+0000", 2023-07-02T14:06:53.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026504399999999998 2023-07-02T14:06:53.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:53.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553729+0000", 2023-07-02T14:06:53.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011173008 2023-07-02T14:06:53.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:53.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564902+0000", 2023-07-02T14:06:53.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051242600000000003 2023-07-02T14:06:53.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:53.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570026+0000", 2023-07-02T14:06:53.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8098999999999999e-05 2023-07-02T14:06:53.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:53.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570104+0000", 2023-07-02T14:06:53.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2323000000000006e-05 2023-07-02T14:06:53.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:53.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570167+0000", 2023-07-02T14:06:53.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.28773701200000001 2023-07-02T14:06:53.681 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.681 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.681 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.681 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20875 2.7 2:e3210266:::rbd_data.125de1449605.00000000000001a3:head [write 0~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:53.681 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:06:53.681 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 28.041115916999999, 2023-07-02T14:06:53.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.25297039599999999, 2023-07-02T14:06:53.682 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:53.682 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:53.682 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:53.682 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:53.682 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:53.682 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20875 2023-07-02T14:06:53.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.683 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:53.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:53.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:06:53.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:53.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:53.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:06:53.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:53.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:53.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321479+0000", 2023-07-02T14:06:53.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015194399999999999 2023-07-02T14:06:53.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:53.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321631+0000", 2023-07-02T14:06:53.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4236999999999999e-05 2023-07-02T14:06:53.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:53.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321645+0000", 2023-07-02T14:06:53.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2160000000000003e-06 2023-07-02T14:06:53.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:53.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321651+0000", 2023-07-02T14:06:53.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3609999999999998e-05 2023-07-02T14:06:53.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321695+0000", 2023-07-02T14:06:53.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1989999999999995e-06 2023-07-02T14:06:53.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:53.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321704+0000", 2023-07-02T14:06:53.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.23542881500000001 2023-07-02T14:06:53.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557133+0000", 2023-07-02T14:06:53.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017359600000000001 2023-07-02T14:06:53.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:53.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557307+0000", 2023-07-02T14:06:53.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00023367599999999999 2023-07-02T14:06:53.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:53.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557540+0000", 2023-07-02T14:06:53.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016744963000000002 2023-07-02T14:06:53.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:53.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574285+0000", 2023-07-02T14:06:53.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1330999999999999e-05 2023-07-02T14:06:53.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:53.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574346+0000", 2023-07-02T14:06:53.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9750000000000002e-05 2023-07-02T14:06:53.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:53.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574406+0000", 2023-07-02T14:06:53.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4647999999999997e-05 2023-07-02T14:06:53.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:53.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574451+0000", 2023-07-02T14:06:53.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.25297039599999999 2023-07-02T14:06:53.692 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.693 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.693 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.693 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20915 2.7 2:e6e64067:::rbd_data.125de1449605.00000000000001a9:head [write 0~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:53.693 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:06:53.693 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 27.977862122000001, 2023-07-02T14:06:53.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.19966078500000001, 2023-07-02T14:06:53.694 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:53.694 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:53.694 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:53.694 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:53.694 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:53.694 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20915 2023-07-02T14:06:53.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.694 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:53.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:53.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:06:53.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:53.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:53.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:06:53.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:53.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:53.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384732+0000", 2023-07-02T14:06:53.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9439999999999999e-05 2023-07-02T14:06:53.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:53.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384802+0000", 2023-07-02T14:06:53.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7572999999999999e-05 2023-07-02T14:06:53.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:53.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384819+0000", 2023-07-02T14:06:53.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1670000000000004e-06 2023-07-02T14:06:53.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:53.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384825+0000", 2023-07-02T14:06:53.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6471999999999998e-05 2023-07-02T14:06:53.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384872+0000", 2023-07-02T14:06:53.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0028e-05 2023-07-02T14:06:53.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:53.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384882+0000", 2023-07-02T14:06:53.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.17674214099999999 2023-07-02T14:06:53.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561624+0000", 2023-07-02T14:06:53.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000135809 2023-07-02T14:06:53.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:53.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561760+0000", 2023-07-02T14:06:53.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020468 2023-07-02T14:06:53.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:53.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561965+0000", 2023-07-02T14:06:53.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012498198 2023-07-02T14:06:53.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:53.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574463+0000", 2023-07-02T14:06:53.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098197509999999998 2023-07-02T14:06:53.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:53.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584283+0000", 2023-07-02T14:06:53.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2056000000000004e-05 2023-07-02T14:06:53.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:53.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584345+0000", 2023-07-02T14:06:53.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0513000000000002e-05 2023-07-02T14:06:53.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:53.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584395+0000", 2023-07-02T14:06:53.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.19966078500000001 2023-07-02T14:06:53.704 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.704 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.704 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.705 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20948 2.7 2:ff0f5f99:::rbd_data.125de1449605.0000000000000006:head [write 139264~2220032] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:53.705 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.450499+0000", 2023-07-02T14:06:53.705 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 27.912097395, 2023-07-02T14:06:53.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13400999399999999, 2023-07-02T14:06:53.705 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:53.705 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:53.706 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:53.706 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:53.706 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:53.706 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20948 2023-07-02T14:06:53.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.706 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:53.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:53.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.450499+0000", 2023-07-02T14:06:53.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:53.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:53.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.450499+0000", 2023-07-02T14:06:53.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:53.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:53.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.450497+0000", 2023-07-02T14:06:53.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043010360000000003 2023-07-02T14:06:53.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:53.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454798+0000", 2023-07-02T14:06:53.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8073000000000001e-05 2023-07-02T14:06:53.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:53.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454816+0000", 2023-07-02T14:06:53.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3769999999999996e-06 2023-07-02T14:06:53.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:53.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454825+0000", 2023-07-02T14:06:53.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2797999999999998e-05 2023-07-02T14:06:53.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454878+0000", 2023-07-02T14:06:53.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0020999999999999e-05 2023-07-02T14:06:53.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:53.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454888+0000", 2023-07-02T14:06:53.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.110043313 2023-07-02T14:06:53.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564931+0000", 2023-07-02T14:06:53.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7185e-05 2023-07-02T14:06:53.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:53.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.565008+0000", 2023-07-02T14:06:53.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014058199999999999 2023-07-02T14:06:53.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:53.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.565149+0000", 2023-07-02T14:06:53.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093221559999999998 2023-07-02T14:06:53.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:53.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574471+0000", 2023-07-02T14:06:53.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099660410000000001 2023-07-02T14:06:53.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:53.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584437+0000", 2023-07-02T14:06:53.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2805000000000001e-05 2023-07-02T14:06:53.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:53.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584480+0000", 2023-07-02T14:06:53.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9393000000000001e-05 2023-07-02T14:06:53.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:53.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584509+0000", 2023-07-02T14:06:53.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13400999399999999 2023-07-02T14:06:53.716 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.716 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.716 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.717 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20949 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 196608~2162688] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:53.717 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.454838+0000", 2023-07-02T14:06:53.717 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 27.907758235999999, 2023-07-02T14:06:53.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12983145400000001, 2023-07-02T14:06:53.717 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:53.718 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:53.718 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:53.718 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:53.718 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:53.718 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20949 2023-07-02T14:06:53.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.718 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:53.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:53.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454838+0000", 2023-07-02T14:06:53.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:53.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:53.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454838+0000", 2023-07-02T14:06:53.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:53.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:53.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454837+0000", 2023-07-02T14:06:53.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003772821 2023-07-02T14:06:53.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:53.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458610+0000", 2023-07-02T14:06:53.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3013999999999999e-05 2023-07-02T14:06:53.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:53.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458623+0000", 2023-07-02T14:06:53.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2789999999999998e-06 2023-07-02T14:06:53.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:53.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458628+0000", 2023-07-02T14:06:53.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7630999999999998e-05 2023-07-02T14:06:53.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458676+0000", 2023-07-02T14:06:53.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1579999999999994e-06 2023-07-02T14:06:53.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:53.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458685+0000", 2023-07-02T14:06:53.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.111536444 2023-07-02T14:06:53.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570221+0000", 2023-07-02T14:06:53.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000120844 2023-07-02T14:06:53.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:53.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570342+0000", 2023-07-02T14:06:53.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000149965 2023-07-02T14:06:53.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:53.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570492+0000", 2023-07-02T14:06:53.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014027843999999999 2023-07-02T14:06:53.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:53.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584520+0000", 2023-07-02T14:06:53.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8446000000000003e-05 2023-07-02T14:06:53.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:53.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584568+0000", 2023-07-02T14:06:53.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6771999999999998e-05 2023-07-02T14:06:53.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:53.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584625+0000", 2023-07-02T14:06:53.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4401999999999999e-05 2023-07-02T14:06:53.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:53.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584670+0000", 2023-07-02T14:06:53.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12983145400000001 2023-07-02T14:06:53.728 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.728 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.728 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.729 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20972 2.7 2:e3210266:::rbd_data.125de1449605.00000000000001a3:head [write 65536~4128768] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:53.729 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.463174+0000", 2023-07-02T14:06:53.729 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 27.899422395999999, 2023-07-02T14:06:53.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12719914800000001, 2023-07-02T14:06:53.729 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:53.729 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:53.730 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:53.730 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:53.730 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:53.730 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20972 2023-07-02T14:06:53.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.730 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:53.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:53.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.463174+0000", 2023-07-02T14:06:53.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:53.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:53.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.463174+0000", 2023-07-02T14:06:53.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:53.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:53.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.463173+0000", 2023-07-02T14:06:53.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069882069999999998 2023-07-02T14:06:53.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:53.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470161+0000", 2023-07-02T14:06:53.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0103999999999999e-05 2023-07-02T14:06:53.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:53.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470181+0000", 2023-07-02T14:06:53.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:06:53.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:53.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470189+0000", 2023-07-02T14:06:53.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0970999999999994e-05 2023-07-02T14:06:53.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470270+0000", 2023-07-02T14:06:53.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4389000000000001e-05 2023-07-02T14:06:53.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:53.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470284+0000", 2023-07-02T14:06:53.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.10421862799999999 2023-07-02T14:06:53.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:53.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574503+0000", 2023-07-02T14:06:53.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2457e-05 2023-07-02T14:06:53.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:53.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574585+0000", 2023-07-02T14:06:53.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013870899999999999 2023-07-02T14:06:53.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:53.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574724+0000", 2023-07-02T14:06:53.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012886246000000001 2023-07-02T14:06:53.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:53.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.587610+0000", 2023-07-02T14:06:53.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026883599999999999 2023-07-02T14:06:53.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:53.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.590298+0000", 2023-07-02T14:06:53.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1850000000000001e-05 2023-07-02T14:06:53.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:53.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.590340+0000", 2023-07-02T14:06:53.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2901999999999998e-05 2023-07-02T14:06:53.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:53.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:53.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:53.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.590373+0000", 2023-07-02T14:06:53.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12719914800000001 2023-07-02T14:06:53.740 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.740 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.740 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.740 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:53.740 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:53.741 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:53.742 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:06:53.743 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:53.742+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:53.795 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:06:53.843 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:53.842+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:53.882 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:53.883 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:53.883 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:53.883 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:53.894 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:06:53.898 INFO:tasks.workunit.client.0.smithi163.stderr:+ tar -C b3fbb028-e267-4d25-a750-93ba18a2316a-mnt -xzf b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4.tar.gz 2023-07-02T14:06:53.943 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:53.943+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:53.944 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:53.961 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:53.979 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:53.997 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:54.016 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:54.022 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:54.022 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:54.032 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:06:54.034 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:54.036 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:06:54.037 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:06:54.037 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:06:54.037 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:06:54.037 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:06:54.037 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:06:54.047 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:54.047+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:54.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:06:54.145 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:54.144+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:54.159 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:54.159 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:54.170 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:06:54.174 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap61 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap62 b3fbb028-e267-4d25-a750-93ba18a2316a@snap62.diff 2023-07-02T14:06:54.242 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:54.242+0000 7f7e2ad5b700 0 -- 172.21.15.163:0/3524319397 >> [v2:172.21.15.22:6808/99977,v1:172.21.15.22:6809/99977] conn(0x7f7e08008780 msgr2=0x7f7e0800abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:54.246 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:54.246+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:54.250 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:54.250+0000 7f7e2b55c700 0 -- 172.21.15.163:0/3524319397 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x559bf4f6b280 msgr2=0x559bf4f6d670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:54.346 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:54.346 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:54.347 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:54.347+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:54.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:06:54.447 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:54.447+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:54.486 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:54.487 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:54.492 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 100% complete...done. 2023-07-02T14:06:54.496 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:06:54.497 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=63 2023-07-02T14:06:54.497 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap63 2023-07-02T14:06:54.498 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:06:54.548 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:54.547+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:54.649 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:54.649+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:54.662 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:54.662 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:54.672 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:06:54.749 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:54.748+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:54.836 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:54.837 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:54.846 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:06:54.851 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:54.851+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:54.951 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:54.951+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:54.970 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:54.970 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:54.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:06:55.052 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:55.052+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:55.104 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:55.105 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:55.115 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:06:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:06:55.159 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:55.158+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:55.181 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:06:55.187 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 2 2023-07-02T14:06:55.254 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:55.254+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:55.256 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:55.256 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:55.259 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 2, 4] out_osds: [5] dead_osds: [] live_osds: [1, 2, 4, 5, 3, 0] 2023-07-02T14:06:55.259 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:06:55.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:06:55.267 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:06:55.355 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:55.355+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:55.432 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:55.433 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:55.442 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:06:55.487 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:55.486+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:55.556 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:55.556+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:55.558 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:06:55.559 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:06:55.566 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:55.566 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:55.570 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2023-07-02T14:06:55.127520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10690255761,"num_objects":3202,"num_object_clones":2676,"num_object_copies":6404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3202,"num_whiteouts":0,"num_read":80492,"num_read_kb":14234972,"num_write":24712,"num_write_kb":10375440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2445,"num_bytes_recovered":8416021391,"num_keys_recovered":178,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24413,"ondisk_log_size":24413,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":21263404,"kb_used_data":20942340,"kb_used_omap":1114,"kb_used_meta":319909,"kb_avail":450595796,"statfs":{"total":483183820800,"available":461410095104,"internally_reserved":0,"allocated":21444956160,"data_stored":24923599838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1141261,"internal_metadata":327587315},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4477090,"num_objects":6,"num_object_clones":5,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1592,"num_read_kb":1088,"num_write":19,"num_write_kb":4379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.841033"},"pg_stats":[{"pgid":"2.7","version":"205'3389","reported_seq":8848,"reported_epoch":205,"state":"active+clean","last_fresh":"2023-07-02T14:06:52.196215+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:06:52.196215+0000","last_peered":"2023-07-02T14:06:52.196215+0000","last_clean":"2023-07-02T14:06:52.196215+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:06:52.196215+0000","last_undegraded":"2023-07-02T14:06:52.196215+0000","last_fullsized":"2023-07-02T14:06:52.196215+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3389,"ondisk_log_size":3389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1418291506,"num_objects":424,"num_object_clones":357,"num_object_copies":848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":424,"num_whiteouts":0,"num_read":52493,"num_read_kb":1825492,"num_write":2846,"num_write_kb":1377580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"203'2768","reported_seq":4714,"reported_epoch":203,"state":"active+clean","last_fresh":"2023-07-02T14:06:47.286988+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:06:47.286988+0000","last_peered":"2023-07-02T14:06:47.286988+0000","last_clean":"2023-07-02T14:06:47.286988+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:06:47.286988+0000","last_undegraded":"2023-07-02T14:06:47.286988+0000","last_fullsized":"2023-07-02T14:06:47.286988+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2768,"ondisk_log_size":2768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1389634832,"num_objects":436,"num_object_clones":370,"num_object_copies":872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":436,"num_whiteouts":0,"num_read":1749,"num_read_kb":1934812,"num_write":2398,"num_write_kb":1347836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"197'2882","reported_seq":3491,"reported_epoch":204,"state":"active+clean","last_fresh":"2023-07-02T14:06:49.113294+0000","last_change":"2023-07-02T14:05:57.093448+0000","last_active":"2023-07-02T14:06:49.113294+0000","last_peered":"2023-07-02T14:06:49.113294+0000","last_clean":"2023-07-02T14:06:49.113294+0000","last_became_active":"2023-07-02T14:05:57.089226+0000","last_became_peered":"2023-07-02T14:05:57.089226+0000","last_unstale":"2023-07-02T14:06:49.113294+0000","last_undegraded":"2023-07-02T14:06:49.113294+0000","last_fullsized":"2023-07-02T14:06:49.113294+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2882,"ondisk_log_size":2882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1348875212,"num_objects":375,"num_object_clones":315,"num_object_copies":750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":375,"num_whiteouts":0,"num_read":561,"num_read_kb":1781911,"num_write":2567,"num_write_kb":1304531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":860565666,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"203'5853","reported_seq":8303,"reported_epoch":203,"state":"active+clean","last_fresh":"2023-07-02T14:06:47.280882+0000","last_change":"2023-07-02T14:06:29.572710+0000","last_active":"2023-07-02T14:06:47.280882+0000","last_peered":"2023-07-02T14:06:47.280882+0000","last_clean":"2023-07-02T14:06:47.280882+0000","last_became_active":"2023-07-02T14:06:29.569824+0000","last_became_peered":"2023-07-02T14:06:29.569824+0000","last_unstale":"2023-07-02T14:06:47.280882+0000","last_undegraded":"2023-07-02T14:06:47.280882+0000","last_fullsized":"2023-07-02T14:06:47.280882+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":5853,"ondisk_log_size":5853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1484600770,"num_objects":435,"num_object_clones":366,"num_object_copies":870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":435,"num_whiteouts":0,"num_read":20722,"num_read_kb":1947733,"num_write":8648,"num_write_kb":1449933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":132,"reported_epoch":203,"state":"active+clean","last_fresh":"2023-07-02T14:06:47.190715+0000","last_change":"2023-07-02T14:05:53.410292+0000","last_active":"2023-07-02T14:06:47.190715+0000","last_peered":"2023-07-02T14:06:47.190715+0000","last_clean":"2023-07-02T14:06:47.190715+0000","last_became_active":"2023-07-02T14:05:53.409913+0000","last_became_peered":"2023-07-02T14:05:53.409913+0000","last_unstale":"2023-07-02T14:06:47.190715+0000","last_undegraded":"2023-07-02T14:06:47.190715+0000","last_fullsized":"2023-07-02T14:06:47.190715+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"199'2651","reported_seq":4012,"reported_epoch":203,"state":"active+clean","last_fresh":"2023-07-02T14:06:47.109094+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:06:47.109094+0000","last_peered":"2023-07-02T14:06:47.109094+0000","last_clean":"2023-07-02T14:06:47.109094+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:06:47.109094+0000","last_undegraded":"2023-07-02T14:06:47.109094+0000","last_fullsized":"2023-07-02T14:06:47.109094+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2651,"ondisk_log_size":2651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1483183230,"num_objects":437,"num_object_clones":366,"num_object_copies":874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":437,"num_whiteouts":0,"num_read":1334,"num_read_kb":1887201,"num_write":2285,"num_write_kb":1439844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"203'2621","reported_seq":3477,"reported_epoch":203,"state":"active+clean","last_fresh":"2023-07-02T14:06:47.121984+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:06:47.121984+0000","last_peered":"2023-07-02T14:06:47.121984+0000","last_clean":"2023-07-02T14:06:47.121984+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:06:47.121984+0000","last_undegraded":"2023-07-02T14:06:47.121984+0000","last_fullsized":"2023-07-02T14:06:47.121984+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2621,"ondisk_log_size":2621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1418987092,"num_objects":413,"num_object_clones":343,"num_object_copies":826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":413,"num_whiteouts":0,"num_read":906,"num_read_kb":1805707,"num_write":2278,"num_write_kb":1377286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"203'2102","reported_seq":3933,"reported_epoch":203,"state":"active+clean","last_fresh":"2023-07-02T14:06:47.274117+0000","last_change":"2023-07-02T14:05:58.098047+0000","last_active":"2023-07-02T14:06:47.274117+0000","last_peered":"2023-07-02T14:06:47.274117+0000","last_clean":"2023-07-02T14:06:47.274117+0000","last_became_active":"2023-07-02T14:05:58.096421+0000","last_became_peered":"2023-07-02T14:05:58.096421+0000","last_unstale":"2023-07-02T14:06:47.274117+0000","last_undegraded":"2023-07-02T14:06:47.274117+0000","last_fullsized":"2023-07-02T14:06:47.274117+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2102,"ondisk_log_size":2102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1107027397,"num_objects":362,"num_object_clones":291,"num_object_copies":724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":1713,"num_read_kb":1609153,"num_write":1811,"num_write_kb":1068023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":340635667,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"199'2147","reported_seq":3234,"reported_epoch":203,"state":"active+clean","last_fresh":"2023-07-02T14:06:47.116938+0000","last_change":"2023-07-02T14:05:56.429093+0000","last_active":"2023-07-02T14:06:47.116938+0000","last_peered":"2023-07-02T14:06:47.116938+0000","last_clean":"2023-07-02T14:06:47.116938+0000","last_became_active":"2023-07-02T14:05:56.427016+0000","last_became_peered":"2023-07-02T14:05:56.427016+0000","last_unstale":"2023-07-02T14:06:47.116938+0000","last_undegraded":"2023-07-02T14:06:47.116938+0000","last_fullsized":"2023-07-02T14:06:47.116938+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2147,"ondisk_log_size":2147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1039655722,"num_objects":320,"num_object_clones":268,"num_object_copies":640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":320,"num_whiteouts":0,"num_read":1014,"num_read_kb":1442963,"num_write":1879,"num_write_kb":1010407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":271880192,"num_keys_recovered":4,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":10690255761,"num_objects":3202,"num_object_clones":2676,"num_object_copies":6404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3202,"num_whiteouts":0,"num_read":80492,"num_read_kb":14234972,"num_write":24712,"num_write_kb":10375440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2445,"num_bytes_recovered":8416021391,"num_keys_recovered":178,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21434077184,"data_stored":24922957602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":24413,"ondisk_log_size":24413,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640319,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640319,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2922592,"kb_used_data":2862360,"kb_used_omap":0,"kb_used_meta":60224,"kb_avail":91449248,"statfs":{"total":96636764160,"available":93644029952,"internally_reserved":0,"allocated":2931056640,"data_stored":3332660617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61669376},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:51 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.44,"15min":0.44400000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.64400000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47399999999999998,"15min":0.47999999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.69999999999999996,"5min":0.877,"15min":0.877},"last":0.32900000000000001}]},{"osd":1,"last update":"Sun Jul 2 14:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.44500000000000001,"15min":0.46600000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.436,"15min":0.44700000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.44900000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.44,"15min":0.45000000000000001},"min":{"1min":0.26500000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.47699999999999998,"15min":0.48399999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.53600000000000003}]},{"osd":3,"last update":"Sun Jul 2 14:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.42399999999999999}]},{"osd":5,"last update":"Sun Jul 2 14:05:53 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.40899999999999997,"15min":0.45300000000000001},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.65400000000000003,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.436},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.44500000000000001,"15min":0.497},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.57599999999999996}]}]},{"osd":3,"up_from":133,"seq":571230650391,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4104580,"kb_used_data":4039484,"kb_used_omap":229,"kb_used_meta":64858,"kb_avail":90267260,"statfs":{"total":96636764160,"available":92433674240,"internally_reserved":0,"allocated":4136431616,"data_stored":4819567734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":66414814},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]},{"osd":1,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.41299999999999998}]},{"osd":2,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.43099999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.39700000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.35199999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640319,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2875392,"kb_used_data":2828664,"kb_used_omap":0,"kb_used_meta":46720,"kb_avail":91496448,"statfs":{"total":96636764160,"available":93692362752,"internally_reserved":0,"allocated":2896551936,"data_stored":3328129383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47841280},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:48 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.29399999999999998,"15min":0.29599999999999999},"min":{"1min":0.218,"5min":0.182,"15min":0.182},"max":{"1min":0.378,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.317,"5min":0.315,"15min":0.314},"min":{"1min":0.23000000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.32600000000000001}]},{"osd":1,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.38800000000000001,"15min":0.36199999999999999},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.2720000000000002,"5min":4.2720000000000002,"15min":4.2720000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.32300000000000001,"15min":0.32000000000000001},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.36899999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:06:04 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2,"5min":2,"15min":2},"last":0.46999999999999997}]},{"osd":4,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.45400000000000001,"15min":0.45100000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.436},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.46899999999999997,"15min":0.46300000000000002},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.51900000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:05:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.48799999999999999,"15min":0.48499999999999999},"min":{"1min":0.34699999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.48499999999999999}]}]},{"osd":0,"up_from":166,"seq":712964571149,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3556848,"kb_used_data":3522472,"kb_used_omap":884,"kb_used_meta":33483,"kb_avail":90814992,"statfs":{"total":96636764160,"available":92994551808,"internally_reserved":0,"allocated":3607011328,"data_stored":4122344272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":905963,"internal_metadata":34286869},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]},{"osd":1,"up_from":11,"seq":47244640319,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7803992,"kb_used_data":7689360,"kb_used_omap":0,"kb_used_meta":114624,"kb_avail":86567848,"statfs":{"total":96636764160,"available":88645476352,"internally_reserved":0,"allocated":7873904640,"data_stored":9320897832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":117374976},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:04:54 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.20200000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.32200000000000001,"15min":0.33000000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.41099999999999998,"5min":0.496,"15min":0.496},"last":0.32200000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.33500000000000002,"15min":0.33100000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.47899999999999998,"5min":1.615,"15min":1.615},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":0.53100000000000003,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.30499999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.42499999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47899999999999998,"15min":0.5},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.495,"15min":0.503},"min":{"1min":0.308,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71999999999999997,"5min":1.524,"15min":1.524},"last":0.51000000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:05:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51300000000000001,"15min":0.51500000000000001},"min":{"1min":0.33400000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.73899999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54800000000000004,"15min":0.54600000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.77600000000000002,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.53100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3605991424,"data_stored":4125572283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7871442944,"data_stored":9319931539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2894090240,"data_stored":3327163090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4133957632,"data_stored":4818596366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2928594944,"data_stored":3331694324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:06:55.576 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:06:55.660 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:55.659+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:55.746 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:55.746 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:55.757 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:06:55.798 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:55.798+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:55.859 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:55.859+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:55.924 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:55.925 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:55.934 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:06:55.960 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:55.960+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:56.048 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:06:56.058 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:56.058 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:56.060 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:56.059+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:56.068 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:06:56.161 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:56.161+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:56.188 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:56.205 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:56.222 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:56.232 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:56.233 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:56.240 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:56.243 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:06:56.257 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:56.261 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:56.261+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:56.275 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:56.277 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:06:56.277 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:06:56.277 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:06:56.277 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:06:56.277 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:06:56.277 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:06:56.362 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:56.361+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:56.367 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:56.367 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:06:56.367 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:06:56.367 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:06:56.367 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:56.377 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:06:56.463 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:56.463+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:56.544 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:06:56.545 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:06:56.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:06:56.545 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:06:56.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.545 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17354 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:56.545 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:56.545 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 73.238336777000001, 2023-07-02T14:06:56.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493, 2023-07-02T14:06:56.546 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:56.546 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:56.546 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:56.546 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:56.546 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:56.546 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17354 2023-07-02T14:06:56.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.547 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:56.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:56.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:56.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:56.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:56.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:56.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:56.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:56.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:06:56.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7440000000000001e-06 2023-07-02T14:06:56.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:56.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302606+0000", 2023-07-02T14:06:56.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:06:56.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:56.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302613+0000", 2023-07-02T14:06:56.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8810000000000004e-06 2023-07-02T14:06:56.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:56.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302618+0000", 2023-07-02T14:06:56.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028613100000000002 2023-07-02T14:06:56.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302904+0000", 2023-07-02T14:06:56.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5278999999999995e-05 2023-07-02T14:06:56.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302979+0000", 2023-07-02T14:06:56.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096904750000000005 2023-07-02T14:06:56.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312669+0000", 2023-07-02T14:06:56.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0328000000000002e-05 2023-07-02T14:06:56.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312700+0000", 2023-07-02T14:06:56.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098019319999999993 2023-07-02T14:06:56.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322502+0000", 2023-07-02T14:06:56.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4669999999999998e-05 2023-07-02T14:06:56.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322536+0000", 2023-07-02T14:06:56.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010023839 2023-07-02T14:06:56.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332560+0000", 2023-07-02T14:06:56.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5559999999999998e-05 2023-07-02T14:06:56.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332596+0000", 2023-07-02T14:06:56.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090401319999999993 2023-07-02T14:06:56.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341636+0000", 2023-07-02T14:06:56.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5373000000000001e-05 2023-07-02T14:06:56.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341681+0000", 2023-07-02T14:06:56.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081994059999999994 2023-07-02T14:06:56.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349881+0000", 2023-07-02T14:06:56.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4319000000000001e-05 2023-07-02T14:06:56.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349915+0000", 2023-07-02T14:06:56.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089388339999999997 2023-07-02T14:06:56.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358854+0000", 2023-07-02T14:06:56.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4319000000000005e-05 2023-07-02T14:06:56.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358928+0000", 2023-07-02T14:06:56.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080234690000000001 2023-07-02T14:06:56.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366952+0000", 2023-07-02T14:06:56.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2082999999999998e-05 2023-07-02T14:06:56.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366984+0000", 2023-07-02T14:06:56.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075015790000000004 2023-07-02T14:06:56.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374485+0000", 2023-07-02T14:06:56.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4480999999999995e-05 2023-07-02T14:06:56.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.563 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.563 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374550+0000", 2023-07-02T14:06:56.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091666620000000008 2023-07-02T14:06:56.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383716+0000", 2023-07-02T14:06:56.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9318000000000003e-05 2023-07-02T14:06:56.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383786+0000", 2023-07-02T14:06:56.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013434115 2023-07-02T14:06:56.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397220+0000", 2023-07-02T14:06:56.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4573999999999997e-05 2023-07-02T14:06:56.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397264+0000", 2023-07-02T14:06:56.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064824649999999998 2023-07-02T14:06:56.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403747+0000", 2023-07-02T14:06:56.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1087999999999993e-05 2023-07-02T14:06:56.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403828+0000", 2023-07-02T14:06:56.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072854449999999998 2023-07-02T14:06:56.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411113+0000", 2023-07-02T14:06:56.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5120000000000001e-05 2023-07-02T14:06:56.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411169+0000", 2023-07-02T14:06:56.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007211505 2023-07-02T14:06:56.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418380+0000", 2023-07-02T14:06:56.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011289399999999999 2023-07-02T14:06:56.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:56.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418493+0000", 2023-07-02T14:06:56.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00051217500000000002 2023-07-02T14:06:56.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:56.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.419005+0000", 2023-07-02T14:06:56.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00327393 2023-07-02T14:06:56.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:56.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422279+0000", 2023-07-02T14:06:56.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041432350000000003 2023-07-02T14:06:56.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:56.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426422+0000", 2023-07-02T14:06:56.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7021000000000003e-05 2023-07-02T14:06:56.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:56.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426469+0000", 2023-07-02T14:06:56.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000190256 2023-07-02T14:06:56.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:56.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426660+0000", 2023-07-02T14:06:56.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493 2023-07-02T14:06:56.574 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.574 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.574 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.574 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17355 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:06:56.575 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:56.575 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 73.238318116000002, 2023-07-02T14:06:56.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001, 2023-07-02T14:06:56.575 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:56.575 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:56.575 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:56.575 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:56.576 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:56.576 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17355 2023-07-02T14:06:56.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.576 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:56.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:56.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:56.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:56.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:56.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:06:56.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:56.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:56.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302621+0000", 2023-07-02T14:06:56.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2260000000000001e-06 2023-07-02T14:06:56.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:56.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302625+0000", 2023-07-02T14:06:56.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8599999999999998e-06 2023-07-02T14:06:56.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:56.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302630+0000", 2023-07-02T14:06:56.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0629999999999999e-06 2023-07-02T14:06:56.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:56.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302635+0000", 2023-07-02T14:06:56.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035731199999999998 2023-07-02T14:06:56.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302992+0000", 2023-07-02T14:06:56.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4164e-05 2023-07-02T14:06:56.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.303026+0000", 2023-07-02T14:06:56.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097115900000000008 2023-07-02T14:06:56.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312738+0000", 2023-07-02T14:06:56.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1701999999999998e-05 2023-07-02T14:06:56.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312819+0000", 2023-07-02T14:06:56.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097363640000000008 2023-07-02T14:06:56.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322556+0000", 2023-07-02T14:06:56.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2517000000000001e-05 2023-07-02T14:06:56.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322588+0000", 2023-07-02T14:06:56.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01002393 2023-07-02T14:06:56.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332612+0000", 2023-07-02T14:06:56.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2960999999999998e-05 2023-07-02T14:06:56.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332645+0000", 2023-07-02T14:06:56.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090669819999999995 2023-07-02T14:06:56.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341712+0000", 2023-07-02T14:06:56.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9288000000000002e-05 2023-07-02T14:06:56.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.586 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.586 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341811+0000", 2023-07-02T14:06:56.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081504460000000004 2023-07-02T14:06:56.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.587 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.587 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349962+0000", 2023-07-02T14:06:56.587 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107321 2023-07-02T14:06:56.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.587 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.350069+0000", 2023-07-02T14:06:56.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088790999999999991 2023-07-02T14:06:56.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358948+0000", 2023-07-02T14:06:56.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1711000000000002e-05 2023-07-02T14:06:56.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358980+0000", 2023-07-02T14:06:56.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080367579999999997 2023-07-02T14:06:56.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367017+0000", 2023-07-02T14:06:56.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6329000000000002e-05 2023-07-02T14:06:56.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367073+0000", 2023-07-02T14:06:56.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075097300000000001 2023-07-02T14:06:56.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374583+0000", 2023-07-02T14:06:56.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5710999999999995e-05 2023-07-02T14:06:56.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374648+0000", 2023-07-02T14:06:56.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009165912 2023-07-02T14:06:56.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383814+0000", 2023-07-02T14:06:56.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3916000000000003e-05 2023-07-02T14:06:56.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383878+0000", 2023-07-02T14:06:56.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013440532999999999 2023-07-02T14:06:56.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397319+0000", 2023-07-02T14:06:56.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7218999999999995e-05 2023-07-02T14:06:56.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397386+0000", 2023-07-02T14:06:56.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065165279999999997 2023-07-02T14:06:56.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403903+0000", 2023-07-02T14:06:56.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8152000000000005e-05 2023-07-02T14:06:56.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404001+0000", 2023-07-02T14:06:56.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072185360000000002 2023-07-02T14:06:56.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411219+0000", 2023-07-02T14:06:56.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:06:56.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411310+0000", 2023-07-02T14:06:56.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010021663 2023-07-02T14:06:56.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421331+0000", 2023-07-02T14:06:56.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0148000000000006e-05 2023-07-02T14:06:56.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421411+0000", 2023-07-02T14:06:56.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053056170000000003 2023-07-02T14:06:56.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426717+0000", 2023-07-02T14:06:56.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8242000000000001e-05 2023-07-02T14:06:56.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.599 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:56.599 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426795+0000", 2023-07-02T14:06:56.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00041308399999999998 2023-07-02T14:06:56.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.600 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:56.600 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.427208+0000", 2023-07-02T14:06:56.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028330849999999999 2023-07-02T14:06:56.600 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.600 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:56.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430041+0000", 2023-07-02T14:06:56.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00072642300000000002 2023-07-02T14:06:56.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:56.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430768+0000", 2023-07-02T14:06:56.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1847000000000002e-05 2023-07-02T14:06:56.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:56.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430800+0000", 2023-07-02T14:06:56.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9308999999999994e-05 2023-07-02T14:06:56.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:56.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430899+0000", 2023-07-02T14:06:56.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001 2023-07-02T14:06:56.603 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.603 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.603 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.603 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:410 2.5 2:bbaed619:::rbd_data.125de1449605.0000000000000109:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:06:56.604 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:56.604 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.090372850999998, 2023-07-02T14:06:56.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003, 2023-07-02T14:06:56.604 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:56.604 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:56.604 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:56.604 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:06:56.605 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:06:56.605 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:06:56.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.605 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:56.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:56.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:56.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:56.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:56.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:56.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:56.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:56.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:06:56.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.635e-06 2023-07-02T14:06:56.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:56.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450568+0000", 2023-07-02T14:06:56.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5970000000000002e-06 2023-07-02T14:06:56.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:56.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450573+0000", 2023-07-02T14:06:56.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3379999999999999e-06 2023-07-02T14:06:56.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:56.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450575+0000", 2023-07-02T14:06:56.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9458000000000001e-05 2023-07-02T14:06:56.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450615+0000", 2023-07-02T14:06:56.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0569999999999993e-06 2023-07-02T14:06:56.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:56.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450623+0000", 2023-07-02T14:06:56.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.63968709199999996 2023-07-02T14:06:56.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090310+0000", 2023-07-02T14:06:56.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020157399999999999 2023-07-02T14:06:56.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:56.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090512+0000", 2023-07-02T14:06:56.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027739190000000001 2023-07-02T14:06:56.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.612 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:56.612 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093285+0000", 2023-07-02T14:06:56.612 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003 2023-07-02T14:06:56.612 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.612 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.612 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.613 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:416 2.5 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:06:56.613 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:56.613 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 60.090343418000003, 2023-07-02T14:06:56.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002, 2023-07-02T14:06:56.613 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:56.613 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:56.613 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:56.614 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:06:56.614 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:06:56.614 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:06:56.614 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.614 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:56.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.614 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:56.614 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:56.615 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:56.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:56.615 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:56.615 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:56.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.616 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:56.616 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:06:56.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.34e-06 2023-07-02T14:06:56.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.616 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.616 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:56.616 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450599+0000", 2023-07-02T14:06:56.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1549999999999998e-06 2023-07-02T14:06:56.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:56.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450604+0000", 2023-07-02T14:06:56.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:06:56.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:56.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450607+0000", 2023-07-02T14:06:56.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5717999999999999e-05 2023-07-02T14:06:56.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450643+0000", 2023-07-02T14:06:56.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1619999999999999e-06 2023-07-02T14:06:56.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:56.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450648+0000", 2023-07-02T14:06:56.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64292291800000001 2023-07-02T14:06:56.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093571+0000", 2023-07-02T14:06:56.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115628 2023-07-02T14:06:56.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:56.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093687+0000", 2023-07-02T14:06:56.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002885552 2023-07-02T14:06:56.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:56.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.096572+0000", 2023-07-02T14:06:56.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002 2023-07-02T14:06:56.621 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.622 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.622 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.622 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18557 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:56.622 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:56.622 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 51.291971039000003, 2023-07-02T14:06:56.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13120991400000001, 2023-07-02T14:06:56.623 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:56.623 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:56.623 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:56.623 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:56.623 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:56.623 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18557 2023-07-02T14:06:56.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.623 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:56.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:56.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:56.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:56.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:56.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:56.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:06:56.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.625 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:56.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:06:56.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8870000000000001e-06 2023-07-02T14:06:56.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:56.626 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248971+0000", 2023-07-02T14:06:56.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9009999999999996e-06 2023-07-02T14:06:56.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:56.626 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248974+0000", 2023-07-02T14:06:56.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9809999999999997e-06 2023-07-02T14:06:56.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:56.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248978+0000", 2023-07-02T14:06:56.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00076989199999999999 2023-07-02T14:06:56.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249748+0000", 2023-07-02T14:06:56.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8245999999999999e-05 2023-07-02T14:06:56.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249787+0000", 2023-07-02T14:06:56.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066332969999999998 2023-07-02T14:06:56.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256420+0000", 2023-07-02T14:06:56.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2566e-05 2023-07-02T14:06:56.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256452+0000", 2023-07-02T14:06:56.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012364714000000001 2023-07-02T14:06:56.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268817+0000", 2023-07-02T14:06:56.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5113000000000001e-05 2023-07-02T14:06:56.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268852+0000", 2023-07-02T14:06:56.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010993899999999999 2023-07-02T14:06:56.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279846+0000", 2023-07-02T14:06:56.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2144999999999998e-05 2023-07-02T14:06:56.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279878+0000", 2023-07-02T14:06:56.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011778958000000001 2023-07-02T14:06:56.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291657+0000", 2023-07-02T14:06:56.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9592000000000001e-05 2023-07-02T14:06:56.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291687+0000", 2023-07-02T14:06:56.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081107220000000008 2023-07-02T14:06:56.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299798+0000", 2023-07-02T14:06:56.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0290999999999998e-05 2023-07-02T14:06:56.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299828+0000", 2023-07-02T14:06:56.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071584489999999999 2023-07-02T14:06:56.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306986+0000", 2023-07-02T14:06:56.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0755999999999999e-05 2023-07-02T14:06:56.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307047+0000", 2023-07-02T14:06:56.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057110260000000001 2023-07-02T14:06:56.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312758+0000", 2023-07-02T14:06:56.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5145999999999999e-05 2023-07-02T14:06:56.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312793+0000", 2023-07-02T14:06:56.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004758361 2023-07-02T14:06:56.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317552+0000", 2023-07-02T14:06:56.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9190000000000002e-05 2023-07-02T14:06:56.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317601+0000", 2023-07-02T14:06:56.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050329420000000003 2023-07-02T14:06:56.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322634+0000", 2023-07-02T14:06:56.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5737999999999998e-05 2023-07-02T14:06:56.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322719+0000", 2023-07-02T14:06:56.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064995679999999998 2023-07-02T14:06:56.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329219+0000", 2023-07-02T14:06:56.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0908000000000003e-05 2023-07-02T14:06:56.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329250+0000", 2023-07-02T14:06:56.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062072029999999997 2023-07-02T14:06:56.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335457+0000", 2023-07-02T14:06:56.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9471999999999998e-05 2023-07-02T14:06:56.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335487+0000", 2023-07-02T14:06:56.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097780239999999997 2023-07-02T14:06:56.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345265+0000", 2023-07-02T14:06:56.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0982999999999994e-05 2023-07-02T14:06:56.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345336+0000", 2023-07-02T14:06:56.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046378799999999996 2023-07-02T14:06:56.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349973+0000", 2023-07-02T14:06:56.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1069999999999999e-05 2023-07-02T14:06:56.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350005+0000", 2023-07-02T14:06:56.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042766870000000004 2023-07-02T14:06:56.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354281+0000", 2023-07-02T14:06:56.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1155000000000002e-05 2023-07-02T14:06:56.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354312+0000", 2023-07-02T14:06:56.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071436820000000002 2023-07-02T14:06:56.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361456+0000", 2023-07-02T14:06:56.647 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9054000000000002e-05 2023-07-02T14:06:56.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361505+0000", 2023-07-02T14:06:56.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072301320000000002 2023-07-02T14:06:56.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.648 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368735+0000", 2023-07-02T14:06:56.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1013999999999994e-05 2023-07-02T14:06:56.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368816+0000", 2023-07-02T14:06:56.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036582490000000001 2023-07-02T14:06:56.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372475+0000", 2023-07-02T14:06:56.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4652000000000004e-05 2023-07-02T14:06:56.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:56.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372539+0000", 2023-07-02T14:06:56.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00031374199999999999 2023-07-02T14:06:56.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:56.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372853+0000", 2023-07-02T14:06:56.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030071299999999998 2023-07-02T14:06:56.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:56.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.375860+0000", 2023-07-02T14:06:56.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041897749999999997 2023-07-02T14:06:56.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:56.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380050+0000", 2023-07-02T14:06:56.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8331000000000003e-05 2023-07-02T14:06:56.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:56.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380088+0000", 2023-07-02T14:06:56.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0626999999999998e-05 2023-07-02T14:06:56.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:56.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380179+0000", 2023-07-02T14:06:56.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13120991400000001 2023-07-02T14:06:56.654 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.654 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.654 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.655 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18558 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:56.655 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:56.655 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 51.291954337, 2023-07-02T14:06:56.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13755567899999999, 2023-07-02T14:06:56.655 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:56.655 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:56.655 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:56.656 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:56.656 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:56.656 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18558 2023-07-02T14:06:56.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.656 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:56.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:56.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:56.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:56.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:56.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:06:56.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:56.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:56.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248985+0000", 2023-07-02T14:06:56.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8630000000000001e-06 2023-07-02T14:06:56.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:56.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248989+0000", 2023-07-02T14:06:56.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0750000000000003e-06 2023-07-02T14:06:56.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:56.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248995+0000", 2023-07-02T14:06:56.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3290000000000004e-06 2023-07-02T14:06:56.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:56.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248999+0000", 2023-07-02T14:06:56.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00080656300000000005 2023-07-02T14:06:56.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249806+0000", 2023-07-02T14:06:56.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2040999999999999e-05 2023-07-02T14:06:56.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249848+0000", 2023-07-02T14:06:56.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066312419999999999 2023-07-02T14:06:56.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256479+0000", 2023-07-02T14:06:56.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3348e-05 2023-07-02T14:06:56.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256513+0000", 2023-07-02T14:06:56.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012391531000000001 2023-07-02T14:06:56.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268904+0000", 2023-07-02T14:06:56.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.347e-05 2023-07-02T14:06:56.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268978+0000", 2023-07-02T14:06:56.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010929772000000001 2023-07-02T14:06:56.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279907+0000", 2023-07-02T14:06:56.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2669999999999997e-05 2023-07-02T14:06:56.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279940+0000", 2023-07-02T14:06:56.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011770332999999999 2023-07-02T14:06:56.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291710+0000", 2023-07-02T14:06:56.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1207000000000001e-05 2023-07-02T14:06:56.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291742+0000", 2023-07-02T14:06:56.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081242199999999997 2023-07-02T14:06:56.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299866+0000", 2023-07-02T14:06:56.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4650000000000002e-05 2023-07-02T14:06:56.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299900+0000", 2023-07-02T14:06:56.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071810270000000004 2023-07-02T14:06:56.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307081+0000", 2023-07-02T14:06:56.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4378000000000002e-05 2023-07-02T14:06:56.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307116+0000", 2023-07-02T14:06:56.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057031909999999998 2023-07-02T14:06:56.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312819+0000", 2023-07-02T14:06:56.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1612999999999997e-05 2023-07-02T14:06:56.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312851+0000", 2023-07-02T14:06:56.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048172470000000002 2023-07-02T14:06:56.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317668+0000", 2023-07-02T14:06:56.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3328e-05 2023-07-02T14:06:56.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317741+0000", 2023-07-02T14:06:56.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050117290000000004 2023-07-02T14:06:56.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322753+0000", 2023-07-02T14:06:56.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9872000000000001e-05 2023-07-02T14:06:56.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322783+0000", 2023-07-02T14:06:56.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064910610000000002 2023-07-02T14:06:56.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329274+0000", 2023-07-02T14:06:56.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0097e-05 2023-07-02T14:06:56.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329304+0000", 2023-07-02T14:06:56.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062163879999999998 2023-07-02T14:06:56.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335520+0000", 2023-07-02T14:06:56.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9513e-05 2023-07-02T14:06:56.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335550+0000", 2023-07-02T14:06:56.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098209969999999997 2023-07-02T14:06:56.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345371+0000", 2023-07-02T14:06:56.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9862999999999999e-05 2023-07-02T14:06:56.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345401+0000", 2023-07-02T14:06:56.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046552030000000001 2023-07-02T14:06:56.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350056+0000", 2023-07-02T14:06:56.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8751999999999993e-05 2023-07-02T14:06:56.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350145+0000", 2023-07-02T14:06:56.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041981450000000003 2023-07-02T14:06:56.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354343+0000", 2023-07-02T14:06:56.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1501000000000002e-05 2023-07-02T14:06:56.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354374+0000", 2023-07-02T14:06:56.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071551740000000003 2023-07-02T14:06:56.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361530+0000", 2023-07-02T14:06:56.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5156e-05 2023-07-02T14:06:56.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361565+0000", 2023-07-02T14:06:56.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072918569999999997 2023-07-02T14:06:56.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368857+0000", 2023-07-02T14:06:56.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5770999999999999e-05 2023-07-02T14:06:56.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368902+0000", 2023-07-02T14:06:56.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057210569999999999 2023-07-02T14:06:56.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374623+0000", 2023-07-02T14:06:56.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3266999999999998e-05 2023-07-02T14:06:56.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374667+0000", 2023-07-02T14:06:56.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055479570000000001 2023-07-02T14:06:56.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380215+0000", 2023-07-02T14:06:56.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0067e-05 2023-07-02T14:06:56.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:56.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380265+0000", 2023-07-02T14:06:56.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00043082299999999999 2023-07-02T14:06:56.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:56.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380696+0000", 2023-07-02T14:06:56.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024853240000000001 2023-07-02T14:06:56.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:56.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.383181+0000", 2023-07-02T14:06:56.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032601280000000002 2023-07-02T14:06:56.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:56.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386441+0000", 2023-07-02T14:06:56.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9748e-05 2023-07-02T14:06:56.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:56.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386471+0000", 2023-07-02T14:06:56.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0531000000000006e-05 2023-07-02T14:06:56.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:56.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386541+0000", 2023-07-02T14:06:56.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13755567899999999 2023-07-02T14:06:56.687 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.688 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.688 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.688 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18559 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:56.688 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:56.688 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 51.291931732999998, 2023-07-02T14:06:56.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.146184591, 2023-07-02T14:06:56.689 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:56.689 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:56.689 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:56.689 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:56.689 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:56.689 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18559 2023-07-02T14:06:56.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.690 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:56.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:56.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:56.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:56.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:56.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:56.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:56.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:56.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:06:56.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.726e-06 2023-07-02T14:06:56.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:56.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249011+0000", 2023-07-02T14:06:56.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7939999999999997e-06 2023-07-02T14:06:56.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:56.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249018+0000", 2023-07-02T14:06:56.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3470000000000001e-06 2023-07-02T14:06:56.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:56.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249023+0000", 2023-07-02T14:06:56.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00084402900000000005 2023-07-02T14:06:56.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249867+0000", 2023-07-02T14:06:56.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9050000000000001e-05 2023-07-02T14:06:56.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249906+0000", 2023-07-02T14:06:56.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066311039999999996 2023-07-02T14:06:56.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256537+0000", 2023-07-02T14:06:56.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2790000000000003e-05 2023-07-02T14:06:56.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256570+0000", 2023-07-02T14:06:56.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012436129000000001 2023-07-02T14:06:56.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269006+0000", 2023-07-02T14:06:56.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0173999999999999e-05 2023-07-02T14:06:56.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269046+0000", 2023-07-02T14:06:56.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010924698 2023-07-02T14:06:56.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279971+0000", 2023-07-02T14:06:56.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3831000000000002e-05 2023-07-02T14:06:56.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280004+0000", 2023-07-02T14:06:56.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011795912 2023-07-02T14:06:56.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291800+0000", 2023-07-02T14:06:56.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5613999999999997e-05 2023-07-02T14:06:56.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291886+0000", 2023-07-02T14:06:56.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080395999999999992 2023-07-02T14:06:56.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299925+0000", 2023-07-02T14:06:56.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0694999999999997e-05 2023-07-02T14:06:56.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299956+0000", 2023-07-02T14:06:56.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072049449999999999 2023-07-02T14:06:56.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307161+0000", 2023-07-02T14:06:56.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5523000000000001e-05 2023-07-02T14:06:56.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307247+0000", 2023-07-02T14:06:56.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056331929999999999 2023-07-02T14:06:56.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312880+0000", 2023-07-02T14:06:56.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1646000000000002e-05 2023-07-02T14:06:56.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312911+0000", 2023-07-02T14:06:56.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048627540000000004 2023-07-02T14:06:56.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317774+0000", 2023-07-02T14:06:56.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2008000000000003e-05 2023-07-02T14:06:56.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317806+0000", 2023-07-02T14:06:56.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050055689999999996 2023-07-02T14:06:56.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322812+0000", 2023-07-02T14:06:56.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3442999999999997e-05 2023-07-02T14:06:56.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322855+0000", 2023-07-02T14:06:56.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006472813 2023-07-02T14:06:56.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329328+0000", 2023-07-02T14:06:56.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0128e-05 2023-07-02T14:06:56.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329358+0000", 2023-07-02T14:06:56.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062156490000000002 2023-07-02T14:06:56.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335574+0000", 2023-07-02T14:06:56.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2357999999999997e-05 2023-07-02T14:06:56.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335606+0000", 2023-07-02T14:06:56.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098318409999999992 2023-07-02T14:06:56.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345438+0000", 2023-07-02T14:06:56.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1174000000000007e-05 2023-07-02T14:06:56.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345509+0000", 2023-07-02T14:06:56.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004669851 2023-07-02T14:06:56.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350179+0000", 2023-07-02T14:06:56.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1559000000000001e-05 2023-07-02T14:06:56.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350211+0000", 2023-07-02T14:06:56.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041865740000000002 2023-07-02T14:06:56.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354397+0000", 2023-07-02T14:06:56.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9850000000000001e-05 2023-07-02T14:06:56.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354427+0000", 2023-07-02T14:06:56.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071608449999999999 2023-07-02T14:06:56.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361588+0000", 2023-07-02T14:06:56.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8346000000000002e-05 2023-07-02T14:06:56.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361626+0000", 2023-07-02T14:06:56.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073175009999999997 2023-07-02T14:06:56.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368944+0000", 2023-07-02T14:06:56.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2879000000000005e-05 2023-07-02T14:06:56.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369017+0000", 2023-07-02T14:06:56.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056850690000000001 2023-07-02T14:06:56.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374702+0000", 2023-07-02T14:06:56.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7693e-05 2023-07-02T14:06:56.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374739+0000", 2023-07-02T14:06:56.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076925149999999996 2023-07-02T14:06:56.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382432+0000", 2023-07-02T14:06:56.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3259e-05 2023-07-02T14:06:56.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382475+0000", 2023-07-02T14:06:56.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041005110000000003 2023-07-02T14:06:56.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386576+0000", 2023-07-02T14:06:56.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2074999999999999e-05 2023-07-02T14:06:56.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:56.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386618+0000", 2023-07-02T14:06:56.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000233692 2023-07-02T14:06:56.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:56.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386851+0000", 2023-07-02T14:06:56.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024022269999999998 2023-07-02T14:06:56.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:56.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.389254+0000", 2023-07-02T14:06:56.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058506469999999996 2023-07-02T14:06:56.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:56.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395104+0000", 2023-07-02T14:06:56.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8807000000000003e-05 2023-07-02T14:06:56.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:56.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395143+0000", 2023-07-02T14:06:56.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9616000000000001e-05 2023-07-02T14:06:56.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:56.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395193+0000", 2023-07-02T14:06:56.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.146184591 2023-07-02T14:06:56.723 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.723 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.723 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.723 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18560 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:56.723 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:56.724 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 51.291913110000003, 2023-07-02T14:06:56.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.15647551900000001, 2023-07-02T14:06:56.724 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:56.724 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:56.724 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:56.724 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:56.724 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:56.724 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18560 2023-07-02T14:06:56.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.725 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:56.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:56.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:56.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:56.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:56.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:06:56.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:56.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:56.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249026+0000", 2023-07-02T14:06:56.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3419999999999999e-06 2023-07-02T14:06:56.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:56.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249030+0000", 2023-07-02T14:06:56.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7869999999999998e-06 2023-07-02T14:06:56.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:56.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249035+0000", 2023-07-02T14:06:56.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9639999999999997e-06 2023-07-02T14:06:56.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:56.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249039+0000", 2023-07-02T14:06:56.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00088477599999999999 2023-07-02T14:06:56.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249924+0000", 2023-07-02T14:06:56.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7333999999999999e-05 2023-07-02T14:06:56.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249962+0000", 2023-07-02T14:06:56.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066323409999999999 2023-07-02T14:06:56.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256594+0000", 2023-07-02T14:06:56.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2595999999999998e-05 2023-07-02T14:06:56.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256627+0000", 2023-07-02T14:06:56.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012449894 2023-07-02T14:06:56.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269076+0000", 2023-07-02T14:06:56.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3942999999999997e-05 2023-07-02T14:06:56.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269110+0000", 2023-07-02T14:06:56.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010923884 2023-07-02T14:06:56.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280034+0000", 2023-07-02T14:06:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7774000000000001e-05 2023-07-02T14:06:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280072+0000", 2023-07-02T14:06:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011850027000000001 2023-07-02T14:06:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291922+0000", 2023-07-02T14:06:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2128000000000001e-05 2023-07-02T14:06:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291954+0000", 2023-07-02T14:06:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080261150000000003 2023-07-02T14:06:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299980+0000", 2023-07-02T14:06:56.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6322999999999998e-05 2023-07-02T14:06:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300017+0000", 2023-07-02T14:06:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072683039999999997 2023-07-02T14:06:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307285+0000", 2023-07-02T14:06:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3309000000000002e-05 2023-07-02T14:06:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307318+0000", 2023-07-02T14:06:56.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056178649999999997 2023-07-02T14:06:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312936+0000", 2023-07-02T14:06:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1866000000000003e-05 2023-07-02T14:06:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312968+0000", 2023-07-02T14:06:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048789970000000004 2023-07-02T14:06:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317847+0000", 2023-07-02T14:06:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0223999999999996e-05 2023-07-02T14:06:56.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317927+0000", 2023-07-02T14:06:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049543649999999996 2023-07-02T14:06:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322882+0000", 2023-07-02T14:06:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1399000000000002e-05 2023-07-02T14:06:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322913+0000", 2023-07-02T14:06:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006534214 2023-07-02T14:06:56.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329447+0000", 2023-07-02T14:06:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9649999999999998e-05 2023-07-02T14:06:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329527+0000", 2023-07-02T14:06:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061026839999999997 2023-07-02T14:06:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335629+0000", 2023-07-02T14:06:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9611000000000002e-05 2023-07-02T14:06:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335659+0000", 2023-07-02T14:06:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098851549999999996 2023-07-02T14:06:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345544+0000", 2023-07-02T14:06:56.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0587999999999999e-05 2023-07-02T14:06:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345575+0000", 2023-07-02T14:06:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046790859999999998 2023-07-02T14:06:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350254+0000", 2023-07-02T14:06:56.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0284000000000002e-05 2023-07-02T14:06:56.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350304+0000", 2023-07-02T14:06:56.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041560540000000002 2023-07-02T14:06:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354460+0000", 2023-07-02T14:06:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2427000000000004e-05 2023-07-02T14:06:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354523+0000", 2023-07-02T14:06:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071268770000000002 2023-07-02T14:06:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361650+0000", 2023-07-02T14:06:56.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0026999999999999e-05 2023-07-02T14:06:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361680+0000", 2023-07-02T14:06:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073733080000000003 2023-07-02T14:06:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369053+0000", 2023-07-02T14:06:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7635999999999997e-05 2023-07-02T14:06:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369091+0000", 2023-07-02T14:06:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056908059999999996 2023-07-02T14:06:56.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374781+0000", 2023-07-02T14:06:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4656000000000002e-05 2023-07-02T14:06:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374816+0000", 2023-07-02T14:06:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077037809999999998 2023-07-02T14:06:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382520+0000", 2023-07-02T14:06:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7858e-05 2023-07-02T14:06:56.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382588+0000", 2023-07-02T14:06:56.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057884019999999998 2023-07-02T14:06:56.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388376+0000", 2023-07-02T14:06:56.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4831000000000001e-05 2023-07-02T14:06:56.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388461+0000", 2023-07-02T14:06:56.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067708480000000003 2023-07-02T14:06:56.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395232+0000", 2023-07-02T14:06:56.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7948000000000002e-05 2023-07-02T14:06:56.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:56.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395290+0000", 2023-07-02T14:06:56.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00027672599999999998 2023-07-02T14:06:56.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:56.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395566+0000", 2023-07-02T14:06:56.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025705569999999998 2023-07-02T14:06:56.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:56.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.398137+0000", 2023-07-02T14:06:56.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072257099999999998 2023-07-02T14:06:56.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:56.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405363+0000", 2023-07-02T14:06:56.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3887999999999998e-05 2023-07-02T14:06:56.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:56.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405397+0000", 2023-07-02T14:06:56.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000105769 2023-07-02T14:06:56.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:56.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405502+0000", 2023-07-02T14:06:56.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.15647551900000001 2023-07-02T14:06:56.759 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.759 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.759 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.759 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18561 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:06:56.759 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:56.760 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 51.291892746000002, 2023-07-02T14:06:56.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744, 2023-07-02T14:06:56.760 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:56.760 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:56.760 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:56.760 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:56.760 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18561 2023-07-02T14:06:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:56.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:56.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:56.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:56.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:06:56.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.563e-06 2023-07-02T14:06:56.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:56.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249050+0000", 2023-07-02T14:06:56.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4500000000000001e-06 2023-07-02T14:06:56.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249057+0000", 2023-07-02T14:06:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1860000000000002e-06 2023-07-02T14:06:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:56.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249061+0000", 2023-07-02T14:06:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00091153399999999998 2023-07-02T14:06:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249972+0000", 2023-07-02T14:06:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7156999999999997e-05 2023-07-02T14:06:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.250010+0000", 2023-07-02T14:06:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006641816 2023-07-02T14:06:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256651+0000", 2023-07-02T14:06:56.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1968999999999998e-05 2023-07-02T14:06:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256683+0000", 2023-07-02T14:06:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012463952 2023-07-02T14:06:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269147+0000", 2023-07-02T14:06:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2855000000000006e-05 2023-07-02T14:06:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269220+0000", 2023-07-02T14:06:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010870418999999999 2023-07-02T14:06:56.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280091+0000", 2023-07-02T14:06:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1921999999999997e-05 2023-07-02T14:06:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280123+0000", 2023-07-02T14:06:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011861682 2023-07-02T14:06:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291984+0000", 2023-07-02T14:06:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5259000000000006e-05 2023-07-02T14:06:56.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.292059+0000", 2023-07-02T14:06:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079709359999999996 2023-07-02T14:06:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300030+0000", 2023-07-02T14:06:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0729999999999998e-05 2023-07-02T14:06:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300071+0000", 2023-07-02T14:06:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072785690000000004 2023-07-02T14:06:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307350+0000", 2023-07-02T14:06:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1948999999999999e-05 2023-07-02T14:06:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307412+0000", 2023-07-02T14:06:56.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055785779999999998 2023-07-02T14:06:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312990+0000", 2023-07-02T14:06:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0957000000000002e-05 2023-07-02T14:06:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.313021+0000", 2023-07-02T14:06:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049246400000000001 2023-07-02T14:06:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317946+0000", 2023-07-02T14:06:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9346e-05 2023-07-02T14:06:56.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317975+0000", 2023-07-02T14:06:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049649469999999999 2023-07-02T14:06:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322940+0000", 2023-07-02T14:06:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9017000000000002e-05 2023-07-02T14:06:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322989+0000", 2023-07-02T14:06:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065907029999999998 2023-07-02T14:06:56.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329580+0000", 2023-07-02T14:06:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5879999999999999e-05 2023-07-02T14:06:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329656+0000", 2023-07-02T14:06:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060166960000000002 2023-07-02T14:06:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335672+0000", 2023-07-02T14:06:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8963e-05 2023-07-02T14:06:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335701+0000", 2023-07-02T14:06:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098989560000000004 2023-07-02T14:06:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345600+0000", 2023-07-02T14:06:56.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5395999999999995e-05 2023-07-02T14:06:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345666+0000", 2023-07-02T14:06:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046589530000000004 2023-07-02T14:06:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350325+0000", 2023-07-02T14:06:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2459000000000002e-05 2023-07-02T14:06:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350357+0000", 2023-07-02T14:06:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042054290000000001 2023-07-02T14:06:56.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354563+0000", 2023-07-02T14:06:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5612999999999999e-05 2023-07-02T14:06:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354598+0000", 2023-07-02T14:06:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070967449999999998 2023-07-02T14:06:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361695+0000", 2023-07-02T14:06:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9314000000000001e-05 2023-07-02T14:06:56.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361724+0000", 2023-07-02T14:06:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007391028 2023-07-02T14:06:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369115+0000", 2023-07-02T14:06:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9163000000000002e-05 2023-07-02T14:06:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369184+0000", 2023-07-02T14:06:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056532500000000003 2023-07-02T14:06:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374838+0000", 2023-07-02T14:06:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5297999999999999e-05 2023-07-02T14:06:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374883+0000", 2023-07-02T14:06:56.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077292189999999998 2023-07-02T14:06:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382612+0000", 2023-07-02T14:06:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1102000000000001e-05 2023-07-02T14:06:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382653+0000", 2023-07-02T14:06:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058515989999999999 2023-07-02T14:06:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388505+0000", 2023-07-02T14:06:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9196000000000006e-05 2023-07-02T14:06:56.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388574+0000", 2023-07-02T14:06:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084245339999999991 2023-07-02T14:06:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.396999+0000", 2023-07-02T14:06:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0596999999999998e-05 2023-07-02T14:06:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:06:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.397039+0000", 2023-07-02T14:06:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084911339999999991 2023-07-02T14:06:56.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405530+0000", 2023-07-02T14:06:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7555000000000001e-05 2023-07-02T14:06:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:56.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405578+0000", 2023-07-02T14:06:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000229607 2023-07-02T14:06:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405808+0000", 2023-07-02T14:06:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026919940000000001 2023-07-02T14:06:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.408500+0000", 2023-07-02T14:06:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053065059999999999 2023-07-02T14:06:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413806+0000", 2023-07-02T14:06:56.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0490000000000001e-05 2023-07-02T14:06:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413837+0000", 2023-07-02T14:06:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000100386 2023-07-02T14:06:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413937+0000", 2023-07-02T14:06:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744 2023-07-02T14:06:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.796 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20768 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch ping cookie 140544751726928 gen 52] snapc 0=[] ondisk+write+known_if_redirected e183)", 2023-07-02T14:06:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.318525477999998, 2023-07-02T14:06:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3751256970000001, 2023-07-02T14:06:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:56.797 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20768 2023-07-02T14:06:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:56.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:06:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7829999999999998e-06 2023-07-02T14:06:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222419+0000", 2023-07-02T14:06:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1380000000000004e-06 2023-07-02T14:06:56.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222425+0000", 2023-07-02T14:06:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2519999999999996e-06 2023-07-02T14:06:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:56.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222431+0000", 2023-07-02T14:06:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2157999999999994e-05 2023-07-02T14:06:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222523+0000", 2023-07-02T14:06:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5147000000000001e-05 2023-07-02T14:06:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222538+0000", 2023-07-02T14:06:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30105419700000002 2023-07-02T14:06:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523592+0000", 2023-07-02T14:06:56.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2316e-05 2023-07-02T14:06:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523605+0000", 2023-07-02T14:06:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.073658953 2023-07-02T14:06:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597264+0000", 2023-07-02T14:06:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000213415 2023-07-02T14:06:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597477+0000", 2023-07-02T14:06:56.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3034999999999995e-05 2023-07-02T14:06:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597540+0000", 2023-07-02T14:06:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3751256970000001 2023-07-02T14:06:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.806 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20769 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 53] snapc 0=[] ondisk+write+known_if_redirected e184)", 2023-07-02T14:06:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.318163022999997, 2023-07-02T14:06:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.380884139, 2023-07-02T14:06:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:56.807 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20769 2023-07-02T14:06:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:56.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:06:56.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222776+0000", 2023-07-02T14:06:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9710000000000003e-06 2023-07-02T14:06:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222781+0000", 2023-07-02T14:06:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9899999999999992e-06 2023-07-02T14:06:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222791+0000", 2023-07-02T14:06:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4240000000000003e-06 2023-07-02T14:06:56.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222795+0000", 2023-07-02T14:06:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2366000000000001e-05 2023-07-02T14:06:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222828+0000", 2023-07-02T14:06:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4790000000000001e-06 2023-07-02T14:06:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222834+0000", 2023-07-02T14:06:56.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30084589299999998 2023-07-02T14:06:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523680+0000", 2023-07-02T14:06:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.87e-06 2023-07-02T14:06:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:56.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523685+0000", 2023-07-02T14:06:56.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.074111042 2023-07-02T14:06:56.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597796+0000", 2023-07-02T14:06:56.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0411000000000001e-05 2023-07-02T14:06:56.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:56.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597856+0000", 2023-07-02T14:06:56.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058045340000000001 2023-07-02T14:06:56.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:56.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.603661+0000", 2023-07-02T14:06:56.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.380884139 2023-07-02T14:06:56.817 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.817 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.817 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.817 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20770 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 54] snapc 0=[] ondisk+write+known_if_redirected e185)", 2023-07-02T14:06:56.817 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:56.818 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 37.016945913000001, 2023-07-02T14:06:56.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0799819349999999, 2023-07-02T14:06:56.818 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:56.818 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:56.818 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:56.818 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:56.818 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:56.818 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20770 2023-07-02T14:06:56.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.819 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:56.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:56.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:56.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:56.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:56.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:06:56.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:56.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:56.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523992+0000", 2023-07-02T14:06:56.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6720000000000004e-06 2023-07-02T14:06:56.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:56.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524000+0000", 2023-07-02T14:06:56.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3711e-05 2023-07-02T14:06:56.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:56.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524014+0000", 2023-07-02T14:06:56.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7849999999999992e-06 2023-07-02T14:06:56.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:56.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524022+0000", 2023-07-02T14:06:56.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2366000000000001e-05 2023-07-02T14:06:56.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524054+0000", 2023-07-02T14:06:56.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8169999999999998e-06 2023-07-02T14:06:56.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:56.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524059+0000", 2023-07-02T14:06:56.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0748338850000001 2023-07-02T14:06:56.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.598893+0000", 2023-07-02T14:06:56.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116968 2023-07-02T14:06:56.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:56.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.599010+0000", 2023-07-02T14:06:56.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049663370000000004 2023-07-02T14:06:56.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:56.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.603976+0000", 2023-07-02T14:06:56.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0799819349999999 2023-07-02T14:06:56.826 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.826 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.826 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.826 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20833 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 55] snapc 0=[] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:56.827 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:06:56.827 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.008826608, 2023-07-02T14:06:56.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.029349206, 2023-07-02T14:06:56.827 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:56.827 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:06:56.827 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:56.827 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:56.827 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:56.828 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20833 2023-07-02T14:06:56.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.828 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:56.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:56.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:06:56.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:56.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:56.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:06:56.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:56.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:56.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532112+0000", 2023-07-02T14:06:56.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7590000000000003e-06 2023-07-02T14:06:56.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:56.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532120+0000", 2023-07-02T14:06:56.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8130000000000001e-05 2023-07-02T14:06:56.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:56.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532139+0000", 2023-07-02T14:06:56.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3299999999999999e-06 2023-07-02T14:06:56.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:56.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532147+0000", 2023-07-02T14:06:56.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9679999999999998e-05 2023-07-02T14:06:56.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532207+0000", 2023-07-02T14:06:56.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0648e-05 2023-07-02T14:06:56.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op must wait for map", 2023-07-02T14:06:56.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532217+0000", 2023-07-02T14:06:56.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002082455 2023-07-02T14:06:56.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.534300+0000", 2023-07-02T14:06:56.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0710000000000001e-05 2023-07-02T14:06:56.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:56.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.534310+0000", 2023-07-02T14:06:56.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.013227235 2023-07-02T14:06:56.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.547538+0000", 2023-07-02T14:06:56.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021904300000000001 2023-07-02T14:06:56.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:56.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.547757+0000", 2023-07-02T14:06:56.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013705784 2023-07-02T14:06:56.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:56.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561462+0000", 2023-07-02T14:06:56.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.029349206 2023-07-02T14:06:56.836 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.836 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.836 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.837 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20865 2.7 2:ff0f5f99:::rbd_data.125de1449605.0000000000000006:head [write 131072~8192] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:56.837 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:06:56.837 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.264203536, 2023-07-02T14:06:56.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.288115762, 2023-07-02T14:06:56.837 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:56.838 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:56.838 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:56.838 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:56.838 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:56.838 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20865 2023-07-02T14:06:56.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.838 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:56.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:56.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:06:56.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:56.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:56.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:06:56.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:56.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:56.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276735+0000", 2023-07-02T14:06:56.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3273e-05 2023-07-02T14:06:56.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:56.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276748+0000", 2023-07-02T14:06:56.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3895e-05 2023-07-02T14:06:56.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:56.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276762+0000", 2023-07-02T14:06:56.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.896e-06 2023-07-02T14:06:56.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:56.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276768+0000", 2023-07-02T14:06:56.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5850999999999999e-05 2023-07-02T14:06:56.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276814+0000", 2023-07-02T14:06:56.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5290000000000003e-06 2023-07-02T14:06:56.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:56.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276821+0000", 2023-07-02T14:06:56.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.27225312600000001 2023-07-02T14:06:56.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549074+0000", 2023-07-02T14:06:56.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000151223 2023-07-02T14:06:56.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:56.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549226+0000", 2023-07-02T14:06:56.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024865400000000001 2023-07-02T14:06:56.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:56.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549474+0000", 2023-07-02T14:06:56.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012039185000000001 2023-07-02T14:06:56.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:56.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561513+0000", 2023-07-02T14:06:56.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003221327 2023-07-02T14:06:56.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:56.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564735+0000", 2023-07-02T14:06:56.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2026000000000005e-05 2023-07-02T14:06:56.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:56.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564807+0000", 2023-07-02T14:06:56.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5303000000000003e-05 2023-07-02T14:06:56.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:56.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564852+0000", 2023-07-02T14:06:56.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.288115762 2023-07-02T14:06:56.849 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.849 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.849 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.850 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20866 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 131072~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:56.850 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:06:56.850 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.258510215000001, 2023-07-02T14:06:56.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.28773701200000001, 2023-07-02T14:06:56.850 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:56.850 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:56.850 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:56.850 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:56.851 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:56.851 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20866 2023-07-02T14:06:56.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.851 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:56.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:56.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:06:56.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:56.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:56.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:06:56.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:56.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:56.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282428+0000", 2023-07-02T14:06:56.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8469999999999998e-05 2023-07-02T14:06:56.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:56.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282517+0000", 2023-07-02T14:06:56.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2974e-05 2023-07-02T14:06:56.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:56.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282530+0000", 2023-07-02T14:06:56.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4530000000000001e-06 2023-07-02T14:06:56.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:56.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282535+0000", 2023-07-02T14:06:56.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1926999999999999e-05 2023-07-02T14:06:56.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282587+0000", 2023-07-02T14:06:56.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3083e-05 2023-07-02T14:06:56.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:56.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282600+0000", 2023-07-02T14:06:56.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.270692971 2023-07-02T14:06:56.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553293+0000", 2023-07-02T14:06:56.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017065599999999999 2023-07-02T14:06:56.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:56.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553464+0000", 2023-07-02T14:06:56.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026504399999999998 2023-07-02T14:06:56.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:56.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553729+0000", 2023-07-02T14:06:56.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011173008 2023-07-02T14:06:56.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:56.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564902+0000", 2023-07-02T14:06:56.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051242600000000003 2023-07-02T14:06:56.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:56.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570026+0000", 2023-07-02T14:06:56.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8098999999999999e-05 2023-07-02T14:06:56.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:56.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570104+0000", 2023-07-02T14:06:56.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2323000000000006e-05 2023-07-02T14:06:56.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:56.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570167+0000", 2023-07-02T14:06:56.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.28773701200000001 2023-07-02T14:06:56.861 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.861 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.861 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.861 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20875 2.7 2:e3210266:::rbd_data.125de1449605.00000000000001a3:head [write 0~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:56.862 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:06:56.862 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.219459399000002, 2023-07-02T14:06:56.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.25297039599999999, 2023-07-02T14:06:56.862 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:56.862 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:56.862 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:56.862 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:56.862 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:56.863 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20875 2023-07-02T14:06:56.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.863 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:56.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:56.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:06:56.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:56.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:56.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:06:56.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:56.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:56.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321479+0000", 2023-07-02T14:06:56.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015194399999999999 2023-07-02T14:06:56.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:56.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321631+0000", 2023-07-02T14:06:56.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4236999999999999e-05 2023-07-02T14:06:56.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:56.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321645+0000", 2023-07-02T14:06:56.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2160000000000003e-06 2023-07-02T14:06:56.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:56.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321651+0000", 2023-07-02T14:06:56.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3609999999999998e-05 2023-07-02T14:06:56.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321695+0000", 2023-07-02T14:06:56.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1989999999999995e-06 2023-07-02T14:06:56.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:56.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321704+0000", 2023-07-02T14:06:56.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.23542881500000001 2023-07-02T14:06:56.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557133+0000", 2023-07-02T14:06:56.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017359600000000001 2023-07-02T14:06:56.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:56.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557307+0000", 2023-07-02T14:06:56.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00023367599999999999 2023-07-02T14:06:56.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:56.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557540+0000", 2023-07-02T14:06:56.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016744963000000002 2023-07-02T14:06:56.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:56.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574285+0000", 2023-07-02T14:06:56.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1330999999999999e-05 2023-07-02T14:06:56.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:56.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574346+0000", 2023-07-02T14:06:56.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9750000000000002e-05 2023-07-02T14:06:56.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:56.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574406+0000", 2023-07-02T14:06:56.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4647999999999997e-05 2023-07-02T14:06:56.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:56.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574451+0000", 2023-07-02T14:06:56.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.25297039599999999 2023-07-02T14:06:56.872 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.873 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.873 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.873 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20915 2.7 2:e6e64067:::rbd_data.125de1449605.00000000000001a9:head [write 0~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:56.873 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:06:56.873 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.156205604, 2023-07-02T14:06:56.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.19966078500000001, 2023-07-02T14:06:56.874 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:56.874 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:56.874 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:56.874 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:56.874 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:56.874 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20915 2023-07-02T14:06:56.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.875 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:56.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:56.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:06:56.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:56.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:56.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:06:56.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:56.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:56.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384732+0000", 2023-07-02T14:06:56.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9439999999999999e-05 2023-07-02T14:06:56.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:56.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384802+0000", 2023-07-02T14:06:56.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7572999999999999e-05 2023-07-02T14:06:56.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:56.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384819+0000", 2023-07-02T14:06:56.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1670000000000004e-06 2023-07-02T14:06:56.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:56.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384825+0000", 2023-07-02T14:06:56.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6471999999999998e-05 2023-07-02T14:06:56.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384872+0000", 2023-07-02T14:06:56.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0028e-05 2023-07-02T14:06:56.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:56.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384882+0000", 2023-07-02T14:06:56.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.17674214099999999 2023-07-02T14:06:56.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561624+0000", 2023-07-02T14:06:56.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000135809 2023-07-02T14:06:56.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:56.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561760+0000", 2023-07-02T14:06:56.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020468 2023-07-02T14:06:56.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:56.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561965+0000", 2023-07-02T14:06:56.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012498198 2023-07-02T14:06:56.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:56.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574463+0000", 2023-07-02T14:06:56.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098197509999999998 2023-07-02T14:06:56.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:56.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584283+0000", 2023-07-02T14:06:56.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2056000000000004e-05 2023-07-02T14:06:56.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:56.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584345+0000", 2023-07-02T14:06:56.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0513000000000002e-05 2023-07-02T14:06:56.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:56.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584395+0000", 2023-07-02T14:06:56.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.19966078500000001 2023-07-02T14:06:56.884 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.884 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.884 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.885 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20948 2.7 2:ff0f5f99:::rbd_data.125de1449605.0000000000000006:head [write 139264~2220032] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:56.885 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.450499+0000", 2023-07-02T14:06:56.885 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.090440876999999, 2023-07-02T14:06:56.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13400999399999999, 2023-07-02T14:06:56.885 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:56.886 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:56.886 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:56.886 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:56.886 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:56.886 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20948 2023-07-02T14:06:56.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.886 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:56.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:56.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.450499+0000", 2023-07-02T14:06:56.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:56.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:56.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.450499+0000", 2023-07-02T14:06:56.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:56.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:56.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.450497+0000", 2023-07-02T14:06:56.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043010360000000003 2023-07-02T14:06:56.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:56.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454798+0000", 2023-07-02T14:06:56.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8073000000000001e-05 2023-07-02T14:06:56.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:56.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454816+0000", 2023-07-02T14:06:56.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3769999999999996e-06 2023-07-02T14:06:56.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:56.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454825+0000", 2023-07-02T14:06:56.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2797999999999998e-05 2023-07-02T14:06:56.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454878+0000", 2023-07-02T14:06:56.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0020999999999999e-05 2023-07-02T14:06:56.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:56.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454888+0000", 2023-07-02T14:06:56.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.110043313 2023-07-02T14:06:56.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564931+0000", 2023-07-02T14:06:56.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7185e-05 2023-07-02T14:06:56.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:56.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.565008+0000", 2023-07-02T14:06:56.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014058199999999999 2023-07-02T14:06:56.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.893 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:56.893 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.565149+0000", 2023-07-02T14:06:56.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093221559999999998 2023-07-02T14:06:56.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:56.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574471+0000", 2023-07-02T14:06:56.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099660410000000001 2023-07-02T14:06:56.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:56.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584437+0000", 2023-07-02T14:06:56.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2805000000000001e-05 2023-07-02T14:06:56.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:56.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584480+0000", 2023-07-02T14:06:56.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9393000000000001e-05 2023-07-02T14:06:56.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:56.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584509+0000", 2023-07-02T14:06:56.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13400999399999999 2023-07-02T14:06:56.896 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.896 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.896 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.897 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20949 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 196608~2162688] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:56.897 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.454838+0000", 2023-07-02T14:06:56.897 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.086101717999998, 2023-07-02T14:06:56.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12983145400000001, 2023-07-02T14:06:56.897 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:56.897 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:56.897 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:56.897 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:56.898 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:56.898 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20949 2023-07-02T14:06:56.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.898 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:56.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:56.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454838+0000", 2023-07-02T14:06:56.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:56.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:56.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454838+0000", 2023-07-02T14:06:56.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:56.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:56.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454837+0000", 2023-07-02T14:06:56.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003772821 2023-07-02T14:06:56.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:56.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458610+0000", 2023-07-02T14:06:56.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3013999999999999e-05 2023-07-02T14:06:56.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:56.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458623+0000", 2023-07-02T14:06:56.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2789999999999998e-06 2023-07-02T14:06:56.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:56.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458628+0000", 2023-07-02T14:06:56.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7630999999999998e-05 2023-07-02T14:06:56.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458676+0000", 2023-07-02T14:06:56.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1579999999999994e-06 2023-07-02T14:06:56.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:56.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458685+0000", 2023-07-02T14:06:56.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.111536444 2023-07-02T14:06:56.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570221+0000", 2023-07-02T14:06:56.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000120844 2023-07-02T14:06:56.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:56.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570342+0000", 2023-07-02T14:06:56.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000149965 2023-07-02T14:06:56.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:56.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570492+0000", 2023-07-02T14:06:56.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014027843999999999 2023-07-02T14:06:56.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:56.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584520+0000", 2023-07-02T14:06:56.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8446000000000003e-05 2023-07-02T14:06:56.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:56.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584568+0000", 2023-07-02T14:06:56.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6771999999999998e-05 2023-07-02T14:06:56.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:56.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584625+0000", 2023-07-02T14:06:56.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4401999999999999e-05 2023-07-02T14:06:56.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:56.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584670+0000", 2023-07-02T14:06:56.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12983145400000001 2023-07-02T14:06:56.908 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.908 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.908 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.908 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20972 2.7 2:e3210266:::rbd_data.125de1449605.00000000000001a3:head [write 65536~4128768] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:06:56.909 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.463174+0000", 2023-07-02T14:06:56.909 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 31.077765878000001, 2023-07-02T14:06:56.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12719914800000001, 2023-07-02T14:06:56.909 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:06:56.909 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:56.909 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:06:56.909 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:06:56.909 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:56.910 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20972 2023-07-02T14:06:56.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.910 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:06:56.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:06:56.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.463174+0000", 2023-07-02T14:06:56.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:06:56.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:06:56.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.463174+0000", 2023-07-02T14:06:56.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:06:56.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:06:56.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.463173+0000", 2023-07-02T14:06:56.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069882069999999998 2023-07-02T14:06:56.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:06:56.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470161+0000", 2023-07-02T14:06:56.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0103999999999999e-05 2023-07-02T14:06:56.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:06:56.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470181+0000", 2023-07-02T14:06:56.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:06:56.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:06:56.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470189+0000", 2023-07-02T14:06:56.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0970999999999994e-05 2023-07-02T14:06:56.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470270+0000", 2023-07-02T14:06:56.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4389000000000001e-05 2023-07-02T14:06:56.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:06:56.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470284+0000", 2023-07-02T14:06:56.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.10421862799999999 2023-07-02T14:06:56.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:06:56.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574503+0000", 2023-07-02T14:06:56.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2457e-05 2023-07-02T14:06:56.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:06:56.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574585+0000", 2023-07-02T14:06:56.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013870899999999999 2023-07-02T14:06:56.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:06:56.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574724+0000", 2023-07-02T14:06:56.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012886246000000001 2023-07-02T14:06:56.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:06:56.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.587610+0000", 2023-07-02T14:06:56.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026883599999999999 2023-07-02T14:06:56.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:56.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.590298+0000", 2023-07-02T14:06:56.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1850000000000001e-05 2023-07-02T14:06:56.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:06:56.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.590340+0000", 2023-07-02T14:06:56.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2901999999999998e-05 2023-07-02T14:06:56.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:06:56.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:06:56.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:06:56.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.590373+0000", 2023-07-02T14:06:56.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12719914800000001 2023-07-02T14:06:56.919 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.920 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.920 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.920 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:06:56.920 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:06:56.920 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:06:56.922 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:06:56.922 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:56.921+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:57.022 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:57.021+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:57.105 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:57.105 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:57.106 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:57.106 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:57.123 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:57.122+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:57.129 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:06:57.142 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:06:57.188 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap62 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap63 b3fbb028-e267-4d25-a750-93ba18a2316a@snap63.diff 2023-07-02T14:06:57.223 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:57.222+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:57.262 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:57.261+0000 7f9010005700 0 -- 172.21.15.163:0/4228251693 >> [v2:172.21.15.22:6808/99977,v1:172.21.15.22:6809/99977] conn(0x7f8fec008780 msgr2=0x7f8fec00abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:57.270 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:06:57.269+0000 7f900f804700 0 -- 172.21.15.163:0/4228251693 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7f8ff0024a40 msgr2=0x7f8ff0026e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:06:57.325 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:57.323+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:57.364 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:57.364 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:57.364 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:06:57.365 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:06:57.365 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:57.380 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:06:57.415 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 100% complete...done. 2023-07-02T14:06:57.419 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:06:57.419 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=64 2023-07-02T14:06:57.419 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap64 2023-07-02T14:06:57.463 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:57.462+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:57.526 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:57.526+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:57.563 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:06:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:06:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18643 2.5 e176/170 2:a9ace6f8:::rbd_data.125de1449605.0000000000000187:head v 176'2314, mlcod=173'2311)", 2023-07-02T14:06:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:06:57.564 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 52.135631586999999, 2023-07-02T14:06:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015027297, 2023-07-02T14:06:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:57.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:06:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:06:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426547+0000", 2023-07-02T14:06:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056864430000000002 2023-07-02T14:06:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432234+0000", 2023-07-02T14:06:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.415e-05 2023-07-02T14:06:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432248+0000", 2023-07-02T14:06:57.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7490000000000001e-06 2023-07-02T14:06:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432255+0000", 2023-07-02T14:06:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3915000000000002e-05 2023-07-02T14:06:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432299+0000", 2023-07-02T14:06:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1214000000000001e-05 2023-07-02T14:06:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432320+0000", 2023-07-02T14:06:57.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0092114950000000001 2023-07-02T14:06:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.441531+0000", 2023-07-02T14:06:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4422999999999998e-05 2023-07-02T14:06:57.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.441576+0000", 2023-07-02T14:06:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015027297 2023-07-02T14:06:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18680 2.2 e176/171 2:5fd2923a:::rbd_data.125de1449605.000000000000018c:head v 176'1639, mlcod=176'1637)", 2023-07-02T14:06:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.489711+0000", 2023-07-02T14:06:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 52.072469144999999, 2023-07-02T14:06:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014407402, 2023-07-02T14:06:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:57.573 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.489711+0000", 2023-07-02T14:06:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.489711+0000", 2023-07-02T14:06:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:06:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.489709+0000", 2023-07-02T14:06:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001991792 2023-07-02T14:06:57.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491701+0000", 2023-07-02T14:06:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7323000000000002e-05 2023-07-02T14:06:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:57.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491738+0000", 2023-07-02T14:06:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2483999999999999e-05 2023-07-02T14:06:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491751+0000", 2023-07-02T14:06:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7896000000000003e-05 2023-07-02T14:06:57.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491809+0000", 2023-07-02T14:06:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0185000000000001e-05 2023-07-02T14:06:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:57.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.491829+0000", 2023-07-02T14:06:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012250901999999999 2023-07-02T14:06:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.504080+0000", 2023-07-02T14:06:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:06:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.504118+0000", 2023-07-02T14:06:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014407402 2023-07-02T14:06:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18881 2.5 e177/170 2:b1731d2f:::rbd_data.125de1449605.0000000000000024:head v 177'2350, mlcod=177'2347)", 2023-07-02T14:06:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:06:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.656265763, 2023-07-02T14:06:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020820709999999999, 2023-07-02T14:06:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:57.581 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:06:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:06:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905913+0000", 2023-07-02T14:06:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0080191519999999999 2023-07-02T14:06:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913932+0000", 2023-07-02T14:06:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3757999999999999e-05 2023-07-02T14:06:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913956+0000", 2023-07-02T14:06:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4009999999999997e-06 2023-07-02T14:06:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913965+0000", 2023-07-02T14:06:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3383e-05 2023-07-02T14:06:57.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914018+0000", 2023-07-02T14:06:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7674e-05 2023-07-02T14:06:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:57.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914046+0000", 2023-07-02T14:06:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012656555999999999 2023-07-02T14:06:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.926702+0000", 2023-07-02T14:06:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2891000000000001e-05 2023-07-02T14:06:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.926735+0000", 2023-07-02T14:06:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020820709999999999 2023-07-02T14:06:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.588 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19073 2.5 e177/170 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:head v 177'2378, mlcod=177'2376)", 2023-07-02T14:06:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:06:57.589 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.288552580000001, 2023-07-02T14:06:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018435625000000001, 2023-07-02T14:06:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:57.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:06:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:06:57.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273626+0000", 2023-07-02T14:06:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046035659999999999 2023-07-02T14:06:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278230+0000", 2023-07-02T14:06:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9386e-05 2023-07-02T14:06:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278259+0000", 2023-07-02T14:06:57.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1972e-05 2023-07-02T14:06:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278271+0000", 2023-07-02T14:06:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017532680000000001 2023-07-02T14:06:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280025+0000", 2023-07-02T14:06:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2367000000000001e-05 2023-07-02T14:06:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280047+0000", 2023-07-02T14:06:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011987028 2023-07-02T14:06:57.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.292034+0000", 2023-07-02T14:06:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9266000000000001e-05 2023-07-02T14:06:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.292063+0000", 2023-07-02T14:06:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018435625000000001 2023-07-02T14:06:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.597 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:20909 2.5 e189/170 2:bb7db89c:::rbd_data.125de1449605.00000000000001a5:head v 189'2620, mlcod=189'2618)", 2023-07-02T14:06:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.368184+0000", 2023-07-02T14:06:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 32.193996503000001, 2023-07-02T14:06:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016727725999999998, 2023-07-02T14:06:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.368184+0000", 2023-07-02T14:06:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:57.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.368184+0000", 2023-07-02T14:06:57.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:57.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.368182+0000", 2023-07-02T14:06:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067720749999999998 2023-07-02T14:06:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:57.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.374954+0000", 2023-07-02T14:06:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4501000000000001e-05 2023-07-02T14:06:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.374969+0000", 2023-07-02T14:06:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4100000000000002e-06 2023-07-02T14:06:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.374976+0000", 2023-07-02T14:06:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.736e-05 2023-07-02T14:06:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.375034+0000", 2023-07-02T14:06:57.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4544000000000001e-05 2023-07-02T14:06:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.375058+0000", 2023-07-02T14:06:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0098134910000000006 2023-07-02T14:06:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.384872+0000", 2023-07-02T14:06:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9687000000000001e-05 2023-07-02T14:06:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.384911+0000", 2023-07-02T14:06:57.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016727725999999998 2023-07-02T14:06:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:21096 2.5 e190/170 2:a0acb434:::rbd_data.125de1449605.00000000000001bd:head v 190'2623, mlcod=189'2621)", 2023-07-02T14:06:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.705165+0000", 2023-07-02T14:06:57.606 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 31.857014667000001, 2023-07-02T14:06:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027008152000000001, 2023-07-02T14:06:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:57.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.705165+0000", 2023-07-02T14:06:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.705165+0000", 2023-07-02T14:06:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:57.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.705164+0000", 2023-07-02T14:06:57.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017192187000000001 2023-07-02T14:06:57.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:57.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722356+0000", 2023-07-02T14:06:57.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0043e-05 2023-07-02T14:06:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722387+0000", 2023-07-02T14:06:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8230000000000001e-06 2023-07-02T14:06:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722395+0000", 2023-07-02T14:06:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9427e-05 2023-07-02T14:06:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:57.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722445+0000", 2023-07-02T14:06:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4321999999999999e-05 2023-07-02T14:06:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722469+0000", 2023-07-02T14:06:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0096570649999999994 2023-07-02T14:06:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.732126+0000", 2023-07-02T14:06:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7490000000000001e-05 2023-07-02T14:06:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.732174+0000", 2023-07-02T14:06:57.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027008152000000001 2023-07-02T14:06:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22204 2.4 e197/169 2:25e15f0e:::rbd_data.125de1449605.0000000000000087:head v 197'2046, mlcod=197'2043)", 2023-07-02T14:06:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.493042+0000", 2023-07-02T14:06:57.615 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.069138404, 2023-07-02T14:06:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016388874000000001, 2023-07-02T14:06:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:57.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493042+0000", 2023-07-02T14:06:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493042+0000", 2023-07-02T14:06:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493041+0000", 2023-07-02T14:06:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9621999999999999e-05 2023-07-02T14:06:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:57.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493070+0000", 2023-07-02T14:06:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3439e-05 2023-07-02T14:06:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493084+0000", 2023-07-02T14:06:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6720000000000004e-06 2023-07-02T14:06:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493092+0000", 2023-07-02T14:06:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4440000000000001e-05 2023-07-02T14:06:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493136+0000", 2023-07-02T14:06:57.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3710000000000002e-05 2023-07-02T14:06:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493160+0000", 2023-07-02T14:06:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016220526999999998 2023-07-02T14:06:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.509380+0000", 2023-07-02T14:06:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0417000000000002e-05 2023-07-02T14:06:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.509431+0000", 2023-07-02T14:06:57.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016388874000000001 2023-07-02T14:06:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22261 2.4 e197/169 2:2a8b9a55:::rbd_data.125de1449605.0000000000000088:head v 197'2053, mlcod=197'2050)", 2023-07-02T14:06:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.616181+0000", 2023-07-02T14:06:57.623 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.945999256, 2023-07-02T14:06:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014816169000000001, 2023-07-02T14:06:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.616181+0000", 2023-07-02T14:06:57.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.616181+0000", 2023-07-02T14:06:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.616180+0000", 2023-07-02T14:06:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00089455100000000005 2023-07-02T14:06:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617074+0000", 2023-07-02T14:06:57.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9324e-05 2023-07-02T14:06:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617093+0000", 2023-07-02T14:06:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2199999999999998e-06 2023-07-02T14:06:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617103+0000", 2023-07-02T14:06:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013166799999999999 2023-07-02T14:06:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617234+0000", 2023-07-02T14:06:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9300000000000001e-05 2023-07-02T14:06:57.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617264+0000", 2023-07-02T14:06:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013684961000000001 2023-07-02T14:06:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:57.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.630949+0000", 2023-07-02T14:06:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8435e-05 2023-07-02T14:06:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.630997+0000", 2023-07-02T14:06:57.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014816169000000001 2023-07-02T14:06:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22265 2.5 e197/170 2:b033b7a2:::rbd_data.125de1449605.000000000000008a:head v 197'2737, mlcod=197'2734)", 2023-07-02T14:06:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.620697+0000", 2023-07-02T14:06:57.631 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.9414835, 2023-07-02T14:06:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016340187999999999, 2023-07-02T14:06:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:57.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.620697+0000", 2023-07-02T14:06:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.620697+0000", 2023-07-02T14:06:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.620695+0000", 2023-07-02T14:06:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029799269999999998 2023-07-02T14:06:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623675+0000", 2023-07-02T14:06:57.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6534000000000001e-05 2023-07-02T14:06:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623692+0000", 2023-07-02T14:06:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0400000000000004e-06 2023-07-02T14:06:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:57.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623699+0000", 2023-07-02T14:06:57.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3189999999999997e-05 2023-07-02T14:06:57.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623752+0000", 2023-07-02T14:06:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2078000000000001e-05 2023-07-02T14:06:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623774+0000", 2023-07-02T14:06:57.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013226343999999999 2023-07-02T14:06:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.637001+0000", 2023-07-02T14:06:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6279000000000003e-05 2023-07-02T14:06:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.637037+0000", 2023-07-02T14:06:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016340187999999999 2023-07-02T14:06:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22352 2.5 e197/170 2:ac0b3e68:::rbd_data.125de1449605.00000000000000a2:head v 197'2748, mlcod=197'2745)", 2023-07-02T14:06:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.836234+0000", 2023-07-02T14:06:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.725946102999998, 2023-07-02T14:06:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015976187999999999, 2023-07-02T14:06:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:57.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.836234+0000", 2023-07-02T14:06:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:57.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.836234+0000", 2023-07-02T14:06:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.836233+0000", 2023-07-02T14:06:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066101179999999999 2023-07-02T14:06:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842843+0000", 2023-07-02T14:06:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9471e-05 2023-07-02T14:06:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:57.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842862+0000", 2023-07-02T14:06:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3270000000000007e-06 2023-07-02T14:06:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842872+0000", 2023-07-02T14:06:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0770999999999998e-05 2023-07-02T14:06:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842933+0000", 2023-07-02T14:06:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0375999999999999e-05 2023-07-02T14:06:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842963+0000", 2023-07-02T14:06:57.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091962380000000007 2023-07-02T14:06:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.852159+0000", 2023-07-02T14:06:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1100999999999997e-05 2023-07-02T14:06:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:57.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.852210+0000", 2023-07-02T14:06:57.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015976187999999999 2023-07-02T14:06:57.647 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.647 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.647 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.647 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22416 2.5 e197/170 2:aa88fbae:::rbd_data.125de1449605.00000000000000af:head v 197'2769, mlcod=197'2766)", 2023-07-02T14:06:57.648 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.954925+0000", 2023-07-02T14:06:57.648 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.60725549, 2023-07-02T14:06:57.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017286838999999998, 2023-07-02T14:06:57.648 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:57.649 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:57.649 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:57.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:57.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.954925+0000", 2023-07-02T14:06:57.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:57.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:57.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.954925+0000", 2023-07-02T14:06:57.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:57.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:57.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.954924+0000", 2023-07-02T14:06:57.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007392755 2023-07-02T14:06:57.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:57.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962316+0000", 2023-07-02T14:06:57.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9266000000000002e-05 2023-07-02T14:06:57.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:57.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962336+0000", 2023-07-02T14:06:57.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0961000000000001e-05 2023-07-02T14:06:57.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:57.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962347+0000", 2023-07-02T14:06:57.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6159999999999998e-05 2023-07-02T14:06:57.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:57.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962403+0000", 2023-07-02T14:06:57.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7605000000000003e-05 2023-07-02T14:06:57.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:57.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962440+0000", 2023-07-02T14:06:57.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097334670000000009 2023-07-02T14:06:57.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:57.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.972174+0000", 2023-07-02T14:06:57.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7759000000000002e-05 2023-07-02T14:06:57.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:57.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.972211+0000", 2023-07-02T14:06:57.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017286838999999998 2023-07-02T14:06:57.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.655 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.655 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.656 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22470 2.5 e197/170 2:af9d8f16:::rbd_data.125de1449605.00000000000000b2:head v 197'2772, mlcod=197'2769)", 2023-07-02T14:06:57.656 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.062494+0000", 2023-07-02T14:06:57.656 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.499686291, 2023-07-02T14:06:57.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018691016000000001, 2023-07-02T14:06:57.656 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:57.657 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:57.657 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:57.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:57.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.062494+0000", 2023-07-02T14:06:57.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:57.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:57.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.062494+0000", 2023-07-02T14:06:57.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:57.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:57.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.062492+0000", 2023-07-02T14:06:57.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010932252999999999 2023-07-02T14:06:57.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:57.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073424+0000", 2023-07-02T14:06:57.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3642000000000001e-05 2023-07-02T14:06:57.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:57.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073438+0000", 2023-07-02T14:06:57.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6270000000000002e-06 2023-07-02T14:06:57.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:57.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073445+0000", 2023-07-02T14:06:57.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0265999999999997e-05 2023-07-02T14:06:57.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:57.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073485+0000", 2023-07-02T14:06:57.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5221000000000001e-05 2023-07-02T14:06:57.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:57.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073510+0000", 2023-07-02T14:06:57.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076289959999999999 2023-07-02T14:06:57.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:57.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.081139+0000", 2023-07-02T14:06:57.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5618999999999998e-05 2023-07-02T14:06:57.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:57.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.081185+0000", 2023-07-02T14:06:57.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018691016000000001 2023-07-02T14:06:57.663 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.663 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.663 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.664 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22516 2.5 e197/170 2:a545e9c7:::rbd_data.125de1449605.00000000000000c3:head v 197'2781, mlcod=197'2778)", 2023-07-02T14:06:57.664 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.146937+0000", 2023-07-02T14:06:57.664 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.415242971000001, 2023-07-02T14:06:57.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015143108000000001, 2023-07-02T14:06:57.664 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:57.664 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:57.665 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:57.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:57.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.146937+0000", 2023-07-02T14:06:57.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:57.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:57.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.146937+0000", 2023-07-02T14:06:57.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:57.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:57.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.146937+0000", 2023-07-02T14:06:57.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8502000000000001e-05 2023-07-02T14:06:57.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:57.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.147025+0000", 2023-07-02T14:06:57.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4660000000000005e-06 2023-07-02T14:06:57.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:57.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.147035+0000", 2023-07-02T14:06:57.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3360000000000002e-06 2023-07-02T14:06:57.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:57.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.147040+0000", 2023-07-02T14:06:57.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004086124 2023-07-02T14:06:57.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:57.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.151126+0000", 2023-07-02T14:06:57.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9261999999999999e-05 2023-07-02T14:06:57.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:57.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.151155+0000", 2023-07-02T14:06:57.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010884783 2023-07-02T14:06:57.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:57.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.162040+0000", 2023-07-02T14:06:57.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0268000000000001e-05 2023-07-02T14:06:57.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:57.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.162080+0000", 2023-07-02T14:06:57.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015143108000000001 2023-07-02T14:06:57.671 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.671 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.671 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22534 2.5 e197/170 2:b58a101b:::rbd_data.125de1449605.00000000000000c2:head v 197'2788, mlcod=197'2783)", 2023-07-02T14:06:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.173613+0000", 2023-07-02T14:06:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.388567473999998, 2023-07-02T14:06:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023211177999999999, 2023-07-02T14:06:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:57.672 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:57.673 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:57.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:57.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.173613+0000", 2023-07-02T14:06:57.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:57.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:57.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.173613+0000", 2023-07-02T14:06:57.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:57.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:57.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.173612+0000", 2023-07-02T14:06:57.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055730149999999997 2023-07-02T14:06:57.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:57.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179185+0000", 2023-07-02T14:06:57.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1826e-05 2023-07-02T14:06:57.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:57.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179197+0000", 2023-07-02T14:06:57.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9130000000000002e-06 2023-07-02T14:06:57.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:57.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179202+0000", 2023-07-02T14:06:57.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1668999999999997e-05 2023-07-02T14:06:57.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:57.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179244+0000", 2023-07-02T14:06:57.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6171999999999999e-05 2023-07-02T14:06:57.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:57.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179270+0000", 2023-07-02T14:06:57.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017512570000000002 2023-07-02T14:06:57.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:57.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.196782+0000", 2023-07-02T14:06:57.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1576999999999999e-05 2023-07-02T14:06:57.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:57.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.196824+0000", 2023-07-02T14:06:57.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023211177999999999 2023-07-02T14:06:57.679 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.679 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.679 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.680 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22539 2.5 e197/170 2:a6e4b3cf:::rbd_data.125de1449605.00000000000000c4:head v 197'2789, mlcod=197'2786)", 2023-07-02T14:06:57.680 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.179446+0000", 2023-07-02T14:06:57.680 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.38273367, 2023-07-02T14:06:57.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017409688999999999, 2023-07-02T14:06:57.680 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:57.680 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:57.680 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:57.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:57.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179446+0000", 2023-07-02T14:06:57.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:57.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:57.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179446+0000", 2023-07-02T14:06:57.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:57.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:57.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179446+0000", 2023-07-02T14:06:57.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082263240000000001 2023-07-02T14:06:57.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:57.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187672+0000", 2023-07-02T14:06:57.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7371e-05 2023-07-02T14:06:57.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:57.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187689+0000", 2023-07-02T14:06:57.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0820000000000005e-06 2023-07-02T14:06:57.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:57.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187698+0000", 2023-07-02T14:06:57.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0510999999999998e-05 2023-07-02T14:06:57.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187749+0000", 2023-07-02T14:06:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4805000000000001e-05 2023-07-02T14:06:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:57.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187774+0000", 2023-07-02T14:06:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.009056049 2023-07-02T14:06:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.196830+0000", 2023-07-02T14:06:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6347e-05 2023-07-02T14:06:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.196856+0000", 2023-07-02T14:06:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017409688999999999 2023-07-02T14:06:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22591 2.5 e197/170 2:bf746601:::rbd_data.125de1449605.00000000000000d6:head v 197'2799, mlcod=197'2795)", 2023-07-02T14:06:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.286442+0000", 2023-07-02T14:06:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.275737901999999, 2023-07-02T14:06:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016215869000000001, 2023-07-02T14:06:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:57.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.286442+0000", 2023-07-02T14:06:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.286442+0000", 2023-07-02T14:06:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.286441+0000", 2023-07-02T14:06:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00513759 2023-07-02T14:06:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291579+0000", 2023-07-02T14:06:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5048000000000001e-05 2023-07-02T14:06:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291604+0000", 2023-07-02T14:06:57.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1429e-05 2023-07-02T14:06:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291615+0000", 2023-07-02T14:06:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6360999999999994e-05 2023-07-02T14:06:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291712+0000", 2023-07-02T14:06:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9581999999999999e-05 2023-07-02T14:06:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291741+0000", 2023-07-02T14:06:57.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010882032999999999 2023-07-02T14:06:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.302623+0000", 2023-07-02T14:06:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4688999999999999e-05 2023-07-02T14:06:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.302658+0000", 2023-07-02T14:06:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016215869000000001 2023-07-02T14:06:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22597 2.5 e197/170 2:a8e921ad:::rbd_data.125de1449605.00000000000000d7:head v 197'2802, mlcod=197'2799)", 2023-07-02T14:06:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.307993+0000", 2023-07-02T14:06:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.254186690000001, 2023-07-02T14:06:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014767641, 2023-07-02T14:06:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:57.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.307993+0000", 2023-07-02T14:06:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.307993+0000", 2023-07-02T14:06:57.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:57.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:57.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.307993+0000", 2023-07-02T14:06:57.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004799523 2023-07-02T14:06:57.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:57.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.312792+0000", 2023-07-02T14:06:57.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0421e-05 2023-07-02T14:06:57.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:57.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.312813+0000", 2023-07-02T14:06:57.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7560000000000005e-06 2023-07-02T14:06:57.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:57.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.312821+0000", 2023-07-02T14:06:57.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4166999999999998e-05 2023-07-02T14:06:57.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:57.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.312876+0000", 2023-07-02T14:06:57.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7472999999999999e-05 2023-07-02T14:06:57.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:57.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.312903+0000", 2023-07-02T14:06:57.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0098211700000000006 2023-07-02T14:06:57.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:57.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.322724+0000", 2023-07-02T14:06:57.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6928999999999998e-05 2023-07-02T14:06:57.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:57.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.322761+0000", 2023-07-02T14:06:57.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014767641 2023-07-02T14:06:57.703 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.703 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.703 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.703 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:23429 2.5 e206/170 2:aa2388f6:::rbd_data.125de1449605.00000000000001c4:head v 206'2898, mlcod=206'2896)", 2023-07-02T14:06:57.704 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:56.993306+0000", 2023-07-02T14:06:57.704 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.56887389300000002, 2023-07-02T14:06:57.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017604624999999999, 2023-07-02T14:06:57.704 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:57.704 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:57.704 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:57.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:57.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:56.993306+0000", 2023-07-02T14:06:57.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:57.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:57.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:56.993306+0000", 2023-07-02T14:06:57.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:57.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:57.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:56.993305+0000", 2023-07-02T14:06:57.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068210199999999997 2023-07-02T14:06:57.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:57.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.000126+0000", 2023-07-02T14:06:57.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1634999999999999e-05 2023-07-02T14:06:57.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:57.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.000147+0000", 2023-07-02T14:06:57.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3950000000000001e-06 2023-07-02T14:06:57.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:57.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.000156+0000", 2023-07-02T14:06:57.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012743300000000001 2023-07-02T14:06:57.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:57.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.000283+0000", 2023-07-02T14:06:57.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6681000000000001e-05 2023-07-02T14:06:57.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:57.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.000310+0000", 2023-07-02T14:06:57.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010566268 2023-07-02T14:06:57.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:57.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.010876+0000", 2023-07-02T14:06:57.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4650999999999997e-05 2023-07-02T14:06:57.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:57.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.010911+0000", 2023-07-02T14:06:57.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017604624999999999 2023-07-02T14:06:57.711 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.711 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.711 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.711 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:23563 2.5 e206/170 2:a468a258:::rbd_data.125de1449605.00000000000001de:head v 206'2917, mlcod=206'2915)", 2023-07-02T14:06:57.712 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:57.318124+0000", 2023-07-02T14:06:57.712 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.24405575600000001, 2023-07-02T14:06:57.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026542948, 2023-07-02T14:06:57.712 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:57.712 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:57.712 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:57.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:57.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318124+0000", 2023-07-02T14:06:57.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:57.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:57.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318124+0000", 2023-07-02T14:06:57.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:06:57.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:57.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318122+0000", 2023-07-02T14:06:57.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013275129 2023-07-02T14:06:57.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331397+0000", 2023-07-02T14:06:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.294e-05 2023-07-02T14:06:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331410+0000", 2023-07-02T14:06:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7190000000000001e-06 2023-07-02T14:06:57.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331418+0000", 2023-07-02T14:06:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1717000000000001e-05 2023-07-02T14:06:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:57.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331460+0000", 2023-07-02T14:06:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6067999999999998e-05 2023-07-02T14:06:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331486+0000", 2023-07-02T14:06:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013124998000000001 2023-07-02T14:06:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.344611+0000", 2023-07-02T14:06:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6536999999999998e-05 2023-07-02T14:06:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:57.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.344667+0000", 2023-07-02T14:06:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026542948 2023-07-02T14:06:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:23565 2.5 e206/170 2:bb77596d:::rbd_data.125de1449605.00000000000001df:head v 206'2919, mlcod=206'2915)", 2023-07-02T14:06:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:57.331506+0000", 2023-07-02T14:06:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.23067460100000001, 2023-07-02T14:06:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016925951000000002, 2023-07-02T14:06:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:57.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331506+0000", 2023-07-02T14:06:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331506+0000", 2023-07-02T14:06:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:57.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331505+0000", 2023-07-02T14:06:57.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045442210000000002 2023-07-02T14:06:57.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:57.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.336049+0000", 2023-07-02T14:06:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8947e-05 2023-07-02T14:06:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.336068+0000", 2023-07-02T14:06:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9129999999999992e-06 2023-07-02T14:06:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.336077+0000", 2023-07-02T14:06:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029420649999999998 2023-07-02T14:06:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.339019+0000", 2023-07-02T14:06:57.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7931000000000002e-05 2023-07-02T14:06:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.339037+0000", 2023-07-02T14:06:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093659829999999996 2023-07-02T14:06:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.348403+0000", 2023-07-02T14:06:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8435000000000001e-05 2023-07-02T14:06:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.348431+0000", 2023-07-02T14:06:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016925951000000002 2023-07-02T14:06:57.726 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.727 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:57.727 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:57.727 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:57.729 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:57.727+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:57.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:06:57.829 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:57.828+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:57.894 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:57.895 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:57.895 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:06:57.895 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:57.904 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:06:57.930 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:57.930+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:58.030 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:58.029+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:58.068 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:58.068 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:06:58.068 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:06:58.068 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:06:58.068 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:58.078 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:06:58.140 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:06:58.141 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:58.140+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:58.206 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:06:58.217 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 4 2023-07-02T14:06:58.232 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:58.231+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:58.288 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:06:58.290 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:58.290 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:06:58.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:06:58.290 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:58.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:06:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 228.834833285, 2023-07-02T14:06:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:06:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.291 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:06:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:58.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:06:58.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:58.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:06:58.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:06:58.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:06:58.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:06:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:06:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:06:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:06:58.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:06:58.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:06:58.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:06:58.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:58.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:06:58.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:06:58.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:06:58.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:06:58.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:58.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:06:58.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:06:58.299 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.299 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.299 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.299 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:06:58.299 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:58.300 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 227.96414027700001, 2023-07-02T14:06:58.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:06:58.300 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.300 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:58.300 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.300 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:06:58.300 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:06:58.300 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:06:58.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.301 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:58.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:06:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:06:58.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:06:58.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:06:58.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:06:58.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:06:58.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:06:58.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:06:58.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:06:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:06:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:06:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:58.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:06:58.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:06:58.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:06:58.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:06:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:06:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:06:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:58.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:06:58.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:06:58.308 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.308 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.308 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.309 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:58.309 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:58.309 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 227.96401555599999, 2023-07-02T14:06:58.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:06:58.309 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.309 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:58.309 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.309 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:58.310 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:58.310 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:06:58.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.310 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:06:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:58.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:06:58.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:06:58.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:06:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:06:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:06:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:06:58.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:06:58.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:06:58.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:06:58.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:06:58.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:58.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:06:58.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:06:58.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:06:58.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:06:58.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:06:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:06:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.317 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:58.318 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:58.318 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 227.957165672, 2023-07-02T14:06:58.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:06:58.318 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.318 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:58.318 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.318 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:58.318 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:58.319 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:06:58.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.319 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:58.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:06:58.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:06:58.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:06:58.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:06:58.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:06:58.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:06:58.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:06:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:06:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:06:58.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:06:58.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:06:58.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:06:58.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:58.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:06:58.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:06:58.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:06:58.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:06:58.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:58.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:06:58.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:06:58.325 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.325 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 227.95710274699999, 2023-07-02T14:06:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:06:58.326 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.327 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:58.327 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.327 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:58.327 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:58.327 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:06:58.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.327 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:58.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:06:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:06:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:06:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:06:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:06:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:06:58.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:06:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:06:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:06:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:06:58.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:06:58.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:58.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:06:58.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:06:58.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:06:58.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:06:58.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:58.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:06:58.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:06:58.334 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.334 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.334 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.334 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:06:58.335 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:58.335 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 227.956802326, 2023-07-02T14:06:58.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:06:58.335 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.335 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:58.335 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.335 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:58.336 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:58.336 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:06:58.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.336 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:58.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:06:58.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:58.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:06:58.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:06:58.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:06:58.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:06:58.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:06:58.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:06:58.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:06:58.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:06:58.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:06:58.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:06:58.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:58.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:06:58.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:06:58.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:06:58.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:06:58.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:58.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:06:58.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:06:58.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:06:58.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:06:58.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:06:58.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:58.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:06:58.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:06:58.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:58.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:06:58.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:06:58.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:58.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:06:58.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:06:58.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:58.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:06:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:06:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:06:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:06:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.346 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:06:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 227.95627782599999, 2023-07-02T14:06:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:06:58.347 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:06:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.348 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:06:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:06:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:06:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:06:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:06:58.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:06:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:06:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:06:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:06:58.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:06:58.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:58.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:06:58.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:06:58.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:06:58.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:06:58.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:58.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:06:58.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:06:58.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:58.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:06:58.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:06:58.355 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.356 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.356 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.356 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:06:58.356 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:58.356 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 200.832258701, 2023-07-02T14:06:58.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:06:58.357 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.357 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:06:58.357 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.357 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:58.357 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:58.357 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:06:58.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.357 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:58.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:06:58.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:58.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:06:58.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:06:58.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:06:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:06:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:06:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:06:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:06:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:06:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:06:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:06:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:06:58.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:06:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:06:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:06:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:06:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:06:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:06:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 200.50677599700001, 2023-07-02T14:06:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:06:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:06:58.365 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:06:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:06:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:06:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:06:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:06:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:06:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:06:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:06:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:06:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:06:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:06:58.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:06:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:06:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:06:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:06:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:06:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:06:58.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:06:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:06:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:06:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:06:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.373 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:06:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:06:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:06:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:06:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:06:58.374 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:06:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:06:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:06:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:58.375 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:06:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:06:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.376 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:17 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 0~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.088500215, 2023-07-02T14:06:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997, 2023-07-02T14:06:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:58.377 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17 2023-07-02T14:06:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:06:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:06:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198431+0000", 2023-07-02T14:06:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2815e-05 2023-07-02T14:06:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198443+0000", 2023-07-02T14:06:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2569999999999999e-05 2023-07-02T14:06:58.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198466+0000", 2023-07-02T14:06:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0223e-05 2023-07-02T14:06:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198476+0000", 2023-07-02T14:06:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1792999999999997e-05 2023-07-02T14:06:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:06:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4249999999999998e-06 2023-07-02T14:06:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198536+0000", 2023-07-02T14:06:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91888663500000001 2023-07-02T14:06:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117423+0000", 2023-07-02T14:06:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223594 2023-07-02T14:06:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117647+0000", 2023-07-02T14:06:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033832900000000001 2023-07-02T14:06:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117985+0000", 2023-07-02T14:06:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997 2023-07-02T14:06:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.385 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:18 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 65536~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.088443004, 2023-07-02T14:06:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998, 2023-07-02T14:06:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.386 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18 2023-07-02T14:06:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:58.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:06:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:06:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7600000000000002e-06 2023-07-02T14:06:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198494+0000", 2023-07-02T14:06:58.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7840000000000004e-06 2023-07-02T14:06:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198502+0000", 2023-07-02T14:06:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6459999999999998e-06 2023-07-02T14:06:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198508+0000", 2023-07-02T14:06:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7395999999999997e-05 2023-07-02T14:06:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198565+0000", 2023-07-02T14:06:58.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6800000000000004e-06 2023-07-02T14:06:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198572+0000", 2023-07-02T14:06:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91965896800000002 2023-07-02T14:06:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118231+0000", 2023-07-02T14:06:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5212999999999998e-05 2023-07-02T14:06:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118326+0000", 2023-07-02T14:06:58.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000238538 2023-07-02T14:06:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118565+0000", 2023-07-02T14:06:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998 2023-07-02T14:06:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:19 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 131072~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.08841878, 2023-07-02T14:06:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002, 2023-07-02T14:06:58.395 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 19 2023-07-02T14:06:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:58.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:06:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1990000000000002e-06 2023-07-02T14:06:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198517+0000", 2023-07-02T14:06:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8789999999999997e-06 2023-07-02T14:06:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198524+0000", 2023-07-02T14:06:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:06:58.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:06:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5545999999999999e-05 2023-07-02T14:06:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2540000000000003e-06 2023-07-02T14:06:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198598+0000", 2023-07-02T14:06:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.919983263 2023-07-02T14:06:58.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118581+0000", 2023-07-02T14:06:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6878999999999994e-05 2023-07-02T14:06:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:58.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118648+0000", 2023-07-02T14:06:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149732 2023-07-02T14:06:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118798+0000", 2023-07-02T14:06:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002 2023-07-02T14:06:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.403 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:20 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 262144~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.08839981600001, 2023-07-02T14:06:58.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998, 2023-07-02T14:06:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20 2023-07-02T14:06:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.405 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:06:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.732e-06 2023-07-02T14:06:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198537+0000", 2023-07-02T14:06:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5259999999999999e-06 2023-07-02T14:06:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198543+0000", 2023-07-02T14:06:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.121e-06 2023-07-02T14:06:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198547+0000", 2023-07-02T14:06:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9591000000000005e-05 2023-07-02T14:06:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:06:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3120000000000004e-06 2023-07-02T14:06:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198621+0000", 2023-07-02T14:06:58.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92020789400000003 2023-07-02T14:06:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118829+0000", 2023-07-02T14:06:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022393386000000001 2023-07-02T14:06:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141223+0000", 2023-07-02T14:06:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000329866 2023-07-02T14:06:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141552+0000", 2023-07-02T14:06:58.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998 2023-07-02T14:06:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:21 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 196608~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:58.413 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.088372181, 2023-07-02T14:06:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995, 2023-07-02T14:06:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21 2023-07-02T14:06:58.414 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:06:58.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0169999999999999e-05 2023-07-02T14:06:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198571+0000", 2023-07-02T14:06:58.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3479999999999998e-06 2023-07-02T14:06:58.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198578+0000", 2023-07-02T14:06:58.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8439999999999997e-06 2023-07-02T14:06:58.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198583+0000", 2023-07-02T14:06:58.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8294999999999997e-05 2023-07-02T14:06:58.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198641+0000", 2023-07-02T14:06:58.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8120000000000001e-06 2023-07-02T14:06:58.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:58.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:06:58.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94293828499999999 2023-07-02T14:06:58.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141583+0000", 2023-07-02T14:06:58.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014607300000000001 2023-07-02T14:06:58.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:58.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141729+0000", 2023-07-02T14:06:58.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00044906600000000003 2023-07-02T14:06:58.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:58.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142178+0000", 2023-07-02T14:06:58.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995 2023-07-02T14:06:58.533 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.533 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.533 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.533 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:22 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 327680~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:58.534 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:58.534 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.08833875099999, 2023-07-02T14:06:58.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004, 2023-07-02T14:06:58.534 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.534 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:58.534 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.534 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:58.534 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:58.535 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 22 2023-07-02T14:06:58.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.535 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:58.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:58.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:58.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:06:58.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4850000000000001e-06 2023-07-02T14:06:58.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198597+0000", 2023-07-02T14:06:58.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:06:58.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198605+0000", 2023-07-02T14:06:58.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4839999999999998e-06 2023-07-02T14:06:58.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198609+0000", 2023-07-02T14:06:58.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3100999999999999e-05 2023-07-02T14:06:58.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198662+0000", 2023-07-02T14:06:58.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9099999999999998e-06 2023-07-02T14:06:58.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:58.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198666+0000", 2023-07-02T14:06:58.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.943539503 2023-07-02T14:06:58.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142206+0000", 2023-07-02T14:06:58.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012255999999999999 2023-07-02T14:06:58.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:58.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142328+0000", 2023-07-02T14:06:58.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208956 2023-07-02T14:06:58.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:58.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142537+0000", 2023-07-02T14:06:58.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004 2023-07-02T14:06:58.542 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.542 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.542 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.543 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:23 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 393216~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:58.543 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:58.543 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.088318771, 2023-07-02T14:06:58.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348, 2023-07-02T14:06:58.543 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.544 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:58.544 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.544 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:58.544 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:58.544 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23 2023-07-02T14:06:58.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.544 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:58.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:58.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:58.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:06:58.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8439999999999998e-06 2023-07-02T14:06:58.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:06:58.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0440000000000001e-06 2023-07-02T14:06:58.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198623+0000", 2023-07-02T14:06:58.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4480000000000002e-06 2023-07-02T14:06:58.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198628+0000", 2023-07-02T14:06:58.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5843e-05 2023-07-02T14:06:58.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:58.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.89e-06 2023-07-02T14:06:58.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:58.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198688+0000", 2023-07-02T14:06:58.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94389171999999999 2023-07-02T14:06:58.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142580+0000", 2023-07-02T14:06:58.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3892000000000004e-05 2023-07-02T14:06:58.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:58.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142644+0000", 2023-07-02T14:06:58.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017608000000000001 2023-07-02T14:06:58.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:58.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142820+0000", 2023-07-02T14:06:58.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348 2023-07-02T14:06:58.551 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.552 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.552 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.552 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:24 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 458752~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:58.552 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:58.552 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.08829069700001, 2023-07-02T14:06:58.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004, 2023-07-02T14:06:58.553 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.553 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:58.553 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.553 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:58.553 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:58.553 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 24 2023-07-02T14:06:58.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.554 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:58.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:06:58.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:58.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198642+0000", 2023-07-02T14:06:58.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.253e-06 2023-07-02T14:06:58.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:06:58.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1289999999999998e-06 2023-07-02T14:06:58.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198652+0000", 2023-07-02T14:06:58.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4510000000000002e-06 2023-07-02T14:06:58.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198657+0000", 2023-07-02T14:06:58.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9131000000000001e-05 2023-07-02T14:06:58.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198706+0000", 2023-07-02T14:06:58.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8890000000000004e-06 2023-07-02T14:06:58.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:58.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198710+0000", 2023-07-02T14:06:58.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94415412200000004 2023-07-02T14:06:58.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142864+0000", 2023-07-02T14:06:58.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8047000000000006e-05 2023-07-02T14:06:58.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:58.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142952+0000", 2023-07-02T14:06:58.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192967 2023-07-02T14:06:58.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:58.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143145+0000", 2023-07-02T14:06:58.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004 2023-07-02T14:06:58.561 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.561 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.561 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.561 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:25 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 524288~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:58.562 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:58.562 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.088271902, 2023-07-02T14:06:58.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001, 2023-07-02T14:06:58.562 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.562 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:58.562 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.562 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:58.562 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:58.563 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25 2023-07-02T14:06:58.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.563 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:58.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:58.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:58.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:06:58.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.615e-06 2023-07-02T14:06:58.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198664+0000", 2023-07-02T14:06:58.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4759999999999997e-06 2023-07-02T14:06:58.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198669+0000", 2023-07-02T14:06:58.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5999999999999998e-06 2023-07-02T14:06:58.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198673+0000", 2023-07-02T14:06:58.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4256999999999999e-05 2023-07-02T14:06:58.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198727+0000", 2023-07-02T14:06:58.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840000000000001e-06 2023-07-02T14:06:58.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:58.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198731+0000", 2023-07-02T14:06:58.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94444241900000003 2023-07-02T14:06:58.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143173+0000", 2023-07-02T14:06:58.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6839000000000002e-05 2023-07-02T14:06:58.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:58.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143240+0000", 2023-07-02T14:06:58.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017053399999999999 2023-07-02T14:06:58.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:58.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143411+0000", 2023-07-02T14:06:58.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001 2023-07-02T14:06:58.570 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.570 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.570 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.571 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:26 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 589824~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:06:58.571 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:58.571 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 107.08824875099999, 2023-07-02T14:06:58.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997, 2023-07-02T14:06:58.571 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.571 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:58.571 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.572 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:06:58.572 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:06:58.572 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 26 2023-07-02T14:06:58.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.572 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:58.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:58.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:58.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:06:58.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0520000000000002e-06 2023-07-02T14:06:58.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198687+0000", 2023-07-02T14:06:58.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1880000000000002e-06 2023-07-02T14:06:58.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198695+0000", 2023-07-02T14:06:58.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9960000000000004e-06 2023-07-02T14:06:58.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198699+0000", 2023-07-02T14:06:58.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2330999999999998e-05 2023-07-02T14:06:58.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198742+0000", 2023-07-02T14:06:58.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7950000000000001e-06 2023-07-02T14:06:58.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:58.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198745+0000", 2023-07-02T14:06:58.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94469398199999999 2023-07-02T14:06:58.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143439+0000", 2023-07-02T14:06:58.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6048e-05 2023-07-02T14:06:58.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:58.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143525+0000", 2023-07-02T14:06:58.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017016700000000001 2023-07-02T14:06:58.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:58.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143696+0000", 2023-07-02T14:06:58.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997 2023-07-02T14:06:58.579 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.580 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.580 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.580 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.6579.0:11 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected e171)", 2023-07-02T14:06:58.580 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:58.580 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 61.128450264999998, 2023-07-02T14:06:58.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998, 2023-07-02T14:06:58.581 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.581 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:06:58.581 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.581 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.6579", 2023-07-02T14:06:58.581 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/554403467", 2023-07-02T14:06:58.581 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11 2023-07-02T14:06:58.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.582 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:58.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:06:58.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:58.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158481+0000", 2023-07-02T14:06:58.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:06:58.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158487+0000", 2023-07-02T14:06:58.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7630000000000006e-06 2023-07-02T14:06:58.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158496+0000", 2023-07-02T14:06:58.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5100000000000001e-06 2023-07-02T14:06:58.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158504+0000", 2023-07-02T14:06:58.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9397000000000002e-05 2023-07-02T14:06:58.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158553+0000", 2023-07-02T14:06:58.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5810000000000003e-06 2023-07-02T14:06:58.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:06:58.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158562+0000", 2023-07-02T14:06:58.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93893079499999998 2023-07-02T14:06:58.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097493+0000", 2023-07-02T14:06:58.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015799299999999999 2023-07-02T14:06:58.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:58.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097651+0000", 2023-07-02T14:06:58.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187746 2023-07-02T14:06:58.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:06:58.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097839+0000", 2023-07-02T14:06:58.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998 2023-07-02T14:06:58.589 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.589 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.589 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.589 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.589 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.589 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:58.591 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:06:58.599 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:58.593+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:58.692 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:58.690+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:58.790 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:06:58.790 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:06:58.790 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.790 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:23985 2.3 2:ddcb94f0:::rbd_data.125de1449605.0000000000000049:head [write 36864~3325952 in=3325952b] snapc 43=[43,42,41,40,3f,3e,3d,3c,3b,3a,39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e207)", 2023-07-02T14:06:58.790 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:58.768278+0000", 2023-07-02T14:06:58.791 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.021134947000000001, 2023-07-02T14:06:58.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021166709999999998, 2023-07-02T14:06:58.791 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.791 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "waiting for sub ops", 2023-07-02T14:06:58.791 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.791 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:58.791 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:58.791 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23985 2023-07-02T14:06:58.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.792 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.792 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.792 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.768278+0000", 2023-07-02T14:06:58.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.792 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.768278+0000", 2023-07-02T14:06:58.793 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:06:58.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.793 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.768276+0000", 2023-07-02T14:06:58.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0050659040000000004 2023-07-02T14:06:58.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.773342+0000", 2023-07-02T14:06:58.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7447e-05 2023-07-02T14:06:58.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.773369+0000", 2023-07-02T14:06:58.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0222e-05 2023-07-02T14:06:58.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.773379+0000", 2023-07-02T14:06:58.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5442999999999998e-05 2023-07-02T14:06:58.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.773425+0000", 2023-07-02T14:06:58.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000116048 2023-07-02T14:06:58.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:58.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.773541+0000", 2023-07-02T14:06:58.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000218972 2023-07-02T14:06:58.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:58.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.773760+0000", 2023-07-02T14:06:58.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015141000999999999 2023-07-02T14:06:58.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:58.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.788901+0000", 2023-07-02T14:06:58.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020622784000000002 2023-07-02T14:06:58.798 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.798 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.798 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.799 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:23986 2.3 2:ddcb94f0:::rbd_data.125de1449605.0000000000000049:head [write 3366912~61440 in=61440b] snapc 43=[43,42,41,40,3f,3e,3d,3c,3b,3a,39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e207)", 2023-07-02T14:06:58.799 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:58.773397+0000", 2023-07-02T14:06:58.799 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.016015813, 2023-07-02T14:06:58.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016117019, 2023-07-02T14:06:58.799 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.799 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "waiting for sub ops", 2023-07-02T14:06:58.800 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.800 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:58.800 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:58.800 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23986 2023-07-02T14:06:58.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.800 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.773397+0000", 2023-07-02T14:06:58.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.801 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.801 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.773397+0000", 2023-07-02T14:06:58.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:06:58.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.773396+0000", 2023-07-02T14:06:58.802 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3949000000000002e-05 2023-07-02T14:06:58.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.802 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.802 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.773440+0000", 2023-07-02T14:06:58.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8890000000000008e-06 2023-07-02T14:06:58.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.773449+0000", 2023-07-02T14:06:58.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6230000000000002e-06 2023-07-02T14:06:58.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.773454+0000", 2023-07-02T14:06:58.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066183049999999997 2023-07-02T14:06:58.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.780073+0000", 2023-07-02T14:06:58.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011861100000000001 2023-07-02T14:06:58.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:58.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.780191+0000", 2023-07-02T14:06:58.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000226579 2023-07-02T14:06:58.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:58.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.780418+0000", 2023-07-02T14:06:58.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0084936430000000004 2023-07-02T14:06:58.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:58.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.788911+0000", 2023-07-02T14:06:58.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015514172999999999 2023-07-02T14:06:58.807 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.807 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.807 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.807 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:23987 2.3 2:ddcb94f0:::rbd_data.125de1449605.0000000000000049:head [write 3428352~45056 in=45056b] snapc 43=[43,42,41,40,3f,3e,3d,3c,3b,3a,39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e207)", 2023-07-02T14:06:58.808 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:58.773461+0000", 2023-07-02T14:06:58.808 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.015952448000000001, 2023-07-02T14:06:58.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016142769000000001, 2023-07-02T14:06:58.808 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:06:58.808 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "waiting for sub ops", 2023-07-02T14:06:58.808 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:06:58.808 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:06:58.809 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:06:58.809 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23987 2023-07-02T14:06:58.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.809 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:06:58.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:06:58.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.773461+0000", 2023-07-02T14:06:58.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:06:58.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:06:58.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.773461+0000", 2023-07-02T14:06:58.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:06:58.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:06:58.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.773460+0000", 2023-07-02T14:06:58.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1542999999999999e-05 2023-07-02T14:06:58.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:06:58.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.773491+0000", 2023-07-02T14:06:58.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5780000000000001e-06 2023-07-02T14:06:58.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:06:58.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.773499+0000", 2023-07-02T14:06:58.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5750000000000002e-06 2023-07-02T14:06:58.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:06:58.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.773504+0000", 2023-07-02T14:06:58.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0086441339999999995 2023-07-02T14:06:58.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:06:58.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.782148+0000", 2023-07-02T14:06:58.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000100995 2023-07-02T14:06:58.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:06:58.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.782249+0000", 2023-07-02T14:06:58.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000190977 2023-07-02T14:06:58.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:06:58.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.782440+0000", 2023-07-02T14:06:58.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0064793079999999996 2023-07-02T14:06:58.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:06:58.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:06:58.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:06:58.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.788919+0000", 2023-07-02T14:06:58.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015458438 2023-07-02T14:06:58.815 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.815 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:06:58.816 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.816 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:06:58.816 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:06:58.816 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 3 2023-07-02T14:06:58.816 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:06:58.817 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:06:58.818 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:06:58.818 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:58.817+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:58.848 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:06:58.884 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:06:58.918 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:58.918+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:58.918 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:06:58.956 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:06:58.988 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:06:58.991 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:06:58.991 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:06:58.991 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:06:58.992 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:06:58.992 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:06:58.992 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:06:59.018 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:06:59.018+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:06:59.049 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:59.049 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:59.067 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:06:59.119 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:59.119+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:59.219 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:59.219+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:59.294 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:59.294 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:59.309 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:06:59.353 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:59.353+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:59.421 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:06:59.420+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:06:59.474 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:59.474 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:59.485 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:06:59.522 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:06:59.521+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:06:59.623 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:06:59.623+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:06:59.691 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:59.691 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:59.704 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:06:59.745 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:59.745+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:06:59.825 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:06:59.823+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:06:59.877 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:06:59.878 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:06:59.889 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:06:59.934 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:06:59.934+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:00.025 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:00.025+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:00.052 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:00.052 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:00.063 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:07:00.126 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:00.125+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:00.153 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:07:00.227 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:00.227+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:00.257 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:00.258 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:00.269 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:07:00.328 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:07:00.327+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:07:00.428 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:00.427+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:00.458 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:00.458 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:00.471 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:07:00.529 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:00.530+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:00.573 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 2, 4] out_osds: [5] dead_osds: [] live_osds: [1, 2, 4, 5, 3, 0] 2023-07-02T14:07:00.573 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:07:00.573 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 3, 2, 4] 2023-07-02T14:07:00.574 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 2023-07-02T14:07:00.611 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:00.612 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:00.622 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:07:00.663 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:00.663+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:00.731 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:00.730+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:00.814 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:00.814 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:00.825 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:07:00.833 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:00.832+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:00.935 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:07:00.932+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:07:00.993 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:00.994 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:01.006 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:07:01.012 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:07:01.033 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:01.033+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:01.077 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-07-02T14:07:01.076+0000 7ff490b05700 -1 mon.a@0(leader).osd e207 definitely_dead 0 2023-07-02T14:07:01.134 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:01.133+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:01.153 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:01.153 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:01.164 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:07:01.192 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:07:01.209 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:07:01.229 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:07:01.235 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:01.235+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:01.246 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:07:01.265 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:07:01.283 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:07:01.284 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:07:01.285 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:07:01.285 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:07:01.285 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:07:01.285 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:07:01.286 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:07:01.319 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:01.319 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:07:01.319 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:07:01.319 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:07:01.320 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:01.330 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:07:01.335 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:01.335+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:01.348 INFO:teuthology.orchestra.run.smithi016.stderr:marked out osd.0. 2023-07-02T14:07:01.349 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:07:01.436 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:01.436+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:01.462 INFO:tasks.workunit.client.0.smithi163.stderr:+ sync b3fbb028-e267-4d25-a750-93ba18a2316a-mnt 2023-07-02T14:07:01.500 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:01.500 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:07:01.500 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:07:01.501 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:07:01.501 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.501 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18643 2.5 e176/170 2:a9ace6f8:::rbd_data.125de1449605.0000000000000187:head v 176'2314, mlcod=173'2311)", 2023-07-02T14:07:01.501 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:07:01.501 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 56.072137456999997, 2023-07-02T14:07:01.501 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015027297, 2023-07-02T14:07:01.502 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.502 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:01.502 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.502 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.502 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.502 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:07:01.502 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.503 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.503 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:07:01.503 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:01.503 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.503 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.504 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426547+0000", 2023-07-02T14:07:01.504 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056864430000000002 2023-07-02T14:07:01.504 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.504 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.504 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432234+0000", 2023-07-02T14:07:01.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.415e-05 2023-07-02T14:07:01.505 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.505 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.505 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.505 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432248+0000", 2023-07-02T14:07:01.505 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7490000000000001e-06 2023-07-02T14:07:01.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.506 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.506 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432255+0000", 2023-07-02T14:07:01.506 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3915000000000002e-05 2023-07-02T14:07:01.506 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.506 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432299+0000", 2023-07-02T14:07:01.507 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1214000000000001e-05 2023-07-02T14:07:01.507 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.507 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.507 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:01.507 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432320+0000", 2023-07-02T14:07:01.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0092114950000000001 2023-07-02T14:07:01.508 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.508 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.508 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:01.508 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.441531+0000", 2023-07-02T14:07:01.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4422999999999998e-05 2023-07-02T14:07:01.509 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.509 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:01.509 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.441576+0000", 2023-07-02T14:07:01.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015027297 2023-07-02T14:07:01.509 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.509 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.510 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.510 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18881 2.5 e177/170 2:b1731d2f:::rbd_data.125de1449605.0000000000000024:head v 177'2350, mlcod=177'2347)", 2023-07-02T14:07:01.510 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:07:01.510 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 54.592771632999998, 2023-07-02T14:07:01.510 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020820709999999999, 2023-07-02T14:07:01.511 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.511 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:01.511 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:07:01.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:07:01.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:01.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905913+0000", 2023-07-02T14:07:01.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0080191519999999999 2023-07-02T14:07:01.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913932+0000", 2023-07-02T14:07:01.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3757999999999999e-05 2023-07-02T14:07:01.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913956+0000", 2023-07-02T14:07:01.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4009999999999997e-06 2023-07-02T14:07:01.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.517 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.517 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913965+0000", 2023-07-02T14:07:01.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3383e-05 2023-07-02T14:07:01.518 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.518 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.518 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.518 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914018+0000", 2023-07-02T14:07:01.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7674e-05 2023-07-02T14:07:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:01.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914046+0000", 2023-07-02T14:07:01.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012656555999999999 2023-07-02T14:07:01.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:01.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.926702+0000", 2023-07-02T14:07:01.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2891000000000001e-05 2023-07-02T14:07:01.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:01.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.926735+0000", 2023-07-02T14:07:01.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020820709999999999 2023-07-02T14:07:01.522 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.523 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.523 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.523 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19073 2.5 e177/170 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:head v 177'2378, mlcod=177'2376)", 2023-07-02T14:07:01.524 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:07:01.524 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 54.225058449999999, 2023-07-02T14:07:01.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018435625000000001, 2023-07-02T14:07:01.524 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.525 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:01.525 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:07:01.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.526 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:07:01.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:01.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.527 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.527 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273626+0000", 2023-07-02T14:07:01.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046035659999999999 2023-07-02T14:07:01.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.528 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278230+0000", 2023-07-02T14:07:01.528 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9386e-05 2023-07-02T14:07:01.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.528 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278259+0000", 2023-07-02T14:07:01.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1972e-05 2023-07-02T14:07:01.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278271+0000", 2023-07-02T14:07:01.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017532680000000001 2023-07-02T14:07:01.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280025+0000", 2023-07-02T14:07:01.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2367000000000001e-05 2023-07-02T14:07:01.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:01.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280047+0000", 2023-07-02T14:07:01.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011987028 2023-07-02T14:07:01.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:01.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.292034+0000", 2023-07-02T14:07:01.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9266000000000001e-05 2023-07-02T14:07:01.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:01.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.292063+0000", 2023-07-02T14:07:01.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018435625000000001 2023-07-02T14:07:01.532 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.533 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.533 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.533 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:20909 2.5 e189/170 2:bb7db89c:::rbd_data.125de1449605.00000000000001a5:head v 189'2620, mlcod=189'2618)", 2023-07-02T14:07:01.533 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.368184+0000", 2023-07-02T14:07:01.533 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 36.130502372999999, 2023-07-02T14:07:01.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016727725999999998, 2023-07-02T14:07:01.534 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.534 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:01.534 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.368184+0000", 2023-07-02T14:07:01.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.535 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.368184+0000", 2023-07-02T14:07:01.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:01.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.536 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.536 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.368182+0000", 2023-07-02T14:07:01.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067720749999999998 2023-07-02T14:07:01.536 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.536 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.374954+0000", 2023-07-02T14:07:01.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4501000000000001e-05 2023-07-02T14:07:01.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.374969+0000", 2023-07-02T14:07:01.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4100000000000002e-06 2023-07-02T14:07:01.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.374976+0000", 2023-07-02T14:07:01.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.736e-05 2023-07-02T14:07:01.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.375034+0000", 2023-07-02T14:07:01.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4544000000000001e-05 2023-07-02T14:07:01.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:01.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.375058+0000", 2023-07-02T14:07:01.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0098134910000000006 2023-07-02T14:07:01.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:01.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.384872+0000", 2023-07-02T14:07:01.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9687000000000001e-05 2023-07-02T14:07:01.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:01.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.384911+0000", 2023-07-02T14:07:01.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016727725999999998 2023-07-02T14:07:01.541 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.542 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.542 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.542 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:21096 2.5 e190/170 2:a0acb434:::rbd_data.125de1449605.00000000000001bd:head v 190'2623, mlcod=189'2621)", 2023-07-02T14:07:01.542 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.705165+0000", 2023-07-02T14:07:01.542 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 35.793520536999999, 2023-07-02T14:07:01.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027008152000000001, 2023-07-02T14:07:01.543 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.543 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:01.543 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.705165+0000", 2023-07-02T14:07:01.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.544 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.544 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.705165+0000", 2023-07-02T14:07:01.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:01.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.545 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.545 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.705164+0000", 2023-07-02T14:07:01.545 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017192187000000001 2023-07-02T14:07:01.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722356+0000", 2023-07-02T14:07:01.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0043e-05 2023-07-02T14:07:01.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722387+0000", 2023-07-02T14:07:01.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8230000000000001e-06 2023-07-02T14:07:01.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722395+0000", 2023-07-02T14:07:01.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9427e-05 2023-07-02T14:07:01.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722445+0000", 2023-07-02T14:07:01.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4321999999999999e-05 2023-07-02T14:07:01.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:01.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722469+0000", 2023-07-02T14:07:01.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0096570649999999994 2023-07-02T14:07:01.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:01.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.732126+0000", 2023-07-02T14:07:01.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7490000000000001e-05 2023-07-02T14:07:01.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:01.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.732174+0000", 2023-07-02T14:07:01.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027008152000000001 2023-07-02T14:07:01.551 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.551 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.551 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.551 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22204 2.4 e197/169 2:25e15f0e:::rbd_data.125de1449605.0000000000000087:head v 197'2046, mlcod=197'2043)", 2023-07-02T14:07:01.551 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.493042+0000", 2023-07-02T14:07:01.552 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 27.005644274000002, 2023-07-02T14:07:01.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016388874000000001, 2023-07-02T14:07:01.552 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.552 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:01.552 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.553 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.553 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493042+0000", 2023-07-02T14:07:01.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.553 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493042+0000", 2023-07-02T14:07:01.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:01.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493041+0000", 2023-07-02T14:07:01.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9621999999999999e-05 2023-07-02T14:07:01.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493070+0000", 2023-07-02T14:07:01.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3439e-05 2023-07-02T14:07:01.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493084+0000", 2023-07-02T14:07:01.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6720000000000004e-06 2023-07-02T14:07:01.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493092+0000", 2023-07-02T14:07:01.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4440000000000001e-05 2023-07-02T14:07:01.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493136+0000", 2023-07-02T14:07:01.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3710000000000002e-05 2023-07-02T14:07:01.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:01.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493160+0000", 2023-07-02T14:07:01.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016220526999999998 2023-07-02T14:07:01.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:01.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.509380+0000", 2023-07-02T14:07:01.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0417000000000002e-05 2023-07-02T14:07:01.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:01.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.509431+0000", 2023-07-02T14:07:01.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016388874000000001 2023-07-02T14:07:01.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.562 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.562 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.563 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22261 2.4 e197/169 2:2a8b9a55:::rbd_data.125de1449605.0000000000000088:head v 197'2053, mlcod=197'2050)", 2023-07-02T14:07:01.563 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.616181+0000", 2023-07-02T14:07:01.563 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.882505126000002, 2023-07-02T14:07:01.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014816169000000001, 2023-07-02T14:07:01.563 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.563 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:01.564 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.616181+0000", 2023-07-02T14:07:01.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.565 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.565 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.616181+0000", 2023-07-02T14:07:01.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:01.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.566 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.616180+0000", 2023-07-02T14:07:01.566 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00089455100000000005 2023-07-02T14:07:01.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.566 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.566 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617074+0000", 2023-07-02T14:07:01.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9324e-05 2023-07-02T14:07:01.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617093+0000", 2023-07-02T14:07:01.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2199999999999998e-06 2023-07-02T14:07:01.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617103+0000", 2023-07-02T14:07:01.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013166799999999999 2023-07-02T14:07:01.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617234+0000", 2023-07-02T14:07:01.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9300000000000001e-05 2023-07-02T14:07:01.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:01.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617264+0000", 2023-07-02T14:07:01.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013684961000000001 2023-07-02T14:07:01.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:01.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.630949+0000", 2023-07-02T14:07:01.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8435e-05 2023-07-02T14:07:01.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:01.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.630997+0000", 2023-07-02T14:07:01.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014816169000000001 2023-07-02T14:07:01.571 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.571 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.571 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.571 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22265 2.5 e197/170 2:b033b7a2:::rbd_data.125de1449605.000000000000008a:head v 197'2737, mlcod=197'2734)", 2023-07-02T14:07:01.571 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.620697+0000", 2023-07-02T14:07:01.572 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.877989370000002, 2023-07-02T14:07:01.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016340187999999999, 2023-07-02T14:07:01.572 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.572 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:01.572 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.620697+0000", 2023-07-02T14:07:01.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.620697+0000", 2023-07-02T14:07:01.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:01.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.620695+0000", 2023-07-02T14:07:01.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029799269999999998 2023-07-02T14:07:01.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.574 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.574 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623675+0000", 2023-07-02T14:07:01.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6534000000000001e-05 2023-07-02T14:07:01.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.575 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.575 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.575 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623692+0000", 2023-07-02T14:07:01.575 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0400000000000004e-06 2023-07-02T14:07:01.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623699+0000", 2023-07-02T14:07:01.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3189999999999997e-05 2023-07-02T14:07:01.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623752+0000", 2023-07-02T14:07:01.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2078000000000001e-05 2023-07-02T14:07:01.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:01.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623774+0000", 2023-07-02T14:07:01.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013226343999999999 2023-07-02T14:07:01.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:01.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.637001+0000", 2023-07-02T14:07:01.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6279000000000003e-05 2023-07-02T14:07:01.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:01.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.637037+0000", 2023-07-02T14:07:01.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016340187999999999 2023-07-02T14:07:01.579 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.579 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.579 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.579 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22352 2.5 e197/170 2:ac0b3e68:::rbd_data.125de1449605.00000000000000a2:head v 197'2748, mlcod=197'2745)", 2023-07-02T14:07:01.580 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.836234+0000", 2023-07-02T14:07:01.580 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.662451973, 2023-07-02T14:07:01.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015976187999999999, 2023-07-02T14:07:01.580 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.580 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:01.580 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.836234+0000", 2023-07-02T14:07:01.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.836234+0000", 2023-07-02T14:07:01.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:01.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.836233+0000", 2023-07-02T14:07:01.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066101179999999999 2023-07-02T14:07:01.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842843+0000", 2023-07-02T14:07:01.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9471e-05 2023-07-02T14:07:01.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.583 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.583 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.583 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842862+0000", 2023-07-02T14:07:01.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3270000000000007e-06 2023-07-02T14:07:01.583 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842872+0000", 2023-07-02T14:07:01.584 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0770999999999998e-05 2023-07-02T14:07:01.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842933+0000", 2023-07-02T14:07:01.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0375999999999999e-05 2023-07-02T14:07:01.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:01.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842963+0000", 2023-07-02T14:07:01.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091962380000000007 2023-07-02T14:07:01.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:01.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.852159+0000", 2023-07-02T14:07:01.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1100999999999997e-05 2023-07-02T14:07:01.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:01.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.852210+0000", 2023-07-02T14:07:01.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015976187999999999 2023-07-02T14:07:01.587 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.587 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.587 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.587 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22416 2.5 e197/170 2:aa88fbae:::rbd_data.125de1449605.00000000000000af:head v 197'2769, mlcod=197'2766)", 2023-07-02T14:07:01.587 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.954925+0000", 2023-07-02T14:07:01.588 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.543761360000001, 2023-07-02T14:07:01.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017286838999999998, 2023-07-02T14:07:01.588 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.588 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:01.588 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.954925+0000", 2023-07-02T14:07:01.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.954925+0000", 2023-07-02T14:07:01.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:01.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.954924+0000", 2023-07-02T14:07:01.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007392755 2023-07-02T14:07:01.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962316+0000", 2023-07-02T14:07:01.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9266000000000002e-05 2023-07-02T14:07:01.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962336+0000", 2023-07-02T14:07:01.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0961000000000001e-05 2023-07-02T14:07:01.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962347+0000", 2023-07-02T14:07:01.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6159999999999998e-05 2023-07-02T14:07:01.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962403+0000", 2023-07-02T14:07:01.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7605000000000003e-05 2023-07-02T14:07:01.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:01.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962440+0000", 2023-07-02T14:07:01.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097334670000000009 2023-07-02T14:07:01.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:01.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.972174+0000", 2023-07-02T14:07:01.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7759000000000002e-05 2023-07-02T14:07:01.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.594 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:01.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.972211+0000", 2023-07-02T14:07:01.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017286838999999998 2023-07-02T14:07:01.595 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.595 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.595 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.595 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.595 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.595 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22470 2.5 e197/170 2:af9d8f16:::rbd_data.125de1449605.00000000000000b2:head v 197'2772, mlcod=197'2769)", 2023-07-02T14:07:01.595 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.062494+0000", 2023-07-02T14:07:01.596 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.436192161000001, 2023-07-02T14:07:01.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018691016000000001, 2023-07-02T14:07:01.596 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.596 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:01.596 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.062494+0000", 2023-07-02T14:07:01.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.062494+0000", 2023-07-02T14:07:01.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:01.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.062492+0000", 2023-07-02T14:07:01.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010932252999999999 2023-07-02T14:07:01.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073424+0000", 2023-07-02T14:07:01.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3642000000000001e-05 2023-07-02T14:07:01.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073438+0000", 2023-07-02T14:07:01.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6270000000000002e-06 2023-07-02T14:07:01.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073445+0000", 2023-07-02T14:07:01.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0265999999999997e-05 2023-07-02T14:07:01.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073485+0000", 2023-07-02T14:07:01.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5221000000000001e-05 2023-07-02T14:07:01.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:01.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073510+0000", 2023-07-02T14:07:01.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076289959999999999 2023-07-02T14:07:01.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:01.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.081139+0000", 2023-07-02T14:07:01.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5618999999999998e-05 2023-07-02T14:07:01.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:01.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.081185+0000", 2023-07-02T14:07:01.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018691016000000001 2023-07-02T14:07:01.602 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.603 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.603 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.603 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22516 2.5 e197/170 2:a545e9c7:::rbd_data.125de1449605.00000000000000c3:head v 197'2781, mlcod=197'2778)", 2023-07-02T14:07:01.603 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.146937+0000", 2023-07-02T14:07:01.603 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.351748840999999, 2023-07-02T14:07:01.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015143108000000001, 2023-07-02T14:07:01.604 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.604 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:01.604 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.604 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.604 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.146937+0000", 2023-07-02T14:07:01.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.604 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.146937+0000", 2023-07-02T14:07:01.605 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:01.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.605 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.146937+0000", 2023-07-02T14:07:01.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8502000000000001e-05 2023-07-02T14:07:01.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.147025+0000", 2023-07-02T14:07:01.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4660000000000005e-06 2023-07-02T14:07:01.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.147035+0000", 2023-07-02T14:07:01.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3360000000000002e-06 2023-07-02T14:07:01.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.147040+0000", 2023-07-02T14:07:01.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004086124 2023-07-02T14:07:01.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.608 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.151126+0000", 2023-07-02T14:07:01.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9261999999999999e-05 2023-07-02T14:07:01.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:01.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.151155+0000", 2023-07-02T14:07:01.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010884783 2023-07-02T14:07:01.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:01.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.162040+0000", 2023-07-02T14:07:01.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0268000000000001e-05 2023-07-02T14:07:01.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:01.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.162080+0000", 2023-07-02T14:07:01.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015143108000000001 2023-07-02T14:07:01.610 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.610 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.610 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.611 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22534 2.5 e197/170 2:b58a101b:::rbd_data.125de1449605.00000000000000c2:head v 197'2788, mlcod=197'2783)", 2023-07-02T14:07:01.611 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.173613+0000", 2023-07-02T14:07:01.611 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.325073344, 2023-07-02T14:07:01.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023211177999999999, 2023-07-02T14:07:01.611 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.611 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:01.612 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.173613+0000", 2023-07-02T14:07:01.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.173613+0000", 2023-07-02T14:07:01.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:01.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.613 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.173612+0000", 2023-07-02T14:07:01.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055730149999999997 2023-07-02T14:07:01.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.614 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179185+0000", 2023-07-02T14:07:01.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1826e-05 2023-07-02T14:07:01.614 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.614 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.614 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179197+0000", 2023-07-02T14:07:01.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9130000000000002e-06 2023-07-02T14:07:01.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179202+0000", 2023-07-02T14:07:01.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1668999999999997e-05 2023-07-02T14:07:01.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179244+0000", 2023-07-02T14:07:01.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6171999999999999e-05 2023-07-02T14:07:01.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:01.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179270+0000", 2023-07-02T14:07:01.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017512570000000002 2023-07-02T14:07:01.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:01.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.196782+0000", 2023-07-02T14:07:01.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1576999999999999e-05 2023-07-02T14:07:01.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:01.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.196824+0000", 2023-07-02T14:07:01.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023211177999999999 2023-07-02T14:07:01.618 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.618 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.618 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.619 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22539 2.5 e197/170 2:a6e4b3cf:::rbd_data.125de1449605.00000000000000c4:head v 197'2789, mlcod=197'2786)", 2023-07-02T14:07:01.619 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.179446+0000", 2023-07-02T14:07:01.619 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.319239540000002, 2023-07-02T14:07:01.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017409688999999999, 2023-07-02T14:07:01.619 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.619 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:01.620 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179446+0000", 2023-07-02T14:07:01.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179446+0000", 2023-07-02T14:07:01.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:01.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179446+0000", 2023-07-02T14:07:01.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082263240000000001 2023-07-02T14:07:01.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187672+0000", 2023-07-02T14:07:01.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7371e-05 2023-07-02T14:07:01.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.622 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187689+0000", 2023-07-02T14:07:01.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0820000000000005e-06 2023-07-02T14:07:01.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.623 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.623 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187698+0000", 2023-07-02T14:07:01.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0510999999999998e-05 2023-07-02T14:07:01.623 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187749+0000", 2023-07-02T14:07:01.624 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4805000000000001e-05 2023-07-02T14:07:01.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:01.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187774+0000", 2023-07-02T14:07:01.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.009056049 2023-07-02T14:07:01.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:01.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.196830+0000", 2023-07-02T14:07:01.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6347e-05 2023-07-02T14:07:01.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:01.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.196856+0000", 2023-07-02T14:07:01.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017409688999999999 2023-07-02T14:07:01.626 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.626 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.626 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.627 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22591 2.5 e197/170 2:bf746601:::rbd_data.125de1449605.00000000000000d6:head v 197'2799, mlcod=197'2795)", 2023-07-02T14:07:01.627 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.286442+0000", 2023-07-02T14:07:01.627 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 26.212243772000001, 2023-07-02T14:07:01.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016215869000000001, 2023-07-02T14:07:01.627 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.627 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:01.627 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.286442+0000", 2023-07-02T14:07:01.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.286442+0000", 2023-07-02T14:07:01.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:01.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.286441+0000", 2023-07-02T14:07:01.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00513759 2023-07-02T14:07:01.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291579+0000", 2023-07-02T14:07:01.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5048000000000001e-05 2023-07-02T14:07:01.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291604+0000", 2023-07-02T14:07:01.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1429e-05 2023-07-02T14:07:01.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291615+0000", 2023-07-02T14:07:01.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6360999999999994e-05 2023-07-02T14:07:01.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291712+0000", 2023-07-02T14:07:01.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9581999999999999e-05 2023-07-02T14:07:01.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:01.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291741+0000", 2023-07-02T14:07:01.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010882032999999999 2023-07-02T14:07:01.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:01.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.302623+0000", 2023-07-02T14:07:01.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4688999999999999e-05 2023-07-02T14:07:01.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:01.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.302658+0000", 2023-07-02T14:07:01.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016215869000000001 2023-07-02T14:07:01.634 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.634 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.634 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.635 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:23429 2.5 e206/170 2:aa2388f6:::rbd_data.125de1449605.00000000000001c4:head v 206'2898, mlcod=206'2896)", 2023-07-02T14:07:01.635 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:56.993306+0000", 2023-07-02T14:07:01.635 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.5053797629999996, 2023-07-02T14:07:01.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017604624999999999, 2023-07-02T14:07:01.635 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.635 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:01.635 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:56.993306+0000", 2023-07-02T14:07:01.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:56.993306+0000", 2023-07-02T14:07:01.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:01.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:56.993305+0000", 2023-07-02T14:07:01.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068210199999999997 2023-07-02T14:07:01.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.000126+0000", 2023-07-02T14:07:01.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1634999999999999e-05 2023-07-02T14:07:01.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.000147+0000", 2023-07-02T14:07:01.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3950000000000001e-06 2023-07-02T14:07:01.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.000156+0000", 2023-07-02T14:07:01.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012743300000000001 2023-07-02T14:07:01.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.000283+0000", 2023-07-02T14:07:01.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6681000000000001e-05 2023-07-02T14:07:01.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:01.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.000310+0000", 2023-07-02T14:07:01.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010566268 2023-07-02T14:07:01.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:01.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.010876+0000", 2023-07-02T14:07:01.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4650999999999997e-05 2023-07-02T14:07:01.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:01.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.010911+0000", 2023-07-02T14:07:01.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017604624999999999 2023-07-02T14:07:01.642 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.642 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.642 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.642 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:23563 2.5 e206/170 2:a468a258:::rbd_data.125de1449605.00000000000001de:head v 206'2917, mlcod=206'2915)", 2023-07-02T14:07:01.642 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:57.318124+0000", 2023-07-02T14:07:01.643 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.1805616260000003, 2023-07-02T14:07:01.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026542948, 2023-07-02T14:07:01.643 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.643 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:01.643 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318124+0000", 2023-07-02T14:07:01.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318124+0000", 2023-07-02T14:07:01.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:07:01.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318122+0000", 2023-07-02T14:07:01.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013275129 2023-07-02T14:07:01.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331397+0000", 2023-07-02T14:07:01.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.294e-05 2023-07-02T14:07:01.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331410+0000", 2023-07-02T14:07:01.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7190000000000001e-06 2023-07-02T14:07:01.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331418+0000", 2023-07-02T14:07:01.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1717000000000001e-05 2023-07-02T14:07:01.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331460+0000", 2023-07-02T14:07:01.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6067999999999998e-05 2023-07-02T14:07:01.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:01.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331486+0000", 2023-07-02T14:07:01.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013124998000000001 2023-07-02T14:07:01.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:01.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.344611+0000", 2023-07-02T14:07:01.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6536999999999998e-05 2023-07-02T14:07:01.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:01.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.344667+0000", 2023-07-02T14:07:01.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026542948 2023-07-02T14:07:01.649 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.651 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.651 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.652 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:23565 2.5 e206/170 2:bb77596d:::rbd_data.125de1449605.00000000000001df:head v 206'2919, mlcod=206'2915)", 2023-07-02T14:07:01.652 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:57.331506+0000", 2023-07-02T14:07:01.652 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 4.167180471, 2023-07-02T14:07:01.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016925951000000002, 2023-07-02T14:07:01.652 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.652 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:01.653 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331506+0000", 2023-07-02T14:07:01.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331506+0000", 2023-07-02T14:07:01.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:01.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331505+0000", 2023-07-02T14:07:01.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045442210000000002 2023-07-02T14:07:01.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.336049+0000", 2023-07-02T14:07:01.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8947e-05 2023-07-02T14:07:01.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.336068+0000", 2023-07-02T14:07:01.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9129999999999992e-06 2023-07-02T14:07:01.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.336077+0000", 2023-07-02T14:07:01.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029420649999999998 2023-07-02T14:07:01.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.339019+0000", 2023-07-02T14:07:01.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7931000000000002e-05 2023-07-02T14:07:01.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:01.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.339037+0000", 2023-07-02T14:07:01.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093659829999999996 2023-07-02T14:07:01.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:01.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.348403+0000", 2023-07-02T14:07:01.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8435000000000001e-05 2023-07-02T14:07:01.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:01.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.348431+0000", 2023-07-02T14:07:01.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016925951000000002 2023-07-02T14:07:01.659 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.659 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.659 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.660 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:23898 2.5 e207/170 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:head v 207'2973, mlcod=207'2967)", 2023-07-02T14:07:01.660 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:58.586875+0000", 2023-07-02T14:07:01.660 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 2.9118109510000001, 2023-07-02T14:07:01.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018821676999999998, 2023-07-02T14:07:01.660 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.660 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:01.660 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.586875+0000", 2023-07-02T14:07:01.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.586875+0000", 2023-07-02T14:07:01.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:01.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.586874+0000", 2023-07-02T14:07:01.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004881214 2023-07-02T14:07:01.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.591755+0000", 2023-07-02T14:07:01.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1639e-05 2023-07-02T14:07:01.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.591767+0000", 2023-07-02T14:07:01.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7150000000000001e-06 2023-07-02T14:07:01.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.591774+0000", 2023-07-02T14:07:01.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000128462 2023-07-02T14:07:01.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.591902+0000", 2023-07-02T14:07:01.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2172999999999999e-05 2023-07-02T14:07:01.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:01.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.591924+0000", 2023-07-02T14:07:01.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01373188 2023-07-02T14:07:01.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:01.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.605656+0000", 2023-07-02T14:07:01.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0467999999999999e-05 2023-07-02T14:07:01.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:01.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.605697+0000", 2023-07-02T14:07:01.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018821676999999998 2023-07-02T14:07:01.667 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.667 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.667 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.667 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:23949 2.5 e207/170 2:be65b5f1:::rbd_data.125de1449605.0000000000000043:head v 207'2981, mlcod=207'2977)", 2023-07-02T14:07:01.667 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:58.698231+0000", 2023-07-02T14:07:01.668 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 2.8004553749999999, 2023-07-02T14:07:01.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017072830000000001, 2023-07-02T14:07:01.668 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.668 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:01.668 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.698231+0000", 2023-07-02T14:07:01.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.698231+0000", 2023-07-02T14:07:01.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:01.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.698230+0000", 2023-07-02T14:07:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034408080000000001 2023-07-02T14:07:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.701671+0000", 2023-07-02T14:07:01.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1169999999999993e-06 2023-07-02T14:07:01.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.701680+0000", 2023-07-02T14:07:01.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9620000000000001e-06 2023-07-02T14:07:01.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.701685+0000", 2023-07-02T14:07:01.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7456e-05 2023-07-02T14:07:01.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.701742+0000", 2023-07-02T14:07:01.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1971e-05 2023-07-02T14:07:01.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:01.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.701764+0000", 2023-07-02T14:07:01.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013496115 2023-07-02T14:07:01.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:01.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.715261+0000", 2023-07-02T14:07:01.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2963999999999997e-05 2023-07-02T14:07:01.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:01.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.715303+0000", 2023-07-02T14:07:01.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017072830000000001 2023-07-02T14:07:01.674 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.675 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.675 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.675 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.675 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.675 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:01.676 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:07:01.677 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:01.676+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:01.777 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:01.777+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:01.802 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:01.802 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:07:01.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.802 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:25356 2.2 2.cd31250a (undecoded) ondisk+write+known_if_redirected e208)", 2023-07-02T14:07:01.803 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:07:01.412517+0000", 2023-07-02T14:07:01.803 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.38944632000000001, 2023-07-02T14:07:01.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.38947029399999999, 2023-07-02T14:07:01.803 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.803 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "delayed", 2023-07-02T14:07:01.803 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:01.803 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:01.803 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:01.804 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25356 2023-07-02T14:07:01.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.804 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.412517+0000", 2023-07-02T14:07:01.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.412517+0000", 2023-07-02T14:07:01.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:01.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.412516+0000", 2023-07-02T14:07:01.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00064120099999999999 2023-07-02T14:07:01.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.413157+0000", 2023-07-02T14:07:01.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6322000000000001e-05 2023-07-02T14:07:01.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.413173+0000", 2023-07-02T14:07:01.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.5690000000000005e-06 2023-07-02T14:07:01.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.413182+0000", 2023-07-02T14:07:01.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0359000000000003e-05 2023-07-02T14:07:01.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.413242+0000", 2023-07-02T14:07:01.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0471e-05 2023-07-02T14:07:01.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:01.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.413253+0000", 2023-07-02T14:07:01.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00073616999999999997 2023-07-02T14:07:01.809 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.809 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.809 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.810 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:25361 2.4 2.ef3adf34 (undecoded) ondisk+write+known_if_redirected e208)", 2023-07-02T14:07:01.810 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:07:01.438662+0000", 2023-07-02T14:07:01.810 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.36330090900000001, 2023-07-02T14:07:01.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36337660799999999, 2023-07-02T14:07:01.810 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.811 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "delayed", 2023-07-02T14:07:01.811 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:01.811 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:01.811 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:01.811 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25361 2023-07-02T14:07:01.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.811 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.438662+0000", 2023-07-02T14:07:01.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.812 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.438662+0000", 2023-07-02T14:07:01.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:01.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.813 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.438661+0000", 2023-07-02T14:07:01.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6209999999999999e-05 2023-07-02T14:07:01.813 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.813 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.813 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.438717+0000", 2023-07-02T14:07:01.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0407e-05 2023-07-02T14:07:01.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.438727+0000", 2023-07-02T14:07:01.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9509999999999997e-06 2023-07-02T14:07:01.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.438732+0000", 2023-07-02T14:07:01.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.261e-05 2023-07-02T14:07:01.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.438775+0000", 2023-07-02T14:07:01.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3550000000000001e-06 2023-07-02T14:07:01.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:01.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.438781+0000", 2023-07-02T14:07:01.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000119023 2023-07-02T14:07:01.816 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.817 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.817 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.817 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:25362 2.4 2.ef3adf34 (undecoded) ondisk+write+known_if_redirected e208)", 2023-07-02T14:07:01.817 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:07:01.438744+0000", 2023-07-02T14:07:01.817 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.36321880600000001, 2023-07-02T14:07:01.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36333580700000001, 2023-07-02T14:07:01.818 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.818 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "delayed", 2023-07-02T14:07:01.818 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:01.818 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:01.818 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:01.818 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25362 2023-07-02T14:07:01.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.819 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.438744+0000", 2023-07-02T14:07:01.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.438744+0000", 2023-07-02T14:07:01.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:01.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.438744+0000", 2023-07-02T14:07:01.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000480302 2023-07-02T14:07:01.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.439224+0000", 2023-07-02T14:07:01.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0040000000000005e-06 2023-07-02T14:07:01.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.439233+0000", 2023-07-02T14:07:01.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7859999999999998e-06 2023-07-02T14:07:01.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.439237+0000", 2023-07-02T14:07:01.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4311000000000002e-05 2023-07-02T14:07:01.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.823 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.439261+0000", 2023-07-02T14:07:01.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0520000000000001e-06 2023-07-02T14:07:01.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.823 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:01.824 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.439266+0000", 2023-07-02T14:07:01.824 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00052192700000000005 2023-07-02T14:07:01.824 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.824 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.824 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.824 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:25348 2.2 2.e69c6322 (undecoded) ondisk+write+known_if_redirected e208)", 2023-07-02T14:07:01.825 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:07:01.390394+0000", 2023-07-02T14:07:01.825 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.41156901099999998, 2023-07-02T14:07:01.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.41172717399999997, 2023-07-02T14:07:01.825 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.825 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "delayed", 2023-07-02T14:07:01.825 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:01.825 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25348 2023-07-02T14:07:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.390394+0000", 2023-07-02T14:07:01.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.390394+0000", 2023-07-02T14:07:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.390392+0000", 2023-07-02T14:07:01.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1748000000000004e-05 2023-07-02T14:07:01.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.390474+0000", 2023-07-02T14:07:01.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2645e-05 2023-07-02T14:07:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.390487+0000", 2023-07-02T14:07:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5760000000000002e-06 2023-07-02T14:07:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.390493+0000", 2023-07-02T14:07:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7759000000000001e-05 2023-07-02T14:07:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.390541+0000", 2023-07-02T14:07:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0299999999999996e-06 2023-07-02T14:07:01.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.390548+0000", 2023-07-02T14:07:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015421900000000001 2023-07-02T14:07:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:25349 2.2 2.e69c6322 (undecoded) ondisk+write+known_if_redirected e208)", 2023-07-02T14:07:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:07:01.390507+0000", 2023-07-02T14:07:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.41145640999999999, 2023-07-02T14:07:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.41165468799999999, 2023-07-02T14:07:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.832 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "delayed", 2023-07-02T14:07:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25349 2023-07-02T14:07:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.390507+0000", 2023-07-02T14:07:01.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.834 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.390507+0000", 2023-07-02T14:07:01.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:01.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.835 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.390506+0000", 2023-07-02T14:07:01.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00047790899999999999 2023-07-02T14:07:01.835 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.835 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.835 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.390984+0000", 2023-07-02T14:07:01.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2045000000000001e-05 2023-07-02T14:07:01.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.390996+0000", 2023-07-02T14:07:01.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8390000000000001e-06 2023-07-02T14:07:01.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.391003+0000", 2023-07-02T14:07:01.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6899999999999998e-05 2023-07-02T14:07:01.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.391090+0000", 2023-07-02T14:07:01.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0565e-05 2023-07-02T14:07:01.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:01.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.391100+0000", 2023-07-02T14:07:01.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00059351499999999997 2023-07-02T14:07:01.838 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.839 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.839 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.839 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:25355 2.2 2.cd31250a (undecoded) ondisk+write+known_if_redirected e208)", 2023-07-02T14:07:01.839 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:07:01.412442+0000", 2023-07-02T14:07:01.839 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 0.389521428, 2023-07-02T14:07:01.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.38975969300000002, 2023-07-02T14:07:01.840 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:01.840 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "delayed", 2023-07-02T14:07:01.840 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:01.840 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:01.840 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:01.840 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25355 2023-07-02T14:07:01.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.840 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:01.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:01.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.412442+0000", 2023-07-02T14:07:01.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:01.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:01.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.412442+0000", 2023-07-02T14:07:01.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:07:01.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:01.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.412440+0000", 2023-07-02T14:07:01.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.059e-05 2023-07-02T14:07:01.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:01.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.412450+0000", 2023-07-02T14:07:01.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6940000000000007e-06 2023-07-02T14:07:01.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:01.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.412460+0000", 2023-07-02T14:07:01.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9060000000000001e-06 2023-07-02T14:07:01.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:01.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.412465+0000", 2023-07-02T14:07:01.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7676999999999999e-05 2023-07-02T14:07:01.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:01.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.412503+0000", 2023-07-02T14:07:01.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0210000000000005e-06 2023-07-02T14:07:01.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:01.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:01.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:01.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:07:01.412512+0000", 2023-07-02T14:07:01.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0167999999999996e-05 2023-07-02T14:07:01.846 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.846 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:01.846 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.846 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:01.846 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:07:01.846 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 6 2023-07-02T14:07:01.846 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:01.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:07:01.877 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:01.878+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:01.971 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:01.971 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:07:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:07:01.972 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:07:01.972 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:01.978 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:01.978+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:01.981 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:07:02.079 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:02.079+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:02.112 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:02.112 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:07:02.113 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:07:02.113 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:07:02.113 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.113 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:07:02.114 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:07:02.114 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 232.65789908100001, 2023-07-02T14:07:02.114 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:07:02.114 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.115 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:07:02.115 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.115 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:02.115 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:02.116 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:07:02.116 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.116 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.117 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.117 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.117 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:07:02.117 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.118 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.118 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.118 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.118 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:07:02.119 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:02.119 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.119 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.119 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.120 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:07:02.120 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:07:02.120 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.120 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.121 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.121 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:07:02.121 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:07:02.121 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.122 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.122 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.122 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:07:02.122 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:07:02.123 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.123 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.123 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.123 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:07:02.124 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:07:02.124 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.124 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.124 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.125 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:07:02.125 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:07:02.125 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.125 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.126 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.126 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:07:02.126 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:07:02.127 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.127 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.127 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.127 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:07:02.128 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:07:02.128 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.128 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.129 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.129 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:07:02.129 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:07:02.129 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.130 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.130 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.130 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.131 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:07:02.131 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:07:02.131 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 231.78720607299999, 2023-07-02T14:07:02.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:07:02.132 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.132 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:02.132 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.132 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:07:02.133 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:07:02.133 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:07:02.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.133 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:07:02.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:07:02.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:02.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:07:02.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:07:02.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:07:02.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:07:02.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.139 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.139 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:07:02.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:07:02.140 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.140 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.140 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.140 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:07:02.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:07:02.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:07:02.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:07:02.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:07:02.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:07:02.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:07:02.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:07:02.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:07:02.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:07:02.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:07:02.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:07:02.146 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.146 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.146 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.147 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:07:02.147 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:07:02.147 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 231.787081352, 2023-07-02T14:07:02.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:07:02.147 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.147 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:07:02.147 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.148 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:02.148 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:02.148 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:07:02.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.148 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:07:02.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:07:02.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:02.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:07:02.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:07:02.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:07:02.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:07:02.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.151 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.151 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:07:02.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:07:02.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:07:02.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:07:02.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.152 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.152 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.152 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:07:02.152 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:07:02.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:07:02.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:07:02.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:07:02.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:07:02.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:07:02.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:07:02.155 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.155 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.155 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.155 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:07:02.155 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:07:02.156 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 231.78023146800001, 2023-07-02T14:07:02.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:07:02.156 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.156 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:07:02.156 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.156 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:02.156 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:02.156 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:07:02.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.157 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:07:02.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:07:02.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:07:02.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:07:02.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:07:02.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:07:02.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:07:02.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:07:02.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:07:02.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:07:02.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:07:02.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:07:02.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:07:02.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.161 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:07:02.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:07:02.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.162 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.162 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.162 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:07:02.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:07:02.162 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:07:02.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:07:02.163 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.163 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.163 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.164 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:07:02.164 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:07:02.164 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 231.780168543, 2023-07-02T14:07:02.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:07:02.164 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.164 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:07:02.164 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.165 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:02.165 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:02.165 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:07:02.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.165 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:07:02.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:07:02.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:02.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:07:02.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:07:02.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:07:02.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:07:02.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:07:02.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:07:02.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:07:02.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:07:02.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:07:02.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:07:02.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:07:02.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:07:02.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:07:02.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:07:02.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:07:02.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:07:02.172 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.172 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.172 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.172 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:07:02.172 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:07:02.172 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 231.77986812200001, 2023-07-02T14:07:02.173 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:07:02.173 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.173 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:02.173 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.173 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:02.173 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:02.173 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:07:02.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.174 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.174 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:07:02.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.174 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.174 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:07:02.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:02.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:07:02.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:07:02.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:07:02.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:07:02.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:07:02.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:07:02.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:07:02.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:07:02.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:07:02.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:07:02.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:07:02.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:07:02.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:07:02.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:07:02.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:07:02.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:07:02.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:07:02.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:07:02.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:07:02.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:02.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:07:02.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:07:02.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:02.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:07:02.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:07:02.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:02.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:07:02.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:07:02.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:02.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:07:02.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:07:02.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:07:02.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:07:02.184 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.184 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.184 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.184 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:07:02.185 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:07:02.185 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 231.779343622, 2023-07-02T14:07:02.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:07:02.185 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.185 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:02.185 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.185 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:02.185 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:02.186 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:07:02.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.186 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.186 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.186 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:07:02.186 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:07:02.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:02.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:07:02.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:07:02.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:07:02.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:07:02.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:07:02.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:07:02.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:07:02.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:07:02.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:07:02.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:07:02.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:07:02.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:07:02.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:07:02.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:07:02.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:07:02.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:07:02.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:07:02.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:07:02.193 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.193 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.193 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.193 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:07:02.194 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:07:02.194 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 204.65532449700001, 2023-07-02T14:07:02.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:07:02.194 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.194 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:07:02.194 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.194 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:02.195 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:02.195 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:07:02.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.195 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:07:02.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:07:02.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:02.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:07:02.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:07:02.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:07:02.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:07:02.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:07:02.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:07:02.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:07:02.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:07:02.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:07:02.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:07:02.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:07:02.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:07:02.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:07:02.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:07:02.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:07:02.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:07:02.201 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.201 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.202 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.202 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:07:02.202 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:07:02.202 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 204.32984179300001, 2023-07-02T14:07:02.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:07:02.202 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.203 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:02.203 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.203 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:02.203 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:02.203 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:07:02.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.203 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:07:02.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:07:02.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:02.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:07:02.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:07:02.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:07:02.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:07:02.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:07:02.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:07:02.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:07:02.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:07:02.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:07:02.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:07:02.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:07:02.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:07:02.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:07:02.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:07:02.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:07:02.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:07:02.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:07:02.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:07:02.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:07:02.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:02.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:07:02.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:07:02.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:02.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:07:02.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:07:02.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:02.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:07:02.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:07:02.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:07:02.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:07:02.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.213 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.213 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.213 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:17 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 0~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:02.214 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:07:02.214 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 110.91156601100001, 2023-07-02T14:07:02.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997, 2023-07-02T14:07:02.214 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.214 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:02.214 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.214 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:02.215 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:02.215 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17 2023-07-02T14:07:02.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.215 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:07:02.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:07:02.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:07:02.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198431+0000", 2023-07-02T14:07:02.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2815e-05 2023-07-02T14:07:02.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198443+0000", 2023-07-02T14:07:02.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2569999999999999e-05 2023-07-02T14:07:02.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198466+0000", 2023-07-02T14:07:02.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0223e-05 2023-07-02T14:07:02.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198476+0000", 2023-07-02T14:07:02.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1792999999999997e-05 2023-07-02T14:07:02.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.219 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:07:02.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4249999999999998e-06 2023-07-02T14:07:02.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198536+0000", 2023-07-02T14:07:02.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91888663500000001 2023-07-02T14:07:02.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117423+0000", 2023-07-02T14:07:02.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223594 2023-07-02T14:07:02.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117647+0000", 2023-07-02T14:07:02.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033832900000000001 2023-07-02T14:07:02.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117985+0000", 2023-07-02T14:07:02.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997 2023-07-02T14:07:02.222 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.222 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.222 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.223 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:18 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 65536~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:02.223 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:07:02.223 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 110.91150879999999, 2023-07-02T14:07:02.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998, 2023-07-02T14:07:02.223 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.223 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:02.223 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.223 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:02.224 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:02.224 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18 2023-07-02T14:07:02.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.224 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:07:02.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:07:02.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:02.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:07:02.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7600000000000002e-06 2023-07-02T14:07:02.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198494+0000", 2023-07-02T14:07:02.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7840000000000004e-06 2023-07-02T14:07:02.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198502+0000", 2023-07-02T14:07:02.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6459999999999998e-06 2023-07-02T14:07:02.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198508+0000", 2023-07-02T14:07:02.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7395999999999997e-05 2023-07-02T14:07:02.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198565+0000", 2023-07-02T14:07:02.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6800000000000004e-06 2023-07-02T14:07:02.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198572+0000", 2023-07-02T14:07:02.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91965896800000002 2023-07-02T14:07:02.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118231+0000", 2023-07-02T14:07:02.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5212999999999998e-05 2023-07-02T14:07:02.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118326+0000", 2023-07-02T14:07:02.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000238538 2023-07-02T14:07:02.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118565+0000", 2023-07-02T14:07:02.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998 2023-07-02T14:07:02.231 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.231 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.231 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.232 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:19 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 131072~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:02.232 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:07:02.232 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 110.91148457600001, 2023-07-02T14:07:02.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002, 2023-07-02T14:07:02.232 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.232 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:02.232 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.232 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:02.233 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:02.233 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 19 2023-07-02T14:07:02.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.233 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:07:02.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:07:02.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:02.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:07:02.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1990000000000002e-06 2023-07-02T14:07:02.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198517+0000", 2023-07-02T14:07:02.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8789999999999997e-06 2023-07-02T14:07:02.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198524+0000", 2023-07-02T14:07:02.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:07:02.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:07:02.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5545999999999999e-05 2023-07-02T14:07:02.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:07:02.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2540000000000003e-06 2023-07-02T14:07:02.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198598+0000", 2023-07-02T14:07:02.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.919983263 2023-07-02T14:07:02.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118581+0000", 2023-07-02T14:07:02.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6878999999999994e-05 2023-07-02T14:07:02.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118648+0000", 2023-07-02T14:07:02.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149732 2023-07-02T14:07:02.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118798+0000", 2023-07-02T14:07:02.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002 2023-07-02T14:07:02.240 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.240 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.240 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.241 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:20 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 262144~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:02.241 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:07:02.241 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 110.911465612, 2023-07-02T14:07:02.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998, 2023-07-02T14:07:02.241 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.241 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:02.241 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.241 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:02.242 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:02.242 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20 2023-07-02T14:07:02.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.242 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:07:02.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:07:02.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:02.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:07:02.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.732e-06 2023-07-02T14:07:02.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198537+0000", 2023-07-02T14:07:02.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5259999999999999e-06 2023-07-02T14:07:02.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198543+0000", 2023-07-02T14:07:02.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.121e-06 2023-07-02T14:07:02.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198547+0000", 2023-07-02T14:07:02.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9591000000000005e-05 2023-07-02T14:07:02.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:07:02.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3120000000000004e-06 2023-07-02T14:07:02.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198621+0000", 2023-07-02T14:07:02.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92020789400000003 2023-07-02T14:07:02.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118829+0000", 2023-07-02T14:07:02.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022393386000000001 2023-07-02T14:07:02.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141223+0000", 2023-07-02T14:07:02.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000329866 2023-07-02T14:07:02.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141552+0000", 2023-07-02T14:07:02.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998 2023-07-02T14:07:02.249 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.249 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.249 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.250 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:21 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 196608~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:02.250 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:07:02.250 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 110.91143797700001, 2023-07-02T14:07:02.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995, 2023-07-02T14:07:02.250 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.250 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:02.250 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.250 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:02.251 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:02.251 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21 2023-07-02T14:07:02.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.251 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:07:02.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:07:02.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:02.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:07:02.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0169999999999999e-05 2023-07-02T14:07:02.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198571+0000", 2023-07-02T14:07:02.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3479999999999998e-06 2023-07-02T14:07:02.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198578+0000", 2023-07-02T14:07:02.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8439999999999997e-06 2023-07-02T14:07:02.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198583+0000", 2023-07-02T14:07:02.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8294999999999997e-05 2023-07-02T14:07:02.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198641+0000", 2023-07-02T14:07:02.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8120000000000001e-06 2023-07-02T14:07:02.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:07:02.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94293828499999999 2023-07-02T14:07:02.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141583+0000", 2023-07-02T14:07:02.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014607300000000001 2023-07-02T14:07:02.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141729+0000", 2023-07-02T14:07:02.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00044906600000000003 2023-07-02T14:07:02.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142178+0000", 2023-07-02T14:07:02.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995 2023-07-02T14:07:02.258 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.258 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.258 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.259 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:22 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 327680~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:02.259 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:07:02.259 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 110.911404547, 2023-07-02T14:07:02.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004, 2023-07-02T14:07:02.259 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.259 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:02.259 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.260 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:02.260 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:02.260 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 22 2023-07-02T14:07:02.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.260 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:07:02.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:07:02.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:02.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:07:02.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4850000000000001e-06 2023-07-02T14:07:02.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198597+0000", 2023-07-02T14:07:02.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:07:02.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198605+0000", 2023-07-02T14:07:02.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4839999999999998e-06 2023-07-02T14:07:02.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198609+0000", 2023-07-02T14:07:02.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3100999999999999e-05 2023-07-02T14:07:02.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198662+0000", 2023-07-02T14:07:02.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9099999999999998e-06 2023-07-02T14:07:02.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198666+0000", 2023-07-02T14:07:02.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.943539503 2023-07-02T14:07:02.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142206+0000", 2023-07-02T14:07:02.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012255999999999999 2023-07-02T14:07:02.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142328+0000", 2023-07-02T14:07:02.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208956 2023-07-02T14:07:02.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142537+0000", 2023-07-02T14:07:02.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004 2023-07-02T14:07:02.267 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.267 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.267 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.268 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:23 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 393216~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:02.268 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:07:02.268 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 110.911384567, 2023-07-02T14:07:02.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348, 2023-07-02T14:07:02.268 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.268 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:02.268 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.269 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:02.269 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:02.269 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23 2023-07-02T14:07:02.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.269 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:07:02.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:07:02.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:02.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:07:02.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8439999999999998e-06 2023-07-02T14:07:02.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:07:02.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0440000000000001e-06 2023-07-02T14:07:02.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198623+0000", 2023-07-02T14:07:02.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4480000000000002e-06 2023-07-02T14:07:02.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198628+0000", 2023-07-02T14:07:02.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5843e-05 2023-07-02T14:07:02.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:07:02.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.89e-06 2023-07-02T14:07:02.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198688+0000", 2023-07-02T14:07:02.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94389171999999999 2023-07-02T14:07:02.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142580+0000", 2023-07-02T14:07:02.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3892000000000004e-05 2023-07-02T14:07:02.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142644+0000", 2023-07-02T14:07:02.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017608000000000001 2023-07-02T14:07:02.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142820+0000", 2023-07-02T14:07:02.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348 2023-07-02T14:07:02.276 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.276 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.276 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.277 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:24 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 458752~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:02.277 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:07:02.277 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 110.911356493, 2023-07-02T14:07:02.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004, 2023-07-02T14:07:02.277 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.277 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:02.277 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.278 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:02.278 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:02.278 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 24 2023-07-02T14:07:02.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.278 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:07:02.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:07:02.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:02.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198642+0000", 2023-07-02T14:07:02.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.253e-06 2023-07-02T14:07:02.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:07:02.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1289999999999998e-06 2023-07-02T14:07:02.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198652+0000", 2023-07-02T14:07:02.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4510000000000002e-06 2023-07-02T14:07:02.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198657+0000", 2023-07-02T14:07:02.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9131000000000001e-05 2023-07-02T14:07:02.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198706+0000", 2023-07-02T14:07:02.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8890000000000004e-06 2023-07-02T14:07:02.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198710+0000", 2023-07-02T14:07:02.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94415412200000004 2023-07-02T14:07:02.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142864+0000", 2023-07-02T14:07:02.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8047000000000006e-05 2023-07-02T14:07:02.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142952+0000", 2023-07-02T14:07:02.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192967 2023-07-02T14:07:02.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143145+0000", 2023-07-02T14:07:02.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004 2023-07-02T14:07:02.285 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.285 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.285 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.286 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:25 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 524288~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:02.286 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:07:02.286 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 110.911337698, 2023-07-02T14:07:02.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001, 2023-07-02T14:07:02.286 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.286 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:02.286 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.287 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:02.287 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:02.287 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25 2023-07-02T14:07:02.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.287 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:07:02.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:07:02.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:02.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:07:02.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.615e-06 2023-07-02T14:07:02.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198664+0000", 2023-07-02T14:07:02.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4759999999999997e-06 2023-07-02T14:07:02.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198669+0000", 2023-07-02T14:07:02.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5999999999999998e-06 2023-07-02T14:07:02.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198673+0000", 2023-07-02T14:07:02.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4256999999999999e-05 2023-07-02T14:07:02.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198727+0000", 2023-07-02T14:07:02.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840000000000001e-06 2023-07-02T14:07:02.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198731+0000", 2023-07-02T14:07:02.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94444241900000003 2023-07-02T14:07:02.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143173+0000", 2023-07-02T14:07:02.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6839000000000002e-05 2023-07-02T14:07:02.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143240+0000", 2023-07-02T14:07:02.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017053399999999999 2023-07-02T14:07:02.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143411+0000", 2023-07-02T14:07:02.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001 2023-07-02T14:07:02.294 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.294 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.294 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.295 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:26 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 589824~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:02.295 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:07:02.295 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 110.911314547, 2023-07-02T14:07:02.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997, 2023-07-02T14:07:02.295 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.295 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:02.295 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.296 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:02.296 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:02.296 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 26 2023-07-02T14:07:02.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.296 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:07:02.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:07:02.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:02.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:07:02.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0520000000000002e-06 2023-07-02T14:07:02.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198687+0000", 2023-07-02T14:07:02.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1880000000000002e-06 2023-07-02T14:07:02.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198695+0000", 2023-07-02T14:07:02.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9960000000000004e-06 2023-07-02T14:07:02.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198699+0000", 2023-07-02T14:07:02.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2330999999999998e-05 2023-07-02T14:07:02.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198742+0000", 2023-07-02T14:07:02.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7950000000000001e-06 2023-07-02T14:07:02.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198745+0000", 2023-07-02T14:07:02.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94469398199999999 2023-07-02T14:07:02.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143439+0000", 2023-07-02T14:07:02.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6048e-05 2023-07-02T14:07:02.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143525+0000", 2023-07-02T14:07:02.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017016700000000001 2023-07-02T14:07:02.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143696+0000", 2023-07-02T14:07:02.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997 2023-07-02T14:07:02.303 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.303 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.303 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.304 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.6579.0:11 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected e171)", 2023-07-02T14:07:02.304 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:07:02.304 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 64.951516061000007, 2023-07-02T14:07:02.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998, 2023-07-02T14:07:02.304 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.304 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:02.304 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.305 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.6579", 2023-07-02T14:07:02.305 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/554403467", 2023-07-02T14:07:02.305 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11 2023-07-02T14:07:02.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.305 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:07:02.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:07:02.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:02.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158481+0000", 2023-07-02T14:07:02.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:07:02.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158487+0000", 2023-07-02T14:07:02.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7630000000000006e-06 2023-07-02T14:07:02.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158496+0000", 2023-07-02T14:07:02.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5100000000000001e-06 2023-07-02T14:07:02.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158504+0000", 2023-07-02T14:07:02.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9397000000000002e-05 2023-07-02T14:07:02.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158553+0000", 2023-07-02T14:07:02.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5810000000000003e-06 2023-07-02T14:07:02.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158562+0000", 2023-07-02T14:07:02.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93893079499999998 2023-07-02T14:07:02.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097493+0000", 2023-07-02T14:07:02.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015799299999999999 2023-07-02T14:07:02.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097651+0000", 2023-07-02T14:07:02.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187746 2023-07-02T14:07:02.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097839+0000", 2023-07-02T14:07:02.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998 2023-07-02T14:07:02.312 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.312 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.312 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.312 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.313 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.313 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:02.314 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap63 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap64 b3fbb028-e267-4d25-a750-93ba18a2316a@snap64.diff 2023-07-02T14:07:02.314 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:07:02.315 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:02.313+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:02.415 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:02.414+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:02.480 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:02.481 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:07:02.481 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:07:02.481 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:02.491 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:07:02.532 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:02.531+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:02.617 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:02.618+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:02.654 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:02.655 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:07:02.655 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:07:02.655 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:07:02.655 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:02.665 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:07:02.718 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:02.717+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:02.791 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:02.791 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:07:02.791 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:07:02.791 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:07:02.792 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.792 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:07:02.792 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:07:02.792 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 241.68173617900001, 2023-07-02T14:07:02.792 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:07:02.792 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.792 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:02.793 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.793 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:02.793 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:02.793 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:07:02.793 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.793 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.793 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.793 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:07:02.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.794 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.794 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.794 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:07:02.794 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:07:02.794 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.795 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:07:02.795 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:07:02.795 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.795 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.795 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:07:02.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:07:02.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.796 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.796 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.796 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:07:02.796 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:07:02.796 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:07:02.797 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:07:02.797 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.797 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.797 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.797 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:07:02.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:07:02.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.798 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.798 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:07:02.798 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:07:02.798 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.798 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:07:02.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:07:02.799 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.799 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.799 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.799 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:07:02.799 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:07:02.800 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.800 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.800 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.800 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:07:02.800 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:07:02.800 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.800 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.801 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.801 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.801 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.801 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:07:02.801 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:07:02.801 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 241.68167544100001, 2023-07-02T14:07:02.801 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:07:02.801 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.802 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:02.802 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.802 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:02.802 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:02.802 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:07:02.802 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.802 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.802 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:07:02.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.803 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.803 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.803 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.803 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:07:02.803 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:02.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.804 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:07:02.804 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:07:02.804 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.804 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.804 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:07:02.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:07:02.805 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.805 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.805 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.805 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:07:02.805 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:07:02.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.806 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:07:02.806 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:07:02.806 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.806 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.806 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:07:02.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:07:02.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.807 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.807 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.807 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:07:02.807 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:07:02.807 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:07:02.808 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:07:02.808 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.808 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.808 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.808 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:07:02.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:07:02.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.809 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:02.809 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:07:02.809 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:07:02.809 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.809 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:02.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:07:02.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:07:02.810 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.810 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.810 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:02.810 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:07:02.810 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:07:02.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:02.811 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:07:02.811 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:07:02.811 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.811 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.811 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.812 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:07:02.812 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:07:02.812 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.812 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.812 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.812 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.812 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.813 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:07:02.813 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:07:02.813 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 231.451221111, 2023-07-02T14:07:02.813 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:07:02.813 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.813 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:02.813 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.813 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:02.814 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:02.814 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:07:02.814 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.814 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.814 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.814 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:07:02.814 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.815 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:07:02.815 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:02.815 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.815 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.815 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:07:02.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:07:02.816 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.816 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.816 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.816 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:07:02.816 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:07:02.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.817 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:07:02.817 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:07:02.817 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.817 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.817 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:07:02.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:07:02.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.818 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.818 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.818 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:07:02.818 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:07:02.818 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:07:02.819 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:07:02.819 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.819 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.819 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:02.819 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:07:02.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:07:02.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.820 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:02.820 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:07:02.820 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:07:02.820 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.820 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:02.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:07:02.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:07:02.821 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.821 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.821 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:02.821 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:07:02.821 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:07:02.822 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.822 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.822 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.822 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:07:02.822 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:07:02.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.822 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.822 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.823 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.823 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.823 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9620 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 589824~65536] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:07:02.823 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:07:02.823 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 157.76720651299999, 2023-07-02T14:07:02.823 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001, 2023-07-02T14:07:02.823 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.824 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:02.824 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.824 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:02.824 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:02.824 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9620 2023-07-02T14:07:02.824 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.824 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.824 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:07:02.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.825 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.825 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.825 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.825 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:07:02.825 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:07:02.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.826 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022203+0000", 2023-07-02T14:07:02.826 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0346000000000001e-05 2023-07-02T14:07:02.826 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.826 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.826 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022263+0000", 2023-07-02T14:07:02.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4657999999999999e-05 2023-07-02T14:07:02.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.827 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.827 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.827 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022278+0000", 2023-07-02T14:07:02.827 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:07:02.827 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022285+0000", 2023-07-02T14:07:02.828 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0096000000000003e-05 2023-07-02T14:07:02.828 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.828 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.828 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.828 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022335+0000", 2023-07-02T14:07:02.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2334000000000001e-05 2023-07-02T14:07:02.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.829 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.829 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022347+0000", 2023-07-02T14:07:02.829 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095653701999999993 2023-07-02T14:07:02.829 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.829 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118001+0000", 2023-07-02T14:07:02.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000185797 2023-07-02T14:07:02.830 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.830 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.830 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.830 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118187+0000", 2023-07-02T14:07:02.830 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026397900000000001 2023-07-02T14:07:02.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:02.831 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118451+0000", 2023-07-02T14:07:02.831 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051832800000000002 2023-07-02T14:07:02.831 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.831 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.831 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:02.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.123634+0000", 2023-07-02T14:07:02.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001227332 2023-07-02T14:07:02.832 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.832 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.832 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:02.832 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124861+0000", 2023-07-02T14:07:02.832 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9493999999999998e-05 2023-07-02T14:07:02.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:02.833 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124901+0000", 2023-07-02T14:07:02.833 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0462000000000001e-05 2023-07-02T14:07:02.833 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.833 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.833 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.834 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124931+0000", 2023-07-02T14:07:02.834 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001 2023-07-02T14:07:02.834 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.834 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.834 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.834 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.834 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.834 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9621 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 655360~1966080] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:07:02.835 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:07:02.835 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 157.76701913100001, 2023-07-02T14:07:02.835 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999, 2023-07-02T14:07:02.835 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.835 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:02.835 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.835 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:02.835 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:02.836 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9621 2023-07-02T14:07:02.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.836 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.836 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.836 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.836 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:07:02.836 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.836 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:07:02.837 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:02.837 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.837 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.837 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.837 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022391+0000", 2023-07-02T14:07:02.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032053939999999999 2023-07-02T14:07:02.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.838 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.838 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025597+0000", 2023-07-02T14:07:02.838 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4307999999999999e-05 2023-07-02T14:07:02.838 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.838 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025611+0000", 2023-07-02T14:07:02.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.032e-06 2023-07-02T14:07:02.839 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.839 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.839 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.839 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025617+0000", 2023-07-02T14:07:02.839 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8977999999999998e-05 2023-07-02T14:07:02.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.840 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.840 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025656+0000", 2023-07-02T14:07:02.840 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:07:02.840 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.840 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:02.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025663+0000", 2023-07-02T14:07:02.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095220739999999998 2023-07-02T14:07:02.841 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.841 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.841 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.841 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120884+0000", 2023-07-02T14:07:02.841 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2525000000000001e-05 2023-07-02T14:07:02.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.842 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120956+0000", 2023-07-02T14:07:02.842 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016232400000000001 2023-07-02T14:07:02.842 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.842 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.842 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:02.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.121119+0000", 2023-07-02T14:07:02.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044007380000000004 2023-07-02T14:07:02.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.843 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.843 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:02.843 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.125519+0000", 2023-07-02T14:07:02.843 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000797209 2023-07-02T14:07:02.843 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:02.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126317+0000", 2023-07-02T14:07:02.844 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1228000000000004e-05 2023-07-02T14:07:02.844 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.844 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.844 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:02.844 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126378+0000", 2023-07-02T14:07:02.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978000000000003e-05 2023-07-02T14:07:02.845 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.845 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.845 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.845 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126430+0000", 2023-07-02T14:07:02.845 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999 2023-07-02T14:07:02.845 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.845 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.846 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.846 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.846 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.846 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:281 2.1 2:901d5a95:::rbd_data.125de1449605.0000000000000118:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:07:02.846 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:07:02.846 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 149.23394302, 2023-07-02T14:07:02.846 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002, 2023-07-02T14:07:02.846 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.847 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:02.847 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.847 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:07:02.847 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:07:02.847 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:07:02.847 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.847 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.847 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:07:02.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.848 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.848 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.848 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.848 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:07:02.848 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:02.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.849 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.849 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:07:02.849 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0469999999999998e-06 2023-07-02T14:07:02.849 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.849 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555471+0000", 2023-07-02T14:07:02.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8860000000000001e-06 2023-07-02T14:07:02.850 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.850 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.850 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.850 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555476+0000", 2023-07-02T14:07:02.850 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3160000000000002e-06 2023-07-02T14:07:02.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.851 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555483+0000", 2023-07-02T14:07:02.851 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989525000000001 2023-07-02T14:07:02.851 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.851 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.851 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591472+0000", 2023-07-02T14:07:02.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3708999999999996e-05 2023-07-02T14:07:02.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.852 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.852 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.852 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591536+0000", 2023-07-02T14:07:02.852 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055109720000000003 2023-07-02T14:07:02.852 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.853 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.853 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.853 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.597047+0000", 2023-07-02T14:07:02.853 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002 2023-07-02T14:07:02.853 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.853 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.853 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.853 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.854 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12985 2.7 2:eeddf24b:::rbd_data.125de1449605.000000000000007e:head [write 65536~4128768] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:07:02.854 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:07:02.854 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 130.00641607399999, 2023-07-02T14:07:02.854 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999, 2023-07-02T14:07:02.854 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.854 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:02.854 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.855 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:02.855 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:02.855 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12985 2023-07-02T14:07:02.855 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.855 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.855 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.855 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.855 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:07:02.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.856 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.856 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:07:02.856 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:02.856 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.856 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782994+0000", 2023-07-02T14:07:02.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019254741999999998 2023-07-02T14:07:02.857 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.857 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.857 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.857 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802248+0000", 2023-07-02T14:07:02.857 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0451e-05 2023-07-02T14:07:02.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.858 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802279+0000", 2023-07-02T14:07:02.858 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3318e-05 2023-07-02T14:07:02.858 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.858 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.858 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802292+0000", 2023-07-02T14:07:02.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109403 2023-07-02T14:07:02.859 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.859 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.859 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.859 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802402+0000", 2023-07-02T14:07:02.859 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012520599999999999 2023-07-02T14:07:02.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.860 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802527+0000", 2023-07-02T14:07:02.860 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187768 2023-07-02T14:07:02.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.860 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.860 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:02.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802715+0000", 2023-07-02T14:07:02.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013192822 2023-07-02T14:07:02.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.861 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.861 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:02.861 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.815907+0000", 2023-07-02T14:07:02.861 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0086451870000000004 2023-07-02T14:07:02.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:02.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824553+0000", 2023-07-02T14:07:02.862 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6759999999999999e-05 2023-07-02T14:07:02.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.862 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.862 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:02.862 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824609+0000", 2023-07-02T14:07:02.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011314700000000001 2023-07-02T14:07:02.863 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.863 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.863 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.863 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824722+0000", 2023-07-02T14:07:02.863 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999 2023-07-02T14:07:02.863 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.863 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.864 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.864 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.864 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.864 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:165 2.3 2:c9c6393a:::rbd_data.125de1449605.000000000000002c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:07:02.864 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:07:02.864 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 125.949633257, 2023-07-02T14:07:02.864 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002, 2023-07-02T14:07:02.865 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.865 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:02.865 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.865 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:07:02.865 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:07:02.865 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 165 2023-07-02T14:07:02.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.865 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:07:02.866 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.866 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.866 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.866 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.866 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:07:02.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:02.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.867 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.867 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839777+0000", 2023-07-02T14:07:02.867 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6740000000000001e-06 2023-07-02T14:07:02.867 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.867 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839780+0000", 2023-07-02T14:07:02.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9030000000000004e-06 2023-07-02T14:07:02.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.868 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.868 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839788+0000", 2023-07-02T14:07:02.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1799999999999998e-06 2023-07-02T14:07:02.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.869 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839792+0000", 2023-07-02T14:07:02.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.031592227000000001 2023-07-02T14:07:02.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871384+0000", 2023-07-02T14:07:02.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013600099999999999 2023-07-02T14:07:02.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871520+0000", 2023-07-02T14:07:02.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082817089999999999 2023-07-02T14:07:02.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.879802+0000", 2023-07-02T14:07:02.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002 2023-07-02T14:07:02.871 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.871 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.871 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.872 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:252 2.7 2:f1d0abf0:::rbd_data.125de1449605.0000000000000083:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:07:02.872 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:07:02.872 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 125.533348287, 2023-07-02T14:07:02.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997, 2023-07-02T14:07:02.872 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.872 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:02.873 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.873 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:07:02.873 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:07:02.873 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 252 2023-07-02T14:07:02.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.873 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:07:02.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.874 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:07:02.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:02.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256062+0000", 2023-07-02T14:07:02.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0149999999999999e-06 2023-07-02T14:07:02.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256065+0000", 2023-07-02T14:07:02.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5269999999999998e-06 2023-07-02T14:07:02.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.876 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.876 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256071+0000", 2023-07-02T14:07:02.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:07:02.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256074+0000", 2023-07-02T14:07:02.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028852909 2023-07-02T14:07:02.877 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.877 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.284927+0000", 2023-07-02T14:07:02.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168274 2023-07-02T14:07:02.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.285096+0000", 2023-07-02T14:07:02.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011036300000000001 2023-07-02T14:07:02.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296132+0000", 2023-07-02T14:07:02.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997 2023-07-02T14:07:02.879 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.879 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.879 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.880 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:261 2.1 2:971731cb:::rbd_data.125de1449605.000000000000008c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:07:02.880 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:07:02.880 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 125.533252812, 2023-07-02T14:07:02.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984, 2023-07-02T14:07:02.880 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.880 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:02.880 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.880 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:07:02.881 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:07:02.881 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 261 2023-07-02T14:07:02.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.881 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:07:02.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:07:02.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:02.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:07:02.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2819999999999999e-06 2023-07-02T14:07:02.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256160+0000", 2023-07-02T14:07:02.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2020000000000003e-06 2023-07-02T14:07:02.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.884 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256165+0000", 2023-07-02T14:07:02.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2100000000000003e-06 2023-07-02T14:07:02.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.884 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256170+0000", 2023-07-02T14:07:02.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025871715999999999 2023-07-02T14:07:02.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.885 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.885 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282041+0000", 2023-07-02T14:07:02.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132364 2023-07-02T14:07:02.885 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282174+0000", 2023-07-02T14:07:02.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014044566 2023-07-02T14:07:02.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296218+0000", 2023-07-02T14:07:02.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984 2023-07-02T14:07:02.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.887 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.887 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.887 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:264 2.3 2:dbf06c9d:::rbd_data.125de1449605.000000000000008f:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:07:02.887 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:07:02.888 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 125.533218694, 2023-07-02T14:07:02.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003, 2023-07-02T14:07:02.888 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.888 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:02.888 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.888 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:07:02.888 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:07:02.889 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 264 2023-07-02T14:07:02.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.889 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:07:02.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:07:02.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:02.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:07:02.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5749999999999999e-06 2023-07-02T14:07:02.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256195+0000", 2023-07-02T14:07:02.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.327e-06 2023-07-02T14:07:02.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256200+0000", 2023-07-02T14:07:02.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0580000000000002e-06 2023-07-02T14:07:02.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.892 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.892 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256203+0000", 2023-07-02T14:07:02.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035707524999999997 2023-07-02T14:07:02.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.291911+0000", 2023-07-02T14:07:02.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014269799999999999 2023-07-02T14:07:02.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.292053+0000", 2023-07-02T14:07:02.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062760309999999996 2023-07-02T14:07:02.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.298329+0000", 2023-07-02T14:07:02.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003 2023-07-02T14:07:02.894 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.895 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.895 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.895 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:341 2.3 2:cee258bd:::rbd_data.125de1449605.00000000000000dc:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:07:02.895 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:07:02.895 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 125.152731367, 2023-07-02T14:07:02.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997, 2023-07-02T14:07:02.896 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.896 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:02.896 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.896 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:07:02.896 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:07:02.896 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 341 2023-07-02T14:07:02.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.897 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:07:02.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:07:02.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:02.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636679+0000", 2023-07-02T14:07:02.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6240000000000002e-06 2023-07-02T14:07:02.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636682+0000", 2023-07-02T14:07:02.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6910000000000002e-06 2023-07-02T14:07:02.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636688+0000", 2023-07-02T14:07:02.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3469999999999999e-06 2023-07-02T14:07:02.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.900 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.900 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636691+0000", 2023-07-02T14:07:02.900 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035625560000000001 2023-07-02T14:07:02.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672317+0000", 2023-07-02T14:07:02.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108391 2023-07-02T14:07:02.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672425+0000", 2023-07-02T14:07:02.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012104976 2023-07-02T14:07:02.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.684530+0000", 2023-07-02T14:07:02.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997 2023-07-02T14:07:02.902 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.902 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.902 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.903 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPushReply(2.7 139/134 [PushReplyOp(2:e3782f39:::rbd_data.125de1449605.0000000000000120:head)])", 2023-07-02T14:07:02.903 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:07:02.903 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 110.862199214, 2023-07-02T14:07:02.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998, 2023-07-02T14:07:02.903 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.904 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:07:02.904 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:07:02.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:07:02.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:02.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927211+0000", 2023-07-02T14:07:02.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3689999999999998e-06 2023-07-02T14:07:02.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927217+0000", 2023-07-02T14:07:02.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7269999999999996e-06 2023-07-02T14:07:02.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927225+0000", 2023-07-02T14:07:02.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:07:02.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927230+0000", 2023-07-02T14:07:02.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:07:02.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927266+0000", 2023-07-02T14:07:02.908 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.069972871000000006 2023-07-02T14:07:02.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.908 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.908 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.997239+0000", 2023-07-02T14:07:02.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998 2023-07-02T14:07:02.909 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.909 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.909 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.909 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19179 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2832, mlcod=177'2827)", 2023-07-02T14:07:02.909 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:07:02.910 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.330669383999997, 2023-07-02T14:07:02.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051928864999999998, 2023-07-02T14:07:02.910 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.910 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:02.910 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:07:02.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:07:02.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:02.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458741+0000", 2023-07-02T14:07:02.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01233852 2023-07-02T14:07:02.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471079+0000", 2023-07-02T14:07:02.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8131e-05 2023-07-02T14:07:02.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471097+0000", 2023-07-02T14:07:02.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.19e-06 2023-07-02T14:07:02.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471104+0000", 2023-07-02T14:07:02.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4463999999999999e-05 2023-07-02T14:07:02.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471158+0000", 2023-07-02T14:07:02.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1723000000000001e-05 2023-07-02T14:07:02.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471190+0000", 2023-07-02T14:07:02.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039438622999999999 2023-07-02T14:07:02.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:02.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510628+0000", 2023-07-02T14:07:02.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2280000000000002e-05 2023-07-02T14:07:02.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.916 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.916 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510671+0000", 2023-07-02T14:07:02.916 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051928864999999998 2023-07-02T14:07:02.917 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.917 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.917 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.917 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.917 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19180 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2833, mlcod=177'2827)", 2023-07-02T14:07:02.917 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:07:02.917 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.318292143999997, 2023-07-02T14:07:02.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039574109000000003, 2023-07-02T14:07:02.918 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.918 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:02.918 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:07:02.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:07:02.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:02.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471118+0000", 2023-07-02T14:07:02.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4799000000000001e-05 2023-07-02T14:07:02.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471143+0000", 2023-07-02T14:07:02.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3960000000000003e-06 2023-07-02T14:07:02.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471149+0000", 2023-07-02T14:07:02.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9230000000000004e-06 2023-07-02T14:07:02.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471153+0000", 2023-07-02T14:07:02.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015076050000000001 2023-07-02T14:07:02.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.922 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.486229+0000", 2023-07-02T14:07:02.922 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8466000000000002e-05 2023-07-02T14:07:02.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.486257+0000", 2023-07-02T14:07:02.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024419020999999999 2023-07-02T14:07:02.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.923 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:02.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510676+0000", 2023-07-02T14:07:02.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6693000000000001e-05 2023-07-02T14:07:02.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510693+0000", 2023-07-02T14:07:02.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039574109000000003 2023-07-02T14:07:02.924 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.924 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.925 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.925 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19181 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2834, mlcod=177'2827)", 2023-07-02T14:07:02.925 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:07:02.925 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.318250882999997, 2023-07-02T14:07:02.925 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039548481000000003, 2023-07-02T14:07:02.925 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.926 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:02.926 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.926 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.926 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:07:02.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:07:02.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:02.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:07:02.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8320999999999999e-05 2023-07-02T14:07:02.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.928 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471188+0000", 2023-07-02T14:07:02.928 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9669999999999998e-06 2023-07-02T14:07:02.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.928 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471194+0000", 2023-07-02T14:07:02.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8709999999999999e-06 2023-07-02T14:07:02.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.929 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.929 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471198+0000", 2023-07-02T14:07:02.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016760634999999999 2023-07-02T14:07:02.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.487959+0000", 2023-07-02T14:07:02.930 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8398000000000002e-05 2023-07-02T14:07:02.930 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.930 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.930 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.487987+0000", 2023-07-02T14:07:02.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022707627000000001 2023-07-02T14:07:02.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:02.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510695+0000", 2023-07-02T14:07:02.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3991999999999999e-05 2023-07-02T14:07:02.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510709+0000", 2023-07-02T14:07:02.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039548481000000003 2023-07-02T14:07:02.932 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.932 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.932 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.933 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19192 2.7 e177/141 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head v 177'2835, mlcod=177'2827)", 2023-07-02T14:07:02.933 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:07:02.933 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 55.318207911000002, 2023-07-02T14:07:02.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039521186999999999, 2023-07-02T14:07:02.933 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.933 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:02.933 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:07:02.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:07:02.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:02.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:07:02.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014525079 2023-07-02T14:07:02.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485728+0000", 2023-07-02T14:07:02.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8372e-05 2023-07-02T14:07:02.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485746+0000", 2023-07-02T14:07:02.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6149999999999997e-06 2023-07-02T14:07:02.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485755+0000", 2023-07-02T14:07:02.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036949270000000002 2023-07-02T14:07:02.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489450+0000", 2023-07-02T14:07:02.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9415e-05 2023-07-02T14:07:02.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.938 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.938 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489469+0000", 2023-07-02T14:07:02.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021241488999999999 2023-07-02T14:07:02.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:02.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510711+0000", 2023-07-02T14:07:02.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3604999999999999e-05 2023-07-02T14:07:02.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510724+0000", 2023-07-02T14:07:02.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039521186999999999 2023-07-02T14:07:02.940 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.940 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.940 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.940 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:20659 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2134016~94208] snapc 35=[35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e179)", 2023-07-02T14:07:02.941 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:07:02.941 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 51.199024528999999, 2023-07-02T14:07:02.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041695300999999997, 2023-07-02T14:07:02.941 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.941 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:02.941 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.941 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:02.941 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:02.942 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20659 2023-07-02T14:07:02.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.942 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:07:02.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:07:02.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:02.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590386+0000", 2023-07-02T14:07:02.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1415999999999994e-05 2023-07-02T14:07:02.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590467+0000", 2023-07-02T14:07:02.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8029999999999999e-06 2023-07-02T14:07:02.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590474+0000", 2023-07-02T14:07:02.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5379999999999998e-06 2023-07-02T14:07:02.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590478+0000", 2023-07-02T14:07:02.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073612449999999998 2023-07-02T14:07:02.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.597839+0000", 2023-07-02T14:07:02.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013387800000000001 2023-07-02T14:07:02.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.597973+0000", 2023-07-02T14:07:02.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022472799999999999 2023-07-02T14:07:02.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.947 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:02.947 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.598197+0000", 2023-07-02T14:07:02.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065294259999999996 2023-07-02T14:07:02.947 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:02.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.604727+0000", 2023-07-02T14:07:02.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027239217999999999 2023-07-02T14:07:02.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:02.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.631966+0000", 2023-07-02T14:07:02.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3194999999999994e-05 2023-07-02T14:07:02.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:02.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632029+0000", 2023-07-02T14:07:02.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2639999999999997e-05 2023-07-02T14:07:02.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632082+0000", 2023-07-02T14:07:02.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041695300999999997 2023-07-02T14:07:02.950 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.950 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.950 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.951 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:20660 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2031616~196608] snapc 35=[35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e179)", 2023-07-02T14:07:02.951 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:07:02.951 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 51.198929509000003, 2023-07-02T14:07:02.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041610001000000001, 2023-07-02T14:07:02.951 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.951 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:02.951 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.952 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:02.952 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:02.952 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20660 2023-07-02T14:07:02.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.952 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:07:02.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:07:02.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:02.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590481+0000", 2023-07-02T14:07:02.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012730599999999999 2023-07-02T14:07:02.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590608+0000", 2023-07-02T14:07:02.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6319999999999997e-06 2023-07-02T14:07:02.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.955 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590617+0000", 2023-07-02T14:07:02.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5209999999999998e-06 2023-07-02T14:07:02.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.955 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590621+0000", 2023-07-02T14:07:02.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018367430000000001 2023-07-02T14:07:02.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592457+0000", 2023-07-02T14:07:02.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016640599999999999 2023-07-02T14:07:02.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592624+0000", 2023-07-02T14:07:02.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028927400000000001 2023-07-02T14:07:02.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.957 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:02.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592913+0000", 2023-07-02T14:07:02.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011797813000000001 2023-07-02T14:07:02.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:02.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.604711+0000", 2023-07-02T14:07:02.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027285997999999999 2023-07-02T14:07:02.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:02.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.631997+0000", 2023-07-02T14:07:02.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8890000000000001e-05 2023-07-02T14:07:02.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:02.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632046+0000", 2023-07-02T14:07:02.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5867e-05 2023-07-02T14:07:02.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.960 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632092+0000", 2023-07-02T14:07:02.960 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041610001000000001 2023-07-02T14:07:02.960 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.960 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.961 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.961 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:23560 2.3 2:c69ae922:::rbd_data.125de1449605.00000000000001da:head [write 65536~4128768] snapc 42=[42,41,40,3f,3e,3d,3c,3b,3a,39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e206)", 2023-07-02T14:07:02.961 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:57.307513+0000", 2023-07-02T14:07:02.961 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 5.481898138, 2023-07-02T14:07:02.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.049163971000000001, 2023-07-02T14:07:02.961 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:02.962 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:02.962 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:02.962 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:02.962 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:02.962 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23560 2023-07-02T14:07:02.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.962 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:02.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:02.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.307513+0000", 2023-07-02T14:07:02.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:02.963 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:02.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.307513+0000", 2023-07-02T14:07:02.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:07:02.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:02.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.307511+0000", 2023-07-02T14:07:02.964 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010796256000000001 2023-07-02T14:07:02.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318307+0000", 2023-07-02T14:07:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2152e-05 2023-07-02T14:07:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318319+0000", 2023-07-02T14:07:02.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8710000000000002e-06 2023-07-02T14:07:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318325+0000", 2023-07-02T14:07:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00065064300000000001 2023-07-02T14:07:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.966 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318976+0000", 2023-07-02T14:07:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4927999999999996e-05 2023-07-02T14:07:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.319071+0000", 2023-07-02T14:07:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000184487 2023-07-02T14:07:02.967 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.319255+0000", 2023-07-02T14:07:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021455622000000001 2023-07-02T14:07:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:02.968 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.340711+0000", 2023-07-02T14:07:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015850748000000001 2023-07-02T14:07:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.356562+0000", 2023-07-02T14:07:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1957000000000002e-05 2023-07-02T14:07:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.356634+0000", 2023-07-02T14:07:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3409999999999999e-05 2023-07-02T14:07:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.356677+0000", 2023-07-02T14:07:02.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.049163971000000001 2023-07-02T14:07:02.971 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.971 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.971 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.971 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:02.971 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:02.971 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:02.972 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:07:02.973 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:02.972+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:03.073 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:03.072+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:03.111 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:03.112 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:07:03.112 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:07:03.112 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:03.123 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:07:03.174 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:03.174+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:03.247 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:03.247 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:07:03.248 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:07:03.248 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:07:03.248 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:03.258 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:07:03.297 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:07:03.299 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:03.299+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:03.368 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:07:03.376 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:03.376+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:03.424 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:03.424 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:07:03.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:07:03.424 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:07:03.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.425 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:07:03.425 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:07:03.425 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 203.525208929, 2023-07-02T14:07:03.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:07:03.425 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:03.425 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:03.426 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:03.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:03.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:07:03.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:03.426 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:03.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:07:03.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:03.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:03.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:07:03.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:07:03.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:03.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:07:03.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:07:03.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:03.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:07:03.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:07:03.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:03.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:07:03.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:07:03.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:03.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:07:03.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:07:03.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:03.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:07:03.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:07:03.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:03.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:07:03.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:07:03.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:03.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:07:03.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:07:03.432 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.432 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.432 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.433 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:07:03.433 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:07:03.433 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 203.521123211, 2023-07-02T14:07:03.433 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:07:03.433 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:03.433 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:03.433 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:03.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:03.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:07:03.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:03.434 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:03.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:07:03.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:03.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:03.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:07:03.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:07:03.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:03.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:07:03.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:07:03.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:03.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:07:03.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:07:03.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:03.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:07:03.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:07:03.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:03.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:07:03.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:07:03.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:03.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:07:03.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:07:03.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:03.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:07:03.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:07:03.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:03.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:07:03.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:07:03.440 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.440 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.440 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.440 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:07:03.441 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:07:03.441 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 203.51725843, 2023-07-02T14:07:03.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:07:03.441 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:03.441 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:03.441 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:03.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:03.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:07:03.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:03.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.442 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:03.442 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:07:03.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:03.442 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:03.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:07:03.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:07:03.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:03.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:07:03.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:07:03.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:03.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:07:03.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:07:03.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:03.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:07:03.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:07:03.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:03.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:07:03.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:07:03.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:03.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:07:03.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:07:03.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:03.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:07:03.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:07:03.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:03.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:07:03.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:07:03.447 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.448 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.448 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.448 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:07:03.448 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:07:03.448 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 203.47802937, 2023-07-02T14:07:03.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:07:03.449 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:03.449 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:03.449 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:03.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:03.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:07:03.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:03.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:03.450 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:07:03.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:03.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.450 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:03.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:07:03.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:07:03.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.451 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:03.451 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:07:03.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:07:03.451 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:03.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:07:03.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:07:03.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:03.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:07:03.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:07:03.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:03.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:07:03.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:07:03.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:03.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:07:03.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:07:03.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:03.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:07:03.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:07:03.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:03.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:07:03.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:07:03.455 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.455 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.456 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.456 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:07:03.456 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:07:03.456 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 203.473372012, 2023-07-02T14:07:03.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:07:03.456 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:03.457 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:03.457 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:03.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:03.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:07:03.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:03.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:03.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:07:03.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:03.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:03.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:07:03.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:07:03.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:03.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:07:03.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:07:03.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:03.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:07:03.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:07:03.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:03.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:07:03.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:07:03.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:03.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:07:03.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:07:03.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:03.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:07:03.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:07:03.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:03.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:07:03.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:07:03.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:03.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:07:03.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:07:03.463 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.463 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.463 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.464 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:07:03.464 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:07:03.464 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 203.47243214900001, 2023-07-02T14:07:03.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:07:03.464 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:03.464 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:03.464 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:03.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:03.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:07:03.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:03.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:03.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:07:03.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:03.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:03.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:07:03.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:07:03.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:03.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:07:03.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:07:03.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:03.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:07:03.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:07:03.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:03.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:07:03.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:07:03.468 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:03.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:07:03.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:07:03.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.469 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:03.469 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:07:03.469 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:07:03.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:03.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:07:03.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:07:03.470 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.470 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.470 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:03.470 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:07:03.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:07:03.471 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.471 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.471 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.471 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:07:03.471 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:07:03.472 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 203.47103427100001, 2023-07-02T14:07:03.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:07:03.472 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:03.472 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:03.472 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:03.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:03.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:07:03.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:03.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:03.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:07:03.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:03.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:03.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:07:03.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:07:03.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:03.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:07:03.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:07:03.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:03.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:07:03.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:07:03.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:03.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:07:03.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:07:03.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:03.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:07:03.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:07:03.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:03.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:07:03.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:07:03.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:03.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:07:03.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:07:03.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:03.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:07:03.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:07:03.478 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.479 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.479 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.479 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:07:03.479 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:07:03.479 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 182.060843775, 2023-07-02T14:07:03.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:07:03.480 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:03.480 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:03.480 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:03.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:03.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:07:03.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:03.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:03.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:07:03.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:03.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:03.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:07:03.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:07:03.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:03.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:07:03.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:07:03.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:03.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:07:03.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:07:03.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:03.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:07:03.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:07:03.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:03.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:07:03.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:07:03.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:03.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:07:03.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:07:03.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:03.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:07:03.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:07:03.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:03.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:07:03.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:07:03.486 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.486 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.486 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.487 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:07:03.487 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:07:03.487 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.694494091, 2023-07-02T14:07:03.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:07:03.487 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:03.487 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:03.488 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:03.488 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:03.488 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:03.488 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:07:03.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.488 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:03.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:03.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:07:03.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:03.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:03.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:07:03.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:03.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:03.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:07:03.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:07:03.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:03.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:07:03.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:07:03.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:03.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:07:03.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:07:03.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:03.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:07:03.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:07:03.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:03.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:07:03.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:07:03.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:03.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:07:03.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:07:03.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:03.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:07:03.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:07:03.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:03.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:07:03.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:07:03.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:03.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:07:03.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:07:03.495 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.495 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.495 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.496 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:07:03.496 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:07:03.496 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.69044674700001, 2023-07-02T14:07:03.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:07:03.496 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:03.496 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:03.497 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:03.497 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:03.497 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:03.497 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:07:03.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.497 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:03.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:03.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:07:03.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:03.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:03.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:07:03.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:03.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:03.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:07:03.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:07:03.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:03.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:07:03.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:07:03.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:03.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:07:03.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:07:03.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:03.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:07:03.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:07:03.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:03.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:07:03.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:07:03.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:03.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:07:03.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:07:03.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:03.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:07:03.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:07:03.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:03.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:07:03.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:07:03.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:03.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:07:03.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:07:03.504 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.504 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.504 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.505 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:07:03.505 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:07:03.505 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.68588292199999, 2023-07-02T14:07:03.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:07:03.505 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:03.506 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:03.506 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:03.506 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:03.506 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:03.506 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:07:03.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.506 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:03.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:03.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:07:03.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:03.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:03.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:07:03.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:03.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:03.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:07:03.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:07:03.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:03.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:07:03.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:07:03.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:03.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:07:03.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:07:03.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:03.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:07:03.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:07:03.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:03.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:07:03.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:07:03.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:03.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:07:03.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:07:03.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:03.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:07:03.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:07:03.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:03.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:07:03.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:07:03.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:03.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:07:03.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:07:03.513 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.513 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.513 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.514 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:07:03.514 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:07:03.514 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.68583305600001, 2023-07-02T14:07:03.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:07:03.514 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:03.515 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:03.515 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:03.515 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:03.515 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:03.515 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:07:03.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.515 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:03.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:03.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:07:03.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:03.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:03.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:07:03.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:03.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:03.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:07:03.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:07:03.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:03.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:07:03.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:07:03.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:03.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:07:03.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:07:03.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:03.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:07:03.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:07:03.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:03.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:07:03.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:07:03.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:03.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:07:03.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:07:03.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:03.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:07:03.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:07:03.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:03.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:07:03.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:07:03.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:03.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:07:03.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:07:03.522 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.522 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.523 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.523 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:07:03.523 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:07:03.523 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 180.684794063, 2023-07-02T14:07:03.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:07:03.523 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:03.524 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:03.524 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:03.524 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:03.524 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:03.524 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:07:03.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.524 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:03.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:03.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:07:03.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:03.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:03.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:07:03.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:03.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:03.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:07:03.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:07:03.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:03.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:07:03.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:07:03.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:03.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:07:03.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:07:03.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:03.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:07:03.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:07:03.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:03.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:07:03.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:07:03.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:03.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:07:03.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:07:03.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:03.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:07:03.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:07:03.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:03.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:07:03.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:07:03.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:03.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:07:03.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:07:03.531 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.531 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.532 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.532 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22312 2.0 e197/193 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 197'5590, mlcod=197'5582)", 2023-07-02T14:07:03.532 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:07:03.532 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 28.674349364000001, 2023-07-02T14:07:03.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021943201999999998, 2023-07-02T14:07:03.532 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:03.533 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:03.533 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:03.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:03.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:07:03.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:03.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:03.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:07:03.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:03.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:03.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748473+0000", 2023-07-02T14:07:03.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00099631099999999994 2023-07-02T14:07:03.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:03.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749469+0000", 2023-07-02T14:07:03.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4279e-05 2023-07-02T14:07:03.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:03.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749483+0000", 2023-07-02T14:07:03.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5289999999999997e-06 2023-07-02T14:07:03.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:03.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749490+0000", 2023-07-02T14:07:03.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047909650000000003 2023-07-02T14:07:03.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:03.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.754281+0000", 2023-07-02T14:07:03.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1999000000000001e-05 2023-07-02T14:07:03.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:03.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.754303+0000", 2023-07-02T14:07:03.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016063747 2023-07-02T14:07:03.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:03.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.770367+0000", 2023-07-02T14:07:03.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0231000000000001e-05 2023-07-02T14:07:03.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:03.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.770417+0000", 2023-07-02T14:07:03.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021943201999999998 2023-07-02T14:07:03.539 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.539 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.539 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.540 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22313 2.0 e197/193 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 197'5592, mlcod=197'5587)", 2023-07-02T14:07:03.540 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:07:03.540 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 28.669095961, 2023-07-02T14:07:03.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026016164000000001, 2023-07-02T14:07:03.540 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:03.540 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:03.540 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:03.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:03.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:07:03.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:03.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:03.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:07:03.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999933 2023-07-02T14:07:03.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:03.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753720+0000", 2023-07-02T14:07:03.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4851999999999995e-05 2023-07-02T14:07:03.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:03.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753785+0000", 2023-07-02T14:07:03.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0216e-05 2023-07-02T14:07:03.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:03.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753796+0000", 2023-07-02T14:07:03.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4249999999999997e-06 2023-07-02T14:07:03.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:03.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753801+0000", 2023-07-02T14:07:03.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070274359999999998 2023-07-02T14:07:03.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:03.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760828+0000", 2023-07-02T14:07:03.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0975000000000002e-05 2023-07-02T14:07:03.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:03.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760849+0000", 2023-07-02T14:07:03.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018855916 2023-07-02T14:07:03.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:03.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.779705+0000", 2023-07-02T14:07:03.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8037999999999997e-05 2023-07-02T14:07:03.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:03.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.779743+0000", 2023-07-02T14:07:03.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026016164000000001 2023-07-02T14:07:03.547 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.547 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.547 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.547 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22314 2.0 e197/193 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 197'5594, mlcod=197'5587)", 2023-07-02T14:07:03.548 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:07:03.548 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 28.666491307000001, 2023-07-02T14:07:03.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027004523999999999, 2023-07-02T14:07:03.548 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:03.548 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:03.548 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:03.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:03.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:07:03.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:03.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:03.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:07:03.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:03.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:03.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756331+0000", 2023-07-02T14:07:03.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4697999999999999e-05 2023-07-02T14:07:03.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:03.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756365+0000", 2023-07-02T14:07:03.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3902e-05 2023-07-02T14:07:03.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:03.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756379+0000", 2023-07-02T14:07:03.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7329999999999997e-06 2023-07-02T14:07:03.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:03.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756386+0000", 2023-07-02T14:07:03.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062861929999999998 2023-07-02T14:07:03.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:03.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.762672+0000", 2023-07-02T14:07:03.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9031e-05 2023-07-02T14:07:03.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:03.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.762711+0000", 2023-07-02T14:07:03.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020589275000000001 2023-07-02T14:07:03.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:03.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783301+0000", 2023-07-02T14:07:03.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5720999999999999e-05 2023-07-02T14:07:03.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:03.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783336+0000", 2023-07-02T14:07:03.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027004523999999999 2023-07-02T14:07:03.555 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.555 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.555 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.555 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22315 2.0 e197/193 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 197'5595, mlcod=197'5587)", 2023-07-02T14:07:03.555 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:07:03.555 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 28.662522035999999, 2023-07-02T14:07:03.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023064345999999999, 2023-07-02T14:07:03.556 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:03.556 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:03.556 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:03.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:03.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:07:03.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:03.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:03.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:07:03.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:03.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:03.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760300+0000", 2023-07-02T14:07:03.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3056000000000002e-05 2023-07-02T14:07:03.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:03.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760353+0000", 2023-07-02T14:07:03.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2825e-05 2023-07-02T14:07:03.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:03.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760366+0000", 2023-07-02T14:07:03.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8290000000000002e-06 2023-07-02T14:07:03.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:03.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760373+0000", 2023-07-02T14:07:03.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075982089999999999 2023-07-02T14:07:03.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:03.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.767971+0000", 2023-07-02T14:07:03.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4096000000000001e-05 2023-07-02T14:07:03.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:03.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.767995+0000", 2023-07-02T14:07:03.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015345908 2023-07-02T14:07:03.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:03.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783341+0000", 2023-07-02T14:07:03.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4273e-05 2023-07-02T14:07:03.562 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.562 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:03.562 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783365+0000", 2023-07-02T14:07:03.562 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023064345999999999 2023-07-02T14:07:03.562 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.562 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.563 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.563 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.563 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.563 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22319 2.0 e197/193 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 197'5597, mlcod=197'5587)", 2023-07-02T14:07:03.563 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:07:03.563 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 28.659019216000001, 2023-07-02T14:07:03.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022862501, 2023-07-02T14:07:03.564 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:03.564 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:03.564 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:03.564 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.564 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:03.564 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:07:03.564 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:03.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:03.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:07:03.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:03.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:03.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763803+0000", 2023-07-02T14:07:03.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052067220000000004 2023-07-02T14:07:03.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:03.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769010+0000", 2023-07-02T14:07:03.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3249e-05 2023-07-02T14:07:03.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:03.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769023+0000", 2023-07-02T14:07:03.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1790000000000008e-06 2023-07-02T14:07:03.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:03.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769031+0000", 2023-07-02T14:07:03.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030611079999999999 2023-07-02T14:07:03.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:03.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.772092+0000", 2023-07-02T14:07:03.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3958e-05 2023-07-02T14:07:03.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:03.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.772116+0000", 2023-07-02T14:07:03.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014504064000000001 2023-07-02T14:07:03.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:03.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.786620+0000", 2023-07-02T14:07:03.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5952999999999999e-05 2023-07-02T14:07:03.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:03.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.786666+0000", 2023-07-02T14:07:03.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022862501 2023-07-02T14:07:03.570 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.570 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.570 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.571 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22335 2.0 e197/193 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 197'5602, mlcod=197'5599)", 2023-07-02T14:07:03.571 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:07:03.571 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 28.631948939000001, 2023-07-02T14:07:03.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024437743000000001, 2023-07-02T14:07:03.571 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:03.571 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:03.572 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:03.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:03.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:07:03.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:03.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:03.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:07:03.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:03.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.573 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:03.573 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790873+0000", 2023-07-02T14:07:03.573 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081672399999999992 2023-07-02T14:07:03.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:03.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799040+0000", 2023-07-02T14:07:03.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5712999999999999e-05 2023-07-02T14:07:03.574 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.574 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.574 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:03.574 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799066+0000", 2023-07-02T14:07:03.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1413999999999999e-05 2023-07-02T14:07:03.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:03.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799077+0000", 2023-07-02T14:07:03.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3102000000000007e-05 2023-07-02T14:07:03.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:03.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799140+0000", 2023-07-02T14:07:03.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.883e-05 2023-07-02T14:07:03.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.576 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:03.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799169+0000", 2023-07-02T14:07:03.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016086700999999998 2023-07-02T14:07:03.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:03.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.815256+0000", 2023-07-02T14:07:03.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5967000000000001e-05 2023-07-02T14:07:03.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:03.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.815312+0000", 2023-07-02T14:07:03.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024437743000000001 2023-07-02T14:07:03.578 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.578 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.578 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.579 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22336 2.0 e197/193 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 197'5603, mlcod=197'5599)", 2023-07-02T14:07:03.579 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:07:03.579 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 28.623713652999999, 2023-07-02T14:07:03.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022713011000000002, 2023-07-02T14:07:03.579 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:03.579 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:03.579 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:03.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:03.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:07:03.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:03.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:03.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:07:03.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:03.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:03.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799108+0000", 2023-07-02T14:07:03.581 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00058943600000000004 2023-07-02T14:07:03.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:03.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799698+0000", 2023-07-02T14:07:03.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6205000000000002e-05 2023-07-02T14:07:03.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:03.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799714+0000", 2023-07-02T14:07:03.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9630000000000001e-06 2023-07-02T14:07:03.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:03.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799721+0000", 2023-07-02T14:07:03.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076846579999999996 2023-07-02T14:07:03.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:03.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.807405+0000", 2023-07-02T14:07:03.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8191000000000002e-05 2023-07-02T14:07:03.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:03.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.807434+0000", 2023-07-02T14:07:03.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014338020999999999 2023-07-02T14:07:03.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:03.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.821772+0000", 2023-07-02T14:07:03.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0868999999999997e-05 2023-07-02T14:07:03.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:03.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:03.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:03.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.821822+0000", 2023-07-02T14:07:03.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022713011000000002 2023-07-02T14:07:03.586 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.586 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.586 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.586 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:03.586 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:03.586 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:03.588 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:07:03.588 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:07:03.588 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:07:03.588 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:07:03.589 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:07:03.589 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:07:03.589 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:07:03.590 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:03.588+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:03.590 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:07:03.590 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:07:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:07:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:07:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:07:03.591 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:07:03.688 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:03.688+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:03.752 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:03.752 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:03.762 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:07:03.804 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:03.804+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:03.890 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:03.889+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:03.926 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:03.926 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:03.936 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:07:03.990 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:07:03.989+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:07:04.054 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:07:04.053+0000 7fb5c354f700 0 -- 172.21.15.163:0/648338979 >> [v2:172.21.15.22:6808/99977,v1:172.21.15.22:6809/99977] conn(0x7fb5a0008780 msgr2=0x7fb5a000abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:07:04.059 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:07:04.058+0000 7fb5c2d4e700 0 -- 172.21.15.163:0/648338979 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7fb5a002a970 msgr2=0x7fb5a002adb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:07:04.088 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:04.088 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:04.091 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:04.090+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:04.100 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:07:04.192 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:04.192+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:04.218 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 87% complete...+ rm -rf b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4 b3fbb028-e267-4d25-a750-93ba18a2316a-mnt/linux-5.4.tar.gz 2023-07-02T14:07:04.251 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:04.251 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:04.263 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:07:04.293 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:04.292+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:04.335 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 100% complete...done. 2023-07-02T14:07:04.338 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:07:04.338 INFO:tasks.workunit.client.0.smithi163.stderr:+ snap_num=65 2023-07-02T14:07:04.338 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd snap create b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap65 2023-07-02T14:07:04.356 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:07:04.410 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:04.409+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:04.447 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:04.447 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:04.460 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:07:04.502 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:04.503+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:04.597 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:04.596+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:04.597 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:04.598 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:04.610 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:07:04.697 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:04.698+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:04.739 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:04.739 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:04.750 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:07:04.798 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:04.798+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:04.884 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:04.884 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:04.895 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:07:04.939 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:04.938+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:05.000 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:05.000+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:05.026 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:05.026 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:05.038 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:07:05.123 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:05.122+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:05.202 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:05.201+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:05.209 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:05.209 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:05.220 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:07:05.303 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:07:05.302+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:07:05.403 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:05.403+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:05.406 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:05.406 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:05.423 INFO:tasks.workunit.client.0.smithi163.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2023-07-02T14:07:05.425 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:07:05.430 INFO:tasks.workunit.client.0.smithi163.stderr:+ sleep 4 2023-07-02T14:07:05.508 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:07:05.507+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:07:05.592 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:07:05.611 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:05.611 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:05.612 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:05.610+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:05.624 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:07:05.640 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i++ )) 2023-07-02T14:07:05.640 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i < 10 )) 2023-07-02T14:07:05.706 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:05.706+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:05.747 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:07:05.766 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:07:05.786 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:07:05.801 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:05.801 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:07:05.801 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:07:05.801 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:05.806 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:05.805+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:05.807 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:07:05.814 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:07:05.827 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:07:05.848 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:07:05.849 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:07:05.849 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:07:05.850 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:07:05.850 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:07:05.850 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:07:05.850 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:07:05.907 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:05.907+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:05.971 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:05.972 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:07:05.972 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:07:05.972 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:07:05.972 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:05.984 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:07:06.027 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:06.026+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:06.108 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:06.107+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:06.129 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:06.130 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:07:06.130 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:07:06.130 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:07:06.130 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.130 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:07:06.130 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:07:06.130 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 245.020039911, 2023-07-02T14:07:06.131 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:07:06.131 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:06.131 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:06.131 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:06.131 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:06.131 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:06.131 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:07:06.131 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.132 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:06.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.132 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:06.132 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:07:06.132 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:06.132 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.132 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:06.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:07:06.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:07:06.133 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.133 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.133 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:06.133 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:07:06.133 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:07:06.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:06.134 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:07:06.134 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:07:06.134 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.134 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.134 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:06.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:07:06.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:07:06.135 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.135 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.135 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.135 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:07:06.135 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:07:06.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.136 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:07:06.136 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:07:06.136 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.136 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.136 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:06.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:07:06.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:07:06.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.137 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.137 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.137 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:07:06.137 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:07:06.137 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:06.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:07:06.138 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:07:06.138 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.138 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.138 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:06.138 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:07:06.139 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:07:06.139 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.139 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.139 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.139 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.139 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.139 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:07:06.140 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:07:06.140 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 245.019979173, 2023-07-02T14:07:06.140 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:07:06.140 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:06.140 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.140 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:06.140 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:06.140 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:06.141 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:07:06.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.141 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:06.141 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.141 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:06.141 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:07:06.141 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:06.141 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:06.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:07:06.142 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:06.142 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.142 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.142 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:06.142 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:07:06.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:07:06.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.143 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:06.143 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:07:06.143 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:07:06.143 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.143 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:06.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:07:06.144 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:07:06.144 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.144 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.144 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.144 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:07:06.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:07:06.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.145 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.145 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:07:06.145 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:07:06.145 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.145 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:06.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:07:06.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:07:06.146 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.146 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.146 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.146 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:07:06.146 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:07:06.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:06.147 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:07:06.147 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:07:06.147 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.147 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.147 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:06.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:07:06.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:07:06.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.148 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.148 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:06.148 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:07:06.148 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:07:06.148 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:06.149 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:07:06.149 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:07:06.149 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.149 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.149 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:06.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:07:06.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:07:06.150 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.150 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.150 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:06.150 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:07:06.150 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:07:06.150 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.151 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.151 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.151 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.151 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.151 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:07:06.151 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:07:06.151 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 234.78952484300001, 2023-07-02T14:07:06.151 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:07:06.152 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:06.152 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.152 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:06.152 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:06.152 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:06.152 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:07:06.152 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.152 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:06.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:06.153 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:07:06.153 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:06.153 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.153 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.153 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:06.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:07:06.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:06.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.154 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.154 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:06.154 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:07:06.154 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:07:06.154 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:06.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:07:06.155 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:07:06.155 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.155 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.155 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:06.155 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:07:06.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:07:06.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.156 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.156 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.156 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:07:06.156 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:07:06.156 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:07:06.157 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:07:06.157 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.157 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.157 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:06.157 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:07:06.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:07:06.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.158 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:06.158 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:07:06.158 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:07:06.158 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.158 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:06.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:07:06.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:07:06.159 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.159 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.159 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:06.159 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:07:06.159 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:07:06.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:06.160 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:07:06.160 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:07:06.160 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.160 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.160 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:06.161 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:07:06.161 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:07:06.161 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.161 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.161 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.161 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.161 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.162 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9620 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 589824~65536] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:07:06.162 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:07:06.162 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 161.105510245, 2023-07-02T14:07:06.162 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001, 2023-07-02T14:07:06.162 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:06.162 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.162 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:06.162 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:06.163 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:06.163 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9620 2023-07-02T14:07:06.163 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.163 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:06.163 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.163 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:06.163 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:07:06.163 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:06.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:06.164 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:07:06.164 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:07:06.164 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.164 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.164 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:06.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022203+0000", 2023-07-02T14:07:06.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0346000000000001e-05 2023-07-02T14:07:06.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.165 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.165 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:06.165 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022263+0000", 2023-07-02T14:07:06.165 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4657999999999999e-05 2023-07-02T14:07:06.165 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:06.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022278+0000", 2023-07-02T14:07:06.166 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:07:06.166 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.166 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.166 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.166 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022285+0000", 2023-07-02T14:07:06.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0096000000000003e-05 2023-07-02T14:07:06.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.167 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.167 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.167 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022335+0000", 2023-07-02T14:07:06.167 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2334000000000001e-05 2023-07-02T14:07:06.167 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:06.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022347+0000", 2023-07-02T14:07:06.168 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095653701999999993 2023-07-02T14:07:06.168 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.168 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.168 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.168 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118001+0000", 2023-07-02T14:07:06.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000185797 2023-07-02T14:07:06.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.169 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:06.169 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118187+0000", 2023-07-02T14:07:06.169 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026397900000000001 2023-07-02T14:07:06.169 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.169 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:06.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118451+0000", 2023-07-02T14:07:06.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051832800000000002 2023-07-02T14:07:06.170 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.170 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.170 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:06.170 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.123634+0000", 2023-07-02T14:07:06.170 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001227332 2023-07-02T14:07:06.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:06.171 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124861+0000", 2023-07-02T14:07:06.171 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9493999999999998e-05 2023-07-02T14:07:06.171 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.171 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.171 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:06.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124901+0000", 2023-07-02T14:07:06.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0462000000000001e-05 2023-07-02T14:07:06.172 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.172 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.172 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:06.172 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124931+0000", 2023-07-02T14:07:06.172 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001 2023-07-02T14:07:06.173 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.173 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.173 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.173 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.173 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.173 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9621 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 655360~1966080] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:07:06.173 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:07:06.173 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 161.105322863, 2023-07-02T14:07:06.174 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999, 2023-07-02T14:07:06.174 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:06.174 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.174 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:06.174 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:06.174 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:06.174 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9621 2023-07-02T14:07:06.174 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.175 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:06.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:06.175 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:07:06.175 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:06.175 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.175 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.175 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:06.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:07:06.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:06.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.176 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.176 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:06.176 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022391+0000", 2023-07-02T14:07:06.176 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032053939999999999 2023-07-02T14:07:06.176 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:06.177 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025597+0000", 2023-07-02T14:07:06.177 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4307999999999999e-05 2023-07-02T14:07:06.177 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.177 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.177 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:06.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025611+0000", 2023-07-02T14:07:06.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.032e-06 2023-07-02T14:07:06.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.178 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.178 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.178 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025617+0000", 2023-07-02T14:07:06.178 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8977999999999998e-05 2023-07-02T14:07:06.178 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025656+0000", 2023-07-02T14:07:06.179 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:07:06.179 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.179 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.179 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:06.179 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025663+0000", 2023-07-02T14:07:06.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095220739999999998 2023-07-02T14:07:06.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.180 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.180 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.180 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120884+0000", 2023-07-02T14:07:06.180 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2525000000000001e-05 2023-07-02T14:07:06.180 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:06.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120956+0000", 2023-07-02T14:07:06.181 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016232400000000001 2023-07-02T14:07:06.181 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.181 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.181 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:06.181 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.121119+0000", 2023-07-02T14:07:06.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044007380000000004 2023-07-02T14:07:06.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.182 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:06.182 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.125519+0000", 2023-07-02T14:07:06.182 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000797209 2023-07-02T14:07:06.182 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.182 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:06.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126317+0000", 2023-07-02T14:07:06.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1228000000000004e-05 2023-07-02T14:07:06.183 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.183 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.183 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:06.183 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126378+0000", 2023-07-02T14:07:06.183 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978000000000003e-05 2023-07-02T14:07:06.184 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.184 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.184 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:06.184 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126430+0000", 2023-07-02T14:07:06.184 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999 2023-07-02T14:07:06.184 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.184 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.184 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.185 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.185 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.185 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:281 2.1 2:901d5a95:::rbd_data.125de1449605.0000000000000118:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:07:06.185 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:07:06.185 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 152.57224675200001, 2023-07-02T14:07:06.185 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002, 2023-07-02T14:07:06.185 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:06.186 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:06.186 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:06.186 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:07:06.186 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:07:06.186 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:07:06.186 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.186 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:06.186 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:06.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:07:06.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:06.187 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.187 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.187 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:06.187 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:07:06.187 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:06.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:06.188 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:07:06.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0469999999999998e-06 2023-07-02T14:07:06.188 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.188 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:06.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555471+0000", 2023-07-02T14:07:06.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8860000000000001e-06 2023-07-02T14:07:06.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.189 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.189 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:06.189 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555476+0000", 2023-07-02T14:07:06.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3160000000000002e-06 2023-07-02T14:07:06.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555483+0000", 2023-07-02T14:07:06.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989525000000001 2023-07-02T14:07:06.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591472+0000", 2023-07-02T14:07:06.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3708999999999996e-05 2023-07-02T14:07:06.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:06.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591536+0000", 2023-07-02T14:07:06.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055109720000000003 2023-07-02T14:07:06.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:06.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.597047+0000", 2023-07-02T14:07:06.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002 2023-07-02T14:07:06.192 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.192 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.192 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.193 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12985 2.7 2:eeddf24b:::rbd_data.125de1449605.000000000000007e:head [write 65536~4128768] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:07:06.193 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:07:06.193 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 133.344719806, 2023-07-02T14:07:06.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999, 2023-07-02T14:07:06.193 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:06.193 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.193 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:06.194 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:06.194 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:06.194 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12985 2023-07-02T14:07:06.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.194 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:06.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:06.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:07:06.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:06.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:06.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:07:06.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:06.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:06.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782994+0000", 2023-07-02T14:07:06.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019254741999999998 2023-07-02T14:07:06.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:06.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802248+0000", 2023-07-02T14:07:06.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0451e-05 2023-07-02T14:07:06.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:06.197 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802279+0000", 2023-07-02T14:07:06.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3318e-05 2023-07-02T14:07:06.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.197 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802292+0000", 2023-07-02T14:07:06.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109403 2023-07-02T14:07:06.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.198 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.198 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.198 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802402+0000", 2023-07-02T14:07:06.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012520599999999999 2023-07-02T14:07:06.198 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:06.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802527+0000", 2023-07-02T14:07:06.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187768 2023-07-02T14:07:06.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:06.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802715+0000", 2023-07-02T14:07:06.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013192822 2023-07-02T14:07:06.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:06.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.815907+0000", 2023-07-02T14:07:06.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0086451870000000004 2023-07-02T14:07:06.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:06.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824553+0000", 2023-07-02T14:07:06.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6759999999999999e-05 2023-07-02T14:07:06.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:06.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824609+0000", 2023-07-02T14:07:06.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011314700000000001 2023-07-02T14:07:06.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:06.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824722+0000", 2023-07-02T14:07:06.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999 2023-07-02T14:07:06.202 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.202 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.203 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.203 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:165 2.3 2:c9c6393a:::rbd_data.125de1449605.000000000000002c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:07:06.203 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:07:06.203 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 129.287936989, 2023-07-02T14:07:06.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002, 2023-07-02T14:07:06.203 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:06.204 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:06.204 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:06.204 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:07:06.204 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:07:06.204 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 165 2023-07-02T14:07:06.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.204 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:06.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:06.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:07:06.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:06.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:06.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:07:06.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:06.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:06.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839777+0000", 2023-07-02T14:07:06.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6740000000000001e-06 2023-07-02T14:07:06.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:06.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839780+0000", 2023-07-02T14:07:06.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9030000000000004e-06 2023-07-02T14:07:06.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:06.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839788+0000", 2023-07-02T14:07:06.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1799999999999998e-06 2023-07-02T14:07:06.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839792+0000", 2023-07-02T14:07:06.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.031592227000000001 2023-07-02T14:07:06.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871384+0000", 2023-07-02T14:07:06.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013600099999999999 2023-07-02T14:07:06.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.209 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:06.209 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871520+0000", 2023-07-02T14:07:06.209 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082817089999999999 2023-07-02T14:07:06.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.210 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.210 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:06.210 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.879802+0000", 2023-07-02T14:07:06.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002 2023-07-02T14:07:06.210 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.210 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.210 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.211 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:252 2.7 2:f1d0abf0:::rbd_data.125de1449605.0000000000000083:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:07:06.211 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:07:06.211 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 128.87165201900001, 2023-07-02T14:07:06.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997, 2023-07-02T14:07:06.211 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:06.211 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:06.212 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:06.212 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:07:06.212 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:07:06.212 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 252 2023-07-02T14:07:06.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.212 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:06.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:06.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:07:06.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:06.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:06.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:07:06.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:06.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:06.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256062+0000", 2023-07-02T14:07:06.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0149999999999999e-06 2023-07-02T14:07:06.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:06.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256065+0000", 2023-07-02T14:07:06.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5269999999999998e-06 2023-07-02T14:07:06.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:06.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256071+0000", 2023-07-02T14:07:06.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:07:06.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256074+0000", 2023-07-02T14:07:06.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028852909 2023-07-02T14:07:06.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.284927+0000", 2023-07-02T14:07:06.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168274 2023-07-02T14:07:06.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:06.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.285096+0000", 2023-07-02T14:07:06.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011036300000000001 2023-07-02T14:07:06.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:06.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296132+0000", 2023-07-02T14:07:06.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997 2023-07-02T14:07:06.218 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.218 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.218 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.218 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:261 2.1 2:971731cb:::rbd_data.125de1449605.000000000000008c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:07:06.219 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:07:06.219 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 128.87155654399999, 2023-07-02T14:07:06.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984, 2023-07-02T14:07:06.219 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:06.219 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:06.219 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:06.219 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:07:06.220 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:07:06.220 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 261 2023-07-02T14:07:06.220 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.220 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:06.220 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.220 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:06.220 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:07:06.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:06.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:06.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:07:06.221 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:06.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:06.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:07:06.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2819999999999999e-06 2023-07-02T14:07:06.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:06.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256160+0000", 2023-07-02T14:07:06.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2020000000000003e-06 2023-07-02T14:07:06.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:06.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256165+0000", 2023-07-02T14:07:06.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2100000000000003e-06 2023-07-02T14:07:06.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256170+0000", 2023-07-02T14:07:06.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025871715999999999 2023-07-02T14:07:06.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282041+0000", 2023-07-02T14:07:06.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132364 2023-07-02T14:07:06.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:06.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282174+0000", 2023-07-02T14:07:06.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014044566 2023-07-02T14:07:06.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:06.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296218+0000", 2023-07-02T14:07:06.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984 2023-07-02T14:07:06.226 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.226 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.226 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.226 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:264 2.3 2:dbf06c9d:::rbd_data.125de1449605.000000000000008f:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:07:06.226 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:07:06.227 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 128.87152242600001, 2023-07-02T14:07:06.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003, 2023-07-02T14:07:06.227 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:06.227 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:06.227 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:06.227 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:07:06.227 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:07:06.227 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 264 2023-07-02T14:07:06.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.228 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:06.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:06.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:07:06.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:06.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:06.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:07:06.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:06.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:06.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:07:06.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5749999999999999e-06 2023-07-02T14:07:06.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:06.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256195+0000", 2023-07-02T14:07:06.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.327e-06 2023-07-02T14:07:06.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:06.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256200+0000", 2023-07-02T14:07:06.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0580000000000002e-06 2023-07-02T14:07:06.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.231 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.231 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256203+0000", 2023-07-02T14:07:06.231 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035707524999999997 2023-07-02T14:07:06.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.232 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.291911+0000", 2023-07-02T14:07:06.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014269799999999999 2023-07-02T14:07:06.232 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.232 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.232 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:06.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.292053+0000", 2023-07-02T14:07:06.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062760309999999996 2023-07-02T14:07:06.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:06.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.298329+0000", 2023-07-02T14:07:06.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003 2023-07-02T14:07:06.233 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.234 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.234 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.234 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:341 2.3 2:cee258bd:::rbd_data.125de1449605.00000000000000dc:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:07:06.234 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:07:06.234 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 128.49103509899999, 2023-07-02T14:07:06.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997, 2023-07-02T14:07:06.235 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:06.235 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:06.235 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:06.235 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:07:06.235 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:07:06.235 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 341 2023-07-02T14:07:06.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.236 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:06.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:06.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:07:06.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:06.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:06.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:07:06.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:06.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:06.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636679+0000", 2023-07-02T14:07:06.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6240000000000002e-06 2023-07-02T14:07:06.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:06.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636682+0000", 2023-07-02T14:07:06.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6910000000000002e-06 2023-07-02T14:07:06.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:06.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636688+0000", 2023-07-02T14:07:06.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3469999999999999e-06 2023-07-02T14:07:06.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636691+0000", 2023-07-02T14:07:06.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035625560000000001 2023-07-02T14:07:06.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672317+0000", 2023-07-02T14:07:06.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108391 2023-07-02T14:07:06.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:06.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672425+0000", 2023-07-02T14:07:06.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012104976 2023-07-02T14:07:06.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:06.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.684530+0000", 2023-07-02T14:07:06.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997 2023-07-02T14:07:06.241 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.241 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.242 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.242 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPushReply(2.7 139/134 [PushReplyOp(2:e3782f39:::rbd_data.125de1449605.0000000000000120:head)])", 2023-07-02T14:07:06.242 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:07:06.242 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 114.200502946, 2023-07-02T14:07:06.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998, 2023-07-02T14:07:06.242 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:06.243 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:07:06.243 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:06.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.243 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:06.243 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:07:06.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:06.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:06.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:07:06.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:06.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.244 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.244 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:06.244 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927211+0000", 2023-07-02T14:07:06.244 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3689999999999998e-06 2023-07-02T14:07:06.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:06.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927217+0000", 2023-07-02T14:07:06.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7269999999999996e-06 2023-07-02T14:07:06.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:06.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927225+0000", 2023-07-02T14:07:06.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:07:06.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927230+0000", 2023-07-02T14:07:06.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:07:06.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927266+0000", 2023-07-02T14:07:06.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.069972871000000006 2023-07-02T14:07:06.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:06.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.997239+0000", 2023-07-02T14:07:06.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998 2023-07-02T14:07:06.248 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.248 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.248 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.248 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19179 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2832, mlcod=177'2827)", 2023-07-02T14:07:06.249 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:07:06.249 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 58.668973115999997, 2023-07-02T14:07:06.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051928864999999998, 2023-07-02T14:07:06.249 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:06.249 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.249 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:06.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:06.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:07:06.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:06.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:06.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:07:06.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:06.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:06.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458741+0000", 2023-07-02T14:07:06.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01233852 2023-07-02T14:07:06.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.251 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:06.251 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471079+0000", 2023-07-02T14:07:06.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8131e-05 2023-07-02T14:07:06.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:06.252 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471097+0000", 2023-07-02T14:07:06.252 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.19e-06 2023-07-02T14:07:06.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471104+0000", 2023-07-02T14:07:06.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4463999999999999e-05 2023-07-02T14:07:06.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471158+0000", 2023-07-02T14:07:06.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1723000000000001e-05 2023-07-02T14:07:06.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:06.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471190+0000", 2023-07-02T14:07:06.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039438622999999999 2023-07-02T14:07:06.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:06.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510628+0000", 2023-07-02T14:07:06.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2280000000000002e-05 2023-07-02T14:07:06.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:06.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510671+0000", 2023-07-02T14:07:06.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051928864999999998 2023-07-02T14:07:06.256 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.256 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.256 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.256 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19180 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2833, mlcod=177'2827)", 2023-07-02T14:07:06.256 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:07:06.257 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 58.656595875999997, 2023-07-02T14:07:06.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039574109000000003, 2023-07-02T14:07:06.257 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:06.257 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.257 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:06.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:06.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:07:06.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:06.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:06.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:07:06.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:06.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:06.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471118+0000", 2023-07-02T14:07:06.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4799000000000001e-05 2023-07-02T14:07:06.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:06.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471143+0000", 2023-07-02T14:07:06.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3960000000000003e-06 2023-07-02T14:07:06.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:06.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471149+0000", 2023-07-02T14:07:06.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9230000000000004e-06 2023-07-02T14:07:06.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471153+0000", 2023-07-02T14:07:06.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015076050000000001 2023-07-02T14:07:06.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.261 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.261 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.486229+0000", 2023-07-02T14:07:06.261 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8466000000000002e-05 2023-07-02T14:07:06.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:06.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.486257+0000", 2023-07-02T14:07:06.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024419020999999999 2023-07-02T14:07:06.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.262 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.262 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:06.262 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510676+0000", 2023-07-02T14:07:06.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6693000000000001e-05 2023-07-02T14:07:06.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:06.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510693+0000", 2023-07-02T14:07:06.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039574109000000003 2023-07-02T14:07:06.263 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.263 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.264 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.264 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19181 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2834, mlcod=177'2827)", 2023-07-02T14:07:06.264 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:07:06.264 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 58.656554614999997, 2023-07-02T14:07:06.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039548481000000003, 2023-07-02T14:07:06.265 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:06.265 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.265 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:06.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:06.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:07:06.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:06.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:06.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:07:06.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:06.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:06.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:07:06.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8320999999999999e-05 2023-07-02T14:07:06.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:06.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471188+0000", 2023-07-02T14:07:06.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9669999999999998e-06 2023-07-02T14:07:06.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:06.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471194+0000", 2023-07-02T14:07:06.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8709999999999999e-06 2023-07-02T14:07:06.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471198+0000", 2023-07-02T14:07:06.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016760634999999999 2023-07-02T14:07:06.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.269 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.487959+0000", 2023-07-02T14:07:06.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8398000000000002e-05 2023-07-02T14:07:06.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.269 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:06.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.487987+0000", 2023-07-02T14:07:06.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022707627000000001 2023-07-02T14:07:06.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.270 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.270 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:06.270 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510695+0000", 2023-07-02T14:07:06.270 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3991999999999999e-05 2023-07-02T14:07:06.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:06.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510709+0000", 2023-07-02T14:07:06.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039548481000000003 2023-07-02T14:07:06.271 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.271 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.271 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.272 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19192 2.7 e177/141 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head v 177'2835, mlcod=177'2827)", 2023-07-02T14:07:06.272 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:07:06.272 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 58.656511643000002, 2023-07-02T14:07:06.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039521186999999999, 2023-07-02T14:07:06.272 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:06.272 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.273 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:06.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:06.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:07:06.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:06.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:06.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:07:06.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:06.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:06.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:07:06.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014525079 2023-07-02T14:07:06.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:06.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485728+0000", 2023-07-02T14:07:06.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8372e-05 2023-07-02T14:07:06.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:06.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485746+0000", 2023-07-02T14:07:06.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6149999999999997e-06 2023-07-02T14:07:06.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485755+0000", 2023-07-02T14:07:06.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036949270000000002 2023-07-02T14:07:06.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489450+0000", 2023-07-02T14:07:06.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9415e-05 2023-07-02T14:07:06.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.277 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:06.277 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489469+0000", 2023-07-02T14:07:06.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021241488999999999 2023-07-02T14:07:06.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:06.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510711+0000", 2023-07-02T14:07:06.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3604999999999999e-05 2023-07-02T14:07:06.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:06.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510724+0000", 2023-07-02T14:07:06.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039521186999999999 2023-07-02T14:07:06.279 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.279 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.279 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.280 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:20659 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2134016~94208] snapc 35=[35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e179)", 2023-07-02T14:07:06.280 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:07:06.280 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 54.537328260999999, 2023-07-02T14:07:06.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041695300999999997, 2023-07-02T14:07:06.280 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:06.280 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.280 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:06.281 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:06.281 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:06.281 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20659 2023-07-02T14:07:06.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.281 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:06.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:06.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:07:06.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:06.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:06.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:07:06.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:06.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:06.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590386+0000", 2023-07-02T14:07:06.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1415999999999994e-05 2023-07-02T14:07:06.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:06.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590467+0000", 2023-07-02T14:07:06.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8029999999999999e-06 2023-07-02T14:07:06.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:06.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590474+0000", 2023-07-02T14:07:06.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5379999999999998e-06 2023-07-02T14:07:06.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590478+0000", 2023-07-02T14:07:06.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073612449999999998 2023-07-02T14:07:06.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.285 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.285 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.597839+0000", 2023-07-02T14:07:06.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013387800000000001 2023-07-02T14:07:06.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:06.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.597973+0000", 2023-07-02T14:07:06.286 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022472799999999999 2023-07-02T14:07:06.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:06.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.598197+0000", 2023-07-02T14:07:06.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065294259999999996 2023-07-02T14:07:06.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:06.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.604727+0000", 2023-07-02T14:07:06.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027239217999999999 2023-07-02T14:07:06.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:06.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.631966+0000", 2023-07-02T14:07:06.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3194999999999994e-05 2023-07-02T14:07:06.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:06.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632029+0000", 2023-07-02T14:07:06.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2639999999999997e-05 2023-07-02T14:07:06.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:06.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632082+0000", 2023-07-02T14:07:06.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041695300999999997 2023-07-02T14:07:06.289 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.289 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.290 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.290 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:20660 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2031616~196608] snapc 35=[35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e179)", 2023-07-02T14:07:06.290 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:07:06.290 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 54.537233241000003, 2023-07-02T14:07:06.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041610001000000001, 2023-07-02T14:07:06.290 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:06.291 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.291 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:06.291 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:06.291 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:06.291 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20660 2023-07-02T14:07:06.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.291 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:06.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:06.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:07:06.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:06.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:06.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:07:06.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:06.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:06.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590481+0000", 2023-07-02T14:07:06.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012730599999999999 2023-07-02T14:07:06.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:06.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590608+0000", 2023-07-02T14:07:06.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6319999999999997e-06 2023-07-02T14:07:06.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:06.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590617+0000", 2023-07-02T14:07:06.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5209999999999998e-06 2023-07-02T14:07:06.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590621+0000", 2023-07-02T14:07:06.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018367430000000001 2023-07-02T14:07:06.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592457+0000", 2023-07-02T14:07:06.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016640599999999999 2023-07-02T14:07:06.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:06.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592624+0000", 2023-07-02T14:07:06.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028927400000000001 2023-07-02T14:07:06.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:06.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592913+0000", 2023-07-02T14:07:06.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011797813000000001 2023-07-02T14:07:06.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:06.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.604711+0000", 2023-07-02T14:07:06.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027285997999999999 2023-07-02T14:07:06.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:06.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.631997+0000", 2023-07-02T14:07:06.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8890000000000001e-05 2023-07-02T14:07:06.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:06.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632046+0000", 2023-07-02T14:07:06.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5867e-05 2023-07-02T14:07:06.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:06.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632092+0000", 2023-07-02T14:07:06.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041610001000000001 2023-07-02T14:07:06.300 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.300 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.300 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.300 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:23560 2.3 2:c69ae922:::rbd_data.125de1449605.00000000000001da:head [write 65536~4128768] snapc 42=[42,41,40,3f,3e,3d,3c,3b,3a,39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e206)", 2023-07-02T14:07:06.300 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:57.307513+0000", 2023-07-02T14:07:06.301 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 8.82020187, 2023-07-02T14:07:06.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.049163971000000001, 2023-07-02T14:07:06.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:06.301 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.301 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:06.301 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:06.301 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:06.301 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23560 2023-07-02T14:07:06.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.302 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:06.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:06.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.307513+0000", 2023-07-02T14:07:06.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:06.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:06.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.307513+0000", 2023-07-02T14:07:06.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:07:06.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:06.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.307511+0000", 2023-07-02T14:07:06.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010796256000000001 2023-07-02T14:07:06.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:06.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318307+0000", 2023-07-02T14:07:06.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2152e-05 2023-07-02T14:07:06.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:06.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318319+0000", 2023-07-02T14:07:06.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8710000000000002e-06 2023-07-02T14:07:06.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318325+0000", 2023-07-02T14:07:06.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00065064300000000001 2023-07-02T14:07:06.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:06.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318976+0000", 2023-07-02T14:07:06.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4927999999999996e-05 2023-07-02T14:07:06.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:06.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.319071+0000", 2023-07-02T14:07:06.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000184487 2023-07-02T14:07:06.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:06.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.319255+0000", 2023-07-02T14:07:06.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021455622000000001 2023-07-02T14:07:06.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:06.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.340711+0000", 2023-07-02T14:07:06.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015850748000000001 2023-07-02T14:07:06.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:06.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.356562+0000", 2023-07-02T14:07:06.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1957000000000002e-05 2023-07-02T14:07:06.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:06.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.356634+0000", 2023-07-02T14:07:06.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3409999999999999e-05 2023-07-02T14:07:06.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:06.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:06.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:06.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.356677+0000", 2023-07-02T14:07:06.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.049163971000000001 2023-07-02T14:07:06.310 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.310 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.310 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.310 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:06.311 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:06.311 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:06.312 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:07:06.313 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:06.312+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:06.365 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 2, 4] out_osds: [5, 0] dead_osds: [] live_osds: [1, 2, 4, 5, 3, 0] 2023-07-02T14:07:06.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-07-02T14:07:06.369 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:07:06.413 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:06.413+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:06.454 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:06.454 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:07:06.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.455 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "MOSDPGPush(2.2 212/209 [PushOp(2:52fb801b:::rbd_data.125de1449605.0000000000000188:33, version: 176'1627, data_included: [2555904~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:52fb801b:::rbd_data.125de1449605.0000000000000188:33@176'1627, size: 4194304, copy_subset: [0~4194304], clone_subset: {}, snapset: 33=[]:{33=[33,32,31,30,2f]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3604480, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2555904, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-07-02T14:07:06.455 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:07:06.452859+0000", 2023-07-02T14:07:06.455 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 0.0016149999999999999, 2023-07-02T14:07:06.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0016627300000000001, 2023-07-02T14:07:06.455 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:06.455 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:07:06.455 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:06.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:06.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:06.452859+0000", 2023-07-02T14:07:06.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:06.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:06.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:06.452859+0000", 2023-07-02T14:07:06.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:06.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:06.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:06.452858+0000", 2023-07-02T14:07:06.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00099749699999999997 2023-07-02T14:07:06.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:06.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:06.453856+0000", 2023-07-02T14:07:06.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5398999999999999e-05 2023-07-02T14:07:06.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:06.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:06.453871+0000", 2023-07-02T14:07:06.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0325e-05 2023-07-02T14:07:06.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.459 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.459 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:06.453882+0000", 2023-07-02T14:07:06.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4477e-05 2023-07-02T14:07:06.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:06.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:06.453936+0000", 2023-07-02T14:07:06.460 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7299999999999996e-06 2023-07-02T14:07:06.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.460 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.460 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:06.460 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:06.453941+0000", 2023-07-02T14:07:06.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0010818900000000001 2023-07-02T14:07:06.461 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.461 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:06.461 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.461 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.461 INFO:teuthology.orchestra.run.smithi022.stdout: ], 2023-07-02T14:07:06.461 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 1 2023-07-02T14:07:06.461 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:06.467 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:07:06.513 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:06.512+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:06.614 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:06.614+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:06.647 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:06.647 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:07:06.647 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:07:06.648 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:07:06.648 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:06.659 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:07:06.742 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:06.741+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:06.816 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:06.815+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:06.845 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:06.845 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:07:06.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:07:06.845 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:07:06.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.846 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:07:06.846 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:07:06.846 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 206.94604321899999, 2023-07-02T14:07:06.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:07:06.846 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:06.846 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.846 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:06.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:06.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:07:06.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:06.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:06.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:07:06.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:06.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:06.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:07:06.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:07:06.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:06.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:07:06.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:07:06.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:06.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:07:06.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:07:06.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:07:06.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:07:06.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:06.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:07:06.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:07:06.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:06.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:07:06.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:07:06.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:06.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:07:06.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:07:06.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:06.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:07:06.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:07:06.853 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.853 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:06.853 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.854 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:07:06.854 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:07:06.854 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 206.94195750099999, 2023-07-02T14:07:06.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:07:06.854 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:06.854 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.854 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:06.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:06.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:07:06.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:06.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:06.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:07:06.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:06.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:06.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:07:06.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:07:06.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:06.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:07:06.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:07:06.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:06.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:07:06.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:07:06.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:07:06.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:07:06.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:06.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:07:06.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:07:06.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:06.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:07:06.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:07:06.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:06.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:07:06.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:07:06.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:06.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:07:06.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:07:06.861 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.861 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:06.861 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.861 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:07:06.862 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:07:06.862 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 206.93809271999999, 2023-07-02T14:07:06.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:07:06.862 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:06.862 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.862 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:06.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:06.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:07:06.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:06.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:06.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:07:06.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:06.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:06.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:07:06.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:07:06.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:06.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:07:06.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:07:06.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:06.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:07:06.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:07:06.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:07:06.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:07:06.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:06.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:07:06.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:07:06.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:06.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:07:06.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:07:06.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:06.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:07:06.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:07:06.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:06.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:07:06.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:07:06.869 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.869 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:06.869 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.869 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:07:06.870 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:07:06.870 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 206.89886365999999, 2023-07-02T14:07:06.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:07:06.870 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:06.870 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.870 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:06.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:06.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:07:06.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:06.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:06.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:07:06.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:06.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:06.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:07:06.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:07:06.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:06.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:07:06.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:07:06.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:06.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:07:06.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:07:06.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:07:06.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:07:06.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:06.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:07:06.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:07:06.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:06.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:07:06.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:07:06.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:06.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:07:06.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:07:06.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:06.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:07:06.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:07:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:07:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:07:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 206.89420630199999, 2023-07-02T14:07:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:07:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:07:06.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:06.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:06.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:07:06.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:06.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:06.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:07:06.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:07:06.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:06.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:07:06.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:07:06.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:06.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:07:06.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:07:06.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:07:06.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:07:06.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:06.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:07:06.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:07:06.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:06.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:07:06.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:07:06.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:06.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:07:06.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:07:06.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:06.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:07:06.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:07:06.884 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.885 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:06.885 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.885 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:07:06.885 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:07:06.885 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 206.893266439, 2023-07-02T14:07:06.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:07:06.886 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:06.886 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.886 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:06.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:06.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:07:06.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:06.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:06.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:07:06.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:06.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:06.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:07:06.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:07:06.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:06.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:07:06.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:07:06.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:06.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:07:06.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:07:06.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:07:06.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:07:06.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:06.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:07:06.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:07:06.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:06.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:07:06.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:07:06.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:06.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:07:06.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:07:06.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:06.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:07:06.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:07:06.892 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.892 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:06.892 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.893 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:07:06.893 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:07:06.893 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 206.891868561, 2023-07-02T14:07:06.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:07:06.893 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:06.893 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.894 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:06.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:06.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:07:06.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:06.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:06.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:07:06.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:06.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:06.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:07:06.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:07:06.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:06.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:07:06.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:07:06.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:06.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:07:06.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:07:06.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:07:06.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:07:06.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:06.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:07:06.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:07:06.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:06.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:07:06.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:07:06.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:06.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:07:06.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:07:06.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:06.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:07:06.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:07:06.900 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.900 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:06.900 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.901 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:07:06.901 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:07:06.901 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 185.48167806500001, 2023-07-02T14:07:06.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:07:06.901 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:06.901 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.901 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:06.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:06.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:07:06.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:06.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:06.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:07:06.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:06.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:06.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:07:06.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:07:06.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:06.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:07:06.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:07:06.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:06.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:07:06.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:07:06.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:07:06.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:07:06.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:06.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:07:06.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:07:06.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:06.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:07:06.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:07:06.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:06.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:07:06.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:07:06.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:06.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:07:06.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:07:06.908 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.908 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:06.908 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.908 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:07:06.909 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:07:06.909 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 184.11532838100001, 2023-07-02T14:07:06.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:07:06.909 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:06.909 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.909 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:06.909 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:06.910 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:06.910 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:07:06.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.910 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:06.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:06.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:07:06.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:06.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:06.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:07:06.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:06.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:06.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:07:06.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:07:06.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:06.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:07:06.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:07:06.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:06.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:07:06.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:07:06.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:07:06.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:07:06.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:06.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:07:06.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:07:06.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:06.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:07:06.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:07:06.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:06.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:07:06.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:07:06.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:06.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:07:06.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:07:06.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:06.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:07:06.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:07:06.917 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.917 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:06.917 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.918 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:07:06.918 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:07:06.918 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 184.111281037, 2023-07-02T14:07:06.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:07:06.918 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:06.918 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.918 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:06.918 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:06.919 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:06.919 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:07:06.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.919 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:06.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:06.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:07:06.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:06.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:06.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:07:06.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:06.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:06.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:07:06.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:07:06.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:06.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:07:06.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:07:06.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:06.922 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:07:06.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:07:06.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:07:06.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:07:06.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:06.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:07:06.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:07:06.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:06.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:07:06.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:07:06.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:06.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:07:06.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:07:06.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:06.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:07:06.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:07:06.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:06.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:07:06.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:07:06.926 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.926 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:06.926 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.927 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:07:06.927 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:07:06.927 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 184.10671721200001, 2023-07-02T14:07:06.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:07:06.927 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:06.927 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.927 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:06.928 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:06.928 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:06.928 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:07:06.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.928 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:06.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:06.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:07:06.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:06.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:06.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:07:06.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:06.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:06.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:07:06.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:07:06.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:06.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:07:06.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:07:06.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:06.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:07:06.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:07:06.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:07:06.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:07:06.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:06.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:07:06.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:07:06.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:06.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:07:06.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:07:06.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:06.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:07:06.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:07:06.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:06.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:07:06.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:07:06.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:06.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:07:06.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:07:06.935 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.935 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:06.935 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.936 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:07:06.936 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:07:06.936 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 184.10666734599999, 2023-07-02T14:07:06.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:07:06.936 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:06.936 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.937 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:06.937 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:06.937 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:06.937 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:07:06.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.937 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:06.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:06.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:07:06.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:06.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:06.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:07:06.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:06.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:06.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:07:06.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:07:06.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:06.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:07:06.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:07:06.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:06.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:07:06.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:07:06.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:07:06.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:07:06.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:06.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:07:06.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:07:06.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:06.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:07:06.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:07:06.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:06.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:07:06.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:07:06.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:06.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:07:06.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:07:06.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:06.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:07:06.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:07:06.944 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.944 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:06.944 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.945 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:07:06.945 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:07:06.945 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 184.10562835299999, 2023-07-02T14:07:06.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:07:06.945 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:06.945 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.946 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:06.946 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:06.946 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:06.946 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:07:06.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.946 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:06.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:06.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:07:06.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:06.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:06.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:07:06.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:06.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:06.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:07:06.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:07:06.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:06.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:07:06.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:07:06.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:06.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:07:06.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:07:06.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:07:06.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:07:06.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:06.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:07:06.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:07:06.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:06.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:07:06.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:07:06.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:06.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:07:06.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:07:06.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:06.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:07:06.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:07:06.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:06.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:07:06.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:07:06.953 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.953 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:06.954 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.954 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22312 2.0 e197/193 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 197'5590, mlcod=197'5582)", 2023-07-02T14:07:06.954 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:07:06.954 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.095183654000003, 2023-07-02T14:07:06.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021943201999999998, 2023-07-02T14:07:06.954 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:06.955 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.955 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:06.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:06.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:07:06.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:06.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:06.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:07:06.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:06.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:06.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748473+0000", 2023-07-02T14:07:06.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00099631099999999994 2023-07-02T14:07:06.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:06.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749469+0000", 2023-07-02T14:07:06.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4279e-05 2023-07-02T14:07:06.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:06.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749483+0000", 2023-07-02T14:07:06.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5289999999999997e-06 2023-07-02T14:07:06.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749490+0000", 2023-07-02T14:07:06.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047909650000000003 2023-07-02T14:07:06.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:06.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.754281+0000", 2023-07-02T14:07:06.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1999000000000001e-05 2023-07-02T14:07:06.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:06.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.754303+0000", 2023-07-02T14:07:06.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016063747 2023-07-02T14:07:06.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:06.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.770367+0000", 2023-07-02T14:07:06.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0231000000000001e-05 2023-07-02T14:07:06.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:06.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.770417+0000", 2023-07-02T14:07:06.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021943201999999998 2023-07-02T14:07:06.961 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.961 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:06.961 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.962 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22313 2.0 e197/193 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 197'5592, mlcod=197'5587)", 2023-07-02T14:07:06.962 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:07:06.962 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.089930250999998, 2023-07-02T14:07:06.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026016164000000001, 2023-07-02T14:07:06.962 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:06.962 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.962 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:06.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:06.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:07:06.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:06.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:06.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:07:06.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999933 2023-07-02T14:07:06.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:06.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753720+0000", 2023-07-02T14:07:06.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4851999999999995e-05 2023-07-02T14:07:06.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:06.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753785+0000", 2023-07-02T14:07:06.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0216e-05 2023-07-02T14:07:06.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:06.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753796+0000", 2023-07-02T14:07:06.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4249999999999997e-06 2023-07-02T14:07:06.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753801+0000", 2023-07-02T14:07:06.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070274359999999998 2023-07-02T14:07:06.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:06.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760828+0000", 2023-07-02T14:07:06.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0975000000000002e-05 2023-07-02T14:07:06.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:06.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760849+0000", 2023-07-02T14:07:06.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018855916 2023-07-02T14:07:06.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:06.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.779705+0000", 2023-07-02T14:07:06.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8037999999999997e-05 2023-07-02T14:07:06.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:06.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.779743+0000", 2023-07-02T14:07:06.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026016164000000001 2023-07-02T14:07:06.969 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.969 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:06.969 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.969 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22314 2.0 e197/193 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 197'5594, mlcod=197'5587)", 2023-07-02T14:07:06.970 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:07:06.970 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.087325597000003, 2023-07-02T14:07:06.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027004523999999999, 2023-07-02T14:07:06.970 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:06.970 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.970 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:06.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:06.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:07:06.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:06.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:06.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:07:06.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:06.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:06.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756331+0000", 2023-07-02T14:07:06.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4697999999999999e-05 2023-07-02T14:07:06.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:06.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756365+0000", 2023-07-02T14:07:06.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3902e-05 2023-07-02T14:07:06.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:06.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756379+0000", 2023-07-02T14:07:06.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7329999999999997e-06 2023-07-02T14:07:06.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756386+0000", 2023-07-02T14:07:06.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062861929999999998 2023-07-02T14:07:06.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:06.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.762672+0000", 2023-07-02T14:07:06.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9031e-05 2023-07-02T14:07:06.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:06.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.762711+0000", 2023-07-02T14:07:06.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020589275000000001 2023-07-02T14:07:06.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:06.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783301+0000", 2023-07-02T14:07:06.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5720999999999999e-05 2023-07-02T14:07:06.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:06.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783336+0000", 2023-07-02T14:07:06.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027004523999999999 2023-07-02T14:07:06.977 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.977 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:06.977 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.977 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22315 2.0 e197/193 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 197'5595, mlcod=197'5587)", 2023-07-02T14:07:06.977 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:07:06.978 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.083356326000001, 2023-07-02T14:07:06.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023064345999999999, 2023-07-02T14:07:06.978 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:06.978 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.978 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:06.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:06.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:07:06.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:06.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:06.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:07:06.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:06.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:06.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760300+0000", 2023-07-02T14:07:06.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3056000000000002e-05 2023-07-02T14:07:06.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:06.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760353+0000", 2023-07-02T14:07:06.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2825e-05 2023-07-02T14:07:06.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:06.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760366+0000", 2023-07-02T14:07:06.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8290000000000002e-06 2023-07-02T14:07:06.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760373+0000", 2023-07-02T14:07:06.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075982089999999999 2023-07-02T14:07:06.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:06.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.767971+0000", 2023-07-02T14:07:06.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4096000000000001e-05 2023-07-02T14:07:06.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:06.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.767995+0000", 2023-07-02T14:07:06.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015345908 2023-07-02T14:07:06.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:06.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783341+0000", 2023-07-02T14:07:06.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4273e-05 2023-07-02T14:07:06.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:06.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783365+0000", 2023-07-02T14:07:06.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023064345999999999 2023-07-02T14:07:06.984 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.985 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:06.985 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.985 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22319 2.0 e197/193 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 197'5597, mlcod=197'5587)", 2023-07-02T14:07:06.985 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:07:06.985 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.079853505999999, 2023-07-02T14:07:06.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022862501, 2023-07-02T14:07:06.986 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:06.986 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.986 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:06.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:06.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:07:06.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:06.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:06.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:07:06.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:06.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:06.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763803+0000", 2023-07-02T14:07:06.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052067220000000004 2023-07-02T14:07:06.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:06.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769010+0000", 2023-07-02T14:07:06.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3249e-05 2023-07-02T14:07:06.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:06.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769023+0000", 2023-07-02T14:07:06.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1790000000000008e-06 2023-07-02T14:07:06.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769031+0000", 2023-07-02T14:07:06.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030611079999999999 2023-07-02T14:07:06.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:06.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.772092+0000", 2023-07-02T14:07:06.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3958e-05 2023-07-02T14:07:06.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:06.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.772116+0000", 2023-07-02T14:07:06.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014504064000000001 2023-07-02T14:07:06.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:06.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.786620+0000", 2023-07-02T14:07:06.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5952999999999999e-05 2023-07-02T14:07:06.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:06.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.786666+0000", 2023-07-02T14:07:06.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022862501 2023-07-02T14:07:06.992 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.992 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:06.992 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:06.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.993 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22335 2.0 e197/193 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 197'5602, mlcod=197'5599)", 2023-07-02T14:07:06.993 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:07:06.993 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.052783228999999, 2023-07-02T14:07:06.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024437743000000001, 2023-07-02T14:07:06.993 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:06.994 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:06.994 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:06.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:06.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:07:06.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:06.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:06.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:07:06.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:06.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:06.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790873+0000", 2023-07-02T14:07:06.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081672399999999992 2023-07-02T14:07:06.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:06.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799040+0000", 2023-07-02T14:07:06.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5712999999999999e-05 2023-07-02T14:07:06.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:06.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799066+0000", 2023-07-02T14:07:06.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1413999999999999e-05 2023-07-02T14:07:06.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:06.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799077+0000", 2023-07-02T14:07:06.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3102000000000007e-05 2023-07-02T14:07:06.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:06.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799140+0000", 2023-07-02T14:07:06.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.883e-05 2023-07-02T14:07:06.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:06.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799169+0000", 2023-07-02T14:07:06.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016086700999999998 2023-07-02T14:07:06.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:06.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:06.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:06.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.815256+0000", 2023-07-02T14:07:06.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5967000000000001e-05 2023-07-02T14:07:06.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.815312+0000", 2023-07-02T14:07:07.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024437743000000001 2023-07-02T14:07:07.000 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.000 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.000 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.001 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22336 2.0 e197/193 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 197'5603, mlcod=197'5599)", 2023-07-02T14:07:07.001 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:07:07.001 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 32.044547942999998, 2023-07-02T14:07:07.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022713011000000002, 2023-07-02T14:07:07.001 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.001 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.001 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:07:07.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:07:07.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:07.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799108+0000", 2023-07-02T14:07:07.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00058943600000000004 2023-07-02T14:07:07.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799698+0000", 2023-07-02T14:07:07.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6205000000000002e-05 2023-07-02T14:07:07.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799714+0000", 2023-07-02T14:07:07.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9630000000000001e-06 2023-07-02T14:07:07.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799721+0000", 2023-07-02T14:07:07.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076846579999999996 2023-07-02T14:07:07.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.807405+0000", 2023-07-02T14:07:07.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8191000000000002e-05 2023-07-02T14:07:07.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.807434+0000", 2023-07-02T14:07:07.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014338020999999999 2023-07-02T14:07:07.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.821772+0000", 2023-07-02T14:07:07.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0868999999999997e-05 2023-07-02T14:07:07.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.821822+0000", 2023-07-02T14:07:07.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022713011000000002 2023-07-02T14:07:07.008 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.008 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.008 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.008 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.008 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.008 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:07.010 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:07:07.010 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:07.008+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:07.111 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:07.110+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:07.194 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:07.195 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:07:07.195 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:07:07.195 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:07.208 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:07:07.249 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:07.248+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:07.313 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:07.312+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:07.367 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:07:07.367 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.1 to 1 (10000) 2023-07-02T14:07:07.382 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-07-02T14:07:07.390 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:07.390 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:07:07.391 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:07:07.391 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:07:07.391 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:07.403 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:07:07.423 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:07.422+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:07.514 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:07.513+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:07.614 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:07.614 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:07:07.614 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:07:07.614 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:07:07.614 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.615 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:07.615 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:07:07.615 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 268.32095305299998, 2023-07-02T14:07:07.615 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:07:07.615 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.615 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.615 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:07.615 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:07.616 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:07.616 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:07:07.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.616 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.616 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.616 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.616 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:07:07.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:07:07.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:07.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:07:07.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:07:07.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:07:07.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:07:07.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:07:07.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:07:07.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:07:07.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:07:07.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:07:07.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:07:07.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:07:07.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:07:07.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:07.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:07:07.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:07:07.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:07.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:07:07.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:07:07.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.623 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:07.623 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:07:07.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:07:07.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:07:07.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:07:07.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:07:07.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:07:07.625 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.625 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.625 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.625 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:07.625 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:07:07.626 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 268.32071803000002, 2023-07-02T14:07:07.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:07:07.626 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.626 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.626 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:07.626 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:07.626 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:07.626 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:07:07.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.627 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.627 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.627 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:07:07.627 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:07:07.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:07.628 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.628 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.628 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.628 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:07:07.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:07:07.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.629 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.629 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:07:07.629 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:07:07.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.629 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:07:07.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:07:07.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:07:07.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:07:07.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:07:07.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:07:07.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:07:07.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:07:07.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:07.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:07:07.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:07:07.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:07.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:07:07.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:07:07.633 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.633 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:07.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:07:07.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:07:07.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.634 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:07:07.634 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:07:07.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.635 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:07:07.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:07:07.635 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.635 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.635 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.636 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.636 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.636 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:07.636 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:07:07.636 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 268.32051889600001, 2023-07-02T14:07:07.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:07:07.636 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.636 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.637 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:07.637 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:07.637 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:07.637 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:07:07.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.637 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:07:07.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:07:07.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:07.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:07:07.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:07:07.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:07:07.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:07:07.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:07:07.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:07:07.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:07:07.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:07:07.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:07:07.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:07:07.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:07:07.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:07:07.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.643 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:07.643 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:07:07.643 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:07:07.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.644 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.644 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:07.644 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:07:07.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:07:07.644 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:07.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:07:07.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:07:07.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:07:07.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:07:07.646 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:07:07.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:07:07.646 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.646 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.647 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.647 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:07.647 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:07:07.647 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 268.32015377200003, 2023-07-02T14:07:07.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:07:07.648 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.648 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.648 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:07.648 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:07.648 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:07.648 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:07:07.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.649 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:07:07.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.649 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.649 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:07:07.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:07.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:07:07.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:07:07.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:07:07.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:07:07.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:07:07.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:07:07.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:07:07.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:07:07.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:07:07.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:07:07.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:07:07.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:07:07.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.654 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:07.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:07:07.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:07:07.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:07.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:07:07.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:07:07.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.655 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:07.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:07:07.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:07:07.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:07:07.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:07:07.656 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.657 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:07:07.657 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:07:07.657 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.657 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.657 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.658 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:07.658 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:07:07.658 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 268.32007271100002, 2023-07-02T14:07:07.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:07:07.658 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.658 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.658 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:07.659 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:07.659 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:07.659 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:07:07.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.659 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:07:07.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:07:07.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:07.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:07:07.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:07:07.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:07:07.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:07:07.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:07:07.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:07:07.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:07:07.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:07:07.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:07:07.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:07:07.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:07:07.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:07:07.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:07.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:07:07.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:07:07.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:07.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:07:07.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:07:07.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:07.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:07:07.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:07:07.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:07:07.667 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:07:07.667 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:07:07.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:07:07.668 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.668 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.668 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.669 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:07.669 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:07:07.669 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 268.31998104000002, 2023-07-02T14:07:07.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:07:07.669 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.669 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.669 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:07.669 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:07.670 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:07.670 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:07:07.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.670 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:07:07.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:07:07.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:07.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:07:07.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:07:07.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:07:07.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:07:07.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:07:07.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:07:07.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:07:07.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:07:07.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:07:07.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:07:07.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:07:07.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:07:07.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:07.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:07:07.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:07:07.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:07.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:07:07.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:07:07.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:07.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:07:07.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:07:07.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:07:07.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:07:07.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:07:07.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:07:07.678 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.679 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.679 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.679 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:07.679 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:07:07.679 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 268.31964159799998, 2023-07-02T14:07:07.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:07:07.680 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.680 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.680 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:07.680 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:07.680 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:07.680 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:07:07.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.681 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:07:07.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:07:07.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:07.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:07:07.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:07:07.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:07:07.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:07:07.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:07:07.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:07:07.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:07:07.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:07:07.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:07:07.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:07:07.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:07:07.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:07:07.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:07.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:07:07.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:07:07.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:07.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:07:07.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:07:07.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:07.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:07:07.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:07:07.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:07:07.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:07:07.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:07:07.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:07:07.689 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.689 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.689 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.690 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:07.690 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:07:07.690 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 268.31945554499998, 2023-07-02T14:07:07.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:07:07.690 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.690 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.691 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:07.691 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:07.691 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:07.691 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:07:07.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.691 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:07:07.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:07:07.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:07.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:07:07.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:07:07.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:07:07.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:07:07.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:07:07.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:07:07.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:07:07.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:07:07.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:07:07.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:07:07.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.696 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.696 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:07:07.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:07:07.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:07.697 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:07:07.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:07:07.697 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.697 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.697 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:07.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:07:07.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:07:07.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:07.698 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:07:07.698 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:07:07.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:07:07.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:07:07.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:07:07.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:07:07.700 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.700 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.700 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.700 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:07.701 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:07:07.701 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 268.31929774299999, 2023-07-02T14:07:07.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:07:07.701 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.703 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.703 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:07.704 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:07.704 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:07.704 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:07:07.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.705 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:07:07.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:07:07.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:07.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:07:07.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:07:07.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:07:07.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:07:07.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:07:07.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:07:07.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:07:07.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:07:07.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:07:07.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:07:07.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:07:07.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:07:07.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:07.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:07:07.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:07:07.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:07.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:07:07.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:07:07.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:07.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:07:07.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:07:07.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:07:07.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:07:07.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:07:07.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:07:07.721 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.721 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.721 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.722 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:07.722 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:07:07.722 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 268.31910596099999, 2023-07-02T14:07:07.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:07:07.722 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.722 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.722 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:07.723 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:07.723 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:07.723 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:07:07.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.723 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:07:07.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:07:07.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:07.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:07:07.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:07:07.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:07:07.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:07:07.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:07:07.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:07:07.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:07:07.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:07:07.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:07:07.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:07:07.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:07:07.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:07:07.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:07.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:07:07.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:07:07.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:07.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:07:07.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:07:07.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:07.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:07:07.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:07:07.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:07:07.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:07:07.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:07:07.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:07:07.732 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.732 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.732 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.732 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:07.733 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:07:07.733 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 268.31893030100002, 2023-07-02T14:07:07.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:07:07.733 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.733 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.733 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:07.733 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:07.733 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:07.734 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:07:07.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.734 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:07:07.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:07:07.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:07.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:07:07.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:07:07.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:07:07.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:07:07.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:07:07.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:07:07.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:07:07.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:07:07.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:07:07.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:07:07.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:07:07.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:07:07.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:07.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:07:07.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:07:07.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:07.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:07:07.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:07:07.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:07.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:07:07.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:07:07.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:07:07.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:07:07.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:07:07.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:07:07.742 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.743 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.743 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.743 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:07.743 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:07:07.743 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 268.31883611299997, 2023-07-02T14:07:07.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:07:07.744 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.744 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.744 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:07.744 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:07.744 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:07.744 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:07:07.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.744 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:07:07.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:07:07.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:07.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.746 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:07:07.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:07:07.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:07:07.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:07:07.747 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.747 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.747 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.747 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:07:07.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:07:07.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:07:07.748 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:07:07.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:07:07.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:07:07.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:07:07.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:07:07.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:07.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:07:07.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:07:07.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:07.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:07:07.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:07:07.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:07.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:07:07.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:07:07.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:07:07.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:07:07.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:07:07.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:07:07.753 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.753 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.753 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.754 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:07.754 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:07:07.754 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 268.31863272700002, 2023-07-02T14:07:07.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:07:07.754 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.754 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.754 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:07.754 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:07.755 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:07.755 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:07:07.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.755 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:07:07.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:07:07.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:07.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:07:07.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:07:07.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:07:07.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:07:07.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:07:07.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:07:07.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:07:07.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:07:07.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:07:07.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:07:07.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:07:07.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:07:07.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:07.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:07:07.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:07:07.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:07.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:07:07.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:07:07.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:07.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:07:07.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:07:07.762 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.762 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.762 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.762 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:07:07.762 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:07:07.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.763 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:07:07.763 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:07:07.763 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.763 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.763 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.764 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:07.764 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:07:07.764 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 268.31848207100001, 2023-07-02T14:07:07.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:07:07.764 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.764 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.765 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:07.765 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:07.765 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:07.765 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:07:07.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.765 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:07:07.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:07:07.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:07.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:07:07.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:07:07.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:07:07.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:07:07.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:07:07.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:07:07.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:07:07.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:07:07.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:07:07.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:07:07.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:07:07.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:07:07.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:07.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:07:07.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:07:07.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:07.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:07:07.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:07:07.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:07.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:07:07.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:07:07.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:07:07.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:07:07.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:07:07.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:07:07.774 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.774 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.774 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.774 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:07.774 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:07:07.774 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 268.318409368, 2023-07-02T14:07:07.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:07:07.775 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.775 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.775 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:07.775 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:07.775 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:07.775 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:07:07.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.776 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:07:07.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:07:07.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:07.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:07:07.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:07:07.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:07:07.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:07:07.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:07:07.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:07:07.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:07:07.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:07:07.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:07:07.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:07:07.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:07:07.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:07:07.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:07.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:07:07.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:07:07.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:07.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:07:07.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:07:07.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:07.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:07:07.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:07:07.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:07:07.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:07:07.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:07:07.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:07:07.784 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.785 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.785 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.785 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:07.785 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:07:07.785 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 268.31829237800002, 2023-07-02T14:07:07.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:07:07.786 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.786 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.786 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:07.786 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:07.786 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:07.786 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:07:07.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.787 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:07:07.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:07:07.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:07.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:07:07.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:07:07.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:07:07.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:07:07.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:07:07.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:07:07.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:07:07.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:07:07.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:07:07.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:07:07.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:07:07.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:07:07.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:07.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:07:07.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:07:07.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:07.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:07:07.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:07:07.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:07.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:07:07.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:07:07.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:07:07.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:07:07.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:07:07.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:07:07.796 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.796 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.796 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.796 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.796 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.796 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:07.797 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:07:07.797 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 268.318142927, 2023-07-02T14:07:07.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:07:07.797 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.797 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.797 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:07.797 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:07.798 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:07.798 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:07:07.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.798 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:07:07.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:07:07.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:07.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:07:07.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:07:07.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:07:07.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:07:07.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:07:07.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:07:07.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:07:07.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:07:07.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:07:07.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:07:07.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:07:07.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:07:07.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:07.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:07:07.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:07:07.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:07.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:07:07.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:07:07.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:07.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:07:07.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:07:07.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:07:07.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:07:07.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:07:07.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:07:07.807 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.807 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.807 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.807 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:07.807 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:07:07.808 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 268.29666695399999, 2023-07-02T14:07:07.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:07:07.808 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.808 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.808 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:07.808 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:07.808 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:07.808 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:07:07.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.809 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:07:07.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:07:07.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:07.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:07:07.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:07:07.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:07:07.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:07:07.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:07:07.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:07:07.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:07:07.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:07:07.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:07:07.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:07:07.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:07:07.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:07:07.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:07:07.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:07:07.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:07:07.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:07.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:07:07.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:07:07.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:07.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:07:07.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:07:07.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:07:07.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:07:07.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:07:07.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:07:07.817 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.817 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.817 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.818 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:07.818 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:07:07.818 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 268.29653441800002, 2023-07-02T14:07:07.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:07:07.818 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.818 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.819 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:07.819 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:07.819 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:07.819 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:07:07.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.819 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:07:07.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:07:07.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:07.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:07:07.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:07:07.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:07:07.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:07:07.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:07:07.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:07:07.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:07:07.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:07:07.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:07:07.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:07:07.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:07:07.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:07:07.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:07:07.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:07:07.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:07:07.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:07.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:07:07.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:07:07.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:07.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:07:07.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:07:07.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:07:07.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:07:07.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:07:07.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:07:07.828 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.828 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.829 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.829 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:07.829 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:07:07.829 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 268.23519775699998, 2023-07-02T14:07:07.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:07:07.829 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:07.830 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:07.830 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:07.830 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:07.830 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:07.830 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:07:07.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.830 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:07.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:07.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:07:07.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:07.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:07.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:07:07.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:07.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:07.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:07:07.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:07:07.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:07.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:07:07.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:07:07.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:07.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:07:07.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:07:07.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:07.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:07:07.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:07:07.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:07.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:07:07.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:07:07.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:07.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:07:07.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:07:07.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:07.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:07:07.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:07:07.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:07.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:07:07.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:07:07.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:07.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:07:07.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:07:07.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:07.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:07:07.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:07:07.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:07.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:07.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:07.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:07:07.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:07:07.840 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.840 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.840 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.840 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:07.840 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:07.840 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:07.842 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:07:07.842 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:07.841+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:07.864 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:07:07.943 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:07.942+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:08.001 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:07:08.018 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:08.018 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:07:08.018 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:07:08.018 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:08.019 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:07:08.030 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:07:08.035 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:07:08.043 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:08.042+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:08.053 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:07:08.072 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:07:08.090 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:07:08.091 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:07:08.091 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:07:08.091 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:07:08.091 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:07:08.091 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:07:08.092 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:07:08.144 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:08.143+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:08.168 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:08.168 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:08.180 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:07:08.245 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:07:08.244+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:07:08.314 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:08.315 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:08.326 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:07:08.369 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:08.368+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:08.375 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.3 to 1 (10000) 2023-07-02T14:07:08.387 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-07-02T14:07:08.447 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:08.446+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:08.457 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:08.457 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:08.468 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:07:08.551 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:08.550+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:08.634 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:08.635 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:08.645 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:07:08.686 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:08.685+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:08.749 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:08.748+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:08.811 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:08.811 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:08.821 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:07:08.865 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:08.864+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:08.946 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:08.946 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:08.950 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:08.949+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:08.956 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:07:09.051 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:09.050+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:09.119 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:09.120 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:09.130 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:07:09.171 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:09.170+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:09.253 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:09.252+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:09.294 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:09.294 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:09.304 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:07:09.354 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:09.353+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:09.375 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.2 to 1 (10000) 2023-07-02T14:07:09.383 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:07:09.390 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2023-07-02T14:07:09.431 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd export-diff --whole-object --from-snap snap64 b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap65 b3fbb028-e267-4d25-a750-93ba18a2316a@snap65.diff 2023-07-02T14:07:09.434 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:09.434 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:09.445 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:07:09.455 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:09.454+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:09.518 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete...2023-07-02T14:07:09.517+0000 7f6bf548c700 0 -- 172.21.15.163:0/2500840251 >> [v2:172.21.15.22:6808/99977,v1:172.21.15.22:6809/99977] conn(0x7f6bd0008780 msgr2=0x7f6bd000abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-07-02T14:07:09.522 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:07:09.521+0000 7f6beffff700 0 -- 172.21.15.163:0/2500840251 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7f6bd401be80 msgr2=0x7f6bd401e270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:07:09.556 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:09.555+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:09.657 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:09.656+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:09.680 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:09.681 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:09.698 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:07:09.757 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:09.756+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:09.833 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:09.833 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:09.843 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:07:09.885 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:09.884+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:09.958 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:09.957+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:10.002 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:10.002 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:10.016 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:07:10.059 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:07:10.058+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:07:10.103 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:07:10.160 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:10.159+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:10.170 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:10.170 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:10.183 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:07:10.255 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:07:10.266 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete...2023-07-02T14:07:10.265+0000 7f6bf5c8d700 0 -- 172.21.15.163:0/2500840251 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x5596f1296ac0 msgr2=0x5596f1298eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:07:10.266 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:10.261+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:10.283 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:07:10.287 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:07:10.286+0000 7f6bf5c8d700 0 -- 172.21.15.163:0/2500840251 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x7f6bd0008780 msgr2=0x7f6bd803bd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:07:10.290 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:07:10.289+0000 7f6bf548c700 0 -- 172.21.15.163:0/2500840251 >> [v2:172.21.15.22:6808/99977,v1:172.21.15.22:6809/99977] conn(0x7f6bd8036c20 msgr2=0x7f6bd8039050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:07:10.310 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:07:10.334 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:07:10.354 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:07:10.353+0000 7f6beffff700 0 -- 172.21.15.163:0/2500840251 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7f6bd401be80 msgr2=0x7f6bd803e230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:07:10.357 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:07:10.363 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:10.362+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:10.379 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:07:10.381 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:07:10.381 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:07:10.382 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:07:10.382 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:07:10.382 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:07:10.382 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:07:10.383 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.4 to 1 (10000) 2023-07-02T14:07:10.384 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:07:10.389 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:10.390 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:07:10.390 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:07:10.390 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:07:10.390 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.391 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:07:10.391 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:07:10.391 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 210.489154254, 2023-07-02T14:07:10.391 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:07:10.391 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:10.391 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:10.392 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:10.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.392 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:10.392 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:07:10.392 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:10.392 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.392 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:10.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:07:10.393 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:10.393 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.393 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.393 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:10.393 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:07:10.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:07:10.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.394 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.394 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:10.394 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:07:10.394 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:07:10.394 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:10.395 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:07:10.395 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:07:10.395 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.395 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.395 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:10.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:07:10.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:07:10.396 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.396 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.396 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:10.396 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:07:10.396 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:07:10.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.397 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:10.397 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:07:10.397 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:07:10.397 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.397 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:10.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:07:10.398 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:07:10.398 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.398 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.398 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:10.398 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:07:10.399 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:07:10.399 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.399 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.399 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.399 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.399 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.399 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:07:10.399 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:07:10.400 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 210.485068536, 2023-07-02T14:07:10.400 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:07:10.400 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:10.400 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:10.400 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:10.400 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.400 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:10.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:07:10.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:10.401 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.401 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.401 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:10.401 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:07:10.401 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:10.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.402 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:10.402 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:07:10.402 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:07:10.402 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.402 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:10.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:07:10.403 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:07:10.403 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.403 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.403 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:10.403 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:07:10.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:07:10.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.404 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:10.404 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:07:10.404 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:07:10.404 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.404 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:10.405 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:07:10.405 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:07:10.405 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.405 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.405 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:10.405 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:07:10.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:07:10.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.406 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.406 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:10.406 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:07:10.406 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:07:10.406 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.407 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.407 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:10.407 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:07:10.407 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:07:10.407 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.407 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.407 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.408 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.408 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.408 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:07:10.408 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:07:10.408 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 210.481203755, 2023-07-02T14:07:10.408 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:07:10.409 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:10.409 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:10.409 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:10.409 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.409 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:10.409 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:07:10.409 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:10.409 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:10.410 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:07:10.410 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:10.410 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.410 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.410 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:10.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:07:10.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:07:10.411 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.411 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.411 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:10.411 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:07:10.411 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:07:10.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.412 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:10.412 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:07:10.412 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:07:10.412 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.412 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:10.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:07:10.413 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:07:10.413 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.413 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.413 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:10.413 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:07:10.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:07:10.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.414 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.414 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:10.414 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:07:10.414 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:07:10.414 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:10.415 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:07:10.415 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:07:10.415 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.415 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.415 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:10.416 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:07:10.416 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:07:10.416 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.416 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.416 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.416 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.416 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.416 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:07:10.417 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:07:10.417 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 210.441974695, 2023-07-02T14:07:10.417 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:07:10.417 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:10.417 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:10.417 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:10.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:10.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:07:10.418 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:10.418 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.418 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.418 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:10.418 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:07:10.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:10.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.419 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.419 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:10.419 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:07:10.419 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:07:10.419 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:10.420 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:07:10.420 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:07:10.420 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.420 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.420 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:10.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:07:10.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:07:10.421 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.421 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.421 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:10.421 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:07:10.421 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:07:10.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.422 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:10.422 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:07:10.422 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:07:10.422 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.422 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:10.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:07:10.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:07:10.423 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.423 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.423 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:10.423 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:07:10.423 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:07:10.424 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.424 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.424 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:10.424 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:07:10.424 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:07:10.424 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.425 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.425 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.425 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.425 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.425 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:07:10.425 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:07:10.425 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 210.437317337, 2023-07-02T14:07:10.425 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:07:10.426 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:10.426 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:10.426 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:10.426 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.426 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:10.426 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:07:10.426 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:10.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:10.427 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:07:10.427 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:10.427 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.427 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.427 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:10.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:07:10.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:07:10.428 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.428 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.428 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:10.428 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:07:10.428 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:07:10.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:10.429 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:07:10.429 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:07:10.429 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.429 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.429 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:10.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:07:10.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:07:10.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.430 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.430 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:10.430 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:07:10.430 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:07:10.430 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:10.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:07:10.431 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:07:10.431 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.431 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.431 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:10.431 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:07:10.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:07:10.432 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.432 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.432 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:10.432 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:07:10.432 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:07:10.432 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.433 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.433 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.433 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.433 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.433 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:07:10.433 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:07:10.433 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 210.43637747400001, 2023-07-02T14:07:10.434 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:07:10.434 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:10.434 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:10.434 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:10.434 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.434 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:10.434 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:07:10.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:10.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.435 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.435 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:10.435 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:07:10.435 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:10.435 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:10.436 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:07:10.436 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:07:10.436 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.436 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.436 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:10.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:07:10.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:07:10.437 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.437 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.437 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:10.437 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:07:10.437 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:07:10.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:10.438 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:07:10.438 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:07:10.438 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.438 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.438 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:10.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:07:10.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:07:10.439 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.439 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.439 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:10.439 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:07:10.439 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:07:10.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.440 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.440 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:10.440 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:07:10.440 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:07:10.440 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.441 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.441 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:10.441 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:07:10.441 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:07:10.441 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.441 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.441 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.441 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.442 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.442 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:07:10.442 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:07:10.442 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 210.43497959600001, 2023-07-02T14:07:10.442 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:07:10.442 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:10.442 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:10.443 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:10.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.443 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:10.443 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:07:10.443 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:10.443 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.443 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:10.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:07:10.444 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:10.444 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.444 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.444 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:10.444 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:07:10.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:07:10.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.445 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.445 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:10.445 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:07:10.445 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:07:10.445 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:10.446 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:07:10.446 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:07:10.446 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.446 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.446 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:10.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:07:10.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:07:10.447 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.447 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.447 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:10.447 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:07:10.447 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:07:10.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.448 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:10.448 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:07:10.448 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:07:10.448 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.448 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:10.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:07:10.449 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:07:10.449 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.449 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.449 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:10.449 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:07:10.450 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:07:10.450 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.450 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.450 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.450 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.450 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.450 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:07:10.451 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:07:10.451 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 189.02478909999999, 2023-07-02T14:07:10.451 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:07:10.451 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:10.451 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:10.451 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:10.451 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:10.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:07:10.452 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:10.452 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.452 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.452 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:10.452 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:07:10.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:10.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.453 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:10.453 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:07:10.453 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:07:10.453 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.453 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:10.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:07:10.454 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:07:10.454 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.454 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.454 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:10.454 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:07:10.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:07:10.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.455 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.455 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:10.455 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:07:10.455 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:07:10.455 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:10.456 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:07:10.456 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:07:10.456 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.456 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.456 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:10.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:07:10.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:07:10.457 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.457 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.457 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:10.457 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:07:10.457 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:07:10.458 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.458 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.458 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:10.458 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:07:10.458 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:07:10.458 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.458 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.459 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.459 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.459 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.459 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:07:10.459 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:07:10.459 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 187.65843941599999, 2023-07-02T14:07:10.459 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:07:10.459 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:10.460 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:10.460 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:10.460 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:10.460 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:10.460 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:07:10.460 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.460 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:10.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:10.461 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:07:10.461 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:10.461 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.461 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.461 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:10.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:07:10.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:10.462 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.462 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.462 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:10.462 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:07:10.462 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:07:10.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.463 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:10.463 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:07:10.463 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:07:10.463 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.463 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:10.464 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:07:10.464 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:07:10.464 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.464 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.464 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:10.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:07:10.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:07:10.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.465 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.465 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:10.465 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:07:10.465 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:07:10.465 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:10.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:07:10.466 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:07:10.466 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.466 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.466 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:10.466 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:07:10.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:07:10.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.467 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.467 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:10.467 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:07:10.467 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:07:10.467 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.468 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.468 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:10.468 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:07:10.468 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:07:10.468 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.469 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.469 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.469 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.469 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.469 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:07:10.469 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:07:10.469 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 187.65439207200001, 2023-07-02T14:07:10.470 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:07:10.470 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:10.470 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:10.470 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:10.470 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:10.470 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:10.470 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:07:10.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.471 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:10.471 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.471 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:10.471 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:07:10.471 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:10.471 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:10.472 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:07:10.472 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:10.472 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.472 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.472 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:10.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:07:10.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:07:10.473 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.473 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:10.473 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:07:10.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:07:10.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.474 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:10.474 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:07:10.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:07:10.474 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.474 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:10.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:07:10.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:07:10.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:10.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:07:10.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:07:10.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:10.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:07:10.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:07:10.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:10.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:07:10.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:07:10.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:10.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:07:10.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:07:10.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:10.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:07:10.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:07:10.479 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.479 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.479 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.479 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:07:10.479 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:07:10.479 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 187.64982824699999, 2023-07-02T14:07:10.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:07:10.480 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:10.480 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:10.480 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:10.480 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:10.480 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:10.480 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:07:10.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.481 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:10.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:10.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:07:10.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:10.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:10.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:07:10.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:10.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:10.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:07:10.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:07:10.483 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:10.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:07:10.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:07:10.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.484 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:10.484 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:07:10.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:07:10.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:10.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:07:10.485 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:07:10.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.485 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.485 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:10.485 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:07:10.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:07:10.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:10.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:07:10.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:07:10.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:10.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:07:10.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:07:10.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:10.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:07:10.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:07:10.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:10.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:07:10.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:07:10.488 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.488 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.489 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.489 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:07:10.489 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:07:10.489 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 187.649778381, 2023-07-02T14:07:10.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:07:10.489 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:10.490 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:10.490 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:10.490 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:10.490 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:10.490 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:07:10.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.490 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:10.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:10.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:07:10.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:10.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:10.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:07:10.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:10.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:10.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:07:10.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:07:10.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:10.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:07:10.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:07:10.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:10.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:07:10.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:07:10.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:10.494 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:07:10.494 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:07:10.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.494 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:10.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:07:10.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:07:10.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.495 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.495 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:10.495 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:07:10.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:07:10.495 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:10.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:07:10.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:07:10.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:10.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:07:10.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:07:10.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:10.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:07:10.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:07:10.497 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.498 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.498 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.498 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:07:10.498 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:07:10.498 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 187.648739388, 2023-07-02T14:07:10.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:07:10.499 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:10.499 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:10.499 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:10.499 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:10.499 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:10.499 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:07:10.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.499 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:10.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:10.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:07:10.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:10.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:10.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:07:10.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:10.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:10.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:07:10.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:07:10.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:10.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:07:10.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:07:10.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:10.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:07:10.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:07:10.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:10.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:07:10.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:07:10.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:10.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:07:10.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:07:10.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:10.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:07:10.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:07:10.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:10.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:07:10.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:07:10.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:10.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:07:10.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:07:10.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:10.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:07:10.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:07:10.507 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.507 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.507 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.507 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22312 2.0 e197/193 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 197'5590, mlcod=197'5582)", 2023-07-02T14:07:10.507 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:07:10.507 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.638294688999999, 2023-07-02T14:07:10.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021943201999999998, 2023-07-02T14:07:10.508 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:10.508 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:10.508 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:10.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:10.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:07:10.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:10.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:10.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:07:10.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:10.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:10.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748473+0000", 2023-07-02T14:07:10.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00099631099999999994 2023-07-02T14:07:10.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:10.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749469+0000", 2023-07-02T14:07:10.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4279e-05 2023-07-02T14:07:10.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:10.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749483+0000", 2023-07-02T14:07:10.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5289999999999997e-06 2023-07-02T14:07:10.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:10.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749490+0000", 2023-07-02T14:07:10.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047909650000000003 2023-07-02T14:07:10.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:10.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.754281+0000", 2023-07-02T14:07:10.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1999000000000001e-05 2023-07-02T14:07:10.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:10.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.754303+0000", 2023-07-02T14:07:10.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016063747 2023-07-02T14:07:10.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:10.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.770367+0000", 2023-07-02T14:07:10.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0231000000000001e-05 2023-07-02T14:07:10.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:10.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.770417+0000", 2023-07-02T14:07:10.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021943201999999998 2023-07-02T14:07:10.514 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.514 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.515 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.515 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22313 2.0 e197/193 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 197'5592, mlcod=197'5587)", 2023-07-02T14:07:10.515 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:07:10.515 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.633041286000001, 2023-07-02T14:07:10.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026016164000000001, 2023-07-02T14:07:10.515 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:10.516 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:10.516 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:10.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:10.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:07:10.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:10.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:10.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:07:10.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999933 2023-07-02T14:07:10.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:10.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753720+0000", 2023-07-02T14:07:10.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4851999999999995e-05 2023-07-02T14:07:10.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:10.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753785+0000", 2023-07-02T14:07:10.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0216e-05 2023-07-02T14:07:10.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:10.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753796+0000", 2023-07-02T14:07:10.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4249999999999997e-06 2023-07-02T14:07:10.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:10.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753801+0000", 2023-07-02T14:07:10.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070274359999999998 2023-07-02T14:07:10.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:10.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760828+0000", 2023-07-02T14:07:10.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0975000000000002e-05 2023-07-02T14:07:10.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:10.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760849+0000", 2023-07-02T14:07:10.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018855916 2023-07-02T14:07:10.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:10.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.779705+0000", 2023-07-02T14:07:10.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8037999999999997e-05 2023-07-02T14:07:10.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:10.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.779743+0000", 2023-07-02T14:07:10.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026016164000000001 2023-07-02T14:07:10.522 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.522 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.522 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.523 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22314 2.0 e197/193 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 197'5594, mlcod=197'5587)", 2023-07-02T14:07:10.523 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:07:10.523 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.630436631999999, 2023-07-02T14:07:10.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027004523999999999, 2023-07-02T14:07:10.523 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:10.523 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:10.523 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:10.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:10.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:07:10.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:10.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:10.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:07:10.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:10.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:10.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756331+0000", 2023-07-02T14:07:10.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4697999999999999e-05 2023-07-02T14:07:10.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:10.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756365+0000", 2023-07-02T14:07:10.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3902e-05 2023-07-02T14:07:10.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:10.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756379+0000", 2023-07-02T14:07:10.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7329999999999997e-06 2023-07-02T14:07:10.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:10.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756386+0000", 2023-07-02T14:07:10.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062861929999999998 2023-07-02T14:07:10.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:10.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.762672+0000", 2023-07-02T14:07:10.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9031e-05 2023-07-02T14:07:10.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:10.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.762711+0000", 2023-07-02T14:07:10.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020589275000000001 2023-07-02T14:07:10.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:10.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783301+0000", 2023-07-02T14:07:10.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5720999999999999e-05 2023-07-02T14:07:10.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:10.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783336+0000", 2023-07-02T14:07:10.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027004523999999999 2023-07-02T14:07:10.530 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.530 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.530 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.531 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22315 2.0 e197/193 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 197'5595, mlcod=197'5587)", 2023-07-02T14:07:10.531 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:07:10.531 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.626467361000003, 2023-07-02T14:07:10.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023064345999999999, 2023-07-02T14:07:10.531 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:10.531 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:10.531 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:10.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:10.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:07:10.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:10.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:10.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:07:10.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:10.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:10.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760300+0000", 2023-07-02T14:07:10.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3056000000000002e-05 2023-07-02T14:07:10.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:10.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760353+0000", 2023-07-02T14:07:10.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2825e-05 2023-07-02T14:07:10.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:10.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760366+0000", 2023-07-02T14:07:10.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8290000000000002e-06 2023-07-02T14:07:10.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:10.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760373+0000", 2023-07-02T14:07:10.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075982089999999999 2023-07-02T14:07:10.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:10.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.767971+0000", 2023-07-02T14:07:10.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4096000000000001e-05 2023-07-02T14:07:10.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.536 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:10.536 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.767995+0000", 2023-07-02T14:07:10.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015345908 2023-07-02T14:07:10.536 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:10.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783341+0000", 2023-07-02T14:07:10.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4273e-05 2023-07-02T14:07:10.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.537 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:10.537 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783365+0000", 2023-07-02T14:07:10.537 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023064345999999999 2023-07-02T14:07:10.538 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.538 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.538 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.538 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22319 2.0 e197/193 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 197'5597, mlcod=197'5587)", 2023-07-02T14:07:10.538 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:07:10.538 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.622964541000002, 2023-07-02T14:07:10.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022862501, 2023-07-02T14:07:10.539 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:10.539 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:10.539 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:10.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:10.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:07:10.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:10.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:10.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:07:10.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:10.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:10.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763803+0000", 2023-07-02T14:07:10.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052067220000000004 2023-07-02T14:07:10.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:10.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769010+0000", 2023-07-02T14:07:10.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3249e-05 2023-07-02T14:07:10.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:10.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769023+0000", 2023-07-02T14:07:10.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1790000000000008e-06 2023-07-02T14:07:10.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:10.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769031+0000", 2023-07-02T14:07:10.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030611079999999999 2023-07-02T14:07:10.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:10.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.772092+0000", 2023-07-02T14:07:10.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3958e-05 2023-07-02T14:07:10.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:10.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.772116+0000", 2023-07-02T14:07:10.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014504064000000001 2023-07-02T14:07:10.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.786620+0000", 2023-07-02T14:07:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5952999999999999e-05 2023-07-02T14:07:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.786666+0000", 2023-07-02T14:07:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022862501 2023-07-02T14:07:10.545 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22335 2.0 e197/193 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 197'5602, mlcod=197'5599)", 2023-07-02T14:07:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:07:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.595894264000002, 2023-07-02T14:07:10.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024437743000000001, 2023-07-02T14:07:10.547 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:10.547 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:10.547 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:10.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.547 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:10.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:07:10.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:10.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:10.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:07:10.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:10.548 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.548 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.548 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:10.548 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790873+0000", 2023-07-02T14:07:10.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081672399999999992 2023-07-02T14:07:10.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.549 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:10.549 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799040+0000", 2023-07-02T14:07:10.549 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5712999999999999e-05 2023-07-02T14:07:10.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:10.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799066+0000", 2023-07-02T14:07:10.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1413999999999999e-05 2023-07-02T14:07:10.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:10.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799077+0000", 2023-07-02T14:07:10.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3102000000000007e-05 2023-07-02T14:07:10.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:10.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799140+0000", 2023-07-02T14:07:10.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.883e-05 2023-07-02T14:07:10.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:10.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799169+0000", 2023-07-02T14:07:10.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016086700999999998 2023-07-02T14:07:10.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:10.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.815256+0000", 2023-07-02T14:07:10.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5967000000000001e-05 2023-07-02T14:07:10.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:10.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.815312+0000", 2023-07-02T14:07:10.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024437743000000001 2023-07-02T14:07:10.553 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.553 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.553 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.554 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22336 2.0 e197/193 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 197'5603, mlcod=197'5599)", 2023-07-02T14:07:10.554 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:07:10.554 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 35.587658978, 2023-07-02T14:07:10.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022713011000000002, 2023-07-02T14:07:10.554 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:10.554 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:10.555 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:10.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:10.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:07:10.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:10.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:10.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:07:10.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:10.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:10.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799108+0000", 2023-07-02T14:07:10.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00058943600000000004 2023-07-02T14:07:10.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:10.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799698+0000", 2023-07-02T14:07:10.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6205000000000002e-05 2023-07-02T14:07:10.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:10.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799714+0000", 2023-07-02T14:07:10.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9630000000000001e-06 2023-07-02T14:07:10.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:10.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799721+0000", 2023-07-02T14:07:10.558 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076846579999999996 2023-07-02T14:07:10.558 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:10.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.807405+0000", 2023-07-02T14:07:10.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8191000000000002e-05 2023-07-02T14:07:10.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:10.559 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.807434+0000", 2023-07-02T14:07:10.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014338020999999999 2023-07-02T14:07:10.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.560 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.560 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:10.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.821772+0000", 2023-07-02T14:07:10.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0868999999999997e-05 2023-07-02T14:07:10.560 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:10.561 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:10.561 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:10.561 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.821822+0000", 2023-07-02T14:07:10.561 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022713011000000002 2023-07-02T14:07:10.561 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.561 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.561 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.561 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:10.562 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:10.562 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:10.564 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:07:10.564 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:10.562+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:10.565 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-07-02T14:07:10.565 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:07:10.663 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:10.662+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:10.764 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:10.763+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:10.766 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:10.766 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:07:10.766 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:07:10.766 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:10.772 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete...2023-07-02T14:07:10.771+0000 7f6beffff700 0 -- 172.21.15.163:0/2500840251 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7f6bd8036c20 msgr2=0x7f6bd8039010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:07:10.779 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:07:10.865 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:07:10.864+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:07:10.941 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:10.942 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:07:10.942 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:07:10.942 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:07:10.942 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:10.944 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:10.946 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:07:10.954 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:07:10.958 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2023-07-02T14:07:10.378845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11892805145,"num_objects":3557,"num_object_clones":2995,"num_object_copies":7114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":354,"num_objects_misplaced":401,"num_objects_unfound":0,"num_objects_dirty":3557,"num_whiteouts":0,"num_read":88256,"num_read_kb":15199597,"num_write":27213,"num_write_kb":11540672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2862,"num_bytes_recovered":9703530998,"num_keys_recovered":186,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26888,"ondisk_log_size":26888,"up":18,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":21655396,"kb_used_data":21313668,"kb_used_omap":229,"kb_used_meta":341466,"kb_avail":355831964,"statfs":{"total":386547056640,"available":364371931136,"internally_reserved":0,"allocated":21825196032,"data_stored":25424386923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":349661406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22749346,"num_objects":16,"num_object_clones":15,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":136,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1953,"num_read_kb":671410,"num_write":89,"num_write_kb":22250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":922657902,"num_keys_recovered":8,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.088950"},"pg_stats":[{"pgid":"2.7","version":"212'3647","reported_seq":9311,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-07-02T14:07:05.528711+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:07:05.528711+0000","last_peered":"2023-07-02T14:07:05.528711+0000","last_clean":"2023-07-02T14:07:05.528711+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:07:05.528711+0000","last_undegraded":"2023-07-02T14:07:05.528711+0000","last_fullsized":"2023-07-02T14:07:05.528711+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3647,"ondisk_log_size":3647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1570191828,"num_objects":466,"num_object_clones":394,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":57412,"num_read_kb":1894335,"num_write":3056,"num_write_kb":1525929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"215'3058","reported_seq":5145,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:10.324843+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:07:10.324843+0000","last_peered":"2023-07-02T14:07:10.324843+0000","last_clean":"2023-07-02T14:07:10.324843+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:07:10.324843+0000","last_undegraded":"2023-07-02T14:07:10.324843+0000","last_fullsized":"2023-07-02T14:07:10.324843+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3058,"ondisk_log_size":3058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535489458,"num_objects":482,"num_object_clones":414,"num_object_copies":964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":0,"num_read":1926,"num_read_kb":2091895,"num_write":2644,"num_write_kb":1489957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"210'3193","reported_seq":3825,"reported_epoch":213,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-07-02T14:07:08.095386+0000","last_change":"2023-07-02T14:07:07.848445+0000","last_active":"2023-07-02T14:07:08.095386+0000","last_peered":"2023-07-02T14:07:08.095386+0000","last_clean":"2023-07-02T14:07:01.043240+0000","last_became_active":"2023-07-02T14:07:04.030791+0000","last_became_peered":"2023-07-02T14:07:04.030791+0000","last_unstale":"2023-07-02T14:07:08.095386+0000","last_undegraded":"2023-07-02T14:07:03.107328+0000","last_fullsized":"2023-07-02T14:07:03.101623+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3193,"ondisk_log_size":3193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1516729292,"num_objects":421,"num_object_clones":355,"num_object_copies":842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":401,"num_objects_unfound":0,"num_objects_dirty":421,"num_whiteouts":0,"num_read":572,"num_read_kb":1826967,"num_write":2838,"num_write_kb":1467491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":909742242,"num_keys_recovered":8,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":401},{"shards":"3","objects":18}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"215'6476","reported_seq":8964,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:10.334213+0000","last_change":"2023-07-02T14:06:29.572710+0000","last_active":"2023-07-02T14:07:10.334213+0000","last_peered":"2023-07-02T14:07:10.334213+0000","last_clean":"2023-07-02T14:07:10.334213+0000","last_became_active":"2023-07-02T14:06:29.569824+0000","last_became_peered":"2023-07-02T14:06:29.569824+0000","last_unstale":"2023-07-02T14:07:10.334213+0000","last_undegraded":"2023-07-02T14:07:10.334213+0000","last_fullsized":"2023-07-02T14:07:10.334213+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":6476,"ondisk_log_size":6476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672852930,"num_objects":488,"num_object_clones":413,"num_object_copies":976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":22953,"num_read_kb":2152050,"num_write":9580,"num_write_kb":1632373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":113,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-07-02T14:07:05.358697+0000","last_change":"2023-07-02T14:07:02.103009+0000","last_active":"2023-07-02T14:07:05.358697+0000","last_peered":"2023-07-02T14:07:05.358697+0000","last_clean":"2023-07-02T14:07:05.358697+0000","last_became_active":"2023-07-02T14:07:02.102654+0000","last_became_peered":"2023-07-02T14:07:02.102654+0000","last_unstale":"2023-07-02T14:07:05.358697+0000","last_undegraded":"2023-07-02T14:07:05.358697+0000","last_fullsized":"2023-07-02T14:07:05.358697+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"215'2964","reported_seq":4463,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:10.302466+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:07:10.302466+0000","last_peered":"2023-07-02T14:07:10.302466+0000","last_clean":"2023-07-02T14:07:10.302466+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:07:10.302466+0000","last_undegraded":"2023-07-02T14:07:10.302466+0000","last_fullsized":"2023-07-02T14:07:10.302466+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2964,"ondisk_log_size":2964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1658680446,"num_objects":491,"num_object_clones":414,"num_object_copies":982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":0,"num_read":1509,"num_read_kb":2028229,"num_write":2550,"num_write_kb":1608728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"215'2867","reported_seq":3734,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:10.282420+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:07:10.282420+0000","last_peered":"2023-07-02T14:07:10.282420+0000","last_clean":"2023-07-02T14:07:10.282420+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:07:10.282420+0000","last_undegraded":"2023-07-02T14:07:10.282420+0000","last_fullsized":"2023-07-02T14:07:10.282420+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2867,"ondisk_log_size":2867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1564710484,"num_objects":455,"num_object_clones":382,"num_object_copies":910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":455,"num_whiteouts":0,"num_read":949,"num_read_kb":1945167,"num_write":2485,"num_write_kb":1518698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"215'2325","reported_seq":5128,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:10.305571+0000","last_change":"2023-07-02T14:07:09.383558+0000","last_active":"2023-07-02T14:07:10.305571+0000","last_peered":"2023-07-02T14:07:10.305571+0000","last_clean":"2023-07-02T14:07:10.305571+0000","last_became_active":"2023-07-02T14:07:09.382190+0000","last_became_peered":"2023-07-02T14:07:09.382190+0000","last_unstale":"2023-07-02T14:07:10.305571+0000","last_undegraded":"2023-07-02T14:07:10.305571+0000","last_fullsized":"2023-07-02T14:07:10.305571+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2325,"ondisk_log_size":2325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1230505737,"num_objects":402,"num_object_clones":325,"num_object_copies":804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":1898,"num_read_kb":1730263,"num_write":2000,"num_write_kb":1186689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":1578968698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"215'2358","reported_seq":3456,"reported_epoch":215,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-07-02T14:07:10.279927+0000","last_change":"2023-07-02T14:07:04.022783+0000","last_active":"2023-07-02T14:07:10.279927+0000","last_peered":"2023-07-02T14:07:10.279927+0000","last_clean":"2023-07-02T14:07:01.026611+0000","last_became_active":"2023-07-02T14:07:04.018152+0000","last_became_peered":"2023-07-02T14:07:04.018152+0000","last_unstale":"2023-07-02T14:07:10.279927+0000","last_undegraded":"2023-07-02T14:07:03.110294+0000","last_fullsized":"2023-07-02T14:07:03.103717+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2358,"ondisk_log_size":2358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1143644970,"num_objects":352,"num_object_clones":298,"num_object_copies":704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":1037,"num_read_kb":1530691,"num_write":2060,"num_write_kb":1110807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":271880192,"num_keys_recovered":4,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":346},{"shards":"1","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":11892805145,"num_objects":3557,"num_object_clones":2995,"num_object_copies":7114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":354,"num_objects_misplaced":401,"num_objects_unfound":0,"num_objects_dirty":3557,"num_whiteouts":0,"num_read":88256,"num_read_kb":15199597,"num_write":27213,"num_write_kb":11540672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2862,"num_bytes_recovered":9703530998,"num_keys_recovered":186,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25840590848,"data_stored":30051234943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":26888,"ondisk_log_size":26888,"up":16,"acting":14,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640323,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640323,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5235024,"kb_used_data":5143176,"kb_used_omap":0,"kb_used_meta":91840,"kb_avail":89136816,"statfs":{"total":96636764160,"available":91276099584,"internally_reserved":0,"allocated":5266612224,"data_stored":6072961790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":94044160},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.307}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44,"15min":0.46400000000000002},"min":{"1min":0.28399999999999997,"5min":0.215,"15min":0.215},"max":{"1min":0.71499999999999997,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.45000000000000001},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.32800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66800000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.39900000000000002}]},{"osd":3,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.38300000000000001},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45900000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.36799999999999999,"15min":0.36299999999999999},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.48799999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.56200000000000006}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.44900000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.433,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59099999999999997,"5min":1.026,"15min":1.026},"last":0.42899999999999999}]}]},{"osd":3,"up_from":133,"seq":571230650394,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4560208,"kb_used_data":4490568,"kb_used_omap":229,"kb_used_meta":69402,"kb_avail":89811632,"statfs":{"total":96636764160,"available":91967111168,"internally_reserved":0,"allocated":4598341632,"data_stored":5339881520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":71067870},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.443}]},{"osd":1,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.42299999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.504}]},{"osd":4,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.49099999999999999}]},{"osd":5,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.52000000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640323,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3192604,"kb_used_data":3138708,"kb_used_omap":0,"kb_used_meta":53888,"kb_avail":91179236,"statfs":{"total":96636764160,"available":93367537664,"internally_reserved":0,"allocated":3214036992,"data_stored":3691892043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55181312},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.28799999999999998}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.42499999999999999,"15min":0.374},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.376,"5min":0.33500000000000002,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.22500000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:07:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.91100000000000003,"5min":1.929,"15min":1.929},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.54800000000000004,"15min":0.55600000000000005},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93999999999999995,"5min":2,"15min":2},"last":0.35999999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.40899999999999997}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68400000000000005,"5min":1.071,"15min":1.071},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":0.432}]}]},{"osd":0,"up_from":0,"seq":712964571152,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":11,"seq":47244640323,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8667560,"kb_used_data":8541216,"kb_used_omap":0,"kb_used_meta":126336,"kb_avail":85704280,"statfs":{"total":96636764160,"available":87761182720,"internally_reserved":0,"allocated":8746205184,"data_stored":10319651570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":129368064},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.32600000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.247},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.224,"5min":0.184,"15min":0.184},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.33800000000000002}]},{"osd":3,"last update":"Sun Jul 2 14:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.49099999999999999,"15min":0.47299999999999998},"min":{"1min":0.30299999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.38600000000000001}]},{"osd":4,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71899999999999997,"5min":1.524,"15min":1.524},"last":0.51000000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.80300000000000005,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84399999999999997,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.42999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3981160448,"data_stored":4543561053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8745132032,"data_stored":10339602553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3211833344,"data_stored":3699260114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4595769344,"data_stored":5338870610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5306695680,"data_stored":6129940613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:07:10.959 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:07:10.999 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:07:10.998+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:07:11.066 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:11.065+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:11.098 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-07-02T14:07:11.109 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:11.109 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:07:11.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:07:11.109 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:07:11.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.110 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:231 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:11.110 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:07:11.110 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 271.81586653099998, 2023-07-02T14:07:11.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906, 2023-07-02T14:07:11.110 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.110 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.110 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.111 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.111 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.111 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 231 2023-07-02T14:07:11.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.111 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:07:11.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290776+0000", 2023-07-02T14:07:11.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:11.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290775+0000", 2023-07-02T14:07:11.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000211626 2023-07-02T14:07:11.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290986+0000", 2023-07-02T14:07:11.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0192e-05 2023-07-02T14:07:11.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.290997+0000", 2023-07-02T14:07:11.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:07:11.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291004+0000", 2023-07-02T14:07:11.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01435646 2023-07-02T14:07:11.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305360+0000", 2023-07-02T14:07:11.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4641000000000006e-05 2023-07-02T14:07:11.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305435+0000", 2023-07-02T14:07:11.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148567 2023-07-02T14:07:11.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:11.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.305584+0000", 2023-07-02T14:07:11.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088575879999999996 2023-07-02T14:07:11.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314441+0000", 2023-07-02T14:07:11.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003131539 2023-07-02T14:07:11.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317573+0000", 2023-07-02T14:07:11.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7297999999999999e-05 2023-07-02T14:07:11.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317630+0000", 2023-07-02T14:07:11.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3571000000000001e-05 2023-07-02T14:07:11.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317664+0000", 2023-07-02T14:07:11.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026887906 2023-07-02T14:07:11.119 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.120 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.120 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.120 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:232 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1703936~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:11.120 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:07:11.120 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 271.81563150800002, 2023-07-02T14:07:11.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624, 2023-07-02T14:07:11.121 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.121 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.121 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.121 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.121 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.121 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 232 2023-07-02T14:07:11.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.121 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:07:11.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291011+0000", 2023-07-02T14:07:11.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:11.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291010+0000", 2023-07-02T14:07:11.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017761099999999999 2023-07-02T14:07:11.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291187+0000", 2023-07-02T14:07:11.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0025e-05 2023-07-02T14:07:11.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291197+0000", 2023-07-02T14:07:11.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8289999999999996e-06 2023-07-02T14:07:11.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291203+0000", 2023-07-02T14:07:11.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015045842 2023-07-02T14:07:11.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306249+0000", 2023-07-02T14:07:11.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:07:11.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306339+0000", 2023-07-02T14:07:11.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148746 2023-07-02T14:07:11.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:11.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306488+0000", 2023-07-02T14:07:11.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080060240000000005 2023-07-02T14:07:11.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314494+0000", 2023-07-02T14:07:11.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031757209999999998 2023-07-02T14:07:11.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317670+0000", 2023-07-02T14:07:11.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8048999999999999e-05 2023-07-02T14:07:11.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317698+0000", 2023-07-02T14:07:11.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1253000000000001e-05 2023-07-02T14:07:11.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317719+0000", 2023-07-02T14:07:11.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026708624 2023-07-02T14:07:11.130 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.130 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.130 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.130 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:233 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1769472~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:11.131 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:07:11.131 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 271.81543237400001, 2023-07-02T14:07:11.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001, 2023-07-02T14:07:11.131 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.131 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.131 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.131 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.131 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.132 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 233 2023-07-02T14:07:11.132 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.132 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.132 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.132 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.132 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:07:11.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291210+0000", 2023-07-02T14:07:11.133 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:11.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291209+0000", 2023-07-02T14:07:11.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000347986 2023-07-02T14:07:11.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291557+0000", 2023-07-02T14:07:11.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5159999999999994e-06 2023-07-02T14:07:11.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291565+0000", 2023-07-02T14:07:11.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6430000000000003e-06 2023-07-02T14:07:11.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291570+0000", 2023-07-02T14:07:11.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015226440000000001 2023-07-02T14:07:11.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306797+0000", 2023-07-02T14:07:11.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3521000000000002e-05 2023-07-02T14:07:11.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.306880+0000", 2023-07-02T14:07:11.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014783099999999999 2023-07-02T14:07:11.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:11.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.307028+0000", 2023-07-02T14:07:11.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075237890000000003 2023-07-02T14:07:11.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314552+0000", 2023-07-02T14:07:11.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031715519999999998 2023-07-02T14:07:11.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317723+0000", 2023-07-02T14:07:11.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3932999999999999e-05 2023-07-02T14:07:11.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317747+0000", 2023-07-02T14:07:11.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8199999999999999e-05 2023-07-02T14:07:11.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317765+0000", 2023-07-02T14:07:11.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026555602000000001 2023-07-02T14:07:11.140 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.140 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.141 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.141 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:234 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1835008~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:11.141 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:07:11.141 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 271.81506725000003, 2023-07-02T14:07:11.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998, 2023-07-02T14:07:11.141 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.142 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.142 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.142 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.142 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.142 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 234 2023-07-02T14:07:11.142 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.142 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:07:11.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.143 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.143 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291575+0000", 2023-07-02T14:07:11.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:11.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.144 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291574+0000", 2023-07-02T14:07:11.144 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8320999999999999e-05 2023-07-02T14:07:11.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.144 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.144 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291633+0000", 2023-07-02T14:07:11.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1949e-05 2023-07-02T14:07:11.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291645+0000", 2023-07-02T14:07:11.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1779999999999997e-06 2023-07-02T14:07:11.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291652+0000", 2023-07-02T14:07:11.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017359039999999999 2023-07-02T14:07:11.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309011+0000", 2023-07-02T14:07:11.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110927 2023-07-02T14:07:11.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309122+0000", 2023-07-02T14:07:11.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000296197 2023-07-02T14:07:11.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:11.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.309418+0000", 2023-07-02T14:07:11.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051804879999999996 2023-07-02T14:07:11.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314598+0000", 2023-07-02T14:07:11.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003171063 2023-07-02T14:07:11.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317769+0000", 2023-07-02T14:07:11.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2887999999999998e-05 2023-07-02T14:07:11.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317792+0000", 2023-07-02T14:07:11.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9893000000000001e-05 2023-07-02T14:07:11.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317812+0000", 2023-07-02T14:07:11.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026237325999999998 2023-07-02T14:07:11.151 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.151 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.151 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.151 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:235 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1900544~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:11.151 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:07:11.151 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 271.81498618900002, 2023-07-02T14:07:11.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001, 2023-07-02T14:07:11.152 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.152 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.152 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.152 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.152 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.152 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 235 2023-07-02T14:07:11.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.153 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:07:11.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291656+0000", 2023-07-02T14:07:11.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:11.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.154 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.154 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291655+0000", 2023-07-02T14:07:11.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0091000000000004e-05 2023-07-02T14:07:11.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291726+0000", 2023-07-02T14:07:11.155 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0744000000000001e-05 2023-07-02T14:07:11.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.155 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.155 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291736+0000", 2023-07-02T14:07:11.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.31e-06 2023-07-02T14:07:11.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291742+0000", 2023-07-02T14:07:11.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019375376999999999 2023-07-02T14:07:11.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311117+0000", 2023-07-02T14:07:11.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4282000000000004e-05 2023-07-02T14:07:11.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311201+0000", 2023-07-02T14:07:11.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012005100000000001 2023-07-02T14:07:11.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:11.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311321+0000", 2023-07-02T14:07:11.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062429470000000004 2023-07-02T14:07:11.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317564+0000", 2023-07-02T14:07:11.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025171299999999998 2023-07-02T14:07:11.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317816+0000", 2023-07-02T14:07:11.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2704000000000001e-05 2023-07-02T14:07:11.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317839+0000", 2023-07-02T14:07:11.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7181999999999999e-05 2023-07-02T14:07:11.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317856+0000", 2023-07-02T14:07:11.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026199902000000001 2023-07-02T14:07:11.161 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.161 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.161 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.161 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:237 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1966080~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:11.162 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:07:11.162 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 271.81489451800002, 2023-07-02T14:07:11.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001, 2023-07-02T14:07:11.162 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.162 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.162 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.162 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.163 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.163 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 237 2023-07-02T14:07:11.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.163 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.163 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.163 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:07:11.163 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.164 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291748+0000", 2023-07-02T14:07:11.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:11.164 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.164 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.164 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291747+0000", 2023-07-02T14:07:11.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3188000000000003e-05 2023-07-02T14:07:11.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291840+0000", 2023-07-02T14:07:11.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1360000000000003e-06 2023-07-02T14:07:11.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291849+0000", 2023-07-02T14:07:11.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.608e-06 2023-07-02T14:07:11.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.291853+0000", 2023-07-02T14:07:11.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.019913888000000001 2023-07-02T14:07:11.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311767+0000", 2023-07-02T14:07:11.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6749000000000007e-05 2023-07-02T14:07:11.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.311844+0000", 2023-07-02T14:07:11.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015639500000000001 2023-07-02T14:07:11.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:11.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.312000+0000", 2023-07-02T14:07:11.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058616550000000003 2023-07-02T14:07:11.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317862+0000", 2023-07-02T14:07:11.169 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3245999999999999e-05 2023-07-02T14:07:11.169 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317905+0000", 2023-07-02T14:07:11.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6220999999999998e-05 2023-07-02T14:07:11.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317931+0000", 2023-07-02T14:07:11.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0545999999999999e-05 2023-07-02T14:07:11.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.317952+0000", 2023-07-02T14:07:11.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026204028000000001 2023-07-02T14:07:11.171 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.171 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.171 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.172 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:239 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2031616~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:11.172 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:07:11.172 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 271.81455507599998, 2023-07-02T14:07:11.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999, 2023-07-02T14:07:11.172 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.172 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.173 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.173 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.173 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.173 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 239 2023-07-02T14:07:11.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.173 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:07:11.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.174 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.174 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292087+0000", 2023-07-02T14:07:11.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:11.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.175 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292086+0000", 2023-07-02T14:07:11.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015478000000000001 2023-07-02T14:07:11.175 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.175 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.175 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292241+0000", 2023-07-02T14:07:11.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5789999999999993e-06 2023-07-02T14:07:11.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292251+0000", 2023-07-02T14:07:11.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.707e-06 2023-07-02T14:07:11.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292256+0000", 2023-07-02T14:07:11.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022369354000000001 2023-07-02T14:07:11.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314625+0000", 2023-07-02T14:07:11.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2325999999999997e-05 2023-07-02T14:07:11.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.314717+0000", 2023-07-02T14:07:11.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001224939 2023-07-02T14:07:11.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:11.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315942+0000", 2023-07-02T14:07:11.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076557439999999999 2023-07-02T14:07:11.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323598+0000", 2023-07-02T14:07:11.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017196609999999999 2023-07-02T14:07:11.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325318+0000", 2023-07-02T14:07:11.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7695000000000003e-05 2023-07-02T14:07:11.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325365+0000", 2023-07-02T14:07:11.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8772000000000001e-05 2023-07-02T14:07:11.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325404+0000", 2023-07-02T14:07:11.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033317000999999999 2023-07-02T14:07:11.182 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.182 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.182 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.182 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:240 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:11.182 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:07:11.182 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 271.81436902299998, 2023-07-02T14:07:11.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936, 2023-07-02T14:07:11.183 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.183 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.183 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.183 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.183 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.183 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 240 2023-07-02T14:07:11.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.184 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:07:11.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:07:11.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:11.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292273+0000", 2023-07-02T14:07:11.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013703599999999999 2023-07-02T14:07:11.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.186 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.186 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292410+0000", 2023-07-02T14:07:11.186 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3719999999999994e-06 2023-07-02T14:07:11.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292419+0000", 2023-07-02T14:07:11.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7089999999999999e-06 2023-07-02T14:07:11.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.187 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.187 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.187 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292425+0000", 2023-07-02T14:07:11.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024068678999999999 2023-07-02T14:07:11.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316493+0000", 2023-07-02T14:07:11.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0166999999999993e-05 2023-07-02T14:07:11.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316573+0000", 2023-07-02T14:07:11.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015840299999999999 2023-07-02T14:07:11.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:11.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.316732+0000", 2023-07-02T14:07:11.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068721240000000003 2023-07-02T14:07:11.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323604+0000", 2023-07-02T14:07:11.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018392440000000001 2023-07-02T14:07:11.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325443+0000", 2023-07-02T14:07:11.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1959000000000003e-05 2023-07-02T14:07:11.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325485+0000", 2023-07-02T14:07:11.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6837e-05 2023-07-02T14:07:11.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325522+0000", 2023-07-02T14:07:11.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033248936 2023-07-02T14:07:11.192 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.192 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.192 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.193 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:241 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2162688~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:11.193 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:07:11.193 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 271.81421122099999, 2023-07-02T14:07:11.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002, 2023-07-02T14:07:11.193 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.193 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.194 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.194 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.194 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.194 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 241 2023-07-02T14:07:11.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.194 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:07:11.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292431+0000", 2023-07-02T14:07:11.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:11.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292430+0000", 2023-07-02T14:07:11.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000169271 2023-07-02T14:07:11.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292599+0000", 2023-07-02T14:07:11.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0340000000000002e-06 2023-07-02T14:07:11.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292608+0000", 2023-07-02T14:07:11.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3119999999999999e-06 2023-07-02T14:07:11.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292615+0000", 2023-07-02T14:07:11.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025417309999999999 2023-07-02T14:07:11.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.198 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.198 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318032+0000", 2023-07-02T14:07:11.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5841000000000003e-05 2023-07-02T14:07:11.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.199 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.199 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318088+0000", 2023-07-02T14:07:11.199 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000156126 2023-07-02T14:07:11.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.199 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:11.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318244+0000", 2023-07-02T14:07:11.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059101960000000004 2023-07-02T14:07:11.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324154+0000", 2023-07-02T14:07:11.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014104479999999999 2023-07-02T14:07:11.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325565+0000", 2023-07-02T14:07:11.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2027000000000003e-05 2023-07-02T14:07:11.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325607+0000", 2023-07-02T14:07:11.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8157000000000001e-05 2023-07-02T14:07:11.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325645+0000", 2023-07-02T14:07:11.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.033213945000000002 2023-07-02T14:07:11.202 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.203 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.203 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.203 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:242 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2228224~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:11.203 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:07:11.203 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 271.81401943899999, 2023-07-02T14:07:11.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001, 2023-07-02T14:07:11.204 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.204 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.204 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.204 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.204 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.204 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 242 2023-07-02T14:07:11.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.205 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:07:11.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292623+0000", 2023-07-02T14:07:11.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:11.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292622+0000", 2023-07-02T14:07:11.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015593599999999999 2023-07-02T14:07:11.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292778+0000", 2023-07-02T14:07:11.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7390000000000005e-06 2023-07-02T14:07:11.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292787+0000", 2023-07-02T14:07:11.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4129999999999997e-06 2023-07-02T14:07:11.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292792+0000", 2023-07-02T14:07:11.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.025911598000000001 2023-07-02T14:07:11.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318704+0000", 2023-07-02T14:07:11.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3715999999999997e-05 2023-07-02T14:07:11.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318777+0000", 2023-07-02T14:07:11.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000155298 2023-07-02T14:07:11.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.210 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:11.210 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.318933+0000", 2023-07-02T14:07:11.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067515400000000003 2023-07-02T14:07:11.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325684+0000", 2023-07-02T14:07:11.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004086264 2023-07-02T14:07:11.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.211 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.211 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.211 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329770+0000", 2023-07-02T14:07:11.211 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00016418800000000001 2023-07-02T14:07:11.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329935+0000", 2023-07-02T14:07:11.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3968999999999999e-05 2023-07-02T14:07:11.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329979+0000", 2023-07-02T14:07:11.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037355881000000001 2023-07-02T14:07:11.213 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.213 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.213 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.213 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:243 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2293760~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:11.214 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:07:11.214 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 271.81384377900002, 2023-07-02T14:07:11.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112, 2023-07-02T14:07:11.214 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.214 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.214 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.214 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.214 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.215 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 243 2023-07-02T14:07:11.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.215 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:07:11.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:07:11.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:11.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292798+0000", 2023-07-02T14:07:11.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1550000000000004e-05 2023-07-02T14:07:11.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292869+0000", 2023-07-02T14:07:11.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6009999999999995e-06 2023-07-02T14:07:11.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292878+0000", 2023-07-02T14:07:11.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6389999999999998e-06 2023-07-02T14:07:11.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292886+0000", 2023-07-02T14:07:11.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027764021 2023-07-02T14:07:11.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320650+0000", 2023-07-02T14:07:11.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104876 2023-07-02T14:07:11.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320755+0000", 2023-07-02T14:07:11.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015257199999999999 2023-07-02T14:07:11.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:11.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.320908+0000", 2023-07-02T14:07:11.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004813749 2023-07-02T14:07:11.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.221 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.325722+0000", 2023-07-02T14:07:11.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042661189999999996 2023-07-02T14:07:11.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329988+0000", 2023-07-02T14:07:11.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1004999999999998e-05 2023-07-02T14:07:11.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.222 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.222 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330029+0000", 2023-07-02T14:07:11.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8733999999999999e-05 2023-07-02T14:07:11.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.223 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.330067+0000", 2023-07-02T14:07:11.223 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.037269112 2023-07-02T14:07:11.223 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.223 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.223 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.224 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:244 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2359296~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:11.224 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:07:11.224 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 271.81374959099998, 2023-07-02T14:07:11.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002, 2023-07-02T14:07:11.224 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.224 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.225 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.225 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.225 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.225 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 244 2023-07-02T14:07:11.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.225 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:07:11.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292893+0000", 2023-07-02T14:07:11.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:11.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.292892+0000", 2023-07-02T14:07:11.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000147918 2023-07-02T14:07:11.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293040+0000", 2023-07-02T14:07:11.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1494000000000001e-05 2023-07-02T14:07:11.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293061+0000", 2023-07-02T14:07:11.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1424999999999999e-05 2023-07-02T14:07:11.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293083+0000", 2023-07-02T14:07:11.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028524966999999998 2023-07-02T14:07:11.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321608+0000", 2023-07-02T14:07:11.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9184000000000006e-05 2023-07-02T14:07:11.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.230 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321687+0000", 2023-07-02T14:07:11.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015631999999999999 2023-07-02T14:07:11.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:11.231 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.321843+0000", 2023-07-02T14:07:11.231 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049091129999999997 2023-07-02T14:07:11.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.231 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326752+0000", 2023-07-02T14:07:11.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076030230000000004 2023-07-02T14:07:11.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334355+0000", 2023-07-02T14:07:11.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7737000000000001e-05 2023-07-02T14:07:11.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334393+0000", 2023-07-02T14:07:11.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4912000000000001e-05 2023-07-02T14:07:11.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334418+0000", 2023-07-02T14:07:11.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041525403000000002 2023-07-02T14:07:11.234 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.234 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.234 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.234 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.234 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:245 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:11.234 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:07:11.235 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 271.81354620500002, 2023-07-02T14:07:11.235 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001, 2023-07-02T14:07:11.235 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.235 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.235 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.235 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.235 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.236 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 245 2023-07-02T14:07:11.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.236 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:07:11.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293096+0000", 2023-07-02T14:07:11.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:11.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293095+0000", 2023-07-02T14:07:11.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000133447 2023-07-02T14:07:11.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293229+0000", 2023-07-02T14:07:11.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1460000000000002e-06 2023-07-02T14:07:11.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293238+0000", 2023-07-02T14:07:11.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.844e-06 2023-07-02T14:07:11.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293242+0000", 2023-07-02T14:07:11.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.029253593000000001 2023-07-02T14:07:11.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.240 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322495+0000", 2023-07-02T14:07:11.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9577000000000001e-05 2023-07-02T14:07:11.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322575+0000", 2023-07-02T14:07:11.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014791399999999999 2023-07-02T14:07:11.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:11.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.322723+0000", 2023-07-02T14:07:11.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040665340000000001 2023-07-02T14:07:11.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326789+0000", 2023-07-02T14:07:11.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076357680000000002 2023-07-02T14:07:11.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334425+0000", 2023-07-02T14:07:11.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2996000000000001e-05 2023-07-02T14:07:11.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334458+0000", 2023-07-02T14:07:11.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4010000000000001e-05 2023-07-02T14:07:11.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334492+0000", 2023-07-02T14:07:11.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041396107000000001 2023-07-02T14:07:11.244 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.244 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.244 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.245 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:246 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:11.245 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:07:11.245 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 271.81339554900001, 2023-07-02T14:07:11.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999, 2023-07-02T14:07:11.245 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.245 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.246 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.246 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.246 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.246 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 246 2023-07-02T14:07:11.246 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.246 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.246 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.246 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:07:11.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293247+0000", 2023-07-02T14:07:11.247 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:11.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293246+0000", 2023-07-02T14:07:11.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7160000000000002e-05 2023-07-02T14:07:11.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293303+0000", 2023-07-02T14:07:11.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0237999999999999e-05 2023-07-02T14:07:11.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293313+0000", 2023-07-02T14:07:11.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.864e-06 2023-07-02T14:07:11.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293316+0000", 2023-07-02T14:07:11.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030315208999999999 2023-07-02T14:07:11.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323631+0000", 2023-07-02T14:07:11.251 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4054999999999995e-05 2023-07-02T14:07:11.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.251 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.251 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323705+0000", 2023-07-02T14:07:11.251 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000148164 2023-07-02T14:07:11.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:11.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.323854+0000", 2023-07-02T14:07:11.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031796160000000001 2023-07-02T14:07:11.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.252 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327033+0000", 2023-07-02T14:07:11.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074684879999999997 2023-07-02T14:07:11.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.253 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334502+0000", 2023-07-02T14:07:11.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3515e-05 2023-07-02T14:07:11.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334535+0000", 2023-07-02T14:07:11.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7885000000000001e-05 2023-07-02T14:07:11.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334573+0000", 2023-07-02T14:07:11.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041326584999999999 2023-07-02T14:07:11.255 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.255 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.255 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.255 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:247 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:11.255 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:07:11.255 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 271.81332284600001, 2023-07-02T14:07:11.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998, 2023-07-02T14:07:11.256 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.256 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.256 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.256 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.256 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.256 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 247 2023-07-02T14:07:11.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.257 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:07:11.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.257 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.257 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:07:11.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:11.258 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.258 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.258 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.258 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293319+0000", 2023-07-02T14:07:11.258 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7052999999999994e-05 2023-07-02T14:07:11.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.259 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293416+0000", 2023-07-02T14:07:11.259 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3107e-05 2023-07-02T14:07:11.259 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.259 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.259 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293429+0000", 2023-07-02T14:07:11.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.641e-06 2023-07-02T14:07:11.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.260 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.260 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.260 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293433+0000", 2023-07-02T14:07:11.260 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.030741177000000001 2023-07-02T14:07:11.260 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.261 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.261 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324174+0000", 2023-07-02T14:07:11.261 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3402999999999997e-05 2023-07-02T14:07:11.261 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.261 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.261 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.261 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324227+0000", 2023-07-02T14:07:11.262 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000152285 2023-07-02T14:07:11.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.262 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.262 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:11.262 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324380+0000", 2023-07-02T14:07:11.262 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026880789999999999 2023-07-02T14:07:11.262 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.263 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.327068+0000", 2023-07-02T14:07:11.263 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075133730000000003 2023-07-02T14:07:11.263 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.263 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.263 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.263 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334581+0000", 2023-07-02T14:07:11.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7907000000000001e-05 2023-07-02T14:07:11.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.264 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.264 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334629+0000", 2023-07-02T14:07:11.264 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0893999999999998e-05 2023-07-02T14:07:11.264 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.264 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.265 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.265 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334670+0000", 2023-07-02T14:07:11.265 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041350553999999998 2023-07-02T14:07:11.265 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.265 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.265 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.265 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.265 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.266 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:248 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [set-alloc-hint object_size 4194304 write_size 4194304,write 327680~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:11.266 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:07:11.266 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 271.81320585600002, 2023-07-02T14:07:11.266 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244, 2023-07-02T14:07:11.266 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.266 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.266 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.267 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.267 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.267 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 248 2023-07-02T14:07:11.267 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.267 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.267 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.267 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.267 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:07:11.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.268 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.268 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.268 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:07:11.268 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:11.268 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293436+0000", 2023-07-02T14:07:11.269 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000125043 2023-07-02T14:07:11.269 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.269 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.269 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.269 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293561+0000", 2023-07-02T14:07:11.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5809999999999999e-06 2023-07-02T14:07:11.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.270 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.270 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293568+0000", 2023-07-02T14:07:11.270 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000001e-06 2023-07-02T14:07:11.270 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.270 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.271 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293572+0000", 2023-07-02T14:07:11.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.031151218000000001 2023-07-02T14:07:11.271 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.271 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.271 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.271 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324723+0000", 2023-07-02T14:07:11.271 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000104412 2023-07-02T14:07:11.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.272 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324827+0000", 2023-07-02T14:07:11.272 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015374099999999999 2023-07-02T14:07:11.272 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.272 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.272 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:11.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.324981+0000", 2023-07-02T14:07:11.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004215547 2023-07-02T14:07:11.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.273 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.273 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.273 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.329196+0000", 2023-07-02T14:07:11.273 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054811249999999999 2023-07-02T14:07:11.273 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334678+0000", 2023-07-02T14:07:11.274 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9750999999999999e-05 2023-07-02T14:07:11.274 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.274 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.274 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.274 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334727+0000", 2023-07-02T14:07:11.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7197999999999999e-05 2023-07-02T14:07:11.275 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.275 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.275 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.275 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334765+0000", 2023-07-02T14:07:11.275 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.041328244 2023-07-02T14:07:11.275 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.275 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.276 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.276 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.276 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.276 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:252 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2490368~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:11.276 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:07:11.276 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 271.813056405, 2023-07-02T14:07:11.276 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003, 2023-07-02T14:07:11.277 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.277 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.277 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.277 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.277 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.277 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 252 2023-07-02T14:07:11.277 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.277 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:07:11.278 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.278 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.278 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.278 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.278 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293586+0000", 2023-07-02T14:07:11.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:11.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.279 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.279 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293585+0000", 2023-07-02T14:07:11.279 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014862400000000001 2023-07-02T14:07:11.279 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.279 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293734+0000", 2023-07-02T14:07:11.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3510000000000008e-06 2023-07-02T14:07:11.280 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.280 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.280 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.280 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293743+0000", 2023-07-02T14:07:11.280 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0998e-05 2023-07-02T14:07:11.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.281 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.281 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.293754+0000", 2023-07-02T14:07:11.281 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.032377296 2023-07-02T14:07:11.281 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.281 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326131+0000", 2023-07-02T14:07:11.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4837000000000003e-05 2023-07-02T14:07:11.282 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.282 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.282 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.282 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326196+0000", 2023-07-02T14:07:11.282 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015642799999999999 2023-07-02T14:07:11.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:11.283 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.326353+0000", 2023-07-02T14:07:11.283 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079942199999999998 2023-07-02T14:07:11.283 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.283 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.283 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334347+0000", 2023-07-02T14:07:11.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00042679300000000002 2023-07-02T14:07:11.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.284 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.284 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.284 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.334774+0000", 2023-07-02T14:07:11.284 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00087257600000000002 2023-07-02T14:07:11.284 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335646+0000", 2023-07-02T14:07:11.285 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8000999999999997e-05 2023-07-02T14:07:11.285 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.285 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.285 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.285 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.335694+0000", 2023-07-02T14:07:11.286 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.042108429000000003 2023-07-02T14:07:11.286 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.286 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.286 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.286 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.286 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.286 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:410 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 851968~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:11.287 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:07:11.287 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 271.79158043199999, 2023-07-02T14:07:11.287 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537, 2023-07-02T14:07:11.287 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.287 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.287 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.287 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.287 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.288 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:07:11.288 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.288 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.288 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.288 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.288 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:07:11.288 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.289 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315062+0000", 2023-07-02T14:07:11.289 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:11.289 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.289 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.289 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315061+0000", 2023-07-02T14:07:11.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107877 2023-07-02T14:07:11.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.290 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.290 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.290 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315169+0000", 2023-07-02T14:07:11.290 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1212999999999999e-05 2023-07-02T14:07:11.290 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315180+0000", 2023-07-02T14:07:11.291 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4279999999999999e-06 2023-07-02T14:07:11.291 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.291 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.291 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.291 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315187+0000", 2023-07-02T14:07:11.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023715681999999998 2023-07-02T14:07:11.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.292 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.292 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338903+0000", 2023-07-02T14:07:11.292 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4929e-05 2023-07-02T14:07:11.292 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.292 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.338978+0000", 2023-07-02T14:07:11.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015764800000000001 2023-07-02T14:07:11.293 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.293 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.293 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:07:11.293 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339136+0000", 2023-07-02T14:07:11.293 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0014048730000000001 2023-07-02T14:07:11.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.294 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.340541+0000", 2023-07-02T14:07:11.294 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001785322 2023-07-02T14:07:11.294 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.294 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.294 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342326+0000", 2023-07-02T14:07:11.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8128000000000003e-05 2023-07-02T14:07:11.295 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.295 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.295 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.295 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342384+0000", 2023-07-02T14:07:11.295 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3072999999999999e-05 2023-07-02T14:07:11.296 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.296 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.296 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.296 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342427+0000", 2023-07-02T14:07:11.296 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.02736537 2023-07-02T14:07:11.296 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.296 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.296 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.297 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.297 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.297 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:411 2.4 2:2cfb5cf7:::rbd_data.125de1449605.0000000000000107:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~65536] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:11.297 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:07:11.297 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 271.79144789600002, 2023-07-02T14:07:11.297 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999, 2023-07-02T14:07:11.297 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.297 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.298 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.298 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.298 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.298 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 411 2023-07-02T14:07:11.298 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.298 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.298 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.298 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:07:11.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.299 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.299 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.299 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315194+0000", 2023-07-02T14:07:11.299 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:11.299 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315193+0000", 2023-07-02T14:07:11.300 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00060660100000000002 2023-07-02T14:07:11.300 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.300 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.300 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.300 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315800+0000", 2023-07-02T14:07:11.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.7089999999999997e-06 2023-07-02T14:07:11.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.301 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.301 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315810+0000", 2023-07-02T14:07:11.301 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:07:11.301 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.301 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.315816+0000", 2023-07-02T14:07:11.302 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023777269 2023-07-02T14:07:11.302 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.302 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.302 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.302 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339593+0000", 2023-07-02T14:07:11.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4329e-05 2023-07-02T14:07:11.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.303 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.303 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339657+0000", 2023-07-02T14:07:11.303 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015517300000000001 2023-07-02T14:07:11.303 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.303 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 0", 2023-07-02T14:07:11.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.339812+0000", 2023-07-02T14:07:11.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026288560000000002 2023-07-02T14:07:11.304 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.304 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.304 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.304 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342441+0000", 2023-07-02T14:07:11.304 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00052620200000000003 2023-07-02T14:07:11.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.305 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.305 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.342967+0000", 2023-07-02T14:07:11.305 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1508999999999998e-05 2023-07-02T14:07:11.305 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.305 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343009+0000", 2023-07-02T14:07:11.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6213e-05 2023-07-02T14:07:11.306 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.306 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.306 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.306 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.343045+0000", 2023-07-02T14:07:11.306 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027850864999999999 2023-07-02T14:07:11.307 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.307 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.307 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.307 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.307 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.307 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:481 2.3 2:d6c371cf:::rbd_data.125de1449605.000000000000010b:head [write 2752512~1441792] snapc 0=[] ondisk+write+known_if_redirected e40)", 2023-07-02T14:07:11.307 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:07:11.307 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 271.73011123499998, 2023-07-02T14:07:11.308 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054, 2023-07-02T14:07:11.308 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.308 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.308 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.308 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.308 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.308 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 481 2023-07-02T14:07:11.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.309 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.309 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.309 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:07:11.309 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.309 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.309 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376531+0000", 2023-07-02T14:07:11.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:11.310 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.310 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.310 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.310 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.376530+0000", 2023-07-02T14:07:11.310 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073298820000000002 2023-07-02T14:07:11.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.311 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383860+0000", 2023-07-02T14:07:11.311 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7178000000000001e-05 2023-07-02T14:07:11.311 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.311 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.311 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383877+0000", 2023-07-02T14:07:11.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6080000000000003e-06 2023-07-02T14:07:11.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.312 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.312 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.312 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383882+0000", 2023-07-02T14:07:11.312 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0275000000000001e-05 2023-07-02T14:07:11.312 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.383943+0000", 2023-07-02T14:07:11.313 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2384000000000004e-05 2023-07-02T14:07:11.313 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.313 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.313 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.313 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384005+0000", 2023-07-02T14:07:11.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017814 2023-07-02T14:07:11.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.314 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 2", 2023-07-02T14:07:11.314 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.384183+0000", 2023-07-02T14:07:11.314 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025742730000000002 2023-07-02T14:07:11.314 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.314 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.386757+0000", 2023-07-02T14:07:11.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015919039999999999 2023-07-02T14:07:11.315 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.315 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.315 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.315 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402677+0000", 2023-07-02T14:07:11.315 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7353e-05 2023-07-02T14:07:11.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.316 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.316 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402714+0000", 2023-07-02T14:07:11.316 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.2433999999999999e-05 2023-07-02T14:07:11.316 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.316 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.317 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.317 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:02:39.402736+0000", 2023-07-02T14:07:11.317 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0262054 2023-07-02T14:07:11.317 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.317 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.317 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.317 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.317 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.318 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:11.319 INFO:tasks.workunit.client.0.smithi163.stderr:+ kill -0 78885 2023-07-02T14:07:11.319 INFO:tasks.workunit.client.0.smithi163.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh: line 65: kill: (78885) - No such process 2023-07-02T14:07:11.319 INFO:tasks.workunit.client.0.smithi163.stderr:+ sudo umount b3fbb028-e267-4d25-a750-93ba18a2316a-mnt 2023-07-02T14:07:11.319 INFO:tasks.workunit.client.0.smithi163.stderr:+ sudo rbd device unmap -t nbd /dev/nbd0 2023-07-02T14:07:11.320 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:07:11.320 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:11.318+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:11.321 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:11.323 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:07:11.334 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2023-07-02T14:07:10.378845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11892805145,"num_objects":3557,"num_object_clones":2995,"num_object_copies":7114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":354,"num_objects_misplaced":401,"num_objects_unfound":0,"num_objects_dirty":3557,"num_whiteouts":0,"num_read":88256,"num_read_kb":15199597,"num_write":27213,"num_write_kb":11540672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2862,"num_bytes_recovered":9703530998,"num_keys_recovered":186,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26888,"ondisk_log_size":26888,"up":18,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":21655396,"kb_used_data":21313668,"kb_used_omap":229,"kb_used_meta":341466,"kb_avail":355831964,"statfs":{"total":386547056640,"available":364371931136,"internally_reserved":0,"allocated":21825196032,"data_stored":25424386923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":349661406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22749346,"num_objects":16,"num_object_clones":15,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":136,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1953,"num_read_kb":671410,"num_write":89,"num_write_kb":22250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":922657902,"num_keys_recovered":8,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.088950"},"pg_stats":[{"pgid":"2.7","version":"212'3647","reported_seq":9311,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-07-02T14:07:05.528711+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:07:05.528711+0000","last_peered":"2023-07-02T14:07:05.528711+0000","last_clean":"2023-07-02T14:07:05.528711+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:07:05.528711+0000","last_undegraded":"2023-07-02T14:07:05.528711+0000","last_fullsized":"2023-07-02T14:07:05.528711+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3647,"ondisk_log_size":3647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1570191828,"num_objects":466,"num_object_clones":394,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":57412,"num_read_kb":1894335,"num_write":3056,"num_write_kb":1525929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"215'3058","reported_seq":5145,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:10.324843+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:07:10.324843+0000","last_peered":"2023-07-02T14:07:10.324843+0000","last_clean":"2023-07-02T14:07:10.324843+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:07:10.324843+0000","last_undegraded":"2023-07-02T14:07:10.324843+0000","last_fullsized":"2023-07-02T14:07:10.324843+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3058,"ondisk_log_size":3058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535489458,"num_objects":482,"num_object_clones":414,"num_object_copies":964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":0,"num_read":1926,"num_read_kb":2091895,"num_write":2644,"num_write_kb":1489957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"210'3193","reported_seq":3825,"reported_epoch":213,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-07-02T14:07:08.095386+0000","last_change":"2023-07-02T14:07:07.848445+0000","last_active":"2023-07-02T14:07:08.095386+0000","last_peered":"2023-07-02T14:07:08.095386+0000","last_clean":"2023-07-02T14:07:01.043240+0000","last_became_active":"2023-07-02T14:07:04.030791+0000","last_became_peered":"2023-07-02T14:07:04.030791+0000","last_unstale":"2023-07-02T14:07:08.095386+0000","last_undegraded":"2023-07-02T14:07:03.107328+0000","last_fullsized":"2023-07-02T14:07:03.101623+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3193,"ondisk_log_size":3193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1516729292,"num_objects":421,"num_object_clones":355,"num_object_copies":842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":401,"num_objects_unfound":0,"num_objects_dirty":421,"num_whiteouts":0,"num_read":572,"num_read_kb":1826967,"num_write":2838,"num_write_kb":1467491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":909742242,"num_keys_recovered":8,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":401},{"shards":"3","objects":18}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"215'6476","reported_seq":8964,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:10.334213+0000","last_change":"2023-07-02T14:06:29.572710+0000","last_active":"2023-07-02T14:07:10.334213+0000","last_peered":"2023-07-02T14:07:10.334213+0000","last_clean":"2023-07-02T14:07:10.334213+0000","last_became_active":"2023-07-02T14:06:29.569824+0000","last_became_peered":"2023-07-02T14:06:29.569824+0000","last_unstale":"2023-07-02T14:07:10.334213+0000","last_undegraded":"2023-07-02T14:07:10.334213+0000","last_fullsized":"2023-07-02T14:07:10.334213+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":6476,"ondisk_log_size":6476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672852930,"num_objects":488,"num_object_clones":413,"num_object_copies":976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":22953,"num_read_kb":2152050,"num_write":9580,"num_write_kb":1632373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":113,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-07-02T14:07:05.358697+0000","last_change":"2023-07-02T14:07:02.103009+0000","last_active":"2023-07-02T14:07:05.358697+0000","last_peered":"2023-07-02T14:07:05.358697+0000","last_clean":"2023-07-02T14:07:05.358697+0000","last_became_active":"2023-07-02T14:07:02.102654+0000","last_became_peered":"2023-07-02T14:07:02.102654+0000","last_unstale":"2023-07-02T14:07:05.358697+0000","last_undegraded":"2023-07-02T14:07:05.358697+0000","last_fullsized":"2023-07-02T14:07:05.358697+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"215'2964","reported_seq":4463,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:10.302466+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:07:10.302466+0000","last_peered":"2023-07-02T14:07:10.302466+0000","last_clean":"2023-07-02T14:07:10.302466+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:07:10.302466+0000","last_undegraded":"2023-07-02T14:07:10.302466+0000","last_fullsized":"2023-07-02T14:07:10.302466+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2964,"ondisk_log_size":2964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1658680446,"num_objects":491,"num_object_clones":414,"num_object_copies":982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":0,"num_read":1509,"num_read_kb":2028229,"num_write":2550,"num_write_kb":1608728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"215'2867","reported_seq":3734,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:10.282420+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:07:10.282420+0000","last_peered":"2023-07-02T14:07:10.282420+0000","last_clean":"2023-07-02T14:07:10.282420+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:07:10.282420+0000","last_undegraded":"2023-07-02T14:07:10.282420+0000","last_fullsized":"2023-07-02T14:07:10.282420+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2867,"ondisk_log_size":2867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1564710484,"num_objects":455,"num_object_clones":382,"num_object_copies":910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":455,"num_whiteouts":0,"num_read":949,"num_read_kb":1945167,"num_write":2485,"num_write_kb":1518698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"215'2325","reported_seq":5128,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:10.305571+0000","last_change":"2023-07-02T14:07:09.383558+0000","last_active":"2023-07-02T14:07:10.305571+0000","last_peered":"2023-07-02T14:07:10.305571+0000","last_clean":"2023-07-02T14:07:10.305571+0000","last_became_active":"2023-07-02T14:07:09.382190+0000","last_became_peered":"2023-07-02T14:07:09.382190+0000","last_unstale":"2023-07-02T14:07:10.305571+0000","last_undegraded":"2023-07-02T14:07:10.305571+0000","last_fullsized":"2023-07-02T14:07:10.305571+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2325,"ondisk_log_size":2325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1230505737,"num_objects":402,"num_object_clones":325,"num_object_copies":804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":1898,"num_read_kb":1730263,"num_write":2000,"num_write_kb":1186689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":1578968698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"215'2358","reported_seq":3456,"reported_epoch":215,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-07-02T14:07:10.279927+0000","last_change":"2023-07-02T14:07:04.022783+0000","last_active":"2023-07-02T14:07:10.279927+0000","last_peered":"2023-07-02T14:07:10.279927+0000","last_clean":"2023-07-02T14:07:01.026611+0000","last_became_active":"2023-07-02T14:07:04.018152+0000","last_became_peered":"2023-07-02T14:07:04.018152+0000","last_unstale":"2023-07-02T14:07:10.279927+0000","last_undegraded":"2023-07-02T14:07:03.110294+0000","last_fullsized":"2023-07-02T14:07:03.103717+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2358,"ondisk_log_size":2358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1143644970,"num_objects":352,"num_object_clones":298,"num_object_copies":704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":1037,"num_read_kb":1530691,"num_write":2060,"num_write_kb":1110807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":271880192,"num_keys_recovered":4,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":346},{"shards":"1","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":11892805145,"num_objects":3557,"num_object_clones":2995,"num_object_copies":7114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":354,"num_objects_misplaced":401,"num_objects_unfound":0,"num_objects_dirty":3557,"num_whiteouts":0,"num_read":88256,"num_read_kb":15199597,"num_write":27213,"num_write_kb":11540672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2862,"num_bytes_recovered":9703530998,"num_keys_recovered":186,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25840590848,"data_stored":30051234943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":26888,"ondisk_log_size":26888,"up":16,"acting":14,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640323,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640323,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5235024,"kb_used_data":5143176,"kb_used_omap":0,"kb_used_meta":91840,"kb_avail":89136816,"statfs":{"total":96636764160,"available":91276099584,"internally_reserved":0,"allocated":5266612224,"data_stored":6072961790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":94044160},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.307}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44,"15min":0.46400000000000002},"min":{"1min":0.28399999999999997,"5min":0.215,"15min":0.215},"max":{"1min":0.71499999999999997,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.45000000000000001},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.32800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66800000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.39900000000000002}]},{"osd":3,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.38300000000000001},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45900000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.36799999999999999,"15min":0.36299999999999999},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.48799999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.56200000000000006}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.44900000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.433,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59099999999999997,"5min":1.026,"15min":1.026},"last":0.42899999999999999}]}]},{"osd":3,"up_from":133,"seq":571230650394,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4560208,"kb_used_data":4490568,"kb_used_omap":229,"kb_used_meta":69402,"kb_avail":89811632,"statfs":{"total":96636764160,"available":91967111168,"internally_reserved":0,"allocated":4598341632,"data_stored":5339881520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":71067870},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.443}]},{"osd":1,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.42299999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.504}]},{"osd":4,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.49099999999999999}]},{"osd":5,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.52000000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640323,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3192604,"kb_used_data":3138708,"kb_used_omap":0,"kb_used_meta":53888,"kb_avail":91179236,"statfs":{"total":96636764160,"available":93367537664,"internally_reserved":0,"allocated":3214036992,"data_stored":3691892043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55181312},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.28799999999999998}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.42499999999999999,"15min":0.374},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.376,"5min":0.33500000000000002,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.22500000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:07:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.91100000000000003,"5min":1.929,"15min":1.929},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.54800000000000004,"15min":0.55600000000000005},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93999999999999995,"5min":2,"15min":2},"last":0.35999999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.40899999999999997}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68400000000000005,"5min":1.071,"15min":1.071},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":0.432}]}]},{"osd":0,"up_from":0,"seq":712964571152,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":11,"seq":47244640323,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8667560,"kb_used_data":8541216,"kb_used_omap":0,"kb_used_meta":126336,"kb_avail":85704280,"statfs":{"total":96636764160,"available":87761182720,"internally_reserved":0,"allocated":8746205184,"data_stored":10319651570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":129368064},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.32600000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.247},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.224,"5min":0.184,"15min":0.184},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.33800000000000002}]},{"osd":3,"last update":"Sun Jul 2 14:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.49099999999999999,"15min":0.47299999999999998},"min":{"1min":0.30299999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.38600000000000001}]},{"osd":4,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71899999999999997,"5min":1.524,"15min":1.524},"last":0.51000000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.80300000000000005,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84399999999999997,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.42999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3981160448,"data_stored":4543561053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8745132032,"data_stored":10339602553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3211833344,"data_stored":3699260114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4595769344,"data_stored":5338870610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5306695680,"data_stored":6129940613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:07:11.335 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-07-02T14:07:11.420 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:11.419+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:11.511 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:11.512 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:07:11.512 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:07:11.512 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:11.520 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:11.519+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:11.525 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:07:11.620 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:11.619+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:11.673 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:11.674 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:07:11.674 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:07:11.674 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:07:11.674 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:11.687 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:07:11.701 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:11.701 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 358/7086 objects degraded (5.052%), 3 pgs degraded","count":3},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":334,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":216,"num_osds":6,"num_up_osds":6,"osd_up_since":1688306752,"num_in_osds":4,"osd_in_since":1688306821,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":2},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":3543,"data_bytes":11876343159,"bytes_used":20448686080,"bytes_avail":366098370560,"bytes_total":386547056640,"degraded_objects":358,"degraded_total":7086,"degraded_ratio":0.050522156364662708,"misplaced_objects":666,"misplaced_total":7086,"misplaced_ratio":0.093988145639288756,"recovering_objects_per_sec":3,"recovering_bytes_per_sec":9966631,"recovering_keys_per_sec":1,"num_objects_recovered":18,"num_bytes_recovered":49176576,"num_keys_recovered":8,"read_bytes_sec":16379029,"write_bytes_sec":1274676,"read_op_per_sec":331,"write_op_per_sec":3},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2023-07-02T14:06:52.287392+0000","services":{}},"progress_events":{"39f7aefe-a986-4f7d-8f18-834c65a827f0":{"message":"Rebalancing after osd.0 marked out (5s)\n [=========...................] (remaining: 10s)","progress":0.32440775632858276,"add_to_ceph_s":false},"c9df9a9d-66a6-47d3-b334-4774a23f2050":{"message":"Global Recovery Event (0s)\n [............................] ","progress":0,"add_to_ceph_s":true}}} 2023-07-02T14:07:11.714 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2023-07-02T14:07:11.715 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:07:11.733 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:11.732+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:11.822 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:11.821+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:11.834 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 78885 2023-07-02T14:07:11.834 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'Exported 65 incrementals' 2023-07-02T14:07:11.835 INFO:tasks.workunit.client.0.smithi163.stdout:Exported 65 incrementals 2023-07-02T14:07:11.836 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( snap_num < 30 )) 2023-07-02T14:07:11.836 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i = 1 )) 2023-07-02T14:07:11.836 INFO:tasks.workunit.client.0.smithi163.stderr:+ (( i <= snap_num )) 2023-07-02T14:07:11.837 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd import-diff b3fbb028-e267-4d25-a750-93ba18a2316a@snap1.diff b3fbb028-e267-4d25-a750-93ba18a2316a-dst 2023-07-02T14:07:11.851 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:11.851 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:07:11.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:07:11.852 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:07:11.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.853 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17354 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:07:11.853 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:07:11.853 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 88.542512149000004, 2023-07-02T14:07:11.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493, 2023-07-02T14:07:11.854 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.854 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.854 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.854 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.855 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.855 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17354 2023-07-02T14:07:11.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.855 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:07:11.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:07:11.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:11.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302603+0000", 2023-07-02T14:07:11.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7440000000000001e-06 2023-07-02T14:07:11.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302606+0000", 2023-07-02T14:07:11.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:07:11.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302613+0000", 2023-07-02T14:07:11.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8810000000000004e-06 2023-07-02T14:07:11.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302618+0000", 2023-07-02T14:07:11.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00028613100000000002 2023-07-02T14:07:11.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302904+0000", 2023-07-02T14:07:11.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5278999999999995e-05 2023-07-02T14:07:11.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302979+0000", 2023-07-02T14:07:11.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0096904750000000005 2023-07-02T14:07:11.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312669+0000", 2023-07-02T14:07:11.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0328000000000002e-05 2023-07-02T14:07:11.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312700+0000", 2023-07-02T14:07:11.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098019319999999993 2023-07-02T14:07:11.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322502+0000", 2023-07-02T14:07:11.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4669999999999998e-05 2023-07-02T14:07:11.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322536+0000", 2023-07-02T14:07:11.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010023839 2023-07-02T14:07:11.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332560+0000", 2023-07-02T14:07:11.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5559999999999998e-05 2023-07-02T14:07:11.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332596+0000", 2023-07-02T14:07:11.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090401319999999993 2023-07-02T14:07:11.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341636+0000", 2023-07-02T14:07:11.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5373000000000001e-05 2023-07-02T14:07:11.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341681+0000", 2023-07-02T14:07:11.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081994059999999994 2023-07-02T14:07:11.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349881+0000", 2023-07-02T14:07:11.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4319000000000001e-05 2023-07-02T14:07:11.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349915+0000", 2023-07-02T14:07:11.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0089388339999999997 2023-07-02T14:07:11.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358854+0000", 2023-07-02T14:07:11.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4319000000000005e-05 2023-07-02T14:07:11.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358928+0000", 2023-07-02T14:07:11.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080234690000000001 2023-07-02T14:07:11.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366952+0000", 2023-07-02T14:07:11.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2082999999999998e-05 2023-07-02T14:07:11.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.366984+0000", 2023-07-02T14:07:11.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075015790000000004 2023-07-02T14:07:11.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374485+0000", 2023-07-02T14:07:11.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4480999999999995e-05 2023-07-02T14:07:11.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374550+0000", 2023-07-02T14:07:11.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0091666620000000008 2023-07-02T14:07:11.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383716+0000", 2023-07-02T14:07:11.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9318000000000003e-05 2023-07-02T14:07:11.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383786+0000", 2023-07-02T14:07:11.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013434115 2023-07-02T14:07:11.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397220+0000", 2023-07-02T14:07:11.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4573999999999997e-05 2023-07-02T14:07:11.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397264+0000", 2023-07-02T14:07:11.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064824649999999998 2023-07-02T14:07:11.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403747+0000", 2023-07-02T14:07:11.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1087999999999993e-05 2023-07-02T14:07:11.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403828+0000", 2023-07-02T14:07:11.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072854449999999998 2023-07-02T14:07:11.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411113+0000", 2023-07-02T14:07:11.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5120000000000001e-05 2023-07-02T14:07:11.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411169+0000", 2023-07-02T14:07:11.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007211505 2023-07-02T14:07:11.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418380+0000", 2023-07-02T14:07:11.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00011289399999999999 2023-07-02T14:07:11.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.418493+0000", 2023-07-02T14:07:11.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00051217500000000002 2023-07-02T14:07:11.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.893 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:11.893 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.419005+0000", 2023-07-02T14:07:11.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00327393 2023-07-02T14:07:11.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.422279+0000", 2023-07-02T14:07:11.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041432350000000003 2023-07-02T14:07:11.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426422+0000", 2023-07-02T14:07:11.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7021000000000003e-05 2023-07-02T14:07:11.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426469+0000", 2023-07-02T14:07:11.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000190256 2023-07-02T14:07:11.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426660+0000", 2023-07-02T14:07:11.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.124056493 2023-07-02T14:07:11.896 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.896 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.896 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.897 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:17355 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e161)", 2023-07-02T14:07:11.897 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:07:11.897 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 88.542493488000005, 2023-07-02T14:07:11.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001, 2023-07-02T14:07:11.897 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.897 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.898 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.898 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.898 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.898 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 17355 2023-07-02T14:07:11.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.898 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:07:11.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302622+0000", 2023-07-02T14:07:11.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:11.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302621+0000", 2023-07-02T14:07:11.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2260000000000001e-06 2023-07-02T14:07:11.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302625+0000", 2023-07-02T14:07:11.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8599999999999998e-06 2023-07-02T14:07:11.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302630+0000", 2023-07-02T14:07:11.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0629999999999999e-06 2023-07-02T14:07:11.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302635+0000", 2023-07-02T14:07:11.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00035731199999999998 2023-07-02T14:07:11.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.302992+0000", 2023-07-02T14:07:11.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4164e-05 2023-07-02T14:07:11.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.303026+0000", 2023-07-02T14:07:11.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097115900000000008 2023-07-02T14:07:11.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312738+0000", 2023-07-02T14:07:11.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1701999999999998e-05 2023-07-02T14:07:11.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.312819+0000", 2023-07-02T14:07:11.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097363640000000008 2023-07-02T14:07:11.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322556+0000", 2023-07-02T14:07:11.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2517000000000001e-05 2023-07-02T14:07:11.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.322588+0000", 2023-07-02T14:07:11.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01002393 2023-07-02T14:07:11.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332612+0000", 2023-07-02T14:07:11.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2960999999999998e-05 2023-07-02T14:07:11.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.332645+0000", 2023-07-02T14:07:11.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090669819999999995 2023-07-02T14:07:11.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341712+0000", 2023-07-02T14:07:11.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9288000000000002e-05 2023-07-02T14:07:11.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.341811+0000", 2023-07-02T14:07:11.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081504460000000004 2023-07-02T14:07:11.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.349962+0000", 2023-07-02T14:07:11.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000107321 2023-07-02T14:07:11.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.350069+0000", 2023-07-02T14:07:11.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0088790999999999991 2023-07-02T14:07:11.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358948+0000", 2023-07-02T14:07:11.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1711000000000002e-05 2023-07-02T14:07:11.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.358980+0000", 2023-07-02T14:07:11.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080367579999999997 2023-07-02T14:07:11.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367017+0000", 2023-07-02T14:07:11.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6329000000000002e-05 2023-07-02T14:07:11.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.367073+0000", 2023-07-02T14:07:11.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075097300000000001 2023-07-02T14:07:11.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374583+0000", 2023-07-02T14:07:11.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5710999999999995e-05 2023-07-02T14:07:11.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.374648+0000", 2023-07-02T14:07:11.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.009165912 2023-07-02T14:07:11.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383814+0000", 2023-07-02T14:07:11.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3916000000000003e-05 2023-07-02T14:07:11.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.383878+0000", 2023-07-02T14:07:11.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013440532999999999 2023-07-02T14:07:11.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397319+0000", 2023-07-02T14:07:11.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7218999999999995e-05 2023-07-02T14:07:11.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.397386+0000", 2023-07-02T14:07:11.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065165279999999997 2023-07-02T14:07:11.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.403903+0000", 2023-07-02T14:07:11.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8152000000000005e-05 2023-07-02T14:07:11.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.404001+0000", 2023-07-02T14:07:11.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072185360000000002 2023-07-02T14:07:11.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411219+0000", 2023-07-02T14:07:11.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0284000000000005e-05 2023-07-02T14:07:11.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.411310+0000", 2023-07-02T14:07:11.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010021663 2023-07-02T14:07:11.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421331+0000", 2023-07-02T14:07:11.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0148000000000006e-05 2023-07-02T14:07:11.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.421411+0000", 2023-07-02T14:07:11.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053056170000000003 2023-07-02T14:07:11.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.922 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426717+0000", 2023-07-02T14:07:11.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8242000000000001e-05 2023-07-02T14:07:11.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.426795+0000", 2023-07-02T14:07:11.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00041308399999999998 2023-07-02T14:07:11.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:11.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.427208+0000", 2023-07-02T14:07:11.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028330849999999999 2023-07-02T14:07:11.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430041+0000", 2023-07-02T14:07:11.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00072642300000000002 2023-07-02T14:07:11.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430768+0000", 2023-07-02T14:07:11.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1847000000000002e-05 2023-07-02T14:07:11.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430800+0000", 2023-07-02T14:07:11.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9308999999999994e-05 2023-07-02T14:07:11.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:43.430899+0000", 2023-07-02T14:07:11.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12827714900000001 2023-07-02T14:07:11.927 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.927 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.927 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.927 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:410 2.5 2:bbaed619:::rbd_data.125de1449605.0000000000000109:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:07:11.927 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:07:11.928 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 75.394548223000001, 2023-07-02T14:07:11.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003, 2023-07-02T14:07:11.928 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.928 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:07:11.928 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.928 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:07:11.928 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:07:11.929 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:07:11.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.929 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:07:11.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:07:11.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:11.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:07:11.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.635e-06 2023-07-02T14:07:11.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450568+0000", 2023-07-02T14:07:11.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5970000000000002e-06 2023-07-02T14:07:11.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450573+0000", 2023-07-02T14:07:11.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3379999999999999e-06 2023-07-02T14:07:11.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450575+0000", 2023-07-02T14:07:11.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9458000000000001e-05 2023-07-02T14:07:11.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450615+0000", 2023-07-02T14:07:11.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0569999999999993e-06 2023-07-02T14:07:11.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:11.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450623+0000", 2023-07-02T14:07:11.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.63968709199999996 2023-07-02T14:07:11.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090310+0000", 2023-07-02T14:07:11.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020157399999999999 2023-07-02T14:07:11.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090512+0000", 2023-07-02T14:07:11.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027739190000000001 2023-07-02T14:07:11.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093285+0000", 2023-07-02T14:07:11.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003 2023-07-02T14:07:11.936 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.936 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.937 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.937 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:416 2.5 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:07:11.937 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:07:11.937 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 75.394518790000006, 2023-07-02T14:07:11.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002, 2023-07-02T14:07:11.938 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.938 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:07:11.938 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.938 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:07:11.938 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:07:11.938 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:07:11.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.939 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:07:11.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:07:11.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:11.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:07:11.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.34e-06 2023-07-02T14:07:11.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450599+0000", 2023-07-02T14:07:11.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1549999999999998e-06 2023-07-02T14:07:11.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450604+0000", 2023-07-02T14:07:11.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:07:11.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450607+0000", 2023-07-02T14:07:11.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5717999999999999e-05 2023-07-02T14:07:11.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450643+0000", 2023-07-02T14:07:11.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1619999999999999e-06 2023-07-02T14:07:11.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:11.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450648+0000", 2023-07-02T14:07:11.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64292291800000001 2023-07-02T14:07:11.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093571+0000", 2023-07-02T14:07:11.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115628 2023-07-02T14:07:11.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093687+0000", 2023-07-02T14:07:11.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002885552 2023-07-02T14:07:11.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.096572+0000", 2023-07-02T14:07:11.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002 2023-07-02T14:07:11.946 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.946 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.946 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.947 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18557 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:07:11.947 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:07:11.947 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 66.596146411000007, 2023-07-02T14:07:11.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13120991400000001, 2023-07-02T14:07:11.947 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.947 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.948 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.948 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.948 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.948 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18557 2023-07-02T14:07:11.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.948 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:07:11.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:07:11.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967296 2023-07-02T14:07:11.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248969+0000", 2023-07-02T14:07:11.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8870000000000001e-06 2023-07-02T14:07:11.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248971+0000", 2023-07-02T14:07:11.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9009999999999996e-06 2023-07-02T14:07:11.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248974+0000", 2023-07-02T14:07:11.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9809999999999997e-06 2023-07-02T14:07:11.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248978+0000", 2023-07-02T14:07:11.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00076989199999999999 2023-07-02T14:07:11.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249748+0000", 2023-07-02T14:07:11.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8245999999999999e-05 2023-07-02T14:07:11.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249787+0000", 2023-07-02T14:07:11.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066332969999999998 2023-07-02T14:07:11.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.954 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256420+0000", 2023-07-02T14:07:11.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2566e-05 2023-07-02T14:07:11.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256452+0000", 2023-07-02T14:07:11.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012364714000000001 2023-07-02T14:07:11.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268817+0000", 2023-07-02T14:07:11.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5113000000000001e-05 2023-07-02T14:07:11.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268852+0000", 2023-07-02T14:07:11.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010993899999999999 2023-07-02T14:07:11.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279846+0000", 2023-07-02T14:07:11.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2144999999999998e-05 2023-07-02T14:07:11.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279878+0000", 2023-07-02T14:07:11.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011778958000000001 2023-07-02T14:07:11.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291657+0000", 2023-07-02T14:07:11.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9592000000000001e-05 2023-07-02T14:07:11.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291687+0000", 2023-07-02T14:07:11.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081107220000000008 2023-07-02T14:07:11.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299798+0000", 2023-07-02T14:07:11.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0290999999999998e-05 2023-07-02T14:07:11.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299828+0000", 2023-07-02T14:07:11.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071584489999999999 2023-07-02T14:07:11.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.306986+0000", 2023-07-02T14:07:11.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0755999999999999e-05 2023-07-02T14:07:11.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307047+0000", 2023-07-02T14:07:11.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057110260000000001 2023-07-02T14:07:11.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312758+0000", 2023-07-02T14:07:11.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5145999999999999e-05 2023-07-02T14:07:11.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312793+0000", 2023-07-02T14:07:11.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004758361 2023-07-02T14:07:11.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317552+0000", 2023-07-02T14:07:11.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9190000000000002e-05 2023-07-02T14:07:11.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317601+0000", 2023-07-02T14:07:11.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050329420000000003 2023-07-02T14:07:11.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322634+0000", 2023-07-02T14:07:11.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5737999999999998e-05 2023-07-02T14:07:11.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322719+0000", 2023-07-02T14:07:11.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064995679999999998 2023-07-02T14:07:11.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329219+0000", 2023-07-02T14:07:11.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0908000000000003e-05 2023-07-02T14:07:11.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329250+0000", 2023-07-02T14:07:11.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062072029999999997 2023-07-02T14:07:11.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335457+0000", 2023-07-02T14:07:11.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9471999999999998e-05 2023-07-02T14:07:11.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335487+0000", 2023-07-02T14:07:11.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097780239999999997 2023-07-02T14:07:11.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345265+0000", 2023-07-02T14:07:11.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0982999999999994e-05 2023-07-02T14:07:11.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345336+0000", 2023-07-02T14:07:11.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046378799999999996 2023-07-02T14:07:11.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.349973+0000", 2023-07-02T14:07:11.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1069999999999999e-05 2023-07-02T14:07:11.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350005+0000", 2023-07-02T14:07:11.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042766870000000004 2023-07-02T14:07:11.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354281+0000", 2023-07-02T14:07:11.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1155000000000002e-05 2023-07-02T14:07:11.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354312+0000", 2023-07-02T14:07:11.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071436820000000002 2023-07-02T14:07:11.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361456+0000", 2023-07-02T14:07:11.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9054000000000002e-05 2023-07-02T14:07:11.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361505+0000", 2023-07-02T14:07:11.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072301320000000002 2023-07-02T14:07:11.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368735+0000", 2023-07-02T14:07:11.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1013999999999994e-05 2023-07-02T14:07:11.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368816+0000", 2023-07-02T14:07:11.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036582490000000001 2023-07-02T14:07:11.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372475+0000", 2023-07-02T14:07:11.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4652000000000004e-05 2023-07-02T14:07:11.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:11.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372539+0000", 2023-07-02T14:07:11.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00031374199999999999 2023-07-02T14:07:11.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:11.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.372853+0000", 2023-07-02T14:07:11.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030071299999999998 2023-07-02T14:07:11.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:11.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.375860+0000", 2023-07-02T14:07:11.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041897749999999997 2023-07-02T14:07:11.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:11.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380050+0000", 2023-07-02T14:07:11.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8331000000000003e-05 2023-07-02T14:07:11.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:11.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380088+0000", 2023-07-02T14:07:11.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0626999999999998e-05 2023-07-02T14:07:11.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:11.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380179+0000", 2023-07-02T14:07:11.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13120991400000001 2023-07-02T14:07:11.981 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.981 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:11.981 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:11.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.981 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18558 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:07:11.981 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:07:11.981 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 66.596129708999996, 2023-07-02T14:07:11.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13755567899999999, 2023-07-02T14:07:11.982 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:11.982 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:11.982 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:11.982 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:11.982 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:11.982 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18558 2023-07-02T14:07:11.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.983 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:11.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:11.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:07:11.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:11.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:11.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248986+0000", 2023-07-02T14:07:11.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:11.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:11.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248985+0000", 2023-07-02T14:07:11.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8630000000000001e-06 2023-07-02T14:07:11.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:11.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248989+0000", 2023-07-02T14:07:11.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0750000000000003e-06 2023-07-02T14:07:11.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:11.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248995+0000", 2023-07-02T14:07:11.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3290000000000004e-06 2023-07-02T14:07:11.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:11.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.248999+0000", 2023-07-02T14:07:11.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00080656300000000005 2023-07-02T14:07:11.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249806+0000", 2023-07-02T14:07:11.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2040999999999999e-05 2023-07-02T14:07:11.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249848+0000", 2023-07-02T14:07:11.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066312419999999999 2023-07-02T14:07:11.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256479+0000", 2023-07-02T14:07:11.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3348e-05 2023-07-02T14:07:11.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256513+0000", 2023-07-02T14:07:11.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012391531000000001 2023-07-02T14:07:11.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268904+0000", 2023-07-02T14:07:11.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.347e-05 2023-07-02T14:07:11.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.268978+0000", 2023-07-02T14:07:11.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010929772000000001 2023-07-02T14:07:11.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279907+0000", 2023-07-02T14:07:11.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2669999999999997e-05 2023-07-02T14:07:11.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279940+0000", 2023-07-02T14:07:11.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011770332999999999 2023-07-02T14:07:11.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291710+0000", 2023-07-02T14:07:11.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1207000000000001e-05 2023-07-02T14:07:11.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291742+0000", 2023-07-02T14:07:11.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081242199999999997 2023-07-02T14:07:11.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299866+0000", 2023-07-02T14:07:11.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4650000000000002e-05 2023-07-02T14:07:11.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299900+0000", 2023-07-02T14:07:11.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071810270000000004 2023-07-02T14:07:11.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307081+0000", 2023-07-02T14:07:11.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4378000000000002e-05 2023-07-02T14:07:11.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307116+0000", 2023-07-02T14:07:11.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057031909999999998 2023-07-02T14:07:11.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312819+0000", 2023-07-02T14:07:11.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1612999999999997e-05 2023-07-02T14:07:11.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312851+0000", 2023-07-02T14:07:11.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048172470000000002 2023-07-02T14:07:11.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317668+0000", 2023-07-02T14:07:11.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3328e-05 2023-07-02T14:07:11.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317741+0000", 2023-07-02T14:07:11.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050117290000000004 2023-07-02T14:07:11.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:11.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322753+0000", 2023-07-02T14:07:11.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9872000000000001e-05 2023-07-02T14:07:11.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:11.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322783+0000", 2023-07-02T14:07:11.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064910610000000002 2023-07-02T14:07:11.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:11.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:11.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329274+0000", 2023-07-02T14:07:12.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0097e-05 2023-07-02T14:07:12.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329304+0000", 2023-07-02T14:07:12.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062163879999999998 2023-07-02T14:07:12.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335520+0000", 2023-07-02T14:07:12.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9513e-05 2023-07-02T14:07:12.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335550+0000", 2023-07-02T14:07:12.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098209969999999997 2023-07-02T14:07:12.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345371+0000", 2023-07-02T14:07:12.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9862999999999999e-05 2023-07-02T14:07:12.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345401+0000", 2023-07-02T14:07:12.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046552030000000001 2023-07-02T14:07:12.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350056+0000", 2023-07-02T14:07:12.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8751999999999993e-05 2023-07-02T14:07:12.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350145+0000", 2023-07-02T14:07:12.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041981450000000003 2023-07-02T14:07:12.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354343+0000", 2023-07-02T14:07:12.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1501000000000002e-05 2023-07-02T14:07:12.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354374+0000", 2023-07-02T14:07:12.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071551740000000003 2023-07-02T14:07:12.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361530+0000", 2023-07-02T14:07:12.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5156e-05 2023-07-02T14:07:12.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361565+0000", 2023-07-02T14:07:12.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072918569999999997 2023-07-02T14:07:12.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368857+0000", 2023-07-02T14:07:12.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5770999999999999e-05 2023-07-02T14:07:12.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368902+0000", 2023-07-02T14:07:12.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057210569999999999 2023-07-02T14:07:12.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374623+0000", 2023-07-02T14:07:12.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3266999999999998e-05 2023-07-02T14:07:12.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374667+0000", 2023-07-02T14:07:12.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055479570000000001 2023-07-02T14:07:12.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380215+0000", 2023-07-02T14:07:12.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0067e-05 2023-07-02T14:07:12.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:12.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380265+0000", 2023-07-02T14:07:12.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00043082299999999999 2023-07-02T14:07:12.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:12.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.380696+0000", 2023-07-02T14:07:12.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024853240000000001 2023-07-02T14:07:12.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:12.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.383181+0000", 2023-07-02T14:07:12.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032601280000000002 2023-07-02T14:07:12.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:12.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386441+0000", 2023-07-02T14:07:12.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9748e-05 2023-07-02T14:07:12.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:12.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386471+0000", 2023-07-02T14:07:12.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0531000000000006e-05 2023-07-02T14:07:12.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:12.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386541+0000", 2023-07-02T14:07:12.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13755567899999999 2023-07-02T14:07:12.014 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.014 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:12.014 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.015 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18559 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:07:12.015 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:07:12.015 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 66.596107105000002, 2023-07-02T14:07:12.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.146184591, 2023-07-02T14:07:12.015 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:12.015 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:12.015 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:12.015 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:12.016 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:12.016 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18559 2023-07-02T14:07:12.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.016 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:12.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:12.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:07:12.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:12.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:12.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:07:12.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:12.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:12.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249008+0000", 2023-07-02T14:07:12.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.726e-06 2023-07-02T14:07:12.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:12.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249011+0000", 2023-07-02T14:07:12.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7939999999999997e-06 2023-07-02T14:07:12.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:12.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249018+0000", 2023-07-02T14:07:12.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3470000000000001e-06 2023-07-02T14:07:12.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:12.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249023+0000", 2023-07-02T14:07:12.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00084402900000000005 2023-07-02T14:07:12.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249867+0000", 2023-07-02T14:07:12.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9050000000000001e-05 2023-07-02T14:07:12.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249906+0000", 2023-07-02T14:07:12.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066311039999999996 2023-07-02T14:07:12.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256537+0000", 2023-07-02T14:07:12.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2790000000000003e-05 2023-07-02T14:07:12.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256570+0000", 2023-07-02T14:07:12.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012436129000000001 2023-07-02T14:07:12.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269006+0000", 2023-07-02T14:07:12.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0173999999999999e-05 2023-07-02T14:07:12.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269046+0000", 2023-07-02T14:07:12.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010924698 2023-07-02T14:07:12.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.279971+0000", 2023-07-02T14:07:12.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3831000000000002e-05 2023-07-02T14:07:12.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280004+0000", 2023-07-02T14:07:12.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011795912 2023-07-02T14:07:12.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291800+0000", 2023-07-02T14:07:12.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5613999999999997e-05 2023-07-02T14:07:12.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291886+0000", 2023-07-02T14:07:12.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080395999999999992 2023-07-02T14:07:12.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299925+0000", 2023-07-02T14:07:12.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0694999999999997e-05 2023-07-02T14:07:12.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299956+0000", 2023-07-02T14:07:12.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072049449999999999 2023-07-02T14:07:12.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307161+0000", 2023-07-02T14:07:12.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5523000000000001e-05 2023-07-02T14:07:12.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307247+0000", 2023-07-02T14:07:12.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056331929999999999 2023-07-02T14:07:12.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312880+0000", 2023-07-02T14:07:12.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1646000000000002e-05 2023-07-02T14:07:12.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312911+0000", 2023-07-02T14:07:12.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048627540000000004 2023-07-02T14:07:12.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317774+0000", 2023-07-02T14:07:12.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2008000000000003e-05 2023-07-02T14:07:12.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317806+0000", 2023-07-02T14:07:12.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050055689999999996 2023-07-02T14:07:12.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322812+0000", 2023-07-02T14:07:12.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3442999999999997e-05 2023-07-02T14:07:12.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322855+0000", 2023-07-02T14:07:12.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006472813 2023-07-02T14:07:12.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329328+0000", 2023-07-02T14:07:12.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0128e-05 2023-07-02T14:07:12.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329358+0000", 2023-07-02T14:07:12.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062156490000000002 2023-07-02T14:07:12.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335574+0000", 2023-07-02T14:07:12.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2357999999999997e-05 2023-07-02T14:07:12.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335606+0000", 2023-07-02T14:07:12.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098318409999999992 2023-07-02T14:07:12.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345438+0000", 2023-07-02T14:07:12.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1174000000000007e-05 2023-07-02T14:07:12.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345509+0000", 2023-07-02T14:07:12.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004669851 2023-07-02T14:07:12.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350179+0000", 2023-07-02T14:07:12.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1559000000000001e-05 2023-07-02T14:07:12.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350211+0000", 2023-07-02T14:07:12.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041865740000000002 2023-07-02T14:07:12.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.039 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.039 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354397+0000", 2023-07-02T14:07:12.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9850000000000001e-05 2023-07-02T14:07:12.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.040 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354427+0000", 2023-07-02T14:07:12.040 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071608449999999999 2023-07-02T14:07:12.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.040 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.040 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361588+0000", 2023-07-02T14:07:12.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8346000000000002e-05 2023-07-02T14:07:12.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361626+0000", 2023-07-02T14:07:12.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073175009999999997 2023-07-02T14:07:12.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.368944+0000", 2023-07-02T14:07:12.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2879000000000005e-05 2023-07-02T14:07:12.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369017+0000", 2023-07-02T14:07:12.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056850690000000001 2023-07-02T14:07:12.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374702+0000", 2023-07-02T14:07:12.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7693e-05 2023-07-02T14:07:12.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374739+0000", 2023-07-02T14:07:12.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076925149999999996 2023-07-02T14:07:12.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382432+0000", 2023-07-02T14:07:12.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3259e-05 2023-07-02T14:07:12.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382475+0000", 2023-07-02T14:07:12.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041005110000000003 2023-07-02T14:07:12.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386576+0000", 2023-07-02T14:07:12.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2074999999999999e-05 2023-07-02T14:07:12.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:12.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386618+0000", 2023-07-02T14:07:12.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000233692 2023-07-02T14:07:12.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:12.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.386851+0000", 2023-07-02T14:07:12.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024022269999999998 2023-07-02T14:07:12.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:12.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.389254+0000", 2023-07-02T14:07:12.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058506469999999996 2023-07-02T14:07:12.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:12.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395104+0000", 2023-07-02T14:07:12.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8807000000000003e-05 2023-07-02T14:07:12.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:12.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395143+0000", 2023-07-02T14:07:12.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9616000000000001e-05 2023-07-02T14:07:12.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:12.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395193+0000", 2023-07-02T14:07:12.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.146184591 2023-07-02T14:07:12.051 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.051 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:12.051 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.051 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18560 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:07:12.051 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:07:12.052 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 66.596088481999999, 2023-07-02T14:07:12.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.15647551900000001, 2023-07-02T14:07:12.052 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:12.052 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:12.052 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:12.052 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:12.053 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:12.053 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18560 2023-07-02T14:07:12.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.053 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:12.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:12.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:07:12.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:12.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:12.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249027+0000", 2023-07-02T14:07:12.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:12.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:12.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249026+0000", 2023-07-02T14:07:12.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3419999999999999e-06 2023-07-02T14:07:12.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:12.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249030+0000", 2023-07-02T14:07:12.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7869999999999998e-06 2023-07-02T14:07:12.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:12.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249035+0000", 2023-07-02T14:07:12.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9639999999999997e-06 2023-07-02T14:07:12.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:12.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249039+0000", 2023-07-02T14:07:12.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00088477599999999999 2023-07-02T14:07:12.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249924+0000", 2023-07-02T14:07:12.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7333999999999999e-05 2023-07-02T14:07:12.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249962+0000", 2023-07-02T14:07:12.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066323409999999999 2023-07-02T14:07:12.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256594+0000", 2023-07-02T14:07:12.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2595999999999998e-05 2023-07-02T14:07:12.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256627+0000", 2023-07-02T14:07:12.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012449894 2023-07-02T14:07:12.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269076+0000", 2023-07-02T14:07:12.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3942999999999997e-05 2023-07-02T14:07:12.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269110+0000", 2023-07-02T14:07:12.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010923884 2023-07-02T14:07:12.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280034+0000", 2023-07-02T14:07:12.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7774000000000001e-05 2023-07-02T14:07:12.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280072+0000", 2023-07-02T14:07:12.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011850027000000001 2023-07-02T14:07:12.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291922+0000", 2023-07-02T14:07:12.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2128000000000001e-05 2023-07-02T14:07:12.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291954+0000", 2023-07-02T14:07:12.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0080261150000000003 2023-07-02T14:07:12.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.299980+0000", 2023-07-02T14:07:12.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6322999999999998e-05 2023-07-02T14:07:12.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300017+0000", 2023-07-02T14:07:12.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072683039999999997 2023-07-02T14:07:12.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307285+0000", 2023-07-02T14:07:12.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3309000000000002e-05 2023-07-02T14:07:12.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307318+0000", 2023-07-02T14:07:12.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056178649999999997 2023-07-02T14:07:12.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312936+0000", 2023-07-02T14:07:12.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1866000000000003e-05 2023-07-02T14:07:12.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312968+0000", 2023-07-02T14:07:12.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048789970000000004 2023-07-02T14:07:12.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317847+0000", 2023-07-02T14:07:12.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0223999999999996e-05 2023-07-02T14:07:12.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317927+0000", 2023-07-02T14:07:12.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049543649999999996 2023-07-02T14:07:12.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322882+0000", 2023-07-02T14:07:12.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1399000000000002e-05 2023-07-02T14:07:12.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322913+0000", 2023-07-02T14:07:12.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006534214 2023-07-02T14:07:12.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329447+0000", 2023-07-02T14:07:12.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9649999999999998e-05 2023-07-02T14:07:12.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329527+0000", 2023-07-02T14:07:12.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061026839999999997 2023-07-02T14:07:12.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335629+0000", 2023-07-02T14:07:12.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9611000000000002e-05 2023-07-02T14:07:12.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335659+0000", 2023-07-02T14:07:12.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098851549999999996 2023-07-02T14:07:12.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345544+0000", 2023-07-02T14:07:12.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0587999999999999e-05 2023-07-02T14:07:12.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345575+0000", 2023-07-02T14:07:12.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046790859999999998 2023-07-02T14:07:12.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350254+0000", 2023-07-02T14:07:12.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0284000000000002e-05 2023-07-02T14:07:12.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350304+0000", 2023-07-02T14:07:12.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041560540000000002 2023-07-02T14:07:12.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354460+0000", 2023-07-02T14:07:12.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2427000000000004e-05 2023-07-02T14:07:12.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354523+0000", 2023-07-02T14:07:12.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071268770000000002 2023-07-02T14:07:12.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361650+0000", 2023-07-02T14:07:12.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0026999999999999e-05 2023-07-02T14:07:12.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361680+0000", 2023-07-02T14:07:12.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073733080000000003 2023-07-02T14:07:12.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369053+0000", 2023-07-02T14:07:12.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7635999999999997e-05 2023-07-02T14:07:12.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369091+0000", 2023-07-02T14:07:12.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056908059999999996 2023-07-02T14:07:12.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374781+0000", 2023-07-02T14:07:12.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4656000000000002e-05 2023-07-02T14:07:12.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374816+0000", 2023-07-02T14:07:12.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077037809999999998 2023-07-02T14:07:12.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382520+0000", 2023-07-02T14:07:12.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7858e-05 2023-07-02T14:07:12.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382588+0000", 2023-07-02T14:07:12.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057884019999999998 2023-07-02T14:07:12.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388376+0000", 2023-07-02T14:07:12.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4831000000000001e-05 2023-07-02T14:07:12.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388461+0000", 2023-07-02T14:07:12.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067708480000000003 2023-07-02T14:07:12.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395232+0000", 2023-07-02T14:07:12.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.7948000000000002e-05 2023-07-02T14:07:12.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:12.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395290+0000", 2023-07-02T14:07:12.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00027672599999999998 2023-07-02T14:07:12.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:12.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.395566+0000", 2023-07-02T14:07:12.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0025705569999999998 2023-07-02T14:07:12.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:12.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.398137+0000", 2023-07-02T14:07:12.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072257099999999998 2023-07-02T14:07:12.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:12.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405363+0000", 2023-07-02T14:07:12.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3887999999999998e-05 2023-07-02T14:07:12.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:12.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405397+0000", 2023-07-02T14:07:12.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000105769 2023-07-02T14:07:12.089 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.090 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:12.090 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405502+0000", 2023-07-02T14:07:12.090 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.15647551900000001 2023-07-02T14:07:12.090 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.090 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:12.090 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.090 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.091 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.091 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18561 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:07:12.091 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:07:12.091 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 66.596068118000005, 2023-07-02T14:07:12.091 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744, 2023-07-02T14:07:12.091 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:12.091 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:12.092 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:12.092 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:12.092 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:12.092 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18561 2023-07-02T14:07:12.092 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.092 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:12.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:12.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:07:12.093 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:12.093 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.093 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.093 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:12.093 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:07:12.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:12.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.094 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.094 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:12.094 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:07:12.094 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.563e-06 2023-07-02T14:07:12.094 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:12.095 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249050+0000", 2023-07-02T14:07:12.095 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4500000000000001e-06 2023-07-02T14:07:12.095 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.095 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.095 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:12.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249057+0000", 2023-07-02T14:07:12.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1860000000000002e-06 2023-07-02T14:07:12.096 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.096 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.096 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:12.096 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249061+0000", 2023-07-02T14:07:12.096 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00091153399999999998 2023-07-02T14:07:12.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.097 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.097 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249972+0000", 2023-07-02T14:07:12.097 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7156999999999997e-05 2023-07-02T14:07:12.097 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.097 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.250010+0000", 2023-07-02T14:07:12.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006641816 2023-07-02T14:07:12.098 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.098 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.098 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.098 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256651+0000", 2023-07-02T14:07:12.098 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1968999999999998e-05 2023-07-02T14:07:12.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.099 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.099 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256683+0000", 2023-07-02T14:07:12.099 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012463952 2023-07-02T14:07:12.099 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.099 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269147+0000", 2023-07-02T14:07:12.100 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2855000000000006e-05 2023-07-02T14:07:12.100 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.100 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.100 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.100 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269220+0000", 2023-07-02T14:07:12.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010870418999999999 2023-07-02T14:07:12.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.101 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.101 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280091+0000", 2023-07-02T14:07:12.101 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1921999999999997e-05 2023-07-02T14:07:12.101 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.101 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280123+0000", 2023-07-02T14:07:12.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011861682 2023-07-02T14:07:12.102 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.102 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.102 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.102 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291984+0000", 2023-07-02T14:07:12.102 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5259000000000006e-05 2023-07-02T14:07:12.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.103 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.103 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.292059+0000", 2023-07-02T14:07:12.103 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079709359999999996 2023-07-02T14:07:12.103 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.103 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300030+0000", 2023-07-02T14:07:12.104 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0729999999999998e-05 2023-07-02T14:07:12.104 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.104 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.104 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.104 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300071+0000", 2023-07-02T14:07:12.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072785690000000004 2023-07-02T14:07:12.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.105 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.105 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.105 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307350+0000", 2023-07-02T14:07:12.105 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1948999999999999e-05 2023-07-02T14:07:12.105 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.106 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307412+0000", 2023-07-02T14:07:12.106 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055785779999999998 2023-07-02T14:07:12.106 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.106 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.106 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312990+0000", 2023-07-02T14:07:12.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0957000000000002e-05 2023-07-02T14:07:12.107 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.107 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.107 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.107 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.313021+0000", 2023-07-02T14:07:12.107 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049246400000000001 2023-07-02T14:07:12.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.108 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.108 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317946+0000", 2023-07-02T14:07:12.108 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9346e-05 2023-07-02T14:07:12.108 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.108 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317975+0000", 2023-07-02T14:07:12.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049649469999999999 2023-07-02T14:07:12.109 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.109 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.109 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.109 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322940+0000", 2023-07-02T14:07:12.109 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9017000000000002e-05 2023-07-02T14:07:12.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.110 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.110 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322989+0000", 2023-07-02T14:07:12.110 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065907029999999998 2023-07-02T14:07:12.110 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.110 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329580+0000", 2023-07-02T14:07:12.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5879999999999999e-05 2023-07-02T14:07:12.111 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.111 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.111 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.111 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329656+0000", 2023-07-02T14:07:12.111 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060166960000000002 2023-07-02T14:07:12.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.112 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.112 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.112 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335672+0000", 2023-07-02T14:07:12.112 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8963e-05 2023-07-02T14:07:12.112 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335701+0000", 2023-07-02T14:07:12.113 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098989560000000004 2023-07-02T14:07:12.113 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.113 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.113 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.113 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345600+0000", 2023-07-02T14:07:12.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5395999999999995e-05 2023-07-02T14:07:12.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.114 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.114 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.114 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345666+0000", 2023-07-02T14:07:12.114 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046589530000000004 2023-07-02T14:07:12.114 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.115 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350325+0000", 2023-07-02T14:07:12.115 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2459000000000002e-05 2023-07-02T14:07:12.115 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.115 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.115 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350357+0000", 2023-07-02T14:07:12.116 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042054290000000001 2023-07-02T14:07:12.116 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.116 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.116 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.116 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354563+0000", 2023-07-02T14:07:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5612999999999999e-05 2023-07-02T14:07:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354598+0000", 2023-07-02T14:07:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070967449999999998 2023-07-02T14:07:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.117 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361695+0000", 2023-07-02T14:07:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9314000000000001e-05 2023-07-02T14:07:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.118 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361724+0000", 2023-07-02T14:07:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007391028 2023-07-02T14:07:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369115+0000", 2023-07-02T14:07:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9163000000000002e-05 2023-07-02T14:07:12.119 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.120 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369184+0000", 2023-07-02T14:07:12.120 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056532500000000003 2023-07-02T14:07:12.120 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.120 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.120 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374838+0000", 2023-07-02T14:07:12.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5297999999999999e-05 2023-07-02T14:07:12.121 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.121 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.121 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.121 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374883+0000", 2023-07-02T14:07:12.121 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077292189999999998 2023-07-02T14:07:12.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.122 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.122 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382612+0000", 2023-07-02T14:07:12.122 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1102000000000001e-05 2023-07-02T14:07:12.122 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.122 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382653+0000", 2023-07-02T14:07:12.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058515989999999999 2023-07-02T14:07:12.123 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.123 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.123 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.123 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388505+0000", 2023-07-02T14:07:12.123 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9196000000000006e-05 2023-07-02T14:07:12.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.124 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.124 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.124 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388574+0000", 2023-07-02T14:07:12.124 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084245339999999991 2023-07-02T14:07:12.124 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.125 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.396999+0000", 2023-07-02T14:07:12.125 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0596999999999998e-05 2023-07-02T14:07:12.125 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.125 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.125 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:12.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.397039+0000", 2023-07-02T14:07:12.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084911339999999991 2023-07-02T14:07:12.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.126 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.126 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.126 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405530+0000", 2023-07-02T14:07:12.126 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7555000000000001e-05 2023-07-02T14:07:12.126 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:12.127 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405578+0000", 2023-07-02T14:07:12.127 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000229607 2023-07-02T14:07:12.127 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.127 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.127 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:12.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405808+0000", 2023-07-02T14:07:12.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026919940000000001 2023-07-02T14:07:12.128 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.128 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.128 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:12.128 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.408500+0000", 2023-07-02T14:07:12.128 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053065059999999999 2023-07-02T14:07:12.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.129 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.129 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:12.129 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413806+0000", 2023-07-02T14:07:12.129 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0490000000000001e-05 2023-07-02T14:07:12.129 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:12.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413837+0000", 2023-07-02T14:07:12.130 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000100386 2023-07-02T14:07:12.130 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.130 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.130 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:12.130 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413937+0000", 2023-07-02T14:07:12.131 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744 2023-07-02T14:07:12.131 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.131 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:12.131 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.131 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.131 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.131 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20768 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch ping cookie 140544751726928 gen 52] snapc 0=[] ondisk+write+known_if_redirected e183)", 2023-07-02T14:07:12.132 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:07:12.132 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 52.622700850000001, 2023-07-02T14:07:12.132 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3751256970000001, 2023-07-02T14:07:12.132 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:12.132 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:07:12.132 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:12.132 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:12.133 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:12.133 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20768 2023-07-02T14:07:12.133 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.133 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:12.133 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.133 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:12.133 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:07:12.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:12.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.134 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:12.134 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:07:12.134 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:12.134 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.134 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:12.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:07:12.135 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7829999999999998e-06 2023-07-02T14:07:12.135 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.135 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.135 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:12.135 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222419+0000", 2023-07-02T14:07:12.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1380000000000004e-06 2023-07-02T14:07:12.136 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.136 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.136 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:12.136 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222425+0000", 2023-07-02T14:07:12.136 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2519999999999996e-06 2023-07-02T14:07:12.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.137 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:12.137 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222431+0000", 2023-07-02T14:07:12.137 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2157999999999994e-05 2023-07-02T14:07:12.137 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.137 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222523+0000", 2023-07-02T14:07:12.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5147000000000001e-05 2023-07-02T14:07:12.138 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.138 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.138 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:12.138 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222538+0000", 2023-07-02T14:07:12.138 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30105419700000002 2023-07-02T14:07:12.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.139 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.139 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523592+0000", 2023-07-02T14:07:12.139 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2316e-05 2023-07-02T14:07:12.139 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.139 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:12.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523605+0000", 2023-07-02T14:07:12.140 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.073658953 2023-07-02T14:07:12.140 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.140 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.140 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.140 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597264+0000", 2023-07-02T14:07:12.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000213415 2023-07-02T14:07:12.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.141 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.141 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:12.141 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597477+0000", 2023-07-02T14:07:12.141 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3034999999999995e-05 2023-07-02T14:07:12.141 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.142 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.142 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:12.142 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597540+0000", 2023-07-02T14:07:12.142 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3751256970000001 2023-07-02T14:07:12.142 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.142 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:12.142 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.143 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.143 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.143 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20769 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 53] snapc 0=[] ondisk+write+known_if_redirected e184)", 2023-07-02T14:07:12.143 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:07:12.143 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 52.622338395, 2023-07-02T14:07:12.143 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.380884139, 2023-07-02T14:07:12.143 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:12.143 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:07:12.144 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:12.144 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:12.144 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:12.144 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20769 2023-07-02T14:07:12.144 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.144 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:12.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:12.145 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:07:12.145 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:12.145 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.145 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.145 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:12.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:07:12.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:12.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.146 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.146 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:12.146 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222776+0000", 2023-07-02T14:07:12.146 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9710000000000003e-06 2023-07-02T14:07:12.146 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:12.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222781+0000", 2023-07-02T14:07:12.147 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9899999999999992e-06 2023-07-02T14:07:12.147 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.147 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.147 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:12.147 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222791+0000", 2023-07-02T14:07:12.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4240000000000003e-06 2023-07-02T14:07:12.148 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.148 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.148 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:12.148 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222795+0000", 2023-07-02T14:07:12.148 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2366000000000001e-05 2023-07-02T14:07:12.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.149 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222828+0000", 2023-07-02T14:07:12.149 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4790000000000001e-06 2023-07-02T14:07:12.149 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.149 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.149 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:12.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222834+0000", 2023-07-02T14:07:12.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30084589299999998 2023-07-02T14:07:12.150 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.150 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.150 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.150 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523680+0000", 2023-07-02T14:07:12.150 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.87e-06 2023-07-02T14:07:12.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.151 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:12.151 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523685+0000", 2023-07-02T14:07:12.151 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.074111042 2023-07-02T14:07:12.151 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.151 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597796+0000", 2023-07-02T14:07:12.152 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0411000000000001e-05 2023-07-02T14:07:12.152 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.152 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.152 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:12.152 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597856+0000", 2023-07-02T14:07:12.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058045340000000001 2023-07-02T14:07:12.153 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.153 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.153 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:12.153 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.603661+0000", 2023-07-02T14:07:12.153 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.380884139 2023-07-02T14:07:12.153 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.153 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:12.154 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.154 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.154 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.154 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20770 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 54] snapc 0=[] ondisk+write+known_if_redirected e185)", 2023-07-02T14:07:12.154 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:07:12.154 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 52.321121284999997, 2023-07-02T14:07:12.154 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0799819349999999, 2023-07-02T14:07:12.154 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:12.155 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:07:12.155 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:12.155 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:12.155 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:12.155 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20770 2023-07-02T14:07:12.155 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.155 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:12.155 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:12.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:07:12.156 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:12.156 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.156 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.156 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:12.156 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:07:12.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:12.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.157 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.157 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:12.157 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523992+0000", 2023-07-02T14:07:12.157 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6720000000000004e-06 2023-07-02T14:07:12.157 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:12.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524000+0000", 2023-07-02T14:07:12.158 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3711e-05 2023-07-02T14:07:12.158 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.158 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.158 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:12.158 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524014+0000", 2023-07-02T14:07:12.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7849999999999992e-06 2023-07-02T14:07:12.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.159 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:12.159 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524022+0000", 2023-07-02T14:07:12.159 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2366000000000001e-05 2023-07-02T14:07:12.159 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.159 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524054+0000", 2023-07-02T14:07:12.160 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8169999999999998e-06 2023-07-02T14:07:12.160 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.160 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.160 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:12.160 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524059+0000", 2023-07-02T14:07:12.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0748338850000001 2023-07-02T14:07:12.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.161 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.161 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.598893+0000", 2023-07-02T14:07:12.161 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116968 2023-07-02T14:07:12.161 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.161 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:12.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.599010+0000", 2023-07-02T14:07:12.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049663370000000004 2023-07-02T14:07:12.162 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.162 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.162 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:12.162 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.603976+0000", 2023-07-02T14:07:12.162 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0799819349999999 2023-07-02T14:07:12.163 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.163 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:12.163 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.163 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.163 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.163 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20833 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 55] snapc 0=[] ondisk+write+known_if_redirected e189)", 2023-07-02T14:07:12.163 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:07:12.163 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 47.313001980000003, 2023-07-02T14:07:12.164 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.029349206, 2023-07-02T14:07:12.164 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:12.164 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:07:12.164 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:12.164 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:12.164 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:12.164 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20833 2023-07-02T14:07:12.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.165 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:12.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.165 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:12.165 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:07:12.165 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:12.165 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.165 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:12.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:07:12.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:12.166 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.166 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.166 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:12.166 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532112+0000", 2023-07-02T14:07:12.166 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7590000000000003e-06 2023-07-02T14:07:12.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:12.167 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532120+0000", 2023-07-02T14:07:12.167 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8130000000000001e-05 2023-07-02T14:07:12.167 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.167 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.167 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:12.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532139+0000", 2023-07-02T14:07:12.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3299999999999999e-06 2023-07-02T14:07:12.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.168 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.168 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:12.168 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532147+0000", 2023-07-02T14:07:12.168 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9679999999999998e-05 2023-07-02T14:07:12.168 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.169 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.169 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.169 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532207+0000", 2023-07-02T14:07:12.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0648e-05 2023-07-02T14:07:12.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.170 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.170 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op must wait for map", 2023-07-02T14:07:12.170 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532217+0000", 2023-07-02T14:07:12.170 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002082455 2023-07-02T14:07:12.170 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.534300+0000", 2023-07-02T14:07:12.171 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0710000000000001e-05 2023-07-02T14:07:12.171 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.171 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.171 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:12.171 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.534310+0000", 2023-07-02T14:07:12.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.013227235 2023-07-02T14:07:12.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.172 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.172 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.172 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.547538+0000", 2023-07-02T14:07:12.172 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021904300000000001 2023-07-02T14:07:12.172 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:12.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.547757+0000", 2023-07-02T14:07:12.173 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013705784 2023-07-02T14:07:12.173 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.173 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.173 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:12.173 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561462+0000", 2023-07-02T14:07:12.174 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.029349206 2023-07-02T14:07:12.174 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.174 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:12.174 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.174 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.174 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.174 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20865 2.7 2:ff0f5f99:::rbd_data.125de1449605.0000000000000006:head [write 131072~8192] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:07:12.174 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:07:12.175 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.568378908, 2023-07-02T14:07:12.175 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.288115762, 2023-07-02T14:07:12.175 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:12.175 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:12.175 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:12.175 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:12.175 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:12.175 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20865 2023-07-02T14:07:12.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.176 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:12.176 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.176 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:12.176 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:07:12.176 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:12.176 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:12.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:07:12.177 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:12.177 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.177 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.177 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:12.177 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276735+0000", 2023-07-02T14:07:12.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3273e-05 2023-07-02T14:07:12.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.178 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.178 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:12.178 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276748+0000", 2023-07-02T14:07:12.178 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3895e-05 2023-07-02T14:07:12.178 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:12.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276762+0000", 2023-07-02T14:07:12.179 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.896e-06 2023-07-02T14:07:12.179 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.179 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.179 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:12.179 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276768+0000", 2023-07-02T14:07:12.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5850999999999999e-05 2023-07-02T14:07:12.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.180 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.180 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.180 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276814+0000", 2023-07-02T14:07:12.180 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5290000000000003e-06 2023-07-02T14:07:12.180 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:12.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276821+0000", 2023-07-02T14:07:12.181 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.27225312600000001 2023-07-02T14:07:12.181 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.181 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.181 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.181 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549074+0000", 2023-07-02T14:07:12.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000151223 2023-07-02T14:07:12.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.182 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:12.182 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549226+0000", 2023-07-02T14:07:12.182 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024865400000000001 2023-07-02T14:07:12.182 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.182 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:12.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549474+0000", 2023-07-02T14:07:12.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012039185000000001 2023-07-02T14:07:12.183 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.183 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.183 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:12.183 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561513+0000", 2023-07-02T14:07:12.183 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003221327 2023-07-02T14:07:12.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.184 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:12.184 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564735+0000", 2023-07-02T14:07:12.184 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2026000000000005e-05 2023-07-02T14:07:12.184 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.184 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:12.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564807+0000", 2023-07-02T14:07:12.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5303000000000003e-05 2023-07-02T14:07:12.185 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.185 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.185 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:12.185 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564852+0000", 2023-07-02T14:07:12.185 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.288115762 2023-07-02T14:07:12.186 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.186 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:12.186 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.186 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.186 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.186 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20866 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 131072~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:07:12.186 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:07:12.186 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.562685586999997, 2023-07-02T14:07:12.187 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.28773701200000001, 2023-07-02T14:07:12.187 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:12.187 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:12.187 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:12.187 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:12.187 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:12.187 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20866 2023-07-02T14:07:12.187 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.188 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:12.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.188 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:12.188 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:07:12.188 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:12.188 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.188 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:12.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:07:12.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:12.189 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.189 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.189 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:12.189 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282428+0000", 2023-07-02T14:07:12.189 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8469999999999998e-05 2023-07-02T14:07:12.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:12.190 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282517+0000", 2023-07-02T14:07:12.190 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2974e-05 2023-07-02T14:07:12.190 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.190 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.190 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:12.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282530+0000", 2023-07-02T14:07:12.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4530000000000001e-06 2023-07-02T14:07:12.191 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.191 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.191 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:12.191 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282535+0000", 2023-07-02T14:07:12.191 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1926999999999999e-05 2023-07-02T14:07:12.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.192 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282587+0000", 2023-07-02T14:07:12.192 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3083e-05 2023-07-02T14:07:12.192 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.192 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.192 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:12.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282600+0000", 2023-07-02T14:07:12.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.270692971 2023-07-02T14:07:12.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.193 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.193 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.193 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553293+0000", 2023-07-02T14:07:12.193 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017065599999999999 2023-07-02T14:07:12.193 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:12.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553464+0000", 2023-07-02T14:07:12.194 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026504399999999998 2023-07-02T14:07:12.194 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.194 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.194 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:12.194 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553729+0000", 2023-07-02T14:07:12.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011173008 2023-07-02T14:07:12.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.195 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.195 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:12.195 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564902+0000", 2023-07-02T14:07:12.195 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051242600000000003 2023-07-02T14:07:12.195 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:12.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570026+0000", 2023-07-02T14:07:12.196 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8098999999999999e-05 2023-07-02T14:07:12.196 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.196 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.196 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:12.196 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570104+0000", 2023-07-02T14:07:12.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2323000000000006e-05 2023-07-02T14:07:12.197 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.197 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.197 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:12.197 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570167+0000", 2023-07-02T14:07:12.197 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.28773701200000001 2023-07-02T14:07:12.197 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.198 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:12.198 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.198 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.198 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.198 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20875 2.7 2:e3210266:::rbd_data.125de1449605.00000000000001a3:head [write 0~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:07:12.198 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:07:12.198 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.523634770999998, 2023-07-02T14:07:12.198 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.25297039599999999, 2023-07-02T14:07:12.199 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:12.199 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:12.199 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:12.199 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:12.199 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:12.199 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20875 2023-07-02T14:07:12.199 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.199 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:12.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:12.200 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:07:12.200 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:12.200 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.200 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.200 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:12.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:07:12.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:12.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.201 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.201 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:12.201 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321479+0000", 2023-07-02T14:07:12.201 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015194399999999999 2023-07-02T14:07:12.201 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:12.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321631+0000", 2023-07-02T14:07:12.202 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4236999999999999e-05 2023-07-02T14:07:12.202 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.202 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.202 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:12.202 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321645+0000", 2023-07-02T14:07:12.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2160000000000003e-06 2023-07-02T14:07:12.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.203 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:12.203 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321651+0000", 2023-07-02T14:07:12.203 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3609999999999998e-05 2023-07-02T14:07:12.203 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.203 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321695+0000", 2023-07-02T14:07:12.204 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1989999999999995e-06 2023-07-02T14:07:12.204 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.204 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.204 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:12.204 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321704+0000", 2023-07-02T14:07:12.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.23542881500000001 2023-07-02T14:07:12.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.205 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.205 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557133+0000", 2023-07-02T14:07:12.205 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017359600000000001 2023-07-02T14:07:12.205 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.205 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:12.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557307+0000", 2023-07-02T14:07:12.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00023367599999999999 2023-07-02T14:07:12.206 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.206 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.206 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:12.206 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557540+0000", 2023-07-02T14:07:12.206 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016744963000000002 2023-07-02T14:07:12.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.207 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:12.207 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574285+0000", 2023-07-02T14:07:12.207 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1330999999999999e-05 2023-07-02T14:07:12.207 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.207 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:12.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574346+0000", 2023-07-02T14:07:12.208 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9750000000000002e-05 2023-07-02T14:07:12.208 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.208 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.208 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:12.208 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574406+0000", 2023-07-02T14:07:12.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4647999999999997e-05 2023-07-02T14:07:12.209 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.209 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.209 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:12.209 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574451+0000", 2023-07-02T14:07:12.209 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.25297039599999999 2023-07-02T14:07:12.209 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.209 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:12.210 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.210 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.210 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.210 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20915 2.7 2:e6e64067:::rbd_data.125de1449605.00000000000001a9:head [write 0~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:07:12.210 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:07:12.210 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.460380976000003, 2023-07-02T14:07:12.210 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.19966078500000001, 2023-07-02T14:07:12.211 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:12.211 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:12.211 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:12.211 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:12.211 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:12.211 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20915 2023-07-02T14:07:12.211 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.211 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:12.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:12.212 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:07:12.212 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:12.212 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.212 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.212 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:12.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:07:12.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:07:12.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.213 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.213 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:12.213 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384732+0000", 2023-07-02T14:07:12.213 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9439999999999999e-05 2023-07-02T14:07:12.213 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:12.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384802+0000", 2023-07-02T14:07:12.214 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7572999999999999e-05 2023-07-02T14:07:12.214 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.214 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.214 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:12.214 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384819+0000", 2023-07-02T14:07:12.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1670000000000004e-06 2023-07-02T14:07:12.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.215 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:12.215 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384825+0000", 2023-07-02T14:07:12.215 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6471999999999998e-05 2023-07-02T14:07:12.215 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.215 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384872+0000", 2023-07-02T14:07:12.216 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0028e-05 2023-07-02T14:07:12.216 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.216 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.216 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:12.216 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384882+0000", 2023-07-02T14:07:12.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.17674214099999999 2023-07-02T14:07:12.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.217 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.217 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561624+0000", 2023-07-02T14:07:12.217 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000135809 2023-07-02T14:07:12.217 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.217 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:12.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561760+0000", 2023-07-02T14:07:12.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020468 2023-07-02T14:07:12.218 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.218 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.218 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:12.218 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561965+0000", 2023-07-02T14:07:12.218 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012498198 2023-07-02T14:07:12.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:12.219 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574463+0000", 2023-07-02T14:07:12.219 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098197509999999998 2023-07-02T14:07:12.219 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.219 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.219 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:12.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584283+0000", 2023-07-02T14:07:12.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2056000000000004e-05 2023-07-02T14:07:12.220 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.220 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.220 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:12.220 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584345+0000", 2023-07-02T14:07:12.220 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0513000000000002e-05 2023-07-02T14:07:12.221 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.221 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.221 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:12.221 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584395+0000", 2023-07-02T14:07:12.221 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.19966078500000001 2023-07-02T14:07:12.221 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.221 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:12.221 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.222 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.222 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.222 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20948 2.7 2:ff0f5f99:::rbd_data.125de1449605.0000000000000006:head [write 139264~2220032] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:07:12.222 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.450499+0000", 2023-07-02T14:07:12.222 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.394616249000002, 2023-07-02T14:07:12.222 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13400999399999999, 2023-07-02T14:07:12.222 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:12.222 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:12.223 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:12.223 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:12.223 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:12.223 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20948 2023-07-02T14:07:12.223 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.223 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:12.223 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.223 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:12.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.450499+0000", 2023-07-02T14:07:12.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:12.224 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.224 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.224 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:12.224 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.450499+0000", 2023-07-02T14:07:12.224 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:07:12.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:12.225 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.450497+0000", 2023-07-02T14:07:12.225 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043010360000000003 2023-07-02T14:07:12.225 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.225 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.225 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:12.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454798+0000", 2023-07-02T14:07:12.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8073000000000001e-05 2023-07-02T14:07:12.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.226 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.226 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:12.226 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454816+0000", 2023-07-02T14:07:12.226 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3769999999999996e-06 2023-07-02T14:07:12.226 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:12.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454825+0000", 2023-07-02T14:07:12.227 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2797999999999998e-05 2023-07-02T14:07:12.227 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.227 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.227 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.227 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454878+0000", 2023-07-02T14:07:12.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0020999999999999e-05 2023-07-02T14:07:12.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.228 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.228 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:12.228 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454888+0000", 2023-07-02T14:07:12.228 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.110043313 2023-07-02T14:07:12.228 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564931+0000", 2023-07-02T14:07:12.229 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7185e-05 2023-07-02T14:07:12.229 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.229 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.229 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:12.229 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.565008+0000", 2023-07-02T14:07:12.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00014058199999999999 2023-07-02T14:07:12.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.230 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:12.230 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.565149+0000", 2023-07-02T14:07:12.230 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0093221559999999998 2023-07-02T14:07:12.230 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.230 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:12.231 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574471+0000", 2023-07-02T14:07:12.231 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099660410000000001 2023-07-02T14:07:12.231 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.231 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.231 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:12.231 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584437+0000", 2023-07-02T14:07:12.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2805000000000001e-05 2023-07-02T14:07:12.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.232 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.232 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:12.232 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584480+0000", 2023-07-02T14:07:12.232 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9393000000000001e-05 2023-07-02T14:07:12.232 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.233 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.233 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:12.233 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584509+0000", 2023-07-02T14:07:12.233 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.13400999399999999 2023-07-02T14:07:12.233 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.233 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:12.233 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.233 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.234 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.234 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20949 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 196608~2162688] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:07:12.234 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.454838+0000", 2023-07-02T14:07:12.234 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.390277089999998, 2023-07-02T14:07:12.234 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12983145400000001, 2023-07-02T14:07:12.234 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:12.234 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:12.234 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:12.235 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:12.235 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:12.235 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20949 2023-07-02T14:07:12.235 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.235 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:12.235 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.235 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:12.235 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454838+0000", 2023-07-02T14:07:12.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:12.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.236 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.236 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:12.236 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454838+0000", 2023-07-02T14:07:12.236 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:12.236 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:12.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.454837+0000", 2023-07-02T14:07:12.237 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003772821 2023-07-02T14:07:12.237 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.237 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.237 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:12.237 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458610+0000", 2023-07-02T14:07:12.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3013999999999999e-05 2023-07-02T14:07:12.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.238 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:12.238 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458623+0000", 2023-07-02T14:07:12.238 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2789999999999998e-06 2023-07-02T14:07:12.238 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.238 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:12.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458628+0000", 2023-07-02T14:07:12.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7630999999999998e-05 2023-07-02T14:07:12.239 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.239 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.239 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.239 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458676+0000", 2023-07-02T14:07:12.239 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1579999999999994e-06 2023-07-02T14:07:12.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.240 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:12.240 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.458685+0000", 2023-07-02T14:07:12.240 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.111536444 2023-07-02T14:07:12.240 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.240 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570221+0000", 2023-07-02T14:07:12.241 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000120844 2023-07-02T14:07:12.241 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.241 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.241 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:12.241 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570342+0000", 2023-07-02T14:07:12.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000149965 2023-07-02T14:07:12.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.242 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:12.242 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570492+0000", 2023-07-02T14:07:12.242 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014027843999999999 2023-07-02T14:07:12.242 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.242 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:12.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584520+0000", 2023-07-02T14:07:12.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8446000000000003e-05 2023-07-02T14:07:12.243 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.243 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.243 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:12.243 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584568+0000", 2023-07-02T14:07:12.243 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6771999999999998e-05 2023-07-02T14:07:12.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.244 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.244 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:12.244 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584625+0000", 2023-07-02T14:07:12.244 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4401999999999999e-05 2023-07-02T14:07:12.244 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.245 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.245 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:12.245 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584670+0000", 2023-07-02T14:07:12.245 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12983145400000001 2023-07-02T14:07:12.245 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.245 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:12.245 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.245 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.246 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.246 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20972 2.7 2:e3210266:::rbd_data.125de1449605.00000000000001a3:head [write 65536~4128768] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:07:12.246 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.463174+0000", 2023-07-02T14:07:12.246 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 46.381941249999997, 2023-07-02T14:07:12.246 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12719914800000001, 2023-07-02T14:07:12.246 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:12.246 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:12.246 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:12.247 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:12.247 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:12.247 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20972 2023-07-02T14:07:12.247 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.247 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:12.247 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.247 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:12.247 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.463174+0000", 2023-07-02T14:07:12.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:12.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.248 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.248 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:12.248 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.463174+0000", 2023-07-02T14:07:12.248 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:12.248 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:12.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.463173+0000", 2023-07-02T14:07:12.249 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069882069999999998 2023-07-02T14:07:12.249 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.249 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.249 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:12.249 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470161+0000", 2023-07-02T14:07:12.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0103999999999999e-05 2023-07-02T14:07:12.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.250 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.250 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:12.250 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470181+0000", 2023-07-02T14:07:12.250 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4259999999999997e-06 2023-07-02T14:07:12.250 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.251 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:12.251 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470189+0000", 2023-07-02T14:07:12.251 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0970999999999994e-05 2023-07-02T14:07:12.251 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.251 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.251 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.251 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470270+0000", 2023-07-02T14:07:12.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4389000000000001e-05 2023-07-02T14:07:12.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.252 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.252 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:12.252 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.470284+0000", 2023-07-02T14:07:12.252 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.10421862799999999 2023-07-02T14:07:12.252 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:12.253 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574503+0000", 2023-07-02T14:07:12.253 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2457e-05 2023-07-02T14:07:12.253 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.253 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.253 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:12.253 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574585+0000", 2023-07-02T14:07:12.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00013870899999999999 2023-07-02T14:07:12.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.254 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:12.254 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574724+0000", 2023-07-02T14:07:12.254 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012886246000000001 2023-07-02T14:07:12.254 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.254 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:12.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.587610+0000", 2023-07-02T14:07:12.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026883599999999999 2023-07-02T14:07:12.255 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.255 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.255 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:12.255 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.590298+0000", 2023-07-02T14:07:12.255 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1850000000000001e-05 2023-07-02T14:07:12.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.256 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:12.256 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.590340+0000", 2023-07-02T14:07:12.256 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2901999999999998e-05 2023-07-02T14:07:12.256 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:12.256 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:12.257 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:12.257 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.590373+0000", 2023-07-02T14:07:12.257 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.12719914800000001 2023-07-02T14:07:12.257 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.257 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:12.257 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.257 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:12.258 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:12.258 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:12.259 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:12.260 INFO:tasks.workunit.client.0.smithi163.stderr: Importing image diff: 37% complete... Importing image diff: 38% complete... Importing image diff: 62% complete... Importing image diff: 99% complete...2023-07-02T14:07:12.127+0000 7fa3cd6f1700 0 -- 172.21.15.163:0/1139503436 >> [v2:172.21.15.22:6808/99977,v1:172.21.15.22:6809/99977] conn(0x7fa3a8008780 msgr2=0x7fa3a800abd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:07:12.261 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:07:12.261 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:12.258+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:12.262 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:07:12.262 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2023-07-02T14:07:10.378845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11892805145,"num_objects":3557,"num_object_clones":2995,"num_object_copies":7114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":354,"num_objects_misplaced":401,"num_objects_unfound":0,"num_objects_dirty":3557,"num_whiteouts":0,"num_read":88256,"num_read_kb":15199597,"num_write":27213,"num_write_kb":11540672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2862,"num_bytes_recovered":9703530998,"num_keys_recovered":186,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26888,"ondisk_log_size":26888,"up":18,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":21655396,"kb_used_data":21313668,"kb_used_omap":229,"kb_used_meta":341466,"kb_avail":355831964,"statfs":{"total":386547056640,"available":364371931136,"internally_reserved":0,"allocated":21825196032,"data_stored":25424386923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":349661406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22749346,"num_objects":16,"num_object_clones":15,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":136,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1953,"num_read_kb":671410,"num_write":89,"num_write_kb":22250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":922657902,"num_keys_recovered":8,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.088950"},"pg_stats":[{"pgid":"2.7","version":"212'3647","reported_seq":9311,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-07-02T14:07:05.528711+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:07:05.528711+0000","last_peered":"2023-07-02T14:07:05.528711+0000","last_clean":"2023-07-02T14:07:05.528711+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:07:05.528711+0000","last_undegraded":"2023-07-02T14:07:05.528711+0000","last_fullsized":"2023-07-02T14:07:05.528711+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3647,"ondisk_log_size":3647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1570191828,"num_objects":466,"num_object_clones":394,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":57412,"num_read_kb":1894335,"num_write":3056,"num_write_kb":1525929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"215'3058","reported_seq":5145,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:10.324843+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:07:10.324843+0000","last_peered":"2023-07-02T14:07:10.324843+0000","last_clean":"2023-07-02T14:07:10.324843+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:07:10.324843+0000","last_undegraded":"2023-07-02T14:07:10.324843+0000","last_fullsized":"2023-07-02T14:07:10.324843+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3058,"ondisk_log_size":3058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535489458,"num_objects":482,"num_object_clones":414,"num_object_copies":964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":0,"num_read":1926,"num_read_kb":2091895,"num_write":2644,"num_write_kb":1489957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"210'3193","reported_seq":3825,"reported_epoch":213,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-07-02T14:07:08.095386+0000","last_change":"2023-07-02T14:07:07.848445+0000","last_active":"2023-07-02T14:07:08.095386+0000","last_peered":"2023-07-02T14:07:08.095386+0000","last_clean":"2023-07-02T14:07:01.043240+0000","last_became_active":"2023-07-02T14:07:04.030791+0000","last_became_peered":"2023-07-02T14:07:04.030791+0000","last_unstale":"2023-07-02T14:07:08.095386+0000","last_undegraded":"2023-07-02T14:07:03.107328+0000","last_fullsized":"2023-07-02T14:07:03.101623+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3193,"ondisk_log_size":3193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1516729292,"num_objects":421,"num_object_clones":355,"num_object_copies":842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":401,"num_objects_unfound":0,"num_objects_dirty":421,"num_whiteouts":0,"num_read":572,"num_read_kb":1826967,"num_write":2838,"num_write_kb":1467491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":247,"num_bytes_recovered":909742242,"num_keys_recovered":8,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":401},{"shards":"3","objects":18}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"215'6476","reported_seq":8964,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:10.334213+0000","last_change":"2023-07-02T14:06:29.572710+0000","last_active":"2023-07-02T14:07:10.334213+0000","last_peered":"2023-07-02T14:07:10.334213+0000","last_clean":"2023-07-02T14:07:10.334213+0000","last_became_active":"2023-07-02T14:06:29.569824+0000","last_became_peered":"2023-07-02T14:06:29.569824+0000","last_unstale":"2023-07-02T14:07:10.334213+0000","last_undegraded":"2023-07-02T14:07:10.334213+0000","last_fullsized":"2023-07-02T14:07:10.334213+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":6476,"ondisk_log_size":6476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672852930,"num_objects":488,"num_object_clones":413,"num_object_copies":976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":22953,"num_read_kb":2152050,"num_write":9580,"num_write_kb":1632373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":113,"reported_epoch":212,"state":"active+clean","last_fresh":"2023-07-02T14:07:05.358697+0000","last_change":"2023-07-02T14:07:02.103009+0000","last_active":"2023-07-02T14:07:05.358697+0000","last_peered":"2023-07-02T14:07:05.358697+0000","last_clean":"2023-07-02T14:07:05.358697+0000","last_became_active":"2023-07-02T14:07:02.102654+0000","last_became_peered":"2023-07-02T14:07:02.102654+0000","last_unstale":"2023-07-02T14:07:05.358697+0000","last_undegraded":"2023-07-02T14:07:05.358697+0000","last_fullsized":"2023-07-02T14:07:05.358697+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"215'2964","reported_seq":4463,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:10.302466+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:07:10.302466+0000","last_peered":"2023-07-02T14:07:10.302466+0000","last_clean":"2023-07-02T14:07:10.302466+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:07:10.302466+0000","last_undegraded":"2023-07-02T14:07:10.302466+0000","last_fullsized":"2023-07-02T14:07:10.302466+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2964,"ondisk_log_size":2964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1658680446,"num_objects":491,"num_object_clones":414,"num_object_copies":982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":0,"num_read":1509,"num_read_kb":2028229,"num_write":2550,"num_write_kb":1608728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"215'2867","reported_seq":3734,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:10.282420+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:07:10.282420+0000","last_peered":"2023-07-02T14:07:10.282420+0000","last_clean":"2023-07-02T14:07:10.282420+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:07:10.282420+0000","last_undegraded":"2023-07-02T14:07:10.282420+0000","last_fullsized":"2023-07-02T14:07:10.282420+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2867,"ondisk_log_size":2867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1564710484,"num_objects":455,"num_object_clones":382,"num_object_copies":910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":455,"num_whiteouts":0,"num_read":949,"num_read_kb":1945167,"num_write":2485,"num_write_kb":1518698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"215'2325","reported_seq":5128,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:10.305571+0000","last_change":"2023-07-02T14:07:09.383558+0000","last_active":"2023-07-02T14:07:10.305571+0000","last_peered":"2023-07-02T14:07:10.305571+0000","last_clean":"2023-07-02T14:07:10.305571+0000","last_became_active":"2023-07-02T14:07:09.382190+0000","last_became_peered":"2023-07-02T14:07:09.382190+0000","last_unstale":"2023-07-02T14:07:10.305571+0000","last_undegraded":"2023-07-02T14:07:10.305571+0000","last_fullsized":"2023-07-02T14:07:10.305571+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2325,"ondisk_log_size":2325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1230505737,"num_objects":402,"num_object_clones":325,"num_object_copies":804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":1898,"num_read_kb":1730263,"num_write":2000,"num_write_kb":1186689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":1578968698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"215'2358","reported_seq":3456,"reported_epoch":215,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-07-02T14:07:10.279927+0000","last_change":"2023-07-02T14:07:04.022783+0000","last_active":"2023-07-02T14:07:10.279927+0000","last_peered":"2023-07-02T14:07:10.279927+0000","last_clean":"2023-07-02T14:07:01.026611+0000","last_became_active":"2023-07-02T14:07:04.018152+0000","last_became_peered":"2023-07-02T14:07:04.018152+0000","last_unstale":"2023-07-02T14:07:10.279927+0000","last_undegraded":"2023-07-02T14:07:03.110294+0000","last_fullsized":"2023-07-02T14:07:03.103717+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2358,"ondisk_log_size":2358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1143644970,"num_objects":352,"num_object_clones":298,"num_object_copies":704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":1037,"num_read_kb":1530691,"num_write":2060,"num_write_kb":1110807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":271880192,"num_keys_recovered":4,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":346},{"shards":"1","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":11892805145,"num_objects":3557,"num_object_clones":2995,"num_object_copies":7114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":354,"num_objects_misplaced":401,"num_objects_unfound":0,"num_objects_dirty":3557,"num_whiteouts":0,"num_read":88256,"num_read_kb":15199597,"num_write":27213,"num_write_kb":11540672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2862,"num_bytes_recovered":9703530998,"num_keys_recovered":186,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25840590848,"data_stored":30051234943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":26888,"ondisk_log_size":26888,"up":16,"acting":14,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640323,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640323,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5235024,"kb_used_data":5143176,"kb_used_omap":0,"kb_used_meta":91840,"kb_avail":89136816,"statfs":{"total":96636764160,"available":91276099584,"internally_reserved":0,"allocated":5266612224,"data_stored":6072961790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":94044160},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.307}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44,"15min":0.46400000000000002},"min":{"1min":0.28399999999999997,"5min":0.215,"15min":0.215},"max":{"1min":0.71499999999999997,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.45000000000000001},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.32800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66800000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.39900000000000002}]},{"osd":3,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.38300000000000001},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45900000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.36799999999999999,"15min":0.36299999999999999},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.48799999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.56200000000000006}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.44900000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.433,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59099999999999997,"5min":1.026,"15min":1.026},"last":0.42899999999999999}]}]},{"osd":3,"up_from":133,"seq":571230650394,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4560208,"kb_used_data":4490568,"kb_used_omap":229,"kb_used_meta":69402,"kb_avail":89811632,"statfs":{"total":96636764160,"available":91967111168,"internally_reserved":0,"allocated":4598341632,"data_stored":5339881520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":71067870},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.443}]},{"osd":1,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.42299999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.504}]},{"osd":4,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.49099999999999999}]},{"osd":5,"last update":"Sun Jul 2 14:06:05 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.52000000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640323,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3192604,"kb_used_data":3138708,"kb_used_omap":0,"kb_used_meta":53888,"kb_avail":91179236,"statfs":{"total":96636764160,"available":93367537664,"internally_reserved":0,"allocated":3214036992,"data_stored":3691892043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55181312},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.28799999999999998}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.42499999999999999,"15min":0.374},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.376,"5min":0.33500000000000002,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.22500000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:07:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.91100000000000003,"5min":1.929,"15min":1.929},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.54800000000000004,"15min":0.55600000000000005},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93999999999999995,"5min":2,"15min":2},"last":0.35999999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.40899999999999997}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68400000000000005,"5min":1.071,"15min":1.071},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":0.432}]}]},{"osd":0,"up_from":0,"seq":712964571152,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":11,"seq":47244640323,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8667560,"kb_used_data":8541216,"kb_used_omap":0,"kb_used_meta":126336,"kb_avail":85704280,"statfs":{"total":96636764160,"available":87761182720,"internally_reserved":0,"allocated":8746205184,"data_stored":10319651570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":129368064},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.32600000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.247},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.224,"5min":0.184,"15min":0.184},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.33800000000000002}]},{"osd":3,"last update":"Sun Jul 2 14:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.49099999999999999,"15min":0.47299999999999998},"min":{"1min":0.30299999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.38600000000000001}]},{"osd":4,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71899999999999997,"5min":1.524,"15min":1.524},"last":0.51000000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.80300000000000005,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84399999999999997,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.42999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3981160448,"data_stored":4543561053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8745132032,"data_stored":10339602553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3211833344,"data_stored":3699260114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4595769344,"data_stored":5338870610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5306695680,"data_stored":6129940613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:07:12.359 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:12.359+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:12.432 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:12.432 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:07:12.433 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:07:12.433 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:12.444 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:07:12.460 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:12.459+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:12.560 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:12.560+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:12.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:07:12.566 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-07-02T14:07:12.572 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:12.572 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:07:12.572 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:07:12.572 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:07:12.573 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:12.584 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:07:12.667 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:12.666+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:12.710 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:07:12.762 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:12.761+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:12.763 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:12.763 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:12.774 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:07:12.800 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:07:12.820 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:07:12.840 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:07:12.858 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:07:12.862 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:12.861+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:12.877 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:07:12.879 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:07:12.879 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:07:12.879 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:07:12.879 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:07:12.879 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:07:12.879 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:07:12.941 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:12.942 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:12.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:07:12.997 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:12.996+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:13.064 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:13.063+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:13.078 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:13.078 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:13.090 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:07:13.102 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is set 2023-07-02T14:07:13.165 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:13.164+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:13.239 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:13.239 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:13.252 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:07:13.293 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:13.292+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:13.365 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:07:13.364+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:07:13.429 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:13.429 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:13.441 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:07:13.482 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:13.481+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:13.567 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:07:13.566+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:07:13.626 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:13.626 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:13.637 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:07:13.668 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:13.667+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:13.769 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:13.768+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:13.808 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:13.808 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:13.819 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:07:13.870 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:13.869+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:13.966 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:13.967 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:13.971 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:13.970+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:13.979 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:07:14.072 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:14.071+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:14.136 INFO:tasks.workunit.client.0.smithi163.stderr: Importing image diff: 100% complete...done. 2023-07-02T14:07:14.145 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:07:14.144+0000 7fa3cd6f1700 0 -- 172.21.15.163:0/1139503436 >> [v2:172.21.15.22:6808/99977,v1:172.21.15.22:6809/99977] conn(0x7fa3b0036c20 msgr2=0x7fa3b0039050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:07:14.155 INFO:tasks.workunit.client.0.smithi163.stderr:++ rbd export b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap1 - 2023-07-02T14:07:14.156 INFO:tasks.workunit.client.0.smithi163.stderr:++ md5sum 2023-07-02T14:07:14.156 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2023-07-02T14:07:14.172 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:14.173 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:14.173 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:14.172+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:14.185 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:07:14.225 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 12% complete... Exporting image: 14% complete...2023-07-02T14:07:14.224+0000 7f3f1ffa1700 0 -- 172.21.15.163:0/4202108232 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7f3efc011f20 msgr2=0x7f3efc014310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:07:14.231 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:07:14.230+0000 7f3f207a2700 0 -- 172.21.15.163:0/4202108232 >> [v2:172.21.15.22:6808/99977,v1:172.21.15.22:6809/99977] conn(0x7f3f04036c00 msgr2=0x7f3f04039030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:07:14.273 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:07:14.272+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:07:14.374 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:14.373+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:14.383 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:14.383 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:14.397 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:07:14.475 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:07:14.474+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:07:14.576 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:14.575+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:14.581 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:14.581 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:14.593 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:07:14.676 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:07:14.675+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:07:14.739 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:14.739 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:14.752 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:07:14.778 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:14.776+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:14.878 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:14.877+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:14.894 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:07:14.918 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:14.919 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:14.931 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:07:14.979 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:07:14.978+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:07:15.045 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:07:15.062 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:07:15.075 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:15.075 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:15.080 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:15.079+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:15.081 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:07:15.088 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:07:15.099 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:07:15.118 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-07-02T14:07:15.121 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:07:15.143 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:07:15.145 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:07:15.145 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:07:15.145 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:07:15.145 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:07:15.145 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:07:15.145 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:07:15.181 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:15.180+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:15.264 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:07:15.277 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:15.277 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:07:15.277 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:07:15.277 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:07:15.278 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:15.281 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:15.281+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:15.288 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:07:15.383 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:15.382+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:15.472 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:07:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:07:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:07:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:410 2.5 2:bbaed619:::rbd_data.125de1449605.0000000000000109:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:07:15.473 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:07:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 79.015299325000001, 2023-07-02T14:07:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003, 2023-07-02T14:07:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:07:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:07:15.474 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:07:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 410 2023-07-02T14:07:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:07:15.475 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:07:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.476 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450567+0000", 2023-07-02T14:07:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.635e-06 2023-07-02T14:07:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:15.477 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450568+0000", 2023-07-02T14:07:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5970000000000002e-06 2023-07-02T14:07:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450573+0000", 2023-07-02T14:07:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3379999999999999e-06 2023-07-02T14:07:15.478 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450575+0000", 2023-07-02T14:07:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9458000000000001e-05 2023-07-02T14:07:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.479 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450615+0000", 2023-07-02T14:07:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0569999999999993e-06 2023-07-02T14:07:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.480 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450623+0000", 2023-07-02T14:07:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.63968709199999996 2023-07-02T14:07:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090310+0000", 2023-07-02T14:07:15.481 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020157399999999999 2023-07-02T14:07:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.090512+0000", 2023-07-02T14:07:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0027739190000000001 2023-07-02T14:07:15.482 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093285+0000", 2023-07-02T14:07:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64271839200000003 2023-07-02T14:07:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.483 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.6552.0:416 2.5 2:a3eaf584:::rbd_data.125de1449605.000000000000010f:30 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e170)", 2023-07-02T14:07:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:07:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 79.015269892000006, 2023-07-02T14:07:15.484 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002, 2023-07-02T14:07:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:07:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.6552", 2023-07-02T14:07:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1999128455", 2023-07-02T14:07:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 416 2023-07-02T14:07:15.485 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:07:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.486 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:07:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:15.487 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450596+0000", 2023-07-02T14:07:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.34e-06 2023-07-02T14:07:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450599+0000", 2023-07-02T14:07:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1549999999999998e-06 2023-07-02T14:07:15.488 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450604+0000", 2023-07-02T14:07:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:07:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.489 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450607+0000", 2023-07-02T14:07:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5717999999999999e-05 2023-07-02T14:07:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450643+0000", 2023-07-02T14:07:15.490 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1619999999999999e-06 2023-07-02T14:07:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:56.450648+0000", 2023-07-02T14:07:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64292291800000001 2023-07-02T14:07:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.491 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093571+0000", 2023-07-02T14:07:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000115628 2023-07-02T14:07:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:15.492 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.093687+0000", 2023-07-02T14:07:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002885552 2023-07-02T14:07:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:05:57.096572+0000", 2023-07-02T14:07:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.64597563800000002 2023-07-02T14:07:15.493 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:18561 2.0 2:1c72c23d:::rbd_object_map.125de1449605:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e176)", 2023-07-02T14:07:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:07:15.494 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 70.216819220000005, 2023-07-02T14:07:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744, 2023-07-02T14:07:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:15.495 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 18561 2023-07-02T14:07:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:07:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:15.496 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:07:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:15.497 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249047+0000", 2023-07-02T14:07:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.563e-06 2023-07-02T14:07:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249050+0000", 2023-07-02T14:07:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4500000000000001e-06 2023-07-02T14:07:15.498 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249057+0000", 2023-07-02T14:07:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1860000000000002e-06 2023-07-02T14:07:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.499 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249061+0000", 2023-07-02T14:07:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00091153399999999998 2023-07-02T14:07:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.500 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.249972+0000", 2023-07-02T14:07:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7156999999999997e-05 2023-07-02T14:07:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.250010+0000", 2023-07-02T14:07:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006641816 2023-07-02T14:07:15.501 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256651+0000", 2023-07-02T14:07:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1968999999999998e-05 2023-07-02T14:07:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.502 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.256683+0000", 2023-07-02T14:07:15.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012463952 2023-07-02T14:07:15.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.503 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.503 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.503 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269147+0000", 2023-07-02T14:07:15.503 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2855000000000006e-05 2023-07-02T14:07:15.503 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.269220+0000", 2023-07-02T14:07:15.504 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010870418999999999 2023-07-02T14:07:15.504 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.504 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.504 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.504 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280091+0000", 2023-07-02T14:07:15.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1921999999999997e-05 2023-07-02T14:07:15.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.505 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.505 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.505 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.280123+0000", 2023-07-02T14:07:15.505 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011861682 2023-07-02T14:07:15.505 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.506 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.291984+0000", 2023-07-02T14:07:15.506 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5259000000000006e-05 2023-07-02T14:07:15.506 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.506 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.506 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.292059+0000", 2023-07-02T14:07:15.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079709359999999996 2023-07-02T14:07:15.507 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.507 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.507 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.507 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300030+0000", 2023-07-02T14:07:15.507 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0729999999999998e-05 2023-07-02T14:07:15.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.508 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.508 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.300071+0000", 2023-07-02T14:07:15.508 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072785690000000004 2023-07-02T14:07:15.508 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.508 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307350+0000", 2023-07-02T14:07:15.509 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1948999999999999e-05 2023-07-02T14:07:15.509 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.509 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.509 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.509 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.307412+0000", 2023-07-02T14:07:15.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055785779999999998 2023-07-02T14:07:15.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.510 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.510 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.510 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.312990+0000", 2023-07-02T14:07:15.510 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0957000000000002e-05 2023-07-02T14:07:15.510 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.313021+0000", 2023-07-02T14:07:15.511 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049246400000000001 2023-07-02T14:07:15.511 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.511 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.511 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.511 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317946+0000", 2023-07-02T14:07:15.512 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9346e-05 2023-07-02T14:07:15.512 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.512 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.512 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.512 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.317975+0000", 2023-07-02T14:07:15.513 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049649469999999999 2023-07-02T14:07:15.513 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.513 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.513 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.513 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322940+0000", 2023-07-02T14:07:15.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9017000000000002e-05 2023-07-02T14:07:15.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.514 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.514 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.322989+0000", 2023-07-02T14:07:15.514 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065907029999999998 2023-07-02T14:07:15.514 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.514 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329580+0000", 2023-07-02T14:07:15.515 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5879999999999999e-05 2023-07-02T14:07:15.515 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.515 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.515 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.515 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.329656+0000", 2023-07-02T14:07:15.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060166960000000002 2023-07-02T14:07:15.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.516 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.516 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.516 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335672+0000", 2023-07-02T14:07:15.516 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8963e-05 2023-07-02T14:07:15.516 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.517 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.517 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.517 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.335701+0000", 2023-07-02T14:07:15.517 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098989560000000004 2023-07-02T14:07:15.517 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345600+0000", 2023-07-02T14:07:15.518 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5395999999999995e-05 2023-07-02T14:07:15.518 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.518 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.518 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.518 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.345666+0000", 2023-07-02T14:07:15.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046589530000000004 2023-07-02T14:07:15.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.519 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.519 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.519 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350325+0000", 2023-07-02T14:07:15.519 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2459000000000002e-05 2023-07-02T14:07:15.519 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.520 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.350357+0000", 2023-07-02T14:07:15.520 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042054290000000001 2023-07-02T14:07:15.520 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.520 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.520 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354563+0000", 2023-07-02T14:07:15.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5612999999999999e-05 2023-07-02T14:07:15.521 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.521 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.521 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.521 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.354598+0000", 2023-07-02T14:07:15.521 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070967449999999998 2023-07-02T14:07:15.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.522 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.522 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361695+0000", 2023-07-02T14:07:15.522 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9314000000000001e-05 2023-07-02T14:07:15.522 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.522 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.523 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.523 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.361724+0000", 2023-07-02T14:07:15.523 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007391028 2023-07-02T14:07:15.523 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.523 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369115+0000", 2023-07-02T14:07:15.524 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9163000000000002e-05 2023-07-02T14:07:15.524 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.524 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.524 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.524 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.369184+0000", 2023-07-02T14:07:15.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056532500000000003 2023-07-02T14:07:15.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.525 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.525 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.525 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374838+0000", 2023-07-02T14:07:15.525 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5297999999999999e-05 2023-07-02T14:07:15.525 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.526 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.374883+0000", 2023-07-02T14:07:15.526 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077292189999999998 2023-07-02T14:07:15.526 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.526 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.526 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382612+0000", 2023-07-02T14:07:15.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1102000000000001e-05 2023-07-02T14:07:15.527 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.527 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.527 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.527 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.382653+0000", 2023-07-02T14:07:15.527 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058515989999999999 2023-07-02T14:07:15.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.528 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.528 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388505+0000", 2023-07-02T14:07:15.528 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9196000000000006e-05 2023-07-02T14:07:15.528 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.528 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.388574+0000", 2023-07-02T14:07:15.529 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084245339999999991 2023-07-02T14:07:15.529 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.529 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.529 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.529 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.396999+0000", 2023-07-02T14:07:15.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0596999999999998e-05 2023-07-02T14:07:15.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.530 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.530 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.530 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.397039+0000", 2023-07-02T14:07:15.530 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0084911339999999991 2023-07-02T14:07:15.530 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.531 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405530+0000", 2023-07-02T14:07:15.531 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7555000000000001e-05 2023-07-02T14:07:15.531 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.531 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.531 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:15.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405578+0000", 2023-07-02T14:07:15.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000229607 2023-07-02T14:07:15.532 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.532 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.532 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:15.532 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.405808+0000", 2023-07-02T14:07:15.532 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0026919940000000001 2023-07-02T14:07:15.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:15.533 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.408500+0000", 2023-07-02T14:07:15.533 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053065059999999999 2023-07-02T14:07:15.533 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.533 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.533 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:15.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413806+0000", 2023-07-02T14:07:15.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0490000000000001e-05 2023-07-02T14:07:15.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.534 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.534 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:15.534 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413837+0000", 2023-07-02T14:07:15.534 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000100386 2023-07-02T14:07:15.534 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.535 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.535 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:15.535 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:05.413937+0000", 2023-07-02T14:07:15.535 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.164889744 2023-07-02T14:07:15.535 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.535 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:15.535 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.535 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.536 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.536 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20768 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch ping cookie 140544751726928 gen 52] snapc 0=[] ondisk+write+known_if_redirected e183)", 2023-07-02T14:07:15.536 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:07:15.536 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 56.243451952000001, 2023-07-02T14:07:15.536 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3751256970000001, 2023-07-02T14:07:15.536 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:15.537 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:07:15.537 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:15.537 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:15.537 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:15.537 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20768 2023-07-02T14:07:15.537 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.537 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:15.537 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:15.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:07:15.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:15.538 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.538 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.538 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:15.538 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:07:15.538 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:15.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.539 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:15.539 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222414+0000", 2023-07-02T14:07:15.539 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7829999999999998e-06 2023-07-02T14:07:15.539 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.539 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:15.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222419+0000", 2023-07-02T14:07:15.540 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1380000000000004e-06 2023-07-02T14:07:15.540 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.540 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.540 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:15.540 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222425+0000", 2023-07-02T14:07:15.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2519999999999996e-06 2023-07-02T14:07:15.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.541 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.541 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:15.541 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222431+0000", 2023-07-02T14:07:15.541 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2157999999999994e-05 2023-07-02T14:07:15.541 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.542 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.542 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.542 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222523+0000", 2023-07-02T14:07:15.542 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5147000000000001e-05 2023-07-02T14:07:15.542 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:15.543 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222538+0000", 2023-07-02T14:07:15.543 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30105419700000002 2023-07-02T14:07:15.543 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.543 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.543 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523592+0000", 2023-07-02T14:07:15.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2316e-05 2023-07-02T14:07:15.544 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.544 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.544 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:15.544 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523605+0000", 2023-07-02T14:07:15.544 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.073658953 2023-07-02T14:07:15.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.545 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.545 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597264+0000", 2023-07-02T14:07:15.545 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000213415 2023-07-02T14:07:15.545 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.545 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:15.546 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597477+0000", 2023-07-02T14:07:15.546 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3034999999999995e-05 2023-07-02T14:07:15.546 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.546 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.546 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:15.547 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597540+0000", 2023-07-02T14:07:15.547 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3751256970000001 2023-07-02T14:07:15.547 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.547 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:15.547 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.547 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.547 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.548 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20769 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 53] snapc 0=[] ondisk+write+known_if_redirected e184)", 2023-07-02T14:07:15.548 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:07:15.548 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 56.243089497, 2023-07-02T14:07:15.548 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.380884139, 2023-07-02T14:07:15.548 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:15.548 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:07:15.548 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:15.549 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:15.549 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:15.549 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20769 2023-07-02T14:07:15.549 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.549 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:15.549 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:15.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:07:15.550 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:15.550 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.550 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.550 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:15.550 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222777+0000", 2023-07-02T14:07:15.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:15.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.551 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.551 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:15.551 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222776+0000", 2023-07-02T14:07:15.551 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9710000000000003e-06 2023-07-02T14:07:15.551 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:15.552 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222781+0000", 2023-07-02T14:07:15.552 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9899999999999992e-06 2023-07-02T14:07:15.552 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.552 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.552 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:15.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222791+0000", 2023-07-02T14:07:15.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4240000000000003e-06 2023-07-02T14:07:15.553 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.553 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.553 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:15.553 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222795+0000", 2023-07-02T14:07:15.553 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2366000000000001e-05 2023-07-02T14:07:15.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.554 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222828+0000", 2023-07-02T14:07:15.554 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4790000000000001e-06 2023-07-02T14:07:15.554 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.554 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.554 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:15.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.222834+0000", 2023-07-02T14:07:15.555 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.30084589299999998 2023-07-02T14:07:15.555 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.555 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.555 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.555 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523680+0000", 2023-07-02T14:07:15.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.87e-06 2023-07-02T14:07:15.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.556 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.556 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:15.556 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523685+0000", 2023-07-02T14:07:15.556 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.074111042 2023-07-02T14:07:15.556 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.557 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.557 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.557 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597796+0000", 2023-07-02T14:07:15.557 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0411000000000001e-05 2023-07-02T14:07:15.557 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.558 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.558 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:15.558 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.597856+0000", 2023-07-02T14:07:15.559 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058045340000000001 2023-07-02T14:07:15.559 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.559 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.559 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:15.560 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.603661+0000", 2023-07-02T14:07:15.560 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.380884139 2023-07-02T14:07:15.560 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.561 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:15.561 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.561 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.562 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.562 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20770 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 54] snapc 0=[] ondisk+write+known_if_redirected e185)", 2023-07-02T14:07:15.562 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:07:15.563 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 55.941872386999997, 2023-07-02T14:07:15.563 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0799819349999999, 2023-07-02T14:07:15.563 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:15.563 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:07:15.564 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:15.564 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:15.564 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:15.564 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20770 2023-07-02T14:07:15.564 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.564 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:15.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:15.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:07:15.565 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:15.565 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.565 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.565 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:15.565 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523994+0000", 2023-07-02T14:07:15.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:15.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.566 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.566 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:15.566 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.523992+0000", 2023-07-02T14:07:15.566 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6720000000000004e-06 2023-07-02T14:07:15.566 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:15.567 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524000+0000", 2023-07-02T14:07:15.567 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3711e-05 2023-07-02T14:07:15.567 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.567 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.567 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:15.568 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524014+0000", 2023-07-02T14:07:15.568 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7849999999999992e-06 2023-07-02T14:07:15.568 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.568 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.568 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:15.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524022+0000", 2023-07-02T14:07:15.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2366000000000001e-05 2023-07-02T14:07:15.569 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.569 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.569 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.569 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524054+0000", 2023-07-02T14:07:15.569 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8169999999999998e-06 2023-07-02T14:07:15.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:15.570 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:19.524059+0000", 2023-07-02T14:07:15.570 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0748338850000001 2023-07-02T14:07:15.570 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.570 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.570 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.598893+0000", 2023-07-02T14:07:15.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116968 2023-07-02T14:07:15.571 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.571 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.571 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:15.571 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.599010+0000", 2023-07-02T14:07:15.571 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049663370000000004 2023-07-02T14:07:15.572 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.572 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.572 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:15.572 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:20.603976+0000", 2023-07-02T14:07:15.572 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0799819349999999 2023-07-02T14:07:15.572 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.573 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:15.573 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.573 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.573 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.573 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20833 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 55] snapc 0=[] ondisk+write+known_if_redirected e189)", 2023-07-02T14:07:15.573 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:07:15.573 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.933753082000003, 2023-07-02T14:07:15.574 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.029349206, 2023-07-02T14:07:15.574 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:15.574 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "started", 2023-07-02T14:07:15.574 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:15.574 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:15.574 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:15.574 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20833 2023-07-02T14:07:15.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.575 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:15.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.575 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:15.575 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:07:15.575 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:15.575 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.575 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.576 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:15.576 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532113+0000", 2023-07-02T14:07:15.576 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:15.576 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.577 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:15.577 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532112+0000", 2023-07-02T14:07:15.577 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7590000000000003e-06 2023-07-02T14:07:15.577 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.577 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:15.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532120+0000", 2023-07-02T14:07:15.578 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8130000000000001e-05 2023-07-02T14:07:15.578 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.578 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.578 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:15.578 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532139+0000", 2023-07-02T14:07:15.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3299999999999999e-06 2023-07-02T14:07:15.579 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.579 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.579 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:15.579 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532147+0000", 2023-07-02T14:07:15.579 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9679999999999998e-05 2023-07-02T14:07:15.580 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.580 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.580 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.580 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532207+0000", 2023-07-02T14:07:15.580 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0648e-05 2023-07-02T14:07:15.581 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.581 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.581 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op must wait for map", 2023-07-02T14:07:15.581 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.532217+0000", 2023-07-02T14:07:15.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002082455 2023-07-02T14:07:15.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.582 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.582 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.582 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.534300+0000", 2023-07-02T14:07:15.582 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0710000000000001e-05 2023-07-02T14:07:15.582 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:15.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:24.534310+0000", 2023-07-02T14:07:15.583 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.013227235 2023-07-02T14:07:15.583 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.583 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.583 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.583 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.547538+0000", 2023-07-02T14:07:15.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00021904300000000001 2023-07-02T14:07:15.584 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.584 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.584 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:15.584 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.547757+0000", 2023-07-02T14:07:15.584 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013705784 2023-07-02T14:07:15.585 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.585 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.585 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:15.585 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561462+0000", 2023-07-02T14:07:15.585 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.029349206 2023-07-02T14:07:15.585 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.585 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:15.585 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.586 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.586 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.586 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20865 2.7 2:ff0f5f99:::rbd_data.125de1449605.0000000000000006:head [write 131072~8192] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:07:15.586 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:07:15.586 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.18913001, 2023-07-02T14:07:15.586 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.288115762, 2023-07-02T14:07:15.587 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:15.587 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:15.587 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:15.587 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:15.587 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:15.587 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20865 2023-07-02T14:07:15.587 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.587 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:15.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.588 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:15.588 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:07:15.588 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:15.588 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.588 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:15.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276736+0000", 2023-07-02T14:07:15.589 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:15.589 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.589 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.589 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:15.589 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276735+0000", 2023-07-02T14:07:15.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3273e-05 2023-07-02T14:07:15.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.590 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:15.590 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276748+0000", 2023-07-02T14:07:15.590 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3895e-05 2023-07-02T14:07:15.590 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.590 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:15.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276762+0000", 2023-07-02T14:07:15.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.896e-06 2023-07-02T14:07:15.591 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.591 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.591 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:15.591 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276768+0000", 2023-07-02T14:07:15.591 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5850999999999999e-05 2023-07-02T14:07:15.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.592 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.592 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.592 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276814+0000", 2023-07-02T14:07:15.592 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5290000000000003e-06 2023-07-02T14:07:15.592 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:15.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.276821+0000", 2023-07-02T14:07:15.593 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.27225312600000001 2023-07-02T14:07:15.593 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.593 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.593 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.593 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549074+0000", 2023-07-02T14:07:15.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000151223 2023-07-02T14:07:15.594 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.594 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.594 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:15.594 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549226+0000", 2023-07-02T14:07:15.594 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00024865400000000001 2023-07-02T14:07:15.595 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.595 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.595 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:15.595 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.549474+0000", 2023-07-02T14:07:15.595 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012039185000000001 2023-07-02T14:07:15.596 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.596 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.596 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:15.596 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561513+0000", 2023-07-02T14:07:15.596 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003221327 2023-07-02T14:07:15.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.597 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:15.597 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564735+0000", 2023-07-02T14:07:15.597 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2026000000000005e-05 2023-07-02T14:07:15.597 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.597 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:15.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564807+0000", 2023-07-02T14:07:15.598 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5303000000000003e-05 2023-07-02T14:07:15.598 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.598 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.598 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:15.598 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564852+0000", 2023-07-02T14:07:15.599 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.288115762 2023-07-02T14:07:15.599 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.599 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:15.599 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.599 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.599 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.599 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20866 2.7 2:f555ee6a:::rbd_data.125de1449605.0000000000000007:head [write 131072~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:07:15.599 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:07:15.600 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.183436688999997, 2023-07-02T14:07:15.600 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.28773701200000001, 2023-07-02T14:07:15.600 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:15.600 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:15.600 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:15.600 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:15.601 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:15.601 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20866 2023-07-02T14:07:15.601 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.601 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:15.601 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.601 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:15.601 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:07:15.601 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:15.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.602 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.602 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:15.602 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282430+0000", 2023-07-02T14:07:15.602 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:15.602 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:15.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282428+0000", 2023-07-02T14:07:15.603 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8469999999999998e-05 2023-07-02T14:07:15.603 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.603 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.603 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:15.603 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282517+0000", 2023-07-02T14:07:15.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2974e-05 2023-07-02T14:07:15.604 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.604 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.604 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:15.604 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282530+0000", 2023-07-02T14:07:15.604 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4530000000000001e-06 2023-07-02T14:07:15.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:15.605 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282535+0000", 2023-07-02T14:07:15.605 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1926999999999999e-05 2023-07-02T14:07:15.605 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.605 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.605 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282587+0000", 2023-07-02T14:07:15.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3083e-05 2023-07-02T14:07:15.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.606 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.606 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:15.606 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.282600+0000", 2023-07-02T14:07:15.606 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.270692971 2023-07-02T14:07:15.606 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.607 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553293+0000", 2023-07-02T14:07:15.607 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017065599999999999 2023-07-02T14:07:15.607 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.607 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.607 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:15.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553464+0000", 2023-07-02T14:07:15.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00026504399999999998 2023-07-02T14:07:15.608 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.608 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.608 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:15.608 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.553729+0000", 2023-07-02T14:07:15.608 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.011173008 2023-07-02T14:07:15.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.609 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:15.609 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.564902+0000", 2023-07-02T14:07:15.609 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051242600000000003 2023-07-02T14:07:15.609 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.609 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:15.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570026+0000", 2023-07-02T14:07:15.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8098999999999999e-05 2023-07-02T14:07:15.610 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.610 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.610 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:15.610 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570104+0000", 2023-07-02T14:07:15.610 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2323000000000006e-05 2023-07-02T14:07:15.611 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.611 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.611 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:15.611 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.570167+0000", 2023-07-02T14:07:15.611 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.28773701200000001 2023-07-02T14:07:15.611 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.611 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:15.612 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.612 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.612 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.612 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20875 2.7 2:e3210266:::rbd_data.125de1449605.00000000000001a3:head [write 0~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:07:15.613 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:07:15.613 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.144385872999997, 2023-07-02T14:07:15.613 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.25297039599999999, 2023-07-02T14:07:15.613 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:15.614 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:15.614 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:15.614 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:15.614 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:15.614 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20875 2023-07-02T14:07:15.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.615 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:15.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.615 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:15.615 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:07:15.615 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:15.615 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.615 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.616 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:15.616 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321480+0000", 2023-07-02T14:07:15.616 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:15.616 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.616 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.616 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:15.616 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321479+0000", 2023-07-02T14:07:15.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00015194399999999999 2023-07-02T14:07:15.617 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.617 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.617 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:15.617 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321631+0000", 2023-07-02T14:07:15.617 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4236999999999999e-05 2023-07-02T14:07:15.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.618 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:15.618 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321645+0000", 2023-07-02T14:07:15.618 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2160000000000003e-06 2023-07-02T14:07:15.618 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.618 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:15.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321651+0000", 2023-07-02T14:07:15.619 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3609999999999998e-05 2023-07-02T14:07:15.619 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.619 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.619 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.619 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321695+0000", 2023-07-02T14:07:15.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1989999999999995e-06 2023-07-02T14:07:15.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.620 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.620 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:15.620 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.321704+0000", 2023-07-02T14:07:15.620 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.23542881500000001 2023-07-02T14:07:15.620 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.621 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557133+0000", 2023-07-02T14:07:15.621 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00017359600000000001 2023-07-02T14:07:15.621 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.621 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.621 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:15.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557307+0000", 2023-07-02T14:07:15.622 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00023367599999999999 2023-07-02T14:07:15.622 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.622 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.622 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:15.622 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.557540+0000", 2023-07-02T14:07:15.623 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016744963000000002 2023-07-02T14:07:15.623 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.623 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.624 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:15.624 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574285+0000", 2023-07-02T14:07:15.624 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1330999999999999e-05 2023-07-02T14:07:15.624 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.624 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.625 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:15.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574346+0000", 2023-07-02T14:07:15.625 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9750000000000002e-05 2023-07-02T14:07:15.625 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.625 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.625 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:15.625 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574406+0000", 2023-07-02T14:07:15.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4647999999999997e-05 2023-07-02T14:07:15.626 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.626 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.626 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:15.626 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574451+0000", 2023-07-02T14:07:15.626 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.25297039599999999 2023-07-02T14:07:15.626 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.627 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:15.627 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.627 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.627 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.628 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:20915 2.7 2:e6e64067:::rbd_data.125de1449605.00000000000001a9:head [write 0~65536] snapc 39=[39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e189)", 2023-07-02T14:07:15.628 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:07:15.628 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 50.081132078000003, 2023-07-02T14:07:15.628 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.19966078500000001, 2023-07-02T14:07:15.628 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:15.628 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:15.629 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:15.629 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:15.629 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:15.629 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 20915 2023-07-02T14:07:15.629 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.629 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:15.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:15.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:07:15.630 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:15.630 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.630 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.630 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:15.630 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384734+0000", 2023-07-02T14:07:15.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999981 2023-07-02T14:07:15.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.631 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.631 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:15.631 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384732+0000", 2023-07-02T14:07:15.631 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9439999999999999e-05 2023-07-02T14:07:15.631 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.632 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:15.632 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384802+0000", 2023-07-02T14:07:15.632 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7572999999999999e-05 2023-07-02T14:07:15.632 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.632 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.633 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:15.633 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384819+0000", 2023-07-02T14:07:15.633 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1670000000000004e-06 2023-07-02T14:07:15.634 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.634 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.634 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:15.635 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384825+0000", 2023-07-02T14:07:15.635 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6471999999999998e-05 2023-07-02T14:07:15.635 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.635 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.636 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.636 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384872+0000", 2023-07-02T14:07:15.636 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0028e-05 2023-07-02T14:07:15.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.637 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.637 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for peered", 2023-07-02T14:07:15.637 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.384882+0000", 2023-07-02T14:07:15.637 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.17674214099999999 2023-07-02T14:07:15.637 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.638 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561624+0000", 2023-07-02T14:07:15.638 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000135809 2023-07-02T14:07:15.638 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.638 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.638 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:15.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561760+0000", 2023-07-02T14:07:15.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00020468 2023-07-02T14:07:15.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.639 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.639 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:15.639 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.561965+0000", 2023-07-02T14:07:15.639 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012498198 2023-07-02T14:07:15.639 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.640 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:15.640 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.574463+0000", 2023-07-02T14:07:15.640 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098197509999999998 2023-07-02T14:07:15.640 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.640 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:15.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584283+0000", 2023-07-02T14:07:15.641 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2056000000000004e-05 2023-07-02T14:07:15.641 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.641 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.641 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:15.641 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584345+0000", 2023-07-02T14:07:15.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0513000000000002e-05 2023-07-02T14:07:15.642 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.642 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.642 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:15.642 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:25.584395+0000", 2023-07-02T14:07:15.642 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.19966078500000001 2023-07-02T14:07:15.643 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.643 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:15.643 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.643 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.643 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.643 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.7359.0:48 2.7 2:f82cfcd4:::rbd_object_map.11fc3a5ad442:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e216)", 2023-07-02T14:07:15.643 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:07:11.913979+0000", 2023-07-02T14:07:15.643 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.5518877240000002, 2023-07-02T14:07:15.644 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.17117800899999999, 2023-07-02T14:07:15.644 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:15.644 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:15.644 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:15.644 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.7359", 2023-07-02T14:07:15.644 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1139503436", 2023-07-02T14:07:15.645 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 48 2023-07-02T14:07:15.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.645 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:15.645 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.645 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:15.645 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.913979+0000", 2023-07-02T14:07:15.645 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:15.645 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.646 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.646 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:15.646 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.913979+0000", 2023-07-02T14:07:15.646 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:15.647 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.647 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.647 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:15.647 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.913978+0000", 2023-07-02T14:07:15.648 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2519999999999996e-06 2023-07-02T14:07:15.648 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.648 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.648 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:15.649 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.913984+0000", 2023-07-02T14:07:15.649 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.9620000000000005e-06 2023-07-02T14:07:15.649 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.650 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.650 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:15.650 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.913994+0000", 2023-07-02T14:07:15.650 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5479999999999998e-06 2023-07-02T14:07:15.650 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:15.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914001+0000", 2023-07-02T14:07:15.651 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005245146 2023-07-02T14:07:15.651 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.651 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.651 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.651 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.919246+0000", 2023-07-02T14:07:15.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.523e-05 2023-07-02T14:07:15.652 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.652 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.652 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.652 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.919281+0000", 2023-07-02T14:07:15.652 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068795219999999999 2023-07-02T14:07:15.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.653 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.926160+0000", 2023-07-02T14:07:15.653 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5011000000000001e-05 2023-07-02T14:07:15.653 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.653 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.653 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.654 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.926195+0000", 2023-07-02T14:07:15.654 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010577570999999999 2023-07-02T14:07:15.654 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.654 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.655 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.655 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.936773+0000", 2023-07-02T14:07:15.655 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5687999999999999e-05 2023-07-02T14:07:15.655 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.656 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.656 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.656 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.936839+0000", 2023-07-02T14:07:15.656 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062738760000000003 2023-07-02T14:07:15.657 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.657 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.657 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.943113+0000", 2023-07-02T14:07:15.658 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9002999999999999e-05 2023-07-02T14:07:15.658 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.658 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.658 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.943152+0000", 2023-07-02T14:07:15.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052419110000000001 2023-07-02T14:07:15.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.659 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.659 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.659 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.948393+0000", 2023-07-02T14:07:15.659 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9255e-05 2023-07-02T14:07:15.659 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.660 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.948463+0000", 2023-07-02T14:07:15.660 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073665079999999999 2023-07-02T14:07:15.660 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.660 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.660 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.955829+0000", 2023-07-02T14:07:15.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4318000000000002e-05 2023-07-02T14:07:15.661 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.661 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.661 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.661 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.955904+0000", 2023-07-02T14:07:15.661 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066155659999999998 2023-07-02T14:07:15.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.662 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.662 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.962519+0000", 2023-07-02T14:07:15.662 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0018999999999994e-05 2023-07-02T14:07:15.662 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.662 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.962599+0000", 2023-07-02T14:07:15.663 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064935230000000002 2023-07-02T14:07:15.663 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.663 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.663 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.663 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.969093+0000", 2023-07-02T14:07:15.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0037999999999999e-05 2023-07-02T14:07:15.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.664 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.664 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.664 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.969133+0000", 2023-07-02T14:07:15.664 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067096320000000001 2023-07-02T14:07:15.664 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.975842+0000", 2023-07-02T14:07:15.665 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.9580999999999995e-05 2023-07-02T14:07:15.665 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.665 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.665 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.665 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.975932+0000", 2023-07-02T14:07:15.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057728399999999996 2023-07-02T14:07:15.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.666 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.666 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.666 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.981705+0000", 2023-07-02T14:07:15.666 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5228999999999998e-05 2023-07-02T14:07:15.666 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.667 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.667 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.667 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.981740+0000", 2023-07-02T14:07:15.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060052109999999999 2023-07-02T14:07:15.668 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.668 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.668 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.668 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.987745+0000", 2023-07-02T14:07:15.668 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8829e-05 2023-07-02T14:07:15.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.669 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.669 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.987824+0000", 2023-07-02T14:07:15.669 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042380559999999996 2023-07-02T14:07:15.669 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.669 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.992062+0000", 2023-07-02T14:07:15.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5153000000000004e-05 2023-07-02T14:07:15.670 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.670 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.670 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.670 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.992137+0000", 2023-07-02T14:07:15.670 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007856811 2023-07-02T14:07:15.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.671 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.671 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.999994+0000", 2023-07-02T14:07:15.671 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1412e-05 2023-07-02T14:07:15.671 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.671 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.000065+0000", 2023-07-02T14:07:15.672 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050351299999999996 2023-07-02T14:07:15.672 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.672 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.672 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.672 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.005101+0000", 2023-07-02T14:07:15.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0498000000000001e-05 2023-07-02T14:07:15.673 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.673 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.673 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.673 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.005181+0000", 2023-07-02T14:07:15.673 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00679118 2023-07-02T14:07:15.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.674 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.011972+0000", 2023-07-02T14:07:15.674 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3603000000000003e-05 2023-07-02T14:07:15.674 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.674 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.674 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.012006+0000", 2023-07-02T14:07:15.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031990019999999998 2023-07-02T14:07:15.675 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.675 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.675 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.675 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.015205+0000", 2023-07-02T14:07:15.675 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1495999999999998e-05 2023-07-02T14:07:15.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.676 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.676 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.015236+0000", 2023-07-02T14:07:15.676 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074766590000000001 2023-07-02T14:07:15.676 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.676 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.022713+0000", 2023-07-02T14:07:15.677 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.96e-05 2023-07-02T14:07:15.677 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.677 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.677 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.677 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.022753+0000", 2023-07-02T14:07:15.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061422200000000003 2023-07-02T14:07:15.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.678 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.678 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.678 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.028895+0000", 2023-07-02T14:07:15.678 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.092e-05 2023-07-02T14:07:15.678 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.679 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.028946+0000", 2023-07-02T14:07:15.679 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056345789999999998 2023-07-02T14:07:15.679 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.679 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.679 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.034580+0000", 2023-07-02T14:07:15.680 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5260999999999997e-05 2023-07-02T14:07:15.680 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.680 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.680 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.680 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.034656+0000", 2023-07-02T14:07:15.681 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005366127 2023-07-02T14:07:15.681 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.681 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.681 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.681 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.040022+0000", 2023-07-02T14:07:15.682 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2359e-05 2023-07-02T14:07:15.682 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.682 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.682 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.682 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.040054+0000", 2023-07-02T14:07:15.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054776820000000002 2023-07-02T14:07:15.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.683 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.683 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.683 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.045532+0000", 2023-07-02T14:07:15.683 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6161999999999994e-05 2023-07-02T14:07:15.683 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.684 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.684 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.045598+0000", 2023-07-02T14:07:15.684 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044752330000000003 2023-07-02T14:07:15.684 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.684 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.050073+0000", 2023-07-02T14:07:15.685 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.286e-05 2023-07-02T14:07:15.685 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.685 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.685 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.685 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.050106+0000", 2023-07-02T14:07:15.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069698379999999999 2023-07-02T14:07:15.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.686 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.686 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.686 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.057076+0000", 2023-07-02T14:07:15.686 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6508999999999994e-05 2023-07-02T14:07:15.686 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.057172+0000", 2023-07-02T14:07:15.687 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0063542579999999998 2023-07-02T14:07:15.687 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.687 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.687 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.687 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.063527+0000", 2023-07-02T14:07:15.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8861000000000003e-05 2023-07-02T14:07:15.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.688 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.688 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.688 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.063606+0000", 2023-07-02T14:07:15.688 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051551619999999996 2023-07-02T14:07:15.688 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.068761+0000", 2023-07-02T14:07:15.689 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1804999999999994e-05 2023-07-02T14:07:15.689 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.689 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.689 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.689 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.068832+0000", 2023-07-02T14:07:15.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054504729999999999 2023-07-02T14:07:15.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.690 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.690 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.074283+0000", 2023-07-02T14:07:15.690 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5518000000000003e-05 2023-07-02T14:07:15.690 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.690 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.074378+0000", 2023-07-02T14:07:15.691 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044083079999999997 2023-07-02T14:07:15.691 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.691 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.691 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.691 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.078787+0000", 2023-07-02T14:07:15.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3823e-05 2023-07-02T14:07:15.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.692 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:15.692 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.078871+0000", 2023-07-02T14:07:15.692 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00044416300000000002 2023-07-02T14:07:15.692 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.692 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:15.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.079315+0000", 2023-07-02T14:07:15.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036548890000000001 2023-07-02T14:07:15.693 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.693 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.693 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:15.693 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.082970+0000", 2023-07-02T14:07:15.693 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0019178660000000001 2023-07-02T14:07:15.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.694 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:15.694 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.084888+0000", 2023-07-02T14:07:15.694 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3066999999999999e-05 2023-07-02T14:07:15.694 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.694 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:15.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.084941+0000", 2023-07-02T14:07:15.695 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000216076 2023-07-02T14:07:15.695 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.695 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.695 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:15.695 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.085157+0000", 2023-07-02T14:07:15.696 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.17117800899999999 2023-07-02T14:07:15.696 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.696 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:15.696 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.696 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.696 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.697 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.7359.0:49 2.7 2:f82cfcd4:::rbd_object_map.11fc3a5ad442:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e216)", 2023-07-02T14:07:15.697 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:07:11.914010+0000", 2023-07-02T14:07:15.697 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.5518559619999999, 2023-07-02T14:07:15.697 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.17609659599999999, 2023-07-02T14:07:15.697 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:15.697 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:15.697 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:15.698 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.7359", 2023-07-02T14:07:15.698 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1139503436", 2023-07-02T14:07:15.698 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 49 2023-07-02T14:07:15.698 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.698 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:15.698 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.698 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:15.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914010+0000", 2023-07-02T14:07:15.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:15.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.699 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.699 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:15.699 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914010+0000", 2023-07-02T14:07:15.699 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:15.699 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:15.700 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914010+0000", 2023-07-02T14:07:15.700 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.459e-06 2023-07-02T14:07:15.700 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.700 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.700 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:15.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914014+0000", 2023-07-02T14:07:15.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0290000000000005e-06 2023-07-02T14:07:15.701 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.701 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.701 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:15.701 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914022+0000", 2023-07-02T14:07:15.701 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7110000000000003e-06 2023-07-02T14:07:15.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.702 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:15.702 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914027+0000", 2023-07-02T14:07:15.702 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052718069999999999 2023-07-02T14:07:15.702 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.702 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.919299+0000", 2023-07-02T14:07:15.703 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4303e-05 2023-07-02T14:07:15.703 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.703 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.703 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.703 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.919333+0000", 2023-07-02T14:07:15.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0069133909999999996 2023-07-02T14:07:15.704 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.704 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.704 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.704 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.926246+0000", 2023-07-02T14:07:15.704 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1404999999999997e-05 2023-07-02T14:07:15.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.705 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.705 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.926328+0000", 2023-07-02T14:07:15.705 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010546692999999999 2023-07-02T14:07:15.705 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.705 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.936875+0000", 2023-07-02T14:07:15.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4690999999999997e-05 2023-07-02T14:07:15.706 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.706 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.706 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.706 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.936909+0000", 2023-07-02T14:07:15.706 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062721649999999997 2023-07-02T14:07:15.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.707 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.707 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.943181+0000", 2023-07-02T14:07:15.707 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0457000000000002e-05 2023-07-02T14:07:15.707 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.707 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.708 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.943222+0000", 2023-07-02T14:07:15.708 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053028490000000001 2023-07-02T14:07:15.708 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.708 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.708 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.948525+0000", 2023-07-02T14:07:15.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.1451999999999994e-05 2023-07-02T14:07:15.709 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.709 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.709 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.709 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.948586+0000", 2023-07-02T14:07:15.709 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073670460000000004 2023-07-02T14:07:15.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.710 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.955953+0000", 2023-07-02T14:07:15.710 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0064999999999997e-05 2023-07-02T14:07:15.710 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.710 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.710 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.955993+0000", 2023-07-02T14:07:15.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066647420000000004 2023-07-02T14:07:15.711 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.711 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.711 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.711 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.962658+0000", 2023-07-02T14:07:15.711 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3088000000000001e-05 2023-07-02T14:07:15.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.712 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.712 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.962741+0000", 2023-07-02T14:07:15.712 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064260640000000004 2023-07-02T14:07:15.712 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.712 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.713 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.969167+0000", 2023-07-02T14:07:15.713 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9378999999999997e-05 2023-07-02T14:07:15.713 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.713 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.713 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.714 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.969207+0000", 2023-07-02T14:07:15.714 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067799640000000003 2023-07-02T14:07:15.714 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.714 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.714 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.975987+0000", 2023-07-02T14:07:15.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4986000000000004e-05 2023-07-02T14:07:15.715 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.715 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.715 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.715 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.976052+0000", 2023-07-02T14:07:15.715 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057279260000000004 2023-07-02T14:07:15.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.716 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.716 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.981779+0000", 2023-07-02T14:07:15.716 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6104000000000003e-05 2023-07-02T14:07:15.716 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.716 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.717 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.981846+0000", 2023-07-02T14:07:15.717 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060164629999999997 2023-07-02T14:07:15.717 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.717 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.717 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.987862+0000", 2023-07-02T14:07:15.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1872000000000002e-05 2023-07-02T14:07:15.718 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.718 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.718 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.718 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.987904+0000", 2023-07-02T14:07:15.718 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042721429999999999 2023-07-02T14:07:15.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.719 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.992176+0000", 2023-07-02T14:07:15.719 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0377000000000003e-05 2023-07-02T14:07:15.719 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.719 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.719 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.992216+0000", 2023-07-02T14:07:15.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079040989999999995 2023-07-02T14:07:15.720 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.720 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.720 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.720 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.000121+0000", 2023-07-02T14:07:15.720 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2204000000000003e-05 2023-07-02T14:07:15.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.721 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.721 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.000183+0000", 2023-07-02T14:07:15.721 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050360509999999997 2023-07-02T14:07:15.721 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.721 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.005219+0000", 2023-07-02T14:07:15.722 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7339000000000003e-05 2023-07-02T14:07:15.722 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.722 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.722 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.722 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.005256+0000", 2023-07-02T14:07:15.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067807880000000003 2023-07-02T14:07:15.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.723 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.723 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.723 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.012037+0000", 2023-07-02T14:07:15.723 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1863000000000003e-05 2023-07-02T14:07:15.723 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.724 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.012069+0000", 2023-07-02T14:07:15.724 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032102210000000001 2023-07-02T14:07:15.724 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.724 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.724 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.015279+0000", 2023-07-02T14:07:15.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2775999999999996e-05 2023-07-02T14:07:15.725 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.725 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.725 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.725 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.015352+0000", 2023-07-02T14:07:15.725 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074204040000000002 2023-07-02T14:07:15.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.726 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.726 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.022772+0000", 2023-07-02T14:07:15.726 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9047000000000002e-05 2023-07-02T14:07:15.726 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.726 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.022801+0000", 2023-07-02T14:07:15.727 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061832800000000002 2023-07-02T14:07:15.727 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.727 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.727 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.727 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.028984+0000", 2023-07-02T14:07:15.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5652999999999999e-05 2023-07-02T14:07:15.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.728 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.728 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.728 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.029020+0000", 2023-07-02T14:07:15.728 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056827459999999998 2023-07-02T14:07:15.728 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.034703+0000", 2023-07-02T14:07:15.729 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4592999999999997e-05 2023-07-02T14:07:15.729 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.729 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.729 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.729 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.034757+0000", 2023-07-02T14:07:15.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053366389999999998 2023-07-02T14:07:15.730 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.730 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.730 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.730 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.040094+0000", 2023-07-02T14:07:15.730 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8977000000000003e-05 2023-07-02T14:07:15.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.731 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.040133+0000", 2023-07-02T14:07:15.731 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055145960000000001 2023-07-02T14:07:15.731 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.731 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.731 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.045648+0000", 2023-07-02T14:07:15.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2254000000000003e-05 2023-07-02T14:07:15.732 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.732 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.732 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.732 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.045720+0000", 2023-07-02T14:07:15.732 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044189279999999999 2023-07-02T14:07:15.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.733 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.733 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.050139+0000", 2023-07-02T14:07:15.733 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1952000000000002e-05 2023-07-02T14:07:15.733 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.733 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.050171+0000", 2023-07-02T14:07:15.734 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070571770000000004 2023-07-02T14:07:15.734 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.734 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.734 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.734 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.057228+0000", 2023-07-02T14:07:15.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1320000000000001e-05 2023-07-02T14:07:15.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.735 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.735 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.057309+0000", 2023-07-02T14:07:15.735 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0063538350000000004 2023-07-02T14:07:15.735 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.735 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.063663+0000", 2023-07-02T14:07:15.736 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2733999999999999e-05 2023-07-02T14:07:15.736 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.736 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.736 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.736 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.063736+0000", 2023-07-02T14:07:15.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051447619999999998 2023-07-02T14:07:15.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.737 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.737 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.737 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.068881+0000", 2023-07-02T14:07:15.737 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4487999999999994e-05 2023-07-02T14:07:15.737 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.738 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.068975+0000", 2023-07-02T14:07:15.738 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054599760000000001 2023-07-02T14:07:15.738 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.738 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.738 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.074435+0000", 2023-07-02T14:07:15.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5309999999999996e-05 2023-07-02T14:07:15.739 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.739 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.739 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.739 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.074510+0000", 2023-07-02T14:07:15.739 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066353530000000001 2023-07-02T14:07:15.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.740 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.081146+0000", 2023-07-02T14:07:15.740 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2584999999999997e-05 2023-07-02T14:07:15.740 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.740 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.740 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.081218+0000", 2023-07-02T14:07:15.741 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0039653620000000001 2023-07-02T14:07:15.741 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.741 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.741 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.741 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.085184+0000", 2023-07-02T14:07:15.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0596000000000001e-05 2023-07-02T14:07:15.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.742 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:15.742 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.085234+0000", 2023-07-02T14:07:15.742 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000285042 2023-07-02T14:07:15.742 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.742 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:15.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.085519+0000", 2023-07-02T14:07:15.743 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043416920000000003 2023-07-02T14:07:15.743 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.743 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.743 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:15.743 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.089861+0000", 2023-07-02T14:07:15.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010830700000000001 2023-07-02T14:07:15.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.744 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.744 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:15.744 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.089969+0000", 2023-07-02T14:07:15.744 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1618e-05 2023-07-02T14:07:15.744 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:15.745 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.090011+0000", 2023-07-02T14:07:15.745 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.6015999999999997e-05 2023-07-02T14:07:15.745 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.745 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.745 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:15.746 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.090107+0000", 2023-07-02T14:07:15.746 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.17609659599999999 2023-07-02T14:07:15.746 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.746 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:15.746 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.746 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.746 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.747 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.7359.0:50 2.7 2:f82cfcd4:::rbd_object_map.11fc3a5ad442:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e216)", 2023-07-02T14:07:15.747 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:07:11.914032+0000", 2023-07-02T14:07:15.747 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.5518345280000001, 2023-07-02T14:07:15.747 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.181539863, 2023-07-02T14:07:15.747 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:15.747 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:15.747 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:15.747 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.7359", 2023-07-02T14:07:15.748 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1139503436", 2023-07-02T14:07:15.748 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 50 2023-07-02T14:07:15.748 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.748 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:15.748 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.748 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:15.748 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914032+0000", 2023-07-02T14:07:15.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:15.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.749 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.749 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:15.749 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914032+0000", 2023-07-02T14:07:15.749 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:15.749 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:15.750 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914031+0000", 2023-07-02T14:07:15.750 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4590000000000003e-06 2023-07-02T14:07:15.750 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.750 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.750 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:15.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914038+0000", 2023-07-02T14:07:15.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7159999999999997e-06 2023-07-02T14:07:15.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.751 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.751 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:15.751 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914045+0000", 2023-07-02T14:07:15.751 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.408e-06 2023-07-02T14:07:15.751 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:15.752 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914049+0000", 2023-07-02T14:07:15.752 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053065129999999997 2023-07-02T14:07:15.752 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.752 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.752 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.919355+0000", 2023-07-02T14:07:15.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6915999999999997e-05 2023-07-02T14:07:15.753 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.753 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.753 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.753 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.919392+0000", 2023-07-02T14:07:15.753 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070034859999999997 2023-07-02T14:07:15.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.754 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.926396+0000", 2023-07-02T14:07:15.754 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8988999999999999e-05 2023-07-02T14:07:15.754 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.754 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.754 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.926475+0000", 2023-07-02T14:07:15.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010474221000000001 2023-07-02T14:07:15.755 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.755 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.755 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.755 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.936949+0000", 2023-07-02T14:07:15.755 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2463000000000001e-05 2023-07-02T14:07:15.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.756 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.756 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.937012+0000", 2023-07-02T14:07:15.756 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062378570000000003 2023-07-02T14:07:15.756 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.756 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.943249+0000", 2023-07-02T14:07:15.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3742000000000003e-05 2023-07-02T14:07:15.757 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.757 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.757 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.757 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.943283+0000", 2023-07-02T14:07:15.757 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053598029999999998 2023-07-02T14:07:15.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.758 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.948643+0000", 2023-07-02T14:07:15.758 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6029e-05 2023-07-02T14:07:15.758 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.758 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.758 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.948719+0000", 2023-07-02T14:07:15.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073148249999999996 2023-07-02T14:07:15.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.759 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.759 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.759 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.956034+0000", 2023-07-02T14:07:15.759 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5241999999999998e-05 2023-07-02T14:07:15.759 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.760 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.956079+0000", 2023-07-02T14:07:15.760 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067187310000000004 2023-07-02T14:07:15.760 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.760 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.760 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.962798+0000", 2023-07-02T14:07:15.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.5866999999999995e-05 2023-07-02T14:07:15.761 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.761 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.761 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.761 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.962894+0000", 2023-07-02T14:07:15.761 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0063470840000000002 2023-07-02T14:07:15.763 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.763 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.763 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.969241+0000", 2023-07-02T14:07:15.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5182999999999998e-05 2023-07-02T14:07:15.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.764 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.764 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.764 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.969276+0000", 2023-07-02T14:07:15.764 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006841059 2023-07-02T14:07:15.764 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.765 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.765 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.976117+0000", 2023-07-02T14:07:15.765 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1842999999999996e-05 2023-07-02T14:07:15.765 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.765 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.976189+0000", 2023-07-02T14:07:15.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056863629999999998 2023-07-02T14:07:15.766 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.766 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.766 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.766 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.981875+0000", 2023-07-02T14:07:15.766 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0577999999999998e-05 2023-07-02T14:07:15.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.767 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.981906+0000", 2023-07-02T14:07:15.767 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060401910000000003 2023-07-02T14:07:15.767 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.767 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.767 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.987946+0000", 2023-07-02T14:07:15.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3609000000000006e-05 2023-07-02T14:07:15.768 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.768 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.768 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.768 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.988020+0000", 2023-07-02T14:07:15.768 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042379039999999998 2023-07-02T14:07:15.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.769 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.992257+0000", 2023-07-02T14:07:15.769 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8497000000000003e-05 2023-07-02T14:07:15.769 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.769 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.769 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.992346+0000", 2023-07-02T14:07:15.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078834179999999997 2023-07-02T14:07:15.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.770 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.770 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.770 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.000229+0000", 2023-07-02T14:07:15.770 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4128999999999994e-05 2023-07-02T14:07:15.770 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.000303+0000", 2023-07-02T14:07:15.771 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049806199999999998 2023-07-02T14:07:15.771 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.771 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.771 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.771 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.005284+0000", 2023-07-02T14:07:15.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3077000000000002e-05 2023-07-02T14:07:15.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.772 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.772 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.005317+0000", 2023-07-02T14:07:15.772 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067827319999999997 2023-07-02T14:07:15.772 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.772 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.012100+0000", 2023-07-02T14:07:15.773 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0080000000000003e-05 2023-07-02T14:07:15.773 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.773 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.773 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.773 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.012140+0000", 2023-07-02T14:07:15.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032486920000000001 2023-07-02T14:07:15.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.774 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.774 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.015389+0000", 2023-07-02T14:07:15.774 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7679000000000002e-05 2023-07-02T14:07:15.774 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.774 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.015436+0000", 2023-07-02T14:07:15.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074112659999999997 2023-07-02T14:07:15.775 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.775 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.775 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.775 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.022848+0000", 2023-07-02T14:07:15.775 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9221000000000006e-05 2023-07-02T14:07:15.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.776 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.776 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.022927+0000", 2023-07-02T14:07:15.776 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061277969999999999 2023-07-02T14:07:15.776 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.776 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.029055+0000", 2023-07-02T14:07:15.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000112797 2023-07-02T14:07:15.777 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.777 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.777 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.777 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.029167+0000", 2023-07-02T14:07:15.777 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056416569999999996 2023-07-02T14:07:15.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.778 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.034809+0000", 2023-07-02T14:07:15.778 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3275999999999993e-05 2023-07-02T14:07:15.778 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.778 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.778 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.034892+0000", 2023-07-02T14:07:15.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052726500000000003 2023-07-02T14:07:15.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.779 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.779 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.779 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.040165+0000", 2023-07-02T14:07:15.779 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2160999999999999e-05 2023-07-02T14:07:15.779 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.780 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.040197+0000", 2023-07-02T14:07:15.780 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055705930000000004 2023-07-02T14:07:15.780 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.780 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.780 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.045768+0000", 2023-07-02T14:07:15.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7665999999999998e-05 2023-07-02T14:07:15.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.781 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.781 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.781 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.045845+0000", 2023-07-02T14:07:15.781 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043533319999999997 2023-07-02T14:07:15.781 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.050199+0000", 2023-07-02T14:07:15.782 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2767e-05 2023-07-02T14:07:15.782 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.782 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.782 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.782 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.050232+0000", 2023-07-02T14:07:15.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071142380000000002 2023-07-02T14:07:15.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.783 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.783 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.783 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.057346+0000", 2023-07-02T14:07:15.783 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2778000000000003e-05 2023-07-02T14:07:15.783 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.057389+0000", 2023-07-02T14:07:15.784 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064065609999999999 2023-07-02T14:07:15.784 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.784 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.784 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.784 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.063795+0000", 2023-07-02T14:07:15.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2032000000000004e-05 2023-07-02T14:07:15.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.785 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.785 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.063867+0000", 2023-07-02T14:07:15.785 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051688569999999998 2023-07-02T14:07:15.785 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.785 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.069036+0000", 2023-07-02T14:07:15.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4841999999999998e-05 2023-07-02T14:07:15.786 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.786 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.786 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.786 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.069121+0000", 2023-07-02T14:07:15.786 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054419610000000004 2023-07-02T14:07:15.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.787 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.787 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.074563+0000", 2023-07-02T14:07:15.787 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9251e-05 2023-07-02T14:07:15.787 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.787 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.074612+0000", 2023-07-02T14:07:15.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066624700000000002 2023-07-02T14:07:15.788 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.788 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.788 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.788 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.081275+0000", 2023-07-02T14:07:15.788 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0149999999999997e-05 2023-07-02T14:07:15.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.789 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.081355+0000", 2023-07-02T14:07:15.789 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059046890000000003 2023-07-02T14:07:15.789 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.789 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.789 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.087259+0000", 2023-07-02T14:07:15.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3191999999999999e-05 2023-07-02T14:07:15.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.790 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.790 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.790 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.087343+0000", 2023-07-02T14:07:15.790 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0028279960000000002 2023-07-02T14:07:15.790 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.090171+0000", 2023-07-02T14:07:15.791 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000105679 2023-07-02T14:07:15.791 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.791 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.791 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:15.791 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.090276+0000", 2023-07-02T14:07:15.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00048451400000000002 2023-07-02T14:07:15.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.792 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.792 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:15.792 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.090761+0000", 2023-07-02T14:07:15.792 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029060520000000001 2023-07-02T14:07:15.792 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:15.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.093667+0000", 2023-07-02T14:07:15.793 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0017563259999999999 2023-07-02T14:07:15.793 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.793 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.793 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:15.793 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.095423+0000", 2023-07-02T14:07:15.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4518000000000003e-05 2023-07-02T14:07:15.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.794 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:15.794 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.095468+0000", 2023-07-02T14:07:15.794 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00010404900000000001 2023-07-02T14:07:15.794 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.794 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.795 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:15.795 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.095572+0000", 2023-07-02T14:07:15.795 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.181539863 2023-07-02T14:07:15.795 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.795 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:15.795 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.795 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.795 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.796 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.7359.0:51 2.7 2:f82cfcd4:::rbd_object_map.11fc3a5ad442:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e216)", 2023-07-02T14:07:15.796 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:07:11.914068+0000", 2023-07-02T14:07:15.796 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.5517986800000001, 2023-07-02T14:07:15.796 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.18689261500000001, 2023-07-02T14:07:15.796 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:15.796 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:15.796 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:15.796 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.7359", 2023-07-02T14:07:15.797 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1139503436", 2023-07-02T14:07:15.797 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 51 2023-07-02T14:07:15.797 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.797 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:15.797 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.797 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:15.797 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914068+0000", 2023-07-02T14:07:15.797 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:15.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.798 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:15.798 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914068+0000", 2023-07-02T14:07:15.798 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:15.798 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.798 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:15.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914067+0000", 2023-07-02T14:07:15.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6650000000000002e-06 2023-07-02T14:07:15.799 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.799 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.799 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:15.799 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914072+0000", 2023-07-02T14:07:15.799 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3379999999999999e-06 2023-07-02T14:07:15.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:15.800 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914079+0000", 2023-07-02T14:07:15.800 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7269999999999998e-06 2023-07-02T14:07:15.800 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.800 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.800 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:15.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914084+0000", 2023-07-02T14:07:15.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053268500000000002 2023-07-02T14:07:15.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.801 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.801 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.801 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.919411+0000", 2023-07-02T14:07:15.801 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5485999999999999e-05 2023-07-02T14:07:15.801 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.919446+0000", 2023-07-02T14:07:15.802 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070717109999999996 2023-07-02T14:07:15.802 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.802 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.802 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.802 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.926518+0000", 2023-07-02T14:07:15.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3957999999999999e-05 2023-07-02T14:07:15.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.803 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.803 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.803 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.926592+0000", 2023-07-02T14:07:15.803 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010455356000000001 2023-07-02T14:07:15.803 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.937047+0000", 2023-07-02T14:07:15.804 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8084999999999997e-05 2023-07-02T14:07:15.804 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.804 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.804 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.804 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.937105+0000", 2023-07-02T14:07:15.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062038750000000002 2023-07-02T14:07:15.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.805 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.805 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.943309+0000", 2023-07-02T14:07:15.805 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1528e-05 2023-07-02T14:07:15.805 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.805 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.943341+0000", 2023-07-02T14:07:15.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054124430000000003 2023-07-02T14:07:15.806 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.806 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.806 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.806 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.948753+0000", 2023-07-02T14:07:15.806 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2969999999999998e-05 2023-07-02T14:07:15.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.807 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.807 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.948786+0000", 2023-07-02T14:07:15.807 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073220619999999998 2023-07-02T14:07:15.807 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.807 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.956108+0000", 2023-07-02T14:07:15.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3821e-05 2023-07-02T14:07:15.808 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.808 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.808 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.808 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.956142+0000", 2023-07-02T14:07:15.808 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068237439999999996 2023-07-02T14:07:15.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.809 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.809 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.962966+0000", 2023-07-02T14:07:15.809 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2082999999999995e-05 2023-07-02T14:07:15.809 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.809 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.963028+0000", 2023-07-02T14:07:15.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062808669999999999 2023-07-02T14:07:15.810 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.810 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.810 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.810 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.969309+0000", 2023-07-02T14:07:15.810 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6241999999999994e-05 2023-07-02T14:07:15.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.811 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.969375+0000", 2023-07-02T14:07:15.811 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068752939999999997 2023-07-02T14:07:15.811 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.811 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.811 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.976250+0000", 2023-07-02T14:07:15.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6419e-05 2023-07-02T14:07:15.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.812 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.812 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.812 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.976337+0000", 2023-07-02T14:07:15.812 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056071630000000001 2023-07-02T14:07:15.812 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.981944+0000", 2023-07-02T14:07:15.813 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9063e-05 2023-07-02T14:07:15.813 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.813 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.813 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.813 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.982003+0000", 2023-07-02T14:07:15.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060721259999999997 2023-07-02T14:07:15.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.814 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.814 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.988075+0000", 2023-07-02T14:07:15.814 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0599000000000005e-05 2023-07-02T14:07:15.814 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.814 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.988166+0000", 2023-07-02T14:07:15.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042290349999999999 2023-07-02T14:07:15.815 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.815 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.815 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.815 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.992395+0000", 2023-07-02T14:07:15.815 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2336000000000003e-05 2023-07-02T14:07:15.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.816 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.816 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.992437+0000", 2023-07-02T14:07:15.816 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0079028689999999999 2023-07-02T14:07:15.816 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.816 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.000340+0000", 2023-07-02T14:07:15.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5964999999999999e-05 2023-07-02T14:07:15.817 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.817 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.817 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.817 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.000376+0000", 2023-07-02T14:07:15.817 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049664929999999998 2023-07-02T14:07:15.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.818 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.005342+0000", 2023-07-02T14:07:15.818 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1257000000000002e-05 2023-07-02T14:07:15.818 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.818 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.818 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.005374+0000", 2023-07-02T14:07:15.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068061930000000003 2023-07-02T14:07:15.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.819 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.819 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.819 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.012180+0000", 2023-07-02T14:07:15.819 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0606000000000002e-05 2023-07-02T14:07:15.819 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.012240+0000", 2023-07-02T14:07:15.820 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032280999999999998 2023-07-02T14:07:15.820 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.820 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.820 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.820 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.015468+0000", 2023-07-02T14:07:15.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5029e-05 2023-07-02T14:07:15.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.821 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.821 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.015503+0000", 2023-07-02T14:07:15.821 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074611570000000004 2023-07-02T14:07:15.821 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.821 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.022965+0000", 2023-07-02T14:07:15.822 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3414000000000002e-05 2023-07-02T14:07:15.822 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.822 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.822 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.822 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.022998+0000", 2023-07-02T14:07:15.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062327700000000003 2023-07-02T14:07:15.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.823 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.823 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.029231+0000", 2023-07-02T14:07:15.823 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7526000000000003e-05 2023-07-02T14:07:15.823 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.823 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.029308+0000", 2023-07-02T14:07:15.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056632219999999999 2023-07-02T14:07:15.824 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.824 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.824 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.824 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.034972+0000", 2023-07-02T14:07:15.824 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5676000000000001e-05 2023-07-02T14:07:15.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.825 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.035017+0000", 2023-07-02T14:07:15.825 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052101029999999998 2023-07-02T14:07:15.825 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.825 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.825 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.040227+0000", 2023-07-02T14:07:15.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2283000000000002e-05 2023-07-02T14:07:15.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.826 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.826 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.826 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.040260+0000", 2023-07-02T14:07:15.826 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056310939999999997 2023-07-02T14:07:15.826 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.045891+0000", 2023-07-02T14:07:15.827 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0764000000000001e-05 2023-07-02T14:07:15.827 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.827 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.827 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.827 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.045971+0000", 2023-07-02T14:07:15.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042858490000000004 2023-07-02T14:07:15.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.828 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.828 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.828 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.050257+0000", 2023-07-02T14:07:15.828 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4209999999999999e-05 2023-07-02T14:07:15.828 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.050302+0000", 2023-07-02T14:07:15.829 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071211089999999996 2023-07-02T14:07:15.829 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.829 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.829 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.829 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.057423+0000", 2023-07-02T14:07:15.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3025999999999999e-05 2023-07-02T14:07:15.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.830 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.830 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.057456+0000", 2023-07-02T14:07:15.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006461474 2023-07-02T14:07:15.830 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.063917+0000", 2023-07-02T14:07:15.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1563999999999994e-05 2023-07-02T14:07:15.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.063999+0000", 2023-07-02T14:07:15.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051867980000000003 2023-07-02T14:07:15.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.069185+0000", 2023-07-02T14:07:15.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7226999999999997e-05 2023-07-02T14:07:15.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.069273+0000", 2023-07-02T14:07:15.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054037249999999998 2023-07-02T14:07:15.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.074676+0000", 2023-07-02T14:07:15.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6379999999999996e-05 2023-07-02T14:07:15.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.074763+0000", 2023-07-02T14:07:15.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066442020000000001 2023-07-02T14:07:15.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.081407+0000", 2023-07-02T14:07:15.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3225000000000005e-05 2023-07-02T14:07:15.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.081470+0000", 2023-07-02T14:07:15.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005925441 2023-07-02T14:07:15.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.087396+0000", 2023-07-02T14:07:15.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8586999999999998e-05 2023-07-02T14:07:15.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.087474+0000", 2023-07-02T14:07:15.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004673239 2023-07-02T14:07:15.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.092148+0000", 2023-07-02T14:07:15.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8097999999999999e-05 2023-07-02T14:07:15.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.092196+0000", 2023-07-02T14:07:15.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0034108010000000002 2023-07-02T14:07:15.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.838 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.838 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.095606+0000", 2023-07-02T14:07:15.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5056999999999999e-05 2023-07-02T14:07:15.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:15.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.095651+0000", 2023-07-02T14:07:15.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025461800000000002 2023-07-02T14:07:15.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:15.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.095906+0000", 2023-07-02T14:07:15.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0037319940000000002 2023-07-02T14:07:15.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:15.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.099638+0000", 2023-07-02T14:07:15.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001144838 2023-07-02T14:07:15.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:15.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.100783+0000", 2023-07-02T14:07:15.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0985999999999998e-05 2023-07-02T14:07:15.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:15.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.100814+0000", 2023-07-02T14:07:15.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000146408 2023-07-02T14:07:15.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:15.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.100960+0000", 2023-07-02T14:07:15.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.18689261500000001 2023-07-02T14:07:15.843 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.843 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:15.843 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.843 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.7359.0:52 2.7 2:f82cfcd4:::rbd_object_map.11fc3a5ad442:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e216)", 2023-07-02T14:07:15.843 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:07:11.914091+0000", 2023-07-02T14:07:15.843 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.5517750719999999, 2023-07-02T14:07:15.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.19175679000000001, 2023-07-02T14:07:15.844 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:15.844 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:15.844 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:15.844 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.7359", 2023-07-02T14:07:15.844 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1139503436", 2023-07-02T14:07:15.844 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 52 2023-07-02T14:07:15.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.845 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:15.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:15.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914091+0000", 2023-07-02T14:07:15.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:15.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:15.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914091+0000", 2023-07-02T14:07:15.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:15.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.846 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:15.846 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914091+0000", 2023-07-02T14:07:15.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.262e-06 2023-07-02T14:07:15.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:15.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914095+0000", 2023-07-02T14:07:15.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.2569999999999996e-06 2023-07-02T14:07:15.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:15.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914104+0000", 2023-07-02T14:07:15.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.4490000000000001e-06 2023-07-02T14:07:15.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:15.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914109+0000", 2023-07-02T14:07:15.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053566509999999996 2023-07-02T14:07:15.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.919465+0000", 2023-07-02T14:07:15.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3865999999999997e-05 2023-07-02T14:07:15.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.919499+0000", 2023-07-02T14:07:15.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071541850000000004 2023-07-02T14:07:15.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.926653+0000", 2023-07-02T14:07:15.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9091000000000006e-05 2023-07-02T14:07:15.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.926732+0000", 2023-07-02T14:07:15.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010415295999999999 2023-07-02T14:07:15.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.937148+0000", 2023-07-02T14:07:15.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3711000000000003e-05 2023-07-02T14:07:15.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.937181+0000", 2023-07-02T14:07:15.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061845679999999997 2023-07-02T14:07:15.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.943366+0000", 2023-07-02T14:07:15.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2375000000000001e-05 2023-07-02T14:07:15.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.943398+0000", 2023-07-02T14:07:15.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054277140000000001 2023-07-02T14:07:15.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.854 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.854 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.948826+0000", 2023-07-02T14:07:15.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6822000000000005e-05 2023-07-02T14:07:15.854 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.854 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.948893+0000", 2023-07-02T14:07:15.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072777930000000003 2023-07-02T14:07:15.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.956171+0000", 2023-07-02T14:07:15.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8863000000000002e-05 2023-07-02T14:07:15.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.956230+0000", 2023-07-02T14:07:15.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068469589999999997 2023-07-02T14:07:15.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.963077+0000", 2023-07-02T14:07:15.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8334000000000003e-05 2023-07-02T14:07:15.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.963115+0000", 2023-07-02T14:07:15.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0063163489999999997 2023-07-02T14:07:15.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.969431+0000", 2023-07-02T14:07:15.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8176000000000005e-05 2023-07-02T14:07:15.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.969509+0000", 2023-07-02T14:07:15.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068610839999999999 2023-07-02T14:07:15.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.976370+0000", 2023-07-02T14:07:15.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3124000000000003e-05 2023-07-02T14:07:15.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.976404+0000", 2023-07-02T14:07:15.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056302130000000002 2023-07-02T14:07:15.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.982034+0000", 2023-07-02T14:07:15.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3654e-05 2023-07-02T14:07:15.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.982067+0000", 2023-07-02T14:07:15.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061538809999999999 2023-07-02T14:07:15.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.988221+0000", 2023-07-02T14:07:15.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1270999999999998e-05 2023-07-02T14:07:15.862 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.862 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.862 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.988273+0000", 2023-07-02T14:07:15.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042049729999999999 2023-07-02T14:07:15.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.992478+0000", 2023-07-02T14:07:15.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9771000000000007e-05 2023-07-02T14:07:15.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.992547+0000", 2023-07-02T14:07:15.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0078699359999999993 2023-07-02T14:07:15.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.000417+0000", 2023-07-02T14:07:15.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0698000000000006e-05 2023-07-02T14:07:15.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.000488+0000", 2023-07-02T14:07:15.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049087000000000002 2023-07-02T14:07:15.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.005397+0000", 2023-07-02T14:07:15.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4388999999999999e-05 2023-07-02T14:07:15.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.005431+0000", 2023-07-02T14:07:15.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068464099999999998 2023-07-02T14:07:15.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.012278+0000", 2023-07-02T14:07:15.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4619000000000002e-05 2023-07-02T14:07:15.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.012312+0000", 2023-07-02T14:07:15.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032166640000000002 2023-07-02T14:07:15.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.015529+0000", 2023-07-02T14:07:15.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6857999999999998e-05 2023-07-02T14:07:15.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.015566+0000", 2023-07-02T14:07:15.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007473108 2023-07-02T14:07:15.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.023039+0000", 2023-07-02T14:07:15.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4019999999999999e-05 2023-07-02T14:07:15.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.870 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.870 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.023113+0000", 2023-07-02T14:07:15.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062513999999999998 2023-07-02T14:07:15.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.029364+0000", 2023-07-02T14:07:15.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9107000000000002e-05 2023-07-02T14:07:15.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.029423+0000", 2023-07-02T14:07:15.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056454729999999998 2023-07-02T14:07:15.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.035069+0000", 2023-07-02T14:07:15.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6847000000000005e-05 2023-07-02T14:07:15.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.035146+0000", 2023-07-02T14:07:15.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051606380000000004 2023-07-02T14:07:15.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.040306+0000", 2023-07-02T14:07:15.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2215000000000002e-05 2023-07-02T14:07:15.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.040338+0000", 2023-07-02T14:07:15.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056807009999999998 2023-07-02T14:07:15.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.046019+0000", 2023-07-02T14:07:15.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1458999999999995e-05 2023-07-02T14:07:15.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.046091+0000", 2023-07-02T14:07:15.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042370699999999999 2023-07-02T14:07:15.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.050328+0000", 2023-07-02T14:07:15.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0397999999999999e-05 2023-07-02T14:07:15.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.050358+0000", 2023-07-02T14:07:15.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071291449999999999 2023-07-02T14:07:15.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.057487+0000", 2023-07-02T14:07:15.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2833999999999998e-05 2023-07-02T14:07:15.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.057520+0000", 2023-07-02T14:07:15.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065307459999999996 2023-07-02T14:07:15.878 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.878 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.878 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.064051+0000", 2023-07-02T14:07:15.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5399000000000002e-05 2023-07-02T14:07:15.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.064116+0000", 2023-07-02T14:07:15.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052051880000000003 2023-07-02T14:07:15.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.069321+0000", 2023-07-02T14:07:15.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4442000000000003e-05 2023-07-02T14:07:15.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.069396+0000", 2023-07-02T14:07:15.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054185789999999998 2023-07-02T14:07:15.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.074814+0000", 2023-07-02T14:07:15.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4079000000000002e-05 2023-07-02T14:07:15.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.074868+0000", 2023-07-02T14:07:15.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066351149999999996 2023-07-02T14:07:15.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.081504+0000", 2023-07-02T14:07:15.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6733000000000002e-05 2023-07-02T14:07:15.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.081540+0000", 2023-07-02T14:07:15.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059915990000000002 2023-07-02T14:07:15.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.087532+0000", 2023-07-02T14:07:15.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3135999999999999e-05 2023-07-02T14:07:15.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.087605+0000", 2023-07-02T14:07:15.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046271810000000002 2023-07-02T14:07:15.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.092232+0000", 2023-07-02T14:07:15.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7982999999999999e-05 2023-07-02T14:07:15.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.092270+0000", 2023-07-02T14:07:15.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055357660000000001 2023-07-02T14:07:15.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.886 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.886 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.097806+0000", 2023-07-02T14:07:15.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.477e-05 2023-07-02T14:07:15.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.097851+0000", 2023-07-02T14:07:15.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031490250000000002 2023-07-02T14:07:15.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.101000+0000", 2023-07-02T14:07:15.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7858000000000002e-05 2023-07-02T14:07:15.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:15.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.101048+0000", 2023-07-02T14:07:15.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00029407899999999999 2023-07-02T14:07:15.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:15.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.101342+0000", 2023-07-02T14:07:15.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030713049999999999 2023-07-02T14:07:15.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:15.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.104413+0000", 2023-07-02T14:07:15.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0012429439999999999 2023-07-02T14:07:15.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:15.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.105656+0000", 2023-07-02T14:07:15.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8126999999999993e-05 2023-07-02T14:07:15.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:15.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.105724+0000", 2023-07-02T14:07:15.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012396600000000001 2023-07-02T14:07:15.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:15.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.105848+0000", 2023-07-02T14:07:15.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.19175679000000001 2023-07-02T14:07:15.891 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.892 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:15.892 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.892 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.7359.0:53 2.7 2:f82cfcd4:::rbd_object_map.11fc3a5ad442:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e216)", 2023-07-02T14:07:15.892 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:07:11.914230+0000", 2023-07-02T14:07:15.892 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.551636792, 2023-07-02T14:07:15.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.19699383500000001, 2023-07-02T14:07:15.893 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:15.893 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:15.893 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:15.893 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.7359", 2023-07-02T14:07:15.893 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1139503436", 2023-07-02T14:07:15.893 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 53 2023-07-02T14:07:15.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.893 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:15.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:15.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914230+0000", 2023-07-02T14:07:15.894 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:15.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.894 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:15.894 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914230+0000", 2023-07-02T14:07:15.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:15.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.895 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:15.895 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914229+0000", 2023-07-02T14:07:15.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8239999999999999e-06 2023-07-02T14:07:15.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.895 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:15.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914234+0000", 2023-07-02T14:07:15.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5229999999999995e-06 2023-07-02T14:07:15.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:15.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914242+0000", 2023-07-02T14:07:15.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1660000000000002e-06 2023-07-02T14:07:15.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:15.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914247+0000", 2023-07-02T14:07:15.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00527035 2023-07-02T14:07:15.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.919518+0000", 2023-07-02T14:07:15.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3513999999999998e-05 2023-07-02T14:07:15.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.919551+0000", 2023-07-02T14:07:15.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072376849999999998 2023-07-02T14:07:15.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.926789+0000", 2023-07-02T14:07:15.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9328999999999999e-05 2023-07-02T14:07:15.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.926868+0000", 2023-07-02T14:07:15.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010352102 2023-07-02T14:07:15.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.937220+0000", 2023-07-02T14:07:15.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8924999999999995e-05 2023-07-02T14:07:15.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.937289+0000", 2023-07-02T14:07:15.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006148199 2023-07-02T14:07:15.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.943437+0000", 2023-07-02T14:07:15.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6278e-05 2023-07-02T14:07:15.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.943484+0000", 2023-07-02T14:07:15.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005455784 2023-07-02T14:07:15.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.948939+0000", 2023-07-02T14:07:15.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7399999999999998e-05 2023-07-02T14:07:15.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.903 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.903 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.949007+0000", 2023-07-02T14:07:15.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072619119999999997 2023-07-02T14:07:15.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.904 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.904 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.956269+0000", 2023-07-02T14:07:15.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4319999999999997e-05 2023-07-02T14:07:15.904 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.904 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.956303+0000", 2023-07-02T14:07:15.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068593739999999997 2023-07-02T14:07:15.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.963162+0000", 2023-07-02T14:07:15.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4866000000000001e-05 2023-07-02T14:07:15.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.963197+0000", 2023-07-02T14:07:15.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006358397 2023-07-02T14:07:15.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.969556+0000", 2023-07-02T14:07:15.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9285e-05 2023-07-02T14:07:15.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.969605+0000", 2023-07-02T14:07:15.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068364300000000001 2023-07-02T14:07:15.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.976441+0000", 2023-07-02T14:07:15.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.0192000000000002e-05 2023-07-02T14:07:15.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.976502+0000", 2023-07-02T14:07:15.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056056789999999997 2023-07-02T14:07:15.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.982107+0000", 2023-07-02T14:07:15.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.6842999999999997e-05 2023-07-02T14:07:15.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.982174+0000", 2023-07-02T14:07:15.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010792629999999999 2023-07-02T14:07:15.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.992967+0000", 2023-07-02T14:07:15.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8032999999999999e-05 2023-07-02T14:07:15.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.993015+0000", 2023-07-02T14:07:15.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075199419999999999 2023-07-02T14:07:15.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.000535+0000", 2023-07-02T14:07:15.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2507999999999999e-05 2023-07-02T14:07:15.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.912 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.912 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.000587+0000", 2023-07-02T14:07:15.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049331000000000002 2023-07-02T14:07:15.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.913 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.913 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.005520+0000", 2023-07-02T14:07:15.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4191999999999997e-05 2023-07-02T14:07:15.913 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.913 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.005595+0000", 2023-07-02T14:07:15.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067464739999999997 2023-07-02T14:07:15.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.012341+0000", 2023-07-02T14:07:15.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2277999999999998e-05 2023-07-02T14:07:15.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.012373+0000", 2023-07-02T14:07:15.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0032227039999999998 2023-07-02T14:07:15.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.015596+0000", 2023-07-02T14:07:15.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0318999999999999e-05 2023-07-02T14:07:15.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.015626+0000", 2023-07-02T14:07:15.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075373539999999996 2023-07-02T14:07:15.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.023164+0000", 2023-07-02T14:07:15.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4638000000000003e-05 2023-07-02T14:07:15.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.023198+0000", 2023-07-02T14:07:15.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062640539999999998 2023-07-02T14:07:15.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.029462+0000", 2023-07-02T14:07:15.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5593000000000003e-05 2023-07-02T14:07:15.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.029498+0000", 2023-07-02T14:07:15.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056884309999999999 2023-07-02T14:07:15.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.035186+0000", 2023-07-02T14:07:15.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1548000000000003e-05 2023-07-02T14:07:15.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.035228+0000", 2023-07-02T14:07:15.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051429190000000001 2023-07-02T14:07:15.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.040371+0000", 2023-07-02T14:07:15.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2172000000000003e-05 2023-07-02T14:07:15.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.921 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.040403+0000", 2023-07-02T14:07:15.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057273819999999996 2023-07-02T14:07:15.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.922 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.046130+0000", 2023-07-02T14:07:15.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.6150000000000003e-05 2023-07-02T14:07:15.922 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.922 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.922 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.046187+0000", 2023-07-02T14:07:15.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041953160000000001 2023-07-02T14:07:15.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.050382+0000", 2023-07-02T14:07:15.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0028000000000001e-05 2023-07-02T14:07:15.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.050412+0000", 2023-07-02T14:07:15.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071395260000000002 2023-07-02T14:07:15.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.057551+0000", 2023-07-02T14:07:15.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1424000000000003e-05 2023-07-02T14:07:15.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.057583+0000", 2023-07-02T14:07:15.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065879320000000003 2023-07-02T14:07:15.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.064171+0000", 2023-07-02T14:07:15.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1884000000000004e-05 2023-07-02T14:07:15.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.064253+0000", 2023-07-02T14:07:15.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052036649999999997 2023-07-02T14:07:15.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.069456+0000", 2023-07-02T14:07:15.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1013999999999995e-05 2023-07-02T14:07:15.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.069527+0000", 2023-07-02T14:07:15.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053928789999999997 2023-07-02T14:07:15.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.074920+0000", 2023-07-02T14:07:15.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9036999999999999e-05 2023-07-02T14:07:15.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.074959+0000", 2023-07-02T14:07:15.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066322229999999996 2023-07-02T14:07:15.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.081592+0000", 2023-07-02T14:07:15.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3004999999999997e-05 2023-07-02T14:07:15.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.081655+0000", 2023-07-02T14:07:15.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006005074 2023-07-02T14:07:15.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.931 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.087660+0000", 2023-07-02T14:07:15.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.8764999999999995e-05 2023-07-02T14:07:15.931 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.931 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.931 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.087738+0000", 2023-07-02T14:07:15.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045723650000000001 2023-07-02T14:07:15.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.092311+0000", 2023-07-02T14:07:15.932 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4039000000000002e-05 2023-07-02T14:07:15.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.092365+0000", 2023-07-02T14:07:15.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055219270000000003 2023-07-02T14:07:15.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.097887+0000", 2023-07-02T14:07:15.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6368000000000001e-05 2023-07-02T14:07:15.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.097923+0000", 2023-07-02T14:07:15.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054401240000000002 2023-07-02T14:07:15.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.103363+0000", 2023-07-02T14:07:15.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.2290000000000002e-05 2023-07-02T14:07:15.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.103415+0000", 2023-07-02T14:07:15.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0024881199999999999 2023-07-02T14:07:15.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.105904+0000", 2023-07-02T14:07:15.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1612000000000004e-05 2023-07-02T14:07:15.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:15.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.105985+0000", 2023-07-02T14:07:15.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00046834600000000002 2023-07-02T14:07:15.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:15.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.106454+0000", 2023-07-02T14:07:15.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002803187 2023-07-02T14:07:15.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:15.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.109257+0000", 2023-07-02T14:07:15.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0018006949999999999 2023-07-02T14:07:15.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:15.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.111057+0000", 2023-07-02T14:07:15.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5685e-05 2023-07-02T14:07:15.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:15.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.111113+0000", 2023-07-02T14:07:15.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000110272 2023-07-02T14:07:15.940 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.940 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:15.940 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.111223+0000", 2023-07-02T14:07:15.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.19699383500000001 2023-07-02T14:07:15.940 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.940 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:15.940 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.941 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.7359.0:54 2.7 2:f82cfcd4:::rbd_object_map.11fc3a5ad442:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e216)", 2023-07-02T14:07:15.941 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:07:11.914252+0000", 2023-07-02T14:07:15.941 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.5516140539999999, 2023-07-02T14:07:15.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.203157264, 2023-07-02T14:07:15.941 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:15.941 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:15.942 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:15.942 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.7359", 2023-07-02T14:07:15.942 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1139503436", 2023-07-02T14:07:15.942 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 54 2023-07-02T14:07:15.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.942 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:15.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:15.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914252+0000", 2023-07-02T14:07:15.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:15.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:15.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914252+0000", 2023-07-02T14:07:15.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:15.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:15.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914252+0000", 2023-07-02T14:07:15.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.01e-06 2023-07-02T14:07:15.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:15.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914255+0000", 2023-07-02T14:07:15.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2809999999999997e-06 2023-07-02T14:07:15.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:15.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914261+0000", 2023-07-02T14:07:15.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9339999999999999e-06 2023-07-02T14:07:15.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:15.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.914265+0000", 2023-07-02T14:07:15.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005297225 2023-07-02T14:07:15.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.919562+0000", 2023-07-02T14:07:15.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6037000000000001e-05 2023-07-02T14:07:15.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.919598+0000", 2023-07-02T14:07:15.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073030029999999998 2023-07-02T14:07:15.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.926901+0000", 2023-07-02T14:07:15.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.0710000000000005e-05 2023-07-02T14:07:15.948 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.948 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.948 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.926982+0000", 2023-07-02T14:07:15.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010335946 2023-07-02T14:07:15.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.937318+0000", 2023-07-02T14:07:15.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1531e-05 2023-07-02T14:07:15.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.937350+0000", 2023-07-02T14:07:15.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061697210000000004 2023-07-02T14:07:15.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.943519+0000", 2023-07-02T14:07:15.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7274999999999998e-05 2023-07-02T14:07:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.943557+0000", 2023-07-02T14:07:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054996339999999998 2023-07-02T14:07:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.949056+0000", 2023-07-02T14:07:15.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6187999999999996e-05 2023-07-02T14:07:15.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.949132+0000", 2023-07-02T14:07:15.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072118709999999999 2023-07-02T14:07:15.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.956344+0000", 2023-07-02T14:07:15.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3379000000000003e-05 2023-07-02T14:07:15.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.956418+0000", 2023-07-02T14:07:15.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006810183 2023-07-02T14:07:15.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.954 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.963228+0000", 2023-07-02T14:07:15.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6143999999999998e-05 2023-07-02T14:07:15.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.963264+0000", 2023-07-02T14:07:15.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064011729999999996 2023-07-02T14:07:15.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.955 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.955 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.969665+0000", 2023-07-02T14:07:15.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0876000000000004e-05 2023-07-02T14:07:15.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.956 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.969736+0000", 2023-07-02T14:07:15.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068130489999999998 2023-07-02T14:07:15.956 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.976549+0000", 2023-07-02T14:07:15.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2723000000000001e-05 2023-07-02T14:07:15.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.976632+0000", 2023-07-02T14:07:15.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055786350000000002 2023-07-02T14:07:15.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.982210+0000", 2023-07-02T14:07:15.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2147999999999998e-05 2023-07-02T14:07:15.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.982243+0000", 2023-07-02T14:07:15.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010927047 2023-07-02T14:07:15.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.993170+0000", 2023-07-02T14:07:15.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3072999999999996e-05 2023-07-02T14:07:15.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.993253+0000", 2023-07-02T14:07:15.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0073788739999999997 2023-07-02T14:07:15.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.000632+0000", 2023-07-02T14:07:15.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4482999999999998e-05 2023-07-02T14:07:15.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.000706+0000", 2023-07-02T14:07:15.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049438269999999996 2023-07-02T14:07:15.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.005650+0000", 2023-07-02T14:07:15.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4211999999999999e-05 2023-07-02T14:07:15.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.005734+0000", 2023-07-02T14:07:15.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0099212329999999998 2023-07-02T14:07:15.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.963 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.963 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.015655+0000", 2023-07-02T14:07:15.963 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2212000000000003e-05 2023-07-02T14:07:15.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.015688+0000", 2023-07-02T14:07:15.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075522000000000002 2023-07-02T14:07:15.964 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.023240+0000", 2023-07-02T14:07:15.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1103e-05 2023-07-02T14:07:15.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.023311+0000", 2023-07-02T14:07:15.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062237689999999997 2023-07-02T14:07:15.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.029535+0000", 2023-07-02T14:07:15.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2706000000000002e-05 2023-07-02T14:07:15.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.029567+0000", 2023-07-02T14:07:15.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057061489999999998 2023-07-02T14:07:15.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.035273+0000", 2023-07-02T14:07:15.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.5473000000000001e-05 2023-07-02T14:07:15.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.035349+0000", 2023-07-02T14:07:15.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005083667 2023-07-02T14:07:15.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.040433+0000", 2023-07-02T14:07:15.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2267999999999997e-05 2023-07-02T14:07:15.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.040465+0000", 2023-07-02T14:07:15.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057718029999999998 2023-07-02T14:07:15.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.046237+0000", 2023-07-02T14:07:15.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0457999999999995e-05 2023-07-02T14:07:15.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.046307+0000", 2023-07-02T14:07:15.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041300219999999997 2023-07-02T14:07:15.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.050437+0000", 2023-07-02T14:07:15.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0629000000000001e-05 2023-07-02T14:07:15.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.971 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.971 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.050468+0000", 2023-07-02T14:07:15.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071488419999999999 2023-07-02T14:07:15.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.057617+0000", 2023-07-02T14:07:15.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2085000000000002e-05 2023-07-02T14:07:15.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.057649+0000", 2023-07-02T14:07:15.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066525819999999998 2023-07-02T14:07:15.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.064301+0000", 2023-07-02T14:07:15.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1353000000000006e-05 2023-07-02T14:07:15.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.064373+0000", 2023-07-02T14:07:15.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052024510000000003 2023-07-02T14:07:15.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.069575+0000", 2023-07-02T14:07:15.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1688999999999998e-05 2023-07-02T14:07:15.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.069647+0000", 2023-07-02T14:07:15.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053450549999999996 2023-07-02T14:07:15.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.074992+0000", 2023-07-02T14:07:15.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2591e-05 2023-07-02T14:07:15.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.075024+0000", 2023-07-02T14:07:15.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0066836200000000004 2023-07-02T14:07:15.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.081708+0000", 2023-07-02T14:07:15.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6578000000000001e-05 2023-07-02T14:07:15.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.081745+0000", 2023-07-02T14:07:15.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060445710000000003 2023-07-02T14:07:15.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.978 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.087789+0000", 2023-07-02T14:07:15.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8800000000000005e-05 2023-07-02T14:07:15.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.979 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.979 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.087858+0000", 2023-07-02T14:07:15.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004539253 2023-07-02T14:07:15.979 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.092397+0000", 2023-07-02T14:07:15.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3791000000000002e-05 2023-07-02T14:07:15.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.092431+0000", 2023-07-02T14:07:15.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055268849999999996 2023-07-02T14:07:15.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.097958+0000", 2023-07-02T14:07:15.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5769999999999997e-05 2023-07-02T14:07:15.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.098004+0000", 2023-07-02T14:07:15.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054336260000000004 2023-07-02T14:07:15.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.103437+0000", 2023-07-02T14:07:15.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1594999999999998e-05 2023-07-02T14:07:15.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.103469+0000", 2023-07-02T14:07:15.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047415039999999997 2023-07-02T14:07:15.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.108210+0000", 2023-07-02T14:07:15.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2256000000000007e-05 2023-07-02T14:07:15.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.108293+0000", 2023-07-02T14:07:15.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0029549649999999999 2023-07-02T14:07:15.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.111248+0000", 2023-07-02T14:07:15.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7308000000000002e-05 2023-07-02T14:07:15.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:15.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.111295+0000", 2023-07-02T14:07:15.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00025653299999999998 2023-07-02T14:07:15.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.986 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:15.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.111551+0000", 2023-07-02T14:07:15.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.002569259 2023-07-02T14:07:15.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:15.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.114121+0000", 2023-07-02T14:07:15.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031857360000000002 2023-07-02T14:07:15.987 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:15.987 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.117306+0000", 2023-07-02T14:07:15.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.3785000000000003e-05 2023-07-02T14:07:15.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:15.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.117340+0000", 2023-07-02T14:07:15.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9338999999999995e-05 2023-07-02T14:07:15.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:15.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.117410+0000", 2023-07-02T14:07:15.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.203157264 2023-07-02T14:07:15.989 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.989 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:15.989 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:15.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.990 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.7359.0:57 2.7 2:f82cfcd4:::rbd_object_map.11fc3a5ad442:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e216)", 2023-07-02T14:07:15.990 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:07:11.928146+0000", 2023-07-02T14:07:15.990 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.5377202919999999, 2023-07-02T14:07:15.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.195341611, 2023-07-02T14:07:15.990 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:15.990 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:15.990 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:15.991 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.7359", 2023-07-02T14:07:15.991 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1139503436", 2023-07-02T14:07:15.991 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 57 2023-07-02T14:07:15.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.991 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:15.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:15.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.928146+0000", 2023-07-02T14:07:15.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:15.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:15.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.928146+0000", 2023-07-02T14:07:15.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:15.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:15.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.928145+0000", 2023-07-02T14:07:15.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3170000000000001e-06 2023-07-02T14:07:15.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:15.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.928150+0000", 2023-07-02T14:07:15.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:07:15.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.994 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.994 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:15.994 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.928157+0000", 2023-07-02T14:07:15.994 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1459999999999998e-06 2023-07-02T14:07:15.994 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:15.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.928161+0000", 2023-07-02T14:07:15.995 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0092227639999999996 2023-07-02T14:07:15.995 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.995 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.995 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.995 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.937384+0000", 2023-07-02T14:07:15.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1967999999999999e-05 2023-07-02T14:07:15.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.996 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.996 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.996 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.937466+0000", 2023-07-02T14:07:15.996 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061252420000000004 2023-07-02T14:07:15.996 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.943591+0000", 2023-07-02T14:07:15.997 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6402999999999997e-05 2023-07-02T14:07:15.997 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.997 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.997 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.997 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.943628+0000", 2023-07-02T14:07:15.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055430030000000003 2023-07-02T14:07:15.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.998 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.998 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.949171+0000", 2023-07-02T14:07:15.998 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7598000000000002e-05 2023-07-02T14:07:15.998 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.998 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:15.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.949208+0000", 2023-07-02T14:07:15.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072440170000000002 2023-07-02T14:07:15.999 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:15.999 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:15.999 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:15.999 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.956452+0000", 2023-07-02T14:07:15.999 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0850999999999999e-05 2023-07-02T14:07:16.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.000 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.956493+0000", 2023-07-02T14:07:16.000 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067986699999999997 2023-07-02T14:07:16.000 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.000 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.000 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.963292+0000", 2023-07-02T14:07:16.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5088e-05 2023-07-02T14:07:16.001 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.001 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.001 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.001 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.963327+0000", 2023-07-02T14:07:16.001 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0064643929999999997 2023-07-02T14:07:16.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.002 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.969791+0000", 2023-07-02T14:07:16.002 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.2089000000000006e-05 2023-07-02T14:07:16.002 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.002 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.002 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.969874+0000", 2023-07-02T14:07:16.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0068295409999999997 2023-07-02T14:07:16.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.003 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.003 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.003 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.976703+0000", 2023-07-02T14:07:16.003 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.4887000000000002e-05 2023-07-02T14:07:16.003 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.976788+0000", 2023-07-02T14:07:16.004 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054977719999999997 2023-07-02T14:07:16.004 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.004 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.004 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.004 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.982286+0000", 2023-07-02T14:07:16.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6649000000000004e-05 2023-07-02T14:07:16.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.005 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.005 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.005 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.982362+0000", 2023-07-02T14:07:16.005 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.01093265 2023-07-02T14:07:16.005 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.993295+0000", 2023-07-02T14:07:16.006 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.625e-05 2023-07-02T14:07:16.006 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.006 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.006 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.006 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.993331+0000", 2023-07-02T14:07:16.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0074294729999999998 2023-07-02T14:07:16.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.007 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.007 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.000761+0000", 2023-07-02T14:07:16.007 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9523000000000004e-05 2023-07-02T14:07:16.007 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.007 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.000840+0000", 2023-07-02T14:07:16.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.004951614 2023-07-02T14:07:16.008 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.008 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.008 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.008 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.005792+0000", 2023-07-02T14:07:16.008 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0895999999999993e-05 2023-07-02T14:07:16.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.009 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.005863+0000", 2023-07-02T14:07:16.009 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0098495319999999994 2023-07-02T14:07:16.009 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.009 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.009 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.015712+0000", 2023-07-02T14:07:16.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9465e-05 2023-07-02T14:07:16.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.010 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.010 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.010 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.015742+0000", 2023-07-02T14:07:16.010 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007705478 2023-07-02T14:07:16.010 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.023447+0000", 2023-07-02T14:07:16.011 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3809999999999999e-05 2023-07-02T14:07:16.011 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.011 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.011 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.011 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.023531+0000", 2023-07-02T14:07:16.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060786770000000002 2023-07-02T14:07:16.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.012 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.012 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.012 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.029610+0000", 2023-07-02T14:07:16.012 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4285999999999997e-05 2023-07-02T14:07:16.012 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.029644+0000", 2023-07-02T14:07:16.013 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057407500000000002 2023-07-02T14:07:16.013 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.013 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.013 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.013 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.035385+0000", 2023-07-02T14:07:16.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5029e-05 2023-07-02T14:07:16.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.014 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.014 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.035420+0000", 2023-07-02T14:07:16.014 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050764490000000002 2023-07-02T14:07:16.014 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.014 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.040496+0000", 2023-07-02T14:07:16.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.3288999999999998e-05 2023-07-02T14:07:16.015 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.015 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.015 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.015 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.040540+0000", 2023-07-02T14:07:16.015 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058053289999999997 2023-07-02T14:07:16.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.016 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.046345+0000", 2023-07-02T14:07:16.016 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5312999999999999e-05 2023-07-02T14:07:16.016 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.016 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.016 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.046380+0000", 2023-07-02T14:07:16.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041126289999999996 2023-07-02T14:07:16.017 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.017 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.017 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.017 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.050493+0000", 2023-07-02T14:07:16.017 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0889999999999997e-05 2023-07-02T14:07:16.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.018 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.018 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.050524+0000", 2023-07-02T14:07:16.018 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071567180000000003 2023-07-02T14:07:16.018 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.018 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.057680+0000", 2023-07-02T14:07:16.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1659000000000003e-05 2023-07-02T14:07:16.019 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.019 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.019 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.019 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.057712+0000", 2023-07-02T14:07:16.019 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006716278 2023-07-02T14:07:16.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.020 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.020 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.020 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.064428+0000", 2023-07-02T14:07:16.020 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4110999999999996e-05 2023-07-02T14:07:16.020 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.064502+0000", 2023-07-02T14:07:16.021 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051838090000000002 2023-07-02T14:07:16.021 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.021 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.021 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.021 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.069686+0000", 2023-07-02T14:07:16.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9461e-05 2023-07-02T14:07:16.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.022 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.022 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.022 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.069736+0000", 2023-07-02T14:07:16.022 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053187 2023-07-02T14:07:16.022 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.023 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.075054+0000", 2023-07-02T14:07:16.023 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2113000000000002e-05 2023-07-02T14:07:16.023 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.023 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.023 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.075087+0000", 2023-07-02T14:07:16.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067066900000000004 2023-07-02T14:07:16.024 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.024 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.024 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.024 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.081793+0000", 2023-07-02T14:07:16.024 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2648000000000003e-05 2023-07-02T14:07:16.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.025 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.025 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.081826+0000", 2023-07-02T14:07:16.025 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0060787899999999997 2023-07-02T14:07:16.025 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.025 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.087905+0000", 2023-07-02T14:07:16.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7435999999999995e-05 2023-07-02T14:07:16.026 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.026 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.026 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.026 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.087982+0000", 2023-07-02T14:07:16.026 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0044806029999999997 2023-07-02T14:07:16.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.027 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.027 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.092463+0000", 2023-07-02T14:07:16.027 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1893000000000001e-05 2023-07-02T14:07:16.027 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.027 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.092495+0000", 2023-07-02T14:07:16.028 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055513460000000004 2023-07-02T14:07:16.028 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.028 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.028 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.028 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.098046+0000", 2023-07-02T14:07:16.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6792999999999994e-05 2023-07-02T14:07:16.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.029 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.029 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.098133+0000", 2023-07-02T14:07:16.029 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0053550849999999999 2023-07-02T14:07:16.029 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.029 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.103488+0000", 2023-07-02T14:07:16.030 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0027000000000001e-05 2023-07-02T14:07:16.030 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.030 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.030 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.030 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.103528+0000", 2023-07-02T14:07:16.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048218669999999996 2023-07-02T14:07:16.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.031 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.031 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.031 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.108350+0000", 2023-07-02T14:07:16.031 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7824999999999993e-05 2023-07-02T14:07:16.031 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.032 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.108438+0000", 2023-07-02T14:07:16.032 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046592220000000002 2023-07-02T14:07:16.032 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.032 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.032 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.113097+0000", 2023-07-02T14:07:16.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7736000000000003e-05 2023-07-02T14:07:16.033 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.033 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.033 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.033 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.113165+0000", 2023-07-02T14:07:16.033 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0042955179999999999 2023-07-02T14:07:16.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.034 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.117460+0000", 2023-07-02T14:07:16.034 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3097999999999996e-05 2023-07-02T14:07:16.034 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.034 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.034 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:16.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.117543+0000", 2023-07-02T14:07:16.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00036729599999999997 2023-07-02T14:07:16.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.035 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.035 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:16.035 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.117910+0000", 2023-07-02T14:07:16.035 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0031789909999999999 2023-07-02T14:07:16.035 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:16.036 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.121089+0000", 2023-07-02T14:07:16.036 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0022672320000000001 2023-07-02T14:07:16.036 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.036 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.036 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:16.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.123357+0000", 2023-07-02T14:07:16.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3028000000000001e-05 2023-07-02T14:07:16.037 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.037 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.037 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:16.037 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.123410+0000", 2023-07-02T14:07:16.037 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7989000000000002e-05 2023-07-02T14:07:16.038 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.038 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.038 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:16.038 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.123488+0000", 2023-07-02T14:07:16.038 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.195341611 2023-07-02T14:07:16.038 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:16.038 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:16.038 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:16.039 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.039 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.039 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.7359.0:62 2.7 2:f82cfcd4:::rbd_object_map.11fc3a5ad442:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected e216)", 2023-07-02T14:07:16.039 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:07:11.931534+0000", 2023-07-02T14:07:16.039 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 3.53433273, 2023-07-02T14:07:16.039 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.197008458, 2023-07-02T14:07:16.039 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:16.040 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.040 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:16.040 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.7359", 2023-07-02T14:07:16.040 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/1139503436", 2023-07-02T14:07:16.040 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 62 2023-07-02T14:07:16.040 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.040 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:16.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:16.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.931534+0000", 2023-07-02T14:07:16.041 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:16.041 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.041 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.041 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:16.041 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.931534+0000", 2023-07-02T14:07:16.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:16.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.042 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.042 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:16.042 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.931532+0000", 2023-07-02T14:07:16.042 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.968e-06 2023-07-02T14:07:16.042 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:16.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.931539+0000", 2023-07-02T14:07:16.043 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0682e-05 2023-07-02T14:07:16.043 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.043 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.043 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:16.043 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.931550+0000", 2023-07-02T14:07:16.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3499999999999997e-06 2023-07-02T14:07:16.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.044 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.044 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.044 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.931558+0000", 2023-07-02T14:07:16.044 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0059246330000000003 2023-07-02T14:07:16.044 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.937483+0000", 2023-07-02T14:07:16.045 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1597000000000001e-05 2023-07-02T14:07:16.045 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.045 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.045 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.045 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.937535+0000", 2023-07-02T14:07:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061258900000000002 2023-07-02T14:07:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.943661+0000", 2023-07-02T14:07:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4956999999999998e-05 2023-07-02T14:07:16.046 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.943695+0000", 2023-07-02T14:07:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0055520860000000003 2023-07-02T14:07:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.047 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.949248+0000", 2023-07-02T14:07:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.1274999999999998e-05 2023-07-02T14:07:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.949319+0000", 2023-07-02T14:07:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0072031839999999996 2023-07-02T14:07:16.048 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.956522+0000", 2023-07-02T14:07:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5784000000000006e-05 2023-07-02T14:07:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.049 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.956588+0000", 2023-07-02T14:07:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067543890000000004 2023-07-02T14:07:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.963342+0000", 2023-07-02T14:07:16.050 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4150999999999999e-05 2023-07-02T14:07:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.963376+0000", 2023-07-02T14:07:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0065304409999999997 2023-07-02T14:07:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.051 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.969907+0000", 2023-07-02T14:07:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9005000000000007e-05 2023-07-02T14:07:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.969986+0000", 2023-07-02T14:07:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006833932 2023-07-02T14:07:16.052 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.976820+0000", 2023-07-02T14:07:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4041999999999993e-05 2023-07-02T14:07:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.053 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.976894+0000", 2023-07-02T14:07:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0054878119999999999 2023-07-02T14:07:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.982382+0000", 2023-07-02T14:07:16.054 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1711000000000002e-05 2023-07-02T14:07:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.982413+0000", 2023-07-02T14:07:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010958554000000001 2023-07-02T14:07:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.055 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.993372+0000", 2023-07-02T14:07:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8614000000000004e-05 2023-07-02T14:07:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:11.993440+0000", 2023-07-02T14:07:16.056 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.007431834 2023-07-02T14:07:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.000872+0000", 2023-07-02T14:07:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2156000000000005e-05 2023-07-02T14:07:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.057 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.000944+0000", 2023-07-02T14:07:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0049890480000000003 2023-07-02T14:07:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.005934+0000", 2023-07-02T14:07:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9731000000000001e-05 2023-07-02T14:07:16.058 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.005993+0000", 2023-07-02T14:07:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0097622739999999996 2023-07-02T14:07:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.059 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.015756+0000", 2023-07-02T14:07:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1656999999999999e-05 2023-07-02T14:07:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.015787+0000", 2023-07-02T14:07:16.060 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0077861720000000001 2023-07-02T14:07:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.023573+0000", 2023-07-02T14:07:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4190999999999995e-05 2023-07-02T14:07:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.061 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.023648+0000", 2023-07-02T14:07:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006015744 2023-07-02T14:07:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.029663+0000", 2023-07-02T14:07:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2189e-05 2023-07-02T14:07:16.062 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.029695+0000", 2023-07-02T14:07:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057536549999999999 2023-07-02T14:07:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.063 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.035449+0000", 2023-07-02T14:07:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.3548999999999997e-05 2023-07-02T14:07:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.035523+0000", 2023-07-02T14:07:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0050347009999999999 2023-07-02T14:07:16.064 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.040557+0000", 2023-07-02T14:07:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1863999999999998e-05 2023-07-02T14:07:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.065 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.040589+0000", 2023-07-02T14:07:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0058235580000000004 2023-07-02T14:07:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.046413+0000", 2023-07-02T14:07:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9206000000000001e-05 2023-07-02T14:07:16.066 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.046482+0000", 2023-07-02T14:07:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0040555030000000002 2023-07-02T14:07:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.067 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.050537+0000", 2023-07-02T14:07:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1489000000000003e-05 2023-07-02T14:07:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.050569+0000", 2023-07-02T14:07:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0071619159999999999 2023-07-02T14:07:16.068 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.057731+0000", 2023-07-02T14:07:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1925999999999999e-05 2023-07-02T14:07:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.069 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.057763+0000", 2023-07-02T14:07:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067665980000000004 2023-07-02T14:07:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.064529+0000", 2023-07-02T14:07:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.2502000000000001e-05 2023-07-02T14:07:16.070 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.064572+0000", 2023-07-02T14:07:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051957949999999996 2023-07-02T14:07:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.071 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.069768+0000", 2023-07-02T14:07:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.6746e-05 2023-07-02T14:07:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.069844+0000", 2023-07-02T14:07:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005261366 2023-07-02T14:07:16.072 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.075106+0000", 2023-07-02T14:07:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8949999999999998e-05 2023-07-02T14:07:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.073 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.075145+0000", 2023-07-02T14:07:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.006706297 2023-07-02T14:07:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.081851+0000", 2023-07-02T14:07:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.1496999999999999e-05 2023-07-02T14:07:16.074 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.081893+0000", 2023-07-02T14:07:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061394409999999998 2023-07-02T14:07:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.075 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.088032+0000", 2023-07-02T14:07:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6759000000000001e-05 2023-07-02T14:07:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.088119+0000", 2023-07-02T14:07:16.076 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0043945379999999999 2023-07-02T14:07:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.092513+0000", 2023-07-02T14:07:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2280000000000003e-05 2023-07-02T14:07:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.077 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.092546+0000", 2023-07-02T14:07:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0056163309999999996 2023-07-02T14:07:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.098162+0000", 2023-07-02T14:07:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4772999999999999e-05 2023-07-02T14:07:16.078 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.098217+0000", 2023-07-02T14:07:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.005535463 2023-07-02T14:07:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.079 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.103752+0000", 2023-07-02T14:07:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2311000000000006e-05 2023-07-02T14:07:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.103824+0000", 2023-07-02T14:07:16.080 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00464482 2023-07-02T14:07:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.108469+0000", 2023-07-02T14:07:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6438000000000001e-05 2023-07-02T14:07:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.081 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.108556+0000", 2023-07-02T14:07:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0046322689999999996 2023-07-02T14:07:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.113188+0000", 2023-07-02T14:07:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.6359000000000002e-05 2023-07-02T14:07:16.082 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.113224+0000", 2023-07-02T14:07:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0063363159999999998 2023-07-02T14:07:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.083 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.119561+0000", 2023-07-02T14:07:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.7528999999999995e-05 2023-07-02T14:07:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for rw locks", 2023-07-02T14:07:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.119648+0000", 2023-07-02T14:07:16.084 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038868000000000002 2023-07-02T14:07:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.123535+0000", 2023-07-02T14:07:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.8840999999999999e-05 2023-07-02T14:07:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.085 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.123624+0000", 2023-07-02T14:07:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000355905 2023-07-02T14:07:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.123980+0000", 2023-07-02T14:07:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0036652999999999998 2023-07-02T14:07:16.086 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.127645+0000", 2023-07-02T14:07:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00083249699999999997 2023-07-02T14:07:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.087 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.128478+0000", 2023-07-02T14:07:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.0609999999999999e-05 2023-07-02T14:07:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.128518+0000", 2023-07-02T14:07:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3937000000000001e-05 2023-07-02T14:07:16.088 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:07:12.128542+0000", 2023-07-02T14:07:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.197008458 2023-07-02T14:07:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:16.089 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:16.090 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:16.090 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:16.092 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:16.093 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:07:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:07:16.094 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2023-07-02T14:07:14.291299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11898068505,"num_objects":3570,"num_object_clones":2998,"num_object_copies":7140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":88995,"num_read_kb":15468831,"num_write":27307,"num_write_kb":11545887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3265,"num_bytes_recovered":11163399618,"num_keys_recovered":186,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26951,"ondisk_log_size":26951,"up":18,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":21664116,"kb_used_data":21321876,"kb_used_omap":229,"kb_used_meta":341978,"kb_avail":355823244,"statfs":{"total":386547056640,"available":364363001856,"internally_reserved":0,"allocated":21833601024,"data_stored":25442403265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":350185694},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21725346,"num_objects":27,"num_object_clones":16,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-6,"num_objects_misplaced":-666,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1058,"num_read_kb":861722,"num_write":167,"num_write_kb":21323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":675,"num_bytes_recovered":2333349946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.921151"},"pg_stats":[{"pgid":"2.7","version":"216'3699","reported_seq":9410,"reported_epoch":216,"state":"active+clean","last_fresh":"2023-07-02T14:07:12.123411+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:07:12.123411+0000","last_peered":"2023-07-02T14:07:12.123411+0000","last_clean":"2023-07-02T14:07:12.123411+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:07:12.123411+0000","last_undegraded":"2023-07-02T14:07:12.123411+0000","last_fullsized":"2023-07-02T14:07:12.123411+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3699,"ondisk_log_size":3699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1575041492,"num_objects":472,"num_object_clones":397,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":58088,"num_read_kb":2022620,"num_write":3141,"num_write_kb":1530739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"215'3058","reported_seq":5145,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:10.324843+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:07:10.324843+0000","last_peered":"2023-07-02T14:07:10.324843+0000","last_clean":"2023-07-02T14:07:10.324843+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:07:10.324843+0000","last_undegraded":"2023-07-02T14:07:10.324843+0000","last_fullsized":"2023-07-02T14:07:10.324843+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3058,"ondisk_log_size":3058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535489458,"num_objects":482,"num_object_clones":414,"num_object_copies":964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":0,"num_read":1926,"num_read_kb":2091895,"num_write":2644,"num_write_kb":1489957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"216'3204","reported_seq":4695,"reported_epoch":216,"state":"active+undersized+remapped","last_fresh":"2023-07-02T14:07:12.410373+0000","last_change":"2023-07-02T14:07:12.410555+0000","last_active":"2023-07-02T14:07:12.410373+0000","last_peered":"2023-07-02T14:07:12.410373+0000","last_clean":"2023-07-02T14:07:01.043240+0000","last_became_active":"2023-07-02T14:07:04.030791+0000","last_became_peered":"2023-07-02T14:07:04.030791+0000","last_unstale":"2023-07-02T14:07:12.410373+0000","last_undegraded":"2023-07-02T14:07:03.107328+0000","last_fullsized":"2023-07-02T14:07:03.101623+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3204,"ondisk_log_size":3204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1517142988,"num_objects":428,"num_object_clones":355,"num_object_copies":856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":635,"num_read_kb":1967916,"num_write":2847,"num_write_kb":1467896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":2369610862,"num_keys_recovered":8,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3],"avail_no_missing":["3","0","4"],"object_location_counts":[{"shards":"3","objects":428}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"215'6476","reported_seq":8964,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:10.334213+0000","last_change":"2023-07-02T14:06:29.572710+0000","last_active":"2023-07-02T14:07:10.334213+0000","last_peered":"2023-07-02T14:07:10.334213+0000","last_clean":"2023-07-02T14:07:10.334213+0000","last_became_active":"2023-07-02T14:06:29.569824+0000","last_became_peered":"2023-07-02T14:06:29.569824+0000","last_unstale":"2023-07-02T14:07:10.334213+0000","last_undegraded":"2023-07-02T14:07:10.334213+0000","last_fullsized":"2023-07-02T14:07:10.334213+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":6476,"ondisk_log_size":6476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1672852930,"num_objects":488,"num_object_clones":413,"num_object_copies":976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":22953,"num_read_kb":2152050,"num_write":9580,"num_write_kb":1632373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":116,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:09.372972+0000","last_change":"2023-07-02T14:07:02.103009+0000","last_active":"2023-07-02T14:07:09.372972+0000","last_peered":"2023-07-02T14:07:09.372972+0000","last_clean":"2023-07-02T14:07:09.372972+0000","last_became_active":"2023-07-02T14:07:02.102654+0000","last_became_peered":"2023-07-02T14:07:02.102654+0000","last_unstale":"2023-07-02T14:07:09.372972+0000","last_undegraded":"2023-07-02T14:07:09.372972+0000","last_fullsized":"2023-07-02T14:07:09.372972+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"215'2964","reported_seq":4463,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:10.302466+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:07:10.302466+0000","last_peered":"2023-07-02T14:07:10.302466+0000","last_clean":"2023-07-02T14:07:10.302466+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:07:10.302466+0000","last_undegraded":"2023-07-02T14:07:10.302466+0000","last_fullsized":"2023-07-02T14:07:10.302466+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2964,"ondisk_log_size":2964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1658680446,"num_objects":491,"num_object_clones":414,"num_object_copies":982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":0,"num_read":1509,"num_read_kb":2028229,"num_write":2550,"num_write_kb":1608728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"215'2867","reported_seq":3734,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:10.282420+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:07:10.282420+0000","last_peered":"2023-07-02T14:07:10.282420+0000","last_clean":"2023-07-02T14:07:10.282420+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:07:10.282420+0000","last_undegraded":"2023-07-02T14:07:10.282420+0000","last_fullsized":"2023-07-02T14:07:10.282420+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2867,"ondisk_log_size":2867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1564710484,"num_objects":455,"num_object_clones":382,"num_object_copies":910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":455,"num_whiteouts":0,"num_read":949,"num_read_kb":1945167,"num_write":2485,"num_write_kb":1518698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"215'2325","reported_seq":5128,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:10.305571+0000","last_change":"2023-07-02T14:07:09.383558+0000","last_active":"2023-07-02T14:07:10.305571+0000","last_peered":"2023-07-02T14:07:10.305571+0000","last_clean":"2023-07-02T14:07:10.305571+0000","last_became_active":"2023-07-02T14:07:09.382190+0000","last_became_peered":"2023-07-02T14:07:09.382190+0000","last_unstale":"2023-07-02T14:07:10.305571+0000","last_undegraded":"2023-07-02T14:07:10.305571+0000","last_fullsized":"2023-07-02T14:07:10.305571+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2325,"ondisk_log_size":2325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1230505737,"num_objects":402,"num_object_clones":325,"num_object_copies":804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":1898,"num_read_kb":1730263,"num_write":2000,"num_write_kb":1186689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":1578968698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"215'2358","reported_seq":3456,"reported_epoch":215,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-07-02T14:07:10.279927+0000","last_change":"2023-07-02T14:07:04.022783+0000","last_active":"2023-07-02T14:07:10.279927+0000","last_peered":"2023-07-02T14:07:10.279927+0000","last_clean":"2023-07-02T14:07:01.026611+0000","last_became_active":"2023-07-02T14:07:04.018152+0000","last_became_peered":"2023-07-02T14:07:04.018152+0000","last_unstale":"2023-07-02T14:07:10.279927+0000","last_undegraded":"2023-07-02T14:07:03.110294+0000","last_fullsized":"2023-07-02T14:07:03.103717+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2358,"ondisk_log_size":2358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1143644970,"num_objects":352,"num_object_clones":298,"num_object_copies":704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":1037,"num_read_kb":1530691,"num_write":2060,"num_write_kb":1110807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":271880192,"num_keys_recovered":4,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":346},{"shards":"1","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":11898068505,"num_objects":3570,"num_object_clones":2998,"num_object_copies":7140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":88995,"num_read_kb":15468831,"num_write":27307,"num_write_kb":11545887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3265,"num_bytes_recovered":11163399618,"num_keys_recovered":186,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25670000640,"data_stored":29831275485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":26951,"ondisk_log_size":26951,"up":16,"acting":14,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640323,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640323,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5235024,"kb_used_data":5143176,"kb_used_omap":0,"kb_used_meta":91840,"kb_avail":89136816,"statfs":{"total":96636764160,"available":91276099584,"internally_reserved":0,"allocated":5266612224,"data_stored":6072961790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":94044160},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.307}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44,"15min":0.46400000000000002},"min":{"1min":0.28399999999999997,"5min":0.215,"15min":0.215},"max":{"1min":0.71499999999999997,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.45000000000000001},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.32800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66800000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.39900000000000002}]},{"osd":3,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.38300000000000001},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45900000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.36799999999999999,"15min":0.36299999999999999},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.48799999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.56200000000000006}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.44900000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.433,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59099999999999997,"5min":1.026,"15min":1.026},"last":0.42899999999999999}]}]},{"osd":3,"up_from":133,"seq":571230650395,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4568928,"kb_used_data":4498776,"kb_used_omap":229,"kb_used_meta":69914,"kb_avail":89802912,"statfs":{"total":96636764160,"available":91958181888,"internally_reserved":0,"allocated":4606746624,"data_stored":5357897862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":71592158},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.45500000000000002}]},{"osd":1,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.372,"15min":0.36599999999999999},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.39600000000000002}]},{"osd":2,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.28699999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.44900000000000001,"15min":0.439},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.47899999999999998}]},{"osd":4,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.35799999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.36199999999999999,"15min":0.35299999999999998},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.35699999999999998}]},{"osd":5,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.39800000000000002},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.433},{"interface":"front","average":{"1min":0.439,"5min":0.39500000000000002,"15min":0.38800000000000001},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.315}]}]},{"osd":2,"up_from":11,"seq":47244640323,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3192604,"kb_used_data":3138708,"kb_used_omap":0,"kb_used_meta":53888,"kb_avail":91179236,"statfs":{"total":96636764160,"available":93367537664,"internally_reserved":0,"allocated":3214036992,"data_stored":3691892043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55181312},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.28799999999999998}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.42499999999999999,"15min":0.374},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.376,"5min":0.33500000000000002,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.22500000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:07:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.91100000000000003,"5min":1.929,"15min":1.929},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.54800000000000004,"15min":0.55600000000000005},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93999999999999995,"5min":2,"15min":2},"last":0.35999999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.40899999999999997}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68400000000000005,"5min":1.071,"15min":1.071},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":0.432}]}]},{"osd":0,"up_from":0,"seq":712964571153,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":11,"seq":47244640323,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8667560,"kb_used_data":8541216,"kb_used_omap":0,"kb_used_meta":126336,"kb_avail":85704280,"statfs":{"total":96636764160,"available":87761182720,"internally_reserved":0,"allocated":8746205184,"data_stored":10319651570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":129368064},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.32600000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.247},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.224,"5min":0.184,"15min":0.184},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.33800000000000002}]},{"osd":3,"last update":"Sun Jul 2 14:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.49099999999999999,"15min":0.47299999999999998},"min":{"1min":0.30299999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.38600000000000001}]},{"osd":4,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71899999999999997,"5min":1.524,"15min":1.524},"last":0.51000000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.80300000000000005,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84399999999999997,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.42999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3802202112,"data_stored":4305599675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8745132032,"data_stored":10339602553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3211833344,"data_stored":3699260114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4604137472,"data_stored":5356872530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5306695680,"data_stored":6129940613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:07:16.096 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-07-02T14:07:16.096 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:16.090+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:16.110 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is set 2023-07-02T14:07:16.192 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:16.191+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:16.248 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:16.248 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:07:16.248 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:07:16.248 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:16.259 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:07:16.300 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:16.299+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:16.393 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:16.392+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:16.440 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:07:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:07:16.440 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:07:16.440 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:16.452 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:07:16.493 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:16.493+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:16.515 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:16.515 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 352/7140 objects degraded (4.930%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":339,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":219,"num_osds":6,"num_up_osds":6,"osd_up_since":1688306752,"num_in_osds":4,"osd_in_since":1688306821,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+undersized+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":3570,"data_bytes":11898068505,"bytes_used":22184054784,"bytes_avail":364363001856,"bytes_total":386547056640,"degraded_objects":352,"degraded_total":7140,"degraded_ratio":0.049299719887955187,"recovering_objects_per_sec":137,"recovering_bytes_per_sec":474147194,"recovering_keys_per_sec":0,"num_objects_recovered":675,"num_bytes_recovered":2333349946,"num_keys_recovered":0,"read_bytes_sec":179308321,"write_bytes_sec":4436919,"read_op_per_sec":214,"write_op_per_sec":33},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2023-07-02T14:06:52.287392+0000","services":{}},"progress_events":{"39f7aefe-a986-4f7d-8f18-834c65a827f0":{"message":"Rebalancing after osd.0 marked out (10s)\n [==============..............] (remaining: 9s)","progress":0.50810569524765015,"add_to_ceph_s":false},"c9df9a9d-66a6-47d3-b334-4774a23f2050":{"message":"Global Recovery Event (5s)\n [=====================.......] (remaining: 1s)","progress":0.77777779102325439,"add_to_ceph_s":true}}} 2023-07-02T14:07:16.530 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2023-07-02T14:07:16.531 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:07:16.594 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:16.593+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:16.626 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:16.626 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:07:16.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:07:16.627 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:07:16.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.627 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18643 2.5 e176/170 2:a9ace6f8:::rbd_data.125de1449605.0000000000000187:head v 176'2314, mlcod=173'2311)", 2023-07-02T14:07:16.627 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:07:16.627 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 71.197305877000005, 2023-07-02T14:07:16.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015027297, 2023-07-02T14:07:16.628 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:16.628 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.628 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:16.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:16.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:07:16.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:16.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:16.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:07:16.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:16.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:16.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426547+0000", 2023-07-02T14:07:16.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056864430000000002 2023-07-02T14:07:16.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:16.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432234+0000", 2023-07-02T14:07:16.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.415e-05 2023-07-02T14:07:16.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:16.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432248+0000", 2023-07-02T14:07:16.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7490000000000001e-06 2023-07-02T14:07:16.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.632 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.632 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432255+0000", 2023-07-02T14:07:16.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3915000000000002e-05 2023-07-02T14:07:16.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.633 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.633 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:16.633 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432299+0000", 2023-07-02T14:07:16.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1214000000000001e-05 2023-07-02T14:07:16.633 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432320+0000", 2023-07-02T14:07:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0092114950000000001 2023-07-02T14:07:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.441531+0000", 2023-07-02T14:07:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4422999999999998e-05 2023-07-02T14:07:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.441576+0000", 2023-07-02T14:07:16.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015027297 2023-07-02T14:07:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18881 2.5 e177/170 2:b1731d2f:::rbd_data.125de1449605.0000000000000024:head v 177'2350, mlcod=177'2347)", 2023-07-02T14:07:16.636 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:07:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 69.717940053000007, 2023-07-02T14:07:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020820709999999999, 2023-07-02T14:07:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:07:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:07:16.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905913+0000", 2023-07-02T14:07:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0080191519999999999 2023-07-02T14:07:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913932+0000", 2023-07-02T14:07:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3757999999999999e-05 2023-07-02T14:07:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:16.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913956+0000", 2023-07-02T14:07:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4009999999999997e-06 2023-07-02T14:07:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913965+0000", 2023-07-02T14:07:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3383e-05 2023-07-02T14:07:16.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914018+0000", 2023-07-02T14:07:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7674e-05 2023-07-02T14:07:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914046+0000", 2023-07-02T14:07:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012656555999999999 2023-07-02T14:07:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.926702+0000", 2023-07-02T14:07:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2891000000000001e-05 2023-07-02T14:07:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:16.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.926735+0000", 2023-07-02T14:07:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020820709999999999 2023-07-02T14:07:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.645 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19073 2.5 e177/170 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:head v 177'2378, mlcod=177'2376)", 2023-07-02T14:07:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:07:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 69.35022687, 2023-07-02T14:07:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018435625000000001, 2023-07-02T14:07:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.646 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:07:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:07:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273626+0000", 2023-07-02T14:07:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046035659999999999 2023-07-02T14:07:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278230+0000", 2023-07-02T14:07:16.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9386e-05 2023-07-02T14:07:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278259+0000", 2023-07-02T14:07:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1972e-05 2023-07-02T14:07:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278271+0000", 2023-07-02T14:07:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017532680000000001 2023-07-02T14:07:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280025+0000", 2023-07-02T14:07:16.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2367000000000001e-05 2023-07-02T14:07:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280047+0000", 2023-07-02T14:07:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011987028 2023-07-02T14:07:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.292034+0000", 2023-07-02T14:07:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9266000000000001e-05 2023-07-02T14:07:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:16.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.292063+0000", 2023-07-02T14:07:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018435625000000001 2023-07-02T14:07:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:20909 2.5 e189/170 2:bb7db89c:::rbd_data.125de1449605.00000000000001a5:head v 189'2620, mlcod=189'2618)", 2023-07-02T14:07:16.654 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.368184+0000", 2023-07-02T14:07:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 51.255670793, 2023-07-02T14:07:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016727725999999998, 2023-07-02T14:07:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.368184+0000", 2023-07-02T14:07:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.368184+0000", 2023-07-02T14:07:16.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.368182+0000", 2023-07-02T14:07:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067720749999999998 2023-07-02T14:07:16.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.374954+0000", 2023-07-02T14:07:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4501000000000001e-05 2023-07-02T14:07:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:16.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.374969+0000", 2023-07-02T14:07:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4100000000000002e-06 2023-07-02T14:07:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.374976+0000", 2023-07-02T14:07:16.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.736e-05 2023-07-02T14:07:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.375034+0000", 2023-07-02T14:07:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4544000000000001e-05 2023-07-02T14:07:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.375058+0000", 2023-07-02T14:07:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0098134910000000006 2023-07-02T14:07:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.384872+0000", 2023-07-02T14:07:16.661 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9687000000000001e-05 2023-07-02T14:07:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.384911+0000", 2023-07-02T14:07:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016727725999999998 2023-07-02T14:07:16.662 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:21096 2.5 e190/170 2:a0acb434:::rbd_data.125de1449605.00000000000001bd:head v 190'2623, mlcod=189'2621)", 2023-07-02T14:07:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.705165+0000", 2023-07-02T14:07:16.663 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 50.918688957000001, 2023-07-02T14:07:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027008152000000001, 2023-07-02T14:07:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.705165+0000", 2023-07-02T14:07:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:16.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.705165+0000", 2023-07-02T14:07:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.705164+0000", 2023-07-02T14:07:16.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017192187000000001 2023-07-02T14:07:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722356+0000", 2023-07-02T14:07:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0043e-05 2023-07-02T14:07:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722387+0000", 2023-07-02T14:07:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8230000000000001e-06 2023-07-02T14:07:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722395+0000", 2023-07-02T14:07:16.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9427e-05 2023-07-02T14:07:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722445+0000", 2023-07-02T14:07:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4321999999999999e-05 2023-07-02T14:07:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722469+0000", 2023-07-02T14:07:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0096570649999999994 2023-07-02T14:07:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.732126+0000", 2023-07-02T14:07:16.670 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7490000000000001e-05 2023-07-02T14:07:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.732174+0000", 2023-07-02T14:07:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027008152000000001 2023-07-02T14:07:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.671 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22204 2.4 e197/169 2:25e15f0e:::rbd_data.125de1449605.0000000000000087:head v 197'2046, mlcod=197'2043)", 2023-07-02T14:07:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.493042+0000", 2023-07-02T14:07:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 42.130812693999999, 2023-07-02T14:07:16.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016388874000000001, 2023-07-02T14:07:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493042+0000", 2023-07-02T14:07:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:16.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493042+0000", 2023-07-02T14:07:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:16.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493041+0000", 2023-07-02T14:07:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9621999999999999e-05 2023-07-02T14:07:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493070+0000", 2023-07-02T14:07:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3439e-05 2023-07-02T14:07:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493084+0000", 2023-07-02T14:07:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6720000000000004e-06 2023-07-02T14:07:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493092+0000", 2023-07-02T14:07:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4440000000000001e-05 2023-07-02T14:07:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493136+0000", 2023-07-02T14:07:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3710000000000002e-05 2023-07-02T14:07:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493160+0000", 2023-07-02T14:07:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016220526999999998 2023-07-02T14:07:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.509380+0000", 2023-07-02T14:07:16.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0417000000000002e-05 2023-07-02T14:07:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.509431+0000", 2023-07-02T14:07:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016388874000000001 2023-07-02T14:07:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.679 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22261 2.4 e197/169 2:2a8b9a55:::rbd_data.125de1449605.0000000000000088:head v 197'2053, mlcod=197'2050)", 2023-07-02T14:07:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.616181+0000", 2023-07-02T14:07:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 42.007673545999999, 2023-07-02T14:07:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014816169000000001, 2023-07-02T14:07:16.680 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.616181+0000", 2023-07-02T14:07:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.616181+0000", 2023-07-02T14:07:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:16.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.616180+0000", 2023-07-02T14:07:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00089455100000000005 2023-07-02T14:07:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617074+0000", 2023-07-02T14:07:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9324e-05 2023-07-02T14:07:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617093+0000", 2023-07-02T14:07:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2199999999999998e-06 2023-07-02T14:07:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617103+0000", 2023-07-02T14:07:16.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013166799999999999 2023-07-02T14:07:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617234+0000", 2023-07-02T14:07:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9300000000000001e-05 2023-07-02T14:07:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617264+0000", 2023-07-02T14:07:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013684961000000001 2023-07-02T14:07:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.630949+0000", 2023-07-02T14:07:16.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8435e-05 2023-07-02T14:07:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.630997+0000", 2023-07-02T14:07:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014816169000000001 2023-07-02T14:07:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.687 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22265 2.5 e197/170 2:b033b7a2:::rbd_data.125de1449605.000000000000008a:head v 197'2737, mlcod=197'2734)", 2023-07-02T14:07:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.620697+0000", 2023-07-02T14:07:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 42.003157790000003, 2023-07-02T14:07:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016340187999999999, 2023-07-02T14:07:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:16.688 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.620697+0000", 2023-07-02T14:07:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.620697+0000", 2023-07-02T14:07:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.620695+0000", 2023-07-02T14:07:16.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029799269999999998 2023-07-02T14:07:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623675+0000", 2023-07-02T14:07:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6534000000000001e-05 2023-07-02T14:07:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623692+0000", 2023-07-02T14:07:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0400000000000004e-06 2023-07-02T14:07:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623699+0000", 2023-07-02T14:07:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3189999999999997e-05 2023-07-02T14:07:16.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623752+0000", 2023-07-02T14:07:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2078000000000001e-05 2023-07-02T14:07:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:16.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623774+0000", 2023-07-02T14:07:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013226343999999999 2023-07-02T14:07:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.637001+0000", 2023-07-02T14:07:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6279000000000003e-05 2023-07-02T14:07:16.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.637037+0000", 2023-07-02T14:07:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016340187999999999 2023-07-02T14:07:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22352 2.5 e197/170 2:ac0b3e68:::rbd_data.125de1449605.00000000000000a2:head v 197'2748, mlcod=197'2745)", 2023-07-02T14:07:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.836234+0000", 2023-07-02T14:07:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.787620392999997, 2023-07-02T14:07:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015976187999999999, 2023-07-02T14:07:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.696 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.836234+0000", 2023-07-02T14:07:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:16.697 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.836234+0000", 2023-07-02T14:07:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.836233+0000", 2023-07-02T14:07:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066101179999999999 2023-07-02T14:07:16.698 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842843+0000", 2023-07-02T14:07:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9471e-05 2023-07-02T14:07:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:16.699 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842862+0000", 2023-07-02T14:07:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3270000000000007e-06 2023-07-02T14:07:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842872+0000", 2023-07-02T14:07:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0770999999999998e-05 2023-07-02T14:07:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.700 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842933+0000", 2023-07-02T14:07:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0375999999999999e-05 2023-07-02T14:07:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842963+0000", 2023-07-02T14:07:16.701 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091962380000000007 2023-07-02T14:07:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.852159+0000", 2023-07-02T14:07:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1100999999999997e-05 2023-07-02T14:07:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.702 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.852210+0000", 2023-07-02T14:07:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015976187999999999 2023-07-02T14:07:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.703 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22416 2.5 e197/170 2:aa88fbae:::rbd_data.125de1449605.00000000000000af:head v 197'2769, mlcod=197'2766)", 2023-07-02T14:07:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.954925+0000", 2023-07-02T14:07:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.668929779999999, 2023-07-02T14:07:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017286838999999998, 2023-07-02T14:07:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:16.704 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.954925+0000", 2023-07-02T14:07:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.954925+0000", 2023-07-02T14:07:16.705 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.954924+0000", 2023-07-02T14:07:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007392755 2023-07-02T14:07:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.706 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962316+0000", 2023-07-02T14:07:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9266000000000002e-05 2023-07-02T14:07:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962336+0000", 2023-07-02T14:07:16.707 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0961000000000001e-05 2023-07-02T14:07:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962347+0000", 2023-07-02T14:07:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6159999999999998e-05 2023-07-02T14:07:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.708 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962403+0000", 2023-07-02T14:07:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7605000000000003e-05 2023-07-02T14:07:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962440+0000", 2023-07-02T14:07:16.709 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097334670000000009 2023-07-02T14:07:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.972174+0000", 2023-07-02T14:07:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7759000000000002e-05 2023-07-02T14:07:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.710 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.972211+0000", 2023-07-02T14:07:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017286838999999998 2023-07-02T14:07:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.711 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22470 2.5 e197/170 2:af9d8f16:::rbd_data.125de1449605.00000000000000b2:head v 197'2772, mlcod=197'2769)", 2023-07-02T14:07:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.062494+0000", 2023-07-02T14:07:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.561360581000002, 2023-07-02T14:07:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018691016000000001, 2023-07-02T14:07:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.712 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.062494+0000", 2023-07-02T14:07:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.062494+0000", 2023-07-02T14:07:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:16.713 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.062492+0000", 2023-07-02T14:07:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010932252999999999 2023-07-02T14:07:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.714 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073424+0000", 2023-07-02T14:07:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3642000000000001e-05 2023-07-02T14:07:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073438+0000", 2023-07-02T14:07:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6270000000000002e-06 2023-07-02T14:07:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073445+0000", 2023-07-02T14:07:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0265999999999997e-05 2023-07-02T14:07:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:16.716 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073485+0000", 2023-07-02T14:07:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5221000000000001e-05 2023-07-02T14:07:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073510+0000", 2023-07-02T14:07:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076289959999999999 2023-07-02T14:07:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.717 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.081139+0000", 2023-07-02T14:07:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5618999999999998e-05 2023-07-02T14:07:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:16.718 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.081185+0000", 2023-07-02T14:07:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018691016000000001 2023-07-02T14:07:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22516 2.5 e197/170 2:a545e9c7:::rbd_data.125de1449605.00000000000000c3:head v 197'2781, mlcod=197'2778)", 2023-07-02T14:07:16.719 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.146937+0000", 2023-07-02T14:07:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.476917260999997, 2023-07-02T14:07:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015143108000000001, 2023-07-02T14:07:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.146937+0000", 2023-07-02T14:07:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.146937+0000", 2023-07-02T14:07:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.721 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.146937+0000", 2023-07-02T14:07:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8502000000000001e-05 2023-07-02T14:07:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.147025+0000", 2023-07-02T14:07:16.722 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4660000000000005e-06 2023-07-02T14:07:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.147035+0000", 2023-07-02T14:07:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3360000000000002e-06 2023-07-02T14:07:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.723 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.147040+0000", 2023-07-02T14:07:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004086124 2023-07-02T14:07:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.151126+0000", 2023-07-02T14:07:16.724 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9261999999999999e-05 2023-07-02T14:07:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.151155+0000", 2023-07-02T14:07:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010884783 2023-07-02T14:07:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.725 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.162040+0000", 2023-07-02T14:07:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0268000000000001e-05 2023-07-02T14:07:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.162080+0000", 2023-07-02T14:07:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015143108000000001 2023-07-02T14:07:16.726 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22534 2.5 e197/170 2:b58a101b:::rbd_data.125de1449605.00000000000000c2:head v 197'2788, mlcod=197'2783)", 2023-07-02T14:07:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.173613+0000", 2023-07-02T14:07:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.450241763999998, 2023-07-02T14:07:16.727 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023211177999999999, 2023-07-02T14:07:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.173613+0000", 2023-07-02T14:07:16.728 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.173613+0000", 2023-07-02T14:07:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.729 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.173612+0000", 2023-07-02T14:07:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055730149999999997 2023-07-02T14:07:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179185+0000", 2023-07-02T14:07:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1826e-05 2023-07-02T14:07:16.730 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179197+0000", 2023-07-02T14:07:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9130000000000002e-06 2023-07-02T14:07:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.731 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179202+0000", 2023-07-02T14:07:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1668999999999997e-05 2023-07-02T14:07:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179244+0000", 2023-07-02T14:07:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6171999999999999e-05 2023-07-02T14:07:16.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179270+0000", 2023-07-02T14:07:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017512570000000002 2023-07-02T14:07:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:16.733 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.196782+0000", 2023-07-02T14:07:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1576999999999999e-05 2023-07-02T14:07:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.196824+0000", 2023-07-02T14:07:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023211177999999999 2023-07-02T14:07:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.734 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22539 2.5 e197/170 2:a6e4b3cf:::rbd_data.125de1449605.00000000000000c4:head v 197'2789, mlcod=197'2786)", 2023-07-02T14:07:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.179446+0000", 2023-07-02T14:07:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.444407959999999, 2023-07-02T14:07:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017409688999999999, 2023-07-02T14:07:16.735 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179446+0000", 2023-07-02T14:07:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.736 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179446+0000", 2023-07-02T14:07:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179446+0000", 2023-07-02T14:07:16.737 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082263240000000001 2023-07-02T14:07:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187672+0000", 2023-07-02T14:07:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7371e-05 2023-07-02T14:07:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.738 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187689+0000", 2023-07-02T14:07:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0820000000000005e-06 2023-07-02T14:07:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187698+0000", 2023-07-02T14:07:16.739 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0510999999999998e-05 2023-07-02T14:07:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187749+0000", 2023-07-02T14:07:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4805000000000001e-05 2023-07-02T14:07:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.740 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187774+0000", 2023-07-02T14:07:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.009056049 2023-07-02T14:07:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.196830+0000", 2023-07-02T14:07:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6347e-05 2023-07-02T14:07:16.741 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.196856+0000", 2023-07-02T14:07:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017409688999999999 2023-07-02T14:07:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.742 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22591 2.5 e197/170 2:bf746601:::rbd_data.125de1449605.00000000000000d6:head v 197'2799, mlcod=197'2795)", 2023-07-02T14:07:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.286442+0000", 2023-07-02T14:07:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 41.337412192000002, 2023-07-02T14:07:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016215869000000001, 2023-07-02T14:07:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.743 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.286442+0000", 2023-07-02T14:07:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.744 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.286442+0000", 2023-07-02T14:07:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.286441+0000", 2023-07-02T14:07:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00513759 2023-07-02T14:07:16.745 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291579+0000", 2023-07-02T14:07:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5048000000000001e-05 2023-07-02T14:07:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291604+0000", 2023-07-02T14:07:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1429e-05 2023-07-02T14:07:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291615+0000", 2023-07-02T14:07:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6360999999999994e-05 2023-07-02T14:07:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.747 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291712+0000", 2023-07-02T14:07:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9581999999999999e-05 2023-07-02T14:07:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291741+0000", 2023-07-02T14:07:16.748 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010882032999999999 2023-07-02T14:07:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.302623+0000", 2023-07-02T14:07:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4688999999999999e-05 2023-07-02T14:07:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.749 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.302658+0000", 2023-07-02T14:07:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016215869000000001 2023-07-02T14:07:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.750 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:23429 2.5 e206/170 2:aa2388f6:::rbd_data.125de1449605.00000000000001c4:head v 206'2898, mlcod=206'2896)", 2023-07-02T14:07:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:56.993306+0000", 2023-07-02T14:07:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.630548182999998, 2023-07-02T14:07:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017604624999999999, 2023-07-02T14:07:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:16.751 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:56.993306+0000", 2023-07-02T14:07:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:56.993306+0000", 2023-07-02T14:07:16.752 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:56.993305+0000", 2023-07-02T14:07:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068210199999999997 2023-07-02T14:07:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.753 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.000126+0000", 2023-07-02T14:07:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1634999999999999e-05 2023-07-02T14:07:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.000147+0000", 2023-07-02T14:07:16.754 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3950000000000001e-06 2023-07-02T14:07:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.000156+0000", 2023-07-02T14:07:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012743300000000001 2023-07-02T14:07:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.755 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.000283+0000", 2023-07-02T14:07:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6681000000000001e-05 2023-07-02T14:07:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.000310+0000", 2023-07-02T14:07:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010566268 2023-07-02T14:07:16.756 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.010876+0000", 2023-07-02T14:07:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4650999999999997e-05 2023-07-02T14:07:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.757 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.010911+0000", 2023-07-02T14:07:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017604624999999999 2023-07-02T14:07:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.758 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:23563 2.5 e206/170 2:a468a258:::rbd_data.125de1449605.00000000000001de:head v 206'2917, mlcod=206'2915)", 2023-07-02T14:07:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:57.318124+0000", 2023-07-02T14:07:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.305730046000001, 2023-07-02T14:07:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026542948, 2023-07-02T14:07:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.759 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318124+0000", 2023-07-02T14:07:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318124+0000", 2023-07-02T14:07:16.760 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:07:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318122+0000", 2023-07-02T14:07:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013275129 2023-07-02T14:07:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.761 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331397+0000", 2023-07-02T14:07:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.294e-05 2023-07-02T14:07:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331410+0000", 2023-07-02T14:07:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7190000000000001e-06 2023-07-02T14:07:16.762 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331418+0000", 2023-07-02T14:07:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1717000000000001e-05 2023-07-02T14:07:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:16.763 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331460+0000", 2023-07-02T14:07:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6067999999999998e-05 2023-07-02T14:07:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331486+0000", 2023-07-02T14:07:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013124998000000001 2023-07-02T14:07:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.764 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.344611+0000", 2023-07-02T14:07:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6536999999999998e-05 2023-07-02T14:07:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:16.765 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.344667+0000", 2023-07-02T14:07:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026542948 2023-07-02T14:07:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:23565 2.5 e206/170 2:bb77596d:::rbd_data.125de1449605.00000000000001df:head v 206'2919, mlcod=206'2915)", 2023-07-02T14:07:16.766 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:57.331506+0000", 2023-07-02T14:07:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 19.292348891, 2023-07-02T14:07:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016925951000000002, 2023-07-02T14:07:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:16.767 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331506+0000", 2023-07-02T14:07:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331506+0000", 2023-07-02T14:07:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.768 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331505+0000", 2023-07-02T14:07:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045442210000000002 2023-07-02T14:07:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.769 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.336049+0000", 2023-07-02T14:07:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8947e-05 2023-07-02T14:07:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.336068+0000", 2023-07-02T14:07:16.770 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9129999999999992e-06 2023-07-02T14:07:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.336077+0000", 2023-07-02T14:07:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029420649999999998 2023-07-02T14:07:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.771 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.339019+0000", 2023-07-02T14:07:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7931000000000002e-05 2023-07-02T14:07:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.339037+0000", 2023-07-02T14:07:16.772 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093659829999999996 2023-07-02T14:07:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.348403+0000", 2023-07-02T14:07:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8435000000000001e-05 2023-07-02T14:07:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.773 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.348431+0000", 2023-07-02T14:07:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016925951000000002 2023-07-02T14:07:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.774 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:23898 2.5 e207/170 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:head v 207'2973, mlcod=207'2967)", 2023-07-02T14:07:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:58.586875+0000", 2023-07-02T14:07:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 18.036979371000001, 2023-07-02T14:07:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018821676999999998, 2023-07-02T14:07:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.775 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.586875+0000", 2023-07-02T14:07:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.586875+0000", 2023-07-02T14:07:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:16.776 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.586874+0000", 2023-07-02T14:07:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004881214 2023-07-02T14:07:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.777 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.591755+0000", 2023-07-02T14:07:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1639e-05 2023-07-02T14:07:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.591767+0000", 2023-07-02T14:07:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7150000000000001e-06 2023-07-02T14:07:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.591774+0000", 2023-07-02T14:07:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000128462 2023-07-02T14:07:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:16.779 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.591902+0000", 2023-07-02T14:07:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2172999999999999e-05 2023-07-02T14:07:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.591924+0000", 2023-07-02T14:07:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01373188 2023-07-02T14:07:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.780 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.605656+0000", 2023-07-02T14:07:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0467999999999999e-05 2023-07-02T14:07:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:16.781 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.605697+0000", 2023-07-02T14:07:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018821676999999998 2023-07-02T14:07:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:23949 2.5 e207/170 2:be65b5f1:::rbd_data.125de1449605.0000000000000043:head v 207'2981, mlcod=207'2977)", 2023-07-02T14:07:16.782 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:58.698231+0000", 2023-07-02T14:07:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 17.925623795, 2023-07-02T14:07:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017072830000000001, 2023-07-02T14:07:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:16.783 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.698231+0000", 2023-07-02T14:07:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.698231+0000", 2023-07-02T14:07:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.784 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.698230+0000", 2023-07-02T14:07:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034408080000000001 2023-07-02T14:07:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.701671+0000", 2023-07-02T14:07:16.785 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1169999999999993e-06 2023-07-02T14:07:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.701680+0000", 2023-07-02T14:07:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9620000000000001e-06 2023-07-02T14:07:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.786 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.701685+0000", 2023-07-02T14:07:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7456e-05 2023-07-02T14:07:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.701742+0000", 2023-07-02T14:07:16.787 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1971e-05 2023-07-02T14:07:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.701764+0000", 2023-07-02T14:07:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013496115 2023-07-02T14:07:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.788 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.715261+0000", 2023-07-02T14:07:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2963999999999997e-05 2023-07-02T14:07:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.715303+0000", 2023-07-02T14:07:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017072830000000001 2023-07-02T14:07:16.789 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:16.790 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:16.790 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:16.791 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:16.790+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:16.792 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:07:16.874 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:16.876 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:07:16.888 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2023-07-02T14:07:16.291696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":96,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":89148,"num_read_kb":15790561,"num_write":27377,"num_write_kb":11561437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3514,"num_bytes_recovered":11979257930,"num_keys_recovered":190,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27019,"ondisk_log_size":27019,"up":18,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":23208200,"kb_used_data":22864872,"kb_used_omap":229,"kb_used_meta":343066,"kb_avail":354279160,"statfs":{"total":386547056640,"available":362781859840,"internally_reserved":0,"allocated":23413628928,"data_stored":27281001637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":351299806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21398562,"num_objects":47,"num_object_clones":9,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-346,"num_objects_misplaced":-305,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":892,"num_read_kb":590964,"num_write":164,"num_write_kb":20765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":652,"num_bytes_recovered":2275726932,"num_keys_recovered":4,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.912850"},"pg_stats":[{"pgid":"2.7","version":"216'3699","reported_seq":9410,"reported_epoch":216,"state":"active+clean","last_fresh":"2023-07-02T14:07:12.123411+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:07:12.123411+0000","last_peered":"2023-07-02T14:07:12.123411+0000","last_clean":"2023-07-02T14:07:12.123411+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:07:12.123411+0000","last_undegraded":"2023-07-02T14:07:12.123411+0000","last_fullsized":"2023-07-02T14:07:12.123411+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3699,"ondisk_log_size":3699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1575041492,"num_objects":472,"num_object_clones":397,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":58088,"num_read_kb":2022620,"num_write":3141,"num_write_kb":1530739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"216'3069","reported_seq":5174,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-07-02T14:07:14.216006+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:07:14.216006+0000","last_peered":"2023-07-02T14:07:14.216006+0000","last_clean":"2023-07-02T14:07:14.216006+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:07:14.216006+0000","last_undegraded":"2023-07-02T14:07:14.216006+0000","last_fullsized":"2023-07-02T14:07:14.216006+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3069,"ondisk_log_size":3069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1538467250,"num_objects":486,"num_object_clones":415,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":1942,"num_read_kb":2141255,"num_write":2654,"num_write_kb":1492749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"216'3204","reported_seq":4695,"reported_epoch":216,"state":"active+undersized+remapped","last_fresh":"2023-07-02T14:07:12.410373+0000","last_change":"2023-07-02T14:07:12.410555+0000","last_active":"2023-07-02T14:07:12.410373+0000","last_peered":"2023-07-02T14:07:12.410373+0000","last_clean":"2023-07-02T14:07:01.043240+0000","last_became_active":"2023-07-02T14:07:04.030791+0000","last_became_peered":"2023-07-02T14:07:04.030791+0000","last_unstale":"2023-07-02T14:07:12.410373+0000","last_undegraded":"2023-07-02T14:07:03.107328+0000","last_fullsized":"2023-07-02T14:07:03.101623+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3204,"ondisk_log_size":3204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1517142988,"num_objects":428,"num_object_clones":355,"num_object_copies":856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":635,"num_read_kb":1967916,"num_write":2847,"num_write_kb":1467896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":2369610862,"num_keys_recovered":8,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3],"avail_no_missing":["3","0","4"],"object_location_counts":[{"shards":"3","objects":428}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"216'6494","reported_seq":9005,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-07-02T14:07:14.216095+0000","last_change":"2023-07-02T14:06:29.572710+0000","last_active":"2023-07-02T14:07:14.216095+0000","last_peered":"2023-07-02T14:07:14.216095+0000","last_clean":"2023-07-02T14:07:14.216095+0000","last_became_active":"2023-07-02T14:06:29.569824+0000","last_became_peered":"2023-07-02T14:06:29.569824+0000","last_unstale":"2023-07-02T14:07:14.216095+0000","last_undegraded":"2023-07-02T14:07:14.216095+0000","last_fullsized":"2023-07-02T14:07:14.216095+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":6494,"ondisk_log_size":6494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1675535810,"num_objects":495,"num_object_clones":414,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":23025,"num_read_kb":2213857,"num_write":9605,"num_write_kb":1635009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":116,"reported_epoch":215,"state":"active+clean","last_fresh":"2023-07-02T14:07:09.372972+0000","last_change":"2023-07-02T14:07:02.103009+0000","last_active":"2023-07-02T14:07:09.372972+0000","last_peered":"2023-07-02T14:07:09.372972+0000","last_clean":"2023-07-02T14:07:09.372972+0000","last_became_active":"2023-07-02T14:07:02.102654+0000","last_became_peered":"2023-07-02T14:07:02.102654+0000","last_unstale":"2023-07-02T14:07:09.372972+0000","last_undegraded":"2023-07-02T14:07:09.372972+0000","last_fullsized":"2023-07-02T14:07:09.372972+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"216'2975","reported_seq":4499,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-07-02T14:07:14.232736+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:07:14.232736+0000","last_peered":"2023-07-02T14:07:14.232736+0000","last_clean":"2023-07-02T14:07:14.232736+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:07:14.232736+0000","last_undegraded":"2023-07-02T14:07:14.232736+0000","last_fullsized":"2023-07-02T14:07:14.232736+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2975,"ondisk_log_size":2975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1661289598,"num_objects":499,"num_object_clones":416,"num_object_copies":998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":1534,"num_read_kb":2096750,"num_write":2559,"num_write_kb":1611276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"216'2884","reported_seq":3770,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-07-02T14:07:14.233750+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:07:14.233750+0000","last_peered":"2023-07-02T14:07:14.233750+0000","last_clean":"2023-07-02T14:07:14.233750+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:07:14.233750+0000","last_undegraded":"2023-07-02T14:07:14.233750+0000","last_fullsized":"2023-07-02T14:07:14.233750+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2884,"ondisk_log_size":2884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1569543764,"num_objects":461,"num_object_clones":383,"num_object_copies":922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":966,"num_read_kb":2013799,"num_write":2501,"num_write_kb":1523310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"216'2328","reported_seq":5146,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-07-02T14:07:14.192116+0000","last_change":"2023-07-02T14:07:09.383558+0000","last_active":"2023-07-02T14:07:14.192116+0000","last_peered":"2023-07-02T14:07:14.192116+0000","last_clean":"2023-07-02T14:07:14.192116+0000","last_became_active":"2023-07-02T14:07:09.382190+0000","last_became_peered":"2023-07-02T14:07:09.382190+0000","last_unstale":"2023-07-02T14:07:14.192116+0000","last_undegraded":"2023-07-02T14:07:14.192116+0000","last_fullsized":"2023-07-02T14:07:14.192116+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2328,"ondisk_log_size":2328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1230645001,"num_objects":405,"num_object_clones":325,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":1910,"num_read_kb":1765337,"num_write":2003,"num_write_kb":1186825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":1578968698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"218'2366","reported_seq":3976,"reported_epoch":218,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-07-02T14:07:15.337452+0000","last_change":"2023-07-02T14:07:12.410957+0000","last_active":"2023-07-02T14:07:15.337452+0000","last_peered":"2023-07-02T14:07:15.337452+0000","last_clean":"2023-07-02T14:07:01.026611+0000","last_became_active":"2023-07-02T14:07:04.018152+0000","last_became_peered":"2023-07-02T14:07:04.018152+0000","last_unstale":"2023-07-02T14:07:15.337452+0000","last_undegraded":"2023-07-02T14:07:03.110294+0000","last_fullsized":"2023-07-02T14:07:03.103717+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2366,"ondisk_log_size":2366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1146537804,"num_objects":358,"num_object_clones":299,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":96,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":1048,"num_read_kb":1569027,"num_write":2067,"num_write_kb":1113633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":330,"num_bytes_recovered":1087738504,"num_keys_recovered":8,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":96},{"shards":"1","objects":254}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":96,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":89148,"num_read_kb":15790561,"num_write":27377,"num_write_kb":11561437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3514,"num_bytes_recovered":11979257930,"num_keys_recovered":190,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":27226423296,"data_stored":31620268175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":27019,"ondisk_log_size":27019,"up":16,"acting":14,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640324,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640324,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6749764,"kb_used_data":6661564,"kb_used_omap":0,"kb_used_meta":88192,"kb_avail":87622076,"statfs":{"total":96636764160,"available":89725005824,"internally_reserved":0,"allocated":6821441536,"data_stored":7847685364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90308608},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.53700000000000003}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44,"15min":0.46400000000000002},"min":{"1min":0.28399999999999997,"5min":0.215,"15min":0.215},"max":{"1min":0.71499999999999997,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.45000000000000001},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.58299999999999996}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.32800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66800000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.54900000000000004}]},{"osd":3,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.38300000000000001},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45900000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.36799999999999999,"15min":0.36299999999999999},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.48799999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.435}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.44900000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.379},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.433,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59099999999999997,"5min":1.026,"15min":1.026},"last":0.41499999999999998}]}]},{"osd":3,"up_from":133,"seq":571230650395,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4568928,"kb_used_data":4498776,"kb_used_omap":229,"kb_used_meta":69914,"kb_avail":89802912,"statfs":{"total":96636764160,"available":91958181888,"internally_reserved":0,"allocated":4606746624,"data_stored":5357897862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":71592158},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.45500000000000002}]},{"osd":1,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.372,"15min":0.36599999999999999},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.39600000000000002}]},{"osd":2,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.28699999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.44900000000000001,"15min":0.439},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.47899999999999998}]},{"osd":4,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.35799999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.36199999999999999,"15min":0.35299999999999998},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.35699999999999998}]},{"osd":5,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.39800000000000002},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.433},{"interface":"front","average":{"1min":0.439,"5min":0.39500000000000002,"15min":0.38800000000000001},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.315}]}]},{"osd":2,"up_from":11,"seq":47244640324,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3200368,"kb_used_data":3146280,"kb_used_omap":0,"kb_used_meta":54080,"kb_avail":91171472,"statfs":{"total":96636764160,"available":93359587328,"internally_reserved":0,"allocated":3221790720,"data_stored":3710342833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55377920},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":4.3650000000000002}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.42499999999999999,"15min":0.374},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":2.0720000000000001},{"interface":"front","average":{"1min":0.376,"5min":0.33500000000000002,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.66500000000000004}]},{"osd":3,"last update":"Sun Jul 2 14:07:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.91100000000000003,"5min":1.929,"15min":1.929},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.54800000000000004,"15min":0.55600000000000005},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93999999999999995,"5min":2,"15min":2},"last":0.55800000000000005}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.50900000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68400000000000005,"5min":1.071,"15min":1.071},"last":2.1230000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":0.57399999999999995}]}]},{"osd":0,"up_from":0,"seq":712964571153,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":11,"seq":47244640324,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8689140,"kb_used_data":8558252,"kb_used_omap":0,"kb_used_meta":130880,"kb_avail":85682700,"statfs":{"total":96636764160,"available":87739084800,"internally_reserved":0,"allocated":8763650048,"data_stored":10365075578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":134021120},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.318},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.26000000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.224,"5min":0.184,"15min":0.184},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.28499999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.49099999999999999,"15min":0.47299999999999998},"min":{"1min":0.30299999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.33400000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71899999999999997,"5min":1.524,"15min":1.524},"last":0.372}]},{"osd":5,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.80300000000000005,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84399999999999997,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.41899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3802202112,"data_stored":4305599675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8761016320,"data_stored":10364040347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3219156992,"data_stored":3709307602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4604137472,"data_stored":5356872530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6839910400,"data_stored":7884448021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:07:16.891 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:16.891+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:16.958 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:16.958 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:07:16.958 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:07:16.958 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:16.968 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:07:17.011 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:17.010+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:17.093 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:17.092+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:17.094 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:07:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:07:17.094 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:07:17.094 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:17.107 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:07:17.157 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:07:17.232 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:17.231+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:17.275 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:17.275 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:07:17.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:07:17.276 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:07:17.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.276 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:07:17.276 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:07:17.276 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 247.82153670100001, 2023-07-02T14:07:17.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:07:17.277 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:17.277 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:07:17.277 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:17.277 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:17.277 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:17.277 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:07:17.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.278 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:17.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.278 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:17.278 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:07:17.278 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:17.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:17.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:07:17.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:17.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:17.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:07:17.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:07:17.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:17.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:07:17.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:07:17.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:17.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:07:17.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:07:17.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:17.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:07:17.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:07:17.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:07:17.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:07:17.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:17.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:07:17.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:07:17.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:07:17.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:07:17.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.284 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:17.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:07:17.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:07:17.284 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.284 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.285 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.285 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.285 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:07:17.285 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:07:17.285 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 246.950843693, 2023-07-02T14:07:17.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:07:17.286 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:17.286 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:17.286 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:17.286 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:07:17.286 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:07:17.286 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:07:17.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.286 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:17.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:17.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:07:17.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:17.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:07:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:07:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:07:17.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:07:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:07:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:07:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:07:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:07:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:07:17.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:07:17.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:07:17.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:17.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:07:17.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:07:17.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.292 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:07:17.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:07:17.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:17.293 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:07:17.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:07:17.293 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.293 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.293 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:17.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:07:17.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:07:17.294 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.294 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.294 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.294 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:07:17.295 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:07:17.295 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 246.950718972, 2023-07-02T14:07:17.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:07:17.295 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:17.295 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:07:17.295 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:17.295 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:17.296 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:17.296 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:07:17.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.296 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:17.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:17.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:07:17.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:17.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:17.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:07:17.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:17.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:17.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:07:17.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:07:17.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:17.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:07:17.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:07:17.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:17.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:07:17.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:07:17.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:17.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:07:17.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:07:17.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.300 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.300 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:07:17.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:07:17.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:17.301 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:07:17.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:07:17.301 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:07:17.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:07:17.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:17.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:07:17.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:07:17.303 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.303 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.303 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.303 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:07:17.303 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:07:17.303 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 246.94386908800001, 2023-07-02T14:07:17.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:07:17.304 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:17.304 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:07:17.304 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:17.304 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:17.304 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:17.304 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:07:17.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.305 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:17.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:17.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:07:17.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:17.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:17.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:07:17.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:07:17.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:17.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:07:17.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:07:17.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.307 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:17.307 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:07:17.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:07:17.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:17.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:07:17.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:07:17.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:17.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:07:17.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:07:17.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:07:17.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:07:17.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:17.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:07:17.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:07:17.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:07:17.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:07:17.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:17.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:07:17.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:07:17.311 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.311 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.312 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.312 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:07:17.312 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:07:17.312 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 246.943806163, 2023-07-02T14:07:17.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:07:17.313 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:17.313 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:07:17.313 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:17.313 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:17.313 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:17.313 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:07:17.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.313 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:17.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:17.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:07:17.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:17.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:17.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:07:17.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:17.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.315 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:17.315 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:07:17.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:07:17.315 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:17.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:07:17.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:07:17.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:17.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:07:17.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:07:17.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:17.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:07:17.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:07:17.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:07:17.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:07:17.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:17.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:07:17.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:07:17.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:07:17.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:07:17.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:17.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:07:17.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:07:17.320 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.320 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.320 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.321 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:07:17.321 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:07:17.321 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 246.94350574200001, 2023-07-02T14:07:17.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:07:17.321 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:17.321 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:17.321 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:17.322 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:17.322 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:17.322 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:07:17.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.322 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:17.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:17.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:07:17.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:17.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.323 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:17.323 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:07:17.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:17.323 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:17.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:07:17.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:07:17.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:17.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:07:17.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:07:17.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:17.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:07:17.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:07:17.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:17.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:07:17.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:07:17.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:07:17.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:07:17.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:17.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:07:17.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:07:17.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:07:17.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:07:17.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:17.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:07:17.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:07:17.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:07:17.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:07:17.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:07:17.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:17.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:07:17.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:07:17.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.330 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:17.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:07:17.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:07:17.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:17.331 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:07:17.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:07:17.331 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:17.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:07:17.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:07:17.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:17.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:07:17.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:07:17.333 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.333 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.333 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.333 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:07:17.333 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:07:17.334 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 246.942981242, 2023-07-02T14:07:17.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:07:17.334 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:17.334 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:17.334 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:17.334 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:17.334 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:17.334 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:07:17.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.335 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:17.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:17.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:07:17.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:17.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:17.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:07:17.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:17.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:17.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:07:17.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:07:17.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:17.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:07:17.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:07:17.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:17.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:07:17.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:07:17.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.338 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:17.338 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:07:17.338 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:07:17.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.339 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:07:17.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:07:17.339 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.339 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.339 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:17.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:07:17.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:07:17.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:07:17.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:07:17.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:17.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:07:17.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:07:17.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:17.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:07:17.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:07:17.342 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.342 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.342 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.342 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:07:17.343 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:07:17.343 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 219.81896211700001, 2023-07-02T14:07:17.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:07:17.343 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:17.343 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:07:17.343 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:17.343 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:17.343 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:17.344 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:07:17.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.344 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:17.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:17.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:07:17.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:17.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:17.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:07:17.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:17.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:17.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:07:17.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:07:17.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:17.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:07:17.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:07:17.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:17.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:07:17.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:07:17.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:17.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:07:17.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:07:17.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:07:17.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:07:17.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:17.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:07:17.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:07:17.349 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.349 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.349 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.349 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:07:17.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:07:17.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:17.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:07:17.350 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:07:17.350 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.350 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.351 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.351 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:07:17.351 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:07:17.351 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 219.49347941299999, 2023-07-02T14:07:17.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:07:17.351 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:17.352 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:17.352 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:17.352 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:17.352 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:17.352 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:07:17.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.352 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:17.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:17.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:07:17.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:17.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:17.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:07:17.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:17.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:17.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:07:17.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:07:17.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:17.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:07:17.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:07:17.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:17.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:07:17.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:07:17.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:17.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:07:17.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:07:17.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:07:17.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:07:17.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:17.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:07:17.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:07:17.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:07:17.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:07:17.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:17.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:07:17.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:07:17.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.359 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:07:17.359 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:07:17.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:07:17.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.360 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:17.360 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:07:17.360 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:07:17.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.360 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:17.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:07:17.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:07:17.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:17.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:07:17.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:07:17.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:17.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:07:17.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:07:17.362 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.363 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.363 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.363 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:17 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 0~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:17.363 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:07:17.363 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.07520363099999, 2023-07-02T14:07:17.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997, 2023-07-02T14:07:17.364 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:17.364 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:17.364 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:17.364 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:17.364 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:17.364 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17 2023-07-02T14:07:17.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.364 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:17.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:17.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:07:17.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:17.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:17.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:07:17.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:07:17.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:17.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198431+0000", 2023-07-02T14:07:17.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2815e-05 2023-07-02T14:07:17.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:17.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198443+0000", 2023-07-02T14:07:17.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2569999999999999e-05 2023-07-02T14:07:17.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:17.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198466+0000", 2023-07-02T14:07:17.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0223e-05 2023-07-02T14:07:17.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:17.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198476+0000", 2023-07-02T14:07:17.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1792999999999997e-05 2023-07-02T14:07:17.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.369 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:07:17.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4249999999999998e-06 2023-07-02T14:07:17.369 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.369 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.369 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:17.370 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198536+0000", 2023-07-02T14:07:17.370 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91888663500000001 2023-07-02T14:07:17.370 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.370 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.371 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117423+0000", 2023-07-02T14:07:17.371 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223594 2023-07-02T14:07:17.371 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.371 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.371 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:17.372 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117647+0000", 2023-07-02T14:07:17.372 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033832900000000001 2023-07-02T14:07:17.372 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.372 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.372 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:17.373 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117985+0000", 2023-07-02T14:07:17.373 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997 2023-07-02T14:07:17.373 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.373 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.373 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.373 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.373 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.374 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:18 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 65536~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:17.374 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:07:17.374 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.07514642, 2023-07-02T14:07:17.374 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998, 2023-07-02T14:07:17.374 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:17.374 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:17.375 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:17.375 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:17.375 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:17.375 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18 2023-07-02T14:07:17.375 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.375 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:17.375 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.375 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:17.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:07:17.376 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:17.376 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.376 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.376 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:17.376 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:07:17.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:17.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.377 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:17.377 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:07:17.377 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7600000000000002e-06 2023-07-02T14:07:17.377 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.377 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:17.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198494+0000", 2023-07-02T14:07:17.378 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7840000000000004e-06 2023-07-02T14:07:17.378 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.378 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.378 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:17.378 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198502+0000", 2023-07-02T14:07:17.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6459999999999998e-06 2023-07-02T14:07:17.379 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.379 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.379 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:17.379 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198508+0000", 2023-07-02T14:07:17.379 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7395999999999997e-05 2023-07-02T14:07:17.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.380 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.380 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.380 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198565+0000", 2023-07-02T14:07:17.380 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6800000000000004e-06 2023-07-02T14:07:17.380 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.381 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:17.381 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198572+0000", 2023-07-02T14:07:17.381 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91965896800000002 2023-07-02T14:07:17.381 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.381 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118231+0000", 2023-07-02T14:07:17.382 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5212999999999998e-05 2023-07-02T14:07:17.382 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.382 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.382 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:17.382 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118326+0000", 2023-07-02T14:07:17.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000238538 2023-07-02T14:07:17.383 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.383 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.383 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:17.383 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118565+0000", 2023-07-02T14:07:17.383 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998 2023-07-02T14:07:17.383 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.384 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.384 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.384 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.384 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.384 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:19 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 131072~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:17.384 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:07:17.385 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.075122196, 2023-07-02T14:07:17.385 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002, 2023-07-02T14:07:17.385 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:17.385 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:17.385 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:17.385 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:17.385 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:17.386 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 19 2023-07-02T14:07:17.386 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.386 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:17.386 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.386 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:17.386 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:07:17.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:17.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.387 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.387 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:17.387 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:07:17.387 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:17.387 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:17.388 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:07:17.388 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1990000000000002e-06 2023-07-02T14:07:17.388 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.388 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.388 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:17.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198517+0000", 2023-07-02T14:07:17.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8789999999999997e-06 2023-07-02T14:07:17.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.389 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.389 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:17.389 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198524+0000", 2023-07-02T14:07:17.389 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:07:17.389 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:17.390 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:07:17.390 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5545999999999999e-05 2023-07-02T14:07:17.390 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.390 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.390 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:07:17.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2540000000000003e-06 2023-07-02T14:07:17.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.391 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.391 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:17.391 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198598+0000", 2023-07-02T14:07:17.391 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.919983263 2023-07-02T14:07:17.391 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.392 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118581+0000", 2023-07-02T14:07:17.392 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6878999999999994e-05 2023-07-02T14:07:17.392 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.392 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.392 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:17.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118648+0000", 2023-07-02T14:07:17.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149732 2023-07-02T14:07:17.393 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.393 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.393 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:17.393 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118798+0000", 2023-07-02T14:07:17.393 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002 2023-07-02T14:07:17.393 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.394 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.394 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.394 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.394 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.394 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:20 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 262144~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:17.394 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:07:17.394 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.075103232, 2023-07-02T14:07:17.395 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998, 2023-07-02T14:07:17.395 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:17.395 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:17.395 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:17.395 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:17.395 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:17.395 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20 2023-07-02T14:07:17.395 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.396 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:17.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.396 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:17.396 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:07:17.396 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:17.396 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.396 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:17.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:07:17.397 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:17.397 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.397 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.397 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:17.397 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:07:17.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.732e-06 2023-07-02T14:07:17.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.398 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:17.398 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198537+0000", 2023-07-02T14:07:17.398 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5259999999999999e-06 2023-07-02T14:07:17.398 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.398 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:17.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198543+0000", 2023-07-02T14:07:17.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.121e-06 2023-07-02T14:07:17.399 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.399 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.399 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:17.399 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198547+0000", 2023-07-02T14:07:17.399 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9591000000000005e-05 2023-07-02T14:07:17.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.400 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.400 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:07:17.400 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3120000000000004e-06 2023-07-02T14:07:17.400 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.400 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:17.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198621+0000", 2023-07-02T14:07:17.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92020789400000003 2023-07-02T14:07:17.401 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.401 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.401 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.401 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118829+0000", 2023-07-02T14:07:17.401 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022393386000000001 2023-07-02T14:07:17.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.402 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:17.402 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141223+0000", 2023-07-02T14:07:17.402 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000329866 2023-07-02T14:07:17.402 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.402 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.403 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:17.403 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141552+0000", 2023-07-02T14:07:17.403 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998 2023-07-02T14:07:17.403 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.403 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.403 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.403 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.403 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.404 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:21 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 196608~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:17.404 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:07:17.404 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.07507559699999, 2023-07-02T14:07:17.404 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995, 2023-07-02T14:07:17.404 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:17.404 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:17.404 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:17.405 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:17.405 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:17.405 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21 2023-07-02T14:07:17.405 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.405 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:17.405 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.405 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:17.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:07:17.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:17.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.406 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.406 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:17.406 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:07:17.406 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:17.406 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:17.407 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:07:17.407 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0169999999999999e-05 2023-07-02T14:07:17.407 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.407 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.407 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:17.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198571+0000", 2023-07-02T14:07:17.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3479999999999998e-06 2023-07-02T14:07:17.408 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.408 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.408 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:17.408 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198578+0000", 2023-07-02T14:07:17.408 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8439999999999997e-06 2023-07-02T14:07:17.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.409 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:17.409 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198583+0000", 2023-07-02T14:07:17.409 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8294999999999997e-05 2023-07-02T14:07:17.409 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.409 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198641+0000", 2023-07-02T14:07:17.410 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8120000000000001e-06 2023-07-02T14:07:17.410 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.410 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.410 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:17.410 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:07:17.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94293828499999999 2023-07-02T14:07:17.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.411 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.411 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141583+0000", 2023-07-02T14:07:17.411 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014607300000000001 2023-07-02T14:07:17.411 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.411 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:17.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141729+0000", 2023-07-02T14:07:17.412 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00044906600000000003 2023-07-02T14:07:17.412 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.412 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.412 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:17.412 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142178+0000", 2023-07-02T14:07:17.413 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995 2023-07-02T14:07:17.413 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.413 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.413 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.413 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.413 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.413 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:22 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 327680~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:17.413 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:07:17.414 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.07504216700001, 2023-07-02T14:07:17.414 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004, 2023-07-02T14:07:17.414 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:17.414 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:17.414 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:17.414 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:17.414 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:17.414 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 22 2023-07-02T14:07:17.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.415 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:17.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.415 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:17.415 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:07:17.415 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:17.415 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.415 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:17.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:07:17.416 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:17.416 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.416 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.416 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:17.416 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:07:17.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4850000000000001e-06 2023-07-02T14:07:17.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.417 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:17.417 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198597+0000", 2023-07-02T14:07:17.417 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:07:17.417 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.417 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:17.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198605+0000", 2023-07-02T14:07:17.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4839999999999998e-06 2023-07-02T14:07:17.418 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.418 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.418 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:17.418 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198609+0000", 2023-07-02T14:07:17.418 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3100999999999999e-05 2023-07-02T14:07:17.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.419 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198662+0000", 2023-07-02T14:07:17.419 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9099999999999998e-06 2023-07-02T14:07:17.419 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.419 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.419 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:17.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198666+0000", 2023-07-02T14:07:17.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.943539503 2023-07-02T14:07:17.420 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.420 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.420 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.420 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142206+0000", 2023-07-02T14:07:17.420 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012255999999999999 2023-07-02T14:07:17.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:17.421 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142328+0000", 2023-07-02T14:07:17.421 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208956 2023-07-02T14:07:17.421 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.421 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.421 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:17.422 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142537+0000", 2023-07-02T14:07:17.422 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004 2023-07-02T14:07:17.422 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.422 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.422 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.422 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.422 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.422 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:23 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 393216~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:17.423 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:07:17.423 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.075022187, 2023-07-02T14:07:17.423 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348, 2023-07-02T14:07:17.423 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:17.423 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:17.423 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:17.423 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:17.424 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:17.424 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23 2023-07-02T14:07:17.424 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.424 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:17.424 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.424 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:17.424 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:07:17.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:17.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.425 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:17.425 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:07:17.425 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:17.425 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.425 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:17.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:07:17.426 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8439999999999998e-06 2023-07-02T14:07:17.426 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.426 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.426 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:17.426 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:07:17.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0440000000000001e-06 2023-07-02T14:07:17.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.427 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:17.427 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198623+0000", 2023-07-02T14:07:17.427 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4480000000000002e-06 2023-07-02T14:07:17.427 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.427 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:17.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198628+0000", 2023-07-02T14:07:17.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5843e-05 2023-07-02T14:07:17.428 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.428 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.428 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.428 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:07:17.428 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.89e-06 2023-07-02T14:07:17.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.429 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:17.429 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198688+0000", 2023-07-02T14:07:17.429 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94389171999999999 2023-07-02T14:07:17.429 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.429 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142580+0000", 2023-07-02T14:07:17.430 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3892000000000004e-05 2023-07-02T14:07:17.430 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.430 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.430 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:17.430 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142644+0000", 2023-07-02T14:07:17.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017608000000000001 2023-07-02T14:07:17.431 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.431 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.431 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:17.431 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142820+0000", 2023-07-02T14:07:17.431 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348 2023-07-02T14:07:17.431 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.431 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.432 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.432 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.432 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.432 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:24 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 458752~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:17.432 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:07:17.432 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.074994113, 2023-07-02T14:07:17.432 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004, 2023-07-02T14:07:17.433 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:17.433 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:17.433 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:17.433 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:17.433 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:17.433 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 24 2023-07-02T14:07:17.433 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.433 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:17.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:17.434 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:07:17.434 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:17.434 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.434 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.434 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:17.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:07:17.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:17.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.435 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.435 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:17.435 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198642+0000", 2023-07-02T14:07:17.435 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.253e-06 2023-07-02T14:07:17.435 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:17.436 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:07:17.436 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1289999999999998e-06 2023-07-02T14:07:17.436 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.436 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.436 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:17.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198652+0000", 2023-07-02T14:07:17.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4510000000000002e-06 2023-07-02T14:07:17.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.437 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.437 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:17.437 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198657+0000", 2023-07-02T14:07:17.437 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9131000000000001e-05 2023-07-02T14:07:17.437 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.438 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.438 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198706+0000", 2023-07-02T14:07:17.438 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8890000000000004e-06 2023-07-02T14:07:17.438 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.438 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:17.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198710+0000", 2023-07-02T14:07:17.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94415412200000004 2023-07-02T14:07:17.439 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.439 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.439 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.439 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142864+0000", 2023-07-02T14:07:17.439 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8047000000000006e-05 2023-07-02T14:07:17.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.440 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:17.440 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142952+0000", 2023-07-02T14:07:17.440 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192967 2023-07-02T14:07:17.440 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.440 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.441 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:17.441 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143145+0000", 2023-07-02T14:07:17.441 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004 2023-07-02T14:07:17.441 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.441 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.441 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.442 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:25 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 524288~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:17.442 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:07:17.442 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.074975318, 2023-07-02T14:07:17.442 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001, 2023-07-02T14:07:17.442 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:17.442 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:17.442 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:17.443 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:17.443 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:17.443 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25 2023-07-02T14:07:17.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.443 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:17.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.443 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:17.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:07:17.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:17.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.444 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:17.444 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:07:17.444 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:17.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:17.445 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:07:17.445 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.615e-06 2023-07-02T14:07:17.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.445 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:17.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198664+0000", 2023-07-02T14:07:17.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4759999999999997e-06 2023-07-02T14:07:17.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.446 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:17.446 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198669+0000", 2023-07-02T14:07:17.446 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5999999999999998e-06 2023-07-02T14:07:17.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:17.447 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198673+0000", 2023-07-02T14:07:17.447 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4256999999999999e-05 2023-07-02T14:07:17.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.447 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198727+0000", 2023-07-02T14:07:17.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840000000000001e-06 2023-07-02T14:07:17.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.448 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:17.448 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198731+0000", 2023-07-02T14:07:17.448 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94444241900000003 2023-07-02T14:07:17.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.449 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143173+0000", 2023-07-02T14:07:17.449 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6839000000000002e-05 2023-07-02T14:07:17.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.449 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:17.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143240+0000", 2023-07-02T14:07:17.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017053399999999999 2023-07-02T14:07:17.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.450 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:17.450 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143411+0000", 2023-07-02T14:07:17.450 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001 2023-07-02T14:07:17.451 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.451 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.451 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.451 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:26 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 589824~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:17.451 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:07:17.451 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 126.07495216700001, 2023-07-02T14:07:17.452 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997, 2023-07-02T14:07:17.452 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:17.452 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:17.452 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:17.452 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:17.452 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:17.452 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 26 2023-07-02T14:07:17.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.453 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:17.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.453 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:17.453 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:07:17.453 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:17.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:17.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:07:17.454 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:17.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.454 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:17.454 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:07:17.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0520000000000002e-06 2023-07-02T14:07:17.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.455 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:17.455 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198687+0000", 2023-07-02T14:07:17.455 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1880000000000002e-06 2023-07-02T14:07:17.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:17.456 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198695+0000", 2023-07-02T14:07:17.456 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9960000000000004e-06 2023-07-02T14:07:17.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.456 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:17.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198699+0000", 2023-07-02T14:07:17.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2330999999999998e-05 2023-07-02T14:07:17.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.457 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.457 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198742+0000", 2023-07-02T14:07:17.457 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7950000000000001e-06 2023-07-02T14:07:17.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:17.458 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198745+0000", 2023-07-02T14:07:17.458 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94469398199999999 2023-07-02T14:07:17.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.458 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143439+0000", 2023-07-02T14:07:17.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6048e-05 2023-07-02T14:07:17.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.459 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.459 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:17.459 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143525+0000", 2023-07-02T14:07:17.459 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017016700000000001 2023-07-02T14:07:17.459 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.460 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.460 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:17.460 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143696+0000", 2023-07-02T14:07:17.460 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997 2023-07-02T14:07:17.460 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.460 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.460 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.461 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.461 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.461 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.6579.0:11 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected e171)", 2023-07-02T14:07:17.461 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:07:17.461 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 80.115153680999995, 2023-07-02T14:07:17.461 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998, 2023-07-02T14:07:17.461 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:17.461 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:17.462 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:17.462 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.6579", 2023-07-02T14:07:17.462 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/554403467", 2023-07-02T14:07:17.462 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11 2023-07-02T14:07:17.462 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.462 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:17.462 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.462 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:17.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:07:17.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:17.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.463 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.463 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:17.463 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:07:17.463 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:17.463 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:17.464 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158481+0000", 2023-07-02T14:07:17.464 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:07:17.464 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.464 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.464 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:17.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158487+0000", 2023-07-02T14:07:17.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7630000000000006e-06 2023-07-02T14:07:17.465 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.465 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.465 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:17.465 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158496+0000", 2023-07-02T14:07:17.465 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5100000000000001e-06 2023-07-02T14:07:17.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.466 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:17.466 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158504+0000", 2023-07-02T14:07:17.466 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9397000000000002e-05 2023-07-02T14:07:17.466 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.466 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158553+0000", 2023-07-02T14:07:17.467 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5810000000000003e-06 2023-07-02T14:07:17.467 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.467 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.467 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:17.467 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158562+0000", 2023-07-02T14:07:17.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93893079499999998 2023-07-02T14:07:17.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.468 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:17.468 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097493+0000", 2023-07-02T14:07:17.468 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015799299999999999 2023-07-02T14:07:17.468 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.468 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:17.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097651+0000", 2023-07-02T14:07:17.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187746 2023-07-02T14:07:17.469 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:17.469 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:17.469 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:17.469 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097839+0000", 2023-07-02T14:07:17.469 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998 2023-07-02T14:07:17.470 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.470 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.470 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.470 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:17.470 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:17.470 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:17.472 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:07:17.473 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:07:17.473 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:07:17.473 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:07:17.473 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:07:17.473 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:07:17.474 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:07:17.474 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:07:17.474 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:07:17.475 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:07:17.475 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:07:17.475 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:07:17.475 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:07:17.475 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:17.471+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:17.572 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:07:17.571+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:07:17.636 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:17.637 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:17.647 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:07:17.673 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:17.673+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:17.774 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:07:17.773+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:07:17.797 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 15% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 23% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 37% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 48% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 62% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 73% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 87% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 98% complete... Exporting image: 100% complete...done. 2023-07-02T14:07:17.803 INFO:tasks.workunit.client.0.smithi163.stderr:+ src_sum=772cfd861b9de6313f9663b1d0f9b297 2023-07-02T14:07:17.804 INFO:tasks.workunit.client.0.smithi163.stderr:++ rbd export b3fbb028-e267-4d25-a750-93ba18a2316a-dst@snap1 - 2023-07-02T14:07:17.804 INFO:tasks.workunit.client.0.smithi163.stderr:++ md5sum 2023-07-02T14:07:17.804 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2023-07-02T14:07:17.811 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:17.811 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:17.821 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:07:17.906 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:17.905+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:17.976 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:17.975+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:17.987 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:17.987 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:17.997 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:07:18.077 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:18.076+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:18.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:07:18.137 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:18.137 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:18.148 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:07:18.191 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:18.190+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:18.272 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:18.272 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:18.278 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:18.277+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:18.282 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:07:18.379 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:18.378+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:18.446 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:18.446 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:18.456 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:07:18.480 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:07:18.479+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:07:18.580 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:18.581 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:18.581 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:18.580+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:18.591 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:07:18.682 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:18.681+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:18.755 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:18.756 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:18.766 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:07:18.782 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:18.782+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:18.883 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:18.882+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:18.930 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:18.930 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:18.940 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:07:18.983 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:18.983+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:19.084 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:19.083+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:19.104 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:19.104 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:19.115 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:07:19.120 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:07:19.197 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:19.197+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:19.279 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:19.279 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:19.285 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:19.284+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:19.289 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:07:19.386 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:19.385+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:19.452 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:19.453 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:19.462 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:07:19.476 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-07-02T14:07:19.504 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:19.503+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:19.587 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:19.587+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:19.591 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:19.591 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:19.602 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:07:19.648 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:07:19.666 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:07:19.683 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:07:19.688 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:19.688+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:19.701 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:07:19.720 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:07:19.731 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:19.731 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:07:19.731 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:07:19.731 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:07:19.731 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:19.737 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:07:19.739 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-07-02T14:07:19.739 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-07-02T14:07:19.739 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-07-02T14:07:19.739 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-07-02T14:07:19.739 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-07-02T14:07:19.740 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-07-02T14:07:19.742 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:07:19.788 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:19.788+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:19.867 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:19.868 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:07:19.868 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:07:19.868 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:07:19.868 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.868 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18643 2.5 e176/170 2:a9ace6f8:::rbd_data.125de1449605.0000000000000187:head v 176'2314, mlcod=173'2311)", 2023-07-02T14:07:19.869 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:07:19.869 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 74.439719613999998, 2023-07-02T14:07:19.869 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015027297, 2023-07-02T14:07:19.869 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:19.869 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:19.869 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:19.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.869 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:19.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:07:19.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:19.870 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.870 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.870 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:19.870 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426549+0000", 2023-07-02T14:07:19.870 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:19.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:19.871 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.426547+0000", 2023-07-02T14:07:19.871 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0056864430000000002 2023-07-02T14:07:19.871 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.871 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.871 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:19.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432234+0000", 2023-07-02T14:07:19.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.415e-05 2023-07-02T14:07:19.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.872 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.872 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:19.872 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432248+0000", 2023-07-02T14:07:19.872 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7490000000000001e-06 2023-07-02T14:07:19.872 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:19.873 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432255+0000", 2023-07-02T14:07:19.873 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3915000000000002e-05 2023-07-02T14:07:19.873 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.873 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.873 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:19.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432299+0000", 2023-07-02T14:07:19.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1214000000000001e-05 2023-07-02T14:07:19.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.874 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.874 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:19.874 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.432320+0000", 2023-07-02T14:07:19.874 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0092114950000000001 2023-07-02T14:07:19.874 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:19.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.441531+0000", 2023-07-02T14:07:19.875 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4422999999999998e-05 2023-07-02T14:07:19.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.875 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.875 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:19.875 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:05.441576+0000", 2023-07-02T14:07:19.876 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015027297 2023-07-02T14:07:19.876 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.876 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:19.876 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.876 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.876 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.876 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:18881 2.5 e177/170 2:b1731d2f:::rbd_data.125de1449605.0000000000000024:head v 177'2350, mlcod=177'2347)", 2023-07-02T14:07:19.877 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:07:19.877 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 72.960353789999999, 2023-07-02T14:07:19.877 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020820709999999999, 2023-07-02T14:07:19.877 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:19.877 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:19.877 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:19.877 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.877 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:19.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:07:19.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:19.878 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.878 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.878 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:19.878 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905914+0000", 2023-07-02T14:07:19.878 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:19.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:19.879 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.905913+0000", 2023-07-02T14:07:19.879 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0080191519999999999 2023-07-02T14:07:19.879 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.879 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.879 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:19.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913932+0000", 2023-07-02T14:07:19.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3757999999999999e-05 2023-07-02T14:07:19.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.880 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.880 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:19.880 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913956+0000", 2023-07-02T14:07:19.880 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4009999999999997e-06 2023-07-02T14:07:19.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:19.881 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.913965+0000", 2023-07-02T14:07:19.881 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3383e-05 2023-07-02T14:07:19.881 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.881 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.881 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:19.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914018+0000", 2023-07-02T14:07:19.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7674e-05 2023-07-02T14:07:19.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.882 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.882 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:19.882 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.914046+0000", 2023-07-02T14:07:19.882 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012656555999999999 2023-07-02T14:07:19.882 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:19.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.926702+0000", 2023-07-02T14:07:19.883 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.2891000000000001e-05 2023-07-02T14:07:19.883 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.883 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.883 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:19.883 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:06.926735+0000", 2023-07-02T14:07:19.884 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.020820709999999999 2023-07-02T14:07:19.884 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.884 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:19.884 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.884 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.884 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19073 2.5 e177/170 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:head v 177'2378, mlcod=177'2376)", 2023-07-02T14:07:19.884 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:07:19.885 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 72.592640607000007, 2023-07-02T14:07:19.885 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018435625000000001, 2023-07-02T14:07:19.885 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:19.885 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:19.885 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:19.885 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.885 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:19.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:07:19.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:19.886 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.886 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.886 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:19.886 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273628+0000", 2023-07-02T14:07:19.886 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:19.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:19.887 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.273626+0000", 2023-07-02T14:07:19.887 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0046035659999999999 2023-07-02T14:07:19.887 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.887 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:19.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278230+0000", 2023-07-02T14:07:19.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9386e-05 2023-07-02T14:07:19.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.888 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:19.888 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278259+0000", 2023-07-02T14:07:19.888 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1972e-05 2023-07-02T14:07:19.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:19.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.278271+0000", 2023-07-02T14:07:19.889 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0017532680000000001 2023-07-02T14:07:19.889 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.889 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.889 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:19.889 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280025+0000", 2023-07-02T14:07:19.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2367000000000001e-05 2023-07-02T14:07:19.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.890 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.890 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:19.890 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.280047+0000", 2023-07-02T14:07:19.890 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011987028 2023-07-02T14:07:19.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:19.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.292034+0000", 2023-07-02T14:07:19.891 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9266000000000001e-05 2023-07-02T14:07:19.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.891 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:19.891 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.292063+0000", 2023-07-02T14:07:19.892 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018435625000000001 2023-07-02T14:07:19.892 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.892 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:19.892 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.892 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.892 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:20909 2.5 e189/170 2:bb7db89c:::rbd_data.125de1449605.00000000000001a5:head v 189'2620, mlcod=189'2618)", 2023-07-02T14:07:19.892 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.368184+0000", 2023-07-02T14:07:19.893 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 54.49808453, 2023-07-02T14:07:19.893 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016727725999999998, 2023-07-02T14:07:19.893 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:19.893 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:19.893 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:19.893 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.893 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:19.893 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.368184+0000", 2023-07-02T14:07:19.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:19.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.894 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:19.894 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.368184+0000", 2023-07-02T14:07:19.894 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:19.894 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:19.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.368182+0000", 2023-07-02T14:07:19.895 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067720749999999998 2023-07-02T14:07:19.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.895 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:19.895 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.374954+0000", 2023-07-02T14:07:19.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4501000000000001e-05 2023-07-02T14:07:19.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.896 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.896 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:19.896 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.374969+0000", 2023-07-02T14:07:19.896 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.4100000000000002e-06 2023-07-02T14:07:19.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:19.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.374976+0000", 2023-07-02T14:07:19.897 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.736e-05 2023-07-02T14:07:19.897 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.897 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:19.897 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.375034+0000", 2023-07-02T14:07:19.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4544000000000001e-05 2023-07-02T14:07:19.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.898 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:19.898 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.375058+0000", 2023-07-02T14:07:19.898 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0098134910000000006 2023-07-02T14:07:19.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:19.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.384872+0000", 2023-07-02T14:07:19.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9687000000000001e-05 2023-07-02T14:07:19.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.899 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.899 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:19.899 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.384911+0000", 2023-07-02T14:07:19.899 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016727725999999998 2023-07-02T14:07:19.900 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.900 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:19.900 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.900 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.900 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.900 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:21096 2.5 e190/170 2:a0acb434:::rbd_data.125de1449605.00000000000001bd:head v 190'2623, mlcod=189'2621)", 2023-07-02T14:07:19.900 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:25.705165+0000", 2023-07-02T14:07:19.901 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 54.161102694, 2023-07-02T14:07:19.901 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027008152000000001, 2023-07-02T14:07:19.901 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:19.901 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:19.901 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:19.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.901 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:19.901 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.705165+0000", 2023-07-02T14:07:19.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:19.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.902 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:19.902 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.705165+0000", 2023-07-02T14:07:19.902 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:19.902 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.902 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:19.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.705164+0000", 2023-07-02T14:07:19.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017192187000000001 2023-07-02T14:07:19.903 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.903 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.903 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:19.903 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722356+0000", 2023-07-02T14:07:19.903 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0043e-05 2023-07-02T14:07:19.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.904 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:19.904 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722387+0000", 2023-07-02T14:07:19.904 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8230000000000001e-06 2023-07-02T14:07:19.904 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:19.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722395+0000", 2023-07-02T14:07:19.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9427e-05 2023-07-02T14:07:19.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.905 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:19.905 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722445+0000", 2023-07-02T14:07:19.905 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4321999999999999e-05 2023-07-02T14:07:19.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:19.906 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.722469+0000", 2023-07-02T14:07:19.906 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0096570649999999994 2023-07-02T14:07:19.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.906 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.906 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:19.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.732126+0000", 2023-07-02T14:07:19.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7490000000000001e-05 2023-07-02T14:07:19.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.907 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:19.907 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:25.732174+0000", 2023-07-02T14:07:19.907 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027008152000000001 2023-07-02T14:07:19.908 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.908 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:19.908 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.908 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.908 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22204 2.4 e197/169 2:25e15f0e:::rbd_data.125de1449605.0000000000000087:head v 197'2046, mlcod=197'2043)", 2023-07-02T14:07:19.908 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.493042+0000", 2023-07-02T14:07:19.908 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.373226430999999, 2023-07-02T14:07:19.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016388874000000001, 2023-07-02T14:07:19.909 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:19.909 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:19.909 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:19.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.909 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:19.909 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493042+0000", 2023-07-02T14:07:19.909 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:19.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:19.910 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493042+0000", 2023-07-02T14:07:19.910 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:19.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.910 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:19.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493041+0000", 2023-07-02T14:07:19.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9621999999999999e-05 2023-07-02T14:07:19.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.911 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.911 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:19.911 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493070+0000", 2023-07-02T14:07:19.911 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3439e-05 2023-07-02T14:07:19.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:19.912 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493084+0000", 2023-07-02T14:07:19.912 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.6720000000000004e-06 2023-07-02T14:07:19.912 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.912 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:19.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493092+0000", 2023-07-02T14:07:19.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4440000000000001e-05 2023-07-02T14:07:19.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.913 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:19.913 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493136+0000", 2023-07-02T14:07:19.913 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.3710000000000002e-05 2023-07-02T14:07:19.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:19.914 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.493160+0000", 2023-07-02T14:07:19.914 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016220526999999998 2023-07-02T14:07:19.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.914 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.914 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:19.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.509380+0000", 2023-07-02T14:07:19.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0417000000000002e-05 2023-07-02T14:07:19.915 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.915 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.915 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:19.915 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.509431+0000", 2023-07-02T14:07:19.915 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016388874000000001 2023-07-02T14:07:19.916 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.916 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:19.916 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.916 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22261 2.4 e197/169 2:2a8b9a55:::rbd_data.125de1449605.0000000000000088:head v 197'2053, mlcod=197'2050)", 2023-07-02T14:07:19.916 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.616181+0000", 2023-07-02T14:07:19.916 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.250087282999999, 2023-07-02T14:07:19.917 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014816169000000001, 2023-07-02T14:07:19.917 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:19.917 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:19.917 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:19.917 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.917 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:19.917 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.616181+0000", 2023-07-02T14:07:19.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:19.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.918 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:19.918 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.616181+0000", 2023-07-02T14:07:19.918 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:19.918 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.918 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:19.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.616180+0000", 2023-07-02T14:07:19.919 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00089455100000000005 2023-07-02T14:07:19.919 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.919 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:19.919 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617074+0000", 2023-07-02T14:07:19.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9324e-05 2023-07-02T14:07:19.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.920 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:19.920 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617093+0000", 2023-07-02T14:07:19.920 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2199999999999998e-06 2023-07-02T14:07:19.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.921 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.921 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:19.921 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617103+0000", 2023-07-02T14:07:19.921 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013166799999999999 2023-07-02T14:07:19.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.922 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:19.923 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617234+0000", 2023-07-02T14:07:19.923 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9300000000000001e-05 2023-07-02T14:07:19.923 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.924 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:19.924 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.617264+0000", 2023-07-02T14:07:19.924 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013684961000000001 2023-07-02T14:07:19.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.925 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:19.925 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.630949+0000", 2023-07-02T14:07:19.926 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8435e-05 2023-07-02T14:07:19.926 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.926 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.927 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:19.927 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.630997+0000", 2023-07-02T14:07:19.927 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014816169000000001 2023-07-02T14:07:19.927 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.928 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:19.928 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.929 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.929 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22265 2.5 e197/170 2:b033b7a2:::rbd_data.125de1449605.000000000000008a:head v 197'2737, mlcod=197'2734)", 2023-07-02T14:07:19.929 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.620697+0000", 2023-07-02T14:07:19.929 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.245571527000003, 2023-07-02T14:07:19.929 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016340187999999999, 2023-07-02T14:07:19.930 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:19.930 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:19.930 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:19.930 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.931 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:19.931 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.620697+0000", 2023-07-02T14:07:19.931 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:19.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.932 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.932 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:19.932 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.620697+0000", 2023-07-02T14:07:19.932 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:19.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:19.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.620695+0000", 2023-07-02T14:07:19.933 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029799269999999998 2023-07-02T14:07:19.933 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.933 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.933 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:19.933 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623675+0000", 2023-07-02T14:07:19.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6534000000000001e-05 2023-07-02T14:07:19.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.934 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:19.934 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623692+0000", 2023-07-02T14:07:19.934 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0400000000000004e-06 2023-07-02T14:07:19.934 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:19.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623699+0000", 2023-07-02T14:07:19.935 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3189999999999997e-05 2023-07-02T14:07:19.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.935 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:19.935 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623752+0000", 2023-07-02T14:07:19.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2078000000000001e-05 2023-07-02T14:07:19.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.936 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.936 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:19.936 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.623774+0000", 2023-07-02T14:07:19.936 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013226343999999999 2023-07-02T14:07:19.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:19.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.637001+0000", 2023-07-02T14:07:19.937 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6279000000000003e-05 2023-07-02T14:07:19.937 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.937 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:19.937 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.637037+0000", 2023-07-02T14:07:19.938 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016340187999999999 2023-07-02T14:07:19.938 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.938 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:19.938 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.938 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22352 2.5 e197/170 2:ac0b3e68:::rbd_data.125de1449605.00000000000000a2:head v 197'2748, mlcod=197'2745)", 2023-07-02T14:07:19.938 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.836234+0000", 2023-07-02T14:07:19.939 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 45.030034129999997, 2023-07-02T14:07:19.939 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015976187999999999, 2023-07-02T14:07:19.939 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:19.939 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:19.939 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:19.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.939 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:19.939 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.836234+0000", 2023-07-02T14:07:19.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:19.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.940 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:19.940 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.836234+0000", 2023-07-02T14:07:19.940 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:19.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:19.941 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.836233+0000", 2023-07-02T14:07:19.941 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0066101179999999999 2023-07-02T14:07:19.941 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.941 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.941 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:19.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842843+0000", 2023-07-02T14:07:19.942 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9471e-05 2023-07-02T14:07:19.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.942 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:19.942 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842862+0000", 2023-07-02T14:07:19.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3270000000000007e-06 2023-07-02T14:07:19.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.943 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:19.943 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842872+0000", 2023-07-02T14:07:19.943 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0770999999999998e-05 2023-07-02T14:07:19.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:19.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842933+0000", 2023-07-02T14:07:19.944 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0375999999999999e-05 2023-07-02T14:07:19.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.944 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.944 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:19.944 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.842963+0000", 2023-07-02T14:07:19.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0091962380000000007 2023-07-02T14:07:19.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.945 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:19.945 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.852159+0000", 2023-07-02T14:07:19.945 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1100999999999997e-05 2023-07-02T14:07:19.945 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.945 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.946 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:19.946 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.852210+0000", 2023-07-02T14:07:19.946 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015976187999999999 2023-07-02T14:07:19.946 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.946 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:19.946 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.947 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22416 2.5 e197/170 2:aa88fbae:::rbd_data.125de1449605.00000000000000af:head v 197'2769, mlcod=197'2766)", 2023-07-02T14:07:19.947 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:34.954925+0000", 2023-07-02T14:07:19.947 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.911343516999999, 2023-07-02T14:07:19.947 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017286838999999998, 2023-07-02T14:07:19.947 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:19.947 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:19.947 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:19.947 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:19.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.954925+0000", 2023-07-02T14:07:19.948 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:19.948 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.948 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.948 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:19.948 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.954925+0000", 2023-07-02T14:07:19.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:19.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.949 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:19.949 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.954924+0000", 2023-07-02T14:07:19.949 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007392755 2023-07-02T14:07:19.949 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:19.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962316+0000", 2023-07-02T14:07:19.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9266000000000002e-05 2023-07-02T14:07:19.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.950 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:19.950 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962336+0000", 2023-07-02T14:07:19.950 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0961000000000001e-05 2023-07-02T14:07:19.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.951 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:19.951 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962347+0000", 2023-07-02T14:07:19.951 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6159999999999998e-05 2023-07-02T14:07:19.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.951 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:19.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962403+0000", 2023-07-02T14:07:19.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7605000000000003e-05 2023-07-02T14:07:19.952 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.952 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:19.952 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.962440+0000", 2023-07-02T14:07:19.952 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0097334670000000009 2023-07-02T14:07:19.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.953 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:19.953 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.972174+0000", 2023-07-02T14:07:19.953 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7759000000000002e-05 2023-07-02T14:07:19.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.954 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:19.954 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:34.972211+0000", 2023-07-02T14:07:19.954 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017286838999999998 2023-07-02T14:07:19.954 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.954 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:19.954 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.954 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.955 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22470 2.5 e197/170 2:af9d8f16:::rbd_data.125de1449605.00000000000000b2:head v 197'2772, mlcod=197'2769)", 2023-07-02T14:07:19.955 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.062494+0000", 2023-07-02T14:07:19.955 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.803774318000002, 2023-07-02T14:07:19.955 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018691016000000001, 2023-07-02T14:07:19.955 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:19.955 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:19.955 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:19.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:19.956 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.062494+0000", 2023-07-02T14:07:19.956 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:19.956 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.956 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.956 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:19.957 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.062494+0000", 2023-07-02T14:07:19.957 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:19.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.958 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:19.958 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.062492+0000", 2023-07-02T14:07:19.958 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010932252999999999 2023-07-02T14:07:19.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.959 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.959 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:19.959 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073424+0000", 2023-07-02T14:07:19.959 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3642000000000001e-05 2023-07-02T14:07:19.960 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.960 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.960 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:19.961 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073438+0000", 2023-07-02T14:07:19.961 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6270000000000002e-06 2023-07-02T14:07:19.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.962 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:19.962 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073445+0000", 2023-07-02T14:07:19.962 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0265999999999997e-05 2023-07-02T14:07:19.962 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.963 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.963 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:19.963 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073485+0000", 2023-07-02T14:07:19.963 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5221000000000001e-05 2023-07-02T14:07:19.964 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.964 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:19.964 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.073510+0000", 2023-07-02T14:07:19.965 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0076289959999999999 2023-07-02T14:07:19.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.965 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:19.966 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.081139+0000", 2023-07-02T14:07:19.966 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5618999999999998e-05 2023-07-02T14:07:19.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.966 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.967 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:19.967 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.081185+0000", 2023-07-02T14:07:19.967 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018691016000000001 2023-07-02T14:07:19.967 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.968 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:19.968 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22516 2.5 e197/170 2:a545e9c7:::rbd_data.125de1449605.00000000000000c3:head v 197'2781, mlcod=197'2778)", 2023-07-02T14:07:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.146937+0000", 2023-07-02T14:07:19.969 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.719330997999997, 2023-07-02T14:07:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015143108000000001, 2023-07-02T14:07:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:19.970 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.146937+0000", 2023-07-02T14:07:19.971 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.972 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.146937+0000", 2023-07-02T14:07:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.973 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:19.974 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.146937+0000", 2023-07-02T14:07:19.974 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8502000000000001e-05 2023-07-02T14:07:19.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.974 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.975 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:19.975 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.147025+0000", 2023-07-02T14:07:19.975 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4660000000000005e-06 2023-07-02T14:07:19.975 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.976 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.976 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:19.976 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.147035+0000", 2023-07-02T14:07:19.977 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3360000000000002e-06 2023-07-02T14:07:19.977 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.977 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:19.978 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.147040+0000", 2023-07-02T14:07:19.978 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004086124 2023-07-02T14:07:19.978 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.978 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.979 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:19.979 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.151126+0000", 2023-07-02T14:07:19.979 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9261999999999999e-05 2023-07-02T14:07:19.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.980 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.980 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:19.980 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.151155+0000", 2023-07-02T14:07:19.981 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010884783 2023-07-02T14:07:19.981 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.981 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:19.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.162040+0000", 2023-07-02T14:07:19.982 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0268000000000001e-05 2023-07-02T14:07:19.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.982 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.982 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:19.982 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.162080+0000", 2023-07-02T14:07:19.983 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015143108000000001 2023-07-02T14:07:19.983 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.983 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:19.983 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.983 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.983 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22534 2.5 e197/170 2:b58a101b:::rbd_data.125de1449605.00000000000000c2:head v 197'2788, mlcod=197'2783)", 2023-07-02T14:07:19.984 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.173613+0000", 2023-07-02T14:07:19.984 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.692655500999997, 2023-07-02T14:07:19.984 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023211177999999999, 2023-07-02T14:07:19.984 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:19.984 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:19.984 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:19.984 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.984 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:19.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.173613+0000", 2023-07-02T14:07:19.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:19.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.985 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:19.985 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.173613+0000", 2023-07-02T14:07:19.985 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:19.985 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:19.986 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.173612+0000", 2023-07-02T14:07:19.986 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055730149999999997 2023-07-02T14:07:19.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.986 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:19.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179185+0000", 2023-07-02T14:07:19.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1826e-05 2023-07-02T14:07:19.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.987 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.987 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:19.987 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179197+0000", 2023-07-02T14:07:19.987 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9130000000000002e-06 2023-07-02T14:07:19.987 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:19.988 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179202+0000", 2023-07-02T14:07:19.988 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1668999999999997e-05 2023-07-02T14:07:19.988 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.988 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.988 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:19.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179244+0000", 2023-07-02T14:07:19.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6171999999999999e-05 2023-07-02T14:07:19.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.989 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:19.989 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179270+0000", 2023-07-02T14:07:19.989 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017512570000000002 2023-07-02T14:07:19.989 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:19.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.196782+0000", 2023-07-02T14:07:19.990 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1576999999999999e-05 2023-07-02T14:07:19.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.990 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:19.990 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.196824+0000", 2023-07-02T14:07:19.991 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023211177999999999 2023-07-02T14:07:19.991 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.991 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:19.991 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.991 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22539 2.5 e197/170 2:a6e4b3cf:::rbd_data.125de1449605.00000000000000c4:head v 197'2789, mlcod=197'2786)", 2023-07-02T14:07:19.991 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.179446+0000", 2023-07-02T14:07:19.992 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.686821696999999, 2023-07-02T14:07:19.992 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017409688999999999, 2023-07-02T14:07:19.992 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:19.992 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:19.992 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:19.992 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.992 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:19.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179446+0000", 2023-07-02T14:07:19.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:19.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.993 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:19.993 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179446+0000", 2023-07-02T14:07:19.993 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:19.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:19.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.179446+0000", 2023-07-02T14:07:19.994 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082263240000000001 2023-07-02T14:07:19.994 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.994 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.994 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:19.994 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187672+0000", 2023-07-02T14:07:19.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7371e-05 2023-07-02T14:07:19.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.995 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:19.995 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187689+0000", 2023-07-02T14:07:19.995 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.0820000000000005e-06 2023-07-02T14:07:19.995 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.995 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:19.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187698+0000", 2023-07-02T14:07:19.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0510999999999998e-05 2023-07-02T14:07:19.996 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.996 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.996 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:19.996 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187749+0000", 2023-07-02T14:07:19.996 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4805000000000001e-05 2023-07-02T14:07:19.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:19.997 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.187774+0000", 2023-07-02T14:07:19.997 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.009056049 2023-07-02T14:07:19.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.997 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.997 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:19.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.196830+0000", 2023-07-02T14:07:19.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6347e-05 2023-07-02T14:07:19.998 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.998 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.998 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:19.998 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.196856+0000", 2023-07-02T14:07:19.998 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017409688999999999 2023-07-02T14:07:19.998 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.999 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:19.999 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:19.999 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:19.999 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:19.999 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:22591 2.5 e197/170 2:bf746601:::rbd_data.125de1449605.00000000000000d6:head v 197'2799, mlcod=197'2795)", 2023-07-02T14:07:19.999 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:35.286442+0000", 2023-07-02T14:07:19.999 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 44.579825929000002, 2023-07-02T14:07:20.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016215869000000001, 2023-07-02T14:07:20.000 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.000 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:20.000 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.000 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.000 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.000 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.286442+0000", 2023-07-02T14:07:20.000 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.001 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.286442+0000", 2023-07-02T14:07:20.001 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:20.001 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.001 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.001 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.286441+0000", 2023-07-02T14:07:20.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00513759 2023-07-02T14:07:20.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.002 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.002 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.002 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291579+0000", 2023-07-02T14:07:20.002 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5048000000000001e-05 2023-07-02T14:07:20.002 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291604+0000", 2023-07-02T14:07:20.003 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1429e-05 2023-07-02T14:07:20.003 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.003 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.003 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.003 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291615+0000", 2023-07-02T14:07:20.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6360999999999994e-05 2023-07-02T14:07:20.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.004 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.004 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.004 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291712+0000", 2023-07-02T14:07:20.004 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.9581999999999999e-05 2023-07-02T14:07:20.004 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.291741+0000", 2023-07-02T14:07:20.005 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010882032999999999 2023-07-02T14:07:20.005 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.005 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.005 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:20.005 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.302623+0000", 2023-07-02T14:07:20.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4688999999999999e-05 2023-07-02T14:07:20.006 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.006 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.006 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.006 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:35.302658+0000", 2023-07-02T14:07:20.006 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016215869000000001 2023-07-02T14:07:20.006 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.006 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.007 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.007 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.007 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.007 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:23429 2.5 e206/170 2:aa2388f6:::rbd_data.125de1449605.00000000000001c4:head v 206'2898, mlcod=206'2896)", 2023-07-02T14:07:20.007 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:56.993306+0000", 2023-07-02T14:07:20.007 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.872961920000002, 2023-07-02T14:07:20.007 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017604624999999999, 2023-07-02T14:07:20.007 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.008 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:20.008 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.008 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.008 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:56.993306+0000", 2023-07-02T14:07:20.008 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.008 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.008 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:56.993306+0000", 2023-07-02T14:07:20.009 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:20.009 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.009 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.009 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.009 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:56.993305+0000", 2023-07-02T14:07:20.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0068210199999999997 2023-07-02T14:07:20.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.010 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.010 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.000126+0000", 2023-07-02T14:07:20.010 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1634999999999999e-05 2023-07-02T14:07:20.010 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.010 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.000147+0000", 2023-07-02T14:07:20.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3950000000000001e-06 2023-07-02T14:07:20.011 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.011 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.011 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.011 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.000156+0000", 2023-07-02T14:07:20.011 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012743300000000001 2023-07-02T14:07:20.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.012 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.000283+0000", 2023-07-02T14:07:20.012 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6681000000000001e-05 2023-07-02T14:07:20.012 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.012 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.012 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.000310+0000", 2023-07-02T14:07:20.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010566268 2023-07-02T14:07:20.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.013 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.013 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:20.013 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.010876+0000", 2023-07-02T14:07:20.013 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4650999999999997e-05 2023-07-02T14:07:20.013 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.014 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.014 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.014 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.010911+0000", 2023-07-02T14:07:20.014 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017604624999999999 2023-07-02T14:07:20.014 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.014 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.014 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.015 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.015 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.015 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:23563 2.5 e206/170 2:a468a258:::rbd_data.125de1449605.00000000000001de:head v 206'2917, mlcod=206'2915)", 2023-07-02T14:07:20.015 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:57.318124+0000", 2023-07-02T14:07:20.015 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.548143783, 2023-07-02T14:07:20.015 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026542948, 2023-07-02T14:07:20.015 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.015 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:20.016 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.016 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318124+0000", 2023-07-02T14:07:20.016 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.016 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.016 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.016 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318124+0000", 2023-07-02T14:07:20.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:07:20.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.017 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.017 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.017 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318122+0000", 2023-07-02T14:07:20.017 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013275129 2023-07-02T14:07:20.017 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331397+0000", 2023-07-02T14:07:20.018 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.294e-05 2023-07-02T14:07:20.018 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.018 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.018 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.018 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331410+0000", 2023-07-02T14:07:20.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7190000000000001e-06 2023-07-02T14:07:20.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.019 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.019 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331418+0000", 2023-07-02T14:07:20.019 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1717000000000001e-05 2023-07-02T14:07:20.019 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.019 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331460+0000", 2023-07-02T14:07:20.020 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6067999999999998e-05 2023-07-02T14:07:20.020 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.020 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.020 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.020 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331486+0000", 2023-07-02T14:07:20.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013124998000000001 2023-07-02T14:07:20.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.021 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:20.021 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.344611+0000", 2023-07-02T14:07:20.021 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6536999999999998e-05 2023-07-02T14:07:20.021 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.021 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.022 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.022 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.344667+0000", 2023-07-02T14:07:20.022 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.026542948 2023-07-02T14:07:20.022 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.022 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.022 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.022 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.022 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.023 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:23565 2.5 e206/170 2:bb77596d:::rbd_data.125de1449605.00000000000001df:head v 206'2919, mlcod=206'2915)", 2023-07-02T14:07:20.023 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:57.331506+0000", 2023-07-02T14:07:20.023 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 22.534762627999999, 2023-07-02T14:07:20.023 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016925951000000002, 2023-07-02T14:07:20.023 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.023 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:20.023 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.023 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331506+0000", 2023-07-02T14:07:20.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.024 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.024 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.024 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.024 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331506+0000", 2023-07-02T14:07:20.024 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:20.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.025 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.025 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.331505+0000", 2023-07-02T14:07:20.025 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0045442210000000002 2023-07-02T14:07:20.025 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.025 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.336049+0000", 2023-07-02T14:07:20.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8947e-05 2023-07-02T14:07:20.026 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.026 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.026 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.026 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.336068+0000", 2023-07-02T14:07:20.026 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.9129999999999992e-06 2023-07-02T14:07:20.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.027 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.336077+0000", 2023-07-02T14:07:20.027 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029420649999999998 2023-07-02T14:07:20.027 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.027 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.027 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.339019+0000", 2023-07-02T14:07:20.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.7931000000000002e-05 2023-07-02T14:07:20.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.028 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.028 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.028 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.339037+0000", 2023-07-02T14:07:20.028 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0093659829999999996 2023-07-02T14:07:20.028 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:20.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.348403+0000", 2023-07-02T14:07:20.029 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8435000000000001e-05 2023-07-02T14:07:20.029 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.029 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.029 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.029 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.348431+0000", 2023-07-02T14:07:20.030 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016925951000000002 2023-07-02T14:07:20.030 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.030 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.030 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.030 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.030 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.030 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:23898 2.5 e207/170 2:b81f02a4:::rbd_data.125de1449605.000000000000003a:head v 207'2973, mlcod=207'2967)", 2023-07-02T14:07:20.030 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:58.586875+0000", 2023-07-02T14:07:20.031 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.279393108000001, 2023-07-02T14:07:20.031 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018821676999999998, 2023-07-02T14:07:20.031 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.031 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:20.031 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.031 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.031 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.031 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.586875+0000", 2023-07-02T14:07:20.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.032 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.032 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.032 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.586875+0000", 2023-07-02T14:07:20.032 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:20.032 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.586874+0000", 2023-07-02T14:07:20.033 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.004881214 2023-07-02T14:07:20.033 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.033 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.033 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.033 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.591755+0000", 2023-07-02T14:07:20.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.1639e-05 2023-07-02T14:07:20.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.034 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.034 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.591767+0000", 2023-07-02T14:07:20.034 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7150000000000001e-06 2023-07-02T14:07:20.034 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.034 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.591774+0000", 2023-07-02T14:07:20.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000128462 2023-07-02T14:07:20.035 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.035 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.035 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.035 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.591902+0000", 2023-07-02T14:07:20.035 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2172999999999999e-05 2023-07-02T14:07:20.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.036 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.591924+0000", 2023-07-02T14:07:20.036 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01373188 2023-07-02T14:07:20.036 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.036 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.036 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:20.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.605656+0000", 2023-07-02T14:07:20.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.0467999999999999e-05 2023-07-02T14:07:20.037 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.037 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.037 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.037 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.605697+0000", 2023-07-02T14:07:20.037 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018821676999999998 2023-07-02T14:07:20.038 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.038 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.038 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.038 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.038 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.038 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:23949 2.5 e207/170 2:be65b5f1:::rbd_data.125de1449605.0000000000000043:head v 207'2981, mlcod=207'2977)", 2023-07-02T14:07:20.038 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:58.698231+0000", 2023-07-02T14:07:20.038 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 21.168037532, 2023-07-02T14:07:20.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017072830000000001, 2023-07-02T14:07:20.039 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.039 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:20.039 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.039 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.039 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.039 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.698231+0000", 2023-07-02T14:07:20.039 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.040 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.698231+0000", 2023-07-02T14:07:20.040 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:20.040 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.040 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.040 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.698230+0000", 2023-07-02T14:07:20.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0034408080000000001 2023-07-02T14:07:20.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.041 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.041 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.041 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.701671+0000", 2023-07-02T14:07:20.041 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1169999999999993e-06 2023-07-02T14:07:20.041 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.701680+0000", 2023-07-02T14:07:20.042 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9620000000000001e-06 2023-07-02T14:07:20.042 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.042 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.042 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.042 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.701685+0000", 2023-07-02T14:07:20.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7456e-05 2023-07-02T14:07:20.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.043 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.043 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.701742+0000", 2023-07-02T14:07:20.043 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1971e-05 2023-07-02T14:07:20.043 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.043 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.701764+0000", 2023-07-02T14:07:20.044 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013496115 2023-07-02T14:07:20.044 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.044 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.044 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:20.044 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.715261+0000", 2023-07-02T14:07:20.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2963999999999997e-05 2023-07-02T14:07:20.045 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.045 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.045 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.045 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:58.715303+0000", 2023-07-02T14:07:20.045 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.017072830000000001 2023-07-02T14:07:20.045 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.045 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.046 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.046 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.046 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.046 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:20.046 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:07:20.048 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:20.046+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:20.049 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-07-02T14:07:20.147 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:20.147+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:20.220 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:20.221 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:07:20.221 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:07:20.221 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:20.231 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-07-02T14:07:20.248 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:20.247+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:20.350 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:20.349+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:20.356 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:20.356 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:07:20.356 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:07:20.356 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:07:20.356 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:20.367 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-07-02T14:07:20.397 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:20.400 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:07:20.412 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":385,"stamp":"2023-07-02T14:07:20.292528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":96,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":89620,"num_read_kb":15828021,"num_write":27382,"num_write_kb":11561441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3514,"num_bytes_recovered":11979257930,"num_keys_recovered":190,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27026,"ondisk_log_size":27026,"up":18,"acting":17,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":23208312,"kb_used_data":22864920,"kb_used_omap":229,"kb_used_meta":343130,"kb_avail":354279048,"statfs":{"total":386547056640,"available":362781745152,"internally_reserved":0,"allocated":23413678080,"data_stored":27281021064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":351365342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":472,"num_read_kb":37460,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.185540"},"pg_stats":[{"pgid":"2.7","version":"218'3701","reported_seq":9437,"reported_epoch":219,"state":"active+clean","last_fresh":"2023-07-02T14:07:16.109103+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:07:16.109103+0000","last_peered":"2023-07-02T14:07:16.109103+0000","last_clean":"2023-07-02T14:07:16.109103+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:07:16.109103+0000","last_undegraded":"2023-07-02T14:07:16.109103+0000","last_fullsized":"2023-07-02T14:07:16.109103+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3701,"ondisk_log_size":3701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1575041492,"num_objects":472,"num_object_clones":397,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":58519,"num_read_kb":2051790,"num_write":3143,"num_write_kb":1530741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"216'3069","reported_seq":5174,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-07-02T14:07:14.216006+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:07:14.216006+0000","last_peered":"2023-07-02T14:07:14.216006+0000","last_clean":"2023-07-02T14:07:14.216006+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:07:14.216006+0000","last_undegraded":"2023-07-02T14:07:14.216006+0000","last_fullsized":"2023-07-02T14:07:14.216006+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3069,"ondisk_log_size":3069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1538467250,"num_objects":486,"num_object_clones":415,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":1942,"num_read_kb":2141255,"num_write":2654,"num_write_kb":1492749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"218'3209","reported_seq":4731,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-07-02T14:07:17.837519+0000","last_change":"2023-07-02T14:07:14.111389+0000","last_active":"2023-07-02T14:07:17.837519+0000","last_peered":"2023-07-02T14:07:17.837519+0000","last_clean":"2023-07-02T14:07:17.837519+0000","last_became_active":"2023-07-02T14:07:14.107764+0000","last_became_peered":"2023-07-02T14:07:14.107764+0000","last_unstale":"2023-07-02T14:07:17.837519+0000","last_undegraded":"2023-07-02T14:07:17.837519+0000","last_fullsized":"2023-07-02T14:07:17.837519+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3209,"ondisk_log_size":3209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1517142988,"num_objects":428,"num_object_clones":355,"num_object_copies":856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":676,"num_read_kb":1976206,"num_write":2850,"num_write_kb":1467898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":2369610862,"num_keys_recovered":8,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"216'6494","reported_seq":9005,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-07-02T14:07:14.216095+0000","last_change":"2023-07-02T14:06:29.572710+0000","last_active":"2023-07-02T14:07:14.216095+0000","last_peered":"2023-07-02T14:07:14.216095+0000","last_clean":"2023-07-02T14:07:14.216095+0000","last_became_active":"2023-07-02T14:06:29.569824+0000","last_became_peered":"2023-07-02T14:06:29.569824+0000","last_unstale":"2023-07-02T14:07:14.216095+0000","last_undegraded":"2023-07-02T14:07:14.216095+0000","last_fullsized":"2023-07-02T14:07:14.216095+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":6494,"ondisk_log_size":6494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1675535810,"num_objects":495,"num_object_clones":414,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":23025,"num_read_kb":2213857,"num_write":9605,"num_write_kb":1635009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":120,"reported_epoch":219,"state":"active+clean","last_fresh":"2023-07-02T14:07:16.109161+0000","last_change":"2023-07-02T14:07:02.103009+0000","last_active":"2023-07-02T14:07:16.109161+0000","last_peered":"2023-07-02T14:07:16.109161+0000","last_clean":"2023-07-02T14:07:16.109161+0000","last_became_active":"2023-07-02T14:07:02.102654+0000","last_became_peered":"2023-07-02T14:07:02.102654+0000","last_unstale":"2023-07-02T14:07:16.109161+0000","last_undegraded":"2023-07-02T14:07:16.109161+0000","last_fullsized":"2023-07-02T14:07:16.109161+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"216'2975","reported_seq":4499,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-07-02T14:07:14.232736+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:07:14.232736+0000","last_peered":"2023-07-02T14:07:14.232736+0000","last_clean":"2023-07-02T14:07:14.232736+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:07:14.232736+0000","last_undegraded":"2023-07-02T14:07:14.232736+0000","last_fullsized":"2023-07-02T14:07:14.232736+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2975,"ondisk_log_size":2975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1661289598,"num_objects":499,"num_object_clones":416,"num_object_copies":998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":1534,"num_read_kb":2096750,"num_write":2559,"num_write_kb":1611276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"216'2884","reported_seq":3770,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-07-02T14:07:14.233750+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:07:14.233750+0000","last_peered":"2023-07-02T14:07:14.233750+0000","last_clean":"2023-07-02T14:07:14.233750+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:07:14.233750+0000","last_undegraded":"2023-07-02T14:07:14.233750+0000","last_fullsized":"2023-07-02T14:07:14.233750+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2884,"ondisk_log_size":2884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1569543764,"num_objects":461,"num_object_clones":383,"num_object_copies":922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":966,"num_read_kb":2013799,"num_write":2501,"num_write_kb":1523310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"216'2328","reported_seq":5146,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-07-02T14:07:14.192116+0000","last_change":"2023-07-02T14:07:09.383558+0000","last_active":"2023-07-02T14:07:14.192116+0000","last_peered":"2023-07-02T14:07:14.192116+0000","last_clean":"2023-07-02T14:07:14.192116+0000","last_became_active":"2023-07-02T14:07:09.382190+0000","last_became_peered":"2023-07-02T14:07:09.382190+0000","last_unstale":"2023-07-02T14:07:14.192116+0000","last_undegraded":"2023-07-02T14:07:14.192116+0000","last_fullsized":"2023-07-02T14:07:14.192116+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2328,"ondisk_log_size":2328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1230645001,"num_objects":405,"num_object_clones":325,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":1910,"num_read_kb":1765337,"num_write":2003,"num_write_kb":1186825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":1578968698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"218'2366","reported_seq":3976,"reported_epoch":218,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-07-02T14:07:15.337452+0000","last_change":"2023-07-02T14:07:12.410957+0000","last_active":"2023-07-02T14:07:15.337452+0000","last_peered":"2023-07-02T14:07:15.337452+0000","last_clean":"2023-07-02T14:07:01.026611+0000","last_became_active":"2023-07-02T14:07:04.018152+0000","last_became_peered":"2023-07-02T14:07:04.018152+0000","last_unstale":"2023-07-02T14:07:15.337452+0000","last_undegraded":"2023-07-02T14:07:03.110294+0000","last_fullsized":"2023-07-02T14:07:03.103717+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2366,"ondisk_log_size":2366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1146537804,"num_objects":358,"num_object_clones":299,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":96,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":1048,"num_read_kb":1569027,"num_write":2067,"num_write_kb":1113633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":330,"num_bytes_recovered":1087738504,"num_keys_recovered":8,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":96},{"shards":"1","objects":254}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":96,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":89620,"num_read_kb":15828021,"num_write":27382,"num_write_kb":11561441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3514,"num_bytes_recovered":11979257930,"num_keys_recovered":190,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26498469888,"data_stored":30723944105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":27026,"ondisk_log_size":27026,"up":16,"acting":15,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640324,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640324,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6749764,"kb_used_data":6661564,"kb_used_omap":0,"kb_used_meta":88192,"kb_avail":87622076,"statfs":{"total":96636764160,"available":89725005824,"internally_reserved":0,"allocated":6821441536,"data_stored":7847685364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90308608},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.53700000000000003}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44,"15min":0.46400000000000002},"min":{"1min":0.28399999999999997,"5min":0.215,"15min":0.215},"max":{"1min":0.71499999999999997,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.45000000000000001},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.58299999999999996}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.32800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66800000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.54900000000000004}]},{"osd":3,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.38300000000000001},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45900000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.36799999999999999,"15min":0.36299999999999999},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.48799999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.435}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.44900000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.379},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.433,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59099999999999997,"5min":1.026,"15min":1.026},"last":0.41499999999999998}]}]},{"osd":3,"up_from":133,"seq":571230650396,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4569040,"kb_used_data":4498824,"kb_used_omap":229,"kb_used_meta":69978,"kb_avail":89802800,"statfs":{"total":96636764160,"available":91958067200,"internally_reserved":0,"allocated":4606795776,"data_stored":5357917289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":71657694},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.42699999999999999}]},{"osd":1,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.372,"15min":0.36599999999999999},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.374},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.46400000000000002}]},{"osd":2,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.28699999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.44900000000000001,"15min":0.439},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.47799999999999998}]},{"osd":4,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.35799999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.36199999999999999,"15min":0.35299999999999998},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.33800000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.39800000000000002},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.39500000000000002,"15min":0.38800000000000001},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.55300000000000005}]}]},{"osd":2,"up_from":11,"seq":47244640324,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3200368,"kb_used_data":3146280,"kb_used_omap":0,"kb_used_meta":54080,"kb_avail":91171472,"statfs":{"total":96636764160,"available":93359587328,"internally_reserved":0,"allocated":3221790720,"data_stored":3710342833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55377920},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":4.3650000000000002}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.42499999999999999,"15min":0.374},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":2.0720000000000001},{"interface":"front","average":{"1min":0.376,"5min":0.33500000000000002,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.66500000000000004}]},{"osd":3,"last update":"Sun Jul 2 14:07:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.91100000000000003,"5min":1.929,"15min":1.929},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.54800000000000004,"15min":0.55600000000000005},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93999999999999995,"5min":2,"15min":2},"last":0.55800000000000005}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.50900000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68400000000000005,"5min":1.071,"15min":1.071},"last":2.1230000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":0.57399999999999995}]}]},{"osd":0,"up_from":0,"seq":712964571154,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":11,"seq":47244640324,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8689140,"kb_used_data":8558252,"kb_used_omap":0,"kb_used_meta":130880,"kb_avail":85682700,"statfs":{"total":96636764160,"available":87739084800,"internally_reserved":0,"allocated":8763650048,"data_stored":10365075578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":134021120},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.318},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.26000000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.224,"5min":0.184,"15min":0.184},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.28499999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.49099999999999999,"15min":0.47299999999999998},"min":{"1min":0.30299999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.33400000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71899999999999997,"5min":1.524,"15min":1.524},"last":0.372}]},{"osd":5,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.80300000000000005,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84399999999999997,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.41899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3074248704,"data_stored":3409275605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8761016320,"data_stored":10364040347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3219156992,"data_stored":3709307602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4604137472,"data_stored":5356872530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6839910400,"data_stored":7884448021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:07:20.413 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-07-02T14:07:20.453 INFO:tasks.ceph.osd.0.smithi016.stderr:2023-07-02T14:07:20.452+0000 7fd57692d700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 115683) UID: 0 2023-07-02T14:07:20.508 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:20.508 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:07:20.508 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:07:20.508 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:07:20.509 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.509 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2807 2.4 2.f678c15c (undecoded) ondisk+write+known_if_redirected e59)", 2023-07-02T14:07:20.509 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:07:20.509 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 251.05441024500001, 2023-07-02T14:07:20.509 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998, 2023-07-02T14:07:20.509 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.509 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:07:20.510 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:20.510 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:20.510 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:20.510 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2807 2023-07-02T14:07:20.510 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.510 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.510 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.510 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:07:20.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.511 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.511 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.511 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452100+0000", 2023-07-02T14:07:20.511 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:20.511 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.512 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452099+0000", 2023-07-02T14:07:20.512 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1730999999999996e-05 2023-07-02T14:07:20.512 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.512 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.512 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452161+0000", 2023-07-02T14:07:20.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2522e-05 2023-07-02T14:07:20.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.513 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.513 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.513 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452173+0000", 2023-07-02T14:07:20.513 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.75e-06 2023-07-02T14:07:20.513 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.514 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452179+0000", 2023-07-02T14:07:20.514 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4567999999999997e-05 2023-07-02T14:07:20.514 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.514 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.514 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452224+0000", 2023-07-02T14:07:20.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6409999999999999e-06 2023-07-02T14:07:20.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.515 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.515 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:20.515 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:09.452229+0000", 2023-07-02T14:07:20.515 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87056857700000001 2023-07-02T14:07:20.515 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322798+0000", 2023-07-02T14:07:20.516 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4849999999999997e-06 2023-07-02T14:07:20.516 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.516 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.516 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.516 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322804+0000", 2023-07-02T14:07:20.517 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.87070430799999998 2023-07-02T14:07:20.517 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.517 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.517 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.517 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.517 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.517 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4861.0:3 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e60)", 2023-07-02T14:07:20.518 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:07:20.518 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 250.183717237, 2023-07-02T14:07:20.518 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663, 2023-07-02T14:07:20.518 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.518 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:20.518 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:20.518 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4861", 2023-07-02T14:07:20.519 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/1039679439", 2023-07-02T14:07:20.519 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3 2023-07-02T14:07:20.519 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.519 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.519 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.519 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.519 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:07:20.519 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.520 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.520 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322793+0000", 2023-07-02T14:07:20.520 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:20.520 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.520 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322792+0000", 2023-07-02T14:07:20.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:07:20.521 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.521 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.521 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.521 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322799+0000", 2023-07-02T14:07:20.521 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2560000000000008e-06 2023-07-02T14:07:20.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.522 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322808+0000", 2023-07-02T14:07:20.522 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.4420000000000003e-06 2023-07-02T14:07:20.522 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.522 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.522 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322815+0000", 2023-07-02T14:07:20.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8288999999999998e-05 2023-07-02T14:07:20.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.523 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.523 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.523 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322863+0000", 2023-07-02T14:07:20.523 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9880000000000004e-06 2023-07-02T14:07:20.523 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:20.524 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322871+0000", 2023-07-02T14:07:20.524 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0298754699999999 2023-07-02T14:07:20.524 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.524 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.524 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352746+0000", 2023-07-02T14:07:20.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00025986800000000001 2023-07-02T14:07:20.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.525 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.525 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.525 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353006+0000", 2023-07-02T14:07:20.525 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048347100000000002 2023-07-02T14:07:20.525 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.526 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.526 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.526 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353490+0000", 2023-07-02T14:07:20.526 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.03069663 2023-07-02T14:07:20.526 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.526 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.526 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.527 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.527 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.527 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2601 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 65536~3801088] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:07:20.527 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:07:20.527 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 250.183592516, 2023-07-02T14:07:20.527 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001, 2023-07-02T14:07:20.527 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.527 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:07:20.528 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:20.528 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:20.528 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:20.528 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2601 2023-07-02T14:07:20.528 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.528 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.528 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:07:20.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.529 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.529 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.529 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.529 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322918+0000", 2023-07-02T14:07:20.529 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:20.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.530 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.322917+0000", 2023-07-02T14:07:20.530 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0067992540000000002 2023-07-02T14:07:20.530 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.530 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.530 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329716+0000", 2023-07-02T14:07:20.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2949e-05 2023-07-02T14:07:20.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.531 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.531 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.531 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329739+0000", 2023-07-02T14:07:20.531 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7230000000000004e-06 2023-07-02T14:07:20.531 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.532 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329747+0000", 2023-07-02T14:07:20.532 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6341000000000001e-05 2023-07-02T14:07:20.532 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.532 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.532 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329803+0000", 2023-07-02T14:07:20.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3662000000000001e-05 2023-07-02T14:07:20.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.533 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.533 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:20.533 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329817+0000", 2023-07-02T14:07:20.533 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0227830609999999 2023-07-02T14:07:20.533 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.534 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352600+0000", 2023-07-02T14:07:20.534 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.3913999999999996e-05 2023-07-02T14:07:20.534 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.534 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.534 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.535 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.352694+0000", 2023-07-02T14:07:20.535 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0297758480000001 2023-07-02T14:07:20.535 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.535 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.535 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.535 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.535 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.535 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2602 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3866624~45056] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:07:20.536 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:07:20.536 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 250.17674263200001, 2023-07-02T14:07:20.536 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001, 2023-07-02T14:07:20.536 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.536 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:07:20.536 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:20.536 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:20.536 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:20.537 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2602 2023-07-02T14:07:20.537 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.537 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.537 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.537 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.537 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:07:20.537 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.538 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329768+0000", 2023-07-02T14:07:20.538 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999895 2023-07-02T14:07:20.538 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.538 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.538 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329757+0000", 2023-07-02T14:07:20.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6297e-05 2023-07-02T14:07:20.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.539 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.539 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.539 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329813+0000", 2023-07-02T14:07:20.539 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1530000000000002e-06 2023-07-02T14:07:20.539 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.540 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329820+0000", 2023-07-02T14:07:20.540 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.506e-06 2023-07-02T14:07:20.540 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.540 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.540 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329823+0000", 2023-07-02T14:07:20.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4910999999999998e-05 2023-07-02T14:07:20.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.541 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.541 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.541 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329858+0000", 2023-07-02T14:07:20.541 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6999999999999997e-06 2023-07-02T14:07:20.541 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:20.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329867+0000", 2023-07-02T14:07:20.542 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0236589039999999 2023-07-02T14:07:20.542 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.542 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.542 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.542 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353526+0000", 2023-07-02T14:07:20.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011072 2023-07-02T14:07:20.543 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.543 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.543 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.543 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353636+0000", 2023-07-02T14:07:20.543 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0238688010000001 2023-07-02T14:07:20.543 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.543 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.544 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.544 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.544 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.544 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2608 2.7 2:e74db836:::rbd_data.125de1449605.00000000000000dd:head [write 3825664~368640] snapc 7=[7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:07:20.544 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:07:20.544 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 250.17667970700001, 2023-07-02T14:07:20.544 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883, 2023-07-02T14:07:20.544 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.545 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:07:20.545 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:20.545 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:20.545 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:20.545 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2608 2023-07-02T14:07:20.545 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.545 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.545 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:07:20.546 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.546 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.546 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.546 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.546 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329831+0000", 2023-07-02T14:07:20.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:20.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.547 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.547 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.329830+0000", 2023-07-02T14:07:20.547 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028275000000000002 2023-07-02T14:07:20.547 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.547 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330112+0000", 2023-07-02T14:07:20.548 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.2979999999999992e-06 2023-07-02T14:07:20.548 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.548 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.548 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.548 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330121+0000", 2023-07-02T14:07:20.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.6519999999999999e-06 2023-07-02T14:07:20.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.549 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.549 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330124+0000", 2023-07-02T14:07:20.549 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3586e-05 2023-07-02T14:07:20.549 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.549 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330158+0000", 2023-07-02T14:07:20.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7150000000000004e-06 2023-07-02T14:07:20.550 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.550 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.550 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:20.550 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330167+0000", 2023-07-02T14:07:20.550 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0235021929999999 2023-07-02T14:07:20.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.551 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353669+0000", 2023-07-02T14:07:20.551 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.8589000000000001e-05 2023-07-02T14:07:20.551 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.551 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.551 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.552 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353767+0000", 2023-07-02T14:07:20.552 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.023936883 2023-07-02T14:07:20.552 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.552 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.552 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.552 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.552 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.553 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2736 2.7 2:e85b1b19:::rbd_data.125de1449605.00000000000000df:head [write 3911680~282624] snapc 7=[7,6,5,4] RETRY=1 ondisk+retry+write+known_if_redirected e60)", 2023-07-02T14:07:20.553 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:07:20.553 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 250.17637928600001, 2023-07-02T14:07:20.553 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417, 2023-07-02T14:07:20.553 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.553 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:20.553 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:20.553 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2736 2023-07-02T14:07:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:07:20.554 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.555 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330131+0000", 2023-07-02T14:07:20.555 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:20.555 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.555 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.555 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330130+0000", 2023-07-02T14:07:20.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00050705699999999999 2023-07-02T14:07:20.556 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.556 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.556 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.556 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330637+0000", 2023-07-02T14:07:20.556 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.3869999999999995e-06 2023-07-02T14:07:20.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.557 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330646+0000", 2023-07-02T14:07:20.557 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4630000000000001e-06 2023-07-02T14:07:20.557 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.557 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.557 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330649+0000", 2023-07-02T14:07:20.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9820999999999997e-05 2023-07-02T14:07:20.558 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.558 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.558 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.558 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330689+0000", 2023-07-02T14:07:20.558 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.6889999999999999e-06 2023-07-02T14:07:20.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:20.559 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330699+0000", 2023-07-02T14:07:20.559 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.02309987 2023-07-02T14:07:20.559 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.559 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.559 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353799+0000", 2023-07-02T14:07:20.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016765899999999999 2023-07-02T14:07:20.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.560 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.560 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.560 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.353966+0000", 2023-07-02T14:07:20.560 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00021509900000000001 2023-07-02T14:07:20.560 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 3,4", 2023-07-02T14:07:20.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.354181+0000", 2023-07-02T14:07:20.561 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0039879729999999997 2023-07-02T14:07:20.561 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.561 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.561 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:20.561 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358169+0000", 2023-07-02T14:07:20.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.6820999999999998e-05 2023-07-02T14:07:20.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.562 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:20.562 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.358216+0000", 2023-07-02T14:07:20.562 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.007707378 2023-07-02T14:07:20.562 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.562 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:20.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365924+0000", 2023-07-02T14:07:20.563 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6579999999999997e-05 2023-07-02T14:07:20.563 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.563 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.563 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:20.563 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.365980+0000", 2023-07-02T14:07:20.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015022500000000001 2023-07-02T14:07:20.564 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.564 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.564 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.564 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366130+0000", 2023-07-02T14:07:20.564 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035999417 2023-07-02T14:07:20.564 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.564 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.565 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.565 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.565 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.565 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2809 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 9] snapc 0=[] ondisk+write+known_if_redirected e60)", 2023-07-02T14:07:20.565 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:07:20.565 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 250.175854786, 2023-07-02T14:07:20.565 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905, 2023-07-02T14:07:20.565 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.566 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:20.566 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:20.566 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:20.566 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:20.566 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2809 2023-07-02T14:07:20.566 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.566 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:07:20.567 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.567 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.567 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.567 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.567 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:07:20.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:20.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.568 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.568 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330655+0000", 2023-07-02T14:07:20.568 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3500000000000001e-06 2023-07-02T14:07:20.568 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.568 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330658+0000", 2023-07-02T14:07:20.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.545e-06 2023-07-02T14:07:20.569 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.569 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.569 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.569 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330665+0000", 2023-07-02T14:07:20.569 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5739999999999997e-06 2023-07-02T14:07:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330669+0000", 2023-07-02T14:07:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9210999999999999e-05 2023-07-02T14:07:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.570 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330728+0000", 2023-07-02T14:07:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:07:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:10.330738+0000", 2023-07-02T14:07:20.571 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.026647184 2023-07-02T14:07:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357385+0000", 2023-07-02T14:07:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000121279 2023-07-02T14:07:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.572 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.357506+0000", 2023-07-02T14:07:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0087652709999999998 2023-07-02T14:07:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.366271+0000", 2023-07-02T14:07:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.035615905 2023-07-02T14:07:20.573 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:4234 2.1 2.c00b9641 (undecoded) ondisk+write+known_if_redirected e79)", 2023-07-02T14:07:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:07:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 223.05183566100001, 2023-07-02T14:07:20.574 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382, 2023-07-02T14:07:20.575 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.575 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:07:20.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:20.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:20.575 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:20.575 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 4234 2023-07-02T14:07:20.575 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.575 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.576 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:07:20.576 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.576 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.576 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.576 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454675+0000", 2023-07-02T14:07:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.454674+0000", 2023-07-02T14:07:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001490344 2023-07-02T14:07:20.577 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456164+0000", 2023-07-02T14:07:20.578 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6677999999999999e-05 2023-07-02T14:07:20.578 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.578 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.578 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.578 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456181+0000", 2023-07-02T14:07:20.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5120000000000002e-06 2023-07-02T14:07:20.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.579 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.579 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456188+0000", 2023-07-02T14:07:20.579 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1499000000000003e-05 2023-07-02T14:07:20.579 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.579 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456240+0000", 2023-07-02T14:07:20.580 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0514e-05 2023-07-02T14:07:20.580 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.580 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.580 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:20.580 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.456250+0000", 2023-07-02T14:07:20.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.62484402400000005 2023-07-02T14:07:20.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.581 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.581 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081094+0000", 2023-07-02T14:07:20.581 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.7460000000000003e-06 2023-07-02T14:07:20.581 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.581 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.582 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.582 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.081101+0000", 2023-07-02T14:07:20.582 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.626426382 2023-07-02T14:07:20.582 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.582 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.582 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.582 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.582 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.583 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:3878 2.1 2:96c43c99:::rbd_data.125de1449605.00000000000000b5:head [write 65536~4128768] snapc d=[d,c,b,a,9,8,7,6,5,4] RETRY=2 ondisk+retry+write+known_if_redirected e79)", 2023-07-02T14:07:20.583 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:07:20.583 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 222.72635295699999, 2023-07-02T14:07:20.583 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003, 2023-07-02T14:07:20.583 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.583 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:20.583 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:20.583 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:20.584 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:20.584 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 3878 2023-07-02T14:07:20.584 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.584 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.584 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.584 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.584 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:07:20.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.585 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.585 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780157+0000", 2023-07-02T14:07:20.585 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:20.585 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.585 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.780156+0000", 2023-07-02T14:07:20.586 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0038233609999999999 2023-07-02T14:07:20.586 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.586 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.586 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.586 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.783980+0000", 2023-07-02T14:07:20.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.1725999999999998e-05 2023-07-02T14:07:20.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.587 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.587 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.587 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784001+0000", 2023-07-02T14:07:20.587 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7049999999999999e-06 2023-07-02T14:07:20.587 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784009+0000", 2023-07-02T14:07:20.588 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8592999999999999e-05 2023-07-02T14:07:20.588 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.588 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.588 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.588 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784078+0000", 2023-07-02T14:07:20.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3528e-05 2023-07-02T14:07:20.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.589 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:20.589 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:37.784091+0000", 2023-07-02T14:07:20.589 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.32217544399999998 2023-07-02T14:07:20.589 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.589 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106267+0000", 2023-07-02T14:07:20.590 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00010935200000000001 2023-07-02T14:07:20.590 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.590 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.590 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.590 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106376+0000", 2023-07-02T14:07:20.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016171800000000001 2023-07-02T14:07:20.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.591 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 0", 2023-07-02T14:07:20.591 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.106538+0000", 2023-07-02T14:07:20.591 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.018680338000000001 2023-07-02T14:07:20.591 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.591 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:20.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.125218+0000", 2023-07-02T14:07:20.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.023610490000000001 2023-07-02T14:07:20.592 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.592 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.592 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:20.592 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.148829+0000", 2023-07-02T14:07:20.592 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00020197600000000001 2023-07-02T14:07:20.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:20.593 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149031+0000", 2023-07-02T14:07:20.593 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4419000000000003e-05 2023-07-02T14:07:20.593 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.593 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.593 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.594 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:38.149075+0000", 2023-07-02T14:07:20.594 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.36891778200000003 2023-07-02T14:07:20.594 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.594 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.594 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.594 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.594 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.594 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:17 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 0~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:20.595 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:07:20.595 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 129.30807717499999, 2023-07-02T14:07:20.595 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997, 2023-07-02T14:07:20.595 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.595 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:20.595 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:20.595 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:20.595 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:20.596 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 17 2023-07-02T14:07:20.596 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.596 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.596 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.596 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.596 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:07:20.596 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.597 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.597 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198433+0000", 2023-07-02T14:07:20.597 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999976 2023-07-02T14:07:20.597 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.597 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198431+0000", 2023-07-02T14:07:20.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2815e-05 2023-07-02T14:07:20.598 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.598 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.598 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.598 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198443+0000", 2023-07-02T14:07:20.598 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2569999999999999e-05 2023-07-02T14:07:20.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.599 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198466+0000", 2023-07-02T14:07:20.599 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0223e-05 2023-07-02T14:07:20.599 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.599 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.599 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198476+0000", 2023-07-02T14:07:20.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1792999999999997e-05 2023-07-02T14:07:20.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.600 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.600 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.600 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:07:20.600 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.4249999999999998e-06 2023-07-02T14:07:20.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:20.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198536+0000", 2023-07-02T14:07:20.601 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91888663500000001 2023-07-02T14:07:20.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.601 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.601 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.601 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117423+0000", 2023-07-02T14:07:20.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000223594 2023-07-02T14:07:20.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.602 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.602 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.602 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117647+0000", 2023-07-02T14:07:20.602 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00033832900000000001 2023-07-02T14:07:20.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.603 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.603 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.603 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.117985+0000", 2023-07-02T14:07:20.603 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91955186499999997 2023-07-02T14:07:20.603 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.603 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.603 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.603 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.604 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.604 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:18 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 65536~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:20.604 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:07:20.604 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 129.30801996400001, 2023-07-02T14:07:20.604 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998, 2023-07-02T14:07:20.604 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.604 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:20.604 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:20.605 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:20.605 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:20.605 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 18 2023-07-02T14:07:20.605 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.605 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.605 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.605 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:07:20.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.606 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.606 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.606 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:07:20.606 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:20.606 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198490+0000", 2023-07-02T14:07:20.607 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7600000000000002e-06 2023-07-02T14:07:20.607 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.607 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.607 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.607 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198494+0000", 2023-07-02T14:07:20.608 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.7840000000000004e-06 2023-07-02T14:07:20.608 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.608 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.608 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198502+0000", 2023-07-02T14:07:20.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6459999999999998e-06 2023-07-02T14:07:20.609 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.609 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.609 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.609 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198508+0000", 2023-07-02T14:07:20.609 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.7395999999999997e-05 2023-07-02T14:07:20.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.610 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198565+0000", 2023-07-02T14:07:20.610 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6800000000000004e-06 2023-07-02T14:07:20.610 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.610 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.610 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:20.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198572+0000", 2023-07-02T14:07:20.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.91965896800000002 2023-07-02T14:07:20.611 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.611 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.611 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.611 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118231+0000", 2023-07-02T14:07:20.611 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5212999999999998e-05 2023-07-02T14:07:20.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.612 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118326+0000", 2023-07-02T14:07:20.612 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000238538 2023-07-02T14:07:20.612 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.612 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.612 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.613 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118565+0000", 2023-07-02T14:07:20.613 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92007426199999998 2023-07-02T14:07:20.613 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.613 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.613 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.613 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.613 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.613 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:19 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 131072~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:20.614 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:07:20.614 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 129.30799574, 2023-07-02T14:07:20.614 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002, 2023-07-02T14:07:20.614 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.614 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:20.614 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:20.614 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:20.615 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:20.615 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 19 2023-07-02T14:07:20.615 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.615 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.615 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.615 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.615 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:07:20.615 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.616 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:07:20.616 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:20.616 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.616 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.616 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198514+0000", 2023-07-02T14:07:20.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1990000000000002e-06 2023-07-02T14:07:20.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.617 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.617 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.617 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198517+0000", 2023-07-02T14:07:20.617 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8789999999999997e-06 2023-07-02T14:07:20.617 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.618 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198524+0000", 2023-07-02T14:07:20.618 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2069999999999997e-06 2023-07-02T14:07:20.618 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.618 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.618 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198528+0000", 2023-07-02T14:07:20.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5545999999999999e-05 2023-07-02T14:07:20.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.619 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.619 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.619 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:07:20.619 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2540000000000003e-06 2023-07-02T14:07:20.619 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:20.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198598+0000", 2023-07-02T14:07:20.620 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.919983263 2023-07-02T14:07:20.620 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.620 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.620 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.620 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118581+0000", 2023-07-02T14:07:20.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6878999999999994e-05 2023-07-02T14:07:20.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.621 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.621 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118648+0000", 2023-07-02T14:07:20.621 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000149732 2023-07-02T14:07:20.621 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.621 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.622 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.622 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118798+0000", 2023-07-02T14:07:20.622 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92028352400000002 2023-07-02T14:07:20.622 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.622 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.622 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.622 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.623 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.623 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:20 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 262144~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:20.623 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:07:20.623 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 129.307976776, 2023-07-02T14:07:20.623 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998, 2023-07-02T14:07:20.623 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.623 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:20.623 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:20.624 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:20.624 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:20.624 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20 2023-07-02T14:07:20.624 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.624 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.624 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.624 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.624 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:07:20.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.625 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.625 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:07:20.625 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:20.625 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.625 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198533+0000", 2023-07-02T14:07:20.626 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.732e-06 2023-07-02T14:07:20.626 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.626 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.626 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.626 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198537+0000", 2023-07-02T14:07:20.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.5259999999999999e-06 2023-07-02T14:07:20.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.627 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.627 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198543+0000", 2023-07-02T14:07:20.627 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.121e-06 2023-07-02T14:07:20.627 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.627 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.628 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198547+0000", 2023-07-02T14:07:20.628 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.9591000000000005e-05 2023-07-02T14:07:20.628 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.628 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.628 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:07:20.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3120000000000004e-06 2023-07-02T14:07:20.629 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.629 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.629 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:20.629 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198621+0000", 2023-07-02T14:07:20.629 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.92020789400000003 2023-07-02T14:07:20.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.630 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.630 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.118829+0000", 2023-07-02T14:07:20.630 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022393386000000001 2023-07-02T14:07:20.630 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.630 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141223+0000", 2023-07-02T14:07:20.631 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000329866 2023-07-02T14:07:20.631 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.631 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.631 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.631 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141552+0000", 2023-07-02T14:07:20.632 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94301902599999998 2023-07-02T14:07:20.632 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.632 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.632 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.632 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.632 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.632 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:21 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 196608~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:20.633 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:07:20.633 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 129.30794914099999, 2023-07-02T14:07:20.633 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995, 2023-07-02T14:07:20.633 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.633 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:20.633 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:20.633 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:20.634 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:20.634 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 21 2023-07-02T14:07:20.634 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.634 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.634 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.634 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.634 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:07:20.634 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.635 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:07:20.635 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:20.635 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.635 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.635 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198561+0000", 2023-07-02T14:07:20.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0169999999999999e-05 2023-07-02T14:07:20.636 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.636 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.636 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.636 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198571+0000", 2023-07-02T14:07:20.636 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.3479999999999998e-06 2023-07-02T14:07:20.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.637 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198578+0000", 2023-07-02T14:07:20.637 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8439999999999997e-06 2023-07-02T14:07:20.637 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.637 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.637 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198583+0000", 2023-07-02T14:07:20.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8294999999999997e-05 2023-07-02T14:07:20.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.638 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.638 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.638 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198641+0000", 2023-07-02T14:07:20.638 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8120000000000001e-06 2023-07-02T14:07:20.638 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:20.639 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:07:20.639 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94293828499999999 2023-07-02T14:07:20.639 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.639 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.639 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141583+0000", 2023-07-02T14:07:20.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014607300000000001 2023-07-02T14:07:20.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.640 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.640 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.640 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.141729+0000", 2023-07-02T14:07:20.640 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00044906600000000003 2023-07-02T14:07:20.640 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.641 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.641 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.641 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142178+0000", 2023-07-02T14:07:20.641 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94361737599999995 2023-07-02T14:07:20.641 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.641 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.641 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.641 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.642 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.642 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:22 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 327680~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:20.642 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:07:20.642 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 129.30791571099999, 2023-07-02T14:07:20.642 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004, 2023-07-02T14:07:20.642 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.642 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:20.642 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:20.643 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:20.643 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:20.643 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 22 2023-07-02T14:07:20.643 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.643 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.643 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.643 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.643 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:07:20.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.644 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.644 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.644 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:07:20.644 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:20.644 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198594+0000", 2023-07-02T14:07:20.645 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4850000000000001e-06 2023-07-02T14:07:20.645 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.645 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.645 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.645 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198597+0000", 2023-07-02T14:07:20.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.204e-06 2023-07-02T14:07:20.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.646 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.646 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198605+0000", 2023-07-02T14:07:20.646 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4839999999999998e-06 2023-07-02T14:07:20.646 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.646 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198609+0000", 2023-07-02T14:07:20.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3100999999999999e-05 2023-07-02T14:07:20.647 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.647 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.647 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.647 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198662+0000", 2023-07-02T14:07:20.647 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9099999999999998e-06 2023-07-02T14:07:20.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:20.648 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198666+0000", 2023-07-02T14:07:20.648 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.943539503 2023-07-02T14:07:20.648 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.648 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.648 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142206+0000", 2023-07-02T14:07:20.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012255999999999999 2023-07-02T14:07:20.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.649 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.649 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.649 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142328+0000", 2023-07-02T14:07:20.649 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000208956 2023-07-02T14:07:20.649 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.650 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.650 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.650 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142537+0000", 2023-07-02T14:07:20.650 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94394266000000004 2023-07-02T14:07:20.650 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.650 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.650 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.651 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.651 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.651 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:23 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 393216~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:20.651 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:07:20.651 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 129.307895731, 2023-07-02T14:07:20.651 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348, 2023-07-02T14:07:20.651 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.651 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:20.652 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:20.652 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:20.652 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:20.652 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23 2023-07-02T14:07:20.652 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.652 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.652 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.652 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:07:20.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.653 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.653 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.653 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:07:20.653 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:20.653 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198614+0000", 2023-07-02T14:07:20.654 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8439999999999998e-06 2023-07-02T14:07:20.654 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.654 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.654 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.654 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198617+0000", 2023-07-02T14:07:20.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0440000000000001e-06 2023-07-02T14:07:20.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.655 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.655 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198623+0000", 2023-07-02T14:07:20.655 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4480000000000002e-06 2023-07-02T14:07:20.655 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.655 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198628+0000", 2023-07-02T14:07:20.656 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5843e-05 2023-07-02T14:07:20.656 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.656 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.656 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.656 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:07:20.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.89e-06 2023-07-02T14:07:20.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.657 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:20.657 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198688+0000", 2023-07-02T14:07:20.657 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94389171999999999 2023-07-02T14:07:20.657 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.657 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142580+0000", 2023-07-02T14:07:20.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3892000000000004e-05 2023-07-02T14:07:20.658 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.658 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.658 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.658 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142644+0000", 2023-07-02T14:07:20.658 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017608000000000001 2023-07-02T14:07:20.659 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.659 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.659 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.659 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142820+0000", 2023-07-02T14:07:20.659 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.944205348 2023-07-02T14:07:20.659 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.659 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.659 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.660 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.660 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.660 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:24 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 458752~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:20.660 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:07:20.660 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 129.307867657, 2023-07-02T14:07:20.660 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004, 2023-07-02T14:07:20.660 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.660 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:20.661 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:20.661 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:20.661 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:20.661 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 24 2023-07-02T14:07:20.661 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.661 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.661 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:07:20.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.662 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.662 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.662 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.662 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198643+0000", 2023-07-02T14:07:20.662 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:20.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.663 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198642+0000", 2023-07-02T14:07:20.663 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.253e-06 2023-07-02T14:07:20.663 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.663 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.663 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198645+0000", 2023-07-02T14:07:20.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1289999999999998e-06 2023-07-02T14:07:20.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.664 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.664 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.664 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198652+0000", 2023-07-02T14:07:20.664 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.4510000000000002e-06 2023-07-02T14:07:20.664 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198657+0000", 2023-07-02T14:07:20.665 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9131000000000001e-05 2023-07-02T14:07:20.665 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.665 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.665 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.665 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198706+0000", 2023-07-02T14:07:20.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8890000000000004e-06 2023-07-02T14:07:20.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.666 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.666 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:20.666 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198710+0000", 2023-07-02T14:07:20.666 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94415412200000004 2023-07-02T14:07:20.666 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142864+0000", 2023-07-02T14:07:20.667 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.8047000000000006e-05 2023-07-02T14:07:20.667 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.667 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.667 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.667 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.142952+0000", 2023-07-02T14:07:20.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000192967 2023-07-02T14:07:20.668 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.668 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.668 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.668 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143145+0000", 2023-07-02T14:07:20.668 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94450251500000004 2023-07-02T14:07:20.668 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.668 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.669 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.669 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.669 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.669 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:25 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 524288~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:20.669 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:07:20.669 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 129.30784886199999, 2023-07-02T14:07:20.669 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001, 2023-07-02T14:07:20.669 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.670 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:20.670 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:20.670 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:20.670 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:20.670 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 25 2023-07-02T14:07:20.670 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.670 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.670 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:07:20.671 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.671 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.671 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.671 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.671 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:07:20.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:20.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.672 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.672 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198661+0000", 2023-07-02T14:07:20.672 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.615e-06 2023-07-02T14:07:20.672 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.672 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198664+0000", 2023-07-02T14:07:20.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4759999999999997e-06 2023-07-02T14:07:20.673 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.673 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.673 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.673 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198669+0000", 2023-07-02T14:07:20.673 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5999999999999998e-06 2023-07-02T14:07:20.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.674 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198673+0000", 2023-07-02T14:07:20.674 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4256999999999999e-05 2023-07-02T14:07:20.674 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.674 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.674 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198727+0000", 2023-07-02T14:07:20.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7840000000000001e-06 2023-07-02T14:07:20.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.675 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.675 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:20.675 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198731+0000", 2023-07-02T14:07:20.675 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94444241900000003 2023-07-02T14:07:20.675 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.676 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143173+0000", 2023-07-02T14:07:20.676 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6839000000000002e-05 2023-07-02T14:07:20.676 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.676 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.676 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143240+0000", 2023-07-02T14:07:20.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017053399999999999 2023-07-02T14:07:20.677 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.677 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.677 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.677 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143411+0000", 2023-07-02T14:07:20.677 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94474915500000001 2023-07-02T14:07:20.677 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.678 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.678 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.678 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.678 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.678 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5561.0:26 2.6 2:744d72b6:::rbd_data.125de1449605.0000000000000100:25 [sparse-read 589824~65536 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e139)", 2023-07-02T14:07:20.678 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:07:20.678 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 129.30782571099999, 2023-07-02T14:07:20.678 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997, 2023-07-02T14:07:20.679 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.679 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:20.679 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:20.679 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5561", 2023-07-02T14:07:20.679 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/3262395932", 2023-07-02T14:07:20.679 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 26 2023-07-02T14:07:20.679 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.679 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:07:20.680 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.680 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.680 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.680 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.680 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:07:20.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:20.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.681 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.681 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198684+0000", 2023-07-02T14:07:20.681 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0520000000000002e-06 2023-07-02T14:07:20.681 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198687+0000", 2023-07-02T14:07:20.682 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1880000000000002e-06 2023-07-02T14:07:20.682 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.682 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.682 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.682 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198695+0000", 2023-07-02T14:07:20.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9960000000000004e-06 2023-07-02T14:07:20.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.683 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.683 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198699+0000", 2023-07-02T14:07:20.683 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2330999999999998e-05 2023-07-02T14:07:20.683 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.683 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198742+0000", 2023-07-02T14:07:20.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.7950000000000001e-06 2023-07-02T14:07:20.684 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.684 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.684 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:20.684 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.198745+0000", 2023-07-02T14:07:20.684 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94469398199999999 2023-07-02T14:07:20.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.685 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143439+0000", 2023-07-02T14:07:20.685 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6048e-05 2023-07-02T14:07:20.685 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.685 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.685 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143525+0000", 2023-07-02T14:07:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00017016700000000001 2023-07-02T14:07:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:12.143696+0000", 2023-07-02T14:07:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.94501115199999997 2023-07-02T14:07:20.686 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.6579.0:11 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected e171)", 2023-07-02T14:07:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:07:20.687 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 83.348027224999996, 2023-07-02T14:07:20.688 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998, 2023-07-02T14:07:20.688 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:20.688 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:20.688 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:20.688 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.6579", 2023-07-02T14:07:20.688 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/554403467", 2023-07-02T14:07:20.688 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 11 2023-07-02T14:07:20.688 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.689 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:20.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:20.689 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:07:20.689 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:20.689 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.689 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.689 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:20.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158483+0000", 2023-07-02T14:07:20.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:20.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.690 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.690 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:20.690 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158481+0000", 2023-07-02T14:07:20.690 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8629999999999996e-06 2023-07-02T14:07:20.690 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:20.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158487+0000", 2023-07-02T14:07:20.691 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7630000000000006e-06 2023-07-02T14:07:20.691 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.691 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.691 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:20.691 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158496+0000", 2023-07-02T14:07:20.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.5100000000000001e-06 2023-07-02T14:07:20.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.692 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.692 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:20.692 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158504+0000", 2023-07-02T14:07:20.692 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.9397000000000002e-05 2023-07-02T14:07:20.692 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158553+0000", 2023-07-02T14:07:20.693 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.5810000000000003e-06 2023-07-02T14:07:20.693 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.693 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.693 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:20.693 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:57.158562+0000", 2023-07-02T14:07:20.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93893079499999998 2023-07-02T14:07:20.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.694 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:20.694 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097493+0000", 2023-07-02T14:07:20.694 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015799299999999999 2023-07-02T14:07:20.694 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.694 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:20.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097651+0000", 2023-07-02T14:07:20.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187746 2023-07-02T14:07:20.695 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:20.695 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:20.695 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:20.695 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:58.097839+0000", 2023-07-02T14:07:20.695 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.93935603499999998 2023-07-02T14:07:20.696 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.696 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.696 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.696 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:20.696 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:20.696 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:20.698 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-07-02T14:07:20.699 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:20.696+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:20.798 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:20.797+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:20.841 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:20.841 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8/7208 objects degraded (0.111%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":343,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":222,"num_osds":6,"num_up_osds":6,"osd_up_since":1688306752,"num_in_osds":4,"osd_in_since":1688306821,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":3604,"data_bytes":11914203707,"bytes_used":23765311488,"bytes_avail":362781745152,"bytes_total":386547056640,"degraded_objects":8,"degraded_total":7208,"degraded_ratio":0.0011098779134295228,"misplaced_objects":96,"misplaced_total":7208,"misplaced_ratio":0.013318534961154271,"recovering_objects_per_sec":59,"recovering_bytes_per_sec":194627700,"recovering_keys_per_sec":0,"num_objects_recovered":249,"num_bytes_recovered":815858312,"num_keys_recovered":4,"read_bytes_sec":87743328,"write_bytes_sec":3799548,"read_op_per_sec":149,"write_op_per_sec":17},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2023-07-02T14:06:52.287392+0000","services":{}},"progress_events":{"39f7aefe-a986-4f7d-8f18-834c65a827f0":{"message":"Rebalancing after osd.0 marked out (15s)\n [=========================...] (remaining: 1s)","progress":0.92656171321868896,"add_to_ceph_s":false},"c9df9a9d-66a6-47d3-b334-4774a23f2050":{"message":"Global Recovery Event (10s)\n [=====================.......] (remaining: 2s)","progress":0.77777779102325439,"add_to_ceph_s":true}}} 2023-07-02T14:07:20.855 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2023-07-02T14:07:20.856 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:07:20.865 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:20.866 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:07:20.866 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2023-07-02T14:07:20.866 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:20.877 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-07-02T14:07:20.899 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:20.899+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:21.000 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:20.999+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:21.042 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:21.043 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2023-07-02T14:07:21.043 INFO:teuthology.orchestra.run.smithi016.stdout: "complaint_time": 30, 2023-07-02T14:07:21.043 INFO:teuthology.orchestra.run.smithi016.stdout: "num_blocked_ops": 0 2023-07-02T14:07:21.043 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:21.054 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-07-02T14:07:21.100 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-07-02T14:07:21.099+0000 7fd936ee3700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83348) UID: 0 2023-07-02T14:07:21.133 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:07:21.171 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:21.173 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:07:21.185 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":385,"stamp":"2023-07-02T14:07:20.292528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":96,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":89620,"num_read_kb":15828021,"num_write":27382,"num_write_kb":11561441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3514,"num_bytes_recovered":11979257930,"num_keys_recovered":190,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27026,"ondisk_log_size":27026,"up":18,"acting":17,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":23208312,"kb_used_data":22864920,"kb_used_omap":229,"kb_used_meta":343130,"kb_avail":354279048,"statfs":{"total":386547056640,"available":362781745152,"internally_reserved":0,"allocated":23413678080,"data_stored":27281021064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":351365342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":472,"num_read_kb":37460,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.185540"},"pg_stats":[{"pgid":"2.7","version":"218'3701","reported_seq":9437,"reported_epoch":219,"state":"active+clean","last_fresh":"2023-07-02T14:07:16.109103+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:07:16.109103+0000","last_peered":"2023-07-02T14:07:16.109103+0000","last_clean":"2023-07-02T14:07:16.109103+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:07:16.109103+0000","last_undegraded":"2023-07-02T14:07:16.109103+0000","last_fullsized":"2023-07-02T14:07:16.109103+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3701,"ondisk_log_size":3701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1575041492,"num_objects":472,"num_object_clones":397,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":58519,"num_read_kb":2051790,"num_write":3143,"num_write_kb":1530741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"216'3069","reported_seq":5174,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-07-02T14:07:14.216006+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:07:14.216006+0000","last_peered":"2023-07-02T14:07:14.216006+0000","last_clean":"2023-07-02T14:07:14.216006+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:07:14.216006+0000","last_undegraded":"2023-07-02T14:07:14.216006+0000","last_fullsized":"2023-07-02T14:07:14.216006+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3069,"ondisk_log_size":3069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1538467250,"num_objects":486,"num_object_clones":415,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":1942,"num_read_kb":2141255,"num_write":2654,"num_write_kb":1492749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"218'3209","reported_seq":4731,"reported_epoch":220,"state":"active+clean","last_fresh":"2023-07-02T14:07:17.837519+0000","last_change":"2023-07-02T14:07:14.111389+0000","last_active":"2023-07-02T14:07:17.837519+0000","last_peered":"2023-07-02T14:07:17.837519+0000","last_clean":"2023-07-02T14:07:17.837519+0000","last_became_active":"2023-07-02T14:07:14.107764+0000","last_became_peered":"2023-07-02T14:07:14.107764+0000","last_unstale":"2023-07-02T14:07:17.837519+0000","last_undegraded":"2023-07-02T14:07:17.837519+0000","last_fullsized":"2023-07-02T14:07:17.837519+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3209,"ondisk_log_size":3209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1517142988,"num_objects":428,"num_object_clones":355,"num_object_copies":856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":676,"num_read_kb":1976206,"num_write":2850,"num_write_kb":1467898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":2369610862,"num_keys_recovered":8,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"216'6494","reported_seq":9005,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-07-02T14:07:14.216095+0000","last_change":"2023-07-02T14:06:29.572710+0000","last_active":"2023-07-02T14:07:14.216095+0000","last_peered":"2023-07-02T14:07:14.216095+0000","last_clean":"2023-07-02T14:07:14.216095+0000","last_became_active":"2023-07-02T14:06:29.569824+0000","last_became_peered":"2023-07-02T14:06:29.569824+0000","last_unstale":"2023-07-02T14:07:14.216095+0000","last_undegraded":"2023-07-02T14:07:14.216095+0000","last_fullsized":"2023-07-02T14:07:14.216095+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":6494,"ondisk_log_size":6494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1675535810,"num_objects":495,"num_object_clones":414,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":23025,"num_read_kb":2213857,"num_write":9605,"num_write_kb":1635009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":120,"reported_epoch":219,"state":"active+clean","last_fresh":"2023-07-02T14:07:16.109161+0000","last_change":"2023-07-02T14:07:02.103009+0000","last_active":"2023-07-02T14:07:16.109161+0000","last_peered":"2023-07-02T14:07:16.109161+0000","last_clean":"2023-07-02T14:07:16.109161+0000","last_became_active":"2023-07-02T14:07:02.102654+0000","last_became_peered":"2023-07-02T14:07:02.102654+0000","last_unstale":"2023-07-02T14:07:16.109161+0000","last_undegraded":"2023-07-02T14:07:16.109161+0000","last_fullsized":"2023-07-02T14:07:16.109161+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"216'2975","reported_seq":4499,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-07-02T14:07:14.232736+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:07:14.232736+0000","last_peered":"2023-07-02T14:07:14.232736+0000","last_clean":"2023-07-02T14:07:14.232736+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:07:14.232736+0000","last_undegraded":"2023-07-02T14:07:14.232736+0000","last_fullsized":"2023-07-02T14:07:14.232736+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2975,"ondisk_log_size":2975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1661289598,"num_objects":499,"num_object_clones":416,"num_object_copies":998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":1534,"num_read_kb":2096750,"num_write":2559,"num_write_kb":1611276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"216'2884","reported_seq":3770,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-07-02T14:07:14.233750+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:07:14.233750+0000","last_peered":"2023-07-02T14:07:14.233750+0000","last_clean":"2023-07-02T14:07:14.233750+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:07:14.233750+0000","last_undegraded":"2023-07-02T14:07:14.233750+0000","last_fullsized":"2023-07-02T14:07:14.233750+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2884,"ondisk_log_size":2884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1569543764,"num_objects":461,"num_object_clones":383,"num_object_copies":922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":966,"num_read_kb":2013799,"num_write":2501,"num_write_kb":1523310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"216'2328","reported_seq":5146,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-07-02T14:07:14.192116+0000","last_change":"2023-07-02T14:07:09.383558+0000","last_active":"2023-07-02T14:07:14.192116+0000","last_peered":"2023-07-02T14:07:14.192116+0000","last_clean":"2023-07-02T14:07:14.192116+0000","last_became_active":"2023-07-02T14:07:09.382190+0000","last_became_peered":"2023-07-02T14:07:09.382190+0000","last_unstale":"2023-07-02T14:07:14.192116+0000","last_undegraded":"2023-07-02T14:07:14.192116+0000","last_fullsized":"2023-07-02T14:07:14.192116+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2328,"ondisk_log_size":2328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1230645001,"num_objects":405,"num_object_clones":325,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":1910,"num_read_kb":1765337,"num_write":2003,"num_write_kb":1186825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":1578968698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"218'2366","reported_seq":3976,"reported_epoch":218,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-07-02T14:07:15.337452+0000","last_change":"2023-07-02T14:07:12.410957+0000","last_active":"2023-07-02T14:07:15.337452+0000","last_peered":"2023-07-02T14:07:15.337452+0000","last_clean":"2023-07-02T14:07:01.026611+0000","last_became_active":"2023-07-02T14:07:04.018152+0000","last_became_peered":"2023-07-02T14:07:04.018152+0000","last_unstale":"2023-07-02T14:07:15.337452+0000","last_undegraded":"2023-07-02T14:07:03.110294+0000","last_fullsized":"2023-07-02T14:07:03.103717+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2366,"ondisk_log_size":2366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1146537804,"num_objects":358,"num_object_clones":299,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":96,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":1048,"num_read_kb":1569027,"num_write":2067,"num_write_kb":1113633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":330,"num_bytes_recovered":1087738504,"num_keys_recovered":8,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":96},{"shards":"1","objects":254}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":96,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":89620,"num_read_kb":15828021,"num_write":27382,"num_write_kb":11561441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3514,"num_bytes_recovered":11979257930,"num_keys_recovered":190,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26498469888,"data_stored":30723944105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":27026,"ondisk_log_size":27026,"up":16,"acting":15,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640324,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640324,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6749764,"kb_used_data":6661564,"kb_used_omap":0,"kb_used_meta":88192,"kb_avail":87622076,"statfs":{"total":96636764160,"available":89725005824,"internally_reserved":0,"allocated":6821441536,"data_stored":7847685364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90308608},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.53700000000000003}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44,"15min":0.46400000000000002},"min":{"1min":0.28399999999999997,"5min":0.215,"15min":0.215},"max":{"1min":0.71499999999999997,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.45000000000000001},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.58299999999999996}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.32800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66800000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.54900000000000004}]},{"osd":3,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.38300000000000001},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45900000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.36799999999999999,"15min":0.36299999999999999},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.48799999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.435}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.44900000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.379},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.433,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59099999999999997,"5min":1.026,"15min":1.026},"last":0.41499999999999998}]}]},{"osd":3,"up_from":133,"seq":571230650396,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4569040,"kb_used_data":4498824,"kb_used_omap":229,"kb_used_meta":69978,"kb_avail":89802800,"statfs":{"total":96636764160,"available":91958067200,"internally_reserved":0,"allocated":4606795776,"data_stored":5357917289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":71657694},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.42699999999999999}]},{"osd":1,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.372,"15min":0.36599999999999999},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.374},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.46400000000000002}]},{"osd":2,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.28699999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.44900000000000001,"15min":0.439},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.47799999999999998}]},{"osd":4,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.35799999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.36199999999999999,"15min":0.35299999999999998},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.33800000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.39800000000000002},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.39500000000000002,"15min":0.38800000000000001},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.55300000000000005}]}]},{"osd":2,"up_from":11,"seq":47244640324,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3200368,"kb_used_data":3146280,"kb_used_omap":0,"kb_used_meta":54080,"kb_avail":91171472,"statfs":{"total":96636764160,"available":93359587328,"internally_reserved":0,"allocated":3221790720,"data_stored":3710342833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55377920},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":4.3650000000000002}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.42499999999999999,"15min":0.374},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":2.0720000000000001},{"interface":"front","average":{"1min":0.376,"5min":0.33500000000000002,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.66500000000000004}]},{"osd":3,"last update":"Sun Jul 2 14:07:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.91100000000000003,"5min":1.929,"15min":1.929},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.54800000000000004,"15min":0.55600000000000005},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93999999999999995,"5min":2,"15min":2},"last":0.55800000000000005}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.50900000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68400000000000005,"5min":1.071,"15min":1.071},"last":2.1230000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":0.57399999999999995}]}]},{"osd":0,"up_from":0,"seq":712964571154,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":11,"seq":47244640324,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8689140,"kb_used_data":8558252,"kb_used_omap":0,"kb_used_meta":130880,"kb_avail":85682700,"statfs":{"total":96636764160,"available":87739084800,"internally_reserved":0,"allocated":8763650048,"data_stored":10365075578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":134021120},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.318},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.26000000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.224,"5min":0.184,"15min":0.184},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.28499999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.49099999999999999,"15min":0.47299999999999998},"min":{"1min":0.30299999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.33400000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71899999999999997,"5min":1.524,"15min":1.524},"last":0.372}]},{"osd":5,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.80300000000000005,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84399999999999997,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.41899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3074248704,"data_stored":3409275605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8761016320,"data_stored":10364040347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3219156992,"data_stored":3709307602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4604137472,"data_stored":5356872530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6839910400,"data_stored":7884448021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:07:21.187 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:21.188 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 20, 2023-07-02T14:07:21.188 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 600, 2023-07-02T14:07:21.188 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [ 2023-07-02T14:07:21.188 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.188 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:968 2.7 2:f1a24948:::rbd_header.125de1449605:head [watch reconnect cookie 140544751726928 gen 6] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:07:21.188 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:07:21.188 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 260.07592765700002, 2023-07-02T14:07:21.189 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658, 2023-07-02T14:07:21.189 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:21.189 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:21.189 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:21.189 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:21.189 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:21.189 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 968 2023-07-02T14:07:21.189 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.190 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:21.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.190 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:21.190 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:07:21.190 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:21.190 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.190 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:21.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107675+0000", 2023-07-02T14:07:21.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:07:21.191 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.191 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.191 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:21.191 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107673+0000", 2023-07-02T14:07:21.191 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1540000000000008e-06 2023-07-02T14:07:21.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:21.192 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107682+0000", 2023-07-02T14:07:21.192 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8538000000000001e-05 2023-07-02T14:07:21.192 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.192 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.192 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:21.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107701+0000", 2023-07-02T14:07:21.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.0458999999999998e-05 2023-07-02T14:07:21.193 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.193 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.193 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.193 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107721+0000", 2023-07-02T14:07:21.193 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8122999999999999e-05 2023-07-02T14:07:21.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.194 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107779+0000", 2023-07-02T14:07:21.194 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3305000000000001e-05 2023-07-02T14:07:21.194 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.194 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.194 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:21.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107793+0000", 2023-07-02T14:07:21.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.0021328220000001 2023-07-02T14:07:21.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.195 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.195 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.195 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.109925+0000", 2023-07-02T14:07:21.195 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00023275499999999999 2023-07-02T14:07:21.195 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:21.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110158+0000", 2023-07-02T14:07:21.196 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018315149999999999 2023-07-02T14:07:21.196 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.196 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.196 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:21.196 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111990+0000", 2023-07-02T14:07:21.197 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.004314658 2023-07-02T14:07:21.197 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.197 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.197 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.197 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.197 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.197 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:969 2.7 2:f1a24948:::rbd_header.125de1449605:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected e55)", 2023-07-02T14:07:21.198 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:07:21.198 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 260.07586691900002, 2023-07-02T14:07:21.198 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009, 2023-07-02T14:07:21.198 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:21.198 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.198 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:21.198 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:21.198 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:21.199 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 969 2023-07-02T14:07:21.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.199 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:21.199 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.199 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:21.199 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:07:21.199 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:21.199 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:21.200 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107736+0000", 2023-07-02T14:07:21.200 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:21.200 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.200 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.200 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:21.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107735+0000", 2023-07-02T14:07:21.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1490000000000002e-06 2023-07-02T14:07:21.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.201 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.201 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:21.201 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107740+0000", 2023-07-02T14:07:21.201 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1419999999999993e-06 2023-07-02T14:07:21.201 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:21.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107748+0000", 2023-07-02T14:07:21.202 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.6980000000000001e-06 2023-07-02T14:07:21.202 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.202 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.202 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.202 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107755+0000", 2023-07-02T14:07:21.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1663999999999997e-05 2023-07-02T14:07:21.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.203 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.203 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107826+0000", 2023-07-02T14:07:21.203 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.8560000000000007e-06 2023-07-02T14:07:21.203 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.203 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:21.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:01.107834+0000", 2023-07-02T14:07:21.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.00313132 2023-07-02T14:07:21.204 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.204 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.204 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.204 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.110966+0000", 2023-07-02T14:07:21.204 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000106267 2023-07-02T14:07:21.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:21.205 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111072+0000", 2023-07-02T14:07:21.205 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00048328500000000002 2023-07-02T14:07:21.205 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.205 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.205 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:21.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.111555+0000", 2023-07-02T14:07:21.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00093633200000000001 2023-07-02T14:07:21.206 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.206 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.206 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:21.206 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112492+0000", 2023-07-02T14:07:21.206 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00024894899999999999 2023-07-02T14:07:21.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:21.207 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112741+0000", 2023-07-02T14:07:21.207 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8675999999999999e-05 2023-07-02T14:07:21.207 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.207 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.207 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:21.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112799+0000", 2023-07-02T14:07:21.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.7532e-05 2023-07-02T14:07:21.208 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.208 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.208 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:21.208 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:02.112827+0000", 2023-07-02T14:07:21.208 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.005091009 2023-07-02T14:07:21.208 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.209 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.209 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.209 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.209 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.209 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:2817 2.3 2:cfca42b1:::rbd_data.125de1449605.00000000000000f3:head [write 917504~28672] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e61)", 2023-07-02T14:07:21.209 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:07:21.209 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 249.84541258900001, 2023-07-02T14:07:21.210 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003, 2023-07-02T14:07:21.210 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:21.210 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.210 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:21.210 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:21.210 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:21.210 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 2817 2023-07-02T14:07:21.210 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.211 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:21.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:21.211 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:07:21.211 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:21.211 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.211 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.211 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:21.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338190+0000", 2023-07-02T14:07:21.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:21.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.212 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.212 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:21.212 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338189+0000", 2023-07-02T14:07:21.212 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6028e-05 2023-07-02T14:07:21.212 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:21.213 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338245+0000", 2023-07-02T14:07:21.213 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.1179999999999998e-06 2023-07-02T14:07:21.213 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.213 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.213 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:21.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338254+0000", 2023-07-02T14:07:21.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5369999999999997e-06 2023-07-02T14:07:21.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.214 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.214 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.214 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.338260+0000", 2023-07-02T14:07:21.214 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.030654608 2023-07-02T14:07:21.214 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.368914+0000", 2023-07-02T14:07:21.215 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000143559 2023-07-02T14:07:21.215 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.215 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.215 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:21.215 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369058+0000", 2023-07-02T14:07:21.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00015913200000000001 2023-07-02T14:07:21.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.216 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:21.216 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.369217+0000", 2023-07-02T14:07:21.216 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0029277420000000001 2023-07-02T14:07:21.216 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.216 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:21.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.372145+0000", 2023-07-02T14:07:21.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011933312 2023-07-02T14:07:21.217 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.217 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.217 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:21.217 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384078+0000", 2023-07-02T14:07:21.217 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1897999999999999e-05 2023-07-02T14:07:21.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:21.218 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384110+0000", 2023-07-02T14:07:21.218 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.2761000000000002e-05 2023-07-02T14:07:21.218 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.218 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.218 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:21.219 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:03:11.384203+0000", 2023-07-02T14:07:21.219 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.046012727000000003 2023-07-02T14:07:21.219 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.219 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.219 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.219 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.219 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.220 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9620 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 589824~65536] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:07:21.220 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:07:21.220 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 176.161397991, 2023-07-02T14:07:21.220 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001, 2023-07-02T14:07:21.220 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:21.220 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.220 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:21.220 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:21.221 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:21.221 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9620 2023-07-02T14:07:21.221 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.221 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:21.221 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.221 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:21.221 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:07:21.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:21.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.222 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:21.222 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022205+0000", 2023-07-02T14:07:21.222 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:07:21.222 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.222 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:21.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022203+0000", 2023-07-02T14:07:21.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.0346000000000001e-05 2023-07-02T14:07:21.223 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.223 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.223 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:21.223 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022263+0000", 2023-07-02T14:07:21.223 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4657999999999999e-05 2023-07-02T14:07:21.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:21.224 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022278+0000", 2023-07-02T14:07:21.224 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1840000000000002e-06 2023-07-02T14:07:21.224 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.224 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.224 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022285+0000", 2023-07-02T14:07:21.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.0096000000000003e-05 2023-07-02T14:07:21.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.225 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.225 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.225 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022335+0000", 2023-07-02T14:07:21.225 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2334000000000001e-05 2023-07-02T14:07:21.225 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:21.226 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022347+0000", 2023-07-02T14:07:21.226 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095653701999999993 2023-07-02T14:07:21.226 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.226 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.226 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118001+0000", 2023-07-02T14:07:21.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000185797 2023-07-02T14:07:21.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.227 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.227 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:21.227 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118187+0000", 2023-07-02T14:07:21.227 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00026397900000000001 2023-07-02T14:07:21.227 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:21.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.118451+0000", 2023-07-02T14:07:21.228 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0051832800000000002 2023-07-02T14:07:21.228 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.228 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.228 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:21.228 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.123634+0000", 2023-07-02T14:07:21.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.001227332 2023-07-02T14:07:21.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.229 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:21.229 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124861+0000", 2023-07-02T14:07:21.229 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9493999999999998e-05 2023-07-02T14:07:21.229 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.229 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:21.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124901+0000", 2023-07-02T14:07:21.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0462000000000001e-05 2023-07-02T14:07:21.230 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.230 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.230 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:21.230 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.124931+0000", 2023-07-02T14:07:21.230 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10272669500000001 2023-07-02T14:07:21.231 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.231 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.231 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.231 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.231 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.231 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:9621 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 655360~1966080] snapc 19=[19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e105)", 2023-07-02T14:07:21.231 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:07:21.232 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 176.16121060899999, 2023-07-02T14:07:21.232 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999, 2023-07-02T14:07:21.232 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:21.232 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.232 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:21.232 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:21.232 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:21.232 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 9621 2023-07-02T14:07:21.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.233 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:21.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.233 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:21.233 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:07:21.233 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:21.233 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.233 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:21.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022392+0000", 2023-07-02T14:07:21.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:21.234 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.234 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.234 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:21.234 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.022391+0000", 2023-07-02T14:07:21.234 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0032053939999999999 2023-07-02T14:07:21.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:21.235 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025597+0000", 2023-07-02T14:07:21.235 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.4307999999999999e-05 2023-07-02T14:07:21.235 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.235 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.235 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:21.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025611+0000", 2023-07-02T14:07:21.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.032e-06 2023-07-02T14:07:21.236 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.236 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.236 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.236 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025617+0000", 2023-07-02T14:07:21.236 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8977999999999998e-05 2023-07-02T14:07:21.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.237 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025656+0000", 2023-07-02T14:07:21.237 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1600000000000001e-06 2023-07-02T14:07:21.237 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.237 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.237 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for peered", 2023-07-02T14:07:21.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.025663+0000", 2023-07-02T14:07:21.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.095220739999999998 2023-07-02T14:07:21.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.238 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.238 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.238 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120884+0000", 2023-07-02T14:07:21.238 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.2525000000000001e-05 2023-07-02T14:07:21.238 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:21.239 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.120956+0000", 2023-07-02T14:07:21.239 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016232400000000001 2023-07-02T14:07:21.239 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.239 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.239 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:21.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.121119+0000", 2023-07-02T14:07:21.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0044007380000000004 2023-07-02T14:07:21.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.240 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.240 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:21.240 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.125519+0000", 2023-07-02T14:07:21.240 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000797209 2023-07-02T14:07:21.240 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:21.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126317+0000", 2023-07-02T14:07:21.241 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.1228000000000004e-05 2023-07-02T14:07:21.241 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.241 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.241 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:21.241 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126378+0000", 2023-07-02T14:07:21.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.1978000000000003e-05 2023-07-02T14:07:21.242 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.242 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.242 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:21.242 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:25.126430+0000", 2023-07-02T14:07:21.242 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.10403772999999999 2023-07-02T14:07:21.242 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.242 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.243 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.243 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.243 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.243 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5303.0:281 2.1 2:901d5a95:::rbd_data.125de1449605.0000000000000118:1b [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e110)", 2023-07-02T14:07:21.243 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:07:21.243 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 167.62813449800001, 2023-07-02T14:07:21.243 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002, 2023-07-02T14:07:21.243 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:21.244 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:21.244 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:21.244 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5303", 2023-07-02T14:07:21.244 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2482313815", 2023-07-02T14:07:21.244 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 281 2023-07-02T14:07:21.244 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.244 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:21.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:21.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:07:21.245 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:21.245 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.245 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.245 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:21.245 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:07:21.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:21.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.246 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:21.246 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555468+0000", 2023-07-02T14:07:21.246 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0469999999999998e-06 2023-07-02T14:07:21.246 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.246 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:21.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555471+0000", 2023-07-02T14:07:21.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8860000000000001e-06 2023-07-02T14:07:21.247 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.247 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.247 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:21.247 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555476+0000", 2023-07-02T14:07:21.247 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3160000000000002e-06 2023-07-02T14:07:21.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.248 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.555483+0000", 2023-07-02T14:07:21.248 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035989525000000001 2023-07-02T14:07:21.248 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.248 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.248 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591472+0000", 2023-07-02T14:07:21.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3708999999999996e-05 2023-07-02T14:07:21.249 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.249 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.249 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:21.249 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.591536+0000", 2023-07-02T14:07:21.249 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0055109720000000003 2023-07-02T14:07:21.250 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.250 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.250 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:21.250 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:33.597047+0000", 2023-07-02T14:07:21.250 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041578886000000002 2023-07-02T14:07:21.250 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.250 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.250 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.251 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.251 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.251 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:12985 2.7 2:eeddf24b:::rbd_data.125de1449605.000000000000007e:head [write 65536~4128768] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e125)", 2023-07-02T14:07:21.251 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:07:21.251 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 148.400607552, 2023-07-02T14:07:21.251 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999, 2023-07-02T14:07:21.251 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:21.251 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.252 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:21.252 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:21.252 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:21.252 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 12985 2023-07-02T14:07:21.252 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.252 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:21.252 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.252 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:21.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:07:21.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:21.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.253 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.253 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:21.253 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782995+0000", 2023-07-02T14:07:21.253 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:21.253 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:21.254 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.782994+0000", 2023-07-02T14:07:21.254 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.019254741999999998 2023-07-02T14:07:21.254 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.254 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.254 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:21.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802248+0000", 2023-07-02T14:07:21.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0451e-05 2023-07-02T14:07:21.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.255 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.255 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:21.255 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802279+0000", 2023-07-02T14:07:21.255 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3318e-05 2023-07-02T14:07:21.255 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802292+0000", 2023-07-02T14:07:21.256 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000109403 2023-07-02T14:07:21.256 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.256 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.256 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.256 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802402+0000", 2023-07-02T14:07:21.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012520599999999999 2023-07-02T14:07:21.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.257 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:21.257 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802527+0000", 2023-07-02T14:07:21.257 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000187768 2023-07-02T14:07:21.257 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.257 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 4", 2023-07-02T14:07:21.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.802715+0000", 2023-07-02T14:07:21.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.013192822 2023-07-02T14:07:21.258 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.258 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.258 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:21.258 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.815907+0000", 2023-07-02T14:07:21.258 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0086451870000000004 2023-07-02T14:07:21.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:21.259 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824553+0000", 2023-07-02T14:07:21.259 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6759999999999999e-05 2023-07-02T14:07:21.259 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.259 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.259 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:21.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824609+0000", 2023-07-02T14:07:21.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00011314700000000001 2023-07-02T14:07:21.260 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.260 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.260 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:21.260 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:52.824722+0000", 2023-07-02T14:07:21.260 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041727371999999999 2023-07-02T14:07:21.261 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.261 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.261 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.261 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.261 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.261 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:165 2.3 2:c9c6393a:::rbd_data.125de1449605.000000000000002c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:07:21.261 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:07:21.261 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 144.343824735, 2023-07-02T14:07:21.262 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002, 2023-07-02T14:07:21.262 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:21.262 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:21.262 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:21.262 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:07:21.262 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:07:21.262 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 165 2023-07-02T14:07:21.262 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.263 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:21.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:21.263 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:07:21.263 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:21.263 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.263 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.263 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:21.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839778+0000", 2023-07-02T14:07:21.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:21.264 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.264 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.264 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:21.264 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839777+0000", 2023-07-02T14:07:21.264 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6740000000000001e-06 2023-07-02T14:07:21.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:21.265 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839780+0000", 2023-07-02T14:07:21.265 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.9030000000000004e-06 2023-07-02T14:07:21.265 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.265 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.265 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:21.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839788+0000", 2023-07-02T14:07:21.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.1799999999999998e-06 2023-07-02T14:07:21.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.266 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.266 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.266 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.839792+0000", 2023-07-02T14:07:21.266 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.031592227000000001 2023-07-02T14:07:21.266 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871384+0000", 2023-07-02T14:07:21.267 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013600099999999999 2023-07-02T14:07:21.267 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.267 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.267 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:21.267 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.871520+0000", 2023-07-02T14:07:21.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0082817089999999999 2023-07-02T14:07:21.268 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.268 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.268 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:21.268 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:56.879802+0000", 2023-07-02T14:07:21.268 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040024268000000002 2023-07-02T14:07:21.268 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.269 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.269 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.269 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.269 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:252 2.7 2:f1d0abf0:::rbd_data.125de1449605.0000000000000083:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:07:21.269 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:07:21.269 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 143.92753976500001, 2023-07-02T14:07:21.269 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997, 2023-07-02T14:07:21.270 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:21.270 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:21.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:21.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:07:21.270 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:07:21.270 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 252 2023-07-02T14:07:21.270 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.270 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:21.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:21.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:07:21.271 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:21.271 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.271 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.271 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:21.271 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256063+0000", 2023-07-02T14:07:21.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:21.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.272 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.272 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:21.272 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256062+0000", 2023-07-02T14:07:21.272 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0149999999999999e-06 2023-07-02T14:07:21.272 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:21.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256065+0000", 2023-07-02T14:07:21.273 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.5269999999999998e-06 2023-07-02T14:07:21.273 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.273 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.273 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:21.273 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256071+0000", 2023-07-02T14:07:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.4309999999999998e-06 2023-07-02T14:07:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256074+0000", 2023-07-02T14:07:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.028852909 2023-07-02T14:07:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.274 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.284927+0000", 2023-07-02T14:07:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000168274 2023-07-02T14:07:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.285096+0000", 2023-07-02T14:07:21.275 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011036300000000001 2023-07-02T14:07:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296132+0000", 2023-07-02T14:07:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040069005999999997 2023-07-02T14:07:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.276 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.277 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.277 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.277 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:261 2.1 2:971731cb:::rbd_data.125de1449605.000000000000008c:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:07:21.277 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:07:21.277 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 143.92744429000001, 2023-07-02T14:07:21.277 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984, 2023-07-02T14:07:21.277 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:21.278 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:21.278 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:21.278 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:07:21.278 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:07:21.278 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 261 2023-07-02T14:07:21.278 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.278 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:21.278 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:21.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:07:21.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:21.279 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.279 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.279 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:21.279 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:07:21.279 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:21.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:21.280 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256158+0000", 2023-07-02T14:07:21.280 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.2819999999999999e-06 2023-07-02T14:07:21.280 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.280 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.280 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:21.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256160+0000", 2023-07-02T14:07:21.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2020000000000003e-06 2023-07-02T14:07:21.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.281 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.281 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:21.281 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256165+0000", 2023-07-02T14:07:21.281 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2100000000000003e-06 2023-07-02T14:07:21.281 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256170+0000", 2023-07-02T14:07:21.282 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.025871715999999999 2023-07-02T14:07:21.282 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.282 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.282 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.282 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282041+0000", 2023-07-02T14:07:21.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000132364 2023-07-02T14:07:21.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.283 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:21.283 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.282174+0000", 2023-07-02T14:07:21.283 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014044566 2023-07-02T14:07:21.283 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.283 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.284 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:21.284 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.296218+0000", 2023-07-02T14:07:21.284 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.040059984 2023-07-02T14:07:21.284 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.284 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.284 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.284 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.285 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.285 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:264 2.3 2:dbf06c9d:::rbd_data.125de1449605.000000000000008f:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:07:21.285 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:07:21.285 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 143.92741017200001, 2023-07-02T14:07:21.285 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003, 2023-07-02T14:07:21.285 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:21.285 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:21.285 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:21.286 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:07:21.286 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:07:21.286 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 264 2023-07-02T14:07:21.286 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.286 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:21.286 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.286 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:21.286 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:07:21.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:21.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.287 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:21.287 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:07:21.287 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:21.287 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.287 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:21.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256192+0000", 2023-07-02T14:07:21.288 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.5749999999999999e-06 2023-07-02T14:07:21.288 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.288 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.288 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:21.288 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256195+0000", 2023-07-02T14:07:21.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.327e-06 2023-07-02T14:07:21.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.289 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:21.289 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256200+0000", 2023-07-02T14:07:21.289 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.0580000000000002e-06 2023-07-02T14:07:21.289 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.289 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.256203+0000", 2023-07-02T14:07:21.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035707524999999997 2023-07-02T14:07:21.290 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.290 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.290 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.290 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.291911+0000", 2023-07-02T14:07:21.290 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00014269799999999999 2023-07-02T14:07:21.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:21.291 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.292053+0000", 2023-07-02T14:07:21.291 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0062760309999999996 2023-07-02T14:07:21.291 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.291 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.291 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:21.292 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.298329+0000", 2023-07-02T14:07:21.292 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.042136799000000003 2023-07-02T14:07:21.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.292 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.292 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.292 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.292 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.292 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.5465.0:341 2.3 2:cee258bd:::rbd_data.125de1449605.00000000000000dc:22 [sparse-read 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+read+known_if_redirected e128)", 2023-07-02T14:07:21.293 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:07:21.293 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 143.54692284500001, 2023-07-02T14:07:21.293 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997, 2023-07-02T14:07:21.293 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:21.293 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "started", 2023-07-02T14:07:21.293 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:21.293 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.5465", 2023-07-02T14:07:21.294 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/654906395", 2023-07-02T14:07:21.294 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 341 2023-07-02T14:07:21.294 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.294 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:21.294 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.294 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:21.294 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:07:21.294 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:21.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:21.295 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636680+0000", 2023-07-02T14:07:21.295 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:21.295 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.295 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.295 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:21.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636679+0000", 2023-07-02T14:07:21.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.6240000000000002e-06 2023-07-02T14:07:21.296 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.296 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.296 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:21.296 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636682+0000", 2023-07-02T14:07:21.296 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.6910000000000002e-06 2023-07-02T14:07:21.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:21.297 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636688+0000", 2023-07-02T14:07:21.297 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.3469999999999999e-06 2023-07-02T14:07:21.297 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.297 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.297 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.636691+0000", 2023-07-02T14:07:21.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.035625560000000001 2023-07-02T14:07:21.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.298 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.298 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.298 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672317+0000", 2023-07-02T14:07:21.298 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000108391 2023-07-02T14:07:21.298 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:21.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.672425+0000", 2023-07-02T14:07:21.299 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.012104976 2023-07-02T14:07:21.299 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.299 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.299 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:21.299 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:04:57.684530+0000", 2023-07-02T14:07:21.300 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.047850197999999997 2023-07-02T14:07:21.300 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.300 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.300 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.300 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.300 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.300 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "MOSDPGPushReply(2.7 139/134 [PushReplyOp(2:e3782f39:::rbd_data.125de1449605.0000000000000120:head)])", 2023-07-02T14:07:21.301 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:07:21.301 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 129.256390692, 2023-07-02T14:07:21.301 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998, 2023-07-02T14:07:21.301 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:21.301 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "reached pg", 2023-07-02T14:07:21.301 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:21.301 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.301 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:21.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:07:21.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:21.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.302 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.302 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:21.302 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927212+0000", 2023-07-02T14:07:21.302 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:21.302 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:21.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927211+0000", 2023-07-02T14:07:21.303 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.3689999999999998e-06 2023-07-02T14:07:21.303 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.303 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.303 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:21.303 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927217+0000", 2023-07-02T14:07:21.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.7269999999999996e-06 2023-07-02T14:07:21.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.304 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:21.304 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927225+0000", 2023-07-02T14:07:21.304 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.7940000000000002e-06 2023-07-02T14:07:21.304 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.304 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927230+0000", 2023-07-02T14:07:21.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5695999999999998e-05 2023-07-02T14:07:21.305 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.305 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.305 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.305 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.927266+0000", 2023-07-02T14:07:21.305 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.069972871000000006 2023-07-02T14:07:21.306 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.306 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.306 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:21.306 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:05:11.997239+0000", 2023-07-02T14:07:21.306 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.070026776999999998 2023-07-02T14:07:21.306 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.306 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.307 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.307 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.307 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.307 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19179 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2832, mlcod=177'2827)", 2023-07-02T14:07:21.307 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:07:21.307 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.724860862, 2023-07-02T14:07:21.307 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051928864999999998, 2023-07-02T14:07:21.307 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:21.308 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.308 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:21.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.308 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:21.308 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:07:21.308 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:21.308 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.308 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:21.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458742+0000", 2023-07-02T14:07:21.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:21.309 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.309 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.309 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:21.309 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.458741+0000", 2023-07-02T14:07:21.309 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.01233852 2023-07-02T14:07:21.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.310 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:21.310 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471079+0000", 2023-07-02T14:07:21.310 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8131e-05 2023-07-02T14:07:21.310 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.310 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:21.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471097+0000", 2023-07-02T14:07:21.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.19e-06 2023-07-02T14:07:21.311 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.311 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.311 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.311 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471104+0000", 2023-07-02T14:07:21.311 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.4463999999999999e-05 2023-07-02T14:07:21.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.312 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471158+0000", 2023-07-02T14:07:21.312 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.1723000000000001e-05 2023-07-02T14:07:21.312 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.312 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.312 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:21.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471190+0000", 2023-07-02T14:07:21.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039438622999999999 2023-07-02T14:07:21.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.313 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.313 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:21.313 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510628+0000", 2023-07-02T14:07:21.313 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.2280000000000002e-05 2023-07-02T14:07:21.313 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.314 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.314 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:21.314 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510671+0000", 2023-07-02T14:07:21.314 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.051928864999999998 2023-07-02T14:07:21.314 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.314 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.314 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.314 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.315 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.315 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19180 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2833, mlcod=177'2827)", 2023-07-02T14:07:21.315 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:07:21.315 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.712483621999993, 2023-07-02T14:07:21.315 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039574109000000003, 2023-07-02T14:07:21.315 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:21.315 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.316 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:21.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:21.316 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:07:21.316 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:21.316 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.316 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.316 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:21.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471119+0000", 2023-07-02T14:07:21.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:21.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.317 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.317 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:21.317 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471118+0000", 2023-07-02T14:07:21.317 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.4799000000000001e-05 2023-07-02T14:07:21.317 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:21.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471143+0000", 2023-07-02T14:07:21.318 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3960000000000003e-06 2023-07-02T14:07:21.318 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.318 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.318 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:21.318 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471149+0000", 2023-07-02T14:07:21.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.9230000000000004e-06 2023-07-02T14:07:21.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.319 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.319 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471153+0000", 2023-07-02T14:07:21.319 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015076050000000001 2023-07-02T14:07:21.319 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.319 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.486229+0000", 2023-07-02T14:07:21.320 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8466000000000002e-05 2023-07-02T14:07:21.320 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.320 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.320 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:21.320 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.486257+0000", 2023-07-02T14:07:21.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.024419020999999999 2023-07-02T14:07:21.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.321 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:21.321 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510676+0000", 2023-07-02T14:07:21.321 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.6693000000000001e-05 2023-07-02T14:07:21.321 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.321 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510693+0000", 2023-07-02T14:07:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039574109000000003 2023-07-02T14:07:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.322 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19181 2.7 e177/141 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head v 177'2834, mlcod=177'2827)", 2023-07-02T14:07:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:07:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.712442361000001, 2023-07-02T14:07:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039548481000000003, 2023-07-02T14:07:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:21.323 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:07:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:07:21.324 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:21.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.325 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:21.325 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471160+0000", 2023-07-02T14:07:21.325 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8320999999999999e-05 2023-07-02T14:07:21.325 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.325 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:21.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471188+0000", 2023-07-02T14:07:21.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.9669999999999998e-06 2023-07-02T14:07:21.326 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.326 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.326 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:21.326 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471194+0000", 2023-07-02T14:07:21.326 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.8709999999999999e-06 2023-07-02T14:07:21.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.327 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471198+0000", 2023-07-02T14:07:21.327 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.016760634999999999 2023-07-02T14:07:21.327 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.327 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.327 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.487959+0000", 2023-07-02T14:07:21.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 2.8398000000000002e-05 2023-07-02T14:07:21.328 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.328 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.328 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:21.328 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.487987+0000", 2023-07-02T14:07:21.328 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.022707627000000001 2023-07-02T14:07:21.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:21.329 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510695+0000", 2023-07-02T14:07:21.329 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3991999999999999e-05 2023-07-02T14:07:21.329 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.329 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.329 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:21.330 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510709+0000", 2023-07-02T14:07:21.330 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039548481000000003 2023-07-02T14:07:21.330 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.330 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.330 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.330 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.330 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.330 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_repop(client.4645.0:19192 2.7 e177/141 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head v 177'2835, mlcod=177'2827)", 2023-07-02T14:07:21.331 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:07:21.331 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 73.712399388999998, 2023-07-02T14:07:21.331 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039521186999999999, 2023-07-02T14:07:21.331 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:21.331 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.331 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:21.331 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.331 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:21.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:07:21.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:21.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.332 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.332 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:21.332 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:07:21.332 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:21.332 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.471203+0000", 2023-07-02T14:07:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.014525079 2023-07-02T14:07:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:21.333 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485728+0000", 2023-07-02T14:07:21.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.8372e-05 2023-07-02T14:07:21.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.334 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:21.334 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485746+0000", 2023-07-02T14:07:21.334 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6149999999999997e-06 2023-07-02T14:07:21.334 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.334 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.485755+0000", 2023-07-02T14:07:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0036949270000000002 2023-07-02T14:07:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.335 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489450+0000", 2023-07-02T14:07:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.9415e-05 2023-07-02T14:07:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.489469+0000", 2023-07-02T14:07:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021241488999999999 2023-07-02T14:07:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.336 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510711+0000", 2023-07-02T14:07:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.3604999999999999e-05 2023-07-02T14:07:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:07.510724+0000", 2023-07-02T14:07:21.337 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.039521186999999999 2023-07-02T14:07:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:20659 2.1 2:9edc58a7:::rbd_data.125de1449605.0000000000000102:head [write 2134016~94208] snapc 35=[35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e179)", 2023-07-02T14:07:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:07:21.338 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 69.593216006999995, 2023-07-02T14:07:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041695300999999997, 2023-07-02T14:07:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20659 2023-07-02T14:07:21.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.340 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:21.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.340 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:21.340 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:07:21.340 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:21.340 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.340 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590387+0000", 2023-07-02T14:07:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.999999 2023-07-02T14:07:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590386+0000", 2023-07-02T14:07:21.341 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.1415999999999994e-05 2023-07-02T14:07:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590467+0000", 2023-07-02T14:07:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.8029999999999999e-06 2023-07-02T14:07:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.342 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590474+0000", 2023-07-02T14:07:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5379999999999998e-06 2023-07-02T14:07:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590478+0000", 2023-07-02T14:07:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0073612449999999998 2023-07-02T14:07:21.343 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.344 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.597839+0000", 2023-07-02T14:07:21.344 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00013387800000000001 2023-07-02T14:07:21.344 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.344 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.344 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:21.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.597973+0000", 2023-07-02T14:07:21.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00022472799999999999 2023-07-02T14:07:21.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.345 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.345 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:21.345 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.598197+0000", 2023-07-02T14:07:21.345 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0065294259999999996 2023-07-02T14:07:21.345 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:21.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.604727+0000", 2023-07-02T14:07:21.346 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027239217999999999 2023-07-02T14:07:21.346 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.346 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.346 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:21.346 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.631966+0000", 2023-07-02T14:07:21.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 6.3194999999999994e-05 2023-07-02T14:07:21.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.347 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:21.347 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632029+0000", 2023-07-02T14:07:21.347 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.2639999999999997e-05 2023-07-02T14:07:21.347 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.347 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.348 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:21.348 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632082+0000", 2023-07-02T14:07:21.348 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041695300999999997 2023-07-02T14:07:21.348 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.348 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.348 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.348 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.348 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.349 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:20660 2.3 2:d298eaed:::rbd_data.125de1449605.0000000000000103:head [write 2031616~196608] snapc 35=[35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e179)", 2023-07-02T14:07:21.349 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:07:21.349 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 69.593120987000006, 2023-07-02T14:07:21.349 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041610001000000001, 2023-07-02T14:07:21.349 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:21.349 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.349 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:21.350 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:21.350 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:21.350 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 20660 2023-07-02T14:07:21.350 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.350 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:21.350 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.350 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:21.350 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:07:21.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:21.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.351 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:21.351 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590482+0000", 2023-07-02T14:07:21.351 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:21.351 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.351 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:21.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590481+0000", 2023-07-02T14:07:21.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00012730599999999999 2023-07-02T14:07:21.352 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.352 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.352 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:21.352 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590608+0000", 2023-07-02T14:07:21.352 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 8.6319999999999997e-06 2023-07-02T14:07:21.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:21.353 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590617+0000", 2023-07-02T14:07:21.353 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 3.5209999999999998e-06 2023-07-02T14:07:21.353 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.353 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.353 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.590621+0000", 2023-07-02T14:07:21.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.0018367430000000001 2023-07-02T14:07:21.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.354 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.354 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.354 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592457+0000", 2023-07-02T14:07:21.354 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00016640599999999999 2023-07-02T14:07:21.354 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:21.355 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592624+0000", 2023-07-02T14:07:21.355 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00028927400000000001 2023-07-02T14:07:21.355 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.355 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.355 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:21.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.592913+0000", 2023-07-02T14:07:21.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.011797813000000001 2023-07-02T14:07:21.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.356 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.356 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:21.356 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.604711+0000", 2023-07-02T14:07:21.356 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.027285997999999999 2023-07-02T14:07:21.356 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:21.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.631997+0000", 2023-07-02T14:07:21.357 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.8890000000000001e-05 2023-07-02T14:07:21.357 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.357 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.357 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:21.357 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632046+0000", 2023-07-02T14:07:21.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.5867e-05 2023-07-02T14:07:21.358 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.358 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.358 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:21.358 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:11.632092+0000", 2023-07-02T14:07:21.358 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.041610001000000001 2023-07-02T14:07:21.358 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.358 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.359 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.359 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.359 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.359 INFO:teuthology.orchestra.run.smithi016.stdout: "description": "osd_op(client.4645.0:23560 2.3 2:c69ae922:::rbd_data.125de1449605.00000000000001da:head [write 65536~4128768] snapc 42=[42,41,40,3f,3e,3d,3c,3b,3a,39,38,37,36,35,34,33,32,31,30,2f,2e,2d,2c,2b,2a,29,28,27,26,25,24,23,22,21,20,1f,1e,1d,1c,1b,1a,19,18,17,16,15,14,13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e206)", 2023-07-02T14:07:21.359 INFO:teuthology.orchestra.run.smithi016.stdout: "initiated_at": "2023-07-02T14:06:57.307513+0000", 2023-07-02T14:07:21.359 INFO:teuthology.orchestra.run.smithi016.stdout: "age": 23.876089616000002, 2023-07-02T14:07:21.359 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.049163971000000001, 2023-07-02T14:07:21.360 INFO:teuthology.orchestra.run.smithi016.stdout: "type_data": { 2023-07-02T14:07:21.360 INFO:teuthology.orchestra.run.smithi016.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.360 INFO:teuthology.orchestra.run.smithi016.stdout: "client_info": { 2023-07-02T14:07:21.360 INFO:teuthology.orchestra.run.smithi016.stdout: "client": "client.4645", 2023-07-02T14:07:21.360 INFO:teuthology.orchestra.run.smithi016.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:21.360 INFO:teuthology.orchestra.run.smithi016.stdout: "tid": 23560 2023-07-02T14:07:21.360 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.360 INFO:teuthology.orchestra.run.smithi016.stdout: "events": [ 2023-07-02T14:07:21.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "initiated", 2023-07-02T14:07:21.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.307513+0000", 2023-07-02T14:07:21.361 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0 2023-07-02T14:07:21.361 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.361 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.361 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "throttled", 2023-07-02T14:07:21.361 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.307513+0000", 2023-07-02T14:07:21.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4294967295.9999981 2023-07-02T14:07:21.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.362 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "header_read", 2023-07-02T14:07:21.362 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.307511+0000", 2023-07-02T14:07:21.362 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.010796256000000001 2023-07-02T14:07:21.362 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.362 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "all_read", 2023-07-02T14:07:21.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318307+0000", 2023-07-02T14:07:21.363 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 1.2152e-05 2023-07-02T14:07:21.363 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.363 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.363 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "dispatched", 2023-07-02T14:07:21.363 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318319+0000", 2023-07-02T14:07:21.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 5.8710000000000002e-06 2023-07-02T14:07:21.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.364 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.364 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318325+0000", 2023-07-02T14:07:21.364 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.00065064300000000001 2023-07-02T14:07:21.364 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.364 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "reached_pg", 2023-07-02T14:07:21.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.318976+0000", 2023-07-02T14:07:21.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 9.4927999999999996e-05 2023-07-02T14:07:21.365 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.365 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.365 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "started", 2023-07-02T14:07:21.365 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.319071+0000", 2023-07-02T14:07:21.365 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.000184487 2023-07-02T14:07:21.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "waiting for subops from 1", 2023-07-02T14:07:21.366 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.319255+0000", 2023-07-02T14:07:21.366 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.021455622000000001 2023-07-02T14:07:21.366 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.366 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.366 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "op_commit", 2023-07-02T14:07:21.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.340711+0000", 2023-07-02T14:07:21.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.015850748000000001 2023-07-02T14:07:21.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.367 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.367 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "sub_op_commit_rec", 2023-07-02T14:07:21.367 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.356562+0000", 2023-07-02T14:07:21.367 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 7.1957000000000002e-05 2023-07-02T14:07:21.367 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "commit_sent", 2023-07-02T14:07:21.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.356634+0000", 2023-07-02T14:07:21.368 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 4.3409999999999999e-05 2023-07-02T14:07:21.368 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:21.368 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:21.368 INFO:teuthology.orchestra.run.smithi016.stdout: "event": "done", 2023-07-02T14:07:21.368 INFO:teuthology.orchestra.run.smithi016.stdout: "time": "2023-07-02T14:06:57.356677+0000", 2023-07-02T14:07:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: "duration": 0.049163971000000001 2023-07-02T14:07:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:21.369 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:21.369 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:21.371 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-07-02T14:07:21.372 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:21.369+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:21.438 INFO:tasks.workunit.client.0.smithi163.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-07-02T14:07:21.442 INFO:tasks.workunit.client.0.smithi163.stderr:+ dst_sum=772cfd861b9de6313f9663b1d0f9b297 2023-07-02T14:07:21.442 INFO:tasks.workunit.client.0.smithi163.stderr:+ [[ 772cfd861b9de6313f9663b1d0f9b297 != \7\7\2\c\f\d\8\6\1\b\9\d\e\6\3\1\3\f\9\6\6\3\b\1\d\0\f\9\b\2\9\7 ]] 2023-07-02T14:07:21.443 INFO:tasks.workunit.client.0.smithi163.stderr:+ check_object_map b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap1 2023-07-02T14:07:21.443 INFO:tasks.workunit.client.0.smithi163.stderr:+ local spec=b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap1 2023-07-02T14:07:21.443 INFO:tasks.workunit.client.0.smithi163.stderr:+ rbd object-map check b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap1 2023-07-02T14:07:21.471 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:21.470+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:21.491 INFO:tasks.workunit.client.0.smithi163.stderr: Object Map Check: 1% complete... Object Map Check: 2% complete... Object Map Check: 3% complete... Object Map Check: 4% complete... Object Map Check: 5% complete... Object Map Check: 6% complete... Object Map Check: 7% complete... Object Map Check: 8% complete...2023-07-02T14:07:21.490+0000 7f9718b74700 -1 librbd::ObjectMapIterateRequest: object map error: object rbd_data.125de1449605.0000000000000022 marked as 0, but should be 1 2023-07-02T14:07:21.491 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:07:21.490+0000 7f9719375700 -1 librbd::ObjectMapIterateRequest: object map error: object rbd_data.125de1449605.0000000000000021 marked as 0, but should be 1 2023-07-02T14:07:21.491 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:07:21.490+0000 7f9718b74700 -1 librbd::ObjectMapIterateRequest: object map error: object rbd_data.125de1449605.0000000000000023 marked as 0, but should be 1 2023-07-02T14:07:21.491 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:07:21.490+0000 7f9718b74700 -1 librbd::ObjectMapIterateRequest: object map error: object rbd_data.125de1449605.0000000000000024 marked as 0, but should be 1 2023-07-02T14:07:21.492 INFO:tasks.workunit.client.0.smithi163.stderr: Object Map Check: 9% complete...2023-07-02T14:07:21.490+0000 7f9718b74700 -1 librbd::ObjectMapIterateRequest: object map error: object rbd_data.125de1449605.0000000000000025 marked as 0, but should be 1 2023-07-02T14:07:21.492 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:07:21.490+0000 7f9718b74700 -1 librbd::ObjectMapIterateRequest: object map error: object rbd_data.125de1449605.0000000000000026 marked as 0, but should be 1 2023-07-02T14:07:21.492 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:07:21.490+0000 7f9718b74700 -1 librbd::ObjectMapIterateRequest: object map error: object rbd_data.125de1449605.0000000000000027 marked as 0, but should be 1 2023-07-02T14:07:21.493 INFO:tasks.workunit.client.0.smithi163.stderr: Object Map Check: 10% complete... Object Map Check: 11% complete... Object Map Check: 12% complete... Object Map Check: 13% complete... Object Map Check: 14% complete... Object Map Check: 15% complete... Object Map Check: 16% complete...2023-07-02T14:07:21.492+0000 7f971a377700 0 -- 172.21.15.163:0/1247239943 >> [v2:172.21.15.16:6816/83361,v1:172.21.15.16:6817/83361] conn(0x7f96fc01c740 msgr2=0x7f96fc01eb30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:07:21.494 INFO:tasks.workunit.client.0.smithi163.stderr: Object Map Check: 17% complete... Object Map Check: 18% complete... Object Map Check: 19% complete...2023-07-02T14:07:21.493+0000 7f971b379700 0 -- 172.21.15.163:0/1247239943 >> [v2:172.21.15.16:6808/83360,v1:172.21.15.16:6809/83360] conn(0x564f22398020 msgr2=0x564f2239a410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-07-02T14:07:21.516 INFO:tasks.workunit.client.0.smithi163.stderr: Object Map Check: 20% complete... Object Map Check: 21% complete... Object Map Check: 22% complete... Object Map Check: 23% complete... Object Map Check: 24% complete... Object Map Check: 25% complete... Object Map Check: 26% complete... Object Map Check: 27% complete... Object Map Check: 28% complete... Object Map Check: 29% complete... Object Map Check: 30% complete... Object Map Check: 31% complete... Object Map Check: 32% complete... Object Map Check: 33% complete... Object Map Check: 34% complete... Object Map Check: 35% complete... Object Map Check: 36% complete... Object Map Check: 37% complete... Object Map Check: 38% complete... Object Map Check: 39% complete... Object Map Check: 40% complete... Object Map Check: 41% complete... Object Map Check: 42% complete... Object Map Check: 43% complete... Object Map Check: 44% complete... Object Map Check: 45% complete... Object Map Check: 46% complete... Object Map Check: 47% complete... Object Map Check: 48% complete... Object Map Check: 49% complete... Object Map Check: 50% complete... Object Map Check: 51% complete... Object Map Check: 52% complete... Object Map Check: 53% complete... Object Map Check: 54% complete... Object Map Check: 55% complete... Object Map Check: 56% complete... Object Map Check: 57% complete... Object Map Check: 58% complete... Object Map Check: 59% complete... Object Map Check: 60% complete... Object Map Check: 61% complete... Object Map Check: 62% complete... Object Map Check: 63% complete... Object Map Check: 64% complete... Object Map Check: 65% complete... Object Map Check: 66% complete... Object Map Check: 67% complete... Object Map Check: 68% complete... Object Map Check: 69% complete... Object Map Check: 70% complete... Object Map Check: 71% complete... Object Map Check: 72% complete... Object Map Check: 73% complete... Object Map Check: 74% complete... Object Map Check: 75% complete... Object Map Check: 76% complete... Object Map Check: 77% complete... Object Map Check: 78% complete... Object Map Check: 79% complete... Object Map Check: 80% complete... Object Map Check: 81% complete... Object Map Check: 82% complete... Object Map Check: 83% complete... Object Map Check: 84% complete... Object Map Check: 85% complete... Object Map Check: 86% complete... Object Map Check: 87% complete... Object Map Check: 88% complete... Object Map Check: 89% complete... Object Map Check: 90% complete... Object Map Check: 91% complete... Object Map Check: 92% complete... Object Map Check: 93% complete... Object Map Check: 94% complete... Object Map Check: 95% complete... Object Map Check: 96% complete... Object Map Check: 97% complete... Object Map Check: 98% complete... Object Map Check: 99% complete...2023-07-02T14:07:21.515+0000 7f9718b74700 -1 librbd::object_map::InvalidateRequest: 0x7f96fc016ec0 invalidating object map in-memory 2023-07-02T14:07:21.516 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:07:21.515+0000 7f9718b74700 -1 librbd::object_map::InvalidateRequest: 0x7f96fc016ec0 invalidating object map on-disk 2023-07-02T14:07:21.517 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:21.517 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:07:21.517 INFO:teuthology.orchestra.run.smithi022.stdout: "num_ops": 0 2023-07-02T14:07:21.517 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:21.520 INFO:tasks.workunit.client.0.smithi163.stderr:2023-07-02T14:07:21.519+0000 7f9719375700 -1 librbd::object_map::InvalidateRequest: 0x7f96fc016ec0 should_complete: r=0 2023-07-02T14:07:21.520 INFO:tasks.workunit.client.0.smithi163.stderr: Object Map Check: 100% complete...done. 2023-07-02T14:07:21.527 INFO:tasks.workunit.client.0.smithi163.stderr:+ local flags 2023-07-02T14:07:21.527 INFO:tasks.workunit.client.0.smithi163.stderr:++ rbd info b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap1 2023-07-02T14:07:21.527 INFO:tasks.workunit.client.0.smithi163.stderr:++ grep 'flags: ' 2023-07-02T14:07:21.528 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-07-02T14:07:21.571 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-07-02T14:07:21.571+0000 7fb390b79700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 99972) UID: 0 2023-07-02T14:07:21.573 INFO:tasks.workunit.client.0.smithi163.stderr:+ flags=' flags: object map invalid, fast diff invalid' 2023-07-02T14:07:21.574 INFO:tasks.workunit.client.0.smithi163.stderr:+ [[ flags: object map invalid, fast diff invalid =~ object map invalid ]] 2023-07-02T14:07:21.574 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'Object map invalid at b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap1' 2023-07-02T14:07:21.574 INFO:tasks.workunit.client.0.smithi163.stdout:Object map invalid at b3fbb028-e267-4d25-a750-93ba18a2316a-src@snap1 2023-07-02T14:07:21.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T14:07:21.575 INFO:tasks.workunit.client.0.smithi163.stderr:+ exit 1 2023-07-02T14:07:21.575 INFO:tasks.workunit:Stopping ['rbd/diff_continuous.sh'] on client.0... 2023-07-02T14:07:21.576 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-07-02T14:07:21.652 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:21.653 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [], 2023-07-02T14:07:21.653 INFO:teuthology.orchestra.run.smithi022.stdout: "complaint_time": 30, 2023-07-02T14:07:21.653 INFO:teuthology.orchestra.run.smithi022.stdout: "num_blocked_ops": 0 2023-07-02T14:07:21.653 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:21.663 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-07-02T14:07:21.672 INFO:tasks.ceph.osd.2.smithi016.stderr:2023-07-02T14:07:21.672+0000 7f0830ee5700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83353) UID: 0 2023-07-02T14:07:21.751 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-07-02T14:07:21.772 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-07-02T14:07:21.772+0000 7fee83c23700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83344) UID: 0 2023-07-02T14:07:21.829 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-07-02T14:07:21.829 INFO:teuthology.orchestra.run.smithi022.stdout: "size": 20, 2023-07-02T14:07:21.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 600, 2023-07-02T14:07:21.830 INFO:teuthology.orchestra.run.smithi022.stdout: "ops": [ 2023-07-02T14:07:21.830 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.830 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6116 2.0 e81/46 2:0d6700ee:::rbd_data.125de1449605.000000000000008e:head v 81'1321, mlcod=81'1318)", 2023-07-02T14:07:21.830 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:07:21.830 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 221.93025254700001, 2023-07-02T14:07:21.830 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998, 2023-07-02T14:07:21.831 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:21.831 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.831 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:21.831 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.831 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:21.831 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:07:21.831 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:21.831 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:21.832 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897614+0000", 2023-07-02T14:07:21.832 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:21.832 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.832 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.832 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:21.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.897613+0000", 2023-07-02T14:07:21.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003722951 2023-07-02T14:07:21.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.833 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.833 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:21.833 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901336+0000", 2023-07-02T14:07:21.833 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1466000000000001e-05 2023-07-02T14:07:21.833 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:21.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901358+0000", 2023-07-02T14:07:21.834 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.0589999999999992e-06 2023-07-02T14:07:21.834 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.834 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.834 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.834 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901367+0000", 2023-07-02T14:07:21.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4468999999999998e-05 2023-07-02T14:07:21.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.835 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.835 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:21.835 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901401+0000", 2023-07-02T14:07:21.835 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7628000000000001e-05 2023-07-02T14:07:21.835 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:21.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901419+0000", 2023-07-02T14:07:21.836 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022360628 2023-07-02T14:07:21.836 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.836 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.836 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:21.836 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923779+0000", 2023-07-02T14:07:21.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.2165000000000001e-05 2023-07-02T14:07:21.837 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.837 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.837 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:21.837 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923812+0000", 2023-07-02T14:07:21.837 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026197460999999998 2023-07-02T14:07:21.837 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.837 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.838 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.838 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.838 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.838 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6112 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1322, mlcod=81'1319)", 2023-07-02T14:07:21.838 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:07:21.838 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 221.92616682900001, 2023-07-02T14:07:21.838 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163, 2023-07-02T14:07:21.838 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:21.839 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.839 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:21.839 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.839 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:21.839 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:07:21.839 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:21.839 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:21.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901700+0000", 2023-07-02T14:07:21.840 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:21.840 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.840 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.840 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:21.840 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901699+0000", 2023-07-02T14:07:21.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0330000000000003e-06 2023-07-02T14:07:21.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.841 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:21.841 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901706+0000", 2023-07-02T14:07:21.841 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2568999999999999e-05 2023-07-02T14:07:21.841 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.841 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:21.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901719+0000", 2023-07-02T14:07:21.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.2659999999999998e-06 2023-07-02T14:07:21.842 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.842 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.842 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.842 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.901725+0000", 2023-07-02T14:07:21.842 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0067987660000000004 2023-07-02T14:07:21.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:21.843 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908524+0000", 2023-07-02T14:07:21.843 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.9617000000000001e-05 2023-07-02T14:07:21.843 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.843 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.843 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:21.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.908553+0000", 2023-07-02T14:07:21.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015263794000000001 2023-07-02T14:07:21.844 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.844 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.844 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:21.844 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923817+0000", 2023-07-02T14:07:21.844 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8983000000000001e-05 2023-07-02T14:07:21.845 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.845 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.845 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:21.845 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.923836+0000", 2023-07-02T14:07:21.845 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022136163 2023-07-02T14:07:21.845 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.845 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.845 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.846 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.846 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.846 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6120 2.0 e81/46 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 81'1324, mlcod=81'1319)", 2023-07-02T14:07:21.846 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:07:21.846 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 221.92230204800001, 2023-07-02T14:07:21.846 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999, 2023-07-02T14:07:21.846 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:21.846 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.847 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:21.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:21.847 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:07:21.847 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:21.847 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.847 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.847 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:21.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905565+0000", 2023-07-02T14:07:21.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:21.848 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.848 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.848 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:21.848 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905563+0000", 2023-07-02T14:07:21.848 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7516999999999997e-05 2023-07-02T14:07:21.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:21.849 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905641+0000", 2023-07-02T14:07:21.849 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0177999999999999e-05 2023-07-02T14:07:21.849 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.849 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.849 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:21.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905661+0000", 2023-07-02T14:07:21.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.5290000000000008e-06 2023-07-02T14:07:21.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.850 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.850 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.850 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.905669+0000", 2023-07-02T14:07:21.850 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0038243869999999998 2023-07-02T14:07:21.850 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:21.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909494+0000", 2023-07-02T14:07:21.851 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6642999999999999e-05 2023-07-02T14:07:21.851 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.851 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.851 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:21.851 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.909520+0000", 2023-07-02T14:07:21.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018535471000000001 2023-07-02T14:07:21.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.852 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:21.852 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928056+0000", 2023-07-02T14:07:21.852 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5486e-05 2023-07-02T14:07:21.852 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.852 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.853 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:21.853 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.928081+0000", 2023-07-02T14:07:21.853 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022516551999999999 2023-07-02T14:07:21.853 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.853 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.853 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.853 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.853 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.854 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6143 2.0 e81/46 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 81'1340, mlcod=81'1333)", 2023-07-02T14:07:21.854 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:07:21.854 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 221.88307298800001, 2023-07-02T14:07:21.854 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557, 2023-07-02T14:07:21.854 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:21.854 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.855 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:21.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.855 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:21.855 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:07:21.855 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:21.855 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.855 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:21.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944794+0000", 2023-07-02T14:07:21.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:21.856 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.856 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.856 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:21.856 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.944793+0000", 2023-07-02T14:07:21.856 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045151979999999998 2023-07-02T14:07:21.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.857 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:21.857 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949308+0000", 2023-07-02T14:07:21.857 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0859000000000001e-05 2023-07-02T14:07:21.857 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.857 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:21.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949319+0000", 2023-07-02T14:07:21.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.0509999999999999e-06 2023-07-02T14:07:21.858 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.858 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.858 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.858 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949326+0000", 2023-07-02T14:07:21.858 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00066007499999999998 2023-07-02T14:07:21.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:21.859 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949986+0000", 2023-07-02T14:07:21.859 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3963000000000001e-05 2023-07-02T14:07:21.859 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.859 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.859 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:21.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950010+0000", 2023-07-02T14:07:21.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022982216999999999 2023-07-02T14:07:21.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.860 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.860 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:21.860 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.972992+0000", 2023-07-02T14:07:21.860 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8858999999999999e-05 2023-07-02T14:07:21.860 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.861 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.861 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:21.861 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973021+0000", 2023-07-02T14:07:21.861 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028227557 2023-07-02T14:07:21.861 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.861 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.861 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.861 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.862 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.862 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6144 2.0 e81/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 81'1341, mlcod=81'1333)", 2023-07-02T14:07:21.862 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:07:21.862 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 221.87841563000001, 2023-07-02T14:07:21.862 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001, 2023-07-02T14:07:21.862 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:21.862 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.863 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:21.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:21.863 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:07:21.863 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:21.863 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.863 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.863 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:21.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949451+0000", 2023-07-02T14:07:21.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:21.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.864 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.864 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:21.864 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949450+0000", 2023-07-02T14:07:21.864 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.8719999999999997e-06 2023-07-02T14:07:21.864 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:21.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949460+0000", 2023-07-02T14:07:21.865 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.456e-06 2023-07-02T14:07:21.865 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.865 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.865 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:21.865 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949468+0000", 2023-07-02T14:07:21.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5119999999999999e-06 2023-07-02T14:07:21.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.866 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.866 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.866 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.949474+0000", 2023-07-02T14:07:21.866 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0057549510000000003 2023-07-02T14:07:21.866 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.867 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:21.867 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955229+0000", 2023-07-02T14:07:21.867 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.9765000000000002e-05 2023-07-02T14:07:21.867 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.867 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:21.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955279+0000", 2023-07-02T14:07:21.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017749141 2023-07-02T14:07:21.868 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.868 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.868 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:21.868 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973028+0000", 2023-07-02T14:07:21.868 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.8927e-05 2023-07-02T14:07:21.869 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.869 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.869 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:21.869 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973047+0000", 2023-07-02T14:07:21.869 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023595699000000001 2023-07-02T14:07:21.869 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.869 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.869 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.870 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.870 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.870 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6145 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1342, mlcod=81'1333)", 2023-07-02T14:07:21.870 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:07:21.870 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 221.87747576699999, 2023-07-02T14:07:21.870 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999, 2023-07-02T14:07:21.870 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:21.871 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.871 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:21.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.871 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:21.871 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:07:21.871 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:21.871 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.871 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:21.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950391+0000", 2023-07-02T14:07:21.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:21.872 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.872 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.872 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:21.872 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950390+0000", 2023-07-02T14:07:21.872 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2221999999999999e-05 2023-07-02T14:07:21.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:21.873 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950402+0000", 2023-07-02T14:07:21.873 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7186000000000001e-05 2023-07-02T14:07:21.873 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.873 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.873 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:21.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950419+0000", 2023-07-02T14:07:21.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.8930000000000002e-06 2023-07-02T14:07:21.874 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.874 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.874 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.874 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.950425+0000", 2023-07-02T14:07:21.874 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0061607500000000004 2023-07-02T14:07:21.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:21.875 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956586+0000", 2023-07-02T14:07:21.875 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9009e-05 2023-07-02T14:07:21.875 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.875 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.875 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:21.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956605+0000", 2023-07-02T14:07:21.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016443284999999998 2023-07-02T14:07:21.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.876 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.876 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:21.876 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973048+0000", 2023-07-02T14:07:21.876 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.9539999999999999e-05 2023-07-02T14:07:21.876 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.877 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.877 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:21.877 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973068+0000", 2023-07-02T14:07:21.877 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022676776999999999 2023-07-02T14:07:21.877 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.877 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.877 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.877 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.878 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.878 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:6147 2.0 e81/46 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 81'1343, mlcod=81'1335)", 2023-07-02T14:07:21.878 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:07:21.878 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 221.87607788899999, 2023-07-02T14:07:21.878 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999, 2023-07-02T14:07:21.878 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:21.878 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.879 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:21.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:21.879 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:07:21.879 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:21.879 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.879 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.879 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:21.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951789+0000", 2023-07-02T14:07:21.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:21.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.880 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.880 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:21.880 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.951788+0000", 2023-07-02T14:07:21.880 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0041975620000000002 2023-07-02T14:07:21.880 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.881 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:21.881 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955985+0000", 2023-07-02T14:07:21.881 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4154e-05 2023-07-02T14:07:21.881 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.881 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:21.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.955999+0000", 2023-07-02T14:07:21.882 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.4440000000000002e-06 2023-07-02T14:07:21.882 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.882 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.882 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.882 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.956007+0000", 2023-07-02T14:07:21.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.001924037 2023-07-02T14:07:21.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.883 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.883 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:21.883 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957931+0000", 2023-07-02T14:07:21.883 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6246e-05 2023-07-02T14:07:21.883 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:21.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.957947+0000", 2023-07-02T14:07:21.884 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015760618000000001 2023-07-02T14:07:21.884 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.884 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.884 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:21.884 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973708+0000", 2023-07-02T14:07:21.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.0800000000000003e-05 2023-07-02T14:07:21.885 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.885 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.885 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:21.885 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:03:39.973738+0000", 2023-07-02T14:07:21.885 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021949600999999999 2023-07-02T14:07:21.885 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.885 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.886 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.886 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.886 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.886 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:7097 2.0 e91/46 2:1c72c23d:::rbd_object_map.125de1449605:head v 91'1585, mlcod=91'1583)", 2023-07-02T14:07:21.886 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:07:21.886 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 200.465887393, 2023-07-02T14:07:21.886 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001, 2023-07-02T14:07:21.887 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:21.887 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.887 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:21.887 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.887 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:21.887 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:07:21.887 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:21.887 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:21.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361979+0000", 2023-07-02T14:07:21.888 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:21.888 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.888 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.888 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:21.888 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361978+0000", 2023-07-02T14:07:21.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.3559999999999993e-06 2023-07-02T14:07:21.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.889 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:21.889 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361987+0000", 2023-07-02T14:07:21.889 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0049e-05 2023-07-02T14:07:21.889 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.889 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:21.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.361997+0000", 2023-07-02T14:07:21.890 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.7949999999999998e-06 2023-07-02T14:07:21.890 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.890 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.890 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.890 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362001+0000", 2023-07-02T14:07:21.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.7425999999999997e-05 2023-07-02T14:07:21.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.891 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:21.891 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362039+0000", 2023-07-02T14:07:21.891 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2762000000000001e-05 2023-07-02T14:07:21.891 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.891 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:21.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.362052+0000", 2023-07-02T14:07:21.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021870686 2023-07-02T14:07:21.892 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.892 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.892 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:21.892 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383922+0000", 2023-07-02T14:07:21.892 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6533000000000003e-05 2023-07-02T14:07:21.893 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.893 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.893 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:21.893 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:01.383969+0000", 2023-07-02T14:07:21.893 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021989496000000001 2023-07-02T14:07:21.893 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.894 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.894 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.894 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.894 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.894 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7705 2.7 2:e39f56ba:::rbd_data.125de1449605.000000000000004b:head [write 53248~3092480] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:07:21.894 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:07:21.894 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 199.099537709, 2023-07-02T14:07:21.895 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002, 2023-07-02T14:07:21.895 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:21.895 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.895 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:21.895 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:21.895 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:21.895 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7705 2023-07-02T14:07:21.895 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.896 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:21.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:21.896 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:07:21.896 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:21.896 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.896 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.896 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:21.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728329+0000", 2023-07-02T14:07:21.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:21.897 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.897 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.897 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:21.897 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.728328+0000", 2023-07-02T14:07:21.897 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.003892099 2023-07-02T14:07:21.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:21.898 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732220+0000", 2023-07-02T14:07:21.898 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.5648999999999999e-05 2023-07-02T14:07:21.898 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.898 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.898 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:21.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732236+0000", 2023-07-02T14:07:21.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.7419999999999997e-06 2023-07-02T14:07:21.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.899 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.899 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.899 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732243+0000", 2023-07-02T14:07:21.899 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.8813000000000002e-05 2023-07-02T14:07:21.899 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:21.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732292+0000", 2023-07-02T14:07:21.900 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000103218 2023-07-02T14:07:21.900 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.900 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.900 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:21.900 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732396+0000", 2023-07-02T14:07:21.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018997714999999998 2023-07-02T14:07:21.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.901 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.901 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:21.901 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751393+0000", 2023-07-02T14:07:21.901 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7219999999999997e-05 2023-07-02T14:07:21.901 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:21.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751460+0000", 2023-07-02T14:07:21.902 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000109981 2023-07-02T14:07:21.902 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.902 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.902 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:21.902 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751570+0000", 2023-07-02T14:07:21.903 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023241421000000002 2023-07-02T14:07:21.903 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.903 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.903 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.903 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.903 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.903 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7708 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 65536~3289088] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:07:21.903 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:07:21.904 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 199.09549036499999, 2023-07-02T14:07:21.904 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001, 2023-07-02T14:07:21.904 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:21.904 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.904 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:21.904 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:21.904 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:21.904 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7708 2023-07-02T14:07:21.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.905 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:21.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.905 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:21.905 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:07:21.905 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:21.905 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.905 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:21.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:07:21.906 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:21.906 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.906 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.906 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:21.906 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.732376+0000", 2023-07-02T14:07:21.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0045273209999999999 2023-07-02T14:07:21.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.907 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:21.907 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736903+0000", 2023-07-02T14:07:21.907 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0429e-05 2023-07-02T14:07:21.907 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.907 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:21.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736923+0000", 2023-07-02T14:07:21.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.2509999999999997e-06 2023-07-02T14:07:21.908 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.908 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.908 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.908 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736931+0000", 2023-07-02T14:07:21.908 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0051924279999999998 2023-07-02T14:07:21.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:21.909 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742123+0000", 2023-07-02T14:07:21.909 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00012258500000000001 2023-07-02T14:07:21.909 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.909 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.909 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:21.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742246+0000", 2023-07-02T14:07:21.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.017098573999999998 2023-07-02T14:07:21.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.910 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.910 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:21.910 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759344+0000", 2023-07-02T14:07:21.910 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.9638000000000001e-05 2023-07-02T14:07:21.910 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:21.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759404+0000", 2023-07-02T14:07:21.911 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.1966999999999995e-05 2023-07-02T14:07:21.911 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.911 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.911 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:21.911 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759496+0000", 2023-07-02T14:07:21.912 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027119500000000001 2023-07-02T14:07:21.912 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.912 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.912 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.912 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.912 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.912 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7709 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3358720~49152] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:07:21.913 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:07:21.913 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 199.09092654, 2023-07-02T14:07:21.913 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002, 2023-07-02T14:07:21.913 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:21.913 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.913 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:21.913 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:21.913 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:21.914 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7709 2023-07-02T14:07:21.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.914 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:21.914 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.914 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:21.914 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:07:21.914 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:21.914 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:21.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736940+0000", 2023-07-02T14:07:21.915 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:21.915 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.915 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.915 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:21.915 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736939+0000", 2023-07-02T14:07:21.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.1350000000000003e-05 2023-07-02T14:07:21.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.916 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:21.916 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736970+0000", 2023-07-02T14:07:21.916 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.3579999999999992e-06 2023-07-02T14:07:21.916 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.916 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:21.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736979+0000", 2023-07-02T14:07:21.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.1769999999999997e-06 2023-07-02T14:07:21.917 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.917 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.917 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.917 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736984+0000", 2023-07-02T14:07:21.917 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.012207638 2023-07-02T14:07:21.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.918 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:21.918 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749192+0000", 2023-07-02T14:07:21.918 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000116069 2023-07-02T14:07:21.918 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.918 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:21.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.749308+0000", 2023-07-02T14:07:21.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.010201979 2023-07-02T14:07:21.919 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.919 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.919 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:21.919 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759510+0000", 2023-07-02T14:07:21.919 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0501000000000003e-05 2023-07-02T14:07:21.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:21.920 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759561+0000", 2023-07-02T14:07:21.920 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 7.9029999999999994e-05 2023-07-02T14:07:21.920 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.920 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.920 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:21.921 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.759640+0000", 2023-07-02T14:07:21.921 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022699545000000002 2023-07-02T14:07:21.921 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.921 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.921 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.921 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.921 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.921 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7724 2.7 2:e95aa193:::rbd_data.125de1449605.000000000000004c:head [write 3407872~65536] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:07:21.922 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:07:21.922 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 199.09087667399999, 2023-07-02T14:07:21.922 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001, 2023-07-02T14:07:21.922 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:21.922 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.922 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:21.922 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:21.923 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:21.923 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7724 2023-07-02T14:07:21.923 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.923 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:21.923 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.923 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:21.923 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:07:21.923 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:21.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:21.924 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736990+0000", 2023-07-02T14:07:21.924 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999995 2023-07-02T14:07:21.924 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.924 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.924 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:21.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.736989+0000", 2023-07-02T14:07:21.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.6499000000000002e-05 2023-07-02T14:07:21.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.925 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.925 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:21.925 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737016+0000", 2023-07-02T14:07:21.925 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8120000000000002e-06 2023-07-02T14:07:21.925 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:21.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737023+0000", 2023-07-02T14:07:21.926 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.6959999999999996e-06 2023-07-02T14:07:21.926 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.926 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.926 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.926 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.737027+0000", 2023-07-02T14:07:21.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013417836000000001 2023-07-02T14:07:21.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.927 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:21.927 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750445+0000", 2023-07-02T14:07:21.927 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.4290999999999996e-05 2023-07-02T14:07:21.927 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.927 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:21.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.750539+0000", 2023-07-02T14:07:21.928 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015093530000000001 2023-07-02T14:07:21.928 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.928 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.928 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:21.928 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765633+0000", 2023-07-02T14:07:21.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5950000000000004e-05 2023-07-02T14:07:21.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.929 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:21.929 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765699+0000", 2023-07-02T14:07:21.929 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 9.789e-05 2023-07-02T14:07:21.929 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.929 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.930 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:21.930 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765797+0000", 2023-07-02T14:07:21.930 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.028806818000000001 2023-07-02T14:07:21.930 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.930 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.930 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.930 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.930 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.931 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_op(client.4645.0:7729 2.7 2:ff9ef6de:::rbd_data.125de1449605.000000000000004d:head [write 53248~3878912] snapc 13=[13,12,11,10,f,e,d,c,b,a,9,8,7,6,5,4] ondisk+write+known_if_redirected e92)", 2023-07-02T14:07:21.931 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:07:21.931 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 199.08983768100001, 2023-07-02T14:07:21.931 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244, 2023-07-02T14:07:21.931 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:21.931 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.931 INFO:teuthology.orchestra.run.smithi022.stdout: "client_info": { 2023-07-02T14:07:21.932 INFO:teuthology.orchestra.run.smithi022.stdout: "client": "client.4645", 2023-07-02T14:07:21.932 INFO:teuthology.orchestra.run.smithi022.stdout: "client_addr": "172.21.15.163:0/2238656381", 2023-07-02T14:07:21.932 INFO:teuthology.orchestra.run.smithi022.stdout: "tid": 7729 2023-07-02T14:07:21.932 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.932 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:21.932 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.932 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:21.932 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:07:21.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:21.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.933 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:21.933 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738029+0000", 2023-07-02T14:07:21.933 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:21.933 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.933 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:21.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.738028+0000", 2023-07-02T14:07:21.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0048793919999999998 2023-07-02T14:07:21.934 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.934 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.934 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:21.934 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742907+0000", 2023-07-02T14:07:21.934 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.7516e-05 2023-07-02T14:07:21.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.935 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:21.935 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742925+0000", 2023-07-02T14:07:21.935 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.055e-06 2023-07-02T14:07:21.935 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.935 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.742933+0000", 2023-07-02T14:07:21.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0090549919999999996 2023-07-02T14:07:21.936 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.936 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.936 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:21.936 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.751988+0000", 2023-07-02T14:07:21.936 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.000106656 2023-07-02T14:07:21.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:21.937 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.752095+0000", 2023-07-02T14:07:21.937 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.013715237999999999 2023-07-02T14:07:21.937 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.937 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.937 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "op_commit", 2023-07-02T14:07:21.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765810+0000", 2023-07-02T14:07:21.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3233999999999999e-05 2023-07-02T14:07:21.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.938 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.938 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:21.938 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765863+0000", 2023-07-02T14:07:21.938 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.6271000000000001e-05 2023-07-02T14:07:21.938 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.939 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.939 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:21.939 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:04:02.765949+0000", 2023-07-02T14:07:21.939 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027920244 2023-07-02T14:07:21.939 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.939 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.939 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.939 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.940 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.940 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22312 2.0 e197/193 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 197'5590, mlcod=197'5582)", 2023-07-02T14:07:21.940 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:07:21.940 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 47.079392982000002, 2023-07-02T14:07:21.940 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021943201999999998, 2023-07-02T14:07:21.940 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:21.940 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.941 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:21.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:21.941 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:07:21.941 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:21.941 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.941 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.941 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:21.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748474+0000", 2023-07-02T14:07:21.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:21.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.942 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.942 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:21.942 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.748473+0000", 2023-07-02T14:07:21.942 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00099631099999999994 2023-07-02T14:07:21.942 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:21.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749469+0000", 2023-07-02T14:07:21.943 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.4279e-05 2023-07-02T14:07:21.943 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.943 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.943 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:21.943 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749483+0000", 2023-07-02T14:07:21.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.5289999999999997e-06 2023-07-02T14:07:21.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.944 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.944 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.944 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.749490+0000", 2023-07-02T14:07:21.944 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0047909650000000003 2023-07-02T14:07:21.944 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:21.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.754281+0000", 2023-07-02T14:07:21.945 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.1999000000000001e-05 2023-07-02T14:07:21.945 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.945 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.945 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:21.945 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.754303+0000", 2023-07-02T14:07:21.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016063747 2023-07-02T14:07:21.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.946 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:21.946 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.770367+0000", 2023-07-02T14:07:21.946 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0231000000000001e-05 2023-07-02T14:07:21.946 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.946 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.947 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:21.947 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.770417+0000", 2023-07-02T14:07:21.947 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.021943201999999998 2023-07-02T14:07:21.947 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.947 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.947 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.947 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.947 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.948 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22313 2.0 e197/193 2:1df58950:::rbd_data.125de1449605.0000000000000098:head v 197'5592, mlcod=197'5587)", 2023-07-02T14:07:21.948 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:07:21.948 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 47.074139578999997, 2023-07-02T14:07:21.948 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026016164000000001, 2023-07-02T14:07:21.948 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:21.948 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.948 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:21.948 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:21.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:07:21.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:21.949 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.949 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.949 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:21.949 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753727+0000", 2023-07-02T14:07:21.949 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999933 2023-07-02T14:07:21.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:21.950 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753720+0000", 2023-07-02T14:07:21.950 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.4851999999999995e-05 2023-07-02T14:07:21.950 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.950 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.950 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:21.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753785+0000", 2023-07-02T14:07:21.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.0216e-05 2023-07-02T14:07:21.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.951 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.951 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:21.951 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753796+0000", 2023-07-02T14:07:21.951 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.4249999999999997e-06 2023-07-02T14:07:21.951 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.952 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.753801+0000", 2023-07-02T14:07:21.952 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0070274359999999998 2023-07-02T14:07:21.952 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.952 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.952 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:21.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760828+0000", 2023-07-02T14:07:21.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.0975000000000002e-05 2023-07-02T14:07:21.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.953 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.953 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:21.953 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760849+0000", 2023-07-02T14:07:21.953 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.018855916 2023-07-02T14:07:21.953 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.954 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:21.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.779705+0000", 2023-07-02T14:07:21.954 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.8037999999999997e-05 2023-07-02T14:07:21.954 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.954 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.954 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:21.954 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.779743+0000", 2023-07-02T14:07:21.955 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.026016164000000001 2023-07-02T14:07:21.955 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.955 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.955 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.955 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.955 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.955 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22314 2.0 e197/193 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 197'5594, mlcod=197'5587)", 2023-07-02T14:07:21.955 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:07:21.956 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 47.071534925000002, 2023-07-02T14:07:21.956 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027004523999999999, 2023-07-02T14:07:21.956 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:21.956 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.956 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:21.956 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.956 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:21.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:07:21.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:21.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.957 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.957 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:21.957 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756332+0000", 2023-07-02T14:07:21.957 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:21.957 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:21.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756331+0000", 2023-07-02T14:07:21.958 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.4697999999999999e-05 2023-07-02T14:07:21.958 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.958 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.958 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:21.958 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756365+0000", 2023-07-02T14:07:21.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3902e-05 2023-07-02T14:07:21.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.959 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:21.959 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756379+0000", 2023-07-02T14:07:21.959 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.7329999999999997e-06 2023-07-02T14:07:21.959 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.959 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.756386+0000", 2023-07-02T14:07:21.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0062861929999999998 2023-07-02T14:07:21.960 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.960 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.960 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:21.960 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.762672+0000", 2023-07-02T14:07:21.960 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.9031e-05 2023-07-02T14:07:21.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:21.961 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.762711+0000", 2023-07-02T14:07:21.961 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.020589275000000001 2023-07-02T14:07:21.961 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.961 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.961 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:21.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783301+0000", 2023-07-02T14:07:21.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 3.5720999999999999e-05 2023-07-02T14:07:21.962 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.962 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.962 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:21.962 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783336+0000", 2023-07-02T14:07:21.962 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.027004523999999999 2023-07-02T14:07:21.962 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.963 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.963 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.963 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.963 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.963 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22315 2.0 e197/193 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 197'5595, mlcod=197'5587)", 2023-07-02T14:07:21.963 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:07:21.963 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 47.067565653999999, 2023-07-02T14:07:21.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023064345999999999, 2023-07-02T14:07:21.964 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:21.964 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.964 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:21.964 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.964 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:21.964 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:07:21.964 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:21.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:21.965 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760301+0000", 2023-07-02T14:07:21.965 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:21.965 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.965 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.965 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:21.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760300+0000", 2023-07-02T14:07:21.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.3056000000000002e-05 2023-07-02T14:07:21.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.966 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.966 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:21.966 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760353+0000", 2023-07-02T14:07:21.966 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.2825e-05 2023-07-02T14:07:21.966 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:21.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760366+0000", 2023-07-02T14:07:21.967 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.8290000000000002e-06 2023-07-02T14:07:21.967 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.967 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.967 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.967 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.760373+0000", 2023-07-02T14:07:21.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0075982089999999999 2023-07-02T14:07:21.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.968 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:21.968 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.767971+0000", 2023-07-02T14:07:21.968 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4096000000000001e-05 2023-07-02T14:07:21.968 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.968 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:21.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.767995+0000", 2023-07-02T14:07:21.969 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.015345908 2023-07-02T14:07:21.969 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.969 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.969 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:21.969 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783341+0000", 2023-07-02T14:07:21.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.4273e-05 2023-07-02T14:07:21.970 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.970 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.970 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:21.970 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.783365+0000", 2023-07-02T14:07:21.970 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.023064345999999999 2023-07-02T14:07:21.970 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.970 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.971 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.971 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.971 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.971 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22319 2.0 e197/193 2:1782b6aa:::rbd_data.125de1449605.000000000000009b:head v 197'5597, mlcod=197'5587)", 2023-07-02T14:07:21.971 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:07:21.971 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 47.064062833999998, 2023-07-02T14:07:21.971 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022862501, 2023-07-02T14:07:21.971 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:21.972 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.972 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:21.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.972 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:21.972 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:07:21.972 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:21.972 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.972 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:21.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763804+0000", 2023-07-02T14:07:21.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.999999 2023-07-02T14:07:21.973 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.973 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.973 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:21.973 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.763803+0000", 2023-07-02T14:07:21.973 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0052067220000000004 2023-07-02T14:07:21.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:21.974 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769010+0000", 2023-07-02T14:07:21.974 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.3249e-05 2023-07-02T14:07:21.974 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.974 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.974 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:21.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769023+0000", 2023-07-02T14:07:21.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 8.1790000000000008e-06 2023-07-02T14:07:21.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.975 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.975 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.975 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.769031+0000", 2023-07-02T14:07:21.975 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0030611079999999999 2023-07-02T14:07:21.975 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:21.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.772092+0000", 2023-07-02T14:07:21.976 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.3958e-05 2023-07-02T14:07:21.976 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.976 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.976 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:21.976 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.772116+0000", 2023-07-02T14:07:21.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014504064000000001 2023-07-02T14:07:21.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.977 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:21.977 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.786620+0000", 2023-07-02T14:07:21.977 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4.5952999999999999e-05 2023-07-02T14:07:21.977 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.977 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.978 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:21.978 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.786666+0000", 2023-07-02T14:07:21.978 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022862501 2023-07-02T14:07:21.978 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.978 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.978 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.978 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.979 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.979 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22335 2.0 e197/193 2:0dba0d84:::rbd_data.125de1449605.000000000000009d:head v 197'5602, mlcod=197'5599)", 2023-07-02T14:07:21.979 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:07:21.979 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 47.036992556999998, 2023-07-02T14:07:21.979 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024437743000000001, 2023-07-02T14:07:21.979 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:21.979 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.979 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:21.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:21.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:07:21.980 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:21.980 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.980 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.980 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:21.980 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790874+0000", 2023-07-02T14:07:21.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:21.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.981 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:21.981 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.790873+0000", 2023-07-02T14:07:21.981 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0081672399999999992 2023-07-02T14:07:21.981 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.981 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:21.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799040+0000", 2023-07-02T14:07:21.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.5712999999999999e-05 2023-07-02T14:07:21.982 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.982 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.982 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:21.982 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799066+0000", 2023-07-02T14:07:21.982 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.1413999999999999e-05 2023-07-02T14:07:21.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.983 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799077+0000", 2023-07-02T14:07:21.983 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.3102000000000007e-05 2023-07-02T14:07:21.983 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.983 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.983 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:21.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799140+0000", 2023-07-02T14:07:21.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.883e-05 2023-07-02T14:07:21.984 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.984 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.984 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:21.984 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799169+0000", 2023-07-02T14:07:21.984 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.016086700999999998 2023-07-02T14:07:21.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:21.985 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.815256+0000", 2023-07-02T14:07:21.985 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.5967000000000001e-05 2023-07-02T14:07:21.985 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.985 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.985 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:21.986 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.815312+0000", 2023-07-02T14:07:21.986 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.024437743000000001 2023-07-02T14:07:21.986 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.986 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.986 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.986 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.986 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.986 INFO:teuthology.orchestra.run.smithi022.stdout: "description": "osd_repop(client.4645.0:22336 2.0 e197/193 2:0167a29b:::rbd_data.125de1449605.000000000000009a:head v 197'5603, mlcod=197'5599)", 2023-07-02T14:07:21.987 INFO:teuthology.orchestra.run.smithi022.stdout: "initiated_at": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:07:21.987 INFO:teuthology.orchestra.run.smithi022.stdout: "age": 47.028757271000003, 2023-07-02T14:07:21.987 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022713011000000002, 2023-07-02T14:07:21.987 INFO:teuthology.orchestra.run.smithi022.stdout: "type_data": { 2023-07-02T14:07:21.987 INFO:teuthology.orchestra.run.smithi022.stdout: "flag_point": "commit sent; apply or cleanup", 2023-07-02T14:07:21.987 INFO:teuthology.orchestra.run.smithi022.stdout: "events": [ 2023-07-02T14:07:21.987 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.987 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "initiated", 2023-07-02T14:07:21.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:07:21.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0 2023-07-02T14:07:21.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.988 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.988 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "throttled", 2023-07-02T14:07:21.988 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799109+0000", 2023-07-02T14:07:21.988 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 4294967295.9999986 2023-07-02T14:07:21.988 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "header_read", 2023-07-02T14:07:21.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799108+0000", 2023-07-02T14:07:21.989 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.00058943600000000004 2023-07-02T14:07:21.989 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.989 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.989 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "all_read", 2023-07-02T14:07:21.989 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799698+0000", 2023-07-02T14:07:21.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 1.6205000000000002e-05 2023-07-02T14:07:21.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.990 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "dispatched", 2023-07-02T14:07:21.990 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799714+0000", 2023-07-02T14:07:21.990 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 6.9630000000000001e-06 2023-07-02T14:07:21.990 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.990 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "queued_for_pg", 2023-07-02T14:07:21.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.799721+0000", 2023-07-02T14:07:21.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.0076846579999999996 2023-07-02T14:07:21.991 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.991 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.991 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "reached_pg", 2023-07-02T14:07:21.991 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.807405+0000", 2023-07-02T14:07:21.991 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 2.8191000000000002e-05 2023-07-02T14:07:21.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.992 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "started", 2023-07-02T14:07:21.992 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.807434+0000", 2023-07-02T14:07:21.992 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.014338020999999999 2023-07-02T14:07:21.992 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.992 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "commit_sent", 2023-07-02T14:07:21.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.821772+0000", 2023-07-02T14:07:21.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 5.0868999999999997e-05 2023-07-02T14:07:21.993 INFO:teuthology.orchestra.run.smithi022.stdout: }, 2023-07-02T14:07:21.993 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-07-02T14:07:21.993 INFO:teuthology.orchestra.run.smithi022.stdout: "event": "done", 2023-07-02T14:07:21.993 INFO:teuthology.orchestra.run.smithi022.stdout: "time": "2023-07-02T14:06:34.821822+0000", 2023-07-02T14:07:21.993 INFO:teuthology.orchestra.run.smithi022.stdout: "duration": 0.022713011000000002 2023-07-02T14:07:21.994 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.994 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.994 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.994 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-07-02T14:07:21.994 INFO:teuthology.orchestra.run.smithi022.stdout: ] 2023-07-02T14:07:21.994 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-07-02T14:07:21.996 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0: {} 2023-07-02T14:07:21.996 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1: {} 2023-07-02T14:07:21.996 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2: {} 2023-07-02T14:07:21.997 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3: {} 2023-07-02T14:07:21.997 INFO:teuthology.orchestra.run.smithi016.stdout:osd.4: {} 2023-07-02T14:07:21.997 INFO:teuthology.orchestra.run.smithi016.stdout:osd.5: {} 2023-07-02T14:07:21.997 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-07-02T14:07:21.998 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_idryomov_ceph_cbd88633a9c19779ad4b2e367bdc17f83b3a065d/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_idryomov_ceph_cbd88633a9c19779ad4b2e367bdc17f83b3a065d/qa/tasks/workunit.py", line 423, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/diff_continuous.sh) on smithi163 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cbd88633a9c19779ad4b2e367bdc17f83b3a065d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' 2023-07-02T14:07:22.066 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=2e2789d5d9d14447b6d04f6a14556966 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_idryomov_ceph_cbd88633a9c19779ad4b2e367bdc17f83b3a065d/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_idryomov_ceph_cbd88633a9c19779ad4b2e367bdc17f83b3a065d/qa/tasks/workunit.py", line 423, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/diff_continuous.sh) on smithi163 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cbd88633a9c19779ad4b2e367bdc17f83b3a065d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh' 2023-07-02T14:07:22.069 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2023-07-02T14:07:22.082 INFO:tasks.thrashosds:joining thrashosds 2023-07-02T14:07:22.087 INFO:teuthology.orchestra.run.smithi016.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-07-02T14:07:22.087 INFO:teuthology.orchestra.run.smithi016.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-07-02T14:07:22.087 INFO:teuthology.orchestra.run.smithi016.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-07-02T14:07:22.088 INFO:teuthology.orchestra.run.smithi016.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-07-02T14:07:22.088 INFO:teuthology.orchestra.run.smithi016.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-07-02T14:07:22.088 INFO:teuthology.orchestra.run.smithi016.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-07-02T14:07:22.089 INFO:tasks.ceph.osd.1.smithi016.stderr:2023-07-02T14:07:21.994+0000 7fee776f6700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83351) UID: 0 2023-07-02T14:07:22.128 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:07:22.214 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:22.215 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:22.225 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-07-02T14:07:22.348 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:22.349 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:22.359 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-07-02T14:07:22.482 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:22.482 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:22.492 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-07-02T14:07:22.617 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:22.617 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:22.627 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-07-02T14:07:22.751 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:22.751 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:22.761 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-07-02T14:07:22.885 INFO:teuthology.orchestra.run.smithi022.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:22.885 INFO:teuthology.orchestra.run.smithi022.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:22.895 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-07-02T14:07:23.022 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:23.022 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:23.032 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-07-02T14:07:23.158 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:23.158 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:23.168 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-07-02T14:07:23.298 INFO:teuthology.orchestra.run.smithi016.stdout:ERROR: (22) Invalid argument 2023-07-02T14:07:23.298 INFO:teuthology.orchestra.run.smithi016.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-07-02T14:07:24.141 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-07-02T14:07:24.187 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:07:24.521 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:24.523 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:07:24.534 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2023-07-02T14:07:24.293272+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90630,"num_read_kb":15880603,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3618,"num_bytes_recovered":12343797996,"num_keys_recovered":190,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27030,"ondisk_log_size":27030,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":23594956,"kb_used_data":23245228,"kb_used_omap":229,"kb_used_meta":349466,"kb_avail":353892404,"statfs":{"total":386547056640,"available":362385821696,"internally_reserved":0,"allocated":23803113472,"data_stored":27731750779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":357853406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-8,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1482,"num_read_kb":90042,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":364540066,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.193635"},"pg_stats":[{"pgid":"2.7","version":"222'3704","reported_seq":9528,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-07-02T14:07:21.570504+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:07:21.570504+0000","last_peered":"2023-07-02T14:07:21.570504+0000","last_clean":"2023-07-02T14:07:21.570504+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:07:21.570504+0000","last_undegraded":"2023-07-02T14:07:21.570504+0000","last_fullsized":"2023-07-02T14:07:21.570504+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3704,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1575041492,"num_objects":472,"num_object_clones":397,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":59417,"num_read_kb":2052552,"num_write":3144,"num_write_kb":1530742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"216'3069","reported_seq":5184,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-07-02T14:07:18.355050+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:07:18.355050+0000","last_peered":"2023-07-02T14:07:18.355050+0000","last_clean":"2023-07-02T14:07:18.355050+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:07:18.355050+0000","last_undegraded":"2023-07-02T14:07:18.355050+0000","last_fullsized":"2023-07-02T14:07:18.355050+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3069,"ondisk_log_size":3069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1538467250,"num_objects":486,"num_object_clones":415,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":1948,"num_read_kb":2154143,"num_write":2654,"num_write_kb":1492749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"221'3210","reported_seq":4801,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-07-02T14:07:21.516569+0000","last_change":"2023-07-02T14:07:14.111389+0000","last_active":"2023-07-02T14:07:21.516569+0000","last_peered":"2023-07-02T14:07:21.516569+0000","last_clean":"2023-07-02T14:07:21.516569+0000","last_became_active":"2023-07-02T14:07:14.107764+0000","last_became_peered":"2023-07-02T14:07:14.107764+0000","last_unstale":"2023-07-02T14:07:21.516569+0000","last_undegraded":"2023-07-02T14:07:21.516569+0000","last_fullsized":"2023-07-02T14:07:21.516569+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3210,"ondisk_log_size":3210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1517142988,"num_objects":428,"num_object_clones":355,"num_object_copies":856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":743,"num_read_kb":1976610,"num_write":2851,"num_write_kb":1467899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":2369610862,"num_keys_recovered":8,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"216'6494","reported_seq":9016,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-07-02T14:07:18.355075+0000","last_change":"2023-07-02T14:06:29.572710+0000","last_active":"2023-07-02T14:07:18.355075+0000","last_peered":"2023-07-02T14:07:18.355075+0000","last_clean":"2023-07-02T14:07:18.355075+0000","last_became_active":"2023-07-02T14:06:29.569824+0000","last_became_peered":"2023-07-02T14:06:29.569824+0000","last_unstale":"2023-07-02T14:07:18.355075+0000","last_undegraded":"2023-07-02T14:07:18.355075+0000","last_fullsized":"2023-07-02T14:07:18.355075+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":6494,"ondisk_log_size":6494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1675535810,"num_objects":495,"num_object_clones":414,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":23032,"num_read_kb":2218337,"num_write":9605,"num_write_kb":1635009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":123,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-07-02T14:07:19.160455+0000","last_change":"2023-07-02T14:07:02.103009+0000","last_active":"2023-07-02T14:07:19.160455+0000","last_peered":"2023-07-02T14:07:19.160455+0000","last_clean":"2023-07-02T14:07:19.160455+0000","last_became_active":"2023-07-02T14:07:02.102654+0000","last_became_peered":"2023-07-02T14:07:02.102654+0000","last_unstale":"2023-07-02T14:07:19.160455+0000","last_undegraded":"2023-07-02T14:07:19.160455+0000","last_fullsized":"2023-07-02T14:07:19.160455+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"216'2975","reported_seq":4513,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-07-02T14:07:19.117837+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:07:19.117837+0000","last_peered":"2023-07-02T14:07:19.117837+0000","last_clean":"2023-07-02T14:07:19.117837+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:07:19.117837+0000","last_undegraded":"2023-07-02T14:07:19.117837+0000","last_fullsized":"2023-07-02T14:07:19.117837+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2975,"ondisk_log_size":2975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1661289598,"num_objects":499,"num_object_clones":416,"num_object_copies":998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":1545,"num_read_kb":2105271,"num_write":2559,"num_write_kb":1611276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"216'2884","reported_seq":3782,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-07-02T14:07:18.738127+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:07:18.738127+0000","last_peered":"2023-07-02T14:07:18.738127+0000","last_clean":"2023-07-02T14:07:18.738127+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:07:18.738127+0000","last_undegraded":"2023-07-02T14:07:18.738127+0000","last_fullsized":"2023-07-02T14:07:18.738127+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2884,"ondisk_log_size":2884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1569543764,"num_objects":461,"num_object_clones":383,"num_object_copies":922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":974,"num_read_kb":2026251,"num_write":2501,"num_write_kb":1523310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"216'2328","reported_seq":5156,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-07-02T14:07:19.121434+0000","last_change":"2023-07-02T14:07:09.383558+0000","last_active":"2023-07-02T14:07:19.121434+0000","last_peered":"2023-07-02T14:07:19.121434+0000","last_clean":"2023-07-02T14:07:19.121434+0000","last_became_active":"2023-07-02T14:07:09.382190+0000","last_became_peered":"2023-07-02T14:07:09.382190+0000","last_unstale":"2023-07-02T14:07:19.121434+0000","last_undegraded":"2023-07-02T14:07:19.121434+0000","last_fullsized":"2023-07-02T14:07:19.121434+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2328,"ondisk_log_size":2328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1230645001,"num_objects":405,"num_object_clones":325,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":1916,"num_read_kb":1773666,"num_write":2003,"num_write_kb":1186825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":1578968698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"218'2366","reported_seq":4203,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-07-02T14:07:19.121484+0000","last_change":"2023-07-02T14:07:18.108807+0000","last_active":"2023-07-02T14:07:19.121484+0000","last_peered":"2023-07-02T14:07:19.121484+0000","last_clean":"2023-07-02T14:07:19.121484+0000","last_became_active":"2023-07-02T14:07:18.106899+0000","last_became_peered":"2023-07-02T14:07:18.106899+0000","last_unstale":"2023-07-02T14:07:19.121484+0000","last_undegraded":"2023-07-02T14:07:19.121484+0000","last_fullsized":"2023-07-02T14:07:19.121484+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2366,"ondisk_log_size":2366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1146537804,"num_objects":358,"num_object_clones":299,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":1055,"num_read_kb":1573773,"num_write":2067,"num_write_kb":1113633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":434,"num_bytes_recovered":1452278570,"num_keys_recovered":8,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90630,"num_read_kb":15880603,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3618,"num_bytes_recovered":12343797996,"num_keys_recovered":190,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25587728384,"data_stored":29691869917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":27030,"ondisk_log_size":27030,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640325,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640325,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7136148,"kb_used_data":7041740,"kb_used_omap":0,"kb_used_meta":94400,"kb_avail":87235692,"statfs":{"total":96636764160,"available":89329348608,"internally_reserved":0,"allocated":7210741760,"data_stored":8298362840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":96665600},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.34000000000000002}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44,"15min":0.46400000000000002},"min":{"1min":0.28399999999999997,"5min":0.215,"15min":0.215},"max":{"1min":0.71499999999999997,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.45000000000000001},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.32800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66800000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.40899999999999997}]},{"osd":3,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.38300000000000001},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45900000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.36799999999999999,"15min":0.36299999999999999},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.48799999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.31900000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.44900000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.433,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59099999999999997,"5min":1.026,"15min":1.026},"last":0.32600000000000001}]}]},{"osd":3,"up_from":133,"seq":571230650397,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4569140,"kb_used_data":4498860,"kb_used_omap":229,"kb_used_meta":70042,"kb_avail":89802700,"statfs":{"total":96636764160,"available":91957964800,"internally_reserved":0,"allocated":4606832640,"data_stored":5357931508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":71723230},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.436}]},{"osd":1,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.372,"15min":0.36599999999999999},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.371}]},{"osd":2,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.28699999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.44900000000000001,"15min":0.439},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.52900000000000003}]},{"osd":4,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.35799999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.502},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.36199999999999999,"15min":0.35299999999999998},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.57199999999999995}]},{"osd":5,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.39800000000000002},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.439,"5min":0.39500000000000002,"15min":0.38800000000000001},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.61299999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640325,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3200480,"kb_used_data":3146328,"kb_used_omap":0,"kb_used_meta":54144,"kb_avail":91171360,"statfs":{"total":96636764160,"available":93359472640,"internally_reserved":0,"allocated":3221839872,"data_stored":3710361843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55443456},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.255}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.42499999999999999,"15min":0.374},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":0.496},{"interface":"front","average":{"1min":0.376,"5min":0.33500000000000002,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.35399999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:07:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.91100000000000003,"5min":1.929,"15min":1.929},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.54800000000000004,"15min":0.55600000000000005},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93999999999999995,"5min":2,"15min":2},"last":0.45900000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.51800000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68400000000000005,"5min":1.071,"15min":1.071},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":0.48199999999999998}]}]},{"osd":0,"up_from":0,"seq":712964571155,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":11,"seq":47244640325,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8689188,"kb_used_data":8558300,"kb_used_omap":0,"kb_used_meta":130880,"kb_avail":85682652,"statfs":{"total":96636764160,"available":87739035648,"internally_reserved":0,"allocated":8763699200,"data_stored":10365094588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":134021120},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.26600000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.311},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.224,"5min":0.184,"15min":0.184},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.30199999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.49099999999999999,"15min":0.47299999999999998},"min":{"1min":0.30299999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.42699999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71899999999999997,"5min":1.524,"15min":1.524},"last":0.46700000000000003}]},{"osd":5,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.80300000000000005,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84399999999999997,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.61799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1795358720,"data_stored":1964340839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8761016320,"data_stored":10364040347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3219156992,"data_stored":3709307602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4604137472,"data_stored":5356872530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":7208058880,"data_stored":8297308599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:07:24.536 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-07-02T14:07:24.536 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-07-02T14:07:24.536 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2023-07-02T14:07:24.537 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-07-02T14:07:24.849 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:24.849 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:24.849 INFO:teuthology.orchestra.run.smithi016.stdout: "epoch": 223, 2023-07-02T14:07:24.849 INFO:teuthology.orchestra.run.smithi016.stdout: "fsid": "9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2", 2023-07-02T14:07:24.850 INFO:teuthology.orchestra.run.smithi016.stdout: "created": "2023-07-02T14:01:37.298060+0000", 2023-07-02T14:07:24.850 INFO:teuthology.orchestra.run.smithi016.stdout: "modified": "2023-07-02T14:07:22.120618+0000", 2023-07-02T14:07:24.850 INFO:teuthology.orchestra.run.smithi016.stdout: "last_up_change": "2023-07-02T14:05:52.397969+0000", 2023-07-02T14:07:24.850 INFO:teuthology.orchestra.run.smithi016.stdout: "last_in_change": "2023-07-02T14:07:01.341473+0000", 2023-07-02T14:07:24.850 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-07-02T14:07:24.851 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_num": 5799936, 2023-07-02T14:07:24.851 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_set": [ 2023-07-02T14:07:24.851 INFO:teuthology.orchestra.run.smithi016.stdout: "pglog_hardlimit", 2023-07-02T14:07:24.851 INFO:teuthology.orchestra.run.smithi016.stdout: "purged_snapdirs", 2023-07-02T14:07:24.851 INFO:teuthology.orchestra.run.smithi016.stdout: "recovery_deletes", 2023-07-02T14:07:24.852 INFO:teuthology.orchestra.run.smithi016.stdout: "sortbitwise" 2023-07-02T14:07:24.852 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:07:24.852 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_version": 6, 2023-07-02T14:07:24.852 INFO:teuthology.orchestra.run.smithi016.stdout: "full_ratio": 0.94999998807907104, 2023-07-02T14:07:24.852 INFO:teuthology.orchestra.run.smithi016.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-07-02T14:07:24.853 INFO:teuthology.orchestra.run.smithi016.stdout: "nearfull_ratio": 0.85000002384185791, 2023-07-02T14:07:24.853 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_snapshot": "", 2023-07-02T14:07:24.853 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_max": 2, 2023-07-02T14:07:24.853 INFO:teuthology.orchestra.run.smithi016.stdout: "max_osd": 6, 2023-07-02T14:07:24.853 INFO:teuthology.orchestra.run.smithi016.stdout: "require_min_compat_client": "luminous", 2023-07-02T14:07:24.854 INFO:teuthology.orchestra.run.smithi016.stdout: "min_compat_client": "luminous", 2023-07-02T14:07:24.854 INFO:teuthology.orchestra.run.smithi016.stdout: "require_osd_release": "pacific", 2023-07-02T14:07:24.854 INFO:teuthology.orchestra.run.smithi016.stdout: "pools": [ 2023-07-02T14:07:24.854 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.854 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 1, 2023-07-02T14:07:24.855 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "device_health_metrics", 2023-07-02T14:07:24.855 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:45.323455+0000", 2023-07-02T14:07:24.855 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 1, 2023-07-02T14:07:24.855 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool", 2023-07-02T14:07:24.855 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:07:24.856 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:07:24.856 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:07:24.856 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:07:24.856 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:07:24.856 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:07:24.856 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:07:24.857 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:07:24.857 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:07:24.857 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:07:24.857 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 1, 2023-07-02T14:07:24.857 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 1, 2023-07-02T14:07:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 1, 2023-07-02T14:07:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 1, 2023-07-02T14:07:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 1, 2023-07-02T14:07:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:07:24.858 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:07:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:07:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:07:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:07:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:07:24.859 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:07:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "14", 2023-07-02T14:07:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:07:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:07:24.860 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:07:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:07:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:07:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 0, 2023-07-02T14:07:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 0, 2023-07-02T14:07:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:07:24.861 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:07:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:07:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:07:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:07:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:07:24.862 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:07:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:07:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:07:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:07:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:07:24.863 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:07:24.864 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:07:24.864 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:07:24.864 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:07:24.864 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:07:24.864 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:07:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:07:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:07:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:07:24.865 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:07:24.866 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:07:24.866 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:07:24.866 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:07:24.866 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:07:24.866 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:07:24.866 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:07:24.867 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:07:24.867 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:07:24.867 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:07:24.867 INFO:teuthology.orchestra.run.smithi016.stdout: "options": { 2023-07-02T14:07:24.867 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_max": 32, 2023-07-02T14:07:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_min": 1 2023-07-02T14:07:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:07:24.868 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr_devicehealth": {} 2023-07-02T14:07:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: "pool": 2, 2023-07-02T14:07:24.869 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_name": "rbd", 2023-07-02T14:07:24.870 INFO:teuthology.orchestra.run.smithi016.stdout: "create_time": "2023-07-02T14:01:47.448897+0000", 2023-07-02T14:07:24.870 INFO:teuthology.orchestra.run.smithi016.stdout: "flags": 8193, 2023-07-02T14:07:24.870 INFO:teuthology.orchestra.run.smithi016.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2023-07-02T14:07:24.870 INFO:teuthology.orchestra.run.smithi016.stdout: "type": 1, 2023-07-02T14:07:24.870 INFO:teuthology.orchestra.run.smithi016.stdout: "size": 2, 2023-07-02T14:07:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: "min_size": 1, 2023-07-02T14:07:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_rule": 0, 2023-07-02T14:07:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_count": 0, 2023-07-02T14:07:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_target": 0, 2023-07-02T14:07:24.871 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_barrier": 0, 2023-07-02T14:07:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-07-02T14:07:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: "object_hash": 2, 2023-07-02T14:07:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_autoscale_mode": "off", 2023-07-02T14:07:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num": 8, 2023-07-02T14:07:24.872 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num": 8, 2023-07-02T14:07:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_placement_num_target": 8, 2023-07-02T14:07:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_target": 8, 2023-07-02T14:07:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_num_pending": 8, 2023-07-02T14:07:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: "last_pg_merge_meta": { 2023-07-02T14:07:24.873 INFO:teuthology.orchestra.run.smithi016.stdout: "source_pgid": "0.0", 2023-07-02T14:07:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: "ready_epoch": 0, 2023-07-02T14:07:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_started": 0, 2023-07-02T14:07:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: "last_epoch_clean": 0, 2023-07-02T14:07:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: "source_version": "0'0", 2023-07-02T14:07:24.874 INFO:teuthology.orchestra.run.smithi016.stdout: "target_version": "0'0" 2023-07-02T14:07:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: "last_change": "217", 2023-07-02T14:07:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend": "0", 2023-07-02T14:07:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_prenautilus": "0", 2023-07-02T14:07:24.875 INFO:teuthology.orchestra.run.smithi016.stdout: "last_force_op_resend_preluminous": "0", 2023-07-02T14:07:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: "auid": 0, 2023-07-02T14:07:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_mode": "selfmanaged", 2023-07-02T14:07:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_seq": 69, 2023-07-02T14:07:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: "snap_epoch": 217, 2023-07-02T14:07:24.876 INFO:teuthology.orchestra.run.smithi016.stdout: "pool_snaps": [], 2023-07-02T14:07:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps": "[]", 2023-07-02T14:07:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_bytes": 0, 2023-07-02T14:07:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: "quota_max_objects": 0, 2023-07-02T14:07:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: "tiers": [], 2023-07-02T14:07:24.877 INFO:teuthology.orchestra.run.smithi016.stdout: "tier_of": -1, 2023-07-02T14:07:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: "read_tier": -1, 2023-07-02T14:07:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: "write_tier": -1, 2023-07-02T14:07:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_mode": "none", 2023-07-02T14:07:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_bytes": 0, 2023-07-02T14:07:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: "target_max_objects": 0, 2023-07-02T14:07:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-07-02T14:07:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-07-02T14:07:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_target_full_ratio_micro": 800000, 2023-07-02T14:07:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_flush_age": 0, 2023-07-02T14:07:24.879 INFO:teuthology.orchestra.run.smithi016.stdout: "cache_min_evict_age": 0, 2023-07-02T14:07:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profile": "", 2023-07-02T14:07:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_params": { 2023-07-02T14:07:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "none" 2023-07-02T14:07:24.880 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_period": 0, 2023-07-02T14:07:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_count": 0, 2023-07-02T14:07:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: "use_gmt_hitset": true, 2023-07-02T14:07:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: "min_read_recency_for_promote": 0, 2023-07-02T14:07:24.881 INFO:teuthology.orchestra.run.smithi016.stdout: "min_write_recency_for_promote": 0, 2023-07-02T14:07:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_grade_decay_rate": 0, 2023-07-02T14:07:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: "hit_set_search_last_n": 0, 2023-07-02T14:07:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: "grade_table": [], 2023-07-02T14:07:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: "stripe_width": 0, 2023-07-02T14:07:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: "expected_num_objects": 0, 2023-07-02T14:07:24.882 INFO:teuthology.orchestra.run.smithi016.stdout: "fast_read": false, 2023-07-02T14:07:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: "options": {}, 2023-07-02T14:07:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: "application_metadata": { 2023-07-02T14:07:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: "rbd": {} 2023-07-02T14:07:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.883 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.884 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:07:24.884 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:07:24.884 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.884 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:07:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "43c446fe-c811-4ee3-99e8-d7452edfc1c1", 2023-07-02T14:07:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:07:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 0, 2023-07-02T14:07:24.885 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 0, 2023-07-02T14:07:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 0.55181884765625, 2023-07-02T14:07:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 11, 2023-07-02T14:07:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 148, 2023-07-02T14:07:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 166, 2023-07-02T14:07:24.886 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 166, 2023-07-02T14:07:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 149, 2023-07-02T14:07:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:07:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:07:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.887 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6800", 2023-07-02T14:07:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 115688 2023-07-02T14:07:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.888 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6801", 2023-07-02T14:07:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 115688 2023-07-02T14:07:24.889 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.890 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.890 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.890 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:07:24.890 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6802", 2023-07-02T14:07:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 115688 2023-07-02T14:07:24.891 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6803", 2023-07-02T14:07:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 115688 2023-07-02T14:07:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.893 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:07:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.894 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6806", 2023-07-02T14:07:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 115688 2023-07-02T14:07:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.895 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6807", 2023-07-02T14:07:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 115688 2023-07-02T14:07:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.896 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:07:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6804", 2023-07-02T14:07:24.897 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 115688 2023-07-02T14:07:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6805", 2023-07-02T14:07:24.898 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 115688 2023-07-02T14:07:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6801/115688", 2023-07-02T14:07:24.899 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6803/115688", 2023-07-02T14:07:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6807/115688", 2023-07-02T14:07:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6805/115688", 2023-07-02T14:07:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:07:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:07:24.900 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:07:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:07:24.901 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b", 2023-07-02T14:07:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:07:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:07:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:07:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:07:24.902 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:07:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:07:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:07:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 220, 2023-07-02T14:07:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:07:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:07:24.903 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:07:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6808", 2023-07-02T14:07:24.904 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:07:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6809", 2023-07-02T14:07:24.905 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:07:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:07:24.906 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6810", 2023-07-02T14:07:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:07:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6811", 2023-07-02T14:07:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:07:24.908 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:07:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.909 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6814", 2023-07-02T14:07:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:07:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.910 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6815", 2023-07-02T14:07:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:07:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.911 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:07:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.912 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6812", 2023-07-02T14:07:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:07:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.913 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6813", 2023-07-02T14:07:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83360 2023-07-02T14:07:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.914 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6809/83360", 2023-07-02T14:07:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6811/83360", 2023-07-02T14:07:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6815/83360", 2023-07-02T14:07:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6813/83360", 2023-07-02T14:07:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:07:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:07:24.915 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:07:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:07:24.916 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "99f8ea0a-5a71-4abe-b087-8e891fd11d1b", 2023-07-02T14:07:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:07:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:07:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:07:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 0.9522247314453125, 2023-07-02T14:07:24.917 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:07:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:07:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:07:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 140, 2023-07-02T14:07:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:07:24.918 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:07:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:07:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.919 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6816", 2023-07-02T14:07:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:07:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.920 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6817", 2023-07-02T14:07:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:07:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:07:24.921 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6818", 2023-07-02T14:07:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:07:24.922 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6819", 2023-07-02T14:07:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:07:24.923 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:07:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.924 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6822", 2023-07-02T14:07:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:07:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.925 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6823", 2023-07-02T14:07:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:07:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.926 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:07:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.927 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6820", 2023-07-02T14:07:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:07:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.928 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:6821", 2023-07-02T14:07:24.929 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83361 2023-07-02T14:07:24.929 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.929 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.929 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.929 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.16:6817/83361", 2023-07-02T14:07:24.930 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.16:6819/83361", 2023-07-02T14:07:24.930 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.16:6823/83361", 2023-07-02T14:07:24.930 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.16:6821/83361", 2023-07-02T14:07:24.930 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:07:24.930 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:07:24.931 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:07:24.931 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.931 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.931 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:07:24.932 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "d285b4d5-0bef-421a-b84a-943e485b10c2", 2023-07-02T14:07:24.932 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:07:24.932 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:07:24.932 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:07:24.932 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:07:24.933 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 11, 2023-07-02T14:07:24.933 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 87, 2023-07-02T14:07:24.933 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 133, 2023-07-02T14:07:24.933 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 217, 2023-07-02T14:07:24.933 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 88, 2023-07-02T14:07:24.933 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:07:24.934 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:07:24.934 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.934 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.934 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.934 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6808", 2023-07-02T14:07:24.935 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 99977 2023-07-02T14:07:24.935 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.935 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.935 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.935 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6809", 2023-07-02T14:07:24.936 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 99977 2023-07-02T14:07:24.936 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.936 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.936 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.936 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:07:24.937 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.937 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.937 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.937 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6810", 2023-07-02T14:07:24.937 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 99977 2023-07-02T14:07:24.938 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.938 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.938 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.938 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6811", 2023-07-02T14:07:24.938 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 99977 2023-07-02T14:07:24.938 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.939 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.939 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.939 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:07:24.939 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.939 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.940 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.940 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6814", 2023-07-02T14:07:24.940 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 99977 2023-07-02T14:07:24.940 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.940 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.941 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.941 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6815", 2023-07-02T14:07:24.941 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 99977 2023-07-02T14:07:24.941 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.941 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.942 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.942 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:07:24.942 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.942 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.942 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.943 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6812", 2023-07-02T14:07:24.943 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 99977 2023-07-02T14:07:24.943 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.943 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.943 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.943 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6813", 2023-07-02T14:07:24.944 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 99977 2023-07-02T14:07:24.944 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.944 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.944 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.944 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6809/99977", 2023-07-02T14:07:24.945 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6811/99977", 2023-07-02T14:07:24.945 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6815/99977", 2023-07-02T14:07:24.945 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6813/99977", 2023-07-02T14:07:24.945 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:07:24.945 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:07:24.946 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:07:24.946 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.946 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.946 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.946 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:07:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "7ba0b0db-e407-4c63-8278-aab09f3082c4", 2023-07-02T14:07:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:07:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 1, 2023-07-02T14:07:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 1, 2023-07-02T14:07:24.947 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 0.23297119140625, 2023-07-02T14:07:24.948 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:07:24.948 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:07:24.948 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:07:24.948 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 120, 2023-07-02T14:07:24.948 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:07:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:07:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:07:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.949 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6800", 2023-07-02T14:07:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:07:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.950 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6801", 2023-07-02T14:07:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:07:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:07:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6802", 2023-07-02T14:07:24.952 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:07:24.953 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.953 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.953 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.953 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6803", 2023-07-02T14:07:24.953 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:07:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:07:24.954 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6806", 2023-07-02T14:07:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:07:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.955 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.956 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.956 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6807", 2023-07-02T14:07:24.956 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:07:24.956 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.956 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:07:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.957 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6804", 2023-07-02T14:07:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:07:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.958 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6805", 2023-07-02T14:07:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83351 2023-07-02T14:07:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.959 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.960 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6801/83351", 2023-07-02T14:07:24.960 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6803/83351", 2023-07-02T14:07:24.960 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6807/83351", 2023-07-02T14:07:24.960 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6805/83351", 2023-07-02T14:07:24.960 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:07:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:07:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:07:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.961 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:07:24.962 INFO:teuthology.orchestra.run.smithi016.stdout: "uuid": "29663bfa-f6fe-4ce7-b45a-a9b7b4424c42", 2023-07-02T14:07:24.962 INFO:teuthology.orchestra.run.smithi016.stdout: "up": 1, 2023-07-02T14:07:24.962 INFO:teuthology.orchestra.run.smithi016.stdout: "in": 0, 2023-07-02T14:07:24.962 INFO:teuthology.orchestra.run.smithi016.stdout: "weight": 0, 2023-07-02T14:07:24.962 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_affinity": 1, 2023-07-02T14:07:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_begin": 0, 2023-07-02T14:07:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: "last_clean_end": 0, 2023-07-02T14:07:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: "up_from": 11, 2023-07-02T14:07:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: "up_thru": 34, 2023-07-02T14:07:24.963 INFO:teuthology.orchestra.run.smithi016.stdout: "down_at": 0, 2023-07-02T14:07:24.964 INFO:teuthology.orchestra.run.smithi016.stdout: "lost_at": 0, 2023-07-02T14:07:24.964 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addrs": { 2023-07-02T14:07:24.964 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.964 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.964 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6816", 2023-07-02T14:07:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:07:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.965 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6817", 2023-07-02T14:07:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:07:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.966 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addrs": { 2023-07-02T14:07:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6818", 2023-07-02T14:07:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:07:24.968 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.968 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.968 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.968 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6819", 2023-07-02T14:07:24.968 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:07:24.969 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.969 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.969 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.969 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addrs": { 2023-07-02T14:07:24.969 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.970 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.970 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.970 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6822", 2023-07-02T14:07:24.970 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:07:24.970 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.971 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.971 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.971 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6823", 2023-07-02T14:07:24.971 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:07:24.971 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.972 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.972 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.972 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addrs": { 2023-07-02T14:07:24.972 INFO:teuthology.orchestra.run.smithi016.stdout: "addrvec": [ 2023-07-02T14:07:24.972 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.973 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v2", 2023-07-02T14:07:24.973 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6820", 2023-07-02T14:07:24.973 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:07:24.973 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.973 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.974 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2023-07-02T14:07:24.974 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.22:6821", 2023-07-02T14:07:24.974 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 83352 2023-07-02T14:07:24.974 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.974 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.974 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.975 INFO:teuthology.orchestra.run.smithi016.stdout: "public_addr": "172.21.15.22:6817/83352", 2023-07-02T14:07:24.975 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_addr": "172.21.15.22:6819/83352", 2023-07-02T14:07:24.975 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_back_addr": "172.21.15.22:6823/83352", 2023-07-02T14:07:24.975 INFO:teuthology.orchestra.run.smithi016.stdout: "heartbeat_front_addr": "172.21.15.22:6821/83352", 2023-07-02T14:07:24.975 INFO:teuthology.orchestra.run.smithi016.stdout: "state": [ 2023-07-02T14:07:24.976 INFO:teuthology.orchestra.run.smithi016.stdout: "exists", 2023-07-02T14:07:24.976 INFO:teuthology.orchestra.run.smithi016.stdout: "up" 2023-07-02T14:07:24.976 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.976 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.976 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:07:24.977 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_xinfo": [ 2023-07-02T14:07:24.977 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.977 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 0, 2023-07-02T14:07:24.977 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:07:24.977 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:07:24.978 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:07:24.978 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:07:24.978 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 65536, 2023-07-02T14:07:24.978 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.001178+0000", 2023-07-02T14:07:24.978 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 149 2023-07-02T14:07:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 1, 2023-07-02T14:07:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:07:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:07:24.979 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:07:24.980 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:07:24.980 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:07:24.980 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.184181+0000", 2023-07-02T14:07:24.980 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:07:24.980 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.981 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.981 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 2, 2023-07-02T14:07:24.981 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:07:24.981 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:07:24.981 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:07:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:07:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:07:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.274854+0000", 2023-07-02T14:07:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:07:24.982 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 3, 2023-07-02T14:07:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:07:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:07:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:07:24.984 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:07:24.984 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:07:24.984 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.186217+0000", 2023-07-02T14:07:24.984 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 88 2023-07-02T14:07:24.984 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 4, 2023-07-02T14:07:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:07:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:07:24.985 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:07:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:07:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 0, 2023-07-02T14:07:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.247442+0000", 2023-07-02T14:07:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:07:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.986 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": 5, 2023-07-02T14:07:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: "down_stamp": "0.000000", 2023-07-02T14:07:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_probability": 0, 2023-07-02T14:07:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: "laggy_interval": 0, 2023-07-02T14:07:24.987 INFO:teuthology.orchestra.run.smithi016.stdout: "features": 4540138314316775423, 2023-07-02T14:07:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: "old_weight": 65536, 2023-07-02T14:07:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: "last_purged_snaps_scrub": "2023-07-02T14:01:44.220075+0000", 2023-07-02T14:07:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: "dead_epoch": 0 2023-07-02T14:07:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.988 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:07:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap": [ 2023-07-02T14:07:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "1.0", 2023-07-02T14:07:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:07:24.989 INFO:teuthology.orchestra.run.smithi016.stdout: 0, 2023-07-02T14:07:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: 3 2023-07-02T14:07:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.990 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.0", 2023-07-02T14:07:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:07:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: 4, 2023-07-02T14:07:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: 1 2023-07-02T14:07:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.991 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.992 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.2", 2023-07-02T14:07:24.992 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:07:24.992 INFO:teuthology.orchestra.run.smithi016.stdout: 1, 2023-07-02T14:07:24.992 INFO:teuthology.orchestra.run.smithi016.stdout: 5 2023-07-02T14:07:24.992 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.993 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.993 INFO:teuthology.orchestra.run.smithi016.stdout: { 2023-07-02T14:07:24.993 INFO:teuthology.orchestra.run.smithi016.stdout: "pgid": "2.4", 2023-07-02T14:07:24.993 INFO:teuthology.orchestra.run.smithi016.stdout: "osds": [ 2023-07-02T14:07:24.993 INFO:teuthology.orchestra.run.smithi016.stdout: 5, 2023-07-02T14:07:24.994 INFO:teuthology.orchestra.run.smithi016.stdout: 0 2023-07-02T14:07:24.994 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2023-07-02T14:07:24.994 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.994 INFO:teuthology.orchestra.run.smithi016.stdout: ], 2023-07-02T14:07:24.994 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_upmap_items": [], 2023-07-02T14:07:24.995 INFO:teuthology.orchestra.run.smithi016.stdout: "pg_temp": [], 2023-07-02T14:07:24.995 INFO:teuthology.orchestra.run.smithi016.stdout: "primary_temp": [], 2023-07-02T14:07:24.995 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklist": {}, 2023-07-02T14:07:24.995 INFO:teuthology.orchestra.run.smithi016.stdout: "range_blocklist": {}, 2023-07-02T14:07:24.995 INFO:teuthology.orchestra.run.smithi016.stdout: "erasure_code_profiles": { 2023-07-02T14:07:24.996 INFO:teuthology.orchestra.run.smithi016.stdout: "default": { 2023-07-02T14:07:24.996 INFO:teuthology.orchestra.run.smithi016.stdout: "crush-failure-domain": "osd", 2023-07-02T14:07:24.996 INFO:teuthology.orchestra.run.smithi016.stdout: "k": "2", 2023-07-02T14:07:24.996 INFO:teuthology.orchestra.run.smithi016.stdout: "m": "1", 2023-07-02T14:07:24.996 INFO:teuthology.orchestra.run.smithi016.stdout: "plugin": "jerasure", 2023-07-02T14:07:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: "ruleset-failure-domain": "osd", 2023-07-02T14:07:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: "technique": "reed_sol_van" 2023-07-02T14:07:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2023-07-02T14:07:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: "removed_snaps_queue": [], 2023-07-02T14:07:24.997 INFO:teuthology.orchestra.run.smithi016.stdout: "new_removed_snaps": [], 2023-07-02T14:07:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "new_purged_snaps": [], 2023-07-02T14:07:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "crush_node_flags": {}, 2023-07-02T14:07:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "device_class_flags": {}, 2023-07-02T14:07:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode": { 2023-07-02T14:07:24.998 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_enabled": false, 2023-07-02T14:07:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_bucket_count": 0, 2023-07-02T14:07:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: "degraded_stretch_mode": 0, 2023-07-02T14:07:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: "recovering_stretch_mode": 0, 2023-07-02T14:07:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: "stretch_mode_bucket": 0 2023-07-02T14:07:24.999 INFO:teuthology.orchestra.run.smithi016.stdout: } 2023-07-02T14:07:25.000 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:25.001 INFO:tasks.thrashosds.thrasher:j is {'epoch': 223, 'fsid': '9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2', 'created': '2023-07-02T14:01:37.298060+0000', 'modified': '2023-07-02T14:07:22.120618+0000', 'last_up_change': '2023-07-02T14:05:52.397969+0000', 'last_in_change': '2023-07-02T14:07:01.341473+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2023-07-02T14:01:45.323455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-07-02T14:01:47.448897+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '217', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 69, 'snap_epoch': 217, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}], 'osds': [{'osd': 0, 'uuid': '43c446fe-c811-4ee3-99e8-d7452edfc1c1', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.55181884765625, 'last_clean_begin': 11, 'last_clean_end': 148, 'up_from': 166, 'up_thru': 166, 'down_at': 149, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6800', 'nonce': 115688}, {'type': 'v1', 'addr': '172.21.15.16:6801', 'nonce': 115688}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6802', 'nonce': 115688}, {'type': 'v1', 'addr': '172.21.15.16:6803', 'nonce': 115688}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6806', 'nonce': 115688}, {'type': 'v1', 'addr': '172.21.15.16:6807', 'nonce': 115688}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6804', 'nonce': 115688}, {'type': 'v1', 'addr': '172.21.15.16:6805', 'nonce': 115688}]}, 'public_addr': '172.21.15.16:6801/115688', 'cluster_addr': '172.21.15.16:6803/115688', 'heartbeat_back_addr': '172.21.15.16:6807/115688', 'heartbeat_front_addr': '172.21.15.16:6805/115688', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 220, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6808', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6809', 'nonce': 83360}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6810', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6811', 'nonce': 83360}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6814', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6815', 'nonce': 83360}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6812', 'nonce': 83360}, {'type': 'v1', 'addr': '172.21.15.16:6813', 'nonce': 83360}]}, 'public_addr': '172.21.15.16:6809/83360', 'cluster_addr': '172.21.15.16:6811/83360', 'heartbeat_back_addr': '172.21.15.16:6815/83360', 'heartbeat_front_addr': '172.21.15.16:6813/83360', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '99f8ea0a-5a71-4abe-b087-8e891fd11d1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9522247314453125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 140, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6816', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6817', 'nonce': 83361}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6818', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6819', 'nonce': 83361}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6822', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6823', 'nonce': 83361}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6820', 'nonce': 83361}, {'type': 'v1', 'addr': '172.21.15.16:6821', 'nonce': 83361}]}, 'public_addr': '172.21.15.16:6817/83361', 'cluster_addr': '172.21.15.16:6819/83361', 'heartbeat_back_addr': '172.21.15.16:6823/83361', 'heartbeat_front_addr': '172.21.15.16:6821/83361', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd285b4d5-0bef-421a-b84a-943e485b10c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 11, 'last_clean_end': 87, 'up_from': 133, 'up_thru': 217, 'down_at': 88, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6808', 'nonce': 99977}, {'type': 'v1', 'addr': '172.21.15.22:6809', 'nonce': 99977}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6810', 'nonce': 99977}, {'type': 'v1', 'addr': '172.21.15.22:6811', 'nonce': 99977}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6814', 'nonce': 99977}, {'type': 'v1', 'addr': '172.21.15.22:6815', 'nonce': 99977}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6812', 'nonce': 99977}, {'type': 'v1', 'addr': '172.21.15.22:6813', 'nonce': 99977}]}, 'public_addr': '172.21.15.22:6809/99977', 'cluster_addr': '172.21.15.22:6811/99977', 'heartbeat_back_addr': '172.21.15.22:6815/99977', 'heartbeat_front_addr': '172.21.15.22:6813/99977', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '7ba0b0db-e407-4c63-8278-aab09f3082c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.23297119140625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 120, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6800', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6801', 'nonce': 83351}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6802', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6803', 'nonce': 83351}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6806', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6807', 'nonce': 83351}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6804', 'nonce': 83351}, {'type': 'v1', 'addr': '172.21.15.22:6805', 'nonce': 83351}]}, 'public_addr': '172.21.15.22:6801/83351', 'cluster_addr': '172.21.15.22:6803/83351', 'heartbeat_back_addr': '172.21.15.22:6807/83351', 'heartbeat_front_addr': '172.21.15.22:6805/83351', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '29663bfa-f6fe-4ce7-b45a-a9b7b4424c42', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 34, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6816', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6817', 'nonce': 83352}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6818', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6819', 'nonce': 83352}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6822', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6823', 'nonce': 83352}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6820', 'nonce': 83352}, {'type': 'v1', 'addr': '172.21.15.22:6821', 'nonce': 83352}]}, 'public_addr': '172.21.15.22:6817/83352', 'cluster_addr': '172.21.15.22:6819/83352', 'heartbeat_back_addr': '172.21.15.22:6823/83352', 'heartbeat_front_addr': '172.21.15.22:6821/83352', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.001178+0000', 'dead_epoch': 149}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.184181+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.274854+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.186217+0000', 'dead_epoch': 88}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.247442+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-07-02T14:01:44.220075+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [0, 3]}, {'pgid': '2.0', 'osds': [4, 1]}, {'pgid': '2.2', 'osds': [1, 5]}, {'pgid': '2.4', 'osds': [5, 0]}], 'pg_upmap_items': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-07-02T14:07:25.002 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:07:25.137 INFO:teuthology.orchestra.run.smithi016.stderr:noscrub is unset 2023-07-02T14:07:25.148 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-07-02T14:07:25.305 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:25.307 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:07:25.319 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":389,"stamp":"2023-07-02T14:07:25.132751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90630,"num_read_kb":15880603,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3618,"num_bytes_recovered":12343797996,"num_keys_recovered":190,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27030,"ondisk_log_size":27030,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":23594956,"kb_used_data":23245228,"kb_used_omap":229,"kb_used_meta":349466,"kb_avail":353892404,"statfs":{"total":386547056640,"available":362385821696,"internally_reserved":0,"allocated":23803113472,"data_stored":27731750779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":357853406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-8,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1010,"num_read_kb":52582,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":364540066,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.840629"},"pg_stats":[{"pgid":"2.7","version":"222'3704","reported_seq":9528,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-07-02T14:07:21.570504+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:07:21.570504+0000","last_peered":"2023-07-02T14:07:21.570504+0000","last_clean":"2023-07-02T14:07:21.570504+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:07:21.570504+0000","last_undegraded":"2023-07-02T14:07:21.570504+0000","last_fullsized":"2023-07-02T14:07:21.570504+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3704,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1575041492,"num_objects":472,"num_object_clones":397,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":59417,"num_read_kb":2052552,"num_write":3144,"num_write_kb":1530742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"216'3069","reported_seq":5184,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-07-02T14:07:18.355050+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:07:18.355050+0000","last_peered":"2023-07-02T14:07:18.355050+0000","last_clean":"2023-07-02T14:07:18.355050+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:07:18.355050+0000","last_undegraded":"2023-07-02T14:07:18.355050+0000","last_fullsized":"2023-07-02T14:07:18.355050+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3069,"ondisk_log_size":3069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1538467250,"num_objects":486,"num_object_clones":415,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":1948,"num_read_kb":2154143,"num_write":2654,"num_write_kb":1492749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"221'3210","reported_seq":4801,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-07-02T14:07:21.516569+0000","last_change":"2023-07-02T14:07:14.111389+0000","last_active":"2023-07-02T14:07:21.516569+0000","last_peered":"2023-07-02T14:07:21.516569+0000","last_clean":"2023-07-02T14:07:21.516569+0000","last_became_active":"2023-07-02T14:07:14.107764+0000","last_became_peered":"2023-07-02T14:07:14.107764+0000","last_unstale":"2023-07-02T14:07:21.516569+0000","last_undegraded":"2023-07-02T14:07:21.516569+0000","last_fullsized":"2023-07-02T14:07:21.516569+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3210,"ondisk_log_size":3210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1517142988,"num_objects":428,"num_object_clones":355,"num_object_copies":856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":743,"num_read_kb":1976610,"num_write":2851,"num_write_kb":1467899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":2369610862,"num_keys_recovered":8,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"216'6494","reported_seq":9016,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-07-02T14:07:18.355075+0000","last_change":"2023-07-02T14:06:29.572710+0000","last_active":"2023-07-02T14:07:18.355075+0000","last_peered":"2023-07-02T14:07:18.355075+0000","last_clean":"2023-07-02T14:07:18.355075+0000","last_became_active":"2023-07-02T14:06:29.569824+0000","last_became_peered":"2023-07-02T14:06:29.569824+0000","last_unstale":"2023-07-02T14:07:18.355075+0000","last_undegraded":"2023-07-02T14:07:18.355075+0000","last_fullsized":"2023-07-02T14:07:18.355075+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":6494,"ondisk_log_size":6494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1675535810,"num_objects":495,"num_object_clones":414,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":23032,"num_read_kb":2218337,"num_write":9605,"num_write_kb":1635009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":123,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-07-02T14:07:19.160455+0000","last_change":"2023-07-02T14:07:02.103009+0000","last_active":"2023-07-02T14:07:19.160455+0000","last_peered":"2023-07-02T14:07:19.160455+0000","last_clean":"2023-07-02T14:07:19.160455+0000","last_became_active":"2023-07-02T14:07:02.102654+0000","last_became_peered":"2023-07-02T14:07:02.102654+0000","last_unstale":"2023-07-02T14:07:19.160455+0000","last_undegraded":"2023-07-02T14:07:19.160455+0000","last_fullsized":"2023-07-02T14:07:19.160455+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"216'2975","reported_seq":4513,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-07-02T14:07:19.117837+0000","last_change":"2023-07-02T14:04:25.120696+0000","last_active":"2023-07-02T14:07:19.117837+0000","last_peered":"2023-07-02T14:07:19.117837+0000","last_clean":"2023-07-02T14:07:19.117837+0000","last_became_active":"2023-07-02T14:04:25.117347+0000","last_became_peered":"2023-07-02T14:04:25.117347+0000","last_unstale":"2023-07-02T14:07:19.117837+0000","last_undegraded":"2023-07-02T14:07:19.117837+0000","last_fullsized":"2023-07-02T14:07:19.117837+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2975,"ondisk_log_size":2975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1661289598,"num_objects":499,"num_object_clones":416,"num_object_copies":998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":1545,"num_read_kb":2105271,"num_write":2559,"num_write_kb":1611276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":232,"num_bytes_recovered":818643316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"216'2884","reported_seq":3782,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-07-02T14:07:18.738127+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:07:18.738127+0000","last_peered":"2023-07-02T14:07:18.738127+0000","last_clean":"2023-07-02T14:07:18.738127+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:07:18.738127+0000","last_undegraded":"2023-07-02T14:07:18.738127+0000","last_fullsized":"2023-07-02T14:07:18.738127+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2884,"ondisk_log_size":2884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1569543764,"num_objects":461,"num_object_clones":383,"num_object_copies":922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":974,"num_read_kb":2026251,"num_write":2501,"num_write_kb":1523310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"216'2328","reported_seq":5156,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-07-02T14:07:19.121434+0000","last_change":"2023-07-02T14:07:09.383558+0000","last_active":"2023-07-02T14:07:19.121434+0000","last_peered":"2023-07-02T14:07:19.121434+0000","last_clean":"2023-07-02T14:07:19.121434+0000","last_became_active":"2023-07-02T14:07:09.382190+0000","last_became_peered":"2023-07-02T14:07:09.382190+0000","last_unstale":"2023-07-02T14:07:19.121434+0000","last_undegraded":"2023-07-02T14:07:19.121434+0000","last_fullsized":"2023-07-02T14:07:19.121434+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2328,"ondisk_log_size":2328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1230645001,"num_objects":405,"num_object_clones":325,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":1916,"num_read_kb":1773666,"num_write":2003,"num_write_kb":1186825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":499,"num_bytes_recovered":1578968698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"218'2366","reported_seq":4203,"reported_epoch":222,"state":"active+clean","last_fresh":"2023-07-02T14:07:19.121484+0000","last_change":"2023-07-02T14:07:18.108807+0000","last_active":"2023-07-02T14:07:19.121484+0000","last_peered":"2023-07-02T14:07:19.121484+0000","last_clean":"2023-07-02T14:07:19.121484+0000","last_became_active":"2023-07-02T14:07:18.106899+0000","last_became_peered":"2023-07-02T14:07:18.106899+0000","last_unstale":"2023-07-02T14:07:19.121484+0000","last_undegraded":"2023-07-02T14:07:19.121484+0000","last_fullsized":"2023-07-02T14:07:19.121484+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2366,"ondisk_log_size":2366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1146537804,"num_objects":358,"num_object_clones":299,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":1055,"num_read_kb":1573773,"num_write":2067,"num_write_kb":1113633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":434,"num_bytes_recovered":1452278570,"num_keys_recovered":8,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90630,"num_read_kb":15880603,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3618,"num_bytes_recovered":12343797996,"num_keys_recovered":190,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25587728384,"data_stored":29691869917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":27030,"ondisk_log_size":27030,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":0,"seq":47244640325,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":11,"seq":47244640325,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7136148,"kb_used_data":7041740,"kb_used_omap":0,"kb_used_meta":94400,"kb_avail":87235692,"statfs":{"total":96636764160,"available":89329348608,"internally_reserved":0,"allocated":7210741760,"data_stored":8298362840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":96665600},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.34000000000000002}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44,"15min":0.46400000000000002},"min":{"1min":0.28399999999999997,"5min":0.215,"15min":0.215},"max":{"1min":0.71499999999999997,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.45000000000000001},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.32800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66800000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.40899999999999997}]},{"osd":3,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.38300000000000001},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45900000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.36799999999999999,"15min":0.36299999999999999},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.48799999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.31900000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.44900000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.433,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59099999999999997,"5min":1.026,"15min":1.026},"last":0.32600000000000001}]}]},{"osd":3,"up_from":133,"seq":571230650397,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4569140,"kb_used_data":4498860,"kb_used_omap":229,"kb_used_meta":70042,"kb_avail":89802700,"statfs":{"total":96636764160,"available":91957964800,"internally_reserved":0,"allocated":4606832640,"data_stored":5357931508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":71723230},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.436}]},{"osd":1,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.372,"15min":0.36599999999999999},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.371}]},{"osd":2,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.28699999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.44900000000000001,"15min":0.439},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.52900000000000003}]},{"osd":4,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.35799999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.502},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.36199999999999999,"15min":0.35299999999999998},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.57199999999999995}]},{"osd":5,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.39800000000000002},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.439,"5min":0.39500000000000002,"15min":0.38800000000000001},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.61299999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640325,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3200480,"kb_used_data":3146328,"kb_used_omap":0,"kb_used_meta":54144,"kb_avail":91171360,"statfs":{"total":96636764160,"available":93359472640,"internally_reserved":0,"allocated":3221839872,"data_stored":3710361843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55443456},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.255}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.42499999999999999,"15min":0.374},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":0.496},{"interface":"front","average":{"1min":0.376,"5min":0.33500000000000002,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.35399999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:07:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.91100000000000003,"5min":1.929,"15min":1.929},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.54800000000000004,"15min":0.55600000000000005},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93999999999999995,"5min":2,"15min":2},"last":0.45900000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.51800000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68400000000000005,"5min":1.071,"15min":1.071},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":0.48199999999999998}]}]},{"osd":0,"up_from":0,"seq":712964571155,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":11,"seq":47244640325,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8689188,"kb_used_data":8558300,"kb_used_omap":0,"kb_used_meta":130880,"kb_avail":85682652,"statfs":{"total":96636764160,"available":87739035648,"internally_reserved":0,"allocated":8763699200,"data_stored":10365094588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":134021120},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.26600000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.311},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.224,"5min":0.184,"15min":0.184},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.30199999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.49099999999999999,"15min":0.47299999999999998},"min":{"1min":0.30299999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.42699999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71899999999999997,"5min":1.524,"15min":1.524},"last":0.46700000000000003}]},{"osd":5,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.80300000000000005,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84399999999999997,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.61799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1795358720,"data_stored":1964340839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8761016320,"data_stored":10364040347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3219156992,"data_stored":3709307602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4604137472,"data_stored":5356872530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":7208058880,"data_stored":8297308599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:07:25.320 INFO:tasks.thrashosds.thrasher:Setting 2.3 to [3, 4, 5, 2] 2023-07-02T14:07:25.321 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.3', '3', '4', '5', '2'] 2023-07-02T14:07:25.321 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.3 3 4 5 2 2023-07-02T14:07:26.141 INFO:teuthology.orchestra.run.smithi016.stderr:set 2.3 pg_upmap_items mapping to [3->4,5->2] 2023-07-02T14:07:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:nodeep-scrub is unset 2023-07-02T14:07:31.159 INFO:tasks.thrashosds.thrasher:inning osd 2023-07-02T14:07:31.160 INFO:tasks.thrashosds.thrasher:Adding osd 5 2023-07-02T14:07:31.160 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2023-07-02T14:07:31.486 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-07-02T14:07:31.485+0000 7ff490b05700 -1 mon.a@0(leader).osd e225 definitely_dead 0 2023-07-02T14:07:32.357 INFO:teuthology.orchestra.run.smithi016.stderr:marked in osd.5. 2023-07-02T14:07:32.370 INFO:tasks.thrashosds.thrasher:Added osd 5 2023-07-02T14:07:32.371 INFO:tasks.thrashosds.thrasher:inning osd 2023-07-02T14:07:32.371 INFO:tasks.thrashosds.thrasher:Adding osd 0 2023-07-02T14:07:32.371 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2023-07-02T14:07:32.689 INFO:tasks.ceph.mon.a.smithi016.stderr:2023-07-02T14:07:32.688+0000 7ff490b05700 -1 mon.a@0(leader).osd e226 definitely_dead 0 2023-07-02T14:07:33.359 INFO:teuthology.orchestra.run.smithi016.stderr:marked in osd.0. 2023-07-02T14:07:33.372 INFO:tasks.thrashosds.thrasher:Added osd 0 2023-07-02T14:07:33.373 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2023-07-02T14:07:33.505 INFO:teuthology.orchestra.run.smithi016.stdout:mon.a: {} 2023-07-02T14:07:33.515 INFO:teuthology.orchestra.run.smithi016.stdout:mon.b: {} 2023-07-02T14:07:33.524 INFO:teuthology.orchestra.run.smithi016.stdout:mon.c: {} 2023-07-02T14:07:33.525 INFO:teuthology.orchestra.run.smithi016.stderr:mon.a: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2023-07-02T14:07:33.525 INFO:teuthology.orchestra.run.smithi016.stderr:mon.b: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2023-07-02T14:07:33.525 INFO:teuthology.orchestra.run.smithi016.stderr:mon.c: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2023-07-02T14:07:33.536 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-07-02T14:07:34.363 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.1 to 1 (10000) 2023-07-02T14:07:34.382 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2023-07-02T14:07:35.366 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.1 primary-affinity to 1 (8655362) 2023-07-02T14:07:35.383 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-07-02T14:07:36.368 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.2 to 1 (10000) 2023-07-02T14:07:36.382 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2023-07-02T14:07:37.375 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.2 primary-affinity to 1 (8655362) 2023-07-02T14:07:37.391 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2023-07-02T14:07:38.385 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.4 to 1 (10000) 2023-07-02T14:07:38.400 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2023-07-02T14:07:39.389 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.4 primary-affinity to 1 (8655362) 2023-07-02T14:07:39.404 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2023-07-02T14:07:40.393 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.5 to 1 (10000) 2023-07-02T14:07:40.406 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2023-07-02T14:07:41.395 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.5 primary-affinity to 1 (8655362) 2023-07-02T14:07:41.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-07-02T14:07:42.397 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.3 to 1 (10000) 2023-07-02T14:07:42.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2023-07-02T14:07:43.401 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.3 primary-affinity to 1 (8655362) 2023-07-02T14:07:43.415 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-07-02T14:07:44.408 INFO:teuthology.orchestra.run.smithi016.stderr:reweighted osd.0 to 1 (10000) 2023-07-02T14:07:44.425 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2023-07-02T14:07:45.414 INFO:teuthology.orchestra.run.smithi016.stderr:set osd.0 primary-affinity to 1 (8655362) 2023-07-02T14:07:45.431 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2023-07-02T14:07:45.432 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2023-07-02T14:07:45.432 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2023-07-02T14:07:45.432 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2023-07-02T14:07:45.432 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-07-02T14:07:45.433 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-02T14:07:45.827 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:45.828 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":239,"fsid":"9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2","created":"2023-07-02T14:01:37.298060+0000","modified":"2023-07-02T14:07:45.402569+0000","last_up_change":"2023-07-02T14:05:52.397969+0000","last_in_change":"2023-07-02T14:07:33.350320+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-07-02T14:01:45.323455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-02T14:01:47.448897+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":69,"snap_epoch":217,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"43c446fe-c811-4ee3-99e8-d7452edfc1c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":148,"up_from":166,"up_thru":227,"down_at":149,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":115688},{"type":"v1","addr":"172.21.15.16:6801","nonce":115688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":115688},{"type":"v1","addr":"172.21.15.16:6803","nonce":115688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":115688},{"type":"v1","addr":"172.21.15.16:6807","nonce":115688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":115688},{"type":"v1","addr":"172.21.15.16:6805","nonce":115688}]},"public_addr":"172.21.15.16:6801/115688","cluster_addr":"172.21.15.16:6803/115688","heartbeat_back_addr":"172.21.15.16:6807/115688","heartbeat_front_addr":"172.21.15.16:6805/115688","state":["exists","up"]},{"osd":1,"uuid":"4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":83360},{"type":"v1","addr":"172.21.15.16:6809","nonce":83360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":83360},{"type":"v1","addr":"172.21.15.16:6811","nonce":83360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":83360},{"type":"v1","addr":"172.21.15.16:6815","nonce":83360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":83360},{"type":"v1","addr":"172.21.15.16:6813","nonce":83360}]},"public_addr":"172.21.15.16:6809/83360","cluster_addr":"172.21.15.16:6811/83360","heartbeat_back_addr":"172.21.15.16:6815/83360","heartbeat_front_addr":"172.21.15.16:6813/83360","state":["exists","up"]},{"osd":2,"uuid":"99f8ea0a-5a71-4abe-b087-8e891fd11d1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":83361},{"type":"v1","addr":"172.21.15.16:6817","nonce":83361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":83361},{"type":"v1","addr":"172.21.15.16:6819","nonce":83361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":83361},{"type":"v1","addr":"172.21.15.16:6823","nonce":83361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":83361},{"type":"v1","addr":"172.21.15.16:6821","nonce":83361}]},"public_addr":"172.21.15.16:6817/83361","cluster_addr":"172.21.15.16:6819/83361","heartbeat_back_addr":"172.21.15.16:6823/83361","heartbeat_front_addr":"172.21.15.16:6821/83361","state":["exists","up"]},{"osd":3,"uuid":"d285b4d5-0bef-421a-b84a-943e485b10c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":87,"up_from":133,"up_thru":234,"down_at":88,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":99977},{"type":"v1","addr":"172.21.15.22:6809","nonce":99977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":99977},{"type":"v1","addr":"172.21.15.22:6811","nonce":99977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":99977},{"type":"v1","addr":"172.21.15.22:6815","nonce":99977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":99977},{"type":"v1","addr":"172.21.15.22:6813","nonce":99977}]},"public_addr":"172.21.15.22:6809/99977","cluster_addr":"172.21.15.22:6811/99977","heartbeat_back_addr":"172.21.15.22:6815/99977","heartbeat_front_addr":"172.21.15.22:6813/99977","state":["exists","up"]},{"osd":4,"uuid":"7ba0b0db-e407-4c63-8278-aab09f3082c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":83351},{"type":"v1","addr":"172.21.15.22:6801","nonce":83351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":83351},{"type":"v1","addr":"172.21.15.22:6803","nonce":83351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":83351},{"type":"v1","addr":"172.21.15.22:6807","nonce":83351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":83351},{"type":"v1","addr":"172.21.15.22:6805","nonce":83351}]},"public_addr":"172.21.15.22:6801/83351","cluster_addr":"172.21.15.22:6803/83351","heartbeat_back_addr":"172.21.15.22:6807/83351","heartbeat_front_addr":"172.21.15.22:6805/83351","state":["exists","up"]},{"osd":5,"uuid":"29663bfa-f6fe-4ce7-b45a-a9b7b4424c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":238,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":83352},{"type":"v1","addr":"172.21.15.22:6817","nonce":83352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":83352},{"type":"v1","addr":"172.21.15.22:6819","nonce":83352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":83352},{"type":"v1","addr":"172.21.15.22:6823","nonce":83352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":83352},{"type":"v1","addr":"172.21.15.22:6821","nonce":83352}]},"public_addr":"172.21.15.22:6817/83352","cluster_addr":"172.21.15.22:6819/83352","heartbeat_back_addr":"172.21.15.22:6823/83352","heartbeat_front_addr":"172.21.15.22:6821/83352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.001178+0000","dead_epoch":149},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.184181+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.274854+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.186217+0000","dead_epoch":88},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.247442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.220075+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[0,3]},{"pgid":"2.0","osds":[4,1]},{"pgid":"2.2","osds":[1,5]},{"pgid":"2.4","osds":[5,0]}],"pg_upmap_items":[],"pg_temp":[{"pgid":"2.4","osds":[1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-02T14:07:45.844 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-07-02T14:07:45.844 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-02T14:07:46.236 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:46.237 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":239,"fsid":"9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2","created":"2023-07-02T14:01:37.298060+0000","modified":"2023-07-02T14:07:45.402569+0000","last_up_change":"2023-07-02T14:05:52.397969+0000","last_in_change":"2023-07-02T14:07:33.350320+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-07-02T14:01:45.323455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-02T14:01:47.448897+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":69,"snap_epoch":217,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"43c446fe-c811-4ee3-99e8-d7452edfc1c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":148,"up_from":166,"up_thru":227,"down_at":149,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":115688},{"type":"v1","addr":"172.21.15.16:6801","nonce":115688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":115688},{"type":"v1","addr":"172.21.15.16:6803","nonce":115688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":115688},{"type":"v1","addr":"172.21.15.16:6807","nonce":115688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":115688},{"type":"v1","addr":"172.21.15.16:6805","nonce":115688}]},"public_addr":"172.21.15.16:6801/115688","cluster_addr":"172.21.15.16:6803/115688","heartbeat_back_addr":"172.21.15.16:6807/115688","heartbeat_front_addr":"172.21.15.16:6805/115688","state":["exists","up"]},{"osd":1,"uuid":"4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":237,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":83360},{"type":"v1","addr":"172.21.15.16:6809","nonce":83360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":83360},{"type":"v1","addr":"172.21.15.16:6811","nonce":83360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":83360},{"type":"v1","addr":"172.21.15.16:6815","nonce":83360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":83360},{"type":"v1","addr":"172.21.15.16:6813","nonce":83360}]},"public_addr":"172.21.15.16:6809/83360","cluster_addr":"172.21.15.16:6811/83360","heartbeat_back_addr":"172.21.15.16:6815/83360","heartbeat_front_addr":"172.21.15.16:6813/83360","state":["exists","up"]},{"osd":2,"uuid":"99f8ea0a-5a71-4abe-b087-8e891fd11d1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":83361},{"type":"v1","addr":"172.21.15.16:6817","nonce":83361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":83361},{"type":"v1","addr":"172.21.15.16:6819","nonce":83361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":83361},{"type":"v1","addr":"172.21.15.16:6823","nonce":83361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":83361},{"type":"v1","addr":"172.21.15.16:6821","nonce":83361}]},"public_addr":"172.21.15.16:6817/83361","cluster_addr":"172.21.15.16:6819/83361","heartbeat_back_addr":"172.21.15.16:6823/83361","heartbeat_front_addr":"172.21.15.16:6821/83361","state":["exists","up"]},{"osd":3,"uuid":"d285b4d5-0bef-421a-b84a-943e485b10c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":87,"up_from":133,"up_thru":234,"down_at":88,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":99977},{"type":"v1","addr":"172.21.15.22:6809","nonce":99977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":99977},{"type":"v1","addr":"172.21.15.22:6811","nonce":99977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":99977},{"type":"v1","addr":"172.21.15.22:6815","nonce":99977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":99977},{"type":"v1","addr":"172.21.15.22:6813","nonce":99977}]},"public_addr":"172.21.15.22:6809/99977","cluster_addr":"172.21.15.22:6811/99977","heartbeat_back_addr":"172.21.15.22:6815/99977","heartbeat_front_addr":"172.21.15.22:6813/99977","state":["exists","up"]},{"osd":4,"uuid":"7ba0b0db-e407-4c63-8278-aab09f3082c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":83351},{"type":"v1","addr":"172.21.15.22:6801","nonce":83351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":83351},{"type":"v1","addr":"172.21.15.22:6803","nonce":83351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":83351},{"type":"v1","addr":"172.21.15.22:6807","nonce":83351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":83351},{"type":"v1","addr":"172.21.15.22:6805","nonce":83351}]},"public_addr":"172.21.15.22:6801/83351","cluster_addr":"172.21.15.22:6803/83351","heartbeat_back_addr":"172.21.15.22:6807/83351","heartbeat_front_addr":"172.21.15.22:6805/83351","state":["exists","up"]},{"osd":5,"uuid":"29663bfa-f6fe-4ce7-b45a-a9b7b4424c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":238,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":83352},{"type":"v1","addr":"172.21.15.22:6817","nonce":83352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":83352},{"type":"v1","addr":"172.21.15.22:6819","nonce":83352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":83352},{"type":"v1","addr":"172.21.15.22:6823","nonce":83352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":83352},{"type":"v1","addr":"172.21.15.22:6821","nonce":83352}]},"public_addr":"172.21.15.22:6817/83352","cluster_addr":"172.21.15.22:6819/83352","heartbeat_back_addr":"172.21.15.22:6823/83352","heartbeat_front_addr":"172.21.15.22:6821/83352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.001178+0000","dead_epoch":149},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.184181+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.274854+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.186217+0000","dead_epoch":88},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.247442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.220075+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[0,3]},{"pgid":"2.0","osds":[4,1]},{"pgid":"2.2","osds":[1,5]},{"pgid":"2.4","osds":[5,0]}],"pg_upmap_items":[],"pg_temp":[{"pgid":"2.4","osds":[1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-02T14:07:46.255 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-07-02T14:07:46.458 INFO:teuthology.orchestra.run.smithi016.stdout:712964571161 2023-07-02T14:07:46.458 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-07-02T14:07:46.658 INFO:teuthology.orchestra.run.smithi016.stdout:47244640331 2023-07-02T14:07:46.659 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-07-02T14:07:46.857 INFO:teuthology.orchestra.run.smithi016.stdout:47244640331 2023-07-02T14:07:46.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-07-02T14:07:47.059 INFO:teuthology.orchestra.run.smithi016.stdout:571230650402 2023-07-02T14:07:47.059 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-07-02T14:07:47.254 INFO:teuthology.orchestra.run.smithi016.stdout:47244640331 2023-07-02T14:07:47.255 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-07-02T14:07:47.452 INFO:teuthology.orchestra.run.smithi016.stdout:47244640331 2023-07-02T14:07:47.453 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-07-02T14:07:47.759 INFO:teuthology.orchestra.run.smithi016.stdout:712964571159 2023-07-02T14:07:47.773 INFO:tasks.ceph.ceph_manager.ceph:need seq 712964571161 got 712964571159 for osd.0 2023-07-02T14:07:48.774 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-07-02T14:07:49.085 INFO:teuthology.orchestra.run.smithi016.stdout:712964571161 2023-07-02T14:07:49.098 INFO:tasks.ceph.ceph_manager.ceph:need seq 712964571161 got 712964571161 for osd.0 2023-07-02T14:07:49.099 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-07-02T14:07:49.402 INFO:teuthology.orchestra.run.smithi016.stdout:47244640331 2023-07-02T14:07:49.416 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640331 got 47244640331 for osd.1 2023-07-02T14:07:49.416 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-07-02T14:07:49.721 INFO:teuthology.orchestra.run.smithi016.stdout:47244640331 2023-07-02T14:07:49.735 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640331 got 47244640331 for osd.2 2023-07-02T14:07:49.736 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-07-02T14:07:50.041 INFO:teuthology.orchestra.run.smithi016.stdout:571230650403 2023-07-02T14:07:50.055 INFO:tasks.ceph.ceph_manager.ceph:need seq 571230650402 got 571230650403 for osd.3 2023-07-02T14:07:50.055 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-07-02T14:07:50.357 INFO:teuthology.orchestra.run.smithi016.stdout:47244640331 2023-07-02T14:07:50.370 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640331 got 47244640331 for osd.4 2023-07-02T14:07:50.371 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-07-02T14:07:50.691 INFO:teuthology.orchestra.run.smithi016.stdout:47244640331 2023-07-02T14:07:50.705 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640331 got 47244640331 for osd.5 2023-07-02T14:07:50.705 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-07-02T14:07:50.706 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:07:51.005 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:51.007 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:07:51.018 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2023-07-02T14:07:50.298241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":178,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90838,"num_read_kb":15854726,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5219,"num_bytes_recovered":17639070407,"num_keys_recovered":203,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27030,"ondisk_log_size":27030,"up":18,"acting":17,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":28351368,"kb_used_data":27913880,"kb_used_omap":1114,"kb_used_meta":435301,"kb_avail":537879672,"statfs":{"total":579820584960,"available":550788784128,"internally_reserved":0,"allocated":28583813120,"data_stored":33216884049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1141261,"internal_metadata":445748723},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-358,"num_objects_misplaced":-544,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":2099713296,"num_keys_recovered":4,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.895415"},"pg_stats":[{"pgid":"2.7","version":"222'3704","reported_seq":9543,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.405172+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:07:44.405172+0000","last_peered":"2023-07-02T14:07:44.405172+0000","last_clean":"2023-07-02T14:07:44.405172+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:07:44.405172+0000","last_undegraded":"2023-07-02T14:07:44.405172+0000","last_fullsized":"2023-07-02T14:07:44.405172+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3704,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1575041492,"num_objects":472,"num_object_clones":397,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":59417,"num_read_kb":2052552,"num_write":3144,"num_write_kb":1530742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"216'3069","reported_seq":5259,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.409689+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:07:44.409689+0000","last_peered":"2023-07-02T14:07:44.409689+0000","last_clean":"2023-07-02T14:07:44.409689+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:07:44.409689+0000","last_undegraded":"2023-07-02T14:07:44.409689+0000","last_fullsized":"2023-07-02T14:07:44.409689+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3069,"ondisk_log_size":3069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1538467250,"num_objects":486,"num_object_clones":415,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":2008,"num_read_kb":2154143,"num_write":2654,"num_write_kb":1492749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"221'3210","reported_seq":5689,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.405167+0000","last_change":"2023-07-02T14:07:41.397514+0000","last_active":"2023-07-02T14:07:44.405167+0000","last_peered":"2023-07-02T14:07:44.405167+0000","last_clean":"2023-07-02T14:07:44.405167+0000","last_became_active":"2023-07-02T14:07:41.396115+0000","last_became_peered":"2023-07-02T14:07:41.396115+0000","last_unstale":"2023-07-02T14:07:44.405167+0000","last_undegraded":"2023-07-02T14:07:44.405167+0000","last_fullsized":"2023-07-02T14:07:44.405167+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3210,"ondisk_log_size":3210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1517142988,"num_objects":428,"num_object_clones":355,"num_object_copies":856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":743,"num_read_kb":1976610,"num_write":2851,"num_write_kb":1467899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1078,"num_bytes_recovered":3879069754,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"216'6494","reported_seq":9011,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.405769+0000","last_change":"2023-07-02T14:07:40.398546+0000","last_active":"2023-07-02T14:07:44.405769+0000","last_peered":"2023-07-02T14:07:44.405769+0000","last_clean":"2023-07-02T14:07:44.405769+0000","last_became_active":"2023-07-02T14:07:40.395869+0000","last_became_peered":"2023-07-02T14:07:40.395869+0000","last_unstale":"2023-07-02T14:07:44.405769+0000","last_undegraded":"2023-07-02T14:07:44.405769+0000","last_fullsized":"2023-07-02T14:07:44.405769+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":6494,"ondisk_log_size":6494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1675535810,"num_objects":495,"num_object_clones":414,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":23017,"num_read_kb":2209627,"num_write":9605,"num_write_kb":1635009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":20,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.415234+0000","last_change":"2023-07-02T14:07:34.365379+0000","last_active":"2023-07-02T14:07:44.415234+0000","last_peered":"2023-07-02T14:07:44.415234+0000","last_clean":"2023-07-02T14:07:44.415234+0000","last_became_active":"2023-07-02T14:07:34.365070+0000","last_became_peered":"2023-07-02T14:07:34.365070+0000","last_unstale":"2023-07-02T14:07:44.415234+0000","last_undegraded":"2023-07-02T14:07:44.415234+0000","last_fullsized":"2023-07-02T14:07:44.415234+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"216'2975","reported_seq":4504,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-07-02T14:07:45.413573+0000","last_change":"2023-07-02T14:07:45.413573+0000","last_active":"2023-07-02T14:07:45.413573+0000","last_peered":"2023-07-02T14:07:45.413573+0000","last_clean":"2023-07-02T14:07:45.413573+0000","last_became_active":"2023-07-02T14:07:45.412476+0000","last_became_peered":"2023-07-02T14:07:45.412476+0000","last_unstale":"2023-07-02T14:07:45.413573+0000","last_undegraded":"2023-07-02T14:07:45.413573+0000","last_fullsized":"2023-07-02T14:07:45.413573+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2975,"ondisk_log_size":2975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1661289598,"num_objects":499,"num_object_clones":416,"num_object_copies":998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":1528,"num_read_kb":2088102,"num_write":2559,"num_write_kb":1611276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":2494793202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"216'2884","reported_seq":3864,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.404722+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:07:44.404722+0000","last_peered":"2023-07-02T14:07:44.404722+0000","last_clean":"2023-07-02T14:07:44.404722+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:07:44.404722+0000","last_undegraded":"2023-07-02T14:07:44.404722+0000","last_fullsized":"2023-07-02T14:07:44.404722+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2884,"ondisk_log_size":2884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1569543764,"num_objects":461,"num_object_clones":383,"num_object_copies":922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":1039,"num_read_kb":2026251,"num_write":2501,"num_write_kb":1523310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"216'2328","reported_seq":6065,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.409671+0000","last_change":"2023-07-02T14:07:40.399680+0000","last_active":"2023-07-02T14:07:44.409671+0000","last_peered":"2023-07-02T14:07:44.409671+0000","last_clean":"2023-07-02T14:07:44.409671+0000","last_became_active":"2023-07-02T14:07:40.398277+0000","last_became_peered":"2023-07-02T14:07:40.398277+0000","last_unstale":"2023-07-02T14:07:44.409671+0000","last_undegraded":"2023-07-02T14:07:44.409671+0000","last_fullsized":"2023-07-02T14:07:44.409671+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2328,"ondisk_log_size":2328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1230645001,"num_objects":405,"num_object_clones":325,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":1983,"num_read_kb":1773668,"num_write":2003,"num_write_kb":1186825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":904,"num_bytes_recovered":2809445763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"218'2366","reported_seq":5080,"reported_epoch":239,"state":"active+recovering+undersized+remapped","last_fresh":"2023-07-02T14:07:46.633604+0000","last_change":"2023-07-02T14:07:43.741538+0000","last_active":"2023-07-02T14:07:46.633604+0000","last_peered":"2023-07-02T14:07:46.633604+0000","last_clean":"2023-07-02T14:07:32.354728+0000","last_became_active":"2023-07-02T14:07:35.595705+0000","last_became_peered":"2023-07-02T14:07:35.595705+0000","last_unstale":"2023-07-02T14:07:46.633604+0000","last_undegraded":"2023-07-02T14:07:46.633604+0000","last_fullsized":"2023-07-02T14:07:35.367623+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2366,"ondisk_log_size":2366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1146537804,"num_objects":358,"num_object_clones":299,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":178,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":1103,"num_read_kb":1573773,"num_write":2067,"num_write_kb":1113633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":703,"num_bytes_recovered":2331465138,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[1],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1","objects":269},{"shards":"1,4","objects":89}],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":178,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90838,"num_read_kb":15854726,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5219,"num_bytes_recovered":17639070407,"num_keys_recovered":203,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28955799552,"data_stored":33654465805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":27030,"ondisk_log_size":27030,"up":16,"acting":15,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640331,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3910404,"kb_used_data":3874108,"kb_used_omap":0,"kb_used_meta":36288,"kb_avail":90461436,"statfs":{"total":96636764160,"available":92632510464,"internally_reserved":0,"allocated":3967086592,"data_stored":4641637735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37158912},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.251},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.26900000000000002}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.42899999999999999,"15min":0.42699999999999999},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.0509999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.45000000000000001,"15min":0.45700000000000002},"min":{"1min":0.28299999999999997,"5min":0.224,"15min":0.224},"max":{"1min":0.80700000000000005,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":0.34300000000000003}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.43099999999999999,"15min":0.42699999999999999},"min":{"1min":0.29699999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.1850000000000001,"5min":2.157,"15min":2.157},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.439,"15min":0.44900000000000001},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73699999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.36899999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.38100000000000001,"15min":0.36299999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.2919999999999998,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.36399999999999999,"15min":0.35899999999999999},"min":{"1min":0.28000000000000003,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.39600000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.39800000000000002,"15min":0.41699999999999998},"min":{"1min":0.27600000000000002,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.99299999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.42899999999999999},"min":{"1min":0.29499999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.65700000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.44700000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640331,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5931760,"kb_used_data":5836520,"kb_used_omap":0,"kb_used_meta":95232,"kb_avail":88440080,"statfs":{"total":96636764160,"available":90562641920,"internally_reserved":0,"allocated":5976596480,"data_stored":6835270655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":97517568},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.30399999999999999}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44,"15min":0.46400000000000002},"min":{"1min":0.28399999999999997,"5min":0.215,"15min":0.215},"max":{"1min":0.71499999999999997,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.45000000000000001},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.35599999999999998}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.32800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66800000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.38300000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.38300000000000001},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45900000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.36799999999999999,"15min":0.36299999999999999},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.48799999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.33300000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.44900000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.433,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59099999999999997,"5min":1.026,"15min":1.026},"last":0.318}]}]},{"osd":3,"up_from":133,"seq":571230650403,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4569460,"kb_used_data":4499052,"kb_used_omap":229,"kb_used_meta":70170,"kb_avail":89802380,"statfs":{"total":96636764160,"available":91957637120,"internally_reserved":0,"allocated":4607029248,"data_stored":5358008840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":71854302},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.30299999999999999}]},{"osd":1,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.372,"15min":0.36599999999999999},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.28199999999999997}]},{"osd":2,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.28699999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.318},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.44900000000000001,"15min":0.439},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.34300000000000003}]},{"osd":4,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.35799999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.36199999999999999,"15min":0.35299999999999998},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.38200000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.39800000000000002},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.372},{"interface":"front","average":{"1min":0.439,"5min":0.39500000000000002,"15min":0.38800000000000001},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.42299999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640331,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3200812,"kb_used_data":3146532,"kb_used_omap":0,"kb_used_meta":54272,"kb_avail":91171028,"statfs":{"total":96636764160,"available":93359132672,"internally_reserved":0,"allocated":3222048768,"data_stored":3710443912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55574528},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.314},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.26400000000000001}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.42499999999999999,"15min":0.374},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.376,"5min":0.33500000000000002,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.30199999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:07:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.91100000000000003,"5min":1.929,"15min":1.929},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.497,"5min":0.54800000000000004,"15min":0.55600000000000005},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93999999999999995,"5min":2,"15min":2},"last":0.53400000000000003}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.375}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68400000000000005,"5min":1.071,"15min":1.071},"last":0.501},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":0.47399999999999998}]}]},{"osd":0,"up_from":166,"seq":712964571161,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2049348,"kb_used_data":1999164,"kb_used_omap":884,"kb_used_meta":48267,"kb_avail":92322492,"statfs":{"total":96636764160,"available":94538231808,"internally_reserved":0,"allocated":2047143936,"data_stored":2306346250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":905963,"internal_metadata":49425685},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.27500000000000002}]},{"osd":2,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.30399999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.442},{"interface":"front","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.34399999999999997}]},{"osd":4,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.375}]},{"osd":5,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.38900000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640331,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8689584,"kb_used_data":8558504,"kb_used_omap":0,"kb_used_meta":131072,"kb_avail":85682256,"statfs":{"total":96636764160,"available":87738630144,"internally_reserved":0,"allocated":8763908096,"data_stored":10365176657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":134217728},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.19},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.27400000000000002}]},{"osd":2,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.224,"5min":0.184,"15min":0.184},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.33700000000000002}]},{"osd":3,"last update":"Sun Jul 2 14:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.49099999999999999,"15min":0.47299999999999998},"min":{"1min":0.30299999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.40600000000000003}]},{"osd":4,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71899999999999997,"5min":1.524,"15min":1.524},"last":0.374}]},{"osd":5,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.80300000000000005,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.441},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84399999999999997,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.46700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2324832256,"data_stored":2632939092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8761016320,"data_stored":10364040347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3219156992,"data_stored":3709307602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4604137472,"data_stored":5356872530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5973704704,"data_stored":6834134345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4072951808,"data_stored":4757171889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:07:51.020 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:07:51.317 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:51.319 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:07:51.330 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2023-07-02T14:07:50.298241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":178,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90838,"num_read_kb":15854726,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5219,"num_bytes_recovered":17639070407,"num_keys_recovered":203,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27030,"ondisk_log_size":27030,"up":18,"acting":17,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":28351368,"kb_used_data":27913880,"kb_used_omap":1114,"kb_used_meta":435301,"kb_avail":537879672,"statfs":{"total":579820584960,"available":550788784128,"internally_reserved":0,"allocated":28583813120,"data_stored":33216884049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1141261,"internal_metadata":445748723},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-358,"num_objects_misplaced":-544,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":2099713296,"num_keys_recovered":4,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.895415"},"pg_stats":[{"pgid":"2.7","version":"222'3704","reported_seq":9543,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.405172+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:07:44.405172+0000","last_peered":"2023-07-02T14:07:44.405172+0000","last_clean":"2023-07-02T14:07:44.405172+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:07:44.405172+0000","last_undegraded":"2023-07-02T14:07:44.405172+0000","last_fullsized":"2023-07-02T14:07:44.405172+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3704,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1575041492,"num_objects":472,"num_object_clones":397,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":59417,"num_read_kb":2052552,"num_write":3144,"num_write_kb":1530742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"216'3069","reported_seq":5259,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.409689+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:07:44.409689+0000","last_peered":"2023-07-02T14:07:44.409689+0000","last_clean":"2023-07-02T14:07:44.409689+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:07:44.409689+0000","last_undegraded":"2023-07-02T14:07:44.409689+0000","last_fullsized":"2023-07-02T14:07:44.409689+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3069,"ondisk_log_size":3069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1538467250,"num_objects":486,"num_object_clones":415,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":2008,"num_read_kb":2154143,"num_write":2654,"num_write_kb":1492749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"221'3210","reported_seq":5689,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.405167+0000","last_change":"2023-07-02T14:07:41.397514+0000","last_active":"2023-07-02T14:07:44.405167+0000","last_peered":"2023-07-02T14:07:44.405167+0000","last_clean":"2023-07-02T14:07:44.405167+0000","last_became_active":"2023-07-02T14:07:41.396115+0000","last_became_peered":"2023-07-02T14:07:41.396115+0000","last_unstale":"2023-07-02T14:07:44.405167+0000","last_undegraded":"2023-07-02T14:07:44.405167+0000","last_fullsized":"2023-07-02T14:07:44.405167+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3210,"ondisk_log_size":3210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1517142988,"num_objects":428,"num_object_clones":355,"num_object_copies":856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":743,"num_read_kb":1976610,"num_write":2851,"num_write_kb":1467899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1078,"num_bytes_recovered":3879069754,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"216'6494","reported_seq":9011,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.405769+0000","last_change":"2023-07-02T14:07:40.398546+0000","last_active":"2023-07-02T14:07:44.405769+0000","last_peered":"2023-07-02T14:07:44.405769+0000","last_clean":"2023-07-02T14:07:44.405769+0000","last_became_active":"2023-07-02T14:07:40.395869+0000","last_became_peered":"2023-07-02T14:07:40.395869+0000","last_unstale":"2023-07-02T14:07:44.405769+0000","last_undegraded":"2023-07-02T14:07:44.405769+0000","last_fullsized":"2023-07-02T14:07:44.405769+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":6494,"ondisk_log_size":6494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1675535810,"num_objects":495,"num_object_clones":414,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":23017,"num_read_kb":2209627,"num_write":9605,"num_write_kb":1635009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":20,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.415234+0000","last_change":"2023-07-02T14:07:34.365379+0000","last_active":"2023-07-02T14:07:44.415234+0000","last_peered":"2023-07-02T14:07:44.415234+0000","last_clean":"2023-07-02T14:07:44.415234+0000","last_became_active":"2023-07-02T14:07:34.365070+0000","last_became_peered":"2023-07-02T14:07:34.365070+0000","last_unstale":"2023-07-02T14:07:44.415234+0000","last_undegraded":"2023-07-02T14:07:44.415234+0000","last_fullsized":"2023-07-02T14:07:44.415234+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"216'2975","reported_seq":4504,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-07-02T14:07:45.413573+0000","last_change":"2023-07-02T14:07:45.413573+0000","last_active":"2023-07-02T14:07:45.413573+0000","last_peered":"2023-07-02T14:07:45.413573+0000","last_clean":"2023-07-02T14:07:45.413573+0000","last_became_active":"2023-07-02T14:07:45.412476+0000","last_became_peered":"2023-07-02T14:07:45.412476+0000","last_unstale":"2023-07-02T14:07:45.413573+0000","last_undegraded":"2023-07-02T14:07:45.413573+0000","last_fullsized":"2023-07-02T14:07:45.413573+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2975,"ondisk_log_size":2975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1661289598,"num_objects":499,"num_object_clones":416,"num_object_copies":998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":1528,"num_read_kb":2088102,"num_write":2559,"num_write_kb":1611276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":2494793202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"216'2884","reported_seq":3864,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.404722+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:07:44.404722+0000","last_peered":"2023-07-02T14:07:44.404722+0000","last_clean":"2023-07-02T14:07:44.404722+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:07:44.404722+0000","last_undegraded":"2023-07-02T14:07:44.404722+0000","last_fullsized":"2023-07-02T14:07:44.404722+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2884,"ondisk_log_size":2884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1569543764,"num_objects":461,"num_object_clones":383,"num_object_copies":922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":1039,"num_read_kb":2026251,"num_write":2501,"num_write_kb":1523310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"216'2328","reported_seq":6065,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.409671+0000","last_change":"2023-07-02T14:07:40.399680+0000","last_active":"2023-07-02T14:07:44.409671+0000","last_peered":"2023-07-02T14:07:44.409671+0000","last_clean":"2023-07-02T14:07:44.409671+0000","last_became_active":"2023-07-02T14:07:40.398277+0000","last_became_peered":"2023-07-02T14:07:40.398277+0000","last_unstale":"2023-07-02T14:07:44.409671+0000","last_undegraded":"2023-07-02T14:07:44.409671+0000","last_fullsized":"2023-07-02T14:07:44.409671+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2328,"ondisk_log_size":2328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1230645001,"num_objects":405,"num_object_clones":325,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":1983,"num_read_kb":1773668,"num_write":2003,"num_write_kb":1186825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":904,"num_bytes_recovered":2809445763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"218'2366","reported_seq":5080,"reported_epoch":239,"state":"active+recovering+undersized+remapped","last_fresh":"2023-07-02T14:07:46.633604+0000","last_change":"2023-07-02T14:07:43.741538+0000","last_active":"2023-07-02T14:07:46.633604+0000","last_peered":"2023-07-02T14:07:46.633604+0000","last_clean":"2023-07-02T14:07:32.354728+0000","last_became_active":"2023-07-02T14:07:35.595705+0000","last_became_peered":"2023-07-02T14:07:35.595705+0000","last_unstale":"2023-07-02T14:07:46.633604+0000","last_undegraded":"2023-07-02T14:07:46.633604+0000","last_fullsized":"2023-07-02T14:07:35.367623+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2366,"ondisk_log_size":2366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1146537804,"num_objects":358,"num_object_clones":299,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":178,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":1103,"num_read_kb":1573773,"num_write":2067,"num_write_kb":1113633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":703,"num_bytes_recovered":2331465138,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[1],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1","objects":269},{"shards":"1,4","objects":89}],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":178,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90838,"num_read_kb":15854726,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5219,"num_bytes_recovered":17639070407,"num_keys_recovered":203,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28955799552,"data_stored":33654465805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":27030,"ondisk_log_size":27030,"up":16,"acting":15,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640331,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3910404,"kb_used_data":3874108,"kb_used_omap":0,"kb_used_meta":36288,"kb_avail":90461436,"statfs":{"total":96636764160,"available":92632510464,"internally_reserved":0,"allocated":3967086592,"data_stored":4641637735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37158912},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.251},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.26900000000000002}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.42899999999999999,"15min":0.42699999999999999},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.0509999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.45000000000000001,"15min":0.45700000000000002},"min":{"1min":0.28299999999999997,"5min":0.224,"15min":0.224},"max":{"1min":0.80700000000000005,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":0.34300000000000003}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.43099999999999999,"15min":0.42699999999999999},"min":{"1min":0.29699999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.1850000000000001,"5min":2.157,"15min":2.157},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.439,"15min":0.44900000000000001},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73699999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.36899999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.38100000000000001,"15min":0.36299999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.2919999999999998,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.36399999999999999,"15min":0.35899999999999999},"min":{"1min":0.28000000000000003,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.39600000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.39800000000000002,"15min":0.41699999999999998},"min":{"1min":0.27600000000000002,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.99299999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.42899999999999999},"min":{"1min":0.29499999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.65700000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.44700000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640331,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5931760,"kb_used_data":5836520,"kb_used_omap":0,"kb_used_meta":95232,"kb_avail":88440080,"statfs":{"total":96636764160,"available":90562641920,"internally_reserved":0,"allocated":5976596480,"data_stored":6835270655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":97517568},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.30399999999999999}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44,"15min":0.46400000000000002},"min":{"1min":0.28399999999999997,"5min":0.215,"15min":0.215},"max":{"1min":0.71499999999999997,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.45000000000000001},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.35599999999999998}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.32800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66800000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.38300000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.38300000000000001},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45900000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.36799999999999999,"15min":0.36299999999999999},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.48799999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.33300000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.44900000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.433,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59099999999999997,"5min":1.026,"15min":1.026},"last":0.318}]}]},{"osd":3,"up_from":133,"seq":571230650403,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4569460,"kb_used_data":4499052,"kb_used_omap":229,"kb_used_meta":70170,"kb_avail":89802380,"statfs":{"total":96636764160,"available":91957637120,"internally_reserved":0,"allocated":4607029248,"data_stored":5358008840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":71854302},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.30299999999999999}]},{"osd":1,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.372,"15min":0.36599999999999999},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.28199999999999997}]},{"osd":2,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.28699999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.318},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.44900000000000001,"15min":0.439},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.34300000000000003}]},{"osd":4,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.35799999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.36199999999999999,"15min":0.35299999999999998},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.38200000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.39800000000000002},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.372},{"interface":"front","average":{"1min":0.439,"5min":0.39500000000000002,"15min":0.38800000000000001},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.42299999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640331,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3200812,"kb_used_data":3146532,"kb_used_omap":0,"kb_used_meta":54272,"kb_avail":91171028,"statfs":{"total":96636764160,"available":93359132672,"internally_reserved":0,"allocated":3222048768,"data_stored":3710443912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55574528},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.314},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.26400000000000001}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.42499999999999999,"15min":0.374},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.376,"5min":0.33500000000000002,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.30199999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:07:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.91100000000000003,"5min":1.929,"15min":1.929},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.497,"5min":0.54800000000000004,"15min":0.55600000000000005},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93999999999999995,"5min":2,"15min":2},"last":0.53400000000000003}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.375}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68400000000000005,"5min":1.071,"15min":1.071},"last":0.501},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":0.47399999999999998}]}]},{"osd":0,"up_from":166,"seq":712964571161,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2049348,"kb_used_data":1999164,"kb_used_omap":884,"kb_used_meta":48267,"kb_avail":92322492,"statfs":{"total":96636764160,"available":94538231808,"internally_reserved":0,"allocated":2047143936,"data_stored":2306346250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":905963,"internal_metadata":49425685},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.27500000000000002}]},{"osd":2,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.30399999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.442},{"interface":"front","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.34399999999999997}]},{"osd":4,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.375}]},{"osd":5,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.38900000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640331,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8689584,"kb_used_data":8558504,"kb_used_omap":0,"kb_used_meta":131072,"kb_avail":85682256,"statfs":{"total":96636764160,"available":87738630144,"internally_reserved":0,"allocated":8763908096,"data_stored":10365176657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":134217728},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.19},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.27400000000000002}]},{"osd":2,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.224,"5min":0.184,"15min":0.184},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.33700000000000002}]},{"osd":3,"last update":"Sun Jul 2 14:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.49099999999999999,"15min":0.47299999999999998},"min":{"1min":0.30299999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.40600000000000003}]},{"osd":4,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71899999999999997,"5min":1.524,"15min":1.524},"last":0.374}]},{"osd":5,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.80300000000000005,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.441},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84399999999999997,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.46700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2324832256,"data_stored":2632939092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8761016320,"data_stored":10364040347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3219156992,"data_stored":3709307602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4604137472,"data_stored":5356872530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5973704704,"data_stored":6834134345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4072951808,"data_stored":4757171889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:07:51.332 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-07-02T14:07:51.718 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:51.718 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":374,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":3},"osdmap":{"epoch":241,"num_osds":6,"num_up_osds":6,"osd_up_since":1688306752,"num_in_osds":6,"osd_in_since":1688306853,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":3604,"data_bytes":11914203707,"bytes_used":29031800832,"bytes_avail":550788784128,"bytes_total":579820584960,"misplaced_objects":178,"misplaced_total":7208,"misplaced_ratio":0.024694783573806882,"recovering_objects_per_sec":107,"recovering_bytes_per_sec":356160369,"recovering_keys_per_sec":0,"num_objects_recovered":633,"num_bytes_recovered":2099713296,"num_keys_recovered":4},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2023-07-02T14:06:52.287392+0000","services":{}},"progress_events":{"17fed3b2-40b5-4240-87fc-c08a0db77ab2":{"message":"Global Recovery Event (10s)\n [=====================.......] (remaining: 2s)","progress":0.77777779102325439,"add_to_ceph_s":true}}} 2023-07-02T14:07:51.732 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2023-07-02T14:07:51.732 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:07:52.029 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:52.031 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:07:52.042 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2023-07-02T14:07:50.298241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":178,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90838,"num_read_kb":15854726,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5219,"num_bytes_recovered":17639070407,"num_keys_recovered":203,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27030,"ondisk_log_size":27030,"up":18,"acting":17,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":28351368,"kb_used_data":27913880,"kb_used_omap":1114,"kb_used_meta":435301,"kb_avail":537879672,"statfs":{"total":579820584960,"available":550788784128,"internally_reserved":0,"allocated":28583813120,"data_stored":33216884049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1141261,"internal_metadata":445748723},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-358,"num_objects_misplaced":-544,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":633,"num_bytes_recovered":2099713296,"num_keys_recovered":4,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.895415"},"pg_stats":[{"pgid":"2.7","version":"222'3704","reported_seq":9543,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.405172+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:07:44.405172+0000","last_peered":"2023-07-02T14:07:44.405172+0000","last_clean":"2023-07-02T14:07:44.405172+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:07:44.405172+0000","last_undegraded":"2023-07-02T14:07:44.405172+0000","last_fullsized":"2023-07-02T14:07:44.405172+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3704,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1575041492,"num_objects":472,"num_object_clones":397,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":59417,"num_read_kb":2052552,"num_write":3144,"num_write_kb":1530742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"216'3069","reported_seq":5259,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.409689+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:07:44.409689+0000","last_peered":"2023-07-02T14:07:44.409689+0000","last_clean":"2023-07-02T14:07:44.409689+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:07:44.409689+0000","last_undegraded":"2023-07-02T14:07:44.409689+0000","last_fullsized":"2023-07-02T14:07:44.409689+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3069,"ondisk_log_size":3069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1538467250,"num_objects":486,"num_object_clones":415,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":2008,"num_read_kb":2154143,"num_write":2654,"num_write_kb":1492749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"221'3210","reported_seq":5689,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.405167+0000","last_change":"2023-07-02T14:07:41.397514+0000","last_active":"2023-07-02T14:07:44.405167+0000","last_peered":"2023-07-02T14:07:44.405167+0000","last_clean":"2023-07-02T14:07:44.405167+0000","last_became_active":"2023-07-02T14:07:41.396115+0000","last_became_peered":"2023-07-02T14:07:41.396115+0000","last_unstale":"2023-07-02T14:07:44.405167+0000","last_undegraded":"2023-07-02T14:07:44.405167+0000","last_fullsized":"2023-07-02T14:07:44.405167+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3210,"ondisk_log_size":3210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1517142988,"num_objects":428,"num_object_clones":355,"num_object_copies":856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":743,"num_read_kb":1976610,"num_write":2851,"num_write_kb":1467899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1078,"num_bytes_recovered":3879069754,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"216'6494","reported_seq":9011,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.405769+0000","last_change":"2023-07-02T14:07:40.398546+0000","last_active":"2023-07-02T14:07:44.405769+0000","last_peered":"2023-07-02T14:07:44.405769+0000","last_clean":"2023-07-02T14:07:44.405769+0000","last_became_active":"2023-07-02T14:07:40.395869+0000","last_became_peered":"2023-07-02T14:07:40.395869+0000","last_unstale":"2023-07-02T14:07:44.405769+0000","last_undegraded":"2023-07-02T14:07:44.405769+0000","last_fullsized":"2023-07-02T14:07:44.405769+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":6494,"ondisk_log_size":6494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1675535810,"num_objects":495,"num_object_clones":414,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":23017,"num_read_kb":2209627,"num_write":9605,"num_write_kb":1635009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":20,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.415234+0000","last_change":"2023-07-02T14:07:34.365379+0000","last_active":"2023-07-02T14:07:44.415234+0000","last_peered":"2023-07-02T14:07:44.415234+0000","last_clean":"2023-07-02T14:07:44.415234+0000","last_became_active":"2023-07-02T14:07:34.365070+0000","last_became_peered":"2023-07-02T14:07:34.365070+0000","last_unstale":"2023-07-02T14:07:44.415234+0000","last_undegraded":"2023-07-02T14:07:44.415234+0000","last_fullsized":"2023-07-02T14:07:44.415234+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"216'2975","reported_seq":4504,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-07-02T14:07:45.413573+0000","last_change":"2023-07-02T14:07:45.413573+0000","last_active":"2023-07-02T14:07:45.413573+0000","last_peered":"2023-07-02T14:07:45.413573+0000","last_clean":"2023-07-02T14:07:45.413573+0000","last_became_active":"2023-07-02T14:07:45.412476+0000","last_became_peered":"2023-07-02T14:07:45.412476+0000","last_unstale":"2023-07-02T14:07:45.413573+0000","last_undegraded":"2023-07-02T14:07:45.413573+0000","last_fullsized":"2023-07-02T14:07:45.413573+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2975,"ondisk_log_size":2975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1661289598,"num_objects":499,"num_object_clones":416,"num_object_copies":998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":1528,"num_read_kb":2088102,"num_write":2559,"num_write_kb":1611276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":2494793202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"216'2884","reported_seq":3864,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.404722+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:07:44.404722+0000","last_peered":"2023-07-02T14:07:44.404722+0000","last_clean":"2023-07-02T14:07:44.404722+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:07:44.404722+0000","last_undegraded":"2023-07-02T14:07:44.404722+0000","last_fullsized":"2023-07-02T14:07:44.404722+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2884,"ondisk_log_size":2884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1569543764,"num_objects":461,"num_object_clones":383,"num_object_copies":922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":1039,"num_read_kb":2026251,"num_write":2501,"num_write_kb":1523310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"216'2328","reported_seq":6065,"reported_epoch":238,"state":"active+clean","last_fresh":"2023-07-02T14:07:44.409671+0000","last_change":"2023-07-02T14:07:40.399680+0000","last_active":"2023-07-02T14:07:44.409671+0000","last_peered":"2023-07-02T14:07:44.409671+0000","last_clean":"2023-07-02T14:07:44.409671+0000","last_became_active":"2023-07-02T14:07:40.398277+0000","last_became_peered":"2023-07-02T14:07:40.398277+0000","last_unstale":"2023-07-02T14:07:44.409671+0000","last_undegraded":"2023-07-02T14:07:44.409671+0000","last_fullsized":"2023-07-02T14:07:44.409671+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2328,"ondisk_log_size":2328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1230645001,"num_objects":405,"num_object_clones":325,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":1983,"num_read_kb":1773668,"num_write":2003,"num_write_kb":1186825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":904,"num_bytes_recovered":2809445763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"218'2366","reported_seq":5080,"reported_epoch":239,"state":"active+recovering+undersized+remapped","last_fresh":"2023-07-02T14:07:46.633604+0000","last_change":"2023-07-02T14:07:43.741538+0000","last_active":"2023-07-02T14:07:46.633604+0000","last_peered":"2023-07-02T14:07:46.633604+0000","last_clean":"2023-07-02T14:07:32.354728+0000","last_became_active":"2023-07-02T14:07:35.595705+0000","last_became_peered":"2023-07-02T14:07:35.595705+0000","last_unstale":"2023-07-02T14:07:46.633604+0000","last_undegraded":"2023-07-02T14:07:46.633604+0000","last_fullsized":"2023-07-02T14:07:35.367623+0000","mapping_epoch":228,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2366,"ondisk_log_size":2366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1146537804,"num_objects":358,"num_object_clones":299,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":178,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":1103,"num_read_kb":1573773,"num_write":2067,"num_write_kb":1113633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":703,"num_bytes_recovered":2331465138,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[1],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1","objects":269},{"shards":"1,4","objects":89}],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":178,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90838,"num_read_kb":15854726,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5219,"num_bytes_recovered":17639070407,"num_keys_recovered":203,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28955799552,"data_stored":33654465805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":27030,"ondisk_log_size":27030,"up":16,"acting":15,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640331,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3910404,"kb_used_data":3874108,"kb_used_omap":0,"kb_used_meta":36288,"kb_avail":90461436,"statfs":{"total":96636764160,"available":92632510464,"internally_reserved":0,"allocated":3967086592,"data_stored":4641637735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37158912},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.251},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.26900000000000002}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.42899999999999999,"15min":0.42699999999999999},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.0509999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.45000000000000001,"15min":0.45700000000000002},"min":{"1min":0.28299999999999997,"5min":0.224,"15min":0.224},"max":{"1min":0.80700000000000005,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":0.34300000000000003}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.43099999999999999,"15min":0.42699999999999999},"min":{"1min":0.29699999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.1850000000000001,"5min":2.157,"15min":2.157},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.439,"15min":0.44900000000000001},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73699999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.36899999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.38100000000000001,"15min":0.36299999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.2919999999999998,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.36399999999999999,"15min":0.35899999999999999},"min":{"1min":0.28000000000000003,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.39600000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.39800000000000002,"15min":0.41699999999999998},"min":{"1min":0.27600000000000002,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.99299999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.42899999999999999},"min":{"1min":0.29499999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.65700000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.44700000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640331,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5931760,"kb_used_data":5836520,"kb_used_omap":0,"kb_used_meta":95232,"kb_avail":88440080,"statfs":{"total":96636764160,"available":90562641920,"internally_reserved":0,"allocated":5976596480,"data_stored":6835270655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":97517568},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.30399999999999999}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44,"15min":0.46400000000000002},"min":{"1min":0.28399999999999997,"5min":0.215,"15min":0.215},"max":{"1min":0.71499999999999997,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.45000000000000001},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.35599999999999998}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.32800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66800000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.38300000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.38300000000000001},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45900000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.36799999999999999,"15min":0.36299999999999999},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.48799999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.33300000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.44900000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.433,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59099999999999997,"5min":1.026,"15min":1.026},"last":0.318}]}]},{"osd":3,"up_from":133,"seq":571230650403,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4569460,"kb_used_data":4499052,"kb_used_omap":229,"kb_used_meta":70170,"kb_avail":89802380,"statfs":{"total":96636764160,"available":91957637120,"internally_reserved":0,"allocated":4607029248,"data_stored":5358008840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":71854302},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.30299999999999999}]},{"osd":1,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.372,"15min":0.36599999999999999},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.28199999999999997}]},{"osd":2,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.28699999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.318},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.44900000000000001,"15min":0.439},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.34300000000000003}]},{"osd":4,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.35799999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.36199999999999999,"15min":0.35299999999999998},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.38200000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.39800000000000002},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.372},{"interface":"front","average":{"1min":0.439,"5min":0.39500000000000002,"15min":0.38800000000000001},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.42299999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640331,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3200812,"kb_used_data":3146532,"kb_used_omap":0,"kb_used_meta":54272,"kb_avail":91171028,"statfs":{"total":96636764160,"available":93359132672,"internally_reserved":0,"allocated":3222048768,"data_stored":3710443912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55574528},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.314},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.26400000000000001}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.42499999999999999,"15min":0.374},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.376,"5min":0.33500000000000002,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.30199999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:07:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.91100000000000003,"5min":1.929,"15min":1.929},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.497,"5min":0.54800000000000004,"15min":0.55600000000000005},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93999999999999995,"5min":2,"15min":2},"last":0.53400000000000003}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.375}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68400000000000005,"5min":1.071,"15min":1.071},"last":0.501},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":0.47399999999999998}]}]},{"osd":0,"up_from":166,"seq":712964571161,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2049348,"kb_used_data":1999164,"kb_used_omap":884,"kb_used_meta":48267,"kb_avail":92322492,"statfs":{"total":96636764160,"available":94538231808,"internally_reserved":0,"allocated":2047143936,"data_stored":2306346250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":905963,"internal_metadata":49425685},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.27500000000000002}]},{"osd":2,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.30399999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.442},{"interface":"front","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.34399999999999997}]},{"osd":4,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.375}]},{"osd":5,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.38900000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640331,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8689584,"kb_used_data":8558504,"kb_used_omap":0,"kb_used_meta":131072,"kb_avail":85682256,"statfs":{"total":96636764160,"available":87738630144,"internally_reserved":0,"allocated":8763908096,"data_stored":10365176657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":134217728},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.19},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.27400000000000002}]},{"osd":2,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.224,"5min":0.184,"15min":0.184},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.33700000000000002}]},{"osd":3,"last update":"Sun Jul 2 14:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.49099999999999999,"15min":0.47299999999999998},"min":{"1min":0.30299999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.40600000000000003}]},{"osd":4,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71899999999999997,"5min":1.524,"15min":1.524},"last":0.374}]},{"osd":5,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.80300000000000005,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.441},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84399999999999997,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.46700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2324832256,"data_stored":2632939092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8761016320,"data_stored":10364040347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3219156992,"data_stored":3709307602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4604137472,"data_stored":5356872530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5973704704,"data_stored":6834134345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4072951808,"data_stored":4757171889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:07:55.045 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:07:55.359 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:55.361 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:07:55.372 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2023-07-02T14:07:54.299271+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90838,"num_read_kb":15854726,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5308,"num_bytes_recovered":17939292043,"num_keys_recovered":203,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27030,"ondisk_log_size":27030,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":28053364,"kb_used_data":27612164,"kb_used_omap":1114,"kb_used_meta":440037,"kb_avail":538177676,"statfs":{"total":579820584960,"available":551093940224,"internally_reserved":0,"allocated":28274855936,"data_stored":32834556229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1141261,"internal_metadata":450598387},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-446,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":722,"num_bytes_recovered":2417727972,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001738"},"pg_stats":[{"pgid":"2.7","version":"222'3704","reported_seq":9545,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.418578+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:07:48.418578+0000","last_peered":"2023-07-02T14:07:48.418578+0000","last_clean":"2023-07-02T14:07:48.418578+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:07:48.418578+0000","last_undegraded":"2023-07-02T14:07:48.418578+0000","last_fullsized":"2023-07-02T14:07:48.418578+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3704,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1575041492,"num_objects":472,"num_object_clones":397,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":59417,"num_read_kb":2052552,"num_write":3144,"num_write_kb":1530742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"216'3069","reported_seq":5261,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.421447+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:07:48.421447+0000","last_peered":"2023-07-02T14:07:48.421447+0000","last_clean":"2023-07-02T14:07:48.421447+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:07:48.421447+0000","last_undegraded":"2023-07-02T14:07:48.421447+0000","last_fullsized":"2023-07-02T14:07:48.421447+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3069,"ondisk_log_size":3069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1538467250,"num_objects":486,"num_object_clones":415,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":2008,"num_read_kb":2154143,"num_write":2654,"num_write_kb":1492749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"221'3210","reported_seq":5691,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.418575+0000","last_change":"2023-07-02T14:07:41.397514+0000","last_active":"2023-07-02T14:07:48.418575+0000","last_peered":"2023-07-02T14:07:48.418575+0000","last_clean":"2023-07-02T14:07:48.418575+0000","last_became_active":"2023-07-02T14:07:41.396115+0000","last_became_peered":"2023-07-02T14:07:41.396115+0000","last_unstale":"2023-07-02T14:07:48.418575+0000","last_undegraded":"2023-07-02T14:07:48.418575+0000","last_fullsized":"2023-07-02T14:07:48.418575+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3210,"ondisk_log_size":3210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1517142988,"num_objects":428,"num_object_clones":355,"num_object_copies":856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":743,"num_read_kb":1976610,"num_write":2851,"num_write_kb":1467899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1078,"num_bytes_recovered":3879069754,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"216'6494","reported_seq":9013,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.419242+0000","last_change":"2023-07-02T14:07:40.398546+0000","last_active":"2023-07-02T14:07:48.419242+0000","last_peered":"2023-07-02T14:07:48.419242+0000","last_clean":"2023-07-02T14:07:48.419242+0000","last_became_active":"2023-07-02T14:07:40.395869+0000","last_became_peered":"2023-07-02T14:07:40.395869+0000","last_unstale":"2023-07-02T14:07:48.419242+0000","last_undegraded":"2023-07-02T14:07:48.419242+0000","last_fullsized":"2023-07-02T14:07:48.419242+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":6494,"ondisk_log_size":6494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1675535810,"num_objects":495,"num_object_clones":414,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":23017,"num_read_kb":2209627,"num_write":9605,"num_write_kb":1635009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":22,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.424665+0000","last_change":"2023-07-02T14:07:34.365379+0000","last_active":"2023-07-02T14:07:48.424665+0000","last_peered":"2023-07-02T14:07:48.424665+0000","last_clean":"2023-07-02T14:07:48.424665+0000","last_became_active":"2023-07-02T14:07:34.365070+0000","last_became_peered":"2023-07-02T14:07:34.365070+0000","last_unstale":"2023-07-02T14:07:48.424665+0000","last_undegraded":"2023-07-02T14:07:48.424665+0000","last_fullsized":"2023-07-02T14:07:48.424665+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"216'2975","reported_seq":4506,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.422315+0000","last_change":"2023-07-02T14:07:45.413573+0000","last_active":"2023-07-02T14:07:48.422315+0000","last_peered":"2023-07-02T14:07:48.422315+0000","last_clean":"2023-07-02T14:07:48.422315+0000","last_became_active":"2023-07-02T14:07:45.412476+0000","last_became_peered":"2023-07-02T14:07:45.412476+0000","last_unstale":"2023-07-02T14:07:48.422315+0000","last_undegraded":"2023-07-02T14:07:48.422315+0000","last_fullsized":"2023-07-02T14:07:48.422315+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2975,"ondisk_log_size":2975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1661289598,"num_objects":499,"num_object_clones":416,"num_object_copies":998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":1528,"num_read_kb":2088102,"num_write":2559,"num_write_kb":1611276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":2494793202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"216'2884","reported_seq":3866,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.418819+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:07:48.418819+0000","last_peered":"2023-07-02T14:07:48.418819+0000","last_clean":"2023-07-02T14:07:48.418819+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:07:48.418819+0000","last_undegraded":"2023-07-02T14:07:48.418819+0000","last_fullsized":"2023-07-02T14:07:48.418819+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2884,"ondisk_log_size":2884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1569543764,"num_objects":461,"num_object_clones":383,"num_object_copies":922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":1039,"num_read_kb":2026251,"num_write":2501,"num_write_kb":1523310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"216'2328","reported_seq":6067,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.421423+0000","last_change":"2023-07-02T14:07:40.399680+0000","last_active":"2023-07-02T14:07:48.421423+0000","last_peered":"2023-07-02T14:07:48.421423+0000","last_clean":"2023-07-02T14:07:48.421423+0000","last_became_active":"2023-07-02T14:07:40.398277+0000","last_became_peered":"2023-07-02T14:07:40.398277+0000","last_unstale":"2023-07-02T14:07:48.421423+0000","last_undegraded":"2023-07-02T14:07:48.421423+0000","last_fullsized":"2023-07-02T14:07:48.421423+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2328,"ondisk_log_size":2328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1230645001,"num_objects":405,"num_object_clones":325,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":1983,"num_read_kb":1773668,"num_write":2003,"num_write_kb":1186825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":904,"num_bytes_recovered":2809445763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"218'2366","reported_seq":4268,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:07:49.426548+0000","last_change":"2023-07-02T14:07:49.426548+0000","last_active":"2023-07-02T14:07:49.426548+0000","last_peered":"2023-07-02T14:07:49.426548+0000","last_clean":"2023-07-02T14:07:49.426548+0000","last_became_active":"2023-07-02T14:07:49.425350+0000","last_became_peered":"2023-07-02T14:07:49.425350+0000","last_unstale":"2023-07-02T14:07:49.426548+0000","last_undegraded":"2023-07-02T14:07:49.426548+0000","last_fullsized":"2023-07-02T14:07:49.426548+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2366,"ondisk_log_size":2366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1146537804,"num_objects":358,"num_object_clones":299,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":1103,"num_read_kb":1573773,"num_write":2067,"num_write_kb":1113633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":792,"num_bytes_recovered":2631686774,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90838,"num_read_kb":15854726,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5308,"num_bytes_recovered":17939292043,"num_keys_recovered":203,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28257357824,"data_stored":32827681303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":27030,"ondisk_log_size":27030,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640332,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4026312,"kb_used_data":3989312,"kb_used_omap":0,"kb_used_meta":36992,"kb_avail":90345528,"statfs":{"total":96636764160,"available":92513820672,"internally_reserved":0,"allocated":4085055488,"data_stored":4773662384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37879808},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.44600000000000001}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.42899999999999999,"15min":0.42699999999999999},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.0509999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.45000000000000001,"15min":0.45700000000000002},"min":{"1min":0.28299999999999997,"5min":0.224,"15min":0.224},"max":{"1min":0.80700000000000005,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":0.46500000000000002}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.43099999999999999,"15min":0.42699999999999999},"min":{"1min":0.29699999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.1850000000000001,"5min":2.157,"15min":2.157},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.439,"15min":0.44900000000000001},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73699999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.39100000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.38100000000000001,"15min":0.36299999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.2919999999999998,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.314},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.36399999999999999,"15min":0.35899999999999999},"min":{"1min":0.28000000000000003,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.36699999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.39800000000000002,"15min":0.41699999999999998},"min":{"1min":0.27600000000000002,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.99299999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.42899999999999999},"min":{"1min":0.29499999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.65700000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.496}]}]},{"osd":4,"up_from":11,"seq":47244640332,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5265052,"kb_used_data":5169492,"kb_used_omap":0,"kb_used_meta":95552,"kb_avail":89106788,"statfs":{"total":96636764160,"available":91245350912,"internally_reserved":0,"allocated":5293559808,"data_stored":5993142242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":97845248},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.441}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44,"15min":0.46400000000000002},"min":{"1min":0.28399999999999997,"5min":0.215,"15min":0.215},"max":{"1min":0.71499999999999997,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.45000000000000001},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.53500000000000003}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.32800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66800000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.55500000000000005}]},{"osd":3,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.38300000000000001},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45900000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.36799999999999999,"15min":0.36299999999999999},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.48799999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.49099999999999999}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.44900000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.433,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59099999999999997,"5min":1.026,"15min":1.026},"last":0.47199999999999998}]}]},{"osd":3,"up_from":133,"seq":571230650404,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4569484,"kb_used_data":4499076,"kb_used_omap":229,"kb_used_meta":70170,"kb_avail":89802356,"statfs":{"total":96636764160,"available":91957612544,"internally_reserved":0,"allocated":4607053824,"data_stored":5358018351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":71854302},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.41099999999999998}]},{"osd":1,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.372,"15min":0.36599999999999999},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.44800000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.28699999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.44900000000000001,"15min":0.439},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.56299999999999994}]},{"osd":4,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.35799999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.498},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.36199999999999999,"15min":0.35299999999999998},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.59499999999999997}]},{"osd":5,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.39800000000000002},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.439,"5min":0.39500000000000002,"15min":0.38800000000000001},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.63}]}]},{"osd":2,"up_from":11,"seq":47244640332,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3200836,"kb_used_data":3146556,"kb_used_omap":0,"kb_used_meta":54272,"kb_avail":91171004,"statfs":{"total":96636764160,"available":93359108096,"internally_reserved":0,"allocated":3222073344,"data_stored":3710453423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55574528},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.17000000000000001},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.20300000000000001}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.42499999999999999,"15min":0.374},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":0.254},{"interface":"front","average":{"1min":0.376,"5min":0.33500000000000002,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.23999999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:07:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.91100000000000003,"5min":1.929,"15min":1.929},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.54800000000000004,"15min":0.55600000000000005},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93999999999999995,"5min":2,"15min":2},"last":0.29899999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.312},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.33700000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68400000000000005,"5min":1.071,"15min":1.071},"last":0.372},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":0.35799999999999998}]}]},{"osd":0,"up_from":166,"seq":712964571162,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2680400,"kb_used_data":2627784,"kb_used_omap":884,"kb_used_meta":51723,"kb_avail":91691440,"statfs":{"total":96636764160,"available":93892034560,"internally_reserved":0,"allocated":2690850816,"data_stored":3042367221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":905963,"internal_metadata":52964629},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.33600000000000002}]},{"osd":2,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.28999999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.438},{"interface":"front","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.377}]},{"osd":4,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.42999999999999999}]},{"osd":5,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.42299999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640332,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8311280,"kb_used_data":8179944,"kb_used_omap":0,"kb_used_meta":131328,"kb_avail":86060560,"statfs":{"total":96636764160,"available":88126013440,"internally_reserved":0,"allocated":8376262656,"data_stored":9956912608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":134479872},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.28799999999999998}]},{"osd":2,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.224,"5min":0.184,"15min":0.184},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.32400000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.49099999999999999,"15min":0.47299999999999998},"min":{"1min":0.30299999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.443}]},{"osd":4,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.433},{"interface":"front","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71899999999999997,"5min":1.524,"15min":1.524},"last":0.39600000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.80300000000000005,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84399999999999997,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.495}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2687934464,"data_stored":3041221400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8373346304,"data_stored":9955766787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3219156992,"data_stored":3709307602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4604137472,"data_stored":5356872530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5290643456,"data_stored":5991996421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4082139136,"data_stored":4772516563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:07:55.374 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-07-02T14:07:55.374 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2023-07-02T14:07:55.695 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-07-02T14:07:55.705 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-07-02T14:07:55.706 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-02T14:07:56.008 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:56.008 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":241,"fsid":"9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2","created":"2023-07-02T14:01:37.298060+0000","modified":"2023-07-02T14:07:49.415411+0000","last_up_change":"2023-07-02T14:05:52.397969+0000","last_in_change":"2023-07-02T14:07:33.350320+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-07-02T14:01:45.323455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-02T14:01:47.448897+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":69,"snap_epoch":217,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"43c446fe-c811-4ee3-99e8-d7452edfc1c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":148,"up_from":166,"up_thru":227,"down_at":149,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":115688},{"type":"v1","addr":"172.21.15.16:6801","nonce":115688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":115688},{"type":"v1","addr":"172.21.15.16:6803","nonce":115688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":115688},{"type":"v1","addr":"172.21.15.16:6807","nonce":115688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":115688},{"type":"v1","addr":"172.21.15.16:6805","nonce":115688}]},"public_addr":"172.21.15.16:6801/115688","cluster_addr":"172.21.15.16:6803/115688","heartbeat_back_addr":"172.21.15.16:6807/115688","heartbeat_front_addr":"172.21.15.16:6805/115688","state":["exists","up"]},{"osd":1,"uuid":"4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":83360},{"type":"v1","addr":"172.21.15.16:6809","nonce":83360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":83360},{"type":"v1","addr":"172.21.15.16:6811","nonce":83360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":83360},{"type":"v1","addr":"172.21.15.16:6815","nonce":83360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":83360},{"type":"v1","addr":"172.21.15.16:6813","nonce":83360}]},"public_addr":"172.21.15.16:6809/83360","cluster_addr":"172.21.15.16:6811/83360","heartbeat_back_addr":"172.21.15.16:6815/83360","heartbeat_front_addr":"172.21.15.16:6813/83360","state":["exists","up"]},{"osd":2,"uuid":"99f8ea0a-5a71-4abe-b087-8e891fd11d1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":83361},{"type":"v1","addr":"172.21.15.16:6817","nonce":83361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":83361},{"type":"v1","addr":"172.21.15.16:6819","nonce":83361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":83361},{"type":"v1","addr":"172.21.15.16:6823","nonce":83361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":83361},{"type":"v1","addr":"172.21.15.16:6821","nonce":83361}]},"public_addr":"172.21.15.16:6817/83361","cluster_addr":"172.21.15.16:6819/83361","heartbeat_back_addr":"172.21.15.16:6823/83361","heartbeat_front_addr":"172.21.15.16:6821/83361","state":["exists","up"]},{"osd":3,"uuid":"d285b4d5-0bef-421a-b84a-943e485b10c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":87,"up_from":133,"up_thru":234,"down_at":88,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":99977},{"type":"v1","addr":"172.21.15.22:6809","nonce":99977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":99977},{"type":"v1","addr":"172.21.15.22:6811","nonce":99977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":99977},{"type":"v1","addr":"172.21.15.22:6815","nonce":99977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":99977},{"type":"v1","addr":"172.21.15.22:6813","nonce":99977}]},"public_addr":"172.21.15.22:6809/99977","cluster_addr":"172.21.15.22:6811/99977","heartbeat_back_addr":"172.21.15.22:6815/99977","heartbeat_front_addr":"172.21.15.22:6813/99977","state":["exists","up"]},{"osd":4,"uuid":"7ba0b0db-e407-4c63-8278-aab09f3082c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":83351},{"type":"v1","addr":"172.21.15.22:6801","nonce":83351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":83351},{"type":"v1","addr":"172.21.15.22:6803","nonce":83351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":83351},{"type":"v1","addr":"172.21.15.22:6807","nonce":83351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":83351},{"type":"v1","addr":"172.21.15.22:6805","nonce":83351}]},"public_addr":"172.21.15.22:6801/83351","cluster_addr":"172.21.15.22:6803/83351","heartbeat_back_addr":"172.21.15.22:6807/83351","heartbeat_front_addr":"172.21.15.22:6805/83351","state":["exists","up"]},{"osd":5,"uuid":"29663bfa-f6fe-4ce7-b45a-a9b7b4424c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":83352},{"type":"v1","addr":"172.21.15.22:6817","nonce":83352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":83352},{"type":"v1","addr":"172.21.15.22:6819","nonce":83352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":83352},{"type":"v1","addr":"172.21.15.22:6823","nonce":83352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":83352},{"type":"v1","addr":"172.21.15.22:6821","nonce":83352}]},"public_addr":"172.21.15.22:6817/83352","cluster_addr":"172.21.15.22:6819/83352","heartbeat_back_addr":"172.21.15.22:6823/83352","heartbeat_front_addr":"172.21.15.22:6821/83352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.001178+0000","dead_epoch":149},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.184181+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.274854+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.186217+0000","dead_epoch":88},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.247442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.220075+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[0,3]},{"pgid":"2.0","osds":[4,1]},{"pgid":"2.2","osds":[1,5]},{"pgid":"2.4","osds":[5,0]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-02T14:07:56.022 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-07-02T14:07:56.022 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:07:56.319 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:56.321 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:07:56.332 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2023-07-02T14:07:56.299823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90838,"num_read_kb":15854726,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5308,"num_bytes_recovered":17939292043,"num_keys_recovered":203,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27030,"ondisk_log_size":27030,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":25712616,"kb_used_data":25269944,"kb_used_omap":1114,"kb_used_meta":441509,"kb_avail":540518424,"statfs":{"total":579820584960,"available":553490866176,"internally_reserved":0,"allocated":25876422656,"data_stored":30191493462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1141261,"internal_metadata":452105715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":66,"apply_latency_ms":66,"commit_latency_ns":66000000,"apply_latency_ns":66000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-178,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":300221636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001969"},"pg_stats":[{"pgid":"2.7","version":"222'3704","reported_seq":9545,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.418578+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:07:48.418578+0000","last_peered":"2023-07-02T14:07:48.418578+0000","last_clean":"2023-07-02T14:07:48.418578+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:07:48.418578+0000","last_undegraded":"2023-07-02T14:07:48.418578+0000","last_fullsized":"2023-07-02T14:07:48.418578+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3704,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1575041492,"num_objects":472,"num_object_clones":397,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":59417,"num_read_kb":2052552,"num_write":3144,"num_write_kb":1530742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"216'3069","reported_seq":5261,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.421447+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:07:48.421447+0000","last_peered":"2023-07-02T14:07:48.421447+0000","last_clean":"2023-07-02T14:07:48.421447+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:07:48.421447+0000","last_undegraded":"2023-07-02T14:07:48.421447+0000","last_fullsized":"2023-07-02T14:07:48.421447+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3069,"ondisk_log_size":3069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1538467250,"num_objects":486,"num_object_clones":415,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":2008,"num_read_kb":2154143,"num_write":2654,"num_write_kb":1492749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"221'3210","reported_seq":5691,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.418575+0000","last_change":"2023-07-02T14:07:41.397514+0000","last_active":"2023-07-02T14:07:48.418575+0000","last_peered":"2023-07-02T14:07:48.418575+0000","last_clean":"2023-07-02T14:07:48.418575+0000","last_became_active":"2023-07-02T14:07:41.396115+0000","last_became_peered":"2023-07-02T14:07:41.396115+0000","last_unstale":"2023-07-02T14:07:48.418575+0000","last_undegraded":"2023-07-02T14:07:48.418575+0000","last_fullsized":"2023-07-02T14:07:48.418575+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3210,"ondisk_log_size":3210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1517142988,"num_objects":428,"num_object_clones":355,"num_object_copies":856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":743,"num_read_kb":1976610,"num_write":2851,"num_write_kb":1467899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1078,"num_bytes_recovered":3879069754,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"216'6494","reported_seq":9013,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.419242+0000","last_change":"2023-07-02T14:07:40.398546+0000","last_active":"2023-07-02T14:07:48.419242+0000","last_peered":"2023-07-02T14:07:48.419242+0000","last_clean":"2023-07-02T14:07:48.419242+0000","last_became_active":"2023-07-02T14:07:40.395869+0000","last_became_peered":"2023-07-02T14:07:40.395869+0000","last_unstale":"2023-07-02T14:07:48.419242+0000","last_undegraded":"2023-07-02T14:07:48.419242+0000","last_fullsized":"2023-07-02T14:07:48.419242+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":6494,"ondisk_log_size":6494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1675535810,"num_objects":495,"num_object_clones":414,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":23017,"num_read_kb":2209627,"num_write":9605,"num_write_kb":1635009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":22,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.424665+0000","last_change":"2023-07-02T14:07:34.365379+0000","last_active":"2023-07-02T14:07:48.424665+0000","last_peered":"2023-07-02T14:07:48.424665+0000","last_clean":"2023-07-02T14:07:48.424665+0000","last_became_active":"2023-07-02T14:07:34.365070+0000","last_became_peered":"2023-07-02T14:07:34.365070+0000","last_unstale":"2023-07-02T14:07:48.424665+0000","last_undegraded":"2023-07-02T14:07:48.424665+0000","last_fullsized":"2023-07-02T14:07:48.424665+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"216'2975","reported_seq":4506,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.422315+0000","last_change":"2023-07-02T14:07:45.413573+0000","last_active":"2023-07-02T14:07:48.422315+0000","last_peered":"2023-07-02T14:07:48.422315+0000","last_clean":"2023-07-02T14:07:48.422315+0000","last_became_active":"2023-07-02T14:07:45.412476+0000","last_became_peered":"2023-07-02T14:07:45.412476+0000","last_unstale":"2023-07-02T14:07:48.422315+0000","last_undegraded":"2023-07-02T14:07:48.422315+0000","last_fullsized":"2023-07-02T14:07:48.422315+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2975,"ondisk_log_size":2975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1661289598,"num_objects":499,"num_object_clones":416,"num_object_copies":998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":1528,"num_read_kb":2088102,"num_write":2559,"num_write_kb":1611276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":2494793202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"216'2884","reported_seq":3866,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.418819+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:07:48.418819+0000","last_peered":"2023-07-02T14:07:48.418819+0000","last_clean":"2023-07-02T14:07:48.418819+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:07:48.418819+0000","last_undegraded":"2023-07-02T14:07:48.418819+0000","last_fullsized":"2023-07-02T14:07:48.418819+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2884,"ondisk_log_size":2884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1569543764,"num_objects":461,"num_object_clones":383,"num_object_copies":922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":1039,"num_read_kb":2026251,"num_write":2501,"num_write_kb":1523310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"216'2328","reported_seq":6067,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.421423+0000","last_change":"2023-07-02T14:07:40.399680+0000","last_active":"2023-07-02T14:07:48.421423+0000","last_peered":"2023-07-02T14:07:48.421423+0000","last_clean":"2023-07-02T14:07:48.421423+0000","last_became_active":"2023-07-02T14:07:40.398277+0000","last_became_peered":"2023-07-02T14:07:40.398277+0000","last_unstale":"2023-07-02T14:07:48.421423+0000","last_undegraded":"2023-07-02T14:07:48.421423+0000","last_fullsized":"2023-07-02T14:07:48.421423+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2328,"ondisk_log_size":2328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1230645001,"num_objects":405,"num_object_clones":325,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":1983,"num_read_kb":1773668,"num_write":2003,"num_write_kb":1186825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":904,"num_bytes_recovered":2809445763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"218'2366","reported_seq":4268,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:07:49.426548+0000","last_change":"2023-07-02T14:07:49.426548+0000","last_active":"2023-07-02T14:07:49.426548+0000","last_peered":"2023-07-02T14:07:49.426548+0000","last_clean":"2023-07-02T14:07:49.426548+0000","last_became_active":"2023-07-02T14:07:49.425350+0000","last_became_peered":"2023-07-02T14:07:49.425350+0000","last_unstale":"2023-07-02T14:07:49.426548+0000","last_undegraded":"2023-07-02T14:07:49.426548+0000","last_fullsized":"2023-07-02T14:07:49.426548+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2366,"ondisk_log_size":2366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1146537804,"num_objects":358,"num_object_clones":299,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":1103,"num_read_kb":1573773,"num_write":2067,"num_write_kb":1113633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":792,"num_bytes_recovered":2631686774,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90838,"num_read_kb":15854726,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5308,"num_bytes_recovered":17939292043,"num_keys_recovered":203,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25858924544,"data_stored":30184618536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":27030,"ondisk_log_size":27030,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640333,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4026312,"kb_used_data":3989312,"kb_used_omap":0,"kb_used_meta":36992,"kb_avail":90345528,"statfs":{"total":96636764160,"available":92513820672,"internally_reserved":0,"allocated":4085055488,"data_stored":4773662384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37879808},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.499}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.42899999999999999,"15min":0.42699999999999999},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.0509999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.45000000000000001,"15min":0.45700000000000002},"min":{"1min":0.28299999999999997,"5min":0.224,"15min":0.224},"max":{"1min":0.80700000000000005,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":0.54900000000000004}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.43099999999999999,"15min":0.42699999999999999},"min":{"1min":0.29699999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.1850000000000001,"5min":2.157,"15min":2.157},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.439,"15min":0.44900000000000001},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73699999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.53600000000000003}]},{"osd":3,"last update":"Sun Jul 2 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.38100000000000001,"15min":0.36299999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.2919999999999998,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.442},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.36399999999999999,"15min":0.35899999999999999},"min":{"1min":0.28000000000000003,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.39200000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.39800000000000002,"15min":0.41699999999999998},"min":{"1min":0.27600000000000002,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.99299999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.42899999999999999},"min":{"1min":0.29499999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.65700000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.40500000000000003}]}]},{"osd":4,"up_from":11,"seq":47244640333,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3882852,"kb_used_data":3786524,"kb_used_omap":0,"kb_used_meta":96320,"kb_avail":90488988,"statfs":{"total":96636764160,"available":92660723712,"internally_reserved":0,"allocated":3877400576,"data_stored":4464061785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":98631680},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.496},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.48899999999999999}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44,"15min":0.46400000000000002},"min":{"1min":0.28399999999999997,"5min":0.215,"15min":0.215},"max":{"1min":0.71499999999999997,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.45000000000000001},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.55000000000000004}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.32800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66800000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.57399999999999995}]},{"osd":3,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.38300000000000001},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45900000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.36799999999999999,"15min":0.36299999999999999},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.48799999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.45000000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.44900000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.433,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59099999999999997,"5min":1.026,"15min":1.026},"last":0.42899999999999999}]}]},{"osd":3,"up_from":133,"seq":571230650404,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4569484,"kb_used_data":4499076,"kb_used_omap":229,"kb_used_meta":70170,"kb_avail":89802356,"statfs":{"total":96636764160,"available":91957612544,"internally_reserved":0,"allocated":4607053824,"data_stored":5358018351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":71854302},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.41099999999999998}]},{"osd":1,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.372,"15min":0.36599999999999999},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.44800000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.28699999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.44900000000000001,"15min":0.439},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.56299999999999994}]},{"osd":4,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.35799999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.498},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.36199999999999999,"15min":0.35299999999999998},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.59499999999999997}]},{"osd":5,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.39800000000000002},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.439,"5min":0.39500000000000002,"15min":0.38800000000000001},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.63}]}]},{"osd":2,"up_from":11,"seq":47244640333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3200836,"kb_used_data":3146556,"kb_used_omap":0,"kb_used_meta":54272,"kb_avail":91171004,"statfs":{"total":96636764160,"available":93359108096,"internally_reserved":0,"allocated":3222073344,"data_stored":3710453423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55574528},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.17000000000000001},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.20300000000000001}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.42499999999999999,"15min":0.374},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":0.254},{"interface":"front","average":{"1min":0.376,"5min":0.33500000000000002,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.23999999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:07:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.91100000000000003,"5min":1.929,"15min":1.929},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.54800000000000004,"15min":0.55600000000000005},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93999999999999995,"5min":2,"15min":2},"last":0.29899999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.312},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.33700000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68400000000000005,"5min":1.071,"15min":1.071},"last":0.372},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":0.35799999999999998}]}]},{"osd":0,"up_from":166,"seq":712964571162,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2680400,"kb_used_data":2627784,"kb_used_omap":884,"kb_used_meta":51723,"kb_avail":91691440,"statfs":{"total":96636764160,"available":93892034560,"internally_reserved":0,"allocated":2690850816,"data_stored":3042367221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":905963,"internal_metadata":52964629},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.33600000000000002}]},{"osd":2,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.28999999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.438},{"interface":"front","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.377}]},{"osd":4,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.42999999999999999}]},{"osd":5,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.42299999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640333,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7352732,"kb_used_data":7220692,"kb_used_omap":0,"kb_used_meta":132032,"kb_avail":87019108,"statfs":{"total":96636764160,"available":89107566592,"internally_reserved":0,"allocated":7393988608,"data_stored":8842930298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":135200768},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.32000000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.224,"5min":0.184,"15min":0.184},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.35999999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.49099999999999999,"15min":0.47299999999999998},"min":{"1min":0.30299999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.39300000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71899999999999997,"5min":1.524,"15min":1.524},"last":0.45300000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.80300000000000005,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.495},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84399999999999997,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.52600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2687934464,"data_stored":3041221400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7391072256,"data_stored":8841784477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3219156992,"data_stored":3709307602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4604137472,"data_stored":5356872530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3874484224,"data_stored":4462915964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4082139136,"data_stored":4772516563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:07:56.334 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:07:56.635 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:56.637 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:07:56.648 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2023-07-02T14:07:56.299823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90838,"num_read_kb":15854726,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5308,"num_bytes_recovered":17939292043,"num_keys_recovered":203,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27030,"ondisk_log_size":27030,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":25712616,"kb_used_data":25269944,"kb_used_omap":1114,"kb_used_meta":441509,"kb_avail":540518424,"statfs":{"total":579820584960,"available":553490866176,"internally_reserved":0,"allocated":25876422656,"data_stored":30191493462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1141261,"internal_metadata":452105715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":66,"apply_latency_ms":66,"commit_latency_ns":66000000,"apply_latency_ns":66000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-178,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":300221636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001969"},"pg_stats":[{"pgid":"2.7","version":"222'3704","reported_seq":9545,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.418578+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:07:48.418578+0000","last_peered":"2023-07-02T14:07:48.418578+0000","last_clean":"2023-07-02T14:07:48.418578+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:07:48.418578+0000","last_undegraded":"2023-07-02T14:07:48.418578+0000","last_fullsized":"2023-07-02T14:07:48.418578+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3704,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1575041492,"num_objects":472,"num_object_clones":397,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":59417,"num_read_kb":2052552,"num_write":3144,"num_write_kb":1530742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"216'3069","reported_seq":5261,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.421447+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:07:48.421447+0000","last_peered":"2023-07-02T14:07:48.421447+0000","last_clean":"2023-07-02T14:07:48.421447+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:07:48.421447+0000","last_undegraded":"2023-07-02T14:07:48.421447+0000","last_fullsized":"2023-07-02T14:07:48.421447+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3069,"ondisk_log_size":3069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1538467250,"num_objects":486,"num_object_clones":415,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":2008,"num_read_kb":2154143,"num_write":2654,"num_write_kb":1492749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"221'3210","reported_seq":5691,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.418575+0000","last_change":"2023-07-02T14:07:41.397514+0000","last_active":"2023-07-02T14:07:48.418575+0000","last_peered":"2023-07-02T14:07:48.418575+0000","last_clean":"2023-07-02T14:07:48.418575+0000","last_became_active":"2023-07-02T14:07:41.396115+0000","last_became_peered":"2023-07-02T14:07:41.396115+0000","last_unstale":"2023-07-02T14:07:48.418575+0000","last_undegraded":"2023-07-02T14:07:48.418575+0000","last_fullsized":"2023-07-02T14:07:48.418575+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3210,"ondisk_log_size":3210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1517142988,"num_objects":428,"num_object_clones":355,"num_object_copies":856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":743,"num_read_kb":1976610,"num_write":2851,"num_write_kb":1467899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1078,"num_bytes_recovered":3879069754,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"216'6494","reported_seq":9013,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.419242+0000","last_change":"2023-07-02T14:07:40.398546+0000","last_active":"2023-07-02T14:07:48.419242+0000","last_peered":"2023-07-02T14:07:48.419242+0000","last_clean":"2023-07-02T14:07:48.419242+0000","last_became_active":"2023-07-02T14:07:40.395869+0000","last_became_peered":"2023-07-02T14:07:40.395869+0000","last_unstale":"2023-07-02T14:07:48.419242+0000","last_undegraded":"2023-07-02T14:07:48.419242+0000","last_fullsized":"2023-07-02T14:07:48.419242+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":6494,"ondisk_log_size":6494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1675535810,"num_objects":495,"num_object_clones":414,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":23017,"num_read_kb":2209627,"num_write":9605,"num_write_kb":1635009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":22,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.424665+0000","last_change":"2023-07-02T14:07:34.365379+0000","last_active":"2023-07-02T14:07:48.424665+0000","last_peered":"2023-07-02T14:07:48.424665+0000","last_clean":"2023-07-02T14:07:48.424665+0000","last_became_active":"2023-07-02T14:07:34.365070+0000","last_became_peered":"2023-07-02T14:07:34.365070+0000","last_unstale":"2023-07-02T14:07:48.424665+0000","last_undegraded":"2023-07-02T14:07:48.424665+0000","last_fullsized":"2023-07-02T14:07:48.424665+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"216'2975","reported_seq":4506,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.422315+0000","last_change":"2023-07-02T14:07:45.413573+0000","last_active":"2023-07-02T14:07:48.422315+0000","last_peered":"2023-07-02T14:07:48.422315+0000","last_clean":"2023-07-02T14:07:48.422315+0000","last_became_active":"2023-07-02T14:07:45.412476+0000","last_became_peered":"2023-07-02T14:07:45.412476+0000","last_unstale":"2023-07-02T14:07:48.422315+0000","last_undegraded":"2023-07-02T14:07:48.422315+0000","last_fullsized":"2023-07-02T14:07:48.422315+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2975,"ondisk_log_size":2975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1661289598,"num_objects":499,"num_object_clones":416,"num_object_copies":998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":1528,"num_read_kb":2088102,"num_write":2559,"num_write_kb":1611276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":2494793202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"216'2884","reported_seq":3866,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.418819+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:07:48.418819+0000","last_peered":"2023-07-02T14:07:48.418819+0000","last_clean":"2023-07-02T14:07:48.418819+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:07:48.418819+0000","last_undegraded":"2023-07-02T14:07:48.418819+0000","last_fullsized":"2023-07-02T14:07:48.418819+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2884,"ondisk_log_size":2884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1569543764,"num_objects":461,"num_object_clones":383,"num_object_copies":922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":1039,"num_read_kb":2026251,"num_write":2501,"num_write_kb":1523310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"216'2328","reported_seq":6067,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.421423+0000","last_change":"2023-07-02T14:07:40.399680+0000","last_active":"2023-07-02T14:07:48.421423+0000","last_peered":"2023-07-02T14:07:48.421423+0000","last_clean":"2023-07-02T14:07:48.421423+0000","last_became_active":"2023-07-02T14:07:40.398277+0000","last_became_peered":"2023-07-02T14:07:40.398277+0000","last_unstale":"2023-07-02T14:07:48.421423+0000","last_undegraded":"2023-07-02T14:07:48.421423+0000","last_fullsized":"2023-07-02T14:07:48.421423+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2328,"ondisk_log_size":2328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1230645001,"num_objects":405,"num_object_clones":325,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":1983,"num_read_kb":1773668,"num_write":2003,"num_write_kb":1186825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":904,"num_bytes_recovered":2809445763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"218'2366","reported_seq":4268,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:07:49.426548+0000","last_change":"2023-07-02T14:07:49.426548+0000","last_active":"2023-07-02T14:07:49.426548+0000","last_peered":"2023-07-02T14:07:49.426548+0000","last_clean":"2023-07-02T14:07:49.426548+0000","last_became_active":"2023-07-02T14:07:49.425350+0000","last_became_peered":"2023-07-02T14:07:49.425350+0000","last_unstale":"2023-07-02T14:07:49.426548+0000","last_undegraded":"2023-07-02T14:07:49.426548+0000","last_fullsized":"2023-07-02T14:07:49.426548+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2366,"ondisk_log_size":2366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1146537804,"num_objects":358,"num_object_clones":299,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":1103,"num_read_kb":1573773,"num_write":2067,"num_write_kb":1113633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":792,"num_bytes_recovered":2631686774,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90838,"num_read_kb":15854726,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5308,"num_bytes_recovered":17939292043,"num_keys_recovered":203,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25858924544,"data_stored":30184618536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":27030,"ondisk_log_size":27030,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640333,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4026312,"kb_used_data":3989312,"kb_used_omap":0,"kb_used_meta":36992,"kb_avail":90345528,"statfs":{"total":96636764160,"available":92513820672,"internally_reserved":0,"allocated":4085055488,"data_stored":4773662384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37879808},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.499}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.42899999999999999,"15min":0.42699999999999999},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.0509999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.45000000000000001,"15min":0.45700000000000002},"min":{"1min":0.28299999999999997,"5min":0.224,"15min":0.224},"max":{"1min":0.80700000000000005,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":0.54900000000000004}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.43099999999999999,"15min":0.42699999999999999},"min":{"1min":0.29699999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.1850000000000001,"5min":2.157,"15min":2.157},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.439,"15min":0.44900000000000001},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73699999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.53600000000000003}]},{"osd":3,"last update":"Sun Jul 2 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.38100000000000001,"15min":0.36299999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.2919999999999998,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.442},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.36399999999999999,"15min":0.35899999999999999},"min":{"1min":0.28000000000000003,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.39200000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.39800000000000002,"15min":0.41699999999999998},"min":{"1min":0.27600000000000002,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.99299999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.42899999999999999},"min":{"1min":0.29499999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.65700000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.40500000000000003}]}]},{"osd":4,"up_from":11,"seq":47244640333,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3882852,"kb_used_data":3786524,"kb_used_omap":0,"kb_used_meta":96320,"kb_avail":90488988,"statfs":{"total":96636764160,"available":92660723712,"internally_reserved":0,"allocated":3877400576,"data_stored":4464061785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":98631680},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.496},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.48899999999999999}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44,"15min":0.46400000000000002},"min":{"1min":0.28399999999999997,"5min":0.215,"15min":0.215},"max":{"1min":0.71499999999999997,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.45000000000000001},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.55000000000000004}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.32800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66800000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.57399999999999995}]},{"osd":3,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.38300000000000001},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45900000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.36799999999999999,"15min":0.36299999999999999},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.48799999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.45000000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.44900000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.433,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59099999999999997,"5min":1.026,"15min":1.026},"last":0.42899999999999999}]}]},{"osd":3,"up_from":133,"seq":571230650404,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4569484,"kb_used_data":4499076,"kb_used_omap":229,"kb_used_meta":70170,"kb_avail":89802356,"statfs":{"total":96636764160,"available":91957612544,"internally_reserved":0,"allocated":4607053824,"data_stored":5358018351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":71854302},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.41099999999999998}]},{"osd":1,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.372,"15min":0.36599999999999999},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.44800000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.28699999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.44900000000000001,"15min":0.439},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.56299999999999994}]},{"osd":4,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.35799999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.498},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.36199999999999999,"15min":0.35299999999999998},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.59499999999999997}]},{"osd":5,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.39800000000000002},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.439,"5min":0.39500000000000002,"15min":0.38800000000000001},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.63}]}]},{"osd":2,"up_from":11,"seq":47244640333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3200836,"kb_used_data":3146556,"kb_used_omap":0,"kb_used_meta":54272,"kb_avail":91171004,"statfs":{"total":96636764160,"available":93359108096,"internally_reserved":0,"allocated":3222073344,"data_stored":3710453423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55574528},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.17000000000000001},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.20300000000000001}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.42499999999999999,"15min":0.374},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":0.254},{"interface":"front","average":{"1min":0.376,"5min":0.33500000000000002,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.23999999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:07:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.91100000000000003,"5min":1.929,"15min":1.929},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.54800000000000004,"15min":0.55600000000000005},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93999999999999995,"5min":2,"15min":2},"last":0.29899999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.312},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.33700000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68400000000000005,"5min":1.071,"15min":1.071},"last":0.372},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":0.35799999999999998}]}]},{"osd":0,"up_from":166,"seq":712964571162,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2680400,"kb_used_data":2627784,"kb_used_omap":884,"kb_used_meta":51723,"kb_avail":91691440,"statfs":{"total":96636764160,"available":93892034560,"internally_reserved":0,"allocated":2690850816,"data_stored":3042367221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":905963,"internal_metadata":52964629},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.33600000000000002}]},{"osd":2,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.28999999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.438},{"interface":"front","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.377}]},{"osd":4,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.42999999999999999}]},{"osd":5,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.42299999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640333,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7352732,"kb_used_data":7220692,"kb_used_omap":0,"kb_used_meta":132032,"kb_avail":87019108,"statfs":{"total":96636764160,"available":89107566592,"internally_reserved":0,"allocated":7393988608,"data_stored":8842930298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":135200768},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.32000000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.224,"5min":0.184,"15min":0.184},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.35999999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.49099999999999999,"15min":0.47299999999999998},"min":{"1min":0.30299999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.39300000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71899999999999997,"5min":1.524,"15min":1.524},"last":0.45300000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.80300000000000005,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.495},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84399999999999997,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.52600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2687934464,"data_stored":3041221400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7391072256,"data_stored":8841784477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3219156992,"data_stored":3709307602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4604137472,"data_stored":5356872530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3874484224,"data_stored":4462915964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4082139136,"data_stored":4772516563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:07:56.650 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-07-02T14:07:56.650 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:07:56.957 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:56.959 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:07:56.970 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2023-07-02T14:07:56.299823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90838,"num_read_kb":15854726,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5308,"num_bytes_recovered":17939292043,"num_keys_recovered":203,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27030,"ondisk_log_size":27030,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":25712616,"kb_used_data":25269944,"kb_used_omap":1114,"kb_used_meta":441509,"kb_avail":540518424,"statfs":{"total":579820584960,"available":553490866176,"internally_reserved":0,"allocated":25876422656,"data_stored":30191493462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1141261,"internal_metadata":452105715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":66,"apply_latency_ms":66,"commit_latency_ns":66000000,"apply_latency_ns":66000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-178,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":300221636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001969"},"pg_stats":[{"pgid":"2.7","version":"222'3704","reported_seq":9545,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.418578+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:07:48.418578+0000","last_peered":"2023-07-02T14:07:48.418578+0000","last_clean":"2023-07-02T14:07:48.418578+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:07:48.418578+0000","last_undegraded":"2023-07-02T14:07:48.418578+0000","last_fullsized":"2023-07-02T14:07:48.418578+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3704,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1575041492,"num_objects":472,"num_object_clones":397,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":59417,"num_read_kb":2052552,"num_write":3144,"num_write_kb":1530742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"216'3069","reported_seq":5261,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.421447+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:07:48.421447+0000","last_peered":"2023-07-02T14:07:48.421447+0000","last_clean":"2023-07-02T14:07:48.421447+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:07:48.421447+0000","last_undegraded":"2023-07-02T14:07:48.421447+0000","last_fullsized":"2023-07-02T14:07:48.421447+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3069,"ondisk_log_size":3069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1538467250,"num_objects":486,"num_object_clones":415,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":2008,"num_read_kb":2154143,"num_write":2654,"num_write_kb":1492749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"221'3210","reported_seq":5691,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.418575+0000","last_change":"2023-07-02T14:07:41.397514+0000","last_active":"2023-07-02T14:07:48.418575+0000","last_peered":"2023-07-02T14:07:48.418575+0000","last_clean":"2023-07-02T14:07:48.418575+0000","last_became_active":"2023-07-02T14:07:41.396115+0000","last_became_peered":"2023-07-02T14:07:41.396115+0000","last_unstale":"2023-07-02T14:07:48.418575+0000","last_undegraded":"2023-07-02T14:07:48.418575+0000","last_fullsized":"2023-07-02T14:07:48.418575+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3210,"ondisk_log_size":3210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1517142988,"num_objects":428,"num_object_clones":355,"num_object_copies":856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":743,"num_read_kb":1976610,"num_write":2851,"num_write_kb":1467899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1078,"num_bytes_recovered":3879069754,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"216'6494","reported_seq":9013,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.419242+0000","last_change":"2023-07-02T14:07:40.398546+0000","last_active":"2023-07-02T14:07:48.419242+0000","last_peered":"2023-07-02T14:07:48.419242+0000","last_clean":"2023-07-02T14:07:48.419242+0000","last_became_active":"2023-07-02T14:07:40.395869+0000","last_became_peered":"2023-07-02T14:07:40.395869+0000","last_unstale":"2023-07-02T14:07:48.419242+0000","last_undegraded":"2023-07-02T14:07:48.419242+0000","last_fullsized":"2023-07-02T14:07:48.419242+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":6494,"ondisk_log_size":6494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1675535810,"num_objects":495,"num_object_clones":414,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":23017,"num_read_kb":2209627,"num_write":9605,"num_write_kb":1635009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":22,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.424665+0000","last_change":"2023-07-02T14:07:34.365379+0000","last_active":"2023-07-02T14:07:48.424665+0000","last_peered":"2023-07-02T14:07:48.424665+0000","last_clean":"2023-07-02T14:07:48.424665+0000","last_became_active":"2023-07-02T14:07:34.365070+0000","last_became_peered":"2023-07-02T14:07:34.365070+0000","last_unstale":"2023-07-02T14:07:48.424665+0000","last_undegraded":"2023-07-02T14:07:48.424665+0000","last_fullsized":"2023-07-02T14:07:48.424665+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"216'2975","reported_seq":4506,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.422315+0000","last_change":"2023-07-02T14:07:45.413573+0000","last_active":"2023-07-02T14:07:48.422315+0000","last_peered":"2023-07-02T14:07:48.422315+0000","last_clean":"2023-07-02T14:07:48.422315+0000","last_became_active":"2023-07-02T14:07:45.412476+0000","last_became_peered":"2023-07-02T14:07:45.412476+0000","last_unstale":"2023-07-02T14:07:48.422315+0000","last_undegraded":"2023-07-02T14:07:48.422315+0000","last_fullsized":"2023-07-02T14:07:48.422315+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2975,"ondisk_log_size":2975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1661289598,"num_objects":499,"num_object_clones":416,"num_object_copies":998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":1528,"num_read_kb":2088102,"num_write":2559,"num_write_kb":1611276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":2494793202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"216'2884","reported_seq":3866,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.418819+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:07:48.418819+0000","last_peered":"2023-07-02T14:07:48.418819+0000","last_clean":"2023-07-02T14:07:48.418819+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:07:48.418819+0000","last_undegraded":"2023-07-02T14:07:48.418819+0000","last_fullsized":"2023-07-02T14:07:48.418819+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2884,"ondisk_log_size":2884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1569543764,"num_objects":461,"num_object_clones":383,"num_object_copies":922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":1039,"num_read_kb":2026251,"num_write":2501,"num_write_kb":1523310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"216'2328","reported_seq":6067,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.421423+0000","last_change":"2023-07-02T14:07:40.399680+0000","last_active":"2023-07-02T14:07:48.421423+0000","last_peered":"2023-07-02T14:07:48.421423+0000","last_clean":"2023-07-02T14:07:48.421423+0000","last_became_active":"2023-07-02T14:07:40.398277+0000","last_became_peered":"2023-07-02T14:07:40.398277+0000","last_unstale":"2023-07-02T14:07:48.421423+0000","last_undegraded":"2023-07-02T14:07:48.421423+0000","last_fullsized":"2023-07-02T14:07:48.421423+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2328,"ondisk_log_size":2328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1230645001,"num_objects":405,"num_object_clones":325,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":1983,"num_read_kb":1773668,"num_write":2003,"num_write_kb":1186825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":904,"num_bytes_recovered":2809445763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"218'2366","reported_seq":4268,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:07:49.426548+0000","last_change":"2023-07-02T14:07:49.426548+0000","last_active":"2023-07-02T14:07:49.426548+0000","last_peered":"2023-07-02T14:07:49.426548+0000","last_clean":"2023-07-02T14:07:49.426548+0000","last_became_active":"2023-07-02T14:07:49.425350+0000","last_became_peered":"2023-07-02T14:07:49.425350+0000","last_unstale":"2023-07-02T14:07:49.426548+0000","last_undegraded":"2023-07-02T14:07:49.426548+0000","last_fullsized":"2023-07-02T14:07:49.426548+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2366,"ondisk_log_size":2366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1146537804,"num_objects":358,"num_object_clones":299,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":1103,"num_read_kb":1573773,"num_write":2067,"num_write_kb":1113633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":792,"num_bytes_recovered":2631686774,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90838,"num_read_kb":15854726,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5308,"num_bytes_recovered":17939292043,"num_keys_recovered":203,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25858924544,"data_stored":30184618536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":27030,"ondisk_log_size":27030,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640333,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4026312,"kb_used_data":3989312,"kb_used_omap":0,"kb_used_meta":36992,"kb_avail":90345528,"statfs":{"total":96636764160,"available":92513820672,"internally_reserved":0,"allocated":4085055488,"data_stored":4773662384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37879808},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.499}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.42899999999999999,"15min":0.42699999999999999},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.0509999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.45000000000000001,"15min":0.45700000000000002},"min":{"1min":0.28299999999999997,"5min":0.224,"15min":0.224},"max":{"1min":0.80700000000000005,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":0.54900000000000004}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.43099999999999999,"15min":0.42699999999999999},"min":{"1min":0.29699999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.1850000000000001,"5min":2.157,"15min":2.157},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.439,"15min":0.44900000000000001},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73699999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.53600000000000003}]},{"osd":3,"last update":"Sun Jul 2 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.38100000000000001,"15min":0.36299999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.2919999999999998,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.442},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.36399999999999999,"15min":0.35899999999999999},"min":{"1min":0.28000000000000003,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.39200000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.39800000000000002,"15min":0.41699999999999998},"min":{"1min":0.27600000000000002,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.99299999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.42899999999999999},"min":{"1min":0.29499999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.65700000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.40500000000000003}]}]},{"osd":4,"up_from":11,"seq":47244640333,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3882852,"kb_used_data":3786524,"kb_used_omap":0,"kb_used_meta":96320,"kb_avail":90488988,"statfs":{"total":96636764160,"available":92660723712,"internally_reserved":0,"allocated":3877400576,"data_stored":4464061785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":98631680},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.496},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.48899999999999999}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44,"15min":0.46400000000000002},"min":{"1min":0.28399999999999997,"5min":0.215,"15min":0.215},"max":{"1min":0.71499999999999997,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.45000000000000001},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.55000000000000004}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.32800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66800000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.57399999999999995}]},{"osd":3,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.38300000000000001},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45900000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.36799999999999999,"15min":0.36299999999999999},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.48799999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.45000000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.44900000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.433,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59099999999999997,"5min":1.026,"15min":1.026},"last":0.42899999999999999}]}]},{"osd":3,"up_from":133,"seq":571230650404,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4569484,"kb_used_data":4499076,"kb_used_omap":229,"kb_used_meta":70170,"kb_avail":89802356,"statfs":{"total":96636764160,"available":91957612544,"internally_reserved":0,"allocated":4607053824,"data_stored":5358018351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":71854302},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:53 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.41099999999999998}]},{"osd":1,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.372,"15min":0.36599999999999999},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.44800000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.28699999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.44900000000000001,"15min":0.439},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.56299999999999994}]},{"osd":4,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.35799999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.498},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.36199999999999999,"15min":0.35299999999999998},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.59499999999999997}]},{"osd":5,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.39800000000000002},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.439,"5min":0.39500000000000002,"15min":0.38800000000000001},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.63}]}]},{"osd":2,"up_from":11,"seq":47244640333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3200836,"kb_used_data":3146556,"kb_used_omap":0,"kb_used_meta":54272,"kb_avail":91171004,"statfs":{"total":96636764160,"available":93359108096,"internally_reserved":0,"allocated":3222073344,"data_stored":3710453423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55574528},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.17000000000000001},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.20300000000000001}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.42499999999999999,"15min":0.374},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":0.254},{"interface":"front","average":{"1min":0.376,"5min":0.33500000000000002,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.23999999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:07:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.91100000000000003,"5min":1.929,"15min":1.929},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.54800000000000004,"15min":0.55600000000000005},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93999999999999995,"5min":2,"15min":2},"last":0.29899999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.312},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.33700000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68400000000000005,"5min":1.071,"15min":1.071},"last":0.372},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":0.35799999999999998}]}]},{"osd":0,"up_from":166,"seq":712964571162,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2680400,"kb_used_data":2627784,"kb_used_omap":884,"kb_used_meta":51723,"kb_avail":91691440,"statfs":{"total":96636764160,"available":93892034560,"internally_reserved":0,"allocated":2690850816,"data_stored":3042367221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":905963,"internal_metadata":52964629},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.33600000000000002}]},{"osd":2,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.28999999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.438},{"interface":"front","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.377}]},{"osd":4,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.42999999999999999}]},{"osd":5,"last update":"Sun Jul 2 14:06:52 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.42299999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640333,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7352732,"kb_used_data":7220692,"kb_used_omap":0,"kb_used_meta":132032,"kb_avail":87019108,"statfs":{"total":96636764160,"available":89107566592,"internally_reserved":0,"allocated":7393988608,"data_stored":8842930298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":135200768},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.32000000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.224,"5min":0.184,"15min":0.184},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.35999999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.49099999999999999,"15min":0.47299999999999998},"min":{"1min":0.30299999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.39300000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71899999999999997,"5min":1.524,"15min":1.524},"last":0.45300000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.80300000000000005,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.495},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84399999999999997,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.52600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2687934464,"data_stored":3041221400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7391072256,"data_stored":8841784477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3219156992,"data_stored":3709307602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4604137472,"data_stored":5356872530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3874484224,"data_stored":4462915964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4082139136,"data_stored":4772516563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:07:56.972 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-02T14:07:57.274 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:07:57.274 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":241,"fsid":"9eaaf0b2-e5a8-42a9-b170-62124ca4d4c2","created":"2023-07-02T14:01:37.298060+0000","modified":"2023-07-02T14:07:49.415411+0000","last_up_change":"2023-07-02T14:05:52.397969+0000","last_in_change":"2023-07-02T14:07:33.350320+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-07-02T14:01:45.323455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-07-02T14:01:47.448897+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":69,"snap_epoch":217,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"43c446fe-c811-4ee3-99e8-d7452edfc1c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":148,"up_from":166,"up_thru":227,"down_at":149,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":115688},{"type":"v1","addr":"172.21.15.16:6801","nonce":115688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":115688},{"type":"v1","addr":"172.21.15.16:6803","nonce":115688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":115688},{"type":"v1","addr":"172.21.15.16:6807","nonce":115688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":115688},{"type":"v1","addr":"172.21.15.16:6805","nonce":115688}]},"public_addr":"172.21.15.16:6801/115688","cluster_addr":"172.21.15.16:6803/115688","heartbeat_back_addr":"172.21.15.16:6807/115688","heartbeat_front_addr":"172.21.15.16:6805/115688","state":["exists","up"]},{"osd":1,"uuid":"4ab25ac5-e47d-4f98-80fa-d0a72ce4a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":83360},{"type":"v1","addr":"172.21.15.16:6809","nonce":83360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":83360},{"type":"v1","addr":"172.21.15.16:6811","nonce":83360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":83360},{"type":"v1","addr":"172.21.15.16:6815","nonce":83360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":83360},{"type":"v1","addr":"172.21.15.16:6813","nonce":83360}]},"public_addr":"172.21.15.16:6809/83360","cluster_addr":"172.21.15.16:6811/83360","heartbeat_back_addr":"172.21.15.16:6815/83360","heartbeat_front_addr":"172.21.15.16:6813/83360","state":["exists","up"]},{"osd":2,"uuid":"99f8ea0a-5a71-4abe-b087-8e891fd11d1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":83361},{"type":"v1","addr":"172.21.15.16:6817","nonce":83361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":83361},{"type":"v1","addr":"172.21.15.16:6819","nonce":83361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":83361},{"type":"v1","addr":"172.21.15.16:6823","nonce":83361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":83361},{"type":"v1","addr":"172.21.15.16:6821","nonce":83361}]},"public_addr":"172.21.15.16:6817/83361","cluster_addr":"172.21.15.16:6819/83361","heartbeat_back_addr":"172.21.15.16:6823/83361","heartbeat_front_addr":"172.21.15.16:6821/83361","state":["exists","up"]},{"osd":3,"uuid":"d285b4d5-0bef-421a-b84a-943e485b10c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":87,"up_from":133,"up_thru":234,"down_at":88,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":99977},{"type":"v1","addr":"172.21.15.22:6809","nonce":99977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":99977},{"type":"v1","addr":"172.21.15.22:6811","nonce":99977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6814","nonce":99977},{"type":"v1","addr":"172.21.15.22:6815","nonce":99977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":99977},{"type":"v1","addr":"172.21.15.22:6813","nonce":99977}]},"public_addr":"172.21.15.22:6809/99977","cluster_addr":"172.21.15.22:6811/99977","heartbeat_back_addr":"172.21.15.22:6815/99977","heartbeat_front_addr":"172.21.15.22:6813/99977","state":["exists","up"]},{"osd":4,"uuid":"7ba0b0db-e407-4c63-8278-aab09f3082c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":233,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":83351},{"type":"v1","addr":"172.21.15.22:6801","nonce":83351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":83351},{"type":"v1","addr":"172.21.15.22:6803","nonce":83351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":83351},{"type":"v1","addr":"172.21.15.22:6807","nonce":83351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":83351},{"type":"v1","addr":"172.21.15.22:6805","nonce":83351}]},"public_addr":"172.21.15.22:6801/83351","cluster_addr":"172.21.15.22:6803/83351","heartbeat_back_addr":"172.21.15.22:6807/83351","heartbeat_front_addr":"172.21.15.22:6805/83351","state":["exists","up"]},{"osd":5,"uuid":"29663bfa-f6fe-4ce7-b45a-a9b7b4424c42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":240,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":83352},{"type":"v1","addr":"172.21.15.22:6817","nonce":83352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6818","nonce":83352},{"type":"v1","addr":"172.21.15.22:6819","nonce":83352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6822","nonce":83352},{"type":"v1","addr":"172.21.15.22:6823","nonce":83352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":83352},{"type":"v1","addr":"172.21.15.22:6821","nonce":83352}]},"public_addr":"172.21.15.22:6817/83352","cluster_addr":"172.21.15.22:6819/83352","heartbeat_back_addr":"172.21.15.22:6823/83352","heartbeat_front_addr":"172.21.15.22:6821/83352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.001178+0000","dead_epoch":149},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.184181+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.274854+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.186217+0000","dead_epoch":88},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.247442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-07-02T14:01:44.220075+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[0,3]},{"pgid":"2.0","osds":[4,1]},{"pgid":"2.2","osds":[1,5]},{"pgid":"2.4","osds":[5,0]}],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-02T14:07:58.288 INFO:tasks.ceph:Scrubbing osd.0 2023-07-02T14:07:58.289 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-07-02T14:07:58.436 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:58.436 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-07-02T14:07:58.436 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:58.448 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-07-02T14:07:58.753 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 0 to deep-scrub 2023-07-02T14:07:58.765 INFO:tasks.ceph:Scrubbing osd.1 2023-07-02T14:07:58.765 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-07-02T14:07:58.907 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:58.907 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-07-02T14:07:58.908 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:58.919 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-07-02T14:07:59.224 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 1 to deep-scrub 2023-07-02T14:07:59.235 INFO:tasks.ceph:Scrubbing osd.2 2023-07-02T14:07:59.236 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-07-02T14:07:59.377 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:59.377 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-07-02T14:07:59.377 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:59.389 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-07-02T14:07:59.692 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 2 to deep-scrub 2023-07-02T14:07:59.703 INFO:tasks.ceph:Scrubbing osd.3 2023-07-02T14:07:59.703 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-07-02T14:07:59.855 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:07:59.856 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-07-02T14:07:59.856 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:07:59.867 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-07-02T14:08:00.237 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 3 to deep-scrub 2023-07-02T14:08:00.250 INFO:tasks.ceph:Scrubbing osd.4 2023-07-02T14:08:00.250 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-07-02T14:08:00.428 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:08:00.428 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-07-02T14:08:00.429 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:08:00.442 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-07-02T14:08:00.821 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 4 to deep-scrub 2023-07-02T14:08:00.838 INFO:tasks.ceph:Scrubbing osd.5 2023-07-02T14:08:00.838 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-07-02T14:08:01.032 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2023-07-02T14:08:01.033 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-07-02T14:08:01.033 INFO:teuthology.orchestra.run.smithi016.stdout:} 2023-07-02T14:08:01.051 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-07-02T14:08:01.484 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 5 to deep-scrub 2023-07-02T14:08:01.499 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:08:01.926 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:08:01.928 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:08:01.943 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2023-07-02T14:08:00.300680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90838,"num_read_kb":15854726,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5308,"num_bytes_recovered":17939292043,"num_keys_recovered":203,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27030,"ondisk_log_size":27030,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":23,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":25712616,"kb_used_data":25269944,"kb_used_omap":1114,"kb_used_meta":441509,"kb_avail":540518424,"statfs":{"total":579820584960,"available":553490866176,"internally_reserved":0,"allocated":25876422656,"data_stored":30191493462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1141261,"internal_metadata":452105715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":63,"apply_latency_ms":63,"commit_latency_ns":63000000,"apply_latency_ns":63000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-178,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":300221636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.883298"},"pg_stats":[{"pgid":"2.7","version":"222'3704","reported_seq":9545,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.418578+0000","last_change":"2023-07-02T14:06:25.548919+0000","last_active":"2023-07-02T14:07:48.418578+0000","last_peered":"2023-07-02T14:07:48.418578+0000","last_clean":"2023-07-02T14:07:48.418578+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:07:48.418578+0000","last_undegraded":"2023-07-02T14:07:48.418578+0000","last_fullsized":"2023-07-02T14:07:48.418578+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3704,"ondisk_log_size":3704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1575041492,"num_objects":472,"num_object_clones":397,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":59417,"num_read_kb":2052552,"num_write":3144,"num_write_kb":1530742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"216'3069","reported_seq":5261,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.421447+0000","last_change":"2023-07-02T14:05:12.118156+0000","last_active":"2023-07-02T14:07:48.421447+0000","last_peered":"2023-07-02T14:07:48.421447+0000","last_clean":"2023-07-02T14:07:48.421447+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:07:48.421447+0000","last_undegraded":"2023-07-02T14:07:48.421447+0000","last_fullsized":"2023-07-02T14:07:48.421447+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3069,"ondisk_log_size":3069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1538467250,"num_objects":486,"num_object_clones":415,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":2008,"num_read_kb":2154143,"num_write":2654,"num_write_kb":1492749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"221'3210","reported_seq":5691,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.418575+0000","last_change":"2023-07-02T14:07:41.397514+0000","last_active":"2023-07-02T14:07:48.418575+0000","last_peered":"2023-07-02T14:07:48.418575+0000","last_clean":"2023-07-02T14:07:48.418575+0000","last_became_active":"2023-07-02T14:07:41.396115+0000","last_became_peered":"2023-07-02T14:07:41.396115+0000","last_unstale":"2023-07-02T14:07:48.418575+0000","last_undegraded":"2023-07-02T14:07:48.418575+0000","last_fullsized":"2023-07-02T14:07:48.418575+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3210,"ondisk_log_size":3210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1517142988,"num_objects":428,"num_object_clones":355,"num_object_copies":856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":743,"num_read_kb":1976610,"num_write":2851,"num_write_kb":1467899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1078,"num_bytes_recovered":3879069754,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"216'6494","reported_seq":9013,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.419242+0000","last_change":"2023-07-02T14:07:40.398546+0000","last_active":"2023-07-02T14:07:48.419242+0000","last_peered":"2023-07-02T14:07:48.419242+0000","last_clean":"2023-07-02T14:07:48.419242+0000","last_became_active":"2023-07-02T14:07:40.395869+0000","last_became_peered":"2023-07-02T14:07:40.395869+0000","last_unstale":"2023-07-02T14:07:48.419242+0000","last_undegraded":"2023-07-02T14:07:48.419242+0000","last_fullsized":"2023-07-02T14:07:48.419242+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":6494,"ondisk_log_size":6494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1675535810,"num_objects":495,"num_object_clones":414,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":23017,"num_read_kb":2209627,"num_write":9605,"num_write_kb":1635009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":22,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.424665+0000","last_change":"2023-07-02T14:07:34.365379+0000","last_active":"2023-07-02T14:07:48.424665+0000","last_peered":"2023-07-02T14:07:48.424665+0000","last_clean":"2023-07-02T14:07:48.424665+0000","last_became_active":"2023-07-02T14:07:34.365070+0000","last_became_peered":"2023-07-02T14:07:34.365070+0000","last_unstale":"2023-07-02T14:07:48.424665+0000","last_undegraded":"2023-07-02T14:07:48.424665+0000","last_fullsized":"2023-07-02T14:07:48.424665+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:46.277830+0000","last_clean_scrub_stamp":"2023-07-02T14:01:46.277830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"216'2975","reported_seq":4506,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.422315+0000","last_change":"2023-07-02T14:07:45.413573+0000","last_active":"2023-07-02T14:07:48.422315+0000","last_peered":"2023-07-02T14:07:48.422315+0000","last_clean":"2023-07-02T14:07:48.422315+0000","last_became_active":"2023-07-02T14:07:45.412476+0000","last_became_peered":"2023-07-02T14:07:45.412476+0000","last_unstale":"2023-07-02T14:07:48.422315+0000","last_undegraded":"2023-07-02T14:07:48.422315+0000","last_fullsized":"2023-07-02T14:07:48.422315+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2975,"ondisk_log_size":2975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1661289598,"num_objects":499,"num_object_clones":416,"num_object_copies":998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":1528,"num_read_kb":2088102,"num_write":2559,"num_write_kb":1611276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":2494793202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"216'2884","reported_seq":3866,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.418819+0000","last_change":"2023-07-02T14:04:24.118435+0000","last_active":"2023-07-02T14:07:48.418819+0000","last_peered":"2023-07-02T14:07:48.418819+0000","last_clean":"2023-07-02T14:07:48.418819+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:07:48.418819+0000","last_undegraded":"2023-07-02T14:07:48.418819+0000","last_fullsized":"2023-07-02T14:07:48.418819+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2884,"ondisk_log_size":2884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1569543764,"num_objects":461,"num_object_clones":383,"num_object_copies":922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":1039,"num_read_kb":2026251,"num_write":2501,"num_write_kb":1523310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"216'2328","reported_seq":6067,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-07-02T14:07:48.421423+0000","last_change":"2023-07-02T14:07:40.399680+0000","last_active":"2023-07-02T14:07:48.421423+0000","last_peered":"2023-07-02T14:07:48.421423+0000","last_clean":"2023-07-02T14:07:48.421423+0000","last_became_active":"2023-07-02T14:07:40.398277+0000","last_became_peered":"2023-07-02T14:07:40.398277+0000","last_unstale":"2023-07-02T14:07:48.421423+0000","last_undegraded":"2023-07-02T14:07:48.421423+0000","last_fullsized":"2023-07-02T14:07:48.421423+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2328,"ondisk_log_size":2328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1230645001,"num_objects":405,"num_object_clones":325,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":1983,"num_read_kb":1773668,"num_write":2003,"num_write_kb":1186825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":904,"num_bytes_recovered":2809445763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"218'2366","reported_seq":4268,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:07:49.426548+0000","last_change":"2023-07-02T14:07:49.426548+0000","last_active":"2023-07-02T14:07:49.426548+0000","last_peered":"2023-07-02T14:07:49.426548+0000","last_clean":"2023-07-02T14:07:49.426548+0000","last_became_active":"2023-07-02T14:07:49.425350+0000","last_became_peered":"2023-07-02T14:07:49.425350+0000","last_unstale":"2023-07-02T14:07:49.426548+0000","last_undegraded":"2023-07-02T14:07:49.426548+0000","last_fullsized":"2023-07-02T14:07:49.426548+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":2366,"ondisk_log_size":2366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1146537804,"num_objects":358,"num_object_clones":299,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":1103,"num_read_kb":1573773,"num_write":2067,"num_write_kb":1113633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":792,"num_bytes_recovered":2631686774,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90838,"num_read_kb":15854726,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5308,"num_bytes_recovered":17939292043,"num_keys_recovered":203,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":25858924544,"data_stored":30184618536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":27030,"ondisk_log_size":27030,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640333,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4026312,"kb_used_data":3989312,"kb_used_omap":0,"kb_used_meta":36992,"kb_avail":90345528,"statfs":{"total":96636764160,"available":92513820672,"internally_reserved":0,"allocated":4085055488,"data_stored":4773662384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37879808},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.499}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.42899999999999999,"15min":0.42699999999999999},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.0509999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.45000000000000001,"15min":0.45700000000000002},"min":{"1min":0.28299999999999997,"5min":0.224,"15min":0.224},"max":{"1min":0.80700000000000005,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":0.54900000000000004}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.43099999999999999,"15min":0.42699999999999999},"min":{"1min":0.29699999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.1850000000000001,"5min":2.157,"15min":2.157},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.439,"15min":0.44900000000000001},"min":{"1min":0.26500000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73699999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.53600000000000003}]},{"osd":3,"last update":"Sun Jul 2 14:07:05 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.38100000000000001,"15min":0.36299999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.2919999999999998,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.442},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.36399999999999999,"15min":0.35899999999999999},"min":{"1min":0.28000000000000003,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.39200000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.39800000000000002,"15min":0.41699999999999998},"min":{"1min":0.27600000000000002,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.99299999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.39100000000000001,"15min":0.42899999999999999},"min":{"1min":0.29499999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.65700000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.40500000000000003}]}]},{"osd":4,"up_from":11,"seq":47244640333,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3882852,"kb_used_data":3786524,"kb_used_omap":0,"kb_used_meta":96320,"kb_avail":90488988,"statfs":{"total":96636764160,"available":92660723712,"internally_reserved":0,"allocated":3877400576,"data_stored":4464061785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":98631680},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.496},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.48899999999999999}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44,"15min":0.46400000000000002},"min":{"1min":0.28399999999999997,"5min":0.215,"15min":0.215},"max":{"1min":0.71499999999999997,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.44500000000000001,"15min":0.45000000000000001},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.79700000000000004,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.55000000000000004}]},{"osd":2,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47599999999999998,"15min":0.48399999999999999},"min":{"1min":0.32800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66800000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.57399999999999995}]},{"osd":3,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.378,"15min":0.38300000000000001},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.45900000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.36799999999999999,"15min":0.36299999999999999},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.48799999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.45000000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.44500000000000001},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.44900000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.433,"15min":0.49299999999999999},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.59099999999999997,"5min":1.026,"15min":1.026},"last":0.42899999999999999}]}]},{"osd":3,"up_from":133,"seq":571230650405,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4569484,"kb_used_data":4499076,"kb_used_omap":229,"kb_used_meta":70170,"kb_avail":89802356,"statfs":{"total":96636764160,"available":91957612544,"internally_reserved":0,"allocated":4607053824,"data_stored":5358018351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":71854302},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:07:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.36899999999999999,"15min":0.36599999999999999},"min":{"1min":0.26100000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.375,"15min":0.371},"min":{"1min":0.26900000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.433}]},{"osd":1,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.372,"15min":0.36599999999999999},"min":{"1min":0.22600000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.26200000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.46000000000000002}]},{"osd":2,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.28699999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.44900000000000001,"15min":0.439},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.47199999999999998}]},{"osd":4,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.35799999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.36199999999999999,"15min":0.35299999999999998},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.47999999999999998}]},{"osd":5,"last update":"Sun Jul 2 14:07:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.39800000000000002},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.496},{"interface":"front","average":{"1min":0.439,"5min":0.39500000000000002,"15min":0.38800000000000001},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.45100000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640333,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3200836,"kb_used_data":3146556,"kb_used_omap":0,"kb_used_meta":54272,"kb_avail":91171004,"statfs":{"total":96636764160,"available":93359108096,"internally_reserved":0,"allocated":3222073344,"data_stored":3710453423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":55574528},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.313,"15min":0.313},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.17000000000000001},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.20300000000000001}]},{"osd":1,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.42499999999999999,"15min":0.374},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":4.5179999999999998,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":0.254},{"interface":"front","average":{"1min":0.376,"5min":0.33500000000000002,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.23999999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:07:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.91100000000000003,"5min":1.929,"15min":1.929},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.54800000000000004,"15min":0.55600000000000005},"min":{"1min":0.30599999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93999999999999995,"5min":2,"15min":2},"last":0.29899999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.312},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.33700000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:06:55 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48799999999999999,"15min":0.48599999999999999},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68400000000000005,"5min":1.071,"15min":1.071},"last":0.372},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.499,"15min":0.48899999999999999},"min":{"1min":0.34300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":0.35799999999999998}]}]},{"osd":0,"up_from":166,"seq":712964571163,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2680400,"kb_used_data":2627784,"kb_used_omap":884,"kb_used_meta":51723,"kb_avail":91691440,"statfs":{"total":96636764160,"available":93892034560,"internally_reserved":0,"allocated":2690850816,"data_stored":3042367221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":905963,"internal_metadata":52964629},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 2 14:07:52 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.36699999999999999,"15min":0.34699999999999998},"min":{"1min":0.20699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.3849999999999998,"5min":3.3849999999999998,"15min":3.3849999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.40500000000000003},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.4100000000000001,"5min":3.4100000000000001,"15min":3.4100000000000001},"last":0.45000000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:07:52 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.34000000000000002,"15min":0.34699999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.45800000000000002,"5min":0.876,"15min":0.876},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.34599999999999997,"15min":0.34100000000000003},"min":{"1min":0.23899999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.432}]},{"osd":3,"last update":"Sun Jul 2 14:07:52 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.46600000000000003,"15min":0.46800000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.45000000000000001,"15min":0.438},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":0.65600000000000003}]},{"osd":4,"last update":"Sun Jul 2 14:07:52 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.49099999999999999,"15min":0.495},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.501,"15min":0.50600000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.73399999999999999,"5min":1.21,"15min":1.21},"last":0.61199999999999999}]},{"osd":5,"last update":"Sun Jul 2 14:07:52 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48699999999999999,"15min":0.48899999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.64000000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.503,"15min":0.50600000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.71899999999999997,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.60099999999999998}]}]},{"osd":1,"up_from":11,"seq":47244640333,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7352732,"kb_used_data":7220692,"kb_used_omap":0,"kb_used_meta":132032,"kb_avail":87019108,"statfs":{"total":96636764160,"available":89107566592,"internally_reserved":0,"allocated":7393988608,"data_stored":8842930298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":135200768},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:06:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.32000000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.36599999999999999,"15min":0.34100000000000003},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.35399999999999998,"15min":0.372},"min":{"1min":0.224,"5min":0.184,"15min":0.184},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.35999999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:07:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.49099999999999999,"15min":0.47299999999999998},"min":{"1min":0.30299999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.39300000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.496,"15min":0.50600000000000001},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.3300000000000001,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.503},"min":{"1min":0.34999999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71899999999999997,"5min":1.524,"15min":1.524},"last":0.45300000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:06:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.80300000000000005,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.495},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.84399999999999997,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.52600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2687934464,"data_stored":3041221400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7391072256,"data_stored":8841784477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3219156992,"data_stored":3709307602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4604137472,"data_stored":5356872530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3874484224,"data_stored":4462915964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4082139136,"data_stored":4772516563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:08:01.947 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-07-02T14:01:48.285721+0000 time.struct_time(tm_year=2023, tm_mon=7, tm_mday=2, tm_hour=14, tm_min=1, tm_sec=48, tm_wday=6, tm_yday=183, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=7, tm_mday=2, tm_hour=14, tm_min=7, tm_sec=57, tm_wday=6, tm_yday=183, tm_isdst=0) 2023-07-02T14:08:01.947 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-07-02T14:01:48.285721+0000 time.struct_time(tm_year=2023, tm_mon=7, tm_mday=2, tm_hour=14, tm_min=1, tm_sec=48, tm_wday=6, tm_yday=183, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=7, tm_mday=2, tm_hour=14, tm_min=7, tm_sec=57, tm_wday=6, tm_yday=183, tm_isdst=0) 2023-07-02T14:08:01.948 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-07-02T14:01:48.285721+0000 time.struct_time(tm_year=2023, tm_mon=7, tm_mday=2, tm_hour=14, tm_min=1, tm_sec=48, tm_wday=6, tm_yday=183, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=7, tm_mday=2, tm_hour=14, tm_min=7, tm_sec=57, tm_wday=6, tm_yday=183, tm_isdst=0) 2023-07-02T14:08:01.948 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-07-02T14:01:48.285721+0000 time.struct_time(tm_year=2023, tm_mon=7, tm_mday=2, tm_hour=14, tm_min=1, tm_sec=48, tm_wday=6, tm_yday=183, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=7, tm_mday=2, tm_hour=14, tm_min=7, tm_sec=57, tm_wday=6, tm_yday=183, tm_isdst=0) 2023-07-02T14:08:01.948 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-07-02T14:01:46.277830+0000 time.struct_time(tm_year=2023, tm_mon=7, tm_mday=2, tm_hour=14, tm_min=1, tm_sec=46, tm_wday=6, tm_yday=183, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=7, tm_mday=2, tm_hour=14, tm_min=7, tm_sec=57, tm_wday=6, tm_yday=183, tm_isdst=0) 2023-07-02T14:08:01.948 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-07-02T14:01:48.285721+0000 time.struct_time(tm_year=2023, tm_mon=7, tm_mday=2, tm_hour=14, tm_min=1, tm_sec=48, tm_wday=6, tm_yday=183, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=7, tm_mday=2, tm_hour=14, tm_min=7, tm_sec=57, tm_wday=6, tm_yday=183, tm_isdst=0) 2023-07-02T14:08:01.949 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-07-02T14:01:48.285721+0000 time.struct_time(tm_year=2023, tm_mon=7, tm_mday=2, tm_hour=14, tm_min=1, tm_sec=48, tm_wday=6, tm_yday=183, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=7, tm_mday=2, tm_hour=14, tm_min=7, tm_sec=57, tm_wday=6, tm_yday=183, tm_isdst=0) 2023-07-02T14:08:01.949 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-07-02T14:01:48.285721+0000 time.struct_time(tm_year=2023, tm_mon=7, tm_mday=2, tm_hour=14, tm_min=1, tm_sec=48, tm_wday=6, tm_yday=183, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=7, tm_mday=2, tm_hour=14, tm_min=7, tm_sec=57, tm_wday=6, tm_yday=183, tm_isdst=0) 2023-07-02T14:08:01.949 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-07-02T14:01:48.285721+0000 time.struct_time(tm_year=2023, tm_mon=7, tm_mday=2, tm_hour=14, tm_min=1, tm_sec=48, tm_wday=6, tm_yday=183, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=7, tm_mday=2, tm_hour=14, tm_min=7, tm_sec=57, tm_wday=6, tm_yday=183, tm_isdst=0) 2023-07-02T14:08:01.950 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-07-02T14:08:21.950 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:08:22.262 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:08:22.263 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:08:22.274 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2023-07-02T14:08:20.304186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90838,"num_read_kb":15854726,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5308,"num_bytes_recovered":17939292043,"num_keys_recovered":203,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4498,"num_omap_keys":88,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30148,"ondisk_log_size":30148,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":23782224,"kb_used_data":23300896,"kb_used_omap":1114,"kb_used_meta":480165,"kb_avail":542448816,"statfs":{"total":579820584960,"available":555467587584,"internally_reserved":0,"allocated":23860117504,"data_stored":27734404004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1141261,"internal_metadata":491689459},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4361,"num_omap_keys":79,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002087"},"pg_stats":[{"pgid":"2.7","version":"241'4176","reported_seq":10024,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:08:14.570747+0000","last_change":"2023-07-02T14:08:14.570747+0000","last_active":"2023-07-02T14:08:14.570747+0000","last_peered":"2023-07-02T14:08:14.570747+0000","last_clean":"2023-07-02T14:08:14.570747+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:08:14.570747+0000","last_undegraded":"2023-07-02T14:08:14.570747+0000","last_fullsized":"2023-07-02T14:08:14.570747+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'4176","last_scrub_stamp":"2023-07-02T14:08:14.570668+0000","last_deep_scrub":"241'4176","last_deep_scrub_stamp":"2023-07-02T14:08:14.570668+0000","last_clean_scrub_stamp":"2023-07-02T14:08:14.570668+0000","log_size":4176,"ondisk_log_size":4176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1575041492,"num_objects":472,"num_object_clones":397,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":59417,"num_read_kb":2052552,"num_write":3144,"num_write_kb":1530742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4241,"num_omap_keys":75,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"216'3069","reported_seq":5265,"reported_epoch":241,"state":"active+clean+scrubbing+deep","last_fresh":"2023-07-02T14:08:15.260264+0000","last_change":"2023-07-02T14:08:15.260264+0000","last_active":"2023-07-02T14:08:15.260264+0000","last_peered":"2023-07-02T14:08:15.260264+0000","last_clean":"2023-07-02T14:08:15.260264+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:08:15.260264+0000","last_undegraded":"2023-07-02T14:08:15.260264+0000","last_fullsized":"2023-07-02T14:08:15.260264+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:01:48.285721+0000","last_clean_scrub_stamp":"2023-07-02T14:01:48.285721+0000","log_size":3069,"ondisk_log_size":3069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1538467250,"num_objects":486,"num_object_clones":415,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":2008,"num_read_kb":2154143,"num_write":2654,"num_write_kb":1492749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"241'3638","reported_seq":6124,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:08:03.218274+0000","last_change":"2023-07-02T14:08:03.218274+0000","last_active":"2023-07-02T14:08:03.218274+0000","last_peered":"2023-07-02T14:08:03.218274+0000","last_clean":"2023-07-02T14:08:03.218274+0000","last_became_active":"2023-07-02T14:07:41.396115+0000","last_became_peered":"2023-07-02T14:07:41.396115+0000","last_unstale":"2023-07-02T14:08:03.218274+0000","last_undegraded":"2023-07-02T14:08:03.218274+0000","last_fullsized":"2023-07-02T14:08:03.218274+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'3638","last_scrub_stamp":"2023-07-02T14:08:03.218189+0000","last_deep_scrub":"241'3638","last_deep_scrub_stamp":"2023-07-02T14:08:03.218189+0000","last_clean_scrub_stamp":"2023-07-02T14:08:03.218189+0000","log_size":3638,"ondisk_log_size":3638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1517142988,"num_objects":428,"num_object_clones":355,"num_object_copies":856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":743,"num_read_kb":1976610,"num_write":2851,"num_write_kb":1467899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1078,"num_bytes_recovered":3879069754,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":137,"num_omap_keys":9,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"241'6989","reported_seq":9513,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:08:08.317238+0000","last_change":"2023-07-02T14:08:08.317238+0000","last_active":"2023-07-02T14:08:08.317238+0000","last_peered":"2023-07-02T14:08:08.317238+0000","last_clean":"2023-07-02T14:08:08.317238+0000","last_became_active":"2023-07-02T14:07:40.395869+0000","last_became_peered":"2023-07-02T14:07:40.395869+0000","last_unstale":"2023-07-02T14:08:08.317238+0000","last_undegraded":"2023-07-02T14:08:08.317238+0000","last_fullsized":"2023-07-02T14:08:08.317238+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'6989","last_scrub_stamp":"2023-07-02T14:08:08.317138+0000","last_deep_scrub":"241'6989","last_deep_scrub_stamp":"2023-07-02T14:08:08.317138+0000","last_clean_scrub_stamp":"2023-07-02T14:08:08.317138+0000","log_size":6989,"ondisk_log_size":6989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1675535810,"num_objects":495,"num_object_clones":414,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":23017,"num_read_kb":2209627,"num_write":9605,"num_write_kb":1635009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":27,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:07:59.574575+0000","last_change":"2023-07-02T14:07:59.574575+0000","last_active":"2023-07-02T14:07:59.574575+0000","last_peered":"2023-07-02T14:07:59.574575+0000","last_clean":"2023-07-02T14:07:59.574575+0000","last_became_active":"2023-07-02T14:07:34.365070+0000","last_became_peered":"2023-07-02T14:07:34.365070+0000","last_unstale":"2023-07-02T14:07:59.574575+0000","last_undegraded":"2023-07-02T14:07:59.574575+0000","last_fullsized":"2023-07-02T14:07:59.574575+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:07:59.574478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:07:59.574478+0000","last_clean_scrub_stamp":"2023-07-02T14:07:59.574478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"241'3474","reported_seq":5012,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:08:07.689398+0000","last_change":"2023-07-02T14:08:07.689398+0000","last_active":"2023-07-02T14:08:07.689398+0000","last_peered":"2023-07-02T14:08:07.689398+0000","last_clean":"2023-07-02T14:08:07.689398+0000","last_became_active":"2023-07-02T14:07:45.412476+0000","last_became_peered":"2023-07-02T14:07:45.412476+0000","last_unstale":"2023-07-02T14:08:07.689398+0000","last_undegraded":"2023-07-02T14:08:07.689398+0000","last_fullsized":"2023-07-02T14:08:07.689398+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'3474","last_scrub_stamp":"2023-07-02T14:08:07.689303+0000","last_deep_scrub":"241'3474","last_deep_scrub_stamp":"2023-07-02T14:08:07.689303+0000","last_clean_scrub_stamp":"2023-07-02T14:08:07.689303+0000","log_size":3474,"ondisk_log_size":3474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1661289598,"num_objects":499,"num_object_clones":416,"num_object_copies":998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":1528,"num_read_kb":2088102,"num_write":2559,"num_write_kb":1611276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":2494793202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"241'3345","reported_seq":4332,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:08:03.289691+0000","last_change":"2023-07-02T14:08:03.289691+0000","last_active":"2023-07-02T14:08:03.289691+0000","last_peered":"2023-07-02T14:08:03.289691+0000","last_clean":"2023-07-02T14:08:03.289691+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:08:03.289691+0000","last_undegraded":"2023-07-02T14:08:03.289691+0000","last_fullsized":"2023-07-02T14:08:03.289691+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'3345","last_scrub_stamp":"2023-07-02T14:08:03.289603+0000","last_deep_scrub":"241'3345","last_deep_scrub_stamp":"2023-07-02T14:08:03.289603+0000","last_clean_scrub_stamp":"2023-07-02T14:08:03.289603+0000","log_size":3345,"ondisk_log_size":3345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1569543764,"num_objects":461,"num_object_clones":383,"num_object_copies":922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":1039,"num_read_kb":2026251,"num_write":2501,"num_write_kb":1523310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"241'2733","reported_seq":6479,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:08:13.699689+0000","last_change":"2023-07-02T14:08:13.699689+0000","last_active":"2023-07-02T14:08:13.699689+0000","last_peered":"2023-07-02T14:08:13.699689+0000","last_clean":"2023-07-02T14:08:13.699689+0000","last_became_active":"2023-07-02T14:07:40.398277+0000","last_became_peered":"2023-07-02T14:07:40.398277+0000","last_unstale":"2023-07-02T14:08:13.699689+0000","last_undegraded":"2023-07-02T14:08:13.699689+0000","last_fullsized":"2023-07-02T14:08:13.699689+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'2733","last_scrub_stamp":"2023-07-02T14:08:13.699595+0000","last_deep_scrub":"241'2733","last_deep_scrub_stamp":"2023-07-02T14:08:13.699595+0000","last_clean_scrub_stamp":"2023-07-02T14:08:13.699595+0000","log_size":2733,"ondisk_log_size":2733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1230645001,"num_objects":405,"num_object_clones":325,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":1983,"num_read_kb":1773668,"num_write":2003,"num_write_kb":1186825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":904,"num_bytes_recovered":2809445763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"241'2724","reported_seq":4631,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:08:09.738499+0000","last_change":"2023-07-02T14:08:09.738499+0000","last_active":"2023-07-02T14:08:09.738499+0000","last_peered":"2023-07-02T14:08:09.738499+0000","last_clean":"2023-07-02T14:08:09.738499+0000","last_became_active":"2023-07-02T14:07:49.425350+0000","last_became_peered":"2023-07-02T14:07:49.425350+0000","last_unstale":"2023-07-02T14:08:09.738499+0000","last_undegraded":"2023-07-02T14:08:09.738499+0000","last_fullsized":"2023-07-02T14:08:09.738499+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'2724","last_scrub_stamp":"2023-07-02T14:08:09.738419+0000","last_deep_scrub":"241'2724","last_deep_scrub_stamp":"2023-07-02T14:08:09.738419+0000","last_clean_scrub_stamp":"2023-07-02T14:08:09.738419+0000","log_size":2724,"ondisk_log_size":2724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1146537804,"num_objects":358,"num_object_clones":299,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":1103,"num_read_kb":1573773,"num_write":2067,"num_write_kb":1113633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":792,"num_bytes_recovered":2631686774,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":120,"num_omap_keys":4,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90838,"num_read_kb":15854726,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5308,"num_bytes_recovered":17939292043,"num_keys_recovered":203,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4498,"num_omap_keys":88,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23842619392,"data_stored":27727529078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":30148,"ondisk_log_size":30148,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640337,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4031944,"kb_used_data":3989312,"kb_used_omap":0,"kb_used_meta":42624,"kb_avail":90339896,"statfs":{"total":96636764160,"available":92508053504,"internally_reserved":0,"allocated":4085055488,"data_stored":4773662384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43646976},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:07:57 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38,"15min":0.38200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.378},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.44,"15min":0.42799999999999999},"min":{"1min":0.252,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.3780000000000001,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":0.44600000000000001}]},{"osd":1,"last update":"Sun Jul 2 14:07:57 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.42999999999999999,"15min":0.42799999999999999},"min":{"1min":0.27300000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.68600000000000005,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.436,"5min":0.44500000000000001,"15min":0.45500000000000002},"min":{"1min":0.26800000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.73999999999999999,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":0.42499999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:07:57 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.434,"15min":0.42799999999999999},"min":{"1min":0.30399999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.77200000000000002,"5min":2.157,"15min":2.157},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.45800000000000002,"15min":0.45500000000000002},"min":{"1min":0.28999999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.52800000000000002}]},{"osd":3,"last update":"Sun Jul 2 14:08:08 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.38300000000000001,"15min":0.36299999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.52600000000000002,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.37,"15min":0.36099999999999999},"min":{"1min":0.27400000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.54500000000000004,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.45900000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:07:57 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.40300000000000002,"15min":0.41899999999999998},"min":{"1min":0.35999999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.59999999999999998,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.40999999999999998,"15min":0.435},"min":{"1min":0.25900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":2.3519999999999999,"5min":2.3519999999999999,"15min":2.3519999999999999},"last":0.495}]}]},{"osd":4,"up_from":11,"seq":47244640337,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3306736,"kb_used_data":3204584,"kb_used_omap":0,"kb_used_meta":102144,"kb_avail":91065104,"statfs":{"total":96636764160,"available":93250666496,"internally_reserved":0,"allocated":3281494016,"data_stored":3733003123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":104595456},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:07:59 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.315},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.40000000000000002,"15min":0.39600000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.28299999999999997}]},{"osd":1,"last update":"Sun Jul 2 14:07:59 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.43099999999999999,"15min":0.46100000000000002},"min":{"1min":0.27700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.71399999999999997,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.32200000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73399999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.40999999999999998}]},{"osd":2,"last update":"Sun Jul 2 14:07:59 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.44,"15min":0.45000000000000001},"min":{"1min":0.32400000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.61799999999999999,"5min":1.389,"15min":1.389},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.38300000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.75700000000000001,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.432}]},{"osd":3,"last update":"Sun Jul 2 14:08:08 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.375,"15min":0.38200000000000001},"min":{"1min":0.26600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.51000000000000001,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.38100000000000001,"15min":0.36699999999999999},"min":{"1min":0.29599999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.379}]},{"osd":5,"last update":"Sun Jul 2 14:07:59 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.375,"15min":0.441},"min":{"1min":0.29099999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.57699999999999996,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.42199999999999999,"15min":0.48999999999999999},"min":{"1min":0.318,"5min":0.246,"15min":0.246},"max":{"1min":0.66900000000000004,"5min":1.026,"15min":1.026},"last":0.372}]}]},{"osd":3,"up_from":133,"seq":571230650409,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4579276,"kb_used_data":4499076,"kb_used_omap":229,"kb_used_meta":79962,"kb_avail":89792564,"statfs":{"total":96636764160,"available":91947585536,"internally_reserved":0,"allocated":4607053824,"data_stored":5358018351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":81881310},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:07:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.36899999999999999,"15min":0.36599999999999999},"min":{"1min":0.26100000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.315},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.375,"15min":0.371},"min":{"1min":0.26900000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.26800000000000002}]},{"osd":1,"last update":"Sun Jul 2 14:08:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.38100000000000001,"15min":0.36899999999999999},"min":{"1min":0.24399999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.55300000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.40300000000000002,"15min":0.40100000000000002},"min":{"1min":0.28199999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.54500000000000004,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.27900000000000003}]},{"osd":2,"last update":"Sun Jul 2 14:08:08 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.41799999999999998,"15min":0.39600000000000002},"min":{"1min":0.29999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.58499999999999996,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.34200000000000003,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.59799999999999998,"5min":1.907,"15min":1.907},"last":0.309}]},{"osd":4,"last update":"Sun Jul 2 14:08:08 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.35299999999999998,"15min":0.35599999999999998},"min":{"1min":0.23999999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.502,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.35499999999999998},"min":{"1min":0.29699999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.59499999999999997,"5min":0.625,"15min":0.625},"last":0.40100000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:08:08 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40000000000000002,"15min":0.39800000000000002},"min":{"1min":0.27900000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.56100000000000005,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.37},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.40300000000000002,"15min":0.39100000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.46100000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640337,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3206148,"kb_used_data":3146556,"kb_used_omap":0,"kb_used_meta":59584,"kb_avail":91165692,"statfs":{"total":96636764160,"available":93353668608,"internally_reserved":0,"allocated":3222073344,"data_stored":3710453423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61014016},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:07:55 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.317,"15min":0.314},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.51300000000000001,"5min":0.87,"15min":0.87},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.36499999999999999,"15min":0.33100000000000002},"min":{"1min":0.20300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":4.3650000000000002,"5min":4.3650000000000002,"15min":4.3650000000000002},"last":0.51500000000000001}]},{"osd":1,"last update":"Sun Jul 2 14:07:55 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45600000000000002,"15min":0.38400000000000001},"min":{"1min":0.24399999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":2.0720000000000001,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.37,"5min":0.34499999999999997,"15min":0.32700000000000001},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.83399999999999996,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.502}]},{"osd":3,"last update":"Sun Jul 2 14:08:06 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.53600000000000003,"15min":0.56899999999999995},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.83299999999999996,"5min":1.929,"15min":1.929},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.53200000000000003,"15min":0.55100000000000005},"min":{"1min":0.29899999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93700000000000006,"5min":2,"15min":2},"last":0.56699999999999995}]},{"osd":4,"last update":"Sun Jul 2 14:07:55 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47399999999999998,"15min":0.45800000000000002},"min":{"1min":0.28899999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.91400000000000003,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.48599999999999999,"15min":0.46899999999999997},"min":{"1min":0.309,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.80300000000000005,"5min":1.905,"15min":1.905},"last":0.59099999999999997}]},{"osd":5,"last update":"Sun Jul 2 14:07:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.505,"15min":0.49199999999999999},"min":{"1min":0.34100000000000003,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.50700000000000001,"15min":0.49099999999999999},"min":{"1min":0.35799999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.874,"5min":1.968,"15min":1.968},"last":0.61899999999999999}]}]},{"osd":0,"up_from":166,"seq":712964571167,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2685456,"kb_used_data":2627784,"kb_used_omap":884,"kb_used_meta":56779,"kb_avail":91686384,"statfs":{"total":96636764160,"available":93886857216,"internally_reserved":0,"allocated":2690850816,"data_stored":3042367221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":905963,"internal_metadata":58141973},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 2 14:07:52 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.36699999999999999,"15min":0.34699999999999998},"min":{"1min":0.20699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.3849999999999998,"5min":3.3849999999999998,"15min":3.3849999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.40500000000000003},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.4100000000000001,"5min":3.4100000000000001,"15min":3.4100000000000001},"last":0.54100000000000004}]},{"osd":2,"last update":"Sun Jul 2 14:07:52 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.34000000000000002,"15min":0.34699999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.45800000000000002,"5min":0.876,"15min":0.876},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.34599999999999997,"15min":0.34100000000000003},"min":{"1min":0.23899999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.49299999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:07:52 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.46600000000000003,"15min":0.46800000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.45000000000000001,"15min":0.438},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":0.42999999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:07:52 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.49099999999999999,"15min":0.495},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.501,"15min":0.50600000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.73399999999999999,"5min":1.21,"15min":1.21},"last":0.45800000000000002}]},{"osd":5,"last update":"Sun Jul 2 14:07:52 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48699999999999999,"15min":0.48899999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.64000000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.503,"15min":0.50600000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.71899999999999997,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.53000000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640337,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5972664,"kb_used_data":5833584,"kb_used_omap":0,"kb_used_meta":139072,"kb_avail":88399176,"statfs":{"total":96636764160,"available":90520756224,"internally_reserved":0,"allocated":5973590016,"data_stored":7116899502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":142409728},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:07:57 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.39300000000000002,"15min":0.40999999999999998},"min":{"1min":0.19,"5min":0.188,"15min":0.188},"max":{"1min":0.39100000000000001,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.309},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.33100000000000002,"15min":0.33500000000000002},"min":{"1min":0.20999999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.36699999999999999,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.34000000000000002}]},{"osd":2,"last update":"Sun Jul 2 14:08:01 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.378,"15min":0.34499999999999997},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.105,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34599999999999997,"15min":0.37},"min":{"1min":0.251,"5min":0.184,"15min":0.184},"max":{"1min":0.53400000000000003,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.29299999999999998}]},{"osd":3,"last update":"Sun Jul 2 14:08:10 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.49199999999999999,"15min":0.47299999999999998},"min":{"1min":0.29899999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.73999999999999999,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28599999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.54300000000000004,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.38200000000000001}]},{"osd":4,"last update":"Sun Jul 2 14:08:01 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.48299999999999998,"15min":0.501},"min":{"1min":0.32600000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.57799999999999996,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":1.2849999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.48499999999999999,"15min":0.5},"min":{"1min":0.36899999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.69799999999999995,"5min":1.524,"15min":1.524},"last":0.41599999999999998}]},{"osd":5,"last update":"Sun Jul 2 14:08:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51400000000000001,"15min":0.51500000000000001},"min":{"1min":0.36499999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.64600000000000002,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.41899999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.64800000000000002,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.502}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2687934464,"data_stored":3041221400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5970673664,"data_stored":7115753681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3219156992,"data_stored":3709307602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4604137472,"data_stored":5356872530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3278577664,"data_stored":3731857302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4082139136,"data_stored":4772516563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:08:22.277 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-07-02T14:01:48.285721+0000 time.struct_time(tm_year=2023, tm_mon=7, tm_mday=2, tm_hour=14, tm_min=1, tm_sec=48, tm_wday=6, tm_yday=183, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=7, tm_mday=2, tm_hour=14, tm_min=7, tm_sec=57, tm_wday=6, tm_yday=183, tm_isdst=0) 2023-07-02T14:08:22.277 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-07-02T14:08:42.278 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-02T14:08:42.587 INFO:teuthology.orchestra.run.smithi016.stdout: 2023-07-02T14:08:42.588 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2023-07-02T14:08:42.599 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":445,"stamp":"2023-07-02T14:08:42.308807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90838,"num_read_kb":15854726,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5308,"num_bytes_recovered":17939292043,"num_keys_recovered":203,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4498,"num_omap_keys":88,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30634,"ondisk_log_size":30634,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":23750224,"kb_used_data":23300896,"kb_used_omap":1114,"kb_used_meta":448165,"kb_avail":542480816,"statfs":{"total":579820584960,"available":555500355584,"internally_reserved":0,"allocated":23860117504,"data_stored":27734404004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1141261,"internal_metadata":458921459},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002515"},"pg_stats":[{"pgid":"2.7","version":"241'4176","reported_seq":10024,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:08:14.570747+0000","last_change":"2023-07-02T14:08:14.570747+0000","last_active":"2023-07-02T14:08:14.570747+0000","last_peered":"2023-07-02T14:08:14.570747+0000","last_clean":"2023-07-02T14:08:14.570747+0000","last_became_active":"2023-07-02T14:06:25.546088+0000","last_became_peered":"2023-07-02T14:06:25.546088+0000","last_unstale":"2023-07-02T14:08:14.570747+0000","last_undegraded":"2023-07-02T14:08:14.570747+0000","last_fullsized":"2023-07-02T14:08:14.570747+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'4176","last_scrub_stamp":"2023-07-02T14:08:14.570668+0000","last_deep_scrub":"241'4176","last_deep_scrub_stamp":"2023-07-02T14:08:14.570668+0000","last_clean_scrub_stamp":"2023-07-02T14:08:14.570668+0000","log_size":4176,"ondisk_log_size":4176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1575041492,"num_objects":472,"num_object_clones":397,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":59417,"num_read_kb":2052552,"num_write":3144,"num_write_kb":1530742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":2670124012,"num_keys_recovered":174,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4241,"num_omap_keys":75,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"241'3555","reported_seq":5754,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:08:18.161049+0000","last_change":"2023-07-02T14:08:18.161049+0000","last_active":"2023-07-02T14:08:18.161049+0000","last_peered":"2023-07-02T14:08:18.161049+0000","last_clean":"2023-07-02T14:08:18.161049+0000","last_became_active":"2023-07-02T14:05:12.116444+0000","last_became_peered":"2023-07-02T14:05:12.116444+0000","last_unstale":"2023-07-02T14:08:18.161049+0000","last_undegraded":"2023-07-02T14:08:18.161049+0000","last_fullsized":"2023-07-02T14:08:18.161049+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'3555","last_scrub_stamp":"2023-07-02T14:08:18.160966+0000","last_deep_scrub":"241'3555","last_deep_scrub_stamp":"2023-07-02T14:08:18.160966+0000","last_clean_scrub_stamp":"2023-07-02T14:08:18.160966+0000","log_size":3555,"ondisk_log_size":3555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1538467250,"num_objects":486,"num_object_clones":415,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":2008,"num_read_kb":2154143,"num_write":2654,"num_write_kb":1492749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":602222916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"241'3638","reported_seq":6124,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:08:03.218274+0000","last_change":"2023-07-02T14:08:03.218274+0000","last_active":"2023-07-02T14:08:03.218274+0000","last_peered":"2023-07-02T14:08:03.218274+0000","last_clean":"2023-07-02T14:08:03.218274+0000","last_became_active":"2023-07-02T14:07:41.396115+0000","last_became_peered":"2023-07-02T14:07:41.396115+0000","last_unstale":"2023-07-02T14:08:03.218274+0000","last_undegraded":"2023-07-02T14:08:03.218274+0000","last_fullsized":"2023-07-02T14:08:03.218274+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'3638","last_scrub_stamp":"2023-07-02T14:08:03.218189+0000","last_deep_scrub":"241'3638","last_deep_scrub_stamp":"2023-07-02T14:08:03.218189+0000","last_clean_scrub_stamp":"2023-07-02T14:08:03.218189+0000","log_size":3638,"ondisk_log_size":3638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1517142988,"num_objects":428,"num_object_clones":355,"num_object_copies":856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":743,"num_read_kb":1976610,"num_write":2851,"num_write_kb":1467899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1078,"num_bytes_recovered":3879069754,"num_keys_recovered":17,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":137,"num_omap_keys":9,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"241'6989","reported_seq":9513,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:08:08.317238+0000","last_change":"2023-07-02T14:08:08.317238+0000","last_active":"2023-07-02T14:08:08.317238+0000","last_peered":"2023-07-02T14:08:08.317238+0000","last_clean":"2023-07-02T14:08:08.317238+0000","last_became_active":"2023-07-02T14:07:40.395869+0000","last_became_peered":"2023-07-02T14:07:40.395869+0000","last_unstale":"2023-07-02T14:08:08.317238+0000","last_undegraded":"2023-07-02T14:08:08.317238+0000","last_fullsized":"2023-07-02T14:08:08.317238+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'6989","last_scrub_stamp":"2023-07-02T14:08:08.317138+0000","last_deep_scrub":"241'6989","last_deep_scrub_stamp":"2023-07-02T14:08:08.317138+0000","last_clean_scrub_stamp":"2023-07-02T14:08:08.317138+0000","log_size":6989,"ondisk_log_size":6989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1675535810,"num_objects":495,"num_object_clones":414,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":0,"num_read":23017,"num_read_kb":2209627,"num_write":9605,"num_write_kb":1635009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":641,"num_bytes_recovered":2185800272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":27,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:07:59.574575+0000","last_change":"2023-07-02T14:07:59.574575+0000","last_active":"2023-07-02T14:07:59.574575+0000","last_peered":"2023-07-02T14:07:59.574575+0000","last_clean":"2023-07-02T14:07:59.574575+0000","last_became_active":"2023-07-02T14:07:34.365070+0000","last_became_peered":"2023-07-02T14:07:34.365070+0000","last_unstale":"2023-07-02T14:07:59.574575+0000","last_undegraded":"2023-07-02T14:07:59.574575+0000","last_fullsized":"2023-07-02T14:07:59.574575+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-02T14:07:59.574478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-02T14:07:59.574478+0000","last_clean_scrub_stamp":"2023-07-02T14:07:59.574478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"241'3474","reported_seq":5012,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:08:07.689398+0000","last_change":"2023-07-02T14:08:07.689398+0000","last_active":"2023-07-02T14:08:07.689398+0000","last_peered":"2023-07-02T14:08:07.689398+0000","last_clean":"2023-07-02T14:08:07.689398+0000","last_became_active":"2023-07-02T14:07:45.412476+0000","last_became_peered":"2023-07-02T14:07:45.412476+0000","last_unstale":"2023-07-02T14:08:07.689398+0000","last_undegraded":"2023-07-02T14:08:07.689398+0000","last_fullsized":"2023-07-02T14:08:07.689398+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'3474","last_scrub_stamp":"2023-07-02T14:08:07.689303+0000","last_deep_scrub":"241'3474","last_deep_scrub_stamp":"2023-07-02T14:08:07.689303+0000","last_clean_scrub_stamp":"2023-07-02T14:08:07.689303+0000","log_size":3474,"ondisk_log_size":3474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1661289598,"num_objects":499,"num_object_clones":416,"num_object_copies":998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":1528,"num_read_kb":2088102,"num_write":2559,"num_write_kb":1611276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":731,"num_bytes_recovered":2494793202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"241'3345","reported_seq":4332,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:08:03.289691+0000","last_change":"2023-07-02T14:08:03.289691+0000","last_active":"2023-07-02T14:08:03.289691+0000","last_peered":"2023-07-02T14:08:03.289691+0000","last_clean":"2023-07-02T14:08:03.289691+0000","last_became_active":"2023-07-02T14:04:24.117759+0000","last_became_peered":"2023-07-02T14:04:24.117759+0000","last_unstale":"2023-07-02T14:08:03.289691+0000","last_undegraded":"2023-07-02T14:08:03.289691+0000","last_fullsized":"2023-07-02T14:08:03.289691+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'3345","last_scrub_stamp":"2023-07-02T14:08:03.289603+0000","last_deep_scrub":"241'3345","last_deep_scrub_stamp":"2023-07-02T14:08:03.289603+0000","last_clean_scrub_stamp":"2023-07-02T14:08:03.289603+0000","log_size":3345,"ondisk_log_size":3345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1569543764,"num_objects":461,"num_object_clones":383,"num_object_copies":922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":1039,"num_read_kb":2026251,"num_write":2501,"num_write_kb":1523310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":666149350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"241'2733","reported_seq":6479,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:08:13.699689+0000","last_change":"2023-07-02T14:08:13.699689+0000","last_active":"2023-07-02T14:08:13.699689+0000","last_peered":"2023-07-02T14:08:13.699689+0000","last_clean":"2023-07-02T14:08:13.699689+0000","last_became_active":"2023-07-02T14:07:40.398277+0000","last_became_peered":"2023-07-02T14:07:40.398277+0000","last_unstale":"2023-07-02T14:08:13.699689+0000","last_undegraded":"2023-07-02T14:08:13.699689+0000","last_fullsized":"2023-07-02T14:08:13.699689+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'2733","last_scrub_stamp":"2023-07-02T14:08:13.699595+0000","last_deep_scrub":"241'2733","last_deep_scrub_stamp":"2023-07-02T14:08:13.699595+0000","last_clean_scrub_stamp":"2023-07-02T14:08:13.699595+0000","log_size":2733,"ondisk_log_size":2733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1230645001,"num_objects":405,"num_object_clones":325,"num_object_copies":810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":1983,"num_read_kb":1773668,"num_write":2003,"num_write_kb":1186825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":904,"num_bytes_recovered":2809445763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"241'2724","reported_seq":4631,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-07-02T14:08:09.738499+0000","last_change":"2023-07-02T14:08:09.738499+0000","last_active":"2023-07-02T14:08:09.738499+0000","last_peered":"2023-07-02T14:08:09.738499+0000","last_clean":"2023-07-02T14:08:09.738499+0000","last_became_active":"2023-07-02T14:07:49.425350+0000","last_became_peered":"2023-07-02T14:07:49.425350+0000","last_unstale":"2023-07-02T14:08:09.738499+0000","last_undegraded":"2023-07-02T14:08:09.738499+0000","last_fullsized":"2023-07-02T14:08:09.738499+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'2724","last_scrub_stamp":"2023-07-02T14:08:09.738419+0000","last_deep_scrub":"241'2724","last_deep_scrub_stamp":"2023-07-02T14:08:09.738419+0000","last_clean_scrub_stamp":"2023-07-02T14:08:09.738419+0000","log_size":2724,"ondisk_log_size":2724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1146537804,"num_objects":358,"num_object_clones":299,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":1103,"num_read_kb":1573773,"num_write":2067,"num_write_kb":1113633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":792,"num_bytes_recovered":2631686774,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":120,"num_omap_keys":4,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":11914203707,"num_objects":3604,"num_object_clones":3004,"num_object_copies":7208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":90838,"num_read_kb":15854726,"num_write":27384,"num_write_kb":11561443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5308,"num_bytes_recovered":17939292043,"num_keys_recovered":203,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4498,"num_omap_keys":88,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23842619392,"data_stored":27727529078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57196,"internal_metadata":0},"log_size":30634,"ondisk_log_size":30634,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640342,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4031944,"kb_used_data":3989312,"kb_used_omap":0,"kb_used_meta":42624,"kb_avail":90339896,"statfs":{"total":96636764160,"available":92508053504,"internally_reserved":0,"allocated":4085055488,"data_stored":4773662384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43646976},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:07:57 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38,"15min":0.38200000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.44,"15min":0.42799999999999999},"min":{"1min":0.252,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.3780000000000001,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":0.65000000000000002}]},{"osd":1,"last update":"Sun Jul 2 14:07:57 2023","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.42999999999999999,"15min":0.42799999999999999},"min":{"1min":0.27300000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.68600000000000005,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.44500000000000001,"15min":0.45500000000000002},"min":{"1min":0.26800000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.73999999999999999,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":0.50700000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:07:57 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.434,"15min":0.42799999999999999},"min":{"1min":0.30399999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.77200000000000002,"5min":2.157,"15min":2.157},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.45800000000000002,"15min":0.45500000000000002},"min":{"1min":0.28999999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.3239999999999998,"5min":2.3239999999999998,"15min":2.3239999999999998},"last":0.58899999999999997}]},{"osd":3,"last update":"Sun Jul 2 14:08:08 2023","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.38300000000000001,"15min":0.36299999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.52600000000000002,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.37,"15min":0.36099999999999999},"min":{"1min":0.27400000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.54500000000000004,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.51800000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:07:57 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.40300000000000002,"15min":0.41899999999999998},"min":{"1min":0.35999999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.59999999999999998,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.40999999999999998,"15min":0.435},"min":{"1min":0.25900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":2.3519999999999999,"5min":2.3519999999999999,"15min":2.3519999999999999},"last":0.624}]}]},{"osd":4,"up_from":11,"seq":47244640342,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3306736,"kb_used_data":3204584,"kb_used_omap":0,"kb_used_meta":102144,"kb_avail":91065104,"statfs":{"total":96636764160,"available":93250666496,"internally_reserved":0,"allocated":3281494016,"data_stored":3733003123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":104595456},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:07:59 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.40000000000000002,"15min":0.39600000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.52000000000000002}]},{"osd":1,"last update":"Sun Jul 2 14:07:59 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.43099999999999999,"15min":0.46100000000000002},"min":{"1min":0.27700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.71399999999999997,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.32200000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73399999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.61399999999999999}]},{"osd":2,"last update":"Sun Jul 2 14:07:59 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.44,"15min":0.45000000000000001},"min":{"1min":0.32400000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.61799999999999999,"5min":1.389,"15min":1.389},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.38300000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.75700000000000001,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.53100000000000003}]},{"osd":3,"last update":"Sun Jul 2 14:08:08 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.375,"15min":0.38200000000000001},"min":{"1min":0.26600000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.51000000000000001,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.38100000000000001,"15min":0.36699999999999999},"min":{"1min":0.29599999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.44}]},{"osd":5,"last update":"Sun Jul 2 14:07:59 2023","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.375,"15min":0.441},"min":{"1min":0.29099999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.57699999999999996,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.42199999999999999,"15min":0.48999999999999999},"min":{"1min":0.318,"5min":0.246,"15min":0.246},"max":{"1min":0.66900000000000004,"5min":1.026,"15min":1.026},"last":0.46000000000000002}]}]},{"osd":3,"up_from":133,"seq":571230650413,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4563020,"kb_used_data":4499076,"kb_used_omap":229,"kb_used_meta":63706,"kb_avail":89808820,"statfs":{"total":96636764160,"available":91964231680,"internally_reserved":0,"allocated":4607053824,"data_stored":5358018351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":235298,"internal_metadata":65235166},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:07:55 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.36899999999999999,"15min":0.36599999999999999},"min":{"1min":0.26100000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.375,"15min":0.371},"min":{"1min":0.26900000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.59899999999999998}]},{"osd":1,"last update":"Sun Jul 2 14:08:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.38100000000000001,"15min":0.36899999999999999},"min":{"1min":0.24399999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.55300000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.40300000000000002,"15min":0.40100000000000002},"min":{"1min":0.28199999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.54500000000000004,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.67400000000000004}]},{"osd":2,"last update":"Sun Jul 2 14:08:08 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.41799999999999998,"15min":0.39600000000000002},"min":{"1min":0.29999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.58499999999999996,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.45700000000000002,"15min":0.441},"min":{"1min":0.34200000000000003,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.59799999999999998,"5min":1.907,"15min":1.907},"last":0.69899999999999995}]},{"osd":4,"last update":"Sun Jul 2 14:08:08 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.35299999999999998,"15min":0.35599999999999998},"min":{"1min":0.23999999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.502,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.36899999999999999,"15min":0.35499999999999998},"min":{"1min":0.29699999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.59499999999999997,"5min":0.625,"15min":0.625},"last":0.621}]},{"osd":5,"last update":"Sun Jul 2 14:08:08 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40000000000000002,"15min":0.39800000000000002},"min":{"1min":0.27900000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.56100000000000005,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.40300000000000002,"15min":0.39100000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.63,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.57899999999999996}]}]},{"osd":2,"up_from":11,"seq":47244640342,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3206148,"kb_used_data":3146556,"kb_used_omap":0,"kb_used_meta":59584,"kb_avail":91165692,"statfs":{"total":96636764160,"available":93353668608,"internally_reserved":0,"allocated":3222073344,"data_stored":3710453423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":61014016},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:07:55 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.317,"15min":0.314},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.51300000000000001,"5min":0.87,"15min":0.87},"last":0.255},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.36499999999999999,"15min":0.33100000000000002},"min":{"1min":0.20300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":4.3650000000000002,"5min":4.3650000000000002,"15min":4.3650000000000002},"last":0.314}]},{"osd":1,"last update":"Sun Jul 2 14:07:55 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45600000000000002,"15min":0.38400000000000001},"min":{"1min":0.24399999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":2.0720000000000001,"5min":4.5179999999999998,"15min":4.5179999999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.37,"5min":0.34499999999999997,"15min":0.32700000000000001},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.83399999999999996,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.30199999999999999}]},{"osd":3,"last update":"Sun Jul 2 14:08:06 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.53600000000000003,"15min":0.56899999999999995},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.83299999999999996,"5min":1.929,"15min":1.929},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.53200000000000003,"15min":0.55100000000000005},"min":{"1min":0.29899999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.93700000000000006,"5min":2,"15min":2},"last":0.61199999999999999}]},{"osd":4,"last update":"Sun Jul 2 14:07:55 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47399999999999998,"15min":0.45800000000000002},"min":{"1min":0.28899999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.91400000000000003,"5min":2.0009999999999999,"15min":2.0009999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.48599999999999999,"15min":0.46899999999999997},"min":{"1min":0.309,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.80300000000000005,"5min":1.905,"15min":1.905},"last":0.63500000000000001}]},{"osd":5,"last update":"Sun Jul 2 14:07:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.505,"15min":0.49199999999999999},"min":{"1min":0.34100000000000003,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.50700000000000001,"15min":0.49099999999999999},"min":{"1min":0.35799999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.874,"5min":1.968,"15min":1.968},"last":0.59699999999999998}]}]},{"osd":0,"up_from":166,"seq":712964571172,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2685456,"kb_used_data":2627784,"kb_used_omap":884,"kb_used_meta":56779,"kb_avail":91686384,"statfs":{"total":96636764160,"available":93886857216,"internally_reserved":0,"allocated":2690850816,"data_stored":3042367221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":905963,"internal_metadata":58141973},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jul 2 14:07:52 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.36699999999999999,"15min":0.34699999999999998},"min":{"1min":0.20699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.3849999999999998,"5min":3.3849999999999998,"15min":3.3849999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.40500000000000003},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.4100000000000001,"5min":3.4100000000000001,"15min":3.4100000000000001},"last":0.32500000000000001}]},{"osd":2,"last update":"Sun Jul 2 14:07:52 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.34000000000000002,"15min":0.34699999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.45800000000000002,"5min":0.876,"15min":0.876},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.34599999999999997,"15min":0.34100000000000003},"min":{"1min":0.23899999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.40100000000000002}]},{"osd":3,"last update":"Sun Jul 2 14:07:52 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.46600000000000003,"15min":0.46800000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.45000000000000001,"15min":0.438},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":0.47299999999999998}]},{"osd":4,"last update":"Sun Jul 2 14:07:52 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.49099999999999999,"15min":0.495},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.501,"15min":0.50600000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.73399999999999999,"5min":1.21,"15min":1.21},"last":0.48499999999999999}]},{"osd":5,"last update":"Sun Jul 2 14:07:52 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48699999999999999,"15min":0.48899999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.64000000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.503,"15min":0.50600000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.71899999999999997,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.53400000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640342,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5956920,"kb_used_data":5833584,"kb_used_omap":0,"kb_used_meta":123328,"kb_avail":88414920,"statfs":{"total":96636764160,"available":90536878080,"internally_reserved":0,"allocated":5973590016,"data_stored":7116899502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":126287872},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jul 2 14:07:57 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.39300000000000002,"15min":0.40999999999999998},"min":{"1min":0.19,"5min":0.188,"15min":0.188},"max":{"1min":0.39100000000000001,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.31},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.33100000000000002,"15min":0.33500000000000002},"min":{"1min":0.20999999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.36699999999999999,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.34499999999999997}]},{"osd":2,"last update":"Sun Jul 2 14:08:01 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.378,"15min":0.34499999999999997},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.105,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34599999999999997,"15min":0.37},"min":{"1min":0.251,"5min":0.184,"15min":0.184},"max":{"1min":0.53400000000000003,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.38400000000000001}]},{"osd":3,"last update":"Sun Jul 2 14:08:10 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.49199999999999999,"15min":0.47299999999999998},"min":{"1min":0.29899999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.73999999999999999,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28599999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.54300000000000004,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.46400000000000002}]},{"osd":4,"last update":"Sun Jul 2 14:08:01 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.48299999999999998,"15min":0.501},"min":{"1min":0.32600000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.57799999999999996,"5min":2.3300000000000001,"15min":2.3300000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.48499999999999999,"15min":0.5},"min":{"1min":0.36899999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.69799999999999995,"5min":1.524,"15min":1.524},"last":0.58699999999999997}]},{"osd":5,"last update":"Sun Jul 2 14:08:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51400000000000001,"15min":0.51500000000000001},"min":{"1min":0.36499999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.64600000000000002,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.624},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.41899999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.64800000000000002,"5min":1.5840000000000001,"15min":1.5840000000000001},"last":0.66200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2687934464,"data_stored":3041221400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53994,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5970673664,"data_stored":7115753681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3219156992,"data_stored":3709307602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4604137472,"data_stored":5356872530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3278577664,"data_stored":3731857302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4082139136,"data_stored":4772516563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-02T14:08:42.602 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-07-02T14:08:42.918 INFO:teuthology.misc:Shutting down mds daemons... 2023-07-02T14:08:42.919 INFO:teuthology.misc:Shutting down osd daemons... 2023-07-02T14:08:42.920 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-07-02T14:08:42.920 INFO:teuthology.orchestra.run:waiting for 300 2023-07-02T14:08:42.977 INFO:tasks.ceph.osd.0:Stopped 2023-07-02T14:08:42.977 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-07-02T14:08:42.978 INFO:teuthology.orchestra.run:waiting for 300 2023-07-02T14:08:43.050 INFO:tasks.ceph.osd.1:Stopped 2023-07-02T14:08:43.051 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-07-02T14:08:43.051 INFO:teuthology.orchestra.run:waiting for 300 2023-07-02T14:08:43.103 INFO:tasks.ceph.osd.2:Stopped 2023-07-02T14:08:43.103 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-07-02T14:08:43.104 INFO:teuthology.orchestra.run:waiting for 300 2023-07-02T14:08:43.149 INFO:tasks.ceph.osd.3:Stopped 2023-07-02T14:08:43.150 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-07-02T14:08:43.150 INFO:teuthology.orchestra.run:waiting for 300 2023-07-02T14:08:43.208 INFO:tasks.ceph.osd.4:Stopped 2023-07-02T14:08:43.209 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-07-02T14:08:43.209 INFO:teuthology.orchestra.run:waiting for 300 2023-07-02T14:08:43.259 INFO:tasks.ceph.osd.5:Stopped 2023-07-02T14:08:43.259 INFO:teuthology.misc:Shutting down mgr daemons... 2023-07-02T14:08:43.260 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-07-02T14:08:43.260 INFO:teuthology.orchestra.run:waiting for 300 2023-07-02T14:08:43.283 INFO:tasks.ceph.mgr.x:Stopped 2023-07-02T14:08:43.283 INFO:teuthology.misc:Shutting down mon daemons... 2023-07-02T14:08:43.284 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-07-02T14:08:43.284 INFO:teuthology.orchestra.run:waiting for 300 2023-07-02T14:08:43.294 INFO:tasks.ceph.mon.a:Stopped 2023-07-02T14:08:43.295 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-07-02T14:08:43.295 INFO:teuthology.orchestra.run:waiting for 300 2023-07-02T14:08:43.344 INFO:tasks.ceph.mon.c:Stopped 2023-07-02T14:08:43.345 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-07-02T14:08:43.345 INFO:teuthology.orchestra.run:waiting for 300 2023-07-02T14:08:43.356 INFO:tasks.ceph.mon.b:Stopped 2023-07-02T14:08:43.356 INFO:tasks.ceph:Checking cluster log for badness... 2023-07-02T14:08:43.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | head -n 1 2023-07-02T14:08:43.415 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi016.front.sepia.ceph.com 2023-07-02T14:08:43.416 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-07-02T14:08:43.741 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi016.front.sepia.ceph.com 2023-07-02T14:08:43.741 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-07-02T14:08:43.838 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi016.front.sepia.ceph.com 2023-07-02T14:08:43.839 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-07-02T14:08:43.916 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi022.front.sepia.ceph.com 2023-07-02T14:08:43.917 DEBUG:teuthology.orchestra.run.smithi022:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-07-02T14:08:44.057 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi022.front.sepia.ceph.com 2023-07-02T14:08:44.057 DEBUG:teuthology.orchestra.run.smithi022:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-07-02T14:08:44.139 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi022.front.sepia.ceph.com 2023-07-02T14:08:44.139 DEBUG:teuthology.orchestra.run.smithi022:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-07-02T14:08:44.213 INFO:tasks.ceph:Archiving mon data... 2023-07-02T14:08:44.214 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366/data/mon.a.tgz 2023-07-02T14:08:44.215 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2023-07-02T14:08:44.232 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.2yvVYQmKpC 2023-07-02T14:08:44.232 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.2yvVYQmKpC 2023-07-02T14:08:44.553 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.2yvVYQmKpC 2023-07-02T14:08:44.635 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.2yvVYQmKpC is 1MB 2023-07-02T14:08:44.738 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.2yvVYQmKpC 2023-07-02T14:08:44.753 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366/data/mon.c.tgz 2023-07-02T14:08:44.753 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2023-07-02T14:08:44.806 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.4EBYLlMuzZ 2023-07-02T14:08:44.806 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.4EBYLlMuzZ 2023-07-02T14:08:45.125 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.4EBYLlMuzZ 2023-07-02T14:08:45.207 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.4EBYLlMuzZ is 1MB 2023-07-02T14:08:45.282 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.4EBYLlMuzZ 2023-07-02T14:08:45.297 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366/data/mon.b.tgz 2023-07-02T14:08:45.297 DEBUG:teuthology.orchestra.run.smithi022:> mktemp 2023-07-02T14:08:45.313 INFO:teuthology.orchestra.run.smithi022.stdout:/tmp/tmp.pWGN6ZpsOu 2023-07-02T14:08:45.314 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.pWGN6ZpsOu 2023-07-02T14:08:45.646 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0666 /tmp/tmp.pWGN6ZpsOu 2023-07-02T14:08:45.728 DEBUG:teuthology.orchestra.remote:smithi022:/tmp/tmp.pWGN6ZpsOu is 1MB 2023-07-02T14:08:45.800 DEBUG:teuthology.orchestra.run.smithi022:> rm -fr /tmp/tmp.pWGN6ZpsOu 2023-07-02T14:08:45.815 INFO:tasks.ceph:Cleaning ceph cluster... 2023-07-02T14:08:45.815 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-07-02T14:08:45.817 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-07-02T14:08:45.856 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-07-02T14:08:45.886 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-07-02T14:08:45.886 DEBUG:teuthology.orchestra.run.smithi016:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-07-02T14:08:45.888 DEBUG:teuthology.orchestra.run.smithi022:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-07-02T14:08:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-07-02T14:08:45.920 DEBUG:teuthology.orchestra.run.smithi163:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-07-02T14:08:45.944 INFO:teuthology.orchestra.run.smithi022.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-07-02T14:08:45.954 INFO:teuthology.orchestra.run.smithi163.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-07-02T14:08:45.960 INFO:tasks.ceph:Archiving crash dumps... 2023-07-02T14:08:45.962 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366/remote/smithi016/crash 2023-07-02T14:08:45.963 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-07-02T14:08:45.989 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366/remote/smithi022/crash 2023-07-02T14:08:45.990 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-07-02T14:08:46.015 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366/remote/smithi163/crash 2023-07-02T14:08:46.016 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-07-02T14:08:46.041 INFO:tasks.ceph:Compressing logs... 2023-07-02T14:08:46.042 DEBUG:teuthology.orchestra.run.smithi016:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-02T14:08:46.044 DEBUG:teuthology.orchestra.run.smithi022:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-02T14:08:46.059 DEBUG:teuthology.orchestra.run.smithi163:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-02T14:08:59.281 INFO:tasks.ceph:Archiving logs... 2023-07-02T14:08:59.282 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366/remote/smithi016/log 2023-07-02T14:08:59.283 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-02T14:09:00.579 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/log/ceph to /home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366/remote/smithi022/log 2023-07-02T14:09:00.580 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-02T14:09:01.252 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/log/ceph to /home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366/remote/smithi163/log 2023-07-02T14:09:01.253 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-02T14:09:01.365 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-07-02T14:09:01.377 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-07-02T14:09:01.378 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-02T14:09:01.380 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-02T14:09:01.383 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-02T14:09:01.519 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-02T14:09:01.532 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-02T14:09:01.533 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-02T14:09:01.536 DEBUG:teuthology.orchestra.run.smithi022:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-02T14:09:01.539 DEBUG:teuthology.orchestra.run.smithi163:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-02T14:09:01.548 INFO:teuthology.orchestra.run.smithi016.stderr:bash: ntpq: command not found 2023-07-02T14:09:01.552 INFO:teuthology.orchestra.run.smithi016.stdout:210 Number of sources = 4 2023-07-02T14:09:01.552 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-02T14:09:01.553 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2023-07-02T14:09:01.553 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +122us[ +122us] +/- 76ms 2023-07-02T14:09:01.554 INFO:teuthology.orchestra.run.smithi016.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -343us[ -310us] +/- 67ms 2023-07-02T14:09:01.554 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 +292us[ +292us] +/- 83ms 2023-07-02T14:09:01.554 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T14:09:01.555 INFO:teuthology.orchestra.run.smithi163.stderr:bash: ntpq: command not found 2023-07-02T14:09:01.556 INFO:teuthology.orchestra.run.smithi022.stderr:bash: ntpq: command not found 2023-07-02T14:09:01.558 INFO:teuthology.orchestra.run.smithi022.stdout:210 Number of sources = 4 2023-07-02T14:09:01.558 INFO:teuthology.orchestra.run.smithi022.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-02T14:09:01.559 INFO:teuthology.orchestra.run.smithi022.stdout:=============================================================================== 2023-07-02T14:09:01.559 INFO:teuthology.orchestra.run.smithi022.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 +125us[ +105us] +/- 76ms 2023-07-02T14:09:01.559 INFO:teuthology.orchestra.run.smithi022.stdout:^* hv02.front.sepia.ceph.com 3 6 377 21 -351us[ -371us] +/- 67ms 2023-07-02T14:09:01.560 INFO:teuthology.orchestra.run.smithi022.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 +300us[ +280us] +/- 83ms 2023-07-02T14:09:01.560 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T14:09:01.562 INFO:teuthology.orchestra.run.smithi163.stdout:210 Number of sources = 4 2023-07-02T14:09:01.562 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-02T14:09:01.562 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2023-07-02T14:09:01.563 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +151us[ +151us] +/- 76ms 2023-07-02T14:09:01.563 INFO:teuthology.orchestra.run.smithi163.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -377us[ -371us] +/- 67ms 2023-07-02T14:09:01.563 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 +273us[ +273us] +/- 83ms 2023-07-02T14:09:01.564 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-02T14:09:01.564 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-02T14:09:01.579 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-02T14:09:01.580 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-02T14:09:01.616 DEBUG:teuthology.orchestra.run.smithi016:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-02T14:09:01.662 DEBUG:teuthology.orchestra.run.smithi022:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-02T14:09:01.771 DEBUG:teuthology.orchestra.run.smithi163:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-02T14:09:01.867 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-02T14:09:01.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T14:09:01.894 DEBUG:teuthology.orchestra.run.smithi022:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-02T14:09:01.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T14:09:01.926 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-02T14:09:01.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T14:09:01.955 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-02T14:09:01.964 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-02T14:09:02.011 INFO:teuthology.task.internal:Duration was 1265.966110 seconds 2023-07-02T14:09:02.012 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-02T14:09:02.061 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-02T14:09:02.061 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-02T14:09:02.063 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-02T14:09:02.065 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-02T14:09:02.108 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-02T14:09:02.109 INFO:teuthology.orchestra.run.smithi022.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-02T14:09:02.110 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-02T14:09:02.510 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-02T14:09:02.511 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2023-07-02T14:09:02.511 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-02T14:09:02.531 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi022.front.sepia.ceph.com 2023-07-02T14:09:02.531 DEBUG:teuthology.orchestra.run.smithi022:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-02T14:09:02.551 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com 2023-07-02T14:09:02.551 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-02T14:09:02.570 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-02T14:09:02.570 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-02T14:09:02.573 DEBUG:teuthology.orchestra.run.smithi022:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-02T14:09:02.593 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-02T14:09:02.648 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-02T14:09:02.648 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-02T14:09:02.650 DEBUG:teuthology.orchestra.run.smithi022:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-02T14:09:02.676 DEBUG:teuthology.orchestra.run.smithi163:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-02T14:09:02.765 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-02T14:09:02.776 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-02T14:09:02.776 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-02T14:09:02.801 DEBUG:teuthology.orchestra.run.smithi022:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-02T14:09:02.826 DEBUG:teuthology.orchestra.run.smithi163:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-02T14:09:02.850 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-02T14:09:02.864 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-02T14:09:02.866 DEBUG:teuthology.orchestra.run.smithi022:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-02T14:09:02.868 DEBUG:teuthology.orchestra.run.smithi163:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-02T14:09:02.889 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2023-07-02T14:09:02.890 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern = core 2023-07-02T14:09:02.913 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = core 2023-07-02T14:09:02.926 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-02T14:09:02.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T14:09:02.957 DEBUG:teuthology.orchestra.run.smithi022:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-02T14:09:02.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T14:09:02.971 DEBUG:teuthology.orchestra.run.smithi163:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-02T14:09:02.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T14:09:02.985 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-02T14:09:02.997 INFO:teuthology.task.internal:Transferring archived files... 2023-07-02T14:09:02.998 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366/remote/smithi016 2023-07-02T14:09:02.999 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-02T14:09:03.030 DEBUG:teuthology.misc:Transferring archived files from smithi022:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366/remote/smithi022 2023-07-02T14:09:03.030 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-02T14:09:03.064 DEBUG:teuthology.misc:Transferring archived files from smithi163:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366/remote/smithi163 2023-07-02T14:09:03.064 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-02T14:09:03.095 INFO:teuthology.task.internal:Removing archive directory... 2023-07-02T14:09:03.095 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-02T14:09:03.097 DEBUG:teuthology.orchestra.run.smithi022:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-02T14:09:03.101 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-02T14:09:03.145 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-02T14:09:03.156 INFO:teuthology.task.internal:Not uploading archives. 2023-07-02T14:09:03.156 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-02T14:09:03.189 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-02T14:09:03.189 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-02T14:09:03.191 DEBUG:teuthology.orchestra.run.smithi022:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-02T14:09:03.193 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-02T14:09:03.203 INFO:teuthology.orchestra.run.smithi016.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 2 14:09 /home/ubuntu/cephtest 2023-07-02T14:09:03.205 INFO:teuthology.orchestra.run.smithi022.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 2 14:09 /home/ubuntu/cephtest 2023-07-02T14:09:03.207 INFO:teuthology.orchestra.run.smithi163.stdout: 262176 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jul 2 14:09 /home/ubuntu/cephtest 2023-07-02T14:09:03.207 INFO:teuthology.orchestra.run.smithi163.stdout: 393313 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Jul 2 14:01 /home/ubuntu/cephtest/mnt.0 2023-07-02T14:09:03.207 INFO:teuthology.orchestra.run.smithi163.stdout: 393314 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Jul 2 14:02 /home/ubuntu/cephtest/mnt.0/client.0 2023-07-02T14:09:03.208 INFO:teuthology.orchestra.run.smithi163.stdout: 528635 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Jul 2 14:07 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-07-02T14:09:03.208 INFO:teuthology.orchestra.run.smithi163.stdout: 528654 399752 -rw-r--r-- 1 ubuntu ubuntu 409339209 Jul 2 14:04 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap18.diff 2023-07-02T14:09:03.208 INFO:teuthology.orchestra.run.smithi163.stdout: 528641 983048 -rw-r--r-- 1 ubuntu ubuntu 1006633512 Jul 2 14:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap5.diff 2023-07-02T14:09:03.208 INFO:teuthology.orchestra.run.smithi163.stdout: 528670 4100 -rw-r--r-- 1 ubuntu ubuntu 4195436 Jul 2 14:05 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap34.diff 2023-07-02T14:09:03.208 INFO:teuthology.orchestra.run.smithi163.stdout: 528685 262148 -rw-r--r-- 1 ubuntu ubuntu 268435636 Jul 2 14:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap49.diff 2023-07-02T14:09:03.209 INFO:teuthology.orchestra.run.smithi163.stdout: 528639 68 -rw-r--r-- 1 ubuntu ubuntu 66666 Jul 2 14:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap3.diff 2023-07-02T14:09:03.209 INFO:teuthology.orchestra.run.smithi163.stdout: 528661 69892 -rw-r--r-- 1 ubuntu ubuntu 71566478 Jul 2 14:04 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap25.diff 2023-07-02T14:09:03.209 INFO:teuthology.orchestra.run.smithi163.stdout: 528664 140292 -rw-r--r-- 1 ubuntu ubuntu 143658288 Jul 2 14:04 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap28.diff 2023-07-02T14:09:03.209 INFO:teuthology.orchestra.run.smithi163.stdout: 528651 4100 -rw-r--r-- 1 ubuntu ubuntu 4195436 Jul 2 14:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap15.diff 2023-07-02T14:09:03.209 INFO:teuthology.orchestra.run.smithi163.stdout: 528677 4100 -rw-r--r-- 1 ubuntu ubuntu 4195436 Jul 2 14:05 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap41.diff 2023-07-02T14:09:03.210 INFO:teuthology.orchestra.run.smithi163.stdout: 528640 68 -rw-r--r-- 1 ubuntu ubuntu 66666 Jul 2 14:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap4.diff 2023-07-02T14:09:03.210 INFO:teuthology.orchestra.run.smithi163.stdout: 528683 4100 -rw-r--r-- 1 ubuntu ubuntu 4195436 Jul 2 14:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap47.diff 2023-07-02T14:09:03.210 INFO:teuthology.orchestra.run.smithi163.stdout: 528667 1086088 -rw-r--r-- 1 ubuntu ubuntu 1112147613 Jul 2 14:04 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap31.diff 2023-07-02T14:09:03.210 INFO:teuthology.orchestra.run.smithi163.stdout: 528701 1050952 -rw-r--r-- 1 ubuntu ubuntu 1076168332 Jul 2 14:07 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap65.diff 2023-07-02T14:09:03.210 INFO:teuthology.orchestra.run.smithi163.stdout: 528649 4 -rw-r--r-- 1 ubuntu ubuntu 44 Jul 2 14:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap13.diff 2023-07-02T14:09:03.211 INFO:teuthology.orchestra.run.smithi163.stdout: 528700 262148 -rw-r--r-- 1 ubuntu ubuntu 268435636 Jul 2 14:07 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap64.diff 2023-07-02T14:09:03.211 INFO:teuthology.orchestra.run.smithi163.stdout: 528674 32772 -rw-r--r-- 1 ubuntu ubuntu 33554493 Jul 2 14:05 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap38.diff 2023-07-02T14:09:03.211 INFO:teuthology.orchestra.run.smithi163.stdout: 528680 1096008 -rw-r--r-- 1 ubuntu ubuntu 1122306781 Jul 2 14:05 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap44.diff 2023-07-02T14:09:03.211 INFO:teuthology.orchestra.run.smithi163.stdout: 528673 4 -rw-r--r-- 1 ubuntu ubuntu 44 Jul 2 14:05 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap37.diff 2023-07-02T14:09:03.211 INFO:teuthology.orchestra.run.smithi163.stdout: 528679 196612 -rw-r--r-- 1 ubuntu ubuntu 201326738 Jul 2 14:05 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap43.diff 2023-07-02T14:09:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: 528644 4 -rw-r--r-- 1 ubuntu ubuntu 42 Jul 2 14:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap8.diff 2023-07-02T14:09:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: 528650 32772 -rw-r--r-- 1 ubuntu ubuntu 33554493 Jul 2 14:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap14.diff 2023-07-02T14:09:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: 528652 8196 -rw-r--r-- 1 ubuntu ubuntu 8389740 Jul 2 14:04 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap16.diff 2023-07-02T14:09:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: 528663 4 -rw-r--r-- 1 ubuntu ubuntu 44 Jul 2 14:04 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap27.diff 2023-07-02T14:09:03.212 INFO:teuthology.orchestra.run.smithi163.stdout: 528681 72452 -rw-r--r-- 1 ubuntu ubuntu 74187935 Jul 2 14:05 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap45.diff 2023-07-02T14:09:03.213 INFO:teuthology.orchestra.run.smithi163.stdout: 528638 32772 -rw-r--r-- 1 ubuntu ubuntu 33554491 Jul 2 14:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap2.diff 2023-07-02T14:09:03.213 INFO:teuthology.orchestra.run.smithi163.stdout: 528660 1179656 -rw-r--r-- 1 ubuntu ubuntu 1207960208 Jul 2 14:04 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap24.diff 2023-07-02T14:09:03.213 INFO:teuthology.orchestra.run.smithi163.stdout: 528672 1086088 -rw-r--r-- 1 ubuntu ubuntu 1112147613 Jul 2 14:05 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap36.diff 2023-07-02T14:09:03.213 INFO:teuthology.orchestra.run.smithi163.stdout: 528669 32772 -rw-r--r-- 1 ubuntu ubuntu 33554493 Jul 2 14:05 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap33.diff 2023-07-02T14:09:03.213 INFO:teuthology.orchestra.run.smithi163.stdout: 528636 168572 -rw-rw-r-- 1 ubuntu ubuntu 172616875 Jan 21 2020 /home/ubuntu/cephtest/mnt.0/client.0/tmp/linux-5.4.tar.gz 2023-07-02T14:09:03.214 INFO:teuthology.orchestra.run.smithi163.stdout: 528692 229380 -rw-r--r-- 1 ubuntu ubuntu 234881187 Jul 2 14:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap56.diff 2023-07-02T14:09:03.214 INFO:teuthology.orchestra.run.smithi163.stdout: 528643 65540 -rw-r--r-- 1 ubuntu ubuntu 67108940 Jul 2 14:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap7.diff 2023-07-02T14:09:03.214 INFO:teuthology.orchestra.run.smithi163.stdout: 528666 98308 -rw-r--r-- 1 ubuntu ubuntu 100663391 Jul 2 14:04 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap30.diff 2023-07-02T14:09:03.214 INFO:teuthology.orchestra.run.smithi163.stdout: 528678 4 -rw-r--r-- 1 ubuntu ubuntu 44 Jul 2 14:05 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap42.diff 2023-07-02T14:09:03.214 INFO:teuthology.orchestra.run.smithi163.stdout: 528653 1015816 -rw-r--r-- 1 ubuntu ubuntu 1040187963 Jul 2 14:04 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap17.diff 2023-07-02T14:09:03.214 INFO:teuthology.orchestra.run.smithi163.stdout: 528646 8196 -rw-r--r-- 1 ubuntu ubuntu 8389739 Jul 2 14:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap10.diff 2023-07-02T14:09:03.215 INFO:teuthology.orchestra.run.smithi163.stdout: 528699 46084 -rw-r--r-- 1 ubuntu ubuntu 47189245 Jul 2 14:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap63.diff 2023-07-02T14:09:03.215 INFO:teuthology.orchestra.run.smithi163.stdout: 528637 88384 -rw-r--r-- 1 ubuntu ubuntu 90502427 Jul 2 14:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap1.diff 2023-07-02T14:09:03.215 INFO:teuthology.orchestra.run.smithi163.stdout: 528698 40964 -rw-r--r-- 1 ubuntu ubuntu 41945277 Jul 2 14:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap62.diff 2023-07-02T14:09:03.215 INFO:teuthology.orchestra.run.smithi163.stdout: 528695 39684 -rw-r--r-- 1 ubuntu ubuntu 40633486 Jul 2 14:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap59.diff 2023-07-02T14:09:03.215 INFO:teuthology.orchestra.run.smithi163.stdout: 528682 32772 -rw-r--r-- 1 ubuntu ubuntu 33554493 Jul 2 14:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap46.diff 2023-07-02T14:09:03.216 INFO:teuthology.orchestra.run.smithi163.stdout: 528697 4 -rw-r--r-- 1 ubuntu ubuntu 44 Jul 2 14:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap61.diff 2023-07-02T14:09:03.216 INFO:teuthology.orchestra.run.smithi163.stdout: 528645 4100 -rw-r--r-- 1 ubuntu ubuntu 4195434 Jul 2 14:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap9.diff 2023-07-02T14:09:03.216 INFO:teuthology.orchestra.run.smithi163.stdout: 528668 4 -rw-r--r-- 1 ubuntu ubuntu 44 Jul 2 14:05 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap32.diff 2023-07-02T14:09:03.216 INFO:teuthology.orchestra.run.smithi163.stdout: 528687 74820 -rw-r--r-- 1 ubuntu ubuntu 76613855 Jul 2 14:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap51.diff 2023-07-02T14:09:03.216 INFO:teuthology.orchestra.run.smithi163.stdout: 657917 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Jul 2 14:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a-mnt 2023-07-02T14:09:03.217 INFO:teuthology.orchestra.run.smithi163.stdout: 528691 197636 -rw-r--r-- 1 ubuntu ubuntu 202376402 Jul 2 14:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap55.diff 2023-07-02T14:09:03.217 INFO:teuthology.orchestra.run.smithi163.stdout: 528657 8196 -rw-r--r-- 1 ubuntu ubuntu 8390828 Jul 2 14:04 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap21.diff 2023-07-02T14:09:03.217 INFO:teuthology.orchestra.run.smithi163.stdout: 528693 430084 -rw-r--r-- 1 ubuntu ubuntu 440403273 Jul 2 14:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap57.diff 2023-07-02T14:09:03.217 INFO:teuthology.orchestra.run.smithi163.stdout: 528686 1048584 -rw-r--r-- 1 ubuntu ubuntu 1073742412 Jul 2 14:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap50.diff 2023-07-02T14:09:03.217 INFO:teuthology.orchestra.run.smithi163.stdout: 528658 4100 -rw-r--r-- 1 ubuntu ubuntu 4195436 Jul 2 14:04 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap22.diff 2023-07-02T14:09:03.218 INFO:teuthology.orchestra.run.smithi163.stdout: 528694 778120 -rw-r--r-- 1 ubuntu ubuntu 796790387 Jul 2 14:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap58.diff 2023-07-02T14:09:03.218 INFO:teuthology.orchestra.run.smithi163.stdout: 528655 32772 -rw-r--r-- 1 ubuntu ubuntu 33554493 Jul 2 14:04 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap19.diff 2023-07-02T14:09:03.218 INFO:teuthology.orchestra.run.smithi163.stdout: 528642 213256 -rw-r--r-- 1 ubuntu ubuntu 218368289 Jul 2 14:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap6.diff 2023-07-02T14:09:03.218 INFO:teuthology.orchestra.run.smithi163.stdout: 528659 32772 -rw-r--r-- 1 ubuntu ubuntu 33554493 Jul 2 14:04 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap23.diff 2023-07-02T14:09:03.218 INFO:teuthology.orchestra.run.smithi163.stdout: 528648 951688 -rw-r--r-- 1 ubuntu ubuntu 974521945 Jul 2 14:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap12.diff 2023-07-02T14:09:03.219 INFO:teuthology.orchestra.run.smithi163.stdout: 528656 4 -rw-r--r-- 1 ubuntu ubuntu 44 Jul 2 14:04 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap20.diff 2023-07-02T14:09:03.219 INFO:teuthology.orchestra.run.smithi163.stdout: 528689 4 -rw-r--r-- 1 ubuntu ubuntu 44 Jul 2 14:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap53.diff 2023-07-02T14:09:03.219 INFO:teuthology.orchestra.run.smithi163.stdout: 528671 173060 -rw-r--r-- 1 ubuntu ubuntu 177211649 Jul 2 14:05 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap35.diff 2023-07-02T14:09:03.219 INFO:teuthology.orchestra.run.smithi163.stdout: 528684 4 -rw-r--r-- 1 ubuntu ubuntu 44 Jul 2 14:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap48.diff 2023-07-02T14:09:03.219 INFO:teuthology.orchestra.run.smithi163.stdout: 528665 102404 -rw-r--r-- 1 ubuntu ubuntu 104858783 Jul 2 14:04 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap29.diff 2023-07-02T14:09:03.219 INFO:teuthology.orchestra.run.smithi163.stdout: 528690 8196 -rw-r--r-- 1 ubuntu ubuntu 8389757 Jul 2 14:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap54.diff 2023-07-02T14:09:03.220 INFO:teuthology.orchestra.run.smithi163.stdout: 528688 32772 -rw-r--r-- 1 ubuntu ubuntu 33554493 Jul 2 14:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap52.diff 2023-07-02T14:09:03.220 INFO:teuthology.orchestra.run.smithi163.stdout: 528696 4 -rw-r--r-- 1 ubuntu ubuntu 44 Jul 2 14:06 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap60.diff 2023-07-02T14:09:03.220 INFO:teuthology.orchestra.run.smithi163.stdout: 528647 360456 -rw-r--r-- 1 ubuntu ubuntu 369098983 Jul 2 14:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap11.diff 2023-07-02T14:09:03.220 INFO:teuthology.orchestra.run.smithi163.stdout: 528676 8196 -rw-r--r-- 1 ubuntu ubuntu 8390828 Jul 2 14:05 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap40.diff 2023-07-02T14:09:03.220 INFO:teuthology.orchestra.run.smithi163.stdout: 528662 32772 -rw-r--r-- 1 ubuntu ubuntu 33554493 Jul 2 14:04 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap26.diff 2023-07-02T14:09:03.221 INFO:teuthology.orchestra.run.smithi163.stdout: 528675 4 -rw-r--r-- 1 ubuntu ubuntu 44 Jul 2 14:05 /home/ubuntu/cephtest/mnt.0/client.0/tmp/b3fbb028-e267-4d25-a750-93ba18a2316a@snap39.diff 2023-07-02T14:09:03.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-02T14:09:03.222 INFO:teuthology.orchestra.run.smithi163.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-07-02T14:09:03.222 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/run_tasks.py", line 192, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi163 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-07-02T14:09:03.229 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-02T14:09:03.241 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-02T14:09:03.290 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi163 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-07-02T14:09:03.290 INFO:teuthology.nuke:Checking targets against current locks 2023-07-02T14:09:03.320 DEBUG:teuthology.nuke:shortname: smithi016 2023-07-02T14:09:03.321 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-02T14:09:03.350 DEBUG:teuthology.nuke:shortname: smithi022 2023-07-02T14:09:03.351 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-02T14:09:03.354 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-02 13:36:35.227001', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKAKQdK43fJ67ZR0UFSSd/x9YyqkWqm7r0EvMSQqiplezjVhkRix4wwgl7eot1veKhyPZmpw4xJNdOF0LHYyJls='} 2023-07-02T14:09:03.377 DEBUG:teuthology.nuke:shortname: smithi163 2023-07-02T14:09:03.378 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-02T14:09:03.380 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi022.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-02 13:36:35.229451', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAJsgII6kEeeCVaBI90i/iehAOPSQrDC/rPxYXSo44UgV74Crx/QB4JOdm21MI75Pv9b9R4Vt8I2AzdTkMoETQA='} 2023-07-02T14:09:03.405 INFO:teuthology.orchestra.console:Power off smithi016 2023-07-02T14:09:03.407 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-02T14:09:03.420 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-07-02_13:35:41-rbd-pacific-distro-default-smithi/7324366', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-02 13:36:35.228345', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlG8ZmXbNG1kKGtt6qrh3TXluF3kt7nELxpD4U3sZvPxNmfk+i05F0KvaqgWwitlcawW5gZzWAqd4jeTIW8MCTKW7jA7Ipp/X6WjdLPEzd3Gl4HBzLACoV13x2/ep16BUlgQgBoDBrf7ye4s5Ud/a2ARRZFB0s5q2Pa+oyjTJjv6SeNkn5rqA5zIwm9+g6bkl7MZtNxHBcm5SPIZpFLJ7P9edWZh5820pOd5LjZEEutwGxUxTUf6TipiRoZYW+oFLTGHrgIFbiUPsZDhhRj9/mJZmhF8JcvkRRTNUQl2Mgl7meT556UkqLDtwbkAaIrOA58XqnvIaTByEaC4Bw1CpuntlENFsJR40BhLj9GgOa7AmDOkurkVwG3yQldNdDQ8EVtX7xRlXDRqHKy6EYh5TMB/zYrNmFiSGFIEKhDzgaRQCAngZ72NVQ3EAlzdqBqQhZlkn3zR71CXe5rzHm4TaZCKIfcFc6UZ+bWSXYs6ob5sTB3YRD788HYsqsuCikcTE='} 2023-07-02T14:09:03.429 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-02T14:09:03.455 INFO:teuthology.orchestra.console:Power off smithi022 2023-07-02T14:09:03.455 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-02T14:09:03.480 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-02T14:09:03.503 INFO:teuthology.orchestra.console:Power off smithi163 2023-07-02T14:09:03.504 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-02T14:09:03.526 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-02T14:09:07.471 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-02T14:09:07.515 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-02T14:09:07.548 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-02T14:09:11.587 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-02T14:09:11.627 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-02T14:09:11.659 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-02T14:09:11.803 INFO:teuthology.orchestra.console:Power off for smithi016 completed 2023-07-02T14:09:11.839 INFO:teuthology.orchestra.console:Power off for smithi022 completed 2023-07-02T14:09:11.871 INFO:teuthology.orchestra.console:Power off for smithi163 completed 2023-07-02T14:09:11.977 INFO:teuthology.run:Summary data: description: rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous} duration: 1265.9661104679108 failure_reason: 'Command failed (workunit test rbd/diff_continuous.sh) on smithi163 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cbd88633a9c19779ad4b2e367bdc17f83b3a065d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/diff_continuous.sh''' flavor: default owner: scheduled_dis@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=2e2789d5d9d14447b6d04f6a14556966 status: fail success: false 2023-07-02T14:09:11.977 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-02T14:09:12.053 INFO:teuthology.run:FAIL